Chapter 4 Implementation and Evaluation
4.3. Evaluation
4.3.8. Analysis of the Results
Navigation Performance
The participants were asked to complete four major tasks during this test. These
four tasks are composed of many sub-tasks, two sub-tasks, 2f and 3d, of Task 2 and
Task 3 are very important for this study. These two sub-tasks were designed
specifically to investigate the participant’s navigation performance from a starting
point to a destination. During the execution of these sub-tasks, the participants can use
any information provided by the mobile interface to orient themselves and navigate to
the destination. From the findings of these two sub-tasks, the usability of TopoNav
can be evaluated.
In order to evaluate the efficiency and effectiveness of TopoNav and Google
Maps, the durations of each participant spent on the navigation from the starting point
to the destination are listed in Table 4.2. The directional arrow in the column “Order”
indicates the sequence of geolocation apps used by the participant. For example, P1
used TopoNav before Google Maps.
All participants have successfully navigated to the destination using both
geolocation apps. The number of participants who completed the navigation with
TopoNav faster is only slightly more than the number of those used Google Maps.
Based on the data collected, three charts (Figure 4.5) are made to visualize the
comparison of the participants’ navigation performance. The first chart (a) presents
the average completion time of the participants, the second chart (b) presents the
minimum completion time, and the third chart (c) presents the maximum completion
time.
Table 4.2 The time each participant used to navigate from the starting point to the destination.
Figure 4.5 Performance of the navigation sub-tasks 2f and 3d.
Comparing with Google Maps, the average completion time of the participants
with TopoNav is shorter. This indicates that the participants performed better with
TopoNav, regardless of the sequence of the two geolocation apps used. The minimum
completion time of TopoNav is slightly shorter than Google Maps. However, in the
case of maximum completion time, the participants performed better with Google
Maps. The fact that all participants have previous experience with Google Maps
before this test should be considered as a possible reason for it. As while all of them
were familiar with the mobile interface of Google Maps, they used TopoNav for the
first time. From the results of these two navigation sub-tasks, the participants
performed much more efficiently and slightly more effectively with TopoNav than
Google Maps.
During the execution of these two navigation sub-tasks, sometimes the
participants had to stop and reorient themselves. They referred to the indicator (blue
dot) on the map to find out their current location in the surrounding environment.
Although individuals may have different levels of navigation and orientation
capabilities, the number of navigation stops can be used as an indicator for the
efficiency of the geolocation apps. The number of stops presented in Table 4.3
excludes those ones triggered by the researcher and only includes the stops made by
the participants themselves. The participants made fewer stops with TopoNav than
Google Maps. A chart (Figure 4.6) was made to visually compare the performance of
TopoNav and Google Maps.
Table 4.3 The number of stops each participant had during the navigation tasks.
Figure 4.6 The average number of stops each participant had during navigation.
Task 1 - Selection of Destination
During this task the participants were required to use the geolocation apps to find
the appointed destination. The first sub-task (1a) was designed to evaluate the
effectiveness of the briefing session. The participants were asked to rate themselves
about their familiarity with the mobile interfaces. Most participants think that they
had enough time to learn about TopoNav; except one participant (P11) stated that he
needed more time to get used to it. As all participants had previous experiences with
Google Maps, no one stated any problems with it.
Sub-task 1b aimed to assess the ability of participants to find the destination
using the interfaces. This sub-task was also intended to investigate the participants’
familiarity with the functions of the geolocation apps. TopoNav only supports text
input, but with Google Maps the participants were given the freedom to choose
between voice or text input. 4 of the participants (P3, P7, P10, and P15) tried to search
using voice input, only two of them had correctly completed the procedure. The voice
recognition feature could not accurately recognize the name that the participants were
trying to input. Most of them had to speak more than once loudly to get it right. In
general, the participants did not have any significant difficulties with text input,
except few minor problems. All of them considered it as an easy sub-task to deliver.
With both interfaces, they could easily type in the appointed text in the textfield,
initiate the search, and select the destination from the suggested list. One of the
participants had difficulties while she had to initiate the search with TopoNav. Instead
of selecting one of the destinations from the suggested list, she typed the name of the
destination and tapped the “Route” button on the keyboard. One possible reason
might be that she was appointed to use TopoNav after Google Maps. As with Google
Maps the participant can initiate search by tapping the “Search” button on keyboard,
but TopoNav does not support this function.
Table 4.4 presents these results in terms of answering the question of whether the
participant has successfully executed the sub-task or not. S refers to succeed, F refers
to failed, and P refers to partially succeed. TN stands for TopoNav, and GM stands for
Google Maps.
Table 4.4 The result for all sub-tasks in Task 1.
Based on the data collected in Table 4.4, the effectiveness chart of Task 1 is
shown in Figure 4.7. There were no obvious performance differences between
TopoNav and Google Maps in sub-task 1a. However, in sub-task 1b, it can be
observed that the performance of TopoNav is better than Google Maps. The results
demonstrate that, unless the voice recognition feature is provided with high accuracy,
voice input is not mandatory for this kind of system.
Figure 4.7 Effectiveness Chart of Task 1.
Task 2 - Route Preview
During this task the participants were asked to preview the calculated route. At the
beginning of this task (sub-task 2a and 2b), they were required to estimate the
distance to the selected destination and the time required to reach it. As both TopoNav
and Google Maps automatically calculated the time and distance, all of the
participants were able to complete these two sub-tasks successfully.
Sub-task 2c aimed to inspect the ability of the participants to recognize their
surroundings on the mobile interfaces. During this sub-task, most participants using
TopoNav were able to recognize the environment on the interface. However, five of
the participants using Google Maps encountered problems. In order to carry out this
The main problem of Google Maps was that most landmarks were hidden under the
map scale for showing the complete route on screen. In addition to this, unlike
TopoNav, no street photos were provided at the starting point with Google Maps.
Sub-task 2d aimed to investigate the effectiveness of the route preview feature
provided by the interfaces. During this sub-task, half of the participants, especially
those who had to use TopoNav before Google Maps, complained about the lacking of
support to street photos in Google Maps. In the case of TopoNav, some suggested that
instead of viewing from the middle of the road, the viewing angle should be from the
sidewalks. Few times the street photos taken at night time were used, and the
participants could not recognize it (because the test was executed during the daytime)
Sub-task 2e aimed to investigate the ability of the participants to identify the
correct direction to the destination. They were asked to walk toward the destination
for a short distance.4 out of 16 participants with Google Maps moved in the wrong
direction (P3, P7, P12, and P15). Three (P3, P7 and P15) headed toward the opposite
direction initially, but corrected their heading after walked for a while. The other
participant (P12) was initially headed to the correct direction, but ended up wrong
after 30 meters. On the other hand, only one participant with TopoNav had problems
while carrying out this sub-task. One stated that “The street photo of the starting point
is very helpful.” Based on these results, the effectiveness of providing street photos
with correct viewing angle at the starting point is impressive.
Table 4.5 presents these results in terms of answering the question of whether the
participant has successfully executed the sub-task or not. S refers to succeed, F refers
to failed, and P refers to partially succeed.
Table 4.5 The result for all sub-tasks in Task 2.
Figure 4.8 Effectiveness Chart of Task 2.
Based on the data collected in Table 4.5, the effectiveness chart of Task 2 is
shown in Figure 4.8. It is observed that there are significant performance differences
between TopoNav and Google Maps in recognizing the environment on the map
interfaces (sub-task 2c), obtaining a cognitive map to understand the calculated route
(sub-task 2d), and identifying the correct heading (sub-task 2e). These results indicate
that topological map with street photos performs better for orientation tasks.
Task 3 - Identification of Turning Points and Travel Decisions
During this task the participants were asked to identify the turning points along
the route by observing the landmarks around their current position. They were also
asked to estimate the remaining distance and time required to reach the destination.
The first sub-task (sub-task 3a) aimed to investigate the ability of participants to
identify the turning points along the route towards their destination. Most of the
participants with TopoNav could easily identify obvious landmarks near the turning
points without any stops. However, as it comes to Google Maps, the participants
generally needed to check the map more often to confirm they were on the correct
path. Five of the participants (P1, P5, P9, P10, and P13) missed the turning points on
their way.
Sub-task 3b and 3c aimed to inspect the ability of the participants to estimate the
remaining distance and time required to reach the destination. As both TopoNav and
Google Maps automatically calculate the distance and time, no participant had
noticeable problem during this sub-task.
Table 4.6 presents these results in terms of answering the question of whether the
participant has successfully executed the sub-task or not. S refers to succeed, F refers
to failed, and P refers to partially succeed.
Table 4.6 The result for all sub-tasks in Task 3.
Figure 4.9 Effectiveness Chart of Task 3.
Based on the data collected in Table 4.6, the effectiveness chart of Task 3 is
shown in Figure 4.9. An obvious performance difference of the participants was
observed between TopoNav and Google Maps on identifying turning points along the
calculated route to the destination. The participants could recognize the landmarks
around the turning points much more easily using TopoNav.
Task 4 - Destination Identification
During this task the participants were asked to recognize the destination and
confirm that they have reached it by referring to the provided geospatial information.
At the beginning of this task, sub-task 4a aimed to evaluate the availability of an
obvious destination indication on the mobile interfaces. Both TopoNav and Google
Maps displayed the destination on the map with large icon; all participants could
successfully confirm their arrival to the destination.
Sub-task 4b aimed to investigate the quality of the destination street photo. 5 out
of the 16 participants with TopoNav complained about the viewing angle of the street
photo. 3 stated that instead of taking the photo from the middle of the road, the photo
should be taken from the sidewalks where the pedestrians were at. 2 participants
stated that they could not easily recognize the destination from the photo in general.
As the photos of destination were not provided under navigation mode of Google
Maps, the participants had to recall the street photo provided previously when they
had to select a destination. 12 participants complained that they had a hard time to
recall the photo from their memory.
Table 4.7 presents these results in terms of answering the question of whether the
participant has successfully executed the sub-task or not. S refers to succeed, F refers
to failed, and P refers to partially succeed.
Table 4.7 The result for all sub-tasks in Task 4.
Figure 4.10 Effectiveness Chart of Task 4.
Based on the data collected in Table 4.7, the effectiveness chart of Task 4 is
shown in Figure 4.10. All participants with both TopoNav and Google Maps had
successfully completed sub-task 4a (verification of the destination arrival on the map).
However, an obvious performance difference of the participants was observed
between TopoNav and Google Maps on the effectiveness of the destination photo. The
main reason for it can be that TopoNav provides a destination photo with correct
viewing angle under its navigation mode.
Chapter 5
Discussion and Conclusion
5.1. Lesson Learned
For a geolocation app like TopoNav, the users typically need to be aware of the
landmarks along the route, especially around the decision points (starting points,
turning points, and destination). The mobile interface should fill up the gap between
the reality and the map for its users. It should present important landmarks with
correct street sizes and patterns, and also accurately display the user’s current
location.
The small size screen is one of the main limitations of geolocation apps. Unlike
traditional paper maps, it is nearly impossible to display both detail and overview
maps at the same time on a small screen. The usability of such a system is reduced as
its users often need to repeatedly zoom in and out to obtain all the geospatial
information needed. It should also be noted that the users usually use the geolocaiton
apps on the go and could be distracted easily.
With a geolocation app, the users often need to figure out their current location
first. In order to orient themselves, they observe their surroundings and try to align it
with the map. The users usually look for a colored dot that represents their current
location, the sizes and patterns of the street, and important landmarks on the map.
Although 3D map is considered to be helpful, 2D map is preferred by the users.
Comparing with the 3D feature, the street photos and landmarks are more important
for the navigation and orientation tasks.
There are typically three main orientation and navigation tasks for the users of
geolocation apps to perform. The first task, initial orientation, is to answer the
question of “Where am I?” In this step, the users often want to know what is around
them and decide on where to go. After selected a destination, the users then perform
the second task of navigation. They often want to know “How far is it?”, “How long
does it take for me to reach it?”, and “Am I moving in the correct direction?” The
third task is the identification of destination. When the users think that they have
arrived at the destination, they often want to know “Is it the correct destination that I
look for?”
From the results of the usability test, the users were satisfied with the efficiency
and effectiveness of TopoNav over Google Maps. During the usability test, the users
found the topological map, landmarks, and street photos of TopoNav helpful.
However, most of them preferred the rotating maps of Google Maps over the north-up
map of TopoNav. Improvements should be made by including the rotating map, street
photos with the viewing angle from sidewalks, and the cross street before the next
turning point.
5.2. Conclusion and Future Works
Street name labels and landmarks icons overlapping problem was one of the
major issues observed during the usability test of TopoNav. This reduces the usability
of the map interface dramatically. The solution for this problem should be carefully
investigated. It can be applying the techniques such as automatic label and icon
placement. In order to understand the user’s needs further, the influence of the
researcher to the participants during the usability test should be reduced. The goal is
to make the participants unaware of the researcher’s presence. This allows them to
behave more naturally when performing the tasks. One of the possible solutions for it
could be an internet-based observation system. With such a system, the participants
can be remotely observed without any influences from the researchers. Instructions
and the geolocation app for the test will be provided to the participants at the
beginning of the experiment. During the test, the participants can use the geolocation
app to execute the test tasks on their own, and a more realistic result can be obtained.
Due to the limitations of this research, iterations of design and development with
the users were not conducted. If the users were involved in the earlier design stages
such as the validation of mock-ups and prototypes of TopoNav, a solution with better
usability can be obtained. However, according the results from the usability test, the
use of topological map with street photos in geolocation apps was proven to have high
potential. More researches should be done to investigate how to apply these solutions
in a more user-friendly way and under different context of use.
Bibliography
[1] Getting, I.A.: ‘Perspective/navigation-The Global Positioning System’, Spectrum, IEEE, 1993, 30, (12), pp. 36-38
[2] Chittaro, L.: ‘Visualizing Information on Mobile Devices’, Computer, 2006, 39, (3), pp. 40-45
[3] Kristoffersen, S., and Ljungberg, F.: ‘“Making place” to make IT work:
empirical explorations of HCI for mobile CSCW’. Proc. Proceedings of the international ACM SIGGROUP conference on Supporting group work, Phoenix, Arizona, USA, 1999, pp. 276-285
[4] Krüger, A., Aslan, I., and Zimmer, H.: ‘The Effects of Mobile Pedestrian Navigation Systems on the Concurrent Acquisition of Route and Survey Knowledge’, in Brewster, S., and Dunlop, M. (Eds.): ‘Mobile
Human-Computer Interaction - MobileHCI 2004’ (Springer Berlin Heidelberg, 2004), pp. 446-450
[5] Goodman, J., Gray, P.D.G., Khammampad, K., and Brewster, S.: ‘Using landmarks to support older people in navigation’, Lecture Notes in Computer Science, 2004, 3160, pp. 38-48
[6] Dudek, G., and Jenkin, M.: ‘Inertial Sensors, GPS, and Odometry’, in Siciliano, B., and Khatib, O. (Eds.): ‘Springer Handbook of Robotics’ (Springer Berlin Heidelberg, 2008), pp. 477-490
[7] Beeharee, A.K., and Steed, A.: ‘A natural wayfinding exploiting photos in pedestrian navigation systems’. Proc. Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, Helsinki, Finland, 2006, pp. 81-88
[8] Hile, H., Vedantham, R., Cuellar, G., Liu, A., Gelfand, N., Grzeszczuk, R., and Borriello, G.: ‘Landmark-based pedestrian navigation from collections of geotagged photos’. Proc. Proceedings of the 7th International Conference on Mobile and Ubiquitous Multimedia, Umeå, Sweden, 2008, pp. 145–152 [9] Chen, B., Neubert, B., Ofek, E., Deussen, O., and Cohen, M.F.: ‘Integrated
videos and maps for driving directions’. Proc. Proceedings of the 22nd annual ACM symposium on User interface software and technology, Victoria, BC, Canada, 2009, pp. 223–232
[10] Holtzblatt, K., and Beyer, H.: ‘Making customer-centered design work for teams’, Commun. ACM, 1993, 36, (10), pp. 92-103
[11] Gould, J.D., and Lewis, C.: ‘Designing for usability: key principles and what designers think’, Commun. ACM, 1985, 28, (3), pp. 300-311
[12] Vredenburg, K., Mao, J.-Y., Smith, P.W., and Carey, T.: ‘A survey of
user-centered design practice’. Proc. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Minneapolis, Minnesota, USA, 2002, pp. 471-478
[13] Norman, D.A., and Draper, S.W.: ‘User Centered System Design; New
[13] Norman, D.A., and Draper, S.W.: ‘User Centered System Design; New