Chapter 4 Implementation and Evaluation
4.3. Evaluation
4.3.2. Test Tasks and Scenarios
A set of tasks was developed for the participants to perform during the evaluation
sessions. These tasks were formulated based on the general scenario of navigation
from a starting point to an unfamiliar destination. It is important for the tasks to
represent as much as possible of the uses under real world context and include the
most important parts of the user interface [21]. Furthermore, the tasks should be
structured in a way allowing the participants to compare and evaluate different user
interfaces, which means the usability of TopoNav is compared against another
geolocation app. Google Maps is selected as the geolocation app for comparison - as
TopoNav uses Google Maps API for its source of geographical data, and they both
share common characteristics such as turn-by-turn navigation
Following are the tasks developed for this evaluation:
Task 1 - Selection of Destination
Scenario 1
You are visiting an unfamiliar city for the first time. You arrived at its central railway
station, and just walked out from one of its exits. You were given the mobile interface
earlier today and tried to familiar yourself with it within 15 minutes. You have limited
amount of time available to navigate around this city and you want to start doing that
as soon as possible. You planned ahead for this trip at home and have built a bucket
list of places you want to see. You use the mobile interface to select the destination
you want to go and find the way to the destination.
Based on this scenario, the first task was divided into several sub-tasks to
evaluate the participants’ level of familiarity with the mobile interface and how the
prototype can support the users to select a destination.
The sub-tasks for both TopoNav and Google Maps are:
(a) Did you have enough time to familiar yourself with the mobile interface?
(b) Please search for the appointed destination using its name.
Task 2 - Route Preview
Scenario 2
Before you leave the start point, you want to get an overview of the calculated route.
You try to preview the step-by-step instructions with street photos on the mobile
interface. You also try to estimate how much time it takes for you to get there and how
far away it is from your current location. To make the navigation easier and prevent
yourself from getting lost, you want to ensure this estimation is accurate and you
know the look of all the major landmarks along the path.
The sub-tasks for TopoNav are:
(a) Please estimate the distance from your current location to the destination.
(b) Please estimate the time needed for you to reach the destination.
(c) Observe the information presented on the mobile interface. Can you recognize the
environment on the map?
(d) Please start preview the calculated route. Swipe to left and preview the instruction
and street photo of next step. Swipe to right and preview the instruction and street
photo of previous step. After you finished, did you obtain a clear overview of the
calculated route?
(e) Please start walking toward the destination for 30 meters and stop. Check your
position on the map; were you moving in the correct direction?
(f) Please navigate to the selected destination.
The sub-tasks for Google Maps are:
(a) Please estimate the distance from your current location to the destination.
(b) Please estimate the time needed for you to reach the destination.
(c) Observe the information presented on the mobile interface. Can you recognize the
environment on the map?
(d) Please tap the “Preview” button on the bottom bar to start preview the calculated
route. On the header, swipe to the left and preview next step on the map, swipe to the
right and preview previous step on the map. After you finished, did you obtain a clear
overview of the calculated route?
(e) Please start walking toward the destination for 30 meters and stop. Check your
position on the map; were you moving in the correct direction?
(f) Please navigate to the selected destination.
Task 3 - Identification of Turning Points and Travel Decisions
Scenario 3
On your way walking toward the selected destination, you want to ensure that you are
following the correct route. You try to identify the turning points by observing the
landmarks along the path and, if necessary, comparing them with the street photos
presented on the mobile interface. You want to check whether the landmarks exist and
visible in reality and is correctly presented in the mobile interface. As always, you try
to estimate the distance of the destination from your current location and how much
time left for you to reach the destination. You want to ensure you are moving toward
the destination in the correct direction by watching your movement on the map.
The sub-tasks for both TopoNav and Google Maps are:
(a) Please observe your surroundings and try to identify the next turning point by
landmarks.
(b) Please estimate the distance from your current location to the destination.
(c) Please estimate the time needed for you to reach the destination.
(d) Please navigate to the selected destination.
Task 4 - Destination Identification
Scenario 4
You have been navigating to the destination with the geolocation app. When you think
you have arrived at the destination, please stop and observe the surroundings.
Compare your observation with the information presented on mobile interface; is it
the destination you look for? Does the mobile interface provide all the required
information for you to identify the destination?
The sub-tasks for both TopoNav and Google Maps are:
(a) When you think that you have arrived at the destination, please stop and observe
the surroundings. Compare your observation with the information presented on
mobile map; is it the destination you look for? Does the mobile interface provide all
the required information for you to identify the destination?
(b) Compare the street image of destination the real environment; does it have the
correct viewing angle?
4.3.3. Participants
The participants of this usability test of TopoNav should represent the target
users of this geolocation app. These users are pedestrian travelers to an unfamiliar city
and they use the geolocation app to obtain the required geospatial information to
perform navigation and orientation tasks.
I started with sending out invitations to people who might be interested in
participating through the online social network Facebook. Free round trip bus tickets
to Taipei City are offered in return to those ones selected to participate. This provides
them with an opportunity to hang out with friends or family in Taipei City after the
test. 52 people accepted the invitation and they are asked to fill out the pre-selection
questionnaire.
Based on the results from pre-selection questionnaires, 16 participants are
recruited to participate in this study. Table 4.1 presents the profiles of these
participants and their experiences with different fields related to this study. 8 of the
participants are female and 8 are male. All participants owned touch-screen
smartphones and had used a geolocation app like Google Maps before. They may
have different levels of experience and knowledge with mobile devices, digital maps,
cartography, navigation, and wayfinding. All of them are unfamiliar with the selected
test area. The average age is 35.7, with a minimum of 18 and maximum of 62. This
distribution is very close to the research of comScore MobiLens on the age of
smartphone users in the United States at April 2011 (Figure 4.2). The age groups of
the participants for this test are shown in Figure 4.3.
Table 4.1 Profiles of the usability test participants.
Figure 4.2 Age of smartphone users in the United States (adapted from comScore MobiLens).
Figure 4.3 Age distribution of the usability test participants.