System Usability and Effectiveness

As part of this work, we wanted to test the usability of the CrowdSMILE.

Questionnaires and Statistical analyses are instruments for gauging user’s response and feedback about the system.

CSUQ [35] is a well-known and standardized questionnaire that is used to evaluate user satisfaction and get feedback from users about a computer system, especially with small sample sizes. The CSUQ survey contains 19 questions which can be assigned into 4 categories.

Table 5-2: The questions of CSUQ 1 Overall, I am satisfied with how easy it is to use this system.

2 It was simple to use this system.

3 I can effectively learn using this system.

4 I am able to learn quickly using this system.

5 I am able to efficiently learn using this system.

6 I feel comfortable using this system.

7 It was easy to learn to use this system.

8 I believe I became productive quickly using this system.

9 The system gives error messages that clearly tell me how to fix problems.

10 Whenever I make a mistake using the system, I recover easily and quickly.

11 The information provided with this system is clear 12 It is easy to find the information I needed.

13 The information provided for the system is easy to understand.

14 The information is effective in helping me complete the tasks and scenarios.

15 The organization of information on the system screens is clear.

16 The interface of this system is pleasant.

17 I like using the interface of this system.

18 This system has all the functions and capabilities I expect it to have.

19 Overall, I am satisfied with this system.

Table 5-3: CSUQ Categories and corresponding items in Questionnaire

Category Items

Overall Satisfaction Items 1 through 19

System Usefulness Items 1 through 8

Information Quality Items 9 through 15

Interface Quality Items 16 through 18

66 5.3.1. Experiment Procedures

A group of 18 users used and tested the system over the period of 2 weeks. Before the start of the test, each user was given an introduction to the system along with an explanation about its purpose. At the end of the test period, they were asked to complete a Computer System Usability Questionnaire (CSUQ) as defined in [35]. The survey contained 19 questions which each being a statement that the user could rate on a Likert scale of 1 – 7, where 1 is strongly disagree and 7 is strongly agree. In addition to CSUQ questions, we asked two more free text questions that required the users to add a note about their negative experience with the system (if any) and a note about their positive experience with the system (if any). A copy of the actual questionnaire is provided in the appendix.

5.3.2. System State at start of Test

The system was populated with 1 Mission and 5 Quests. Each Quest had at 3 Learning Objects. The users were asked to use the system to explore and learn about the Mission available. They were further asked to contribute to the system by providing learning content for each of the Quests. They were also encouraged to create their own Missions.

Table 5-4: State at Start of User Test

Missions 1

Quests 5

Learning Objects 15

5.3.3. System State at end of Test

Table 5-5: State at End of User Test

Missions 4

Quests 12

Learning Objects 54

As can be seen, the users were actively using the system over the test period.

Figure 5-6: Difference in content in system (at the start of user test and at the end)

0

67 5.3.4. Survey Results

The results of the CSUQ are shown below, where N is the number of responses; AVG is the average value, DEV is the standard deviation value, MED is the median, MAX is the maximum, and MIN is the minimum.

Table 5-6: CSUQ Survey Results by Category and User Average

Category AVG DEV MED MAX MIN

Overall Satisfaction 5.56 0.26 6 7 2

System Usefulness 5.63 0.07 6 7 4

Information Quality 5.50 0.24 5.5 7 2 Interface Quality 5.39 0.25 6 7 2

Table 5-7: CSUQ Survey Results by Category and User Average as % Score out of 7 Category Avg as %

Overall Satisfaction 79%

System Usefulness 80%

Information Quality 79%

Interface Quality 77%

Table 5-8: CSUQ Survey Results by Question

N AVG DEV MED MAX MIN

Whenever I make a mistake using the system, I

recover easily and quickly 18 5.06 1.35 5 7 2

The information provided with this system is clear (documentation, online help, on screen messages, etc.)

18 5.39 1.21 5.5 7 3 It is easy to find the information I needed 18 5.44 1.07 5.5 7 3 The information provided for the system is

easy to understand 18 5.78 1.08 6 7 3

68 The information is effective in helping me

complete the tasks and scenarios 18 6.06 0.70 6 7 5

The organization of information on the system

screens is clear 18 5.83 0.83 6 7 4 Table 5-9: CSUQ Survey Results by User

Overall

Table 5-10: Survey – Some Negative aspects of the system Not Enough Missions

Some phones aren't compatible with the app Finding contents and uploading it

The interface can be more friendly and informative on how to use the system, maybe pop up messages and better fonts.

I really don't like the way that the system post information on my Facebook timeline. I want to use the system without link it to my social network or limit its access to my timeline.

Once inside the tasks, the UI is a little cluttered. You have to scroll down to find info The system was good...but uses a lot of data bandwidth at times

Too few missions available

Table 5-11: Survey – Some Positive aspects of the system Easy to use. Very interesting and useful idea.

69 Easy to use and user friendly.

Easy to use. If good and enough content uploaded it saves a lot of time searching for info about the POIs. It can be used as a tourist app with Missions (or Categories) about

sightseeing, food, fun, nightlife and other stuff, so people can just take their phones out and see what POIs are around them. Over all it's pretty cool and useful.

The interface is clean, clear and very easy to use. It just needs some polishing in terms of graphical aspects

The fact it's already integrated with Facebook because, you see which missions your friends have joined making the user curious to try those missions too.

The speed of information display for the located sites, given the device's GPS works correctly The overall assessment of the users was positive. All 4 areas scored over 77%. There were no overall negative scores. The results also show that the users could easily use the system and that they found the system to be useful. There were a few users who were not happy with the interface given that it did not work on their phone. This was due to their phone not meeting the minimum requirements for the MAR environment and was expected. The lowest question score received was 4.94 regarding error messages. This was due to the fact that when the mobile device was not able to communicate with the server, it would fail with a non-descriptive error and not provide the user with any clue about the error. This can easily be fixed to support a better user experience.

70