IX. Findings
9.4 Implications of this Research
91
different problem categories as indicated by the number of hits. Other factors reported by writers such as the simpler format of HE1 and the use of abstract concepts in HE2 did not appear to make one HE more usable than the other.
Results seem to point to the benefits of more specific heuristics. If a potential problem is mentioned directly, in specific detail, and possibly mentioned repeatedly, then it is more likely to result in a hit. The higher inter-rater agreement for HE2
suggests this is likely for all evaluators. However, greater specificity entails some risks.
First, it may result in more false positives, or the mistaken identification of a usability problem. By suggesting to an evaluator that an element is part of good document design, the evaluator may be easily led to accepting that the absence of such an element is a usability problem. A possible solution to this issue may be the design of an HE which combines general and specific heuristics in a comprehensive list, along with reminders to evaluators to consider specific recommendations in light of user and task requirements, and to make space available after each heuristic category for more open-ended responses. Such an HE may enhance the advantages of specified heuristics while minimizing the drawbacks they entail, thus making the documentation more usable for overseas users.
9.4 Implications of this Research
The results of my research have implications for Taiwanese product designers and usability practitioners, as well as implications for usability theory. Taiwanese producers cannot assume that their products will be equally usable overseas. It has been shown that U.S. users make more errors and become more dissatisfied with a product than Taiwanese users. Markets also vary according to the mix of usability attributes they value. While generally, effectiveness is important for most users in all the markets examined, the importance of efficiency is higher for cultures that value the achievement of individual goals such as the U.S. In contrast, preference for
satisfaction is more related to non-task oriented goals such as status and face-saving needs, and maintaining group relationships, such as the Philippines. Product designers need to keep in mind the differences in each market and focus their product design to better match the requirements of their users.
Differences in attitudes towards usability due to culture also affect the attitudes of usability practitioners involved in identifying and rating the severity of usability problems. Such usability evaluators need to be aware of the possibility of error due to
the cultural difference between them and the user they are testing. In order to minimize the possibility of error, when rating the severity of a usability problem, they need to rely on data-driven descriptions of the usability problems, and minimize the need to exercise judgment on the severity of a usability problem. There is also the possibility that, if they become used to basing severity ratings on ISO9241-defined usability attributes, effectiveness, efficiency and user satisfaction, they will achieve greater accuracy in their ratings. Finally, when using usability evaluation methods such as heuristic evaluations, usability evaluators need to select heuristic evaluations that are detailed in their criteria of what makes a good product. In short, a reduction of the reliance on subjective judgments by usability evaluators and greater use of detailed and data-driven criteria, is required to make better usability evaluations in a cross-cultural context.
Finally, my research proposes enhancements to existing models of culture and usability. As mentioned, the model of culturability is a synthesis of existing usability models and definitions, combined with the findings of my research . The diagram below describes the important interactions between culture, usability, usability attributes, and usability measurement. My definition of usability is based on a
combination of ISO9241's definition of usability attributes - effectiveness, efficiency, and user satisfaction, with Hornbaek’s Usability Aspects Model, which describes the subjective and objective aspects of these usability attributes. Finally, this usability definition is combined with the MUSIC Model, which describes the interaction of a user with features of product design, to perform a task given a specific context of use.
This interaction culminates in a level of usability which can be measured based on Hornbaek’s adaption of the ISO9241 definition of usability.
93
Figure IX-1 Proposed Model of Culturability
9.5 Limitations of this Research
While attempts have been made to make this research as accurate and free from error as possible, there still exist factors which may influence the accuracy of my results. Firstly, sample sizes in all studies have varied and have ranged from eight technical writers in the final study into heuristic evaluations, to several hundred respondents of different nationalities in my survey on the importance of usability factors. However, in all statistics reported, the p-value has been given. This indicates the probability a result is not random and takes into account sample size. Often p-values have been in the region of highly significant, suggesting the robustness of my results.
Secondly, while the samples on which these studies are based were selected for their ability to represent the population under consideration, limitations naturally exist.
These occur in two areas - the cultural values considered in the survey on the
importance of usability attributes, and in the type of products considered. Taking the latter first, in the four studies made, a number of consumer products were examined including an MP3 player, a cell phone, and software documentation. This selection does not represent the full range of products in any way. In fact, it would be impossible to test the effect of culture on usability for a full range of products. The only option is for future studies to ensure that alternative products are tested, and in this way a picture of usability is developed that may better represent the range of products types available. Similarly, the full range of cultural values also have not been examined. However, the cultural values selected may better represent those available.
The countries selected represent Western and Eastern cultures, two major cultural regions of the world, and the cultural values considered include the results of two major global studies into culture, when typically in similar research only one set of results are examined (those of Hostede). By doing so, it is hoped that the impact of any inaccuracies in this research have been minimized and that results offer an acceptable level of robustness.
95
X. Conclusion
To conclude, this research has proposed a model of the interaction of culture and usability which bases itself on current models of culture and usability but more clearly explains the interaction of culture and usability attributes, and culture and usability measurement. Research shows that culture can affect the level of effectiveness, efficiency and satisfaction that users experience and perceive when using a product, and that also culture affects the importance users place on each of these usability attributes. Secondly, my research shows that culture affects how usability practitioners measure usability problems. However, some recommendations are made to possibly reduce the impact of culture on the accuracy of usability evaluation, thus supporting Taiwanese product designers and Taiwanese industry as whole to make the most of the opportunities presented by overseas markets.
References
1. Andre, T.S., Hartson, H.R., Belz, S.M., McCreary, F.A., "The user action framework: a reliable foundation for usability engineering support tools", International Journal of Human-Computer Studies, 54, pp. 107-136, 2001.
2. Badre, A.N., "The effects of cross cultural interface design orientation on World Wide Web user performance", GVU Tech Reports, 2000.
3. Barber, W., Badre, A., "Culturability: the merging of culture and usability", Human Factors and the Web 4th Conference, 1998.
4. Bevan, N., "The MUSIC methodology for usability measurement", 1992 SIGCHI
Conference on Human Factors in Computing Systems, pp. 123-124, Monterey, California, 1992.
5. Bevan, N., "International standards for HCI and usability", International Journal of Human-Computer Studies, 55, pp. 533-552, 2001.
6. Borgman, C.L., "The user's mental model of an information retrieval system: an
experiment on a prototype online catalog", International Journal of Man-Machine Studies, 24, pp. 47-64, 1986.
7. Bourges-Waldegg, P., Scrivener, S.A.R., "Meaning, the central issue in cross-cultural HCI design", Interacting with Computers, 9, pp. 287-309, 1998.
8. Capra, M.G., Smith-Jackson, T.L., 2005. Developing guidelines for describing usability problems, Technical Report #ACE/HCI-2005-002. Virginia Tech, Blacksburg.
9. Chau, P.Y.K., Cole, M., Massey, A.P., Montoya-Weiss, M.M., O'Keefe, R.M., "Cultural differences in the online behavior of consumers", Communications of the ACM, 45, pp.
138-143, 2002.
10. Choi, B., Jeon, Y., Kim, J., Lee, I., "A qualitative cross-national study of cultural influences on mobile data service design", Proceedings of CHI 2005 Portland, pp. 661-670, 2005.
11. Clemmensen, T., "Towards a theory of cultural usability: A comparison of ADA and CM-U theory", Lecture Notes in Computer Science, 5619, pp. 416-425, 2009.
12. Clemmensen, T., Hertzum, M., Hornbaek, K., Shi, Q., Yammiyavar, P., "Cultural cognition in usability evaluation", Interacting with Computers, 21, pp. 212-220, 2009.
13. Clemmensen, T., Roese, K., "An overview of a decade of journal publications about Culture and Human - Computer Interaction (HCI)", IFIP Advances in Information and
97
14. Cockton, G., Woolrych, A., "Sale must end: should discount methods be cleared off HCI's shelves?", interactions, 9, pp. 13 - 18, 2002.
15. Connell, I.W., Hammond, N.V., "Comparing usability evaluation principles with heuristics", INTERACT'99 - Human-Computer Interaction, pp. 621-629, 1999.
16. DeBoard, D., 2004. Heuristics to evaluate online help, Usability interface.
17. Evers, V., Day, D., "The role of culture in interface acceptance", IFIP TC13 International Conference on Human–Computer Interaction INTERACT’97, Chapman & Hall, Ltd., Sydney, Australia, pp. 260 - 267, 1997.
18. Ficarra, F.V.C., "Evaluation of multimedia components", Multimedia Computing and Systems '97 Ottawa, Ont., Canada, pp. 557-564, 1997.
19. Fillion, F.M., Boyle, C.D.B., "Important issues in hypertext documentation usability", 9th annual international conference on Systems documentation. ACM Chicago, pp. 59 - 66, Illinois, United States, 1991.
20. Fitzgerald, W., "Evaluation of multimedia components", Multimedia Computing and Systems '97 Proceedings IEEE, Ottawa, Ont., Canada, pp. 557-564, 2004.
21. Folmer, E., Bosch, J., "Architecting for usability", Journal of Systems and Software, 70, pp. 61-78, 2004.
22. Ford, G., Gelderblom, H., "The effects of culture on performance achieved through the use of human computer interaction", SAICSIT '03 Proceedings of the 2003 annual research conference of the South African institute of computer scientists and information technologists on Enablement through technology. ACM, pp. 218-230, 2003.
23. Ford, G., Kotzé, P., "Designing usable interfaces with cultural dimensions", Lecture Notes in Computer Science, 3585, pp. 713-726, 2005.
24. Ford, G., Kotzé, P., "Researching Culture and Usability — A Conceptual Model of Usability. In: McEwan, T., Benyon, D., and Gulliksen, J., (Eds.)", People and Computers XIX — The Bigger Picture, Springer, pp. 317-333, 2005.
25. Frandsen-Thorlacius, O., Hornbæk, K., Hertzum, M., Clemmensen, T., "Non-universal usability? A survey of how usability is understood by Chinese and Danish users", Conference on Human Factors in Computing Systems: Proceedings of the 27th
international conference on human factors in computing systems ACM Press, pp. 41-50, Boston, U.S.A., 2009.
26. Frøkjær, E., Hertzum, M., Hornbæk, K., "Measuring usability: Are effectiveness, efficiency, and satisfaction really correlated?", Proceedings of the ACM CHI 2000
Conference on Human Factors in Computing Systems ACM Press, pp. 345-352, New York, 2000.
27. Fu, L., Salvendy, G., Turley, L., "Effectiveness of user testing and heuristic evaluation as a function of performance classification", Behaviour & Information Technology, 21, pp.
137-143, 2002.
28. Garzotto, F., Mainetti, L., Paolini, P., "Hypermedia design, analysis, and evaluation issues", Communications of the ACM, 38, pp. 74-86, 1995.
29. Gerhardt-Powals, J., "Cognitive engineering principles for enhancing human-computer performance", International Journal of Human-Computer Interaction, 8, pp. 189 - 211, 1996.
30. Gray, W.D., Salzman, M.C., "Damaged merchandise? A review of experiments that
compare usability testing methods", Human-Computer Interaction, 13, pp. 203-261, 1998.
31. Grayling, T., "If we build it will they come? A usability test of two browser-based embedded help systems", Technical communication, 49, 2002.
32. Grice, R.A., Ridgway, L.S., "Usability and hypermedia: Toward a set of usability criteria and measures", Technical Communication, pp. 429-437, 1993.
33. Hall, E., Beyond Culture, Anchor Press, Garden City, N.Y., 1976.
34. Han, S.H., Hwan Yun, M.H., Kwahk, J., Hong, S.W., "Usability of consumer electronic products", International Journal of Industrial Ergonomics, 28, pp. 143-151, 2001.
35. Hartson, H.R., Andre, T.S., Williges, R.C., "Criteria for evaluating usability evaluation methods", International journal of human-computer interaction, 13, pp. 373-410, 2001.
36. Hassenzahl, M., "Prioritizing usability problems: data-driven and judgment-driven severity estimates", Behaviour & Information Technology, 19, pp. 29-42, 2000.
37. Hertzum, M., "Problem prioritization in usability evaluation: From severity assessments towards impact on design", International Journal of Human Computer Interaction, 21, pp.
125-146, 2006.
38. Hertzum, M., Clemmensen, T., Hornbæk, K., Kumar, J., Shi, Q., Yammiyavar, P.,
"Usability constructs: A cross-cultural study of how users and developers experience their use of information systems", Lecture Notes in Computer Science, 4559, pp. 317-326, 2007.
39. Hertzum, M., Jacobsen, N.E., "The evaluator effect: A chilling fact about usability evaluation methods", International journal of human-computer interaction, 13, pp. 421-443, 2001.
99
40. Hofstede, G., Culture's consequences: international difference in work related values, Sage, London and Beverly Hills, 1984.
41. Hofstede, G., Bond, M.H., "The Confucius connection: From cultural roots to economic growth", Organisational Dynamics, pp. 5-21, 1998.
42. Hofstede, G., Minkov, M., Vinken, H., Values Survey Module 2008 Manual, 2008.
43. Hoft, N.L., Developing a cultural model, International users interface, John Wiley &
Sons, Inc, New York, NY, USA, pp. 41 - 73, 1996.
44. Hornbæk, K., "Current practice in measuring usability: Challenges to usability studies and research", International journal of human-computer studies, 64, pp. 79-102, 2006.
45. Hornbæk, K., Law, E.L.-C., Meta-analysis of correlations among usability measures, Proceedings of the SIGCHI conference on Human factors in computing systems San Jose, California, USA, pp. 617-626, 2007.
46. ISO, 1998. ISO 9241-11, Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability. International Organization for Standardization (ISO), Geneva, Switzerland.
47. ISO, 2001. ISO 9126-1, Software Engineering - Product Quality - Part 1: Quality Model.
International Organization for Standardization (ISO), Geneva, Switzerland.
48. Jacobsen, N.E., Hertzum, M., John, B., "The evaluator effect in usability studies: problem detection and severity judgments, Proceedings of the Human Factors and Ergonomics Society", Human Factors and Ergonomics Society, Chicago, U.S.A, pp. 1336-1340, 1998.
49. John, B.E., Marks, S.J., "Tracking the effectiveness of usability evaluation methods", Behaviour & Information Technology, 16, 188-202, 1997.
50. Kantner, L., Shroyer, R., Rosenbaum, S., "Structured heuristic evaluation of online documentation", Professional Communication Conference, pp. 331-342, 2002.
51. Kuroso, M., Kashimura, K., "Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability", Conference on Human Factors in Computing Systems, ACM, pp. 292-293, Denver, Colorado, United States, 1995.
52. Kurosu, M., Kashimura, K., "Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability", Conference on Human Factors in Computing Systems Denver, pp. 292 - 293, Colorado, United States, 1995.
53. Law, E.L.-C., Hvannberg, E.T., "Complementarity and convergence of heuristic evaluation and usability tests: A case study of UNIVERSAL brokerage platform", NordiCHI 2002 Aarhus, pp. 71-80, Denmark, 2002.
54. Law, E.L.-C., Hvannberg, E.T., "Analysis of strategies for improving and estimating the effectiveness of heuristic evaluation", NordiCHI 2004 Tampere, pp. 241-250, Finland, 2004.
55. Lee, I., Choi, G.W., Kim, J., Kim, S., Lee, K., Kim, D., Han, M., Park, S.Y., An, Y.,
"Cultural dimensions for user experience: Cross-country and cross-product analysis of users’ cultural characteristics", 22nd British HCI Group Annual Conference on HCI 2008:
People and Computers XXII: Culture, Creativity, Interaction British Computer Society Liverpool, pp. 3-12, United Kingdom, 2008.
56. Lund, A.M., Measuring usability with the USE questionnaire, Usability Interface, 2001.
57. Mankoff, J., Dey, A.K., Hsieh, G., Kientz, J., Lederer, S., Ames, M., "Heuristic
evaluation of ambient displays", Human factors in computing systems, ACM Press, pp.
169-176, 2003.
58. Marcus, A.. "Cross-cultural user-experience design", Lecture Notes in Computer Science, 4045/2006, pp.16-24, 2006.
59. Marcus, A., Gould, E.W., "Crosscurrents - Cultural dimensions and global web user-interface design", Interactions, 7, pp. 32-46, 2000.
60. Mayhew, D.J., Mantei, M., A basic framework for cost-justifying usability engineering, In: Bias, R. G., and Mayhew, D. J., (Eds.), Cost Justifying Usability Harcourt Brace and Co., Boston, pp. 111-122, 1994.
61. McGee, M., Rich, A., Dumas, J., "Understanding the usability construct: User-perceived usability", Human Factors and Ergonomics Society Annual Meeting Human Factors and Ergonomics Society, pp. 907-911, 2004.
62. McSweeney, B., "Hofstede’s model of national cultural differences and their
consequences: A triumph of faith – a failure of analysis", Human relations, 55, pp. 89-118, 2002.
63. Mehlenbacher, B., "Software usability: Choosing appropriate methods for evaluating online systems and documentation", SIGDOC Conference 1993, Getting in Touch—
Staying in Touch, pp. 209-222, 1993.
64. Molich, R., Dumas, J.S., "Comparative usability evaluation (CUE-4)", Behaviour &
Information Technology, 27, pp. 263-281, 2008.
65. Molich, R., Ede, M.R., Kaasgaard, K., Karyukin, B., "Comparative usability evaluation", Behaviour & Information Technology, 23, pp. 65-74, 2004.
66. Molich, R., Nielsen, J., "Heuristic evaluation of user interfaces", ACM CHI'90 Seattle,
101
67. Molich , R., Nielsen, J., "Heuristic evaluation of user interfaces", ACM CHI'90 Seattle, pp. 249-256, WA, 1990.
68. Nantel, J., Glaser, E., "The impact of language and culture on perceived website
usability", Journal of Engineering and Technology Management, 25, pp. 112-122, 2008.
69. Nielsen, J., Usability 101.
70. Nielsen, J., "Finding usability problems through heuristic evaluation", ACM CHI'92 Monterey, pp. 676-680, CA., 1992.
71. Nielsen, J., "Reliability of severity estimates for usability problems found by heuristic evaluation", Conference on Human Factors in Computing System, pp. 676-680, Monterey, California, 1992.
72. Nielsen, J., "Usability Engineering", Academic Press, New York, 1993.
73. Novick, D.G., Ward, K., "What users say they want in documentation", 24th Annual ACM International Conference on Design of Communication, pp. 84-91, Myrtle Beach, SC, 2006.
74. O'Keefe, R.M., Cole, M., Chau, P.Y.K., Massey, A., Montoya-Weiss, M., Perry, M.,
"From the user interface to the consumer interface: results from a global experiment", International Journal of Human-Computer Studies, 53, pp. 611-628, 2000.
75. Olie, R., "The 'culture' factor in personnel and organization policies", In: A., H., and London, V. R. J., (Eds.), International Human Resource Management: An integrated approach. Sage pp. 124-143, 1995.
76. Purho, N., "Heuristic Inspections for Documentation – 10 Recommended Documentation Heuristics", Usability Interface, 2002.
77. Salminen, A., Tiitinen, P., Lyytikäinen, V., "Usability evaluation of a structured
document archive", Proceedings of the 32nd Hawaii International Conference on System Sciences, pp. 1-10, 1999.
78. Sauro, J., Lewis, J.R., "Correlations among prototypical usability metrics: evidence for the construct of usability", Conference on Human Factors in Computing Systems, ACM, pp. 1609-1618, Boston, MA, USA, 2009.
79. Schwartz, S.H., "Mapping and interpreting cultural differences around the world", In:
Vinken, H., Soeters, J., and Ester, P., (Eds.), Comparing cultures, Dimensions of culture in a comparative perspective. Brill., Leiden, The Netherlands, pp. 43-73, 2004.
80. Shackel, B., "Usability—context, framework, definition, design and evaluation", Human factors for Informatics Usability, pp. 21 - 37, 1991.
81. Shen, S.T., Woolley, M., Prior, S., "Towards culture-centered design", Interacting with Computers, 18, pp. 820-852, 2006.
82. Shi, Q., Clemmensen, T., "Communication patterns and usability problem finding in cross-cultural thinking aloud usability testing", Conference on Human Factors in Computing Systems, pp. 2811-2816, Florence, Italy, 2008.
83. Singh, N.. "Culture and the World Wide Web: a cross-cultural analysis of web sites from France, Germany and USA", American Marketing Association, 14, pp. 30-31, 2003.
84. Straub, D., Loch, K., Evaristo, R., Karahanna, E., Srite, M., "Toward a theory-based measurement of culture", Journal of Global Information Management, 10, pp. 13–23, 2002.
85. Tan, W.-s., Liu, D., Bishu, R., "Web evaluation: Heuristic evaluation vs. user testing", International journal of industrial ergonomics, 39, pp. 321-327, 2009.
86. Tractinsky, N., "Aesthetics and apparent usability: empirically assessing cultural and methodological issues", SIGCHI Conference on Human Factors in Computing Systems.
ACM Press, pp. 115–122, Atlanta, Georgia, United States, 1997.
87. Triandis, H.C., The Analysis of Subjective Culture, Wiley, New York, N.Y, 1972.
88. Trompenaars, F., Riding the Waves of Culture: Understanding Diversity in Global Business, Irwin Professional Publishing, New York, 1994.
89. van Biljon, J., Kotzé, P., "Cultural factors in a mobile phone adoption and usage model", Journal of Universal Computer Science, 14, pp. 2650-2679, 2008.
90. Vatrapu, R., Pérez-Quiñones, M., Culture and usability evaluation: The effects of culture in structured interviews, Journal of Usability Studies, 1, 156-170, 2006.
91. Virzi, R.A., "Refining the test phase of usability evaluation: How many subjects is enough?", Human Factors, 34, pp. 457-468, 1992.
92. Vöhringer-Kuhnt, T., "The influence of culture on usability", Freie Universität Berlin, Masters Thesis, 2002.
93. Walker, M.A., Fromer, J., Mestel, C., Hindle, D., "What can I say?: evaluating a spoken language interface to Email", CHI 98 Conference on Human Factors in Computing Systems. ACM, pp. 582-589, 1998.
94. Winter, S., Wagner, S., Deissenboeck , F., "A comprehensive model of usability", Lecture Notes In Computer Science, Berlin, Heidelberg, pp. 106 - 122, 2008.
95. Witold, A., Suryn , W., Khelifi , A., Rilling, J., Seffah, A., Robert, F., "Consolidating the ISO usability models", 11th International Software Quality Management Conference,
103
96. Woolrych, A., Cockton, G., "Why and when five test users aren't enough", In:
Vanderdonckt, J., Blandford, A., and Derycke, A., (Eds.), Proceedings of IHM-HCI 2001 Conference, pp. 105- 108, Cépadèus, Toulouse, France, 2001.