Yani Shi
City University of Hong Kong yanishi2@cityu.edu.hk
Chuan-Hoo Tan
City University of Hong Kong ch.tan@cityu.edu.hk
Choon Ling Sia
City University of Hong Kong iscl@cityu.edu.hk
Abstract：Online product recommendation is viewed as an important way to facilitate online shoppers’ decision-making. A recommendation is deemed to be successful if it persuades a consumer to accept the suggested product. To this end, it is of great importance to understand a consumer’s cognitive process during online shopping in the presence of product recommendation agent. This study employs think-aloud method and performs verbal protocol analysis to examine the impact of third-party (other consumers and/or domain experts) recommendations on
consumer decision-making process. The findings of the impact of third-party recommendation on this cognitive process contribute to both recommendation agent literature and practice.
Keywords: Recommendation Agent, Think Aloud, Verbal Protocol, Consumer Review, Expert Review
Introduction
Online retailers often employ recommendation agents (RAs) to provide online product recommendations, with the objective of not only to support a consumer’s decision-making but also to positively influence product choice(Xiao & Benbasat, 2007). However, product recommendations are not always well accepted and sometime consumers may even react negatively(Fitzsimons & Lehmann, 2004). A likely cause of it is the lack of sufficient reasons provided to justify for the recommendations; thus they are not perceived to be persuasively enough. A way to address this is to anchor on the third-party product reviews, such as the expert reviews and the consumer reviews, because such information is often utilized by consumers to make their purchase decisions (Y. Chen & Xie, 2005; Y. B. Chen & Xie, 2008). Expert reviews refer to the product evaluations made by product domain experts while the consumer reviews are written by product customers reporting their post-consumption assessment. Since third-party generated information serves as an important source of reference for consumer decision-making, how the additions of such information in RA’s recommendation influence consumer decision-making process?
To answer this question, we need to understand the cognitive processes during which a consumerassesses the product recommendations. Most previous studies on recommendation agents mainly leverage on variables such as the decision effort (e.g., decision time, extent of product search), the product evaluation (e.g., the consumer’s ratings of the product alternatives
59
recommended) (Todd & Benbasat, 1987; Xiao & Benbasat, 2007), among others. The understanding of the entire cognitive thinking process remains lacking. Elaborately, the intervening process of online product recommendation is still considered as an unexplored
“black box”. For this reason, this research employs the “process tracing” method so as to gain a more wholesome understanding of how consumers go about making decisions when recommendations, which are based on third-party reviews.
Literature Review
Recommendations that are made based on expert reviews are termed as expert recommendation while those that are provided based on consumer reviews are, likewise, labeled as consumer recommendation. An expert recommendation,which is based on product evaluation or assessment by domain experts (Gershoff, Mukherjee, & Mukhopadhyay, 2003), is typically made with product attributes as the focus of justification (Lee, Kim, & Chan-Olmsted, 2010).
Comparatively, a consumer recommendation, which is based on post-product consumption evaluation, is typically offered with the experience as the focus of justification (Senecal & Nantel, 2003). Indeed, a consumer review reflects a consumer’s usage situations and evaluation of the product performance from a user’s perspective (Bickart & Schindler, 2001). It is highlighted that when consumers have difficulties in evaluating product attribute information and expert review information, online consumer review could be a good information source (Alba & Hutchinson, 1987).
There is a lack of studies on embedding product reviews into RA, thus we are unable to reliably predict their consequential impact on consumer decision-making. Indeed, our understanding of the works on product reviews reveals two gaps that also serve as opportunities for research. First, previous studies mainly focus on the impacts of expert/consumer review on product sales or consumers’ product judgments(Bickart & Schindler, 2001; Park, Lee, & Han, 2007), and there are few studies that investigate leveraging expert reviews and consumer reviews as recommendation source in online product recommendation. Second, some studies suggested that consumer review is likely to be more credible while other studies believe that expert review is of higher expertise(Brown & Reingen, 1987; Reddy, Swaminathan, & Motley, 1998).
Research Methodology
In order to understand consumer decision-making process, we adopted the “think aloud” method to capture their verbal protocols in a lab experiment setting. “Think aloud” method was developed based on protocol analysis techniques coined by Ericsson and Simon (Ericsson &
Simon, 1993; Van Someren, Barnard, & Sandberg, 1994). Think aloud protocols involve subjects thinking aloud as they are performing tasks. Subjects are asked to speak out whatever they are thinking, doing and feeling during the process of task performing (Cooper-Martin, 1993; Todd &
Benbasat, 1992).
In our experiment, 5-6 subjects were randomly assigned to every treatment group. Four types of recommendations were presented for four treatment groups: (1) arecommendation with only product information, (2) a recommendation with product information and consumer reviews, (3)
60
a recommendation with product information and expert reviews, and (4) a recommendation with product information and both consumer reviews and expert reviews. Before conducting the experiment tasks, subjects were primed with a shopping scenario in which they were going to purchase digital products for themselves. They were required to choose one product from each online store. There were four online stores selling cell phones, digital cameras, laptops and mp3 players, respectively. The sequence of purchasing tasks was randomly assigned. These four product categories were selected because they frequently appear in online shopping websites. A product recommendation was presented when a subject has short-listed several options in the shopping car before making a final decision. Real product data was used in the experiment.
Product information, consumer reviews and expert reviews were gathered from an IT portal website using a self-developed web crawler.
Verbal protocols were recorded concurrently with the experimental session via Morae Recorder (software by TechSmith). Following the instruction of administering the “think aloud” method, if there is a period of silence (more than 10 seconds typically), the experimenter would prompt the subject to verbalize, which is the only intervention during think aloud session (Todd & Benbasat, 1987; Van Someren, et al., 1994). The prompt should be neutral and unobtrusive by simply asking the subject to speak out what he/she is doing and thinking during the task session.
Data Analysis and Research Findings
Subjects’ individual characteristics, such as age, gender, computer experience and online shopping experience, were controlled through randomization. Further checks indicate that there is no significant differences among subjects in all four treatments in terms of age (F=1.08, p>0.1), computer experience (F=3.018, p>0.05), and online shopping experience (F=0.081, p>0.1).
There was no significant difference across treatment groups in terms of gender ratio, based on the Kruskal-Wallis test (χ2=4.313, p>0.1).
Manipulation check was conducted to ensure that our manipulation of recommendation source was successful in the experiment.Recommendation source manipulation was checked by asking the subjects whether they thought the recommendation source was from consumer reviews, expert reviews or both. All subjects correctly answered the recommendation sources. As a result, our manipulation of the two independent variables was successful.
An initial data analysis, scanning, was conducted based on the verbal protocols collected in the experiment. As the most straightforward method, scanning “examines the verbal protocols for (frequently anecdotal) information that assists in interpreting quantitative observations” (Todd &
Benbasat, 1992).
When a recommendation was presented without third-party product reviews, we found that the subjects did not pay much attention to the recommendation. For instance, a subject doubted by articulating “why the website recommended this product to me?” and closed the recommendation window immediately. This confirms that recommendation is not persuasive if no supportive information or explanation is provided (Gregor & Benbasat, 1999).
Verbal protocols provide rich information on the effect of expert recommendation. For example, a subject switched his choice to an expert recommended laptop. By comprehensively describing
61
the product features, the expert recommendation provided detailed information on the recommended laptop and comparisons with other products. After reading the recommendation, a subject could speak out several advantages of the recommended product and found that the recommended product would perform better than other alternatives in his consideration set. As a result, he chose the recommended product finally. However, some subjects may not be persuaded by the expert recommendation because they were not interested in the advantages mentioned in the expert review, which suggests the importance of content matching in reviews with consumer’s preference
Compared to expert recommendation, consumer recommendation performed worse. Consumer reviews were perceived as subjective rather than objective. Some subjects said that, although other consumers’ reviews were honest, they were mainly based on personal preference and there was a lack of supportive evidences provided with review comments.
The subjects, who received recommendations based on both consumer review and expert review, mentioned that “there were so many reviews but I did not want to read all of them”. The recommendation was a little bit late, which could be one of the reasons that the subject did not want to spend much time in reading the reviews carefully.It is suggested that too much information may not always benefit consumer decision-making. We also found that subjects had already formed specific preferences and the recommended products often did not match their preferences. It was more difficult for them to change preferences so that recommended products were less likely to be added into consideration set, no matter it was consumer recommendation or expert recommendation.
Discussion and Conclusion
There are limitations in this study which serve as suggestions for future research. First, in this experiment, the recommendation was presented toward the later period of consumer decision-making process. The situations may be different if the recommendation is provided at other stages of consumer decision making process, which deserves further investigation. Second, because of the nature of think aloud method, this study may have limited generalizability as the sample size is small. However, the findings could be leveraged in a large-scaled study to examine its generalizability.
This study contributes to the existing literature by examining the effectiveness of third-party sources in online product recommendation. It is found that expert recommendations play a more important role in influencing consumer decision-making than consumer recommendation. It is also suggested to practitioners that they should attach importance to the selection of reviews with product recommendation. The study also implies the value of understanding the cognitive process via think aloud method and verbal protocol analysis. The use of protocol analysis to understand the principles of consumer information processing could contribute to the future development of online RAs.
References
Alba, J. W., & Hutchinson, J. W. (1987). Dimensions of Consumer Expertise. Journal of Consumer Research,
62 13(4), 411-454.
Bickart, B., & Schindler, R. M. (2001). Internet Forums as Influential Sources of Consumer Information.
Journal of Interactive Marketing, 15(3), 31-40. doi: 10.1002/dir.1014
Bowman, D., & Narayandas, D. (2001). Managing Customer-Initiated Contacts With Manufacturers: The Impact on Share of Category Requirements and Word-of-Mouth Behavior. Journal of Marketing Research, 38(3), 281-297.
Brown, J. J., & Reingen, P. H. (1987). Social Ties and Word-of-Mouth Referral Behavior. The Journal of Consumer Research, 14(3), 350-362.
Chen, Y., & Xie, J. (2005). Third-Party Product Review and Firm Marketing Strategy. Marketing Science, 24(2), 218-240.
Chen, Y. B., & Xie, J. H. (2008). Online Consumer Review: Word-of-Mouth as a News Element of Marketing Communication Mix. Management Science, 54(3), 477-491. doi: Doi 10.1287/Mnsc.1070.0810 Cooper-Martin, E. (1993). An Extension of the Congruence Hypothesis: The Effects of Real Products,
Branching Format, Similarity, and Involvement. Psychology and Marketing, 10(5), 433-447.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal Reports as Data: the MIT Press.
Fitzsimons, G. J., & Lehmann, D. R. (2004). Reactance to Recommendations: When Unsolicited Advice Yields Contrary Responses. Marketing Science, 23(1), 82-94. doi: Doi 10.1287/Mksc.1030.0033
Gershoff, A. D., Mukherjee, A., & Mukhopadhyay, A. (2003). Consumer Acceptance of Online Agent Advice:
Extremity and Positivity Effects. Journal of Consumer Psychology, 13(1-2), 161-170. doi: Doi:
10.1207/s15327663jcp13-1&2_14
Gregor, S., & Benbasat, I. (1999). Explanations from Intelligent Systems: Theoretical Foundations and Implications for Practice. Mis Quarterly, 23(4), 497-530.
Lee, C., Kim, J., & Chan-Olmsted, S. M. (2010). Branded Product Information Search on the Web: The Role of Brand Trust and Credibility of Online Information Sources. Journal of Marketing Communications, First published on: 05 October 2010 (iFirst).
Park, D. H., Lee, J., & Han, I. (2007). The Effect of Online Consumer Reviews on Consumer Purchasing Intention: The Moderating Role of Involvement. International Journal of Electronic Commerce, 11(4), 125-148. doi: Doi 10.2753/Jec1086-4415110405
Reddy, S. K., Swaminathan, V., & Motley, C. M. (1998). Exploring the determinants of Broadway show success. Journal of Marketing Research, 35(3), 370-383.
Senecal, S., & Nantel, J. (2003). Online Influence of Relevant Others: A Framework. Paper presented at the Sixth International Conference on Electronic Commerce Research, Dallas, TX.
Todd, P., & Benbasat, I. (1987). Process Tracing Methods in Decision Support Systems Research: Exploring The Black Box. MIS Quarterly, 493-512.
Todd, P., & Benbasat, I. (1992). The Use of Information in Decision Making: an Experimental Investigation of the Impact of Computer-Based Decision Aids. MIS Quarterly, 373-393.
Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes: Citeseer.
Xiao, B., & Benbasat, I. (2007). E-commerce product recommendation agents: Use, characteristics, and impact.
Mis Quarterly, 31(1), 137-209.
63