6. Opinion Spam and Utility of Opinions
6.2 Utility of Reviews
Determining the utility of reviews is usually formulated as a regression problem. The learned model then assigns a utility value to each review, which can be used in review ranking. In this area of research, the ground truth data used for both training and testing are usually the user-helpfulness feedbacks given to each review, which as we discussed above are provided for each review at many review aggregation sites.
So unlike fake review detection, the training and testing data here is not an issue.
Researchers have used many types of data features for model building [31, 49, 110]. Example features include review length, review ratings (the number of stars), counts of some specific POS tags, opinion words, tf-idf weighting scores, wh-words, product attribute mentions, product brands, comparison with product specifications, and comparison with editorial reviews, and many more. Subjectivity classification is also applied in [31]. In [57], Liu et al. formulated the problem slightly differently, as a binary classification problem. Instead of using the original helpfulness feedbacks as the classification target or dependent variable, they performed manual annotation based on whether a review comments on many product attributes/features or not.
Finally, we note again that review utility regression/classification and review spam detections are different concepts. Not-helpful or low quality reviews are not necessarily fake reviews or spam, and helpful reviews may not be non-spam. A user often determines whether a review is helpful or not based on whether the review expresses opinions on many attributes/features of the product. A spammer can satisfy this requirement by carefully crafting a review that is just like a normal helpful review. Using the number of helpful feedbacks to define review quality is also problematic because user feedbacks can be spammed too. Feedback spam is a sub-problem of click fraud in search advertising, where a person or robot clicks on some online advertisements to give the impression of real customer clicks. Here, a robot or a human spammer can also click on helpful feedback button to increase the helpfulness of a review.
Another important point is that a low quality review is still a valid review and should not be discarded, but a spam review is untruthful and/or malicious and should be removed once detected.
7. Conclusions
This chapter gave an introduction to sentiment analysis and subjectivity (or opinion mining). Due to many challenging research problems and a wide variety of practical applications, it has been a very active research area in recent years. In fact, it has spread from computer science to management science [e.g., 2, 11, 17, 32, 37, 58, 74]. This chapter first presented an abstract model of sentiment analysis, which formulates the problem and provides a common framework to unify different research directions. It then discussed the most widely studied topic of sentiment and subjectivity classification, which determines whether a document or sentence is opinionated, and if so whether it carries a positive or negative opinion.
We then described feature-based sentiment analysis which exploits the full power of the abstract model.
After that we discussed the problem of analyzing comparative and superlative sentences. Such sentences represent a different type of evaluation from direct opinions which have been the focus of the current research. The topic of opinion search or retrieval was introduced as well, as a parallel to the general Web search. Last but not least, we discussed opinion spam, which is increasingly becoming an important issue as more and more people are relying on opinions on the Web for decision making. This gives more and more incentive for spam. There is still no effective technique to combat opinion spam.
Finally, we conclude the chapter by saying that all the sentiment analysis tasks are very challenging. Our understanding and knowledge of the problem and its solution are still limited. The main reason is that it is a natural language processing task, and natural language processing has no easy problems. Another reason may be due to our popular ways of doing research. We probably relied too much on machine learning algorithms. Some of the most effective machine learning algorithms, e.g., support vector machines and conditional random fields, produce no human understandable results such that although they may achieve improved accuracy, we know little about how and why apart from some superficial knowledge gained in the manual feature engineering process. However, that being said, we have indeed made significant progresses over the past few years. This is evident from the large number of start-up companies that offer sentiment analysis or opinion mining services. There is a real and huge need in the industry for such services because every company wants to know how consumers perceive their products and services and those of their competitors. The same can also be said about consumers because whenever one wants to buy something, one wants to know the opinions of existing users. These practical needs and the technical challenges will keep the field vibrant and lively for years to come.
Acknowledgements
I am very grateful to Theresa Wilson for her insightful and detailed comments and suggestions, which have helped me improve the chapter significantly. I thank my former and current students for working with me on this fascinating topic: Xiaowen Ding, Murthy Ganapathibhotla, Minqing Hu, Nitin Jindal, Guang Qiu (visiting student from Zhejiang University) and Lei Zhang. I would also like to express my gratitude to Birgit König (McKinsey&Company) for many valuable discussions which have helped shape my understanding of the practical side of sentiment analysis and its related issues.
References
[1]. A. Andreevskaia and S. Bergler, “Mining WordNet for a fuzzy sentiment: Sentiment tag extraction from WordNet glosses.” Proceedings of the European Chapter of the Association for Computational Linguistics (EACL), 2006.
[2]. N. Archak, A. Ghose, and P. Ipeirotis, “Show me the money! Deriving the pricing power of product features by mining consumer reviews,” Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2007.
[3]. A. Aue and M. Gamon, “Customizing sentiment classifiers to new domains: A case study,”
Proceedings of Recent Advances in Natural Language Processing (RANLP), 2005.
[4]. P. Beineke, T. Hastie, C. Manning, and S. Vaithyanathan. “Exploring sentiment summarization,”
Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text, AAAI technical report SS-04-07, 2004.
[5]. S. Bethard, H. Yu, A. Thornton, V. Hatzivassiloglou, and D. Jurafsky, “Automatic extraction of opinion propositions and their holders,” Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text, 2004.
[6]. J. Blitzer, M. Dredze, and F. Pereira, “Biographies, Bollywood, boom-boxes and blenders:
Domain adaptation for sentiment classification,” Proceedings of the Association for Computational Linguistics (ACL), 2007.
[7]. J. Bos, and M. Nissim, “An Empirical Approach to the Interpretation of Superlatives.”
Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2006.
[8]. E. Breck, Y. Choi, and C. Cardie, “Identifying expressions of opinion in context,” Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2007.
[9]. G. Carenini, R. Ng, and A. Pauls, “Multi-document summarization of evaluative text,”
Proceedings of the European Chapter of the Association for Computational Linguistics (EACL), pp. 305–312, 2006.
[10]. G. Carenini, R. T. Ng, and E. Zwart, “Extracting knowledge from evaluative text,” Proceedings of International Conference on Knowledge Capture (K-CAP), pp. 11-18, 2005.
[11]. Y. Chen and J. Xie, “Online consumer review: Word-of-mouth as a new element of marketing communication mix,” Management Science, vol. 54, pp. 477–491, 2008.
[12]. P. Chesley, B. Vincent, L. Xu, and R. Srihari, “Using verbs and adjectives to automatically classify blog sentiment,” in AAAI Symposium on Computational Approaches to Analysing Weblogs (AAAI-CAAW), pp. 27–29, 2006.
[13]. Y. Choi, E. Breck, and C. Cardie, “Joint extraction of entities and relations for opinion recognition,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2006.
[14]. Y. Choi, C. Cardie, E. Riloff, and S. Patwardhan, “Identifying sources of opinions with conditional random fields and extraction patterns,” Proceedings of the Human Language Technology Conference and the Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), 2005.
[15]. S. R. Das and M. Y. Chen, “Yahoo! for Amazon: Sentiment extraction from small talk on the Web,” Management Science, vol. 53, pp. 1375–1388, 2007.
[16]. K. Dave, S. Lawrence, and D. M. Pennock, “Mining the peanut gallery: Opinion extraction and semantic classification of product reviews,” Proceedings of WWW, pp. 519–528, 2003.
[17]. C. Dellarocas, X. Zhang, and N. F. Awad, “Exploring the value of online product ratings in revenue forecasting: The case of motion pictures,” Journal of Interactive Marketing, vol. 21, pp.
23–45, 2007.
[18]. A. Devitt and K. Ahmad, “Sentiment analysis in financial news: A cohesion based approach,”
Proceedings of the Association for Computational Linguistics (ACL), pp. 984–991, 2007.
[19]. X. Ding, B. Liu, and P. S. Yu, “A holistic lexicon-based approach to opinion mining,”
Proceedings of the Conference on Web Search and Web Data Mining (WSDM), 2008.
[20]. K. Eguchi and V. Lavrenko, “Sentiment retrieval using generative models,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 345–354, 2006.
[21]. A. Esuli and F. Sebastiani, “Determining the semantic orientation of terms through gloss analysis,” Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), 2005.
[22]. A. Esuli and F. Sebastiani, “Determining term subjectivity and term orientation for opinion mining,” Proceedings of the European Chapter of the Association for Computational Linguistics (EACL), 2006.
[23]. A. Esuli and F. Sebastiani, “SentiWordNet: A publicly available lexical resource for opinion mining,” Proceedings of Language Resources and Evaluation (LREC), 2006.
[24]. A. Esuli and F. Sebastiani, “PageRanking WordNet synsets: An application to opinion mining,”
Proceedings of the Association for Computational Linguistics (ACL), 2007.
[25]. C. Fellbaum, ed., Wordnet: An Electronic Lexical Database. MIT Press, 1998.
[26]. M. Fiszman, D. Demner-Fushman, F. Lang, P. Goetz and T. Rindflesch, “Interpreting comparative constructions in biomedical text,” BioNLP, 2007.
[27]. M. Gamon, “Sentiment classification on customer feedback data: Noisy data, large feature vectors, and the role of linguistic analysis,” Proceedings of the International Conference on Computational Linguistics (COLING), 2004.
[28]. M. Gamon, A. Aue, S. Corston-Oliver, and E. Ringger, “Pulse: Mining customer opinions from free text,” Proceedings of the International Symposium on Intelligent Data Analysis (IDA), pp.
121–132, 2005.
[29]. G. Ganapathibhotla and B. Liu. “Identifying Preferred Entities in Comparative Sentences,”
Proceedings of the International Conference on Computational Linguistics, COLING, 2008.
[30]. R. Ghani, K. Probst, Y. Liu, M. Krema, and A. Fano, “Text mining for product attribute extraction,” SIGKDD Explorations Newsletter, vol. 8, pp. 41–48, 2006.
[31]. A. Ghose and P. G. Ipeirotis, “Designing novel review ranking systems: Predicting usefulness and impact of reviews,” Proceedings of the International Conference on Electronic Commerce (ICEC), 2007.
[32]. A. Ghose, P. G. Ipeirotis, and A. Sundararajan, “Opinion mining using econometrics: A case study on reputation systems,” Proceedings of the Association for Computational Linguistics (ACL), 2007.
[33]. N. Godbole, M. Srinivasaiah, and S. Skiena, “Large-scale sentiment analysis for news and blogs,”
Proceedings of the International Conference on Weblogs and Social Media (ICWSM), 2007.
[34]. V. Hatzivassiloglou and K. McKeown, “Predicting the semantic orientation of adjectives,”
Proceedings of the Joint ACL/EACL Conference, pp. 174–181, 1997.
[35]. V. Hatzivassiloglou and J. Wiebe, “Effects of adjective orientation and gradability on sentence subjectivity,” Proceedings of the International Conference on Computational Linguistics (COLING), 2000.
[36]. M. Hu and B. Liu, “Mining and summarizing customer reviews,” Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 168–177, 2004.
[37]. N. Hu, P. A. Pavlou, and J. Zhang, “Can online reviews reveal a product’s true quality?: Empirical findings and analytical modeling of online word-of-mouth communication,” Proceedings of Electronic Commerce (EC), pp. 324–330, USA, New York, NY: ACM, 2006.
[38]. N. Jindal and B. Liu, “Identifying comparative sentences in text documents,” Proceedings of the ACM Special Interest Group on Information Retrieval (SIGIR), 2006.
[39]. N. Jindal and B. Liu, “Mining comparative sentences and relations,” Proceedings of AAAI, 2006.
[40]. N. Jindal and B. Liu, “Review spam detection,” Proceedings of WWW, 2007. (Poster paper).
[41]. N. Jindal and B. Liu, “Opinion spam and analysis,” Proceedings of the Conference on Web Search and Web Data Mining (WSDM), pp. 219–230, 2008.
[42]. N. Kaji and M. Kitsuregawa, “Building lexicon for sentiment analysis from massive collection of HTML documents,” Proceedings of the Joint Conference on Empirical Methods in Natural
Language Processing and ComputationalNatural Language Learning (EMNLP-CoNLL), pp.
1075–1083, 2007.
[43]. J. Kamps, M. Marx, R. J. Mokken and M. de Rijke. Using WordNet to measure semantic orientation of adjectives. In Proc. of LREC’04, pp. 1115-1118, 2004.
[44]. H. Kanayama and T. Nasukawa, “Fully automatic lexicon expansion for domain-oriented sentiment analysis,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 355–363, July 2006.
[45]. A. Kennedy and D. Inkpen, “Sentiment classification of movie reviews using contextual valence shifters,” Computational Intelligence, vol. 22, pp. 110–125, 2006.
[46]. S.-M. Kim and E. Hovy, “Determining the sentiment of opinions,” Proceedings of the International Conference on Computational Linguistics (COLING), 2004.
[47]. S.-M. Kim and E. Hovy, “Automatic identification of pro and con reasons in online reviews,”
Proceedings of the COLING/ACL Main Conference Poster Sessions, pp. 483–490, 2006.
[48]. S.-M. Kim and E. Hovy, “Crystal: Analyzing predictive opinions on the web,” Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), 2007.
[49]. S.-M. Kim, P. Pantel, T. Chklovski, and M. Pennacchiotti, “Automatically assessing review helpfulness,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 423–430, Sydney, Australia, July 2006.
[50]. N. Kobayashi, K. Inui and Y. Matsumoto. Extracting Aspect-Evaluation and Aspect-of Relations in Opinion Mining. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1065–1074, 2007.
[51]. N. Kobayashi, K. Inui, Y. Matsumoto, K. Tateishi, and T. Fukushima. Collecting evaluative expressions for opinion extraction. Proceedings of the 1st International Joint Conference on Natural Language Processing (IJCNLP), pages 584–589, 2004.
[52]. L.-W. Ku, Y.-T. Liang, and H.-H. Chen, “Opinion extraction, summarization and tracking in news and blog corpora,” in AAAI Symposium on Computational Approaches to Analysing Weblogs (AAAI-CAAW), pp. 100–107, 2006.
[53]. L.-W. Ku, Y.-T. Liang, and H.-H. Chen, Novel Relationship Discovery Using Opinions Mined from the Web. AAAI 2006.
[54]. J. Lafferty, A. McCallum, and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” Proceedings of ICML, pp. 282–289, 2001.
[55]. B. Liu, Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data. Springer, 2006.
[56]. B. Liu, M. Hu, and J. Cheng, “Opinion observer: Analyzing and comparing opinions on the web,”
Proceedings of WWW, 2005.
[57]. J. Liu, Y. Cao, C.-Y. Lin, Y. Huang, and M. Zhou. “Low-quality product review detection in opinion summarization,” Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 334–
342, 2007. (Poster paper).
[58]. Y. Liu, J. Huang, A. An, and X. Yu. “ARSA: A sentiment-aware model for predicting sales performance using blogs,” Proceedings of the ACM Special Interest Group on Information Retrieval (SIGIR), 2007.
[59]. Y. Lu and C. X. Zhai. Opinion integration through semi-supervised topic modeling. Proceedings of 2008 International WWW Conference (WWW’08), pp. 121–130, 2008.
[60]. C. MacDonald, I. Ounis, and I. Soboroff. Overview of the TREC2007 Blog Track. 2007.
http://trec.nist.gov/pubs/trec16/papers/BLOG.OVERVIEW16.pdf.
[61]. J. Martin and P. White. The Language of Evaluation, Appraisal in English, Palgrave Macmillan, London, New York, 2005.
[62]. R. McDonald, K. Hannan, T. Neylon, M. Wells, and J. Reynar, “Structured models for fine-to-coarse sentiment analysis,” Proceedings of the Association for Computational Linguistics (ACL), pp. 432–439, Prague, Czech Republic: June 2007.
[63]. Q. Mei, X. Ling, M. Wondra, H. Su, and C. X. Zhai, “Topic sentiment mixture: Modeling facets and opinions in weblogs,” Proceedings of WWW, pp. 171–180, 2007.
[64]. R. Mihalcea, C. Banea, and J. Wiebe, “Learning multilingual subjective language via cross-lingual projections,” Proceedings of the Association for Computational Linguistics (ACL), pp. 976–983, 2007.
[65]. S. Morinaga, K. Yamanishi, K. Tateishi, and T. Fukushima, “Mining product reputations on the Web,” Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 341–349, 2002. (Industry track).
[66]. T. Nasukawa and J. Yi, “Sentiment analysis: Capturing favorability using natural language processing,” Proceedings of the Conference on Knowledge Capture (K-CAP), 2003.
[67]. V. Ng, S. Dasgupta, and S. M. N. Arifin, “Examining the role of linguistic knowledge sources in the automatic identification and classification of reviews,” Proceedings of the COLING/ACL Main Conference Poster Sessions, pp. 611–618, Sydney, Australia: July 2006.
[68]. X. Ni, G.-R. Xue, X. Ling, Y. Yu, and Q. Yang, “Exploring in the weblog space by detecting informative and affective articles,” Proceedings of WWW, 2007.
[69]. K. Nigam and M. Hurst, “Towards a robust metric of polarity,” in Computing Attitude and Affect in Text: Theories and Applications, number 20 in The Information Retrieval Series, (J. G.
Shanahan, Y. Qu, and J. Wiebe, eds.), 2006.
[70]. B. Pang and L. Lee, “A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts,” Proceedings of the Association for Computational Linguistics (ACL), pp. 271–278, 2004.
[71]. B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” Proceedings of the Association for Computational Linguistics (ACL), pp.
115–124, 2005.
[72]. B. Pang and L. Lee, “Opinion mining and sentiment analysis.” Foundations and Trends in Information Retrieval 2(1-2), pp. 1–135, 2008.
[73]. B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? Sentiment classification using machine learning techniques,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 79–86, 2002.
[74]. D.-H. Park, J. Lee, and I. Han, “The effect of on-line consumer reviews on consumer purchasing intention: The moderating role of involvement,” International Journal of Electronic Commerce, vol. 11, pp. 125–148, 2007.
[75]. W. Parrott. Emotions in Social Psychology, Psychology Press, Philadelphia, 2001.
[76]. L. Polanyi and A. Zaenen, “Contextual lexical valence shifters,” Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text, 2004.
[77]. A.-M. Popescu and O. Etzioni, “Extracting product features and opinions from reviews,”
Proceedings of the Human Language Technology Conference and the Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), 2005.
[78]. G. Qiu, B. Liu, J. Bu and C. Chen. Expanding Domain Sentiment Lexicon through Double Propagation, International Joint Conference on Artificial Intelligence (IJCAI-09), 2009.
[79]. E. Riloff, S. Patwardhan, and J. Wiebe, “Feature subsumption for opinion analysis,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2006.
[80]. E. Riloff and J. Wiebe, “Learning extraction patterns for subjective expressions,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2003.
[81]. E. Riloff, J. Wiebe, and T. Wilson, “Learning subjective nouns using extraction pattern bootstrapping,” Proceedings of the Conference on Natural Language Learning (CoNLL), pp. 25–
32, 2003.
[82]. S. Sarawagi, “Information extraction,” to appear in Foundations and Trends in Information Retrieval, 2009.
[83]. Y. Seki, K. Eguchi, N. Kando, and M. Aono, “Opinion-focused summarization and its analysis at DUC 2006,” Proceedings of the Document Understanding Conference (DUC), pp. 122–130, 2006.
[84]. B. Snyder and R. Barzilay. Multiple Aspect Ranking Using the Good Grief Algorithm.
Proceedings of Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, 2007 HLT-NAACL 2007: 300-307.
[85]. X. Song, Y. Chi, K. Hino, and B. Tseng, “Identifying opinion leaders in the blogosphere,”
Proceedings of the Conference on Information and Knowledge Management (CIKM), pp. 971–
974, 2007.
[86]. A. Stepinski and V. Mittal, “A fact/opinion classifier for news articles,” Proceedings of the ACM Special Interest Group on Information Retrieval (SIGIR), pp. 807–808, 2007.
[87]. P. J. Stone. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press, 1966.
[88]. V. Stoyanov and C. Cardie, “Partially supervised coreference resolution for opinion summarization through structured rule learning,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 336–344, Sydney, Australia: July 2006.
[89]. Q. Su, X. Xu, H. Guo, X. Wu, X. Zhang, B. Swen, and Z. Su. Hidden Sentiment Association in Chinese Web Opinion Mining. Proceedings of WWW’08, pp. 959-968, 2008.
[90]. H. Takamura, T. Inui, and M. Okumura, “Extracting semantic orientations of phrases from dictionary,” Proceedings of the Joint Human Language Technology/North American Chapter of the ACL Conference (HLT-NAACL), 2007.
[91]. L. Tesnière, éléments de syntaxe structurale, Klincksieck, Paris 1959.
[92]. M. Thomas, B. Pang, and L. Lee, “Get out the vote: Determining support or opposition from congressional floor-debate transcripts,” Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 327–335, 2006.
[93]. I. Titov and R. McDonald. A Joint Model of Text and Aspect Ratings for Sentiment Summarization. Proceedings of 46th Annual Meeting of the Association for Computational Linguistics (ACL’08), 2008.
[94]. R. M. Tong, “An operational system for detecting and tracking opinions in on-line discussion.”
Proceedings of the Workshop on Operational Text Classification (OTC), 2001.
[95]. P. Turney, “Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews,” Proceedings of the Association for Computational Linguistics (ACL), pp. 417–424, 2002.
[96]. X. Wan. “Using bilingual knowledge and ensemble techniques for unsupervised Chinese sentiment analysis.” In Proceedings of EMNLP08, pp. 553-561, 2008.
[97]. J. Wiebe, “Learning subjective adjectives from corpora,” Proceedings of AAAI, 2000.
[98]. J. Wiebe, R. F. Bruce, and T. P. O’Hara. “Development and use of a gold standard data set for subjectivity classifications.” Proceedings of the Association for Computational Linguistics (ACL), pp. 246–253, 1999.
[99]. J. Wiebe and R. Mihalcea. “Word sense and subjectivity.” Proceedings of the Conference on Computational Linguistics / Association for Computational Linguistics (COLING/ACL), 2006.
[100]. J. Wiebe and T. Wilson, “Learning to disambiguate potentially subjective expressions,”
Proceedings of the Conference on Natural Language Learning (CoNLL), pp. 112–118, 2002.
[101]. J. Wiebe, T. Wilson and C. Cardie. Annotating expressions of opinions and emotions in language.
Language Resources and Evaluation, 1(2), 2005.
[102]. J. Wiebe, T. Wilson, R. Bruce, M. Bell, and M. Martin, “Learning subjective language,”
Computational Linguistics, vol. 30, pp. 277–308, September 2004.
[103]. T. Wilson, J. Wiebe, and R. Hwa, “Just how mad are you? Finding strong and weak opinion clauses,” Proceedings of AAAI, pp. 761–769, 2004.
[103]. T. Wilson, J. Wiebe, and R. Hwa, “Just how mad are you? Finding strong and weak opinion clauses,” Proceedings of AAAI, pp. 761–769, 2004.