8. Predicting political tendency of posts on Facebook
8.3 Predictive analysis
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
(a) TDM_TF (b) TDM_TFIDF Figure 47. The word cloud for the right-wing politics
8.3 Predictive analysis
In this section, we focus on posts with messages (i.e., 24,642 posts) for sentiment analysis. We use a popular data mining software package, Weka [120]. Regarding classification algorithms, we use top ones [104]: Naïve Bayes, kNN (where k=1), SVM, AdaBoost, C4.5 Decision Tree, and CART. We compare the classification performance.
Regarding the measure for prediction performance, we mainly use the classic F1-score, which is a measure that combines two commonly used measures, precision, and recall, as shown in (15).
𝐹𝐹1 − 𝑢𝑢𝑠𝑠𝑜𝑜𝑛𝑛𝑒𝑒 = 2 ×𝑃𝑃𝑛𝑛𝑒𝑒𝑠𝑠𝑖𝑖𝑢𝑢𝑖𝑖𝑜𝑜𝑛𝑛 × 𝑅𝑅𝑒𝑒𝑠𝑠𝑎𝑎𝑙𝑙𝑙𝑙
𝑃𝑃𝑛𝑛𝑒𝑒𝑠𝑠𝑖𝑖𝑢𝑢𝑖𝑖𝑜𝑜𝑛𝑛 + 𝑅𝑅𝑒𝑒𝑠𝑠𝑎𝑎𝑙𝑙𝑙𝑙 (15)
Furthermore, we use 10-fold cross-validation. It divides the dataset into 10 disjoint subsets. It uses 9 subsets to create a new dataset, and use the new dataset to train a classifier. Then, it uses the remaining 1 subset to test the classifier. It repeats the above two steps 10 times, and each time it uses a different subset. The final result is an
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
aggregate of the 10 test results. Cross-validation is almost the standard way to evaluate classifiers and compare classification algorithms and to find an optimal set of parameters for a classification algorithm.
We use word frequency between documents and sentiment analysis to predict posts’
political tendency. For each group of political fan pages, each word is given TF value and a political label, l or r. Similarly, we use TF-IDF instead of TF.
We use classification algorithms to predict posts’ political tendency. When SVM and CART classifiers are used, we have errors indicating that the dataset is too huge on TF and TF-IDF. However, while we analyze political sentiment of posts, SVM and CART classifiers can be applied to this dataset. Figure 48 shows our experiment result. For TDM_TFIDF, the best classifier is Decision Tree; the F1-score is 0.95. For TDM_TF, the best classifiers are Decision Tree and AdaBoost but some classification algorithms would not perform well. On TF and TF-IDF, Decision Tree outperforms others. Our results show that TF-IDF is sensitive to these classifiers. Totally, the classification performance on TF-IDF is better than the classification performance on TF.
Figure 48. F1-scores given by TDM_TF and TDM_TFIDF
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
We use TF and TF-IDF to classify all posts, and left- and right-wing political posts.
Figure 49 shows our experiment result using TF to predict posts’ political tendency.
For left-wing politics, AdaBoost and Decision Tree are the best classifiers. For right-wing politics, the best classification algorithm is Naïve Bayes.
Figure 49. F1-scores given by TDM_TF
Figure 50 shows our experiment result using TF-IDF. For left-wing politics, Decision Tree is the best classifier. For right-wing politics, the best classification algorithm is also Decision Tree. The F1-scores are over 0.9. A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences.
It is one way to display an algorithm that only contains conditional control statements.
Decision Tree is the best classifier on TF and TF-IDF.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 50. F1-scores given by TDM_TFIDF
We analyze political sentiment of posts using the two lexical databases. We compute ratios of positive words, negative words, and objective words for each post. Then, we predict posts’ political tendency using classifiers. Posts contain some positive words or negative words regardless of the left- and right-wing politics. In Figure 51, we predict all posts, and left- and right-wing political posts. Totally, 1NN is the best classification algorithm for the opinion lexicon; Naïve Bayes is the best one for SentiWordNet.
However, regardless of classifiers and lexical databases, the classification performance for left-wing politics is better than the classification performance for right-wing politics.
It may be that the number of words of left-wing political posts is over 2 times larger than the number of words of left-wing political ones. For left-wing politics, Naïve Bayes, kNN, SVM, AdaBoost, and Decision Tree are the best classifiers using the two lexical databases; the F1-scores are over 0.7. For right-wing politics, the best classification algorithm is 1NN but some classification algorithms would not perform well.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
(a) The opinion lexicon
(b) SentiWordNet
Figure 51. F1-scores given by the two lexical databases
We analyze interaction features for classification. We classify all posts, and left- and right-wing political posts. We use the original interaction values in Figure 52.
similarity_L denotes the similarity between a number and the average of left-wing politics numbers. It calculates the distance between the value of an interaction feature
‧
of posts and the average value of the feature of left-wing political posts. Similarly, similarity_R is for ring-wing political posts. The similarity is calculated with respect to a class. We also use logarithm, normalization, standardization, similarity_L and similarity_R functions for adjustments in Figure 53, Figure 54, Figure 55, Figure 56 and Figure 57, respectively. For all posts, when the values of the interaction features are adjusted by the normalization and standardization functions and when CART is used, the F1-scores are 0.7, the best result. SVM would not perform well but CART and Decision Tree would. 1NN is insensitive to these adjustments. Adjustments have a positive effect or at least do not have a negative effect on the performance of Naïve Bayes and AdaBoost. For Naïve Bayes and AdaBoost, adjustments with the logarithm and similarity functions have a positive effect; adjustments with the normalization and standardization function do not have an effect.
For left-wing political posts, when the values of the interaction features are adjusted by the logarithm function and when Decision Tree is used, the F1-score is 0.8, the best result. Results of SVM are close to those of CART and the F1-scores. SVM and CART are insensitive to these adjustments. Adjustments with the logarithm function have a positive effect and adjustments with the normalization and similarity functions have a negative effect on the performance of Decision Tree. For Naïve Bayes, the performance increase given by using the similarity function is 7 times larger than that given by using the normalization function and original data. The classification performance of the left-wing political posts is larger than the performance of all posts, except when the original data and adjustments with the normalization function are used with Naïve Bayes.
For right-wing political posts, when the values of the interaction features are adjusted by the standardization function and when CART is used, the F1-score is 0.66, the best result. Overall, the classification of the right-wing political posts would not be as good as that of the left-wing political posts.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 52. F1-scores given by the interaction features
Figure 53. F1-scores given by logarithm function of the interaction features
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 54. F1-scores given by normalization function of the interaction features
Figure 55. F1-scores given by standardization function of the interaction features
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 56. F1-scores given by similarity_L of the interaction features
Figure 57. F1-scores given by similarity_R of the interaction features