Content Ranking using Social Metrics

Given that CrowdSMILE leverages crowdsourcing to generate learning content, there arises a challenge of providing relevant content to the learner and not useless content. To solve the problem, we propose content ranking by using SNS features.

When a user submits new Learning content to a Mission, the link to the content with other information gets publicly posted in the Facebook group of the Mission. As the post is public, other learners may Like the link, they may Share it and of course, click on it. We propose using a combination of Likes, Shares and View counts to rank the learning content.

The rank is then used as a determining factor in the listing order of content to the learner.

The number of Likes and Shares of content posted on FB can be acquired using Facebook graph API. For View counts, we use a redirector link hosted on the CrowdSMILE server that keeps a track of visits to the link while forwarding the user to the actual content.

The links posted actually point to the CrowdSMILE Server with a hash key in the URL that refers to the actual URL in a database on the server. Once the link is clicked, the user is taken to the Server that logs the view as a +1 to total number of views and then redirects user to actual URL of the content.

5.2.1. Definition of Social Index

“Social Index” is the rank value of learning content posted in the CrowdSMILE system. We have come up 6 ways for calculating the Social index. We arrived at these combinations strictly by trial and error. The 6 combinations are:

SI1(i) = Likes + (Shares*2) + (Views * 1.5) (5.1)

62

Once we decide which SI we are going to use, we further define the SI of a Quest and Mission respectively. Let Q be Learning Quest. The SI of Q is defined as follows in equation 5.7:

SI(Q) = ∑

where i ∈ { Learning Objects of Quest Q }

Social Index for Quests

Let M be a Learning Mission. The SI of M is defined in equation 5.8 as follows:

(5.7)

SI(M) = ∑̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅

where Qⁱ ∈ { Quests of Mission M }

Social Index for Missions

(5.8)

We propose that once the Social Index of a Mission and its Quests have been calculated, we may use that for ranking content.

5.2.2. Experiment Setup

To evaluate our content ranking method, we conduct the following experiment. We consider user’s opinions as a manual rank. We then compare the manual rank with the SI ranks. The experiment details as follows:

1) Facebook group with 50 users was setup.

2) Posted 15 Links related to a single topic in the group page. Posted around 2 links per day and monitored the activity.

3) After one week, we analysed the Likes, Shares and Views of each link posted and calculated the Social Index (SI) for each link using the 6 different methods proposed.

We then randomly chose 20 out of 50 users in the group. Each user is asked to visit each link one by one and rank the content on a scale of 1 to 10, where 1 is a low rank and 10 is a high rank. We call this the manual rank.

The hypothesis is that if the Social Index ranks are similar to this manual manual rank, we may be able to use the SI ranking method as an effective content ranking method.

63

4) We then compared the manual rank with the 6 Social Index based ranks. We then perform statistical analysis and compare them to find their correlation strength.

5.2.3. Kendall’s Tau for Statistical Correlation

Kendall’s tau [30] correlation coefficient is designed to capture the association between two variables. Its estimate (denoted τ) can be defined and expressed as follows [49]:

Let (x1, y1), (x2, y2), …, (xn, yn) be a set of observations of the joint random variables X and Y respectively, such that all the values of (xi) and (yi) are unique. Any pair of observations (xi, yi) and (xj, yj) are said to be concordant if the ranks for both elements agree:

that is, if both xi > xj and yi > yj or if both xi < xj and yi < yj. They are said to be discordant, if xi > xj and yi < yj or if xi < xj and yi > yj. If xi = xj or yi = yj, the pair is neither concordant nor discordant.

(5.9)

5.2.4. Test Results

Rank A is the manual ranking method Rank B1 = SI1(i)

Rank B2 = SI2(i) As can be seen,

the correlation between Rank A and Rank B1 is τ = .453 the correlation between Rank A and Rank B2 is τ = .485

Table 5-1: Kendall’s Tau Correlation Strengths

τ = 0 - .4 Weak Correlation

τ = .4 - .7 Moderate Correlation τ = .7 - 1 Strong Correlation

64

Referring to table 5-1 as a guide we can see that we have a moderate correlation between Rank A and Rank B1 (SI1(i)). Similarly, we have a moderate correlation between Rank A and Rank B2 (SI2(i)).

Figure 5-5: Visual comparision of ranking of items using the different methods.

Figure 5-5 above shows that there exists some correlation between SI1(i) and Rank A, and between SI2(i) and Rank A. If the graph is studied carefully, we can see than Rank A’s curve follows all the other Rank’s curves. It indicates that there is some relation between all of them.

However, given the small sample size and the lack of a strong correlation, we would have to conclude that while there is correlation, it is not enough to be used solely as a basis for ranking.

Exploring alternative means from Facebook features may give other effective ranking methods. For example, data mining of Facebook comments is a potential solution that can be combined with our proposed method above.

0

65