Recommendation Performance - RESULTS AND EVALUATION

CHAPTER 5 RESULTS AND EVALUATION

5.3 Recommendation Performance

In this sector, we analyze the accuracy for our recommendation results of different types of jobs. We classify the job query from users in different industries and calculate the user’s satisfaction for each categories. The result is displayed as Figure 20. What we can figure out from the figure is that obviously the users who are still student think the outcome of this mechanism benefit their job hunting the most. Besides those group of people, those users who works in the information technology or semi conduct related also regard this mechanism is helpful. The reason may be obvious since we can see that in our sample, both in our participants or users, the population whose jobs is about information technologies and semi conduct are the largest group from Figure 15 and Figure 16. With more possible candidate as the selection base, the more people we can recommend and of course the better result we might reach and deliver.

AccuracyAccuracy

Figure 20. The Performance of different kinds of job query

Figure 21. Comparison for single components 5.4 Factors Performance

We compare the distinct candidate list generated from different approaches by comparing the accuracy. Here we use ST, ES, JI and SI stands for social tie strength, experience similarity, job influence and social influence correspondingly in the following description. We ask user to evaluate how they think the outcome results generated from the combination of four factors are really helpful or not. Then we respectively look deep into the influence on result for each factor.

First we exam these factors respectively with the candidate lists produced by single components and random pick-up. The result is displayed as below Figure 21:

Surprisingly, this figure shows that the factor of job relevance plays the best among all the results from single factors, instead of the social tie, which most users consider the most important in the questionnaire showed at 5.2.The other finding we can see from this Figure 22 is the satisfaction that only generated by one factors is far below the expectation compared with the result from all of four factors.

Accuracy

ST+ES ST+JR ST+SC ES+JR ES+SC JR+SC ST+ES+JRST+ES+SCST+JR+SC ES+JR+SC All

The result shows that in two components scenarios, users are more satisfied with the result from the combination of experience similarity and job relevance. We can also conclude from this figure that even though with more factors considered the better the performance is, the factor of social capital here contributes little to the improvement of results. Finally, a statistical test is conducted using the paired sample t-test as Table 5.

Paired Group Mean

ES+SC 2.48571 0.94398 0.11283 22.031 0

JR+SC 1.42857 1.001013 0.11965 11.940 0

ST+ES+JR 0.14286 0.87287 0.10433 1.369 0.175

ST+ES+SC 1.74286 0.73594 0.08796 19.814 0

ST+JR+SC 0.85714 0.96738 0.11562 7.413 0

ES+JR+SC 0.8000 0.86141 0.10296 7.770 0

JR+SC 1.42857 1.001013 0.11965 11.940 0

Table 5 .Statistical verification results of factor combinations

As shown in Table 5, we can see except for the combination of ST+ES+JR, all the other pair are significant under 0.05, which we can conclude that our proposed model is better than the other listed model. However, we find out something interesting in the combination of ST+ES+JR, which we do not have enough evidence that the result of four-factor model is significant better than that. In other words, we may consider only use this combination enough applied in our model.

5.5 User’s rating

In this section, we measure how our referral mechanism improves the new services by comparing to well-known web services representing the traditional job websites and business social network websites as our benchmark. Since the output of our system is totally different from those two benchmark websites, here we measure this difference by the user’s

satisfaction in the way of Likert scale questionnaire. There are two aspects of evaluation we consider in the questionnaire: (1) providing more information (2) facilitating for the job hunting activities.

We measure above questions by the questionnaire, asking how those job seekers think about the result of candidate list and how they think this platform is going to support their job-hunting works. In order to fairly compare, we invite the users who have used LinkedIn, 104 job websites and our system to rate our performance. Here we use the job website of 104 human resources bank and LinkedIn as our benchmark to conduct evaluation. The partial content of questionnaire is shown as below:

Question 1: Do you think this social referral service benefits your job-hunting activities?

Question 2: Do you think the service provide by 104 human resource bank benefits your job-hunting activities?

Table 6. User evaluation Questionnaire

The questionnaire is composed of two different aspects. Maybe it is easier to depict as the 3x2 matrix to understand this kind of evaluation. The rows are our benchmark and our proposed system, and the columns are the two aspects we planned to compare. Here we use the Likert scale of five points to do the quantitative analysis. The higher value represents the more satisfied users feel about certain services. We summarized the result from the users’

questionnaires are displayed as following Figure 23:

From the previous figures, we can observe that our system performs well in every aspect, whether being informative and help the job hunting, compared with other current job related information websites. Another interesting findings from the survey is that though LinkedIn is very popular in the United States and other countries, it is not that prevalent in Taiwan due to lacking of adverting and also the limitation of languages.104 human resource bank, on the

Average of Satisfaction Score

Figure 23. Comparison for Useful Information Dimension

other side, is still the first choice for most users when they think about finding jobs.

5.6 Recommendation list

As we mentioned in section 4.1, we will recommend the distribution list which is composed of mixture of both strong-tie and weak-tie friends to the job seeker. Here we further analyze the effectiveness of network construction by different ranking of the list.

There are three different types of ranking list: ranked from the strongest to the weakest, ranked from the weakest to the strongest, the random ranked. We use the definition the same as Granovetter[5]: for those people who directly contact at least twice a week, we define as strong tie, others are weak tie. We will choose the strong-tie and weak-tie friend by this concept. As for random, we will randomly select people from the friend list of user to from the recommended list. Sincere in our experiment this list only works in the period of network construction, we use the size of network as the measurement of those three methods. The result shows as Figure 24.

Figure 24. The average number of participant network

131

From the above figure we can see the real result matches the sociological theory. Though strong-tie friends are definitely willing to help the job seeker more, there would be more possible to have people overlapped in the scope of social network. On the other hand,

weak-tie friends might be less directly interact, though, somehow they are still happy to give a hand when some occasion, like job hunting, happens. Thus, we would larger distribution result benefiting from this kind of relationship.

CHAPTER 6 DISCUSSION AND CONCLUSION

Social power has long been comprehensively used in various activities in the life of people, especially in the field of diffusion the request or asking for someone help. With the popularity of social media, people tend to utilize this huge digital media to publicize their needs, such as job seeking or consulting information. On the other side, from the view of business companies, they has already started to implement their recruiting activities by the support of internet and websites, and even use the social platform as the new channel to promote their own companies and their job, and even an abundant source to find and search the data of candidates to do further evaluation. Combining those trends from the recruiters and job seeker, this mechanism perfectly solves this problem by the matching platform for both job providers and job seekers. The result of the experiment shows that this proposed platform has better performance than other previous benchmark approaches. Moreover, according to our experimental results and evaluations, this new mechanism can effectively mine the potential referral from their social network on the base of social search and appraisal as well as the best efficient social path to reach to referrals.

6.1 Research Contributions

This study makes several significant contributions as follows. At first, from the methodological aspect, our research propose the concept of using the power of social search to find out referral candidates from the social network to provide more job information more than the official description in the job websites. With this key element, a searching mechanism that utilized the power of social network is proposed to find all the possible candidates on the base of willingness and influence. In order to calculate the result with these elements, a classical ANP method is used to generate the final recommendation result.

Secondly, from the empirical aspect, we discover that not all the factors are equally important in the performance of social referral. Actually, the social tie strength, experience

similarity and job relevance are the three factors that truly matters for the result. Among those three factors, job relevance is the most important, which is surprisingly against what users think. Besides, according to the users’ feedback, we can see the recommendation performance of different job industries would be largely dependent on the occupation distribution of participants.

Thirdly, from the practical aspect, for most users joined our test, they all amazingly find out that actually they can reach those one who owns the information they long for concerning certain job openings or are worked in their dreaming companies by the support of our social platform, which does not show on any job websites so far. People can gain more information other than the job information itself but with the real people who they can ask or interact with.

Furthermore, they can even easily search and access those targeted people by the “social path”, the people connection through his or her own friends and the extended social network.

By consideration the real-time and daily interaction occurred on Facebook, our social path is more feasible and practical for those users, which is different from the function LinkedIn has already provided.

6.2 Research Limitations

Though this application has supported the recruiting activities for recruiters and job-hunting for job seekers, there are some limitations we encountered while conducting this research. First, due to the basic assumption and core concept of this mechanism, we need to ensure all the participants are linked by the real existing social links. All of links between every pair of people are tractable and measureable. The web-based app can only be distributed and filled by the people who really know each other. The job seekers have to request all of his or her friends to provide their information and even ask them to send out the same request to their friends (friends of friends) to maximize the distribution and information gathering in order to locate the best referral candidates. This process is very critical in our

system but also very challenging and definitely faced lots of troubles about practical implementation, such as the amount of involving users and the time it takes to gather all of that information.

Moreover, the social interactions we can gather by Facebook are just limited on the online interaction. Although we do all know actually the real face-to-face interaction matters more than virtual ones, little information we can gain from that due to the limitation of the nature of social digital media. For example, user A may click like on the Facebook wall of his friend B frequently and seldom do the same of the Facebook wall of friend C. Based on the data we can collect, we thus assume the social tie strength between A and B is closer than A and C. However, maybe the true reason is A meets C everyday and C seldom updates his wall while A never meets B in the real life. Besides, the privacy issues about social media also troubles the process of evaluation the social tie strength happened between every two people.

This mechanism needs the information such as education experiences and current job description about each person involving in the social network of job seekers.

Thirdly, for the picture of the whole benefit of this mechanism, the more users use this system, the larger the database we have, and thus the more candidates we can select and recommend. It is predicted that this system must encounter the problems lacking of enough users at first. But with the network effect, there would be more alternatives to choose and calculate after more users contributed their social network data to the mechanism.

Lastly, though plenty of sociology researches have proved that weak tie could provide more new information and would be better resources to consult when searching for a new job, how to fully apply to strength to the process of job seeking in a proper way is something worth considering. In this research, we support a list of strong-tie and weak-tie friends and wish users to get information from those people as they could, however in real world in most cases friends in social-tie are more likely to help the query mission. How to conquer this gap

between

6.3 Future Works

There are several related issues, which could be further studied to facilitate the analysis process to best recommend the proper candidates for referrals.

We will check those issues from three different dimensions, the mechanism itself, the job seekers and the purpose of this application.

First, in order to enhance the recommendation mechanism, there are some strategies we could apply to do more accurate analysis and try to stimulate the real decision process. We may take the relationship between different companies into considered and constantly update those correlations by the latest news. For example, take TSMC, HP, and MediaTek for consideration. Though it looks like there are totally different companies, actually HP is the vendor for TSMC IT department and MediaTek is the client side that assigns TSMC to complete their weaver producing. The calculation model will be precise if we consider all of those complicated and dynamic relationship between different companies and the collaboration between those departments in the companies in the advanced version of this application.

Secondly, we could also free the limitation of online media in the further use. We could ask our users to provide more detailed descriptions in the offline interaction among his or her own social network and even the context or scenario happened in the interaction. After calculation the list of possible candidates, the user may provide the real interaction information about the relationship of that social link to further optimize the process based on the true situations in the real world. For example, candidate A may be the classmate who meets three days a week and candidate B just someone meet in the party once.

Thirdly, as we mentioned in the limitation part, we need more users to be involved and willing to provide the information about their own working, education experiences as well as

their social interaction records. For the future, we could design mechanism to encourage more users engaged, like the membership and point system. For example, user can get some extra points while one of his or her friends decide to provide their information for the recommendation process.

Fourthly, we also focus on the distribution of users. From our experiment, we can see due to the users of Facebook, most users of our application are younger generation, like the student just graduated or the employers whose working experience is less than three years.

Maybe for the future we could pay more attention on the users of different ages, such as 40~50 year-old users and figure out some other ways to attract them.

Lastly, since we have already build on such powerful system, in the future we might expand the purpose of request for more widely use, such as finding experts or recommend male or female for date. Though the factors in the computation model may be vary with distinct application domain, the basic concept of social search is still the same. We might use this system as the base and develop more application.

REFERENCE

[1] iLogos Study Research, Global 500 Web Site Recruiting 2001 Survey, Available at:[http://www.thefreelibrary.com/iLogos+Research+Study+Shows+Global+500+Increase+

Web+Site+Recruiting...-a075511718], Accessed: 13/05/2014.

[2] Forrester Research Institute, e-Recruitment, Available at [http://en.wikipedia.org/wiki/E-recruitment], Accessed: 13/05/2014.

[3] Wang, I.Z, (2000). Companies focus on the professional skills while students focus on the package and interest, Young, 20-24.

[4] Society of Human Resource Management, (2001), Society for Human Resource Management Research. Available at:[http://www.shrm.org/Research/SurveyFindings/

Documents/Search%Tactic%20Poll.pdf] Accesses at:13/05/2014

[5] Granovetter, M. (1983). "The Strength of Weak Ties: A Network Theory Revisited". Sociological Theory 1: 201~233.

[6] facebook. Facebook Reports Fourth Quarter and Full Year 2013 Results

Available at: [http://investor.fb.com/releasedetail.cfm?ReleaseID=821954] Accessed at 13/05/2014

[7] Mark-Shane, S., (2008).Facebook as a social search engine and the implications for libraries in the twenty-first century. Emerald,. 26(0): p. 540 - 556.

[8] Hempel, Jesssi,(2013).”LinkedIn: How Its’ Changing Business.”. Fortune. pp.

69-74.

[9] Nishar,Deep.(2013).”200 Million Members!” LinkedIn Blog. LinkedIn.

Available at:[ http://blog.linkedin.com/2013/01/09/linkedin-200-million/]. Accesses at:

14/05/2014

[10] The Buntin Group and Survey Sampling International, (2013), the Lost Art of

Getting Together: Quantitative Exploration of Consumer Attitudes and Behavior”. Available at:[ http://www.mychinet.com/uploads/lostart/Results.pdf ] Accessed at:14/05/2014

[11] S. Brin and L. Page. (1998). The anatomy of a large-scale hyper textual Web search engine, WWW.

[12] M.R. Morris, J.Teevan, K.Panovich.(2010). What do people ask their social networks, and why?: a survey study of status message Q&A behavior. CHI.

[13] B. Smyth, (2007). A Community-Based Approach to Personalizing Web Search, in IEEE Computer, 40(8), pp.42-50.

[14] Glance, N.S. (2001). Community search assistant. In Proc. intelligent user interfaces (pp. 91–96).

[15] Amershi, S., Morris, M.R., (2008). CoSearch: A system for collocated collaborative web search. In Proc. CHI’08 (pp. 1647–1656). ACM Press.

[16] J. Kleinberg. (2006). Complex networks and decentralized search.In Proc. of

the Intl. Congress of Mathematicians (ICM).

[17] A.Trias,I. Mansilla,(2013), Question Waves: A multicast query routing algorithm for social search, Information Sciences,253,pp.1-25.

[18] A. Banerjee and S. Basu, (2008), “A social query model for decentralized search,” in SNAKDD.

[19] J. Kleinberg. (2000).The small-world phenomenon: An algorithmic perspective. Proc. 32nd ACM Symposium on Theory of Computing.

[20] C. Lin, N. Cao, S.X. Liu, S. Papadimitriou, J. Sun, X. Yan. (2009),SmallBlue:

Social Network Analysis for Expertise Search and Collective Intelligence, International Conference On Data Engineering, Shanghai, IEEE, pp. 1483–1486.

[21] H.Fang.,C.X.Zhai.,(2007), Probabilistic models for expert finding, ECIR, pp.418-430.

[22] Zhang, Z., et al., (2013), User community discovery from multi-relational networks. Decision Support Systems, 54(2): p. 870-879.

[23] L. A. Adamic and E. Adar. (2005).How to search a social network. Social Networks, 27(3):187–203.

[24] Maamar, Z., et al.,(2011), Using Social Networks for Web Services Discovery.

Internet Computing, IEEE. 15(4): p. 48-54.

[25] C.T.Li,M.K.Shan,&S.D.Lin,(2011), Context-based People Search in Labeled Social Networks,ACM International Conference on Information and Knowledge

在文檔中社群評鑑機制之求職推薦應用 (頁 50-0)