In this subsection, we collected the information from workers who participant in our HITs assessment and analyze with Panos Ipeirotis‘s demographic data [9, 10]. We focus on four attributes, which are workers‘ Country, Gender, Age and Education level. Figure 11a shows
5 Experiment 19
(a) Workers Age (b) spending time Fig. 10: AMT results 3/3
the changes of demographics composition, since 2008 until 2013, the Indians has become the major group of AMT workers, due to the AMT allow workers in India receiving their payment in rupees, only workers from US and India can have their payment by their own country currency. In figure 11b, male workers number has catch up females and almost overrun for double in 2013, the ratio for male and female workers in India does not have much change, but in US, the male workers number have also catch up with females, too.
(a) Country (b) Gender
Fig. 11: Demographics of AMT worker 1/2
Workers‘ age structure is almost the same which shows in figure 12a and 12b, 20 ∼ 40 year-old workers are still the main groups, especially 20 ∼ 30, this result is also consistent with figure 12c and 12d which demonstrate that bachelor level has the most workers number in AMT. In our study, from 2010 to 2013, there are more workers enter the master degree
5 Experiment 20
after they graduated from college in India (fig.12d).
(a) Age (b) Age
(c) EDU (d) EDU
Fig. 12: Demographics of AMT worker 2/2
6 Related works 21
6 Related works
Crowdsourcing is a hot technique during this Internet explosion and human overpopulation centry [8], it can access scalable workforce and distributed problem solving on-line [23], users only need to set the requests, prepare the funds and find the proper crowdsourcing platform, workers then handle the rest of works; there are many of crowdsourcing platform, such as IStockPhoto [11], uTest [21], TopCoder [20], PeerToPatent [18] and Amazon me-chanical turk. By choosing the correct platform and efficiently implement, crowdsourcing can have low cost and less time-consuming advantages. There are lot of research topics surround it, such as disable people assistant [4, 5, 12], activities learning [25, 26], real time process [3], speech recognition [16, 17], audio quality assessment by crowds [19]. and video annotation [15, 22]. Crowdsourcing is suitable for tasks which are easy for human, hard for computers, such as human activities recognition (AR) [14] using camera monitor-ing or body-worn surveillance with crowdsourcmonitor-ing, make human AR more deployable and extend conveniently. Human AR system needs training dataset, this is time consuming and repetitive process, via crowdsourcing, batches of training processes can be distributed to workers on the Internet, and be trained synchronously, which reduce the lengthy training procedure. Furthermore, model-driven crowdsourcing incisively trigger the crowdsourcing method in human AR [13], save the cost and time consumption to the lowest bound; they use model to predict the next possible activity, which is Hidden Markov model (HMM), only use crowdsourcing when the probabilities cannot support the predictions, based on this, system performance and cost reduction can also be improved. In CLAS, we propose DAWM-driven crowdsourcing, implement our system on AMT, to use crowdsourcing in human behavior assessment — students‘ concentration level.
7 Conclusion 22
7 Conclusion
In this study, we propose a model-driven based crowdsourcing method to assess the students concentration level in class, introduce the Doze-and-Wake model, which use its conditional probabilities to predict the students‘ concentration states, and trigger the crowdsourcing when model‘s probabilities are not support for the predictions. We implement our system on Amazon mechanical Turk and hired the non-restriction and untrained workers to assess the students‘ faces class video, through the majority voting, our results show that even the untrained workers can have reliability assessments. The results also show that our CLAS can have high accuracy with proper parameter setting, workers quantity and DAWM-driven crowdsourcing.
8 Future works 23
8 Future works
Our CLAS can output the students‘ or audiences‘ concentration level distribution during the class or a presentation. Based on the results, we can integrate CLAS with online course or class video online platform, help students who doze off in class to review the part of the lesson that they missed out, we can also give some advices to the lecturers, point out the abstruse / boring part of the presentations, and help them to have better performance for their next lectures. Due to the CLAS has advantages of easy deployment, we can build our system into office surveillance, help directors to manage their works, adjust the office break time or find the workers who work conscientiously for the future promotions, improve companys‘ productivity.
8 Future works 24
References
[1] Amazon. amazon mechanical turk.
[2] Amazon. Wake up anti-doze earpiece alarm.
[3] M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds:
Enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 33–42, 2011.
[4] J. Bigham, E. Brady, S. White, and C. Esposti. Human-backed access technology.
Proceedings of the CHI 2011, 2011.
[5] E. Brady, M. R. Morris, Y. Zhong, S. White, and J. P. Bigham. Visual challenges in the everyday lives of blind people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2117–2126. ACM, 2013.
[6] P.-Y. Chen, P.-H. Wu, W. J. Ong, Y.-J. Huang, W.-C. Lin, and T.-L. Pan. Development of a brand new system using rfid combining with wireless sensor network (wsns) for real-time doze alarm. In Anti-counterfeiting, Security, and Identification in Communi-cation, 2009. ASID 2009. 3rd International Conference on, pages 197–201, 2009.
[7] Y.-L. Chien. Eye opening detection with application for in-class attention monitoring.
Master’s thesis, National Taiwan Normal University, 7 2012.
[8] J. Howe. The rise of crowdsourcing. Wired magazine, 14(6):1–4, 2006.
[9] P. Ipeirotis. Mechanical turk: The demographics. 2008.
[10] P. Ipeirotis. Demographics of mechanical turk. Working paper CeDER-10-01, New York University, Stern School of Business. Available at http://archive.nyu.edu/handle/2451/29585, 2010.
[11] iStockphoto. http://www.istockphoto.com/.
8 Future works 25
[12] W. Lasecki, C. Miller, A. Sadilek, A. Abumoussa, D. Borrello, R. Kushalnagar, and J. Bigham. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology, pages 23–34.
ACM, 2012.
[13] W. S. Lasecki, Y. C. Song, H. Kautz, and J. P. Bigham. Real-time crowd labeling for deployable activity recognition. In Proceedings of the 2013 conference on Computer supported cooperative work, CSCW ’13, pages 1203–1212, New York, NY, USA, 2013. ACM.
[14] L.-V. Nguyen-Dinh, C. Waldburger, D. Roggen, and G. Tr¨oster. Tagging human ac-tivities in video by crowdsourcing. In Proceedings of the 3rd ACM conference on International conference on multimedia retrieval, pages 263–270. ACM, 2013.
[15] S. Nowak and S. R¨uger. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In Proceedings of the international conference on Multimedia information retrieval, pages 557–566. ACM, 2010.
[16] G. Parent and M. Eskenazi. Toward better crowdsourced transcription: Transcription of a year of the let’s go bus information system data. In Spoken Language Technology Workshop (SLT), 2010 IEEE, pages 312–317, 2010.
[17] G. Parent and M. Eskenazi. Speaking to the crowd: looking at past achievements in using crowdsourcing for speech and predicting future challenges. In Proceedings of Interspeech, 2011.
[18] PEERTOPATENT. http://peertopatent.org/.
[19] F. Ribeiro, D. Florˆencio, C. Zhang, and M. Seltzer. Crowdmos: An approach for crowdsourcing mean opinion score studies. In Acoustics, Speech and Signal Process-ing (ICASSP), 2011 IEEE International Conference on, pages 2416–2419. IEEE, 2011.
8 Future works 26
[20] TopCoder. http://www.topcoder.com/.
[21] uTest. http://www.utest.com/.
[22] C. Vondrick and D. Ramanan. Video annotation and tracking with active learning. In Neural Information Processing Systems (NIPS), 2011.
[23] M. Vukovic. Crowdsourcing for enterprises. In Proceedings of the 2009 Congress on Services - I, SERVICES ’09, pages 686–692, Washington, DC, USA, 2009. IEEE Computer Society.
[24] Wikipedia. Virtual learning environment.
[25] L. Zhao, G. Sukthankar, and R. Sukthankar. Incremental relabeling for active learn-ing with noisy crowdsourced annotations. In Privacy, security, risk and trust (passat), 2011 ieee third international conference on and 2011 ieee third international confer-ence on social computing (socialcom), pages 728–733, 2011.
[26] L. Zhao, G. Sukthankar, and R. Sukthankar. Robust active learning using crowd-sourced annotations for activity recognition. In AAAI 2011 Workshop on Human Com-putation, 2011.