E VALUATION OF HSER MODEL

In order to validate our HSER model of different smartphone, we conducted another round of survey. We prepared several video clips about eight operations, shown as Table 4, under normal CPU utilization on three different smartphones. They are HTC hero with 2.2.1 Android platform, Huawei U8860 with 2.3 Android platform, and Nexus S with 4.1.2 Android platform. In this survey, we had 45 volunteers to grade each video with “smooth” or “non-smooth” and used the formula (1) and (2) to compute the questionnaire result, denoted as , for each smartphone. The s are shown in Figure 13(a), in which Huawei U8860 is smoother than HTC hero and Nexus S. The 95% confidence interval of each survey is also shown in Figure 13(b).

Consider that the online questionnaire have the influence of network delay, we collected 10 volunteers to grade each video with the offline questionnaire. As a result, the influence of the offline questionnaire results, whose ranges are in 95% confidence interval, is lower than 10% shown in Figure 13(a). Therefore, users have good judgment even in the circumstances with network delay. We then adopted our regression result, shown in Table 7, to evaluate the smoothness of each smartphone.

As Figure 14 shows, the error rate, which is the error between and predicted result ( ) from our model for each smartphone, is obtained by

| |

| | .

Our HSER model can have 10% error rate below for each individual behavior.

32

(a) The satisfaction of smartphones (b) The confidence interval of smartphones

Figure 13 The satisfaction and confidence interval of smartphones

Figure 14 The error rates between the models 55%

33

Chapter 6 Conclusion and Future Work

In this work, we developed the handheld smoothness evaluation over regression (HSER) model to fairly benchmark the smoothness of smartphones. We first measured BQoS by extracting key indexes. They are the mean of frame intervals (MFI), variance of frame intervals (VFI), maximal frame interval (MaxFI), frame no response (FNR), times of maximal frame interval (TMaxFI) and the number of frame intervals (NFI). Since the indexes may not always be measurable, especially when the changes between frames are fast, we further developed a tool, named extract device operation sequence (Ex-DOS), to obtain necessary information. Based on obtained behavior-based smoothness quality of services (BQoS), we then designed a questionnaire to determine the relationship between BQoS and behavior-based smoothness quality of experience (BQoE). Finally, we converted the BQoE to handheld smoothness QoE (HQoE) by considering how frequently each behavior is performed in daily life.

In order to evaluate the effectiveness of the proposed method, we conducted several experiments on three different smartphones, HTC hero, Huawei U8860 and Nexus S. We investigated the applicability of the HSER model in different user scenarios. Some user scenarios are timing sensitive while others are not. We validated the correctness of the HSER model by comparing it to our questionnaire results.

According to our experiment results, the correlation of MFI, VFI, FNR and TMaxFI is higher than 71.5% in logarithmic relationship. To avoid the collinearity problem, MaxFI and NFI are used to be the indexes for our HSER model. MaxFI and NFI also are good indexes for the “non-smooth” situations of the long waiting time and the fragmentary frames. For individual behavior, the average is close to 1. In

34

particular, for the behavior of viewing gallery and playing game, the is up to 0.986 and 0.973. Also, the error rate of HSER is less than 9%. It implies that our regression model can be used to fairly evaluate the smoothness of a smartphone. In addition, the error rate of HTC hero (9%) is higher than other two smartphones (5%).

The reason may be the variation that users grade the videos with “smooth” or

“non-smooth”. The same video for different users will get the different perception.

In the future, we plan to investigate other indexes and collect more users’

experience in order to further enhance the accuracy of our model. Possible indexes include the speed of fling and scroll operations. We also plan to improve the accuracy of Ex-DOS tool by detecting non-static objects in a video.

35

References

[1] Dong Tian, Lansun Shen, and Zhiheng Yao, “Motion Activity Based Wireless Video Quality Perceptual Metric,” Proceedings of 2001 International Symposium

on Intelligent Multimedia, Video and Speech Processing, 2001.

[2] Ruifen Xu, Mingwei Zhang, Lin Jia, and Yongli Li, “A Novel Algorithm for Video Smoothness,” Proceedings of Evaluation Advanced Computer Theory and

Engineering, August 2008.

[3] Rohani Bakar, Muhammad Ibrahim, and D.M. Ali, “Performance Measurement of VoIP over WiMAX 4G Network,” Proceedings of IEEE 8th International

Colloquium on Signal Processing and its Applications (CSPA), March 2012.

[4] Yu-Chun Chang, Kuan-Ta Chen, Chen-Chi Wu, Chien-Ju Ho, and Chin-Laung Lei, “Online Game QoE Evaluation using Paired Comparisons,” Proceedings of

IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR), June 2010.

[5] Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei, “Effect of Network Quality on Player Departure Behavior in Online Games,” Proceedings of IEEE Transactions

on Parallel and Distributed Systems, May 2009.

[6] Kuan-Ta Chen, Polly Huang, Chin-Laung Lei, and Chun-Ying Huang,

“Quantifying Skype User Satisfaction,” Proceedings of ACM SIGCOMM 2006, September 2006.

[7] Hyeon-Ju Yoon, “A Study on the Performance of Android Platform,” Proceedings

of International Journal on Computer Science and Engineering, pp. 532–537,

[8] AnTuTu-Benchmark. [online], available from World Wide Web;

https://play.google.com/store/apps/details?id=com.antutu.ABenchMark [9] SmartBench. [online], available from World Wide Web;

https://play.google.com/store/apps/details?id=com.smartbench.twelve

[10] Quantify and Optimize the User Interactions with Android* Devices. [online], available from World Wide Web;

http://software.intel.com/en-us/articles/quantify-and-optimize-the-user-interaction s-with-android-devices

[11] Miller, R. B., “Response Time in Man-computer Conversational Transactions,”

Proceedings of Fall Joint Computer Conference, 1968.

[12] Jakob Nielsen, “Usability Engineering,” published by Morgan Kaufmann, San Francisco, 1993.

[13] Android development. [online], available from World Wide Web;

36

http://developer.android.com/training/articles/perf-anr.html

[14] Screencast Video Recorder Demo. [online], available from World Wide Web;

https://play.google.com/store/apps/details?id=com.ms.screencastfree

[15] Chung-You Wei, Jen-Wei Hsieh, Tei-Wei Kuo, I-Hsiang Lee, Yian-Nien Wu, and Mei-Chin Tsai, “Resource Reservation and Enforcement for

Framebuffer-Based Devices,” Proceedings of Ninth International Conference on

Real-Time and Embedded Computing Systems and Applications, 2003.

[16] Yu-Chun Chang, Kuan-Ta Chen, Chen-Chi Wu and Chin-Laung Lei, “A Crowdsourceable QoE Evaluation Framework for Multimedia Content,”

Proceedings of ACM Multimedia, October 2009.

[17] Android keylogger (AKL). [online], available from World Wide Web;

http://www.ebl.org.tw/.

[18] Free Video to JPG Converter Tool. [online], available from World Wide Web;

http://www.dvdvideosoft.com/products/dvd/Free-Video-to-JPG-Converter.htm [19] Nikhil R Pal and Sankar K Pal, “A review on image segmentation techniques,”

Pattern Recognition 26 (1993) 1277–1294.

[20] H. Verkasalo, “Analysis of Smartphone User Behavior,” Proceedings of IEEE

2010 Ninth International Conference on Mobile Business and 2010 Ninth Global Mobility Roundtable (ICMB-GMR), 2010.

[21] Odewale J.O., Agho Collins, Ataga C.D., Odiowaya G., Hamza A., Uwadiae E.O.

and Ahanon M.J., “Study of Some Fruit and Seed Traits Relationship and Assessment of Multicollinearity in Date Palm (Phoenix Dactylifera L) Accessions of Nigeria by Correlation and Principal Component Analysis, ” Proceedings of

Greener Journal of Agricultural Sciences, ISSN: 2276-7770 Vol. 3 (2), pp.

164-175, February 2013.