Performance evaluation of this study stresses on the time responsiveness and the CPU usage of tested e-readers. Response time is the main concern of a user since it affects directly his/her reading experience. CPU utilization is also a key evaluation parameter due to its immediate impact on power consumption and on the smart-phone condition.
To measure the performance, we used an iPhone3G [2] (8GB model) with latest OS version 3.1.3 and a performance analysis tool, called Instruments [1], from the iPhone SDK. In our testing environment, as shown in Figure 2.3, Instruments runs on a MacBook Pro with MacOS version 10.5.8. We use the “Activity Monitor” instrument to record a sequence of user actions and collect data over multiple runs.
“User manual” is the tested e-book for this study. This e-book is in EPUB format without DRM protection. The size of the e-book is 760 kB and the language is traditional Chinese.
To be fair in the comparison, we exported the “user manual” in advance and then imported it into Stanza e-reader via the Stanza Desktop application. Then we connected the iPhone with the MacBook Pro and launched these e-readers in multiple runs mode to gather data in Instruments application. We extracted the response time and the CPU utilization of each application. We repeated the measurements for at least ten runs for each set of test to obtain accurate data.
Figure 2.3 Testing Environment
The measured raw data provided by Instrument were exported into excel tables. Finally they are processed to get the comparison charts and results. In this study, we lead examination on four aspects of the e-reader applications.
2.4.1 Application Loading Results
As described in Section 2.1 and 2.2 the application loading process of Stanza and X-Reader is different. X-X-Reader application needs to login and synchronize with the application server each time the user launches the application. Stanza does not have this registration phase.
To compare fairly the application loading step, we distinguish X-Reader’s application loading phase and registration phase in order to ignore the latter one. To do that, we run tests under different network connections: WiFi, 3G and none (airplane mode). We clearly identify those two phases in Figure 2.4.
Figure 2.4 X-Reader Application Loading
By extracting phase one, the application loading phase, we can compare it with Stanza application loading. Figure 2.5 shows the test results.
X-Reader Stanza
Response Time 1.9s 1.7s
Average CPU utilization 51.7% 44.3%
Figure 2.5 Comparison of Application Loading
From the chart and the table, we observe that on the average data of ten measured runs, Stanza reaches its homepage faster than X-Reader, by 0.2 second. Stanza’s average CPU utilization is also lower by 17.4%.
2.4.2 E-Book Loading Results
When the user clicks an e-book on the bookshelf or Library, the e-book loading starts.
During the e-book loading step, the e-readers have to extract the same EPUB file and load the related files as described in Section 1.2.1.
Figure 2.6 shows the differences in response time and CPU utilization between the two e-readers under comparison. Note that as the application loading, Stanza still has better average performance of e-book loading than X-Reader by around 20%.
X-Reader Stanza
Response time 1.8s 1.6s
Average CPU utilization 45.8% 36.8%
Figure 2.6 Comparison of E-Book Loading 2.4.3 Chapter Loading Results
In an EPUB file, book chapters are separated into different .html files, as mentioned in Section 1.2.1. Figure 2.7 depicts the comparison of e-book chapter loading. For this test, the e-readers render the first chapter of the “User manual”. The results of loading other chapters are similar. Stanza seems to have once more better performance than X-reader on both time responsiveness and CPU utilization.
X-Reader Stanza
Response Time 1s 0.7s
Average CPU utilization 40 % 32.5%
Figure 2.7 Comparison of Chapter Loading
The response time of chapter loading depends on the size of the chapter file and if it contents images, since images are saved in different files. Obviously, a bigger file and a bigger number of images induce a longer response time on loading and rendering.
2.4.4 Page Loading Results
Table 2.4 shows that the page loading time on both e-readers is less than 0.2 second. Since, the data is loaded chapter by chapter, the page loading step does not involve any other file loading. This explains how short the response time can be even though both e-readers perform page-turning animations.
TABLE 2.4. PAGE LOADING COMPARISON
X-Reader Stanza
Response time 0.2s < 0.2s
Average CPU utilization 10.3% 45%
Another observation from Table 2.4 is that, contrary to the other testing steps, X-Reader has a far less CPU consumption than that Stanza on the page loading.
Generally speaking, the tests running on application, e-book and chapter loading steps show that Stanza have better performance than X-Reader. The time response of X-Reader is about 20% longer than Stanza’s. In the same way, the CPU utilization of X-Reader is higher than Stanza by around 20%. On page loading, the response time is about the same for both e-readers. However, X-Reader has a much lower CPU utilization.
This study aimed to compare e-book readers from both functionality and performance points of view. The two e-readers under comparison, Stanza and X-reader, show that they have only little functionality differences. The key feature of Stanza application is its capability to support many ways of importing e-books. X-Reader’s main strengths are its synchronization feature and its proprietary DRM protection. From the performance point of view, the two e-readers are comparable. In this study, we focused on the reading experience of an end-user. The result shows that reading a common e-book on the same iPhone, the global performance of Stanza is better than X-Reader.
We did not address the performance issue on mobility, because most e-readers can access the downloaded e-books without any network connection. The proposed paper highlights what has to be improved in e-readers. The evaluation method is also applicable to other mobile devices, e.g. Android. An e-reader developer should make a trade-off between performance and user-interface design. Furthermore, if the e-reader needs to support DRM protection, evaluating the induced delay is essential to prevent from impacts on the reading experience.
Chapter 3
User Experience of E-Readers for NCTU Students
User experiences and opinions are valuable elements to evaluate e-readers. Many studies about the usefulness of e-readers among university students have been conducted in recent years [16, 24]. This chapter collects the results of two survey studies about the e-reader user experience among NCTU students. Those studies concern users’ reading behavior of e-books as well as their opinions about e-readers.
3.1 Context of the Studies
The first and the more thorough study is done thanks to the participation of the students of the undergraduate course “Mobile Application and Services” taught by Prof. Lin in the spring semester 2010. During this course, 27 students answered a survey about their reading experience and behavior towards e-books and electronic resources in general.
Moreover, students had to experience the use of different kind of e-readers either on smart-phone and on dedicated devices. This chapter collects their feedbacks and user experiences. The expected results focus mainly on if e-readers are easy and convenient to use. The conclusion suggests solutions and gave ideas about what can be done to make e-readers better.
The second study is done thanks to the collaboration of NCTU library. Indeed, the library provided smart-phones for loan in order for NCTU students to experience reading e-books through smart-phone applications. We collected their opinions and user experiences through a survey. For that purpose, an event was created with the support of NCTU library to target more NCTU students to answer the survey. The agenda of the event and the survey can be found in the appendix of this thesis. Finally, 46 students took part in the event.