Automated GUI (Graphical User Interface) testing tools are software programs used to test application user interface and to verify the functionalities. In the process of testing embedded software, engineers first design a test case consist of operations, which include several GUI operations and a set of conditions devised to determine whether an application works correctly or not. After engineers convert the test cases to a script file, the script performs predefined operations on a device under test (DUT), such as a smart phone or tablet PC. To verify the result, the DUT captures the screen and sends it to host PC, where an automated GUI testing tool performs verify operation. Take a popular open source automated GUI tool Sikuli, an Android device controlling tool AndroidScreenCast, and an Android smart phone for examples. Software engineers first write a Sikuli script to describe the timing and order of GUI operations, such as scroll screen and key press actions. At runtime, each action of the Sikuli script is performed on DUT screenshot window provided by AndroidScreenCast. These actions are interpreted into multiple motion events and key press events and transmitted to an Android smart phone, which is the DUT. After performing all received events, the AndroidScreenCast captures the screen of DUT and sends it back to the host PC, where the Sikuli verifies the correctness.
However, due to the uncertainty of runtime execution environment, such as timing delay variation in communication, interpreted events may not be reproduced at the DUT on time. As a result, intervals between events may be different from those expected. The non-deterministic event sequences may lead to an incorrect GUI operation. For example, Android Fling action happens when user scrolls on touch panel and then quickly lifts his finger. A sequence of motion events is used to represent the action. When replaying these event sequences, each motion event should be triggered on time in order to reproduce the Fling action with the same scrolling speed. Otherwise, the scrolling speed of the reproduced
filing action will be different from what is expected and therefore result in an incorrect result.
In order to address the issue of non-deterministic events, a commonly used method is to use trackball instead of Fling action. However, trackball is not always equipped with a smart phone.
Uncertain runtime execution environment may cause another problem because it may interfere or delay the execution of application, especially under the circumstance that the DUT is in a heavy load condition. A delayed application may fail to process an event correctly if the response to previous event is not yet completed. For example, an event may be dropped if AUT receives the event ahead of time and is not ready to process it yet. To solve the problem, an intuitive method is to delay the executing of the operations. However, it requires experienced engineers to set the delay for each operation properly, so that the application can receive the reproduced events.
The other problem is how to verify test results efficiently. Traditional pixel-based takes relative long time on image transferring and processing. The situation becomes even worse for smart phones due to limited computation and communication capacities. This creates a strong need to develop a new method for automatically verifying the response of GUI operations.
In summary, based on our observations, automated GUI testing for smart phones faces three major challenges: non-deterministic events, execution interference and non-deterministic layout. In order to overcome the challenges, we design a Smart Phone Automated GUI testing tool (SPAG), which is based on Sikuli. To avoid non-deterministic events, we batch the event sequence and reproduce them on the DUT. In addition, SPAG can monitor the CPU usage of target application at runtime and dynamically change the timing of next operation so that all event sequences and verifications can be performed on time, even though the DUT is under a heavy load condition. Finally, SPAG adopts Android accessibility service to get necessary information for verifying DUT status. While cooperating with
traditional image matching method, SPAG can automatically generate most verify operations and check the testing results faster.
We conducted several experiments on a popular Acer Liquid smartphone in order to investigate the applicability and performance of SPAG. We compared our method with monkeyrunner[1]. In our experiments, we first investigated the effect of resource utilization of system during test. In addition, we studied the improvement of test accuracy achieved by our batch event and smart wait method. Finally, we explored how our hybrid verifying method reduces the cost of generating test cases.
The rest of the work is organized as follows. Chapter 2 mentions a GUI automated tool named Sikuli, and gives a survey of related works. Chapter 3 gives the definition of variables we used in this work, and describes our statement our problem. Chapter 4 details our solutions of batch event, smart wait and hybrid verifying. Chapter 5 is the implementation details of these three solutions based on Android system. Chapter 6 presents the experiment results and discussion. Finally, Chapter 7 concludes this work and future works.