The chapter first describes the challenges of automated GUI testing. It then introduces two well-known open source projects of automatic GUI testing. Finally, related works are described.
2.1 Challenges of automated GUI testing
Based on the automated GUI testing process, we divide it into two parts: reproducing the interaction between human and DUT, and verifying testing results. For a simple event, such as pressing a hardware key, it can be reproduced by inserting a proper event. However, for multi-touch and complicated gestures, it becomes non-trivial to reproduce the interaction between human and DUT due to the timing constraints. In addition to reproduce predefined events accurately, it is also challenging to verify results. There are three commonly used methods for verifying testing results: bitmap comparison, objects identification and optical character recognition (OCR). Although bitmap comparison is easy to implement, it is extremely sensitive to the changes of GUI. The sensitivity can lead to extra maintenance cost, especially for immature software, which is frequently modified over the period of development. Objects identification relies on system provided API (Application Programming Interface) to obtain object information of the screen. By compared with the reference objects, we can evaluate the correctness of the testing results. However, system-provided APIs are very different from OS to OS, which limits the portability of this method. In addition, not all OS provide enough APIs for objects identification. OCR is an alternative method to reduce sensitivity and increase portability, which converts scanned images into machine-encoded text.
However, OCR may be slow and inaccurate, especially for non-text content identification.
2.2 Sikuli
Sikuli is a framework [2-4] which automates and tests GUI applications by using images (screenshots) on multiple platforms. Sikuli framework includes Sikuli Script, which is a visual
scripting API for Jython. Sikuli Script supports three types of visual scripting APIs: Find, Actions, and Event Observation. All visual scripting APIs requires an image as an operating target. By using find-type APIs, Sikuli can find target's location, check the existence of target, and wait until the target appears or disappears. In addition, Sikuli can adopt action-type APIs to search and click target, hover mouse pointer to target, or drag-and-drop between two targets. Furthermore, with Event-Observation-type APIs, programmers can register their event handling function to wait target to appear or vanish, or wait for the changes of the GUI contents.
Sikuli framework also contains an IDE, which is an integrated development environment for writing visual scripts with screenshots. By using Sikuli IDE, engineers can easily write GUI test cases, execute the script, automate GUI operations on desktop and verify GUI elements presented on screenshot. Although Sikuli provides several handy functions to ease the process of GUI testing, it’s pixel-based image search and comparison mechanism can consume significantly system resources and prolong the testing process. As a result, Sikuli may not be able to apply to embedded systems with limited computation resources. In this work, based on Sikuli, we develop SPAG to support the automated GUI testing of embedded systems.
2.3 Android Software Testing
There are several GUI testing utilities for developers to test their programs, such as Android Instrumentation Framework provided by Android software development kit (SDK) and Robotium. Profiling codes are inserted into proper locations in order to collect necessary information at runtime. These utilities require partial or full source codes of tested applications and are not suitable for black box testing. On the contrary, SPAG does not rely on the availability of source codes. Monkeyrunner is another testing tool provided by Android SDK. This tool can reproduce predefined actions, such as key press and screen touch, by generating associated events. However, Monkeyrunner is sensitive to external interference,
especially under the circumstance that the DUT is in a heavy load condition. SPAG, however, monitors the status of AUT at runtime and dynamically changes the timing to issue commands.
Android 1.6 provides new accessibility features to help users with disabilities use GUI applications. The accessibility features also allow developers to create accessibility services that work in the background and receive notifications of various GUI events. For example, special events are triggered when the state of the activities is changed or some GUI components are focused. These events provide useful hints about the widget where the event originated, such as the type of widget and its text content. However, Android accessibility currently does not allow programmers to list the full contents of the screen. This omission clearly limits the usefulness of accessibility because a GUI screen might some important widgets, labels or objects that are not focusable.
Hierarchy Viewer is the other utility provided by Android SDK that allows application developers to examine the layout of the Android GUI. Hierarchy Viewer communicates with an Android emulated device through the Android Debug Bridge (adb). Hierarchy Viewer provides the detail information of GUI layout, such as the ID, type, text content, location and size of all the GUI widgets in the screen. However, Hierarchy Viewer can only run on the top of an emulator, which limits its applicability.
2.4 Related Work
They have been many research efforts dedicated to automated GUI testing. The most common approach of automated GUI testing is model-based testing (MBT), which models the behaviors of target software and then uses the test cases generated from the models to validate the DUT[5] [6] [7]. T. Takala et al. adopted Monkey and Window services to generate GUI events [6]. L. Zhifang [8] utilizes the concept of virtual devices to test applications. Their method relies on image-based pattern matching which is sensitive to the quality of images. On the contrary, SPAG uses GUI components for pattern matching in order to improve the
stability and the speed of the validation.
Constructing a universal testing framework has been discussed in [8-11]. Several techniques and architectures were developed to realize complex application test. MoGuT [8], a variant of the FSM based test framework, used image flow to describe event changes and screen response. However, it lacks flexibility. Gray-box testing adopted APIs to construct calling context and parameters from input files [11]. Based on a logging mechanism, the gray-box testing verifies testing results. This method is simple and powerful for testing predictable software components. However, for complex software, it becomes difficult to describe the testing logic and calling context. MoibleTest [7, 9, 10] is a SOA based framework, which includes extendable script interpreter, universal communication interface and agent-based testing mechanism. Although MobileTest can be used for testing mobile devices, the accuracy of the test results is not clear.
Accessibility technologies provide different aids to disabled computer users. Most computing platforms nowadays support accessibility functionalities because it is mandated by the law [12] in U.S.. For GUI testing, the accessibility technology is a useful mechanism that provides an interface for programs to access to GUI objects. Grechanik, et al. [13] used accessibility technologies to obtain GUI structure information of the GUI-based applications (GAPs) for maintaining and evolving test scripts. They also used GUI metadata to generate programming objects in order to automate GUI testing [14] on windows. These research results demonstrate that accessibility technologies are applicable to black-box GUI testing.
Recently Chang, et al. [4] claimed that accessibility API can be used to assist pixel-based GUI interpreting in order to obtain a more accurate association between the visual representation and internal structure of a GUI. However, the major limitation of the accessibility-based methods is coverage, which differs from machine to machine. For example, accessibility API on Android 1.6 does not allow developers to list the full contents of the screen. A GUI view might also contain certain widgets or labels that are not focusable, and are thus inaccessible.
As a result, there is a clear need to develop a new method to overcome the limitation.