Related Implementations

Here we introduce two implementations of soware testing framework: D-Cloud and ETICS.

ey are both designed for automatic testing, and want to give a solution for developers to figure out if there are some defects in soware systems. We also summarize the cons of these two implementations, and try to solve these inadequacies in our design.

2.4.1 D-Cloud

D-Cloud [22] is a soware testing environment using cloud computing technology and virtual machines, proposed in 2010. As shown in Figure 6, D-Cloud takes a pre-defined scenario as input. e testing environments such as hardware, soware, or network are set through structured configurations. D-Cloud sets up a test environment on the cloud resources and executes tests automatically according to a given scenario. It provides fault injection facility to test the system working status and performance.

Figure 6: Flow of D-Cloud (source: D-Cloud paper)

2.4.2 ETICS

ETICS [23, 24] is an integrated infrastructure for the automated configuration, build and testing of Grid and distributed soware, proposed in 2007. e architecture of ETICS is shown in Figure 7. It provides a service for soware projects by integrating well-established procedures, tools and resources in a coherent framework and adapting them to the special needs of distributed projects.

Figure 7: e ETICS System Architecture (source: ETICS paper)

2.4.3 Inadequacies

Both the designs of D-Cloud and ETICS have following cons.

• Applicable testing platforms are restricted to Windows and Linux systems.

• Testing targets are restricted to non-interactive soware that don’t need human manipulations, such as web or database servers.

• Do not support GUI soware testing since it usually needs interaction.

• Cannot accept parameterizable inputs, so users need to make another test case when they just want to change some factors.

• Testing functionalities are non-extensible or hard to extend.

Chapter 3 Method

In order to build a regression testing framework meeting our needs, we first illustrate the scenarios. By conceived the scenarios, we can know what requirements are needed to be provided in our system, such as what factors should be parameterized for flexibility and extensibility, what procedure should be designed differently for different testing targets, and what material should be provided to build a new test case.

With comparison to the current method, we also know what processes are needed as well as what processes could be pruned to save costs. And then we can start to build our system, design the architecture of the regression testing framework, and choose the techniques to be used.

3.1 Scenarios

For future extensibility, the applicability of the regression testing framework not only focus on current testing targets like Windows and Linux, but we hope it can also apply to more targets that use QEMU as a testing environment. We enumerate several testing targets here to demonstrate the using context but not limited to these scenarios.

3.1.1 Windows and Linux

We test vulnerable soware on Windows and Linux, and trying to find usable exploit by feeding crash inputs to them.

On the whole, the testing processes taken in Windows and Linux are describe in section 1.2.

At stage 3, we need to take operations in guest OS, through Linux command line via SSH login session or via VNC [25] (Virtual Network Computing) session if we are required to operate GUI environment.

In the soware development cycle, we may change our symbolic or exploit generation tools to make it works efficiently, or we want to change the crash input file, GUI operation order, soware startup parameters, etc. We will make these factors to be parameterized in our framework design.

3.1.2 Web Application

Automatic web testing method is also mainly based on symbolic execution to automate the web exploit generation process. Symbolic socket [26] is used to propagate symbolic execution through socket between applications. In QEMU, we set up web and database (DB) server with target web application running on it. Symbolic socket is implemented by replacing each crawled HTTP request with symbolic data which will be sent to web and DB server. In the other hand, there are also corresponding handlers for HTTP and DB query response to transferring received data to exploit generator at host OS. e overall architecture is shown in Figure 8.

QEMU

Figure 8: Overall architecture for web automatic testing

In web automatic testing, we will change the input query from web client, and the handler behavior may be changed, too. So make these part tunable in the framework will let the testing process reusable.

3.1.3 Android

As the smartphone market arising, there are more and more mobile applications (mobile apps) appear every day. is booming industry indicates the necessity of soware testing.

e app developers need to test their product work normally no maer what version of mobile OS^* or apps are. Also, mobile soware distribution platforms providers need to know whether the apps uploaded by developers are vulnerable or risky.

S²E QEMU (x86) Android (x86) App

(dalvikVM)

App (dalvikVM)

App

(dalvikVM) ...

Figure 9: e architecture of x86-Android on top of S²E

In Figure 9 we can see, by symbolic execution and fuzz testing, we can satisfy these requirements.

To run test on all versions of mobile OS and for each release of apps is nearly impossible in the past. Developers can only choose some popular combination to test if their product may work well. But with a regression testing framework, the problem are solved.

We can build the test cases one time, and run them easily whenever we make changes in a mobile system, either OS version, apps version, or other factors. e regression testing framework will be great beneficial to the mobile apps industry.

Since Android is an open source system and its marketing policy is open too, we will take Android apps as our testing target. By verifying apps not harm to us, users can get more protection.

*Mobile OS: i.e., mobile operating system. e most common mobile OS are Android, BlackBerry 10, iOS, etc.

3.1.4 Fuzzer

Fuzz testing is one of the common techniques of soware testing. It oen threats the tested applications under test as a black box, so it’s especially useful in analyzing closed source soware and proprietary systems since in most cases it does not require any access to the source code.

Fuzzer is a kind of fuzz testing tools, which will generate data or events to repeatedly feed the application with random input. e fuzzing result will be a valid input to the application, but will cause the application work improperly or even terminate unexpectedly. If the soware crashes, we can use the input, i.e., crash input as material in automatic exploit generation. By feeding appropriate data as input instead of using truly random sequence, we are more likely to find soware oversights or vulnerabilities. Improving fuzzing method is what the fuzzing technique are dedicated to.

We would be pleased if there is a testing framework that can help us test the efficacy and usability of new fuzzing method. If we run fuzzer in QEMU environment, we can achieve this.

3.1.5 Malware

Malware is soware used by aackers to disrupt computer operation, gather sensitive information, or gain access to private computer systems. ey may have bugs because they are also soware, but current researches rarely focus on this field.

We are interested in find malware vulnerabilities and usable exploits cause that most of the malware users only focus on ”aack” but not ”defense”. If we successfully find exploit worked on malware, we may have the ability to anti-aack.

On the other hand, one common way of anti-malware strategies is using anti-virus or anti-malware soware. e method of detecting malware aacking is oen by some specific behavior or functioning paern. e detecting technique may also need regression testing aer applying new method.

3.1.6 Embedded System

Embedded system is the most commonly present forms of computers in the world. e method of factory and acceptance testing in development period is usually black box testing by human or component unit testing on specific functions. If we can run embedded system soware on QEMU-based emulator, we can have a reusable and quick testing method through our regression testing framework.