In this chapter, we first show the software testing process of modern times and the
problem that we are going to solve. An automatic method of testing process is introduced in
next section. Finally, we briefly state our motivation and objective.
1.1. Software Testing
Software quality nowadays seems crucially essential. In software engineering, the most
significant process to ensure software quality is testing. The key point that how to carry
testing process out yet depends on the testing strategies. Figure 1 shows a general type of
testing. It consists of three stages: input testing cases, execute programs and compare outputs
with correct results.
Figure 1, General testing process
In the last stage, we usually perform the comparison of results manually. But it is very
expensive to verify the program outputs manually. The Quality Assurance Institute in 1995
proposed a comparison of manual and automatic testing which introduced about 1750 testing
cases and 700 defects. Table 1 shows the result about time consumption for testing processes
(in hours). The improvement of the test case development, test execution and test result
analyses are 55%, 95% and 50%, respectively.
Table 1, Manual vs. automated testing
Test step Manual testing Automatic testing Percent Improvement
Test plan development 32 40 -25%
Test case development 262 117 55%
Test execution 466 23 95%
Test result analyses 117 58 50%
Defect tracking 117 23 80%
Report creation 96 16 83%
Total hours 1090 277 75%
It encouraged us to develop ALERT, an automatic testing tool that automates these test
steps including test case development, test execution and test result analyses. In order to
generate test cases, ALERT must handle the logic information about program source and
static semantics. In addition to generating test cases, the creation of test interface is required
to automate test execution. To demonstrate all system functionalities in a direct way, we have
to test every component one by one, such a method named unit testing. Finally, due to the
automation of the whole test process, we need to generate test case for the remained paths
when performing test result analyses, imagining by figure 2.
Figure 2, Unit testing with inputs generation
1.2. Test Strategies
Traditionally, test strategies method divided into two classes, static and dynamic analyses,
based on the source code and the runtime execution, respectively. Both have been maturely
developed in modern times. Static analyses examine program source code or binary image
without invoking the programs. It is recommended to apply static analyses to program
systems with complex interaction or heavy running overhead. As a consequence, the static
method is extensively used in large scale software for inconsistency checking. Yet, the static
method could cause large number of false alarms. It is because we can’t verify and perform
any feature revealed through static analyses. Static analyses report too many features that the
practicability of those features is unknown.
Dynamic analyses involve the execution of programs. We apply the dynamic analysis on
programs whose environment can be simply reconstructed and the runtime overheads are
tolerable. Obviously, the difficulties of the dynamic method are test input development.
Appropriate test inputs enable program states to shift to some specified condition that exactly
satisfies the constraints to perform features. The quality of test cases decides the efficiency of
testing tools. It is an extremely difficult work to manually generate testing inputs upon
thousand of testing runs.
1.3. Motivation
If we can find the input cases for a given code location and environmental content, we
are able to eliminate the false positives of static method by examining the feasibility of each
reported feature. For dynamic method, with the support of test cases which enable programs
to perform the remained paths, we can improve the efficiency of whole analysis process, with
better coverage.
With the automatic generation of particular inputs from another running instance, we can
easily apply debug method with only failure running instances. It is an advanced integration
of testing and debugging that we resolve the program run-time logic and produce special test
cases for debugging tools to make decisions more precisely.
1.3.1. Feasible Inputs
We propose a mechanism to automatically generate test cases and ensure feasibility of
given paths. Static methods check source code consistency, but if we want to suppress false
alarms we must execute programs and check the features reachable. On the other hand,
dynamic methods execute programs many times, and require test inputs to go through all
branches. In order to generate feasible inputs according to program semantics, we base on
program logic to transform programs into special models for of possible test case resolution.
1.4. Objective
Our objective is to extract logic models from execution instances and create test cases
that cover as many as possible control branches by checking logic models, as figure 3 shows.
We focus on C language, which is one of the most popular imperative languages and present
the logic models with rule-based language. Clearly, there is some gap between imperative
language and rule-based language, and it is impossible to transform imperative programs into
equivalent rule-based models. However, we discover that it is feasible to make a one-way
transformation from run-time instances of imperative programs into rule-based ones.
Figure 3, Unit test with model extraction
We perform the test process by reasoning on the program behaviors, similar to manually
trace-back of program logic, interleaving with symbolic evaluation and concrete evaluations.
Two goals will be fulfilled: logic model extraction and run-time predicates resolution.
1.4.1. Logic constraints extraction
We transform a program running instance view to a list of rules which have the ability to
represent the dynamic states upon codes. We are going to build an executable logic model that
can be evaluated exactly like the original program running. There is still one difference: an
executable model doesn't surround by real program environment. To fix this, the incomplete
predicates of logic models would be resolved by symbolic evaluation of runtime traces.
1.4.2. Runtime predicates resolution
What we want to resolve during logic model construction are those variables and
date-structures that exist in concrete memory space, including function arguments took by
procedures and addresses that pointers refer to. Because logic rules can not express the state
transactions of concrete memory space, we mark those predicates as unknown symbols and
resolve their value dependencies by symbolic evaluation.
We can evaluate each line of source code, and substitute the symbolic meaning of
variables. Of course, the resolving operation is not invertible. We expect to show how the next
state comes from according to the current state. It is used to illustrate running instance of
program iteration.
1.5. Synopsis
The rest of our thesis is organized as follows. The related works and state of the art are
reviewed in chapter 2. The overview and basic concept of ALERT are discussed in chapter 3.
Chapter 4 presents the detail and implementation of ALERT. The result and evaluation of our
work is showed in chapter 5. In the last chapter is the conclusion and future work.