1.3 Difficulties of Testing Concurrent Programs

Chapter 1 Introduction

Software testing is an important process in software quality assurance(SQA). For software development, high-quality, defect-free are the general goal for all developers.

And the cost of software testing can exceed the cost of design and programming. This is the reason that the methodologies, techniques and tools used for testing are the keys to efficient development of high-quality software.[1]

1.1 Testing and Debugging

Program testing can be used to show the presence of bugs, but never to show their absence.[2]

-- E. W. Dijkstra

As one of the key process during software development, software testing is a process used to identify the correctness, completeness and quality of developed computer software. Actually, testing can never establish the correctness of computer software. It can only find defects, not prove that there are none.

Unlike testing intend to determine the program contains defect or not, the purpose of debugging is locating and fixing the defect code responsible for a symptom violating a known specification. Typically, the debugger need to cooperate with testing tools and repeat the execution with the modified code. And finally, the debugger can confirm that the defect code is fixed properly.

Figure 1.1: Static testing , dynamic testing and debugging in a typical software development process.[3]

Existing software testing methodologies are divided into categories: static testing and dynamic testing.[4] In static testing, the requirements documents, design documents and the code are analyzed, either manually or automatically, without actually executing the code. Due to the testing results are not acquired at the runtime, such testing is also called static analysis or just analysis briefly. In dynamic testing, the requirements and program is actually executed in a controlled and systematic way to demonstrate that required functions are present. In this paper, we call dynamic testing as testing briefly if there is no indication accompanied.

System Requirements

Software Requirements

Analysis

Design

Coding

Testing

Production/Deployment Dynamic

Testing Verification

Debugging Static

Testing

With the rapid development of processor during these decades, the cooperation among the devices with different processing speed becomes important. Concurrent programs are widely implemented to meet such requirements and provide better response time. Therefore, such as multitasking operating systems or lots of sever programs, like: web servers, database servers, etc. are all the concurrent programs. For a concurrent program, more than one processes are executed concurrently on one host computer and each process may compete for some shared devices, such as memory space or communication channels to accomplish their tasks. Thus, the outcome of the execution depends on the particular order in which the access takes place. Such is called a race condition.[5]

Oppose to concurrent programs, if programs are designed such that the execution outcomes are independent to the execution order of accesses within processes, we call such programs are sequential programs.[6] And we can treat a concurrent program as a set of ordinary sequential programs, which are executed in abstract parallelism and communicate via shared variables, synchronization signal, message passing or all the other possible kinds of interprocess communications. To distinguish from the sequential programs without any form of interprocess communications, we define that a concurrent program is composed of a tuple of sequential processes.

Definition: A concurrent program P is an N-tuple (p1, p2, p3, ..., pn), which each pi is a sequential process. All the possible and legal input set of a concurrent program P, denoted INPUT(P), is the set of inputs which users expect P to execute on. If P=(p1, p2, p3, …, pn) and X∈INPUT(P), then X can be represented as X=(x1, x2, x3, …, xn) where xk is the input for the k^th process.

Figure 1.2: The construction of a concurrent program.

As the race condition that has been mentioned previously, the outcome of execution depends on the particular order of the operations which access to some shared variables.

As [7] has mentioned, the relation of these operations would be an irreflexive partial ordering. And some certain partial ordered sequences maybe as requirements to produce the expected execution results. Thus, we can say that these operations need to be executed with some partial order synchronously. Such operations would be called as

"synchronization operations". The required partial order to produce expected results can be a set sequences to each processes within the concurrent program, and we will call the set of sequences as “synchronization sequence”, or just SYN-sequence in brief.

1.2.1 Repeat Execution and Nondeterministic Behavior

Before introducing repeat execution of concurrent programs, let's see how the repeat execution take effect when testing sequential programs. Unlike concurrent programs, all

A Concurrent Program Sequential Process

Input Sequential Process

Input Sequential Process Input

...

Sequential Process Input

sequential program with the same input data repeatedly, all the outcomes would be the same. That is, the execution of sequential programs is deterministic.

Figure 1.3: Repeat execution testing method.[6]

Repeat execution testing method is effective only if the execution of target program is deterministic, and the erroneous behavior can be reproduced easily with the same input. But it would be useless when we change the target to concurrent programs. First, the erroneous behavior is usually unreproducible since no execution sequences would be recorded accompanied with executions. Second, the execution sequences can be various but only a little fraction of them would be exercised.

Finally we know the fact that when exercising a concurrent program, the partial order of these synchronization operations can be any combination of them if there is no special execution control tool or processes scheduler of operating system. Therefore, the outcomes of concurrent program are also cannot be determined, same as the

Execution (arbitrarily) Modified User

Program after Compiling

Event History Execution Output

Expected Output

Verify Test

Debugging

Chosen Input

correct incorrect

SYN-sequences are. Such characteristic of concurrent programs is called as nondeterministic behavior.

1.2.2 Probe Effect

Since repeat execution is useless without SYN-sequences as one part of input, how to determine and record SYN-sequences at within runtime is the next topic would be considered naturally. Recall that when we intend to measure the status of object, any operation of measurement would be intrusive to the object. Therefore, when we do one measurement precisely, the successive measurement would be inaccuracy. This is called probe effect or Heisenberg uncertainty principle. It also exists in the monitoring of concurrent programs. In software engineering, debugger need to collect sufficient information to reproduce the erroneous behavior. With software approaches, all of them will consume CPU processing time and alter the original processes scheduler. Thus, it becomes on the horns of dilemma that both collecting information and avoiding intrusion to the target concurrent program.

1.3 Difficulties of Testing Concurrent Programs

According to the characteristics of nondeterministic behavior and probe effect have been mentioned previously, testing concurrent programs becomes a difficult task during software verification and validation. First of all, due to nondeterministic behavior of concurrent programs, the SYN-sequences are must be as one part of input. This kind of method is also called deterministic testing and the erroneous can be reproducible. But

combine the deterministic testing and nondeterministic testing, and can exercise the execution path coverage exhaustively.

Beside dynamic testing, here we will discuss another method called analysis. As we have mentioned previously, analysis will not execute the programs actually.

Therefore, it need to review the source code of target program and generate one global finite state machine(FSM). Finally, by examination of each state from the global FSM we can ensure the outcomes of actual execution will corresponding to our expected ones. Obviously, analysis avoids the nondeterministic behavior and probe effect during execution. And the global FSM can be generated from the source code, thus the execution path coverage is depend on the way to generate global FSM. These are all the advantages of static analysis. But there is one critical problem: with the growth of target program, the size of global FSM will be explosive and difficult to handle. This is one of the main rezone to choose dynamic testing as our research.

1.3 Reachability Testing

Reachability testing is one method which combines deterministic testing and nondeterministic testing. It consists of two major phase: monitor phase and replay phase. Similar to repeated execution, reachability testing let the target programs execute arbitrarily first and record the SYN-sequences. But it is unavoidable that the record operation will be interference to the normal execution of SYN-sequence. That is the probe effect we have mentioned previously. Therefore, reachability testing design one mechanism called “race-variants analyzer” to derive all the other reachable race-variants from the recorded SYN-sequences. Finally, it can perform exhaustive testing by replay the target program with these race-variants.

Figure 1.4: Reachability testing method.[6]

1.3.1 Problems When Dealing with Busy-Waiting Loops

We can discuss the problems when dealing with busy-waiting loops with the two phase of reachability resting respectively. In the monitoring phase, because of reachability testing will let target programs executed arbitrarily, the execution may get into locked and size of recorded SYN-sequence can be explosive. This is a critical problem for all the testing methods. Furthermore, some of the locked condition may be exercised with tiny probability when executed arbitrarily and caused by some certain SYN-sequences. But these condition will not be ignore because that reachability testing do the exhaustive testing, the race-variant analyzer will derive all the possible

Execution Controlling

Tool Modified User

Program after Compiling

Event History Execution Output

Expected Output

Verify Test

Debugging

Chosen Input

correct incorrect

data feedback

1.4 Organization of Thesis

In this thesis, we will introduce the testing of concurrent programs and reachability testing. And what problems would be occur when dealing with busy-waiting loops. In chapter 2, we classified the occurrences of busy-waiting loops by their behavior. This is important that which one will be handled in the monitor phase and other will be handled by race-variant analyzer. In chapter 3 and chapter 4, we propose the new monitor protocol and mechanism of race-variant analyzer respectively. And finally conclusion future works will be in the chapter 5.