Work on dynamic testing of concurrent programs without the assistance of static analysis and model checking includes several random testing techniques for concurrent programs (these techniques are also called noise makers). Although random testing for concurrent programs was proposed in the 1980s, a considerable amount of related work has been published since 2000. Stoller proposed a scheme that transforms a given Java program by inserting calls to a scheduling function at selected points [10]. The scheduling function either does nothing or causes a context switch. Because it does not control the schedule directly, it can only advise the scheduler to make a thread switch;
that is, it cannot force a thread switch. Ben-Asher et al. proposed a similar scheme, in which random context switching is performed at accesses to contended shared variables [11]. Edelstein et al. designed a tool to seed the tested program with some primitives at shared memory accesses and events [12]. At run-time, the tool makes random or coverage-based decisions as to whether the seeded primitive is to be executed, which increases the probability of finding concurrent faults. Sen’s team developed a tool for testing concurrent Java programs, called CalFuzzer [39,40], that can explicitly control the scheduler based on random schedules. The CalFuzzer applies dynamic partial order reduction to avoid the unnecessary interleavings for independent statements. In addition, it attempts to force schedules in which concurrency bugs can be detected. ConTest is a tool from IBM Research for testing multithreaded Java applications [41,42] that generates different interleavings to reveal concurrent faults by seeding the program with conditional sleep statements at shared memory access and synchronization events. Since it cannot guarantee that all the possible interleavings are explored, it is recommended
that the programmer runs the testing tool for a long time (e.g., overnight) in the hope of executing all possible interleavings [43]. Random testing techniques are generally not systematic schemes. The proposed schemes attempt to generate different interleavings as rapidly as possible, but they can neither guarantee state-cover testing nor determine if they have actually achieved state-cover testing during a testing procedure.
2.1. Reachability testing
Systematic and exhaustive techniques have been developed for testing nondeterministic concurrent programs to explore different interleavings or SYN-sequences. Hwang et al. proposed the idea of reachability testing for dealing with the problem of nondeterministic behavior of concurrent programs [30,31]. Reachability testing is an approach that can perform dynamic testing on a concurrent program and explore different interleavings or SYN-sequences without static analysis of the target source program. The SYN-sequences generated during testing are analyzed so that the subsequent tests explore different interleavings, thereby producing different SYN-sequences. The major limitation of reachability testing is that it is problematic to apply to testing concurrent programs with an infinite number of SYN-sequences.
Hwang et al. showed how to perform reachability testing of concurrent programs that use read and write operations on shared memory and semaphores for synchronization [30,31]. Tai described how to apply reachability testing to concurrent programs that use message passing [33]. Lei and Tai showed how to exercise all the possible SYN-sequences for a message passing program [34]. The sequences/variants must be stored, however, to prevent a sequence from being exercised more than once. In [35,36], a new reachability testing algorithm is presented that exercises all the (partially ordered)
12
SYN-sequences of a program exactly once, without requiring the sequences to be stored.
The true-concurrency model used in [35,36] guarantees that reachability testing never exercises two sequences that differ only in the order of concurrent events. Reference [34]
presents a general execution model that allows the algorithm in [35,36] to be applied to programs that use semaphores, locks, monitors, and/or message passing. Hwang et al.
described how to apply reachability testing to the exhaustive testing of a client/server database application that exhibits nondeterministic behavior [32]. In [44], it presents the design and implementation of a distributed reachability testing algorithm for a cluster of workstations. Lei et al. presented a testing strategy, called t-way reachability testing, that selectively exercises a subset of SYN-sequences [45].
It has been proven that reachability testing can derive all the feasible SYN-sequences of a concurrent program if the program has a finite number of SYN-sequences with a given input [31,32]. Reachability testing terminates when all the SYN-sequences have been tested. However, the common assumption of these works is that the concurrent program does not contain busy-waiting loops or iterative statements in which shared objects are repeatedly accessed and updated at the same program location. In cases where this assumption does not hold, reachability testing cannot stop because it continues to explore some identical program states. We illustrate why concurrent programs with iterative statements cannot be handled by reachability testing in Chapter 3. Sen and Agha presented a testing scheme that uses simultaneous concrete and symbolic executions to consider both inputs and schedules for message-passing distributed programs [37]. Their programming model requires that the number of possible inputs and schedules that can be exhibited by the target concurrent program be finite. In general,
all the above schemes are model-checking free.
2.2. CHESS and dynamic partial order reduction
CHESS is a concurrent testing tool that repeatedly runs a concurrent test whilst ensuring that every run takes a different interleaving [28,29]. If an interleaving results in an error, CHESS can reproduce the interleaving (for improved debugging). CHESS uses model checking techniques to systematically generate all interleavings of a given scenario. The implementation of CHESS needs a thin wrapper layer between the program under test and the concurrency API. Musuvathi and Qadeer proposed the concept of fair stateless model checking, a technique for systematic testing of nonterminating programs [46].
Related algorithms were implemented in CHESS. They also proposed iterative context-bounding for effectively searching the state space of a multithreaded program [47].
State-space storage and state matching are important in software model checking.
During the state search, once states have been visited they are stored. It is impossible to determine which states are visited only once before the search is completed. Storing states avoids redundant explorations of parts of the state space. It would be preferable not to store them in order to decrease the memory requirements. State-space caching creates a restricted cache of selected system states that have already been visited [48].
When the cache fills up, old states are deleted to accommodate new ones. Godefroid et al.
proposed a method which utilizes the properties of sleep sets and show that the memory requirement of the cache can be strongly decreased [49]. Dillinger et al. proposed a scheme by employing a compact hash table to offer a faster state matching with good accuracy [50]. In dynamic effective testing, only reiterated states discovered in a single
14
SYN-sequence are stored. Since most of the states are reached only once, there is no need to allocate a large memory space to store identified reiterated states.
Dynamic partial order reduction uses dynamic information to reduce the search space of states [51]. Dynamic partial order reduction and reachability testing share a similar operation model. They start by executing the concurrent program until completion.
Information that was collected during the run is used to identify branching points and the branching points are used to derive alternative executions that need to be explored. In [51], it only supports stateless explorations for acyclic state spaces. In the presence of cycles, the depth of the search has to be bounded somehow. Previous works on reachability testing also only support testing of concurrent programs with acyclic state spaces [31,32,36]. Yang et al. proposed a solution which applies the dynamic partial order reduction to concurrent programs with cyclic state spaces [52]. It consists of a light-weight method for storing abstract states and a stateful dynamic partial order reduction algorithm. As we have mentioned, dynamic effective testing only has to deal with recording and matching of reiterated states.
15