An Empirical Evaluation and Analysis of the Fault-Detection Capability of MUMCUT for General Boolean Expressions

全文

(1)Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. An Empirical Evaluation and Analysis of the Fault-Detection Capability of MUMCUT for General Boolean Expressions Chang-ai Sun 1, Kwan Yong Sim 2, T.H. Tse 3, and T.Y. Chen 4 1, 4 2 School of Information Technology School of Engineering Swinburne University of Technology Swinburne University of Technology Melbourne, VIC 3122, Australia 93576 Kuching, Sarawak, Malaysia { csun, tchen }@it.swin.edu.au ksim@swinburne.edu.my 3. Department of Computer Science The University of Hong Kong Pokfulam, Hong Kong thtse@hku.hk. according to the coverage criteria [16]. From the perspective of the coverage, structural Boolean expression-oriented testing can be further classified into decision coverage, condition coverage, decision /condition coverage and path coverage [9]. In decision coverage, for example, test cases are generated so that every program decision has taken the values True and False. In fault-based testing, test cases are generated to detect specified types of fault in a specification or program [2, 10, 15]. There have been studies related to fault-based testing of Boolean expressions. Weyuker et al. [15] proposed a meaningful impact strategy for testing Boolean formulas in Irreducible Disjointed Normal Form (IDNF). Offutt and Liu [10] proposed a procedure to generate test data for SOFL specification where conditions or predicates are assumed to be in Disjointed Normal Form (DNF). Kuhn [8] described a method for computing the conditions that must be covered by a test set for the test set to guarantee detection of the particular fault class. Tatsuhiro [13] improved on Kuhn’s fault hierarchy. Chen and Lau [2] proposed a set of more efficient test case generation and selection strategies for Boolean specifications in IDNF. We observe that existing fault-based testing techniques are developed for Boolean expressions in DNF or IDNF. In the real world, however, Boolean expressions written by a designer or programmer are usually in General Form (GF, also called arbitrary form [11]). In additional, we have empirically evaluated fault relationships between GF and IDNF in previous experiments and the result shows that, in up to 75.7% of the cases, one fault in GF can result in more than one fault in its equivalent IDNF [5]. Hence, it is of great interest to see how efficient MUMCUT will be for Boolean expressions in GF. We can almost draw a conclusion that there is a big. Abstract Boolean expressions are extensively used in software specifications. It is important to generate a small-sized test set for Boolean expressions without sacrificing the fault-detection capability. MUMCUT is an efficient test case generation strategy for Boolean expressions in Irreducible Disjointed Normal Form (IDNF). In the real world, however, Boolean expressions written by a software designer or programmer are not normally in IDNF. In this paper, we apply MUMCUT to generate test cases for general Boolean expressions and develop a mutation-based empirical evaluation on the effectiveness of this application. The experimental data show that MUMCUT can still detect single seeded faults in up to 98.20% of general Boolean expressions. We also analyze patterns where test cases generated by MUMCUT cannot detect the seeded faults. Keywords: Test Case Generation, Specifications, Software Testing. Boolean. 1. Introduction Boolean expressions are extensively used to represent the decisions/conditions in a specification or program. It is important to check whether they are implemented correctly for the purpose of quality assurance, since they play an important role in the specification or program. In the last decade, a lot of research work has been devoted to the testing on Boolean expressions, logic formulas or predicates [8]. The work on the testing of Boolean expressions can be divided into two major categories: structural approach and faultbased approach. In the structural approach, the basic idea is to generate a test suite to cover the elements of decision. 926.

(2) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. gap between the existing research and realistic situations. This also restricts the applicability of the existing work. Our main motivation in this paper is to investigate how efficient MUMCUT is when applied to general Boolean expressions. MUMCUT [2] is a fault-based test case generation strategy for Boolean expressions in IDNF, and the test suite generated by MUMCUT can detect seven single faults in a Boolean expression in IDNF. For general Boolean expressions, we want to see whether a test suite generated by MUMCUT can still detect a seeded single fault and empirically evaluate the fault detection capability of MUMCUT. The rest of this paper is organized as follows: Section 2 introduces MUMCUT and related techniques. Section 3 presents the approach to applying MUMCUT, and Section 4 introduces an empirical evaluation on the effectiveness of MUMCUT. Section 5 introduces the related work and Section 6 concludes the paper.. Negation Fault (ENF), Literal Negation Fault (LNF), Term Omission Fault (TOF), Term Negation Fault (TNF), Operator Reference Fault (ORF), Literal Omission Fault (LOF), Literal Insert Fault (LIF) and Literal Reference Fault (LRF)1. IDNF Transformation Given a Boolean expression, it can be represented in several forms. A Boolean expression in DNF is formed by the disjunctive terms, and a disjunctive term can be formed by the conjunctive literals (Boolean variables). A Boolean expression in DNF is said to be in IDNF when none of the Boolean literals or terms can be deleted without altering the value of the Boolean expression for some test cases [2, 15]. Given a general Boolean expression, it can always be transformed into an equivalent one in DNF using some laws, such as distribution law, commutative law and so on. A Boolean expression in DNF can also be transformed into an equivalent one in IDNF using the algorithm in [7]. For example, a Boolean expression in GF “a * (!c + !b + !d) + !a * (c + b + d) + b * (!c + !d) + !b * (c + d) + !c * d + c * !d” can be transformed into “a * !c + !a * c + a * !b + b * !c + !a * b + !b * c + !a * d + !b * d + !c * d + a * !d + b * !d + c * !d” in DNF. It can also be transformed into two different equivalent IDNFs “!a * d + !b * d + !c * d + a * !d + b * !d + c * !d” and “!a * d + !b * d + b * !d + c * !d + a * !c”.. 2. Background In this section, we introduce the basic concepts and previous work related to this research. Test Cases and Test Case Adequacy In software testing practice, testers are required to generate test cases to execute the program. A test case is an input on which the program under test is executed during testing. A test set/suite is a set of test cases for testing a program [16]. Given a test criterion, we can judge whether a test set is adequate. For Boolean expressions, a test case is an ordered truth-value list where each value is the assignment of a Boolean variable. For example, consider Boolean expression B1 {B1 = a + b}, a test case for B1 is {a = 0, b = 0}, and the complete test suite for B1 is {00, 01, 11, 10}. If the decision coverage criterion is used, {00, 10}, {00, 11} and {00, 10} are the adequate test sets.. MUMCUT MUMCUT is a fault-based test case selection strategy for generating test cases from Boolean specifications, proposed by Chen and Lau [2]. It is the integration of the MUTP strategy [4], the MNFP strategy [2] and the CUTPNFP strategy [3]. These strategies were developed based on two important concepts, namely Unique True Point (UTP) and Near False Point (NFP) [15]. Accordingly, UTP and NFP are defined on Boolean expressions in IDNF. Compared with the existing test case selection strategies for Boolean expression in IDNF, the MUMCUT strategy is more efficient. For example, MAX-A and MAX-B are two strategies developed by Weyuker et al. [15] based on UTP and NFP. Under the assumption that only one fault is introduced into the implementation, the empirical research shows that the MUMCUT strategy uses on the average 74% and 67% of the test cases required. Fault-based testing and Mutation Analysis Software testing often aims at detecting faults in a program [16]. If it is assumed there are some specific fault types in a specification or program, and if test cases are generated to detect these faults, then this approach is called fault-based testing. In fault-based testing, mutation analysis [6] is widely used to verify the adequacy of a test suite based on the specific testing criteria. Given a Boolean expression B, a derivation M is obtained by seeding faults into B, M is called a mutant of B, and the process to obtain M from B is called mutation. In our experiment, mutation technique is used to derive the Boolean expression mutants. These mutants contain only one fault when they are compared with the original Boolean expressions. The faults concerned in this paper include Expression. 1. Compare these with the seven types of fault defined in [2]. The eight faults here are defined in the context of GF rather than IDNF. Detailed definitions and examples are available in [5].. 927.

(3) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. by the MAX-A and the MAX-B strategies, respectively [2].. mutants with one fault can be distinguished from the original Boolean expression under these test cases. If answer is yes, we say that the mutant is killed. Given a Boolean expression O, a mutant M, and a test suite TS (t1, t2, …, tn ), we say that M is killed if and only if at least one test case ti from TS can differentiate the truth value of M and O, noted as Killed(M, O, TS). In other words,. Two related experiments Recognizing that the realistic Boolean expressions are more often written in arbitrary form, we have conducted an empirical research on the fault relationship between GF and IDNF [5]. In more detail, we investigated statistically that one fault introduced into a Boolean expression in GF will result in how many faults in its equivalent in IDNF. We also developed an algorithm and a tool to generate a benchmark of Boolean expressions in arbitrary form for the subsequent experiments [12]. We shall not repeat the discussion of some techniques that have been discussed in our previous experiments and also required in this experiment, provided this does not affect the comprehension of this paper.. ∃ti ∈ TS : O(ti ) ≠ M (ti ) Given a Boolean expression O, a mutant M, and a MUMCUT-adequate test suite TS (t1, t2, …, tn) generated based on M, we say that MUMCUT is effective for this mutant if and only if at least one test case ti from TS can kill the mutant M with O as the oracle. In this experiment, we employ mutation technique to obtain a mutant M of the original Boolean expression in GF O. If one or more test cases from the generated test suite TS can distinguish the difference between O and M (that is, M is killed when O is used as a test oracle), we say MUMCUT is effective. Figure 1 shows how to evaluate whether a MUMCUT-adequate test suite TS can detect a fault in a Boolean expression in GF M (that is, kill M) with O as the test oracle.. 3. Applying MUMCUT to the Testing of General Boolean Expressions Given a Boolean expression in arbitrary form, we first use an algorithm to transform it into one IDNF, then we use MUMCUT to generate a set of test cases, finally these test cases can be used as a test suite to detect some specific faults in Boolean expression in arbitrary form. In the view of users, they do not need to know about this process, since what they need to do is providing Boolean expressions in arbitrary form and acquiring test cases generated by MUMCUT strategy. However, MUMCUT is not always effective in the context of general Boolean expressions. For a Boolean expression in IDNF B, a test suite T is generated by MUMCUT, it can guarantee that all seven single types of fault mentioned in [2] can be detected when T is used as test cases for B. For general Boolean expressions, one fault will result in simultaneous occurrence of several faults in its equivalent in IDNF. In this situation, the generated test cases satisfying MUMCUT cannot guarantee the detection of all faults introduced into a Boolean expression in IDNF.. MUMCUT is still effective? Boolean expression in GF O. Mutation: Seed a fault. Mutant in GF M. IDNF Transform. Evaluation. A MUMCUTadequate Test Suite TS. Test Case generation. Mutant in IDNF MI. Figure 1. A mutation-based evaluation of the extension of MUMCUT Several critical steps involved in the experiment include: 1) Boolean Expression Samples. To be more convictive, a large number of Boolean expressions in GF are required. In our previous experiment, we have developed a parameterized generator to generate general Boolean expressions [6]. In this experiment, we use this tool to generate Boolean expression samples.. 4. An Empirical Evaluation In this section, we present an empirical evaluation on the fault detection capability of MUMCUT in the context of general Boolean expressions.. 4.1 Principle We statistically check whether one fault in general Boolean expressions can be detected when a set of test cases generated by MUMCUT (also called MUMCUT-adequate test cases) are used. From the perspective of mutation, it is equal to whether the. 2) Mutation. To control the size of mutants and at the same time satisfy mutation adequacy to some extent, we also developed a set of mutation strategies for mutant generation [5]. It is noted only one fault is seeded into mutants when the mutation is conducted.. 928.

(4) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. In this experiment, eight types of fault mentioned in section 2 are considered.. 4.2 Settings When a Boolean expression contains more than 12 Boolean variables, its IDNF transformation will be very slow. It will also take a long time for BEAT to generate a MUMCUT-adequate test suite for a Boolean expression consisting of more than 15 Boolean variables. To be practical, we use the Boolean expression generator BEGen [12] to generate 800 samples of Boolean expressions in GF as the experiment object. We restrict the parameter settings of BEGen as follows: The maximum number of literals is 12 (namely, the characters from a to l as positive literals and their negations such as “!a” as negative literals). The maximum number of terms of a Boolean expression is 12. The maximum number of literals in a term is 6. The seven operators are “*”, “+”, “!”, “+ ( )”, “+ !( )”, “* ( )” and “* !( )”.. 3) IDNF Transformation. The Converter [5], a tool developed by us for IDNF transformation, is used to transform Boolean expressions in GF into ones in IDNF. 4) Test Case Generation. Given a Boolean expression in IDNF, BEAT [1], a tool is developed to implement MUMCUT strategy, is used to generate a MUMCUT-adequate test suite. 5) Evaluation. A MUMCUT-adequate test suite TS, that is generated based on the mutant in IDNF MI, is used to evaluate whether the M can be killed with O as test oracle. Here, M is the mutant of O, and MI is the IDNF of M. In the implementation, we will use TS to kill MI with OI (OI is the IDNF of O) as test oracle because of the following observation. O is a Boolean expression in GF, M is a mutant of O, OI and MI is the IDNF of O and M, respectively, TS is a test suite. Then, Killed (MI, OI, TS ) = Killed (M, O, TS). 4.3 Result and analysis Table 1 shows the result of the empirical evaluation on the effectiveness of MUMCUT for general Boolean expressions.. Our previous experiments have also involved the first three steps mentioned above [5, 12]. In this experiment, we focus on evaluation and use an instance to illustrate the evaluation process as follows. 1) A Boolean expression OG in arbitrary form is “!e * c + c * (h * !g * i * !e + g * !i * !d) * !d * !e + g”. 2) Assume an Operator Reference Fault happens to the first operator and change it from “*” to “+”. A mutant MG like “!e + c + c * (h * !g * i * !e + g * !i * !d) * !d * !e + g” is obtained. 3) Transform OG and MG into Boolean expressions in IDNF. We get O(“!e * c + g”) and M(“!e + c + g”). 4) Generate a MUMCUT-based test suite T of M using BEAT. T = {000, 010, 110, 011} 5) Get t1 from T (t1 = “000”) and create the literal-assign pair table LA = {c = “0”, e = “0”, g = “0”}. 6) Evaluate the truth-value TM of M with LA and TM = “1” (since !e + c + g = 1 + 0 + 0 = 1). Similarly, we evaluate the truth-value TO of O with LA and TO = “0”(since !e * c + g = 1 * 0 + 0 = 0). 7) “True” is returned since TO ≠TM. In this example, a MUMCUT-adequate test suite kills an ORF mutant. In other words, MUMCUT is still effective to detect the seeded fault. In some situations, however, MUMCUT may not be effective. Our experiment is to investigate the failure rate of MUMCUT and why it fails.. Table 1. The result of empirical evaluation Muta -tion Type ENF LNF TOF TNF LOF LIF LRF ORF Total. BE Sample. Valid Mutant. Killed. Not Killed. 100 100 100 100 100 100 100 100 800. 91 775 295 382 263 235 336 686 3063. 91 770 270 377 261 229 332 678 3008. 0 5 25 5 2 6 4 8 55. Effective ness % 100.00% 98.97% 91.53% 98.695 99.24% 97.45% 98.81% 98.83% 98.20%. In Table 1, 800 Boolean expression samples are evenly divided into eight groups and each of them is used for one kind of fault mutation. Column 3 shows the valid mutants of a variety of mutation. Since there may be some redundant components in a Boolean expression, mutants without these redundant component is still equivalent of the original Boolean expression. For example, Given a Boolean expression in GF BG “!e + c + !e * (h * !g + g * !d) + g”, its equivalent IDNF BI is “!e + c + g”. Assume a mutant MG of BG is “!e + c + !e * (h * !g ) + g”. Compared with BG, a Term Omission Fault happens to MG (namely the term “g * !d” is omitted.) The equivalent IDNF MI of MG is “!e + c + g”. In this situation, MG is not a valid mutant since MI is equal to BI. Further more, some implicit equal IDNFs are difficult to identify. For example, assume that BI is “!a*!b + a*b + a*c” and MI is “!a*!b +. 929.

(5) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. a*b + !b*c”. It seems to be a Literal Reference Fault (LRF), but these two IDNFs are equivalent. In our experiment, we have identified and discarded these equivalent mutants. The empirical evaluation result shows that in 98.20% situations MUMCUT is effective when it is used for the fault-based testing on Boolean expressions in GF where there is only one fault is seeded. It means that MUMCUT is still effective in testing Boolean expressions in the realistic situation. In our experiment, there are totally 55 not-killed mutants. When we further examine those instances that cannot be killed by test cases generated by MUMCUT, we discover five failure patterns as follows. Certainly, test cases generated by MUMCUT cannot detect faults in the combination of these five failure patterns too.. Pattern 5: Concurrent Term and Literal Omission without Losing Literals Original Boolean expression: “ab!c + a!bc + !abc” Æ Mutant Boolean expression: “ab + ac”. Pattern extension includes: (ab!c + a!bc + !abc)T Æ (ab + ac)T. Here, “T” is a null term or a term without the occurrence of “a”, “b” or “c”. The empirical evaluation result also indicates that the fault-detection capability of MUMCUT varies with fault types. As for Term Omission Fault (TOF), MUMCUT can only detect faults in 91.53 situations. In our previous work [5], we found when one fault is seeded into a general Boolean expression, in most situations it resulted in more than one fault in its equivalent IDNF. Further, there are more chances where the multiple-literal term is omitted in their equivalent IDNF and above-mentioned patterns occur with the higher frequency. This explains why the fault-detection capacity of MUMCUT for TOF is lower than others. As to Expression Negation Fault (ENF), any test case is able to kill a mutant, since the truth-value of mutant and the original Boolean expression are always different. The fault-defection capability of MUMCUT for the remaining six types of fault is very close. Table 2 demonstrates the occurrence of these five patterns in different fault types.. Pattern 1: Two-Reflective-literal Conjunctive Term Omission Without Losing Literals Original Boolean expression: “abc + AB” Æ Mutant Boolean expression: “abc”. Pattern extension includes: “abcT + !a!b” Æ “abcT” and “(abc + !a!b)T” Æ “abcT”. Here, “T” is a null term or a term without the occurrence of “a”, “b” or “c”. Pattern 2: Disjunctive Term Omission Without Losing Literals Original Boolean expression “ab + bc + ac = ab + c (a + b)” Æ Mutant Boolean expression “ab + c”. Pattern extension includes: “ab + c(a + b) T” Æ “ab + c T” and “(ab + c(a + b))T” Æ“(ab + c)T”. Here, “T” is a null term or a term without the occurrence of “a”, “b” or “c”.. Table 2. The distribution of five failure patterns Mutation Type ENF LNF TOF TNF LOF LIF LRF ORF2 Total. Pattern 3: Term Omission With Losing Literals Original Boolean expression: “ab + bc” Æ Mutant Boolean expression: “ab” Pattern extension includes: “abT + bc” Æ “abT”, “ab + bcT” Æ “ab” and “(ab + bc)T” Æ “abT”. Here, “T” is a null term or a term without the occurrence of “a”, “b” or “c”.. P1. P2. P3. P4. P5. Total. 0 0 3 0 0 2 1 5 11. 0 2 4 0 1 3 2 2 14. 0 2 15 5 1 1 1 4 29. 0 0 3 0 0 0 0 1 4. 0 1 0 0 0 0 0 0 1. 0 5 25 5 2 6 4 8+4 55 + 4. Further, the result also provides a useful guideline for developers. We should pay more attention to the faults difficult to detect, such as TOF.. 4.4 Limitations In this experiment we developed a mutation strategy to produce the limited size of mutants because of the practicability concerns. The restricted number of mutants may be a limitation of the validity of the empirical evaluation. Another possible limitation in the evaluation of overall effectiveness comes from the mutant ratio of different faults, since. Pattern 4: Distribution One-Complemental-Literal Conjunctive Term Omission Without Losing Literals Original Boolean expression: “acde + bc + ab!d” Æ Mutant Boolean expression: “acde + bc”. Pattern extension includes: “acdeT + bc + ab!d” Æ “acdeT + bc”, “acde + bc + ab!dT” Æ “acde + bc” and “(acde + bc + ab!d)T “Æ “(acde + bc)T”. Here, “T” is a null term or a term without the occurrence of “a”, “b”, “c”, “d” and “e”.. 2. In this type of fault, the combination of failure patterns occurs in four mutants.. 930.

(6) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. it is hard to predict which type of fault is more frequent in the real world. Finally, the number of general Boolean expressions may also affect the validity of this evaluation.. effectiveness of 98.20%, while the empirical evaluation in [2] reported an effectiveness of 99.8% using 20 Boolean expression samples from [15].. 6. Conclusion 5. Related Work We have applied MUMCUT to generate test cases for general Boolean expressions and reported an empirical evaluation of the fault detection capability of MUMCUT in the realistic situation. MUMCUT is a fault-based test case generation strategy for Boolean expressions in IDNF. When it is applied to a general Boolean expression B, we first transform it into an IDNF I, and then employ the MUMCUT strategy to generate a test suite T for I. Subsequently, the test suite T can be used as test cases of the original Boolean expression B. We have also developed a mutation–based experiment to empirically evaluate effectiveness of MUMCUT. Given a general Boolean expression, mutation technique is used to generate a set of mutants where only one fault is seeded. A test suite generated by MUMCUT is used to decide whether a mutant can be killed and the original Boolean expression is used as an oracle. Our empirical evaluation showed that, in 98.20% of the situations, MUMCUT was effective in the context of faultbased testing of general Boolean expressions. As future work, we plan to conduct another empirical evaluation on the effectiveness of MUMCUT when more than one fault is seeded into general Boolean expressions. We also plan to develop test case generation strategies or techniques for the failure patterns observed in this study.. This work complements and extends research in fault-based testing of Boolean specifications. For the most part, the research has focused on the systematic test generation, selection, and empirical evaluation of test suites, and these test suites are used to detect several special fault types of Boolean expressions in the specific forms, such as DNF, CNF and IDNF. We outline below the work related to our project. Weyuker et al. [15] investigated and proposed a family of meaning impact strategies for automatically generating test cases for any implementation intended to satisfy a given specification that is a Boolean formula. These strategies are based on two basic concepts, namely unique true points and near false point. These strategies are effective to detect five operator faults, such as variable negation faults. They also require that the Boolean expressions under test should be in IDNF. Chen and Lau [3, 4] proposed a set of more efficient test case generation strategies called MUMCUT for Boolean expressions in IDNF, including MUTP, MNFP and CUTPNFP. These strategies are also based on the unique true point and near false point, and can detect seven single faults in a Boolean expression. MUMCUT also requires that Boolean expressions under test should be in IDNF. IDNF Boolean expressions are a small subset of all Boolean expressions in the real world [2, 5, 11]. In this paper, we have applied MUMCUT to generate test cases for general Boolean expressions. The generated test cases are expected to detect common faults. Finally, our work enables MUMCUT to obtain its application in the whole set of Boolean expressions. To evaluate the effectiveness of the proposed strategy, a lot of empirical studies have been reported in [3, 4, 14, 15]. As we know, all existing empirical studies have used the same set of Boolean expression samples, first used by Weyuker et al. [15], to examine the effectiveness of their strategy. These samples consist of twenty Boolean specifications taken from the specification for a real aircraft collision avoidance system (TCAS II). Chen and Lau [3, 4] also used these Boolean specifications samples in their previous empirical evaluation related to MUMCUT, such as MUTP and CUTPNFP. In this paper, we use a generator [12] developed by us to produce a large number of Boolean expression samples in arbitrary forms. This enhances the representativeness of experimental samples. Furthermore, our empirical evaluation shows an. Acknowledgements This research is supported in part by an ARC Discovery Grant (Project No. DP0345147) and a grant of the Research Grants Council of Hong Kong (Project No. 1083/00E).. References [1] T.Y. Chen, D.D. Grant, M.F. Lau, S.P. Ng, and V.R. Vasa, BEAT: Boolean expression fault-based test case generation, in Proceedings of International Conference on Information Technology: Research and Education, IEEE Computer Society Press, Los Alamitos, California, 2003, pp. 64-69 [2] T.Y. Chen and M.F. Lau. Test case selection strategies based on Boolean Specification, Software Testing, Verification and Reliability, Vol. 11, 2001, pp. 165-180 [3] T.Y. Chen and M.F. Lau, An empirical evaluation on the greedy CUTPNFP strategy for Boolean specification based testing, in Proceedings of the 5th Joint conference on Information Science (JCIS), 2000, pp. 627-630. 931.

(7) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan. [11] V. Okun, P.E. Black, and Y. Yesha, Comparison of fault classes in specification-based testing, Information and Software Technology, Vol. 46, No. 8, 2004, pp. 525533. [4] T.Y. Chen and M.F. Lau, An empirical study on the effectiveness of the greedy MUTP strategy, in Proceedings of Software Engineering Research and Practice (SERP ’98), IEEE Computer Society Press, Los Alamitos, California, 1998, pp. 338-344. [12] C.A. Sun and K.Y. Sim, A FSM-based parameterized generator for Boolean expressions, To appear in the proceedings of ICENCO-2004, Dec. 27-30 2004, Cairo, Egypt.. [5] T.Y. Chen, K.Y. Sim, and C.A. Sun, A simulation analysis of fault relationship between GF and IDNF, submitted for publication. [13] T. Tsuchiya and T. Kikuno, On fault classes and error detection capability of specification-based testing, ACM Transactions on Software Engineering and Methodology, Vol. 11, No. 1, 2002, pp. 58-62. [6] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, Hints on test data selection: help for the practicing programmer, IEEE Computer, Vol. 11, No. 4, 1978, pp. 34-41 [7] R. Galivanche and S.M. Reddy, A parallel PLA minimization program, in Proceedings of the 24th ACM/IEEE Design Automation Conference, ACM Press, New York, 1987, pp. 600-607. [14] M.A. Vouk, A. Paradkar, and K.-C. Tai, Empirical studies of predicate-based software testing, in Proceedings of International Symposium on Software Reliability Engineering (ISSRE), IEEE Computer Society Press, Los Alamitos, California, 1996, pp. 55-65. [8] K. R. Kuhn, Fault classes and error detection capability of specification-based testing, ACM Transactions on Software Engineering and Methodology, Vol. 8, No. 4, 1999, pp. 411-424. [15] E.J. Weyuker, T. Goradia, and A. Singh, Automatically generating test data from a Boolean specification, IEEE Transactions on Software Engineering, Vol. 20, No. 5, 1994, pp. 353-363. [9] N. Juristo, A. M. Moreno, and S. Vegas, Reviewing 25 years of testing technique experiments, Empirical Software Engineering, Vol. 9, No. 1, 2004, pp. 7-44. [16] H. Zhu, A.V. Hall, and H.R. May, Software unit test coverage and adequacy, ACM Computing Survey, Vol. 29, No. 4, 1997, pp. 366-427. [10] J. Offutt and S. Liu, Generating test data from SOFL specifications, Journal of Systems and Software, Vol. 49, No. 1, 1999, pp. 49-62. 932.

(8)