Chapter 4.Extending A Test Case Set Based On Duplication
4.3. Proof Of The Duplicate Algorithm…
To prove the correctness of the duplicate algorithm, some propositions are introduced firstly.
Definition 4.5. (value independent): If all the pairs generated for two parameters Pi
and Pj are distinct in the test case set TS, Pi and Pj are claimed "value independent"
in the TS.
For example, there are three parameters P1 = {ν11, ν12, ν13}, P2 = {ν21, ν22,} and P3 = {ν31,
ν32, ν33} and the test case set TS = {[ν11, ν21, ν31], [ν11, ν22, ν32], [ν12, ν21, ν33], [ν12, ν22, ν32],
[ν13, ν21, ν31], [ν13, ν22, ν32]}. Each pair (ν1i, ν3j) is distinct in the TS, 1≤ i, j≤ 3, and parameter P1 and P3 are value independent in the TS. Because (ν21, ν31) and (ν22, ν32) appear twice, parameter P2 and P3 are not value independent in the TS.
To prove the following propositions, an Incremental-TS-extension (ITE) method is introduced. The ITE method is used to duplicate the permutation of pairs for a parameter in the test case set.
Incremental- TS -extension (ITE) method
Input: A TS covers all pairs which are generated by the k parameters. Let the parameter Pi be one of the k parameters, and Pk+1 is a new parameter which is waited to be added to TS, |Vi|≤ |Vk+1|.
00 Begin
01 ∀ test case Ta∈ TS 02 {
03 νij is the jth value of parameter Pi, 1≤ j≤ |Vi|.
04 if(νij∈ Ta)
05 extend Ta by adding ν(k+1)j
06 } 07 End
Proposition 4.2: Let two parameters Pi and Pj be value independent in the test case set TS, 1≤ i, j≤ k. When a new parameter Pk+1 is added, |Vk+1|≥ |Vi|. After executing the ITE method of the input parameters are Pi andPk+1. The two parameters Pj and
Pk+1 are also value independent in the extended test case set.
Proof: Assume that Pi and Pj are value independent in the extended test case set, but parameters Pj and Pk+1 are not value independent. There are some pairs (vjm, v(k+1)n) where 1≤
m≤ |Vj|, 1≤ n≤ |Vk+1|, which appear repeatedly in the extended test case set. The pairs (vin, vjm) also appear repeatedly in the extended test case set, and parameters Pi and Pj are not value independent. That is a contradiction, and therefore parameters Pj and Pk+1 are value independent.□
Proposition 4.3: Let the test case set TS cover pairwise perfectly for n. For any two test cases in different blocks, there is only one value in common.
Proof: ∀ test case Ta ∈ BBi, Tb ∈ Bj,B i≠ j, if there are two values in common in Ta and Tb, the pair generated by the two values appears twice and TS doesn’t cover pairwise perfectly for n.
That is a contradiction.
Because test cases are divided into blocks according to the value of parameter P1, the value of P1 is different for any twotest cases in different block. Consider the values of parameter P2 to Pn+1, because TS cover pairwise perfectly for n, all values appear once in a block. ∀each value vkl in the test case Ta∈ BBi where k≠ 1 and1≤ i ,l≤ n, the test case Tb∈ BjB can be found that vkl is in Tb, i≠ j. For any two test cases in different blocks, there is only one variable that has the same value.
Proposition 4.4: Let the test case set TS cover pairwise perfectly for n. If an n-valued parameter is added, the extended test case set lose n(n-1) pairs at least without increasing the number of test cases.
Proof: W.L.O.G, assume that the first value is v11 foreach test case in the first block and figure 16 is the corresponding pairwise graph. When parameter Pn+2 is added, two cases are discussed as follows. In case 1, all test cases in the first block are extended by adding n values of Vn+2. In case 2, test cases in the first block are extended by adding j values of Vn+2, 0≤ j≤ n-1.
Case1: The ith test case in the first block is extended by adding v(n+2)j, 1 ≤ i, j≤ n. There are n+1 space for n+2 parameters and n-1 space for n values in pairwise graph. For each complete path except the first block, if i = j and the last node is n(n+2)k, the node nik can be found in the same complete path and is shown in the case 1 of figure 17. The pair (vik, v(n+2)k) appears repeatedly with the pair in the first block. If i≠ j, the proof is the same with above and the repeated pairs become (vik, v(n+2)l), k≠ l, and the corresponding graph is the case2 of Figure 17. ∀ test case Ta∈BBk, k≠ 1, each extended test case has least one repeated pairs and the extended test case set loses n(n-1) pairs at least.
Case2: Test cases in the first block are extended by adding j values of Vn+2, 0≤ j≤ n-1. If all test cases in the first block are extended, there are n-j repeated pairs in the first block.
According to proposition 4.3, there is one same value for any two test cases in different blocks. For the remainder n(n-1) test cases, only n-j test cases have no repeated pairs, and each of the n2-2n+j test cases has least one repeated pair, so the total lost pairs are n2-2*n+ j+
n-j = n (n-1).□
Figure 16 Pairwise graph for n+2 n-valued parameters
Figure 17 Form of the Repeated covered pairs
Proof of the correctness of the Duplicate Algorithm: A test case set TS covers pairwise perfectly for n. m n-valued new parameters are waited to be extended, 1≤
m≤ n+1. After extension by using duplicate algorithm, the test case set TS loses n(n-1) pairs for adding a new parameter and the test cases set has the most pairwise coverage than other test case sets with the same size.
Proof: Because the test case set TS covers pairwise perfectly for n, any two parameters between P1 and Pn+1 are value independent in TS. According to proposition 4.2, the parameter Pn+i+1 is value independent with former parameters except parameter Pi, 1≤ i≤ m, and the number of repeated pairs generated by Pi and Pn+i+1 is n (n-1). According to proposition 4.4, the least number of lost pairs is n (n-1) when adding a new parameter.
Therefore, the TS extended by the duplicate algorithm still cover the most pairs without increasing number of test cases.□
The advantage for using the duplicate method is when new parameters are added, the test cases can be extended quickly. The duplicate method reduces large time in extension and retains high coverage rate.v1i,
Finally, some situations are discussed as follows. Let the test case set TS cover pairwise perfectly for n. There are k*n(n+1) new pairs are generated when a k-valued parameter is added, k≠ n. Without increasing the number of test cases, how many pairs are lost at least after extension for k< n, n< k≤ n2 and k>n2?
k< n: the least number of lost pairs is zero. There are k*n(n+1) new pairs are generated when adding a new parameter and the extended test cases can cover n2(n+1) new pairs at most, so the least number of lost pairs is zero.
n< k≤ n2: the least number of lost pairs is n(n+1)(k-n)+n2-k. If there are no repeated pairs in the extended test cases, the extended test cases cover n2(n+1) new generated pairs.
Because k≤ n2, repeated pairs are not found in only k extended test cases, and the remainder n2- k test cases have more than one repeated pair. The reason is same with the case 2 of proposition 4.4, so the number of lost pairs is n(n+1)k-n2(n+1) + n2-k.
k>n2: the least number of lost pairs is n(n+1)(k-n). If there are no repeated pairs in the extended test cases, the extended test cases cover most n2(n+1) new pairs, so the number of lost pairs is n(n+1)k-n2(n+1) = n(n+1)(k-n).
Chpater 5 Conclusion and Future Work
Pairwise testing is a special case of n-way testing. In this paper, MIVO test generation strategy for pairwise testing is presented and implemented. With our simulation, to reach the same coverage, MIVO generates fewer test cases than IPO, i.e., in test cases generation, MIVO is more effective.
The duplicate algorithm is also introduced and clearly proved in this paper. Using the duplicate algorithm, the extended test cases retain high pair’s coverage. With the advantage of using the algorithm is that test cases can be easily extended and large amount of computing time for extension can be saved
Much works also remains to be done to validate the result of MIVO. In the future, we hope that MIVO testing strategy can be implemented to test real software systemes and more empirical results are gathered to confirm the usefulness of MIVO.
References
[1] Y.Lei and K.C. Tai, “In-Parameter-Order: A Test Generation Strategy for Pairwise Testing,” Technical Report TR-2001-03, Dept. of Computer Science, North Carolina State Univ., Raleigh, North Carolina, May, 2001.
[2] Y.Lei and K.C. Tai. ”A Test Generation Strategy for Pairwise Testing”, IEEE Trans of Software Engineering, Vol 28, NO.1, Jan 2002.
[3] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, “The AETG System: An Approach to Testing Based on Combinational Design,” IEEE Trans. Software Eng., vol. 23, no.7, pp.437-443, July 1997.
[4]D. M. Cohen, S. R. Dalal, M.L. Fredman, and G.C. Patton, “The combinatorial design approach to automatic test generation.” IEEE Software, pages 83-87, September 1996.
[5]Glenford J. Myers, “The Art of Software Testing,” Senior Staff Member, IBM Systems Research Institute, Lecturer in Computer Science, Polytechinic Institute of New York.
[6]Roger S. Pressman, Ph.D.,”Software Engineering: A Practitioner’s Approach”, Sixth Edition. 2005.
[7]R.Mandl, ”Orthogonal Latin squares: An application of experiment design to compiler testing”, Communications of the ACM, Vol. 28 NO. 10(October 1985) pp.1054-1058.
[8]Dennis. Jeffrey and Neelam Gupta,” Test Case Prioritization Using Relevant Slices”, Annual International Computer Software and Applications Conference (COMPSAC’06), IEEE, 2006.
[9] Renee C. Bryce and Charles J. Colbourn, “Test Prioritization for Pairwise Interaction Coverage”, A-MOST’05, May 15-21,2005, St. Louis, Missouri, USA.
Copyright 2005 ACM.
[10] James A. Jones and Mary Jean Harrold, “Test-Suite Reduction and Prioritization for Modified Condition/ Decision Coverage”, IEEE Trans., Software Eng ., 2003.
[11]Soumen Maity and Amiya Nayak “Improved Test Generation Algorithms for Pair-wise Testing” 2005 IEEE, Software Reliability Engineering.
[12]S. Elbaum, A. Malishevsky, and G. Rothermel, ”Prioritizing Test Cases for Regression Testing,” Proc. ACM Software Testing and Analysis, pp. 102-112, Aug. 2000.
[13] D. M. Cohen, S. R. Dalal, A. Kajla, and G.C. Patton, “The automatic Efficient Test Generator (AETG) System”, Proceedings of the 5th IEEE International Symposium on Software Reliability Engineering, pages 303-309, Nov. 1994
[14] D. M. Cohen, S. R. Dalal, J. Parelius, and G.C. Patton, “Combination Design Approach to Automatic Test Generation”, IEEE Software, 13: pages 83-88,1996
Appendix A
The input order of the example for simulation E1: five parameters (5 3-valued).
E2: sevenparameters (5, 6, 3, 2, 3, 4, 5) E3: ten parameters (7, 4, 6, 5, 7, 5, 5, 6, 4, 4)
E4: twenty-five parameters (3, 3, 4 ,7, 2, 5, 4, 2, 7, 3, 5, 2, 2, 2, 4, 3 ,3 ,7 ,2 ,4 ,3, 5, 5, 7, 7) E5: forty parameters (5, 4, 2, 3, 7, 2, 4, 4, 5, 3, 2, 2, 5, 7, 4,7, 3,3,4,5, 3,4,5,2, 2,7,2,3, 7,7,5,4,7,2,3,3,4,5,5,7)
E8: twenty parameters (20 4-valued).
E9: twenty-five parameters (11, 11, 13, 7, 9, 9, 13, 13, 11, 7, 7, 7, 13, 11, 9, 7, 9, 7, 13, 11, 7, 9, 9, 13, 7, 9)
E10: thirty parameters(10, 13,10,15,13,13,10, 15,13,15,10,10,15,13,13,10, 15,15,10,13, 10,15, 13,10,15,10,13, 13,15,15)
E11: forty-three parameters (6, 5, 5, 4, 3, 2, 6, 8, 4, 2, 6, 5, 3, 3, 10, 4, 2, 8, 2, 2, 6, 8, 3, 10, 2, 6, 6 , 5, 4, 4, 10, 2, 8, 8, 6, 4, 10, 2, 3, 3, 6, 8, 10).
E12: fifty-four parameters(6, 9, 2, 10, 3, 5, 5, 7, 6, 3, 2, 4, 4, 9, 8, 2, 10, 7, 3, 9, 5, 5, 8, 4, 8, 10, 6, 3, 3, 8, 9, 5, 7, 6, 7, 5, 3, 10, 10, 2, 8, 6, 7, 4, 7, 6, 8, 4, 10, 2, 2, 9, 4, 9).