Chapter 3. Web application testing model
3.3. State transition diagram
3.3.3 State Transition Diagram (STD) construction algorithm
From the purpose of speeding up collaborative testing, we first apply Breadth-First Search (BFS) on the page-level of value-oriented dependence graph to parallelly distribute folk testers. At each new visiting page, we further drill down to code-level of value-oriented dependence graph to identify different states based on distinct program flow. In the meanwhile, the proper label is created for these new states. The details of state transition diagram are presented below:
26
State transition diagram algorithm Input:
Value-oriented Dependence graph G,
C[i]: tainted variable in branch predicate C of each page PAi, S0:initial state which Label 0,
N[i]:start basic block of each page PAi color(i,c): node c is in code level of page PAi
Output: State transition diagram STD D=(S, δ, S0) Method:
Initial: S=∅, δ=∅ , j=1, k=1
Step1: for each page number P which from small to large ∈ adj [S0.N] in G ConstructionNextState (S, P)
Step2: For each new state s‘ in S 2.1: k=1
2.2: for each page number P which from small to large ∈ adj[s‘.N] in G ConstructionNextState (s‘, P)
Step3: Repeat Step2 until there is no new state.
Subroutine: ConstructionNextState Input:
S: State P: Page number
Output: null
Step1: IF S0 modify global value update global variable value of C[i]
ELSE update global variable value with S0 of C[p]
Step2: add the variable of edge into C[p]
Step3: for each node c Color (p,c)=white
Step4: StateCreation (S0,p, C[p],N[p],0)
Step5: Change back global variable value of C[p]
27
Step1: IF the first part of s exists IF the second part of s exists
Step1: IF number of out-degree of c0 is zero or Color is black 1.1: color (p, c0)=black
1.2: Add p into first part of S‘
1.3: Add coming edge condition into second part of S‘
1.4: IsANewState(SP,S‘,i)
Step2: for each basic block c‘ in neighborhood of c0
IF condition of coming edge of c‘ accords with C[p]
2.1: add condition of coming edge of c‘ into second part of S‘
2.2: color (p, c0)=black
2.3: StateCreation (SP ,p, C[p],c‘,i)
28
3.3.4 Example of Constructing State Transition Diagram
The Example 3 is given to illustrate the state transition diagram construction algorithm, where Figures 9-1 to 9-2 show the result after executing steps 1 to 2, respectively.
Example 3:
The Example 3 continues Example 2. According to the value-oriented dependence graph, tainted variables in branch predicate of each page, and initial state, we can construct the state transition diagram.
Since the first part of initial state, we find outgoing edge in page-level of VDG, and then according to tainted variable in branch predicate of the page we search code-level of VDG, after executing step 1, we can construct the new state of initial state. Second, if the state doesn‘t exist then label and add to state transition diagram as shown in Figure 9-1.
Default Pay Sales
Figure 9-1. State transition diagram after executing Step 1 of STD algorithm S0
2, {(q1,<=0),(q2,null),(Session
*“total”+, null)-
,(q1,<=0),(q2,null),(Session *“total”+, null)-
1, ,(Session *“total”+, null)- 0
29
Since the first part of the given new state we find outgoing edge in page-level of VDG, and then according to tainted variable in branch predicate of the page we search code-level of VDG, after executing step 2, we can construct new state of the given new state. Second, if the state doesn‘t exist then label and add to state transition diagram as shown in Figure 9-2.
1,{(Session[“total”],null) } {(q1,<=0) ,(q2,null),(Session[“total”],null}
2,{(q1,<=0) ,(q2,null),(Session[“total”],null} 2,{(q1,> 0),(q2,null),(Session[“total”],null)} 3,{ (Session[“total”],null)}
{(q1,> 0),(q2,null),(Session[“total”],null)}
2,{(q1,<=0) ,(q2,null),(Session[“total”],null} 2,{(q1,> 0),(q2,null),(Session[“total”],null)} 3,{ (Session[“total”],null)}
{(q1,> 0),(q2,null),(Session[“total”],null)}
Figure 9-2. State transition diagram after executing Step 2 of STD algorithm
Repeat step 2 until has no new states and state transition diagram as shown in Figure 9-1.
3,{(Session[“total”], !null)}
The new state: S1
2
, ,(q1,<=0),(q2,null),(Session *“total”+, null)-30
Chapter 4 Dynamic stubbing technique for collaborative testing
In the chapter, we first provide an example to illustrate that different assignment leads to distinct test cost in collaborative testing. The formal problem formulation of Minimum Test Cost Problem (MTCP) in collaborative testing is presented in this chapter. We also prove MTCP is NP-complete and hence propose a heuristic-based dynamic stubbing algorithm to solve this optimization problem.
4.1 Motivating example
Figure 10 shows a simple Web application which contains nine states and eight sub-problems. Assume that each sub-problem needs to be executed once for completing test and three folk testers A, B, and C are involved in this test. The completion time of each sub-problem by different folk tester is listed in Table 3. The due time of collaborative testing is set to be the last folk tester finishing the assigned jobs.
Problem 0
Figure 10. Problem decomposition scenario
31
Table 3. Sub-problem completion time of each folk tester matrix
0.1.1.1 0.1.1.2 0.2 0.3.1 0.3.2
Tester A 160 sec 160 sec 80 sec 120 sec 120 sec Tester B 165 sec 165 sec 85 sec 125 sec 125 sec Tester C 170 sec 170 sec 90 sec 130 sec 130 sec
Two different assignments of the example are shown in Table 4 and Table 5. In each table, entryij=1 represents that the jth job is assigned to ith tester. Assignment 1 represents that three folk testers have centralized preference on sub-problem 0.1.1.1 and assignment 2 is unbiased on testers‘ preferences. The due time of these two assignments are 390 sec and 300 sec as shown in Figure 11 and 12. This example illustrates that different assignment leads to distinct test cost (due time) and the centralized preferences of folk testers will delay the whole collaborative testing.
Table 4. Assignment 1 for shopping Web-site testing
0.1.1.1 0.1.1.2 0.2 0.3.1 0.3.2
Tester A 1 0 0 1 0
Tester B 1 1 0 0 0
Tester C 1 0 1 0 1
Table 5. Assignment 2 for shopping Web-site testing
0.1.1.1 0.1.1.2 0.2 0.3.1 0.3.2
Tester A 1 0 0 1 0
Tester B 0 0 0 0 1
Tester C 0 1 1 0 0
32
Figure 11. Due time of assignment 1
Figure 12. Due time of assignment 2
4.2 Minimum Test Cost Problem in Collaborative Testing
Under Internet environment, there are a large number of free and experienced folk human resources. Collaborative testing (or called Beta test) is usually used in online game and open source software to reduce test cost in software development stage. Based on the idea of beta test, we use folk testers in Internet to help us finding and reporting bugs. There are some constraints while applying collaborative testing. First, folk testers may not start to test at the
0 200 400 600
Tester A Tester B Tester C
0 100 200 300 400 Tester A
Tester B
Tester C
33
same time and they may delay to start the next test job after finishing current jobs. Second, we only consider those folk testers contributing themselves to collaborative testing. Therefore, each folk tester solves one sub-problem at least. Third, due to different complexity of each sub-problem, each sub-problem has distinct support threshold for completion. And then, the quality of folk testers (trustworthy) is different in Internet; therefore, it needs to consider the accumulation of the testers‘ trustworthy, rather than the number of testing. Final, each folk tester has different workload.
Before presenting our problem formulation, we first introduce the notations in Table 6.
The variables i represents the ith folk tester, the variable j represents jth sub-problem, and the variable k represents kth assignment of the sub-problems. We use the binary variable. σijk equals to 1 represents that the jth sub-problem is assigned to the ith folk tester in the kth
assignment. Otherwise, σijk equals to 0. TDe(i, j) stands for the delay time of the ith folk tester start to solve the jth sub-problem, TEx(I, j) stands for the jth sub-problem execution time of the ith folk tester, and T (i, j) stands for the jth sub-problem completion time of ith folk tester. Wi represents the trustworthy of ith folk tester and Hi represents the workload of the ith folk tester.
STj represents the support threshold of jth sub-problem.
34
Table 6. Notations of MTCP in collaborative testing i = ith folk tester
j = jth sub-problem
k= kth assignment of the sub-problems
σijk= ith folk tester does the jth sub-problem in the kth assignment TDe(i, j)= the jth sub-problem delay time of the ith folk tester TEx(i, j)= the jth sub-problem execution time of the ith folk tester T (i, j)= the jth sub-problem completion time of ith folk tester Wi= the trustworthy weight of ith folk tester
Hi= the available time limit of the ith folk tester STj=support threshold of the jth sub-problem
4.2.1 Problem formulation
The Minimum Test Cost Problem (MTCP) in collaborative testing can be formulated as IP-formulation. The objective function is the minimum of due time on different assignment, when due time is the maximum of the sum of sub-problem‘s time of each folk tester. The constraint 6 is the complete condition that each sub-problem need to be tested at least support threshold. The formulation of MTCP is presented below:
35
Definition 7: Minimum Test Cost Problem (MTCP) definition Objective function:
36
4.2.2 NP-Complete problem
In the section, we introduce an NP-complete problem, Job Assignment Problem (JAP) [30]. Then, JAP can be reduced in polynomial time to MTCP to complete the proof as followed.
The corresponding decision problem of Minimum Test Cost Problem (MTCP):
MTCP = {<D, J, U, ST, W, H, T, t>}
D= (S, δ, S0) is a directed tree.
J is a set of sub-problems in D.
U is a set of folk testers.
ST is a function form JR+ W is a function form U(0, 1]
H is a function form UR+ T is a function form UJR+
And there is an assignment with due time at most t
Theorem 1
Minimum Test Cost Problem is NP-Complete.
Proof:
First, we show that MTCP belongs to NP. Given an instance of the problem, the verification algorithm checks that sum of trustworthy Wi of assigned sub-problems of every folk tester i of each sub-problem j exceeds the support threshold STj, the sum of the completion time of assigned sub-problems of each folk tester i does not exceed Hi, the assignment of the sub-problems differs from the other assignment of the sub-problems, and checks whether the maximum of the sum of the completion time of assigned sub-problems of each folk tester is at most t. This process can certainly be done in polynomial time.
37
Second, to prove that MTCP is NP-Hard, we show that JAPp MTCP. Let G=(V‘,E‘), J‘={Pi|i=1,…,n}, U‘={Ui|i=1,…,m}, d(i, j)=0, the confirm function S(k) ∀k∈Pi, the trustworthy function w by w(i, j) where i ∈U‘and j ∈J‘, the human resource function H by H(i)
∀i∈U‘, the execution time function T by T(Pi, j) ∀Pi∈J‘, j∈U‘, and the maximum total cost at most t of JAP. We construct an instance of MTCP as follows. We form the tree D= (S, δ, S0) where S=Pi, δ=0, S0=Pi and we define the test sub-problem set J=J‘, the folk tester U=U‘, the support threshold function ST by STj=1 ∀j∈J, the trustworthy function W by Wi=w(i,j) ∀i∈U, the available time function H by Hi=H(i) ∀i∈U, the completion time function T by T(i, j)=T(Pi, j) ∀i∈U, ∀j∈J.
The instance of MTCP is then <D, J, U, ST, W, H, T, t>, where is easily formed in polynomial time.
We now show that graph G‘ has an assignment δ of the maximum total cost at t if and only if the tree D‘ has an assignmentσ‗ of the maximum due time at most t. Suppose there is an assignmentδwith maximum cost at most t. Therefore, there exists an assignment σ‘ such that ijk 1 if ij 1, the support threshold STj=1 ∀j∈J , the trustworthy weight of folk tester Wi=w(i,j) ∀i∈U, the available time Hi=H(i), the completion time T(i, j)=T(Pi,j) ∀i∈U,
∀j∈J. Thus, the assignment σ‘ is feasible solution and the maximum due time is t. Conversely, suppose that there is an assignment σ‘ with the maximum due time is t. Then, there existed an assignment a such thatij ijk, the trustworthy w(i, j)=Wi ∀i∈U‘ and j∈J‘, the human resource H(i)=Hi ∀i∈U‘, the execution time T(Pi, j)=T(i,j) ∀Pi∈J‘, ∀j∈U‘. Thus, the assignment is a feasible solution and the maximum total cost is t. Hence, MTCP is NP-Complete #
38
4.3 Dynamic stubbing algorithm for Minimum Test Cost Problem
Because Minimum Test Cost problem (MTCP) is an NP-Complete problem, we propose heuristic approach to solve MTCP. First heuristic is to assign new coming tester the job which requires the most effort to compete. This heuristic can speed up the whole testing. However, the sub-problem completion time of different folk testers is distinct. Therefore, we further predict the completion time to balance the following job assignments. Second heuristic is to assign tester with high trustworthy the most doubting job. Since the quality of folk testers is not the same, there may be opposite result on the same sub-problem. The second heuristic is used to improve the quality of test report.
Before presenting our heuristic-based approach, the used notations are introduced in Table 7. U represents the set of the folk testers, and Wi represents the trustworthy of the ith
folk tester. Mj represents the complexity of jth sub-problem. Sj represents the testing support of the jth sub-problem and STj represents the support threshold of the jth sub-problem. TA (i, j) stands for the actual execution time of the jth sub-problem done by the ith folk tester, and TEv(i, j) stands for the evaluation time of the jth sub-problem done by the ith folk tester. FP (Sk) represents the average trustworthy of folk testers which report bugs at the state k, and FN (Sk) represents the average trustworthy of folk tester which don‘t report bugs at the state k. The details of these notations and the following proposed dynamic stubbing algorithm are listed below:
39
Table 7. Notations of Dynamic stubbing algorithm Folksonomy user:
U= {i | i is the ith tester} is a set of folk testers Trustworthy weight of the ith folk tester:
Wi ∈ (0, 1]
Complexity:
Mj=#line of code in sub-problem j Support of the jth sub-problem:
Sj=
Support threshold weight of the jth sub-problem:
STj=Mj*c, where c is a constant
Actual execution time of the jth sub-problem, which had done by the ith folk tester:
TA (i, j) ∈ R+
Evaluation time of the jth sub-problem by the ith folk tester:
TEv (i, j) ∈ R+
Average trustworthy weight of folk testers which report bug at the state k FP (Sk) =
Average trustworthy weight of folk testers which don‘t report bug at the state k FN (Sk) =
40
Dynamic stubbing algorithm Input:
User Profile
TEv(i, j):evaluation time of sub-problem j of tester i State Transition Diagram D=(S, δ, S0)
S (PR): a set of all sub-problems PR: a set of sub-problems Output:
Testing time Method:
Initial: PR=null, for each sub-problem j in S (PR) InitialMetadata(j)
Step1: for each sub-problem in S (PR) find the set of sub-problem S (PR) which the most required tested
1.1: PR=FindTestedProblem(S (PR)) Step2: GuidingTester(PR,Ui)
Step3: IF the jth sub-problem of the ith tester has done 3.1: updating Sj=Sj + (TA (i, j)-TEv(i, j))*Wi
Step4: IF (Sj>STj, ∀j) Testing Finish ELSE
Go to step1
41
Subroutine: InitialMetadata Input: Sub-problem j
Output: Sub-problem j‘
Method:
Step1: According to the code complexity of j set up the STj of sub-problem j STj <- X
Sj <- 0 Step2: return j‘
Subroutine: FindTestedProblem Input: S (PR): a set of all sub-problems Output: a set of sub-problem PR
Method:
Initial: B=S (PR)
Step1: for each problem jin B
1.1: T (j) = (length (j) +1) [(Sj/STj) +α
42
Subroutine: GuidingFolkTester Input:
State transition diagram D F: all user profile
A set of sub-problem PR A set of folk tester Ui Output:
Method:
Initial: B=PR
Step1: for each sub-problem j in PR
Return arc min{
( ( ) ( ))| }Assign j to the most trustworthy of tester i in Ui
ELSE
Assign arc max{length(j)}
j to the tester i in Ui
2.2: Sj=Sj+TEv(i,j)*Wi
Step3: According to j from left to right
3.1: apply D and F to assign input value which is not tested to the Tester 3.2: for each link L in the tested page
Block link button except the link of entering to the next tested page.
3.3: remove i from Ui 3.4: remove j from B
Step4: repeat step1 until Ui is null
43
Chapter 5 Implementation and Experiment 5.1 System architecture and implementation 5.1.1 System architecture
In this section, we propose our two-phase collaborative testing system architecture including preprocessing phase and testing phase, as shown in Figure 13. In phase I, we convert the ASP.NET with C# language of the web applications into value-oriented dependence graph by value-oriented dependence graph construction algorithm. And then, we transform value-oriented dependence graph into state transition diagram for problem decomposition by considering users‘ runtime behaviors. In phase II, according to the state transition diagram, the tester profiles and portfolios, dynamic stubbing algorithm assigns jobs to each new coming tester. After the testing finishing, we analyze these bugs which folk testers reported based on report analysis algorithm.
Phase I. Pre-progressing Phase Phase II. Testing Phase
Web Site
Figure 13. Intelligent collaborative testing system architecture
44
5.1.2 System implementation
We implement a prototype system, intelligent collaborative testing system (ICTS), to support collaborative testing. When using ICTS, new tester has to register by filling out a simple questionnaire. This questionnaire can be used to estimate the trustworthy of tester and the completion time of each sub-problem. The screenshot of register page of ICTS is shown in Figure 14. The tutorial of ICTS, shown in Figure 15, is provided for testers to understand how to test on ICTS. After understanding how to test on ICTS, testers can start to test Web-site.
ICTS guides testers to test state of the most required and sets all values of tainted variables in branch predicate for testers. The guide of ICTS, shown in Figure 16, the main frame is the current tested page and the left column records tester‘s feedback on current page. Finally, complete information, shown in Figure 17, occurs if the test criterion is met.
45
Register testing account 1.
2.
Register successfully and login
Figure 14. Screen shot of register page of ICTS
46
3.
3.1
Figure 15. Screenshot of ICTS tutorial
3.
Figure 16. Screenshot of ICTS guiding
47 4.
Figure 17. Screenshot of complete information of ICTS
5.2 Experimental design and results 5.2.1 Experimental design
We demonstrate our proposed approach on a open source Web application ―BookStore‖
[http://www.gotocode.com/], which contains nine pages (AdvSearch page, BookDetail page, Books page, Default page, Login page, MyInfo page, Registration page, ShoppingCart page, and ShoppingCartRecord page). We convert ―BookStore‖ into value-oriented dependence graph by the value-oriented dependence graph construction algorithm, and then transform value-oriented dependence graph into state transition diagram. The corresponding statics of value-oriented dependence graph of ―Bookstore‖ are provided in Table 8, respectively. And then we transform value-oriented dependence graph into state transition diagram with 28 states.
48
Table 8. Value-oriented dependence graph statistics
Level Type Quantity
Page Level Node 9
Edge 52
Function Level Node 146
Edge 136
Code Level Node 351
Edge 546
Total Node 506
Edge 734
In the experimental design, we design a real testing environment of the web application
―BookStore‖. The folk testers of this experiment are gathered via social network sites such as Facebook and msn. Therefore, these folks have basic internet access skills. The ages of testers are between 15 and 30. These folk testers are further split into control group (85 testers) and experimental group (59 testers). The testing periods of control group and experimental group are 2011/6/1~2011/6/6 and 2011/6/1~2011/6/3, respectively. The stopping criterion of these tests is that the support of each state exceeds its support threshold.
5.2.2 Experimental results
Experimental result I-Efficiency evaluation
Centralized preferences of folk testers are the major cause of the delay of collaborative testing. Figure 18 shows that folk testers in control group prefer to test the first state and then second, 6th, 10th, 14th, 20th, 21th and 22th state. Compared to control group, our proposed intelligent collaborative testing system can reduce this kind of bias. Table 9 shows that experimental group has much less standard deviation in state complete degree than control group (0.2928 v.s. 0.9268). This points out that our proposed algorithm can balance the job assignment.
49
Table 9. State complete degree of testing comparing
Group Mean S.D.
Experimental group 0.6907 0.2928
Control group 0.9404 0.9268
Figure 18. Comparison of state complete degree of testing
Table 10 shows the average contribution (test time) of each folk tester on this experiment.
We further apply independent t-test on control group and experimental group. There is no significant difference (p=0.1240) in contribution of each folk tester between control and experimental groups. In fact, testers in experimental group contribute less. Table 11 shows the comparison of unit number of online folk testers of control group and experimental group.
This result shows that the contributions of testers in unit time of two groups are the same.
According to the above results, these experiments are fair for experimental group and control group. Based on these fair comparisons, the due time of the experimental group can be reduced to 50% of the control group, shown in Figure 19. Hence, our proposed dynamic stubbing algorithm can speed up collaborative testing.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Control Experimental
50
Table 10. Comparison of test time between two groups
Group N Mean S.D. p-value
Experimental group 59 150.6271 236.1593 0.1240
Control group 85 244.5647 482.3225
Table 11. Comparison of the folk testers per due time
Control group Experiment group
Folk testers/Due time 0.0050 0.0061
Figure 19. Due time comparison
0 2000 4000 6000 8000 10000 12000 14000 16000 18000
Control group Experiment group
Control group Experiment group