As the quantity and breadth of Web-based software systems continue to grow rapidly, assuring the quality and reliability of this software domain is becoming critical.
Low reliability of software can result in serious detrimental effects for businesses, consumers, and the government as they increasingly depend on the Internet for routine daily operations. A major impediment to producing reliable software is the labor and resource-intensive nature of software testing. Software testing processes, in general, are labor-intensive processes and involve substantial collaboration between testers, developers, and even users. In testing, testers are requested to assess software in restricted time, and report bugs back to the development team. In collaborative testing, testers need to share the test result and test status to the others. Since the test results are needed to report to developer and users, the communication is necessary under test processes. In some online collaborative software testing, such as Web application testing, game beta testing, open-source software testing, how to utilize a large amount of human resources on the Internet to perform test is interesting, so a collaborative tool to support collaborative testing is necessary. Under the situation of a complex architecture of Web application, the problem of how to perform testing rapidly and support collaborative work automatically should be considered for Web applications testing.
Recently, many approaches have been proposed to address Web application testing and software engineering collaboration. In [31], Whitehead classified collaboration tools of software engineering into four categories, model-based collaboration tools, processes support tools, awareness tools, and collaboration infrastructure. However, to the best of our knowledge all existing collaborative testing tools just focus on testing processes and generating bug report. There are no tools supporting collaboration test
with the consideration of communication requirement and testing resource constraints under test scheduling. Under Internet environment, there are a large amount of various and experienced human resource. And it is appropriate for collaborative testing of software functionality and usability by utilizing folk resource on the Internet. To test Web application with folk testers on Internet, we proposed a folksonomy-based approach to support collaboration and constructed a prototype tool with proposed approach.
In collaborative testing, we are concerned with the following issues [18][24][34][38]. (1) Software testing are inherently cooperative, requiring many testing engineers to coordinate their efforts to perform a software system testing. To coordinate this effort, we guide test activities and share information each other over the entire testing process. How to propose a collaboration technology within testing process is an interesting issue in collaborative testing. (2) In Web application testing, the combination of path executions and input parameters may cause test case explosion problem. With a large amount of test cases and restricted resources, how to assign the required test cases or jobs to corresponding testers becomes an important research topic.
(3) Web applications typically involve complex, multi-tiered, heterogeneous architectures including Web sites, applications, database servers, and clients. How to perform Web application modeling for testing and consider a variety of situations to be able to handle the testing of various components in these architectures is also a challenging issue.
3
constructions and information flows of Web application with the defined three levels program dependence graphs, code-level dependence graph, function-level dependence graph, and page-level dependence graph. Phase 2 contains folksonomy-based collaborative testing. We implement a prototype system to evaluate our proposed approach. With our collaborative testing tool, we can collect user sessions logs of participated testers during testing processes. To support collaborative testing, we further analyze the user session logs and guide the test activities. The results will be analyzed and utilized to improve the way of collaborative testing. In the experiment, we simulate the collaborative test activities with our proposed model. The experimental results show that our approach is effective and well performance in testing and defects revealing.
And the average reduction rate for testing effort is almost 90%.
The main contributions of this research are as follows:
1. Model Web application with the dependencies, data dependence and control dependence, and propose the formulation of the Web application for collaborative testing.
2. Define the collaborative testing problem as ILP-formulation and prove the problem is NP-Complete.
3. Propose a folksonomy-based approach to support collaborative test of Web applications and implement a prototype tool based upon the approach.
4. Evaluate the experiments of the effectiveness of the proposed folksonomy-based approach.
The remainder of this research is organized as follows: Chapter 2 discusses related works. Chapter 3 gives an overview of the proposed approach. Chapter 4 explains how the test approach is performed in test scenarios. Chapter 5 explains how the experimental design and experimental result. Section 6 presents the conclusion and proposes future work.
Chapter 2. Related Work
2.1 Web application testing approach
Recently, many approaches have been proposed to address Web application testing.
We briefly describe relevant studies as follows. Ricca et al. [17] described an approach based on UML model for high level abstraction Web application. Similar to their approach, Jia et al. [48] presented a technique based on formal specification instead of UML model. These approaches are not appropriate to cope with dynamic behavior of modern Web application. Kung et al. [12] presented an approach based on multiple models of Web application under test. Andrews et al. [1] proposed a system-level testing technique that combines test generation based on finite state machines with constraints. These approaches have state space explosion problem if the Web application is very complicated.
Deng et al. [49] proposed an approach based on static analysis for recovering the model of Web application. In their approach, the model of the Web application is built by scanning its source code to identify links and names of input parameters. However, their approach only identifies the names of input parameters without considering its domain information, such as parameter types, relevant values of the parameters, so it might not be applied directly to generate test case. William et al. [47] presented a technique for automatically discovering Web application interfaces based on a two-phrase static analysis algorithm. To generate test cases, the problem of how to model the behavior of Web application should be considered in Web applications
5
construct test cases for the collected user sessions [42], [30]. The promising results demonstrated the fault detection capabilities and cost effectiveness of user-session-based testing. In addition, they observed that the effectiveness of user-session-based testing can be improved as the number of collected sessions increases; however, the cost of collecting, analyzing, and replaying test cases also increases. User-session-based testing techniques are complementary to the testing performed during the development phase of the application [8], [20], [17], [29], [7], [49], [34].
It is a serious problem that how to select test cases from a large number of test cases when testing the application. Since test suite reduction has several advantages, such as reducing the cost of executing, validating, and managing test suites as the application evolves, the test suite reduction is important for Web application testing.
Several test suite reduction techniques have been proposed [37], [45], [28], [26], [13], [41], [14], [15]. Harrold et al. [37] developed a test suite reduction technique that employs a heuristic based on the minimum cardinality hitting set to select a representative set of test cases that satisfies a set of testing requirements. Harder et al.
[36] proposed an operational-abstraction-based minimization technique that can be executed incrementally, but dynamically generating operational abstractions can be costly. In this research, we propose a novel approach to support collaborative testing for Web application and construct a new model with folksonomy-based approach.
2.2 Collaborated-Based approach in Software Engineering
In software engineering, participants have adopted a wide range of communication and collaboration technologies to assist in the coordination of project work[31].
Engineers have developed a wide range of model oriented technologies to support collaborative work on their projects. These technologies span the entire lifecycle, including collaborative requirements tools [3], collaborative UML diagram creation, software configuration management systems and bug tracking systems. Process modeling and enactment systems have been created to help manage the entire lifecycle, supporting managers and developers in assignment of work, monitoring current progress, and improving processes [19] [5] . In the commercial sphere, there are many examples of project management software, including Microsoft Project and Rational Method Composer. Several efforts have created standard interfaces or repositories for software project artifacts, including WebDAV/DeltaV [32] [16] and PCTE [33].
Web-based integrated development environments serve to integrate a range of model-based (SCM, bug tracking systems) and unstructured (discussion list, web pages) collaboration technologies.
There are many researches related to collaborative testing. However, most collaborative testing tools above just focus on testing processes and generating bug report. There were no tools supporting collaboration test with considering communication requirement and testing resource constraints under test scheduling in fact. To test Web application with folk testers on Internet, we proposed a
7
2.3 Folksonomy-Based approach
A folksonomy is a system of classification derived from the practice and method of collaboratively creating and managing tags to annotate and categorize content; this practice is also known as collaborative tagging, social classification, social indexing, and social tagging. Folksonomy, a term coined by Thomas Vander Wal, is a portmanteau of folk and taxonomy.
General overviews on folksonomy systems and their strengths and weaknesses are given in [46][4]. In [40], Mika defined a model of semantic social networks for extracting lightweight ontologies from del.icio.us. Recently, work on more specialized topics such as structure mining on folksonomies—e. g. to visualize trends [35] and patterns [9] in users’ tagging behavior—as well as ranking of folksonomy contents [2], analyzing the semiotic dynamics of the tagging vocabulary, or the dynamics and semantics [23] have been presented.
In the literatures, the existing approaches usually focus on the collaborative filtering and information retrieval areas. AutoTag [21], e.g., is a tool that suggests tags for weblog posts using information retrieval techniques. Xu et al. [50] introduced a collaborative tag suggestion approach based on the HITS algorithm [27]. A goodness measure for tags, derived from collective user authorities, is iteratively adjusted by a reward-penalty algorithm. Benz et al. [11] introduced a collaborative approach for bookmark classification based on a combination of nearest-neighbor classifiers. A keyword recommender plays the role of a collaborative tag recommender, but it is just a component of the overall algorithm. Besides, the standard tag recommenders, in practice, are services that provide the most-popular tags used for a particular resource.
This is usually done by means of tag clouds where the most frequent used tags are depicted in a larger font or otherwise emphasized.
The approaches described above address important aspects of the folksonomy,
collaborative intelligence. Collective intelligence is a shared or group intelligence that emerges from the collaboration and competition of many individuals and appears in consensus decision making in bacteria, animals, humans and computer networks.
9
2.4 Program slicing and program dependence diagram
Much related work has been performed over the past ten years in the area of dependence-based program representations. The traditional syntactic view of software dependency had its origins in compiler optimizations, and focused on control and dataflow relationships [25]. This approach extracts relational information between specific units of analysis such as statements, functions or methods, and source-code files. Dependencies are discovered, typically, by analysis of source code or from an intermediate representation such as bytecodes or abstract syntax trees. These relationships can be represented either by a data-related dependency (e.g., a particular data structure modified by a function and used in another function) or by a functional dependency (e.g., method A calls method B).
The work by Hutchens and Basili [22] and Selby and Basili [43] represent of the first use of dependency data in the context of a system’s propensity for failure. Based on the concepts of coupling and cohesion proposed by Stevens et al. [44], Hutchens and Basili [22] presented metrics to assess the structure of a system in terms of data and functional relationships, which were called bindings. The authors used clustering methods to evaluate the modularization of a particular system. Selby and Basili [43] used the data binding measure to relate system structure to errors and failures. They found that routines and subsystems with lower coupling were less likely to exhibit defects than those with higher levels of coupling. Similar results have been reported in object-oriented systems. Chidamber and Kemerer [10] proposed a set of measures that captures different aspects of the system of relationships between classes. Briand et al.
[6] found that the measures of coupling proposed by Chidamber and Kemerer were positively associated with failure proneness of classes of objects.
In this research, we use program slicing techniques to construct dependence graphs.
To represent Web application, the three-level dependence graphs are constructed in
Chapter 3. And different graphs stand for different semantic meanings of software testing.
11
Chapter 3. Folksonomy-based approach for collaborative testing
In this chapter, we will introduce our proposed approach to support collaborative testing for Web application. First, we describe a motivating example of job assignment problem. Then, we define the job assignment problem based upon mathematical definition. We construct our problem model as ILP-formulation by considering resources constrains and job assignment under testing. The problem is proved as an NP-Complete problem. We reduce the problem to minimal representative set problem and prove the problem is NP-Complete. In the last section of Chapter 3, we propose the heuristic ideas, dependence graphs and folksonomy-based approach, to solve the problem.
3.1 Motivation Example
Under Internet environment, there are a large amount of various and experienced human resource. And it is appropriate for collaborative testing of software functionality and usability by utilizing folk resource on the Internet. However, people usually execute some specific popular function of the application when they face a new system without interact each other. When participated testers focus on specific popular functions, the resources are wasted in overlapped and duplicated testing. And the lack of testing of the other pages will result in unequal distribution of resources and slow test convergence. In this situation, it seems that the problem is associated with job scheduling and resource allocation.
Figure 1. Folksonomy testing scenario
13
Table 1. Job-Page covered matrix
Page 1 Page 2 Page 3 Page 4 Page 5
Job 1 1 1 1 0 0
Job 2 1 0 0 1 1
Job 3 1 0 1 1 0
Job 4 1 1 1 1 0
Table 2. Job execution time matrix
Tester A Tester B Tester C open-source project, ―Bookstore‖[http://www.gotocode.com/], which contains five Web pages and 4 jobs, where each job consists of several pages and possible executed paths.
For example, the Job 1 in Figure 1 can be represented as (P1P2P3). As a result, the job-page covered matrix is formed as Table 1. The testing finishes when all of pages are executed at least one time. There are four testers participated in this testing and the job executing time for each tester can be estimated by summarizing corresponding page executing times. Tester A executes Job 1, consisted of page 1, 2, and 3, with 80 seconds in this testing. The Job execution time matrix is shown in Table 2 by summarizing execution times of testers.
We want to find out the minimal job quantity which can cover the whole page and make the minimal execution time of the maximum execution time of the user. When
testing begins, tester A, tester B, and tester C execute pages respectively. The testing is terminated after all of the pages can be executed at least one time. For the sake of testing, the testers should coordinate the jobs respectively and cooperate to get work done quickly. One of the solutions of this case is shown below:
Job 1: Tester A spends 80 Seconds.
Job 2: Tester A spends 120 Seconds.
However, we should further consider other factors in our testing according to the real environment. First, what is ―testing done‖ under testing? Each page should have same standard or not? We create the confirmed criteria for each page based on different complexity. In the page confirmed matrix, each page has he own threshold of testing quantity substitute for test once. Second, should we trust the testing results of testers?
Based on folksonomy-based approach, we create the tester trustworthy matrix with testers. By considering these constraints, the problem is more complex than original one.
Table 3. Page confirmed matrix
Page 1 Page 2 Page 3 Page 4 Page 5
Confirm condition 10 10 10 8 8
Table 4. Tester trustworthy matrix
Tester A Tester B Tester C
Trustworthy 1 0.5 0.7
15
―minimal‖ total completion time. It is a job assignment problem with resource constrains. And we can conclude some proposed problems below:
(1) Testing resource constrains should be considered in testing.
(2) What is the stop criteria of testing?
(3) How can we trust the testing results of participated testers?
3.2 Problem definition: job assignment in collaborative testing 3.2.1 Problem definition
In this section, we define the proposed problem we mentioned above with mathematical formulation. In Table 5, we show the notation definition used in the formulation. The model of proposed problem, job assignment problem, is formulated as ILP formulation in Definition 1.
We adopt three variables to represent jobs, testers, and Web pages. The variable i represents the ith job sequence, the variable j represents the jth tester, and the variable k represents the kth page of the web application. We use three binary decision variables to represent the status of variables i, j, and k. The variable ij equals 1 if the ith job assign to tester j and response successfully. The variable i equals 1 if job I is legal. And the variable dnm equals 1 if page n has a dependence edge link to page m. Some variables
are defined according to the constraints of above variable. The variable Hj stands for testers’ total working hours, and is used to represent the total completion time limit of jth tester. The function T(Pi,j) stands for the ith job completion time of tester j. The variable Wj stands for the trustworthy weight of tester j. And each tester has his own trustworthy weight when he executes testing works. The variable Sk stands for the support threshold of kth page, each page has his own testing terminate condition.
17
Table 5. Notations of Job assignment problem definition i : ith job sequence ; j : jth tester ; k : kth page
ij= ith job assign to tester j and response successfully )
, (P j
T i = ith job costing time of tester j
Wij= trustworthy weight of tester j to do ith job SK = support threshold of kth page
i = job i is legal
dnm= page n has a dependence edge to page m Hj= total testing time limit of jth tester
Definition1: Job assignment problem definition
The constraint (2) ensures the testing job is assigned. The constraint (3) ensures the job costing time is a positive real number. The constraint (4) ensures each tester has
19
3.2.2 NP-Complete problem
This section shows the job assignment problem in this study domain is a NP-Complete problem. First, we will show this problem is an NP-problem. Second, we reduce the ―Optimum representative set‖[39] NP-Hard problem to our problem to show it is an NP-Complete problem.
The corresponding decision problem of job assignment problem:
JAP={<G,J,U,D,S,W,H,T,k>}
G=(V,E) is a complete directed graph.
J is a set of walks in G.
And there is an assignment with total cost at most k.
Thm Job Assignment Problem (JAP) is NP-Complete Proof:
We first show that JAP belongs to NP. Given an instance of the problem, the verification algorithm checks that the sum of the trustworthy Wi of assigned legal test paths of every tester I of each page k exceeds the threshold SK, the sum of the execution time of assigned legal test paths of each tester I does not exceed the threshold Hi, and checks whether the maximum of the sum of the execution time of assigned legal test
We first show that JAP belongs to NP. Given an instance of the problem, the verification algorithm checks that the sum of the trustworthy Wi of assigned legal test paths of every tester I of each page k exceeds the threshold SK, the sum of the execution time of assigned legal test paths of each tester I does not exceed the threshold Hi, and checks whether the maximum of the sum of the execution time of assigned legal test