In the previous chapter we described how each component of SPAG-C works, now we proceed to explain how each component was implemented.
5.1 Oracle Client
As stated before the oracle client is coupled with the record-replay component. In our case that component is SPAG. Since SPAG is implemented in Java, the oracle client is also implemented in Java.
Most of the code of the oracle client is automatically created by the tool “wsdl2java”
which is part of the Apache CXF framework (an open source services framework) [22].
“wsdl2java” simply takes the WSDL file exposed by a Web service (in this case the oracle synchronizer), and automatically generates Java code from which to call the service. With that code in place, we added event listeners to SPAG to automatically call the oracle synchronizer after any mouse event in order to perform the automatic verification process described in the previous chapter. During the replay phase, when the test case execution reaches the verification command, the client will make a call to the oracle synchronizer asking it to perform the requested action.
5.2 Oracle Verifier
The oracle verifier makes use of OpenCV (via EmguCV) and .NET framework functionality to compare, crop and rotate images.
EmguCV is as cross-platform .NET wrapper for the OpenCV image processing library [24] that allows calling OpenCV functions from any of the .NET compatible languages (in this case C#). OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision [25]. In this work we only use OpenCV’s functions to extract and compare SURF features, to calculate and compare image Histograms, to perform template matching, and to detect Canny Edges.
Detecting ROI (Region of Interest)
In order improve accuracy, we remove the undesired content of the images by detecting the ROI on the device’s screen in two ways: manually and automatically. Since the tester can see in the computer the input of the camera, he can simply drag a rectangle around the desired ROI. After that, the oracle verifier will automatically crop any image captured. The automatic method, illustrated in Figure 8, makes use of an image processing technique called Canny
22
Edges to detect the borders of the screen. However, because Canny Edges will detect all edges in the image, it is important to make the device’s screen as white as possible. After edges are detected, as shown in Figure 8 above “Canny edges”, we find contours on the image but filter them in order to keep only quadrilateral figures. Then, the quadrilateral figures are filtered by the angle of their vertices; because we are looking for the screen of the device (i.e., a rectangle), we only keep those figures with angles between 80 and 100 degrees. Finally, we filter the rest of the figures by the size of their area, since the desired ROI should be the biggest rectangle detected. The result after the filter process is shown in Figure 8 above “Area detected”.
Figure 8: Process used to detect the device's screen (ROI) SURF
In the previous chapter, we described most of the SURF comparison process; however, there are some implementation details that are worth mentioning. There are many ways to match SURF features. In this work we use OpenCV’s BruteForceMatcher to perform KnnMatch (k-nearest neighbor match). KnnMatch returns the K nearest neighbors, K = 2 in our case, using a Euclidean Distance (L2 Distance). After the match is performed, we filter the matched features by using OpenCV’s function VoteForUniqueness, which discards non-unique matches, with a non-uniqueness threshold of 0.9; and VoteForSizeAndOrientation, which discards those features whose size and orientation does not match the majority’s size and orientation, using a scale increment of 1.5 and 20 rotation bins.
Histogram
In order to perform color histogram comparison, we need to first extract the color
23
histogram of each image. However, we first reduce the colors of the images to get a reliable similarity measure and improve computation efficiency [25] on each one of the three channels of the image: red, green, blue (RGB). After reducing the colors, we calculate the color histogram by calling OpenCV’s DenseHistogram.Calculate method on each one of channels and compare the histograms by calling the cvCompareHist method.
Template match
As described in Chapter 2, template matching finds a given small image in a larger image.
However, since our automatic verification automatically decides what images will be used as the expected states the tester cannot provide the templates. Therefore, we automatically split the expected state into smaller images, and match each of those small images against the current state. This means that several template matches are performed; if at least one of the matches has a value smaller than the provided similarity threshold then the overall match is considered a failure.
OpenCV implements different methods to match templates. In this work, we use the MatchTemplate function using CV_TM_CCORR_NORMED, which returns a value between -1 and 1, where the higher the value the better the match, and vice versa.
5.3 Oracle Synchronizer
We use Microsoft WCF and C# to implement the oracle synchronizer. Microsoft WCF is a framework for building service-oriented applications [23] that facilitates building interoperable Web services using different standards.
The oracle synchronizer exposes three methods: AddCheckpoint, VerifyState and CaptureScreen. Each of the methods receives a different set of parameters. The AddCheckpoint method is only called during the record phase, it receives the difference threshold as well as the previously captured screenshots 𝑐𝑗𝑝 and 𝑐𝑗𝑝−1, and performs the automatic verification process. The VerifyState method is only called during the replay phase, it receives the expected state 𝑠𝑗𝑝, captured during the record phase, and the similarity threshold, and performs the verification process. Finally, the CaptureScreen method is called during both phases, it does not receive any parameters, it simply captures a screenshot and returns its path.
Additionally, the oracle synchronizer is in charge of logging all relevant information each time a call to the service is done. For example, during the verification process, it will log what images have been compared, what similarity thresholds we had, how long the image
24
comparison process took and whether the verification process failed or passed.
5.4 Image maintenance
SPAG-C captures a lot of images but not all of them are required in order to replay a test case. In order to automate image maintenance, we name images randomly by using .NET’s Path.GetRandomFileName method, and we prefix image names with the word “record”, in case of 𝑠𝑗𝑝, and “replay”, in case of 𝑠′𝑗𝑝.
Of all the images captured during the record phase, only the expected states are kept, and the rest are all discarded automatically. Expected states are required in order to replay a test case; therefore, these images are not deleted.
Each time a test case is replayed, a new set of current states will be captured. This means that a lot of duplicated images will be stored on the host computer. In order to alleviate this problem, testers can simply call the helper method deleteUnnecessaryImages to remove all images captured during previous rounds, hence keeping only the images that will be captured during the current round.
25