Chapter 3 Related Work 9
3.3 ARToolKitPlus
ARToolKitPlus (ARTK+) [9] [10], developed by Daniel Wagner, is a revision from ARTK.
The main purpose of ARTK+ is to improve the performance of ARTK and apply the AR tech-nique on the mobile devices. Since accuracy and efficiency are always a tradeoff, ARTK+ uses a special detection technique which allows a user to choose the best detection method for differ-ent purposes. A user can choose high efficiency but less robust detection method for the mobile devices or use robust method for high computer power PCs. The flexible structure of ARTK+
is illustrated in Figure 3.4.
The structure flow of ARTK+ can be separated into three phases. The first phase determines a threshold value. One of the choices is specifying fixed global threshold, which is adopted by ARTK. Another choice is to use dynamic global thresholding, which uses the random number
The second phase is in charge of marker detection. A user can choose the template marker and choose pattern matching method used in original ARTK, which is an efficient but less robust method. Another choice is to use the ID marker system used by ARTag. Such a choice supports up to 4096 different markers. The last phase is to estimate pose. The pose estimation contains two sub phases: signal marker and multiple markers estimation. However, ARTK+ can only detect fix arranged multiple markers, which needs to set up the module before detection. It can't detect stably for separated multiple markers since the pose estimation can only estimate the pose of one marker in this type of detection.
ARTK+ truly improves the performance of original ARTK and gives more reliable result under many circumstances. However, ARTK+ does not fit to detect separated multiple markers, and does not performs well for detecting multiple markers under unbalanced light conditions.
In addition, since the detection algorithm is similar to ARTK, ARTK+ also can't detect partial occluded markers.
3.4 Summary
We introduced three popular AR systems in this chapter, and discussed the problems for these systems. ARTK cannot handle unbalanced light detection and partial occluded marker detection. ARTag uses gradient methods to solve these two problems, but still can't detect inner pattern occluded markers. Another system, ARTK+, is a revision from ARTK and intends to improve the drawback of the unbalanced light detection. However, it is still unable to detect multiple markers under unbalanced light and the partial occluded markers. We will detail our designs to improve these problems in the next chapter.
Chapter 4
Selectable Thresholding Augmented Reality System
The section interprets the structure of STAR, including the labeling phase, marker detection phase, and pattern matching phase.
4.1 Tool Design
Selectable Thresholding Augmented Reality System (STAR) is a tool to deal with unbal-anced light detection and partial occluded marker detection. STAR is based on ARTK [6] [7]
which is an open source AR system and provides complete and detailed reference files. These characteristics make ARTK a suitable foundation than other existing AR systems. The purpose for STAR is to solve two problems for existing AR systems using planar markers (1) detect single and multiple markers under unbalanced lights, and (2) detect partial occluded markers.
Moreover, the detection methods we use must be efficient to fulfill the real-time characteristic of AR definitions.
Figure 4.1 shows the structure of STAR, which is very similar to the structure of ARTK, but we add new steps and modify some original steps to improve the detection performance. The structure can be divided into three phases: (1) labeling, (2) marker detection, and (3) pattern matching. We will mention the details of these three phases in the following sections.
Flat area removal
The main purpose for this phase is to find the number of objects in an image, and assign different labels to identify these objects. Thus, we can check these objects one by one to find marker candidates in the following phases. This phase is composed of three steps: (1) get thresh-old, (2) binarize image, and (3) component labeling. The first step includes two methods for a user to choose: (1) dynamic global thresholding, and (2) adaptive thresholding. After getting the threshold value, we can binarize the image and find the objects by applying the sequential labeling algorithm [19]. While we want to solve the detection under unbalanced lights, we focus
on the two methods to get thresholds and detail them in the following sections.
4.2.1 Dynamic Global Thresholding
Dynamic global thresholding is a fast and simple method, which is suitable for single marker detection. This method can get the correct threshold in a short period of time under the changing light conditions, and it can also cope with general unbalanced light conditions. Different with ARTK+ [9], our dynamic global thresholding method doesn't change threshold randomly, but change it in three values. These threshold values are used to handle dark, normal and bright conditions. To find the three values, we test the detection rate under common illuminance range (from 100 Lux to 2000 Lux) by using different thresholds, and note the results in Table 4.1.
From the test results, we choose the three thresholds as 70, 140, and 250 to deal with dark, normal and bright conditions separately.
Table 4.1: Detection under common illuminance range with different threshold
illuminance
\threshold
20 40 70 100 140 170 210 250100 100% 100% 100% 0% 0% 0% 0% 0%
During the marker detection, we will modify the threshold while the light condition changes.
Since we don't use optical sensors, we change the threshold while our system can't find markers in the previous frame. We won't stop changing the threshold until we find a marker in the frame, in order to detect the new-incoming markers under any light conditions. Figure 4.2 shows the examples of using dynamic global thresholding to detect markers under dark, normal and bright
Dark condition
(250 Lux) Normal condition (870 Lux)
Light condition (1840 Lux) Figure 4.2: Detection under different illuminance with dynamic global threshold
conditions.
4.2.2 Adaptive Thresholding
The adaptive thresholding method for STAR is based on the Bradley's [16] method we've mentioned in Chapter 2. In this method, we regard the center pixel which intensity is 5% lower than the region average as black. This method can provide correct results for multiple markers detection under unbalanced light conditions as showed in Figure 4.3. Nevertheless, this method is designed to process grey level images originally, which may get wrong results for processing color images. For example, the skin color which should be recognized as white color may be recognized as black color by applying the original Bradley's method. Thus, our adaptive thresholding method doesn't calculate one integral image with grey level but calculate three integral images with red, green and blue colors separately. A pixel is set to black while the three color intensities are less than the weighted average calculated by each integral image. This method may increase processing time, but it still suitable for real-time video process.
4.3 Marker Detection Phase
The marker detection phase is to find an object is whether a possible marker or not. This phase will consume a lot of time if we check every object one by one. Hence, we eliminate the
Original image Global threshold Adaptive threshold
Figure 4.3: Using global threshold and adaptive threshold to detect multiple markers under unbalance light
improper objects by the area first. Objects with too large or too small area are eliminated before this phase. The structure in this phase includes the following methods: (1) flat area removal, (2) boundary detection, (3) quadrangle detection, (4) boundary reconstruction, and (5) finding Inner boundary. The boundary detection method in the second step is the boundary following method [17][18] we've introduced in Chapter 2. We will introduce other new designed methods in the following sections.
4.3.1 Flat Area Removal
The boundary following algorithm can only detect outer boundaries of the template markers.
The outer boundary detection can failed easily when a marker is partially occluded. In these cases, we want to use the inner boundary to rebuild the marker. In this step, we try to reveal the hollow objects' inner boundaries by using Prewitt mask [20][18] to remove objects' flat area.
Figure 4.4 shows the Prewitt masks for both vertical and horizontal direction. If the pixel belongs to flat area, the results would be zero for both masks. We remove these pixels and reveal the outer and inner boundaries of the objects. These two boundaries will be checked in the following steps to determine an object is whether a marker candidate or not.
-1 -1 -1
Figure 4.4: (a) Horizontal Prewitt mask (b) Vertical Prewitt mask
4.3.2 Quadrangle Detection
While we use the square-shaped planar markers, it is easy to find marker candidates by finding square objects in the image frame. Nevertheless, with the affection of perspective pro-jection, the markers won't be in square shape but in quadrangle shape in the grabbed frame image. In this step, we try to find these quadrangle shaped objects by locating the corner points of their boundaries. The corner points stand for the points with the slope changes the most in the near areas. When we locate all of the corner points, we can figure out the shape of the objects' boundaries and find the quadrangle shaped objects.
Figure 4.5 illustrates an example of our quadrangle detection method. First, we take the lowest point of an object's boundary as a start point. In the next step, we try to sample the boundary by 50 sample points. After sampling, the slopes of each sample points are calculated.
Comparing the changing of these slopes, we can find a part includes the corner point. We call this part as "corner part", which is showed as the green segments in the Figure 4.5 (d). Then, we link a line between the start point and the second sample point of the corner part. With checking the distance from every point belongs to the corner part to the linked line, we can find the point with the longest distance as the corner point. After finding the corner point, we take the second sample point of the corner part as a new start point and reiterate the above steps until we find all corner points of the boundary. An object with its boundary has four corners is regarded as a
0.5 0.6 0.40.6 -4.5
(a) (b) (c)
(d) (e) (f)
Figure 4.5: Example of quadrangle detection (a) The start point (b) The sample points (c) Calculate the slopes of each sample points (d) The part includes corner point (e) The correct corner position (f) The object is regarded as quadrangle while four corners are found
candidate of complete marker, which can be recognized by pattern matching directly.
4.3.3 Boundary Reconstruction
The quadrangle detection method can find candidates of complete markers, but can't find the candidates of partial occluded markers. We want to find whether an object is a candidate of occluded marker or not in this step. Generally speaking, we can find more than four corners for the boundaries of occluded markers. However, an object with too many corners is impossible to be a marker. Hence, we only reconstruct the objects with their boundaries have more than four corners but less than fifteen corners.
Figure 4.6 shows an example of boundary reconstruction. With a marker occluded by a white triangle, we can find ten corners in the quadrangle detection step. We signed these ten corners as red circles and other sample points as blue circles in Figure 4.6 (a). First, we have to find two neighbored sides which are not occluded by the white triangle as the standard sides to
1.2 -0.8
(a) (b) (c)
(d) (e) (f)
Figure 4.6:Example of boundary reconstruction (a) The object with detected corners and sample points (b) The longest side of the boundary and the longer neighbored side (c) The three correct points (d) The slopes of the two shorter sides next to the standard sides (e) The intersection point of the two line equations (f) The reconstructed boundary
reconstruct boundary. We take the longest side of the boundary and the longer side next to it as the standard sides. Three correct corners can be found in these two standard sides. In the next step, we calculate the line equations of the two shorter sides next to the two standard sides. The intersection point of these two equations is the missing corner position. At the last step, we can rebuild the quadrangle boundary by three correct corners and one calculated missing corner.
This method can cope with the general occlude conditions and give stable results, but it may fail or give wrong results when the three sides of the inner boundary are occluded. Under this condition, it is useless to rebuild the boundary while the major part of inner pattern may also be occluded, making it impossible to be recognized by the partial pattern matching method in the next phase.
Boundary detection Quad detection
Figure 4.7: The example of failed outer boundary reconstruction
4.3.4 Finding Inner Boundary
Inner boundary is important for the occluded marker detection and very useful when the outer boundary reconstruction failed. This situation occurs while finding too many corners in the outer boundary or calculating incorrect slope or incorrect missing point position during the reconstruction step. Figure 4.7 shows an example of failed outer boundary reconstruction while detecting too many corners for the circular occlude object.
This step includes two sub-steps (1) remove the outer boundary, and, (2) find the upper left corner of the inner boundary. While we have removed the flat area, the inner boundary is left after removing the outer boundary in the first sub-step. We can use the boundary following method again to find pixels belong to the inner boundary after finding the upper left corner in the second sub-step. Like the detection steps for outer boundary, the inner boundary is also checked by the quadrangle detection step and, if needed, the boundary reconstruction step. If an object with quadrangle-shaped inner boundary is detected, it would be regarded as a candidate of occluded marker. The candidate will be checked by using the partial pattern matching in the next phase. Figure 4.8 shows the inner boundary detection example, which is the following steps of the example in Figure 4.7. With recheck the inner boundaries, we can deal with much more occlusion situations than methods only detect the outer boundaries.
Boundary detection Quad detection Boundary reconstruction Find inner contour
Success! Fail! Success!
Partial pattern matching
Figure 4.8: The following steps of Figure 4.7
4.4 Pattern Matching Phase
The marker detection phase can find marker candidates from an image frame, and we have to ensure the candidates are true markers by matching the inner patterns. This phase can not only eliminate the incorrect candidates but also recognize different markers to each other. The former is very important while our boundary reconstruction method may detect the background objects as marker candidates which need to be eliminated in this phase.
The pattern matching method we use is the Normalized Cross Correlation method [18] we've introduced in Chapter 2. An inner pattern will be matched by the template in four different ori-entations to find the direction of marker's rotation. However, this method doesn't give reliable matching results while the inner pattern is occluded. The solution is using partial pattern match-ing method for boundary reconstructed candidates while usmatch-ing the original method to match the ordinary candidates.
4.4.1 Partial Pattern Matching
The original pattern matching method is not robust for partial occluded markers. To increase the robustness, we use the partial pattern matching method, which is very similar to the original pattern matching method. We divide the inner pattern into four parts, and only match the parts
Hiro
Figure 4.9: An example of an occluded marker. The orange area shows the parts for pattern matching.
which are not occluded. We show an example of occluded marker and the matching parts in Figure 4.9.
The first step in partial pattern matching is to find the position of occluded parts. In the marker detection phase, we can find a missing point of the occluded marker, which also indi-cates the position of occluded part. The second step is to recalculate the new templates in four directions. Figure 4.10 shows the four templates for the former example. After calculating the new templates, we can use NCC method to match the partial inner pattern and the new templates.
The partial pattern matching method can match markers with occluded inner pattern, but it can't match correctly when more than quarter of inner pattern is occluded. These markers with large occluded area can't provide enough features and can be confused with the background objects while detection. It is not a good idea to regard the background objects as markers while we try to detect these broken markers. Thus, we regard markers with more than a quarter of their inner patterns are occluded as background objects, not the markers we intend to detect.
Hiro
H iro Hiro
H ir o
Figure 4.10: The four recalculated templates for the Figure 4.9 example
4.5 Summary
In this chapter, we detailed the structure of our AR system, STAR, and the new designed methods to improve the problems for other systems. The two thresholding methods in the la-beling phase are to solve the unbalanced light detection problems. Besides, the new quadrangle detection method and boundary reconstruction method make our system rebuild the boundaries of partial occluded markers. If the outer boundary reconstruction failed, we can also use the inner boundary to rebuild the marker. Finally, the partial pattern matching method increases the stability of the partial occluded marker detection results. We will demonstrate the STAR's ability by the experiments in the next chapter.
Chapter 5
Experiment Design
The section demonstrates the experiment results, including lighting immunity, partial oc-clusion immunity, and computational performance, and discusses the performance of each AR system.
5.1 Experimental Environments
The camera we used for grabbing images is Logitech Webcam Pro 9000. This camera can capture 2-megapixel high definition video and grab up to 30 frames per second (FPS) video.
It also equips autofocus and auto exposure compensation functions which can also be adjust manually. With these two functions, we can emulate the experiment environments we need.
The computer platform we use is an ASUS AS-D760 with 2.93GHz Intel Core2 Duo E7500 Dual-core processer and 2GB RAM with Windows XP SP3. This is the general equipment of the current Dual-core computers.
We conduct every experiment by the following steps:
1. Attach the 5×5 (cm2) sized marker on the vertical white wall.
2. Fix the camera at the distance of 40cm away from the marker.
3. Capture the VGA sized (640×480 pixels) video with frame rate at 30FPS.
4. Use the test program of different systems to detect the markers.
nity, and (3) computational performance separately. Before demonstrating these experiments in the following sections, one thing is worth mentioned. While ARTag only provides a demo execution file, we can't modify the source code to calculate the information we need, such as the detection rate or frame rate. In the following experiments, we use the "
⃝" sign to show ARTag
can well detect markers while the "×" sign means the system can barely detect the markers in
the experiment. And in the speed performance experiment, since we can't measure the frame rate manually, the frame rate of ARTag is signed as "not available".5.2 Immunity to Light Condition
Light condition is always an important issue for marker detection. To find the immunity of different light conditions for every planar marker AR system, we design two experiments finding the detection rates of single marker and multiple markers detection. In addition, we will test STAR by using both dynamic global thresholding method and adaptive thresholding method separately to show the performance of these two methods for different types of detection.
Light condition is always an important issue for marker detection. To find the immunity of different light conditions for every planar marker AR system, we design two experiments finding the detection rates of single marker and multiple markers detection. In addition, we will test STAR by using both dynamic global thresholding method and adaptive thresholding method separately to show the performance of these two methods for different types of detection.