Chapter 4 Selectable Thresholding Augmented Reality System 14
4.4 Pattern Matching Phase
4.4.1 Partial Pattern Matching
The original pattern matching method is not robust for partial occluded markers. To increase the robustness, we use the partial pattern matching method, which is very similar to the original pattern matching method. We divide the inner pattern into four parts, and only match the parts
Hiro
Figure 4.9: An example of an occluded marker. The orange area shows the parts for pattern matching.
which are not occluded. We show an example of occluded marker and the matching parts in Figure 4.9.
The first step in partial pattern matching is to find the position of occluded parts. In the marker detection phase, we can find a missing point of the occluded marker, which also indi-cates the position of occluded part. The second step is to recalculate the new templates in four directions. Figure 4.10 shows the four templates for the former example. After calculating the new templates, we can use NCC method to match the partial inner pattern and the new templates.
The partial pattern matching method can match markers with occluded inner pattern, but it can't match correctly when more than quarter of inner pattern is occluded. These markers with large occluded area can't provide enough features and can be confused with the background objects while detection. It is not a good idea to regard the background objects as markers while we try to detect these broken markers. Thus, we regard markers with more than a quarter of their inner patterns are occluded as background objects, not the markers we intend to detect.
Hiro
H iro Hiro
H ir o
Figure 4.10: The four recalculated templates for the Figure 4.9 example
4.5 Summary
In this chapter, we detailed the structure of our AR system, STAR, and the new designed methods to improve the problems for other systems. The two thresholding methods in the la-beling phase are to solve the unbalanced light detection problems. Besides, the new quadrangle detection method and boundary reconstruction method make our system rebuild the boundaries of partial occluded markers. If the outer boundary reconstruction failed, we can also use the inner boundary to rebuild the marker. Finally, the partial pattern matching method increases the stability of the partial occluded marker detection results. We will demonstrate the STAR's ability by the experiments in the next chapter.
Chapter 5
Experiment Design
The section demonstrates the experiment results, including lighting immunity, partial oc-clusion immunity, and computational performance, and discusses the performance of each AR system.
5.1 Experimental Environments
The camera we used for grabbing images is Logitech Webcam Pro 9000. This camera can capture 2-megapixel high definition video and grab up to 30 frames per second (FPS) video.
It also equips autofocus and auto exposure compensation functions which can also be adjust manually. With these two functions, we can emulate the experiment environments we need.
The computer platform we use is an ASUS AS-D760 with 2.93GHz Intel Core2 Duo E7500 Dual-core processer and 2GB RAM with Windows XP SP3. This is the general equipment of the current Dual-core computers.
We conduct every experiment by the following steps:
1. Attach the 5×5 (cm2) sized marker on the vertical white wall.
2. Fix the camera at the distance of 40cm away from the marker.
3. Capture the VGA sized (640×480 pixels) video with frame rate at 30FPS.
4. Use the test program of different systems to detect the markers.
nity, and (3) computational performance separately. Before demonstrating these experiments in the following sections, one thing is worth mentioned. While ARTag only provides a demo execution file, we can't modify the source code to calculate the information we need, such as the detection rate or frame rate. In the following experiments, we use the "
⃝" sign to show ARTag
can well detect markers while the "×" sign means the system can barely detect the markers in
the experiment. And in the speed performance experiment, since we can't measure the frame rate manually, the frame rate of ARTag is signed as "not available".5.2 Immunity to Light Condition
Light condition is always an important issue for marker detection. To find the immunity of different light conditions for every planar marker AR system, we design two experiments finding the detection rates of single marker and multiple markers detection. In addition, we will test STAR by using both dynamic global thresholding method and adaptive thresholding method separately to show the performance of these two methods for different types of detection.
5.2.1 Single Marker Detection
In this experiment, we try to find the detection rate of existing AR systems detecting single marker under unbalanced lights. As Figure 5.1 shows, we set a light source and the illuminance of each part is different, confirm the definition of unbalanced lights. Then, we use the illumi-nance meter to find the positions with the illumiillumi-nance at 200, 500, 1000, 1500 and 2000 Lux.
We attach planar marker to these positions and set the camera to grab images. We also disable the auto exposure compensation function to ensure the light condition won't be changed by the camera. During the detection, the detection rate is calculated by the percentage of the correct
Marker
Illuminance
200 500 1000 1500
light
(Lux)
Figure 5.1: The single marker detection under different illuminance
Table 5.1: The detection rate for single marker detection under different illuminance
Illuminance (Lux) ARTK ARTag ARTK+ STAR (Global) STAR (Adaptive)
200 100.00%
⃝
100.00% 100.00% 100.00%500 100.00%
⃝
100.00% 99.90% 100.00%1000 0.00%
⃝
99.70% 99.80% 100.00%1500 0.00%
⃝
20.10% 99.80% 99.76%2000 0.00%
⃝
21.00% 99.80% 2.06%detection times in 1000 measurements, we record the average of 5 detection rates in Table 5.1.
Comparing the results in Table 5.1, we can find ARTag and STAR with dynamic thresh-olding method can give stable detection results under general illumination range while ARTK and ARTK+ can't. Since ARTK uses the fixed global thresholding method, it is hard to detect markers under bright conditions. ARTK+ tries to improve this drawback by using the random dynamic thresholding method, but still can't detect stably for markers in the bright environ-ments. It's because the random method needs to guess many times to find the right threshold in bright conditions, and it has to guess again while the chosen threshold doesn't provide stable detection results. Differ from AETK+, our dynamic global thresholding method detects well under different light conditions and only needs one or two frames to find the correct and stable
light
Figure 5.2: The multiple markers detection under different illuminance
this method can't detect markers for illuminance exceeds 1500 Lux. It's because the adaptive thresholding method regards the color of the points which intensity is 5% lower than the region average as black. Nevertheless, in excessive bright conditions, the real black and white color are very close to each other in the grabbed images, makes it hard to classify these two colors by using adaptive thresholding method.
5.2.2 Multiple Markers Detection
The multiple markers detection under unbalanced light conditions is different to the single marker detection. The illumination for every marker is different to each other in the multiple markers detection, which increases the difficulty to detect every marker at the same time. In this experiment, we test each AR system using planar markers to detect six markers as illustrated in Figure 5.2. This diagram also shows the illuminations for each marker. We calculate the average detection rate for the six markers in 5000 times measurements, and note the results in Table 5.2.
From the data in Table fig:fig5.2, we know ARTag and STAR with adaptive thresholding method can find every marker in the same time while other systems or methods only provide
Table 5.2: The detection rate for multiple markers detection under different illuminance
ARTK ARTag ARTK+ STAR (Global) STAR (Adaptive)
52.28%
⃝
66.56% 76.64% 99.92%unstable results. Although the detection rate of our dynamic thresholding method is not as good as the adaptive thresholding method, it still performs better than original ARTK and ARTK+.
In these two experiments, we simulate the unbalanced light conditions and detect single marker and multiple markers, separately. In single marker detection part, the changing of the illuminance in the marker's region is not as dramatically as the changing in the whole frame, making the system can use the global thresholding method to detect marker. However, when the illuminance changes too much in marker's region, the global thresholding method cannot detect this marker correctly. This is an unusual situation and seldom happens in the daily life.
Hence, we think global thresholding method suitable for single marker detection under unbal-anced lights. In the multiple markers experiment, the adaptive thresholding method is the most appropriate way to detect these markers under unbalanced lights.
5.3 Immunity to Partial Occlusion
This experiment is to find the maxima occluded area while the AR systems can still provide stable results. For some of the marker detection methods, such as Hirzer's method [13], can only detect markers occluded by light color objects. Thus, in the experiment, we will test the systems to detect markers occluded by different sized white triangles and black triangles. Figure 5.3 shows the makers occluded by different size of black triangles. The triangle with 2% of marker area only occludes the white periphery. And triangles with 5% and 10% marker area occlude the outer boundary of the black periphery while the 25% sized triangle can occlude the inner
2
Figure 5.3: Markers occluded by different sized black triangles
Table 5.3: The detection rate for marker occluded by different sized white triangles
Occluded area ARTK ARTag ARTK+ STAR
0% 100.00%
⃝
99.92% 100.00%boundary. The inner pattern is occluded by using triangles with size larger than 32% of the marker area. The detection rates of 5000 measurements for marker occluded by white and black objects are note in Table 5.3 and Table 5.4.
From the results in Table 5.3 and Table 5.4, most systems can't detect stably when the marker's inner pattern is occluded except STAR. ARTK and ARTK+ can't detect black periphery
Table 5.4: The detection rate for marker occluded by different sized black triangles
Occluded area ARTK ARTag ARTK+ STAR
0% 100.00%
⃝
99.92% 100.00%(a) (b)
Figure 5.4: Two examples of partial occluded marker detection by using STAR. (a) Use white object occlude marker's inner boundary, and (b) Use black object occlude marker's inner pattern.
occluded markers since the boundaries are not quadrangle shaped anymore, makes the makers regarded as background objects in the quadrangle detection step. Another system, ARTag, can detect markers while the outer boundaries are occluded, but still can't cope with inner pattern occluded markers. It is because the ID marker used in ARTag can't provide correct digital data to calculate marker ID while the inner pattern is occluded. And for our system, STAR, can detect markers with less than quarter of inner pattern area is occluded. With the help of boundary re-construction method and partial pattern matching techniques, STAR can give more stable results than other systems in this experiment. We show the two examples of partial occluded marker detection by using STAR in Figure 5.4.
5.4 Computational Performance
In this experiment, we test the computational performance for each AR system to find whether the systems can interact in the real time or not, which is one of the definitions of AR techniques. The computational performance is evaluated by calculating the frame rates for each
Table 5.5: The frame rate for different AR systems
ARTK ARTag ARTK+ STAR (global) STAR (adaptive)
29.74 NA 29.80 29.74 19.71
system. We execute test programs on the Dual-core desktop which merge a green square on the marker after detecting it, decreasing the time to draw complex visual objects. The computational performance of 5000 frames is recorded in Table 5.5 by using FPS.
The computational performances of ARTK, ARTK+ and STAR with dynamic global thresh-olding method are very close to 30 FPS, the fastest frame rate supported by the camera. Cer-tainly, these AR systems can interact in real time. For ARTag, the system we can't measure the frame rate, can actually interact in real time while we using the demo program in the previous experiments. The last system, STAR with adaptive thresholding method, is slower than other AR systems. It's because the method needs to calculate three integral images, makes the com-putational performance not as good as other systems. However, the frame rate of this method is higher than 16 FPS, the lowest frame rate for human persistence of vision, and reaches the real time definition for AR technologies.
5.5 Summary
We demonstrated STAR's performance by conducting three experiments in this chapter. For unbalanced light detection, STAR improves the drawbacks of ARTK and ARTK+ and performs as well as ARTag. On the other hand, STAR can detect inner pattern occluded markers while other systems can't provide stable and correct results. Besides, STAR is suitable for general desktop computers to process in real time. In conclusion, STAR has better immunities for both light and partial occlusion than other AR systems.
Chapter 6
Conclusion and Future Work
In this research, we design an AR system, STAR, solving two major problems for existing AR systems using planar markers (1) unbalanced light detection, and (2) partial occluded marker detection. This system is based on ARTK system, but we design a series of new detection methods to replace the original ones, which can rebuild the missing boundaries while markers are partial occluded. To deal with the detection unbalanced lights, we propose a dynamic global thresholding method and an adaptive thresholding method for a user to select by the requirements and purposes. Besides, we also design three experiments to evaluate (1) lighting immunity, (2) partial occlusion immunity, and (3) computational performance for our system and other existing AR systems. These experiments demonstrate our system can deal with two major problems and performs better than other existing AR systems.
In the next step, we will make effort to improve the computational performance of the adap-tive thresholding method, and find other thresholding methods to detect multiple markers under unbalanced lights efficiently and stably. Simultaneously, we are trying to apply our AR sys-tem on human pose recognizers and human-computer interaction syssys-tems, hoping to get fruitful results in the near future.
References
[1] R. Azuma et al., ''A survey of augmented reality,'' Presence-Teleoperators and Virtual
Environments, vol. 6, no. 4, pp. 355--385, 1997.
[2] C. Kirner, E. Zorzal, and T. Kirner, ''Case Studies on the Development of Games Using Augmented Reality,'' in IEEE International Conference on Systems, Man and Cybernetics,
2006. SMC'06, vol. 2, 2006.
[3] D. C. P.-L. Mariano Alcaniz, Manuel Contero and M. Ortega, ''Augmented reality technol-ogy for education,'' 2010.
[4] S. White, D. Feng, and S. Feiner, ''Interaction and presentation techniques for shake menus in tangible augmented reality,'' in Proceedings of 2009 IEEE International Symposium on
Mixed and Augmented Reality, 2009, pp. 39--48.
[5] F. Zhou, H. Duh, and M. Billinghurst, ''Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR,'' in Proceedings of the 2008 7th IEEE/ACM
International Symposium on Mixed and Augmented Reality-Volume 00.
IEEE Computer Society, 2008, pp. 193--202.[6] H. Kato, K. Tachibana, M. Billinghurst, and M. Grafe, ''A registration method based on texture tracking using artoolkit,'' in The Second IEEE International Augmented Reality
Toolkit Workshop, Nishi-Waseda Campus, Waseda University, Tokyo, Japan, 7th October,
2003.[7] H. Kato and M. Billinghurst, ''Marker tracking and hmd calibration for a video-based
aug-mented reality conferencing system,'' in iwar. Published by the IEEE Computer Society, 1999, p. 85.
[8] ''Artoolkit,'' http://www.hitl.washington.edu/artoolkit/.
[9] D. Wagner and D. Schmalstieg, ''Artoolkitplus for pose tracking on mobile devices,'' in
Computer Vision Winter Workshop.
Citeseer, 2007, pp. 6--8.[10] ''Artoolkitplus,'' http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php.
[11] T. Pintaric, ''An adaptive thresholding algorithm for the augmented reality toolkit,'' in Proc.
of the Second IEEE Int'l Augmented Reality Toolkit Workshop (ART03).
Citeseer, 2003.[12] M. Maidi, F. Ababsa, and M. Mallem, ''Vision-inertial tracking system for robust fiducials registration in augmented reality,'' in Computational Intelligence for Multimedia Signal
and Vision Processing, 2009. CIMSVP '09. IEEE Symposium on, march 2009, pp. 83 --90.
[13] M. Hirzer, ''Marker detection for augmented reality applications,'' 2008.
[14] M. Fiala, ''Designing Highly Reliable Fiducial Markers,'' IEEE Transactions on Pattern
Analysis and Machine Intelligence, 2009.
[15] ''Artag,'' http://www.artag.net/.
[16] D. Bradley and G. Roth, ''Adaptive thresholding using the integral image,'' Journal of
Graphics, GPU, Game Tools, vol. 12, no. 2, pp. 13--21, 2007.
[17] G. Moore and M. Gaithersburg, ''Automatic scanning and computer processes for the quan-titative analysis of micrographs and equivalent subjects,'' in Pictorial pattern recognition:
proceedings.
Thompson Book Co., 1968, p. 275.[18] R. C. Gonzalez and R. E. Woods, Digital Image Processing (3rd Edition). Upper Saddle River, NJ, USA: Prentice-Hall, Inc., 2006.
[19] A. Rosenfeld and J. L. Pfaltz, ''Sequential operations in digital picture processing,'' J. ACM, vol. 13, no. 4, pp. 471--494, 1966.
[20] J. Prewitt, ''Object enhancement and extraction,'' Picture processing and Psychopictorics, pp. 75--149, 1970.