In this chapter, we demonstrate the effectiveness of our perspective correction and structure enhanced inpainting algorithm on architectural images. We also compare the results of our method and original stereoscopic inpainting [WAN08].
The iterative inpainting and consistency-checking framework [WAN08] can reduce many of the inappropriate filled pixels. More iterations will recover a more coincident result pair. Practically, the progress converges to a visually consistent result after four or five iterations.
Without perspective correction, perspective artifacts may still remain after many iterations and are sensitive to human eyes. Inpainting on a perspective corrected space can recover a smoother result on the attachment where structures meet. The structure-enhanced patch searching algorithm can further maintain the architectural structure on under or over textured area.
31
7.1 Results
Figure 12: Input stereo images and user defined obstacle pixels (red pixels).
32
Figure 13: Four iterations of original stereoscopic inpainting.
Top to bottom: iteration 1 to 4.
33
Figure 14: Fourth iteration of one view in Figure 13.
34
Figure 15: Four iterations of our method without structure enhancement.
Top to bottom: iteration 1 to 4.
35
Figure 16: Fourth iteration of one view in Figure 15.
36
Figure 17: Four iterations of our method with structure enhancement.
Top to bottom: iteration 1 to 4.
37
Figure 18: Fourth iteration of one view in Figure 17.
Figure 19: Comparison of three approaches.
Left to right: Original stereoscopic inpainting, our method without structure enhancement, our method with structure enhancement.
38
Figure 20: Input stereo images and user defined obstacle pixels (red pixels).
39
Figure 21: Four iterations of original stereoscopic inpainting.
Top to bottom: iteration 1 to 4.
40
Figure 22: Fourth iteration of one view in Figure 21.
41
Figure 23: Four iterations of our method without structure enhancement.
Top to bottom: iteration 1 to 4.
42
Figure 24: Fourth iteration of one view in Figure 23.
43
Figure 25: Four iterations of our method with structure enhancement.
Top to bottom: iteration 1 to 4.
44
Figure 26: Fourth iteration of one view in Figure 25.
Figure 27: Comparison of three approaches.
Left to right: Original stereoscopic inpainting, our method without structure enhancement, our method with structure enhancement.
45
Figure 28: Input stereo images and user defined obstacle pixels (red pixels).
46
Figure 29: Four iterations of original stereoscopic inpainting.
Top to bottom: iteration 1 to 4.
47
Figure 30: Fourth iteration of one view in Figure 29.
48
Figure 31: Four iterations of our method without structure enhancement.
Top to bottom: iteration 1 to 4.
49
Figure 32: Fourth iteration of one view in Figure 31.
50
Figure 33: Four iterations of our method with structure enhancement.
Top to bottom: iteration 1 to 4.
51
Figure 34: Fourth iteration of one view in Figure 33.
Figure 35: Comparison of three approaches .
Left to right: Original stereoscopic inpainting, our method without structure enhancement, our method with structure enhancement.
52
Figure 36: Input stereo images and user defined obstacle pixels (red pixels).
53
Figure 37: Four iterations of original stereoscopic inpainting.
Top to bottom: iteration 1 to 4.
54
Figure 38: Fourth iteration of one view in Figure 37.
55
Figure 39: Four iterations of our method without structure enhancement.
Top to bottom: iteration 1 to 4.
56
Figure 40: Fourth iteration of one view in Figure 39
57
Figure 41: Four iterations of our method with structure enhancement.
Top to bottom: iteration 1 to 4.
58
Figure 42: Fourth iteration of one view in Figure 41.
Figure 43: Comparison of three approaches.
Left to right: Original stereoscopic inpainting, our method without structure enhancement, our method with structure enhancement.
59
7.2 Discussion
We showed results of different input stereo images. The backgrounds of these images are structural artificiality. Different types of obstacle are marked by user and removed from the images. The image completion results of the original stereoscopic inpainting are showed in Figure 13, 21, 29, 37. We can see the perspective artifacts on the connection of structure line still remain after four iterations of inpainting. Using our method without structure enhancement leads to more reasonable results (Figure 15, 23, 31, 39). The perspective artifacts are effectively reduced but some ghost effects showed up in Figure 16 and Figure 32. Using structure enhanced patch searching in our method, the ghost effects are eliminated (Figure 18 and Figure 34) and the structure of the background is better preserved (Figure 19, 27, 35, 43).
60
Chapter 8 Conclusion
We proposed an automatic image completion method for architectural scene images.
Our system takes stereo images and disparity maps as input. Foreground obstacles are defined by users. The stereoscopic inpainting scheme is used for reducing removed pixels from two-view information and detecting unreliable filling pixels. The unreliable pixels are re-inpainted to preserve parallax consistency of stereo images. Since human eyes are sensitive to the structure of artificiality, we improved the inpainting algorithm using vanishing point and vanishing line prediction to project the image to perspective corrected space. Exemplar-based inpainting is performed on the perspective corrected space, and the perspective artifacts are effectively alleviated. We also applied a structure-enhanced patch searching method to exemplar-based inpainting to better preserve the structure of buildings.
The results of our method are reasonable and natural.
61
References
[ADV03] A. Almansa,A. Desolneux, S. Vamech. “Vanishing point detection without any a priori information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.
25, Issue 4, pp. 502-507, 2003.
[ALC05] D. G. Aguilera, J. G. Lahoz, J. F. Codes, “A new method for vanishing points detection in 3d reconstruction from a single view,” Proceedings of the ISPRS Commission, 2005.
[BAR83] S. T. Barnard, “Interpreting perspective images,” Artificial Intelligence 21, pp.435-462, 1983.
[BAR02] W. A. Barrett and A. S. Cheney, “Object-based image editing,” ACM Transactions on Graphics, Vol. 21, Issue 3, pp. 777-784, 2002.
[BBS01] M. Bertalmio, A. L. Bertozzi, G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” Computer Vision and Pattern Recognition, Vol. 1, pp. 355-362, 2001.
[BCV01] C. Ballester, V. Caselles, J. Verdera, M. Bertalmi, G. Sapiro, “A Variational Model for Filling-In Gray Level and Color Images,” International Conference on Computer Vision, Vol. 1, pp.10, 2001.
62
[BHA07] P. Bhat, C. L. Zitnick, N. Snavely, A. Agarwala, M. Agrawala, M. Cohen, B.
Curless, S. Bing, “Using Photographs to Enhance Videos of a Static Scene,” Symposium A Quarterly Journal In Modern Foreign Literatures, pp. 327–338, 2007.
[BSC00] M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, “Image inpainting,” Analysis and Machine Intelligence, Vol. 8, pp. 679–698, 1986.
[CHA01] T. F. Chan and J. Shen, “Nontexture Inpainting by Curvature-Driven Diffusions,”
Journal of Visual Communication and Image Representation, Vol. 12, Issue 4, pp. 436-449, 2001.
[CHA02] T. F. Chan, S. Soatto, “Inpainting from Multiple Views,” Proceedings First International Symposium on 3D Data Processing Visualization and Transmission, Vol. 1, pp.
622-625, 2002.
[COM02] D. Comaniciu, P. Meer, “Mean shift: A robust approach toward feature space analysis.” Ieee Trans. On Pattern Analysis and Machine Imtelligence, Vol. 24, Issue 5, pp.
603-619, 2002
63
[CPT03] A. Criminisi, P. Pérez, K. Toyama, “Object removal by exemplar-based inpainting,”
IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, pp. II-721-II-728, 2003.
[CY99] J. M. Coughlan and A. L. Yuille, “Manhattan world : Compass direction from a single image by Bayesian inference,” In International Conference on Computer Vision, 1999
[CY03] J. M. Coughlan and A. L. Yuille, “Manhattan world :Orientation and outlier detection by Bayesian inference,” Neural Computation, Vol. 15, no. 5, pp. 1063-1088, 2003.
[DCY03] I. Drori, D. Cohen-Or, H. Yeshurun, “Fragment-based image completion,” ACM Transactions on Graphics, Vol. 22, Issue 3, pp. 303, 2003.
[DOH72] Duda, R. O. and P. E. Hart, “Use of the Hough Transformation to Detect Lines and Curves in Pictures,” Comm. ACM, Vol. 15, pp. 11–15, January, 1972
[DVL06] P. Darko, S. Volker and K. Leif, “Interactive image completion with perspective correction.” The Visual Computer, Vol. 22, Issue 9, pp. 671-681, 2006
[FIS81] M. A. Fischler, R. C. Bolles. “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.” Communications of the ACM, Vol. 24, Issue 6, pp. 381-395, 1981
[GON03] M. Gong, Y. Yang, “Fast stereo matching using reliability-based dynamic programming and consistency constraints.” Int. Conf. on Computer Vision, pp. 610-617, 2003
64
[HAY07] J. Hays and A. A. Efros, “Scene completion using millions of photographs,” ACM Transactions on Graphics, Vol. 26, Issue 3, pp. 4, 2007.
[MAS98] S. Masnou and J. M. Morel, “Level lines based disocclusion,” Int. Conf. Image Processing, Vol. III, pp. 259-263, 1998.
[MMB97] W. R. Mark, L. McMillan, G. Bishop, “Post-Rendering 3D Warping.” Symp. On Interactive 3D Graphics, pp. 7-16, 1997
[PSK06] D. Pavić, V. Schönefeld, L. Kobbelt. “Interactive image completion with perspective correction,” The Visual Computer, Vol. 22, Issue 9, pp. 671-681, 2006.
[SYJ05] J. Sun, L. Yuan, J. Jia, H. Y. Shum. “Image completion with structure propagation,”
ACM Transactions on Graphics, Vol. 24, Issue 3, pp. 861, 2005.
[WAN08] L. Wang, J. Hailin, Y. Ruigang, G. Minglun “Stereoscopic inpainting: Joint color and depth completion from stereo images,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[WBT05] M. Wilczkowiak, G. J. Brostow, B. Tordoff, R. Cipolla, “Hole Filling Through Photomontage,” Proc British Machine Vision, pp. 492–501, 2005.