CHATPER 5 AUGMENTED STEREO PANORAMAS
5.4. E XPERIMENTAL R ESULTS OF A UGMENTED S TEREO P ANORAMAS
Fig. 47 shows the stitched results of a stereo panorama from photos taken in our laboratory.
Fig. 48 shows the result of integrating a stereo OM into the stereo panorama. The shadow is properly rendered under the inserted object and the perceived depth of the OM is consistent with its nearby scene objects. Fig. 49 shows the consecutive views of rotating the stereo OM in the stereo panorama.
67 (a)
(b)
Fig. 47: Stitching result of a stereo panorama.
(a) (b)
Fig. 48: Result of the augmented panorama with a stereo OM. (a) shows the rendered left view, and (b) shows the right view.
(a) (b)
(c) (d)
Fig. 49 Rotating the OM in the augmented stereo panorama. (a) and (c) are the left views. (b) and (d) are the right views.
68
Chatper 6
Conclusion and Future Work
This work proposes methods of acquiring high quality OMs including object rig calibration, and background removal. Furthermore, to allow additional applications, this work also develops a 3D reconstruction method to obtain the 3D information of the object, and a new method called augmented stereo panoramas to construct interactive 3D virtual worlds.
To calibrate a motorized object rig, this work first applies the CPC kinematic model to formulate the 3D configuration of the device, and then proposes a method of estimating the parameters of the CPC model of the device. Furthermore, a visual tool is provided to guide users to adjust the controllable axes of the rig according to the estimated results. The proposed method has two major advantages. First, only a small number of images of the calibration object is required. Second, the camera parameters of any views can be obtained with the estimated parameters after calibration.
Since fully automatic segmentation method remains an open problem, this work develops interactive segmentation methods for minimizing the user intervention. First, the initial segmentation results are automatically obtained based on observed characteristics of OM. If some segmentation results do not satisfy user expectations, then the user can modify misclassified pixels in only a few images, and propagate the corrected result to all frames through spatial and temporal coherence. In contrast with other segmentation methods, the proposed method incorporates the shape prior to the image segmentation process. The shape prior introduced into each image of the OM is extracted from the 3D model reconstructed using the volumetric graph cuts algorithm. Experimental results demonstrate the shape
69
information is indeed critical to eliminating segmentation errors, and ensures that the segmentation method is robust to background noises. Moreover, the proposed OM segmentation process requires only a small amount of user intervention, namely selecting a subset of acceptable segmentations of the OM following the initial segmentation process.
Some graph-cut-based methods have recently been proposed to reconstruct 3D models from multi-view images, and can yield acceptable results. However, such methods have two problems namely that concavity-convex features and silhouettes are not preserved. This work proposes a two-phase approach is proposed to solve these problems. In the first phase, a modified volumetric graph cuts algorithm is applied to obtain a silhouette-preserved 3D surface.
This algorithm starts from volumetric graph cuts algorithm and enhances the reconstruction result by iteratively adjusting the optimization cost based on the output of volumetric graph cuts.
These iterative steps are performed until the silhouettes of the obtained 3D surface match the input silhouette images. In the second phase, the 3D surface is refined using gradient descent optimization. The positions of the vertices of the 3D model are adjusted along the normal directions to ensure that he surface has high photo-consistency.
Panoramas and OMs are conventionally adopted image-based techniques for modeling and rendering 3D scenes and objects. This work presents a method that allows users to generate an augmented stereo panorama by interactively integrating stereo OMs into a stereo panorama. A user can directly browse the stereo object movies of interest by navigating in the augmented stereo panorama with a stereoscopic display. The augmented stereo panoramas enhance the user’s interactive experience by elevating better depth perception.
A major limitation of the proposed method for 3D reconstruction is that it cannot effectively handle specular objects, because the zero-mean normalized cross correlation (ZNCC), which is adopted to measure the photo-consistency score, is not robust to specular reflection. Future work will be to apply to the OM some diffuse-specular separation techniques before 3D reconstruction. Relighting also can be performed with the reconstructed 3D models
70
after separating the reflection components. Another plan is to further reduce the user intervention by analyzing the energy of the minimum cut after the initial segmentation, and then automatically identifying a subset of acceptable segmented images.
71
References
[1] Adobe Inc. [Online]. Available: http://www.adobe.com/products/photoshop/
[2] M. Agrawal and L. S. Davis, “Complete Camera Calibration Using Spheres : A Dual-Space Approach,” International Conference on Computer Vision, vol. 2, 2003.
[3] K. S. Arun, T. S. Huang and S. D. Blostein, “Least-Squares Fitting of Two 3-D Point Sets,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 9, no. 5, pp.
698-700, 1987.
[4] T. Beier and S. Neely, “Feature-Based Image Metamorphosis,” Computer Graphics, vol. 26, no. 2, pp. 35-42, 1992.
[5] J.-Y. Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker Descrip-tion of the Algorithm,” Intel Corporation, Microprocessor Research Labs, OpenCV Documents, 1999.
[6] Y. Boykov and M.-P. Jolly, “Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images,” International Conference on Computer Vision, 2001, pp. 105–112.
[7] C. Chang, G. Bishop, and A. Lastra, “LDI tree: A hierarchical representation for image-based rendering,” Computer Graphics, pp. 291–298, August 1999.
[8] C.-S. Chen, C. K. Yu, Y.-P. Hung, “New Calibration-free Approach for Augmented Real-ity Based on Parameterized Cuboid Structure,” International Conference on Computer Vision, (1999) 30-37
[9] C.-S. Chen and W.-Y. Chang, “On Pose Recovery for Generalized Visual Sensors,”
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 7, pp.
848-861, July 2004.
[10] C.-W. Chen, L.-W. Chan, Y.-P. Tsai, and Y.-P. Hung, “Augmented stereo panoramas,”
Asian Conference on Computer Vision, vol. 1, pp. 41–49, 2006.
[11] S. Chen and L.Williams, “View interpolation for image synthesis,” Computer Graphics, pp. 279–288, August 1993.
[12] S. E. Chen, “QuickTimeVR – an image-based approach to virtual environment navigation,” Computer Graphics, pp. 29-38, August 1995.
[13] Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski, “A bayesian approach to digital matting,” Conference on Computer Vision and Pattern Recognition, pp. 264-271, December 2001.
72
[14] O. Cuisenaire, “Distance transformations: Fast algorithms and applications to medical image processing,” Ph.D. dissertation, Universit´e Catholique de Louvain, Belgium, 1999.
[15] X. Déecoret, F. Durand, F. X. Sillion, and J. Dorsey, “Billboard clouds for extreme model simplification,” Computer Graphics, vol. 22, no. 3, pp. 689-696, 2003.
[16] J. Denavit and R. S. Hartenberg, “A Kinematic notation for lower-pair mechanisms based on matrices,” Journal of Applied Mechanics , vol. 22, no. 2, pp. 215-221, June 1955.
[17] D. Freedman and T. Zhang, “Interactive graph cut based segmentation with shape priors,” Conference on Computer Vision and Pattern Recognition, pp. 755–762, 2005.
[18] D. B. Goldman, B. Curless, A.n Hertzmann, and S. M. Seitz, “Shape and Spatially-Varying BRDFs from Photometric Stereo,” International Conference on Computer Vision, 2005.
[19] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” Computer Graphics, pp. 43-54, 1996.
[20] C. Gu and Lee, M. C., “Semi-automatic segmentation and tracking of semantic video objects,” IEEE Transaction on Circuits and Systems for Video Technology, vol. 8, no. 5, pp.
572-584, 1998.
[21] R. I. Hartley, E. Hayman, L. de Agapito and I. D. Reid. “Camera calibration and the search for infinity,” International Conference on Computer Vision, pp. 510-517, September 1999.
[22] A. Hornung, and L. Kobbelt, “Hierarchical Volumetric Multi-view Stereo Reconstruction of Manifold Surfaces based on Dual Graph Embedding,” Conference on Computer Vision and Pattern Recognition, 2006.
[23] H. C. Huang, and Y. P.: Hung, “Panoramic Stereo Imaging System with Automatic Disparity Warping and Seaming,” Graphical Models and Image Processing, vol. 60, no. 3, pp.196-208, 1998.
[24] C. R. Huang, C. S. Chen and P. C. Chung, “Tangible Photo-Realistic Virtual Museum,”
IEEE Computer Graphics and Applications, vol. 25, no.1, pp. 15-17, 2005.
[25] Y. P. Hung, C. S. Chen, Y. P. Tsai, and S. W. Lin, “Augmenting Panoramas with Object Movies by Generating Novel Views with Disparity-Based View Morphing,” Journal of Visualization and Computer Animation, Special Issue on Hallucinating the Real World from Real Images, vol. 13, pp. 237-247, 2002
[26] Y. P. Hung and Y. P. Tsai, “Trail-dependent intelligent scissors based on multi-scale image segmentation,” Asian Conference on Computer Vision, pp. 539–544, 2002.
[27] H. Kato and M. Billinghurst, “Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferenciing,” 2nd IEEE and ACM International Workshop on Augmented Reality, IWAR, 1999.
[28] A. Laurentini, “The visual hull concept for silhouette based image understanding.”
IEEE Transactions on Pattern Analysis and Machine Intelligence, 16.2 (1994), 150-162.
73
[29] M. Levoy and P. Hanrahan, “Light Field Rendering,” Computer Graphics, pp.31-42, 1996.
[30] S. Z. Li, Markov Random Field Modeling in Computer Vision, Springer-Verlag, Tokyo, 1995.
[31] Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum, “Lazy snapping,” Computer Graphics, pp.
303–308, 2004.
[32] W.-Y. Lo, Y.-P. Tsai, C.-W. Chen, and Y.-P. Hung, “Stereoscopic Kiosk for Virtual Museum,” International Computer Symposium, 2004.
[33] W. E. Lorensen and H. E.Cline, “Marching Cubes: a high resolution 3D surface construction algorithm,” Computer Graphics, pp.163-169, 1987.
[34] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
[35] D.G. Lowe, “Fitting parameterized three–dimensional models to images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 5, pp. 441-450, 1991.
[36] H. Luo and A. Eleftheriadis, “An interactive authoring system for video object segmentation and annotation,” Signal Processing: Image Communication, vol. 17, p559-572, 2001.
[37] S. P. Mallick, T. E. Zickler, D. J. Kriegman, and P. N. Belhumeur, “Beyond lambert:
reconstructing specular surfaces using color,” Conference on Computer Vision and Pattern Recognition, 2005.
[38] B. Marcotegui, P. Correia, F. Marques, R. Mech, R. Rosa, M. Wollborn, F. Zanoguera
"A Video Object Generator Tool Allowing Friendly User Interaction" International Conference of Image Processing ,1999.
[39] S. J. Maybank and O. D. Faugeras, “A Theory of Self-Calibration of a Moving Camera,” International Journal of Computer Vision, International Journal of Computer Vision, vol. 8. no. 2, pp. 123-152, Aug. 1992.
[40] E. N. Mortensen and W. A. Barrett, “Intelligent scissors for image composition,”
Computer Graphics, pp.191–198, 1995.
[41] E. N. Mortensen and W. A. Barrett, “Toboggan-based intelligent scissors with a four-parameter edge model,” Conference on Computer Vision and Pattern Recognition, pp.
2452–2458, 1999.
[42] S. Nayar, S. Nene, and H. Murase, “Subspace methods for robot vision,” IEEE Transactions on Robotics and Automation, vol. 12, no. 5, pp. 750-758, 1996.
[43] M.T. Orchard, C.A. Bouman, “Color Quantization of Image,” IEEE Transactions on Signal Processing, vol. 39, no. 12, pp. 2677-2690, 1991.
[44] I. Patras, E. Hendriks, and I. Lagendijk, “Video Segmentation by MAP Labeling of Watershed Segments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.
74
23, no. 3, pp. 326-332, 2001.
[45] S. Peleg, and M. Ben-Ezra, “Stereo Panorama with a Single Camera,” Computer Vision and Pattern Recognition, pp.395-401, 1999
[46] K. S. Roberts, “A New Representation for a Line,” Computer Vision and Pattern Recognition, pp. 635-640, June 1988.
[47] C. Rother, V. Kolmogorov, and A. Blake, “GrabCut: interactive foreground extraction using iterated graph cuts,” Computer Graphics, pp. 309–314, 2004.
[48] D. Ruprecht and H. Muller, "Image Warping with Scattered Data Interpolation", IEEE Computer Graphics and Applications, vol. 3, pp. 37-43. 1995.
[49] S. M. Seitz and C. M. Dyer, “Photorealistic scene reconstruction by voxel coloring,”
Computer Vision and Pattern Recognition, 1997.
[50] S. M. Seitz and C. M. Dyer, “View morphing,” Computer Graphics, pp. 21–30, August 1996.
[51] J., Shade S. Gortler, L.-W. He, and R. Szeliski, “Layered depth images,” Computer Graphics, pp. 231–242, July 1998.
[52] J. Shi and C. Tomasi, “Good features to track,” Computer Vision and Pattern Recognition, 1994, pp.593–600.
[53] S. W. Shih, Kinematic and Camera Calibration of Reconfigurable Binocular Vision Systems, Ph.D. thesis, National Taiwan University, 1995.
[54] H. Y. Shum, and L. W. He, “Rendering with Concentric Mosaics,” Computer Graphics, (1999) 299-306.
[55] H. Y. Shum and R. Szeliski, “Stereo Reconstruction from Multiperspective Panoramas,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, pp.
45-62, Jan 2004.
[56] H.-Y. Shum, S. B. Kang, S.-C. Chan, “Survey of Image-Based Representations and Compression Techniques,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 11, pp. 1020-1037, 2003.
[57] S. Tran, and L. Davis, “3D surface reconstruction using graph cuts with surface constraints,” European Conference on Computer Vision, 2006
[58] R. Y. Tsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,” IEEE Journal of Robotics and Automation, vol. 3, no. 4, pp. 323-344, Aug. 1987.
[59] G. Turk and M. Levoy, “Zippered polygon meshes from range images,” Computer Graphics, pp. 311-318, 1994
[60] L. Vincent and P. Soille, “Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 6, pp. 583-598, 1991.
75
[61] G. Vogiatzis, C. Hernández, and R. Cipolla, “Reconstruction in the round using photometric normals,” CVPR 2006.
[62] G. Vogiatzis, P. H. S. Torr, and R. Cipolla, “Multi-view stereo via volumetric graph-cuts,” Computer Vision and Pattern Recognition, 2005
[63] D. Wang, “Unsupervised Video Segmentation Based on Watersheds and Temporal Tracking,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 8, no. 5, pp.
539-546, 1998.
[64] J. Wang and M. F. Cohen, “An iterative optimization approach for unified image segmentation and matting.” International Conference on Computer Vision, pp. 936-943, 2005.
[65] D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. H. Salesin, and W.
Stuetzle, “Surface light fields for 3D photography,” Computer Graphics, pp.287–296, 2000.
[66] Z. Zhang, “A Flexible New Technique for Camera Calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, Nov 2000.
[67] H. Zhong, L. Wenyin, S. Li, “Interactive Tracker – A Semiautomatic System for Video Object Segmentation,” International Conference on Multimedia and Expo, pp. 645-648, 2001.
[68] Z. Zhu, E. M. Riseman, and A. R. Hanson, “Parallel-Perspective Stereo Mosaics,”
International Conference on Computer Vision, pp. 345-352, 2001.
[69] H. Zhuang, Z. S. Roth and F. Hamano, “A Complete and Parametrically Continuous Kinematic Model for Robot Manipulators,” IEEE Transactions on Robotics and Automation, vol. 8, no. 4, pp.451-463, Aug 1992.
76
Appendix
Singularity-Free Line Representation
Let a line, called B-Line, be in 3D space, and a plane, called B-plane, be perpendicular to the line and passes through the origin of reference coordinate system {O, xr, yr, zr}, as shown in Figure A. Let br=
(
bx,by,bz)
be the unit vector along B-Line and lie in the upper half-space of {O, xr , yr, zr} coordinate system, where bx and by are the x and y components of the br
vector defined on the {O, xr, yr, zr} coordinate system. Note that bz, the z component of the direction unit vector br
can be obtained by equation A.1, if bx
and by are known,
(
1 x2 y2)
1/2z b b
b = − − (A.1)
Therefore the bx and by is enough to represent the orientation of B-Line.
Fig. A. Representation of a line B in 3D space.
Next, we define another coordinate system {O, xr', yr', zr'}, where xr' and yr'are 2-D Cartesian coordinate system defined on the B-plane, passing through the same origin O of {O, xr, yr, zr} coordinate system. Let the unit vector kr
denote the common normal of zr
and br
. Thus, we have
xr
zr yr
' xr
' yr ' zr
br B-Line
B-Plane
lx ly
P α
o
bx
by
77 obtained by rotating an angle αabout the axis kr
, where
Let R be the rotation matrix Rot(kr
,α), and it can be calculated by equation (A.4).