A novel 3D planar object reconstruction from multiple
uncalibrated images using the plane-induced homographies
H.L. Chou, Z. Chen
*Department of Computer Science and Information Engineering, National Chiao Tung University, Hsinchu, Taiwan Received 4 April 2003; received in revised form 27 December 2003
Available online 1 July 2004
Abstract
A computer vision method is proposed to determine all the visible 3D planar surfaces in a scene from uncalibrated images and locate them in a single 3D projective space. Most of the existing methods for reconstructing planar objects use point correspondences to estimate the fundamental matrix and derive the compatible projection equations, before they apply the standard triangulation technique to find the 3D points and fit the planes to the 3D points. This type of approaches is generally slow and less accurate because the 3D points are estimated separately, making them vulnerable to image error. We present a plane based reconstruction method to estimate the 3D projective structure using the planar homographies estimated from the plane features in the images. First, we estimate the homography for each visible plane, and then we use the homographies of two primary planes to compute an epipole. We proceed to represent the epipolar geometry for each image pair using the estimated homography and epipole, together with a specified reference plane coefficient vector. Next, we show that the 3D plane coefficient vector of any plane visible in each image pair can be determined with respect to the reference plane coefficient vector once its planar homography is found. Finally, the reconstruction results obtained in individual projective spaces are integrated within a common projective space. To this end, we use the homography and plane equation information of two planes and the epipole associated to derive the coordinate transformation matrix between two involved projective spaces. To evaluate the performance of our method, we apply our method to the synthetic images and real images. All the results indicate the method works successfully. 2004 Published by Elsevier B.V.
Keywords: Computer vision; 3D projective reconstruction; Plane-based projective reconstruction; Uncalibrated camera; Homography; Projective geometry; Reconstruction integration
1. Introduction
In the physical world (especially the man-made world) planar surfaces such as walls, windows, table, roof, road, and terrace can be found in the indoor as well as the outdoor scenes. Our task is to reconstruct the 3D planar surfaces in a scene from multiple uncalibrated images taken by a camera
0167-8655/$ - see front matter 2004 Published by Elsevier B.V. doi:10.1016/j.patrec.2004.05.018
*
Corresponding author. Tel.: 573-1875; fax: +886-3-572-3148.
E-mail address:zchen@csie.nctu.edu.tw(Z. Chen).
placed at different viewpoints. In general, the methods for 3D projective or uncalibrated recon-struction (Mohr and Arbogast, 1991; Faugeras, 1992, 1993; Hartley et al., 1992, 1994; Beardsley et al., 1997) are point-based. They estimate the fundamental matrix from a sufficient number of corresponding point pairs first, and then derive the epipole and the canonical geometric representation for projective views using the fundamental matrix. Then, for each pair of corresponding points, they use a triangulation technique or bundle adjust-ment technique to compute the 3D point coordi-nates in the projective space. Finally, for the determination of the uncalibrated planar scene structure (Luong and Faugeras, 1993; Sawhney, 1994; Criminisi and Zisserman, 1998; Irani et al., 1998; Szeliski and Torr, 1998; Fradkin et al., 1999; Johansson, 1999; Zelnik-Manor and Irani, 2000), the 3D points found are fitted by planes. However, it is desirable to derive the 3D planar scene structure in terms of plane features in the images directly, for these features are more reliable than the point or line features (Luong and Faugeras, 1993). The estimation of the 3D projective planar structure based on the projected plane feature information exclusively has not yet received much attention, although it is known that the corre-sponding projected plane regions in a pair of stereo images induce a homography. It is also known that homographies are useful to many other practical applications including:
(a) Fundamental matrix estimation or canonical projective geometry representation (Luong and Vieville, 1996; Luong and Faugeras, 1993).
(b) 2D image mosaicing or view synthesis
(Szeliski, 1996).
(c) Plane + parallax analysis (Irani et al., 1998; Criminisi and Zisserman, 1998; Sawhney, 1994).
(d) Planar motion estimation and ego-motion (Irani et al., 1997; Szeliski and Torr, 1998; Zelnik-Manor and Irani, 2000).
Recently, two methods have been proposed for the 3D projective reconstruction of planes and cameras. The first method assumes all planes are
visible in all images and the second method as-sumes a reference plane is visible in all images (Rother et al., 2002, 2003). In practice, it is not realistic to have all planes or even one plane visible in all images unless a very large ground plane is available. When there is no reference plane visible in all images, the reconstruction problem cannot be formulated within a common projective space and the reconstruction results will be inevitably obtained in different projective spaces.
We shall recover the 3D scene planar structure from the uncalibrated images using the plane-induced homographies without assuming that all planes or one plane must be seen in all images. To obtain the homographies, we must locate the projected regions of planar surfaces in the images. There are methods for detecting regions corre-sponding to planar surfaces in the image (Sinclair and Blake, 1996; Hamid and Cipolla, 1997; Theiler and Chabbi, 1999). After the image regions of planar surfaces have been extracted, we use the Gabor filtering technique (Sun et al., 2002) to identify at least four point correspondences for every plane in the stereo images in order to obtain the initial value of the homography. Then we iteratively refine the homography based on a nonlinear minimization method given in (Szeliski, 1996). Next, we use two homographies to compute the epipole and to find the compatible projection equations in terms of the estimated homography and an assigned plane coefficient vector of a re-ference plane, together with the estimated epipole. With the projection equations thus derived we then prove that the 3D equation of any other plane visible in the stereo images can be computed with respect to the reference plane equation as long as its homography is determined. Finally, we merge or integrate all reconstructed plane equations found in individual projective spaces within a common space through the coordinate (or space) transformations. Again, each required coordinate transformation matrix is expressed by the homo-graphy and plane coefficient vector information of two planes visible in the involved image pairs. Fig. 1 shows the flow diagram of our method.
The remaining sections of the paper are orga-nized as follows. Section 2 is the preliminaries and mathematical notations for the projective
recon-struction. Section 3 shows how the 3D equations of all planar surfaces visible in the stereo images can be determined from their homographies. Sec-tion 4 presents the integraSec-tion of the reconstruc-tion results obtained in different projective spaces through the coordinate transformations. Section 5 shows the estimation of the plane-induced homo-graphies and the related epipole. Section 6 reports the experimental results on both the synthetic and real images. Section 7 is the concluding re-marks.
2. Preliminaries and mathematical notations for projective reconstruction
Consider any two consecutive images (Ii;Ij) in
an image sequence for reconstructing the visible planar surfaces. Let Ri, ~ti be the extrinsic
para-meters and Mi be the 3· 3 upper triangular
intrinsic camera matrix of the ith camera. Then the coordinates of a 3D point ~pE¼ ½ xE yE zE
T
and its 2D projection point ~ui¼ ½ ui vi
T
in image Ii
are related by a pinhole camera model (Hartley et al., 1992; Faugeras, 1993; Hartley and Zisser-man, 2000): ui vi 1 2 4 3 5 ffi Mi½Rij~ti xE yE zE 1 2 6 6 4 3 7 7 5:
To represent the point in the projective space or the homogeneous coordinate system, we use the vectors with a tilde to denote the homogeneous coordinates of the 3D points and its image pro-jection point such that ~ui¼ ~uTi 1
T
and ~pE¼
~pT
E 1
T
. The symbolffi indicates an equality up to a nonzero scale in the homogeneous coordinate system.
Assume the world coordinate system is chosen to be the ith camera coordinate system; namely, Ri¼ I and ~ti¼ ~0.
~
uiffi Mi ½Ij~0 ~pED Mi ~pE: ð1Þ
Similarly, let Mj, Rj, ~tjbe the camera parameters
of the jth camera. For image Ij, we have
~
ujffi Mj ½Rjj~tj ~pE: ð2Þ
Since the epipole on image Ij is given by
~ ejffi Mj Rj~tj 0 0 0 1 ½ T ffi Mj~tj or ke~ej¼
Mj~tj (ke is the lens depth parameter), we rewrite
Eq. (2) as ~
ujffi ½MjRjjke~ej ~pE: ð3Þ
Consider a plane PA, which does not pass
through the optical center of the ith camera (otherwise, its image will be degenerated into a line). Let its plane equation be ~aT
E~pEþ 1 ¼ 0 with
~aT
E¼ ½ aE1 aE2 aE3 T
. After eliminating the vari-able ~pE in the two projection Eqs. (1) and (3), we
can obtain a homography Aij as follows (Tsai and
Huang, 1982; Luong and Vieville, 1996; Szeliski, 1996): ~ ujffi Aij~ui with Aijffi MjRjM1i n ke~ej~aTEM 1 i o :
The homography Aij from image Iito image Ijis
said to be induced by plane PA. In Section 5 we
shall show how to compute the homography Aij
from image pair (Ii, Ij).
For an uncalibrated camera the intrinsic and extrinsic camera parameters in Eqs. (1) and (3) cannot be estimated. We need to replace these two equations by some new parameters that can be estimated. This is done as follows:
Let Aij be rewritten as Aij¼ kA MjRjM1i
ke~ej~aTEM
1 i g, then Homography estimation for each image pair
Epipole computation from homographies
Plane equation computation for planes visible in each image pair
Integration of all plane equations under a common projective space Image sequence
Final 3D planar structure
MjRj¼
1 kA
AijMiþ ke~ej~aTE:
Eq. (3) can be rewritten as
where ~aij(with aij46¼ 0) is assumed to be a nonnull
column vector. Then
ð4Þ
A similar formulation of Eq. (4) has been de-rived in (Luong and Vieville, 1996). In this way, the original Euclidean point ~pE becomes point ~
pij¼ ½~pT
ij pij4T in the new projective space,
de-noted by f~pijg, which describes the projective geometry associated with images i and j.
The coordinate transformation from the
Euclidean spacef½½~pT E 1
T
g to the projective space f½½~pT ij pij4 T g is given by ~pij¼ Mi~pE; pij4¼ kAke aij4 ~aTE~pE þ 1~a T ij aij4 Mi~pE: Also, ~aTij~pijþ aij4pij4¼ ~aijT~pijþ ðkAkeÞ ~aTE~pE þ 1 ~aTij~pij ¼ kAke ~aTE~pE þ 1¼ 0: It implies that ~aT
ij~pijþ aij4pij4¼ 0 is the new 3D
equation of the plane in the projective space ~pij
. Since the projective structure can only be deter-mined up to a 4· 4 nonsingular projective matrix (Hartley and Zisserman, 2000), the new plane coefficient vector ~aij¼ ~aTij aij4
T
can take on some general value, say, ½ 1 1 1 1T (more discussion on the values of ~aij is given in Section
6). In this new space the parameters including homography Aij, epipole ~ej and plane coefficients
~
aij ¼ ~aTij aij4 T
involved in Eq. (4) are now all known.
Next, we shall describe how to obtain the projective reconstruction for the other planes vis-ible in the image pair (Ii, Ij) in the newly defined
projective spacef~pijg.
3. Reconstruction of all visible planes from a given image pair
In the new projective space the projection equations become
ð5Þ
Similarly, for any other plane PB visible in (Ii,
Ij) the induced homography between the plane
regions in image pair (Ii, Ij) is expressed by
Bij ¼ kB MjRjM1i n ke~ej~bTEM 1 i o ð6Þ
with the plane equation of PBbeing ~bTE~pEþ 1 ¼ 0.
Next, we shall prove the fact that the relation between plane coefficient vectors of planes PB and
PA is determined once their homographies Aij and
Bij are found. From above we have
MjRjM1i ¼ 1 kA Aijþ ke~ej~aTEM 1 i ¼ 1 kB Bijþ ke~ej~bTEM 1 i
or Aij¼ kA kB Bijþ kAke~ej ~bTE h ~aT E i M1i DkA kB Bijþ ~ej~gTij; ð7Þ where ~gij¼ kAke½~bE ~aEM1i . We can apply the
least-squares method to estimate the unknowns ~gij
andkA
kBin a system of nine linear equations; here the
epipole ~ejcan be determined in advance from the
two homographies Aij and Bij based on the fact
that BT½~ejAðffi B T
½~ejB) is skew symmetric.
Therefore,
Substituting Aij ¼kkABBijþ ~ej~gTij into the above
equation, we have
Then
Since aij46¼ 0 and bij46¼ 0 (i.e., planes PA and PB
do not contain the lens center), we obtain,
Thus kB kA ~gT ij þkB kA ~aT ij kB kA aij4 ffi ~bTijbij4 h i :
In other words, the relationship between two plane coefficient vectors is given by
~ bij ffi ~aij þ ~gij 0 : ð8Þ
Thus, ~bijcan be determined with respect to ~aijonce
the planar homography Bij is known.
4. Integration of planes reconstructed from different image pairs
Next, we consider the integration of recon-structed planes obtained from different image pairs (Ii, Ij) and (Ij, Ik), which contain the projections
of two commonly visible planes. We shall use the plane-based coordinate transformation method for integrating the reconstruction results defined in different spaces.
Let the 4· 4 coordinate transformation matrix Hijk, mapping the points in the projective space
f~pijg to the points in the projective space f~pjkg, be defined by
~
pjk¼ Hijk~pij:
Then, the plane coefficient vectors ~cij, ~cjk of a
common plane, which are respectively defined in the two different projective spacesf~pijg and f~pjkg,
will be related by: ~
cjkffi HTijk~cij: ð9Þ
Thus, it requires the information of five common planes in the two different projective spaces in order to solve for the transformation matrix Hijk.
It is usually not very practical to find five common planes in the image pairs.
On the other hand, the two respective 3· 4 projection matrices associated with image Ij
de-fined in the two projective spacesf~pijg and f~pjkg are
related directly by the matrix Hijk(Fitzgibbon and
Zisserman, 1998). This relationship provides 11 linear equations in the 15 matrix elements in Hijk.
Then, it is reduced to a need of two plane infor-mation to provide six additional linear equations to solve for the 15 unknowns. In the following we shall give a system of 24 linear equations using the information of two planes for solving for the 15 unknowns; the result will be more reliable.
Combining this equation with a plane equation ~
bTij~pij ¼ 0, we have
Thus,
Similarly, for the same point on plane PB, but
represented as ~pjk in the different projective space
f~pjkg, we can relate it to the same 4 · 1 vector
lj~uTj 0
T
by
Then
After some algebraic manipulation, this can be reduced to I bT jk=bjk4 " # ~ uj¼ kBHijk B1ij bT ijB 1 ij =bij4 " # ~ uj:
Since this equality holds for all image points on plane PB, it further implies:
I bT jk=bjk4 " # Bij¼ kBHijk I bT ij=bij4 " # :
This leads to a system of 12 linear equations in 16 unknowns: 15 from the matrix Hijk plus one
from kB. Therefore, we need another system of
equations provided by a second visible plane, say, PG: I gT jk=gjk4 " # Gij¼ kGHijk I gT ij=gij4 " # :
Combining the above two systems of equations, we have a total of 24 linear equations in 17 un-knowns. Here we give a least squared solution by placing the two systems of equations in the fol-lowing form Hijk I b*Tij=bij4 2 6 4 3 7 5 I g*Tij=gij4 2 4 3 5 3 5 2 6 4 ffi I b*Tjk=bjk4 2 6 4 3 7 5Bij 2 6 4 kB kG I g*Tjk=gjk4 2 4 3 5Gij 3 7 5 ð10Þ where the ratio of kB=kGhas been estimated during
the plane reconstruction phase (see Eq. (7)). We can find the matrix Hijk using the pseudo-inverse
matrix of the 4· 6 matrix on the left-hand side of the above equation.
5. Computation of homographies
We need to estimate Aij from the image data
associated with the planar surface PA. We shall
use the region-based matching, instead of point-based matching, to find the homography. First of all, we use the Gabor filtering technique (Sun et al., 2002) to identify at least four point corre-spondences in order to obtain the initial solution of the homography. We then use the Levenberg– Marquardt iterative nonlinear minimization algo-rithm (Szeliski, 1996) to minimize the sum of the squared intensity differences of the transformed and original image points due to the plane PA in
the image pair
E¼X k fIjððuikÞ 0;ðvi kÞ 0Þ I iðuik; v i kÞg 2 :
Here the transformed location ðui kÞ 0; ðvi kÞ 0; 1 T is obtained from the image point (ui
k; v i
k) using an
estimated Aij, and Ij ðuikÞ 0;ðvi
kÞ 0
is the intensity obtained by a bilinear interpolation from the ori-ginal image Ij. The intensity values of the image
points in the common region of the two images are normalized to remove the possible illumination
difference. The above minimization method con-verges in a few iterations.
After finding two homographies, recall that we can compute the epipole ~ej from the skew
sym-metry property of BT½~ejA. Also, in turn, we can
compute the fundamental matrix F using the epi-pole ~ej as follows: ½~ejAffi ½~ej MjRjM1i ke~ej~aTEM 1 i n o ffi ½~ejMjRjM1i ¼ F: 6. Experimental results 6.1. Experiment 1
In the first experiment we use a synthetic tower whose feature points and schematic diagram are given in Table 1 and Fig. 2. We take a sequence of six pictures to cover all aspects of the tower using a virtual camera looking down from the upper positions. The image resolution is 640· 480 in pixel. Three consecutive images of the sequence, I1,
I2, and I3, are shown in Fig. 3. We apply the
reconstruction process to this data set. We employ a linear least-squares method based on eight cor-responding image point pairs available in the synthetic data to get the true homography for each of the five planes, PGr, PA, PB, PE, PF
vis-ible in the pair (I1, I2). In addition, to handle the
possible problems caused by data translation and scaling change, we also use the normaliza-tion transform proposed by Hartley (1997) to compute the homographies. We choose PGr as
the reference plane. During the reconstruction process, we find the plane coefficient vectors with respect to the reference plane PGr vector
desig-nated as½ 1 1 1 1T. To check the correctness of the final 3D projective reconstruction result, we convert the 3D camera centered projective space back to the 3D object centered Euclidean space using the 3D Euclidean data of the tower available in Table 1 to measure the reconstruction errors in the metric space. The computation times for esti-mating the plane coefficient vectors and the coor-dinate transformation matrix for space integration
are within a second. Ta
ble 1 3D object center ed coord inates of the towe r featu re po ints P o in t 12345678 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 X ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 20 ) 13.333 ) 13.333 ) 6.667 ) 6.667 0 0 ) 50 ) 70 ) 50 0 5 0 7 0 Y 0 0 120 120 5 0 5 0 7 0 7 0 8 0 8 0 110 110 140 140 160 1 60 180 0 0 0 0 0 0 0 Z ) 20 20 20 ) 20 ) 10 10 10 ) 10 ) 15 15 15 ) 15 ) 13.333 13.333 ) 6.667 6.667 0 ) 70 ) 50 0 5 0 7 0 5 0 0
In what follows we assume the noise is uni-formly distributed over the interval [R; R], where Rindicates the noise strength or level. We generate 500 copies of noisy image data using the given noise model with R¼ 0:5, 1.0, 1.5 and 2.0 pixels, respectively. Then we find the 500 reconstruction results and compute the mean and standard devi-ation of the differences between the true and the estimated values of the 3D coordinates of the tower feature points. Table 2 lists the statistics of the relative distance errors into the x, y, and z components. The results indicate our reconstruc-tion method is quite stable in the presence of the noise.
Since the rest of the planes visible in the images are estimated relative to the reference plane, we shall examine the effect of the assigned value of the reference plane coefficient vector on the recon-struction. Five hundreds of the reference plane coefficient vectors are uniformly generated from the range [103, 102]; we also randomly select the
Table 2
The statistics of the distance errors of the reconstruction results
Error value Noise level R (in pixels)
0 0.5 1 1.5 2
Error type Distance mean error x 6.624e)5 0.0547 0.1096 0.2108 0.2505
y 2.492e)5 0.0896 0.1811 0.2944 0.3923
z 7.341e)5 0.0761 0.1514 0.3379 0.3798
Standard deviation x 2.027e)4 0.0855 0.1707 0.3655 0.3850
y 3.188e)5 0.1323 0.2673 0.4416 0.5745
z 1.192e)4 0.1165 0.2320 0.6085 0.9093
Fig. 3. Three distinct images I1, I2and I3taken at a distance of about 500 in. The visible planes in the three images are PGr, PA, PB,
PE, PFin I1and I2, and PGr, PB, PF, PC, PGin I3. 1 2 8 7 5 6 11 12 9 4 13 17 16 15 14 Gr A B Π F E 3 10 18 19 20 23 22 21 24 z y x Π Π Π Π
Fig. 2. The schematic diagram of the tower. The dimensions of the tower are 40 in. in depth (the x-direction), 40 in. in width (the z-direction) and 180 in. in height (the y-direction).
positive or negative sign for the coefficients. Fig. 4 depicts the reconstruction results under the effects of the random selection of the reference plane coefficient vector and the noise at different levels. The horizontal axis indicates the trial number of the reconstruction process and the vertical axis indicates the resulting distance errors. The various marks ‘‘.’’, ‘‘o’’, ‘‘x’’, ‘‘+’’ and ‘‘’’ stand for mean errors of the computed relative distances associ-ated with the uniform noise levels of R¼ 0, 0.5, 1, 1.5 and 2 pixels, respectively. The figure indicates the reconstruction results are virtually not affected by the random selection of the reference plane coefficient vector under the given specified noise. A remark is in order here. That is, we must avoid using ð1; 1; 1; 0ÞT for the reference plane
coeffi-cient vector, since the camera origin ð0; 0; 0; 1Þ is supposed not to lie on the plane.
6.2. Experiment 2
For a comparison between our method and the point-based method employing the fundamental matrix estimated from two arbitrary planes with the aid of hallucinated points (Szeliski and Torr, 1998), we use the same setup as in the previous experiment and run the experiment 500 times with a uniform distribution at different noise levels. The reconstructed Euclidean position errors are com-puted and tabulated in Table 3. The position errors are in the unit of inch.
From this table, we observe that in the noiseless cases where the noise level is 0, all the recon-struction results obtained by the two methods are almost equally good and very small; the errors are due to the rounding/truncation errors arising from numerical computations. As the noise level
Table 3
The mean errors of the reconstructed Euclidean point positions for different setups
Method Noise level (in pixel)
0 0.2 0.5 1.0 Ours 0.0000301259 0.2030772297 0.4996064197 1.2803144642 ð4; 2; 2Þa 0.0000781737 1.6271384793 14.5382797879 47.6350445089 ð4; 4; 1Þa 0.0000125658 0.2113535985 0.5662224535 1.4202651215 ð4; 4; 2Þa 0.0000134805 0.2128984139 0.5343441024 1.4456471793 a
(n; m; p): n, m are the respective numbers of points on the two planes, p is the number of points hallucinated per plane. Fig. 4. The effects of the uniformly generated reference plane coefficient vectors and the noise at different levels on the reconstruction result.
increases from 0 to 1.0 for all the (n; m; p) cases, we notice that the homography estimation using the four noisy data points varies dramatically, and, thus, the fundamental matrix computation with the resulting noisy data points is bad. These lead to the final reconstruction results with large errors. For a more interesting comparison, we compute the homography using the four outmost data points and then use the estimated homo-graphy to generate the hallucinated points located inside the area surrounded by the four outmost points; we denote these hallucinated points as the p (p¼ 1 or 2) points. In our method since the homographies are iteratively estimated in a re-gion-based way, so our reconstruction results are good even in the presence of image noise. In these simulations, the reconstruction results of the two methods are nearly equally good. Even so, our method is better in the sense that we can effi-ciently find each 3D plane without the need of computing 3D points, while the authors in the other method alternated a plane estimation stage with the point reconstruction stage. Thus, their method conducted two kinds of estimations: plane and points.
6.3. Experiment 3
In this experiment the real images of a polyhe-dral, depicted in Fig. 5, are used to reconstruct the model of seven major planar surfaces. We go through the whole reconstruction process as we did in Experiment 1. The line parallelism and perpendicularity properties of the scene are used to
9 10 12 10 2 3 5 1 6 14 13 8 17 16 15 19 18 20 A B G r G F E D C Π Π Π Π Π Π Π Π
Fig. 5. The indices of the vertices and planes of the object.
compute the projective-to-Euclidean coordinate transformation matrix. First, the line parallelism is used to compute the plane at infinity which is then used to transform the reconstruction results from projective space to affine space. Secondly, the line perpendicularity is used to transform the recon-struction results from affine space to metric space. Further details can be found in (Daniilidis and Ernst, 1996; Zhang et al., 1998). Fig. 6 shows the four new views of the reconstructed model, which look like the real ones. Besides, the metric angles between individual object plane and the ground plane are shown in Table 4. The reconstructed object is found to be rather close to the true one.
7. Conclusions
An uncalibrated planar object reconstruction method has been described in which we rely on the plane information. We first estimate the homo-graphy for all planar surfaces using the region features of planar surfaces, and then we use the homographies induced by two planes to compute the epipole. We represent explicitly the compatible projection equations for the stereo images using the information of planar homographies and an assigned reference plane coefficient vector. We continue to derive the 3D equations of the planes visible in the stereo images with respect to the assigned reference plane once the planar homo-graphy is determined. Finally, to integrate the reconstruction results obtained from different image pairs under a unified projective space, we use the homography and the plane coefficient
vector information of two planes to derive the coordinate transformation matrix. We then com-pute the new plane equation for the planes in the unified projective space. In the experiments we conduct the sensitivity analysis on our method by introducing image noise. We also consider the ef-fect of assigning the different values of the refer-ence plane coefficient vector on the reconstruction results. Experimental results on the synthetic and real images indicate the reconstruction method works quite successfully. In the future, we shall consider combining this plane based reconstruc-tion method with other methods to determine the 3D structure of more complex objects.
References
Beardsley, P.A., Zisserman, A., Murray, D., 1997. Sequential updating of projective and affine structure from motion. Int. J. of Comput. Vis. 23 (3), 235–259.
Criminisi, R., Zisserman, A., 1998. Duality, rigidity and planar parallax. In: Proceedings of European Conference on Computer Vision. vol. 2. pp. 846–861.
Daniilidis, K., Ernst, J., 1996. Active intrinsic calibration using vanishing points. Pattern Recogn. Lett. 17, 1179–1189. Faugeras, O.D., 1992. What can be seen in three dimensions
with an uncalibrated stereo rig? In: Proceedings of Euro-pean Conference on Computer Vision. pp. 563–578. Faugeras, O.D., 1993. Three-Dimensional Computer Vision: A
Geometric Viewpoint. MIT Press, Cambridge, MA. Fitzgibbon, A.W., Zisserman, A., 1998. Automatic camera
recovery for closed or open image sequences. In: Proceed-ings of European Conference on Computer Vision. pp. 311– 326.
Fradkin, M., Roux, M., Maitre, H., Lelogu, U.M., 1999. Surface reconstruction from multiple aerial images in dense urban areas. In: Proceedings of International Conference on Computer Vision. pp. 262–267.
Hartley, R.I., Gupta, R., Chang, T., 1992. Stereo from uncalibrated cameras. In: Proceedings of International Conference on Computer Vision and Pattern Recognition. pp. 761–764.
Hartley, R.I., 1994. Projective reconstruction and invariants from multiple images. IEEE Trans. Pattern Anal. Mach. Intell. 16 (10), 1036–1041.
Hartley, R.I., 1997. In defense of the eight-point algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 19 (6), 580–593. Hartley, R.I., Zisserman, A., 2000. Multiple View Geometry in
Computer Vision. Cambridge University Press, Cambridge, UK. Chapter 9.
Hamid, N.H.G., Cipolla, R., 1997. Identifying planar regions in a scene using uncalibrated stereo vision. In: Proceedings of British Machine Vision Conference.
Table 4
The estimated angles between the object planes and the ground plane
Planes form the angle Angle
Estimated value Actual value
(PGr;PA) 92.21 90 (PGr;PB) 92.69 90 (PGr;PC) 97.09 90 (PGr;PD) 91.67 90 (PGr;PE) 1.69 0 (PGr;PF) 45.31 45 (PGr;PG) 1.66 0
Irani, M., Rousso, B., Peleg, S., 1997. Recovery of ego-motion using region alignment. IEEE Trans. Pattern Anal. Mach. Intell. 19 (3), 268–272.
Irani, M., Anandan, P., Weinshall, D., 1998. From refer-ence frames to referrefer-ence planes: multi-view parallax geometry and applications. In: Proceedings of Euro-pean Conference on Computer Vision, vol. 2. pp. 829– 845.
Johansson, B., 1999. View synthesis and 3D reconstruction of piecewise planar scenes using intersection lines between the planes. Proceedings of European Conference on Computer Vision. pp. 54–59.
Luong, Q.T., Vieville, T., 1996. Canonic representations for the geometries of multiple projective views. Comput. Vis. Image Und. 64 (2), 193–229.
Luong, Q.-T., Faugeras, O.D.,1993. Determining the funda-mental matrix with planes: instability and new algorithms. In: Proceedings of Conference on Computer Vision and Pattern Recognition. pp. 489–494.
Mohr, R., Arbogast, E., 1991. It can be done without camera calibration. Pattern Recogn. Lett. 12 (1), 39–43.
Rother, C., Carlsson, S., Tell, D., 2002. Projective factorization of planes and cameras in multiple views. In: Proceedings of International Conference on Pattern Recognition. pp. 737– 740.
Rother, C., 2003. Multi-View Reconstruction and Camera Recovery using a Real or Virtual Reference Plane. Doctoral Dissertation. ISBN 91-7283-422-6, KTH, Stockholm, Sweden.
Szeliski, R., 1996. Video mosaics for virtual environments. IEEE Comput. Graph. Appl. 6 (March), 22–30.
Szeliski, R., Torr, P.H.S., 1998. Geometrically constrained structure from motion: Points on planes. In: Proceedings of European Workshop on 3D Structure from Multiple Images of Large-Scale Environments. pp. 171–186.
Sinclair, D., Blake, A., 1996. Quantitative planar region detection. Int. J. Comput. Vis. 18 (1), 77–91.
Sawhney, H.S., 1994. 3D geometry from planar parallax. In: Proceedings of Conference on Computer Vision and Pattern Recognition. pp. 929–934.
Sun, S.K., Chen, Z., Chia, T.L., 2002. Invariant feature extraction and object shape matching using Gabor filtering. In: Proceedings of International Conference on Visual Information System, Hsinchu, Taiwan. pp. 95–104. Tsai, R.Y., Huang, T.S., 1982. Estimating three-dimensional
motion parameters of a rigid planar patch, II: singular value decomposition. IEEE Transactions on Acoustic, Speech and Signal Processing, vol. 30.
Theiler, L., Chabbi, H., 1999. Projective decomposition of planar facets. In: Proceedings of International Conference on Image Analysis and Processing. pp. 556–561.
Zelnik-Manor, L., Irani, M., 2000. Multi-frame estimate of planar motion. IEEE Trans. Pattern Anal. Mach. Intell. 22 (10), 1105–1116.
Zhang, Z., Isono, K., Akamatsu, S., 1998. Euclidean structure from uncalibrated images using fuzzy domain knowledge: Application to facial images synthesis. In: Proceedings of International Conference on Computer Vision. pp. 784–789.
