This section describes the process of identifying the half-planes in a world scene based upon the corresponding lines and points within a sequence of images of the scene. In order to robustly estimate the planes, the computation process is based on an inter-image homography approach. The computation procedure involves two stages, namely (1) computing the homographies amongst the multiple images of the scene based on corresponding lines and points, and (2) parameterizing and verifying the planar facets.
Baillard et al. [1] proposed a planar reconstruction method in which the half-planes were parameterized by a single 3D line L and an angle of rotation 𝜇 with 180 possibilities (see Fig. 4.2(a)). The half-plane was then verified by computing the intensity similarity of every pixel contained within it. However, this approach incurs a considerable computational cost since it is necessary to search a total of 180 possible angles of rotation. Moreover, when the region within the world plane is smooth and monotonic, the proposed method failed to determine the correct half-plane due to the false high similarity scores produced by the smooth regions. To reduce the complexity of the half-plane search process and to resolve the mismatch problem, the reconstruction method proposed in this study modifies the half-plane parameterization method proposed in [1] by replacing the angle of rotation parame-ter, 𝜇, by a 3D feature point X (note that the use of the single 3D line L is retained, see Fig. 4.2(b)). Without loss of generality, an assumption is made that each cor-rect half-plane contains at least one corresponding point in the multiple images. In order to reduce the computation time, the half-planes in the image are obtained
by searching only those feature points which satisfy prescribed region and coplanar constraints for each 3D line L. Note that here, the region constraint restricts the range over which the search process is performed, and the coplanar constraint is to limits the selection of candidate half-planes to those half-planes in which all the feature points are coplanar in the 3D world scene.
(a) (b)
Figure 4.2: One-parameter family of half-planes spanned by 3D line L: (a) half-plane parame-terized by angle 𝜇; (b) half-plane parameparame-terized by feature point X.
4.2.1 Region Constraint
To improve the efficiency of the search process, it is desirable to minimize the amount of information required to establish the possible half-planes in the real-world scene.
Since the feature points lying on the world plane are generally distributed around the corresponding 3D line L, the half-plane search process commences by obtaining a set 𝑆𝑟 of all the corresponding feature points within a restricted region around L.
The restricted region is bounded by a quadrangle extended from l1, where l1 is the 2D line produced by projecting L onto the first image. Let 𝑆𝑝 be the set of all pixel points within the restricted quadrangle region 𝑟(𝐿, 𝑊 ) , where 𝐿 is the length of the restricted region and is equal to the length of l1 and 𝑊 is the width. In addition, let the long side of the restricted quadrangle region be located along l1. The set containing all the corresponding points in the range 𝑟(𝐿, 𝑊 ) is given by
𝑆𝑟 = {𝑆𝑝∩ 𝑆1} (4.1)
where 𝑆1 contains all the corresponding points in the first image. Since the value of 𝐿 is fixed, the range of restricted region depends on the value of 𝑊 . In practice,
the width is defined as 𝑊 = 𝑛 × (𝐿/5), where 𝑛 is initially set to a small value such as 1. Since at least one corresponding point is required for verifying the half-plane, if the resultant number of members in set 𝑆𝑟 is 0, the value of 𝑛 is increased by 1. This process is repeated iteratively until |𝑆𝑟| ≥ 1. Furthermore, to prevent the restricted region from overextending, the value of 𝑊 is limited to be less than the value of 𝐿. If there is still no corresponding point included in the restricted region which has reached its maximum size, the current 3D line L will be discarded.
4.2.2 Coplanar Constraint
Once all the points in 𝑆𝑟satisfying the region constraint in the first image have been extracted, the set S𝑟 containing the 3D points is constructed from the points within 𝑆𝑟 in the first image and those within the corresponding set 𝑆𝑟′ in another image.
Having obtained the set S𝑟, an extraction procedure is performed to identify the set S containing all of the points which satisfy the coplanar constraint.
Common sense dictates that of all the half-planes spanned by the 3D line L, the more realistic half-planes, i.e., the half-planes which more accurately resemble the real-world plane, are those which contain more coplanar feature points than the others. Thus, for each point X𝑗 in S𝑟, the set containing points lying on the plane parameterized by the 3D line L and the feature point X𝑗 is determined as
S𝑗 = {X𝑖| 𝑑(X𝑖, 𝜋(L, X𝑗)) < 𝜖, ∀ X𝑖 ∈ S𝑟} (4.2) where 𝜋(L, X𝑗) is the plane parameterized by L and X𝑗, and 𝑑(X, 𝜋) is the function measures the distance from the point X to the plane 𝜋. If the distance is smaller than the threshold 𝜖, the point X is considered as on the plane 𝜋. Consequently, the set S𝑗 which contains the greatest number of coplanar points is regarded as the final set S, that is
S = arg max
S𝑗 |S𝑗|. (4.3)
4.2.3 Verifying Half-Plane Regions
The procedure described above yields a set S of constrained feature points associated with the 3D line L. The next step in the reconstruction process is to identify the correct half-plane region associated with L. As shown in Fig. 4.2(b), the possible half-planes of L are constructed by L and all the feature points X𝑘 belonging to the set S (i.e., {𝜋(L, X𝑘) | X𝑘 ∈ S}). In order to identify the correct half-plane amongst the set of all possible half-planes, the image intensity similarity of each half-plane region is estimated over the multiple views using the following similarity score function and the homography H𝑖(𝜋) introduced by the half-plane 𝜋 between the first view and the 𝑖th view is computed. Meanwhile, the set POI (i.e., the Points of Interest in the first view) is obtained by the Canny edge detector [59] with a low threshold setting in order to make the maximum use of the texture information available in the image. In addition, 𝐶𝑜𝑟(x, x𝑖) is the normalized cross-correlation between point x in the first view and point x𝑖 in the 𝑖th view within a local 𝑛 × 𝑛 window. In this study, the homography H(𝜋) introduced by the half-plane 𝜋 between two different views is computed directly from the equation of 𝜋 [35]. Let the projection matrix associated with the first view be denoted as P = [I|0], and let the projection matrix of any of the other views be denoted as P′ = [A′|a′]. Finally, let the planar equation
After estimating the score for each feature point X𝑘, the point X with the highest similarity score is considered to be the best solution for the half-plane, i.e.
X = arg max
X𝑘∈S 𝑆𝑖𝑚 (X𝑘) . (4.6)
The total number of points in S satisfying both the region and the coplanar constraints is far less than 180. Consequently, the number of half-plane search trials
is significantly reduced, and thus the efficiency of the search process is improved.
Algorithm 1 presents the pseudo-code for the half-plane search algorithm described in this section.
Algorithm 1 Half-plane search algorithm.
Objective: Given 2D and 3D corresponding lines and points, find half-planes 𝜋 by searching the corresponding points which satisfy both the region and coplanar constraints.
Algorithm:
8: If |𝑆
𝑟
| = 0, then discard this line and continue using other line;9:
10: // Coplanar Constraint //
11: Obtain S
𝑟
which contains the 3D points constructed from 𝑆𝑟
and 𝑆𝑟 ′
; 12: Estimate S which contains the most coplanar points using Eq. (4.2);13: If |S| = 0, then discard this line and continue using other line;
14:
15: // Verifying Half-Plane Regions //
16: for each point X
𝑘
∈ S do17: Determine the half-plane 𝜋(L, X
𝑘
);18: Compute the homography H introduced by 𝜋 using Eq. (4.5);
19: Compute the similarity score 𝑆𝑖𝑚(X
𝑘
) using Eq. (4.4);20: end for
21: Select the point X with the highest score as the best solution;
22: end for