Automatic 3D House-layout Construction by a Vision-based Autonomous Vehicle Using a Two-camera Omni-directional Imaging System
Bo-Jhih You Institute of Multimedia Engineering,
National Chiao Tung University, Hsinchu, Taiwan 30010 E-mail: [email protected]
Wen-Hsiang Tsai Dept. of Computer Science, National Chiao Tung University,
Hsinchu, Taiwan 30010 E-mail: [email protected]
Abstract—An automatic house-layout construction system via vision-based autonomous vehicle navigation without human involvement is proposed. The system can be used to acquire the floor layout and flat objects on walls like doors and windows of an empty room. First, a new type of omni-directional camera system is designed, which consists of two omni-cameras aligned coaxially and back to back. The proposed house layout construction process consists of three stages, vehicle navigation, floor layout construction, and 3-D graphic house layout display.
The vehicle navigation stage is conducted to follow the wall mopboards. Then, a global optimization method is proposed to construct a floor layout from all the mopboard edges in the second stage. In the last stage, doors and windows are detected from the omni-images taken in the vehicle navigation stage.
With the above-mentioned data, a graphic form of the house configuration for 3-D display from any viewpoint can be obtained. Finally, experimental results are shown to to demonstrate the feasibility of the proposed approach.
Keywords-autonomous vehicle navigation, omni-directional camera, automatic house layout construction.
I. INTRODUCTION
In recent years, related applications of the autonomous vehicle have been developed intensively by many researchers to help human beings in various areas of automation. There are many empty pre-owned houses with uncertain interior space configurations, and a house agent may want to obtain the layout of each empty room in a house in advance before getting the house ready for sale. It is inconvenient and sometimes dangerous for the house agent to measure the house layout line by line and room by room by hand. Besides, this task usually includes measurement of the positions of doors and windows in rooms, which might be so numerous that the task becomes too time-consuming to be endurable. A possible way out is to conduct the task automatically without human involvement!
To achieve this goal, it is desired to design a vision-based autonomous vehicle to possess an ability to navigate by mopboard following in an unknown empty room space and acquire automatically during navigation the information of the room layout and the structures (positions and heights) of the doors and windows in the room. This is feasible because most houses have mopboards at the roots of the walls. Use of the mopboard for vehicle guidance also simplifies the design of the vehicle navigation work for solving the above-
mentioned automatic house-layout construction problem.
Therefore, object localization for estimating mopboard positions is required.
Various visual sensing equipments such as cameras and ultrasonic sensors were used to acquire the 3-D information in vehicle surroundings in the past studies [1-2]. For depth information estimation, the most common stereo matching method may be used, which is based on the triangulation principle to obtain the relation between a 3-D point and multiple cameras, but matching of corresponding points between images is often a difficult problem in this approach.
Use of the laser range finder together with conventional imaging sensors were also used frequently [3-4]. However, the laser range finder has a drawback, that is, it can only scan a 2D plane at a specific height. Jeng and Tsai [5] proposed a space-mapping method for the omni-camera image, which can be used to estimate the location of an object. In this study, we propose a method which is based on two concepts.
The first is the use of the triangulation principle to measure depths of objects (i.e, their ranges). The other is the use of the space-mapping technique proposed by Jeng and Tsai [5]
to get the location for a concerned object. In this way, we can estimate the positions of objects on the floor and walls in a room.
Autonomous vehicles or mobile robots usually suffer from accumulations of mechanical errors during navigation, which cause inaccurate measures of the moving distances and orientations yielded by the odometer in the vehicle. Chen and Tsai [6] proposed a mechanical error correction method by curve-fitting the erroneous deviations from correct paths.
In this study, we utilize the curve-fitting method proposed by Chen and Tsai [6] to correct the positions of the vehicle and use the edge information estimated from images to correct the direction of the vehicle. Analyzing the omni-images taken by different types of camera is an important topic. Kim and Oh [4] proposed a method for extracting vertical lines from an omni-image with the help of the horizontal lines generated by a laser range finder.
The goal of this study is to design a system for automatic house-layout construction in an empty indoor room space using a vision-based autonomous vehicle. It is assumed that the adjacency walls in the room are perpendicular to each other. In order to achieve this goal, the system needs not only a capability of automatic navigation but also one to acquire environment information automatically. For a vehicle to
have such capabilities, an imaging device with two vertically-aligned omni-cameras connected in a bottom-to- bottom fashion is designed in this study. With the environment information collected from the taken images, a method for constructing a 3-D house layout in graphic form by estimating the locations of mopboard edges, doors, and windows is also proposed. The system consists of three major phases: setup of the imaging system, vehicle navigation by mopboard following, and 3-D construction of room space. The details will be introduced in the following sections.
II. SYSTEM CONFIGURATION
In the proposed system, we make use of a Pioneer 3-DX vehicle made by MobileRobots Inc. as a test bed. The vehicle is equipped with an imaging system composed of two catadioptric omni-directional cameras which are connected and vertically-aligned in a bottom-to-bottom fashion. The imaging system is not only part of the vehicle system but also plays an important role of gathering environment information and locating the vehicle. A diagram illustrating the configuration of this system is shown in Fig.
1(a), and Figs. 1(b) and 1(c) show an example of the omni- images captured by the imaging system.
The overall framework of the proposed system is illustrated in Fig. 2. The major stages in proposed 3D house- layout construction by vehicle navigation include: (1) vehicle navigation by mopboard following; (2) floor-layout construction; and (3) 3-D house-layout construction.
(b)
(a) (c)
Figure 1. Illustration of system configuration. (a) The two-camera imaging system is equipped on the vehicle. (b) An acquired image of the ceiling using the upper camera. (c) An acquired image of the floor using the lower camera.
Figure 2. Flowchart of proposed system.
Before conducting vehicle navigation, we must perform the setup of the system which includes space-mapping calibration and mechanic error correction. For camera calibration, we establish a space-mapping table which is proposed by Jeng and Tsai [5] for each omni-camera by finding the relations between specific points (on a specially- designed calibration object) in 2-D omni-images and the corresponding points in 3-D space. In other words, the space- mapping table specifies a relation among the corresponding space-image coordinates pairs (Pi, pi). The relation between the omni-camera and image coordinate systems is shown in Fig. 3, and the example of mapping table is shown in Table 1.
As a result, the vehicle which carries the imaging system has the abilities to estimate the distances between the vehicle and an object (the mopboard here) utilizing the omni-images.
Figure 3. Relation between omni-camera and image coordinate systems.
Table 1. Example of pano-mapping table of size M×N [5].
When creating the space mapping table, the focal point of the hyperboloidal mirror is viewed as the origin in the CCS.
However, the focal point may not be located on the bottom of the omni-camera. We propose in this study a method to find the focal point of the hyperboloidal mirror. As shown in Fig. 4, we use two different landmarks L1 and L2 which have the same corresponding image point p with known heights and horizontal distances from the transverse axis of the hyperboloidal mirror. We assume that Ow is at (0, 0, 0). Then, as shown in Fig. 4, the position of the focal point can be computed by the following equation:
1 2 1
1 2 1
tan H O Om w H H
D D D
θ − −
= =
− ; (1)
2 1
1 1 1 1
2 1
m w tan
H H
O O H D H D
D D
θ −
= − × = − ×
− . (2) Accumulations of mechanical errors during navigation usually will lead to wrong control instructions and inaccurate
vehicle localization information, and it is desired to conduct a calibration task to eliminate such errors. Chen and Tsai [6]
proposed a mechanical error correction method by curve- fitting the erroneous deviations from correct paths. In this study, we use this method for correcting the vehicle odometer readings.
Figure 4. Finding out the focal point Om.
III. 3-DDATA MEASUREMENT MEHTOD
In this study, we design a new type of omni-directional camera to achieve acquisition of environment images. Note that the two omni-cameras are connected and vertically- aligned in a bottom-to-bottom fashion, which means that none of the space points can be projected on both image planes at the same time. As a result, the most common stereo matching method that is based on the triangulation principle to obtain the relation between a 3-D point and multiple cameras does not work here at all. We need another theory to deal with the stereo matching problem here. One way proposed in this study is to estimate the horizontal distance to the desired object point, and subsequently to utilize both the distance and the elevation angle of the space point with respect to the local camera coordinate system CCSlocal to get relevant 3-D data. As shown in Fig. 5, assume that both space points P1 and P2 lie on the vertical line L which is perpendicular to the x-y plane in the global coordinate system GCS and that P1 is the intersection point of L and the x-y plane in the GCS. Assume also that P1 and P2 lie on the light rays with the azimuth-elevation angel pairs of (θ1, ρ1) and (θ2, ρ2) with respect to the upper camera coordinate system CCS1 and the lower camera coordinate system CCS2, respectively, where Om1 is the focal point of the lower mirror, and Om2 is the focal point of the upper mirror. Assume finally that the point Olocal with coordinates (0, 0, 0) is the origin of the CCSlocal
.
Referring to Fig. 5 and by the triangulation principle, we have1
tan 1
D MH
= α , (3)
where D is the horizontal distance between Olocal and P1, and MH1 is the height of the focal point Om1. As a result, the height H2 of the space point P2 can be computed by
2 2 tan 2
H =MH + ×D α (4 )
where MH2 is the height of the focal point Om2
.
According to the rotational invariance property of the omni-camera, we can derive the following equations:
Om1
Om2
Ol o c a l(0,0,0) P2
P1
Z Y X α2
α1
D M H 1
M H 2
L
Z2
Y2
X2
X1 Y1
Z1
(a)
(b)
Figure 5. (a) Computation od depth using the two-camera omni- directional imaging system. (b) Details of (a).
1 1
1 2 2 2 2
1 1 1 1
cos X u
X Y u v
θ = =
+ + ; (5 )
1 1
1 2 2 2 2
1 1 1 1
sin Y v
X Y u v
θ = =
+ + ; (6 )
2 2
2 2 2 2 2
2 2 2 2
cos X u
X Y u v
θ = =
+ + ; (7 )
2 2
2 2 2 2 2
2 2 2 2
sin Y v
X Y u v
θ = =
+ + . (8 )
Because the vertical line L is perpendicular to the X1-Y1 and X2-Y2 planes and the positive direction of the X1-axis in the CCS1 is opposite to the positive direction of the X2-axis in the CCS2, it can be found that X1
X2 and
1 2 1 2
cosθ = −cos ; sinθ θ =sinθ . (9 ) Now, we can utilize (3), (4), and (9) to calculate the coordinates (X Y Zp1, p1, p1) of P1 with respect to the CCSlocal
as follows:
1 1
1
1 1 1 1
1 1 1 1
cos ( / tan ) cos ;
sin ( / tan ) sin ;
0,
p p
p
X D MH
Y D MH
Z
θ α θ
θ α θ
= − × = − ×
= × = − ×
=
(10 )
and the coordinates (Xp2,Yp2,Zp2) of P2 with respect to the CCSlocal as follows:
2 1
2 1
2
2 1 1 2
2 1 1 2
2 2 2
2 1 1 2
cos ( / tan ) cos ;
sin ( / tan ) sin ;
tan ( / tan ) tan .
p p
p p
p
X D MH X
Y D MH Y
Z H MH D MH MH
θ α θ
θ α θ
α
α α
= × = × = −
= × = × =
= = + ×
= + ×
(11)
IV. HOUSE LAYOUT CONSTRUCTION
As mentioned previously, it is desired to conduct the house-layout construction task by driving the vehicle around the room and collect environment data. More specifically, we discuss the major stages of the house-layout construction task in the following.
A. Vehicle navigation by mopboard following
Most houses have mopboards at the roots of their walls.
The mopboard feature is used for vehicle navigation in this paper. In this section, we describe the method we use to detect mopboard edges in omni-images and the entire navigation strategy.
According to the rotational invariance property of the omni-image, all points lying on each straight line L, which is parallel to the Z1-axis in the CCS1, are projected onto the image plane of the upper image coordinate system ICS1 with the azimuth θ1, forming a straight line in the ICS1 and passing the origin I1 of ICS1, as shown in Fig. 6. And because mopboards occupy the bottom parts of walls, it results in a fact that the projected pixels of the mopboards in the omni-image also have an obvious band. We utilize these two properties to find out the mopboard edge points which are located on the edge between the mopboard and the floor.
Assuming that the house is empty, we can transform an acquired grayscale image into a binary one by a pre-defined threshold and perform erosion and dilation operations on it.
In this way, the mopboard will remain in the image and the mopboard edge points can be detected by using a scanning line starting from the image center. An algorithm for the detection of the mopboard edges is described as follows.
Figure 6. Vertiine L in the OCSlocal and corresponding line l in the ICS1
Algorithm 1: Mopboard detection
Input: An omni-image Ik taken from the lower omni-camera;
a desired scanning range; and a vehicle mask.
Output: A set Mk of image coordinates of mopboard edges.
Steps:
1) Transform Ik into a grayscale version IG
.
2) Reset the gray values gpi of each pixel pi in the gray image IG by comparing gpi with a pre-defined threshold T as follows:
if gpi ≥ T, set gpi as a white pixel;
else, set gpi as a black pixel.
3) Perform erosion on IG to obtain a new image IG′.
4) Perform dilation on IG′ to obtain a second new image IG′′.
5) For each scan line Lθ in a pre-selected angle scanning range with certain angular scanning steps, perform the following steps.
a) Start from the image center of IG′′, traverse along the line Lθ, and find the first intersection black pixel p of IG′′ with Lθ
.
b) Check if p is followed by 10 or more consecutive black pixels along Lθ: if so, then add the coordinates of p into Mk (which is set empty initially).
6) Repeat Step 5 until the scan range is exhausted.
After detecting the mopboard edge points, we have a set Mk which includes the image coordinates of the detected mopboard edge points. Denote the set Mk in more detail byMk =
{
( , )u vt t}
, where (ut, vt) are the detected mopboard edge coordinates in the ICS1. In order to estimate the corresponding locations of these detected edge points, we first choose an element (ut, vt) from Mk and calculate the corresponding radius rt and azimuth θt by transforming (ut, vt) into the corresponding polar coordinates (rt, θt). Then, we determine a column index of the pano-mapping table by finding out which azimuth interval the θt lies on. For example, if there exists i satisfying the following equation:(2 / ) ( 1) (2 / ) 1
i i M t i M i
θ = × π ≤ < + ×θ π =θ+ , (12)
then the column index is taken to be i. Also, we compare rt
with rs in the entry Eis of the pano-mapping table, and keep the nearest radius rj. As a result, the corresponding ρj can be found out. Once the ρj is known, the 3-D position estimation task can be carried out.
For vehicle navigation, the distance to the wall is important navigation information. The main idea is to navigate and keep the navigation path parallel to the wall. In this way, we choose three regions RL, RR, and RF on the CCSlocal, as shown in Fig. 7, so that the walls which are close to the vehicle can be detected. And we define the three sets SL, SR , and SF by
{ }
{ }
{ }
| and ;
| and ;
| and
L L L k L L
R R R k R R
F F F k F F
S P P M P R
S P P M P R
S P P M P R
= ∈ ∈
= ∈ ∈
= ∈ ∈
(13)
where the set Mk includes the detected mopboard edge points with coordinates in the CCSlocal.
Figure 7. An illustration of the pre-defined detect regions.
However, for example, the edge points which are on the left wall with respect to the vehicle may appear both in RL
and in RF at the same time, resulting in the wrong distance estimation. In order to transform the detected mopboard edge point locations into useful navigation information and environment data, a pattern classification technique is proposed. Once we can assign each of the edge points to its corresponding wall by this classification technique, we can adjust the direction of the vehicle to keep the vehicle’s navigation path parallel to each wall and estimate the distances to the nearby walls. Let the new sets derived from (13) after the pattern classification process be described as follows:
{ }
{ }
{ }
' ' ' '
' ' ' '
' ' ' '
| and ;
| and ;
| and .
L L L L L L
R R R R R R
F F F F F F
S P P S P W
S P P S P W
S P P S P W
= ∈ ∈
= ∈ ∈
= ∈ ∈
(14 )
As a result, the distances to each wall can be estimated more accurately, and are used as navigation information to guide the vehicle.
As mentioned above, the estimated positions of the detected mopboard edge points are derived in a raster- scanning order in the omni-image. Based on this order, the classification process would be simplified because any of these estimated points can be easily divided into two parts.
The purpose of using a pattern classification technique in this study is to find an edge point locating at the corner of two adjacent walls that can divide the edge points into two parts.
With a property that the adjacent walls are perpendicular to each other, we can conduct the pattern classification process by the LSE criterion to find such a corner point, if it exists.
Let such a corner point, denoted as Sk, divides the set of edge points in different walls into two sets, denote as Sk1 and Sk2. A reasonable assumption is made here, that is, the two lines which are used to fit the points in Sk1 and Sk2 by the LSE criterion, respectively, are perpendicular to each other. In this way, such a corner point will yield a minimum error.
Before driving the vehicle to go forward or to turn around, it is desired that the vehicle can keep its path to be parallel with the walls. Otherwise, the vehicle may hit the wall. For this purpose, the detected mopboard edges can be used, by a line fitting process, to generate a straight line which describes the direction of these detected edges. With the direction of the line, the vehicle can adjust its pose such
that it can keep an appropriate distance with respect to the wall. The entire process is illustrated in Fig. 8.
(a) (b)
Figure 8. Vehicle direction adjustment. (a) θ
90o. (b) θ 90o.Before the vehicle starts a navigation session, it will estimate the distances to the nearby walls which are within the relevant angular scanning ranges. Based on the distances, we can know if it can turn to the left, turn to the right, or go forward, and then issue an appropriate control instruction to the vehicle to drive it to navigate safely. The major rules for controlling the vehicle are described as follows:
• For the work of the imaging system, keeping an appropriate distance with the wall is needed.
• The vehicle can go forward until the left wall is not detected further or until it is blocked by the frontal wall.
Both situations are illustrated in Fig. 9, and the algorithm of issuing a control instruction to the vehicle for each navigation cycle is described as follows. Note that the distance DLB is assigned a value only under the condition that the vehicle exceeds the corner point, as shown in Fig. 9(b).
For gathering enough information, it is desired that the vehicle goes forward a fixed distance moveDistfix if it is moveable, and then collect information. Refer to Fig. 9 for notations used in the algorithm. DL, DF, and DR are the distances between the vehicle and the left wall, the frontal wall, and the right wall, respectively, if the corresponding walls are detected. The threshold values passLengthnear and passLengthfar are used to decide whether the vehicle can move forward or not.
B. Floor-layout construction
As mentioned, it is reasonable to use two mutually perpendicular lines to fit the edge points which belong to adjacent mutually perpendicular walls. It means that if we choose one fitting line and adjust the direction of it, the directions of other fitting lines will change, too. As a result, using the fitting lines to fit the edge points will incur fitting errors. Based on this idea, we propose a global LSE
(a) (b)
Figure 9. Illustration of the safe ranges. (a) Approaching to the frontal wall. (b) Exceeding the corner.
optimization method to minimize the fitting error and the entire process is described in the following algorithm.
Algorithm 2: Floor-layout construction
Input: n sets of detected mopboard edge points, S1, S2, …, Sn
of n walls, W1, W2, …, Wn. Output: A floor layout.
Steps:
1) (Line fitting for each wall) Fit the points in Sk of Wk with a line Lk by the LSE curve fitting scheme and compute its mean point Mk where 1 ≤ k ≤ n.
2) (Optimal fitting with respect to a chosen line) Perform the following steps to obtain globally optimal fitting with respect to a selected fitting line of a certain wall.
a) Choose a fitting line Lk, starting from k =1 until k = n, and compute its direction angle θk.
b) Adjust θk by adding a small angle ε (initially adding
−10o and then adding 0 . 1 ο each time later until adding up to +10o), resulting in θk′, to generate a new line Lk′ such that Lk′ passes the point Mk with direction angle θk′.
c) (Generation of other fitting lines) Generate a sequence of lines Lk+1′, Lk+2′, …, L(k+n−2)mod n′ with each Lk+i′ perpendicular to its former line Lk+(i−1)′ and passing its original mean point Mk+i (i.e., every two neighboring lines are mutually perpendicular).
d) (Computing the sum of fitting errors of all lines) Compute the error ei of fitting all the points in Si of Wi to line Li′ obtained in the last two steps (Steps b) and c) above), and sum the errors up to get a total error ek′ for Lk′.
e) Repeat b) through d) until the range of angular adjustment, (−10o, + 10o), is exhausted.
f) Find the minimum of all the total errors ek′ and denote it as emin, k.
g) Repeat Steps a) through f) to compute the emin, k for all k = 1, 2, …, n.
h) Find the global minimum error emin, ko as the one which is the minimum of all the emin, k.
3) Take all the lines with adjusted angles corresponding to emin, ko as the desired floor layout.
C. 3-D house-layout construction
Only creating a floor layout is insufficient for use as a 3- D model of the indoor room space. The objects on walls such as doors and windows must also be detected and be drawn to appear in the desired 3-D room model. Therefore, it is indispensible to analyze the omni-images taken by the imaging system to extract such objects. However, an omni- image covers a large range, but the further the distance is, the larger the estimation error is. How to retrieve the desired information is an important task. For this, we propose a method to determine a scanning range with two direction angles for each pair of omni-images based on the floor layout edge equation. With the scanning region for each omni- image, we can retrieve appropriate 3-D information from different omni-images. Note that each object which is detected by the scanning region of each omni-image is regarded as an individual one. Therefore, adjacent objects
which appear in the consecutive omni-images taken from the same omni-camera (the upper or the lower one) must be combined into one based on, e.g., the information of their positions. Also, due to the configuration of the imaging system, some objects on the wall, such as windows and doors, may appear in the pair of omni-images (the upper and the lower ones). To solve it, we propose a method to recognize doors and windows from the combined objects.
As shown in Fig. 10, for each wall Wk, there are multiple navigation imaging spots N1, N2, …, and Nm for taking omni- images as the environment information. In order to determine a scanning region for each omni-image taken at N1
through Nm, we calculate the midpoint Mij of Ni and Nj for all i = 1, 2, …, n− 1 and j = i + 1. Then, we project each midpoint Mij onto the line Lk which is the result of the floor- layout construction process mentioned previously, resulting in a projection point Mij′. And based on the positions of the projection points and the navigation imaging spots, we can determine the cover ranges. With the cover range and the direction of the vehicle at the navigation spot, we can determine the scanning range with two direction angles for each pair of omni-images.
However, if we conduct the detection of objects within each scanning range, the mopboard will be detected as an object on the wall. Because the objects to be detected are doors and windows, and the mopboard is regarded as a feature of a wall, the mopboard should not appear in the detected objects. For this reason, we define a scanning region excluding the mopboard in the scanning range. In order to exclude the mopboard part in the scanning range, we use the space mapping table in a reverse way to estimate the image coordinates of a concerned space point at a known position.
More specifically, we use the distance to a detected edge point, and a pre-defined height of the mopboard to calculate the elevation of a point which is on the top edge of the mopboard with respect to the CCS1. Then, we can look up the space mapping table to find its radius in the omni-image.
(a) (b)
Figure 10. Illustration of determining scanning range. (a) Decide the cover range. (b) Relevant scanning range on omni-image.
As a result, the mopboard part can be excluded in the omni-image, forming a desired scanning region. In this way, the object detection process can be carried out within this region.
With the above-mentioned scanning region, we can detect the objects in each omni-image by the proposed two-
way angular scanning scheme which extends the method in Algorithm 1. The two-way scheme is used instead of the original one described in Algorithm 1 because the scheme can achieve a better estimation. We first perform Steps 1 through 4 of Algorithm 1 for the pair of images, then traverse along the line Lθ from the outer boundary to the inner one and from the inner to the outer boundary within the scanning regions, and find the first intersection black pixel which is followed by 10 or more consecutive black pixels, respectively. The first found black pixel in each direction will be regarded as an element of the boundary of the detected object. According to the two first scanned pixels which are detected by traversing along the scanning line with opposite directions, we utilize the information within the scanning line bounded by the two scanned pixels to determine whether the two pixels are boundary points of the object or not. Besides, objects may occupy the omni-image and are detected for some certain continuous angular interval.
We utilize the property to combine the detected objects which are detected from certain continuous scanning lines into an individual one. In this way, there may be some individual objects in the scanning region. As a result, according to these pixels which are boundary points of the object, and by utilizing the 3-D position estimation method by looking up the space mapping table, the average heights at the bottom and the top of the object can be estimated.
There are some major rules which may be adopted for combing the detected objects.
• First, we traverse along the scanning line Lθ within the scanning region R′ from opposite directions and find the pixels pi and po of the object boundaries, respectively.
• By counting the numbers nb and nsum of black pixels and all pixels between the two detected pixels along Lθ, if found, and by a threshold and the ratio nb/ nsum, we can determine whether the pixels pi and po are on the boundary of the object or not.
• In R′, if all the pixels pi and po for each scanning line Lθ
in an angular interval are all on the boundary of an object, then such a largest interval can used to describe the same object. Besides, in R′, those combined objects may be combined again according to their positions.
With the above rules, all individual objects can be determined in the same scanning region of an omni-image.
However, an object on a wall may appear to cross two or more scanning regions of corresponding source omni-images taken by the same upper or lower omni-camera. Besides, an object also has a possibility to appear in the pairs of omni- images simultaneously. For the above reason, we have to combine those detected individual objects which belong to the same one. Here, we denote O1 and O2 as the sets of the individual objects detected from the omni-images taken by the lower and upper omni-cameras, respectively. As shown in Fig. 11, at first, we conduct the combination task for all the objects in O1 and O2, according to their positions to form two new sets O1′ and O2′, respectively. Then, we conduct the reorganization task. The process is described in the following algorithm.
Algorithm 3: Objects reorganization
Input: O1′ and O2′ including the combined objects on walls.
Output: Window objects set OW, and door objects set OD. Steps:
1) (Objects recognition for each wall) For each floor- layout edge Fk, perform the following steps to recognize the objects on wall Wk
.
a) Choose an object o2,i from O2′, to find an object in O1′ at a similar location.
i. If such object o1,j is found, then check if it is connected to the mopboard by its location:
1. if yes, then recognize o1,j together with o2,i as a door, and add it to OD;
2. otherwise, recognize o1,j together with o2,i as a window, and add it to OW.
ii. If such an object is not found, recognize o2,i as a window, and add it to OW.
b) Repeat Step (a) until the objects of O2′ are exhausted.
c) Recognize the remaining objects in O1′ as widows, and add it to OW.
2) Repeat Step (1) until all the floor-layout edges are exhausted.
V. EXPERIMENTAL RESULTS
We show some experimental results of the proposed automatic house-layout construction system by autonomous vehicle navigation based on mopboard following in indoor environments. The experimental environment was an empty indoor room with mopboards at the bottom of the walls. Fig.
12 shows an example of the resulting images of mopboard detection in which the red dots show the detected mopboard edge points and the result of the pattern classification procedure. Fig. 13 shows the estimated mopboard edge points of all walls and the floor layout which is constructed by the globally optimal fitting method. In Table 2, we show the errors in percentage between the actual widths of the walls and the estimated data of 9 times of navigations using the proposed system. The average error of the wall widths is 2.75% and the average error in percentage of the estimated total perimeter of the floor layout with respect to the real data is 0.21%. We simulated a window in our experimental environment by creating a black frame and attaching it on a wall. The window is not so low that only the upper omni- camera can “see” it. An example of the images of the door and window and the detection results are shown in Fig. 14.
An example of house-layout construction in graphic form is shown in Fig. 15.
VI. CONCLUSSIONS
A system for automatic house-layout construction by vision-based autonomous vehicle navigation in an empty room space has been proposed. To achieve acquisition of images, a new type of omni-directional camera has been designed for this study, which consists of two omni-cameras aligned coaxially and back to back. The proposed automatic house layout construction process consists of three major stages: (1) vehicle navigation by mopboard following; (2) floor layout construction; and (3) 3-D house layout
construction. he entire house layout construction process is fully automatic, requiring no human involvement, and so is very convenient for real applications. The experimental results show the feasibility of the proposed method.
(a) (b)
(c) (d)
Figure 11. Illustration of object combinations. (a) Scanning regions. (b) Individual objects. (c) Combined objects for each omni-camera. (d) Reorganization of objects.
(a) (b)
(c) (d)
Figure 12. Environment data collection. (a) Experimental environment.
(b) Detected mopboard edge points. (c) The detected mopboard points.
(d) Result of the classification (the points belonging to the upper wall).
(a) (b)
Figure 13. Illustration of global optimization. (a) Etimated mopboard edge points of all walls. (b) A floor layout fitting the points in (a).
(a) (b) (c)
(d) (e) (f)
Figure 14. Images and door detection result. (a) Image of the door taken by the upper omni-camera. (b) Image of the door taken by the lower omni-camera. (c) Image of the window taken by the upper omni-camera.
(d) Door detection result of (a). (e) Door detection result of (b). (f) Window detection result of (c).
Figure 15. Graphic display of constructed house layout. (a) Viewing from the top (green rectangle is a door and yellow one is a window). (b) Viewing from the back of the window
REFERENCES
[1] A. Ohya, A. Kosaka, and A. Kak, “Vision-based Navigation by a Mobile Robot with Obstacle Avoidance Using Single-Camera Vision and Ultrasonic Sensing,” IEEE Transactions on Robotics and Automation, Vol. 14, No. 6, pp. 969-978, 1998.
[2] J. Gluckman, S. K. Nayar, and K. J. Thoresz, “Real-Time Omnidirectional and Panoramic Stereo,” Proceedings of DARPA98, pp.299–303, 1998.
[3] B. S. Kim, Y. M. Park and K. W. Lee, “A experiment of 3D reconstruction using laser range finder and CCD camera,”
Proceedings of IEEE 2005 International Geoscience and Remote Sensing Symposium , Seoul, Korea, pp.1442-1445 , July 25-29, 2005.
[4] S. Kim and S. Y. Oh, “SLAM in Indoor Environments using Omni- directional Vertical and Horizontal Line Features,” Journal of Intelligent and Robotic Systems, v.51 n.1, p.31-43, January 2008.
[5] S. W. Jeng and W. H. Tsai, “Using Pano-mapping Tables for Unwrapping of Omni-images into Panoramic and Perspective-view Images,” Journal of IET Image Processing, Vol. 1, No. 2, pp. 149- 155, June 2007.
[6] M. C. Chen and W. H. Tsai, "Vision-based Security Patrolling in Indoor Environments Using Autonomous Vehicles, ”Proceedings of 2005 Conference on Computer Vision, Graphics and Image Processing, pp. 811-818, Taipei, Taiwan, Republic of China, August 2005
[7] C. J. Wu and W. H. Tsai, “Location Estimation for Indoor Autonomous Vehicle Navigation by Omni-Directional Vision Using Circular Landmarks on Ceilings,” Robotics and Autonomous Systems, Vol. 57, No. 5, pp. 546-555, May 2009.
Table 2. Precision of estimated wall widths and their error percentages.
Exp. No. Average estimated wall width error % Estimated perimeter error %
1 2.345 0.36
2 2.586 0.03
3 2.948 0.02
4 3.129 0.35
5 2.090 0.37
6 3.137 0.28
7 2.387 0.09
8 2.749 0.23
9 3.335 0.13
average 2.745 0.21
