Chapter 1 Introduction
1.4 Dissertation Organization
In the remainder of this thesis, we describe the proposed methods for various applications in the chapters, respectively. In Chapter 2, we propose a method for location estimation for indoor autonomous vehicle navigation by omni-directional vision using circular landmarks on ceilings. In Chapter 3, we propose a systematic approach to indoor vision-based robot localization using corner features in omni- images. In Chapter 4, we propose an omni-vision based self-localization method for
17
automatic helicopter landing on standard helipads. In Chapter 5, we propose a method for omni-vision based localization of lateral vehicles for car driving assistance. In Chapter 6, we propose a technique for adaptation of space-mapping methods for object location estimation to camera setup changes. In Chapter 7, we propose a method for unwarping of images taken by misaligned omni-cameras without camera calibration by curved quadrilateral morphing using quadratic pattern classifiers. In each chapter, we describe relevant techniques and applications of the proposed methods, and include experimental results to show the feasibility of the methods.
Discussions and suggestions for future studies are also included.
18
Chapter 2 Location Estimation for Indoor
Autonomous Vehicle Guidance by
Omni-vision Using Circular Landmarks on Ceilings
2.1 Idea of Proposed Method
Vehicle localization is essential for guidance of autonomous vehicles in many indoor navigation applications. Most existing vision-based techniques deal only with frontal scenes acquired by traditional cameras and are easily interfered by unexpected objects around the vehicle. A feasible solution to this problem is to use an omni-camera which looks upward at certain landmarks attached on the ceiling [7].
This solution has the unique advantage of providing wide-angle views with fewer objects appearing in the FOV, thus reducing the guidance error coming from landmark occlusion, noise inference, etc. This is important for applications of intelligence robots such as cleaning robots, pet robots, tour guide robots, etc., which must work among humans or objects at close distances. On the other hand, even though obtaining the distance and orientation of the circular landmarks on the ceiling can be easily realized with a traditional perspective camera [8][9], a well-designed single omni-camera system may be used to replace several standard cameras so far as the image taking range is concerned.
In this study, a location estimation method for indoor autonomous vehicle guidance using omni-images of circular landmarks on ceilings is proposed. Analysis of circular shapes in omni-images is not well studied so far. It is found in this study
19
that a circular shape, which becomes an irregular shape in an omni-image with no known shape descriptor, can be well approximated analytically by an elliptical shape.
Consequently, it is appropriate to guide a vehicle equipped with an upward-looking omni-camera using a circular shape attached on a ceiling as a landmark, as is done in this study. Several merits can be identified in this approach, including: (1) the circular-shaped landmark attached on the ceiling is identically observable from every direction; (2) the circular shape, being elliptical when imaged, is easier to detect in low-resolution omni-images; (3) the elliptical shape provides more precise parameters for location estimation; (4) the elliptical shape does not get mixed up easily with other shapes found in the environment. Owing to these merits, stable and precise relative vehicle location estimation can be achieved for navigation. An illustration of the experimental navigation environment for this study, including a vehicle, a ceiling, and a landmark, is shown in Fig. 2.1.
In the proposed method, at first an upward-looking omni-camera on a vehicle is used to take an image of a circular-shaped landmark attached on the ceiling of an indoor space. An ellipse detection algorithm [17] is applied next to detect the projected shape of the landmark in the image. The irregular shape formed from the circular shape in the omni-image is approximated by an elliptical shape. The location of the landmark, including its distance and orientation, with respect to the camera on the vehicle are then derived analytically in terms of the major axis length and the center coordinates of the approximating ellipse. Finally, the move distance and the orientation change of the vehicle between two consecutive observations of the landmark are derived, which are useful for a number of autonomous vehicle applications.
The remainder of this chapter is organized as follows. In Section 2.2, we derive the analytic equations for approximating as an elliptical shape the irregular shape in
20
an omni-image taken of a circular-shaped landmark. In Section 2.3, we describe how we estimate the vehicle location from the acquired image using the derived analytic equation, and show an application of the results to autonomous vehicle guidance. In Section 2.4, some experimental results are described to show the precision and feasibility of the proposed method. Finally, some conclusions are given in Section 2.5.
2.2 Approximation of Irregular Shape in Omni-image Taken of Circular-shaped Landmark by Ellipse
The circular shape attached on the ceiling of the vehicle navigation environment for use as a landmark becomes irregular with no mathematical shape descriptor in an omni-image taken with a hyperboloidal omni-camera. We can approximate the Fig. 2.1 Relative positions of camera, ceiling, and circular landmark for providing
sufficient field of view and avoiding unexpected objects and humans appearing in acquired images.
Camera 1.0m~6.5m
Ceiling Circular
0.5~1.5m
21
irregular shape well by an ellipse, as mentioned previously, and this fact will be proved here. Specifically, an equation of the approximating elliptical shape in the image will be derived. The validity of this ellipse approximation will become clear in the derivation. The precision of the approximation will also be proved by some experimental results.
In Section 2.2.1, the projection transformation between the camera coordinate system and the image coordinate system will be described first. Then, the coordinate systems will be rotated horizontally to derive the equation of an ellipse in the image.
In Section 2.2.2 a simulated shape of the circular landmark computed with the derived equation will be compared with the shape obtained by an imaging projection based on [5] to show the effectiveness of the proposed elliptical shape approximation.
2.2.1. Approximation of Distorted Circular Shapes in Omni-images by Ellipses
The camera and image coordinate systems involved in this study using a hyperboloidal omni-camera are depicted in Fig. 2.2, with their coordinates specified by (X, Y, Z), and (u, v), respectively. The hyperbolic shape of the omni-directional mirror in the camera coordinate system may be described as:
2 1 relationship between the image coordinates (u, v) and the camera coordinates (X, Y, Z) can be described as follows [5][34][35][36]:
22
where f is the focal length of the camera.
In Fig. 2.2, let the circular-shaped landmark and its center and radius be denoted by W, Pw and Rw, respectively. And let the image of W in the hyperboloidal image be denoted by Q. Also, let (Xw, Yw, Zw) denote the camera coordinates of Pw. In this study, the normal vector of the landmark is assumed to be parallel to the optical axis of the camera. To simplify the derivation described later, we rotate, as shown in Fig. 2.3, horizontally the camera coordinate system and the image coordinate system through an angle of θw defined by:
Then, the relation between the original camera coordinates (X, Y, Z) and the resulting ones (X′, Y′, Z′) may be described by:
X′ = Xcosθw + Ysinθw, Y′ = Ycosθw − Xsinθw, Z′ = Z (2.4) and the relation between the original image coordinates (u, v) and the resulting ones
(u′, v′) may be described by:
u′ = ucosθw + vsinθw, v′ = usinθw − vcosθw. (2.5) Also, after this rotation, the circular shape of W in the new camera coordinate system
may be expressed by:
(X′ − Xw′)2 + (Y′ − Yw′)2 = Rw2, Z′ = Zw′,
23
Fig. 2:2 Coordinate systems involved in this study.
(X′ − Xw′)2 + (Y′ − Yw′)2 = Rw2, Z′ = Zw′,
given that the center point Pw of W is located at (Xw′, Yw′, Zw′) with Xw′ = Xwcosθw + Ywsinθw, Yw′ = Xwsinθw − Ywcosθw, Zw′ = Zw according to (2.4). Notice that Yw′ is now zero after the rotation according to Fig. 2.3, so that the above equation becomes
(X′ − Xw′)2 + Y′2 = Rw2, Z′ = Zw′. (2.6) Also, by the optical geometry of the camera described by (2.2), we have
Z'
=Y' u'
v' , (2.7)
or equivalently,
24 X'Y'
= u' '
v . (2.8)
Eq. (2.8) will be used in Section 2.3.1 later.
We are now ready to prove the previously-mentioned fact that the irregular shape of the circular landmark W appearing in the omni-image may be well approximated by an ellipse. After horizontally rotating the camera and the image coordinate systems for the angle of θw = tan−1(Yw/Xw) described by Eq. (2.3), Eq. (2.2) Fig 2.3 Top view from the Z direction showing the relationship between new and
original coordinate system with the new image coordinate system (u′, v′) obtained by rotating the u-axis through an angle of θw = tan−1(Yw/Xw) with respect to the center of the circular-shaped landmark W.
25
In addition, the new Y-coordinates Yw' of the landmark circle center is 0. Then, by assuming that the horizontal distance from the origin of the camera coordinate system to the landmark is much larger than the radius of the landmark, we have Y' <<
X' and the circle (X' − Xw')2 + (Y' − Yw')2 = Rw2 with Yw' = 0 may be regarded relatively as a point which is its center located at (Xw', Yw') so that X'2 + Y'2 ≈ Xw'2 + Yw'2 = Xw'2. Also, Z' is a constant (denoted as hw now) because the ceiling on which the landmark is attached is assume to be parallel to the camera coordinate system. As a consequence, the second Eq. in (2.2A) above for v' can be simplified to
2 2
The other coordinate u′ of each shape pixel of the landmark W may also be derived by approximation, but in a different way. Under the same assumption mentioned above that the radius of the landmark is relatively very small with respect to the horizontal distance from the origin of the camera coordinate system to the landmark, the magnitude of X′ of each shape pixel of the circular landmark W in the camera coordinate system is much larger than that of Y′. Therefore, we may neglect the influence of the magnitude of Y′ in the computation of u′ described by the first equation in (2.2A) so that
26
and compute u' just in terms of X'. Regarding the above equation in the form u' = F(X′), we may use the Taylor series to expand the function around Xw' as
u' = F(X′) = F(Xw′) + [(X′ − Xw′)/1!]F'(Xw′) + [(X′ − Xw′)2/2!]F''(Xw′) + ….
Ignoring the terms after the second, we have
u' ≈ F(Xw′) + [(X′ − Xw′)/1!]F'(Xw′) = uw' + (X′ − Xw′)F'(Xw′). (2.10) Eq. (2.10) may be transformed easily into
X′ ≈ Xw′ + (u′ − uw′)/F'(Xw′) (2.11) with the first derivative F' calculated to be:
F'(Xw′) = 1 2
Now with X' and Y' available, we come to the final stage of the derivation of the equation of the ellipse for approximating the distorted circular shape of the landmark in the omni-image. By substituting Eqs. (2.9) and (2.11) into Eq. (2.6) and rearranging the result, we can get
27
which obviously specifies exactly an elliptical shape centered at (uw′, 0) with the lengths of the major and minor axes being RwF'(Xw′) and Rw/M, respectively. This completes the proof.
2.2.2. Effectiveness of Shape Approximation
To check the effectiveness of the approximation of the circular shape of the landmark by the elliptical shape using Eq. (2.13), we show in Fig. 2.4 an example of the simulation results obtained in this study, in which both the original circular shape and the approximating elliptical one are drawn and superimposed on each other for comparison: the former shape being drawn by Eq. (2.6) and then projected into the image plane by Eq. (2.2), and the latter being drawn directly by Eq. (2.13). The outer big circle in the figure marks the field of view of the camera. Inside the big circle, the original distorted circular shapes of W at different positions are drawn with white pixels, and the approximate elliptical shapes are computed using the coordinates of the white pixels and drawn by black pixels.
From the figure, we can see that each black ellipse overlaps the corresponding white distorted circle quite well. This shows that the distorted circular shape of the landmark in the omni-image indeed may be approximated by the ellipse described by Eq. (2.13). This discovery offers great helps, as found in this study, in utilizing this kind of circular landmark to provide the location information for vehicle guidance, as described in the following section.
28
(a)
(b)
Fig. 2.4 Simulation of a series of circular shapes of the landmark at different places, showing that the distorted landmark shape may be approximated well by ellipses. (a) illustration of the simulation results. (b) partially enlarged view of (a).
2.3 Vehicle Location Estimation
In this section we describe the proposed method for vehicle location estimation using the detected landmark as a known reference point. From each acquired image of the ceiling, we extract the circular landmark shape by image processing techniques,
29
including thresholding, edge detection, and ellipse detection. For thresholding, since the artificially-made circular-shaped landmark contrasts well with the background in the image, a threshold value is selected to segment the distorted circular shape of the landmark in the image. And for edge detection in the image, the Sobel edge detector is applied. Since the landmark shape in the image can be regarded as an ellipse according to Section 2.2, an ellipse detection algorithm [17] is employed to extract the landmark, yielding an approximating ellipse with its axis lengths computed.
2.3.1. Vehicle Location Estimation by Axis Lengths of Ellipse
We will derive here the location of the landmark, including its distance and orientation, with respect to the camera coordinate system on the vehicle for use in vehicle guidance. We derive first the horizontal distance of the landmark in terms of the lengths of the two axes of the approximating ellipse. As depicted in Fig 2.3, the X′
axis is directed to the landmark center Pw′. For a better view of the involved situation, Fig.2.3 is redrawn as Fig. 2.5 in which d is the horizontal distance between the camera and the center of the landmark W to be derived; Pα′Pβ′ is a diameter of W are the coordinates of Pα′ in the (X′, Y′) coordinate system. Similarly, we have
'
30
Fig. 2.5 Top view from the Z direction illustrating the relation between the axes of the approximating ellipse and the horizontal distance of the circular-shaped landmark.
are the coordinates of Pβ′ in the (X′, Y′) coordinate system. Since Pα′Pβ′ is the diameter of W perpendicular to the X′ axis, we have Xα′ = Xβ′ and so uα′ = uβ′.
Therefore, Eqs. (2.14) and (2.15) can be merged and rearranged to result in v ' v ' Y ' Y ' of the approximating ellipse in the image plane; and uα' is the u' coordinate of the center of the ellipse. The latter two parameters I'αI'β and uα' can be computed right after the approximating ellipse is obtained. Also, as seen from the figure, Xα' is just
31
the desired relative horizontal distance d between the camera and the center of the landmark W, or equivalently, the desired horizontal distance d of the landmark with respect to the vehicle. Therefore, we can derive from Eq. (2.16) the desired value of d as
Furthermore, using the ellipse makes it easier to solve the orientation θw of the landmark with respect to the camera coordinate system, or equivalently, with respective to the vehicle. Given that the center of the found ellipse in the image is located at (uw, vw), from Eq. (2.3) and the following equation derived from Equation (2.7),
A merit of the above procedure is that there is no need of the vertical height value Zw of the landmark with respect to the camera coordinate system in computing the distance d and the orientation θw of the landmark. This provides an advantage of allowing dynamic changes of the camera height for the purpose of avoiding camera view occlusion by surrounding people or objects. Taking this advantage, we have included a shaft in the vehicle system for adjusting the height of the camera dynamically: if the landmark shape cannot be well extracted from the image taken at a certain camera height to yield an approximating ellipse, then the camera is lifted up automatically and gradually until a good result is obtained.
32
Another merit of vehicle location estimation by Eqs. (2.17) and (2.18) is that the exact hyperbolic mirror shape of the omni-camera, as described in Eq. (2.1), need not be considered. This reduces the estimation error caused by using the possibly imprecise shape parameters of the mirror obtained from calibration, and simplifies the computation process involved in the estimation. Also, it provides a nature of generality of the proposed location estimation process using a given type of camera, so that it is unnecessary to re-design the location estimation process when a camera with a different hyperbolic mirror shape is used.
2.3.2. Estimation of Vehicle Moving Distances and Orientation Changes
In the previous section, we show how to estimate the distance and orientation of the landmark with respect to the vehicle. In this section we show how to find out, as an application of the previous results, the relative distance and orientation of the vehicle in a navigation cycle with respect to its location in the previous cycle, which we call the move distance and orientation change of the vehicle, respectively.
Derivations of these parameters are useful for a number of autonomous vehicle applications, including recording of navigation paths, path planning for vehicle guidance, measurement of guidance errors, etc.
As shown in Figs. 2.6(a) and 2.6(b), we denote the move distance of the vehicle between two consecutive observation times T1 and T2 as D. Using the approximating ellipse, we can, according to Section 2.3.1, calculate the horizontal distances d1, d2
and the orientations θ1, θ2 of the landmark at T1 and T2, respectively, with respect to the vehicle. Based on the cosine theorem, we have
d12 + D2 − 2d1×D×cosθ1 = d22,
33
which may be solved to get the desired move distance D as
2 Next, to find the desired orientation change of the vehicle, which we denote as γ,
caused by a vehicle turning between the two consecutive observations, we acquire first, using Eq. (2.18), the orientations of the landmark with respect to the vehicle before and after the vehicle turning, denoted as θ2 and φ, respectively, as depicted in Figs. 2.6 (c) and 2.6(d). Then, γ can be computed easily by
γ = θ2 − φ. (2.21)
2.4 Experimental Results
The effectiveness of the proposed location estimation method has been tested by some experiments conducted in this study, which include two parts: (1) using computer simulations to test if the circular shape of the landmark in the acquired images taken with omni-cameras with different shapes of hyperboloidal mirrors can be detected by the proposed ellipse approximation method; (2) using real images to determine the precision of the estimated vehicle locations relative to the landmark.
(A) Simulations
In the first experiment of landmark shape approximation by ellipses, the first step was to create landmark shapes at different locations. A series of virtual circular-shaped landmarks with a radius of 20cm were created and projected onto the image plane, in which the centers of these landmarks are located in a range of 300cm in intervals of 50cm with the orientation angles θ ranging from 0o to 360o in intervals of 5o. Two kinds of common distortion, the barrel and the pincushion distortion as
34
described in [18][19], were added to make the simulation more realistic. In our simulations, the original image coordinates (u, v) are inherently normal and distortion-free, as described by Eq. (2.2).
D
Fig. 2.6 Illustration of the relative ALV location estimation. (a) and (b): the displacement D of the vehicle. (c) and (d): the orientation of the vehicle.
Then, we add barrel and pincushion distortion to them by the following equations: control the shape of the geometric distortion. The coordinates (u(, v() then are taken as input to our method.
35
Second, an ellipse detection algorithm [17] was used to detect these landmarks.
Third, a rate of successful ellipse detections was computed. And at last, the three steps were repeated to compute the rates of successful ellipse detections for different kinds of omni-cameras which were created virtually only by changing the parameters of their hyperboloidal mirrors. More specifically, we changed only the mirror parameter b in (1) while maintaining the value of c invariant (= 20 in our simulations) since the position and the focus of the cameras need not be changed. The
smallest value of b was taken to be 146mm since otherwise the landmark image will exceed the range of the image plane. And the largest value of b was taken to be
smallest value of b was taken to be 146mm since otherwise the landmark image will exceed the range of the image plane. And the largest value of b was taken to be