Chapter 1 Introduction
1.5 Thesis Organization
The remainder of this thesis is organized as follows. In Chapter 2, we introduce the configuration of the proposed system and the system process in detail. In Chapter 3, the proposed training methods for vehicle to learn the guidance parameters and the navigation path are described. In Chapter 4, we introduce the navigation strategies including the ideas, the proposed guidance techniques, and detailed navigation algorithms. In Chapter 5, a new space line technique is proposed and the proposed techniques of hydrant and light pole detections are described. In Chapter 6, the navigation techniques of line following and obstacle avoidance are introduced. In Chapter 7, some experimental results to show the feasibility of the proposed techniques for vehicle navigation are shown. At last, conclusions and some suggestions for future works are given in Chapter 8.
Chapter 2
System Design and Processes
2.1 Idea of System Design
For a blind person to walk safely on a sidewalk, a vision-based autonomous vehicle system is a good substitute for a guide dog, as mentioned previously. Because of the advantages of possessing good mobility and long-time navigation capabilities, autonomous vehicles have become more and more popular in recent years for many applications. Equipped with cameras, an autonomous vehicle is able to “see” like a human being. Moreover, both the autonomous vehicle and the cameras may be connected to a main control system, which have the capabilities to integrate information, analyze data, and make decisions. In this study, we have designed an autonomous vehicle system of this kind for use as a machine guide dog. The entire configuration of the system will be introduced in detail in Section 2.2, and 3D data acquisition using the camera will be described in Section 2.3.
In an unknown environment, the autonomous vehicle system still has to be
“trained” before it can navigate by itself. Specifically, it should be “taught” to know the information of the navigation path; how to navigate in this path; and how to handle different conditions on the way. Moreover, secure navigation strategies should also be established for the vehicle to protect the blind and itself. In the end, the vehicle should be able to navigate in the same path repetitively with the learned data and the navigation strategies. The system processes designed to achieve these functions on the proposed autonomous vehicle system will be described in Section 2.3,
including the learning process described in Section 2.3.1 and the navigation process in Section 2.3.2.
2.2 System Configuration
In this section, we will introduce the configuration of the proposed system. We use Pioneer 3, an intelligent mobile vehicle made by MobileRobots Inc. as shown in Figure 2.1, as a test bed for this study. The autonomous vehicle and other associated hardware devices will be introduced in more detail in Section 2.2.1. In addition, a particularly-designed stereo omni-camera is employed in this study and equipped on the autonomous vehicle. We will describe the structure of the camera system in Section 2.2.2. Finally, the configuration of the software we use as the development tool will be introduced in Section 2.2.3.
Figure. 2.1 Autonomous vehicle, Pioneer 3 produced by MobileRobots Inc., used in this study.
2.2.1 Hardware configuration
The hardware architecture of the proposed autonomous vehicle system can be divided into three parts. The first is the vehicle system; the second the camera system,
and the third the control system. The latter two are installed on the first, the vehicle system, as shown in Figure 2.2. We will introduce these systems one by one subsequently.
(b)
(a)
(c)
Figure. 2.2 Three different views of the used autonomous vehicle, which includes a vehicle, a stereo camera, and a notebook PC for use as the control unit. (a) A 45o view. (b) A front view.
(c) A side view.
The vehicle, Pioneer 3, has an aluminum body with the size of 44cm×38cm×
22cm, two wheels of the same diameter of 16.5cm, and one caster. Also, 16 ultrasonic sensors are installed on the vehicle, half of them in front of the body and the other half behind. When navigating on flat floors, Pioneer 3 can reach its maximum speed 1.6 meters per second. Also, it has the maximum rotation speed of 300 degrees per second, and can climb up a ramp with the largest slope of 30 degrees. The vehicle is able to
carry payloads up to 23kg at a slower speed. It has three 12V rechargeable lead-acid batteries and can run constantly for 18 to 24 hours if all of the batteries are fully charged initially. The vehicle also provides the user some parameter information of the system, such as the vehicle speed, the battery voltage, etc.
The camera system, called a two-mirror omni-camera, consists of one perspective camera, one lens, and two reflective mirrors of different sizes. The perspective camera, ARCAM-200SO, is produced by the ARTRAY company with a size of 33mm×33mm×50mm and the maximum resolution of 2.0M pixels. With the maximum resolution, the frame rate can reach 8 fps. The CMOS visual sensor in the camera has a size of 1/2 inches (33mm×33mm). The lens is produced by Sakai Co.
and has a variable focal length of 6-15mm. The two reflective mirrors are produced by Micro-Star International Co. A detailed view of the entire camera system is shown in Figure. 2.3, and the camera and the lens are shown in Figures 2.3(a) and 2.4(b), respectively. Other details about the camera structure will be described in the next section.
(a) (b) Figure. 2.3 The two-mirror omni-camera used in this study. (a) A full view of the camera equipped
on the vehicle. (b) A closer view.
About the final part, we use a laptop computer as the control system. It is of the model of ASUS W7J produced by ASUSTek Computer Inc. as shown in Figure. 2.5.
For the computer to communicate with the other parts, we connected it with the autonomous vehicle by an RS-232, and with the camera system by a USB.
(a) (b) Figure. 2.4 The used camera and lens. (a) The camera of model Arcam 200so produced by ARTRAY
Co. (b) The lens produced by Sakai Co.
Figure. 2.5 The laptop computer of model ASUS W7J used in this study.
2.2.2 Structure of used two-mirror omni-camera
The structure of the two-mirror omni-camera used in this study and the projection of a space point P onto the system are illustrated in Figure 2.6. We call the
bigger and higher mirror Mirror A, and the smaller and lower one Mirror B subsequently, and both of them are made to be of the shape of a hyperboloid with parameters shown in Table 2.1. One of the focal points of Mirror A is on fa and one of the focal points of Mirror B is on fb. Both mirrors have another focal point on the same position fc which is the center of the lens. In addition, the line segment , which we call the baseline, has a length in 20cm.
f fa b
Table 2.1 Specifications of the used two hyperboloidal-shaped mirrors.
radius Parameter a Parameter b
Mirror A 12 cm 11.46cm 9.68cm
Mirror B 2cm 2.41cm 4.38cm
An important optical property of the hyperboloidal-shaped mirror is: if a light ray goes through one focal point, it must be reflected by the mirror to the other focal point.
As illustrated in Figure 2.7, two light rays which go through fa and fa are both reflected to the same focal point fc by the specially-designed mirrors, Mirrors A and B, respectively. Based on these property of the omni-camera and as illustrated in Figure 2.6, a space point P will be projected onto two different positions in the CMOS sensor in the camera along the blue light ray and the red light ray reflected by Mirrors A and B, respectively, so that the range data of P can be computed according to the two resulting distinct image points.
Furthermore, the way of placement of the two-mirror omni-camera on the vehicle has been carefully considered. The camera was originally placed in such a way that the optical axis going through Mirrors A and B is parallel to the ground as shown in Figure 2.8(a). In the omni-image acquired by the two-mirror omni-camera,
because of the existence of the image region caused by Mirror B, the image region of the field of view (FOV) reflected by Mirror A became smaller. We can see that the overlapping region on the ground where range data can be computed was not large enough in this situation (as shown in the green dotted region in the figure). However, for a navigation system, the front FOV is very important for the vehicle to avoid collisions. Moreover, it is desired that the vehicle can find obstacles at distances as large as possible in the navigation process. Due to these reasons, the camera was later slanted up for an angle of as shown in Figure 2.8(b). We can see in the figure that the region of overlapping is now bigger than before.
Figure 2.6 The prototype of the two-mirror omni-camera and a space point projected on the CMOS sensor of the camera.
Figure 2.7 The reflection property of the two hyperboloidal-shaped mirrors in the camera system.
(a)
Mirror A
Mirror B
(b)
Figure 2.8 Two different placements of the two-mirror omni-camera on the vehicle and the region of overlapping. (a) The optical axis going through the two mirrors is parallel to the ground. (b) The optical axis through the two mirrors is slanted up for an angle of .
2.2.3 Software configuration
The producer, MobileRobots Inc., of the autonomous vehicle used in this study provides an application interface, called ARIA (Advanced Robotics Interface Application), for the user to control the vehicle. The ARIA is an object-oriented
interface which can be used under the Win32 or Linux operating system using the C++
language. Therefore, we can utilize the ARIA to communicate with the embedded sensor system in the vehicle and obtain the vehicle state to control the pose of the vehicle.
For the camera system, the ARTAY provides a development tool called Capture Module Software Developer Kit (SDK). This SDK is an object-oriented interface and its application interface is written in several computer languages like C, C++, VB.net, C#.net and Delphi. We use the SDK to capture image frames with the camera and can change many parameters of the camera, such as the exposure. In the control system, we use Borland C++ Builder 6, which is a GUI-based interface development environment, to develop our system processes on the Windows XP operating system.
2.3 3D Data Acquisition by the Two–mirror Omni-camera
2.3.1 Review of imaging principle of the two-mirror omni-camera
Before derivation of the formulas for range data computation by the use of the two-mirror omni-camera, we review first the imaging principle of a simple omni-camera consisting of a hyperboloidal-shaped mirror and a projective camera.
First of all, we introduce two coordinate systems as shown in Figure 2.9 where the image coordinate system (ICS) is a two-dimensional coordinate system coincident with the omni-image plane with its origin being the center of the omni-image. The camera coordinate system (CCS) in Figure 2.9 is a three-dimensional coordinate system with the origin being located at a focal point of the mirror.
According to the optical property of the hyperboloidal-shaped mirror, a space point P at coordinates (x, y, z) in the CCS is projected onto an omni-image point I at coordinates (u, v) in the ICS as shown in Figure 2.9. In more detail, assume that a light ray from P goes through a focal point in the mirror center Om. Then, reflected by the hyperboloidal-shaped mirror, the light ray goes through another focal point at the lens center Oc. It is finally projected onto an image point I at coordinates (u, v) on the image plane. In this way of imaging space points, therefore, for each image point I, we can find a corresponding light ray with a specific elevation angle and a specific azimuth angle θ (shown in the figure by the red and the green characters, respectively) to represent the image point I.
Figure 2.9 Imaging principle of a space point P using an omni-camera.
2.3.2 3D data computation for used two-mirror omni-camera
Before deriving the 3D data from the omni-image acquired by the two-mirror
omni-camera, we define a camera coordinate system (CCS) as shown in Figure 2.10. The origin of is the focal point of Mirror A, and the Z-axis coincident with the optical axis going through the two mirrors. As shown in the figure, there is a space point Q at coordinates (X, Y, Z) in CCS which is projected respectively by the two mirrors onto two image points, Is at coordinates (u1, v1) and Ib
at coordinates (u2, v2), in the omni-image. By the geometry of the camera oprtics, we may compute the 3D position of Q by the following way.
local
Figure 2.10 The cameras coordinate system , and a space point Q projected on the omni-image acquired by the two-mirror omni-camera.
local
CCS
Firstly, following the two light rays which go through Mirror A’s center and Mirror B’s center, respectively, we obtain two different elevation angles a and b as shown in Figure 2.11(a). Also, the points Oa,Ob, and Q form a triangle OaObQ which we especially illustrate in Figure 2.11(b). The distance between Oa and Ob, which is the length of the baseline defined previously, is known to b, while the distance d between Oa and Q is an unknown parameter. By the law of sines based on geometry, we can compute d as follows:
sin(90o b) sin( b) Figure 2.11 An illustration of the relation between a space point Q and the two mirrors in the used
Secondly, we may compute the azimuth angles of the two light rays. According to the
s
1 1
camera. (a) A side view of Q projected onto the two mirrors. (b) A triangle OaObQ used in deriving 3D data.
property of rotational invariance of the omni-image, these two azimuth angles actually are equal, which we denote by . From Figure 2.12, by the use of the image point I at coordinates (u , v ), we can derive θ by the following equations:
1 1
Thirdly, with the distance d derived by Equation (2.2) and the azimuth angle θ obtained by Equation (2.3), we can compute the position of Q in according to geometry illustrated in Figure 2.12 as follows:
local
CCS
X ＝ d × cosαa × sinθ,
Y ＝ d × cosαa × cosθ,
Z ＝ d × sinαa. (2.4)
Figure 2.12 An illustration of a space point Q at coordinates (X, Y, Z) in CCSlocal.
However, as mentioned previously, the optical axis going through the two mirrors is slanted up so that it is not parallel to the ground. It is desired that the Z-axis of could be parallel to the ground. As shown in Figure 2.13, we define another camera coordinate system CCS, which coincidences w Slocal except that the Z-axis is slanted for an angle of toward the Y-axis along the X-axis. Finally, the coordinates of Q is translated to a new coordinates (X’, Y’, Z’), which we want to obtain, in the CCS by the use of a rotation matrix R by following equations:
local
CCS
ith CC
1 0 0
Figure 2.13 The relation between the two camera coordinate systems CCS and CCSlocal.
2.4 System Processes
2.4.1 Learning process
The goal in the learning process is to “teach” the autonomous vehicle to know how to navigate automatically in a pre-specified path. The entire learning process proposed in this study is shown in Figure 2.14. Discussed in the following is some information which the vehicle should “memorize.” First, as mentioned in Chapter 1, the autonomous vehicle has to conduct self-localization by some pre-selected landmarks in the specified path, so the first type of information the vehicle have to record is the landmark locations along the path. Next, for our study, the experimental
environment is on the sidewalk, and this has an advantage that the vehicle may navigate along the curb of the sidewalk. For this reason, line following along the sidewalk curb, called sidewalk following, is a proper navigation method instead of using the odometer only. Thus, the second type of information which has to be recorded is the vehicle navigation information along the path (path nodes, the navigation distance between two nodes, etc.). Finally, the environment information at different locations on the navigation path also has to be recorded.
For the purpose of training an autonomous vehicle easily, a user learning interface is constructed for the trainer and can be used to control the autonomous vehicle as well as construct learning navigation information. First, at the beginning of each section of the navigation path, the trainer should establish a set of corresponding navigation rules in advance, and the vehicle will follow them and conduct navigation in the learning process as well as in the navigation process. Then, the current vehicle pose obtained from the odometer and some current environment information like the illumination are also recorded. Next, when the mode, navigation by following the sidewalk, is selected, a semi-automatic learning process will proceed until reaching the next node assigned by the trainer. Otherwise, the trainer is required to guide the vehicle manually to the next path node by the use of the learning interface.
In addition, the trainer can decide where to localize the vehicle by a selected landmark in the learning process. After guiding the vehicle to a proper pose for detecting the landmark (close enough to the landmark, “looking” at the landmark from the right direction, etc.), the trainer then has to establish relevant rules for landmark detection. Some parameters for landmark detection can be appropriately adjusted by the trainer before the detection work is started. Next, landmark localization is conducted by a space line detection technique described in Chapter 5.
After possibly multiple times of detecting and collecting adequate information of the
landmark, its position is finally computed automatically and recorded.
At last, after bringing the autonomous vehicle to the destination, the learning process is finished, and the learned information is organized into a learned path composed of several path nodes with guidance parameters. Combining it with landmark information and environment information, we obtained an integrated path map which finally is stored in the memory of the vehicle navigation system.
2.4.2 Navigation process
With the map information obtained in the learned process, the autonomous vehicle can continually analyze the current location using various stored information and navigate to an assigned goal node on the learned path in the navigation process.
The entire navigation process proposed in this study is shown in Figure 2.15.
According to the learned information data retrieved from the storage, the autonomous vehicle continually analyzes the current environment node by node to navigate to the goal. At first, before starting to navigate to the next node, the autonomous vehicle checks if the image frame is too dark or too bright according to the learned environment parameter data, and then dynamically adjusts the exposure of the camera if necessary.
After that, the autonomous vehicle always checks if any obstacle exists in front of the vehicle. As soon as an obstacle is found and checked to be too close to the vehicle, a procedure of collision avoidance is started automatically to perform collision avoidance. Then, based on the learned navigation rules, the autonomous vehicle checks the corresponding navigation mode and follows it to navigate forward.
In the meantime, the vehicle checks whether it has arrived at the next node; whether the node is the destination; or whether the vehicle has to localize its current position.
Learning of landmarks
In addition, if a self-localization node is expected, the autonomous vehicle will adjust its pose and relevant parameters into an appropriate condition and conduct landmark detection. For landmark detection, the autonomous vehicle uses the corresponding technique in accordance with the property of the landmark. If a desired landmark is found and localized successfully, its location then is used to modify the position of the vehicle; if not, some remedy for recovering the landmark
In addition, if a self-localization node is expected, the autonomous vehicle will adjust its pose and relevant parameters into an appropriate condition and conduct landmark detection. For landmark detection, the autonomous vehicle uses the corresponding technique in accordance with the property of the landmark. If a desired landmark is found and localized successfully, its location then is used to modify the position of the vehicle; if not, some remedy for recovering the landmark