Chapter 2 System Design and Processes
2.4 System Processes
2.4.2 Navigation process
With the map information obtained in the learned process, the autonomous vehicle can continually analyze the current location using various stored information and navigate to an assigned goal node on the learned path in the navigation process.
The entire navigation process proposed in this study is shown in Figure 2.15.
According to the learned information data retrieved from the storage, the autonomous vehicle continually analyzes the current environment node by node to navigate to the goal. At first, before starting to navigate to the next node, the autonomous vehicle checks if the image frame is too dark or too bright according to the learned environment parameter data, and then dynamically adjusts the exposure of the camera if necessary.
After that, the autonomous vehicle always checks if any obstacle exists in front of the vehicle. As soon as an obstacle is found and checked to be too close to the vehicle, a procedure of collision avoidance is started automatically to perform collision avoidance. Then, based on the learned navigation rules, the autonomous vehicle checks the corresponding navigation mode and follows it to navigate forward.
In the meantime, the vehicle checks whether it has arrived at the next node; whether the node is the destination; or whether the vehicle has to localize its current position.
Learning of landmarks
In addition, if a self-localization node is expected, the autonomous vehicle will adjust its pose and relevant parameters into an appropriate condition and conduct landmark detection. For landmark detection, the autonomous vehicle uses the corresponding technique in accordance with the property of the landmark. If a desired landmark is found and localized successfully, its location then is used to modify the position of the vehicle; if not, some remedy for recovering the landmark will be conducted, such as changing the pose of the vehicle to detect the landmark again.
Figure 2.15 Navigation process.
Chapter 3
3.1.1
3.1.2
Learning Guidance Parameters and Navigation Paths
3.1 Introduction
Before the autonomous vehicle can navigate, some works has to be conducted in the learning process. First, the camera system should be calibrated. Then, a path and a set of landmarks should be selected, and each landmark location should be recorded into the path, resulting in a learned path. Finally, adopted guidance parameters have to be “trained” and then recorded.
Camera calibration
As mentioned in Chapter 1, instead of calibrating the camera’s intrinsic and extrinsic parameters, we adopt a space-mapping technique [25], called pano-mapping, to calibrate the two-mirror omni-camera used in this study. We will describe the adopted technique in Section 3.2.
Selection of landmarks for navigation guidance
For the purpose to localize the position of the vehicle during the navigation process, some objects should be selected as landmarks to conduct vehicle localization.
Two types of objects, hydrant and light pole, as shown in Fig 3.1 are selected in this study as landmarks for vehicle localization during sidewalk navigation. By the use of proposed hydrant and light pole localization techniques, introduced later in Chapter 5,
we can guide the vehicle to learn the positions of pre-selected hydrants and light poles in the learning process.
Figure 3.1 The hydrant (left) and the light pole (right) used as landmarks in this study.
3.1.3 Learning of guidance parameters
For complicated outdoor environments, the trainer should train some parameters for use in vehicle guidance, such as environment parameters and image segmentation thresholds, in the learning process. We will introduce the proposed techniques for learning environment parameters in Sections 3.4. Also, some image segmentation parameters for landmark image analysis and the techniques proposed to learn them will be introduced in Sections 3.5. Finally, a scheme proposed to create the learned navigation path will be described in Section 3.6.
3.2 Camera Calibration by Space-mapping Approach
We utilize the pano-mapping technique proposed by Jeng and Tsai [25] for image unwarping to calibrate the camera system used in this study. The main idea is to establish a so-called pano-mapping table to record the relation between image points and corresponding real-world points. More specifically, as illustrated in Figure 3.2, a
light ray going through a world-space point P with the elevation angle α and the azimuth angle θ is projected onto a specific point p at coordinates (u, v) in the omni-image. The pano-mapping table specifies the relation between the coordinates (u, v) of the image point p and the azimuth-elevation angle pair ( of the corresponding world-space point P. The table is established in advance and can be looked up to retrieve 3D information forever. Accordingly, we construct two pano-mapping tables for Mirrors A and B, respectively, by the following steps, assuming an omni-image I has been taken as the input.
Algorithm 3.1 Construction of pano-mapping tables.
Step 1. Manually select in advance six known image points pi at coordinates (ui, vi,) on the Mirror A region in omni-image I and the six corresponding known world-space points Pi at coordinates (xi, yi, zi), where i is 1 through 6.
Step 2. Select similarly six known image points qj at coordinates (Uj, Vj,) on the Mirror B region in omni-image I and the six corresponding known world-space points Qj at coordinates (Xj, Yj, Zj), where j is 1 through 6.
Step 3. For image points pi and qj, compute the radial distances ri and Rj in the image plane with respect to the image center respectively by the following equations:
2 2; 2 2
i i i i i i
r u v R U V . (3.1)
Step 4. Compute the elevation angles αi and βi for the corresponding world-space points Pj and Qj by the following equations:
resulting in six pairs of radial distances and corresponding elevation angles
for Mirrors A and B, respectively.
Step 5. Under the assumption that the surface geometries of Mirrors A and B are radially symmetric in the range of 360 degrees, use two radial stretching functions, denoted as fA and fB, to describe the relationship between the radial distances ri and the elevation angles αi as well as that between Rj and βj, respectively, by the following equations:
1 2 3 4 six radial-distance pairs for Mirrors A and B, respectively, obtained in Step 4 using a numerical method to obtain the coefficients a0 through a5 and b0
through b5.
Step 7. By the use of the function fA with the known coefficients a0 through a5, construct the pano-mapping table for Mirror A in a form as that shown in Figure 3.3(a) according to the following rule:
for each world-space point Pij with the azimuth-elevation pair (θi, αj), compute the corresponding image coordinates (uij, vij) by the following equations:
cos ; sin
ij j i ij j i
u r v r . (3.4)
Step 8. In a similar way, construct the pano-mapping table for Mirror B by the use of the function fB with the known coefficients b0 through b5 in a form as that shown in Figure 3.3(b).
Figure 3.2 The relation between a space point P and the relevant elevation angle and azimuth.
Figure 3.3 Two pano-mapping tables used for the two-mirror omni-camera used in this study. (a) Pano-mapping table used for Mirror A. (b) Pano-mapping used for Mirror B.
3.3 Coordinate Systems
In this section, we will introduce the coordinate systems used in this study, which describe the relations between the used devices and concerned landmarks in the navigation environment. Furthermore, the used odometer and some involved coordinate transformations are introduced also. The following are four coordinate systems used in this study.
(1). Image coordinate system (ICS): denoted as (u, v). The u-v plane coincides with
(2). Vehicle coordinate system (VCS): denoted as (VX, VY). The VX-VY plane coincides with the ground and the origin OV of the VCS is located at the center of the autonomous vehicle.
(3). Global coordinate system (GCS): denoted as (MX, MY). The MX-MY plane coincides with the ground. The origin OG of this system is always placed at the start position of the vehicle in the navigation path.
(4). Camera coordinated system (CCS): denoted as (X, Y, Z). The origin OC of the CCS is placed at the focal point of Mirror A. The X-Z plane is parallel to the ground and the Y-axis is perpendicular to the ground.
Navigation Environment
MX
OG MY
(a) (b) Figure 3.4 Two coordinate systems used in this study. (a) The ICS. (b) The GCS.
In this study, the navigation path is specified by the GCS as shown in Figure 3.4(b). The relationship between the GCS and the VCS is illustrated in Figure 3.5. At the beginning of the navigation, the VCS coincides with the GCS, and then the VCS follows the movement of the current vehicle position as well as the CCS. In addition, it is emphasized that the vehicle uses an odometer to localize its position in the GCS.
As illustrated in the figure, the reading of the vehicle odometer is denoted as (Px, Py, Pth) where Px and Py represent the current vehicle position with respect to its original
position on the ground, and Pth represents the rotation angle of the vehicle axis with respect to the GCS.
As shown in Figure 3.6, assume that the vehicle is at a position V at world coordinates (Cx, Cy) with a rotation angle θ. We can derive the coordinate transformation between the coordinates (MX, MY) of the VCS and the coordinates (VX, VY) of the GCS by the following equations:
cos sin
X X Y x
M V V C ;
sin cos
Y X Y y
M V V C . (3.5)
In addition, the relationship between the CCS and the VCS is illustrated in Figure 3.7. As shown in the figure, the projection of the origin of the CCS onto the ground does not coincident with the origin of the VCS, and there is a horizontal distance between the two origins, which we denote as Sy. Thus, the coordinate transformation between the CCS and the VCS can be derived in the following way:
X ;
V X VY Z Sy. (3.6)
jiFigure 3.5 An illustration of the relation between the GCS and the VCS.
Figure 3.6 A vehicle at coordinates (Cx, Cy) with a rotation angle θ with respect to the GCS.
Figure 3.7 An illustration of the relation between the GCS and the VCS.
3.4 Learning of Environment Parameters
3.4.1 Definition of environment windows on images
In the process of navigation, the vehicle conducts several works including following sidewalk curbs, finding landmarks, obstacle detection, etc. In general, each desired landmark is projected onto a specific region in the image. By this property, we can consider only the region of interest in the image instead of the whole image, and two advantages can be obtained from this approach as follows:
1. reducing computation time for each navigation cycle;
2. the environment appearing within the image region is similar in each navigation cycle so that we can analyze the environment information in this region with a fixed scheme.
To be more specific, an environment window, as we call hereafter, is predefined by the trainer, which specifies a rectangular region in the image. Two environment windows including a small one in the region of Mirror B and a big one in the region of Mirror A, denoted as winS and winB, respectively, are considered to form a set for use in conducting specific image analysis works when the vehicle is navigating. For instance, a set of environment windows is defined for the hydrant detection work, as shown is Figure 3.8 (the blue rectangles). Besides, it is pointed out that this scheme of defining a pair of search windows for use in this study follows the property of rotational invariance of the omni-image, which is useful to reduce the redundancy region where we cannot get relevant 3D information, as will be elaborated later in this section.
Figure 3.8 An example of a pair of environment windows for hydrant detection
3.4.2 Learning of environment intensity by environment windows
In outdoor environments, varying lighting conditions influence the results of our navigation environment analysis work, according to our experimental experience. For example, as shown in Figure 3.9(a), the feature of the curb is not obvious enough to be recognized because of the overexposure due to the lighting condition. Also, using a fixed value of the exposure, some landmark detection works cannot be completed successfully. For example, the hydrant and the light pole become undetectable in images because of the overexposure and underexposure, as shown the examples in Figures 3.10(a) and 3.11(a), respectively.
(a) (b) Figure 3.9 Two different illuminations in the image for curb detection and the environment windows. (a)
An instance of overexposure. (B) A suitable case.
(a) (b) Figure 3.10 Two different illuminations in the image for hydrant detection and the environment windows.
(a) An unclear case. (B) An appropriate case.
(a) (b) Figure 3.11 Two different illuminations in the image for light pole detection and the environment windows.
(a) A blurred case. (B) A proper case.
For this reason, in this study we design the system in such a way to allow the trainer in the learning phase to determine a suitable illumination parameter by manually adjusting the exposure of the camera for the purpose to detect desired objects successfully. By adjusting the illumination parameter to a suitable value, we mean that the desired landmark feature can be extracted well in the same illumination afterward. Then, this image illumination parameter is recorded into the path information as part of the learning result. To be more specific, for each environment analysis work using a landmark, we learn a value of suitable image intensity, called environment intensity hereafter, on the image in relevant environment windows during the path learning process. The detail of this scheme of getting proper environment intensity parameters is described in the following algorithm.
Algorithm 3.2 Learning of the environment intensity parameter at a path node.
Input: a relevant set of environment windows Winen for a certain path node with a pre-selected landmark under the assumption that the vehicle reaches this node currently.
Output: an environment intensity parameter Ien. Steps.
Step 1. Adjust the exposure of the camera and acquire an appropriate image Icur. Step 2. Check if the desired landmark feature is well imaged in the resulting
illumination. If not, go to Step 1.
Step 3. For each pixel Ii in Icur with color (R, G, B) in winB of Winen, calculate its
Some examples of suitable illuminations for navigation tasks are shown in Figures 3.9(b), 3.10(b), and 3.11(b), and the environment intensity parameters learned in the above way for them will be recorded as part of the learning result of landmark detection described later.
3.5 Learning of Landmark Segmentation Parameters
In this study, we utilize some segmentation methods for image analysis in landmark detection which we describe in detail later in Chapters 5 and 6. In this section, we introduce the process for learning the parameters used in landmark segmentation. Firstly, we introduce three sets of segmentation parameters for landmark segmentation which we propose for use in this study as follows.
(1) In sidewalk curb segmentation, we use the color information (hue and saturation) and the image thresholding technique to find the curb feature in the image utilizing the HSI color model. The thresholds for hue and saturation values are collected as a set of curb segmentation parameters.
(2) In hydrant segmentation, just like what we do in sidewalk curb detection, we use the HSI color model to extract the hydrant shape. The threshold values for hue and saturation and also the contour of the hydrant described by the principal components obtained from principal component analysis is collected as a set of hydrant segmentation parameters.
(3) In light pole segmentation, we adopt the Canny edge detection technique to extract the light pole shape. The threshold values used to detect the light pole in the image are collected as a set of light pole segmentation parameters.
Next, as shown in Figure 3.12, when conducting landmark learning, the trainer can detect a desired landmark by the use of a user interface of the system, and adjust the values of the related set of segmentation parameters. After obtaining a proper result from the landmark detection process, the used set of segmentation parameters and the learned landmark information are recoded together as part of the learned path.
Figure 3.12 The process for learning landmark segmentation parameters.
3.6 Learning Processes for Creating a Navigation Path
In this section, we introduce the proposed method for learning a navigation path in the learning process. As usual, we use the odometer to localize the vehicle position and estimate the position of a detected landmark in the learning process. The proposed strategy for learning landmarks for vehicle localization is introduced in Section 3.6.1.
In addition, in the navigation path, some obstacles on the sidewalk, which may not be recognized easily by the camera system, could also block the vehicle. An example of obstacles, a sewer cover with uprising handles, which might hinder autonomous vehicle navigation, is shown in Figure 3.13. Thus, we propose in this study a method to learn the positions of such obstacles, called fixed obstacles hereafter. The method is described in Section 3.6.2. Finally, the entire proposed procedure to learn a navigation path is described in Section 3.6.3.
Figure 3.13 A fixed obstacle in a navigation path which may block the autonomous vehicle.
3.6.1 Strategy for learning landmark positions and related vehicle poses
In this section, we introduce the proposed strategy for learning a landmark and its position. Simply speaking, for a landmark to be learned well, we guide the vehicle
to appropriate positions to detect it. To increase the accuracy of the learned landmark position, we take images of the landmark a number of times from a number of different directions after guiding the vehicle to a number of different locations. The reason why we take multiple images from a fixed direction at a fixed position is that the weather condition might cause the taken images to be all different, especially when there are clouds floating across the sun in the sky during the noon time. After analyzing the collected multiple images, a more precise landmark position can be obtained, which, together with the corresponding vehicle pose (including the vehicle position and orientation on the path), is recorded as part of the learned navigation path.
To be more specific, after detecting the landmark in acquired omni-images for a multiple times with the vehicle in a number of poses, we calculate the mean of all the detected landmark positions as an estimated landmark position, denoted as Plandmark. Furthermore, we choose the vehicle pose among the multiple ones, which is closest to the one to yield the estimated Plandmark, for use as the learned pose, denoted as Pvehicle, corresponding to the estimated Plandmark. The detailed algorithm for the above process is described in the following.
Algorithm 3.2 Learning of the landmark position and related vehicle pose.
Input: A landmark type of the appointed landmark to be learned.
Output: an estimated landmark position Plandmark and a corresponding vehicle pose Pvehicle.
Steps.
Step 1. Initialize three parameters i, j and k to be all zero, where i, j and k represent the k-th landmark detection, the j-th vehicle orientation, and the i-th vehicle position, respectively.
Step 2. Guide the vehicle to a position Vi = (Pxi, Pyi) and record this vehicle position Vi into a set SV.
Step 3. Turn the vehicle into an orientation Thij and record this orientation into a set STh.
Step 4. According to the landmark type, localize the landmark by the use of the corresponding localization technique (described in subsequent chapters) to obtain the landmark position pijk = (xijk, yijk), and record this landmark position pijk into a set SL.
Step 5. Go to Step 4 for K times as needed, and record the number of recoded landmark positions in the j-th vehicle orientation and the i-th vehicle position, denoted as Nij = K.
Step 6. Go to Step 3 for J times as needed, and record the number of different vehicle orientations in the i-th vehicle position, denoted as Ni = J.
Step 7. Go to Step 2 for I times as needed, and record the number of the different vehicle position, denoted as N = I.
Step 8. Compute the desired landmark position Plandmark using the set SL by the following equation:
c vehicle vehicle c c c
. (3.9)
distance to P computed in terms of the Euclidean distance.
Choose a median orientation Th from all Th in S , where a is 1 through N , and set the desired vehicle pose P as P = (Px , Py , Th ).
3.6.2 Learning of fixed obstacles in a navigation path
In this study, we propose a function in the learning interface which can be used to learn fixed obstacles. After guiding the vehicle to a proper location where a fixed obstacle is projected onto the image region of both Mirrors A and B, we utilize this function to learn the fixed obstacle. As shown in Figure 3.14, we can select the fixed
In this study, we propose a function in the learning interface which can be used to learn fixed obstacles. After guiding the vehicle to a proper location where a fixed obstacle is projected onto the image region of both Mirrors A and B, we utilize this function to learn the fixed obstacle. As shown in Figure 3.14, we can select the fixed