Chapter 4 Calibrations of Vehicle Odometer and Proposed
4.4 Experimental Results
In this section, we show some experimental results of the proposed calibration
method for the proposed two-mirror omni-camera. Result 1 is the coefficients of the proposed non-linear function. Result 2 is the elevation angles corresponding to the pixels. Result 3 is the error ratios in landmark location estimations.
Result 1:
The obtained coefficients of the proposed non-linear function fr ( i ) in Equation (4.12) are listed in the Table 4.3, and in the computation we use radians instead of degrees as units for the elevation angles .
Table 4.3 The coefficients of fr.
a0 820.955200195313 a3 -33481.98046875
a1 -3812.36010742188 a4 34658.2734375
a2 15473.1533203125 a5 -13728.5224609375
Result 2:
The elevation angles are shown in Figure 4.9. We use degrees of saturation of red color to indicate the different magnitudes of the elevation angles, with red color with the higher saturation representing larger elevation angles and vice versa. There are three circles in Figure 4.9, the biggest one representing the omni-image portion reflected by the bigger mirror, the medium one representing the omni-image portion reflected by the smaller mirror, and the smallest one representing the omni-image portion reflected by the camera lens in the omni-camera. The elevation angles from far to nearby with respect to the center of the circles are given red color from less saturated to more.
Result 3:
The error ratios in landmark location estimations using the result of the proposed calibration method are shown in Tables 4.4 and 4.5. Table 4.4 includes the locations of the calibration landmark and the estimated data of their locations, Table 4.5 is the computed error ratios in these location estimation data. As can be seen, the errors are
relatively small, and will not affect the effectiveness of the proposed two-mirror omni-camera in our application of vehicle navigation for person guidance.
Figure 4.9 The graphical representation identifies the elevation angles.
Table 4.4 Locations (X, Y, Z) of the calibration landmarks and the estimated location (estX, estY, estZ).
X Y Z estX estY estZ
-120 -57 -80 -102.96 -45.203 -77.43
-100 -57 -80 -95.47 -52.73 -82.13
-80 -57 -80 -82 -57.37 -84.11
-60 -57 -80 -64.11 -61.43 -87.62
-40 -57 -80 -42.38 -60.72 -85.96
-20 -57 -80 -20.69 -59.42 -85.01
0 -57 -80 1.039356 -55.1732 -79.40129212
20 -57 -80 20.37 -54.45 -78.44
40 -57 -80 37.84 -52.02 -75.45
60 -57 -80 52.89 -48.01 -72.38
80 -57 -80 77.28 -51.71 -78.62
100 -57 -80 97.4 -50.91 -81
120 -57 -80 104.99 -42.91 -76
-120 -57 -120 -116.55 -54.86 -112.1
-100 -57 -120 -64.39 -27.5 -69.2
-80 -57 -120 -81.1 -57.069 -124.32
-60 -57 -120 -62.6 -59.49 -128.34
-40 -57 -120 -41.39 -59.01 -127.76
-20 -57 -120 -19.1 -57.46 -124.34
0 -57 -120 -0.32 -60.1 -128.62
20 -57 -120 20.04 -53.32 -115.89
40 -57 -120 37.9 -46.22 -109.54
60 -57 -120 55.92 -48.93 -109.78
80 -57 -120 74.57 -50.141 -111.2
100 -57 -120 84 -43.76 -101
120 -57 -120 103.35 -45.17 -103.92
Table 4.5 Error ratios in location estimations with calibration landmarks.
Chapter 5
Supervised Learning of Navigation Path by Semi-automatic Navigation and Hand Pose Guidance
5.1 Idea of Proposed Supervised Learning Method
The main purpose of the learning procedure is to create a path map while instructing the vehicle to navigate in an environment to learn. In this chapter, we propose two techniques to detect the guide line (the curbstone) in the outdoor environment. And if the color of the curbstone is too similar to the ground color to be detected or if there is no special color in the environment, we propose a new technique of detecting the hand pose in front of or on top of the camera enclosure to instruct the vehicle to navigate properly in the environment.
In addition, when navigating in outdoor environments, the vehicle often encounters the problem of varying light intensity in the environment which has a serious influence on the image analysis work. We propose a method, called dynamic exposure adjustment, to adjust the exposure of the camera automatically and propose another method, called dynamically thresholding adjustment, to adjust the image threshold value used in the hand pose detection procedure in the outdoor environment.
An illustration of the learning procedure is in Figure 5.1.
Figure 5.1 An illustration of the learning procedure.
5.2 Coordinate Systems
In this section, we summarize the four coordinate systems utilized in this study to describe the relationship between the devices and the navigation environment. The coordinate systems are illustrated in Figure 5.2 and the definitions of all the coordinate systems are described in the following.
(1) Image coordinate system (ICS): denoted as (u, v). The u-v plane coincides with the image plane and the origin OI of the ICS is placed at the center of the image plane.
(2) Vehicle coordinate system (VCS): denoted as (Vx, Vy). The Vx-Vy plane is coincident with the ground. The Vx-axis of the VCS is parallel to the line segment joining the two driving wheels. The origin OV is placed at the middle of
origin OV.
(3) Global coordinate system (GCS): denoted as (x, y). The origin OG of the GCS is a pre-defined point on the ground. In this study, we define OG as the starting position of the vehicle navigation.
(4) Camera coordinate system (CCS): denoted as (X, Y, Z). The origin O1 of the CCS is a focal point of Mirror 1. And the X-Y plane coincides with the image plane and the Z-axis coincides with the optical axis of the camera.
The GCS is determined when a navigation session is started. The CCS and VCS follow the vehicle during navigation. The coordinate transformation between the VCS and the GCS can be described by the following equations:
x Vx × cosθ Vy × sinθ xp, (5.1) y Vx × sinθ Vy × cosθ yp, (5.2)
where (xp, yp) represent the coordinates of the vehicle in the GCS, and θ is the directional angle between the positive direction of the x-axis in the GCS and the positive direction of the Vx-axis in VCS.
5.3 Proposed Semi-automatic Vehicle Navigation for Learning
5.3.1 Ideas and problems of semi-automatic vehicle navigation
In the thesis study, we use the curbstone features to guide the autonomous vehicle to navigate in the outdoor environment, and we have at least three problems in the following to solve before this idea can be implemented.
Features are not easy to detect with varying light intensities in the outdoor environment, and how should the autonomous vehicle adjust itself appropriately to deal with the varying light intensities?
How does the autonomous vehicle extract curbstone features in the outdoor environment automatically?
How does the autonomous vehicle compute range data to move in the outdoor environment automatically?
We propose several techniques to solve these problems in the following sections.
(a) (b)
Figure 5.2 The coordinate systems used in this study. (a) The image coordinate system. (b) The global coordinate system. (c) The vehicle coordinate system. (d) The camera coordinate system.
5.3.2 Adjustment of the exposure value of the camera
In the outdoor environment, the light is changing. The varying light influences the navigation experiment very much. The dynamic exposure adjustment we propose to
as discussed in the following:
1. create a so-called basic image where the features can be detected as well and then the local intensity Ibasic can be recorded into the memory storage;
2. record the intensity of each input image Ii with different exposure values Expi
before the navigation procedure started;
3. generate a linear function fexp to fit Ii and Expi using a line fitting technique;
4. adjust the exposure value dynamically in the navigation procedure.
To increase the efficiency, we use a bisecting method to adjust the exposure value of the camera automatically when generating fexp. The line fitting scheme we used is like that we used in Section 4.2.3 and is described in the following.
1. Generate a line equation:
y ＝mx b, (5.3)
Figure 5.3(a) shows the relationship between the exposure Expi of the camera and the intensity value Ii of the input image for 105 data items. Figure 5.3(b) shows the result of line fitting using the data of Figure 5.3(a).
0
Figure 5.3 The relationship between the exposure of the camera and the average of the intensity of the input image. (a) The distributed data. (b) The distributed data with a fitting line.
Algorithm 5.1. Establishment of the model for dynamic exposure adjustment.
Input: the intensity Ibasic of the base image; the input images Iinput; the range of the exposure value of the camera [Emin, Emax].
Output: the exposure value Eexp of the camera and the mapping function fexp. Steps:
Step 1. Set the termination condition to be | Eexp－Emax | < 1 or | Eexp－Emin | < 1.
Step 2. Computer Eexp＝(Emin＋Emax) / 2.
Step 3. Compute the intensity Iinput of the image and compare it with Ibasic; if Iinput
－Ibasic < 0, then set Emin＝Eexp; else, set Emax＝Eexp. Step 4. Record Iinput and Eexp.
Step 5. Repeat Steps 2 through 4 until the termination condition is satisfied.
Step 6. Create a mapping function fexp(x) = ax b, where x = Iinput and fexp(x) =
In the navigation procedure, the system adjusts the exposure value of the camera at the beginning of each cycle to deal with the varying light condition. The adjustment equation is defined in the following:
m = (Expnew Expnow) (Ybase Ynow), (5.6) or equivalently,
Expnew = (Ybase Ynow)×m Expnow (5.7) where m is the slope of the fitting line, Expnew is the new exposure value, Expnow is the current exposure value, Ynow is the current intensity value of the input image and Ybase is the intensity value Ibase of the base image.
5.3.3 Single-class classification of HSI colors for sidewalk detection
Two methods are used to classify the curbstone features on the sidewalk in this study. First, we use the special color of the curbstone at the sidewalk to detect the guide line. As illustrated in Figure 5.4, a cylinder is used to represent the HSI color model. It translates the RGB space into three channels, called hue, saturation and intensity, forming an HSI color space with hue representing the perceived colors in 0o to 360o, saturation representing the brightness of the color, and intensity representing the gradient of the gray level images.
Figure 5.4 The HSI color model [23].
Method 1.
We use the hue value and the saturation value to describe the color of the curbstone as shown in Figure 5.5. The ground of the sidewalk looks unlike the red color, but as we classify the curbstone from the sidewalk using only the hue channel, the result is shown in Figure 5.5(b) which is not good. Fortunately, we found the saturations of the ground are much lower than those of the curbstone because of the different materials. Therefore, we use both hue and saturation values to extract the curbstone of the sidewalk. The classification rule for extracting the curbstone is designed to be in the following:
"if the input pattern x satisfied the following conditions:
thH ≦ h(x) ≦ thH2, thS ≦ s(x) ≦ thS2,
then assign x to the class c; else to the class g, " (5.8) where c is the class of the curbstone, and g is the class of the ground of the sidewalk; the functions h and s are used to translate the input pattern x to the hue value and saturation value respectively; thH1 and thH2 are the threshold values in the hue channel; and thS1 and thS2 are the threshold values in the saturation channel.
Method 2.
We extract the guide line using the ground features instead of the curbstone in this case. In the learning process, we use two-dimension features of the hue and saturation and design a single-class classification rule to extract the curbstone:
"if d(X) < thre, then assign the input pattern X to the class g; else,
to the class c," (5.9)
where X is the input pattern, c is the class of the curbstone, g is the class of the ground
distance which is defined as follows:
d2 = (X M)TΣ1(X M)
where M is the mean vector of g and Σ is the covariance matrix of g. The resulting image of Method 2 is shown in Figure 5.7.
Method 2 is more general than Method 1 but the computation cost of it is higher.
If the color of the curbstone is different, the system can still extract out the guide line by using Method 2. In order to adapt to the real environment, Method 2 is adopted in this study.
5.3.4 Proposed method for guide line detection
In this section, we will describe four steps for detecting the guide line features and finding the corresponding points in the following. We call this procedure as guide line detection.
1. Defining the extraction windows to detect the features.
We define two pairs of extraction windows, each pair has two extraction windows, to detect the guide line features of the curbstone, and one pair is used for the lateral direction and the other pair is used for the frontal direction. In order to increase the efficiency, the extraction windows in this study are defined as the rectangular shapes instead of them as the fan shapes.
In the lateral direction, the extraction windows are defined to cover a shape extending from 223o to 256o in the ICS. The extraction window Wsmall is located from the top-left point at coordinates (701, 713) to the bottom-right one at coordinates (755, 759) with the size of 46 pixels by 54 pixels and the other extraction window Wbig is located from the top-left point at coordinates (463, 932) to the bottom-right point at coordinates (685, 1116) with the size of 184 pixels by 222 pixels. Their shapes
appearing in the omni-image in accordance with the rotational invariance property is illustrated in Figure 5.6(a).
(a)
(b)
(c)
Figure 5.5 Experimental results of extracting curbstone features. (a) An input image. (b) The curbstone features extracted by the hue channel. (c) The curbstone features extracted by the hue and saturation channels.
In the frontal direction, the pair of the extraction windows is defined from 244o to 299o in the ICS. The extraction window Wsmall is located from the top-left point at coordinates (753, 730) to the bottom-right point at coordinates (811, 798) with the
size of 68 pixels by 58 pixels and the other extraction window Wbig is located from the top-left point at coordinates (692, 929) to the bottom-right point at coordinates (882, 1124) with the size of 195 pixels by 190 pixels by the rotational invariance property as illustrated in Figure 5.6(b). Each of the four corners of Wbig and that of Wsmall are aligned in the same radial direction.
2. Detecting the features of the curbstones.
By the proposed classification rules, we can extract curbstone features well and we also use some techniques in imaging process such as connected components, dilation and erosion to eliminate noise in the image after detecting the features of the curbstone. The resulting image is shown in Figure 5.7.
(a)
Figure 5.6 Two extraction windows used to detect guide line. (a) A extraction window used for detection of lateral direction. (b) A extraction window used for detection of frontal direction.
(b)
Figure 5.7 Two extraction windows used to detect guide line. (a) A extraction window used for detection of lateral direction. (b) A extraction window used for detection of frontal direction. (cont’d)
Figure 5.8 The result of detecting the curbstone features in the experimental environment.
3. Extracting the guide line of the curbstones.
Many curbstone features are hard to use to detect the corresponding points in Wbig, so we propose a method to detect the guide line of the curbstone instead of total features. After detecting the curbstone features, we scan the image in Wsmall from right to left and from top to bottom to find the guide line at the inner position of the sidewalk. After detecting the first feature pixel with the scan order from right to left, label the pixel as a guide line feature point and then detect the next feature pixel in the next row. An illustration is shown in Figure 5.8.
4. Finding the corresponding points.
In [10] and [23], an omni-image is transformed into two panoramic images for two mirrors, then the bilinear interpolation technique is used to refine the panoramic images, and corresponding points are found by some matching algorithms, and range data are computed finally. A disadvantage is that the computation cost of this process is high.
Figure 5.9 The guide line (the red dots) extracted by the proposed method.
We propose in this study a method based on the rotational invariance property of
the omni-image without transforming the omni-image into two panoramic images, as illustrated in Figure 5.9. The corresponding point is in the same radial direction of the feature point, so we scan the feature points along the radial direction (the red dotted line in Figure 5.9(a)) and take the corresponding point to be the last feature pixel in the image portion corresponding to the large mirror, as illustrated by Figure 5.9(a).
Algorithm 5.3. Computation of corresponding feature points.
Input: an image Iinput.
Output: an image with corresponding points marked.
Steps:
Step 1. Scan Iinput in Wsmall and Wbig to extract the curbstone features based on the proposed classification rules.
Step 2. Scan in Wsmall from right to left and from top to bottom to find the guide line at the inner sidewalk. If the first curbstone feature on a row is found, label the pixel as a guide line feature Fg, and then scan the next row until finished.
Step 3. For each Fg, compute the azimuth angle θ, and scan Iinput accordingly on the same radial direction of Fg from near to far with respect to the center in the area of Wbig.
Step 4. Label as a corresponding point of Fg the last feature pixel found in the from-near-to-far scanning with respect to the center of the omni-image.
5.3.5 Line fitting technique for sidewalk following
After detecting the guide line features and their corresponding points, we can compute the range data by Equations (3.15) and (3.16). And in this section, we use the line fitting technique described by Equations (5.3) through (5.5) to refine the range
data. We also use the technique to compute the directions of the guide line. The proposed schemes are illustrated in Figure 5.10, and the procedure is listed in the following.
Algorithm 5.4. Refining the range data by the line fitting technique.
Input: a set of range data, R.
Output: a refined set of range data R'.
Steps:
Step 1. Use a line equation L to fit all the range data by Equations (5.3) through (5.5).
Step 2. Calculate the distance di from each range data item ri to L.
Step 3. Compute a threshold value dmean which is the average value of all computed distances di.
Step 4. Collect the range data item ri into R' if the corresponding distance di < dmean. Step 5. Take the final R′ as the output.
(a)
Figure 5.10 Finding the corresponding point for the feature point. (a) The scanning order. (b) The experimental results.
(b)
Figure 5.11 Finding the corresponding point for the feature point. (a) The scanning order. (b) The experimental results. (cont’d)
(a)
(b)
(c)
Figure 5.12 An illustration of the proposed scheme to refine the range data. (a) Fitting the range data. (b) Computing the distance from each data to the line. (c) Refined result.
After refining the range data, we obtain the turning angle of the vehicle and the distance d from the fitting line to the mirror center by
= tan1(1/m), (5.11)
5.3.6 Proposed method for semi-automatic navigation
In the navigation procedure, we adopt a strategy to instruct the vehicle to follow a safer navigation route as illustrated in Figure 5.11. If the vehicle navigates in any of the damage zones as shown in the figure, its speed will be slowed down (and so to increase the cycle time of the navigation procedure).
Figure 5.13 The dangerous (damage) zones along the guide line.
Algorithm 5.5. Process of semi-automatic navigation.
Input: current image Iinput and the mapping function fexp for exposure correction.
Output: the vehicle commands with and d.
Steps:
Step 1. Compute the average intensity of Iinput.
Step 2. Adjust the exposure of the camera automatically by fexp described in Section 5.3.2.
Step 3. Extract the curbstone features by the proposed techniques described in Section 5.3.3.
Step 4. Extract the guide line feature points by the techniques described in Section 5.3.4.
Step 5. Find the corresponding points of each guide line feature point.
Step 6. Calculate the 3D range data of the guide line by the technique described in Section 3.2.3.
Step 7. Refine the range data R' using the line fitting technique by Algorithm 5.4.
Step 8. Conduct line fitting for R' again and get the line equation L by Algorithm 5.4.
Step 9. Compute and d of L by Equations (5.11) and (5.12).
Step 10. Slow down the speed of the vehicle if it is in the damage zone detected by the use of d.
Step 11. Perform the turning command by the use of .
5.4 Detection of Hand Poses as Guidance Commands
5.4.1 Idea of proposed method for hand pose detection
In the previous section, we described the proposed scheme to extract the guide
feature point in the environment or if the extracted curbstone feature points are similar to the feature points of the ground, then it will become hard to extract out the guide line feature points. Therefore, we propose a human interaction technique using hand poses to solve this problem. The hand poses are detected by the technique as commands
feature point in the environment or if the extracted curbstone feature points are similar to the feature points of the ground, then it will become hard to extract out the guide line feature points. Therefore, we propose a human interaction technique using hand poses to solve this problem. The hand poses are detected by the technique as commands