Chapter 3. Vehicle Surrounding Image
3.1. Lens Distortion Correction
3.1.2. Distortion Correction Model
In this thesis, we utilize the Field of View correction model mentioned in [20] to rectify the distortion line into the straight line. We assume the object point P in the world coordinate, it project on the image plane m, and the undistorted projection point m, which is shown in Fig.3-5. The relation between p and p can be described by the formula. Where x is the horizontal distance between image center to the pixel p and y is the vertical distance between image center to the pixel p . Where x is the distance frame image center to the virtual undistorted pixel p. Where r is the undistorted radius and the r is the distorted radius. According to the FOV model, the lens distortion can be fitted excellently in the field of view distortion model by finding the w parameter. In Fig. 3-6, we demonstrate the original distort image and the rectified result.
3.2
coord
and d
Fig.
29
Fig. 3- 12: Image registration result in four cameras
3.3. Image Fusion and Lookup Table Generation
3.3.1. Image Fusion
In image fusion, we deal with the image stitching result. For image stitching, the remarkable illumination difference between two source images, an obviously seam will appear after the stitching process. In Fig.3-13 shows boundary seam caused by illumination different. Here, we apply the pixel weighting function to handle the effect. Supposed two overlapped region in the target image I , and reference image I is respectively A and B, Where A x, y I , B x, y I . The fusion image is I, and then we can obtain equation (3.3)
I x, y
I , x, y I
A 1 r B r , x, y I I
I , x, y I
A I , B I (3.3) Among Eq. (3.16), r expresses the weight value, which is determined by overlapped region width d, and the range of r is 1→0 which increases based on r. We can know when r changes slowly from 1 to 0, then the image will slowly smooth translation from I to I , therefore eliminates illumination boundary. In Fig. 3-14, we show the image fusion result of Fig. 3-13.
3.3.
31
Chapter 4. Obstacle Detection
In this chapter, we describe the obstacle detection which applies on the vehicle sounding image. First of all, we extract appropriate feature points by combining road detection to ensure these points which are on the road. And, in order to reduce the false alarm caused by ground texture, we assume that the ground texture movement is same to ground movement.
According to selected feature points mentioned above, we propose ground movement estimation method WFPM to estimate the movement of ground. By the estimated movement, we construct a compensated image to model the ground texture movement in driving. The obstacle will be detected by the difference of vehicle surrounding image and compensated image. Finally, we locate the obstacle by searching radioactive line around the car center to alarm the driver. The flow chart of the obstacle detection is shown Fig. 4-1.
Fig. 4- 1: Obstacle detection system flow chart
32
4.1. Feature Point Extraction
Feature points are used to estimate ground movement by its corresponding relationship between one frame and subsequent frame. Considering the objective, to select an appropriate point is important for us. If we choose a point on a large blank wall then it won’t be easy to find that same point in the next frame of a video. If all points on the wall are identical or even very similar, then we won’t have much luck tracking that point in subsequent frames. On the other hand, if we find a point that is unique then we have a pretty good chance of finding that point again.
In practice, the point or feature we select should be unique, or nearly unique. It should be parameterizable and it can be compared to other points in another image. As a result, we might be tempted to look for points that have some significant change within neighboring local area. We say that is the good features which have a strong derivative in spatial domain.
Another characteristic of features is about the position of the image.
Considering that the objective of the following procedure is to estimate the ground movement information. Features lie on the ground region is useful criteria for the following ground movement estimation algorithm. Due to above analysis, a good feature to track should have two characteristics. First, a feature should have strong derivative which could assist us to track them and obtain a precise motion. Then, the position of feature should be restricted on the road region (non-obstacle region). The features which we will use them to estimate the ground movement information should conform the above two characteristic, these features will be suitable for estimating ground movement information.
We proposed a feature point extraction method employ road detection procedure to support in getting ground features.
To utilize the result of the road detection, major road color is compared to the feature point’s color to ensure these good features within road region. By integrating road detection, the more
33
useful ground features could be extracted and could improve results of ground movement effectively. The next chapter 4.1.2 will introduce the detail of road detection and describe what feature point will be selected.
4.1.1. Road Detection
The proposed feature point extraction technique is integrating a road detection procedure [23]. This procedure uses an on-line color model that we can train an adaptive color model to fit road color. The main objective of road detection is to discriminate the road and non-road region roughly, because the result is used to support feature extraction not used to extract obstacle regions. However, we adopt an on-line learning model that allows continuously update during driving, through the training method that can enhance plasticity and ensure the feature is on the road region.
Due to the color appearance in the driving environment, we have to select the color features and using these color features to build a color model of the road. Therefore, we have to choose a color space which has uniform, little correlation, concentrated properties in order to increase the accuracy of the model. In computer color vision, all visible colors are represented by vectors in a three-dimensional color space. Among all the common color spaces, RGB color space is the most common color feature selected because it is the initial format of the captured image without any distortion. However, the RGB color feature is high correlative, and the similar colors spread extensively in the color space. As a result, it is difficult to evaluate the similarity of two color from their 1-norm or Euclidean distance in the color space.
The other standard color space HSV is supposed to be closer to the way of human color perception. Both HSV and L*a*b* resist to the interference of illumination variation such as the shadow when modeling the road area. However, the performance of HSV model is not as good as L*a*b* model because the road color cause the HSV model not uniform that lead to
34
the HSV color model not as uniform as the L*a*b* color model. There are many reasons attribute this result. Firstly, HSV is very sensitive and unstable when lightness is low.
Furthermore, the Hue is computed by dividing (Imax - Imin) in which Imax = max(R,G,B), Imin = min(R,G,B), therefore when a pixel has a similar value of Red, Green and Blue components, the Hue of the pixel may be undetermined. Unfortunately, most of the road surface is in similar gray colors with very close R, G, and B values. If using HSV color space to build road color model, the sensitive variation and fluctuation of Hue will generate inconsistent road colors and decrease the accuracy and effectiveness. L*a*b* color space is based on data-driven human perception research that assumes the human visual system owing to its uniform, little correlation, concentrate characteristics are ideally developed for processing natural scenes and is popular for color-processed rendering. L*a*b* color space also possesses these characteristics to satisfy our requirement. It maps similar colors to the reference color with about the same differences by Euclidean distances measure and demonstrates more concentrated color distribution than others. Then considering the advantaged properties of L*a*b* for general road environment, the L*a*b* color space for road detection is adopted.
The RGB-L*a*b* conversion is described as follow equations:
1. RGB-XYZ conversion:
0.431 0.342 0.178
X = ⋅ +R ⋅ +G ⋅ B
0.222 R+0.707 G+0.071 B
Y = ⋅ ⋅ ⋅
0.020 0.130 0.939 Z = ⋅ +R ⋅ +G ⋅ B 2. Cube-root transformation:
35
where Y Z are tristimulus values of reference white poi
⎧ ⎛ ⎞
By modeling and updating of the L*a*b* color model, the built road color model can be used to extract the road region. The L*a*b* model is constituted of K color balls, and each color ball mi is formed by a center on (Lmi,*ami,*bmi) and a fixed radiusλmax =5 as seen in Fig. 4-2.
Fig. 4- 2: A color ball i in the L*a*b* color model whose center is at (Lm, *am, *bm) and with radius λmax
36
In order to train a color model, we set a fixed area around the car in the vehicle surrounding image by manual and assume pixels in this area are the road samples. In the beginning few frames are used to initialize the color model for each pixel in the sample region.
And update the model every fixed frame to increase processing speed but still maintain high accurate performance.
The sampling area is used to be modeled by a group of K weighted color balls. We denote the weight and the counter of the mi th color ball at a time instant t by Wm ti, and
i,
Counterm t, and the weight of each color ball represents the stability of the color. The color ball which more on-line samples belonged to over time accumulated a bigger weight value shown in Fig. 4-3. Adopting the weight module increases robustness of the model.
Fig. 4- 3 : Sampling area and color ball with a weight which represents the similarity to current road color.
The weight of each color ball is updated by its counter when the new sample is coming which is called one iteration. Therefore, the counter would be initialized to zero at the beginning of iteration. The counter of each color ball records the number of pixels added from
37
the on-line samples in the iteration. The first thing to do is that which color ball is chosen to be added. We measure the similarity between new pixel xt and the existing K color balls using a Euclidean distance measure (4-1). The maximum value of K is also set to limit the color ball number.
If a new pixel xt was covered by any of the color ball in the model, one will be added to the counter of best matching color at this iteration as the equation (4-2). After entire new sample pixels at this iteration undertake the matching procedures mentioned above, the weights of every color ball are updated according to their current counter and their weight at last iteration. The updating method is as follows:
i mi i i max
, where αwis user-defined learning rate, N sample is the sampling area
Then using the weight to decide which color ball of the model most adapt and resemble current road. The color balls are sorted in a decreasing order according to their weights. As a result, the most probable road color features are at the top of the list. The first B color balls are selected to be enabled as standard color for road detection, and these color balls with a higher weight has more importance in detection step. Road detection is achieved via comparison of the new pixel xt with the existing B standard color balls selected at the previous instant of time shown in Fig. 4-4. If no match is found, the pixel xt is considered as non-road. On the contrary, the pixel xt is detected as road.
38
Fig. 4- 4 : Pixel matched with first B weight color balls which are the most represent standard color.
4.1.2. Feature Extraction
As mentioned above, we consider two characteristics which are strong derivative and ground feature. The road region and strong gradient points are selected to be feature points.
Therefore, the first criterion of feature point extraction is to extract the major strong gradient points using Sobel edge detector. For each extracted feature points, we compare its color information to the road color model to distinguish the feature points on road or on the obstacle.
And the nearest position constraint is considered together to enhance the selection. Then the feature points are collected completely by these road boundary and high gradient features. In Fig. 4-5 shows the result of feature point extraction. By employing road detection to support feature point extraction, the more useful ground features can be extracted.
Fig. 4- 5 : The feature point extraction result
39
4.2. Ground Movement Estimation
Considering the road texture on the ground will cause false alarm rate of obstacle detection system. Assume that the ground texture movement attached on ground is same to the ground movement. Hence, we propose a WFPM method to estimate the ground movement vector. By compensating the ground plane movement with estimated vector we can reduce the false alarm caused by road texture. The following will brief describe the estimation method WFPM.
4.2.1. Weighted Feature Point Matching
Our estimation method WFPM (Weighted Feature Point Matching) is similar to FPM (Feature Point Matching) algorithm [24] to calculate the ground movement. The FPM technique is proposed to solve the digital image stabilization. Here, the original equation of FPM is shown in Fig. 4-6. The correlation calculation of FPM is by
, | 1, , , , |
Fig. 4- 6: The feature point movement
The N is the total number of feature point, I(t-1,x,y) is the intensity of the representative point (x , y) at frame t-1 , and , is the correlation measure for a shift (p , q) between the representative points in image at frame t-1 and the relative shifting points at frame t.
Assuming , is the minimum correlation value in image,
40
, , , , the ground movement vector v that produces the minimum
correlation value, , , , , .
We propose a new method called Weighted Feature Point Matching method which is based on the FPM technique to estimate the ground movement. Here, the road detection information will be used to enhance the matching correctness effectively. As mentioned above, the feature point extraction in section 4.1, we utilize the road color model to distinguish the feature points on road or not. We set up a fixed threshold to reselect the feature points.
However, these feature points which exceed the threshold still be different for the similarity of road color. Due to this reason, the feature point more similar to the road color, the more reliability should be assign to the feature point. In here, for every pixel exceeds the threshold, we give a road reliable weight in following criterion.
Quantitative criterion:
# .
_
Also we think about reliability on the feature points exceed the intensity threshold. If the intensity higher, the more reliability weight should be assign to feature point. The following is the intensity reliable weight criterion.
Quantitative criterion:
# .
_
We apply the road reliable weight and intensity reliable weight on the feature point matching method to enhance the matching. Therefore, the FPM correlation calculation equation rewrite to the following equation:
_ _
∑ _ , _ 1
41
_ _
∑ _ , _ 1
, 1, , , , _ _
4.2.2. Motion Control
By considering the temporal constraint, the scene will not change a lot during the few frames. For these reason, value of ground movement vector is anomalous relative to the neighboring frames would indicate the compensation information is not correct.
Through observing the recent frames can assist us to ensure the ground movement information is correct or not. Because the correctness of ground movement is greatly determining the results of following detection procedure, the verification is essential and worth to undertake. If the equation is conformed, that is indicating anomalous amount of obstacle candidate and the ground movement information in the current frame possibly erroneous. Then the previous compensation information will be utilized to compensate.
Quantitative criterion:
.
(4-6) Where Thd. is user-defined threshold
By applying the motion control technique, we can reduce the error of estimation information and validate the compensation information more reliable.
4.3. Obstacle Localization
4.3.1. Image Difference
The obstacle residue map can be obtained by the difference between current image and the compensated image, shown in Fig.4-7. It can be determined whether a point is on the ground
plane
43
4.3.2. Obstacle Localization
The special characteristic of the obstacle are used to detect obstacle in bird view coordinate, shown in Fig.4-9. The object significantly stand on the ground plane will appear radioactive line in the bird view image.
Fig. 4- 9: Radioactive line in the vehicle surrounding image
To utilize the characteristic in obstacle, we divide the image every four degree as a scan region. By moving a bounding box in this region, we can add up the radioactive line number in each region. If the radioactive line numbers in the bounding box are over the threshold we set, we locate the lowest position of the radioactive line as the obstacle to vehicle, shown in Fig.4-10.
44
Fig. 4- 10: Search every region with a bounding box
Finally, we use the proportional scale (1 pixel: 2.5 cm) which is mentioned in section 3-2 to calculate the real distance in world coordinate. Here, the distance between vehicle and obstacle will be separated in different emergency color. The red color label means the obstacle nears vehicle less than two meters. The yellow color label means the obstacle nears vehicle between two meters to four meters. The green color label means the obstacle nears vehicle longer four meters.
45
Chapter 5. Experimental Results
In this chapter we will demonstrate our research results. The experimental environment will be introduced at first. Next, we will separately demonstrate the vehicle surrounding image and its detection result. In compensation evaluation, we will generate a ground truth to evaluate proposed method.
5.1. Experimental Environment
The environmental vehicle is Mitsubishi Savrin. The four wide angle cameras is setup separately around the vehicle, shown in Fig.5-1.
(a)
(b)
Fig. 5-1 : The system setup environment (a) The front camera is mount on the mark of vehicle (b) In left side and right side, camera is mounted below the rear-view mirror
46
Our algorithm is implemented on the platform of PC with Intel Core2 Duo 2.2GHz and 2GB RAM. Borland C++ Builder is our developing tool and operated on Windows XP. All of our testing inputs are uncompressed AVI video files. The resolution of video frame is 320*240.
5.2. Vehicle Surrounding Monitoring
Lens distortion correction parameters of four cameras are shown in Table 5-1. The corresponding homography matrix of cameras is shown separately in Table 5-2.
Table 5- 1 : Camera correction parameter w and image center separately
Camera W Image Center
Front 0.0056 160 98
Back 0.00558 160 92
Left 0.00565 163 98
Right 0.00569 158 98 Table 5- 2 : Homography matrix of cameras
Front Back 0.119 0.702 125.23 -1.227 4.905 2175.25
0.002 0.402 71.17 0.204 6.27 2387.83 7.074 0.003 0.435 0.0002 0.0255 9.0298
Left Right 3.047 -0.01 -505.4 -1.088 -0.161 607.8
2.2967 0.72894 -459.69 0.9595 0.335 96.00
0.01531 -0.0010 -1.375 -0.0063 0.0006 2.851
47
The following is the vehicle surrounding image without obstacle function in driving scene which is shown in Fig. 5-2.
Fig. 5-2 : Vehicle surrounding image in driving scene
By applying the obstacle detection on the vehicle surrounding image, we detect the obstacle when user driving and parking to prevent collision accident. Fig. 5-3 and Fig. 5-4 demonstrate the experimental results of obstacle detection in vehicle surrounding image for various conditions.
48
49
Fig. 5- 3 Vehicle surrounding monitoring in driving scene
Fig. 5- 4: Vehicle surrounding monitoring in parking scene
50
5.3. Compensation Evaluation
In order to verify the accuracy of proposed ground movement estimation technique, an experiment is designed to evaluate compensation of ground movement. At first, we have to establish ground truth manually that can assist us to evaluate the estimation results.
Considering that we are going to evaluate the compensation of ground movement, we pick up three ground points for each frame and corresponding positions in previous frame. As shown
Considering that we are going to evaluate the compensation of ground movement, we pick up three ground points for each frame and corresponding positions in previous frame. As shown