Chapter 1 Introduction
1.5 Organization of this thesis
The rest of this thesis is organized as follows: Chapter 2, describes how to crop a useful area for reducing computing area; Chapter 3, describes how to detect lane segment with our accurate algorithm and gets correct lane model; Chapter 4, describes how to detect the front vehicle and its position by shadow and C-means algorithm; Chapter 5, provides an algorithm about how to map the front vehicle distance from 2D image to real world 3D coordinate;
Chapter 6, describes how to use the parameters got from pre-chapter to make a decision;
Chapter 7, gives the experimental result; and Chapter 8 concludes our works and remark on some possible future works.
Chapter 2
Road Cropping
Road cropping stage is used to reduced calculate area. Base on our application, we only need to calculate road area in the image. The composition of a high way image is very complex in road area but clear in sky and cabin area such as in Figure 2.1. From Figure 2.1, we can realize that lanes are comprised of a lot of oblique line segments. So, we can use ±45° line edge filter to filter out lane segments and do a horizontal histogram to isolate road area.
Figure 2.1: A High Way Image
2.1 HSV Color Model
In our system, we use color information and line structure to do the lane and vehicle detections. The true color image is hard to do computer vision especially to the object outline and the edge analysis. In general, the gray image is good for edge detection. Therefore, we need to transform image form the true color to the gray level image. We chose HSV color space to be our process domain.
Road Area Sky Area
Cabin Area
The HSV model is commonly used in the computer graphics applications. In various application contexts, a user must choose a color to be applied to a particular graphical element.
When used in this way, the HSV color wheel in Figure 2.2 is mostly used. In it, the hue is represented by a circular region; a separate triangular region may be used to represent the saturation and the value. Typically, the vertical axis of the triangle indicates saturation while the horizontal axis corresponds to value. In this way, a color can be chosen by first picking the hue from the circular region, and then selected the desired saturation and value from the triangular region.
Figure 2.2: HSV color wheel
Another visualization method of the HSV model is the cone in Figure 2.3. In this representation, the hue is depicted as a three-dimensional conical formation of the color wheel.
The saturation is represented by the distance from the center of a circular cross-section of the cone, and the value is the distance from the pointed end of the cone. This method is well-suited to visualize the entire HSV color space in a single object.
Figure 2.3: HSV cone.
Here is the formula of conversion form RGB color model to HSV color model. In our application the value range of Red, Green and Blue is 0~255.
h =
⎩⎪
⎨
⎪⎧0, if max = min 60° × + 360° mod 360°, if max = r
°× °, °× °,
(2.1)
s = 0, if max = 0
= 1 − , otherwise (2.2) v = max (2.3) l r, g, b ∈ [0,255].
l h: The hue angle, where h ∈ [0,360].
l s: The saturation.
l v: The lightness.
l max: The greatest of r, g and b.
l min: The least of r, g and b.
2.2 Lane Filter
From Figure 2.1, we realized that the road area composes with the lanes, vehicle, traffic signs, plants and so on. The composition is very complex. The sky area does not have obvious line segments. And the cabin area just has some fix horizontal lines. Therefore, we will apply a lane filter to the gray scale image and do a horizontal histogram from the edge image. From the histogram, we can segment out the road area easily. From Figure 2.1, we also understood that the lane is composes with the oblique lines. The vector of lanes in the left part of image is positive and negative in the right part of image. So, we used two 3x3 matrices to filter out these line segments. Below is our filters and formula.
g(x, y) = ∑ ∑ LM(s, t)f(x + s, y + t), if x ≤ (width 2 ) addition to figure out the edge, marks the lane attribute is the other important process.
2.3 Cache Enhance Lane Filter Coding
The RISC CPU always has weak cache system. The CPU on our experiment device is Qualcomm 7201A. It is ARM 11 family and just has 16K bytes I-Cache and D-Cache. Where the original data is expensive to fetch (owing to longer access time) or to do the computing, compared to the cost of reading the cache. In other words, a cache operate is used as a temporary storage area where frequently accessed data can be stored for rapid access. Once the data is stored in the cache, it can be used in the future by accessing the cached copy rather than re-fetching or re-computing the original data. Hence we should avoid the cache miss as much
current stack which should be in cache memory and do the 3x3 filter each times. After the filter process has done we can move the memory point, copy one processed line to the target image memory and just copy next new line from the memory to the memory in the cache. According to our experiment, this skill can be improved about 30% in the performance than operating the memory directly while doing the filter process. In the below is our coding sample of this cache enhancement skill on lane filter.
static unsigned char *proc_line;
static unsigned char *up_line;
static unsigned char *bt_line;
static unsigned char *cp_line;
static unsigned char *p_proc_line;
static unsigned char *p_up_line;
static unsigned char *p_bt_line;
static unsigned char *p_cp_line;
static unsigned char *p_temp;
static unsigned char linecache[960];
p_up_line = up_line = linecache;
p_proc_line = proc_line = linecache+240;
p_bt_line = bt_line = linecache+480;
for(int Y=m_ImgTop+1; Y<m_ImgBottom-1; Y++){
addr = Y*ImgWidth+1;
addr_w = (Y+2)*ImgWidth;
memcpy(p_cp_line,&pGrayImg[addr_w],240);
for(int X=1; X<ImgWidth-1; X++){
2.4 Lane Probed Algorithm
From Figure 2.5, we can see that the lane pattern is symmetry. So, we use this attribute to develop an algorithm to keep the problem lane marker. We call it lane probed algorithm. The main concept is, if we probed a series positive edge pixel following a series space pixel and negative pixels with some proportions, we then regard that as a lane marker. In our system, we assume that a series pixel is smaller than 3 pixels. Then we stuff white color in the positive and negative lane edges for subsequence algorithm operation. The Figure 2.4 (c) shows that the image processed with lane probe algorithm will get a clear lane image. It will help us to calculate the road area later in this chapter. Following is the pseudo code of lane probe algorithm.
for(Y = m_ImgTop; Y<m_ImgBottom; Y++){
Count3 = 0; Count1 = 0; Count2 = 0;
(a) (b) (c)
Figure 2.4: (a) Original Image. (b) The image applied the lane filter. (c) The image processed with lane probe algorithm.
(a) (b) Figure 2.5: (a) Lane pattern. (b) Others
2.5 Road Area Conjecture
After the previous processes, we will use those processed images to conjecture the road area. In this thesis, we propose a histogram algorithm to conjecture the road area. At first, we overlap every image which was processed by lane probe algorithm. In our experiment experiences, overlaping 50 images is enough to do the conjecture. Figure 2.5 (a) is the result of overlapping 50 images. In this image we can see that, the lane was marked on the image very clearly. But there are still some noises. Maybe we can use some advanced algorithm to rule out these noises. However it is not efficient on mobile device, so we do not remove these noises
directly. We use statistic to vote the road area. Do a horizontal histogram on overlapping images is our counterplot. Figure 2.5 (b) is the histogram of figure 2.5 (a).
(a) (b)
Figure 2.6: (a) Overlap image. (b). Horizontal histogram.
From this histogram, it is easily to understand that the road area is the maximum group in the histogram. Therefore, we just need to find out the maximum group in the histogram and we can get the road area. Following is the pseudo code about how to get the maximum group in the histogram.
1 26 51 76 101 126 151 176 201 226 251 276 301
for(s=0; s<Length; s++) {
Chapter 3
Accurate Lane Detection
We have proposed some lane filters and extraction lane algorithms. However it still has a lot of noise information in the processed image. Most of papers use a second order model to model the lane. This is:
y = a + a x + a x (3.1) They always use least square fitting to find the coefficients. If the image just has lane markers and this algorithm can find the correct lane fit. But, if there are too many noise markers in the image such as car shadow, traffic signs, guardrail and so on, this kind of fitting should be failed. So, these fitting like papers are not accurate practical in our project. The other people use hardware to do the brute force voting the lane curve. But, this kind of algorithm is very expensive and hard to implement on the embedded system. In this chapter, we proposed a smart voting system which is merges with simple linear regression.
3.1 Image Normalization
From Figure 2.4(c), we can see that if we just use filter to treat the image, it is hard to extract the lane marker cleanly. Some papers used Gaussian Filter with Sobel Filter or get its gradient in image. Of course, it can get a good lane marker image, but they are very expensive for CPU. We have done an experiment which deal with a gradient process on an image with 240x320 and it consumed 220 ms in our system. So, we propose that using a color information to pick up lane and car object and do the filter process in chapter 2.2. It is very efficient, because we can do the color probe at the color transform stage conveniently. But, the color range is hard to be defined. There are too many factors cause the image with different color temperature, contrast, brightness and saturation. Therefore, we need to normalize the input image at first. Following is our normalization algorithm.
S = 255 × ∑ (3.2) l k = 0,1,2, … ,255
l nj : Number of pixel with intensity j.
l n: Total number of pixels.
l For every pixel if I(im,i,j) = k then I(imhe,i,j) = Sk
In our algorithm, we do not use full image to do the normalization as we worry about the lane color and black shadow. If we use full image to do the normalization process, the most white and most black should not appear in road area. And therefore result in the lane color or car shadow hard to be detected. For the reason, we will just use the road crop area to do the normalization analysis. In Figure 3.1 (a) is the original image. We can see that the image is partial blue and the lane is not white in the image. In Figure 3.1 (b) is the normalized image.
The lane in the image is pure white and the shadow is the most black. That is helpful for us to use color information to do the lane and car detection.
(a) (b) Figure 3.1: (a) Original image. (b) Normalization image.
3.2 Lane and Vehicle Detection
The gray level image is one of the most important source for computer vision, because it is easily to be done with some edge detections. In this system, we need to make the best of CPU resource. We think that we should do something in the color transform stage at the same time. Using object’s color information to do the object detection is a good choice in this system.
In Taiwan, lanes are with white, yellow and red color. Vehicle has deep black shadow in the daytime [8]. In this thesis, we propose to use HSV color domain to do the color object detection. Using RGB color domain to do the color object detection is hard to define what is white, yellow or red. If used HSV color domain, we can define that white is V bigger than 192 and S is small than 64. Yellow is S bigger than 60 and H between 40 and 60. Red is S bigger than 60 and H between 350 and 10. Deep black is V smaller than 50 and S bigger than 86. We can get the gray scale image from V and separate the lane object and vehicle object images at this stage conveniently. Figure 3.2 shows that we can separate the vehicle and lane with color transform at the same time.
(a) (b)
(b) (d)
Figure 3.2: (a) Original image. (b) Normalization image. (c) Separated lane image from normalization image. (d) Separated vehicle image from normalization image.
3.3 Extract Lane
In this stage, we need to extract lane markers as much as possible as we will use these lane markers to do the linear fitting. Otherwise, if the markers are not clear enough the result will be wrong. In Figure 3.2 (b), the separated lane image is not clear enough because it contains the wrong markers caused by halo and reflected light. Hence, we need to apply lane filter and lane probed algorithms on the separated lane images. But the lane probed algorithm in this stage has some difference. We do not fill in the gap with negative and positive edge.
That will cause too many lane markers and lead the linear fitting hard to work on embedded system. Thus, we just keep the middle point of negative and positive edge. Figure 3.3 (a) is an extracted lane image.
3.4 Lane Model
The application in this thesis, lane fitting is an auxiliary function. We used lane to find the front vehicle and driving shift. The lane in the high way is closed to straight line even if in curve. If we just went to find front vehicle, the absolute lane fitting is not necessary. In addition, the calculation power of RISC CPU is very week for quadratic curve operation.
Therefore, we just used a linear equation to do the lane fitting. And in our experiment, we proved that a linear equation is good enough in this project. Following is our lane model.
y = ax + b (3.3) After got the separated lane images, we collected the potential lane pixels and did the least squares fitting to derivate the lane model. In this case, the two sets of linear equations Lr and Ll could be then determined by using the information of the right and left lane coordinates.
Take the right lane markings for example, let the extracted points be a set of N observations
= 2 ∑ (y − ax − b)(−x) = 0 (3.8)
= 2 ∑ (y − ax − b)(−1) = 0 (3.9)
−1 × ∑ x y + a × ∑ x + b × ∑ x = 0 (3.10)
−1 × ∑ y + a × ∑ x + b × ∑ i = 0 (3.11)
a × ∑ x + b × ∑ x = ∑ x y (3.12) a × ∑ x + b × N = ∑ y (3.13)
These equations are nicely represented in matrix form. The parameters of the best line are found by solving the equation (3.14).
∑ x ∑ x
∑ x N ab = ∑ x y
∑ y (3.14)
Following shows the liner fitting result:
(a) (b)
Figure 3.3: (a) Lane marker image. (b) Linear fitting result.
3.5 Slope Histogram
Using least squares fitting to derivate the lane model is an efficient way. But this method should be with a necessary condition that a clear lane separated image is required. If we do not get a clear lane separated image like in Figure 3.4 (a) then we will get a wrong linear fitting result such as in Figure 3.4 (b). However, to get an absolute clear separated lane image needs a lot of preprocess on the input images. It is very consumption of the resource on smart phone system. Thus, we did not use linear fitting to find the lane model at the decision stage. Instead, we used the vote algorithm to get the lane model.
In this section, we need to do some things for the accelerating vote speed. We think that, if user set the camera up at the cabin and the view angle is fixed, the slop of right and left lanes is located at two normality values if the driver drives at the middle of the lane. We did the statics of factor “a” in equation (3.3) with some frames and get the slope “a” which is the most voted. Then we defined two new linear equations with capital letter “A” and “B” which is means the maximum voted slope on left lane and right lane.
y = Ax + by = Bx + c (3.15)
After getting these two lane model equations, we need to redefine the road area by equation (3.15). The top boundary “Top” is such as following equation.
Top = A × + b. (3.16) And bottom “Bot” is such as following equation.
Bot = min (b, c) (3.17)
3.6 Lane Voting
In this thesis, we proposed to use the lane voting algorithm to get an accurate lane model equation. The benefit of lane voting algorithm is that we do not need to get an absolutely clear lane marker image but can get accurate lane model equation easily. The first stage is only to get lane marker image and following with the follow chart of getting pre-process lane image for the lane voting.
Figure 3.4: The flow chart of pre-process lane image for lane voting.
In Figure 3.4, the stages of “Image Normalization”, “Lane Detection in HSV Color Domain” and “Apply Lane Filter” are related to previous chapters.
After getting the lane mark images, we can start to do the lane voting process. From chapter 3.4, we have got two equations. One is the left lane model and another is right. The idea is to use these equations to scan full lane mark image and take the maximum matched equation to be the lane model equation. (Figure 3.5)
Get Color Image
Image Normalization
Lane Detection in HSV Color Domain
Apply Lane Filter
.
Figure 3.5: (a) Matching left lane. (b) Matching right lane.
. (a)
(b)
Figure 3.5: (a) Matching left lane. (b) Matching right lane.
.
.
…
.
… … …
Figure 3.5: (a) Matching left lane. (b) Matching right lane.
After getting the maximum matching, we matched lane model is just the closest equation
we fix the parameter “b” in the equation (3.3) and do some addition parameter “a” to get potential maximum match.
the maximum matching, we need to do some angle fine tun matched lane model is just the closest equation to the real lane marker. In our fine tune
in the equation (3.3) and do some additions and subtraction to get potential maximum match. In Figure 3.6 is our matching result.
Figure 3.6: Lane voting result.
need to do some angle fine tunings. The real lane marker. In our fine tune process,
and subtractions on igure 3.6 is our matching result.
The vehicle detection is very important in this system.
in the image demands a complexity algorithm to do it. In this algorithm to detect a vehicle in the
4.1 Detect Vehicle by the Shadow
In the chapter 3.1, we separated
stage. In this chapter, we need to do more calculation to extract the vehicle at the front. We do not care the neighborhood vehicle
up of the lane models. The blue area in the following
4.2 C-means
We used shadow to conjecture shape feature so it is hard to use
the shape. Here, we use C-means to decide the position of algorithm.
V(x, y) = ∑ xi , ∑
N: The markers of vehicle in the searching area.
In our system, we just need to get the y position. So, we can simplify the equation (4.1) as following.
V(y) = ∑ yi
Figure 4.2 is our search result.
Chapter 4
Vehicle Detection
ehicle detection is very important in this system. A robust vehicle detection process a complexity algorithm to do it. In this thesis, we propose
in the front, but it bases on the searching assistance
separated the land and vehicle images at image color transform stage. In this chapter, we need to do more calculation to extract the vehicle at the front. We do
t care the neighborhood vehicles but just the front one. So, the search area is just in the
t care the neighborhood vehicles but just the front one. So, the search area is just in the