Chapter 2 Road Cropping
2.5 Road Area Conjecture
(a) (b) (c)
Figure 2.4: (a) Original Image. (b) The image applied the lane filter. (c) The image processed with lane probe algorithm.
(a) (b) Figure 2.5: (a) Lane pattern. (b) Others
2.5 Road Area Conjecture
After the previous processes, we will use those processed images to conjecture the road area. In this thesis, we propose a histogram algorithm to conjecture the road area. At first, we overlap every image which was processed by lane probe algorithm. In our experiment experiences, overlaping 50 images is enough to do the conjecture. Figure 2.5 (a) is the result of overlapping 50 images. In this image we can see that, the lane was marked on the image very clearly. But there are still some noises. Maybe we can use some advanced algorithm to rule out these noises. However it is not efficient on mobile device, so we do not remove these noises
directly. We use statistic to vote the road area. Do a horizontal histogram on overlapping images is our counterplot. Figure 2.5 (b) is the histogram of figure 2.5 (a).
(a) (b)
Figure 2.6: (a) Overlap image. (b). Horizontal histogram.
From this histogram, it is easily to understand that the road area is the maximum group in the histogram. Therefore, we just need to find out the maximum group in the histogram and we can get the road area. Following is the pseudo code about how to get the maximum group in the histogram.
1 26 51 76 101 126 151 176 201 226 251 276 301
for(s=0; s<Length; s++) {
Chapter 3
Accurate Lane Detection
We have proposed some lane filters and extraction lane algorithms. However it still has a lot of noise information in the processed image. Most of papers use a second order model to model the lane. This is:
y = a + a x + a x (3.1) They always use least square fitting to find the coefficients. If the image just has lane markers and this algorithm can find the correct lane fit. But, if there are too many noise markers in the image such as car shadow, traffic signs, guardrail and so on, this kind of fitting should be failed. So, these fitting like papers are not accurate practical in our project. The other people use hardware to do the brute force voting the lane curve. But, this kind of algorithm is very expensive and hard to implement on the embedded system. In this chapter, we proposed a smart voting system which is merges with simple linear regression.
3.1 Image Normalization
From Figure 2.4(c), we can see that if we just use filter to treat the image, it is hard to extract the lane marker cleanly. Some papers used Gaussian Filter with Sobel Filter or get its gradient in image. Of course, it can get a good lane marker image, but they are very expensive for CPU. We have done an experiment which deal with a gradient process on an image with 240x320 and it consumed 220 ms in our system. So, we propose that using a color information to pick up lane and car object and do the filter process in chapter 2.2. It is very efficient, because we can do the color probe at the color transform stage conveniently. But, the color range is hard to be defined. There are too many factors cause the image with different color temperature, contrast, brightness and saturation. Therefore, we need to normalize the input image at first. Following is our normalization algorithm.
S = 255 × ∑ (3.2) l k = 0,1,2, … ,255
l nj : Number of pixel with intensity j.
l n: Total number of pixels.
l For every pixel if I(im,i,j) = k then I(imhe,i,j) = Sk
In our algorithm, we do not use full image to do the normalization as we worry about the lane color and black shadow. If we use full image to do the normalization process, the most white and most black should not appear in road area. And therefore result in the lane color or car shadow hard to be detected. For the reason, we will just use the road crop area to do the normalization analysis. In Figure 3.1 (a) is the original image. We can see that the image is partial blue and the lane is not white in the image. In Figure 3.1 (b) is the normalized image.
The lane in the image is pure white and the shadow is the most black. That is helpful for us to use color information to do the lane and car detection.
(a) (b) Figure 3.1: (a) Original image. (b) Normalization image.
3.2 Lane and Vehicle Detection
The gray level image is one of the most important source for computer vision, because it is easily to be done with some edge detections. In this system, we need to make the best of CPU resource. We think that we should do something in the color transform stage at the same time. Using object’s color information to do the object detection is a good choice in this system.
In Taiwan, lanes are with white, yellow and red color. Vehicle has deep black shadow in the daytime [8]. In this thesis, we propose to use HSV color domain to do the color object detection. Using RGB color domain to do the color object detection is hard to define what is white, yellow or red. If used HSV color domain, we can define that white is V bigger than 192 and S is small than 64. Yellow is S bigger than 60 and H between 40 and 60. Red is S bigger than 60 and H between 350 and 10. Deep black is V smaller than 50 and S bigger than 86. We can get the gray scale image from V and separate the lane object and vehicle object images at this stage conveniently. Figure 3.2 shows that we can separate the vehicle and lane with color transform at the same time.
(a) (b)
(b) (d)
Figure 3.2: (a) Original image. (b) Normalization image. (c) Separated lane image from normalization image. (d) Separated vehicle image from normalization image.
3.3 Extract Lane
In this stage, we need to extract lane markers as much as possible as we will use these lane markers to do the linear fitting. Otherwise, if the markers are not clear enough the result will be wrong. In Figure 3.2 (b), the separated lane image is not clear enough because it contains the wrong markers caused by halo and reflected light. Hence, we need to apply lane filter and lane probed algorithms on the separated lane images. But the lane probed algorithm in this stage has some difference. We do not fill in the gap with negative and positive edge.
That will cause too many lane markers and lead the linear fitting hard to work on embedded system. Thus, we just keep the middle point of negative and positive edge. Figure 3.3 (a) is an extracted lane image.
3.4 Lane Model
The application in this thesis, lane fitting is an auxiliary function. We used lane to find the front vehicle and driving shift. The lane in the high way is closed to straight line even if in curve. If we just went to find front vehicle, the absolute lane fitting is not necessary. In addition, the calculation power of RISC CPU is very week for quadratic curve operation.
Therefore, we just used a linear equation to do the lane fitting. And in our experiment, we proved that a linear equation is good enough in this project. Following is our lane model.
y = ax + b (3.3) After got the separated lane images, we collected the potential lane pixels and did the least squares fitting to derivate the lane model. In this case, the two sets of linear equations Lr and Ll could be then determined by using the information of the right and left lane coordinates.
Take the right lane markings for example, let the extracted points be a set of N observations
= 2 ∑ (y − ax − b)(−x) = 0 (3.8)
= 2 ∑ (y − ax − b)(−1) = 0 (3.9)
−1 × ∑ x y + a × ∑ x + b × ∑ x = 0 (3.10)
−1 × ∑ y + a × ∑ x + b × ∑ i = 0 (3.11)
a × ∑ x + b × ∑ x = ∑ x y (3.12) a × ∑ x + b × N = ∑ y (3.13)
These equations are nicely represented in matrix form. The parameters of the best line are found by solving the equation (3.14).
∑ x ∑ x
∑ x N ab = ∑ x y
∑ y (3.14)
Following shows the liner fitting result:
(a) (b)
Figure 3.3: (a) Lane marker image. (b) Linear fitting result.
3.5 Slope Histogram
Using least squares fitting to derivate the lane model is an efficient way. But this method should be with a necessary condition that a clear lane separated image is required. If we do not get a clear lane separated image like in Figure 3.4 (a) then we will get a wrong linear fitting result such as in Figure 3.4 (b). However, to get an absolute clear separated lane image needs a lot of preprocess on the input images. It is very consumption of the resource on smart phone system. Thus, we did not use linear fitting to find the lane model at the decision stage. Instead, we used the vote algorithm to get the lane model.
In this section, we need to do some things for the accelerating vote speed. We think that, if user set the camera up at the cabin and the view angle is fixed, the slop of right and left lanes is located at two normality values if the driver drives at the middle of the lane. We did the statics of factor “a” in equation (3.3) with some frames and get the slope “a” which is the most voted. Then we defined two new linear equations with capital letter “A” and “B” which is means the maximum voted slope on left lane and right lane.
y = Ax + by = Bx + c (3.15)
After getting these two lane model equations, we need to redefine the road area by equation (3.15). The top boundary “Top” is such as following equation.
Top = A × + b. (3.16) And bottom “Bot” is such as following equation.
Bot = min (b, c) (3.17)
3.6 Lane Voting
In this thesis, we proposed to use the lane voting algorithm to get an accurate lane model equation. The benefit of lane voting algorithm is that we do not need to get an absolutely clear lane marker image but can get accurate lane model equation easily. The first stage is only to get lane marker image and following with the follow chart of getting pre-process lane image for the lane voting.
Figure 3.4: The flow chart of pre-process lane image for lane voting.
In Figure 3.4, the stages of “Image Normalization”, “Lane Detection in HSV Color Domain” and “Apply Lane Filter” are related to previous chapters.
After getting the lane mark images, we can start to do the lane voting process. From chapter 3.4, we have got two equations. One is the left lane model and another is right. The idea is to use these equations to scan full lane mark image and take the maximum matched equation to be the lane model equation. (Figure 3.5)
Get Color Image
Image Normalization
Lane Detection in HSV Color Domain
Apply Lane Filter
.
Figure 3.5: (a) Matching left lane. (b) Matching right lane.
. (a)
(b)
Figure 3.5: (a) Matching left lane. (b) Matching right lane.
.
.
…
.
… … …
Figure 3.5: (a) Matching left lane. (b) Matching right lane.
After getting the maximum matching, we matched lane model is just the closest equation
we fix the parameter “b” in the equation (3.3) and do some addition parameter “a” to get potential maximum match.
the maximum matching, we need to do some angle fine tun matched lane model is just the closest equation to the real lane marker. In our fine tune
in the equation (3.3) and do some additions and subtraction to get potential maximum match. In Figure 3.6 is our matching result.
Figure 3.6: Lane voting result.
need to do some angle fine tunings. The real lane marker. In our fine tune process,
and subtractions on igure 3.6 is our matching result.
The vehicle detection is very important in this system.
in the image demands a complexity algorithm to do it. In this algorithm to detect a vehicle in the
4.1 Detect Vehicle by the Shadow
In the chapter 3.1, we separated
stage. In this chapter, we need to do more calculation to extract the vehicle at the front. We do not care the neighborhood vehicle
up of the lane models. The blue area in the following
4.2 C-means
We used shadow to conjecture shape feature so it is hard to use
the shape. Here, we use C-means to decide the position of algorithm.
V(x, y) = ∑ xi , ∑
N: The markers of vehicle in the searching area.
In our system, we just need to get the y position. So, we can simplify the equation (4.1) as following.
V(y) = ∑ yi
Figure 4.2 is our search result.
Chapter 4
Vehicle Detection
ehicle detection is very important in this system. A robust vehicle detection process a complexity algorithm to do it. In this thesis, we propose
in the front, but it bases on the searching assistance
separated the land and vehicle images at image color transform stage. In this chapter, we need to do more calculation to extract the vehicle at the front. We do
t care the neighborhood vehicles but just the front one. So, the search area is just in the up of the lane models. The blue area in the following Figure 4.1 is the searched area.
Figure 4.1 Vehicle searched area.
conjecture the front vehicle, however the shadow t is hard to use feature definition to make decision about the
means to decide the position of the front vehicle.
yi N: The markers of vehicle in the searching area.
In our system, we just need to get the y position. So, we can simplify the equation (4.1) as
A robust vehicle detection process , we proposed an easy assistance at pre-chapter.
at image color transform stage. In this chapter, we need to do more calculation to extract the vehicle at the front. We do but just the front one. So, the search area is just in the fence
igure 4.1 is the searched area.
the shadow do not have any to make decision about the authenticity of front vehicle. Following is our
(4.1)
In our system, we just need to get the y position. So, we can simplify the equation (4.1) as
(4.2)
(a) (b) Figure 4.2: (a) Real image. (b) Vehicle markers image.
Chapter 5
Auto Calibration
Using computer vision to detect distance from 2D image in previous papers is very complex for users. Generally, it is hard to detect image from 2D image directly. People need to provide camera parameter and the coordinates of camera setup. We provide a user friendly distance measure method assists with GPS.
5.1 GPS
GPS is a common device in the smart phone now. It can provide the coordinate of longitude and time information. We can get the speed and time information through Microsoft standard API and use it to do the automatic 2D distance measure calibration.
5.2 Auto Calibration
In this thesis, we proved an algorithm to do the distance measurement. This algorithm has an important supposition that is front vehicle is const speed cruise. For this reason, we cannot calibration it immediately, we need to static the parameters set which derived from the algorithm in a while. Finally, we chosen the maximum statistic one to be the calibration result.
The accuracy of GPS is not very exact. We do the calibration at once in one second interval.
The parameters got in an image are time, speed and vehicle coordinate in the image. Figure 5.1 is the sketch map of the got parameters.
Figure 5.1: The sketch map of calibration parameter
By the anti-perspective projection, people can map points on the 2D plane to 3D world system. But in our system, users do not need to input camera parameter and setup coordination.
It is hard to do the mapping directly by this theory. In Figure 5.2, we can see that the x-axis project to y-axis on 2D plane is approaching to the parabolic curve. Thus, we propose to use parabolic curve (Y − b) = aX to get the mapping of 2D plane in coordinate with 3D world coordinate system for distance measuring.
Figure 5.2: Perspective projection of ground plane and the relationship of parabolic curve.
Following is our derivation about how to get the parabolic curve approach from three images with 1 second interval by themselves.
Image t1 front vehicle position
Step1. We can get the acceleration “A” by the equation (5.1).
A = (5.1) Step2. The shift of our car “S” is got by equation (5.2).
S = V1 + A
S = V2 + A (5.2) Step3. Then solve the following equation (5.3) to get the parameter “a” and “b” in the
parabolic curve equation (Y − b) = aX .
( ) ( )
(5.3)
=>
a =( ) ( )
a =( ) ( ) (5.4)
=>( ) ( ) =( ) ( ) (5.5)
=> b = (5.6)
In the calibration stage, we needed to do the parabolic curve approach for a while.
Because we expected that the front vehicle was with a constant speed and we had the acceleration. If not fitting in these conditions, the result will be wrong. Therefore, we need to do the approach to static the parameter “a” and “b” for finding the maximum candidate to be our parabolic curve approach result.
Chapter 6
Distance Measurement
After dealing with the parabolic approach curve parameter “a” in the front chapter, we now can get front vehicle distance easily. The decision stage is just to find the front vehicle in the sequencial input images and use the following equation (6.1) to get the real world distance.
X = ( ) (6.1) Figure 6.1 show that we can measure the distance of front car is 120 meter.
Figure 6.1: The example of front car distance measuring result.
Finally, using this distance to make a decision and do some warning alarms to driver when the car is too close to the front vehicle. In our implement, we can make a decision in 100 milliseconds. This should good enough to help driver to avoid some overtaking collision danger.
Chapter 7
Experiment
The aim of this thesis is to build up a Driver Assistance System which is suitable for mobile devices. Currently there are four smart phone systems on the market Windows Mobile, RIM, iPhone and Android. Though iPhone has a complete develop API supporting and well performance but it is belong to high-end smart phone. RIM is just popular at Europe and the developed tools are not very complete. Android was not release when we created this project.
So we chose Windows Mobile system to be our target developing system.
7.1 Hardware Environment Description
Even though Microsoft has define a camera module access path from Direct Show after windows mobile 5.0, but there are still not many window mobile system smart phones supporting this feature. After having done the many tests, we just found that the HTC has supporting Direct Show completely. Therefore, we chose HTC Touch Diamond P3702 to be our developing environment. Figure 7.1 is its appearance.
Figure 7.1: HTC Touch Diamond P3702
Table 7.1 shows the detailed description of the system, it consists of the processor, memory, camera and GPS.
Table 7.1: Hardware system information.
Camera 320 Mega Pixels with auto focus
GPS GPS and A-GPS
7.2 Software Environment Description
Microsoft has a complete windows mobile developer center website.
http://msdn.microsoft.com/en-us/windowsmobile/default.aspx Our software system is listed at following table 7.2.
Table 7.2: Software system information.
Component Information
OS Windows Vista SP1
IDE Visual Studio 2008 Professional with SP1
SDK Windows Mobile 6 SDKs
Synchronization Windows Mobile Device Center .
7.3 Get Image by Direct Show
Getting image from camera is very hard from the previous to the present. Generally, the smart phone developer needed to get the camera SDK from camera vender for camera application development. But it is not release to end users. So we need to get image by Direct Show. Direct Show is a convent technique, users only need to assemble the components and finish its work. In Windows Mobile 6 SDKs, it supported “Camera Driver ” and “Video Capture” filters for getting image from camera. But, unfortunately it just has “Video Render” filter for display. It lacked “Video
Getting image from camera is very hard from the previous to the present. Generally, the smart phone developer needed to get the camera SDK from camera vender for camera application development. But it is not release to end users. So we need to get image by Direct Show. Direct Show is a convent technique, users only need to assemble the components and finish its work. In Windows Mobile 6 SDKs, it supported “Camera Driver ” and “Video Capture” filters for getting image from camera. But, unfortunately it just has “Video Render” filter for display. It lacked “Video