Chapter 1 Introduction
1.4 Organization
This thesis is organized as follows. In Chapter 2, the camera configuration and the vehicle blind-spot is introduced along with the preliminary knowledge of the vision-based system. Chapter 3 describes our algorithm of lane detection. The approaches about lane departure warning and drowsiness prediction of the driver are proposed in Chapter 4. The experimental results are exhibited in Chapter 5. Finally, the conclusions of our system are presented in Chapter 6.
2 Chapter 2 Preliminary
In this chapter, the preliminary knowledge of the whole system will be introduced. In the beginning, the difference of the visual characteristics between the frontal and lateral place mounted on the camera is discussed. Then, the vehicle’s blind spot based on the intrinsic and extrinsic limitation of the human and rearview mirror will be described. The related principles of the lane detection and departure warning system are presented finally.
2.1 Camera Configuration
In this section, the geometric relationship and transformation between the image coordinate and the realistic vehicle coordinate are explained in detail. Furthermore, the applications about the frontal and lateral view of the camera are introduced here.
2.1.1 Perspective Geometry
To extract the image information of road plane on the side of the vehicle, a single camera is mounted near the rearview mirror. In the vision-based configuration, each objects captured by the image sensor in the camera coordinate system can be projected onto the image pane in the image coordinate system. This geometric relationship can be described as the perspective projection, and the camera configuration for the proposed system is shown in Figure 2-1 with the height of the camera H and the tilt angleα.
α
H
Fig. 2-1 : Camera configuration.
Before computing the transformation between the image coordinate and the vehicle coordinate, some assumption must be established. At first, the condition in this section is only considered that the ground plane is almost flat. In general, we ignore the specific environment when the vehicle drives on the mountain road or other rugged surface. Second, the optical distortion of the camera lens can not be considered in this deduced process of the geometric transformation.
The spatial relationship between the vehicle coordinate and image coordinate system are shown in Figure 2-2. For practicality, the pan and tilt angle of the camera must be taken into account for this systematic configuration. The tilt angle α is an included angle from road plane to the optical axis. On the other hand, the pan angle
β is an included angle from the moving direction of the vehicle (Y-axis) to the projection of the optical axis onto the road plane. In general, the camera can be modeled as the pin-hole model. The distance between the optical center (OC) and the central point of the image plane ( , ) determines the focal length. Moreover, according to the known camera height H and the information of pan-tilt angle, we can deduce where the object contained by the road surface are projected onto the image plane from the perspective geometry.
u0 v0
Fig. 2-2 : Vehicle and image coordinate systems.
For further analysis, we discuss the spatial relation between the vehicle coordinate and the image coordinate system through two different points of view.
Figure 2-3 and Figure 2-4 are the side view and bird’s eye view of the geometric chart between the vehicle coordinate and the image coordinate system.
Fig. 2-3 : Side view of the geometric relation between the vehicle coordinate and the image coordinate system.
Fig. 2-4 : Bird’s eye view of the geometric relation between the vehicle coordinate and the image coordinate system.
Before explaining the formulation of the transformation between the two coordinate systems, some annotation must be introduced about Fig. 2-3 and Fig. 2-4 in advance.
f: Focal length of the camera
f 、u f : The scaling factors of the image plane in the horizontal and vertical axis v H: The distance from the road plane to the camera
α : Tilt angle of the camera β : Pan angle of the camera
( , ): The corresponding point in the image plane is projected from the road surface u v
0 0
( , )u v : The central point of the image plane
In Fig. 2-3, ΔCPR and are similar triangles. With this property, the spatial relation between two coordinate systems can be derived as follows.
' '
0
= f is the camera constant of the image plane in the vertical axis
0
Similar to the above process, the deducing details about Fig. 2-4 will be described in the following.
= f is the camera constant of the image plane in the horizontal axis
0 tan
Equation (2-1) and (2-2) are the transformation from the point ( u , ) in the image plane to that (x, y, 0) within the road surface in the vehicle coordinate system.
However, parts of the parameters in this formulation are unknown. We must make use of some probable approach to estimate those values if the precision of the perspective
v
phenomenon is adequate.
2.1.2 Applications of Frontal-View Lane Detection
For several years, many researchers worked on driving assistance problem by using the concept of artificial vision. Among those applications about intelligent transportation, automatic navigation has been taken seriously in recent years. For accomplishing this objective with efficient performance, most vision-based methods are to extract the road information by mounting the image sensor on the windshield of the vehicle. Such as the human eye, forward-looking camera can extract the widest field of view than other mounting position around the car body. In general, by detecting the contrast between the white lines and the road, the lane boundary in front of the vehicle can be sufficient to detect. Furthermore, the variation of the lane’s curvature can be predicted in time without resulting in the erroneous following of the vehicle. Besides, some obstacles captured by the camera can be recognized with 2D or 3D techniques of computer vision. Other related works such as keeping the secure distance ahead of the car are based on this system configuration.
2.1.3 Applications of Lateral-View Lane Detection
Some risks of road traffic which occur on highway during the lane-changed maneuver happen easily if another vehicle besides the own one has been overlooked.
In other words, drivers have not assured accurately if there is no other vehicle alongside in the blind spot of the lateral view. During the general driving procedure, drivers must keep notifying the frontal field of view so that they forget to check the information of the lateral blind-spot at the same time. In order to overcome this kind
of traffic hazard with efficiency, a camera is mounted at the driver’s outside rear-view mirror to monitor the blind spot and the alongside lane. Approaching vehicles should be detected in time and tracked until they leave the blind spot by this configuration. In addition, this system can restrain the intended lane -changed maneuver and maintain the distance from the lane boundary in the blind spot to the realistic car body without a significant amount of the potential collisions.
2.1.4 Comparison with Two Applications on Vehicle
In addition to the distinct effect of the geometric projection onto the image plane, there are still other different factors and applications between the frontal-view and lateral-view configurations. The four reasons are listed as follows.
(1) The initial purpose has influence on where the camera is mounted:
As explained in section 2.1.2 and 2.1.3, road images extracted from the forward sight of the vehicle can yield more driving information to track the real-time road curvature by the lane-marking modeling. Furthermore, the related data of them has effectively contributed to the system with respect to the assistant navigation.
On the other word, the major objective about mounting the camera on the side of the car is to adjust how much is the detecting range of the blind-spot region. This configuration only puts emphasis on judging the approaching car or the lane trajectory near the vehicle, and the variation of the forward road information can be not considered.
(2) The diverse sensation of the driver with respect to two mounting position:
In general, to extract the forward visual information as far as possible, the camera was almost fixed to the windshield. This setting location could easily reduce the eyesight of the driver whether the size of the camera is so small or not. The
disadvantage resulting from the driver’s unfamiliar looking will be concerned with the research about driver analysis. Nevertheless, due to the position of the camera near the rear-view mirror when focusing on extracting the lateral-view content of the vehicle, drivers can be not confused with this experimental environment. In other words, the camera added to the vehicle can not affect the original driving habit of the driver, and the data collected by driver analysis system will still be higher accuracy.
(3) The different extrinsic factors of two locations of the sensing device
Compared with the initial purpose of two configurations, the camera mounted in front of the vehicle must have farther distance from its optical center to the specified lane portion on the road plane than that on the side of the car because of the perspective geometry. In addition, the overtaking cars which crossing the lane are almost captured by the frontal-view image sequences. Therefore, the information of the lane trajectory extracted by the sideward camera can be more complete than the forward one throughout the driving experiment on highway.
However, with the headlight switched in the gloomy driving situation, the video collected by the frontal camera can still hold more acceptable luminance information in night vision.
2.2 Definition of Vehicle Blind Spot
Blind spots, in the context of driving an automobile, are the areas of the road that cannot be seen while looking forward or through either the rear-view or side mirrors.
Detection of vehicles or other objects in blind spots may also be aided by systems such as video cameras or distance sensors. Throughout the notation in this thesis, the
area of blind spot is only regarded as the rear of the vehicle on both sides. The introduction in this section not only describes the causes of traffic accidents resulted from the blind spot, but discuss how to resonablely establish the region of blind spot by the inherent limitation of the human vision and rear-view mirror.
2.2.1 Traffic Accident Causes of Vehicle Blind Spot
In Taiwan, the types of traffic accidents between two cars on highway are listed in Table 2.1 from [17].
Table 3 : Causes of traffic accidents between two cars on highway.
Year Collision by the Backward Car
As shown in foregoing statistics, we can conclude that the lateral and rubbed collisions are both the principal causes of the traffic accidents between the cars. There have been numerous topics focused on how to avoid the forward or backward collision for the vehicle, but the related research for lateral collision is little. When vehicles in the adjacent lanes of the road fall into the range of lateral blind spots, the driver will be unable to see them with only the car’s mirrors. Due to the above reason, drivers must actively rotate their head to extract more information within the region of blind spot. However, the probability of car accident can be raised simultaneously.
Therefore, vision-based system can be developed to assist the drivers in keeping away from the lateral danger of vehicles by the image sensor alongside the rear-view mirror.
2.2.2 Limitation of View by Human-Vision
The eyesight of people has obvious difference between the static and dynamic environment due to the variation of the vehicle velocity. In general, the view-angle of the single eyeshot is about 160 degrees when people lie in the stationary scene; the maximum view-angle of the double field of view is enlarged about 180 degrees.
Flannagan [18] proposed that the people’s double eyesight should reach to 320 degrees by adding the rotating motion for the head and body of human. According to the statistics from [19], the realistically clear field of view contained two eyes is only about 70 degrees when a normal person situates in the static environment.
Nevertheless, the human’s eyesight could frequently vary when people are in the dynamic conditions such as the internal part of the moving vehicle thanks to the tunnel-vision effect. The relationship between the range of human eyesight and the variation of the vehicle velocity is in Table 2.2; the range of field of view between the static or dynamic environment is shown in Fig. 2.5.
Table 4 : The relationship between the field of view and the vehicle velocity.
Speed (km/hr) 40 70 100
Field of View (degrees) 100D 65D 40D
70km/hr 65D 100km/hr 40D
40km/hr 100D
210D
Fig. 2-5 : The diagram of the driver’s field of view.
As the information shown in Fig. 2-5, the eyesight becomes greatly narrow when the vehicle is driven at high speed. In other words, the driver can not judge whether there are other vehicles moving on the adjacent road surface or not only by his/her remaining eyeshot on highway as the car velocity raises to 100 km/hr. In this way, drivers induced by the blind-spot hazard will be easily in danger.
2.2.3 Limitation of View by Rear-View Mirrors
In general, the side mirrors of the vehicle are almost used by the planar type.
Therefore, the formation of image about the normal rearview mirror is still followed by the principle which describes that the angle of incidence (θi) is the same as that of reflection (θr). In other words, the field of image produced by the rearview mirror is stretched to 2θ (θ =θi=θr) view-angle projecting into the road surface.
The relationship between the field of view of the side mirror and that of the driver is shown in Fig. 2-6.
70km/hr 65D
40km/hr 100D
100km/hr 40D
Fig. 2-6 : The relation about field of view between the side mirror and the driver.
By the geometric relation from Fig. 2-6, when driving at high speed, in order to make the eyesight overlap the reflected field of rearview mirror, the driver must rotate his/her head so as to extract the lateral information as much as possible. However, due to this unnatural motion, the driver’s inattention will not keep his/her eye for the forward state of the vehicle for a long time with the occurrence of traffic accident.
There are two general approaches to extend the range of field of the rearview mirror. The first approach is to increase the distance between the side mirror and the driver. Due to the fixed car-body, this improving effect will be restricted. The second approach is to replace the traditional planar mirror with the curved one. Nevertheless, the distortion effect of the reflected image will be serious due to the curvature of the lens. Through the above discussion, the blind-spot region between the side mirror and the driver can not be easily resolved. For this reason, adding the camera on the side of the car with intelligent vision-based algorithm will still be regarded as the important device of the assistant system for the driver’s safety.
2.3 Principles of Lane Detection
The objective of lane detection method we expected in this thesis is to extract the lane markers without knowing the internal or external parameters of the camera alongside the vehicle in advance. Besides, the sensitivity of the image sensor easily disturbed by the light condition must be suppressed as much as possible. Therefore, developing an adaptive lane-finding system is essential to satisfy the previous demands. First, our system can automatically extract the ROI contained by the road surface only by the image content despite the unknown environmental information of camera. Second, the preprocessing tasks will be able to effectively restrain the noise when driving in the nighttime. Through the property for the view-angle of blind spot, the improving edge operator will be added to acquire the clear lane boundary. Not depending on the distortion of the camera lens which results in the obviously curved lane trajectory even if people drive on the straight road, a piece-wise edge linking model will be developed to mark all information of lanes shown in image sequence.
2.4 Principles of Lane Departure
Warning and Drowsiness Prediction
The part for lane departure warning is to provide some triggers for caution with respect to the driving-off-road behavior through the lateral information of the lane extracted by the lane detection algorithm. After measuring the lateral velocity from the consecutive frames, the warning system will determine when the departure driving occurs based on the lateral displacement and TLC (time to lane crossing.)
On the other hand, the part for drowsiness prediction will try to combine the
experimental results of BRC (brain research center) from NCTU with the realistic driving video. In order to estimate the lateral location of lane where the driver gets used to navigate on the straight road, we construct the single Gaussian model to simulate the stable-state range about the lane position. Then, the additional updating mechanism will contribute to the systematic adaptation even if the driver changes his/her driving habits. At last, the proportional gauge of the drowsy degree we proposed will show if the driver has higher or lower probability in the drowsy state at that moment with the amount of reflection time measured by the lane position over the stable-state region.
3 Chapter 3
Lane Detection 3.1 Overview
Figure 3-1 shows the flow chart of lane detection. At the beginning of this architecture, because we merely aim at the monochromatic information of each frame to process, the RGB coordinate will be transformed into the YCbCr one so that the illumination component will be totally retained. Then, the automatic mechanism about searching the ROI (region of interest) of the image content will be described in Section 3.2.1. The preprocessing step about de-noising will be presented in Section 3.2.2.
Next to the processing step, the flow will enter the principal detection parts. Due to the mounting position of camera on the side of the car, the image captured by that device will contain most of the lateral-view information next to the wheels. In other words, only one lane trajectory which is the most closed to the vehicle can be apparently seen. An edge detection operator will be developed to adapt to the geometry relationship of the camera based on the property of view-angle in Section 3.3. In addition, the binarization step we proposed in this section will depend on the spatial relation with respect to the perspective effect. To eliminate the blind-spot region as much as possible, we choose the fish-eye camera for enlarging the field of view with some obvious distortion result. Therefore, the adaptive edge-linking model demonstrated in Section 3.4 will overcome the serious problem whether the lane boundary in the image sequences is straight or not.
Fig. 3-1 : The flow chart of lane detection.
3.2 Preprocessing
3.2.1 Automatic ROI Extraction
Before discussing how to search for the lane-marking, the step of color transformation must be executed. In general, most of the algorithms shown in the past theses with respect to lane detection are only considered the grey-level component.
Before discussing how to search for the lane-marking, the step of color transformation must be executed. In general, most of the algorithms shown in the past theses with respect to lane detection are only considered the grey-level component.