Piece-Wise Edge Linking Model

Qing Li [22] and C. H. Yeh [24] still apply the Hough transform to track the lane markers which can not be deformed in the image captured by the normal camera.

However, due to the distinct curvature with the fish-eye lens, it is impossible to take Hough transform into our system. Hence, the novel approach for lane modeling needs to be considered the geometric effect of ROI and the connectivity of the lane markers with robustness and adaptation.

The flow chart of the piece-wise edge linking model is shown in Fig. 3-16(b).

(a) (b)

Fig. 3-16 : (a) Seven sub-regions automatically segmented within ROI. (b) The flow chart of the piece-wise edge linking model.

(a) (b) B

C D E F G

A

Fig. 3-17 : (a) Seven sub-regions segmented within ROI. (b) The flow chart of the piece-wise edge linking model.

Figure 3-17 shows the two different size of ROI is caused by the variation of the intrinsic and extrinsic setting of camera. In general, the width of ROI depends on the yaw angle of camera, and the height of that depends on the pitch angle or the distance from the mounting position near the rearview mirror to the road plane. Although those parameters can not be taken in our system, we still find the property that the lane boundary in the image must extend to the upper-left part of ROI even if the lateral position estimated from the lane marker is not the same through the image sequences.

KeyAngle−RegAngle>θTH

Fig. 3-18 : The flow chart for finding the line-shape in the bottom sub-region (A).

2

KeyAngle RegAngle− >θTH

Fig. 3-19 : The flow chart for finding the line-shape in sub-regions from (B) to (G).

By using the perspective effect that lane markers almost converge near the region of vanishing point, the included angle from the diagonal of the ROI to the vertical boundary of that can be determined the maximum searching range of angles for Hough transform. This mechanism will be regarded as the initial step in the piece-wise linking model as shown in Fig. 3-16(b). To overcome the irregular curvature of lane trajectory from the fish-eye lens distortion, the seven sub-regions automatically segmented in Fig. 3-16(a) contribute to fit the edge pixels of lane since its boundary information contained in it can be regarded as the line-shape. Therefore, the principle of Hough transform described in Section 3.4.2 is directly used for the bottom sub-region (A) as demonstrated in Fig. 3-17. The details of parameters in Fig.

3-18 and Fig.3-19 are explained as follows:

St_X, Ed_X:

The coordinate values of x-axis in the bottom and top border of the sub-region determined by Hough transform. Ed_X situated in the bottom border of the next sub-region, such as the same location as the bottom border of sub-region (B) and the top border of sub-region (A), can become the fixed point for searching the line edge pixels only by the angle θ as the flow chart in Fig. 3-19.

SkipTh:

Its size depends on the vertical pixel-width of the sub-region (A) in Fig. 3-18.

For some circumstances like the rapidly lane changing maneuver, the lane marker may be discontinuous for each sub-region in the image. The threshold is to control when the lane modeling procedure is performed and observe if the edge pixels in the bottom sub-region (A) have adequate amounts to composite the lane trajectory.

KeyAngle, RegAngle,θ_TH,θ_TH2, δ, Δθ, Lw:

KeyAngle and RegAngle are the angles about appropriate orientation of line boundary in sub-regions induced by the current and previous frame. Based on the connectivity and continuity of lane markers on the road surface, and are the thresholds to limit if the difference between KeyAngle and RegAngle is small enough. In addition, must be smaller than since the searching angles with sub-region (B) to (G) is restricted by the previous detecting results from the bottom sub-region (A). δ and Δθ are the slight range for detection with Hough Transform from sub-region (B) to (G) where the computation power can be reduced. At last, Lw is a revised parameter to restart the seeking area in the x-axis when the number of line pixels is zero in Fig. 3-19.

θTH θ_TH2

TH2

θ θ_TH

To simply the geometric circumstance that the distance between the vehicle and lane trajectory with some curvature in the image is much different, especially the effect of fish-eye lens distortion, we use LSR (least square regression) to make the curved a lane boundary approximate a straight line. The LSR can be induced as below:

Equation (3-13), (3-14) can be simplified as

{ ^∑ ∑

^∑ ∑

^∑

a X b X X Y

a X b N Y (3-15)

2

Fig. 3-20 : LSR approximation.

According to the parameter information showed in Fig. 3-20 (a), the linear model can be constructed by the equations (3-16). The approximating straight lane boundary is displayed in Fig. 3-20 (b), which is directly reflected since the image contents acquired by the camera mounted on the opposite side of the vehicle are almost the same except for the reflective property.

4 Chapter 4

Lane Departure Warning and Drowsiness Estimation

4.1 Overview

Fig. 4-1 : The flow chart for the whole system.

After extracting the lane boundary from the previous chapter, the lateral information of lane markers can be used to judge when the lane change maneuver occurs for the driver. In this chapter, the LDW (lane departure warning) system is constructed by measuring the displacement, instantaneous velocity and TLC (time to lane crossing) of the lane to form the warning triggers for alarms. The standard for drivers’ drowsy state is based on the reflection time when people start to turn the

steering wheel after they drive off the stable-state regions which are dependent on the habit of them for strait-line driving. Trying to combine the experimental data from BRC (brain research center) in NCTU with the realistic scenes, and a gauge of drowsy degree will be proposed to show the possibility with respect to the drowsiness of drivers. The flow chart of the whole system is shown in Fig. 4-1.

4.2 Lane Departure Warning

4.2.1 The Warning Algorithm

As described in Section 1.2.2, some algorithm has been developed to predict when the driver is in danger of departing the road but not annoy the driver sensitively.

In other words, extending the interval of warning time can receive the more correct driving maneuver, but the number of nuisance alarms will increase apparently. Lee [26] and Ruder [31] considered that LDW does not necessarily need the precise offset and position information from each frame to add the computing load since it only assists the human driver and passively responds to the circumstance such as when the lane-departure occurs. In order to balance the systematic efficiency and acceptable detection rate in our LDW system, only two representative measures are selected to trigger the warning message. The two judging conditions are discussed as follows:

(1) Lateral displacement:

If the lane boundary is excessively close to the vertical borders of ROI, the driver will be in danger with higher possibility. We will regard this as a dangerous departing behavior even if it may be only someone’s habit of driving. There, the safe region which contains the normal lateral offset of lanes is defined as follows.

1 4

(2) TLC (time to lane crossing):

TLC which was first proposed by Godthelp [35], is a measure of the time remaining before a vehicle on a given trajectory will depart the road. It can provide more reliable information than the lateral position merely due to the factor for lateral velocity can be considered. In our system, the definition of TLC is a ratio of lateral offset smaller than the width of ROI to the lateral velocity at the moment.

The classification for the dangerous degree of warning alarms and the deducing process of TLC are explained in details in the next section.

4.2.2 Evaluation

To prevent the noisy effect such as high frequency variances of the lateral offset of lane markers in each frame from measuring error, we take five frames processed by lane detection to estimate only one weighted average result for departure judgment such like a causal temporal filter. (In practice, there is always one frame only for Gaussian smoothing between the two frames used for lane detection in our system. In other words, consecutive five numbers of lateral positions occupy about 0.33 seconds for 30fps.) The values of weights are {0.22, 0.21, 0.20, 0.19, 0.18} from the present and the last four processed frames. The flow chart for TLC computation is shown in Fig. 4-2.

By the two obvious measures, the degree of departure warning can be classified with the color of alarms as the following:

1 4 If the current lateral offset is inside the safe region: { ROI, ROI}

4 5

Fig. 4-2 : The flow chart for TLC estimation.

4.3 Drowsiness Estimation for Driver

In recent years, preventing accidents caused by drowsiness has become a major focus of active safety driving in recent years. The major challenges in developing a real-time system for drowsiness prediction include: 1) the lack of significant index for detecting drowsiness and 2) complicated and pervasive noise interferences in a

realistic driving environment. Therefore, the BRC (Brain Research Center) in National Chiao Tung University has developed a drowsiness-estimation system based on electroencephalogram (EEG) to estimate a driver’s cognitive state when he/she drives in a virtual reality (VR)-based dynamic simulator. The definition of the driving error in this experimental environment is the deviations between the center of the vehicle and the center of the cruising lane in the lane-keeping driving task.

In this section, the system architecture of BRC will be introduced in Section 4.3.1. The relationship between the reaction time and driver’s drowsiness will be explained in Section 4.3.2. Before trying to reasonably and effectively integrate the measuring result from the VR-based driving environment into the lane detection system in this thesis, some changeful factors of the realistic image-based system must be discussed. In Section 4.3.3, a stable-state range can be constructed to determine the lane’s lateral position where someone gets used to driving in a straight road-path for a long time. Then, a gauge of the drowsy degree successfully combine the experimental result evaluated by the EEG-based analysis [27] with the realistic and dynamic LDW system successfully is proposed in this thesis, as described in Section 4.3.4. Finally, in order to adaptively extend the experimental framework to the practical driving environment, we estimate the average velocity within the interval of reaction time by deducing the ratio of the lane-width on the realistic road plane to that in the video, as explained in Section 4.3.5.

4.3.1 Experimental Architecture of BRC

In general, measuring the precise data for human consciousness in dynamic driving environment is not easy. There may be some perturbations from the external noise or suddenly interference caused by the traffic variations affecting the data

accuracy. In other words, strict training of human operators by the actual machines or vehicles in real sites not only has high demands in space, time, and money to perform such a training job, but also leads to another phase of the measuring problem. To overcome the above dilemma, the worldwide trend is to use the virtual-reality (VR) technology to meet the requirements of public security in training and censoring of human operators. It can provide a realistic safety environment, which allows subjects to make on-line decisions by directly interacting with a virtual object rather than monotonic auditory and visual signals. Besides, VR is also an excellent candidate for brain research on real-time tasks become of its low cost, saving time, less space, and condition control to avoid the risk of operating on the actual machines, and thus extends the applications of possible brain computer interfaces to general populations.

Fig. 4-3 : The VR-based dynamic driving simulation laboratory.

Fig. 4-4 : The details about the width information of each lane, road, and car.

The experimental environment constructed by BRC is shown in Fig. 4.3. The VR-based four lane highway scene is projected on a 120 degree-surround screen (304.1-cm wide and 228.1-cm high), which is 350 cm away from the driving cabin.

The four lanes from to right are separated by a median stripe. The distance from the left side to the right side of the road is equally divided into 256 points (digitized into values 0-255), where the width of each lane and the car is 60 and 32 units, respectively. The frame rate of highway scene is 60 fps. All the descriptions are depicted in Fig. 4-4.

4.3.2 Predictive Mechanism for Drowsiness Effect

Before executing the experimental step, we have to find the relationship between the measured EEG signal and the subject’s behavior performance. One point should be taken as a quantified index as the deviation between the center of the vehicle and that of the cruising lane [36]. By examining the video recordings, the pilot experimental studies show that when the subject is drowsy, the driving performance will decrease and vice versa. In this experiment, the subjects participated in the highway-driving simulation after lunch in the early afternoon when the alertness may easily diminish within one-hour monotonous working [37].

All the subjects were instructed to keep the car at the center of cruising lane by controlling a steering wheel. In all sessions, the subjects drive the car continuously for 60 minutes and were asked to try their best to stay alert. Participants then returned on different days to complete a second 60-minute driving session or more if necessary.

To mimic the consequences of a non-ideal road surface, the car is randomly drifted away from the center of the cruising lane every 5 or 10 minutes. So the driver must maintain high attention to immediately correct the direction of vehicle in the cruising

lane due to the 60 pixels per second for the deviating velocity. When the driver is drowsy, the reaction time between the onset of deviation and steering wheel is increased. This event can be used for ERP analysis of different drowsiness states using 30-channel EEG signals [27].

In general, the reaction behavior should be increasingly slower when people start to enter the drowsy state. In other words, the higher possibility for the measurement shows that the subject is drowsy when his/her average reaction time is gradually longer in a section of time interval. To avoid the fluctuation of drowsiness signal, the measured data for reaction time must be smoothed by a causal 90-second square moving average filter advancing at 2-seconds steps. The experimental trials are sorted according to the length of reaction time and equally divided into five groups as the index for drowsiness estimation in Fig. 4-5, where each group has 20 percentages of trials in order. This statistics evaluated by the EEG analysis [27] can be regarded as the reference implemented into our vision based lane departure warning system.

Fig. 4-5 : The trials collected from the VR-based experiment are sorted according to the degree of reaction time.

4.3.3 Construct the Stable-Driving Region with Different Driver’s Habit

According to the above experimental condition, the definition of reaction time is the duration between the onset of deviation and the occurrence for steering-wheel.

Subjects have to move the vehicle’s center back to the cruising lane to wait for the next testing deviation produced by the computer when they have been informed in advance. However, the restarting action is not easy to be determined due to the variation of different driving habits, especially the loose drivers which have a larger spread in lateral position so that the distance between the wheel and lane marker can not exactly fixed in the straight-road driving [15]. Therefore, the algorithm to extract the stable-state driving region must be developed before constructing the drowsiness estimation mechanism.

The standard for stable-state range determination is described as below: (1) the lateral position of lane markers within this region should be close to each other; (2) the TLC is larger; (3) The lateral offsets found by the LDW system in Section 4.2 must be situated in this region for a long period.

According to the above properties, first of all, we take the lateral offsets with larger TLC about consecutive N frames processed by the LDW system. Second, by the previous statistics, the mean and standard deviation estimated by them with the clustering method are used to model the region as a normal distribution. At last, the updating method is developed to adjust the size and location of the range to the changed driving habit for a driver. The flow chart for stable-state region determination is demonstrated in Fig. 4-6.

To rapidly and precisely find out the optimal parameters of each normal

distribution, we choose k-means to initially classify the statistics of N lateral offsets.

Fig. 4-6 : The flow chart for stable-state region determination.

The error function which determines the clustering center point of each group is shown as follows:

In Fig. 4-6, μ is the mean value of each distribution; δ is the standard deviation of each distribution; w is the weight determined by the probability of each group.

After initializing for each distribution model, we find that the N lateral offsets can be approximately modeled by only three normal distributions, which are respectively located on the points nearby the mean value and 1.5 standard deviations with high probably, as shown in Fig. 4-7. Therefore, we choose K=3 as the initial clustering numbers.

0

Fig. 4-7 : The distribution of N lateral offsets and three approximately Gaussian model (N=200 in this Figure.)

Not the same as the adaptive background model [32], the human habit can last within a steady behavior style for a long time. Based on this psychological property, we only use a single normal distribution with some update mechanism to model the adaptive stable-state driving region to avoid its unreasonable fluctuation. Updating the parameters of the stable-state model can adapt to the changed driving habit if the lateral offset is within 2.25 standard deviations of this distribution. The parameters of the distribution which matches the new observation for human habit are updated as follows:

By observing equation (4-5), the influence for this stable-driving distribution will be unapparent when the distance between the current lateral offset and the mean value of the model is so far. This property can effectively maintain the stability of this region.

4.3.4 Data Collection and Adjustment for the Realistic Environment

After selecting the suitable driving region for the driver, the experimental statistics evaluated by EEG analysis from BRC can be integrated into our lane departure system. Not the same as experimental condition which stipulated that the reaction behavior can be increasingly slower when the subject starts to enter the drowsy state by observing the trend of reaction time for a long period (about 90 sec), the demand for drowsy estimation mechanism in our system should provide real-time prediction if the driver is still on the alert. Therefore, we design a gauge chart to estimate and display the current driver’s drowsy degree as much as possible, as shown in Fig. 4-8 (b).

In Fig. 4-8 (a), the difference in lateral offset between (B) and (C) is 52.45 pixels, the mean value (A) of stable-driving region is located at pixel value of 123.23, and the reaction time counted from (D) and (E) is 1.65sec, as shown in Fig. 4-8 (c).

As described in Section 4.3.2, the definition of reaction time is the time interval of deviation between the center of the vehicle and that of the cruising lane in the VR-based experimental environment. In other words, the value of deviation can be the

As described in Section 4.3.2, the definition of reaction time is the time interval of deviation between the center of the vehicle and that of the cruising lane in the VR-based experimental environment. In other words, the value of deviation can be the