The Alignment of LMR Boundary Window

In this section, we describe how to accomplish the differential energy optimization and align the LMR boundary window to the appropriate location when it deviates from boundary. The system accumulate boundary feature in Mn frames before initiating alignment. Therefore, LMR boundary window doesn’t need to align to boundary every frame in order to increase the consuming time and can accumulate more comprehensive boundary feature. When the system capture the Mn frames, it accumulates boundary energy of L, M, R blocks in the LMR boundary window and updates the color model of each LMR boundary window shown in the Fig. 4-13.

Fig. 4-13: The flow chart of alignment of LMR boundary window.

Fig. 4-14: Left boundary is continuous line with comprehensive boundary energy (c) and region ratio (a); The LMR boundary windows in yellow circle on right boundary (b) don’t accumulate boundary features, such as boundary energy (d) and region ratio (a).

After accumulating Mn frames, it starts to align LMR boundary window from last position to current boundary position. Firstly, it needs to check whether the boundary has passed through the LMR boundary window. If there is no boundary passing through, the window will not accumulate boundary feature. For example, when boundary is constituted of broken lines, some LMR boundary windows are absence of boundary feature in Mn frames shown in Fig. 4-13. Therefore, we proposed an exclusive accumulator window for each LMR boundary window. The accumulator window collectively records pixels belonged to the boundary in a series of subsequent Mn frames. If any of the accumulator windows has detected the boundary as white pixels in each window shown in Fig. 4-15, it will process the boundary alignment (optimization). Alternatively, in an absence of boundary, it will

process boundary detection (searching). Next, we illustrate how to achieve boundary alignment (optimization) and boundary detection (searching) as follows I and II.

Finally, if the LMR boundary window align to a new position, it will be verify by alignment checking as follows III.

Fig. 4-15: Checking the accumulator window which records the boundary passing through the LMR boundary window. If any of the accumulator windows has detected the boundary, it will process the boundary alignment (optimization). Alternatively, it will process boundary detection (searching).

I. Boundary Alignment (Optimization)

Any of the accumulator windows has detected the boundary as white pixels in each window shown in Fig. 4-15, it will process the boundary alignment. This step is called optimization because we align the M block of LMR boundary window to position of boundary if the boundary deviates from each M block. When the boundary resides in M block, it is most stable. Therefore, the boundary alignment process is like the idea of optimization.

Firstly, we can utilize the boundary energy to check whether the boundary has deviated from M block. If maximum boundary energy is not resided in M block, it represent the deviation of boundary from the center of LMR boundary window. The

[ ]

( ) ( ( ), ( ), ( ))

( ) arg ( )

represent any of theLMR boundary window

i

MovingEnergy i Max BEnergy L BEnergy M BEnergy R MovingDirection i MovingEnergy i

i

=

=

relative moving direction, MovingDirection(i) shown in (4-4), of boundary and LMR boundary window is determined by where the maximum boundary energy resides. If the MovingDirection(i) is either R(right) or L(left), we relocate the position of boundary by following the MovingDirection(i). Next, we will discuss the relocate method.

(4-4)

The relocating process is tracking the boundary location in the last frame of a series of subsequent Mn frames. We use an example to explain the relocating process.

When the boundary moves to the right side of a LMR boundary window shown in Fig.

4-16 (a) to (b), the accumulator window have detected the boundary moving trace as yellow pixels in Fig. 4-16 (c). Therefore it processes boundary alignment. The MovingDirection of a LMR boundary window is detected as R(right) because of its maximum boundary energy shown in Fig. 4-16 (d). As a result we have realized the boundary moves to right side and we will track the boundary from the position of R block. Firstly, we detect whether the boundary resides R block of LMR boundary window in the last frame called current window checking. If the current window detects the boundary shown as yellow pixels in current window shown in Fig. 4-16 (e), we find the last position of boundary by the mean of the x-axis position of every boundary pixels. Finally, we align the LMR boundary window to the last position as Fig. 4-16 (g) to (h) to complete the boundary alignment (optimization) process.

Moreover, if the position of boundary moves extensively in a series of subsequent Mn frames during encountering sharp curvature, the last boundary position might deviate from LMR boundary window shown as Fig. 4-16 (i) to (j).

Therefore the current window setting at R block will detected no boundary

appearance shown in Fig. 4-16 (k). Then we set the current window on the outer of R block shown in Fig. 4-16 (l) , the size of current window is same as the block of this LMR boundary window, and keep tracking forwarding MovingDirecition, right side, until the current window detect the boundary and using the same method mentioned above to align the LMR boundary window to the last position as Fig. 4-16 (g) to (h) to complete the boundary alignment (optimization) process.

(a) (b)

(c) (d)

(e) (f)

(g) (h)

(i) (j)

(k) (l)

Fig. 4-16: The steps of boundary alignment (optimization) II. Boundary Detection (Searching)

On the other hand, if there is no boundary passing through the LMR boundary window in the series of subsequent Mn frames, its accumulator window detects no boundary trace as yellow pixels in Fig. 4-16 (c) but as Fig. 4-17. This situation can be attributed by the following two reasons. Firstly, the boundary is constituted of broken line shown in Fig. 4-18 (a). Secondly, the boundary has deviated from LMR boundary

window shown in Fig. 4-19 (a). Therefore, it processes boundary detection (searching) procedure to find where the boundary deviates to or to check whether the broken line of boundary just have not passed through a LMR boundary window. We discuss the method in detail in the following.

(a) (b)

Fig. 4-17: (a) A LMR boundary and (b) its accumulator window with no boundary trace. Therefore, it performs the boundary detection procedure.

We utilize the last positions of far side and near side neighbor LMR boundary windows which have processed the boundary alignment (optimization) to complete searching process of the LMR boundary window. Firstly, we set the searching range for detecting the deviated boundary or to realizing the broken line condition. If the neighbor LMR boundary window which has process boundary alignment is right to it, the searching range is determined from the most extreme right side of this LMR boundary window to the center position of that neighbor LMR boundary window. On the other hand, If the neighbor LMR boundary window which has process boundary alignment is left to it, the searching range is determined from the most extreme left side of this LMR boundary window to the center position of that neighbor LMR boundary window. Secondly, we follow the similar tracking method to boundary alignment (optimization) process. The current window searches the boundary from near side to far side of the searching range. Once the current window detects the last boundary, we find the last position of boundary by the mean of the x-axis position of

every boundary pixels in the current frame. Finally, we align the LMR boundary window to the last position as shown in Fig. 4-19 (b). If the current can not find the boundary in the searching range, it represents that there is no boundary passing through the neighbor area in the series of subsequent Mn frames. It is because the segment of broken line boundary has not passed through yet. Therefore, The LMR boundary will remain at original position and wait for the incoming segment of the broken line shown in Fig. 4-18 (b).

(a)

(b)

Fig. 4-18: (a): The boundary is broken line so it detects no boundary appear in the searching range. (b): The LMR boundary will remain at original position and wait for the incoming segment of the broken line.

(a)

(b)

Fig. 4-19: (a): The boundary have deviated the range of LMR boundary window, so the accumulator window detected no boundary trace. (b): The LMR boundary tracks the last boundary position in the searching range and aligns to it.

III. Alignment checking

The boundary alignment (optimization) and boundary detecting (searching) just consider the boundary energy feature to realize the location and the relative moving direction of the boundary. The boundary energy categorized edge feature, but the feature is not robust enough when a LMR boundary window encounters strong edge intensity appearance which does not belong to road boundary. These edge are called fake edge, especially, the fake edge resulted from obvious shadow in the high light environment, a pond of water on the ground, and the objects around the path. These cause to mistake the boundary energy, so the LMR boundary will align to the wrong position. If a LMR boundary window correct alignment corresponds to position of road boundary, its Region Ratio Features, RoadPower and NonRoadPower, will pass the alignment checking terms (4-5). On the contrary, one of the Region Ratio Features will be conflicted. If the new region ratio features pass the alignment checking, the region ratio will be updated in (4-6). We compare three examples to explain the problem of boundary alignment. In Fig. 4-20,

Fig. 4-21 and

Fig. 4-22, the pink area represents the road region inside of the driving path and the red region area is outside of driving path.The demarcation between the two areas is the driving path boundary. If the boundary energy is only attributed by road boundary, the LMR boundary window will align to the boundary. Then, the RoadPower and NonRoadPower can fit the thresholds, because the region around boundary always is with similar region ratio. Besides, the terms (4-5) take the last region ratio into account so it is very adaptive without any prior knowledge or predefined parameters.

In Fig. 4-20, after alignment the RoadPower and NonRoadPower regress to about 90~100% and 65~80%. On the other hand, when LMR boundary window align to

( 1) ( )

( 1) ( )

t+1: the region ratio of the LMR boundary window after alignment t: the last region ratio of the LMR boundary

Road

wrong position, its region ratio must differ from previous one which fixed position on the road boundary shown as green blocks in Fig. 4-21 and Fig. 4-22. Therefore, the LMR boundary has to be checked by the Region Ratio Features after it alignment to boundary.

Fig. 4-20: (a): L block with the maximum boundary; (b): LMR boundary move toward left side to align to boundary. The correct alignment regress the region ratio, both RoadPower and NonRoadPower, to previous effective range.

(a)

(b)

Fig. 4-21: (a): The boundary alignment to left is in error because the fake edge resulted from the prominent shadow. (b): After the wrong alignment, the incorrect position of LMR boundary window can be checked by the NonRoadPower of Region Ratio Feature shown in green block.

(a)

(b)

Fig. 4-22: (a): The boundary alignment to right is in error because the fake edge resulted from the object outside path. (b): After the wrong alignment, the incorrect position of LMR boundary window can be checked by the RoadPower of Region Ratio Feature shown in green block.

5 Chapter 5

Experimental Results

In this chapter, we will show our results of boundary detection and tracking algorithm in Section 5.1 and Section 5.2. We implemented out algorithm on the platform of PC with P4 2.66GHz and 1GB RAM. The software we used is Borland C++ Builder on Windows XP OS. All of the testing videos are uncompressed AVI standard files. The resolution of video frame is 320×240.

We use CCD camera as the sensor which is mounted in the middle of the test vehicle’s roof. The tested vehicle platform can be seen in Fig. 5-1.

Fig. 5-1: The tested vehicle

5.1 Experimental Results of Driving Path Boundary Detection

In the following , we show our experimental results of driving path boundary detection. It is the initial process in our system in order to find the boundaries and locate the LMR boundary window for tracking later. At first, we present the environment of the road as the left image of the raw in Fig. 5-2, and show the location of LMR boundary windows set by the driving path boundary detection algorithm as

the right image of the raw in Fig. 5-2. In Fig. 5-2, they include many type of road boundary and succeed in detecting the boundary in the night where is with or without street light. Besides, this initial process can work when encountering the curl road due to the modified methods proposed.

(a) (b)

(c) (d)

(e) (f)

(g) (h)

(i) (j)

(k) (l)

Fig. 5-2: Experimental results of driving boundary detection.

5.2 Experimental Results of Boundary Tracking

Here, we demonstrate some boundary tracking examples. In Fig. 5-3 we can see that the system can tracking steadily by align the LMR boundary to the last boundary

position. Moreover, it can work when encountering sharp curve (d)(e)(f) or in the night with or without street light (f)(h). If the vehicle is changing the line, the LMR boundary windows will follow the new path boundary to process the alignment. For example, when the vehicle attempts to change to the right line, the LMR boundary windows on the original right side will transfer to track the left boundary. On the other hand, the new right boundary appears on the right side of current lane, so the LMR

boundary window will detect it immediately and continue tracking show in Fig. 5-4

(a) (b)

(c) (d)

(e) (f)

(g) (h) Fig. 5-3: Experimental results of boundary tracking

(a) (b)

(b) (d) Fig. 5-4: Experimental results of change line situation.

6 Chapter 6

Conclusion

We present a new method of efficient road following to achieve real-time performance. At first, we locate the main driving path precisely, and then perform the boundary tracking algorithm. In the algorithm, we utilize the color distribution and edge difference features in the proposed LMR boundary window. By on-lined L*a*b color model, we can extract the road color region and observe the color distribution of the LMR boundary window. This approach is accomplished by using the proposed region ratio feature. We also present a boundary energy feature to find the most distinct edge difference which most matches the boundary property. Using these both main features, we can track the boundary efficiently and effectively. Therefore, the system integrates the concepts of region-based and boundary-based to realize the whole processing.

Experiments were conducted on different scenes which including the different type of road boundaries. The system succeeds in detecting and tracking the regular type of boundary with obvious lane line or complicated types of boundary including broken lane lines, absence of lane line and various types of country roads. Our system can also accommodate the lane change movement and relocate the new main driving path boundary. Through the experiment, we realize the overall excursion time is very swift and can handle driving in the night. In the future, we believe that when the system ports to the platform; it can achieve real-time performance as well.

6.1 Contribution

In this paper, we proposed a new method which combines color feature and edge feature. Therefore, we enhances the preciseness of the system. In addition, our algorithm has modified most disadvantages of the region-based and boundary-based methods. Firstly, we proposed the guide snake to sample the nearest region of each LMR boundary window. Therefore, it can solve the problem when encountering non-homogenous surface. Besides, the L*a*b color feature can resist the light variation and possess great luminance difference toleration. After applying the updating module, we can train the on-lined color model to increase its plasticity. As a result, the adaptive model can preserve accurate detection within unstable feature condition. On the other hand, we decrease the region of interest to LMR boundary windows so we can enhance the speed of computation time. Specifically for these disadvantages, we summarize the modification we mentioned above shown in Fig. 6-1.

Secondly, we solve the main problems of boundary-based methods. The majority of the problems are caused by unconstructed road without lane line, strong edge of non-boundary objects, and the sharp curvature on the road. In our system, we focus solely on the LMR boundary windows and adopt the general boundary feature so we can prevent these problems occurred. Because the LMR boundary windows locate near by road boundary, it would not interfere with other stronger edge of the non-boundary objects. Furthermore, our tracking algorithm can effectively handle many complicated scenes such as broken lane line, and unclear boundary. Some boundary-based methods need many parameters and horizon position when converting to bird view image by inverse perspective mapping (IPM) or processing model template matching. It is very impractical due to it requires many prior knowledge. In fact, the pre-defined parameters and knowledge are not necessary in

our system. Specifically for the above disadvantages of boundary-based methods, we summarize the modification we mentioned above shown in Fig. 6-2.

Fig. 6-1: The main features and disadvantages of region-based methods. Our algorithm can modify and solve the problems.

Fig. 6-2: The main features and disadvantages of boundary-based methods. Our algorithm can modify and solve the problems.

6.2 Future Works

To further improve the performance and the robustness of our algorithm, some enhancements or trials can be made in the future. Firstly, if the edge boundary is very ambiguous, the edge information will be inconsistent. Therefore, we can aim at the LMR boundary windows to enhance the edge feature by stretching gradient difference.

Secondly, we can express the boundary location more visually by using the least square error technique. The least square error method processes the boundary candidate pixels and finds the optimal boundary contour of linear or parabolic model which is with minimum errors. Thirdly, we can exploit the boundary detection results to realize the lateral departure warning (LDW) function and integrate into our system.

Finally, with the effectiveness of locating boundary, the driver can clearly understand the vehicle position. Therefore, we will consolidate the proposed road features and the ability of driving path boundary detection to continue to research the obstacles and vehicles detection. According the boundary position, the system can comprehend the relative position, frontal or lateral, between the vehicle and these obstacles, Moreover, the distance estimation from the vehicle to the frontal or lateral objects is another important research topic. After accomplishing the overall functions mentioned above, the system will very applicable and we will transplant from PC based to DSP platform for more commercial certification and appealing.

References

[1] J. Crisman and C. Thorpe, SCARF:A Color Vision System that Tracks Roads and Intersections, Conf. Proc. IEEE International Conf. on Robotics and Automation, IEEE, 1993

[2] J. Crisman and C. Thorpe, UNSCARF: A Color Vision System for the Detection of Unstructured Roads, Conf. Proc. IEEE International Conf. on Robotics and Automation, 1991.

[3] J.Huang and B Kong, A New Method of Unstructured Road Detection Based on HSV Color Space and Road Features, International Conf. on Information Acquisition, 2007.

[4] Y.Alon, A.Ferencz and A.Shashua, Off-road path following using region classification and geometric projection constrains, Computer Vision and Pattern Recognition, IEEE Computer Soc. Conf, 2006.

[5] C.Rasmussen, Grouping dominant orientations for ill-structured road following, Proc. IEEE Soc. Conf. Computer Vision and Pattern Recognition, 2004.

[6] Y.He, H.Wang, and B. Zhang, Color-Road Detection in urban traffic Scenes, IEEE Trans. Intelligent Transportation System, 2004.

[6] Y.He, H.Wang, and B. Zhang, Color-Road Detection in urban traffic Scenes, IEEE Trans. Intelligent Transportation System, 2004.