CHAPTER 3. FISHEYE LENS INVERSE PERSPECTIVE MAPPING
3.3. F ISHEYE L ENS I NVERSE P ERSPECTIVE M APPING
3.3.3. The Complete Fisheye Lens Inverse Perspective Mapping
According to the combination of the results in Section 3.2.3 and Section 3.3.2, we will propose a complete fisheye lens inverse perspective mapping algorithm. This algorithm can be divided into two sections: (1) forward mapping algorithm, (2) backward mapping algorithm. The consideration and detailed algorithm would be shown below.
The objective of forward mapping algorithm is to search the remapped image dimensions or range. The illustration is shown in Fig. 3-12.
(a)
θ1
β β 1 -m u 2
(b)
Fig. 3-12: The original and adjusted scope
The influence factor of dimensions of scope is solely concerned with filed of view of camera whether the lens is normal lens or fisheye lens with fixed tile and pan angle.
In order to reduce the confusion of computation in computing the tangent and secant triangular functions, we limit the scope of camera by condensing the filed of view.
Though this action will lead to lose far and fringe information, we still retain the most scope as possible. Furthermore, we condense the angle of view by Snell’s Law like equation (3-13), where IR simulates the index of refraction and control the scope of
result range. The range of IR is between 1.3~1.7 to simulate the index of refraction of
The angle θ1andθ2 are substituted for the equation (3-7) to compute the extreme value of X,Y coordinate value, hence we obtain the dimension of the remapped image.
The flow of backward mapping algorithm is different from forward one; it is because the backward mapping algorithm needs a radial distortion correction step.
Before explaining the backward mapping algorithm, we consider to explain the idea of backward mapping algorithm by the Fig. 3-13 first.
Fig. 3-13: The (a) real scene image (b) distorted image (c) desired image
We observe that the image captured by the fisheye camera in Fig. 3-13(a) which has perspective effect and distortion simultaneously. After the inverse perspective mapping, we can expect to obtain the Fig. 3-13(b) whose perspective effect has been removed. On the image Fig. 3-13(c), the perspective effect and distortion are completely removed. By following the sequences of Fig. 3-13(a),(b),(c), we can derived a backward mapping algorithm.
First of all, we modify the equation (3-8) to equation (3-14).
Then, we utilize the angle θ1andθ2 directly and equation (3-12) to perform the distortion correction. By tuning the parameter of IR and , we will finally obtain the undistorted and perspective effect removed image.
k1
Chapter 4.
Obstacle Detection Algorithm
Broggi et al. [5] utilized his inverse perspective mapping method and stereo cameras to detect obstacles in front of vehicle. The left and right images captured by the camera at corresponding positions were differenced to obtain a difference image. In that image, the planar objects such as lane markings were eliminated and prominent objects were reserved as quasi-triangle pairs. By detecting these quasi-triangle pairs, the position of obstacles could be found. Nevertheless, the performance of Broggi’s detection method was obviously dependent on the obstacle height, width, distance and shape.
In this chapter, we develop an obstacle detection algorithm using spatial and temporal fisheye lens inverse perspective mapping method. We use the fisheye lens inverse perspective mapping method mentioned in Section 3.3 and single fisheye camera to detect obstacles by the side of vehicle. The definition of obstacle in this thesis excludes objects whose height shorter than a threshold and objects whose edges are not quasi-vertical. The image preprocessing process that selects a proper pattern image as the input to the feature searching procedure will be described in Section 4.1. As mentioned in Section 3.2.3, a vertical straight line in the image which represented the vertical edge of obstacle in the world coordinate system is projected to a straight line whose prolongation will pass the camera vertical projection point on the world surface.
The algorithm of searching features whose definition is described above will be presented in Section 4.2. Histogram relative procedure and object tracking and information extraction are described in Section 4.3 and 4.4 respectively.
4.1. Image pre-processing
Fig. 4-1: The flow chart of image pre-processing
Since we have obtained the perspective effect removed and distortion corrected image from the fisheye lens inverse perspective mapping step, we will execute some preprocessing to extract the simplified pattern image for further process. At first, the remapped image will be smoothed by mean filter in order to reduce the noise caused from the process of image capturing and fisheye lens inverse perspective mapping transformation; hence we will obtain a blurred remapped image. After above step, we will judge the variation situation in the image by differencing the previous frame and current frame. If the objects in the world and our camera are motionless relatively, the difference image will be empty. Then we will utilize the profile image to extract the feature segments; otherwise we will use the obstacle-sensitive temporal fisheye lens inverse perspective mapping difference image to extract the features. We will discuss the flows of obtaining the profile image and temporal fisheye lens inverse perspective mapping difference image in Section 4.1.1 and Section 4.1.2 respectively.
4.1.1. Profile image
In order to extract the profile of image, we need the obvious edges of obstacles.
Nevertheless, the image has been blurred by mean filter, some edges become
unapparent, thus we should emphasize the edges once again by the unsharp mask. The diagram of unsharp mask is shown in Fig. 4-2.
Fig. 4-2: The unsharp mask illustration
In Fig. 4-2, we can observe that a sharp edge E is smoothed to form the edge A, and then we will perform mean filter to smooth the blurred edge again to form the edge B. The difference of edge A and B will be done and added to the original blurred edge A, thus an obvious edge A+(A-B) will appear in the image. The result of sharpening process is shown in Fig. 4-3.
(a) (b)
Fig. 4-3: The results of sharpening process. (a) the blurred image (b) the sharpened image
An edge detection process will be operated after the sharpening stage, Gx and Gy stand for the horizontal and vertical direction 3 by 3 Sobel edge mask respectively. It
will be performed on the edge-sharpened image. Equation (4-1) show the spatial illustration and mathematic operation of Sobel edge detection,
1)]
where the x, y is the coordinate of each pixel in the horizontal and vertical axis, and f(x,y) is the intensity of this pixel.
After detecting the edges of the remapped image, a threshold will be performed on the image in order to obtain a binary image. However the edges may not be continuous, therefore we should execute the morphological dilation and erosion process to connect these edges and discard the edges which maybe caused by noise. Equation (4-2) shows the spatial relationship and mathematic operation of morphological dilation and erosion,
X0 X1 X2
where the D and E are the notation of dilation and erosion value of a 3 by 3 region in a binary image. The results of edge detection and morphology operation are shown in Fig.
4-4.
Fig. 4-4: The results of edge detection and morphology operation
To extract the feature segments, we should execute the thinning algorithm [26] to thin these binary edges. However, the computational load of thinning algorithm is so huge; we perform this algorithm in another form alternatively. The modified thinning algorithm (single direction) is listed as follows.
N(x4)= x2+x3+x4+x5+x6+x7+x8+x9
S(x4) is the number of value changing of the cycle from x2,x3,x4 to x8,x9,x2 If
x9 x2 x3
x8 x1 x4
x7 x6 x5
2<=N(x1)<=6 AND S(x1)=1 AND x2∩x4∩x6=0 AND x4∩x6∩x8=0
THEN x1=1
N(x1)<2 means that x1 is an end point or N(x1)>6 means that x1 is an inner point, thus we should not delete x1. S(x1)>1 stands for that x1 is a bridge point, therefore we should not erase x1 as well. The original algorithm will delete the x1 (x1=0) to thin the pattern while above conditions are satisfied, and execute this algorithm until the pattern was unit size. Contrarily, we set x1=1 to extract the exterior profile of a pattern and perform the algorithm once. Fig. 4-5 shows the result of profile searching.
(a) (b)
Fig. 4-5: The results of profile searching. (a) the remapped image (b) the profile image
After the profile searching stage, we obtain the profile image for further use.
Although the results of segment searching on the profile image is not so good as the results of segment searching on the skeleton-like image, we can add some processes described later to increase the effect of segment searching.
4.1.2. Temporal fisheye lens inverse perspective mapping difference image
The objective of temporal fisheye lens inverse perspective mapping difference process is that we would like to simulate the function of stereo vision. The advantages of stereo inverse perspective mapping are that the planar object patterns in the image could be removed by differencing the left and right remapped image and the non-planar object patterns would be reserved.
Fig. 4-6: The idea illustrations of temporal fisheye lens inverse perspective mapping difference image. Upper: planar object patterns. Lower: non-planar object patterns
According to stereo inverse perspective mapping method [5], two cameras were set apart from a distance to acquire enough overlapped information. If we would like to simulate the stereo effect by using a single camera, we need to capture the image apart from a time interval. From Fig. 4-6, there are important issues need to solve, such as the choice of time interval and the shift displacement of the remapped image. The time interval must be moderate, too long interval will produce too large region and make detection error; conversely, too short interval will produce unapparent region and bring about difficulty of obstacle detection. Similarly, the shift displacement of previous remapped image controls the coincidence of planar object patterns.
Since the moderate time interval are decided by experiential adjusting, we can compute some feature points displacement in this two remapped images and take the average displacement into the shift displacement of temporal fisheye lens inverse perspective mapping method. After thresholding process, we will obtain a stereo effect-like binary difference image for further use.
(a) (b)
Fig. 4-7: The results of temporal fisheye lens inverse perspective mapping process.
(a) the remapped image (b) the temporal difference image
4.2. Feature Searching algorithm
As mentioned at the beginning of this chapter, the features we would like to search are merely the segments in the remapped image which represent the quasi-vertical edge of objects in the world coordinate system. Due to these feature segments prolongation will pass through the camera vertical projection point, we will develop a searching algorithm and scan the polar histogram.
Starting from the camera vertical projection point defined as CP on the remapped image, we scan angle by angle and scan radius outward for each angle on the pattern image such as profile and difference image from last step. In order to reduce the loss of segment points, we use a mask and set a voting threshold to search the points satisfied
with the pixel value and between distance conditions. For each angle, a segment will be output as long as its length is long enough; next segment starting point at the same angle must be close enough to the last segment in order that we merely concerned about the segment closet to CP. After searching and highlighting the segment points at each angle, we can obtain the number of segment points at the same time. These point number at each angle will be used to produce a polar histogram for further process. Some results of feature searching procedure are show in Fig. 4-8 and Fig. 4-9.The complete flow chart of feature searching is shown in Fig. 4-10.
(a) (b)
(c) (d)
Fig. 4-8: The result of feature searching procedure using profile image. (a)The sharpened remapped image, (b)The profile image, (c)The scanned feature
segments, (d)The scanned feature segments superposed on the sharpened remapped image
(a) (b)
(c) (d)
Fig. 4-9: The result of feature searching procedure using temporal difference fisheye lens inverse perspective mapping image.(a)The remapped image, (b)The temporal difference fisheye lens inverse perspective mapping image, (c)The scanned feature segments, (d)The scanned feature segments superposed on the remapped image
Fig. 4-10: The flow chart of feature searching
4.3. Histogram Post-processing
Fig. 4-11: The flow chart of histogram post-processing
As we have obtained the polar histogram of feature segments, we still need some procedure to eliminate the histogram we are not concerned, such as the planar object segment and noise segment. Due to the imperfectness of planar object segments reduction in the pattern image obtaining procedure, planar object segments will still appear and be computed at the polar histogram step. In this thesis, we barely define the lane marking as the planar object. By observing the polar histogram, we can discover that the trapezoid shape of these histogram columns means the planar objects and then delete them. To avoid the noise histogram columns, we reject the too short columns.
After eliminating planar objects and noise, we will search the local maximum position which represents the non-planar object segment position in the polar histogram.
In order to reduce the confusion of too many obstacles appearing in a frame simultaneously, we pick a peak column merely at an angular interval. Now, we obtain the positions of obstacle candidates and then entry the tracking process to confirm them.
Some results of histogram post-processing procedure are show in Fig. 4-12 and Fig.
4-13.
Fig. 4-12: The trapezoid shape histogram caused by lane markings
(a) (b)
(c)
Fig. 4-13: The process of histogram post-processing (a) The polar histogram of Fig.4-9, (b) The histogram of (a) after the trapezoid shape histogram elimination, low singleton histogram elimination and peak searching procedure, (c) Local peak histogram
4.4. Object Tracking, Confirmation and Information
Fig. 4-14: The flow chart of object tracking, confirmation and information extraction
The objective of tracking procedure is to confirm and improve the correctness of object we have detected. We choose the displacement and angle variation of feature segment on the image to be the judgment conditions, then we can track the feature segment continuously. If the feature segment passes the tracking procedure, we will judge whether this feature segment belongs to the planar object or not once more by simple pattern matching. The pattern represents the planar object is shown in Fig. 4-14.
After the tracking and confirmation procedure, we can sure the feature segment we have detected is an obstacle segment and then point out the position of feature segment. Thus we will extract and store the relative information for system further use.
Chapter 5. Experimental Results
5.1. Results and Comparison of Normal Lens Inverse Perspective Mapping Method
In section 3.2.3, we proposed a modified forward and backward normal lens inverse perspective mapping equation pairs. In order to discriminate the results, we will show the experimental results of Broggi’s equations and our equations in Fig. 5-1 with normal lens camera. The experimental environment is indoor space and the image contents are square tiles and some objects.
(a)
(b)
(c)
Fig. 5-1: The results of normal lens inverse perspective mapping equations (a) original captured image (b) bird’s view image using Broggi’s equations (c) bird’s view image using our equations
From Fig. 5-1, we note that the image captured by normal lens camera is accurately transformed to the bird’s view image by using our equations. The perspective effect in the original image leads the prolongation of vertical straight lines in the world to intersect a point called vanish point. In Fig. 5-1(b) and (c), we can observe the perspective effect is eliminated by both of Broggi’s and our equations. Nevertheless, a horizontal straight line in the world will be transformed to an arc by implementing Broggi’s equations as shown in Fig. 5-1(b). With our modified equations, a horizontal straight line in the world will still be transformed to a horizontal straight line as shown in Fig. 5-1(c).
5.2. The Experimental Configuration
5.2.1. The Experimental Setup
Without losing the generality and experimental convenience to detect the obstacles at the side of vehicle, we mounted a fisheye lens camera at the central position of two
side doors with an appropriate height. The illustration is shown in Fig. 5-2.
Fig. 5-2: The setup diagram of camera.
In order to avoid the disorder of a frame, we will merely detect the objects whose heights are more than a threshold and whose edges are quasi-vertical. The objects such as sidewalk, small balls and so on are excluded in our detection system. The experimental environment is constrained to the brighter environment and the speed of vehicle should not too fast so that the objects in frames would not move too quickly.
5.2.2. The Program Interface and Platform Information
(D) (E)
(C) (B)
(A)
Fig. 5-3: The program interface in the PC platform
Fig. 5-3 shows the program interface in the PC platform. Block (A) shows the input frame which is added the rectangle and line to indicate the position of obstacles. Block (B) contains the video relative information. Block (C) contains the extrinsic and intrinsic parameter of camera and fisheye lens inverse perspective mapping look up table production. Block (D) shows the image we used in the obstacle detection algorithm. The upper image is the remapped image which is added the rectangle and line to indicate the position of feature segments. The central image is the judgment image to judge the static situation as described in Section 4.1, and the lower image shows the input image to feature segment searching stage. Block (E) shows the histogram we have described in Section 4.3. The detailed platform information is listed in Table 3.
Table 3: The specification of platform information
CPU Intel Core Duo T2050 1.6GHz
Memory 1GB DDR2 RAM
Programming Tool Borland C++ Builder 6.0
Operation System Microsoft Windows XP
Video Resolution 640x480
Frame Rate 30fps
5.3. Results of Distinct Environment
5.3.1. Results of Fisheye Lens Inverse Perspective Mapping and Obstacle Detection
As shown in Fig. 5-4, we detected larger obstacles whose edges are quasi-vertical at the lateral side of our vehicle by using fisheye lens inverse perspective mapping method described in Section 3.3 and obstacle detection algorithm in Chapter 4. For
instance, bicyclist, street light, railings, vehicles, pedestrian and so on are detected accurately by our system.
(a) Scenery1: bicyclist, street light.
(b) Scenery 2: railings.
(c) Scenery 3: multiple vehicles in parking area.
(d) Scenery 4: pedestrians, close vehicle.
(e) Scenery 5: multiple vehicles.
Fig. 5-4: The results of fisheye lens inverse perspective mapping and obstacle detection in different scenery
5.3.2. Results of Obstacle Tracking
Since we have detected the obstacle in a frame, we should track this obstacle in successive frames continuously by checking the displacement and angular shift in the image as demonstrated in Section 4.4. The successive tracking results are shown in Fig.
5-5 and Fig. 5-6.
# Frame 314 # Frame 318
# Frame 322 # Frame 326
Fig. 5-5: Results of obstacle tracking in Scenery 1
# Frame 172 # Frame 188
# Frame 204 # Frame 220 Fig. 5-6: Results of obstacle tracking in Scenery 3
5.3.3. Results of Obstacle Warning
Thanks to the fisheye lens inverse perspective mapping method, the three-dimension world coordinate value could be estimated from the remapped image.
In other words, when we detected the obstacle in the remapped image, we could also estimate the position information. If the obstacle is so overly approaching our vehicle exceeding a threshold, that system will alert us by arresting markings.
(a)
(b)
Fig. 5-7: Results of obstacle warning
5.4. Discussion
Though the performance of our obstacle detection system based on fisheye lens inverse perspective mapping method is satisfying, it is still subject to some disturbance factors as shown below. In Fig. 5-8 (a), the street light in the remapped image becomes unapparent due to the texture of remapped grassland behind it.
(a)
(b)
Fig. 5-8: Some example of detecting error in our system
In addition, the completeness of obstacle shape in the profile or temporal fisheye lens inverse perspective mapping difference image is critical for following obstacle detection process. Fig. 5-8 (b) show the broken shape of obstacle in the temporal
In addition, the completeness of obstacle shape in the profile or temporal fisheye lens inverse perspective mapping difference image is critical for following obstacle detection process. Fig. 5-8 (b) show the broken shape of obstacle in the temporal