Flowchart of the proposed vehicle surrounding monitoring system is depicted in Fig. 2. We first calibrate the fisheye cameras to obtain their intrinsic parameters.
Then these parameters can be used for correcting distortion and transforming the fisheye images into perspective ones. There perspective images are registered to a ground plane by using planar homographies. The objects coplanar with the ground plane can be registered perfectly in this way. For 3-D objects, some ad-vanced registration process is necessary. We select a seam between each pair of adjacent images according to the residual error obtained in previous calibration
(a) Sample view of bird’s-eye view vision system 208 Y.-C. Liu, K.-Y. Lin, and Y.-S. Chen
Fig. 1. The proposed vehicle surrounding monitoring system using multiple cameras.
(a) Images I1, I2, ..., and I6 captured from the corresponding cameras C1, C2, ..., and C6, respectively. (b) The bird’s-eye view image synthesized from the six fisheye images.
approach is barely feasible because it is very dangerous to hang a camera high above a vehicle. The Matsushita company proposed an image synthesis display system [1] which uses several cameras around the vehicle and synthesizes the images captured from these cameras into a whole picture. Because they simply average the image pixel values in the overlapped regions, the ghost artifact is severe and the driver cannot know the actual location of the objects. Ehlgen and Pajdla mounted four omnidirectional cameras on a truck and find several subdivision ways for dividing the overlap regions to construct the bird’s-eye view image [2]. No matter how to choose the subdivision way, there are some blind spots in the subdivision boundaries.
In this paper we propose a novel method of synthesizing a seamless bird’s-eye view image of vehicle surroundings from six fisheye cameras mounted around the vehicle. As shown in Fig. 1(a), these six fisheye cameras cover the whole surrounding area and have some overlapped areas between adjacent cameras.
Our goal is to stitch the six distorted images captured from the fisheye cameras and provide an image of vehicle surroundings from a bird’s eye viewpoint, as shown in Fig. 1(b).
2 Vehicle Surrounding Monitoring System
Flowchart of the proposed vehicle surrounding monitoring system is depicted in Fig. 2. We first calibrate the fisheye cameras to obtain their intrinsic parameters.
Then these parameters can be used for correcting distortion and transforming the fisheye images into perspective ones. There perspective images are registered to a ground plane by using planar homographies. The objects coplanar with the ground plane can be registered perfectly in this way. For 3-D objects, some ad-vanced registration process is necessary. We select a seam between each pair of adjacent images according to the residual error obtained in previous calibration (b) Virtual perspective view and cameras setup of this system
Figure 1.10: Sample views of bird’s-eye view vision system. (a) The camera setup and the virtual perspective bird’s-eye view image. (b) The sample view of bird’s-eye view vision system. Bird’s-Eye View Vision System for Vehicle Surrounding Monitoring 217
(a)
(b)
Fig. 6. Image sequences of a car moving in reverse (a) and moving back into a parking space (b)
as shown in Fig. 4(b). As long as the cameras remain fixed, the perspective transformation from each rectified images to the bird’s eye view and the optimal seam between each pair of adjacent images are invariant. Hence, the images of ground-level objects around the vehicle could be stitching into a bird’s eye view, as shown in Fig. 4(c). But when a non-ground-level object exists in the surrounding scene, such as the car on the left side of the image shown in Fig. 4(d), and the image of this object cross the seam in the composite, a misalignment will occur on the seam. This misalignment is then removed by the DIW registration along the seam and the final composite image after the exposure compensation and weighted blending are shown in Figs. 4(e) and 4(f), respectively.
Some problems may occur in practice when all surrounding images are stitched in perspective presentation, such as the distortions of non-ground-level objects, low image resolution and amplified vibration in the farther surrounding area (see Fig. 5(a)). Warping the composite in a distortion level by placing the virtual fish-eye camera at certain height can cope with this problem, as shown in (Fig 5(b).
In Figs. 6(a) and 6(b), we show two sampled image sequences in which the vehicle is moving in reverse and moving back into a parking space, respectively.
4 Conclusion
In this paper, we have presented a driving assistant system which can provide the bird’s eye view image of vehicle surrounding. The multiple fisheye cameras are mounted around the vehicle to capture images of the surroundings in all
(a) Sample views of bird’s-eye view vision system
Bird’s-Eye View Vision System for Vehicle Surrounding Monitoring 217
(a)
(b)
Fig. 6. Image sequences of a car moving in reverse (a) and moving back into a parking space (b)
as shown in Fig. 4(b). As long as the cameras remain fixed, the perspective transformation from each rectified images to the bird’s eye view and the optimal seam between each pair of adjacent images are invariant. Hence, the images of ground-level objects around the vehicle could be stitching into a bird’s eye view, as shown in Fig. 4(c). But when a non-ground-level object exists in the surrounding scene, such as the car on the left side of the image shown in Fig. 4(d), and the image of this object cross the seam in the composite, a misalignment will occur on the seam. This misalignment is then removed by the DIW registration along the seam and the final composite image after the exposure compensation and weighted blending are shown in Figs. 4(e) and 4(f), respectively.
Some problems may occur in practice when all surrounding images are stitched in perspective presentation, such as the distortions of non-ground-level objects, low image resolution and amplified vibration in the farther surrounding area (see Fig. 5(a)). Warping the composite in a distortion level by placing the virtual fish-eye camera at certain height can cope with this problem, as shown in (Fig 5(b).
In Figs. 6(a) and 6(b), we show two sampled image sequences in which the vehicle is moving in reverse and moving back into a parking space, respectively.
4 Conclusion
In this paper, we have presented a driving assistant system which can provide the bird’s eye view image of vehicle surrounding. The multiple fisheye cameras are mounted around the vehicle to capture images of the surroundings in all
(b) Sample views of reversing parking
Figure 1.11: Sample views of bird’s-eye view vision system in parking lot with a pedes-trian]. (a) The sample view of bird’s-eye view vision system. (b) The sample view of bird’s-eye view vision system while reversing parking.
1.3 Overview of the proposed method 13
1.3 Overview of the proposed method
We propose a low cost but efficiency embedded vehicle surveillance system that only need four fisheye cameras, a quad splitter and a low cost DSP embedded system to generate bird’s-eye view image in real time. Also we propose a dynamic boundary idea to switch image source according the image contents, and ensure driver can see figures of obstacles in bird’s-eye view image.
Our work can be divided in two parts. First task is to build a lookup table, which de-fined the mapping relationship from bird’s-eye view image to four fisheye camera images (Figure 1.12). The procedures of generating such a table include fisheye image distortion model, fisheye image rectification, 2D image projection (homography), and image format conversion between PC and embedded system. The second task is the implementation of real time image processing in embedded system. The lookup table is used to speed up the image processing procedures in embedded system, and we also apply the pipeline mecha-nism to reduce the data transition time between cache and SRAM. Moreover, we implement the dynamic boundary idea to this system to ensure the driver can see obstacles near the vehicle. (Figure 1.12) is the flow chart of lookup table generation, and (Figure 1.13) is the flow chart of real time image processing in embedded system.
1.4 Thesis Organization
In chapter 2, we introduce the first part of the proposed method for finding the map-ping relationship between fisheye images and the bird’s-eye view image. We also generate a lookup table for embedded system to reduce the computation of image processing. In chapter 3, we introduce the embedded system and the real time image processing applica-tion. The implementation details are revealed in this chapter, too. Chapter 4 is the results of our work. Finally, the conclusions are in last chapter.
14 Introduction
Figure 1.12: Flow chart of generating the lookup table for embedded system. This flow chart illustrate the procedures for generating the lookup table. The procedures includes image format conversion, fisheye distortion principle, fisheye image rectification, image warping, and 2D image projection.
1.4 Thesis Organization 15
ADV7183 video decoder
ADV7179 video encoder
Video signal in Video signal Out
DMA move frame data to L2 SRAM
DSP core fills sub‐frames according to table which built by PC
After all sub‐frames generated, DMA move output frame data to encoder
Figure 1.13: BlackFin-561 video processing flow chart. This flow chart illustrate the video processing sequence in blackfin-561. When receive the input frame from quad split-ter, the decoder decode video signals into itu-656 format frame data. Then, the DMA move the frame data into the SRAM through the PPIs(parallel peripheral interfaces). The DSP core generate the bird’s-eye view image and saved in SRAM. After the whole bird’s-eye view image has been generated, another DMA will move the result to the encoder. Then, the bird’s-eye view image frame immediately show on screen.
16 Introduction