1.1. Motivation
Digital video sequences acquired by image captured devices are usually affected by undesired motions which can be classified into several aspects. The image captured by compact and lightweight video cameras, i.e. hand held devices, are usually affected by unstable camera holding or moving platform. The image captured by in-car video cameras are usually affected by a bumpy ride or by the steering of the driver. The image captured by video surveillance systems are usually affected by wind blowing, bird jumping or earthquake. The unwanted positional fluctuations of the video sequence will affect the visual quality and impede the subsequent processes for various applications such as motion coding, video compression, feature tracking, etc. The challenges of a digital image stabilization system (DIS) are: how to compensate for the unwanted shaking of the devices without being affected by large moving objects, ill conditions in the image, and the panning motion of the camera. In this dissertation, the related techniques of DIS will be proposed to tackle these challenges.
1.2. Related Works Review
The image stabilization systems can be classified into three major types: (1) the electronic image stabilizers (EIS) [1]; (2) the optical image stabilizers (OIS) [2]; (3) the digital image stabilizers (DIS) [5]. The EIS stabilizes the image sequence by employing motion sensors to detect the camera movement for compensation. The OIS employs a prism assembly that moves opposite the shaking of the camera for stabilization. Because both EIS and OIS are hardware dependant, the applications are restricted to device built-in on-line process. The (DIS) is the process of removing the undesired motion effects to generate a compensated image sequence by using digital image processing techniques without any mechanical devices such as gyro sensors or fluid prism [4]. The major advantages of DIS are:
(1) no restriction of on/off-line applications, (2) suitable for miniature hardware
implementation (since the mechanical device is not required for compensation) [5]. The DIS can be performed either as post-processing after the video sequence was acquired, or in real-time during the acquisition process, depending on the applications. Archive films with undesired shaking effects require post-processing for the video sequences, while camcorders require a real-time compensation process.
The DIS system is generally composed of two processing units. One is the motion estimation unit and the other is the motion compensation unit. The purpose of motion estimation unit is to estimate the reliable camera global movement through three processing steps on the acquired image sequence. (1) evaluation of local motion vectors (LMVs) is the first step of the process; (2) detection of unreliable motion vector components is the next step.
(3) determination of the global motion vector (GMV). Following the motion estimation, the motion compensation unit generates the compensating motion vector and shifts the current picking window according to the compensating motion vector to obtain a smoother image sequence. Fig. 1.1 shows the motion compensation schematics. The window of frame( 1)t− is the previous compensated image. The compensating motion vector v is generated by the DIS according to the global motion vector between two consecutive images. The window of frame( )t is the picking window according to the compensating motion vector v to minimize the shaking effect.
Various algorithms had been developed to estimate the local motion vectors in DIS applications such as representative point matching (RPM) [3][5][6], edge pattern matching (EPM) [7][8], bit-plane matching (BPM) [4][9] and others [10][11][15][16][17]. It had also been demonstrated that the DIS could reduce the bit rate for video communication [18]. The major objective of these algorithms is to reduce the computational complexity, in comparison to the large area full-search block-matching method, without losing too much accuracy. In general, the RPM can greatly reduce the complexity of the computation in comparison with the other methods. However, it is sensitive to irregular conditions such as moving objects and intentional panning, etc. [9]. Therefore, the reliability evaluation is necessary to screen the undesired motion vectors for the RPM method. In [6], a fuzzy-logic-based approach was proposed to discriminate the reliable motion vector from the local motion vectors. This method produced two discriminating signals based on some image information such as contrast, moving object, and scene changing to determine the global motion vector. However, these two signals cannot widely cover various irregular conditions such as the lack of features or containing large moving objects in the images, and it is also hard to determine an optimum
threshold for discrimination with these various conditions. Some researchers estimate local motion vector using feature based techniques, which track a small number of image features (points, lines, and contours or certain object, etc.) to evaluate the motion vector. This makes it efficient and available for real-time implementation. But the difficulty is that, especially for outdoor applications, it can not stably and accurately find available features in the image [19].
Based on the optical flow technique, a fundamental approach in computer vision, many methods have been proposed in literature to solve different types of problems. The estimation of optical flow is based on the assumption that the intensity of the object (or specified pixel) in the image sequence is constant. The difficulty is that most consumer video camcorders have an auto-shutter function that adjusts itself to average intensity dynamically, so that maintaining constant intensity of the object becomes impossible in real applications. In this paper, a reliable local motion vector extraction method is proposed to determine the global motion vectors for practical applications.
In the motion compensation of DIS, accumulated motion vector estimation [7] and frame position smoothing (FPS) [20][21][22] are the two most popular approaches. The accumulated motion vector estimation needs to compromise stabilization and intentional panning (constant motion) preservation since the panning condition causes a steady-state lag in the motion trajectory [20]. The FPS accomplished the smooth reconstruction of an actual long-term camera motion by filtering out jitter components based on the concept of designing the filter with appropriated cut-off frequency. The disadvantage of FPS is that it does not guarantee the availability of the determined compensating motion vector when the specified-bound is restricted to preserve the effective image area in the DIS applications.
Compensating Motion Vector
Window Shifting Allowance
Window of Frame(t-1) Window of Frame(t) Image Captured Area
Fig. 1.1. Motion compensation schematics.
1.3. Overview of This Dissertation
Local Motion Vectors Estimation
Reliability Evaluation of Motion
Vectors Refined Motion
Vector
Global Motion Vector Estimation
Motion Compensation
Original Images Stabilized Images
Fig. 1.2. Schematic overview of the digital image stabilization developing techniques Fig. 1.2 describes a schematic overview of the digital image stabilization developing techniques. It illustrates the three main issues of DIS developments. These issues are local motion estimation, global motion estimation and motion compensation. The issues presented in this dissertation ,such as motion estimation, which include local motion vectors and global motion vector, and motion compensation will be addressed in separate chapters, each with its own introduction, literature overview and algorithm development. This will make it possible to read each chapter without having to cross-reference to the other chapters.
In chapter 2, we start with the issues of local motion vector estimation and global motion vector estimation. Due to these two issues being mutually related in their information process and development technique, it is hard to separate them. Based on the difference in dynamic and real video characteristics, we propose several techniques to tackle irregular images that contain large moving objects, low-contrast area and lack of features to improve robustness and accuracy. The advantages of our proposed methods in the different functions are:
Use representative point matching to dramatically reduce the computation complexity. This makes it possible to build in the regular processors for realization.
Propose an inverse triangle method to discriminate the reliability of motion estimation. Based on the inverse triangle method, the related development techniques are:
Discriminate the reliability of each local motion vector with respect to each axis.
Form a refined motion vector.
Determine the background-based evaluation area by coarsely detecting the skyline.
Determine the optimum representative points for stationary video surveillance system. This will improve the computation efficiency.
Use background based evaluation to determine the global motion vector.
In chapter 3, we address the issue of motion compensation. The objective of motion compensation is to reduce the jiggled image phenomenon by generating a compensating motion vector to stabilize the image sequence. The major work in this part is to improve the existing compensation method within the specified requirements such as limited window shifting allowance, panning condition etc. Therefore, an inner feedback-loop integrator and fuzzy inference have been applied to approach the problem. Quantitative evaluation terms, such as motion trajectory and smoothness index, have been developed for final results comparison as well.
In chapter 4, the experimental results of the algorithms developed in chapter 2 & 3 will be interpreted. First, the accuracy of global motion vector estimation will be compared by the root mean square error (RMSE) method. In this part of the experiments, video sequences captured by different applications of hand-held camcorders and in-car video capture devices will be compared with the RPM_FUZZY method. Secondly, motion compensation with the inner feedback-loop integrator will be demonstrated to show the improvements of the motion trajectory and the smoothness index in panning conditions. Furthermore, we describe the results of applying the fuzzy inference algorithm to stabilize the image sequences in various conditions, which can reduce the drawbacks and keep the merits of each motion compensation method.