影像感測器之色彩濾波陣列補插與輪式機器人之視覺追蹤控制設計

(1)

國立交通大學

電機與控制工程學系

博士論文

影像感測器之色彩濾波陣列補插與

輪式機器人之視覺追蹤控制設計

Color Filter Array Interpolation for an Image

Sensor and Visual Tracking Control Design of

a Wheeled Mobile Robot

研究生：蔡奇謚

指導教授：宋開泰博士

(2)

影像感測器之色彩濾波陣列補插與

輪式機器人之視覺追蹤控制設計

Color Filter Array Interpolation for an Image Sensor and

Visual Tracking Control Design of a Wheeled Mobile

Robot

研究生：蔡奇謚 Student: Chi-Yi Tsai

指導教授：宋開泰博士 Advisor: Dr. Kai-Tai Song

國立交通大學

電機與控制工程學系博士論文

A Dissertation

Submitted to Department of Electrical and Control Engineering College of Electrical and Computer Engineering

National Chiao Tung University in Partial Fulfillment of the Requirements

for the Degree of Doctor of Philosophy

in

Electrical and Control Engineering July 2008

Hsinchu, Taiwan, Republic of China

(3)

影像感測器之色彩濾波陣列補插與

輪式機器人之視覺追蹤控制設計

學生:蔡奇謚指導教授:宋開泰博士

國立交通大學電機與控制工程學系

摘要

本論文的目標為發展一新穎的色彩濾波陣列(Color Filter Array, CFA)補插演算法用以解決單一影像感測器之色彩重現問題，以及一具有強健性的視覺追蹤控制系統用以解決輪式移動型機器人之以視覺為基礎的移動控制問題。目前大部分的數位相機均採用一貝爾模式(Bayer Pattern)之 CFA 覆蓋於單一影像感測器上，用以擷取一貝爾模式之馬賽克影像(Bayer Mosaic Image)。一般所見的全彩影像則是將擷取到的貝爾模式之馬賽克影像，經由一色彩重現處理後，重建出所有像素上的色彩資訊。此色彩重現處理一般稱為 CFA 補插或 CFA 解馬賽克。為了重建出具有高逼真度的全彩影像，本論文提出一新穎的異次投影硬性決定(Heterogeneity-Projection Hard-Decision, HPHD)演算法並結合一新的以色差為基礎之邊緣適應性(Color-Difference Based Edge-Adaptive, CDEA)CFA 補插方

法來解決貝爾模式之馬賽克影像的色彩重現問題。在此所提出的HPHD 演算法主要目的

為估測出最佳的補插方向並且執行硬性決定捕差法，此方法的主要特點在於補插的方向

決定於執行補插動作之前，如此可提高色彩補插的效率。另一方面，所提出的CDEA CFA

補插方法致力於將綠色色彩平面之高頻資訊加入其他色彩平面中，用以重現出較少色彩遺物的色彩值。

(4)

在視覺追蹤控制的設計中，本論文提出一應用於配有一傾斜式相機之輪式移動型機器人的強健型視覺追蹤控制系統，此系統包含一視覺追蹤控制器(Visual Tracking Controller, VTC)及一視覺狀態估測器(Visual State Estimator, VSE)。首先，一新穎的雙重 Jacobian 視覺互動模型被推導出來，用以幫助 VTC 及 VSE 的設計。VSE 主要目的為在影像平面中直接估測出最佳系統狀態以及目標物之移動，VTC 則利用 VSE 所估測出的結果，接著計算出機器人的控制速度。為了掌握實際視覺追蹤系統中所遭受到的不確定性，VSE 可以克服由影像雜訊及短暫遮蔽不確定性所造成之干擾。另一方面，VTC 不但對於系統參數之不確定性具有某種程度的強健性，並且也能克服速度命令中無法模型化的量化影響。因此，結合所提出的VTC 及所提出的 VSE，此視覺追蹤控制系統對於影像雜訊、系統參數、速度量化及短暫遮蔽所造成之不確定性影響皆具有強健性。電腦模擬及實驗結果驗證所提出的視覺追蹤控制系統之效果，包含追蹤性能、系統收斂性及系統強健性。

(5)

Color Filter Array Interpolation for an Image Sensor and

Visual Tracking Control Design of a Wheeled Mobile

Robot

Student: Chi-Yi Tsai

Advisor: Dr. Kai-Tai Song

Department of Electrical and Control Engineering

National Chiao Tung University

ABSTRACT

The objective of this thesis is to develop a novel color filter array (CFA) interpolation algorithm for color reproduction of a single image sensor and a robust visual tracking control system for vision-based motion control of a wheeled mobile robot. Most digital cameras employ a single image sensor covered with a Bayer CFA to capture a Bayer mosaic image. A full-color image is then reconstructed from the captured Bayer mosaic image through a color reproduction process, commonly known as CFA interpolation or CFA demosaicing. To reconstruct the full-color images with high fidelity, a novel heterogeneity-projection hard-decision (HPHD) algorithm combined with a new color-difference based edge-adaptive (CDEA) CFA interpolation method is proposed for color reproduction of Bayer mosaic images. The proposed HPHD algorithm aims to estimate the optimal interpolation direction and perform hard-decision interpolation, in which the direction of interpolation is decided before performing the interpolation. On the other hand, the proposed CDEA CFA

(6)

interpolation method devotes to reproduce color values with fewer color artifacts by adding the high-frequency information of green channel to other color channels. Compared with three recently reported CFA interpolation techniques, the proposed HPHD-CDEA method outperforms all of them in both quantitative and visual comparisons by utilizing twenty-five natural images from Kodak PhotoCD.

In the design of visual tracking control, a robust visual tracking control system, which consists of a visual tracking controller (VTC) and a visual state estimator (VSE), is proposed for a wheeled mobile robot equipped with a tilt camera. A novel dual-Jacobian visual interaction model is first derived to help the design of VTC and VSE. The VSE aims to estimate the optimal system state and target motion in the image plane directly, and the VTC then calculates the robot’s control velocities by using the estimation results from VSE. To handle the uncertainties encountered in practical visual tracking control system, the VSE can overcome the disturbances caused by both image noise and temporary occlusion uncertainties. On the other hand, the VTC not only possesses some degree of robustness against the system model uncertainties, but also overcomes the unmodelled quantization effect in the velocity commands. Therefore, by combining the proposed VTC with the proposed VSE, the visual tracking control system is robust to the uncertainties of image noise, system model, velocity quantization and temporary occlusion. Computer simulations and experimental results validate the effectiveness of the proposed visual tracking control system, in terms of tracking performance, system convergence, and robustness.

(7)

誌謝

博士班的訓練是一條辛苦又漫長的研究之路。衷心感謝我的指導教授宋開泰博士，感謝他多年來在專業上及論文寫作上的指導，真摯誠意且不厭其煩的給我意見及修正，使我受益良多，也讓本論文得以順利完成。接著，必須感謝論文口試委員－傅立成教授、王文俊教授、李迪章教授、陳建祥教授、王聖智教授、楊谷洋教授，對於本論文的建議與指引，強化本論文的嚴整性與可讀性。再來，要感謝交通大學電機與控制工程研究所的教授們－李祖添教授、吳炳飛教授、林昇甫教授、林清安教授、以及林心宇教授對於學業上的教導以及觀念上的啟發。感謝 ISCI 實驗室的學弟們富聖、振揚、俊瑋、濬尉及志昇所提供的協助，也感謝學長戴任詔博士和博士班學弟嘉豪及孟儒對本論文的建議與討論，以及感謝已畢業的學弟們松峙、維祥、鎮源、崇民及柏秋的相互鼓勵及在生活上帶來的樂趣。另外，特別感謝我的父母，由於他們辛苦栽培，在生活上給予我細心地關愛與照料，使得我才得以順利完成此論文；也感謝我女友在我的身邊全力的支持我，在我最無助及失意的時候給予我意見及鼓勵。最後，感謝全能的上帝，將一切榮耀歸給上帝。

(8)

List of Figures

Fig. 1-1: Structure of the thesis. ... 10 Fig. 2-1: The Bayer pattern... 14 Fig. 2-2: The concept of horizontal heterogeneity-projection from a 1×5 row data of a Bayer mosaic image. ... 19 Fig. 2-3: The concept of (a) horizontal and (b) vertical adaptive filtering using a 1×3

rectangular window... 21 Fig. 2-4: Test images used in the experiment. ... 24 Fig. 2-5: Flowchart for demonstrating the assumption of color-difference model... 25 Fig. 2-6: (a) The red color-difference value on blue pixel and (b) the red color-difference

value on green pixel of a central pixel to be estimated. ... 29 Fig. 2-7: Two cases of missing green value on the central pixel. (a) The green value on blue pixel and (b) the green value on red pixel of a central pixel to be estimated... 32 Fig. 2-8: Flowchart of the proposed HPHD-CDEA CFA interpolation algorithm. ... 36 Fig. 2-9: Illustration of execution steps of the proposed HPHD-CDEA CFA interpolation

algorithm. (a) Original Bayer mosaic image of small Lighthouse image (384x256). (b) Horizontal heterogeneity map Hh_map (N =24). (c) Vertical heterogeneity map Hv_map. (d) Filtered horizontal heterogeneity map *

_ map

h

H . (e) Filtered vertical heterogeneity map *

_ map

v

H . (f) Three decided subsets in the image (α =0.8). The gray region is the horizontal subset Ω , the white region is the vertical subset _h Ω , _v and the black region is the smooth subset Ω . (g) Interpolation result using the _s proposed CDEA CFA interpolation presented in Section 2.3... 37 Fig. 2-10: (a) Zoom-in of the original Lighthouse image in the fence region. Zoom-in of the demosaicing results with parameters N =24 and (b) α =0, (c) α =0.5, (d)

8 . 0 =

(13)

Fig. 3-1: (a) A model of the wheeled mobile robot and the target in the world coordinate frame. (b) Side view of the wheeled mobile robot with a tilt camera mounted on top of it. 43

Fig. 3-2: World, camera and image coordinate frames of robotic visual interaction. ... 45

Fig. 3-3: The definition of observed system state in the image plane. ... 47

Fig. 3-4: Depicts the concept of dual-Jacobian visual interaction model (3.11). ... 49

Fig. 3-5: Physical meaning of the singularity condition (3.26)... 54

Fig. 3-6: Projection error in dx... 55

Fig. 4-1: Concept of time series linear-least-squares regression. ... 71

Fig. 4-2: Architecture of the proposed VSE-WoTV... 72

Fig. 5-1: Flowchart of the experiment. In the interpolation step, we compare the performance of bilinear, Lu’s, Gunturk’s, Li’s and proposed HPHD-CDLi, HPHD-CDEA methods. In post-processing step, Lu’s post-processing method is adopted into each CFA interpolation method... 76

Fig. 5-2: Zoom-in demosaicing results of test image No. 1. (a) Original picture; Demosaiced result in interpolation step: (b) Gunturk’s method, (c) Lu’s method, (d) Li’s method, (e) HPHD-CDLi method, and (f) HPHD-CDEA method... 80

Fig. 5-3: Zoom-in demosaicing results of test image No. 20. (a) Original picture; Demosaiced result in interpolation step: (b) Gunturk’s method, (c) Lu’s method, (d) Li’s method, (e) HPHD-CDLi method, and (f) HPHD-CDEA method... 81

Fig. 5-4: Two experimental mobile robots developed in the Intelligent System Control Integration (ISCI) Lab, National Chiao Tung University. ... 84

Fig. 5-5: Implemented VTC (presented in Section 3.3) combined with the VSE-WTV (presented in Section 4.2) to test in the visual tracking control experiments... 84

Fig. 5-6: Computer simulation results of experiment 1 without velocity quantization error. (a) Robot trajectory in the world coordinates. (b) Tracking errors in the image plane. (c) Control velocities of the center point and tilt camera of tracking robot... 88

(14)

Fig. 5-7: Experimental results of experiment 1 without using the proposed robust control law (3.46). (a) Robot trajectory in the world coordinates. (b) Tracking errors the in the image plane. (c) Control velocities of the center point and tilt camera of tracking robot... 89 Fig. 5-8: Experimental results of experiment 1 using the proposed robust control law (3.46). (a) Robot trajectory in the world coordinates. (b) Tracking errors the in the image plane. (c) Control velocities of the center point and tilt camera of tracking robot.... 90 Fig. 5-9: Experimental results of experiment 2 using control gains (α1,α2,α3)_S =

(5/64,6/16,4/64). (a) Robot trajectory in the world coordinates. (b) Tracking errors in the image plane. (c) Control velocities of the center point and tilt camera of tracking robot... 92 Fig. 5-10: Experimental results of experiment 2 using control gains (α1,α2,α3)_L =

(5/16,6/8,4/16). (a) Robot trajectory in the world coordinates. (b) Tracking errors in the image plane. (c) Control velocities of the center point and tilt camera of tracking robot... 94 Fig. 5-11: Simulation setup for the performance evaluation of the proposed VSE-WoTV

presented in Section 4.3... 95 Fig. 5-12: The computer simulation results of the proposed VTC combined with the

VSE-WoTV. (a) Robot trajectory in the world coordinate frame. (b) Control velocities of the center point and tilt camera of tracking robot. (c) Tracking errors with random noise. (d) Tracking errors estimated by the VSE-WoTV. (e) Estimated target image velocity. (f) Estimation errors. ... 97 Fig. 5-13: Experimental results of tracking a moving target when it is temporarily partially

occluded. (a) Before partial occlusion. (b)-(d) Partial occlusion occurred. (e)-(f) After partial occlusion, the moving target was still under tracking... 99 Fig. 5-14: Experimental results of tracking a moving target when it is temporarily fully

(15)

occluded. (a) Before full occlusion. (b)-(e) Full occlusion occurred. The moving target is estimated only using prediction information. (f) After fully occlusion, the moving target was still under tracking. ... 99 Fig. 5-15: Estimated tracking errors compared with observed tracking errors. ... 100 Fig. 5-16: Experimental mobile robot used to test the tracking performance of the proposed VTC combined with the VSE-WoTV presented in Section 4.3. ... 101 Fig. 5-17: Block diagram of the visual tracking control system tested in the experiment of

visual tracking of a moving person. ... 102 Fig. 5-18: Experimental results. (a1-a7): Image sequence recorded from a DV camera. (b1-b7):

Corresponding image sequence recorded from on-board USB camera. (c-d): Recorded tracking errors in the image plane. (f-h): Target image velocity estimates. (i-j): Command linear and angular velocities of the mobile robot. (k): Command velocity of the tilt camera. ... 103 Fig. 5-19: Estimated tracking errors compared with observed tracking errors. ... 104 Fig. 5-20: Experimental pan-tile platform used to demonstrate the robust property of the

proposed visual tracking scheme... 106 Fig. 5-21: The experimental results of occlusion using VSE-WoTV. (a)-(c) Recorded camera view, observation states and propagation states, (d) variance of propagation states, (e) variance of observation states... 106 Fig. A-1: Experimental results of tuning parameters in each step. (a) Evolution of PSNR_Avg as

the parameter N increases. (b) Evolution of PSNR_Avg as the parameter α increases.

(c) Influence of the parameters (N,α) on the performance gap ∆PSNR_Avg between post-processing and interpolation steps... 114 Fig. B-1: Flowchart of the (a) soft-decision; (b) enhanced soft-decision and (c) proposed

(16)

Fig. C-1: Test images used in the extended visual comparison.(a)Lighthouse.(b)Window. .. 119 Fig. C-2: (a) and (d) show the zoom-in of the original Lighthouse image in the fence and

house regions, respectively. (b) and (e) show the zoom-in of the demosaicing results using parameters (Nopt,αopt)=(11,0.6) . (c) and (f) show the zoom-in of the demosaicing results using parameters (N,α)=(24,0.8)... 119 Fig. C-3: Zoom-in demosaicing results of Lighthouse image in fence region. (a) Original

picture; Demosaiced result in interpolation step: (b) Lu’s method, (c) Gunturk’s method, (d) Li’s method, (e) Muresan’s method, (f) Grossman’s method, (g) Omer’s method, (h) HPHD-CDEA method... 121 Fig. C-4: Zoom-in demosaicing results of Lighthouse image in house region. (a) Original

picture; Demosaiced result in interpolation step: (b) Lu’s method, (c) Gunturk’s method, (d) Li’s method, (e) Muresan’s method, (f) Grossman’s method, (g) Omer’s method, (h) HPHD-CDEA method... 122 Fig. C-5: Zoom-in demosaicing results of Window image. (a) Original picture; Demosaiced

result in interpolation step: (b) Lu’s method, (c) Gunturk’s method, (d) Li’s method, (e) Muresan’s method, (f) Grossman’s method, (g) Omer’s method, (h) HPHD-CDEA method. ... 123 Fig. D-1: Simulation result of the distance between mobile robot and motion target, || Xf ||. 126

(17)

List of Tables

Table 2-1: Comparison of mean squared error at each step in Fig. 2-5... 26

Table 5-1: Performance comparison among recent proposed methods: PSNR (dB) and _∆ * ab E measures of demosaiced images in the interpolation and post-processing steps. .. 77

Table 5-2: Calculations performed for reconstructing one color pixel... 82

Table 5-3: Computational cost per color pixel of different CFA interpolation techniques... 82

Table 5-4: Parameters used in the experiment of visual tracking of a moving robot. ... 87

Table 5-5: Parameters used in the simulations of visual state estimation. ... 96

(18)

Chapter 1 Introduction

1.1 Motivation

An intelligent robot uses its on-board sensors to collect information from the surroundings and react to the changes of its immediate environment. In recent years, vision systems become one of the major on-board sensors for autonomous robots. Most vision systems utilize a monocular digital camera to capture full-color images of the environment for several purposes. A full-color image usually is composed of three color planes and required three separate image sensors to measure each color plane. In order to reduce the cost, many cameras employ a single image sensor covered with a color filter array (CFA), which consists

of a set of spectrally selective filters that are arranged in an interleaved pattern so that each sensor pixel samples one of three primary color components, to capture the mosaic images. To render a full-color image from a mosaic image, a color reproduction process, commonly known as CFA interpolation or CFA demosaicing, is required to estimate for each pixel its

two missing color values. Therefore, the researches on CFA interpolation for color reproduction have become an important topic in digital camera pipeline process [1-11].

The rendered full-color images of the environment can be used in several robotic applications, for example, recognition, navigation, localization, tracking control, etc. In recent years, computer vision techniques act a key role in robotic systems for making robot motion control and object tracking efficiently. Thus the study of visual tracking control (which means

the vision-based robot motion control to track a target of interest) has gained increasing attention in recent years [12-36]. Based on the motion constraints of the robot, the research on visual tracking control can be classified into visual servoing for holonomic manipulators and

(19)

visual tracking for nonholonomic mobile robots. Although visual servoing of holonomic manipulators has been discussed extensively and many results can be found in the literature [12-14], mobile robots are commonly nonholonomic and the visual servoing results for holonomic manipulators are unsuitable for the mobile platform [15]. Hence, the researches on mobile robot visual tracking control have been an active area in robotic researches [15-36]. According to the reasons above, this thesis aims to investigate the methodologies of CFA interpolation for a single image sensor and visual tracking control for a wheeled mobile robot.

1.2 Literature Survey

1.2.1 CFA Interpolation for a Single Image Sensor

Digital color images from single-chip digital cameras are obtained by interpolating the output from a CFA. The simplest CFA interpolation methodologies apply well-known image interpolation techniques, such as nearest-neighbor replication, bilinear interpolation, and cubic spline interpolation, to each color channel separately. However, these single-channel algorithms usually introduce severe color artifacts and blurs around sharp edges [1]. These drawbacks motivate the need of more advanced algorithms for improving demosaicing performance. An excellent review on advanced CFA interpolation algorithms can be found in [2].

In recent years, there have been investigations on more sophisticated CFA interpolation algorithms. In [3], Lu and Tan presented an improved hybrid CFA interpolation method that consists of two successive steps: an interpolation step to render full-color images and a post-processing step to suppress visible demosaicing artifacts. Muresan and Parks proposed an improved edge-directed CFA interpolation algorithm based on optimal recovery interpolation of grayscale images [4]. They first utilized a grayscale image interpolation algorithm based on optimal recovery estimation theory to interpolate the green plane. The

(20)

red/blue channels were interpolated using inter-channel color difference adaptive filtering. These two CFA interpolation algorithms in general produce high quality visual results, especially in reconstructing sharp or well-defined edges of the image. However, in fine details or texture regions, where edges tend to be short and in different directions, these algorithms introduce undesirable errors and give degraded performance.

Meanwhile, two iterative CFA interpolation techniques were proposed by Gunturk et al.

[5] and Li [6], respectively. In [5], a projection-onto-convex-set (POCS) technique was presented to estimate the missing color values in red and blue channels using alternating projection scheme based on high inter-channel correlation. In [6], Li formulated the CFA interpolation as a problem of reconstructing correlated signals from decimated versions and proposed a successive approximation strategy by adopting color difference interpolation iteratively. Although these iterative CFA interpolation algorithms perform well in texture regions and reveal low computational complexity, they cannot produce satisfactorily high quality visual results in well-defined edges of the image.

Another recent CFA interpolation approach divides the demosaicing procedure into interpolation stage and decision stage [7-10]. In the interpolation stage, horizontally and vertically interpolated images are produced respectively. In the latter decision stage, a soft-decision method, in which the interpolation must be performed before the decision procedure, was employed for choosing the pixels interpolated in the direction with fewer artifacts. Because the decision stage is essential for these CFA interpolation approaches, we refer them as decision-based CFA interpolation algorithm. For the decision stage, Hirakawa et al. proposed a homogeneity metric to measure the misguidance level of color artifacts

presented in interpolated images [7]. Based on this measurement, the interpolation decision is made by choosing the region with larger homogeneity values. In [8], Wu et al. adopted the

Fisher’s linear discriminant technique to determine the optimal interpolation direction in a local window. In [9], Grossmann and Eldar utilized the YIQ color space as a tool to select the

(21)

reconstructed regions with a smoother chrominance component. Recently, Omer and Werman proposed an enhanced decision-based CFA interpolation algorithm that combines the decision process with the standard CFA interpolation algorithm such as edge-directed scheme [11] to improve its performance in places the standard algorithm tends to fail [10]. The decision-based CFA interpolation algorithm performs well not only in texture regions, but also in well-defined edges of the image. However, the main drawback of these CFA interpolation algorithms is that they are not efficient in the interpolation stage because each pixel needs to be interpolated at least twice, one in horizontal direction and the other in vertical direction, for the next soft-decision procedure. This drawback also greatly increases the computing efforts in the latter decision stage. Therefore, it is still a challenge in CFA interpolation design to develop an efficient CFA interpolation method with high performance in both texture and edge regions.

1.2.2 Visual Tracking Control for a Wheeled Mobile Robot

The visual tracking control problem addressed in this thesis focuses on the visual tracking control of a unicycle-modeled (usually termed as wheeled) mobile robot equipped with an on-board monocular vision system. Due to the high number of different mobile robot

visual tracking control methods, we classify the reported methods into four groups based on the type of the target to be tracked. Many efforts focus on the first group which aims to track a static target, such as a ground line, landmark, or reference image, for the purpose of mobile robot navigation or regulation (so-called homing) [15-28]. To track the ground line, Ma et al.

formulated the visual tracking control problem as controlling the shape of a ground curve in the image plane and proposed a closed-loop vision-guided control system for a nonholonomic mobile robot [16]. Coulaud et al. proposed a simple and stable feedback controller design,

which avoids sophisticated image processing and control algorithms, for a mobile robot equipped with a fixed camera to track a line on the ground [17]. In the case of tracking the

(22)

landmark, the reported controllers usually modify the visual servoing technique to satisfy the nonholonomic constraint for the motion control of the mobile robot [18-21]. In [22], Zhang and Ostrowski utilized an optimal control method to solve the visual motion-planning problem by generating a virtual trajectory in the image plane and the corresponding optimal control signals for the robot to follow. Nierobisch et al. proposed a visual tracking control

method for a mobile robot with a pan-tilt camera to track visual reference landmarks in the acquired views during autonomous navigation [23]. Recently, the homography-based [24, 25] and epipolar-based [15, 26-28] visual tracking control approaches were proposed for a mobile robot equipped with a pinhole or an omni-directional (so-called central catadioptric) camera to track a reference image toward a desired configuration. These two approaches consider the mobile robot visual tracking control problem as a visual servoing regulation or visual homing problem. In [24], Chen et al. developed a visual tracking controller based on the Euclidean

homography to track a desired time-varying trajectory defined by a prerecorded image sequence of a stationary target viewed by the on-board camera as the mobile robot moves. However, the stability of their result is restricted by the non-zero reference velocity condition of the desired trajectory. To overcome this drawback, Fang et al. exploited Lyapunov-based

techniques to construct a homography-based visual servoing regulation controller for proving asymptotic regulation of the mobile robot [25]. In [26], Mariottini et al. exploited the epipolar

geometry defined by the current and desired camera views to develop a two-step visual servoing regulation controller. They also extend this design to the visual servoing regulation control of a mobile robot with a central catadioptric camera [27]. In [28], Goedemé et al.

developed a vision-only navigation and homing system for mobile robots with an omni-directional camera. Their method divides the visual homing operation into two phases and computes visual homing vector based on epipolar geometry estimation. Although these approaches of the first group provide appropriate solutions for static target visual tracking control problem, they cannot guarantee to solve the moving (non-static) target visual tracking

(23)

control problem.

The second group aims to track other robot teammates in a robot group for the formation control purpose [29, 30]. The proposed approaches in this group usually are designed based on the central catadioptric camera model in order to detect all robot teammates at the same instant. The subject of the third group is to track a predictable moving target, such as a projectile or straight moving ball, for mobile robot interception purpose [31, 32]. In [31], Borgstadt et al. utilized a human vision-based strategy to guide a mobile robot to intercept a

projectile ball. Similarly, Capparella et al. extended the concept of human-like strategy to

develop a vision-based two-level interception approach, which contains a lower level controller to control the on-board pan-tilt camera and a higher lever controller to operate the mobile robot platform, for intercepting a straight moving ball [32]. A common point of the second and third group is that the motion of the target of interest is known and predictable. However, in some robotic applications, a mobile robot requires to track a dynamic and unpredictable motion target, such as a human’s face, for the purpose of pursuit or interaction. Thus, the existent methods of the aforementioned two groups are not suitable to solve the dynamic moving target visual tracking control problem.

The purpose of the fourth group aims to solve the problem of tracking a dynamic moving target [33-37]. In [33], Wang et al. proposed an adaptive backstepping control law based on

an image-based camera-target visual interaction model to track a dynamic moving target with unknown height parameter. Although the approach in [33] guarantees the asymptotic stability of closed-loop visual tracking control system in tracking a dynamic moving target, the case of tracking a static target cannot be guaranteed due to the non-zero restrictions on the reference velocity of the mobile robot. In [34], Song et al. combined a face detection algorithm with a

PID controller to track a moving person in a home setting. The main disadvantage of their method is that it cannot guarantee the stability of the closed-loop visual tracking system based on a stability criterion. In [35], Malis et al. integrated template-based visual tracking

(24)

algorithms and model-free vision-based control techniques to build a flexible and robust visual tracking control system for various robotic applications. Because their visual tracking result is based on the homography estimation, which requires two images of the target pattern to estimate the optimal homography, the reported system only overcomes the partial occlusion problem but fails in the fully occlusion problem. In [36], Han et al. proposed an image-based

visual tracking control scheme for a mobile robot to estimate the position of the target in the next image and track the target to the central area of the image. Since their method utilized the differential approximation method to estimate the velocity of target in the image plane, the estimation result is very sensitive to the image noise. Recently, a visual interaction controller had been proposed for a unicycle-modeled mobile robot to track a dynamic moving target such as human’s face [37]. The drawback of this method is that the controller requires the target’s 3D motion velocity, which is difficult to estimate when only a monocular camera is used.

Therefore, from the literature survey, one of the most important challenges in mobile robot visual tracking control design is to develop a visual tracking control system to estimate the motion of the dynamic moving target and track it based on a stability criterion. Further, in realization of the control schemes, it has been noted that the disturbances of image noise, velocity quantization error and temporary (partial/full) occlusion degrade the performance of the controller and might make the system unstable. These problems have not yet been clarified in many existent related works and hence motivated us to investigate the robustness of the visual tracking control system against the uncertainties of image noise, system model, velocity quantization and temporary occlusion.

1.3 Research Objectives

The objective of this thesis is to develop a novel CFA interpolation algorithm for color reproduction of a single image sensor and a robust visual tracking control system for

(25)

vision-based motion control of a wheeled mobile robot. For color reproduction, a novel heterogeneity-projection hard-decision (HPHD) CFA interpolation algorithm is proposed to reconstruct the full-color images from Bayer mosaic images [38]. The proposed HPHD CFA interpolation algorithm can combine with many existent image interpolation methods to reconstruct each color plane. However, in CFA interpolation process, it should be noticed that color artifacts existing around edges and fine textures of the demosaiced image are a factor limiting performance. Color artifacts are caused primarily by aliasing error in high-frequency regions such as edges or fine textures, and existing algorithms are unable to resolve color artifacts in these regions effectively to obtain demosaiced results with high visual quality. To effectively reduce color artifacts in demosaiced images, a novel color-difference based edge-adaptive (CDEA) CFA interpolation algorithm is then proposed and combined with the HPHD CFA interpolation algorithm to reproduce color values by exploiting the green plane information for making high-frequency components of red and blue planes similar to the green plane, in which the aliasing errors are usually much smaller than those in red and blue planes.

For visual tracking control, a novel design of a robust visual tracking control system, which consists of a visual tracking controller (VTC) and a visual state estimator (VSE), is proposed for a wheeled mobile robot equipped with a tilt camera. In order to resolve the problem of visual tracking a dynamic moving target and guarantee the stability of the closed-loop visual tracking system, a novel dual-Jacobian visual interaction model is derived to help the design of VTC and VSE. The VSE is constructed by a real-time self-tuning Kalman filter and aims to estimate the optimal system state and target motion in the image plane directly for later use by the VTC. The VTC then calculates the robot’s control velocities in the image plane directly. To handle the uncertainties encountered in practical visual tracking control system, the VSE can estimate the optimal target state from the observed image with disturbances to overcome both image noise and temporary occlusion uncertainties.

(26)

On the other hand, the VTC not only possesses some degree of robustness against the system model uncertainties, but also overcomes the unmodelled quantization effect in the velocity commands. Therefore, the proposed system is robust to the uncertainties of image noise, system model, velocity quantization and temporary occlusion. This advantage enhances the reliability of the proposed visual tracking control system in practical applications.

1.4 Organization of the Thesis

The remainder of this thesis is organized as follows: In Chapter 2, a novel HPHD CFA interpolation algorithm is first developed based on the concept of spectral-spatial correlation. A new CDEA CFA interpolation algorithm is then proposed to combine with HPHD CFA interpolation algorithm for color reproduction of Bayer mosaic images. In Chapter 3, a novel dual-Jacobian visual interaction model is derived in order to help the design of mobile robot visual tracking control for tracking a dynamic moving target. Accordingly, the results of VTC design with robustness analysis are developed by using the proposed visual interaction model. Furthermore, the robust control law to overcome the velocity quantization error encountered in practical systems will also be presented. Chapter 4 develops the VSE using Kalman filter with self-tuning algorithm to estimate the optimal system state in the image plane for handling the uncertainties caused by image noise and temporary occlusion. Simulation and experimental results of the proposed CFA interpolation algorithm and the proposed mobile robot visual tracking control system are reported and discussed in Chapter 5. Chapter 6 concludes the contributions of this work and provides the recommendations for future research. In Appendix A, an experiment of tweaking parameters is presented to find the local optimal parameters for the proposed HPHD-CDEA CFA interpolation algorithm described in Chapter 2. Appendixes B and C present more discussions and visual comparisons of the proposed CFA interpolation algorithm. An extended discussion on the proposed VTC scheme is presented in Appendix D. Figure 1-1 shows the simplified overview of the thesis.

(27)

(28)

Chapter 2 CFA Interpolation for Color Reproduction of Bayer

Mosaic Images

2.1 Introduction

In this chapter, the design of the proposed HPHD-CDEA CFA interpolation algorithm is presented for color reproduction from Bayer mosaic images. The proposed algorithm consists of HPHD algorithm and CDEA CFA interpolation. The proposed HPHD algorithm aims to estimate the optimal interpolation direction before performing CFA interpolation. Because the decision stage is performed before the interpolation stage (termed as hard-decision interpolation), each pixel only needs to be interpolated once. To do so, a new heterogeneity-projection scheme based on a novel spectral-spatial correlation concept is proposed to estimate the best interpolation direction directly from the original Bayer mosaic image. Using the proposed heterogeneity-projection scheme, a hard-decision rule can be decided before performing CFA interpolation.

In order to effectively reduce color artifacts in CFA interpolation process, the proposed CDEA CFA interpolation algorithm is then combined with the HPHD algorithm to reconstruct the red and blue color planes by exploiting the green plane information for making high-frequency components of red and blue planes similar to the green plane, in which the aliasing errors are usually much smaller than those in red and blue planes. To do so, the red and blue channels are first reconstructed using bilinear interpolation and then edge-adaptive filtered in color-difference space. To reconstruct the green plane, any existent image interpolation methods can be employed to combine with the CDEA CFA interpolation

(29)

algorithm. Moreover, we also present a new edge-adaptive interpolation method to reconstruct the green channel from CFA samples. The advantages of the proposed HPHD-CDEA CFA interpolation algorithm are summarized as follows:

1) Many existent CFA interpolation methods can be combined with the proposed HPHD algorithm to reconstruct each color plane. For example, the proposed heterogeneity-projection scheme can combine with existent decision-based CFA interpolation algorithms. More specifically, the proposed heterogeneity-projection scheme can adopt into the decision step of existent decision-based CFA interpolation algorithms. 2) Each pixel only has to be interpolated once. Therefore, the proposed algorithm is much

more efficient than other decision-based schemes.

3) Any existing image interpolation methods can be combined with the proposed CDEA CFA interpolation algorithm to reconstruct the green plane.

4) The proposed HPHD-CDEA CFA interpolation algorithm performs well not only in texture regions, but also in well-defined edges of the image.

The rest of this chapter is organized as follows: In Section 2.2, the proposed HPHD CFA interpolation algorithm is designed by using a novel concept of spectral-spatial correlation. Section 2.3 presents the proposed CDEA CFA interpolation algorithm based on color-difference model. A new edge-adaptive interpolation method to reconstruct the green plane from CFA samples is also presented in this section. Section 2.4 describes the complete HPHD-CDEA CFA interpolation algorithm and gives an example study. Section 2.5 summarizes the contributions of this work. An experiment of tweaking parameters to find the local optimal parameters for the proposed CFA interpolation algorithm is presented in Appendix A. More discussions and visual comparisons are presented in Appendixes B and C.

(30)

2.2 Heterogeneity-Projection Hard-Decision (HPHD) CFA interpolation

Figure 2-1 shows the most used CFA pattern, the Bayer pattern [38], where R, G and B denote, respectively, the pixels having only red, green and blue color values. We limit our discussion to the Bayer pattern in this chapter because it is so popular. In the following, image spectral and spatial correlations are first introduced. A novel spectral-spatial correlation is then derived based on these two correlations.

2.2.1 Spectral and Spatial Correlations

Many existing demosaicing methods are developed using image spectral and/or spatial correlation. The concept of spectral correlation is based on the assumption that the color difference signals are locally constant in chrominance smooth areas [39]. Let [R G B] denote three color planes of a nature color image, the concept of spectral correlation leads to the following assumption.

A1) The color differences between green and red/blue channels satisfy the following conditions ) , ( ) , ( ) , (x y G x y A x y R = + _rg and B(x,y)=G(x,y)+A_bg(x,y),

where A_rg( yx, ) and A_bg( yx, ) are piecewise constant within the boundary of a given object.

The spatial correlation reflects the fact that within a homogeneous image region, neighboring pixels share similar color values [40]. In other words, the difference between neighboring pixel values along an edge direction in spatial domain is a constant. Thus we have the following assumption based on the concept of spatial correlation [3].

(31)

Fig. 2-1:The Bayer pattern.

A2) The rate of change of neighboring pixel values along an edge direction is a constant.

To illustrate this, let us consider the interpolation of R in Fig. 2-1. Suppose that the pixel ₃₃

33

R is located on a horizontal edge. Based on A1), the neighboring pixels of R along the ₃₃

horizontal direction have the following relationship between green and red/blue pixel values

) , 1 ( ) , ( ) , 1 (x y A x y A x y A_rg − = _rg = _rg + and ) , 1 ( ) , ( ) , 1 (x y A x y A x y A_bg − = _bg = _bg + . (2.1) So we have 43 43 33 33 23 23 G R G R G R − = − = − and B₂₃−G₂₃ =B₃₃−G₃₃ =B₄₃ −G₄₃. (2.2) The assumption A2) gives the following relationship on horizontal edges

h dR R R R R₂₃− ₃₃ = ₃₃ − ₄₃ ≡ , h dG G G G G₂₃− ₃₃ = ₃₃− ₄₃ ≡ , and (2.3) h dB B B B B₂₃− ₃₃ = ₃₃ − ₄₃ ≡ ,

where G₃₃, R₂₃, R₄₃, B₂₃, B₃₃, and B₄₃ denote the missing color values at the respective pixel locations. dG , _h dR and _h dB are constants. _h

(32)

2.2.2 Spectral-Spatial Correlation (SSC)

A significant characteristic of Bayer pattern is that for each pixel, the surrounding pixels are one of the primary components in different channels. It is then interesting to investigate the relationship between neighboring pixels in different color channels. Consider the following situation: on a horizontal edge, two green pixels surround a red pixel on horizontal direction. Take the difference between the center red pixel and right green pixel, we then have

[

( , ) ( , )

] [

( , ) ( 1, )

]

) , 1 ( ) , (x y G x y R x y G x y G x y G x y R − + = − + − + , (2.4)

where G( yx, ) denotes the missing green value at center red pixel location. Recall assumptions A1) and A2), expression (2.4) becomes such that

h rg x x h rg R x y G x y A x y dG S ( , +1) ≡ ₍ _, ₎− ₍ +₁_, ₎= ₍ _, ₎+ _._(2.5) Similarly, the difference between a blue pixel and its right green pixel is given by

h bg x x h bg B x y G x y A x y dG S ( , +1) ≡ ₍ _, ₎− ₍ +₁_, ₎= ₍ _, ₎+ _._(2.6) The same results also can be obtained along vertical direction on a vertical edge such that

v rg y y v rg R x y G x y A x y dG S ( , +1) ≡ ₍ _, ₎− ₍ _, +₁₎= ₍ _, ₎+ _{, and} v bg y y v bg B x y G x y A x y dG S ( , +1) ≡ ₍ _, ₎− ₍ _, +₁₎= ₍ _, ₎+ _. _(2.7) Expressions (2.5)-(2.7) show that the difference between surrounding pixels in different color channels is equal to the summation of spectral and spatial correlations. We refer these relationships (2.5)-(2.7) as spectral-spatial correlation (SSC). SSC has two important

characteristics. First, SSC can be easily and directly calculated from the original Bayer mosaic image. Second, SSC inherits the characteristics of spectral and spatial correlations. In other words, SSC is also piecewise constant within the boundary of a given object or along an edge direction. Therefore, we have the following assumption based on these observations:

(33)

direction is also piecewise constant.

Assumption A3) is a significant clue for us to find the directional smooth regions in Bayer mosaic images directly before performing the interpolation. In the following section, we will present the method of heterogeneity-projection based on A3).

2.2.3 Heterogeneity-Projection for Bayer Mosaic Images

The proposed heterogeneity-projection scheme transfers the original Bayer mosaic image directly into horizontal and vertical heterogeneity maps, respectively. Using these two heterogeneity maps, the interpolation direction can be determined easily by choosing the smallest heterogeneity values.

Assumption A3) implies that the n-order directional finite derivative of SSC along an edge direction tends toward a small value. For example, consider a red pixel R( yx, ) locates on a horizontal edge, the SSC values of R( yx, ) and its neighboring pixels along horizontal direction can be found such that

h rg x x h rg A x y dG S ( , +1) = ₍ _, ₎+ _, h rg x x h gr A x y dG S ( +1, +2) =− ₍ +₂_, ₎+ _, _(2.8) where )_Sh(x 1,x 2) _G(_x 1,_y) _R(_x 2,_y gr ≡ + − + +

+ _{. Based on the basic definition of the first-order}

derivative of a one-dimensional discrete function, the first-order horizontal derivative of SSC are given by [41] ) , 2 ( ) , ( ) 3 , 2 ( ) 1 , ( ) 3 , ( _S _S _A _x _y _A _x _y dS h x x _rg _rg rg x x h rg x x h rg ≡ − = − + + + + + _, ) , 2 ( ) , 4 ( ) 4 , 3 ( ) 2 , 1 ( ) 4 , 1 ( _S _S _A _x _y _A _x _y dS h x x _rg _rg gr x x h gr x x h gr ≡ − = + − + + + + + + + _{. (2.9)}

Recall A1) and A3), one can see that h(x,x+3)

rg

dS and h(x+ x1, +4)

gr

(34)

along this horizontal edge. Because the higher-order derivative of a discrete function is a linear combination of the first-order ones, it implies the higher-order horizontal derivative of SSC will also approach to zero along the horizontal edge. Thus we have the following assumption.

A4) If pixels locate on a directional edge, then the corresponding nth-order directional derivative of SSC along the edge direction approaches to zero.

Assumption A4) poses a question that how the nth-order directional derivative of SSC can be directly calculated from Bayer mosaic image. To resolve this problem, a heterogeneity-projection scheme is developed to transfer the row data of Bayer mosaic image directly into nth-order directional derivative of SSC. Note that the value of nth-order directional derivative of SSC is defined as heterogeneity measure, because it leads to a small

value within a directional smooth region.

Denote RG₁_×_N =[R₁ G₂ R₃ _L]₁_×_N as a row data of Bayer mosaic image, N is the presetting window size, and H is the corresponding horizontal heterogeneity value. To _h

calculate the horizontal heterogeneity value H from _h RG₁_×_N, we propose the following steps. First, the row data RG₁_×_N is transferred into a 1× N( −3) vector of first-order horizontal derivative of SSC using a linear transformation such that

1 ) 3 ( 1 ) 3 ( 1 ) 6 , 3 ( ) 5 , 2 ( ) 4 , 1 ( 3 ( 1× − =[ ]×N− = ×N N× N− h rg h gr h rg N dS dS dS RG T dS _L , (2.10) where 1

[

1 1 1 1

]

( 3) ) 3 ( − = − − ⊗ − × eye N T T N

N , ⊗ denotes the 2D convolution operator

and eye(M)denotes a M×M identity matrix. Second, because the high-order derivative of a discrete function is derived by the linear combination of its first-order ones, the horizontal

(35)

heterogeneity value H , the _h (N−3)th-order horizontal derivative of SSC, is obtained such that [41] 2 1 ) 3 ( ) 3 ( 1 ) 6 , 3 ( 3 ) 5 , 2 ( 2 ) 4 , 1 ( 1 ) , 3 ( 5 ) 1 , 2 ( 5 ) 2 , 1 ( 5 ) , 2 ( 4 ) 1 , 1 ( 4 ) , 1 ( 3 2 × − − × − − − − − − − − − = + + + = + − = − = ≡ N N h rg h gr h rg N h rg N N h gr N N h rg N N h gr N N h rg N N h rg N h T dS dS w dS w dS w S d S d S d S d S d S d H L M (2.11) where

∏

−

[

]

= × − = − ⊗ − − 4 1 2 1 ) 3 ( 1 1 ( 3 ) N i T N eye N i

T is a (N −3)×1 coefficient vector which transfers vector dS₁_{× N}₍ ₋₃₎ into the (N−3)th-order derivative value through Euclidean inner product [42]. Next, substituting (2.10) into (2.11) yields

1 1 2 1 ) 3 ( 1 ) 3 ( 1× × − − × = × × = _N _N _N _N _N _N h RG T T RG P H , (2.12) where 2 1 ) 3 ( 1 ) 3 ( 1 × − − × × = N N N N T T

P is a N×1 vector and referred as heterogeneity vector.

Expression (2.12) shows that the horizontal heterogeneity value H is the projection of the _h

row data of Bayer mosaic image onto the heterogeneity vector P . Thus expression (2.12) _N_×₁

is termed as horizontal heterogeneity-projection. Figure 2-2 illustrates an example of

horizontal heterogeneity-projection from a 1×5 row data of Bayer mosaic image. Using (2.12), the heterogeneity vector P is obtained as _N_×₁

[

]

T T T P 2 1 2 0 2 1 1 2 1 2 5 1 5× = × × = − − .

The horizontal heterogeneity value H of _h R is then given by ₃₃

53 43 23 13 1 5 5 1 ) 5 , 1 ( 2_S _RG _P _R ₂_G ₂_G _R d H h rg h = = × × = − + − .

Similarly, the vertical heterogeneity value H is the projection of Bayer mosaic image’s _v

(36)

Fig. 2-2: The concept of horizontal heterogeneity-projection from a 1× row data of a Bayer mosaic 5 image. 1 1 × × = T _N N v RG P H , (2.13) where T N N R G R

RG _×₁=[ ₁ ₂ ₃ _L] _×₁ is a column data of Bayer mosaic image. Finally, based on (2.12) and (2.13), the horizontal and vertical heterogeneity maps, H_h_{_}_map and H_v_{_}_map

are obtained, respectively by

T N map

h Bayer P

H _{_} = ⊗ _×₁ , and H_v_{_}_map = Bayer⊗P_N_×₁ , (2.14) where Bayer denotes the original Bayer mosaic image. One can see from (2.14) that the horizontal and vertical heterogeneity maps are derived directly from the Bayer mosaic image via horizontal and vertical heterogeneity-projection, respectively.

2.2.4 Directional Adaptive Filtering for Error Reduction

Assumption A4) states that the directional heterogeneity-projection along an edge direction leads to a small heterogeneity value. However, a small heterogeneity measure does not imply the directional heterogeneity-projection along a right edge direction. This problem will induce the estimation errors in the initial estimated heterogeneity maps. In order to reduce

(37)

the estimation errors, a directional adaptive filter, whose behavior changes based on the statistical characteristics inside a local window, is proposed to reduce the estimation error and estimate the optimal heterogeneity maps. Moreover, since each heterogeneity measure in the initial heterogeneity maps is static, this estimation problem is equivalent to the static estimation problem, in which the estimation errors are modeled as the zero mean Gaussian noises with non-zero variance. According to [43], the minimum mean square-error (MMSE) solution of the static estimation problem can be estimated using a predictor-corrector filter. Therefore, the design of the proposed directional adaptive filter adopts the structure of predictor-corrector filter to obtain the MMSE estimates.

The proposed directional adaptive filter is divided into horizontal and vertical adaptive filters. For the horizontal heterogeneity map, only the horizontal adaptive filter is applied to it. Figure 2-3(a) illustrates the concept of horizontal adaptive filter. In Fig. 2-3(a), the center pixel H is to be adaptively filtered along the horizontal direction based on statistical _h

measures of surrounding pixels R h

H and L h

H . The simplest statistical measures of R h

H and

L h

H are their mean and variance in a local window [41]. For instance, if a 1×3 rectangular window defines the window size, the local mean and variance of R

h H and L h H are, respectively, given by 3 ) ( RR h R h h R h H H H H = + + , [( )2 ( )2 ( RR)2] 3 h R h R h R h h R h R h H H H H H H H = − + − + − δ . (2.15) 3 ) ( LL h L h h L h H H H H = + + , [( )2 ( )2 ( LL)2] 3 h L h L h L h h L h L h H H H H H H H = − + − + − δ . (2.16)

Using (2.15) and (2.16), the adaptively filtered pixel *

h H is obtained below ) ( * L h R h R h L h L h L h h H H H H H H H − + + = δ δ δ . (2.17)

In (2.17), the local mean L h

H is the predictor term with an associated error variance L h

H

δ , and the local mean R

h

H is the corresponding corrector term with error variance R h

H

(38)

(a) (b)

Fig. 2-3: The concept of (a) horizontal and (b) vertical adaptive filtering using a 1× rectangular 3 window.

(2.17) provides the MMSE estimate of horizontal heterogeneity measure in a local window. Figure 2-3(b) illustrates an example of vertical adaptive filter for vertical heterogeneity map. Using the same procedure discussed above, the adaptively filtered pixel *

v H is obtained as follows ) ( * U v D v D v U v U v U v v H H H H H H H − + + = δ δ δ , (2.18) where ( , U) v U v H H δ and ( , D) v D v H

H δ are the local mean and variance of U v

H and D v

H . Similarly, (2.18) also provides the MMSE estimate of vertical heterogeneity measure in a local window. After adopting the horizontal and vertical adaptive filters presented above into horizontal and vertical heterogeneity maps, respectively, the MMSE estimates of horizontal and vertical heterogeneity maps *

_ map h H and * _ map v H are obtained.

2.2.5 Hard-Decision CFA interpolation

With the horizontal and vertical heterogeneity maps, a hard-decision rule is applied for CFA interpolation. First, we classify three subsets in the image such that

(39)

{

( , )| ( , ) * ( , )

}

_ * _ x y H x y H y x _h _map _v _map h ≡ <α Ω ,

{

( , )| ( , ) * ( , )

}

_ * _ x y H x y H y x _v _map _h _map v≡ <α Ω , (2.19)

{

h v

}

s ≡ x y x y ∉Ω x y ∉Ω Ω ( , )|( , ) ,( , ) ,

where Ω , _h Ω , and _v Ω denote the horizontal, vertical, and smooth subsets, respectively. _s α is a positive constant satisfying 0≤α ≤1. Second, based on (2.19), the concept of hard-decision rule for CFA interpolation is obtained

channel. color missing each on pixels g neighborin of Perform channel; color missing each on ion interpolat Perform ) , ( channel; color missing each on ion interpolat Perform ) , ( averaging weight else vertical y x elseif horizontal y x if v h Ω ∈ Ω ∈ (2.20)

In the following discussion, the CFA interpolation method is developed based on the hard-decision rule (2.20).

Remark 2.1: The parameter α in (2.19) determines the size of smooth subset in the image.

A small (large) α leads to a large (small) smooth subset in the image. For example, if

0 =

α , the image only contains smooth subset without horizontal and vertical subsets. Based on (2.20), the interpolation of image only adopts the weight averaging of neighboring pixels on each missing color channel [3, 11, 40]. On the other hand, for α =1, the image only contains horizontal and vertical subsets but without smooth subset and the interpolation of image only adopts horizontal and vertical interpolations on each missing color channel [7-9]. Therefore, for 0<α <1, the hard-decision rule (2.20) is characterized by features of weight averaging and directional interpolating.

(40)

2.3 Color-Difference Based Edge-Adaptive (CDEA) CFA interpolation

In this section, a novel CDEA CFA interpolation algorithm is proposed to reproduce the missing color values in Bayer mosaic images. The proposed CFA interpolation algorithm aims to exploit the green channel information for making high-frequency components of red and blue channels similar to the green channel, which is useful to effectively reduce color artifacts in demosaiced images. Any of the existing image interpolation methods can be combined with the proposed algorithm to reconstruct the green channel; but, a new edge-adaptive interpolation method to interpolate the green channel from CFA samples will also be presented in this section. The red and blue channels are first reconstructed using bilinear interpolation and then edge-adaptive low-pass filtered in color-difference space by the assistance of the reconstructed green channel.

2.3.1 Color-Difference Approach to CFA interpolation

In a Bayer pattern, green samples are obtained on a quincunx, while red and blue samples are obtained in rectangular lattices. The density of red and blue samples is one-half that of the green ones, and the aliasing error of high-frequency components in green channel is likely to be less than that in red and blue channels. Thus, a common problem in demosaicing is that the visible color artifacts in high-frequency regions are caused primarily by aliasing in the red and blue channels. Fortunately, there is usually high inter-channel correlation in high-frequency regions among red, green, and blue channels for natural color images [5]. This implies that the red, green, and blue channels are quite similar at fine texture and edge locations with all three colors. Therefore, a valid assumption can be made that object boundaries are the same in all three color channels. More specifically, we have the following assumption.

(41)

Fig. 2-4: Test images used in the experiment.

high-frequency regions of the green channel.

In order to validate assumption A5), we utilize twenty-five natural images from the Kodak PhotoCD (see Fig. 2-4), which have been used as test images for several demosaicing studies [3-10].

Figure 2-5 shows the flowchart for demonstrating the assumption of color-difference model mentioned above. The key concept is to replace the high-frequency components of red and blue planes by using those of green plane, and compare then the mean squared error (MSE) between the original and reconstructed color planes. A low-pass filter is utilized for red and blue planes and a high-pass filter for the green plane. We utilize 2-D ideal low-pass and high-pass filters in this procedure. Their transfer functions are given by [41]:

⎩ ⎨ ⎧ > ≤ = 0 0 ) , ( , 0 ) , ( , 1 ) , ( D v u D if D v u D if v u H_lowpass and ⎩ ⎨ ⎧ > ≤ = 0 0 ) , ( , 1 ) , ( , 0 ) , ( D v u D if D v u D if v u H_highpass ,

影像感測器之色彩濾波陣列補插與輪式機器人之視覺追蹤控制設計

國立交通大學

電機與控制工程學系

博士論文

影像感測器之色彩濾波陣列補插與

輪式機器人之視覺追蹤控制設計

Color Filter Array Interpolation for an Image

Sensor and Visual Tracking Control Design of

a Wheeled Mobile Robot

研 究 生：蔡奇謚

指導教授：宋開泰 博士

影像感測器之色彩濾波陣列補插與

輪式機器人之視覺追蹤控制設計

Color Filter Array Interpolation for an Image Sensor and

Visual Tracking Control Design of a Wheeled Mobile

Robot

研 究 生：蔡奇謚 Student: Chi-Yi Tsai

指導教授：宋開泰 博士 Advisor: Dr. Kai-Tai Song

影像感測器之色彩濾波陣列補插與

輪式機器人之視覺追蹤控制設計

學生:蔡奇謚 指導教授:宋開泰 博士

國立交通大學電機與控制工程學系

摘要

Color Filter Array Interpolation for an Image Sensor and

Visual Tracking Control Design of a Wheeled Mobile

Robot

Student: Chi-Yi Tsai

Advisor: Dr. Kai-Tai Song

Department of Electrical and Control Engineering

National Chiao Tung University

ABSTRACT

誌謝

Contents

List of Figures

List of Tables

Chapter 1

Introduction

1.1 Motivation

1.2 Literature Survey

1.2.1 CFA Interpolation for a Single Image Sensor

1.2.2 Visual Tracking Control for a Wheeled Mobile Robot

1.3 Research Objectives

1.4 Organization of the Thesis

Chapter 2

CFA Interpolation for Color Reproduction of Bayer

Mosaic Images

2.1 Introduction

2.2 Heterogeneity-Projection Hard-Decision (HPHD) CFA interpolation

2.2.1 Spectral and Spatial Correlations

2.2.2 Spectral-Spatial Correlation (SSC)

[

] [

]

2.2.3 Heterogeneity-Projection for Bayer Mosaic Images

[

]

∏

[

]

[

]

2.2.4 Directional Adaptive Filtering for Error Reduction

2.2.5 Hard-Decision CFA interpolation

{

}

{

}

{

}

2.3 Color-Difference Based Edge-Adaptive (CDEA) CFA interpolation

2.3.1 Color-Difference Approach to CFA interpolation

研究生：蔡奇謚

指導教授：宋開泰博士

研究生：蔡奇謚 Student: Chi-Yi Tsai

指導教授：宋開泰博士 Advisor: Dr. Kai-Tai Song

學生:蔡奇謚指導教授:宋開泰博士