Organization of the Thesis - 影像感測器之色彩濾波陣列補插與輪式機器人之視覺追蹤控制設計

Chapter 1 Introduction

1.4 Organization of the Thesis

The remainder of this thesis is organized as follows: In Chapter 2, a novel HPHD CFA interpolation algorithm is first developed based on the concept of spectral-spatial correlation.

A new CDEA CFA interpolation algorithm is then proposed to combine with HPHD CFA interpolation algorithm for color reproduction of Bayer mosaic images. In Chapter 3, a novel dual-Jacobian visual interaction model is derived in order to help the design of mobile robot visual tracking control for tracking a dynamic moving target. Accordingly, the results of VTC design with robustness analysis are developed by using the proposed visual interaction model.

Furthermore, the robust control law to overcome the velocity quantization error encountered in practical systems will also be presented. Chapter 4 develops the VSE using Kalman filter with self-tuning algorithm to estimate the optimal system state in the image plane for handling the uncertainties caused by image noise and temporary occlusion. Simulation and experimental results of the proposed CFA interpolation algorithm and the proposed mobile robot visual tracking control system are reported and discussed in Chapter 5. Chapter 6 concludes the contributions of this work and provides the recommendations for future research. In Appendix A, an experiment of tweaking parameters is presented to find the local optimal parameters for the proposed HPHD-CDEA CFA interpolation algorithm described in Chapter 2. Appendixes B and C present more discussions and visual comparisons of the proposed CFA interpolation algorithm. An extended discussion on the proposed VTC scheme is presented in Appendix D. Figure 1-1 shows the simplified overview of the thesis.

Fig. 1-1: Structure of the thesis.

Chapter 2 CFA Interpolation for Color Reproduction of Bayer Mosaic Images

2.1 Introduction

In this chapter, the design of the proposed HPHD-CDEA CFA interpolation algorithm is presented for color reproduction from Bayer mosaic images. The proposed algorithm consists of HPHD algorithm and CDEA CFA interpolation. The proposed HPHD algorithm aims to estimate the optimal interpolation direction before performing CFA interpolation. Because the decision stage is performed before the interpolation stage (termed as hard-decision interpolation), each pixel only needs to be interpolated once. To do so, a new heterogeneity-projection scheme based on a novel spectral-spatial correlation concept is proposed to estimate the best interpolation direction directly from the original Bayer mosaic image. Using the proposed heterogeneity-projection scheme, a hard-decision rule can be decided before performing CFA interpolation.

In order to effectively reduce color artifacts in CFA interpolation process, the proposed CDEA CFA interpolation algorithm is then combined with the HPHD algorithm to reconstruct the red and blue color planes by exploiting the green plane information for making high-frequency components of red and blue planes similar to the green plane, in which the aliasing errors are usually much smaller than those in red and blue planes. To do so, the red and blue channels are first reconstructed using bilinear interpolation and then edge-adaptive filtered in color-difference space. To reconstruct the green plane, any existent image interpolation methods can be employed to combine with the CDEA CFA interpolation

algorithm. Moreover, we also present a new edge-adaptive interpolation method to reconstruct the green channel from CFA samples. The advantages of the proposed HPHD-CDEA CFA interpolation algorithm are summarized as follows:

1) Many existent CFA interpolation methods can be combined with the proposed HPHD algorithm to reconstruct each color plane. For example, the proposed heterogeneity-projection scheme can combine with existent decision-based CFA interpolation algorithms. More specifically, the proposed heterogeneity-projection scheme can adopt into the decision step of existent decision-based CFA interpolation algorithms.

2) Each pixel only has to be interpolated once. Therefore, the proposed algorithm is much more efficient than other decision-based schemes.

3) Any existing image interpolation methods can be combined with the proposed CDEA CFA interpolation algorithm to reconstruct the green plane.

4) The proposed HPHD-CDEA CFA interpolation algorithm performs well not only in texture regions, but also in well-defined edges of the image.

The rest of this chapter is organized as follows: In Section 2.2, the proposed HPHD CFA interpolation algorithm is designed by using a novel concept of spectral-spatial correlation.

Section 2.3 presents the proposed CDEA CFA interpolation algorithm based on color-difference model. A new edge-adaptive interpolation method to reconstruct the green plane from CFA samples is also presented in this section. Section 2.4 describes the complete HPHD-CDEA CFA interpolation algorithm and gives an example study. Section 2.5 summarizes the contributions of this work. An experiment of tweaking parameters to find the local optimal parameters for the proposed CFA interpolation algorithm is presented in Appendix A. More discussions and visual comparisons are presented in Appendixes B and C.

2.2 Heterogeneity-Projection Hard-Decision (HPHD) CFA interpolation

Figure 2-1 shows the most used CFA pattern, the Bayer pattern [38], where R, G and B denote, respectively, the pixels having only red, green and blue color values. We limit our discussion to the Bayer pattern in this chapter because it is so popular. In the following, image spectral and spatial correlations are first introduced. A novel spectral-spatial correlation is then derived based on these two correlations.

2.2.1 Spectral and Spatial Correlations

Many existing demosaicing methods are developed using image spectral and/or spatial correlation. The concept of spectral correlation is based on the assumption that the color difference signals are locally constant in chrominance smooth areas [39]. Let [

R G B

] denote three color planes of a nature color image, the concept of spectral correlation leads to the following assumption.

A1) The color differences between green and red/blue channels satisfy the following

conditions

) , ( ) , ( ) ,

(x y G x y A x y

R = + _rg and B(x,y)=G(x,y)+A_bg(x,y),

where A_rg( yx, ) and A_bg( yx, ) are piecewise constant within the boundary of a given object.

The spatial correlation reflects the fact that within a homogeneous image region, neighboring pixels share similar color values [40]. In other words, the difference between neighboring pixel values along an edge direction in spatial domain is a constant. Thus we have the following assumption based on the concept of spatial correlation [3].

Fig. 2-1: The Bayer pattern.

A2) The rate of change of neighboring pixel values along an edge direction is a constant.

To illustrate this, let us consider the interpolation of

R in Fig. 2-1. Suppose that the pixel

₃₃

R is located on a horizontal edge. Based on A1), the neighboring pixels of

R along the

₃₃ horizontal direction have the following relationship between green and red/blue pixel values

) , 1 ( ) , ( ) , 1

(x y A x y A x y

A_rg − = _rg = _rg + and

) , 1 ( ) , ( ) , 1

(x y A x y A x y

A_bg − = _bg = _bg + . (2.1) So we have

43 43 33 33 23

23 G R G R G

R − = − = − and B₂₃−G₂₃ =B₃₃−G₃₃ =B₄₃ −G₄₃. (2.2) The assumption A2) gives the following relationship on horizontal edges

dRh

R R R

R₂₃− ₃₃ = ₃₃ − ₄₃ ≡ ,

dGh

G G G

G₂₃− ₃₃ = ₃₃− ₄₃ ≡ , and (2.3)

dBh

B B B

B₂₃− ₃₃ = ₃₃ − ₄₃ ≡ ,

where G₃₃, R₂₃, R₄₃, B₂₃, B₃₃, and B₄₃ denote the missing color values at the respective pixel locations.

dG ,

dR and

dB are constants.

2.2.2 Spectral-Spatial Correlation (SSC)

A significant characteristic of Bayer pattern is that for each pixel, the surrounding pixels are one of the primary components in different channels. It is then interesting to investigate the relationship between neighboring pixels in different color channels. Consider the following situation: on a horizontal edge, two green pixels surround a red pixel on horizontal direction. Take the difference between the center red pixel and right green pixel, we then have

[

⁽ ^, ⁾ ⁽ ^, ⁾

] [

⁽ ^, ⁾ ⁽ ¹^, ⁾

]

) , 1 ( ) ,

(

x y G x y R x y G x y G x y G x y

R

− + = − + − + , (2.4)

where

G

( y

x

, ) denotes the missing green value at center red pixel location. Recall assumptions A1) and A2), expression (2.4) becomes such that

h rg

x x h

R x y G x y A x y dG

S

⁽ ^, ⁺¹⁾ ≡ ( , )− ( +1, )= ( , )+ . (2.5) Similarly, the difference between a blue pixel and its right green pixel is given by

h bg

x x h

B x y G x y A x y dG

S

⁽ ^, ⁺¹⁾ ≡ ( , )− ( +1, )= ( , )+ . (2.6) The same results also can be obtained along vertical direction on a vertical edge such that

v rg

y y v

R x y G x y A x y dG

S

⁽ ^, ⁺¹⁾ ≡ ( , )− ( , +1)= ( , )+ , and

v bg

y y v

B x y G x y A x y dG

S

⁽ ^, ⁺¹⁾ ≡ ( , )− ( , +1)= ( , )+ . (2.7) Expressions (2.5)-(2.7) show that the difference between surrounding pixels in different color channels is equal to the summation of spectral and spatial correlations. We refer these relationships (2.5)-(2.7) as spectral-spatial correlation (SSC). SSC has two important characteristics. First, SSC can be easily and directly calculated from the original Bayer mosaic image. Second, SSC inherits the characteristics of spectral and spatial correlations. In other words, SSC is also piecewise constant within the boundary of a given object or along an edge direction. Therefore, we have the following assumption based on these observations:

A3) The SSC defined in (2.5)-(2.7) within the boundary of a given object or along an edge

direction is also piecewise constant.

Assumption A3) is a significant clue for us to find the directional smooth regions in Bayer mosaic images directly before performing the interpolation. In the following section, we will present the method of heterogeneity-projection based on A3).

2.2.3 Heterogeneity-Projection for Bayer Mosaic Images

The proposed heterogeneity-projection scheme transfers the original Bayer mosaic image directly into horizontal and vertical heterogeneity maps, respectively. Using these two heterogeneity maps, the interpolation direction can be determined easily by choosing the smallest heterogeneity values.

Assumption A3) implies that the n-order directional finite derivative of SSC along an edge direction tends toward a small value. For example, consider a red pixel R( yx, ) locates on a horizontal edge, the SSC values of R( yx, ) and its neighboring pixels along horizontal direction can be found such that

h derivative of a one-dimensional discrete function, the first-order horizontal derivative of SSC are given by [41]

along this horizontal edge. Because the higher-order derivative of a discrete function is a linear combination of the first-order ones, it implies the higher-order horizontal derivative of SSC will also approach to zero along the horizontal edge. Thus we have the following assumption.

A4) If pixels locate on a directional edge, then the corresponding n

^th-order directional derivative of SSC along the edge direction approaches to zero.

Assumption A4) poses a question that how the n^th-order directional derivative of SSC can be directly calculated from Bayer mosaic image. To resolve this problem, a heterogeneity-projection scheme is developed to transfer the row data of Bayer mosaic image directly into n^th-order directional derivative of SSC. Note that the value of n^th-order directional derivative of SSC is defined as heterogeneity measure, because it leads to a small value within a directional smooth region.

Denote

RG

₁_×_N =[

R

₁

G

₂

R

₃ L]₁_×_N as a row data of Bayer mosaic image, N is the presetting window size, and

H is the corresponding horizontal heterogeneity value. To

_h calculate the horizontal heterogeneity value

H from

RG

₁_×_N, we propose the following steps. First, the row data

RG

₁_×_N is transferred into a 1× N( −3) vector of first-order horizontal derivative of SSC using a linear transformation such that

1 ) 3 ( 1 ) 3 ( 1 ) 6 , 3 ( ) 5 , 2 ( ) 4 , 1 ( 3

(

1_×_N₋ =[

dS

_rg^h

dS

_gr^h

dS

_rg^h ]_×_N₋ =

RG

_×_N

T

_N_× _N₋

dS

L , (2.10)

where

T

_N¹_×₍_N₋₃₎ =

[

1 −1 −1 1

]

^T ⊗

eye

(

N

−3), ⊗ denotes the 2D convolution operator and eye(M)denotes a M×M identity matrix. Second, because the high-order derivative of a discrete function is derived by the linear combination of its first-order ones, the horizontal

heterogeneity value

H , the

_h (N−3)^th-order horizontal derivative of SSC, is obtained such product [42]. Next, substituting (2.10) into (2.11) yields

Expression (2.12) shows that the horizontal heterogeneity value

H is the projection of the

_h row data of Bayer mosaic image onto the heterogeneity vector

P . Thus expression (2.12)

_N_×₁ is termed as horizontal heterogeneity-projection. Figure 2-2 illustrates an example of horizontal heterogeneity-projection from a 1×5 row data of Bayer mosaic image. Using (2.12), the heterogeneity vector

P is obtained as

_N_×₁

Similarly, the vertical heterogeneity value

H is the projection of Bayer mosaic image’s

_v column data onto the heterogeneity vector

P such that

_N_×₁

Fig. 2-2: The concept of horizontal heterogeneity-projection from a 1× row data of a Bayer mosaic 5 image.

1 ×

= ^T_N× _N

v RG P

H , (2.13)

where RG_N_×₁=[R₁ G₂ R₃ L]^T_N_×₁ is a column data of Bayer mosaic image. Finally, based

on (2.12) and (2.13), the horizontal and vertical heterogeneity maps, H_h_{_}_map and H_v_{_}_map are obtained, respectively by

T N map

h Bayer P

H _{_} = ⊗ _×₁ , and

H

_v_{_}_map =

Bayer

⊗

P

_N_×₁ , (2.14)

where Bayer denotes the original Bayer mosaic image. One can see from (2.14) that the horizontal and vertical heterogeneity maps are derived directly from the Bayer mosaic image via horizontal and vertical heterogeneity-projection, respectively.

2.2.4 Directional Adaptive Filtering for Error Reduction

Assumption A4) states that the directional heterogeneity-projection along an edge direction leads to a small heterogeneity value. However, a small heterogeneity measure does not imply the directional heterogeneity-projection along a right edge direction. This problem will induce the estimation errors in the initial estimated heterogeneity maps. In order to reduce

the estimation errors, a directional adaptive filter, whose behavior changes based on the statistical characteristics inside a local window, is proposed to reduce the estimation error and estimate the optimal heterogeneity maps. Moreover, since each heterogeneity measure in the initial heterogeneity maps is static, this estimation problem is equivalent to the static estimation problem, in which the estimation errors are modeled as the zero mean Gaussian noises with non-zero variance. According to [43], the minimum mean square-error (MMSE) solution of the static estimation problem can be estimated using a predictor-corrector filter.

Therefore, the design of the proposed directional adaptive filter adopts the structure of predictor-corrector filter to obtain the MMSE estimates.

The proposed directional adaptive filter is divided into horizontal and vertical adaptive filters. For the horizontal heterogeneity map, only the horizontal adaptive filter is applied to it.

Figure 2-3(a) illustrates the concept of horizontal adaptive filter. In Fig. 2-3(a), the center pixel

H is to be adaptively filtered along the horizontal direction based on statistical

_h measures of surrounding pixels H_h^R and H_h^L. The simplest statistical measures of H_h^R and

Hh are their mean and variance in a local window [41]. For instance, if a 1×3 rectangular window defines the window size, the local mean and variance of H_h^R and H_h^L are, respectively, given by

3 ) ( _h _h^R _h^RR

h H H H

H = + + , δH_h^R =[(H_h^R −H_h)² +(H_h^R −H_h^R)²+(H_h^R −H_h^RR)²] 3. (2.15) 3

) ( _h _h^L _h^LL

h H H H

H = + + , δH_h^L =[(H_h^L−H_h)² +(H_h^L−H_h^L)² +(H_h^L−H_h^LL)²] 3. (2.16) Using (2.15) and (2.16), the adaptively filtered pixel H_h^* is obtained below

)

* ( L

h R R h h L h

L L h

H H

H H H H

H

−

+ +

= δ δ

δ . (2.17)

In (2.17), the local mean H_h^L is the predictor term with an associated error variance δH_h^L, and the local mean H_h^R is the corresponding corrector term with error variance δH_h^R. Thus,

(a) (b)

Fig. 2-3: The concept of (a) horizontal and (b) vertical adaptive filtering using a 1× rectangular 3 window.

(2.17) provides the MMSE estimate of horizontal heterogeneity measure in a local window.

Figure 2-3(b) illustrates an example of vertical adaptive filter for vertical heterogeneity map.

Using the same procedure discussed above, the adaptively filtered pixel H_v^* is obtained as follows

)

* ( U

v D D v v U v

U U v

H H

H

−

+ +

= δ δ

δ , (2.18)

where (H_v^U,δH^U_v ) and (H_v^D,δH_v^D) are the local mean and variance of H_v^U and H_v^D. Similarly, (2.18) also provides the MMSE estimate of vertical heterogeneity measure in a local window. After adopting the horizontal and vertical adaptive filters presented above into horizontal and vertical heterogeneity maps, respectively, the MMSE estimates of horizontal and vertical heterogeneity maps

H

^*_h_{_ map} and

H

_v^*_{_ map} are obtained.

2.2.5 Hard-Decision CFA interpolation

With the horizontal and vertical heterogeneity maps, a hard-decision rule is applied for CFA interpolation. First, we classify three subsets in the image such that

{

⁽ ^, ⁾^| * ⁽ ^, ⁾ ^*_ ⁽ ^, ⁾

}

hard-decision rule for CFA interpolation is obtained

channel.

In the following discussion, the CFA interpolation method is developed based on the hard-decision rule (2.20).

Remark 2.1: The parameter α in (2.19) determines the size of smooth subset in the image.

A small (large) α leads to a large (small) smooth subset in the image. For example, if

α , the image only contains smooth subset without horizontal and vertical subsets.

Based on (2.20), the interpolation of image only adopts the weight averaging of neighboring pixels on each missing color channel [3, 11, 40]. On the other hand, for α =1, the image only contains horizontal and vertical subsets but without smooth subset and the interpolation of image only adopts horizontal and vertical interpolations on each missing color channel [7-9]. Therefore, for 0<α <1, the hard-decision rule (2.20) is characterized by features of weight averaging and directional interpolating.

2.3 Color-Difference Based Edge-Adaptive (CDEA) CFA interpolation

In this section, a novel CDEA CFA interpolation algorithm is proposed to reproduce the missing color values in Bayer mosaic images. The proposed CFA interpolation algorithm aims to exploit the green channel information for making high-frequency components of red and blue channels similar to the green channel, which is useful to effectively reduce color artifacts in demosaiced images. Any of the existing image interpolation methods can be combined with the proposed algorithm to reconstruct the green channel; but, a new edge-adaptive interpolation method to interpolate the green channel from CFA samples will also be presented in this section. The red and blue channels are first reconstructed using bilinear interpolation and then edge-adaptive low-pass filtered in color-difference space by the assistance of the reconstructed green channel.

2.3.1 Color-Difference Approach to CFA interpolation

In a Bayer pattern, green samples are obtained on a quincunx, while red and blue samples are obtained in rectangular lattices. The density of red and blue samples is one-half that of the green ones, and the aliasing error of high-frequency components in green channel is likely to be less than that in red and blue channels. Thus, a common problem in demosaicing is that the visible color artifacts in high-frequency regions are caused primarily by aliasing in the red and blue channels. Fortunately, there is usually high inter-channel correlation in high-frequency regions among red, green, and blue channels for natural color images [5]. This implies that the red, green, and blue channels are quite similar at fine texture and edge locations with all three colors. Therefore, a valid assumption can be made that object boundaries are the same in all three color channels. More specifically, we have the following assumption.

A5) The high-frequency regions are similar in all three channels and close to the

Fig. 2-4: Test images used in the experiment.

high-frequency regions of the green channel.

In order to validate assumption A5), we utilize twenty-five natural images from the Kodak PhotoCD (see Fig. 2-4), which have been used as test images for several demosaicing studies [3-10].

Figure 2-5 shows the flowchart for demonstrating the assumption of color-difference model mentioned above. The key concept is to replace the high-frequency components of red and blue planes by using those of green plane, and compare then the mean squared error (MSE) between the original and reconstructed color planes. A low-pass filter is utilized for red and blue planes and a high-pass filter for the green plane. We utilize 2-D ideal low-pass and high-pass filters in this procedure. Their transfer functions are given by [41]:

⎩⎨

Fig. 2-5: Flowchart for demonstrating the assumption of color-difference model.

where

D is a specified nonnegative quantity; and

₀ D( vu, )is the distance from point ( vu, ) to the origin of the frequency plane. We set

D equal to 128 in this test. After filtering in

₀ each color plane, the new red and blue planes, R and B , are reconstructed respectively by adding the high-frequency components of the green plane to their low-frequency components.

Table 2-1 records the MSE comparison results of each step. The first and second columns show the MSE between original and low-pass filtered red (blue) planes

R (

_low

B ). The third

_low and fourth columns show the MSE between original and reconstructed red (blue) planes

R ( B ). From the test results, it is clear that the MSE is reduced effectively by adding the

high-frequency regions of the green plane Ghigh to the low-pass filtered red (blue) planes. This implies that the high-frequency regions of red and blue planes are similar and close to the high-frequency regions of the green plane. Thus, assumption A5) is validated. Based on assumption A5), our motivation in this study is to reduce the color artifacts in high-frequency regions by adding the high-frequency information of green channel to other color channels.

As described below, this can be achieved by utilizing the color-difference model.

Let ][

R

G

B

^d denote three color planes of a demosaiced image. The Fourier spectrum of each color plane can be described as follows:

Table 2-1: Comparison of mean squared error at each step in Fig. 2-5.

Image No. MSE(

R, R

_low) MSE(

B, B

_low) MSE(

R, R

) MSE(

B, B

) 1 147.3784 123.2171 37.0369 8.7712 2 260.2865 263.2530 4.2306 2.9049 3 73.0098 66.0994 9.8449 1.6476 4 59.1224 55.4568 3.1630 4.0863 5 75.0895 71.0032 8.0543 1.6137 6 307.8661 293.3846 6.3342 9.2040 7 178.8668 174.3140 2.1157 4.2843 8 76.3988 76.0140 2.8284 3.3626 9 539.2621 526.6953 8.2720 7.8362 10 81.2787 77.1119 2.6365 3.9927 11 86.7402 89.3761 2.8525 4.2737 12 149.5650 139.4246 4.1582 2.2200 13 70.4837 74.3636 3.1482 2.9343 14 415.0555 421.2574 3.3856 10.3724 15 168.1822 141.2964 9.1458 8.2062 16 112.1141 103.0222 12.2228 3.6253 17 82.9777 80.8168 1.3618 1.8066 18 81.5764 84.8154 2.2567 3.9789 19 187.1993 182.8144 6.1265 10.0651 20 179.4598 163.5454 2.7882 3.8415 21 119.7853 109.3889 2.4227 6.2912 22 155.7798 160.6295 2.6352 4.9741 23 124.3185 129.3350 7.7838 7.3834 24 57.6438 55.0500 4.2972 4.7799 25 232.1675 302.3846 10.3759 19.5268

h d l

d F R F R

F[ ]= [ ] + [ ] , F[G^d]=F[G^d]_l +F[G^d]_h, F[B^d]= F[B^d]_l +F[B^d]_h, (2.21) where F[•] denotes the 2-D discrete Fourier transform; and the underscores l and h stand for low-frequency and high-frequency components, respectively. The color-difference models of the demosaiced image are defined such that

d d

R G

R

= − ,

B

_g =

B

^d −

G

^d. (2.22) Let L{•} denote a linear low-pass filtering process, and

R

~_g

and

B

~_g

denote the low-frequency regions of the color differences corresponding to R_g and B_g. Suppose that the high-frequency components of the color differences R_g and B_g can be removed by the low-pass filtering process, the Fourier spectrum of

R

~_g

and

B

~_g

can be described such that

l d l

d g

L F R F R F G

R

F

~ ] { [ ]} [ ] [ ]

[ = = − ,

F B

~_g]

L

{

F

[

B

_g]}

F

[

B

^d]_l

F

[

G

^d]_l

[ = = − . (2.23)

Subsequently, the new red and blue planes of the demosaiced image,

R and

B , can be

^d obtained respectively, by adding

R

~_g

and

B

~_g

with G^d. Their Fourier spectra are given by

h d l

d d

F R G F R F G

R

F

~ ] [ ] [ ]

[ ]

[ = + = + ,

F B

~_g

G

^d]

F

[

B

^d]_l

F

[

G

^d]_h [

]

[ = + = + . (2.24)

It is clear from (2.24) that the high-frequency components of the new red and blue planes of the demosaiced image are replaced by the high-frequency components of the green plane.

Because the aliasing error in the green plane is usually much smaller than those in red and blue planes, based on the assumption described above, the aliased errors in red and blue channels can be efficiently reduced by linear low-pass filtering in the color-difference spaces and adding the results with green channel to obtain the new ones. This observation leads to the development of an efficient CFA interpolation algorithm based on color-difference that

在文檔中影像感測器之色彩濾波陣列補插與輪式機器人之視覺追蹤控制設計 (頁 26-0)