Demosaicing: Heterogeneity-Projection Hard-Decision Adaptive
Interpolation Using Spectral-Spatial Correlation
Chi-Yi Tsai and Kai-Tai Song
Department of Electrical and Control Engineering, National Chiao Tung University
1001 Ta Hsueh Road, Hsinchu 300, Taiwan
E-mail: [email protected]; [email protected]
ABSTRACT
A novel heterogeneity-projection hard-decision adaptive interpolation (HPHD-AI) algorithm is proposed in this paper for color reproduction from Bayer mosaic images. The proposed algorithm aims to estimate the optimal interpolation direction and perform hard-decision interpolation, in which the decision is made before interpolation. To do so, a new heterogeneity-projection scheme based on spectral-spatial correlation is proposed to decide the best interpolation direction from the original mosaic image directly. Exploiting the proposed heterogeneity-projection scheme, a hard-decision rule can be designed easily to perform the interpolation. We have compared this technique with three recently proposed demosaicing techniques: Lu’s, Gunturk’s and Li’s methods, by utilizing twenty-five natural images from Kodak PhotoCD. The experimental results show that HPHD-AI outperforms all of them in both PSNR values and S-CIELab *
ab E
∆ measures.
Keywords: Color reproduction, CFA demosaicing, color artifacts, adaptive filtering, digital cameras.
1. INTRODUCTION
Digital color images from single-chip digital still cameras are obtained by interpolating the output from a color filter array (CFA), in which each sensor pixel only samples one of three primary color components. These sparsely sampled color values are termed mosaic images. A full-color image is reproduced from a mosaic image by estimating two missing color values for each pixel. This image reconstruction process is commonly known as CFA interpolation or CFA demosaicing. The simplest CFA demosaicing methods apply well-known interpolation techniques to each color channel separately such as bilinear interpolation and cubic spline interpolation. However, these single-channel algorithms usually introduce severe color artifacts and blurs around sharp edges [1]. These drawbacks motivate the need of more specialized algorithms for advanced demosaicing performance. An excellent literature survey on advanced demosaicing algorithms can be found in [2].
In recent years, there have been researches on more sophisticated demosaicing algorithms. In [3], Lu and Tan presented an improved hybrid CFA demosaicing method that consists of interpolation and post-processing steps to render full-color images and suppress visible demosaicing artifacts. In [4], the authors utilized a projection-onto-convex-set (POCS) technique to estimate the missing color values in red and blue channels using alternating projection scheme based on high inter-channel correlation. In [5], Li proposed a successive approximation demosaicing strategy by adopting color difference interpolation iteratively. Another recent demosaicing approach, termed as decision-based demosaicing algorithm, divides the demosaicing procedure into interpolation step and decision step [6, 7]. In the interpolation step, they produce respectively horizontally interpolated and vertically interpolated images. In decision step, a soft decision method was adopted for choosing the pixels interpolated in the direction with fewer artifacts. For the decision step, Hirakawa et al proposed the color image homogeneity metric to measure the level of misguidance color artifacts presented in these two images [6]. Based on this measurement, the interpolation decision is made by choosing the region with larger homogeneity map values. In [7], Wu et al adopted the Fisher’s linear discriminant technique to determine the optimal interpolation direction under two hypotheses, one for horizontal structure and the other for vertical structure, in a local window. The decision-based demosaicing algorithm performs well not only in textured regions, but also in well-defined edges of the image. However, the main drawback of decision-based demosaicing algorithms is that they are not efficient in the interpolation step because each pixel has to interpolate twice,
one in horizontal direction and the other in vertical direction, before applying the soft decision method. Therefore, to develop an efficient color interpolation algorithm with high performance in both textured and edge regions is still a challenge in CFA demosaicing research.
In this paper, a novel heterogeneity-projection hard-decision adaptive interpolation (HPHD-AI) method is proposed for color reproduction from Bayer mosaic images. The proposed algorithm aims to decide the optimal interpolation direction before performing interpolation. To do so, a new heterogeneity-projection scheme based on spectral-spatial correlation is proposed to estimate the best interpolation direction from the original Bayer mosaic images directly. Based on the proposed heterogeneity-projection scheme, a hard-decision rule can be designed easily to perform the interpolation. The advantage of the proposed demosaicing algorithm is threefold. First, the proposed heterogeneity-projection scheme can be combined with existent decision-based demosaicing algorithms. Second, the decision is made before interpolation and thus each pixel only has to interpolate once in the interpolation step. Finally, the proposed demosaicing algorithm also performs well not only in textured regions, but also in well-defined edges of the image.
2. SPECTRAL-SPATIAL CORRELATION
Fig. 1 shows the most used CFA pattern, the Bayer pattern [8], where R, G and B denote, respectively, the pixels having only red, green and blue color values. We limit our discussion in this paper to the Bayer pattern because it is popular. In the following, we will introduce a novel spectral-spatial correlation based on two popular image correlations: spectral and spatial correlations.
Many existent demosaicing methods are developed using image spectral or spatial correlation, or both. The concept of spectral correlation is based on the assumption that the color difference signals are locally constant in chrominance smooth areas [9]. The spatial correlation refers to the fact that within a homogeneous image region, neighboring pixels share similar color values [3, 10]. In other words, the difference between neighboring pixel values along an edge direction in spatial domain is a constant. Spectral and spatial correlations of a natural image describe the relationship between different color channels. However, in Bayer mosaic image, it is difficult to calculate the spectral and spatial correlations directly because each pixel only contains one primary component. This problem motivates us to find a more efficient criterion instead of spectral and spatial correlations for Bayer mosaic images.
A significant characteristic of Bayer pattern is that for each pixel, the surrounding pixels are one of the primary components in different channels. This causes us to investigate the relationship between neighboring pixels in different color channels. Consider the following situation: on a horizontal edge, two green pixels surround a red pixel on horizontal direction. Take the difference between the center red pixel and right green pixel, we then have
R(x,y)−G(x+1,y)=
[
R(x,y)−G(x,y)] [
+ G(x,y)−G(x+1,y)]
, (1) where G( yx, ) denotes the missing green value at center red pixel location. By the assumption of spectral and spatial correlations, expression (1) becomes such thath rg x x h rg R x y G x y A x y dG S ( ,+1) ≡ ( , )− ( +1, )= ( , )+ . (2)
Similarly, the difference between a blue pixel and its right green pixel is given by
h bg x x h bg B x y G x y A x y dG S ( ,+1) ≡ ( , )− ( +1, )= ( , )+ . (3)
The same results also can be obtained along vertical direction on a vertical edge such that
v rg y y v rg R x y G x y A x y dG S ( ,+1)≡ ( , )− ( , +1)= ( , )+ , and v bg y y v bg B x y G x y A x y dG S ( ,+1)≡ ( , )− ( , +1)= ( , )+ . (4)
where Arg( yx, ) and Abg( yx, ) are piecewise constant functions; dG and h dG are constants. Expressions (1)-(4) v
tell us that the difference between surrounding pixels in different color channels is equal to the summation of spectral and spatial correlations. We refer these relationships (1)-(4) as spectral-spatial correlations (SSC). SSC has two important characteristics. First, SSC can be easily and directly calculated from Bayer mosaic images. Second, SSC inherits the characteristics of spectral and spatial correlations. In other words, SSC is also piecewise constant within the boundary of a given object or along an edge direction. SSC acts as a significant clue for us to find the directional smooth regions in Bayer mosaic images directly before performing the interpolation. In the following section, we will present the proposed heterogeneity-projection based on these observations.
Fig. 1: Bayer color filter array pattern (Bayer pattern)
3. HETEROGENEITY-PROJECTION FOR BAYER MOSAIC IMAGES
The aim of this section is to derive the heterogeneity-projection formulation based on SSC. The proposed heterogeneity-projection scheme can directly transfer the original Bayer mosaic image into horizontal and vertical heterogeneity maps. Using these two heterogeneity maps, the decision of interpolation direction can be determined easily by choosing the smallest heterogeneity values.
A. Heterogeneity-Projection
Because SSC is piecewise constant along an edge direction, the nth-order directional finite derivative of SSC along the edge direction tends toward a small value. For example, let’s consider the interpolation of R33 in Fig. 1. Suppose
that the pixel R33 is located on a horizontal edge. The SSC values of pixel R33 and its neighboring pixels along
horizontal direction can be found such that
h rg h rg A dG S (1,2) = (1,3)+ , h rg h rg A dG S (3,4) = (3,3)+ , rg h h gr A dG S (2,3) =− (2,3)+ , h rg h gr A dG S (4,5)=− (4,3)+ , (5) where Sh(x,x1) G(x,y) R(x 1,y) gr ≡ − +
+ . Define the first-order horizontal finite derivative of SSC such that
) 3 , 3 ( ) 3 , 1 ( ) 4 , 3 ( ) 2 , 1 ( ) 4 , 1 ( rg rg h rg h rg h rg S S A A dS ≡ − = − , and (2,5) (2,3) (4,5) (4,3) (2,3) rg rg h gr h gr h gr S S A A dS ≡ − = − . (6)
Because Arg(x,y) is piecewise constant function,
) 4 , 1 ( h rg dS and h(2,5) rg
dS both will approach to zero along this horizontal edge. Consequently, the second-order horizontal finite derivative of SSC
) 3 , 4 ( ) 3 , 3 ( ) 3 , 2 ( ) 3 , 1 ( ) 5 , 2 ( ) 4 , 1 ( ) 5 , 1 ( 2 rg rg rg rg h gr h rg h rg dS dS A A A A S d ≡ − = + − −
will also tend toward zero along the horizontal edge. This observation poses a question that how the nth-order directional finite derivative of SSC can be directly calculated from a Bayer mosaic image. To resolve this problem, a heterogeneity-projection scheme has been developed to transfer row data of a Bayer mosaic image into nth-order directional finite derivative of SSC directly. Note that we refer the value of nth-order directional finite derivative of SSC as heterogeneity value because it leads to a small value within a directional smooth region.
Denote RG1×N =[R1 G2 R3 L]1×N as row data of a Bayer mosaic image, N is the presetting window size, and h
H is the corresponding horizontal heterogeneity value. To calculate the horizontal heterogeneity value Hh from N
RG1× , we have the following steps. First, the row data RG1×N is transferred into a 1× N( −3) vector of first-order
horizontal finite derivative of SSC using a linear transformation such that
1 ) 3 ( 1 ) 3 ( 1 ) 6 , 3 ( ) 5 , 2 ( ) 4 , 1 ( ] [ ×N− = ×N N×N− h rg h gr h rg dS dS RG T dS L , (7) where 1
[
1 1 1 1]
( 3) ) 3 ( − = − − ⊗ − × eye N T T NN , ⊗ denotes the 2D convolution operator and eye(M) denotes a
M
M× identity matrix. Second, the horizontal heterogeneity value Hh can be calculated using Euclidean inner
product [11] 2 1 ) 3 ( ) 3 ( 1 ) 6 , 3 ( ) 5 , 2 ( ) 4 , 1 ( ] [ × − − × = N N h rg h gr h rg h dS dS dS T H L , (8) where
∏
−[
]
= × − = − ⊗ − − 4 1 2 1 ) 3 ( 1 1 ( 3 ) N i T N eye N i1 1 2 1 ) 3 ( 1 ) 3 ( 1× × − − × = × × = N N N N N N h RG T T RG P H , (9) where 2 1 ) 3 ( 1 ) 3 ( 1 × − − × × = N N N N T T
P is a N×1 vector and referred as heterogeneity vector. Expression (9) shows that the horizontal heterogeneity value Hh is the projection of the row data of Bayer mosaic image onto the heterogeneity
vector PN×1. Thus expression (9) is termed as horizontal heterogeneity-projection of Bayer mosaic image’s row data.
Similarly, the vertical heterogeneity value Hv is the projection of Bayer mosaic image’s column data onto the
heterogeneity vector PN×1 such that
1 1 × × = N T N v RG P H , (10) where T N N R G R
RG×1=[ 1 2 3 L] ×1 is a column data of Bayer mosaic image. Finally, based on (9) and (10), the
horizontal and vertical heterogeneity maps, Hh_map and Hv_map can be obtained, respectively by T
N map
h Bayer P
H _ = ⊗ ×1 , and Hv_map= Bayer⊗PN×1 , (11)
where Bayer denotes the original Bayer mosaic image. We see from (11) that the horizontal and vertical heterogeneity maps are derived directly from the Bayer mosaic image via horizontal and vertical heterogeneity-projection, respectively.
B. Directional Adaptive Filtering
The directional heterogeneity-projection along an edge direction leads to a small heterogeneity value; however, it may also obtain a small heterogeneity value when the directional heterogeneity-projection performs along a wrong edge direction. This problem will cause a wrong decision in the interpolation step. In order to overcome this problem, a directional adaptive filter whose behavior changes based on statistical characteristics of the image inside a local window is designed to reduce the estimation error in horizontal and vertical heterogeneity maps.
The proposed directional adaptive filter is divided into horizontal and vertical adaptive filters. For horizontal heterogeneity map, only the horizontal adaptive filter is applied to it without the vertical one. The concept of directional adaptive filter is to perform adaptive filtering based on statistical measures of surrounding pixels along one direction. The simplest statistical measures are the mean and variance in a local window [12]. For instance, consider the horizontal adaptive filtering of a pixel Hh on the horizontal heterogeneity map; the adaptively filtered pixel *
h H is obtained by ) ( * L h R h R h L h L h L h h H H H H H H H − + + = δ δ δ . (12) where L h H and R h
H , respectively, denote the left and right neighboring pixels of Hh; ( , L)
h L h H H δ and ( , R) h R h H H δ
are the local mean and variance of L h
H and R h
H , respectively. Similarly, the vertical adaptive filter for the pixel H v on the vertical heterogeneity map is given by
) ( * U v D v D v U v U v U v v H H H H H H H − + + = δ δ δ , (13) where U h H and D h
H , respectively, denote the up and down neighboring pixels of Hh; ( , U)
v U v H H δ and ( , D) v D v H H δ are the local mean and variance of U
v
H and D v
H , respectively. After adopting, respectively the horizontal and vertical adaptive filters presented above into horizontal and vertical heterogeneity maps, the filtered horizontal and vertical heterogeneity maps * _ map h H and * _ map v H are obtained.
4. HARD-DECISION ADAPTIVE INTERPOLATION
When the horizontal and vertical heterogeneity maps are obtained, a hard-decision rule is employed for color interpolation. First, we define three subsets in the image such that
{
( , )| ( , ) * ( , )}
_ * _ x y H x y H y x h map v map h≡ <α Ω ,{
( , )| ( , ) * ( , )}
_ * _ x y H x y H y x v map h map v ≡ <α Ω ,{
h v}
s≡ x y x y ∉Ω x y ∉Ω Ω ( , )|( , ) ,( , ) , (14)where Ω , h Ω , and v Ω denote the horizontal, vertical, and smooth subsets, respectively. s α is a positive constant
satisfying 0≤α≤1. The parameter α in (14) controls the size of a smooth subset in the image. A small (large) α leads to a large (small) smooth subset in the image. For example, if α=0, the image only contains smooth subset without horizontal and vertical subsets. On the contrary, for α =1, the image only contains horizontal and vertical subsets but without smooth subset.
Second, based on (14), the concept of hard-decision rule for interpolation is obtained
channel. color missing each on pixels g neighborin of Perform channel. color missing each on ion interpolat Perform ) , ( channel. color missing each on ion interpolat Perform ) , ( averaging weight else vertical y x elseif horizontal y x if v h Ω ∈ Ω ∈ (15)
The color interpolation method is performed based on the hard-decision rule (15). We first interpolate green channel because the green plane possesses most spatial information of the image. Each missing green value Gmiss is to be
estimated from its four surrounding green pixels by the following expression
left down right up left left down down right right up up miss e e e e G e G e G e G e G + + + + + + = ˆ ˆ ˆ ˆ , (16)
where Gˆ{up,right,down,left} denote the color-adjusted green values of four surrounding green pixels, and e{up,right,down,left} denote
the corresponding edge indicators. However, in our method, the following modification on edge indicators is adopted according to the hard-decision rule (15) such that
.) 0 , 0 ( ) , ( ) , ( ). 0 , 0 ( ) , ( ) , ( = Ω ∈ = Ω ∈ left right v down up h e e y x elseif e e y x if (17)
In other words, the hard-decision adaptive interpolation for green channel is summarized as follows
⎪ ⎪ ⎪ ⎪ ⎩ ⎪⎪ ⎪ ⎪ ⎨ ⎧ Ω ∈ + + + + + + Ω ∈ + + Ω ∈ + + = s left down right up left left down down right right up up v down up down down up up h left right left left right right miss y x if e e e e G e G e G e G e y x if e e G e G e y x if e e G e G e G ) , ( , ˆ ˆ ˆ ˆ ) , ( , ˆ ˆ ) , ( , ˆ ˆ (18)
Note that the formulation of each surrounding color-adjusted green value in (18) adopts the approach proposed in [3] while the corresponding edge indicator can be referred from among the references [3], [10], and [13]. Hereafter, the color-adjusted value of each color pixel and the corresponding edge-indicator are determined as in [3].
When the green channel has been fully recovered, it can be used to assist the interpolation of red and blue channels. The interpolation procedure for red and blue channels consists of two sub-steps: 1) interpolating the missing red/blue values at blue/red pixels, and 2) interpolating the rest of the missing red/blue values at green pixels. In our method, we only apply the hard-decision rule (15) to the sub-step 2) because there is not enough information to perform horizontal and vertical interpolations in sub-step 1). Since the same procedure is utilized to interpolate the red and blue channels, only the red channel interpolation will be presented.
Let b miss
R denote a missing red value at a blue pixel. It is estimated from its four neighboring red pixels by the following formulation left up left down right down right up left up left up left down left down right down right down right up right up b miss e e e e R e R e R e R e R − − − − − − − − − − − − + + + + + + = ˆ ˆ ˆ ˆ , (19)
where Rˆ{up−right,down−right,down−left,up−left} denote the color-adjusted red values of four neighboring red pixels, and }
, , ,
{uprightdownrightdownleftupleft
green pixels will be proceeded. As the same procedure is performed in green channel, each missing red value at a green pixel g
miss
R can be estimated from its four surrounding red pixels by the following hard-decision adaptive interpolation
⎪ ⎪ ⎪ ⎪ ⎩ ⎪⎪ ⎪ ⎪ ⎨ ⎧ Ω ∈ + + + + + + Ω ∈ + + Ω ∈ + + = s left down right up left left down down right right up up v down up down down up up h left right left left right right g miss y x if e e e e R e R e R e R e y x if e e R e R e y x if e e R e R e R ) , ( , ˆ ˆ ˆ ˆ ) , ( , ˆ ˆ ) , ( , ˆ ˆ (20)
where Rˆ{up,right,down,left} denote the color-adjusted red values of four surrounding red pixels, and e{up,right,down,left} are the
corresponding edge indicators. Finally, a full-color image can be obtained after applying the same interpolation processes described above on each missing blue value.
5. COMPARATIVE STUDY ON EXPERIMENTAL RESULTS
Fig. 2 shows twenty-five Kodak photographic images employed in the experiments for demonstrating the demosaicing performance. According to [14], the CFA operations in digital camera pipeline usually introduce a demosaiced image post-processing framework to provide more pleasing color output. Therefore, the experiments also apply a post-processing framework to complete the comparisons. Fig. 3 illustrates the flowchart of the experiment, which contains interpolation and post-processing steps. In interpolation step, the demosaiced results of the proposed HPHD adaptive interpolation (HPHD-AI) method are compared with those using three recent published methods: Lu’s [3], Gunturk’s [4], and Li’s [5] methods. For Gunturk’s method, we make use of one-level (1-L) decomposition with eight projection iterations in the experiment. For Li’s method, the universal threshold value (δl=δh=4), suggested threshold value (δl =4,δh=0.05), and maximum iteration number iter=20 are chosen in the experiment. For the
proposed method, the presetting window size and positive constant are chosen as N=9 and α=0.8, respectively. All
test images are down-sampled to obtain the Bayer pattern (as shown in Fig. 1) and then reconstructed using the demosaicing methods under comparison in RGB color space.
To evaluate the quality of the demosaiced images, two performance measures are adopted in the experiments: PSNR metric and S-CIELab *
ab E
∆ metric [3, 15]. The PSNR (in dB) metric in this paper is defined as
⎪⎭ ⎪ ⎬ ⎫ ⎪⎩ ⎪ ⎨ ⎧ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − = − ≤ ≤ ≤≤
∑ ∑
1 1 1 2 2 10 ( , ) ( , ) 1 255 log 10 ) ( M v uN v u D v u O MN dB PSNR , (21)where M , N are the total column and row number of the image; O( vu, ) is the color vector at the ( vu, )th position of the original color image; D( vu, ) is the corresponding color vector in the demosaiced color image. Note that, for a demosaiced image, high fidelity implies high PSNR and small S-CIELab *
ab E
∆ measures. A. Quantitative Comparison Using PSNR and S-CIELab Measures
Table I records the PSNR values and S-CIELab *
ab E
∆ measures of the demosaiced results obtained by the proposed interpolation method together with these by other methods for comparison. The bold-type font denotes the highest PSNR and smallest *
ab E
∆ values across each row. From Table I, it can be seen that HPHD-AI method generates improved demosaiced fidelity in most of the test images in the interpolation step. Moreover, in the post-processing step, HPHD-AI method not only has significant improvement, but also obtains the best demosaiced results in most of the test images compared with other methods.
B. Visual Comparison
The demosaiced results shown in Figs. 4-5 evaluate the performance of the proposed HPHD-AI method in edge regions and fine textures. Figs. 4(a)-5(a) show the zoom-in of the test image No. 16 and 20, respectively. These scenes contain many long edges and fine detail regions such as fine fiber patterns (Fig. 4) and picket fences (Fig. 5). These features can effectively challenge the performance of demosaicing methods. Figs. 4(b)-5(b) and 4(c)-5(c) are,
jiiiI I I
II' I i
A/
-.5
. nui
I- - S
-r
'S CFA Samples P CFA Demosaicing Final • Demosaiced ImageFig. 2: Test images used in the experiment.
Fig. 3: Flowchart of the experiment. In the interpolation step, we compare the performance of Lu’s, Gunturk’s, Li’s and proposed HPHD adaptive interpolation (HPHD-AI) methods. In post-processing step, Lu’s post-processing method is adopted into each demosaicing method.
respectively, the demosaiced results obtained from Lu’s and Gunturk’s methods. Figs. 4(d)-5(d) and 4(e)-5(e) are the demosaiced results obtained from Li’s method with the universal threshold value (UTV) and suggested threshold value (STV), respectively. Figs. 4(f)-5(f) are the demosaiced results obtained from HPHD-AI methods. From visual comparison, we observe that the Lu’s, Gunturk’s and Li’s methods induce more color artifacts in edge and textured regions than HPHD-AI do. Therefore, these experimental results validate that proposed HPHD-AI method performs satisfactorily not only in textured regions, but also in well-defined edges of the image.
6. CONCLUSIONS
A novel heterogeneity-projection hard-decision adaptive interpolation (HPHD-AI) algorithm has been developed based on spectral-spatial correlation. The proposed HPHD-AI method effectively reconstructs the fine detail features in both edge and texture regions of demosaiced images. One merit of the proposed algorithm is that it can be combined with many existing image interpolation methods such as decision-based algorithm (set α =1), edge-directed interpolation, adaptive interpolation, linear interpolation, etc. The performance of HPHD-AI method has been compared with three recent published demosaicing methods. Experimental results show that HPHD-AI method not only outperforms all of them in PSNR (dB) and *
ab E
∆ measures, but also gives superior demosaiced fidelities in visual comparison with other methods.
Table I: Performance comparison among recent proposed methods: PSNR (dB) and S-CIELab *
ab E
∆ measures of demosaiced images in the interpolation and post-processing steps.
Step Interpolation Step Post-Processing Step
Method Lu [3] Gunturk [4] Li with UTV [5] Li with STV [5] HPHD-AI Lu [3] Gunturk [4] Li with UTV [5] Li with STV [5] HPHD-AI 1 31.0257 1.5357 29.3765 1.7666 28.4957 1.8899 27.5149 1.9636 31.2606 1.5202 30.7940 1.5466 29.2676 1.7845 28.3192 1.9083 27.4982 1.9795 31.0342 1.5233 2 31.6889 1.7135 33.2296 1.5972 33.6676 1.5396 33.6663 1.5374 31.4653 1.7480 33.8433 1.4668 33.6595 1.5445 33.9846 1.4974 34.0382 1.4871 34.0739 1.4473 3 35.7152 1.4910 34.7577 1.6598 35.2213 1.5958 34.9041 1.6084 35.8612 1.4691 35.7232 1.4943 34.6331 1.6721 35.0579 1.6008 34.8693 1.6089 36.0636 1.4581 4 37.3966 0.9094 36.6168 0.9774 36.3808 0.9766 35.4652 0.9977 37.7831 0.8869 38.0096 0.8576 36.7206 0.9635 36.3960 0.9615 35.6678 0.9774 38.2998 0.8385 5 35.4482 1.3020 34.9839 1.3508 34.8997 1.3260 34.6224 1.3333 35.3821 1.3132 36.1356 1.1861 34.9657 1.3075 34.7714 1.3213 34.5816 1.3261 36.2018 1.1804 6 32.7081 2.0318 32.6411 2.1864 31.8126 2.3790 30.8156 2.4840 32.5609 2.0839 33.7802 1.8551 32.6069 2.1709 31.6062 2.3857 30.7851 2.4808 33.8550 1.8533 7 32.4465 1.2998 34.0239 1.2157 33.8198 1.2266 33.8272 1.2288 33.7754 1.1676 34.0965 1.1592 34.3593 1.1896 34.5397 1.1590 34.6761 1.1528 35.7373 1.0157 8 37.9098 0.9885 36.8763 1.1338 36.7725 1.1444 36.1020 1.1711 37.8379 0.9859 38.1854 0.9694 36.6670 1.1576 36.4265 1.1699 35.9278 1.1954 38.2823 0.9626 9 29.7212 1.8327 30.8332 1.7679 31.2495 1.7192 30.8557 1.7774 30.3530 1.7553 31.3071 1.6277 31.1581 1.7214 31.4196 1.6968 31.1759 1.7479 32.1709 1.5285 10 36.8133 0.8758 36.7662 0.8925 37.2501 0.8255 36.1575 0.8629 37.1048 0.8562 37.8106 0.7919 37.0662 0.8491 37.2927 0.8226 36.3276 0.8594 38.1879 0.7698 11 36.8098 0.8715 36.7975 0.8954 37.0956 0.8286 36.7442 0.8377 36.8842 0.8720 37.5213 0.7926 37.0497 0.8536 37.0952 0.8263 36.8198 0.8343 37.6985 0.7843 12 33.8725 1.4666 34.5407 1.4748 34.4102 1.4275 33.7818 1.4515 34.0164 1.4450 35.2610 1.3140 34.6820 1.4288 34.7541 1.3622 34.2524 1.3772 35.6644 1.2713 13 37.3884 0.6695 37.8205 0.6731 37.7569 0.6760 37.3173 0.6841 38.0053 0.6463 38.3279 0.6267 37.9377 0.6665 37.8628 0.6610 37.5068 0.6690 39.0792 0.5981 14 27.8600 2.4652 29.7386 2.5595 30.4264 2.4457 30.5734 2.4330 27.7554 2.8844 30.2549 2.3619 30.2466 2.8077 30.8242 2.3680 31.1554 2.3196 30.4845 2.3558 15 32.4833 1.7491 30.8370 1.9406 29.6090 2.1114 28.4714 2.2016 32.4883 1.7547 32.6128 1.6518 30.6644 1.9284 29.3860 2.1159 28.5487 2.1994 32.8477 1.6275 16 34.4161 1.3868 34.4301 1.4764 34.3050 1.4804 33.8643 1.4972 34.5715 1.3814 34.9354 1.3388 34.3523 1.4682 34.2067 1.4704 33.8927 1.4846 35.0711 1.3264 17 35.6650 1.0971 37.3602 0.9964 37.0917 1.0009 37.0862 0.9993 37.5058 0.9665 37.2329 0.9865 37.6885 0.9740 37.8239 0.9477 37.8704 0.9431 39.2991 0.8526 18 35.7449 1.4857 36.2947 1.4628 36.4685 1.3340 36.0800 1.3471 35.7404 1.4897 36.8960 1.3056 36.5932 1.3572 36.6429 1.3100 36.2771 1.3213 36.9402 1.3029 19 31.6767 2.2879 32.3393 2.3592 32.3295 2.3903 32.0304 2.4202 31.3846 2.3443 32.9921 2.0898 32.5119 2.3137 32.2416 2.4326 32.0507 2.4546 32.9077 2.1113 20 34.5020 1.3409 34.9738 1.3061 35.2707 1.2493 34.9622 1.2727 34.9280 1.3294 35.7424 1.1902 35.2671 1.2452 35.5570 1.2094 35.3051 1.2304 36.4388 1.1637 21 35.8899 1.0016 35.7991 1.0396 35.7714 1.0294 35.2282 1.0521 35.7142 1.0179 36.8055 0.9230 36.0108 1.0077 35.9894 0.9971 35.5311 1.0146 36.8352 0.9239 22 33.0809 1.3691 34.0980 1.3142 33.8535 1.3468 33.8285 1.3536 32.9655 1.3895 34.6893 1.2138 34.3656 1.2900 34.4198 1.2757 34.5202 1.2687 34.8393 1.2031 23 33.5303 1.3922 32.8830 1.5024 32.9540 1.5250 32.4965 1.5490 33.3529 1.4115 33.7291 1.3651 32.8127 1.5307 32.8188 1.5364 32.5483 1.5548 33.7452 1.3662 24 38.0689 0.8977 37.0203 0.9664 37.0820 0.9820 36.7586 0.9894 38.1592 0.8901 38.1993 0.8945 36.9022 0.9867 36.9148 0.9956 36.7148 1.0017 38.4097 0.8834 25 29.4449 1.4432 29.8870 1.4933 30.0755 1.5055 29.8573 1.5331 29.6185 1.4594 30.0984 1.3599 30.0602 1.4858 30.0909 1.5123 29.9332 1.5353 30.3096 1.3566 Avg. 34.0523 1.4099 34.1970 1.4403 34.1628 1.4382 33.7204 1.4635 34.2590 1.4027 34.9994 1.2948 34.3300 1.4145 34.2577 1.4217 33.9390 1.4410 35.3791 1.2682
(a) (b) (c)
(d) (e) (f)
Fig. 4: Zoom-in demosaicing results of test image No. 16. (a) Original picture; Demosaiced result in the interpolation step using (b) Lu’s method, (c) Gunturk’s method, (d) Li’s method with UTV, (e) Li’s method with STV, and (f) proposed HPHD-AI method.
(a) (b) (c)
(d) (e) (f)
Fig. 5: Zoom-in demosaicing results of test image No. 20. (a) Original picture; Demosaiced result in the interpolation step using (b) Lu’s method, (c) Gunturk’s method, (d) Li’s method with UTV, (e) Li’s method with STV, and (f) proposed HPHD-AI method.
7. ACKNOWLEDGMENT
The authors would like to thank Prof. B. K. Gunturk of Louisiana State University, USA; Prof. Yap-Peng Tan of Nanyang Technological University, Singapore; and Prof. Xin Li of West Virginia University, Morgantown, USA for providing us their CFA demosaicing programs. This work was supported by the National Science Council of Taiwan, ROC under grant NSC 92-2213-E-009-007.
8. REFERENCES
[1] D. R. Cok, “Reconstruction of CCD images using template matching,” in Proc. IS&T Ann. Conf./ICPS., pp.380-385, 1994.
[2] B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Processing Magazine, Vol. 22, No. 1, pp. 44-54, January 2005.
[3] W.-M. Lu and Y.-P. Tan, “Color filter array demosaicking: new method and performance measures,” IEEE transactions on Image Processing, Vol. 12, No.10, pp. 1194-1210, October 2003.
[4] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, “Color plane interpolation using alternating projections,” IEEE transactions on Image Processing, Vol. 11, No.9, pp. 997-1013, September 2002.
[5] X. Li, “Demosaicing by successive approximation,” IEEE transactions on Image Processing, Vol. 14, No.3, pp. 370-379, March 2005.
[6] K. Hirakawa and T. W. Parks, “Adaptive homogeneity-directed demosaicing algorithm,” IEEE transactions on Image Processing, Vol. 14, No.3, pp. 360-369, March 2005.
[7] X.-L. Wu and N. Zhang, “Primary-consistent soft-decision color demosaicking for digital cameras (patent pending),” IEEE transactions on Image Processing, Vol. 13, No.9, pp. 1263-1274, March 2004.
[8] B. Bayer, Color imaging array, U.S. Patent 3,971,065, 1976.
[9] S.-C. Pei and I.-K. Tam, “Effective color interpolation in CCD color filter arrays using signal correlation,” IEEE transactions on Circuits and Systems Video Technol., Vol. 13, No.6, pp. 503-513, Jun. 2003.
[10] L.-L. Chang and Y.-P. Tan, “Effective use of spatial and spectral correlations for color filter array demosaiking,” IEEE transactions on Consumer Electronics, Vol. 50, No.1, pp.355-365, January 2004.
[11] H. Stark and Y. Yang, Vector Space Projections: A Numerical Approach to Signal and Image Processing, Neural Nets, and Optics, New York, John Wiley&Sons, INC., 1998.
[12] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd edition, NJ: Prentice-Hall, 2002.
[13] R. Kimmel, “Demosaicking: Image reconstruction from color CCD samples,” IEEE transactions on Image Processing, Vol. 8, No. 9, pp.1221-1228, 1999.
[14] R. Lukac, K. Martin, and K. N. Plataniotis, “Demosaicked image postprocessing using local color ratios,” IEEE Transactions on Circuits and Systems Video Technol., Vol. 14, No. 6, pp.914-920, 2004.
[15] M. Mahy, E. Van Eyckden, and O. Oosterlinck, “Evaluation of uniform color spaces developed after the adoption of CIELAB and CIELUV,” Color Res. Applicat., Vol. 19, No. 2, pp. 105-121, 1994.
