• 沒有找到結果。

中 華 大 學

N/A
N/A
Protected

Academic year: 2022

Share "中 華 大 學"

Copied!
89
0
0

加載中.... (立即查看全文)

全文

(1)

中 華 大 學 碩 士 論 文

以色差平面為基礎有效率的彩色濾光片陣列 影像解馬賽克內插演算法

Effective color-difference-based interpolation algorithm for CFA Images

demosaicking

系 所 別:資訊工程學系碩士班 學號姓名:E09702003 鄭盛義 指導教授:黃 雅 軒 博士

中 華 民 國 九十九 年 八 月

(2)

致 謝

感 謝 黃 雅 軒 教 授 , 在 學 生 求 學 的 過 程 當 中 , 對 學 生 報 告 時 的 教 誨,讓 學 生 能 夠 清 楚 明 瞭 在 學 校 上 台 報 告 以 及 在 業 界 工 作 上 作 簡 報 是 如 此 的 不 同 、 是 如 此 的 細 膩 及 嚴 謹 ; 另 外 也 由 於 黃 雅 軒 教 授 對 學 生 論 文 時 的 指 導 , 引 導 學 生 在 做 研 究 時 如 何 能 夠 更 準 確 的 看 到 問 題 點 , 使 得 做 研 究 更 有 效 率 。

對 於 學 生 論 文 的 完 成,也 感 謝 黃 教 授 的 耳 提 面 命,以 及 做 研 究 時 所 必 須 要 的 嚴 謹 , 在 學 生 寫 論 文 的 時 候 , 不 時 的 提 醒 學 生 在 論 文 陳 述 的 過 程 中 要 盡 可 能 讓 複 雜 的 演 算 法 及 流 程 、 變 得 淺 顯 易 懂 。 要 盡 可 能 種 點 式 的 將 計 算 式 背 後 的 物 理 意 義 表 答 出 來、且 在 論 文 文 章 撰 寫 的 時 候 必 須 要 注 意 文 章 的 起 承 轉 合 , 以 及 很 多 的 小 細 節 , 讓 學 生 懂 得 做 論 文 時 必 須 要 注 意 到 很 多 以 前 沒 有 注 意 到 的 小 細 節 , 使 學 生 做 事 更 細 心 , 以 及 學 習 到 耐 心 。

感 謝 黃 教 授 , 花 費 很 多 寶 貴 的 時 間 跟 學 生 討 論 論 文 , 以 及 不 辭 辛 勞 的 替 學 生 修 改 論 文 。 這 也 讓 學 生 在 求 學 過 程 中 或 是 未 來 受 益 無 窮 。 感 謝 業 界 同 事 所 提 供 的 協 助 , 使 得 我 能 夠 結 合 學 術 上 的 知 識 以 及 業 界 的 經 驗 、 讓 我 在 做 研 究 的 過 程 中 , 在 思 考 問 題 的 時 候 能 夠 更 全 面 性 的 思 考 問 題 , 哪 些 地 方 應 該 加 強 。

最 後,我 要 感 謝 我 親 愛 的 家 人。感 謝 父 母 的 栽 培 與 鼓 勵,你 們 是 我 一 直 成 長 的 最 大 動 力 來 源 , 還 有 要 感 謝 我 親 愛 的 太 太 ─ 瑞 祺 , 有 妳 的 鼓 勵 與 默 默 奉 獻 , 才 讓 我 可 以 無 後 顧 之 憂 的 完 成 我 的 學 業 。 最 後 ,

(3)

中 文 摘 要

在 這 篇 論 文 , 提 出 了 以 色 差 平 面 為 基 礎 的 一 個 有 效 的 補 色 演 算 法 (ECDB)。為 了 提 高 補 色 後 的 影 像 品 質,ECDB 演 算 法 在 補 綠 色 點 時 , 會 先 進 行 分 析 周 邊 的 樣 本 點 (或 是 方 向 ), 進 而 選 擇 適 當 的 樣 本 點 去 補 該 缺 少 的 綠 色 像 素 ; 在 補 滿 綠 色 頻 道 後 , ECDB演 算 法 利 用 紅 色 、 綠 色 以 及 藍 色 這 三 個 色 彩 頻 道 之 間 的 色 彩 相 互 關 係 去 產 生 出 紅 綠 色 彩 差 異 頻 道 跟 藍 綠 色 彩 差 異 頻 道,進 而 利 用 這 兩 個 色 彩 差 異 頻 道 去 建 構 出 完 整 的 紅 色 以 及 藍 色 頻 道。其 中 這 紅 綠 跟 藍 綠 色 彩 差 異 頻 道 屬 於 低 頻 訊 號,所 以 在 插 補 的 過 程 中 可 以 利 用 低 頻 訊 號 的 特 性 降 低 插 補 過 程 中 所 產 生 的 雜 訊 以 提 升 品 質 。

很 明 顯 的 , 一 個 小 區 域 中 色 彩 差 異 頻 道 在 不 只 是 在 視 覺 上 比 單 獨 看 紅 、 綠 、 藍 頻 道 平 滑 , 而 且 利 用 了 綠 色 頻 道 的 資 訊 去 建 構 紅 色 跟 藍 色 頻 道 , 使 得 補 色 出 來 的 結 果 更 佳 的 精 確 。 因 此 , ECDB演 算 法 利 用 紅 綠 色 彩 差 異 頻 道 跟 藍 綠 色 彩 差 異 頻 道 以 及 配 合 一 個 式 子 去 補 色 , 而 這 個 式 子 結 合 水 平 、 垂 直 以 及 對 角 方 向 的 鄰 近 點 資 訊 去 提 升 所 猜 出 來 的 色 素 的 精 確 性 ; 使 得 在 補 色 後 的 影 像 看 起 來 很 自 然 。 最 後 在 實 驗 結 果 上 , 無 論 是 實 驗 數 據 或 圖 片 , 在 這 篇 論 文 中 所 提 出 來 的 方 法 都 比 這 裡 論 文 中 所 比 較 了 的 其 它 方 法 更 好 。

關 鍵 字 : 色 彩 濾 光 片 陣 列 , 解 馬 賽 克 , 內 插

(4)

Abstract

This thesis proposes an effective color-difference-based (ECDB) interpolation algorithm for CFA Image demosaicking. A CFA image consists of a set of spectrally selective filters which are arranged in an interleaved pattern such that only one of color component is sampled at each pixel location. To improve the quality of reconstructed full-color images from color filter array (CFA) images, the ECDB algorithm first analyzes the neighboring samples around a green missing pixel to determine suitable samples for interpolating the value of this green missing pixel. After finishing the interpolation operations of all the green missing pixels, a complete green plane (i.e G plane) can be obtained. The ECDB algorithm then makes use of the high correlation between R, G, and B planes to produce the red-green and blue-green color difference planes and further reconstructs the red and the blue planes in successive operations. Because of the green plane provides twice information than red and blue planes, the algorithm exploits the information of green plane more than that of red/blue plane so that the full color image can be reconstructed more accurately. In essence, the ECDB algorithm not only ingeniously combines several up-to-date mainstream interpolation techniques but also designs an effective pixel categorization mechanism to develop a high efficient interpolation mechanism. Co mpared with other

(5)

position relationship and then it effectively applies different interpolation mechanisms for different categories; 2, it classifies the pixel variation of a local region in more detail, and designs the effective interpolation mechanism with suitable neighboring pixels and appropriate weighting technique for each kind of pixel variation; and 3, instead of using only the first-order gradient of a certain color, it makes use of the second-order gradient of other color to estimate the missing value of this certain color, because the second-order gradient of other color provides the extra color variation trend which is useful to interpolate the missing color value. The experimental results performed on the well-known Kodak 3-CCD image database demonstrate that the proposed algorithm has outstanding performance.

Keywords: Color filter array, demosaicking, interpolation.

I V

(6)

Table of Contents

致 謝 ………..I 中 文 摘 要 ……….II

Abstract……….……III Table of Contents………..……....V List of Figures……….…VII List of Tables……….……….X

Chapter 1 Introduction………..1

Chapter 2 Review of Color Filter Array Interpolation Algorithms………..4

2.1 Bilinear Interpolation………...5

2.2 Gradient Based Interpolation………..………6

2.3 Effective Color Interpolation………..8

Chapter 3 An efficient demosaicking algorithm

for CFA images……….….. 12

(7)

3.2 Proposed scheme………..………..……18

3.2.1 The demosaicking procedure of green plane………….…..18 3.2.2 Principle of variable naming………...20 3.2.3 The demosaicking procedure of R-G/B-G color difference

planes….………..………29 3.2.4 Value estimation of the missing diagonal-class R-G

pixels……….…...31 3.2.5 Value estimation of the missing vertical-class R-G

pixels……….…...…...38 3.2.6 Demosaicked red plane and blue plane………...40

Chapter 4 Experimental and performance

discussion ………..…………41

4.1 Experimental results..…………..……….………41 4.2 Complexity analysis and performance discussion……66

Chapter 5 Conclusions and future work…….…..71 Reference……….………...73

V I

(8)

List of Figures

Figure 2.1. CCD sense an image via CFA……….….…...4 Figure 2.2. The central blue sample B5 has no green color which should be estimated……….….5 Figure 2.3. The central blue sample B7 has no green color value which should be estimated with the neighboring color values….…...7 Figure 2.4. (a) A full-color test image; (b), (c), and (d) are red, green, and

blue components of the test image, respectively; (e) kr image; (f)

kb image………..…...……9

Figure 2.5. Interpolating the green color of the central red sample R7.…10 Figure 3.1. Three primary color components for each pixel……..…...…12 Figure 3.2. Color filter array (CFA) with single color component at each

of sample location based on the Bayer pattern…………...13 Figure 3.3. (a) The original image of fence, (b) false color image of fence,

(c) original image of fence, (d) the zipper effect of fence....15 Figure 3.4. The flowchart of the ECDB algorithm……….………...…21 Figure 3.5. (a) to (d) corresponding to Eq. 3-12 cases 1 to 4,

respectively……….23 Figure 3.6. The configuration of related pixels for estimating the green

value of pixel located at “c1”... ……...23 Figure 3.7. (a) The demosaicked green image by the bilinear interpolation

algorithm, (b) the demosaicked green image by the ECDB algorithm……..…………..………...29 Figure 3.8. R-G = Red - Demosaicked Green………...31 Figure 3.9. (a) to (d) corresponding to Eq. 3-20 cases 1 to 4,

respectively……….32 Figure 3.10. The to-be-estimated R-G value on the blue sample B ……..33

(9)

Figure 3.12 demosaicked G plane +demosaicked R-G plane = demosaicked R plane, demosaicked G plane +demosaicked B-G plane = demosaicked B plane………..……….40 Figure 4.1. The 24 Test images (1) brickWood, (2) wall, (3) hats, (4)

RedRidingHood, (5) motocross, (6) boatZaotime, (7) flowers&sill, (8) house, (9) sailboats, (10) sailboat, (11) ZenTimeAtThePeer, (12) beachBums, (13) mountainStream, (14) Rafting, (15) Kid, (16) landAhoy, (17) GoldenBall, (18) ContryStyle, (19) fence, (20) six-shooter, (21) lightHouse, (22) RusticDream, (23) Parrots, and (24) 2house, image database source: http://r0k.us/graphics/kodak/...….45 Figure 4.2. Enlarged parts of the demosaicked image corresponding to the

fence picture: (a) the original image, (b) BI, (c) ACPI, (d) ECI, (e) the proposed ECDB.………...52 Figure 4.3. Enlarged parts of the demosaicked image corresponding to the

fence picture: (a) the original image, (b) BI,(c) ACPI, (d) ECI, (e) the proposed ECDB..……….………...….53 Figure 4.4. Enlarged parts of the demosaicked image corresponding to the brickwood: (a) the original image, (b) BI, (c) ACPI, (d) ECI, (e) the proposed ECDB...54 Figure 4.5. Enlarged parts of the demosaicked image corresponding to the house: (a) the original image, (b) BI, (c) ACPI, (d) ECI, (e) the proposed ECDB………..….………...54 Figure 4.6. Enlarged parts of the demosaicked image corresponding to the lightHouse: (a) the original image, (b) BI, (c) ACPI, (d) ECI, (e) the proposed ECDB…..………..…….…………..55 Figure 4.7. Enlarged parts of the demosaicked image corresponding to the 2house: (a) the original image, (b) BI, (c) ACPI, (d) ECI, (e) the proposed ECDB……….….……….56 Figure 4.8. Enlarged parts of the demosaicked image corresponding to the

Parrots: (a)The original image, (b) BI, (c) MFI, (d) CHI, (e) ECI, (f) the proposed ECDB…………..…….……….57 Figure 4.9. Enlarged parts of the demosaicked image corresponding to the Kid: (a) The original image, (b) BI, (c) MFI, (d) CHI, (e) ECI, (f) the proposed ECDB...57

(10)

Figure 4.11. Enlarged parts of the demosaicked image corresponding to the Fence: (a) The original image, (b) AP, (c) EMI, (d) KA, (e) PVM, (f) the proposed ECDB...59 Figure 4.12. Enlarged parts of the demosaicked image corresponding to the Fence: (a) the original image, (b) AP, (c) EMI, (d) KA, (e) PVM, (f) the proposed ECDB...60 Figure 4.13. Enlarged parts of the demosaicked image corresponding to the Rafting: (a) the original image, (b) AP, (c) BI, (d) MFI, (e) PVM, (f) the proposed ECDB...61 Figure 4.14. Enlarged parts of the demosaicked image corresponding to the fence (a) The original image, (b) the noise added image, (c) GBI, (d) ECI (e) proposed ECDB…...63 Figure 4.15. Enlarged parts of the demosaicked image corresponding to the fence, (a) The original image, (b) the noise added image, (c) GBI, (d) ECI (e) the proposed ECDB...64 Figure 4.16. Enlarged parts of the demosaicked image corresponding to the house (a) The original image, (b) the noise added image, (c) GBI, (d) ECI (e) proposed ECDB………...65

(11)

List of Tables

Table 3.1: List of suffix attribute and symbol………..19

Table 4.1: difference methods comparison of image No.1~8………..…..47

Table 4.2: difference methods comparison of image No.9~16…...48

Table 4.3: difference methods comparison of image No.17~24…...49

Table 4.4: difference methods comparison of all image’s average...50

Table 4.5: Rustic...50

Table 4.6: Parrots...50

Table 4.7: Kid…...51

Table 4.8: Fence...51

Table 4.9: Rafting...52

Table 4.10: Results of Noise sensitivity test with difference methods….62 Table 4.11: The total number of computations…...67

Table 4.12: Complexity comparison of the proposed algorithm and various algorithms...68

X

(12)

Chapter 1 Introduction

In nowadays, Most of electronic devices such as digital still cameras, mobile phones, and PDAs use a single image sensor to capture digital images, which usually consist of three color components (red, green, and blue) at each pixel location. However, the surface of single image sensor is covered with a color filter array (CFA). A CFA image consists of a set of spectrally selective filters which are arranged in an interleaved pattern such that only one of color component is sampled at each pixel location. In order to render a full color image, image interpolation algorithm is performed to estimate the other two missing color components and the image interpolation is called as CFA demosaicking algorithm (spectral interpolation).

To obtain a good demosaicked color image, a lot of demosaicking algorithms have been proposed, such as the Cubic Spline Interpolation (CSI) [22], the Nearest-Neighbor replication (NNI) [20][21], the Alternative Projection (AP) [04], the Bilinear Interpolation (BI) [05], the Edge Map Interpolation (EMI) [06], Kimmel’s Algorithm (KA) [07], the Median Filter Interpolation (MFI) [08], the Principle Vector Method (PVM) [09], the

(13)

Interpolation (ECI) [12] etc.

Although, a lot of demosaicking algorithms have been proposed, but in general they all still suffer two major problems in fine texture areas: (1) there are considerable false color artifacts, and (2) there are obvious zipper effect. We realize that (1) if we could classify the pixel variation in a local region more detail and effectively, then interpolation of the missing color values could be more accurate; (2) if we can category each pixel based on the accordingly geometrical position relation, then it will be much easier t o design the appropriate interpolation mechanism for each pixel category.

Based on the above realizations, we devoted myself to this color demosaicking research and hoped to further improve the false color artifact and the zipper effect at the fine texture areas.

In this thesis, we propose an effective demosaicking algorithm to reconstruct full-color images from CFA images based on Bayer CFA pattern, and the reconstructed image can provide very good visual quality. The proposed demosaicking algorithm is a new kind of Effective Color-Difference-Based interpolation (ECDB) demosaicking algorithm. The ECDB algorithm starts from green plan e to produce reconstructed full-color image, because of the green plane provides twice information than red and blue planes. Therefore, when interpolating the green missing colors, the ECDB algorithm will make use of neig hboring pixels to analyze the unifor m direction of each pixel, here high uniform means low variation among pixel

(14)

values and uniform direction means pixels along a direction (such as vertical, horizontal and diagonal) are low variation. Then, according to the speciall y designed operations with suitably selective neighboring pixels, the missing green color can be effectively estimated. Once the green plane is produced, the ECDB algorithm exploits both the demosaicked green plane and the original red/blue plane to produce green-red and green-blue color difference planes, and then the color difference planes are interpolated by using neighboring samples with suitable weighting technique. Finally, by using the green plane, red-green and blue-green color difference planes high-quality demosaicked full-color images can be reconstructed.

The rest of this thesis is organized as follows. The section 2 introduces the most representatively existed interpolation methods. Section 3 describes the proposed ECDB method, and Sectio n 4 demonstrates the experimental results, the complexity analysis and performance discussion of three CFA demosaicking methods including ECI, AP, and the proposed method in detail.

Finally, conclusions and future works are drawn in Section 5.

(15)

Chapter 2

Review of Color Filter Array Interpolation Algorithms

In general, the flowchart of a demosaicking process is shown as Figure 2.1.

Color Filter Array

Figure 2.1. CCD sense an image via CFA

There are three interpolation polices to reconstruct a full color image.

The first policy is to interpolate each color plane independently; the second policy is to interpolate each plane by exploiting color difference plane; and the third policy is to interpolate each plane not only by exploiting color difference plane but also by considering the variation trend of other color in a local area. In this chapter, we will briefly introduce the most representative interpolation methods for each policy, which are Bilinear Interpolation (policy 1), Gradient Based Interpolation (policy 2), and Effective Color Interpolat ion (policy 3). In fact, the three representative methods are the main comparison methods with the proposed method in the

(16)

experiment chapter of this thesis.

2.1 Bilinear Interpolation

The Bilinear Interpolation (BI) is one of the well-known interpolation algorithms, which estimates the missing color value of one pixel by simply averaging its surrounding pixel values. Therefore, BI is very easy to implement and can performs very fast. The BI makes use of neighboring samples to interpolate the missing color. Referring to figure 2.2, the missing green color (G5) located at B5 is interpolated as following:

4 ,

8 6 4 2 5

G G G

G G + + +

=

(2-1)

And the missing red color (R5) located at B5 is calculated as:

4 ,

9 7 3 1 5

R R R

R = R + + +

(2-2)

The missing red colors (R2) on the green sample location are interpolated as:

2 . R

2 ,

7 1 4 3 1 2

R R R

R = R + = +

(2-3)

(17)

estimated.

Next, the other missing red/blue colors are calculated in the same manner. However, BI does not consider the uniform direction information of each pixel and it probably chooses unsuitable samples to interpolate the missing color, so that it usually generates a lot false colors and zipper effect around the fine texture areas.

2.2 Gradient Based Interpolation

Gradient Based Interpolation (GBI) is proposed by Laroche and Prescott [28]. GBI starts the interpolation operations at the green plane, and then it takes advantage of the color difference planes to reconstruct red/blue plane so that the quality of demosaicked images can be accordingly improved. As shown in Figure 2.3, the missing color G7 located at the blue sample B7 is needed to estimate, GBI starts with calculating ce and cv, which are given below:

2 . cv B

2 ,

7 13 1

7 9 5

B B B B ce B

+ −

= + −

=

(2-4)

(2-5)

where ce and cv are individually the second-order derivative of the horizontal and vertical directions, which are used to determine whether the estimated pixel belongs to the vertical or horizontal direction. The value of G7 is calculated as follows:

(18)

. if

4 ,

; if

2 ,

G

; if

2 ,

11 8

6 3

11 3

8 6

7

⎪ ⎪

⎪⎪

+ = + +

+ >

+ <

=

cv G ce

G G G

cv G ce

cv G ce

G

G

(2-6)

Figure 2.3. The central blue sample B7 has no green color value which should be estimated with the neighboring color values.

The remaining missing green colors located on red samples are processed in the similar manner. Once the green plane is produced, GBI takes advantage of the red-green and blue-green color difference planes to interpolate the missing red/blue colors. The missing red colors are produced as follows:

6 10 10 2

2 6

3 4

4 2

2 3

2

) (

) (

2

) (

) (

G G R G

R R

G G R G

R R

− +

− +

− +

− + +

= −

− + +

= −

(2-7)

(19)

The missing blue colors are interpol ated in the same approach. After the interpolation process, the complete full-color image is generated.

2.3 Effective Color Interpolation

Effective Color Interpolation (ECI) [12] exploits the high correlation between the R, G, and B planes to produce R-G and B-G color difference planes which are denoted as kr and kb, respectively. Figure 2.4 illustrates an example that kr and kb are much smooth than the original R, G, and B planes;

and kr and kb are quite flat over local regions. These properties are very useful for interpolating missing pixels to reduce the estimated errors and improve the visual quality of demosaicked images.

(a) (b)

(20)

(c) (d)

(e) (f)

(21)

Because of the CFA image provided twice information on green channel than the red and blue color channel, ECI starts to reconstruct the full-color image with the green plane first and then the red or blue plane. As shown in Figure 2.5, to interpolate the missing green color G7 at the red sample (R7) position, ECI starts with calculating the kr values around R7, i.e. kr 3, kr 6, kr 8, and kr 11. For example, kr 3 and kr 6 are given as blow.

2 . ) (

2 , ) (

7 5 6

6

7 1 3 3

R G R

k

R G R

k

r r

− +

=

− +

=

(2-8)

(2-9)

Figure 2.5. Interpolating the green color of the central red sample R7 .

Because of the symmetry in the Bayer pattern, kr 8 and kr 11 can be calculated similarly as Equations (2-8) and (2-9), respectively. Then the missing green color (G7) located at the central position R7 can be calculated by Equation (2-10). As for the missing green colors at the blue sample positions, they are calculated in the same way.

(22)

4 .

)

(

3 6 8 11

7 7

r r r

r

k k k

R k

G = + + + +

(2-10)

After generating the green plane, ECI reconstructs the red and blue planes based on the same image calculation model. Referring to Figure 2.5, the interpolated red and blue colors are produced by using kr and kb as

4 .

) (

2 , ) (

12 10

4 2 7

7

7 1 3

3

b b

b b

r r

k k

k G k

B

k G k

R

+ +

− +

=

− +

=

(2-11)

(2-12)

(23)

Chapter 3

An efficient demosaicking algorithm for CFA images

Digital electronic devices become very popular over capturing digital images that are convenient for transmission or maintenance. A pixel in a full-color image has consists of three primary color components (red, green, blue) as shown in Figure 3.1, so a digital device needs three sensors to record each color component in the digital format. The sensor is an expensive component of digital electronic devices. Therefore, most digital electronic devices nowadays such as personal digital assistants (PDAs), mobile phones etc., only use a single sensor called color filter array (CFA) [15] to capture digital images. The CFA based on Bayer pattern consists of a set of interleaving filters, as shown in Figure 3.2, to sample one of three primary red, green and blue color components.

R G B

Figure 3.1. Three primary color components for each pixel

(24)

Figure 3.2. Color filter array (CFA) with single color component at each of sample location based on the Bayer pattern

To render a full-color image, we interpolate the other two missing pixels on a sample location. The interpolation processes are cal led the CFA demosaicking (spectral interpolation). There are well-known interpolation methods to reconstruct an image into a full-color type, such as bilinear interpolation, pixel duplication, and cubic spline interpolation. These methods are all signal plane interpolation ones and provide satisfactory results in smooth regions especially. However, these methods may not address the problem of false color and zipper effects deriving from the high frequency regions. Figure 3.3(a) is an original image of fence and Figure 3.3(b) shows the false color image of fence; Figure 3.3(c) is an original image of fence and Figure 3.3(d) shows the zipper effect occurring in the sharp regions of the fence.

(25)

(a)

(b)

(26)

(c)

(d)

Figure 3.3. (a) The original image of fence, (b) false color image of fence, (c)

(27)

3.1 Previous

To obtain better results in terms of visual quality, the adaptive CFA demosaicking methods was proposed by using the high spectral-spatial correlation between different color planes such as the R-G and B-G color difference planes [18][19]. In a localized image area, the R-G and B-G color difference planes have an equal difference and a constant ratio. In addition, the color difference planes are smoother than the original R, G, and B planes and low pass signal; the estimated errors after the performance of interpolation operations can be reduced.

However, the human visual system is sensitive to the edge structures in an image so an edge sensitive interpolator is used to direct the interpolation along sharp edges to increase the spatial resolution. Hamiliton and Adams proposed an interpolation scheme based on the second-order Laplacian filter [16] [17]. This scheme uses the se cond-order gradients of blue or red samples and the first-order gradients of green samples to estimate the missing green colors. The second-order gradients of blue or red samples in a vertical or horizontal direction are used as correction factors. Once the green samples are filled, the red and blue samples are interpolated similar with the correction of the second-order gradients of the green channel.

This thesis proposed a new effective color-difference-based

(28)

render a better resolution of demosaicked full-color images. In a raw CFA image, the sampling frequency of green channel is as twice as that of red or blue channel. So, the green planes can be demosaicked and interpolated more accurately than red and blue planes. Inspired from this, the ECDB algorithm starts interpolating operation on the green plane.

When reconstructing the green plane, the ECDB algorithm will analyze the neighboring values around the missing green color to be interpolated for selecting suitable (directional) neighboring samples to estimate the missing green color. Then, the ECDB algorithm uses the demosaicked green plane and the original CFA image to produce R-G and B-G color difference images.

However, the produced R-G and B-G color difference images are not complete. There exist some missing pixels on the color difference planes.

Thus, the ECDB algorithm calculates the missing pixels on the color difference planes by combining the neighboring samples with appropriate weights determined by basing on the spectral difference among samples to estimate the missing values. Finally, the ECDB algorithm combines the demosaicked green plane with complete R-G and B-G color difference planes to recover the red and blue planes to produce a full-color demosaicked image.

(29)

3.2 Proposed scheme

Although, the novel algorithm ECI already improved the false-color-artifact and the zipper-effect problems, but it still has some false color artifacts at the fine detail areas. Therefore, we proposed the ECDB algorithm. Based on the Bayer pattern, a CFA image holds twice number of green samples than those of red and blue samples. So the green plane has a great effect upon the quality of the final demosaicked image. The ECDB algorithm starts from the interpolation of the green plane. When the edge structure of an image object becomes blurred or seriously distorted, our visual system will perceive the image quality is poor. Therefore, the human visual system is very sensitive to the edge structure of an image, Hence, in order to reconstruct the demosaicked image with a satisfactory visual quality and to reduce the false color or the zipper effect, we propose an effective CFA interpolation method by considering the edge structure to reconstruct the green plane.

3.2.1. Principle of variable naming

In the proposed algorithm we use quite a few variables. In order to let variables can be understood more easily, we introduce the variable naming principle in this section. When we define a variable, usually it contains an

(30)

strength”, ‘w’ denotes “weight”, and ‘d’ denotes “demosaicked value”. For superscript, ‘r ’, ‘g’, and ‘b’ denote the red color, the green color and the blue color, respectively. Also, ‘r-g’ denotes the color difference of the red color and the green color, ‘b-g’ denotes the color difference of the blue color and the green color, ‘gb’ denotes in the green color channel reference to the blue color value, ‘rb’ denotes in the color difference ‘r-g’ channel reference to the blue color value, ‘rg’ denotes in the color difference ‘r-g’

channel reference to the green color value, and ‘a’ denotes “adaptive” to represent more than one color channel are used to interpolate a missing value. For suffix, only the direction attribute, weight attribute, and the summation attribute will appear. In suffix, symbol ‘w’ denotes the weight attribute, symbol ‘s’ denotes the summation attribute, and there are 12 direction-related attributes which are listed in Table 3.1 with their names and their corresponding symbols. For example, variable denotes the vertical edge strength computed from the green color channel.

g

e|

Table 3.1 List of suffix attribute and symbol.

Suffix attribute symbol

Vertical |

Horizontal -

Diagonal /

(31)

Bottom ↓

Left ←

Right →

Top right → ↑

Top left ← ↑

Bottom right → ↓ Bottom left ← ↓

3.2.2. The demosaicking procedure of green plane

Figure 3.4 shows the flowchart of the proposed ECDB algorithm. The ECDB algorithm starts with the interpolation of the demosaicked green plane, then it exploits the R-G and B-G color difference planes to reconstruct the red and blue planes, respectively.

(32)

1

2

3

4 G

B-G

R

B

Demosaicked G plane

R-G

6

Demosaicked R-G plane

Demosaicked B-G plane

5

+

+

CFA Image

Figure 3.4 The flowchart of the ECDB algorithm.

During interpolating the green plane, we will take consideration for the neighboring pixels around a currently processing pi xel. In general, high continuity neighboring pixels will have higher attribute (such as color or texture) consistency than low continuity neighboring pixels. To interpolate the missing color value of a pixel A we should choose only those pixels having high attribute consistency from all the neighboring pixels of A and

(33)

he edge strength to measure the continuity and consistency among its eighboring pixels, and also use the edge strengths of different directions to etermine the consistency direction (horizontal or vertical) of its t

n d

corresponding region E. Obviously, the direction having the weaker edge strength probably corresponds to the consistency direction. If both vertical and horizontal edge strengths are very weak, then the region will be determined to be a smooth region. On the contrary, if both vertical and horizontal edge strengths are strong, then the region will be determined to be a complex region. Conceptually to interpolate the missing color value of pixel A, only those pixels along the consistency direction will be used.

However, even along the consistency direction probably only a part of pixels are truly consistent and the other are not. Therefore, we further judge the region E into five different cases so that more appropriate pixels can be chosen. In Figure 3.5, it conceptually show the (a) to (d) shows that corresponding case 1 to case 4, respectively. Case 1, the consistency direction is horizontal but only the right or left half of E is consistent. Case 2, the consistency direction is vertical but only the upper or lower half of E is consistent. Case 3, the consistency direction is horizontal and both the right and left half of E are consistent. Case 4, the consistency direction i s vertical and both the upper and lower half of E are consistent. Case 5, both horizontal and vertical direction are either both consistent or both not consistent. With the above judgment, we design the following mechanism to interpolate the missing green color. In the designed mechanism, we also developed a novel weight technique to give each chosen pixel sample a

(34)

Figure 3.6. The configuration of related pixels for estimating the green value of pixel located at “c1

suitable weight value for calculation.

Figure 3.5. (a) to (d) corresponding to Eq. 3-12 cases 1 to 4, respectively.

”.

Figure 3.6 displays the configuration of related pixels for estimating the green value of pixel located at “c1”. Because Bayer pattern have only

(c) (d)

(a) (b)

c1

(35)

interpolate green color value on the blue plane in detail, and it will be similar to interpolate the green color value on the red plane.

If the estimated value on the horizontal direction has high attribute consistence for both side (left side and right side), we give an equal weight value to both side; otherwise, we give different weight values to both sides.

edge strengt h, green color upper edge strength and reen color lower edge strength of this configuration, respectively. Let

an

The same concept can be applied on vertical direction. Let

e

| , e and e be the blue color vertical

gb g g

g

e

gb- ,

eg d eg be the blue color horizontal edge strength, green color right edge strength and green color left edge strength of this configuration, respectively.

The values of

e

gb| ,e ,g e ,g

e

gb,e and g e are computed by Eq. 3-1 as g

. G G

* G

G e

, G G

* G

G e

B B

*B e

g g gb

8 4 4

2

2

2 , 2

− +

=

− +

=

=

(3-1)

Also, let eg| and eg be the edge strength va lues of the vertical and horizontal directions of this configuration, respectively. The value of g

acco ng to the calculation of

e

gb, e and g e , and the value of g e is g

, G G

* G

G e

, G G

* G

G e

, B B

*B e

g

6 2 4

2

5 3 1

7 3 3

1

5 1 3

1 gb

2 2 2

− +

=

− +

=

=

is rdi

g

4 2 1

|

e|

(36)

e com

according to the calculation of

e

gb, eg and eg . The above two edge strengths can b puted by Eq. 3-2 as

. , e e e

e

e e e

e

g g gb g

g g gb

1 2 1

|

2

g

|

+ + +

=

+ + +

=

Let , and db be horizontal-direction les which are used to calculate the estimated missing green value .

a ig

v

d

T ,

a

(3-2)

dg db three estimated

variab

d ˆ

|

w

g nd

w

g are the we ht values of db and db . d|g is the estimated

alues of vertical direction. db is the difference of B1 and B2. db is the ifference of B1 and B4. wg and w are the weight values of db and db.

he 10 parameters (d, d, d, w, d| d, d, w and w ) re calculated as bellow:

g

g b b g

w , g g b b g g

4 . 4

2

4 , 4

2

4 1 2

1 3

g 1

|

5 1 3

1 4

2

B d B

B , d B

G , d G

B d B

B , d B

G , d G

b b

b b

g

= −

= −

= +

= −

= −

= +

(3-3)

, w e , w

e

w

g

e

g g g

g

g

1

= −

= +

(3-4)

(37)

d

l d ns ly

(3-5) (3-6)

Then, the central green missing sample “c1” ca n be estimated by the following Equation:

Let and dwa| be the estimated values of horizontal and vertical directions with weight, respectively. da and da| are the estimate values of horizontal and vertica irectio without weight, respective . It can be calculated as bellow:

a

dw

,

, wa| g| g b g b

b g b g g a

w d w d w d d d w d w d

d = + × + × = + × + × .

, | a g| b b

b b g

a d d d d d d d

d = + + = + +

⎪⎪

⎪⎪

+

<=

<

>

+

>

<=

<

>

+

>

>

<

>

+

>

>

>

=

, w

D e

e T

e T e e R /e e , d

D e

e T

e T e e R /e e , d

D e e T

e T e e R /e e , d

D e e /e

e , d

d

g a a

eg g

g g

g a

eg g

g g

g g

a

eg g g g

g a

w

eg g g g

a w

g

; and

and )

( and if

; and

and )

( and if

; and

and )

( and if

; if

ˆ

g

|

2 g

| 1 g

| g

|

|

2 1

g

| g

|

2 g

| 1 g

| g

|

|

g

|

where dˆ is the estimated value for the green missing pixel on the red or t “c1

<

>

+e T e T

e

R and( g| g) 1and g 2and

×

× d w .

d| otherwise

(3-7)

blue sample location a

g

”.

In Eq. 3-7, R, T1 , T2 and De g are four threshold values. R is the threshold for the ratio of and . If the value of is larger than R, it means the configuration of “c1

eg| eg eg| /eg

” has relativel gh continuity in the horizontal direction, else if the value of is larger than R, then it

y hi eg/eg|

(38)

tion of “c1

means the configura ” has relatively high continuity on the vertical direction, otherwise, there is no obvious horizontal and vertical continuity at position “c1”. If the edge strength | + | is larger than T1, it means the central pixel at “c1

eg| eg

” position is located at a non-smooth area with low continuity in either horizontal or vertic rection; otherwise

ixel “c1

al di

” is located at a smooth area with high continuity in both p

horizontal and vertical directions. If the edge strength (either eg or eg| ) is less than T2, it means pixel “c1” is located at a non-complex a with h continuity in either horizontal or vertical direction; otherwise, “

rea hig c1” is

lo

tr g

cated at a complex area with low continuity in both horizontal and vertical direction. eg and eg are the edge s en ths of the horizontal direction for the right side and the left side at “c1”, respectively. eg and

e are the edge strengths of the vertical direction for the top side and the g

bottom side, respectively . If the value of |eg -eg | is larger than D

e g, it means the edge strengths of the right and the left are significantly different, otherwise, the two values have no large difference. or the vertical side, the same principle can be applied to decide whether the edge strengths of the top side and the bottom side are different enough by comparing |eg-eg| with D .

F

e g

(39)

s

al

strength (i.e. >R), (2) the total horizontal and vertical edge strength the hori

The second, third, and fourth situations correspond to a vertically half-

ila

continuity in the respectively vertical and horizontal direction to give each direction the suitable weight. So that we interpolate the central green missing value according to and their weights and by the following Equation.

configuration which satisfie the following four conditions: (1) the horizontal edge strength should be relatively smaller than the vertic edge

|

should be not too small (i.e. | + >T eg/eg

eg eg 1), (3) zontal edge strength

should be small enough (i.e. eg<T2), and (4) only half of the horizontal direction (either the right-half or the left-half) is uniform and the other half is non-uniform (i.e. |eg -eg | >De g). If all above four conditions are satisfied, then we estimate the missing value

d ˆ

g by the value of dwa.

side uniform configuration, a horizontally two-side uniform configuration, and a vertically two-side uniform configuration, respectively.

If the green missing pixel does not belong to the above four situations, it then belongs to the fifth situation which corresponds to either a non-uniform or a uniform configuration in both horizontal and vertical directions. Readers can easily understand the required conditions of each situation because they are sim r to the condition illustration of the first situation. In the fifth situation we will base on the individual green color

da,da| wg| wg

(40)

. w w

e , e

e g

g g

g g

| g

+

After each green missing pixel value has been estimated by the Equation 3-7, the full demosaicked green plane can be produced, which is called

w| = =1− | (3-8)

G . Compared with a demosaicked green plane Figure 3.7(a) reconstructed by the bilinear interpolation algorithm, visually the demosaicked green plane Figure 3.7(b) reconstructed by the ECDB algorithm show obviously better quality with much little zipper effect.

(a) (b)

Figure 3.7. (a) The demosaicked green image by the bilinear interpolation algorithm, (b) the demosaicked green image by the ECDB algorithm.

3.2.3. The demosaicking procedure of R-G/B-G color difference

(41)

demosaicked green plane and B-G denote the color difference plane of the blue plane and the demosaicked green plane. That is R-G = R-G and B-G = B- G . Both R-G and B-G inherently are low-pass signal which can be utilized to reduce the estimated errors of the demosaicked images, and the ECDB algorithm uses them to reconstruct the red and blue planes, individually. Since both R-G and B are derived by similar operation , only the deriva

many ixels will accordingly have no values in the R-G plane. These pixels are

class, vertical class, and horizontal class. Figure 3.8 shows that a R-G plane is derived from subtracting the red plane with the demosaicked

green plane, of which the gray pixels have their R-G values and the white

R-G R-G

have no red values and their values should be further estimated. For a missing R-G pixel, we will decide its clas s first and accordingly estimate its R-G pixel value. The class criteria are (1) if its left and right R-G pixels have values, it is attributed to the horizontal class; (2) if its top and bottom R-G pixels have values, it is attributed to the vertical class; (3) otherwise,

it is attributed to the diagonal class. Because in our design when reconstructing a R-G pixel value of an either horizontal-class or vertical-class pixel, the values of some pixels belonging to diagonal class will be referenced; but the R-G pixel value of a diagonal-class pixel can be

-G s

tion of R-G is described here.

Because there are many red-missing pixels in the red plane R, p

called the “missing R-G pixels” which can be categorized into three classes:

diagonal

pixels are the missing pixels which have no values because they

(42)

3

interpolate a missing green color value, but the neighboring pixels in the

diagonal ate the missing

hows that corresponding case 1 to

estimated directly without referencing the values of horizontal-class and vertical-class pixels. Therefore, the diagonal-class pixels should be estimated first, and the pixels of the other two classes are estimated later.

.2.4. Value estimation of the missing diagonal-class R-G pixels

During interpolating the R-G value at a blue sample location, we will take consideration for the neighboring pixels around a currently processing pixel. In general, the principle to interpolate the missing R-G pixel is very similar to that of interpolate the missing green color which has been explained in section 3.2.2. The main difference between them is that only the neighboring pixels in the vertical and horizontal directions are used to

and reverse diagonal directions are used to interpol green color value. Referring to Figure 3.9. (a) to (d) s

case 4, respectively.

Figure 3.8. R-G = Red - Demosaicked Green

(43)

F

f a missing R-G pixel, Figure .10 is provided to show the geometric relations of the missing R-G pixels.

For estimating the R- alue of B1, let , and be the edge strengths in the vertical, horizontal, diagonal and reverse diagona l directions, respectively. The four edge strengths (

(a)

(c)

igure 3.9. (a) to (d) corresponding to Eq. 3-20 cases 1 to 4, respectively.

In order to describe the estimated value o (b)

(d)

3

G v ea| ,

e

a e/a e\a

{ | , , /, \ }

i

a

,

i

∀ ∈ −

e

) are

defined as

.

rg

,

G G G G G G e

G G G G G G e

rg

= − + − + −

− +

− +

=

,

* 2

,

* 2

2 .

,

2 ,

,

19 3 3 1 1 13

|

9 5 1

\ 3

7 1 /

8 4 1 2

6 1

|

\

\

/ /

rb

| rg

| a

|

B B B e

B B B e

e e e e

e e e

e e e e

e e e

rb rb

rb r

r a rb

rg a

rb r

r a

=

=

+ +

= +

=

+ +

= +

=

→↓

←↑

←↓

→↑

(3-9)

,

* 2

,

*

2 B B B e B B B

e

rb

= − −

rb

= − −

(3-10)

(44)

igure 3.10. The to-be-estimated R-G value on the blue sample B1.

Let , ←↑ and be the edge strengths of the missing R-G pixels on the top-right, the bottom-left, the top-left, and the bottom-right direction, respectively. The values of , and

can be given by

a

| rg

|

, e e

a

r e

e→↑, /

a

r e

e

F

e

→↑r

e

←↓r ,

e

r

e

→↓r

e

→↑r

e

←↓r ,

e

←↑r

e

→↓r

,

* 2

,

*

2

1 6

1 3

R R R

R e

R R R

R e

r r

− +

=

− +

→↑

=

a rg

e e

,

←↑, \

a rg

e e

,

a

r e

e←↓, / e→↓r ,e\a

a

| rg

|

,e

e

參考文獻

相關文件

• Color the nodes representing literals by their truth values (color 0 for false and color 1 for true).. – We are dealing with the a-triangles here, not the

EQUIPAMENTO SOCIAL A CARGO DO INSTITUTO DE ACÇÃO SOCIAL, Nº DE UTENTES E PESSOAL SOCIAL SERVICE FACILITIES OF SOCIAL WELFARE BUREAU, NUMBER OF USERS AND STAFF. ᑇؾ N

The resulting color at a spot reveals the relative levels of expression of a particular gene in the two samples, which may be from different tissues or the same tissue under

Rugged, high resolution, full-color, video-rate displays enable a multitude.

• Given a finite sample of some texture, the goal is to synthesize other samples from that same is to synthesize other samples from that same texture...

– S+U can also preserve annotations of synthetic images – Refined images really help improving the testing result – Generate &gt; 1 images for each synthetic

Experiment a little with the Hello program. It will say that it has no clue what you mean by ouch. The exact wording of the error message is dependent on the compiler, but it might

Multiple images from a sequence tracked with 6DOF SLAM on a client, while a localization server provides the global pose used to overlay the building outlines with transparent