基於人類視覺系統之混合雜訊消除技術

(1)

國立交通大學

電機與控制工程學系

博

士

論

文

基於人類視覺系統之混合雜訊消除技術

Human-Visual-System-Based Mixed-Noise

Removal Techniques

研究生：盧世茂

指導教授：林進燈

張志永

中華民國九十五年九月

(2)

基於人類視覺系統之混合雜訊消除技術

Human-Visual-System-Based Mixed-Noise

Removal Techniques

研究生：盧世茂 Student：Shih-Mao Lu

指導教授：林進燈博士 Advisor：Dr. Chin-Teng Lin

張志永博士

Dr. Jyh-Yeong Chang

國立交通大學

電機與控制工程學系

博士論文

A Dissertation

Submitted to Department of Electrical and Control Engineering

College of Electrical Engineering and Computer Science

National Chiao Tung University

in partial Fulfillment of the Requirements

for the Degree of

Doctor of Philosophy

in

Electrical and Control Engineering

Septemper 2006

Hsinchu, Taiwan, Republic of China

中華民國九十五年九月

(3)

基於人類視覺系統之混合雜訊消除技術

研究生：盧世茂指導教授：林進燈

張志永

國立交通大學電機與控制工程研究所

摘要

雜訊常是嚴重損害影像品質並且破壞重要細節的因素。附加在影像上的雜訊，大體上可以用兩種雜訊模型來模擬表現：突波雜訊及高斯雜訊。本論文提出兩級式的架構，相繼消除混合雜訊影像中非線性的突波雜訊及線性的高斯雜訊。在第一級消除突波雜訊中，基於決策機制的適應性中位數濾波器，應用於消除椒鹽雜訊 (Salt-Pepper noise)，而另一個適應性類神經網路的架構，則用於消除隨機的突波雜訊。然後，我們提出以人類視覺系統為基礎的影像品質增強系統，再補償修改過的圖素。在第二級消除高斯雜訊中，我們提出了線性改良式模糊規則為基礎的濾波器 (MFRB) 消除線性的高斯雜訊，並盡可能的保留影像的邊緣及細節部分。為了考量實際狀況，我們設計了幾個通用的 MFRB 濾波器來處理各種不同摻雜程度的高斯雜訊。利用所估測到影像雜訊值的大小，我們選擇對應到雜訊摻雜程度最接近的 MFRB 濾波器，去除影像中的高斯雜訊。根據實驗結果顯示，所提出的方法不論在客觀的評比數據(PSNR)或主觀的視覺感知上，都優於其他所比較的雜訊消除技術。

(4)

Human-Visual-System-Based Mixed-Noise

Removal Techniques

Student: Shih-Mao Lu Advisor: Chin-Teng Lin Jyh-Yeong Chang

Department of Electrical and Control Engineering National Chiao-Tung University

Abstract

Noise always significantly damages an image and can corrupt most important details. Two noise models can adequately represent most noise added to images: additive impulse noise and Gaussian noise. In this thesis, we propose a two-stage filtering method to sequentially remove the mixed noises of images corrupted with nonlinear impulse and linear Gaussian noises as well. In the first stage, the decision-based recursive adaptive median filter is applied to remove the Salt-Pepper noise, and an adaptive two-level neural network noise reduction procedure is applied to remove the random-valued noise. Then, an HVS-directed neural-network-based image quality enhancement is applied to compensate the modified pixels. In the second stage, we derive a linear modified fuzzy rule-based (MFRB) filter to remove the linear type Gaussian noise while preserving the image edges and details as much as possible. For practical consideration, we design several sets of universal MFRB filters, to be utilized in correspondence to the estimated values of contaminated Gaussian noise variance in the image. The correspondent MFRB filter closest to the estimated Gaussian noise level will be selected to remove the Gaussian noise of the processed image. According to the experiment results, the proposed method is superior, both quantitatively and visually, compared to several other techniques.

(5)

誌謝

首先，感謝指導教授林進燈院長及張志永教授多年來的指導。無論是在專業研究上或是生活上的教導，都使我獲益良多。兩位教授敏銳宏觀的思維、創新前瞻技術的研究、努力不懈、實事求是、行事嚴謹的特質，都是非常值得我學習的地方。此外，也非常感謝諸位口試委員寶貴的意見，使得本論文得以更加完備。在家人方面，感謝我的雙親盧龍修與林麗華，以及弟弟俊宇和妹妹薏如多年來的支持與鼓勵，使我得以無後顧之憂的專心於學業方面。在學校方面，感謝梁勝富博士、蒲鶴章博士、張俊隆博士、秦群立博士等學長及同學們在學業上與生活上的幫忙與照顧。也感謝實驗室的伙伴們-文昌學長、建文、孝羽、宇文、芳誼、曉佩、得平… 及研究生團契的輔導-家齊老師、珊如老師和弟兄姊妹們-宥達、京荃、朝旺、佩琪、玉芳、安琪…，因為你們的出現及參與，使得我在交通大學的生活更加的多采多姿。在教會方面，感謝陳正雄牧師、李啟誠牧師及小家人們-慧美姐、毓琴、有廷、其華、敬業、彥欽…的支持與鼓勵，感謝神一路走來不斷的引導及看顧，使我得以能夠順利完成博士學位。在此，謹以本論文獻給我的家人與關心我的師長與朋友們。

(6)

List of Figures

Fig. 2.1. A low frequency spatial grating (top) and a higher frequency grating (bottom).

Both gratings have the same contrast. ...10

Fig. 2.2. A low contrast spatial grating (top) and a higher contrast grating (bottom). Both grating are of the same spatial frequency. ...12

Fig. 2.3. Visibility thresholds corresponding to different background luminance...13

Fig. 2.4. Typical human contrast sensitivity function. The arrow points to the cutoff frequency. ...14

Fig. 2.5. Visibility thresholds corresponding to different background luminance...15

Fig. 2.6. The 4×4 sliding (overlapping) window blocks applied to the image quality enhancement...16

Fig. 2.7. An illustration of the relation between SD parameter and the distribution of pixels in a sliding block. ...18

Fig. 2.8. Portions of (a) the sliding block containing texture structure, (b) the sliding block containing edge structure...19

Fig. 2.9. (a)-(d) Membership functions of fuzzy sets on input variables VD, SD, CD, and output variable Mo, respectively...20

Fig. 2.10. Flow diagram of angle evaluation. ...22

Fig. 2.11. The proposed feedforward neural network for image quality enhancement. ...24

Fig. 3.1. The proposed two-step impulse noise removal algorithm. ...29

Fig. 3.2. The decision-based adaptive recursive median filtering scheme...30

Fig. 3.3. Histogram statistics of Lena with 20% impulse noise within different noise range. (a) range = 5 [(0−4); (251−255)]; (b) range =10 [(0−9); (246−255)]; (c) range = 15 [(0−14); (241−255)]. ...32

Fig. 3.4. Elements of the 3×3 window centered around x(n). (a) Non-recursive; (b) recursive. x (n) represent original pixels and i y (n) represent processed i pixels. ...34

Fig. 3.5. The proposed scheme for impulse noise removal. ...35

Fig. 3.6. The procedure diagram of the first-level impulse noise removal. ...35

Fig. 3.7. The procedure diagram of the second-level impulse noise removal. ...36

Fig. 3.8. The proposed neural network for noise detection. ...37 Fig. 3.9. Schematic block diagram of the proposed image quality enhancement system.

(9)

...41 Fig. 3.10. (a) Original image; (b) Lena with 20% of impulse noise; (c) the 3×3

standard median filter; (d) the 3×3 recursive standard median filter; (e) the recursive CWM filter with weight = 3; (f) the recursive Tri-state median filter with threshold = 25; (g) Li’s method with threshold = 32; (h) our proposed method. ...46 Fig. 3.11. (a) Original image; (b) Peppers with 40% of impulse noise; (c) the 3×3

standard median filter; (d) the 3×3 recursive standard median filter; (e) the recursive CWM filter with weight = 3; (f) the recursive Tri-state median filter with threshold = 25; (g) Li’s method with threshold = 32; (h) our proposed method. ...47 Fig. 3.12. (a) 30% Impulse Salt-Pepper noise [63]; (b) the 3×3 standard median

filter; (c) the recursive Tri-state median filter with threshold = 25; (d) Li’s method with threshold = 32; (e) DMMD Denoise Software [65]; (f) our proposed method. ...48 Fig. 3.13. (a) Original image; (b) Lena with 40% of impulse noise; (c) image after processing of the 1st step (impulse noise removal); (d) image after the processing of 2nd step ( image quality enhancement)...49 Fig. 3.14. (a) Lena with 20% of white impulse noise; (b) processed by our proposed method...49 Fig. 3.15. The performance comparisons of different noise removal methods

with/without the proposed noise range assumption applying to “Lena” corrupted with 40% impulse noise. (a) and (d) the 3×3 recursive standard median filter with and without the noise range assumption, respectively. (b) and (e) the recursive Tri-state median filter (threshold = 25) with and without the noise range assumption, respectively. (c) and (f) Li’s method (threshold = 32) with and without the noise range assumption, respectively.

...50 Fig. 3.16. Performance comparison of the matrix-based estimation method and the

proposed neural network for the 2nd step compensation with respect to the Lena image with 40% of impulse noise. (a) Image after the processing of the 1st stage; (b) and (c) images after the 2nd step compensation by using the resultant weighting matrices of the matrix-based estimation and the proposed neural network, respectively. ...50 Fig. 3.17. (a) Original image; (b) Lena with 20% of fixed-valued impulse noise; (c) the3×3standard median filter; (d) the 3×3 recursive median filter; (e) the recursive tri-state median filter with threshold = 25; (f) optimal weighted median filter; (g) Li’s edge preserving method with threshold = 32; (h) PBM

(10)

filter; (i) Zhang’s fuzzy techniques; (j) Trilateral filter; (k) 1st step of the proposed method; (l) The proposed two-step method. ...57 Fig. 3.18. (a) Original image; (b) Boat with 25% of random-valued impulse noise; (c) the3×3standard median filter; (d) the 3×3 recursive median filter; (e) the recursive tri-state median filter with threshold = 25; (f) optimal weighted median filter; (g) Li’s edge preserving method with threshold = 32; (h) PBM filter; (i) Zhang’s fuzzy techniques; (j) Trilateral filter; (k) 1st step of the proposed method; (l) The proposed two-step method. ...58 Fig. 3.19. (a) Original image; (b) Barbara with 25% of random-valued impulse noise; (c) the3×3standard median filter; (d) the 3×3 recursive median filter; (e) the recursive tri-state median filter with threshold = 25; (f) optimal weighted median filter; (g) Li’s edge preserving method with threshold = 32; (h) PBM filter; (i) Zhang’s fuzzy techniques; (j) Trilateral filter; (k) 1st step of the proposed method; (l) The proposed two-step method. ...59 Fig. 3.20. (a) Original image; (b) Baboon with 25% of random-valued impulse noise; (c) the3×3standard median filter; (d) the 3×3 recursive median filter; (e) the recursive tri-state median filter with threshold = 25; (f) optimal weighted median filter; (g) Li’s edge preserving method with threshold = 32; (h) PBM filter; (i) Zhang’s fuzzy techniques; (j) Trilateral filter; (k) 1st step of the proposed method; (l) The proposed two-step method. ...60 Fig. 3.21. (a) Original image; (b) Peppers with 25% of random-valued impulse noise; (c) twice applying of 3×3 standard median filter; (d) twice applying of

3

3× recursive median filter; (e) twice applying of the recursive tri-state median filter with threshold = 25; (f) optimal weighted median filter; (g) twice applying of Li’s edge preserving method with threshold = 32; (h) twice applying of the PBM filter; (i) Zhang’s fuzzy techniques; (j) twice applying of the Trilateral filter; (k) result after 1st step processing of the proposed method; (l) results of the proposed two-step method. ...61 Fig. 4.1. The probability distribution of several typical images for the Variable [a]. ..72 Fig. 4.2. Elements and their subscripts in the 5×5 filter window...73 Fig. 4.3. A typical local variance σn probability distribution for the Variable [c]. ...74

Fig. 4.4. Overlapping

7 ×

7

sliding window for local variance estimation, based on

3

3× sub-area. ...77 Fig. 4.5. (a) Original “Boats” image; (b) Noisy image of “Boats” with Gaussian noise

only of σ=20; ...78 Fig. 4.6. Filtered “Boats” images of Fig. 4.5(b) using (a) 3 × 3 Wiener filter; (b)

5

5 × Wiener filter; (c) FWLS [47]; (d) MFF [48]; (e) FLF [49]; (f) PBM

(11)

Fig. 4.7 (a) Original “Bridge” image; (b) Noisy image of “Bridge” with 10% impulses and Gaussian noise of σ=10; (c) Original “Lena” image; (d) Noisy image of “Lena” with 20% impulses and Gaussian noise of σ=15. ...82 Fig. 4.8. Filtered “Bridge” images of Fig. 4.7(b) using (a) FK-NN with 3 × 3

Wiener filter; (b) FK-NN with 5 × 5 Wiener filter; (c) FWLS [47]; (d) MFF [48]; (e) FLF [49]; (f) PBM [21]; (g) Trilateral Filter [31]; (h) First Stage; and (i) Our Two-Stage Method. ...83 Fig. 4.9. Filtered “Lena” images of Fig. 4.7(d) using (a) FK-NN with 3 × 3 Wiener

filter; (b) FK-NN with 5 × 5 Wiener filter; (c) FWLS [47]; (d) MFF [48]; (e) FLF [49]; (f) PBM [21]; (g) Trilateral Filter [31]; (h) First Stage; and (i) Our Two-Stage Method...84 Fig. 4.10. The corrupted images (a) Original “Airplane” image; (b) Airplane with 20% impulses and Gaussian noise of σ= 13; (c) Original “Elaine” image; (d) Elaine with 17% impulses and Gaussian noise of σ= 18. ...86 Fig. 4.11. Filtered “Airplane” images of Fig. 4.10(b) using (a) FK-NN with 3 × 3

Wiener filter; (b) FK-NN with 5 × 5 Wiener filter; (c) FWLS [47]; (d) MFF [48]; (e) FLF [49]; (f) PBM [21]; (g) Trilateral Filter [31]; (h) First Stage; and (i) Our Two-Stage Method. ...87 Fig. 4.12. Filtered “Elaine” images of Fig. 4.10(d) using (a) FK-NN with 3 × 3

Wiener filter; (b) FK-NN with 5 × 5 Wiener filter; (c) FWLS [47]; (d) MFF [48]; (e) FLF [49]; (f) PBM [21]; (g) Trilateral Filter [31]; (h) First Stage; and (i) Our Two-Stage Method. ...88

(12)

List of Tables

Table 2.1 The eight weighting matrices obtained from the trained ...26 Table 3.1 Quantitative comparisons of different noise removal methods applied to the images with various percentages of impulse noise...51 Table 3.2 Compensation ability of our adaptive median filter in the 1st step and the

image quality enhancement system in the 2nd step with respect to Lena...52 Table 3.3 Noise detection accuracy of the proposed method with respect to “Lena”

image corrupted with various range and density of impulse noise. ...52 Table 3.4 Speed Comparison for Various Algorithms (unit: sec) ...52 Table 3.5 Comparative results of various noise detection algorithms applied to ...62 Table 3.6 Comparative results in PSNR of different algorithms applied to various

kinds of images corrupted with 20% fixed-valued impulse noise...63 Table 3.7 Comparative results in PSNR of different algorithms applied to image

“Elaine” corrupted by various rates of random-valued impulse noise. ...63 Table 3.8 Comparative results in PSNR of different algorithms applied to various

kinds of images corrupted with 25% of random-valued impulse noise...64 Table 3.9 Speed Comparison for Various Algorithms (sec.) ...64 Table 4.1 Comparative Results of PSNR in the Cases of Corruption by Gaussian

Noise (σ = 20)...80 Table 4.2 Comparative Results of PSNR in the Cases of Corruption by Mixed

Gaussian (σ = 10) and Impulse Noise (p = 10%) ...81 Table 4.3 Comparative Results of PSNR in the Cases of Corruption by Mixed

Gaussian (σ = 15) and Impulse Noise (p = 20%) ...81 Table 4.4 Comparative Results of PSNR in the Cases of Corruption by Mixed

Gaussian (σ = 20) and Impulse Noise (p = 30%) ...82 Table 4.5 The Estimated Gaussian Noise STD of σ=10, 20, and 30...89 Table 4.6 Comparative Results of PSNR in the Cases of Random Mixed Noise...89

(13)

1. Introduction

1.1. Motivation

In the real-life, images are often contaminated by mixture of impulse and Gaussian noises of varying noise intensities due to the imperfection of sensors and communication channels when transmitted. The objectives of image noise removal are to remove the mixed noise and to retain the edges or other salient structures in the original image. Noise smoothing and edge enhancement are inherently conflicting processes, since smoothing a region will destroy an edge and sharpening edges might lead to enhance the unnecessary noise. Thus it is a difficult work for a universal algorithm that can remove different kinds and intensities of noise from images and preserve their sharpness and details. In this thesis, we propose a two-stage fuzzy filtering method to sequentially remove the mixed noises of images corrupted with nonlinear impulse and linear Gaussian noises well.

1.2. Impulse and Gaussian Noise Model

The additive impulse noise and Gaussian noise models can adequately represent most noise added to images. Impulse noise is the pixels randomly misfired and replaced by other values in an image. Such noise can be introduced due to transmission errors. With the noise ratio p, only p percent of the pixels in the image are replaced and others keep noise-uncorrupted. For example, when u is an image, ui,jwill represent the intensity value of u at the pixel location (i,j) in the

(14)

related to the original image u0 by ⎪⎩ ⎪ ⎨ ⎧ − = p) (1 y probabilit with p y probabilit with 0 , , , j i j i j i u n u _(1.1)

where the impulse noise is denoted by n.

In a variety of impulse noise models for images, fixed and random-valued impulse noises are mostly discussed. Fixed-valued impulse noise, known as the “Salt-Pepper” noise, is the corrupted pixels whose values are replaced with values equal to the maximum or minimum (255 or 0) of the allowable range with equal probability (p/2). The random-valued impulse noise is the corrupted pixels whose values are replaced by random values uniformly distributed in the range within [0, 255].

Additive Gaussian noise is characterized by adding to each image pixel a value which is drawn from a zero-mean Gaussian distribution. Such noise is usually introduced during image acquisition. For the case of additive Gaussian noise, the noisy image u is related to the original image u0 by

j i j i j i u n u _, = 0_, + _, _(1.2)

1.3. Previous Research

Nowadays, image processing techniques have been well developed, but there are still some bottlenecks that have not been solved. Many image processing algorithms cannot work well in a noisy environment; therefore, the noise removal algorithm is adopted as a preprocessing module. A number of approaches have been developed for noise removal and listed as follows.

(15)

1.3.1. Existing Impulse Noise Removal Algorithms

The nonlinear filtering technique, standard median (SM) [1]−[2] filter, based on order statistic has been demonstrated generally superior to linear filtering (moving average) on suppressing impulse noise. However, median filter still tends to blur fine details and destroy edges while removing out the impulse noise. To achieve better performance, median filter has been modified in many ways, such as weighted median (WM) [3]−[4], center weight median filters (CWM) [5], adaptive-length median filter [6], the recursive medians [7]−[8] and the alpha-trimmed mean filter [9]. They were expected to increase the signal preservation but relatively decrease the noise suppression ability. Applying these algorithms altogether across the whole image without identification would inevitably remove the uncorrupted detail pixels, destroy the image quality, and cause additional blur.

For that reason, the decision-making schemes [10]−[12] were proposed in which only the identified noisy pixels are processed. Besides, the switching schemes [13]− [17] provide adaptive decision to recover the noisy pixels based on several filters to remain noise-free pixels unchanged. In [18], the progressive switching median filter was proposed for only fixed-valued impulse noise removal. Also, the weighting-average linear combinations of nonlinear median-based filters through learning-rule optimization have been proposed [19]−[21]. Although satisfactory results have been obtained, they tend to remove fine details or retain too much of the noise due to undetection or misdetection of the noise [22]. In addition, since the noisy pixels are replaced without taking into account local features, details and edges are not recovered satisfactorily, especially when the noise level is high. The thresholding filtering [23] which composed of new efficient noise detectors was proposed to prevent the misclassification of noise-free pixels. The edge-directed noise detection

(16)

and suppression strategy was proposed to preserve the details and edges [24]. Two stage approaches that combine noise identification and edge preserving supplementary have been proposed trying to remove the noise cleanly and keep the detail information well [25]−[26].

1.3.2. Existing Gaussian Noise Removal Algorithms

On the other hand, linear filtering algorithms, such as Wiener [27] and Kalman [28] filters are essentially low-pass filters and well-known for their ability to remove the Gaussian noise, but they cannot remove the impulse noise well and tend to blur the fine features in the image. To remove the mixed noise in an image, a combination approach or hybrid filters [29]−[30] have become a promising approach. Garnett et al. [31] introduced the local image static - Rank-Ordered Absolute Differences (ROAD) to quantify how different in intensity the particular pixels are from their most similar neighbors, and proposed the trilateral filter, modified from the bilateral filter [32], for removing mixed Gaussian and impulse noise.

1.3.3. Existing Neural and Fuzzy Noise Removal Algorithms

Since neural networks have the ability to learn from examples and fuzzy systems have the ability of reasoning to deal with uncertainty, they also have a growing number of applications in image noise removal in the past few years [33]−[40]. Zhang

et al. [33] proposed the fuzzy techniques to detect the impulse noise and to find the

ultimate remote window to replace the noise-liked pixel with a linear combination of the pixels. Schulte et al. [34] proposed a fuzzy derivative estimation with fuzzy rules in the first stage for noise detection and fuzzy smoothing of neighboring pixels in the second stage. Lee et al. [35] proposed a fuzzy image filter based on genetic learning

(17)

process. Russo [36]−[37] proposed a recursive Neuro-Fuzzy filter with specifically designed original multiple output network structure and learned the parameters based on the genetic algorithm. M. E. Yüksel et al. [38] proposed a new impulse noise detector comprises two identical Neuro-Fuzzy subdetectors combined with a decision maker.

On the other hand, there are mainly three kinds of fuzzy approaches used in mixed noise removal of an image. The first kind is the fuzzy weighted average filter [41] and fuzzy weighted median filter [42]. Peng [43]−[44] proposed a multi-level adaptive fuzzy (MLAF) filter, which uses fuzzy sets to adaptively combine simple linear and nonlinear filters to remove varying mixed noise with different levels. The second kind is the fuzzy logic filter, which suggests that individual pixels should not be uniformly fired by each of the fuzzy rules. Choi et al. [47] derived three different filters for each of the three objectives using the fuzzy weighted least squares (FWLS) method. And he defined the criteria for selecting each of the filter based on the local context using the fuzzy rules. Taguchi [48] proposed the modified fuzzy filter (MFF) with new local characteristic calculated with fuzzy rules by using multiple difference values between arbitrary pixels in the filter window. Farbiz et al. [49] proposed the fuzzy logic filter (FLF), which adopted the general structure of fuzzy if-then-else rules mechanism. The S-type fuzzy function enables the non-uniform firing of the basic fuzzy rules. For the third kind uses fuzzy reasoning which is embedded into the neural network structure through genetic learning algorithm. It is able to adapt the filtering action to different distributions of mixed noise.

1.3.4. Existing Human Visual System Algorithms

(18)

years. The characteristics of the HVS have been incorporated into the digital image processing such as watermark encoder design, digital image compression, and image recognition etc [51]−[59]. The perceptual redundancies inherent in a still image are basically due to the inconsistency in sensitivity of the HVS to stimuli of varying levels of contrast and luminance changes in the spatial domain. It was found that the perception of HVS is more sensitive to luminance contrast rather than the uniform brightness [60]. In addition to the magnitude difference between object and the background, different structures of images also cause different visual perceptions for HVS. Many features have been proposed based on the block DCT in frequency domain and Wavelet [61]−[62]. A novel fuzzy decision system inspired by HVS is proposed to classify the image into human perception sensitive and non-sensitive regions.

1.4. Brief Summary

In this chapter, several noise removal algorithms are presented. As mentioned above, it is difficult to remove the mixed noises without blurring the edge and details information. The two tasks, involving suppressing the impulse noise and removing the Gaussian noise in an image, are very different in characteristics because each of them respectively facilitates the nonlinear and linear filtering operations. Besides, the presence of impulse noise can seriously degrade the performance of a restoration technique that is designed mainly to remove Gaussian type noise. Consequently, we propose in this thesis a two-stage filtering technique to remove the nonlinear impulse and linear Gaussian noises sequentially. In the first stage, the decision-based recursive adaptive median filter is applied to remove the Salt-Pepper noise and an adaptive two-level neural network noise reduction procedure is applied to remove the

(19)

random-valued noise described in Chapter 3. Then, an HVS-directed neural-network- based image quality enhancement described in Chapter 2 is applied to compensate the modified pixels. In the second stage, we derive a linear modified fuzzy rule-based (MFRB) filter to remove the linear type Gaussian noise while best preserving image details, and we combine the two stage filtering and proposed the universal MFRB filter for mixed noise removal described in Chapter 4. Finally, conclusions and perspectives are described in Chapter 5.

(20)

2. Human Visual System Based Image Quality

Enhancement

In this chapter, we proposed an HVS-directed neural-network-based image quality enhancement. The fuzzy decision rules inspired by human visual system (HVS) are proposed to classify pixels of the image into human perception sensitive class and non-sensitive class. A neural network is proposed to enhance the sensitive regions to perform better visual quality. In addition, the proposed fuzzy decision rules combined with the neural network can balance the trade-off between speed and quality for different applications by just adjusting a threshold parameter. The ideas of using the fuzzy system to implement human visual system and using the neural network for image enhancement are new. The learning results can be directly implemented through the look up table (LUT) to reduce the computational cost and the system complexity for practical applications. It can be combined with any other filtering approach to enhance the visual quality.

2.1. Introduction

The visual acuity of the eye is generally regarded as the most important factor for the ability of the eye for seeing objects. The acuity of the eye is usually measured by acuity tests where the minimum visible separation is measured of black rings with a small interrupted part. These tests are used for decisions that play a role in the properties of the human visual system.

Objects can generally be better distinguished from each other or from their background, if the difference in luminance or color is large. Of these two factors,

(21)

luminance plays the most important role. In practice, it appears that it is not the absolute difference in luminance that is important, but the relative difference. This relative difference can be expressed by the ratio between two luminance values, which is called contrast ratio, or by the difference between two luminance values divided by the sum of them, which is simply called contrast. Objects that only have a small contrast with respect to their background are difficult to observe. The eye is more sensitive for the observation of objects, if the required amount of contrast is lower. The reciprocal of minimum contrast required for detection is called contrast sensitivity.

In this chapter, Section 2.2 studied the characteristics of spatial visual for improvement in applications of digital image processing. The structure of the image quality enhancement system is described in Section 2.3. Section 2.4 gives conclusions of this chapter.

2.2. Characteristic of Spatial Vision

When we speak of spatial vision, we refer to the visual system’s ability to detect and analyze changes in brightness across space.

2.2.1. Characteristics of Sine Wave Grating

Comparing the two gratings in Fig. 2.1, the grating on the bottom shows more alterations between light and dark than the grating on the top. In a given space (the space taken up by the photograph), there are more alterations, or cycles, in the bottom grating. Therefore, this bottom grating has a higher spatial frequency than the top grating. In other words, it has more cycles within unit given space. The top grating shows a relatively lower spatial frequency; it has fewer cycles per space.

(22)

It is common to refer to the spatial frequency of a grating in terms of cycles per degree rather then cycles per space. The number of cycles per degree of visual angle is specified rather than the number of cycles per unit of space. As we shall see, the specification of spatial frequency in terms of cycles per degree offers many practical advantages.

Fig. 2.1. A low frequency spatial grating (top) and a higher frequency grating (bottom). Both gratings have the same contrast.

(23)

2.2.2. Characteristics of Contrast

The top of Fig. 2.2 shows a grating of low contrast along with its luminance profile. The bottom of this figure shows a grating of the same spatial frequency but of a higher contrast. The dashed line across the luminance profiles represents the average luminance of the gratings. Note that the average luminance l_ave for both gratings is the same; however, the grating on the bottom has a greater difference between its peak and the average luminance value. Consequently, it has a higher contrast.

Contrast is often defined by way of a formula: Contrast= ave l l ∆ , (2.1) where l

∆ : The difference in luminance between the peak or trough and the average luminance lave.

ave

l : The average luminance of the grating. (the average of the light peaks and dark troughs)

This formula states that contrast is the ratio of the difference between the peak luminance and average luminance to the average luminance. Although this formula is useful in defining and understanding contrast, it is not very practical for the measurement of contrast. In the laboratory, the scientist may find it easiest to measure the peak luminance l_max and the minimum luminance l_min to calculate the contrast. The formula used is

Contrast= min max min max l l l l + − , (2.2) where l l l_max = _ave+∆ l l lmin = ave−∆

(24)

It can be demonstrated that Eq. (2.2) provides the same result as Eq. (2.1). Substituting for l_max and l_min we have

ave ave ave ave ave l l l l l l l l l l l l l l ∆ = ∆ − + ∆ + ∆ − − ∆ + = + − ) ( ) ( ) ( ) ( min max min max . (2.3)

It should be clear from examining Fig. 2.2 and the preceding formulae that contrast ranges between 0% and 100%. Contrast cannot be greater then 100% because of the physical impossibility of making ∆l greater than l_ave. The trough of the luminance profile is at zero luminance when ∆l=lave. It is not possible to have less

than zero luminance.

l

∆

l

∆

Fig. 2.2. A low contrast spatial grating (top) and a higher contrast grating (bottom). Both grating are of the same spatial frequency.

(25)

2.2.3. The Human Visibility Threshold

It was found that the perception of HVS is more sensitive to luminance contrast rather than the uniform brightness. Fig. 2.3 shows the actual visibility thresholds called just-noticeable-distortion (JND) corresponding to different background luminance and it was verified by a subjective experiment [60]. The experiments were conducted in a dark room and a square area was located in the center of a flat field of constant gray level. Through varying the amplitude of the object, the visibility threshold for each gray level was determined when the object was just noticeable. The ability of human eyes to tell the magnitude difference between an object and its background depends on the average value of background luminance. We can find that the visibility threshold is lower when the background luminance is within the interval from 70 to 150, and the visibility threshold will increase if the background luminance becomes darker or brighter away from this interval. In addition, high visibility threshold will occur when the background luminance is in very dark region.

(26)

2.2.4. The Contrast Sensitivity Function

To determine a human contrast sensitivity function (HCSF), the subject is presented with a sine wave grating of a given spatial frequency. A typical HCSF is shown in Fig 2.4. Note that the coordinates are plotted in log units. The HCSF is a band pass function: it shows distinct peak sensitivity and decreasing sensitivity on either side of this peak. A typical HCSF peaks in the region of 4 cycles/degree. The human visual system is most sensitive to this frequency; it will detect a grating of this frequency at lower contrast levels than it will detect gratings of other frequencies.

Fig. 2.4. Typical human contrast sensitivity function. The arrow points to the cutoff frequency.

(27)

2.3. Structure of the Image Quality Enhancement

System

For image analysis, we make use of the properties of human visual system (HVS) to obtain the features of images. Therefore, we could realize which region would be worth quality enhancement, since human eyes would be usually more sensitive to this region. For sensitive regions, we propose an adaptive neural network to enhance the visual quality to match the characteristics of human visual perception.

2.3.1. HVS-Directed Image Analysis

There are three input variables, visibility degree (VD), structural degree (SD), complexity degree (CD), and one output variable (Mo) in the proposed fuzzy decision system.

2.3.1.1. Visibility Degree (VD)

To apply the human visibility threshold as mentioned is Section 2.2.3, the actual data and approximated nonlinear equation of visibility thresholds corresponding to different background luminance was shown in Fig. 2.5.

BL 008 . 0 BL 03 . 0 66 . 20 e− +e

(28)

Fig. 2.6. The 4×4 sliding (overlapping) window blocks applied to the image quality enhancement.

In order to obtain the input variables corresponding to each sliding block shown in Fig. 2.6, two index parameters called background luminance (BL) and difference (D) are defined at first. BL is the average luminance of the sliding block proposed to approximate the actual background luminance and can be calculated by

∑∑

− = =− × = 2 1 2 1 ) , ( ) , ( 23 1 BL i j j i B j i O , (2.4) where ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = 1 1 1 1 1 2 2 2 1 2 0 2 1 2 2 2 ) , ( ji B , (2.5)

and the denominator 23 in Eq. (2.4) is the weighted sum of all elements in Eq. (2.5) for normalization.

Feature D is the difference between the maximum pixel value and the minimum pixel value in the sliding block and can be calculated by

)) , ( min( )) , ( max( D = O i j − O i j _{. (2.6)}

(29)

A nonlinear function V(BL) is also designed to approximate the relation between the visibility threshold and background luminance (as Fig. 2.5), and can be represented as BL 008 . 0 BL 03 . 0 66 . 20 V(BL)= e− +e . (2.7) The first input variable of the fuzzy decision system, VD, is defined as the difference between D and V (BL) and can be represented as

V(BL) D

VD = − . (2.8) If VD>0, it means the magnitude difference between the object and its background exceeds the visibility threshold and the object is sensible. Otherwise, this object is not sensible.

The other two input variables, SD and CD, are used to indicate whether the pixels in the sliding block own edge structure.

2.3.1.2. Structural Degree (SD)

SD shows if the sliding block is a high contrast region and the pixels in the block can be obviously separated into two clusters. It is calculated by

))) , ( min( )) , ( max( ))] , ( min( -) ) , ( [mean( -) ) , ( mean( -) ) , ( max( SD j i O j i O j i O j i O j i O j i O − = , (2.9) where ) ) , ( mean(O i j =

∑ ∑

= =− 2 1 -2 1 ) , ( 16 1 i j j i O . (2.10) An illustration of Eq. (2.9) is shown in Fig. 2.7. According to Fig. 2.7, Eq. (2.9) can be expressed as σ1−σ2

/

(σ1+σ2 ), where σ1 =max(O(i,j))-mean(O(i,j))

and )σ2 =mean(O(i,j))-min(O(i,j) . So the SD has been normalized to [0,1] and this rule can also be applied to images with different intensity range. If SD is small

(30)

2 σ σ1 1 σ 2 σ (a) (b)

Fig. 2.7. An illustration of the relation between SD parameter and the distribution of pixels in a sliding block.

(close to 0), σ and 2 σ are close (see Fig. 2.7(a)), it means the pixels in the 1

block can be separated into two even clusters. The block may contain edge or texture structure. On the contrary, if SD is a large value, 0<<σ1−σ2 (see Fig. 2.7(b)), it

means pixel number of one cluster and that of the other cluster are not even, thus, the block may contain noise.

2.3.1.3. Complexity Degree (CD)

Fig. 2.8(a) and (b) show a texture structure and a delineated edge structure in a sliding block, respectively. In these two plots, pixel numbers of the two clusters are the same. Therefore, the SD values corresponding to these two structures are close. Since the proposed neural network is used to compensate the sensitive regions such as Fig. 2.8(b), CD input variable based on differential process is employed to tell the delineated edge structure from texture structure. It is calculated by

∑ ∑

= =− − + + + − + + − = 2 1 -2 1 ' ' ' ' ' )] 1 , ( ) 1 , ( ) , 1 ( ) , 1 ( [ ) , ( 4 CD i j j i O j i O j i O j i O j i O , (2.11)

(31)

Sliding block

(a) (b)

Fig. 2.8. Portions of (a) the sliding block containing texture structure, (b) the sliding block containing edge structure.

gray value of the sliding block, O'(i,j) is defined as :

⎪⎩ ⎪ ⎨ ⎧ ≥ = . otherwise , 0 ), ( ) , ( if , 1 ) , ( ' i j O i j mean O O (2.12)

In Eq. (2.11), each pixel in the 4×4 sliding block takes the 4-directional local gradient operation and CD is the summation of the 16 local gradient values. If CD is a large value, it means the block may contain texture structure. On the contrary, if CD is a small value, the block may contain delineated edge structure.

2.3.1.4. Fuzzy Decision System

In the proposed HVS-based Fuzzy decision system, the input variable VD has two fuzzy sets, N (negative) and P (positive). The input variable SD has three fuzzy sets S (small), M (medium), and B (Big). The input variable CD has three fuzzy sets, S (small), M (medium), and B (Big). The membership functions corresponding to VD, SD, and CD are shown in Fig. 2.10(a)−(c), respectively. In order to determine the fuzzy membership functions, seven nature images were used to generate the model. The images were separated into smooth, texture and edge regions by the admission of the majority (seven of ten subjects). Then the ranges of VD, CD and SD proposed in

(32)

Eqs. (2.8), (2.9) and (2.11) corresponding to these regions were evaluated. Finally, the membership functions of VD, CD and SD could be designed according to the distribution ranges of the parameters in these regions, respectively. Mo is the output variable and the membership functions corresponding to Mo are shown in Fig. 2.10(d). It has two fuzzy sets, NN (neural network) and OP (original pixel).

Seven fuzzy decision rules are used in the proposed fuzzy system and represented as follows:

1. If VD is N then Mo is OP 2. If SD is B then Mo is OP 3. If CD is B then Mo is OP

4. If VD is P and SD is S and CD is S then Mo is NN 5. If VD is P and SD is S and CD is M then Mo is NN 6. If VD is P and SD is M and CD is S then Mo is NN 7. If VD is P and SD is M and CD is M then Mo is OP.

VD 0 N P VD µ SD 0.36 S M B 0.4 0.32 SD µ (a) (b) CD 13 S M B 16 10 CD µ 5 10 OP NN Mo MO µ (c) (d)

Fig. 2.9. (a)-(d) Membership functions of fuzzy sets on input variables VD, SD, CD, and output variable Mo, respectively.

(33)

The numerical value of Mo after defuzzification is compared with a threshold value, Th, where Th is preferably set as the value 5 by experiments. When Mo≥ Th, the adaptive neural-network (NN) compensation module with angle evaluation would be chosen; otherwise, the original pixel (OP) value would be used.

2.3.2. Angle Evaluation

As Mo≥ Th, the fuzzy system identifies the reference pixel as sensible delineated edge and the trained adaptive neural-network model is chosen for quality enhancement according to its corresponding edge angle. The angle evaluation is performed to determine the dominate orientation of the sliding block. The flow diagram of angle evaluation is shown in Fig. 2.10 to compute the orientation angle of each neighborhood of the original image pixel. When the orientation angle of O( ji, ) denoted as A( ji, ) is computed, the luminance values of the original pixels nearby

) , ( ji

O are used for the following computations: )) 1 , 1 ( ) , 1 ( 2 ) 1 , 1 ( ( ) 1 , 1 ( ) , 1 ( 2 ) 1 , 1 ( ) , ( + + + + + − + − + − + − + − − = j i O j i O j i O j i O j i O j i O j i Dx , (2.13) )) 1 , 1 ( ) 1 , ( 2 ) 1 , 1 ( ( ) 1 , 1 ( ) 1 , ( 2 ) 1 , 1 ( ) , ( + + + + + + − − − + + − + − − = j i O j i O j i O j i O j i O j i O j i Dy , (2.14) )] ) , ( ) , ( ( [tan 180 ) , ( 1 j i Dx j i Dy j i A =− − π , (2.15) where −1≤i≤2 and −1≤ j≤2.

The obtained angle of each pixel in the sliding window is quantized into eight quantization sectors such as _θ = 22.5×k degrees, where k = 0, 1, ... , 7. Assuming _θ is the most frequently quantized angle in the window; it is regarded as the dominant orientation of the reference edge pixel. The corresponding weighting coefficient W _θ

(34)

Reference Image Pixel's Neighborhood Compute the Quantized Orientations θ of Current

Pixel in the Sliding Block Have All Pixels Been Processed?

To Gather the Most Frequently Occurring Quantized Orientation θ Yes No Move to the Next Pixel

Send the Orientation θ to the Proposed

neural network

Fig. 2.10. Flow diagram of angle evaluation.

2.3.3. Neural-Network-Based Image Compensation

The function of the proposed neural network is to obtain the weights W_θ

defined in Eq. (2.4), where θ represents the quantized dominant orientation of the reference pixel. Thus, the proposed neural network is used to obtain 8 sets of weighting matrices through training. Each weighting matrix W_θ can be represented as ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = − − − − − − − − 22 21 20 1 2 12 11 10 1 1 02 01 00 1 0 12 11 10 1 1 ) , ( w w w w w w w w w w w w w w w w j i W_θ . (2.16)

In order to use supervised learning algorithms to train the proposed neural network, we have to obtain the desired input-output patterns such that the differences of network outputs and the corresponding desired outputs can be used to define the cost function as the goal to minimize. In this section, several clean image portions with dominant orientation are used as training patterns. Assuming a clean image

(35)

portion is denoted as I, the noise-corrupted version of I has been processed by the proposed noise removal method in the first stage and the filtered result is denoted as

'

I . According to Fig. 2.11, let I'(i,j) be the reference pixel, where O(0,0)=I'(i,j), and it is classified as an edge pixel with dominant orientation θ after angle evaluation. The input of the neural network can be defined as IP=θ and the network output is the compensated pixel value of I'(i,j). The pixel value of

) ,

I( ji obtained from the clean original image is used as the desired output of the neural network for training.

When the input-output patterns are given, the following task is to train a neural network to match the input-output relations. A new neural network as shown in Fig. 2.11 is proposed for image compensation. It is a 4-layer network with two hidden layers. The input layer consists of one node corresponding to θ . The second layer (1st hidden layer) consists of M nodes denoted as g(z_i), where M is 200 in our experiments, and the bipolar sigmoid function is used as the activation function. The weighting vector between the first and the second layer is denoted as U . The third layer (2nd hidden layer) includes 16 nodes and the bipolar sigmoid function is also used as the activation function. The weighting vector between the second and the third layers is denoted as V . The output value of each node in the third layer is denoted as

) (s_i

y and represents an element of the weighting matrix W_θ given in Eq. (2.16), wherey(si)=wjk,i=4(j+1)+k+2, 1≤ i ≤ 16, -1≤ j ≤ 2, and -1 ≤ k ≤ 2. The fourth

layer is the output layer with one output node and its output value represents the compensated pixel value of I' (i,j). The vector between the third and the fourth layers is denoted as '

I_θ. It represents the vector of the sixteen neighborhood pixels of the reference pixel I'(i,j) with dominant orientation θ as follows:

(36)

U

' 1 I_θ

V

1

u

M

u

16 M

v

11 v ) (s1 y ) (s2 y ) (s3 y

M

θ

_Y ' 2 I_θ ' 3 I_θ ' 16 I_θ ) , ( I i j D = ) (s16 y '

I

θ ) (z1 g ) (z2 g ) (z3 g ) (zM g

M

Fig. 2.11. The proposed feedforward neural network for image quality enhancement.

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ + + − − + − + − − − = ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = ) 2 , 2 ( I : ) , 1 ( I ) 1 , 2 ( I ) 1 , 1 ( I ) 1 , ( I ) 1 , 1 ( I I : I I I I I I ' ' ' ' ' ' 16 ' 5 ' 4 ' 3 ' 2 ' 1 ' ' j i j i j i j i j i j i θ θ θ θ θ θ θ . (2.17)

Then the system estimation output can be calculated by

' 16 1 I ) ( x x x s y Y

∑

_θ = ⋅ = , (2.18)

and the corresponding desired output D can be obtained by

) , ( I i j

D= . (2.19) It should be noted that the weighting vectors need to be updated in the training stage are only U and V . If a reference pixel I'(i,j) is given, the neighborhood

pixel vector '

(37)

compensation. This unique operating rule is the major difference between the proposed neural network and the common feed-forward neural networks and is specially designed for the image-compensation application.

In the training stage, the updating rules of weights, v_ab∈V , u_b∈U, can be derived by the back-propagation learning method as

) ( ] 2 / )) ( 1 ))( ( 1 ( I )[ ( ) ( ) 1 ( ab 'a a a b ab t v t D Y y s y s g Z v + = +η − _θ + − × , (2. 20)

[

]

, 2 / )) ( 1 ))( ( 1 ( ] 2 )) ( 1 ))( ( 1 ( ) (I ) [( ) ( ) 1 ( ' 16 1 IP Z g Z g v s y s y Y D t u t u b b ib i i i i b b − + × ⎭ ⎬ ⎫ ⎩ ⎨ ⎧ ₋ + − + = +

∑

= θ η (2.21) where η is the learning constant which determines the rate of learning.

Thirty nature images were used to train the proposed NN for image compensation. The edge regions in these training images are separated into 8 different quantized angles. The variations may be caused by the quantization error (11.25o) and the characteristics of different images and regions. In addition, the vector between the third and the fourth layers of the neural network for image quality enhancement represents the sixteen neighborhood pixels of the reference pixel and it is the filtered results of the first stage (noise removal). This will also cause the variation and nonlinearity in the training. In order to reduce the cost function (MSE) to 1% of the intensity range, i.e. 255×0.01≅2.5, 200 nodes in the first hidden layer were required to achieve this goal with the learning rate η= 0.2 in our experiments. However, if we release the goal (MSE) to achieve from 2.5 to 5, the hidden nodes in the first hidden layer can be reduced to 80 without affecting the visual quality heavily. When the training process is finished, 8 different input values, θ , can be inputted to the trained network, and the corresponding weighting matrices W can be obtained to build a _θ

(38)

look up table combined with Eq. (2.4) for image compensation to reduce the computational cost.

Table 2.1 shows the obtained weighting matrices for the 8 orientations (W1~W8).

W1 and W3 look the rotated versions of W5 and W7, respectively. W2, W4, W6 and W8 look the rotated version of each other. It means the proposed neural network can

catch the directional characteristics of edges through automatic training.

Table 2.1 The eight weighting matrices obtained from the trained neural network for image quality enhancement.

W1 )(0o W2 (22.5o) ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − − 00179080 . 0 00104945 . 0 0.00009103 0.00060843 00184474 . 0 00078190 . 0 00081938 . 0 00004062 . 0 25007752 . 0 24900746 . 0 24986401 . 0 25111893 . 0 00004264 . 0 00056130 . 0 00069366 . 0 00270896 . 0 ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − − − − − − 01947648 . 0 07172672 . 0 01565117 . 0 05561969 . 0 04396327 . 0 03941444 . 0 06377091 . 0 19398496 . 0 12049203 . 0 23237932 . 0 24242122 . 0 11473297 . 0 20418046 . 0 3393475 . 0 12737456 . 0 03263883 . 0 W3 )(45o W4 (67.5o) ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − − − − − − 00070582 . 0 02140568 . 0 05580725 . 0 17292993 . 0 02869632 . 0 05967028 . 0 16688901 . 0 16927022 . 0 05673975 . 0 16765814 . 0 16861688 . 0 154362 . 0 . 0 17003769 . 0 17198606 . 0 03348986 . 0 0.01068067 ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − − − − − − 01901695 . 0 04451190 . 0 12011293 . 0 20509279 . 0 07309391 . 0 04103377 . 0 23072700 . 0 03393999 . 0 01620973 . 0 06267749 . 0 24180996 . 0 12803707 . 0 05537197 . 0 19406541 . 0 11399747 . 0 03404918 . 0 W5 )(90o W6 (112.5o) ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ −0.00097231 0.00257733 25100192 . 0 00171057 . 0 00040739 . 0 00158754 . 0 24998878 . 0 00183930 . 0 00018051 . 0 00131791 . 0 25283754 . 0 00008099 . 0 00410132 . 0 00225166 . 0 2525427052 . 0 0409030 . 0 ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − − − − − − 05560378 . 0 19355151 . 0 11399426 . 0 03281431 . 0 01579252 . 0 06352055 . 0 24167401 . 0 12803423 . 0 07141998 . 0 04032709 . 0 23192659 . 0 03346962 . 0 02036048 . 0 04483605 . 0 11994475 . 0 20460927 . 0 W7 )(135o W8 (147.5o) ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 25065422 . 0 00049473 . 0 00054735 . 0 00058684 . 0 00108892 . 0 25014690 . 0 00041066 . 0 00043473 . 0 00056688 . 0 00000905 . 0 25005892 . 0 00074467 . 0 00050129 . 0 00055420 . 0 00072645 . 0 25051364 . 0 ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − − − − − − 05482485 . 0 01556357 . 0 07150226 . 0 0200380 . 0 19427589 . 0 06342979 . 0 04002404 . 0 04462188 . 0 11438927 . 0 24146707 . 0 23204653 . 0 12006646 . 0 03285340 . 0 12767382 . 0 03368954 . 0 20489226 . 0

(39)

2.4. Brief Summary

In this chapter, we proposed an HVS-directed neural-network-based image quality enhancement. A fuzzy decision system inspired by the HVS is proposed to classify the input image into human perception sensitive regions and non-sensitive regions. If a pixel is in the perception sensitive region, the proposed neural-network module is applied to this pixel for further compensation. The proposed image quality enhancement system can be combined with any other filtering approach to enhance the visual quality. It can also provide a quite stable performance over a wide variety of images with various noise corrupted probabilities will be discussed in the latter chapter.

(40)

3. Image Impulse Noise Removal

In this chapter, a novel noise removal algorithm to deal with impulse noise is proposed. Firstly, the decision-based recursive adaptive median filter is applied to remove the Salt-Pepper noise and an adaptive two-level neural network noise reduction procedure is applied to remove the random-valued noise, and keep the uncorrupted information well. Then the fuzzy decision rules inspired by human visual system (HVS) described in Chapter 2 is applied to compensate the blur of the edge and the destruction caused by median filter. According to the experiment results, the proposed method is superior to conventional methods in perceptual image quality as well as the clarity and smoothness in edge regions.

3.1. Introduction

As mentioned in Sections 1.3, the common median-type methods suffer from the trade-off between cleanness of noise removal and preserving of edge sharpness. They will suffer from the side effect of median filter such as blur and edge destruction especially for images are highly corrupted. These methods exhibit relatively better performance but require more computation and memory cost. It is desired to improve the quality of noise removal and reduce the time consumption at the same time.

In this chapter, the decision-based recursive adaptive median filter to remove the Salt-Pepper noise is described in Section 3.2, and a novel framework for random- valued noise removal is described in Section 3.3. Since the precise noise detection procedure is the key point in noise reduction, we propose a neural network for noise detection such that various widespread densities of noisy pixels can be distinguished

(41)

from the detail edge pixels well. After suppressing the impulse noise, the image quality enhancement system is applied to compensate the corrupted pixels described in Section 3.4. If a noise-corrupted pixel is in the sensitive region, the proposed neural-network module is applied to this pixel for further compensation. Section 3.5 shows the experimental results with various comparisons to other schemes. Finally, the conclusions are summarized in Section 3.6.

3.2. Salt-Pepper Impulse Noise Removal

3.2.1. System Architecture

In order to remove the noisy images without blurring the edge, we divide the process into two steps as shown in Fig. 3.1. In the first step, the impulse noise is removed cleanly without losing too much detail information and then image quality enhancement is applied to compensate the edge sharpness in the second step.

Fig. 3.2 shows the process in the first step, called the decision-based recursive adaptive median filtering scheme. The proposed system consists of noise range detection, decision-based rules, and the adaptive median filter with two different window sizes excluding from noisy pixels, and the recursive algorithm. The noise range estimation and decision-based rules decide whether the pixel is possibly corrupted by noise or not, and the median filter is applied to the possibly noise-corrupted pixels only. The window size of median filter is chosen depending on

Impulse

Noise

Removal

Image Quality

Enhancement

(42)

how heavily the neighboring pixels are corrupted by noise. Then we can remove the noise entirely without destroying the detail and edges of the image. The recursive algorithm replaces the value of noisy pixel with the processing output of the adaptive median filter before the window shifts to next possible noisy pixel so that we can remove the noise more cleanly when the window shifts to the following positions.

Yes 5*5 Median Filter Yes 3*3 Median Filter No No Change No The End Yes Moving to the Next Pixel If the Pixel Within the Noise

Range?

If the Number of Noisy Pixels >=5 in the

3 * 3 Neighborhood ? Is the Whole Image Processed ? Original Image No Noise Range Detection

(43)

3.2.2. The Strategy of Salt-Pepper Noise Removal

In the first step, our goal is to totally remove the Salt-Pepper noises and keeping the edge and detailed information as much as possible so that we can well compensate the corrupted pixels by using the relationship of the neighborhood in the second step. The techniques used in the first step are introduced as follows.

3.2.2.1. Noise Range Estimation

The noise model of additive Salt-Pepper impulse noise, as mentioned in Section 1.2, is a popular and practical problem in image processing. Since the noise range is random assigned, estimating the distribution range of noise is the first task. As the histogram statistics of nature images are distributed uniformly, we can observe that the boundary between the values of noise and the normal pixels are abrupt as shown in Fig. 3.3. Therefore, we apply the histogram statistics to find the gap thresholds and estimate the noise ranges near the maximal and minimal gray level adaptively. It is calculated by

{

() ( 1) 0 30

}

max − + ≤ ≤ = His i His i i ThL , (3.1)

{

( 1) ( ) 225 255

}

max + − ≤ ≤ = His i His i i ThH , (3.2)

where His(i) is the histogram statistic of noisy image at gray value i , ThH and ThL are the maximum upper and lower gap thresholds, respectively. If the gray level of the pixel is within the noise range, smaller than ThL or greater than ThH, we classify it as a suspected noisy pixel and execute the noise removal process in Fig. 3.2. The initial values of ThH and ThL are 256 and –1 respectively, to deal with the case that only maximum or minimum threshold is existed such as the white noise that only maximum noisy pixels are found.

(44)

(a) (b) (c)

Fig. 3.3. Histogram statistics of Lena with 20% impulse noise within different noise range. (a) range = 5 [(0−4); (251−255)]; (b) range =10 [(0−9); (246−255)]; (c) range = 15 [(0−14); (241−255)].

3.2.2.2. Decision Rules

It is very time consuming and even worse effective to filter all pixels in an image. As a matter of fact, most pixels in the image are uncorrupted and should not be modified. In the detection step, each pixel is classified into the possible noisy pixel or the uncorrupted pixel. The uncorrupted pixels are retained without any modification to avoid blurring caused by unnecessary processing.

In our strategy, the pixels within the detected noise range are regarded as possibly corrupted noisy pixels and processed by the adaptive median filter. In such decision rule, noisy pixels will be completely detected but the original pixels in these intervals will also be misidentified as noisy pixels. Although some pixels are false detected, our algorithm will not destroy the image quality too much. As the false detections of noise are in the smooth regions, gray values of nearby pixels are very close, so the result of median filtering is also very close to the original pixel value. Even if the pixels are false detected in the edge area, our second step − image quality enhancement system will well compensate the jaggy edges.

3.2.2.3. The Noise-Exclusive Adaptive Median Filtering

(45)

blurring with large window sizes (such as 5×5) and suffers in insufficient noise suppression with small window sizes (such as 3×3). Although an image may be degraded with up to 20 or 30 percent impulse noise, the random excited pixels make the distribution of noise uneven. When images are highly corrupted, a large number of noisy pixels may connect into noise blotches, and it will still retain the noise after the processing of median filter with 3×3 window size.

In this chapter, an adaptive median filter is proposed to achieve superior performance of noise suppression and to preserve sufficient image sharpness and detailed information. We first analyze the neighboring 3×3 region of the possible noisy pixel. If there are more than four other possible noisy pixels in this block, it is identified as a highly corrupted region and the 5×5 median filter is applied for processing. Otherwise, the noisy pixel is processed by the 3×3 median filter. The noise-exclusive scheme allows only the clean pixels inside the window to participate in median processing [11].

3.2.2.4. Recursive Method

The recursive algorithm replaces the value of noisy pixel with the processing output before the 3×3 window shifts to the next noisy pixel. It can efficiently reduce the number of noisy pixels and remove the noise more cleanly when the window shifts to the following processing positions. Fig. 3.4 shows the difference between recursive and non-recursive methods. In Fig. 3.4, x_i(n) represent original pixels and y_i(n) represent processed pixels. We can find almost half of pixels in each block have been processed by the recursive method (as shown in Fig. 3.4(b)) such that the number of pixels contaminated by the noise is reduced and the noise can be easily removed by the 3×3 median filter. The drawback of the recursive algorithm is that it

基於人類視覺系統之混合雜訊消除技術

國 立 交 通 大 學

電機與控制工程學系

博

士

論

文

基於人類視覺系統之混合雜訊消除技術

Human-Visual-System-Based Mixed-Noise

Removal Techniques

研 究 生：盧 世 茂

指導教授：林 進 燈

張 志 永

中 華 民 國 九 十 五 年 九 月

基於人類視覺系統之混合雜訊消除技術

Human-Visual-System-Based Mixed-Noise

Removal Techniques

研 究 生：盧世茂 Student：Shih-Mao Lu

指導教授：林進燈 博士 Advisor：Dr. Chin-Teng Lin

張志永 博士

Dr. Jyh-Yeong Chang

國 立 交 通 大 學

電 機 與 控 制 工 程 學 系

博 士 論 文

A Dissertation

Submitted to Department of Electrical and Control Engineering

College of Electrical Engineering and Computer Science

National Chiao Tung University

in partial Fulfillment of the Requirements

for the Degree of

Doctor of Philosophy

in

Electrical and Control Engineering

Septemper 2006

Hsinchu, Taiwan, Republic of China

中華民國九十五年九月

基於人類視覺系統之混合雜訊消除技術

研究生：盧世茂 指導教授：林進燈

張志永

國立交通大學電機與控制工程研究所

摘要

Human-Visual-System-Based Mixed-Noise

Removal Techniques

Abstract

誌謝

Contents

List of Figures

7

×

7

List of Tables

1.

Introduction

1.1.

Motivation

1.2.

Impulse and Gaussian Noise Model

1.3.

Previous Research

1.3.1.

Existing Impulse Noise Removal Algorithms

1.3.2.

Existing Gaussian Noise Removal Algorithms

1.3.3.

Existing Neural and Fuzzy Noise Removal Algorithms

1.3.4.

Existing Human Visual System Algorithms

1.4.

Brief Summary

2.

Human Visual System Based Image Quality

Enhancement

2.1.

Introduction

2.2.

Characteristic of Spatial Vision

2.2.1.

Characteristics of Sine Wave Grating

2.2.2.

Characteristics of Contrast

國立交通大學

研究生：盧世茂

指導教授：林進燈

張志永

中華民國九十五年九月

研究生：盧世茂 Student：Shih-Mao Lu

指導教授：林進燈博士 Advisor：Dr. Chin-Teng Lin

張志永博士

國立交通大學

電機與控制工程學系

博士論文

研究生：盧世茂指導教授：林進燈