Abstract—In this paper, a novel two-stage noise removal
algo-rithm to deal with impulse noise is proposed. In the first stage, an adaptive two-level feedforward neural network (NN) with a back-propagation training algorithm was applied to remove the noise cleanly and keep the uncorrupted information well. In the second stage, the fuzzy decision rules inspired by the human visual system (HVS) are proposed to classify the image pixels into human per-ception sensitive class and nonsensitive class, and to compensate the blur of the edge and the destruction caused by the median fil-ter. An NN is proposed to enhance the sensitive regions with higher visual quality. According to the experimental results, the proposed method is superior to conventional methods in perceptual image quality as well as the clarity and smoothness in edge regions.
Index Terms—Fuzzy decision system, human visual system
(HVS), impulse noise, neural network (NN), noise removal.
I. INTRODUCTION
I
MAGES are often corrupted by impulse noise due to noisy sensors or channel transmission errors. The objectives of noise removal are to suppress the noise, as well as possibly to preserve the sharpness of edge and detail information. The nonlinear filtering technique—standard median (SM) [1], [2] filter—based on order statistic, has been demonstrated to be generally superior to linear filtering (moving average) in sup-pressing impulse noise. However, the median filter tends to blur fine details and destroy edges while removing out the im-pulse noise. To achieve better performance, the median filter has been modified in many ways, such as weighted median (WM) filters [3], [4], center weight median filters (CWM) [5], adaptive-length median filters [6], recursive medians [7], [8], and the alpha-trimmed mean filters [9]. These were expected to increase the signal preservation but relatively decrease the noise suppression ability. Applying these algorithms altogetherManuscript received August 14, 2006; revised May 30, 2007; accepted July 29, 2007. This work was supported in part by the National Science Council, Taiwan, under Grant NSC 95-2221-E-009-210, Grant NSC 96-2221-E-009-058, and Grant NSC 95-2752-E-009-011-PAE.
S.-F. Liang is with the Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 701, Taiwan, R.O.C. (e-mail: [email protected]).
S.-M. Lu and J.-Y. Chang are with the Department of Electrical and Control Engineering, National Chiao-Tung University, Hsinchu 300, Taiwan, R.O.C. (e-mail: [email protected]; [email protected]).
C.-T. Lin is with the Department of Electrical and Control Engineering/ Department of Computer Science, National Chiao-Tung University, Hsinchu 300, Taiwan, R.O.C., and also with the Brain Research Center, NCTU Branch, University System of Taiwan, Hsinchu 300, Taiwan, R.O.C. (e-mail: [email protected]).
Digital Object Identifier 10.1109/TFUZZ.2008.917297
across the whole image without identification would inevitably remove the uncorrupted detail pixels, destroy the image quality, and cause additional blur.
For that reason, decision-making schemes [10]–[13] were proposed in which only the identified noisy pixels are processed. Luo [14] proposed a method that uses the alpha-trimmed mean only in impulse noise detection, and its value is replaced by a linear combination of its original value and the median of its local window. Besides, the switching schemes [15]–[20] pro-vide adaptive decision to recover the noisy pixels based on several filters that keep noise-free pixels unchanged. Also, the weighting-average linear combinations of nonlinear median-based filters through learning-rule optimization have been pro-posed [21], [22]. Although satisfactory results have been ob-tained, they tend to remove fine details or retain too much of the noise due to undetection or misdetection of the noise [23], [24]. In addition, since the noisy pixels are replaced without taking into account local features, details and edges are not recov-ered satisfactorily, especially when the noise level is high. The thresholding filtering [25], which is composed of new efficient noise detectors, was proposed to prevent the misclassification of noise-free pixels. A trilateral filter combined with an impulse detector that detects the impulse noise according to the local image static was also proposed [26]. The edge-directed noise detection and suppression strategy was proposed to preserve the details and edges [27]. Two-stage approaches that combine noise identification and edge-preserving supplementary have been proposed for trying to remove the noise cleanly and keep the detail information well [28]–[32].
Since neural networks (NNs) have the ability to learn from examples, and fuzzy systems have the ability to deal with un-certainty, they also have a growing number of applications in image noise removal in the past few years [33]–[44]. Zhang
et al. [33] proposed the fuzzy techniques to detect the impulse
noise and to remove the noise based on long-range correlation within different parts of the image. Schulte et al. [34], [35] pro-posed a fuzzy derivative estimation for noise detection and a fuzzy smoothing of neighboring pixels for noise removal. Lee
et al. proposed a fuzzy image filter based on the genetic learning
process [36]. Neurofuzzy systems that combine both the advan-tages of NNs and fuzzy systems have also been applied to noise removal [38]–[44]. Y¨uksel [43] proposed a hybrid filter obtained by appropriately combining a median filter, an edge detector, and a neurofuzzy network in which the internal parameters are adap-tively optimized by training. These methods exhibit relaadap-tively better performance but require more computation and memory 1063-6706/$25.00 © 2008 IEEE
Fig. 1. Procedure diagram of the two-level impulse noise removal.
cost. It is desired to improve the quality of noise removal and reduce the time consumption at the same time.
In this paper, a novel two-stage noise removal algorithm to deal with impulse noise is proposed. An easily implemented NN is designed for fast and accurate noise detection such that var-ious widespread densities of noisy pixels can be distinguished from the detail edge pixels well. After suppressing the impulse noise, the image quality enhancement is applied to compensate the corrupted pixels to improve the visual quality of the resul-tant images. It consists of fuzzy decision rules based on the human visual system (HVS) for image analysis and an NN for image quality enhancement. If a noise-corrupted pixel is in the perception sensitive region, the proposed NN module is applied to this pixel for further quality compensation. According to the experimental results, the proposed two-stage impulse noise re-moval technology is vastly superior to the conventional methods in processing speed as well as quantitative and visual quality of the processed images.
This paper is organized as follows. Section II introduces the integral system architecture of the proposed impulse noise re-moval algorithm. The procedure of impulse noise rere-moval is pre-sented in Section III. The HVS-directed image analysis method and the NN for image compensation are proposed in Section IV. Section V presents the experimental results for demonstration, and Section VI concludes the paper.
Fig. 2. Procedure diagram of the image quality enhancement.
II. SYSTEMARCHITECTURE
Optimal noise removal should delete the visible noise as cleanly as possible and maintain the detail information and natu-ral appearance to obtain a natunatu-ral-looking image. In order to re-move the impulse noise cleanly from input images without blur-ring the edge, we divide the process into two stages—impulse noise removal and image quality enhancement. In the first stage, the impulse noise is removed cleanly without losing too much detail information, and then, the image quality enhancement is applied to compensate the edge sharpness in the second stage.
The first-stage, the two-level NN noise removal procedure, is shown in Fig. 1. Inside the first level, only the noisy pixels identified by the NN detection are processed with the 3× 3 median filter. The second-level noise removal procedure is used to detect and remove the misclassified and the detected but unremoved noise pixels in the first-level noise removal process with an adaptive median filter. The 3× 3 window [see Fig. 3(a)] is applied at this stage to obtain the features corresponding to the pixel P (0, 0) for noise detection. The more detailed discussion is given in Section III.
Fig. 2 shows the schematic block diagram of the second-stage image quality enhancement system. The proposed sys-tem consists of a fuzzy decision module, an angle evalua-tion module, and an adaptive compensaevalua-tion module. A fuzzy decision module based on the HVS classifies each reference pixel O(0, 0) [as shown in Fig. 3(b)] as sensible delineated edge or not. Based on this classification, the proposed adap-tive NN compensation module is applied to the sensible de-lineated edge region. When the adaptive NN compensation is actuated, the angle evaluation module will compute the
Fig. 3. Two sliding (overlapping) window blocks. (a) The 3× 3 window applied to the first stage for impulse noise removal. (b) The 4× 4 window applied to the second stage for image quality enhancement.
dominant orientation of the original image located in the sliding block as the input data of the proposed NN. The 4× 4 window [see Fig. 3(b)] is applied at this stage to obtain the features corresponding to the pixel O(0, 0) for HVS-based image com-pensation. The weighted compensation of O(0, 0) is applied to the noise-corrupted pixel F (m, n) at the position (m, n) in the sensible delineated edge region and can be presented as
F(m, n) = 2 i=−1 2 j =−1 O(i, j)Wθ(i, j) (1)
where Wθis derived from an NN after offline training. The NN
is trained according to the edge angle of the reference image pixel to obtain the corresponding weights.
III. IMPULSENOISEREMOVAL
A. Impulse Noise Model
Impulse noise is when the pixels are randomly misfired and replaced by other values in an image. The image model contain-ing impulse noise can be described as follows:
Xij =
Nij, with probability p
Sij, with probability 1− p
(2) where Sij denotes the noiseless image pixel and Nij denotes
the noise substituting for the original pixel (OP). With the noise ratio p, only p percent of the pixels in the image are replaced and others keep noise uncorrupted. In a variety of impulse noise models for images, fixed- and random-valued impulse noises are mostly discussed. Fixed-valued impulse noise, known as the “salt-and-pepper” noise, is made up of corrupted pixels whose values are replaced with values equal to the maximum or min-imum (255 or 0) of the allowable range with equal probability (p/2). The random-valued impulse noise is made up of corrupted pixels whose values are replaced by random values uniformly distributed in the range within [0, 255]. In this paper, both fixed-and rfixed-andom-valued impulse noises are adopted as the noise model to test the system robustness.
B. NN for Noise Detection
Since the residual noise will strongly affect human perception, precise noise detection is the first important step for the noise
Fig. 4. Proposed NN for noise detection.
removal. It is found that noise is more annoying in smooth and edge areas [45], [46]. Most algorithms work well on low noise-density images but fail to detect noise pixels in the edge region. The decision-based algorithms for noise detection can be di-vided into three types. The first type is to detect whether the pixel is contaminated by noise according to the local features. Florencio et al. [10] proposed a decision measure based on the second-order statistic called normalized deviation to detect the noise by threshold. Zhang et al. [18] proposed a detection tech-nique by four convolutions using the 1-D Laplacian operator. The second-type decision measure considers the differences of adjacent pixel values in the rank-ordered median filtering se-quence [12], [13]. The third-type approach, called switching schemes [15], [16], first applies several types of rank-ordered filters, and then, detects the noise pixels by their relationships with the gray level of the origin pixel.
In this paper, an NN with high precision and capability of dealing with images corrupted by various noise densities is pro-posed for noise detection. It is a 3-layer NN with one hidden layer, as shown in Fig. 4. The input layer consists of three nodes corresponding to the gray-level difference (GD), average back-ground difference (ABD), and accumulation complexity differ-ence (ACD) in the 3× 3 sliding window. The second layer is the hidden layer that consists of six nodes, and the bipolar sig-moid function is used as the activation function. The weighting vectors between the first and second layers, and between the second and third layers, are denoted as S and R, respectively. The output layer includes one node that represents the identified attribution of the pixel: “noise” or “non-noise,” and the bipolar sigmoid function is also used as the activation function. The three features in the input layer are discussed as follows.
1) Gray-Level Difference (GD): The GD represents the
ac-cumulated variations between the central pixel for identification and each surrounding local pixel. It is defined by
GD = 1 i=−1 1 j =−1 (i,j )=(0,0) |P (0, 0) − P (i, j)| (3)
where P (0, 0) is the reference pixel and P (i, j) is the surround-ing local pixel.
The feature GD is mainly considered to detect the noise over a flat area. It is expected that the corrupted pixels would yield much bigger differences as compared with the uncorrupted pix-els. However, the pixels in edge and texture areas will also get
high GD values so that the obscure region identification be-tween the noise, edge, and texture pixels relies on the other two assistant features, ABD and ACD.
2) Average Background Difference (ABD): Averaging the
surrounding pixels as the background luminance (BL) of the sliding block to compare with the central pixel is another assis-tant feature to detect the noise. This feature, called the ABD, representing the overall average variation with the central pixel in the block, is defined by
ABD = P (0, 0)− 1 i=−1 1 j =−1 (i,j )=(0,0) P (i, j) 8 . (4)
The corrupted pixels will yield bigger differences as com-pared with the clean ones. For the pixels in the texture area, the GD value is large but the ABD feature will be small.
3) Accumulation Complexity Difference (ACD):
Accumu-lating the difference between each pixel in the 3× 3 sliding block and its four neighboring pixels as defined next shows the structure information of the block
ACD = 1 i=−1 1 j =−1 |4 × P (i, j) − P (i − 1, j) − P (i + 1, j) − P (i, j − 1) − P (i, j + 1)|. (5) In the edge area, the summation is lower than that in the noise-pixel area, though the GD difference might be similar. So, it provides an assistant feature between the edge and noise pixels.
In order to train the proposed NN for noise detection, the 512× 512 of the gray-scale Lena image with 20% of impulse noise generated uniformly within [0, 255] is used as a reference pattern for training. Also, 3000 noisy pixels and 3000 uncor-rupted pixels uniformly distributed in the image are adopted as the training data. We also establish a noise table corresponding to these 6000 training data as the desired output for supervised training. The desired output for noise pixels is 1, and that for the clean pixels is−1. The goal is to reduce the mean square er-ror (MSE) to 0.1. The backpropagation learning method is used to derive the updating rules of weights. In our experiments, six nodes in the hidden layer are enough to achieve this goal and the learning rate was 0.1. Experimental results show that our NN owns the highest detection precision of other compared methods and our detection procedure also gives a better tradeoff between the undetection and misdetection rates. More detailed results will be demonstrated in Section V.
C. Noise Removal Algorithm
As per the proposed two-level noise removal procedure shown in Fig. 1, after the first level, we can estimate the image noise density to decide whether the second level is necessary or not by the precise detection procedure. By the experiments, we observe that when the noise density is below 10%, only a one-level noise removal process is enough. As the noise density increases, more misidentified and residual noises will occur. In this case, the
second-level noise removal process is necessary to detect and remove the residual noises.
Since the local characteristics may influence the correct-ness of the detection part and the median filter may still retain some noises, the residual noise pixels are detected and removed with an adaptive median filter in the second level. If there are more than 30% noisy pixels in this image, it is identified as a highly corrupted region and the 5× 5 median filter is applied for processing. Otherwise, the noisy pixel is processed by the 3× 3 median filter. The proposed adaptive two-level noise re-moval algorithm is very effective to suppress the impulse noise as well as to preserve the sharpness of edges and detail infor-mation.
IV. IMAGEQUALITYENHANCEMENT
It is well known that conventional median filtering techniques often suffer from blurring details and cause artifacts around edges. In order to compensate the edge sharpness, image quality enhancement is applied to the modified pixels. Since the process of the first stage has removed the visible noise as cleanly as possible, the second stage focuses the image enhancement on the edge region. For image analysis, we make use of the properties of the HVS to obtain the features of images. Therefore, we can realize which region would be worth quality enhancement, since human eyes would be usually more sensitive to this region. For sensitive regions, we propose an adaptive NN to enhance the visual quality to match the characteristics of human visual perception.
A. HVS-Directed Image Analysis
Many researches have been made on discovering the char-acteristics of the HVS for years. The perceptual redundancies inherent in a still image are basically due to the inconsistency in sensitivity of the HVS to stimuli of varying levels of contrast and luminance changes in the spatial domain. The noise is much more annoying to the human perception in the smooth and edge areas that have lower, just noticeable, distortion (JND) values compared with the JND in the texture area [47]. The magnitude difference between the object and the background, as well as different structures of images, also cause different visual per-ceptions for the HVS. In this paper, a novel fuzzy decision system inspired by the HVS is proposed to classify the image into human perception sensitive and nonsensitive regions.
There are three input variables: visibility degree (VD); struc-tural degree (SD); and complexity degree (CD), and one output variable (Mo) in the proposed fuzzy decision system.
1) Visibility Degree (VD): The ability of human eyes to tell
the magnitude difference between an object and its background depends on the BL. Fig. 5 shows the actual visibility thresh-olds called JND corresponding to different BLs, and they were verified by a subjective experiment [47]. The experiments were conducted in a dark room and a square area was located in the center of a flat field of constant gray level. Through varying the amplitude of the object, the visibility threshold for each gray level was determined when the object was just noticeable.
Fig. 5. Visibility thresholds corresponding to different BLs.
According to Fig. 5, we can find that the visibility threshold is lower when the BL is within the interval from 70 to 150, and the visibility threshold will increase if the BL becomes darker or brighter away from this interval. In addition, a high visibility threshold will occur when the BL is in a very dark region.
In order to obtain the input variables corresponding to each sliding block, as shown in Fig. 3(b), two index parameters called BL and difference (D) are defined at first. BL is the average luminance of the sliding block proposed to approximate the actual BL and can be calculated by
BL = 1 23 2 i=−1 2 j =−1 O(i, j)× B(i, j) (6) where B(i, j) = 2 2 2 1 2 0 2 1 2 2 2 1 1 1 1 1 (7)
and the denominator 23 in (6) is the weighted sum of all ele-ments in (7) for normalization. The weighting coefficients of B decrease as the corresponding distance away from the reference pixel increases to estimate the average BL. Feature D is the difference between the maximum and minimum pixel values in the sliding block and can be calculated by
D = max (O(i, j))− min (O(i, j)). (8) A nonlinear function V (BL) is also designed to approximate the relation between the visibility threshold and BL (as Fig. 5), and can be represented as
V (BL) = 20.66e−0.03BL + e0.008BL. (9) The parameter of 20.66 is obtained by substituting 0 for BL in the nonlinear approximation equation by setting the coefficient of e0.008 BL to be 1.
The first input variable of the fuzzy decision system, VD, is defined as the difference between D and V (BL) and can be represented as
VD = D− V (BL). (10)
If VD > 0, it means the magnitude difference between the object and its background exceeds the visibility threshold and the object is sensible. Otherwise, this object is not sensible.
Fig. 6. Illustration of the relation between the SD parameter and the distribu-tion of pixels in a sliding block.
Fig. 7. Portions of (a) the sliding block including texture structure, and (b) the sliding block including edge structure.
The other two input variables, SD and CD, are used to indicate whether the pixels in the sliding block own the edge structure.
2) Structural Degree (SD): SD shows if the sliding block
is a high contrast region, and the pixels in the block can be obviously separated into two clusters. It is calculated by SD
= |max(O(i, j))−mean(O(i, j))−[mean(O(i, j))−min(O(i, j))]| max(O(i, j))−min(O(i, j))) (11) where mean(O(i, j)) = 1 16 2 i=−1 2 j =−1 O(i, j). (12)
An illustration of (11) is shown in Fig. 6. According to Fig. 6, (11) can be expressed as|σ1− σ2|/(σ1+ σ2). So, the SD has
been normalized to [0, 1] and this rule can also be applied to images with a different intensity range. If SD is small (close to 0), and σ2 and σ1 are close [see Fig. 6(a)], it means the pixels
in the block can be separated into two even clusters. The block may contain edge or texture structure. On the contrary, if SD is a large value, 0 |σ1− σ2| [see Fig. 6(b)], it means the pixel
number of one cluster and that of the other cluster are not even; thus, the block may contain noise.
3) Complexity Degree (CD): Fig. 7(a) and (b) shows a
tex-ture structex-ture and a delineated edge structex-ture in a sliding block, respectively. In these two plots, pixel numbers of the two clus-ters are the same. Therefore, the SD values corresponding to these two structures are close. Since the proposed NN is used to compensate the sensitive regions, such as Fig. 7(b), a CD input variable based on the differential process is employed to tell the delineated edge structure from the texture structure. It is
Fig. 8. (a)–(d) Membership functions of fuzzy sets on input variables VD, SD, CD, and output variable Mo, respectively.
calculated by CD = 2 i=−1 2 j =−1 |4O(i, j)− [O(i + 1, j) + O(i− 1, j) + O(i, j + 1) + O(i, j− 1)]| (13) where O(i, j) is the binarized version of O(i, j). Assuming mean(O) is the mean gray value of the sliding block, O(i, j) is defined as
O(i, j) =
1, if O(i, j)≥ mean(O)
0, otherwise. (14)
In (13), each pixel in the 4× 4 sliding block takes the 4-directional local gradient operation and the CD is the summation of the 16 local gradient values. If the CD is a large value, it means the block may contain texture structure. On the contrary, if the CD is a small value, the block may contain delineated edge structure.
In the proposed HVS-based fuzzy decision system, the input variable VD has two fuzzy sets, negative (n) and positive (p). The input variable SD has three fuzzy sets, small (S), medium (M ), and big (B). The input variable CD has three fuzzy sets, S,
M , and B. The membership functions corresponding to the VD,
SD, and CD are shown in Fig. 8(a)–(c), respectively. In order to determine the fuzzy membership functions, seven nature images were used to generate the model. The images were separated into smooth, texture, and edge regions by the admission of the majority (seven of ten subjects). Then, the ranges of VD, CD, and SD proposed in (10), (11), and (13) corresponding to these regions were evaluated. Finally, the membership functions of the VD, CD, and SD could be designed according to the distribution ranges of the parameters in these regions, respectively. Mo is the output variable, and the membership functions corresponding to Mo are shown in Fig. 8(d). It has two fuzzy sets, NN and OP.
Seven fuzzy decision rules are used in the proposed fuzzy system and represented as follows:
1) If VD is N then Mo is OP; 2) If SD is B then Mo is OP; 3) If CD is B then Mo is OP;
4) If VD is P and SD is S and CD is S then Mo is NN;
Fig. 9. Flow diagram of angle evaluation.
5) If VD is P and SD is S and CD is M then Mo is NN; 6) If VD is P and SD is M and CD is S then Mo is NN; 7) If VD is P and SD is M and CD is M then Mo is OP.
The numerical value of Mo after defuzzification is compared with a threshold value, Th, where Th is preferably set as the value 5 by experiments. When Mo ≥ Th, the adaptive NN compensation module with angle evaluation would be chosen; otherwise, the OP value would be used.
B. Angle Evaluation
As Mo≥ Th, the fuzzy system identifies the reference pixel as sensible delineated edge and the trained adaptive NN model is chosen for quality enhancement according to its corresponding edge angle. The angle evaluation is performed to determine the dominant orientation of the sliding block. The flow diagram of angle evaluation is shown in Fig. 9 to compute the orientation angle of each neighborhood of the original image pixel. When the orientation angle of O(i, j) denoted as A(i, j) is computed, the luminance values of the OPs nearby O(i, j) are used for the following computations:
Dx(i, j) = O(i− 1, j − 1) + 2O(i − 1, j) + O(i − 1, j + 1) − (O(i + 1, j − 1) + 2O(i + 1, j) + O(i + 1, j + 1))
(15)
Dy(i, j) = O(i− 1, j − 1) + 2O(i, j − 1) + O(i + 1, j − 1) − (O(i − 1, j + 1) + 2O(i, j + 1) + O(i + 1, j + 1))
(16) A(i, j) =−180 π tan−1 Dy(i, j) Dx(i, j) (17) where−1 ≤ i ≤ 2 and −1 ≤ j ≤ 2.
The obtained angle of each pixel in the sliding window is quantized into eight quantization sectors such as θ = 22.5× k (in degrees), where k = 0, 1, . . . , 7. Assuming θ is the quan-tized angle for most pixels in the window; it is regarded as
Fig. 10. Proposed feedforward NN for image quality enhancement.
the dominant orientation of the reference edge pixel. The cor-responding weighting coefficient Wθ derived from the offline
training NN is adopted for compensation filtering.
C. NN-Based Image Compensation
The function of the proposed NN is to obtain the weights
Wθ defined in (1), where θ represents the quantized dominant
orientation of the reference pixel. Thus, the proposed NN is used to obtain eight sets of weighting matrices through training. Each weighting matrix Wθcan be represented as
Wθ(i, j) = w−1−1 w−10 w−11 w−12 w0−1 w00 w01 w02 w1−1 w10 w11 w12 w2−1 w20 w21 w22 . (18) In order to use supervised learning algorithms to train the proposed NN, several clean image portions with dominant ori-entation are used as training patterns. Assuming a clean image portion is denoted as I, the noise-corrupted version of I has been processed by the proposed noise removal method in the first stage and the filtered result is denoted as I. According to Fig. 10, let I(i, j) be the reference pixel, where O(0, 0) = I(i, j), and it is classified as an edge pixel with dominant orientation θ after angle evaluation. The input of the NN can be defined as IP = θ and the network output is the compensated pixel value of I(i, j). The pixel value of I(i, j) obtained from the clean original image is used as the desired output of the NN for training.
When the input–output patterns are given, the following task is to train an NN to match the input–output relations. A new four-layer NN, as shown in Fig. 10, is proposed for image com-pensation. The bipolar sigmoid function is used as the activation function. The input layer consists of one node corresponding to
θ. The second layer consists of M nodes denoted as g(zi), where
M is 200 in our experiments, and the weighting vector between
the first and the second layer is denoted as U . The third layer includes 16 nodes, and the weighting vector between the sec-ond and the third layers is denoted as V . The output value of each node in the third layer is denoted as y(si) and represents
an element of the weighting matrix Wθ given in (18), where
y(si) = wj k, i = 4(j + 1) + k + 2, 1 ≤ I ≤ 16, −1 ≤ j ≤
2, and−1 ≤ k ≤ 2. The fourth layer is the output layer with one output node, and its output value represents the
compen-Iθ = Iθ 4 Iθ 5 : Iθ 16 = I(i + 2, j− 1) I(i− 1, j) : I(i + 2, j + 2) . (19)
Then, the system estimation output can be calculated by
Y = 16
x= 1
y(sx)· Iθ x (20)
and the corresponding desired output D can be obtained by
D = I(i, j). (21)
It should be noted that the weighting vectors that need to be updated in the training stage are only U and V . If a reference pixel I(i, j) is given, the neighborhood pixel vector ¯Iθof I(i, j) can be regarded as an extra input vector for compensation. This unique operating rule is the major difference between the pro-posed NN and the common feedforward NNs and is specially designed for the image-compensation application.
In the training stage, the updating rules of weights, vab ∈ V
and ub ∈ U, can be derived by the backpropagation learning
method as
vab(t + 1) = vab(t) + η(D− Y )[Iθ a(1 + y(sa))(1− y(sa))/2]
× g(Zb) (22) ub(t + 1) = ub(t) + η 16 i= 1 (D− Y )(Iθ i ) × (1 + y(si))(1− y(si)) 2 vib × [(1 + g(Zb))(1− g(Zb))/2] IP. (23)
where η is the learning constant that determines the rate of learning.
Thirty nature images were used to train the proposed NN for image compensation. The edge regions in these training images are separated into eight different quantized angles. The varia-tions may be caused by the quantization error (11.25◦) and the characteristics of different images and regions. In addition, the vector between the third and fourth layers of the NN for image quality enhancement represents the 16 neighborhood pixels of the reference pixel, and it is the filtered results of the first stage (noise removal). This will also cause the variation and nonlin-earity in the training. In order to reduce the cost function (MSE) to 1% of the intensity range, i.e., 255× 0.01 ∼= 2.5, 200 nodes in the first hidden layer were required to achieve this goal with
TABLE I
COMPARATIVERESULTS OFVARIOUSNOISEDETECTIONALGORITHMSAPPLIED TORANDOM-VALUEDNOISE-CORRUPTEDIMAGESWITH25% NOISEDENSITY
the learning rate η = 0.2 in our experiments. However, if we re-lease the goal (MSE) to achieve from 2.5 to 5, the hidden nodes in the first hidden layer can be reduced to 80 without affecting the visual quality heavily. When the training process is finished, eight different input values, θ, can be inputted to the trained network, and the corresponding weighting matrices Wθ can be
obtained to build a look-up table combined with (1) for image compensation to reduce the computational cost.
V. EXPERIMENTALRESULTS
A. Impulse Noise Detection
We first demonstrate the performance of the proposed NN for noise detection. Three algorithms [12], [13], [18] were im-plemented to compare with our NN and the detection results as shown in Table I. Three detection measures are defined as follows: “Total Correct Classification” [48] means the noisy pixels and uncorrupted pixels are correctly identified. “Undetec-tion” [25] means noise pixels that lead to residual noise are not identified. “Misdetection” [25] means clean pixels are misiden-tified such that an unnecessary filtering operation causes image blurring.
Experimental results show that our NN owns the highest de-tection precision of the other methods and the network is trained by only using the Lena image with 20% of random-valued noise as the training data. For Zhang’s method [18], though the low misdetection can preserve the detail information, a large number of residual noises will damage the image seriously. On the con-trary, for Pok’s method [12], the high misdetection will blur the edge sharpness. Our detection procedure gives a better tradeoff between the undetection and misdetection. Besides, our two-level impulse noise removal construction (see Fig. 1) will fur-ther eliminate the residual noise pixels to get a near noise-free image.
B. Impulse Noise Removal
The performance of our algorithm has been examined on a variety of noise-corrupted testing images corrupted by various noise densities. The peak signal-to-noise ratio (PSNR) defined
TABLE II
COMPARATIVERESULTS INPSNROFDIFFERENTALGORITHMSAPPLIED TO
IMAGE“ELAINE” CORRUPTED BYVARIOUSRATES OFRANDOM-VALUED
IMPULSENOISE
TABLE III
COMPARATIVERESULTS INFOMOFDIFFERENTALGORITHMSAPPLIED TO
IMAGE“ELAINE” CORRUPTED BYVARIOUSRATES OFRANDOM-VALUED
IMPULSENOISE as PSNR = 10 log10 Nn = 12552 N n = 1( ∧ p−p)2 (24)
is used as a quantitative performance indication, where n is the total number of pixels in an image. The numbers indicate the overall image quality with errors between the original and simu-lation results pixel-by-pixel. The figure of merit (FOM) [49] was also utilized to compare edge preservation performances of dif-ferent noise reduction schemes. The FOM [49] ranges between 0 and 1, and is defined by
FOM = 1 max{ ˆN , Nideal} ˆ N i= 1 1 1 + d2 iα (25) where ˆN and Nideal are the number of detected and ideal edge
pixels, respectively, diis the Euclidean distance between the ith-detected edge pixel and the nearest ideal edge pixel, and α is a constant typically set to 1/9. FOM value is unity for ideal edge detection.
Our proposed method is compared with several existing meth-ods, and various images with fixed- or random-valued impulse noises are used for testing. Tables II–IV show the quantitative comparison of the proposed method and the existing methods with respect to images corrupted with random-valued impulse noise. According to these quantitative comparisons, the parti-tion belief median (PBM) [23], trilateral filter [27], and the pro-posed method can produce better and more robust quantitative
Fig. 11. (a) Original image. (b) Lena with 20% of fixed-valued impulse noise. (c) The 3× 3 standard median filter. (d) The 3 × 3 recursive median filter. (e) PBM filter. (f) Li’s edge preserving method with threshold = 32. (g) Trilateral filter with five iterations. (h) Zhang’s fuzzy techniques. (i) Luo’s method. (j) FRINRM. (k) First stage of the proposed method. (l) The proposed two-stage method.
performance than the other methods. In Table II, we can find the trilateral filter [27], the Zhang’s method [33], the Luo’s method [14], and the fuzzy random impulse noise reduction method (FRINRM) [35], and our proposed method can produce better PSNR performance. It can also be found that if the noise rate of the image for compensation is higher than 20%, the sec-ond stage of the proposed method will obviously improve the metric performance. In Table III, we can find that the FOMs of
Fig. 12. (a) Original image. (b) Boat with 25% of random-valued impulse noise. (c) The 3× 3 standard median filter. (d) The 3 × 3 recursive median filter. (e) PBM filter. (f) Li’s edge preserving method with threshold = 32. (g) Trilat-eral filter with five iterations. (h) Zhang’s fuzzy techniques. (i) Luo’s method. (j) FRINRM. (k) First stage of the proposed method. (l) The proposed two-stage method.
the FRINRM [35] are not as good as its PSNR performance. Ac-cording to Table IV, we can find that the PSNR performance of Zhang’s fuzzy approach [33] is not robust and varies depending on the image characteristics.
Fig. 11 shows the experimental results of different noise re-moval techniques applied to the “Lena” with 20% fixed-valued noise. Fig. 12 shows the processed results of “Boat” with 25% random-valued impulse noise. We can find that the Li’s edge-preserving filtering method [28] can retain more edge sharpness but cannot remove the noise very well. The edge-preserving filtering method might even misjudge some noisy pixels as the edge and then increase the size of some noises. Generally, the threshold adjustment for different images is another problem that needs to be solved in these methods. The PBM filter [23] per-forms well in most regions except for the noises along the edge area. In addition, the PBM filter should be trained separately to determine the filter parameter for images with fixed-valued or random-valued noise, respectively. The fuzzy approach [33] can remove the noise well but it destroys some edges and detail infor-mation very much, especially for the images with high-density
noise. In addition, it is very time consuming to find the ultimate remote window for noise cancellation, and the coefficients need to be adjusted depending on the image and noise types. The tri-lateral filter [27] cannot remove the noises along the edges very well, since the linear-type weighted average filtering algorithm is not suitable for removal of nonlinear-type impulse noise. In addition, this solution would require a deep statistical study for the automatic selection of control parameters and the best way to do so is still not clear. Luo’s method [14] and FRINRM [35] also cannot remove the noises along the edges very well and destroy some edges and detail information.
Above all, most of the algorithms cannot find the balance between noise removal and edge sharpness very well, and they meet the same problem that as the noise rate increases, their noise suppression ability decreases due to noise misidentifi-cation, edge destroying, and blurring. Obviously, the proposed method produces effective and robust results over various noise-corrupted images with different noise densities, and achieves a better result both in noise reduction and detail preservation than the other methods. Our HVS-based image enhancement algorithm produces a more visually natural looking image with smooth and sharp edges even though it might lose some su-periority in the quantitative metric compared with the original image. The proposed method applies noise removal to the iden-tified noisy pixels, and the image quality enhancement process is also applied to the noise-corrupted pixels in the visual sensi-tive region only, so the computation time can be greatly reduced. According to all the experimental results, it is demonstrated that the proposed method is superior to the existing methods both in perceptual image quality and time consumption.
VI. CONCLUSION
In this paper, a novel two-stage noise removal algorithm was proposed to deal with impulse noise. In the first stage, a two-level noise removal procedure with NN-based noise detection was applied to remove the noise cleanly and keep the uncor-rupted information as well as possible. In the second stage, a fuzzy decision rule inspired by the HVS was proposed to clas-sify pixels of the image into human perception sensitive and nonsensitive classes. An NN is proposed to enhance the sensi-tive regions to perform better visual quality. According to the experimental results, the proposed method is superior to the conventional methods in perceptual image quality, and it can provide a quite a stable performance over a wide variety of images with various noise densities.
REFERENCES
[1] T. A. Nodes and N. C. Gallagher, “Median filters: Some modifications and their properties,” IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-30, no. 5, pp. 739–746, Oct. 1982.
[2] G. R. Arce and R. J. Crinon, “Median filters: Analysis for two-dimensional recursively filtered signals,” in Proc. Int. Conf. Acoust., Speech, Signal
Process., 1984, pp. 20.11.1–20.11.4.
[3] D. Brownrigg, “The weighted median filter,” Commun. Assoc. Comput., vol. 27, no. 8, pp. 807–818, Mar. 1984.
[4] R. Yang and L. Yin, “Optimal weighted median filtering under structural constrains,” IEEE Trans. Signal Process., vol. 43, no. 3, pp. 591–604, Mar. 1995.
[5] S. J. Ko and Y. H. Lee, “Center weighted median filters and their applica-tions to image enhancement,” IEEE Trans. Circuits Syst., vol. 38, no. 9, pp. 984–993, Sep. 1991.
[6] H. M. Lin and A. N. Willson Jr., “Median filters with adaptive length,”
IEEE Trans. Circuits Syst., vol. 35, no. 6, pp. 675–690, Jun. 1988.
[7] M. P. McLoughlin and G. R. Arce, “Deterministic properties of the re-cursive separable median filter,” IEEE Trans. Acoust., Speech, Signal
Process., vol. ASSP-35, no. 1, pp. 98–106, Jan. 1987.
[8] G. Qiu, “An improved recursive median filtering scheme for image pro-cessing,” IEEE Trans. Image Process., vol. 5, no. 4, pp. 646–648, Apr. 1996.
[9] J. B. Bednar and T. L. Watt, “Alpha-trimmed means and their relation-ship to median filters,” IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-32, no. 1, pp. 145–153, Feb. 1984.
[10] D. A. F. Florencio and R. W. Schafer, “Decision-based median filter using local signal statistics,” in Proc. SPIE Symp. Vis. Commun. Image Process., 1994, vol. 2308, pp. 268–275.
[11] H. Kong and L. Guan, “A noise-exclusive adaptive filtering framework for removing impulse noise in digital images,” IEEE Trans. Circuits Syst.
II, Analog Digit. Signal Process., vol. 45, no. 3, pp. 422–428, Mar. 1998.
[12] G. Pok and J. C. Liu, “Decision-based median filter improved by predic-tions,” in Proc. IEEE Int. Conf. Image Process., 1999, pp. 410–413. [13] E. Abreu and S. K. Mitra, “A signal-dependent rank ordered mean
(SD-ROM) filter—A new approach for removal of impulses from highly cor-rupted images,” in Proc. Int. Conf. Acoust., Speech, Signal Process., May 1995, vol. 4, pp. 2371–2374.
[14] W. Luo, “An efficient detail-preserving approach for removing impulse noise in images,” IEEE Signal Process. Lett., vol. 13, no. 7, pp. 413–416, Jul. 2006.
[15] T. Chen, K. K. Ma, and L. H. Chen, “Tri-state median filter for image de-noising,” IEEE Trans. Image Process., vol. 8, no. 12, pp. 1834–1838, Dec. 1999.
[16] T. Chen and H. R. Wu, “Impulse noise removal by multi-state median filtering,” in Proc. Int. Conf. Acoust., Speech, Signal Process., Jun. 2000, vol. 4, pp. 2183–2186.
[17] H. L. Eng and K. K. Ma, “Noise adaptive soft-swtching median filter,”
IEEE Trans. Image Process., vol. 10, no. 2, pp. 242–251, Feb. 2001.
[18] S. Zhang and M. A. Karim, “A new impulse detector for switching median filters,” IEEE Signal Process. Lett., vol. 9, no. 11, pp. 360–363, Nov. 2002. [19] Z. Wang and D. Zhang, “Progressive switching median filter for the re-moval of impulse noise from highly corrupted images,” IEEE Trans.
Circuits Syst. II, Analog Digit. Signal Process., vol. 46, no. 1, pp. 78–80,
Jan. 1999.
[20] P. E. Ng and K. K. Ma, “A switching median filter with boundary dis-criminative noise detection for extremely corrupted images,” IEEE Trans.
Image Process., vol. 15, no. 6, pp. 1506–1516, Jun. 2006.
[21] C. T. Chen and L. G. Chen, “A self-adjusting weighted median filter for removing impulse noise in image,” in Proc. IEEE Int. Conf. Image
Process., Sep. 1996, vol. 1, pp. 419–422.
[22] G. R. Arce and J. L. Paredes, “Recursive weighted median filters admitting negative weights and their optimization,” IEEE Trans. Signal Process., vol. 48, no. 3, pp. 768–779, Mar. 2000.
[23] T. Chen and H. R. Wu, “Application of partition-based median type filters for suppressing noise in images,” IEEE Trans. Image Process., vol. 10, no. 6, pp. 829–836, Jun. 2001.
[24] Z. Ma, H. R. Wu, and D. Feng, “Partition-based vector filtering technique for suppression of noise in digital color images,” IEEE Trans. Image
Process., vol. 15, no. 8, pp. 2324–2342, Aug. 2006.
[25] K. Kondo, M. Haseyama, and H. Kitajima, “An accurate noise detector for image restoration,” in Proc. Int. Conf. Image Process., Sep. 2002, vol. 1, pp. I-321–I-324.
[26] I. Aizenberg, C. Butakoff, and D. Paliy, “Impulsive noise removal using threshold Boolean filtering based on the impulse detecting functions,”
IEEE Signal Process. Lett., vol. 12, no. 1, pp. 63–66, Jan. 2005.
[27] R. Garnett, T. Huegerich, C. Chui, and W. He, “A universal noise removal algorithm with an impulse detector,” IEEE Trans. Image Process., vol. 14, no. 11, pp. 1747–1754, Nov. 2005.
[28] X. Li and M. Orchard, “True edge-preserving filtering for impulse noise removal,” in presented at the 34th Asilomar Conf. Signals, Syst. Comput., Pacific Grove CA, Oct. 2000.
[29] R. H. Chan, C. Hu, and M. Nikolova, “An iterative procedure for removing random-valued impulse noise,” IEEE Signal Process. Lett., vol. 11, no. 12, pp. 921–924, Dec. 2004.
[30] R. H. Chan, C.-W. Ho, and M. Nikolova, “Salt-and-pepper noise removal by median-type noise detectors and detail-preserving regularization,”
[35] S. Schulte, V. De Witte, M. Nachtegael, D. Van der Weken, and E. E. Kerre, “Fuzzy random impulse noise reduction method,” Fuzzy
Sets Syst., vol. 158, no. 3, pp. 270–283, Feb. 2007.
[36] C. S. Lee, S. M. Guo, and C. Y. Hsu, “Genetic-based fuzzy image filter and its application to image processing,” IEEE Trans. Syst., Man Cybern.,
B, Cybern., vol. 35, no. 4, pp. 694–711, Aug. 2005.
[37] Y. Nie and K. E. Barner, “Fuzzy rank LUM filters,” IEEE Trans. Image
Process., vol. 15, no. 12, pp. 3636–3654, Dec. 2006.
[38] F. Russo, “Noise removal from image data using recursive neurofuzzy filter,” IEEE Trans. Instrum. Meas., vol. 49, no. 2, pp. 307–314, Apr. 2000.
[39] F. Russo, “An image enhancement technique combining sharpening and noise reduction,” IEEE Trans. Instrum. Meas., vol. 51, no. 4, pp. 824–828, Aug. 2002.
[40] M. E. Y¨uksel and E. Bes¸dok, “A simple neuro-fuzzy impulse detector for efficient blur reduction of impulse noise removal operators for digital images,” IEEE Trans. Fuzzy Syst., vol. 12, no. 6, pp. 854–865, Dec. 2004. [41] E. Bes¸dok, P. Civicioglu, and M. Alci, “Using an adaptive neuro-fuzzy inference system-based interpolant for impulsive noise suppression from highly distorted images,” Fuzzy Sets Syst., vol. 150, no. 3, pp. 525–543, Mar. 2005.
[42] H. Qin and S. X. Yang, “Nonlinear noise cancellation for image with adaptive neuro-fuzzy inference systems,” Electron. Lett., vol. 41, no. 8, pp. 474–475, Apr. 2005.
[43] M. E. Y¨uksel, “A hybrid neuro-fuzzy filter for edge preserving restoration of images corrupted by impulse noise,” IEEE Trans. Image Process., vol. 15, no. 4, pp. 928–936, Apr. 2006.
[44] P. Civicioglu, “Using uncorrupted neighborhoods of the pixels for im-pulsive noise suppression with ANFIS,” IEEE Trans. Image Process., vol. 16, no. 3, pp. 759–773, Mar. 2007.
[45] B. Azeddine, B. B. Kamel, and B. Abdelouahab, “Low-level vision treat-ments inspired from human visual system,” in Proc. 5th Int. Symp. Signal
Process. Appl., ISSPA 999, Brisbane, Australia, Aug., pp. 313–316.
[46] J. Johnston, N. Jayant, and R. Safranek, “Signal compression based on models of human perception,” in Proc. IEEE, Oct. 1993, vol. 81, no. 10, pp. 1325–1422.
[47] C. H. Chou and Y. C. Li, “A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile,” IEEE Trans.
Fuzzy Syst., vol. 3, no. 3, pp. 467–476, Dec. 1995.
[48] E. S. Hore, B. Qiu, and H. R. Wu, “Adaptive noise detection for image restoration with a multiple window configuration,” in Proc. Int. Conf.
Image Process., Sep. 2002, vol. 1, pp. 329–334.
[49] W. K. Pratt, Digital Image Processing. New York: Wiley, 1977.
Sheng-Fu Liang was born in Tainan, Taiwan, R.O.C.,
in 1971. He received the B.S. and M.S. degrees in con-trol engineering, in 1994 and 1996, respectively, and the Ph.D. degree in electrical and control engineer-ing in 2000, all from National Chiao-Tung University (NCTU), Hsinchu, Taiwan.
From 2001 to 2005, he was a Research Assis-tant Professor in Electrical and Control Engineering at NCTU. In 2005, he joined the Department of Bi-ological Science and Technology, NCTU, where he served as an Assistant Professor. Currently, he is an Assistant Professor in the Department of Computer Science and Information Engineering, National Cheng-Kung University, Tainan, and is also a collabo-rative researcher of the Brain Research Center, NCTU. His current research interests include biomedical engineering, biomedical signal/image processing, machine learning, and multimedia signal processing.
Jyh-Yeong Chang (S’84–M’86) received the B.S.
degree in control engineering in 1976 and the M.S. degree in electronic engineering in 1980, both from National Chiao-Tung University (NCTU), Hsinchu, Taiwan, R.O.C., and the Ph.D. degree in electrical engineering from North Carolina State University, Raleigh, in 1987.
During 1976–1978 and 1980–1982, he was a Re-search Fellow at Chung Shan Institute of Science and Technology, Lung-Tan, Taiwan. From 1987, he was an Associate Professor in the Department of Electri-cal and Control Engineering, NCTU, where he is currently a Professor. His current research interests include neural fuzzy systems, video processing and surveillance, and bioinformatics.
Chin-Teng (CT) Lin (S’88–M’91–SM’99–F’05)
re-ceived the B.S. degree from National Chiao-Tung University (NCTU), Hsinchu, Taiwan, R.O.C., in 1986, and the Ph.D. degree in electrical engineer-ing from Purdue University, West Lafayette, IN, in 1992.
He is currently the Chair Professor of Electri-cal and Computer Engineering, Dean of Academic Affairs, and Director of the Brain Research Center at NCTU. He served as the Director of the Research and Development Office of NCTU from 1998 to 2000, the Chairman of the Department of Electrical and Control Engineering of NCTU from 2000 to 2003, the Associate Dean of the College of Electrical Engineer-ing and Computer Science from 2003 to 2005, and the Dean of the College of Computer Science of NCTU from 2005 to 2007. He is the author or coauthor of more than 100 journal papers in the areas of neural networks (NNs), fuzzy systems, multimedia hardware/software, and soft computing, including about 70 IEEE journal papers. He is the coauthor of Neural Fuzzy System—A
Neuro-Fuzzy Synergism to Intelligent Systems (Prentice-Hall, 1996), and the author of Neural Fuzzy Control Systems with Structure and Parameter Learning (World
Scientific, 1994). His current research interests include fuzzy NNs, NNs, fuzzy systems, cellular NNs, neural engineering, algorithms and very large-scale inte-gration (VLSI) design for pattern recognition, intelligent control, and multime-dia (including image/video and speech/audio) signal processing, and intelligent transportation systems.
Prof. Lin is a member of Tau Beta Pi, Eta Kappa Nu, and Phi Kappa Phi honorary societies. He served on the Board of Governors of the IEEE Circuits and Systems (CAS) Society in 2005 and the IEEE Systems, Man, Cybernetics (SMC) Society during 2003–2005. He was the Distinguished Lecturer of the IEEE CAS Society from 2003 to 2005. He was the International Liaison of the IEEE International Symposium of Circuits and Systems (ISCAS) 2005 in Japan, the Special Session Co-Chair of the ISCAS 2006 in Greece, and the Program Co-Chair of the IEEE International Conference on Systems, Man, and Cyber-netics (SMC) 2006 in Taiwan. He has been the President of the Asia Pacific Neural Network Assembly since 2004. He currently serves as the Associate Editor of the IEEE TRANSACTIONS ONCIRCUITS ANDSYSTEMS—I: REGULAR
PAPERSand IEEE TRANSACTIONS ONCIRCUITS ANDSYSTEMS—II: EXPRESS
BRIEFS, IEEE TRANSACTIONS ONSYSTEMS, MAN,ANDCYBERNETICS, IEEE TRANSACTIONS ONFUZZYSYSTEMS, and International Journal of Speech
Tech-nology. He was the recipient of numerous awards, including the Outstanding
Research Award granted by the National Science Council, Taiwan, from 1997 to present, the Outstanding Electrical Engineering Professor Award granted by the Chinese Institute of Electrical Engineering in 1997, the Outstanding En-gineering Professor Award granted by the Chinese Institute of EnEn-gineering in 2000, and the Taiwan Outstanding Information-Technology Expert Award in 2002. He was also elected to be one of the 38th Ten Outstanding Rising Stars in Taiwan in 2000.
