Classifier-Augmented Median Filters
for Image Restoration
Jyh-Yeong Chang and Jia-Lin Chen
Abstract—Developed in this paper is a new approach that aug-ments a fuzzy classifier to determine whether or not the operating pixel, centered in the sliding window, should be involved with the impulse noise filtering process. Owing to the inclusion of the fuzzy -nearest neighbor ( -NN) scheme, any central operating pixel that is not noise corrupted can be effectively detected and then left unchanged. Thus, the unnecessary pixel replacement can be avoided and the details and signal structure of the image will be best retained. If the center point is found to be noise corrupted, the proposed classifier-augmented median filter facilitates the filtering action only on a subset of pixels, which are not noise contaminated in the window. Due to this impulse pixel exclusion, the biased es-timation caused from impulses can be eliminated and, thus, ob-tains a better estimation of the center pixel. Experimental results showed that this new approach largely outperformed several ex-isting schemes for image noise removal.
Index Terms—Fuzzy -nearest neighbor ( -NN), image restoration, median filters, nonlinear filters.
I. INTRODUCTION
I
N AN ERA of multimedia communication, image data are playing an ever-increasing role in our daily life. The success in producing an excellent multimedia system hinges heavily on an effective instrumentation and measurement scheme on the image data. Since an image formation and its subsequent pro-cessing are inevitably corrupted by noise, image noise removal becomes one of the key preprocessing techniques of common interests. In particular, when an image is coded and transmitted over a noisy channel, or degraded by electrical sensor noise, degradation often appears as “salt-and-pepper” noise (i.e., pos-itive and negative impulses) [1]. The median, or in a more gen-eral term, order statistic, filter has been widely recognized as an effective technique to remove impulsive noise from images [2]. A median-based approach, however, makes use of only the rank-order information of the input data within the filter window, and it suffers from the shortcoming of ignoring their original temporal-order information. Even though me-dian-based filtering can usually produce effective impulse noise removal, it may destroy the image signal structure and the image details. In order to utilize both rank- and temporal-order information of input data, several classes of combination filters have been developed. These filters integrate the ideas of linear Manuscript received June 6, 2002; revised November 15, 2003. This work was supported in part by the Ministry of Education under Grant EX-91-E-FA06-4-4, Program for Promoting University Academic Excellence, Taiwan, R.O.C.The authors are with the Department of Electrical and Control Engineering, National Chiao Tung University, Hsinchu, Taiwan 300, R.O.C. (e-mail: jychang@cc.nctu.edu.tw).
Digital Object Identifier 10.1109/TIM.2003.822716
and filters by considering the output as a weighted sum of filters concerning both the sample’s rank and temporal position. For example, FIR-median hybrid (FMH) filters [3], filters [4], weighted order statistics (WOS) filters [5], FIR-WOS hybrid (FWH) filters [6], and stack filters [7] were introduced along this line of reasoning. It has been shown that these filters were efficient in their filtering action under Gaussian as well as outlier noise contamination. These approaches, however, are typically implemented across an image uniformly and are apt to modify pixels that were undisturbed by impulse noise. As a result, although they tend to have a better noise suppression ca-pability, they risk the loss of integrity of edge and finer details. The effective removal of impulses is often accompanied by the expense of blurred and distorted image. Therefore, removal of impulse noise while preserving the integrity of the edge and details of the image is an essential issue in image filtering.
Since its inception, fuzzy logic has been highly reputed to be an effective tool in dealing with any physical and/or processing systems with typical uncertainty and intrinsic vagueness. Image noise cancellation is surely one of filtering systems with per-ceivable fuzziness. For example, fuzziness exists within image signal and unwanted noise. Consequently, fuzzy-based image filtering schemes have become the focus of numerous research efforts recently. Comprehensive survey papers can be found in Russo’s seminal contributions [8], [9], in which fuzzy infer-ence ruled by else-action (FIRE) filters [10], fuzzy weight fil-ters [11], and fuzzified classical methods [12] were introduced to our attention. These fuzzy logic-based approaches have pro-duced very promising results because the intrinsic fuzzy nature of image filtering, the vagueness and uncertainty existing in the image data, and the processing steps have been considered in their algorithmic designs.
To best preserve the signal structure in the noise cancella-tion, the pixel value will be estimated only if it is detected to be noise corrupted. This scheme, called outlier noise-cleaning method, matches our intuition closely and has drawn much at-tention [13]–[15] recently. Following this scheme and observing the effectiveness of median filters, we propose in this paper a classifier-augmented median filter for image impulse noise removal. Moreover, on account of the success and suitability of exploiting fuzzy set theory in image noise cancellation de-scribed above, the powerful fuzzy -nearest neighbor ( -NN) algorithm [16] was adopted to determine the similarity between the center operating pixel and its neighborhood in a soft fashion. Based on the gray level distribution of pixels in the window, the fuzzy -NN decision rule will detect the center pixel to be noise corrupted or not. If the center pixel is detected not noise 0018-9456/04$20.00 © 2004 IEEE
contaminated, then the center pixel would bypass the filtering procedure. Otherwise, the median filtering over the subset con-taining noise-free pixels in the window is executed to restore the center pixel. In this way, our scheme can integrate the me-dian filtering with the outlier detection in a flexible and effec-tive manner. The classifier-augmented median filter can remove noise effectively and preserve the edge well. Hence, the undesir-able effect of losing image quality and integrity causing by the nonlinear noise removal of median filters has been minimized through this fuzzy classifier inclusion.
II. IMPULSENOISEMODEL ANDMEDIANFILTERS In the experimental impulsive noise model for an image, the source image was corrupted by additive impulse noise with a probability . The impulses take on positive and negative values with an equal probability 1/2 . This model is also known as a salt-and-pepper noise model. For 8-bit images, if a pixel is corrupted, it is replaced by a positive or negative impulse values. Conventionally, impulse noise assumes 0 for negative impulses and 255 for positive impulses. In a more practical situation, the impulse noise values are fixed, but may vary within a dynamic range [17]. In this study, depending on a negative or positive impulse noise to be added, the corrupted pixel’s gray level were randomly generated between 0 and 10, and between 245 and 255, respectively.
To define the method, it is necessary to denote variables for the median filter. Let and be the input and output pixel of the median filter, respectively; then
median (1)
where is a square window of size 2 1 2 1 given
by .
III. PROPOSEDMETHOD
The order statistic theory is the design rationale behind the generalized median-type approaches for noise removal. Although these order statistic-based approaches demonstrated robust estimation capability and can remove the impulse noise effectively, they also produce the undesirable effect of removing the details and the signal structure of the image, caused mainly by the following two defects.
1) Since these approaches are typically implemented uni-formly across an image, they tend to modify pixels that are undisturbed by impulse noise. In other words, all pixels in an image are processed regardless of impulse noise corrupted or not.
2) The impulses are included in the restoration procedure which in turn translates into a bias estimation. This would certainly degrade the correctness of pixel value estimation and results in a blurred output image.
This paper proposes a median-based filter that was aug-mented with a classification process to reduce the defects above. The following filtering procedure is repeatedly applied
to the raster-scanned pixel ; 0 1 1,
0 1 1 in an image as follows.
Step 1) -NN Definition.
The -NN of an input operating pixel are first de-fined. The raster scanned input pixel was centered with a sliding window mask of size 2 1
2 1 . Each input operating pixel in a sliding window would specify its own -NN by its asso-ciated window mask. In the proposed system, the -NN of the operating pixel are defined to be the pixels inside the associated window, with the center pixel being excluded.
Step 2) Clustering.
As noted in Section II, an image may be prac-tically corrupted by positive and negative impul-sive noises of fixed values but varying within a dy-namic range, the histogram of this image would con-tain two surges concentrated at both ends of the gray level axis. Consequently, the first valley from the right side (the brightest side) of the histogram can be used as a suitable threshold value for distinguishing the positive impulse pixels from the clean ones. On the other hand, the first valley from the left side (the darkest side) can be used as the threshold value for distinguishing the negative impulse pixels from the clean ones. Note that only a rough estimation of the valleys of the histogram is re-quired, since determining the precise location of the valley is relatively difficult. The rough estimation of valleys could be offset by the powerful median fil-tering action done subsequently. With the estimated and , the pixels in the working window can be categorized by the labeling function into the fol-lowing three classes:
if if else
(2)
where and are indices that specify all the pixels inside the sliding window defined in (1). After the classification above, let denote the class that contains the maximal pixel samples, where takes an value of 1, 2, or 3; and is called the
ma-jority class hereafter. It follows from (2) that the
pixel samples labeled in class would contain most of the positive impulse pixels, whereas those la-beled in class would contain mainly negative pulse pixels. It is notable that, to be realistic, the im-pulse occurrence rate of an image median filtering problem should be less than 50% [2, p. 77]. Hence, from a statistical standpoint, the pixel members in classes or altogether should be less than half of the pixel samples in the sliding window, exclu-sive of the central pixel. On account of almost equal chance in the occurrence of positive and negative impulses, the pixels in either class or class would probably be less than 25% of the pixels in the window. Consequently, class is most likely the majority class, whose composing pixel samples are
of good chance to be noise-free. If the noise corrup-tion rate is smaller (than 50%), the chance of to be the majority class will be more enhanced. The prob-able tendency of class to be the majority class still holds for the corrupted case of only positive im-pulses, although not as effective. Naturally, this be-havior also applies to the corrupted case of only neg-ative impulses too.
Step 3) Fuzzy -NN Classification.
Unlike -NN of assigning a sample definitely to a particular class, the sample in a fuzzy -NN al-gorithm [16] is associated with each class a different class membership degree, which, for instance, can be very likely, likely, or unlikely. As a consequence, a sample is not definitely ascribed to a particular class, it rather belongs to each class with a certain degree of confidence. Since fuzzy -NN yields a soft partition of an input sample among the classes of interests and it has demonstrated an improved performance over its crisp counterpart [16], this is the reason why we adopt the fuzzy version of -NN algorithm in this contribution. Turing back to our image filtering al-gorithm, we will apply the fuzzy -NN algorithm with the crisp membership initialization [16] to de-termine to which class, , , or obtained in Step 2), should the center pixel belong. The assigned initial memberships are further influenced by the inverse of the Euclidean distances, i.e., set-ting 2 as in [16], between the nearest neigh-bors and the input pixel. In the final classification phase, the center pixel is assigned to the class that accumulates the maximum membership. It is possible that there will be a tie among classes with the same maximum membership degree being as-signed to . To prevent the arbitrary assignment when a tie condition occurs, the class assignment of the input pixel follows the following priority,
, in this study. After the fuzzy -NN clas-sification process, we assume that the center pixel is categorized to class , , 2, or 3. We have also adopted fuzzy three-nearest neighbor
initialization [16] in our experiments, almost equal
restoration performances have been produced on the various contaminated images tested. Hence, the sim-pler crisp membership initialization was utilized in this research.
Step 4) Restoration.
If the input pixel class equals the majority class , which is very likely class containing noise-free pixels mostly, then the input pixel would share the same property of the majority class, i.e., it is very probably to be also noise-free. Hence, the central pixel is unlikely to be impulse corrupted, and the filtering process can be bypassed by
let-ting . Owing to this bypassing,
the unnecessary replacement can be avoided and de-fect 1) noted earlier can be remedied. Consequently, the image details and signal structure could thus
Fig. 1. Noiseless 2562 256 test images of (a) “Lena,” (b) “Baboon,” and (c) “Peppers.”
be better retained. Otherwise, the operating pixel is not in the majority class and belongs to either class or class . Since both and are outlier classes, the input pixel would probably be an out-lier, i.e., a noise contaminated pixel. Hence we in-troduce the median filtering by replacing the center pixel with the median of the majority class , i.e.,
median and
, where is the sliding window defined in (1). Owing to the median filtering restoration over the majority class, whose members are mostly noise-free, a biased estimation, i.e., de-fect 2) noted earlier, caused from impulsive pixels could be suitably eliminated. The proposed median filtering on the majority class is in spirit similar to the -trimmed mean [18] and center weighted me-dian (CWM) [19] methods.
IV. EXPERIMENTALRESULTS
In this section, the proposed algorithm was considered to restore images corrupted by various impulse noise levels. As shown in Fig. 1, three 8-bit, 256 256 test images, “Lena,” “Baboon,” and “Peppers,” were utilized in our experimental simulations. Several criteria in terms of mean square error (MSE), mean absolute error (MAE), and peak signal-to-noise ratio (PSNR) [13] are provided to evaluate quantitative per-formance of our model in comparison to a number of existing nonlinear techniques. In addition, perceptual assessment visually on the restored images were also given. For brevity, only a sample of the processed “Lena” images containing both smooth regions and detail-rich regions was plotted. In the simulation, the restoration performance were compared on images degraded by positive and negative interval-valued impulse noise, sampled respectively from the range [0, 10] and
TABLE I
MSE, MAE,ANDPSNR COMPARISONSUSINGDIFFERENTFILTERS ONDIFFERENTLYCORRUPTED“LENA” IMAGES
TABLE II
MSE, MAE,ANDPSNR COMPARISONSUSINGDIFFERENTFILTERS ONDIFFERENTLYCORRUPTED“BABOON” IMAGES
TABLE III
MSE, MAE,ANDPSNR COMPARISONSUSINGDIFFERENTFILTERS ONDIFFERENTLYCORRUPTED“PEPPERS” IMAGES
[245, 255] at random, with various probabilities ranging from 0.1 to 0.3.
Quantitative evaluation of the proposed filter was given below. With respect to differently corrupted “Lena” images, Table I reported the MSE, MAE, and PSNR obtained, re-spectively, using the median filter, central-weighted median (CWM) [19], signal dependent-rank order mean (SD-ROM) [13], and the proposed technique acting on a 3 3 and/or a 5 5 window. In all cases, the algorithms are implemented either nonrecursively, denoted as 0, or recursively, denoted as 1, according to the approach which provided the best results. Similarly, thresholds, ’s, 1 4, and central weight coefficient, , were chosen to produce the best restora-tion of the corrupted images for each method. In this study, it was founded that the best thresholds for restoring “Lena”
by SD-ROM were obtained when 8, 20, 40,
and 50, which were also exploited in the restoration of “Baboon” and “Peppers” images. The condition for the best
performance in each case demonstrated certain consistency and this condition was summarized in the comparison table. In the case which deviated from the best condition shown, the superscript was used to specify the difference. For example, the 5 5 nonrecursive CWM restored the contaminated images best when center weight (2 1 center pixels being used) was equal to 2 and this condition, 0, 2, was denoted at the top of the sixth column of Tables I–III. This best condition, however, changed to 1 for “Peppers” images corrupted with 20% and 30% noise rates and it was noted by superscript in Table III. From Table I, SD-ROM reported a very good restoration performance with high reliability, in comparison to the results obtained from median-based approaches. The proposed method was able to significantly outperform the other filters. Simulating on restoring other benchmark images, “Baboon” and “Peppers,” their quantitative performance mea-sures were tabulated in Tables II and III, respectively. Similar to the results obtained in “Lena,” the best performance still
(a) (b)
(c) (d)
(e) (f)
(g) (h)
Fig. 2. (a) Image “Lena,” corrupted by impulse noise ofp = 0.3. Filtered images using (b) 32 3 median, R = 1; (c) 5 2 5 median, R = 0; (d) 3 2 3 CWM;k = 1, R = 1; (e) 5 2 5 CWM; k = 2, R = 0; (f) 3 2 3 SD-ROM, R = 1; (g) 3 2 3 ours, R = 1; (h) 5 2 5 ours, R = 1.
went to the proposed method except SD-ROM restores slightly better than our method in 10% contaminated “Peppers” image. Although the stack and FMH filters were not included in the comparisons, the SD-ROM method [13] has been extensively testified and reported to outperform these filters by 1–4 dB in the PSNR measurements.
Aside from the quantitative performance measures above, we showed several restored “Lena” images for subjective evalua-tion. Fig. 2(a) showed the corrupted version, 0.3, of the original image “Lena” as shown in Fig. 1(a). Under a 3 3
window and/or 5 5 selection, Fig. 2(b)–(h) were the resulting images of Fig. 2(a) by the median, CWM, SD-ROM, and the proposed methods. It can be observed from these simulated im-ages that the 3 3 CWM cannot remove noisy pixels effec-tively, even when the noise rate is as low as 10%. Moreover, 3 3 median-based filtering needed to be implemented recur-sively to enhance its limited noise suppression capability caused by such a small window size being adopted. Although recursive implementation canceled more noise at the price of excessive annoying blurring of the image details and textures. 5 5 me-dian and CWM filters generally demonstrated better noise can-cellation capability than their 3 3 counterparts, but they de-graded the image details and signal structure seriously. CWM filters improve median filters in detail preservation at the cost of a reduced noise suppression ability. Qualitative assessment SD-ROM approach visually, it demonstrated good noise can-cellation and edge-preserving, which is in compliance with the brilliant quantitative figures displayed in the table. As can be seen from these images, our proposed method has been found to largely outperform the other methods both in impulse noise suppression and edge and finer details preservation. It is also to be noted that the new method, SD-ROM as well, avoided very annoying blurring in the output image’s edge and details, which was observed in recursive median and CWM filters. We have re-peated the simulations on “Baboon” and “Peppers” images cor-rupted by varying percentages of impulse noise, our new tech-nique yielded superior restoration results, similar to the “Lena” sample shown above.
Finally, it is worthy to be remarked that efficient me-dian-based schemes for computer-generated “artificial impulse noise” elimination has been shown to work equally well in a “real” noise situation. As illustrated in [17], Kong and Guan has proposed a median-based filter to reduce impulsive noise in an image. Testing on removing synthetic impulse noise added to images, their method outperformed other median based filters, both quantitatively by normalized mean square error measures and qualitatively by visual inspection. To further evaluate the new method in resolving real problems, this method, together with the methods above were applied to remove annoying spots in an image captured from cable TV. Similar to the performance obtained from the artificial noise case, this method performed much better than others in this real-world impulse noise suppression attempt. The median-based approaches to reduce both artificial and real impulse noises are also commonly investigated in the literature. Interested readers may refer to, for example, [20] and [21] for more details on this subject.
V. CONCLUSION
For impulse noise removal from images, a new approach, called classifier-augmented median filtering, is proposed in this paper. The fuzzy -NN decision rule was initially introduced to determine whether the operating pixel belonged to the ma-jority class or not. If the operating pixel was in the mama-jority class, then it was left unchanged because it would probably be a noise-free pixel as the ones in the majority class. However, if the operating pixel was not in the majority class, then it was very
likely to be an outlier, i.e., a corrupted pixel. The operating pixel was then replaced by the median of the majority class. Through this augmentation of fuzzy classifier into the median filter, the image structure and details can be best retained because only a detected outlier was restored. Moreover, the impulse exclu-sive median filtering could further offset a biased estimation of pixel value if the median filter was executed directly. Experi-mentally, the proposed method has been shown to significantly outperform well-known techniques in impulse noise removal and detail preservation of contaminated images. Our proposed scheme can be easily extended to the more general noise re-moval problem contaminated by long-tailed noise distributions.
REFERENCES
[1] K. J. Kerpez, “Minimum mean squared error impulse noise estima-tion and cancellaestima-tion,” IEEE Trans. Signal Processing, vol. 43, pp. 1651–1662, July 1995.
[2] I. Pitas and A. N. Venetasanopoulos, Nonlinear Digital Filters:
Princi-ples and Applications. Boston, MA: Kluwer, 1990.
[3] H. Zhou, B. Zeng, and Y. Neuvo, “Weighted FIR-median hybrid filters for image processing,” in Proc. IEEE Int. Symp. Circuits and Systems, Shenzhen, China, June 1991, pp. 793–796.
[4] F. Palmieri and C. G. Boncelect Jr., “Ll-filters a new class of order statistic filters,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 691–701, May 1989.
[5] N. Himayet and S. A. Kassam, “A structure for adaptive order statis-tics filtering,” IEEE Trans. Image Processing, vol. 3, pp. 192–206, Mar. 1994.
[6] L. Yin and Y. Neuvo, “Fast adaptation and performance characteristics of FIR-WOS hybrid filters,” IEEE Trans. Signal Processing, vol. 42, pp. 1610–1628, July 1994.
[7] J. H. Lin and E. J. Coyle, “Minimum mean absolute error estimation over the class of generalized stack filters,” IEEE Trans. Acoust., Speech,
Signal Processing, vol. 38, pp. 663–678, Apr. 1990.
[8] F. Russo and G. Ramponi, “Fuzzy systems in instrumentation: fuzzy signal processing,” IEEE Trans. Instrum. Meas., vol. 45, pp. 683–689, Apr. 1996.
[9] F. Russo, “Recent advances in fuzzy techniques for image enhance-ment,” IEEE Trans. Instrum. Meas., vol. 47, pp. l428–1434, Dec. 1998. [10] F. Russo and G. Ramponi, “Nonlinear fuzzy operators for image
pro-cessing,” Signal Processing, vol. 38, pp. 429–440, Aug. 1994. [11] Y. Choi and R. Krishinapuran, “A robust approach to image
enhance-ment based on fuzzy logic,” IEEE Trans. Image Processing, vol. 6, pp. 808–825, June 1997.
[12] A. Taguchi, “Removal of mixed noise by using fuzzy rules,” in 1998
IEEE Int. Conf. Knowledge-Based Intelligent Electronic Systems,
Ade-laide, Australia, Apr. 1998, pp. 176–179.
[13] E. Abreu, M. Lightstone, S. K. Mitra, and K. Arakawa, “A new effi-cient approach for the removal of impulse noise from highly corrupted images,” IEEE Trans. Image Processing, vol. 5, pp. 1012–1025, June 1996.
[14] Y. Xu and G. Grebbin, “A robust adaptive estimator for filtering noise in images,” IEEE Trans. Image Processing, vol. 4, pp. 694–699, May 1995.
[15] Y. Xu and E. M. Lai, “Restoration of images contaminated by mixed Gaussian and impulse noise using a recursive minimum-maximum method,” Proc. Inst. Elect. Eng., vol. 145, pp. 264–270, 1998. [16] J. M. Keller, M. R. Gray, and J. A. Givens, “A fuzzyK-nearest neighbor
algorithm,” IEEE Trans. Syst. Man, Cybern., vol. 15, pp. 580–585, July 1985.
[17] H. Kong and L. Guan, “A noise-exclusive adaptive filtering framework for removing impulse noise in digital images,” IEEE Trans. Circuits
Syst. II, vol. 45, pp. 422–428, Mar. 1998.
[18] J. B. Bednar and T. L. Watt, “Alpha-trimmed means and their relation-ship to median filters,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 32, pp. 145–153, Feb. 1984.
[19] S. Ko and Y. H. Lee, “Center weighted median filters and their appli-cations to image enhancement,” IEEE Trans. Circuits Syst., vol. 35, pp. 984–993, Sept. 1991.
[20] J. S. Kim and H. W. Park, “Adaptive 3-D median filtering for restoration of an image sequence corrupted by impulse noise,” Signal Processing:
Image Commun., vol. 16, pp. 657–668, 2001.
[21] L. Yin, R. Yang, M. Gabbouj, and Y. Neuvo, “Weighted median filters: a tutorial,” IEEE Trans. Circuits Syst. II, vol. 43, pp. 157–192, Mar. 1996.
Jyh-Yeong Chang received the B.S. degree in control engineering and the M.S.
degree in electronic engineering from National Chiao Tung University, Hsinchu, Taiwan, R.O.C., in 1976 and 1980, respectively, and the Ph.D. degree in elec-trical engineering from North Carolina State University, Raleigh, in 1987.
From 1976 to 1978, and from 1980 to 1982, he was a Research Fellow with the Chung Shan Institute of Science and Technology (CSIST), Taiwan. Since 1987, he has been an Associate Professor with the Department of Electrical and Control Engineering, National Chiao Tung University. His research interests in-clude fuzzy sets and systems, image processing, pattern recognition, and neural network applications.
Jia-Lin Chen received the B.S. degree in electrical engineering from Feng-Chia
University, Taichung, Taiwan, R.O.C., in 1994, the M.S. and Ph.D. degrees from the Department of Electrical and Control Engineering, National Chiao Tung University, Hsinchu, Taiwan, in 1996 and 2000, respectively.
He was with Century Semiconductor, Inc., Taiwan. His current research in-terests include pattern recognition, fuzzy neural theory, and VLSI design.
