Image enhancement approach using the
just-noticeable-difference model of the
human visual system
Chang-Hsing Lee
Pei-Ying Lin
Ling-Hwei Chen
Wei-Kang Wang
Image enhancement approach using the
just-noticeable-difference model of the human
visual system
Chang-Hsing Lee Chung Hua UniversityDepartment of Computer Science and Information Engineering 707, Section 2, WuFu Road
Hsinchu, Taiwan 30012 Pei-Ying Lin Ling-Hwei Chen Wei-Kang Wang National Chiao Tung University Institute of Multimedia Engineering
1001 University Road Hsinchu, Taiwan 30010 E-mail: [email protected]
Abstract. The low-contrast images taken by digital cameras or camera phones are not always satisfactory due to the limitation of the capturing devices or improper illumination/exposure conditions. Conventional image contrast enhancement methods may either fail to produce satisfactory and undistorted images, or they cannot improve every region of interest appropriately, especially faces. In this paper, a histogram equalization (HE) approach exploiting the just-noticeable-difference (JND) model of the human visual system (HVS), denoted by JND-HE, is proposed for generic image contrast enhancement. Further, the proposed JND-HE approach is combined with the exposure correction (EC) method (denoted by JND-HE-EC) for face image enhancement. The proposed JND-HE-EC approach can improve the contrast in face regions and provide proper illumina-tion in the background. Experimental results on both generic images and faces have shown that our proposed approach can produce more pleasing and appealing enhanced images than other methods. © 2012 SPIE and IS&T. [DOI:10.1117/1.JEI.21.3.033007]
1 Introduction
In recent years, digital cameras and camera phones have been widely used in our daily lives. People may take many unsatisfactory images due to the limitation of the capturing devices or improper illumination/exposure conditions. In addition, the contrast of an image may be greatly affected if the images were taken in bad weather, especially foggy or rainy conditions. The common defects appearing in real-life images include (1) a dark image due to poor illumi-nation, (2) an image with an underexposed background and an overexposed subject, (3) an image with a dark subject and a bright background due to backlighting, and (4) a low-contrast image owing to the influence of fog or
moisture. Many software1,2 or image enhancement meth-ods3–10were developed to cope with these problems. In gen-eral, these methods can be classified into three categories: histogram-based methods,3–6 transform-based methods,3,7,8 and exposure-based methods.9,10
Histogram equalization (HE),3 which is based on the adjustment of histograms, is the best-known and most widely used technique for image contrast enhancement. HE uses a nonlinear mapping function to produce an image with a histogram approximating a uniform distribu-tion. This method will spread out the values that occur more frequently and compress those values that occur less fre-quently to get an image of higher contrast. However, HE fails to produce satisfactory pictures owing to three com-mon drawbacks:11 (1) false contour [see Fig. 1(b)]; (2) amplified noises [see Fig.2(b)]; and (3) washed-out appear-ance [see Fig.3(b)]. Several advanced HE methods4–6have been proposed to improve the image contrast. The global HE methods, such as bi-HE4 and dualistic subimage HE,5first divide the histogram into two parts and then inde-pendently apply HE on each subhistogram. These methods can preserve the original brightness to some extent, which is essential for consumer electronic products, but they may also produce unpleasant artifacts for some images. Pizer et al.6 proposed a local HE method called adaptive HE. First, an image is divided into several blocks, and then HE is applied to each block. Finally, the enhanced image blocks are fused together using bilinear interpolation. The main drawback of these HE-based methods is that some regions in the images may be enhanced excessively. For transform-based methods,3,7,8 a transformation function (e.g., the power-law transformation function or logarithm transformation function) is defined to map an input luminance value into an output one. These methods were widely provided in many commercial devices and
Paper 11195 received Jul. 22, 2011; revised manuscript received Apr. 19, 2012; accepted for publication Jun. 11, 2012; published online Jul. 13, 2012.
Fig. 1 Comparison of different enhancement approaches on an image with overexposed and underexposed regions. (a) Original image; (b) HE-enhanced image; (c) Picasa software—enhanced image; (d) EC-enhanced image; (e) LCC-enhanced image; (f) SC-enhanced image; and (g) JND-HE—enhanced image.
Fig. 2 Comparison of different enhancement approaches on a dark scene. (a) Original image; (b) HE-enhanced image; (c) Picasa software-enhanced image; (d) EC-software-enhanced image; (e) LCC-software-enhanced image; (f) SC-software-enhanced image; (g) JND-HE-software-enhanced image.
software.3 However, some device-dependent parameters need to be determined in advance. Transform-based methods generally can produce a properly enhanced image for either underexposed images or overexposed images by selecting appropriate parameters.3 However, if an image has both underexposed and overexposed regions, transform-based methods fail to produce appropriate contrast on both regions. Moroney7 proposed an enhancement approach based on pixel-wise gamma correction with a nonlinear masking. The gamma correction of each pixel depends on the values of its neighboring pixels. However, it may produce halo effects near the edges. Thus, Schettini et al.8proposed the local con-trast correction method, which is based on a local and image-dependent gamma correction with parameters determined automatically by image statistics analysis, to enhance over-exposed and underover-exposed regions simultaneously. The bilateral filter is used as the mask for gamma correction in an attempt to reduce halo effects. Finally, a contrast enhancement module consisting of stretching, clipping, and saturation preserving is developed to increase the contrast. Although overexposed and underexposed regions can be well enhanced, the global contrast of the whole image also was reduced.
Exposure-based methods9,10 tried to adjust the exposure level of an image using a mapping function between the light values and the pixel values of interested objects. Battiato et al.9proposed a content-dependent exposure cor-rection approach using the camera response curve to adjust the exposure levels. The algorithm first identifies the visually relevant regions and calculates the offset between the average luminance value of visually relevant regions and the desired
target luminance value. The offset value was then employed to modify the luminance value of every pixel in the image. Since this approach was specifically designed for interested regions, it can produce pleasing results in interested regions, but it may lead to poorer illumination in other regions [see Fig. 3(d)]. Safonov et al.10 developed an enhancement method for global and local correction of various exposure defects. Their approach is based on contrast stretching and alpha-blending of both the brightness of the original image and the estimated reflectance.
In this paper, enhancement approaches for real-life images with and without face regions will be proposed. For generic images, an HE approach exploiting the just-noticeable-difference (JND) model of the human visual system (HVS), denoted by JND-HE, will be proposed. For face image enhancement, the proposed JND-HE approach will be combined with the exposure correction (EC) method,9 denoted by JND-HE-EC, to produce a pleasing image with appropriate contrast in background regions and proper illumination in face regions.
In the next section, we will describe the proposed JND-HE image enhancement approach. Section3 gives the pro-posed JND-HE-EC face image enhancement approach. Some experimental results and performance comparisons will be shown in Sec.4. Finally, Sec.5gives a conclusion and a discussion of the future prospects of this work. 2 Proposed Generic Image Enhancement
Approach
In this paper, HE will be integrated with the JND model of the HVS for generic image enhancement. Before describing
Fig. 3 Comparison of different enhancement approaches on a face image taken under backlight conditions. (a) Original image; (b) HE-enhanced image; (c) Picasa software—enhanced image; (d) EC-enhanced image; (e) LCC-enhanced image; (f) SC-enhanced image; and (g) JND-HE-EC-enhanced image.
the proposed approach, we will first give a brief review of HE and analyze its defects.
2.1 Histogram Equalization
HE3tries to increase the contrast of an image using a non-linear mapping function to make the enhanced histogram approximate a uniform distribution. For color images, HE is typically applied to the histogram of the luminance value, Y, converted from the red, green, and blue color values using the following conversion function:
Yðx;yÞ ¼ 0.299 · Rðx; yÞ þ 0.587 · Gðx;yÞ
þ 0.114 · Bðx;yÞ; (1)
where Rðx; yÞ, Gðx; yÞ, and Bðx; yÞ denote the red, green, and blue color values of a pixel located atðx; yÞ. Let variable k represent the luminance value in the interval (0, 255). Then, the occurrence probability of the luminance value k, pðkÞ, in an image is defined as follows:
pðkÞ ¼nk
N; (2)
where N is the total number of pixels in the image, and nkis
the number of pixels with luminance value k. Then, the non-linear transformation function T that maps a luminance value k into a reconstructed luminance value s is given by:
s¼ TðkÞ ¼ 255X k i¼0 pðiÞ ¼ 255X k i¼0 ni N: (3)
From Eq. (3), the difference between the enhanced values of k and kþ r, denoted by TðkÞ and Tðk þ rÞ, can be repre-sented as follows: dEðk;rÞ ¼ Tðk þ rÞ − TðkÞ ¼ 255 × Xkþr i¼0 pðiÞ − 255 ×X k i¼0 pðiÞ ¼ 255 × Xkþr i¼kþ1 pðiÞ: (4)
Thus, the gap dEðk; rÞ between the enhanced values TðkÞ
and Tðk þ rÞ is directly proportional to the accumulative probability between kþ 1 and k þ 1. If some pðiÞ, for i∈ fk þ 1; : : : ; k þ rg, is abnormally large, a larger gap dEðk; rÞ can be obtained. As a result, false contours might
appear in the enhanced image. Specifically, for a dark image in which most pixels have low luminance values, the con-trast-stretching result using HE will excessively enhance the low-luminance part and push the medium-luminance values toward the higher ones. As a result, the noises hidden in the dark regions will be amplified and the whole image exhibits a washed-out appearance. In conclusion, when the histogram of an image has very high peaks at some specific values or concentrates in a small range of levels, HE fails to produce pleasingly enhanced images. To prevent these defects, the JND model of the HVS will be integrated with HE to produce well-enhanced images.
2.2 The Proposed JND-based Histogram Equalization (JND-HE) Approach
Typically, an image enhancement method should not only produce high-contrast images, but also prevent the intro-duction of any noticeable distortion/noise. In the proposed JND-HE approach, we design an evaluation function, based on the JND model of the HVS, to assess whether HE will introduce any noticeable distortion/noise in the enhanced images.
JND is a quantitative measure for distinguishing the lumi-nance change perceived by the HVS. In other words, JND gives the maximum difference of the luminance values that can be perceived by human eyes. In this paper, the JND model proposed by Chou and Li12 will be employed to design the evaluation function. The perceptual function for evaluating the visibility threshold of the JND model can be described by the following equation:
JNDðkÞ ¼
T0½1 − ðk∕127Þ0.5 þ 3; k ≤ 127
γðk − 127Þ þ 3; otherwise; (5)
where k is the luminance value within (0, 255) and the parameters T0andγ depend on the viewing distance between
a tester and the monitor. T0 denotes the visibility threshold
when the background gray level is 0, andγ denotes the slope of the line that models the JND visibility threshold function at higher background luminance. In this study, T0andγ are
set to be 17 and 3∕128, based on the subjective experiments conducted by Chou and Li.12Figure4 shows the visibility threshold for different gray values computed by using Eq. (5). It can be seen that HVS is relatively sensitive to the change of luminance at the medium of the luminance range. In contrast, HVS is less sensitive to the change of lumi-nance for dark or bright regions. In this figure, the smallest visibility threshold is 3 when k is 127.
Note that the visibility threshold evaluated in this study considers only the effect of average background luminance, without taking into account spatial masking (i.e, spatial non-uniformity of the background luminance). In general, if both average background luminance and spatial masking are con-sidered, the evaluated visibility threshold will be larger than that depicted in Fig.4. However, it is impossible to use the spatial masking concept on the HE approach because HE maps pixels of the same value into identical output values without considering their spatial contexts. Thus, to incorpo-rate the JND model into HE directly, we used only the
average background luminance concept to evaluate the vis-ibility threshold.
As shown in Eq. (4), the difference dEðk; rÞ between the
enhanced values TðkÞ and Tðk þ rÞ of two nearby values k and kþ r represents the amplified contrast value resulting from HE. If the difference between k and kþ r is smaller than the JND threshold value evaluated on k [that is, r < JNDðkÞ], k and k þ r will be indistinguishable by human eyes. To avoid excessive contrast stretching, we have to make sure that dEðk; rÞ should also be smaller than the
JND threshold value evaluated on TðkÞ, for 0 ≤ k ≤ 255. This constraint will ensure that for every two nearby lumi-nance values, the corresponding amplified contrast value will be indistinguishable to human eyes. Therefore, we represent the evaluation function in the following form:
dEðk; rÞ < JND½TðkÞ ∀ k; 0 ≤ k ≤ 255; 1 ≤ r ≤ rmax; (6) or equivalently, Xkþr i¼kþ1 pðiÞ < JND½TðkÞ∕255 ∀ k; 0 ≤ k ≤ 255; 1 ≤ r ≤ rmax (7) calculated by using Eq. (4), where rmax is defined as the
smallest visibility threshold evaluated for all luminance values; that is,
rmax¼ min
0≤k≤255JNDðkÞ: (8)
In fact, rmax¼ 3, calculated by using Eq. (5). This JND
evaluation function can help to determine whether smooth regions in the enhanced image will exhibit any perceived dis-tortion or not. In this paper, we will use the evaluation func-tion to determine whether HE can be directly employed for image enhancement. If the JND evaluation function is satis-fied for every luminance value k, HE will be used to obtain an enhanced image without producing any artifacts. Other-wise, the enhanced result will definitely exhibit some notice-able artifacts. Thus, a histogram curve adjustment procedure will be designed to generate a qualified histogram curve before HE. Figure 5 shows the flowchart of the proposed JND-HE approach.
2.3 Histogram Curve Adjustment
In general, it is more probable that the JND evaluation func-tion will be satisfied if the peak values in the histogram are attenuated. Based on this idea, we remove the highest peaks in the histogram and uniformly redistribute the correspond-ing eliminated values among all histogram components. This adjustment will spread the excessively concentrated probabilities at some certain histogram components over the other ones with lower probabilities. The procedure for histogram curve adjustment consists of three stages: sorting, shifting, and redistribution. A detailed description of these three stages is given next.
2.3.1 Sorting
The histogram component, pðkÞ, represents the occurrence probability of luminance value k, 0≤ k ≤ 255. Let L denote the number of histogram components, and let n denote the iteration number of the histogram curve adjustment process conducted so far, with n¼ 1 initially. First, the histogram is sorted in descending order. Let index i denote the index of the ith sorted component. Thus, index(1) and indexðLÞ are the indices having the largest and the smallest probability values, respectively, among all histogram components. 2.3.2 Shifting
First, reset the n largest component values to 0. For example, if n¼ 2, the component values corresponding to indices of index(1) and index(2) are set to 0. That is, p½indexð1Þ ¼ p½indexð2Þ ¼ 0. Next, shift each sorted histogram compo-nent by n values. That is, each sorted compocompo-nent value will be replaced with the next nth sorted component value and the last n values are reset to 0:
pshift n ½indexðiÞ ¼ p½indexði þ nÞ; 1 ≤ i ≤ L − n 0; otherwise : (9) 2.3.3 Redistribution
Since some histogram component values are eliminated in the new shifted histogram, the sum of the remaining compo-nent values will be less than 1. To deal with this problem, we uniformly redistribute the sum of these eliminated compo-nent values to all histogram compocompo-nents using the following equation:
predis
n ðiÞ ¼ pshiftn ðiÞ þ predisavg ; 1≤ i ≤ L; (10)
where predis
avg is the average of the eliminated component
values given by predis avg ¼ 1 L Xn i¼1 p½indexðiÞ: (11)
As a result, the shifted and redistributed histogram will become smoother than the original one. If this histogram satisfies the JND evaluation function defined in Eq. (7), ter-minate the histogram adjustment procedure and apply the HE mapping function [Eq. (3)] on this new histogram to get the enhanced image. Otherwise, set n¼ n þ 1 and perform the shifting and redistribution operation repeatedly until a qua-lified histogram is obtained. It should be noted that the larger the value of the iteration number (n) is, the smoother the his-togram is. This is because as n increases, more hishis-togram peaks will be eliminated. Hence, a qualified histogram that satisfies the JND evaluation function finally can be obtained after a number of iterations. Table1gives an exam-ple of the histogram curve adjustment procedure with eight histogram components. We can see from this that the redis-tributed histogram is smoother than the original one. 2.4 Prevention of Over-adjustment
By repeating the histogram adjustment procedure described above, we can get a qualified image in which no visible distortion will appear in the enhanced image. However, a smoother histogram curve will deteriorate the contrast-stretching ability of HE. In order to avoid this over-adjustment phenomenon, an interpolation histogram can be derived using a weighted average of the shifted histograms obtained in the current iteration and the previous iteration: pint
n½indexðiÞ ¼ ωpshiftn ½indexðiÞ
þ ð1 − ωÞpshift
n−1½indexðiÞ; 1 ≤ i ≤ L; (12)
where 0≤ ω ≤ 1. In this paper, we set pshift
0 ½indexðiÞ ¼
p½indexðiÞ, 1 ≤ i ≤ L. Note that the sum of all component
values in the interpolation histogram is not equal to 1. Thus, the averaged difference will be evenly added to all interpola-tion histogram components to get the corrected interpolainterpola-tion histogram components:
pcint
n ½indexðiÞ ¼ pintn ½indexðiÞ þ pintavg; 1≤ i ≤ L; (13)
where pint
avg is the averaged difference given by
pintavg¼ 1 L 1−X L i¼1 pintn ½indexðiÞ : (14)
In this paper, the binary search algorithm will be used to find the smallest interpolation parameterω between 0 and 1 with which the interpolation histogram will satisfy the JND evaluation function. Finally, HE will be applied to the cor-rected interpolation histogram to obtain the enhanced image. An example showing the results of applying the histogram adjustment procedure is shown in Fig.6. First, an input dark scene image is shown in Fig.6(a). Using Eq. (7) as the JND evaluation function for r¼ 1, we can find that its histogram is not qualified. Thus, the histogram curve adjustment procedure will be conducted iteratively. By repeatedly applying the shift-ing and redistribution operations to the original histogram, we can find a qualified histogram after five iterations (n¼ 5). Figure6(b)through6(g)shows the enhanced results for dif-ferent numbers of adjustment operations (0≤ n ≤ 5). Note that the case for n¼ 0 denotes that no histogram curve adjust-ment operation is conducted before HE. It can be seen that the noises in the background are gradually attenuated and the overexposed areas are greatly improved. Figure6(h) is the final enhanced image obtained by interpolating the histograms corresponding to n¼ 4 and n ¼ 5 with α ¼ 0.6. Figure6(i)
compares the original histogram and the adjusted histogram. We can see that the high peaks in the histogram are greatly attenuated, whereas the other histogram components are increased slightly and evenly.
Table 1 An example illustrating the histogram curve adjustment operation forn ¼ 1, given eight histogram components. Original
histogram Sorted index Sorted histogram Shifted histogram
Redistributed histogram
predis
avg 0.400∕8 0.05
pð0Þ ¼ 0.250 indexð1Þ ¼ 1 p½indexð1Þ ¼ 0.400 p½indexð1Þ ¼ 0.250 p½indexð1Þ ¼ 0.250 þ 0.05 ¼ 0.300 pð1Þ ¼ 0.400 indexð2Þ ¼ 0 p½indexð2Þ ¼ 0.250 p½indexð2Þ ¼ 0.125 p½indexð2Þ ¼ 0.125 þ 0.05 ¼ 0.175 pð2Þ ¼ 0.125 indexð3Þ ¼ 2 p½indexð3Þ ¼ 0.125 p½indexð3Þ ¼ 0.100 p½indexð3Þ ¼ 0.100 þ 0.05 ¼ 0.150 pð3Þ ¼ 0.045 indexð4Þ ¼ 7 p½indexð4Þ ¼ 0.100 P½indexð4Þ ¼ 0.045 p½indexð4Þ ¼ 0.045 þ 0.05 ¼ 0.095 pð4Þ ¼ 0.030 indexð5Þ ¼ 3 p½indexð5Þ ¼ 0.045 P½indexð5Þ ¼ 0.030 p½indexð5Þ ¼ 0.030 þ 0.05 ¼ 0.080 pð5Þ ¼ 0.020 indexð6Þ ¼ 4 p½indexð5Þ ¼ 0.030 p½indexð5Þ ¼ 0.030 p½indexð5Þ ¼ 0.030 þ 0.05 ¼ 0.080 pð6Þ ¼ 0.030 indexð7Þ ¼ 6 p½indexð7Þ ¼ 0.030 p½indexð7Þ ¼ 0.020 p½indexð7Þ ¼ 0.020 þ 0.05 ¼ 0.070 pð7Þ ¼ 0.100 indexð8Þ ¼ 5 p½indexð8Þ ¼ 0.020 p½indexð8Þ ¼ 0.000 p½indexð8Þ ¼ 0.000 þ 0.05 ¼ 0.050
2.5 Linear Image Fusion
Figure7compares the enhanced images using the JND eva-luation function for different values of r. Note that the param-eter r dparam-etermines how many histogram components will be accumulated in the JND evaluation function defined in Eq. (7). The larger the parameter r is, the stricter the JND evaluation function is. That is, a large r value will eliminate a larger number of histogram peaks in the histogram curve adjustment process. As a result, the adjusted histogram becomes flatter, and thus, the subsequent HE will not modify the input image too much. It can be seen that for r¼ 1, the noise around the streetlight is amplified slightly [see Fig.7(a)]. We can also see that for r¼ 3, the noise is attenuated [see Fig. 7(c)], whereas the contrast is reduced as well (see the
red car). In this study, we will perform a linear image fusion on the enhanced images for different r values by using pixel-by-pixel averaging to yield a fused image with low noise and better contrast. Let Y1ðx; yÞ, Y2ðx; yÞ, and Y3ðx; yÞ denote the
enhanced luminance images for r¼ 1, 2, and 3, respectively. The fused image can be obtained using the following equation: YJND-HEðx; yÞ ¼ 1 3Y1ðx; yÞ þ 1 3Y2ðx; yÞ þ 1 3Y3ðx; yÞ: (15) Figure7(d)shows the fused image. It can be seen that the fused image provides a better trade-off between low noise and high contrast.
Fig. 6 An example showing the results of applying the histogram adjustment procedure for different numbers of adjustment operations (n) using the JND evaluation function forr ¼ 1. (a) Original dark scene image; (b) enhanced image for n ¼ 0; (c) enhanced image for n ¼ 1; (d) enhanced image forn ¼ 2; (e) enhanced image for n ¼ 3; (f) enhanced image for n ¼ 4; (g) enhanced image for n ¼ 5; (h) the final enhanced image; (i) the original and final adjusted histogram curves.
2.6 Color Reconstruction
After obtaining the enhanced luminance value YJND-HEðx; yÞ,
the R, G, B color values will be reconstructed by using the following formula to prevent relevant hue shift and color de-saturation:13
Rrðx; yÞ ¼1
2
YJND-HEðx; yÞ
Yðx; yÞ ½Rðx; yÞ þ Yðx; yÞ þ Rðx; yÞ − Yðx; yÞ
; (16) Grðx;yÞ ¼1 2 YJND-HEðx;yÞ
Yðx;yÞ ½Gðx;yÞ þ Yðx;yÞ þ Gðx;yÞ − Yðx;yÞ ; (17) Brðx; yÞ ¼1 2 YJND-HEðx; yÞ
Yðx; yÞ ½Bðx; yÞ þ Yðx; yÞ þ Bðx; yÞ − Yðx; yÞ
; (18)
3 The Proposed Face Image Enhancement Method In real-life images, the face regions are usually the most rele-vant parts in terms of human perception. Conventional image enhancement methods3–8,10,11 do not specifically treat face regions in improper illumination/exposure environments. Hence, these enhancement techniques fail to offer proper con-trast or alleviate improper illumination in face regions. As a result, face regions will exhibit a washed-out appearance or unnatural look [see Fig. 8(b),8(c), and 8(e)]. On the other hand, some enhancement techniques9,14,15 were particularly designed to enhance the face regions. From Fig. 8(d), we can see that the EC method9can produce a satisfying exposure
in face regions, whereas it might cause improper illumination in background regions. Therefore, a face image enhancement approach, which combines our proposed JND-HE approach with the EC method, denoted by JND-HE-EC, will be designed to enhance both face and nonface regions simulta-neously. First, according to the mean luminance value of all detected face skin pixels, EC is employed to obtain an enhanced image, denoted by YEC. Meanwhile, JND-HE is
used to get another enhanced image, denoted by YJND-HE.
Finally, a distance map is constructed to fuse together YEC and YJND-HE to obtain the final enhanced image YJND-HE.
The flowchart of the proposed JND-HE-EC approach for face image enhancement is shown in Fig.9.
3.1 Face Skin Pixel Identification
First, the face regions in the input image are detected using the face detection framework proposed by Viola and Jones,16,17 which can quickly and robustly detect frontal, upright faces. Then, the skin pixels in the detected face regions are identified. In this paper, the skin locus model,18 which is robust under multiple lighting sources or varying illumination environments, is employed to detect all possible skin pixels. This model is based on theðr; gÞ-plane obtained by normal-izing the color values (R, G, B) of each pixel, given by r¼ R∕ðR þ G þ BÞ and g ¼ G∕ðR þ G þ BÞ. From the statisti-cal analysis on the color distribution of skin pixels, Soriano et al. found that in theðr; gÞ-plane, the color values of skin pixels tend to cluster together in a shell-shaped area,18 which is defined by a pair of quadratic functions denoting its upper curve and lower curve:
gupperðrÞ ¼ −1.3767r2þ 1.0743r þ 0.1452 (19)
glowerðrÞ ¼ −0.7760r2þ 0.5601r þ 0:1766: (20)
Fig. 7 Comparison of the enhanced images using the JND evaluation function for different values ofr. (a) r ¼ 1; (b) r ¼ 2; (c) r ¼ 3; and (d) the fused image.
A pixel is detected as a skin pixel if its r and g values satisfy the following constraints:
0.2 < r < 0.6 and glowerðrÞ < g < gupperðrÞ and W > 0.0004;
(21) where W¼ ðr − 0.33Þ2þ ðg − 0.33Þ2is provided to prevent
from labeling whitish pixels as skin pixels. As a result, a face skin map, denoted by Mfs, will be generated to indicate whether
a pixel is a face skin pixel [denoted by Mfsðx; yÞ ¼ 1] or not
[denoted by Mfsðx; yÞ ¼ 0]. Then, the morphological
opera-tion3is conducted on the face skin map to eliminate the noises and fill the holes in the face skin map. Figure10shows some input images, their detected face regions, and identified face skin pixels. We can see that most face regions and face skin pix-els can be correctly detected except for the central face in the rightmost image because its pose is far from upright. Thus, the face detection approach proposed by Viola and Jones16,17fails to detect this face region.
3.2 The EC Method
The EC method9tries to adjust the illumination in an image based on the difference between the average luminance of all reference pixels and the desired luminance value. A simu-lated camera response curve,19showing the relation between a light value q (called“light quantity”) and the pixel value I transformed by the camera sensor, was developed to adjust the exposure level. This camera response curve f can be represented by the following equation:
I¼ fðqÞ ¼ 255
ð1 þ e−AqÞC; (22)
where parameters A and C are used to control the shape of the curve. Thus, the original light quantity q can be derived from a reference luminance value using the inverse transfor-mation of the camera response curve f. In this study, the mean luminance value of all face skin pixels was computed as the reference luminance value. First, the average
Fig. 8 Comparison of different enhancement approaches on a dark scene image using a photoflash. (a) Original image; (b) HE-enhanced image; (c) Picasa software-enhanced image; (d) EC-enhanced image; (e) LCC-enhanced image; (f) SC-enhanced image; and (g) JND-HE-EC-enhanced image.
luminance Yavgof all detected face skin pixels is computed.
The simulated camera response curve f is then used to mea-sure the light quantity difference between Yavgand a
prede-fined ideal luminance value Yideal:
qoffset¼ f−1ðYidealÞ − f−1ðYavgÞ: (23)
The luminance value of each pixel can then be modified using the following equation:
YECðx; yÞ ¼ f
f−1½Yðx; yÞ þ qoffset
(24)
3.3 Estimation of the Distance Map
To fuse together the independently enhanced images YECand
YJND-HE into one image, we use a weighted sum of the two
images. In this paper, the weight for fusing these two images is based on a distance map, denoted by Mdist, which records
the distance between each pixel and its nearest face skin pixel. If a pixel located at ðx; yÞ is close to one of the connected face regions, the distance value recorded in the distance map corresponding to the pixel location ðx; yÞ, Mdistðx; yÞ, is small. On the other hand, if a pixel is far
from any connected face region, a large distance value will be recorded in Mdistðx; yÞ. To construct the distance
map, we first use the region growing algorithm [3] to label each connected face region based on the detected face skin map Mfs. Let there be S connected face regions
in the input image, denoted by Rfs;1, Rfs;2; : : : ; Rfs;S.
Asso-ciated with the ith face region Rfs;i, 1≤ i ≤ S, a
correspond-ing face distance map, denoted by Mdist;i, will be constructed
independently. The overall face distance map Mdist can be
obtained by fusing all individual face distance maps. For each connected face region Rfs;i, we iteratively use the
mor-phological dilation operation3to estimate Mdist;i, 1≤ i ≤ S.
Typically, if a pixel is near a connected face region, it will be dilated earlier. Thus, the iteration number at which the pixel is dilated will be regarded as the face distance be-tween the pixel and a connected face region. For example, if a pixel located at ðx; yÞ is dilated at the tth iteration, set Mdist;iðx; yÞ ¼ t. Note that for the same pixel, the face
dis-tance values recorded in different connected face regions may be distinct. That is, Mdist;iðx; yÞ ≠ Mdist;jðx; yÞ for i ≠ j.
Hence, the overall face distance value, Mdistðx; yÞ, is defined
as the minimum among all face distance maps:
Mdistðx; yÞ ¼ min
1≤i≤SMdist;iðx; yÞ: (25)
In this paper, a maximum number of dilation operations, T, will be defined to reduce the computation time. After T iterations, those pixels that are not dilated yet will be assigned a value of Tþ 1, indicating that they are too far from any connected face region. A detailed description will be given next.
Notation and initialization: Let t denote the number of the dilation operations performed so far, and initially set t¼ 0. Let Rfs;i and Mdist;i, denote the ith connected
face region and the corresponding distance map, respec-tively. Initially, set Mdist;iðx; yÞ ¼ 0 if Mfsðx; yÞ ¼ 1 and
ðx; yÞ∈ Rfs;i; otherwise, set Mdist;iðx; yÞ ¼ T þ 1.
Step 1: Set t¼ t þ 1. Perform the morphological dilation operation on each connected skin region Rfs;i
using a disk structuring element and set t as the dis-tance values of all pixels dilated at the present itera-tion, Mdist;iðx; yÞ ¼ T.
Step 2: If t < T, go to Step 1.
Step 3: Compute the overall distance map Mdist using
Eq. (25).
It should be noted that the face skin map Mfs is scaled
down by a factor of 1∕4. Thus, the face distance map Mdist is also 1∕4 the size of the original image. To fuse
together the independently enhanced images YEC and
YJND-HE, the face skin map must be scaled up to the same
size as its original. In this paper, a bilinear interpolation by a factor of 4 is used for this purpose.
3.4 Fusion of the Independently Enhanced Images We define the fused luminance value of the pixel located at ðx; yÞ, YJND-HE-ECðx; yÞ, as a weighted sum of YECðx; yÞ and
YJND-HEðx; yÞ, given as
YJND-HE-ECðx; yÞ ¼ wðx; yÞYJND-HEðx; yÞ
þ ½1 − wðx; yÞYECðx; yÞ; (26)
where the weight value wðx; yÞ is represented by a power-law function of the distance value Mdistðx; yÞ, given by
Fig. 10 Some example images illustrating the results of face detection and skin pixel identification. The upper row shows the test images and the detected face regions; the bottom row shows the identified face skin pixels.
wðx; yÞ ¼ Mdistðx; yÞ Tþ 1 β : (27)
Therefore, if Mdistðx; yÞ is very small, this enhanced value
tends to close to YECðx; yÞ. We set the power-law curve with
parameterβ ¼ 0.4 in these experiments because such a set-ting can map a narrow range of distance values into a wider range of weight values. As a result, the boundary regions between face and nonface areas will become sharper and
thus will not generate any halo effect. Figure11shows an input image as well as the face skin map, distance map, and weight map associated with the input image. Finally, the reconstructed color image can be obtained using Eqs. (16)– (18) by replacing YJND-HEðx; yÞ with YJND-HE-ECðx; yÞ.
4 Experimental Results
In this paper, some images with the following problems will be used for performance comparison: (1) overexposed
Fig. 11 An example of face region detection, face skin pixel identification, distance map construction, and weighted value evaluation. (a) An input image; (b) detected face regions; (c) a detected face skin map; (d) a morphologically refined face skin map; (e) the distance map with brighter pixels having smaller distance values; and (f) the weight map.
Fig. 12 Comparison of different enhancement approaches on a normal image. (a) Original image; (b) HE-enhanced image; (c) Picasa software— enhanced image; (d) EC-enhanced image; (e) LCC-enhanced image; (f) SC-enhanced image; and (g) JND-HE—enhanced image.
and/or underexposed problems, (2) low-contrast problems, and (3) normal images with proper illumination/exposure. The proposed approach will be compared to some other methods, including (1) HE,3 (2) Picasa software,2 (3) Battiato’s EC algorithm,9(4) Schettini’s local gamma correc-tion (LCC) algorithm,8 and (5) Safonov’s shadow correc-tion (SC) algorithm.10 The comparison of experimental results on generic images, as well as face images, also will be given.
4.1 Experimental Results on Generic Images
Four different kinds of generic images, including a normal image, an image with shadow areas, an image taken in a backlight condition, and a dark scene image will be used for performance comparison.
Figure12(a)shows an original image with proper expo-sure and the enhanced images obtained using different approaches. We can see that HE yields a sharper image than the original [see Fig.12(b)]. From Fig.12(c)and12(f), we can see that the enhanced images using Picasa software and Safonov’s SC algorithm are similar to the original. From Fig.12(d), we can see that the lotus leave area is overexposed using Battiato’s EC algorithm. In Fig.12(e), the global con-trast decreases although the concon-trast in local regions is appro-priate. By comparing all enhanced images, we can see that our JND-HE approach [see Fig.12(g)] yields a high-contrast image that looks more vivid than the others.
Figure13shows an input image with shadow areas and the enhanced images obtained using different approaches. From these images, we can see that our JND-HE approach
and HE produced proper contrast and illumination in the shadow areas.
Figure1 shows an image having both overexposed and underexposed regions, as well as the enhanced images obtained using different approaches. From Fig. 1(b), we can see that HE produces obviously false contours in the overexposed sky area, although the underexposed area (the foreground building) is clearly enhanced. The image enhanced using Picasa software is almost identical to the original [see Fig.1(c)]. In Fig.1(d), Battiato’s EC algorithm has enhanced the dark area appropriately but lost the detail information in the sky area. Schettini’s LCC algorithm and Safonov’s SC algorithm can enhance the shadow areas to some extent, but the global contrast is not appealing [see Fig. 1(e) and 1(f)]. In Fig. 1(g), we can see that JND-HE produces proper illumination and high contrast in the fore-ground building and keeps the details of the sky area.
Figure2 shows a dark scene and the enhanced images obtained using different approaches. From Fig. 2(b), we can see that HE amplifies the background noises and pro-duces a halo effect near the streetlight. Schettini’s LCC algo-rithm also amplifies the background noises and produces the halo effect near the streetlight to some degree [see Fig.2(e)]. The improvement of the enhanced images produced by using Picasa software and Safonov’s SC algorithm is not clear [see Fig. 2(c) and 2(f)]. Battiato’s EC algorithm produces a washed-out appearance and also suffers from an obvious halo effect near the streetlight [see Fig.2(d)]. Our JND-HE method produces obvious improvement in the dark area with-out generating any artifacts [see Fig.2(g)]. Further, we can see that the dark area looks clearer using our JND-HE method.
Fig. 13 Comparison of different enhancement approaches on an image with shadow areas. (a) Original image; (b) HE-enhanced image (c) Picasa software—enhanced image; (d) EC-enhanced image; (e) LCC-enhanced image; and (f) SC-enhanced image; (g) JND-HE—enhanced image.
4.2 Experimental Results on Face Images
Typically, the exposure problems that most frequently occur in images with face regions happen in the following situa-tions: (1) the images were taken at night using a photoflash, and (2) the images were taken in backlight conditions. In the first case, the human faces appear clearly, but the background is unclear since the photoflash gives less illumination on the background. In the second case, the human faces are gener-ally dark and unclear due to relatively insufficient illumina-tion in the foreground face regions. In this paper, some images suffering from these problems will be used for performance comparison.
Figure8shows a face image with dark background and the enhanced images obtained using different approaches. To improve the illumination in the background region, HE also amplifies the background noises and overexposes the face regions [see Fig. 8(b)]. Picasa software and Battiato’s EC algorithm enhance the face regions slightly, but the back-ground is still unclear [see Fig. 8(c) and 8(d)]. From Fig.8(e), Schettini’s LCC algorithm can produce more dis-tinguishable details in the background, but the face regions become unnatural due to insufficient contrast. Safonov’s SC algorithm and our JND-HE-EC approach enhance both the background and face regions [see Fig. 8(f) and 8(g)]. However, we can see that the image enhanced using our JND-HE-EC approach yields a clearer background and proper illumination in face regions.
Figure3shows a face image taken in backlight conditions and the enhanced images obtained using different approaches. In Fig. 3(b), we can see that HE produces a washed-out appearance in the face region, and thus, the face looks unnatural. The images enhanced using Picasa software, Schettini’s LCC algorithm, and Safonov’s LC algorithm exhibit insufficient illumination in the face region [see Fig.3(c),3(e), and3(f)]. In Fig.3(d), we see that Battiato’s EC algorithm enhances the face region appropriately but excessively enhances the background. Our JND-HE-EC method provides a clear face and produces an appropriate background as well.
5 Conclusion and Future Work
In this paper, we have proposed automatic and effective con-trast enhancement approaches for generic images as well as faces. The JND-HE method, which exploits the JND model of the HVS, is proposed for generic image enhancement. An evaluation function based on the JND model is designed to assess whether HE can be employed directly to enhance an input image without introducing any noticeable distortion or artifacts. To improve face images, we proposed the JND-HE-EC method, which combines our JND-HE approach with Battiato’s EC method9to obtain an enhanced image that pro-vides appropriate contrast in face regions and proper illumi-nation in background areas. Experimental results on both generic images and faces have shown that our proposed methods produce more pleasingly enhanced images than other methods, such as HE, Picasa software, Battiato’s EC algorithm, Schettini’s LCC algorithm, and Safonov’s SC algorithm.
Based on the experimental results, we can see that the parameter r used in Eq. (6) will affect the quality of the enhanced images to some degree. A smaller r value will pro-duce a higher-contrast image, whereas the background noise
is also amplified. On the other hand, a larger r value will attenuate the background noise at the cost of reduced con-trast. Thus, a simple pixel-by-pixel averaging of the images enhanced using different r values is employed to provide a better trade-off between low noise and high contrast. In the future, we will try to assign different weights to these images adaptively according to the context and/or contrast analysis on these enhanced images, or by utilizing image fusion tech-niques20,21on these images to produce an image with higher contrast than each individual one.
It can be noted that the JND-HE approach is based only on the luminance values of image pixels, not all color values. Thus, in the near future, we will try to integrate the JND model with the color HE method22 in order to achieve more appealingly enhanced images.
Acknowledgments
This research was supported in part by the National Science Council, R.O.C., under Contract NSC 98-2221-E-009-117-MY2. The authors would like to thank the anonymous reviewers for their valuable comments that improved the representation and quality of this paper.
References
1. Adobe Systems Inc.,“Image editor software-Adobe Photoshop,”http:// www.adobe.com/products/photoshop.
2. Google, Inc.,“Picasa software,”http://picasa.google.com.
3. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2d ed., Prentice-Hall, Upper Saddle River, New Jersey (2002).
4. Y. T. Kim,“Contrast enhancement using brightness preserving bi-histo-gram equalization,”IEEE Trans. Consum. Electron.43(1), 1–8 (1997).
5. Y. Wang, Q. Chen, and B. Zhang,“Image enhancement based on equal area dualistic sub-image histogram equalization method,”IEEE Trans. Consum. Electron.45(1), 68–75 (1999).
6. S. M. Pizer et al.,“Adaptive histogram equalization and its variations,”
Comp. Vision, Graph. Image Proc.39(3), 355–368 (1987). 7. N. Moroney,“Local colour correction using non-linear masking,” in
Proc. IS&T/SID Eighth Color Imaging Conf., pp. 108–111, IS&T, Springfield, VA (2000).
8. R. Schettini et al.,“Contrast image correction method,” J. Electron. Imag.19(2), 023025 (2010).
9. S. Battiato et al.,“Automatic image enhancement by content dependent exposure correction,” EURASIP J. Appl. Sig. Proc. 2004(12), 1849–1860 (2004).
10. I. V. Safonov et al.,“Automatic correction of exposure problems in photo printer,” in Proc. IEEE Tenth Int. Symp. Consum. Electron., pp. 1–6, IEEE, Piscataway, NJ (2006).
11. Z. Y. Chen et al.,“A generalized and automatic image contrast enhance-ment using gray level grouping,” in Proc. IEEE Int. Conf. Acous., Speech, Sig. Proc., Vol. 2, pp. 965–968, IEEE, Piscataway, NJ (2006). 12. C. H. Chou and Y. C. Li,“A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile,”IEEE Trans. Circuits Syst. Video Technol.5(6), 467–476 (1995).
13. S. Sakaue et al.,“Adaptive gamma processing of the video cameras for the expansion of the dynamic range,”IEEE Trans. Consum. Electron.
41(3), 555–562 (1995).
14. F. Naccari et al.,“Natural scenes classification for color enhancement,”
IEEE Trans. Consum. Electron.51(1), 234–239 (2005).
15. C. Shi et al.,“Automatic image quality improvement for videoconfer-encing,” in Proc. IEEE Int. Conf. Acous., Speech, Sig. Proc., Vol. 3, pp. 701–704, IEEE, Piscataway, NJ (2004).
16. P. Viola and M. Jones,“Rapid object detection using a boosted cascade of simple features,” in Proc. IEEE Conf. Comp. Vision Patt. Recog., Kauai, Hawaii, pp. 511–518, IEEE, Piscataway, NJ (2001). 17. P. Viola and M. Jones,“Robust real-time face detection,”Intl. J. Comp.
Vis.57(2), 137–154 (2004).
18. M. Soriano et al.,“Skin color modeling under varying illumination conditions using the skin locus for selecting training pixels,” in Proc. Workshop Real-time Image Seq. Anal., pp. 43–49, Oulu, Finland (2000). 19. S. Battiato, A. Castorina, and M. Mancuso, “High dynamic range imaging for digital still camera: an overview,”J. Elec. Imag. 12(3), 459–469 (2003).
20. Z. Wang et al.,“A comparative analysis of image fusion methods,”
IEEE Trans. Geosci. Remote Sens.43(6), 1391–1402 (2005). 21. G. Pajares and J. M. de la Cruz, “A wavelet-based image fusion
22. E. Pichon, M. Niethammer, and G. Sapiro,“Color histogram equaliza-tion through mesh deformaequaliza-tion,” in Proc. Int. Conf. Image Proc., Vol. 2, pp. 117–120, IEEE, Piscataway, NJ (2003).
Chang-Hsing Lee received his BS and PhD degrees in computer and information science from National Chiao Tung University, Hsinchu, Taiwan, in 1991 and 1995, respectively. He is currently an associate professor in the Department of Computer Science and Information Engineering at Chung Hua University, Hsinchu, Taiwan. His main research interests include audio/music classification, image processing, pattern recognition, and multimedia information retrieval.
Pei-Ying Lin received a BS in computer science and information engineering from National Cheng Kung University, Tainan, Taiwan, in 2005 and an MS at the Institute of Multimedia Engineering from National Chiao Tung University, Hsinchu, Taiwan, in 2007. Since September 2007, she has been working at the Taiwan Semiconductor Manufacturing Company as a digital IC design engineer. Her main research interests include image processing and digital design.
Ling-Hwei Chen received a BS in mathematics and an MS in applied mathematics from National Tsing Hua University, Hsinchu, Taiwan, in 1975 and 1977, respectively, and a PhD in computer engineering from National Chiao Tung University, Hsinchu, Taiwan, in 1987. From 1977 to 1979, she worked as a research assistant at the Chung-Shan Insti-tute of Science and Technology, Taoyan, Taiwan. From 1979 to 1981, she worked as a research associate at the Electronic Research and Service Organization, Industry Technology Research Institute, Hsinchu, Taiwan. From 1981 to 1983, she worked as an engineer in the Institute of Information Industry, Taipei, Taiwan. She is now a professor in the Department of Computer Science at the National Chiao Tung University. Her current research interests include image processing, pattern recognition, multimedia compression, content-based retrieval, and multimedia steganography.
Wei-Kang Wang received a BS in computer science from National Chiao Tung University, Hsinchu, Taiwan, in 2010. He is currently a graduate student at the Institute of Multimedia Engineering, National Chiao Tung University, Hsinchu, Taiwan. His main research interest is image processing.
