CHAPTER 1 INTRODUCTION
1.3 Organization of the Thesis
This thesis is composed of five chapters. In Chapter 1, previous works are introduced. In Chapter 2 and Chapter 3, we will propose solutions to automatically improve image quality for non-face and face images,
respectively. Several experimental results and comparisons will be shown in Chapter 4. Finally, we will give a conclusion and future work in Chapter 5.
CHAPTER 2
THE PROPOSED NON-FACE ENHANCEMENT METHOD
The objective of image enhancement is to produce a more suitable result than an original image for a specific purpose. For the application of real-life image enhancement, keeping the sense of reality of the image without artifacts is the most important criterion. For example, if we take a picture at night, we do not hope this photo look like daytime after using image enhancement techniques.
As mentioned previously, HE fails to produce pleasant pictures owing to three drawbacks: (1) wash-out appearance; (2) false contour; (3) amplified noises. Before explaining the proposed method, we will analyze what factors cause these defects. According to the two histograms given in Fig. 1.2, because of the occurrence probability value of histogram bin 255 is abnormally high in the original image, HE causes a serious gap for those pixels with gray values 254 and 255 in the original image. If these two kinds of pixels are neighbors, this will cause false contours. In Fig. 1.3, most pixels have low luminance in the original image. This makes the contrast-stretching by HE very excessive in the low luminance and the medium luminance push toward to the higher grayscales. This will amplify the noises in the dark regions and look like the wash-out appearance in the whole image. In conclusion, when the histogram of an image has the characteristic of high amplitude at few peaks or excessive concentration at small continuous gray
levels, it is difficult to produce a proper result by the HE technique.
Based on the discussion above, we regard JND as a constraint on contrast stretching. If the original histogram curve exceeds a limitation, it is possible that the image is characterized by the problems mentioned above. Therefore, we try to modify the histogram curve to become not only similar to the original histogram curve but also qualify to the restriction. In other words, we use the modified HE mapping to make the contrast as greater as possible without producing artifacts.
In this chapter, a new approach of image enhancement for real-life image is presented. We use the HE technique and regard the JND of HVS model as a contrast-stretching constraint to avoid the artifacts mentioned previously. In Section 2.1, we will introduce the conventional histogram equalization method. In Section 2.2, the contrast-stretching constraint based on JND is provided. In Section 2.3, we present a new approach of adjustment of histogram curve in Section 2.4. In section 2.5, the color construction method is proposed. Fig. 2.1 is the flowchart of our proposed method.
Furthermore, for real-life images, skin regions are the most relevant parts by human perception. We will improve our framework to treat skin problems especially and describe this in Chapter 3.
Start
Use HE to get the transform function
Satisfy the constraint?
Read in the image and extract the luminance
Examine by the contrast-stretching
constraint
Adjust the histogram curve
End No
Yes
Reconstruct the color image to get
the final image
Fig. 2.1 The flowchart of the proposed non-face enhancement method.
2.1 Histogram equalization
Histogram equalization (HE) [3] is a non-linear mapping which approximately produces the discrete equivalence of a uniform probability density function. Before presenting the HE technique, we need to extract the luminance Y at spatial coordinate (x, y) of the original image by
)
After obtaining the gray values of the original image, let variable k represent the gray level in the interval [0, 255] of an image. Then, the probability p of the occurrence of gray level k in an image is calculated with
, )
( N
k n
p = k (2.1-2)
where N is the total number of pixels in the image, and nk is the number of pixels of gray value k. Next, we describe the mapping function of the form s
= T(k), where T is a transform function that maps a gray level k into a gray level s. The transform function is denoted as follows.
.
2.2 Contrast-stretching constraint based on JND
Before explaining the constraint, we introduce the concept of JND in advance. In the field of image processing, just-noticeable-difference (JND) is the ability of distinguishing the luminance change with human visual system (HVS). In other words, JND is the smallest difference of the luminance change which is detectable by human eyes. We adopt the JND model proposed by Chou and Li [12]. Fig. 2.2 illustrates the smallest detectable difference of each possible gray level by their experiment. As can be indicated by Fig. 2.2, it is relatively sensitive to the change of medium gray level by human visual system. On the contrary, it is relatively not sensitive to the change of dark or bright background. The perceptual model for evaluating the visibility threshold of JND is denoted by
( )
depend on the viewing distance between testers and the monitor taken by experiment.
Fig. 2.2 Visibility thresholds related to the luminance.
Real-life image enhancement should not only focus on contrast stretching, but also try to prevent enhancing the areas where the difference of illumination is unnoticeable from producing artifacts. We use this concept to design an examination function which can use to judge if the HE technique is suitably applied to an image. The HE technique uses the information of the occurrence probability of each gray level to determine the amplitude of contrast stretching. Therefore, for any two consecutive gray levels k-1 and k, the mapping function by HE yields corresponding gray levels T(k-1) and T(k).
The difference between T(k-1) and T(k) represents the amplitude of contrast stretching. In order to avoid over-contrast-stretching by the HE technique, we give a constraint that for every two consecutive gray levels k-1 and k, which is not distinguishable by human eyes, the difference between T(k-1) and T(k) should not be larger than the JND value of T(k). Based on this constraint, the examination function is denoted as follows:
)
The examination function can help us to avoid enhancing regions with unnoticeable difference to be perceived. Through the examination function, we can determine whether the HE technique is suitably applied to an image for enhancement.
2.3 Adjustment of histogram curve
As indicated in previous discussion, we can use the examination function to determine if it is suitable to apply the HE technique in an image to do enhancement. If it is suitable, the HE technique can be applied to enhance the image directly without producing artifacts. Otherwise, we must adjust the occurrence probability values to qualify the constraint. If the adjusted histogram curve is smoother, the constraint is more possibly satisfied.
Note that keeping the monotonic property of the histogram of an image is the most important criterion. This means that for any two gray levels h and k, if p(k)>p(h), then p(T(k))>p(T(h)). As a result, our adjustment approach is accomplished by making the histogram curve smoother and preserving the monotonic property. We eliminate the higher probability of occurrence of histogram and then redistribute the eliminated value over all grayscales uniformly. This method can help eliminate the high peaks of histogram or spread out the over-concentration probabilities at small continuous gray levels with lower probabilities. The procedure of the adjustment approach contains three phases: sort, shift, and redistribution. These three phases are described as follows:
z Sort: The histogram component, p(i), defines the probability of
) (i p′
occurrence of gray level i, and represents the adjusted histogram component. Let n denote the current shift times and initialize to be zero.
In order to preserve the monotonic property, we need to sort the distinct histogram components in descending order. Let L denote the number of distinct histogram components and ordern(i) be the ith element sorted histogram components, where i is within the interval [1, L] and n is the current shift times. Therefore, and represent the biggest and smallest original probability value respectively without adjustment. Fig. 2.3(a) gives an example of a histogram with 7 histogram components.
)
There are 7 distinct histogram components (L=7) and its ordered set (see Fig. 2.3(b)) is
z Shift: After sorting, we shift the histogram components according to the following rules. First, the n-biggest values in sorted histogram components are discarded. That is, if n=2, order (1) and order0 0(2) are discarded. Next, each sorted component substitute is by the next n sorted component. Then, the shifted and sorted components are transformed into a histogram as follows:
)
The procedure for shifting the histogram components in Fig. 2.3(b) (n=1) is shown in the following.
.
Zero padding
Figs. 2.3(c) and (d) illustrate the shifted sorted histogram components and the corresponding histogram.
z Redistribution: After obtaining the new shifted histogram components, the difference between original and shifted histogram components are redistributed uniformly over all grayscales. This redistribution function is denoted by
where g is the number of grayscales. Therefore, the examples of redistributed results are given as follows.
.
Fig. 2.3(e) shows the results treated by different shift times. It can be noted that the more shift times there are, the smoother the curve is. As a result, through increasing the shift times, there should be a smallest shift times such that the adjusted histogram is qualified.
(a) (b)
(c) (d)
(e)
Fig. 2.3 An example to illustrate the proposed adjustment method. (a) Original histogram. (b) The sorted histogram components of (a). (c) The result of shifting (b) one time. (d) The corresponding histograms after shifting one time and redistributing. (e) Several results treated by different shift times.
2.4 Prevention for over-adjustment
The more number of shift times are, the greater changing of original histogram components is. However, the smoother histogram curve leads to the less amplitude of contrast-stretching. Through the adjustment approach described in previous section, we can get the smallest shifted times n such that the shifted histogram is qualified. In order to avoid over-adjustment, we get an interpolated histogram by
. components and the new histogram components are redistributed uniformly to get the final histogram components.
)
We adopt bisection method to find the smallest α such that the shifted histogram can qualify the constraint. Fig. 2.4 illustrates an example of the procedure. The red curve and blue line show the sorted histogram components treated by n = 0 and n = 1 respectively. The black curve shows the interpolation of the two curves. (α= 0.5)
Fig. 2.4 An example to illustrate the procedure of avoiding over-adjustment.
2.5 Color reconstruction
After previous steps, we can get the modified gray values . Then, color reconstruction is done by using the following formulation [13] to prevent relevant hue shift and color desaturation. This method is denoted by
Y ′
where R, G, and B are the input color values.
Finally, there is an example to illustrate the whole proposed method in Fig. 2.5. First, there is a dark scene in Fig. 2.5(a). Through the examination function, its corresponding histogram is not qualified, so the adjustment method is applied. Through increasing the shift times, we can find a smallest shift times (n = 4.6) such that the adjusted histogram can be qualified. Figs.
2.5(b) through (h) show the results treated by different shift times. It can be seen that the noises in the background are decreasing and the over-exposed areas are being improved. Fig. 2.5(i) shows the original histogram curve and the adjusted curve applied by our proposed adjustment approach.
(a) Original image
(b) n = 0 (c) n = 1
(d) n = 2 (e) n = 3
Fig. 2.5 An example to illustrate the proposed method. (a) Original image.
(b)-(e) The enlargement of part of the results treated by 0-3 shift times.
(continues)
(f) n = 4 (g) n = 5
(h) n = 4, α=0.6 (i)
Fig. 2.5 An example to illustrate the proposed method. (f)-(g) The
enlargement of part of the results treated by 4 and 5 shift times. (h) The enlargement of part of the results treated by 4.6 shift times for avoiding over-adjustment. (i) The original and the final adjusted histogram curve.
CHAPTER 3
THE PROPOSED FACE ENHANCEMENT METHOD
It is straightforward that skin regions, especially faces, in real-life images are the most visually interesting areas. Most conventional image enhancement techniques [3-8, 10-11] do not give a special treat for the improper lighting condition in skin regions. Hence, these enhancement techniques either improve obviously for the unsuitable illumination in skin regions or fail to offer sufficient contrast in skin regions. If the contrast in skin regions is insufficient, the skin regions may seem to be wash-out appearance and unnatural (see Figs. 3.1(b) through (e)). On the other hand, some techniques [9, 14-15] were provided to enhance face part in an image, for the other part, they can not provide a satisfied result.
Battiato et al. [9] proposed an exposure correction with camera response curve improved by skin dependent techniques. The basis of adjusting illumination in a whole image is based on the difference between the average luminance of skin regions and the ideal pre-defined luminance. This technique can produce a satisfied result in skin regions, but it may fail to produce a suitable result in non-skin regions. Fig. 3.2 shows the satisfied illumination in skin regions, but the background regions are distorted.
(a)
(b)
(c)
(d)
(e)
Fig. 3.1 Some results using different enhancement methods with the images
in the right column being the enlarged parts of the images in the left column.
(a) Original image. (b) HE [3]. (c) Capra’s algorithm [8]. (d) Our proposed non-face enhancement method. (e) Picasa software [2].
(a)
(b)
Fig. 3.2 A result using exposure correction method with the images in the
right column being the enlarged parts of the images in the left column. (a) Original image. (b) Battiato’s algorithm.
Since our proposed non-face enhancement method can only treat non-face images well, for those face images, the skin part may not be treated well. In this chapter, we will provide a method to treat both face and non-face regions. The proposed method integrates our non-face enhancement and the exposure correction method proposed by Battiato et al [9]. This exposure correction method uses the mean luminance of skin regions as a reference point. Then, we can apply exposure correction method by skin content to get an image with satisfied skin regions. After obtaining the results of non-face enhancement method defined Ynon-skin, and exposure correction method defined Yskin, a distance map can help us to fuse these two results. Fig. 3.3 shows the flowchart of the proposed method for face images.
Fig. 3.3 The flowchart of the proposed face enhancement method.
3.1 Skin recognition by skin locus model
Before applying the exposure correction by skin dependent techniques, we should recognize skin pixels in advance. We would like to determine all possible skin pixels with multiple light source or changing illumination conditions. Therefore, we choose the skin locus model defined in [16]. To reduce the illumination dependence, this technique is based on the
plane by normalized RGB space (
) , (r g
) , (
)
( R G B
g g B G R r R
+
= + +
= + ).
Through the experiment, Fig. 3.4 shows the r-g skin histogram in diverse illumination condition from a 1CCD camera.
Fig. 3.4 Statistic of skin locus
By this statistical information, the skin color cluster in plane occupies a shell-shape curve. A membership function to the skin locus is a pair of quadratic functions denoting the upper and lower bound of the cluster.
Pixels can be labeled as skin pixels using the skin locus constraint denoted as follows:
where skin=1 representing a skin pixel and skin=0 representing a non-skin pixel. The variable W is to avoid labeling whitish pixels as skin. Therefore, we can process an image pixel-by-pixel using this constraint and record the result in a skin map, , with = 1 representing a skin pixel and
= 0 representing a non-skin pixel(see Fig. 3.5(b)). Then, the skin map should be refined by Morphological operation [3] to eliminate the noises and fill the holes (see Fig. 3.5(c)). In order to speed up the latter procedure described in Section 3.3, we can use the bilinear interpolation of a factor of
Mskin Mskin(x,y) )
, (x y Mskin
1/4 to scale the original image at first to obtain a smaller size of Mskin.
(a)
(b) (c)
Fig. 3.5 An example of skin recognition. (a) Original image. (b) Recognized
skin map by the skin locus. (c) Refined skin map after morphological processing.
3.2 Exposure correction method
The exposure correction method is accomplished by a simulated camera response curve [17]. This curve shows an evaluation of light value q, called
“light quantity”, transformed to the final pixel values I by the camera sensor (see Fig. 3.6). This camera response curve f can be presented by
C
e Aq
I q
f (1 )
) 255
( −
= +
= (3.2-1)
where parameters A and C can be utilize for controlling the curve shape.
Fig. 3.6 A simulated camera response curve
Therefore, the exposure correction technique [9] utilized the transformation between light quantity and final luminance to simulate controlling how much light the camera will capture. Based on this transformation, the exposure correction method uses the mean luminance of skin regions as a reference point. First, we extract luminance Y with Eq.
(2.1-1) in an original image. After labeling skin pixels, we can get the average luminance Yavg of the skin pixels. A simulated camera response curve f defined in Eq. (3.2-1) can be used to offset the light quantity difference between Yavg and pre-defined ideal luminance Yideal. The offset of light quantity is denoted by
) ( )
( 1
1
avg
ideal f Y
Y f
offset= − − − (3.2-2) The original luminance values can be modified by
))) , ( ( (
) ,
(x y f offset f 1 Y x y
Yskin = + − (3.2-3) Therefore, the Yskin is the result of the exposure correction method. There is an example to illustrate the result of exposure correction method in Fig 3.7.
(a) (b)
Fig. 3.7 An example of the exposure correction method (a) Original image. (b)
Result of the exposure correction method.
3.3 Measurement of distance map
In this section, we define a measurement of a distance map Mdistance. Mdistance means a distance map recording the distance between each pixel and its nearest skin pixel. The Yskin and Ynon-skin can be fused together using a distance map. Before presenting the fusion method, we describe the measurement of a distance map at first.
After labeling the skin pixels in the Mskin, there should be several connected components of skin regions. Therefore, we use the dilation operation [3] iteratively to estimate the Mdistance. It is the reason that if a pixel is nearer connected components, it will be dilated earlier. Because the distance between a pixel and different connected components is distinct, the smallest distance should be selected. Therefore, each connected component should be dilated individually and recorded the smallest distance for each pixel. A pre-defined threshold t would be given for setting the number of times of performing dilation procedure. When dilation procedure is stop, the pixels which are not dilated yet represent that they are too far from skin
regions. Therefore, they are all assigned to be t+1 which is the farthest distance recorded in Mdistance. This method is described as follows and there will be an example to illustrate to procedure in Fig. 3.8.
Notation and initialization: Let d denote the current dilation times and
initialize to be zero. Denote Mdisn tance to be the distance map for the nth connected component where 1≤n≤C and C is the number of connected components in a skin map. Initialize the
where it is a skin pixel at coordinate belonging to the n
0 ) ,
tan (x y =
Mdisn ce (x,y)
th connected component. Other pixels are initialized to be infinite (see Figs. 3.8(a) and (b)).
Step 1: Add one to the variable d. Dilate each connected component in
one time by a disk structuring element and record the current dilated pixel at to be d.
n ce
Mdistan
) ,
tan (x y Mdisn ce
Step 2: If , go back to step 1. Otherwise, combine all
with the smallest value at the same position to obtain the distance map M
t
d < Mdisn tance
distance. (see Figs. 3.8 (c) throught (e))
Step 3: Replace all infinite values in Mdistance with value t+1. (see Fig.
Step 3: Replace all infinite values in Mdistance with value t+1. (see Fig.