LIGHT COMPENSATION
Chih-Fong Chang (
張致豐), Chiou-Shann Fuh (
傅楸善)
Dept. of Computer Science and Information Engineering, National Taiwan University E-mail: r96922104@ntu.edu.tw
ABSTRACT
Digital camera often produces undesired result when working under improper light condition. The details of dark regions are too obscure to be recognized, and it would not be a good picture that the photographers wish.
To solve the problem, we enhance the image with two phases. First we enhance the image by adjusting the brightness of the image; second we enhance the major details and the details in dark region to be more distinct.
Our results show that we can light up the dark regions, and keep the bright regions from overexposure.
1. INTRODUCTION
Nowadays, film cameras are already replaced by digital cameras. Digital cameras capture and store photographs digitally, and people can design methods to enhance pictures. If the output results are not good enough, the very common problem is underexposure. Figure 1.1 shows the image captured in dim light environment.
Figure 1: Image capture in dim light environment.
In order to solve the underexposure problem, analyzing the image properties before enhancing is necessary. The idea of Human Visual System (HVS) was proposed by many researchers.
1.1. Rods and Cones
From biology, the rods and cones are the most important parts in the retina. The rods discriminate scene at low luminance level, and cones provide color vision at high luminance level.
The rods have smaller threshold than the cones, when log background luminance is less than 0. Then if the luminance is too low, the luminance information will be more important than color information.
1.2. Image Enhancement
Global contrast enhancement takes short computing time, but requires high memory, and the result is worse than local one. Local contrast enhancement usually takes long computing time, but the result is usually better than algorithms using global information, especially in enhancing object detail.
Histogram Equalization (HE) has been a well known technique that globally enhances the image quality by rearranging the unbalanced light distribution and increasing the dynamic range of the image. Nevertheless, in some cases the exaggerated enhancement of contrast produces unfavorable artifacts. Figure 1.2 shows the unnatural looking.
Figure 2: Image processed with histogram equalization looks unnatural.
1.3 Contrast
Contrast is the difference in visual properties that makes an object distinguishable from other objects and the background. In visual perception of the real world, contrast is determined by the difference in the color and brightness of the object and other objects within the same field of view. Human visual system is more sensitive to contrast than absolute luminance, if we want to achieve the same contrast, brighter regions require larger luminance gradients than darker regions.
1.4 Halo Artifact
Local contrast enhancement has been proposed in several methods. Most of these techniques compute the local adaptive luminance by finding the geometric or arithmetic mean of pixel luminance in a local window.
These methods usually produce halo artifacts around region of high contrast. Halo artifacts happen on the bright side of edge with high gradient, because the local mean luminance is overestimated from the effect of bright pixels. In other word, the mean luminance of the dark side is underestimated due to the effect of bright pixels. Figure 3 shows the image with halo artifact.
Figure 3: Image with halo artifact.
2. RELATED WORK
2.1. Logarithm Curve
The difference between logarithm curve and gamma curve is that logarithm curve obeys the Weber-Fechner law of JND response in human vision. Gamma curve has high slope for low luminance, if γ is too large, it usually enhances the noise of the dark region, resulting in overexposure in light region; If γ is too small, the light enhancement would be insufficient. Followed by Stockham’s suggestion [8], a simple form of logarithm curve is used.
Figure4: The logarithm curve. Left: comparison between deferent β. Right: comparison between logarithm curve and gamma curve [8].
2.2 Detail Decomposition
In computational photography, images are often decomposed into a piecewise smooth base layer and one or more detail layers [3]. The base layer captures the larger scale variations intensity, and is typically computed by applying an edge-preserving smoothing operator to the image. The detail layers may be manipulated separately in various ways, depending on the application and possibly recombined to yield the final result.
R. Fattal introduced a new way to construct edge- preserving multi-scale image decompositions, based on the bilateral filter [3]:
∑
∈Ω+ +
+ = ⋅ − ⋅
q
j q p j p j
q p j r j
s j
p g q g I I I
I k1 (| |) ( ) ,
, ,
1 σ σ
, (| |) , ( j j)
s j r j p q p
q
k gσ q gσ I + I
∈Ω
=
∑
⋅ −) / exp(
)
( 2 σ2
σ x x
g = −
where p is a pixel coordinate; gσs,jand gσ are the r,j widths of the spatial and range Gaussians respectively;
and q is an offset relative to p that runs across the support of the spatial Gaussian. The repeated convolution by gσs,j increases the spatial smoothing at each scale j.
We can compute a set of detail images as difference between successive levels of these bilateral filtered images Dj =Ij−Ij−1 for j=1, 2, 3, …, m. Thus,
I j retains the strongest edges in the image as preserved by the bilateral filter and the detail layers
D
j contain the smaller changes in intensity.Figure 2 shows the result of bilateral decomposition;
I (n, 0) is the original image; and I (n, i) is the ith times after bilateral filter; and D(n, i) is the difference between I (n, i) and I (n, i-1).
By the bilateral decomposition, we generate the image as follows:
) log(
) 1 ) 1 ( ) , ( ) log(
, (
'
ββ
− +
= I x y × y
x
I
m m j
j
I
D
I = ∑
=+
1
After the bilateral decomposition, there is a synthesis method used to enhance detail:
where β is a trade-off emphasis of the detail image versus the base images.
Figure 5: The result of bilateral decomposition [3].
Figure 6: The result after synthesis method [3]. Right:
input image. (a) At β= 3.0 the flower is very smooth. (b) Reducing β to 0.8 (the default value) and then (c) to 0.2 brings in more detail.
2.3 Smart Tone Reproduction
An image model I(x, y) is commonly regarded as a product of reflectance R(x, y) and luminance L(x, y) at each point of coordinates (x, y). Moreover, the enhanced image I’(x, y) becomes [6]:
) , ( ' ) , ( ' ) , (
' x y R x y L x y
I = ×
To avoid image luminance being extreme after enhancement, D. Y. Hsiao expresses this characteristic as follows [6]:
, )
, (
'
maxmin
+ δ < L x y < G − φ G
where Gmin and Gmax are the minimum and maximum values of a dynamic range; and δ and φ are two constants to prevent extreme luminance values.
Weber’s Law points out: a contrast is perceivable only when it is greater than predetermined threshold [6].
Therefore, he concludes that the details of an image can be identified only when its local contrast is large enough.
Therefore the following can be derived:
, ))
, (Region(
), , ( / ) , ( ) , ( ' / ) , ( '
threshold y
x Contrast
y x L y x R y x L y x R
>
>
Figure 7: The result of smart tone reproduction [6].
2.4 H. T. Chen’s Method
H. T. Chen [1] uses two stages to adjust the brightness.
First he derived a method which enhances the dark regions much stronger than the bright ones:
) , ( ))) , ( 2 log(
1 ( )
,
(x y Scaler I x y xI x y
IBrighter = × + − in in
where Iin(x, y) is the brightness channel of the input image represented in HSV color space and has a range in [0, 1]; and Scaler controls the strength of the enhancement. Scaler and Gain control the strength of enhancement.
After first stage, the bright regions of the original image will be burned out. To overcome this problem, he exploits the information of each pixel’s brightness and saturation in HSV color space to decreasing the result image processed by first stage.
2 / ) 1 ) , ( (
) , (
)) , ( 1
/(
1 ) , (
) , ( )
, ( )
, (
+
= +
=
×
=
y x AvgSat y
x D
y x I y
x D
y x D
y x D
y x I
Saturation
power Origin
Brightness
Saturation Brightness
Dark
where IOrigin represents average brightness of the pixel values covered by the 5*5 block located at (x, y); and power is the strength of this function.
Figure 8: The result of H. T. Chen’s Method [1].
1 0
, < <
⋅ +
=
Detailβ
Baseβ
Result
I I
I
2.5 ORMIT
Based on the human visual system, Apical developed Orthogonal Retina-Morphic Image Transform (ORMIT) [2]. Human eye has relatively high dynamic range extending well into the dark region, so the algorithm is preferably asymmetric in the brightness domains, pixels in dark regions have to be enhanced more than relatively bright regions.
The algorithm is:
) ) 1 ( )) ( ( ))) ( ( ( ) ( ( '
0
∑
=×
− +
×
×
= N
i
i i
i
i I LPF P F I Q F I I
I α α
where Pi(x) is orthogonal basis of functions of x defined in the range, 0≤x≤1; Qi is anti-derivatives of P; LPF is operator of Low-Pass spatial Filtering; F is weight function similar to gamma correction; N is number of bands; and α is the strength of the transformation varied with the intensity of I. The following equation is an example of α (I) in [2]:
−
∆+
∆+
−
= 2
) 1 1 log(
) 1 log(
4 2tanh 1 2 ) 1 (
I α I
where ∆ controls the strength of asymmetric effect of this transform on dark and bright part. That is, this algorithm is more active in the dark regions than in the bright regions. The range of ∆ is 0.001 < ∆ < 0.1 in practice. The weighting function F(I) is used to approximate the property of human eye and can be written as a logarithmic transfer function:
) log(
) 1 log(
) log(
) ) log(
( +∆ − ∆
∆
−
∆
= I + I
F
ORMIT employs Legendre polynomials as the orthogonal functions P (I) which can be given by the recurrence equation:
P0 (x)=1, P1 (x)=x, Pi (x)=aiPi-2 (x)+bi × Pi-1 (x), and
Figure 8: The result of ORMIT [2].
3. PROPOSED METHOD
3.1 Light Enhancement
In Chapter 2, we introduced a logarithm-curve method to enhance image, but it can not satisfy pictures in different illuminations. Therefore, we need a method to produce initial β automatically.
We want to make the image captured in dim light more distinct, but we don’t want to enhance too much if the image is captured in appropriate light environment.
So we calculate the numbers of dark pixel as reference, and we can produce a suitable β.
The number of pixel darker than threshold Tdark are
darkpixel
N
; Ntotalpixel is the numbers of total pixels; and βmin and βmax are selection max and min β valueThen, we can enhance the image by β’ and the logarithm curve:
Unfortunately, even with this measurement, image cannot avoid underexposure or overexposure completely, so we have to add back the original image.
where θ is the strength of transform, 0≤θ≤1 (default θ=
I). and the final equation becomes:
(a) (b) (c)
(d) (e) (f)
Figure 9: comparison output images with different β. (a) and (d) are two input images; (b) and (d) use β counting by our method; (c) and (f) use fixed β=15. the result (f) is not enhanced enough at the stairs.
θ θ + ×
−
×
= I I
I'' ' (1 ) ) ' log(
) 1 ) 1 ' ( ) , ( ) log(
, (
' β
β− +
= I x y × y
x I
θ θ
β =I × − +I × I
LE( , ) ' (1 )
min min
max
) ( / )
(
'
β β ββ
= − ×
Ndarkpixel Ntotalpixel+
i b i i
ai i i 2 1
1, = −
= −
Figure 10: comparison output images with adjustment.
Left: input image. Middle: light enhancement without adjustment. Right: light enhancement with adjustment.
3.2 Detail Enhancement
According to the detail decomposition in Section 2.3, we can decompose an image into a smooth base layer and one or more detail layers. Moreover, we can enhance them in different methods.
We decompose an image I into a base layer LBase
and k detail layers Li, 1≤i≤k :
In Section 2.2, we have a synthesis method as:
It enhances the detail but makes the image darker. We try to adjust it as:
Therefore, the equation becomes:
Figure 11: the result of our method. Left: input image.
Middle: image after light enhancement. Right: image after full stage.
4. THE SPEED-UP METHOD
Nowadays, digital image is usually bigger than 4 megapixels, it takes more than 4 second with our method, it is too slow and it can not satisfy our requirement.
There are some easier ways to reduce the execute time:
(1) Reduce the number of layer.
(2)
Reduce the bilateral filter mask
(3) Use a pre-make table to search the suitable brightness in light enhancement phase
Those method are useful and easy to execute, but the cost time is still not good enough, so we try to redesign the decomposition.
The original bilateral filter has two Gaussian functions. Without spatial Gaussians, halo artifact would be produced after the detail enhancement; without range Gaussians, the image would be less smooth after enhancement.
Figure 12: left: the decomposition without range Gaussian; right: the decomposition without spatial Gaussian.
Halo artifact is hardly tolerated, so we try to modify the filter as:
Figure 13: comparison with original method and the speed-up method. Left: input image. Middle: original method. Right: speed-up method.
∑ = +
= k
i L L
I Base 1 i
i LBase
k L
i k k
i Result k
I ∑ +
= ⋅
+
− +
= ⋅
1 2 3
) 2 ( 2
∑ = +
− + + ⋅
= k
i LE L
k k
i L k
LE
Iout Base 1 ( i, )
2 3
) 2 ( ) 2
,
( β β
1 0
, < <
⋅ +
= L
Detail βL
Base βI
Result∑
∈Ω + ++
= − ⋅
q
j q p j p j
q p j r j
p
g I I I
I k 1 ( )
,
1
σ
)
,
(
j p j
q p j
r
I I
g
k =
σ +−
5. EXPERIMENTS AND RESULTS
First we introduce our experiment environment:
CPU: AMD Athlon 64 X2 Dual core.
Memory: 2GB.
System: Windows XP Professional Service Pack 2.
Programming Language: Visual Studio C++
Our method takes 0.8 second to process an image with resolution of 800*600 pixels; the speed-up method takes 0.4 second; and the method without detail enhancement takes 0.18 second.
(a) Input image. (b) Our method.
(c) Smart tone reproduction.
(d) H. T. Chen’s Method.
Figure 14: (c) looks unnatural, and (d) is too white at the windows.
In Figure 14, H. T. Chen’s Method enhanced windows region too much, and the result is too white at the windows.
(a) Input image. (b) Our method.
(c)Smart tone reproduction. (d) H. T. Chen’s Method.
Figure 15: Image (c) is over-exposed inside the house.
In Figure 15, the result of smart tone reproduction method burned out the region inside house where we hardly recognized after enhancement.
(a) Input image. (b) Our method.
(c) ORMIT. (d) H. T. Chen’s Method.
Figure 16: (c) has halo artifact; (d) enhances the brightness too much.
In Figure 16, image (d) enhanced the image too much, so it makes the image too bright.
(a) Input image. (b) Our method.
(c) ORMIT. (d) H. T. Chen’s Method.
Figure 17: image (d) enhances the brightness too much.
In Figure 17, the grass region of image (d) enhanced too much, and the details disappear.
(a) Input image. (b) Our method.
(c) ORMIT. (d) H. T. Chen’s Method.
Figure 18: image (b) and (d) is too white at the eyes
In Figure 18, image (b) and (d) both enhanced the eye too much, and so the eyes look like white after enhancement.
(a) Input image. (b) Our method.
(c) ORMIT. (d) H. T. Chen’s Method.
Figure 19: image (c) has halo artifact.
In Figure 19, image (c) has artifact around the human body.
H. T.
Chen’s Method
Smart Tone Reproducti on
ORMIT Our method
1 5 7 9
2 8 2 11
3 9 0 12
4 5 3 13
5 5 8 8
6 6 7 8
7 8 3 10
8 8 3 10
9 2 11 8
10 9 1 11
11 4 10 7
12 9 2 10
13 0 16 5
14 4 8 9
15 6 7 8
Table 1: vote result.
Table 1 show the vote result with twenty-one people.
6. CONCLUSION AND FUTURE WORK
We have proposed a simple and effective method for light compensation. We lighten the whole image through the logarithm function, and then we attenuate the brightness with proper brightness. Then we enhance the detail, and image becomes more distinct.
The experiments show that our method can make it possible to enhance the brightness of the dark regions and meanwhile can preserve the details of the bright regions.
However, the detail enhancing method usually enhances the noise too. The noise can be reduced after the noise reduction, but the detail may diminish too.
Figure 20 shows a example about it, image (b) is the result after our method, there is some noises enhanced after our method. After doing noise reduction, noise are got clear, but some of the detail are also disappear too.
There should be a good method to trade-off between the noise and detail that we have to solve in future.
(a) (b) (c)
Figure 20: about noise reduction. Left: input image.
Middle: image after enhancement. Right: image after enhancement and noise reduction.
REFERENCES
[1] H. T. Chen, “An Algorithm for Light Compensation,” Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, 2008.
[2] V. Chesnokov, “Dynamic Range Compression Preserving Local Image Contrast,” UK Patent GB 2 417381 A, 2006.
[3] R. Fattal, M. Agrawala, and S. Rusinkiewicz,
“Multiscale shape and detail enhancement from multi-light image collections,” ACM Transactions on Graphics, Vol. 26, No. 3, 2007.
[5] R. C. Gonzales and R. E. Woods, “Digital Image Processing”, 2nd , Prentice Hall, Upper Saddle River, NJ, 2002.
[6] D. Y. Hsiao and H. Y. M. Liao, “Smart Tone Reproduction,” Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, Las Vegas, IMDSP-P11.F10, 2008.
[7] I. V. Safonov, M. N. Rychagov, K. Kang, and S. H.
Kim, “Automatic Correction of Exposure Problems in Photo Printer,” Proceedings of IEEE International Symposium on Consumer Electronics, St. Petersburg, Russia, pp. 1 – 6, 2006.
[8] T. G. Stockham, “Image Processing in the Context of a Visual Model,” Proceedings of IEEE, Vol. 60, No.
7, pp. 828-842, 1972.
