利用直方圖實現一個新的顏色轉移之方法

國立交通大學

多媒體工程研究所

碩 士 論 文

利用直方圖實現一個新的顏色轉移之方法

A Novel Method For Color Transfer Using Histogram

研 究 生：張翔竣

指導教授：施仁忠 教授

利用直方圖實現一個新的顏色轉移之方法

A Novel Method For Color Transfer Using Histogram

研 究 生：張翔竣

Student：Shiang-Chun Chang

指導教授：施仁忠

Advisor：Zen-Chun Shih

國 立 交 通 大 學

多 媒 體 工 程 研 究 所

碩 士 論 文

Submitted to Institute of Multimedia Engineering College of Computer Science

National Chiao Tung University in Partial Fulfillment of the Requirements

for the Degree of Master

in

Computer Science June 2009

Hsinchu, Taiwan, Republic of China

利用直方圖實現一個新的顏色轉移之方法

A Novel Method For Color Transfer Using Histogram

研究生：張翔竣

指導教授：施仁忠

國立交通大學多媒體工程研究所

摘 要

A Novel Method For Color Transfer Using Histogram

Student : Shiang-Chun Chang

Advisor : Dr. Zen-Chung Shih

Institute of Multimedia Engineering

National Chiao Tung University

ABSTRACT

In this thesis, we propose a novel method for color transfer using histogram. We can construct the histograms individually for both source and reference images based on the statistical analysis of them. After the construction of histograms, we perform segmentation in order to separate the histograms into multiple sections. According to the relationship of corresponding sections between the source and target images, we do color transfer from target to source. At the same time, there will be parameter interpolation between pairs of sections to solve the artifacts along the section boundary. In this way, we can transfer the color information from target image to source image and the result is better than the current methods.

Acknowledgements

First of all, I would like to thank my advisors, Dr. Zen-Chung Shih for their supervision and helps in this work. Then I want to thank all the members in Computer Graphics & Virtual Reality Laboratory for their comments and instructions. I want to dedicate the achievement of this work to my family. Without their support, I couldn’t fully on my study.

Content

Chapter 2 Related Work ……….7

Chapter 3 Histogram Segmentation ………...7

3.1 Segmentation Threshold ………..12

3.2 Parameter Interpolation ………...14

Chapter 4 Structure Control Term ………....20

4.1 Structure Control Term Motivation………..20

4.2 Structure Control Term Definition………...22

Chapter 5 Result ………...24

Chapter 6 Conclusions And Future Work ………....31

List of Figures

Figure 1.1: System flow chart ………...5

Figure 2.1: The color transferred result by Reinhard’s result ………7

Figure 2.2: The colorized result by Welsh’s method ……….8

Figure 2.3: The result of Levin’s method ………..9

Figure 2.4: Irony’s method ………..10

Figure 2.5: Wen’s method ………11

Figure 3.1: Concept of histogram ………13

Figure 3.2: The concept of multiple section color transfer ……….15

Figure 3.3: Our method without interpolation ………16

Figure 3.5: Section parameter interpolation ………19

Figure 4.1: Extreme case example ………..21

Figure 5.1: Input and Result ………25

Figure 5.2: Input and Result ………26

Figure 5.3: Input and Result ………27

Figure 5.4: Input and Result ………29

Chapter 1

Introduction

1.1 Motivation

Color transfer is a task that a source image copies color characteristics from

a reference image. It is one of the most common topics in image processing. In fact, with the help of powerful post-processing software, experts can adjust the image’s color characteristic in order to change the image’s style but it is complex and tedious. On the other hand, not everyone is familiar with the professional software.

Fortunately, digital cameras are very popular nowadays, and people take photos wherever they like. In this way, there will be a lot of images with different color characteristics. In this thesis, we propose a method to transfer user desired color characteristics of an image, called reference image, to the other, called source image whose color characteristics want to be modified. For transferring

color characteristics of the reference image to source image, Reinhard et al. [2] proposed a simple but effective method using statistical analysis. They use the mean value and standard deviation of both the source and reference images doing shifting and scaling for every source image pixels in each l, α, and β channel.

Although Reinhard’s method successfully transfers the reference image’s color characteristics to the source image, the transferring algorithm depending only on the mean value and standard deviation for each image makes the final result lose its original feature or obscure. The scaling term according to the deviation ratio breaks the original structure of the source image in the extreme case, for instance, the standard deviation of reference image equals 0.

In our approach, we apply histograms to Reinhard’s method and segment it into three sections. Each histogram section has its own mean value and standard deviation. Then we transfer the color characteristics of reference image to the source image based on their corresponding sections. In this way, not only the original feature of source image can be maintained, but also the reference color characteristics can be transferred more matching to the source. During the color transfer between pairs of sections, there will be unnatural artifacts along the section boundary. So we interpolate the mean value and the standard deviation in the same histogram with different sections. In order to solve the extreme case, we

apply a structure control term into the scaling ratio whose concept comes from the HDR linear function. With the enhancement we proposed above, our color transfer algorithm has better results and suitable for all the applications of Reinhard’s method.

1.2 Overview

The flow of our proposed method is shown in Figure 1.1. First, user inputs a source image and a reference image. For example, if we want to modify a sunny seascape image with clouds to a dusk feeling, we can provide a nightfall image as the reference.

To enter the core of our algorithm, we have to convert these two images into certain color space. In this thesis, we use l α β color space, an orthogonal color space which is suitable for apply different operations in different color channels with confidence that undesirable cross channel artifacts won’t occur.

After color space conversion, construct the histograms in each l α β channels for both source and reference images, and find out the left and right thresholds for each histogram. According to the threshold, we can segment each histogram into three sections, called the left section, the middle section and the right section. In the same color channel, a histogram section of the source image has its corresponding section in the reference image. We will do color transfer from

the reference image to the source image section by section.

When we apply color transfer based on the section correspondence, section parameter interpolation is certainly needed. That means we can’t transfer color characteristics only from one section to another, the section will interpolate its parameter with another section which next to it, and then transfer it.

Finally, the color characteristics of reference image with dusk feeling will be transferred to the source image. Then we can get the nightfall seascape image with clouds.

The major contributions of this thesis are as follows: First, we apply histogram segmentation and section color characteristics transfer based on Reinhard’s method and have a better result. Second, the structure control term which is added to the scaling ratio makes the result maintain the original feature of the source image more correctly. Third, our approach is suitable for current application of Reinhard’s theory because of the same concept.

1.3 Thesis Organization

The rest of this thesis is organized as follows: In Chapter 2, we review related works about color transfer and colorization techniques. In Chapter 3, we introduce how we decide the segmentation threshold to do histogram segmentation and section parameter interpolation. Chapter 4 presents the structure control term to maintain the source image structure. The results and implementation are shown in Chapter 5. Finally, conclusions and future works are discussed in Chapter 6.

Chapter 2

Related Works

In this chapter, we review previous researches related to our work. We focus on colorization techniques and color transfer applications.

Figure 2.1

The color transferred result by Reinhard’s result Left: The source image

Middle: The reference image Right: The result

Reinhard et al. [2] describe an automatic system for transferring the color characteristics from one color image to another. This system works by modifying the mean and the standard deviation of colors in the image that matches the

three-dimensional distribution of color values between the images and then transforms the color distribution of the reference image to match the distribution of the source image. Figure 2.1 shows the result image which its color characteristic is transferred from the reference image to the source image.

Welsh et al. [3] colorized a gray image by transferring color from a source, color image, to a destination, gray image. First, each image is converted into the l

α β color space. Next, each pixel in the grayscale image arrange in scan-line order

and select the best matching sample in the color image using neighborhood statics according to the luminance channel. The chromaticity values (α , β channels) of the best matching pixels are then transferred to the grayscale image by using Reinhard’s method to form the final mage. Figure 2.2 shows the results. In general, Welsh’s method works well on scenes where the image is divide into distinct luminance clusters. However, their current technique does not work very well with the image of similar luminance.

Figure 2.2

The colorized result by Welsh’s method. Left: The source image; Middle: The reference image; Right: The colorized result.

Levin et al. [4] supposed that luminance is related to color. Their method is based on a simple premise: neighboring pixels in space-time that have similar

image with a few color strokes, and the indicated colors are automatically propagated in both space and time to produce a colorized image or sequence by solving an optimization problem. Figure 2.3 shows a gray image marked by the user’s color strokes and the corresponding colorization result.

Figure 2.3

The result of Levin’s method. Left: The original image and user drawing stroke; Right: The colorized result image

Finding a good match between a pixel and its neighborhood in a grayscale image and a pixel in the reference image is not sufficient for a successful colorization. Pixels with the same luminance value and similar neighborhood statistics may appear in different regions of the reference image, which may have different semantics and different colors. Moreover, it is tedious to scribble on an image with complex scenes. In Irony et al., a method is presented for automatically generating such scribbles from an example image provided by the user.

Irony et al. [5] uses a supervised learning technique to classify in the feature-space and a voting technique to increase the spatial consistency of the colorized image. Next, transfer color only to pixels with high - level of

confidence, and feed the colored pixels as “micro-scribbles” to the optimization algorithm of Levin et al., which produces the final result. This technique assumes that a similar example image is pre-segmented. Otherwise, the task of segmenting the example image itself can be almost as hard as colorizing the input image.

Figure 2.4 shows the Irony’s method.

(a) Reference image along with a partial segmentation

(b) Classification and resulting colorization

Figure 2.4 Irony’s method requires an example image with a few user-marked or automatically segmented regions

To supply more user control, an application of Reinhard’s algorithm , Wen et al. [6] presented a multiple local color transfer system with a stroke-based user interface. Through their system the user can easily enhance a defect (source) photo by referring to some other good quality (reference) images simply by drawing some strokes. The user draws several strokes on the source and target images to indicate corresponding regions and also the regions that want to preserve. Then the system will perform the multiple local color transfer

automatically. Finally, the defect image can be enhanced. Figure 2.5 shows the process of Wen’s method.

(a) (b)

(c) (d) Figure 2.5

(a) Strokes on the source photo. The light blue stroke was drawn for preserving the background; (b)The reference image; (c)The source image; (d)The result

Based on Reinhard’s method, all above researches work very well not only on colorization but also on color characteristics transfer. As we mentioned before, the improvement on Reinhard’s algorithm by applying histogram to it and adding a structure control term to the scaling ratio can generate better quality results. In this way, all of the methods which are based on Reinhard’s algorithm also can be improved

Chapter 3

Histogram Segmentation

In this chapter, we show how to segment the histogram of source and reference images. The selection of threshold that is used to do histogram segmentation is discussed in section 3.1. Section 3.2 introduces the parameter interpolation before we do color transfer and the algorithm of our proposed method.

3.1 Segmentation Threshold

In this thesis, we assume that the histogram of each color channel consists of three sections: the major section, the left extreme section and the right extreme

section. The major section is a cluster of color channel values that close to the

represents the border characteristic. Figure 3.1 shows the concept of our histogram.

Figure 3.1 Blue buckets comprise the main color section, the left extreme section is yellow, and the green is the right extreme section.

In the segmentation, we need to find out two thresholds, called the left and right

thresholds to segment the histogram into three sections. As mentioned before, the

major section collects the color values that close to the mean of the whole channel. However, the left extreme and right extreme sections collect the color values that close to the borders. First, we compute the mean value m of the whole color channel, and m separates the histogram into two temporary sections, the left and right sections. Then, compute the mean value of these two sections mL and mR. The mL

and mR are the left and right thresholds of the histogram.

section. On the other hand, color channel values that larger than the right threshold constitute the extreme right section. Finally, the remaining color values that between the left and right threshold form the major section. In this way, we can find out the major characteristic and the border characteristic of the color channel.

3.2 Parameter Interpolation

Generally, Reinhard et al. do color transfer by using non-segmented histograms. It means that the entire histogram belongs to one section. They shift the mean value of the histogram of the source image to that of the target image. Then transform the color distribution of the histogram of the reference image to match the distribution of the histogram of the source image according to the ratio of the source image’s standard deviation and the reference image’s standard deviation. Based on their algorithm, color characteristics can be transferred between “pairs of sections” successfully.

In comparison with our method, we segment each histogram into three sections, represent one major section and two border sections respectively. Transferring color characteristics based on the section names (main to main, border to border) make the main color characteristics of the source image matches the main color characteristics of the reference image and so as the border. In this way, we can get better result than Reinhard’s method.

Although the local color characteristics can be transferred successfully, the global color characteristics between the sections that in the same image may be destroyed because of the histogram segmentation. Such artifact which caused by transferring color characteristics between multiple sections with different mean value and different standard deviation does not exist in Reinhard’s method that using one section. Figure 3.2 shows the problem of multiple section color transfer.

Figure 3.2 The concept of multiple section color transfer. (a) The source image without histogram segmentation; (b) Source image with histogram segmentation; (c) The Reinhard’s method after color transfer; (d)Our method without interpolation after color transfer

ratio after color transfer in Reinhard’s method but destroyed in our method.

Figure 3.3 shows the result of our method without parameter interpolation compare to Reinhard’s method.

Figure 3.3

The source image (a) and the reference image (b). Reinhard’s result (c) does not have artifacts compare to our multiple section color transfer (d).

To overcome this artifact, we do parameter interpolation before color transfer. Consider Reinhard’s color transfer function for each color channel defined as:

p

_{= r p − 𝑚}

𝑠

+ 𝑚

, p ∈ source image

(1),

where ms is the mean value of color channel of the source image, mr is the mean

value of color channel of the reference image, and r is the scaling ratio term that

𝑟 =

𝜎𝑠

(2),

σ is the standard deviation. According to Reinhard’s algorithm, each pixel applies Eq.(1) in l α β channels and then gets the final transferred result. In our method, color channel is segmented into three sections, each section has its own mean value and standard deviation. If we apply Eq.(1) directly to these sections, the artifacts will occur. So we do parameter interpolation with respect to the ratio term r and the mean value ms,mr between section and sections.

An additional parameter cm which is called the middle value is used to be the

threshold when we do parameter interpolation. It is defined by the average of the minimum and maximum in the color channel. Before we apply color transfer to the color channel value which is smaller than the middle value, the scaling ratio and the shifting term in Eq.(1) will be interpolated with the r, ms,mr terms that in the major

section and the left extreme section, and so as the color channel value that greater than the middle value whose scaling ratio and shifting term will be interpolated with the terms in the major section and the right extreme section.

the average of the minimum and maximum of that section. The weigh w that we use to do linear interpolation based on the distance between the color channel value and the section middle value in the source image. The following algorithm shows the interpolation process. The shifting term is interpolated by the mean value m in two of the histogram sections with w and we interpolate the standard deviation σ with the same w in order to get the scaling ratio r.

Algorithm 1: Parameter Interpolation

1: for all color channel value v < middle value 2: dvl = distance (v, cl) ; 3: dvm = distance (v, cm) ; 4: dtotal = dvl + dvm ; 5: wvl = weight(dtotal , dvl) ; 6: wvm = weight(dtotal , dvm) ; 7: ms = interpolate(wvl, wvm, msl, msm) ; 8: mr = interpolate(wvl, wvm, mrl, mrm) ; 9: σs = interpolate(wvl, wvm, σsl, σsm) ; 10: σr = interpolate(wvl, wvm, σrl, σrm) ; 11: r = σ_σr s ; 12: end for

,where dvl means the distance from color channel value v to the middle value of left

extreme section and msl means the mean value in the left extreme section of the source

image. If the color value v larger than the middle value, then all of the parameter in the left extreme section will be replaced by the right extreme section parameters. After parameter interpolation, we use the new ms,mr and r to do color transfer. Figure 3.4

shows the transferred result after parameter interpolation. The distance weight respect to the source image, not only remove the artifact between sections but the local color characteristics transferred successfully.

Figure 3.4

Left : Multiple section color transfer without section parameter interpolation; Right : Multiple section color transfer with section parameter interpolation.

Chapter 4

Structure Control Term

The standard deviation of reference image equals to 0 breaks the structure of source image in Reinhard’s method because of the definition of scaling term based on the ratio of standard deviation about source and reference images. To solve this problem, we add a structure control term to the scaling term to maintain the original structure of source image.

In this chapter, we briefly describe the relation between the scaling term and the structure control term. The reason that makes the scaling term to zero will be

discussed in section 4.1. And the definition of structure control term will be show in section 4.2.

4.1 Structure Control Term Motivation

1 can be simplified as:

p

_{= 𝑚}

𝑟

(3),

in this way, the final color transferred image equals to the pure average color of the reference image. Figure 4.1 shows an example.

Figure 4.1. Left top: The source image; Right top: The reference image; Left down: Reinhard’s method; Right down: Our method with structure control term

We can find out that if the reference image is a pure color image, and the final transferred image is a pure color image, too. The standard deviation of a pure color

way, the final result lost its original structure even though the standard deviation of the source image not equals to zero.

To avoid this phenomenon, we add a structure control term into the scaling term

r that maintain the source image structure according to the ratio of the standard

deviation of both source and reference image. We describe the definition of the structure control term in the following section.

4.2 Structure Control Term Definition

When the standard deviation of reference image closes to zero, we use the structure control term to maintain the structure of source image. On the other hand, if the standard deviation of reference image far away from zero, we use the structure control term to transfer the color characteristics according to the ratio of standard deviation like Reinhard’s method. Equation 4 shows our definition of the structure control term.

c =

δ

∗ σ

σ

+ 1

4

The δ is a percentage parameter that for user to control the effect of structure control term on the scaling term. Note that the standard deviation of reference image closes to zero makes the structure control term c closes to δ * σs. And the bigger the

σr , the smaller the c is. This concept is similar to the HDR linear function which is

mentioned by Reinhard et. al [7]. Equation 5 shows the final color transfer function with the structure control term.

p

_{= r + c p − 𝑚}

𝑠

+ 𝑚

, p ∈ source image (5)

With the structure control term, we can maintain the original structure of source image as the right down image of Figure 4.1 does.

Chapter 5

Results

The results of our proposed thesis are presented in this chapter. We compare our method to Reinhard’s algorithm. Figure 5.1~5.3 show the results about the scenery. Figure 5.1(a) is the source image: the seascape in the sunny day. Figure 5.1(b) is the reference image: another seascape in the dusk. Figure 5.1(c) shows the Reinhard’s result. Figure 5.1(d) is our result.Figure 5.2(a) is the source image: the same as Figure 5.1(a). Figure 5.2(b) is the reference image: a seascape in a cloudy day. Figure 5.2(c) is the Reinhard’s result. Figure 5.2(d) shows our result.

(a) (b)

(c)

(d)

Figure 5.1. (a) The source image; (b) The reference image; (c) Reinhard’s result; (d) Our result.

(a) (b)

(c)

(d)

Figure 5.2. (a) The source image; (b) The reference image; (c) Reinhard’s result; (d) Our result.

(a) (b)

(c)

(d)

According to Figure 5.1 and Figure 5.2, the lightning effect on the cloud that in the source image is blurred and decreased in Reinhard’s method. With our histogram segmentation algorithm, we generate better results. Figure 5.3 shows a result that transfer the sunny day to the cloudy day.

Figure 5.4 and 5.5 shows the result about the plant. We not only transfer the flower color but successfully maintain the structure of the flower stamen in Figure 5.4. Compare to Reinhard’s method in Figure 5.5, our method transfers the color correctly.

(a) (b)

(c)

(d)

Figure 5.4. (a) The source image; (b) The reference image; (c) Reinhard’s result; (d) Our result.

(a) (b)

(c)

(d)

Figure 5.5. (a) The source image; (b) The reference image; (c) Reinhard’s result; (d) Our result.

Chapter 6

Conclusions And Future

Work

In this thesis, we proposed a novel method for color characteristics transfer using histograms. The proposed method is composed of three main ideas: histogram

segmentation, section interpolation, and the structure control term.

The histogram segmentation and section interpolation can produce good color characteristics transfer results by matching color characteristics according to the histogram sections. With the structure control term, our method successfully

maintained the structure of source image even in extreme cases. These contributions make the thesis of color transfer more integrity.

Although our algorithm generates better results than Reinhard’s method, there exist improvements. Consider the three color characteristics sections that we used, if we segment the histogram into more than three sections, the quality of the final color

histogram sections for an image becomes one of the future works about our algorithm. The method that we used to interpolate between histogram sections can also be

improved even that current linear interpolation method is acceptable. Histogram sections parameter interpolation by using the color distance metric which is similar to Gooch’s method[8] becomes a considerable method when we interpolate histogram parameters.

Finally, not only the single image color transfer method we improved, but the application on video contains additional time complexity is our goal in the future.

Reference

[1] D.L. Ruderman, T.W. Cronin, and C.C. Chiao, “Statistics of Cone Responses to Natural Images: Implications for Visual Coding,” J. Optical Soc. of America, vol. 15, no. 8, 1998, pp. 2036-2045.

[2] Reinhard E., Ashikhimin M., Gooch B., Shirley P.: Color transfer between images.

IEEE Computer Graphics and Applications 21, 5 (2001), 34–41.

[3] Welsh T., Ashikhmin M., Mueller K.: Transferring color to greyscale images.

ACM Transactions on Graphics 21, 3 (2002), 277–280. (SIGGRAPH 2002

Conference Proceedings).

[4] Levin A., Lischinski D., Weiss Y.: Colorization using optimization. ACM

Transactions on Graphics 23, 3 (2004), 689–694. (SIGGRAPH 2004 Conference

Proceedings).

[5] Irony R., Cohen-Or D., Lischinski D., “Colorization by example”, Eurographics

Symposium on Rendering, pp. 277–280, 2005

[6] Chung-Lin Wen, Chang-His Hsieh, Bing-Yu Chen, Ming Ouhyoung. Example-based Multiple Color Transfer by Strokes. Pacific Graphics 2008.

[7] Reinhard E., Stark M., Shirley P., Ferwerda J. Photograpic Tone Reproduction for Digital Images. ACM Transactions on Graphics (SIGGRAPH 2002 Conference Proceedings).

[8] Gooch Amy A., Olsen Seven C., Tumblin J., Gooch B. Color2Gray: Salience-Preserving Color Removal. SIGGRAPH 2005.

[9] KONUSHIN V., VEZHNEVETS V., “Interactive image colorization and recoloring based on coupled map lattices” Graphicon, pp. 231–234, 2006 [10] Luan Q., Wen F., “Nature Image Colorization”, Eurographics Symposium on

Rendering, 2007

[11] QU Y., WONG T., HENG P., “Manga colorization”, ACM Transactions on

Graphics , pp. 1214–1220, 2006

[12] Wang Z. M., You S. A., Huang M. L., “An Efficient Colorization Algorithm to Chinese Ink Paintings Using A Fuzzy Segmentation Technique”

[13] YATZIV L., SAPIRO G., “Fast image and video colorization using chrominance blending”, IEEE Transaction on Image Processing, 2006

[14] D.R.J. Laming, Sensory Analysis, Academic Press, London, 1986. [15] G. Wyszecki and W.S. Stiles, Color Science: Concepts and Methods,

Quantitative Data and Formulae, 2nd ed., John Wiley & Sons, New York, 1982.

[16] P. Flanagan, P. Cavanagh, and O.E. Favreau, “Independent Orientation-Selective Mechanism for The Cardinal Directions of Colour Space,” Vision Research, vol. 30, no. 5, 1990, pp. 769-778.

[17] S.N. Pattanaik et al., “A Multiscale Model of Adaptation and Spatial Vision for Realistic Image Display,” Computer Graphics (Proc. Siggraph 98), ACM Press, New York, 1998, pp. 287-298.