CHAPTER 1 INTRODUCTION
1.3 T HESIS O RGANIZATION
The rest of this thesis is organized as follows. In Chapter 2, we review the related works in non-photorealistic rendering. Then, in Chapter 3, we introduce characterized Chinese fine-brushwork painting on animals. Chapter 4 describes the details of the component of our stroke generator. The rendering procedure is presented in Chapter 5. The experimental results of our technique are demonstrated in Chapter 6. Finally, conclusions and future works are given in Chapter 7.
Figure 1.1 The system flowchart.
Chapter 2
Related Works
2.1 Stroke-Based Rendering
Haeberli [11] developed an interactive system, which enables a user to place strokes on a canvas. The stroke distribution is given with an input device such as a mouse. A stroke color is chosen from the colors in the corresponding region of the source image. A stroke orientation as well as size can be controlled by the user. In addition to that, he proposed a method to control the stroke orientation automatically using the intensity gradient of the source image.
Litwinowicz [14] takes a video sequence as its input. This work concentrates on exploiting the frame coherence for a painterly animation sequence. The jittered grid is used as the initial stroke distribution. A stroke color is determined by taking from the source image just as Haeberli’s work. The stroke size may vary, because a stroke may be clipped by the edges detected in the source image to preserve the contours of objects. However, the stroke width is constant over the canvas. The orientation is determined by the intensity gradient, which is similar to Haeberli’s method [11] as well. The painting order is randomized to obtain a hand-crafted touch.
Hertzmann [12] introduced a method to automatically paint still images with various stroke sizes and shapes and then extended the technique to video [13]. His method composes an image with several layers. Each layer is painted by the strokes with its own constant width.
The upper layer is painted with narrower strokes. As a result, the canvas image gradually gets close to the source image. The resulting image contains strokes with different widths. The orientation of spline curves are controlled by the intensity gradient.
2.2 Chinese Ink Painting
The researches about traditional Chinese painting are not as many as the Western painting arts like watercolor [7], pen-and-ink [18][19][20][26][27], oil painting[12] and so on.
In order to simulate stroke characteristics in Chinese Landscape Painting, Weng et al [25]
introduced a two-dimensional brush model. The bristles are distributed uniformly as concentric circles, and adding some physics properties such as pressure and ink decline. Later, Huang [15] proposed a new method physically imitating the behavior of ink diffusion in paper structure. He used particles representing carbon and water, simulating the physical phenomenon of diffusion on the paper.
Way and Shih [23] focused on Chinese landscape painting of Tang dynasty (618-907 AD) and Song dynasty (960-1279 AD) and synthesized rock textures by drawing Hemp-Fiber texture strokes and Axe-Cut texture strokes respectively. Way et al [24] then developed another application to simulate the Chinese painting of trees by using silhouette and texture strokes. Three-dimensional tree models were used as their source models, and they can draw various styles of bark texture by defining the texture patterns.
Borrowing from [12] [14], we intend to develop a semi-automatic rendering system to composite the painting style of the specific Chinese fine-brushwork painting on animals.
Chapter 3 Background
3.1 The History about Chinese Painting on Animals
In the Tang dynasty (618-907 AD), in addition to figures and landscapes paintings that had already reached the extremely high realm, flowers, birds and animals also formed the individual style of drawing. And also many artists were fond of still life paintings on animals.
At that time, Ts’ao Pa（曹霸）was one of the most famous master with the divine horse painting. One of his student, Han Kan（韓幹）, as a Chinese proverbial saying “the pupil surpasses the master”, was an expert at painting in according to actual live horses and was the first one who promoted still life painting in China.[3]
In the middle period of Qing Dynasty, the animal and flower painter, Lang Shining（郎士 寧） (1688~1766), or Giuseppe Castiglione as his Italian name, came to China as a Jesuit missionary and brought his skill of European painting. He turned a Chinese court artist with blending of Chinese and Western painting techniques.
In the twentieth century, more and more drawings of animals came out. Xu Beihong（徐 悲鴻）was a celebrated artist and the painted horses are full of passion and power underneath
his brief and semiabstract strokes. Figure 3.1 shows representative works of these artists.
(a)
(b)
(c)
(d)
Figure 3.1 (a) Grazing horses by Han Kan. (b) Horse by Xu Beihong [4]. (c) Eight horses by Lang Shining. (d) Rabbits (A year of good harvest) by Chang Ke-Chi（張克齊）[2].
In this thesis, we focus on highly realistic depiction in describing nature objectivity and catching accurate form on animals as combining Western drawing with Chinese calligraphic strokes. One example is shown in Figure 3.1(d).
3.2 Fine-Brushwork Painting on Animals
There are diverse skills to paint an animal in Chinese fine-brushwork. Referring to [1], this book introduces several skills to paint the head and the hair of cats and tigers. In regard to the hair, we simulate one method which is presented in the book. This method is called “hair painted with a flattened brush”（破鋒絲毛法）. This method is described as follows and takes cats for its painting subject.
3.2.1 Hair Painted with a Flattened Brush
When painting the hair of the cat with a flattened brush, the brushwork should be expressive of the texture of the hair.
1. The sketch of the form of the cat starts from the head.
2. Then proceed to the neck, the back, the abdomen, the legs, and the tail.
3. When rendering the hair, the procedure is the same as above.
There is a sense of dryness in the flattened brushwork. Therefore, after the hair is rendered with flattened brushwork, apply wet washes of ink mixed with cyanine to the whole body of the cat to enrich the texture. Examples are illustrated in Figure 3.2.
(1) (2) (3)
(4) (5)
Figure 3.2 Examples of hair painted with a flattened brush excerpted from [1].
We observe the painting style carefully and summarize it as follows. The orientation of brush is greatly related to the contour coherence of the subject figure. The stroke distribution is regularly near the edges. The variation of each stroke length is small. Although the colored wet brushes are used in gradation to flesh out the contour, the hair brushes are still apparent and noticeable. These heuristics are carried out by means of image analysis and processing in our system.
Chapter 4
Stroke Generation
In this chapter, we introduce the stroke generator component of our system. The overview of our stroke generator is shown in Figure 4.1.
The stroke generator we presented takes four objects as input: 1. the reference image; 2.
the precision map; 3. the color fusion image and its region map; and 4. the style parameters.
The reference image is a photograph of the subject, such as a dog or a cat. In Section 4.1, the precision map specified by the user is to control the tone and the fineness of the stroke distribution. In Section 4.2, the color fusion image and its region map are generated from the reference image used to extract edges. The relevant style parameters used to generate artistic renders are introduced in Section 4.3. Then we describe the kernel of our stroke generator in Section 4.4.
Figure 4.1 Overview of our stroke generator.
Reference Image Fusion Image
& Region Map
Stroke Generator
Color Fusion
Parameters
Precision Map
Layers of Strokes
4.1 User-Specified Precision Map
Inspired by [9], our system proposes a simple model of precision, controlled spatially through the precision map. It is common artistic practice to use coarse tones for regions of little importance, and fine spatial details for regions of focus. Take animals for example, the artist would like to keep the detail information for facial features like the eyes, the nose and the teeth, and leave the other regions, say, the body, in rough.
To make the precision map as shown in Figure 4.2, the user first draws the regions of interest (ROI) interactively. By the mouse drawing, the user directly draws the shape of regions over the reference image. To facilitate the designation, we provide three different type of drawing: freehand, rectangular and elliptic. After making ROIs, an intermediate image is created by first filling the background in black and then drawing the pixels within the ROIs in white. In case it causes some aliasing near the boundaries of ROIs, we blur the intermediate image by Gaussian filter and thus the precision map is. The precision map is used as a gray-level mask in the stroke generator and the wash renderer.
(a) (b) (c) (d)
Figure 4.2 The user specified precision map. (a)(c) Original images. (b)(d) The corresponding precision map defined by the user.
4.2 Color Fusion
In the fusion step, extensive use was made of region adjacency graphs (RAG) and graph contraction with a union-find algorithm [21]. The RAG provides a spatial view of the image.
A RAG consists of vertices (or nodes) associated with each region and lines joining each pair of adjacency regions. By definition, this region adjacency graph provides a simple-connectivity view of the image as illustrated in Figure 4.3(b).
(a) (b)
Figure 4.3 Example of region adjacency graph (RAG).
The initial RAG was built from the reference image. The fusion was performed as a transitive closure operation on the graph, under the condition that the color difference between adjacent nodes should not exceed hr/2 where hr is the user-defined threshold. At convergence, the color of the regions was recomputed and the transitive closure was again performed. After at most three iterations the final labeling of the image (segmentation) was obtained. Small regions (the minimum size, M is defined by the user) were then allocated to the nearest neighbor in the color space. The algorithm is as described in Table 4.1.
Input: for each p do begin { Traverse the image labeling each new region encountered } if Labels(p) = -1 { If this region has not yet been labeled, label it }
Labels(p) = ++label;
Modes(p) = LUV(p) { Copy region color into modes } Fill(p, label);
{ Populate labels with label for this region via an 8-connected fill}
end end
iteration = 0; deltaRC = 0; { deltaRC is difference of region # between iteration } repeat { Apply transitive closure iteratively to the regions }
TransitiveClosure();
{ Update Labels(p) & Modes(p) until the each neighboring region color < sigmaR } until deltaRC > 0 or iteration > 10
Prune(minRegion);
{ Prune spurious regions (regions whose area is under MinRegion) } Table 4.1 The algorithm of color fusion.
After applying the color fusion operation, we will get a color-fused image and its region map. The color-fused image will be used to guide the stroke distribution, and the region map will be used to avoid spreading strokes in the background region while the background region index is defined. Figure 4.4 illustrates examples of color fusion with different parameters.
(a) (b) (c) (d) Figure 4.4 Examples of color fusion. (a) Original image of a cat. (b) (sigmaR, minRegion)
= (6.5, 20) of (a). (c) Original image of a puppy. (d) (sigmaR, minRegion) = (6.5, 20) of (c).
4.3 Primitives and Style Parameters
In this section, we introduce the primitives and parameters used in our system.
4.3.1 Strokes, Layers and Canvas
Each brush stroke has several properties that are determined by the analysis of input images. Our brush stroke definition most closely resembles that of [14]. Each brush stroke is an object with the following properties: 1. Anchor point in image coordinates; 2. Angle of orientation in radius; 3. Width in pixel; 4. Lengths in both directions from the anchor point in pixels; 5. Color (R, G, B). Anchor, angle, width, and lengths are kept in floating-point coordinates. A brush stroke object is illustrated in Figure 4.5.
Figure 4.5 Brush stroke object.
In order to emulate the coarse-to-fine painting process, we extend the concept of Layers
which is introduced in [12]. Layers are disjoint groups of brush strokes representing successive passes of refinement in a painting. As used here, brush strokes are created within a Layer only when perceptually necessary as determined by the presence of edges at the frequency band corresponding to each Layer. We set an additional layer of ROI to draw the finest strokes in the region within the precision map in order to keep the greatest detail of the animal features.
Output is rendered onto an arbitrarily large canvas. Using a canvas of higher resolution than an input image is analogous to a painter filling an entire canvas according to a small reference photograph. It allows the output rendering to retain the structural coherence of the input image while still displaying considerable artistic styling.
4.3.2 Style Parameters
In order to produce a painterly rendering, the user should first pick a style. Here a style is an encapsulation of parameters that control the analysis of input images. Our style parameters are shown in Table 4.2.
Brush Width wb Width of the brush stroke
Stroke Generation Width wr
The minimum window size of unoccupied space on the canvas for which a new brush stroke will be created to fill
Color Brush Texture TCb Set of brush texture and mask to use in wash rendering
Hair Brush Texture THb Set of brush texture and mask to use in hair rendering
Color Fusion sigmaR
minRegion
sigmaR is the color difference between neighboring regions, and minRegion is the minimum size of a region
Background Region Index rbg Background region index to avoid spreading strokes in the background region
Gradient Threshold Tg Threshold of gradient magnitude used at stroke orientation stage.
Basis Point Radius Rbpt Radius used at stroke orientation stage to control the dense of basis points
Canny Edge Detector Tlow,
Tup
Lower threshold and upper threshold used in Canny edge detector
Mean Shift Filter hs
hr
hs is the spatial bandwidth and hr is the color bandwidth for mean shift filtering
Canvas Size Cs Canvas size scale factor
Convolution Size m1
m2
m1 is Median convolution size used for detail and m2 is used as Gaussian blur size for non-detail parts of a image after wash brush rendering
Alpha Blending alpha Alpha blending value used in median
interpolation Table 4.2 Style parameters used to generate artistic renders.
These parameters are defined for each layer. The number of layers is also defined by the user. The wash brush textures and hair brush textures are especially made by a real Chinese painting brush.
4.4 Stroke Generation in Detail
Now we describe in detail each step in our process of the stroke generation. There are five stages of processing:
1. Initialization;
2. Stroke Anchor Generation;
3. Orientation;
4. Edge Clipping;
5. Color Extraction.
An overview of these stages is illustrated in Figure 4.6.
Figure 4.6 The stroke generation procedure.
4.4.1 Initialization
First all layers are initialized. The number of layers depends on the style which is determined by the user. Each layer contains an array of brush strokes which is initially empty. The layers will be filled, as necessary, in the next step.
4.4.2 Anchor Generation
In this step, we generate the anchor position of each stroke with the help of the region map. From the color fusion process, we get a color-fused image and its region map which labels where a pixel belongs to. By indicating the index of the background region, it is easy to avoid spreading stroke anchor in the background. We simply look up the region map (table) while trying to create a new stroke position. If the region of the position we picked is not the background region, the creation is success; otherwise it fails and goes on finding another position.
At the bottom layer, the canvas is searched and new brush strokes are created where gaps are found. Gaps are simply defined as area wr across that contains no brush stroke anchor point. To make strokes less disorder, we restrict the bottom layer strokes to areas “near” the edges but not “on” the edges. If the stroke generation width, wr, is made larger than the actual width of a brush stroke, it will allow the canvas to show through, that is, the effect of “white- reserving.” Oppositely, making wr smaller will make the brush strokes appear more crowded. The number of strokes is approximated by
The anchor point positions are generated randomly and are examined whether it is within areas near edges and is outside the background region. The canvas is searched in pseudo-random order (versus scan-line) in order to avoid certain mechanical looking artifacts that can result if brush strokes are generated in sequence and end up perfectly spaced.
The same process is performed on the higher layers (except the highest), but the generation is restricted to areas near edges of the appropriate frequency (see subsection 4.4.4). That is, we allow strokes generated “on” the edges in these layers.
For example, brush strokes in the middle layer are placed on or near the middle frequency edges in the color-fused image. The frequency edge images are illustrated in Figure 4.7. The dark green in the edge images are the detected edges at each frequency. The light green border is the area, proportional to wr, that brush strokes will be created to fill. Note how smoothly shaded and out of focus regions are refined while remaining perceptually similar to the input.
On the highest layer, called ROI layer, the generation is restricted to areas within the regions of interest (ROIs). ROIs are specified by the user during the precision map definition. The ROIs is searched and new brush strokes are created where gaps are found. Here the wr of the highest layer is relatively small to make strokes in dense.
This is to emulate the painter placing emphasis on some regions as animal facial features.
We believe proximity to edges is a more perceptually accurate description of how an artist might refine a painting than what previous techniques employed. In [12], it creates brush strokes in higher layers when a rendered version of the lower layer
differs too much in color from the original image. In Chinese fine-brushwork painting or other styles, painters worry about refining the structural accuracy of an image rather than correcting color in otherwise empty regions.
(b) (c)
(d) (e)
(a)
Figure 4.7 Frequency edge images. (a) Color-fused image. (b)- (e) are different frequency edges, from low to high.
4.4.3 Orientation
Painters typically paint by following contours of features in an image. Instead of orienting each brush stroke largely based on the local gradient estimates as suggested by [12][14], we believe that brush stroke orientation should be globally dictated only by the brush stroke lying on the strongest gradients. We achieve this by using radial basis functions (RBFs) to globally interpolate X and Y gradients from only the strongest gradients [5].
For each layer, the reference image is blurred by a Gaussian kernel of width proportional to wb. Gradients on the layer are then estimated across the entire image with a Sobel filter which is commonly used in image processing technology [10] to approximate the differentiation of the image in the continuous domain. The gradient of an image f(x, y) at location (x, y) is the vector:
And the magnitude of this vector which generally referred to simply as the gradient and denoted as ∇fmag:
The computation of gradient based on the Sobel kernel is
⎥⎥
And the stroke located at (x, y) is determined to be “Strong” or not as
⎩⎨
with a gradient threshold Tg, and not near other “Strong” brush strokes. The radius Rbpt keeps off the dense and numerous of strong strokes. Once the stroke is “Strong”, we set its gradient to be (Sx, Sy).
These “Strong” brush strokes, typically about 10 to 200 per layer, become the basis points for a radial basis function. Typically, all layers contribute “Strong” brush strokes to the same radial basis function in order to capture the gradients at different
These “Strong” brush strokes, typically about 10 to 200 per layer, become the basis points for a radial basis function. Typically, all layers contribute “Strong” brush strokes to the same radial basis function in order to capture the gradients at different