Chapter 1 Introduction
1.3 Thesis Organization
The rest of this thesis is organized as follows: In Chapter 2, we review related works about texture synthesis with control and solid texture synthesis. In Chapter 3, we present our approach for synthesizing solid textures from a 2D texture. Chapter 4 presents the proposed anisometric synthesis approach for solid textures based on vector field control. Chapter 5 shows the implementation and results. Finally, conclusions and future works are discussed in Chapter 6.
4
Input 2D Texture Vector Field
Feature Vector Generation
Anisometric Field
Inverse Anisometric Field Similarity Set
Generation
Pre-process
Upsampling
Jitter Synthesis Process
Correction
Result
Figure 1.1 System flowchart 3D-candidate
Generation
5
Chapter 2
Related Works
In this chapter, we review some recent and representative works.
2.1 Texture Synthesis with Control Mechanism
Ashikhmin [1] presented a texture synthesis algorithm for natural textures. He provided an interactive interface for users to control the texture synthesis process. He introduced an idea of “shifted-candidates” for neighborhood matching and found the best similar candidate. This approach uses a smaller neighborhood to obtain the quality characteristics of a larger neighborhood and maintains the coherence of the results.
Users can use painting-style interface to indicate large-scale properties of the texture.
Furrhermore, this method is fast and straightforward for the users. But it could not obtain good results if the user’s control does not contain significant amount of high frequency components.
6
Lefebvre and Hoppe [12] introduced a high-quality and parallel pyramid synthesis algorithm. Their method includes upsampling step to maintain patch coherence, jittering of exemplar coordinates to increase the randomness of the texture, and an order-independent correction step to keep the similarities between synthesized results and input textures. They obtained high-quality and efficient results thanks to the order-independent correction step. It corrects the pixel coordinate for more accurate neighborhood matching and the whole step can be divided into several subpasses to promote performance. However, it still has one drawback: when the features are too large to be captured by small neighborhoods, it could perform poorly. This drawback is also a well-known problem for other neighborhood-based per-pixel synthesis methods.
Lefebvre and Hoppe [13] proposed a structure for exemplar-based texture synthesis with anisometric control. They replaced traditional RGB values with appearance vectors for neighborhood matching. Their appearance space reduces runtime neighborhood vectors from 5×5 grids to only 4 locations, so the synthesis is more efficient, and the quality of results is good because of the information-rich appearance vectors. In addition, they combined their pyramid synthesis with appearance vectors to accelerate neighborhood matching and proposed novel methods for coherent anisometric synthesis which makes arbitrary affine deformation on textures. They presented a convenient way to control textures.
Kwatra et al. [9] introduced a method to control flows on 2D textures and presented an algorithm to perform texture control on 3D surfaces [10]. They proposed a vector advection technique with global texture synthesis to accomplish dynamically changing
7
fluid surfaces. Users can define fluid velocity field to control the texture results on 3D surfaces. The neighborhood construction step in the process considers orientation coherent with the user-defined velocity field. This method not only makes the synthesized results similar to the input texture, but also keeps temporally coherent.
However, it is difficult for users to define an orientation velocity field which is smooth everywhere.
2.2 Solid Texture Synthesis
Jagnow et al. [7] used traditional stereological methods to synthesize 3D solid textures from 2D images. They synthesized solid textures for spherical particles first and then extended the technique to apply to arbitrary-shaped particles. Their approach needs cross-section images to record the distribution of circle sizes on 2D slices, and then builds the relationship of 2D profile density and 3D particle density. Users could use the particle density to add one particle at a time to reconstruct the volume data, so it means the step is manual. This method uses many 2D profiles to construct 3D density for volume results, which are good for marble textures. However, their system is not automatic and only for particle textures.
Chiou and Yang [2] improved the above system to automatic process. They divided the synthesis into two parts: 2D analysis phase and 3D synthesis phase. First, they collected essential statistics to develop a probability model. Then they used this probability model to control the variation in particle size through the 3D synthesis procedure. However, this system inherits the above system so it is only for particle textures.
8
Qin et al. [16] introduced an image-based solid texturing in terms of basic gray-level aura matrices (BGLAMs) framework. They replaced traditional gray-level histograms with BGLAMs for neighborhood matching. They defined aura matrices from input exemplars and generated a solid texture from multiple view directions. In the volume result, they will only consider the pixels on the three orthogonal slices for neighborhood matching. The whole system is fully automatic without user interactions.
Furthermore, they can generate reliable results for both stochastic and structural textures.
However, it needs large storages for large matrices and the results are not good for color textures.
Kopf et al. [11] provided a solid texture synthesis method from 2D exemplars. They took advantage of 2D texture optimization techniques to do 3D solid texture synthesis and then preserved global statistical properties by histogram matching to achieve optimization. For each voxel, they only considered the neighborhood coherence in three orthogonal slices, and increased the similarity between the solid textures and the exemplar iteratively. Their approach could do well for wide range of textures. But they synthesized the texture with the information on the slices.
Takayama et al. [17] presented a method which fills a model with anisotropic textures. They had some volume textures and defined it how to map to 3D objects. Then they pasted solid texture exemplars repeatedly on the 3D object. Volumetric tensor over the mesh can be set by users, and the texture patches are located according to these fields. This method still has drawbacks. The patch seams became noticeable when using a texture with strong low-frequency components and the blurring artifacts appeared
9
when using highly structured textures.
Dong et al. [4] introduced a new algorithm to restrict synthesis to a subset of the voxels, while enforcing spatial determinism. They reduced the dependency chain of neighborhood matching, so that each voxel only depended on a small number of other voxels. They synthesized a volume by using pre-computed 3D-candidates, each being a triple of interleaved 2D neighborhoods. These 3D-candidates are selected carefully to form consistent triples. Their approach could generate good results efficiently. However, if the three exemplarsdo not define a coherent 3D volume, the quality of the result will be poor.
2.3 Texture Synthesis and Vector Field
Many patterns are created by interactions between texture elements and surface geometry, so Turk [18] synthesized a texture directly on the surface of the model. An orientation field must be specified over the surface in order to preserve the directional nature of textures. They did this by allowing users to pick the directions at several locations and then interpolating vectors over the rest of the surface.
Ying et al. [22] presented a method that synthesized the texture directly on the surface, rather than synthesizing a texture image and mapping it to the surface. A 2D vector field was used to specify the correspondence between orientation on the surface and orientation in the domain of the example texture. A pair of orthogonal tangent vector fields is used for the purpose. They got one field first, and the second field is computed as the cross-product of the first field and the oriented surface normal.
10
Taponecco et al. [19] used Markov Random Field (MRF) texture synthesis method to implement 2D vector field visualization. In their approach, they defined a vector field by magnitude and direction. The magnitude could be easily computed using an appropriate norm of the values. Assigning the direction required a projection of the vector onto the image plane, and then the angle of the tangent in the vector field relative to the x-axis is gotten. The magnitude and direction were used to scale and rotate the image.
Wu et al. [21] proposed an approach to directly synthesize texture on arbitrary surface with texture sample. It had a tangential vector field which indicated the desired growing orientation of the texture on the surface. First, users specified tangential vectors at a few seed triangles, and then they interpolated vectors at the remaining triangles to build a tangential vector field. They recursively mapped triangles to texture space until the whole surface is mapped completely based on the vector field.
11
Chapter 3
Solid Synthesis Process
In this section, we present our approach for synthesizing solid textures from 2D textures. In Section 3.1, we describe feature vectors in appearance space and how we get the feature vectors. Then we use the similarity set to accelerate neighborhood matching in Section 3.2. In Section 3.3, we keep the color coherence of the three orthogonal slices by computing 3D-candidates. We introduce how to apply 2D pyramid texture synthesis to solid texture synthesis in Section 3.4. The upsampling process increases the texture sizes between different levels, that each voxel in parent level generates eight voxels in children level. The jitter step perturbs the textures to achieve deterministic randomness. The last step in pyramid solid synthesis is voexl correction, using neighborhood matching to make the results more similar to the exemplar.
3.1 Feature Vector Generation
Solid texture synthesis using RGB color values for neighborhood matching needs
12
larger neighborhood size and data. Appearance vectors have been proved that they are continuous and low-dimensional for neighborhood matching. Hence, we transform the texture data values in color space into feature vectors in appearance space. As shown in Fig. 3.1, we transform texture data T into appearance space texture data T'.
Figure 3.1 Overview of texture data transformation
According to Lefebvre and Hoppe [13], we take the RGB values in 5×5 windows (Fig. 3.2 (b)) to construct appearance vectors for each pixel of an input texture T(Fig.
3.2 (a)). The exemplar T' consists of the feature vectors at each pixel. There are 75 dimensions (25 for grids and 3 for RGB) for each pixel in T', and then we perform principal component analysis (PCA) to reduce the dimensions for a transformed exemplar T~'
(Fig. 3.2 (c)).
Figure 3.2 The process for feature vector generation:
(b)
(a) (c)
13
(a) input texture data T (b) 5×5 windows structure for feature vectors (c) transformed exemplar T~'
3.2 Similarity Set Generation
By the k -coherence search method [24], searching from the candidates can accelerate neighborhood matching because we do not have to search from each pixel in the exemplar for neighborhood matching. Thus, we construct a similarity set to record the candidates similar to each pixel.
Based on the principle of coherence synthesis [1], searching candidates from the n
n× neighbors of pixel p in the exemplar T can accelerate the synthesis process.
Hence, we find the k most similar pixels from the n×n neighbors of pixel p in the transformed exemplar T~'
to construct the similarity set C1l...k(p) for pixel p, where k is a user-defined parameter, and l is the pyramid level, C1l(p)=p. Note that n is a user-defined parameter to control the window size for coherent synthesis. In the experiments, n is set as 7.
3.3 3D-Candidate Generation
First, we repeat the input texture thrice as three directional exemplarsTx, Tyand Tz. By Dong et al. [4], we generate a small set of 3D-candidates for each pixel of the three exemplars to build the relation between the three exemplars. The 3D-candidate is by three 5×5 slices (2D neighborhoods) (Fig. 3.3 (a)). It is important that a suitable candidate should be consistent across the crossbar. The crossbar is the strip that is intersected by two slices (Fig. 3.3 (b)). Therefore, we seek to minimize the color
14
disparity between the strips shared by interleaved slices.
Figure 3.3 The diagram of 3D-candidate:
(a) Three input exemplars Tx, Ty, Tz and a corresponding 3D-candidate.
(b) The crossbars are defined by the three slices.
(c) A consistent triple from each exemplar.
We define the triple to be the three coordinates which is pointed by the center pixel of the three slices. For each pixel of each exemplar, we form triples using the pixel and two neighborhoods from the other two exemplars. We select the triples which have the smaller crossbar error. The 3D-candidate set is composed of these triples. For instance, as shown in Fig. 3.3(c), assuming we want to compute a 3D-candidate set for
(a)
(b)
(c)
15
pixel p in Tx, we first find in Ty the K pixel strips best matching the orange line from
Tx, and in Tz the K pixel strips best matching the green line from Tx. We use the current pixel p as the third coordinate to produce all possible K2 triples, and then order them based on the crossbar error which is the sum of color differences for the three of pixel strips. In the experiments, we keep the 12 best triples as 3D-candidates for each pixel and set a value K as 65.
3.4 Pyramid Solid Texture Synthesis
3.4.1 Pyramid Upsampling
The pyramid synthesis method [6] synthesizes textures from coarse level to fine level. There are l+1 levels in synthesis process, l=0~log2m, where m is the size of the target texture. We apply this 2D pyramid synthesis method to 3D space.
Figure 3.4 Synthesis from one voxel to m×m×m solid texture
In our approach, we synthesize from one voxel to a m×m×m solid texture, from
SL
S0 ~ , where L=log2m, as shown in Fig. 3.4. We synthesize a volume data S in which each voxel S
[ ]
v stores three coordinate values, indicating ~'T , x ~' T , y T~z' S 0 S1 S2
=0
l l =1 l=2
16
exemplar’s pixel, respectively. First, we build a voxel and assign value (1,1), (1,1), (1,1) to it as triple coordinates. Then, we upsample the coordinates of parent voxels for next level, assigning each eight children the scaled parent coordinates plus child-dependent offset as
17 after constructing feature vectors. For each voxel v, we collect the feature values of its neighbors to obtain the neighborhood vectors Ns v x direction to obtain the neighborhood vectors Ns v x
l( ) , Ns v y
Fig. 3.5 shows that each direction has 4 diagonal locations for each voxel v .
18
We average the appearance values from 3 synthesized voxels nearby v+∆x as the new feature value at voxel v+∆x, and then use the new feature values at 4 diagonal voxel to construct neighborhood vector Ns v x
l( ) . Also, Ns v y
19
Fig. 3.6 shows the locations of 3 synthesized voxels for each neighbor.
Figure 3.6 Three sub-neighbors for each neighbor of voxel v : (a) in Ns v x
l( ) (b) in Ns v y
l( ) (c) in Ns v z
l( )
We use the similarity sets and coherence synthesis method in the searching process, utilizing the 4 voxels for each direction nearby voxel v . Taking direction x for example, for the neighbor voxel ix (ix =1~4), we can get the most similar 3 pixels (ix1,ix2,ix3) in exemplar T for voxel x ix from the similarity set. And then we the offset between voxel ix and voxel v from 3D space to 2D space to infer the
(a) (b) (c)
20
candidates (ix1v,ix2v,ix3v) in exemplar T for voxel v (Fig. 3.7(b)). In order to keep x color consistence in three directions, we use the 3D-candidate set to infer the other two coordinates (i(x→y)v , i(x→z)v ) in exemplars Ty and T (Fig. 3.7(c)). In addition, z directions y , z are done by the same steps.
Now, for each v , we can get a set of triple candidates TC1...k which point towards pixels in exemplars Tx, Ty, T . We compute the neighborhood vectors z Ns TC k x
l( 1... ) ,
y k
s TC
Nl( 1... ) , Ns TC k z
l( 1... ) by the averaged feature values from the 4 nearby pixels.
Finally, we sum up the total difference between Ns TC k x
l( 1... ) , Ns TC k y
l( 1... ) ,
z k
s TC
Nl( 1... ) and Ns v x
l( ) , Ns v y
l( ) , Ns v z
l( ) , then replace the triple coordinate for voxel v with the best matching triple candidate.
21
Figure 3.7 The process of forming triple candidates in direction x : (a) A voxel v is corrected.
(b) The candidates in exemplar Tx.
(c) Find the other two candidates in exemplar Ty and Tz.
(a)
(b)
T
xT
yT
z(c) i x1
i x2
i x3
ix1v
ix2v
ix3v
y)v
i(x→
z)v
i(x→
22
Chapter 4
Anisometric Synthesis Process
In this section, we present the proposed anisometric synthesis approach for solid textures with vector field control. In Section 4.1, we introduce 3D vector fields and how we generate anisometric fields and inverse anisometric fields with the 3D vector fields.
In Section 4.2, we introduce the differences between solid synthesis and anisometric synthesis. These different steps are upsampling and correction. The jitter step is the same as it in the solid synthesis process.
4.1 3D Vector Field
We need the user-defined 3D vector fields to implement anisometric solid texture synthesis. We use these 3D vector fields to control the result.
First, we design a 3D space which contains three orthogonal axes at every point, then we use mathematics formulas to control the three axes. Fig. 4.1 shows a 3D vector field with orthogonal axes at each point, and the space size is 5×5×5. We make the three
23
axes various, and expect that the texture results would be changed with the fields. For example, we design a circular field, and there will be a circular pattern on the texture.
Fig. 4.2 shows the 3D vector field with a circular pattern on XY plane. The vector field should be the same size as the texture result.
(a) (b)
(c)
Figure 4.1 5×5×5 3D vector field with orthogonal axes (a) XY plane (b) XZ plane
(c) three orthogonal axes at each point
24
(a) (b)
(c)
Figure 4.2 5×5×5 3D vector field with a circle pattern on XY plane (a) XY plane (b) and (c) are three axes at each point
For each level, we have to make the anisometric field A and the inverse anisometric field A−1 based on the user-defined 3D vector field. The anisometric field A is created by downsampling the 3D vector field, and we obtain Al for each level.
Afterward we inverse the Al to get inverse anisometric field Al−1 for each level. In anisometrc synthesis process, the upsampling and correction steps will refer to the fields
25
Al and Al−1 at each level.
4.2 Anisometric Solid Texture Synthesis
4.2.1 Pyramid Upsampling
The goal for upsmapling step in anisometric synthesis is the same as it in isometric synthesis. It synthesizes from coarse level to fine level. We upsample the coordinate values of parent voxels for the next level.
The difference is that the child-dependent offset for upsmapling step is dependent on the anisometric field A. The anisometric field A is used to compute the distance for spacing.
26 transformed exemplars ~'
Tx , ~' T , y ~'
Tz to make the result similar to the exemplars Tx, Ty, T according to the 3D vector field. z
Lefebvre and Hoppe [13] presented a method for anisometric synthesis which is able to reproduce arbitrary affine deformations, including shears and non-uniform scales.
They only accessed immediate neighbors of pixel p to construct the neighborhood vector N ( p)
sl . They used the Jacobian field J and the inverse Jacobian field J−1 to infer which pixel neighbors to access, and the results will be transformed by the inverse Jacobian field J−1 at the current point. We apply this method to 3D space.
First, we have to know which 4 voxel neighbors in each direction to voxel v. We use the inverse anisometric field Al−1 to infer the 4 warped neighbors for voxel v, and construct the three warped neighborhood vectors Ns v x
l( )
27
Fig. 4.3 shows the 4 warped neighbors for each voxel v for each direction. Their locations are changed from diagonal locations because of the inverse anisometric fieldAl−1.
28 warped synthesized voxels for each warped neighbor.
]
29
(a) (b) (c) Figure 4.4 Three warped sub-neighbors for warped neighbors of voxel v
We utilize the 4 warped voxels for each direction nearby voxel v . Taking direction x for example, the warped neighbor voxel i′x (i′x =1~4), we can get the most similar 3 voxels (i′ ,x1 i′ ,x2 i′ ) in exemplar x3 T for voxel x i′ from the similarity set. Then we x use the warped relationship with the anisometric fields A between voxel i′ and x voxel v to infer the candidate voxels (ix′1v,ix′2v,ix′3v) in exemplar Tx for voxel v , as shown in Fig. 4.5(b). We want to keep the color consistence in three directions, so we use the precomputed 3D-candidate set to infer the other two coordinates (i(′x→y)v,i(′x→z)v)
30
triple coordinate for voxel v with the best matching triple candidate.
Figure 4.5 The process of forming warped triple candidates in direction x: (a) A voxel v is corrected.
31
Chapter 5
Implementation and Results
We implement our system on a PC with 3.00GHz and 3.00GHz Core2 Extreme CPU and 8.0GB of system memory. We use MATLAB to implement our method. We always use 128×128 input texture, and we can synthesize to any target size which we want. It needs about 2.5 seconds to construct a transformed exemplar from feature
We implement our system on a PC with 3.00GHz and 3.00GHz Core2 Extreme CPU and 8.0GB of system memory. We use MATLAB to implement our method. We always use 128×128 input texture, and we can synthesize to any target size which we want. It needs about 2.5 seconds to construct a transformed exemplar from feature