Chapter 1 Introduction
1.3 Thesis Organization
The rest of this thesis is organized as follows: In Chapter 2, we review related works.
In Chapter 3, we present our approach for synthesizing 3D textures from a 2D texture based on weighted color and GLCP features. Chapter 4 shows the implementation and results. Finally, conclusions and future works are discussed in Chapter 5.
Figure 1.1 System flowchart Input 2D Texture
Result Synthesis Process Similarity Set
Generation
3D-candidate Generation Feature Vector Generation
Weighted Color and GLCP Features
Grey Level Co-occurrence Probabilities Grey Level Texture
Grey Level Co-occurrence Matrix
Statistic Selection Color
Pixel Color
Appearance Vector
Dimension Reduction
6
Chapter 2
Related Works
In this chapter, we review related works related to our approach. In Section 2.1, we review previous approaches about 3D texture synthesis based on neighborhood matching.
In Section 2.2, we review related works about grey level co-occurrence matrices.
2.1 3D Texture Synthesis Based on Neighborhood Matching
Wei [26] first adapted 2D neighborhood matching synthesis schemes to 3D textures.
The key idea is to consider three 2D exemplars for each direction from an input 2D texture. In each voxel of the output 3D texture, three interleaved 2D neighborhoods are extracted (see Fig. 2.1 [7]). The best matches are found independently in each of the three 2D exemplars. And then they attempted to converge toward the best color for all three directions.
Figure 2.1 [7] 3D neighborhood matching synthesis schemes
(a) The three exemplars E , x Ey, Ez and a corresponding 3-neighborhood.
(b) the crossbar defined by the 3-neighborhood.
Qin and Yang [22] presented a method for generating 3D textures from input examples. They replaced traditional gray-level histograms with basic gray-level aura matrices for neighborhood matching. Their method characterized each input example as a set of aura matrices and generated a 3D texture from multiple view directions. For every voxel in the volume result, they only considered the pixels on the three orthogonal slices for neighborhood matching. Furthermore, their approach can generate reliable results for both stochastic and structural textures. However, it needs large storages for large matrices.
Kopf et al. [14] introduced a 3D texture synthesis method from 2D exemplars. They extended 2D texture optimization techniques to synthesize 3D textures, and integrated
(a) (b)
8
with histogram matching which forces the global statistics of the synthesized 3D texture.
They also introduced a reweighting scheme for histogram matching. For each voxel, they only considered the neighborhood coherence in three orthogonal slices, and iteratively increased the similarity between the 3D textures and the exemplar. Their approach could generate good results for wide range of textures. However, they synthesized the texture with the information on the slices.
Dong et al. [7] introduced a method with the unique ability to restrict synthesis to a subset of the voxels. They synthesized a volume from a set of pre-computed 3D candidates. Their pre-computed 3D candidates improved synthesis efficiency and significantly reduced the dependency chain required to compute voxel colors. Their approach generates good results efficiently. However, if the three exemplars do not define a coherent 3D volume, the quality of the synthesized result will be poor.
Chen and Wang [3] presented a method for synthesizing solid textures from 2D exemplars. They analyzed the relevant factors for further improvements of the synthesis quality. They adopted an optimization framework with the k-coherence search and the discrete solver for 3D texture synthesis. Their approach was integrated with two kinds of histogram matching methods, position and index histogram matching. They effectively
cause the global statistics of the synthesized 3D textures to match those of the exemplars.
2.2 Grey Level Co-occurrence Matrix
The grey level co-occurrence method provides a second-order approach for generating features [9, 12]. Given a spatial window within the grey level image, the GLCP method computes the conditional joint probabilities, Cij , of all pairwise
combinations of grey levels i and j, given the inter-pixel displacement vector ( ,), which represents the relationship of the pixel pairs, where is the interpixel distance, and is the orientation. The set of GLCPs are defined as
1
0 , G j
i ij
ij
ij P
C P ,
where Pij is the frequency of occurrence between two grey levels, i and j, for a given
displacement vector ( ,), for the given window size; G is the number of quantized grey levels. The probabilities are stored in a grey level co-occurrence matrix (GLCM), where index (i, j) in the matrix is probability Cij. Statistics are applied to the GLCM to
generate texture features which are assigned to the center pixel of the image window. For example, as shown in Fig. 2.2, we use a 55 window size for the grey image and G is set as 4 (Fig. 2.2(a)). The GLCM is generated with the inter-pixel displacement vector (1,
10
0) (Fig. 2.2(b)) and then GLCP (Fig. 2.2(c)) is computed.
Figure 2.2 The process for computing grey level co-occurrence probabilities
(a) a 55 window size for the grey image and G is set as 4
(b) GLCM is generated with the inter-pixel displacement vector (1, 0) (c) GLCP is computed from (b)
Haralick et al. [9] first presented a texture analysis method, grey level co-occurrence (a)
(b)
(c) 0
0 1 2 3
1
2
3
0 1 2 3
0 1 2 3
i i
j
j
matrix (GLCM). The GLCM method is a way of extracting second order statistical texture features. The approach has been used in various of applications [2, 12, 13, 18]. The key idea is to describe the probability of any grey level occurring spatially relative to any other grey level, within a given image window.
Jobanputra and Clausi [12] extended the GLCM method. They proposed a Gaussian weighting scheme. Co-occurring pixel pairs closer to the center of the image window are assigned larger co-occurring probabilities according to a Gaussian distribution. The generated features are referred to as weighted GLCM texture features.
Jobanputra and Clausi [13] presented a weighted GLCM method for preserving boundaries for image texture segmentation. They selected suitable GLCM parameters for
improved boundary preservation. From their tests, weighted GLCM features provide
improved boundary preservation and segmentation accuracy at a computational cost.
Chen [2] presented a GLCM method for face detection. Face detection is one of the most important researches in face recognition. They extracted texture features from a color image by wavelet transform, and then extracted the suitable statistics by the GLCM method. They used the statistics to execute the face detection process.
12
Lu [18] presented a GLCM method for image segmentation. The key idea is to perform image segmentation by combining the analytic result of color and texture. In the textural part, they proposed an approach to solve the problem in GLCM. Using fixed window size is unable to accurately segment textures in the image. Therefore, adaptive windows are used to analyze features of the image textures.
Chapter 3
3D Texture Synthesis
In this chapter, we present our approach for synthesizing 3D textures. In Section 3.1, we describe how to obtain the feature vectors. In Section 3.2, we use similarity sets to accelerate neighborhood matching. And then we compute 3D-candidates which keep the color coherence of the three orthogonal slices. In Section 3.3, we introduce 3D pyramid texture synthesis to our system. It is divided into three parts: the upsampling step, the jitter step, and the correction step. The upsampling step is to increase the synthesized 3D texture sizes from coarse level to fine level, and each voxel in parent level generates eight voxels in children level. The jitter step is to perturb the textures to achieve deterministic randomness. The correction step is to use neighborhood matching to make the results more similar to the exemplar.
3.1 Feature Vector Generation
14
First, we extract the features from an input 2D texture. The feature extraction approach is divided into two parts: color feature extraction method and the grey level co-occurrence method.
3.1.1 Color Features
3D texture synthesis using RGB color values for neighborhood matching needs larger neighborhood size and numerous data. Appearance vectors [17] are continuous and low-dimensional than RGB color values for neighborhood matching. Therefore, we transform the texture data values in color space into feature vectors in appearance space.
As shown in Fig. 3.1, we transform the input 2D texture T into the transformed exemplar T.
According to Lefebvre and Hoppe [19], we take the RGB color values in 5×5 windows to construct appearance vectors for each pixel of an input texture T. The transformed exemplar T consists of the feature vectors at each pixel. There are 75 dimensions (25 for grids and 3 for RGB) for each pixel in T. We perform PCA to reduce the dimensions to obtain a transformed exemplar T~. It means that we project the exemplar Tusing PCA to obtain the transformed exemplar T~. We reduce the 75-dimensional to 8-dimensional feature vectors.
Figure 3.1 Overview of color feature extraction
We suppose that the data on each side is connected with them on the opposite side.
For the pixels on the border, we will treat the pixels on the opposite border as its neighbors, and then take their RGB color values to construct feature vectors.
3.1.2 Grey Level Co-occurrence Probability Features
Here we convert the input 2D texture to a grey image, and extract GLCP features from the grey image. Extracting GLCP features from a grey image requires the parameters as follows: (1) window size (nx, ny), (2) image quantization (G), (3)
displacement vector ( ,) and (4) statistic selection.
For window size (nx, ny), we use a window size of 15 15 to compute GLCP
features. Because large window sizes are necessary to gather sufficient data to
T 2D Texture
T
Color Space Appearance Space
RGB for each pixel Feature vector for each pixel
Dimension Reduced T~
16
characterize local texture regions, and small window sizes will result in poorly sampled
co-occurring probabilities and will produce incoherent features [13]. As shown in Fig.
3.2, we compute GLCP features for the window size 15 15.
Figure 3.2 (a) grey level image generated from the input 2D texture
(b) 15×15 window for each pixel (c) GLCP features computed from the window.
(b)
(a) (c)
For image quantization (G), according to Clausi [5], the larger values of G are
deemed excessive. Therefore, we set G as 32 for our experiments. For displacement vector ( ,), we select the suitable displacement vector. The interpixel distance is set as 1 and the orientations are set as 0,45,90,135 . Because we consider the characteristic of an input 2D texture for each direction and consider the influence of the distance of the pixel pairs.
For statistic selection, we will select the suitable statistic. Many of the statistics suggested by Haralick [9] produce highly correlated texture features which are not desirable. Clausi [5] studied the relationship of the statistical parameters and concluded that entropy, contrast and correlation compose a preferred set of statistical parameters.
Therefore, we use three grey level shift invariant statistics in our approach as follows:
18
where x and yare the means in the the x- and y-directions, respectively; x and
yare the standard derivations in the the x- and y-directions, respectively. For entropy statistics, it means the level of spatial disorder of gray levels in the GLCM. For contrast
statistics, it means the contrast of spatial disorder of gray levels in the GLCM. For
correlation statistics, it means the linear dependency of gray levels on those of neighboring pixels in the GLCM. Therefore, we extract 12 GLCP features for our experiments (4 orientations for each statistical parameter).
3.1.3 Weighted Color and GLCP Features
After color and GLCP features are extracted, we propose weighted color and GLCP features for 3D texture synthesis. Different weights are assigned to color and GLCP features for a given input 2D texture. The weighted color and GLCP features are defined as
Weight CF Weight GLCPF
WF CF , GLCPF ,
where WeightCF and WeightGLCPF are the weights, and they are changed according to
the characteristics of the input 2D texture. CF and GLCPF are the color feature vectors and GLCP feature vectors, respectively.
3.2 Similarity Set and 3D-Candidate Construction
Based on [1, 24], we find the k most similar candidates in the exemplar for each pixel p, and then search from the candidates for neighborhood matching. The method can accelerate neighborhood matching because we do not have to search from each pixel in the exemplar for neighborhood matching. Therefore, we have to construct a similarity set to record the k candidates similar to each pixel.
Furthermore, we repeat the input texture thrice as three directional exemplars Tx,
Ty, Tz. By Dong et al. [7], we construct a small set of 3D-candidates for each pixel of the three exemplars to build the relation between the three exemplars. A 3D-candidate is defined by three 5×5 slices (2D neighborhoods). A suitable candidate should be consistent across the crossbar which is the strip that is intersected by two slices.
Therefore, we minimize the color disparity between the strips shared by interleaved slices.
3.3 3D Pyramid Synthesis 3.3.1 Pyramid Upsampling
The 3D pyramid synthesis approach [7, 10] synthesizes 3D textures from coarse
20
level to fine level. There are l1 levels in the whole synthesis process, l0~log2 m, where m is the size of the target 3D texture.
In the proposed approach, we synthesize from one voxel to a 3D texture, S ~0 Sl. A
where h is the regular output spacing of exemplar coordinates, and l ijk signifies the
3.3.3 Voxel Correction
After we jitter the coordinates for achieving deterministic randomness, we make the coordinates similar to those in the exemplars Tx, Ty, Tz. We take the jittered results to
recreate neighborhood. There are feature values for each pixel after constructing feature vectors. For every voxel v, we respectively collect the feature values of its neighborhood
22
to obtain the neighborhood vectors for each direction. We search the most similar pixel from the transformed exemplars to make the result similar to the exemplars Tx, Ty, Tz.
In neighborhood matching, 4 diagonal locations in each direction are taken for voxel v to obtain the neighborhood vectors Ns v x
l( ) , Ns v y appearance values from 3 synthesized voxels nearby vx as the new feature value to replace the voxel vx. And then, we use the new feature values at 4 diagonal voxel to
]
Using the similarity sets and 3D-candidate sets, we utilize the 4 voxels (ix = 1~4) for each direction nearby voxel v. For the neighborhood voxel ix (ix = 1~4) in direction x , we can obtain the most similar 3 pixels (ix1, ix2, ix3) in exemplar T for voxel ix x from the
similarity set. Then, we transform the offset between voxel ix and voxel v from 3D space to 2D space to infer the candidates (ix1v, ix2v, ix3v) in exemplar Tx for voxel v. To keep
color consistence in three directions, we use the 3D-candidate set to infer the other two coordinates (ixyv,ixzv) in exemplars Ty and Tz. Additionally, directions y and
z can be done by the same step.
24
Finally, we can obtain a set of triple candidates TC1...k which point towards pixels
in exemplars Tx , Ty , Tz . We compute the neighborhood vectors Ns TC k x
l( 1... ) ,
y k
s TC
N l( 1... ) , Nsl(TC1...k)z by the averaged feature values from the 4 nearby pixels. And Then, we compute the total difference among Nsl(TC1...k)x, Nsl(TC1...k)y, Nsl(TC1...k)z and Nsl(v)x, Ns v y
l( ) , Nsl(v)zto replace the triple coordinate for voxel v with the best matching triple candidate.
Chapter 4
Implementation and Results
In Section 4.1, we describe the implementation of our method. Then, we show synthesis results and comparisons in Section 4.2.
4.1 Implementation
Our system is implemented on a PC with 3.00GHz and 3.00GHz Core2 Extreme CPU and 8.0GB of system memory. We use MATLAB to implement our method. The input 2D texture size is 128×128, and the output 3D texture size is 128×128×128.
In feature vector generation, we use a 5×5 window for extracting color features and a 15×15 window for extracting GLCP features for each pixel from the input 2D texture.
Furthermore, we set image quantization G as 32 and displacement vector ( , ) as (1, 0 ). Then, we use 7×7 neighborhood for constructing similarity sets. In 3D pyramid
26
synthesis step, the parameter for jitter step is set to 0.7.
4.2 Results
It takes about 2 minutes and 22 seconds to generate weighted color and grey level co-occurrence probability features. It takes different times to construct similarity sets based on characteristics of an input 2D texture. However, it does not exceed 1.5 hours.
And it takes about 3 hours 20 minutes to construct a 3D-candidate set. Once the feature vectors and similarity sets are constructed, they are used in syntheses process with different target sizes for results. It takes about 1 day and 1 hour for synthesis process.
We compare our approach with the previous method which only extracts color features. As we can see in Fig. 4.1~Fig. 4.8, our results preserve more features and structures better than the results synthesized by the previous approach. This is because we can obtain more information than the previous method. The previous method only extracts color features to generate feature vectors from input 2D texture, but our approach extracts not only color features but also GLCP features. The grey level co-occurrence method is one of the statistical methods. GLCP features describe the probability of any grey level occurring spatially relative to any other grey level within a given window for each pixel. From the distribution, we can obtain the information of various characteristics
for each pixel from input 2D texture. For irregular textures, color features are hard to record various characteristics accurately for each pixel. However, GLCP features record accurately the various characteristics because of the statistical relatedness among pixel pair within a given window for each pixel.
The input 2D texture in Fig. 4.1(a) is a stochastic and marble-like texture. And the input 2D texture in Fig. 4.2(a) is a particle-like texture. They only contain two kinds of colors, and they are vivid. Fig. 4.1(b) shows the result synthesized by the previous approach and Fig. 4.1(c) shows our result. As we can see from the results, the quality of our method is almost the same as that of the previous method. And our result is continuous and not the duplication of the input 2D texture. As long as there are few complete particle patterns in the input 2D texture, our approach and the previous method can synthesize the desired results, as shown in Fig. 4.2(b) and Fig. 4.2(c). From Fig. 4.1 and Fig. 4.2, our approach and the previous method can preserve features and structures.
The input 2D texture in Fig. 4.3(a) is a stochastic and marble-like texture. The input 2D texture in Fig. 4.4(a) is a kind of stochastic textures. The input 2D texture in Fig.
4.5(a) is a stochastic and camouflage-like texture. In Fig. 4.3(c), our result preserves more white features in whole area. But in the previous result (Fig. 4.3(b)), it preserves
28
less white features. And our result is more colorful than that of the previous method. In Fig. 4.4(c), our result preserves complete features in whole area. But the previous result (Fig. 4.4(b)), it does not preserve features as we can see in Fig. 4.5(c), our result preserves more features than that of the previous method (Fig. 4.5(b)). From Fig. 4.3 and Fig. 4.4 and Fig. 4.5, we can preserve more features than the previous method.
The input 2D textures in Fig. 4.6(a), Fig. 4.7(a), and Fig. 4.8(a) are grey level textures. They are stochastic patterns as we can see in Fig. 4.6(b), our result preserves more black features in whole area. But the previous result (Fig. 4.6(b)) preserves less black features. In Fig. 4.7(c), color variation of our result is better than that of the previous method (Fig. 4.7(b)). Our result is information-rich. The previous result does not keep the features as we can see in Fig. 4.8(c), our result preserves white features in whole area. But the previous result (Fig. 4.8(b)) preserves less white features. From Fig.
4.6, Fig. 4.7 and Fig. 4.8, the quality of our approach is better than that of the previous method.
We can see in Fig. 4.1 and Fig. 4.2, the results of our approach and the previous method are similar. However, we can see in Fig 4.3 and 4.8, the results of our approach are better than the previous method. This is because they are regular textures in Fig. 4.1(a)
and Fig. 4.2(a), so we can synthesize good results by only using color features. But in Fig 4.3(a) and 4.8(a), they are irregular textures, but color features can not record the information of various characteristics accurately. GLCP features can solve the above problem because of the statistical relatedness among pixel pair within a given window for each pixel. Therefore, our approach can synthesize well than previous method.
30
Figure 4.1 The comparison between the previous method and our method (a) input 2D texture
(b) output 3D texture by the previous method (c) output 3D texture by our method
(a)
(b) (c)
Figure 4.2 The comparison between the previous method and our method (a) input 2D texture
(b) output 3D texture by the previous method (c) output 3D texture by our method
(a)
(b) (c)
32
Figure 4.3 The comparison between the previous method and our method (a) input 2D texture
(b) output 3D texture by the previous method (c) output 3D texture by our method
(a)
(b) (c)
Figure 4.4 The comparison between the previous method and our method
Figure 4.4 The comparison between the previous method and our method