CMU-RI-04-01, 2004

(1)

January 2004

School of Computer Science Carnegie Mellon University

College of Computing

^∗

Georgia Institute of Technology

° Carnegie Mellon Universityc

(2)

Abstract

In this report, we compare the performance of four texture synthesis algorithms on synthesizing regular and near-regular textures. Our results show that near-regular texture synthesis remains a challenging problem. This is because a near-regular texture demonstrates both global regularity and local randomness in its texture pattern. It is difficult to preserve both properties in the synthetic textures. The comparison indicates that a specially-designed texture synthesis algorithm that respects the nature of near-regular textures can produce more faithfully synthesized textures than general purpose state of the art synthesis algorithms.

(3)

Texture Spectrum

regular stochastic

regular near-regular irregular

Figure 1: A texture spectrum on which textures are arranged according to the regularity of their structural variations, where irregular textures refers to geometrically-irregular near-regular textures.

The goal of this report is to compare the performance of texture synthesis algorithms in a more objective and consistent way. To satisfy this goal, testing samples should cover an appropriate scope of textures on the spectrum, and more importantly, the testing samples should also have a consistent property so that it is easy to verify whether the property is faithfully preserved in the synthesis process. For these reasons, we consider two particular types of textures in this report:

regular and near-regular textures[13].

Regular textures are simply periodic patterns where the color/intensity and shape of all texture elements are repeating in equal intervals. That is, a texture element is a unit tile in a regular texture, which can be synthesized by tiling the space with the unit tile¹. An example of regular textures is wallpaper. In the real-world, however, few textures are exactly regular. Most of the time, the textures we see in the real-world are near-regular, such as cloth, basket, windows, brick walls,

1The tile and texture element refer to the same concept and will be used interchangeably in this report.

(4)

building columns, carpet, blanket, honeycomb, etc. Near-regular textures can be considered as departures of regular textures in different spaces with different degrees. For example, in brick wall textures, the major departure happens in the color/intensity space as the shape of each brick is regular but the color/intensity may vary. On the other hand, the perspectively-distorted window texture in Figure 2 departs mainly in the geometry space while the color/intensity of each window is almost regular. Figure 1 shows a texture spectrum where the textures are arranged according to the regularity variation in geometry space.

In this report, we will only consider a subclass of near-regular textures: geometrically-regular near-regular textures, of which the texture elements only vary in color/intensity but are almost regular in geometry. In other words, the texture elements can be defined by a regular lattice. For simplicity, we will call geometrically-regular near-regular textures ”near-regular textures (NRT)”

in the rest of the report.

The global regularity of the geometrically-regular near-regular textures is a consistent property that can be used to evaluate the synthesis results. Moreover, the departures in color/intensity space of near-regular textures provides another property to evaluate synthesis algorithms.

wallpaper

wallpaper brickwallbrickwall windowswindows

snake skin snake skin honeycomb

honeycomb alligator skinalligator skin

perspectively -distorted windows

Figure 2: Examples of regular and near-regular textures.

2 Selected texture synthesis algorithms

Work in texture synthesis has achieved impressive results in a variety of cases[1, 2, 3, 4, 5, 6, 7, 9, 14, 15, 16, 17]. These algorithms can be roughly divided into two groups: the statistical-model-

(5)

patch-based approach developed by one of the authors. We briefly describe these algorithms here.

Readers who are interested in these algorithms are encouraged to read the original papers for details.

2.1 Graph cuts texture synthesis

Kwatra et al.[7] demonstrate a very effective general texture synthesis algorithm. Texture is synthesized by overlaying the entire input texture onto the synthetic texture at various offsets and using a graph cut algorithm to find the optimal region to add to the synthetic texture. The graph cut algorithm avoids the need for a fixed, a-priori patch size, and scales well to any dimension (such as video). However, for near regular textures the choice of offsets is as important as finding low-error seams. If the input texture is copied onto the synthesized texture at an offset that is inconsistent with the periodicity of the texture, any selection of seams will still violate the global regularity of the texture. Kwatra et al. describe patch placement algorithms which do a fair job of finding low error offsets. The error is defined as the sum of squared difference (SSD) between the pixels in the overlapping region of the input texture and the texture being synthesized. They treat the input texture as a template and compute the correlation between the template and the texture being synthesized to find the low error offsets. The minimum error (or maximum correlation) offsets often, but not always, correspond to the offsets preserving the periodicity of the input texture.

2.2 Near-regular texture synthesis

Liu et al.[13] propose a texture synthesis algorithm for geometrically-regular near-regular textures². The basic idea is to utilize the translational symmetry property[10][11] of a near-regular texture to find the underlying lattice structure of the texture patterns and locate the texture elements. These tiles represent the smallest parallelogram-shaped region on a regular texture that can reproduce the texture pattern under the texture’s translation subgroup. For a regular texture, only

2Although the name of the synthesis algorithm is near-regular texture synthesis, the algorithm only deals with geometrically-regular near-regular textures.

(6)

one tile is needed for recovering the full texture. For a near-regular texture, one needs a set of tiles collected by sampling the input texture in a principled manner to preserve both the geomet- ric regularity and color/intensity variations in the input texture[12]. The tiles in the tile set have roughly the same size and shape but varied color/intensity. The output texture is synthesized by randomly picking a tile from the tile set and pasting the tile to the synthesizing image with overlapping on lattice points. Dynamic programming and image blending techniques are applied to the overlapping regions to stitch the tiles.

2.3 Patch-based texture synthesis

Liang et al.[9] develop a patch-based synthesis algorithm. The basic idea of the algorithm is to synthesize textures by directly copying image patches from the input texture. The major difference from other image-based approaches is that they apply a modified approximate nearest neighbor technique to speed up the search for the best matched patch. With this improved search speed, the algorithm can run in real-time and reach similar image quality as other image-based synthesis algorithms. The image feathering technique is used in the patch-based synthesis approach to blend the overlapping regions of patches. This might blur the overlapping region slightly compared to the dynamic programming technique used in near-regular texture synthesis or the graph cut technique in the graph cuts synthesis approach.

Patch placement in the patch-based approach is very different from that in near-regular texture synthesis. In the patch-based approach, the patch is rectangular and the patches are pasted in a scan-line order. Since the patch size and placement offset are arbitrarily defined by a user, they may not match the lattice structure of the input near-regular texture.

2.4 Regularized patch-based texture synthesis

We develop a regularized patch-based texture synthesis algorithm to deal with near-regular textures³in which each texture element may not be well circumscribed by a parallelogram. We allow the parallelograms on a regular lattice to be deformed to quadrilaterals so that the texture elements can be separated by the deformed lattice. In other words, we deform a geometrically-irregular near-regular texture to a geometrically-regular near-regular texture. We then apply a modified patch-based approach to synthesize the geometrically-regular texture. Our modification to the patch-based approach allows patches to be pasted along the lattice axis direction and allow the patch shape to be a parallelogram rather than a rectangle. The patch-size and lattice construction vectors are provided by a user who identifies the underlying lattice structure of the input near- regular texture. A synthesized inverse deformation is used to warp the synthesized regular texture to a near-regular texture. We describe the details of the algorithm in the appendix.

(7)

region are determined when extracting image pixels from the input texture, while patch placement and stitching refer to how the patches are placed and stitched in the synthesized texture.

To conclude this section, we summarize these four algorithms in Table 1. We compare the patch shape/size determination, patch placement, and patch stitching methods used in these algo- rithms, where a patch is a 2D sample of neighboring pixels extracted from the input texture. Patch shape/size refers to how the shape and size of the region are determined when extracting image pixels from the input texture, while patch placement and stitching refer to how the patches are placed and stitched in the synthesized texture.

3 Results

We compare the synthesis results of these four algorithms on regular textures (Figure 3-7) and near- regular textures (Figure 8-45). For the convenience of comparison, we summarize the synthesis performance of these four algorithms on regular textures in Table 2 and near-regular textures in Table 3. These two tables record whether the global regularity is preserved in the synthesized texture. In addition, we include the synthesis results by the image quilting approach[4] for six textures on their website⁴(Figure 16, 18, 22, 27, 35, 36). Among the six tested textures, two synthesized results of image quilting preserve the global regularity(Figure 22 and 27).

From Table 2, we find that all four algorithms can handle the regular textures very well, but they perform quite differently on near-regular textures, as shown in Table 3. It may not be surprising that the near-regular texture synthesis algorithm performs best among the four because it is specially designed for near-regular texture synthesis. The graph cuts approach and patch-based approach do not preserve the global regularity well most of the time. This is because these two algorithms do not explicitly model the underlying lattice structure of the input texture. It is interesting to compare the patch-based approach and the regularized patch-based approach. Because the latter utilizes a user-specified lattice of the input texture in the synthesis process, it preserves the global regularity

4http://www.cs.berkeley.edu/ efros/research/quilting.html.

(8)

Textures Description Graph cuts Near-regular Regularized Patch-based synthesis patch-based

Figure 3 wallpaper X X X X

Figure 6 jigsaw puzzle X X X X

Figure 7 pavement tiles X X X X

success rate preserved/total 100% 100% 100% 100%

Table 2: Comparison of whether four algorithms preserve global regularity in regular textures, where Xdenotes that regularity is preserved.

much better than the patch-based approach.

Although the graph cuts approach does not utilize the lattice structure in the synthesis process, it still performs well on a few near-regular textures and much better than the patch-based approach(Table 3). The reason for this is that the algorithm incorporates a correlation technique to determine the best pasting location so that the underlying periodicity, if it exists, can be preserved.

This correlation-based patch placement works well on regular textures and some near-regular textures; however, for near-regular textures in which the color/intensity of texture elements is not regular, the correlation technique can not guarantee the regularity to be maintained globally. This is especially true when the input texture contains an interlocked structure, like woven fabric or brick walls (Figure 23, 28, 34, 36, 37, 38, 40, 41). Figure 46 shows several synthesis results of a brick wall texture by different patch placement settings in the graph cuts. This example shows that the correlation technique can not work well on discovering the regularity of near-regular textures in this brick wall texture. Synthesis of the same texture by the near-regular synthesis approach is shown in Figure 36.

A limitation of the near-regular texture synthesis algorithm is that the input texture sample must contain at least two complete tiles so that the underlying lattice and the tile set are well defined. If no complete tile exists in the texture, the algorithm can not produce good results. This is the situ- ation in Figure 31, where the input texture contains two overlapping of periodic patterns (squares and hexagons), but the sampling area is too small to have a complete tile that covers a full period of the composed periodic pattern⁵. The near-regular texture synthesis algorithm preserves the square pattern because it demonstrates stronger periodicity, but the hexagon net is discontinuous between squares. Another failure example of the near-regular texture synthesis algorithm is the thin line jigsaw puzzle texture in Figure 33. In this case, the input texture does not contain a complete tile either. This can be observed from the upper-right image in which the pattern in the circled region only appears once in the texture.

(9)

Figure 19 fabric X X X X

Figure 22 knotted mat X X X ×

Figure 23 pie × X × ×

Figure 24 fabric X X X ×

Figure 25 toothpastes X X X X

Figure 26 windows X X X ×

Figure 27 windows X X X ×

Figure 28 fabric × X X ×

Figure 29 basket X X × ×

Figure 30 fabric X X N/A N/A

Figure 31 squares&hexagons × × N/A N/A

Figure 32 mosaic × X N/A N/A

Figure 33 jigsaw puzzle X × N/A N/A

Figure 34 fabric × X N/A N/A

Figure 35 cracker × X N/A N/A

Figure 36 brick wall × X N/A N/A

Figure 39 brick wall X X N/A N/A

Figure 42 carpet X X N/A N/A

Figure 43 rug × X N/A N/A

Figure 44 rug × X N/A N/A

Figure 45 cans × X N/A N/A

success rate preserved/total 53% 95% 86% 23%

(10)

4 Conclusion

In this report, we compare the performance of four texture synthesis algorithms on regular and near-regular textures. Because of the global regularity property of the regular and near-regular textures, we are able to provide a more consistent and objective criterion to evaluate the synthesis results.

This comparison shows that near-regular texture synthesis remains a challenge for several state- of-the-art algorithms, such as the graph cuts[7], patch-based[9], and image quilting methods[4].

The results from the regularized patch-based approach and the near-regular texture synthesis approach show that the global regularity of the near-regular textures can be better preserved if the synthesis algorithm analyzes the underlying lattice structure and uses this information in the synthesis process. In fact, the synthesis results by the near-regular texture synthesis algorithm demonstrate that both the global regularity and local randomness of a near-regular texture can be faithfully preserved.

Acknowledgment

The authors would like to thank Yanghai Tsin for providing his code for an earlier version of the near-regular texture synthesis algorithm. This research is partially funded by NSF grant #IIS- 0099597. The regularized-patch-based method was implemented when Wu was an intern at Mi- crosoft Research Asia, jointly advised by Dr. Liu and Dr. Harry Shum.

(11)

graph cuts near-regular synthesis

regularized patch-based patch-based

Figure 3: Synthesis results of a wallpaper texture.

(12)

input

(13)

(14)

input

Figure 6: Synthesis results of a jigsaw puzzle texture.

(15)

Figure 7: Synthesis results of a pavement tile texture.

(16)

input

Figure 8: Synthesis results of a punched card texture. Global regularity is not preserved in the patch-based result.

(17)

Figure 9: Synthesis results of a hexagonal net texture.

(18)

input

Figure 10: Synthesis results of a metal texture. Global regularity is not preserved in the graph cuts and patch-based results.

(19)

Figure 11: Synthesis results of a ceramic tile texture. The result by the patch-based approach does not preserve the regularity of the tile size in the middle part of the bottom rows.

(20)

input

Figure 12: Synthesis results of a fish tile texture. The patch-based result does not preserve the global regularity.

(21)

Figure 13: Synthesis results of a shaded wall texture. The patch-based result does not preserve the global regularity.

(22)

input

Figure 14: Synthesis results of a texture of squares. Regularity is not preserved in the graph cuts and patch-based results.

(23)

Figure 15: Synthesis results of a pavement tile texture. Only the near-regular synthesis result preserves the global regularity of the squares and vertical/horizontal line pattern.

(24)

input image quilting

Figure 16: Synthesis results of a texture of cans. Global regularity is not preserved in the patch- based and image quilting results. The cans in the third column of the patch-based result and the third column from the right of the image quilting approach are not synthesized correctly.

(25)

Figure 17: Synthesis results of a swirl texture. The regularity is not preserved in the bottom rows of the patch-based synthesis result.

(26)

Figure 18: Synthesis results of a basket texture. Global regularity is not preserved in the graph cuts, patch-based, and image quilting approaches. The interwoven structure is not maintained in the lower-left portion of the graph cuts and the central bottom part of the image quilting approach.

(27)

Figure 19: Synthesis results of a fabric texture.

(28)

input

graph cuts near-regular synthesis 1

(29)

(30)

(31)

Figure 23: Synthesis results of a pie texture. The near-regular synthesis result almost preserves the interwoven structure, except there are some artifact edges in the intersections. Regularity is not preserved in the graph cuts, regularized patch-based, and patch-based results.

(32)

input

Figure 24: Synthesis results of a fabric texture. Global regularity is preserved in graph cuts, near- regular synthesis, and regularized patch-based results, but not the patch-based result

(33)

Figure 25: Synthesis results of a toothpaste texture.

(34)

input

Figure 26: Synthesis results of a window texture. The size of windows is varied in the patch-based result.

(35)

(36)

input

(37)

Figure 29: Synthesis results of a basket texture. Regularity is preserved in the graph cuts and near- regular synthesis results, but not the regularized patch-based and patch-based results. Note that the orientation of texture in the regularized patch-based result is not correctly synthesized.

(38)

input

(39)

Figure 31: The near-regular texture synthesis result preserves the regularity of the squares, but the hexagon net structure is not maintained. Therefore, both algorithms failed in this texture.

(40)

input

Figure 32: Synthesis results of a mosaic texture. The regularity of the squares is not preserved in the graph cuts result.

(41)

Figure 33: Synthesis results of a jigsaw puzzle texture. The global regularity of this texture is not easy to observe. Near-regular texture synthesis can not preserve the regularity of this texture. The upper-right image shows that the input texture does not contain a complete tile as the pattern in the circled region only appears once in the texture.

(42)

input

Figure 34: Synthesis results of a fabric texture. Global regularity is not preserved in the graph cuts result.

(43)

Figure 35: Synthesis results of a cracker texture. There are some misplaced small holes in the image quilting and the graph cuts results.

(44)

Figure 36: Synthesis results of a brick wall texture. The regularity of the grout structure is not maintained in the image quilting and graph cuts results.

(45)

Figure 37: Synthesis results of a brick wall texture, where every sixth row consists of smaller bricks. The graph cuts result does not preserve this regularity.

(46)

input

Figure 38: Synthesis results of a brick wall texture. In the graph cuts result, some bricks in the 5th and 6th row are 50% larger than the others.

(47)

Figure 39: Synthesis results of a brick wall texture. Global regularity is preserved in both results.

(48)

input

Figure 40: Synthesis results of a brick wall texture. The graph cuts result does not preserve the global regularity.

(49)

Figure 41: Synthesis results of a brick wall texture. The grout structure is not preserved in the graph cuts result.

(50)

input

graph cuts

near-regular synthesis

Figure 42: Synthesis results of a carpet texture.

(51)

Figure 43: Synthesis results of a damaged rug texture. Global regularity is not preserved in the bottom portion of the graph cuts result.

(52)

input

(53)

Figure 45: Synthesis results of a texture of cans. Some cans in the 5th and 6th rows of the graph cuts result are not synthesized correctly (bottom-right area).

(54)

(a)input (b)pasting randomly

(c)pasting at local minima (d)pasting at local minima and maxima

Figure 46: This figure shows the results due to different settings for patch placement in the graph cuts approach. In the synthesis process, an error map is computed using seam cost around each pixel and then the location around which we want to paste the new patch is picked by sampling from this error map. The results shown are obtained respectively by (b)picking the pasting location randomly, (c)computing a local minima for these error values and picking a location from one of these, (d)computing both local minima and maxima and picking a location from one of these. The synthesis result of the same texture by the near-regular synthesis approach is shown in Figure 36.

(55)

[5] A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In ICCV (2), pages 1033–1038, 1999.

[6] D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. In ACM SIGGRAPH, pages 229–238, 1995.

[7] V. Kwatra, A. Sch¨odl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: Image and video synthesis using graph cuts. In ACM SIGGRAPH, pages 277–286, 2003.

[8] S.-Y. Lee, K.-Y. Chwa, and S. Y. Shin. Image metamorphosis using snakes and free-form deformations.

In ACM SIGGRAPH, pages 439–448, 1995.

[9] L. Liang, C Liu, Y. Xu, B. Guo, and H.Y. Shum. Real-time texture synthesis by patch-based sampling.

ACM Transactions on Graphics, 20(3):127–150, 2001.

[10] Y. Liu and R. T. Collins. A computational model for repeated pattern perception using frieze and wallpaper groups. In IEEE Conference on Computer Vision and Pattern Recognition, pages 537–544, June 2000.

[11] Y. Liu, R. T. Collins, and Y. Tsin. A computational model for periodic pattern perception based on frieze and wallpaper groups. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(3):354–371, March 2004.

[12] Y. Liu and Y. Tsin. The promise and perils of near-regular texture. In The 2nd International Workshop on Texture Analysis and Synthesis, 2002.

[13] Y. Liu, Y. Tsin, and W.-C. Lin. The promise and perils of near-regular texture. International Journal of Computer Vision, 2004. To appear.

[14] J. Portilla and E. P. Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision, 40(1):49–71, 2000.

[15] L.-Y. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. In ACM SIGGRAPH, pages 479–488, 2000.

(56)

[16] L.-Y. Wei and M. Levoy. Texture synthesis over arbitrary manifold surfaces. In ACM SIGGRAPH, pages 355–360, 2001.

[17] J. Zhang, K. Zhou, L. Velho, B. Guo, and H.-Y. Shum. Synthesis of progressively-variant textures on arbitrary surfaces. In ACM SIGGRAPH, pages 295–302, 2003.

A Regularized patch-based approach

In near-regular texture synthesis, it is desired to extract the ”texton”, which is the basic element in the texture image. We consider the near-regular texture as an array of arbitrarily deformed textons.

We build a regular lattice and warp the original deformed textons to the regular shapes. Then we can synthesize the regular texture. Finally, we compute a deformation field and warp the regular texture to a near-regular texture.

The whole system includes two parts. One is for analysis and the other is for synthesis. The analysis process is semi-automatic. We design an interface for users to select textons, build and deform the regular lattice. Then we can automatically synthesize the regular texture by using the patch based method, and the deformed near-regular texture by warping. Figure 47 illustrates how to select a texton, which corresponds to a quadrilateral or hexagon. We use two vectors to specify a quadrilateral. Then we can automatically build a regular lattice.

(a) (b)

Figure 47: This figure illustrates how to select a texton, which corresponds to a quadrilateral or hexagon. (a) The user draws two vectors to specify a quadrilateral. (b) a regular lattice is then automatically constructed based on the two vectors.

(57)

Figure 48: After a texture is regularized, the patch-based approach is used to synthesis this texture.

The right window shows the result.

In order to generate the near-regular texture (deformed texture), we need to compute the deformation field. Inspired by the theory of Markov Random Field, for a quadrilateral texton, each vertex in the deformed lattice is affected by its eight neighbors. For example, in Figure 49, the yellow knot represents the vertex M , and eight orange knots represent the neighbors of M . Each neigh- bor can be discriminated by its relative position to the center vertex, such as left-top, top, right-top, right, right-bottom, bottom, and left-bottom. We can build a distance density map p_i(x−x₀, y −y₀) for each neighbor. For each center vertex M (x₀, y0), if its neighbor Ni(x, y) exists,

p_i(x − x₀, y − y₀) = 1

Z exp(−dist(M(x₀, y₀), N_i(x, y))

2σ² ) (1)

(58)

Figure 49: The upper-left window shows the definition of a vertex (yellow knot) and its neighbors (orange knots) on the lattice.

Then we can generate a deformation field as follows.

1. Initialize the deformed lattice by randomly moving each vertex in the regular lattice within a small window.

2. Deform the lattice iteratively:

In each round, given a vertex M (x₀, y₀) and a small window [x₀−σ_x, x₀+σ_x, y₀−σ_y, y₀−σ_y], its neighbors are N_i(x_i, y_i), i = 1, 2, ..., 8. we can compute the likelihood as follows: