Activity analysis for adaptive patch-based synthesis approach

Performance of patch-based synthesis approaches strongly depends on patch size [15], generally, the large patch can preserve more texture characteristics but smooth the details of objects structure especially in their shapes and the small patch works just the opposite. In order to demonstrate the patch size effect, an example is illustrated in Figure 13. For oil painting, to synthesis with large patch size, the presented styles in Figure 13(c) are clearly better than that of Figure 13(d), however, the details preservation is worse. Therefore, in order to render in source image style and preserve the target image content, synthesis by variable patch size is an appropriate choice. In summary, the patch-based texture rendering suffers from the following fundamental problems: do not incur apparent block patterns, smooth transition for the overlapping boundary zones and low time cost during the

synthesized procedure. In this Section, we will derive an adaptive patch size scheme to gain the both advantages in large and small patch size. The new scheme is also different from the iterative or pixel-based texture transfer method used in [14, 27-28]. In the meantime, the performance of artistic transfer quality and speed can be improved simultaneously. The detailed adaptive algorithm is explained as follows.

(a) Styled texture. (b) Target texture. (c) Patch size:24×24 (d) Patch size:12×12 Figure 13 The comparisons of transferred effect based on different patch sizes.

Although the transfer of desired artistic styles highly dependents user subjective which It is thus difficult to clearly define. Howe, from the viewpoint of features, the artistic style can be considered as various combinations of low level features such as colors, lines, edges, corners, texture, etc. In other words, the general idea of artistic style transfer is to preserve large features (i.e., main structure) of the target image but replacing high-frequency details to match those of the artistic images [12]. Therefore, the idea behind our approach is that the large patch is used to render the artistic style in less variation region and the small patch used to quick variation region in target image, respectively. Finally, the overall performance is

satisfactory for both artistic style in source image and contents in target image. Based on aforementioned observations, the developed scheme activity analysis is the criteria for adaptive synthesis process to achieve better transformation of style and preservation content in the target image at the same time. In [29], a distance-based interpolation method (Radial Basis Functions) [43] was proposed to calculate the significant feature in target image to guide texture synthesis for artistic transfer. In our approach, similar idea is employed to detect the activity in target image and then the patch size is determined accordingly. Since human perceptual system is very sensitive to the features of abrupt change such as edges, corners and some specific salient structures in image, thus to select such features to describe image is a simple way. Meanwhile, we assume the pixels that close to these features have with more important information of the image, and the information drop off gradually when move away from these features. Therefore, we denote the edge-based feature field distribution to represent the activity of the target image.

In our algorithm, the first step of this method is to extract the line information from sample texture. From preprocessing of section 3.1, we obtained skeleton map of an image.

We can define the results of skeleton as the salient structure of an image. As above-mentioned, we assume the pixels that close to the salient structure have more important information of the image, and the information drop off gradually when move away from these features.

Therefore, we can simply dilate the salient structure and then low-pass filter (RBF) to obtain

the high activity region of the image, and we define as salient map hereafter. For visual clarification, we illustrate the distribution of significant feature points shown in Figure 14. In this example, the white regions in Figure 14(d) represent the salient structures with high activity of an image; the black regions might carry less amount of information and we can assume that the pixels which are background of an image.

(a) Original image (b) Skeleton (c) Dilation twice (c) Low-pass filter Figure 14 Salient map of an image

Since the user desired result should look similar to the content of the target image and infuse novel style (high-frequency details) from the styled example. Thus, the approximate activity analysis is suitable to derive an adaptive patch-based sampling approach and a reasonable estimation valuesto control the output texture making it preserve more content of target texture or styled image for the texture transfer process. For the adaptive patch size based on activity analysis, the following condition should be considered:

𝑎𝑐𝑡𝑖𝑣𝑖𝑡𝑦_𝑖𝑛𝑑𝑒𝑥 = ^{∑ 𝑞∈}_𝑞 ^𝑝∩𝑠𝑎𝑙𝑖𝑒𝑛𝑡 𝑚𝑎𝑝

⌊_𝑝⌋ (10)

where |_𝑝| is the area of a given patch which centered at point p. Since the large value of

activity-index indicate that the pixels are nearer to the features and carry more information of an image [15]. Therefore, a relative smaller patch is adopted to preserve the details of objects in target image; otherwise, a larger patch means better transferring the style to the background of target by capturing more characteristics of source image. Since human visual perception is more sensitive to low frequency components of an image, thus the adaptive patch-based synthesis approach will effectively enhance the visual effect.