PDT synthesis

If the image contains known shapes with explicit or implicit linear structure, we might want to look for groups of line pixels having this specific shape. In our approach, we use Hough transform [22] to search for such known shapes. Furthermore, in order to detect the possible line structures effectively and reduce the noise inference, some necessary pre-processing should be performed as follows.

3.1.1 Preprocessing

The first step of this method is to extract the line information from PDT. However, in order to extract the significant control structure, we need to reduce the inference due to color variation. For simplicity, we firstly re-quantize the RGB components in texture image into 8 levels (the entire domain is re-quantized from (0, 0, 0) through (255, 255, 255) to (0, 0, 0) through (1, 1, 1), quantized criterion = 125). The details that are not related to the structure can be removed effectively in this step. Then, a well-known Sobel operator is applied to extract edge pixels. Furthermore, the morphology operation of closing (dilation and erosion) is performed to remove the isolated edge pixels or noises.

Furthermore, the skeleton technique is introduced to describe the global properties of objects and reduce the original image into a more compact representation [38]. The above

steps for a perspective distorted example are illustrated in Figure 6. Later on, the Hough transform will be utilized for main slanted lines detection.

(a) Input Texture (b) Color Quantization (c) Sobel operator

(d) Dilation (e) Erosion (f) Skeleton

Figure 6 The preprocessing for perspective distorted texture synthesis.

3.1.2 Control Structure detection

In our approach, the control structure that we are interested in PDT is composed of implicit or explicit slanted lines. Thus, after the pre-processing, the Hough transform is applied to detect the lines in the edge map image. The main idea in Hough transform is to detect the peaks in distance-angle space as shown in Equation (1) [25, 39].

ρ = x cos 𝜃 + 𝑦 sin 𝜃. (1)

where 𝜌 is the distance from the origin, and 𝜃 is the angle it makes with the x axis. That is, the point(s) in the parameter space (𝜌, 𝜃) having the most number of accumulations are the most likely to be candidates of lines. In addition, as shown in Figure 1, we can find some important characteristics in PDT. First, the detected straight lines in PDT are usually not parallel in specific direction (i.e., camera axis) due to perspective transform, and the difference of directions between two lines is very small. However, the non-parallel phenomenon is more and more significant when the distance between two lines increases.

Because the line in PDT would be implicit and the edge detections are noisy, the error in line detection is unavoidable, which significantly degrades the accuracy of transform model estimation especially when two lines are very close. Therefore, two step procedures are needed for control structure detection.

1. The major lines detection: Two criteria for major lines detection are presented: (a) The line distance is as large as possible, and (b) The difference in angle for non-frontal-parallel lines should be restricted in a small range. For main slanted lines detection, the significant maximum in Hough transform domain is selected as the first line. Then, the second criterion is defined according to the first detected line;

that is, the angle (𝜃) of detected line is set as the search center. According to the search center, we would like to define the search area in both horizontal and vertical

directions, respectively. In horizontal direction, the span range of search area is set from -30° to 30°. In the vertical direction, the angle of search center at 𝜃, is defined as reference direction and the maximum distance in the transform space along the reference direction is selected as the span range. Then, we divide the search area vertically into two equal sub-regions; one sub-region contains the first line, and we search the second dominant line in the other sub-regions. The detection of major lines is illustrated in Figure 7.

Figure 7 The major lines detection through Hough transform.

(b) Perspective distorted texture (c) Binary image (a) Main slanted lines detection

(d) Hough transform for perspective texture.

2. The minor lines detection: After the set of major lines are extracted, the remaining two lines of the control structure need to be detected. We eliminate the search area of major lines, and use the same procedure as in major lines detection to define the search area of the minor lines. However, when the horizontal range exceeds 90 degree or less than -90 degree, we will expand the 𝜃 plane toroidally to form the search area as shown in Figure 8(b). The detection results of the two minor lines are shown in Figure 8.

Figure 8 The minor lines detection for perspective distorted texture.

(a) Secondary structural lines detection.

(b) Hough transform for perspective texture.

For visual clarification, we use a distorted texture to illustrate the aspect of our approach in the following. In this example, the significant maxima (A1 in Figure 7(d)) in the transform domain with respect to the one of true slanted lines; meanwhile, the other symmetrical slanted line (B1) can be detected in the candidate peaks for a specific search area. In Figure 7(c), the detected corresponding structure features (A1, B1) are shown in the image space.

In order to derive the control points {a, b, c, d} in the map plane, the next step is to detect another pair of lines. Excluding the previous blue search region and using the aforementioned technique; the secondary structural features (C1, D1) can be identified in the horizontal and the locations in the parameter space ( , )  of C1( 278, 90 )  and D1

( 25, 90 )  as shown in red rectangle box in Figure 8(b). The control structure in PDT is composed of major and minor lines. By the control structure, the perspective transform can be solved accordingly, which will be explained in the next section.

3.1.3 Texture rectification

The distortion in PDT can be described by perspective transform model with eight transform coefficients. In order to find the model coefficients, we should define the perspective transform in PDT by using the detected control structure. The control structure can be viewed as a sample of perspective distortion in PDT, and we will define the reference structure for PDT rectification. There are two possible cases in our work: (a). The perspective

distortion texture has the dominant direction inclined to x or y axis; (b). The perspective distortion texture has no dominant direction in both axes.

Figure 9 The texture rectification for perspective distortion with the dominant direction inclined to x or y axis texture.

For case (a), as shown in Figure 9(b), the detected control structure is a quadrilateral with four line segments (A1, B1, C1, D1), and the control points are the intersections of the four lines. Let the coordinates of control points be presented as a(x₁, y₁), b(x₂, y₂), c(x₃, y₃), d(x₄, y₄) respectively. Meanwhile, the four vertices coordinates of the reference structure are defined as (x^’₁, y^’₁), (x^’₂, y^’₂), (x^’₃, y^’₃), (x^’₄, y^’₄). In our work, the reference structure is defined

(c) (b) (a)

as a maximal bounding rectangle in the control structure. According to control structure, we sort the coordinates of control points in x and y axes, respectively. Then, we select the middle values as the coordinates of the reference rectangle. The selected scheme is expressed as follows.

(2)

After obtaining the control structure (𝑥_𝑖, 𝑦_𝑖) and reference structure (𝑥_𝑖^′, 𝑦_𝑖^′), i = 1, 2, 3, 4, we can calculate the parameters of the perspective transform with Equation (3). Then, applying the 8 transformed parameters, the distorted real texture can be rectified as shown in Figure 10(c).

Figure 10 Texture rectification for real texture image in case (a).

(a) Angle convention (b) The converted reference structure (marked by red box)

Figure 11 The reference structure detection for case (b).

For case (b), the perspective distortion texture has no dominant direction in both axes as shown in Figure 11. In this case, the above scheme will produce inconsistent corresponding reference structure. Thus, it cannot apply the selected scheme directly. In our approach, a

(a) Perspective distorted texture

(b) Deriving the coordinates for texture rectification

(c) Rectified texture

simple pre-processing is developed to address the problem. Firstly, we calculate the angle θ= cos^{-1 n}

r, where n and r are shown in Figure 11(a). Then, we rotate the control structure by θ degree counterclockwise or clockwise depending on the direction of r, i.e., if the slope of r is negative then rotate counterclockwise otherwise clockwise. Finally, we apply the same scheme as in case (a) to find the corresponding reference structure. Once the reference structure is calculated, we inversely rotate the same angle and the correct reference can be obtained.

Figure 12 System flowchart of our framework for perspective distorted texture synthesis

blending schemes in section 3.2-3.3, the similar characteristic textures of arbitrary size can be generated based on the conventional patch-based algorithm [15] efficiently. The problem that renders a perspective distorted texture with corresponding slant degree as the source texture can be regarded as the reverse mapping process of texture rectification as mentioned in section 3.1.3. Figure 12 demonstrates the block diagram of our synthesis framework for perspective distorted texture. The experimental results demonstrated that our texture rectification through Hough transform approach can work well for various types of PDT synthesis.