CHAPTER 1 INTRODUCTION
1.3 Organization of this Thesis
The detail of our method to cope with the puzzle solving problem will be described in Chapter 2, and then the progress and difficulty in our research and the user interactive method are discussed in Chapter 3. In Chapter 4, the experimental results will be displayed. Finally, Chapter 5 makes a conclusion.
Table 1. Categorization of methods on handling puzzle solving problems.
Method
7
CHAPTER 2
THE PROPOSED METHOD
In this chapter, we will describe our proposed method in detail. Fig. 2 shows the block diagram of the proposed method. As mentioned in Chapter 1.2, our method contains four steps:
(1) Puzzle Piece Extraction.
(2) Feature Extraction.
(3) Common Boundary Finding.
(4) Adjacent Pieces Merging.
Fig. 2. The block diagram of the proposed method.
8
2.1 Puzzle Piece Extraction
At the beginning of puzzle solving, a series of images are inputted. Each image contains one puzzle piece (see Fig. 3(a)). In order to extract all puzzle pieces, each puzzle piece is placed over a background layer with a predefined color. This color should be chosen to make the contour of a puzzle piece be extracted easily on the input image; therefore we select the color, excluding those boundary colors, as the background color.
The proposed method is quoted from Makridis and Papamarkos [17]. First, it uses the background color to differentiate the background region from the puzzle piece region. A binary image BI will be built by the following Equation:
background and some noises on the background might be considered as a part of the puzzle piece (see Fig. 3(b)). To overcome this problem, all connected components on BI are found.
Based on the hypothesis that the biggest connected component is either the puzzle piece or the background, all erroneous smaller connected components are reclassified.
More concretely, the input image is classified into two classes: (1) pixels with the background color and (2) all other pixels. Then, all connected components are found for both classes. After that, there will be two classes of objects; one for puzzle piece objects and
9
another for background ones. The biggest object of each class is considered as either puzzle piece or background. Each smaller object will be reclassified if it lies inside the biggest object, but belongs to the different class. Fig. 3 gives an example to illustrate puzzle piece extraction.
Fig. 3. Puzzle piece extraction. (a) The input image. (b) Binarization based on the predefined background color. (c) Extracted puzzle piece.
2.2 Feature Extraction
After correctly extracting all puzzle pieces, we will locate the boundary of each puzzle piece. All boundary points extracted are referred as sample points, which are chosen following the boundary of a piece pixel by pixel by selecting the next pixel from the 8-neighbors counterclockwise. However, two sets of sample points corresponding to the common boundary of two adjacent pieces may be different, that is, pixels on the common boundary of one piece are unmatched to those in the common boundary of the adjacent piece (see Fig.
4(a)). The occurrence of the above situation is due to the curvature of the boundary. Here, we
10
apply the method proposed by Nielsen et al. [15] to solve this problem. Their method uses the edge of the boundary to take the place of pixels on the boundary as sample points (see Fig.
4(b)).
Fig. 4. An example of sample points on two adjacent pieces. (a) Sample points extracted from pixels on the common boundary with black color standing for one piece and white for the other piece. (b) Sample points extracted from the edge of the common boundary with circles denoting sample points.
2.2.1. Shape Feature Extracting
The technique used to extract shape information is an adaptation of Chain code [20] with 8-neighbors. It extracts the direction of each sample point on the boundary as the shape feature. For each sample point X, points P1 and P2 are located at the right and left sides of X with the same distance (see Fig. 5(a)). Then, the direction from P1 to P2 is considered as the direction of X. Here, we design eight direction codes based on our coding map, which normalizes all directions to eight values 0 ~ 7 (see Fig. 5(b)).
11
Fig. 5. An example to illustrate direction code extraction. (a) A sample point X with P1 and P2 located at its right and left sides. (b) The direction coding map.
After obtaining the direction codes of all sample points on each puzzle piece, we group these codes to form a shape feature string to represent the shape information of the puzzle piece. Note that the proposed method wants to find adjacent pieces, and then these adjacent pieces are merged to form the original image. Each pair of adjacent pieces should have a common boundary. This boundary will correspond to a segment of shape feature string of each piece. The segments representing the common boundary of two adjacent pieces will be similar.
In order to reduce the complexity of finding pairs of similar segments between two puzzle pieces, the run-length coding (RLC) algorithm is used to encode all shape feature strings; the result is called run length coding string (RLCS). However, there are some small runs in RLCS; these will increase the complexity of finding pairs of similar segments. These small runs are produced from those sample points with directions near the boundary of two neighbor direction codes. A technique is provided to eliminate these small runs.
12
For each RLCS S with S
R1,R2,...,,Rj,...,Rm
, where Rj
Cj,Lj
represents the jth run with direction code C and run length j L . First, we classify all runs into four categories j by threshold T1, as shown in the following:
Category3 will be held. Runs in Category4 will be merged together with their neighbor runs in Category4. This process will be executed until no run in Category4. To treat the above merging process, the number of direction codes will be increased to 16 corresponding to
0,01,1,12,2,23,3,34,4,45,5,56,6,67,7,70
. Note that the new direction code C1C2 iscreated to represent the direction value of two merged runs with direction codes C1 and C2. For the convenience of using these values, we reset these direction codes to 0~15 (see Fig. 6).
After that, if there are still small runs in RLCSs, that is, those runs are classified into Category2 or Category3. They will be averagely reassigned to the neighbors on both sides.
13
Fig. 6. The new 16 direction codes. (a) Original 8 direction codes. (b) After merging each pair of two neighboring direction codes. (c) The new direction codes. (d) New direction coding map.
2.2.2. Color Feature Extracting
The puzzles discussed here contain not only shape but also color information. In order to enhance the accuracy of reconstructing puzzle pieces, we consider the color feature around the boundary. According to the research of Nielsen et al. [15], using all HSI color components (Hue, Saturation, and Intensity) as the color feature can get the most correct result on restoring puzzle pieces, so we choose the HSI color model to represent colors of an input image. Because shadow exists on the boundary of a puzzle piece, this makes colors of boundary points darker than expected ones. Fig. 7 shows an example. To solve this problem, we take an inside point near a sample point to get a color representing the color of the sample point. Here, a distance D is used for preventing from the shadow effect. Note that D depends on the input image. If the image is produced by the scanner, then D is set to 2. Otherwise, if the image is created by a person, then D is set to 0.
14
Fig. 7. An example to show that colors on boundary of a puzzle piece are darker due to shadows on the boundary.
For a sample point X, two sample points Y and Z are taken on its both sides within the same distance to get a line segment . Locate the point P inside the piece, and the line segment is perpendicular to with a distance D. Then, the average hue, saturation,
and intensity values of pixels within the 33window centered at P are computed. These values are considered as the color features of X. Finally, color features of all sample points are grouped to form a color feature string. Fig.8 illustrates the color feature extraction.
Fig. 8. An Example of color extraction with D 3.5.
15
2.3 Common Boundary Finding
To reconstruct the puzzle image, the first thing we have to do is to find a possible common boundary between two puzzle pieces. Note that this boundary will correspond to a pair of similar segments in these two pieces’ RLCSs. In this section, we will describe how to find the most similar segment pair of two puzzle pieces. In some situations, it is difficult to find the correct pair of similar segments in two adjacent puzzle pieces by only using shape information, since there might be more than one pair of similar segments in two puzzle pieces (see Fig. 9). To treat the above problem, a common boundary finding technique is provided based on both shape and color features.
Fig. 9. Examples of multiple similar segments with only shape information used. (a) and (c) Results of correct pairs of similar segments. (b) and (d) Results of incorrect pairs of similar segments.
16
In the proposed technique, first, for two puzzle pieces, several possible common boundaries, which are apt to merge these two puzzle pieces, are found based on their RLCSs.
Second, for each possible common boundary, the matching points of this boundary, which can align two corresponding similar segments representing the possible boundary, are located based on the shape and color feature strings. Finally, the best possible common boundary will be extracted. Following is the details of our common boundary finding method.
First, the proposed method attempts to find several possible common boundaries between two puzzle pieces based on their RLCSs. This method is an adaption of Smith-Waterman algorithm (SWA) [21], which is designed for determining similar regions between two nucleotide or protein sequences. It performs local sequence alignment, that is, SWA can find a pair of similar matching segments between two strings. Note that each element in a string is a single symbol. However, our purpose is to find several pairs of similar matching segments based on RLCSs, in which each element contains two values: direction code and its run length. Thus, SWA is modified to meet this situation. The details are described as follows.
Let S1
R11,...,R1m
andS2
R21,...,R2n
be the RLCSs of two puzzle pieces, where m is the number of runs in S1, and n is the number of runs in S2. Assume that mn, = (R11,R12,…,R1m,R11,R12,…,R1m) and = (R21,R22,…,R2n,R21,R22,…,R2m). and are created to be circular strings, they are used to deal with the problem that for a similar segment17
pair, each segment may consist of two separated substrings, one in the front of a RLC string, the other in the rear of this string. Fig. 10 shows an example to illustrate this problem.
Fig. 10. An example of a real similar segment pair in which each segment consists of two separated substrings, one in the front of a RLCS, and the other in the rear of this string. Solid line represents the front substring, and dotted line represents the rear substring.
Then, a (2m1)(mn1)matrix H is built as follows:
18
Where w(a,b) is a gap-scoring scheme, a,bS1, S2 ,or
'' , '' stands for missing element. R1i(R2j) is the ith (jth) run in S1 (S2). SimiMat is a similarity measure matrix to represent the similarity of two direction codes (see Table 2).Table 2. The SimiMat matrix values with codes C1 and C2.
2
1 C
C 0 1 2 3 4 5 6 7
SimiMat[C1][C2] 10 5 -1 -4 -8 -10 -10 -10
2
1 C
C 8 9 10 11 12 13 14 15
SimiMat[C1][C2] -10 -10 -10 -10 -8 -4 -1 5
Note that in SWA, two similar strings may have some insertions and deletions. But in puzzle solving, two similar strings should represent the common boundary of two adjacent puzzle pieces in the original puzzle image. Thus, only few insertions and deletions are allowed. This means that for a pair of similar segments, the number of consecutive unmatched elements should be less than a threshold T2, which depends on the size of puzzle pieces.
Therefore, on the process of building matrix H, a counter is used to record the number of consecutive unmatched elements. If the counter is larger than T2 in calculating the value of
) , (i j
H , the value of H(i,j) will be reset to 0.
Note that H( ji, ) stands for the maximum similarity score between segment MS1 )
..., ,
(R1i' R1i and segment MS2(R2j',...,R2j), where R1i (R2j) is the beginning run of MS1
19
(MS2). The bigger the value of H(i, j) is, the more similar the pair of segments is.
To locate R1i and R2j for each similar segment pairs, we start from position
i, j , then, we go backwards to one of positions
i1,j
,
i,j1
, and
i1,j1
depending on the direction of movement used to build H( ji, ). Keep the process until it reaches a position
i' j, ' with H(i',j')0 or position
0,0 .Based on matrix H we can extract several similar segments. The top 10 largest values in H are extracted, each of which corresponds to a pair of similar segments. However, there may
be some duplicates in these ten pairs since strings ( ) is a concatenation of S1 (S2).
Therefore, all pairs of similar segments are examined and duplications are discarded.
After several pairs of similar segments between two puzzle pieces are found, our method attempts to find a pair of matching points in RLCSs for each pair of similar segments, so that we can merge these two puzzle pieces based on the matching points.
Note that a pair of similar segments found might consist of matching runs and several insertions or deletions (see Fig. 11). That is each segment can be split into several sub-segments by these insertions or deletions. Based on this phenomenon, we take the longest sub-segment in each segment; it has the maximum similarity. Assume that the locations of matching points will be situated at the longest sub-segments. Then, the shape and color features on these two sub-segments are used to find the best pair of matching points on these two segments. More details are described as follows.
20
Fig. 11. A pair of similar segments with arrows standing for matching runs.
Let LS1 (LS2) be the longest sub-segment of MS1 (MS2), LsS1 (LsS2) be the result string of run-length decoding (RLD) for LS1 (LS2) with length l (l'), LsS1(C'11,C'12,...,C'1l) (LsS2(C'21,C'22,...,C'2l')), and LcS1 (LcS2) be the sub-string of color features of pixels on LS1 (LS2) with length l (l'), LcS1(CS'11,CS'12,...,CS'1l) (LcS2(CS'21,CS'22,...,CS'2l')).
LsS1 and LsS2 are used to estimate a Local Shape Similarity (LSS), and LcS1 and LcS2 are used to estimate a Local Color Difference (LCD). In case these two sub-segments LsS1 and LsS2 are not perfectly aligned, that is ll', the shorter sub-segment may be shifted relative to the longer one to find the maximum similarity measure (see Fig. 12).
21
Where index u denotes the number of times the shorter sub-segment shifts. After obtaining all LSSu and LCDu, the proposed method finds the maxima LSSmax and LCDmax, and the minima LSSmin and LCDmin, then uses these values to estimate the Local Similarity Measure (LSM) by Equation (8) for each possible pair of matching points.
Where α is set to determine the contribution of shape and color features for LSM. The pair
22
of points with maximum LSM will be considered as the pair of matching points for segments MS1 and MS2.
According to the above method, a pair of matching points is found for each similar segment pair. After that, Shape Similarity (SS) and Color Difference (CD) are estimated for each pair of similar segments. Let (x, y) be the pair of matching points for a pair of similar segments MS1 and MS2, MsS1 and MsS2 be the result sub-string of RLD for MS1 and MS2 with length k and k’, and McS1 and McS2 be the corresponding color feature substrings of MS1 and MS2 with length k and k’, MsS1 = (C11,C12,…,C1k), MsS2 = (C21,C22,…,C2k’), McS1 = (CS11,CS12,…,CS1k), and McS2 = (CS21,CS22,…,CS2k’). Assume that x < y and k < k’, the SS and CD will be calculated as follows:
all SSs and CDs. Finally, the maximum SM is considered as the similarity measure of the pair23
of two pieces, and the pair of similar segments with maximum SM will be considered as the possible common boundary between these two pieces. If SM is close to 1 for a possible common boundary, the corresponding two puzzle pieces are more suitable to be merged.
2.4 Adjacent Pieces Merging
The final destination is to correctly restore the puzzle pieces; the aforementioned SM is used to reconstruct the puzzle piece by piece. First of all, the similarity measures of all pairs of puzzle pieces are ranked from maximum to minimum. Secondly, according to the order, our merging procedure executes the following steps:
(1) Take the pair of two puzzle pieces with maximum SM and merge them based on their matching points.
(2) Based on the merged piece, take the next pair of puzzle pieces, where one of them is a merged piece. If the overlap of the unmerged piece with the merged piece occupies 20% of the unmerged piece, it would not be merged. Otherwise, this piece will be merged with the merged piece based on the pair of matching points.
(3) Repeat Step (2) until all pieces have been merged.
The result will be considered as the original puzzle image.
24
CHAPTER 3
SEMI-AUTOMATIC PUZZLE SOLVER
The proposed method attempts to reconstruct all kinds of puzzle pattern without any common puzzle conditions. According to the adjacent pieces merging step described in Chapter 2.4, our method will automatically select a pair of puzzle pieces in each step, and merges them by their matching points, until all the pieces have been merged. However, our method might select few incorrect pairs of matching pieces. These errors occur when nonadjacent puzzle pieces have texture or similar colors and shapes near boundary. Fig. 13 demonstrates these situations. In Fig. 13(a), two nonadjacent puzzle pieces are merged due to that they have texture boundaries. In Figs. 13(b) and (c), two nonadjacent puzzle pieces are merged due to that their boundaries have similar shapes and colors. These errors are hard to be solved automatically, and this will result in that our solver cannot restore the puzzle pieces to the original image completely.
25
Fig. 13. Incorrect matching pairs of pieces selected by our method.
In order to overcome this problem, we propose a semi-automatic solver to successfully restore puzzle pieces because of these errors may be correctly recognized by a person. This solver uses the proposed method described in Chapter 2, but it allows a user deciding if the current merging step is correct or not. If a pair of matching pieces is recognized as an incorrect merging, this pair of puzzle pieces will not be merged. Fig. 14 gives an example.
Nevertheless, there are still incorrect matching pieces that cannot be recognized by the user, so our solver implements a backtracking method that can remove some error merging. Finally, we can get the correct result of reconstructing puzzle pieces. Fig. 15 shows an example of the backtracking method. In Fig. 15(a), piece R2 is selected and the user determines that the merging of R1 and R2 is correct. After several mergings, R2, R3, and R4 are merged. Then, in Fig. 15(b), piece R5 is selected, and the user finds that the merging of R1 and R2 is incorrect;
the previous mergings including R2 are removed. The result is shown in Fig 15(c).
26
Fig. 14. User decision on the pieces merging step of puzzle solver. (a) D5 is selected in the merging step, and the user determines that this step is correct. (b) C3 is selected in the next step. (c) D2 is selected in the merging step, and the user determines that this step is incorrect.
(d) D2 is discarded by the solver, and A1 is selected in the next step.
Fig. 15. An example of backtracking method. (a) Piece R2 is selected and merged with R1.
(b) When R5 is selected, the merging of R2 and R1 is found incorrect. (c) The result after backtracking with R2, R3, and R4 removing.
27
To sum up, the block diagram of the semi-automatic puzzle solver is shown in Fig. 16 and it is depicted as follows:
(1) Our solver automatically extracts shape and color features for all puzzle pieces.
(2) It calculates a similarity measure for each pair of pieces.
(2) It calculates a similarity measure for each pair of pieces.