Adjacent Pieces Merging

The final destination is to correctly restore the puzzle pieces; the aforementioned SM is used to reconstruct the puzzle piece by piece. First of all, the similarity measures of all pairs of puzzle pieces are ranked from maximum to minimum. Secondly, according to the order, our merging procedure executes the following steps:

(1) Take the pair of two puzzle pieces with maximum SM and merge them based on their matching points.

(2) Based on the merged piece, take the next pair of puzzle pieces, where one of them is a merged piece. If the overlap of the unmerged piece with the merged piece occupies 20% of the unmerged piece, it would not be merged. Otherwise, this piece will be merged with the merged piece based on the pair of matching points.

(3) Repeat Step (2) until all pieces have been merged.

The result will be considered as the original puzzle image.

24

CHAPTER 3

SEMI-AUTOMATIC PUZZLE SOLVER

The proposed method attempts to reconstruct all kinds of puzzle pattern without any common puzzle conditions. According to the adjacent pieces merging step described in Chapter 2.4, our method will automatically select a pair of puzzle pieces in each step, and merges them by their matching points, until all the pieces have been merged. However, our method might select few incorrect pairs of matching pieces. These errors occur when nonadjacent puzzle pieces have texture or similar colors and shapes near boundary. Fig. 13 demonstrates these situations. In Fig. 13(a), two nonadjacent puzzle pieces are merged due to that they have texture boundaries. In Figs. 13(b) and (c), two nonadjacent puzzle pieces are merged due to that their boundaries have similar shapes and colors. These errors are hard to be solved automatically, and this will result in that our solver cannot restore the puzzle pieces to the original image completely.

25

Fig. 13. Incorrect matching pairs of pieces selected by our method.

In order to overcome this problem, we propose a semi-automatic solver to successfully restore puzzle pieces because of these errors may be correctly recognized by a person. This solver uses the proposed method described in Chapter 2, but it allows a user deciding if the current merging step is correct or not. If a pair of matching pieces is recognized as an incorrect merging, this pair of puzzle pieces will not be merged. Fig. 14 gives an example.

Nevertheless, there are still incorrect matching pieces that cannot be recognized by the user, so our solver implements a backtracking method that can remove some error merging. Finally, we can get the correct result of reconstructing puzzle pieces. Fig. 15 shows an example of the backtracking method. In Fig. 15(a), piece R2 is selected and the user determines that the merging of R1 and R2 is correct. After several mergings, R2, R3, and R4 are merged. Then, in Fig. 15(b), piece R5 is selected, and the user finds that the merging of R1 and R2 is incorrect;

the previous mergings including R2 are removed. The result is shown in Fig 15(c).

26

Fig. 14. User decision on the pieces merging step of puzzle solver. (a) D5 is selected in the merging step, and the user determines that this step is correct. (b) C3 is selected in the next step. (c) D2 is selected in the merging step, and the user determines that this step is incorrect.

(d) D2 is discarded by the solver, and A1 is selected in the next step.

Fig. 15. An example of backtracking method. (a) Piece R2 is selected and merged with R1.

(b) When R5 is selected, the merging of R2 and R1 is found incorrect. (c) The result after backtracking with R2, R3, and R4 removing.

27

To sum up, the block diagram of the semi-automatic puzzle solver is shown in Fig. 16 and it is depicted as follows:

(1) Our solver automatically extracts shape and color features for all puzzle pieces.

(2) It calculates a similarity measure for each pair of pieces.

(3) It ranks these similarity measures from maximum to minimum and restores the puzzle image based on this order.

(4) This solver implements the adjacent pieces merging step described in Chapter 2.4, but it requests the user to determine if two selected puzzle pieces should be merged in each merging.

Fig. 16. The block diagram of the semi-automatic puzzle solver.

28

CHAPTER 4

EXPERIMENTAL RESULTS

The final goal of the semi-automatic puzzle solver is to restore all kinds of puzzles into their original images. Here, we will calculate the number of times for incorrect merging. Note that the solver will keep solving until all puzzle pieces are placed in their correct locations.

In this thesis, we present six puzzle images. Four of them are produced by factitiously splitting original puzzle images into various shapes of pieces. The other two are real-world puzzle pieces derived from the database of Nielsen et al. [16] which scan real-world puzzle pieces by a scanner. These puzzles have been chosen to show that this solver can cope with any kinds of puzzle pieces. The parameter  is set to 0.5. And if the puzzle images are created by a person, D is set to 0. Otherwise, if they are created by a scanner D is set to 2.

Example 1 (Blockhouse): In this example, the original puzzle image has been artificially divided into 13 pieces with different shapes (see Fig. 17). The proposed solver restores this puzzle without any incorrect merging. The result is shown in Fig. 18.

Example 2 (Lake): In this example, the original puzzle image has been artificially

divided into 31 varied pieces (see Fig. 19). This puzzle is restored with 6 incorrect mergings.

The result is shown in Fig. 20.

29

Example 3 (Leopard): This puzzle is a real-world puzzle of 24 pieces. The proposed

solver restores this puzzle with 16 incorrect mergings. The result is shown in Fig. 21.

Example 4 (Benjamin): This puzzle is a real-world puzzle of 54 pieces. This puzzle is

restored with 22 incorrect mergings. The result is shown in Fig. 22.

Example 5 (Construction50, Construction160): This puzzle image depicts a construction

with varies colors and texture (see Figs. 23 and 24); it is derived from Nielsen et al. [16]. In this example, we divide this image into 50 and 160 rectangular pieces. The piece size of the first puzzle image is 154198 pixels, and the other is100104 pixels. The First puzzle is restored without any incorrect merging, and the result is shown in Fig. 25. The second puzzle of 160 pieces is also successfully restored with 59 incorrect mergings.

In our proposed solver,  value is the key parameter to affect the number of incorrect mergings. This value is set to determine the contribution of shape and color information, and it is normalized to [0, 1]. If  is close to 1, the proposed solver coping with the puzzle mainly depends on shape information. On the contrary, if  is close to 0, our solver coping with the puzzle mainly depends on color information. In Table 3, five puzzles (Benjamin, Blockhouse, Construction50, Lake, Leopard) are used to analyze the effect of  by computing the number of incorrect mergings occurred. From this table, we can see that each puzzle can be restored with few incorrect mergings by using a specific  value. The selection of  will be based on the characteristics of each puzzle, if the colors and textures

30

in a puzzle are similar, then this puzzle should be restored using the shape information, that is

 is set to close 1, like Leopard. On the other hand, if the shapes of pieces are similar, then

this puzzle should be restored using the color information, that is  is set to close 0, like Construction50. We can also see that when 0.5 our puzzle solver can restore all puzzles effectively. Therefore  is suggested to set to 0.5.

Table 3. Analysis results on different . Each value in this table is the number of incorrect merging to restore the designated puzzle. ‘-‘ means that this puzzle cannot be restored.

Puzzle

 ^Benjamin Blockhouse Construction50 Lake Leopard

0.1 45 0 0 - 98

0.2 29 0 0 11 86

0.3 19 0 0 6 56

0.4 20 0 0 2 33

0.5 22 0 0 6 16

0.6 30 0 8 10 16

0.7 45 2 55 16 16

0.8 49 2 - 28 2

0.9 63 2 - 49 4

31

Fig. 17. Puzzle pieces of puzzle Blockhouse.

Fig. 18. The restored image of puzzle Blockhouse.

32

Fig. 19. Puzzle pieces of puzzle Lake.

Fig. 20. The restored image of puzzle Lake.

33

Fig. 21. The restored image of puzzle Leopard.

Fig. 22. The restored image of puzzle Benjamin.

34

Fig. 23. Puzzle pieces of puzzle Construction50.

Fig. 24. Puzzle pieces of puzzle Construction160.

35

Fig. 25. The restored image of puzzle Construction.

36

CHAPTER 5 CONCLUSIONS

This thesis provides a semi-automatic puzzle solver, which is based on shape and color information. The target puzzles are not restricted by some specific patterns. And the solver does not use any common puzzle limitations. The experiments were performed with real-world and artificial puzzle images. The experimental results show that our method is successful for restoring all kinds of puzzle pieces.

In this method, first, the shape and color features are extracted; they are next used to estimate the similarity measure of a possible common boundary for each pair of puzzle pieces.

Then, a simple user interactive method is provided to allow a user deciding if two selected pieces are adjacent or not. Our solver will select another pair of puzzle pieces which may be adjacent as soon as the user decides previous selected pieces are not adjacent. This method makes any kinds of puzzle pieces be restored completely. And a user can easily restore a puzzle by our puzzle solver.

37

REFERENCES

[1] H. C. da Gama Leitao and J. Stolfi, “Automatic reassembly of irregular fragments,” Univ.

Campinas, Campinas, Brazil, Tech. Rep. IC-98-06, 1998.

[2] H. C. da Gama Leitao and J. Stolfi, “Information contents of fracture lines,” Univ.

Campinas, Campinas, Brazil, Tech. Rep. IC-99-24, 1999.

[3] A. R. Willis and D. B. Cooper, “Computational reconstruction of ancient artifacts,”

IEEE Signal Process. Mag., Vol. 25, No. 4, pp. 65–83, Jul. 2008.

[4] F. S. Kuhl, G. M. Crippen, and D. K. Friesen. “A combinatorial algorithm for calculating ligand binding,” Journal of Computational Chemistry, Vol. 5, No. 1, pp. 24-34, 1984.

[5] Elaine C. Meng and Irwin D. Kuntz, “Molecular docking: a tool for ligand discovery and design,” Drug Information Journal, Vol. 28, pp. 735-749, 1994.

[6] Y.-X. Zhao, M.-C. Su, Z.-L. Chou, and J. Lee, “A puzzle solver and its application in speech descrambling,” In ICCEA, 2007.

[7] L. Zhu, Z. Zhou, and D. Hu, “Globally consistent reconstruction of ripped-up documents,” IEEE TPAMI, 2008.

[8] H. Freeman and L. Gardner, “A pictorial jigsaw puzzles: The computer solution of a problem in pattern recognition,” IEEE Trans. Electron. Comput., Vol. EC-13, No. 2, pp.

118–127, Apr. 1964.

38

[9] G. Radack and N. Badler, “Jigsaw puzzle matching using a boundary-centered polar encoding,” Comput. Graphics Image Process. 19, pp. 1–17, 1982.

[10] H. Wolfson, E. Schonberg, A. Kalvin, and Y. Lamdan, “Solving jigsaw puzzles by computer,” Ann. Oper. Res., Vol. 12, No. 1, pp. 51–64, Dec. 1988.

[11] D. Goldberg, C. Mallon, and M. Bern, “A global approach to automatic solution of jigsaw puzzles,” in Proc. 18th Annu. Symp. Comput. Geometry, Barcelona, Spain, pp.

82–87, 2002.

[12] D. Kosiba, P. Devaux, S. Balasubramanian, T. Gandhi, and R. Kasturi, “An automatic jigsaw puzzle solver,” In: Proceedings 12th IAPR International Conference on Computer

Vision and Image Processing, Jerusalem, Vol. 1, pp. 616–618, October 1994.

[13] M. G. Chung, M. M. Fleck, and D. A. Forsyth, “Jigsaw puzzle solver using shape and color,” in Proc. 4th Int. Conf. Signal Process., pp. 877–880, 1998.

[14] F. H. Yao and G. F. Shao, “A shape and image merging technique to solve jigsaw puzzles,” Pattern Recognit. Lett., Vol. 24, No. 12, pp. 1819–1835, Aug. 2003.

[15] T. R. Nielsen, P. Drewsen, and K. Hansen, “Solving jigsaw puzzles using image features,”

Pattern Recognit. Lett., Vol. 29, No. 14, pp. 1924–1933, Oct. 2008.

[16] M. Makridis and N. Papamarkos, “A new technique for solving puzzles,” IEEE Trans. On systems, man, and cybernetics-part B: cybernetics, Vol. 40, No. 3, June 2010.

39

[17] D. Chetverikov and Z. Szabo, “A simple and efficient algorithm for detection of high curvature points in planar curves,” Robust Vis. Ind. Appl., Vol. 128, pp. 175–184, 1999.

[18] T. Kohonen, “Self-Organizing Maps”, 2nd ed. Berlin, Germany: Springer-Verlag, 1997.

[19] H. Freeman, “On the encoding of arbitrary geometric configurations,” IRE Trans.

Electron. Comput., Vol. 10, No. 2, pp. 260–268, Jun. 1961.

[20] T. F. Smith and M. S. Waterman, "Identification of Common Molecular Subsequences,”

Journal of Molecular Biology 147, pp. 195–197, 1981.