An Improved Similarity Measure for Image Database Based on 2D C+-string

全文

(1)Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. An Improved Similarity Measure for Image Database Based on 2D C+ -string Yeong-Ren Jean. Hao-Yu Lo. Department of Computer Science and Information Management, Providence University, Taichung, Taiwna, R.O.C. [email protected] and [email protected] Abstract -In an image database system, the spatial knowledge representation is a technique of abstraction to describe an image. The 2D string and its variants are based on this concept. One of the variants, called 2D C+ -string, considers sizes of and distances between objects. This method presents three advantages: (1) more accuracy in picture representation and reconstruction; (2) less ambiguity in similarity retrieval; (3) reasoning about relative locations for a symbolic picture is possible. However, the similarity measure based on 2D C+ -string doesn’t consider the ratios about sizes of and distances between objects on x- and y-axis together. This neglect may cause distorted result of similarity retrieval. In this paper, we modify the original similarity measure for taking down the variation of ratios between two symbolic pictures and furthermore propose two new types of similarity measure for discriminating pictures more precise. By exploiting the ratios, the pictorial query is more flexible for certain demands. Keywords: Spatial knowledge,2D C+ -string, Similarity retrieval, Pictorial query.. 1. Introduction An image database system is different from conventional database systems in two respects. First is the raw data of image is a visual information rather than pure numeral or alphabet. Therefore, it is very important to describe some features from the original image for representation and image indexing. Second is the system needs to have the capability of similarity retrieval for adapting human perception. In the research of image representation, one technique is to abstract an original image as a symbolic picture. A symbolic picture is constituted by some symbols or vocabularies, and these symbols or vocabularies represent iconic objects or real entities in an image. Chang et al. [2] proposed the 2D string to represent a symbolic picture based on the knowledge structure of symbolic projections. The basic idea of 2D string is that iconic objects will be pro-. 547. E B A (a). u: A=B<C<D=E v: A=D<B=C<E. C D (b). Figure 1. (a) an original image (b) its symbolic picture and 2D string. jected on the rectangular coordinate plane along xand y-coordinates respectively to build two strings for describing relative positions among objects on xand y-axis. This approach provides an efficient and natural way to construct iconic indexes for symbolic pictures. A symbolic picture and its 2D string are shown in Figure 1. For the capability of similarity retrieval, three types of 2D subsequence (type-i, i=0, 1, 2) based on 2D string are defined. Therefore, the problem of image retrieval then can be envisioned as a problem of 2D subsequecne matching [2]. Subsequently, many spatial knowledge representations for image were proposed, such as 2D G-string, 2D C-string, 2D C+ -string and so on [1, 4, 6]. And there are corresponding similarity measures based on these representations. However, the similarity measure based on 2D C+ string doesn’t consider the ratios about sizes of and distances between objects on x- and y-axis together. This problem may cause distorted pictures of typei1 and type-i2 are retrieved. In this paper, we improve the similarity measure based on 2D C+ -string. We modify the equations of type-i1 and type-i2 for taking down the variation of ratios and then explicit the ratios to propose two new types of similarity measure. The improved method provides two major advantages: 1. More precision in similarity retrieval because the additional two new types of similarity measure can discriminate the case of the ratios about sizes on x- and y-axis are equal, so dose the case of distances. 2. More flexibility in the pictorial query because we can assign the specific ratios about sizes of and distances between objects for certain demands. The remainder of this paper is organized as fol-.

(2) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. (1) A<B. objects will cause ambiguity in picture representation, reconstruction, and similarity retrieval. Thus, Huang et al. [4] proposed 2D C+ -string, which extends the work of Lee et al. [6,7] by attaching metric information to the strings. Such information can be easily counted by pixels in an image. In the 2D C+ string, each object in a symbolic picture is given two pairs of begin-bounds and end-bounds, one for x-axis and the other for y-axis to calculate the sizes and the distances for objects.. (2) B<A. A. A B. (3) A|B. B (4) B|A A. A B. B (6) B%A. (5) A%B A. A. B. B. (7) A[B. Table 1. The definitions of spatial operators. Notations Conditions A<B end(A) < begin(B) A|B end(A) = begin(B) A%B begin(A) < begin(B), end(A) > end(B) A[B begin(A) = begin(B), end(A) > end(B) A]B begin(A) < begin(B), end(A) = end(B) A/B begin(A) > begin(B), end(A) > end(B) A=B begin(A) = begin(B), end(A) = end(B). (8) B[A A. A. B. B. (9) A]B. (10) B]A A. A B. B (12) B/A. (11) A/B. A. A B. B. (13) A=B A B. Figure 2. The 13 patterns of spatial relations between two objects in one dimension.. The symbolic pictures and their corresponding 2D C+ -string extended from 2D C-string with metric information in one dimension are represented as follows: 1. As denote object A with size s where s=end(A)-begin(A). Following object is represented by A6 . 3 9. A lows. Section 2 reviews some variants of 2D string. Section 3 introduces the similarity measure based on 2D C+ -string and its drawback. The improved similarity measure based on 2D C+ -string and an example are presented in section 4. Finally, the conclusions are given in the last section.. 2. Related Works After 2D string was proposed, Chang et al. [1] extended the idea of 2D string by adding a cutting mechanism that called 2D G-string to describe more complex spatial relations among non-zero sized objects. However, more cutting performed in an image will cause more subparts, and then the length of 2D G-string will increase. To solve the disadvantage of 2D G-string, Lee et al. [6] proposed 2D C-string with a new cutting mechanism based on thirteen patterns of spatial relations between two objects in one dimension. The length of 2D C-string is shorter significantly than 2D G-string because the number of cutting is reduced. The thirteen patterns of spatial relations are shown in Figure 2 and the definitions of spatial operators are shown in Table 1. Although the length of 2D C-string is shortened, the neglect about the sizes of and distances between. 548. 2. Operator “<” with distance d between object A and B, denoted as A< dB, d=begin(B)-end(A). Following pattern is represented by A< 3 B.. 6. 9. A. B. 3.Operator “%” with distance d between object A and B, denoted as A%dB, d=begin(B)-end(A). Following pattern is represented by A%5 B.. 7 B. 2 A. 4.Other operators: no metric information is needed. However, the similarity measure based on 2D C+ string may cause distorted result of similarity retrieval.. 3. Similarity Measure Based on 2D C+ -string and Its Drawback The similarity retrieval is very important since pictures are represented slightly different in image.

(3) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. E. E (6,9). D. C. D. A C (2,6) (1,5). (17,9). A. (5,6). (8,6). (20,6). (1,5). B. B (21,1). (6,1). f1. f2. f1 : u : A2 ]C1 |C1 <1 D4 %1 (B2 = E2 ) f2 : u : A8 ]C1 |C7 <1 E7 ](D4 ]B3 )|D14 [B1. v : B1 <3 A2 ](C1 = D1 )|C1 = D1 <1 E2 v : B1 <3 A2 ](C1 = D1 )|C1 = D1 <1 E2. Figure 3. An example of the original similarity measure between f1 and f2 . database may regard as the same by a user. Therefore, the system needs a similarity measure for evaluating the similarity of images. The following notations will be used in defining the similarity measure based on 2D C+ -string: [4] • (AB ), a pair objects A and B, it is regarded as the type-ij similar pair. • C AB , the spatial relation for (AB ) corresponds to the category that classified by Lee et al [7]. u • rAB , the spatial relation for (AB ) on x-axis. v , the spatial relation for (AB ) on y-axis. • rAB. • S Au , the size of object A on x-axis. • S Av , the size of object A on y-axis. u • DAB , the ditance between object A and B on x-axis. v , the ditance between object A and B on • DAB y-axis.. The definition of similar type between two symbolic pictures is the picture f ’ is a type-ij similar unit picture of a picture f. The picture f and f ’ both contain (AB ), represented as 1. In f , u u: A rAB B u SAu DAB SBu. v v : A rAB B v SAv DAB SBv. 2. In f ’ , u0 B u: A rAB u0 SAu0 DAB SBu0. v B v : A rAB v0 SAv0 DAB SBv0. 5. (type-11): type-10 and (SAu /SBu = SAu0 /SBu0 and SAv /SBv = SAv0 /SBv0 ) 6. (type-21): type-20 and (SAu /SBu = SAu0 /SBu0 and SAv /SBv = SAv0 /SBv0 ) 0. u u 7. (type-02): type-01 and (SAu /DAB = SAu0 /DAB 0 v v and SAv /DAB = SAv0 /DAB ) 0. u u = SAu0 /DAB 8. (type-12): type-11 and (SAu /DAB 0 v v and SAv /DAB = SAv0 /DAB ) 0. u u = SAu0 /DAB 9. (type-22): type-21 and (SAu /DAB 0 v v and SAv /DAB = SAv0 /DAB ). According to the definition of most similar picture between two pictures was proposed by Lee et al. [7] and the similarity retrieval algorithm based on 2D C+ -string [4]. We illustrate the drawback of similarity measure based on 2D C+ -string with the example shown in Figure 3. There are five objects (A, B, C, D, and E) between two pictures f1 and f2 respectively. The type-20 similar subpicture of f1 and f2 contains objects A, B, C, and D. Then type-21 similar subpicture contains objects A, B, and C. The finest granularity, type-22 similar subpicture contains objects A and B. Obviously, type21 and type-22 similar subpictures between f1 and f2 are distorted critically because the ratios about sizes and distances of objects on x- and y-axis aren’t taken into consideration together.. 0. The nine types of similarity (type-ij, i =0, 1, 2, and j =0, 1, 2) that the pair objects (AB ) both in picture f and f 0 are defined as follows: 0. 0. 0. 0. 0. 0. u v u v 1. (type-00): C(rAB , rAB ) = C(rAB , rAB ) v u v u ) and , rAB ) = C(rAB , rAB 2. (type-10): C(rAB 0 0 v v u u (rAB = rAB or rAB = rAB ) u v u v 3. (type-20): C(rAB , rAB ) = C(rAB , rAB ) and v0 v u0 u (rAB = rAB and rAB = rAB ). 4. (type-01): type-00 and (SAu /SBu = SAu0 /SBu0 and SAv /SBv = SAv0 /SBv0 ). 549. 4. An Improved Similarity Measure Based on 2D C+ -string In the improved similarity measure, we change the original equation (SAu /SBu = SAu0 /SBu0 and SAv /SBv = SAv0 /SBv0 ) in type-i 1 into a new equation (SAu /SAu0 = SBu /SBu0 and SAv /SAv0 = SBv /SBv0 ). The original u u0 v equation (SAu /DAB = SAu0 /DAB and SAv /DAB = 0 v SAv0 /DAB ) in type-i 2 is modified as a new equau u0 tion (SAu /SAu0 = DAB /DAB and SAv /SAv0 = 0 v v DAB /DAB ). The new equations can compute the ratios about sizes of and distances between.

(4) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan. objects.Then we add a parameter k for representing the variation of ratios between two symbolic pictures. By exploiting the parameter k, we can retrieve pictures more precise if the ratios on xand y-axis are equal. Notice that the notations of parameter k have different meanings in size and distance. • kus , the ratio about size on x-axis for (AB ) between two symbolic pictures. • kvs , the ratio about size on y-axis for (AB ) between two symbolic pictures. • kusd , the ratio about size and distance on x-axis for (AB ) between two symbolic pictures. • kvsd , the ratio about size and distance on y-axis for (AB ) between two symbolic pictures. The improved similarity measure between the pair objects (AB ) in a picture f and the same pair objects (AB ) in another pictures f 0 are defined as follows: 0. 0. 0. 0. 0. 0. v u v u ) , rAB ) = C(rAB , rAB 1. (type-00): C(rAB u v u v 2. (type-10): C(rAB , rAB ) = C(rAB , rAB ) and v0 v u0 u (rAB = rAB or rAB = rAB ) u v u v 3. (type-20): C(rAB , rAB ) = C(rAB , rAB ) and u u0 v v0 (rAB = rAB and rAB = rAB ). 4. (type-01): type-00 and (SAu /SAu0 = SBu /SBu0 =1/kus and SAv /SAv0 = SBv /SBv0 = 1/kvs ) 5. (type-11): type-10 and (SAu /SAu0 = SBu /SBu0 =1/kus and SAv /SAv0 = SBv /SBv0 = 1/kvs ) 6. (type-21): type-20 and (SAu /SAu0 = SBu /SBu0 =1/kus and SAv /SAv0 = SBv /SBv0 = 1/kvs ) 0. u u 7. (type-02): type-01 and (SAu /SAu0 = DAB /DAB 0 sd v v sd =1/ku and SAv /SAv0 = DAB /DAB = 1/kv ) 0. u u 8. (type-12): type-11 and (SAu /SAu0 = DAB /DAB 0 sd v v sd =1/ku and SAv /SAv0 = DAB /DAB = 1/kv ) 0. u u /DAB 9. (type-22): type-21 and (SAu /SAu0 = DAB 0 sd v v sd =1/ku and SAv /SAv0 = DAB /DAB = 1/kv ). 10. (type-31): type-21 and kus = kvs 11. (type-32): type-22 and kusd = kvsd For clear understanding, we divide the improved similarity measure into four parts for explaining. First, type-i 0 similarity is the same as proposed by Huang et al. [4]. Second, although the original equation (SAu /SBu =SAu0 /SBu0 ) and the new equation (SAu /SAu0 =SBu /SBu0 ) in type-i 1 have the same result in the form of (SAu ∗SBu0 = SAu0 ∗SBu ), they mean two different implications. The original equation compares the ratios about sizes for a pair objects (AB ) on x-axis in one picture to another that also contains the same pair objects (AB ). The improved equation compares the ratios about sizes. 550. for objects A between two pictures to the ratios about sizes for objects B between the same two pictures on x-axis. The new one can help us acquire the ratios of pair objects (AB ) on x-axis between two pictures. Then, we add a notation 1/kus for representing the ratio about sizes for (AB ) on x-axis. The same modifications are applied to y-axis, such as (SAv /SAv0 = SBv /SBv0 = 1/kvs ). Notice that the notations 1/kus and 1/kvs directly add after original equation can’t acquire the ratios about sizes. An example is shown in Figure 3. Based on the improved type-21 similarity, we can realize the ratios about sizes on x- and y-axis for (AC ) are different since (SAu /SAu0 = SCu /SCu0 ) ⇒2/8=2/8 and (SAv /SAv0 = SCv /SCv0 ) ⇒2/2=2/2. However, based on the original type-21 similarity, we can’t realize the ratios about sizes on x- and y-axis for (AC ) are different since (SAu /SBu =SAu0 /SBu0 ) ⇒2/2=8/8 and (SAv /SBv =SAv0 /SBv0 ) ⇒2/2=2/2. The reason is the ratios about sizes of objects on x- and y-axis aren’t considered together. Third, similar modifications also perform on the original type-i 2. The new equation of u u0 type-i 2 is (SAu /SAu0 = DAB /DAB =1/kusd and 0 v v sd SAv /SAv0 = DAB /DAB =1/kv ). Finally, we can propose two new types of similarity measure for retrieving pictures more precise because we have the ratios about sizes and distances for each pair objects on x- and y-axis. The definition of type-31 is (AB ) satisfies type-21 and the ratios about sizes on x- and y-axis are equal. The type-32 satisfies type-22 and the ratios about sizes and distances on x- and y-axis are equal. There is a clear summary is shown in Table 2. Notice that the sign of ”×” in Table 2 means no corresponding type because type-i 0 doesn’t take any metric information into consideration. Table 2. The improved similarity measure based on 2D C+ -string. spatial relations(i) metric info(j) the rations category either both relation x- or x- and on x- and y-axis y-axis are yequal axis 3 2 1 0 × 20 10 00 0 31 21 11 01 size 1 32 22 12 02 size and 2 distance In this paper, we also propose a typeij Similarity Retrieval algorithm based on the concept of “query-by-example” [3]. This algorithm is similar to the one’s proposed by Huang et al. [5]. We will introduce some notations that will be used in defining the similarity retrieval..

(5) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. A. Of q = {A, B, C, D} Cf q = {AdB, ApC, AdD, BdC, BdD, CdD} Rfuq = {A2 <3 B1 , A2 /1 C2 , A2 <2 D4 , C 2 < 2 B1 , D 4 % 1 B1 , C 2 < 1 D 4 } Rfv q = {B1 <1 A2 , A2 /1 C2 , A2 /1 D2 , B1 <2 C2 , B1 <2 D2 , C2 = D2 }. D. C. (2,4) (1,3). (5,4). B (6,1). C. D. (1,4). (5,4). Of 1 = {C, D} Cf 1 = {CdD} Of q 6⊂ Of 1. f1. fq. E. C A. D. (8,4). (30,4). (1,3). (14,3). B (33,1). f2 Rfu2 = {A8 <24 B4 , A8 /7 C8 , A8 <21 D16 , A8 <17 B4 , D16 %3 B4 , E18 <1 B4 , C8 <14 D16 , C8 /2 E18 , E18 /16 D16 } Rfv 2 = {B1 <1 A2 , A2 /1 C2 , A2 /1 D2 , A2 [E4 , B1 <2 C2 , B1 <2 D2 , B1 <1 E4 , C2 = D2 , E4 %1 C2 , E4 %1 D2 }. E D. C. A (5,4). (17,4). (1,3). B. (6,2) (21,1). Rfu3 = {A8 <12 B4 , A8 /4 C8 , A8 <8 D16 , A8 %5 E2 , C8 <8 B4 , D16 %4 B4 , E2 <13 B4 , C8 <4 D16 , C8 /1 E2 , E2 <9 D16 } Rfv 3 = {B1 <1 A2 , A2 /1 C2 , A2 /1 D2 , E6 %1 A2 , B1 <2 C2 , B1 <2 D2 , B1 |E6 , C2 = D2 , E6 %2 C2 , E6 %3 D6 }. f3. E (6,10). C. A (3,7). D (9,7). (1,5). Rfu4 = {A4 <6 B2 , A4 /2 C4 , A4 <4 D8 , A4 <1 E4 , C4 <4 B2 , D8 %2 B2 , E4 <1 B2 , C4 <2 D8 , C4 /3 E4 , E4 <3 D8 } Rfv 4 = {B2 <2 A4 , A4 /2 C4 , A4 /2 D4 , A4 <1 E4 , B2 <4 C4 , B2 <4 D4 , B2 <7 E6 , C4 = D4 , C4 /3 E4 , D4 /3 E4 }. B (10,1). f4. Figure 4. An example of type-ij Similarity Retrival based on improved similarity measure. |, %, [, ], /, =}.. • fq , a query picture in pictorial query. • fd , pictures in image database. • Of q , the set of all objects in a query picture fq . • Of d , the set of all objects in a database picture fd . • cAB , category relation for (AB ) in a picture, cAB ∈{DISJOINT, JOINT, PART OVLP, CONTAIN, BELONG}, they can be abbreviated as d, j, p, c, b [7]. • Cf q , the set of all category relations for each pair objects in fq . • Cf d , the set of all category relations for each pair objects in fd . u v • rAB and rAB , the spatial relations based on 2D + C -string for (AB ) on x- and y-axis, rAB ∈{<,. 551. • Rfuq and Rfv q , the set of all spatial relations for each pair objects on x- and y-axis respectively in fq . • Rfud and Rfv d , the set of all spatial relations for each pair objects on x- and y-axis respectively in fd . Then we review a notation that a pair objects (AB ) is regarded as the type-ij similar pair. Def inition4.1 A picture fd in image database is type-ij similar to a query picture fq if for every two objects A, B ∈fq , there exist a type-ij similar pair (AB )∈fd . Algorithm:type-ij Similarity Retrieval Input:A query picture fq and a sequence of 2D C+ string representing picture in image database.

(6) Int. Computer Symposium, Dec. 15-17, 2004, Taipei, Taiwan.. (6,12) (10,10). G. (8,9). (4,8). P. R. B. (7,6). (3,6). (9,5) (5,4). G. R (4,3). (6,3). B. (5,6) (7,6) (9,6). (11,6). W. (11,3). (8,3). W (2,0). fq. (2,0). f1 Figure 5. An example of flexible query by using parameter k.. Output:type-ij Similar List (pictures in image database similar to fq ) 1. Generate fq [Cf q , Rfuq ,Rfv q ]. /* fq [Cf q , Rfuq ,Rfv q ] denotes Cf q , Rfuq ,Rfv q for fq */ 2. /* search pictures in image database */ For each picture fd in image database if Of q ⊆Of d then (a) Derive fd [Cf d , Rfud ,Rfv d ]. /* fd [Cf d , Rfud ,Rfv d ] denotes Cf d , Rfud ,Rfv d for fd */ (b) Apply type-ij similarity measure to fq [Cf q , Rfuq ,Rfv q ] and fd [Cf d , Rfud ,Rfv d ]. if fd is type-ij similar to fq add fd to type-ij Similar List 3. Return type-ij Similar List To illustrate the power of the improved method, let’s look at an example shown in Figure 4. Suppose that we have a fq and four fd . By applying the algorithm, type-00 Similar List of fq is {f1 , f2 , f3 , f4 }, type-21 Similar List of fq is {f2 , f3 , f4 }, and type-22 Similar List of fq is {f3 , f4 }. Obviously, we can look at f3 is significantly different form fq . Thus, we can further eliminate f3 by applying type32 and precisely discriminate f4 is more similar to fq . Beside the power of retrieving pictures more precise, we will illustrate the parameter k can provide flexible query. An example is shown in Figure 5. We recognize the fq as a cake with three candles of different color (R, G, B) and the base is white (W ). Then we want to retrieve fd containing candles and base that is twice as taller as fq . Therefore, we apply type-21 to fq and fd , and we assign kus =1, kvs =2. Take pair objects (RG) for example, (SRu /SRu0 =SGu /SGu0 =1/kus ⇒1/SRu0 = 1/SGu0 , drives SRu0 = 1 and SGu0 = 1). The size of object R on y-axis is 6 and object G is 4 because (SRv /SRv0 =SGv /SGv0 =1/kvs ⇒3/SRv0 = 2/SGv0 , drives SRv0 = 6 and SG0 = 4). The calculation also conducts other pairs of objects. Finally, we retrieve f1 from database.. 5. Conclusions In this paper, we propose two new types of similarity measure based on 2D C+ -string to retrieve images more precise than the previous method. The. 552. improved method modifies the equations of original type-i1 and type-i2 simialrity measure and adds the parameter k to take down the ratios about relative sizes and distances for objects on x- and y-axis. The problem of distortion between two symbolic pictures can be aware if the ratios on x- and y-axis are quite different. Additional, the parameter k let user can assign the specific value about ratios for certain flexibility of pictorial query.. References [1] S. K. Chang, E. Jungert, and Y. Li. “Representation and Retrieval of Symbolic Pictures Using Generalized 2D strings”. Technical report, University of Pittsburg, 1988. [2] S. K. Chang, Q. Y . Shi, and C. W. Yan. “Iconic Indexing by 2-D Strings”. IEEE Trans. Pattern Anal. Mach. Intell., PAMI-9, pp. 413–428, 1987. [3] S. K. Chang, C. W. Yan, D. C. Dimitroff, and T. Arndt. “An Intelligent Image Database System”. IEEE Trans. Software Engng., vol. 14, no. 5, pp. 681–688, 1988. [4] P. W. Huang and Y. R. Jean. “Using 2D C+ strings as Spatial Knowledge Representation for Image Database Systems”. Pattern Recognition, vol. 27, no. 9, pp. 1249–1257, 1994. [5] P. W. Huang and Y. R. Jean. “Spatial Reasoning and Similarity Retrieval for Image Database Systems Based on RS-strings”. Pattern Recognition, vol. 29, no. 12, pp. 2103–2114, 1996. [6] S. Y. Lee and F. J. Hsu. “2D C-string: A New Spatial Knowledge Representation for Image Database Systems”. Pattern Recognition, vol. 23, no. 10, pp. 1077–1087, 1990. [7] S. Y. Lee and F. J. Hsu. “Spatial Reasoning and Similarity Retrieval of Images Using 2D C–string Knowledge Representation”. Pattern Recognition, vol. 25, no. 3, pp. 305–318,1992..

(7)