Multidimensional interval ®lter: A new indexing method for
subpicture query of image retrieval
Man-Kwan Shan
*, Suh-Yin Lee
Institute of Computer Science and Information Engineering, National Chiao Tung University, 1001 Ta Hsueh Road, HsinChu, Taiwan, ROC
Received 23 June 1997; received in revised form 26 June 1998
Abstract
In this paper a new indexing method, called multidimensional interval ®lter, is proposed to speed up processing of subpicture query. The basic idea is the transformation of spatial information of each image into a multidimensional rectangle. Processing of subpicture query becomes that of rectangle containment query which can be further speeded up by some well-developed spatial access methods such as R-Trees. Ó 1998 Elsevier Science B.V. All rights reserved.
Keywords: 2D string; Content-base image retrieval; Image databases; Spatial access method
1. Introduction
Content-based image retrieval is one of the important issues in the design of multimedia information system, digital library and visual information system. Approaches of recent works on content-based image retrieval include retrieval by color, by shape, by texture, by sketch and by spatial constraints (Gudivada and Raghavan, 1995). Image retrieval by spatial constraints retrieves images based on the spatial relationships among the objects in images. This paper only focuses on retrieval by spatial constraints.
Approaches of image retrieval by spatial constraints were originated from the 2D String approach (Chang et al., 1987). In this approach, objects and their spatial relationships in an image are represented as a spatial data structure, 2D String. Processing of subpicture query is achieved by 2D subsequence matching. However, usually there are a great number of images in an image database. It is essential to develop an indexing method to avoid exhaustive 2D subsequence matching.
In this paper, a new indexing method for subpicture query is proposed. The basic idea of this indexing method is to prune o a large amount of unquali®ed images by a fast ®lter mechanism. The spatial in-formation of each image is transformed to a multidimensional rectangle. Those images whose corre-sponding rectangles do not contain the query rectangle are ®ltered out. The beauty of the proposed ®lter mechanism lies in that rectangle containment query can be speeded up by some well-developed spatial indexing structures, such as R-Trees, TV-Trees (Guttman, 1984; Lin et al., 1994).
*Corresponding author. Fax: +886-35-724176; e-mail: [email protected].
0167-8655/98/$ ± see front matter Ó 1998 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 6 5 5 ( 9 8 ) 0 0 1 0 3 - 2
The remainder of this paper is organized as follows. In Section 2, a brief review of 2D String approaches is given. Section 3 describes the proposed indexing method. In Section 4, experimental results are pre-sented. The conclusions are presented in Section 5.
2. Review of 2D String approaches
The 2D String describes objects and their spatial relationships according to projection of the image along x- and y-axis direction, respectively. The ®rst part of the 2D String describes the left±right relationship while the second part of that describes the bottom±top relationship. A permutation function may be added to the 2D String to prevent ambiguity when there are multiple identical objects in the image. For example, the symbolic picture I0 in Fig. 1 can be represented as the 2D String (A < B D < C, B < A < D < C,
2134).
Because a symbolic picture is represented as a 2D String, a picture query can also be speci®ed as a 2D String. The problem of subpicture query then becomes the problem of 2D subsequence matching. Chang et al. (1987) have de®ned type-0, type-1 and type-2 2D subsequence as follows.
De®nition 1. A String U0is a type-i 1D subsequence of string U, if (1) U0is contained in U and (2) if a 1w1b1
is a substring of U0, a
2 and b2 are symbols of U, a1matches a2in U and b1matches b2in U, then
type-0 r b2 ÿ r a2 P r b1 ÿ r a1 or r b1 ÿ r a1 0;
type-1 r b2 ÿ r a2 P r b1 ÿ r a1 > 0 or r b2 ÿ r a2 r b1 ÿ r a1 0;
type-2 r b2 ÿ r a2 r b1 ÿ r a1;
where r(x), the rank of symbol x, is de®ned as one plus the number of ``<'' preceding this symbol x. De®nition 2. The 2D String (U0, V0) is a type-i 2D subsequence of the 2D String (U, V) if U0is a type-i 1D
subsequence of U and V0 is a type-i 1D subsequence of V.
For example, in Fig. 1, I1, I2and I3are all type-0 subpictures of I0, I1and I2are type-1 subpictures of I0.
Only I1 is a type-2 subpicture of I0. The problem of 2D subsequence matching has been proven to be
NP-complete (Tucci et al., 1991).
Since the invention of 2D String, some variants have been proposed to represent more complex spatial relationships among nonzero sized objects of symbolic pictures. There are generalized 2D G-String (Chang et al., 1988), 2D C-String (Lee and Hsu, 1991), 2D B-String (Lee et al., 1992), 2D C-String
(Huang and Jean, 1994), 2D H-String (Chang and Li, 1988), Adaptive 2D H-String (Chang and Lin, 1996) and 2D N-String (Chou et al., 1997), et al. The basic concept of our proposed indexing method can be applied to any of these variants. Therefore, for the sake of clarity, we only consider the original 2D String representation.
3. The proposed indexing method
The basic idea of the proposed indexing method is a fast ®lter mechanism. The spatial information of each image is transformed to a multidimensional rectangle. The spatial information of the query image is also transformed to a multidimensional rectangle. Those images whose corresponding rectangles do not contain the query rectangle are ®ltered out. Only those images whose rectangles contain the query rectangle are processed further by 2D subsequence matching. Rectangle containment query can be speeded up by some well-developed spatial indexing structures such as R-Trees.
Of course, the ®lter mechanism may incur false drops. False drops denote those images, which are quali®ed by the rectangle containment matching, but actually are not quali®ed by 2D subsequence matching. False drops cause unnecessary 2D subsequence matching and should be minimized as small as possible.
There are two steps for the transformation of spatial information into a rectangle. The ®rst step transforms the spatial information of each image to a set of spatial strings. The second step transforms each set of spatial strings into a rectangle. We ®rst introduce the de®nition of the spatial string.
De®nition 3. A type-i 1D spatial code VABis a code describing the spatial relationship between symbols A and
B in the 1D String, whereas
type 0 VAB \0" if r A r B; VAB \1" and VAB \0" if r A < r B; VAB \2" and VAB \0" if r A > r B; type 1 VAB \0" if r A r B; VAB \1" if r A < r B; VAB \2" if r A > r B;
type 2 VAB \0" str r A±r B if r A r B;
VAB \1" str r A±r B if r A < r B;
VAB \2" str r A±r B if r A > r B;
where r(X) is the rank of symbol X, symbol ``+'' denotes the string concatenating operator, str(Y) is a transformation function which returns the string form of integer Y.
De®nition 4. A type-i 2D spatial string TAB of objects A and B is a string by concatenating symbol A, B
and type-i spatial character Vx
AB and VABy, where the alphanumeric order of the object A is smaller than or
equal to that of the object B, Vx
AB; VABy is the spatial code along the x-dimension and the y-dimension,
respectively.
De®nition 5. A type-i spatial string collection P I for an image I is de®ned as the collection fTABj 8A; B 2 Ig.
Note that it is permitted that there may exist duplicate copies of the spatial string in the spatial string collection.
Example 1. Given an image represented as 2D String (A < B C, B < A < C), the type-0, type-1, type-2 spatial string collections are (type-0): {`AB02', `AB01', `AB12', `AC01', `AC10', `AC11', `BC01'}, (type-1): {`AB12', `AC11', `BC01'}, (type-2): {`AB1121', `AC1111', `BC0012'}.
The methodologies for the fast indexing method of the three types of subpicture query are all the same. Therefore, for the clarity of explanation, we only deal with the type-1 subpicture query in the following. Other types of subpicture query can be processed in the same way as that of type-1 subpicture query. In the following description, the term ``subpicture query'' denotes the type-1 subpicture query.
It is obvious that if a query image Q is a subpicture of an image I in the image database, then the spatial string collection of Q, P(Q), is a subset of P(I). The reverse is not always true. If P(Q) is a subset of P(I), it is possible that the query image Q is not a subpicture of database image I. This occurs when there are multiple same objects in the image. However, since the proposed method is a ®lter mechanism which prunes most, not all, of the unquali®ed images, we can apply subset testing for the indexing of subpicture query.
Having transformed the process of subpicture query into that of subset matching, the next step tries to avoid exhaustive subset testing of all the spatial string collection. Some strategies may be employed. The ®rst strategy, interval ®lter, assigns each spatial string in the image database a unique number. Each spatial string collection P(I) is represented as an interval R(I) which denotes the range of the assigned numbers of spatial strings contained in P(I). Given a query image Q, if R(I) does not contain R(Q), then P(Q) is not a subset of P(I) and I must be an unquali®ed image.
The assigned numbers of the spatial strings can be determined arbitrarily, as long as the numbers are assigned uniquely.
Example 2. Assume that in the image database, there are eight images shown in Fig. 2. The 2D Strings corresponding to these pictures are I0: C < A < B; B < C < A; 312, I1: C < A < B; A < C < B; 213, I2:
A B < C; B < C < A; 231, I3: C < B < A < B; B < B < C < A; 4213, I4: C < A < B < C; A < C <
C < B; 2413, I5: A B < C C; B C < C < A; 2341, I6: C < B < A; B < C < A; 213, I7: C < B <
A; A < C < B; 312. The spatial string collection corresponding to each image is shown in the second column of Table 1. Assume that each spatial string in the image database is assigned a unique number shown in the third column of Table 2, the third column of Table 1 lists the interval for each image.
It is not necessary to assign numbers to those spatial strings which do not exist in the example image database. These absent spatial strings are not shown in Table 2. Given the query image Q represented as 2D String (C < B < A, B < C < A), then P(Q) {`AB22', `AC22', `BC21'} and R Q (5±13). I2 is ®ltered
out by interval ®lter. Actually, I3and I6 are quali®ed images. I0, I1, I4, I5 and I7are false drops.
The ®ltering eect of interval ®lter may be improved by the second strategy, multidimensional interval ®lter. Instead of assigning a unique number, we can assign each spatial string a unique multidimensional vector in the Euclidean vector space. Therefore, each spatial string collection P(I), the spatial information of image I, is represented as a multidimensional interval R(I). In terms of geometry, each spatial string is represented as a multidimensional point, each spatial string collection is represented as a multidimensional rectangle. With multidimensional interval ®lter, the false drop probability is lowered down.
Table 1
Spatial string collections and intervals of images for the example image database Image Spatial string collection Interval
®lter 2-dimensionalinterval ®lter 2-dimensional interval ®lterwith clustering 2-dimensional interval®lter with clustering after ordering I0 AC22, BC21, AB12 (313) (313, 914) ()1.01 ´ 10ÿ15)8.72 ´ 10ÿ16, )8.60 ´ 10ÿ17)7.45 ´ 10ÿ17) (16, 15) I1 AC21, BC22, AB11 (214) (214, 515) (3.44 ´ 10ÿ13.61 ´ 10ÿ1, 1.25 ´ 10ÿ172.25 ´ 10ÿ17) (1416, 810) I2 AB02, AC12, BC11 (111) (111, 16) (1.25 ´ 10ÿ161.43 ´ 10ÿ16, 3.91 ´ 10ÿ14.31 ´ 10ÿ1) (1012, 1520)
I3 AB12, AB22, AC22, BC21,
BC21, BB12, BB21 (316) (316, 419) ()1.01 ´ 10
ÿ15)8.72 ´ 10ÿ16,
)8.60 ´ 10ÿ17)7.45 ´ 10ÿ17) (16, 16)
I4 AB11, AC11, AC21, BC22,
BC12, CC12, CC21 (220) (220, 220) (3.44 ´ 10
ÿ13.95 ´ 10ÿ1,
1.25 ´ 10ÿ173.14 ´ 10ÿ17) (1420, 814)
I5 AB02, AC12, AC12, BC10,
BC11, CC01, CC02 (118) (118, 118) (7.69 ´ 10 ÿ171.43 ´ 10ÿ16, 3.91 ´ 10ÿ14.31 ´ 10ÿ1) (712, 1520) I6 AB22, AC22, BC21 (513) (513, 1117) ()1.01 ´ 10ÿ15)8.72 ´ 10ÿ16, )8.60 ´ 10ÿ17)7.45 ´ 10ÿ17) (16, 16) I7 AB21, AC21, BC22 (414) (414, 1015) (8.54 ´ 10ÿ23.44 ´ 10ÿ1, )1.01 ´ 10ÿ171.25 ´ 10ÿ17) (1316, 79) Table 2
Vector assignment of spatial strings in the example image database Spatial
string Occurrence vectorI Assigned vector
0 I1 I2 I3 I4 I5 I6 I7 Interval
®lter Two-dimensionalinterval ®lter Two-dimensional interval ®lterwith clustering Two dimensionalinterval ®lter with clustering after ordering AB02 0 0 1 0 0 1 0 0 1 (1, 1) (1.43 ´ 10ÿ16, 3.91 ´ 10ÿ1) (11, 15) AB11 0 1 0 0 1 0 0 0 2 (2, 5) (3.61 ´ 10ÿ1, 2.25 ´ 10ÿ17) (14, 10) AB12 1 0 0 1 0 0 0 0 3 (3, 9) ()8.76 ´ 10ÿ16, )7.45 ´ 10ÿ17) (4, 5) AB21 0 0 0 0 0 0 0 1 4 (4, 13) (8.45 ´ 10ÿ2, )1.01 ´ 10ÿ17) (13, 7) AB22 0 0 0 1 0 0 1 0 5 (5, 17) ()8.76 ´ 10ÿ16, )7.45 ´ 10ÿ17) (5, 6) AC11 0 0 0 0 1 0 0 0 6 (6, 2) (3.95 ´ 10ÿ1, 3.14 ´ 10ÿ17) (17, 11) AC12 0 0 1 0 0 2 0 0 7 (7, 6) (1.25 ´ 10ÿ16, 4.31 ´ 10ÿ1) (10, 20) AC21 0 1 0 0 1 0 0 1 8 (8, 10) (3.44 ´ 10ÿ1, 1.25 ´ 10ÿ17) (15, 8) AC22 1 0 0 1 0 0 1 0 9 (9, 14) ()8.72 ´ 10ÿ16, )7.59 ´ 10ÿ17) (6, 4) BC10 0 0 0 0 0 1 0 0 10 (10, 18) (7.69 ´ 10ÿ17, 4.12 ´ 10ÿ1) (7, 17) BC11 0 0 1 0 0 1 0 0 11 (11, 3) (1.43 ´ 10ÿ16, 3.91 ´ 10ÿ1) (12, 16) BC12 0 0 0 0 1 0 0 0 12 (12, 7) (3.95 ´ 10ÿ1, 3.14 ´ 10ÿ17) (18, 12) BC21 1 0 0 2 0 0 1 0 13 (13, 11) ()1.01 ´ 10ÿ15, )8.60 ´ 10ÿ17) (1, 1) BC22 0 1 0 0 1 0 0 1 14 (14, 15) (3.44 ´ 10ÿ1, 1.25 ´ 10ÿ17) (16, 9) BB12 0 0 0 1 0 0 0 0 15 (15, 19) ()9.68 ´ 10ÿ16, )7.93 ´ 10ÿ17) (2, 2) BB21 0 0 0 1 0 0 0 0 16 (16, 4) ()9.68 ´ 10ÿ16, )7.93 ´ 10ÿ17) (3, 3) CC01 0 0 0 0 0 1 0 0 17 (17, 8) (7.69 ´ 10ÿ17, 4.12 ´ 10ÿ1) (8, 18) CC02 0 0 0 0 0 1 0 0 18 (18, 12) (7.69 ´ 10ÿ17, 4.12 ´ 10ÿ1) (9, 19) CC12 0 0 0 0 1 0 0 0 19 (19, 16) (3.95 ´ 10ÿ1, 3.14 ´ 10ÿ17) (19, 13) CC21 0 0 0 0 1 0 0 0 20 (20, 20) (3.95 ´ 10ÿ1, 3.14 ´ 10ÿ17) (20, 14)
Example 3. For the same example image database in Example 2, each spatial string is assigned a unique 2-dimensional vector as shown in the fourth column of Table 2. Therefore, the interval corresponding to each image is shown in the fourth column of Table 1. Fig. 3 plots the corresponding two-dimensional rectangles. The numbers in the four corners of each rectangle denote the corresponding image numbers. The query rectangle is the shadowed block. Given the query image Q represented as 2D String C < B < A; B < C < A, then R Q (513, 1117), which is enclosed by R I3, R I4, R I5, R I6. I0, I1, I2and
I7are ®ltered out. I3and I6are actually quali®ed, I4and I5are false drops. For the same query image, two
false drops are generated by multidimensional interval ®lter while four false drops are generated by interval ®lter.
In Example 3, it can be found that most of the rectangles overlap each other. Certainly, two rectangles overlap if the corresponding images have some common spatial strings. For example, in Example 3, R(I1)
and R(I7) overlap because I1 and I7 have the common spatial strings `BC22' and `AC21'. However, it is
possible that two overlapping rectangles have no common spatial strings. For example, R(I2) and R(I4)
overlap but there is no common spatial string between I2 and I4. This phenomenon comes from the
as-signment of multidimensional vector for the spatial strings. If the query rectangle falls into these over-lapping region, it is expected to incur false drops.
The third strategy, multidimensional interval ®lter with clustering improves the ®ltering eect by ecient assignment of multidimensional vector. The goal of ecient assignment tries to minimize the areas of multidimensional rectangles of the images in the database, because larger rectangles are more likely to cover the query rectangle and produce more false drops. This goal can be achieved heuristically by clustering the coordinates of spatial strings in multidimensional space. The criterion of clustering depends on the cor-relation among the spatial strings in the image database. Those spatial strings which tend to appear
concurrently are clustered. On the contrary, those spatial strings which seldom appear concurrently are dispersed. This will reduce the false drop probability. We ®rst give some de®nitions concerning the cor-relation between spatial strings.
De®nition 6. The occurrence vector of a spatial string TABis de®ned as a vector of images. The ith element of
the vector represents the occurrences of TAB in the spatial string collection of image i.
De®nition 7. Consider two vectors vi and vj which are the occurrence vectors of two spatial strings, the
correlation cij between these two spatial strings is measured as vi vj=jvij jvjj, where `' is the inner
product of two vectors and jvij stands for the Euclidean norm of the vector.
Therefore, the problem to be solved is stated as follows. Given that in the image database, totally there are b objects, m images and n spatial strings. The goal is to ®nd the vectors of the n spatial strings in multidimensional space such that the Euclidean distances between vectors of spatial strings satisfy the following distance criterion. For any three spatial strings with occurrence vectors vi, vj, vk, if cij> cik, then
dij< dik, where cij, dij is the correlation, distance between spatial strings corresponding to vi, vj,
respectively.
To solve this problem for multidimensional case, the multidimensional vector of the spatial string with occurrence vector vi is assigned the normalized occurrence vector, vi=jvij. That is, the assigned
multidi-mensional vectors of the spatial strings are unit vectors. Given three unit vectors ur, us, ut, if ur us> ur ut,
we have crs> crt. Since, ur, us, ut are normalized, so crs6 1 and crt6 1. Therefore, 1ÿcrs< 1ÿcrt which
implies drs< drt. It is obvious that the assignment satis®es the above criterion.
The assignment stated above satis®es the distance criterion. However, usually the number of images, m, is large. It is impractical to assign each spatial string an m-dimensional vector. The reason comes not only from the storage space consideration but also from the eciency consideration of existing spatial indexing structure. Most of the developed spatial indexing structures, such as R-Trees, work well when the number of dimensions is less than 20 (Faloutsos, 1994). It is necessary to ®nd the solution for a speci®c number of dimension, say k. Of course, it is possible that the solution doesn't exist. In this situation, the goal is not so strict but to retain the distance criterion as well as possible.
We use the singular value decomposition (SVD) (Leon, 1990) to achieve this goal. Before giving an introduction to SVD, we ®rst de®ne a spatial information matrix.
De®nition 8. Given an image database with m images and b objects, the spatial information matrix X0
for this image database is de®ned as a matrix of n spatial strings by m images. Each row represents the m-dimensional normalized occurrence vector of the corresponding spatial string. That is, for each entry of X0, xij equals yij=
Rm
j1yij2
q
, where yij is the number of occurences of ith spatial string in the jth
image.
The spatial information matrix represents the spatial information in an image database. The dot product between two rows represents the correlation between corresponding spatial strings. The spatial information matrix is taken as the input to the process of SVD. We then give a brief description of SVD.
Given an n m matrix X0with rank f, SVD involves factoring X0into a product U0S0VT0, where U0is an
n f orthonormal matrix, V0 is an f m orthonormal matrix, and S0 is an f f matrix whose o
di-agonal entries are all 0's and whose didi-agonal elements r1; r2; . . . rf satisfy r1P r2P P rfP 0.
The orthonormal columns of U0and V0 are called left and right singular vectors, respectively. The
values which is useful for optimal approximation. If the ®rst k largest singular values and corresponding left and right orthonormal vectors are kept, then the original matrix X0is approximated by X USVT, where U
is an n k orthonormal matrix, V is a k m orthonormal matrix, and S is a k k diagonal matrix (Leon, 1990).
The dot product between two rows of X0re¯ects the correlation between two spatial strings. Therefore,
the symmetric matrix X0XT0 represents the correlation between each pair of spatial strings, where
X0X0T XXT USVT USVTT USVT VSTUT:
Because V is an orthonormal matrix, S is a diagonal matrix XXT USVT VSTUTT USSTUT US UST:
Thus, we can assign the row of US as the k-dimensional vector for the spatial string. Besides, instead of taking US, we can take U as the k-dimensional vectors. This is because S is diagonal; the positions of the points are the same except that each of the axes has been stretched or shrunk in proportion to the cor-responding diagonal elements of S.
Example 4. The ®fth column of Table 2 shows the two-dimensional vector of each spatial strings by using the SVD for the example image database in Example 2. The interval corresponding to each image is shown in the ®fth column of Table 1. Given the query image Q represented as 2D String C < B < A; B < C < A, then R Q ÿ1:01 10ÿ15 ÿ8:72 10ÿ16; ÿ8:60 10ÿ17 ÿ7:45 10ÿ17, which is enclosed by R(I
0),
R(I3) and R(I6). I1, I2, I4, I5and I7are ®ltered out. I3and I6are actually quali®ed while I0is a false drop. For
the same query image, one false drop is incurred while two false drops are generated by multidimensional interval ®lter.
Observe that in Example 4 the elements of assigned vectors are all real numbers. It takes the storage space and the query processing time. Therefore, for each dimension of the assigned vectors, we impose a partial ordering on the corresponding coordinate of the assigned vectors and take the order numbers as the coordinate of assigned vectors. Two spatial strings with the same singular value are assigned with distinct order numbers.
Example 5. The sixth column of Table 2 lists the two-dimensional vectors of the spatial strings after ordering. The interval corresponding to each image is shown in the sixth column of Table 1. Given the query image Q represented as 2D String C < B < A; B < C < A, then R(Q) (16, 16), which is enclosed by R(I3) and R(I6). I0, I1, I2, I4, I5and I7 are ®ltered out. I3and I6are actually quali®ed. No false
drops are incurred. Fig. 4 plots the two-dimensional rectangles of the eight images in Example 5. Comparing Fig. 4 with Fig. 3, it is observed that less overlapping rectangles exist in the former. In Fig. 4, two rectangles overlap only if the corresponding images have some common spatial strings. It is expected that the false drop probability is much lower.
4. Performance analysis
We measure the performance of the proposed indexing method by the false drop probability. The false drop probability Fddenotes the probability that an image rectangle quali®es, given that the image does not
actually qualify. High false drop probability incurs unnecessary 2D subsequence matching. Therefore false drop probability of an eective ®lter should be as small as possible.
We implement our multidimensional ®lter with clustering algorithm and carry out experiments in order to demonstrate the eectiveness of our proposed indexing method. The design parameters which aect the false drop probability are listed in Table 3. The software package SVDPACKC developed in University of Tennessee (Berry et al., 1993) was used for computing the SVD of large sparse matrices using ANSI C.
Given m, b and h, database images are generated randomly. In other words, h objects are randomly selected from the b objects for each image and randomly placed in a symbolic picture of 10 by 10 cells. The query images are generated from the database images. For a set of m database images, m query images are generated. Each query image is generated by randomly selecting g objects from a database image. This way of generating query images guarantees that at least one image is quali®ed for the query image.
Fig. 4. The two-dimensional rectangles of the eight images in Example 5.
Table 3
Design parameters of simulation
Symbol De®nition
m Total number of images in the image database
b Total number of objects in the image database
h The number of objects in each image
k The number of dimensions
Figs. 5±8 demonstrate the average false drop probability as the function of the design parameters. Note that in these ®gures, the value of y-axis denotes the base 10 logarithm of false drop probability. Besides, there exists the case with no false drops in our simulation, and hence the false drop probability is zero. It is meaningless for logarithm of zero, therefore we approximate the curve by regression analysis.
Fig. 5 shows the average false drop probability as a function of the number of objects in the query image for dierent number of objects in the image database (b equals 26, 52 and 78). From the analysis, it can be seen that the average false drop probability decreases with increasing number of query objects. Increasing the number of query objects would produce larger size of query rectangle and reduce the false drop probability. Besides, given a speci®c number of query objects, the false drop probability decreases with increasing number of objects in the image database.
Fig. 6 gives the average false drop probability as a function of the number of objects in the database image for total number of objects in the image database b being equal to 26, 52, 78, respectively. It is obvious that the average false drop probability increases with increasing number of objects in the database image. Increasing number of objects in the database image would produce larger database image rectangle, and hence increase the false drop probability. Fig. 7 demonstrates the eect of the number of dimensions for the number of query objects g being equal to 2, 3, 4, 5, 6, 7, 8, 9, 10, respectively. Of course, the average false drop probability decreases with increasing number of dimensions.
Fig. 5. Average false drop probability versus the number of objects in the query image for total number of objects in the image da-tabase.
Fig. 6. Average false drop probability versus the number of objects in the database image for total number of objects in the image database.
Fig. 8 compares the average false drop probability between the image database with 1000 images and that with 2000 images. The number of dimension is set to 20. Obviously, the false drop probability of the case with 2000 images is higher than that with 1000 images. The relationship among the number of images, the number of dimensions and average false drop probability are critical to the performance of our system. In practice, the number of dimensions must be chosen to achieve a good retrieval performance.
5. Conclusions
In this paper, we propose a new indexing method for subpicture query of iconic image databases. The basic mechanism of this indexing method is a multidimensional ®lter which is able to prune out most of the unquali®ed images. The spatial information of each image is transformed into a multidimensional rect-angle. Those images whose corresponding rectangles do not contain the query rectangle are ®ltered out. We also present the method using SVD to improve the eectiveness of multidimensional ®lter.
Fig. 7. Average false drop probability versus the number of dimensions for the number of query objects.
The advantages of the proposed indexing method lies in the eectiveness and eciency of ®lter process. For the eciency, the proposed indexing method can adopt a spatial searching structure such as R-Trees to speed up rectangle containment test in a query process. The results of simulations have shown that the average false drop probability depends on the number of objects in the image database, the number of objects in each image, the number of objects in the query image, the number of dimensions and the number of images in the image database. In a typical case with eight objects in each image, 2000 images in the image database, three query objects in the query image, the average false drop probability for 20 dimensions is only 0.003, which is very eective.
Future research includes the derivation of the formula of false drop probability as a function of the design parameters and the performance analysis for the other variants of 2D string representation. The other variants capture more complex spatial relationships. It is worth to investigate the eect of more complex spatial relationships on false drop probability.
Acknowledgements
We thank Prof. P.W. Huang for his valuable suggestion and corrections. Appendix A
This appendix presents the detailed numerical results for the multidimensional ®lter with clustering using the example image database in Example 4. The spatial information matrix X0is presented below. Note that
the corresponding rows for those spatial strings that are absent in the image database are not shown in this matrix. It does not matter, because the assigned multidimensional vectors (left singular vectors) for these absent spatial strings, generated from the SVD of the spatial information matrix, are all zero vectors.
Recall that SVD involves factoring X0into a product U0S0VT0. These matrices U0, S0and V0are shown
in the next page. We can verify that X0 U0S0VT0 except for small rounding errors. To generate the
two-dimensional vectors of the spatial strings, only the ®rst two columns of the left singular matrix are kept. In other words, we approximate X0 by keeping only the ®rst two singular values and the corresponding
columns from the left singular and right singular matrices U and V.
AB02 AB11 AB12 AB21 AB22 AC11 AC12 AC21 AC22 BB12 BB21 BC10 BC11 BC12 BC21 BC22 CC01 CC02 CC12 CC21 1:43 10ÿ16 3:91 10ÿ1 ÿ 1:18 10ÿ17 ÿ 2:98 10ÿ17 5:18 10ÿ1 4:35 10ÿ15 ÿ1:43 10ÿ20 ÿ1:89 10ÿ15 3:61 10ÿ1 2:25 10ÿ17 ÿ 9:32 10ÿ16 ÿ4:93 10ÿ2 ÿ1:09 10ÿ16 6:58 10ÿ17 ÿ6:5 10ÿ1 9:88 10ÿ16 ÿ8:76 10ÿ16 ÿ7:45 10ÿ17 ÿ 3:9 10ÿ1 ÿ 1:03 10ÿ16 ÿ1:26 10ÿ15 1:62 10ÿ1 7:21 10ÿ22 ÿ7:07 10ÿ1 8:54 10ÿ2 ÿ1:01 10ÿ17 ÿ 3:99 10ÿ16 ÿ6:58 10ÿ1 ÿ1:52 10ÿ16 1:65 10ÿ17 5:81 10ÿ1 ÿ8:13 10ÿ15 ÿ8:76 10ÿ16 ÿ7:45 10ÿ17 ÿ 3:9 10ÿ1 ÿ 1:03 10ÿ16 ÿ1:26 10ÿ15 1:62 10ÿ1 7:21 10ÿ22 ÿ7:07 10ÿ1 3:95 10ÿ1 3:14 10ÿ17 ÿ 9:42 10ÿ16 2:30 10ÿ1 ÿ6:39 10ÿ17 7:16 10ÿ17 2:02 10ÿ1 ÿ1:65 10ÿ15 1:25 10ÿ16 4:31 10ÿ1 ÿ 4:51 10ÿ17 ÿ 7:43 10ÿ18 1:56 10ÿ1 1:36 10ÿ15 1:57 10ÿ20 ÿ4:53 10ÿ16 3:44 10ÿ1 1:25 10ÿ17 ÿ 9:91 10ÿ16 ÿ 4:2 10ÿ1 ÿ1:77 10ÿ16 6:32 10ÿ17 ÿ1:96 10ÿ1 ÿ3:89 10ÿ15 ÿ8:72 10ÿ16 ÿ7:59 10ÿ17 ÿ4:01 10ÿ1 ÿ 3:62 10ÿ17 ÿ4:59 10ÿ15 5:75 10ÿ1 7:41 10ÿ22 ÿ8:48 10ÿ15 ÿ9:68 10ÿ16 ÿ7:93 10ÿ17 ÿ4:08 10ÿ1 ÿ 2:29 10ÿ16 4:41 10ÿ15 ÿ5:4 10ÿ1 7:55 10ÿ22 3:51 10ÿ15 ÿ9:68 10ÿ16 ÿ7:93 10ÿ17 ÿ4:08 10ÿ1 ÿ 2:29 10ÿ16 4:41 10ÿ15 ÿ5:4 10ÿ1 7:55 10ÿ22 3:51 10ÿ15 7:69 10ÿ17 4:12 10ÿ1 ÿ 8:41 10ÿ17 2:56 10ÿ17 3:83 10ÿ1 ÿ3:1 10ÿ15 1:5 10ÿ20 1:67 10ÿ15 1:43 10ÿ16 3:91 10ÿ1 ÿ 1:18 10ÿ17 ÿ 2:98 10ÿ17 5:18 10ÿ1 4:35 10ÿ15 1:43 10ÿ20 ÿ1:89 10ÿ15 3:95 10ÿ1 3:14 10ÿ17 ÿ 9:42 10ÿ16 2:30 10ÿ1 ÿ6:39 10ÿ17 7:16 10ÿ17 2:02 10ÿ1 ÿ1:65 10ÿ15 ÿ1:01 10ÿ15 ÿ 8:6 10ÿ17 ÿ 4:5 10ÿ1 ÿ 1:19 10ÿ16 ÿ1:45 10ÿ15 1:86 10ÿ1 8:32 10ÿ22 ÿ4:63 10ÿ15 3:44 10ÿ1 1:25 10ÿ17 ÿ 9:91 10ÿ16 ÿ 4:2 10ÿ1 ÿ1:77 10ÿ16 6:32 10ÿ17 ÿ1:96 10ÿ1 ÿ3:89 10ÿ15 7:69 10ÿ17 4:12 10ÿ1 ÿ 8:41 10ÿ17 2:56 10ÿ17 3:83 10ÿ1 ÿ3:1 10ÿ15 1:5 10ÿ20 1:67 10ÿ15 7:69 10ÿ17 4:12 10ÿ1 ÿ 8:41 10ÿ17 2:56 10ÿ17 3:83 10ÿ1 ÿ3:1 10ÿ15 1:5 10ÿ20 1:67 10ÿ15 3:95 10ÿ1 3:14 10ÿ17 ÿ 9:42 10ÿ16 2:30 10ÿ1 ÿ6:39 10ÿ17 7:16 10ÿ17 2:02 10ÿ1 ÿ1:65 10ÿ15 3:95 10ÿ1 3:14 10ÿ17 ÿ 9:42 10ÿ16 2:30 10ÿ1 ÿ6:39 10ÿ17 7:16 10ÿ17 2:02 10ÿ1 ÿ1:65 10ÿ15 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 ; S0 2:38 0 0 0 0 0 0 0 0 2:30 0 0 0 0 0 0 0 0 2:19 0 0 0 0 0 0 0 0 1:32 0 0 0 0 0 0 0 0 0:85 0 0 0 0 0 0 0 0 0:82 0 0 0 0 0 0 0 0 0:78 0 0 0 0 0 0 0 0 0:71 2 6 6 6 6 6 6 6 6 6 6 6 6 6 4 3 7 7 7 7 7 7 7 7 7 7 7 7 7 5 ; VT 0 ÿ6:45 10ÿ16 2:74 10ÿ1 2:99 10ÿ16 ÿ2:3 10ÿ15 9:4 10ÿ1 1:83 10ÿ16 ÿ6:45 10ÿ16 2:03 10ÿ1 ÿ6:01 10ÿ17 9:61 10ÿ19 3:24 10ÿ1 ÿ1:82 10ÿ16 7:2 10ÿ17 9:46 10ÿ1 ÿ6:01 10ÿ17 ÿ2:32 10ÿ17 ÿ3:15 10ÿ1 ÿ8:25 10ÿ16 1:48 10ÿ16 ÿ8:96 10ÿ1 ÿ2:07 10ÿ15 ÿ1:85 10ÿ16 ÿ3:15 10ÿ1 ÿ8:75 10ÿ16 1:09 10ÿ16 ÿ3:94 10ÿ1 ÿ8:94 10ÿ17 ÿ3:01 10ÿ16 3:03 10ÿ1 3:37 10ÿ17 1:09 10ÿ16 ÿ8:68 10ÿ1 ÿ5:24 10ÿ15 ÿ7:66 10ÿ17 9:46 10ÿ1 3:74 10ÿ15 ÿ5:42 10ÿ17 ÿ3:24 10ÿ1 ÿ5:24 10ÿ15 ÿ1:29 10ÿ16 6:33 10ÿ1 1:77 10ÿ17 7:62 10ÿ15 ÿ4:45 10ÿ1 5:9 10ÿ17 ÿ2:55 10ÿ15 6:33 10ÿ1 1:36 10ÿ17 2:07 10ÿ22 ÿ8:77 10ÿ1 4:03 10ÿ21 5:9 10ÿ22 1:58 10ÿ1 1:17 10ÿ20 2:07 10ÿ22 4:54 10ÿ1 ÿ7:07 10ÿ1 2:16 10ÿ15 ÿ3:07 10ÿ15 2:48 10ÿ15 ÿ1:17 10ÿ15 1:18 10ÿ15 7:07 10ÿ1 ÿ5:75 10ÿ15 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 ;
U AB02 AB11 AB12 AB21 AB22 AC11 AC12 AC21 AC22 BB12 BB21 BC10 BC11 BC12 BC21 BC22 CC01 CC02 CC12 CC21 1:43 10ÿ16 3:91 10ÿ1 3:61 10ÿ1 2:25 10ÿ17 ÿ8:76 10ÿ16 ÿ7:75 10ÿ17 8:54 10ÿ2 ÿ1:01 10ÿ17 ÿ8:76 10ÿ16 ÿ7:75 10ÿ17 3:95 10ÿ1 3:14 10ÿ17 1:25 10ÿ16 4:31 10ÿ1 3:44 10ÿ1 1:25 10ÿ17 ÿ8:72 10ÿ16 ÿ7:59 10ÿ17 ÿ9:68 10ÿ16 ÿ7:93 10ÿ17 ÿ9:68 10ÿ16 ÿ7:93 10ÿ17 7:69 10ÿ17 4:12 10ÿ1 1:43 10ÿ16 3:91 10ÿ1 3:95 10ÿ1 3:14 10ÿ17 ÿ1:01 10ÿ15 ÿ8:6 10ÿ17 3:44 10ÿ1 1:25 10ÿ17 7:69 10ÿ17 4:12 10ÿ1 7:69 10ÿ17 4:12 10ÿ1 3:95 10ÿ1 3:14 10ÿ17 3:95 10ÿ1 3:14 10ÿ17 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 ; S 2:38 0 0 2:30 ; DT ÿ6:45 10ÿ16 2:74 10ÿ1 2:99 10ÿ16 ÿ2:30 10ÿ15 9:4 10ÿ1 1:83 10ÿ16 ÿ6:45 10ÿ16 2:03 10ÿ1 ÿ6:01 10ÿ17 9:61 10ÿ19 3:24 10ÿ1 ÿ1:82 10ÿ16 7:2 10ÿ17 9:46 10ÿ1 ÿ6:01 10ÿ17 ÿ2:32 10ÿ17 " # : References
Berry M., et al., 1993. SVDPACKC (Version 1.0) user's guide. Technical Report CS-93-194, Department of Computer Science, University of Tennessee.
Chang, C.C., Lin, D.C., 1996. A spatial data representation: an adaptive 2D-H string. Pattern Recognition Letters 17 (2), 175±185. Chang, S.K., Li, Y., 1988. Representation of multi-resolution symbolic and binary pictures using 2D-H string. In: Proceedings of IEEE
Workshop on Languages for Automata, Maryland, pp. 190±195.
Chang, S.K., Shi, Q.Y., Yang, C.W., 1987. Iconic indexing by 2-D strings. IEEE Transactions Pattern Analysis and Machine Intelligence PAMI-9 (3), 413±428.
Chang, S.K., Jungert, E., Li, Y., 1988. Representation and retrieval of symbolic pictures using generalized 2D strings. Technical Report, PA15260, University of Pittsburgh.
Chou, A.Y.H., Chang, C.C., Yang, W.P., 1997. Symbolic indexing by N-strings. International Journal of Pattern Recognition and Arti®cial Intelligence (submitted).
Gudivada, V.N., Raghavan, V.V., 1995. Content-based image retrieval systems. Guest editor's introduction. IEEE Computer 28 (9), 3±7.
Guttman, A., 1984. R-trees: A dynamic index structure for spatial searching. In: Proceedings of ACM-SIGMOD 1984, International Conference on Management of Data, pp. 47±57.
Huang, P.W., Jean, Y.R., 1994. Using 2D C-string as spatial knowledge representation for image database systems. Pattern
Recognition 27 (9), 1249±1257.
Lee, S.Y., Hsu, F.J., 1991. 2D C-string: A new spatial knowledge representation for image database systems. Pattern Recognition 12 (7), 425±435.
Lee, S.Y., Yang, M.C., Chen, J.W., 1992. 2D B-string spatial knowledge representation and picture retrieval for image database. In: Proceedings of Second International Computer Science Conference, Data and Knowledge Engineering: Theory and Applications. Leon, S.J., 1990. Linear Algebra with Applications, 3rd edition, Macmillan, New York.
Lin, K.I., Jagadish, H.V., Faloutsos, C., 1994. The TV-tree: An index structure for high-dimensional data. VLDB Journal 3 (4), 517± 542.
Tucci, M., Costagliola, G., Chang, S.K., 1991. A remark on NP-completeness of picture matching information. Information Processing Letters 39 (5), 241±243.
