Chapter 2 Previous Research
2.4 D ISTANCE F UNCTIONS FOR S IMILARITY M EASURE
2.4.5 The DPF Enhancement
Here, we select the features that have a difference smaller than θ belong to the similar feature set A . The Thresholding distance of image pair (X,Y) is then defined as θ
1/
2.4.5.2 Sampling Method
The main idea of Sampling-DPF is to try different m settings simultaneously and to guess that some of the sampling points can be near the optimal m. Suppose we sample N settings of m, denoted as m , 1,...,n n= N . At each sampling m , a ranking list, denoted as n
( ,Φ )
Rn X , is generated. Finally, the sampling method obtains N ranking lists and produces the final ranking, denoted as Rr( ,Φ X . )
Two methods are suggested, one is using the smallest rank as the final rank (SR). The other is using the average distance (or rank) as the final distance (AR). The AR distance is defined as
For the Sampling-Thresholding method, its AR distance is defined as
1/
2.4.5.3 Weighting Method
The Weighting method can be used with Sampling-DPF, Sampling-Thresholding, or other methods. It considers that different types of features can have different degrees of importance for measuring similarity. Let ωi denote the weight of the i th similar feature. In the Weighted-Sampling-DPF method, the distance of image pair (X,Y) is defined as
1/
and in the Weighted-Sampling-Thresholding method, it is defined as
1/
1
1 1
( , )
θ θ
ω
= Δ ∈
⎛ ⎞
⎜ ⎟
= Δ
⎜ ⎟
⎝ ⎠
∑ ∑
i n
n
r N
r
i i
n d A
D X Y d
N A ( 42 )
Chapter 3
The Proposed Method
Overview of our retrieval system is described in section 3.1. In section 3.2, we will introduce a new feature, granulometric histogram, which represents the size distribution of an image. Its similarity is described in section 3.3.1. We also introduce the smallest DPF to be the similarity of Morphological Primitive and describe it in section 3.3.2.
3.1 Overview
Fig-3.1 shows the architecture of our retrieval system. Within it, processing and similarity measure are the key components. There are many processing methods for the purpose of image retrieval. In this thesis, we employe morphological primitive extraction, color moment primitive extraction, morphological granulometric distribution, and morphological granulometric histogram. We plot them in Fig-3.2.
Images in image database Images in image database
Indexing
Fig. 3-1 Architecture of our retrieval system.
YIQ Color
Fig. 3-2 (a)-(e) Different processing methods.
3.2 Morphological Granulometry
3.2.1 Distribution of Morphological Granulometry
Before applying morphological granulometry, we transform the down-sampled image from RGB to YIQ color model. Assume Cx,y =
(
Yx,y,Ix,y,Qx,y)
be the pixel value at location( )
x,y of the down-sampled image, and B={
b1,b2,...,bL}
be a set of structuring elements with increasing sizes. According to section 2.3.5, we have chosen the pattern spectrum as the granulometry function.component of YIQ color space with structuring element bi,i=1,...,L. Therefore, we obtain L
M =3* openings for a single image.
Second, we compute the distribution of morphological granulometry as follows:
}
}
3.2.2 Primitives of Granulometric Distribution
As morphological primitive mentioned in 2.3.6, we first transform RGB color model into YIQ model, then we down-sample each image. Let f(x,y)=
(
Yxy,Ixy,Qxy)
denote the color value of the down-sampled image located at( )
x,y . We define the following Z-tuple vector as the morphological granulometry context of a pixel located at( )
x,y :[
xy xy Zxy]
) 1 (
*
3 L
Z = + . If Z =6 . Yx,y,Ix,y,Q are the pixel value of Y, I, Q color channels respectively; )Ob(Yx,y),Ob(Ix,y),Ob(Qx,y are the openings of Y, I, Q color channels respectively. Consequently, cx4,y,c5x,y,c6x,y represent the pattern spectrum.
3.2.3 Histogram of Morphological Granulometry
The granulometric histograms are obtained by the following four steps.
First, we down-sampled the target image to remove the noise and reduce the computation time. Suppose Cx,y =
(
Rx,y,Gx,y,Bx,y)
represents the pixel value at location( )
x,y of the down-sampled image.Second, letB=
{
b1,b2,...,bL}
be a set of structuring elements with increasing sizes. We do a series of morphological openings, denoted O (R),O (G),O (B)i i
i b b
b , to each component of
color space with structuring element bi,i =1,...,L. Therefore, we obtain M =3*L openings for a single image.
Third, we count the histogram Hj, j=1,...,M of each opening, says the granulometric histograms. H1,H2,...,HL are histograms of O R j L
bj( ), =1,..., , respectively,
L L
L H H
H +1, +2,..., 2* are histograms of O G j L
bj( ), =1,..., , respectively,
L L
L H H
H2* +1, 2* +2,..., 3* are histograms of O B j L
bj( ), =1,..., , respectively.
Fourth, we normalize the histogram. So Hj =Hj/num , 1,...,j= M where
255
0 i j i
num H
=
=
∑
represents the size of down-sampled image. After this step, a histogram count turns into a probability of that color bin. Obviously, it removes the size of an image.Morphological opening or closing could enhance, suppress or smooth some areas.
Opening operation of a particular size of structuring element has the most effect on regions of the input image that contain particles of that size.
In RGB color space, it is easy to treat similar colors as different colors and then similar objects become different objects. Opening of different image may be similar under RGB color space.
Take Fig.3-3 for example. The space between trunks of trees and space separated by legs of horses in Fig.3-3(a) are then becoming blocks of colors after opening. The chinks in the leaves, color changing in the trees, and the stripes of the bus are causing blockings after opening. We may see those blocks in Fig.3-4 and Fig.3-5. These phenomenon leads to false retrieval results.
(a) (b)
Fig. 3-3 Example of different image that has similar opening.
(a) R component (b) opening of (a) with b1 (c) opening of (a) with b2
(d) G component (e) opening of (d) with b1 (f) opening of (d) with b2
(g) B component (h) opening of (g) with b1 (i) opening of (h) with b2
Fig. 3-4 Openings of .Fig.3-3(a).
(a) R component (b) opening of (a) with b1 (c) opening of (a) with b2
(d) G component (e) opening of (d) with b1 (f) opening of (d) with b2
(g) B component (h) opening of (g) with b1 (i) opening of (h) with b2 Fig. 3-5 Openings of Fig.3-3(b).
We improve this problem by changing RGB into YIQ color model for the sake of human perception and linear transform. In this color space, the false segmentation of one object with some similar colors can be avoided.
Use the same images mentioned above for example. The openings of Fig.3-3 under YIQ are listed in Fig-3.6 and Fig-3.7. Only Y component, the luminance part has similar blocking effect as which under RGB color model. We can see the blur shape of horse and bus with light gray in I and Q components.
Therefore, the granulometry histogram gives more information about size of objects than luminance information of objects. If the size of objects in query image and that of matching images are closed, they will be retrieved as similar images. Since the histogram has been normalized, it is independent of image size. It depends only if the ratio of object size and image size are very different. That is, this histogram says dissimilar when the same object in different images has large variations in ratio of object size and image size, or says similar when different objects in different images have the same ratio of object size and image size.
(a) Y component (b) opening of (a) with b1 (c) opening of (a) with b2
(d) I component (e) opening of (d) with b1 (f) opening of (d) with b2
(g) Q component (h) opening of (g) with b1 (i) opening of (h) with b2
Fig. 3-6 Openings of .Fig.3-3(a) under YIQ.
(a) Y component (b) opening of (a) with b1 (c) opening of (a) with b2
(d) I component (e) opening of (d) with b1 (f) opening of (d) with b2
(g) Q component (h) opening of (g) with b1 (i) opening of (h) with b2 Fig. 3-7 Openings of .Fig.3-3(b) under YIQ.
3.3 Similarity
3.3.1 Similarity for Morphological Granulometry
We use absolute difference as the similarity measure for granulometric distribution.
Therefore, similarity of granulometric distribution equals the sum of M absolute difference between target and matching images.
Besides, we choose intersection as the similarity measure for granulometric histogram.
Hence, similarity of granulometric histogram equals M minus the sum of M histogram intersection.
3.3.2 DPF for Primitives
Here, we propose a new similarity measure for primitives, the DPF. The distance between two N-tuple primitive vectors is the sum of the smallest βbins where β < . N This idea comes from some problem of thresholding method. It is difficult to set a fixed threshold that is suitable for every image. And once a threshold T has been set, it is possible that non of the distance within the bins of the feature vectors is less than T . To make sure at least one feature bin has been considered, we choose some smallest bin distance as the
Follow the definitions of primitive similarity in section 2.4.2. We only update the distance between PC and kq PCλs into the following equation:
∑
== β
λ
1 2 ,
_ ,
j j s
q
k d
PC
D ( 44 )
where d is the smallest j th element of j
{
pck iq, −pcλs,i|i=1,...,N}
. The rest are the same as the primitive similarity.Chapter 4
Experimental Results
In this chapter, we will present some experimental results obtained by applying the proposed methods. We also compare the results with color histogram, morphological primitives [10] and color moment primitives [6].
4.1 Experimental environment
The 1000 test images are selected from Corel’s database, and classified to 10 types of models, beach, buildings, buses, dinosaurs, elephants, flowers, horses, underwater worlds, and foods. Each type contains 100 images. Fig. 4-1 show some examples of each type from the database. The image size is either 384*256 or 256*384 pixels.
Images have similar object size in each of type 1, 4, 5, 6, 7, 8, and 9, respectively. Only type 1, 5, 9 have clean background. We define that an image has a clean background for there isn’t too many variant colors or objects inside and with a large area of some similar colors.
Objects in images of type 2, 3, 10 are very complicated. But every image has blue sky and ocean in type 2, has buildings in type 3, and has plates in type 10. Some images in type 3 also have blue sky.
(a) Type 1 (b) Type 2 (c) Type 3 (d) Type 4 (e) Type 5
(f) Type 6 (g) Type 7 (h) Type 8 (i) Type 9 (j) Type 10
Fig. 4-1 Examples of the image database.
The proposed methods are implemented on PC with Pentium4 2.8GHz, RAM 1G.. The operating system is Microsoft Windows XP SP2. The program was developed in the C++
language and compiled under Borland C++ Builder version 6.0. Our feature database is built in MySQL for the sake of its combination with PHP in the network.
The first experiment compares the results of morphological primitives (MP) using smallest DPF and original primitive similarity measure. The second experiment shows the results of morphological granulometric distribution (GD) and its primitives (GDP). The third experiment demonstrates the results of the granulometry histogram under RGB (GHrgb) and YIQ (GHyiq) color model. The fourth experiment compares granulometry histogram, granulometric distribution, primitives of granulometric distribution, morphological primitives
(MP), primitives of color moments (CMP), and color histogram (CH). And finally, we introduce the fifth experiment to verify the robustness of our image retrieval system.
4.2 Experimental results
We use precision and recall to measure our image retrieval system. The definitions of them are described in the following text. Assume that R is the number of retrieved relevant images, T is the total number of relevant images and K is the number of retrieved images.
Then,
K
= R
Precision ( 45 )
T
= R
Recall ( 46 )
4.2.1 Morphological Primitives
Morphological Primitive using smallest-8 DPF has improved the precision of all types in image database except type5 dinosaur. In Fig.4-2, MP means MP using similarity in section 2.4.2, MP8 and MP7 means MP using the smallest-8 and smallest-7 DPF, respectively. We only show the results of three types (model, dinosaur, and horse) in Fig.4-2.
In type1 model, the curve of MP7 is slightly lower than the other two curves when recall is less than 75, but larger than the other two when recall is larger than 75. When retrieve for the top 97, precisions of MP, MP8, and MP7 are 38%, 50%, 70%, respectively. Precision of MP7 for top 97 increases 32% to compare with MP. In type5 dinosaur, the curve of MP8 and MP7 are both lower than that of MP.
The consistency of regular object size and clean background upgrades the precision-recall curve. Although type1 and type5 both have clean background, models have two object sizes while dinosaurs have only one. There is space for improvement in type1 not in type5. That’s why two types with similar options are not both upgraded when using DPF.
Type1: Model
Type5: Dinosaur
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 10 20 30 40 50 60 70 80 90 100 Recall
Precision MP
MP8 MP7
(c)
Fig. 4-2 The Precision-Recall curves when using morphological primitives. (a) The results of Type1 (b) The results of Type7 (c) The results of Type5.
4.2.2 Granulometric Distribution and its Primitives
We use five different numbers of sets of structuring elements to experiment on granulometric distribution. The precision-recall curves are very close with each other.
Consequently, we choose the set of seven structuring elements to attain the average precision and take less computation time. Then, we try on its primitives with Z =6,9. On average, precisions of GDPs are better than those of GDs, but all lower than those of GHs (see next section). Precision curves in Fig. 4-3 show that GH has the best performance.
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10
Type Number
Precision
GH1rgb GH7rgb GH1yiq GD7 GDP6 GDP9
Fig. 4-3 The precision comparison for each type of image database among the proposed methods, granulometric histogram, granulometric distribution, and primitives of granulometric distribution (R=20).
4.2.3 Granulometric Histogram
We suppose M =1,2,...,7 for GH under RGB, and M =1,2,3 for GH under YIQ. In Fig. 4-4, GHk, k =1,2,...,7 means GH under RGB with M =k, and GHyiqk, k =1,2,3 means GH under YIQ with M =k.
From the results in Fig. 4-4, we may conclude that it is more suitable to take GH under YIQ as an object size feature than GH under RGB in most cases. Precision-recall curves of images with variant object size like type2 beach and type 9 under water will be degenerated.
The sizes of two types, flowers and elephants, are very close. Therefore, the mismatching of these two type images is caused. The first type models is a little bit degenerated for the reason that images of models have not unique but two major sizes. The rest of types are all superior to those under RGB.
Type1: Model
Type8: Horse
Fig. 4-4 Comparison of GH under RGB and YIQ.
4.2.4 Comparisons
To show the performance of the proposed methods, the retrieval results are compared with those using morphological primitives, color moment primitives, and color histogram. Fig.
4-5 shows the precision-recall curves of each type in image database using different method.
And Fig. 4-6 list the precisions of each method whenR=20. Due to the factor of object size, GH is a good feature for type1, 5, 6, 7, and 9. It is only below CH but above all other methods for type 3 and 8. And those are lower than other methods for type 2, 4 and 10.
We may see more clearly by compute the difference of precisions between the proposed method GH and other methods which are listed in Table 4-1. The best improvement of GH when compared with MP, CMP, and CH is 47%, 38%, and 37%, respectively, and the worst degradation is 5%, 14%, and 17%, respectively.
Type1: Model
(c)
Fig. 4-5 The Precision-Recall curves of different processing methods.
20%
Fig. 4-6 The precision comparison for each type of image database among the proposed methods and other methods (R=20).
Table 4-1 Differences of precisions between proposed method GH and other methods (MP9, CMP, and CH).
MP9 CMP CH
4.2.5 Example of Retrieval Results
We give an example of a flower image by retrieve the top relevant 20 images with methods mentioned above and show the results in Fig. 4-7. Precisions of CH, CMP, GH with YIQ, GH with RGB, MP7, MP8, and MP9 are 95%, 90%, 95%, 100%, 95%, 100%, and 90%,
(a) Color Histogram
(b) Color Moment Primitives
(c) GH under YIQ with 1 structuring element
(d) GH under RGB with 1 structuring element
(e) Morphological Primitives with smallest-7 DPF
(f) Morphological Primitives with smallest-8 DPF
(g) Morphological Primitives
Fig. 4-7 Image retrieval using (a) Color Histogram (b) Color Moment Primitives (c) GH under YIQ with 1 structuring element (d) GH under RGB with 1 structuring element (e) Morphological Primitives with smallest-7 DPF (f) Morphological Primitives with smallest-8 DPF (g) Morphological Primitives for the top relevant 20 images.
4.2.6 Processsing Time
In Fig. 4-8, we show the average feature extraction time and average similarity measure time of methods discussed above. Our method takes less feature extraction time than MP and CMP, but almost three times of search time of CH since GH does intersection of three granulometric histogram.
Feature Extraction Time
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08
CH CMP MP GH
seconds
(a)
Search Time
0 1 2 3 4 5
CH CMP MP GH
Seconds
(b)
Processing Time
0 1 2 3 4 5
CH CMP MP GH
seconds
(c)
Fig. 4-8 Histograms of processing time.
Chapter 5
Conclusions and Future Works
5.1 Conclusions
In this study, we propose a new method of image retrieval using morphological granulometric histogram. We also propose a new method of morphological primitives with dynamic partial function. By computing the granulometry histogram, we get the color, shape, and object size feature of an image. And it can represent a feature of an image effectively. On the other hand, the similarity, dynamic partial function, improves the precision of morphological primitives.
From the results shown in Chapter 4, we can conclude that our method has some advantages as described below.
1. Morphological operators involve simple logical operations and can be implemented in parallel, making real-time applications possible. [10]
2. Morphological granulometric histograms can represent features of images effectively.
3. Our proposed method reduces time to feature extraction of morphological primitives and color moment primitives.
4. Our proposed method performs good precisions.
5.2 Future works
We introduce the future works in the following topics. First, construct a hybrid image retrieval system that using our proposed method after a region segmentation. Second, seek for
faster similarity for histogram. Third, the relevant feedback shall be designed to make the interaction with users. Finally, to construct an on-line search engine one can use the technique of network and image retrieval.
REFERENCE
[1] Rafael C. Gonzalez, Richard E. Woods, “Digital Image Processing”, Prentice Hall, 2001.
[2] Ze-Nian Li, Mark S. Drew, “Fundamentals of Multimedia”, Prentice Hall, 2004.
[3] Theo Gevers, “Color in Image Search Engines”, Principles of Visual Information Retrieval, editor Michael S. Lew, Spring-Verlag, London, ISBN 1-852333-381-2, February 2001.
[4] M.W. Richardson, “Multidimensional Psychophysics,” Psychological Bull., vol. 35, pp.
639-660, 1938.
[5] Arun Qamra, Yan Meng, and Edward Y. Chang, Senior Member, IEEE, “Enhanced Perceptual Distance Functions and Indexing for Image Replica Recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, No. 3, pp.379-391, March 2005
[6] J.-L. Shih and L.-H. Chen, “Color image retrieval based on primitives of color moments”, IEE Proceedings Vision, Image and Signal Processing, pp370-376, 2002.
[7] C. R. Giardina and E. R. Dougherty, “Morphological Methods in Image and Signal Processing,” Prentice Hall, New Jersey, 1988.
[8] Matheron, G., “Randoms sets and integral equation”, New York, Wiley, 1978.
[9] Cecilia Di Ruberto, Andrew Dempster, Shahid Khan, bill Jarra, “Segmentation of Blood Images Using Morphological Operators”, IEEE, pp397-400. 2000.
[10] Jao-Shian Wu, “Color Image Retrieval Based on Morphological Primitives”, master thesis in NCTU, 2003.
[11] B.S. Manjunathe, Jens-Rainer Ohm, Vinod V. Vasudevan, and Akio Yamada, “Color and Texture Descriptors”, IEEE Transactions on circuits and systems for video technology, Vol. 11, No. 6, pp703-715, 2001.
[12] Jose M. Martinez, “MPEG-7 Overview”, ISO/IEC JTC1/SC29/WG11 N4980, 2002.
[13] Byoung Chul Ko and Hyeran Byun, “FRIP: A Region-Based Image Retrieval Tool Using Automatic Image Segmentation and Stepwise Boolean and Maching”, IEEE transactions on multimedia, Vol. 7, No. 1, pp.105-113, 2005.