CHAPTER 3 THE PROPOSED METHOD
3.1 Segmentation
3.1.1 Flower and stamen bounding rectangles location
In order to reduce the limitations of input images and to extract the correct
flower boundary, an interactive method is provided for user to bind the query flower.
First, a rectangle is drawn by user to bind an interested flower using mouse click and
drag controls. After this operation, the system will detect four edge points of the
(a) (b)
Fig. 5. An example of wrong extraction in overlapping flowers. (a)Original image.
(b) The extracted flower boundary.
flower automatically. Since, some flowers contain obvious stamen in the flower center
and the color of stamen differs from the petal. This will cause a strong edge response
in the stamen which may confuse us when detecting edge points. To treat this problem,
a stamen bounding rectangle is located automatically, and the size is 1/9 of flower
bounding rectangle. Then, the edge points within the stamen bounding rectangle are
ignored when locating four flower edge points. Fig. 7 shows flowers with stamen Original
flower image
Draw the flower and stamen bounding
rectangles
Detect four start/end tracing edge points
Modify tracing point(s) if necessary Search the
flower boundary
Fig. 6. Flow diagram of the segmentation steps.
parts and the located stamen bounding rectangle. Note that the stamen bounding
rectangle actually contains the stamen part.
3.1.2 Start/end tracing edge points locating
In this subsection, we will describe the edge point detection method used to
locate four edge points. These four edge points will be used as start/end tracing points
for boundary searching. These four edge points will divide the flower boundary into
four parts; we trace each part separately to speed up segmentation processing time.
Before describing the method, the local cost of each pixel is defined as
Local cost= 1+MG−G, (1) Fig. 7. Flowers with stamen parts and located bounding rectangles.
where G denotes the gradient magnitude for the pixel and MG denotes the
operators (see Fig. 8). The approximation of the gradient magnitude G for the pixel
y
gradient magnitude for each color band and calculate the maximum over the three
respective outputs to produce a single valued local cost image.
According to the above definitions, the edge point detection method is described
next. We scan vertical and horizontal lines in the middle of the flower bounding
rectangle. One edge point for each scan line will be detected. On the flower bounding
rectangle, five points are taken: the flower’s center P0 and the middle points
edge point. The one closest to the border of the flower bounding rectangle is taken as P4
Fig. 9. Four scan lines in the flower bounding rectangle.
the edge point (see e1, e2, e3 and e4 in Fig. 10). Note that for a small percentage of
pictures, the method aboved may be failed to get correct start/end tracing edge points
on each scan line. Fig. 11 is an example. The reason is that some flower image a
stamen part is smaller than the defined one (1/9 of the flower rectangle region) which
causing the real edge be excluded outside. In such a case, point is modified by user
using a mouse to drag the point to the correct position.
P
4Fig. 10. An example of profile on sun flower.
3.1.3 Boundary search algorithm
In this subsection, a boundary search algorithm will be described which divides
the boundary searching into four parts to speed up segmentation process. After
obtaining the four start/end tracing edge points, the flower boundary will be found
next. The pixel within the flower bounding rectangle will be traced as a graph by
considering each pixel as a vertex in the graph. Edges in the graph connect adjacent
pixels. Each pixel is adjacent to 8 neighbors. the cost of an edge is defined from one
pixel to its neighbor as the neighbor pixel’s local cost.
The proposed algorithm modified the 2-D dynamic programming graph search
algorithm proposed by Mortensen et al. [7]. The concept of average path cost is used
in the proposed algorithm. The average cost is only considered on a partial average
cost from the previous pixel and the next pixel to decide which direction to move. The
e
1Fig. 11. An example of wrong edge detected. (a) Wrong detection result. (b) The wrong point corrected by user.
partial average cost is updated by adding the average of the previous pixel cost and
the next pixel cost. The boundary searching is divided into four parts and use 1/4
sub-region (see R1, R2, R3 and R4 in Fig. 12) of the flower bounding rectangle
as the search region for each search to reduce the computation loading. The 4 pairs of
edge points (e1, e2), (e2, e3), (e3, e4) and (e4, e1) are considered as the start and end
tracing points for each search respectively. And the proposed algorithm will start from
the start point and stop when the end point is reached. After 4 times boundary
searching, a flower boundary is received (see the yellow curve in Fig. 12).The
modified 2-D graph search algorithm is as follows:
Algorithm: 2-D graph search (Continued)
Input:
s Start pixel.
f End pixel.
c(q,r) Cost function for link between pixels q and r.
Data Structures:
L List of active pixels (i.e. pixels are not determined with the minimum average cost yet and will be chosen as candidates to expand at next step) sorted by average cost (initially empty).
N(q) Neighborhood set of q (contains 8 neighbors of pixel).
e(q) Boolean function indicating if q has been expanded/processed.
g(q) Cost function from start pixel to q.
Output:
p Pointers from each pixel indicating the minimum cost path.
Algorithm:
g(s)=0; L=L+{s}; { Add active list with zero cost start pixel.}
while (L≠φ and not e(f))do begin { While still points to expend.}
q←min(L); { Remove minimum cost pixel q from active list.}
e(q)=TRUE; { Mark q as expanded (i.e. processed).}
for each r∈N(q) with not e(r) do begin
temp=(g(q)+c(q,r))/2;{ Compute average cost to neighbor.}
if r∈L and temp<g(r) then
L=L-{r}; { Remove higher cost neighbor from list.}
if r∉L then begin { If neighbor not on list, }
g(r)=temp; { assign neighbor’s average cost, } p(r)=q; { set (or reset) back pointer, }
L=L+{r}; { and place on (or return to) active list.}
end end end
3.2 Feature extraction
The most obvious features in flowers are shape and color. 3 shape features and Fig. 12. Four sub-regions of the flower bounding rectangle.
R
4R
1R
3R
211 color features are used and the first six color features are proposed by Saitoh et al.
[6].
3.2.1 Shape features
For the convenient explanation, we call the internal region bounded by the
extracted boundary as flower region. Based upon the extracted boundary, the gravity
center (gx,gy) of the flower region is defined to be
∑
F1： A ratio which indicates the relevance sharpness of the petals to the flower.
The feature is defined as
F2： The average distance from flower points in boundary to the gravity center.
The feature is defined as
10
F3： Roundness which indicates that how much the boundary shape is closer
to a circle. The feature is defined as
F2 4 2 L πS
= (7)
where L is the boundary length of the flower and S is the total number of
pixels in the flower region.
When the feature is close to 1, it means that the flower shape is close to a circle.
3.2.2 Color features
Flower images are represented in the RGB color model. Owing to that the
images are taken in different days and weather, the RGB values are coverted into the
HSV (hue, saturation and value) values [8] in order to reduce the illumination
variation. The primary, secondary and thirdly flower colors and the stamen color are
taken as our features. First, the HS space is divided into 12x6 cells represented by C , i
=
i 1, 2, …, 72.( see Fig. 13). Let P(Ci) be the probability of pixels with colors in
cellC . Leti DC1, DC2, DC be the cells with the largest, the second largest and the 3
third largest probabilities, respectively. The coordinates of the centers of the three
cellsDC1, DC2, DC and the three corresponding probabilities are taken as our 3
features. The details are described as follows:
)
recognition. So, the color distribution in stamen area which is a square area with
length of 2/3 petal length around the gravity is taken as our feature, where petal length
is defined by R . Let 90 SC be the cell with the largest probability of pixels with
color in stamen area. The center of the cell SC, is taken as our features. The details
are described as follows:
F13： the x coordinate of the center of the color cell SC,
F14： the y coordinate of the center of the color cell SC.
3.3 Recognition
Since the variation of flowers, a classification method does not used for grouping
species and recognition. For example, orchid may contain different colors in petal. If
these instances’ features are grouped together, it may mess up this color feature’s
discrimination ability. Hence, recognition is accomplished using the k-nearest Fig. 13. The HS space divided into 12x6 cells.
neighbor algorithm.
3.3.1 K-nearest neighbor algorithm
In the k-nearest neighbor algorithm, each flower image is considered as an
individual class. First, the recognition system calculates the distance between query
and all flower images in the database. All features are normalized to [0, 1] and the
initial weights for F1 to F14 are set according to the separability of each feature. The
initialized weight w of the i ith feature is defined as
The distance is calculated by Euclidean distance
(10)
3.3.2 Relevence feedback algorithm
Since the result does not always meet the user’s expectation, a feedback
mechanism is provided to adjust the weights of all features in response to each user’s
subjective point of view. A relevence feedback algorithm is provided to automatically
adjust the weight of each feature according to the current retrieval result and the
user’s response. After revising weights, system returns new candidates to user.
Steps for relevence feedback algorithm are presented as follows:
1. The user chooses u alike flower images,A1,A2,...,Au, and t unlike flower
4. Revising the kth feature weight wk
alike k unlike k
wk
σ
=σ k=1, 2, ..., 14 (14)
5. Running recognition engine again with new weights to get new candidate
images.
Note that when some images are selected as alike and unlike images, there will
exist some features, which are similar in those alike images, these features in the
unlike images will be dissimilar to the alike images. These features have high
separability and should have high weights. From the above definition, we can see that
for the kth feature, if σkalike is small and σkunlike is large, this means that the feature
is similar to each other in alike-class but dissimilar with unlike-class. Then, the
feature’s weight will be increased according to the formula (10). Increasing this
feature’s weight will let retrieved images farther from those images in unlike-class
and closer to the alike-ones. After adjusting all of the weights, the new result will be
closer to what the user really wants [9].
CHAPTER 4
EXPERIMENTAL RESULTS
In this chapter, experiments are conducted to evaluate the performance of the
proposed method. First, the separability of each feature is showed. Next, recognition
results are presented based on two databases which are our database of 348 flower
images consisting of 24 species and Zou-Nagy’s [4] database of 612 flower images
consisting of 102 species. The performance will be compared with Zou-Nagy’s
method. Besides, two databases are combined to a bigger species database consisting
of 114 species and the performance of the proposed method will be shown. Finally,
the effectiveness of using the feedback mechanism will be shown.
We take one feature each time in recognition and receive the top 5 accuracy rate
of each feature. Top 5 accuracy means that one of the 1st, 2nd, 3rd, 4th and 5th candidate
species is the same species of the query one. The separability of each feature in our
database is shown in Table 1. Note that every database has its own initialized weight
for each feature.
Each flower image in our database is resized to 400 x 300 pixels. Each flower
image in the database is taken as our query image and the remaining 347 flower
images are considered as training data. The result is shown in Table 2. The average
processing time is 4.15 seconds which includes user interaction and boundary
segmentation time.
Table 1. The initialized weight for each feature in our flower database.
Feature
Top 5(%)
Accuracy Top5 / Sum of top 5 accuracy Initialized weight
F1 58.05 0.0708 7.1
Table 2. Performance on our own database.
Recognition rate (%)
from [10]. All images have size of 300 x 240 pixels. Each species contains six images.
Some pictures are quite out of focus, and several pictures contain multiple, tiny,
overlapping flowers. The result is shown in Table 3. From the table, we can see that
the processing time of our method is twice faster than Zou-Nagy’s method. And the
Top-3 recognition rate (96.2%) is much higher than Zou-Nagy’s (79%) with 8.5
seconds user’s rose-curve adjustments before labeling the flower to the class.
Table 3. Performance on Zou-Nagy’s database Recognition rate (%) Time
(s) Top 1 Top 2 Top 3 Top 4 Top 5
Our method 4.2 74.4 92.2 96.2 97.4 98.4
Zou-Nagy’s method
(before labeling) 8.5 52 79
Zou-Nagy’s method 10.7 93
These two databases are combined into a bigger one and the performance of the
proposed method is shown in Table 4. 684 flower images collected from 114 distinct
species are used to test the proposed algorithm.
Finally, an example is further presented to show the effectiveness of using the
feedback mechanism (see Figs. 14 and 15). In Fig. 14, the first glory bush is used as
the query image and the others are the twenty most similar images search by the
proposed method without user’s feedback. Similar color and shape flowers are
propofdfdfdfdfdfdfd Table 4. Performance on combined database Recognition rate (%)
Time
(s) Top-1 Top-2 Top-3 Top-4 Top-5
number of images
number of species
4.5 74 90.1 94.2 96.2 97.2 684 114
retrieved. If the third image is selected as the alike one and the fifth image as the
unlike one, and then the feedback mechanism is applied, a better retrieval result is
obtained (see Fig. 15). We have 18 images of the glory bush in the database and 5
images of the glory bush are retrieved initially. After feedback mechanism, 16 images
of the glory bush will be retrieved and the candidate images from top 1 to top 10 is the
same species with the query one consequently.
Fig. 14. The initial retrieval result of the glory bush without using the feedback mechanism.
Fig. 15. The retrieval result of the glory bush after using feedback mechanism.
A recognition system written in C# language on a PC is built. Figs. 16 and 17 show
the main interface. Fig. 16 is the interface for user’s interactive operations and Fig. 17
is the interface for recognition result. After the candidate flower images are retrieved,
if there is one image looks like the query one then clicking on the image, the
information about the flower (name, scientific name and other relative species
information) will be showed to the user. Therefore, the system only need one image in
the retrieval results which is the same species with the query one that can recognize
the query flower correctly.
Fig. 16. Interface for user interactive operations.
Fig. 17. Interface for recognition result.
CHAPTER 5 CONCLUSIONS
In this thesis, we have presented an interactive method for flower recognition
which can deal with photographs with fewer constraints. First, a simple user
interactive process is applied to bind the searching area for segmentation. Then, the
system detects the edge points automatically and runs the segmentation engine.
Fourteen features are extracted from the segmented flower image. In the end, a
similarity measure is provided to do recognition. Furthermore, to make the
recognition result more close to the user’s need, a relevance feedback algorithm is
provided to automatically determine the importance of each feature. Based on the
algorithm, a system can be built to recognize flower with photographs. Experiments
designed for assessment of performance showed the accuracy and efficiency of the
proposed method.
REFERENCES
[1]T. Saitoh and T. Kaneko, “Automatic Recognition of Wild Flowers,” Proc.
International Conference on Pattern Recognition, Vol. 2, pp. 507-510, 2000.
[2]M. Das, R. Manmatha, and E-M. Riseman, “Indexing Flower Patent Images Using
Domain Knowledge,” IEEE Intelligent Systems, Vol. 14, pp. 24-33, 1999.
[3]A. Hong, G. Chen, J. Li, Z. Chi, and D. Zhang, “A Flower Image Retrieval
Method Based on ROI Feature,” Journal of Zhejiang University (Science) Vol. 5,
pp. 764–772, 2004.
[4]J. Zou and G. Nagy, “Evaluation of Model-Based Interactive Flower Recognition,”
Proc. International Conference on Pattern Recognition, Vol. 2, pp. 311-314,
2004.
[5]M-E. Niksback and A. Zisserman, “A Visual Vocabulary for flower classification,”
Proc. Computer Vision and Pattern Recognition, Vol. 2, pp. 1447-1454, 2006.
[6]T. Saitoh, K. Aoki and T. Kaneko, “Automatic Recognition of Blooming Flowers,”
Proc. International Conference on Pattern Recognition, Vol. 1, pp. 27-30, 2004.
[7]E-N. Mortensen and W-A. Barrett, “Intelligent Scissors for Image Composition,”
Proc. Computer Graphics and Interactive Techniques, pp. 191-198, 1995.
[8]HSL and HSV, http://en.wikipedia.org/wiki/HSI_color_space.
[9]J. L. Shih and L. H. Chen, “A Context-based approach for color image retrieval,”
International Journal of Pattern Recognition and Artificial Intelligence, Vol. 16,
pp. 239-255, 2002.
[10] http://www.ecse.rpi.edu/doclab/flowers.