CHAPTER 3 THE PROPOSED METHOD
3.4 Feedback
of dominant color Cn. Then, as for the image distance, while the numbers of dominant colors varies by different butterfly images, a new distance function is provided. The distance between the database image i with M dominant colors and the query image q with N dominant colors will be
m
where denotes the mth dominant color of database image i, and denotes the nth dominant color of query image q.
i
Cm Cnq
Finally, the recognition system ranks the distances and returns the top 20 nearest neighbors to the user. These returned images are called candidates.
3.4 Feedback
Since the result does not always meet the user’s expectation, two feedback
37
mechanisms are provided to adjust the weights of features in response to each user’s subjective point of view. The two mechanisms use the same relevance feedback algorithm to different candidates, which are the top 20 nearest neighbors and the nearest neighbor of each species respectively.
Our features are based on the dominant color; hence, the relevance feedback algorithm is especially designed for this property. The steps of our algorithm are presented as follows:
Step 1. User chooses u alike butterfly images, A1, A2, …, Au, and t unlike butterfly images, N1, N2, …, Nt, from candidates.
Step 2. By previous similarity function, we can obtain the difference between a
candidate image and the query image. Hence, for each alike image, calculate the difference produced by the kth feature. Summarize the differences Dkalike and obtain its percentage Pkalike, as follows:
Step 3. Repeat step2 on unlike images to obtain its percentage Pkunlike. Step 4. Use the current kth feature weight wk to obtain the new weightwknew
As user’s expectation, these alike images should be similar with the query image, and the unlike images would be dissimilar with the query image. Therefore, for the kth feature, if is smaller than , which means the feature can represent the similarity relation correctly and whose weight should be increased. On the other hand, if is larger than , which means the feature cannot represent the similarity relation well and whose weight should be decreased. During the revising process, the weight will be adjusted smoothly since we multiply the original weight with higher percentage in order to keep the system stable. After adjusting these weights, the new result will be closer to what the user anticipates.
alike
Pk Pkunlike
alike
Pk Pkunlike
39
CHAPTER 4
EXPERIMETAL RESULTS
In this chapter, experiments are conducted to evaluate the performance of our system. First, the separability of each feature is showed and will be used as our initial feature weight. Next, the three testing datasets will be introduced. Their related recognition results and discussions will also be revealed. Finally, the system’s interfaces and the feedback mechanisms will be shown.
Before the experiments start, the initial feature weights should be acquired first.
We take one feature each time in recognition and receive the top 1 accuracy rate of each feature. Top 1 accuracy means that the first candidate is the same species of the query one. Then, the separability of each feature can be obtained, shown in Table 2, and the initial feature weights can be decided also.
Table 2. The initial weight of each feature in our database.
Feature Top 1(%) Accuracy /
Initial weight Feature Top 1(%) Accuracy / Initial weight
F1 0.2378 F5 0.0747
F2 0.0853 F6 0.0558
F3 0.0564 F7 0.0865
F4 0.0744 F8 0.0904
40
The recognition results are presented based on three testing datasets. The three datasets are constructed by the sample images, the natural images from field guides and the natural images from internet. In the sample testing dataset, the images are taken in the simulation condition as the database images, but on different butterfly (partial) or under different shooting direction and light environment. In order to test each case of the designed model, we take two sample images for each case, and there will be total 3484 images in the sample testing set.
As for the natural images, due to different life cycle and habitats, we could not collect clear images of all kinds of butterflies by ourselves. To overcome this shortage, we obtain the natural images from field guides [14-20] as our field guide testing dataset. The field guide testing set includes 26 species and total 130 images. At the same time, we also acquire the natural images from internet as our network testing dataset. The network testing dataset includes 26 species and 60 images of each species.
The details of three datasets are shown in Table 3. (These images are only used in this study for academic purpose, and will not be spread or be used in other way. Therefore, the copyright is not infringed. Besides, all images shown in this paper are either taken by ourselves, or obtained from the field guides [14-20].)
The result of three testing datasets is shown in Table 4. The average processing time is 8.71 seconds including user interaction and boundary segmentation time.
41
Table 3. The composition of three testing datasets.
Testing dataset Number of species Number of images
Sample testing dataset 26 3484
Field guide testing dataset 26 130
Network testing dataset 26 1560
Table 4. Performance on three testing datasets.
Testing dataset Recognition rate (%)
Top 1 Top 3 Top 5 Top 10 Top 20 Sample testing dataset 94.72 98.54 99.54 99.91 99.97 Field guide testing dataset 41.54 54.62 60.00 70.77 77.69 Network testing dataset 46.92 64.23 71.54 79.74 86.86
Considering the effect of three butterfly groups with similar appearances, we do the experiment again by combining each group as a species. The combining result shown in Table 5 is better in some degree than the original result, which means our method can solve part of the similarity problem.
Table 5. The combing result of three testing datasets.
Testing dataset Recognition rate (%)
Top 1 Top 3 Top 5 Top 10 Top 20 Sample testing dataset 94.72 98.54 99.54 99.91 99.97 Field guide testing dataset 48.46 59.23 63.85 73.08 79.23 Network testing dataset 52.24 67.56 74.29 81.54 87.50
42
We have built our system on a PC by Java language. The system interfaces are shown in Fig. 13. Fig. 13(a) is the segmentation interface with user’s interactive operations, and Fig. 13(b) is the result interface which shows the top 20 candidate images after recognition. Fig. 13(b) also provides the first feedback mechanism.
Through simple operations, user can choose the alike and unlike candidates easily, and the feature weights will be adjusted automatically. Another feedback mechanism for the most related candidate of each species is shown in Fig. 13(c). This mechanism provides another way to help user find the query species. As shown in Fig. 13(d), user can obtain clear butterfly images and the related information (Chinese name, English name, scientific name, family and other useful information) from a candidate by simply click on that image. Therefore, the system only needs one image in the retrieval results to be the correct species of the query one.
43
(a)
(b)
Fig. 13. The system interfaces. (a) Segmentation interface. (b) Recognition interface with the 1st feedback mechanism. (c) The 2nd feedback mechanism interface.
(d) The related information of the chosen candidate image. (continued.)
44
(c)
(d)
Fig. 13. The system interfaces. (a) Segmentation interface. (b) Recognition interface with the 1st feedback mechanism. (c) The 2nd feedback mechanism interface.
(d) The related information of the chosen candidate image.
45
CHAPTER 5 CONCLUTIONS
In the thesis, we have proposed an interactive method for butterfly recognition on natural image. First, the study designs and constructs the useful database model of natural butterfly images. This concept and related research have not been revealed yet.
Second, a simple user interactive segmentation method is provided. The method is designed for the problems on natural images, which has been known as a challenging work for a long period. After segmentation, the system automatically extracts the dominant colors by the provided ARGBT and AET-K-means methods. Then, eight corresponding color and distribution features of each dominant color are obtained. In the end, a similarity measure is provided for recognition. Besides, to make the recognition result more close to the user’s requirement, two relevance feedback mechanisms are provided to automatically determine the importance of each feature.
We hope our system can give more contribution to the related researches, and popularize the education of ecology protection well.
The system may be improved in the future from two aspects: the application on mobile devices and the further better features. The concept of popularizing ecological education of butterfly through mobile system and internet has been gradually revealed
46
with time. [24-25] introduced the mobile system application which helps students learn ecological knowledge and promote the academic work. [26-28] were the butterpree and analyses of Prof. Hsiang’s system [4]. [29] described virtual butterfly museum through internet, which provide a web-learning environment. From these researches, we can realize that if our system can be applied on the mobile devices, it will help greatly on both convenience and the ecological education popularization.
Besides current features, we can increase our features in two ways as the future work too. The first one is combining other features, which lead to nice results in other researches. The second one is considering the suitable shape and texture features for butterfly recognition. We have researched on some texture features, but have not obtained satisfying results yet. We hope these methods can improve our system, and maybe one day makes the world a better place.
47
APPENDIX A
The detail of the designed database model
In Table A.1, the wing-spreading angle is the angle between two wings of butterfly. As the butterfly spreads wings, the angle will increase from 0 to 180, which mean the close condition and the fully spread condition respectively. The angles of shooting direction can be explained by the 3-D diagram shown in Fig. A.1(a). The horizontal angle α means the angle at x-y plane, and the vertical angle β represents the angle along z-axis. We set the left side of butterfly (see Fig. A.1(b)) as angle 0 of the x-y plane, and set the horizontal, on which the butterfly is placed, as angle 0 along the z-axis. Besides, the shooting cases which are taken from the top (β
= 90) or the bottom (β = -90) of butterfly will be recorded in their notations. As for the case description, we will not take pictures for the unused case, which is marked by the symbol “X”. The unused case is unnecessary for the database since its picture is similar with other cases or cannot give us enough information to recognize its species.
The reason of each unused case will be recorded in notation. Finally, for the consultation convenience, the example images of whole 67 cases are shown in Fig.
A.2.
48
㎏
(a) (b) Fig. A.1 The example images of shooting direction. (a) The 3-D diagram of
shooting direction. (b) The left side example image.
Table A.1. The detail of the designed database model. (continued)
Wing-spreading
45 X Insufficient Information 0 X Insufficient Information -45 X Insufficient Information 135
Table A.1. The detail of the designed database model. (continued)
45 X Insufficient Information 0 X Insufficient Information -45 X Insufficient Information 315
0 X Insufficient Information -45 X Similar with 001
45 X Insufficient Information 0 X Insufficient Information -45 X Insufficient Information 135
0 X Insufficient Information -45 X Similar with 002
45 X Insufficient Information 0 X Insufficient Information -45 X Insufficient Information 315
Table A.1. The detail of the designed database model. (continued)
-45 X Insufficient Information 135
0 X Insufficient Information -45 X Insufficient Information 315
0 X Insufficient Information -45 X Insufficient Information 45
45 058
0 X Insufficient Information -45 X Insufficient Information 90
45 059
0 X Insufficient Information -45 X Insufficient Information
51
Table A.1. The detail of the designed database model.
0 X Insufficient Information -45 X Insufficient Information 180
45 061
0 X Insufficient Information -45 X Insufficient Information 225
45 062
0 X Insufficient Information -45 063
270
45 X Insufficient Information 0 X Insufficient Information -45 X Insufficient Information 315
45 064
0 X Insufficient Information -45 065
90 066 Top
-90 067 Bottom
Fig. A.2. The example images of the 67 cases. (continued)
52
Fig. A.2. The example images of the 67 cases. (continued)
53
Fig. A.2. The example images of the 67 cases.
54
REFERENCES
[1] Li-Jen Chen, Ling-Hwei Chen, “A Fast Automatic Segmentation Algorithm based on Local Color-Distribution,” Master Thesis, Institute of Multimedia and Engineering, National Chiao Tung University, Taiwan, ROC, 2007.
[2] M. Nagao, K. Suzuki, A. Muraki, H. Ikeda, Y. Shimodaira, M. Yamazaki,
“Retrieval of Butterfly from Its Sketched Image Utilizing Image Navigation Directory on Multimedia Network,” Proc. IEEE ICECS’98, Vol. 2, pp. 95-98, Sep.
1998.
[3] K. Suzuki, M. Nagao, A. Muraki, H. Ikeda, Y. Shimodaira, M. Yamazaki,
“Reduction of Processing Time in Distributed Image Database System Utilizing Directory,” IEEE ICCT’98, Vol. 12, No. 7, pp. 629–639, Oct. 1998.
[4] J. Hong, H. Chen, J. Hsiang, “A Digital Museum of Taiwanese Butterflies,”
Proceedings of the fifth ACM conference on Digital libraries, pp. 260-261, 2000.
[5] B. Chen, J. Hsiang, “Content-Based Image Retrieval of Butterflies,” Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2000.
[6] S. Lai, J. Hsiang, “Color Retrieval of Butterfly Images,” Master Thesis,
55
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2002.
[7] C. Hung, J. Hsiang, “Texture Retrieval of Butterfly Images,” Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2002.
[8] T. Huang, J. Hsiang, “MPEG-7 Based Butterfly Image Retrieval on Shape,”
Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2002.
[9] W. Liu, J. Hsiang, “Model, Implementation, and Experiments for Fine-grained Image Retrieval,” Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2003.
[10] T. Hyuga, I. Nishikawa, “Implementing the Database System of Butterfly Specimen Image by Self-Organizing Maps,” Journal of Japan Society for Fuzzy Theory and Systems, Vol. 14, No. 1, pp. 74-81, 2002.
[11] J. Lim, J. Cho, T. Nam, S. Kim, “Development of A Classification Algorithm for Butterflies and Ladybugs,” 2006 IEEE Region 10 Conference TENCON 2006, pp.
1-3, Nov. 2006.
[12] O. Starostenko, C. K. Cruz, A. Chávez-Aragon, R. Contreras, “A Novel Shape Indexing Method for Automatic Classification of Lepidoptera”, XVII Int. Conf. on
56
Electronics, Communications, and Computers, Publ. IEEE Computer Society, pp.1-6, Feb. 2007.
[13] O. Starostenko, J. Chávez-Aragón, J. Sánchez, et al, “An Hybrid Approach for Image Retrieval with Ontological Content-Based Indexing,” Progress in pattern recognition, image analysis and applications, Springer-Verlag, Germany, Vol.
3773, pp. 997-1004, 2005.
[14] Y. Chang, “陽明山國家公園---賞蝶篇(上/下、導覽),” Yangmingshan National Park, Taiwan, 1994.
[15] The Forestry Bureau of Council of Agriculture Executive Yuan, “雙流蝶影,” The Forestry Bureau of Council of Agriculture Executive Yuan, Taiwan, 1998.
[16] Society of Wildlife And Nature, “愛蝴蝶護台灣,” Society of Wildlife And Nature, Taiwan, 1999.
[17] Y. Chang, “台灣賞蝶圖鑑,” Morning Star, Taiwan, 2000.
[18] Y. Chang, “台灣賞蝶地圖,” Morning Star, Taiwan, 2002.
[19] Butterfly Conservation Society of Taiwan, “台灣常見的蝴蝶,” Taiwan, 2003.
[20] Y. Chang, “蝴蝶100：台灣常見100種蝴蝶野外觀察及生活史全紀錄,”
Yuan-Liou, Taiwan, 2007.
[21] E. Hamano, Ecological Encyclopedia of TAIWANESE BUTTERFLIES, Japan 1986.
57
[22] N. Otsu, “A Threshold Selection Method from Gray-level Histogram,” IEEE Transactions on Systems, Man, Cybernetics 8, Vol. 9, pp. 62-66,1978.
[23] D. Margulis, “Photoshop Lab Color: The Canyon Conundrum and Other Adventures in the Most Powerful Colorspace,” Peachpit Press, 2005.
[24] Y. Chen, T. Kao, G. Yu, J. Sheu, “A Mobile Butterfly-Watching Learning System for Supporting Independent Learning,” Proceedings of the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE'04), pp.
11-18, 2004.
[25] Y Yu, J. A. Stamberger, A. Manoharan, A. Paepcke, “EcoPod: A Mobile Tool for Community Based Biodiversity Collection Building,” In: Proceedings of the 6th ACM/IEEE-CS JCDL, pp. 344–353. Chapel Hill, NC, 2006.
[26] C. Yu, J. Hsiang, “Butterpree-Preparing Course Material from a Digital Museum of Butterflies,” Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2000.
[27] P. Wu, L. Chan, “The Evaluation of the Design of Web-based Learning Environment--A Case Study of Butterfly Digital Museum,” Master Thesis, Graduate Institute of Library & Information Science, National Chung Hsing University, Taiwan, ROC, 2002.
[28] S. Chen, J. Hsiang, “A System for Creating Course Material from a Digital
58
59
Museum,” Master Thesis, Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, ROC, 2007.
[29] S. Chen, J. Hsiang, “Development of Internet Virtual Butterfly Museum,” Proc.
of Int. Conf. on Computers in Education (ICCE’02), Vol. 2, pp. 1139-1143, 2002.