An Image Annotation Approach Using Location References to Enhance Geographic Knowledge Discovery

(1)

An image annotation approach using location references to enhance

geographic knowledge discovery

Chung-Hong Lee

a,⇑

_{, Hsin-Chang Yang}

b

_{, Shih-Hao Wang}

a

Dept. of Electrical Engineering, National Kaohsiung University of Applied Sciences, Taiwan

b

Dept. of Information Management, National University of Kaohsiung, Taiwan

a r t i c l e

i n f o

Keywords: Image annotation Image classiﬁcation Image processing Text mining

a b s t r a c t

With the continuously increasing needs of location information for users around the world, applications of geospatial information have gained a lot of attention in both research and commercial organizations. Extraction of semantics from the image content for geospatial information seeking and knowledge dis-covery has been thus becoming a critical process. Unfortunately, the available geographic images may be blurred, either too light or too dark. It is therefore often hard to extract geographic features directly from images. In this paper, we describe our developed methods in applying local scale-invariant features and bag-of-keypoints techniques to annotating images, in order to carry out image categorization and geographic knowledge discovery tasks. First, local scale-invariant features are extracted from geographic images as representative geographic features. Subsequently, the bag-of-keypoints methods are used to construct a visual vocabulary and generate feature vectors to support image categorization and annota-tion. The annotated images are classiﬁed by using geographic nouns. The experimental results show that the proposed approach is sensible and can effectively enhance the tasks of geographic knowledge discovery.

1. Introduction

Location information provided on the web is a valuable contri-bution and useful resource for a multitude of applications and is becoming an important element of contents in web search. Recently several studies have reported that 5–20% of all user que-ries express a geographic information need, and an estimate of up to 20% of all web pages contain location references (Himmelstein, 2005; Kamvar & Baluja, 2006; McCurley, 2001; Sanderson & Kohler, 2004). In particular, a huge of geographic information are utilized and generated by various technologies, such as loca-tion-based services (LBS) and location-aware technologies (LAT), and are represented in the forms of images or texts on the inter-net. Under such a circumstance, geospatial search engines there-fore have to identify location-relevant content of web pages and analyze their semantics of location and related images. Going fur-ther, geoparsing approaches can be used to evaluate its location relevance.

Geographic knowledge discovery (GKD) is emerging as a novel and broad application research topic, covering ﬁelds of geospatial information science, environment management, national security,

and applications of location-based services, etc. Examples of geo-spatial data include data that describe the evolution of natural phe-nomena, earth science data that describe spatiotemporal phenomena in the atmosphere, or data that describe the location of individuals in the geographic space as a function of time. Geographic data represent an embedding to the Euclidean space, and therefore give distance information and topological informa-tion. In essence, the domain of the attributes is normally the real numbers. Also, geographic data sets can record additional informa-tion, such as environmental parameters and location-based tour-ism information, therefore defining high dimensional spaces of attributes that are highly correlated. As a result, geographic data mining methods offer solutions for finding and describing patterns in the geospatial data collections, which was previously unknown and is not explicitly stored in the database. In particular, it is worth mentioning that, the geographic knowledge discovery concerns not only the exploration and analysis of professional spatial data sets but also the methods in finding more general knowledge related to specific geographical locations.

The view is taken, therefore in this work the application sce-nario for the attempted solution is a location-based information system for mobile or pedestrian users. We aim to identify loca-tion references in the forms of texts and images at a ﬁne granu-larity level of individual buildings or tourist attraction that is directly applicable to a mobile user or retrieval and analysis tasks

⇑Corresponding author. Tel.: +886 7 3814526.

E-mail addresses: leechung@mail.ee.kuas.edu.tw (C.-H. Lee), yanghc@nuk. edu.tw(H.-C. Yang),wsh@dml.ee.kuas.edu.tw(S.-H. Wang).

Contents lists available atScienceDirect

Expert Systems with Applications

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / e s w a

(2)

at this geographical granularity. The location references include not only photos (images), but also texts. To process the resources of location information for exploring the document-to-location relationships, it is often considered using a hybrid solution involving text and image resources to tackle the issue. Clearly, it is necessary to develop a combination of features such as geo-location, text tags, and image content as geographic references for image annotation. In this work, we are particularly concentrating on exploring the methods of image processing for geographic knowledge discovery. An example of photo mapping illustrating the relationship of geographic images associated with specific locations is shown inFig. 1. However, extracting geographic infor-mation from images is a challenging work. The major difficulties encountered are that the geographic objects, unrelated characters, and cluttered background may occur in the image simulta-neously. In addition, the illumination of images, image clarity, and the angles of geographic objects in the image may also influ-ence the performance of extraction of geographic information from time to time. In this work, the aim is to incorporate the pro-posed techniques together to give a unified treatment of the problems and effective solutions to them.

In this work, we developed a novel method using local scale-invariant features and bag-of-keypoints techniques to perform geographic image annotation. Initially the collected geographic image sets were divided into two parts: labeled and unlabeled images. Subsequently the images were processed to generate key-points by utilizing Difference of Gaussian (DoG) and Scale-invariant Feature Transform (SIFT) approaches. Furthermore, we used the Afﬁnity Propagation (AP) clustering technique, and constructed a vi-sual vocabulary to enable similar keypoints of the images to be aggregated together. The resulting clustered keypoints in the geo-graphic images can be calculated to generate feature vectors. By utilizing Fuzzy ARTMAP (FAM), these feature vectors will be employed as our training and test samples for classifying images by using geographic nouns.

The rest of the paper is organized as follows: Section 2 dis-cusses related researches of technologies and concepts. Section

3 presents the architecture of our approach. Section 4 shows our experimental results, and ﬁnally we conclude this paper in Section5.

2. Related work

In this work there are several important disciplines and approaches related to the system development, such as image fea-ture detection/extraction, image annotation/classiﬁcation, geospa-tial/geographic data mining, and geographic information retrieval. They are described in details in the following subsections. 2.1. Image feature detection/extraction

Image feature detection and extraction are important tech-niques to recognize objects in the image. Related approaches in these fields are widely applied in industry, astronomy, military affairs and other various domains.Lowe (2004)presented an ap-proach so-called SIFT for extracting distinctive invariant features from images that can be utilized to perform reliable matching be-tween different views of an object or scene, and also proposed a method to perform object recognition using these features. By implementing SIFT approach for recognize object, SIFT can robustly identify objects in a clutter and occlusion background and reach near real-time performance. Lindeberg (1996)addressed a mech-anism for automatic selection of scale levels to solve the problem of the information type of extracted features. Unlike traditional ap-proaches, the proposed approach was to define the concept of edge and ridge as one-dimensional curves in the three-dimensional scale-space. Manjunath and Ma (1996) proposed a scheme so called Gabor wavelet based texture analysis to implement a multi-resolution representation based on Gabor filter. By using Gabor fil-ters in various factors, image features were extracted as data sources in order to construct digital libraries and multimedia dat-abases for browsing and retrieval of large image data. As reported in their experimental results, Gabor features are capable of pattern retrieval and browsing in large image databases accurately.

Manjunath, Shekhar, and Chellappa (1996)developed an approach to detect features based on scale-interaction model. Unlike para-metric model which requires graphics and image processing, their image model was not used to detect image feature points. The ap-proach was robustness in detecting image features and was able to be applied in image registration, face recognition, and motion cor-respondence. Harris and Stephens (1988) proposed a hybrid

(3)

method to combine corner and edge detectors, using the local auto-correlation function for satisfying image regions with texture and isolated features. Compared with other methods, the com-bined approach has good consistency on natural imagery. Lowe

(1999) presented a new approach so-called the Scale Invariant

Feature Transform (SIFT) for image feature generation, involving the construction of feature vectors which are invariant to image translation, scaling, rotation, and illumination change. By means of the developed approach, robust object recognition can be reached in cluttered partially-occluded images. Jones, Hassan,

and Diaz (2008)presented a new measure of phase congruency,

and uses wavelets in identifying and compensating for the level of noise in an image. In their approach, scale is varied by utilizing high-pass ﬁltering, rather than low-pass or band-pass ﬁltering methods, as well as feature locations remain constant over scale.

Tomasi and Kanade (1991) tackled the issues associated with

choosing appropriate feature windows which are best suited for feature tracking, by means of a tracking algorithm. Sallaberry,

Etcheverry, and Marquesuzaa (2006)addressed a new framework

based on SIFT for improving SIFT descriptor by means of integrat-ing color and global information. Ledwich and Williams (2004)

demonstrated an approach for reducing SIFT feature size, complex-ity and matching time, and applied this approach in indoor image retrieval and robot localization for making them efﬁciently and practically. Their experimental results indicated that the reduction method beneﬁts the image retrieval, and effectively decreased the size of image descriptors and matching time.

2.2. Image annotation/classiﬁcation

Image annotation and classification can be regard as an impor-tant research field for establishing hybrid approaches combining texts and images. In information retrieval research domain, such concepts can be used to search related image. Csurka, Dance, Fan, Willamowski, and Bray (2004)presented a bag of keypoints method for visual categorization, and try to solve the problem of identifying the object content of natural images while generalizing across variations inherent to the object class. As shown in the experimental results, such a method is simple, computationally efficient, and intrinsically invariant. Qin, Zheng, Jiang, Huang, and Gao (2006)proposed a novel approach to classify texture col-lections that without require experts to provide annotated training sets. By extracting a set of invariant descriptors from each image, each image is vector-quantized, and generated the texture image by bag-of-keypoints vector. Final Probabilistic Latent Semantic Indexing (PLSI) and Non-negative Matrix Factorization (NMF) are implemented to perform image classification. Cusano, Ciocca, and Schettini (2004)described an image annotation approach for classifying image regions. By multi-class Support Vector Machines, an annotation is performed, and this approach could be applied in large image and video databases for indexing images and videos automatically.Goh, Chang, and Li (2005)proposed an annotation method for supporting keyword retrieval of images. By employing one-class, two-class and multiclass SVMs, such concept was imple-mented, and they proposed a confidence-based dynamic ensemble (CDE) for ascertaining the correctness of an annotation. As results of experiments, their study was very effective on large real-world data set.Varma and Zisserman (2005)presented a novel approach to material classification by unknown viewpoint and illumination. Unlike previous 3D texton representations, they tried to employ rotationally invariant filters and cluster in an extremely low dimensional space. By constructing a texton dictionary, such an ap-proach was implemented to classify a single image without requir-ing any previous knowledge about the viewrequir-ing or illumination conditions.Qi and Han (2007)proposed a novel automatic annota-tion system by integrating multiple instance learning (MIL)-based

SVMs and global-feature-based SVMs. By using MIL on the image blocks, the MIL-based bag features are obtained.Carneiro, Chan, Moreno, and Vasconcelos (2007)proposed a probabilistic approach for semantic image annotation and retrieval, and tried to solve the problem that each class is deﬁned as the group of database images labeled with a common semantic label. By constructing this one-to-one correspondence between semantic labels and classes, a minimum probability of error annotation and retrieval are feasible. According to their experimental results, this approach is fairly ro-bust to parameter tuning. Ros, Laurent, and Lefebvre (2006) pre-sented an architecture for natural image classiﬁcation or visual object recognition. By a distribution of local prototype features obtained from projecting local signatures on a self-organizing map, the image content is described, and such signatures describe singularities around interesting points.

2.3. Geospatial/geographic data mining

In the direction of geospatial/geographic data mining, most of researches focused on discovering geographic knowledge.

Uryupina (2003)presented an approach to discover geographical gazetteers automatically from internet. By utilizing bootstrapping techniques, new gazetteers are learned starting from a small set of preclassiﬁed instances. This presented approach is helpful for the Named Entity Recognition task in language.Ding, Stepinski, Parmar, Jiang, and Eick (2009)proposed a supervised clustering ap-proach for discovering feature-based hot spots. Such a method is relied on supervised clustering to produce a list of hot spots re-gions. By employing a ﬁtness function, the data set is subdivided optimally, and try to rank using the interestingness of clusters. The relationship between hot spots and top ranked clusters is high-er.Goldberg, Wilson, and Knoblock (2009)described a methodol-ogy by generating highly complete and detailed regional gazetteers from Internet sources automatically for solving the problem of gazetteers are not complete and measures of their accuracy. By utilizing information extraction and integration tech-niques, geographic features, associated footprints, and widely available online data are obtained, and then such data can be used to create a gazetteer for nearly any area.McCurley (2001) investi-gated several various approaches to discovering geographic con-text for web pages, and a navigational tool is described for web browsing by geographic proximity. Olga (2002)described an algo-rithm for knowledge extraction in the geographic information. In order to classify places into different location types and determine for a give place name, text mining approach is applied to the Inter-net. As results of experiments, such an approach is able to create gazetteers for Named Entities Recognition tools automatically.

Zong, Wu, Sun, Lim, and Goh (2005) gave spatial semantics to

web pages by assigning place names. This assignment task is di-vided into three parts: place name extraction, place name disam-biguation, and place name assignment. And this approach works well for geo/geo ambiguities.Doerr and Papagelis (2007)presented a statistical model to solve the problem of missing information in the gazetteer, multiple matches, or false positive matches for inte-grating place names with actual locations. The model was based on statistical analysis of the place names mapping process and with-out any other background data. Such an approach has been applied to a real-world case study.

2.4. Geographic information retrieval

Geographic information retrieval (GIR) can be regarded as a spe-cialized branch of traditional information retrieval. It covers all of the research areas that have traditionally make up the core of research into information retrieval, but in addition has an empha-sis on spatial and geographic indexing and retrieval. For instance,

(4)

Sallaberry et al. (2006)proposed a core model to formally present geographic information, and also developed a PIV system that com-bined information extraction, information retrieval and informa-tion visualizainforma-tion according to geographic characteristics of documents. Lee, Yang, and Wang (2010)presented an ontology of place that combines limited coordinate data with spatial rela-tionships between places, and be employed to obtain semantic dis-tance measures in geographically-referenced information retrieval.

Buscaldi, Rosso, and Peris (2006)integrated data from gazetteers (GNS and GNIS), the WordNet general domain ontology, and Wikipedia, to produce an ontology that can be used as a source for the Geographical Information Retrieval task. By the extension of vector space model, such model is applied in Expert Search task.

Jones et al. (2008)organized large-scale search engine relevance experiments, using the 12% of queries that containing placenames, matching the placenames to places in the documents, and examin-ing the impact of geographic features on the retrieval relevance.

Cardoso, Cruz, Chaves, and Silva (2008)presented the work of par-ticipation in the University of Lisbon at the 2007 GeoCLEF. They adopt a novel approach for GIR; this approach is focused on han-dling geographic features and features types on both queries and documents, and generating signatures with multiple geographic concepts as a scope of interest.McCurley (2001)investigated sev-eral various approaches to discovering geographic context for web pages, and a navigational tool is described for web browsing by geographic proximity.

3. Approach and system framework

Based on the requirements mentioned above, in this section the system framework and development steps are described. There are three main processes in our system, including tasks of feature detection/description of geographic images, visual vocabulary con-struction/feature vector generation, and image annotation. The system framework is illustrated inFig. 2.

3.1. Overview of the approach

As shown inFig. 2, we ﬁrst collected images related to various geographic locations, and then start to extract geographic informa-tion from the collected images. However, geographic spots in images may have various shooting angles and brightness, which often leads to extract incorrect image semantics. To solve this is-sue, we employed the extraction method of bag-of-keypoints to extract local scale-invariant features and used such techniques to discover correct image object as our geographic information.

The concept of bag-of-keypoints (bok) (Csurka et al., 2004) is a vocabulary of keypoints from images. As the presentation of documents, the bok follows dictionary-based approach, and each document is regarded as a set of keywords. It means each docu-ment contained some proper nouns in the dictionary. In this study, the steps of system implementation are addressed as follows:

1. Feature detection and description of geographic images. 2. Extracting similar keypoints and constructing a visual dictionary

by the concept of bag-of-keypoints.

3. Generating geographic image feature vectors and annotating images as a source for geospatial text mining.

For geographic features detection and description of images, the Difference of Gaussian (DoG) and Scale-invariant Feature Transform (SIFT) (Lowe, 2004, 1999) methods were employed to extract and describe geographic information. They are described as follows.

3.2. Feature detection and description of geographic images

Local scale-invariant features are widely used in the applica-tions of object recognition and image registration. As afﬁne-invari-ant keypoints, such features are stable under lighting and viewpoint changes, and enable match keypoints between various images accurately. In this work, some image features are used to

(5)

sandy parts of the world. In fact, one of the challenges in our work is that geospatial data (speciﬁcally, vector and image data) obtained from various data sources may have different projec-tions, different accuracy levels, and different inconsistencies. To some extent, the proposed approaches that integrate infor-mation from various geospatial data, images, and text informa-tion sources do provide a feasible soluinforma-tion to partially overcome these inconsistencies for such real-world applications. While the experimental results of system performance looked quite promising, much work remains to be done. In our work, geo-graphic objects are represented in the images. Unfortunately, in some cases, a large range of geographic spot may contain various geographic objects. As a result, various geographic objects occurred in same image set might be regarded as a geographic spot sample. It is thus hard to annotate images in such cases. Furthermore, we will try to experiment with other approaches for feature extraction, such as Maximally Stable Extremal Regions (MSER) as our local scale-invariant feature descriptors and add more samples of geographic spots to implement our system. References

Buscaldi, D., Rosso, P., & Peris, P. (2006). Inferring geographical ontologies from multiple resources for geographical information retrieval. In Proceedings of the 3rd GIR workshop, SIGIR 2006.

Cardoso, N., Cruz, D., Chaves, M., & Silva, M. J. (2008). Using geographic signatures as query and document scopes in geographic IR. Advances in Multilingual and Multimodal Information Retrieval, 5152, 802–810.

Carneiro, G., Chan, A. B., Moreno, P. J., & Vasconcelos, N. (2007). Supervised learning of semantic classes for image annotation and retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(3), 394–410.

Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., & Rosen, D. B. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3(5), 698–713.

Csurka, G., Dance, C. R., Fan, L., Willamowski, J., & Bray, C. (2004). Visual categorization with bags of keypoints. In Proceedings workshop on statistical learning in computer vision (pp. 1–22).

Cusano, C., Ciocca, G., & Schettini, R. (2004). Image annotation using SVM. In Proceedings of SPIE (Vol. 5304, pp. 330–338).

Ding, W., Stepinski, T. F., Parmar, R., Jiang, D., & Eick, C. F. (2009). Discovery of feature-based hot spots using supervised clustering. Computers & Geosciences, 35(7), 1508–1516.

Doerr, M., & Papagelis, M. (2007). A method for estimating the precision of placename matching. IEEE Transactions on Knowledge and Data Engineering, 19(8), 1089–1101.

Frey, B. J., & Dueck, B. (2007). Clustering by passing messages between data points. Science, 315(5814), 972–976.

Goh, K. S., Chang, E. Y., & Li, B. (2005). Using one-class and two-class SVMs for multiclass image annotation. IEEE Transactions on Knowledge and Data Engineering, 17(10), 1333–1346.

Goldberg, D. W., Wilson, J. P., & Knoblock, C. A. (2009). Extracting geographic features from the Internet to automatically build detailed reginal gazetteers. International Journal of Geographic Information Science, 23(1), 93–128. Harris, C., & Stephens, M. (1988). A combines corner and edge detection. In Alvey

vision conference (pp. 147–151).

Himmelstein, M. (2005). Local search: The Internet is the yellow pages. IEEE Computer, 26–34.

Jones, R., Hassan, A., & Diaz, F. (2008). Geographic features in web search retrieval. In Proceedings of the 2nd international workshop on geographic information retrieval (pp. 57–58).

Kamvar, M., & Baluja, S. (2006). A large scale study of wireless search behavior: Google mobile search. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 701–709).

Ledwich, L., & Williams, S. (2004). Reduced SIFT features for image retrieval and indoor localization. In Australian conference on robotics and automation. Lee, C. H., Yang, H. C., & Wang, S. H. (2010). A location based text mining method

using ANN for geospatial KDD process. In Proceedings of the 7th international symposium on neural networks (ISNN-2010).

Lindeberg, T. (1996). Edge detection and ridge detection with automatic scale selection. In Proceedings CVPR ‘96, 1996 IEEE computer society conference on computer vision and pattern recognition (pp. 465–470).

Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proceedings of the international conference on computer vision (pp. 1150–1157). Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints.

International Journal of Computer Vision, 60(2), 91–110.

Manjunath, B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8), 837–842.

Manjunath, B. S., Shekhar, C., & Chellappa, R. (1996). A new approach to image feature detection with applications. Pattern Recognition, 29(4), 627–640. McCurley, K. S. (2001). Geospatial mapping and navigation of the web. In

Proceedings of the 10th international conference on world wide web, Hong Kong (pp. 221–229).

Olga, O. (2002). Extracting geographical knowledge from the Internet. In Proceedings of ICDM-AM international workshop on active mining ACDM-AM.

Qi, X., & Han, Y. (2007). Incorporating multiple SVMs for automatically image annotation. Pattern Recognition, 40(2), 728–741.

Qin, L., Zheng, Q., Jiang, S., Huang, Q., & Gao, W. (2006). Unsupervised texture classiﬁcation: Automatically discover and classify texture patterns. In Proceedings of international conference on pattern computing (pp. 433–436). Ros, J., Laurent, C., & Lefebvre, G. (2006). A cascade of unsupervised and supervised

neural networks for natural image classiﬁcation. In Proceedings of the international conference on image and video retrieval (pp. 92–101).

Sallaberry, C., Etcheverry, P., & Marquesuzaa, C. (2006). Information retrieval and visualization based on documents’ geospatial semantics. In Proceedings of international conference on information technology: Research and education, Tel Aviv, Israel (pp. 277–282).

Sanderson, M., & Kohler, J. (2004). Analyzing geographica queries. In Proceedings of the workshop on geographic information retrieval, Shefﬁeld, UK (pp. 25–29). Song, K., Tian, Y., Gao, W., & Huang, T. (2006). Diversifying the image retrieval

results. In Proceedings of the 14th annual ACM international conference on multimedia (MULTIMEDIA-06) (pp. 707–710).

Tomasi, C., & Kanade, T. (1991). Detection and tracking of point features. Technical Report CMU-CS-91-132, Carnegie Mellon University.

Uryupina, O. (2003). Semi-supervised learning of geographical gazetteers from the Internet. In Proceedings of the HLT-NAACL 2003 workshop on analysis of geographic references (pp. 18–25).

Varma, M., & Zisserman, A. (2005). Classifying images of materials: Achieving viewpoint and illumination independence. IEEE Transactions on Knowledge and Data Engineering, 17(10), 1333–1346.

Yang, H. C., & Lee, C. H. (2008). Image semantics discovery from web pages for semantic-based image retrieval using self-organizing maps. Expert Systems with Applications, 34(1), 266–279.

Zhai, C. X., Cohen, W., & Lafferty, J. (2003). Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR-03) (pp. 10–17).

Zong, W., Wu, D., Sun, A., Lim, E. P., & Goh, D. H. L. (2005). On assigning place names to geography related web pages. In Proceedings of the 5th ACM/IEEE-CS joint conference on digital libraries (pp. 354–362).