Mobile merchandise evaluation service using novel information retrieval and image recognition technology

(1)

Mobile merchandise evaluation service using novel information retrieval

and image recognition technology

Chi-Chun Lo

a

_{, Ting-Huan Kuo}

b

_{, Hsu-Yang Kung}

_b,*

_{, Hsiang-Ting Kao}

a

_{, Chi-Hua Chen}

a

_{, Che-I Wu}

b

_,

Ding-Yuan Cheng

a

Institute of Information Management, National Chiao-Tung University, 1001 University Road, Hsinchu, Taiwan 300, Taiwan, ROC

b_{Department of Management Information System, National Pingtung University of Science and Technology, 1, Shuefu Road, Neipu, Pingtung, Taiwan 912, Taiwan, ROC}

a r t i c l e

i n f o

Article history:

Received 27 October 2009

Received in revised form 26 March 2010 Accepted 29 March 2010

Available online 2 April 2010 Keywords:

Information retrieval Region adjacency graph Self-organizing maps

Multiple document summarization Mobile device

a b s t r a c t

Consumers’ purchasing behavior has obviously changed in recent years with developments in social eco-nomics. This change has been evident in the decreased ratio of planned purchases but not in the increase of planned (or spontaneous) purchases. This act of spontaneous or otherwise unplanned purchasing is called ‘‘impulse buying”. However, buying under these conditions costs more money always comes with nega-tive responses, such as complaints and regret. Therefore, we propose and have designed a new merchan-dise recommendation system, the Mobile Merchanmerchan-dise Evaluation Service Platform (MMESP). This is a three-tier system composed of Real-time Merchandise Identifying System (RMIS), Real-time Merchandise Evaluation System (RMES), and Real-time Merchandise Recommendation System (RMRS). With this sys-tem, Mobile Users (MUs) take pictures of merchandise and send them to MMESP, RMIS integrates Region Adjacency Graph (RAG) and Self-Organizing Maps (SOM) to gather information on the merchandise through those photographs, and. RMES and RMRS provide Intelligence Agents (IAs) and Multiple Docu-ment Summarization (MDS) to summarize recommendations on merchandise for MUs, all in real time.

1. Introduction

Consumers’ purchasing behavior has obviously changed in re-cent years with developments in social economics. This change has been evident in the decreased ratio of planned purchases, but not in the increase of unplanned (spontaneous) purchases. This act of spontaneous or otherwise unplanned purchase is so-called ‘‘impulse buying”[2,18,21].

Consumers’ purchasing might be very hurried because of work stress and the lack of leisure time. Furthermore, the purchasing environment, such as others’ recommendations and special dis-counts lead the consumers to purchase on an impulse. Solomon indicates that more than 70% of cosmetics buying decisions are not on the plan and the impulse buyers make their shopping deci-sions very quickly[17]. However, buying under this condition is more expensive, and it always comes with negative responses such as complaint and regret[16].

The traits of impulse buying are: 1. Fast shopping decision making 2. Information deﬁciency

3. Expensive dealing price 4. Below expectations products

Based on all of the above, Mobile Merchandise Evaluation Ser-vice Platform (MMESP) oriented designing principles and functions should include:

1. Identifying the product instantaneously

2. Accessing the product’s information immediately

3. Displaying the product’s recommendations and evaluations from former consumers

4. Providing price comparison and recommendation

Therefore, this research was designed to integrate the Image Identifying System (IIS), Self-Organizing Maps (SOM), Intelligent Agent (IA), and Multiple Document Summarization (MDS) in the information system. Taking make-up and magazine as examples, we provide instant Merchandise Evaluation Service (MES). We de-signed a 3-tier structure of Real-time MMESP, including Real-time Merchandise Identifying System (RMIS), Real-time Merchandise Evaluation System (RMES), and Real-time Merchandise Recom-mendation System (RMRS).

The remainder of the thesis is structured as follows. In Section2

we provide background knowledge through the description of

*Corresponding author. Tel.: +886 8 7703202/7908; fax: +886 8 7740306. E-mail address:[email protected](H.-Y. Kung).

Contents lists available atScienceDirect

Computer Communications

(2)

related technologies, such as the concept of IIS, SOM, IA, and MDS. In Section3we propose the approach of the adaptable MES search combined RMIS, RMES, and RMRS. The complete structure and implementation of MMESP we proposed is explained in Section

4. Finally, conclusion and the future work are given in Section5. 2. Related work

The Mobile Merchandise Evaluating Service Platform (MMESP) is designed to provide (i) real-time merchandise identiﬁcation, (ii) real-time access to related information on the merchandise, (iii) instant browsing the product’s information, and (iv) price com-parison and recommendation. Necessary research background and relevant technology include: (1) Image Identifying System (IIS), (2) Intelligent Agent (IA), and (3) Multiple Document Summarization (MDS).

2.1. Image identifying system (IIS)

As a result of the rapid development of technology, the Internet not only provides a medium for transfer of text data, but also, more importantly, for multimedia. To deal with impulse buying, MMEPS integrates an image identifying system to help users evaluate products via picture taking.

A common feature in many application systems is the key com-ponent needed to do the image identification. Many recognition systems and algorithms have been proposed. There are some tradi-tional methodologies in the image identification research area, such as dynamic programming, elastic matching, minimum dis-tance classifier, string matching by correlation and deformable templates. Recently, some researchers have used genetic algorithm and hidden Markov mode. Today, the image identification tech-niques have made great progress in alphanumeric recognition. Moreover, there exist many improvements in image retrieval plus identification techniques and recognition algorithms. Yang et al.

[22]and Perlibakas[12]used PCA for appearance-based face repre-sentation and recognition. Li and Tang[9]used support vector ma-chines (SVM) to enhance Bayesian recognition. Therefore, we used SOM algorithm to improve the dynamic image retrieval and accel-erate the recognition in this study. Furthermore, we implement the mobile merchandise evaluation service platform to facilitate its use by mobile users.

2.2. Intelligent agent (IA)

The deﬁnition presented here is adapted from Wooldridge and Jennings (Wooldridge, 1995). ‘‘An agent is a computer system that is situated in some environment, and that is capable of autono-mous action in this environment in order to meet its design objec-tives”. The functionality of each intelligent agent is described as follows. (i) Blog Content Retrieval Agent (BCRA), (ii) Merchandise Information Retrieval Agent (MIRA), (iii) Geographical Information Agent (GIA), (iv) User Interface Agent (UIA), (v) Coordinating Agent (CA), and (vi) Resources Monitor Agent (RMA). Six intelligent agents provide mobile clients with the customized information.

The applications with MMESP are shown as follows: 2.2.1. E-commerce

MMESP is user friendly because it can automatically provide judgments and support decision-making from the viewpoint of user’s preferences and requirements.

2.2.2. Information collection and retrieval

The current web information is too complicated, it is also time-consuming and labor-intensive in ﬁnding speciﬁc information for

the user. Inference engine technology is becoming stronger so that data collection and retrieval are more convenient and intuitive. However, due to its inefﬁciency, the search will display accessed information repeatedly, and cannot accurately meet user needs. MMESP combines the agent mechanism to search out the desired information more accurately by storing an index value or the his-torical record of user browsing.

2.2.3. Information ﬁltering

The intelligent agent uses pre-defined filtering rules to filter information in advance in order to reduce the amount of data and extract more useful information.

2.3. Multiple document summarization (MDS)

Blogs have increased greatly in recent years because of the rapid development of computer technology and broader access to the Internet. Users may receive a huge amount of information from blogs through the intelligent agency. However, it is difﬁcult for the user to ﬁlter useless and repeated information. To solve this problem, MMESP uses Multiple Document Summarization (MDS) to simplify and eliminate repeated information, so the user can save search time and derive needed important information.

MMESP is combined with MEAD which is a public domain por-table multi-document summarization system based on Linux. MEAD, whose process is shown inFig. 1, is implemented by Perl programming language[15].

The main procedures of MEAD are:.

(1) Preprocess: The intelligent agent retrieves the contents of the Blog using HTML format to segment sentences in the original document in order to facilitate follow-up by the weight computing. (2) Feature selection: In this paper, MDS is designed to consider several features to compute the weight of each sentence by words and phrases. The main three features are centrality, sentence length, and position[1,13,14,23,24].

(3) Classiﬁer: The score of each sentence is computed as the to-tal weight of each feature[8].

(4) Reranker: The Classiﬁer only functions to score sentence similarity and sorting. This may lead to a problem when there is a high degree of similarity between sentences, especially in mul-ti-document summarization. MEAD designs a Reranker mechanism to recalculate sentences with syntactic similarity and set the threshold to ﬁlter out important sentences to reduce the

(3)

dancy ratio. Finally, a summary value is derived by extracting sen-tences from original document by the compression ratio[3].

(5) Summarization: Summarization retrieves and recombines words and phrases in the original document according to the rank-ing of the sentences by Reranker sortrank-ing.

(6) Evaluation: MMESP measures the performance of the text summarization system, including the effect of output results as well as users’ satisfaction[11].

3. System structure

Mobile Merchandise Evaluate Service Platform (MMESP) inte-grates image recognition and relevant Information Retrieval tech-niques. It provides (1) thematic inferences, (2) merchandise optimization inferences, and (3) multi-document summarization, to recommend merchandise suitability to the user. The system de-sign principles are followed. The MMESP which is dede-signed with three-tier architecture is published to open platforms (e.g. Face-book, Google, Yahoo, and other application stores). As shown in

Fig. 2, MMESP includes Real-time Merchandise Identifying System (RMIS), Merchandise Evaluation System (MES), and Merchandise Recommendation System (MRS). Users can use any kind of mobile device, such as 3G cell-phone, personal digital assistant (PDA), lap-top, or tablet pc to access the merchandise evaluate service through application or browser.

MMESP includes intelligence agents, image identifying system, self-organized neural network, multiple document summarization, and merchandise recommendation system. The recommendation results would be presented on the hand set device, and a mobile user can receive the merchandise evaluation service supported by software and hardware in a wireless communication environ-ment. MMESP is based on the 3G cell-phone platform providing

wireless communication for a mobile user designed by Java. Users connect to a World Wide Web (WWW) server which is developed by Java Server Pages (JSP). System manager could build and man-age the system in the back-end side.

3.1. Real-time merchandise identifying system (RMIS)

RMIS is at the front end of internet, it mainly processes multi-media and provides services such as image recognition, which in-cludes Real-time Multimedia Transmission (RMT) and Image Identifying system (IIS).

When users rush to buy something, they can use cell-phone to take pictures of the product and send them through J2ME tech-nique (MIDP2.0) via mobile communicate internet (3G or IEEE802.11b) to the server. RMIS will integrate the Region Adja-cency Graph (RAG) and Self-Organizing Maps (SOM) as soon as it receives the image. At the same time, it processes and classiﬁes the image in order to recognize the merchandise. The ﬂow path is shown inFig. 3.

3.1.1. Real-time multimedia transmission (RMT)

Text and pictures are transmissible media data. The user can use 3G cell-phone or PDA to take pictures of the product they are interested in. These pictures will be sent back to the back-end RMIS for image recognition, to provide real-time and convenient mer-chandise evaluation service.

3.1.2. Image identifying system (IIS)

RMIS integrates the Region Adjacency Graph (RAG) and Self-Organizing Maps (SOM) techniques to process the multimedia image. The procedures are (i) transformed to the RGB matrix (ii) fuzzy process (iii) color degrade (iv) histogram (v) region and

Fig. 2. System structure.

(4)

adjacent relation (vi) region merging (vii) Region Adjacency Graph (RAG) (viii) Self-Organizing Maps (SOM) and (ix) similarity determination.

First, RAG calculates the merchandise pictures and derives the merchandise characteristics, then puts them into SOM to train the Artiﬁcial Neural Network. When the system gets the image, it can compare and classify the image and recognize the merchandise in order to get the relevant information to do the commenting and recommendation.

This research is based on Java, and integrates image processing via Region Adjacency Graph (RAG) technology to prevent the image identiﬁcation from being affected by the rotation, enlargement, or condensation of the image or the partial sub-image. The main pro-cesses are as follows[7].

(1) Transforming to RGB matrix: First, the input image I is trans-formed into RGB matrix to execute the calculation. For example, there is a cosmetic product ‘‘Clinique Moisture Surge Extended Thirst Relief” shown inFig. 4.

(2) Fuzzy process: The input image I is smoothed by a linear, typically Gaussian, filter. This linear filtering eliminates any pla-teaus, defined as uniform regions, by converting the image to a floating point representation with a gradient magnitude operator and yields the gradient magnitude image (GMI). The numerous gradient operators of GMI can be utilized as follows:

IGMI¼k

r

ðG Ið ÞÞk ð1Þ

where G() andr() are the Gaussian ﬁltering operation and gradi-ent operator, respectively, and kk denotes magnitude[6]. The GMI, shown inFig. 5can be viewed as a geographical surface where the largest gradient magnitude values are interpreted as watershed lines, and the regions separated by the watersheds as catchment

ba-sins. Thus, as a physical analogy, any water falling on one side of a watershed line will ﬂow downhill until it reaches a local intensity minimum (LIM). To formally deﬁne the LIMs, let ig(n) 2 IGMIbe a

sample in the GMI image at index n = [n1,n2], and deﬁne the N N

neighborhood (N odd) of samples centered at n as follow[5]. IN n¼ igðmÞ 2 IGMI:m 2

X

Nn n o ð2Þ where

X

N n¼ m 2 Z2 : jn1 m1j 6ðN1Þ2 ;jn2 m26ðN1Þ2 ( ) ð3Þ Then a sample ig(n) 2 IGMIis a LIM if

igðnÞ < igðmÞ

8

igðmÞ 2 I3n; m–n ð4Þ Fig. 6shows the example image as processed by GMI.

(3) Color reducing: In this step, it reduces the processed image’s colors from RGB value. For example, it can reduce the number of colors from 256 to 4 by dividing into 64, and the number of colors will diminish from 2563_{= 16,777,216 to 4}3_{= 64. The example}

im-age of reduced color is shown asFig. 7.

(4) Histogram making: It uses a histogram to display frequen-cies of pixels of 64 colors and examine the image process for failure or inaccuracy.

(5) Region adjacency relations: The adjacent points with the same color are brought into the same segments.

(6) Segment merging: In this step, we merge the ﬁxed ratio seg-ments with the larger segseg-ments around them because the fracture

Fig. 4. The input image.

Fig. 5. The presentation of GMI (N = 3).

Fig. 6. The image as processed by GMI.

(5)

of segments might reduce the processing speed.Fig. 8shows the example image after merging segments.

(7) Region adjacency graph (RAG) computing: After the above processes, it selects the top k large segments, which contain the colors, the measure of area, shapes, and the number of adjacent segments’ into a two-dimensional array, and stores the k segments’ neighborhoods in to RAG.Tables 1 and 2show the results of exam-ple imaging after RAG computing.

(8) Experimental self-organizing maps (SOM): It projects the different RAG data into SOM and classiﬁes the products’ character-istic marks derived from SOM technique.

(9) Determining the similarity: Finally, we determine the simi-larity between the new incoming image and the classiﬁcations that have resulted from SOM algorithm. This research determines the similarity with Cosine Theorem, and determines the similar prod-ucts for later computing service.

3.2. Real-time merchandise evaluation system (RMES)

RMES provides functions including Blog Content Retrieval Agent (BCRA) and Automatic Multiple Document Summarization (AMDS) and so on.

When the RMIS ﬁnishes the merchandise recognition, the sys-tem transmits the merchandise information to the RMES. BCRA will search the relevant blog content through the Google Blog Search, and save the Crawler and Parser as the Blog Corpus. Finally, the Multiple Document Summarization technique will retrieve the relevant Blog Corpus from the merchandise comment and summa-rize if for users in order to make their buying decision. The ﬂow path is showed inFig. 9.

3.2.1. Blog content retrieval agent (BCRA)

Blog Content Retrieval Agent (BCRA) provides functions such as Fuzzy Search, HTML Crawler, and HTML Parser; the details of the functions are as follows:

(1) Fuzzy search: It provides fuzzy arithmetic and judgment to build the relevant key word vocabulary, and it actively uses the key words to search through Google Blog Search.

(2) HTML crawler: It sends the content of blog after the Google Blog Search is ﬁnished, it also traces the relevant links and stores the HTML content.

(3) HTML parser: It interprets the HTML with HTML tag from HTML Crawler and derives the information. It also effectively de-letes the relevant special characters (such as single quotation mark and double quotation mark) to avoids SQL injection attack, and it builds the Blog Corpus in order to do the multiple document sum-marization inference.

3.2.2. Automatic multiple document summarization (AMDS) RMES combines the AMDS by instantaneously summarizing the blogs’ relevant merchandise comments. It efﬁciently reduces the amount of information so the users can conveniently view the comment and experience from the previous buyers AMDS is done by the MEAD package, which puts the relevant merchandise com-ment from Blog Corpus based on MEAD module. The processing procedure is (I) Predeﬁne, (II) Preprocess, (III) Feature Selection,

Fig. 8. Segments.

Table 1

The RAG result of example image contains k segment and average RGB values.

Target Value Segments 1419 Average red 2.196912 Average green 2.025552 Average blue 2.045424 Table 2

The RAG results of segmentation contains the shape and RGB when the number of the largest segments (k) is set as 5.

Segment No. Shape ¼ _{ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi}Perimeter The measure of area

p R G B 1 7.528407 3 3 3 2 7.785796 0 0 0 3 16.14663 3 3 3 4 7.850506 3 3 3 5 7.528407 3 3 3

(6)

(IV) Classiﬁer, (V) Reranker, and (VI) Summary. The details are as follows:

(1) Predeﬁne: In this section, we predeﬁne and establish the Merchandise Comment Term Ontology (MCTO) which contains Merchandise Comment Term Set (MCTS) and Relevant Comment Term Set (RCTS) based on Natural Language Processing (NLP) and knowledge representation techniques for automatic summariza-tion[20].Fig. 10shows the procedure for MCTO establishment.

I. Merchandise comment term set (MCTS): First, BCRA crawls and parses many merchandise blog articles from Google blog search engine (path (1) inFig. 10). These articles are sent to several domain experts to generate the summaries as training data for MCTS establishment. Afterward BCRA uses the Stanford Log-linear Part-Of-Speech (POS) Tagger library[19]to analyze the POS (e.g. noun, verb, adjective, and adverb) from the experts’ summaries (path (2) inFig. 10). In this paper, we select noun terms and adjec-tive terms to establish the MCTS (path (3) inFig. 10).

II. Relevant comment term set (RCTS): For RCTS establishment, we use WordNet library[10]to ﬁnd out the hypernyms and hyp-onyms of the noun terms and the synhyp-onyms and anthyp-onyms of the adjective terms according to MCTS (path (4) inFig. 10), respec-tively. Finally, the MCTS is extended the relevant comment terms to establish RCTS (path (5) inFig. 10).

(2) Preprocess: The preprocess transfers the format of original HTML documents from blog. Then it sets the document ID and Sen-tence ID sequentially in order to maintain the weight of senSen-tences in each document and production.

(3) Feature selected: After that, MMESP uses several features, (i) Centrality, (ii) Sentence Length, and (iii) Position to calculate the weight of each sentence.

I. Centrality: We use the Vector Space Model (VSM) to carry out the similarity calculation. In this paper, the keywords which are contained by RCTS only will be considered. The maximum cosine value represents the centroid vector of the cluster. The following expression is used to calculate the value of the sentence s. ScoreCentralityðsÞ

¼jkeywords in s \ keywords in other sentencesj keywords in s [ keywords in other sentences

j j

8

keywords 2 RCTS ð5Þ

II. Sentence length: If the length of the sentence is above a given threshold, it is 1. Otherwise, the sentence length is 0. The following expression is used to calculate the value of the sentence length.

ScoreLengthðsÞ ¼

1; LengthðsÞ > n 0; LengthðsÞ 6 n

ð6Þ III. Position: Position is to assign weight by the position of the sentence in the document. For this purpose the weight is divided into 10 levels: 0–9. 0: the sentence does not belong to a summary;

1–9: the sentence belongs to a summary. The importance: 1 is weakest, and 9 is strongest. The following expression used to cal-culate the value of the position of the sentence in the document. ScorePositionðsÞ ¼ Pðs 2 SjPositioniÞ

A

v

erage of Positioni

9:0 ð7Þ

(4) Classiﬁer: In the third step, we select some important fea-tures to set the different weight according to those feafea-tures. We summarize those features and their weight to calculate the score of each sentence.

If Centrality weight is w1, Position weight is w2. The expression

is as follows:

ScoreOverallðsÞ ¼ ½w1 ScoreCentralityðsÞ þ w2 ScorePositionðsÞ

ScoreLengthðsÞ ð8Þ

(5) Reranker: In this step, we judge the correlation in sentences which will decrease the redundancy. Next, we set the threshold for ﬁltering, and set the compression to extract.

(6) Summary: After that, we get the extract from Reranker and map the extract to summarize from Document ID and Sentence ID in this step of preprocessing. Finally, Multiple Document Sum-marization technology provides a summary of comments for users.

3.3. Real-time merchandise recommendation system (RMRS) RMRS provides functions that include Merchandise Information Retrieval Agent (MIRA) and Price Comparing Recommendation Mechanism (PCRM).

When RMIS ﬁnishes the recognition, it sends the data informa-tion to the RMRS, and it searches the price from the price compar-ing websites and the relevant website. Accordcompar-ing to the merchandise price information from a different e-commerce web-site, it saves the Crawler and Parser as the merchandise price cor-pus. Finally, it extracts the price information from the relevant merchandise price Corpus through the price comparing mecha-nism and recommends the better merchandise after price compar-ison. The user can learn the merchandise’s reasonable price, avoiding wasting money. The RMRS procedure is shown asFig. 11.

3.3.1. Merchandise information retrieval agent (MIRA)

Merchandise Information Retrieval Agent (MIRA) provides func-tions such as Fuzzy Search, HTML Crawler, and HTML Parser, the details of which are as follows:

(1) Fuzzy search: This provides fuzzy arithmetic and judgment to build the relevant key word vocabulary, and it actively uses the key word vocabulary in searching through Google Blog Search. (2) HTML crawler: It sends the content of blog after ﬁnishing the Google Blog Search., It also traces the relevant links and stores the HTML content.

(3) HTML parser: It interprets the HTML with HTML tag from HTML Crawler and derives the information. It also effectively de-letes the relevant special characters (such as single quotation mark and double quotation mark) to avoid SQL injection attack and it builds the Blog Corpus in order to accomplish the multiple docu-ment summarization inference.

3.3.2. Price comparing recommendation mechanism (PCRM)

RMRS provides Price Comparing Recommendation Mechanism (PCRM), which compares the price from different e-commerce websites through Merchandise Price Information Extracting Agent. It is a price-oriented recommendation so that the user can under-stand the price spectrum, determine the cheapest merchandise, and decide if he/she will to buy it.

Fig. 10. The procedure for Merchandise Comment Term Ontology (MCTO) establishment.

(7)

4. System evaluation and implementation

We report our experimental results and implement the archi-tecture and approaches to merchandise evaluation service as an example in this section.

4.1. Evaluation

In experiments, we evaluate the merchandise identifying time, merchandise identifying accuracy, and merchandise evaluation accuracy in the following results.

4.1.1. Merchandise identifying time

The length of merchandise identifying time is according to the size of resolution. If the image size is bigger, the merchandise iden-tifying time will be longer by IIS. Therefore, we compare the mer-chandise identifying times with different common formats of resolution (e.g. 240 320, 480 320, 640 480, 800 480, 800 600, 1024 768, and 1280 960). In experiments, the test-ing is performed 100 times per each format of resolution.Fig. 12

shows the average merchandise identifying times are 0.38 s and 5.46 s in 240 320 cases and 1280 960 cases, respectively. 4.1.2. Merchandise identifying accuracy

There are m images for n kinds of merchandise (such as mer-chandise image 1, mermer-chandise image 2, . . ., mermer-chandise image m) in the domain of cosmetics, and those data are collected from several web sites or vendors. We measure the performance of our approach in the way called k-fold cross-validation [4]. In experiments, we focus on the moisture creams (e.g. Clinique Moisture Surge Extended Thirst Relief) and select 240 images for

24 kinds of merchandise. The training and testing are performed 240 times (i.e., n = 24, m = 240). In iteration i, merchandise image i is selected as the test corpus, and the other merchandise images are collectively used to train the values for each kind of merchandise.

In these experiments, the feasibility of applying Gradient Mag-nitude Image (GMI), Region Adjacency Graph (RAG), k-Nearest Neighbor (kNN), or Self-Organizing Maps (SOM) to merchandise image identifying is evaluated.Table 3shows image identification by kNN or SOM combined with GMI and RAG. Consider kNN first; it can be observed that its classification performances are 22.08%, 36.67%, 43.33%, 47.92%, and 48.33% when the different numbers of the largest segments (k) are from 5 to 9. Second, we consider SOM to implement and evaluate that its classification perfor-mances are 76.25%, 78.33%, 80.83%, 81.25%, and 81.25%. The result of SOM algorithm combined GMI and RAG to improve accuracy rates is shown inFig. 13. Therefore, we set the number of the larg-est segments as 8 and apply SOM with GMI and RAG in MMESP.

Fig. 11. The process of RMRS.

Fig. 12. The result of the merchandise identifying time with the different common formats of resolution.

Table 3

The result of GMI, RAG, kNN, or SOM combination with the different numbers of the largest segments.

The number of the largest segments (k)

GMI + RAG + kNN (%)

GMI + RAG + SOM (%) 5 22.08 76.25 6 36.67 78.33 7 43.33 80.83 8 47.92 81.25 9 48.33 81.25

Fig. 13. The result of GMI, RAG, kNN, or SOM combination with the different numbers of the largest segments.

(8)

4.1.3. Merchandise evaluation accuracy

In this experiment, we randomly select 12 articles from Google blog search engine. These articles are sent to 4 domain experts for summarization. Moreover, we evaluate the performance of AMDS in the way called k-fold cross-validation[4]. In iteration j, blog arti-cle j is selected as the test corpus, and the other 11 blog artiarti-cles are collectively used to train the values by the method proposed in Section3.2.2.Table 4shows the results of which the training and testing are performed in 12 runs. About 78% summaries which match with experts’ summaries are enough to help users to under-stand the merchandise. Therefore, the result of the experiment shows that AMDS can be successfully applied to automatic summarization.

4.2. System implementation and case study

Step 1: When a user would like to buy a particular product, he (or she) can photograph the product by cell-phone. Step 2: After taking pictures, he (or she) can upload the photos to

MMESP, and connect to MMESP, as shown asFig. 14a–c with different versions. Click the browse button of to choose the picture.

Step 3: After receiving the photo, RMIS would automatically identify the merchandise, and then combine the two methods of Region Adjacency Graph (RAG) and Self-Organizing Maps (SOM) to process and classify the images. While identifying the mapping merchandise, MMESP would list the related information of the mer-chandise to users as shown inFig. 15.

Step 4: RMES accesses the judgment and use experience infor-mation from Blog Corpus.

Step 5: Multiple document summarization technique would retrieve the recommendation to process the summariza-tion of content, and send the result to the cell-phone of user shown as shown inFig. 16.

Step 6: RMES retrieves the price of the merchandise from all the e-commerce web sites.

Step 7: Price comparison and recommendation mechanism would list the price information. The same product may have different prices at several price comparison web-sites, which provides the merchandise price enquiry shown inFig. 17. Users could learn the range of price information by querying the price to evaluate whether they wish to buy the product or not.

Step 8: User clicks the photo, which would list information on the merchandise in detail.

Step 9: After RMIS carries out the identiﬁcation process, it would send the merchandise information to RMRS. Next, the merchandise price information retrieval agent would not only search the price of related products on a price comparison website and Internet, but also crawl and parse the price information into Corpus.

Fig. 14. (a) Real-time merchandise identification (mobile phone version). (b) Real-time merchandise identification (Facebook version). (c) Real-time merchandise identification (Google Gadget version).

Fig. 15. Real-time merchandise identiﬁcation.

Table 4

The result of automatic summarization accuracy. Total expert’s

sentences

Sentences match expert’s sentences Accuracy rate (%) Expert 1 15 11 73 Expert 2 9 6 67 Expert 3 5 5 100 Expert 4 16 13 81 Total 45 35 78

(9)

Step 10: Finally, it activates the price comparison recommenda-tion mechanism to retrieve the price informarecommenda-tion in Cor-pus, as shown inFig. 18. RMRS would compare the price and recommend merchandise to the users, who can see the range of price and avoid wasting money.

5. Conclusions and future work

Because Impulse Buying is unplanned and happens quickly, the buying decision is always done without thinking and is easily inﬂu-enced by the surrounding environment. MMESP can provide mo-bile transmission service, so that users can take pictures of the merchandise from their cell-phone and send the pictures to the back-end server to activate real-time calculation. MMESP will send back the merchandise’s recognition information, consumer’s com-ment and relevant merchandise price comparison to the user for a buying decision.

Through a portable device (such as 3G cell-phone and PDA) and a wireless communication network environment (3G or IEEE 802.11b), users do not need machine major computer processor but can derive useful information, such as merchandise evaluation service, from MMESP while shopping.

Acknowledgements

The research is supported by the National Science Council of Taiwan under the Grant Nos. NSC 96-2416-H-009-008-MY3, NSC 98-2220-E-020-001, and NSC 98-2220-E-020-002.

References

[1] C.N. Chang, S.Y. Chen, J.Y. Yeh, M.J. Hwang, H.R. Ke, W.P.Yang, A concept extraction approach for document clustering, in: Proceedings of 2008 International Conference on Advanced Information Technologies, Taiwan, 2008.

[2] C.J. Cobb, W.D. Hoyer, Planned versus impulse purchase behavior, Journal of Retailing 62 (4) (1986) 384–409.

[3] G. Erkan, D.R. Radev, LexPageRank: prestige in multi-document text summarization, in: Proceedings of EMNLP 2004, Barcelona, Spain, 2004, pp. 365–371.

[4] J. Han, M. Kamber, Data Mining: Concepts and Techniques, second ed., Morgan Kaufman, Los Altos, CA, 2005.

[5] K. Haris, S. Efstratiadis, N. Maglaveras, A. Katsaggelos, Hybrid image segmentation using watersheds and fast region merging, IEEE Transactions on Image Process 7 (12) (1998) 1684–1699.

[6] S.E. Hernandez, K.E. Barner, Y. Yuan, Region merging using homogeneity and edge integrity for watershed-based image segmentation, Optical Engineering 44 (1) (2005) 1–14.

[7] Y.L. Huang, J.P. Lee, G.L. Chen, Z.Z. Lee, M.P. Chang, X.J. Huang, Image retrieval by using region adjacency relations, Tunghai Science 6 (2004) 17–41. [8] K. Kaikhah, Text summarization using neural networks, WSEAS Transactions

on Systems 3 (2) (2004) 960–963.

[9] Z. Li, X. Tang, Using support vector machines to enhance the performance of Bayesian face recognition, IEEE Transactions on Information Forensics and Security 2 (2) (2007) 174–180.

[10] G.A. Miller, WordNet: a lexical database for English, Communications of the ACM 38 (11) (1995) 39–41.

[11] S.H. Myaeng, D. Jang, Development and evaluation of a statistically based document system, in: I. Mani, M. May Bury (Eds.), Advances in Automated Text Summarization, MIT Press, Cambridge, MA, 1999, pp. 61–70.

[12] V. Perlibakas, Distance measures for PCA-based face recognition, Pattern Recognition Letters 25 (6) (2004) 711–724.

[13] D.R. Radev, A. Winkel, M. Topper, Multi document centroid-based text summarization, in: Processing of ACL 2002, Philadelphia, PA, 2002. [14] D.R. Radev, H. Jing, M. Budzikowska, Centroid-based summarization of

multiple documents, Information Processing and Management 40 (6) (2004) 919–938.

[15] D.R. Radev, T. Allison, S. Blair-Goldensohn, J. Blitzer, A.C. Elebi, S. Dimitrov, E. Drabek, A. Hakim, W. Lam, D. Liu, J. Otterbacher, H. Qi, H. Saggion, S. Teufel, M. Topper, A. Winkel, Z. Zhu, MEAD – a platform for multidocument multilingual text summarization, in: Proceedings of LREC 2004, Lisbon, Portugal, May 2004. [16] D.W. Rook, The buying impulse, Journal of Consumer Research 14 (2) (1987)

189–199.

[17] M.R. Solomon, Consumer Behavior: Buying, Having, and Being, ﬁfth ed., Prentice-Hall, Englewood Cliffs, NJ, 2007.

[18] H. Stern, The signiﬁcance of impulse buying today, Journal of Marketing 26 (2) (1962) 59–62.

[19] K. Toutanova, C.D. Manning, Enriching the knowledge sources used in a maximum entropy part-of-speech tagger, in: Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-2000), Hong Kong, 2000.

[20] H.C. Wang, C.T. Liu, T.H. Huang, A concept-level ontology construction method for automatic summarization, Journal of Internet Technology 8 (4) (2007) 381– 387.

[21] P. Weinberg, W. Gottwald, Impulsive consumer buying as a result of emotions, Journal of Business Research 10 (1) (1982) 43–57.

[22] J. Yang, D. Zhang, A.F. Frangi, J.Y. Yang, Two-dimensional PCA: a new approach to appearance-based face representation and recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 26 (1) (2004) 131–137.

[23] J.Y. Yeh, A study on automated text summarization and its application on Chinese documents, M.S. Thesis, Department of Computer and Information Science, National Chiao-Tung University, Hsinchu, Taiwan, 2002.

[24] J.Y. Yeh, H.R. Ke, W.P. Yang, I.H. Meng, Text summarization using a trainable summarizer and latent semantic analysis, Information Processing and Management 41 (1) (2005) 75–95.

Fig. 18. Merchandise price comparison. Fig. 17. Merchandise recommendation system.