Results and Discussions - 跨語言跨媒體資訊檢索研究(2/2)

Table 1 shows experimental results of official runs in ImageCLEFphoto2006. We compare performance of the runs using textual query only, and the runs using both textual and visual queries

(i.e., Text Only vs. Text + Annotation and Text + Ontology). In addition, we also compare the runs using word-image ontology and the runs using annotated image corpus (i.e., Text + Ontology vs. Text + Annotation). The runs whose performance is better than that of baseline (i.e., Text Only) will be marked in bold. The results show all runs using annotated image corpus are better than the baseline. In contrast, only two runs using word-image ontology are better.

Table 1. Performance of Official Runs (T=text only, A=annotated

image corpus, O=word-image ontology)

Query Language

MAP Description Runs

Portuguese 0.1630 T NTU-PT-EN-AUTO-NOFB-TXT

0.2854

T+A

NTU-PT-EN-AUTO-FB-TXTIMG-T-WEprf

0.1580 T+O

NTU-PT-EN-AUTO- NOFB-TXTIMG-T-IOntology

Russian 0.1630 T

NTU-RU-EN-AUTO-NOFB-TXT

0.2789

T+A

NTU-RU-EN-AUTO-FB-TXTIMG-T-Weprf

0.1591 T+O

NTU-RU-EN-AUTO- NOFB-TXTIMG-T-IOntology

Spanish 0.1595 T

NTU-ES-EN-AUTO-NOFB-TXT

0.2775

T+A

NTU-ES-EN-AUTO-FB-TXTIMG-T-Weprf

0.1554 T+O

NTU-ES-EN-AUTO- NOFB-TXTIMG-T-IOntology

French 0.1548 T

NTU-FR-EN-AUTO-NOFB-TXT

0.2758

T+A

NTU-FR-EN-AUTO-FB-TXTIMG-T-WEprf

0.1525 T+O

NTU-FR-EN-AUTO-Cross-Language Image Retrieval 23

NOFB-TXTIMG-T-IOntology

Simplified 0.1248 T NTU-ZHS-EN-AUTO-NOFB-TXT

Chinese 0.2715 T+A NTU-ZHS-EN-AUTO-FB-TXTIMG-T-Weprf

0.1262

T+O

NTU-ZHS-EN-AUTO- NOFB-TXTIMG-T-IOntology

Japanese 0.1431 T NTU-JA-EN-AUTO-NOFB-TXT

0.2705

T+A

NTU-JA-EN-AUTO-FB-TXTIMG-T-Weprf

0.1396 T+O

NTU-JA-EN-AUTO-NOFB-TXTIMG-T-IOntology Traditional 0.1228 T

NTU-ZHT-EN-AUTO-NOFB-TXT

Chinese 0.2700 T+A NTU-ZHT-EN-AUTO-FB-TXTIMG-T-Weprf

0.1239

T+O

NTU-ZHT-EN-AUTO- NOFB-TXTIMG-T-IOntology

Italian 0.1340 T NTU-IT-EN-AUTO-NOFB-TXT

0.2616

T+A

NTU-IT-EN-AUTO-FB-TXTIMG-T-Weprf

0.1287 T+O

NTU-IT-EN-AUTO-NOFB-TXTIMG-T-IOntology

The reason why the word-image ontology does not perform as our expectation may be that the images in the word-image ontology come from the web and the images in the web still contain much noise even after filtering. To deal with this problem, a better method in the image filtering is indispensable.

Since the example images in this task are in the image collection, the CBIR system always correctly maps the example images into themselves at mapping step. We made some extra experiments to

examine the performance of our intermedia approach. In the experiments, we took out the example images from the image collection when mapping example images into intermedia. Table 2 shows the experiment results. Comparing Table 1 and Table 2 we find the performance of Table 2 is lower than that of Table 1. It shows the performance of CBIR in mapping stage will influence the final result and that is very critical. From Table 2, we also find that the approaches of annotated image corpus are better than the runs using textual query only. It shows even mapping stage have some errors, the annotated image corpus can still work well.

Table 2. MAP of Runs by Removing Example Images from the

Collection

Query Language

Text Only Text + Annotated image corpus

Portuguese 0.1630

0.1992

Russian 0.1630

0.1880

Spanish 0.1595

0.1928

French 0.1548

0.1848

Simplified Chinese

0.1248

0.1779

Japanese 0.1431

0.1702

Traditional Chinese

0.1228

0.1757

Italian 0.1340

0.1694

Table 3 shows the experiment results of monolingual runs. Using both textual and visual queries are still better than runs using textual query only. The performance of the runs by taking out the example images from collection beforehand is still better than the runs use textual query only. From this table, we also find the runs using textual query only does not perform well even in monolingual runs. This may be because the image captions of this year are shorter and we do not have enough information when we use textual information only. In addition, when image captions are short too, the little differences in vocabularies between query and document may influence the results a lot. Therefore, German monolingual run and English monolingual run perform so different.

Cross-Language Image Retrieval 25

Table 4 shows the experiment of runs that using visual query and annotated image corpus only, i.e., the textual query is not used. When example images were kept in the image collection, we can always map the example images into the right images. Therefore, the translation from visual information into textual information will be more correctly.

The experiment shows the performance of visual query runs is better than that of textual query runs when the transformation is correct.

Table 3. Performance of Monolingual Image Retrieval

Query

Language

MAP Description Runs

English 0.1787 T

NTU-EN-EN-AUTO-NOFB-TXT

German 0.1294 T

NTU-DE-DE-AUTO-NOFB-TXT (+example

images)

0.3109

T+A NTU-DE-DE-AUTO-FB-TXTIMG

(-example images)

0.1608

T+A NTU-DE-DE-AUTO-FB-TXTIMG-NoE

Table 4. Performance of Visual Query

MAP Description Runs 0.1787 T (monolingual)

NTU-EN-EN-AUTO-NOFB-TXT

0.2757

V+A (+example images)

NTU-AUTO-FB-TXTIMG-Weprf 0.1174 V+A (-example images)

NTU-AUTO-FB-TXTIMG-Weprf-NoE

6. Conclusion

The experiments show visual query and intermedia approaches are useful. Comparing the runs using textual query only with the runs merging textual query and visual query, the latter improved 71%~119%

of performance of the former. Even in the situation which example images are removed from the image collection, the performance can still be improved about 21%~43%. We find visual query in image retrieval is important. The performance of the runs using visual query only can be even better than the runs using textual only if we translate visual information into textual one correctly. In this year the word-image ontology built automatically still contain much noise. We will investigate how to filter out the noise and explore different methods.

References

1. Besançon, R., Hède, P., Moellic, P.A., & Fluhr, C. (2005). Cross-media feedback strategies: Merging text and image information to improve image retrieval. In Peters, C.; Clough, P.; Gonzalo, J.;

Jones, G.J.F.; Kluck, M.; Magnini, B. (Eds.), Proceedings of 5th

Workshop of the Cross-Language Evaluation Forum, LNCS 3491,

(pp. 709-717). Berlin: Springer.

2. Clough, P., Sanderson, M. & Müller, H. (2005). The CLEF 2004 cross language image retrieval track. In Peters, C.; Clough, P.;

Gonzalo, J.; Jones, G.J.F.; Kluck, M.; Magnini, B. (Eds.),

Proceedings of 5th Workshop of the Cross-Language Evaluation Forum, LNCS 3491, (pp. 597-613). Berlin: Springer.

3. Clough, P., Müller, H., Deselaers, T., Grubinger, M., Lehmann, T.M., Jensen, J., & Hersh, W. (2006). The CLEF 2005 cross-language image retrieval track, Proceedings of 6^th

Workshop of the Cross Language Evaluation Forum, Lecture Notes in Computer

Science, 2006.

4. Jones, G.J.F., Groves, D., Khasin, A., Lam-Adesina, A., Mellebeek, B., & Way, A. (2005). Dublin City University at CLEF 2004:

Experiments with the ImageCLEF St Andrew's Collection. In Peters, C.; Clough, P.; Gonzalo, J.; Jones, G.J.F.; Kluck, M.;

Magnini, B. (Eds.), Proceedings of 5th Workshop of the

Cross-Cross-Language Image Retrieval 27

Language Evaluation Forum, LNCS 3491, (pp. 653-663). Berlin:

Springer.

5. Lin, W.C., Chang, Y.C. and Chen, H.H. (forthcoming). “Integrating Textual and Visual Information for Cross-Language Image Retrieval: A Trans-Media Dictionary Approach.” Information

Processing and Management, Special Issue on Asia Information

Retrieval Research.

6. Zinger, S. (2005). “Extracting an Ontology of Portable Objects from WordNet.” Proceedings of the MUSCLE/ImageCLEF

Workshop on Image and Video Retrieval Evaluation.

96 年 5 月 21 日報告人姓名

陳信希服務機構

及職稱

國立台灣大學資訊工程學系教授

時間會議地點

96 年 5 月 14 日-5 月 19 日日本東京

本會核定補助文號

NSC 95-2221-E-002-334

會議名稱

(中文)

(英文) The 6th NTCIR Workshop on Evaluation of Information Access Technologies

發表論文題目

(英文)

(1) Overview of CLIR Task at the Sixth NTCIR Workshop

(2) Overview of the NTCIR-6 Cross-Lingual Question Answering (CLQA) Task (3) Overview of Opinion Analysis Pilot Task at NTCIR-6

(4) Using Opinion Scores of Words for Sentence-Level Opinion Extraction

附件三

表 Y04

一、參加會議經過

The 6th NTCIR Workshop Meeting on Evaluation of Information Access Technologies (第六屆 NTCIR 資訊存取評估會議，NTCIR-6)，5 月 15 日-5 月 18 日在日本東京 National Institute of Informatics 大會堂舉行，會議前後(5 月 14 日和 5 月 19 日)也舉行 NTCIR-7 圓桌會議，

討論 NTCIR-7 相關事宜。筆者於 5 月 13 日搭乘中華航空 CI 104 班機，抵達東京，5 月 20 日 NTCIR-6 國際會議結束後，搭乘中華航空 CI 101 班機返回台北。

二、與會心得

EVIA 2007 (1^st International Workshop on Evaluating Information Access)是NTCIR-6 會前會，本次會議邀請亞洲中日韓語之外的語言檢索研究人員參與，探討簡體中文資訊檢索評比、越南文件檢索、印度文資訊檢索評比規劃、和泰文搜索引擎評比，以及新的評估方法研究。

正式議程包括 QAC, CLIR, CLQA, PATENT, Opinion, 和 MuST 等六個評比項目的整體報告(評比過程、使用測試集、參與的研究團隊、採用的方法、和效能分析)，以及參與評比研究團隊的系統和技術報告。此外，會議單位也邀請另外兩個資訊檢索國際評比：

TREC 和 CLEF 做專題報告，以經驗分享。會議中也舉行數次圓桌會議，針對評比內容、

程序、評估方式等方面，由 task organizers 和 task participants 進行面對面交流，以作為下次會議的參考。

下年度共有 8 個規劃提案：Complex CLQA (CCLQA)、CLIR For Blog (CLIRB)、

Multilingual Opinion Analysis (MOAT)、Multimodal Summarization for Trend Information (MuST)、Patent Processing (translation, mining) (PAT)、Question Answering Challenge (QAC5)、Simplified Chinese IR (CLIR-SC)、和 User Satisfaction Task (USAT)中，在 PC meeting 中，選出 Complex CLQA、Multilingual Opinion Analysis (MOAT)、和 Patent Processing (PAT)，作為 NTCIR-7 的評比項目。其中 Multilingual Opinion Analysis (MOAT) 為筆者、Yohei Seki、和 David Kirk Evans 所合作提出。

比較可惜的是舉辦數年的 CLIR 評比，下年度不再為獨立項目，而是併入到 MOAT 中。

主要的原因之ㄧ是：CLIR 近年來在技術層面上，並沒有明顯的進展，雖然我們擬引進部落格語料，但是語料的特徵並沒有很突顯出來。另一個可惜的事是 CLQA 過去兩屆是由林川傑教授和 Yutaka Sasaki 博士共同舉辦，由於 Sasaki 博士轉到曼徹斯特大學任教，

工作繁忙，下年度轉由 CMU 的 Mitamura 和 Nyberg 舉辦。

三、建議

NTCIR 是國際三大資訊檢索評比，台灣大學過去 5 年參與規劃舉辦 CLIR、CLQA、和 Opinion Analysis 三項主題，有很多國際研究團隊參加評比，實驗系統的效能，未來持續的參與，才能發揮影響力。

四、攜回資料名稱及內容

Proceedings of The 6th NTCIR Workshop Meeting on Evaluation of Information Access Technologies, and CD-ROM

在文檔中跨語言跨媒體資訊檢索研究(2/2) (頁 22-30)