Chinese Word Segmentation and Information Retrieval

In this paper we present results of experiments with Chinese word segmentation and information retrieval. Our experiments with three different word segmentation algorithms indicate that accurate segmentation measurably improves retrieval performance. We discuss the evaluation of word segmentation algorithms for the purpose of better indexing segmented texts for retrieval. Introduction The increased interest in crosslingual and multilingual information retrieval has revealed the new challenges inherent in retrieval in multiple languages. English IR has been extensively engineered for 30 years, with the development of stop lists, stemming, etc., but such resources are not available for many languages. Recent Text Retrieval Conferences (TREC-4 (Harman 1996b) and TREC-5 (Harman 1996a)) have included separate tracks for evaluation of retrieval in languages such as Spanish and Chinese, and a crosslingual track involving English, French, and German is planned for TREC-6. Traditionally, infor..