Information

Abstract In this paper, we propose an unsupervised seg-mentation approach, named "n-gram mutual information", or NGMI, which is used to segment Chinese documents into n-character words or phrases, using language statistics drawn from the Chinese Wikipedia corpus. The approach allevi-ates the tremendous effort that is required in preparing and maintaining the manually segmented Chinese text for train-ing purposes, and manually maintaining ever expanding lex-icons. Previously, mutual information was used to achieve automated segmentation into 2-character words. The NGMI approach extends the approach to handle longer n-character words. Experiments with heterogeneous documents from the Chinese Wikipedia collection show good results