Segmentation algorithm for Chinese based on extraction of context information

汉语分词在汉语文本处理过程中是一个特殊而重要的组成部分。传统的基于词典的分词算法存在很大的缺陷,无法对未登录词进行很好的处理。基于概率的算法只考虑了训练集语料的概率模型,对于不同领域的文本的处理不尽如人意。文章提出一种基于上下文信息提取的概率分词算法,能够将切分文本的上下文信息加入到分词概率模型中,以指导文本的切分。这种切分算法结合经典n元模型以及EM算法,在封闭和开放测试环境中分别取得了比较好的效果。Chinese segmentation is a special and important issue in Chinese texts processing.The traditional segmentation methods based on an existing dictionary have an obvious defect when they are used to segment texts which may contain words unknown to the dictionary.And the probabilistic methods those consider the probabilistic model of the training set only also do a bad job on the texts of a specific domain.In this paper,a probabilistic segmentation method based on extracting context information was proposed,which adds the context information of the segmenting text into the segmentation probabilistic model so as to guide the processing.The method combining n-gram model and EM algorithm achieves a good effect in the close and opening test.国家863计划资助项目(2002AA117010