Pause and stop labeling for Chinese sentence boundary detection

The fuzziness of Chinese sentence boundary makes discourse analysis more challenging. Moreover, many articles posted on the Internet are even lack of punctuation marks. In this pa-per, we collect documents written by masters as a reference corpus and propose a model to label the punctuation marks for the given text. Conditional random field (CRF) models trained with the corpus determine the correct delimiter (a comma or a full-stop) between each pair of successive clauses. Different tag-ging schemes and various features from differ-ent linguistic levels are explored. The results show that our segmenter achieves an accuracy of 77.48 % for plain text, which is close to the human performance 81.18%. For the rich for-matted text, our segmenter achieves an even better accuracy of 82.93%.