Automatic Sentence Break Disambiguation for Thai

Unlike English, there is no explicit sentence marker in Thai language. Conventionally, a space is placed at the end of the sentence when written in Thai. But it does not mean that a space always indicates the sentence boundary. In this paper, we propose the algorithm, which is a feature-based approach, to extract sentences from a paragraph by detecting the appropriate sentence breaking spaces. The algorithm considers the context around a space for determining the space as whether a sentence breaking space or not. The previous method, probabilistic POS trigram approach, considers only the coarse information of part-of-speech in a limited range of context whereas the feature-based approach considers as many features as possible. A feature can be anything that examines a specific information in the context around the target word sequence, such as context words and collocations. To automatically extract such features from a training corpus, we employ the learning algorithm, namely Winnow. The experimental results showed the effectiveness of Winnow comparing with POS trigram, and also demonstrated that Winnow is superior to POS trigram in our task