ArabicSeg: An Arabic News Story Segmentation

Text segmentation is a very critical step to many applications and while it has been addressed extensively for the English language, work on text segmentation for other languages is still lagging behind. In this paper the ArabicSeg system for segmenting Arabic news stories is presented. The developed system is based on a linguistic technique called lexical chaining which measures the cohesion between textual units. In conjunction with this technique, a set of error reduction filters have been introduced and were found to significantly reduce segmentation errors in the detection of borders in Arabic based news stories. The results of evaluation experiments carried out on an Arabic Reuters news story dataset are presented. An analysis of the effect of introducing each of the proposed error reduction filters is also provided. The evaluation shows that the segmentation results produced by the presented system are not only comparable to the results reported by English based segmentation algorithms, but also outperform them