Experiments to Evaluate Probabilistic Models for Automatic Stemmer Generation and Query Word Translation

Abstract. The paper describes statistical methods and experiments for stemming and for the translation of query words used in the monolin-gual and bilingual tracks in CLEF 2003. While there is still room for im-provement in the method proposed for the bilingual track, the approach adopted for the monolingual track makes it possible to generate stem-mers which learn directly how to stem the words in a document from a training word list extracted from the document collection, with no need for language-dependent knowledge. The experiments suggest that sta-tistical approaches to stemming are as effective as classical algorithms which encapsulate predefined linguistic rules