Corpus-Specific Stemming using Word Form Co-occurrence

Stemming is used in many information retrieval (IR) systems to reduce word forms to common roots. It is one of the simplest and most successful applications of natural language processing for IR. Current stemming algorithms are, however, either inflexible or difficult to adapt to the specific characteristics of a text corpus, except by the manual definition of exception lists. We propose a technique for using corpus-based word co-occurrence statistics to modify a stemmer. Experiments show that this technique is effective and is very suitable for query-based stemming. 1 Introduction Stemming is a common form of language processing in most information retrieval systems [4]. It is similar to the morphological processing used in natural language processing, but has somewhat different aims. In an information retrieval system, stemming is used to reduce different word forms to common roots, and thereby improve the ability of the system to match query and document vocabulary. The variety in ..