A corpus-based finite-state morphological toolkit for contemporary Arabic

We develop an open-source large-scale finite-state morphological processing toolkit (AraComLex) for Modern StandardArabic (MSA) distributed under the GPLv3 license (http://aracomlex.sourceforge.net). The morphological transducer is based on a lexical database specifically constructed for this purpose. In contrast to previous resources, the database is tuned to MSA, eliminating lexical entries no longer attested in contemporary use. The database is built using a corpus of 1,089,111,204 word tokens, a pre-annotation tool, machine learning techniques and knowledge-based pattern matching to automatically acquire lexical knowledge. Our morphological transducer is evaluated and compared to LDC’s SAMA(StandardArabic Morphological Analyser). We also develop a finite-state morphological guesser as part of a methodology for extracting unknown word forms, lemmatizing them, and giving them a priority weight for inclusion in the lexicon