A study of commonly used tokenizers in Corpus Linguistics

Tokenization and segmentation are steps performed in the earlier stages of most text analysis. It is normally a fast process done using deterministic algorithms [3] to establish token and sentence boundaries [5]. Inexact tokenization can negatively affect later processes and applications with the corpus. For instance, applying a dependency parser to a badly tokenized sequence yields errors beyond the span of the problematic token. Also, depending on the input (e.g. language, variety, register) and on the purpose of the study, distinct tokenization decisions might be expected. For instance, multiword expressions and ambiguous separators such as hashtags can be approached in different ways. This study explores three widely used tokenizers - TreeTagger [8], FreeLing [7], SpaCy [1] - by analyzing their output when processing three different corpora, BNC2014 [6], Corifa [4], ukWac [2]. These corpora were chosen to range for a great variation on size, writing style, and register conventions, to account for a better representativity of the community use. By discussing the different outputs, we intend to shed some light on the importance of having control (and awareness) of all the stages of corpus preparation