A Graph-based Approach for Contextual Text Normalization

The informal nature of social media text renders it very difficult to be automati-cally processed by natural language pro-cessing tools. Text normalization, which corresponds to restoring the non-standard words to their canonical forms, provides a solution to this challenge. We introduce an unsupervised text normalization approach that utilizes not only lexical, but also con-textual and grammatical features of social text. The contextual and grammatical fea-tures are extracted from a word association graph built by using a large unlabeled so-cial media text corpus. The graph encodes the relative positions of the words with re-spect to each other, as well as their part-of-speech tags. The lexical features are ob-tained by using the longest common sub-sequence ratio and edit distance measures to encode the surface similarity among words, and the double metaphone algo-rithm to represent the phonetic similarity. Unlike most of the recent approaches that are based on generating normalization dic-tionaries, the proposed approach performs normalization by considering the context of the non-standard words in the input text. Our results show that it achieves state-of-the-art F-score performance on standard datasets. In addition, the system can be tuned to achieve very high precision with-out sacrificing much from recall.