Improving Named Entity Recognition in Tweets via Detecting Non-Standard Words

Most previous work of text normalization on informal text made a strong assumption that the system has already known which tokens are non-standard words (NSW) and thus need normalization. However, this is not realistic. In this paper, we propose a method for NSW detection. In addi-tion to the information based on the dic-tionary, e.g., whether a word is out-of-vocabulary (OOV), we leverage novel in-formation derived from the normalization results for OOV words to help make deci-sions. Second, this paper investigates two methods using NSW detection results for named entity recognition (NER) in social media data. One adopts a pipeline strat-egy, and the other uses a joint decoding fashion. We also create a new data set with newly added normalization annota-tion beyond the existing named entity la-bels. This is the first data set with such annotation and we release it for research purpose. Our experiment results demon-strate the effectiveness of our NSW detec-tion method and the benefit of NSW detec-tion for NER. Our proposed methods per-form better than the state-of-the-art NER system.