Code-mixing: A challenge for language identification in the language of social media

In social media communication, multilin-gual speakers often switch between lan-guages, and, in such an environment, au-tomatic language identification becomes both a necessary and challenging task. In this paper, we describe our work in progress on the problem of automatic language identification for the language of social media. We describe a new dataset that we are in the process of cre-ating, which contains Facebook posts and comments that exhibit code mixing be-tween Bengali, English and Hindi. We also present some preliminary word-level language identification experiments using this dataset. Different techniques are employed, including a simple unsuper-vised dictionary-based approach, super-vised word-level classification with and without contextual clues, and sequence la-belling using Conditional Random Fields. We find that the dictionary-based approach is surpassed by supervised classification and sequence labelling, and that it is im-portant to take contextual clues into con-sideration.