Using a Single Framework for Computational Modeling of Linguistic Similarity for Solving Many NLP Problems

In this paper we show how a single framework for computational modeling of linguistic similarity can be used for solving many problems. Similarity can be measured within or across languages and at various linguistic levels. We model linguistic similarity in three stages: surface similarity, contextual similarity and distributional similarity. We have successfully used the framework for several applications like spell checking and text normalization, unsupervised shallow morphological analysis, improving information retrieval, transliteration, cognate identification, language identification and sentence alignment etc. For all these applications, we have been able to obtain results comparable with the state of the art.