Improving cross-domain n-gram language modelling with skipgrams

© 2016 Association for Computational Linguistics. In this paper we improve over the hierarchical Pitman-Yor processes language model in a cross-domain setting by adding skipgrams as features. We find that adding skipgram features reduces the perplexity. This reduction is substantial when models are trained on a generic corpus and tested on domain-specific corpora. We also find that within-domain testing and crossdomain testing require different backoff strategies. We observe a 30-40% reduction in perplexity in a cross-domain language modelling task, and up to 6% reduction in a within-domain experiment, for both English and Flemish-Dutch.Onrust L., van den Bosch A., Van hamme H., ''Improving cross-domain n-gram language modelling with skipgrams'', Proceedings 54th annual meeting of the Association for Computational Linguistics - ACL 2016, vol. 2, pp. 137-142, August 7-12, 2016, Berlin, Germany.status: publishe