Adaptation, Context-aware Modeling and Rescoring Methods for Neural Language Models in Automatic Speech Recognition

Language models are a critical component of an automatic speech recognition (ASR) system. Neural network-based language models have demonstrated their superior ability to model long-range dependency compared with n-gram models and have been widely used in ASR systems. This dissertation focuses on improving neural language models from two aspects in the context of ASR. The first aspect is about neural language model adaptation and contextual modeling. The second is about making the applications of neural language models in ASR systems more efficient and accurate. The inherent reason for language model adaptation is that language use is strongly influenced by contextual factors including domain, user preference, topic, and related text, etc. It is usually not possible to obtain adequate training data in every possible combination of these context variables. And sometimes these variables are not known when the model is trained. Besides, it is still challenging for even advanced models, for example, LSTM and Transformer, to perform well on modeling rare words. Thus, in the first part of the dissertation, we focus on neural language model adaptation and contextual modeling methods. Specifically, we aim to develop general and lightweight approaches for real ASR systems. The second part of the dissertation focuses on lattice rescoring, a major application of neural language models in ASR systems. Balancing accuracy and efficiency is crucial for lattice rescoring since exact rescoring is infeasible in practice. We propose methods to simplify the implementation of lattice rescoring in the weighted finite-state transducer framework, and improve its efficiency and accuracy with neural language models. Specifically, we propose a parallel lattice rescoring strategy for efficiency and a novel lattice expansion method to improve recognition accuracy. The approaches and models proposed in this dissertation provide efficient and novel solutions for real problems. Though they are developed in the context of neural language modeling and speech recognition, some methods can be applied for other fields as well, such as machine translation and optical character recognition