Fallback Variable History NNLMs: Efficient NNLMs by precomputation and stochastic training

[EN] This paper presents a new method to reduce the computational cost when using Neural Networks as Language Models, during recognition, in some particular scenarios. It is based on a Neural Network that considers input contexts of different length in order to ease the use of a fallback mechanism together with the precomputation of softmax normalization constants for these inputs. The proposed approach is empirically validated, showing their capability to emulate lower order N-grams with a single Neural Network. A machine translation task shows that the proposed model constitutes a good solution to the normalization cost of the output softmax layer of Neural Networks, for some practical cases, without a significant impact in performance while improving the system speed.This work was partially supported by the Spanish MINECO and FEDER founds under project TIN2017-85854-C4-2-R (to MJCB). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Zamora Martínez, FJ.; España Boquera, S.; Castro-Bleda, MJ.; Palacios Corella (2018). Fallback Variable History NNLMs: Efficient NNLMs by precomputation and stochastic training. PLoS ONE. 13(7). https://doi.org/10.1371/journal.pone.020088413