Closing the gap between word and character based language models using fixed and learned hierarchies

Language Modeling (LM) is a complex task that has been preferably addressed with word level RNNs and, recently, attention-based models. LM at character level has shown inferior performance as it entails more complex training, but it also brings along several desirable conceptual modelling advantages such as no closed-vocabulary limitation and better exploitation of words' similarities. In this work we identify the main causes of the performance gap between word and character level LM and we propose solutions to address them. Our experiments make up considerable ground to word level models; we prove that the input can be processed at character level without losing anything in performance and that processing the output at character level results only in a minor performance loss. We also investigate the problem of learning the hierarchical structure of data in a totally unsupervised fashion. We propose a novel approach to learn the segmentation based on the Gumbel Softmax Reparameterization for Bernoulli variables. We have successfully evaluated our model on natural language tasks (PTB, text8) as well as on other types of data, e.g. xml dumps (enwik8) and images (MNIST, CIFAR10)