Character-Aware Neural Language Models Yoon Kim† †School of Engineering and Applied Sciences Harvard University

We describe a simple neural language model that re-lies only on character-level inputs. Predictions are still made at the word-level. Our model employs a con-volutional neural network (CNN) and a highway net-work over characters, whose output is given to a long short-term memory (LSTM) recurrent neural net-work language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60 % fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model out-performs word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for lan-guage modeling. Analysis of word representations ob-tained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information