Word embeddings through hellinger PCA

Word embeddings resulting from neural language models have been shown to be a great asset for a large variety of NLP tasks. However, such architecture might be difficult and time-consuming to train. Instead, we propose to drastically sim-plify the word embeddings computation through a Hellinger PCA of the word co-occurence matrix. We compare those new word embeddings with some well-known embeddings on named entity recognition and movie review tasks and show that we can reach similar or even better perfor-mance. Although deep learning is not re-ally necessary for generating good word embeddings, we show that it can provide an easy way to adapt embeddings to spe-cific tasks.