Analytical Methods for Interpretable Ultradense Word Embeddings

Word embeddings are useful for a wide vari- ety of tasks, but they lack interpretability. By rotating word spaces, interpretable dimensions can be identified while preserving the informa- tion contained in the embeddings without any loss. In this work, we investigate three meth- ods for making word spaces interpretable by rotation: Densifier (Rothe et al., 2016), linear SVMs and DensRay, a new method we pro- pose. In contrast to Densifier, DensRay can be computed in closed form, is hyperparameter- free and thus more robust than Densifier. We evaluate the three methods on lexicon induc- tion and set-based word analogy. In addition we provide qualitative insights as to how inter- pretable word spaces can be used for removing gender bias from embeddings