Evaluation methods for unsupervised word embeddings

We present a comprehensive study of eval-uation methods for unsupervised embed-ding techniques that obtain meaningful representations of words from text. Differ-ent evaluations result in different orderings of embedding methods, calling into ques-tion the common assumption that there is one single optimal vector representation. We present new evaluation techniques that directly compare embeddings with respect to specific queries. These methods re-duce bias, provide greater insight, and allow us to solicit data-driven relevance judgments rapidly and accurately through crowdsourcing.