Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective

Recently significant advances have been witnessed in the area of distributed word representations based on neural networks, which are also known as word embeddings. Among the new word embed-ding models, skip-gram negative sampling (SGNS) in the word2vec toolbox has attracted much atten-tion due to its simplicity and effectiveness. Howev-er, the principles of SGNS remain not well under-stood, except for a recent work that explains SGNS as an implicit matrix factorization of the pointwise mutual information (PMI) matrix. In this paper, we provide a new perspective for further understanding SGNS. We point out that SGNS is essentially a rep-resentation learning method, which learns to repre-sent the co-occurrence vector for a word. Based on the representation learning view, SGNS is in fact an explicit matrix factorization (EMF) of the words’ co-occurrence matrix. Furthermore, extended su-pervised word embedding can be established based on our proposed representation learning view.