Distilling Word Embeddings: An Encoding Approach

Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restricted systems. This paper addresses the problem of distilling word embeddings for NLP tasks. We propose an encoding approach to distill task-specific knowledge from a set of high-dimensional embeddings, so that we can reduce model complexity by a large margin as well as retain high accuracy, achieving a good compromise between efficiency and performance. Experiments reveal the phenomenon that distilling knowledge from cumbersome embeddings is better than directly training neural networks with small embeddings.CPCI-S(ISTP)doublepower.mou@gmail.com; jiaran1994@gmail.com; xuyan14@sei.pku.edu.cn; lige@sei.pku.edu.cn; zhanglu@sei.pku.edu.cn; zhijin@sei.pku.edu.cn1977-198