Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can’t See What I Mean

Models that acquire semantic represen-tations from both linguistic and percep-tual input are of interest to researchers in NLP because of the obvious parallels with human language learning. Perfor-mance advantages of the multi-modal ap-proach over language-only models have been clearly established when models are required to learn concrete noun concepts. However, such concepts are comparatively rare in everyday language. In this work, we present a new means of extending the scope of multi-modal models to more commonly-occurring abstract lexical con-cepts via an approach that learns multi-modal embeddings. Our architecture out-performs previous approaches in combin-ing input from distinct modalities, and propagates perceptual information on con-crete concepts to abstract concepts more effectively than alternatives. We discuss the implications of our results both for op-timizing the performance of multi-modal models and for theories of abstract con-ceptual representation.