Visual Bilingual Lexicon Induction with Transferred ConvNet Features

This paper is concerned with the task of bilingual lexicon induction using image-based features. By applying features from a convolutional neural network (CNN), we obtain state-of-the-art performance on a standard dataset, obtaining a 79 % relative improvement over previous work which uses bags of visual words based on SIFT features. The CNN image-based approach is also compared with state-of-the-art lin-guistic approaches to bilingual lexicon in-duction, even outperforming these for one of three language pairs on another stan-dard dataset. Furthermore, we shed new light on the type of visual similarity met-ric to use for genuine similarity versus re-latedness tasks, and experiment with using multiple layers from the same network in an attempt to improve performance.