visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306

We present a model that generates natural language de-scriptions of images and their regions. Our approach lever-ages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between lan-guage and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architec-ture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in re-trieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions sig-nificantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations. 1