Exploiting language models for visual recognition

The problem of learning language models from large text corpora has been widely stud-ied within the computational linguistic com-munity. However, little is known about the performance of these language models when applied to the computer vision domain. In this work, we compare representative models: a window-based model, a topic model, a distri-butional memory and a commonsense knowl-edge database, ConceptNet, in two visual recognition scenarios: human action recog-nition and object prediction. We examine whether the knowledge extracted from texts through these models are compatible to the knowledge represented in images. We de-termine the usefulness of different language models in aiding the two visual recognition tasks. The study shows that the language models built from general text corpora can be used instead of expensive annotated images and even outperform the image model when testing on a big general dataset.