ReferItGame: Referring to Objects in Photographs of Natural Scenes

In this paper we introduce a new game to crowd-source natural language referring expressions. By designing a two player game, we can both collect and verify refer-ring expressions directly within the game. To date, the game has produced a dataset containing 130,525 expressions, referring to 96,654 distinct objects, in 19,894 pho-tographs of natural scenes. This dataset is larger and more varied than previous REG datasets and allows us to study referring expressions in real-world scenes. We pro-vide an in depth analysis of the resulting dataset. Based on our findings, we design a new optimization based model for gen-erating referring expressions and perform experimental evaluations on 3 test sets.