Information Structure Prediction for Visual-world Referring Expressions

We investigate the order of mention for objects in relational descriptions in visual scenes. Existing work in the visual do-main focuses on content selection for text generation and relies primarily on tem-plates to generate surface realizations from underlying content choices. In contrast, we seek to clarify the influence of visual perception on the linguistic form (as op-posed to the content) of descriptions, mod-eling the variation in and constraints on the surface orderings in a description. We find previously-unknown effects of the vi-sual characteristics of objects; specifically, when a relational description involves a vi-sually salient object, that object is more likely to be mentioned first. We conduct a detailed analysis of these patterns using logistic regression, and also train and eval-uate a classifier. Our methods yield signif-icant improvement in classification accu-racy over a naive baseline.