Sugar House

In this paper we present a novel approach to generating referring expressions (GRE) that is tailored to a model of the visual context the user is attending to. The ap-proach integrates a new computational model of visual salience in simulated 3-D environments with Dale and Reiter’s (1995) Incremental Algorithm. The advantage of our GRE framework are: (1) the context set used by the GRE algorithm is dynamically computed by the vi-sual saliency algorithm as a user navigates through a simulation; (2) the integration of visual salience into the generation process means that in some instances un-derspecified but sufficiently detailed descriptions of the target object are generated that are shorter than those generated by GRE algorithms which focus purely on adjectival and type attributes; (3) the integration of vi-sual saliency into the generation process means that our GRE algorithm will in some instances succeed in gen-erating a description of the target object in situations where GRE algorithms which focus purely on adjecti-val and type attributes fail