The Roles of Haptic-Ostensive Referring Expressions in Cooperative, Task-based Human-Robot Dialogue∗

Generating referring expressions is a task that has received a great deal of attention in the natural-language generation community, with an increasing amount of recent effort tar-geted at the generation of multimodal referring expressions. However, most implemented systems tend to assume very little shared knowledge between the speaker and the hearer, and therefore must generate fully-elaborated linguistic refer-ences. Some systems do include a representation of the phys-ical context or the dialogue context; however, other sources of contextual information are not normally used. Also, the generated references normally consist only of language and, possibly, deictic pointing gestures. When referring to objects in the context of a task-based interaction involving jointly manipulating objects, a much richer notion of context is available, which permits a wider range of referring options. In particular, when conversa-tional partners cooperate on a mutual task in a shared en-vironment, objects can be made accessible simply by ma-nipulating them as part of the task. We demonstrate that such expressions are common in a corpus of human-human dialogues based on constructing virtual objects, and then describe how this type of reference can be incorporated into the output of a humanoid robot that engages in similar joint construction dialogues with a human partner