Continual planning for crossmodal situated clarification in human-robot interaction

Abstract — Robots do not fully understand the world they are situated in. This includes what humans talk to them about. A fundamental problem is thus how a robot can clarify such a lack of understanding. This paper addresses the issue of how a robot can create a plan for resolving a need for clarification. It characterises situated clarification as an information need which may arise in any sensory-motoric modality required to interpret the situated context of the robot, or any deliberative modality referring to that context. It then focuses on how, once a clarification need has been identified, the robot can create a plan in which one or more modalities are used to resolve it. Modalities are involved on the basis of the types of information they can provide. These information types are identified in the ontologies the modalities use to interconnect their content with content of other modalities (via information fusion). We take a continual approach to planning and execution monitoring. This provides the ability to re-plan depending on modality availability and success in resolving (part of) a clarification need. We illustrate the implementation on several examples. I