Learning from Unscripted Deictic Gesture and Language for Human-Robot Interactions

As robots become more ubiquitous, it is increasingly important for untrained users to be able to interact with them intuitively. In this work, we investigate how peo-ple refer to objects in the world during relatively un-structured communication with robots. We collect a cor-pus of deictic interactions from users describing objects, which we use to train language and gesture models that allow our robot to determine what objects are being indicated. We introduce a temporal extension to state-of-the-art hierarchical matching pursuit features to sup-port gesture understanding, and demonstrate that com-bining multiple communication modalities more effec-tively capture user intent than relying on a single type of input. Finally, we present initial interactions with a robot that uses the learned models to follow commands.