Dynamic Generation of Spatial Referring Expressions for Social Robots

Parts of this thesis have been published by the author: Wallbridge, C. D., Lemaignan, S., & Belpaeme, T. (2017). Qualitative review of object recognition techniques for tabletop manipulation. In Proceedings of the 5th International Conference on Human Agent Interaction, (pp. 359-363). ACM Wallbridge, C. D., Lemaignan, S., Senft, E., Edmunds, C., & Belpaeme, T. (2018a). Spatial referring expressions in child-robot interaction: Let’s be ambiguous! In 4th Workshop on Robots for Learning (R4L) - Inclusive Learning @HRI 2018 Lemaignan, S., Sallami, Y., Wallbridge, C., Clodic, A., Belpaeme, T., & Alami, R. (2018). Underworlds: Cascading situation assessment for robots. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),(pp. 7750-7757). IEEE Belpaeme, T., Vogt, P., Van den Berghe, R., Bergmann, K., Göksun, T., De Haas, M., Kanero, J., Kennedy, J., Küntay, A. C., Oudgenoeg-Paz, O., et al. (2018). Guidelines for designing social robots as second language tutors. International Journal of Social Robotics, 10(3), 325–341 Wallbridge, C. D., van den Berghe, R., Hernández Garcia, D., Kanero, J., Lemaignan, S.,Edmunds, C., & Belpaeme, T. (2018b). Using a robot peer to encourage the production of spatial concepts in a second language. In Proceedings of the 6th International Conference on Human-Agent Interaction, (pp. 54-60). ACM Wallbridge, C. D., Lemaignan, S., Senft, E., & Belpaeme, T. (2019b). Towards generating spatial referring expressions in a social robot: Dynamic vs non-ambiguous. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI),(pp. 616-617). IEEE Vogt, P., van den Berghe, R., de Haas, M., Hoffman, L., Kanero, J., Mamus, E., Montanier, J.-M., Oranç, C., Oudgenoeg-Paz, O., García, D. H., et al. (2019a). Second language tutoring using social robots: A large-scale study. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), (pp. 497–505). IEEE Wallbridge, C. D., Lemaignan, S., Senft, E., & Belpaeme, T. (2019a). Generating spatial referring expressions in a social robot: Dynamic vs non-ambiguous. Frontiers in Robotics and AI, 6, 67 Wallbridge, C. D., Smith, A., Giuliani, M., Melhuish, C., Belpaeme, T., & Lemaignan, S. (2020). The Effectiveness of Dynamically Processed Incremental Descriptions in Human Robot Interaction. Submitted to the ACM journal Transactions on Human-Robot Interaction.Robots are an increasing part of our daily lives. As robots become more pervasive it is important that we are able to interact with them naturally. The field of Human Robot Interaction (HRI) seeks to improve interactions between human and robot. People spend many years in their childhood learning to communicate the locations of objects naturally to other people. When trying to communicate the location of objects, people generate under-specified statements and then generate further repair if necessary to guide the listener as part of an interactive dialogue. The focus in HRI up until now has been on trying to generate non-ambiguous statements to refer to objects or locations. I create here a dynamic method of generating spatial referring expressions, based on under-specified statements followed if necessary by repair, as a step towards more interactive dialogue. I present the following thesis: A robot that is able to use dynamic description methods –using vague initial language with the ability to further repair for generating spatial referring expressions as well as reducing the problem of combinatorial explosion, will be a more effective tool for collaborating with people than using static non-ambiguous descriptions. This kind of dynamic form of description is new to the field of HRI. In socio-linguistics this form of communication is thought to lead to a least collaborative effort, with both partners in a conversation contributing to a description. To ensure the validity of my work, I base my work on potential real use case scenarios for a social robot in a number of studies. I start by looking at a Robot Assisted Language Learning scenario, in which the robot attempts to encourage the use of spatial language in a quiz based game. As another use case I look at a nuclear waste disposal task. I also present the initial study in which I noticed the discrepancy between how we have been attempting to generate referring expressions, and how people communicate. I describe how I created the dynamic systems based on human-human interactions. By looking at two people solving the task we gather data on position to represent the state of the action, and what participants are saying in that state. I use this to build a classifier that determines what the robot should say at a given state of the interaction. This system allows a robot to successfully guide a person to the correct object/location. In my studies I find that this dynamic form of communication is more efficient in terms of time, and distance travelled when trying to complete a task that requires spatial referring expressions when compared to static non-ambiguous descriptions. I also find that it is possible for people to prefer this form of communication in a complex real world task