Towards Robot Adaptability in New Situations

We present a system that integrates robot task execution with user input and feedback at multiple abstraction lev-els in order to achieve greater adaptability in new en-vironments. The user can specify a hierarchical task, with the system interactively proposing logical action groupings within the task. During execution, if tasks fail because objects specified in the initial task description are not found in the environment, the robot proposes substitutions autonomously in order to repair the plan and resume execution. The user can assist the robot by reviewing substitutions. Finally, the user can train the robot to recognize and manipulate novel objects, either during training or during execution. In addition to this single-user scenario, we propose extensions that lever-age crowdsourced input to reduce the need for direct user feedback