Inferring Maps and Behaviors from Natural Language Instructions

Abstract. Natural language provides a flexible, intuitive way for people to command robots, which is becoming increasingly important as robots transition to working alongside people in our homes and workplaces. To follow instructions in unknown environments, robots will be expected to reason about parts of the environments that were described in the in-struction, but that the robot has no direct knowledge about. This paper proposes a probabilistic framework that enables robots to follow com-mands given in natural language, without any prior knowledge of the environment. The novelty lies in exploiting environment information im-plicit in the instruction, thereby treating language as a type of sensor which is used to formulate a prior distribution over the unknown parts of the environment. The algorithm then uses this learned distribution to infer a sequence of actions that are most consistent with the command, updating our belief as we gather more metric information. We evaluate our approach through simulation as well as experiments on two mobile robots; our results demonstrate the algorithm’s ability to follow naviga-tion commands with performance comparable to that of a fully-known environment.