Situated Learning and Understanding of Natural Language

Thesis (Ph.D.)--University of Washington, 2015Robust language understanding systems have the potential to transform how we interact with computers. However, significant challenges in automated reasoning and learning remain to be solved before we achieve this goal. To accurately interpret user utterances, for example when instructing a robot, a system must jointly reason about word meaning, grammatical structure, conversation history and world state. Additionally, to learn without prohibitive data annotation costs, systems must automatically make use of weak interaction cues for autonomous language learning. We present a framework that uses situated interactions to learn to map sentences to rich, logical meaning representations. Our approach induces a Combinatory Categorial Grammar (CCG), while relying on various learning cues, such as easily gathered demonstrations and even raw conversations without any additional annotation effort. It achieves state-of-the-art performance on a number of tasks, including robotic interpretation of navigational directions and learning to understand user utterances in dialog systems. Such an approach, when integrated into complete systems, has the potential to achieve continuous, autonomous learning by participating in interactions with users. We first describe an approach to induce a CCG from automatically recorded conversations. Next, we show that jointly reasoning about language meaning and system response improves instruction following, and describe an approach to induce compact grammars in such scenarios. Finally, with the goal of studying linguistic phenomena not addressed by existing corpora, we present a broad-coverage CCG parser to recover rich formal representations. Together, our techniques lower or eliminate annotation overhead, improve situated language understanding and enable grammar induction for recovering logical forms on a scale not possible before