Grounding Linguistic Analysis in Control Applications

This thesis addresses the problem of grounding linguistic analysis in control applica-tions, such as automated maintenance of computers and game playing. We assume access to natural language documents that describe the desired behavior of a con-trol algorithm, either via explicit step-by-step instructions, via high-level strategy advice, or by specifying the dynamics of the control domain. Our goal is to develop techniques for automatically interpreting such documents, and leveraging the textual information to effectively guide control actions. We show that in this setting, langauge analysis can be learnt effectively via feed-back signals inherent to the control application, obviating the need for manual anno-tations. Moreover we demonstrate how information automatically acquired from text can be used to improve the performance of the target control application. We apply our ideas to three applications of increasing linguistic and control com-plexity – interpreting step-by-step instructions into commands in a graphical user interface; interpreting high-level strategic advice to play a complex strategy game