Nlify: Third-party programming support for spoken natural language interfaces

This paper presents the design and implementation of a pro-gramming system that enables third-party developers to add spoken natural language (SNL) interfaces to mobile appli-cations. Existing systems either restrict SNL capabilities to first-party applications or limit developer-defined spoken in-teractions to keyphrases rather than broad natural language. An examination of expert workflow reveals that the primary challenge is in gathering comprehensive sets of paraphrases for each command and in selecting and tuning corresponding statistical models for speech and language processing. We address the former problem by integrating automated statis-tical machine paraphrasing and webscale crowdsourcing into the developer workflow. We address the latter by developing a classifier architecture designed to be robust across app do-mains. We have realized our design fully as an extension to the Visual Studio IDE. Based on a new benchmark dataset with 3500 spoken instances of 27 commands from 20 sub-jects and a small developer study, we establish the promise of our approach and the impact of various design choices. 1