Toward interactive training and evaluation

Machine learning often relies on costly labeled data, and this impedes its application to new classification and infor-mation extraction problems. This has motivated the devel-opment of methods for leveraging abundant prior knowledge about these problems, including methods for lightly super-vised learning using model expectation constraints. Building on this work, we envision an interactive training paradigm in which practitioners perform evaluation, analyze errors, and provide and refine expectation constraints in a closed loop. In this paper, we focus on several key subproblems in this paradigm that can be cast as selecting a represen-tative sample of the unlabeled data for the practitioner to inspect. To address these problems, we propose stratified sampling methods that use model expectations as a proxy for latent output variables. In classification and sequence labeling experiments, these sampling strategies reduce ac-curacy evaluation effort by as much as 53%, provide more reliable estimates of F1 for rare labels, and aid in the speci-fication and refinement of constraints