Programmatic gold: Targeted and scalable quality assurance in crowdsourcing

Crowdsourcing is an effective tool for scalable data annota-tion in both research and enterprise contexts. Due to crowd-sourcing’s open participation model, quality assurance is crit-ical to the success of any project. Present methods rely on EM-style post-processing or manual annotation of large gold standard sets. In this paper we present an automated quality assurance process that is inexpensive and scalable. Our novel process relies on programmatic gold creation to provide tar-geted training feedback to workers and to prevent common scamming scenarios. We find that it decreases the amount of manual work required to manage crowdsourced labor while improving the overall quality of the results.