Article Finding a Comparison Group: Is Online Crowdsourcing a Viable Option?

This article explores the viability of online crowdsourcing for creating matched-comparison groups. This exploratory study compares survey results from a randomized control group to survey results from a matched-comparison group created from Amazon.com’s MTurk crowdsourcing service to determine their comparability. Study findings indicate that online crowdsourcing, a process that allows access to many participants to complete specific tasks, is a potentially viable resource for evaluation designs where access to comparison groups, large budgets, and/or time are limited. The article highlights the strengths and limitations of the online crowdsourcing approach and describes ways that it could potentially be used in evaluation practice. Keywords matched comparison group, crowdsourcing, quasi-experimental design, experimental design, evaluation methods In the mid-19th century, one of the first editors of the Oxford English Dictionary, Herbert Coleridge, sent letters to universities and schools describing the goal of the project and the need for volunteers to help find quotes in various works of literature that would help the editors illustrate the meanings of words in context (Winchester, 2003). Through this process, the Oxford English Dictionary was able to recruit thousands of individuals over the course of four decades to help them complete this project. This effort illustrates the potential of accessing many individuals to complete specific objectives— something we now call ‘‘crowdsourcing’ ’ (Howe, 2006). Crowdsourcing is operationally defined as the ‘‘Paid recruitment of... an independent global workforce for the objective of working on a specifically defined task or set of tasks’ ’ (Behrend, Sharek, Meade, & Wiebe, 2011, p. 800). Using modern tools of technology, tasks such as tagging pictures, translating text, or reviewing website content can be submitted to an online system, and subscribed participants can elect to complete set tasks for specified payments. This article examine