What can machine learning do? Implications for citizen scientists

Citizen science projects set up in research fields such as astronomy, ecology and biodiversity, biology, and neuroimaging produce large datasets; thus they hold promise for applying artificial intelligence for the social and environmental spheres. Human-machine integration in citizen science can harness the contributions of many human observers and use machine learning (ML) to process their contributed data. Several citizen science projects have designed complex human-machine systems, taking advantage of the complementarity of the strengths of humans and machines, and aiming to optimize for efficiency and human engagement. Using document analysis of 12 citizen science projects deploying ML techniques to optimize classification tasks, we describe the distribution of work between citizens and researchers and between humans and algorithms, as well as configurations of human-in-the-loop. The results indicate that experts are involved in every aspect of the loop, from annotating or labeling data to giving them to algorithms to train and make decisions from such predictions. Experts also test and validate models to improve accuracy by scoring their outputs when algorithms are not able to make the right decisions. While experts are the humans mainly involved in the loop, citizens are also involved at various stages of the process. We present three main examples of citizens-in-the-loop: (a) when algorithms provide incorrect suggestions; (b) when algorithms do not know to perform classification, and (c) when algorithms are active learners. We contend that, unlike automated systems that tend to remove or reduce the need for humans, the examined projects are heteromated systems that do not function without the indispensable human mediation of engaged citizens