Metacognitively Wise Crowds

Aggregates of many judgments tend to outperform each of the individual judgments that compose the aggregate, termed the Wisdom of Crowds effect. Metacognition has played an understudied role in the efficacy of these crowds and so in a series of experiments I explore how metacognition and self-direction can be used to improve crowd wisdom. I first demonstrate empirically that individuals can leverage their metacognitive abilities to improve the performance of crowds when they are allowed to opt-in to questions of their choosing. I develop a Bayesian framework wherein latent contextual knowledge describes how crowd members make opt-in decisions to elucidate the relationship between these cognitive and metacognitive processes. I then show that metacognitive ability can be estimated by asking questions with no correct response options and create metacognitively wise crowds which achieve more accurate responses despite incorporating fewer crowd members. I discuss my contributions to a geopolitical forecasting competition in which I developed models that combine human and algorithmic judgments to create highly accurate forecasts of the future. In this competition, I evaluated the effects of attenuating forecaster self-direction in an applied setting. These findings collectively demonstrate the importance of metacognition in forming accurate aggregate judgments and clarify the underlying metacognitive processes involved in self-direction