Guessing the Effect?: A Latent Class Model for Measuring Learning

Note: All scripts and data necessary for replicating the analyses in the manuscript will be made publicly available upon publication of the manuscript. Guessing on closed-ended knowledge items is common. Under likely to hold assumptions, in presence of guessing, the most common estimator of learning, difference between pre- and post-process scores, is negatively biased. To account for guessing related error, we develop a latent class model of how people respond to knowledge questions, and identify the model with the mild assumption that people do not lose knowledge over short periods of time. A Monte Carlo simula-tion over a broad range of informative processes and knowledge items shows that the simple difference score is negatively biased and that our method is unbiased. We apply our model to data from the Deliberative Polls. Expectedly, we find that estimates of learning once adjusted for guessing are considerably higher. Not only that, adjusting for guessing also eliminates the gender gap in learning, and halves the pre-treatment gender gap on political knowledge