Alternative Outcomes, Surprise, and Predicting Replication of Psychological Research

When we read research findings, what facilitates consideration of different possible outcomes? 112 Amazon Mechanical Turk Workers read about four psychological studies, and either predicted the results, received the results without explanation, or received the results with a plausible explanation. We hypothesized that 1. receiving results would increase the difficulty of explaining alternative outcomes, and that this would be amplified by receiving explanations, 2. As difficulty explaining alternatives increases, participants would be less surprised by actual results and predict higher likelihood of replication. We did not find the expected differences in difficulty across conditions. However, across all four research studies, surprise was negatively correlated with likelihood of replication, indicating that more surprising results are considered less likely to replicate. We consider reasons why our manipulation did not affect difficulty of explaining alternative outcomes, and discuss implications for how research can best be presented to promote consideration of alternative outcomes