Disagreements about the strength of evidence1

Ultimately, nearly every researcher selects data for analysis—I certainly do, indeed almost none of the studies I’ve ever been involved in have been preregistered—so my point here is not to condemn but to understand. To put it another way, in the real world where we live, the scientific claims that we notice have been selected through a complex filtering process involving funders, researchers, scientific journals, and the news media. So the question to us, as statisticians and consumers of research, is how to interpret what we do happen to see. From a statistical perspective, our conclusions about the world, based as they are on the research that we hear about, implicitly condition on certain selected data summaries rather than on all the potentially relevant data. Interpretation of research claims can be contentious and often involves asymmetrical battles. For example, when Malcolm Gladwell credulously told the story of a mathematician who claimed to be able to predict marriage breakups with 90 % accuracy, or reported a research study on learning difficulties, without noting that it failed to replicate on “a larger and more representative sample of subjects ” (as noted by psychology researcher Christopher Chabris), the two sides had completely differen