Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics

This is the final version. Available from the American Economic Association via the DOI in this recordThe credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-indifferences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that: (1) Papers published in the ‘Top 5’ journals are different to others; (2) The journal ‘revise and resubmit’ process mitigates the problem; (3) Things are improving through tim