Credibility in Empirical Legal Analysis

Empirical analysis is central in both legal scholarship and litigation, but it is not credible. Researchers can manipulate data to arrive at any conclusion they wish to obtain. A practice known as data fishing—searching for and selectively reporting methods and results that are favorable to the researcher—entirely invalidates a study’s results by giving rise to false positives and false impressions. Nevertheless, it is prevalent in law, leading to false claims, incorrect verdicts, and destructive policy. In this article, I examine the harm that data fishing in empirical legal research causes. I then build on methods in the sciences to develop a framework for eliminating data fishing and restoring confidence in empirical analysis in legal scholarship and litigation. This framework—which I call DASS (an acronym for Design, Analyze, Scrutinize, and Substantiate)—is designed to be simple, flexible, and practical for application in legal settings. It provides a concrete method for researchers to use to safeguard against data fishing and for consumers of empirical analysis to use to evaluate a researcher’s empirical claims. Finally, after describing the DASS framework and its application in various legal settings, I consider its implications for the “hired-gun” problem and other difficulties related to the reliability of expert evidence