AF T The Fallacy of Placing Confidence in Confidence Intervals

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated samples, on average. The width of confidence intervals is thought to index the precision of an estimate; the parameter values contained within a CI are thought to be more plausible than those outside the interval; and the confidence coefficient of the interval (typically 95%) is thought to index the plausibility that the true parameter is included in the interval. We show in a number of examples that CIs do not necessarily have any of these properties, and generally lead to incoherent inferences. For this reason, we recommend against the use of the method of CIs for inference. “You keep using that word. I do not think it means what you think it means.” Inigo Montoya, The Princess Bride (1987) The development of statistics over the past century has seen the proliferation of methods designed to make infer-ences from data. Methods vary widely in their philosophical foundations, the questions they are supposed to address, and their frequency of use in practice. One popular and widely-promoted class of methods are interval estimates, which in-clude frequentist confidence intervals, Bayesian credible in-tervals and highest posterior density (HPD) intervals, fiducial intervals, and likelihood intervals. These procedures differ in their philosophical foundation and computation, but infor-mally are all designed to be estimates of a parameter that ac-count for measurement or sampling uncertainty by yielding a range of values for the parameter instead of a single value. Of the many kinds of interval estimates, the most popular is the confidence interval (CI). Confidence intervals are intro-duced in almost all introductory statistics texts; they are rec-ommended or required by the methodological guidelines of many prominent journals (e.g., Psychonomics Society, 2012