The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results

• Although there is a growing understanding of the importance of statistical power considerations when designing studies and of the value of confidence intervals when interpreting data, confusion exists about the reverse arrangement: the role of confidence intervals in study design and of power in interpretation. Confidence intervals should play an important role when setting sample size, and power should play no role once the data have been collected, but exactly the opposite procedure is widely practiced. In this commentary, we present the reasons why the calculation of power after a study is over is inappropriate and how confidence intervals can be used during both study design and study interpretation. Med. 1994;121:200-206. From Johns Hopkins University School of Medicine, Baltimore, Maryland; and the University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania. For current author addresses, see end of text. Ann Intern The increasing statistical sophistication of medical researchers has heightened their awareness of the importance of having appropriate statistical power in a clinical experiment. Power is the probability that, given a specified true difference between two groups, the quantitative results of a study will be deemed statistically significant. Freiman and colleagues (1) sensitized researchers to the possibility that many so-called "negative" trials of medical interventions might have too few participants to produce statistically significant findings even if clinically important effects actually existed. These authors calculated the statistical power of 71 "negative" studies that compared two treatments. They showed that for most of the studies, the power to detect a 50% improvement in success rates was quite low. This kind of analysis has been applied in various specialty fields, with similar results (2-6). Studies with low statistical power have sample sizes that are too small, producing results that have high statistical variability (low precision). Confidence intervals are a convenient way to express that variability. Numerous articles, commentaries, and editorials (7-13) have appeared in the biomedical literature during the past decade showing how confidence intervals are informative data summaries that can be used in addition to or instead of P values in reporting statistical results. Although there is a growing understanding of the role of power in designing studies and the role of confidence intervals in study interpretation, confusion exists about the reverse arrangement, the role of confidence intervals in designing studies and of power estimates in interpreting study findings. Confidence intervals should play an important role when setting sample size, and power should play no role once the data have been collected, but exactly the opposite is widely practiced. When "no significant difference" between two compared treatments is reported, a common question posed in journal clubs, on rounds, in letters to the editor, and in reviews of manuscripts is, "What was the power of the study to detect the observed difference?" A widely used instrument for assessing the quality of a clinical trial penalizes studies if post hoc power calculations of this type are omitted Although several writers Confidence Intervals A confidence interval can be thought of as the set of true but unknown differences that are statistically compatible with the observed difference. The standard convention for this statistical compatibility is the two-sided 95% confidence interval. A confidence interval is typically reported in the following way: "There was a 10% (95% CI, 2% to 18%) difference in mortality." This means that even though the observed mortality difference was 10% (for example, 70% compared with 60%), the data are statistically compatible with a true mortality difference as small as 2% or as large as 18%. True differences that lie outside the 95% confidence interval are not impossible; they merely have less statistical evidence supporting them than values within it. The 20