Binomial confidence intervals for rare events: importance of defining margin of error relative to magnitude of proportion

Confidence interval performance is typically assessed in terms of two criteria: coverage probability and interval width (or margin of error). In this work we detail the importance of defining the margin of error relative to the magnitude of the estimated proportion when the success probability is small. We compare the performance of four common proportion estimators: the Wald, Clopper-Pearson, Wilson and Agresti-Coull, in the context of rare-event probabilities. We show that incompatibilities between the margin of error and the proportion results in very narrow intervals (requiring extremely large sample sizes), or intervals that are too wide to be practically useful. We propose a relative margin of error scheme that is consistent with the order of magnitude of the proportion. Confidence interval performance is thus assessed in terms of satisfying this relative margin of error, in conjunction with achieving a desired coverage probability. We show that when adherence to this relative margin of error is considered as a requirement for satisfactory interval performance, all four interval estimators perform somewhat similarly for a given sample size and confidence level. The proposed relative margin of error scheme is evaluated by way of its application to a number of recent studies from the literature.Comment: 25 pages, 3 figure