Computing Highly Accurate Confidence Limits from Discrete data using Importance Sampling

For discrete parametric models, approximate confidence limits perform poorly from a strict frequentist perspective. In principle, exact and optimal confidence limits can be computed using the formula of Buehler (1957), Lloyd and Kabaila (2003). So-called profile upper limits (Kabaila \& Lloyd, 2001) are closely related to Buehler limits and have extremely good properties. Both profile and Buehler limits depend on the probability of a certain tail set as a function of the unknown parameters. Unfortunately, this probability surface is not computable for realistic models. In this paper, importance sampling is used to estimate the surface and hence the confidence limits. Unlike the recent methodology of Garthwaite and Jones (2009), the new method (a) allows for nuisance parameters, (b) is an order of magnitude more efficient that the Robbins-Monro bound, (c) does not require any simulation phases or tuning constants, (d) give a straightforward simulation standard error for the target limit and (e) includes a simple diagnostic for simulation breakdown