Using surrogate distributions to improve the convergence properties of gibbs-type samplers

Gibbs-type samplers are widely used tools for obtaining Monte Carlo samples from posterior distributions under complicated Bayesian models. Standard Gibbs samplers update component quantities of the parameter by sequentially sampling their conditional distributions under the target joint distribution. However, this strategy can be slow to converge if the components are highly correlated. We formalize a general strategy to construct more efficient samplers by replacing some of the conditional distributions with conditionals of a surrogate distribution. The surrogate distribution is designed to share certain marginal distributions with the target, but with lower correlations among its components. Although not necessarily recognized when they were introduced, a number of existing strategies for improving Gibbs can be formulated in this way (e.g., Marginal Data Augmentation, Partially Collapsed Gibbs sampling, Ancillarity-Sufficiency Interweaving Strategy, etc.). The use of surrogate distributions in Gibbs-type samplers may lead to incompatible conditional distributions and thus sensitivity to the order of the component draws. We propose a framework to combine different strategies involving surrogate distributions into a single coherent sampler that maintains the target stationary distribution and outperforms any of its component algorithms in terms of convergence. We use both theoretical arguments and numerical examples to illustrate the implementation and efficiency of our strategy. A problem in supernova cosmology has motivated our work and serves as a realistic testing ground for our methods. Finally, we correct two errors in the related Marginal Data Augmentation algorithms of Imai and van~Dyk (2005) that are quite popular for fitting multinomial probit models.Open Acces