Monte Carlo algorithsm for Bayesian inference

This thesis addresses several issues appearing in Bayesian statistics. Firstly, computations for approximating Bayesian posteriors are often performed using Markov chain Monte Carlo (MCMC) methods. However, standard MCMC algorithms tend to perform poorly when the posterior distribution has multiple modes. Secondly, performance of MCMC methods heavily depends on the choice of hyperparameters. It is therefore beneficial to design adaptive methods which update the hyperparameters on the fly, as the algorithm runs. Another topic explored in this thesis is associated with potential misspecification of the considered Bayesian model. It has been well-understood that standard Bayesian inference, when applied to misspecified models, leads to misleading uncertainty quantification. This issue motivates departures from the standard Bayesian inference, and looking into more robust approaches. In Chapter 1 we provide an introduction to the thesis and the literature review covering the above topics. In Chapter 2, based on Pompe et al. (2020), we propose a new Monte Carlo method for sampling from multimodal distributions. The idea of this technique is based on splitting the task into two: finding the modes of a target distribution π and sampling, given the knowledge of the locations of the modes. The sampling algorithm relies on steps of two types: local ones, preserving the mode; and jumps to regions associated with different modes. Moreover, the method learns the optimal parameters of the algorithm as it runs, without requiring user intervention. In order to design an adaptive scheme that facilitates both local and jump moves, we introduce an auxiliary variable representing each mode and we define a new target distribution π˜ on an augmented state space X × I, where X is the original state space of π and I is the set of the modes. As the algorithm runs and updates its parameters, the target distribution π˜ also keeps being modified. This motivates a new class of algorithms, Auxiliary Variable Adaptive MCMC, generalizing Adaptive MCMC. We prove ergodic results for the class and apply them to our specific algorithm. Chapter 3 is based on Pompe (2021). We propose Posterior Bootstrap, a set of algorithms extending Weighted Likelihood Bootstrap, to properly incorporate prior information and address the problem of model misspecification in Bayesian inference. We consider two approaches to incorporating prior knowledge: the first is based on penalization of the Weighted Likelihood Bootstrap objective function, and the second uses pseudo-samples from the prior predictive distribution. We also propose methodology for hierarchical models, which was not previously considered for methods based on Weighted Likelihood Bootstrap. We use Edgeworth expansions to gain new insight on properties of Weighted Likelihood Bootstrap and its extensions, as well as to guide the choice of hyperparameters in our algorithms. In Chapter 4, based on Pompe and Jacob (2021), we focus on Bayesian models built of several components, called modules. Under standard Bayesian inference, misspecification of any module might propagate to other parts of the model, leading to unsatisfactory results. Cut distributions have been proposed as a remedy, where the information is prevented from flowing along certain directions. We consider cut distributions from an asymptotic perspective, find the equivalent of the Laplace approximation, and notice a lack of frequentist coverage for the associated credible regions. We propose algorithms based on the Posterior Bootstrap that deliver credible regions with the nominal frequentist asymptotic coverage. In Chapter 5 we provide a discussion and indicate potential future research directions.