Approximating Bayes: Inference and Modeling

This thesis focuses on sources of error in modern Bayesian analysis and machine learning in the ``big data'' era. We develop new theoretical tools for analyzing and controlling different sources of error. Our work leads to new theory and methodology for providing performance guarantees for modern Bayesian methods and machine learning algorithms. The first two contributions of this thesis are new tools for studying the complexity/hardness of achieving approximation guarantees for Markov chain Monte Carlo (MCMC) in high-dimensional settings. The third contribution of this thesis is a theoretical framework for Bayesian analysis in the face of model misspecification that makes the analysis of different practical Bayesian methods possible. The fourth contribution of this thesis is a bridge between PAC-Bayes theory and Rademacher complexity, two important theoretical tools for developing the so-called generalization bounds in machine learning theory. Through the analysis of various sources of error, this thesis makes contributions to the study of: convergence of high-dimensional MCMC; optimization of high-dimensional MCMC; Bayesian model misspecification; and statistical learning theory.Ph.D