Exploiting the Statistics of Learning and Inference

Abstract. When dealing with datasets containing a billion instances or with sim-ulations that require a supercomputer to execute, computational resources be-come part of the equation. We can improve the efficiency of learning and infer-ence by exploiting their inherent statistical nature. We propose algorithms that exploit the redundancy of data relative to a model by subsampling data-cases for every update and reasoning about the uncertainty created in this process. In the context of learning we propose to test for the probability that a stochastically es-timated gradient points more than 180 degrees in the wrong direction. In the con-text of MCMC sampling we use stochastic gradients to improve the efficiency of MCMC updates, and hypothesis tests based on adaptive mini-batches to decide whether to accept or reject a proposed parameter update. Finally, we argue that in the context of likelihood free MCMC one needs to store all the information revealed by all simulations, for instance in a Gaussian process. We conclude that Bayesian methods will remain to play a crucial role in the era of big data and big simulations, but only if we overcome a number of computational challenges