Enabling scalable stochastic gradient-based inference for Gaussian processes by employing the Unbiased LInear System SolvEr (ULISSE

In applications of Gaussian processes where quantification of uncertainty is of primary in-terest, it is necessary to accurately character-ize the posterior distribution over covariance pa-rameters. This paper proposes an adaptation of the Stochastic Gradient Langevin Dynamics al-gorithm to draw samples from the posterior dis-tribution over covariance parameters with negli-gible bias and without the need to compute the marginal likelihood. In Gaussian process re-gression, this has the enormous advantage that stochastic gradients can be computed by solving linear systems only. A novel unbiased linear sys-tems solver based on parallelizable covariance matrix-vector products is developed to acceler-ate the unbiased estimation of gradients. The re-sults demonstrate the possibility to enable scal-able and exact (in a Monte Carlo sense) quantifi-cation of uncertainty in Gaussian processes with-out imposing any special structure on the covari-ance or reducing the number of input vectors. 1