Sparse Variational Inference for Generalized Gaussian Process Models

Gaussian processes (GP) provide an attrac-tive machine learning model due to their non-parametric form, their flexibility to capture many types of observation data, and their generic infer-ence procedures. Sparse GP inference algorithms address the cubic complexity of GPs by focusing on a small set of pseudo-samples. To date, such approaches have focused on the simple case of Gaussian observation likelihoods. This paper de-velops a variational sparse solution for GPs under general likelihoods by providing a new character-ization of the gradients required for inference in terms of individual observation likelihood terms. In addition, we propose a simple new approach for optimizing the sparse variational approxima-tion using a fixed point computation. We demon-strate experimentally that the fixed point operator acts as a contraction in many cases and therefore leads to fast convergence. An experimental eval-uation for count regression, classification, and or-dinal regression illustrates the generality and ad-vantages of the new approach. 1