Deep Gaussian processes

In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief net-work based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent vari-able model (GP-LVM). We perform inference in the model by approximate variational marginal-ization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically ap-plied to relatively large data sets using stochas-tic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model se-lection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.