Optimization of Gaussian Process Hyperparameters

Abstract. Gaussian processes are a powerful tool for non-parametric re-gression. Training can be realized by maximizing the likelihood of the data given the model. We show that Rprop, a fast and accurate gradient-based optimization technique originally designed for neural network learning, can outperform more elaborate unconstrained optimization methods on real world data sets, where it is able to converge more quickly and reliably to the optimal solution. 1 Gaussian Process Regression Gaussian processes (GP) are defined as a finite collection of jointly Gaussian distributed random variables. For regression problems these random variables represent the values of a function f(x) at input points x. Prior beliefs about the properties of the latent function are encoded by the mean function m(x) and covariance function k(x,x′). Thereby, all function classes that share the same prior assumptions are covered and inferences can be made directly in function space. In order to make predictions based on data, we consider the joint Gaussian prior of the noisy training observations y and the test outputs f∗. We derive the posterior distribution by conditioning the prior on the training observations, such that the conditional distribution of f ∗ only contains those functions from the prior that are consistent with the training data. Assuming the prior mean to be zero, the following predictive equations for GP regression are obtained: f̄ ∗ = K(X∗,X) K(X,X) + σ2n