Optimizing the Evidence

Since Bayesian learning for neural networks was introduced by MacKay it was applied to real world problems with varying success. Despite of the fact that Bayesian learning provides an elegant theory to prevent neural networks from overfitting, it is not as commonly used as it should be. In this paper we focus on two problems that arise in practice: (1) The evidence p(Djff) of the hyperparameter ff does not monotonically increase during the learning process and (2) the correlation coefficient between the evidence and the generalization performance is usually positive but significantly different from 1. The latter problem is solved in practice by forming a committee of networks with reasonably high evidence, thus reducing the influence of outliers. Based on good choice of the prior of the hyperparameters, which was crucial for the convergence of the algorithm in our experiments, we exploit in the following the positive correlation between the evidence and the generalization performance b..