Empirical Evaluation of Bayesian Sampling for Neural Classifiers

Adopting a Bayesian approach and sampling the network parameters from their posterior distribution is a rather novel and promising method for improving the generalisation performance of neural network predictors. The present empirical study applies this scheme to a set of different synthetic and real-world classification problems. The paper focuses on the dependence of the prediction results on the prior distribution of the network parameters and hyperparameters, and provides a critical evaluation of the automatic relevance determination (ARD) scheme for detecting irrelevant inputs. 1 Introduction Consider a K-fold classification problem, where an m-dimensional feature vector x t is assigned to one of K classes fC 1 ; : : : ; CK g indicated by a label vector y t = (y 1 t ; : : : ; y K t ); y k t = 1 if x t 2 C k ; y k t = 0 if x t 62 C k ; ky t k = 1: For a neural network (NN) with K softmax 1 units in the final layer, the network outputs f k (x t ; w) 2 [0; 1] can be interp..