Communicated by David Haussler A Practical Bayesian Framework for Backpropagation Networks

A quantitative and practical Bayesian framework is described for learn-ing of mappings in feedforward networks. The framework makes possible (1) objective comparisons between solutions using alternative network architectures, (2) objective stopping rules for network prun-ing or growing procedures, (3) objective choice of magnitude and type of weight decay terms or additive regularizers (for penalizing large weights, etc.), (4) a measure of the effective number of well-determined parameters in a model, (5) quantified estimates of the error bars on net-work parameters and on network output, and (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian “evidence ” automatically embod-ies ”Occam’s razor,’ ’ penalizing overflexible and overcomplex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalization ability and the Bayesian evi-dence is obtained. This paper makes use of the Bayesian framework for regularization and model comparison described in the companion paper “Bayesian Inter-polation ” (MacKay 1992a). This framework is due to Gull and Skillin