Comparing the inductive biases of simple neural networks and Bayesian models

Understanding the relationship between connectionist and probabilistic models is important for evaluating the compati-bility of these approaches. We use mathematical analyses and computer simulations to show that a linear neural network can approximate the generalization performance of a probabilis-tic model of property induction, and that training this network by gradient descent with early stopping results in similar per-formance to Bayesian inference with a particular prior. How-ever, this prior differs from distributions defined using discrete structure, suggesting that neural networks have inductive bi-ases that can be differentiated from probabilistic models with structured representations