Priors for synaptic weights improve generalization capability.

<p><b>A</b>: The training set, consisting of five samples of a handwritten <i>1</i>. Below a cartoon illustrating the network architecture of the restricted Boltzmann machine (RBM), composed of a layer of 784 visible neurons <b>x</b> and a layer of 9 hidden neurons <b>z</b>. <b>B</b>: Examples from the test set. It contains many different styles of writing that are not part of the training set. <b>C</b>: Evolution of 50 randomly selected synaptic weights throughout learning (on the training set). The weight histogram (right) shows the distribution of synaptic weights at the end of learning. 80 histogram bins were equally spaced between -4 and 4. <b>D</b>: Performance of the network in terms of log likelihood on the training set (blue) and on the test set (red) throughout learning. Mean values over 100 trial runs are shown, shaded area indicates std. The performance on the test set initially increases but degrades for prolonged learning. <b>E</b>: Evolution of 50 weights for the same network but with a bimodal prior. The prior <i>p</i>(<i>w</i>) is indicated by the blue curve. Most synaptic weights settle in the mode around 0, but a few larger weights also emerge and stabilize in the larger mode. Weight histogram (green) as in (C). <b>F</b>: The log likelihood on the test set maintains a constant high value throughout the whole learning session, compare to (D).