Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network

Introduction Backpropagation and contrastive Hebbian learning (CHL) are two supervised learning algorithms for training networks with hidden neurons. They are of interest, because they are generally applicable to wide classes of network architectures. In backpropagation (Rumelhart, Hinton, & Williams, 1986b, 1986a), an error signal for the output neurons is computed and propagated back into the hidden neurons through a separate teacher network. Synaptic weights are updated based on the product between the error signal and network activities. CHL updates the synaptic weights based on the steady states of neurons in two different phases: one with the output neurons clamped to the desired values and the other with the output neurons free (Movellan, 1990; Baldi & Pineda, 1991). Clamping the output neurons Neural Computation 15, 441--454 (2003) c 2002 Massachusetts Institute of Technology causes the hidden neurons to change their activities, and this change constitutes the basis for th