LETTER TO THE EDITOR On-line backpropagation in two-layered neural networks

Abstract We present an emcl analysis of ieaming a rule by on-line gradient descent in a two-layered neural network with adjustable hidden-to-output weights (backpropagation of error). Results are compared with the training of networks having the same architecmre but fixed weights in the second layer. The ability of neural networks to learn arule from examples [I] has been studied successfully in a statistical mechanics context, see e.g. [2-4] for recent reviews. So far most of the analysis has been restricted to very simple networks such as the single layer perceptron [l] or networks with one layer of hidden units and a fixed hidden-to-output relation, e.g. the so-called committee machine 131. In the following we extend the recent investigation of learning by on-line gradient descent [8, IO] to two-layered networks with adjustable weights connecting the hidden units and the output. This topic is of crucial importance as systems with variable hidden-to-output relations can realize more complex classification schemes and are commonly used in practical applications of neural networks [1,5]. The learning prescription corresponds to an on-line version of the so-calle