Learning by Gradient Descent in Function Space

: Traditional connectionist networks have homogeneous nodes wherein each node executes the same function. Networks where each node executes a different function can be used to achieve efficient supervised learning. A modified back-propagation algorithm for such networks, which performs gradient descent in "function space," is presented and its advantages are discussed. The benefits of the suggested paradigm include faster learning and ease of interpretation of the trained network. 1 Introduction Connectionist networks (Rosenblatt, 1962; Grossberg, 1981; Rumelhart, McClelland & the PDP Research Group, 1986) are usually thought of to be graph-like interconnections of processing elements. Each processing element is capable of a simple computation such as summation or summation coupled with thresholding. In such networks, the delta rule has been used for learning linear concepts and the generalized delta rule (Rumelhart, McClelland & the PDP Research Group, 1986) has been employed for lea..