Diagrammatic Methods for Deriving and Relating Temporal Neural Network Algorithms

Introduction Deriving the appropriate gradient descent algorithm for a new network architecture or system configuration normally requires numerous chain rule expansions, diligent bookkeeping, and careful manipulation of terms. For example, the celebrated backpropagation algorithm for training feedforward neural networks was derived by repeatedly applying chain rule expansions backward through the network (Rumelhart et al., 1986; Werbos, 1974; Parker, 1982). However, the actual implementation of backpropagation may be viewed as a simple reversal of signal flow through the network. Another popular algorithm, backpropagation-throughtime for recurrent networks, can be derived by Euler-Lagrange or ordered derivative methods, and involves both a signal flow reversal and time reversal (Werbos, 1992; Nguyen and Widrow, 1989). For both these algorithms, there is a reciprocal nature to the forward propagation of states and the backward p