NOTE Communicated by Nicol Schraudolph Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks

The Levenberg-Marquardt (LM) learning algorithm is a popular algo-rithm for training neural networks; however, for large neural networks, it becomes prohibitively expensive in terms of running time andmemory requirements. The most time-critical step of the algorithm is the calcula-tion of the Gauss-Newton matrix, which is formed by multiplying two large Jacobian matrices together. We propose a method that uses back-propagation to reduce the time of this matrix-matrix multiplication. This reduces the overall asymptotic running time of the LM algorithm by a factor of the order of the number of output nodes in the neural network.