Applying Levenberg-Marquardt algorithm with block-diagonal Hessian approximation to recurrent neural network training.

by Chi-cheong Szeto.Thesis (M.Phil.)--Chinese University of Hong Kong, 1999.Includes bibliographical references (leaves 162-165).Abstracts in English and Chinese.Abstract --- p.iAcknowledgment --- p.iiTable of Contents --- p.iiiChapter Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Time series prediction --- p.1Chapter 1.2 --- Forecasting models --- p.1Chapter 1.2.1 --- Networks using time delays --- p.2Chapter 1.2.1.1 --- Model description --- p.2Chapter 1.2.1.2 --- Limitation --- p.3Chapter 1.2.2 --- Networks using context units --- p.3Chapter 1.2.2.1 --- Model description --- p.3Chapter 1.2.2.2 --- Limitation --- p.6Chapter 1.2.3 --- Layered fully recurrent networks --- p.6Chapter 1.2.3.1 --- Model description --- p.6Chapter 1.2.3.2 --- Our selection and motivation --- p.8Chapter 1.2.4 --- Other models --- p.8Chapter 1.3 --- Learning methods --- p.8Chapter 1.3.1 --- First order and second order methods --- p.9Chapter 1.3.2 --- Nonlinear least squares methods --- p.11Chapter 1.3.2.1 --- Levenberg-Marquardt method ´ؤ our selection and motivation --- p.13Chapter 1.3.2.2 --- Levenberg-Marquardt method - algorithm --- p.13Chapter 1.3.3 --- "Batch mode, semi-sequential mode and sequential mode of updating" --- p.15Chapter 1.4 --- Jacobian matrix calculations in recurrent networks --- p.15Chapter 1.4.1 --- RTBPTT-like Jacobian matrix calculation --- p.15Chapter 1.4.2 --- RTRL-like Jacobian matrix calculation --- p.17Chapter 1.4.3 --- Comparison between RTBPTT-like and RTRL-like calculations --- p.18Chapter 1.5 --- Computation complexity reduction techniques in recurrent networks --- p.19Chapter 1.5.1 --- Architectural approach --- p.19Chapter 1.5.1.1 --- Recurrent connection reduction method --- p.20Chapter 1.5.1.2 --- Treating the feedback signals as additional inputs method --- p.20Chapter 1.5.1.3 --- Growing network method --- p.21Chapter 1.5.2 --- Algorithmic approach --- p.21Chapter 1.5.2.1 --- History cutoff method --- p.21Chapter 1.5.2.2 --- Changing the updating frequency from sequential mode to semi- sequential mode method --- p.22Chapter 1.6 --- Motivation for using block-diagonal Hessian matrix --- p.22Chapter 1.7 --- Objective --- p.23Chapter 1.8 --- Organization of the thesis --- p.24Chapter Chapter 2 --- Learning with the block-diagonal Hessian matrix --- p.25Chapter 2.1 --- Introduction --- p.25Chapter 2.2 --- General form and factors of block-diagonal Hessian matrices --- p.25Chapter 2.2.1 --- General form of block-diagonal Hessian matrices --- p.25Chapter 2.2.2 --- Factors of block-diagonal Hessian matrices --- p.27Chapter 2.3 --- Four particular block-diagonal Hessian matrices --- p.28Chapter 2.3.1 --- Correlation block-diagonal Hessian matrix --- p.29Chapter 2.3.2 --- One-unit block-diagonal Hessian matrix --- p.35Chapter 2.3.3 --- Sub-network block-diagonal Hessian matrix --- p.35Chapter 2.3.4 --- Layer block-diagonal Hessian matrix --- p.36Chapter 2.4 --- Updating methods --- p.40Chapter Chapter 3 --- Data set and setup of experiments --- p.41Chapter 3.1 --- Introduction --- p.41Chapter 3.2 --- Data set --- p.41Chapter 3.2.1 --- Single sine --- p.41Chapter 3.2.2 --- Composite sine --- p.42Chapter 3.2.3 --- Sunspot --- p.43Chapter 3.3 --- Choices of recurrent neural network parameters and initialization methods --- p.44Chapter 3.3.1 --- "Choices of numbers of input, hidden and output units" --- p.45Chapter 3.3.2 --- Initial hidden states --- p.45Chapter 3.3.3 --- Weight initialization method --- p.45Chapter 3.4 --- Method of dealing with over-fitting --- p.47Chapter Chapter 4 --- Updating methods --- p.48Chapter 4.1 --- Introduction --- p.48Chapter 4.2 --- Asynchronous updating method --- p.49Chapter 4.2.1 --- Algorithm --- p.49Chapter 4.2.2 --- Method of study --- p.50Chapter 4.2.3 --- Performance --- p.51Chapter 4.2.4 --- Investigation on poor generalization --- p.52Chapter 4.2.4.1 --- Hidden states --- p.52Chapter 4.2.4.2 --- Incoming weight magnitudes of the hidden units --- p.54Chapter 4.2.4.3 --- Weight change against time --- p.56Chapter 4.3 --- Asynchronous updating with constraint method --- p.68Chapter 4.3.1 --- Algorithm --- p.68Chapter 4.3.2 --- Method of study --- p.69Chapter 4.3.3 --- Performance --- p.70Chapter 4.3.3.1 --- Generalization performance --- p.70Chapter 4.3.3.2 --- Training time performance --- p.71Chapter 4.3.4 --- Hidden states and incoming weight magnitudes of the hidden units --- p.73Chapter 4.3.4.1 --- Hidden states --- p.73Chapter 4.3.4.2 --- Incoming weight magnitudes of the hidden units --- p.73Chapter 4.4 --- Synchronous updating methods --- p.84Chapter 4.4.1 --- Single λ and multiple λ's synchronous updating methods --- p.84Chapter 4.4.1.1 --- Algorithm of single λ synchronous updating method --- p.84Chapter 4.4.1.2 --- Algorithm of multiple λ's synchronous updating method --- p.85Chapter 4.4.1.3 --- Method of study --- p.87Chapter 4.4.1.4 --- Performance --- p.87Chapter 4.4.1.5 --- Investigation on long training time: analysis of λ --- p.89Chapter 4.4.2 --- Multiple λ's with line search synchronous updating method --- p.97Chapter 4.4.2.1 --- Algorithm --- p.97Chapter 4.4.2.2 --- Performance --- p.98Chapter 4.4.2.3 --- Comparison of λ --- p.100Chapter 4.5 --- Comparison between asynchronous and synchronous updating methods --- p.101Chapter 4.5.1 --- Final training time --- p.101Chapter 4.5.2 --- Computation load per complete weight update --- p.102Chapter 4.5.3 --- Convergence speed --- p.103Chapter 4.6 --- Comparison between our proposed methods and the gradient descent method with adaptive learning rate and momentum --- p.111Chapter Chapter 5 --- Number and sizes of the blocks --- p.113Chapter 5.1 --- Introduction --- p.113Chapter 5.2 --- Performance --- p.113Chapter 5.2.1 --- Method of study --- p.113Chapter 5.2.2 --- Trend of performance --- p.115Chapter 5.2.2.1 --- Asynchronous updating method --- p.115Chapter 5.2.2.2 --- Synchronous updating method --- p.116Chapter 5.3 --- Computation load per complete weight update --- p.116Chapter 5.4 --- Convergence speed --- p.117Chapter 5.4.1 --- Trend of inverse of convergence speed --- p.117Chapter 5.4.2 --- Factors affecting the convergence speed --- p.117Chapter Chapter 6 --- Weight-grouping methods --- p.125Chapter 6.1 --- Introduction --- p.125Chapter 6.2 --- Training time and generalization performance of different weight-grouping methods --- p.125Chapter 6.2.1 --- Method of study --- p.125Chapter 6.2.2 --- Performance --- p.126Chapter 6.3 --- Degree of approximation of block-diagonal Hessian matrix with different weight- grouping methods --- p.128Chapter 6.3.1 --- Method of study --- p.128Chapter 6.3.2 --- Performance --- p.128Chapter Chapter 7 --- Discussion --- p.150Chapter 7.1 --- Advantages and disadvantages of using block-diagonal Hessian matrix --- p.150Chapter 7.1.1 --- Advantages --- p.150Chapter 7.1.2 --- Disadvantages --- p.151Chapter 7.2 --- Analysis of computation complexity --- p.151Chapter 7.2.1 --- Trend of computation complexity of each calculation --- p.154Chapter 7.2.2 --- Batch mode of updating --- p.155Chapter 7.2.3 --- Sequential mode of updating --- p.155Chapter 7.3 --- Analysis of storage complexity --- p.156Chapter 7.3.1 --- Trend of storage complexity of each set of variables --- p.157Chapter 7.3.2 --- Trend of overall storage complexity --- p.157Chapter 7.4 --- Parallel implementation --- p.158Chapter 7.5 --- Alternative implementation of weight change constraint --- p.158Chapter Chapter 8 --- Conclusions --- p.160References --- p.16