Locally connected recurrent neural networks.

by Evan, Fung-yu Young.Thesis (M.Phil.)--Chinese University of Hong Kong, 1993.Includes bibliographical references (leaves 161-166).List of Figures --- p.viList of Tables --- p.viiList of Graphs --- p.viiiAbstract --- p.ixChapter Part I --- Learning AlgorithmsChapter 1 --- Representing Time in Connectionist Models --- p.1Chapter 1.1 --- Introduction --- p.1Chapter 1.2 --- Temporal Sequences --- p.2Chapter 1.2.1 --- Recognition Tasks --- p.2Chapter 1.2.2 --- Reproduction Tasks --- p.3Chapter 1.2.3 --- Generation Tasks --- p.4Chapter 1.3 --- Discrete Time v.s. Continuous Time --- p.4Chapter 1.4 --- Time Delay Neural Network (TDNN) --- p.4Chapter 1.4.1 --- Delay Elements in the Connections --- p.5Chapter 1.4.2 --- NETtalk: An Application of TDNN --- p.7Chapter 1.4.3 --- Drawbacks of TDNN --- p.8Chapter 1.5 --- Networks with Context Units --- p.8Chapter 1.5.1 --- Jordan's Network --- p.9Chapter 1.5.2 --- Elman's Network --- p.10Chapter 1.5.3 --- Other Architectures --- p.14Chapter 1.5.4 --- Drawbacks of Using Context Units --- p.15Chapter 1.6 --- Recurrent Neural Networks --- p.16Chapter 1.6.1 --- Hopfield Models --- p.17Chapter 1.6.2 --- Fully Recurrent Neural Networks --- p.20Chapter A. --- EXAMPLES OF USING RECURRENT NETWORKS --- p.22Chapter 1.7 --- Our Objective --- p.25Chapter 2 --- Learning Algorithms for Recurrent Neural Networks --- p.27Chapter 2.1 --- Introduction --- p.27Chapter 2.2 --- Gradient Descent Methods --- p.29Chapter 2.2.1 --- Backpropagation Through Time (BPTT) --- p.29Chapter 2.2.2 --- Real Time Recurrent Learning Rule (RTRL) --- p.30Chapter A. --- RTRL WITH TEACHER FORCING --- p.32Chapter B. --- TERMINAL TEACHER FORCING --- p.33Chapter C. --- CONTINUOUS TIME RTRL --- p.33Chapter 2.2.3 --- Variants of RTRL --- p.34Chapter A. --- SUB GROUPED RTRL --- p.34Chapter B. --- A FIXED SIZE STORAGE 0(n3) TIME COMPLEXITY LEARNGING RULE --- p.35Chapter 2.3 --- Non-Gradient Descent Methods --- p.37Chapter 2.3.1 --- Neural Bucket Brigade (NBB) --- p.37Chapter 2.3.2 --- Temporal Driven Method (TO) --- p.38Chapter 2.4 --- Comparison between Different Approaches --- p.39Chapter 2.5 --- Conclusion --- p.41Chapter 3 --- Locally Connected Recurrent Networks --- p.43Chapter 3.1 --- Introduction --- p.43Chapter 3.2 --- Locally Connected Recurrent Networks --- p.44Chapter 3.2.1 --- Network Topology --- p.44Chapter 3.2.2 --- Subgrouping --- p.46Chapter 3.2.3 --- Learning Algorithm --- p.47Chapter 3.2.4 --- Continuous Time Learning Algorithm --- p.50Chapter 3.3 --- Analysis --- p.51Chapter 3.3.1 --- Time Complexity --- p.51Chapter 3.3.2 --- Space Complexity --- p.51Chapter 3.3.3 --- Local Computations in Time and Space --- p.51Chapter 3.4 --- Running on Parallel Architectures --- p.52Chapter 3.4.1 --- Mapping the Algorithm to Parallel Architectures --- p.52Chapter 3.4.2 --- Parallel Learning Algorithm --- p.53Chapter 3.4.3 --- Analysis --- p.54Chapter 3.5 --- Ring-Structured Recurrent Network (RRN) --- p.55Chapter 3.6 --- Comparison between RRN and RTRL in Sequence Recognition --- p.55Chapter 3.6.1 --- Training Sets and Testing Sequences --- p.56Chapter 3.6.2 --- Comparison in Training Speed --- p.58Chapter 3.6.3 --- Comparison in Recalling Power --- p.59Chapter 3.7 --- Comparison between RRN and RTRL in Time Series Prediction --- p.59Chapter 3.7.1 --- Comparison in Training Speed --- p.62Chapter 3.7.2 --- Comparison in Predictive Power --- p.63Chapter 3.8 --- Conclusion --- p.65Chapter Part II --- ApplicationsChapter 4 --- Sequence Recognition by Ring-Structured Recurrent Networks --- p.67Chapter 4.1 --- Introduction --- p.67Chapter 4.2 --- Related Works --- p.68Chapter 4.2.1 --- Feedback Multilayer Perceptron (FMLP) --- p.68Chapter 4.2.2 --- Back Propagation Unfolded Recurrent Rule (BURR) --- p.69Chapter 4.3 --- Experimental Details --- p.71Chapter 4.3.1 --- Network Architecture --- p.71Chapter 4.3.2 --- Input/Output Representations --- p.72Chapter 4.3.3 --- Training Phase --- p.73Chapter 4.3.4 --- Recalling Phase --- p.73Chapter 4.4 --- Experimental Results --- p.74Chapter 4.4.1 --- Temporal Memorizing Power --- p.74Chapter 4.4.2 --- Time Warping Performance --- p.80Chapter 4.4.3 --- Fault Tolerance --- p.85Chapter 4.4.4 --- Learning Rate --- p.87Chapter 4.5 --- Time Delay --- p.88Chapter 4.6 --- Conclusion --- p.91Chapter 5 --- Time Series Prediction --- p.92Chapter 5.1 --- Introduction --- p.92Chapter 5.2 --- Modelling in Feedforward Networks --- p.93Chapter 5.3 --- Methodology with Recurrent Networks --- p.94Chapter 5.3.1 --- Network Structure --- p.94Chapter 5.3.2 --- Model Building - Training --- p.95Chapter 5.3.3 --- Model Diagnosis - Testing --- p.95Chapter 5.4 --- Training Paradigms --- p.96Chapter 5.4.1 --- A Quasiperiodic Series with White Noise --- p.96Chapter 5.4.2 --- A Chaotic Series --- p.97Chapter 5.4.3 --- Sunspots Numbers --- p.98Chapter 5.4.4 --- Hang Seng Index --- p.99Chapter 5.5 --- Experimental Results and Discussions --- p.99Chapter 5.5.1 --- A Quasiperiodic Series with White Noise --- p.101Chapter 5.5.2 --- Logistic Map --- p.103Chapter 5.5.3 --- Sunspots Numbers --- p.105Chapter 5.5.4 --- Hang Seng Index --- p.109Chapter 5.6 --- Conclusion --- p.112Chapter 6 --- Chaos in Recurrent Networks --- p.114Chapter 6.1 --- Introduction --- p.114Chapter 6.2 --- Important Features of Chaos --- p.115Chapter 6.2.1 --- First Return Map --- p.115Chapter 6.2.2 --- Long Term Unpredictability --- p.117Chapter 6.2.3 --- Sensitivity to Initial Conditions (SIC) --- p.118Chapter 6.2.4 --- Strange Attractor --- p.119Chapter 6.3 --- Chaotic Behaviour in Recurrent Networks --- p.120Chapter 6.3.1 --- Network Structure --- p.121Chapter 6.3.2 --- Dynamics in Training --- p.121Chapter 6.3.3 --- Dynamics in Testing --- p.122Chapter 6.4 --- Experiments and Discussions --- p.123Chapter 6.4.1 --- Henon Model --- p.123Chapter 6.4.2 --- Lorenz Model --- p.127Chapter 6.5 --- Conclusion --- p.134Chapter 7 --- Conclusion --- p.135Appendix A Series 1 Sine Function with White Noise --- p.137Appendix B Series 2 Logistic Map --- p.138Appendix C Series 3 Sunspots Numbers from 1700 to 1979 --- p.139Appendix D A Quasiperiodic Series with White Noise --- p.141Appendix E Hang Seng Daily Closing Index in 1991 --- p.142Appendix F Network Model for the Quasiperiodic Series with White Noise --- p.143Appendix G Network Model for the Logistic Map --- p.144Appendix H Network Model for the Sunspots Numbers --- p.145Appendix I Network Model for the Hang Seng Index --- p.146Appendix J Henon Model --- p.147Appendix K Network Model for the Henon Map --- p.150Appendix L Lorenz Model --- p.151Appendix M Network Model for the Lorenz Map --- p.159Bibliography --- p.16