Theoretical and Experimental Aspects of Supervised Learning in Artificial Neural Networks

The topic of supervised learning within the conceptual framework of artificial neural network (ANN) models is addressed. An ANN is a parallel distributed processing system that consists of many computationally simple processing elements interconnected through unidirectional weighted connections. Such networks, which are roughly patterned after biological nervous systems, have been proposed for use in areas in which the traditional von Neumann computer architecture has been relatively unsuccessful. Learning in these networks is accomplished through the use of algorithms that adjust the values of the connection weights. The work presented here addresses the issue of improving the rate at which ANNs can learn to achieve the mapping of an input pattern to a desired output pattern. The most successful learning algorithms for accomplishing this task are based on gradient descent error minimization techniques. However, the large amount of training time that such algorithms require is currently one of the factors that limits the use of ANN systems. A number of improvements to existing algorithms were developed and analyzed through the use of simulation. Furthermore, a highly flexible simulation environment based on an object-oriented software paradigm was developed and successfully used to implement and test these novel learning algorithms. A comparison of this object-oriented approach with the traditional techniques used by existing ANN simulation systems is performed. Finally, the issue of simulating ANN models on parallel hardware is considered. An associative memory ANN was implemented on a systolic array architecture. The memory requirements and processing times of this implementation are analyzed based upon the number of processing elements required by the ANN and the number of training patterns. Compared with other digital implementations, this design is shown to yield significant improvements in runtime performance and offers the capability of using ANN associative memories in real-time applications