Decomposing & mapping neural systems onto general-purpose parallel machines

Simulations of neural systems on sequential computers are computationally expensive. For example, a single experiment for a typical financial application, e.g. exchange rate time series analysis, requires about ten hours of CPU time on a Sun workstation. Neural systems are, however inherently parallel, and would thus benefit from parallel implementations. Therefore, this thesis investigates the problem of decomposing and mapping neural systems onto general-purpose parallel machines. It presents a Mapping System capable of decomposing neural network systems, and mapping them onto a range of general-purpose parallel machines; both MIMD and SIMD. Firstly, taxonomies of neural network systems and parallel machines are provided, as well as descriptions of typical examples of both. Secondly, parallelism in neural systems and machines is analysed. The different types of parallelism exhibited by neural systems are identified. This allows a classification of neural systems on the basis of their parallelism and in accordance with their taxonomy. Parallel machines and the approaches to parallel programming are then analysed to identify the features of parallel machines involved in the mapping process. From this analysis of parallelism in neural systems and machines, the characteristics required for decomposing and mapping neural systems are identified, and a Mapping System for decomposing and mapping neural systems onto general-purpose parallel machines is described. The Mapping System consists of two modules; a machine independent Abstract Decomposition module, and a Machine Dependent Decomposition module. The Abstract Decomposition (AD) module describes a specification for neural systems. The AD specifies a finite set of schemes for decomposing neural systems according to the required exploitation of neural systems' parallelism; e.g. connection, neuron, cluster. The Machine Dependent Decomposition (MDD) analyses the decomposition schemes in conjunction with the features of parallel machines; e.g. processors' features, communications schemes, and specifies the most suitable mapping scheme to implement. To validate the Mapping System, prototype mapping software for MIMD machines has been implemented. The MIMD mapping software is able to automatically map static neural systems onto a 48- processor Parsys SN1000 Transputer machine. This mapping software was developed as part of the CEC-funded Esprit II Pygmalion Neurocomputing Project, and is incorporated in the Pygmalion Neural Network Environment. The Machine Dependent Decomposition (MDD) module is improved by the development of an analytical framework for evaluating the speedup of neural systems' mapping schemes, based on the integration of machine-dependent features with the alternative decomposition schemes. The various mapping schemes for the classical backpropagation neural systems were hand-coded onto a 64x64-processor Distributed Array of Processors (DAP). The analytical framework is then used to evaluate the speedups of the different mapping schemes. This shows that the expected speedups agree with the results obtained when implementing the mapping schemes. A formal specification for neural network systems which uses the Abstract Syntax Notation One (ASN. 1) as the syntactic construct is then presented. The innovative use of ASN. l, previously dedicated to the specification of communication protocols by the Open System Interconnection, provides a formal basis for specifying neural systems and their parallelism. This thesis develops a solution for decomposing and mapping neural systems onto general-purpose parallel machines. Working top down, this thesis makes three major contributions: i) the innovative use of ASN. l as the notational support for specifying neural systems with explicit support for parallelism, ii) an analytical framework for evaluating the speedup of alternative neural system' mappings, and iii) a general-purpose Mapping System for mapping neural systems onto parallel machines