A Scalable Sparse Distributed Neural Memory Model

For a number of years, artificial neural networks have been used for a variety of applications to automate tasks not suitable for the conventional computing model, such as pattern­recognition. Their inherent non­linearity and parallelism makes them suitable for approximating a variety of functions and graphs. This work is an attempt to develop a scalable sparse distributed neural mem­ory model, which is capable of storing and retrieving sequences of patterns. The model developed is robust, scalable, generic, self error correcting, with low av­erage neural activity resulting in lower power usage, and capable of learning in one ­shot. Patterns are encoded in the memory as an ordered code. The model is inspired by Pentti Kanerva's book on sparse distributed memories, in which he described a memory model where data are stored as addresses in the memory, and which could be implemented through neurons. The basic neural model used is a modified version of Kanerva's model, which uses a special encoding scheme to encode information. N of M codes are used in this scheme, and increased information content is attempted through rank order coding. An experimental investigation is made into the capacity of such a mem­ory, and its ability to store and retrieve sequences of characters and patterns. An attempt is made to incorporate some form of state information in the memory, so that it can store and retrieve multiple sequences having some similar components, and retrieve a whole sequence on being given a part of a sequence, or even one with errors. In this work, a comparative analysis has been done of various memory models with respect to information bearing capacity, state space and ease of hardware im­plementation. A number of experiments have been done on the resulting memory model to determine its memory capacity and the ability to recognise sequences of characters. A discussion has been made of the issues involving the dynamics of neurons in large systems with feedback. Possible applications of the model have also been explored. The memory model developed has got certain similarities with the cerebellar cortex, a part of the mammalian brain specialising in sensory­motor coordination. Thus it is neurobiologically plausible. It tries to show it is possible to build a large scale neural network involving millions of neurons and utilise it to do useful tasks. The work should be of interest to neural network researchers as well as those seeking a new computational model having more robust and scalable characteristics