A Biologically Motivated Connectionist System for Predicting the Next Word in Natural Language Sentences

Abstract-- Recent artificial neural network models lack many physiological properties of the neuron (Rocha 1992; Rosa 2001). Current learning algorithms are more oriented to computational performance than to biological credibility. The aim of this paper is to propose an artificial neural network system, called Bio-Pred, to take care of natural language processing word prediction, in a biologically inspired connectionist approach. Instead of the well-known biologically implausible back-propagation algorithm (Crick 1989; Rumelhart, Hinton, and Williams 1986), a neurophysiologically motivated one is employed (O’Reilly 1996) in a bi-directional connectionist architecture to account for next word prediction in natural language sentences. In addition, several features concerning biological plausibility are also included, for instance, distributed representations. Comparisons are made between Bio-Pred and a system that uses the same word representation and the same next word prediction (Rosa 2002). The differences lie in the architecture employed- bi-directional architecture versus simple recurrent network (Elman 1990)- and in the learning algorithm- a neurophysiologically inspired procedure versus the biologically implausible back-propagation. The main contribution of Bio-Pred is to make an attempt to restore biological inspiration of current connectionist systems