On-Line Learning in a Discrete State Space

On-line learning of a rule given by an N-dimensional Ising perceptron, is considered for the case when the student is constrained to take values in a discrete state space of size L N . For L = 2 no on-line algorithm can achieve a finite overlap with the teacher in the thermodynamic limit. However, if L is on the order of p N , Hebbian learning does achieve a finite overlap. Artificial neural networks are usually trained by a set of examples [4]. After the training phase such a network (="student") has achieved some knowledge about the rule (="teacher") which has generated the examples. The difference between the outputs of the student and the teacher for a random input vector defines the generalization error. There are two basic kinds of training algorithms: 1. In batch mode the complete set of examples is stored and iteratively used to change the synaptic weights of the student network. 2. In on-line mode each example is used only once. At each training step a new example is prese..