A method of training multi-layer networks with heaviside characteristics using internal representations

Two algorithms have recently been reported for training multi-layer networks of neurons with Heaviside characteristics. In these the weights are treated as continuous random variables and the output of a neuron is probabilistic with a distribution that is a differentiable function of the weight parameters. A limitation of these methods is that they are restricted to networks with two layers of variable weights. Other algorithms have been developed for this problem which use internal representations to train networks with Heaviside characteristics. However these suffer from the need to perform "bit flipping" on the internal representations in an effort to reduce the output error and as a result these methods do not guarantee that the network will converge to a solution. We present a learning algorithm that employs internal representations, which are continuous random variables, for the training of multi-layer networks whose neurons have Heaviside characteristics. This algorithm is an improvement over the others in that it is applicable to networks with any number of layers of variable weights and does not require "bit flipping" on the internal representations to reduce output error. The algorithm is extended to apply to recurrent networks. Some illustrative results are given