Unsupervised Learning for Boltzmann Machines

An unsupervised learning algorithm for a stochastic recurrent neural network based on the Boltzmann Machine architecture is formulated in this paper. The maximization of the Mutual Information between the stochastic output neurons and the clamped inputs is used as an unsupervised criterion for training the network. The resulting learning rule contains two terms corresponding to Hebbian and anti-Hebbian learning. It is interesting that these two terms are weighted by the amount of information transmitted in the learning synapse, giving an information-theoretic interpretation of the proportionality constant of Hebb's biological rule. The anti-Hebbian term, which can be interpreted as a forgetting function, supports the optimal coding. In this way, optimal non-linear and recurrent implementation of data compression of boolean patterns are obtained. As an example, the encoder problem is simulated and trained in an unsupervised way in a one layer network. Compression of nonuniform distributed binary data is included. Unsupervised classification even for continuous inputs is shown for the cases of 4 overlapping gaussian spots and for a real world example of thyroid diagnosis. In comparison with other techniques, the present model requires for the classification problem an exponentially smaller number of weights. 1.