Information theory and neural network learning algorithms

There have been a number of recent papers on information theory and neural networks, especially in a perceptual system such as vision.� Some of these approaches are examined, and their implications for neural network learning algorithms are considered.� Existing supervised learning algorithms such as Back Propagation to minimize mean squared error can be viewed as attempting to minimize an upper bound on information loss.� By making an assumption of noise either at the input or the output to the system, unsupervised learning algorithms such as those based on Hebbian (principal component analysing) or anti-Hebbian (decorrelating) approaches can also be viewed in a similar light.� The optimization of information by the use of interneurons to decorrelate output units suggests a role for inhibitory interneurons and cortical loops in biological sensory systems