Cascade Correlation: Derivation of a More Numerically Stable Update Rule

We discuss the weight update rule in the Cascade Correlation neural net learning algorithm. The weight update rule implements gradient descent optimization of the correlation between a new hidden unit's output and the previous network's error. We present a derivation of the gradient of the correlation function and show that our resulting weight update rule results in slightly faster training. We also show that the new rule is mathematically equivalent to the one presented in the original Cascade Correlation paper and discuss numerical issues underlying the difference in performance. Since a derivation of the Cascade Correlation weight update rule was not published, this paper should be useful to those who wish to understand the rule. In International Conference on Neural Networks, Perth, Western Australia, 1995, IEEE Press. 1. Introduction The Cascade Correlation algorithm of Fahlman & Lebiere (1990) is an elegant, simple, and powerful tool for classification and regression. It is a..