Constructive Neural Networks with Regularization

Abstract: In this paper we present a regularization approach to the training of all the network weights in cascade-correlation type constructive neural networks. Especially, the case of regularizing the output neuron of the net-work is presented. In this case, the output weights are trained by employing a regularized objective function con-taining a penalty term which is proportional to the weight values of the unit being trained. It is shown that the training can still be done with the pseudo-inverse method of linear regression if the output unit employs linear activation function. The degree of regularization and the smoothness of network mapping can be adjusted by changing the value of the regularization parameter. The investigated algorithms were Cascade-Correlation, Mod-ified Cascade-Correlation, Cascade, and Fixed Cascade Error. The algorithms having the regularization of the hidden and output units were compared with the ones having only the regularization of the hidden units and with those having no regularization at all. The simulation results show that the regularization of the output unit is highly beneficial. It leads to better generalization performance and in many cases to lower computational costs when compared to the partially and non-regulated versions of the same algorithms. Key-Words: Constructive neural networks, regularization, generalization, cascade-correlation, classification, regression