DWINA: Depth and Width Incremental Neural Algorithm

This paper presents DWINA: an algorithm for depth and width design of neural architectures in the case of supervised learning with noisy data. Each new unit is trained to learn the error of the existing network and is connected to it such that it does not affect its previous performance. Criteria for choosing between increasing width or increasing depth are proposed. The connection procedure for each case is also described. The stopping criterion is very simple and consists in comparing the residual error signal to the noise signal. Preliminary experiments point out the efficacy of the algorithm especially to avoid spurious minima and to design a network with a well-suited size. The complexity of the algorithm (number of operations) is on average the same as that needed in a convergent run of the BP algorithm on a static architecture having the optimal number of parameters. Moreover, it is found that no significant difference exist between networks having the same number of parameters ..