Comparing Adaptive and Non-Adaptive Connection Pruning With Pure Early Stopping

Abstract|Neural network pruning methods on the level of individual network parameters (e.g. connection weights) can improve generalization, as is shown in this empirical study. However, an open problem in the pruning methods known today (OBD, OBS, autoprune, epsiprune) is the selection of the number of parameters to be removed in each pruning step (pruning strength). This work presents a pruning method lprune that automatically adapts the pruning strength to the evolution of weights and loss of generalization during training. The method requires no algorithm parameter adjustmentby the user. Results of statistical signi cance tests comparing autoprune, lprune, and static networks with early stopping are given, based on extensive experimentation with 14 di erent problems. The results indicate that training with pruning is often signi cantly better and rarely signi cantly worse than training with early stopping without pruning. Furthermore, lprune is often superior to autoprune (which is superior to OBD) on diagnosis tasks unless severe pruning early in the training process is required. 1 Pruning and Generalization The principal idea of pruning is to reduce the number of free parameters in the network by removin