Adaptive Dropout for Training Deep Neural Networks

Recently, it was shown that deep neural networks perform very well if the activities of hidden units are regularized during learning, e.g, by randomly dropping out 50% of their activities. We describe a method called "standout" in which a binary belief network is overlaid on a neural network and is used to regularize of its hidden units by selectively setting activities to zero. This "adaptive dropout network" can be trained jointly with the neural network by approximately computing local expectations of binary dropout variables and computing derivatives using back-propagation. Interestingly, experiments suggest that a good dropout network regularizes activities according to magnitude. When evaluated on the MNIST and NORB datasets, we found that our method achieves lower classification error rates than other feature learning methods, including standard dropout, and RBM. We also present the discriminative learning results using our method on the MNIST, NORB and CIFAR-10 datasets.M.A.S