Accelerated learning for Restricted Boltzmann Machine with momentum term

Abstract. Restricted Boltzmann Machines are generative models which can be used as standalone feature extractors, or as a parameter ini-tialization for deeper models. Typically, these models are trained using Contrastive Divergence algorithm, an approximation of the stochastic gradient descent method. In this paper, we aim at speeding up the con-vergence of the learning procedure by applying the momentum method and the Nesterov’s accelerated gradient technique. We evaluate these two techniques empirically using the image dataset MNIST