How to Pretrain Deep Boltzmann Machines in Two Stages

Abstract A deep Boltzmann machine (DBM) is a recently introduced Markov ran-dom field model that has multiple layers of hidden units. It has been shown empiri-cally that it is difficult to train a DBM with approximate maximum-likelihood learn-ing using the stochastic gradient unlike its simpler special case, restricted Boltzmann machine (RBM). In this paper, we propose a novel pretraining algorithm that con-sists of two stages; obtaining approximate posterior distributions over hidden units from a simpler model and maximizing the variational lower-bound given the fixed hidden posterior distributions. We show empirically that the proposed method over-comes the difficulty in training DBMs from randomly initialized parameters and results in a better, or comparable, generative model when compared to the conven-tional pretraining algorithm.