Stochastic Gradient Estimate Variance in Contrastive Divergence and Persistent Contrastive Divergence Mathias Berglund

Contrastive Divergence (CD) and Persistent Con-trastive Divergence (PCD) are popular methods for training the weights of Restricted Boltzmann Machines. However, both methods use an ap-proximate method for sampling from the model distribution. As a side effect, these approxima-tions yield significantly different biases and vari-ances for stochastic gradient estimates of indi-vidual data points. It is well known that CD yields a biased gradient estimate. In this pa-per we however show empirically that CD has a lower stochastic gradient estimate variance than exact sampling, while the mean of subsequent PCD estimates has a higher variance than exac