Adaptively Setting the Learning Rate in Stochastic Variational Inference

Stochastic variational inference is a promising method for fitting large-scale probabilistic models with hidden structures. Different from traditional stochastic learning, stochastic variational inference uses the natural gradient, which is particularly efficient for computing probabilistic distributions. One of the issues in stochastic variational inference is to set an appropriate learning rate. Inspired by a recent approach for setting the learning rate for stochastic learning (Schaul et al., 2012), we present a strategy for setting the learning rate for stochastic variational inference and demonstrate it is effective in learning large-scale complex models.