An adaptive learning rate for stochastic variational inference

Stochastic variational inference finds good pos-terior approximations of probabilistic models with very large data sets. It optimizes the vari-ational objective with stochastic optimization, following noisy estimates of the natural gradi-ent. Operationally, stochastic inference itera-tively subsamples from the data, analyzes the subsample, and updates parameters with a de-creasing learning rate. However, the algorithm is sensitive to that rate, which usually requires hand-tuning to each application. We solve this problem by developing an adaptive learning rate for stochastic inference. Our method re-quires no tuning and is easily implemented with computations already made in the algo-rithm. We demonstrate our approach with latent Dirichlet allocation applied to three large text corpora. Inference with the adaptive learning rate converges faster and to a better approxima-tion than the best settings of hand-tuned rates.