Incremental Variational Inference applied to Latent Dirichlet Allocation

We introduce incremental variational inference, which generalizes incremental EM and provides an alternative to stochastic variational inference. It also natu-rally extends to the distributed setting. We apply incremental variational infer-ence to LDA and show that there is a benefit to do multiple passes over the data in the large-scale setting. Incremental inference does not require to set a learning rate, converges faster to a local optimum of the variational bound and enjoys the attractive property of monotonically increasing it like its batch counterpart.