Memoized Online Variational Inference for Dirichlet Process Mixture Models

Variational inference algorithms provide the most effective framework for large-scale training of Bayesian nonparametric models. Stochastic online approaches are promising, but are sensitive to the chosen learning rate and often converge to poor local optima. We present a new algorithm, memoized online variational inference, which scales to very large (yet finite) datasets while avoiding the com-plexities of stochastic gradient. Our algorithm maintains finite-dimensional suf-ficient statistics from batches of the full dataset, requiring some additional mem-ory but still scaling to millions of examples. Exploiting nested families of varia-tional bounds for infinite nonparametric models, we develop principled birth and merge moves allowing non-local optimization. Births adaptively add components to the model to escape local optima, while merges remove redundancy and im-prove speed. Using Dirichlet process mixture models for image clustering and denoising, we demonstrate major improvements in robustness and accuracy.