Reliable and scalable variational inference for the hierarchical Dirichlet process

We introduce a new variational inference ob-jective for hierarchical Dirichlet process ad-mixture models. Our approach provides novel and scalable algorithms for learning nonparametric topic models of text docu-ments and Gaussian admixture models of im-age patches. Improving on the point esti-mates of topic probabilities used in previous work, we define full variational posteriors for all latent variables and optimize parameters via a novel surrogate likelihood bound. We show that this approach has crucial advan-tages for data-driven learning of the num-ber of topics. Via merge and delete moves that remove redundant or irrelevant topics, we learn compact and interpretable models with less computation. Scaling to millions of documents is possible using stochastic or memoized variational updates.