Journal of Machine Learning Research x (2009) x-xx Submitted 6/08; Published xx/xx Distributed Algorithms for Topic Models

Editor: We describe distributed algorithms for two widely-used topic models, namely the Latent Dirich-let Allocation (LDA) model, and the Hierarchical Dirichet Process (HDP) model. In our distributed algorithms the data is partitioned across separate processors and inference is done in a parallel, dis-tributed fashion. We propose two distributed algorithms for LDA. The first algorithm is a straight-forward mapping of LDA to a distributed processor setting. In this algorithm processors concur-rently perform Gibbs sampling over local data followed by a global update of topic counts. The al-gorithm is simple to implement and can be viewed as an approximation to Gibbs-sampled LDA. The second version is a model that uses a hierarchical Bayesian extension of LDA to directly account for distributed data. This model has a theoretical guarantee of convergence but is more complex to implement than the first algorithm. Our distributed algorithm for HDP takes the straightforward mapping approach, and merges newly-created topics either by matching or by topic-id. Using five real-world text corpora we show that distributed learning works well in practice. For both LDA and HDP, we show that the converged test-data log probability for distributed learning is indistinguish-able from that obtained with single-processor learning. Our extensive experimental results include learning topic models for two multi-million document collections using a 1024-processor parallel computer