Incorporating Word Correlation Knowledge into Topic Modeling

This paper studies how to incorporate the ex-ternal word correlation knowledge to improve the coherence of topic modeling. Existing topic models assume words are generated in-dependently and lack the mechanism to utilize the rich similarity relationships among words to learn coherent topics. To solve this prob-lem, we build a Markov Random Field (MRF) regularized Latent Dirichlet Allocation (LDA) model, which defines a MRF on the latent topic layer of LDA to encourage words la-beled as similar to share the same topic label. Under our model, the topic assignment of each word is not independent, but rather affected by the topic labels of its correlated words. Simi-lar words have better chance to be put into the same topic due to the regularization of MRF, hence the coherence of topics can be boosted. In addition, our model can accommodate the subtlety that whether two words are similar depends on which topic they appear in, which allows word with multiple senses to be put into different topics properly. We derive a vari-ational inference method to infer the poste-rior probabilities and learn model parameters and present techniques to deal with the hard-to-compute partition function in MRF. Exper-iments on two datasets demonstrate the effec-tiveness of our model.