Decomposing background topics from keywords by principal component pursuit

Low-dimensional topic models have been proven very use-ful for modeling a large corpus of documents that share a relatively small number of topics. Dimensionality reduc-tion tools such as Principal Component Analysis or La-tent Semantic Indexing (LSI) have been widely adopted for document modeling, analysis, and retrieval. In this pa-per, we contend that a more pertinent model for a docu-ment corpus as the combination of an (approximately) low-dimensional topic model for the corpus and a sparse model for the keywords of individual documents. For such a joint topic-document model, LSI or PCA is no longer appropriate to analyze the corpus data. We hence introduce a powerful new tool called Principal Component Pursuit that can effec-tively decompose the low-dimensional and the sparse com-ponents of such corpus data. We give empirical results on data synthesized with a Latent Dirichlet Allocation (LDA) mode to validate the new model. We then show that for real document data analysis, the new tool significantly reduces the perplexity and improves retrieval performance compared to classical baselines