Document clustering using an LSI probabilistic subspace model

Feature representation and selection are major challenges for automatic document clustering. We developed a novel method based on an LSI (Latent Semantic Indexing) probabilistic subspace model for this problem. The top ranking conceptual terms or term clusters are selected to represent the corpora according to their global and local statistical contribution to the LSI term space. Then, each term or document is defined as a signature which represents the distribution of its local statistical contribution on the top ranking LSI concept dimensions. Finally, two novel similarity measures are applied between the concept signatures and the document signatures, which bridge the LSI subspaces and significantly improve the performance of the clustering algorithms such as basic K-means and Self-organizing Mapping (SOM) efficiently and effectively. The two similarity measures are also potentially useful for automatically identification of the cluster labels. The evaluation suggests that the top ranking conceptual terms dominate the performance of the two basic clustering algorithms. Moreover, the subsets generated by the algorithms allow learning to reach the maximum using even less than 12% of the overall conceptual terms