Automatically Determining Number of Clusters

Automatically determining number of clusters in the data is an unsolved/unexplored problem. First I'll show why we need to do this, and whether this is a reasonable problem in text clustering in particular. Then starting from simple 1-d/2-d study, I find BIC (Bayesian Information Criterion) is a useful measure, which by penalizing model fitness by model complexity, usually tells the right number of clusters. Experiments on EM clustering of 1-d/2-d data are presented. In text clustering, I find inter-document similarity matrix, when correctly organized & visualized, a very good representation of the collection. Based on that I tried a Similarity-matrix based clustering algorithm, which gives visually appealing results. However, BIC measure can't be easily extended to text clustering case. 1 Motivation 1.1 Why do we need to know the number of clusters Given a large collection of documents, you don't have time to read through them one by one. Here naturally arises the need to cluster th..