Determining the number of clusters via iterative consensus clustering

We use a cluster ensemble to determine the num-ber of clusters, k, in a group of data. A consen-sus similarity matrix is formed from the ensem-ble using multiple algorithms and several val-ues for k. A random walk is induced on the graph defined by the consensus matrix and the eigenvalues of the associated transition proba-bility matrix are used to determine the num-ber of clusters. For noisy or high-dimensional data, an iterative technique is presented to re-fine this consensus matrix in way that encour-ages a block-diagonal form. It is shown that the resulting consensus matrix is generally superior to existing similarity matrices for this type of spectral analysis.