A Flexible Framework for Consensus Clustering

A novel framework for consensus clustering is presented which has the ability to determine both the number of clusters and a final solution us-ing multiple algorithms. A consensus similarity matrix is formed from an ensemble using multiple algorithms and several values for k. A variety of dimension reduction techniques and clustering algorithms are considered for analysis. For noisy or high-dimensional data, an iterative technique is presented to refine this consensus matrix in way that encourages algo-rithms to agree upon a common solution. We utilize the theory of nearly uncoupled Markov chains to determine the number, k, of clusters in a dataset by considering a random walk on the graph defined by the consen-sus matrix. The eigenvalues of the associated transition probability ma-trix are used to determine the number of clusters. This method succeeds at determining the number of clusters in many datasets where previous methods fail. On every considered dataset, our consensus method pro-vides a final result with accuracy well above the average of the individual algorithms.