Optimized clustering with modified K-means algorithm

huge data is a big challenge. Clustering technique is able to find hidden patterns and to extract useful information from huge data. Among the techniques, the k-means algorithm is the most commonly used technique for determining optimal number of clusters (k). However, the choice of k is a prominent problem in the process of the k-means algorithm. In most cases, for clustering huge data, k is pre-determined by researchers and incorrectly chosen k, could end with wrong interpretation of clusters and increase computational cost. Besides, huge data often face with correlated variables which lead to incorrect clustering process. In order to obtain the optimum number of clusters and at the same time could deal with correlated variables in huge data, modified k-means algorithm was proposed. The proposed algorithm utilised a distance measure to compute the between groups’ separation to accelerate the process of identifying an optimal number of clusters using k-means. Two distance measures were considered namely Euclidean and Manhattan distances. In dealing with correlated variables, PCA was embedded in the proposed algorithm. The developed algorithms were tested on uncorrelated and correlated simulated data sets, generated under various conditions. Besides, some real data sets were examined to validate the proposed algorithm. Empirical evidences based on simulated data sets indicated that the proposed modified k-means algorithm is able to recognise the optimum number of clusters for uncorrelated data sets. While, the PCA based on modified k-means managed to identify the optimum number of clusters for correlated data sets. Also, the results revealed that the modified k-means algorithm with Euclidean distance yields optimum number of clusters compared to the Manhattan distance. Testing on real data sets showed consistency results as the simulated ones. Generally, the proposed modified k-means algorithm is able to determine the optimum number of clusters for huge data