Efficient Algorithm to Find Information Rich Subset in High Dimensional Data

cluster analysis of data with anywhere from a few dozens to many thousands of dimensions. High-dimensional data spaces are often encountered in areas such as medicine, where DNA micro array technology can produce a large number of measurements at once and the clustering of text documents where if a word-frequency vector is used, the number of dimensions equals the size of the dictionary. Subspace clustering is the task of detecting all clusters in all subspaces. It is an extension of traditional clustering that seeks to find clusters in different subspaces within a data set. In high dimensional data many of the dimensions are often irrelevant. These irrelevant dimensions confuse clustering algorithm by hiding clusters in noisy data. In very high dimensions it is common for all the instances in a dataset to be nearly equidistant from each other completely masking the clusters. Subspace clustering algorithms localize the search for relevant dimensions allowing them to find clusters that exist in multiple possibly overlapping subspaces. The existing Information Rich Subset (IRS) algorithm can find the effective information rich subspaces trends but here K-means algorithm has been used to find the suitable clusters. K-means can be applied only when the mean of a cluster is defined. This may not be the case in some applications such as when data with categorical attributes are involved. The K-means method is sensitive to noise and outlier data points because a small number of such data can substantially influence the mean value. Here a extension work has been proposed to optimize the IRS algorithm to find the suitable subspaces and also effectiveness is evaluated with the existing different subspace clustering algorithms