Estimating optimal feature subsets using efficient estimation of high-dimensional mutual information

Abstract—A novel feature selection method using the concept of mutual information (MI) is proposed in this paper. In all MI based feature selection methods, effective and efficient estimation of high-dimensional MI is crucial. In this paper, a pruned Parzen window estimator and the quadratic mutual information (QMI) are combined to address this problem. The results show that the proposed approach can estimate the MI in an effective and effi-cient way. With this contribution, a novel feature selection method is developed to identify the salient features one by one. Also, the appropriate feature subsets for classification can be reliably esti-mated. The proposed methodology is thoroughly tested in four dif-ferent classification applications in which the number of features ranged from less than 10 to over 15000. The presented results are very promising and corroborate the contribution of the proposed feature selection methodology. Index Terms—Feature selection, Parzen window esti-mator, quadratic mutual information (QMI), supervised data compression. I