Variable selection in multivariate data analysis using regularization

We provide methods that find sparse projection directions in a class of multivariate analysis methods, which numerically amount to a generalized eigenvalue problem. Here sparse means that the direction vectors have many zero components. Our approach is based on regularization of generalized eigenvalue problems that produces sparse eigenvectors. We first apply the general approach to Fisher\u27s linear discriminant analysis (LDA), which is typically used as a feature extraction or dimension reduction step before classification. We further this dimension reduction technique by incorporating variable selection into LDA. When the sample size is smaller than the dimension of the input data, the standard Fisher\u27s LDA is not directly applicable. To deal with these cases, we formalize two generalizations of Fisher\u27s LDA and for these generalizations we develop methods to find the sparse projection directions. Simulation and real data examples show that our methods can not only identify the important variables, but also improve the discriminant performance in the presence of redundant variables. We also apply the general approach to supervised dimension reduction. Effective dimension reduction in the regression context seeks several projection directions such that these projections of the predictor vector contain all information about the response variable. Our general approach is applicable to a large class of existing dimension reduction methods, including SIR, SAVE and pHd. The proposed method is illustrated using simulated and real data examples. It substantially improves the accuracy in estimating the dimension reduction directions when the directions are sparse