SELF-ADAPTIVE SUPPORT VECTOR MACHINES SELF ADAPTIVE SUPPORT VECTOR MACHINES AND AUTOMATIC FEATURE SELECTION By

We handle the problem of model and feature selection for Support Vector Machines (SVMs) in this thesis. To select the best model for a given training set, we embed the standard convex quadratic problem of SVM training in an upper level problem where the optimal objective value of the quadratic problem is minimized over the feasible range of the kernel parameters. This leads to a bi-level optimization problem for which the optimal solution al-ways exists. The problem of feature selection can be solved simultaneously under the same framework except that the kernel function is extended by introducing independent kernel parameters for each features in the original space. Two solution strategies to solve the bi-level problems of this Self-Adaptive SVMs (SASVMs) are studied. Under the ¯rst strategy, the lower level prob-lems are replaced by their corresponding Karush-Kuhn-Tucker (KKT) con-ditions. Then, the bi-level problems are solved as non-linear non-convex one-level problems by general non-linear solvers. Experiment results show that this strategy can only handle small-sized training sets. The second strategy is designed around the derivative free optimization (DFO) method. The main idea of this strategy is to minimize a quadratic model, which is constructed from the solution of the lower level problem at discrete values of kernel parameters, of the optimal objective value over a controlled range of kernel parameter. At last, SASVMs are applied to several benchmark data sets. They are multi-spectral and hyper-spectral Remote Sensing images, the MNIST data set of hand-written digits and breast cancer classi¯cation data set. The accuracies of SASVMs are compared with those achieved by other classi¯ers. i