Continuous Hyper-parameter Learning for Support Vector Machines

Abstract. In this paper, we address the problem of determining optimal hyper-parameters for support vector machines (SVMs). The standard way for solv-ing the model selection problem is to use grid search. Grid search constitutes an exhaustive search over a pre-defined discretized set of possible parameter values and evaluating the cross-validation error un-til the best is found. We developed a bi-level opti-mization approach to solve the model selection prob-lem for linear and kernel SVMs, including the ex-tension to learn several kernel parameters. Using this method, we can overcome the discretization of the parameter space using continuous optimization, and the complexity of the method only increases lin-early with the number of parameters (instead of ex-ponentially using grid search). In experiments, we determine optimal hyper-parameters based on dif-ferent smooth estimates of the cross-validation error and find that only very few iterations of bi-level opti-mization yield good classification rates. 1