Kernel Learning at the First Level of Inference

Kernel learning methods, whether Bayesian or frequentist, typically involve mul-tiple levels of inference, with the coefficients of the kernel expansion being deter-mined at the first level and the kernel and regularisation parameters carefully tuned at the second level, a process known as model selection. Model selec-tion for kernel machines is commonly performed via optimisation of a suitable model selection criterion, often based on cross-validation or theoretical perfor-mance bounds. However, if there are a large number of kernel parameters, as for instance in the case of automatic relevance determination (ARD), there is a substantial risk of over-fitting the model selection criterion, resulting in poor generalisation performance. In this paper we investigate the possibility of learning the kernel, for the Least-Squares Support Vector Machine (LS-SVM) classifier, at the first level of inference, i.e. parameter optimisation. The kernel parameters and the coefficients of the kernel expansion are jointly optimised at the first level of inference, minimising a training criterion with an additiona