Learning with kernels

Support Vector (SV) Machines combine several techniques from statistics, machine learning and neural networks. One of the most important ingredients are kernels, i.e. the concept of transforming linear algorithms into nonlinear ones via a map into feature spaces. The present work focuses on the following issues: (1) Extensions of Support Vector Machines. (2) Extensions of kernel methods to other algorithms such as unsupervised learning. (3) Capacity bounds which are particularly well suited for kernel methods. After a brief introduction to SV regression it is shown how the classical epsilon-insensitive loss function can be replaced by other cost functions while keeping the original advantages or adding other features such as automatic parameter adaptation. Moreover the connection between kernels and regularization is pointed out. A theoretical analysis of several common kernels follows and criteria to check Mercer's condition more easily are presented. Further modifications lead to semiparametric models and greedy approximation schemes. Next three different types of optimization algorithms, namely interior point codes, subset selection algorithms, and sequential minimal optimization (including pseudocode) are presented. The primal-dual framework is used as an analytic tool in this context. Unsupervised learning is an extension of kernel methods to new problems. Besides Kernel PCA one can use the regularization to obtain more general feature extractors. A second approach leads to regularized quantization functionals which allow a smooth transition between the Generative Topographic Map and Principal Curves. The second part deals with uniform convergence bounds for the algorithms and concepts presented so far. It starts with a brief self contained overview and an introduction to functional analytic tools which play a crucial role in this problem. By viewing the class of kernel expansions as an image of a linear operator one may give bounds on the generalization ability of kernel expansions even when standard concepts like the VC dimension fail or give too conservative estimates. In particular it is shown that it is possible to compute the covering numbers of the given hypothesis classes directly instead of taking the detour via the VC dimension. Applications of the new tools to SV machines, convex combinations of hypotheses (i.e. boosting and sparse coding), greedy approximation schemes, and principal curves conclude the presentation