Leveraging Kernels for Unsupervised Learning

Kernel methods have been a central part of the machine learning arsenal for several decades. Within this framework, unsupervised learning has been a particularly challenging area. This is due to the inherent nature of unsupervised learning tasks, where important information about the structure of the data is unknown to the user, and as such it is difficult to design a kernel or system to solve the problem at hand. This thesis aims to bridge this knowledge gap on a multitude of challenges within the field. Firstly, we address an important challenge within kernel methods for unsupervised learning, namely that of kernel parameter sensitivity. The process of finding the best parameter for the problem at hand usually depends on information which is unavailable for unsupervised tasks.Inspired by ideas from ensemble learning, we design a kernel for vectorial data with missing elements that automatically adapts to the inherent structures of the data, in order to decouple the parameter choice from the problem at hand. We perform experiments on spectral clustering and unsupervised ranking tasks with promising results. Next, we develop a kernelized approximation method for unsupervised ranking using the Personalized PageRank (PPR). This method is based on novel insights on the PPR, which through a PPR-specific low-rank representation akin to Kernel PCA naturally leads to an out-of-sample approximation. The method is based on the spectrum of a specific matrix. We provide error bounds for the approximation which is used to order the eigenvectors/eigenvalues to minimize the approximation error. We perform a range of experiments to support our novel insights and show that our method may even outperform the PPR. The final part of this thesis synergetically combines kernel methods and neural networks in various unsupervised tasks. Firstly, we design a kernelized autoencoder that incorporates similarities between datapoints through a kernel function in order to learn meaningful representations in code space. Secondly, we propose a novel deep learning approach to clustering, utilizing kernel-based information theoretic losses, with promising experimental results when compared to state--of--the art methods on challenging problems. Finally, we incorporate an unsupervised dimensionality reduction method (e.g. Kernel PCA) in-between the reservoir and readout layer of an Echo State Network in order to capture the dynamics of the time series while reducing the dimensionality of the trainable parameters and improving accuracy