Kernel Embeddings of Conditional Distributions

Many modern applications of signal processing and machine learning, ranging from com-puter vision to computational biology, require the analysis of large volumes of high-dimensional continuous-valued measurements. Complex statistical features are commonplace, including multi-modality, skewness, and rich dependency structures. Such problems call for a flexible and robust modeling framework that can take into account these diverse statistical features. Most existing approaches, including graphical models, rely heavily on parametric assumptions. Variables in the model are typically assumed to be discrete-valued, or to be multivariate Gaus-sians; and linear relations between variables are often used. These assumptions can result in a model far different from the data generating process. In this paper, we will review the recent development of nonparametric inference with ker-nel embedding of conditional distributions, and provide a novel, unified kernel framework for addressing a broad class of challenging problems in graphical models. The key idea is to map conditional distributions into infinite-dimensional feature spaces using kernels, such that sub-sequent comparisons and manipulations of these distributions are achieved via feature space operations: inner products, distances, projections, linear transformations, and spectral analysis. Basic probabilistic rules, such as the Sum Rule, Product Rule, and Bayes ’ Rule, then become linear operations on matrices of kernel values. We demonstrate the effectiveness of this ker-nel framework for inference, in a nonparametric belief propagation algorithm and in a hidden Markov model problem. 1