Just-In-Time Kernel Regression for Expectation Propagation

We propose an efficient nonparametric strategy for learning a message operator in expectation propagation (EP), which takes as input the set of incoming messages to a factor node, and pro-duces an outgoing message as output. This learned operator replaces the multivariate inte-gral required in classical EP, which may not have an analytic expression. We use kernel-based re-gression, which is trained on a set of probabil-ity distributions representing the incoming mes-sages, and the associated outgoing messages. The kernel approach has two main advantages: first, it is fast, as it is implemented using a novel two-layer random feature representation of the input message distributions; second, it has prin-cipled uncertainty estimates, and can be cheaply updated online, meaning it can request and in-corporate new training data when it encounters inputs on which it is uncertain. In experiments, our approach is able to solve learning problems where a single message operator is required for multiple, substantially different data sets (logis-tic regression for a variety of classification prob-lems), where it is essential to accurately assess uncertainty and to efficiently and robustly update the message operator. 1