Mean Field Networks

The mean field algorithm is a widely used approximate inference algorithm for graphical models whose exact inference is intractable. In each iteration of mean field, the approximate marginals for each variable are updated by get-ting information from the neighbors. This pro-cess can be equivalently converted into a feed-forward network, with each layer representing one iteration of mean field and with tied weights on all layers. This conversion enables a few nat-ural extensions, e.g. untying the weights in the network. In this paper, we study these mean field networks (MFNs), and use them as infer-ence tools as well as discriminative models. Pre-liminary experiment results show that MFNs can learn to do inference very efficiently and perform significantly better than mean field as discrimina-tive models. 1. Mean Field Networks In this paper, we consider pairwise MRFs defined for ran-dom vector x on graph G = (V, E) with vertex set V and edge set E of the following form, p(x) = 1 Z exp(E(x; θ)), (1) where the energy function E(x; θ) is a sum of unary (fs) and pairwise (fst) potentials E(x; θ) = s∈V fs(xs; θ) + (s,t)∈E fst(xs, xt; θ) (2) θ is a set of parameters in E and Z = x exp(E(x; θ)) is a normalizing constant. We assume for all s ∈ V, xs takes values from a discrete set X, with |X | = K. Note tha