Mean Field Theory for Sigmoid Belief Networks

We develop a mean field theory for sigmoid belief networks based on ideas from statistical mechanics. Our mean field theory provides a tractable approximation to the true probability distribution in these networks; it also yields a lower bound on the likelihood of evidence. We demonstrate the utility of this framework on a benchmark problem in statistical pattern recognition---the classification of handwritten digits. 1. Introduction Bayesian belief networks (Pearl, 1988; Lauritzen & Spiegelhalter, 1988) provide a rich graphical representation of probabilistic models. The nodes in these networks represent random variables, while the links represent causal influences. These associations endow directed acyclic graphs (DAGs) with a precise probabilistic semantics. The ease of interpretation afforded by this semantics explains the growing appeal of belief networks, now widely used as models of planning, reasoning, and uncertainty. Inference and learning in belief networks are possible ins..