Neural Variational Inference and Learning in Belief Networks

Highly expressive directed latent variable mod-els, such as sigmoid belief networks, are diffi-cult to train on large datasets because exact in-ference in them is intractable and none of the approximate inference methods that have been applied to them scale well. We propose a fast non-iterative approximate inference method that uses a feedforward network to implement effi-cient exact sampling from the variational poste-rior. The model and this inference network are trained jointly by maximizing a variational lower bound on the log-likelihood. Although the naive estimator of the inference network gradient is too high-variance to be useful, we make it practi-cal by applying several straightforward model-independent variance reduction techniques. Ap-plying our approach to training sigmoid belief networks and deep autoregressive networks, we show that it outperforms the wake-sleep algo-rithm on MNIST and achieves state-of-the-art re-sults on the Reuters RCV1 document dataset. 1