A Deep and Tractable Density Estimator

The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimen-sions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate infer-ence. In this work we introduce an efficient pro-cedure to simultaneously train a NADE model for each possible ordering of the variables, by shar-ing parameters across all these models. We can thus use the most convenient model for each infer-ence task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our train-ing procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance. 1