Deep generative stochastic networks trainable by backprop

We introduce a novel training principle for prob-abilistic models that is an alternative to max-imum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previ-ous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised func-tion approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic inter-pretation of denoising autoencoders and obtain along the way an interesting justification for de-pendency networks and generalized pseudolike-lihood, along with a definition of an appropri-ate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an ar-chitecture that mimics the Deep Boltzmann Ma-chine Gibbs sampler but allows training to pro-ceed with simple backprop, without the need for layerwise pretraining