Approximate Inference in Variational Autoencoders

A deep latent variable model is a powerful tool for modelling complex distributions. However, in order to train this model, we must perform approximate inference of the latent variable. A variational autoencoder (VAE) is a framework for learning both the generative and inference models for a latent variable model. This thesis provides novel analyses, applications, and interpretations of approximate inference in VAEs. This thesis reviews many of the recent developments made to improve VAEs. One such improvement is the importance-weighted autoencoder. The standard interpretation of importance-weighted autoencoders is that they maximize a tighter lower bound on the marginal likelihood than the standard evidence lower bound. The first contribution of this thesis is to provide an alternative interpretation: that it optimizes the standard variational lower bound, but using a stochastic importance-weighted variational distribution. In order to improve approximate inference in VAEs, it is important to understand what causes inference to be suboptimal. Two important factors that determine the quality of approximate inference in VAEs are: a) whether the variational distribution is expressive enough to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. This thesis contributes to our understanding of approximate inference in VAEs by using these factors to diagnose suboptimal inference across a number of settings. Finally, we investigate the task of using a latent variable to learn a joint distribution over two data modalities: images and text. The contribution of this work is to develop a model for this task and analyze how to perform approximate inference in this model. Specifically, we identify a problem arising from the mismatch between the posteriors of each modality and we demonstrate how the problem can be largely addressed by modelling the aggregate of the image posteriors.Ph.D