On Training Deep Generative Models with Latent Variables

Humans understand the world through concepts. They form high-level abstractions to represent sensory information in a simple way. Conceptual thinking is one of the central aspects of human intelligence as it allows knowledge reuse, simplifies the understanding of cause-effect relationships, and empowers creativity. We argue that further progress in our quest for artificial intelligence critically depends on the development of machine learning algorithms that can infer concepts from data and fantasize new data based on those concepts. Deep generative models with latent variables (DGLs) provide a unified framework for both (i) representation learning and (ii) data synthesis. Despite remarkable recent progress in this area, many practical challenges prevent DGLs from attaining their full potential. The goal of this thesis is to highlight those challenges and propose novel algorithmic solutions to address them. The first part of this thesis studies DGLs in the context of sequential data such as text sequences. DGLs for sequences are typically trained via maximum likelihood estimation (MLE), but this renders models with uninformative latent variables. To regularize degenerate MLE training, we propose an importance-weighted dropout scheme, implemented using an adversarial approach. In contrast to standard dropout, our method obtains a better trade-off between representation learning and sequence modeling. In the second part, we discuss DGLs for images in the form of variational autoencoders (VAEs). VAEs are generally regarded as inferior to other generative models concerning both density estimation and image generation quality. By augmenting VAEs with the newly introduced spatial dependency layers, we considerably close this performance gap that hinders the widespread adoption of VAEs. In the third part, we integrate our image VAE into a framework for automatic analysis of sleep patterns from brain signals. We apply VAE to detect anomalies in sleep recordings and combine it with a supervised classifier of sleep stages. The resulting framework is implemented as a web server, serving sleep labs around the world