Designing Deep Generative Models with Symbiotic Composition

Generative models, especially ones that are parametrized by deep neural networks, are powerful unsupervised learning tools towards understanding complex data without label. Deep generative models have achieved tremendous success in recent years, with applications in various tasks including sample generation, image editing, visual domain adaptation, data augmentation for discriminative models and solving inverse problems. Parallel endeavors have been made along various directions – such as generative adversarial networks (GAN), variational autoencoders (VAE), normalizing flows, energy-based methods, autoregressive models, and diffusion models – and we are now able to generate increasingly photorealistic images using deep neural networks. Although these models have distinct formulations and properties, it is critical to have a clear view of fundamental deep generative models, understanding their pros and cons as well as knowing the reasons behind that. With a good understanding of existing generative learning frameworks, we can design new models that can maintain the advantages while getting rid of the limitations of previous models. Figure 1 is an illustration of the main theme of this dissertation: we give a panoramic view of the landscape of deep generative models, and design new models based on the landscape. Following the theme, the dissertation can be divided into two parts. In the first part (Chapter 1 and 2), we give a high-level overview of deep generative models and dive deep into several important models, introducing their formulations and analyzing their pros and cons carefully. Motivated by the analysis, in the second part (Chapter 3, 4, 5 and 6), we introduce four advances in the direction of designing new generative models by combining existing ones. For each new model we propose, we carefully present the formulation and explain the motivation behind the composition. We conduct comprehensive experiments to show that our proposed models can be seen as symbiotic compositions of two different generative models: the two components in each composition help each other to get rid of the limitations while keeping the advantages. It is our hope that our findings may serve as a minor contribution to the development of deep generative models