Geometric and spectral limitations in generative adversarial networks

Generative Adversarial Networks (GANs) have become one of the most successful and popular generative models in the recent years, with a wide variety of applications across the robust intelligence domains, such as image manipulation, text and audio synthesis, style transfer, and semi-supervised learning, to name a few. The main advantage of GANs over their classical counterparts stems from the use of Deep Neural Networks (DNNs), which can utilize the ongoing revolution in the availability of data and computation power to effectively discover complex patterns. Yet, with this exceptional power, comes an exceptional limitation: the black-box behavior associated with DNNs. This lack of understanding and clarity not only places the profound promise of GANs under a shadow of mistrust, but also greatly hinders any effort to increase their efficiency. As such, studying GANs' limitations and biases is perhaps as important, if not more important, as advancing their design and performance. The main focus of this dissertation is to study two fundamental limitations in GANs, namely a geometric and a spectral limitation. We investigate these limitations in depth, both empirically and theoretically, unveil their causes and consequences across different applications, and finally provide solutions to these limitations. We start by providing an introduction to density estimation and generative modeling in Chapter 1. In this chapter, we review different approaches to density estimation, highlight the advantages and disadvantages of each method, and discuss the issues that motivated the development of modern DNN-based generative models. The main goal of this chapter is to draw both a historic and a pragmatic line from classical density estimation methods to the modern GANs. Chapters 2 and 3 elaborate and extend on the results presented in Khayatkhoei et al. (2018). In Chapter 2, we expose and study the limitation of GANs in learning distributions with disconnected support. We first discuss why having a disconnected support is not a singular and pathological phenomenon, rather a common property of many real world data distributions. Then we theoretically and empirically illustrate the difficulties of GANs in learning such distributions, and its ramifications for the practitioner. In Chapter 3, we propose and evaluate an approach for dealing with the geometric limitation discussed in the previous chapter. This model is based on using an ensemble of generative DNNs, each of which will learn to focus on one connected component of the distribution's support. Moreover, a prior learning approach is proposed to address the problem of how to choose the ``best'' ensemble for a given distribution. The final GAN model, denoted DM-WGAN, trains end-to-end, can learn distributions supported on connected and disconnected manifolds, and infers the required number of members in the ensemble automatically without any explicit supervision. We conclude this chapter by reviewing several existing variants of GANs and their relation with the introduced geometric limitation and our proposed solution. Chapters 4 and 5 elaborate and extend on the results presented in Khayatkhoei and Elgammal (2020). In Chapter 4, we uncover another fundamental limitation in GANs: a spatial frequency bias. Specifically, we empirically and theoretically show that GANs' performance is not indifferent to the frequency of the underlying signal that carries a distribution. This chapter provides an insight into which datasets and domains are more prone to sub-optimal learning when GANs are used, and perhaps more importantly, what part of a signal is more likely to be missed by GANs. The findings are particularly crucial to the applications that use GANs to manipulate high resolution data, such as in medical and satellite imaging, or where GANs are used to augment or extrapolate data, such as in semi-supervised learning and simulation. In Chapter 5, we propose an efficient approach for matching the spatial bias of GANs to the known biases of a distribution. This approach, denoted Frequency Shifted Generators, utilizes the observation that the spatial frequency bias is not a fixed bias and can be efficiently translated to construct a generative DNN that is specifically targeted at a desired spatial frequency. We also show that it is possible to construct an ensemble of such shifted generators, each focusing on a specific frequency, to address the spatial frequency bias in a more general sense. Finally, in Chapter 6, we connect our separate discussions of the two fundamental limitations in the previous chapters, and discuss the broader impact of our findings on the bigger picture of distribution learning and generative modeling. We particularly comment on the open questions and directions of future research into the limitations of GANs, and more generally, of DNNs.Ph.D.Includes bibliographical reference