Robustness, structure and hierarchy in deep generative models

Deep learning provides us with ever-more-sophisticated neural networks that can be tuned via gradient ascent to maximise some objective. Bayesian statistics provides us with a principled and unified approach to specify statistical models and to perform inference. One productive way to pair these two methodologies results in Deep Generative Models (DGMs), where the mappings between the statistical parameters in a probabilistic model are themselves parameterised using neural networks. In this thesis we study both the ways in which this approach can be brought to bear on various problems in machine learning and the properties of the resulting models. There are three recurrent themes in this thesis, robustness, structure and hierarchy, that reemerge throughout. We begin by studying how we can structure a deep generative model to learn in a novel learning regime we propose called semi-unsupervised learning. It is an extreme case of semi-supervised learning, where for some classes of data there are no labelled examples given whatsoever. In learning to partition the data into distinct components, different ground truth classes, the model must be able to cluster over the unlabelled classes and perform semi-supervised learning over the classes for which some labelled examples are given. We show how this can be done for a range of standard datasets. From handling one discrete latent, cluster assignment, we then study models with a hierarchy of discrete latent variables. We propose a novel way to parameterise latent variables in models of this type, relaxed responsibility vector quantisation that enables training of very deep hierarchies of layer of latent variables. This method achieves state-of-the-art results at maximising a lower bound on the evidence of the data (train and test set) for hierarchical discrete DGMs trained end-to-end, on a range of standard datasets. In doing so, these models help to close the gap between hierarchical DGMs with discrete latents and those with continuous latents, as well as providing extremely stable training. We then swap to a different problem, how to structure a model to effectively learn statistically independent latent representations from high dimensional data. We propose a hierarchical approach, where we use a bijective function, a flow, to produce an intermediate representation that a highly-constrained linear Independent Component Analysis (ICA) model then acts on. This leads to superior performance on various toy and real datasets compared to other approaches. We then study the hitherto-unconsidered problem of how to render DGMs robust to adversarial attack. We demonstrate that regularising the latent space of these models can reliably induce robustness, and obtain even more robust models by applying this regularisation to hierarchical DGMs. Finally, we then study the problem of DGM robustness from a theoretical standpoint. We define $r$-robustness, a novel criterion for DGM robustness, and then derive a margin on that criterion within which a model can be said to be robust. Combined with new theory about the optimal model for a variety of DGM where the latent space is regularised, the form of this margin sheds insight into how this regularisation increases robustness. The work presented in this thesis shows how productive the combination of deep learning and Bayesian statistics can be, as well as providing insights into the nature of the models made by their combination. This opens up new research in both of these directions—for new models that build on the work presented and also new avenues for theoretical work studying deep generative models.