Understanding Gaussian noise injections in neural networks

Deep Learning (read neural networks) has emerged as one of the most exciting and powerful tools in the statistical machine learning toolbox. Neural networks over the past decade have defeated incumbent state of the art models in computer vision, natural language processing, and reinforcement learning; amongst a myriad of other fields. Though these achievements are impressive, neural networks remain surprisingly brittle, especially in the case of smaller network architectures. They can easily overfit, yielding poor performance on unseen data, and are remarkably susceptible to small perturbations and noise on their inputs. White noise on images has been shown to consistently alter the predictions of neural networks trained for classification. In a more targeted adversarial setting, where a hypothetical adversary has agency on the exact form of the input perturbation, these adverse effects are even more drastic. As these networks are increasingly deployed on our devices and in critical decision making systems, it is in the community’s broader interest to develop methods that not only allow smaller neural networks to generalise, but to develop methods that are robust to random and adversarially targeted perturbations on data. An obvious candidate method to do this is to train neural networks with random perturbations on their training data, with the hopes that they are then able to make predictions that are robust to these perturbations. The limitations of this approach are well documented, as it only confers a strong degree of robustness to the exact perturbation distribution used at training time. As such, it doesn’t confer a ‘broad spectrum’ robustness that one might hope. However, using these methods at test-time can confer this sought out certifiable robustness, one which we study at length in the context of Variational Autoencoders, a class of deep probabilistic model which experience Gaussian noise injections by design. These methods also confer other overlooked benefits. For analytic simplicity, due to their finite moments, we focus our analysis on Gaussian Noise Injections (GNIs); but this analysis paves the way for the study of other forms of noise added to neural networks. As we demonstrate, in the case of GNIs during training, neural networks are regularised to learn a function that is biased towards lower frequencies in the Fourier domain. This tends to yield networks that not only generalise better to unseen data, but also have better calibrated predictions in the classification setting, meaning that the predictions they output are inherently more trustworthy. We also nuance this portrait, by showing that neural networks trained with gradient descent related methods also experience bias when trained with GNI, meaning that they will learn functions that aren’t true minimizers of the objective function they are optimising for. To present a unifying view with which to study GNI, this work ends with a framework using tools of Harmonic analysis, that explains both the lower frequency bias and the adversarial robustness conferred by GNI on neural networks. This work presents a novel perspective from which to study deep probabilistic models (that of neural networks experiencing noise on their inputs), provides a comprehensive review of the positive and negative effects of training neural networks with GNI, and paves the way for the study of the effects of other types of noise on neural network training.