Invariance and Invertibility in Deep Neural Networks

Machine learning is concerned with computer systems that learn from data instead of being explicitly programmed to solve a particular task. One of the main approaches behind recent advances in machine learning involves neural networks with a large number of layers, often referred to as deep learning. In this dissertation, we study how to equip deep neural networks with two useful properties: invariance and invertibility. The first part of our work is focused on constructing neural networks that are invariant to certain transformations in the input, that is, some outputs of the network stay the same even if the input is altered. Furthermore, we want the network to learn the appropriate invariance from training data, instead of being explicitly constructed to achieve invariance to a pre-defined transformation type. The second part of our work is centered on two recently proposed types of deep networks: neural ordinary differential equations and invertible residual networks. These networks are invertible, that is, we can reconstruct the input from the output. However, there are some classes of functions that these networks cannot approximate. We show how to modify these two architectures to provably equip them with the capacity to approximate any smooth invertible function