Input and Model Compression for Adaptive and Robust Neural Networks

This dissertation explores compression techniques for neural networks to enable control of resource usage, accelerate training, and increase robustness against adversarial as well as out-of-distribution examples. The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. Our new band-limiting method artificially constrains the frequency spectra of these filters and data during training and inference. The frequency-domain constraints apply to both the feed-forward and back-propagation steps. Experimental results confirm that band-limited models can effectively control resource usage (GPU and memory). The band-limited method with 50% compression in the frequency domain results in only a 1.5% drop in accuracy for the ReNet-18 model trained on CIFAR-10 data while reducing the GPU memory usage by 40% and the computation time by 30% in comparison to their full-spectra counterparts. The models trained with band-limited layers retain high prediction accuracy and require no modification to existing training algorithms or neural network architectures, unlike other compression schemes. Since band-limited models naturally reject high-frequency noise, they are useful for studying the adversarial robustness of neural networks. I show that band-limited models are actually one instance of a broader family of adversarial defenses called perturbation-based defenses. These defenses, whether random or deterministic, are essentially equivalent in their efficacy, and all share similar weaknesses to adaptive attacks. I also illustrate applications to out-of-distribution robustness of models in terms of their generalization and detection of anomalous inputs in the NLP (Natural Language Processing) domain. The band-limited models have practical applications in a number of time-series and signal processing domains. I present a new solution for a fair wireless co-existence between Wi-Fi Access Points (APs) and LTE-U (Long Term Evolution-Unlicensed) Base Stations (BS) using band-limited neural networks and demonstrate that such networks can accurately learn distinct patterns for the energy distributions in Wi-Fi AP transmissions. Our method results in higher accuracy (close to 99% in all cases) as compared to the existing auto-correlation (AC) and energy detection (ED) approaches