Improved model design and training techniques for efficient DNN inference

The rapid growth of artificial intelligence and deep learning in recent years has led to significant progress in various application domains, including computer vision (CV), natural language processing (NLP), and recommendation models (RM). As the complexity and size of Deep Neural Networks (DNNs) increase, the demand for more effective techniques to train and deploy these models also rises. This dissertation introduces three techniques aimed at enhancing the performance and efficiency of DNNs in three disjoint settings. First, we describe a technique for improving DNN model quality via a training time technique applied to RM-type models. Then, we describe two techniques for model-hardware co-design: (1) Quantization for resource-constrained deployments; (2) Training/tuning for deployments on analog Processing-in-Memory (PIM) accelerators. The dissertation first focuses on the development of better embedding training algorithms for recommendation models. Recommendation systems are widely used in various applications, such as e-commerce, online advertising, and content recommendation. These models often rely on large embedding layers, which can result in substantial memory and computational requirements. We first show that a strikingly simple factorization technique at training can noticeably improve the model size-accuracy trade-off. We show that its quality continues to improve when the inner dimension of factorization increases. The proposed factorization technique enables more efficient and accurate recommendation models without increasing their embedding size. Because factorization does not enlarge the search space of embedding training, its superiority and dependency on the inner dimension is puzzling. To understand the working mechanism behind the proposed factorization, we present a theory based on the dynamics of embedding updates in gradient descent training. Our theory suggest that the factorization leads to dense embedding updates with momentum-like effects, which are not present in conventional training. Then, the dissertation advances the state of the art in mixed-precision quantization algorithms for general DNNs. Quantization is a widely used technique that compresses DNNs by mapping their weights and activations to lower precision values. Conventional quantization methods assign the same precision to all layers in DNNs. Mixed-precision quantization (MPQ) algorithms allow different quantization bitwidths in layers to reduce the numerical precision of neural network weights and activations, leading to lower memory requirements and computational complexity while maintaining minimal loss in accuracy. By improving MPQ techniques, we can enhance the efficiency of DNNs and facilitate their deployment on resource-constrained devices and specialized hardware. We propose a technique based on exploring interactions effects between different layers during quantization. We first show, through empirical studies on several commonly used DNNs, that cross-layer quantization error interactions are non-negligible and that the assumption of layer-independent quantization error in prior work leads to sub-optimal MPQ solutions. Then, we present the first post-training and backpropogation-free MPQ algorithm that is aware of the cross-layer quantization effects. The algorithm is based on the second-order Taylor expansion of the quantization loss and formulates the MPQ problem as an integer quadratic programming (IQP) problem. The formulated IQP is fast to solve and delivers state-of-the-art MPQ performance. Finally, the dissertation addresses the challenges associated with analog PIM DNN accelerators, which have emerged as a promising solution to reduce data movement and improve energy efficiency in memory-intensive workloads. Analog PIM architectures face issues such as device variability and limited precision. To overcome these challenges, we first propose a novel training algorithm that makes use of straight-through estimator and variability sampling to reduce the degradation due to quantization and variability While prior work generally assumes variations are independent, we study a more realistic setting where variation present spatially correlated variability patterns that occur on a chip-to-chip basis. Under correlated variations, training alone is not able to mitigate the destructive degradation. We propose a innovative inference-time tuning architecture to first estimate the degree of variation and then correct the error in outputs based on the estimation. Our proposed training algorithm and inference-time tuning architecture significantly enhance the robustness and accuracy of analog PIM accelerators, while maintaining the benefits of in-memory processing.Electrical and Computer Engineerin