Efficient Vision Transformers : a report leveraging low-rank NAS and optimization for compact and high-performance models

In recent years, Vision Transformers (ViTs) have emerged as a promising approach for various computer vision tasks. However, their high computational cost poses a significant challenge for practical deployment. This report presents a comprehensive study of two distinct yet complementary methods for compressing and optimizing ViTs. First, we introduce a unified ViT compression framework that combines pruning, layer skipping, and knowledge distillation in a budget-constrained, end-to-end optimization process. The primal-dual algorithm is employed to solve the optimization problem, and experimental results on the ImageNet dataset demonstrate competitive performance with reduced computational overhead. Second, we present an automatic low-rank approximation framework based on neural architecture search, which efficiently searches for optimal low-rank configurations. The low-rank compression approach leverages weight inheritance and a hierarchical search space inspired by computer architecture’s cache memory to balance exploration and exploitation while reducing supernet training overhead. By integrating these two methods, we aim to provide a comprehensive solution for the effective compression and optimization of Vision Transformers, paving the way for their broader applicability in real-world scenarios.Electrical and Computer Engineerin