FALCON: FAst and Lightweight CONvolution for Compressing and Accelerating Convolutional Neural Networks

학위논문(석사)--서울대학교 대학원 :공과대학 컴퓨터공학부,2019. 8. Kang, U.How can we efficiently compress Convolution Neural Networks (CNN) while maintaining the accuracy of classification tasks? One of the promising approaches is based on depthwise separable convolution which replaces a standard convolution with a depthwise convolution and pointwise convolution. However, previous works based on the depthwise separable convolution are limited since 1) they are mostly heuristic approaches without precise understanding of their relations to the standard convolution, and 2) their accuracies cannot match that of the standard convolution. In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel. Such interpretation leads to developing a generalized version rank-k FALCON which further improves the accuracy while sacrificing a bit of compression and computation. Experiments show that FALCON outperforms 1) existing methods based on depthwise separable convolution, and 2) the standard CNN model by up to 8× compression and 8× computation reduction while ensuring similar accuracy. We also demonstrate that rank-k FALCON provides even better accuracy than the standard convolution in many cases, while using smaller numbers of parameters and floating point operations.I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 II. Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1 Convolution Neural Network . . . . . . . . . . . . . . . . . . . . 4 2.2 Depthwise Separable Convolution . . . . . . . . . . . . . . . . . . 6 2.3 Methods Based on Depthwise Separable Convolution . . . . . . . 9 2.4 Hadamard Product . . . . . . . . . . . . . . . . . . . . . . . . . . 10 III. Proposed Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1 Extended Hadamard Product (EHP) . . . . . . . . . . . . . . . . 12 3.2 Depthwise Separable Convolution and EHP . . . . . . . . . . . . 14 3.3 FAst and Lightweight CONvolution (Falcon) . . . . . . . . . . . 16 3.4 Rank-k Falcon . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.5 Quantitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . 20 3.5.1 Falcon . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.5.2 Rank-k Falcon . . . . . . . . . . . . . . . . . . . . . . . 22 IV. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.2 Fitting Convolution Unit into Model . . . . . . . . . . . . . . . . 25 4.3 Accuracy vs. Compression . . . . . . . . . . . . . . . . . . . . . . 32 4.4 Accuracy vs. Computation . . . . . . . . . . . . . . . . . . . . . . 32 4.5 Rank-k Falcon . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 V. Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 VI. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 A Generality of EHP . . . . . . . . . . . . . . . . . . . . . . . . . . 43 B Parameters and FLOPs . . . . . . . . . . . . . . . . . . . . . . . 44 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Maste