엣지 컴퓨팅에서 모델 압축을 위한 Attentive Tutor를 이용한 다중 레벨 지식 증류

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 산업공학과, 2018. 8. 조성준.Deep neural networks have showed success in many areas of study. However, implementing such models may be impractical due to its large size and slow inference speed. Thus studies in reducing the size of neural networks while maintaining its performance have recently been active. One such area of study is knowledge distillation in which a large and well performing teacher network helps a smaller and faster student network learn and perform close to that of the teacher network. In this paper we propose a new knowledge distillation method, Attentive tutor, that utilizes multi-level feature outputs through skip connections and attention. It learns how effective the information in the layers of the teacher network are in helping the student network learn and distills knowledge from intermediate layers with different feature sizes as well as the last classification layer. Attentive tutor is evaluated on MNIST, SVHN, and CIFAR-10 and shows significantly improved results and imple- mentation possibilities.Chapter 1 Introduction 1 Chapter 2 Related Work 6 2.1 Model Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.1 Parameter Pruning and Sharing . . . . . . . . . . . . . . . . 6 2.1.2 Low-rank Factorization . . . . . . . . . . . . . . . . . . . . . 7 2.1.3 Specific Architectures . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Knowledge Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Knowledge Distillation with Logits . . . . . . . . . . . . . . . 8 2.2.2 Knowledge Distillation with Soft Labels . . . . . . . . . . . . 9 2.2.3 Knowledge Distillation with Noise Added to Logits . . . . . . 10 2.2.4 Knowledge Distillation with Hint Layers . . . . . . . . . . . . 11 2.3 Multi-level Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3.1 Fully Convolutional Networks . . . . . . . . . . . . . . . . . . 13 2.3.2 Hypercolumns . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3.3 DAG-CNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.4 Attention Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4.1 Additive Attention . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4.2 Dot-product Attention . . . . . . . . . . . . . . . . . . . . . . 16 Chapter 3 Proposed Methodology 17 3.1 Tutor: Multi-level Features . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 Attentive Tutor: Attention . . . . . . . . . . . . . . . . . . . . . . . . 19 3.3 Attentive Tutor: Knowledge Distillation . . . . . . . . . . . . . . . . 21 Chapter 4 Experiments and Results 23 4.1 Preliminary Experiments on MNIST and SVHN . . . . . . . . . . . 23 4.1.1 Dataset and Training Details . . . . . . . . . . . . . . . . . . 23 4.1.2 Performance evaluation . . . . . . . . . . . . . . . . . . . . . 24 4.1.3 Effect of Temperature . . . . . . . . . . . . . . . . . . . . . . 26 4.2 Detailed experiments on CIFAR-10 . . . . . . . . . . . . . . . . . . . 28 4.2.1 Dataset and Training Details . . . . . . . . . . . . . . . . . . 28 4.2.2 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . 28 4.2.3 Effect of Attention . . . . . . . . . . . . . . . . . . . . . . . . 29 4.2.4 Evaluation on Different Student Networks . . . . . . . . . . . 31 4.2.5 Further Discussion . . . . . . . . . . . . . . . . . . . . . . . . 32 Chapter 5 Conclusion 34 Bibliography 36 국문초록 43Maste