Towards Lightweight Neural Networks with Few Data via Knowledge Distillation

The advancement of deep learning technology has been concentrating on deploying end-to-end solutions using high dimensional data, such as images. With the rapid increase in benchmark performance comes the significant resource requirements to train the network and make inferences with it. Deep learning models that achieve state-of-the-art benchmark result may require a huge amount of computing resources and data. To alleviate this problem, knowledge distillation with teacher-student learning has drawn much attention in compressing neural networks on low-end edge devices, such as mobile phones and wearable watches. However, current teacher-student learning algorithms mainly assume that the complete dataset for the teacher network is also available for the training of the student network. However, for real-world scenarios, users may only access to part of training examples due to commercial profits or data privacy, and severe over-fitting issues would happen as a result. In this study, we tackle the challenge of learning student networks with few data by investigating the ground-truth data-generating distribution underlying these few data. Taking Wasserstein distance as the measurement, we assume that this ideal data distribution lies in a neighborhood of the discrete empirical distribution induced by the training examples. Thus we propose to safely optimize the worst-case cost within this neighborhood to boost the generalization. Furthermore, with theoretical analysis, we derive a novel and easy-to-implement loss for training the student network in an end-to-end fashion. Our analysis is empirically validated through experiments using benchmark datasets, and the result indicates the effectiveness of our proposed method