Improving Transfer Learning via Large-Scale Model Pre-Training

Transfer learning was successfully employed already at the very early rise of deep neural networks to obtain strong models when using only small amounts of data. Recently, evidence was obtained that increasing the model size while at the same time increasing the amount of data and compute for pre-training results in very large models that have even stronger generalization and transfer capabilities. In the talk, we will review evidence for quality of transfer on natural or medical images and its variation due to model and data size used during pre-training. We show evidence that in particular for low data regime or few-shot transfer large models pre-trained on large data (e.g. ImageNet-21k or larger) may provide strong benefits. We will then motivate necessity for systematic experiments that may deliver scaling laws for transfer performance dependent on model, data size, compute budget and composition of large source dataset used for pre-training. Such experiments require vast compute resources and proper utilization of supercomputing facilities. As an outlook, we will introduce COVIDNetX initiative that aims on studying large-scale transfer learning on a specific use case of performing medical imaging based COVID-19 diagnostics