Deep learning augmentation for medical image analysis

Deep learning is at the center of the current rise of computer aided diagnosis in medical imaging. This technology has the ability to mimic extremely complex mathematical functions for predictive tasks. These functions are encoded as computational models that are learned directly from data. Deep learning models are known to achieve human-level accuracy for predictive tasks. However, such a performance requires that the model is trained on a huge amount of training data. For computer aided diagnosis tasks, the relevant training data needs to be carefully annotated by medical experts. This process is laborious and expensive, which generally results in limited amount of training data for deep learning. Moreover, the data often suffers from the practical constraint of class-imbalance. These issues severely degrade the performance of deep learning in medical image analysis. The common strategy to sidestep the limited training data problem in medical imaging is to ‘transfer’ a model from the natural image domain to the medical image domain with the available limited data, and use that model to make predictions. Though useful, the transferred models still lack acceptable performance levels for medical tasks. This thesis develops a range of novel techniques to enhance deep learning based computer aided diagnosis performance, especially within the context of transfer learning paradigm. The research for this thesis was mainly conducted during the COVID-19 pandemic. Hence, COVID-19 detection and classification has received the main attention as evaluation tasks for the proposed techniques, among other thoracic diseases. Opening the research with an extensive literature review of deep learning in medical image analysis (Chapter 2), the thesis identifies lack of large-scale annotated data as the central challenge of effectively employing deep learning for the medical imaging tasks. Hence, it first develops a method to augment deep learning for a better transfer of natural image models to the medical image domain (Chapter 3). This technique also enhances the transferred models performance by reinforcing the model predictions with a sparse representation method. Our analysis revealed that the large domain gap between the natural images and the medical image data is a major source of transfer learning performance degradation. We hypothesize that first transferring a natural image model to the medical image domain with a large-scale data of possibly an irrelevant task, and subsequently transferring that model for the target task, can help. We verify this hypothesis with a novel hierarchical transfer learning method that used large-scale chest X-ray images to finally detect COVID-19 with computed tomography (CT) images (Chapter 4). In parallel, we made an important discovery that due to deep learning hype and urgency of developments in COVID-19 research, the performance of transfer learning for CT-based COVID-19 detection is severely over-estimated in the existing literature. We provide an extensive transparent study to reset the baseline of transfer learning performance for CT-based COVID-19 detection (Chapter 5). The thesis also introduces a novel concept of pre-text representation transfer that enables harnessing large amount of unlabelled data to improve transfer learning performance on balanced and imbalanced limited training data (Chapter 6). With this method, we are able to use un-annotated CT scan images from public domains and transfer the representation of natural image models to the CT data with these plentiful images. We use the resulting transferred representation for subsequent transfer learning with limited annotated COVID-19 CT images, achieving considerable performance gain over the conventional transfer learning