NLIP-Essex-ITESM at ImageCLEFcaption 2021 task: Deep learning-based information retrieval and multi-label classification towards improving medical image understanding

This work presents the NLIP-Essex-ITESM team's participation in the concept detection sub-task of the ImageCLEFcaption 2021 task. We developed a method to predict health outcomes from medical images by processing concepts from radiology reports and their associated medical images. Our aim is to improved medical image understanding and provide sophisticated tools to automate the thorough analysis of multi-modal medical images. In this paper, two deep learning- and k-NN-based methods of a) Information Retrieval and b) Multi-label Classification were developed and assessed. In addition, a Densenet-121 and an EfficientNet were used to train and extract imaging features. Our team achieved the second-highest score when the Information Retrieval method was used (F1-score bench-marking was 0.469). Further investigations are underway in the setting of improving health outcome predictions from multi-modal medical images. Code and pre-trained models are available at https://github.com/fjpa121197/ImageCLEF2021