Onthology-based Masking Loss for Improved Generalization in Remote Sensing Semantic Image Retrieval

Semantic image retrieval can significantly reduce the time required to process the vast amounts of remote sensing image data and support applications such as detection of illegal fishing or logging or analysis of growth and change in residential and industrial areas. In this work we propose a novel method for remote sensing semantic retrieval based on a masking loss for convolutional neural networks which allows multi-dataset training despite different and incompatible semantic classes in different datasets. We achieve improved generalization when the trained model is applied in a realistic cross-dataset setting. In addition to this we perform a thorough evaluation of several design choices which are popular in other retrieval tasks, most notably the impact of specialized ranking losses, and formulate guidelines for future research. Our trained models are evaluated on the recent PatternNet dataset and the established WHU-RS19 and UCM dataset. We outperform the state-of-the-art on PatternNet, UCM, and WHU-RS19 by 29.3%, 17.2%, and 4.5%, respectively