Flood Detection in Social Media Using Multimodal Fusion on Multilingual Dataset

Social media has emerged as a information source for natural disaster detection and assessment, such as flooding. Although visual features from images and textual information have been used for this purpose before, research shows that fusion of these two modes can further improve the detection and assessment. Moreover, due to advancement of social media, users now can write in these social media using their native language. Therefore, it is important to design a methodology that can handle multiliguality, specially for global natural disasters assessment such as flooding. In addition, social media contents, such as image and text, contain dissimilar credibility based on the nature of the media. Here, we propose a fusion methodology to fuse visual features and textual features using a novel machine learning approach, and test it in multiple language datasets, to detect flooding in three different types of social media. The results show that the proposed methodology fuses the two modes efficiently and improves the classification performance in all experimented languages.