VELDA: Relating an Image Tweet’s Text and Images

Image tweets are becoming a prevalent form of social media, but little is known about their content – textual and visual – and the relationship between the two medi-ums. Our analysis of image tweets shows that while vi-sual elements certainly play a large role in image-text relationships, other factors such as emotional elements, also factor into the relationship. We develop Visual-Emotional LDA (VELDA), a novel topic model to cap-ture the image-text correlation from multiple perspec-tives (namely, visual and emotional). Experiments on real-world image tweets in both En-glish and Chinese and other user generated content, show that VELDA significantly outperforms existing methods on cross-modality image retrieval. Even in other domains where emotion does not factor in image choice directly, our VELDA model demonstrates good generalization ability, achieving higher fidelity model-ing of such multimedia documents.