Cross-Modal Learning for Audio-Visual Emotion Recognition in Acted Speech

International audienceHuman affects and automatic emotions detection has been an active research topic for several years, with outcomes that have been beneficial to a number of different applications, including human-machine interaction, health-care, e-education, etc. Indeed, humans perceive and express emotions in a multi-modal manner. In order to capture this multimodal emotional content, robust features need to be extracted and combined efficiently. In this paper, we propose a cross-modal deep learning framework that leverages audio and visual information (speech and facial emotions) for emotion recognition in acted speech. The proposed method learns the spatio-temporal information of facial expressions and the audio features in an end-to-end fashion. The experiments were conducted on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). Proposed method achieves a classification accuracy of 87.5% outperforming state-of-the-art works on RAVDES