Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice

Emotion expression associated with human communica-tion is known to be a multimodal process. In this work, we investigate the way that emotional information is conveyed by facial and vocal modalities, and how these modalities can be effectively combined to achieve improved emotion recognition accuracy. In particular, the behaviors of differ-ent facial regions are studied in detail. We analyze an emo-tion database recorded from ten speakers (five female, five male), which contains speech and facial marker data. Each individual modality is modeled by Gaussian Mixture Mod-els (GMMs). Multiple modalities are combined using two different methods: a Bayesian classifier weighting scheme and support vector machines that use post classification ac-curacies as features. Individual modality recognition per-formances indicate that anger and sadness have compara-ble accuracies for facial and vocal modalities, while happi-ness seems to be more accurately transmitted by facial ex-pressions than voice. The neutral state has the lowest per-formance, possibly due to the vague definition of neutral-ity. Cheek regions achieve better emotion recognition ac-curacy compared to other facial regions. Moreover, classi-fier combination leads to significantly higher performance, which confirms that training detailed single modality clas-sifiers and combining them at a later stage is an effective approach. 1