The integration of facial and auditory affect: An emotional McGurk effect?

Research investigating human emotion processing has typically studied either auditory (vocal) or visual (facial) information in isolation. However, speech perception literature supports integration of auditory and visual phonetic information by way of the McGurk effect. This study hypothesized an emotional McGurk effect (i.e., integration of auditory and visual emotional information). To evaluate this hypothesis, emotionally incongruent auditory-visual stimuli (e.g., a joyful voice with a sad face) were developed on a sample of 120 participants. These stimuli were then presented to 30 additional participants who categorized them according to one of eight common emotions. Results indicated significant (p \u3c .001) Condition by Emotion interaction effects, suggesting that emotional information from auditory and visual sources is integrated during the perception of auditory emotion. This integration appears to occur in a predictable manner. To understand emotion perception, future studies will need to consider the interaction of auditory and visual emotional information