Multisensory integration of speech in social context

Speech perception in social contexts requires multisensory integration (MSI) regarding auditory and visual information produced by a speaker. Despite a large scientific literature concerning MSI of simple, well-controlled stimuli, more research is needed to further our understanding of MSI “in the wild”: of complex, dynamic, context-embedded stimuli. Here, we investigated gestural and visual-speech enhancement of auditory speech perception, using stimuli that are embedded in a social context, and are therefore more ecologically valid. Participants (N=36) watched a recording of a videoconference with three actors playing an (simulated) turn-taking game to name actions performed in given contexts (e.g., in a restaurant we. eat), supporting action words with associated gestures. Words were presented under audio-visual (AV), unisensory visual (V) or auditory (A) conditions (simulating disconnections of the camera or microphone). The task was to identify the action word communicated by the actor, from a list of four alternatives words displayed on the screen. Participants’ behavioural (accuracy and reaction times) and neuronal (EEG) data were collected. As expected, the data revealed improved accuracy and shortening of reaction times in the bimodal condition (AV) in comparison to both unimodal conditions. At the neuronal level, we observed stronger suppression in the alpha band in the AV condition as compared to the sum of A and V. Alpha suppression had been previously linked to integration processes of visual cues in speech (lip reading and gestures). Here, we demonstrate that this MSI alpha suppression cannot be accounted for by simple additive processing of the partial, audio and visual, information. Together, these results confirm that audiovisual speech accompanied by iconic gestures is readily integrated to the benefit of speech perception and extrapolates the findings to dynamic, complex, and naturalistic social situations