Time course of speech-gesture integration in comprehension: Insights from gating experiments

When we speak, we often spontaneously produce hand gestures as well. The current study investigated the time course of how the listener/viewer integrates information from speech and gesture in two experiments, using the "gating paradigm". The stimuli consisted of three types of video clips: those with a spoken verb (e.g., to rotate) and an iconic gesture (a circling hand depicting rotation) (multi-modal condition), those with just a spoken verb (audio-only condition), those with just a gesture (visual only condition). Participants were first presented with the initial part of a video clip, and then with successively longer clips. In Experiment 1, the participants were asked to recognise the verb in the multimodal and audio-only conditions. In Experiment 2, the participants were asked to indicate the meaning of the gesture in the multimodal and visual-only conditions. It was found that multimodal presentation facilitated word recognition and gesture interpretation (Experiments 1 and 2). Furthermore, multimodal presentation initially increased the number of candidate word types (Experiment 1), but it did not increase the number of candidate gesture interpretation types (Experiment 2). The results indicate that gesture and speech are incrementally integrated with each other, with qualitatively different consequences for each modality initially