Real-time integration of gesture and speech during reference resolution

There is some disagreement among researchers about the role of gesture in comprehension; whether it is ignored, processed separately from speech, used only when speakers are having difficulty, or immediately integrated with the content of the co-occurring speech. The present experiment provides evidence in support of immediate integration. In our experiment participants watched videos of a woman describing simple shapes on a display in which the video was surrounded by four potential referents: the target, a speech competitor, a gesture competitor, and an unrelated foil. The task was to “click on the shape that the speaker was describing”. In half of the videos the speaker used a natural combination of speech and gesture. In the other half, the speaker’s hands remained in her lap. Reaction time and eye-movement data from this experimen