The role of synchrony and ambiguity in speech-gesture integration during comprehension

■ During face-to-face communication, one does not only hear speech but also see a speakerʼs communicative hand move-ments. It has been shown that such hand gestures play an important role in communication where the two modalities in-fluence each otherʼs interpretation. A gesture typically tempo-rally overlaps with coexpressive speech, but the gesture is often initiated before (but not after) the coexpressive speech. The present ERP study investigated what degree of asynchrony in the speech and gesture onsets are optimal for semantic integra-tion of the concurrent gesture and speech. Videos of a person gesturing were combined with speech segments that were either semantically congruent or incongruent with the gesture. Although gesture and speech always overlapped in time, gesture and speech were presented with three different degrees of asynchrony. In the SOA 0 condition, the gesture onset and the speech onset were simultaneous. In the SOA 160 and 360 con-ditions, speech was delayed by 160 and 360 msec, respectively. ERPs time locked to speech onset showed a significant difference between semantically congruent versus incongruent gesture– speech combinations on the N400 for the SOA 0 and 160 con-ditions. No significant difference was found for the SOA 360 condition. These results imply that speech and gesture are inte-grated most efficiently when the differences in onsets do not exceed a certain time span because of the fact that iconic ges-tures need speech to be disambiguated in a way relevant to the speech context.