About the integration of gesture and speech: semantic and syntactic aspects

About the integration of gesture and speech: semantic and syntactic aspects. Thomas C. Gunter MPI for Human Cognitive and Brain Sciences, Germany In everyday face-to-face conversation, speakers not only use speech to transfer information but also rely on facial expression, body posture and gestures. In this talk I will take a closer look on how gestures potentially influence language processing on a semantic (part 1) as well as on a syntactic (part 2) level. The first part of the talk sketches some of the semantic influences of gestures and focus on how iconic gestures affect speech comprehension. Iconic gestures have a close formal relationship to the semantic content of speech. For instance, a speaker might perform a typing movement with her fingers while saying: “Yesterday I wrote the letter”. Clearly, a listener can extract additional information from these gestures (e.g. we know that the letter was written on a keyboard and not with a pen). Although there is no doubt that iconic gestures are communicative and can be integrated online with speech, little is known about the nature of this process and how our own communicative abilities and also our environment influence this integration process. In order to shed some light on these issues I will review several ERP-experiments which looked at the influence of task, timing and environment on gesture-speech integration. The second part of the talk will explore the possibility that gesture influences the syntactic aspect of language and will focus on beat-gestures. A beat gesture is a short, rhythmic movement or series of movements of the hand. They have been suggested to accent or emphasize portions of the co-expressive speech and may therefore influence which syntactic structure is assigned to a sequence of words. Recently we have carried out several ERP-experiments that suggest that indeed such gestures (and not other types of emphasis) can disambiguate ambiguous syntactic structures. Taken all the experimental evidence together, I will suggest that most effective communication not only involves the mouth, but also the hands