On the pragmatics of multi-modal communication: Gesture, speech and gaze in the coordination of mental states and social interaction

Coordination is at the heart of human conversation. In order to interact with one another through talk, we must coordinate at many levels, first and foremost at the level of our mental states, intentions and conversational contributions. In this talk, I will present findings on the pragmatics of multi-modal communication from both production and comprehension studies. In terms of production, I will talk about (1) how co-speech gestures are used in the coordination of meaning allowing interactants to arrive at a shared understanding of the things they talk about, as well as on (2) how gesture and gaze are employed in the coordination of speaking turns in conversation, with special reference to the psycholinguistic and cognitive challenges that turn-taking poses. In terms of comprehension, I will focus on the interplay of ostensive (social gaze) and semantic (gesture) signals in the context of intention perception and language processing. My talk will bring different sets of findings together to argue for richer research paradigms that capture more of the complexities and sociality of face-to-face conversational interaction. Advancing the field of multi-modal communication research in this direction will allow us to more fully understand the psycholinguistic processes that underlie human language use and language comprehension