Multimodal constructions for dialogue management: On the role of eye gaze and gesture in dyadic and triadic interactions

It is a well-known fact that managing the flow of a dialogue in terms of sequentially organized speaker turns is a complex undertaking that requires a high degree of synchronized behavior across interlocutors. Early work in conversation analysis (e.g. Schegloff 1972) already pointed at the intrinsic multimodal nature of dialogue management, but nevertheless, the focus has primarily been on (para)verbal signals guiding the speech exchange system (cf. Schmitt 2005 for a review). Only recently, the role of other semiotic channels (gesture, posture, gaze) has been studied more systematically. Most notably, there is a growing interest in the emergence of composite signals (Clark 1996, McNeill 2006) involving multiple channels jointly contributing to dialogue management. A key feature is the sequential organization of the different channels, as illustrated e.g. in Mondada (2007) for pointing-gestures as a resource in establishing speakership. The present study pursues this line of research and focuses on the role of eye gaze patterns in relation to other signal systems in dialogue management. Existing studies on the role of eye gaze in face-to-face interaction, from the early pioneering work by Kendon (1967) to the more recent studies by Vertegaal et al. (2001), Bailly et al. (2010) and Jokinen (2010), have revealed general distributional patterns across speakers and hearers. In this study, we focus more specifically on (i) microphenomena, or detailed gaze events of about 200- 400ms (e.g. brief moment of gaze aversion), associated with specific dialogue acts (e.g. holding the floor); (ii) the temporal positioning of these gaze events vis- vis other relevant markers in speech and gesture. An analysis of co-occurrence patterns across different modalities was performed on a multimodal video corpus (InSight Interaction Corpus, authors 2013). This corpus consists of two- and three-party interactions, with headmounted scene cameras and eye-trackers tracking all participants’visual behavior, providing a unique “speaker-internal” perspective on the conversation. The analysis reveals (i) specific patterns of gaze distribution related to the temporal organization of dialogue acts. Different dialogue acts typically display specific gaze events at crucial points in time, as e.g. in the case of brief gaze aversion associated with turnholding, and shared gaze between interlocutors at the critical point of turn-taking; (ii) a strong correlation and temporal synchronization between eye gaze, speech and gesture in the realization of specific dialogue acts, as shown by means of a series of cross-recurrence analyses for specific turn-holding mechanisms (e.g. verbal hesitation markers co-occurring with gesture and brief gaze aversion) . The results are indicative of the existence of multimodal discourse constructions, defined along the lines of McNeill (2006)- as recurrent patterns in the physical co-occurrence, functional trade-off and interactive set-up of multiple modes of representation in face-to-face settings.status: publishe