A multimodal study of feedback in interpreter-mediated talk

Since the pioneering work by Kendon (1967), researchers across disciplines have shown a growing interest in the role of gaze behaviour in interactional discourse. The present study contributes to this line of multimodal interaction research by inquiring into the role of eye-gaze patterns in correlation with speech, gesture and posture in interpreter-mediated interaction. More specifically, we are interested in the collaborative process of grounding as described by Clark (1996). This grounding process, in which interlocutors try to establish mutual understanding, is most evident in the acceptance phase, when the addressee provides some kind of (verbal or nonverbal) feedback to the speaker’s utterance (cf. Clark 1996). In this study we analyzed participants’ gaze behavior in relation to other verbal and nonverbal signals in providing and eliciting feedback in interpreter-mediated dialogue. Our approach builds on existing research on grounding in monolingual face-to-face conversation through gaze and other multimodal resources (Bavelas 2002, McNeill 2006). For the pilot study, we recorded an interpreter-mediated psychotherapeutic session by using mobile eye-tracking technology (Brône & Oben 2013). Eye-tracking allows for the detailed study of interlocutors’ gaze patterns and at the same time captures other semiotic channels, such as gesture and posture. The recording was transcribed and analysed based on the workflow developed in Brône et al. (2013) for face-to-face dialogues. The results of the analysis of gaze patterns are indicative of a specific gaze distribution related to grounding signals in the interpreted dialogue. We also find indications of a strong correlation between gaze, speech and gesture in the establishment of common ground. The preliminary results thus show that common ground is established at different (verbal and nonverbal) levels of communication between all participants in interpreter-mediated interaction. REFERENCES Bavelas, J., L. Coates and T. Johnson (2002) “Listener Responses as a Collaborative Process: The role of gaze”, Journal of Communication: 566-580. Brône, G. and B. Oben (2015) “InSight Interaction. A multimodal and multifocal dialogue corpus”, Language Resources and Evaluation 49: 195-214. Clark, H.H. (1996) Using Language. Cambridge: Cambridge University Press. McNeill, D. (2006) Gesture, Gaze and Ground. In S. Renals and S. Bengio (eds) Proceedings of Machine Learning for Multimodal Interaction: Second international workshop 2005. Berlin/Heidelberg: Springer Verlag, 1-14. Oben, B. and G. Brône (2015) “What You See is What You Do. On the relationship between gaze and gesture in multimodal Alignment”, Language & Cognition (accepted)