Multimodal feedback mechanisms in interpretermediated interaction

In interpreter-mediated dialogues, the interpreter’s overall task is to create mutual understanding of the subject matter and establish communicative contact and rapport between the primary participants who have no access to each other’s language (Linell, Wadensjö & Jönsson 1992, Pasquandrea 2011). However, the presence of the interpreter may result in difficulties for the primary speakers to address each other directly and to display signs of acceptance or mutual understanding (Linell, Wadensjö & Jönsson 1992, Davidson 2002). It is therefore assumed that multimodal resources such as nods, hand gestures and specific gaze patterns are of particular importance in this kind of complex interaction (cf. Pasquandrea 2011), especially for the collaborative process of grounding as described by Clark (1996). This grounding process, in which interlocutors try to establish mutual understanding, becomes most evident in the acceptance phase, when the addressee provides some kind of (verbal or non-verbal) feedback to the speaker’s utterance (cf. Clark 1996). Previous studies on grounding mechanisms in interpreter-mediated interaction have mainly paid attention to verbal feedback (Linell, Wadensjö & Jönsson 1992; Davidson 2002). Davidson (2002) provides a collaborative model of interpreted discourse based on the stipulation that there is not one, but two separate sets of common ground constructed between the interpreter and each of the two interpretees (Davidson 2002). However, what is missing in this model is the process of grounding between the two primary participants, which, we argue, may be explored on the non-verbal level. In the present study, we focus on the role of eye gaze in relation to other (non-)verbal signals in providing and eliciting feedback signals in the specific institutionalized setting of interpreter-mediated dialogue. Our study builds on existing research on grounding in monolingual face-to-face conversation through gaze and other multimodal resources (Bavelas 2002, McNeill 2006, Allwood & Cerrato 2003). In order to determine how much communicative contact is displayed by both interpreter and primary participants on verbal and non-verbal level, the present study provides a detailed account of temporal dynamics and multimodal packaging (McNeill 2006) of feedback signals. We also take into account how specific feedback patterns are linked to the communicative context of the interaction and to the specific interactional roles (cf. Goffman 1981, Wadensjö 1998, Roy 1999) in the interpreter-mediated interaction. We recorded a set of interpreted encounters by making use of unobtrusive mobile eye-tracking glasses (Gullberg & Kita 2009, Jokinen 2010, Brône & Oben 2013). Eye-tracking allows for the detailed study of interlocutors’ gaze patterns and at the same time captures other semiotic channels, such as gesture and posture. The analysis of our data reveals specific patterns of gaze distribution related to grounding signals in the interpreted dialogue. We also find indications of a strong temporal correlation between eye gaze, speech and gesture in the establishment of common ground. Our study not only reveals multimodal feedback patterns between the interpreter and the two primary speakers, but also between the primary participants themselves. The results are thus indicative of the establishment of common ground at different levels of communication in interpreter-mediated interaction. REFERENCES Allwood, J. & Cerrato, L. 2003. A study of gestural feedback expressions. In: Paggio, P., et al. (eds.) Proceedings of the First Nordic Symposium on Multimodal Communication: 7-22. Bavelas, J., Coates, L. & Johnson, T. 2002. Listener responses as a collaborative process: The role of gaze. Journal of Communication, 566-580. Brône, G. & Oben, B. 2013. InSight Interaction. A multimodal and multifocal dialogue corpus. Language Resources and Evaluation. Clark, H. H. 1996. Using Language. Cambridge: Cambridge University Press. Davidson, G. 2002. A model for the construction of conversational common ground in interpreted discourse. Journal of Pragmatics 34: 1273-1300. Goffman, E. 1981. Forms of Talk. Philadelphia: University of Pennsylvania Press. Gullberg, M. & Kita, S. 2009. Attention to speech-accompanying gestures: Eye movements and information uptake. Journal of Nonverbal Behaviour 33: 251-277. Jokinen, K. 2010. Non-verbal signals for turn-taking & feedback. Proc. of 7th Int. Conf. on Language Resources & Evaluation (LREC) International Universal Communication Symposium. Linell, P., Wadensjö, C. & Jönsson, L. (1992): Establishing Communicative Contact through a Dialogue Interpreter. In: Grindsted, A., Wagner, J. (eds.) Communication for Specific Purposes - Fachsprachliche Kommunikation. Tübingen: Gunter Narr Verlag, 125-142. McNeill, D. 2006. Gesture, gaze and ground. In: Renals, S., Bengio, S. (eds.) Proceedings of Machine learning for multimodal interaction: Second international workshop 2005, Berlin/Heidelberg: Springer Verlag, 1-14. Oben, B. & Brône, G. forthc. What you see is what you do. On the relationship between gaze and gesture in multimodal alignment. Language & Cognition. Pasquandrea, S. 2011. Managing multiple actions through multimodality: Doctors’ involvement in interpreter-mediated interactions. Language in Society 40(4): 455-481. Roy, Cynthia, 1999. Interpreting as a discourse process. Oxford Studies in Sociolinguistics. Oxford University Press, New York. Wadensjö, C. 1998. Interpreting as interaction. London and New York: Longman