In your face? Exploring multimodal resonance patterns involving facial responses to verbal and gestural stance-taking expressions

This contribution reports on a corpus-based interdisciplinary study, in which scholars from psychology, linguistics, gesture studies and FACS-analysis jointly investigate multimodal trade-off patterns between verbal, gestural and facial stance-taking expressions in dyadic face-to-face interactions. The general purpose of this study is to explore whether certain stance-taking phenomena may be characterized by an increased, multimodal connectedness of interaction partners while they are engaged in a spontaneous face-to-face conversation (AliLeGra-corpus, 2012). Following the premise that any facial motor response qualifies as a resonance marker relating to what was said by the interaction partner, we studied the effects of verbal and/or gestural stance-taking expressions on a wide array of facial muscle movements. For this we used Facial Action Coding (FACS, Ekman et al., 2002), which allows to code and quantify movements of individual muscles or small groups of muscles independently of any supposed meaning of these. Data are taken from 24 dyads of male participants, who had initially been unknown to each other and were engaged in seven minutes of spontaneous face-to-face conversation, which took place under controlled circumstances. Synchronized close-up and distant shot video recordings allowed a multimodal analysis. On the level of verbal and gestural expressions, stance-taking utterances expressing meanings like subjective obviousness, surprise, disbelief, impatience, but also certain types of humor (comical hypotheticals, stereotypes) were identified. Two certified FACS encoders identified all apexes (maximal expressions of individual facial muscle movements) of 9 upper face, 18 lower face, and 14 miscellaneous action units occurring in the conversations according to standard FACS conventions. Resonance in terms of facial motor responses was quantified as the number of apexes immediately following instances of a certain stance-taking type produced by the interaction partner. The presentation will exemplify and differentiate verbal-gestural stance-taking phenomena that increase or decrease interaction partners' resonance in the motor modality. Similar to direct facial mimicry and emotion-specific facial muscle responses to emotion words (cf. Foroni & Semin, 2009), resonance in terms of unspecific facial motor responses may increase the connectedness and involvement of interaction partners and facilitate understanding and sympathy between them. References: Authors. 2012. AliLeGra Corpus. Alignment Leuven Graz. Dept. of Linguistics, KU Leuven Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system: The manual. Salt Lake City, Utah: A Human Face. Foroni, F., & Semin, G.R. (2009). Language that puts you in touch with your bodily feelings. The multimodal responsiveness of affective expressions. Psychological Science, 20, 974-980.status: publishe