Multimodal Dynamics of Extended Communication

Language is a multimodal performance of remarkable coordination. Previous research has found this coordination responds to word-level variables as well as sentence-level variables. Coordination at the longer time scales of the discourse-level, however, is less studied. Bridging the rapidly changing multimodal behaviors of language to its diverse discursive contexts and pragmatic intentions is fundamental for our understanding of language use in social contexts. This dissertation takes initial steps in this direction by studying the dynamic organization and coordination of body movement and prosody over the extended time scales of diverse performances. Chapter 2 explores the dynamics and multimodal patterns that speakers produce in the context of an academic talk. We analyze the organization and coordination of body movement, prosody, and PowerPoint slide transitions. Results show weak regularities in the coordination and organization of the modalities as a result of the shared discursive context, but also highlight the role of individual constraints in shaping the multimodal behaviors of speakers. Chapter 3 contrasts the multimodal multiscale coordination of the movements and sounds of solo music performances and speech monologues. Results evidence different coordination patterns depending on performance goals, with higher local sound-movement synchrony and stronger multiscale coordination for speech compared to music. Coordination also varies across the discursive contexts analyzed, but not across instruments of interpretation. Chapter 4 studies the effects that the limitations of videoconferencing have on interpersonal and multimodal coordination. The continuous perturbations introduced by videoconferencing reduce interpersonal coordination during remote conversations in ways consistent with the reduction of signal quality as compared to in-person interactions. Multimodal coordination, which is not mediated by the Zoom’s audiovisual signals, is maintained, while speech convergence is reduced, and movement convergence is disrupted. Chapter 5 outlines a proposal to connect the multimodal coordination patterns of language use to the pragmatic goals of social interactions. I argue that the context sensitivity and rapid adaptability of multimodal coordination are consistent with the characteristics of synergies. I propose the multimodal patterns of language result from metastable multimodal synergies that simultaneously provide stability for pragmatic goals while also dynamically adapting to ever-changing constraints and goals of conversations