‘Seeing tunes.’ The role of visual gestures in tune interpretation

One of the unresolved questions in audiovisual prosody is the relative contribution of acoustic and visual cues to the expression of prosodic meaning. Though the majority of studies on audiovisual prosody have found a complementary mode of processing whereby sight provides relatively weak and redundant information in comparison with strong auditory cues, other work has found that sight provides information more efficiently than hearing. In Catalan, a pitch range contrast in a rising-falling nuclear configuration conveys a difference between a contrastive focus statement and an echo question. The main goal of this study is to investigate the relative contribution of visual cues in conveying this distinction. Twenty native speakers of Central Catalan participated in two identification tasks in which they had to decide between a focus statement and a question interpretation. Experiment 1 used a pitch range auditory continuum combined with two congruent and incongruent videotapes showing the facial gestures that are characteristic of the two pragmatic meanings. Experiment 2 used the same auditory continuum in combination with another continuum for facial gestures produced using a digital imagemorphing technique. The responses and reaction times obtained in both experiments revealed a consistent reliance on visual cues in the listener’s decisions, but also a consistent effect of the auditory stimulus. We argue that although facial gestures are the most influential elements that Catalan listeners rely on to decide between contrastive focus and echo question interpretations, bimodal integration with the acoustic cues is necessary for perceptual processing to be accurate and fast. Finally, we discuss the implications of these results for models of audiovisual processing.We are grateful to Carme de la Mota, Itziar Laka, Lluís Payrató, Josep Quer, Núria Sebastián-Gallés, Maria-Josep Solé, Marc Swerts and Eric Vatikiotis-Bateson for their comments on an earlier version of this paper. We would like to thank the members of the Group of Prosodic Studies (UPF-UAB, Barcelona) for their help in managing E-Prime scripts and statistics. We would also like to thank the subjects who took part in the two experiments and the recordings of the audiovisual materials, especially Eva Estebas-Vilaplana, for their comments on the gestural materials. We also thank the editors of this special issue, Ian Maddieson and Caroline Smith, as well as Beatriz Raposo and two anonymous referees for their insightful comments, which helped clarify and strengthen this paper. This research has been funded by projects FFI2009-07648/ FILO and CONSOLIDER-INGENIO 2010 Programme CSD2007-00012 ( both awarded by the Ministerio de Ciencia e Innovación) and by project 2009 SGR 701 (awarded by the Generalitat de Catalunya)