Exploring the contribution of prosody and gesture to the perception of focus using an animated agent

Speech prosody has traditionally been analyzed in terms of acoustic features. Although visual features have been shown to enhance linguistic processing, the conventional view is that facial and body gesture information in oral (non-signed) languages tends to be redundant and has the role of helping the hearer recover the meaning of an utterance. Though prosodic information in face-to-face communication is produced with concurrent visual information, little is known about their audiovisual multisensory interactions. We conducted two perception experiments modeled after the McGurk paradigm with a 3D animated character, in which varying degrees of discordance between auditory and visual information were created to investigate two related questions regarding the detection of contrastive-corrective focus: (a) how important are gestural cues with respect to auditory cues and (b) what is the relevance of the different gestural movements involved (i.e., head nodding, eyebrow raising)? Participants were presented with combinations of auditory and visual cues for both information and Contrastive Focus Statements (Experiment 1, with the corresponding unimodal control experiments) or combinations of two visual cues (namely combinations of competing eyebrow and head movements) without auditory information (Experiment 2), and were asked to identify whether the utterance presented was a statement or a correction. Results of Experiment 1 showed that (a) the presence of either acoustic or gestural features of contrastive focus were key in guiding the listener towards one interpretation or another, and (b) listeners were more sensitive to one of the modalities when the other was weaker. Results of Experiment 2 showed that (a) both types of visual cues (head and eyebrow movements) contributed individually to the perception of contrastive focus, and (b) head nods were more informative than eyebrow movements for focus identification. Overall, our findings suggest that prosodic and visual information work in a complementary fashion and are not integrated in the same way as auditory and visual information during segmental perception.A preliminary version of this article was presented at Interspeech 2011 (Florence, August 27–31). We are grateful to the audience at that conference, and especially to David House, for very helpful comments. We are particularly indebted to Aurora Bel for her help in recruiting participants for the experimental sessions and to Javier Agenjo for his assistance with the NINOs platform. This research was supported by the following grants: BFU2012-31995 and TIN2008-05995 Learn3 CICYT (awarded by the Spanish Ministry of Science and Innovation), 2014 SGR 925 (Generalitat de Catalunya), and iMP (European Union 7th Framework project). The second author also acknowledges a MAEC-AECID Grant (Spanish Ministry of Foreign Affairs and Cooperation)