Speech/hand coordination in the production of prosodic focus

International audienceManual gestures are naturally produced in spoken communication (see e.g. Mcneill, 1992; Kendon, 1997). The links between hand gestures and speech were mainly analyzed qualitatively and most studies showed that brachio-manual gestures and speech are tightly bound. In particular, it appears that gestures can be linked to prosody (e.g. Mcclave, 1998). Some studies have tried to shed light on the coordination of speech and manual gestures using motion capture. Levelt et al. (1985) studied the coordination of speech (noun phrase) and pointing gestures and found that speech seems to adapt its timeline to that of the deictic gesture. de Ruiter (1998) found that speech/pointing coordination (speech: noun phrase or single word) was influenced by contrastive but not lexical stress. Rochet-Capellan et al. (2008) found that the stressed syllable (within a twosyllable non-word) is always included in the part of the pointing gesture that shows (index finger extended and pointed at target). The aim of this study is to analyze the coordination between speech and manual gestures in the production of prosodic contrastive focus. Focus consists in putting forward (i.e. designating and thus, in a sense, pointing at: see Loevenbruck et al., 2009) a word or a group of words in an utterance (ex: THOMASf ate the candy). It is thus related to manual pointing. This study addresses several key questions: 1 How does the position of focus in the utterance influence gesture/speech coordination ? 2 Is this coordination dependent on the relationship between gesture and speech i.e. on the type of gesture? Methodology - We compared two prosodic focus conditions: subject focus (SF) vs. object focus (OF); and four gesture conditions: no gesture (speech only) vs. index-finger pointing (deictic communicative) vs. beat gesture (non-deictic communicative) vs. control gesture (button press; non-deictic non-communicative). We used a correction task to elicit the production of focus. The participants sat in a chair facing a translucid screen on which visual targets appeared. The motion of their mouth and right hand were tracked using a motion capture device (Optotrak). The vocal productions were recorded using a microphone. Four points of interest were annotated for gestures (beginning, apex, return, end). Articulatory lip targets (protrusion and aperture peaks) and acoustic cues (fundamental frequency and intensity peaks, boundaries of syllable production) were also annotated. Ten adults (8F, 2M) participated in the experiment. All were right-handed and native speakers of French. Results - Focus condition (SF vs. OF) has a significant effect on the timing of gesture production (for all four points of interest: F(1,9) È 52, p Ç 10¡4). The position of the gesture is thus influenced by the position of focus. Gesture condition has a significant effect on the timing of gesture apices and ends for all gesture types (F(1,9) È 9,p Ç 10¡2).The timing of the acoustic cues of focus are not influenced by the production of a gesture nor by gesture type except for intensity (F(1,9) Ç 3.1, n.s. for all acoustic cues except intensity). The positions of the gestures' points of interest were compared to those of the articulatory targets and acoustic cues. The pointing gesture is more strictly coordinated with focus than the other gestures. In particular, the pointing gesture apex is tightly aligned with articulatory lip targets corresponding to vocalic gestures. Control and beat gestures appear to be either more difficult to elicit or less precise. Speech/gesture coordination is clearly influenced by the type of gesture. Moreover the data show that speech onset is influenced by the production of a gesture but that the utterance's internal organization is not influenced. Gesture onset is always influenced by the focus condition. However, the gesture's internal organization is influenced by the focus condition only for pointing. Conclusion - This study allows for a more precise characterization of speech/gesture coordination spoken communication (entire sentences and dialog context). It also shows that speech/gesture coordination is dependent on the functional relationship between gesture and speech