How Does Feedback Impact Training in Audio-Visual Speech Perception?

Integration of visual and auditory speech cues is a process used by listeners in compromised listening situations, as well as in normal environments, as exemplified by the McGurk effect (McGurk and McDonald, 1976). Audio-visual integration of speech appears to be a skill independent of the ability to process auditory or visual speech cues alone. Grant and Seitz (1998) argued for independence of this process based on their findings that integration abilities could not be predicted from auditory-only or visual-only performance. Gariety (2009) and James (2009) further supported this argument by training listeners in the auditory-only modality with degraded speech syllables, then testing those listeners in the auditory-only, visual-only, and audio-visual conditions. Their results showed an increase in auditory-only performance, but no improvement in integration. Recently, DiStefano (2010) conducted a training study in which listeners were trained in the audio-visual modality with degraded speech syllables. Results showed that the performance increased only for the audio-visual conditions, and did not increase in the auditory-only or visual-only conditions. Interestingly, performance improved only for stimulus pairs that were “congruent” (i.e. the auditory and visual inputs were the same syllable) and did not increase for “discrepant” stimuli (i.e. the auditory and visual inputs were different syllables). It is possible that the feedback provided in DiStefano’s study impacted their pattern of results. However, the question remains as to whether integration of discrepant stimuli can be trained. In the present study, five listeners received ten hours of training sessions in the audio-visual condition with degraded speech signals similar to those used by Shannon et al. (1995). The feedback given during training was designed to encourage McGurk-type combination and fusion responses, in contrast to DiStefano’s study, in which feedback was given to encourage responses that matched the auditory signal. A comparison of pre-training and post-training scores showed little to no improvement in auditory-only performance, and a slight decrease in visual-only performance for congruent stimuli. Further, a substantial increase in McGurk-type responses was seen from pre- to post-test for discrepant stimuli. These results provide further support that integration is an independent process, and that the feedback provided strongly influences response patterns. This strong lack of generalization from training also should be incorporated into designing effective integration training programs for aural rehabilitation.A Division of Social and Behavioral Sciences Undergraduate Research Scholarship.No embarg