Visual perceptual assimilation of non-native speech

© 2017 Dr. Inn BrodskyStudies of auditory speech perception show that native language exerts a powerful influence on the perception of a second language. This influence is manifested in the assimilation of non-native sounds into native speech categories. However, speech is a multimodal experience, incorporating visual aspects of speech as well as sounds. The novel question addressed here is whether these aspects are assimilated naturally in the audio-visual (AV) perception of speech. The main hypothesis is that the visual aspects of non-native speech are assimilated into native categories, leading to the prediction that visual aspects of non-native speech that are dissimilar to the native language will be identified more readily than visual aspects of non-native speech that are similar to the native language. In several experiments, English-speaking participants were presented with native and non-native vowels and syllables that were experimentally validated and ranked according to their similarity to the native (English) language. Surprisingly, the results showed no evidence of visual perceptual assimilation for similar or dissimilar visual features of non-native speech. The study also investigated perception of visual aspects of non-native speech at the lexical level of cognate words and sentences compared to visual aspects of the native language. The results showed that participants were unable to discriminate native and non-native languages based on visual-only input. When native and non-native sentences were temporally reversed, participants were unable to discriminate veridical from temporally-reversed sentences in a visual-only mode condition. Furthermore, in cross-modal matching experiments, participants were unable to translate a visual image into the corresponding auditory signal, unless a written cue was provided or an auditory signal was present. Moreover, slowing the presentation of stimuli during AV speech perception did not facilitate processing of the visual aspects of non-native speech. Overall, the current study found a remarkably limited influence from the visual-only aspects of speech on non-native perception when compared to auditory-only input but did show for the first time that participants were more likely to falsely accept non-native stimuli as native in the multi-modal AV condition when compared to the auditory-only and visual-only conditions. The results therefore extend prior reports about the influence of visual aspects of speech on perception of sounds in a non-native language. These findings are discussed in terms of models of non-native speech perception and of applicability for teaching non-native languages to second language learners