Integration of auditory and visual information in fast recognition of realistic objects

cote interne IRCAM: Suied08dNone / NoneNational audienceRecent studies have used complex and meaningful stimuli to investigate the neural mechanisms involved in the processing of sensory information necessary for object perception. However, relatively few studies have focused on behavioural measures of multisensory objects processing. Realistic objects are of particular interest in the study of multisensory integration, since a given object can generally be identified through any of several single sensory modalities. The fact that the same semantic knowledge can be accessed through different modalities allows us to explore the different processing levels that underlie retrieval of supramodal object concepts. Here we studied the influence of semantic congruence on auditory-visual object recognition in a go/no-go task. Participants were asked to react as fast as possible to a target object presented in the visual and/or the auditory modality, and to inhibit their response to a distractor object. The experiment was run under an immersive and realistic virtual environment including 3D images and free-field audio. Reaction times were significantly shorter for semantically congruent bimodal stimuli than predicted by independent processing of information about the objects presented unimodally. Interestingly, this effect was twice as large as found in previous studies that used information-rich stimuli. The processing of bimodal objects was also influenced by their semantic congruence: reaction times were significantly shorter for semantically congruent bimodal stimuli (i.e., visual and auditory stimuli from the same object target) than for semantically incongruent bimodal stimuli (i.e. target represented in only one sensory modality and distractor presented in the other modality). Importantly, an interference effect was observed (i.e. longer reaction times to semantically incongruent stimuli than to the corresponding unimodal stimulus) only when the distractor was auditory. When the distractor was visual, the semantic incongruence did not impair recognition. Our results show that immersive displays may provide large multimodal integration effects, and reveal a possible asymmetry in the attentional filtering of irrelevant auditory and visual information