Auditory influence on visual object categorization: an fMRI study

In our daily life, we are often confronted with objects that give rise to signals in multiple sensory modalities. The brain's challenge is to integrate information from multiple senses into a unified percept. Combining psychophysics and fMRI, the present study investigates how task-irrelevant object source sounds affect category-selective activations for visual faces in the fusiform and landmarks in the parahippocampal gyri. In a visual selective attention paradigm, subjects categorized degraded object pictures as landmarks or faces while ignoring intact object source sounds that were either semantically congruent or incongruent. The 2X2 factorial design manipulated (i) Visual category: Animal vs. Landmark, and (ii) Auditory category: Animal vocalization vs. Sound associated with Landmark. Behaviorally, incongruent trials were associated with longer response times. This effect emerged due to an interference of incongruent sounds associated with landmarks on the categorization of visual faces. At the neural level, only landmark-selective activation in the parahippocampal gyrus, but not face selective activations in the fusiform gyrus were modulated by the congruency of an irrelevant object sound. More specifically, activations in the anterior parahippocampal gyrus showed additive effects of both visual and auditory category information. Effective connectivity analysis indicated that parahippocampal responses are amplified by incongruent auditory inputs via enhanced coupling between auditory and occipito-temporal cortices. Collectively, these results suggest that a region in the parahippocampal gyrus integrates information about object categories from multiple senses. In contrast, face-selective responses in fusiform gyrus are more robust and less influenced by task-irrelevant sounds, even in the context of a behavioural audiovisual interference effect