Semantic representations for written words in the human brain

Semantic processing from different input modalities is a fundamental issue in cognitive neuroscience research. There is no agreement yet about the contribution of the different brain regions belonging to the associative-semantic network, such as ventral occipitotemporal, posterior or anterior temporal or inferior frontal cortex, let alone the differential contribution of sub-regions within these association zones. In this doctoral research project, I investigated the role of the perirhinal cortex, a sub-region of the anterior temporal lobe (ATL), in representing the semantic meaning of concrete entities in different input modalities, as well as that of the pars triangularis of the left inferior frontal gyrus (LIFG). I addressed this topic by applying multivoxel pattern analysis (MVPA) and representational similarity analysis (RSA), techniques which have heralded a new era in the study of the representation of concrete entities in the human brain. RSA provides an opportunity to directly test neural representational content and its dependence on input-modality and can reach a level of cognitive specificity and item-by-item granularity that cannot be attained by previous approaches. The primary objectives of the first fMRI study of this project were to evaluate whether the semantic similarity effect for written words in left perirhinal cortex (Bruffaerts et al., 2013b) extended to the spoken modality. During fMRI, healthy volunteers performed a property verification task for written words and spoken words. They were asked to decide whether a property was applicable to an entity. Whilst we found semantic similarity coding for the written modality, no semantic similarity effects were detected for the spoken words in the left perirhinal cortex. These contrastive results raised two questions: 1. At some point in the brain the pathways for written and spoken words should converge as they need to access the same semantic representations. If the perirhinal cortex is not a convergence zone, then where do the modality-specific semantic pathways converge? 2. To what extent does the semantic similarity effect in perirhinal cortex depend on the type of information that has to be retrieved? In order to address the former question, a second fMRI experiment was conducted that aimed at localizing the regions of interest. The fMRI data of the previous study was used as independent dataset where the RSA was performed in all localized regions. 38 healthy volunteers were asked to perform a property verification task for written words and spoken words. Not only did we replicate the semantic similarity effect in perirhinal cortex for the written modality, but we also detected a cross-modal semantic similarity effect in the pars triangularis of the LIFG. In line with a semantic working memory account of the LIFG (Gabrieli et al., 1998), we hypothesized that the convergence of spoken and written word input in pars triangularis serves as a gateway to the distributed processing of meaning in the brain. In a third fMRI experiment we determined to which extent the semantic similarity effect in perirhinal cortex depends on the type of information retrieved. Healthy volunteers performed a property verification task for written words and pictures, where each entity was combined with either a visual or a nonvisual property (e.g. Entity: dog. Visual property question: Does it have 4 paws? Nonvisual property question: Can it be bred?). In addition, pictures were characterized by means of Alexnet Deep Convolutional Neural Network (DCNN) (Krizhevsky et al., 2012). We not only showed that the semantic similarity effect in perirhinal cortex for written words was replicable for a new set of nouns and properties, but also that a semantic similarity effect was resent for written words when probed by visual properties. Moreover, the fMRI activity pattern evoked by written words in perirhinal cortex correlated significantly with layers 4 to 8 of the DCNN for the corresponding pictures. At a whole brain level, while the explicit retrieval of visual properties relied on visual processing areas, the explicit retrieval of non-visual properties relied on a distributed left-hemispheric network involving the inferior parietal lobule, superior temporal sulcus along its posterior-anterior axis and the anterior-inferior frontal gyrus, corresponding to a more verbal-linguistic network. Taken together, the results of these fMRI experiments provide strong evidence for the role of left perirhinal cortex in processing the meaning of written words.status: publishe