An event-related potential investigation of the relationship between semantic and perceptual levels of representation

The present study was conducted to investigate relationships between semantic and perceptual levels of representation. A picture - word repetition paradigm was used in which we manipulated the semantic relationship between pictures and words. Experiment I involved two types of trials, one with words that had the same meaning as pictures and one with words that were unrelated to pictures. In Experiment 2 we replaced words that were identical in meaning with words that were semantically associated to pictures. In both experiments, visually presented probe stimuli were used to determine the presence of perceptual effects within the visual system, originating from the semantic interaction between words and pictures. In both experiments, conditions with unrelated picture - word pairs generated a search process following the N400 which included activity overlying the visual cortex. Probe stimuli were found to attenuate the amplitude of the search related negativity. The latency of the interaction, which was significant at the time of the NI response to the probe, suggested that the attempt to find a relationship between the picture and the word involved processing within extrastriate visual areas. UVF probes provided stronger attenuation, possibly because the UVF has direct transmission to the ventral processing stream which is believed to be involved in visual semantic processing. Semantic interactions between matching picture - word pairs in Experiment I were found to have an effect on the ERPs to probes presented at the same location as pictures. Probes presented under these conditions showed a stronger P2 over frontal areas followed by a more negative P3 over occipital areas. Although we had expected beforehand to find earlier effects in the latency of the probes' PI and NI responses, this result is consistent with the idea that retinotopic levels of object representation are linked with the semantic level of object description. Unlike Experiment 1, same location probes presented in associated picture - word conditions of Experiment 2 did not result in any specific ERP effects on the P2 and P3 components. This suggest that semantic interactions between pictures and words do not automatically propagate to the perceptual level, unless there is direct reference from the word to the visual representation of the object. (C) 2003 Elsevier Science (USA). All rights reserved.