Connecting Brain and Mind with Formal Concept Analysis: a Data-Driven Investigation of the Semantic, Explicit Coding Hypothesis

Understanding how semantic information is represented in the brain has been an important research focus of neuroscience in the past few years. Unlike 'traditional' neural (de)coding approaches, which study the relationship between stimulus and neural response, we are interested in higher-order relational coding: we ask how perceived relationships between stimuli (e.g. similarity) are connected to corresponding relationships in the neural activity. Our approach addresses the semantical problem, i.e. how terms (here stimuli) come to have their (possibly subjective) meaning, from the perspective of the network theory of semantics (Churchland 1984). This theory posits that meaning arises from the network of concepts within which a given term is embedded. We showed previously (Endres et al 2010, AMAI) that Formal Concept Analysis (FCA, (Ganter & Wille 1999)) can reveal interpretable semantic information (e.g. specialization hierarchies, or feature-based representation) from electrophysiological data. Unlike other analysis methods (e.g. hierarchical clustering), FCA does not impose inappropriate structure on the data. FCA is a mathematical formulation of the explicit coding hypothesis (Foldiak, 2009, Curr. Biol.) Here, we investigate whether similar findings can be obtained from fMRI BOLD responses recorded from human subjects. While the BOLD response provides only an indirect measure of neural activity on a much coarser spatio-temporal scale than electrophysiological recordings, it has the advantage that it can be recorded from humans, which can be questioned about their perceptions during the experiment, thereby obviating the need of interpreting animal behavioural responses. Furthermore, the BOLD signal can be recorded from the whole brain simultaneously. In our experiment, a single human subject was scanned while viewing 72 grayscale pictures of animate and inanimate objects in a target detection task (Siemens Trio 3T scanner, GE-EPI, TE=40ms, 38 axial slices, TR=3.08s, 48 sessions, amounting to a total of 10,176 volume images). These pictures comprise the formal objects for FCA. We computed formal attributes by learning a hierarchical Bayesian classifier, which maps BOLD responses onto binary features, and these features onto object labels. The connectivity matrix between the binary features and the object labels can then serve as the formal context. In line with previous reports, FCA revealed a clear dissociation between animate and inanimate objects in a high-level visual area (inferior temporal cortex, IT), with the inanimate category including plants. The inanimate category was subdivided into plants and non-plants when we increased the number of attributes extracted from the fMRI responses. FCA also highlighted organizational differences between the IT and the primary visual cortex, V1. We show that subjective familiarity and similarity ratings are strongly correlated with the attribute structure computed from the fMRI signal (Endres et al. 2012, ICFCA)