Watching the brain build memories across eye movements: an EEG - eye-tracking coregistration study

Introduction Episodic memory allows us to revisit our past and recollect inter-related elements that characterize life events (Tulving, 1983). When forming such relational memories, we only apprehend a small part of our visual field in full acuity at a time. This limitation is overcome by shifting visual attention over a sequence of eye movements. We bind these visual “samples” of the world into a coherent episodic memory. However, little is known about how we build coherent episodic memories across eye movements. Previous studies have established crucial roles of theta and alpha oscillations in memory formation (Hanslmayr et al. 2012; Herweg et al., 2020), but lacked the necessary precision to elucidate how coherent representations are formed over time across eye movements. Here, we combine EEG and eye-tracking to investigate the buildup of episodic memory representations in natural viewing behavior. Methods Participants (N = 28) were asked to encode events with elements from three categories (faces, places, objects, Fig. 1). In each block (N = 6), nine events were presented for 10 s each. After a distractor task, memory was tested for all event-specific combinations of elements. EEG and eye movements were simultaneously recorded. Only data from the encoding phase was analyzed. A major problem in coregistration of EEG and eye movements in free viewing concerns overlapping effects of sequential saccades on EEG (Dimigen et al., 2011; Nikolaev et al., 2016). In order to overcome this, we used the deconvolution approach (Ehinger & Dimigen 2019). This regression-based method involves modelling not only experimental conditions of interest but also confounding eye movement characteristics and ordering factors, such as fixation rank, which can be “regressed out”. Before deconvolution, EEG was filtered in the theta and alpha frequency bands. The theta and alpha power were analyzed in epochs from -200 ms to +500 ms relative to the fixation onset. Results During the 10 s encoding, participants made, on average, 5.1 gaze transitions between categories. Subsequent memory performance increased with the cumulative number of gaze transitions during encoding (p < 0.001). Theta power in the interval -100+300 ms relative to the fixation onset over the frontal channels increased with time (p = 0.03) and performance (p = 0.02) (Fig. 2). Conversely, alpha power in the interval -100+300 ms over all electrodes prominently decreased with time (p = 0.002), irrespective of memory performance (Fig. 2). Next, we compared EEG for the first fixation on a category (i.e., after an inter-category saccade) with the last fixation on that category (i.e., after an intra-category saccade). The theta power in the interval -100+300 ms was significantly higher for the first than the last fixation (p < 0.001) and was more prominent for high than low memory performance (p < 0.001) (Fig. 1). The alpha power in the interval 0+500 ms was lower for the last than for the first fixation (p < 0.001), and lower for high than low memory performance at the last fixation (p = 0.02). Conclusions The frontal theta increase over the time of the event and for successful retrieval support the proposed pivotal role of theta synchronization in the formation of episodic memories (Clouter et al. 2017) and extends it to the binding of event elements across eye movements. The prominent theta effects in the interval of saccade execution and the larger theta power for fixations after inter- than intra-category saccades suggest that binding occurs via association of pre- and post-saccadic information at gaze transitions between categories. Alpha desynchronization over the time course of encoding may reflect an increasing amount of information (Hanslmayr et al. 2016) that is captured at each sequential fixation. The larger desynchronization associated with higher performance for fixations after intra- than inter-category saccades indicates active engagement of cortical areas in information acquisition during encoding