To look and look again: Memory accumulation across sequential eye movements and related brain activity

When we visually explore the world, our eyes are constantly moving in order to sample the information available in front of us. The timing of these movements, as well as where our gaze is directed at, can tell us a great deal about how this information is acquired. Even more can be learned from combining the study of eye movement with that of brain activity. In this dissertation, I co-registered eye movements and electrical brain activity (EEG) in free viewing behavior. Co-registration is instrumental for studying how visual memory accumulates information during free viewing. To address this issue, I developed a task, consisting of multiple target visual search followed by change detection. In a first study, I assessed eye movement parameters as possible indicators of memory load. Pupil size is known as a reliable measure of memory load during fixed-gaze viewing conditions. Hence, pupil size was a primary candidate for free viewing conditions as well. Pupil size reflected memory load only when it exceeded working memory capacity. Exhibiting a slow physiological response, pupil size also was unable to reveal instantaneous effects of memory load based on the task-relevance of the fixated item. I, therefore, also tested the effect of working memory load on fixation duration. Fixation duration increased with memory load for fixations on task-relevant (target) but not on task-irrelevant (distractor) items. Since effects of memory accumulation were observed in eye movement, an effect on brain activity could be expected. With co-registered EEG and eye movements, fixations and saccades become markers to segment EEG activity. However, a solution is needed to the problem that, in free viewing, EEG related to cognitive processes is confounded by activity related to sequential eye movements. I explored two approaches to solving this problem. The first involves selecting EEG epochs across experimental conditions while matching on eye movement parameters such as saccade size and fixation duration. When eye movement parameters are matched, effects on EEG related to eye movements are balanced across conditions, allowing us to study the effect of the cognitive process of interest. I developed a procedure for matching conditions across eye movement parameters using Mahalanobis distance. The second approach involves generalized additive mixed models (GAMM), which estimates nonlinearities in EEG in terms of smoothing splines. To obtain the estimates, eye movement parameters were used as predictors along with experimental conditions. The partial effects of the experimental condition on EEG could then be separately investigated. Using the first approach, I matched eye movement parameters across memory load conditions. Subsequent target fixations were used as criterion to distinguish increases in memory load. I compared EEG in the intervals pre-saccadic and post-saccadic to subsequent target fixations. However, no reliable EEG correlate of visual working memory load could be found. I concluded that effects of single items in working memory could have been overshadowed by more complex memory processes and memorization strategies. To elucidate these memory-encoding strategies, I analyzed refixation behavior in free viewing. To do this, two approaches were taken. In the first, traditional approach, refixations on targets and distractors were counted. Target refixations increased with memory load, whereas, distractor refixations decreased with memory load. There were more target fixations during correct responses than incorrect responses, while the opposite was seen for distractors. In the second approach, eye movement patterns were investigated using recurrence quantification analysis. Repetition of eye movement sequences increased with memory load. This result implies that along with item information, eye movements are also encoded into working memory, in agreement with scanpath theory. Overall, the study of refixation behavior revealed that oculomotor strategies are flexibly adapted to the requirements of the task. Finally, I investigated the neural mechanisms of refixation behavior. The dependence of saccade planning on task-relevance was investigated in the presaccadic potentials of first visits and refixations of items. Refixations differed from first visits to targets in their effect on planning of the subsequent saccade. No such difference was seen for distractors. These presaccadic potentials showed that task-relevance affected memory processes and consequently saccade planning. In sum, my studies have contributed to the understanding of accumulation of visual information during free viewing. This process was seen to involve complex memory strategies that are continually adapted during the task, not only affecting eye movements but also employing them for memorization.status: publishe