Processing and coding of natural time-varying images in the retina

We investigate the hypothesis that the early visual system efficiently codes natural time varying images, first by tracking part of the image, then by matching the spatiotemporal properties of the neural pathway to those of the tracked image. We formulate a representation for the time varying image which consists of two spatiotemporal components: a velocity field component and a stationary component. The velocity field component characterizes geometric motion in a scene, and the stationary component characterizes time variations not associated with motion such as flicker. We derive the space-time variant autocorrelation function and spectrum of the time varying image based on the two component model. Included in the derivation is the effect of tracking eye movements, where tracking eye movements are modeled as a linear, time variant filter placed between the image and the retina. We test and refine the statistical model using sequences of natural images to show that the spatiotemporal spectrum and other attributes of natural images markedly differ before and after tracking. The temporal frequency bandwidth and velocity distribution of the velocity field component are diminished in the region of tracking and broaden with increasing eccentricity from this region. The spectrum of the stationary component is unaffected by tracking. Using the theoretical and computational results, we compare the properties of the tracked image to the spatiotemporal characteristics of the retinal M and P pathways as described in the literature. The M and P pathways are matched to different attributes of the image. The results indicate that the combination of tracking and specialized neural pathways constitute an efficient coding scheme in the early visual system. We also investigate the link between types of tracking eye movements (smooth pursuit or full field tracking) and packing of retinal receptors. Certain combinations of tracking eye movements and retinal receptor packing maximizes the extraction of information from the environment. These combinations agree with observations of comparative physiology across species