Are Topographic Deep Convolutional Neural Networks Better Models of the Ventral Visual Stream?

Neural computations along the ventral visual stream, -- which culminates in the inferior temporal (IT) cortex -- enable humans and monkeys to recognize objects quickly. Primate IT is organized topographically: nearby neurons have similar response properties. Yet the best models of the ventral visual stream - deep artificial neural networks (ANNs) – have “IT” layers that lack topography. We built Topographic Deep ANNs (TDANNs) by incorporating a proxy wiring cost alongside the standard ImageNet categorization cost in the two “IT-like” layers of AlexNet (Lee et al., 2018), by specifying that “neurons” that have similar response properties should be physically close to each other. This cost both induced topographic structure and altered tuning characteristics of model IT neurons. We presented 2560 naturalistic images to monkeys and to ANNs. We found that, relative to the base (nontopographic) model, the “neurons” in the “IT” layer of some of the TDANN models matched actual IT neurons slightly better, and the dimensionality of the TDANN “IT” neural population was much closer to that of the measured monkey IT neural population. We also found that, while TDANNs did not show a statistically significant better match to human object discrimination behavior, detailed analysis suggests a trend in that direction. Taken together, TDANNs may better capture properties of IT cortex and wiring costs might be the cause of topographic organization in primate IT.Wellcome Trust (Award 206521/Z/17/Z)National Institutes of Health (Grant DP1HD091947)Simons Foundation (Grants SCGB 325500, 542965)NSF (Award CCF-1231216