Statistical decomposition of neural networks: Information transfer between correlation functions

Uncovering principles of information processing in neural systems continues to be an active field of research. For the visual system it is well known that it processes signals in a hierarchical manner [1,2]. Commonly used models in machine learning that perform hierarchical computations are feed-forward networks. We here study deep feed-forward networks with the aim of deducing general functional aspects of such systems. These networks implement a mapping between probability distributions, where the probability distribution is iteratively transformed from layer to layer. We develop a formalism for expressing signal transformations in each layer as transfers of information between different orders of correlation functions (see Fig. (a)). We show that the processing within internal network layers is captured by correlations up to second order. In addition, we demonstrate how the input layer also extracts higher order correlations from the data. Thus, by presenting different correlation orders in the input, we identify key statistics in the data (see Fig. (b)-(d)). As a next step, we consider recurrent time-continuous networks, reminiscent of biological neuronal networks (NeuralODEs, [3]). We derive a Fokker-Planck equation describing the evolution of the probability distribution. This formulation allows us to study time-dependent information flow between different interaction terms. In summary, this work provides insights into functional principles of information processing in neural networks.References:[1] Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1), 106.[2] Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., & Yamins, D. L. (2021). Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), e2014196118.[3] Chen, R. T., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31