Dynamics and Computation in Heterogeneous Neural Networks

Networks composed of a large number of interacting neurons form a basis of complex, real-time computation in biological organisms and have spurred a machine learning revolution in the past decade. Empirical studies have revealed the ubiquity of heavy-tailed, heterogeneous coupling across both biological and pretrained artificial neural networks, suggesting that the rich, complex dynamics emerging from heterogeneity may form a general principle for computation underlying both systems. Meanwhile, theoretical mean-field models have predicted the emergence of ordered and chaotic regimes of network activity using the assumption of homogeneous, Gaussian connectivity. In this thesis, we draw from and contribute a number of advances to non-equilibrium statistical physics, condensed matter physics and random matrix theory, and build a theory of heterogeneously coupled neural networks across a variety of spiking, rate-based and artificial deep architectures. We find an extended dynamical regime alongside the classical ordered and chaotic phases which displays hallmarks of criticality and extends the classical edge of chaos arising in the Gaussian limit. The extended critical regime emerging from heterogeneous neural networks accounts for a variety of biologically realistic neural dynamics without fine-tuning, provides advantages in a reservoir computing context over classical chaos, and balances the compression and expansion of internal neural representations. We show how these dynamical features of heterogeneous neural networks yield a general principle of computation regardless of architecture or problem context, allowing us to provide theoretically-informed design guidelines to improve the performance of deep neural networks which we numerically verify. These results indicate that our theory provides a foundation for understanding shared dynamical and computational principles of biological and artificial intelligent systems