Reproducible neural network simulations: model validation on the level of network activity data

Modern neuroscience relies on simulations of neural network models to bridge the gap between the experimentally observed activity dynamics in the brain and the theory of neural networks. The quantitative assessment of both experimental and simulated data is the basis for a rigorous validation practice and an indispensable part of any simulation workflow. Moreover, the variety of simulation tools and frameworks, and levels of model description also requires the validation of model implementations against each other to evaluate their equivalence. Despite rapid advances in the formalized description of models [1], data [2], and analysis workflows [3,4], there is no accepted consensus regarding the terminology and practical implementation of validation workflows. This situation prevents the generic, unbiased, and rigorous comparison between published models and data sets, which is a key element of building biologically realistic models in computational neuroscience.Here, we relate terminology from the literature on validation to our domain and argue for the establishment of standardized statistical test metrics that enable formalized quantitative validation workflows. Furthermore, we emphasize the importance of validation testing on the level of network activity. Despite the importance of validating the elementary components of a simulation, such as single cell dynamics for networks of spiking neurons [5], constructing networks from validated building blocks does not entail the validity of the simulation on the network scale. Therefore, we here describe and discuss a set of complementary tests based on features of network dynamics, ranging from mono- and bivariate statistics such as firing rates and correlation coefficients to more model specific features such as statistics of spatiotemporal spiking patterns. Furthermore, we also present the formal implementation of these tests in the Python module NetworkUnit (RRID:SCR_016543) and the Elephant (RRID:SCR_003833) analysis framework.We illustrate a corresponding example workflow that practically demonstrates the iterative comparison, or [*i*]substantiation[*/i*], of a spiking neural network model simulation against its reproduction on the SpiNNaker [6] neuromorphic system, validating the ability of the SpiNNaker system to accurately simulate network models [7,8,9]. Using that framework we also showcase the application to the validation of a network model on experimental data [10].References1] Davison, A. et al. (2008) Front. Neuroinf. 2, 112] Zehl, L. et al. (2016) Front. Neuroinf. 10, 263] Denker, M., and Grün, S. (2016) In: Brain-Inspired Computing, Springer, 58–724] Senk, J. et al. (2016) In: JARA High-Performance Computing Symposium, Springer, 243-256 5] scidash.github.io/neuronunit.html6] Furber, S. et al. (2013) IEEE Transactions on Computers 62, 2454–24677] van Albada, S. J. et al. (2018) Front. Neurosci. 12, 2918] Trensch, G. et al. (sub.) 9] Gutzen R. et al. (sub.)10] Brochier, T. et al. (2018) Sci. Data 5, 180055AcknowledgmentsEU Horizon 2020 Framework Programme for Research and Innovation under Specific Grant Agreements No. 720270 and No. 785907 (Human Brain Projects SGA1 and SGA2). DFG Grants DE 2175/2-1 and GR 1753/4-2 Priority Program (SPP 1665). LIA “Vision for Action” between CNRS and Aix-Marseille Univ, Marseille, France, and Research Centre Juelich, Germany. Helmholtz Association Initiative and Networking Fund under project number ZT-I-0003