POSTER PRESENTATION Open Access Estimating numerical error in neural network simulations on Graphics Processing Units

Modern graphics processing units (GPUs) are becoming a popular hardware substrate for spiking neural network simulations [1-3], due to their massive parallelism and impressive cost-to-speed ratio. However, verifying and interpreting the results of a GPU simulation can be dif-ficult because the results are never exactly reproducible, unlike those from an equivalent serial simulation on a CPU; not only is the simulation subject to the usual rounding errors of floating-point arithmetic, but there are also elements of stochasticity due to the non-deter-minism of the thread scheduling mechanism on the hard-ware, alongside the non-associativity of floating-point addition and multiplication. Consider, for example, a typical postsynaptic integration step for the summation of incoming synapse currents, executed on a GPU. If there are multiple threads, which are each simulating an incoming synapse, there is no guarantee for the order in which each synapse thread’s current will be accumulated into the total current. Therefore, rounding errors will be different depending on this order and the result could be different every time the simulation runs. Such effects would initially be small but can be amplified in unstable or chaotic systems to a degree that the final results appear completely random across different runs (see Figure 1). When comparing runs between GPU and CPU imple-mentations there are additional sources of divergence. There are subtle differences in the way each architectur