Fundamental aspects of noise in analog-hardware neural networks

International audienceWe study and analyze the fundamental aspects of noise propagation inrecurrent as well as deep, multilayer networks. The motivation ofour study is neural networks in analog hardware; yet, the methodology provides insight into networks in general. Considering noisy linearnodes, we investigate the signal-to-noise ratio at the network’s outputs, which determines the upper limit of computational precision. We con-sider additive and multiplicative noise, which can be purely local as well as correlated across populations of neurons. This covers the chiefinternal-perturbations of hardware networks, and noise amplitudes were obtained from a physically implemented neural network. Analyticallyderived descriptions agree exceptionally well with numerical data, enablingclear identification of the components critical for management andmitigation of noise. We find that analog neural networks are surprisingly robust, in particular, against noisy neurons. Their uncorrelated pertur-bations are almost fully suppressed, while correlated noise can accumulate. Our work identifies notoriously sensitive points while highlightinga surprising robustness of such computational systems