Beyond Gaussian Processes: On the Distributions of Infinite Networks

A general analysis of the limiting distribution of neural network functions is performed, with emphasis on non-Gaussian limits. We show that with i.i.d. symmetric stable output weights, and more generally with weights distributed from the normal domain of attraction of a stable variable, that the neural functions converge in distribution to stable processes. Condi-tions are also investigated under which Gaussian limits do occur when the weights are independent but not identically distributed. Some par-ticularly tractable classes of stable distributions are examined, and the possibility of learning with such processes.