Why Deep Neural Networks: A Possible Theoretical Explanation

Abstract. In the past, the most widely used neural networks were 3-layer ones. These networks were preferred, since one of the main ad-vantages of the biological neural networks – which motivated the use of neural networks in computing – is their parallelism, and 3-layer net-works provide the largest degree of parallelism. Recently, however, it was empirically shown that, in spite of this argument, multi-layer (“deep”) neural networks leads to a much more efficient machine learning. In this paper, we provide a possible theoretical explanation for the somewhat surprising empirical success of deep networks. 1 Formulation of the Problem Why neural networks. In spite of all the progress in computer-based recog-nition algorithms, we humans still perform many recognition tasks much faster (and often much more reliably) than computer programs. And we perform faster in spite of the fact that the fastest of our brain’s data processing units – neurons – has reaction time 10 msec, while computer components operate in nanosec