Analog versus Discrete Neural Networks

We show that neural networks with three-times continuously differentiable activation functions are capable of computing a certain family of n-bit Boolean functions with two gates, whereas networks composed of binary threshold functions require at least \Omega\Gammaast n) gates. Thus, for a large class of activation functions, analog neural networks can be more powerful than discrete neural networks, even when computing Boolean functions. 1 Introduction. Artificial neural networks have become a popular model for machine learning and many results have been obtained regarding their application to practical problems. Typically, the network is trained to encode complex associations between inputs and outputs during supervised training cycles, where the associations are encoded by the weights of the network. Once trained, the network will compute an input/output mapping which (hopefully) is a good approximation of the original mapping. 1 Partially supported by NSF Grant CCR-9114545 In thi..