On the Activation Function and Fault Tolerance in Feedforward Neural Networks

The influence of the activation function on fault tolerance property of the feedforward neural networks is empirically investigated. The simulation results show that the activation function largely influences the fault tolerance and the generalization property of neural networks. The neural networks with symmetric sigmoid activation function is largely fault tolerant than the networks with asymmetric sigmoid function. The close relation between the fault tolerance and the the generalization property was not observed and the networks with asymmetric activation function slightly generalize better than the network with the symmetric activation function. An XOR-like problem that allows a practical investigation of the fault tolerance property of the networks with different activation functions is presented. Then the results are evaluated on character recognition problem on which the generalization ability is investigated. 1 Introduction Feedforward neural networks (NNs), trained with ba..