Comparison of activation functions in multilayer neural network for pattern classification

This paper discusses properties of activation functions in multilayer neural network applied to pattern classification. A rule of thumb for selecting activation functions or their combination is proposed. The sigmoid, Gaussian and sinusoidal functions are selected due to their independent and fundamental space division properties. The sigmoid function is not effective for a single hidden unit. On the contrary, the other functions can provide good performance. When several hidden units are employed, the sigmoid function is useful. However, the convergence speed is still slower than the others. The Gaussian function is sensitive to the additive noise, while the others are rather insensitive. As a result, based on convergence rates, the minimum error and noise sensitivity, the sinusoidal function is most useful for both without and with additive noise. Property of each function is discussed based on the internal representation, that is the distributions of the hidden unit inputs and outputs. Although this selection depends on the input signals to be classified, the periodic function can be effectively applied to a wide range of application fields