Simultaneous Evolution of Structure and Activation Function Types in Generalized Multi-Layer Perceptrons

Abstract:- The most common (or even only) choice of activation functions for multi–layer perceptrons (MLPs) widely used in research, engineering and business is the logistic function. Among the reasons for this popularity are its boundedness in the unit interval, the function’s and its derivative’s fast computability, and a number of amenable mathematical properties in the realm of approximation theory. However, considering the huge variety of problem domains MLPs are applied in, it is intriguing to suspect that specific problems call for single or a set of specific activation functions. Also, biological neural networks (BNNs) with their enormous variety of neurons mastering a set of complex tasks may be considered to motivate this hypothesis. We present a number of experiments evolving structure and activation function types (AFTs) of generalized multi–layer perceptrons (GMLPs) using the parallel netGEN system to train the evolved architectures. The number of network parameters subjected to evolution is increased in various steps from learning parameters only for a GMLP of fixed architecture to simultaneous evolution of structure and activation function types. For experimental comparisons we utilize a synthetic and a real–world classification problem, and a chaotic time series prediction task