Fitting Multi-Layer Perceptron Models Without Derivatives

The Nelder--Mead simplex algorithm (J. A. Nelder and R. Meade, Computer Journal, vol 7, pages 308-- 313, 1965) for function minimization is explored as a method for fitting multi--layer perceptron models without derivatives. While it is not as successful in some examples as the most sophisticated methods from the numerical analysis literature, for a method that uses neither derivatives nor a line search algorithm, it is surprisingly effective. Some instances where it might be the method of choice are indicated. 1 Introduction Learning in a neural network consists in minimizing the discrepancies between the internal representation and the external assignation. To implement this in a neural network, we impose a penalty function, ae = ae(output,target), that has a minimum value when the output values equal the target values. We then minimize ae. Minimization techniques can be divided into two types: local, which find a local minimum; and global 1 , which attempt to find a global mini..