Neural Networks and Global Optimization

Abstract – Training a neural network is a difficult optimization problem because of numerous local minimums. As an alternative to local search algorithms, many global search algorithms have been used to train neural networks. However, local search algorithms are more efficient with computational resources, and therefore numerous random restarts with a local algorithm may be more effective than a global algorithm at obtaining a low value of the objective function. This study uses Monte-Carlo simulations to determine the relative efficiency of a local search algorithm to 9 stochastic global algorithms: 2 simulated annealing algorithms, 1 simple random stochastic algorithm, 1 genetic algorithm and 5 evolutionary strategy algorithms. The computational requirements of the global algorithms are several times higher than the local algorithm and there is little or no gain in using the global algorithms to train neural networks