Neural networks for solving linear and quadratic programming problems with modified Newton’s and Levenberg-Marquardt methods

Constrained optimization problems entail the minimization or maximization of a linear or quadratic objective function that is subject to linear equality and inequality restrictions. They are very important, appearing in real world scenarios including signal processing, identification, the design of filters and function approximation. In recent years, Artificial Neural Networks (ANNs) have been applied to several classes of constrained optimization problems, with promising results. Most of the ANN methods involve an iterative process, in which a feasible direction that decreases the objective function is determined at each step and a one-dimensional optimization is performed along this direction until a predetermined stopping criterion is satisfied. Normally, programming problems are solved through penalty function methods, often entailing a sequence of unconstrained optimization problems for different values of penalty parameters to ensure convergence. Currently, there are two main neural-network models for solving linear programming (LP) and quadratic programming (QP) problems. The first employs the steepest descent method and the penalty function method with an objective function that can be viewed as an "inexact" penalty function that can only obtain truly optimal solutions when the penalty parameter is infinite. In the second approach, the solution is formed by two mutually exclusive subsystems. Both of these specialized networks can solve LP and QP problems in execution times that are several orders of magnitude faster than the most popular numerical algorithms for general purpose digital computers. Nevertheless, the use of penalty function methods has been found to be somewhat disadvantageous due to inherent numerical instabilities. It is generally impossible to choose a very large penalty parameter in the networks making it very difficult to obtain approximate solutions and impossible to find accurate solutions. Moreover, the existing models concentrate on the steepest descent method. Here, drawbacks in this method are discussed, followed by a choice of two popular numerical algorithms, Newton's method and the Levenberg-Marquardt (L-M) method, as the focus of the investigation. After demonstrating the operation of these algorithms in a neural network setting, modifications to them are introduced. Benchmark LP and QP problems are utilized to illustrate the progress of the steepest descent algorithm and subsequently the superior performance of the modified Newton's and L-M methods