Online Learning of Optimal Control Solutions Using Integral Reinforcement Learning and Neural Networks 1

Abstract — In this paper we introduce an online algorithm that uses integral reinforcement knowledge for learning the continuous-time optimal control solution for nonlinear systems with infinite horizon costs and partial knowledge of the system dynamics. This algorithm is a data based approach to the solution of the Hamilton-Jacobi-Bellman equation and it does not require explicit knowledge on the system’s drift dynamics. The adaptive algorithm use the structure of policy iteration, and it is implemented on an actor/critic structure. Both actor and critic neural networks are adapted simultaneously and a persistence of excitation condition is required to guarantee convergence of the critic to the actual optimal value function. &ovel tuning algorithms are given for both critic and actor networks, with extra terms in the actor tuning law being required to guarantee closed-loop dynamical stability. The convergence to the optimal controller is proven, and stability of the system is also guaranteed. Simulation example support the theoretical result. I