Optimal Tracking Control of Uncertain Systems: On-Policy and Off-Policy Reinforcement Learning Approaches

This chapter presents adaptive solutions to the optimal tracking problem of nonlinear discrete-time and continuous-time systems. These methods find the optimal feedback and feedforward parts of the control input simultaneously, without requiring complete knowledge of the system dynamics. First, an augmented system composed of the error system dynamics and the reference trajectory dynamics is formed to introduce a new nonquadratic discounted performance function for the optimal tracking control problem. This encodes the input constrains caused by the actuator saturation into the optimization problem. Then, the tracking Bellman equation and the tracking Hamilton-Jacobi-Bellman equation for both discrete-time and continuous-time systems are derived. Finally, to obviate the requirement of complete knowledge of the system dynamics in finding the Hamilton-Jacobi-Bellman solution, integral reinforcement learning and off-policy reinforcement learning algorithms are developed for continuous-time systems, and a reinforcement learning algorithm on an actor-critic structure is developed for discrete-time systems