Caution, probing, and the value of information in the control of uncertain systems

This paper discusses the con trot of nonlinear stochastic systems and, in particular, linear systems with unknown parameters. It is shown how the optimal policy utilizes preposierior analysis to obtain the control values- The stochastic nature of the problem leads to the probing and caution properties of the control. Explicit expressions of the probing and caution terms in a stochastic control problem are presented. These terms are obtained by a closed- loop approximation of the stochastic dynamic programming equation. An approximate value of information can be evaluated and the benefit to be derived from probing (experimentation) can be traded off against its cost. The interplay between caution and probing is illustrated by an example The performance of the closed-loop control obtained from the above approximation of the stochastic dynamic programming is compared with severalothersuboptimalcontrols as well as the optimal one. 1