The value of observation for monitoring dynamic systems

We consider the fundamental problem of monitor-ing (i.e. tracking) the belief state in a dynamic sys-tem, when the model is only approximately correct and when the initial belief state might be unknown. In this general setting where the model is (perhaps only slightly) mis-specified, monitoring (and con-sequently planning) may be impossible as errors might accumulate over time. We provide a new characterization, the value of observation, which allows us to bound the error accumulation. The value of observation is a parameter that gov-erns how much information the observation pro-vides. For instance, in Partially Observable MDPs when it is 1 the POMDP is an MDP while for an unobservable Markov Decision Process the param-eter is 0. Thus, the new parameter characterizes a spectrum from MDPs to unobservable MDPs de-pending on the amount of information conveyed in the observations.