Finite Memory Estimation and Control of Finite Probabilistic Systems

A finite probabilistic system (FPS) is a stationary discrete-time controlled stochastic dynamical process, having finite input, output, and (internal) state sets. The partially-observable Markov decision process is an example of such a system. FPS formulations provide a convenient framework for the study of problems of state estimation, statistical decision, or control, where state information is available only through a finite memoryless channel, and observation dynamics may depend on the inputs selected. Notions of reachability and detectability in FPS's (similar to controllability and observability in linear systems) are made precise. It is shown that every FPS can be reduced to components that are either reachable and detectable, or transient, or null-recurrent. It is well known that the information vector (whose i-th entry is the a posteriori probability that the system is in state i) is a sufficien