The numerical exploitation of periodicity in Markov decision processes

Periodicity is a simple form of nonstationarity in Markov decision processes. In this paper successive approximations are considered for discounted and undiscounted periodic Markov decision processes. For this type of process iteration steps can be performed without any loss of efficiency, provided the type of procedure is sensibly chosen. Moreover, and most important, it is possible to derive sharp bounds for the value function. Numerical evidence is provided