On the Complexity of Solving Markov Decision Problems

Markov decision problems (MDPs) provide the foundations for a number of problems of interest to AI researchers studying automated planning and reinforcement learning. In this paper, we summarize results regarding the complexity of solving MDPs and the running time of MDP solution algorithms. We argue that, although MDPs can be solved efficiently in theory, more study is needed to reveal practical algorithms for solving large problems quickly. To encourage future research, we sketch some alternative methods of analysis that rely on the structure of MDPs. 1 INTRODUCTION A Markov decision process is a controlled stochastic process satisfying the Markov property with costs assigned to state transitions. A Markov decision problem is a Markov decision process together with a performance criterion. A solution to a Markov decision problem is a policy , mapping states to actions, that (perhaps stochastically) determines state transitions to minimize the cost according to the performance crite..