A (Revised) Survey of Approximate Methods for Solving Partially Observable Markov Decision Processes

Partially observable Markov decision processes (POMDPs) are interesting because they provide a general framework for learning in the presence of multiple forms of uncertainty. We survey methods for learning within the POMDP framework. Because exact methods are intractable we concentrate on approximate methods. We explore two versions of the POMDP training problem: learning when a model of the POMDP is known, and the much harder problem of learning when a model is not available. The methods used to solve POMDPs are sometimes referred to as reinforcement learning algorithms because the only feedback provided to the agent is a scalar reward signal at each time step.