Reinforcement Learning for Problems with Hidden State

In this paper, we describe how techniques from reinforcement learning might be used to approach the problem of acting under uncertainty. We start by introducing the theory of partially observable Markov decision processes (POMDPs) to describe what we call hidden state problems. After a brief review of other POMDP solution techniques, we motivate reinforcement learning by considering an agent with no previous knowledge of the environment model. We describe two major groups of reinforcement learning techniques: those the learn a value function over states of world, and those that search in the space of policies directly. Finally, we discuss the general problems with these methods, and suggest promising avenues for future research.