Forgetful Inference in a Sophisticated World Model

Humans and other animals are able to discover underlying statistical structure in their environ- ments and exploit it to achieve efficient and effective performance. However, the largest scale structures such as ‘world models’ are often difficult to learn and use because they are obscure, involving long-range temporal dependencies. Here, we analyzed behavioral data from a lengthy experiment with rats, showing that subjects discovered such hidden structure, using it to respond more quickly to rewarding states whilst responding more slowly, or not at all, to unrewarding states. We also identified surprising occasions where subjects responded rapidly to unrewarding states, despite the structure of the task seemingly having been learned. We attributed these instances to immediate inferential imperfections caused by the partial observ- ability of hidden states. To describe this process statistically, we built a hidden Markov model (HMM) of the subjects’ models of the experiment, describing overall behavior as integrating recent observations with the recollections of an imperfect memory. Over the course of training, we found that subjects came to track their progress through the task more accurately, indicating an improved ability to infer state. Model fits attributed this improvement to decreased forgetting of the previous state. This ‘learning to remember’ decreased reliance on more recent observations, which can be misleading, in favor of a more dependable memory