Using heuristic models to understand human and optimal decision-making on bandit problems

We study bandit problems in which a decision-maker gets reward-or-failure feedback when choosing repeat-edly between two alternatives, with fixed but unknown reward rates, over a short sequence of trials. We col-lect data across a number of types of bandit problems to analyze five heuristics—four seminal heuristics from machine learning, and one new model we develop—as models of human and optimal decision-making. We find that the new heuristic, known as τ-switch, which assumes a latent search state is followed by a latent stand state to control decision-making on key trials, is best able to mimic optimal decision-making, and best account for the decision-making of the majority of our experimental par-ticipants. We show how these results allow human and optimal decision-making to be characterized and com-pared in simple, psychologically interpretable ways, and discuss some theoretical and practical implications