Representation of Choice and Reward Action Values in Decision- Making Neural Circuits

We studied the effects of past choices and rewards in decision-making. Reinforcement learning paradigm estimates action value that represents the amount of expected reward associated with a particular action and this value determines the choice of future action in a probabilistic manner. Hence, the estimation of action value is critical for understanding animal’s behavior and its relationship to neural activity. Previous studies estimated action value based on past rewards (1, 2). However, it is unclear whether past rewards are sufficient or other factors should be considered for accurate estimation of action value. A recent study has shown that behavioral choices of animal are well predicted by a dynamic model that takes into account not only past rewards but also past choices (3), suggesting that the effect of past choices must be included in the estimation of action values. In this work, we divided action value into two parts- choice and reward action values that quantify the effects of past choices and rewards, respectively. Past choices are represented as the length of the same choices (run-length). The choice action value for a given run-length can be estimated based on behavioral data during steady state, in which mean reward contingency associated with each choice stays constant, using a generalized linear model, because the reward action value term becomes constant in this condition (3). Assuming animal’s behavior depends on the latest run-length, the coefficient for the latest run-lengt