Which temporal difference learning algorithm best reproduces dopamine activity in a multi-choice task

Abstract. The activity of dopaminergic (DA) neurons has been hypoth-esized to encode a reward prediction error (RPE) which corresponds to the error signal in Temporal Difference (TD) learning algorithms. This hypothesis has been reinforced by numerous studies showing the rele-vance of TD learning algorithms to describe the role of basal ganglia in classical conditioning. However, recent recordings of DA neurons dur-ing multi-choice tasks raised contradictory interpretations on whether DA’s RPE signal is action dependent or not. Thus the precise TD algo-rithm (i.e. Actor-Critic, Q-learning or SARSA) that best describes DA signals remains unknown. Here we simulate and precisely analyze these TD algorithms on a multi-choice task performed by rats. We find that DA activity previously reported in this task is best fitted by a TD error which has not fully converged, and which converged faster than observed behavioral adaptation