Experienced rewards and modeled behavioral response.

<p> <b>(A)</b> The switch from matching shoulders (MS) to rising optimum (RO) reward structures was signaled by a large decrease in immediate reward return (∼60%). However, the switch from the flat returns (FR) structure to the pseudorandom (PR) condition did not elicit a similar change in experienced reward. Reward S.E. is indicated by vertical bars at each choice. <b>(B)</b> Subject decisions were predicted using a reinforcement learning model with two different methods to determine the probability to choose a certain action (ε-greedy method and sigmoid method). For both methods, we assume that subjects maintained independent estimates of the reward expected for each choice, A and B, and updated these values based on experienced rewards using the Rescorla-Wagner learning algorithm. Choices were assumed to be 1) probabilistically related to choice values according to a sigmoid function (softmax method, green curve) or 2)have a fixed probability of 1-ε/2 for choice associated with bigger weight (ε-greedy method, pink curve). Panel B indicates the relationships between predicted and actual choices. Decisions were binned (x-axis) based on predicted likelihood that subjects would choose A. Y-values indicate the actual average allocation to A for all choices within each bin. Linear regression shows there is a strong correlation between predicted and actual choices. (MS: r = 0.97, RO: r = 0.99, FR: r = 0.97, PR: r = 0.97 for softmax method; MS: r = 0.97, RO: r = 0.99, FR: r = 0.95, PR: r = 0.99 for ε-greedy method). ε-greedy method restrict each subject only have one probability to choose certain choice (A or B) over 250 trials, while sigmoid method allows subject to generate different probability to choose specific choice for each trial in the 250 trial task. This is why ε-greedy method does not cover all the spectrum of X-axis as sigmoid method does.