schulz_2020_finding

bandit psych-101

Five online multi-armed bandit experiments, each run on mechanical Turk: participants repeatedly choose among 8 arms arranged spatially (left-to-right), collecting noisy rewards whose expected value follows a latent function of arm position — linear gradient (Exp 1, N=119), scrambled rewards (Exp 2, N=82), linear with per-round random intercept (Exp 3, N=131), dynamics of structure learning with SRS/RSR block schedules (Exp 4, N=113), and change-point functions (Exp 5, N=159). Responses are single-key arm choices on each of 10 trials per round, with 30 fresh bandits (rounds) per participant; reward outcome and sometimes response time are recorded per trial. The research question is whether humans detect and exploit latent functional structure among options to guide exploration–exploitation (learning-to-learn across rounds); analysis compares hierarchical Bayesian models of Gaussian-process function learning, clustering of reward distributions, and uncertainty-guided exploration, and all three headline effects of Experiment 1 (structure reward benefit, within-round learning, and learning across rounds) reproduce on these files.

Responses
143,196
14,136 held out
CORE effects
–
no effects simulated
CORE test NLL (avg)
0.641
rank 3 of 7 models
Best model
Centaur
test NLL (avg) 0.637

Likelihood of human responses

modelPNLLtest NLL (avg)test responses train NLL (avg)train responses
CORE9,057.80.640814,1360.6472129,060
CORE (no prediction errors)9,109.20.644414,1360.6452129,060
CORE (no phonemes)9,039.70.639514,1360.6310129,060
CORE (no forgetting)9,941.00.703214,1360.7162129,060
CORE (no hierarchy)13,194.90.933414,1360.9638129,060
Transformer10,923.30.772714,1360.7967129,060
Centaur best9,003.00.636914,136––

Experiments

exptask classparticipantsresponsesconditionssimulatedeffects
exp0LinearBandit11824,7101no–
exp1ScrambledBandit8218,8381no–
exp2ShiftedBandit13129,0811no–
exp3DynamicsBandit11125,3671no–
exp4ChangepointBandit15945,2002no–

Behavioral effects

No behavioral effects were simulated for this study.

Transcript

In this game you play 30 rounds of a slot-machine task. Each round, eight boxes are arranged left to right and labeled with the keys you press to choose them: A, S, D, F, J, K, L, and ; (semicolon), leftmost to rightmost. Each round has 10 trials. On each trial you press one key to sample that box, and you then see how many points that box gave you this time. Your goal is to gain as many points as possible. After each round the game resets: the same keys can now give different points, so treat every round afresh.

Round 1 of 30.
Trial 1: You press A and gain 5.98 points.
Trial 2: You press S and gain 11.22 points.
Trial 3: You press D and gain 16.76 points.
Trial 4: You press F and gain 22.92 points.
Trial 5: You press J and gain 28.46 points.
Trial 6: You press K and gain 34.17 points.
Trial 7: You press L and gain 40.21 points.
Trial 8: You press ; and gain 45.33 points.
Trial 9: You press A and gain 6.32 points.
Trial 10: You press ; and gain 45.35 points.

Round 2 of 30.
Trial 1: You press A and gain 2.7 points.
Trial 2: You press ; and gain 43.57 points.
Trial 3: You press J and gain 25.96 points.
Trial 4: You press ; and gain 43.59 points.
Trial 5: You press D and gain 14.67 points.
Trial 6: You press ; and gain 43.74 points.
Trial 7: You press L and gain 36.98 points.
Trial 8: You press ; and gain 43.47 points.
Trial 9: You press ; and gain 42.87 points.
Trial 10: You press ; and gain 43.46 points.

Round 3 of 30.
Trial 1: You press ; and gain 40.3 points.
Trial 2: You press ; and gain 40.26 points.
Trial 3: You press ; and gain 40.21 points.
Trial 4: You press F and gain 12.86 points.
Trial 5: You press ; and gain 40.24 points.
Trial 6: You press ; and gain 39.75 points.
Trial 7: You press ; and gain 39.94 points.
Trial 8: You press J and gain 28.77 points.
Trial 9: You press ; and gain 39.57 points.
Trial 10: You press ; and gain 40.22 points.

Round 4 of 30.
Trial 1: You press ; and gain 14.33 points.
Trial 2: You press F and gain 33.09 points.
Trial 3: You press A and gain 37.77 points.
Trial 4: You press D and gain 42.83 points.
Trial 5: You press K and gain 19.02 points.
Trial 6: You press S and gain 28.55 points.
Trial 7: You press D and gain 42.63 points.
Trial 8: You press F and gain 33.33 points.
Trial 9: You press D and gain 42.52 points.
Trial 10: You press D and gain 43.09 points.

Round 5 of 30.
Trial 1: You press D and gain 18.87 points.
Trial 2: You press K and gain 36.18 points.
Trial 3: You press A and gain 7.52 points.
Trial 4: You press L and gain 42.03 points.
Trial 5: You press ; and gain 47.7 points.
Trial 6: You press ; and gain 47.62 points.
Trial 7: You press ; and gain 47.49 points.
Trial 8: You press ; and gain 47.95 points.
Trial 9: You press ; and gain 47.96 points.
Trial 10: You press ; and gain 47.65 points.

Round 6 of 30.
Trial 1: You press D and gain 35.49 points.
Trial 2: You press J and gain 24.22 points.
Trial 3: You press S and gain 39.81 points.
Trial 4: You press S and gain 40.13 points.
Trial 5: You press A and gain 44.99 points.
Trial 6: You press A and gain 45.54 points.
Trial 7: You press A and gain 45.81 points.
Trial 8: You press A and gain 45.55 points.
Trial 9: You press A and gain 46.1 points.
Trial 10: You press A and gain 45.33 points.

Round 7 of 30.
Trial 1: You press J and gain 28.08 points.
Trial 2: You press L and gain 40.67 points.
Trial 3: You press ; and gain 47.31 points.
Trial 4: You press ; and gain 47.54 points.
Trial 5: You press ; and gain 47.67 points.
Trial 6: You press ; and gain 47.52 points.
Trial 7: You press ; and gain 47.18 points.
Trial 8: You press ; and gain 47.57 points.
Trial 9: You press ; and gain 47.22 points.
Trial 10: You press ; and gain 47.15 points.

Round 8 of 30.
Trial 1: You press A and gain 47.7 points.
Trial 2: You press F and gain 30.44 points.
Trial 3: You press A and gain 47.39 points.
Trial 4: You press A and gain 46.73 points.
Trial 5: You press A and gain 47.14 points.
Trial 6: You press A and gain 47.49 points.
Trial 7: You press A and gain 46.84 points.
Trial 8: You press A and gain 47.33 points.
Trial 9: You press A and gain 46.98 points.
Trial 10: You press A and gain 47.65 points.

Round 9 of 30.
Trial 1: You press F and gain 35.58 points.
Trial 2: You press K and gain 20.34 points.
Trial 3: You press A and gain 30.23 points.
Trial 4: You press S and gain 45.98 points.
Trial 5: You press S and gain 45.73 points.
Trial 6: You press S and gain 45.71 points.
Trial 7: You press S and gain 45.47 points.
Trial 8: You press S and gain 46.32 points.
Trial 9: You press S and gain 46.48 points.
Trial 10: You press S and gain 45.98 points.

Round 10 of 30.
Trial 1: You press A and gain 13.81 points.
Trial 2: You press ; and gain 25.45 points.
Trial 3: You press D and gain 47.1 points.
Trial 4: You press D and gain 47.77 points.
Trial 5: You press F and gain 19.07 points.
Trial 6: You press D and gain 47.52 points.
Trial 7: You press D and gain 47.5 points.
Trial 8: You press D and gain 47.72 points.
Trial 9: You press D and gain 47.91 points.
Trial 10: You press D and gain 47.34 points.

Round 11 of 30.
Trial 1: You press A and gain 34.75 points.
Trial 2: You press ; and gain 7.88 points.
Trial 3: You press S and gain 18.3 points.
Trial 4: You press L and gain 29.12 points.
Trial 5: You press J and gain 2.81 points.
Trial 6: You press D and gain 14.05 points.
Trial 7: You press F and gain 23.99 points.
Trial 8: You press K and gain 39.71 points.
Trial 9: You press L and gain 29.17 points.
Trial 10: You press K and gain 40.2 points.

Round 12 of 30.
Trial 1: You press A and gain 44.08 points.
Trial 2: You press A and gain 43.74 points.
Trial 3: You press A and gain 43.84 points.
Trial 4: You press A and gain 43.56 points.
Trial 5: You press A and gain 43.51 points.
Trial 6: You press A and gain 44.1 points.
Trial 7: You press A and gain 44.28 points.
Trial 8: You press A and gain 43.74 points.
Trial 9: You press A and gain 43.47 points.
Trial 10: You press A and gain 43.38 points.

Round 13 of 30.
Trial 1: You press A and gain 4.59 points.
Trial 2: You press ; and gain 42.37 points.
Trial 3: You press ; and gain 41.7 points.
Trial 4: You press ; and gain 41.7 points.
Trial 5: You press ; and gain 41.82 points.
Trial 6: You press ; and gain 42.29 points.
Trial 7: You press ; and gain 41.53 points.
Trial 8: You press ; and gain 42.03 points.
Trial 9: You press ; and gain 41.67 points.
Trial 10: You press ; and gain 41.82 points.

Round 14 of 30.
Trial 1: You press ; and gain 43.08 points.
Trial 2: You press ; and gain 43.9 points.
Trial 3: You press ; and gain 43.86 points.
Trial 4: You press ; and gain 43.63 points.
Trial 5: You press ; and gain 43.44 points.
Trial 6: You press ; and gain 43.54 points.
Trial 7: You press ; and gain 43.91 points.
Trial 8: You press ; and gain 44.02 points.
Trial 9: You press ; and gain 43.77 points.
Trial 10: You press ; and gain 43.59 points.

Round 15 of 30.
Trial 1: You press ; and gain 6.33 points.
Trial 2: You press A and gain 46.03 points.
Trial 3: You press A and gain 46.06 points.
Trial 4: You press A and gain 45.75 points.
Trial 5: You press A and gain 45.84 points.
Trial 6: You press A and gain 45.6 points.
Trial 7: You press A and gain 45.88 points.
Trial 8: You press A and gain 46.22 points.
Trial 9: You press A and gain 45.88 points.
Trial 10: You press A and gain 45.43 points.

Round 16 of 30.
Trial 1: You press ; and gain 5.07 points.
Trial 2: You press A and gain 34.58 points.
Trial 3: You press J and gain 11.31 points.
Trial 4: You press S and gain 45.61 points.
Trial 5: You press S and gain 45.69 points.
Trial 6: You press S and gain 46.04 points.
Trial 7: You press S and gain 46.53 points.
Trial 8: You press S and gain 46.17 points.
Trial 9: You press S and gain 46.24 points.
Trial 10: You press S and gain 45.99 points.

Round 17 of 30.
Trial 1: You press A and gain 48.01 points.
Trial 2: You press A and gain 48.11 points.
Trial 3: You press A and gain 48.02 points.
Trial 4: You press A and gain 48.34 points.
Trial 5: You press A and gain 47.71 points.
Trial 6: You press A and gain 47.29 points.
Trial 7: You press A and gain 47.78 points.
Trial 8: You press A and gain 47.48 points.
Trial 9: You press A and gain 48.16 points.
Trial 10: You press A and gain 47.68 points.

Round 18 of 30.
Trial 1: You press A and gain 5.69 points.
Trial 2: You press ; and gain 47.5 points.
Trial 3: You press ; and gain 47.13 points.
Trial 4: You press ; and gain 46.65 points.
Trial 5: You press ; and gain 47.21 points.
Trial 6: You press ; and gain 47.22 points.
Trial 7: You press ; and gain 47.18 points.
Trial 8: You press ; and gain 47.36 points.
Trial 9: You press ; and gain 48.21 points.
Trial 10: You press ; and gain 47.06 points.

Round 19 of 30.
Trial 1: You press ; and gain 2.84 points.
Trial 2: You press A and gain 43.09 points.
Trial 3: You press A and gain 43.49 points.
Trial 4: You press A and gain 43.64 points.
Trial 5: You press A and gain 43.37 points.
Trial 6: You press A and gain 43.3 points.
Trial 7: You press A and gain 43.41 points.
Trial 8: You press A and gain 43.62 points.
Trial 9: You press A and gain 44.18 points.
Trial 10: You press A and gain 43.31 points.

Round 20 of 30.
Trial 1: You press A and gain 36.18 points.
Trial 2: You press S and gain 14.61 points.
Trial 3: You press ; and gain 31.17 points.
Trial 4: You press L and gain 9.26 points.
Trial 5: You press F and gain 25.9 points.
Trial 6: You press J and gain 4.66 points.
Trial 7: You press K and gain 19.93 points.
Trial 8: You press D and gain 41.89 points.
Trial 9: You press D and gain 41.69 points.
Trial 10: You press D and gain 42.04 points.

Round 21 of 30.
Trial 1: You press D and gain 17.58 points.
Trial 2: You press J and gain 26.63 points.
Trial 3: You press F and gain 22.67 points.
Trial 4: You press A and gain 7.89 points.
Trial 5: You press ; and gain 40.63 points.
Trial 6: You press ; and gain 40.71 points.
Trial 7: You press ; and gain 40.54 points.
Trial 8: You press ; and gain 40.63 points.
Trial 9: You press ; and gain 40.67 points.
Trial 10: You press ; and gain 40.55 points.

Round 22 of 30.
Trial 1: You press ; and gain 46.39 points.
Trial 2: You press ; and gain 46.0 points.
Trial 3: You press ; and gain 46.37 points.
Trial 4: You press ; and gain 46.18 points.
Trial 5: You press ; and gain 46.44 points.
Trial 6: You press ; and gain 45.91 points.
Trial 7: You press ; and gain 46.13 points.
Trial 8: You press ; and gain 46.28 points.
Trial 9: You press ; and gain 46.42 points.
Trial 10: You press ; and gain 46.0 points.

Round 23 of 30.
Trial 1: You press ; and gain 9.82 points.
Trial 2: You press A and gain 42.48 points.
Trial 3: You press A and gain 42.2 points.
Trial 4: You press A and gain 42.34 points.
Trial 5: You press A and gain 42.86 points.
Trial 6: You press A and gain 42.24 points.
Trial 7: You press A and gain 42.89 points.
Trial 8: You press A and gain 42.86 points.
Trial 9: You press A and gain 42.95 points.
Trial 10: You press A and gain 43.22 points.

Round 24 of 30.
Trial 1: You press A and gain 46.96 points.
Trial 2: You press A and gain 47.19 points.
Trial 3: You press A and gain 47.13 points.
Trial 4: You press A and gain 47.71 points.
Trial 5: You press A and gain 47.13 points.
Trial 6: You press A and gain 47.3 points.
Trial 7: You press A and gain 47.7 points.
Trial 8: You press A and gain 47.28 points.
Trial 9: You press A and gain 47.26 points.
Trial 10: You press A and gain 47.14 points.

Round 25 of 30.
Trial 1: You press A and gain 40.16 points.
Trial 2: You press A and gain 40.21 points.
Trial 3: You press A and gain 40.4 points.
Trial 4: You press A and gain 39.79 points.
Trial 5: You press A and gain 39.75 points.
Trial 6: You press A and gain 39.42 points.
Trial 7: You press A and gain 40.07 points.
Trial 8: You press A and gain 40.26 points.
Trial 9: You press A and gain 40.35 points.
Trial 10: You press A and gain 40.66 points.

Round 26 of 30.
Trial 1: You press L and gain 18.82 points.
Trial 2: You press ; and gain 46.72 points.
Trial 3: You press ; and gain 46.5 points.
Trial 4: You press ; and gain 47.1 points.
Trial 5: You press ; and gain 46.57 points.
Trial 6: You press ; and gain 47.1 points.
Trial 7: You press ; and gain 46.89 points.
Trial 8: You press ; and gain 46.9 points.
Trial 9: You press ; and gain 46.81 points.
Trial 10: You press ; and gain 46.68 points.

Round 27 of 30.
Trial 1: You press ; and gain 7.97 points.
Trial 2: You press A and gain 19.85 points.
Trial 3: You press K and gain 2.04 points.
Trial 4: You press D and gain 32.19 points.
Trial 5: You press S and gain 44.6 points.
Trial 6: You press S and gain 43.51 points.
Trial 7: You press S and gain 43.38 points.
Trial 8: You press S and gain 43.64 points.
Trial 9: You press S and gain 43.98 points.
Trial 10: You press S and gain 43.84 points.

Round 28 of 30.
Trial 1: You press A and gain 45.57 points.
Trial 2: You press A and gain 45.36 points.
Trial 3: You press A and gain 45.84 points.
Trial 4: You press A and gain 45.44 points.
Trial 5: You press A and gain 45.19 points.
Trial 6: You press A and gain 46.06 points.
Trial 7: You press A and gain 45.13 points.
Trial 8: You press A and gain 45.33 points.
Trial 9: You press A and gain 45.44 points.
Trial 10: You press A and gain 45.65 points.

Round 29 of 30.
Trial 1: You press A and gain 37.77 points.
Trial 2: You press S and gain 11.26 points.
Trial 3: You press ; and gain 32.86 points.
Trial 4: You press D and gain 22.33 points.
Trial 5: You press L and gain 27.41 points.
Trial 6: You press K and gain 16.7 points.
Trial 7: You press J and gain 5.26 points.
Trial 8: You press ; and gain 32.37 points.
Trial 9: You press A and gain 38.04 points.
Trial 10: You press A and gain 38.17 points.

Round 30 of 30.
Trial 1: You press A and gain 2.82 points.
Trial 2: You press ; and gain 41.84 points.
Trial 3: You press ; and gain 41.28 points.
Trial 4: You press ; and gain 41.39 points.
Trial 5: You press ; and gain 40.99 points.
Trial 6: You press ; and gain 41.65 points.
Trial 7: You press ; and gain 41.1 points.
Trial 8: You press ; and gain 41.22 points.
Trial 9: You press ; and gain 40.54 points.
Trial 10: You press ; and gain 41.14 points.