gershman_2018_deconstructing
Two two-armed bandit experiments on Amazon Mechanical Turk (Exp 1: N = 45; Exp 2: N = 44) in which participants chose between two slot machines for 20 blocks of 10 trials each, with Gaussian reward feedback and each block introducing a fresh bandit (newly drawn arm means). In Exp 1 a stochastic arm (mean ~ N(0,10), variance 10) competed with a fixed 0-point arm; in Exp 2 both arms were stochastic with means ~ N(0,100). Per trial the data record the participant's choice, reward in points, and response time. The paper tests whether human exploration reflects directed exploration (UCB-style uncertainty bonuses, visible as a bias shift in a probit regression) and random exploration (Thompson sampling, visible as a slope shift), using per-block Kalman-filter value/uncertainty estimates; Bayesian model comparison favors a hybrid of UCB and Thompson sampling.
Responses
17,800
1,600 held out
CORE effects
1 / 1
CORE test NLL (avg)
0.267
rank 2 of 7 models
Best model
Centaur
test NLL (avg) 0.259
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE | 426.5 | 0.2666 | 1,600 | 0.2809 | 16,200 |
| CORE (no prediction errors) | 434.9 | 0.2718 | 1,600 | 0.2869 | 16,200 |
| CORE (no phonemes) | 428.1 | 0.2675 | 1,600 | 0.2784 | 16,200 |
| CORE (no forgetting) | 461.7 | 0.2886 | 1,600 | 0.3132 | 16,200 |
| CORE (no hierarchy) | 621.1 | 0.3882 | 1,600 | 0.4418 | 16,200 |
| Transformer | 540.3 | 0.3377 | 1,600 | 0.3724 | 16,200 |
| Centaur best | 414.1 | 0.2588 | 1,600 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | TwoArmedBandit | 45 | 9,000 | 1 | yes | first_trial_uncertainty_preference |
| exp1 | DualStochasticBandit | 44 | 8,800 | 1 | yes | first_trial_uncertainty_preference |
Behavioral effects
| effect | experiment | paper | CORE | CORE (no prediction errors) | CORE (no phonemes) | CORE (no forgetting) | CORE (no hierarchy) |
|---|---|---|---|---|---|---|---|
first_trial_uncertainty_preference | exp0,exp1 | +1.443 | ✓ yes +0.310 | ✗ no +0.669 | ✓ yes +1.005 | ✗ no -0.015 | ✗ no -0.824 |
on the first trial of each block participants prefer the risky arm 1 in Exp 1 (P(arm 1) > .5; paper: t(44)=9.68, p < .001) and show NO such preference in Exp 2, where both arms have equal prior uncertainty (paper: p = .95; reproduces as p > .05). | |||||||
Transcript
Welcome. In this task you have a choice between two slot machines, represented by colored buttons. When you choose the left (variable) machine, you will win or lose points. The left machine will not always give you the same points, but it will tend to give points around its average value. When you choose the right (fixed) machine, you will always get 0 points. Your goal is to choose the slot machine that will give you the most points. After making your choice, you will receive feedback about the outcome. You will play 20 games, each with a different left (variable) slot machine (the right, fixed machine will always stay the same). Each game will consist of 10 trials. Press A to choose the left (variable) machine and B to choose the right (fixed) machine. Game 1: a new pair of slot machines appears. You press A. You receive 2 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive -1 points. You press B. You receive 0 points. You press A. You receive 0 points. Game 2: a new pair of slot machines appears. You press B. You receive 0 points. You press A. You receive 1 points. You press A. You receive -1 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive 2 points. Game 3: a new pair of slot machines appears. You press A. You receive -2 points. You press A. You receive -1 points. You press B. You receive 0 points. You press A. You receive -3 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. Game 4: a new pair of slot machines appears. You press A. You receive -2 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive -1 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. Game 5: a new pair of slot machines appears. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive -1 points. You press A. You receive -1 points. You press A. You receive -1 points. You press A. You receive 1 points. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive 1 points. Game 6: a new pair of slot machines appears. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive -1 points. You press A. You receive -1 points. You press A. You receive -2 points. You press A. You receive -2 points. Game 7: a new pair of slot machines appears. You press A. You receive 3 points. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive 2 points. Game 8: a new pair of slot machines appears. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive -2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive -2 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive 1 points. Game 9: a new pair of slot machines appears. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive 1 points. Game 10: a new pair of slot machines appears. You press A. You receive -3 points. You press B. You receive 0 points. You press A. You receive -3 points. You press A. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. Game 11: a new pair of slot machines appears. You press A. You receive 0 points. You press A. You receive -1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive -1 points. You press A. You receive -1 points. You press A. You receive -2 points. Game 12: a new pair of slot machines appears. You press A. You receive 2 points. You press A. You receive 2 points. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 3 points. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 3 points. You press A. You receive 0 points. Game 13: a new pair of slot machines appears. You press A. You receive -1 points. You press A. You receive 0 points. You press B. You receive 0 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive 1 points. Game 14: a new pair of slot machines appears. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 3 points. You press A. You receive 3 points. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive 1 points. Game 15: a new pair of slot machines appears. You press A. You receive -2 points. You press A. You receive -2 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. Game 16: a new pair of slot machines appears. You press A. You receive -1 points. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive -1 points. You press A. You receive 1 points. You press A. You receive -1 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. Game 17: a new pair of slot machines appears. You press A. You receive -2 points. You press A. You receive -1 points. You press A. You receive -2 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. You press B. You receive 0 points. Game 18: a new pair of slot machines appears. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive -3 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive 1 points. You press A. You receive -1 points. You press A. You receive -1 points. Game 19: a new pair of slot machines appears. You press A. You receive 2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 3 points. You press A. You receive 1 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive 0 points. Game 20: a new pair of slot machines appears. You press A. You receive 0 points. You press A. You receive 2 points. You press A. You receive 2 points. You press A. You receive 2 points. You press A. You receive 0 points. You press A. You receive 0 points. You press A. You receive 2 points. You press A. You receive 3 points. You press A. You receive 2 points. You press A. You receive 4 points.
Welcome. In this task you have a choice between two slot machines, represented by colored buttons. When you click one of the buttons, you will win or lose points. Choosing the same slot machine will not always give you the same points, but one slot machine is always better than the other. Your goal is to choose the slot machine that will give you the most points. After making your choice, you will receive feedback about the outcome. You will play 20 games, each with a different pair of slot machines. Each game will consist of 10 trials. Press A to choose the left machine and B to choose the right machine. Game 1: a new pair of slot machines appears. You press B. You receive 0 points. You press A. You receive -16 points. You press B. You receive -1 points. You press B. You receive 0 points. You press B. You receive -1 points. You press B. You receive 0 points. You press A. You receive -17 points. You press B. You receive -1 points. You press B. You receive -1 points. You press B. You receive -1 points. Game 2: a new pair of slot machines appears. You press A. You receive -5 points. You press A. You receive -4 points. You press B. You receive 4 points. You press B. You receive 4 points. You press B. You receive 4 points. You press B. You receive 5 points. You press B. You receive 6 points. You press B. You receive 5 points. You press B. You receive 2 points. You press B. You receive 6 points. Game 3: a new pair of slot machines appears. You press B. You receive 3 points. You press B. You receive 3 points. You press B. You receive 1 points. You press B. You receive 0 points. You press B. You receive 2 points. You press B. You receive 1 points. You press B. You receive 3 points. You press B. You receive 3 points. You press B. You receive 1 points. You press B. You receive 1 points. Game 4: a new pair of slot machines appears. You press B. You receive 17 points. You press B. You receive 16 points. You press B. You receive 16 points. You press B. You receive 18 points. You press B. You receive 17 points. You press B. You receive 18 points. You press B. You receive 15 points. You press B. You receive 17 points. You press B. You receive 14 points. You press B. You receive 15 points. Game 5: a new pair of slot machines appears. You press B. You receive 2 points. You press B. You receive 3 points. You press B. You receive 3 points. You press B. You receive 4 points. You press B. You receive 3 points. You press B. You receive 2 points. You press B. You receive 3 points. You press B. You receive 4 points. You press B. You receive 3 points. You press B. You receive 2 points. Game 6: a new pair of slot machines appears. You press B. You receive 5 points. You press B. You receive 5 points. You press B. You receive 6 points. You press B. You receive 8 points. You press B. You receive 6 points. You press B. You receive 6 points. You press B. You receive 7 points. You press B. You receive 4 points. You press B. You receive 6 points. You press B. You receive 6 points. Game 7: a new pair of slot machines appears. You press B. You receive 5 points. You press B. You receive 6 points. You press B. You receive 5 points. You press B. You receive 6 points. You press B. You receive 7 points. You press B. You receive 7 points. You press B. You receive 8 points. You press B. You receive 7 points. You press B. You receive 5 points. You press B. You receive 8 points. Game 8: a new pair of slot machines appears. You press B. You receive 0 points. You press B. You receive 2 points. You press B. You receive 2 points. You press B. You receive 2 points. You press B. You receive 2 points. You press A. You receive 9 points. You press A. You receive 10 points. You press A. You receive 11 points. You press A. You receive 10 points. You press A. You receive 10 points. Game 9: a new pair of slot machines appears. You press A. You receive 13 points. You press A. You receive 13 points. You press A. You receive 14 points. You press A. You receive 13 points. You press A. You receive 14 points. You press A. You receive 15 points. You press A. You receive 13 points. You press A. You receive 14 points. You press A. You receive 13 points. You press A. You receive 14 points. Game 10: a new pair of slot machines appears. You press A. You receive -1 points. You press B. You receive -7 points. You press A. You receive -1 points. You press A. You receive -2 points. You press A. You receive 1 points. You press A. You receive 1 points. You press A. You receive 1 points. You press B. You receive -4 points. You press A. You receive 0 points. You press A. You receive 1 points. Game 11: a new pair of slot machines appears. You press B. You receive -18 points. You press A. You receive 1 points. You press A. You receive -1 points. You press B. You receive -16 points. You press A. You receive 0 points. You press A. You receive -2 points. You press A. You receive 0 points. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive 0 points. Game 12: a new pair of slot machines appears. You press A. You receive 19 points. You press A. You receive 19 points. You press A. You receive 18 points. You press A. You receive 18 points. You press A. You receive 19 points. You press A. You receive 19 points. You press A. You receive 19 points. You press A. You receive 21 points. You press A. You receive 18 points. You press A. You receive 17 points. Game 13: a new pair of slot machines appears. You press A. You receive 10 points. You press A. You receive 10 points. You press A. You receive 10 points. You press A. You receive 10 points. You press A. You receive 9 points. You press A. You receive 9 points. You press A. You receive 9 points. You press A. You receive 10 points. You press A. You receive 12 points. You press A. You receive 9 points. Game 14: a new pair of slot machines appears. You press A. You receive -3 points. You press B. You receive -6 points. You press A. You receive -2 points. You press A. You receive -2 points. You press B. You receive -8 points. You press A. You receive -2 points. You press A. You receive -2 points. You press A. You receive -3 points. You press B. You receive -8 points. You press A. You receive -2 points. Game 15: a new pair of slot machines appears. You press A. You receive -6 points. You press B. You receive 3 points. You press B. You receive 1 points. You press B. You receive 4 points. You press B. You receive 1 points. You press B. You receive 4 points. You press B. You receive 4 points. You press B. You receive 3 points. You press B. You receive 3 points. You press B. You receive 4 points. Game 16: a new pair of slot machines appears. You press B. You receive -9 points. You press A. You receive -2 points. You press A. You receive -1 points. You press B. You receive -9 points. You press A. You receive -2 points. You press A. You receive -1 points. You press A. You receive 0 points. You press A. You receive -2 points. You press A. You receive -3 points. You press A. You receive 0 points. Game 17: a new pair of slot machines appears. You press A. You receive 1 points. You press B. You receive -3 points. You press A. You receive 1 points. You press B. You receive -3 points. You press A. You receive 0 points. You press A. You receive 0 points. You press B. You receive 0 points. You press B. You receive -1 points. You press A. You receive 0 points. You press A. You receive 0 points. Game 18: a new pair of slot machines appears. You press B. You receive 10 points. You press A. You receive 14 points. You press B. You receive 9 points. You press A. You receive 12 points. You press A. You receive 15 points. You press A. You receive 13 points. You press B. You receive 10 points. You press A. You receive 13 points. You press A. You receive 12 points. You press A. You receive 13 points. Game 19: a new pair of slot machines appears. You press B. You receive -1 points. You press A. You receive 1 points. You press B. You receive -1 points. You press A. You receive 1 points. You press A. You receive 0 points. You press A. You receive -1 points. You press B. You receive -1 points. You press B. You receive -1 points. You press A. You receive 3 points. You press A. You receive 1 points. Game 20: a new pair of slot machines appears. You press A. You receive -11 points. You press B. You receive -6 points. You press B. You receive -7 points. You press A. You receive -12 points. You press B. You receive -8 points. You press B. You receive -6 points. You press B. You receive -8 points. You press B. You receive -6 points. You press B. You receive -7 points. You press A. You receive -11 points.