bahrami_2020_arm

bandit psych-101

One online restless multi-armed (4-arm) bandit task following the Daw et al. (2006) paradigm, run with N = 965 participants who each completed 150 trials. On each trial a participant chooses one of four arms and receives reward feedback; the true reward of each arm drifts over time under one of three predefined drift schedules (between-subject, mapped to `version`). Responses are discrete choices (arms 1-4, 0-indexed in `response`) plus reaction times in milliseconds. The research question concerns how humans balance exploration versus exploitation when the value of options changes continuously. Primary behavioral findings reproduced on this data: best-arm selection well above the 25% chance baseline, reward-following choice accuracy improving over trials, and slower reaction times on exploratory (non-best) compared with best choices.

Responses
139,816
13,867 held out
CORE effects
2 / 2
CORE test NLL (avg)
0.681
rank 1 of 7 models
Best model
CORE
test NLL (avg) 0.681

Likelihood of human responses

modelPNLLtest NLL (avg)test responses train NLL (avg)train responses
CORE best9,443.90.681013,8670.6442125,949
CORE (no prediction errors)9,522.70.686713,8670.6501125,949
CORE (no phonemes)9,497.80.684913,8670.6435125,949
CORE (no forgetting)9,688.00.698613,8670.6673125,949
CORE (no hierarchy)11,184.80.806613,8670.8071125,949
Transformer10,330.90.745013,8670.7221125,949
Centaur9,521.20.686613,867––

Experiments

exptask classparticipantsresponsesconditionssimulatedeffects
exp0FourArmBandit965139,8163yesbest_choice_above_chance, choice_accuracy_improves

Behavioral effects

effectexperimentpaperCORECORE (no prediction errors)CORE (no phonemes)CORE (no forgetting)CORE (no hierarchy)
best_choice_above_chanceexp0+0.250✓ yes
+0.615
✓ yes
+0.633
✓ yes
+0.612
✓ yes
+0.632
✓ yes
+0.447
choice_accuracy_improvesexp0+0.000✓ yes
+0.046
✓ yes
+0.049
✓ yes
+0.045
✓ yes
+0.058
✗ no
+0.002

Transcript

Welcome to the 4-Arm Bandit experiment.
On each trial you will see four slot machines, labeled A, B, C, and D. Choose exactly one machine by pressing the corresponding key. After your choice you will see the reward you obtained from that machine. You will not see the rewards the other machines would have paid. Each machine's payoff drifts slowly over time, so keep trying different machines to find out which is paying well, while mostly choosing the ones that have been good recently. You have 4 seconds to respond; if you do not respond in time the trial is skipped and you earn no reward. The task lasts 150 trials. Press one of A, B, C, or D to choose a machine.

Trial 1: You choose machine B and receive 87 points.
Trial 2: You choose machine D and receive 18 points.
Trial 3: You choose machine C and receive 53 points.
Trial 4: You choose machine B and receive 81 points.
Trial 5: You choose machine D and receive 28 points.
Trial 6: You choose machine A and receive 75 points.
Trial 7: You choose machine B and receive 78 points.
Trial 8: You choose machine C and receive 58 points.
Trial 9: You choose machine D and receive 30 points.
Trial 10: You choose machine A and receive 70 points.
Trial 11: You choose machine B and receive 91 points.
Trial 12: You choose machine B and receive 90 points.
Trial 13: You choose machine B and receive 79 points.
Trial 14: You choose machine A and receive 66 points.
Trial 15: You choose machine D and receive 35 points.
Trial 16: You choose machine B and receive 75 points.
Trial 17: You choose machine C and receive 54 points.
Trial 18: You choose machine B and receive 82 points.
Trial 19: You choose machine B and receive 87 points.
Trial 20: You choose machine A and receive 62 points.
Trial 21: You choose machine B and receive 87 points.
Trial 22: You choose machine B and receive 87 points.
Trial 23: You choose machine B and receive 79 points.
Trial 24: You choose machine A and receive 57 points.
Trial 25: You choose machine C and receive 47 points.
Trial 26: You choose machine B and receive 74 points.
Trial 27: You choose machine B and receive 82 points.
Trial 28: You choose machine D and receive 41 points.
Trial 29: You choose machine B and receive 73 points.
Trial 30: You choose machine B and receive 71 points.
Trial 31: You choose machine A and receive 65 points.
Trial 32: You choose machine B and receive 62 points.
Trial 33: You choose machine A and receive 63 points.
Trial 34: You choose machine B and receive 58 points.
Trial 35: You choose machine C and receive 54 points.
Trial 36: You choose machine D and receive 30 points.
Trial 37: You choose machine B and receive 64 points.
Trial 38: You choose machine A and receive 48 points.
Trial 39: You choose machine B and receive 64 points.
Trial 40: You choose machine B and receive 60 points.
Trial 41: You choose machine C and receive 47 points.
Trial 42: You choose machine A and receive 55 points.
Trial 43: You choose machine D and receive 17 points.
Trial 44: You choose machine B and receive 56 points.
Trial 45: You choose machine A and receive 52 points.
Trial 46: You choose machine B and receive 61 points.
Trial 47: You choose machine C and receive 49 points.
Trial 48: You choose machine B and receive 51 points.
Trial 49: You choose machine A and receive 59 points.
Trial 50: You choose machine D and receive 14 points.
Trial 51: You choose machine B and receive 52 points.
Trial 52: You choose machine A and receive 67 points.
Trial 53: You choose machine A and receive 71 points.
Trial 54: You choose machine A and receive 71 points.
Trial 55: You choose machine A and receive 69 points.
Trial 56: You choose machine B and receive 46 points.
Trial 57: You choose machine C and receive 47 points.
Trial 58: You choose machine B and receive 60 points.
Trial 59: You choose machine A and receive 63 points.
Trial 60: You choose machine A and receive 58 points.
Trial 61: You choose machine B and receive 53 points.
Trial 62: You choose machine A and receive 53 points.
Trial 63: You choose machine B and receive 61 points.
Trial 64: You choose machine D and receive 26 points.
Trial 65: You choose machine C and receive 52 points.
Trial 66: You choose machine B and receive 54 points.
Trial 67: You choose machine B and receive 56 points.
Trial 68: You choose machine A and receive 63 points.
Trial 69: You choose machine B and receive 71 points.
Trial 70: You choose machine A and receive 61 points.
Trial 71: You choose machine B and receive 56 points.
Trial 72: You choose machine B and receive 51 points.
Trial 73: You choose machine A and receive 49 points.
Trial 74: You choose machine B and receive 61 points.
Trial 75: You choose machine C and receive 51 points.
Trial 76: You choose machine B and receive 68 points.
Trial 77: You choose machine A and receive 42 points.
Trial 78: You choose machine B and receive 59 points.
Trial 79: You choose machine C and receive 48 points.
Trial 80: You choose machine B and receive 56 points.
Trial 81: You choose machine A and receive 43 points.
Trial 82: You choose machine B and receive 59 points.
Trial 83: You did not respond in time, so the trial was skipped and you received no reward.
Trial 84: You choose machine B and receive 55 points.
Trial 85: You choose machine A and receive 38 points.
Trial 86: You choose machine B and receive 61 points.
Trial 87: You choose machine C and receive 51 points.
Trial 88: You choose machine B and receive 58 points.
Trial 89: You choose machine A and receive 45 points.
Trial 90: You choose machine B and receive 56 points.
Trial 91: You choose machine A and receive 39 points.
Trial 92: You choose machine B and receive 53 points.
Trial 93: You choose machine C and receive 50 points.
Trial 94: You choose machine B and receive 57 points.
Trial 95: You choose machine B and receive 54 points.
Trial 96: You choose machine B and receive 49 points.
Trial 97: You choose machine A and receive 33 points.
Trial 98: You choose machine C and receive 71 points.
Trial 99: You choose machine C and receive 78 points.
Trial 100: You choose machine C and receive 79 points.
Trial 101: You choose machine C and receive 79 points.
Trial 102: You choose machine C and receive 85 points.
Trial 103: You choose machine C and receive 80 points.
Trial 104: You choose machine C and receive 82 points.
Trial 105: You choose machine C and receive 83 points.
Trial 106: You choose machine C and receive 85 points.
Trial 107: You choose machine C and receive 71 points.
Trial 108: You choose machine C and receive 74 points.
Trial 109: You choose machine C and receive 83 points.
Trial 110: You choose machine C and receive 69 points.
Trial 111: You choose machine B and receive 51 points.
Trial 112: You choose machine C and receive 64 points.
Trial 113: You choose machine A and receive 33 points.
Trial 114: You choose machine C and receive 74 points.
Trial 115: You choose machine C and receive 80 points.
Trial 116: You choose machine C and receive 70 points.
Trial 117: You choose machine C and receive 73 points.
Trial 118: You choose machine C and receive 66 points.
Trial 119: You choose machine C and receive 71 points.
Trial 120: You choose machine C and receive 70 points.
Trial 121: You choose machine C and receive 71 points.
Trial 122: You choose machine C and receive 65 points.
Trial 123: You choose machine C and receive 71 points.
Trial 124: You choose machine C and receive 73 points.
Trial 125: You choose machine C and receive 65 points.
Trial 126: You choose machine C and receive 73 points.
Trial 127: You choose machine C and receive 66 points.
Trial 128: You choose machine C and receive 59 points.
Trial 129: You choose machine B and receive 37 points.
Trial 130: You choose machine D and receive 47 points.
Trial 131: You choose machine A and receive 32 points.
Trial 132: You choose machine C and receive 63 points.
Trial 133: You choose machine C and receive 74 points.
Trial 134: You choose machine C and receive 70 points.
Trial 135: You choose machine C and receive 77 points.
Trial 136: You choose machine C and receive 70 points.
Trial 137: You choose machine C and receive 78 points.
Trial 138: You choose machine C and receive 74 points.
Trial 139: You choose machine C and receive 70 points.
Trial 140: You choose machine C and receive 68 points.
Trial 141: You choose machine C and receive 67 points.
Trial 142: You choose machine C and receive 63 points.
Trial 143: You choose machine C and receive 65 points.
Trial 144: You choose machine C and receive 60 points.
Trial 145: You choose machine C and receive 60 points.
Trial 146: You choose machine C and receive 62 points.
Trial 147: You choose machine C and receive 61 points.
Trial 148: You choose machine C and receive 70 points.
Trial 149: You choose machine C and receive 60 points.
Trial 150: You choose machine C and receive 56 points.