anvari_2024_testing
A registered report testing whether five explore–exploit behavioural tasks converge on a general tendency to explore, generalize across task domains, and are stable over about one month. 678 UK Prolific participants completed the whole study at time1 (April 2023) and 254 of them repeated it at time2 about one month later. Each CSV pools both waves (`session`), and one `participant_id` identifies the same person in both waves and in all five experiments, so the paper's test–retest and cross-task analyses can be run from these files. Participants who failed a task's comprehension check six times skipped that task and have no rows in that experiment. exp0 (N = 672): five-armed bandit, per wave 1 practice block of 20 trials and 4 incentivized blocks of 40 trials. exp1 (N = 671): sampling paradigm, per wave 1 practice and 5 incentivized blocks of free sampling (2 to 100 samples) closed by the choice of the button to be paid from. exp2 (N = 674): observe-or-bet, per wave 1 practice block of 25 trials and 2 incentivized blocks of 50 trials. exp3 (N = 678): optional stopping with recall, per wave 1 practice and 8 incentivized blocks of up to 20 box openings. exp4 (N = 675): alien game (search on an NK payoff landscape), per wave 1 practice block of 3 trials and 3 incentivized blocks of 10 trials, each opened by a displayed starting combination. The paper's headline analyses are CFA/latent-factor SEM and cross-task correlations; two within-task convergent-validity effects reproduce from the trial-level data (bandit switch rate with exploit-complement, paper r = 0.47; alien Hamming distance with active search, paper r = 0.52).
Responses
264,435
19,265 held out
CORE effects
–
no effects simulated
CORE test NLL (avg)
0.635
rank 1 of 7 models
Best model
CORE
test NLL (avg) 0.635
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE best | 12,226.5 | 0.6346 | 19,265 | 0.5791 | 245,170 |
| CORE (no prediction errors) | 12,628.4 | 0.6555 | 19,265 | 0.5950 | 245,170 |
| CORE (no phonemes) | 12,330.4 | 0.6400 | 19,265 | 0.5724 | 245,170 |
| CORE (no forgetting) | 13,389.2 | 0.6950 | 19,265 | 0.6330 | 245,170 |
| CORE (no hierarchy) | 17,939.2 | 0.9312 | 19,265 | 0.8651 | 245,170 |
| Transformer | 15,785.8 | 0.8194 | 19,265 | 0.7434 | 245,170 |
| Centaur | 12,456.5 | 0.6466 | 19,265 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | FiveArmedBandit | 672 | 166,497 | 1 | no | – |
| exp1 | SamplingParadigm | 671 | 97,938 | 1 | no | – |
Behavioral effects
No behavioral effects were simulated for this study.
Transcript
You play a game with five buttons. On each trial you click one button and it reveals a number of points; your goal is to earn as many points as possible. The screen shows the trial number, how many times each button has been selected, the average points per click for each button, and your total points so far. For every 100 points you earn a bonus of 1p. On each trial press the letter for the button you choose: Button 1 = D, Button 2 = A, Button 3 = C, Button 4 = B, Button 5 = E. practice block begins with five new buttons. practice trial: you press C; it pays 58 points. practice trial: you press C; it pays 63 points. practice trial: you press D; it pays 39 points. practice trial: you press E; it pays 62 points. practice trial: you press B; it pays 58 points. practice trial: you press A; it pays 53 points. practice trial: you press C; it pays 67 points. practice trial: you press B; it pays 48 points. practice trial: you press E; it pays 55 points. practice trial: you press C; it pays 66 points. practice trial: you press B; it pays 47 points. practice trial: you press E; it pays 63 points. practice trial: you press A; it pays 54 points. practice trial: you press C; it pays 58 points. practice trial: you press E; it pays 62 points. practice trial: you press A; it pays 50 points. practice trial: you press C; it pays 58 points. practice trial: you press E; it pays 62 points. practice trial: you press C; it pays 54 points. practice trial: you press E; it pays 50 points. incentivized block begins with five new buttons. incentivized trial: you press D; it pays 55 points. incentivized trial: you press E; it pays 61 points. incentivized trial: you press A; it pays 48 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press C; it pays 54 points. incentivized trial: you press E; it pays 56 points. incentivized trial: you press E; it pays 59 points. incentivized trial: you press E; it pays 62 points. incentivized trial: you press B; it pays 55 points. incentivized trial: you press B; it pays 63 points. incentivized trial: you press B; it pays 54 points. incentivized trial: you press E; it pays 54 points. incentivized trial: you press E; it pays 65 points. incentivized trial: you press E; it pays 61 points. incentivized trial: you press E; it pays 56 points. incentivized trial: you press D; it pays 60 points. incentivized trial: you press D; it pays 49 points. incentivized trial: you press D; it pays 57 points. incentivized trial: you press D; it pays 42 points. incentivized trial: you press D; it pays 58 points. incentivized trial: you press D; it pays 50 points. incentivized trial: you press E; it pays 63 points. incentivized trial: you press E; it pays 54 points. incentivized trial: you press E; it pays 62 points. incentivized trial: you press B; it pays 59 points. incentivized trial: you press E; it pays 58 points. incentivized trial: you press B; it pays 57 points. incentivized trial: you press B; it pays 51 points. incentivized trial: you press B; it pays 56 points. incentivized trial: you press E; it pays 58 points. incentivized trial: you press E; it pays 59 points. incentivized trial: you press E; it pays 50 points. incentivized trial: you press E; it pays 55 points. incentivized trial: you press C; it pays 47 points. incentivized trial: you press C; it pays 54 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 56 points. incentivized block begins with five new buttons. incentivized trial: you press E; it pays 68 points. incentivized trial: you press E; it pays 55 points. incentivized trial: you press E; it pays 50 points. incentivized trial: you press E; it pays 60 points. incentivized trial: you press E; it pays 54 points. incentivized trial: you press B; it pays 50 points. incentivized trial: you press B; it pays 62 points. incentivized trial: you press B; it pays 56 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press D; it pays 50 points. incentivized trial: you press B; it pays 50 points. incentivized trial: you press B; it pays 54 points. incentivized trial: you press B; it pays 54 points. incentivized trial: you press B; it pays 57 points. incentivized trial: you press B; it pays 64 points. incentivized trial: you press B; it pays 41 points. incentivized trial: you press C; it pays 47 points. incentivized trial: you press C; it pays 51 points. incentivized trial: you press C; it pays 52 points. incentivized trial: you press E; it pays 67 points. incentivized trial: you press E; it pays 56 points. incentivized trial: you press E; it pays 58 points. incentivized trial: you press E; it pays 64 points. incentivized trial: you press E; it pays 66 points. incentivized trial: you press E; it pays 59 points. incentivized trial: you press E; it pays 65 points. incentivized trial: you press E; it pays 54 points. incentivized trial: you press E; it pays 66 points. incentivized trial: you press C; it pays 44 points. incentivized trial: you press C; it pays 47 points. incentivized trial: you press A; it pays 44 points. incentivized trial: you press E; it pays 57 points. incentivized trial: you press E; it pays 58 points. incentivized trial: you press E; it pays 61 points. incentivized trial: you press E; it pays 53 points. incentivized trial: you press E; it pays 60 points. incentivized block begins with five new buttons. incentivized trial: you press E; it pays 48 points. incentivized trial: you press B; it pays 44 points. incentivized trial: you press C; it pays 64 points. incentivized trial: you press C; it pays 51 points. incentivized trial: you press C; it pays 50 points. incentivized trial: you press C; it pays 64 points. incentivized trial: you press D; it pays 47 points. incentivized trial: you press C; it pays 60 points. incentivized trial: you press A; it pays 49 points. incentivized trial: you press C; it pays 50 points. incentivized trial: you press C; it pays 65 points. incentivized trial: you press C; it pays 65 points. incentivized trial: you press C; it pays 65 points. incentivized trial: you press C; it pays 54 points. incentivized trial: you press C; it pays 63 points. incentivized trial: you press C; it pays 62 points. incentivized trial: you press C; it pays 64 points. incentivized trial: you press C; it pays 61 points. incentivized trial: you press C; it pays 62 points. incentivized trial: you press C; it pays 50 points. incentivized trial: you press C; it pays 67 points. incentivized trial: you press C; it pays 70 points. incentivized trial: you press C; it pays 55 points. incentivized trial: you press C; it pays 58 points. incentivized trial: you press C; it pays 57 points. incentivized trial: you press C; it pays 52 points. incentivized trial: you press C; it pays 55 points. incentivized trial: you press C; it pays 58 points. incentivized trial: you press C; it pays 54 points. incentivized trial: you press C; it pays 54 points. incentivized trial: you press C; it pays 54 points. incentivized trial: you press C; it pays 61 points. incentivized trial: you press C; it pays 60 points. incentivized trial: you press C; it pays 57 points. incentivized trial: you press C; it pays 59 points. incentivized trial: you press C; it pays 60 points. incentivized trial: you press C; it pays 51 points. incentivized trial: you press C; it pays 50 points. incentivized trial: you press C; it pays 64 points. incentivized trial: you press C; it pays 58 points. incentivized block begins with five new buttons. incentivized trial: you press E; it pays 60 points. incentivized trial: you press D; it pays 49 points. incentivized trial: you press E; it pays 47 points. incentivized trial: you press B; it pays 69 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 63 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 53 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 57 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 50 points. incentivized trial: you press B; it pays 54 points. incentivized trial: you press B; it pays 62 points. incentivized trial: you press B; it pays 53 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 59 points. incentivized trial: you press B; it pays 56 points. incentivized trial: you press B; it pays 58 points. incentivized trial: you press B; it pays 74 points. incentivized trial: you press B; it pays 56 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 65 points. incentivized trial: you press B; it pays 55 points. incentivized trial: you press B; it pays 56 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 59 points. incentivized trial: you press B; it pays 60 points. incentivized trial: you press B; it pays 57 points. incentivized trial: you press B; it pays 59 points. incentivized trial: you press B; it pays 53 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press B; it pays 46 points. incentivized trial: you press B; it pays 61 points. incentivized trial: you press B; it pays 56 points. incentivized trial: you press B; it pays 64 points. incentivized trial: you press E; it pays 62 points.
You see two buttons, each with an unknown chance of paying a fixed number of points. You can sample the buttons as many times as you like (up to 100 per block) to learn what they pay; each sample shows what that button would have paid. You must sample each button at least once. When you stop sampling you choose which button to be paid out from, and it pays once more. For each point you earn a bonus of 5p. Press B to sample Button 1, C to sample Button 2, A to stop and be paid from Button 1, or D to stop and be paid from Button 2. practice block begins. You press B and sample Button 1. You press B and sample Button 1. You press C and sample Button 2. You press A and choose to be paid from Button 1; it pays 1 points. incentivized block begins. You press B and sample Button 1. You press C and sample Button 2. You press A and choose to be paid from Button 1; it pays 4 points. incentivized block begins. You press B and sample Button 1. You press C and sample Button 2. You press B and sample Button 1. You press C and sample Button 2. You press A and choose to be paid from Button 1; it pays 10 points. incentivized block begins. You press B and sample Button 1. You press C and sample Button 2. You press A and choose to be paid from Button 1; it pays 3 points. incentivized block begins. You press B and sample Button 1. You press C and sample Button 2. You press A and choose to be paid from Button 1; it pays 3 points. incentivized block begins. You press B and sample Button 1. You press C and sample Button 2. You press B and sample Button 1. You press C and sample Button 2. You press D and choose to be paid from Button 2; it pays 0 points.