xu_2021_novelty
A single experiment with 12 human participants (N=12; subject 10 quit) performing a sequential grid-world decision-making task. Participants navigate from a start state to a goal across 5 episodes in each of two blocks, making one discrete 4-action choice per trial; reward arrives only on reaching the goal. Block 1 (env 3) contains many novel states to probe novelty-driven exploration; block 2 (env 4) swaps the transitions of two states to test surprise-driven re-adaptation. The paper asks whether a novelty (exploration bonus) is distinct from surprise (prediction-error-driven learning), fitting the SurNoR model-based RL algorithm and comparing 13 RL models against the 3,047 trials of choice data.
Responses
3,047
253 held out
CORE effects
–
no effects simulated
CORE test NLL (avg)
1.007
rank 2 of 7 models
Best model
CORE (no phonemes)
test NLL (avg) 0.947
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE | 254.8 | 1.0072 | 253 | 0.8819 | 2,794 |
| CORE (no prediction errors) | 255.8 | 1.0110 | 253 | 0.8798 | 2,794 |
| CORE (no phonemes) best | 239.5 | 0.9466 | 253 | 0.8130 | 2,794 |
| CORE (no forgetting) | 264.7 | 1.0463 | 253 | 0.9518 | 2,794 |
| CORE (no hierarchy) | 352.9 | 1.3950 | 253 | 1.3953 | 2,794 |
| Transformer | 328.6 | 1.2990 | 253 | 1.2326 | 2,794 |
| Centaur | 264.8 | 1.0467 | 253 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | NoveltySwitchState | 12 | 3,047 | 1 | no | – |
Behavioral effects
No behavioral effects were simulated for this study.
Transcript
You are navigating a maze of 11 rooms, each shown as an image on screen. One of the rooms is the goal image; your task is to find the shortest path to it. At each room you see four grey buttons below the image. Button 1 is pressed with B, button 2 with C, button 3 with D, button 4 with A. Pressing a button moves you to the next room. You receive no feedback along the way; when you reach the goal image you earn a reward and the episode ends. You will run several episodes, each starting in a new room. Episode 1 starts in a new room. You are in room 5. You press B. You move to room 9. You get no reward. You are in room 9. You press B. You move to room 7. You get no reward. You are in room 7. You press B. You move to room 8. You get no reward. You are in room 8. You press B. You move to room 7. You get no reward. You are in room 7. You press B. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press B. You move to room 9. You get no reward. You are in room 9. You press B. You move to room 7. You get no reward. You are in room 7. You press B. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press B. You move to room 8. You get no reward. You are in room 8. You press B. You move to room 7. You get no reward. You are in room 7. You press C. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press B. You move to room 7. You get no reward. You are in room 7. You press B. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press C. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press D. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press B. You move to room 4. You get no reward. You are in room 4. You press C. You move to room 9. You get no reward. You are in room 9. You press C. You move to room 8. You get no reward. You are in room 8. You press D. You move to room 9. You get no reward. You are in room 9. You press A. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press C. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press A. You move to room 1. You get no reward. You are in room 1. You press D. You move to room 7. You get no reward. You are in room 7. You press A. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press A. You move to room 1. You get no reward. You are in room 1. You press D. You move to room 7. You get no reward. You are in room 7. You press A. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press A. You move to room 8. You get no reward. You are in room 8. You press D. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press D. You move to room 3. You get no reward. You are in room 3. You press C. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press A. You move to room 7. You get no reward. You are in room 7. You press A. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You reach the goal image and earn a reward. Episode 2 starts in a new room. You are in room 8. You press D. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You reach the goal image and earn a reward. Episode 3 starts in a new room. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You reach the goal image and earn a reward. Episode 4 starts in a new room. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press B. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You reach the goal image and earn a reward. Episode 5 starts in a new room. You are in room 7. You press C. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press A. You move to room 1. You get no reward. You are in room 1. You press D. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 2. You get no reward. You are in room 2. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You reach the goal image and earn a reward. Episode 6 starts in a new room. You are in room 5. You press B. You move to room 9. You get no reward. You are in room 9. You press B. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You move to room 7. You get no reward. You are in room 7. You press A. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press C. You move to room 8. You get no reward. You are in room 8. You press D. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press D. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press B. You move to room 4. You get no reward. You are in room 4. You press C. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press A. You move to room 5. You get no reward. You are in room 5. You press D. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press A. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press C. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press D. You move to room 3. You get no reward. You are in room 3. You press A. You move to room 8. You get no reward. You are in room 8. You press B. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press A. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press B. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press A. You move to room 1. You get no reward. You are in room 1. You press D. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press B. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press A. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press B. You move to room 4. You get no reward. You are in room 4. You press C. You move to room 9. You get no reward. You are in room 9. You press C. You move to room 8. You get no reward. You are in room 8. You press B. You move to room 7. You get no reward. You are in room 7. You press A. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press B. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press B. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press C. You move to room 8. You get no reward. You are in room 8. You press B. You move to room 7. You get no reward. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press D. You move to room 6. You get no reward. You are in room 6. You press C. You move to room 8. You get no reward. You are in room 8. You press D. You move to room 9. You get no reward. You are in room 9. You press B. You move to room 7. You get no reward. You are in room 7. You press B. You move to room 8. You get no reward. You are in room 8. You press A. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press B. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press C. You move to room 0. You get no reward. You are in room 0. You press A. You move to room 8. You get no reward. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press A. You move to room 5. You get no reward. You are in room 5. You press B. You move to room 9. You get no reward. You are in room 9. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 2. You get no reward. You are in room 2. You press B. You reach the goal image and earn a reward. Episode 7 starts in a new room. You are in room 8. You press C. You move to room 0. You get no reward. You are in room 0. You press C. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 2. You get no reward. You are in room 2. You press B. You reach the goal image and earn a reward. Episode 8 starts in a new room. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 2. You get no reward. You are in room 2. You press B. You reach the goal image and earn a reward. Episode 9 starts in a new room. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 2. You get no reward. You are in room 2. You press B. You reach the goal image and earn a reward. Episode 10 starts in a new room. You are in room 7. You press D. You move to room 0. You get no reward. You are in room 0. You press D. You move to room 1. You get no reward. You are in room 1. You press C. You move to room 6. You get no reward. You are in room 6. You press A. You move to room 3. You get no reward. You are in room 3. You press B. You move to room 4. You get no reward. You are in room 4. You press D. You move to room 5. You get no reward. You are in room 5. You press C. You move to room 2. You get no reward. You are in room 2. You press B. You reach the goal image and earn a reward.