kool_2016_when

two-step-task psych-101

Two two-step sequential decision experiments test when model-based control pays off. In each trial participants make a stage-1 choice between two stimuli that transition (common/rare, 70/30) to a second stage, then choose among two options yielding points with drifting reward probabilities (Gaussian random walk). Experiment 1 (exp0, N=206 in data; paper reports 207) uses the original Daw two-step structure; Experiment 2 (exp1, N=200 in data; 199 completers reported) uses a novel task with four stage-1 choices and integer reward magnitudes from -4 to +5 points, designed so model-based planning yields an accuracy-demand trade-off; participants spontaneously increased model-based control there. Responses are stage-1 and stage-2 stimulus choices (plus RTs); the paper shows by simulation that standard two-step tasks lack this trade-off and fits hybrid model-based/model-free reinforcement-learning models (weighted Q-values, softmax) to quantify each participant's reliance on model-based control.

Responses
89,054
8,652 held out
CORE effects
2 / 3
CORE test NLL (avg)
0.556
rank 2 of 7 models
Best model
CORE (no phonemes)
test NLL (avg) 0.555

Likelihood of human responses

modelPNLLtest NLL (avg)test responses train NLL (avg)train responses
CORE4,814.50.55658,6520.547480,402
CORE (no prediction errors)4,834.10.55878,6520.546680,402
CORE (no phonemes) best4,800.00.55488,6520.537680,402
CORE (no forgetting)4,900.10.56648,6520.557380,402
CORE (no hierarchy)5,354.20.61888,6520.619880,402
Transformer5,098.70.58938,6520.582180,402
Centaur4,883.10.56448,652––

Experiments

exptask classparticipantsresponsesconditionssimulatedeffects
exp0TwoStepExp020660,0781yesdaw_stay_reward_maineffect, daw_stay_reward_common_interaction
exp1NovelTwoStep20028,9761yesnovel_stay_same_vs_diff_after_reward

Behavioral effects

effectexperimentpaperCORECORE (no prediction errors)CORE (no phonemes)CORE (no forgetting)CORE (no hierarchy)
daw_stay_reward_maineffectexp0+0.549✓ yes
+0.087
✓ yes
+0.076
✓ yes
+0.072
✓ yes
+0.072
✓ yes
+0.024
daw_stay_reward_common_interactionexp0+0.383✗ no
-0.019
✗ no
-0.008
✗ no
+0.030
✗ no
-0.011
✗ no
+0.007
novel_stay_same_vs_diff_after_rewardexp1+0.711✓ yes
+0.106
✓ yes
+0.106
✓ yes
+0.159
✓ yes
+0.116
✗ no
-0.004

Transcript

You are an astronaut flying a spaceship from Earth to collect space treasure on two different planets. Each planet has two aliens on it, and each alien has its own space treasure mine. If an alien has a good mine, it is likely to share a piece of space treasure with you; if its mine is bad, it usually will not. The quality of each mine drifts slowly over time, so you must keep paying attention. On each trial, two spaceships appear. Choose one by pressing A or B. Each spaceship mostly flies to one of the planets and sometimes to the other; this does not change during the game. When you arrive at a planet, two aliens appear. Choose one by pressing A or B to ask it for treasure. You then learn whether you found space treasure (worth 1 point) or not (0 points). First you play practice trials with no time limit. Then the real game starts, with 2 seconds to make each choice. Press A or B to choose the option you want.
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the yellow planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the green planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the yellow planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the yellow planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the green planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the yellow planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the green planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the green planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the yellow planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the green planet. On the planet you see two aliens. You press B. You find no space treasure.
The practice trials are over. In the real game you will find new planets, new aliens and new mines, but the rules and the spaceships are the same.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. You fail to respond in time.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find no space treasure.
You see two spaceships. You press A. You travel to the purple planet. On the planet you see two aliens. You press A. You find no space treasure.
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press A. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press B. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the purple planet. On the planet you see two aliens. You press A. You find a piece of space treasure!
You see two spaceships. You press B. You travel to the red planet. On the planet you see two aliens. You press A. You find a piece of space treasure!