akata_2025_playing
This is a single human study (N=195 Prolific participants) in which each participant was randomly assigned to play a finitely repeated Prisoner's Dilemma (PD) and a Battle of the Sexes (BoS) game against either a base or chain-of-thought ("Prompted") GPT-4, over 20 rounds (10 PD + 10 BoS, counterbalanced). On each round participants made a binary option choice (0/1) and received a per-round payoff (score); after each game they gave an opponent-belief judgment (whether they thought the other player was human or an LLM), stored as a response row of its own. The source also ships a per-round coordination flag the authors derived from the outcome. The research question is how LLMs behave and cooperate in repeated social interactions relative to humans, prominently showing that chain-of-thought-prompted GPT-4 elicits better scores and more successful coordination from human partners than base GPT-4.
Responses
3,900
380 held out
CORE effects
–
no effects simulated
CORE test NLL (avg)
0.536
rank 3 of 7 models
Best model
CORE (no prediction errors)
test NLL (avg) 0.533
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE | 203.8 | 0.5362 | 380 | 0.5038 | 3,520 |
| CORE (no prediction errors) best | 202.5 | 0.5330 | 380 | 0.5068 | 3,520 |
| CORE (no phonemes) | 203.6 | 0.5359 | 380 | 0.4962 | 3,520 |
| CORE (no forgetting) | 207.4 | 0.5459 | 380 | 0.5157 | 3,520 |
| CORE (no hierarchy) | 260.5 | 0.6855 | 380 | 0.6514 | 3,520 |
| Transformer | 211.4 | 0.5564 | 380 | 0.5327 | 3,520 |
| Centaur | 208.8 | 0.5495 | 380 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | RepeatedGames | 195 | 3,900 | 2 | no | – |
Behavioral effects
No behavioral effects were simulated for this study.
Transcript
You are going to play two games against an artificial agent or another fellow human. Your opponent is randomly assigned and changes at the start of each game. Your goal is to gain as many points as possible in both games. Each game is a 2x2 game played with 2 players: you and the other player each choose between two options, Option F and Option J. Each game consists of 10 rounds. On each round, choose one of the two options by typing F or J (type F to choose Option F, J to choose Option J). You have 20 seconds to choose. After each game you will be asked whether you think your opponent was a human or an artificial agent; answer by typing Human or LLM. The game is a Battle of the Sexes. The rules are: If you choose J and the other player chooses J, you win 7 points and the other wins 10 points. If you choose J and the other player chooses F, you win 0 points and the other wins 0 points. If you choose F and the other player chooses J, you win 0 points and the other wins 0 points. If you choose F and the other player chooses F, you win 10 points and the other wins 7 points. You will play 10 rounds. Round 1: you press F. You won 0 points. Round 2: you press F. You won 10 points. Round 3: you press J. You won 0 points. Round 4: you press F. You won 0 points. Round 5: you press F. You won 0 points. Round 6: you press F. You won 0 points. Round 7: you press J. You won 7 points. Round 8: you press F. You won 10 points. Round 9: you press J. You won 0 points. Round 10: you press J. You won 0 points. Who do you think was your opponent in the previous game? You answer LLM. The game is a Prisoner's Dilemma. The rules are: If you choose J and the other player chooses J, you win 8 points and the other wins 8 points. If you choose J and the other player chooses F, you win 0 points and the other wins 10 points. If you choose F and the other player chooses J, you win 10 points and the other wins 0 points. If you choose F and the other player chooses F, you win 5 points and the other wins 5 points. You will play 10 rounds. Round 1: you press J. You won 8 points. Round 2: you press J. You won 8 points. Round 3: you press J. You won 8 points. Round 4: you press J. You won 8 points. Round 5: you press J. You won 8 points. Round 6: you press J. You won 8 points. Round 7: you press J. You won 8 points. Round 8: you press J. You won 8 points. Round 9: you press J. You won 8 points. Round 10: you press J. You won 8 points. Who do you think was your opponent in the previous game? You answer Human.
You are going to play two games against an artificial agent or another fellow human. Your opponent is randomly assigned and changes at the start of each game. Your goal is to gain as many points as possible in both games. Each game is a 2x2 game played with 2 players: you and the other player each choose between two options, Option F and Option J. Each game consists of 10 rounds. On each round, choose one of the two options by typing F or J (type F to choose Option F, J to choose Option J). You have 20 seconds to choose. After each game you will be asked whether you think your opponent was a human or an artificial agent; answer by typing Human or LLM. The game is a Prisoner's Dilemma. The rules are: If you choose J and the other player chooses J, you win 8 points and the other wins 8 points. If you choose J and the other player chooses F, you win 0 points and the other wins 10 points. If you choose F and the other player chooses J, you win 10 points and the other wins 0 points. If you choose F and the other player chooses F, you win 5 points and the other wins 5 points. You will play 10 rounds. Round 1: you press J. You won 8 points. Round 2: you press F. You won 10 points. Round 3: you press F. You won 5 points. Round 4: you press J. You won 8 points. Round 5: you press F. You won 10 points. Round 6: you press J. You won 0 points. Round 7: you press F. You won 5 points. Round 8: you press F. You won 10 points. Round 9: you press F. You won 5 points. Round 10: you press J. You won 0 points. Who do you think was your opponent in the previous game? You answer Human. The game is a Battle of the Sexes. The rules are: If you choose J and the other player chooses J, you win 7 points and the other wins 10 points. If you choose J and the other player chooses F, you win 0 points and the other wins 0 points. If you choose F and the other player chooses J, you win 0 points and the other wins 0 points. If you choose F and the other player chooses F, you win 10 points and the other wins 7 points. You will play 10 rounds. Round 1: you press F. You won 0 points. Round 2: you press F. You won 0 points. Round 3: you press F. You won 0 points. Round 4: you press J. You won 7 points. Round 5: you press F. You won 10 points. Round 6: you press F. You won 10 points. Round 7: you press F. You won 10 points. Round 8: you press J. You won 0 points. Round 9: you press F. You won 10 points. Round 10: you press F. You won 10 points. Who do you think was your opponent in the previous game? You answer Human.