dentella_2023_systematic
This study evaluates whether large language models (ChatGPT-3.5, GPT-4, GLM-130B) exhibit human-like grammatical competence, using grammar-adherence and meaning-adherence tests; the human comparison experiment (the only human behavioral data, uploaded here, n=80 native English Prolific speakers) is a binary grammaticality-judgment task. Each participant was assigned one of 8 counterbalanced lists, i.e. one linguistic phenomenon: its 5 grammatical and 5 ungrammatical sentences were each presented 10 times to measure response stability, plus 2 attention-check sentences presented 5 times each, giving 110 trials per participant and 8,800 trial-level rows. The research question is whether LMs match human accuracy, absence of yes-response bias, and response stability in grammaticality judgments.
Responses
8,800
880 held out
CORE effects
2 / 3
CORE test NLL (avg)
0.313
rank 3 of 7 models
Best model
CORE (no forgetting)
test NLL (avg) 0.300
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE | 275.5 | 0.3131 | 880 | 0.2250 | 7,920 |
| CORE (no prediction errors) | 287.4 | 0.3266 | 880 | 0.2316 | 7,920 |
| CORE (no phonemes) | 312.5 | 0.3552 | 880 | 0.1749 | 7,920 |
| CORE (no forgetting) best | 264.2 | 0.3003 | 880 | 0.2321 | 7,920 |
| CORE (no hierarchy) | 454.6 | 0.5166 | 880 | 0.4793 | 7,920 |
| Transformer | 305.4 | 0.3470 | 880 | 0.2498 | 7,920 |
| Centaur | 265.5 | 0.3017 | 880 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | GrammaticalityJudgment | 80 | 8,800 | 1 | yes | overall_accuracy_above_chance, accuracy_above_chance_ungrammatical, human_response_stability |
Behavioral effects
| effect | experiment | paper | CORE | CORE (no prediction errors) | CORE (no phonemes) | CORE (no forgetting) | CORE (no hierarchy) |
|---|---|---|---|---|---|---|---|
overall_accuracy_above_chance | exp0 | +0.500 | ✓ yes +0.535 | ✓ yes +0.553 | ✓ yes +0.594 | ✓ yes +0.508 | ✗ no +0.500 |
human mean accuracy across all test sentences is significantly above chance (0.5). One-sample t-test on per- participant accuracy; expected direction: mean > 0.5. | |||||||
accuracy_above_chance_ungrammatical | exp0 | +0.500 | ✗ no +0.244 | ✓ yes +0.585 | ✓ yes +0.645 | ✓ yes +0.649 | ✓ yes +0.644 |
human accuracy is above chance even for ungrammatical sentences (unlike the tested LMs, which perform at chance there). One-sample t-test on per-participant ungrammatical accuracy; expected direction: mean > 0.5. | |||||||
human_response_stability | exp0 | +0.500 | ✓ yes +0.129 | ✓ yes +0.205 | ✓ yes +0.202 | ✓ yes +0.097 | ✓ yes +0.113 |
humans give stable responses — the rate of oscillations (a change from the previous judgment of the same sentence) is significantly below 0.5. One-sample t-test on per-participant oscillation rate (stability==1 encodes a change); expected direction: mean < 0.5. | |||||||
Transcript
You will be shown 110 sentences in random order. For each sentence, indicate whether the sentence is grammatically correct in English. The question is always: "Is the following sentence grammatically correct in English?" Press C if the sentence is grammatically correct, or press N if it is not grammatically correct. Sentence: "The new door is red". You press C. Sentence: "The new door is red". You press C. Sentence: "The new door is red". You press C. Sentence: "The new door is red". You press C. Sentence: "The new door is red". You press C. Sentence: "Red is new door the". You press N. Sentence: "Red is new door the". You press N. Sentence: "Red is new door the". You press N. Sentence: "Red is new door the". You press N. Sentence: "Red is new door the". You press N. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press N. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press N. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The ancient manuscript that the grad student who the new card catalog had confused a great deal was studying in the library was missing a page". You press C. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press C. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The novel that the horror author who the publishing company had recently fired was banned by the local library". You press N. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The lullaby that the famous country singer who the record label had signed to a big contract was singing yesterday was written seventy years ago". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press C. Sentence: "The game that the child who the lawnmower had startled in the yard was playing in the morning lasted for hours". You press N. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The crime that the gangster who the story had profiled had planned for weeks was quickly solved". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The picture that the artist who the school had expelled for cheating was hurriedly copying was printed in a magazine". You press C. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The trophy that the athlete who the restaurant had hired as a spokesman was stolen later". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The apartment that the maid who the service had sent over was well decorated". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The shirt that the seamstress who the immigration officer had investigated last week needed to be dry cleaned". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press C. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N. Sentence: "The lecture that the professor who the newspaper story had just profiled in detail was not well attended". You press N.