jansen_2021_rational

self-assessment psych-101 psych-201

Two large online studies replicate a seminal Dunning–Kruger experiment: N = 4083 participants in exp0 (Grammar) and N = 4081 in exp1 (Logical Reasoning / LSAT) each answer 20 five-option multiple-choice questions and, both before and after the test, estimate how many of the 20 they will/did answer correctly (0-20), rate their ability and their performance relative to the other participants (0-100 percentile sliders), and rate the task's difficulty for themselves and for the average participant (1-10), together with two comprehension checks and demographics. The paper fits a rational Bayesian self-assessment model (ability prior plus sensitivity to evidence ε) and shows that comparing model variants supports asymmetric sensitivity to correctness evidence — low performers are less sensitive — over biased priors as the mechanism behind the Dunning–Kruger effect. Response type: one multiple-choice answer per question plus numeric self-ratings (0-20 count, 0-100 percentiles, 1-10 difficulty).

Responses
174,184
2,200 held out
CORE effects
–
no effects simulated
CORE test NLL (avg)
1.090
rank 2 of 7 models
Best model
CORE (no phonemes)
test NLL (avg) 1.082

Likelihood of human responses

modelPNLLtest NLL (avg)test responses train NLL (avg)train responses
CORE2,398.21.09012,2001.0769171,984
CORE (no prediction errors)2,411.41.09612,2001.0826171,984
CORE (no phonemes) best2,380.51.08212,2001.0505171,984
CORE (no forgetting)2,431.81.10542,2001.0979171,984
CORE (no hierarchy)3,196.31.45292,2001.4565171,984
Transformer2,512.21.14192,2001.1513171,984
Centaur2,711.91.23272,200––

Experiments

exptask classparticipantsresponsesconditionssimulatedeffects
exp0GrammarSelfAssessment3,98786,7451no–
exp1LogicalReasoningSelfAssessment4,01087,4391no–

Behavioral effects

No behavioral effects were simulated for this study.

Transcript

You are taking part in a study on how people perceive their own abilities. You first read an informed-consent statement and then decide to continue. The task instructions say: 'In this task, you're going to be answering a set of 20 questions about grammar. In each question, some part of each sentence is underlined; sometimes the whole sentence is underlined. Five choices for rephrasing the underlined part follow each sentence; one choice repeats the original, and the other four are different. For each sentence, consider the requirements of standard written English. Your choice should be a correct and effective expression, not awkward or ambiguous. Focus on grammar, sentence structure, punctuation, wordiness, and word choice. If a choice changes the meaning of the original sentence, do not select it.' On every question you press the letter of the option you choose (A, B, C, ...). First you must pass a short comprehension test. Question: 'What subject are you going to be answering questions about?' Options: A) Grammar B) Logical reasoning C) Creative writing You press A. Question: 'What will be done to a part of each sentence?' Options: A) It will be underlined B) It will be deleted C) It will be all caps You press A. You are now asked a few questions before the grammar test. Question: Compared to other participants in this study, how would you rate  your overall ability to recognize correct grammar? (a number from 0 to 100) You answer 86. Question: How many of the 20 questions do you think you will answer correctly? (a number from 0 to 20) You answer 18. Question: Compared to other participants in this study, how well do you think you will do? (a number from 0 to 100) You answer 94. Question: How difficult is recognizing correct grammar for you? (1 = very easy, 10 = very difficult) You answer 1. Question: How difficult is recognizing correct grammar for the average participant? (1 = very easy, 10 = very difficult) You answer 6. Next question. The school-age child faces a formidable task when during the first few years of classroom experiences  (underlined: he or she is expected to master the printed form of language). Options: E) he or she is expected to master the printed form of language. A) he or she expects to master the printed form of language. C) he or she faces expectations of mastering the printed form of language. D) mastery of the printed form of language is expected of him or her. B) mastery of print is expected by his or her teacher. You press D. Next question. He came to the United States as a young  (underlined: man, he found) a job as a coal miner. Options: E) man, he found A) man and found C) man and there he was able to find D) man and then finding B) man and had found You press A. Next question. To a large degree,  (underlined: poetry, along with all the other arts, is) a form of imitation. Options: E) poetry, along with all the other arts, is A) poetry along with all the other arts is C) poetry, along with all the other arts, are D) poetry, and other arts, is B) poetry and art are You press E. Next question. Delegates to the political convention found  (underlined: difficulty to choose) a candidate from among the few nominated. Options: E) difficulty to choose A) it difficult in making the choice of C) it difficult to choose D) choosing difficult when selecting B) making a choice difficult in selecting You press C. Next question. Reading in any language can be viewed as a developmental task much the same as learning to walk, to cross the street independently, to care for one's possessions, or  (underlined: accepting responsibility for one's own decisions.) Options: E) accepting responsibility for one's own decisions. A) accepting one's own decisions responsibly. C) to accept responsibility for one's own decisions. D) accepting responsibility and making one's own decisions. B) to make one's own decisions. You press E. Next question. Sea forests of giant kelp, which fringe only one coastline in the Northern Hemisphere,  (underlined: is native to shores) throughout the Southern Hemisphere. Options: E) is native to shores A) is native to most shores C) are native only in shores D) are native B) are native to shores You press B. Next question. Taking an occasional respite between chapters or assignments is more desirable  (underlined: than a long, continuous period of study.) Options: E) than a long, continuous period of study. A) than a period of long, continuous study. C) than a long period of continuous study. D) than studying for a long, continuous period. B) than a study period long and continuous. You press D. Next question. Like so many characters in Russian fiction,  (underlined: Crime and Punishment exhibits) a behavior so foreign to the American temperament that many readers find the story rather incredible. Options: E) Crime and Punishment exhibits A) those in Crime and Punishment exhibit C) those in Crime and Punishment exhibits D) often exhibiting B) characterized by You press A. Next question. Don Quixote provides a cross section of Spanish life, thought, and  (underlined: portrays the feelings of many Spaniards) at the end of the chivalric age. Options: E) portrays the feelings of many Spaniards A) portrayal of the feelings of many Spaniards C) feelings portrayed by Spaniards D) feelings B) Spanish feelings You press A. Next question. Hamlet, Prince of Denmark thought several times of killing Claudius  (underlined: and finally succeeding) in doing so. Options: E) and finally succeeding A) that finally was successful C) finally a successful attempt D) being finally successful B) and finally succeeded You press B. Next question. The lamb  (underlined: had laid on the hay beside its mother and had begun to nurse as soon as the boy had sat) the lantern on the table. Options: E) had laid on the hay beside its mother and had begun to nurse as soon as the boy had sat A) had lain on the hay beside its mother and had begun to nurse as soon as the boy had set C) had laid on the hay beside its mother and had begun to nurse as soon as the boy had set D) had lain on the hay besides its mother and had begun to nurse as soon as the boy had set B) had lain on the hay beside it's mother and had begun to nurse as soon as the boy had set You press B. Next question. An infant,  (underlined: whether lying alone in the crib or enjoying the company of adults, is consistently fascinated at) the movement of toes and fingers. Options: E) whether lying alone in the crib or enjoying the company of adults, is consistently fascinated at A) alone or in company, is consistently fascinated at C) whether lying alone in the crib or enjoying the company of adults, is constantly fascinated at D) whether lying alone in the crib or enjoying the company of adults, is consistently fascinated by B) lonely in the crib and enjoying the company of adults is consistently fascinated at You press D. Next question. A policeman of proven valor,  (underlined: the city council designated him) the "Outstanding Law Enforcement Officer of the Year." Options: E) the city council designated him A) the city council's designating him C) the city council will designate him D) he designated the city council B) he was designated by the city council You press B. Next question. The supervisor asked,  (underlined: "Bob have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock?") Options: E) "Bob have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock?" A) "Bob, have you checked with our office in Canton, Ohio to see if it stocks slate, flagstone, and feather rock?" C) "Bob, have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock?" D) Bob, have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock? B) "Bob have you checked with our office in Canton, Ohio, to see if it stocks slate flagstone and feather rock?" You press C. Next question.  (underlined: If the room would have been brighter), I would have been more successful in my search for the lost earrings. Options: E) If the room would have been brighter A) If rooms were brighter C) If the room could have been brighter D) If the room had been brighter B) If the room was brighter You press D. Next question. After announcing that no notes could be used during the final exam, the instructor was compelled to fail  (underlined: two students because they used notes anyway.) Options: E) two students because they used notes anyway. A) two students because of their notes. C) two students because of them using notes. D) two students whose notes were used. B) two students due to the use of their notes. You press B. Next question. The respiratory membranes,  (underlined: through which exchange of gases occurs), are the linings of the lungs. Options: E) through which exchange of gases occurs A) through which exchange of gas occurs C) after gases are exchanged D) occurs through the exchange of gases B) through which gas is exchanged You press E. Next question. Jeff is one of those  (underlined: who tends to resist any attempt at) classification or regulation. Options: E) who tends to resist any attempt at A) whose tendency to resist any attempt at C) who tend to resist any attempt at D) who tends to resist any attempt to B) who tends to resistance of any attempt at You press E. Next question.  (underlined: The amount of water in living cells vary), but it is usually 65 percent and in some organisms may be as high as 96 percent or more of the total substance. Options: E) The amount of water in living cells vary A) The amount of water varies C) The amount of water in cells vary D) The amount of water in living cells varies B) The amounts of water varies in living cells You press D. Next question.  (underlined: The belief of ancient scientists was) that maggots are generated from decaying bodies and filth and are not formed by reproduction. Options: E) The belief of ancient scientists was A) The ancient scientists beliefs were C) The ancient scientists believe D) The belief of ancient scientists were B) The ancient belief of scientists was You press E. After the test you answer the same questions about your performance. Question: Compared to other participants in this study, how would you rate  your overall ability to recognize correct grammar? (a number from 0 to 100) You answer 88. Question: How many of the 20 grammar questions you just completed do you think you answered correctly? (a number from 0 to 20) You answer 17. Question: Compared to other participants in this study, how well do you think you performed? (a number from 0 to 100) You answer 92. Question: How difficult was recognizing correct grammar for you? (1 = very easy, 10 = very difficult) You answer 3. Question: How difficult was recognizing correct grammar for the average participant? (1 = very easy, 10 = very difficult) You answer 8. You also complete a short demographics questionnaire.