jansen_2021_rational
Two large online studies replicate a seminal Dunning–Kruger experiment: N = 4083 participants in exp0 (Grammar) and N = 4081 in exp1 (Logical Reasoning / LSAT) each answer 20 five-option multiple-choice questions and, both before and after the test, estimate how many of the 20 they will/did answer correctly (0-20), rate their ability and their performance relative to the other participants (0-100 percentile sliders), and rate the task's difficulty for themselves and for the average participant (1-10), together with two comprehension checks and demographics. The paper fits a rational Bayesian self-assessment model (ability prior plus sensitivity to evidence ε) and shows that comparing model variants supports asymmetric sensitivity to correctness evidence — low performers are less sensitive — over biased priors as the mechanism behind the Dunning–Kruger effect. Response type: one multiple-choice answer per question plus numeric self-ratings (0-20 count, 0-100 percentiles, 1-10 difficulty).
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE | 2,398.2 | 1.0901 | 2,200 | 1.0769 | 171,984 |
| CORE (no prediction errors) | 2,411.4 | 1.0961 | 2,200 | 1.0826 | 171,984 |
| CORE (no phonemes) best | 2,380.5 | 1.0821 | 2,200 | 1.0505 | 171,984 |
| CORE (no forgetting) | 2,431.8 | 1.1054 | 2,200 | 1.0979 | 171,984 |
| CORE (no hierarchy) | 3,196.3 | 1.4529 | 2,200 | 1.4565 | 171,984 |
| Transformer | 2,512.2 | 1.1419 | 2,200 | 1.1513 | 171,984 |
| Centaur | 2,711.9 | 1.2327 | 2,200 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | GrammarSelfAssessment | 3,987 | 86,745 | 1 | no | – |
| exp1 | LogicalReasoningSelfAssessment | 4,010 | 87,439 | 1 | no | – |
Behavioral effects
No behavioral effects were simulated for this study.
Transcript
You are taking part in a study on how people perceive their own abilities. You first read an informed-consent statement and then decide to continue. The task instructions say: 'In this task, you're going to be answering a set of 20 questions about grammar. In each question, some part of each sentence is underlined; sometimes the whole sentence is underlined. Five choices for rephrasing the underlined part follow each sentence; one choice repeats the original, and the other four are different. For each sentence, consider the requirements of standard written English. Your choice should be a correct and effective expression, not awkward or ambiguous. Focus on grammar, sentence structure, punctuation, wordiness, and word choice. If a choice changes the meaning of the original sentence, do not select it.' On every question you press the letter of the option you choose (A, B, C, ...). First you must pass a short comprehension test. Question: 'What subject are you going to be answering questions about?' Options: A) Grammar B) Logical reasoning C) Creative writing You press A. Question: 'What will be done to a part of each sentence?' Options: A) It will be underlined B) It will be deleted C) It will be all caps You press A. You are now asked a few questions before the grammar test. Question: Compared to other participants in this study, how would you rate your overall ability to recognize correct grammar? (a number from 0 to 100) You answer 86. Question: How many of the 20 questions do you think you will answer correctly? (a number from 0 to 20) You answer 18. Question: Compared to other participants in this study, how well do you think you will do? (a number from 0 to 100) You answer 94. Question: How difficult is recognizing correct grammar for you? (1 = very easy, 10 = very difficult) You answer 1. Question: How difficult is recognizing correct grammar for the average participant? (1 = very easy, 10 = very difficult) You answer 6. Next question. The school-age child faces a formidable task when during the first few years of classroom experiences (underlined: he or she is expected to master the printed form of language). Options: E) he or she is expected to master the printed form of language. A) he or she expects to master the printed form of language. C) he or she faces expectations of mastering the printed form of language. D) mastery of the printed form of language is expected of him or her. B) mastery of print is expected by his or her teacher. You press D. Next question. He came to the United States as a young (underlined: man, he found) a job as a coal miner. Options: E) man, he found A) man and found C) man and there he was able to find D) man and then finding B) man and had found You press A. Next question. To a large degree, (underlined: poetry, along with all the other arts, is) a form of imitation. Options: E) poetry, along with all the other arts, is A) poetry along with all the other arts is C) poetry, along with all the other arts, are D) poetry, and other arts, is B) poetry and art are You press E. Next question. Delegates to the political convention found (underlined: difficulty to choose) a candidate from among the few nominated. Options: E) difficulty to choose A) it difficult in making the choice of C) it difficult to choose D) choosing difficult when selecting B) making a choice difficult in selecting You press C. Next question. Reading in any language can be viewed as a developmental task much the same as learning to walk, to cross the street independently, to care for one's possessions, or (underlined: accepting responsibility for one's own decisions.) Options: E) accepting responsibility for one's own decisions. A) accepting one's own decisions responsibly. C) to accept responsibility for one's own decisions. D) accepting responsibility and making one's own decisions. B) to make one's own decisions. You press E. Next question. Sea forests of giant kelp, which fringe only one coastline in the Northern Hemisphere, (underlined: is native to shores) throughout the Southern Hemisphere. Options: E) is native to shores A) is native to most shores C) are native only in shores D) are native B) are native to shores You press B. Next question. Taking an occasional respite between chapters or assignments is more desirable (underlined: than a long, continuous period of study.) Options: E) than a long, continuous period of study. A) than a period of long, continuous study. C) than a long period of continuous study. D) than studying for a long, continuous period. B) than a study period long and continuous. You press D. Next question. Like so many characters in Russian fiction, (underlined: Crime and Punishment exhibits) a behavior so foreign to the American temperament that many readers find the story rather incredible. Options: E) Crime and Punishment exhibits A) those in Crime and Punishment exhibit C) those in Crime and Punishment exhibits D) often exhibiting B) characterized by You press A. Next question. Don Quixote provides a cross section of Spanish life, thought, and (underlined: portrays the feelings of many Spaniards) at the end of the chivalric age. Options: E) portrays the feelings of many Spaniards A) portrayal of the feelings of many Spaniards C) feelings portrayed by Spaniards D) feelings B) Spanish feelings You press A. Next question. Hamlet, Prince of Denmark thought several times of killing Claudius (underlined: and finally succeeding) in doing so. Options: E) and finally succeeding A) that finally was successful C) finally a successful attempt D) being finally successful B) and finally succeeded You press B. Next question. The lamb (underlined: had laid on the hay beside its mother and had begun to nurse as soon as the boy had sat) the lantern on the table. Options: E) had laid on the hay beside its mother and had begun to nurse as soon as the boy had sat A) had lain on the hay beside its mother and had begun to nurse as soon as the boy had set C) had laid on the hay beside its mother and had begun to nurse as soon as the boy had set D) had lain on the hay besides its mother and had begun to nurse as soon as the boy had set B) had lain on the hay beside it's mother and had begun to nurse as soon as the boy had set You press B. Next question. An infant, (underlined: whether lying alone in the crib or enjoying the company of adults, is consistently fascinated at) the movement of toes and fingers. Options: E) whether lying alone in the crib or enjoying the company of adults, is consistently fascinated at A) alone or in company, is consistently fascinated at C) whether lying alone in the crib or enjoying the company of adults, is constantly fascinated at D) whether lying alone in the crib or enjoying the company of adults, is consistently fascinated by B) lonely in the crib and enjoying the company of adults is consistently fascinated at You press D. Next question. A policeman of proven valor, (underlined: the city council designated him) the "Outstanding Law Enforcement Officer of the Year." Options: E) the city council designated him A) the city council's designating him C) the city council will designate him D) he designated the city council B) he was designated by the city council You press B. Next question. The supervisor asked, (underlined: "Bob have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock?") Options: E) "Bob have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock?" A) "Bob, have you checked with our office in Canton, Ohio to see if it stocks slate, flagstone, and feather rock?" C) "Bob, have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock?" D) Bob, have you checked with our office in Canton, Ohio, to see if it stocks slate, flagstone, and feather rock? B) "Bob have you checked with our office in Canton, Ohio, to see if it stocks slate flagstone and feather rock?" You press C. Next question. (underlined: If the room would have been brighter), I would have been more successful in my search for the lost earrings. Options: E) If the room would have been brighter A) If rooms were brighter C) If the room could have been brighter D) If the room had been brighter B) If the room was brighter You press D. Next question. After announcing that no notes could be used during the final exam, the instructor was compelled to fail (underlined: two students because they used notes anyway.) Options: E) two students because they used notes anyway. A) two students because of their notes. C) two students because of them using notes. D) two students whose notes were used. B) two students due to the use of their notes. You press B. Next question. The respiratory membranes, (underlined: through which exchange of gases occurs), are the linings of the lungs. Options: E) through which exchange of gases occurs A) through which exchange of gas occurs C) after gases are exchanged D) occurs through the exchange of gases B) through which gas is exchanged You press E. Next question. Jeff is one of those (underlined: who tends to resist any attempt at) classification or regulation. Options: E) who tends to resist any attempt at A) whose tendency to resist any attempt at C) who tend to resist any attempt at D) who tends to resist any attempt to B) who tends to resistance of any attempt at You press E. Next question. (underlined: The amount of water in living cells vary), but it is usually 65 percent and in some organisms may be as high as 96 percent or more of the total substance. Options: E) The amount of water in living cells vary A) The amount of water varies C) The amount of water in cells vary D) The amount of water in living cells varies B) The amounts of water varies in living cells You press D. Next question. (underlined: The belief of ancient scientists was) that maggots are generated from decaying bodies and filth and are not formed by reproduction. Options: E) The belief of ancient scientists was A) The ancient scientists beliefs were C) The ancient scientists believe D) The belief of ancient scientists were B) The ancient belief of scientists was You press E. After the test you answer the same questions about your performance. Question: Compared to other participants in this study, how would you rate your overall ability to recognize correct grammar? (a number from 0 to 100) You answer 88. Question: How many of the 20 grammar questions you just completed do you think you answered correctly? (a number from 0 to 20) You answer 17. Question: Compared to other participants in this study, how well do you think you performed? (a number from 0 to 100) You answer 92. Question: How difficult was recognizing correct grammar for you? (1 = very easy, 10 = very difficult) You answer 3. Question: How difficult was recognizing correct grammar for the average participant? (1 = very easy, 10 = very difficult) You answer 8. You also complete a short demographics questionnaire.
You are taking part in a study on how people perceive their own abilities. You first read an informed-consent statement and then decide to continue. The task instructions say: 'In this task, you're going to be answering a set of 20 questions about logical reasoning. You will be presented with brief passages or statements and will be required to evaluate their reasoning or determine what inferences you can logically draw from the passage. In each case, select the best answer choice, even though more than one choice may present a possible answer.' On every question you press the letter of the option you choose (A, B, C, ...). First you must pass a short comprehension test. Question: 'What subject are you going to be answering questions about?' Options: A) Grammar B) Logical reasoning C) Creative writing You press B. Question: 'Which answer choice should you select?' Options: A) The best choice B) The longest choice C) All of the choices that are correct You press A. You are now asked a few questions before the test. Question: Compared to other participants in this study, how would you rate your general logical reasoning ability? (a number from 0 to 100) You answer 65. Question: How many of the 20 logical reasoning problems do you think you will solve correctly? (a number from 0 to 20) You answer 14. Question: Compared to other participants in this study, how well do you think you will do? (a number from 0 to 100) You answer 60. Question: How difficult is solving logical reasoning problems for you? (1 = very easy, 10 = very difficult) You answer 6. Question: How difficult is solving logical reasoning problems for the average participant? (1 = very easy, 10 = very difficult) You answer 6. Next question. Life imitates art. Which of the following, if true, most strongly supports the previous statement? Options: D) When Warren Beatty filmed Reds, he tried to suggest not only the chaos of the Russian Revolution but also its relationship to the present. C) The number of professional ballet companies has increased over the last five years, but the number of dance majors has decreased. E) On Tuesday, the business section of the newspaper had predicted the drop in interest rates that occurred on Friday. B) Truman Capote wrote In Cold Blood as a result of a series of brutal slayings by two crazed killers. A) Soon after the advent of color television, white shirts became less popular as dressy attire for men, and pastel-colored shirts began to sell well. You press A. Next question. On average, federal workers receive salaries 35.5 percent higher than private-sector salaries. For instance, federal workers in California average $19,206 a year, 25 percent higher than the average pay in the private sector, which is $15,365. This information would best support which of the following opinions? Options: D) Private-sector salaries in California are above average. C) The private sector is being paid fairly. E) Federal jobs are more secure than private-sector jobs. B) Public-sector work is more difficult than private-sector work. A) Federal pay is out of line. You press B. Next question. No high jumper entered the track meet unless he or she was a track club member. No track club member both entered the meet and was a high jumper. Which of the following conclusions can be correctly drawn from the two previous sentences? Options: D) No one but high jumpers entered the meet. C) Only track club members entered the meet. E) No track club members entered the meet. B) No high jumper entered the meet. A) Some track club members entered the meet. You press E. Next question. About 33% of American men between 25 and 50 are overweight. Research has shown that in most cases men between 25 and 50 who are overweight are more subject to heart disease than men who are not overweight. Which of the following is the most logical conclusion to this argument? Options: D) Therefore, 33% of the American men between 25 and 50 should lose weight. C) Therefore, if 33% of the American men between 25 and 50 were to lose weight, they would reduce their risk of heart disease. E) Therefore, if the men between 25 and 50 who are overweight were to lose weight, they would reduce their risk of heart disease by 33%. B) Therefore, if 33% of American men were to lose weight, they would reduce their risk of heart disease. A) Therefore, if the overweight men between 25 and 50 were to lose weight, their risk of heart disease would be reduced. You press A. Next question. All computer geniuses are also brilliant mathematicians. Therefore, some computer geniuses don’t require calculators for simple multiplication facts. Which of the following is the least necessary assumption for the previous conclusion to be logically correct? Options: D) Some brilliant mathematicians don’t require calculators for simple multiplication facts. C) All brilliant mathematicians don’t require calculators for simple multiplication facts. E) Some brilliant mathematicians require calculators for simple multiplication facts. B) All computer geniuses who require calculators for simple multiplication facts are brilliant mathematicians. A) Only computer geniuses are also brilliant mathematicians. You press A. Next question. A researcher has concluded that women are just as capable as men in math but that their skills are not developed because society expects them to develop other and more diverse abilities. Which of the following is a basic assumption of the researcher? Options: D) Ability in math is more important than ability in more diverse subjects. C) Ability in math is less important than ability in more diverse subjects. E) Women and men should be equally capable in math. B) Women might be more capable than men in math. A) Women tend to conform to social expectations. You press A. Next question. Jonathan Swift said, “Laws are like cobwebs which may catch small flies but let wasps and hornets break through.” Jonathan Swift would most likely believe that Options: D) prosecutors should be tough on criminals C) pesticides should be used to deter large insects E) small crimes should not be prosecuted B) the powerful can often avoid serious criminal sentences A) laws do not stop people from committing crimes You press B. Next question. When Mom cooks, we all eat a delicious dinner. But Mom didn’t cook today, so we won’t be eating a delicious dinner. Which of the following is logically most similar to the previous argument? Options: D) When food is cooked, it always burns. But our food isn’t burned, so therefore it wasn’t cooked. C) When silverware is used at the dining table, we usually have guests. Today we have guests, so we are using the silverware. E) When the dog has fleas, he always scratches. But the dog doesn’t have fleas, so he won’t be scratching. B) When a person is fortunate, he or she has great good luck. So a fortunate person will always be lucky. A) When a university finishes admitting entering students, the freshman class is complete. Since the freshman class is not complete, the university has not finished admitting entering students. You press E. Next question. Without sign ordinances, everyone with the price of a can of spray paint can suddenly decide to publicly create their own personal Picassos, and soon the entire town would start to look like something out of Alice in Wonderland. Therefore we need sign ordinances. All of the following are assumptions underlying the previous argument EXCEPT Options: D) spray paint can be used to create graffiti C) the town looking like Alice in Wonderland is undesirable E) sign ordinances are effective B) no other effective means of deterring graffiti presently exist A) sign ordinances are rarely, if ever effective You press A. Next question. When Louis Pasteur said, “Chance favors the prepared mind,” the famous French scientist most nearly meant Options: D) take a chance only if you’re prepared C) pasteurization was a chance that Pasteur prepared for E) being prepared will be favorable to those who take chances B) happenstance will be more beneficial to those who are prepared A) we all have a chance to be prepared You press B. Next question. All acts have consequences. Given this fact, we may wish to play it safe by never doing anything. The speaker implies that Options: D) we may prefer to live safely C) all acts have consequences E) consequentiality is not safe B) doing nothing has lesser consequences A) not doing anything is not an act You press B. Next question. Voltaire once said, “Common sense is not so common.” Which of the following most nearly parallels Voltaire’s statement? Options: D) God must have loved the common man; he certainly made enough of them. C) The common good is not necessarily best for everyone. E) Jumbo shrimp may not actually be very big. B) Good people may not necessarily have good sense. A) Truth serum cannot contain the truth. You press B. Next question. It has been proven that the “lie detector” can be fooled. If one is truly aware that one is lying, when in fact one is, then the “lie detector” is worthless. The author of this argument implies that Options: D) the lie detector is a useless device C) a good liar can fool the device E) a lie detector is often inaccurate B) the lie detector is sometimes worthless A) no one can fool the lie detector all of the time You press C. Next question. It has been proven that the “lie detector” can be fooled. If one is truly aware that one is lying, when in fact one is, then the “lie detector” is worthless. This argument would be strengthened most by Options: D) demonstrating that one’s awareness of truth or falsity is always undetectable C) citing evidence that there are other means of measuring truth which are consistently less reliable than the lie detector E) citing the number of cases in which the lie detector mistook falsehood for truth B) claiming that ordinary, unbiased people are the best “lie detectors” A) showing that the “truth” of any statement always relies on a subjective assessment You press E. Next question. It has been proven that the “lie detector” can be fooled. If one is truly aware that one is lying, when in fact one is, then the “lie detector” is worthless. Without contradicting his or her own statements, the author of the above statement might present which of the following arguments as a strong point in favor of the lie detector? Options: D) The methodology used by investigative critics of the lie detector is itself highly flawed. C) Circumstantial evidence might be more useful in a criminal case than is personal testimony. E) The very threat of a lie-detector test has led to a significant number of criminals to confess. B) People are never “truly unaware” that they are lying. A) Law-enforcement agencies have purchased too many detectors to abandon them now. You press E. Next question. English automobiles leak oil. All sportscars need some repair every month. Since the vehicle I recently purchased leaks oil and needs repair every month, I must have purchased an English sportscar. Which of the following, if true, would logically weaken the previous conclusion? Options: D) Only English sportscars need repair every month. C) Not all English sedans leak oil. E) Some sportscars need repair every two weeks. B) Danish automobiles also leak oil. A) American sportscars never need repair. You press B. Next question. By appropriating bailout money for the depressed housing industry, Congress is opening the door to a flood of special relief programs for other recession-affected businesses. The author’s attitude toward the Congress’ action is probably Options: D) neutral C) disapproving E) confused B) happy A) irate You press C. Next question. We have been warned that if we stop watering our lawn, then not only will our grass die and our trees turn brown, but also the gophers will find the dry, hard soil a stimulus for ravaging whatever vegetation happens to survive. Therefore, we have decided to continue to water our lawn. All of the following can be reasonably inferred as goals of the author except Options: D) the grass is not dying C) the trees not turning brown E) the gophers not ravaging remaining vegetation B) the soil not becoming dry and hard A) water conservation You press A. Next question. According to a recent study by the National Academy of Public Administration, postal patrons are regularly affronted by out-of-order stamp vending machines, branch post office lobbies locked at night, and twenty-nine cent letters that take as long to get there as thirteen-cent letters did a decade ago. Which of the following, if true, would weaken the implication of one of the author’s observations? Options: D) Most out-of-order vending machines are located in run-down neighborhoods. C) Late-night vandalism has plagued post offices nationwide. E) Postage rates rose over a hundred percent from 1983 to 1993, but the cost of first class mail is still cheaper in the US than anywhere else. B) As a public corporation, the Postal Service has increased its capital assets by $3 billion. A) Ten years ago, most letters reached their destination within twenty-four hours. You press A. Next question. If I do not get at least a B on the final exam, I will definitely fail my geology course. From the previous statement, it most logically follows that if I do get a B on the final exam in geology, I then Options: D) may or may not pass the course C) will definitely pass the course E) will probably not pass the course B) will probably pass the course A) will definitely not pass the course You press A. After the test you answer the same questions about your performance. Question: Compared to other participants in this study, how would you rate your general logical reasoning ability? (a number from 0 to 100) You answer 65. Question: How many of the 20 logical reasoning problems you just completed do you think you answered correctly? (a number from 0 to 20) You answer 12. Question: Compared to other participants in this study, how well do you think you performed? (a number from 0 to 100) You answer 60. Question: How difficult was solving these logical reasoning problems for you? (1 = very easy, 10 = very difficult) You answer 8. Question: How difficult was solving these logical reasoning problems for the average participant? (1 = very easy, 10 = very difficult) You answer 8. You also complete a short demographics questionnaire.