gnther_2022_patterns
Two crowdsourced (Prolific) rating experiments test whether distributed semantic-vector representations (whole-word, additive, and CAOSS composition) predict people's relational interpretations of English nominal compounds. In each trial a participant selects 1 of 16 possible relational meanings (e.g., X MADE OF Y, X HAS Y) for a presented compound. Experiment 1 (familiar compounds, ~575 items, data reused from Schmidtke et al. 2018 — present only as per-item aggregate counts) and Experiment 2 (408 novel compounds, N = 400 participants at 55 items each: 51 novel compounds from one of eight lists plus 4 catch trials) both yield per-compound frequency distributions over the 16 relations; a multivariate least-squares regression mapping system trained on the vectors under leave-one-out cross-validation predicts these distributions. The available trial-level data covers Experiment 2 (400 participants × 55 trials = 22,000 rows), where participants made a 16-relation forced-choice judgment per trial. Response is the selected relational-interpretation label.
Likelihood of human responses
| model | PNLL | test NLL (avg) | test responses | train NLL (avg) | train responses |
|---|---|---|---|---|---|
| CORE | 5,555.6 | 2.5253 | 2,200 | 2.4706 | 19,800 |
| CORE (no prediction errors) | 5,700.7 | 2.5912 | 2,200 | 2.5369 | 19,800 |
| CORE (no phonemes) | 6,231.3 | 2.8324 | 2,200 | 0.6510 | 19,800 |
| CORE (no forgetting) | 5,857.8 | 2.6627 | 2,200 | 2.6408 | 19,800 |
| CORE (no hierarchy) | 6,136.3 | 2.7892 | 2,200 | 2.7629 | 19,800 |
| Transformer | 6,197.7 | 2.8172 | 2,200 | 2.4668 | 19,800 |
| Centaur best | 5,482.5 | 2.4921 | 2,200 | – | – |
Experiments
| exp | task class | participants | responses | conditions | simulated | effects |
|---|---|---|---|---|---|---|
| exp0 | CompoundRelationalJudgment | 400 | 22,000 | 1 | yes | catch_trial_accuracy, relations_systematic_nonuniform, high_interpretational_variability |
Behavioral effects
| effect | experiment | paper | CORE | CORE (no prediction errors) | CORE (no phonemes) | CORE (no forgetting) | CORE (no hierarchy) |
|---|---|---|---|---|---|---|---|
catch_trial_accuracy | exp0 | +0.085 | ✗ no +0.000 | ✗ no +0.000 | ✓ yes +0.120 | ✗ no +0.000 | ✗ no +0.000 |
the four embedded catch trials (snow man, stone ware, sweat band, tea time) have well-defined normative answers; participants answer them at far above the 1/16 chance level, and only 34 of 400 (8.5%) participants are non-compliant (<3 of 4 correct). Test: binomial proportion vs. chance; direction: accuracy >> chance. | |||||||
relations_systematic_nonuniform | exp0 | +0.062 | ✗ no +0.068 | ✗ no +0.068 | ✓ yes +0.196 | ✗ no +0.066 | ✗ no +0.070 |
across the 16 possible relational interpretations, participants' choices are strongly non-uniform / systematic rather than random. Test: chi-square goodness-of-fit of the pooled relation counts against a uniform distribution; direction: observed distribution differs strongly from uniform, top relation far above the 1/16 chance share. | |||||||
high_interpretational_variability | exp0 | +2.338 | ✓ yes +2.609 | ✓ yes +2.607 | ✓ yes +2.017 | ✓ yes +2.611 | ✓ yes +2.605 |
item-level relational entropy of the 408 novel compounds is high (paper reports M = 2.338), i.e., participants do not converge on a single relation but spread across several — substantial across-subject variation. Test: one-sample t-test that mean item entropy exceeds a near-unanimous baseline; direction: entropy clearly > 0 / > 1, p < .05. | |||||||
Transcript
You are taking part in a study of how people interpret novel English compounds: words made of two familiar words combined, which you have almost certainly never seen before. In each trial you see one such novel compound written as two words. The FIRST word is the modifier; the SECOND word is the head. You must choose the one relation that best expresses what the whole compound means, from sixteen possibilities: A: the head causes the modifier B: the head is the modifier C: the head is during the modifier D: the head is used by the modifier E: the modifier is located at the head F: the modifier has the head G: the head is about the modifier H: the head is caused by the modifier I: the head is located at the modifier J: the head uses the modifier K: the head is made of the modifier L: the head is for the modifier M: the head is by the modifier N: the head has the modifier O: the head makes the modifier P: the head is from the modifier You answer by typing the letter that names the relation you choose, e.g. type the single letter next to your chosen relation. You see the novel compound 'brick steak'. What does it mean? You press K. You see the novel compound 'loin robe'. What does it mean? You press L. You see the novel compound 'sand quake'. What does it mean? You press H. You see the novel compound 'meadow luck'. What does it mean? You press H. You see the novel compound 'band rock'. What does it mean? You press H. You see the novel compound 'hearth belly'. What does it mean? You press E. You see the novel compound 'stone ware'. What does it mean? You press K. You see the novel compound 'mast hoe'. What does it mean? You press I. You see the novel compound 'byte miner'. What does it mean? You press L. You see the novel compound 'cash summer'. What does it mean? You press J. You see the novel compound 'beef broker'. What does it mean? You press N. You see the novel compound 'sea buoy'. What does it mean? You press I. You see the novel compound 'bob load'. What does it mean? You press F. You see the novel compound 'rose mill'. What does it mean? You press M. You see the novel compound 'back boat'. What does it mean? You press I. You see the novel compound 'hunter poll'. What does it mean? You press L. You see the novel compound 'bag street'. What does it mean? You press B. You see the novel compound 'info finch'. What does it mean? You press N. You see the novel compound 'skate band'. What does it mean? You press G. You see the novel compound 'birth wife'. What does it mean? You press N. You see the novel compound 'consul son'. What does it mean? You press F. You see the novel compound 'tea time'. What does it mean? You press L. You see the novel compound 'milk friend'. What does it mean? You press K. You see the novel compound 'switch job'. What does it mean? You press H. You see the novel compound 'tree soil'. What does it mean? You press F. You see the novel compound 'use record'. What does it mean? You press C. You see the novel compound 'ion riding'. What does it mean? You press H. You see the novel compound 'lunch day'. What does it mean? You press L. You see the novel compound 'win slot'. What does it mean? You press A. You see the novel compound 'day fish'. What does it mean? You press C. You see the novel compound 'saw fuel'. What does it mean? You press L. You see the novel compound 'snow man'. What does it mean? You press K. You see the novel compound 'tram fill'. What does it mean? You press L. You see the novel compound 'special owl'. What does it mean? You press B. You see the novel compound 'shop room'. What does it mean? You press B. You see the novel compound 'rein shine'. What does it mean? You press H. You see the novel compound 'strait timer'. What does it mean? You press B. You see the novel compound 'bubble step'. What does it mean? You press P. You see the novel compound 'gilt round'. What does it mean? You press B. You see the novel compound 'bog term'. What does it mean? You press C. You see the novel compound 'shin click'. What does it mean? You press J. You see the novel compound 'cheese road'. What does it mean? You press K. You see the novel compound 'sweat band'. What does it mean? You press L. You see the novel compound 'anchor domain'. What does it mean? You press J. You see the novel compound 'hair dress'. What does it mean? You press L. You see the novel compound 'meeting jar'. What does it mean? You press L. You see the novel compound 'sea storm'. What does it mean? You press I. You see the novel compound 'dance swing'. What does it mean? You press P. You see the novel compound 'jet giving'. What does it mean? You press G. You see the novel compound 'goal coach'. What does it mean? You press G. You see the novel compound 'waist set'. What does it mean? You press L. You see the novel compound 'sand tan'. What does it mean? You press P. You see the novel compound 'review onset'. What does it mean? You press F. You see the novel compound 'wine button'. What does it mean? You press L. You see the novel compound 'folk line'. What does it mean? You press P.