Domain Report - Annotation Quality

Domain: annotation-quality

Roughly one-third of crowdworkers used LLMs on an LLM-advantaged text-production task (33-35% on Prolific, July 2023; ~34% Prolific self-report in a 2025 follow-up by Zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). Load-bearing

EVIDENCE: Veselovsky et al. ran two preregistered Prolific studies (n=168 and n=720, medical-abstract summarization) with a calibrated classifier plus self-report and keystroke heuristics; prevalence 33.3-35.4% baseline (95% CIs ~[26%,43%]); a 3x3 factorial of requests x hurdles halved but did not eliminate use. The June 2026 community survey (arXiv 2606.04924) corroborates: 30-40% estimated reliance and 34% Prolific self-report, and notes mitigations 'reduce rather than eliminate' LLM-assisted responses. SOURCE: Prevalence and Prevention of Large Language Model Use in Crowd Work (CACM; arXiv 2310.15683) | https://arxiv.org/abs/2310.15683 | 2025-02-18 | academic IMPLICATION: AutoQA must assume a contamination base rate of ~15-35% on any text-heavy attempter task even after friction measures; per-item content detection and 'please don't' policies cannot be the primary defense, and mitigation friction has a measurable quality cost that the QA layer must not misattribute to attempter skill.

An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. Load-bearing

EVIDENCE: Westwood (PNAS, Nov 20 2025; 43,800 evaluations across 9 LLMs) shows instruction-following checks, logic puzzles, reCAPTCHA, and rare-event probes all fail; persona calibration even masks superhuman-accuracy tells. Recommended defenses are entirely provenance/economics-based: identity revalidation, per-worker throttling, completion-history transparency, pass/fail history, location/VPN checks - 'there is no magical fix.' The 2026 crowdsourcing survey (arXiv 2606.04924) independently confirms detectors degrade under paraphrase and out-of-domain, and mixed human-AI authorship is harder still. SOURCE: The potential existential threat of large language models to online survey research (PNAS 122(47)) | https://www.pnas.org/doi/10.1073/pnas.2518075122 | 2025-11-20 | academic IMPLICATION: Directly answers taxonomy Q8: wholesale-LLM-submission defense must be economics/provenance-based (rate limits, work-history consistency, randomized deep audits, process telemetry), not per-item text classification; any AutoQA spam layer built on stylistic detection will be silently beaten within the cost asymmetry.

Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. Load-bearing

EVIDENCE: Elangovan et al. (Amazon), ICLR 2025 (arXiv 2410.03775, opened and verified): Krippendorff's alpha / kappa were designed for human-human reliability and their assumptions break for machine labels; the paper's fix is (1) stratify meta-evaluation by human-label uncertainty, (2) use binned Jensen-Shannon divergence for perception-type (inherently subjective) tasks, (3) report 'perception charts' instead of one correlation. Code open-sourced (amazon-science/BeyondCorrelation). SOURCE: Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025) | https://arxiv.org/abs/2410.03775 | 2025-05-01 | academic IMPLICATION: Answers taxonomy Q1 directly: meta-evaluation must stratify gold/validation items by measured human agreement level and report per-stratum judge performance; 'accuracy vs. humans' as a single number should be banned from reporting, and judge validity claims are only meaningful on the high-human-agreement stratum.

In human preference/rating data, the majority of annotator disagreements are attributable to task underspecification and response-style preferences - not random noise and not primarily expertise gaps - and standard aggregation (Bradley-Terry reward modeling) plus LLM-as-judge evaluation both fail to account for this divergence. Load-bearing

EVIDENCE: Zhang et al., ICML 2025 (arXiv 2410.14632, opened and verified): built a 10-category, 4-class taxonomy of disagreement sources on preference datasets; abstract states 'the majority of disagreements are due to factors such as task underspecification or response style,' explicitly challenging the noise assumption; they show LLM evaluations are heavily swayed by divisive style features and develop divergence-identification methods. Converges with Jiang & de Marneffe (TACL 2022) who found the same structure in NLI. SOURCE: Diverging Preferences: When do Annotators Disagree and do Models Know? (ICML 2025) | https://arxiv.org/abs/2410.14632 | 2025-07-15 | academic IMPLICATION: Supports root-cause-1 primacy and taxonomy Q2: the verdict ontology needs a first-class 'instructions underdetermine this case' class detected via measured divergence, and a cheap variance-attribution pilot is empirically justified because published decompositions find underspecification (fixable by rubric compilation) dominates over noise (fixable by averaging).

Even on a nominally objective task (NLI), annotator disagreement decomposes into a 10-category taxonomy across three high-level sources - uncertainty in sentence meaning, underspecification in guidelines, and annotator behavior - proving that a fraction of inter-reviewer variance is item-intrinsic and survives perfect rubric operationalization.

EVIDENCE: Jiang & de Marneffe, TACL 2022, developed and hand-applied the taxonomy to high-disagreement NLI items; disagreement persisted among trained annotators on items with genuine interpretive ambiguity. D3CODE (EMNLP 2024; 4.5K sentences, 4K+ annotators, 21 countries) adds that residual variance is systematically structured by annotators' moral values and region, not random. SOURCE: Investigating Reasons for Disagreement in Natural Language Inference (TACL 10) | https://aclanthology.org/2022.tacl-1.78/ | 2022-12-01 | academic IMPLICATION: The target metric cannot be raw agreement: some disagreement is item-intrinsic signal. The variance-attribution pilot should code disagreements against a taxonomy like this to split rubric-fixable variance (underspecification) from irreducible variance (ambiguity/values) before setting build order.

Chance-corrected agreement coefficients (Cohen's kappa, and analogously Krippendorff's alpha) can be near zero despite very high raw agreement when class prevalence is extreme (the Feinstein-Cicchetti paradoxes), so under the 90%+ pass rates typical of QA pipelines, single-coefficient reliability reporting is misleading in the opposite direction from raw percent agreement.

EVIDENCE: Feinstein & Cicchetti (J Clin Epidemiol, 1990) formalized the two kappa paradoxes (high agreement/low kappa under imbalance; asymmetric marginals inflating kappa); the effect is prevalence-driven, extensively replicated (Quarfoot & Levine, Am. Statistician 2016; Gwet's AC1 literature). This is exactly the regime of QA pass/fail data with low failure base rates. SOURCE: High agreement but low kappa: I. The problems of two paradoxes (Feinstein & Cicchetti) | https://doi.org/10.1016/0895-4356(90)90158-L | 1990-01-01 | academic IMPLICATION: Answers taxonomy Q1 stats question: standardize on reporting a chance-corrected coefficient PLUS per-class recall at fixed prevalence PLUS CIs; ban both raw percent-agreement AND any single coefficient in isolation, because at extreme pass rates alpha/kappa collapse even for a well-functioning judge.

Classical label-aggregation models (Dawid-Skene 1979, MACE 2013 with its explicit spammer latent variable, GLAD) all assume annotators are conditionally independent given the true label - an assumption measurably violated when the 'annotators' are LLM judges sharing data/architectures/prompts, causing miscalibrated posteriors and confidently wrong aggregate verdicts; Feb 2026 work replaces this with dependence-aware Ising-model aggregation.

EVIDENCE: Balasubramanian, Podkopaev & Kasiviswanathan (Amazon/UC Davis, arXiv 2601.22336, Feb 2 2026) state the CI assumption 'is often violated by LLM judges due to shared data, architectures, prompts, and failure modes' and that ignoring dependencies 'can yield miscalibrated posteriors and even confidently incorrect predictions'; they build aggregation via Ising models. MACE (Hovy et al., NAACL 2013) remains the canonical spam-aware human aggregator, with production implementations in Toloka's crowd-kit. SOURCE: Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models (arXiv 2601.22336) | https://arxiv.org/pdf/2601.22336 | 2026-02-02 | academic IMPLICATION: If AutoQA uses k-sample voting or multi-judge ensembles, naive Dawid-Skene/majority aggregation will overstate confidence because same-family judge samples share failure modes; confidence estimates must model correlated errors (or use genuinely different model families), and MACE-style competence/spam latent variables remain the right tool for the human-reviewer side.

LLM contamination in low-dimensional annotation data (multiple-choice labels, ratings) can be evaluated at the worker level without any ground truth, using peer-prediction scores that condition on LLM-generated reference labels - a training-free mechanism with theoretical guarantees under an LLM-collusion model, published at NeurIPS 2025 - because text-based detectors are inapplicable to short label outputs.

EVIDENCE: Zhang, Pang, Zhu & Liu (NeurIPS 2025; arXiv 2506.06991, opened and verified): existing detectors need high-dimensional text; their mechanism scores the informativeness of worker answers via inter-worker correlation conditioned on a subset of LLM labels, explicitly modeling collusion (multiple workers pasting from the same LLM), and empirically detects low-effort cheating on real crowdsourcing datasets. Scoring is per-worker, not per-item. SOURCE: Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth (NeurIPS 2025) | https://arxiv.org/abs/2506.06991 | 2025-12-01 | academic IMPLICATION: For rating/label-type attempter outputs where stylistic detection is impossible, the viable contamination sensor is worker-level statistical scoring (cross-worker correlation vs. an LLM-conditioned baseline) run continuously - reinforcing that provenance/behavioral defense operates at the annotator level, not the item level.

The NLP field has institutionalized disagreement-as-signal: the third Learning-With-Disagreements shared task (LeWiDi-2025, EMNLP 2025) standardizes dual evaluation - soft-label (predict the population label distribution, Wasserstein/Manhattan distance) and perspectivist (recover individual annotators' labels) - across four datasets that all ship per-annotator labels rather than adjudicated gold.

EVIDENCE: Leonardelli et al. (arXiv 2510.08460, Oct 2025; le-wi-di.github.io) describe the two complementary paradigms and public leaderboard; lineage runs from Aroyo & Welty's 'truth is a lie' perspectivism through DICES (NeurIPS 2023 safety dataset with up to 100+ raters/item) and jury learning (Gordon et al., CHI 2022, which composes verdicts from explicitly selected rater subpopulations); Frenda et al.'s survey (LREV 2024) documents the dataset ecosystem. SOURCE: LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task | https://arxiv.org/pdf/2510.08460 | 2025-10-09 | academic IMPLICATION: The AutoQA data model should preserve rater identity and full label distributions (never only adjudicated verdicts), and for judgment-laden axes the judge's target should be the distribution/jury composition, not a forced scalar - this is now the field-standard evaluation contract, giving ready-made metrics (Wasserstein/Manhattan soft-label distance) for 'consistency-of-application' monitoring.

Gold/test labels themselves are wrong at material rates - a lower-bound average of 3.3% label errors across 10 canonical benchmarks (at least 6% in the ImageNet validation set), validated by human review of confident-learning-flagged candidates (51% flag precision) - and error rates this size are enough to flip model rankings.

EVIDENCE: Northcutt, Athalye & Mueller (NeurIPS 2021 Datasets & Benchmarks; arXiv 2103.14749, opened and verified): algorithmic flagging via confident learning + crowdsourced verification; ResNet-18 beats ResNet-50 on corrected ImageNet if mislabeled prevalence rises just 6%. Cleanlab is the maintained open-source implementation. SOURCE: Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (NeurIPS 2021 D&B) | https://arxiv.org/abs/2103.14749 | 2021-11-01 | academic IMPLICATION: Answers taxonomy Q1's false-agreement question in the only measured form available: several percent of any 'gold' set is wrong, so judge-vs-gold disagreement at that rate is uninformative; the gold set needs its own standing audit channel (confident-learning-style flagging + expert re-adjudication) as a permanent component, and judge certification margins must exceed the gold error floor.

Showing annotators LLM suggestions makes them anchor on the suggestions - significantly shifting the label distribution versus unassisted baseline, raising self-reported confidence without making them faster - and evaluating models against LLM-assisted labels significantly inflates reported model performance.

EVIDENCE: Schroeder, Roy & Kabbara (Findings of ACL 2025, July 2025; pre-registered, 350 annotators, 7,000 annotations, 4 conditions x 2 models x 2 datasets, opened and verified): annotators 'strongly took the LLM suggestions'; the paper warns homogenization/lessened variation is a direct risk when LLMs are inserted into annotation loops, and that 'human-approved' LLM-assisted labels change downstream conclusions. SOURCE: Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks (Findings of ACL 2025) | https://aclanthology.org/2025.findings-acl.1323/ | 2025-07-27 | academic IMPLICATION: Direct evidence for taxonomy Q8 monoculture and Q9 induced drift: if AutoQA feedback (or any AI verdict) is visible to attempters/reviewers before they judge, distributions converge on judge-pleasing labels and self-evaluation inflates - so human touches must be blind-first, and population-level label-distribution drift is the right monoculture sensor.

RLVR-style reasoning training degrades an LLM's ability to model human annotator disagreement, while naive chain-of-thought on RLHF models improves it - evaluated across 60 setups on 3 tasks - meaning stronger 'reasoning' judges are measurably worse at knowing when humans legitimately diverge.

EVIDENCE: Ni et al., EACL 2026 main (arXiv 2506.19467 v3, Jan 12 2026, opened and verified): systematic evaluation over model sizes, distribution-expression and steering methods using variance-correlation and distributional-alignment metrics; authors explicitly warn about 'the potential risk of replacing human annotators with reasoning LLMs' where disagreement carries signal; author communication adds that small fine-tuned models (ModernBERT-class) with human labels often beat large reasoning models at this. SOURCE: Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? (EACL 2026) | https://arxiv.org/abs/2506.19467 | 2026-01-12 | academic IMPLICATION: For the 'instructions-underdetermine / route-to-human' triage lane, do not assume the strongest reasoning judge is the best ambiguity detector: disagreement-prediction may need a dedicated small fine-tuned model (or CoT-on-RLHF configuration) separate from the verdict judge.

Tools & artifacts

• Cleanlab (confident learning): Open-source library implementing confident learning for algorithmic label-error detection in labeled datasets | https://github.com/cleanlab/cleanlab | Ready-made machinery for the standing gold-set audit channel: flag suspect gold/QA labels for expert re-adjudication (51% flag precision measured in the NeurIPS 2021 study).
• crowd-kit (Toloka): Maintained Python library of aggregation/truth-inference algorithms: Dawid-Skene, GLAD, MACE, Wawa, ZeroBasedSkill, IBCC/EBCC | https://github.com/Toloka/crowd-kit | Production implementations for aggregating multiple human reviewers (and multi-judge samples) with per-worker competence and spammer modeling - but note the conditional-independence caveat for same-family LLM judges.
• LeWiDi-2025 datasets + leaderboard: Four benchmark datasets with per-annotator labels plus standardized soft-label (Wasserstein/Manhattan) and perspectivist evaluation metrics, EMNLP 2025 shared task | https://le-wi-di.github.io/ | Off-the-shelf metrics and data format for evaluating disagreement-aware judge components and for the 'consistency-of-application vs agreement-with-truth' distinction.
• DICES dataset (Google): Diversity-in-Conversational-AI-Safety dataset: 990 conversations with dense parallel safety ratings (up to 100+ demographically diverse raters per item) | https://github.com/google-research-datasets/dices-dataset | Reference dataset for measuring how much rater variance survives on a judgment-laden axis with a fixed rubric - useful for calibrating the variance-attribution pilot.
• BeyondCorrelation (Amazon Science): Code for uncertainty-stratified meta-evaluation of automatic evaluators: perception charts, binned Jensen-Shannon divergence | https://github.com/amazon-science/BeyondCorrelation | Directly implements the recommended meta-evaluation reporting standard (stratify by human agreement) for certifying the AutoQA judge.
• Can Crowdsourcing Survive the LLM Era? (community survey): June 2026 arXiv survey (2606.04924) synthesizing prevalence, detection, and mitigation literature on LLM use in human data collection | https://arxiv.org/html/2606.04924v1 | Most current single map of the LLM-contamination literature; documents detector degradation under paraphrase and the state of mitigation practice as of mid-2026.

Disagreements

• Detectability of LLM-generated submissions: Veselovsky et al. (2023/2025) achieved usable classifier-based detection on a long-form text task, and TU Delft's prompt-injection work (TOCHI 2025) claims in-instrument countermeasures work; Westwood (PNAS 2025) and the community survey (arXiv 2606.04924, June 2026) show detection and quality checks are comprehensively evaded by agents and degrade under paraphrase/out-of-domain, with mixed human-AI authorship hardest. The recent evidence favors Westwood, but the sources genuinely conflict on whether content-side detection retains any value as a secondary signal.
• Nature of disagreement: the aggregation tradition (Dawid-Skene, MACE, majority vote) models disagreement as noise around one latent truth to be recovered; the perspectivist tradition (Aroyo & Welty, DICES, LeWiDi-2025, Diverging Preferences) treats much of it as legitimate signal to be preserved. Neither side claims universality, but they prescribe opposite data models (adjudicated gold vs. per-rater distributions) - the design must pick per-axis, and no source offers a validated automatic classifier for which regime an axis falls into.
• Agreement statistics under skew: Krippendorff defends alpha as the general-purpose reliability standard; Gwet and successors show alpha/kappa collapse under extreme prevalence and push AC1/AC2; Elangovan et al. (ICLR 2025) argue all human-reliability coefficients are misapplied to machine-vs-human comparison in the first place. There is no consensus single statistic - only convergence that one-number reporting is wrong.
• Mitigation of crowdworker LLM use: Veselovsky et al. measured that request+hurdle interventions halve usage (a meaningful effect worth deploying); Westwood concludes no instrument-level mitigation withstands agent-based completion ('no magical fix'). These are consistent about direction (partial reduction) but conflict sharply on whether prevention is worth its measured quality cost.

Gaps

• No published measurement of the false-agreement rate (judge and attempter agreeing while both wrong per expert panel) in an annotation-QA setting; the nearest proxy is Northcutt's 3.3%+ gold-label error floor. This number would have to be produced in-house via an independent expert-audit channel.
• No public 2025-2026 prevalence data for LLM use specifically among paid annotators/reviewers on managed AI-training-data platforms (Scale, Surge, Outlier, Prolific-managed RLHF work); all published prevalence numbers come from MTurk/Prolific survey and summarization tasks, which may under- or over-state prevalence for critique/rating work.
• Item-level (as opposed to worker-level) detection of LLM-generated annotations for short/low-dimensional outputs remains unsolved; the NeurIPS 2025 peer-prediction mechanism scores workers, not items, and requires overlapping assignments across workers.
• No annotation-science source found that gives sample-size arithmetic for certifying a judge against a gold set under low failure base rates (per-project vs pooled hierarchical validation); this appears only in general eval-statistics literature outside this domain's charter.
• No longitudinal studies found measuring whether QA feedback changes annotator behavior over time (repeat-error rates per criterion); the feedback-efficacy question (taxonomy Q7) has essentially no direct evidence in the annotation-quality literature.
• Could not open the full text of the Diverging Preferences taxonomy to extract exact percentage splits across its 10 disagreement categories (abstract confirms only that underspecification + style form the majority); exact fractions would sharpen the variance-attribution prior.

Verifications

• CLAIM: Roughly one-third of crowdworkers used LLMs on an LLM-advantaged text-production task (33-35% on Prolific, July 2023; ~34% Prolific self-report in a 2025 follow-up by Zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). VERDICT: partially_confirmed | Core claim VERIFIED against the full PDF of arXiv 2310.15683 (Veselovsky, Horta Ribeiro, Cozzolino, Gordon, Rothschild, West): Study 1, n=168 Prolific workers, 3 July 2023, medical-abstract summarization; prevalence 33.3% [25.9,40.1] (classify-and-count), 35.2% [29.8,40.6] (probabilistic), 35.4% [27.8,43.0] (corrected) - matches "33.3-35.4%, CIs ~[26%,43%]". Study 2, n=720, 23 July 2023, 3x3 factorial (request: none/indirect/direct x hurdle: none/image/no-Ctrl-C+V); classifier estimate dropped 27.6% -> 15.9% (direct+image) and 15.8% (direct+Ctrl C+V) - matches "roughly halved, not eliminated" (paper: "reduced LLM use by nearly 50%, it could not fully prevent it"). Direct request decreased keyword retention by 6.2% (p=0.009) - exact match. Classifier was finetuned e5-base-v2, calibrated (Card & Smith), with self-report and high-precision heuristics (completion time + paste artifacts) - matches. The June 2026 community survey arXiv 2606.04924 (Velutharambath et al., 155 researchers, survey run Aug 2025-Mar 2026) does cite "Veselovsky et al. estimate 30-40%", cites "Zhang et al. (2025): 34% of Prolific participants self-report using LLMs for open-ended questions", and states mitigations "reduce rather than eliminate LLM-assisted responses" - so the follow-up attribution to Zhang et al. 2025 is consistent with that survey's citations (note: I verified the citation exists in the survey, not the underlying Zhang et al. paper itself). ISSUES: (1) Date "2025-02-18" is wrong for the arXiv link given - arXiv has only v1, dated 24 Oct 2023; the published version is Communications of the ACM 68(3):42-47, March 2025 issue (DOI 10.1145/3685527). (2) "Preregistered" is not supported by the paper text - no preregistration is mentioned in the arXiv PDF. (3) Minor nuance: "never to zero" - by the classifier measure true, but the high-precision heuristics table actually shows 0.0% in the direct+Ctrl C+V cell; the paper's own framing ("cannot fully prevent") still supports the claim's spirit. (4) 44% of surveyed researchers in the 2026 survey observed LLM use in their data - newer corroborating, not superseding, evidence; no later work found that overturns the prevalence estimates. Sources: https://arxiv.org/abs/2310.15683, https://arxiv.org/abs/2606.04924, https://dl.acm.org/doi/10.1145/3685527. CORRECTED: Roughly one-third of Prolific crowdworkers used LLMs on an LLM-advantaged text-production task (33.3-35.4% across three estimators, July 2023, n=168; Veselovsky et al., arXiv 2310.15683 v1 posted 24 Oct 2023, published in Communications of the ACM 68(3), March 2025). In a second study (n=720, 3x3 factorial), the best mitigations (direct request + image presentation or + copy-paste disable) roughly halved classifier-estimated usage from 27.6% to 15.9%/15.8% without eliminating it, and directly requesting non-use reduced keyword retention by 6.2% (p=0.009). The studies were not described as preregistered. A June 2026 community survey of 155 researchers (arXiv 2606.04924, Velutharambath et al.) corroborates, citing Veselovsky et al.'s 30-40% estimate and Zhang et al. (2025)'s 34% Prolific self-report, and finding 44% of researchers observed LLM use in their crowdsourced free-text data; it echoes that mitigations reduce rather than eliminate LLM-assisted responses.
• CLAIM: An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. VERDICT: confirmed | Every quantitative element of the CLAIM verifies against the primary source (Westwood, "The potential existential threat of large language models to online survey research," PNAS 122(47), published Nov 20, 2025; PubMed 41264250, cited 81x by Jul 2026; won a 2025 Cozzarelli Prize). Extracted paper text confirms verbatim: (1) 99.8% pass rate across 6,000 trials (20 ACQ types x 300 repetitions), only 10 errors, perfect on 18/20 check types; (2) execution layer "simulates realistic reading times calibrated to the persona's education level, generates human-like mouse movements, and types open-ended responses keystroke-by-keystroke, complete with plausible typos and corrections"; (3) strategically declines 97.7% of "reverse shibboleth" tasks; (4) ~$0.05 per survey with commercial models vs $1.50 payment (>96.8% margin), marginal cost near zero with local open-weight models; (5) 43,800 total evaluations, 139 questions, 6,700 trials, with a key subset validated across nine LLMs (DeepSeek R1, Mistral Large, Claude 3.7 Sonnet, Grok 3, Gemini 2.5, Qwen Plus, Llama 4 Maverick, GPT-4.1, o4-mini); (6) 0% error rate on impossible-event/rare-event probes. The paper's abstract and press framing support the interpretive conclusion that coherence-/content-based detection is no longer tenable and defenses must be provenance/economic. Supersession check: no 2026 work refutes it - follow-ups (Westwood & Frederick 2026 Prolific audit; a 2026 SAGE AMPPS review of AI-mediated contamination) build on it and treat detection as an unsolved provenance problem. Two minor caveats, both in the EVIDENCE SUMMARY rather than the claim: (a) arXiv 2606.04924 ("Can Crowdsourcing Survive the LLM Era?", Jun 2026) is a community survey of 155 researchers' practices/experiences, not a technical detector benchmark; its abstract does not state the paraphrase/out-of-domain/mixed-authorship degradation findings attributed to it (those likely come from a different detection-benchmark paper). (b) On reCAPTCHA the paper says the system "is designed to accommodate tools for bypassing" reCAPTCHA - an architectural capability, slightly weaker than "reCAPTCHA... fail[s]" as an empirically demonstrated result. Neither caveat affects the claim text itself, which is fully accurate.
• CLAIM: Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. VERDICT: partially_confirmed | Substance CONFIRMED on every checked point against the arXiv record (https://arxiv.org/abs/2410.03775): (1) abstract states machine labels "may superficially appear to have similar or better correlation with the human majority" when human uncertainty is high, and machine-human correlation falls "well below" human-human correlation as human label consistency increases; (2) it flags Krippendorff's alpha / Randolph's kappa as designed for human-human reliability with assumptions inapplicable to machine labels; (3) proposed fixes match exactly - stratification by human label uncertainty, binned Jensen-Shannon divergence for perception-type tasks, and 'perception charts'; (4) venue is ICLR 2025 per the arXiv comments field; (5) code is open-sourced at github.com/amazon-science/BeyondCorrelation. The only inaccuracy is the SOURCE date: the claim says 2025-05-01, but the latest arXiv version is v3 dated 2025-01-27 (original submission 2024-10-03) - no May 2025 version exists on the record. Supersession check: a follow-up search surfaced related later work - "Validating LLM-as-a-Judge Systems under Rating Indeterminacy" (arXiv 2503.05965), "Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking" (arXiv 2604.11581), and "CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation" (arXiv 2603.00039, Feb 2026). These extend and reinforce the human-uncertainty/measurement-error critique rather than refute it; nothing found supersedes or contradicts the core finding. Verdict is partially_confirmed solely because of the incorrect source date; the claim itself stands. CORRECTED: Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. Source: Elangovan et al., "Beyond correlation..." (Amazon Science), accepted at ICLR 2025, arXiv 2410.03775 (v1 2024-10-03, latest v3 2025-01-27), code at github.com/amazon-science/BeyondCorrelation.