Domain Report - Critique Models

Domain: critique-models

Human+critic-model teams occupy a strictly better operating point than either alone: in OpenAI's CriticGPT study, Human+CriticGPT teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (FSBS length penalty), not a fixed property. Load-bearing

EVIDENCE: CriticGPT critiques were preferred over human contractor critiques in 63% of cases on naturally occurring LLM errors (>80% Elo win rate on inserted bugs); both ChatGPT and CriticGPT caught substantially more inserted bugs than paid contractors (median ~50 min/review, ~5 yrs Python experience). Humans alone had far fewer hallucinated bugs/nitpicks; Human+CriticGPT teams landed between, 'moving beyond the model-only Pareto frontier.' Force Sampling Beam Search scores candidates by rm_score + LENGTH_MODIFIER x num_highlights, tracing a precision/comprehensiveness Pareto curve selectable at deployment without retraining. Verified directly from the arXiv HTML full text. SOURCE: LLM Critics Help Catch LLM Bugs (CriticGPT, OpenAI) | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary IMPLICATION: Validates the hybrid one-touch foundation: AI critic drafts findings, human filters hallucinated nitpicks. Make the precision/comprehensiveness operating point an explicit per-project config knob (Q2/Q4/Q7 false-accusation SLO), since the same trained judge can be run strict or comprehensive per project.

Critic-assisted review of items already rated 'flawless' by a first human found real, rating-changing problems in 24% of flagged cases, versus 6% found by an unassisted second human review of the same data - a 4x lift on the exact 'confident pass' slice.

EVIDENCE: CriticGPT (RL-only) was run over ChatGPT training data rated flawless by a first annotator; where the critique flagged a problem, reviewers agreed it substantially lowered the rating in 24% of cases, vs 6% in a no-critique second-rater replication - and this held out-of-distribution (mostly non-code tasks). Prioritizing by critique reward-model score further raised catch rates and cut hallucinations. Verified from arXiv HTML full text. SOURCE: LLM Critics Help Catch LLM Bugs (CriticGPT, OpenAI) - 'flawless' data experiment | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary IMPLICATION: Direct evidence for Q1's false-agreement problem and Q6's routing: unassisted human re-review of passed items misses ~3/4 of what critic-assisted review catches, so the standing expert-audit channel over confident passes should itself be critic-assisted, and critic-confidence scores can prioritize the audit queue.

In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong. Load-bearing

EVIDENCE: 1,918 expert-labeled fact-verification items; tool-using AI rater with majority-vote-over-50-samples confidence (well calibrated). AI 87.7% vs individual humans 75.1% (majority vote 80.6%). Hybridization at threshold 0.62: 89.3%; on the routed low-confidence slice humans beat the AI 71.3% vs 60.5% (real complementarity region). Evidence-only assistance: 79.3% when AI right (vs 71.3% baseline) with no significant harm when AI wrong; conditions showing Judgments+Reasoning/Confidence produced significant over-reliance decrements (e.g. beta = -0.768, p < .001); AI-debate condition was numerically worst. Verified from arXiv HTML full text. SOURCE: Human-AI Complementarity: A Goal for Amplified Oversight (Google DeepMind) | https://arxiv.org/abs/2510.26518 | 2025-10-30 | primary IMPLICATION: Answers Q6 concretely: spend the one human touch on AI-low-confidence items, and give that human the judge's collected evidence spans WITHOUT the judge's verdict or rationale (blind-ish adjudication over curated evidence). Requires the judge to emit calibrated confidence and quoted evidence as first-class outputs (Q3), and argues against 'human verifies AI verdict' as the default interaction.

A Nature Human Behaviour meta-analysis of 100+ experiments (300+ effect sizes) found human-AI combinations on average performed significantly WORSE than the best of human or AI alone on decision/judgment tasks, with gains only where humans outperformed the AI solo - naive human-checks-AI designs destroy value.

EVIDENCE: Vaccaro, Almaatouq & Malone systematic review/meta-analysis: average human-AI combo underperformed the stronger party alone; losses concentrated in decision-making tasks, gains in content-creation tasks; combos helped when humans beat AI alone. DeepMind's complementarity paper explicitly frames its confidence-routing + evidence-only design as the answer to this result. Verified from arXiv abstract. SOURCE: When combinations of humans and AI are useful: A systematic review and meta-analysis | https://arxiv.org/abs/2405.06087 | 2024-10-28 | contrarian IMPLICATION: The Q6 fork ('is one shallow touch better than zero touch plus deep audits?') is empirically live: an automation-biased shallow verify step is the modal failure in the literature. The touch only pays where the human plausibly beats the judge (routed low-confidence/contested items), which mandates closed-loop routing on observed overturn rates rather than fixed per-item review.

In the largest deployed RCT of AI critiquing human evaluative work (ICLR 2025, feedback on >20,000 peer reviews), 27% of reviewers who received LLM feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated LLM reliability tests. Load-bearing

EVIDENCE: Randomized controlled study at ICLR 2025 (~22.5k feedback-arm vs ~22.4k control reviews): Review Feedback Agent flagged vague comments, content misunderstandings, unprofessional remarks; guardrail LLM tests gated delivery; 26.6-27% update rate, avg +80 words among updaters, blinded informativeness gains, longer author-reviewer discussions. Verified from arXiv abstract page; corroborated by the official ICLR blog. SOURCE: Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 | https://arxiv.org/abs/2504.09737 | 2025-04-13 | academic IMPLICATION: Q7's central bet is supported at deployment scale: constructive, item-specific AI feedback measurably changes the behavior of human evaluators, and the shipped pattern held feedback to its own automated verification gate before delivery (fail-closed feedback). Also gives a realistic effect-size prior: ~1 in 4 recipients act on feedback - plan longitudinal metrics around that base rate.

LLMs are structurally weak at detecting omissions: on AbsenceBench, average F1 drops 56.9 points versus detecting the same content as insertions (best model 71.2% overall, 40.0% on code diffs, vs ~99.5% needle-in-haystack), and inserting explicit placeholders at gap sites recovers ~35.7 points - absence has no attention key to attend to. Load-bearing

EVIDENCE: 4,302 instances across poetry, numerical sequences, GitHub PRs (~5K token contexts): models must list deliberately deleted elements given original+modified documents. Gemini-2.5-flash (thinking) best at 71.2% avg F1; Mixtral-8x7B scores 99%+ on NIAH but 14.7% here; thinking tokens buy only ~7.9%; '' placeholders add up to +81.8%. NeurIPS 2025 Datasets & Benchmarks. Verified from arXiv HTML full text. SOURCE: AbsenceBench: Language Models Can't Tell What's Missing | https://arxiv.org/abs/2506.11440 | 2025-06-13 | academic IMPLICATION: Caps what the judge can verify autonomously (Q3/Q5): 'comprehensiveness'/'complete' praise claims and omission-type attempter failures are the judge's structurally weakest lane. The rubric compiler should convert completeness criteria into explicit enumerable checklists (placeholder effect = give absence a token), and free-form omission judgment belongs in the human or human-assisted lane, not judge-autonomous.

Critique quality is itself quantifiable at usable reliability: MetaCritique decomposes critiques into atomic information units and scores precision/recall against references, with GPT-4 AIU-level judgments at 85-89% accuracy and Meta-F1 correlating with human gold at Pearson 0.84-0.89 - and measured baselines show human critiques are high-precision/low-recall (87.6%/48.7%) while LLM critiques are the inverse-ish (71.9%/53.3%).

EVIDENCE: ACL Findings 2024 framework: precision = fraction of critique AIUs judged factual; recall = coverage of reference-critique AIUs; each judgment carries a natural-language rationale. Choosing critiques by Meta-F1 yields refinements that win 51%/lose 27% by human eval, vs GPT-4 pairwise picks losing more than winning. ~28% of LLM-critique AIUs were non-factual vs ~12% for humans; LLM critiques carried 2.4x the information volume (8.10 vs 3.31 AIUs). Verified from arXiv HTML full text. SOURCE: The Critique of Critique (MetaCritique) | https://arxiv.org/abs/2401.04518 | 2024-01-09 | academic IMPLICATION: Q7's reflexive-grounding requirement is buildable today: run AutoQA feedback through AIU-level precision scoring as its own admission gate, and report critique-precision/recall as first-class system metrics. Also sets expectations: the AI side of the pipeline supplies coverage, the human side supplies precision - mirroring CriticGPT.

Verdict accuracy and critique validity dissociate badly: in CriticBench-THU data 24.8% of items got the correct verdict with a low-quality critique, and open-loop verdict-agreement metrics compress a 27.1-point real error-identification gap (ProcessBench: o1-mini 88.9 vs Qwen2.5-72B 61.8) into a 1.3-point verdict-F1 gap - so critique quality must be evaluated closed-loop by whether the critique drives a successful correction.

EVIDENCE: RealCritic (Jan 2025, arXiv 2501.14492): closed-loop critique-then-correct design over 8 reasoning benchmarks; classical LLMs (incl. GPT-4) lose accuracy under self-critique (-1.8 to -5.1 avg; up to -35.6 domain drops) while o1-mini is the only model with positive self-critique delta; failure modes 'superficial success' (right verdict, wrong analysis) and 'contradictory output' documented. Independently confirmed at scale in 2026: an ICML 2026 paper (RM-NLHF, arXiv 2601.07349) finds outcome-rewarded generative reward models 'guess correct outcomes without sound critiques' and uses similarity-to-human-critique as a process reward to fix it. RealCritic verified via arXiv abstract + HTML (HTML render partially draft-quality; abstract and appendix numbers consistent). SOURCE: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques | https://arxiv.org/abs/2501.14492 | 2025-01-24 | academic IMPLICATION: For Q1 meta-evaluation: never certify the AutoQA on verdict agreement alone - a judge can match pass/fail labels while its rationales are wrong, which poisons the feedback channel and attempter trust. Validation must score the rationale (does the cited evidence entail the finding; does acting on the feedback fix the item), i.e., Q7's explainability contract is also the correct validation instrument.

A June 2026 study of 21 LLM judges (~541k judgments) shows raw percent-agreement overstates chance-corrected agreement by 33.8-41.3 points (85% agreement = kappa ~0.48), judge rankings flip by up to 15 positions across benchmarks, and the most reproducible judges are among the least valid (test-retest 0.99 with position bias 0.19) - leading the authors to prescribe a pre-deployment Minimum Viable Validation Protocol.

EVIDENCE: 'Reliability without Validity' (UC Berkeley, arXiv 2606.19544, evals run March-April 2026): MT-Bench/JudgeBench/RewardBench, 118 runs at temperature 0; kappa deflation universal across all 21 judges; consistency-bias paradox instantiated by Qwen3-8B and Gemini 2.5 Flash; MVVP = report kappa/Krippendorff alpha as headline, AB+BA position swaps, >=3 replicates, >=2 benchmarks spanning preference- and correctness-style labels, and audit that high stability is not just high bias. Verified from arXiv HTML full text. SOURCE: Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic IMPLICATION: Directly answers Q1's statistics question (ban raw percent-agreement; standardize on chance-corrected stats) and Q5's ship-gate question (perturbation/position-swap + replicate harness is now published prescriptive practice). Run-to-run consistency must never be reported as evidence of validity - a perfectly consistent judge can be laundering a bias.

The closest production analog to the proposed AutoQA already exists: Toloka's deployed 'LLM QA' runs a tool-using agent per quality-metric on every human annotation submission, emits a strict three-way Pass / Fail / Unable-to-verify verdict (the abstain class exists specifically to prevent hallucinated verdicts), coaches annotators with Socratic feedback, and explicitly accepts lower precision on Fail because a false pass costs more than escalating a genuine pass.

EVIDENCE: Toloka engineering blog (bylined 2026-03-30): agentic autocheck with web/image/audio/Python/bash tools; 'one metric, one entity, one verdict' scoping (separate agent instance per criterion); internal benchmark of 300+ real submissions from 20+ live projects with senior-QA ground truth (numbers in an image, not extractable); reported constraint: performance 'entirely bounded by task design' - ambiguous guidelines produce walls of Unable-to-verify. Verified by opening the post. SOURCE: LLM QA: Scaling data quality assurance technologically (Toloka) | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | 2026-03-30 | practitioner IMPLICATION: Validates several foundational choices from production: criterion-scoped judge instances (Q3 decomposition at the per-axis level), a first-class abstain/escalate verdict distinct from pass/fail (Q2 verdict ontology), asymmetric error costs favoring false-flags over false-passes, and rubric quality as the binding constraint (Q4: tighten criteria before writing guidelines).

The foundational 2022 result behind the whole lineage: model-written critiques helped human evaluators find flaws in summaries they would otherwise have missed (including planted flaws in deliberately misleading human-written summaries), and models exhibit a discriminator-critique gap - they can often detect that something is wrong better than they can articulate why.

EVIDENCE: Saunders et al. (OpenAI, arXiv 2206.05802): topic-based summarization assistance experiments; critiques helped on both model- and human-written summaries; framework comparing generation, discrimination, and critique ability found 'even large models may still have relevant knowledge they cannot or do not articulate as critiques'; larger models critique and self-refine better. Verified from arXiv abstract. SOURCE: Self-critiquing models for assisting human evaluators (OpenAI) | https://arxiv.org/abs/2206.05802 | 2022-06-12 | primary IMPLICATION: The discriminator-critique gap argues for a two-stage judge: use cheap discrimination (scores/uncertainty) for routing and prioritization even where articulated critique is unreliable, and hold only the articulated-critique layer to the evidence-quoting bar (Q3/Q7). Judge-assisted human review of HUMAN-written work is validated at the root of this lineage, not just AI-output critique.

A deployed LLM compliance-checker was gamed by its subjects in a single shot: the NeurIPS 2024 author checklist assistant (234 papers) was rated useful by >70% of authors, but the organizers found the system 'not robust to gaming' by authors and concluded it is a poor substitute for human review.

EVIDENCE: Goldberg, Ullah, Guyon, Shah et al. (arXiv 2411.03417; NeurIPS blog Dec 2024): GPT-4-turbo checklist verification offered pre-submission; surveys plus qualitative analysis showed genuine improvements in some submissions alongside demonstrated manipulation of the assistant (authors could satisfy the checker without satisfying the standard); access was restricted to authors partly to avoid biasing review. Verified via arXiv abstract and organizer blog summary. SOURCE: Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment | https://arxiv.org/abs/2411.03417 | 2024-11-05 | academic IMPLICATION: Q8's decision-boundary-leak concern has deployment evidence even without repeated feedback cycles: pay/acceptance-motivated subjects will optimize against a visible checker. Defenses (rotating seeded probes, held-out human-graded golden items to detect AutoQA-vs-gold divergence, abstracted rather than checker-revealing feedback) must be designed in from day one, not retrofitted.

Tools & artifacts

• Review Feedback Agent: Open-source multi-LLM pipeline (zou-group/review_feedback_agent, CC BY 4.0 paper) that critiques human peer reviews for vagueness, content misunderstanding, and unprofessionalism, with automated LLM reliability tests gating feedback delivery. | https://github.com/zou-group/review_feedback_agent | Directly reusable architecture for the constructive-feedback channel: feedback taxonomy + fail-closed guardrail tests that feedback must pass before delivery (Q7).
• MetaCritique: Framework + dataset (GAIR-NLP/MetaCritique) scoring critique precision/recall/F1 over atomic information units with per-unit rationales. | https://github.com/GAIR-NLP/MetaCritique | Off-the-shelf machinery for holding AutoQA feedback to its own grounding standard and for meta-evaluating critique quality during validation (Q1, Q7).
• AbsenceBench: NeurIPS 2025 D&B benchmark (harvey-fin/absence-bench) measuring whether models can identify deliberately omitted content across poetry, sequences, and code diffs. | https://github.com/harvey-fin/absence-bench | Ready-made competence-limit test for the omission/comprehensiveness lane of any candidate judge model before assigning it completeness-verification authority (Q5 lane assignment).
• RealCritic: Closed-loop critique-evaluation benchmark (tangzhy/RealCritic): critique quality scored by correction success, with self-/cross-/iterative-critique modes. | https://github.com/tangzhy/RealCritic | Template for validating the AutoQA judge on rationale effectiveness rather than verdict agreement (Q1 meta-evaluation design).
• CTRL / Critique-RL / DeepCritic (2025 critic-training lineage): RL-trained critic models: CTRL (arXiv 2502.03492, critique via RL without human critique labels, up to 106% relative code-gen gains), Critique-RL (arXiv 2510.24320, two-stage RL: discriminability then helpfulness, no stronger supervisor), DeepCritic (arXiv 2505.00662, deliberate step-level math critique). | https://arxiv.org/abs/2502.03492 | State of the art for training a dedicated critic without the expensive expert-critique annotation dependency (Q4's aligned-expert concern); Critique-RL's discriminate-first-then-explain staging mirrors the discriminator-critique gap design.
• On scalable oversight with weak LLMs judging strong LLMs (Kenton et al., DeepMind, NeurIPS 2024): Protocol comparison (debate vs consultancy vs direct QA) with weaker judges overseeing stronger agents across QA/math/code/logic asymmetries. | https://arxiv.org/abs/2407.04622 | Evidence base for arbitration design (Q6): debate beats consultancy everywhere tested, and open debate makes a weak judge much harder to convince of a wrong answer - a candidate protocol for contested-item escalation.
• Minimum Viable Validation Protocol (MVVP): Five-step pre-deployment judge validation checklist from 'Reliability without Validity' (2026): chance-corrected headline stats, AB+BA position swaps, >=3 replicates, >=2 benchmark styles, consistency-bias paradox audit. | https://arxiv.org/abs/2606.19544 | Directly adoptable as the per-project judge-config ship gate the design taxonomy asks for (Q5).
• Toloka LLM QA: Production agentic QA system for human annotation submissions: per-metric tool-using agent instances, Pass/Fail/Unable-to-verify verdicts, Socratic annotator feedback. | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | The closest commercial deployment to the proposed AutoQA; its verdict ontology, criterion scoping, and guideline-quality constraint are field-tested reference points (Q2, Q4, Q6).

Disagreements

• Does adding a human to an AI judge help at all? Vaccaro et al. (Nature Human Behaviour 2024) find human-AI combos on average significantly WORSE than the best solo party on decision tasks, while CriticGPT (2024) finds Human+AI teams beyond the model-only Pareto frontier and DeepMind (2025) gets 91.3% hybrid vs 87.7% AI-alone. Reconciliation offered by DeepMind: complementarity only appears with confidence-based routing to a slice where humans genuinely beat the AI and with non-leading assistance formats - but the DeepMind-adjacent follow-up ('Toward Human-AI Complementarity Across Diverse Tasks') found baseline hybridization added only +0.4pp when model confidence failed to identify the complementarity region. The one-touch design's value is conditional, not guaranteed.
• What should the human be shown? DeepMind (2025) finds showing the AI's judgment/reasoning/confidence causes over-reliance and that evidence-only assistance is the only safe format, whereas CriticGPT's protocol and the ICLR 2025 feedback agent delivered full natural-language critiques and reported net gains. Task framing differs (assisting a second-party judge vs coaching the first-party author), which suggests the AutoQA should show full critiques to ATTEMPTERS but evidence-only to human ADJUDICATORS - no single source tests both arms.
• Are LLM critics better or worse than humans? CriticGPT: model critiques preferred 63% and catch more bugs than paid contractors. MetaCritique: human critiques have much higher factual precision (87.6% vs 71.9% of atomic units factual) while LLM critiques have higher coverage. Both true simultaneously - the disagreement is about which axis (precision vs comprehensiveness) a QA system should privilege, and sources implicitly pick opposite defaults.
• Can critics increase comprehensiveness of review? CriticGPT/Saunders show critics make human review MORE comprehensive (catching missed flaws), but AbsenceBench (2025) shows LLMs are structurally near-blind to omissions (56.9 F1 drop). Reconciliation: the critic gains are demonstrated on commission-type errors (inserted/tampered bugs, incorrect statements); no source demonstrates critic gains on omission-type attempter failures, and AbsenceBench predicts they won't appear without checklist/placeholder scaffolding.
• What metric certifies a judge? Open-loop verdict-agreement benchmarks (CriticBench 2024, CriticEval 2024, MT-Bench-style agreement) vs RealCritic's closed-loop position that verdict accuracy without correction-effectiveness is 'superficial success' (24.8% right-verdict-wrong-critique), reinforced by RM-NLHF (ICML 2026) and by 'Reliability without Validity' (2026) showing judge rankings flip up to 15 places depending on which benchmark style you trust.

Gaps

• No public post-deployment report from OpenAI on CriticGPT's integration into the production RLHF labeling pipeline (announced intent, June 2024) - no follow-up metrics, and no officially branded successor found through July 2026; the lineage continued in academic work (CTRL, DeepCritic, Critique-RL, MultiCritique) rather than vendor deployment reports.
• No published RCT of AI critique/feedback specifically on human DATA-ANNOTATION QA work. Closest analogs are peer review (ICLR 2025) and fact-verification rater assistance (DeepMind 2025). No longitudinal data anywhere on repeat-error-rate reduction per annotator under sustained AI feedback - Q7's efficacy metric has no published precedent to calibrate against.
• No measured multi-cycle adversarial study of annotators gaming a deployed critic (Q8's decision-boundary leakage): the NeurIPS'24 checklist gaming is single-shot and author-side; citation-theater/evidence-swap perturbation tests on critics were not found.
• Toloka's deployment report - the single closest production analog - publishes its benchmark design (300+ submissions, 20+ projects) but its quantitative precision/recall table is embedded in an image and was not extractable; no independent audit of any commercial annotation-QA LLM system was found.
• False-agreement rate (judge and attempter both wrong vs expert panel) is unmeasured in the literature; the CriticGPT 24%-vs-6% flawless-data result is the nearest proxy and is one study, one domain (mostly), one vendor.
• Positive-claim/praise verification (is 'accurate/complete/clear' praise warranted?) has no dedicated benchmark or precision/recall measurements - MetaCritique-style AIU machinery has only been applied to flaw-finding critiques. This is a genuine hole given the mission's requirement to evaluate positive statements.
• Semantic Scholar returned HTTP 429 during literature-graph search (OpenAlex/Crossref coverage only for that call); one WebSearch call was flagged by a per-call safeguard and was successfully re-run via Exa - no coverage lost, noted for provenance.

Verifications

• CLAIM: Human+critic-model teams occupy a strictly better operating point than either alone: in OpenAI's CriticGPT study, Human+CriticGPT teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (FSBS length penalty), not a fixed property. VERDICT: confirmed | Source verified: arXiv:2407.00215 "LLM Critics Help Catch LLM Bugs" (McAleese et al., OpenAI), submitted 2024-06-28 - date matches. Abstract directly confirms: 63% preference for model critiques on naturally occurring LLM errors; models catch more bugs than paid human reviewers; human-machine teams catch similar bug counts to LLM critics while hallucinating less than LLMs alone. Full-text details corroborated via independent secondary coverage: critic-assisted contractors wrote more comprehensive critiques than unassisted contractors while reducing hallucination/nitpick rate relative to the model, described as moving beyond the model-only Pareto frontier; FSBS scores candidates by rm_score + LENGTH_MODIFIER x num_highlights, giving a deployment-time precision/comprehensiveness dial without retraining. Minor reading note: "strictly better than either alone" holds in the paper's Pareto sense - teams matched (did not exceed) the model's bug-catch/comprehensiveness while beating its hallucination rate, and exceeded human comprehensiveness; the claim's own wording states exactly this, so no correction needed. Supersession check: no official OpenAI follow-up found; later work (CodeCriticBench 2025; 2026 papers on LLM reviewer overcorrection) extends but does not contradict these findings.
• CLAIM: In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong. VERDICT: confirmed | Every quantitative element checks out against the arXiv HTML full text (2510.26518). Dataset: 1,918 expert-labeled tuples - confirmed. AI rater 87.7%, individual humans 75.1%, human majority vote 80.6% - confirmed. Confidence via 50 samples/sentence (avg 33.25 passing format check), described as fairly well calibrated - confirmed. Unassisted hybridization at threshold T=0.62: 89.3% (beta=0.413, p=.012) - confirmed. Low-confidence routed slice (280 items): humans 71.3% vs AI 60.5% (p=.006) - confirmed. The headline 91.3% is specifically the ASSISTED hybrid (AI when confident + evidence-assisted humans on the low-confidence slice), vs 89.3% unassisted hybrid - exactly as the claim words it, so 91.3% is correctly attributed. Evidence-only assistance: 79.3% vs 71.3% baseline when AI correct (beta=0.446, p=.009), no significant harm when AI wrong (64.0% vs 61.5%); paper explicitly calls it "the only form of assistance that achieves the ideal of helping when correct and not hurting when wrong" - confirmed. Over-reliance when AI wrong: Evidence&Reasoning&Judgments beta=-0.768 (SE=0.174, z=-4.413, p<.001), ER&J&Confidence beta=-0.636 p<.001, Judgments&Confidence beta=-0.356 p=.042 - the quoted beta=-0.768 matches the Evidence&Reasoning&Judgments condition exactly. Debate was numerically worst (64.9%, only format below baseline) - confirmed. Date: v1 submitted 2025-10-30 as claimed. Currency: a v2 was posted 2026-06-25 and the paper was published at ACM FAccT '26 (DOI 10.1145/3805689.3812308); this is publication/revision, not supersession - no later work contradicting the findings found. DeepMind blog notes ongoing follow-up combining hybridization+assistance on an internal rating task, unpublished as of 2026-07-14. Minor caveat only: whoever cites this should prefer the v2/FAccT version in case numbers shifted slightly in revision (v1 numbers verified here).
• CLAIM: In the largest deployed RCT of AI critiquing human evaluative work (ICLR 2025, feedback on >20,000 peer reviews), 27% of reviewers who received LLM feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated LLM reliability tests. VERDICT: confirmed | Source verified directly (arXiv:2504.09737, v1 submitted 2025-04-13 - date matches). Abstract states: Review Feedback Agent deployed at ICLR 2025 as a large randomized controlled study on >20,000 randomly selected reviews; feedback targeted vague comments, content misunderstandings, unprofessional remarks; 27% of reviewers who received feedback updated their reviews; >12,000 suggestions incorporated (12,222 in the paper); +80 words average among updaters; more informative per blinded evaluators; longer author-reviewer discussions; feedback delivered only if it passed a suite of automated LLM reliability tests. Arm sizes corroborated from paper figures: 22,467 selected for feedback vs 22,364 control; 18,946 successfully received feedback, of whom 26.6% updated (the abstract rounds to 27%). Two nuances, neither contradicting the claim: (1) the superlative "largest deployed RCT of AI critiquing human evaluative work" is the researcher's framing, not verbatim in the abstract - no larger counterexample found, and it is consistent with the study's scale; (2) not superseded but now formally published: Nature Machine Intelligence, Feb 23, 2026, as "A large-scale randomized study of large language model feedback in peer review" (https://www.nature.com/articles/s42256-026-01188-x) - the NMI version is the preferred citation going forward. A related follow-up mixed-methods study on reviewer perceptions exists (arXiv:2602.13817) but does not alter these results.