Domain Report - Hybrid Statistical Methods

Domain: hybrid-statistical

Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a GPT-4 judge in 'AutoEval Done Right'. Load-bearing

EVIDENCE: arXiv:2403.07008 (Boyeau, Angelopoulos, Yosef, Malik, Jordan, UC Berkeley) abstract confirmed: 'increase the effective human-labeled sample size by up to 50% on experiments with GPT-4'; v3 camera-ready posted 2026-06-01. The unconditional-coverage property is restated and empirically demonstrated in the GLIDE paper (arXiv:2605.31278) and is the foundation of the entire 2026 LLM-judge debiasing literature (PRECISE/AAAI 2026, arXiv:2606.05308 ranking extension). SOURCE: AutoEval Done Right: Sample-Efficient Human Evaluation via Prediction-Powered Inference | https://arxiv.org/abs/2403.07008 | 2024-03 (v3 camera-ready 2026-06-01) | academic IMPLICATION: The AutoQA meta-evaluation layer should treat human gold labels as bias-correctors for judge-derived population metrics (pass rates, per-attempter/per-criterion error rates), never report raw judge numbers. Because validity survives a bad judge, pooled/hierarchical cross-project validation is statistically safe even where per-project gold sets are tiny - directly resolving taxonomy Q1's gold-set-arithmetic fork toward pooled certification with per-project bias correction.

Cascaded Selective Evaluation (Trust or Escalate, ICLR 2025 Oral) delivers a provable, user-specified human-agreement guarantee for LLM judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a Chatbot Arena subset where GPT-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models. Load-bearing

EVIDENCE: arXiv:2407.18370 abstract fetched directly (80% agreement / ~80% coverage claim confirmed); ICLR 2025 Oral status confirmed at iclr.cc/virtual/2025/oral/31838; fixed-sequence-testing calibration mechanism confirmed via OpenReview PDF snippets and secondary reviews (83 citations as of 2026-07). Its 'Simulated Annotators' method (in-context simulation of diverse annotators) is the confidence estimator that makes the guarantee achievable at high coverage. SOURCE: Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement (ICLR 2025 Oral) | https://arxiv.org/abs/2407.18370 | 2024-07-25 (ICLR 2025-04) | academic IMPLICATION: This is the direct architectural blueprint for 'which items get the single human touch': the human queue is exactly the abstention set of a confidence-calibrated cascade, and the abstention rate is a tunable dial trading human budget against a guaranteed agreement level. Caveat for taxonomy Q1: the guarantee is agreement-with-human-majority, so this component's validity is capped at human panel reliability - it certifies consistency-of-application, not truth.

The minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n(1-rho^2) where n is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with R^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes. Load-bearing

EVIDENCE: arXiv:2605.16354v1 (Jane Paik Kim, Stanford Psychiatry, 2026-05-08) HTML fetched: derives sample-size formulas from the doubly-robust estimator's asymptotic variance; worked examples confirmed (n*=200, R^2=0.7, N=2000 -> 65 human labels; N=400 -> 100). Explicit caveats: pilot overestimation of R^2 breaks precision guarantees (use conservative values), framework assumes human ratings are a reliable gold standard, single-rater scope. SOURCE: Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need? | https://arxiv.org/html/2605.16354v1 | 2026-05-08 | academic IMPLICATION: Answers taxonomy Q1's gold-set question with arithmetic instead of convention: run a small pilot to estimate per-axis judge-human correlation, then size each project's gold set from the formula with a conservative correlation estimate. Because the floor depends on rho, per-axis correlation heterogeneity argues for stratified gold sampling by criterion, and low-rho axes are where gold budget must concentrate.

Using the LLM's own confidence signals to choose WHICH items receive human annotation (Confidence-Driven Inference, building on Active Statistical Inference) cut required human annotations by >25% in all three tested settings while keeping provably valid confidence intervals that remain safe even if the LLM annotations are poor - but a 2026 ICLR paper finds the opposite in the sequential regime, where near-uniform sampling at the budget ceiling beat uncertainty-driven querying. Load-bearing

EVIDENCE: arXiv:2408.15204 (Gligoric, Zrnic, Lee, Candes, Jurafsky; NAACL 2025) abstract fetched: '>25% reduction in each of three settings' (politeness, stance, bias) with validity guarantees regardless of LLM quality. Foundation: Active Statistical Inference (Zrnic & Candes, ICML 2024, arXiv:2403.03208): label where the model is uncertain, provably valid CIs, fewer labels than uniform. Counter-evidence verified separately (see disagreements): arXiv:2604.18569 (ICLR 2026). SOURCE: Can Unconfident LLM Annotations Be Used for Confident Conclusions? (NAACL 2025) + Active Statistical Inference (ICML 2024) | https://arxiv.org/abs/2408.15204 | 2024-08-27 (v2 2025-02-08; NAACL 2025-04) | academic IMPLICATION: Uncertainty-routed allocation of the scarce human touch is the statistically defensible default and comes with a safety net (validity even when the judge is wrong), but the gain over random sampling is regime-dependent and contested - so the foundation should make the routing policy (uncertainty-weighted vs uniform vs stratified) a measured, per-project choice with a built-in A/B against uniform, not a hard-wired assumption.

GLIDE (May 2026) is an open-source industrial library unifying PPI++, stratified PPI, predict-then-debias bootstrap, and active inference with a decision tree for method selection, and it quantifies when PPI helps: effective-sample gain is ~1.0x at judge-human correlation rho=0.1, ~2.2x at rho=0.9; CLT-based estimators need roughly >=50 human labels per stratum (below that use bootstrap PTD); in a real agent-safety case (rho=0.59, ~13-point judge bias) 100 human labels became worth 143-157.

EVIDENCE: arXiv:2605.31278v2 HTML fetched (Martinon, Merad, Raki; Emerton Data): Monte Carlo with 500 human + 1000 proxy labels shows effective n ~500 at rho=0.1 vs ~1100 at rho=0.9; R-Judge case study with Claude Sonnet as judge; library at github.com/EmertonData/glide with reproducible validation notebooks; limitations: means/proportions only, i.i.d. assumptions, no anytime-valid inference. SOURCE: Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation | https://arxiv.org/html/2605.31278v2 | 2026-05 (v2 2026-06) | practitioner IMPLICATION: Ready-made tooling exists for the statistical layer; more importantly it publishes the correlation-vs-benefit curve and the >=50-labels-per-stratum threshold the foundation can budget against: if a project's judge-human correlation on an axis is below ~0.3, PPI buys nothing there and gold labels must carry that axis alone.

For estimating scores from noisy judges, PPI++/efficient-influence-function estimators produce near-identical, shortest confidence intervals - reported 3-15x narrower than Rogan-Gladen-style misclassification-correction estimators depending on the human-labeling ratio, with the advantage largest when the judge is close to random guessing.

EVIDENCE: arXiv:2601.05420 (Chen, Lu, Li, Guo, Li; 2026-01-08) abstract fetched: unifies measurement-error correction and PPI classes via semiparametric efficiency theory, derives EIF-based estimators, characterizes when PPI-style strictly dominates; the 3-15x interval-width figure comes from the paper's empirical section as surfaced in search-indexed full text (not independently recomputed). Code: github.com/yiqunchen/debias-llm-as-a-judge. SOURCE: Efficient Inference for Noisy LLM-as-a-Judge Evaluation | https://arxiv.org/abs/2601.05420 | 2026-01-08 | academic IMPLICATION: Standardize the estimation layer on PPI++/EIF-style residual calibration rather than sensitivity/specificity (confusion-matrix) correction when the target is a rate or mean - but note the certification literature pushes the other way for hypothesis tests (see disagreements).

For certifying that a failure rate is below a threshold (the low-base-rate pass/fail regime), the 'Noisy but Valid' framework (ICLR 2026) uses a small human calibration set to estimate judge TPR/FPR, applies a variance-corrected test to the large judge-labeled stream with finite-sample Type-I error control, and derives exact conditions under which judge-based testing has HIGHER statistical power than direct human evaluation.

EVIDENCE: arXiv:2601.20913 (Feng, Shen, Balashankar, Gerner-Beuerle, Rodrigues; submitted 2026-01-28, accepted ICLR 2026 per arXiv page) abstract fetched: variance-corrected critical threshold, finite-sample Type-I control under calibration uncertainty, validation on Jigsaw/Hate Speech/SafeRLHF, and a quantified 'oracle gap' - the power cost of having to estimate judge reliability rather than knowing it. SOURCE: Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges | https://arxiv.org/abs/2601.20913 | 2026-01-28 | academic IMPLICATION: For attempter-level or batch-level certification ('this attempter's error rate is below X'), a judge-plus-small-calibration design can be provably MORE powerful than human-only review - reframing the human panel's job as estimating judge error rates, not per-item adjudication. This is direct evidence for the zero-per-item-touch + calibration-audit branch of taxonomy Q6's fork.

Naive reporting of judge-scored evaluations is systematically biased by judge sensitivity/specificity; a plug-in correction with confidence intervals propagating BOTH test-set and calibration-set uncertainty fixes it, remains unbiased under distribution shift between calibration and test data, includes adaptive allocation of calibration labels, and characterizes regimes where corrected judge-based evaluation is more reliable than human-only evaluation.

EVIDENCE: arXiv:2511.21140 (Lee, Zeng, Jeong, Sohn, Kangwook Lee; submitted 2025-11-26, v4 2026-05-31, accepted ICML 2026 per arXiv page) abstract fetched. The calibration-to-test distribution-shift robustness is the distinctive property versus standard PPI's i.i.d. assumption. SOURCE: How to Correctly Report LLM-as-a-Judge Evaluations | https://arxiv.org/abs/2511.21140 | 2025-11-26 (v4 2026-05-31) | academic IMPLICATION: Supports the taxonomy Q1 proposal to ban raw judge-agreement numbers from all reporting: every reported metric should be the corrected estimator with a CI that includes calibration uncertainty. The shift-robustness result matters specifically because attempter adaptation (taxonomy Q8) shifts the live item distribution away from the calibration set over time.

Conformal prediction now provides per-item uncertainty for LLM-as-judge ratings: EMNLP 2025 work builds guaranteed-coverage score intervals from a single judging run (with an ordinal boundary adjustment and a lower-bias midpoint score), and April 2026 work shows conformal set width is a genuine per-instance reliability signal (r_s=+0.576 with reliability, n=1,918; widths correlate ~0.32-0.38 across different judges) with reliability driven more by the CRITERION than the judge model (relevance avg set size ~3.0 vs fluency/consistency ~4.9 on a 1-5 scale).

EVIDENCE: Sheng, Liu, He, Zhao, Kang, 'Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction', EMNLP 2025 main (aclanthology.org/2025.emnlp-main.569), abstract confirmed via NeurIPS 2025 workshop listing. Gupta & Kumar, arXiv:2604.15302 (2026-04-16, under review): split conformal on SummEval; also finds 33-67% of documents contain intransitive judge preference cycles despite low aggregate violation rates. SOURCE: Analyzing Uncertainty of LLM-as-a-Judge (EMNLP 2025) + Diagnosing LLM Judge Reliability (arXiv:2604.15302) | https://arxiv.org/abs/2604.15302 | 2025-11 (EMNLP) / 2026-04-16 | academic IMPLICATION: Conformal set width is a cheap, distribution-free routing statistic for the human-touch queue that transfers across judges, and the criterion-dominates-judge finding is empirical support for taxonomy Q2/Q5's axis-triage: per-criterion reliability ceilings must be measured and some axes routed human-only regardless of judge choice.

Judge-vs-system drift can be attributed with anytime-valid statistics: a frozen human-labeled anchor set periodically re-scored by the live judge, monitored with betting e-processes, correctly attributed a silent judge version bump in 60/60 runs with zero misattribution (rolling z-test baseline false-alarmed on 75% of drift-free streams), at 0.21-0.64x the cost of strong-judging every item.

EVIDENCE: arXiv:2606.15474 (Yitao Li, 2026-06-13) abstract fetched: 'one-way identification (only the judge can move the anchors)'; three-verdict output (none/system/judge); strict-prompt change attributed in 110/120 runs; replicated on TL;DR summarization 240/240. Single-author preprint, not yet peer-reviewed - treat quantitative claims as provisional. SOURCE: Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines | https://arxiv.org/abs/2606.15474 | 2026-06-13 | academic IMPLICATION: Taxonomy Q9's three-moving-parts problem has an existing statistical design for one axis: freeze a per-project human-labeled anchor set at onboarding, interleave it continuously into the judge stream, and use e-process alarms to separate 'judge changed' from 'attempter population changed' before triggering rubric revision or de-graduation.

In sequential prediction-powered mean estimation, uncertainty-based active querying contributed little: the smallest confidence widths occurred when query probabilities were near-constant at the budget ceiling, and theory shows optimized query probabilities converge to the maximum allowed constant rate when chosen without reference to covariates.

EVIDENCE: arXiv:2604.18569 (Sfyraki & Wang, submitted 2026-04-20, accepted ICLR 2026 per arXiv page) abstract fetched: empirical finding that the uncertainty-blend weight near the constant term minimizes CI width, plus non-asymptotic analysis and no-regret query-probability selection corroborating near-uniform optimality in this regime. SOURCE: Revisiting Active Sequential Prediction-Powered Mean Estimation | https://arxiv.org/abs/2604.18569 | 2026-04-20 | contrarian IMPLICATION: Direct counter-evidence to uncertainty-routed human sampling for population estimation in streaming settings: the foundation must not assume uncertainty routing dominates - measure the routing policy's marginal value per project, and expect uniform-at-budget to be near-optimal when the goal is estimation rather than per-item verdict quality.

Commercial tooling for confidence-scored LLM judgments with human calibration is mature: Cleanlab's Trustworthy Language Model attaches a real-time trustworthiness score to every LLM response, supports custom evaluation criteria, explicitly documents calibrating trust scores against human quality ratings, and publishes benchmarks (Nov 2025) for auto-flagging incorrect structured outputs for human review.

EVIDENCE: Cleanlab TLM documentation (help.cleanlab.ai/tlm/: quickstart, custom-eval calibration-against-human-ratings tutorial, advanced usage with low-score explanations) and 2025-11-18 structured-outputs benchmark blog post, all fetched via search with page content confirmed. Vendor source: benchmark numbers are self-reported. SOURCE: Cleanlab Trustworthy Language Model (TLM) documentation and benchmark | https://help.cleanlab.ai/tlm/ | 2025-11-18 (benchmark); docs current 2026 | practitioner IMPLICATION: The 'score every judgment, calibrate the score against a human-rated set, route low-trust items to humans' pattern is already productized - the foundation can specify this as a component contract (per-verdict trust score + human-calibration hook) rather than inventing it, while keeping vendor-neutrality.

Tools & artifacts

• ppi_py: Reference open-source Python library for prediction-powered inference (Angelopoulos, Zrnic et al.): means, quantiles, GLMs, PPI++ power tuning. | https://github.com/aangelopoulos/ppi_py | Drop-in statistical layer for computing bias-corrected pass rates and attempter metrics from many judge labels + few human labels; covers estimand types GLIDE does not (quantiles, regressions).
• GLIDE: Open-source industrial PPI library (Emerton Data, May 2026): PPI++, stratified PPI++, predict-then-debias bootstrap, active statistical inference, uniform/stratified/active/cost-optimal samplers, plus a method-selection decision tree. | https://github.com/EmertonData/glide | Closest existing implementation of this domain's whole playbook, including the sampler that decides which items get human labels and effective-sample-size ROI reporting; validated on agent-safety trajectories with a Claude judge.
• active-inference: Code for Active Statistical Inference (Zrnic & Candes, ICML 2024): uncertainty-guided label collection with provably valid confidence intervals. | https://github.com/tijana-zrnic/active-inference | Reference implementation for uncertainty-routed allocation of the scarce human annotation budget.
• debias-llm-as-a-judge: Code for Chen et al. 2026 (arXiv:2601.05420): efficient-influence-function and PPI++ estimators for noisy LLM-judge evaluation of scores and win rates. | https://github.com/yiqunchen/debias-llm-as-a-judge | Implements the estimator class this domain recommends standardizing on for judge-score estimation, with the Rogan-Gladen comparison built in.
• Cleanlab Trustworthy Language Model (TLM): Commercial system that attaches a calibratable trustworthiness score to every LLM response, with custom evaluation criteria and tutorials for calibrating scores against human ratings (benchmark published 2025-11). | https://help.cleanlab.ai/tlm/ | Productized version of the per-verdict-confidence + human-calibration + route-low-trust-to-human pattern; useful as a component contract template even if not adopted as a vendor.
• Cascaded Selective Evaluation (Trust or Escalate): ICLR 2025 Oral method: judge cascade with fixed-sequence-testing calibration delivering user-specified human-agreement guarantees; Simulated Annotators confidence estimation. | https://arxiv.org/abs/2407.18370 | The architectural template for the AutoQA's escalation spine: guarantee level and abstention rate become the two governing dials for the human-touch budget.

Disagreements

• Uncertainty-routed vs uniform human sampling: Zrnic & Candes (ICML 2024) and Gligoric et al. (NAACL 2025) report provably valid inference with >25% fewer human labels when the model's uncertainty guides which items humans label; Sfyraki & Wang (ICLR 2026, arXiv:2604.18569) find in the sequential regime that the uncertainty component contributes little and near-constant query probabilities at the budget ceiling give the tightest intervals. Both sides are peer-reviewed; the honest read is that active-sampling gains are regime-dependent (batch design + informative confidence signals vs sequential streaming), so routing policy must be validated per project, not assumed.
• Explicit judge-error modeling vs black-box residual calibration: Chen et al. (arXiv:2601.05420) show Rogan-Gladen-style TPR/FPR-correction estimators yield intervals reportedly 3-15x wider than PPI++/EIF for score ESTIMATION; Feng et al. (ICLR 2026, arXiv:2601.20913) argue the opposite direction for CERTIFICATION - explicitly modeling judge TPR/FPR yields better-powered hypothesis tests than treating PPI as a black box. Partly reconcilable (different targets: estimation vs testing), but they issue opposing default recommendations and the synthesis should pick per use-case: PPI++/EIF for dashboards and rates, TPR/FPR-corrected tests for threshold certification.
• What the human labels are for - target vs corrector: Trust or Escalate guarantees agreement with human majority preference (capping the system's validity at human panel reliability, i.e., consistency-of-application); the PPI/certification line treats human labels as gold-standard truth for bias correction, and both 'Noisy but Valid' and 'How to Correctly Report' prove regimes where corrected judge-based evaluation is strictly MORE reliable/powerful than human-only evaluation, while Kim (arXiv:2605.16354) explicitly warns her framework breaks when human raters are themselves unreliable. No paper in this literature resolves what to do when the gold itself is noisy - the taxonomy Q1 construct choice (instruction-satisfaction vs consensus-prediction) is genuinely open in the statistics literature.

Gaps

• No 2025-2026 paper found that applies PPI, selective escalation, or conformal routing specifically to QA of HUMAN-produced annotations (reviewing human attempters); every application evaluates model outputs with humans as gold. Transfer is plausible but unvalidated, especially under adversarial attempter adaptation, which violates the i.i.d. assumptions most PPI variants rely on (only arXiv:2511.21140 claims calibration-to-test shift robustness).
• All PPI-family guarantees are population-level (rates, means, rankings) and selective-evaluation guarantees are marginal, not conditional: a system can meet its aggregate agreement/coverage target while concentrating its failures on specific attempters, criteria, or hard-item subgroups. Conditionally valid (per-subgroup) escalation guarantees for LLM judges remain an open problem in this literature.
• The specific fork 'one shallow human touch per item vs zero-touch plus randomized deep audits' has no direct head-to-head study measuring total error caught per human-hour; the statistical literature implicitly favors the zero-touch+audit side (valid conclusions with human labels on a small fraction of items) but only for measurement/certification objectives, not for per-item verdict correction or attempter coaching.
• Calibration economics for rubric-based QA: Trust or Escalate's Simulated Annotators and fixed-sequence testing were validated on pairwise preference tasks; required calibration-set sizes and achievable coverage for per-criterion rubric verdicts (the AutoQA's actual workload) are not established anywhere I found.
• Could not open the Trust or Escalate OpenReview forum (CAPTCHA); handling of abstained items and calibration-set-size details confirmed only from abstract-level and secondary sources, not reviewer discussion. The 3-15x interval-width figure in Chen et al. was corroborated from search-indexed paper text, not independently recomputed.
• No published head-to-head comparison of conformal-width routing vs Simulated-Annotators-style confidence routing vs verbalized-confidence routing for deciding the human-escalation queue - the three candidate mechanisms have never been benchmarked against each other.

Verifications

• CLAIM: Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a GPT-4 judge in 'AutoEval Done Right'. VERDICT: partially_confirmed | Verified directly against arXiv:2403.07008: authors (Boyeau, Angelopoulos, Yosef, Malik, Jordan) correct; abstract verbatim contains 'increase the effective human-labeled sample size by up to 50% on experiments with GPT-4'; abstract claims methods 'improve sample efficiency while remaining unbiased'; version history confirms v1 2024-03-09 and v3 2026-06-01 with comment 'camera-ready paper version' - dates as claimed. GLIDE (arXiv:2605.31278, Martinon/Merad/Raki, v1 2026-05-29, ICML 2026 workshop) exists and states PPI 'combines both into debiased estimates with valid confidence intervals', consistent with the claim, though the exact phrase 'regardless of judge quality' was not verifiable in its abstract. The 2026 PPI-for-LLM-judge literature is real and active (PRECISE/Amazon Science, arXiv:2606.05308 ranking extension, arXiv:2601.05420, arXiv:2601.20913), so the claim is not superseded - it is being extended. Two corrections: (1) 'always-valid confidence intervals' is a misuse of a term of art - 'always-valid'/anytime-valid refers to sequential inference; PPI/PPI++ intervals are fixed-n ASYMPTOTIC (CLT-based) intervals, so coverage is guaranteed only asymptotically and finite-sample coverage can degrade with very small gold sets. (2) The judge-quality-independence property is a general PPI/PPI++ property (Angelopoulos et al. 2023) rather than something stated in the AutoEval abstract itself; the claim's framing correctly captures its substance (worse judge -> wider intervals via lower correlation, coverage preserved) but slightly overstates its strength and its provenance in this specific source. CORRECTED: Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with asymptotically valid confidence intervals whose coverage does not depend on judge accuracy (a worse judge widens intervals rather than breaking coverage, subject to standard asymptotic/regularity conditions and a sufficiently large gold set); 'AutoEval Done Right' (arXiv:2403.07008, Boyeau, Angelopoulos, Yosef, Malik & Jordan) reports increasing the effective human-labeled sample size by up to 50% in experiments with GPT-4.
• CLAIM: Cascaded Selective Evaluation (Trust or Escalate, ICLR 2025 Oral) delivers a provable, user-specified human-agreement guarantee for LLM judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a Chatbot Arena subset where GPT-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models. VERDICT: confirmed | All load-bearing elements check out against the primary sources. (1) arXiv:2407.18370 abstract (fetched directly) states Cascaded Selective Evaluation provides a provable human-agreement guarantee at a user-specified level, and that on a Chatbot Arena subset "where GPT-4 almost never achieves 80% human agreement," the method "guarantees over 80% human agreement with almost 80% test coverage" even using Mistral-7B. (2) Paper body (arxiv.org/html/2407.18370v1) confirms the mechanism: threshold calibration via fixed sequence testing (Bauer, 1991) on a small human calibration set (|D_cal|=500 for ChatArena/TL;DR, 392 for Auto-J), cascade escalation from cheap to strong judges on low confidence, abstention (output empty set) when even the strongest judge is unconfident, and Simulated Annotators as the confidence estimator enabling high coverage. (3) The ~88% figure is exact: "79.1% of all samples, among which 88.1% are evaluated by substantially cheaper Mistral-7B or GPT-3.5 instead of GPT-4" (Table 3: GPT-4 handles only 17.5%). (4) ICLR 2025 Oral confirmed at iclr.cc/virtual/2025/oral/31838 (Jung, Brahman, Choi); dates correct (v1 2024-07-25; ICLR 2025 in April 2025). Two trivial nuances, neither rising to a correction: the paper defines abstention as returning empty set / excluding from coverage - deferral "to humans" is the intended framing but abstained items are not actually routed to human annotators in the experiments; and the guarantee holds with probability 1-delta over calibration-set sampling (standard for such guarantees, and consistent with "provable, user-specified"). Supersession check: not superseded - but a newer alternative exists: SCOPE (arXiv:2602.13110, ICML 2026 poster, Feb 2026) uses conformal calibration with a Bidirectional Preference Entropy signal and reports better calibration/coverage than Simulated Annotators; it complements rather than invalidates the claim about this paper. The "83 citations" figure was not independently verified but is plausible and not load-bearing.
• CLAIM: The minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n(1-rho^2) where n is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with R^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes. VERDICT: confirmed | All elements verified against the source (https://arxiv.org/html/2605.16354v1, Jane Paik Kim, Stanford Dept. of Psychiatry and Behavioral Sciences, submitted 2026-05-08 - date and attribution correct). (1) Floor formula: paper states required human reviews converge to n(1-rho^2) as N increases - exact match. (2) Worked example: n=200, R^2=0.70, N=2000 -> 65 human labels, confirmed verbatim; the N=400/100 figure is stated in the paper as "with a budget of 100 human ratings, N can be reduced to 400 while achieving target power" - same numbers, slightly different framing (human budget fixed, N solved) but mathematically equivalent to the claim. (3) Stratification savings: paper reports "up to 12.9%" reduction when R^2 gap is large (0.8 vs 0.1) - matches "~13%", with the paper adding that moderate gaps yield only ~2%. (4) Diminishing returns from more judge labels: explicitly stated ("each successive increase in N produces diminishing reductions in n"). (5) All three caveats in the evidence summary (pilot R^2 overestimation breaking precision guarantees, human-as-gold-standard assumption, single-rater scope) appear in the paper's limitations. Supersession check: two searches found no v2 revision, no citing papers, and no later work replacing or contradicting the result as of 2026-07-14. Minor precision notes only: "up to ~13%" should strictly be "up to 12.9% in the extreme R^2-gap case, ~2% for moderate gaps," and the floor is asymptotic (approached as N grows), not attainable at finite N - the claim's own phrasing ("converge to a floor," "up to") already reflects both.