{ "batch": 3, "sources": [ { "url": "https://arxiv.org/abs/2606.19544", "source_title": "Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias", "source_date": "2026-06-17", "source_type": "academic", "claims": [ { "id": "AJ-03", "domain": "academic-judges", "claim": "The largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including April-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected Cohen's kappa is 33-41 percentage points lower on MT-Bench - and judge rankings shift by up to 14 positions depending on which benchmark you use.", "evidence": "Norman, Rivera, Hughes (arXiv 2606.19544, abstract opened and confirmed). Four cohort-wide findings: universal kappa deflation; benchmark-dependent rankings (up to 14 position shifts across MT-Bench/JudgeBench/RewardBench); a 'consistency-bias paradox' where test-retest reliability >0.95 coexists with position bias >0.10 in two production judges; verbosity bias small (<0.011) under a single controlled pairwise rubric. They distill a Minimum Viable Validation Protocol.", "in_corpus_verdict": "confirmed" }, { "id": "CM-09", "domain": "critique-models", "claim": "A June 2026 study of 21 LLM judges (~541k judgments) shows raw percent-agreement overstates chance-corrected agreement by 33.8-41.3 points (85% agreement = kappa ~0.48), judge rankings flip by up to 15 positions across benchmarks, and the most reproducible judges are among the least valid (test-retest 0.99 with position bias 0.19) - leading the authors to prescribe a pre-deployment Minimum Viable Validation Protocol.", "evidence": "'Reliability without Validity' (UC Berkeley, arXiv 2606.19544, evals run March-April 2026): MT-Bench/JudgeBench/RewardBench, 118 runs at temperature 0; kappa deflation universal across all 21 judges; consistency-bias paradox instantiated by Qwen3-8B and Gemini 2.5 Flash; MVVP = report kappa/Krippendorff alpha as headline, AB+BA position swaps, >=3 replicates, >=2 benchmarks spanning preference- and correctness-style labels, and audit that high stability is not just high bias. Verified from arXiv HTML full text.", "in_corpus_verdict": null }, { "id": "CT-01", "domain": "contrarian", "claim": "Raw percent-agreement systematically overstates LLM-judge ability: in the largest judge meta-evaluation to date (21 judges, 9 providers, ~541,000 judgments, including April-2026 frontier models), Cohen's kappa runs 33-41 percentage points below exact-match agreement on MT-Bench, judge rankings shift by up to 14 positions across benchmarks, and two production-deployed judges combine test-retest reliability >0.95 with severe position bias >0.10 (a 'consistency-bias paradox').", "evidence": "Norman, Rivera & Hughes ran 118 runs across MT-Bench, JudgeBench, RewardBench under three protocols (agreement, consistency, bias audit); kappa deflation was universal across the cohort; they distill a 'Minimum Viable Validation Protocol'. Verified from the arXiv abstract page. Corroborated by the NeurIPS 2025 position paper 'Neither Valid nor Reliable? Investigating the Use of LLMs as Judges' (arXiv 2508.18076, Chehbouni et al.), which argues via social-science measurement theory that LLJ adoption has outpaced scrutiny of validity/reliability and that human-agreement proxying is an unvalidated assumption.", "in_corpus_verdict": "confirmed" } ] }, { "url": "https://arxiv.org/abs/2410.03775", "source_title": "Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025)", "source_date": "2025-05-01", "source_type": "academic", "claims": [ { "id": "AQ-03", "domain": "annotation-quality", "claim": "Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target.", "evidence": "Elangovan et al. (Amazon), ICLR 2025 (arXiv 2410.03775, opened and verified): Krippendorff's alpha / kappa were designed for human-human reliability and their assumptions break for machine labels; the paper's fix is (1) stratify meta-evaluation by human-label uncertainty, (2) use binned Jensen-Shannon divergence for perception-type (inherently subjective) tasks, (3) report 'perception charts' instead of one correlation. Code open-sourced (amazon-science/BeyondCorrelation).", "in_corpus_verdict": "partially_confirmed" } ] }, { "url": "https://arxiv.org/abs/2405.06087", "source_title": "When combinations of humans and AI are useful: A systematic review and meta-analysis", "source_date": "2024-10-28", "source_type": "contrarian", "claims": [ { "id": "CM-04", "domain": "critique-models", "claim": "A Nature Human Behaviour meta-analysis of 100+ experiments (300+ effect sizes) found human-AI combinations on average performed significantly WORSE than the best of human or AI alone on decision/judgment tasks, with gains only where humans outperformed the AI solo - naive human-checks-AI designs destroy value.", "evidence": "Vaccaro, Almaatouq & Malone systematic review/meta-analysis: average human-AI combo underperformed the stronger party alone; losses concentrated in decision-making tasks, gains in content-creation tasks; combos helped when humans beat AI alone. DeepMind's complementarity paper explicitly frames its confidence-routing + evidence-only design as the answer to this result. Verified from arXiv abstract.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/abs/2406.19276", "source_title": "VeriScore (EMNLP Findings 2024) + FactBench/VERIFY (ACL 2025)", "source_date": "2024-06 (VeriScore, foundational); 2025-07 (FactBench ACL 2025)", "source_type": "academic", "claims": [ { "id": "GF-05", "domain": "grounding-faithfulness", "claim": "Verifiability triage before entailment is established machinery: VeriScore extracts and scores ONLY verifiable claims because FActScore/SAFE 'assume that every claim is verifiable', which breaks on real long-form text containing opinions and unverifiable content; FactBench's VERIFY pipeline (ACL 2025) further labels content units supported/unsupported/UNDECIDABLE against retrieval, with 4,467 human-annotated units released for validation.", "evidence": "Fetched arXiv:2406.19276 abstract (VeriScore, EMNLP Findings 2024): human evaluation found VeriScore's extracted claims 'more sensible' than competitors across 8 long-form tasks. Fetched launchnlp/FactBench GitHub (ACL 2025 paper 2410.22257): three-way supported/unsupported/undecidable labeling according to retrieval results, benchmarked against FActScore, SAFE, Factcheck-GPT.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/html/2605.31278v2", "source_title": "Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation", "source_date": "2026-05 (v2 2026-06)", "source_type": "practitioner", "claims": [ { "id": "HS-05", "domain": "hybrid-statistical", "claim": "GLIDE (May 2026) is an open-source industrial library unifying PPI++, stratified PPI, predict-then-debias bootstrap, and active inference with a decision tree for method selection, and it quantifies when PPI helps: effective-sample gain is ~1.0x at judge-human correlation rho=0.1, ~2.2x at rho=0.9; CLT-based estimators need roughly >=50 human labels per stratum (below that use bootstrap PTD); in a real agent-safety case (rho=0.59, ~13-point judge bias) 100 human labels became worth 143-157.", "evidence": "arXiv:2605.31278v2 HTML fetched (Martinon, Merad, Raki; Emerton Data): Monte Carlo with 500 human + 1000 proxy labels shows effective n ~500 at rho=0.1 vs ~1100 at rho=0.9; R-Judge case study with Claude Sonnet as judge; library at github.com/EmertonData/glide with reproducible validation notebooks; limitations: means/proportions only, i.i.d. assumptions, no anytime-valid inference.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/html/2604.01375v2", "source_title": "RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics (Snorkel AI)", "source_date": "2026-04-20", "source_type": "academic", "claims": [ { "id": "RR-06", "domain": "rubrics-recent", "claim": "RIFT (Snorkel AI, Apr 2026) provides the first rubric failure-mode taxonomy - 8 modes in 3 categories (Reliability: Subjective, Non-Atomic, Ungrounded; Content Validity: Misaligned/Rigid, Missing Criteria; Consequential Validity: Hackable, Low Signal, Redundant) - and shows failure-mode density correlates with judge-human misalignment (r=0.162, p=0.0021); synthetic rubrics skew Subjective (86.7% vs 52.6% for human-crafted) while human rubrics skew Misaligned/Rigid (63.2% vs 20.0%); automated LLM detection reaches F1 0.925 for Subjective but ~0.000 for Hackable.", "evidence": "Full HTML (arXiv 2604.01375v2) confirmed all numbers: 85 rubrics / 255 expert annotations across 5 sources (human: AdvancedIF, ResearchRubrics; synthetic: WildChecklists, OpenRubrics, AutoRubrics); taxonomy built via grounded theory, mean Cohen's kappa 0.64; diagnostics include LLM-as-judge classifiers (GPT-5.2, Gemini 3 Pro), inter-rater reliability across labeler models, alignment to a strong reference, and reward-variance probes.", "in_corpus_verdict": null } ] }, { "url": "https://sacra.com/c/surge-ai", "source_title": "Sacra: Surge AI revenue, funding & news", "source_date": "2026-04-21", "source_type": "practitioner", "claims": [ { "id": "IP-06", "domain": "industry-practice", "claim": "Surge AI (~$1.2B revenue 2024, ~50k contractors, bootstrapped) runs QA as real-time dashboards over gold-standard accuracy, inter-annotator agreement, and per-worker trust ratings, with flagged labels automatically reassigned to other annotators; its published AdvancedIF verifier work with Meta reports F1 0.728 versus human judgments and +13% RL gains from rubric-based rewards.", "evidence": "Fetched Sacra's Surge AI company report (updated 2026-04-21; figures are Sacra estimates). QA description: gold-standard accuracy + IAA + per-worker trust ratings on live dashboards; automated reassignment of low-quality labels; premium per-minute pay (~$0.30-0.40/min) as a quality incentive; vetting = domain tests + background checks + ongoing performance evaluation. Hemingway-bench used 5,000+ blind pairwise expert comparisons. Risks noted: ~12-customer concentration, CA misclassification suit alleging unpaid training time and tight task timers.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/abs/2503.05061", "source_title": "No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding", "source_date": "2025-03-07", "source_type": "academic", "claims": [ { "id": "CT-08", "domain": "contrarian", "claim": "LLM judges give the weakest signal exactly where QA needs them most: they cannot reliably grade responses to questions they cannot answer themselves (poor signal on the hardest items in a benchmark), and in expert domains subject-matter experts agreed with LLM-judge picks only 64% (mental health) to 68% (dietetics) of the time, with the judge favoring superficially actionable detail.", "evidence": "'No Free Labels' (Kim et al., arXiv 2503.05061, Mar 2025) shows judge quality collapses on the most difficult items and that human-written reference answers improve agreement and reduce self-preference; Szymanski et al. (ACM IUI 2025, DOI 10.1145/3708359.3712091) measured the 64/68% SME agreement in pairwise comparisons on domain tasks. Both reported consistently across two independent search engines; abstracts inspected.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/html/2603.00077v1", "source_title": "Autorubric: A Unified Framework for Rubric-Based LLM Evaluation (arXiv 2603.00077)", "source_date": "2026-03-02", "source_type": "academic", "claims": [ { "id": "F26-12", "domain": "frontier-2026", "claim": "Autorubric (arXiv 2603.00077, March 2026) is an open-source unified rubric-evaluation framework that packages the 2026 judge-reliability defaults -- per-criterion atomic evaluation, binary/ordinal/nominal criteria with weights, multi-judge ensembles with configurable vote aggregation, few-shot calibration with verdict-balanced sampling, and built-in mitigations for position bias (option shuffling), verbosity bias (length penalties), and criterion conflation.", "evidence": "arXiv HTML abstract read; feature list confirmed, validated on three benchmarks including a contributed 100-sample ground-truth dataset (CHARM-100). Notably, an AI data vendor's banking meta-eval used Autorubric as the representative text-only rubric-judge family -- and found it architecture-limited (cannot inspect artifacts), placing a measured ceiling on this whole verifier class for evidence-bearing tasks.", "in_corpus_verdict": null } ] }, { "url": "https://www.tspa.org/curriculum/ts-fundamentals/content-moderation-and-operations/content-moderation-quality-assurance/", "source_title": "TSPA: Content Moderation Quality Assurance + Metrics for Content Moderation", "source_date": "2022-09-16", "source_type": "practitioner", "claims": [ { "id": "CG-14", "domain": "critic-and-gapfill", "claim": "The Trust & Safety Professional Association (industry body staffed by Meta/Google/TikTok alumni) codifies exactly two complementary QA channels - forward audit sampling (weighted re-review samples by peers or a dedicated quality team) and 'reverse quality sampling' (pre-reviewed golden/known-verdict items injected through the regular review process) - with the binding constraint that seeded items 'must look exactly like regular reviews to be effective', plus appeals/overturns as a cheap but population-biased third signal, and a four-way error taxonomy (false positive, false negative, wrong-selection, technical error).", "evidence": "Fetched both TSPA curriculum pages. QA page: golden seeding gives 'full control' to probe chosen edge cases but fails when user history is part of the review or when indistinguishability breaks; appeals surface false positives 'at a low cost' but 'the population of appeals is often different from the population of all decisions.' Metrics page: overturn rate 'can be a difficult metric to interpret' alone (low overturn = correct decisions OR users gave up); 'consistency' (multi-review mismatch rate) is tracked as a distinct metric from error rate; prevalence measurement via random sampling is called a 'premium metric' due to cost.", "in_corpus_verdict": null } ] } ] }