{
  "batch": 5,
  "sources": [
    {
      "url": "https://arxiv.org/abs/2606.03043",
      "source_title": "The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment",
      "source_date": "2026-06-02",
      "source_type": "academic",
      "claims": [
        {
          "id": "AJ-05",
          "domain": "academic-judges",
          "claim": "On subjective rubrics, LLM judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-to-human), judges use only 0.3-0.5x the human score spread, and inter-LLM agreement (r~0.35) exceeds LLM-human agreement (r~0.27-0.32) - while on a rubric with a verifiable factual answer the same judges fall back into the human range (58.5 degrees, r=0.519).",
          "evidence": "Mukherjee, Hamna, Bali, Sitaram (arXiv 2606.03043, abstract opened). 41 LLM judges, 4 datasets, 8 Indic languages, geometric analysis with bootstrap CIs. Fine-tuning/preference optimization recovers score spread (0.32 to 1.08) but barely moves the axis; only post-hoc calibration on a small human-anchored set improved all rubrics. Scope caveat: Indic-language community-health data, but the verifiable-vs-subjective contrast is the mechanism of interest.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://aclanthology.org/2022.tacl-1.78/",
      "source_title": "Investigating Reasons for Disagreement in Natural Language Inference (TACL 10)",
      "source_date": "2022-12-01",
      "source_type": "academic",
      "claims": [
        {
          "id": "AQ-05",
          "domain": "annotation-quality",
          "claim": "Even on a nominally objective task (NLI), annotator disagreement decomposes into a 10-category taxonomy across three high-level sources - uncertainty in sentence meaning, underspecification in guidelines, and annotator behavior - proving that a fraction of inter-reviewer variance is item-intrinsic and survives perfect rubric operationalization.",
          "evidence": "Jiang & de Marneffe, TACL 2022, developed and hand-applied the taxonomy to high-disagreement NLI items; disagreement persisted among trained annotators on items with genuine interpretive ambiguity. D3CODE (EMNLP 2024; 4.5K sentences, 4K+ annotators, 21 countries) adds that residual variance is systematically structured by annotators' moral values and region, not random.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2506.11440",
      "source_title": "AbsenceBench: Language Models Can't Tell What's Missing",
      "source_date": "2025-06-13",
      "source_type": "academic",
      "claims": [
        {
          "id": "CM-06",
          "domain": "critique-models",
          "claim": "LLMs are structurally weak at detecting omissions: on AbsenceBench, average F1 drops 56.9 points versus detecting the same content as insertions (best model 71.2% overall, 40.0% on code diffs, vs ~99.5% needle-in-haystack), and inserting explicit placeholders at gap sites recovers ~35.7 points - absence has no attention key to attend to.",
          "evidence": "4,302 instances across poetry, numerical sequences, GitHub PRs (~5K token contexts): models must list deliberately deleted elements given original+modified documents. Gemini-2.5-flash (thinking) best at 71.2% avg F1; Mixtral-8x7B scores 99%+ on NIAH but 14.7% here; thinking tokens buy only ~7.9%; '<missing line>' placeholders add up to +81.8%. NeurIPS 2025 Datasets & Benchmarks. Verified from arXiv HTML full text.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2512.10791",
      "source_title": "The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality",
      "source_date": "2025-12",
      "source_type": "academic",
      "claims": [
        {
          "id": "GF-07",
          "domain": "grounding-faithfulness",
          "claim": "Grounding judges have a measured agreement-default asymmetry: in Google's FACTS Leaderboard (Dec 2025), grounding judges score ~85 F1 on the positive (grounded) class but only ~46 F1 on the negative (ungrounded) class, and the grounding metric is gameable by vague, short responses that avoid unsupported claims - countered by a mandatory eligibility gate that scores non-responsive answers as inaccurate.",
          "evidence": "Fetched full text of arXiv:2512.10791v1 (The FACTS Leaderboard, Google, Dec 2025). Best judge-prompt combos reach only ~65 macro-F1 (gemini-2.5-flash + v2 prompt: 65.33) against a 320-example human-adjudicated held-out set; positive-class ~85 vs negative-class ~46 F1; eligibility check explicitly added because 'grounding metrics can be hacked via vague, short responses'.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2601.20913",
      "source_title": "Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges",
      "source_date": "2026-01-28",
      "source_type": "academic",
      "claims": [
        {
          "id": "HS-07",
          "domain": "hybrid-statistical",
          "claim": "For certifying that a failure rate is below a threshold (the low-base-rate pass/fail regime), the 'Noisy but Valid' framework (ICLR 2026) uses a small human calibration set to estimate judge TPR/FPR, applies a variance-corrected test to the large judge-labeled stream with finite-sample Type-I error control, and derives exact conditions under which judge-based testing has HIGHER statistical power than direct human evaluation.",
          "evidence": "arXiv:2601.20913 (Feng, Shen, Balashankar, Gerner-Beuerle, Rodrigues; submitted 2026-01-28, accepted ICLR 2026 per arXiv page) abstract fetched: variance-corrected critical threshold, finite-sample Type-I control under calibration uncertainty, validation on Jigsaw/Hate Speech/SafeRLHF, and a quantified 'oracle gap' - the power cost of having to estimate judge reliability rather than knowing it.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/html/2504.01848v2",
      "source_title": "PaperBench: Evaluating AI's Ability to Replicate AI Research (OpenAI)",
      "source_date": "2025-04-02",
      "source_type": "primary",
      "claims": [
        {
          "id": "RR-08",
          "domain": "rubrics-recent",
          "claim": "PaperBench (OpenAI, Apr 2025) demonstrates the extreme end of decomposition: 8,316 individually-graded binary leaf criteria across 20 papers, sibling-relative manual weights, rubrics co-developed with each paper's original authors over multiple weeks per rubric; an o3-mini judge then grades leaves at F1 = 0.83 vs expert human judgments for ~$66/paper versus ~12 hours of expert grading time.",
          "evidence": "Full paper (arXiv 2504.01848) confirmed: each leaf is pass/fail; parent scores are weighted averages propagating to a root score; weights reflect importance relative to siblings, not difficulty; JudgeEval is a separate benchmark for judges (o1 scored 0.84 but ~$830/paper - o3-mini chosen as cost-effective); human expert grading estimated at 'tens of hours' per paper.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals",
      "source_title": "Braintrust: LLM-as-a-judge vs human-in-the-loop evals: When to use each",
      "source_date": "2026-04-03",
      "source_type": "practitioner",
      "claims": [
        {
          "id": "IP-08",
          "domain": "industry-practice",
          "claim": "Braintrust's published cost arithmetic makes the human-budget fork concrete: an LLM judge over 10,000 outputs costs ~$5-15 while a domain expert reviewing 500 of them costs ~$800-1,800 (~100x per output), which is why its prescribed architecture is deterministic checks on everything, LLM judges on everything, and humans only on flagged/low-confidence/disagreement slices plus random spot-checks of confident passes.",
          "evidence": "Fetched Braintrust article (2026-04-03) and human-review docs. The hybrid loop is explicit: trace production -> continuous deterministic + judge scoring -> flag low-confidence/scorer-disagreement -> human review -> reviewed findings become new scorers/rubric refinements -> CI regression gates. They name the biggest practitioner mistake as treating judge scores as ground truth without human validation, and note untrained reviewers with vague rubrics can produce labels worse than a decent judge.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://aclanthology.org/2025.emnlp-main.144.pdf",
      "source_title": "NUTMEG: Separating Signal From Noise in Annotator Disagreement",
      "source_date": "2025-11-04",
      "source_type": "academic",
      "claims": [
        {
          "id": "CT-11",
          "domain": "contrarian",
          "claim": "Annotator disagreement contains recoverable systematic signal, not just noise: NUTMEG, a Bayesian model separating annotator-competence noise from subpopulation-level systematic disagreement, produces downstream models that significantly outperform both majority-vote aggregation and fully disaggregated training - meaning aggregation to a single 'true' label destroys measurable information.",
          "evidence": "Ivey, Gauch & Jurgens, EMNLP 2025 main (pp. 2874-2887); PDF and poster abstract inspected: NUTMEG estimates true labels per subpopulation and beats MACE/majority-vote at replicating subgroup label distributions on politeness and offensiveness. Sits atop the perspectivist NLP literature (NLPerspectives workshops 2022-2025) which rejects single-gold-label resolution for subjective tasks.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://doi.org/10.1177/026553229801500205",
      "source_title": "Using FACETS to model rater training effects (Weigle, Language Testing 15(2))",
      "source_date": "1998-04",
      "source_type": "academic",
      "claims": [
        {
          "id": "CG-02",
          "domain": "critic-and-gapfill",
          "claim": "Rater training on real scoring behavior improves within-rater consistency but does NOT equalize severity: after formal training of 16 raters in an operational ESL writing exam, significant between-rater severity differences remained even though consistency improved for most raters.",
          "evidence": "Weigle 1998 (Language Testing, 363 citations; abstract opened and confirmed verbatim): 'rater training is more successful in helping raters give more predictable scores (i.e., intra-rater reliability) than in getting them to give identical scores (i.e., inter-rater reliability).' This is the canonical, repeatedly replicated finding of the language-testing rater literature (Lumley & McNamara 1995 same conclusion via MFRM).",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/html/2603.18053",
      "source_title": "Auditing the Auditors: Does Community-based Moderation Get It Right? (arXiv 2603.18053)",
      "source_date": "2026-03-17",
      "source_type": "academic",
      "claims": [
        {
          "id": "CG-16",
          "domain": "critic-and-gapfill",
          "claim": "Auditing raters by rewarding agreement with the eventual consensus outcome ('consensus-based auditing', as X's Community Notes has done since Sept 2022 by tying participation eligibility to agreement with the final aggregate) measurably induces strategic conformity - minority contributors' evaluations drift toward the majority and their participation share falls precisely on controversial items - and a two-stage alternative that weights contributors by the stability of their past residuals (predictability relative to a latent-factor model) rather than majority-agreement improves out-of-sample predictive performance.",
          "evidence": "Fetched arXiv 2603.18053 (Alimohammadi, Huang, Borgs, Chayes, Mar 2026) abstract and framing via Exa crawl: empirical evidence from Community Notes data plus a behavioral model where contributors trade off private beliefs against anticipated penalties for disagreement; the proposed method gives influence to consistently-informative contributors 'even when they disagree with the prevailing consensus.' Senior authors (Borgs h-54, Chayes h-60). Preprint, not yet peer-reviewed.",
          "in_corpus_verdict": null
        }
      ]
    }
  ]
}