{
  "batch": 6,
  "sources": [
    {
      "url": "https://arxiv.org/abs/2606.13685",
      "source_title": "The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation",
      "source_date": "2026-04-23",
      "source_type": "academic",
      "claims": [
        {
          "id": "AJ-06",
          "domain": "academic-judges",
          "claim": "Single-trial LLM judging is measurably noisy: across 29 tasks with 50 repeated trials, pairwise preferences flipped on average 13.6% of the time (28% of questions exceeded 20% flip rate), semantically equivalent prompt templates changed majority outcomes in 25% of tested cases, and ~11 repeated trials were needed for majority vote to recover the reference verdict with 95% probability.",
          "evidence": "Yagubyan (arXiv 2606.13685, abstract opened). GPT-4o-mini and GPT-4.1-mini judges; also found significant first-position bias (72% A-majority, p=0.024), cross-judge agreement only 76% (kappa 0.51), and a pairwise-pointwise gap where judges pick winners even when their own scalar scores show no meaningful difference. Caveat: both judges from one provider, mini-tier models; the Reliability-without-Validity cohort shows some production judges reach test-retest >0.95.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://doi.org/10.1016/0895-4356(90)90158-L",
      "source_title": "High agreement but low kappa: I. The problems of two paradoxes (Feinstein & Cicchetti)",
      "source_date": "1990-01-01",
      "source_type": "academic",
      "claims": [
        {
          "id": "AQ-06",
          "domain": "annotation-quality",
          "claim": "Chance-corrected agreement coefficients (Cohen's kappa, and analogously Krippendorff's alpha) can be near zero despite very high raw agreement when class prevalence is extreme (the Feinstein-Cicchetti paradoxes), so under the 90%+ pass rates typical of QA pipelines, single-coefficient reliability reporting is misleading in the opposite direction from raw percent agreement.",
          "evidence": "Feinstein & Cicchetti (J Clin Epidemiol, 1990) formalized the two kappa paradoxes (high agreement/low kappa under imbalance; asymmetric marginals inflating kappa); the effect is prevalence-driven, extensively replicated (Quarfoot & Levine, Am. Statistician 2016; Gwet's AC1 literature). This is exactly the regime of QA pass/fail data with low failure base rates.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2401.04518",
      "source_title": "The Critique of Critique (MetaCritique)",
      "source_date": "2024-01-09",
      "source_type": "academic",
      "claims": [
        {
          "id": "CM-07",
          "domain": "critique-models",
          "claim": "Critique quality is itself quantifiable at usable reliability: MetaCritique decomposes critiques into atomic information units and scores precision/recall against references, with GPT-4 AIU-level judgments at 85-89% accuracy and Meta-F1 correlating with human gold at Pearson 0.84-0.89 - and measured baselines show human critiques are high-precision/low-recall (87.6%/48.7%) while LLM critiques are the inverse-ish (71.9%/53.3%).",
          "evidence": "ACL Findings 2024 framework: precision = fraction of critique AIUs judged factual; recall = coverage of reference-critique AIUs; each judgment carries a natural-language rationale. Choosing critiques by Meta-F1 yields refinements that win 51%/lose 27% by human eval, vs GPT-4 pairwise picks losing more than winning. ~28% of LLM-critique AIUs were non-factual vs ~12% for humans; LLM critiques carried 2.4x the information volume (8.10 vs 3.31 AIUs). Verified from arXiv HTML full text.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2501.03200",
      "source_title": "FACTS Grounding Leaderboard (v1 paper + v2 in FACTS Leaderboard)",
      "source_date": "2025-01 (v1); 2025-12 (v2)",
      "source_type": "academic",
      "claims": [
        {
          "id": "GF-08",
          "domain": "grounding-faithfulness",
          "claim": "Cross-family judge ensembles are the standard mitigation for self-preference: FACTS Grounding v1 (Jan 2025) aggregated three frontier judges from different families (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) explicitly because 'models are biased towards favorably judging their own outputs', and Grounding v2 (Dec 2025) kept a two-family ensemble (Gemini 2.5 Flash + GPT-5).",
          "evidence": "Fetched arXiv:2512.10791v1 which states the v2 judge-ensemble rationale and composition; v1 design documented in arXiv:2501.03200 (Jan 2025) and the DeepMind blog (Dec 17, 2024). FACTS Parametric suite separately validated that a single-judge setup preserved rankings vs a mixed panel (Gemini 2.5 Pro, o3, Grok 4) - i.e., ensemble necessity is task-dependent and testable.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2511.21140",
      "source_title": "How to Correctly Report LLM-as-a-Judge Evaluations",
      "source_date": "2025-11-26 (v4 2026-05-31)",
      "source_type": "academic",
      "claims": [
        {
          "id": "HS-08",
          "domain": "hybrid-statistical",
          "claim": "Naive reporting of judge-scored evaluations is systematically biased by judge sensitivity/specificity; a plug-in correction with confidence intervals propagating BOTH test-set and calibration-set uncertainty fixes it, remains unbiased under distribution shift between calibration and test data, includes adaptive allocation of calibration labels, and characterizes regimes where corrected judge-based evaluation is more reliable than human-only evaluation.",
          "evidence": "arXiv:2511.21140 (Lee, Zeng, Jeong, Sohn, Kangwook Lee; submitted 2025-11-26, v4 2026-05-31, accepted ICML 2026 per arXiv page) abstract fetched. The calibration-to-test distribution-shift robustness is the distinctive property versus standard PPI's i.i.d. assumption.",
          "in_corpus_verdict": null
        },
        {
          "id": "F26-09",
          "domain": "frontier-2026",
          "claim": "A plug-in statistical framework now exists for reporting judge-based pass rates correctly: because an imperfect judge (sensitivity q1, specificity q0) biases the naive pass-rate estimate (upward at low true rates, downward at high), the Rogan-Gladen-style corrected estimator plus a confidence interval propagating BOTH test-set and calibration-set uncertainty should replace raw judge-agreement numbers, with an adaptive algorithm allocating calibration labels between true-pass and true-fail classes (roughly m0 ~ (1/p-1)*sqrt(kappa)*m1) -- e.g., ~200 labels per class for CI width <0.1 in the paper's worked regime.",
          "evidence": "Full paper text crawled and read, including derivations, Monte Carlo validation (naive CI coverage near zero; corrected CI holds 95% across all true accuracies), and released Python implementation (UW-Madison-Lee-Lab/LLM-judge-reporting). Posted Nov 26, 2025 -- just before the charter window -- but it is the current statistical-reporting layer being circulated in 2026 (alphaXiv feature May 2026) and no 2026 successor supersedes it.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://developers.openai.com/api/docs/guides/graders",
      "source_title": "OpenAI API documentation: Graders",
      "source_date": "2026-07-14",
      "source_type": "primary",
      "claims": [
        {
          "id": "RR-09",
          "domain": "rubrics-recent",
          "claim": "OpenAI's official grader documentation prescribes that every model grader be validated on its own eval built from expert-graded answers (grader score ordering must reproduce expert ranking), and names the canonical reward-hacking detection signature: the graded model scores well on the model-grader eval while doing poorly on expert human evaluations.",
          "evidence": "Fetched live doc (developers.openai.com/api/docs/guides/graders, accessed 2026-07-14): grader taxonomy = string_check, text_similarity, score_model (LLM judge with numeric range), python (sandboxed), multigrader (formula-combined sub-graders, RFT only); guidance: iterate grader prompts, use few-shot examples of good/fair/poor answers, prefer smooth over binary scores, add edge cases to the grader eval over time, 'guard against reward hacking'. Page carries a deprecation note for graders within the evals/fine-tuning workflows they support.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2602.18583",
      "source_title": "Luna-2: Scalable Single-Token Evaluation with Small Language Models (arXiv 2602.18583) + Patronus Lynx launch materials",
      "source_date": "2026-02-20 (Lynx 2024-07-11)",
      "source_type": "primary",
      "claims": [
        {
          "id": "IP-09",
          "domain": "industry-practice",
          "claim": "Fine-tuned specialist judges have commoditized binary grounding checks: Patronus Lynx (fine-tuned Llama-3-70B) beats GPT-4o on HaluBench faithfulness detection and lifted one customer's hallucination detection from 0.375 to 0.69, while Galileo's Luna-2 (arXiv 2026-02-20) runs hundreds of per-metric LoRA adapters on one small-model backbone at >80x lower cost and >20x lower latency than LLM-as-judge at claimed parity accuracy, in production on 100M+ sessions.",
          "evidence": "Patronus launch post + SiliconANGLE (2024-07-11) give HaluBench numbers (Lynx 70B +8.3% vs GPT-4o on PubMedQA inaccuracies; 8B +24.5% vs GPT-3.5); Patronus evaluators page gives the Algomo 0.375->0.69 customer number and Gamma's 1,000+ manual eval hours saved; Percival extends this to tracing where in a multi-step agent run the failure occurred. Fetched Luna-2 arXiv abstract (2602.18583): single-token SLM evaluation, >80x cost / >20x latency reduction, 100M+ sessions and 100B tokens/month in production, $30M+ claimed annual eval savings. Vendor-reported numbers; treat magnitudes, not decimals.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://mackinstitute.wharton.upenn.edu/2025/new-in-nature-chatgpt-decreases-idea-diversity-in-brainstorming/",
      "source_title": "New in Nature: ChatGPT Decreases Idea Diversity in Brainstorming (Meincke, Nave & Terwiesch, Nature Human Behaviour)",
      "source_date": "2025-05",
      "source_type": "academic",
      "claims": [
        {
          "id": "CT-12",
          "domain": "contrarian",
          "claim": "AI assistance homogenizes human output at the population level even while raising individual quality: in a Nature Human Behaviour brainstorming study 94% of ChatGPT-assisted participants' ideas shared overlapping concepts (nine independently produced the same product name) while human-only ideas were entirely unique, and the 'Artificial Hivemind' study found different vendors' frontier models converge on near-identical phrasings (~81% average similarity between DeepSeek-V3 and GPT-4o).",
          "evidence": "Meincke, Nave & Terwiesch (Nature Human Behaviour, 2025; Wharton Mack Institute summary opened and read); Jiang et al. (UW/CMU/AI2 'Artificial Hivemind', reported by The Decoder) documents intra-model repetition plus inter-model homogeneity. Consistent with Doshi & Hauser (Science Advances 2024) and Wan et al. 2026 (diverse AI personas partially mitigate homogenization). All are adjacent-domain (ideation/writing), not annotation-QA-feedback studies.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://doi.org/10.1111/j.2044-8325.2011.02045.x",
      "source_title": "Rater training revisited: An updated meta-analytic review of frame-of-reference training (Roch et al.)",
      "source_date": "2012-06",
      "source_type": "academic",
      "claims": [
        {
          "id": "CG-03",
          "domain": "critic-and-gapfill",
          "claim": "Frame-of-reference (FOR) training, the best-validated rater-training method, has a meta-analytic effect on rating accuracy of about Cohen's d = 0.50 (moderate) - down from the d = 0.83 estimated by the earlier, smaller Woehr & Huffcutt 1994 meta-analysis - i.e., training helps but removes nowhere near all rater error.",
          "evidence": "Roch, Woehr, Mishra & Kieszczynska (J. Occup. Organ. Psychol., 2012; abstract opened) is the updated meta-analysis with 4x the studies of Woehr & Huffcutt 1994; it finds FOR training effective, with Borman's differential accuracy and behavioural accuracy most improved. The d=0.50 overall figure confirmed via two independent citing sources (Tsai et al. 2019 SMU full-text PDF: 'medium-to-large effect on improving rating accuracy (d = 0.50, Roch et al., 2012; d = 0.83, Woehr & Huffcutt 1994)'; Loignon et al. 2017). Note: classic 'rater error training' (teaching raters to avoid halo/leniency) reduces those errors but can reduce accuracy - FOR (anchoring raters to a shared performance theory with practice+feedback) is the variant that works.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2306.07466",
      "source_title": "Statistical Methods for Auditing the Quality of Manual Content Reviews (arXiv 2306.07466, ICLR 2023 Tiny Papers)",
      "source_date": "2023-06-12",
      "source_type": "academic",
      "claims": [
        {
          "id": "CG-17",
          "domain": "critic-and-gapfill",
          "claim": "Google's audit-quality method (ICLR 2023 Tiny Paper, Google LLC authors) decomposes inter-rater agreement per rubric question rather than per item - in their worked example one question had Fleiss kappa 0.094 against a 0.475 overall, isolating it as the ambiguous criterion - and identifies sequential non-blind review (a later reviewer seeing the earlier verdict) as a distinct, measurable source of audit risk, testable via A/B or difference-in-differences.",
          "evidence": "Extracted full PDF text via pdftotext. Methods: per-question Fleiss kappa to locate high-disagreement criteria; chi-square to test criterion-verdict linkage; t-test/ANOVA to detect review teams systematically deviating from ground truth; binomial CIs to extrapolate sampled error rates. Caveats: 2-page tiny paper, synthetic dataset (3 reviewers, 9-question rubric, 1,528 products), no production deployment claimed. Code at https://github.com/xuanyang0607/openreviewpaper.",
          "in_corpus_verdict": null
        }
      ]
    }
  ]
}