{
  "batch": 11,
  "sources": [
    {
      "url": "https://arxiv.org/abs/2410.20266",
      "source_title": "Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks",
      "source_date": "2024-10 (IUI 2025)",
      "source_type": "academic",
      "claims": [
        {
          "id": "AJ-11",
          "domain": "academic-judges",
          "claim": "In expert-knowledge domains, LLM judges agreed with subject-matter experts only 68% (dietetics, registered dietitians) and 64% (mental health, clinical psychologists) on overall pairwise preference, with agreement varying further across domain-specific aspect questions.",
          "evidence": "Szymanski et al. (arXiv 2410.20266; ACM IUI 2025, ~220 citations). Mixed-methods pairwise comparison study; authors conclude LLMs alone lack the depth for complex knowledge-specific tasks and experts must stay in the evaluation loop. Full abstract text opened and confirmed.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://aclanthology.org/2025.findings-acl.1323/",
      "source_title": "Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks (Findings of ACL 2025)",
      "source_date": "2025-07-27",
      "source_type": "academic",
      "claims": [
        {
          "id": "AQ-11",
          "domain": "annotation-quality",
          "claim": "Showing annotators LLM suggestions makes them anchor on the suggestions - significantly shifting the label distribution versus unassisted baseline, raising self-reported confidence without making them faster - and evaluating models against LLM-assisted labels significantly inflates reported model performance.",
          "evidence": "Schroeder, Roy & Kabbara (Findings of ACL 2025, July 2025; pre-registered, 350 annotators, 7,000 annotations, 4 conditions x 2 models x 2 datasets, opened and verified): annotators 'strongly took the LLM suggestions'; the paper warns homogenization/lessened variation is a direct risk when LLMs are inserted into annotation loops, and that 'human-approved' LLM-assisted labels change downstream conclusions.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://llm-aggrefact.github.io/",
      "source_title": "LLM-AggreFact Leaderboard (MiniCheck project)",
      "source_date": "2025 (accessed 2026-07-14)",
      "source_type": "primary",
      "claims": [
        {
          "id": "GF-01",
          "domain": "grounding-faithfulness",
          "claim": "Grounded claim-verification is commoditized but ceilinged: the best model on the LLM-AggreFact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7B model, Bespoke-MiniCheck-7B, at 77.4 average balanced accuracy, with 0.4-0.8B specialized checkers (FactCG, MiniCheck-Flan-T5-L) within ~2.5 points of frontier LLMs like Claude-3.5-Sonnet (77.2) and GPT-4o (75.9).",
          "evidence": "Fetched the live LLM-AggreFact leaderboard: top-10 shows Bespoke-MiniCheck-7B 77.4, Claude-3.5-Sonnet 77.2, Granite Guardian 3.3 8B 76.5, FactCG-DeBERTa-L (0.4B) 75.6, MiniCheck-Flan-T5-L (0.8B) 75.0, Llama-3.1-405B 74.4. Benchmark measures binary supported/unsupported vs grounding docs. Confirmed independently by Paladin-mini (June 2025 arXiv) citing Bespoke-MiniCheck as leaderboard SOTA.",
          "in_corpus_verdict": "confirmed"
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2403.07008",
      "source_title": "AutoEval Done Right: Sample-Efficient Human Evaluation via Prediction-Powered Inference",
      "source_date": "2024-03 (v3 camera-ready 2026-06-01)",
      "source_type": "academic",
      "claims": [
        {
          "id": "HS-01",
          "domain": "hybrid-statistical",
          "claim": "Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a GPT-4 judge in 'AutoEval Done Right'.",
          "evidence": "arXiv:2403.07008 (Boyeau, Angelopoulos, Yosef, Malik, Jordan, UC Berkeley) abstract confirmed: 'increase the effective human-labeled sample size by up to 50% on experiments with GPT-4'; v3 camera-ready posted 2026-06-01. The unconditional-coverage property is restated and empirically demonstrated in the GLIDE paper (arXiv:2605.31278) and is the foundation of the entire 2026 LLM-judge debiasing literature (PRECISE/AAAI 2026, arXiv:2606.05308 ranking extension).",
          "in_corpus_verdict": "partially_confirmed"
        }
      ]
    },
    {
      "url": "https://arxiv.org/html/2505.08775v1",
      "source_title": "HealthBench: Evaluating Large Language Models Towards Improved Human Health (OpenAI)",
      "source_date": "2025-05-13",
      "source_type": "primary",
      "claims": [
        {
          "id": "RR-01",
          "domain": "rubrics-recent",
          "claim": "OpenAI HealthBench meta-evaluates its LLM grader per-criterion against physician majority grades using Macro F1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: GPT-4.1 grader MF1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician MF1 is only 0.569-0.730 with wide individual spread.",
          "evidence": "Full paper text (Section 8.1, Tables 5-6) confirmed: 60,896 meta-examples over 34 physician-consensus criteria; 'typical physician' baseline computed by scoring each physician against the others exactly as the model is scored; grader model choice matters (GPT-4.1 0.709 > o4-mini 0.692 > o3 0.681 > GPT-4.1-nano 0.580); prompt AND individual criterion phrasings were tuned 'so that their intent was unmistakable to the grader'. Overall benchmark std across 16 full runs is ~0.002.",
          "in_corpus_verdict": "confirmed"
        },
        {
          "id": "RR-02",
          "domain": "rubrics-recent",
          "claim": "HealthBench's production rubric format is per-item (conversation-specific) criteria written by 262 physicians (48,562 criteria), each an independently-judged binary met/not-met check with a nonzero weight from -10 to +10 (negative criteria encode pitfalls), aggregated as a weighted sum normalized by max attainable positive points.",
          "evidence": "Confirmed in fetched full text: 'Each rubric criterion has an associated nonzero point value between -10 and 10, with negative points used for criteria that are undesirable'; grader judges each criterion independently; item score can go negative; the vast majority of criteria were written specifically for one example.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "[link withheld]",
      "source_title": "Your verifier is probably the bottleneck. We built one that isn't. (the vendor's agent-judge verifier and banking verifier meta-eval)",
      "source_date": "2026-05-27",
      "source_type": "primary",
      "claims": [
        {
          "id": "IP-02",
          "domain": "industry-practice",
          "claim": "An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only F1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting F1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice.",
          "evidence": "Fetched the vendor's full research post (2026-05-27). The banking meta-eval is explicitly a meta-eval of verifiers, not agents; text-only 'autorubric' judges fail because evidence lives in spreadsheet formulas/tool state they cannot see; snapshot serialization loses formula references; the vendor's agent-judge (open-sourced, OpenHands SDK) opens files and follows references at inference time. Cheapest agent-judge config (~$42) beat the best baseline (~$422) by ~3 F1 at one-tenth cost; within one model family the architecture gap was 9.5 F1. They prescribe measuring every verifier against adjudicated expert labels before trusting it to score work or produce rewards.",
          "in_corpus_verdict": "confirmed"
        },
        {
          "id": "F26-02",
          "domain": "frontier-2026",
          "claim": "An AI data vendor's agent-judge verifier (May 27, 2026) shows verifier ARCHITECTURE beats verifier MODEL: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on F1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models.",
          "evidence": "Full primary post opened and read. The banking meta-eval fixes completed rollouts and rubrics and compares automated verifiers against practicing-banker labels; baselines were Autorubric (text-only rubric judge), Archipelago/APEX snapshot grading, and Agent-as-a-Judge. Key framing: 'verifiability is a relationship between a criterion and the verifier available to check it' -- a criterion invisible to a text-only judge is effectively ungraded, and weak verifiers push benchmark/rubric design toward only what they can reach. The vendor's agent-judge verifier code is open-sourced; the banking meta-eval dataset release planned.",
          "in_corpus_verdict": "confirmed"
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2410.13341",
      "source_title": "Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data",
      "source_date": "2024-10-17",
      "source_type": "academic",
      "claims": [
        {
          "id": "CT-03",
          "domain": "contrarian",
          "claim": "There is a proven theoretical ceiling on judge-based validation: when the judge is no more accurate than the model/content being evaluated, no debiasing method using gold labels can cut the required amount of ground-truth data by more than a factor of two, and empirical savings are smaller than the 2x bound.",
          "evidence": "Dorner, Nastl & Hardt (ICLR 2025) prove the bound for methods that combine cheap judge scores with a small gold-label set to correct judge biases such as self-preference; verified from the arXiv abstract page including the central quote 'when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth' by more than half.",
          "in_corpus_verdict": "confirmed"
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2607.05391",
      "source_title": "LLM-as-a-Verifier: A General-Purpose Verification Framework (arXiv 2607.05391)",
      "source_date": "2026-07-06",
      "source_type": "academic",
      "claims": [
        {
          "id": "F26-07",
          "domain": "frontier-2026",
          "claim": "LLM-as-a-Verifier (July 6, 2026; Mirhoseini/Finn/Pavone/Stoica groups) reframes verification as a scaling axis: computing the expectation over scoring-token logits yields continuous scores that improve monotonically along three dimensions -- finer score granularity, repeated evaluation, and criteria decomposition -- reaching reported SOTA on verification benchmarks (Terminal-Bench V2 86.5%, SWE-Bench Verified 78.2%, RoboRewardBench 87.4%, MedAgentBench 73.3%) without any judge training.",
          "evidence": "arXiv abstract fetched and verified (2607.05391, v2 July 7, 2026). Finer granularity separates good from bad solutions better; repeated evaluation and decomposition boost accuracy via variance and complexity reduction; includes a cost-efficient candidate-ranking algorithm and progress-estimation use.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://aipolicylab.se/2025/03/25/potential-impact-of-the-eu-platform-work-directive-on-ai-labelers/",
      "source_title": "Directive (EU) 2024/2831 Art 2 & 7 (EUR-Lex) + AI Policy Lab, 'Potential impact of the EU Platform Work Directive on AI labelers'",
      "source_date": "2025-03-25",
      "source_type": "academic",
      "claims": [
        {
          "id": "CG-09",
          "domain": "critic-and-gapfill",
          "claim": "The directive's scope explicitly covers online-performed work - the 'digital labour platform' definition applies 'irrespective of whether that work is performed online or in a certain location' - recital 19 names tagging/crowdwork, and Chapter III algorithmic-management rules (Arts 7-11) apply to genuinely self-employed persons performing platform work, from the start of recruitment, regardless of where the platform is established.",
          "evidence": "Definition verified verbatim from EUR-Lex (Art 2: service provided at a distance by electronic means, at the recipient's request, involving organisation of work for payment, using automated monitoring/decision systems). Article 7(2) verified: 'shall apply to all persons performing platform work from the start of the recruitment or selection procedure.' Chapter III's application to the genuinely self-employed rests on the Art 16(2) TFEU data-protection legal basis (ETUI, Countouris & De Stefano 2025). Lund AI Policy Lab (Mar 2025) analysis specifically concludes AI labelers are covered. Extraterritoriality confirmed by Ius Laboris and Omnivoo (2026): applies to platforms established outside the EU when the work is performed in the EU.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full",
      "source_title": "The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research",
      "source_date": "2020-01-22",
      "source_type": "academic",
      "claims": [
        {
          "id": "CG-22",
          "domain": "critic-and-gapfill",
          "claim": "In the largest educational feedback meta-analysis (435 studies, k=994, N>61,000), information content is the dominant moderator: high-information feedback (task + process + self-regulation content) yields d=0.99 [0.82-1.15] versus d=0.46 for corrective right/wrong feedback and d=0.24 for bare reinforcement/punishment - with overall d=0.48 masking huge heterogeneity (I2=83%) and 17% of raw effects negative.",
          "evidence": "Fetched the open-access Frontiers article (Wisniewski, Zierer & Hattie 2020). Feedback-type moderator significant (QB=41.52, p<0.0001); outcome moderator significant: cognitive d=0.51 vs motivational d=0.33; of negative motivational effects, 86% came from uninformative (reward/punishment-style) feedback. Authors conclude feedback 'cannot be understood as a single consistent form of treatment.'",
          "in_corpus_verdict": null
        }
      ]
    }
  ]
}