{ "batch": 11, "checked_at": "2026-07-15", "sources": [ { "url": "https://arxiv.org/abs/2410.20266", "retrieval": "abstract", "fetched": ["https://arxiv.org/abs/2410.20266"], "resolved_title": "Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks", "resolved_date": "2024-10-26 (submitted; Comments cite ACM IUI 2025)", "title_match": true, "claims": [ { "id": "AJ-11", "verdict": "confirmed", "checked": [ "68% SME-LLM agreement in dietetics (registered dietitians)", "64% SME-LLM agreement in mental health (clinical psychologists)", "overall pairwise preference comparison; agreement varied across domain-specific aspect questions", "two fields: dietetics with registered dietitian experts, mental health with clinical psychologist experts", "conclusion: keep human experts in the evaluation loop; LLMs alone lack depth for complex knowledge-specific tasks" ], "not_visible": [], "quote": "SMEs agreed with LLM judges 68% of the time in the dietetics domain and 64% in mental health", "notes": "Abstract page fully supports the claim's core assertion and both headline percentages. Title and Oct-2024 date match; IUI 2025 venue appears in the Comments field." } ] }, { "url": "https://aclanthology.org/2025.findings-acl.1323/", "retrieval": "abstract", "fetched": ["https://aclanthology.org/2025.findings-acl.1323/"], "resolved_title": "Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks", "resolved_date": "July 2025 (Findings of ACL 2025; pp. 25771-25795)", "title_match": true, "claims": [ { "id": "AQ-11", "verdict": "confirmed", "checked": [ "annotators anchored on LLM suggestions, significantly changing label distribution vs baseline", "did NOT make them faster but improved self-reported confidence", "using LLM-assisted labels for evaluation significantly increases reported model performance", "pre-registered experiment", "350 unique annotators, 7,000 annotations, 4 conditions, 2 models, 2 datasets" ], "not_visible": [ "'homogenization/lessened variation is a direct risk' framing (from the evidence narrative) is not present in the abstract; abstract instead says label-distribution shifts can affect conclusions drawn even from human-approved datasets" ], "quote": "annotators strongly took the LLM suggestions, significantly changing the label distribution compared to the baseline", "notes": "Every core assertion and headline number in the claim is confirmed by the abstract. Only the 'homogenization' phrasing (which is in the verifier's evidence rationale, not the claim text) is not visible in the abstract; it may appear in the full paper (not fetched)." } ] }, { "url": "https://llm-aggrefact.github.io/", "retrieval": "landing", "fetched": ["https://llm-aggrefact.github.io/"], "resolved_title": "LLM-AggreFact Leaderboard", "resolved_date": "no explicit date (live leaderboard; accessed 2026-07-15)", "title_match": true, "claims": [ { "id": "GF-01", "verdict": "confirmed", "checked": [ "Bespoke-MiniCheck-7B tops the board at 77.4 average", "Claude-3.5 Sonnet 77.2", "Granite Guardian 3.3 8B 76.5", "gpt-4o-2024-05-13 75.9", "FactCG-DeBERTa-L (0.4B) 75.6", "MiniCheck-Flan-T5-L (0.8B) 75.0", "Llama-3.1-405B-Instruct 74.4", "11 datasets; grounded factuality / hallucination (claim supported vs source documents)" ], "not_visible": [], "quote": "Aggregates 11 datasets on grounded factuality (i.e., hallucination) evaluation", "notes": "All seven cited leaderboard numbers and the 11-dataset scope match exactly. 0.4-0.8B specialized checkers (75.6, 75.0) are within ~2.4 points of top model (77.4), consistent with the claim's '~2.5 points'." } ] }, { "url": "https://arxiv.org/abs/2403.07008", "retrieval": "fulltext", "fetched": ["https://arxiv.org/abs/2403.07008", "https://arxiv.org/pdf/2403.07008", "https://arxiv.org/html/2403.07008v3"], "resolved_title": "AutoEval Done Right: Using Synthetic Data for Model Evaluation", "resolved_date": "v1 2024-03-09; v3 (camera-ready) 2026-06-01", "title_match": false, "claims": [ { "id": "HS-01", "verdict": "partially_confirmed", "checked": [ "core method is prediction-powered inference (PPI) and the optimized variant PPI++", "combines a small amount of human data with a large amount of synthetic/judge data to increase effective human sample size without compromising validity", "estimators remain unbiased; PPI/PPI++ yield valid, calibrated confidence intervals across all labeled set sizes", "robust to judge quality: even with a very poor annotator model, PPI++ performs at least as well as the classical approach (worse judge widens, never invalidates)", "v3 camera-ready dated 2026-06-01" ], "not_visible": [ "the specific figure '50% with a GPT-4 judge': the LLM/judge experiment used gpt-4o-mini and gave only 20-35% ESS improvement; the ~50% ESS gain came from the ImageNet and protein-fitness experiments, not the GPT-judge experiment", "GLIDE (arXiv:2605.31278), PRECISE/AAAI 2026, arXiv:2606.05308 -- separate papers, not part of this source" ], "quote": "even with a very poor annotator model, PPI++ performs at least as well as the classical approach", "notes": "TITLE MISMATCH: source_title says 'Sample-Efficient Human Evaluation via Prediction-Powered Inference' but the actual (v3) title is 'Using Synthetic Data for Model Evaluation'. PPI/PPI++, unbiasedness, valid/calibrated CIs, and coverage-robustness-to-judge-quality are all directly confirmed in the full HTML text. The abstract does say 'up to 50% on experiments with GPT-4', but the body attributes ~50% ESS to ImageNet/protein experiments while the actual LLM/GPT-style judge (gpt-4o-mini) experiment yielded only 20-35% -- so the '50% with a GPT-4 judge' specificity is imprecise. Kept at partially_confirmed for that discrepancy plus the title mismatch." } ] }, { "url": "https://arxiv.org/html/2505.08775v1", "retrieval": "fulltext", "fetched": ["https://arxiv.org/html/2505.08775v1"], "resolved_title": "HealthBench: Evaluating Large Language Models Towards Improved Human Health", "resolved_date": "2025-05-13 (arXiv 2505.08775v1)", "title_match": true, "claims": [ { "id": "RR-01", "verdict": "confirmed", "checked": [ "per-criterion Macro F1 (met/not-met, class-balanced) meta-evaluation of the grader", "GPT-4.1 grader MF1 = 0.709; exceeds average physician score in five out of seven themes (verbatim)", "physician-vs-physician weighted MF1 range 0.569 (Response depth) to 0.730 (Health data tasks)", "o4-mini 0.692, o3 0.681, GPT-4.1-nano 0.580 (GPT-4.1-mini 0.661)", "over 60,896 meta-examples", "typical-physician baseline: each physician scored against the others, excluding their own grades", "std ~0.002 across 16 full runs (Table 7: 0.0016-0.0029)" ], "not_visible": [], "quote": "It \"exceeds the average physician score in five out of seven themes\" — verbatim confirmed", "notes": "All eight cited numbers match verbatim. NUANCE: the grading ground truth is each INDIVIDUAL physician's grade, not a majority-vote label; majority agreement was used earlier to assign consensus criteria to examples, not as the grading target. The claim's phrase 'physician majority grades' is therefore a minor mischaracterization, but does not affect any headline number or the core method." }, { "id": "RR-02", "verdict": "confirmed", "checked": [ "rubrics created by 262 physicians", "48,562 unique criteria across all conversations; conversation-specific", "each criterion nonzero point value between -10 and 10; negative for undesirable/pitfalls", "grader judges each criterion independently, binary met/not-met", "aggregated as weighted sum divided by maximum possible score; per-example score can go negative" ], "not_visible": [], "quote": "an associated nonzero point value between -10 and 10, with negative points used for criteria that are undesirable", "notes": "Full text confirms all elements: 262 physicians, 48,562 criteria, -10..10 point range, independent binary judging, weighted-sum normalization by max attainable, and negative-capable item scores." } ] }, { "url": "[link withheld]", "retrieval": "landing", "fetched": ["[link withheld]"], "resolved_title": "Your verifier is probably the bottleneck. We built one that isn't. (the vendor's agent-judge verifier and banking verifier meta-eval)", "resolved_date": "not surfaced verbatim in fetch (claimed 2026-05-27); content references 2026-era models (GPT-5.4, Gemini 3 Pro)", "title_match": true, "claims": [ { "id": "IP-02", "verdict": "confirmed", "checked": [ "The banking meta-eval is a meta-eval of verifiers, not an agent benchmark", "3,204 expert-graded (practicing-banker) pass/fail judgments", "inter-annotator agreement 89.5%; disagreements adjudicated; a subset dual-coded", "The vendor's agent-judge F1 range 0.633-0.664 on artifact-heavy banking tasks", "The vendor's agent-judge F1 0.951 on OpenClaw (stateful tools), 7.5 points ahead of next-best", "Cheapest agent-judge (GPT-5.4 Nano ~$42) beats best baseline (Archipelago/Gemini 3 Pro, F1 0.604, ~$422) by ~3 F1 at ~one-tenth cost", "within-family architecture gap 9.5 F1 (agent-judge/GPT-5.4 Nano vs Archipelago/GPT-5.4)", "The vendor's agent-judge is a reactive agent-as-judge running inside the rollout environment; open-sourced; OpenHands SDK harness", "verifier architecture (view + path) matters more than backing model" ], "not_visible": [], "quote": "F1 range, from 0.633 to 0.664, sits above the highest-F1 non-agent-judge run", "notes": "Every cited figure is present in the post. Minor framing: 89.5% is the overall inter-annotator agreement (with a separate dual-coded subset for rubric-application consistency), which is consistent with the claim's wording." }, { "id": "F26-02", "verdict": "confirmed", "checked": [ "The vendor's banking verifier meta-eval fixes completed rollouts and rubrics and compares automated verifiers to practicing-banker labels", "3,204 expert-graded judgments; 21 tasks sampled from the vendor's banking agent benchmark", "inter-annotator agreement 89.5%; disagreements adjudicated", "reactive agent-judge runs inside the environment and chooses at inference time what evidence to inspect; beats text-only/snapshot/workflow verifiers on F1 at ~10x lower cost", "gap between verifier architectures larger than gap between backing models", "baselines: Autorubric (text-only), Archipelago from APEX (snapshot), Agent-as-a-Judge (AAAJ)", "framing: verifiability is a relationship between a criterion and the verifier available to check it", "The vendor's agent-judge verifier code open-sourced" ], "not_visible": [ "'the banking meta-eval dataset release planned' -- not explicitly surfaced in the fetched summary (the vendor's agent-judge code open-sourcing is confirmed; the banking meta-eval dataset release status not quoted)" ], "quote": "verifiability is not only a property of the task. It is a relationship between a criterion and the verifier available to check it", "notes": "Architecture-over-model thesis, all counts (3,204 / 21 / 89.5%), baseline names, and the framing quote confirmed. The '9.5 F1' architecture gap and ~10x cost advantage are supported (see IP-02 checked items)." } ] }, { "url": "https://arxiv.org/abs/2410.13341", "retrieval": "abstract", "fetched": ["https://arxiv.org/abs/2410.13341"], "resolved_title": "Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data", "resolved_date": "v1 2024-10-17; v3 2026-01-06 (Comments: ICLR 2025)", "title_match": true, "claims": [ { "id": "CT-03", "verdict": "confirmed", "checked": [ "proven bound on label savings from judge-based debiasing", "when the judge is no more accurate than the evaluated model, no debiasing method can decrease required ground truth by more than half (factor of two)", "'won't beat twice the data' appears verbatim in the title", "self-preferencing named as a distorting bias", "empirical/practical savings even more modest than the theoretical limit", "authors Dorner, Nastl, Hardt; ICLR 2025" ], "not_visible": [], "quote": "no debiasing method can decrease the required amount of ground truth labels by more than half", "notes": "Core assertion (2x ceiling), the no-more-accurate-than-evaluated-model condition, self-preferencing, and 'empirical savings smaller than the bound' all confirmed from the abstract. Authors and ICLR 2025 venue match." } ] }, { "url": "https://arxiv.org/abs/2607.05391", "retrieval": "abstract", "fetched": ["https://arxiv.org/abs/2607.05391"], "resolved_title": "LLM-as-a-Verifier: A General-Purpose Verification Framework", "resolved_date": "v1 2026-07-06; v2 2026-07-07", "title_match": true, "claims": [ { "id": "F26-07", "verdict": "confirmed", "checked": [ "reframes verification as a new scaling axis", "computes the expectation over the distribution of scoring-token logits to generate continuous scores", "three dimensions: (1) score granularity, (2) repeated evaluation, (3) criteria decomposition", "SOTA: Terminal-Bench V2 86.5%, SWE-Bench Verified 78.2%, RoboRewardBench 87.4%, MedAgentBench 73.3%", "no additional judge training required", "cost-efficient candidate-ranking (best-of-N) algorithm and task-progress estimation", "authors include Finn, Pavone, Stoica, Mirhoseini" ], "not_visible": [ "the word 'monotonically': the abstract says repeated evaluation and decomposition 'consistently lead to additional gains' -- 'monotonically improving' is the claim's paraphrase, not the abstract's wording" ], "quote": "Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%)", "notes": "All four SOTA numbers, the logit-expectation mechanism, the three scaling dimensions, the no-training property, and the named authors match. Only the 'monotonically' descriptor is a slight paraphrase of 'consistently lead to additional gains'." } ] }, { "url": "https://aipolicylab.se/2025/03/25/potential-impact-of-the-eu-platform-work-directive-on-ai-labelers/", "retrieval": "landing", "fetched": ["https://aipolicylab.se/2025/03/25/potential-impact-of-the-eu-platform-work-directive-on-ai-labelers/"], "resolved_title": "Potential impact of the EU Platform Work Directive on AI labelers", "resolved_date": "2025-03-25 (byline Mariia Lesina, Lund University)", "title_match": true, "claims": [ { "id": "CG-09", "verdict": "partially_confirmed", "checked": [ "the post concludes AI labelers/annotators are covered by the directive when work is conducted through a digital platform in the EU", "recital/intro 19 mentions tagging as a form of crowd work that can be conducted remotely (page labels it 'Article 19 of Introduction')" ], "not_visible": [ "verbatim Art 2 'digital labour platform' definition language ('irrespective of whether that work is performed online or in a certain location') -- not on this page (attributed to EUR-Lex, not fetched)", "Article 7(2) quote 'shall apply to all persons performing platform work from the start of the recruitment or selection procedure' -- does not appear on this page", "Chapter III / Articles 7-11 applying to genuinely self-employed persons from the start of recruitment -- not stated on this page (post discusses Arts 10 and 12 only)", "Article 16(2) TFEU data-protection legal basis -- not mentioned", "extraterritoriality (applies to platforms established outside the EU) -- attributed to Ius Laboris/Omnivoo, not on this page" ], "quote": "The EU Directive recognizes AI labeling as a form of platform work if it is conducted through a digital platform within the EU", "notes": "Only the AI Policy Lab page is fetchable per constraints. It supports the core conclusion (AI labelers are covered) and the tagging-as-crowdwork-in-recital-19 point. The many verbatim legal specifics in the compound claim (Art 2 definition, Art 7(2), Chapter III scope to self-employed, Art 16(2) TFEU, extraterritoriality) are attributed to EUR-Lex/ETUI/Ius Laboris/Omnivoo and are NOT present on this page. The page also conflates recitals with articles ('Article 19 of Introduction', 'Article 47/44'), weakening any verbatim-quote reliance. Partially_confirmed on the retrieved page alone." } ] }, { "url": "https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full", "retrieval": "fulltext", "fetched": ["https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full"], "resolved_title": "The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research", "resolved_date": "2020-01-22 (Frontiers in Psychology, Vol. 10-2019)", "title_match": true, "claims": [ { "id": "CG-22", "verdict": "confirmed", "checked": [ "435 studies, k = 994 effect sizes, N > 61,000", "high-information feedback d = 0.99 [0.82-1.15] (k=42)", "corrective right/wrong feedback d = 0.46 [0.39-0.55] (k=238)", "reinforcement/punishment d = 0.24 [0.06-0.43] (k=39)", "overall weighted d = 0.48 (outlier-cleaned; CL 0.44-0.51), I2 = 83.40%", "17% of raw effects negative (full set before outlier removal)", "feedback-type moderator QB = 41.52, df=2, p < 0.0001", "cognitive outcomes d = 0.51 vs motivational d = 0.33", "86% of negative motivational effects from uninformative (reward/punishment) feedback", "'feedback cannot be understood as a single consistent form of treatment' (verbatim)" ], "not_visible": [], "quote": "feedback cannot be understood as a single consistent form of treatment", "notes": "Every headline number matches. Minor caveat surfaced in text: d=0.48 is the outlier-adjusted estimate (initial integration d=0.55 with 17% negative); the claim uses the 0.48 cleaned figure and the 17%-negative full-set figure, both consistent with the paper." } ] } ] }