{
  "batch": 7,
  "checked_at": "2026-07-15",
  "sources": [
    {
      "url": "https://en.papernotes.org/ACL2025/llm_nlp/llm_vs_human_judges_study/",
      "retrieval": "fulltext",
      "fetched": ["https://en.papernotes.org/ACL2025/llm_nlp/llm_vs_human_judges_study/"],
      "resolved_title": "LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (ACL 2025 paper note)",
      "resolved_date": "2026-05-08",
      "title_match": true,
      "claims": [
        {
          "id": "AJ-07",
          "verdict": "confirmed",
          "checked": [
            "Judge-Bench with 20 datasets (70k+ instances), 11 LLM judges",
            "GPT-4o overall leader",
            "GPT-4o Categorical Avg kappa = 0.28 +/- 0.32 (Table 1)",
            "GPT-4o Graded Avg rho = 0.50 +/- 0.21 (Table 1)",
            "higher correlation with non-expert than expert evaluators",
            "every model struggles on engagingness",
            "safety guardrail refusals -> negative correlations",
            "task-specific human validation remains critical"
          ],
          "not_visible": ["author name 'Bavaresco' (not present on this secondary paper-note page)"],
          "quote": "Table 1 lists GPT-4o Categorical Avg kappa as 0.28 +/- 0.32 ... GPT-4o Graded Avg rho as 0.50 +/- 0.21. 'Models exhibit higher correlation with non-expert evaluators.' 'Achieves negative correlations on DICES and Medical-safety due to overactive guardrails.'",
          "notes": "Secondary source (a paper note dated 2026-05-08 discussing the ACL 2025 paper, arXiv 2406.18403 cited on the page); the claim self-identifies as read via a paper note. All headline numbers (kappa 0.28+/-0.32, rho 0.50+/-0.21) and the non-expert>expert and safety-refusal findings match verbatim. source_date '2025-07 (ACL 2025)' is the paper's venue; page metadata date is 2026-05-08."
        }
      ]
    },
    {
      "url": "https://arxiv.org/pdf/2601.22336",
      "retrieval": "abstract",
      "fetched": ["https://arxiv.org/abs/2601.22336"],
      "resolved_title": "Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models",
      "resolved_date": "2026-01-29",
      "title_match": true,
      "claims": [
        {
          "id": "AQ-07",
          "verdict": "partially_confirmed",
          "checked": [
            "classical methods assume annotators are conditionally independent",
            "assumption often violated by LLM judges due to shared data, architectures, prompts, and failure modes",
            "ignoring dependencies yields miscalibrated posteriors and even confidently incorrect predictions",
            "dependence-aware aggregation via Ising graphical models",
            "Dawid-Skene named"
          ],
          "not_visible": [
            "MACE (2013)",
            "'spammer' latent variable",
            "GLAD",
            "Dawid-Skene '1979' date",
            "Toloka crowd-kit production implementations"
          ],
          "quote": "Most classical methods, e.g., Dawid-Skene or (weighted) majority voting, assume annotators are conditionally independent ... Ignoring such dependencies can yield miscalibrated posteriors and even confidently incorrect predictions.",
          "notes": "Both direct quotes the claim attributes to the paper are confirmed verbatim, and the Ising-model core is confirmed. Retrieved abstract lists CI examples as 'Dawid-Skene or (weighted) majority voting' only; MACE, GLAD, the spammer latent variable, and the crowd-kit implementation note are the claim author's additions and are not visible in retrieved text. Submitted 2026-01-29 (source_date 2026-02-02)."
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2501.14492",
      "retrieval": "abstract",
      "fetched": ["https://arxiv.org/abs/2501.14492"],
      "resolved_title": "RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques",
      "resolved_date": "2025-01-24",
      "title_match": true,
      "claims": [
        {
          "id": "CM-08",
          "verdict": "partially_confirmed",
          "checked": [
            "closed-loop methodology evaluating quality of corrections generated from critiques",
            "eight challenging reasoning tasks",
            "classical LLMs significantly lag o1-mini across all critique scenarios",
            "classical LLMs can fall below their own baselines under self/iterative critique"
          ],
          "not_visible": [
            "CriticBench-THU 24.8%",
            "27.1-point error-identification gap",
            "ProcessBench o1-mini 88.9 vs Qwen2.5-72B 61.8",
            "1.3-point verdict-F1 gap",
            "-1.8 to -5.1 avg self-critique delta",
            "-35.6 domain drop",
            "'superficial success' failure-mode name",
            "'contradictory output' failure-mode name",
            "RM-NLHF / arXiv 2601.07349 (separate source)"
          ],
          "quote": "our approach employs a closed-loop methodology that evaluates the quality of corrections generated from critiques ... classical LLMs significantly lag behind the advanced reasoning-based model o1-mini across all critique scenarios",
          "notes": "RealCritic's core design (closed-loop critique-then-correct over 8 reasoning tasks; o1-mini only model with positive self-critique; classical LLMs drop below baselines) is confirmed. However essentially every headline number in the claim is drawn from OTHER benchmarks (CriticBench-THU 24.8%, ProcessBench 88.9/61.8) or from RealCritic detailed results not present in the abstract; none are visible in the retrieved text."
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2506.01829",
      "retrieval": "abstract",
      "fetched": ["https://arxiv.org/abs/2506.01829"],
      "resolved_title": "CiteEval: Principle-Driven Citation Evaluation for Source Attribution",
      "resolved_date": "2025-06-02",
      "title_match": true,
      "claims": [
        {
          "id": "GF-09",
          "verdict": "confirmed",
          "checked": [
            "NLI binary/ternary supportiveness judged 'a suboptimal proxy for citation evaluation'",
            "evaluates full retrieval context, user query, and generated text (not cited sources alone)",
            "CiteBench multi-domain human-annotated benchmark",
            "CiteEval-Auto model-based metrics with strong correlation to human judgments"
          ],
          "not_visible": [
            "'ALCE' (not named on page)",
            "explicit head-to-head numeric comparison 'better than NLI-based metrics' (entailed by 'suboptimal proxy' + 'strong correlation' but not stated as a number)",
            "CiteGuard (ACL 2026) corroboration (separate source, out of scope)"
          ],
          "quote": "current frameworks mainly rely on Natural Language Inference (NLI) to assess binary or ternary supportiveness from cited sources, which we argue is a suboptimal proxy for citation evaluation ... not only the cited sources but the full retrieval context, user query, and generated text",
          "notes": "Core assertion and every substantive element (suboptimal NLI proxy, full retrieval context, CiteBench, CiteEval-Auto human-correlation) confirmed verbatim in the abstract."
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2604.15302",
      "retrieval": "abstract",
      "fetched": ["https://arxiv.org/abs/2604.15302"],
      "resolved_title": "Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations",
      "resolved_date": "2026-04-16",
      "title_match": true,
      "claims": [
        {
          "id": "HS-09",
          "verdict": "partially_confirmed",
          "checked": [
            "conformal set width is a per-instance reliability indicator, r_s = +0.576, N = 1,918",
            "cross-judge width agreement r-bar = 0.32-0.38",
            "relevance avg set size ~3.0 vs fluency/consistency ~4.9",
            "criterion matters more than the judge model",
            "split conformal prediction over 1-5 Likert scores on SummEval"
          ],
          "not_visible": [
            "EMNLP 2025 Sheng/Liu/He/Zhao/Kang paper: guaranteed-coverage intervals from a single judging run, ordinal boundary adjustment, lower-bias midpoint score (this is a DIFFERENT paper at aclanthology.org/2025.emnlp-main.569, not fetchable from this arXiv URL)"
          ],
          "quote": "set width serving as a per-instance reliability indicator (r_s = +0.576, N = 1,918, p < 10^-100) ... consistent cross-judge agreement (r-bar = 0.32-0.38)",
          "notes": "The claim bundles TWO papers; only the arXiv:2604.15302 (April 2026, Gupta & Kumar) half is at the fetched URL and it is fully confirmed. The EMNLP 2025 conformal-interval work (Sheng et al.) with the ordinal boundary adjustment and lower-bias midpoint score is a separate source not retrievable here, hence its specifics are not visible."
        },
        {
          "id": "F26-05",
          "verdict": "confirmed",
          "checked": [
            "relevance most reliable, avg conformal set size ~3.0 of 5",
            "coherence moderate ~3.9",
            "fluency and consistency unreliable ~4.9",
            "aggregate transitivity violations 0.8-4.1%",
            "33-67% of documents contain at least one directed 3-cycle (intransitive cycle)",
            "conformal set width correlates with reliability r_s = +0.576, N = 1,918",
            "cross-judge width agreement 0.32-0.38",
            "four judges, four criteria on SummEval; criterion > judge"
          ],
          "not_visible": ["verbatim word 'intransitive' (paper expresses it as 'directed 3-cycle')"],
          "quote": "relevance judged most reliably (avg. set size approx 3.0) ... fluency and consistency remain unreliable (avg. set size approx 4.9) ... low aggregate violation rates (rho-bar = 0.8-4.1%) ... 33-67% of documents exhibiting at least one directed 3-cycle",
          "notes": "All headline numbers confirmed verbatim in the retrieved abstract. 'Intransitive preference cycle' corresponds to the paper's 'directed 3-cycle'."
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2602.01511",
      "retrieval": "abstract",
      "fetched": ["https://arxiv.org/abs/2602.01511"],
      "resolved_title": "Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training",
      "resolved_date": "2026-02-02",
      "title_match": true,
      "claims": [
        {
          "id": "RR-10",
          "verdict": "partially_confirmed",
          "checked": [
            "Rubric-ARM jointly optimizes a rubric generator and a judge via reinforcement learning from preference feedback",
            "prior methods 'rely on static rubrics or disjoint training pipelines'",
            "alternating optimization strategy to mitigate non-stationarity",
            "reduces gradient variance; SOTA on benchmarks and improved policy alignment"
          ],
          "not_visible": [
            "'adaptability' wording (not found on page)",
            "'co-evolve' framing",
            "'exploitable reward specification under optimization pressure' framing",
            "EvoRubrics, ARBOR, AMARIS (separate 2026 papers, not at this URL; claim states these are search-only)"
          ],
          "quote": "jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback ... Unlike existing methods that rely on static rubrics or disjoint training pipelines",
          "notes": "Only Rubric-ARM is at this URL and its mechanism is confirmed verbatim. The 'cluster of 2026 papers converging' (EvoRubrics/ARBOR/AMARIS) is not verifiable here and the claim itself flags it as search-only. Actual title lacks the 'Rubric-ARM:' prefix and the '(plus 2026 co-evolution cluster)' suffix of source_title, but the substantive title matches. v1 2026-02-02, v2 2026-02-11."
        }
      ]
    },
    {
      "url": "https://deepeval.com/docs/metrics-dag",
      "retrieval": "fulltext",
      "fetched": ["https://deepeval.com/docs/metrics-dag"],
      "resolved_title": "DAG (Deep Acyclic Graph)",
      "resolved_date": "not shown on docs page",
      "title_match": true,
      "claims": [
        {
          "id": "IP-10",
          "verdict": "partially_confirmed",
          "checked": [
            "DAG lets you 'easily build deterministic decision trees for evaluation'",
            "positioned above G-Eval: 'the DAGMetric will give you much greater control'",
            "'more deterministic control' over GEval; produces deterministic scores",
            "nodes are LLM-judge calls (BinaryJudgementNode, NonBinaryJudgementNode; 'reduce LLM-judge variance')",
            "breakdown into more atomic units for evaluation"
          ],
          "not_visible": [
            "'conversational' variant for multi-turn work (word not found on page)",
            "Confident-AI 2025-02-09 engineering post ('holistic LLM scores too noisy') - separate URL, not fetched",
            "exact phrase 'for more control' (page says 'much greater control' / 'more deterministic control')",
            "docs page publication/last-updated date"
          ],
          "quote": "You can still use GEval in the DAGMetric, but the DAGMetric will give you much greater control ... easily build deterministic decision trees for evaluation",
          "notes": "Core G-Eval -> DAG progression and decomposed/deterministic-node judging confirmed on the docs page. The asserted 'conversational variant' and the Confident-AI blog motivation (source_date 2025-02-09) are not verifiable from this docs URL."
        }
      ]
    },
    {
      "url": "https://openreview.net/forum?id=mzh9d3dooN",
      "retrieval": "failed",
      "fetched": ["https://openreview.net/forum?id=mzh9d3dooN", "https://api.openreview.net/notes?forum=mzh9d3dooN"],
      "resolved_title": null,
      "resolved_date": null,
      "title_match": false,
      "claims": [
        {
          "id": "F26-01",
          "verdict": "not_retrievable",
          "checked": [],
          "not_visible": [
            "68% -> 9% detection drop",
            "44% -> 88% false-positive rise",
            "80% buggy-evaluator detection",
            "200 web-agent items / 2,400 audits / 3 production models",
            "re-solving mechanism"
          ],
          "quote": "",
          "notes": "Both the forum page and the OpenReview API return a Cloudflare Turnstile 'Verifying your browser' challenge (API: HTTP 403 ChallengeRequiredError). No paper content retrievable via WebFetch or curl. The batch file carried a pre-set in_corpus_verdict of 'confirmed', but nothing could be independently verified from this source."
        }
      ]
    },
    {
      "url": "https://doi.org/10.1186/s40468-021-00134-5",
      "retrieval": "fulltext",
      "fetched": ["https://doi.org/10.1186/s40468-021-00134-5", "https://languagetestingasia.springeropen.com/articles/10.1186/s40468-021-00134-5"],
      "resolved_title": "The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution",
      "resolved_date": "2021-10-01",
      "title_match": true,
      "claims": [
        {
          "id": "CG-04",
          "verdict": "confirmed",
          "checked": [
            "student-by-task-by-method (STM:B) = 31.8% of total variance",
            "student-by-rater-by-task-by-method (SRTM:B) = 26.5% of total variance",
            "student-by-rater-by-method (SRM:B) = 17.6% of total variance",
            "these three interactions are the major/largest variance sources",
            "student-by-rater (SR:B) = 99.2% of error variance (rater-by-background = 0.8%)",
            "120 students ('One hundred and twenty students, 90 females and 30 males')",
            "EFL learners; G-theory / generalizability theory"
          ],
          "not_visible": ["the literal labeled term 'rater main effect' (rater-related variance appears as SR/RB interactions; RB ~8% of error variance)"],
          "quote": "the student by task by method of scoring (nested in background of education) interaction (STM:B) with 31.8% contribution to the total variance ... (SR:B) and rater by background of education with 99.2% and 0.8% contribution to the error variance",
          "notes": "All four headline percentages confirmed with their exact interaction mappings, plus the 120-student sample. The claim's interpretation that the rater MAIN effect is minor is supported: dominant components are all high-order student x rater x task x method interactions, and 99.2% of error variance is the student-by-rater interaction; severity/bias shows up as the SR/RB interaction (~8% of error). Reached via server-redirect chain (doi -> springeropen BMC host) fetched by curl; WebFetch on the springer.com leg bounced to an idp auth wall. Published 2021-10-01 (source_date 2021-11)."
        }
      ]
    },
    {
      "url": "https://theconversation.com/hard-labour-conditions-of-online-moderators-directly-affect-how-well-the-internet-is-policed-new-study-261386",
      "retrieval": "fulltext",
      "fetched": ["https://theconversation.com/hard-labour-conditions-of-online-moderators-directly-affect-how-well-the-internet-is-policed-new-study-261386"],
      "resolved_title": "Hard labour conditions of online moderators directly affect how well the internet is policed - new study",
      "resolved_date": "2025-07-22",
      "title_match": true,
      "claims": [
        {
          "id": "CG-18",
          "verdict": "confirmed",
          "checked": [
            "removing posts required only two steps; de-ranking involved four steps",
            "moderator skipped/avoided de-ranking to save time ('Would never recommend de-ranking content as it would take time')",
            "removing tool-flagged words without evaluating context",
            "moderators develop a simplified list of 'dos and don'ts'",
            "commercial content moderators in India; screen-share observation",
            "study published in New Media & Society; author Tania Chatterjee"
          ],
          "not_visible": [
            "literal word 'clicks' (article says 'steps': two steps vs four steps)",
            "literal word 'throughput' (concept present via targets/time pressure)",
            "quantitative error rates (none - study is qualitative, as the claim acknowledges)"
          ],
          "quote": "removing posts required only two steps ... reducing the visibility of content (de-ranking) involved four steps ... Would never recommend de-ranking content as it would take time.",
          "notes": "Core mechanism confirmed: the action-selection UI (2-step removal vs 4-step de-ranking) plus throughput pressure shifted outcomes independent of policy fit ('To save time, she skipped the content flagged to be de-ranked'; 'removing flagged words without evaluating the context'; 'dos and don'ts'). Only wording nuance: article says 'steps' not 'clicks'."
        }
      ]
    }
  ]
}