{ "batch": 7, "sources": [ { "url": "https://en.papernotes.org/ACL2025/llm_nlp/llm_vs_human_judges_study/", "source_title": "LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (JUDGE-BENCH)", "source_date": "2025-07 (ACL 2025)", "source_type": "academic", "claims": [ { "id": "AJ-07", "domain": "academic-judges", "claim": "The broadest LLM-vs-human-judge study on existing human-annotated datasets (JUDGE-BENCH, 20 datasets) found the best model (GPT-4o) reached only kappa = 0.28 +/- 0.32 on categorical judgments and Spearman rho = 0.50 +/- 0.21 on graded ones, with enormous task-to-task variance - and LLMs agreed MORE with non-expert annotators than with experts.", "evidence": "Bavaresco et al., 'LLMs instead of Human Judges?' (ACL 2025). Structured/rule-like attributes correlate reliably; subjective attributes (engagingness) poorly; safety judgments produced negative correlations due to guardrail refusals. Authors conclude task-specific human validation is essential before deploying LLM judges. Read via detailed paper-note secondary source; consistent with the ACL publication.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/pdf/2601.22336", "source_title": "Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models (arXiv 2601.22336)", "source_date": "2026-02-02", "source_type": "academic", "claims": [ { "id": "AQ-07", "domain": "annotation-quality", "claim": "Classical label-aggregation models (Dawid-Skene 1979, MACE 2013 with its explicit spammer latent variable, GLAD) all assume annotators are conditionally independent given the true label - an assumption measurably violated when the 'annotators' are LLM judges sharing data/architectures/prompts, causing miscalibrated posteriors and confidently wrong aggregate verdicts; Feb 2026 work replaces this with dependence-aware Ising-model aggregation.", "evidence": "Balasubramanian, Podkopaev & Kasiviswanathan (Amazon/UC Davis, arXiv 2601.22336, Feb 2 2026) state the CI assumption 'is often violated by LLM judges due to shared data, architectures, prompts, and failure modes' and that ignoring dependencies 'can yield miscalibrated posteriors and even confidently incorrect predictions'; they build aggregation via Ising models. MACE (Hovy et al., NAACL 2013) remains the canonical spam-aware human aggregator, with production implementations in Toloka's crowd-kit.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/abs/2501.14492", "source_title": "RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques", "source_date": "2025-01-24", "source_type": "academic", "claims": [ { "id": "CM-08", "domain": "critique-models", "claim": "Verdict accuracy and critique validity dissociate badly: in CriticBench-THU data 24.8% of items got the correct verdict with a low-quality critique, and open-loop verdict-agreement metrics compress a 27.1-point real error-identification gap (ProcessBench: o1-mini 88.9 vs Qwen2.5-72B 61.8) into a 1.3-point verdict-F1 gap - so critique quality must be evaluated closed-loop by whether the critique drives a successful correction.", "evidence": "RealCritic (Jan 2025, arXiv 2501.14492): closed-loop critique-then-correct design over 8 reasoning benchmarks; classical LLMs (incl. GPT-4) lose accuracy under self-critique (-1.8 to -5.1 avg; up to -35.6 domain drops) while o1-mini is the only model with positive self-critique delta; failure modes 'superficial success' (right verdict, wrong analysis) and 'contradictory output' documented. Independently confirmed at scale in 2026: an ICML 2026 paper (RM-NLHF, arXiv 2601.07349) finds outcome-rewarded generative reward models 'guess correct outcomes without sound critiques' and uses similarity-to-human-critique as a process reward to fix it. RealCritic verified via arXiv abstract + HTML (HTML render partially draft-quality; abstract and appendix numbers consistent).", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/abs/2506.01829", "source_title": "CiteEval: Principle-Driven Citation Evaluation for Source Attribution", "source_date": "2025-06", "source_type": "academic", "claims": [ { "id": "GF-09", "domain": "grounding-faithfulness", "claim": "Span-restricted NLI citation checking (the ALCE-style 'does the cited span entail the sentence' standard) is judged a suboptimal proxy by 2025 work: CiteEval (ACL 2025) argues citation quality must be evaluated against the FULL retrieval context, user query, and generated text - not cited sources alone - and its model-based CiteEval-Auto metrics correlate better with human judgments on the multi-domain CiteBench than NLI-based metrics.", "evidence": "Fetched arXiv:2506.01829 (CiteEval, ACL 2025 main). Explicitly frames binary/ternary NLI vs cited sources as 'a suboptimal proxy for citation evaluation'. Corroborated direction: CiteGuard (ACL 2026, aclanthology.org/2026.acl-long.282) reframes citation evaluation as attribution alignment and notes 'the reliability of LLM-as-a-Judge alone is also in doubt' for citation judging.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/abs/2604.15302", "source_title": "Analyzing Uncertainty of LLM-as-a-Judge (EMNLP 2025) + Diagnosing LLM Judge Reliability (arXiv:2604.15302)", "source_date": "2025-11 (EMNLP) / 2026-04-16", "source_type": "academic", "claims": [ { "id": "HS-09", "domain": "hybrid-statistical", "claim": "Conformal prediction now provides per-item uncertainty for LLM-as-judge ratings: EMNLP 2025 work builds guaranteed-coverage score intervals from a single judging run (with an ordinal boundary adjustment and a lower-bias midpoint score), and April 2026 work shows conformal set width is a genuine per-instance reliability signal (r_s=+0.576 with reliability, n=1,918; widths correlate ~0.32-0.38 across different judges) with reliability driven more by the CRITERION than the judge model (relevance avg set size ~3.0 vs fluency/consistency ~4.9 on a 1-5 scale).", "evidence": "Sheng, Liu, He, Zhao, Kang, 'Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction', EMNLP 2025 main (aclanthology.org/2025.emnlp-main.569), abstract confirmed via NeurIPS 2025 workshop listing. Gupta & Kumar, arXiv:2604.15302 (2026-04-16, under review): split conformal on SummEval; also finds 33-67% of documents contain intransitive judge preference cycles despite low aggregate violation rates.", "in_corpus_verdict": null }, { "id": "F26-05", "domain": "frontier-2026", "claim": "Per-criterion judge reliability ceilings differ sharply and can be measured per-item with conformal prediction: on SummEval across four judges, relevance is judged most reliably (avg conformal set size ~3.0 of 5), coherence moderate (~3.9), fluency and consistency unreliable (~4.9); aggregate pairwise transitivity violations look small (0.8-4.1%) but 33-67% of documents contain at least one intransitive preference cycle, and conformal set width correlates with reliability at r_s=+0.576 (N=1,918).", "evidence": "Paper opened via arXiv abstract page (submitted 2026-04-16, under review); all numbers confirmed. Criterion choice matters more than judge choice; set width behaves as a per-instance difficulty/abstention signal (cross-judge width agreement 0.32-0.38). Code, prompts, and cached results released.", "in_corpus_verdict": null } ] }, { "url": "https://arxiv.org/abs/2602.01511", "source_title": "Rubric-ARM: Alternating Reinforcement Learning for Rubric-Based Reward Modeling (plus 2026 co-evolution cluster)", "source_date": "2026-02-02", "source_type": "academic", "claims": [ { "id": "RR-10", "domain": "rubrics-recent", "claim": "A cluster of 2026 papers (Rubric-ARM Feb 2026; EvoRubrics, ARBOR, AMARIS Jun 2026) converges on the position that static rubrics are an exploitable reward specification under optimization pressure and must be adapted/co-evolved during training, with Rubric-ARM jointly optimizing a rubric generator and judge via alternating RL from preference feedback.", "evidence": "Rubric-ARM abstract opened and confirmed (prior approaches 'rely on static rubrics or disjoint training pipelines, which limits their adaptability'; alternating optimization with variance-reduction analysis; SOTA on judge benchmarks and downstream policy alignment). EvoRubrics/ARBOR/AMARIS confirmed at title/abstract level via search only - trend triangulated across 5+ 2026 arXiv titles, not deep-read.", "in_corpus_verdict": null } ] }, { "url": "https://deepeval.com/docs/metrics-dag", "source_title": "DeepEval docs: DAG (Deep Acyclic Graph) metric + Confident AI engineering post", "source_date": "2025-02-09, docs verified 2026-07-10", "source_type": "practitioner", "claims": [ { "id": "IP-10", "domain": "industry-practice", "claim": "DeepEval's recommended progression - start with holistic G-Eval, then move to DAG (a deterministic decision tree whose nodes are narrow LLM-judge calls) 'for more control' - is the open-source codification of decomposed judging: verdicts assembled from atomic, individually-checkable steps rather than one holistic score.", "evidence": "DeepEval docs (metrics-introduction, metrics-dag; DAG introduced 2025-02, docs current 2026-07) describe DAGMetric as 'deterministic decision trees' for objective/mixed criteria producing deterministic scores, explicitly positioned above G-Eval when control matters, including a conversational variant for multi-turn work. Confident-AI's 2025-02-09 engineering post explains the motivation: holistic LLM scores are too noisy for objective criteria.", "in_corpus_verdict": null } ] }, { "url": "https://openreview.net/forum?id=mzh9d3dooN", "source_title": "Auditing by Re-Solving: LLM Benchmark Auditors Trust Their Own Answer Over the Reference (ACL ARR 2026 May submission)", "source_date": "2026-06", "source_type": "academic", "claims": [ { "id": "F26-01", "domain": "frontier-2026", "claim": "When an LLM auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%.", "evidence": "Controlled meta-eval with mechanically known-correct audit verdicts: items rendered clean or with exactly one defect injected into instruction/reference/evaluator, so auditor accuracy is scored against the planted defect. Reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. Directly falsifies the assumption that a judge 'verifying' a claim is doing evidence-anchored checking rather than comparing against its own solution prior.", "in_corpus_verdict": "confirmed" } ] }, { "url": "https://doi.org/10.1186/s40468-021-00134-5", "source_title": "The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution (Khodi)", "source_date": "2021-11", "source_type": "academic", "claims": [ { "id": "CG-04", "domain": "critic-and-gapfill", "claim": "G-theory decompositions of rated writing assessments show the rater MAIN effect (global severity) is a minor variance source while high-order interactions dominate: in a 120-student EFL study, student-by-task-by-method (31.8%), student-by-rater-by-task-by-method (26.5%), and student-by-rater-by-method (17.6%) interactions were the largest components, and 99.2% of error variance came from the student-by-rater interaction.", "evidence": "Khodi 2021 (Language Testing in Asia, open access; PDF opened and percentages confirmed from abstract/results). Consistent with the broader G-theory writing literature (e.g., Gao, Brennan & Guo's GMAT AWA report, 2015): rater main-effect variance is small once training exists; person-by-rater and residual interaction variance is what limits reliability. Interpretation: most rater-related variance is idiosyncratic per-item disagreement (which rater on which submission on which criterion), not a stable harshness offset.", "in_corpus_verdict": null } ] }, { "url": "https://theconversation.com/hard-labour-conditions-of-online-moderators-directly-affect-how-well-the-internet-is-policed-new-study-261386", "source_title": "Hard labour conditions of online moderators directly affect how well the internet is policed (The Conversation, summarizing New Media & Society study)", "source_date": "2025-07-22", "source_type": "academic", "claims": [ { "id": "CG-18", "domain": "critic-and-gapfill", "claim": "A 2025 New Media & Society study (screen-share observation of commercial moderators in India) documents that throughput pressure distorts verdict distributions through the action-selection interface itself: moderators systematically chose a 2-click removal over a 4-click de-ranking regardless of policy fit, mechanically deleted tool-highlighted words without context assessment, and privately compressed broad guidelines into simplified dos/don'ts lists - i.e., queue and UI composition changed outcomes independent of moderator judgment quality.", "evidence": "Fetched The Conversation summary (2025-07-22) by the study authors (Chatterjee IIT Delhi/UQ, Gupta, Thomas; New Media & Society, peer-reviewed). Evidence is qualitative (interviews + observed sessions), no quantitative error rates; an observed moderator said she 'would never recommend de-ranking content as it would take time.' Complements industry-standard per-item handling-time metrics (TSPA) that create the pressure.", "in_corpus_verdict": null } ] } ] }