{
  "batch": 2,
  "sources": [
    {
      "url": "https://neurips.cc/virtual/2025/poster/117308",
      "source_title": "Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025)",
      "source_date": "2025-09-19",
      "source_type": "academic",
      "claims": [
        {
          "id": "AJ-02",
          "domain": "academic-judges",
          "claim": "Standard LLM-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial LLMs, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy.",
          "evidence": "Guerdan, Barocas, Holstein, Wallach, Wu, Chouldechova (NeurIPS 2025 poster, abstract opened and confirmed). They formalize 'rating indeterminacy' - items where multiple ratings are reasonable - and show humans and LLMs resolve forced choices differently, so aggregated gold labels systematically mis-rank judges. They provide concrete recommendations for elicitation and aggregation.",
          "in_corpus_verdict": "partially_confirmed"
        }
      ]
    },
    {
      "url": "https://www.pnas.org/doi/10.1073/pnas.2518075122",
      "source_title": "The potential existential threat of large language models to online survey research (PNAS 122(47))",
      "source_date": "2025-11-20",
      "source_type": "academic",
      "claims": [
        {
          "id": "AQ-02",
          "domain": "annotation-quality",
          "claim": "An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025.",
          "evidence": "Westwood (PNAS, Nov 20 2025; 43,800 evaluations across 9 LLMs) shows instruction-following checks, logic puzzles, reCAPTCHA, and rare-event probes all fail; persona calibration even masks superhuman-accuracy tells. Recommended defenses are entirely provenance/economics-based: identity revalidation, per-worker throttling, completion-history transparency, pass/fail history, location/VPN checks - 'there is no magical fix.' The 2026 crowdsourcing survey (arXiv 2606.04924) independently confirms detectors degrade under paraphrase and out-of-domain, and mixed human-AI authorship is harder still.",
          "in_corpus_verdict": "confirmed"
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2510.26518",
      "source_title": "Human-AI Complementarity: A Goal for Amplified Oversight (Google DeepMind)",
      "source_date": "2025-10-30",
      "source_type": "primary",
      "claims": [
        {
          "id": "CM-03",
          "domain": "critique-models",
          "claim": "In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong.",
          "evidence": "1,918 expert-labeled fact-verification items; tool-using AI rater with majority-vote-over-50-samples confidence (well calibrated). AI 87.7% vs individual humans 75.1% (majority vote 80.6%). Hybridization at threshold 0.62: 89.3%; on the routed low-confidence slice humans beat the AI 71.3% vs 60.5% (real complementarity region). Evidence-only assistance: 79.3% when AI right (vs 71.3% baseline) with no significant harm when AI wrong; conditions showing Judgments+Reasoning/Confidence produced significant over-reliance decrements (e.g. beta = -0.768, p < .001); AI-debate condition was numerically worst. Verified from arXiv HTML full text.",
          "in_corpus_verdict": "confirmed"
        }
      ]
    },
    {
      "url": "https://aclanthology.org/2025.emnlp-main.1205/",
      "source_title": "DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation",
      "source_date": "2025-11",
      "source_type": "academic",
      "claims": [
        {
          "id": "GF-04",
          "domain": "grounding-faithfulness",
          "claim": "Decomposition and decontextualization are in direct tension - isolating atomic facts strips the context needed to verify them, while adding context back creates multi-fact claims where the verifier may credit or penalize the wrong content; DnDScore resolves this by verifying the original subclaim WITH the added information treated as context rather than as content to be verified.",
          "evidence": "Fetched ACL Anthology abstract of DnDScore (Wanner, Van Durme, Dredze; EMNLP 2025 main, pp. 23609-23626). The paper evaluates combinations of decomposition, decontextualization, and verification strategies and finds the strategy choice materially changes factuality scores.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2408.15204",
      "source_title": "Can Unconfident LLM Annotations Be Used for Confident Conclusions? (NAACL 2025) + Active Statistical Inference (ICML 2024)",
      "source_date": "2024-08-27 (v2 2025-02-08; NAACL 2025-04)",
      "source_type": "academic",
      "claims": [
        {
          "id": "HS-04",
          "domain": "hybrid-statistical",
          "claim": "Using the LLM's own confidence signals to choose WHICH items receive human annotation (Confidence-Driven Inference, building on Active Statistical Inference) cut required human annotations by >25% in all three tested settings while keeping provably valid confidence intervals that remain safe even if the LLM annotations are poor - but a 2026 ICLR paper finds the opposite in the sequential regime, where near-uniform sampling at the budget ceiling beat uncertainty-driven querying.",
          "evidence": "arXiv:2408.15204 (Gligoric, Zrnic, Lee, Candes, Jurafsky; NAACL 2025) abstract fetched: '>25% reduction in each of three settings' (politeness, stance, bias) with validity guarantees regardless of LLM quality. Foundation: Active Statistical Inference (Zrnic & Candes, ICML 2024, arXiv:2403.03208): label where the model is uncertain, provably valid CIs, fewer labels than uniform. Counter-evidence verified separately (see disagreements): arXiv:2604.18569 (ICLR 2026).",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://arxiv.org/abs/2404.12272",
      "source_title": "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences",
      "source_date": "2024-10-13",
      "source_type": "academic",
      "claims": [
        {
          "id": "RR-05",
          "domain": "rubrics-recent",
          "claim": "EvalGen ('Who Validates the Validators', UIST 2024) established criteria drift: evaluation criteria cannot be fully specified a priori because grading outputs itself changes the criteria - 'users need criteria to grade outputs, but grading outputs helps users define criteria' - and some criteria are dependent on the specific outputs observed.",
          "evidence": "Fetched abstract/paper confirmed the catch-22 formulation, the finding that criteria appear dependent on observed outputs rather than definable in advance, and the conclusion that judge-alignment is inherently iterative and subjective; EvalGen operationalizes this by generating candidate assertions/judge prompts and selecting those best aligned with accumulated human grades.",
          "in_corpus_verdict": null
        },
        {
          "id": "CT-04",
          "domain": "contrarian",
          "claim": "Human evaluation criteria are output-dependent and unstable: even when graders define criteria before grading, the act of grading changes their criteria and they retroactively revise earlier grades ('criteria drift'), implying evaluation criteria for LLM-output quality cannot be fully determined prior to observing outputs.",
          "evidence": "Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran & Arawjo (UIST 2024, EvalGen system study); abstract verified via arXiv crawl: 'we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria'; the authors argue there is reason to believe criteria never fully settle because they adapt to the observed output distribution.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "[link withheld]",
      "source_title": "TechCrunch: Report on an AI data vendor's acquisition of a data-quality startup",
      "source_date": "2026-01-28",
      "source_type": "practitioner",
      "claims": [
        {
          "id": "IP-05",
          "domain": "industry-practice",
          "claim": "An AI data vendor acquired a data-quality startup (announced 2026-01-28) specifically for algorithms that 'flag incorrect data without a second human reviewer,' after competing bids from other data-labeling companies - signaling that algorithmic second-pass QA replacing the human review layer is now an acquisition-grade core capability for human-data vendors.",
          "evidence": "TechCrunch (2026-01-28), the startup's CEO letter, and the vendor's own announcement triangulate: 9 staff of the startup including all three MIT-PhD co-founders join the vendor's research team; rationale quotes name confident learning, data-centric AI, and LLM evaluation as the acquired capabilities; an AI data vendor serves 8 top labs at ~$300M annualized run-rate (end 2025). The startup's core product audited human-labeler output quality algorithmically.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://aclanthology.org/2025.findings-emnlp.1361.pdf",
      "source_title": "Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks",
      "source_date": "2025-11-04",
      "source_type": "academic",
      "claims": [
        {
          "id": "CT-07",
          "domain": "contrarian",
          "claim": "LLM judges have low intra-rater reliability - identical items re-scored across runs with identical settings produce inconsistent, 'almost arbitrary in the worst case' ratings - and the obvious fix (temperature-0 determinism) measurably REDUCES agreement with human judgment; meanwhile the human baseline itself is weak (SummEval inter-annotator kappa: 0.492 crowd, 0.413 expert first round, 0.71 only after a second adjudication round).",
          "evidence": "Haldar & Hockenmaier, Findings of EMNLP 2025 (pp. 24986-25004); verified by reading the paper PDF (pp. 1-3): contributions are (1) low agreement of LLM ratings across runs, (2) disabling sampling hurts human-agreement, (3) phenomenon persists across SummaC, SummEval, MT-Bench; the SummEval human kappa figures are quoted in their Section 3.1.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://www.langchain.com/resources/llm-as-a-judge",
      "source_title": "How to Calibrate LLM-as-Judge with Human Corrections (LangChain)",
      "source_date": "2026-03-10",
      "source_type": "practitioner",
      "claims": [
        {
          "id": "F26-11",
          "domain": "frontier-2026",
          "claim": "The 2026 production norm for trusting an LLM judge is a closed calibration loop on sampled human corrections: humans review a slice of judge verdicts, corrections become ground truth for measuring alignment and are recycled as few-shot exemplars into the judge prompt (LangChain, March 10, 2026); practitioner guides converge on LLM-judge-first with a ~5-10% sampled human verification rate and periodic kappa re-checks as the default large-volume labeling pattern.",
          "evidence": "LangChain resource (dated March 10, 2026) describes the correction-collection -> alignment-measurement -> few-shot-rebuild loop as learned from production evaluator deployments; FutureAGI's 2026 guide independently states the '5-10% human verification' default and monthly kappa re-sampling for judge drift. Practitioner-grade evidence, not controlled studies.",
          "in_corpus_verdict": null
        }
      ]
    },
    {
      "url": "https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona",
      "source_title": "The Trauma Floor: The secret lives of Facebook moderators in America (The Verge, Casey Newton)",
      "source_date": "2019-02-25",
      "source_type": "primary",
      "claims": [
        {
          "id": "CG-13",
          "domain": "critic-and-gapfill",
          "claim": "Facebook's production QA at Cognizant ran a three-layer blind audit: ~50-60 of each moderator's ~1,500 weekly decisions (~3-4%) randomly re-reviewed by a dedicated QA worker (paid $1/hr more), with full-time Facebook employees auditing a subset of QA decisions; 'accuracy' was computed purely as agreement with the auditor against a 95% target (other sites reported 98%), actual scores ran high-80s to 92, and misses triggered a remediation program that often ended in termination.",
          "evidence": "Verified in The Verge's original text (fetched): 'From Miguel's 1,500 or so weekly decisions, Facebook will randomly select 50 or 60 to audit... Full-time Facebook employees then audit a subset of QA decisions.' Moderator quote: 'Accuracy is only judged by agreement. If me and the auditor both allow the obvious sale of heroin, Cognizant was correct... This number is fake.' Also confirmed: moderators lobbied QAs off-book to reverse decisions despite prohibition, and pre-firing 'coaching' often served as pretext for managing workers out.",
          "in_corpus_verdict": null
        }
      ]
    }
  ]
}