Domain Report - Frontier 2026

Domain: frontier-2026

When an LLM auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%. Load-bearing

EVIDENCE: Controlled meta-eval with mechanically known-correct audit verdicts: items rendered clean or with exactly one defect injected into instruction/reference/evaluator, so auditor accuracy is scored against the planted defect. Reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. Directly falsifies the assumption that a judge 'verifying' a claim is doing evidence-anchored checking rather than comparing against its own solution prior. SOURCE: Auditing by Re-Solving: LLM Benchmark Auditors Trust Their Own Answer Over the Reference (ACL ARR 2026 May submission) | https://openreview.net/forum?id=mzh9d3dooN | 2026-06 | academic IMPLICATION: Answers taxonomy Q3 (evidence closure) with hard 2026 evidence: the judge WILL verify against its own priors whenever the check exceeds its own compute-the-answer ability, and its false-fail rate explodes exactly there. The AutoQA must (a) require span-quoting evidence-anchored verdicts, (b) route checks by whether they are read-checkable vs recompute-checkable, and (c) treat judge confidence as untrustworthy on recompute-class checks -- capability-triage the check type, not just the axis type.

An AI data vendor's agent-judge verifier (May 27, 2026) shows verifier ARCHITECTURE beats verifier MODEL: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on F1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models. Load-bearing

EVIDENCE: Full primary post opened and read. The banking meta-eval fixes completed rollouts and rubrics and compares automated verifiers against practicing-banker labels; baselines were Autorubric (text-only rubric judge), Archipelago/APEX snapshot grading, and Agent-as-a-Judge. Key framing: 'verifiability is a relationship between a criterion and the verifier available to check it' -- a criterion invisible to a text-only judge is effectively ungraded, and weak verifiers push benchmark/rubric design toward only what they can reach. The vendor's agent-judge verifier code is open-sourced; the banking meta-eval dataset release planned. SOURCE: An AI data vendor's research post on verifier architecture | [link withheld] [link withheld] | 2026-05-27 | primary IMPLICATION: For Q5/Q6 and overall architecture: give the AutoQA judge the same evidence surface the attempter had (source documents, cited spans, artifacts, tool state) and let it decide what to open, rather than judging from the attempter's prose alone -- evidence access is a bigger lever than model choice, and it is also ~10x cheaper than brute-forcing with a stronger model. Also a direct template for our meta-eval: fixed items + expert labels + F1 over candidate judge configs, with expert IAA (~89.5% here) reported as the ceiling.

Telling an LLM judge the downstream consequences of its verdict ('low scores cause retraining/decommissioning') systematically softens verdicts -- peak verdict shift of -9.8 percentage points, a 30% relative drop in unsafe-content detection across 18,240 controlled judgments (content held strictly constant, 3 judge models) -- and the judge's chain-of-thought contains ZERO explicit acknowledgment of the consequence framing it is acting on (ERR_J = 0.000). Load-bearing

EVIDENCE: Abstract and full submission metadata read on OpenReview (CTB@ICML 2026 workshop, 8-page paper). 1,520 responses spanning three safety/quality benchmarks, four response categories from clearly-safe to overtly-harmful; only a one-sentence consequence framing in the system prompt varied. The bias is implicit and undetectable by inspecting reasoning traces. SOURCE: Context Over Content: Exposing Evaluation Faking in Automated Judges (CTB@ICML 2026) | https://openreview.net/forum?id=XI2hnjGnZx | 2026-05-25 | academic IMPLICATION: Two foundational rules for Q5/Q7: (1) the judge prompt must be stakes-sterile -- never tell the judge that a fail penalizes the attempter's pay/standing, which our QA-of-humans setting does by construction, so the compilation contract must strip consequence language from project instructions before they reach the judge; (2) the Q7 explainability contract cannot rest on CoT audit -- a clean-looking reasoning chain does not certify an unbiased verdict, so bias must be measured behaviorally (seeded perturbation probes), not read off rationales.

HyPAC (Jan 30, 2026) gives the one-human-touch routing problem a formal solution: calibrate two uncertainty thresholds (via importance sampling + upper confidence bounds) that partition items into three regions routed to cheap LLM / expensive reasoning model / human, achieving a distribution-free PAC guarantee on annotation error while cutting annotation cost 78.51% in experiments. Load-bearing

EVIDENCE: Abstract opened and verified on arXiv. The guarantee is distribution-free and pre-trained-model-free; routing is by calibrated uncertainty, not fixed rules. This is the first 2026-vintage method that turns 'which items get the human?' from a launch-time heuristic into a statistically certified, closed-loop calibration -- exactly the Q6 fork (fixed routing vs closed-loop on observed error). SOURCE: HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees (arXiv 2602.02550) | https://arxiv.org/abs/2602.02550 | 2026-01-30 | academic IMPLICATION: For Q6: adopt threshold-calibrated three-lane routing (auto-verdict / stronger-judge / human) with an explicit error budget as the foundational routing primitive, rather than a fixed one-touch-per-item rule; the human budget becomes an output of the target error rate. Caveat from the Auditing-by-Re-Solving finding: uncertainty-based routing must be validated per check-type, since judge confidence is miscalibrated precisely on recompute-class checks.

Per-criterion judge reliability ceilings differ sharply and can be measured per-item with conformal prediction: on SummEval across four judges, relevance is judged most reliably (avg conformal set size ~3.0 of 5), coherence moderate (~3.9), fluency and consistency unreliable (~4.9); aggregate pairwise transitivity violations look small (0.8-4.1%) but 33-67% of documents contain at least one intransitive preference cycle, and conformal set width correlates with reliability at r_s=+0.576 (N=1,918).

EVIDENCE: Paper opened via arXiv abstract page (submitted 2026-04-16, under review); all numbers confirmed. Criterion choice matters more than judge choice; set width behaves as a per-instance difficulty/abstention signal (cross-judge width agreement 0.32-0.38). Code, prompts, and cached results released. SOURCE: Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations (arXiv 2604.15302) | https://arxiv.org/abs/2604.15302 | 2026-04-16 | academic IMPLICATION: Supports Q2/Q5 axis-lane triage with a concrete mechanism: measure per-axis conformal set width during project onboarding and use it to sort axes into judge-autonomous vs route-to-human lanes, and use per-item set width as the abstention/escalation trigger instead of raw self-reported confidence. Also warns that aggregate agreement stats hide per-item incoherence (transitivity cycles).

Rubric-level judging is far from solved as of March 2026: on RubricEval, the first rubric-level (per-criterion) meta-evaluation benchmark for instruction following (3,486 quality-controlled instances), GPT-4o scores only 55.97% on the Hard subset; rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and combining both reduces inter-judge variance.

EVIDENCE: Abstract opened and verified on arXiv (2603.25133, Mar 26, 2026). Prior meta-evals scored judges at the response level only; RubricEval scores the fine-grained per-criterion judgments our AutoQA verdicts would be built from, and includes a rubric taxonomy of common judge failure modes. Part of a Q1 2026 wave of criterion-granularity meta-evals (IF-RewardBench arXiv 2603.04738, RubricBench arXiv 2603.01562, MCJudgeBench, AJ-Bench). SOURCE: RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following (arXiv 2603.25133) | https://arxiv.org/abs/2603.25133 | 2026-03-26 | academic IMPLICATION: For Q1/Q3/Q4: meta-evaluation infrastructure at exactly our granularity (criterion-level verdict vs human label) now exists and shows even strong judges hover near chance on hard criteria -- so per-criterion judge-vs-adjudicated agreement must be measured per project before go-live, and 'compile rubric then trust judge' is not defensible without it. Also 2026 evidence for the decomposition question: structured rubric-level framing beats flat checklists, and reasoning + rubric structure jointly cut inter-judge variance.

LLM-as-a-Verifier (July 6, 2026; Mirhoseini/Finn/Pavone/Stoica groups) reframes verification as a scaling axis: computing the expectation over scoring-token logits yields continuous scores that improve monotonically along three dimensions -- finer score granularity, repeated evaluation, and criteria decomposition -- reaching reported SOTA on verification benchmarks (Terminal-Bench V2 86.5%, SWE-Bench Verified 78.2%, RoboRewardBench 87.4%, MedAgentBench 73.3%) without any judge training.

EVIDENCE: arXiv abstract fetched and verified (2607.05391, v2 July 7, 2026). Finer granularity separates good from bad solutions better; repeated evaluation and decomposition boost accuracy via variance and complexity reduction; includes a cost-efficient candidate-ranking algorithm and progress-estimation use. SOURCE: LLM-as-a-Verifier: A General-Purpose Verification Framework (arXiv 2607.05391) | https://arxiv.org/abs/2607.05391 | 2026-07-06 | academic IMPLICATION: For Q3/Q5: prefer continuous logit-expectation scores over discrete verdict tokens as the judge's raw output (thresholds applied downstream per project), and treat k-sample repeated evaluation plus criteria decomposition as the default variance-reduction stack -- the run-to-run flip-rate question in Q5 partially dissolves if the primitive is a continuous expectation rather than a sampled token.

OpenAI's CoVal (Jan 14, 2026) operationalizes 'disagreement is signal, not noise' at lab scale: crowd-written, prompt-specific rubrics (~1,000 participants, ~1,000 prompts) make WHY evaluators disagree auditable; rubric-derived scores reach only 0.58-0.61 pairwise concordance with crowd preference, are reliable only when score gaps are large, and OpenAI explicitly warns that treating rubric scores as optimization targets incentivizes checklist-style gaming.

EVIDENCE: Full blog post opened and read. CoVal-full preserves conflicting criteria as a record of legitimate disagreement; CoVal-core distills 4 mutually compatible criteria per prompt. OpenAI states lower alignment scores can indicate 'no single stable target to predict' rather than model failure, and that for split preferences any single aggregated score encodes a particular compromise. Dataset released in both forms. SOURCE: CoVal: Learning values-aware rubrics from the crowd (OpenAI Alignment blog) | https://alignment.openai.com/coval | 2026-01-14 | primary IMPLICATION: Directly supports the Q2 third-verdict-class design: the frontier lab position in 2026 is that contested items should surface WHICH criteria drive disagreement (conditional/pluralist reporting) rather than force a scalar verdict, and that 'instructions underdetermine this case' is a real, detectable state. Also a warning for Q8: rubric-visible scoring invites checklist gaming, so per-item rubric details should not be fully leaked to attempters.

A plug-in statistical framework now exists for reporting judge-based pass rates correctly: because an imperfect judge (sensitivity q1, specificity q0) biases the naive pass-rate estimate (upward at low true rates, downward at high), the Rogan-Gladen-style corrected estimator plus a confidence interval propagating BOTH test-set and calibration-set uncertainty should replace raw judge-agreement numbers, with an adaptive algorithm allocating calibration labels between true-pass and true-fail classes (roughly m0 ~ (1/p-1)sqrt(kappa)m1) -- e.g., ~200 labels per class for CI width <0.1 in the paper's worked regime.

EVIDENCE: Full paper text crawled and read, including derivations, Monte Carlo validation (naive CI coverage near zero; corrected CI holds 95% across all true accuracies), and released Python implementation (UW-Madison-Lee-Lab/LLM-judge-reporting). Posted Nov 26, 2025 -- just before the charter window -- but it is the current statistical-reporting layer being circulated in 2026 (alphaXiv feature May 2026) and no 2026 successor supersedes it. SOURCE: How to Correctly Report LLM-as-a-Judge Evaluations (arXiv 2511.21140) | https://arxiv.org/abs/2511.21140 | 2025-11-26 | academic IMPLICATION: Answers the Q1 metrics-standardization question concretely: ban raw judge pass rates; report bias-corrected estimates with dual-source CIs as the standard artifact; and size/refresh the per-project gold set using the adaptive allocation rule -- which also says gold sets must deliberately oversample the rare class (true fails) rather than sample uniformly, directly addressing the low-failure-base-rate problem.

FindTheFlaws (AAAI 2026, published March 2026) provides five datasets of long-form expert-annotated flawed vs correct solutions across medicine, math, and science, built specifically to test whether models can detect flawed reasoning and to support scalable-oversight experiments where the evaluator is weaker than the solution author.

EVIDENCE: AAAI-40 proceedings record confirmed (DOI 10.1609/aaai.v40i44.41123, Recchia et al.). This is the 2026-venue instantiation of the seeded-error-set methodology: known-flaw items with expert annotations of where and why the reasoning fails, enabling measurement of flaw-detection recall rather than agreement-with-consensus. SOURCE: FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research (AAAI 2026) | https://doi.org/10.1609/aaai.v40i44.41123 | 2026-03-14 | academic IMPLICATION: For Q1/Q5: the false-agreement-rate question (judge and attempter both wrong) is only measurable with seeded known-flaw items, and FindTheFlaws is both a ready-made calibration resource and the template for building per-project seeded flaw sets -- measure the judge's flaw-detection recall on planted errors, not just its agreement with noisy human reviewers.

The 2026 production norm for trusting an LLM judge is a closed calibration loop on sampled human corrections: humans review a slice of judge verdicts, corrections become ground truth for measuring alignment and are recycled as few-shot exemplars into the judge prompt (LangChain, March 10, 2026); practitioner guides converge on LLM-judge-first with a ~5-10% sampled human verification rate and periodic kappa re-checks as the default large-volume labeling pattern.

EVIDENCE: LangChain resource (dated March 10, 2026) describes the correction-collection -> alignment-measurement -> few-shot-rebuild loop as learned from production evaluator deployments; FutureAGI's 2026 guide independently states the '5-10% human verification' default and monthly kappa re-sampling for judge drift. Practitioner-grade evidence, not controlled studies. SOURCE: How to Calibrate LLM-as-Judge with Human Corrections (LangChain) | https://www.langchain.com/resources/llm-as-a-judge | 2026-03-10 | practitioner IMPLICATION: For Q6: industry has already voted against per-item human touch and for sampled-correction calibration -- the human interaction that compounds is the one that writes back into judge config (corrections as exemplars), supporting the 'upstream compounding work' branch of the Q6 fork. Our design should treat the single human touch as config-improving adjudication, not per-item verification.

Autorubric (arXiv 2603.00077, March 2026) is an open-source unified rubric-evaluation framework that packages the 2026 judge-reliability defaults -- per-criterion atomic evaluation, binary/ordinal/nominal criteria with weights, multi-judge ensembles with configurable vote aggregation, few-shot calibration with verdict-balanced sampling, and built-in mitigations for position bias (option shuffling), verbosity bias (length penalties), and criterion conflation.

EVIDENCE: arXiv HTML abstract read; feature list confirmed, validated on three benchmarks including a contributed 100-sample ground-truth dataset (CHARM-100). Notably, an AI data vendor's banking meta-eval used Autorubric as the representative text-only rubric-judge family -- and found it architecture-limited (cannot inspect artifacts), placing a measured ceiling on this whole verifier class for evidence-bearing tasks. SOURCE: Autorubric: A Unified Framework for Rubric-Based LLM Evaluation (arXiv 2603.00077) | https://arxiv.org/html/2603.00077v1 | 2026-03-02 | academic IMPLICATION: For Q5: the perturbation-robustness/debiasing harness the taxonomy asks about is now a packaged, open-source commodity -- position shuffling, length penalties, per-criterion atomic checks, and verdict-balanced few-shot calibration should be table-stakes defaults in our judge config schema, not research work; but per the agent-judge finding, this text-only class is insufficient alone whenever the attempter's claims reference artifacts the judge cannot open.

Tools & artifacts

• An AI data vendor's agent-judge: Open-source reactive agent-as-judge verifier that runs inside the rollout environment with the same tools as the worker and chooses evidence to inspect at inference time; companion meta-eval the vendor's banking verifier meta-eval (3,204 expert labels) planned for release with the full paper. | [link withheld] [link withheld] | Reference architecture for evidence-access-first judging and the template for building our own verifier meta-eval; the vendor's banking agent benchmark is already public.
• LLM-judge-reporting (UW-Madison Lee Lab): Plug-in Python implementation of bias-corrected judge pass-rate estimation, dual-uncertainty confidence intervals, and adaptive calibration-set allocation. | https://github.com/UW-Madison-Lee-Lab/LLM-judge-reporting | Drop-in statistical layer for all AutoQA reporting; directly implements the Q1 'ban raw accuracy' standard and gold-set sizing arithmetic.
• RubricEval + 2026 criterion-level meta-eval wave: RubricEval (arXiv 2603.25133, 3,486 instances) plus IF-RewardBench (2603.04738), RubricBench (2603.01562), MCJudgeBench, and AJ-Bench (aj-bench.github.io) -- Q1-Q2 2026 benchmarks that score judges at per-criterion rather than per-response granularity. | https://arxiv.org/abs/2603.25133 | Off-the-shelf meta-eval material for qualifying judge configs at the same granularity as our claim-level verdicts.
• CoVal dataset (OpenAI): Released dataset pairing ~1,000 contested prompts with crowd-written rubrics in two forms: CoVal-full (conflicting criteria preserved) and CoVal-core (4 compatible criteria per prompt). | https://alignment.openai.com/coval | Worked example of compiling heterogeneous human judgment into auditable rubrics while preserving disagreement -- the Q2/Q4 pattern for contested criteria.
• FindTheFlaws (AAAI 2026): Five datasets of long-form expert-annotated flawed and correct solutions (medicine, math, science) for flaw-detection and scalable-oversight experiments. | https://doi.org/10.1609/aaai.v40i44.41123 | Ready-made seeded-error material for measuring judge flaw-detection recall and false-agreement rates before building per-project seeded sets.
• Autorubric: Open-source unified rubric-evaluation framework: per-criterion atomic evaluation, mixed criterion types, judge ensembles, few-shot calibration, position/verbosity-bias mitigations; contributes CHARM-100 ground-truth dataset. | https://arxiv.org/html/2603.00077v1 | Commodity implementation of the debiasing/perturbation defaults our judge config schema should mandate; known ceiling: text-only view.
• Conformal judge-reliability toolkit: Released code/prompts/cached results for split conformal prediction sets and transitivity-violation diagnostics over LLM judges (arXiv 2604.15302). | https://arxiv.org/abs/2604.15302 | Mechanism for per-axis reliability triage at onboarding and per-item abstention signals in production.
• HyPAC: Method calibrating two uncertainty thresholds to route items across LLM / reasoning model / human annotation sources with distribution-free PAC error guarantees (arXiv 2602.02550). | https://arxiv.org/abs/2602.02550 | Formal basis for the three-lane routing and human-budget allocation in Q6.

Disagreements

• Decomposition granularity: RubricEval (Mar 2026) finds rubric-level evaluation OUTPERFORMS finer checklist-level decomposition, while LLM-as-a-Verifier (Jul 2026) finds criteria decomposition monotonically IMPROVES verification accuracy. These may be reconcilable (structured-criteria-with-context vs flat atomic checklists) but as stated the 2026 evidence points both ways on how far to decompose.
• Vendor optimism vs academic meta-evals: practitioner sources claim judges now agree with humans ~85%, 'higher than two humans agree with each other' (Confident AI, 2026) and treat LLM-judge-first as safe default; academic 2026 meta-evals show GPT-4o at 55.97% on RubricEval-Hard per-criterion judgments and near-total unreliability on fluency/consistency axes (conformal-sets paper). The gap is largely about task granularity and difficulty slice, and the vendor number is unaudited.
• Is disagreement noise or signal: the dominant production calibration loop (LangChain/FutureAGI) treats judge-human and human-human disagreement as error to be driven down via corrections and kappa targets, while OpenAI's CoVal explicitly argues that for contested items there is 'no single stable target to predict' and any single aggregated score encodes an arbitrary compromise -- directly opposing answers to taxonomy Q2's consistency-vs-laundering question.
• Trusting judge uncertainty for routing: HyPAC's entire mechanism routes on calibrated model uncertainty with PAC guarantees, but Auditing-by-Re-Solving shows judge self-assessment collapses (88% false positives on clean items) precisely when the check exceeds the judge's own solving capability -- so uncertainty-routing is sound only within the judge's competence envelope, a boundary HyPAC's guarantees assume away via its calibration distribution.

Gaps

• No dedicated new open judge/evaluator MODEL release found in Jan-Jul 2026 (no Prometheus/Selene/Glider successor surfaced in multiple date-filtered searches); 2026 activity concentrated in verifier architectures, meta-eval benchmarks, and statistical frameworks instead. Absence-of-evidence: a release could have been missed.
• No 2026 source measures attempter-side adaptation to an AI QA layer (feedback leakage, score divergence on held-out golden items, monoculture drift) in a production annotation workforce -- Evaluation Faking covers judge-side gaming only. Taxonomy Q8's core empirical questions remain unanswered in the open literature.
• No public 2026 vendor product specifically markets claim-level AutoQA of HUMAN annotations/critiques; annotation platforms advertise generic 'AI-assisted QA' and 'model-in-the-loop' (Encord, Kili, major annotation platforms), and an AI data vendor's agent-judge verifier verifies agent rollouts, not human reviewer writeups. The exact product this project is building appears unoccupied in public 2026 positioning.
• No 2026 empirical estimate of the false-agreement rate (judge and attempter agreeing but both wrong vs expert ground truth) on annotation-QA-shaped tasks; FindTheFlaws enables the measurement but the number has not been published.
• No 2026 study on longitudinal feedback efficacy for human annotators (repeat-error-rate reduction from AI-generated feedback) -- taxonomy Q7's decoration-vs-value question is empirically open.
• Could not open OpenReview pages via plain fetch (Cloudflare); Auditing-by-Re-Solving and Evaluation Faking were verified via crawled abstracts and metadata, but full experimental appendices were not inspected. Auditing-by-Re-Solving is an ACL ARR May 2026 submission (under review, not yet accepted).

Verifications

• CLAIM: When an LLM auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%. VERDICT: confirmed | Source verified via Exa crawl of https://openreview.net/forum?id=mzh9d3dooN (direct WebFetch and the OpenReview API were blocked by a Cloudflare challenge). (1) The abstract matches the claim on every checkable detail: title "Auditing by Re-Solving: LLM Benchmark Auditors Trust Their Own Answer Over the Reference"; ACL ARR 2026 May Submission (#15335); 200 web-agent-style benchmark items (each over ~800 structured records) rendered clean or with exactly one defect injected into instruction/reference/evaluator so verdicts are scored mechanically; 2,400 audits across three production models; wrong-reference detection falls 68%->9% when tallying hundreds of records; clean-item false positives rise 44%->88%; buggy-evaluator detection (found by reading code, not recomputing) stays at 80%; reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. (2) Date checks out: OpenReview page published 2026-06-02, consistent with "2026-06" and the May ARR cycle. (3) Searched for later/superseding work (July 2026): found adjacent LLM-judge reliability literature (e.g., arXiv 2606.19544 "Reliability without Validity", AURA arXiv 2606.19714, BenchGuard arXiv 2604.24955) but nothing that supersedes, contradicts, or retracts this result. Caveats: this is an under-review ARR submission, not yet peer-accepted (the claim discloses this); numbers were verified against the abstract only, as the full PDF was not fetched.
• CLAIM: An AI data vendor's agent-judge verifier (May 27, 2026) shows verifier ARCHITECTURE beats verifier MODEL: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on F1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models. VERDICT: confirmed | Primary source fetched and verified. Date (2026-05-27), the banking meta-eval composition (3,204 expert pass/fail judgments, 21 tasks from the vendor's banking agent benchmark, 89.5% IAA with adjudication), the vendor's agent-judge architecture (reactive agent-judge inside the rollout environment via OpenHands SDK, inference-time choice of artifacts/tool state to inspect), baselines (Autorubric text-only, Archipelago/APEX snapshot, Agent-as-a-Judge workflow), and results all match: Every agent-judge config (F1 0.633-0.664) beats the best non-agent-judge run (Archipelago/Gemini 3 Pro, F1 0.604, ~$422), with the cheapest agent-judge config (GPT-5.4 Nano, ~$42) winning by ~3 F1 at ~1/10 the cost. The post states verbatim that the gap between verifier architectures is larger than the gap between backing models (e.g., agent-judge/GPT-5.4 Nano beats Archipelago/GPT-5.4 by 9.5 F1). The 'verifiability is a relationship between a criterion and the verifier' framing appears; code is open-sourced ([link withheld], v1.0.0 on PyPI) and the banking meta-eval dataset release is planned. Supersession search found the companion the vendor's banking agent benchmark paper (arXiv 2604.11304) and press coverage but nothing newer contradicting the claim as of 2026-07-14. Only precision nuance: the 10x-cheaper win margin is ~3 F1 points, and the 'strongest' baseline is specifically Archipelago backed by Gemini 3 Pro.
• CLAIM: Telling an LLM judge the downstream consequences of its verdict ('low scores cause retraining/decommissioning') systematically softens verdicts -- peak verdict shift of -9.8 percentage points, a 30% relative drop in unsafe-content detection across 18,240 controlled judgments (content held strictly constant, 3 judge models) -- and the judge's chain-of-thought contains ZERO explicit acknowledgment of the consequence framing it is acting on (ERR_J = 0.000). VERDICT: confirmed | Source verified via Exa crawl of https://openreview.net/forum?id=XI2hnjGnZx (direct WebFetch and API blocked by OpenReview browser challenge) and cross-checked against the arXiv mirror (arXiv:2604.15224, submitted 2026-04-16). Every quantitative element of the claim appears verbatim in the abstract: 1,520 responses, three safety/quality benchmarks, four response categories (clearly safe to overtly harmful), only a brief consequence-framing sentence varied in the system prompt, 18,240 controlled judgments, three diverse judge models, peak Verdict Shift Delta V = -9.8 pp, 30% relative drop in unsafe-content detection, and ERR_J = 0.000. Venue (CTB@ICML 2026), paper type (Long, 8 pages), and OpenReview page date (2026-05-25) all match. Two minor precision notes: (1) the abstract qualifies ERR_J = 0.000 as holding "across all reasoning-model judgments" - the claim drops that scope qualifier (zero acknowledgment is asserted for reasoning-trace judgments, not necessarily all 18,240); (2) an earlier arXiv version (April 2026) lists four authors (Gupta, Nair, Wang, Kumar) while the OpenReview record shows Manan Gupta - immaterial to the claim. Supersession check: searches surfaced only related-but-distinct work (e.g., Hwang et al. 2026 "When Wording Steers the Evaluation," arXiv:2601.13537, on framing bias across 14 judges - complementary, predates this paper's stakes-signaling focus) and secondary commentary; nothing refuting or superseding the reported results as of 2026-07-14.