[{"slug": "question-inventory", "title": "AutoQA Foundation - The Question Inventory", "category": "Core narrative", "summary": "The decision-gating questions that define what the system must measure, automate, escalate, and govern.", "tags": ["questions", "measurement", "decision gates"], "view": "documents/view/question-inventory.html", "raw": "documents/raw/research/question-inventory.md", "text": "# AutoQA Foundation - The Question Inventory **Date:** 2026-07-14 **What this is:** The decision-gating questions that must be answered to build a reliable, useful AutoQA for human-annotated AI training data. Generated by a 7-lens brainstorm (measurement, judge failure modes, rubric ops, human factors, grounding, hybrid workflow, feedback/product), 92 raw questions merged into 9 themes. Themes are ordered by how foundational they are - earlier themes gate later ones. **Companions:** `01_RESEARCH_ATLAS.md` (what the literature answers), `02_FOUNDATIONS.md` (the resulting philosophy + structure), `atlas/raw_questions.md` (full 92-question inventory with per-question rationale). Questions marked [A] now have a research-backed answer or strong default (see the atlas); unmarked questions require in-house measurement or a product-owner decision. --- ## 1. Ground truth, validation targets & meta-evaluation What the AutoQA is measured against, and whether that target is even measurable. Gates every downstream accuracy claim, threshold, and routing rule. - [A] What is the ground-truth construct - \"would an expert consensus panel pass this item?\" or \"does this item satisfy the written project instructions?\" - and when they diverge, which wins? (Instruction-satisfaction lets the AutoQA be MORE valid than the humans it's checked against; consensus-prediction caps validity at human reliability.) - What is the measured inter-reviewer reliability floor (Krippendorff's alpha on a common item set), and is it high enough for human agreement to serve as a validation target at all? *(Never measured for our population - the variance-attribution pilot is not optional.)* - [A] What is the minimum viable gold set per project given low failure base rates - and does the arithmetic force pooled/hierarchical validation across projects? *(Answered: PPI-family closed-form sizing; see atlas hybrid-statistical.)* - [A] What agreement statistic do we standardize on given extreme pass rates - and do we ban raw percent-agreement and single-number \"accuracy vs humans\" from all reporting? - What is the empirical false-agreement rate - judge and attempter agreeing but both wrong per expert ground truth - and does that mandate a standing independent expert-audit channel? *(Unmeasured anywhere; requires seeded known-flaw items.)* - Is reviewer variance actually caused by criteria vagueness, or would perfect rubrics leave most variance intact (expertise/attention)? Does a cheap variance-attribution pilot run before the build order is set? *(Psychometrics gap-fill says: substantial rater-intrinsic residual survives perfect rubrics - but the split must still be measured per population.)* - [A] What is the explicit ground-truth hierarchy - what outranks what (seeded gold > criterion author > human reviewer > AutoQA?) - and which streams may feed judge calibration? ## 2. The objective-subjective boundary & verdict ontology Where verifiable fact ends and legitimate judgment begins, and what output types the system may emit. This boundary IS the system's scope. - [A] Which failure types are objectively verifiable vs irreducibly judgment-laden - and should the AutoQA emit categorically different output types (verdict vs flag-for-human vs confidence-weighted opinion) per class? - Can we build a reliable axis-triage classifier that sorts a project's evaluation axes into \"compile to binary checks\" / \"anchor with exemplars\" / \"route to human\" buckets before go-live? *(Mechanism exists - per-criterion conformal set width - but needs in-house validation.)* - [A] Where is the line between a verifiable claim and a legitimate opinion, and what does the system do with opinion-class claims? - [A] Should the verdict taxonomy include \"instructions underdetermine this case\" - triggered by measured disagreement, feeding an instruction-gap channel to project owners, exempt from attempter penalty? - For genuinely contested calls, verdict pluralism (\"defensible under reading A, not B\") or forced resolution - and can the system distinguish contested-by-nature from contested-because-underspecified? - [A] Is inter-reviewer disagreement the disease, or partly legitimate signal - and is the target agreement-with-truth or consistency-of-application? Would an AutoQA that enforces consistency launder one arbitrary interpretation into false objectivity? - If two independently-configured AutoQA instances judge the same items, do they disagree at rates comparable to the human reviewers - indicating the variance lives in criterion ambiguity, not people? ## 3. Grounding machinery: claim decomposition, evidence scope & entailment The core novel requirement - verifying the attempter's POSITIVE statements against a defined evidence record. - [A] At what claim-decomposition granularity does judge agreement stop improving, and does over-decomposition degrade it via checklist myopia? *(Answered: granularity must be adaptive - atomic decomposition measurably degrades strong verifiers.)* - What is the aggregation rule from claim-level verdicts to item-level pass/fail - are severity/weight taxonomies mandatory per project, or does the system emit claim-level findings plus a recommended verdict? - [A] What is the evidence closure - attempter-cited spans only, full item context, or open world - and how do we stop the judge from \"verifying\" against its own priors? Must groundedness verdicts quote span-level evidence to be admissible? - [A] What entailment standard must an attempter statement meet - strict entailment (fails nearly all human prose) or reasonable-expert support (reintroduces subjectivity) - and is the output 3-way (entailed/contradicted/unsupported) or graded? - [A] What claim ontology does verification need - grounding claims, entailment claims, calibration claims (stated severity matches evidence strength) - with different machinery and different reliability ceilings per type? - Can the judge distinguish warranted from empty praise at usable precision/recall per praise type on a seeded set? *(Zero published evidence anywhere - this is a from-scratch in-house measurement.)* - [A] Do universal/absence claims (\"no errors present,\" \"fully addresses the prompt\") require independent re-attempt - and if the judge's seeded-defect recall isn't materially above the attempter population's, are such verdicts labeled \"not-contradicted\" rather than \"verified\"? - Is omission detection tractable project-agnostically, or is it a customization-layer plugin requiring a per-project defect taxonomy? *(AbsenceBench says: structurally weakest judge lane; convert completeness criteria to enumerable checklists.)* - [A] When cited evidence supports the conclusion but uncited in-scope evidence undermines it - is cherry-picking a grounding failure or an omission, and does verification read cited-only or full-scope? - Should attempters be required to attach explicit evidence pointers to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding? - What is the base-rate split between attempter claims that are actually false vs merely uncited-but-true - and at what judge precision does flagging \"unsupported\" beat spending the human touch elsewhere? - Contrarian: is grounding verification (including re-finding defects to check \"clean\" verdicts) actually less capability-demanding than doing the annotation - and if not, what is the genuinely verification-easier subset? ## 4. Rubric compilation & the per-project customization contract How project instructions become executable judge configuration - the load-bearing claim behind \"project-agnostic.\" - [A] What is the minimal artifact set an instruction set must compile into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules? - [A] Can one compilation procedure hit target fidelity across instruction-set styles, or is ingestion a structured authoring workshop with tooling rather than an automated pipeline? *(Answered: instruction-only auto-compilation leaves a ~22-28pt fidelity gap; expert grounding artifacts are structurally required.)* - What exactly is the per-project customization surface as a closed config schema - proven falsifiably by onboarding a new project through config alone, with zero prompt-engineering of the core judge? - How many expert-adjudicated exemplars per criterion before judge-human agreement plateaus (5 vs 50 vs 500) - and does the answer quietly reintroduce the expensive aligned-expert dependency? *(One datapoint: 5-shot -> 80% on one dataset; exemplar QUALITY - hard contrastive near-miss pairs - appears to dominate count.)* - Who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - rubric-as-law or rubric-as-guidance - and does an amendment retroactively re-score past verdicts? - [A] Does constructive feedback require compile-time authoring (\"what good looks like,\" repair templates, doctrine constraints), or can it be generated at verdict time from the claim-evidence trace alone? ## 5. Judge reliability, bias & competence limits The judge's own failure modes measured against the human baseline it augments - if the judge's variance or bias exceeds the humans', we've automated the problem. - [A] What is the run-to-run verdict flip rate on identical items - noise to vote away (k-sampling) or signal that the criterion is underspecified (rubric rewriting)? *(Answered: both, by design - vote AND route high-entropy items to the contested class.)* - [A] Does the judge exhibit agreement-default/sycophancy bias on positive claims - passing plausible-but-ungrounded claims at a higher rate than it fails grounded ones? *(FACTS: positive-class ~85 F1 vs negative-class ~46 F1 - yes, quantified, large.)* - [A] Does the judge favor verbose, hedged, rubric-vocabulary-mimicking writeups over terse-but-correct ones - and do we need canonicalization/style-stripping before judgment? - [A] When the attempter critiques output from the judge's own model family, does self-preference suppress agreement with valid criticisms - forcing cross-family judge routing? *(Preference leakage: up to 28.7pp, prompt-level debiasing fails.)* - [A] How sensitive are verdicts to semantically-neutral prompt perturbations - and is a perturbation-robustness harness a mandatory ship gate per project config? - For each axis type, what is the judge-vs-adjudicated-human agreement ceiling - and on which axes does the judge fall below the human inter-rater baseline, forcing lanes? *(Must be assembled per project; published anchors exist only for grounding.)* - What is the judge's blind-spot profile on expertise-heavy content - can it detect its own incompetence and abstain, and are some projects AutoQA-ineligible? - [A] Does a multi-model ensemble buy error reduction, or do frontier judges share correlated blind spots - such that cross-family disagreement should be an escalation signal rather than a vote? - Is judge-judge agreement on grounding verdicts actually higher than the human reviewer-reviewer agreement we're fixing - as a falsifiable go/no-go gate before deploying at all? ## 6. Human-budget allocation, routing & arbitration With at most one human touch per item, routing IS the system. - [A] Is one-human-interaction-per-item even the right target - would zero per-item touch plus randomized deep blind audits outperform one shallow, automation-biased touch? *(Fork resolved by evidence + law: see FOUNDATIONS Section 5 - consequential adverse verdicts require a meaningful human decision for EU attempters; measurement is best served by sampled audits.)* - [A] What is the single interaction spent on - blind re-judgment, verdict verification, targeted micro-adjudication of contested claims, or the attempter's contest? *(Evidence-only assisted adjudication of the routed slice wins; showing the AI verdict causes measured over-reliance.)* - [A] Where in the verdict distribution does the touch buy most reliability - and must routing be closed-loop on observed overturn rates? - [A] Should scarce human minutes go to per-item adjudication at all, versus upstream compounding work (rubric disambiguation, exemplar-generating edge-case adjudications that write back into config)? - [A] Does escalation route the whole item or the specific contested claim/criterion? - Are escalation queues organized by criterion (specialist reviewers) or by item (generalist) - and does specialist routing measurably close the expertise-gap root cause? - [A] Does the escalated human judge blind or anchored - and at what point does a falling overturn rate signal rubber-stamping? - [A] Is the escalation threshold set to a fixed human-capacity budget or a target residual error rate? *(Both exist as dial profiles: error-target mode outputs the budget; pace-bound mode outputs the reported residual error - FOUNDATIONS P11.)* - [A] When contractual pace makes the standard threshold unaffordable, what may the quality/leniency dial legitimately move (aggregation cutoffs, escalation rate, verification depth, enforcement disposition) - and what may it never touch (judge perception, integrity verdicts, seeded gold/audit channels, the legal floor)? - [A] Is the dial one-sided by construction - dispositions loosen while measurement persists, so an expedited batch can be re-filtered to any strictness later from recorded scores (attempter-disposition vs dataset-admission consumers)? - [A] How do the Goodhart/divergence sensors condition on the active dial setting, so a leniency period is not misread as attempter improvement (or a strictness change as attempter decay)? - Who has authority to change a project's active operating profile, is every change logged, prospective-only, and announced to attempters - and is the stamped residual-error estimate disclosed to the data buyer? *(product-owner decision - FOUNDATIONS Section 8.6.)* - Which confidence proxy (stated confidence, k-sample agreement, paraphrase flip rate, ensemble disagreement, conformal set width, rationale-entailment score) actually predicts human overturn at routable AUROC? *(Never benchmarked head-to-head - in-house measurement.)* - [A] What is the asymmetric, cost-weighted decision rule for auto-pass / auto-fail / escalate - is auto-FAIL held to a stricter bar than auto-PASS, and who bears each error's cost? - [A] When human and AutoQA persistently disagree, what arbitrates - and how do we prevent the override stream from becoming unvalidated calibration data? - [A] What statistical model fuses sparse human labels with dense AI judgments? *(PPI++/EIF for rates and dashboards; TPR/FPR-corrected tests for threshold certification.)* - What is the cold-start protocol for a zero-label project, and what quantitative graduation criterion flips it into confidence-routed mode? ## 7. Constructive feedback: generation, grounding & efficacy The feedback channel held to its own standards - and falsifiable by attempter behavior change. - [A] Is the judge's feedback held reflexively to the same grounding standard it enforces - every criticism citing entailing evidence, passing its own verification gate before delivery - and what false-accusation precision SLO keeps attempter trust? - [A] Must every feedback statement be traceable to a specific clause in the project instructions - and what happens to real problems that map to no written criterion? - [A] What is the minimal feedback unit attempters actually act on - e.g., the 4-tuple (quoted claim, evidence span, criterion clause, concrete fix)? *(Feedback science: high-information task-referenced feedback ~ doubles effect vs bare verdicts; person/rank-referenced feedback backfires.)* - [A] What is the explainability contract for a failing verdict - verbatim evidence spans, checkable reasoning chain, fail-closed if the bar can't be met? *(Also a legal requirement for EU attempters: plain-language explanation + human review on request.)* - Does feedback need a different confidence bar than the verdict - can the system ship coaching on auto-passed items, and what is the harm model when a feedback claim is itself wrong? - [A] Does feedback measurably change attempter behavior - what longitudinal metric do we commit to, and after how long with no movement do we conclude decoration and reallocate? *(Priors now exist: median modest, concentrated in low performers, ~1/4 act on it; instrument per-attempter pre/post error-rate deltas and expect backfire in a minority.)* ## 8. Adversarial dynamics, incentives & consequences The system operates on paid workers who adapt to it. The moment verdicts carry consequences, the system is adversarial by construction. - Does feedback leak the judge's decision boundary - after N cycles, do attempter scores rise on the AutoQA while staying flat on held-out human-graded golden items? *(The single most important standing sensor; no published precedent - must build.)* - [A] Which shortcut heuristics does the judge use as pass/fail proxies, can pay-motivated attempters reverse-engineer them blind, and is the defense designed in from day one? *(Universal adversarial phrases transfer to unseen judges - filtering rationales is insufficient alone.)* - [A] Can the judge distinguish genuine grounding from citation theater - if we perturb or swap the cited evidence, does the verdict flip? - [A] How do we prevent feedback-induced monoculture - annotations converging on judge-pleasing patterns that destroy the diversity the training data exists to capture - and should feedback ever include stylistic guidance? *(Default: no stylistic guidance; population-level diversity drift metric from day one.)* - [A] For wholesale LLM-generated submissions, is the defense content-based detection or economics-based gating? *(Resolved: economics/provenance-based; content detection is beaten - agent passes 99.8% of checks at $0.05/task.)* - Do we care about LLM-assistance provenance or only content validity - is a correct, well-grounded, heavily-LLM-assisted annotation a pass, a fail, or per-project policy? *(product-owner/per-project decision; process telemetry has legal constraints - see FOUNDATIONS Section 10.)* - [A] Are AutoQA verdicts advisory or enforcement-grade - and if consequences attach, what precision floor, audit trail, and contest rights does each verdict need? *(Enforcement-grade adverse verdicts trigger the legal floor: human decision-maker, plain-language explanation, appeal with SLA.)* - [A] What appeal mechanism absorbs disputes without re-importing full human review load - and what appeal-overturn band signals a healthy vs broken judge? ## 9. Drift, monitoring & lifecycle integrity Keeping the certificate valid after launch. - [A] How do we detect drift in three distinct moving parts - attempter adaptation, instruction revisions, judge-model updates - with a different sensor and response for each? *(Frozen human-labeled anchor set + anytime-valid attribution separates \"judge moved\" from \"population moved.\")* - What falsifiable signal triggers a rubric revision (per-axis disagreement trend, override cluster density, appeal rate) at what threshold? *(No published drift-rate parameters; instrument and learn.)* - [A] Are judge model versions pinned per project with golden-set regression gates on upgrade? - [A] Once a project runs trusted-standalone, what randomized audit rate and drift statistic triggers automatic de-graduation - and is audit sampling unpredictable enough that attempters cannot write for the judge? - [A] Does the AutoQA prefilter shift the human reviewer's base rates enough to cause induced criterion drift - and does queue design mandate seeded known-verdict items and easy-case dilution? *(Content-moderation practice: yes, both are standard; seeded items must be indistinguishable.)* - [A] What keeps the calibration workforce competent once the AutoQA writes verdicts - does reviewer skill decay under reconcile-only work, and is a standing blind-judgment quota mandatory? *(Psychometrics: rater calibration decays within a single session; rolling-window reviewer-effect estimates, not one-time certification.)* --- ## Cross-cutting tensions Places where two legitimate goals pull in opposite directions. These are managed, not solved; FOUNDATIONS states the default posture for each. 1. **Standardization vs legitimate disagreement.** Enforcing consistent verdicts kills the reviewer-variance problem but risks laundering one arbitrary reading of a contested criterion into false objectivity; admitting ambiguity classes preserves legitimate disagreement but can become an excuse bucket. 2. **Decomposition vs holistic validity.** Atomic claim-level checks are tractable, auditable, falsifiable - and strip context (checklist myopia), re-importing subjectivity through the aggregation rule. 3. **Feedback usefulness vs Goodhart resistance.** The more specific and actionable the feedback, the more it publishes a gradient toward the judge's decision boundary that paid attempters will descend. 4. **Human as ground truth vs human as noise source.** The system exists because human reviewers are unreliable, yet escalation, overrides, appeals, and calibration all quietly treat a single human's judgment as truth. Every human touch chooses between efficient anchored review (contaminated labels) and expensive blind independence (usable calibration signal). 5. **Per-item accuracy vs compounding upstream investment.** The touch spent adjudicating this item maximizes per-item quality; the same minutes spent on rubric disambiguation improve all future items - but leave today's items unconfirmed. 6. **Grounding strictness vs false-flag rate.** Strict entailment fails nearly all human prose (which compresses and generalizes); loose \"reasonable support\" reintroduces the subjectivity the system was built to remove. 7. **Project-agnostic foundation vs per-project fidelity.** A closed config surface keeps \"agnostic\" falsifiable - but compilation fidelity, judge domain competence, and defect taxonomies constantly pressure the boundary. 8. **Enforcement weight vs error tolerance and trust.** Consequential verdicts create real deterrence and collapse acceptable error rates (and trigger legal duties); advisory verdicts preserve trust and tolerate error but lose incentive force. 9. **Contractual pace vs quality floor** *(added 2026-07-14 from the product owner's throughput requirement)*. Delivery obligations sometimes demand a looser operating point than the standard threshold; refusing any dial forces the loosening to happen off the books, while a dial that quietly loosens *measurement* (not just disposition) destroys the system's evidentiary value. Managed by pricing leniency: dispositions move, measurement doesn't, and every batch ships stamped with its profile and estimated residual error (FOUNDATIONS P11). "}, {"slug": "philosophy-and-structure", "title": "AutoQA Foundations - Philosophy & Structure", "category": "Core narrative", "summary": "The eleven principles, eight-stage architecture, operating model, legal floor, and unresolved leadership decisions.", "tags": ["philosophy", "architecture", "governance"], "view": "documents/view/philosophy-and-structure.html", "raw": "documents/raw/research/philosophy-and-structure.md", "text": "# AutoQA Foundations - Philosophy & Structure (draft for product-owner review) **Date:** 2026-07-14 **Status:** Draft v1, synthesized from `00_QUESTIONS.md` and `01_RESEARCH_ATLAS.md`. Every principle carries its evidence anchor; positions the evidence does not settle are marked **[default]** (my recommended posture, reversible) or **[the product owner]** (needs your call). **Scope:** Project-agnostic foundation for QA of human-annotated AI training data. Implementation infra (the Vercel app) is out of scope here. --- ## 0. The problem, restated precisely Human attempters produce evaluative work products (annotations, ratings, critiques with rationales) under project instructions. Human reviewers who QA them exhibit unexplained variance in pass/fail rates and reasons. Root causes, ranked: (1) subjective interpretation of evaluation axes, (2) expertise gaps, (3) attention to detail. Attempters are mandatory; the QA layer is what we augment. The psychometric literature sharpens this: after training, raters' *global* severity differences persist but are a minor variance component - the dominant variance is **rater x item x criterion interaction**: which reviewer, on which submission, on which criterion, this time. That variance cannot be fixed by better rubrics or calibration meetings alone. It can only be attacked by verifying each specific claim on each specific item against evidence - which is exactly what an LLM layer can do at scale and humans at scale cannot. That is the reason this system should exist, stated in the field's own terms. *(Atlas Section 26)* ## 1. The construct: what a verdict means **Quality = satisfaction of the written project instructions + grounding in the defined evidence record.** Not \"would the median reviewer pass this.\" Human-consensus prediction caps the system's validity at the reviewers' own (unmeasured, evidently low) reliability ceiling; instruction-satisfaction lets the AutoQA be *more* valid than the layer it augments, and makes every verdict traceable to a clause and a span. *(Atlas Section 16-18; peer-reviewed rating-indeterminacy work (NeurIPS 2025) shows forced-choice consensus gold labels can heavily bias judge selection - empirically up to 31% worse - when criteria admit multiple valid readings.)* Consequences of this choice: - Gold labels come from **adjudicated expert panels judging against the instructions**, not from single-reviewer or majority crowd labels. - When the AutoQA and a human reviewer disagree, neither is presumptively right; the arbiter is the instruction text plus the evidence record, adjudicated. - Where the instructions genuinely underdetermine an item, the correct output is not a verdict but an **instruction-gap report**. Reviewer disagreement on such items is signal about the instructions, not error in the people. *(Diverging Preferences: most annotator disagreement traces to task underspecification and style, not noise or incompetence.)* - The **ground-truth hierarchy** is explicit: seeded known-verdict items > adjudicated expert panel > criterion author's written amendment > individual human reviewer > AutoQA. Only the first three may feed judge calibration. Overrides by individual humans are logged as claims, not truth. ## 2. The eleven principles **P1 - Claims, not vibes.** The unit of analysis is the typed claim, not the item. Attempter writeups are decomposed into claims of distinct types, each verified in the lane built for it, each verdict carrying quoted evidence. Item verdicts are transparent, auditable roll-ups. *(Holistic judging of subjective quality measurably fails: near-orthogonal judge/human axes; ~55% on hard per-criterion checks; long-form instability. Decomposed per-criterion reasoning buys +7-12pp and lower cross-judge variance.)* **P2 - The claim ontology is the type system.** Every extracted statement is triaged before any verification: **(a) grounding claims** (cited evidence exists and says this), **(b) entailment claims** (evidence supports the conclusion drawn), **(c) calibration claims** (stated severity/confidence matches evidence strength), **(d) completeness/absence claims** (\"no errors,\" \"fully addresses\"), **(e) opinions** - never truth-scored, but checked with real machinery via two reusable passes: *stance-contradiction* (the evidence the attempter cites must not contradict the stated stance) and *stance-calibration* (stance strength must not exceed evidence strength - type-(c) machinery). Opinion verdicts are `consistent` / `inconsistent-with-cited-evidence` (a fail-able finding: the opinion's *warrant* fails grounding, not the opinion itself) / `uncheckable` (never penalized) - **(f) undecidable-given-evidence**. Each type has different machinery and a different reliability ceiling, reported per type. **Entailment standard [default]:** reasonable-expert support, not strict logical entailment - three-way output per claim (`supported` / `contradicted` / `unsupported`), where `contradicted` is the high-confidence fail trigger and `unsupported` alone triggers an evidence request or escalation rather than an automatic fail. Strict entailment fails nearly all competent human prose; loose support reintroduces the variance we're removing - so the false-flag rate under this standard is a measured ship-gate output (Section 4.3), not an assumption. *(VeriScore/FactBench triage is field-standard; FACTS shows the positive-class asymmetry; AbsenceBench caps type (d).)* **P3 - Evidence closure is explicit and judge priors are not evidence.** Every project config defines the evidence record: what the attempter saw, what they cited, what's in scope. Verdicts must quote spans from that record to be admissible; \"true in the world but unsupported in the record\" is `unsupported`, not `pass`. Checks are routed by whether they are **read-checkable or recompute-checkable** - judge confidence is untrustworthy on checks that exceed its own solving ability, where it silently re-solves and trusts itself over the record. **[default - provisional:** the re-solving evidence is a single under-review 2026 study; experiment 3 replicates it internally before this hard-gates routing.**]** Cherry-picking is caught by running both span-closed and context-closed citation checks. The judge gets the same evidence surface the attempter had - architecture beats model. *(Auditing-by-Re-Solving; CiteEval; agent-judge/the banking meta-eval.)* **P4 - Positive claims are verified by enumerated class, with honest labels.** \"Warranted praise\" verification is restricted to checkable praise types compiled per project (accuracy-praise -> grounding check; completeness-praise -> enumerable checklist; clarity-praise -> opinion class). Absence claims get a seeded-defect protocol: unless the judge's planted-flaw recall on this project materially exceeds the attempter population's, its concurrence is labeled **`not-contradicted`, never `verified`**. A vacuousness/eligibility gate precedes grounding checks so hedged empty writeups can't pass by asserting nothing. *(FACTS eligibility gate; AbsenceBench; zero published praise-verification numbers - this is the first in-house measurement.)* **P5 - Typed verdicts; abstention and contestation are first-class.** The verdict set is `pass` / `fail` / `unable-to-verify` / `instructions-underdetermine`. High vote-entropy across k samples, cross-family judge disagreement, and clustered reviewer disagreement route items to the contested class - which feeds the instruction-gap channel and never penalizes the attempter. This is how the system enforces consistency where consistency is real, without laundering one arbitrary reading of a contested criterion into false objectivity. *(Toloka's deployed Pass/Fail/Unable-to-verify; CoVal; NUTMEG; rating indeterminacy.)* **P6 - The human touch is spent where it compounds, and it is blind.** Routing is confidence-calibrated three-lane (auto-verdict / stronger judge / human) with the human budget derived from a target residual error rate, not a fixed per-item quota. The escalated human sees the claim, the criterion clause, and the judge's **collected evidence spans - not its verdict or rationale** (verdict-visible review produces measured over-reliance; evidence-only is the sole format that helps when the AI is right without hurting when it's wrong). Escalation routes the contested *claim*, not the whole item. Every human adjudication writes back - in two tiers that keep the ground-truth hierarchy honest: a single-touch adjudication produces **candidate** artifacts (logged claims, provisional exemplars), and only panel-adjudicated or owner-signed artifacts enter judge calibration and compiled config; the standing audit channel batch-promotes candidates. In enforcement mode the human budget is max(target-residual-error escalations, consequential-fail volume - Section 5); under a pace-bound profile (P11) capacity binds instead, and the residual error rate becomes the reported output rather than the target - never pretend to optimize both. The evidence-only-for-adjudicators result is measured; combining it with full critiques to attempters (P9) is an inferred, untested split **[default]**, A/B'd in experiment 6. Overturn-rate monitoring proves the human lane adds signal; an overturn rate at the judge's error floor means rubber-stamping, and the naive human-verifies-AI design is the modal failure in the literature. *(Trust-or-Escalate/HyPAC; DeepMind complementarity; Vaccaro meta-analysis; Braintrust/LangChain/micro1 write-back practice.)* **P7 - The QA layer is measured exactly like what it measures.** The judge is enrolled as one more rater facet in the same measurement model as the humans (MFRM-style severity/centrality/fit per rater, human or machine, per project). Scope honestly: MFRM needs 3-5 ratings/output of rater-item linkage, so it runs fully on the dual-review pilot and continuously on the judge facet (which rates everything); production human-reviewer severity is estimable only where deliberate overlap is injected - a low-rate stream of double-reviewed seeded items, a budgeted cost, not a free byproduct. All reporting uses chance-corrected statistics, per-class recall at fixed prevalence, and PPI-corrected population estimates with dual-uncertainty CIs - raw percent-agreement and single-number \"accuracy vs humans\" are banned artifacts. Feedback passes the same grounding gate it enforces before delivery (fail-closed). Validation stratifies by measured human-agreement level; verdict-agreement alone never certifies a judge, because right-verdict-wrong-rationale poisons feedback and trust (in one benchmark, 24.8% of items got the right verdict with a low-quality critique - RealCritic). **Staging [default]:** v1 ships chance-corrected agreement, PPI-corrected point estimates with CIs, and k-sample entropy routing; MFRM production enrollment, anytime-valid e-processes, and conformal routing are phase-2 refinements - the full stack is the target, not the launch gate. *(MFRM-for-LLM-judges; kappa-deflation cohort; BeyondCorrelation; RealCritic; MetaCritique.)* **P8 - Adversarial by construction, from day one.** Attempters are paid optimizers; some are agents. Defenses are structural, not detective: stakes-sterile judge prompts (consequence framing measurably softens verdicts with zero CoT trace); comparative/anchored scoring over absolute scales (transfer attacks inflate absolute scores blind); evidence-perturbation probes as a standing citation-theater detector (swap the cited span -> verdict must flip); blind seeded gold injected into live queues (indistinguishability is the binding constraint); **the divergence sensor** - attempter scores rising on the AutoQA while flat on held-out human-graded goldens - as the canonical Goodhart alarm; spam defense that is economics/provenance-based (throttling, work-history consistency, worker-level peer-prediction scoring, randomized deep audits), with content detection demoted to a weak auxiliary signal. Never score anyone on agreement-with-final-verdict - consensus-rewarded auditing trains your best dissenters to conform exactly where independent judgment matters most. *(Evaluation Faking; universal adversarial phrases; TSPA; Westwood PNAS; Community Notes conformity study.)* **P9 - Feedback is an instrumented intervention, not decoration and not free.** The feedback unit is the 4-tuple: quoted claim, evidence span, criterion clause, concrete fix - task-referenced always, person- or rank-referenced never (a third of feedback interventions reduce performance; high-information task feedback roughly doubles the effect of bare verdicts). Benchmark against exemplar work, never ranked peers. No stylistic guidance by default - feedback is a homogenization pump aimed at the diversity the training data exists to capture, so a population-level output-diversity drift metric ships with the channel. Efficacy is measured per-attempter (repeat-error-rate per criterion, pre/post), expecting modest median effects concentrated in low performers, and separating within-attempter learning from attrition. If high-information feedback doesn't move repeat-error rates in a defined window, conclude decoration and reallocate. *(Kluger & DeNisi; Wisniewski; Cochrane 2025; Dow CSCW 2012; monoculture evidence.)* **P10 - Everything drifts; every moving part gets its own sensor.** Frozen human-labeled anchor sets interleaved into the live stream attribute drift (judge moved vs population moved) with anytime-valid statistics. Judge versions are pinned per project with golden-set regression gates on upgrade; rubric revision is a versioned loop with explicit retroactive re-scoring policy (adaptation lives in the compilation layer, pinning at the judge layer - that's the resolution of the static-vs-co-evolving fork). Reviewer calibration decays within a session, so reviewer-effect estimates run in rolling windows with hard-item interleaving and order randomization; graduated projects carry randomized, unpredictable audit rates with automatic de-graduation triggers. Queue composition is managed (seeded known-verdict items, easy-case dilution) because prefiltered queues measurably shift reviewer strictness. *(Anchor-set e-process work; DRIFT/OSCE; Pinterest dual monitors; content-moderation practice.)* **P11 - Quality is a priced dial on disposition, never on perception.** Contractual pace sometimes makes the standard quality threshold unaffordable; the system supports that honestly instead of forcing the loosening to happen off the books. The judge always measures the same way - dial-sterile prompts, same claim verification, same evidence dossier (consequence framing measurably corrupts verdicts with zero CoT trace, and a \"be lenient\" judge destroys score comparability, calibration, and the divergence sensor simultaneously). Leniency is applied downstream, as thresholds over the continuous scores the pipeline already emits. A project's active **operating profile** (e.g., `strict` / `standard` / `expedited`) moves exactly four things: aggregation cutoffs (how many/how severe findings fail an item), routing thresholds (escalation rate, human budget), verification depth (k-samples, deep-verify sampling fraction, expensive lanes like context-closed citation checks), and enforcement disposition (findings -> fail vs findings -> coaching-only). Two constraints can bind: in error-target mode the residual error rate is the target and human budget is the output; in pace-bound mode (contractual throughput) capacity binds and **the residual error rate becomes the reported output** - either way the number is computed and stamped, never hidden, with CIs that honestly widen when verification depth is thinned. Because measurement persists under every profile, the dial has two independent consumers: the **attempter-disposition dial** (prospective only, logged, announced - the bar people are judged against never moves silently or retroactively) and the **dataset-admission dial** (re-tunable at any time - an expedited batch can be re-filtered to any strictness later from the recorded scores, without re-running QA; the data buyer's strictness is not coupled to the deadline that produced the batch). Floors the dial can never touch: integrity-class verdicts (fabrication, spam, falsehood), seeded-gold injection and audit sampling, the divergence sensor (which conditions on the active profile and reads seeded/audit outcomes, not pass rates), and the Section 5 legal floor. Every delivered batch carries its profile and its corrected residual-error estimate - leniency becomes a contractual artifact instead of quiet degradation. *(Evaluation Faking -> dial-sterile prompts; CriticGPT's FSBS precision/comprehensiveness dial as the precedent for inference-time operating points; Trust-or-Escalate's coverage dial; HyPAC error budgets; corrected-estimator reporting.)* ## 3. The architecture Eight stages. Stages 0-2 are deterministic-ish and cheap; 3 is where models spend; 4-7 close the loops. ``` project instructions --> [1] COMPILE (workshop loop, versioned config) | attempter submission --> [0] INTAKE GATE --> [2] DECOMPOSE & TYPE --> [3] VERIFY (lanes) | +-------------------------------------------+ v [4] AGGREGATE (typed verdict + evidence dossier) | [5] ROUTE (auto / stronger judge / human, calibrated) | [6] FEEDBACK (4-tuple, self-verified, fail-closed) | [7] META-EVAL & MONITOR (gold, drift, divergence, dashboards) ``` **Stage 0 - Intake & provenance gate.** Worker-level, not item-level: rate/velocity limits, work-history consistency, peer-prediction information scores, session-bounded telemetry (allowlist schema - on-task signals only; see Section 5). Item-level: eligibility/vacuousness check (is this a responsive, contentful submission at all). No LLM-text detection verdicts. **Stage 1 - Compilation layer (the per-project customization contract).** Project instructions compile into a closed config schema: - **Axis triage:** every evaluation axis classified into `deterministic` (string/regex/programmatic checks), `judge-groundable` (evidence-anchored verdicts), `judge-assisted` (human decides with judge-collected evidence), `human-only`, or `AutoQA-ineligible` - using per-axis conformal set width and pilot judge-vs-adjudicated agreement as the sorting evidence. - **Rubric criteria:** <= ~13 active criteria per judge call, each atomic, RIFT-linted; **pitfall/negative criteria human-authored** (the one artifact synthesis reliably can't produce); hard constraints get gating semantics, not soft-criterion listing. - **Contrastive exemplars:** hard near-miss pass/fail pairs per criterion - discriminative exemplars, not descriptive \"what good looks like\" prose. These anchor the judge few-shot (the measured-best grounding architecture) and double as reviewer training material. - **Evidence-closure rules:** what the judge may read, per claim type; citation requirements on load-bearing attempter claims. - **Severity/weight taxonomy + aggregation formula** (explicit, auditable - accepting the known performance tax vs implicit aggregation). - **Routing costs and thresholds:** the asymmetric error prices (false-fail vs false-pass), the target residual error rate, calibration set. - **Operating profiles (the quality/leniency dial, P11):** named threshold sets (`strict` / `standard` / `expedited`) taken from the ship gate's measured operating-point curve, each stamped with its expected fail-class recall and false-pass rate. Profile changes are owner-authorized, logged, prospective-only, and announced to attempters whenever the enforcement bar moves. Compilation is a **loop, not a compile step**: owner grades real items, criteria drift is expected, every version is kept, and sign-off happens on graded items, not abstract criteria. Compilation fidelity is the binding per-project cost - budget accordingly. **Stage 2 - Decompose & type.** Claim extraction at adaptive granularity (claim + explicit context annotations, verifier told what's under test; never free-floating atomic rewrites); ontology typing per P2; canonicalization/style-stripping so verbosity and rubric-vocabulary mimicry don't leak into judgment. **Stage 3 - Verification lanes.** - *Deterministic lane:* compiled checks run on everything, always. - *Screening tier:* cheap specialized checkers (MiniCheck-class entailment, per-metric small models) score every claim at ~1/100 frontier cost - triage, never verdicts. - *Verdict tier:* frontier reasoning judge, per-criterion calls, exemplar-anchored, cross-family from the models attempters plausibly used, stakes-sterile prompts, k-sampled with continuous scores (logit-expectation rather than sampled tokens where available), position/order randomized. - *Opinion-consistency lane:* stance-contradiction + stance-calibration checks against the attempter's cited evidence, with the P2(e) verdict semantics - runs in the same screening/verdict tiers as other claim types. - *Contested detector:* vote entropy, paraphrase flips, cross-family disagreement -> `instructions-underdetermine` candidates. **Stage 4 - Aggregation.** Claim verdicts -> item verdict via the compiled formula; error-tolerant (a ~20% claim-level error rate means naive AND-aggregation is wrong by construction); output is the **verdict + evidence dossier** (every finding with quoted spans and criterion clauses) - the dossier is simultaneously the audit trail, the appeal record, and the feedback source. **Stage 5 - Routing & the human lane.** Calibrated dual thresholds (HyPAC-style) with the human queue = the abstention set; auto-FAIL held to a stricter bar than auto-PASS (false fails burn trust and pay; consequential fails have a legal floor - Section 5); blind evidence-only adjudication UI; claim-level micro-adjudication; adjudications write back per P6's two-tier rule; overturn telemetry closed-loop. Queue organization (specialist-per-criterion vs generalist-per-item) is a compilation-layer decision informed by the pilot's expertise-share finding - if expertise gaps dominate a project's variance, specialist claim-level routing is the mechanism that closes root cause #2 on the human side. **Cold start** for a zero-label project: the judge runs shadow-mode under full human review until the calibration set reaches the PPI-arithmetic size and thresholds are fitted; **graduation** to confidence-routing happens when calibrated thresholds hit the target residual error on a rolling window - mirrored by P10's de-graduation triggers. **Stage 6 - Feedback channel.** Per P9. Generated from the dossier (verdict machinery and feedback share one artifact), passed through its own grounding gate, delivered only if it survives. **Stage 7 - Meta-evaluation & monitoring.** Per-project ship gate before go-live (Section 4); continuous: blind gold injection, frozen anchor set, divergence sensor, per-criterion agreement trends, appeal-overturn bands, population diversity metric, reviewer rolling-window effects, corrected-estimator dashboards. The standing expert-audit channel samples confident passes - **critic-assisted** (a critic-assisted second look finds 4x what an unassisted one does on the \"flawless\" slice) - and is permanent, because when judge accuracy <= attempter accuracy no statistical cleverness can substitute for gold (the 2x theorem), and false agreement is predicted to rise as model errors correlate. ## 4. The ship gate (per project config, before go-live) A project's judge config ships only after: 1. **Variance-attribution pilot:** dual-review a common item set; measure the human floor (chance-corrected, per criterion - per-criterion kappa localizes ambiguous criteria); code disagreements against an underspecification-vs-expertise-vs-attention taxonomy. This decides the build emphasis per project and provides the calibration set. 2. **MVVP-style judge validation:** chance-corrected agreement vs adjudicated labels, >=3 replicates, AB/BA position swaps, >=2 gold-set designs, bias audit separate from consistency (a judge can be perfectly repeatable and severely biased). 3. **Perturbation harness:** meaning-preserving paraphrase/reorder/noise on claims AND evidence; evidence-swap probes; verdicts that flip get routed, not shipped. 4. **Seeded-flaw recall:** planted defects (FindTheFlaws-style) measuring fail-class recall at realistic prevalence - the number that actually matters under 90%+ pass rates, where kappa itself misleads (Feinstein-Cicchetti). 5. **Schematic adherence:** do per-criterion scores vary independently and does the rationale's cited schema explain the verdict (halo/factor-collapse check) before per-criterion output is exposed to anyone. 6. **Gold-set arithmetic:** size from the PPI closed form off pilot judge-human correlation per axis (conservative estimates; ~50+ labels per stratum floor; oversample the fail class); low-correlation axes carry human-only until correlation improves. 7. **Beat-the-humans gate:** per axis, judge-vs-adjudicated reliability must exceed the pilot's measured human-reviewer-vs-adjudicated reliability. Axes where the judge doesn't beat the humans it augments are human-only lanes - automating them adds cost and subtracts validity. This is the go/no-go the whole premise rests on, and it is comparative, not absolute. Beyond go/no-go, the ship gate's data product is the **operating-point curve** - thresholds -> fail-class recall and false-pass rate at pilot prevalence. This is what calibrates P11's dial profiles, so even the `expedited` setting ships with a priced, not guessed, error rate. ## 5. The legal floor (binding if the workforce touches the EU) **Gate first [the product owner]:** determine whether the attempter population includes EU-resident workers. If yes, this section is a launch constraint; if the pilot is US-only, treat it as forward-looking scope and revisit before any EU expansion. From 2026-12-02 (Platform Work Directive transposition), for attempters in the EU - employee or self-employed, platform established anywhere: - Decisions of significant detriment (account/pay restriction, termination, or equivalent) **must be taken by a human**; no consent exception. Auto-pass is fine; consequential auto-fail is not. Whether a single-item pay denial qualifies is legally untested - **[default]** treat consequential item fails as requiring the human lane until counsel says otherwise. - Plain-language explanation of any automated-supported decision, a competent human contact, human review with substantiated written reply within two weeks, rectification/compensation duties. The evidence dossier and 4-tuple feedback ARE this explanation - design them to that bar. - Telemetry: allowlist only (task-session-bounded behavioral signals); no emotional-state inference, no private-communication monitoring, no off-task collection; DPIA required. - SCHUFA doctrine: a score with a \"determining role\" is itself an automated decision, and rubber-stamp review counts as zero-touch - the human lane must have authority, evidence access, and calibrated competence. Enforcement is active. - Map quality dimensions and verdict evidence onto ISO/IEC 5259 vocabulary now (cheap) rather than retrofit later (expensive). Convenient truth: the legally mandated design (meaningful human on adverse decisions, explanations, appeals) is the same design the complementarity evidence recommends anyway. ## 6. What this system deliberately does not do - No per-item LLM-text spam classification as a primary defense (beaten economically and technically). - No holistic single-call item judging; no per-criterion output without schematic-adherence validation. - No raw percent-agreement or single-number accuracy anywhere in reporting. - No truth verdicts on opinion-class claims; no `verified` label on absence claims without seeded-recall evidence. - No stylistic feedback; no visible attempter leaderboards/rank feedback. - No consequence language in judge prompts; no trusting CoT rationales as bias evidence. - No scoring of humans (attempters, reviewers, or auditors) on agreement-with-final-verdict. - No judge calibration from unadjudicated override streams. - No leniency via judge prompting - the dial moves thresholds, depth, and disposition, never the judge's perception (dial-sterile is the same discipline as stakes-sterile). No dial setting touches integrity floors, seeded gold, audit sampling, or the legal floor. No silent or retroactive changes to the bar attempters are judged against. - No claim that the judge validates itself - the expert-audit channel never sunsets. ## 7. First experiments (ordered; each kills or funds a component) 1. **Variance-attribution pilot** on one live project (dual review + disagreement coding). Decides whether rubric compilation or expertise routing is the bigger lever *for our population*, and produces the first calibration set. Everything else calibrates against this. 2. **Positive-claim seeded set:** build warranted/empty-praise pairs per praise type; measure judge precision/recall per type. Nothing published exists - this is both the riskiest premise and the differentiator. Include absence-claim recall (planted defects behind \"no errors\" writeups). 3. **Transfer check:** take the compiled-config judge and measure per-criterion agreement against adjudicated labels on *human evaluative writeups* (not model outputs) - the number every published anchor is missing. Include a re-solving probe: does our judge's wrong-reference detection collapse on recompute-class checks (internal replication of the under-review 2026 result P3 provisionally leans on)? 4. **Confidence-proxy shootout:** verbalized confidence vs k-sample entropy vs conformal width vs cross-family disagreement, scored on predicting human overturn - this picks the routing statistic. Include the uniform-sampling baseline. 5. **Divergence sensor dry run:** stand up held-out human-graded goldens + AutoQA score tracking from the first live week, so Goodhart is measurable before it's a problem. 6. **Feedback tier A/B:** verdict-only vs 4-tuple high-information feedback, outcome = per-criterion repeat-error rate, attrition separated. Priors: modest median, concentrated in low performers, backfire in a minority. Also A/B the assistance-format split (full critiques to attempters vs evidence-only) - the combination P6/P9 defaults to is untested in any single study. 7. **Agnosticism falsification:** onboard a second project through config alone. Any edit to the core judge prompts falsifies the project-agnostic claim - the fix is moving that surface into the config schema, not patching the core. ## 8. Open decisions **[the product owner]** 1. **Enforcement weight at launch:** advisory/coaching-only vs consequential verdicts. Changes the precision floor, the legal posture (Section 5), and attempter adversarial pressure. My read: launch advisory, attach consequences only per-lane after seeded-recall numbers exist. 2. **Construct authority when project owner intuition contradicts their own compiled rubric:** rubric-as-law with forced amendment **[default]**, or standing owner overrides (which quietly re-import unexplainable variance). 3. **LLM-assistance provenance policy default:** content-validity-only vs provenance-gated (per-project override either way). Interacts with telemetry legality and with the reality that ~1/3 of text workers already use LLMs. 4. **Judge sourcing:** frontier API judges per project (cross-family routing) only, or also an owned fine-tuned screening tier. The evidence supports two-tier; the question is whether to own the bottom tier. 5. **Where the first pilot runs** - which project's data, whether its owner will fund the adjudicated gold set (the one cost that cannot be automated away), and whether that attempter population includes EU-resident workers (switches Section 5 from forward-looking to binding). 6. **Dial governance (P11):** who may change a project's active operating profile (project owner alone, or with client sign-off), and whether expedited batches are disclosed to the data buyer with their stamped residual-error estimate. My read: disclose - a priced error rate converts deadline pressure from hidden liability into a contractual artifact, and it is the posture consistent with the ISO 5259 mapping in Section 5. --- *Lineage: brainstorm (92 questions, 7 lenses) -> taxonomy (9 themes, 8 tensions) -> 9-domain recency-biased sweep + 27 adversarial verifications + 4 gap-fill literatures -> this synthesis. Full provenance in `atlas/`.* "}, {"slug": "raw-questions", "title": "Raw Questions by Lens", "category": "Research working papers", "summary": "The full 92-question brainstorm across measurement, judge failure modes, rubric operations, human factors, grounding, workflow, and feedback.", "tags": ["questions", "brainstorm", "research agenda"], "view": "documents/view/raw-questions.html", "raw": "documents/raw/research/raw-questions.md", "text": "# Raw questions by lens ## Lens: measurement ### What is the actual ground-truth construct we're measuring - 'would an expert consensus panel pass this item?' or 'does this item satisfy the written project instructions?' - and when those two diverge, which one wins? WHY: Human reviewers already disagree, so 'agreement with human reviewers' is a moving target. If the construct is instruction-satisfaction, the AutoQA can be MORE valid than the humans it's checked against; if it's human-consensus prediction, validity is capped at the humans' own reliability ceiling. GATES: Whether gold labels come from adjudicated expert panels vs. instruction-derived rubric checks; whether disagreement-with-humans is treated as AutoQA error or human error during meta-evaluation. ### Given that human reviewers show unexplained variance, what is the measured inter-reviewer reliability floor (e.g., Krippendorff's alpha on a common item set), and is it even high enough for human agreement to serve as a validation target at all? WHY: If human-human alpha is, say, 0.35, then judge-human agreement of 0.4 is indistinguishable from perfect performance and 0.7 is impossible - every downstream accuracy claim is bounded by this number. Without measuring it first, we can't tell whether the AutoQA is failing or the target is noise. GATES: Whether meta-evaluation uses raw human labels, adjudicated labels, or noise-corrected agreement statistics; the sample sizes and acceptance thresholds for the AutoQA's own validation. ### At what granularity of claim decomposition (whole-review pass/fail vs. per-criterion verdict vs. per-atomic-claim groundedness check) does judge reliability stop improving - and does aggregation of atomic verdicts back to an item-level decision introduce its own error mode? WHY: The design requires evaluating positive statements ('is this claim grounded in cited evidence?'), which implies decomposition. But decomposition trades one subjective judgment for N smaller ones plus an aggregation rule; if the aggregation rule (any-fail, weighted, threshold) is itself unvalidated, we've moved the subjectivity, not removed it. GATES: The core unit of analysis for the AutoQA verdict schema and the aggregation function from claim-level checks to item-level pass/fail/feedback. ### Which failure types are objectively verifiable (fabricated citation, claim contradicted by evidence, instruction violation with a bright line) vs. irreducibly judgment-laden (severity ratings, 'tone', borderline quality), and should the AutoQA emit categorically different output types (verdict vs. flag-for-human vs. confidence-weighted opinion) for each? WHY: Treating all checks as one verdict type forces a single reliability standard on axes with wildly different measurability. The one-human-interaction budget should be spent exclusively on the judgment-laden residue, which requires knowing where the objective/subjective boundary actually sits per project. GATES: The output taxonomy (auto-decide vs. escalate), and how the per-project customization layer classifies each rubric axis at onboarding. ### Are the AutoQA's confidence scores actually calibrated per-project and per-axis (does 80% confidence mean 80% agreement with adjudicated truth?), and what recalibration mechanism runs when a new project's instruction set shifts the distribution? WHY: The escalation policy (which items consume the one human interaction) is only as good as the calibration behind it. LLM judges are notoriously miscalibrated and calibration does not transfer across domains; an uncalibrated escalation threshold silently converts into either rubber-stamping or full human re-review. GATES: Whether to build a per-project calibration/threshold-fitting step into onboarding, what the minimum labeled-item count for it is, and whether raw model confidence is ever exposed to humans. ### What is the minimum viable gold set per project (size, sampling design, refresh rate), given base rates - if only 5-10% of attempts are genuinely bad, how many adjudicated items do we need before we can distinguish a 90%-precision judge from a 70% one? WHY: Low base rates of spam/falsehood mean precision/recall estimates have huge confidence intervals at plausible gold-set sizes; a 200-item gold set with 15 true failures cannot certify the system. Sample-size arithmetic decides whether per-project validation is feasible or whether we must pool evidence across projects. GATES: Gold-set construction protocol, whether validation is per-project or hierarchical/pooled, and the honest error bars we can put on any launch claim. ### How do we detect drift in three distinct moving parts - attempter behavior (including adversarial adaptation to the QA), project instruction revisions, and judge-model updates - and which statistic (verdict-rate shift, escalation-rate shift, gold-set score drop) triggers re-validation for each? WHY: A static validation certificate decays: attempters learn what the AutoQA checks and game it, instructions get amended mid-project, and the underlying model gets swapped. Each drift source needs a different sensor and a different response; conflating them means either constant false alarms or silent decay. GATES: The monitoring dashboard's core metrics, the re-validation trigger policy, and whether gold items must be continuously injected into the live stream (blind gold) vs. periodically batch-audited. ### For groundedness checks on positive statements, what counts as the evidence closure - only sources the attempter explicitly cited, the full item context, or external knowledge the judge happens to have - and how do we stop the judge from 'verifying' claims against its own priors rather than the record? WHY: The stated goal is 'is this claim grounded in the evidence they cite,' which is a verifiability judgment relative to a defined corpus. If the evidence boundary is undefined, the judge will substitute its parametric knowledge, producing verdicts that are confident, unauditable, and wrong in exactly the expertise-gap cases the system exists to catch. GATES: The judge's context-construction rules (what it may and may not see), and whether groundedness verdicts must quote span-level evidence to be admissible. ### How do we meta-evaluate the constructive-feedback channel - by downstream attempter improvement (revision acceptance rate, repeat-error rate) rather than feedback 'quality' ratings - and are we willing to run the controlled comparison (feedback vs. verdict-only) to find out if feedback does anything? WHY: Feedback quality has no natural gold label and pairwise 'this feedback seems good' ratings inherit all the subjectivity problems we started with. The only defensible criterion is behavioral: does the attempter's next submission improve? If we can't or won't measure that, the feedback component is unfalsifiable decoration. GATES: Whether feedback generation gets its own outcome-based evaluation loop with attempter-level longitudinal tracking, and what data the platform must log to enable it. ### What is the deployment decision rule that translates measured judge performance into routing - at what precision/recall on which failure class does the AutoQA get to auto-fail an item without a human, auto-pass one, or must escalate - and who bears the asymmetric costs of each error? WHY: False auto-fails burn attempter trust and pay; false auto-passes poison training data downstream. These costs are asymmetric and project-specific, so a single accuracy threshold is wrong by construction; the routing policy must be an explicit cost-weighted decision rule, not a vibe. GATES: The three-way routing policy (auto-pass / auto-fail / escalate), its per-project cost parameters, and which decisions are reversible via appeal. ### Contrarian: is inter-reviewer disagreement actually the disease, or is some of it legitimate signal about genuinely ambiguous items and underspecified instructions - and would an AutoQA that enforces consistency be laundering one arbitrary interpretation into a false appearance of objectivity? WHY: High agreement is trivially achievable by collapsing to a crude, consistent-but-invalid rule (reliability without validity). If disagreement concentrates on items where the instructions are genuinely underdetermined, the right output is 'flag instruction gap for the project owner,' not a confident verdict - and the AutoQA becomes an instrument for measuring instruction quality, not just attempter quality. GATES: Whether the system includes an 'instructions underdetermine this case' verdict class and a feedback loop to project owners, vs. always forcing a pass/fail. ### When the AutoQA and the single human interaction disagree on an item, what is the adjudication and logging protocol - does the human always win, and how do we prevent the human-override stream from silently becoming the new (unvalidated) training/calibration signal? WHY: The one-human-per-item budget means most overrides come from a single reviewer - exactly the noisy, poorly-aligned population we started with. If overrides automatically count as ground truth for recalibration, we recycle human inconsistency back into the judge; if they don't, we have no learning loop. GATES: The override-handling policy, whether disagreement items get periodic second-opinion audits (spending extra human budget on a sampled subset), and what feeds the calibration/fine-tuning data. ### Which agreement statistic and evaluation design do we standardize on given that pass rates are extreme and vary by project - and do we explicitly ban raw percent-agreement and single-number 'accuracy vs. humans' from all reporting? WHY: At a 92% pass rate, a judge that passes everything scores 92% agreement and kappa behaves pathologically (the base-rate paradox); different per-project prevalences make cross-project comparisons of any single statistic meaningless. Choosing the metric contract up front prevents every later performance claim from being an artifact. GATES: The standard meta-evaluation report format: chance-corrected statistic choice (e.g., Krippendorff's alpha + per-class recall at fixed prevalence), stratified-by-failure-class reporting, and minimum CI-width requirements before any threshold change ships. ## Lens: judge-failure-modes ### What is the measured run-to-run verdict flip rate for our judge on identical items at fixed temperature, and does the AutoQA architecture treat that number as noise to average away (k-sample voting) or as signal that the criterion itself is underspecified? WHY: If flip rate on a criterion exceeds the human reviewer disagreement rate we are trying to fix, the judge adds variance rather than removing it. Distinguishing stochastic noise from criterion ambiguity determines whether the fix is ensembling or rubric rewriting - two completely different builds. GATES: Whether to spend the inference budget on k-run self-consistency voting versus a rubric-decomposition pipeline; also which criteria get routed to the single allowed human interaction. ### When the judge verifies a POSITIVE attempter claim (\"this statement is grounded in the cited evidence\"), does it exhibit an agreement-default bias - passing plausible-sounding but ungrounded claims at a higher rate than it fails grounded ones - and how large is that asymmetry on a seeded test set? WHY: Verification of positive statements is the core novel requirement here, and sycophancy/plausibility bias runs exactly against it: judges are known to rubber-stamp fluent assertions. If the false-pass rate on seeded ungrounded claims is high, the architecture needs adversarial framing (e.g., 'find the flaw in this claim' prompts, or claim-evidence entailment decomposition) rather than holistic pass/fail judging. GATES: Whether positive-claim verification is built as holistic judgment, as a decomposed entailment check per cited-evidence pair, or as a debate/red-team prompt structure. ### Does the judge systematically favor verbose, hedged, or rubric-vocabulary-mimicking attempter writeups over terse-but-correct ones, and by how many points of pass-rate delta on length/style-controlled pairs? WHY: Verbosity and style bias would train attempters (who adapt fast to QA signals) to pad and mimic rather than improve substance - the AutoQA would actively corrupt the annotation pool it polices. A measured delta tells us whether we need style-normalization (summarize-then-judge, content extraction) before judgment. GATES: Whether to insert a canonicalization/extraction layer that strips style before the judge sees the annotation, versus judging raw attempter text. ### When the attempter's annotation critiques an AI output produced by the same model family as our judge, does self-preference bias suppress the judge's agreement with valid criticisms - and does routing those items to a different model family close the gap? WHY: The annotated content is AI output; attempters saying 'this AI response is wrong' will be second-guessed by a judge that shares the AI's blind spots or prefers its own family's outputs. This is a structural conflict of interest unique to QA-of-AI-critique settings. GATES: Model-choice policy: whether the judge must be a different family than the generator of the content under annotation, and whether that requires per-project judge routing. ### For each evaluation-axis type (factual grounding, instruction compliance, tone/style quality, holistic ratings), what is the judge-vs-adjudicated-human agreement ceiling - and on which axis types does the judge fall below the human-reviewer inter-rater agreement we already consider broken? WHY: Root cause #1 is subjective interpretation of criteria. If the judge is no better than humans on subjective axes (likely), the design must partition axes: judge-autonomous, judge-assisted-human, and human-only lanes. Pretending one judge handles all axes reproduces the exact variance problem. GATES: The routing taxonomy: which criterion types the AutoQA decides alone, which consume the one human interaction, and which it must abstain on. ### At what rubric decomposition granularity does judge-human agreement stop improving - and is there a granularity beyond which agreement degrades because atomized checks lose context (the judge passes every micro-item while missing the writeup being globally wrong)? WHY: The reflexive fix for judge unreliability is finer rubrics, but decomposition has a measurable optimum and a failure mode (checklist myopia). The project-agnostic foundation must specify how project instructions get compiled into judge criteria, and that compiler needs a target granularity. GATES: The design of the per-project customization layer: how instruction sets are decomposed into judge prompts, and whether a holistic synthesis pass sits on top of atomic checks. ### How sensitive are verdicts to semantically-neutral perturbations of the judge prompt (criterion ordering, pass/fail label ordering, few-shot exemplar choice, evidence presentation order), and do we require paraphrase-invariance testing as a gate before any project's judge config ships? WHY: Position and prompt-sensitivity effects can dwarf real quality signal. Since every project customizes the prompt, an unstable base template means each project's customization silently re-rolls the judge's biases - variance across projects that looks exactly like the human variance we set out to kill. GATES: Whether the foundation includes a mandatory perturbation-robustness test harness and randomized-order/position-debiasing defaults in the base template, versus trusting per-project prompt authors. ### What calibration signal does the judge emit per verdict (stated confidence, sample agreement rate, logit margin), and is any of them actually predictive of judge error well enough to drive the route-to-human decision at a fixed one-touch budget? WHY: The one-human-interaction constraint makes triage the load-bearing mechanism. If judge confidence is uncalibrated (common - stated confidence often near-useless), the human touch gets spent on the wrong items and the system's error rate is set by the worst unrouted verdicts. GATES: The escalation mechanism design: self-reported confidence vs. k-sample disagreement vs. dual-judge disagreement as the routing trigger, and what abstention thresholds to set. ### Which shortcut heuristics does the judge use as pass/fail proxies (presence of citations regardless of relevance, rubric keyword echo, annotation length, score extremity), and can attempters who see the feedback reverse-engineer them within a few submission cycles? WHY: The AutoQA emits constructive feedback to attempters, which means it publishes a gradient toward its own shortcuts. Goodharting risk is not hypothetical: attempters are paid on pass rates. The defense (rotating probes, held-out seeded items, feedback that explains without revealing decision boundaries) has to be designed in from the start. GATES: Whether the feedback channel exposes full judge reasoning or a filtered version, and whether the foundation mandates ongoing seeded-item probes to detect gaming. ### Is the judge's constructive feedback held to the same grounding standard as the attempter work it critiques - i.e., what fraction of judge-generated feedback statements are themselves hallucinated or ungrounded, and do we verify feedback before delivering it? WHY: A judge confidently telling an attempter to fix a non-existent problem destroys trust faster than a wrong pass/fail verdict, and hallucinated critique is a known failure mode of critic models. If feedback needs its own verification pass, that roughly doubles the pipeline and changes cost and latency assumptions. GATES: Whether the architecture includes a feedback-verification stage (judge-of-judge or evidence-linking requirement on every feedback claim) before feedback reaches humans. ### Does a multi-model ensemble actually buy error reduction on the failure modes that matter here, or do frontier judges share correlated blind spots on subjective criteria such that the ensemble converges confidently on the same wrong answer? WHY: Ensembling is the default reliability move, but it only helps against decorrelated errors. If disagreement between model families predicts item difficulty better than it predicts judge error, the right design uses cross-family disagreement as an escalation signal, not a voting mechanism - a very different architecture at very different cost. GATES: Ensemble-vs-single-judge-plus-router architecture, and the per-item inference budget. ### Contrarian: on the subjective criteria where human reviewers disagree most, is 'judge bias' even distinguishable from legitimate criterion ambiguity - and would the honest system output 'this criterion is underspecified, here are the two defensible readings' back to the project owner rather than a verdict? WHY: Our lens assumes judge failure modes are the problem to fix, but root cause #1 is subjective interpretation - if the ground truth is genuinely contested, a perfectly debiased judge still can't be 'right,' only consistently arbitrary. Consistent-but-arbitrary may actually be acceptable (it fixes variance) - or it may launder ambiguity into false authority. This choice defines what the product is. GATES: Whether the foundation includes an instruction-ambiguity feedback loop to project owners as a first-class output, and whether the target metric is agreement-with-truth or consistency-of-application. ### What is the judge's blind-spot profile on expertise-heavy content - specifically, at what domain-difficulty level does the judge's ability to verify attempter claims fall below the attempter's own reliability, making AutoQA on those items net-negative? WHY: Root cause #2 is expertise gaps, and the judge has its own. A judge that confidently fails correct expert annotations (or passes confident expert-sounding errors) is worse than no QA on that slice. Knowing where the competence boundary sits per domain determines the abstention policy. GATES: Whether the foundation requires per-project judge-competence calibration (seeded expert items) before enabling autonomous verdicts, and the abstain-vs-verdict boundary. ### Do pass/fail verdicts and diagnostic reasons drift when the underlying judge model is upgraded, and does the foundation pin model versions per project with a regression suite, or accept drift as the price of improvement? WHY: The system's whole pitch is reducing unexplainable variance across reviewers. A silent model upgrade that shifts pass rates 10 points reintroduces exactly that variance, now across time instead of across people, and invalidates attempter feedback histories. GATES: Versioning and regression-testing policy in the foundation: frozen judge versions + golden-set gates on upgrade, versus continuous latest-model usage. ## Lens: rubric-ops ### At what decomposition granularity do rubric items stop improving judge-human-expert agreement - and does over-decomposition actively degrade it by destroying holistic judgments (e.g., 'this critique is fair overall') that only exist at the composite level? WHY: There is a real possibility that agreement vs. granularity is an inverted-U: too coarse reproduces the subjective-interpretation problem, too fine produces checklists that pass while the annotation is globally wrong. Without knowing where the curve peaks per criterion type, we can't set a decomposition policy. GATES: The rubric compiler's target granularity (atomic checkable claims vs. mid-level judgment clusters) and whether the schema allows/requires holistic roll-up items alongside atomic ones. ### Which evaluation-axis types are compilable into evidence-linked binary claims at all, and which are irreducibly gradient/contested - and can we build a reliable classifier (human or model) that sorts a project's axes into 'compile', 'anchor-with-exemplars', and 'route-to-human' buckets before the project goes live? WHY: The whole architecture forks on this taxonomy: compilable axes get automated verdicts, exemplar-anchored axes get calibrated model judgment, contested axes consume the one-human-touch budget. If we can't sort axes reliably up front, every project's failure mode is silent misrouting. GATES: The ingestion pipeline's axis-triage stage and the routing policy for the single allowed human interaction. ### What is the minimal set of artifacts a project instruction set must be compiled into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules - such that a fresh judge (model or human) reaches target agreement with the project owner without any conversation with them? WHY: This defines the 'rubric compilation contract.' If exemplars are load-bearing (likely, given that criteria text alone is what humans already disagree on), then a text-only compiler is doomed and ingestion must extract or elicit exemplars, which changes the whole intake workflow. GATES: The schema of the compiled AutoQA config and the required inputs at project onboarding (can we ingest instructions-as-written, or must we demand adjudicated examples?). ### Who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - and do we force the rubric to be amended (rubric-as-law) or allow standing owner overrides (rubric-as-guidance), given that override accumulation silently recreates the unexplainable-variance problem we're solving? WHY: Every rubric system dies one of two deaths: legalism (rubric ossifies, owners route around it) or override rot (rubric becomes fiction). The governance choice determines whether AutoQA verdicts are appealable, who can amend, and what an amendment invalidates retroactively. GATES: The amendment/override protocol, verdict finality semantics, and whether past verdicts are re-scored when a rubric changes. ### How do we detect criteria drift operationally - what specific signal (judge-human disagreement trending on one axis, override cluster density, exemplar staleness score, attempter-appeal rate) triggers a rubric revision, and at what threshold - versus drift in the item distribution or the attempter population masquerading as rubric drift? WHY: Without a falsifiable drift trigger, rubrics are either revised on vibes (reintroducing subjectivity) or never (ossification). Confusing population drift with criteria drift leads to 'fixing' a rubric that was fine, which churns all downstream calibration. GATES: Which telemetry the system must log per verdict from day one, and the design of the drift-detection monitor as a first-class component vs. a later add-on. ### For verifying an attempter's POSITIVE claims ('the response's argument in section 2 is sound'), what claim-typing does the compiler need - grounding claims (evidence exists), entailment claims (evidence supports the stated conclusion), and calibration claims (stated severity/rating matches evidence strength) - and do these three require different verification machinery with different reliability ceilings? WHY: Positive-statement verification is the stated differentiator, and 'is the opinion consistent with cited evidence' (calibration) is plausibly much harder than 'does the cited evidence exist' (grounding). If reliability ceilings differ sharply, verdict confidence must be reported per claim type, and the human touch should target the weakest type. GATES: The claim ontology in the compiled rubric schema and per-claim-type confidence gating on the human-escalation router. ### What is the empirical false-agreement rate - cases where the AutoQA judge and the human attempter agree with each other but both are wrong per expert ground truth - and does rubric decomposition reduce it or merely make shared errors better-documented? WHY: The seductive failure mode of rubric-ops is measuring judge-attempter agreement as success. If both parties share a plausible misreading of a criterion, decomposed rubrics can score high agreement while shipping bad data. This determines whether we need an independent expert-audit sampling loop as a permanent fixture rather than a bootstrap phase. GATES: Whether a standing gold-set / expert-audit sampling channel is a required foundational component (consuming budget) or an optional per-project add-on. ### For genuinely contested judgment calls, do we design for verdict pluralism (record 'defensible under reading A, not B' and pass with annotation) or forced resolution (every item gets pass/fail) - and can the system reliably distinguish 'contested by the criterion's nature' from 'contested because the rubric is underspecified'? WHY: Forced resolution on genuinely contested items manufactures the exact reviewer-variance noise we're eliminating, just with a machine's false consistency. But pluralism without the underspecification distinction becomes an excuse bucket that swallows fixable rubric gaps. GATES: The verdict schema (binary vs. verdict-plus-reading), the feedback template for contested items, and whether contested-item volume feeds the rubric-revision queue. ### Where does the one human interaction per item buy the most rubric-ops value: pre-verdict adjudication of the single most-contested claim, post-verdict confirmation of borderline fails, or asynchronous adjudication of exemplar-generating edge cases that improve the rubric for all future items? WHY: The third option means the human budget compounds (each touch improves the compiled rubric) rather than being spent per-item. But it also means some items ship with unconfirmed verdicts. This is a fundamental resource-allocation choice that shapes escalation design. GATES: The escalation router's objective function: per-item verdict accuracy vs. rubric-improvement rate, and whether human touches write back into the compiled config. ### Can the rubric compiler itself be trusted to be project-agnostic - i.e., does one compilation prompt/procedure hit target fidelity across instruction-set styles (terse axis lists, 40-page guideline docs, exemplar-heavy specs), or does compilation fidelity vary so much by input style that we need per-style compilation strategies and a human sign-off gate on every compiled rubric? WHY: A miscompiled rubric poisons every downstream verdict on that project, and the errors are systematic, not random - worse than no AutoQA. If compilation needs human sign-off anyway, the 'foundational philosophy + per-project customization' story changes from automated ingestion to a structured authoring workflow with tooling. GATES: Whether ingestion is an automated pipeline with spot checks or a human-led compilation workshop with model assistance, and what the compiled-rubric acceptance test is. ### What is the compiled rubric's unit of address - the attempter's deliverable as a whole, each discrete claim within it, or each (claim, criterion) pair - and how do we aggregate claim-level verdicts into the item-level pass/fail plus feedback without an aggregation rule that itself becomes a contested subjective judgment? WHY: Aggregation is where subjectivity re-enters through the back door: 'three minor grounding lapses vs. one severe misalignment' is exactly the kind of weighing on which human reviewers currently diverge. If aggregation rules can't be specified per project, item-level verdicts inherit the variance we tried to remove. GATES: Whether severity/weight taxonomies are mandatory fields in every compiled rubric, and whether the system emits item-level verdicts at all vs. claim-level findings plus a recommended verdict. ### Contrarian: is reviewer disagreement actually caused by criteria vagueness - or would perfectly operationalized rubrics leave most of the variance intact because it stems from expertise gaps and attention (root causes 2 and 3), meaning rubric-ops investment has a hard ceiling we should measure before building the compiler? WHY: The ranked root-cause list is a hypothesis, not a measurement. If a cheap experiment (give current human reviewers a hand-operationalized rubric for one project and measure variance reduction) shows small gains, the foundation should weight evidence-verification and expertise-substitution over rubric decomposition - a different system. GATES: The build order and budget split between the rubric-compilation subsystem and the evidence-verification/feedback subsystems; whether a variance-attribution pilot is a mandatory phase-zero. ### What does constructive feedback require from the compiled rubric beyond verdict machinery - do rubric items need paired 'what good looks like' statements and repair templates authored at compile time, or can feedback be generated at verdict time from the claim-evidence trace alone without drifting into advice the project owner would disown? WHY: Feedback generated freestyle at verdict time can contradict project doctrine (recommending fixes the project forbids), which destroys attempter trust faster than a wrong verdict. If feedback needs compile-time authoring, the rubric schema roughly doubles and so does ingestion cost. GATES: Whether the compiled-rubric schema includes mandatory feedback-side fields (exemplar repairs, approved phrasings, doctrine constraints) or feedback is a downstream generation layer over verdict traces. ## Lens: human-factors ### Does the constructive feedback we return to attempters leak the judge's decision boundary - i.e., after N feedback cycles, do attempter scores rise on the AutoQA while staying flat or falling on held-out human-graded golden items? WHY: Constructive feedback is a stated design goal, but it is also the highest-bandwidth channel for Goodharting: every explanation of why an item failed is a gradient the attempter can descend without improving true quality. If the divergence (judge score up, golden score flat) appears quickly, feedback must be abstracted (principle-level, delayed, or sampled) rather than item-specific. GATES: Feedback granularity and timing: item-specific rationales vs aggregated/principle-level feedback, and whether feedback text is generated from the same rubric the verdict uses or from a decoupled surface. ### Can the judge distinguish genuine evidence-grounding from 'citation theater' - specifically, if we perturb or swap the evidence an attempter cites, does the AutoQA verdict flip at a rate that shows it is actually checking the citation-claim link rather than rewarding grounded-sounding prose? WHY: The core positive-evaluation requirement ('is this statement actually grounded?') is exactly the surface attempters will learn to imitate: hedged, citation-dense, judge-pleasing prose. A perturbation test is the falsifiable check; if verdicts don't flip when cited evidence is invalidated, the grounding check is decorative and will be gamed within weeks of deployment. GATES: Whether grounding verification is implemented as semantic holistic judgment or as an explicit claim-evidence entailment step with adversarial perturbation testing in the eval harness. ### For wholesale LLM-generated submissions, is our defense content-based detection or economics-based throttling - i.e., what is the measured cost ratio between an attempter generating a passing submission with a frontier model and us detecting it, and at what ratio do we abandon detection for identity/throughput/behavioral gates? WHY: By mid-2026, frontier-model submissions can be indistinguishable from strong human work on content alone; a detection arms race we lose by construction wastes the QA budget. If the cost asymmetry favors the spammer, the architecture must shift defenses to rate limits, work-history consistency, provenance signals, and randomized deep audits rather than per-item classification. GATES: Whether the AutoQA pipeline includes an AI-detection module at all, versus spending that complexity budget on behavioral/longitudinal attempter modeling. ### Do we actually care about LLM assistance provenance, or only content validity - is a correct, well-grounded annotation produced with heavy LLM help a pass, a fail, or a policy question that must be settled per-project before the foundation is designed? WHY: This is unresolved in most annotation pipelines and it changes the objective function of the entire QA layer. If content-only, spam filtering collapses into quality filtering (simpler system). If provenance matters (e.g., because training data must reflect human judgment, not model self-distillation), we need signals the content itself cannot provide - process telemetry, timing, edit traces - which is a different data-collection contract with attempters. GATES: Whether the foundation's spam/falsehood axis is defined over content properties alone or requires process/provenance instrumentation as a first-class input. ### In what order does the single human interaction see the AI's work - and at what verdict-presentation format does the human overturn rate fall below the AutoQA's measured error rate (the signature of rubber-stamping)? WHY: Automation bias is the failure mode that silently converts 'human-in-the-loop' into 'human-as-liability-shield.' Whether the human judges blind-first then reconciles, sees the AI verdict with rationale, or sees rationale without verdict, measurably changes overturn behavior. If overturn rate < AI error rate, the human is adding negative information and the interaction budget is being wasted. GATES: The core review UX: blind-first independent judgment vs AI-verdict-first confirmation, and whether AI confidence/rationale is shown before or after the human commits. ### Where does the one-human-interaction budget buy the most reliability per item: verifying AI-flagged failures, adjudicating low-confidence/disputed items, or spot-auditing AI passes - and is the answer stable across the AutoQA's confidence calibration or does it invert when calibration drifts? WHY: With exactly one human touch, routing IS the system. Spending it on confirmed-looking failures duplicates the AI; spending it on confident passes catches judge blind spots but at low hit rate. The optimal allocation depends on the judge's calibration curve, which will drift as attempters adapt - so the routing policy must be designed as a function of measured calibration, not fixed at launch. GATES: The triage/routing architecture: which confidence bands and verdict types route to the human, and whether routing thresholds are static or closed-loop on observed overturn rates. ### When the judge's domain competence is below the attempter's (frontier expertise, niche domains), can the AutoQA detect its own incompetence and abstain - and at what abstention rate does the system stop net-saving human effort for that project? WHY: Expertise mismatch cuts both ways: the AutoQA confidently 'correcting' an expert attempter is worse than no QA, because it burns attempter trust and pollutes the record with false fails. A judge that cannot self-diagnose incompetence forces either per-domain capability gating or per-project human fallback, which changes the project-agnostic claim of the foundation. GATES: Whether the foundation includes a mandatory abstain/escalate verdict class with per-project competence calibration, and whether some projects are declared AutoQA-ineligible. ### Does the AutoQA prefilter shift the human reviewer's base rates enough to cause criterion drift - when the queue reaching the human is enriched with hard/adversarial cases, do reviewers become systematically harsher or more vigilance-fatigued than on an unfiltered queue, and does seeding known-verdict items into the queue correct it? WHY: Filtering changes the population the human sees, which changes the human. A reviewer seeing 80% genuinely-borderline items recalibrates their pass threshold and fatigues faster; their disagreement with the AI then looks like AI error when it's induced drift. Without seeded gold items you cannot even measure this, because there is no fixed reference in the post-filter queue. GATES: Whether the queue design mandates seeded known-verdict items and easy-case dilution, and how reviewer accuracy is measured post-filter. ### If AutoQA verdicts affect attempter pay or tenure, what appeal mechanism absorbs disputes without either re-importing full human review load or becoming a rubber stamp - concretely, does an appeal route to a human, a stronger/differently-prompted AI pass, or evidence-required re-adjudication, and what appeal-success rate signals a healthy vs broken system? WHY: The moment verdicts carry economic consequence, the system becomes adversarial and appeals are the trust release-valve. An appeal path that costs a full human review breaks the one-interaction budget at scale; one that never succeeds destroys attempter trust and drives silent gaming instead of open dispute. The target appeal-success band (too low = judge unaccountable, too high = judge unreliable) must be chosen before launch because it defines the verdict's enforcement weight. GATES: Whether AutoQA verdicts are advisory or enforcement-grade, and the architecture of the appeal tier (human, escalated AI, or evidence-gated). ### When the single human reviewer and the AutoQA persistently disagree, what arbitrates - given that noisy, misaligned human reviewers are the original problem, does escalation to another human just re-import the variance we built the system to remove? WHY: The system's stated origin is human reviewer misalignment, yet most HITL designs quietly treat the human as ground truth for disagreements. If human arbitration is itself ~60% reliable, disagreement resolution needs a different substrate: golden exemplars, criterion-author rulings, or multi-judge consensus with documented rationale. Choosing wrong means the AutoQA slowly trains toward the noise it was meant to filter. GATES: The ground-truth hierarchy of the foundation: what outranks what (golden set > criterion author > human reviewer > AutoQA?), and how disagreements feed back into judge calibration. ### Is the observed reviewer variance actually a human-factors problem at all - if we run two independently-configured AutoQA instances on the same items, do they disagree at a rate comparable to the human reviewers, indicating the variance lives in criterion ambiguity rather than in humans? WHY: This is the contrarian check on the whole lens: if the criteria themselves are underdetermined, no amount of incentive design, bias mitigation, or fatigue management fixes pass/fail variance - and the AutoQA will simply be misaligned with itself in a way that looks authoritative. The falsifiable test (judge-judge disagreement vs human-human disagreement per criterion) tells us whether budget goes to per-project rubric disambiguation/decision-rule authoring or to the QA layer itself. GATES: Whether the foundation's first-class artifact is the judge, or a per-project criterion-operationalization step (decision rules, boundary exemplars) that both judge and humans consume. ### Is the one-human-interaction-per-item goal itself the right target - would zero per-item human touch plus randomized deep audits and drift monitoring outperform one shallow, automation-biased touch per item on total error caught per human-hour? WHY: One guaranteed shallow interaction per item invites rubber-stamping and gives a false sense of human oversight, while consuming the entire human budget on breadth. Concentrating the same hours into deep audits of a random sample yields unbiased error estimates, resists automation bias (auditors work blind), and creates real deterrence for gaming - but sacrifices per-item human accountability. This is a fork in the foundation, not a tuning knob. GATES: The fundamental human-budget allocation model: per-item review vs sampled deep audit vs confidence-routed hybrid, which determines the entire pipeline shape. ### What keeps the human reviewers calibrated and engaged once the AutoQA writes the verdicts and the constructive feedback - measurably, does reviewer skill and overturn quality decay over months of reconcile-only work, and if so, who remains competent to calibrate the judge? WHY: The judge needs an ongoing supply of high-quality human judgments to detect its own drift, but the system as designed converts reviewers from judgment-producers into verdict-confirmers - the exact role that atrophies judgment. If reviewer skill decays, the calibration signal degrades precisely as attempter gaming pressure grows, a compounding failure. The design may need deliberate blind-judgment quotas purely to preserve the calibration workforce. GATES: Whether the foundation mandates a standing blind-review stream (reviewers judging without AI output) as calibration infrastructure, and how reviewer performance itself is QA'd. ## Lens: grounding ### At what claim-decomposition granularity does judge-human agreement on grounding verdicts stop improving - holistic review-level, per-rubric-axis, per-sentence, or atomic claim - and what is the cost curve per level? WHY: Decomposition is the single biggest architectural choice: atomic claims make entailment checks tractable and auditable but multiply judge calls and can strip context that changes a claim's meaning; holistic judging is cheap but unfalsifiable. Without knowing where agreement plateaus, we cannot size the pipeline or the per-item cost. GATES: Whether the pipeline's first stage is a claim extractor feeding N entailment checks, or a single structured holistic judge; also the shape of the verdict data model. ### Is the evidence universe closed (AI output under review + task instructions + attempter-cited spans) or open (world knowledge, external references) - and what verdict do we assign to attempter claims that are true-in-the-world but unsupported in the closed corpus? WHY: A closed universe makes verification a well-defined entailment problem; an open universe requires retrieval and inherits the judge's own hallucination risk. Many attempter claims (e.g. 'this API call is deprecated') are world-knowledge claims with no in-corpus evidence, and how we score them determines the false-flag rate. GATES: Whether the foundation includes a retrieval/grounding-corpus component per project, and the definition of the 'unsupported' verdict class. ### What entailment standard does an attempter statement have to meet - strict logical entailment from cited evidence, or 'reasonable-expert support' - and is the output a 3-way verdict (entailed / contradicted / unsupported) or a graded support score with thresholds? WHY: Strict entailment will fail nearly all human-written prose (humans compress, paraphrase, and generalize); loose support standards reintroduce exactly the subjective variance we're trying to remove. The standard chosen defines what 'grounded' means for the whole system and must be written into every per-project customization. GATES: The verdict taxonomy, the calibration target for the judge, and the wording of the grounding criterion given to projects. ### Can a judge distinguish warranted praise from empty praise at usable reliability - concretely, on a seeded set where half the praised properties are genuinely present and half are absent or fabricated, what precision/recall does the judge hit per praise type ('accurate', 'complete', 'clear', 'idiomatic')? WHY: Positive-claim verification is the stated differentiator of this system, but praise types differ radically in checkability: 'the code compiles' is verifiable, 'the explanation is clear' is barely falsifiable. If the judge only verifies the checkable subset, we need to know which subset and say so, rather than pretend all praise is verified. GATES: Whether positive-claim verification ships as a hard pass/fail gate, a flag-only signal, or is restricted to an enumerated set of checkable praise types. ### Does verifying a positive claim like 'no errors present' or 'fully addresses the prompt' require the judge to independently re-attempt the task (exhaustive defect search), and if so, is the judge's own miss rate on seeded defects low enough that its 'confirmed clean' verdict adds information over the attempter's? WHY: Universal/negative-existential claims can't be verified by citation checking - they require proving absence. If the judge's defect-finding recall is not materially above the attempter population's, 'AutoQA confirmed the pass' is theater, and pass-verdicts should be treated as unverifiable rather than verified. GATES: Whether the architecture includes an independent re-attempt/defect-search stage, and how pass/clean attempter verdicts are labeled in output (verified vs. not-contradicted). ### Is omission detection tractable project-agnostically - i.e., can a judge generate the reference set of 'what should have been said' from instructions + artifact alone at usable recall, or does credible omission detection require a per-project gold defect taxonomy, making it customization work rather than foundation work? WHY: Detecting what's missing requires a completeness reference that doesn't exist in the attempter's text. If that reference can only come from project-specific defect enumeration, then omission detection belongs in the per-project layer and the foundation should only define its interface - a very different build than a universal omission detector. GATES: Whether omission detection is a core foundation module or a per-project plugin with a defined contract; also whether project onboarding must include building a defect taxonomy. ### Where is the line between a verifiable claim and a legitimate subjective judgment call, and what does the system do with claims classified as opinion - check only internal consistency with the evidence the attempter cites, or decline to score them at all? WHY: Root cause #1 is subjective criterion interpretation; if the grounding layer tries to fact-check genuine judgment calls it will manufacture false disagreements, and if it exempts too much as 'opinion' it verifies nothing. The claim-type classifier's boundary IS the system's scope, and misrouting here is the main source of attempter-facing injustice. GATES: Whether the pipeline needs a claim-type classification stage, and the verdict semantics for opinion-class claims (consistency-checked vs. unscored). ### When the attempter's cited evidence supports their conclusion but uncited evidence elsewhere in the artifact undermines it, do we verify against cited-evidence-only or the full artifact - and are cherry-picking failures scored as grounding failures or omissions? WHY: Cited-only verification is cheap and precise but blesses cherry-picking, which is exactly the failure mode of a lazy-but-plausible attempter. Full-artifact verification catches it but is an order of magnitude more work per claim and blurs into omission detection. This choice determines what the entailment checker actually reads. GATES: The evidence scope of the entailment checker and the taxonomy boundary between 'ungrounded' and 'incomplete' verdicts. ### Should the foundation mandate an upstream instruction change requiring attempters to attach explicit evidence pointers (quotes, line refs) to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding and other gaming responses? WHY: Requiring citations could be the highest-leverage move in the whole design: it turns the hardest problem (locating evidence) into the attempter's job and makes verification nearly mechanical. But it changes the attempter product, adds friction, and invites decorative citations that pass surface checks - which would corrupt the very signal we depend on. GATES: Whether the AutoQA spec includes required changes to attempter-facing instructions, or must work with free-form uncited reviews as given. ### What is the empirical base-rate split between attempter claims that are actually false versus merely uncited-but-true, and at what judge precision does flagging 'unsupported' claims for the one allowed human interaction beat spending that interaction elsewhere? WHY: If most unsupported claims turn out true, aggressive flagging burns the human-interaction budget on exoneration and erodes attempter trust; if falsehoods are common, permissive thresholds let spam through. The threshold cannot be set without the base rate, and the base rate differs from the error-only-QA world because we now audit positive statements too. GATES: Verdict thresholds, whether 'unsupported' auto-fails or only flags, and the triage function that allocates the single human interaction per item. ### Is judge-judge agreement on grounding verdicts (same item, resampled/paraphrased prompts, sibling models) actually higher than the human reviewer-reviewer agreement we're trying to fix - measured per verdict class - and what consistency mechanism (self-consistency voting, fixed rubric anchors, deterministic decomposition) closes the gap if not? WHY: The system's entire justification is reducing unexplained reviewer variance. An LLM judge has its own stochastic variance, and if its verdict flip rate on borderline grounding calls matches human disagreement rates, we've automated the problem rather than solved it. This must be a falsifiable pre-ship gate, not an assumption. GATES: Whether verdicts require ensembling/multi-sample agreement (with its cost multiplier), and the go/no-go criterion for deploying the judge at all. ### Do we apply the grounding standard reflexively to AutoQA's own feedback - every criticism it sends an attempter must itself cite entailing evidence and pass the same entailment check - and what false-accusation rate is low enough that attempters keep trusting and acting on the feedback? WHY: Constructive feedback is a stated goal, and an ungrounded accusation from the QA system is more corrosive than an ungrounded claim from an attempter - it trains attempters to dispute or ignore the system. Reflexive self-verification roughly doubles judge cost, so it must be a deliberate decision with a trust-threshold target, not an afterthought. GATES: Whether the feedback generator gets its own verification pass before emission, and the precision SLO for attempter-facing criticisms. ### Contrarian: if the judge can reliably verify whether an attempter's statements are grounded - including re-finding defects to check 'clean' verdicts - does grounding verification actually require less capability than doing the annotation itself, and if not, what evidence supports keeping mandatory human attempters upstream of a judge that could replace them, rather than inverting the design (model attempts, human grounds)? WHY: The lens presumes verification is easier than generation, which is true for closed-form checks but unproven for omission detection and absence-proving, where the judge must out-attempt the attempter. If verification is capability-equivalent to attempting, the honest design conclusion is different: either scope AutoQA down to the genuinely-easier-to-verify subset, or acknowledge the human's value is legal/accountability cover and design the QA layer around that instead. GATES: The fundamental scope of the grounding module - full verification vs. only the verification-easier-than-generation subset - and how the system's value proposition is framed to stakeholders. ## Lens: hybrid-workflow ### What exactly is the single human interaction spent on: blind re-judgment of the item, verification of the AI's verdict+rationale, or adjudication of specific AI-flagged claims - and which of these maximizes error reduction per minute of human time? WHY: These three interaction types produce different label independence, different cost per item, and different anchoring risk. The whole workflow, UI, and downstream statistics change depending on which one we pick. GATES: The core unit of work in the human queue and the data schema of what a human touch produces. ### Which measurable confidence proxy from the LLM judge (self-consistency across k samples, verdict flip-rate under paraphrase, ensemble disagreement, rationale-evidence entailment score) actually predicts human overturn with AUROC high enough to route on - and what is that threshold? WHY: Confidence-based routing is worthless if the confidence signal is uncorrelated with actual judge error. This is empirically falsifiable per project and determines whether routing is principled or theater. GATES: Whether we build routing on judge-internal signals at all, and which signal the router consumes. ### Should the escalation threshold be set to a fixed human-capacity budget (top-N most uncertain items) or to a target residual error rate (escalate until expected post-QA error < X%), and which quantity is actually the binding constraint? WHY: Budget-first routing gives predictable staffing but uncontrolled quality; error-first routing gives quality guarantees but unpredictable queue depth. You cannot optimize both; picking one dictates the router's objective function. GATES: The router's optimization target and how project owners configure it. ### What is the asymmetric evidence bar for standalone AutoQA verdicts: what measured false-FAIL rate (on adjudicated gold) is acceptable before the system may fail an attempter with zero human touch, and is auto-PASS held to a different bar than auto-FAIL? WHY: Failing a human attempter has pay/reputation consequences; a wrong pass just leaks noise into training data. These harms are asymmetric, so a single trust threshold is almost certainly wrong. GATES: Whether the autonomous lane emits both verdict polarities or only one, and the gold-set size needed to certify it. ### Contrarian: given the stated root problem is that human reviewers disagree with each other for unexplainable reasons, does escalating to one human reduce error at all - or does it re-inject the exact variance we are removing while laundering AI verdicts with a human signature? What overturn-quality audit would prove the human touch adds signal? WHY: The lens presumes human escalation improves outcomes. If a second human agrees with the escalation human no more often than with the AI, the hybrid layer is cost without benefit and the design should shift toward better autonomous calibration instead. GATES: Whether per-item human escalation exists in v1 at all, versus humans only in calibration/gold roles. ### Should the scarce human interactions be spent on per-item adjudication at all, versus spent upstream on building per-criterion calibration sets and disambiguating rubric language - and at what point does one hour of rubric disambiguation beat one hour of item review in agreement gains? WHY: Root cause #1 is subjective interpretation of criteria. If interpretation variance dominates, per-item escalation treats symptoms while the same human minutes spent codifying criterion boundary cases would raise autonomous accuracy across all items. GATES: The allocation policy for the human-interaction budget: adjudication queue vs. calibration/rubric-refinement queue. ### Does the escalated human judge blind (no AI rationale shown) or anchored (AI verdict and cited evidence shown) - and is the measured anchoring cost smaller than the independence value of blind labels for calibrating the judge? WHY: Anchored review is faster and catches AI reasoning errors directly, but anchored labels cannot be used to measure judge accuracy or drift because they are contaminated. This is a one-way door for the statistics. GATES: Whether escalation labels double as calibration data, or a separate blind-label stream must be funded. ### At what granularity does escalation route: the whole item, or the specific contested claim/criterion the judge is uncertain about - and can claim-level adjudication (answer 2-3 pointed micro-questions) stay within the one-interaction budget while whole-item review cannot? WHY: The system evaluates positive statements claim-by-claim (groundedness, criterion-alignment, evidence-consistency). Judge uncertainty is usually localized; sending the whole item wastes the human on parts the AI already resolved confidently. GATES: Queue item schema and the escalation UI contract (full item vs. targeted micro-adjudications). ### Should escalation queues be organized by criterion (specialist reviewers per evaluation axis) or by item (generalist review), and does criterion-specialist routing measurably close the expertise-gap root cause where generalist routing does not? WHY: Root cause #2 is expertise gaps. A reviewer strong on factual grounding may be weak on style criteria. Specialist queues change reviewer staffing, assignment logic, and how verdicts are reassembled from partial adjudications. GATES: Queue partitioning, reviewer assignment model, and verdict-assembly logic. ### What statistical model combines sparse human labels with dense AI judgments - per-reviewer reliability weighting (Dawid-Skene family), per-project Bayesian calibration of judge scores, or simple hierarchical override - and how many human labels per project-criterion cell does the chosen model need before its posteriors beat the raw judge? WHY: Below the convergence threshold, the fusion model is worse than trusting the judge alone; above it, it corrects systematic judge bias per criterion. The answer sets the mandatory human-label spend during ramp-up and whether reviewer identity must be tracked. GATES: The verdict-fusion architecture and the minimum viable human-label collection plan per project. ### What is the cold-start protocol for a new project with zero human labels - a 100%-escalation burn-in window, seeded gold items authored during project setup, or transferred priors from similar projects - and what quantitative graduation criterion (e.g., judge-human kappa over the last N adjudications) flips the project into confidence-routed mode? WHY: Since the system is project-agnostic and customized per instruction set, every project starts uncalibrated. Without an explicit graduation gate, projects either stay expensively human-heavy forever or go autonomous on faith. GATES: Project onboarding lifecycle, burn-in cost model, and the automated mode-transition rule. ### Does constructive feedback delivered to attempters need a different confidence bar than the pass/fail verdict - i.e., can the system ship AI-generated coaching on items whose verdict was auto-passed, and what is the harm model when specific feedback claims (e.g., 'your citation does not support this') are themselves wrong? WHY: Feedback contains checkable factual claims about the attempter's work; a wrong specific accusation damages trust more than a bare wrong verdict. If feedback needs human review, the one-interaction budget gets split between adjudication and feedback vetting. GATES: Whether feedback generation is in the autonomous lane, the escalated lane, or gated by its own groundedness check. ### Once a project runs in trusted-standalone mode, what ongoing randomized audit rate and drift statistic (e.g., CUSUM on judge-vs-audit disagreement per criterion) triggers automatic de-graduation back to heavier escalation - and does audit sampling need to be unpredictable enough that attempters cannot learn to write for the judge? WHY: An autonomous lane fails silently: judge drift, instruction changes, and attempter adaptation to the judge's known preferences all degrade quality without any surfaced signal. Confidence-routing itself is a gameable surface if deterministic. GATES: The monitoring subsystem, audit sampling design (including randomization), and the automatic trust-revocation rule. ## Lens: product-feedback ### At what verdict granularity (binary pass/fail, per-axis scores, or claim-level grounded/ungrounded judgments on each attempter statement) does judge-human agreement stop improving, and does claim-level verification measurably reduce the unexplained reviewer variance we're trying to fix? WHY: The whole premise is that humans disagree at the holistic-verdict level. If decomposing to claim-level checks doesn't recover agreement, the AutoQA inherits the same subjectivity problem and we've just automated the noise. If it does, the entire data model must be claim-first with verdicts as derived aggregates. GATES: The core output schema: whether verdicts are primitive objects or computed rollups of atomic claim judgments, and whether the feedback surface is organized around claims or around holistic scores. ### For positive-statement verification ('is this attempter claim grounded in the cited evidence?'), what is the falsifiable acceptance bar - e.g., does the judge's grounded/ungrounded label match an expert adjudicator at >=X% on a golden set - and what X makes the system trustworthy enough to auto-pass items without human review? WHY: Verifying positive claims is harder than spotting errors; without a measured precision bar we can't decide which items get the one allowed human interaction versus none. This number determines the human-in-the-loop routing policy and the escalation threshold. GATES: The triage policy: which confidence band auto-passes, which auto-fails, and which consumes the single human interaction. ### Should the single permitted human interaction be spent on adjudicating the judge's uncertain verdicts, or on the attempter responding to feedback (contest/revise), and does that choice differ by root cause (subjectivity vs expertise gap vs attention lapse)? WHY: One human touch per item is a hard budget. Spending it on verdict adjudication improves label quality; spending it on attempter response improves future behavior and catches judge errors the attempter can see. These are different products with different data flows. GATES: Where the human sits in the pipeline (before verdict finalization vs after feedback delivery) and what UI/workflow that interaction needs per project. ### What is the minimal structure of a feedback unit that attempters actually act on - e.g., (quoted attempter claim, specific evidence span, criterion clause violated, concrete fix) - and can we falsify 'more detail is better' by measuring revision quality against feedback length and structure? WHY: Constructive feedback is a stated goal, but unstructured LLM feedback tends toward verbose, hedged prose that attempters don't read closely. If a rigid 4-tuple outperforms free text on revision quality, feedback generation becomes a constrained extraction task, which is cheaper, more verifiable, and project-portable. GATES: The feedback schema and generation approach: templated evidence-linked tuples vs free-form critique, and how feedback quality itself gets QA'd. ### Must every piece of feedback be traceable to a specific clause in the project instruction set (criterion-linked), and what happens to judge observations that are real problems but map to no written criterion - surface them, suppress them, or queue them as instruction-gap reports to the project owner? WHY: Root cause #1 is subjective interpretation of criteria. If the judge can cite un-written standards, it amplifies the subjectivity problem and attempters can't contest it. But suppressing real issues loses signal. The gap-report path turns judge disagreement into instruction improvement, attacking the root cause rather than the symptom. GATES: Whether the feedback generator is hard-constrained to cite instruction clauses, and whether an instruction-gap feedback channel to project owners is a first-class output of the system. ### How do we distinguish 'attempter is wrong' from 'criterion is ambiguous' in the verdict itself - should the AutoQA emit a third outcome class (ambiguous-under-instructions) with measured inter-judge disagreement as its trigger, rather than forcing a pass/fail? WHY: Forcing binary verdicts on genuinely ambiguous items is exactly what makes human reviewers look misaligned. If ensemble judge self-disagreement reliably predicts human reviewer disagreement, ambiguity becomes detectable and routable instead of masquerading as attempter failure. GATES: The verdict taxonomy (2-class vs 3-class), the ensemble/self-consistency architecture needed to detect ambiguity, and whether ambiguous items are exempt from attempter penalty. ### What is the explainability contract for a failing verdict: does every fail require verbatim evidence spans and a reproducible reasoning chain an attempter could independently check, and do we reject (not deliver) judge verdicts that can't meet that bar even when the verdict is likely correct? WHY: A correct-but-unexplainable fail destroys attempter trust and is uncontestable, recreating the opaque-human-reviewer problem. Making explainability a delivery gate trades some recall for legitimacy; whether that trade is acceptable changes the judge's output requirements fundamentally. GATES: Whether explainability is a hard output constraint (fail-closed) or best-effort annotation, and the verification step that checks feedback self-consistency before delivery. ### Does delivered feedback measurably change future attempter behavior - what longitudinal metric (repeat-error rate on the same criterion per attempter) do we commit to, and after how many items of no improvement do we conclude feedback is decoration and the real product is filtering? WHY: This is the uncomfortable question for the feedback lens: the constructive-feedback goal presumes attempters read, understand, and apply it. Marketplace annotation workforces often optimize throughput, not learning. If repeat-error rates don't move, feedback generation is wasted spend and the design should shift budget to routing, filtering, and attempter selection instead. GATES: Whether feedback generation is a core deliverable with a learning-loop KPI, or a lightweight byproduct - this reallocates most of the product surface. ### Should attempters be able to contest AutoQA verdicts, and does a contest consume the one human interaction - i.e., is the appeals channel the human-in-the-loop, and what contest-overturn rate would indicate the judge (not the attempters) is the problem? WHY: A contest channel is both a trust mechanism and the cheapest source of judge-error labels. But it competes for the single human-touch budget and can be gamed by throughput-motivated attempters. The overturn-rate threshold gives a falsifiable tripwire for judge recalibration. GATES: Whether appeals exist, who adjudicates them within the one-interaction budget, and the monitoring metric that triggers per-project judge retuning. ### What exactly is the per-project customization surface - is it only (instruction set + criterion decomposition + calibration exemplars + severity weights), and can we prove the foundation is truly project-agnostic by onboarding a new project with zero prompt-engineering of the core judge? WHY: Project-agnostic is the stated constraint, but every per-project prompt hack that leaks into the core erodes it. Defining the customization surface as a closed configuration schema makes 'agnostic' falsifiable: a new project either onboards through config alone or the abstraction is wrong. GATES: The boundary between the frozen foundation and the per-project config schema, and the onboarding procedure (config authoring vs bespoke judge tuning). ### How many calibrated exemplars per criterion (expert-adjudicated pass/fail/ambiguous anchors) does a new project need before judge-human agreement plateaus, and who produces them - because if the answer is 'a large expert-labeled set,' the system quietly reintroduces the expensive aligned-human-reviewer dependency it was built to replace? WHY: Calibration exemplars are the standard fix for criterion subjectivity, but their cost scales with project count and criterion churn. Knowing the plateau point (5 vs 50 vs 500 per criterion) determines whether per-project onboarding is a day of an SME's time or an ongoing labeling program. GATES: The per-project onboarding cost model and whether calibration is a one-time setup artifact or a continuous pipeline with its own QA. ### Should the AutoQA verdict feed attempter-level consequences (ratings, pay, removal) or remain item-level coaching only - and if consequences attach, what precision floor and audit trail does each verdict need to be defensible against a wrongly-failed attempter? WHY: The moment verdicts affect livelihoods, error tolerance collapses and attempters rationally start gaming the judge (writing to please the model rather than to be correct). This changes required precision, explainability depth, contest rights, and how feedback language is phrased. GATES: Verdict consumption policy downstream (coaching-only vs consequential), which sets the precision/audit requirements for the entire judge pipeline. ### How do we prevent feedback-induced monoculture - if attempters learn the judge's stylistic preferences, do annotations converge on judge-pleasing patterns that reduce the diversity and independent judgment the training data exists to capture, and what drift metric would detect this? WHY: The feedback loop is the product's point, but its failure mode is Goodharting the annotation workforce: evidence-citation boilerplate, hedge phrasing, and criterion-parroting that passes QA while degrading the underlying data. A measurable diversity/drift signal decides whether feedback should ever include stylistic guidance at all. GATES: What categories of feedback are permitted (substance-only vs substance+style), and whether a population-level drift monitor is a required system component from v1. "}, {"slug": "all-domains", "title": "Combined Domain Reports", "category": "Research working papers", "summary": "All domain reports concatenated into one source atlas, with evidence summaries, tools, disagreements, gaps, and verification notes.", "tags": ["research", "all domains", "source atlas"], "view": "documents/view/all-domains.html", "raw": "documents/raw/research/all-domains.md", "text": "# AutoQA Research Atlas -- Combined Domain Reports Concatenation of the 10 per-domain source reports underlying 01_RESEARCH_ATLAS.md, in Source Index order. Each section below is one original file with its heading demoted one level; content is otherwise unmodified except for ASCII normalization. ## Contents - [Domain: academic-judges](#domain-academic-judges) -- judge meta-evaluation benchmarks and capability limits - [Domain: annotation-quality](#domain-annotation-quality) -- annotation science, disagreement, contamination - [Domain: critique-models](#domain-critique-models) -- CriticGPT lineage, human+critic teams, omission detection - [Domain: grounding-faithfulness](#domain-grounding-faithfulness) -- entailment checkers, decomposition, citation evaluation - [Domain: hybrid-statistical](#domain-hybrid-statistical) -- PPI, selective escalation, conformal routing, drift attribution - [Domain: rubrics-recent](#domain-rubrics-recent) -- rubric practice 2025-26, compilation, rubric QA - [Domain: industry-practice](#domain-industry-practice) -- vendor QA stacks, eval tooling, spam economics - [Domain: contrarian](#domain-contrarian) -- the case against, adversarial attacks, validity critiques - [Domain: frontier-2026](#domain-frontier-2026) -- 2026-only developments - [Completeness critic](#completeness-critic) -- critic's weak-spot audit + psychometrics, law/standards, content moderation, feedback science --- ## Domain: academic-judges *judge meta-evaluation benchmarks and capability limits* ### On hard, objectively-verifiable judging tasks (JudgeBench, ICLR 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while GPT-4o scored 50.9-56.6% (near random), the best reward model (Skywork-Reward-Gemma-2-27B) hit 64.3%, and fine-tuned judges like PandaLM fell at or below random. [LOAD-BEARING] EVIDENCE: JudgeBench uses 350 rigorously verified response pairs across knowledge/reasoning/math/coding where correctness is checkable. Verified numbers: GPT-4o vanilla 50.9% overall, Arena-Hard pipeline 56.6%, o1-preview 75.4% (85.7% math/coding), Skywork-Reward-27B 64.3%, reward models 60-64%, ChatEval multi-agent ~34%. Fine-tuned preference judges plateau near random on knowledge/reasoning. SOURCE: JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025) + EmergentMind topic synthesis (updated Jan 2026) | https://arxiv.org/abs/2410.12784 | 2024-10 (ICLR 2025; synthesis updated 2026-01) | academic IMPLICATION: The AutoQA core judge should be a frontier reasoning model, not an off-the-shelf fine-tuned judge or scalar reward model, because annotation-QA requires reasoning about correctness, not preference matching. Reasoning-model headroom (75% vs 51% for the same lab's non-reasoning model) is the single biggest capability lever. ### Standard LLM-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial LLMs, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy. [LOAD-BEARING] EVIDENCE: Guerdan, Barocas, Holstein, Wallach, Wu, Chouldechova (NeurIPS 2025 poster, abstract opened and confirmed). They formalize 'rating indeterminacy' - items where multiple ratings are reasonable - and show humans and LLMs resolve forced choices differently, so aggregated gold labels systematically mis-rank judges. They provide concrete recommendations for elicitation and aggregation. SOURCE: Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025) | https://neurips.cc/virtual/2025/poster/117308 | 2025-09-19 | academic IMPLICATION: Directly answers design Q1/Q2: the ground-truth construct cannot be a single forced pass/fail human label. The gold set must elicit response-set labels ('which verdicts are defensible?'), and the verdict ontology needs a first-class 'instructions underdetermine this case' class. Validation against collapsed consensus labels will select the wrong judge configuration. ### The largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including April-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected Cohen's kappa is 33-41 percentage points lower on MT-Bench - and judge rankings shift by up to 14 positions depending on which benchmark you use. [LOAD-BEARING] EVIDENCE: Norman, Rivera, Hughes (arXiv 2606.19544, abstract opened and confirmed). Four cohort-wide findings: universal kappa deflation; benchmark-dependent rankings (up to 14 position shifts across MT-Bench/JudgeBench/RewardBench); a 'consistency-bias paradox' where test-retest reliability >0.95 coexists with position bias >0.10 in two production judges; verbosity bias small (<0.011) under a single controlled pairwise rubric. They distill a Minimum Viable Validation Protocol. SOURCE: Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic IMPLICATION: Answers design Q1's metric question: ban raw percent-agreement and single-number accuracy from all AutoQA reporting; standardize on chance-corrected statistics per benchmark. Also answers Q5: a judge that is perfectly self-consistent can still be systematically biased - consistency monitoring and bias audits are separate mandatory sensors, and per-project validation must use multiple gold-set designs since rankings do not transfer. ### At the rubric level - the exact granularity AutoQA would operate at - even frontier judges achieve only ~55-56% balanced accuracy on hard rubric-verification instances (GPT-4o 55.97%, Claude-Sonnet-4.5 55.65%), but rubric-level evaluation with explicit chain-of-thought reasoning beats checklist-level evaluation by 7-12 percentage points and reduces cross-judge variance. [LOAD-BEARING] EVIDENCE: RubricEval (arXiv 2603.25133, full alphaXiv overview read). Binary satisfied/not-satisfied verification of atomic rubrics against responses; EASY subset ~90% accuracy, HARD subset ~55%. Compositional (conditional-logic) instructions, Role Persona, and Format Structure rubrics hardest. Their multi-model arbitration labeling pipeline (RAF) reached 85.0% agreement with humans, Cohen's kappa 0.702. Reasoning-intensive models (o3) notably outperform standard instruct models. SOURCE: RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following | https://www.alphaxiv.org/overview/2603.25133 | 2026-03-26 | academic IMPLICATION: Answers design Q3 granularity: decompose to atomic criteria and require per-criterion explicit reasoning (worth +7-12pp), but budget for the cost multiplier and expect a hard residual class (~45% error on contested rubrics) that must route to humans. Easy/hard stratification should be built in: judge consensus at the coarse pass identifies which claims need escalation - the RAF arbitration cascade is a directly reusable architecture for gold-label production. ### On subjective rubrics, LLM judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-to-human), judges use only 0.3-0.5x the human score spread, and inter-LLM agreement (r~0.35) exceeds LLM-human agreement (r~0.27-0.32) - while on a rubric with a verifiable factual answer the same judges fall back into the human range (58.5 degrees, r=0.519). EVIDENCE: Mukherjee, Hamna, Bali, Sitaram (arXiv 2606.03043, abstract opened). 41 LLM judges, 4 datasets, 8 Indic languages, geometric analysis with bootstrap CIs. Fine-tuning/preference optimization recovers score spread (0.32 to 1.08) but barely moves the axis; only post-hoc calibration on a small human-anchored set improved all rubrics. Scope caveat: Indic-language community-health data, but the verifiable-vs-subjective contrast is the mechanism of interest. SOURCE: The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment | https://arxiv.org/abs/2606.03043 | 2026-06-02 | academic IMPLICATION: Multi-judge ensembles and inter-judge agreement are NOT validity evidence on subjective axes - consensus can reflect a shared collapsed subspace. The objective-subjective boundary (design Q2) is empirically real and large: verifiable criteria are judge-autonomous territory; subjective criteria need human-anchored calibration sets, not more judges. ### Single-trial LLM judging is measurably noisy: across 29 tasks with 50 repeated trials, pairwise preferences flipped on average 13.6% of the time (28% of questions exceeded 20% flip rate), semantically equivalent prompt templates changed majority outcomes in 25% of tested cases, and ~11 repeated trials were needed for majority vote to recover the reference verdict with 95% probability. EVIDENCE: Yagubyan (arXiv 2606.13685, abstract opened). GPT-4o-mini and GPT-4.1-mini judges; also found significant first-position bias (72% A-majority, p=0.024), cross-judge agreement only 76% (kappa 0.51), and a pairwise-pointwise gap where judges pick winners even when their own scalar scores show no meaningful difference. Caveat: both judges from one provider, mini-tier models; the Reliability-without-Validity cohort shows some production judges reach test-retest >0.95. SOURCE: The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation | https://arxiv.org/abs/2606.13685 | 2026-04-23 | academic IMPLICATION: Answers design Q5's flip-rate fork with 'both': k-sample voting (k around 5-11) plus position randomization is baseline hygiene, AND per-item flip-rate is a cheap, free signal of criterion underspecification - items with high vote entropy are exactly the 'instructions underdetermine this' class. A perturbation-robustness harness is justified as a ship gate. ### The broadest LLM-vs-human-judge study on existing human-annotated datasets (JUDGE-BENCH, 20 datasets) found the best model (GPT-4o) reached only kappa = 0.28 +/- 0.32 on categorical judgments and Spearman rho = 0.50 +/- 0.21 on graded ones, with enormous task-to-task variance - and LLMs agreed MORE with non-expert annotators than with experts. EVIDENCE: Bavaresco et al., 'LLMs instead of Human Judges?' (ACL 2025). Structured/rule-like attributes correlate reliably; subjective attributes (engagingness) poorly; safety judgments produced negative correlations due to guardrail refusals. Authors conclude task-specific human validation is essential before deploying LLM judges. Read via detailed paper-note secondary source; consistent with the ACL publication. SOURCE: LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (JUDGE-BENCH) | https://en.papernotes.org/ACL2025/llm_nlp/llm_vs_human_judges_study/ | 2025-07 (ACL 2025) | academic IMPLICATION: Chance-corrected agreement with humans on subjective axes is far lower than the practitioner '85% agreement' narrative. The agree-more-with-non-experts finding is a red flag for the expertise-gap root cause: an uncalibrated judge may replicate crowd-level judgment, not expert judgment - expert-adjudicated (not crowd-consensus) gold labels are required for validation. ### Preference leakage is a measured, large contamination bias: a judge favors outputs of models trained on data from a related generator by up to 28.7 percentage points (Preference Leakage Score), with a relatedness gradient - same model 23.6% average, fine-tuned descendants 19-22%, same family 2.8-8.9% - and judges cannot detect their own students (near-chance) while a BERT classifier can (82.4%). EVIDENCE: Li et al., ICLR 2026 (arXiv 2502.01534). SFT transmits the bias strongly (23.6% avg PLS) vs DPO (5.2%); effect scales roughly linearly with synthetic-data mixing ratio with no threshold; style/format removal cuts PLS sharply; only contextual calibration mitigated it (17.8 to 7.3); prompting-based mitigation failed. SOURCE: Preference Leakage: A Contamination Problem in LLM-as-a-judge (ICLR 2026) | https://arxiv.org/abs/2502.01534 | 2026-02 (ICLR 2026; arXiv 2025-02) | academic IMPLICATION: Answers design Q5 self-preference: when attempters critique (or produce with the help of) outputs from model family X, the judge should not be family X or its distillation lineage - and since lineage is often undisclosed, cross-family judge routing plus style-stripping/canonicalization is the practical defense. Prompt-level debiasing will not fix this. ### Dedicated trained judges advanced substantially in 2025: Meta's J1 (RL-trained thinking-judge, May 2025) at 32B outperforms o1-mini, o3, and 671B DeepSeek-R1 on some judge benchmarks; CompassJudger-2-7B (July 2025) matches far larger generalists on judge benchmarks; Skywork-Reward-V2-Llama-3.1-8B (July 2025) topped seven reward benchmarks including JudgeBench. EVIDENCE: J1 arXiv 2505.10320 (verified): unified verifiable-reward RL over 22K synthetic pairs, mitigates position bias by construction, gains from test-time self-consistency. CompassJudger-2 arXiv 2507.09104: verifiable-reward + margin-loss training; 32B scores 80.9% accuracy on JudgerBenchV2. Skywork-Reward-V2 arXiv 2507.01352: 26M human-AI-curated preference pairs; 8B model led RewardBench v1/v2, PPE, RM-Bench, JudgeBench among reward models. SOURCE: J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (with CompassJudger-2, Skywork-Reward-V2) | https://arxiv.org/abs/2505.10320 | 2025-05 to 2025-07 | academic IMPLICATION: Small trained judges are now viable for high-volume, well-specified sub-checks (cost tier), but their wins are on preference-style benchmarks; for reasoning-heavy annotation QA the JudgeBench evidence still favors frontier reasoning models. A two-tier design (cheap trained judge for triage, frontier reasoner for verdicts) is supported; the J1 training recipe (verifiable rewards, position-bias-symmetric training) is reusable if a custom judge is ever trained. ### RewardBench 2 (Ai2, June 2025) made reward-model evaluation ~20 points harder via best-of-4 format (random = 25%) and unseen human prompts; top models score below 40% on Precise Instruction Following, yet benchmark scores correlate 0.87 (Pearson) with downstream best-of-N performance across 113 reward models. EVIDENCE: arXiv 2506.01937 (verified via abstract, Ai2 dataset page, and Lambert's analysis post). 1,865 cases across Factuality, Focus, Math, Precise IF, Safety, Ties. Lambert (author) notes generative LLM-as-judge approaches remain 'weaker than expected relative to standard reward models' at output ranking, and that frontier models still fail trivial rankings ('name a color in the rainbow'). SOURCE: RewardBench 2: Advancing Reward Model Evaluation | https://arxiv.org/abs/2506.01937 | 2025-06-02 | academic IMPLICATION: Precise-instruction-following is the weakest judged capability (<40% at best-of-4) - exactly the 'is this annotation compliant with written project instructions?' skill AutoQA needs. Do not assume instruction-compliance checking is a solved sub-problem; it needs its own gold set, and deterministic/programmatic checks should absorb whatever can be compiled to rules. ### In expert-knowledge domains, LLM judges agreed with subject-matter experts only 68% (dietetics, registered dietitians) and 64% (mental health, clinical psychologists) on overall pairwise preference, with agreement varying further across domain-specific aspect questions. EVIDENCE: Szymanski et al. (arXiv 2410.20266; ACM IUI 2025, ~220 citations). Mixed-methods pairwise comparison study; authors conclude LLMs alone lack the depth for complex knowledge-specific tasks and experts must stay in the evaluation loop. Full abstract text opened and confirmed. SOURCE: Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks | https://arxiv.org/abs/2410.20266 | 2024-10 (IUI 2025) | academic IMPLICATION: For projects requiring genuine domain expertise (root cause 2), expect judge-expert agreement in the 60s on holistic preference - below any reasonable autonomy bar. Expertise-heavy axes belong in the judge-assisted or human-only lane, and the axis-triage classifier (design Q2) should treat 'requires specialized professional knowledge' as an explicit routing feature. ### LLM judges are measurably less reliable on long-form outputs: LongJudgeBench (June 2026) finds current judges unstable across real-world long-form scenarios requiring document-level assessment of organization, coverage, and cross-section consistency, and rubrics or references help but are 'not always sufficient'. EVIDENCE: Chen et al., arXiv 2606.01629 (abstract opened and confirmed). First meta-evaluation benchmark targeting long-form judging specifically; systematically evaluates judges across base models and judging protocols; code public. Complements the short-form-centric older benchmarks (MT-Bench, LLMBar, JudgeBench). SOURCE: Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation (LongJudgeBench) | https://arxiv.org/abs/2606.01629 | 2026-06-01 | academic IMPLICATION: Attempter writeups/critiques are long-form documents; holistic single-pass judging of them inherits this instability. Supports the decomposition-first architecture (per-claim verification then aggregation) over holistic item scoring, and flags 'global coherence of the writeup' as a check that per-claim decomposition alone will miss (checklist myopia risk from design Q3 is real in both directions). ### Tools & artifacts - JudgeBench: Meta-evaluation benchmark (350 verified response pairs, objective correctness ground truth) with live HuggingFace leaderboard | https://github.com/ScalerLab/JudgeBench | Canonical hard test for whether a candidate AutoQA judge can detect actual errors rather than preferences; leaderboard at huggingface.co/spaces/ScalerLab/JudgeBench - RewardBench 2: Ai2's best-of-4 reward model benchmark (1,865 cases, 6 subsets incl. Factuality and Precise Instruction Following) | https://huggingface.co/datasets/allenai/reward-bench-2 | The Precise-IF and Factuality subsets are the closest public proxies for instruction-compliance and grounding checks in AutoQA - RubricEval + Rubric Arbitration Framework (RAF): Rubric-level meta-evaluation benchmark plus a multi-model arbitration pipeline for producing gold labels (85% human agreement, kappa 0.702) | https://www.alphaxiv.org/overview/2603.25133 | RAF's disagreement-cascade (base judges -> rationale re-evaluation -> reasoning meta-judges) is a directly reusable architecture for AutoQA gold-set production and escalation routing - J1 (Meta): RL training recipe for thinking-judges using verifiable rewards over synthetic pairs; position-bias-mitigating by construction | https://arxiv.org/abs/2505.10320 | The reference recipe if a custom project-tuned judge is ever trained; also evidence that judge chain-of-thought quality is the optimizable variable - CompassJudger-2 (7B/32B, open weights): OpenCompass generalist judge models trained with verifiable rewards + margin loss; includes JudgerBenchV2 (10k questions, 10 scenarios) | https://github.com/open-compass/CompassJudger | Best open-weight generative judge candidate for a cheap triage tier - Skywork-Reward-V2 (0.6B-8B, open weights): Scalar Bradley-Terry reward models trained on 26M human-AI-curated preference pairs | https://github.com/SkyworkAI/Skywork-Reward-V2 | State-of-art scalar scorers; useful for high-volume ranking/filtering but not for criteria-grounded verdicts with rationales - Atla Selene Mini (8B, open weights): General-purpose fine-tuned evaluator model (Jan 2025), promptable with custom criteria | https://arxiv.org/abs/2501.17195 | Open-weight promptable evaluator baseline; claims high agreement with domain experts on FinanceBench/CRAFT-MD but predates the 2025-26 judge generation - LongJudgeBench: First meta-evaluation benchmark for judging long-form outputs (June 2026), code public | https://github.com/cjj826/LongJudgeBench | Test harness for the long-form writeup judging AutoQA must do; measures rubric/reference sufficiency - llm-as-a-judge.github.io: Maintained living survey and paper list for the LLM-as-judge field (tracks 2025-2026 papers incl. Preference Leakage, ToolPRMBench) | https://llm-as-a-judge.github.io | Ongoing monitoring source for judge research; companion to Gu et al. 'A survey on LLM-as-a-judge' (The Innovation, 2026, ~1,667 citations) - Minimum Viable Validation Protocol (from arXiv 2606.19544): A distilled judge-validation protocol: chance-corrected agreement + consistency protocol + bias audit, from the 541k-judgment study | https://arxiv.org/abs/2606.19544 | Template for the AutoQA per-project ship gate: what to measure before any project's judge config goes live ### Disagreements - Dedicated trained judges vs frontier generalists: JudgeBench (ICLR 2025) shows fine-tuned judges and reward models plateau near random-to-64% on hard reasoning-based judging while frontier reasoning models lead (o1-preview 75.4%); but J1 (Meta, 2025-05), CompassJudger-2 (2025-07), and Skywork-Reward-V2 (2025-07) each claim small trained models beat o1-mini/o3-class models on judge/reward benchmarks; and Lambert (RewardBench 2 author, 2025-06) argues scalar reward models STILL beat generative LLM judges at output ranking. The conflict is largely benchmark-dependent (preference-style vs verifiable-reasoning tasks) - no source resolves which wins for criteria-based QA of human annotations, which resembles neither exactly. - Judge-human agreement levels: practitioner sources claim ~85% agreement 'higher than human-human agreement' (Confident AI, 2026) and '>80%, matching human-to-human levels' (Galileo); academic chance-corrected measurements show best-model kappa of 0.28 +/- 0.32 across 20 tasks (JUDGE-BENCH, ACL 2025), universal 33-41pp kappa deflation vs raw agreement (arXiv 2606.19544, 2026-06), and near-orthogonal judge-vs-human evaluation axes on subjective rubrics (arXiv 2606.03043, 2026-06). Much of the gap is percent-agreement vs chance-corrected statistics plus task subjectivity - the practitioner numbers are not wrong so much as unfalsifiably framed. - Verbosity bias magnitude: the 2606.19544 large-scale audit found verbosity bias small (<0.011) across 21 judges under a single controlled pairwise rubric, whereas the broader literature (Zheng 2023 MT-Bench, Gu et al. survey 2026, arXiv 2510.12462) treats verbosity bias as a major systematic failure. Possible reconciliation: bias magnitude depends heavily on rubric structure and elicitation format, meaning it is a controllable design variable rather than a constant. - Run-to-run consistency: The Coin Flip Judge (2026-04) reports 13.6% average pairwise flip rate and recommends ~11-vote majorities (mini-tier OpenAI judges only), while Reliability without Validity (2026-06) measured test-retest reliability >0.95 for some production judges - consistency is highly judge-specific, and the latter paper explicitly warns high consistency coexists with severe position bias (consistency is not validity). ### Gaps - No benchmark meta-evaluates judges on the AutoQA task itself - verifying HUMAN-written evaluative claims (grounding of praise, evidence-conclusion entailment in critiques, severity calibration). Closest proxies are rubric-verification (RubricEval), factuality judging (RewardBench 2 Factuality), and critique evaluation inside J1/CompassJudger training; the positive-claim/'warranted vs empty praise' verification question (design Q3) has no published precision/recall numbers anywhere I could find. - Could not obtain a verified July-2026 leaderboard snapshot for the newest frontier models (GPT-5.x, Claude Opus 4.x, Gemini 3) on JudgeBench or RewardBench 2 - both official leaderboards render via dynamic JS (HuggingFace Spaces) and did not yield data to fetches; the largest systematic evaluation covering 'the April 2026 frontier' (arXiv 2606.19544) reports cohort-level findings but does not name per-model winners in accessible text. Best-available anchor numbers are therefore early-2026 or older. - No single study reports judge-vs-adjudicated-expert agreement broken out per evaluation-axis type (factual grounding vs instruction compliance vs tone vs holistic) - the per-axis agreement ceiling matrix needed for design Q5's lane assignment must be assembled from fragments (RewardBench 2 subsets, JUDGE-BENCH task variance, RubricEval categories) or measured in-house. - Human inter-reviewer reliability floors (Krippendorff's alpha) for annotation-QA reviewers specifically were out of this domain's scope and not covered by the judge meta-evaluation literature; the rating-indeterminacy framework (NeurIPS 2025) is the right machinery but its 11 tasks are generic rating tasks, not training-data QA. - Atla (Selene) company status as of mid-2026 unverified - Selene Mini (8B, open weights, Jan 2025) claims remain reproducible from the model card, but I found no 2026 successor release, so treat Selene as a frozen artifact rather than a maintained product line. ### Verifications - CLAIM: On hard, objectively-verifiable judging tasks (JudgeBench, ICLR 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while GPT-4o scored 50.9-56.6% (near random), the best reward model (Skywork-Reward-Gemma-2-27B) hit 64.3%, and fine-tuned judges like PandaLM fell at or below random. VERDICT: confirmed | Every quantitative claim checks against the paper's own tables (arXiv:2410.12784 v2, ICLR 2025 camera-ready), read directly from the PDF: GPT-4o vanilla 50.86% / Arena-Hard 56.57% overall (Table 1), o1-preview 75.43% overall with 85.71% on both math and coding (Table 2), Skywork-Reward-Gemma-2-27B 64.29% as best reward model (Table 3, full range 59.43-64.29, paper says \"approximately 59% to 64%\" so the claim's \"60-64%\" is a minor round-up at the low end), PandaLM 13.14% (far below random; paper: all fine-tuned judges except Skywork significantly below random), ChatEval 34.00%. 350 verified pairs across knowledge (154)/reasoning (98)/math (56)/coding (42) confirmed - note this is the GPT-4o-generated split; a separate 270-pair Claude-3.5-Sonnet split exists. Dates confirmed: arXiv v1 2024-10-16, ICLR 2025, EmergentMind synthesis updated 2026-01-09. Two contextual nuances, neither refuting: (1) in the v2 tables o1-preview is not the single best judge - o3-mini (high) scores 80.86% and DeepSeek-R1 73.14%, which strengthens rather than weakens the \"reasoning models dominate\" thesis; (2) 2025-2026 follow-on work (J1, RM-R1, meta-judging, RRD) has pushed JudgeBench accuracies to 77-81%+, so \"GPT-4o near random\" describes the late-2024 snapshot of non-reasoning judges, not the current frontier. The claim as stated about the paper's findings is accurate. - CLAIM: Standard LLM-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial LLMs, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy. VERDICT: partially_confirmed | Core claim confirmed against the actual NeurIPS poster page (https://neurips.cc/virtual/2025/poster/117308): authors, rating-indeterminacy framing, forced-choice vs multi-label response-set elicitation, 11 tasks, 9 commercial LLMs, and the \"as much as 31% worse\" figure all match verbatim. Two corrections: (1) \"provably biased\" overstates the paper's own language - it says differing resolution of indeterminacy \"can heavily bias\" validation, and the 31% gap is empirical, not a theorem (the theory links performance measures/elicitation schemes); (2) the 2025-09-19 date is not shown on the poster page - plausibly the acceptance date, but unverified; the paper dates to arXiv March 2025 and NeurIPS Dec 2025, with an ML@CMU blog post 2025-12-09. Search for later work (as of July 2026) found follow-on papers (IRT-based judge reliability, arXiv 2602.00521; grading-scale alignment, arXiv 2601.03444) that build on, not refute or supersede, this result. Code: https://github.com/lguerdan/indeterminacy. CORRECTED: Guerdan, Barocas, Holstein, Wallach, Wu & Chouldechova (NeurIPS 2025; arXiv:2503.05965, first posted March 2025) show that standard LLM-judge validation via forced-choice human gold labels can be heavily biased when rating criteria admit multiple valid interpretations (\"rating indeterminacy\"): across 11 real-world rating tasks and 9 commercial LLMs, forced-choice-based validation selected judge systems performing as much as 31% worse than those selected by their framework using multi-label \"response set\" ratings. The bias mechanism is supported by a theoretical framework, but the 31% suboptimality figure is an empirical finding, not a proof. - CLAIM: The largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including April-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected Cohen's kappa is 33-41 percentage points lower on MT-Bench - and judge rankings shift by up to 14 positions depending on which benchmark you use. VERDICT: confirmed | Fetched https://arxiv.org/abs/2606.19544 directly. Every quantitative element checks out against the abstract: 21 judges from 9 providers, 118 runs / ~541,000 judgments across MT-Bench/JudgeBench/RewardBench; kappa deflation of 33-41 percentage points on MT-Bench across all 21 models; ranking shifts up to 14 positions across benchmarks; consistency-bias paradox (test-retest reliability >0.95 with position bias >0.10 in two production judges); verbosity bias <0.011; Minimum Viable Validation Protocol; April 2026 frontier models included. Submission date June 17, 2026 matches; authors Norman, Rivera, Hughes match. \"Largest to date\" is the authors' own framing, but two independent searches (July 14, 2026) found no larger or superseding meta-evaluation; follow-on work (AURA arXiv 2606.19714, Apple correlated-panels paper, \"Below the Reliability Floor\" on OpenReview) complements rather than supersedes it. Caveats: preprint with 0 citations; authors themselves limit dataset authority to the March-April 2026 measurement window due to hosted-endpoint drift. --- ## Domain: annotation-quality *annotation science, disagreement, contamination* ### Roughly one-third of crowdworkers used LLMs on an LLM-advantaged text-production task (33-35% on Prolific, July 2023; ~34% Prolific self-report in a 2025 follow-up by Zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). [LOAD-BEARING] EVIDENCE: Veselovsky et al. ran two preregistered Prolific studies (n=168 and n=720, medical-abstract summarization) with a calibrated classifier plus self-report and keystroke heuristics; prevalence 33.3-35.4% baseline (95% CIs ~[26%,43%]); a 3x3 factorial of requests x hurdles halved but did not eliminate use. The June 2026 community survey (arXiv 2606.04924) corroborates: 30-40% estimated reliance and 34% Prolific self-report, and notes mitigations 'reduce rather than eliminate' LLM-assisted responses. SOURCE: Prevalence and Prevention of Large Language Model Use in Crowd Work (CACM; arXiv 2310.15683) | https://arxiv.org/abs/2310.15683 | 2025-02-18 | academic IMPLICATION: AutoQA must assume a contamination base rate of ~15-35% on any text-heavy attempter task even after friction measures; per-item content detection and 'please don't' policies cannot be the primary defense, and mitigation friction has a measurable quality cost that the QA layer must not misattribute to attempter skill. ### An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. [LOAD-BEARING] EVIDENCE: Westwood (PNAS, Nov 20 2025; 43,800 evaluations across 9 LLMs) shows instruction-following checks, logic puzzles, reCAPTCHA, and rare-event probes all fail; persona calibration even masks superhuman-accuracy tells. Recommended defenses are entirely provenance/economics-based: identity revalidation, per-worker throttling, completion-history transparency, pass/fail history, location/VPN checks - 'there is no magical fix.' The 2026 crowdsourcing survey (arXiv 2606.04924) independently confirms detectors degrade under paraphrase and out-of-domain, and mixed human-AI authorship is harder still. SOURCE: The potential existential threat of large language models to online survey research (PNAS 122(47)) | https://www.pnas.org/doi/10.1073/pnas.2518075122 | 2025-11-20 | academic IMPLICATION: Directly answers taxonomy Q8: wholesale-LLM-submission defense must be economics/provenance-based (rate limits, work-history consistency, randomized deep audits, process telemetry), not per-item text classification; any AutoQA spam layer built on stylistic detection will be silently beaten within the cost asymmetry. ### Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. [LOAD-BEARING] EVIDENCE: Elangovan et al. (Amazon), ICLR 2025 (arXiv 2410.03775, opened and verified): Krippendorff's alpha / kappa were designed for human-human reliability and their assumptions break for machine labels; the paper's fix is (1) stratify meta-evaluation by human-label uncertainty, (2) use binned Jensen-Shannon divergence for perception-type (inherently subjective) tasks, (3) report 'perception charts' instead of one correlation. Code open-sourced (amazon-science/BeyondCorrelation). SOURCE: Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025) | https://arxiv.org/abs/2410.03775 | 2025-05-01 | academic IMPLICATION: Answers taxonomy Q1 directly: meta-evaluation must stratify gold/validation items by measured human agreement level and report per-stratum judge performance; 'accuracy vs. humans' as a single number should be banned from reporting, and judge validity claims are only meaningful on the high-human-agreement stratum. ### In human preference/rating data, the majority of annotator disagreements are attributable to task underspecification and response-style preferences - not random noise and not primarily expertise gaps - and standard aggregation (Bradley-Terry reward modeling) plus LLM-as-judge evaluation both fail to account for this divergence. [LOAD-BEARING] EVIDENCE: Zhang et al., ICML 2025 (arXiv 2410.14632, opened and verified): built a 10-category, 4-class taxonomy of disagreement sources on preference datasets; abstract states 'the majority of disagreements are due to factors such as task underspecification or response style,' explicitly challenging the noise assumption; they show LLM evaluations are heavily swayed by divisive style features and develop divergence-identification methods. Converges with Jiang & de Marneffe (TACL 2022) who found the same structure in NLI. SOURCE: Diverging Preferences: When do Annotators Disagree and do Models Know? (ICML 2025) | https://arxiv.org/abs/2410.14632 | 2025-07-15 | academic IMPLICATION: Supports root-cause-1 primacy and taxonomy Q2: the verdict ontology needs a first-class 'instructions underdetermine this case' class detected via measured divergence, and a cheap variance-attribution pilot is empirically justified because published decompositions find underspecification (fixable by rubric compilation) dominates over noise (fixable by averaging). ### Even on a nominally objective task (NLI), annotator disagreement decomposes into a 10-category taxonomy across three high-level sources - uncertainty in sentence meaning, underspecification in guidelines, and annotator behavior - proving that a fraction of inter-reviewer variance is item-intrinsic and survives perfect rubric operationalization. EVIDENCE: Jiang & de Marneffe, TACL 2022, developed and hand-applied the taxonomy to high-disagreement NLI items; disagreement persisted among trained annotators on items with genuine interpretive ambiguity. D3CODE (EMNLP 2024; 4.5K sentences, 4K+ annotators, 21 countries) adds that residual variance is systematically structured by annotators' moral values and region, not random. SOURCE: Investigating Reasons for Disagreement in Natural Language Inference (TACL 10) | https://aclanthology.org/2022.tacl-1.78/ | 2022-12-01 | academic IMPLICATION: The target metric cannot be raw agreement: some disagreement is item-intrinsic signal. The variance-attribution pilot should code disagreements against a taxonomy like this to split rubric-fixable variance (underspecification) from irreducible variance (ambiguity/values) before setting build order. ### Chance-corrected agreement coefficients (Cohen's kappa, and analogously Krippendorff's alpha) can be near zero despite very high raw agreement when class prevalence is extreme (the Feinstein-Cicchetti paradoxes), so under the 90%+ pass rates typical of QA pipelines, single-coefficient reliability reporting is misleading in the opposite direction from raw percent agreement. EVIDENCE: Feinstein & Cicchetti (J Clin Epidemiol, 1990) formalized the two kappa paradoxes (high agreement/low kappa under imbalance; asymmetric marginals inflating kappa); the effect is prevalence-driven, extensively replicated (Quarfoot & Levine, Am. Statistician 2016; Gwet's AC1 literature). This is exactly the regime of QA pass/fail data with low failure base rates. SOURCE: High agreement but low kappa: I. The problems of two paradoxes (Feinstein & Cicchetti) | https://doi.org/10.1016/0895-4356(90)90158-L | 1990-01-01 | academic IMPLICATION: Answers taxonomy Q1 stats question: standardize on reporting a chance-corrected coefficient PLUS per-class recall at fixed prevalence PLUS CIs; ban both raw percent-agreement AND any single coefficient in isolation, because at extreme pass rates alpha/kappa collapse even for a well-functioning judge. ### Classical label-aggregation models (Dawid-Skene 1979, MACE 2013 with its explicit spammer latent variable, GLAD) all assume annotators are conditionally independent given the true label - an assumption measurably violated when the 'annotators' are LLM judges sharing data/architectures/prompts, causing miscalibrated posteriors and confidently wrong aggregate verdicts; Feb 2026 work replaces this with dependence-aware Ising-model aggregation. EVIDENCE: Balasubramanian, Podkopaev & Kasiviswanathan (Amazon/UC Davis, arXiv 2601.22336, Feb 2 2026) state the CI assumption 'is often violated by LLM judges due to shared data, architectures, prompts, and failure modes' and that ignoring dependencies 'can yield miscalibrated posteriors and even confidently incorrect predictions'; they build aggregation via Ising models. MACE (Hovy et al., NAACL 2013) remains the canonical spam-aware human aggregator, with production implementations in Toloka's crowd-kit. SOURCE: Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models (arXiv 2601.22336) | https://arxiv.org/pdf/2601.22336 | 2026-02-02 | academic IMPLICATION: If AutoQA uses k-sample voting or multi-judge ensembles, naive Dawid-Skene/majority aggregation will overstate confidence because same-family judge samples share failure modes; confidence estimates must model correlated errors (or use genuinely different model families), and MACE-style competence/spam latent variables remain the right tool for the human-reviewer side. ### LLM contamination in low-dimensional annotation data (multiple-choice labels, ratings) can be evaluated at the worker level without any ground truth, using peer-prediction scores that condition on LLM-generated reference labels - a training-free mechanism with theoretical guarantees under an LLM-collusion model, published at NeurIPS 2025 - because text-based detectors are inapplicable to short label outputs. EVIDENCE: Zhang, Pang, Zhu & Liu (NeurIPS 2025; arXiv 2506.06991, opened and verified): existing detectors need high-dimensional text; their mechanism scores the informativeness of worker answers via inter-worker correlation conditioned on a subset of LLM labels, explicitly modeling collusion (multiple workers pasting from the same LLM), and empirically detects low-effort cheating on real crowdsourcing datasets. Scoring is per-worker, not per-item. SOURCE: Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth (NeurIPS 2025) | https://arxiv.org/abs/2506.06991 | 2025-12-01 | academic IMPLICATION: For rating/label-type attempter outputs where stylistic detection is impossible, the viable contamination sensor is worker-level statistical scoring (cross-worker correlation vs. an LLM-conditioned baseline) run continuously - reinforcing that provenance/behavioral defense operates at the annotator level, not the item level. ### The NLP field has institutionalized disagreement-as-signal: the third Learning-With-Disagreements shared task (LeWiDi-2025, EMNLP 2025) standardizes dual evaluation - soft-label (predict the population label distribution, Wasserstein/Manhattan distance) and perspectivist (recover individual annotators' labels) - across four datasets that all ship per-annotator labels rather than adjudicated gold. EVIDENCE: Leonardelli et al. (arXiv 2510.08460, Oct 2025; le-wi-di.github.io) describe the two complementary paradigms and public leaderboard; lineage runs from Aroyo & Welty's 'truth is a lie' perspectivism through DICES (NeurIPS 2023 safety dataset with up to 100+ raters/item) and jury learning (Gordon et al., CHI 2022, which composes verdicts from explicitly selected rater subpopulations); Frenda et al.'s survey (LREV 2024) documents the dataset ecosystem. SOURCE: LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task | https://arxiv.org/pdf/2510.08460 | 2025-10-09 | academic IMPLICATION: The AutoQA data model should preserve rater identity and full label distributions (never only adjudicated verdicts), and for judgment-laden axes the judge's target should be the distribution/jury composition, not a forced scalar - this is now the field-standard evaluation contract, giving ready-made metrics (Wasserstein/Manhattan soft-label distance) for 'consistency-of-application' monitoring. ### Gold/test labels themselves are wrong at material rates - a lower-bound average of 3.3% label errors across 10 canonical benchmarks (at least 6% in the ImageNet validation set), validated by human review of confident-learning-flagged candidates (51% flag precision) - and error rates this size are enough to flip model rankings. EVIDENCE: Northcutt, Athalye & Mueller (NeurIPS 2021 Datasets & Benchmarks; arXiv 2103.14749, opened and verified): algorithmic flagging via confident learning + crowdsourced verification; ResNet-18 beats ResNet-50 on corrected ImageNet if mislabeled prevalence rises just 6%. Cleanlab is the maintained open-source implementation. SOURCE: Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (NeurIPS 2021 D&B) | https://arxiv.org/abs/2103.14749 | 2021-11-01 | academic IMPLICATION: Answers taxonomy Q1's false-agreement question in the only measured form available: several percent of any 'gold' set is wrong, so judge-vs-gold disagreement at that rate is uninformative; the gold set needs its own standing audit channel (confident-learning-style flagging + expert re-adjudication) as a permanent component, and judge certification margins must exceed the gold error floor. ### Showing annotators LLM suggestions makes them anchor on the suggestions - significantly shifting the label distribution versus unassisted baseline, raising self-reported confidence without making them faster - and evaluating models against LLM-assisted labels significantly inflates reported model performance. EVIDENCE: Schroeder, Roy & Kabbara (Findings of ACL 2025, July 2025; pre-registered, 350 annotators, 7,000 annotations, 4 conditions x 2 models x 2 datasets, opened and verified): annotators 'strongly took the LLM suggestions'; the paper warns homogenization/lessened variation is a direct risk when LLMs are inserted into annotation loops, and that 'human-approved' LLM-assisted labels change downstream conclusions. SOURCE: Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks (Findings of ACL 2025) | https://aclanthology.org/2025.findings-acl.1323/ | 2025-07-27 | academic IMPLICATION: Direct evidence for taxonomy Q8 monoculture and Q9 induced drift: if AutoQA feedback (or any AI verdict) is visible to attempters/reviewers before they judge, distributions converge on judge-pleasing labels and self-evaluation inflates - so human touches must be blind-first, and population-level label-distribution drift is the right monoculture sensor. ### RLVR-style reasoning training degrades an LLM's ability to model human annotator disagreement, while naive chain-of-thought on RLHF models improves it - evaluated across 60 setups on 3 tasks - meaning stronger 'reasoning' judges are measurably worse at knowing when humans legitimately diverge. EVIDENCE: Ni et al., EACL 2026 main (arXiv 2506.19467 v3, Jan 12 2026, opened and verified): systematic evaluation over model sizes, distribution-expression and steering methods using variance-correlation and distributional-alignment metrics; authors explicitly warn about 'the potential risk of replacing human annotators with reasoning LLMs' where disagreement carries signal; author communication adds that small fine-tuned models (ModernBERT-class) with human labels often beat large reasoning models at this. SOURCE: Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? (EACL 2026) | https://arxiv.org/abs/2506.19467 | 2026-01-12 | academic IMPLICATION: For the 'instructions-underdetermine / route-to-human' triage lane, do not assume the strongest reasoning judge is the best ambiguity detector: disagreement-prediction may need a dedicated small fine-tuned model (or CoT-on-RLHF configuration) separate from the verdict judge. ### Tools & artifacts - Cleanlab (confident learning): Open-source library implementing confident learning for algorithmic label-error detection in labeled datasets | https://github.com/cleanlab/cleanlab | Ready-made machinery for the standing gold-set audit channel: flag suspect gold/QA labels for expert re-adjudication (51% flag precision measured in the NeurIPS 2021 study). - crowd-kit (Toloka): Maintained Python library of aggregation/truth-inference algorithms: Dawid-Skene, GLAD, MACE, Wawa, ZeroBasedSkill, IBCC/EBCC | https://github.com/Toloka/crowd-kit | Production implementations for aggregating multiple human reviewers (and multi-judge samples) with per-worker competence and spammer modeling - but note the conditional-independence caveat for same-family LLM judges. - LeWiDi-2025 datasets + leaderboard: Four benchmark datasets with per-annotator labels plus standardized soft-label (Wasserstein/Manhattan) and perspectivist evaluation metrics, EMNLP 2025 shared task | https://le-wi-di.github.io/ | Off-the-shelf metrics and data format for evaluating disagreement-aware judge components and for the 'consistency-of-application vs agreement-with-truth' distinction. - DICES dataset (Google): Diversity-in-Conversational-AI-Safety dataset: 990 conversations with dense parallel safety ratings (up to 100+ demographically diverse raters per item) | https://github.com/google-research-datasets/dices-dataset | Reference dataset for measuring how much rater variance survives on a judgment-laden axis with a fixed rubric - useful for calibrating the variance-attribution pilot. - BeyondCorrelation (Amazon Science): Code for uncertainty-stratified meta-evaluation of automatic evaluators: perception charts, binned Jensen-Shannon divergence | https://github.com/amazon-science/BeyondCorrelation | Directly implements the recommended meta-evaluation reporting standard (stratify by human agreement) for certifying the AutoQA judge. - Can Crowdsourcing Survive the LLM Era? (community survey): June 2026 arXiv survey (2606.04924) synthesizing prevalence, detection, and mitigation literature on LLM use in human data collection | https://arxiv.org/html/2606.04924v1 | Most current single map of the LLM-contamination literature; documents detector degradation under paraphrase and the state of mitigation practice as of mid-2026. ### Disagreements - Detectability of LLM-generated submissions: Veselovsky et al. (2023/2025) achieved usable classifier-based detection on a long-form text task, and TU Delft's prompt-injection work (TOCHI 2025) claims in-instrument countermeasures work; Westwood (PNAS 2025) and the community survey (arXiv 2606.04924, June 2026) show detection and quality checks are comprehensively evaded by agents and degrade under paraphrase/out-of-domain, with mixed human-AI authorship hardest. The recent evidence favors Westwood, but the sources genuinely conflict on whether content-side detection retains any value as a secondary signal. - Nature of disagreement: the aggregation tradition (Dawid-Skene, MACE, majority vote) models disagreement as noise around one latent truth to be recovered; the perspectivist tradition (Aroyo & Welty, DICES, LeWiDi-2025, Diverging Preferences) treats much of it as legitimate signal to be preserved. Neither side claims universality, but they prescribe opposite data models (adjudicated gold vs. per-rater distributions) - the design must pick per-axis, and no source offers a validated automatic classifier for which regime an axis falls into. - Agreement statistics under skew: Krippendorff defends alpha as the general-purpose reliability standard; Gwet and successors show alpha/kappa collapse under extreme prevalence and push AC1/AC2; Elangovan et al. (ICLR 2025) argue all human-reliability coefficients are misapplied to machine-vs-human comparison in the first place. There is no consensus single statistic - only convergence that one-number reporting is wrong. - Mitigation of crowdworker LLM use: Veselovsky et al. measured that request+hurdle interventions halve usage (a meaningful effect worth deploying); Westwood concludes no instrument-level mitigation withstands agent-based completion ('no magical fix'). These are consistent about direction (partial reduction) but conflict sharply on whether prevention is worth its measured quality cost. ### Gaps - No published measurement of the false-agreement rate (judge and attempter agreeing while both wrong per expert panel) in an annotation-QA setting; the nearest proxy is Northcutt's 3.3%+ gold-label error floor. This number would have to be produced in-house via an independent expert-audit channel. - No public 2025-2026 prevalence data for LLM use specifically among paid annotators/reviewers on managed AI-training-data platforms (Scale, Surge, Outlier, Prolific-managed RLHF work); all published prevalence numbers come from MTurk/Prolific survey and summarization tasks, which may under- or over-state prevalence for critique/rating work. - Item-level (as opposed to worker-level) detection of LLM-generated annotations for short/low-dimensional outputs remains unsolved; the NeurIPS 2025 peer-prediction mechanism scores workers, not items, and requires overlapping assignments across workers. - No annotation-science source found that gives sample-size arithmetic for certifying a judge against a gold set under low failure base rates (per-project vs pooled hierarchical validation); this appears only in general eval-statistics literature outside this domain's charter. - No longitudinal studies found measuring whether QA feedback changes annotator behavior over time (repeat-error rates per criterion); the feedback-efficacy question (taxonomy Q7) has essentially no direct evidence in the annotation-quality literature. - Could not open the full text of the Diverging Preferences taxonomy to extract exact percentage splits across its 10 disagreement categories (abstract confirms only that underspecification + style form the majority); exact fractions would sharpen the variance-attribution prior. ### Verifications - CLAIM: Roughly one-third of crowdworkers used LLMs on an LLM-advantaged text-production task (33-35% on Prolific, July 2023; ~34% Prolific self-report in a 2025 follow-up by Zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). VERDICT: partially_confirmed | Core claim VERIFIED against the full PDF of arXiv 2310.15683 (Veselovsky, Horta Ribeiro, Cozzolino, Gordon, Rothschild, West): Study 1, n=168 Prolific workers, 3 July 2023, medical-abstract summarization; prevalence 33.3% [25.9,40.1] (classify-and-count), 35.2% [29.8,40.6] (probabilistic), 35.4% [27.8,43.0] (corrected) - matches \"33.3-35.4%, CIs ~[26%,43%]\". Study 2, n=720, 23 July 2023, 3x3 factorial (request: none/indirect/direct x hurdle: none/image/no-Ctrl-C+V); classifier estimate dropped 27.6% -> 15.9% (direct+image) and 15.8% (direct+Ctrl C+V) - matches \"roughly halved, not eliminated\" (paper: \"reduced LLM use by nearly 50%, it could not fully prevent it\"). Direct request decreased keyword retention by 6.2% (p=0.009) - exact match. Classifier was finetuned e5-base-v2, calibrated (Card & Smith), with self-report and high-precision heuristics (completion time + paste artifacts) - matches. The June 2026 community survey arXiv 2606.04924 (Velutharambath et al., 155 researchers, survey run Aug 2025-Mar 2026) does cite \"Veselovsky et al. estimate 30-40%\", cites \"Zhang et al. (2025): 34% of Prolific participants self-report using LLMs for open-ended questions\", and states mitigations \"reduce rather than eliminate LLM-assisted responses\" - so the follow-up attribution to Zhang et al. 2025 is consistent with that survey's citations (note: I verified the citation exists in the survey, not the underlying Zhang et al. paper itself). ISSUES: (1) Date \"2025-02-18\" is wrong for the arXiv link given - arXiv has only v1, dated 24 Oct 2023; the published version is Communications of the ACM 68(3):42-47, March 2025 issue (DOI 10.1145/3685527). (2) \"Preregistered\" is not supported by the paper text - no preregistration is mentioned in the arXiv PDF. (3) Minor nuance: \"never to zero\" - by the classifier measure true, but the high-precision heuristics table actually shows 0.0% in the direct+Ctrl C+V cell; the paper's own framing (\"cannot fully prevent\") still supports the claim's spirit. (4) 44% of surveyed researchers in the 2026 survey observed LLM use in their data - newer corroborating, not superseding, evidence; no later work found that overturns the prevalence estimates. Sources: https://arxiv.org/abs/2310.15683, https://arxiv.org/abs/2606.04924, https://dl.acm.org/doi/10.1145/3685527. CORRECTED: Roughly one-third of Prolific crowdworkers used LLMs on an LLM-advantaged text-production task (33.3-35.4% across three estimators, July 2023, n=168; Veselovsky et al., arXiv 2310.15683 v1 posted 24 Oct 2023, published in Communications of the ACM 68(3), March 2025). In a second study (n=720, 3x3 factorial), the best mitigations (direct request + image presentation or + copy-paste disable) roughly halved classifier-estimated usage from 27.6% to 15.9%/15.8% without eliminating it, and directly requesting non-use reduced keyword retention by 6.2% (p=0.009). The studies were not described as preregistered. A June 2026 community survey of 155 researchers (arXiv 2606.04924, Velutharambath et al.) corroborates, citing Veselovsky et al.'s 30-40% estimate and Zhang et al. (2025)'s 34% Prolific self-report, and finding 44% of researchers observed LLM use in their crowdsourced free-text data; it echoes that mitigations reduce rather than eliminate LLM-assisted responses. - CLAIM: An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. VERDICT: confirmed | Every quantitative element of the CLAIM verifies against the primary source (Westwood, \"The potential existential threat of large language models to online survey research,\" PNAS 122(47), published Nov 20, 2025; PubMed 41264250, cited 81x by Jul 2026; won a 2025 Cozzarelli Prize). Extracted paper text confirms verbatim: (1) 99.8% pass rate across 6,000 trials (20 ACQ types x 300 repetitions), only 10 errors, perfect on 18/20 check types; (2) execution layer \"simulates realistic reading times calibrated to the persona's education level, generates human-like mouse movements, and types open-ended responses keystroke-by-keystroke, complete with plausible typos and corrections\"; (3) strategically declines 97.7% of \"reverse shibboleth\" tasks; (4) ~$0.05 per survey with commercial models vs $1.50 payment (>96.8% margin), marginal cost near zero with local open-weight models; (5) 43,800 total evaluations, 139 questions, 6,700 trials, with a key subset validated across nine LLMs (DeepSeek R1, Mistral Large, Claude 3.7 Sonnet, Grok 3, Gemini 2.5, Qwen Plus, Llama 4 Maverick, GPT-4.1, o4-mini); (6) 0% error rate on impossible-event/rare-event probes. The paper's abstract and press framing support the interpretive conclusion that coherence-/content-based detection is no longer tenable and defenses must be provenance/economic. Supersession check: no 2026 work refutes it - follow-ups (Westwood & Frederick 2026 Prolific audit; a 2026 SAGE AMPPS review of AI-mediated contamination) build on it and treat detection as an unsolved provenance problem. Two minor caveats, both in the EVIDENCE SUMMARY rather than the claim: (a) arXiv 2606.04924 (\"Can Crowdsourcing Survive the LLM Era?\", Jun 2026) is a community survey of 155 researchers' practices/experiences, not a technical detector benchmark; its abstract does not state the paraphrase/out-of-domain/mixed-authorship degradation findings attributed to it (those likely come from a different detection-benchmark paper). (b) On reCAPTCHA the paper says the system \"is designed to accommodate tools for bypassing\" reCAPTCHA - an architectural capability, slightly weaker than \"reCAPTCHA... fail[s]\" as an empirically demonstrated result. Neither caveat affects the claim text itself, which is fully accurate. - CLAIM: Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. VERDICT: partially_confirmed | Substance CONFIRMED on every checked point against the arXiv record (https://arxiv.org/abs/2410.03775): (1) abstract states machine labels \"may superficially appear to have similar or better correlation with the human majority\" when human uncertainty is high, and machine-human correlation falls \"well below\" human-human correlation as human label consistency increases; (2) it flags Krippendorff's alpha / Randolph's kappa as designed for human-human reliability with assumptions inapplicable to machine labels; (3) proposed fixes match exactly - stratification by human label uncertainty, binned Jensen-Shannon divergence for perception-type tasks, and 'perception charts'; (4) venue is ICLR 2025 per the arXiv comments field; (5) code is open-sourced at github.com/amazon-science/BeyondCorrelation. The only inaccuracy is the SOURCE date: the claim says 2025-05-01, but the latest arXiv version is v3 dated 2025-01-27 (original submission 2024-10-03) - no May 2025 version exists on the record. Supersession check: a follow-up search surfaced related later work - \"Validating LLM-as-a-Judge Systems under Rating Indeterminacy\" (arXiv 2503.05965), \"Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking\" (arXiv 2604.11581), and \"CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation\" (arXiv 2603.00039, Feb 2026). These extend and reinforce the human-uncertainty/measurement-error critique rather than refute it; nothing found supersedes or contradicts the core finding. Verdict is partially_confirmed solely because of the incorrect source date; the claim itself stands. CORRECTED: Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. Source: Elangovan et al., \"Beyond correlation...\" (Amazon Science), accepted at ICLR 2025, arXiv 2410.03775 (v1 2024-10-03, latest v3 2025-01-27), code at github.com/amazon-science/BeyondCorrelation. --- ## Domain: critique-models *CriticGPT lineage, human+critic teams, omission detection* ### Human+critic-model teams occupy a strictly better operating point than either alone: in OpenAI's CriticGPT study, Human+CriticGPT teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (FSBS length penalty), not a fixed property. [LOAD-BEARING] EVIDENCE: CriticGPT critiques were preferred over human contractor critiques in 63% of cases on naturally occurring LLM errors (>80% Elo win rate on inserted bugs); both ChatGPT and CriticGPT caught substantially more inserted bugs than paid contractors (median ~50 min/review, ~5 yrs Python experience). Humans alone had far fewer hallucinated bugs/nitpicks; Human+CriticGPT teams landed between, 'moving beyond the model-only Pareto frontier.' Force Sampling Beam Search scores candidates by rm_score + LENGTH_MODIFIER x num_highlights, tracing a precision/comprehensiveness Pareto curve selectable at deployment without retraining. Verified directly from the arXiv HTML full text. SOURCE: LLM Critics Help Catch LLM Bugs (CriticGPT, OpenAI) | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary IMPLICATION: Validates the hybrid one-touch foundation: AI critic drafts findings, human filters hallucinated nitpicks. Make the precision/comprehensiveness operating point an explicit per-project config knob (Q2/Q4/Q7 false-accusation SLO), since the same trained judge can be run strict or comprehensive per project. ### Critic-assisted review of items already rated 'flawless' by a first human found real, rating-changing problems in 24% of flagged cases, versus 6% found by an unassisted second human review of the same data - a 4x lift on the exact 'confident pass' slice. EVIDENCE: CriticGPT (RL-only) was run over ChatGPT training data rated flawless by a first annotator; where the critique flagged a problem, reviewers agreed it substantially lowered the rating in 24% of cases, vs 6% in a no-critique second-rater replication - and this held out-of-distribution (mostly non-code tasks). Prioritizing by critique reward-model score further raised catch rates and cut hallucinations. Verified from arXiv HTML full text. SOURCE: LLM Critics Help Catch LLM Bugs (CriticGPT, OpenAI) - 'flawless' data experiment | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary IMPLICATION: Direct evidence for Q1's false-agreement problem and Q6's routing: unassisted human re-review of passed items misses ~3/4 of what critic-assisted review catches, so the standing expert-audit channel over confident passes should itself be critic-assisted, and critic-confidence scores can prioritize the audit queue. ### In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong. [LOAD-BEARING] EVIDENCE: 1,918 expert-labeled fact-verification items; tool-using AI rater with majority-vote-over-50-samples confidence (well calibrated). AI 87.7% vs individual humans 75.1% (majority vote 80.6%). Hybridization at threshold 0.62: 89.3%; on the routed low-confidence slice humans beat the AI 71.3% vs 60.5% (real complementarity region). Evidence-only assistance: 79.3% when AI right (vs 71.3% baseline) with no significant harm when AI wrong; conditions showing Judgments+Reasoning/Confidence produced significant over-reliance decrements (e.g. beta = -0.768, p < .001); AI-debate condition was numerically worst. Verified from arXiv HTML full text. SOURCE: Human-AI Complementarity: A Goal for Amplified Oversight (Google DeepMind) | https://arxiv.org/abs/2510.26518 | 2025-10-30 | primary IMPLICATION: Answers Q6 concretely: spend the one human touch on AI-low-confidence items, and give that human the judge's collected evidence spans WITHOUT the judge's verdict or rationale (blind-ish adjudication over curated evidence). Requires the judge to emit calibrated confidence and quoted evidence as first-class outputs (Q3), and argues against 'human verifies AI verdict' as the default interaction. ### A Nature Human Behaviour meta-analysis of 100+ experiments (300+ effect sizes) found human-AI combinations on average performed significantly WORSE than the best of human or AI alone on decision/judgment tasks, with gains only where humans outperformed the AI solo - naive human-checks-AI designs destroy value. EVIDENCE: Vaccaro, Almaatouq & Malone systematic review/meta-analysis: average human-AI combo underperformed the stronger party alone; losses concentrated in decision-making tasks, gains in content-creation tasks; combos helped when humans beat AI alone. DeepMind's complementarity paper explicitly frames its confidence-routing + evidence-only design as the answer to this result. Verified from arXiv abstract. SOURCE: When combinations of humans and AI are useful: A systematic review and meta-analysis | https://arxiv.org/abs/2405.06087 | 2024-10-28 | contrarian IMPLICATION: The Q6 fork ('is one shallow touch better than zero touch plus deep audits?') is empirically live: an automation-biased shallow verify step is the modal failure in the literature. The touch only pays where the human plausibly beats the judge (routed low-confidence/contested items), which mandates closed-loop routing on observed overturn rates rather than fixed per-item review. ### In the largest deployed RCT of AI critiquing human evaluative work (ICLR 2025, feedback on >20,000 peer reviews), 27% of reviewers who received LLM feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated LLM reliability tests. [LOAD-BEARING] EVIDENCE: Randomized controlled study at ICLR 2025 (~22.5k feedback-arm vs ~22.4k control reviews): Review Feedback Agent flagged vague comments, content misunderstandings, unprofessional remarks; guardrail LLM tests gated delivery; 26.6-27% update rate, avg +80 words among updaters, blinded informativeness gains, longer author-reviewer discussions. Verified from arXiv abstract page; corroborated by the official ICLR blog. SOURCE: Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 | https://arxiv.org/abs/2504.09737 | 2025-04-13 | academic IMPLICATION: Q7's central bet is supported at deployment scale: constructive, item-specific AI feedback measurably changes the behavior of human evaluators, and the shipped pattern held feedback to its own automated verification gate before delivery (fail-closed feedback). Also gives a realistic effect-size prior: ~1 in 4 recipients act on feedback - plan longitudinal metrics around that base rate. ### LLMs are structurally weak at detecting omissions: on AbsenceBench, average F1 drops 56.9 points versus detecting the same content as insertions (best model 71.2% overall, 40.0% on code diffs, vs ~99.5% needle-in-haystack), and inserting explicit placeholders at gap sites recovers ~35.7 points - absence has no attention key to attend to. [LOAD-BEARING] EVIDENCE: 4,302 instances across poetry, numerical sequences, GitHub PRs (~5K token contexts): models must list deliberately deleted elements given original+modified documents. Gemini-2.5-flash (thinking) best at 71.2% avg F1; Mixtral-8x7B scores 99%+ on NIAH but 14.7% here; thinking tokens buy only ~7.9%; '<missing line>' placeholders add up to +81.8%. NeurIPS 2025 Datasets & Benchmarks. Verified from arXiv HTML full text. SOURCE: AbsenceBench: Language Models Can't Tell What's Missing | https://arxiv.org/abs/2506.11440 | 2025-06-13 | academic IMPLICATION: Caps what the judge can verify autonomously (Q3/Q5): 'comprehensiveness'/'complete' praise claims and omission-type attempter failures are the judge's structurally weakest lane. The rubric compiler should convert completeness criteria into explicit enumerable checklists (placeholder effect = give absence a token), and free-form omission judgment belongs in the human or human-assisted lane, not judge-autonomous. ### Critique quality is itself quantifiable at usable reliability: MetaCritique decomposes critiques into atomic information units and scores precision/recall against references, with GPT-4 AIU-level judgments at 85-89% accuracy and Meta-F1 correlating with human gold at Pearson 0.84-0.89 - and measured baselines show human critiques are high-precision/low-recall (87.6%/48.7%) while LLM critiques are the inverse-ish (71.9%/53.3%). EVIDENCE: ACL Findings 2024 framework: precision = fraction of critique AIUs judged factual; recall = coverage of reference-critique AIUs; each judgment carries a natural-language rationale. Choosing critiques by Meta-F1 yields refinements that win 51%/lose 27% by human eval, vs GPT-4 pairwise picks losing more than winning. ~28% of LLM-critique AIUs were non-factual vs ~12% for humans; LLM critiques carried 2.4x the information volume (8.10 vs 3.31 AIUs). Verified from arXiv HTML full text. SOURCE: The Critique of Critique (MetaCritique) | https://arxiv.org/abs/2401.04518 | 2024-01-09 | academic IMPLICATION: Q7's reflexive-grounding requirement is buildable today: run AutoQA feedback through AIU-level precision scoring as its own admission gate, and report critique-precision/recall as first-class system metrics. Also sets expectations: the AI side of the pipeline supplies coverage, the human side supplies precision - mirroring CriticGPT. ### Verdict accuracy and critique validity dissociate badly: in CriticBench-THU data 24.8% of items got the correct verdict with a low-quality critique, and open-loop verdict-agreement metrics compress a 27.1-point real error-identification gap (ProcessBench: o1-mini 88.9 vs Qwen2.5-72B 61.8) into a 1.3-point verdict-F1 gap - so critique quality must be evaluated closed-loop by whether the critique drives a successful correction. EVIDENCE: RealCritic (Jan 2025, arXiv 2501.14492): closed-loop critique-then-correct design over 8 reasoning benchmarks; classical LLMs (incl. GPT-4) lose accuracy under self-critique (-1.8 to -5.1 avg; up to -35.6 domain drops) while o1-mini is the only model with positive self-critique delta; failure modes 'superficial success' (right verdict, wrong analysis) and 'contradictory output' documented. Independently confirmed at scale in 2026: an ICML 2026 paper (RM-NLHF, arXiv 2601.07349) finds outcome-rewarded generative reward models 'guess correct outcomes without sound critiques' and uses similarity-to-human-critique as a process reward to fix it. RealCritic verified via arXiv abstract + HTML (HTML render partially draft-quality; abstract and appendix numbers consistent). SOURCE: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques | https://arxiv.org/abs/2501.14492 | 2025-01-24 | academic IMPLICATION: For Q1 meta-evaluation: never certify the AutoQA on verdict agreement alone - a judge can match pass/fail labels while its rationales are wrong, which poisons the feedback channel and attempter trust. Validation must score the rationale (does the cited evidence entail the finding; does acting on the feedback fix the item), i.e., Q7's explainability contract is also the correct validation instrument. ### A June 2026 study of 21 LLM judges (~541k judgments) shows raw percent-agreement overstates chance-corrected agreement by 33.8-41.3 points (85% agreement = kappa ~0.48), judge rankings flip by up to 15 positions across benchmarks, and the most reproducible judges are among the least valid (test-retest 0.99 with position bias 0.19) - leading the authors to prescribe a pre-deployment Minimum Viable Validation Protocol. EVIDENCE: 'Reliability without Validity' (UC Berkeley, arXiv 2606.19544, evals run March-April 2026): MT-Bench/JudgeBench/RewardBench, 118 runs at temperature 0; kappa deflation universal across all 21 judges; consistency-bias paradox instantiated by Qwen3-8B and Gemini 2.5 Flash; MVVP = report kappa/Krippendorff alpha as headline, AB+BA position swaps, >=3 replicates, >=2 benchmarks spanning preference- and correctness-style labels, and audit that high stability is not just high bias. Verified from arXiv HTML full text. SOURCE: Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic IMPLICATION: Directly answers Q1's statistics question (ban raw percent-agreement; standardize on chance-corrected stats) and Q5's ship-gate question (perturbation/position-swap + replicate harness is now published prescriptive practice). Run-to-run consistency must never be reported as evidence of validity - a perfectly consistent judge can be laundering a bias. ### The closest production analog to the proposed AutoQA already exists: Toloka's deployed 'LLM QA' runs a tool-using agent per quality-metric on every human annotation submission, emits a strict three-way Pass / Fail / Unable-to-verify verdict (the abstain class exists specifically to prevent hallucinated verdicts), coaches annotators with Socratic feedback, and explicitly accepts lower precision on Fail because a false pass costs more than escalating a genuine pass. EVIDENCE: Toloka engineering blog (bylined 2026-03-30): agentic autocheck with web/image/audio/Python/bash tools; 'one metric, one entity, one verdict' scoping (separate agent instance per criterion); internal benchmark of 300+ real submissions from 20+ live projects with senior-QA ground truth (numbers in an image, not extractable); reported constraint: performance 'entirely bounded by task design' - ambiguous guidelines produce walls of Unable-to-verify. Verified by opening the post. SOURCE: LLM QA: Scaling data quality assurance technologically (Toloka) | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | 2026-03-30 | practitioner IMPLICATION: Validates several foundational choices from production: criterion-scoped judge instances (Q3 decomposition at the per-axis level), a first-class abstain/escalate verdict distinct from pass/fail (Q2 verdict ontology), asymmetric error costs favoring false-flags over false-passes, and rubric quality as the binding constraint (Q4: tighten criteria before writing guidelines). ### The foundational 2022 result behind the whole lineage: model-written critiques helped human evaluators find flaws in summaries they would otherwise have missed (including planted flaws in deliberately misleading human-written summaries), and models exhibit a discriminator-critique gap - they can often detect that something is wrong better than they can articulate why. EVIDENCE: Saunders et al. (OpenAI, arXiv 2206.05802): topic-based summarization assistance experiments; critiques helped on both model- and human-written summaries; framework comparing generation, discrimination, and critique ability found 'even large models may still have relevant knowledge they cannot or do not articulate as critiques'; larger models critique and self-refine better. Verified from arXiv abstract. SOURCE: Self-critiquing models for assisting human evaluators (OpenAI) | https://arxiv.org/abs/2206.05802 | 2022-06-12 | primary IMPLICATION: The discriminator-critique gap argues for a two-stage judge: use cheap discrimination (scores/uncertainty) for routing and prioritization even where articulated critique is unreliable, and hold only the articulated-critique layer to the evidence-quoting bar (Q3/Q7). Judge-assisted human review of HUMAN-written work is validated at the root of this lineage, not just AI-output critique. ### A deployed LLM compliance-checker was gamed by its subjects in a single shot: the NeurIPS 2024 author checklist assistant (234 papers) was rated useful by >70% of authors, but the organizers found the system 'not robust to gaming' by authors and concluded it is a poor substitute for human review. EVIDENCE: Goldberg, Ullah, Guyon, Shah et al. (arXiv 2411.03417; NeurIPS blog Dec 2024): GPT-4-turbo checklist verification offered pre-submission; surveys plus qualitative analysis showed genuine improvements in some submissions alongside demonstrated manipulation of the assistant (authors could satisfy the checker without satisfying the standard); access was restricted to authors partly to avoid biasing review. Verified via arXiv abstract and organizer blog summary. SOURCE: Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment | https://arxiv.org/abs/2411.03417 | 2024-11-05 | academic IMPLICATION: Q8's decision-boundary-leak concern has deployment evidence even without repeated feedback cycles: pay/acceptance-motivated subjects will optimize against a visible checker. Defenses (rotating seeded probes, held-out human-graded golden items to detect AutoQA-vs-gold divergence, abstracted rather than checker-revealing feedback) must be designed in from day one, not retrofitted. ### Tools & artifacts - Review Feedback Agent: Open-source multi-LLM pipeline (zou-group/review_feedback_agent, CC BY 4.0 paper) that critiques human peer reviews for vagueness, content misunderstanding, and unprofessionalism, with automated LLM reliability tests gating feedback delivery. | https://github.com/zou-group/review_feedback_agent | Directly reusable architecture for the constructive-feedback channel: feedback taxonomy + fail-closed guardrail tests that feedback must pass before delivery (Q7). - MetaCritique: Framework + dataset (GAIR-NLP/MetaCritique) scoring critique precision/recall/F1 over atomic information units with per-unit rationales. | https://github.com/GAIR-NLP/MetaCritique | Off-the-shelf machinery for holding AutoQA feedback to its own grounding standard and for meta-evaluating critique quality during validation (Q1, Q7). - AbsenceBench: NeurIPS 2025 D&B benchmark (harvey-fin/absence-bench) measuring whether models can identify deliberately omitted content across poetry, sequences, and code diffs. | https://github.com/harvey-fin/absence-bench | Ready-made competence-limit test for the omission/comprehensiveness lane of any candidate judge model before assigning it completeness-verification authority (Q5 lane assignment). - RealCritic: Closed-loop critique-evaluation benchmark (tangzhy/RealCritic): critique quality scored by correction success, with self-/cross-/iterative-critique modes. | https://github.com/tangzhy/RealCritic | Template for validating the AutoQA judge on rationale effectiveness rather than verdict agreement (Q1 meta-evaluation design). - CTRL / Critique-RL / DeepCritic (2025 critic-training lineage): RL-trained critic models: CTRL (arXiv 2502.03492, critique via RL without human critique labels, up to 106% relative code-gen gains), Critique-RL (arXiv 2510.24320, two-stage RL: discriminability then helpfulness, no stronger supervisor), DeepCritic (arXiv 2505.00662, deliberate step-level math critique). | https://arxiv.org/abs/2502.03492 | State of the art for training a dedicated critic without the expensive expert-critique annotation dependency (Q4's aligned-expert concern); Critique-RL's discriminate-first-then-explain staging mirrors the discriminator-critique gap design. - On scalable oversight with weak LLMs judging strong LLMs (Kenton et al., DeepMind, NeurIPS 2024): Protocol comparison (debate vs consultancy vs direct QA) with weaker judges overseeing stronger agents across QA/math/code/logic asymmetries. | https://arxiv.org/abs/2407.04622 | Evidence base for arbitration design (Q6): debate beats consultancy everywhere tested, and open debate makes a weak judge much harder to convince of a wrong answer - a candidate protocol for contested-item escalation. - Minimum Viable Validation Protocol (MVVP): Five-step pre-deployment judge validation checklist from 'Reliability without Validity' (2026): chance-corrected headline stats, AB+BA position swaps, >=3 replicates, >=2 benchmark styles, consistency-bias paradox audit. | https://arxiv.org/abs/2606.19544 | Directly adoptable as the per-project judge-config ship gate the design taxonomy asks for (Q5). - Toloka LLM QA: Production agentic QA system for human annotation submissions: per-metric tool-using agent instances, Pass/Fail/Unable-to-verify verdicts, Socratic annotator feedback. | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | The closest commercial deployment to the proposed AutoQA; its verdict ontology, criterion scoping, and guideline-quality constraint are field-tested reference points (Q2, Q4, Q6). ### Disagreements - Does adding a human to an AI judge help at all? Vaccaro et al. (Nature Human Behaviour 2024) find human-AI combos on average significantly WORSE than the best solo party on decision tasks, while CriticGPT (2024) finds Human+AI teams beyond the model-only Pareto frontier and DeepMind (2025) gets 91.3% hybrid vs 87.7% AI-alone. Reconciliation offered by DeepMind: complementarity only appears with confidence-based routing to a slice where humans genuinely beat the AI and with non-leading assistance formats - but the DeepMind-adjacent follow-up ('Toward Human-AI Complementarity Across Diverse Tasks') found baseline hybridization added only +0.4pp when model confidence failed to identify the complementarity region. The one-touch design's value is conditional, not guaranteed. - What should the human be shown? DeepMind (2025) finds showing the AI's judgment/reasoning/confidence causes over-reliance and that evidence-only assistance is the only safe format, whereas CriticGPT's protocol and the ICLR 2025 feedback agent delivered full natural-language critiques and reported net gains. Task framing differs (assisting a second-party judge vs coaching the first-party author), which suggests the AutoQA should show full critiques to ATTEMPTERS but evidence-only to human ADJUDICATORS - no single source tests both arms. - Are LLM critics better or worse than humans? CriticGPT: model critiques preferred 63% and catch more bugs than paid contractors. MetaCritique: human critiques have much higher factual precision (87.6% vs 71.9% of atomic units factual) while LLM critiques have higher coverage. Both true simultaneously - the disagreement is about which axis (precision vs comprehensiveness) a QA system should privilege, and sources implicitly pick opposite defaults. - Can critics increase comprehensiveness of review? CriticGPT/Saunders show critics make human review MORE comprehensive (catching missed flaws), but AbsenceBench (2025) shows LLMs are structurally near-blind to omissions (56.9 F1 drop). Reconciliation: the critic gains are demonstrated on commission-type errors (inserted/tampered bugs, incorrect statements); no source demonstrates critic gains on omission-type attempter failures, and AbsenceBench predicts they won't appear without checklist/placeholder scaffolding. - What metric certifies a judge? Open-loop verdict-agreement benchmarks (CriticBench 2024, CriticEval 2024, MT-Bench-style agreement) vs RealCritic's closed-loop position that verdict accuracy without correction-effectiveness is 'superficial success' (24.8% right-verdict-wrong-critique), reinforced by RM-NLHF (ICML 2026) and by 'Reliability without Validity' (2026) showing judge rankings flip up to 15 places depending on which benchmark style you trust. ### Gaps - No public post-deployment report from OpenAI on CriticGPT's integration into the production RLHF labeling pipeline (announced intent, June 2024) - no follow-up metrics, and no officially branded successor found through July 2026; the lineage continued in academic work (CTRL, DeepCritic, Critique-RL, MultiCritique) rather than vendor deployment reports. - No published RCT of AI critique/feedback specifically on human DATA-ANNOTATION QA work. Closest analogs are peer review (ICLR 2025) and fact-verification rater assistance (DeepMind 2025). No longitudinal data anywhere on repeat-error-rate reduction per annotator under sustained AI feedback - Q7's efficacy metric has no published precedent to calibrate against. - No measured multi-cycle adversarial study of annotators gaming a deployed critic (Q8's decision-boundary leakage): the NeurIPS'24 checklist gaming is single-shot and author-side; citation-theater/evidence-swap perturbation tests on critics were not found. - Toloka's deployment report - the single closest production analog - publishes its benchmark design (300+ submissions, 20+ projects) but its quantitative precision/recall table is embedded in an image and was not extractable; no independent audit of any commercial annotation-QA LLM system was found. - False-agreement rate (judge and attempter both wrong vs expert panel) is unmeasured in the literature; the CriticGPT 24%-vs-6% flawless-data result is the nearest proxy and is one study, one domain (mostly), one vendor. - Positive-claim/praise verification (is 'accurate/complete/clear' praise warranted?) has no dedicated benchmark or precision/recall measurements - MetaCritique-style AIU machinery has only been applied to flaw-finding critiques. This is a genuine hole given the mission's requirement to evaluate positive statements. - Semantic Scholar returned HTTP 429 during literature-graph search (OpenAlex/Crossref coverage only for that call); one WebSearch call was flagged by a per-call safeguard and was successfully re-run via Exa - no coverage lost, noted for provenance. ### Verifications - CLAIM: Human+critic-model teams occupy a strictly better operating point than either alone: in OpenAI's CriticGPT study, Human+CriticGPT teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (FSBS length penalty), not a fixed property. VERDICT: confirmed | Source verified: arXiv:2407.00215 \"LLM Critics Help Catch LLM Bugs\" (McAleese et al., OpenAI), submitted 2024-06-28 - date matches. Abstract directly confirms: 63% preference for model critiques on naturally occurring LLM errors; models catch more bugs than paid human reviewers; human-machine teams catch similar bug counts to LLM critics while hallucinating less than LLMs alone. Full-text details corroborated via independent secondary coverage: critic-assisted contractors wrote more comprehensive critiques than unassisted contractors while reducing hallucination/nitpick rate relative to the model, described as moving beyond the model-only Pareto frontier; FSBS scores candidates by rm_score + LENGTH_MODIFIER x num_highlights, giving a deployment-time precision/comprehensiveness dial without retraining. Minor reading note: \"strictly better than either alone\" holds in the paper's Pareto sense - teams matched (did not exceed) the model's bug-catch/comprehensiveness while beating its hallucination rate, and exceeded human comprehensiveness; the claim's own wording states exactly this, so no correction needed. Supersession check: no official OpenAI follow-up found; later work (CodeCriticBench 2025; 2026 papers on LLM reviewer overcorrection) extends but does not contradict these findings. - CLAIM: In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong. VERDICT: confirmed | Every quantitative element checks out against the arXiv HTML full text (2510.26518). Dataset: 1,918 expert-labeled tuples - confirmed. AI rater 87.7%, individual humans 75.1%, human majority vote 80.6% - confirmed. Confidence via 50 samples/sentence (avg 33.25 passing format check), described as fairly well calibrated - confirmed. Unassisted hybridization at threshold T=0.62: 89.3% (beta=0.413, p=.012) - confirmed. Low-confidence routed slice (280 items): humans 71.3% vs AI 60.5% (p=.006) - confirmed. The headline 91.3% is specifically the ASSISTED hybrid (AI when confident + evidence-assisted humans on the low-confidence slice), vs 89.3% unassisted hybrid - exactly as the claim words it, so 91.3% is correctly attributed. Evidence-only assistance: 79.3% vs 71.3% baseline when AI correct (beta=0.446, p=.009), no significant harm when AI wrong (64.0% vs 61.5%); paper explicitly calls it \"the only form of assistance that achieves the ideal of helping when correct and not hurting when wrong\" - confirmed. Over-reliance when AI wrong: Evidence&Reasoning&Judgments beta=-0.768 (SE=0.174, z=-4.413, p<.001), ER&J&Confidence beta=-0.636 p<.001, Judgments&Confidence beta=-0.356 p=.042 - the quoted beta=-0.768 matches the Evidence&Reasoning&Judgments condition exactly. Debate was numerically worst (64.9%, only format below baseline) - confirmed. Date: v1 submitted 2025-10-30 as claimed. Currency: a v2 was posted 2026-06-25 and the paper was published at ACM FAccT '26 (DOI 10.1145/3805689.3812308); this is publication/revision, not supersession - no later work contradicting the findings found. DeepMind blog notes ongoing follow-up combining hybridization+assistance on an internal rating task, unpublished as of 2026-07-14. Minor caveat only: whoever cites this should prefer the v2/FAccT version in case numbers shifted slightly in revision (v1 numbers verified here). - CLAIM: In the largest deployed RCT of AI critiquing human evaluative work (ICLR 2025, feedback on >20,000 peer reviews), 27% of reviewers who received LLM feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated LLM reliability tests. VERDICT: confirmed | Source verified directly (arXiv:2504.09737, v1 submitted 2025-04-13 - date matches). Abstract states: Review Feedback Agent deployed at ICLR 2025 as a large randomized controlled study on >20,000 randomly selected reviews; feedback targeted vague comments, content misunderstandings, unprofessional remarks; 27% of reviewers who received feedback updated their reviews; >12,000 suggestions incorporated (12,222 in the paper); +80 words average among updaters; more informative per blinded evaluators; longer author-reviewer discussions; feedback delivered only if it passed a suite of automated LLM reliability tests. Arm sizes corroborated from paper figures: 22,467 selected for feedback vs 22,364 control; 18,946 successfully received feedback, of whom 26.6% updated (the abstract rounds to 27%). Two nuances, neither contradicting the claim: (1) the superlative \"largest deployed RCT of AI critiquing human evaluative work\" is the researcher's framing, not verbatim in the abstract - no larger counterexample found, and it is consistent with the study's scale; (2) not superseded but now formally published: Nature Machine Intelligence, Feb 23, 2026, as \"A large-scale randomized study of large language model feedback in peer review\" (https://www.nature.com/articles/s42256-026-01188-x) - the NMI version is the preferred citation going forward. A related follow-up mixed-methods study on reviewer perceptions exists (arXiv:2602.13817) but does not alter these results. --- ## Domain: grounding-faithfulness *entailment checkers, decomposition, citation evaluation* ### Grounded claim-verification is commoditized but ceilinged: the best model on the LLM-AggreFact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7B model, Bespoke-MiniCheck-7B, at 77.4 average balanced accuracy, with 0.4-0.8B specialized checkers (FactCG, MiniCheck-Flan-T5-L) within ~2.5 points of frontier LLMs like Claude-3.5-Sonnet (77.2) and GPT-4o (75.9). [LOAD-BEARING] EVIDENCE: Fetched the live LLM-AggreFact leaderboard: top-10 shows Bespoke-MiniCheck-7B 77.4, Claude-3.5-Sonnet 77.2, Granite Guardian 3.3 8B 76.5, FactCG-DeBERTa-L (0.4B) 75.6, MiniCheck-Flan-T5-L (0.8B) 75.0, Llama-3.1-405B 74.4. Benchmark measures binary supported/unsupported vs grounding docs. Confirmed independently by Paladin-mini (June 2025 arXiv) citing Bespoke-MiniCheck as leaderboard SOTA. SOURCE: LLM-AggreFact Leaderboard (MiniCheck project) | https://llm-aggrefact.github.io/ | 2025 (accessed 2026-07-14) | primary IMPLICATION: A cheap, deterministic, reproducible entailment-check stage (sub-1B to 7B checker) for 'is this attempter statement supported by the cited evidence' is off-the-shelf and costs ~1/100 of frontier calls - but ~22% claim-level error means item verdicts cannot be a naive AND over claim checks; error-tolerant aggregation and confidence-routing to humans are structurally required. ### On adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on FaithBench, prior detectors hit ~50% accuracy (negligible), HHEM-2.1-Open 66.7% and Bespoke-MiniCheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while FaithJudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 F1. [LOAD-BEARING] EVIDENCE: Fetched full text of arXiv:2505.04847v2 (Vectara, EMNLP 2025 Industry Track, v2 Nov 2025). Table 1: baseline detector numbers; Table 2: FaithJudge results; FaithJudge beats the FACTS Grounding judging prompt head-to-head on all four RAGTruth/FaithBench splits (e.g., FaithBench 70.8 vs 54.3 F1). Documented failure modes: judge underpredicts hallucinations for some generator families; 'benign'/'questionable' ternary labels classified unreliably so only binary is used; specificity drops as more in-context examples are added. SOURCE: Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards (FaithJudge) | https://arxiv.org/abs/2505.04847 | 2025-11 | academic IMPLICATION: The strongest 2025-measured grounding-judge architecture is exactly the AutoQA rubric-compilation contract: a per-project pool of human-adjudicated exemplar judgments (annotated spans + labels on comparable items) injected few-shot, not zero-shot judging and not a fixed fine-tuned checker. Also: severity-graded verdicts (benign/questionable) are where both humans and judges lose reliability - keep the machine verdict binary and treat severity as a separate, human-anchored layer. ### Claim decomposition helps weak verifiers but actively degrades strong ones: with MiniCheck as verifier on WiCE, no-decomposition scores 80.01 balanced accuracy while FActScore-style atomic decomposition drops it to 71.11; with the weaker AlignScore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity. [LOAD-BEARING] EVIDENCE: Fetched full text of 'Decomposition Dilemmas' (NAACL 2025, arXiv:2411.02400). Four-way decomposition-error taxonomy from manual inspection: (A) omission of context/logical relations, (B) ambiguity (unclear pronouns/references), (C) over-decomposition, (D) alteration of original meaning. FActScore-style atomicity produces the most over-decomposition errors; VeriScore-style tends to omit context. On FELM, decomposition raised MiniCheck F1 48.1->~68 but dropped GPT-4o-mini F1 71.6->54.3. SOURCE: Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? | https://aclanthology.org/2025.naacl-long.320/ | 2025-05 | academic IMPLICATION: Directly answers the granularity design question: there is no universally-best decomposition level. Granularity must be tuned per verifier strength and per item complexity - atomic-claim pipelines with a strong judge are measurably WORSE than judging larger spans. The 'checklist myopia' risk is real and has a named mechanism (context omission + meaning alteration). Budget a granularity-calibration step per project rather than fixing atomic decomposition in the foundation. ### Decomposition and decontextualization are in direct tension - isolating atomic facts strips the context needed to verify them, while adding context back creates multi-fact claims where the verifier may credit or penalize the wrong content; DnDScore resolves this by verifying the original subclaim WITH the added information treated as context rather than as content to be verified. EVIDENCE: Fetched ACL Anthology abstract of DnDScore (Wanner, Van Durme, Dredze; EMNLP 2025 main, pp. 23609-23626). The paper evaluates combinations of decomposition, decontextualization, and verification strategies and finds the strategy choice materially changes factuality scores. SOURCE: DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation | https://aclanthology.org/2025.emnlp-main.1205/ | 2025-11 | academic IMPLICATION: When the AutoQA checks an attempter's sentence, the verification unit should be 'claim + explicit context annotations' with the verifier told which part is under test - not a free-floating atomic rewrite. This is a concrete spec for the claim-extraction layer's output schema. ### Verifiability triage before entailment is established machinery: VeriScore extracts and scores ONLY verifiable claims because FActScore/SAFE 'assume that every claim is verifiable', which breaks on real long-form text containing opinions and unverifiable content; FactBench's VERIFY pipeline (ACL 2025) further labels content units supported/unsupported/UNDECIDABLE against retrieval, with 4,467 human-annotated units released for validation. EVIDENCE: Fetched arXiv:2406.19276 abstract (VeriScore, EMNLP Findings 2024): human evaluation found VeriScore's extracted claims 'more sensible' than competitors across 8 long-form tasks. Fetched launchnlp/FactBench GitHub (ACL 2025 paper 2410.22257): three-way supported/unsupported/undecidable labeling according to retrieval results, benchmarked against FActScore, SAFE, Factcheck-GPT. SOURCE: VeriScore (EMNLP Findings 2024) + FactBench/VERIFY (ACL 2025) | https://arxiv.org/abs/2406.19276 | 2024-06 (VeriScore, foundational); 2025-07 (FactBench ACL 2025) | academic IMPLICATION: The claim ontology question has a field-standard answer: a front-stage classifier that routes each attempter statement into {verifiable claim, opinion/unverifiable, undecidable-given-evidence} BEFORE any entailment check. Opinion-class statements get consistency-with-cited-evidence checks at most, never truth verdicts; 'undecidable' is a first-class output, not a forced pass/fail. ### Holistic LLM judging of context-grounded outputs is far weaker than claim-level checking: on ContextualJudgeBench (2,000 pairs, 8 splits over RAG/summarization with conditional criteria like 'faithfulness first, then completeness'), the best of 20 judge models tested (OpenAI o1) barely reaches 55% consistent accuracy. EVIDENCE: Fetched arXiv:2503.15620 (Salesforce; ACL 2025). 11 specialized judge models + 9 general-purpose models evaluated; the benchmark encodes criteria hierarchies and finds contextual assessment poses a significant challenge even to SOTA models. Contrast: the same model class hits 77-84% on single-claim grounding tasks (LLM-AggreFact, FaithJudge). SOURCE: Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings | https://arxiv.org/abs/2503.15620 | 2025-03 | academic IMPLICATION: Do not architect the AutoQA as one holistic 'judge this writeup against the rubric' call - measured reliability roughly halves versus decomposed per-criterion, per-claim checks. Criteria priority ordering (which humans dispute) must be compiled explicitly into the pipeline structure, because judges fail at applying conditional criteria internally. ### Grounding judges have a measured agreement-default asymmetry: in Google's FACTS Leaderboard (Dec 2025), grounding judges score ~85 F1 on the positive (grounded) class but only ~46 F1 on the negative (ungrounded) class, and the grounding metric is gameable by vague, short responses that avoid unsupported claims - countered by a mandatory eligibility gate that scores non-responsive answers as inaccurate. [LOAD-BEARING] EVIDENCE: Fetched full text of arXiv:2512.10791v1 (The FACTS Leaderboard, Google, Dec 2025). Best judge-prompt combos reach only ~65 macro-F1 (gemini-2.5-flash + v2 prompt: 65.33) against a 320-example human-adjudicated held-out set; positive-class ~85 vs negative-class ~46 F1; eligibility check explicitly added because 'grounding metrics can be hacked via vague, short responses'. SOURCE: The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality | https://arxiv.org/abs/2512.10791 | 2025-12 | academic IMPLICATION: Two foundational mandates: (1) the sycophancy/agreement-default bias on positive claims is quantified and large - the AutoQA must be evaluated on per-class recall (especially fail-class recall at fixed prevalence), never single-number accuracy; (2) an eligibility/responsiveness gate must precede grounding checks or attempters can pass by writing vacuous hedged annotations - the exact adversarial dynamic the design taxonomy flags. ### Cross-family judge ensembles are the standard mitigation for self-preference: FACTS Grounding v1 (Jan 2025) aggregated three frontier judges from different families (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) explicitly because 'models are biased towards favorably judging their own outputs', and Grounding v2 (Dec 2025) kept a two-family ensemble (Gemini 2.5 Flash + GPT-5). EVIDENCE: Fetched arXiv:2512.10791v1 which states the v2 judge-ensemble rationale and composition; v1 design documented in arXiv:2501.03200 (Jan 2025) and the DeepMind blog (Dec 17, 2024). FACTS Parametric suite separately validated that a single-judge setup preserved rankings vs a mixed panel (Gemini 2.5 Pro, o3, Grok 4) - i.e., ensemble necessity is task-dependent and testable. SOURCE: FACTS Grounding Leaderboard (v1 paper + v2 in FACTS Leaderboard) | https://arxiv.org/abs/2501.03200 | 2025-01 (v1); 2025-12 (v2) | academic IMPLICATION: When attempters critique outputs from a given model family, the AutoQA judge should default to a different family or an ensemble spanning families; but ensemble overhead can be dropped where a single-judge-vs-panel ranking-preservation test passes - a cheap per-project validation, not a dogma. ### Span-restricted NLI citation checking (the ALCE-style 'does the cited span entail the sentence' standard) is judged a suboptimal proxy by 2025 work: CiteEval (ACL 2025) argues citation quality must be evaluated against the FULL retrieval context, user query, and generated text - not cited sources alone - and its model-based CiteEval-Auto metrics correlate better with human judgments on the multi-domain CiteBench than NLI-based metrics. EVIDENCE: Fetched arXiv:2506.01829 (CiteEval, ACL 2025 main). Explicitly frames binary/ternary NLI vs cited sources as 'a suboptimal proxy for citation evaluation'. Corroborated direction: CiteGuard (ACL 2026, aclanthology.org/2026.acl-long.282) reframes citation evaluation as attribution alignment and notes 'the reliability of LLM-as-a-Judge alone is also in doubt' for citation judging. SOURCE: CiteEval: Principle-Driven Citation Evaluation for Source Attribution | https://arxiv.org/abs/2506.01829 | 2025-06 | academic IMPLICATION: Evidence-closure design: checking attempter claims only against the spans they cite will miss the dominant human failure of citing the wrong/incomplete evidence when better evidence existed in scope. The AutoQA needs two distinct checks - (a) does cited evidence entail the claim (span-closed), and (b) is the citation the right one given the full in-scope corpus (context-closed) - with (b) also catching true-but-wrongly-cited claims. ### Fact-verification LLMs are brittle to semantically-minor input perturbations: FactEval (NAACL 2025) tested 17 realistic word- and character-level perturbations plus 4 subpopulations on FEVER across zero-shot/few-shot/CoT setups and found LLMs 'brittle to small input changes' with performance varying across subpopulations. EVIDENCE: Fetched ACL Anthology page for 2025.naacl-long.534 (Mamta & Cocarascu). Abstract is qualitative; per-perturbation numbers are in the full paper. Consistent with FaithJudge's finding that adding more in-context examples shifts sensitivity/specificity, and with FACTS' judge-prompt sensitivity (macro-F1 varies by prompt version). SOURCE: FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models | https://aclanthology.org/2025.naacl-long.534/ | 2025-04 | academic IMPLICATION: A perturbation-robustness harness (paraphrase, reorder, typo-level noise on both claims and evidence) belongs in the per-project ship gate; verdicts that flip under meaning-preserving edits should be auto-routed to human review rather than delivered. ### 2026 SOTA joins verdict and explanation in one cheap model: FaithLens (ACL 2026 Findings) is an 8B faithfulness-hallucination detector that jointly outputs a binary prediction AND an explanation, trained via cold-start fine-tuning on filtered synthetic data plus rule-based RL rewarding both prediction correctness and explanation quality, and outperforms GPT-5.2 and o3 across 12 tasks. EVIDENCE: Fetched ACL Anthology entry 2026.findings-acl.689 (Si et al., Findings of ACL 2026, pp. 14068-14099). Abstract claims superiority over GPT-5.2/o3 and 'a distinctive balance of trustworthiness, efficiency, and effectiveness'; per-task numbers require the full PDF. SOURCE: FaithLens: Detecting and Explaining Faithfulness Hallucination | https://aclanthology.org/2026.findings-acl.689/ | 2026-07 (ACL 2026) | academic IMPLICATION: Constructive-feedback generation and grounding verdicts do not need separate machinery: an explanation-rewarded detector produces the evidence-citing rationale as a first-class output, at 8B cost. This is the current-generation template for the AutoQA's 'every criticism must carry its evidence' contract, and shows explanation quality can be an explicit training/reward target rather than a post-hoc add-on. ### RAGAS-style prompt-chain faithfulness scoring materially underperforms finetuned/frontier judges on hallucination detection: on the 15K-sample HaluBench, RAGAS Faithfulness scored 66.9% accuracy versus 87.4% for finetuned Lynx-70B and 86.5% for GPT-4o; Patronus has since shipped Lynx 2.0 (8B, long-context, 8 hallucination subtypes including coreference and calculation errors). EVIDENCE: Confirmed via Patronus AI announcement and Lynx paper coverage (July 11, 2024) with the HaluBench table (Lynx-70B 87.4, GPT-4o 86.5, GPT-4-Turbo 85.0, Llama-3-70B 80.1, RAGAS 66.9); Lynx 2.0 details from Patronus docs. Caveat: vendor-reported numbers on the vendor's own benchmark, though HaluBench is public on HuggingFace. RAGAS remains the most widely adopted OSS RAG-eval framework per 2026 practitioner surveys (atlan.com RAG evaluation guide, 2026). SOURCE: Patronus AI Lynx / HaluBench results | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | 2024-07 (Lynx 1.0, foundational); Lynx 2.0 later update | practitioner IMPLICATION: Do not build the grounding stage on RAGAS-style statement-extraction prompt chains despite their ubiquity in 2026 tooling - the popularity/accuracy gap is ~20 points. If an off-the-shelf component is wanted, current-generation finetuned detectors (Bespoke-MiniCheck, Lynx 2.0, Granite Guardian, FaithLens) dominate at equal or lower cost. ### Tools & artifacts - Bespoke-MiniCheck-7B / MiniCheck family: Specialized grounded-claim fact-checking models (0.4B-7B), SOTA on LLM-AggreFact at 77.4 bacc; available on HuggingFace and Ollama, ~400x cheaper than GPT-4-class verification | https://github.com/Liyan06/MiniCheck | Default candidate for the cheap deterministic claim-vs-evidence entailment stage of the AutoQA - FaithJudge (Vectara): Open-source LLM-as-judge framework that anchors judgment with human-annotated peer examples on the same source; repo actively pushed through Nov 2025 | https://github.com/vectara/faithjudge | The measured-best architecture pattern for the AutoQA's per-project exemplar-anchored judge; directly reusable code - FACTS Grounding public set + FACTS Leaderboard: 860 public grounding examples (32k-token docs) plus Google's Dec 2025 four-suite factuality benchmark with Kaggle-held private splits | https://huggingface.co/datasets/google/FACTS-grounding-public | Source of judge-prompt designs, the eligibility-gate pattern, and human-adjudicated grounding labels for meta-evaluating our own judge - FactBench / VERIFY: ACL 2025 factuality pipeline with supported/unsupported/undecidable labeling and 4,467 human-annotated content units released | https://github.com/launchnlp/FactBench | Template + validation data for the verifiability-triage front stage (fact vs opinion vs undecidable) - VeriScore + VeriFastScore: Verifiable-claim-only extraction/scoring for long-form text (EMNLP Findings 2024) and its single-model accelerated successor (EMNLP Findings 2025) that collapses extraction+verification into one fine-tuned pass | https://github.com/Yixiao-Song/VeriScore | Claim-extraction spec that skips opinions; VeriFastScore is the cost-reduction pattern if per-item pipeline latency matters - HaluBench + Lynx 2.0 (Patronus): 15K-sample hallucination detection benchmark with hard/ambiguous cases, plus an 8B long-context finetuned detector covering 8 hallucination subtypes | https://huggingface.co/PatronusAI | Seeded-set material for measuring our checker's fail-class recall; Lynx 2.0 is an off-the-shelf detector option - ContextualJudgeBench (Salesforce): 2,000-pair benchmark for LLM judges in RAG/summarization settings with conditional criteria hierarchies (ACL 2025) | https://arxiv.org/abs/2503.15620 | Meta-evaluation instrument for any holistic judge configuration we consider; quantifies why we should not use one - CiteBench / CiteEval-Auto: Multi-domain human-annotated citation-quality benchmark and model-based metrics that evaluate citations against full retrieval context (ACL 2025) | https://arxiv.org/abs/2506.01829 | Blueprint for the context-closed 'is this the right citation given everything in scope' check, beyond span-closed NLI - HHEM-2.1-Open + Vectara Hallucination Leaderboard: Open small hallucination-detection model and the longest-running (2023-present, refreshed Nov 2025 with 7,700 articles up to 32k tokens) HHEM-scored LLM leaderboard | https://huggingface.co/spaces/vectara/leaderboard | Cheapest available checker tier, but its measured 66.7 bacc on hard cases (FaithBench) bounds where it can be trusted - screening, not verdicts ### Disagreements - HHEM's fitness as an automated grounding judge: Vectara's own next-generation hallucination leaderboard (blog, 2025-11-19) still uses commercial HHEM as the sole scorer and reports rates like Gemini-2.5-flash-lite 3.3% vs Gemini-3-pro 13.6%, while Vectara's own research paper (FaithJudge, EMNLP Industry 2025) shows HHEM-2.1-Open at only 66.7% balanced accuracy on FaithBench - near-random on hard hallucinations - and builds an LLM-judge replacement because of 'limitations observed in current hallucination detection methods'. Detector choice materially changes model rankings; automated hallucination-rate leaderboards from different detectors disagree and cannot be treated as interchangeable ground truth. - Atomic decomposition: the FActScore/SAFE/RAGAS lineage (2023-2024) treats finer-grained atomic claims as strictly better for verification, but Decomposition Dilemmas (NAACL 2025) shows decomposition DEGRADES strong verifiers (80.0 -> 71.1 bacc on WiCE with MiniCheck) and DnDScore (EMNLP 2025) shows atomization and decontextualization actively conflict. The field has not converged: long-form factuality pipelines still default to atomic claims while the 2025 analysis papers say granularity must be adaptive. - Specialized small detectors vs frontier LLM judges: LLM-AggreFact (through 2025) and FaithLens (ACL 2026) show small specialized models beating frontier LLMs (Bespoke-MiniCheck-7B > Claude-3.5/GPT-4o; FaithLens-8B > GPT-5.2/o3), but FaithJudge (2025) shows frontier reasoning judges WITH human-annotated exemplars beating all specialized detectors on adversarial sets (84.0 vs 71.2 bacc). Both are right in their regime: fixed checkers win zero-shot on broad distributions; exemplar-anchored frontier judges win on hard, domain-specific cases - which is an argument for a two-tier architecture rather than either alone. - Citation-support standard: ALCE-style span-closed NLI citation precision/recall remains the de-facto automated standard, but CiteEval (ACL 2025) calls NLI-vs-cited-spans 'a suboptimal proxy' and CiteGuard (ACL 2026) additionally doubts LLM-as-judge reliability for citation evaluation - i.e., the two dominant automated approaches each have a 2025-2026 paper attacking them, with no settled successor. ### Gaps - No published measurements of run-to-run verdict flip rates (self-consistency) for grounding checkers or LLM faithfulness judges on identical inputs - the taxonomy's 'flip rate: noise vs underspecification signal' question has no literature anchor found; specialized classifiers (MiniCheck/HHEM) are deterministic by construction but that property is unstated in evaluations. - Positive-claim verification of EVALUATIVE statements (warranted vs empty praise, 'stated severity matches evidence strength' calibration claims) is essentially unstudied: the entire 2024-2026 literature verifies factual/descriptive claims against documents; no benchmark tests whether a checker can verify claims like 'this response is complete' or 'this citation is the strongest available'. The AutoQA's core use case - verifying human evaluative writeups about AI outputs - is a domain transfer no found benchmark covers. - No dedicated 2025-2026 study found on evidence-swap/citation-theater probes for verifiers (does the verdict flip when cited evidence is replaced with equally-plausible irrelevant text?). FactEval perturbs claims, not evidence; FACTS documents gaming by response vagueness, not by decorative citation. This test would need to be built in-house. - Aggregation from claim-level verdicts to item-level pass/fail with severity weighting is under-researched: FActScore-style %-supported and FACTS' all-or-nothing eligibility+grounding are the only documented rules; RAGTruth/FaithBench severity taxonomies (benign/questionable/unwanted, evident/subtle) exist as LABELS but FaithJudge found judges classify the middle severity classes unreliably and retreated to binary - no validated severity-weighted aggregation scheme found. - Judge-vs-adjudicated-human agreement ceilings per axis type are only available for grounding (~65 macro-F1 in FACTS on held-out human labels; 84 bacc for FaithJudge on FaithBench) - no comparable 2025-2026 numbers found for instruction-compliance or tone/style axes from the faithfulness literature, so the lane-assignment decision (judge-autonomous vs assisted vs human-only per axis) lacks published anchors outside factual grounding. ### Verifications - CLAIM: Grounded claim-verification is commoditized but ceilinged: the best model on the LLM-AggreFact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7B model, Bespoke-MiniCheck-7B, at 77.4 average balanced accuracy, with 0.4-0.8B specialized checkers (FactCG, MiniCheck-Flan-T5-L) within ~2.5 points of frontier LLMs like Claude-3.5-Sonnet (77.2) and GPT-4o (75.9). VERDICT: confirmed | Fetched https://llm-aggrefact.github.io/ on 2026-07-14; every number matches exactly: Bespoke-MiniCheck-7B 77.4 (rank 1), Claude-3.5 Sonnet 77.2, Granite Guardian 3.3 8B 76.5, gpt-4o-2024-05-13 75.9, FactCG-DeBERTa-L (0.4B) 75.6, MiniCheck-Flan-T5-L (0.8B) 75.0, Llama-3.1-405B 74.4. Benchmark is 11 datasets of grounded factuality (supported/unsupported vs grounding docs), avg balanced accuracy - as claimed. The \"within ~2.5 points\" framing is accurate and even conservative: FactCG (75.6) is only 1.6 below Claude-3.5-Sonnet and above GPT-4o. Supersession search found nothing beating 77.4: ACV (May 2026) reports 76.5 training-free; an ACL 2026 paper ranks second to a post-trained metric; Paladin-mini beats Bespoke-MiniCheck only on its own separate benchmark subsets, not LLM-AggreFact overall. Caveats on the interpretive \"ceilinged\" framing: (1) the leaderboard's frontier entries are 2024-era models (Claude-3.5-Sonnet, gpt-4o-2024-05-13, Llama-3.1-405B) - no Claude 4/GPT-5-class/o3 entries, so \"specialized ~ frontier\" reflects 2024 frontiers; (2) \"Verifying the Verifiers\" (arXiv 2506.13342, June 2025) finds ~16% of benchmark labels ambiguous/incorrect and that few-shot frontier LLMs reach top-tier performance, suggesting the ~77 plateau is partly benchmark label noise rather than a pure task ceiling; (3) \"Verify with Caution\" (arXiv 2501.14883) shows models with similar aggregate BAcc make very different instance-level predictions. None of these contradict the stated facts. - CLAIM: On adversarially-hard hallucination sets, specialized detector"}, {"slug": "critic-and-gapfill", "title": "Completeness Critic & Gap-Fill Reports", "category": "Research working papers", "summary": "Adversarial audit of the original sweep, plus psychometrics, legal, content moderation, and feedback-science gap fills.", "tags": ["critique", "psychometrics", "law", "feedback"], "view": "documents/view/critic-and-gapfill.html", "raw": "documents/raw/research/critic-and-gapfill.md", "text": "# Completeness critic ## Psychometrics and rater-training science (educational/language assessment, many-facet Rasch measurement, generalizability theory). The sweep's 9 domains are all 2020s LLM/annotation literature, but human-rater variance has 40+ years of quantitative treatment: rater severity/leniency modeling, rater drift over scoring sessions, frame-of-reference vs rater-error training effect sizes, G-theory decomposition of variance into rater/item/criterion facets. This directly answers taxonomy Q1's variance-attribution question ('would perfect rubrics leave variance intact?') and Q9's reviewer-skill-decay question with measured effect sizes, instead of leaving them as unanchored pilots. WHY: The build order and budget split hinge on whether reviewer variance is rubric-fixable or rater-intrinsic (root cause 1 vs 2-3). Rasch/G-theory literature has actual numbers on how much variance rubric operationalization and rater training each remove, and validated statistical machinery (rater severity as a modeled parameter, not noise) that the entire LLM-judge literature reinvents badly. Ignoring it risks re-running expensive in-house pilots for questions psychometrics settled in the 1990s-2010s. QUERY: many-facet Rasch measurement rater severity drift frame-of-reference training effect size rubric calibration generalizability theory variance components raters ## Legal/regulatory constraints on algorithmic management of the attempter workforce, plus data-quality standards. The AutoQA issues automated pass/fail verdicts affecting paid workers' income and proposes process telemetry (timing, edit traces) as a provenance contract - this is squarely regulated territory: GDPR Article 22 (automated decisions with significant effects require human review - which may legally mandate the one-human-touch the design treats as optional), the EU Platform Work Directive (transposition deadline December 2026, explicitly governs algorithmic management, worker access to decision logic, and human oversight of automated account/pay decisions), and ISO/IEC 5259 (data quality for analytics and ML, published 2024-2025) which large buyers may contractually require. Zero regulatory/standards sources appear in any domain. WHY: This is a foundational-architecture constraint, not compliance trim: if GDPR Art 22 / Platform Work Directive apply to attempters in the EU, 'zero-touch plus randomized audits' (taxonomy Q6's fork) may be illegal for adverse verdicts, contest/appeal channels become mandatory rather than optional, and telemetry collection needs a lawful basis. Discovering this after the philosophy document ships forces a redesign of routing and consequences. QUERY: EU Platform Work Directive 2024 algorithmic management automated decision human oversight crowdwork data annotation GDPR Article 22 ISO 5259 data quality ## Content-moderation QA programs - the largest deployed analog of humans-reviewing-human-judgment at scale. Meta/TikTok/Google have run decade-old programs auditing moderator decisions: golden-set seeding into live queues, audit sampling rates, overturn/appeal-rate monitoring, QA-of-the-QA layers, measured reviewer base-rate and fatigue effects, and published transparency-report methodology plus academic studies (e.g., on moderator agreement and audit design). The industry-practice domain covered only AI-training-data vendors; this adjacent industry has already answered several taxonomy Q6/Q9 questions operationally (seeded known-verdict items, easy-case dilution, de-graduation triggers). WHY: It is the only place where the exact mechanisms the design speculates about - blind gold injection into live streams, closed-loop routing on overturn rates, queue-composition effects on reviewer harshness - have run in production for years with public postmortems and litigation-disclosed internals. Cheaper to import their measured failure modes than rediscover them. QUERY: content moderation quality assurance program golden set audit sampling overturn rate reviewer agreement queue composition fatigue Meta TikTok transparency methodology ## Feedback-efficacy science from education and organizational psychology. Every domain independently reports 'no evidence that QA feedback changes attempter behavior' as a gap, yet feedback-intervention research is a mature field: Kluger & DeNisi's feedback intervention theory (a third of feedback interventions REDUCE performance, with known moderators), formative-assessment literature on feedback specificity/timing, and workplace studies on feedback under pay-linked evaluation (which predicts gaming/monoculture, taxonomy Q8). The sweep searched only for annotation-specific longitudinal studies and found none - the general literature was never consulted. WHY: Taxonomy Q7's entire branch (minimal feedback unit, feedback-vs-verdict-only comparison, when to conclude feedback is decoration) currently has zero evidential anchor, so the design would commit to feedback machinery on faith. FIT's moderators (task-focused vs self-focused feedback, specificity, normative comparison) give testable priors for the feedback 4-tuple design and for predicting when feedback backfires - before spending pipeline cost on self-verified coaching. QUERY: feedback intervention theory Kluger DeNisi meta-analysis feedback specificity performance improvement gaming incentivized workers longitudinal repeat error rate ## Weak spots - The single most load-bearing transfer assumption is unverified everywhere: every quantitative judge result in the sweep is measured on grading MODEL outputs, while the AutoQA grades HUMAN evaluative writeups (grounding of praise, severity calibration, evidence-conclusion entailment). All nine domains flag this and then the numbers get cited anyway - no domain even found a partial-transfer experiment, so the anchor numbers (77.4 bacc grounding, 55-56% rubric-hard, kappa 0.28) may be systematically off in either direction for the actual workload. - The headline meta-evaluation figures (33-41pp kappa deflation, 14-15 position ranking flips, consistency-bias paradox) all trace to one un-peer-reviewed, 0-citation June 2026 preprint ('Reliability without Validity', arXiv 2606.19544), cited independently by four domains as if it were four sources. Cross-domain repetition is masquerading as corroboration. - Much of the frontier-2026 and rubrics-recent evidence was verified only at abstract/title level (rubric co-evolution cluster, Auditing-by-Re-Solving, Evaluation Faking, Trust or Escalate details, Toloka's precision/recall table locked in an image) - several architecturally decisive claims (agent-in-environment verifiers beat text-only 10x cheaper; consequence-framing softens verdicts with zero CoT trace) rest on numbers nobody in the sweep recomputed or read in full. - The spam-economics narrative (Scale/Bulba, paid gibberish for 11 months, account black market) is sourced via Futurism quoting Inc. documents the researchers could not access (403), is disputed on record by Scale, and dates to the pre-LLM-judge QA era - yet it anchors the taxonomy Q8 conclusion that content-based detection has 'already lost'. Westwood's agent study is the solid leg; the Scale story is journalism-of-journalism. - CriticGPT's 24%-vs-6% flawless-slice result - the best evidence for the whole human+critic premise - is one study, one vendor, mostly one domain (code), with no post-deployment report and no replication in 24+ months. The critique-models domain flags this but the claim still functions as the design's cornerstone. - Several 'disagreement resolutions' offered across domains are inferred reconciliations, not tested ones: verbosity bias as 'a controllable design variable', RaR-vs-RubricBench reconciled by 'grounding' as the variable, atomicity 'helps with per-criterion calls, hurts in long checklists' (explicitly labeled 'inferred, not directly tested'), and 'show critiques to attempters but evidence-only to adjudicators'. If the synthesis treats these as findings, the design inherits speculation as doctrine. - The premise metric is missing on both sides: no measured Krippendorff's alpha exists for the actual human reviewer population this system replaces (vendors publish only thresholds), and none was gathered in-house per the mission context. The system is being designed against 'unexplainable variance' that has never been quantified - the variance-attribution pilot in taxonomy Q1 is not optional and no sweep evidence substitutes for it. - Judge-model evidence is stale relative to the decision date: no verified July-2026 leaderboard for current frontier models (both major leaderboards unfetchable), so any per-model judge-selection guidance in the synthesis is anchored to early-2026-or-older cohorts. - Positive-claim/praise verification - an explicit mission requirement - has literally zero direct evidence in any domain (all six domain gap-lists say so). The synthesis has no precision/recall prior at all for the 'is this praise warranted' machinery and should present it as a from-scratch in-house measurement, not extrapolate from flaw-detection numbers, especially given AbsenceBench predicts critics are near-blind to omissions (empty praise is an omission-shaped failure). - Non-US/non-English practice was never searched: the Chinese data-labeling industry (Baidu/Alibaba crowdsourcing QA, government-backed labeling bases) and Japanese/Korean/Indian vendor practice are absent, so 'industry practice' claims generalize from a US/EU vendor sample. # Gap-fill reports ## Area: Psychometrics and rater-training science (educational/language assessment, many-facet Rasch measurement, generalizability theory). The sweep's 9 domains are all 2020s LLM/annotation literature, but human-rater variance has 40+ years of quantitative treatment: rater severity/leniency modeling, rater drift over scoring sessions, frame-of-reference vs rater-error training effect sizes, G-theory decomposition of variance into rater/item/criterion facets. This directly answers taxonomy Q1's variance-attribution question ('would perfect rubrics leave variance intact?') and Q9's reviewer-skill-decay question with measured effect sizes, instead of leaving them as unanchored pilots. ### Adjusting for rater severity/centrality with a Many-Facet Rasch Model (MFRM) materially changed which AI systems ranked best in the OpenAI RLHF summarization dataset: trained raters' agreement was only QWK .31-.50, and MFRM-adjusted scores flipped raw-mean rankings so two human-feedback policies rose above human-written reference summaries. EVIDENCE: Casabianca & Beiting-Parrish (LAK 2026 LLM Psychometrics workshop) fit MFRM to 6,312 ratings from 15 trained raters on 639 summaries across 19 policies. Full text opened and confirmed: raters R02/R08/R06 systematically lenient, R04/R10/R12 severe, R10 aberrantly central; raw-score rankings 'misrepresent the relative standing of models.' Paper also specifies the estimability requirement: 3-5 ratings per output with rater-item overlap/linkage, and proposes percentile-based flagging (top/bottom 2.5%) for rater remediation. SOURCE: Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach (arXiv 2602.22585) | https://arxiv.org/html/2602.22585v1 | 2026-02 | academic IMPLICATION: AutoQA should treat each human reviewer's severity, centrality, and criterion-specific thresholds as modeled parameters, not noise: use adjusted (fair) scores for pass/fail decisions and use per-rater parameter estimates as the constructive-feedback channel. This forces a data-collection constraint into the foundation: some deliberate rater-item overlap must be designed in, because a single-review-per-item regime with disjoint rater pools is statistically unidentifiable. ### Rater training on real scoring behavior improves within-rater consistency but does NOT equalize severity: after formal training of 16 raters in an operational ESL writing exam, significant between-rater severity differences remained even though consistency improved for most raters. EVIDENCE: Weigle 1998 (Language Testing, 363 citations; abstract opened and confirmed verbatim): 'rater training is more successful in helping raters give more predictable scores (i.e., intra-rater reliability) than in getting them to give identical scores (i.e., inter-rater reliability).' This is the canonical, repeatedly replicated finding of the language-testing rater literature (Lumley & McNamara 1995 same conclusion via MFRM). SOURCE: Using FACETS to model rater training effects (Weigle, Language Testing 15(2)) | https://doi.org/10.1177/026553229801500205 | 1998-04 | academic IMPLICATION: Directly answers taxonomy Q1: perfect rubrics plus training will still leave stable between-reviewer severity variance intact. Budget rubric/training work for what it actually buys (self-consistency, fewer misfitting raters) and route severity alignment to statistical adjustment in the AutoQA layer instead of more calibration meetings. ### Frame-of-reference (FOR) training, the best-validated rater-training method, has a meta-analytic effect on rating accuracy of about Cohen's d = 0.50 (moderate) - down from the d = 0.83 estimated by the earlier, smaller Woehr & Huffcutt 1994 meta-analysis - i.e., training helps but removes nowhere near all rater error. EVIDENCE: Roch, Woehr, Mishra & Kieszczynska (J. Occup. Organ. Psychol., 2012; abstract opened) is the updated meta-analysis with 4x the studies of Woehr & Huffcutt 1994; it finds FOR training effective, with Borman's differential accuracy and behavioural accuracy most improved. The d=0.50 overall figure confirmed via two independent citing sources (Tsai et al. 2019 SMU full-text PDF: 'medium-to-large effect on improving rating accuracy (d = 0.50, Roch et al., 2012; d = 0.83, Woehr & Huffcutt 1994)'; Loignon et al. 2017). Note: classic 'rater error training' (teaching raters to avoid halo/leniency) reduces those errors but can reduce accuracy - FOR (anchoring raters to a shared performance theory with practice+feedback) is the variant that works. SOURCE: Rater training revisited: An updated meta-analytic review of frame-of-reference training (Roch et al.) | https://doi.org/10.1111/j.2044-8325.2011.02045.x | 2012-06 | academic IMPLICATION: Sets the ceiling on root-cause-1 fixes: even the best-in-class training intervention delivers ~half a standard deviation of accuracy gain. If the AutoQA program funds training, fund FOR-style training (shared exemplars, practice ratings, feedback against reference judgments) - which the AutoQA itself can generate as a byproduct - and plan for the residual variance to be caught by per-item verification. ### G-theory decompositions of rated writing assessments show the rater MAIN effect (global severity) is a minor variance source while high-order interactions dominate: in a 120-student EFL study, student-by-task-by-method (31.8%), student-by-rater-by-task-by-method (26.5%), and student-by-rater-by-method (17.6%) interactions were the largest components, and 99.2% of error variance came from the student-by-rater interaction. EVIDENCE: Khodi 2021 (Language Testing in Asia, open access; PDF opened and percentages confirmed from abstract/results). Consistent with the broader G-theory writing literature (e.g., Gao, Brennan & Guo's GMAT AWA report, 2015): rater main-effect variance is small once training exists; person-by-rater and residual interaction variance is what limits reliability. Interpretation: most rater-related variance is idiosyncratic per-item disagreement (which rater on which submission on which criterion), not a stable harshness offset. SOURCE: The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution (Khodi) | https://doi.org/10.1186/s40468-021-00134-5 | 2021-11 | academic IMPLICATION: This is the strongest psychometric argument for the AutoQA's core design: global severity calibration and better rubrics attack only the small main-effect components. The dominant variance lives at the rater-times-item-times-criterion level, which can only be attacked by evidence-grounded evaluation of each specific claim on each specific item - exactly the 'is this statement grounded in the cited evidence' positive-verification layer. It also means severity-adjustment (finding 1) is necessary but not sufficient. ### Rater severity is nonstationary within a single scoring session: in an OSCE, later time-slots received systematically higher ratings (regression coefficient 0.88, 95% CI 0.38-1.38, p=.001), with the drift 2.4x larger on difficult stations (1.24 vs 0.52), and excluding warm-up stations did not remove it; the psychometric field has named machinery (DRIFT - differential rater functioning over time, Wolfe et al. 2001; Myford & Wolfe monitoring frameworks) for detecting it. EVIDENCE: PubMed abstract of McLaughlin et al., Medical Education 2009, opened and confirmed (coefficients above). Wolfe & Moulder 2001 (J. Applied Measurement) established Rasch-based procedures for detecting drift types (primacy/recency, centrality drift, practice/fatigue). Drift within and across scoring days is a repeated finding in operational scoring programs (Lunz & Stahl 1990 across grading periods). SOURCE: The effect of differential rater function over time (DRIFT) on objective structured clinical examination ratings (Medical Education 43(10)) | https://pubmed.ncbi.nlm.nih.gov/19769648 | 2009-10 | academic IMPLICATION: Answers taxonomy Q9 with measured evidence: reviewer calibration decays on the timescale of a single session, not just months, and decays faster on hard items. AutoQA should compute reviewer-effect estimates in rolling windows (per session/day) and alert on drift, rather than certifying a reviewer once at onboarding; item-order randomization and hard-item interleaving are cheap design mitigations. ### The MFRM toolkit is now being applied symmetrically to LLM judges: a 2025 study fit MFRM to 10 LLMs plus human expert raters scoring the same writing tasks and found LLMs exhibit measurable severity/centrality rater effects that differ by model, with GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet showing the highest accuracy and fewest rater effects. EVIDENCE: Jiao, Song & Lee, arXiv 2505.18486 (May 2025), abstract opened and confirmed: QWK vs human scores, Cronbach alpha across prompts, and MFRM rater-effect estimates computed identically for human and LLM raters. Parallel 2025 work (Wang et al., Computers and Education: AI, Sept 2025) builds a full psychometric reliability/validity framework for LLM raters in large-scale writing assessment. This is the literature bridge: the same model audits both layers. SOURCE: Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model (Jiao, Song, Lee) | https://arxiv.org/abs/2505.18486 | 2025-05-24 | academic IMPLICATION: The AutoQA's own LLM-judge component should be enrolled as just another 'rater' facet in the same measurement model as the humans it reviews. This gives a project-agnostic, self-auditing structure: one MFRM per project instruction-set yields severity/centrality/fit for every judge, human or machine, and flags when the LLM layer itself drifts after a prompt or model change. - DISAGREEMENT: Size of the training effect: Woehr & Huffcutt 1994 estimated FOR training accuracy gain at d=0.83; Roch et al. 2012, with 4x the studies, revised it down to d~0.50. Separately, Noh & Matore 2022 (Frontiers in Psychology, 164 teacher-raters, MFRM) found prior training experience made NO difference to rating quality on a speaking assessment while rating and teaching experience did - generic/one-shot training may buy nothing; the meta-analytic d~0.50 applies to structured FOR protocols specifically. - DISAGREEMENT: How pathological rater pools are: Casabianca & Beiting-Parrish 2026 cite evidence (Nieto & Casabianca 2019) that large professional testing-organization rater pools show minimal rater effects, versus their own demonstration that in a small trained pool (OpenAI's 15 raters) a few aberrant raters materially distorted system rankings. The severity of the problem depends on pool size, professionalization, and tenure - crowdsourced/DaaS annotation pools sit at the bad end of this spectrum. - DISAGREEMENT: Where the fixable variance lives: MFRM studies foreground the rater severity main effect (a stable, correctable offset), while G-theory decompositions (Khodi 2021; GMAT AWA tradition) show the main effect is small and idiosyncratic rater-by-item interaction dominates. These are complementary lenses, but they imply different remedies (statistical adjustment vs per-item verification), and the literature does not agree on the split's typical proportions across contexts. - GAP: No meta-analysis isolates how much variance rubric operationalization ALONE removes (rubric vs no-rubric variance-component comparison). Individual studies exist (rating-scale vs rubric vs holistic; criteria-order effects on halo, Kim 2020) but no pooled effect size - so 'root cause 1 vs 2-3' cannot be settled from literature alone, though the Weigle/Khodi pattern strongly suggests substantial rater-intrinsic residual. - GAP: Long-horizon reviewer skill decay (months-scale, the Q9 timescale) is not measured in this literature: DRIFT studies cover within-session and across-day/grading-period drift. No published effect sizes for calibration decay over weeks-to-months in annotation-industry settings were found through 2026-07. - GAP: Minimal-linkage requirements under annotation-economics constraints are underspecified: MFRM needs rater-item overlap, but no published guidance found on the cheapest overlap design (e.g., what fraction of items double-reviewed, seeded common items) sufficient for stable severity estimates in pools with high rater churn - this likely needs a small in-house simulation, which is cheap since the models are standard. - GAP: Semantic Scholar returned HTTP 429 during the sweep; coverage relied on OpenAlex/Crossref/Exa/Tavily. A dedicated pass over Language Testing and Assessing Writing 2025-2026 issues could surface additional recent MFRM-for-annotation work. ## Area: Legal/regulatory constraints on algorithmic management of the attempter workforce, plus data-quality standards. The AutoQA issues automated pass/fail verdicts affecting paid workers' income and proposes process telemetry (timing, edit traces) as a provenance contract - this is squarely regulated territory: GDPR Article 22 (automated decisions with significant effects require human review - which may legally mandate the one-human-touch the design treats as optional), the EU Platform Work Directive (transposition deadline December 2026, explicitly governs algorithmic management, worker access to decision logic, and human oversight of automated account/pay decisions), and ISO/IEC 5259 (data quality for analytics and ML, published 2024-2025) which large buyers may contractually require. Zero regulatory/standards sources appear in any domain. ### Platform Work Directive Article 10(5) requires that any decision to restrict, suspend, or terminate the contractual relationship or account of a person performing platform work - or any other decision of equivalent detriment - be taken by a human being, with no consent or contractual-necessity exception (stricter than GDPR Art 22(2)). EVIDENCE: Verified against the directive text and multiple law-firm/academic analyses (Wolters Kluwer, Taylor Wessing, ETUI). Article 10 covers decisions 'taken or supported' by automated systems, deliberately closing the GDPR gap for hybrid human+algorithm pipelines. Unlike GDPR Art 22, the PWD permits no consent-based or contract-based exception for these decisions. SOURCE: Directive (EU) 2024/2831, Official Journal 11 Nov 2024 (EUR-Lex), corroborated by Wolters Kluwer Global Workplace Law & Policy analysis | https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024L2831 | 2024-11-11 | primary IMPLICATION: For EU-based attempters, 'zero-touch plus randomized audits' is illegal for verdicts that cost workers access to work, pay, or their account once national transposition applies (from 2 Dec 2026). The one-human-touch is not a design option for adverse consequences - it is a legal floor. AutoQA can auto-pass but must route auto-fail-with-consequences through a human decision-maker. This is stricter than GDPR: consent cannot buy the exception back. ### Platform Work Directive Article 11 grants persons performing platform work the right to (a) a plain-language explanation of ANY decision taken or supported by an automated decision-making system, (b) a designated competent human contact person to discuss the decision, and (c) human review with a substantiated written reply within two weeks of request; decisions that infringed rights must be rectified within two weeks or compensated. EVIDENCE: Verified verbatim from directive text via extraction of the EUR-Lex page: explanation 'in a transparent and intelligible manner, using clear and plain language'; contact persons must have 'the competence, training and authority necessary'; substantiated reply 'in any event within two weeks of receipt of the request'; rectification/compensation duty plus obligation to modify or discontinue the offending automated system. Carve-out: for workers who are 'business users' under the P2B Regulation (EU) 2019/1150, that regulation's human-review provisions prevail - but P2B itself mandates statements of reasons and an internal complaint-handling system for restriction/suspension/termination. SOURCE: Directive (EU) 2024/2831, Article 11 (EUR-Lex full text) | https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024L2831 | 2024-11-11 | primary IMPLICATION: Contest/appeal channels are legally mandatory, not optional, with a hard two-week SLA. The AutoQA's constructive-feedback output doubles as the legally required explanation - so verdict rationales must be written in plain language grounded in the project criteria (not model logits or opaque scores), and the system must log enough evidence per verdict to support a substantiated human reply on appeal. The 'modify or discontinue the system' duty means systematic AutoQA errors surfaced via appeals trigger a legal remediation obligation. ### The directive's scope explicitly covers online-performed work - the 'digital labour platform' definition applies 'irrespective of whether that work is performed online or in a certain location' - recital 19 names tagging/crowdwork, and Chapter III algorithmic-management rules (Arts 7-11) apply to genuinely self-employed persons performing platform work, from the start of recruitment, regardless of where the platform is established. EVIDENCE: Definition verified verbatim from EUR-Lex (Art 2: service provided at a distance by electronic means, at the recipient's request, involving organisation of work for payment, using automated monitoring/decision systems). Article 7(2) verified: 'shall apply to all persons performing platform work from the start of the recruitment or selection procedure.' Chapter III's application to the genuinely self-employed rests on the Art 16(2) TFEU data-protection legal basis (ETUI, Countouris & De Stefano 2025). Lund AI Policy Lab (Mar 2025) analysis specifically concludes AI labelers are covered. Extraterritoriality confirmed by Ius Laboris and Omnivoo (2026): applies to platforms established outside the EU when the work is performed in the EU. SOURCE: Directive (EU) 2024/2831 Art 2 & 7 (EUR-Lex) + AI Policy Lab, 'Potential impact of the EU Platform Work Directive on AI labelers' | https://aipolicylab.se/2025/03/25/potential-impact-of-the-eu-platform-work-directive-on-ai-labelers/ | 2025-03-25 | academic IMPLICATION: Classifying attempters as independent contractors does NOT exempt the AutoQA from the algorithmic-management chapter - the human-decision, explanation, appeal, and data-limitation rules attach to self-employed annotators too, and to a US-based platform with EU attempters. The foundational philosophy must treat these as baseline constraints for any EU-touching deployment, effective 2 Dec 2026 (about 5 months after this design ships). ### Platform Work Directive Article 7 prohibits automated systems from processing personal data on workers' emotional or psychological state, private conversations (including worker-to-worker exchanges), any data collected while the person is not offering or performing platform work, data predicting exercise of fundamental rights (e.g., organizing), or inferring protected characteristics; Article 8 additionally mandates a data protection impact assessment. EVIDENCE: Verified verbatim from the directive text via EUR-Lex extraction (Art 7(1)(a)-(c) and recital 40). Art 7(3) extends these limits to ANY automated system 'taking or supporting decisions that affect persons performing platform work in any manner' - not just formally designated monitoring systems. SOURCE: Directive (EU) 2024/2831, Article 7 (EUR-Lex full text) | https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024L2831 | 2024-11-11 | primary IMPLICATION: The proposed process-telemetry provenance contract (timing, edit traces) is lawful only if scoped to on-task activity: no collection while the attempter is off-task, no sentiment/frustration inference from edit patterns or communications, no monitoring of attempter-to-attempter channels. Telemetry feeding the AutoQA requires a DPIA. Design the telemetry schema as an explicit allowlist (task-session-bounded behavioral signals) rather than a general activity log. ### Under GDPR Article 22 as interpreted in CJEU SCHUFA (C-634/21, 7 Dec 2023), an automated score that plays a 'determining role' in a downstream decision is itself an Article 22 decision, and human involvement only removes a decision from Art 22 scope if it is meaningful - a reviewer with real authority, data access, and competence to override; rubber-stamping does not count. Enforcement is active: Italy's DPA sanctioned automated rider deactivations, and Hamburg's DPA fined a company ~EUR 490,000 in September 2025 for automated rejections without adequate explanation of the logic. EVIDENCE: SCHUFA three-condition test and 'determining role' doctrine confirmed across TLT, Bird & Bird, IAPP, and Oxford Industrial Law Journal ('Scores as Decisions?', 2024, applying it to the labour context and Uber litigation). EDPB-endorsed WP29 guidance holds that consent is generally an invalid basis in work contexts due to power imbalance, and that human involvement must be substantive. Hamburg fine and Italian rider cases reported in 2025 enforcement roundups; regulator attention to automated rejections continuing into mid-2026. SOURCE: CJEU Case C-634/21 SCHUFA analysis + 'Scores as Decisions? Article 22 GDPR ... in the Labour Context', Industrial Law Journal | https://academic.oup.com/ilj/article/53/4/840/7745471 | 2024-09 | academic IMPLICATION: Two consequences beyond the PWD: (1) even where AutoQA runs as a vendor scoring layer feeding a client's pass/fail call, the SCHUFA logic makes the score itself an Art 22 decision if the client defers to it - outsourcing the final click doesn't escape regulation; (2) the one-human-touch must be designed as genuine review (authority to overturn, access to the evidence, calibrated competence), or it legally counts as zero-touch. Worker consent cannot serve as the lawful basis for telemetry or automated verdicts. ### ISO/IEC 5259 'Data quality for analytics and ML' is now a complete five-part published series - Parts 1-4 published 2024 (overview/terminology; data quality measures; data quality management requirements & guidelines; process framework) and Part 5 (data quality governance framework) published 2025 - sold by ISO as an 'AI data quality management bundle'. EVIDENCE: Confirmed from ISO.org catalog pages: ISO/IEC 5259-1:2024 (81088), 5259-2:2024 (81860, 'specifies a data quality model, data quality measures and guidance on reporting data quality'), 5259-3:2024 (81092, management requirements), 5259-4:2024 (81093, process framework covering data labeling among ML data processes), 5259-5:2025 (84150, governance). Full text paywalled; no direct evidence found of large AI-data buyers contractually mandating 5259 yet, but 5259-3/-4 are certifiable-style requirements documents a buyer can reference in contract. SOURCE: ISO/IEC 5259 series catalog pages (ISO.org) / AI data quality management bundle | https://www.iso.org/publication/PUB200525.html | 2025 | primary IMPLICATION: The AutoQA philosophy document should map its quality dimensions and verdict evidence onto 5259-2's measurement/reporting vocabulary and 5259-4's process framework, so a per-project customization can be presented as 5259-conformant when a buyer asks - cheap to do now, expensive to retrofit. Treat 5259 as the neutral shared vocabulary for 'what a quality measure is' across projects. - DISAGREEMENT: Scope breadth of the algorithmic-management rules: CXC Global (Apr 2026) claims the directive's algorithmic-management rules 'apply to all workers, not only those engaged through digital labour platforms,' but the directive text limits Chapter III to 'persons performing platform work'; ETUC's transposition manual (Mar 2026) confirms extension to all workers is only an ADVOCACY position for national transposition, not the directive's requirement. Some member states may gold-plate; the EU floor covers platform work only. - DISAGREEMENT: Whether online annotation workers will effectively get the employment presumption: AI Policy Lab (Mar 2025) says AI labelers benefit when platforms control workflows/evaluation, but commentators (Verfassungsblog, ILO working papers) caution the control criteria are anchored in traditional notions that fit poorly with subtle microtask monitoring, and Fairwork warns weak transpositions (e.g., Italy) may leave annotators 'self-employed'. Note this dispute concerns the employment presumption only - Chapter III algorithmic-management rights attach regardless of status. - DISAGREEMENT: Strictness of national implementations: law-firm trackers (Ius Laboris Apr 2026, employsome May 2026) expect strict transpositions in France/Germany/Italy/Spain/Netherlands but narrow ones elsewhere (Hungary had taken no steps as of the CMS review; Ireland has no draft legislation), creating a 2026-2027 patchwork - sources disagree on whether to design to the strictest member state or per-country. - GAP: No enforcement action or national transposition text specifically addressing AI data-annotation QA verdicts was found - the directive's application to a QA layer rejecting individual work items (vs. account suspension) is untested; whether a single item pass/fail (payment denial for one task) is a 'decision of equivalent detriment' under Art 10(5) versus merely an Art 11 reviewable decision is unresolved in the sources. - GAP: Whether a BPO/vendor arrangement (annotators employed by an outsourcing firm, AutoQA run by the AI lab or its vendor) falls within the 'digital labour platform' definition is unexamined in available sources; recital 20 excludes platforms that merely connect providers to clients, and the intermediary/subcontractor liability question is flagged by ETUC as a transposition battleground. - GAP: ISO/IEC 5259 full text is paywalled - could not verify its specific annotation-quality measures (e.g., inter-annotator agreement treatment) or confirm any AI-data buyer contractually requiring it; the 'buyers may require it' premise remains plausible but unevidenced. - GAP: No dedicated EDPB guideline on automated decision-making in employment exists as of July 2026 - the operative guidance is still the 2018 WP29 guidelines (EDPB-endorsed); the EDPB Work Programme 2026-2027 (adopted 11 Feb 2026) was not verified in detail for a pending ADM-in-employment item. - GAP: Non-EU jurisdictions were out of scope of this pass: no findings on US state algorithmic-management bills, UK's DUAA ADM regime (expected effective 2026), or Brazil/India annotation-workforce rules - relevant if the attempter pool is global. ## Area: Content-moderation QA programs - the largest deployed analog of humans-reviewing-human-judgment at scale. Meta/TikTok/Google have run decade-old programs auditing moderator decisions: golden-set seeding into live queues, audit sampling rates, overturn/appeal-rate monitoring, QA-of-the-QA layers, measured reviewer base-rate and fatigue effects, and published transparency-report methodology plus academic studies (e.g., on moderator agreement and audit design). The industry-practice domain covered only AI-training-data vendors; this adjacent industry has already answered several taxonomy Q6/Q9 questions operationally (seeded known-verdict items, easy-case dilution, de-graduation triggers). ### Facebook's production QA at Cognizant ran a three-layer blind audit: ~50-60 of each moderator's ~1,500 weekly decisions (~3-4%) randomly re-reviewed by a dedicated QA worker (paid $1/hr more), with full-time Facebook employees auditing a subset of QA decisions; 'accuracy' was computed purely as agreement with the auditor against a 95% target (other sites reported 98%), actual scores ran high-80s to 92, and misses triggered a remediation program that often ended in termination. EVIDENCE: Verified in The Verge's original text (fetched): 'From Miguel's 1,500 or so weekly decisions, Facebook will randomly select 50 or 60 to audit... Full-time Facebook employees then audit a subset of QA decisions.' Moderator quote: 'Accuracy is only judged by agreement. If me and the auditor both allow the obvious sale of heroin, Cognizant was correct... This number is fake.' Also confirmed: moderators lobbied QAs off-book to reverse decisions despite prohibition, and pre-firing 'coaching' often served as pretext for managing workers out. SOURCE: The Trauma Floor: The secret lives of Facebook moderators in America (The Verge, Casey Newton) | https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona | 2019-02-25 | primary IMPLICATION: Directly answers taxonomy Q6/Q9 with a decade-scale postmortem: (a) ~3-4% random audit + a QA-of-QA layer is the deployed baseline; (b) agreement-with-auditor as the accuracy metric is blind to correlated errors and gameable via dispute lobbying; (c) punitive per-item scoring makes the QA relationship adversarial. AutoQA should score attempter statements against grounded evidence (not reviewer agreement), log auditor-attempter deliberations, and decouple constructive feedback from de-graduation triggers. ### The Trust & Safety Professional Association (industry body staffed by Meta/Google/TikTok alumni) codifies exactly two complementary QA channels - forward audit sampling (weighted re-review samples by peers or a dedicated quality team) and 'reverse quality sampling' (pre-reviewed golden/known-verdict items injected through the regular review process) - with the binding constraint that seeded items 'must look exactly like regular reviews to be effective', plus appeals/overturns as a cheap but population-biased third signal, and a four-way error taxonomy (false positive, false negative, wrong-selection, technical error). EVIDENCE: Fetched both TSPA curriculum pages. QA page: golden seeding gives 'full control' to probe chosen edge cases but fails when user history is part of the review or when indistinguishability breaks; appeals surface false positives 'at a low cost' but 'the population of appeals is often different from the population of all decisions.' Metrics page: overturn rate 'can be a difficult metric to interpret' alone (low overturn = correct decisions OR users gave up); 'consistency' (multi-review mismatch rate) is tracked as a distinct metric from error rate; prevalence measurement via random sampling is called a 'premium metric' due to cost. SOURCE: TSPA: Content Moderation Quality Assurance + Metrics for Content Moderation | https://www.tspa.org/curriculum/ts-fundamentals/content-moderation-and-operations/content-moderation-quality-assurance/ | 2022-09-16 | practitioner IMPLICATION: Imports the settled industry answer to blind gold injection: it is standard practice, its failure mode is detectability (seeded items must be indistinguishable - hard when context/history is part of the task), and it is complementary to (not a substitute for) forward sampling. The wrong-selection error class (right verdict, wrong cited criterion) maps directly to the AutoQA requirement to check that an attempter's statement is aligned with the specific criterion they invoke, not just that the verdict is right. ### Pinterest's deployed Decision Quality Evaluation Framework (Feb 2026) scores both human moderators and LLM agents against an SME-adjudicated, versioned, immutable golden set selected by inverse-propensity sampling (XGBoost on PinCLIP embeddings, prioritizing low-propensity/novel items), using a two-axis diagnostic - Cohen's kappa for reliability plus correctness vs the golden set - where 'high reliability + low correctness' is read as systematic policy misunderstanding; measured results show 3x-human majority vote adds only +3.6pp accuracy over a single non-expert human, and current LLMs perform on par with a single non-expert human (GPT-5 the only config with positive accuracy delta, +0.9pp). EVIDENCE: Fetched full HTML of arXiv 2602.15809. Additional verified mechanics: policy changes handled by dual-labeling the existing golden set under old and new guidelines and visualizing the 'policy delta' as label flips; QA-of-the-QA is two continuous monitors - content-drift (evaluate on newest golden items each release) and system-stability (re-run fixed prompt on fixed golden version to catch pipeline non-determinism); switching prevalence measurement from 3x-human to LLM gave 'over 30x cost savings and 10x turnaround reduction.' Gemini 2.5 configs showed +22pp recall but +48-57pp false-positive rate. SOURCE: Decision Quality Evaluation Framework at Pinterest (arXiv 2602.15809) | https://arxiv.org/abs/2602.15809 | 2026-02-17 | primary IMPLICATION: The closest 2026 production analog to the AutoQA foundation: (a) reliability and correctness must be separated - kappa alone cannot distinguish shared misunderstanding from noise; (b) golden sets should be deliberately non-representative (oversample rare/hard cases) with coverage measured in embedding space; (c) policy evolution requires versioned relabeling, not patching; (d) an LLM QA layer needs both a drift monitor and a determinism monitor; (e) expect the LLM judge to be ~single-non-expert-human quality with an FPR skew unless calibrated. ### Auditing raters by rewarding agreement with the eventual consensus outcome ('consensus-based auditing', as X's Community Notes has done since Sept 2022 by tying participation eligibility to agreement with the final aggregate) measurably induces strategic conformity - minority contributors' evaluations drift toward the majority and their participation share falls precisely on controversial items - and a two-stage alternative that weights contributors by the stability of their past residuals (predictability relative to a latent-factor model) rather than majority-agreement improves out-of-sample predictive performance. EVIDENCE: Fetched arXiv 2603.18053 (Alimohammadi, Huang, Borgs, Chayes, Mar 2026) abstract and framing via Exa crawl: empirical evidence from Community Notes data plus a behavioral model where contributors trade off private beliefs against anticipated penalties for disagreement; the proposed method gives influence to consistently-informative contributors 'even when they disagree with the prevailing consensus.' Senior authors (Borgs h-54, Chayes h-60). Preprint, not yet peer-reviewed. SOURCE: Auditing the Auditors: Does Community-based Moderation Get It Right? (arXiv 2603.18053) | https://arxiv.org/html/2603.18053 | 2026-03-17 | academic IMPLICATION: Directly constrains de-graduation trigger design: if attempter or QA standing is scored by agreement-with-final-verdict, the AutoQA will train its best dissenters to conform exactly where independent judgment matters most (the subjective-interpretation root cause #1). Score contributors on the stability/informativeness of their residuals against a grounded reference, and never penalize evidence-backed disagreement per se. ### Google's audit-quality method (ICLR 2023 Tiny Paper, Google LLC authors) decomposes inter-rater agreement per rubric question rather than per item - in their worked example one question had Fleiss kappa 0.094 against a 0.475 overall, isolating it as the ambiguous criterion - and identifies sequential non-blind review (a later reviewer seeing the earlier verdict) as a distinct, measurable source of audit risk, testable via A/B or difference-in-differences. EVIDENCE: Extracted full PDF text via pdftotext. Methods: per-question Fleiss kappa to locate high-disagreement criteria; chi-square to test criterion-verdict linkage; t-test/ANOVA to detect review teams systematically deviating from ground truth; binomial CIs to extrapolate sampled error rates. Caveats: 2-page tiny paper, synthetic dataset (3 reviewers, 9-question rubric, 1,528 products), no production deployment claimed. Code at https://github.com/xuanyang0607/openreviewpaper. SOURCE: Statistical Methods for Auditing the Quality of Manual Content Reviews (arXiv 2306.07466, ICLR 2023 Tiny Papers) | https://arxiv.org/abs/2306.07466 | 2023-06-12 | academic IMPLICATION: For root cause #1 (subjective interpretation of axes): measure agreement per evaluation criterion, not per item - this converts 'reviewers disagree' into 'criterion 3 is ambiguous, rewrite it', which is the closed-loop the AutoQA needs to feed back into project instruction sets. Also: keep any second/QA review blind to the first verdict to avoid anchoring. ### A 2025 New Media & Society study (screen-share observation of commercial moderators in India) documents that throughput pressure distorts verdict distributions through the action-selection interface itself: moderators systematically chose a 2-click removal over a 4-click de-ranking regardless of policy fit, mechanically deleted tool-highlighted words without context assessment, and privately compressed broad guidelines into simplified dos/don'ts lists - i.e., queue and UI composition changed outcomes independent of moderator judgment quality. EVIDENCE: Fetched The Conversation summary (2025-07-22) by the study authors (Chatterjee IIT Delhi/UQ, Gupta, Thomas; New Media & Society, peer-reviewed). Evidence is qualitative (interviews + observed sessions), no quantitative error rates; an observed moderator said she 'would never recommend de-ranking content as it would take time.' Complements industry-standard per-item handling-time metrics (TSPA) that create the pressure. SOURCE: Hard labour conditions of online moderators directly affect how well the internet is policed (The Conversation, summarizing New Media & Society study) | https://theconversation.com/hard-labour-conditions-of-online-moderators-directly-affect-how-well-the-internet-is-policed-new-study-261386 | 2025-07-22 | academic IMPLICATION: Reviewer 'harshness' variance is partly an artifact of per-item cost asymmetries, not belief: any AutoQA that infers attempter/reviewer quality from verdict distributions must first control for the click/effort cost of each verdict option, and the one-human-touch interaction in the hybrid loop should make the correct action the cheapest action. - DISAGREEMENT: Facebook's moderator accuracy target: The Verge's original Phoenix reporting (Feb 2019, fetched) states a 95% target with actual scores in the high-80s to 92; CNBC's Tampa follow-up (Jun 2019) and The Irish Times' Dublin reporting (Feb 2020) both state a 98% target. Likely site/contract variation, but the primary sources genuinely conflict on the number; all agree the target was chronically missed. - DISAGREEMENT: What low reviewer agreement means: production vendor practice (Facebook/Cognizant, Accenture/CPL) treats disagreement-with-auditor as individual reviewer error and scores/fires on it, while Musubi Labs (Nov 2025, practitioner), Pinterest's framework (2026), and the Google tiny paper treat low agreement primarily as a policy/criterion-ambiguity signal ('don't force agreement'); the Community Notes paper (Mar 2026) goes further, showing agreement-rewarded auditing actively corrupts the signal via strategic conformity. - DISAGREEMENT: Platform self-reported accuracy vs external measurement: TikTok's fifth DSA transparency report claims 99.2% moderation accuracy (self-defined, sample re-review), while academic audits of the DSA transparency database found up to ~50-percentage-point inconsistencies in TikTok's self-reported automation figures, and moderator testimony ('this number is fake - accuracy is only judged by agreement') argues agreement-based accuracy overstates true correctness. Self-reported accuracy figures from this industry should not be used as calibration anchors. - DISAGREEMENT: Value of majority vote: Pinterest measured 3x-human majority at only +3.6pp accuracy over a single non-expert human (arguing redundancy is a weak, expensive lever), whereas Roblox's published practice treats >=80% multi-moderator alignment as the gate for scaled consistency - different uses (measurement vs gating) but opposite implicit views on how much signal replication buys. - GAP: No public quantitative study of queue-composition effects on reviewer harshness in content moderation specifically (easy-case dilution shifting strictness, prevalence-induced criterion drift): the mechanism is well established in adjacent vigilance/low-prevalence-effect literature, but I found no moderation-industry measurement of it; the 2025 New Media & Society evidence is qualitative only. This taxonomy question remains empirically open even in the largest deployed analog. - GAP: Gold-injection density is nowhere disclosed: no platform, vendor, or paper states what fraction of a live queue is seeded known-verdict items, only that seeding exists and must be indistinguishable (TSPA) and that golden sets should oversample hard cases (Pinterest, Musubi). - GAP: Meta's current (2025-2026) internal audit sampling rates and Community Standards Enforcement Report reviewer-accuracy methodology are not publicly disclosed at mechanic level; the best internals remain 2019-2020 journalism. Kenya/Meta litigation (Majorel/Sama) disclosures center on labor harms, not QA mechanics - I found no litigation-produced QA-design documents newer than that reporting. - GAP: De-graduation thresholds are only known via journalism (miss accuracy target -> remedial program -> termination); no published numeric trigger (e.g., N misses in window, minimum sample before action) from any platform. - GAP: Pinterest's paper, the single best 2026 primary source, omits absolute golden-set size, SME adjudication protocol, and absolute (non-delta) accuracy values, so its numbers transfer as design patterns, not calibration constants. ## Area: Feedback-efficacy science from education and organizational psychology. Every domain independently reports 'no evidence that QA feedback changes attempter behavior' as a gap, yet feedback-intervention research is a mature field: Kluger & DeNisi's feedback intervention theory (a third of feedback interventions REDUCE performance, with known moderators), formative-assessment literature on feedback specificity/timing, and workplace studies on feedback under pay-linked evaluation (which predicts gaming/monoculture, taxonomy Q8). The sweep searched only for annotation-specific longitudinal studies and found none - the general literature was never consulted. ### Feedback intervention theory's core empirical result stands: feedback improves performance on average (K&D 1996: 131 studies, ~12,000+ participants, mean d approximately 0.38-0.41) but MORE THAN ONE THIRD of feedback interventions REDUCE performance, and effectiveness declines as feedback cues move attention from the task toward the self (praise, person-level evaluation, normative comparison). EVIDENCE: Verified in the authors' own summary (Kluger & DeNisi 1998, Current Directions in Psychological Science): 'although FIs improve performance on average, they reduce performance in more than one third of the cases.' Independently corroborated by Wisniewski et al. 2020, which cites K&D as 131 studies, >12,000 participants, average effect 0.38 with roughly a third of effects negative. FIT's mechanism: feedback that directs attention to meta-task/self processes (threat to self, praise, social comparison) depletes task attention and backfires; task- and process-focused cues help. SOURCE: Feedback Interventions (Kluger & DeNisi 1998), summarizing K&D 1996 Psychological Bulletin meta-analysis | https://doi.org/10.1111/1467-8721.ep10772989 | 1998-06 (meta-analysis 1996-03; foundational) | academic IMPLICATION: The AutoQA feedback 4-tuple must be strictly task/criterion-referenced and evidence-anchored, never person-referenced or rank-referenced; a feedback channel is not presumptively net-positive, so the design should treat 'feedback reduces this attempter's subsequent quality' as an expected outcome for a substantial minority and instrument for it (per-attempter pre/post error-rate deltas), not assume monotone benefit. ### The March 2025 Cochrane update on audit-and-feedback (292 studies, 678 arms, healthcare professionals) finds median absolute improvement in desired practice of only 2.7% (IQR 0.0 to 8.6; weighted meta-analytic mean +6.2%, 95% CI 4.1-8.2, moderate certainty), with effects larger for low baseline performers, individual-level (not team-level) data, comparison to TOP peers or a benchmark (comparison to peer average showed no significant effect), a trusted local source, and action plans with specific advice - while repeated delivery was associated with LOWER effect size. EVIDENCE: Fetched the Cochrane summary (updated review of CD000259, published 2025-03-25): '292 studies with 678 arms'; median absolute improvement '2.7%, with an IQR of 0.0 to 8.6'; weighted mean '6.2% (95% CI 4.1 to 8.2; moderate-certainty evidence)'; OR 1.47. Moderator list quoted directly, including the counterintuitive 'repeated delivery was associated with lower effect size' and 'comparison to top-peers or a benchmark increased effects; comparing against the average of all peers did not.' SOURCE: Audit and feedback: effects on professional practice (Cochrane review update, Ivers et al.) | https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD000259.pub4/full | 2025-03-25 | academic IMPLICATION: This is the largest causal evidence base that verdict-plus-feedback changes skilled professionals' behavior: expect a real but modest median effect with a fat right tail, concentrated in currently-low performers. Design levers with evidence: target feedback at low-baseline attempters first, deliver individual-level data, pair every failed criterion with a specific corrective action, and benchmark against top-quality exemplars rather than cohort averages. Do NOT assume higher feedback frequency improves uptake. ### In the closest annotation-analog RCT (105 analyzed Mechanical Turk workers writing product reviews), both timely external expert feedback and rubric-based self-assessment significantly improved work quality vs no feedback (expert ratings 6.01 and 6.35 vs 5.69 on a 9-point scale, p<0.05) with NO quality difference between external and self-assessment; external feedback uniquely drove revision behavior (56.5% revised vs 24.8% for self-assessment) and more output, and self-assessors over-rated their own work by 1.8 points (7.9 self vs 6.1 expert). EVIDENCE: Read the full Dow, Kulkarni, Klemmer & Hartmann CSCW 2012 PDF. Between-subjects, blind-to-condition expert grading; self-assessment condition showed significant learning over the task series (slope 0.25, p=0.001) vs borderline for external (0.10, p=0.08) and null for none. Also: crowd-peer assessors had low agreement with the expert (Kappa=0.20 aggregated), and the paper explicitly did not measure long-term learning. Attrition was higher under assessment (Self 78%, External 61%, vs None 47% incompleteness), so gains partly reflect weak performers dropping out plus learning. SOURCE: Shepherding the Crowd Yields Better Work (Dow, Kulkarni, Klemmer, Hartmann; CSCW 2012) | https://www.cs.cmu.edu/~spdow/files/Crowds-Shepherd-CSCW12.pdf | 2012-02-11 (foundational; only direct crowdwork feedback RCT found) | academic IMPLICATION: Directly refutes 'no evidence QA feedback changes attempter behavior' for paid micro-task workers: rubric-mediated, task-specific, synchronous feedback is the minimal effective unit. A concrete per-criterion rubric surfaced to attempters may capture most of the quality gain of expensive external feedback (making the AutoQA-generated feedback the 'external expert' at near-zero marginal cost), but attempter self-ratings cannot serve as measurement, and part of any observed 'improvement' will be selection (weak attempters exiting) - the design's efficacy telemetry must separate within-attempter learning from attrition. ### In the largest educational feedback meta-analysis (435 studies, k=994, N>61,000), information content is the dominant moderator: high-information feedback (task + process + self-regulation content) yields d=0.99 [0.82-1.15] versus d=0.46 for corrective right/wrong feedback and d=0.24 for bare reinforcement/punishment - with overall d=0.48 masking huge heterogeneity (I2=83%) and 17% of raw effects negative. EVIDENCE: Fetched the open-access Frontiers article (Wisniewski, Zierer & Hattie 2020). Feedback-type moderator significant (QB=41.52, p<0.0001); outcome moderator significant: cognitive d=0.51 vs motivational d=0.33; of negative motivational effects, 86% came from uninformative (reward/punishment-style) feedback. Authors conclude feedback 'cannot be understood as a single consistent form of treatment.' SOURCE: The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research | https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full | 2020-01-22 | academic IMPLICATION: This is the direct evidential anchor for taxonomy Q7's feedback-vs-verdict-only branch: a pass/fail verdict is the reinforcement/corrective tier (expected d approximately 0.24-0.46), while explaining WHAT is wrong against the criterion, WHY (process), and HOW to self-check next time (self-regulation) roughly doubles the expected effect. The minimal feedback unit worth building is therefore criterion-cited + evidence-grounded + process-level; verdict-only QA forfeits most of the achievable behavior change, and 'feedback is decoration' should only be concluded if high-information feedback (not verdicts) fails to move within-attempter error rates. ### The 2025 Annual Review of Organizational Psychology's 25-year retrospective concludes the science of workplace feedback 'is not yet a story of coherent and cumulative progress': definitions are generic, assumptions diverge across six disconnected research substreams, and simple universal rules about feedback effectiveness do not survive contact with organizational reality. EVIDENCE: Crawled the full open-access review (Anseel & Sherf, Annu. Rev. Organ. Psychol. Organ. Behav. 12:19-43, published 2025-01-21). Abstract states insights 'often appear disconnected from the way feedback is practiced and experienced in organizations'; the review calls for explicated assumptions and paradigms mirroring complex realities. The companion 2025 systematic review (Heine, Stouten & Liden, J. Organ. Behav., 2025-10-26) reaches the same verdict for supervisor performance feedback: most studies fail even to specify feedback valence, and feedback quality/accuracy findings rest on inconsistent constructs. SOURCE: A 25-Year Review of Research on Feedback in Organizations: From Simple Rules to Complex Realities (Anseel & Sherf) | https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-110622-031927 | 2025-01-21 | academic IMPLICATION: Tempering prior for the whole Q7 branch: the general literature supplies directional moderators (task-focus, specificity, information content, source credibility) but NO validated plug-in recipe, and effect heterogeneity is the norm. The project-agnostic foundation should therefore ship feedback design as parameterized hypotheses with built-in efficacy measurement (per-project A/B of feedback tiers against repeat-error rate), not as fixed doctrine imported from any single meta-analysis. ### Under incentives, relative-rank feedback is a double-edged lever: lab and field economics find rank feedback raises output in flat-wage settings (Charness et al. 2014; Tafkov 2013) but induces costly sabotage and cheating to improve rank that offsets the gains, and in at least one field experiment (Barankay 2012) REMOVING rank feedback improved performance. EVIDENCE: Confirmed via the literature synthesis in a peer-reviewed Leadership Quarterly article ('Feedback quality and performance in organisations', 2021), which states: Charness et al. (2014) found 'offering relative rank feedback increases output... and subjects are willing to engage in costly sabotage and cheating activities to improve their relative rank, thus offsetting the positive effects'; Barankay (2012) is cited as the exception where rank feedback hurt. Primary Charness/Barankay texts not independently opened, so treated as well-sourced secondary evidence. SOURCE: Feedback quality and performance in organisations (Leadership Quarterly; synthesizing Charness et al. 2014, Tafkov 2013, Barankay 2012) | https://www.sciencedirect.com/science/article/abs/pii/S1048984321000394 | 2021 (synthesizing 2012-2016 primaries) | academic IMPLICATION: For taxonomy Q8 (gaming/monoculture under pay-linked evaluation): if AutoQA outputs become visible rank or pass-rate leaderboards tied to pay, the literature predicts optimization of the metric (score-hacking, mimicry of known-passing templates) rather than quality. Keep attempter-facing feedback private, criterion-referenced, and decoupled from visible peer ranking; note the tension with Cochrane's top-peer-benchmark moderator (see disagreements) - benchmark against exemplar WORK, not against ranked PEOPLE. - DISAGREEMENT: Feedback frequency: pre-2025 audit-and-feedback guidance (Ivers 2012 Cochrane, Hysong 2006) held that repeated/more frequent delivery increases effect; the 2025 Cochrane update finds repeated delivery associated with LOWER effect size. Unresolved - could be confounding (repeated A&F deployed where problems persist) or genuine habituation. - DISAGREEMENT: Normative comparison: the 2025 Cochrane update finds comparison to top peers or a benchmark INCREASES behavior change in healthcare professionals, while FIT (Kluger & DeNisi) predicts normative comparison shifts attention to self and degrades performance, and incentive economics (Charness 2014, Barankay 2012) finds rank feedback triggers gaming/sabotage under competitive stakes. Plausible reconciliation: comparison to an exemplar standard helps when stakes are professional-norm-based; comparison as interpersonal rank hurts when pay/status is on the line - but no study directly adjudicates this. - DISAGREEMENT: Self-assessment vs external feedback: Dow 2012 found rubric self-assessment equal to external expert feedback for quality improvement (arguing feedback machinery could be replaced by surfaced rubrics), but the same study found self-ratings inflated by 1.8/9 points and education literature (Winstone 2016 recipience work) holds self-assessment only works when later external verification is believed to occur. External QA may be load-bearing as a credibility backstop even if the information could be self-generated. - DISAGREEMENT: Effect magnitude: education meta-analyses report medium standardized effects (d approximately 0.48), while the healthcare A&F median is a small 2.7% absolute improvement on already-trained professionals. For skilled adult annotators the healthcare prior (small median, heterogeneous, concentrated in low performers) is likely the better calibration than the education prior. - GAP: Still no longitudinal study of QA feedback effects on paid ANNOTATION workers' repeat-error rates: Dow 2012 explicitly did not measure long-term learning, and no 2024-2026 annotation-platform study of feedback efficacy was found (searched Exa, Tavily, OpenAlex/Crossref). The original sweep's gap is real; what changed is that adjacent causal literature (Cochrane A&F 2025, crowdwork RCT) transfers with stated caveats. - GAP: No study found on AI-GENERATED feedback to human annotators under pay-linked evaluation - the exact AutoQA deployment condition. Closest analogs are AI-feedback-to-employees work (e.g., Tong et al. 2021 SMJ, disclosure reduces effect) which was not deep-dived here. - GAP: Charness et al. 2014 and Barankay 2012 primaries were not opened (claims verified only through a peer-reviewed secondary synthesis); exact effect sizes for gaming-offset were not extracted. - GAP: Feedback-specificity tradeoff (Goodman & Wood 2004/2011: high specificity aids immediate performance but can impair exploration and transfer to novel cases) was identified in the Anseel & Sherf reference base but not independently verified - relevant to whether highly prescriptive AutoQA feedback creates template-following monoculture (Q8) and worth one follow-up read. - GAP: The 2025 individual-differences meta-analysis (Condrea & Iliescu, EJWOP, 2025-12-16) on reactions to feedback was located but is too new to have accessible full text; could sharpen per-attempter moderation (feedback orientation, self-esteem) but was not extractable."}, {"slug": "domain-academic-judges", "title": "Domain Report - Academic Judges", "category": "Domain research", "summary": "Academic judge benchmarks, task-dependent capability limits, bias, and model-selection evidence.", "tags": ["domain report", "academic judges"], "view": "documents/view/domain-academic-judges.html", "raw": "documents/raw/research/domain-academic-judges.md", "text": "# Domain: academic-judges ## On hard, objectively-verifiable judging tasks (JudgeBench, ICLR 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while GPT-4o scored 50.9-56.6% (near random), the best reward model (Skywork-Reward-Gemma-2-27B) hit 64.3%, and fine-tuned judges like PandaLM fell at or below random. [LOAD-BEARING] EVIDENCE: JudgeBench uses 350 rigorously verified response pairs across knowledge/reasoning/math/coding where correctness is checkable. Verified numbers: GPT-4o vanilla 50.9% overall, Arena-Hard pipeline 56.6%, o1-preview 75.4% (85.7% math/coding), Skywork-Reward-27B 64.3%, reward models 60-64%, ChatEval multi-agent ~34%. Fine-tuned preference judges plateau near random on knowledge/reasoning. SOURCE: JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025) + EmergentMind topic synthesis (updated Jan 2026) | https://arxiv.org/abs/2410.12784 | 2024-10 (ICLR 2025; synthesis updated 2026-01) | academic IMPLICATION: The AutoQA core judge should be a frontier reasoning model, not an off-the-shelf fine-tuned judge or scalar reward model, because annotation-QA requires reasoning about correctness, not preference matching. Reasoning-model headroom (75% vs 51% for the same lab's non-reasoning model) is the single biggest capability lever. ## Standard LLM-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial LLMs, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy. [LOAD-BEARING] EVIDENCE: Guerdan, Barocas, Holstein, Wallach, Wu, Chouldechova (NeurIPS 2025 poster, abstract opened and confirmed). They formalize 'rating indeterminacy' - items where multiple ratings are reasonable - and show humans and LLMs resolve forced choices differently, so aggregated gold labels systematically mis-rank judges. They provide concrete recommendations for elicitation and aggregation. SOURCE: Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025) | https://neurips.cc/virtual/2025/poster/117308 | 2025-09-19 | academic IMPLICATION: Directly answers design Q1/Q2: the ground-truth construct cannot be a single forced pass/fail human label. The gold set must elicit response-set labels ('which verdicts are defensible?'), and the verdict ontology needs a first-class 'instructions underdetermine this case' class. Validation against collapsed consensus labels will select the wrong judge configuration. ## The largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including April-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected Cohen's kappa is 33-41 percentage points lower on MT-Bench - and judge rankings shift by up to 14 positions depending on which benchmark you use. [LOAD-BEARING] EVIDENCE: Norman, Rivera, Hughes (arXiv 2606.19544, abstract opened and confirmed). Four cohort-wide findings: universal kappa deflation; benchmark-dependent rankings (up to 14 position shifts across MT-Bench/JudgeBench/RewardBench); a 'consistency-bias paradox' where test-retest reliability >0.95 coexists with position bias >0.10 in two production judges; verbosity bias small (<0.011) under a single controlled pairwise rubric. They distill a Minimum Viable Validation Protocol. SOURCE: Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic IMPLICATION: Answers design Q1's metric question: ban raw percent-agreement and single-number accuracy from all AutoQA reporting; standardize on chance-corrected statistics per benchmark. Also answers Q5: a judge that is perfectly self-consistent can still be systematically biased - consistency monitoring and bias audits are separate mandatory sensors, and per-project validation must use multiple gold-set designs since rankings do not transfer. ## At the rubric level - the exact granularity AutoQA would operate at - even frontier judges achieve only ~55-56% balanced accuracy on hard rubric-verification instances (GPT-4o 55.97%, Claude-Sonnet-4.5 55.65%), but rubric-level evaluation with explicit chain-of-thought reasoning beats checklist-level evaluation by 7-12 percentage points and reduces cross-judge variance. [LOAD-BEARING] EVIDENCE: RubricEval (arXiv 2603.25133, full alphaXiv overview read). Binary satisfied/not-satisfied verification of atomic rubrics against responses; EASY subset ~90% accuracy, HARD subset ~55%. Compositional (conditional-logic) instructions, Role Persona, and Format Structure rubrics hardest. Their multi-model arbitration labeling pipeline (RAF) reached 85.0% agreement with humans, Cohen's kappa 0.702. Reasoning-intensive models (o3) notably outperform standard instruct models. SOURCE: RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following | https://www.alphaxiv.org/overview/2603.25133 | 2026-03-26 | academic IMPLICATION: Answers design Q3 granularity: decompose to atomic criteria and require per-criterion explicit reasoning (worth +7-12pp), but budget for the cost multiplier and expect a hard residual class (~45% error on contested rubrics) that must route to humans. Easy/hard stratification should be built in: judge consensus at the coarse pass identifies which claims need escalation - the RAF arbitration cascade is a directly reusable architecture for gold-label production. ## On subjective rubrics, LLM judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-to-human), judges use only 0.3-0.5x the human score spread, and inter-LLM agreement (r~0.35) exceeds LLM-human agreement (r~0.27-0.32) - while on a rubric with a verifiable factual answer the same judges fall back into the human range (58.5 degrees, r=0.519). EVIDENCE: Mukherjee, Hamna, Bali, Sitaram (arXiv 2606.03043, abstract opened). 41 LLM judges, 4 datasets, 8 Indic languages, geometric analysis with bootstrap CIs. Fine-tuning/preference optimization recovers score spread (0.32 to 1.08) but barely moves the axis; only post-hoc calibration on a small human-anchored set improved all rubrics. Scope caveat: Indic-language community-health data, but the verifiable-vs-subjective contrast is the mechanism of interest. SOURCE: The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment | https://arxiv.org/abs/2606.03043 | 2026-06-02 | academic IMPLICATION: Multi-judge ensembles and inter-judge agreement are NOT validity evidence on subjective axes - consensus can reflect a shared collapsed subspace. The objective-subjective boundary (design Q2) is empirically real and large: verifiable criteria are judge-autonomous territory; subjective criteria need human-anchored calibration sets, not more judges. ## Single-trial LLM judging is measurably noisy: across 29 tasks with 50 repeated trials, pairwise preferences flipped on average 13.6% of the time (28% of questions exceeded 20% flip rate), semantically equivalent prompt templates changed majority outcomes in 25% of tested cases, and ~11 repeated trials were needed for majority vote to recover the reference verdict with 95% probability. EVIDENCE: Yagubyan (arXiv 2606.13685, abstract opened). GPT-4o-mini and GPT-4.1-mini judges; also found significant first-position bias (72% A-majority, p=0.024), cross-judge agreement only 76% (kappa 0.51), and a pairwise-pointwise gap where judges pick winners even when their own scalar scores show no meaningful difference. Caveat: both judges from one provider, mini-tier models; the Reliability-without-Validity cohort shows some production judges reach test-retest >0.95. SOURCE: The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation | https://arxiv.org/abs/2606.13685 | 2026-04-23 | academic IMPLICATION: Answers design Q5's flip-rate fork with 'both': k-sample voting (k around 5-11) plus position randomization is baseline hygiene, AND per-item flip-rate is a cheap, free signal of criterion underspecification - items with high vote entropy are exactly the 'instructions underdetermine this' class. A perturbation-robustness harness is justified as a ship gate. ## The broadest LLM-vs-human-judge study on existing human-annotated datasets (JUDGE-BENCH, 20 datasets) found the best model (GPT-4o) reached only kappa = 0.28 +/- 0.32 on categorical judgments and Spearman rho = 0.50 +/- 0.21 on graded ones, with enormous task-to-task variance - and LLMs agreed MORE with non-expert annotators than with experts. EVIDENCE: Bavaresco et al., 'LLMs instead of Human Judges?' (ACL 2025). Structured/rule-like attributes correlate reliably; subjective attributes (engagingness) poorly; safety judgments produced negative correlations due to guardrail refusals. Authors conclude task-specific human validation is essential before deploying LLM judges. Read via detailed paper-note secondary source; consistent with the ACL publication. SOURCE: LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (JUDGE-BENCH) | https://en.papernotes.org/ACL2025/llm_nlp/llm_vs_human_judges_study/ | 2025-07 (ACL 2025) | academic IMPLICATION: Chance-corrected agreement with humans on subjective axes is far lower than the practitioner '85% agreement' narrative. The agree-more-with-non-experts finding is a red flag for the expertise-gap root cause: an uncalibrated judge may replicate crowd-level judgment, not expert judgment - expert-adjudicated (not crowd-consensus) gold labels are required for validation. ## Preference leakage is a measured, large contamination bias: a judge favors outputs of models trained on data from a related generator by up to 28.7 percentage points (Preference Leakage Score), with a relatedness gradient - same model 23.6% average, fine-tuned descendants 19-22%, same family 2.8-8.9% - and judges cannot detect their own students (near-chance) while a BERT classifier can (82.4%). EVIDENCE: Li et al., ICLR 2026 (arXiv 2502.01534). SFT transmits the bias strongly (23.6% avg PLS) vs DPO (5.2%); effect scales roughly linearly with synthetic-data mixing ratio with no threshold; style/format removal cuts PLS sharply; only contextual calibration mitigated it (17.8 to 7.3); prompting-based mitigation failed. SOURCE: Preference Leakage: A Contamination Problem in LLM-as-a-judge (ICLR 2026) | https://arxiv.org/abs/2502.01534 | 2026-02 (ICLR 2026; arXiv 2025-02) | academic IMPLICATION: Answers design Q5 self-preference: when attempters critique (or produce with the help of) outputs from model family X, the judge should not be family X or its distillation lineage - and since lineage is often undisclosed, cross-family judge routing plus style-stripping/canonicalization is the practical defense. Prompt-level debiasing will not fix this. ## Dedicated trained judges advanced substantially in 2025: Meta's J1 (RL-trained thinking-judge, May 2025) at 32B outperforms o1-mini, o3, and 671B DeepSeek-R1 on some judge benchmarks; CompassJudger-2-7B (July 2025) matches far larger generalists on judge benchmarks; Skywork-Reward-V2-Llama-3.1-8B (July 2025) topped seven reward benchmarks including JudgeBench. EVIDENCE: J1 arXiv 2505.10320 (verified): unified verifiable-reward RL over 22K synthetic pairs, mitigates position bias by construction, gains from test-time self-consistency. CompassJudger-2 arXiv 2507.09104: verifiable-reward + margin-loss training; 32B scores 80.9% accuracy on JudgerBenchV2. Skywork-Reward-V2 arXiv 2507.01352: 26M human-AI-curated preference pairs; 8B model led RewardBench v1/v2, PPE, RM-Bench, JudgeBench among reward models. SOURCE: J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (with CompassJudger-2, Skywork-Reward-V2) | https://arxiv.org/abs/2505.10320 | 2025-05 to 2025-07 | academic IMPLICATION: Small trained judges are now viable for high-volume, well-specified sub-checks (cost tier), but their wins are on preference-style benchmarks; for reasoning-heavy annotation QA the JudgeBench evidence still favors frontier reasoning models. A two-tier design (cheap trained judge for triage, frontier reasoner for verdicts) is supported; the J1 training recipe (verifiable rewards, position-bias-symmetric training) is reusable if a custom judge is ever trained. ## RewardBench 2 (Ai2, June 2025) made reward-model evaluation ~20 points harder via best-of-4 format (random = 25%) and unseen human prompts; top models score below 40% on Precise Instruction Following, yet benchmark scores correlate 0.87 (Pearson) with downstream best-of-N performance across 113 reward models. EVIDENCE: arXiv 2506.01937 (verified via abstract, Ai2 dataset page, and Lambert's analysis post). 1,865 cases across Factuality, Focus, Math, Precise IF, Safety, Ties. Lambert (author) notes generative LLM-as-judge approaches remain 'weaker than expected relative to standard reward models' at output ranking, and that frontier models still fail trivial rankings ('name a color in the rainbow'). SOURCE: RewardBench 2: Advancing Reward Model Evaluation | https://arxiv.org/abs/2506.01937 | 2025-06-02 | academic IMPLICATION: Precise-instruction-following is the weakest judged capability (<40% at best-of-4) - exactly the 'is this annotation compliant with written project instructions?' skill AutoQA needs. Do not assume instruction-compliance checking is a solved sub-problem; it needs its own gold set, and deterministic/programmatic checks should absorb whatever can be compiled to rules. ## In expert-knowledge domains, LLM judges agreed with subject-matter experts only 68% (dietetics, registered dietitians) and 64% (mental health, clinical psychologists) on overall pairwise preference, with agreement varying further across domain-specific aspect questions. EVIDENCE: Szymanski et al. (arXiv 2410.20266; ACM IUI 2025, ~220 citations). Mixed-methods pairwise comparison study; authors conclude LLMs alone lack the depth for complex knowledge-specific tasks and experts must stay in the evaluation loop. Full abstract text opened and confirmed. SOURCE: Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks | https://arxiv.org/abs/2410.20266 | 2024-10 (IUI 2025) | academic IMPLICATION: For projects requiring genuine domain expertise (root cause 2), expect judge-expert agreement in the 60s on holistic preference - below any reasonable autonomy bar. Expertise-heavy axes belong in the judge-assisted or human-only lane, and the axis-triage classifier (design Q2) should treat 'requires specialized professional knowledge' as an explicit routing feature. ## LLM judges are measurably less reliable on long-form outputs: LongJudgeBench (June 2026) finds current judges unstable across real-world long-form scenarios requiring document-level assessment of organization, coverage, and cross-section consistency, and rubrics or references help but are 'not always sufficient'. EVIDENCE: Chen et al., arXiv 2606.01629 (abstract opened and confirmed). First meta-evaluation benchmark targeting long-form judging specifically; systematically evaluates judges across base models and judging protocols; code public. Complements the short-form-centric older benchmarks (MT-Bench, LLMBar, JudgeBench). SOURCE: Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation (LongJudgeBench) | https://arxiv.org/abs/2606.01629 | 2026-06-01 | academic IMPLICATION: Attempter writeups/critiques are long-form documents; holistic single-pass judging of them inherits this instability. Supports the decomposition-first architecture (per-claim verification then aggregation) over holistic item scoring, and flags 'global coherence of the writeup' as a check that per-claim decomposition alone will miss (checklist myopia risk from design Q3 is real in both directions). ## Tools & artifacts - JudgeBench: Meta-evaluation benchmark (350 verified response pairs, objective correctness ground truth) with live HuggingFace leaderboard | https://github.com/ScalerLab/JudgeBench | Canonical hard test for whether a candidate AutoQA judge can detect actual errors rather than preferences; leaderboard at huggingface.co/spaces/ScalerLab/JudgeBench - RewardBench 2: Ai2's best-of-4 reward model benchmark (1,865 cases, 6 subsets incl. Factuality and Precise Instruction Following) | https://huggingface.co/datasets/allenai/reward-bench-2 | The Precise-IF and Factuality subsets are the closest public proxies for instruction-compliance and grounding checks in AutoQA - RubricEval + Rubric Arbitration Framework (RAF): Rubric-level meta-evaluation benchmark plus a multi-model arbitration pipeline for producing gold labels (85% human agreement, kappa 0.702) | https://www.alphaxiv.org/overview/2603.25133 | RAF's disagreement-cascade (base judges -> rationale re-evaluation -> reasoning meta-judges) is a directly reusable architecture for AutoQA gold-set production and escalation routing - J1 (Meta): RL training recipe for thinking-judges using verifiable rewards over synthetic pairs; position-bias-mitigating by construction | https://arxiv.org/abs/2505.10320 | The reference recipe if a custom project-tuned judge is ever trained; also evidence that judge chain-of-thought quality is the optimizable variable - CompassJudger-2 (7B/32B, open weights): OpenCompass generalist judge models trained with verifiable rewards + margin loss; includes JudgerBenchV2 (10k questions, 10 scenarios) | https://github.com/open-compass/CompassJudger | Best open-weight generative judge candidate for a cheap triage tier - Skywork-Reward-V2 (0.6B-8B, open weights): Scalar Bradley-Terry reward models trained on 26M human-AI-curated preference pairs | https://github.com/SkyworkAI/Skywork-Reward-V2 | State-of-art scalar scorers; useful for high-volume ranking/filtering but not for criteria-grounded verdicts with rationales - Atla Selene Mini (8B, open weights): General-purpose fine-tuned evaluator model (Jan 2025), promptable with custom criteria | https://arxiv.org/abs/2501.17195 | Open-weight promptable evaluator baseline; claims high agreement with domain experts on FinanceBench/CRAFT-MD but predates the 2025-26 judge generation - LongJudgeBench: First meta-evaluation benchmark for judging long-form outputs (June 2026), code public | https://github.com/cjj826/LongJudgeBench | Test harness for the long-form writeup judging AutoQA must do; measures rubric/reference sufficiency - llm-as-a-judge.github.io: Maintained living survey and paper list for the LLM-as-judge field (tracks 2025-2026 papers incl. Preference Leakage, ToolPRMBench) | https://llm-as-a-judge.github.io | Ongoing monitoring source for judge research; companion to Gu et al. 'A survey on LLM-as-a-judge' (The Innovation, 2026, ~1,667 citations) - Minimum Viable Validation Protocol (from arXiv 2606.19544): A distilled judge-validation protocol: chance-corrected agreement + consistency protocol + bias audit, from the 541k-judgment study | https://arxiv.org/abs/2606.19544 | Template for the AutoQA per-project ship gate: what to measure before any project's judge config goes live ## Disagreements - Dedicated trained judges vs frontier generalists: JudgeBench (ICLR 2025) shows fine-tuned judges and reward models plateau near random-to-64% on hard reasoning-based judging while frontier reasoning models lead (o1-preview 75.4%); but J1 (Meta, 2025-05), CompassJudger-2 (2025-07), and Skywork-Reward-V2 (2025-07) each claim small trained models beat o1-mini/o3-class models on judge/reward benchmarks; and Lambert (RewardBench 2 author, 2025-06) argues scalar reward models STILL beat generative LLM judges at output ranking. The conflict is largely benchmark-dependent (preference-style vs verifiable-reasoning tasks) - no source resolves which wins for criteria-based QA of human annotations, which resembles neither exactly. - Judge-human agreement levels: practitioner sources claim ~85% agreement 'higher than human-human agreement' (Confident AI, 2026) and '>80%, matching human-to-human levels' (Galileo); academic chance-corrected measurements show best-model kappa of 0.28 +/- 0.32 across 20 tasks (JUDGE-BENCH, ACL 2025), universal 33-41pp kappa deflation vs raw agreement (arXiv 2606.19544, 2026-06), and near-orthogonal judge-vs-human evaluation axes on subjective rubrics (arXiv 2606.03043, 2026-06). Much of the gap is percent-agreement vs chance-corrected statistics plus task subjectivity - the practitioner numbers are not wrong so much as unfalsifiably framed. - Verbosity bias magnitude: the 2606.19544 large-scale audit found verbosity bias small (<0.011) across 21 judges under a single controlled pairwise rubric, whereas the broader literature (Zheng 2023 MT-Bench, Gu et al. survey 2026, arXiv 2510.12462) treats verbosity bias as a major systematic failure. Possible reconciliation: bias magnitude depends heavily on rubric structure and elicitation format, meaning it is a controllable design variable rather than a constant. - Run-to-run consistency: The Coin Flip Judge (2026-04) reports 13.6% average pairwise flip rate and recommends ~11-vote majorities (mini-tier OpenAI judges only), while Reliability without Validity (2026-06) measured test-retest reliability >0.95 for some production judges - consistency is highly judge-specific, and the latter paper explicitly warns high consistency coexists with severe position bias (consistency is not validity). ## Gaps - No benchmark meta-evaluates judges on the AutoQA task itself - verifying HUMAN-written evaluative claims (grounding of praise, evidence-conclusion entailment in critiques, severity calibration). Closest proxies are rubric-verification (RubricEval), factuality judging (RewardBench 2 Factuality), and critique evaluation inside J1/CompassJudger training; the positive-claim/'warranted vs empty praise' verification question (design Q3) has no published precision/recall numbers anywhere I could find. - Could not obtain a verified July-2026 leaderboard snapshot for the newest frontier models (GPT-5.x, Claude Opus 4.x, Gemini 3) on JudgeBench or RewardBench 2 - both official leaderboards render via dynamic JS (HuggingFace Spaces) and did not yield data to fetches; the largest systematic evaluation covering 'the April 2026 frontier' (arXiv 2606.19544) reports cohort-level findings but does not name per-model winners in accessible text. Best-available anchor numbers are therefore early-2026 or older. - No single study reports judge-vs-adjudicated-expert agreement broken out per evaluation-axis type (factual grounding vs instruction compliance vs tone vs holistic) - the per-axis agreement ceiling matrix needed for design Q5's lane assignment must be assembled from fragments (RewardBench 2 subsets, JUDGE-BENCH task variance, RubricEval categories) or measured in-house. - Human inter-reviewer reliability floors (Krippendorff's alpha) for annotation-QA reviewers specifically were out of this domain's scope and not covered by the judge meta-evaluation literature; the rating-indeterminacy framework (NeurIPS 2025) is the right machinery but its 11 tasks are generic rating tasks, not training-data QA. - Atla (Selene) company status as of mid-2026 unverified - Selene Mini (8B, open weights, Jan 2025) claims remain reproducible from the model card, but I found no 2026 successor release, so treat Selene as a frozen artifact rather than a maintained product line. ## Verifications - CLAIM: On hard, objectively-verifiable judging tasks (JudgeBench, ICLR 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while GPT-4o scored 50.9-56.6% (near random), the best reward model (Skywork-Reward-Gemma-2-27B) hit 64.3%, and fine-tuned judges like PandaLM fell at or below random. VERDICT: confirmed | Every quantitative claim checks against the paper's own tables (arXiv:2410.12784 v2, ICLR 2025 camera-ready), read directly from the PDF: GPT-4o vanilla 50.86% / Arena-Hard 56.57% overall (Table 1), o1-preview 75.43% overall with 85.71% on both math and coding (Table 2), Skywork-Reward-Gemma-2-27B 64.29% as best reward model (Table 3, full range 59.43-64.29, paper says \"approximately 59% to 64%\" so the claim's \"60-64%\" is a minor round-up at the low end), PandaLM 13.14% (far below random; paper: all fine-tuned judges except Skywork significantly below random), ChatEval 34.00%. 350 verified pairs across knowledge (154)/reasoning (98)/math (56)/coding (42) confirmed - note this is the GPT-4o-generated split; a separate 270-pair Claude-3.5-Sonnet split exists. Dates confirmed: arXiv v1 2024-10-16, ICLR 2025, EmergentMind synthesis updated 2026-01-09. Two contextual nuances, neither refuting: (1) in the v2 tables o1-preview is not the single best judge - o3-mini (high) scores 80.86% and DeepSeek-R1 73.14%, which strengthens rather than weakens the \"reasoning models dominate\" thesis; (2) 2025-2026 follow-on work (J1, RM-R1, meta-judging, RRD) has pushed JudgeBench accuracies to 77-81%+, so \"GPT-4o near random\" describes the late-2024 snapshot of non-reasoning judges, not the current frontier. The claim as stated about the paper's findings is accurate. - CLAIM: Standard LLM-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial LLMs, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy. VERDICT: partially_confirmed | Core claim confirmed against the actual NeurIPS poster page (https://neurips.cc/virtual/2025/poster/117308): authors, rating-indeterminacy framing, forced-choice vs multi-label response-set elicitation, 11 tasks, 9 commercial LLMs, and the \"as much as 31% worse\" figure all match verbatim. Two corrections: (1) \"provably biased\" overstates the paper's own language - it says differing resolution of indeterminacy \"can heavily bias\" validation, and the 31% gap is empirical, not a theorem (the theory links performance measures/elicitation schemes); (2) the 2025-09-19 date is not shown on the poster page - plausibly the acceptance date, but unverified; the paper dates to arXiv March 2025 and NeurIPS Dec 2025, with an ML@CMU blog post 2025-12-09. Search for later work (as of July 2026) found follow-on papers (IRT-based judge reliability, arXiv 2602.00521; grading-scale alignment, arXiv 2601.03444) that build on, not refute or supersede, this result. Code: https://github.com/lguerdan/indeterminacy. CORRECTED: Guerdan, Barocas, Holstein, Wallach, Wu & Chouldechova (NeurIPS 2025; arXiv:2503.05965, first posted March 2025) show that standard LLM-judge validation via forced-choice human gold labels can be heavily biased when rating criteria admit multiple valid interpretations (\"rating indeterminacy\"): across 11 real-world rating tasks and 9 commercial LLMs, forced-choice-based validation selected judge systems performing as much as 31% worse than those selected by their framework using multi-label \"response set\" ratings. The bias mechanism is supported by a theoretical framework, but the 31% suboptimality figure is an empirical finding, not a proof. - CLAIM: The largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including April-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected Cohen's kappa is 33-41 percentage points lower on MT-Bench - and judge rankings shift by up to 14 positions depending on which benchmark you use. VERDICT: confirmed | Fetched https://arxiv.org/abs/2606.19544 directly. Every quantitative element checks out against the abstract: 21 judges from 9 providers, 118 runs / ~541,000 judgments across MT-Bench/JudgeBench/RewardBench; kappa deflation of 33-41 percentage points on MT-Bench across all 21 models; ranking shifts up to 14 positions across benchmarks; consistency-bias paradox (test-retest reliability >0.95 with position bias >0.10 in two production judges); verbosity bias <0.011; Minimum Viable Validation Protocol; April 2026 frontier models included. Submission date June 17, 2026 matches; authors Norman, Rivera, Hughes match. \"Largest to date\" is the authors' own framing, but two independent searches (July 14, 2026) found no larger or superseding meta-evaluation; follow-on work (AURA arXiv 2606.19714, Apple correlated-panels paper, \"Below the Reliability Floor\" on OpenReview) complements rather than supersedes it. Caveats: preprint with 0 citations; authors themselves limit dataset authority to the March-April 2026 measurement window due to hosted-endpoint drift."}, {"slug": "domain-annotation-quality", "title": "Domain Report - Annotation Quality", "category": "Domain research", "summary": "Annotation disagreement, rater uncertainty, LLM contamination, label error, and human-assistance effects.", "tags": ["domain report", "annotation quality"], "view": "documents/view/domain-annotation-quality.html", "raw": "documents/raw/research/domain-annotation-quality.md", "text": "# Domain: annotation-quality ## Roughly one-third of crowdworkers used LLMs on an LLM-advantaged text-production task (33-35% on Prolific, July 2023; ~34% Prolific self-report in a 2025 follow-up by Zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). [LOAD-BEARING] EVIDENCE: Veselovsky et al. ran two preregistered Prolific studies (n=168 and n=720, medical-abstract summarization) with a calibrated classifier plus self-report and keystroke heuristics; prevalence 33.3-35.4% baseline (95% CIs ~[26%,43%]); a 3x3 factorial of requests x hurdles halved but did not eliminate use. The June 2026 community survey (arXiv 2606.04924) corroborates: 30-40% estimated reliance and 34% Prolific self-report, and notes mitigations 'reduce rather than eliminate' LLM-assisted responses. SOURCE: Prevalence and Prevention of Large Language Model Use in Crowd Work (CACM; arXiv 2310.15683) | https://arxiv.org/abs/2310.15683 | 2025-02-18 | academic IMPLICATION: AutoQA must assume a contamination base rate of ~15-35% on any text-heavy attempter task even after friction measures; per-item content detection and 'please don't' policies cannot be the primary defense, and mitigation friction has a measurable quality cost that the QA layer must not misattribute to attempter skill. ## An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. [LOAD-BEARING] EVIDENCE: Westwood (PNAS, Nov 20 2025; 43,800 evaluations across 9 LLMs) shows instruction-following checks, logic puzzles, reCAPTCHA, and rare-event probes all fail; persona calibration even masks superhuman-accuracy tells. Recommended defenses are entirely provenance/economics-based: identity revalidation, per-worker throttling, completion-history transparency, pass/fail history, location/VPN checks - 'there is no magical fix.' The 2026 crowdsourcing survey (arXiv 2606.04924) independently confirms detectors degrade under paraphrase and out-of-domain, and mixed human-AI authorship is harder still. SOURCE: The potential existential threat of large language models to online survey research (PNAS 122(47)) | https://www.pnas.org/doi/10.1073/pnas.2518075122 | 2025-11-20 | academic IMPLICATION: Directly answers taxonomy Q8: wholesale-LLM-submission defense must be economics/provenance-based (rate limits, work-history consistency, randomized deep audits, process telemetry), not per-item text classification; any AutoQA spam layer built on stylistic detection will be silently beaten within the cost asymmetry. ## Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. [LOAD-BEARING] EVIDENCE: Elangovan et al. (Amazon), ICLR 2025 (arXiv 2410.03775, opened and verified): Krippendorff's alpha / kappa were designed for human-human reliability and their assumptions break for machine labels; the paper's fix is (1) stratify meta-evaluation by human-label uncertainty, (2) use binned Jensen-Shannon divergence for perception-type (inherently subjective) tasks, (3) report 'perception charts' instead of one correlation. Code open-sourced (amazon-science/BeyondCorrelation). SOURCE: Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025) | https://arxiv.org/abs/2410.03775 | 2025-05-01 | academic IMPLICATION: Answers taxonomy Q1 directly: meta-evaluation must stratify gold/validation items by measured human agreement level and report per-stratum judge performance; 'accuracy vs. humans' as a single number should be banned from reporting, and judge validity claims are only meaningful on the high-human-agreement stratum. ## In human preference/rating data, the majority of annotator disagreements are attributable to task underspecification and response-style preferences - not random noise and not primarily expertise gaps - and standard aggregation (Bradley-Terry reward modeling) plus LLM-as-judge evaluation both fail to account for this divergence. [LOAD-BEARING] EVIDENCE: Zhang et al., ICML 2025 (arXiv 2410.14632, opened and verified): built a 10-category, 4-class taxonomy of disagreement sources on preference datasets; abstract states 'the majority of disagreements are due to factors such as task underspecification or response style,' explicitly challenging the noise assumption; they show LLM evaluations are heavily swayed by divisive style features and develop divergence-identification methods. Converges with Jiang & de Marneffe (TACL 2022) who found the same structure in NLI. SOURCE: Diverging Preferences: When do Annotators Disagree and do Models Know? (ICML 2025) | https://arxiv.org/abs/2410.14632 | 2025-07-15 | academic IMPLICATION: Supports root-cause-1 primacy and taxonomy Q2: the verdict ontology needs a first-class 'instructions underdetermine this case' class detected via measured divergence, and a cheap variance-attribution pilot is empirically justified because published decompositions find underspecification (fixable by rubric compilation) dominates over noise (fixable by averaging). ## Even on a nominally objective task (NLI), annotator disagreement decomposes into a 10-category taxonomy across three high-level sources - uncertainty in sentence meaning, underspecification in guidelines, and annotator behavior - proving that a fraction of inter-reviewer variance is item-intrinsic and survives perfect rubric operationalization. EVIDENCE: Jiang & de Marneffe, TACL 2022, developed and hand-applied the taxonomy to high-disagreement NLI items; disagreement persisted among trained annotators on items with genuine interpretive ambiguity. D3CODE (EMNLP 2024; 4.5K sentences, 4K+ annotators, 21 countries) adds that residual variance is systematically structured by annotators' moral values and region, not random. SOURCE: Investigating Reasons for Disagreement in Natural Language Inference (TACL 10) | https://aclanthology.org/2022.tacl-1.78/ | 2022-12-01 | academic IMPLICATION: The target metric cannot be raw agreement: some disagreement is item-intrinsic signal. The variance-attribution pilot should code disagreements against a taxonomy like this to split rubric-fixable variance (underspecification) from irreducible variance (ambiguity/values) before setting build order. ## Chance-corrected agreement coefficients (Cohen's kappa, and analogously Krippendorff's alpha) can be near zero despite very high raw agreement when class prevalence is extreme (the Feinstein-Cicchetti paradoxes), so under the 90%+ pass rates typical of QA pipelines, single-coefficient reliability reporting is misleading in the opposite direction from raw percent agreement. EVIDENCE: Feinstein & Cicchetti (J Clin Epidemiol, 1990) formalized the two kappa paradoxes (high agreement/low kappa under imbalance; asymmetric marginals inflating kappa); the effect is prevalence-driven, extensively replicated (Quarfoot & Levine, Am. Statistician 2016; Gwet's AC1 literature). This is exactly the regime of QA pass/fail data with low failure base rates. SOURCE: High agreement but low kappa: I. The problems of two paradoxes (Feinstein & Cicchetti) | https://doi.org/10.1016/0895-4356(90)90158-L | 1990-01-01 | academic IMPLICATION: Answers taxonomy Q1 stats question: standardize on reporting a chance-corrected coefficient PLUS per-class recall at fixed prevalence PLUS CIs; ban both raw percent-agreement AND any single coefficient in isolation, because at extreme pass rates alpha/kappa collapse even for a well-functioning judge. ## Classical label-aggregation models (Dawid-Skene 1979, MACE 2013 with its explicit spammer latent variable, GLAD) all assume annotators are conditionally independent given the true label - an assumption measurably violated when the 'annotators' are LLM judges sharing data/architectures/prompts, causing miscalibrated posteriors and confidently wrong aggregate verdicts; Feb 2026 work replaces this with dependence-aware Ising-model aggregation. EVIDENCE: Balasubramanian, Podkopaev & Kasiviswanathan (Amazon/UC Davis, arXiv 2601.22336, Feb 2 2026) state the CI assumption 'is often violated by LLM judges due to shared data, architectures, prompts, and failure modes' and that ignoring dependencies 'can yield miscalibrated posteriors and even confidently incorrect predictions'; they build aggregation via Ising models. MACE (Hovy et al., NAACL 2013) remains the canonical spam-aware human aggregator, with production implementations in Toloka's crowd-kit. SOURCE: Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models (arXiv 2601.22336) | https://arxiv.org/pdf/2601.22336 | 2026-02-02 | academic IMPLICATION: If AutoQA uses k-sample voting or multi-judge ensembles, naive Dawid-Skene/majority aggregation will overstate confidence because same-family judge samples share failure modes; confidence estimates must model correlated errors (or use genuinely different model families), and MACE-style competence/spam latent variables remain the right tool for the human-reviewer side. ## LLM contamination in low-dimensional annotation data (multiple-choice labels, ratings) can be evaluated at the worker level without any ground truth, using peer-prediction scores that condition on LLM-generated reference labels - a training-free mechanism with theoretical guarantees under an LLM-collusion model, published at NeurIPS 2025 - because text-based detectors are inapplicable to short label outputs. EVIDENCE: Zhang, Pang, Zhu & Liu (NeurIPS 2025; arXiv 2506.06991, opened and verified): existing detectors need high-dimensional text; their mechanism scores the informativeness of worker answers via inter-worker correlation conditioned on a subset of LLM labels, explicitly modeling collusion (multiple workers pasting from the same LLM), and empirically detects low-effort cheating on real crowdsourcing datasets. Scoring is per-worker, not per-item. SOURCE: Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth (NeurIPS 2025) | https://arxiv.org/abs/2506.06991 | 2025-12-01 | academic IMPLICATION: For rating/label-type attempter outputs where stylistic detection is impossible, the viable contamination sensor is worker-level statistical scoring (cross-worker correlation vs. an LLM-conditioned baseline) run continuously - reinforcing that provenance/behavioral defense operates at the annotator level, not the item level. ## The NLP field has institutionalized disagreement-as-signal: the third Learning-With-Disagreements shared task (LeWiDi-2025, EMNLP 2025) standardizes dual evaluation - soft-label (predict the population label distribution, Wasserstein/Manhattan distance) and perspectivist (recover individual annotators' labels) - across four datasets that all ship per-annotator labels rather than adjudicated gold. EVIDENCE: Leonardelli et al. (arXiv 2510.08460, Oct 2025; le-wi-di.github.io) describe the two complementary paradigms and public leaderboard; lineage runs from Aroyo & Welty's 'truth is a lie' perspectivism through DICES (NeurIPS 2023 safety dataset with up to 100+ raters/item) and jury learning (Gordon et al., CHI 2022, which composes verdicts from explicitly selected rater subpopulations); Frenda et al.'s survey (LREV 2024) documents the dataset ecosystem. SOURCE: LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task | https://arxiv.org/pdf/2510.08460 | 2025-10-09 | academic IMPLICATION: The AutoQA data model should preserve rater identity and full label distributions (never only adjudicated verdicts), and for judgment-laden axes the judge's target should be the distribution/jury composition, not a forced scalar - this is now the field-standard evaluation contract, giving ready-made metrics (Wasserstein/Manhattan soft-label distance) for 'consistency-of-application' monitoring. ## Gold/test labels themselves are wrong at material rates - a lower-bound average of 3.3% label errors across 10 canonical benchmarks (at least 6% in the ImageNet validation set), validated by human review of confident-learning-flagged candidates (51% flag precision) - and error rates this size are enough to flip model rankings. EVIDENCE: Northcutt, Athalye & Mueller (NeurIPS 2021 Datasets & Benchmarks; arXiv 2103.14749, opened and verified): algorithmic flagging via confident learning + crowdsourced verification; ResNet-18 beats ResNet-50 on corrected ImageNet if mislabeled prevalence rises just 6%. Cleanlab is the maintained open-source implementation. SOURCE: Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (NeurIPS 2021 D&B) | https://arxiv.org/abs/2103.14749 | 2021-11-01 | academic IMPLICATION: Answers taxonomy Q1's false-agreement question in the only measured form available: several percent of any 'gold' set is wrong, so judge-vs-gold disagreement at that rate is uninformative; the gold set needs its own standing audit channel (confident-learning-style flagging + expert re-adjudication) as a permanent component, and judge certification margins must exceed the gold error floor. ## Showing annotators LLM suggestions makes them anchor on the suggestions - significantly shifting the label distribution versus unassisted baseline, raising self-reported confidence without making them faster - and evaluating models against LLM-assisted labels significantly inflates reported model performance. EVIDENCE: Schroeder, Roy & Kabbara (Findings of ACL 2025, July 2025; pre-registered, 350 annotators, 7,000 annotations, 4 conditions x 2 models x 2 datasets, opened and verified): annotators 'strongly took the LLM suggestions'; the paper warns homogenization/lessened variation is a direct risk when LLMs are inserted into annotation loops, and that 'human-approved' LLM-assisted labels change downstream conclusions. SOURCE: Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks (Findings of ACL 2025) | https://aclanthology.org/2025.findings-acl.1323/ | 2025-07-27 | academic IMPLICATION: Direct evidence for taxonomy Q8 monoculture and Q9 induced drift: if AutoQA feedback (or any AI verdict) is visible to attempters/reviewers before they judge, distributions converge on judge-pleasing labels and self-evaluation inflates - so human touches must be blind-first, and population-level label-distribution drift is the right monoculture sensor. ## RLVR-style reasoning training degrades an LLM's ability to model human annotator disagreement, while naive chain-of-thought on RLHF models improves it - evaluated across 60 setups on 3 tasks - meaning stronger 'reasoning' judges are measurably worse at knowing when humans legitimately diverge. EVIDENCE: Ni et al., EACL 2026 main (arXiv 2506.19467 v3, Jan 12 2026, opened and verified): systematic evaluation over model sizes, distribution-expression and steering methods using variance-correlation and distributional-alignment metrics; authors explicitly warn about 'the potential risk of replacing human annotators with reasoning LLMs' where disagreement carries signal; author communication adds that small fine-tuned models (ModernBERT-class) with human labels often beat large reasoning models at this. SOURCE: Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? (EACL 2026) | https://arxiv.org/abs/2506.19467 | 2026-01-12 | academic IMPLICATION: For the 'instructions-underdetermine / route-to-human' triage lane, do not assume the strongest reasoning judge is the best ambiguity detector: disagreement-prediction may need a dedicated small fine-tuned model (or CoT-on-RLHF configuration) separate from the verdict judge. ## Tools & artifacts - Cleanlab (confident learning): Open-source library implementing confident learning for algorithmic label-error detection in labeled datasets | https://github.com/cleanlab/cleanlab | Ready-made machinery for the standing gold-set audit channel: flag suspect gold/QA labels for expert re-adjudication (51% flag precision measured in the NeurIPS 2021 study). - crowd-kit (Toloka): Maintained Python library of aggregation/truth-inference algorithms: Dawid-Skene, GLAD, MACE, Wawa, ZeroBasedSkill, IBCC/EBCC | https://github.com/Toloka/crowd-kit | Production implementations for aggregating multiple human reviewers (and multi-judge samples) with per-worker competence and spammer modeling - but note the conditional-independence caveat for same-family LLM judges. - LeWiDi-2025 datasets + leaderboard: Four benchmark datasets with per-annotator labels plus standardized soft-label (Wasserstein/Manhattan) and perspectivist evaluation metrics, EMNLP 2025 shared task | https://le-wi-di.github.io/ | Off-the-shelf metrics and data format for evaluating disagreement-aware judge components and for the 'consistency-of-application vs agreement-with-truth' distinction. - DICES dataset (Google): Diversity-in-Conversational-AI-Safety dataset: 990 conversations with dense parallel safety ratings (up to 100+ demographically diverse raters per item) | https://github.com/google-research-datasets/dices-dataset | Reference dataset for measuring how much rater variance survives on a judgment-laden axis with a fixed rubric - useful for calibrating the variance-attribution pilot. - BeyondCorrelation (Amazon Science): Code for uncertainty-stratified meta-evaluation of automatic evaluators: perception charts, binned Jensen-Shannon divergence | https://github.com/amazon-science/BeyondCorrelation | Directly implements the recommended meta-evaluation reporting standard (stratify by human agreement) for certifying the AutoQA judge. - Can Crowdsourcing Survive the LLM Era? (community survey): June 2026 arXiv survey (2606.04924) synthesizing prevalence, detection, and mitigation literature on LLM use in human data collection | https://arxiv.org/html/2606.04924v1 | Most current single map of the LLM-contamination literature; documents detector degradation under paraphrase and the state of mitigation practice as of mid-2026. ## Disagreements - Detectability of LLM-generated submissions: Veselovsky et al. (2023/2025) achieved usable classifier-based detection on a long-form text task, and TU Delft's prompt-injection work (TOCHI 2025) claims in-instrument countermeasures work; Westwood (PNAS 2025) and the community survey (arXiv 2606.04924, June 2026) show detection and quality checks are comprehensively evaded by agents and degrade under paraphrase/out-of-domain, with mixed human-AI authorship hardest. The recent evidence favors Westwood, but the sources genuinely conflict on whether content-side detection retains any value as a secondary signal. - Nature of disagreement: the aggregation tradition (Dawid-Skene, MACE, majority vote) models disagreement as noise around one latent truth to be recovered; the perspectivist tradition (Aroyo & Welty, DICES, LeWiDi-2025, Diverging Preferences) treats much of it as legitimate signal to be preserved. Neither side claims universality, but they prescribe opposite data models (adjudicated gold vs. per-rater distributions) - the design must pick per-axis, and no source offers a validated automatic classifier for which regime an axis falls into. - Agreement statistics under skew: Krippendorff defends alpha as the general-purpose reliability standard; Gwet and successors show alpha/kappa collapse under extreme prevalence and push AC1/AC2; Elangovan et al. (ICLR 2025) argue all human-reliability coefficients are misapplied to machine-vs-human comparison in the first place. There is no consensus single statistic - only convergence that one-number reporting is wrong. - Mitigation of crowdworker LLM use: Veselovsky et al. measured that request+hurdle interventions halve usage (a meaningful effect worth deploying); Westwood concludes no instrument-level mitigation withstands agent-based completion ('no magical fix'). These are consistent about direction (partial reduction) but conflict sharply on whether prevention is worth its measured quality cost. ## Gaps - No published measurement of the false-agreement rate (judge and attempter agreeing while both wrong per expert panel) in an annotation-QA setting; the nearest proxy is Northcutt's 3.3%+ gold-label error floor. This number would have to be produced in-house via an independent expert-audit channel. - No public 2025-2026 prevalence data for LLM use specifically among paid annotators/reviewers on managed AI-training-data platforms (Scale, Surge, Outlier, Prolific-managed RLHF work); all published prevalence numbers come from MTurk/Prolific survey and summarization tasks, which may under- or over-state prevalence for critique/rating work. - Item-level (as opposed to worker-level) detection of LLM-generated annotations for short/low-dimensional outputs remains unsolved; the NeurIPS 2025 peer-prediction mechanism scores workers, not items, and requires overlapping assignments across workers. - No annotation-science source found that gives sample-size arithmetic for certifying a judge against a gold set under low failure base rates (per-project vs pooled hierarchical validation); this appears only in general eval-statistics literature outside this domain's charter. - No longitudinal studies found measuring whether QA feedback changes annotator behavior over time (repeat-error rates per criterion); the feedback-efficacy question (taxonomy Q7) has essentially no direct evidence in the annotation-quality literature. - Could not open the full text of the Diverging Preferences taxonomy to extract exact percentage splits across its 10 disagreement categories (abstract confirms only that underspecification + style form the majority); exact fractions would sharpen the variance-attribution prior. ## Verifications - CLAIM: Roughly one-third of crowdworkers used LLMs on an LLM-advantaged text-production task (33-35% on Prolific, July 2023; ~34% Prolific self-report in a 2025 follow-up by Zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). VERDICT: partially_confirmed | Core claim VERIFIED against the full PDF of arXiv 2310.15683 (Veselovsky, Horta Ribeiro, Cozzolino, Gordon, Rothschild, West): Study 1, n=168 Prolific workers, 3 July 2023, medical-abstract summarization; prevalence 33.3% [25.9,40.1] (classify-and-count), 35.2% [29.8,40.6] (probabilistic), 35.4% [27.8,43.0] (corrected) - matches \"33.3-35.4%, CIs ~[26%,43%]\". Study 2, n=720, 23 July 2023, 3x3 factorial (request: none/indirect/direct x hurdle: none/image/no-Ctrl-C+V); classifier estimate dropped 27.6% -> 15.9% (direct+image) and 15.8% (direct+Ctrl C+V) - matches \"roughly halved, not eliminated\" (paper: \"reduced LLM use by nearly 50%, it could not fully prevent it\"). Direct request decreased keyword retention by 6.2% (p=0.009) - exact match. Classifier was finetuned e5-base-v2, calibrated (Card & Smith), with self-report and high-precision heuristics (completion time + paste artifacts) - matches. The June 2026 community survey arXiv 2606.04924 (Velutharambath et al., 155 researchers, survey run Aug 2025-Mar 2026) does cite \"Veselovsky et al. estimate 30-40%\", cites \"Zhang et al. (2025): 34% of Prolific participants self-report using LLMs for open-ended questions\", and states mitigations \"reduce rather than eliminate LLM-assisted responses\" - so the follow-up attribution to Zhang et al. 2025 is consistent with that survey's citations (note: I verified the citation exists in the survey, not the underlying Zhang et al. paper itself). ISSUES: (1) Date \"2025-02-18\" is wrong for the arXiv link given - arXiv has only v1, dated 24 Oct 2023; the published version is Communications of the ACM 68(3):42-47, March 2025 issue (DOI 10.1145/3685527). (2) \"Preregistered\" is not supported by the paper text - no preregistration is mentioned in the arXiv PDF. (3) Minor nuance: \"never to zero\" - by the classifier measure true, but the high-precision heuristics table actually shows 0.0% in the direct+Ctrl C+V cell; the paper's own framing (\"cannot fully prevent\") still supports the claim's spirit. (4) 44% of surveyed researchers in the 2026 survey observed LLM use in their data - newer corroborating, not superseding, evidence; no later work found that overturns the prevalence estimates. Sources: https://arxiv.org/abs/2310.15683, https://arxiv.org/abs/2606.04924, https://dl.acm.org/doi/10.1145/3685527. CORRECTED: Roughly one-third of Prolific crowdworkers used LLMs on an LLM-advantaged text-production task (33.3-35.4% across three estimators, July 2023, n=168; Veselovsky et al., arXiv 2310.15683 v1 posted 24 Oct 2023, published in Communications of the ACM 68(3), March 2025). In a second study (n=720, 3x3 factorial), the best mitigations (direct request + image presentation or + copy-paste disable) roughly halved classifier-estimated usage from 27.6% to 15.9%/15.8% without eliminating it, and directly requesting non-use reduced keyword retention by 6.2% (p=0.009). The studies were not described as preregistered. A June 2026 community survey of 155 researchers (arXiv 2606.04924, Velutharambath et al.) corroborates, citing Veselovsky et al.'s 30-40% estimate and Zhang et al. (2025)'s 34% Prolific self-report, and finding 44% of researchers observed LLM use in their crowdsourced free-text data; it echoes that mitigations reduce rather than eliminate LLM-assisted responses. - CLAIM: An autonomous LLM agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused AI-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. VERDICT: confirmed | Every quantitative element of the CLAIM verifies against the primary source (Westwood, \"The potential existential threat of large language models to online survey research,\" PNAS 122(47), published Nov 20, 2025; PubMed 41264250, cited 81x by Jul 2026; won a 2025 Cozzarelli Prize). Extracted paper text confirms verbatim: (1) 99.8% pass rate across 6,000 trials (20 ACQ types x 300 repetitions), only 10 errors, perfect on 18/20 check types; (2) execution layer \"simulates realistic reading times calibrated to the persona's education level, generates human-like mouse movements, and types open-ended responses keystroke-by-keystroke, complete with plausible typos and corrections\"; (3) strategically declines 97.7% of \"reverse shibboleth\" tasks; (4) ~$0.05 per survey with commercial models vs $1.50 payment (>96.8% margin), marginal cost near zero with local open-weight models; (5) 43,800 total evaluations, 139 questions, 6,700 trials, with a key subset validated across nine LLMs (DeepSeek R1, Mistral Large, Claude 3.7 Sonnet, Grok 3, Gemini 2.5, Qwen Plus, Llama 4 Maverick, GPT-4.1, o4-mini); (6) 0% error rate on impossible-event/rare-event probes. The paper's abstract and press framing support the interpretive conclusion that coherence-/content-based detection is no longer tenable and defenses must be provenance/economic. Supersession check: no 2026 work refutes it - follow-ups (Westwood & Frederick 2026 Prolific audit; a 2026 SAGE AMPPS review of AI-mediated contamination) build on it and treat detection as an unsolved provenance problem. Two minor caveats, both in the EVIDENCE SUMMARY rather than the claim: (a) arXiv 2606.04924 (\"Can Crowdsourcing Survive the LLM Era?\", Jun 2026) is a community survey of 155 researchers' practices/experiences, not a technical detector benchmark; its abstract does not state the paraphrase/out-of-domain/mixed-authorship degradation findings attributed to it (those likely come from a different detection-benchmark paper). (b) On reCAPTCHA the paper says the system \"is designed to accommodate tools for bypassing\" reCAPTCHA - an architectural capability, slightly weaker than \"reCAPTCHA... fail[s]\" as an empirically demonstrated result. Neither caveat affects the claim text itself, which is fully accurate. - CLAIM: Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. VERDICT: partially_confirmed | Substance CONFIRMED on every checked point against the arXiv record (https://arxiv.org/abs/2410.03775): (1) abstract states machine labels \"may superficially appear to have similar or better correlation with the human majority\" when human uncertainty is high, and machine-human correlation falls \"well below\" human-human correlation as human label consistency increases; (2) it flags Krippendorff's alpha / Randolph's kappa as designed for human-human reliability with assumptions inapplicable to machine labels; (3) proposed fixes match exactly - stratification by human label uncertainty, binned Jensen-Shannon divergence for perception-type tasks, and 'perception charts'; (4) venue is ICLR 2025 per the arXiv comments field; (5) code is open-sourced at github.com/amazon-science/BeyondCorrelation. The only inaccuracy is the SOURCE date: the claim says 2025-05-01, but the latest arXiv version is v3 dated 2025-01-27 (original submission 2024-10-03) - no May 2025 version exists on the record. Supersession check: a follow-up search surfaced related later work - \"Validating LLM-as-a-Judge Systems under Rating Indeterminacy\" (arXiv 2503.05965), \"Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking\" (arXiv 2604.11581), and \"CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation\" (arXiv 2603.00039, Feb 2026). These extend and reinforce the human-uncertainty/measurement-error critique rather than refute it; nothing found supersedes or contradicts the core finding. Verdict is partially_confirmed solely because of the incorrect source date; the claim itself stands. CORRECTED: Correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. Source: Elangovan et al., \"Beyond correlation...\" (Amazon Science), accepted at ICLR 2025, arXiv 2410.03775 (v1 2024-10-03, latest v3 2025-01-27), code at github.com/amazon-science/BeyondCorrelation."}, {"slug": "domain-critique-models", "title": "Domain Report - Critique Models", "category": "Domain research", "summary": "CriticGPT lineage, human-plus-critic evidence, omission limits, and feedback validation.", "tags": ["domain report", "critique models"], "view": "documents/view/domain-critique-models.html", "raw": "documents/raw/research/domain-critique-models.md", "text": "# Domain: critique-models ## Human+critic-model teams occupy a strictly better operating point than either alone: in OpenAI's CriticGPT study, Human+CriticGPT teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (FSBS length penalty), not a fixed property. [LOAD-BEARING] EVIDENCE: CriticGPT critiques were preferred over human contractor critiques in 63% of cases on naturally occurring LLM errors (>80% Elo win rate on inserted bugs); both ChatGPT and CriticGPT caught substantially more inserted bugs than paid contractors (median ~50 min/review, ~5 yrs Python experience). Humans alone had far fewer hallucinated bugs/nitpicks; Human+CriticGPT teams landed between, 'moving beyond the model-only Pareto frontier.' Force Sampling Beam Search scores candidates by rm_score + LENGTH_MODIFIER x num_highlights, tracing a precision/comprehensiveness Pareto curve selectable at deployment without retraining. Verified directly from the arXiv HTML full text. SOURCE: LLM Critics Help Catch LLM Bugs (CriticGPT, OpenAI) | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary IMPLICATION: Validates the hybrid one-touch foundation: AI critic drafts findings, human filters hallucinated nitpicks. Make the precision/comprehensiveness operating point an explicit per-project config knob (Q2/Q4/Q7 false-accusation SLO), since the same trained judge can be run strict or comprehensive per project. ## Critic-assisted review of items already rated 'flawless' by a first human found real, rating-changing problems in 24% of flagged cases, versus 6% found by an unassisted second human review of the same data - a 4x lift on the exact 'confident pass' slice. EVIDENCE: CriticGPT (RL-only) was run over ChatGPT training data rated flawless by a first annotator; where the critique flagged a problem, reviewers agreed it substantially lowered the rating in 24% of cases, vs 6% in a no-critique second-rater replication - and this held out-of-distribution (mostly non-code tasks). Prioritizing by critique reward-model score further raised catch rates and cut hallucinations. Verified from arXiv HTML full text. SOURCE: LLM Critics Help Catch LLM Bugs (CriticGPT, OpenAI) - 'flawless' data experiment | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary IMPLICATION: Direct evidence for Q1's false-agreement problem and Q6's routing: unassisted human re-review of passed items misses ~3/4 of what critic-assisted review catches, so the standing expert-audit channel over confident passes should itself be critic-assisted, and critic-confidence scores can prioritize the audit queue. ## In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong. [LOAD-BEARING] EVIDENCE: 1,918 expert-labeled fact-verification items; tool-using AI rater with majority-vote-over-50-samples confidence (well calibrated). AI 87.7% vs individual humans 75.1% (majority vote 80.6%). Hybridization at threshold 0.62: 89.3%; on the routed low-confidence slice humans beat the AI 71.3% vs 60.5% (real complementarity region). Evidence-only assistance: 79.3% when AI right (vs 71.3% baseline) with no significant harm when AI wrong; conditions showing Judgments+Reasoning/Confidence produced significant over-reliance decrements (e.g. beta = -0.768, p < .001); AI-debate condition was numerically worst. Verified from arXiv HTML full text. SOURCE: Human-AI Complementarity: A Goal for Amplified Oversight (Google DeepMind) | https://arxiv.org/abs/2510.26518 | 2025-10-30 | primary IMPLICATION: Answers Q6 concretely: spend the one human touch on AI-low-confidence items, and give that human the judge's collected evidence spans WITHOUT the judge's verdict or rationale (blind-ish adjudication over curated evidence). Requires the judge to emit calibrated confidence and quoted evidence as first-class outputs (Q3), and argues against 'human verifies AI verdict' as the default interaction. ## A Nature Human Behaviour meta-analysis of 100+ experiments (300+ effect sizes) found human-AI combinations on average performed significantly WORSE than the best of human or AI alone on decision/judgment tasks, with gains only where humans outperformed the AI solo - naive human-checks-AI designs destroy value. EVIDENCE: Vaccaro, Almaatouq & Malone systematic review/meta-analysis: average human-AI combo underperformed the stronger party alone; losses concentrated in decision-making tasks, gains in content-creation tasks; combos helped when humans beat AI alone. DeepMind's complementarity paper explicitly frames its confidence-routing + evidence-only design as the answer to this result. Verified from arXiv abstract. SOURCE: When combinations of humans and AI are useful: A systematic review and meta-analysis | https://arxiv.org/abs/2405.06087 | 2024-10-28 | contrarian IMPLICATION: The Q6 fork ('is one shallow touch better than zero touch plus deep audits?') is empirically live: an automation-biased shallow verify step is the modal failure in the literature. The touch only pays where the human plausibly beats the judge (routed low-confidence/contested items), which mandates closed-loop routing on observed overturn rates rather than fixed per-item review. ## In the largest deployed RCT of AI critiquing human evaluative work (ICLR 2025, feedback on >20,000 peer reviews), 27% of reviewers who received LLM feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated LLM reliability tests. [LOAD-BEARING] EVIDENCE: Randomized controlled study at ICLR 2025 (~22.5k feedback-arm vs ~22.4k control reviews): Review Feedback Agent flagged vague comments, content misunderstandings, unprofessional remarks; guardrail LLM tests gated delivery; 26.6-27% update rate, avg +80 words among updaters, blinded informativeness gains, longer author-reviewer discussions. Verified from arXiv abstract page; corroborated by the official ICLR blog. SOURCE: Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 | https://arxiv.org/abs/2504.09737 | 2025-04-13 | academic IMPLICATION: Q7's central bet is supported at deployment scale: constructive, item-specific AI feedback measurably changes the behavior of human evaluators, and the shipped pattern held feedback to its own automated verification gate before delivery (fail-closed feedback). Also gives a realistic effect-size prior: ~1 in 4 recipients act on feedback - plan longitudinal metrics around that base rate. ## LLMs are structurally weak at detecting omissions: on AbsenceBench, average F1 drops 56.9 points versus detecting the same content as insertions (best model 71.2% overall, 40.0% on code diffs, vs ~99.5% needle-in-haystack), and inserting explicit placeholders at gap sites recovers ~35.7 points - absence has no attention key to attend to. [LOAD-BEARING] EVIDENCE: 4,302 instances across poetry, numerical sequences, GitHub PRs (~5K token contexts): models must list deliberately deleted elements given original+modified documents. Gemini-2.5-flash (thinking) best at 71.2% avg F1; Mixtral-8x7B scores 99%+ on NIAH but 14.7% here; thinking tokens buy only ~7.9%; '<missing line>' placeholders add up to +81.8%. NeurIPS 2025 Datasets & Benchmarks. Verified from arXiv HTML full text. SOURCE: AbsenceBench: Language Models Can't Tell What's Missing | https://arxiv.org/abs/2506.11440 | 2025-06-13 | academic IMPLICATION: Caps what the judge can verify autonomously (Q3/Q5): 'comprehensiveness'/'complete' praise claims and omission-type attempter failures are the judge's structurally weakest lane. The rubric compiler should convert completeness criteria into explicit enumerable checklists (placeholder effect = give absence a token), and free-form omission judgment belongs in the human or human-assisted lane, not judge-autonomous. ## Critique quality is itself quantifiable at usable reliability: MetaCritique decomposes critiques into atomic information units and scores precision/recall against references, with GPT-4 AIU-level judgments at 85-89% accuracy and Meta-F1 correlating with human gold at Pearson 0.84-0.89 - and measured baselines show human critiques are high-precision/low-recall (87.6%/48.7%) while LLM critiques are the inverse-ish (71.9%/53.3%). EVIDENCE: ACL Findings 2024 framework: precision = fraction of critique AIUs judged factual; recall = coverage of reference-critique AIUs; each judgment carries a natural-language rationale. Choosing critiques by Meta-F1 yields refinements that win 51%/lose 27% by human eval, vs GPT-4 pairwise picks losing more than winning. ~28% of LLM-critique AIUs were non-factual vs ~12% for humans; LLM critiques carried 2.4x the information volume (8.10 vs 3.31 AIUs). Verified from arXiv HTML full text. SOURCE: The Critique of Critique (MetaCritique) | https://arxiv.org/abs/2401.04518 | 2024-01-09 | academic IMPLICATION: Q7's reflexive-grounding requirement is buildable today: run AutoQA feedback through AIU-level precision scoring as its own admission gate, and report critique-precision/recall as first-class system metrics. Also sets expectations: the AI side of the pipeline supplies coverage, the human side supplies precision - mirroring CriticGPT. ## Verdict accuracy and critique validity dissociate badly: in CriticBench-THU data 24.8% of items got the correct verdict with a low-quality critique, and open-loop verdict-agreement metrics compress a 27.1-point real error-identification gap (ProcessBench: o1-mini 88.9 vs Qwen2.5-72B 61.8) into a 1.3-point verdict-F1 gap - so critique quality must be evaluated closed-loop by whether the critique drives a successful correction. EVIDENCE: RealCritic (Jan 2025, arXiv 2501.14492): closed-loop critique-then-correct design over 8 reasoning benchmarks; classical LLMs (incl. GPT-4) lose accuracy under self-critique (-1.8 to -5.1 avg; up to -35.6 domain drops) while o1-mini is the only model with positive self-critique delta; failure modes 'superficial success' (right verdict, wrong analysis) and 'contradictory output' documented. Independently confirmed at scale in 2026: an ICML 2026 paper (RM-NLHF, arXiv 2601.07349) finds outcome-rewarded generative reward models 'guess correct outcomes without sound critiques' and uses similarity-to-human-critique as a process reward to fix it. RealCritic verified via arXiv abstract + HTML (HTML render partially draft-quality; abstract and appendix numbers consistent). SOURCE: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques | https://arxiv.org/abs/2501.14492 | 2025-01-24 | academic IMPLICATION: For Q1 meta-evaluation: never certify the AutoQA on verdict agreement alone - a judge can match pass/fail labels while its rationales are wrong, which poisons the feedback channel and attempter trust. Validation must score the rationale (does the cited evidence entail the finding; does acting on the feedback fix the item), i.e., Q7's explainability contract is also the correct validation instrument. ## A June 2026 study of 21 LLM judges (~541k judgments) shows raw percent-agreement overstates chance-corrected agreement by 33.8-41.3 points (85% agreement = kappa ~0.48), judge rankings flip by up to 15 positions across benchmarks, and the most reproducible judges are among the least valid (test-retest 0.99 with position bias 0.19) - leading the authors to prescribe a pre-deployment Minimum Viable Validation Protocol. EVIDENCE: 'Reliability without Validity' (UC Berkeley, arXiv 2606.19544, evals run March-April 2026): MT-Bench/JudgeBench/RewardBench, 118 runs at temperature 0; kappa deflation universal across all 21 judges; consistency-bias paradox instantiated by Qwen3-8B and Gemini 2.5 Flash; MVVP = report kappa/Krippendorff alpha as headline, AB+BA position swaps, >=3 replicates, >=2 benchmarks spanning preference- and correctness-style labels, and audit that high stability is not just high bias. Verified from arXiv HTML full text. SOURCE: Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic IMPLICATION: Directly answers Q1's statistics question (ban raw percent-agreement; standardize on chance-corrected stats) and Q5's ship-gate question (perturbation/position-swap + replicate harness is now published prescriptive practice). Run-to-run consistency must never be reported as evidence of validity - a perfectly consistent judge can be laundering a bias. ## The closest production analog to the proposed AutoQA already exists: Toloka's deployed 'LLM QA' runs a tool-using agent per quality-metric on every human annotation submission, emits a strict three-way Pass / Fail / Unable-to-verify verdict (the abstain class exists specifically to prevent hallucinated verdicts), coaches annotators with Socratic feedback, and explicitly accepts lower precision on Fail because a false pass costs more than escalating a genuine pass. EVIDENCE: Toloka engineering blog (bylined 2026-03-30): agentic autocheck with web/image/audio/Python/bash tools; 'one metric, one entity, one verdict' scoping (separate agent instance per criterion); internal benchmark of 300+ real submissions from 20+ live projects with senior-QA ground truth (numbers in an image, not extractable); reported constraint: performance 'entirely bounded by task design' - ambiguous guidelines produce walls of Unable-to-verify. Verified by opening the post. SOURCE: LLM QA: Scaling data quality assurance technologically (Toloka) | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | 2026-03-30 | practitioner IMPLICATION: Validates several foundational choices from production: criterion-scoped judge instances (Q3 decomposition at the per-axis level), a first-class abstain/escalate verdict distinct from pass/fail (Q2 verdict ontology), asymmetric error costs favoring false-flags over false-passes, and rubric quality as the binding constraint (Q4: tighten criteria before writing guidelines). ## The foundational 2022 result behind the whole lineage: model-written critiques helped human evaluators find flaws in summaries they would otherwise have missed (including planted flaws in deliberately misleading human-written summaries), and models exhibit a discriminator-critique gap - they can often detect that something is wrong better than they can articulate why. EVIDENCE: Saunders et al. (OpenAI, arXiv 2206.05802): topic-based summarization assistance experiments; critiques helped on both model- and human-written summaries; framework comparing generation, discrimination, and critique ability found 'even large models may still have relevant knowledge they cannot or do not articulate as critiques'; larger models critique and self-refine better. Verified from arXiv abstract. SOURCE: Self-critiquing models for assisting human evaluators (OpenAI) | https://arxiv.org/abs/2206.05802 | 2022-06-12 | primary IMPLICATION: The discriminator-critique gap argues for a two-stage judge: use cheap discrimination (scores/uncertainty) for routing and prioritization even where articulated critique is unreliable, and hold only the articulated-critique layer to the evidence-quoting bar (Q3/Q7). Judge-assisted human review of HUMAN-written work is validated at the root of this lineage, not just AI-output critique. ## A deployed LLM compliance-checker was gamed by its subjects in a single shot: the NeurIPS 2024 author checklist assistant (234 papers) was rated useful by >70% of authors, but the organizers found the system 'not robust to gaming' by authors and concluded it is a poor substitute for human review. EVIDENCE: Goldberg, Ullah, Guyon, Shah et al. (arXiv 2411.03417; NeurIPS blog Dec 2024): GPT-4-turbo checklist verification offered pre-submission; surveys plus qualitative analysis showed genuine improvements in some submissions alongside demonstrated manipulation of the assistant (authors could satisfy the checker without satisfying the standard); access was restricted to authors partly to avoid biasing review. Verified via arXiv abstract and organizer blog summary. SOURCE: Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment | https://arxiv.org/abs/2411.03417 | 2024-11-05 | academic IMPLICATION: Q8's decision-boundary-leak concern has deployment evidence even without repeated feedback cycles: pay/acceptance-motivated subjects will optimize against a visible checker. Defenses (rotating seeded probes, held-out human-graded golden items to detect AutoQA-vs-gold divergence, abstracted rather than checker-revealing feedback) must be designed in from day one, not retrofitted. ## Tools & artifacts - Review Feedback Agent: Open-source multi-LLM pipeline (zou-group/review_feedback_agent, CC BY 4.0 paper) that critiques human peer reviews for vagueness, content misunderstanding, and unprofessionalism, with automated LLM reliability tests gating feedback delivery. | https://github.com/zou-group/review_feedback_agent | Directly reusable architecture for the constructive-feedback channel: feedback taxonomy + fail-closed guardrail tests that feedback must pass before delivery (Q7). - MetaCritique: Framework + dataset (GAIR-NLP/MetaCritique) scoring critique precision/recall/F1 over atomic information units with per-unit rationales. | https://github.com/GAIR-NLP/MetaCritique | Off-the-shelf machinery for holding AutoQA feedback to its own grounding standard and for meta-evaluating critique quality during validation (Q1, Q7). - AbsenceBench: NeurIPS 2025 D&B benchmark (harvey-fin/absence-bench) measuring whether models can identify deliberately omitted content across poetry, sequences, and code diffs. | https://github.com/harvey-fin/absence-bench | Ready-made competence-limit test for the omission/comprehensiveness lane of any candidate judge model before assigning it completeness-verification authority (Q5 lane assignment). - RealCritic: Closed-loop critique-evaluation benchmark (tangzhy/RealCritic): critique quality scored by correction success, with self-/cross-/iterative-critique modes. | https://github.com/tangzhy/RealCritic | Template for validating the AutoQA judge on rationale effectiveness rather than verdict agreement (Q1 meta-evaluation design). - CTRL / Critique-RL / DeepCritic (2025 critic-training lineage): RL-trained critic models: CTRL (arXiv 2502.03492, critique via RL without human critique labels, up to 106% relative code-gen gains), Critique-RL (arXiv 2510.24320, two-stage RL: discriminability then helpfulness, no stronger supervisor), DeepCritic (arXiv 2505.00662, deliberate step-level math critique). | https://arxiv.org/abs/2502.03492 | State of the art for training a dedicated critic without the expensive expert-critique annotation dependency (Q4's aligned-expert concern); Critique-RL's discriminate-first-then-explain staging mirrors the discriminator-critique gap design. - On scalable oversight with weak LLMs judging strong LLMs (Kenton et al., DeepMind, NeurIPS 2024): Protocol comparison (debate vs consultancy vs direct QA) with weaker judges overseeing stronger agents across QA/math/code/logic asymmetries. | https://arxiv.org/abs/2407.04622 | Evidence base for arbitration design (Q6): debate beats consultancy everywhere tested, and open debate makes a weak judge much harder to convince of a wrong answer - a candidate protocol for contested-item escalation. - Minimum Viable Validation Protocol (MVVP): Five-step pre-deployment judge validation checklist from 'Reliability without Validity' (2026): chance-corrected headline stats, AB+BA position swaps, >=3 replicates, >=2 benchmark styles, consistency-bias paradox audit. | https://arxiv.org/abs/2606.19544 | Directly adoptable as the per-project judge-config ship gate the design taxonomy asks for (Q5). - Toloka LLM QA: Production agentic QA system for human annotation submissions: per-metric tool-using agent instances, Pass/Fail/Unable-to-verify verdicts, Socratic annotator feedback. | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | The closest commercial deployment to the proposed AutoQA; its verdict ontology, criterion scoping, and guideline-quality constraint are field-tested reference points (Q2, Q4, Q6). ## Disagreements - Does adding a human to an AI judge help at all? Vaccaro et al. (Nature Human Behaviour 2024) find human-AI combos on average significantly WORSE than the best solo party on decision tasks, while CriticGPT (2024) finds Human+AI teams beyond the model-only Pareto frontier and DeepMind (2025) gets 91.3% hybrid vs 87.7% AI-alone. Reconciliation offered by DeepMind: complementarity only appears with confidence-based routing to a slice where humans genuinely beat the AI and with non-leading assistance formats - but the DeepMind-adjacent follow-up ('Toward Human-AI Complementarity Across Diverse Tasks') found baseline hybridization added only +0.4pp when model confidence failed to identify the complementarity region. The one-touch design's value is conditional, not guaranteed. - What should the human be shown? DeepMind (2025) finds showing the AI's judgment/reasoning/confidence causes over-reliance and that evidence-only assistance is the only safe format, whereas CriticGPT's protocol and the ICLR 2025 feedback agent delivered full natural-language critiques and reported net gains. Task framing differs (assisting a second-party judge vs coaching the first-party author), which suggests the AutoQA should show full critiques to ATTEMPTERS but evidence-only to human ADJUDICATORS - no single source tests both arms. - Are LLM critics better or worse than humans? CriticGPT: model critiques preferred 63% and catch more bugs than paid contractors. MetaCritique: human critiques have much higher factual precision (87.6% vs 71.9% of atomic units factual) while LLM critiques have higher coverage. Both true simultaneously - the disagreement is about which axis (precision vs comprehensiveness) a QA system should privilege, and sources implicitly pick opposite defaults. - Can critics increase comprehensiveness of review? CriticGPT/Saunders show critics make human review MORE comprehensive (catching missed flaws), but AbsenceBench (2025) shows LLMs are structurally near-blind to omissions (56.9 F1 drop). Reconciliation: the critic gains are demonstrated on commission-type errors (inserted/tampered bugs, incorrect statements); no source demonstrates critic gains on omission-type attempter failures, and AbsenceBench predicts they won't appear without checklist/placeholder scaffolding. - What metric certifies a judge? Open-loop verdict-agreement benchmarks (CriticBench 2024, CriticEval 2024, MT-Bench-style agreement) vs RealCritic's closed-loop position that verdict accuracy without correction-effectiveness is 'superficial success' (24.8% right-verdict-wrong-critique), reinforced by RM-NLHF (ICML 2026) and by 'Reliability without Validity' (2026) showing judge rankings flip up to 15 places depending on which benchmark style you trust. ## Gaps - No public post-deployment report from OpenAI on CriticGPT's integration into the production RLHF labeling pipeline (announced intent, June 2024) - no follow-up metrics, and no officially branded successor found through July 2026; the lineage continued in academic work (CTRL, DeepCritic, Critique-RL, MultiCritique) rather than vendor deployment reports. - No published RCT of AI critique/feedback specifically on human DATA-ANNOTATION QA work. Closest analogs are peer review (ICLR 2025) and fact-verification rater assistance (DeepMind 2025). No longitudinal data anywhere on repeat-error-rate reduction per annotator under sustained AI feedback - Q7's efficacy metric has no published precedent to calibrate against. - No measured multi-cycle adversarial study of annotators gaming a deployed critic (Q8's decision-boundary leakage): the NeurIPS'24 checklist gaming is single-shot and author-side; citation-theater/evidence-swap perturbation tests on critics were not found. - Toloka's deployment report - the single closest production analog - publishes its benchmark design (300+ submissions, 20+ projects) but its quantitative precision/recall table is embedded in an image and was not extractable; no independent audit of any commercial annotation-QA LLM system was found. - False-agreement rate (judge and attempter both wrong vs expert panel) is unmeasured in the literature; the CriticGPT 24%-vs-6% flawless-data result is the nearest proxy and is one study, one domain (mostly), one vendor. - Positive-claim/praise verification (is 'accurate/complete/clear' praise warranted?) has no dedicated benchmark or precision/recall measurements - MetaCritique-style AIU machinery has only been applied to flaw-finding critiques. This is a genuine hole given the mission's requirement to evaluate positive statements. - Semantic Scholar returned HTTP 429 during literature-graph search (OpenAlex/Crossref coverage only for that call); one WebSearch call was flagged by a per-call safeguard and was successfully re-run via Exa - no coverage lost, noted for provenance. ## Verifications - CLAIM: Human+critic-model teams occupy a strictly better operating point than either alone: in OpenAI's CriticGPT study, Human+CriticGPT teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (FSBS length penalty), not a fixed property. VERDICT: confirmed | Source verified: arXiv:2407.00215 \"LLM Critics Help Catch LLM Bugs\" (McAleese et al., OpenAI), submitted 2024-06-28 - date matches. Abstract directly confirms: 63% preference for model critiques on naturally occurring LLM errors; models catch more bugs than paid human reviewers; human-machine teams catch similar bug counts to LLM critics while hallucinating less than LLMs alone. Full-text details corroborated via independent secondary coverage: critic-assisted contractors wrote more comprehensive critiques than unassisted contractors while reducing hallucination/nitpick rate relative to the model, described as moving beyond the model-only Pareto frontier; FSBS scores candidates by rm_score + LENGTH_MODIFIER x num_highlights, giving a deployment-time precision/comprehensiveness dial without retraining. Minor reading note: \"strictly better than either alone\" holds in the paper's Pareto sense - teams matched (did not exceed) the model's bug-catch/comprehensiveness while beating its hallucination rate, and exceeded human comprehensiveness; the claim's own wording states exactly this, so no correction needed. Supersession check: no official OpenAI follow-up found; later work (CodeCriticBench 2025; 2026 papers on LLM reviewer overcorrection) extends but does not contradict these findings. - CLAIM: In DeepMind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (AI rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% AI-alone and 75.1% human-alone - and showing humans the AI's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the AI was right without hurting when it was wrong. VERDICT: confirmed | Every quantitative element checks out against the arXiv HTML full text (2510.26518). Dataset: 1,918 expert-labeled tuples - confirmed. AI rater 87.7%, individual humans 75.1%, human majority vote 80.6% - confirmed. Confidence via 50 samples/sentence (avg 33.25 passing format check), described as fairly well calibrated - confirmed. Unassisted hybridization at threshold T=0.62: 89.3% (beta=0.413, p=.012) - confirmed. Low-confidence routed slice (280 items): humans 71.3% vs AI 60.5% (p=.006) - confirmed. The headline 91.3% is specifically the ASSISTED hybrid (AI when confident + evidence-assisted humans on the low-confidence slice), vs 89.3% unassisted hybrid - exactly as the claim words it, so 91.3% is correctly attributed. Evidence-only assistance: 79.3% vs 71.3% baseline when AI correct (beta=0.446, p=.009), no significant harm when AI wrong (64.0% vs 61.5%); paper explicitly calls it \"the only form of assistance that achieves the ideal of helping when correct and not hurting when wrong\" - confirmed. Over-reliance when AI wrong: Evidence&Reasoning&Judgments beta=-0.768 (SE=0.174, z=-4.413, p<.001), ER&J&Confidence beta=-0.636 p<.001, Judgments&Confidence beta=-0.356 p=.042 - the quoted beta=-0.768 matches the Evidence&Reasoning&Judgments condition exactly. Debate was numerically worst (64.9%, only format below baseline) - confirmed. Date: v1 submitted 2025-10-30 as claimed. Currency: a v2 was posted 2026-06-25 and the paper was published at ACM FAccT '26 (DOI 10.1145/3805689.3812308); this is publication/revision, not supersession - no later work contradicting the findings found. DeepMind blog notes ongoing follow-up combining hybridization+assistance on an internal rating task, unpublished as of 2026-07-14. Minor caveat only: whoever cites this should prefer the v2/FAccT version in case numbers shifted slightly in revision (v1 numbers verified here). - CLAIM: In the largest deployed RCT of AI critiquing human evaluative work (ICLR 2025, feedback on >20,000 peer reviews), 27% of reviewers who received LLM feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated LLM reliability tests. VERDICT: confirmed | Source verified directly (arXiv:2504.09737, v1 submitted 2025-04-13 - date matches). Abstract states: Review Feedback Agent deployed at ICLR 2025 as a large randomized controlled study on >20,000 randomly selected reviews; feedback targeted vague comments, content misunderstandings, unprofessional remarks; 27% of reviewers who received feedback updated their reviews; >12,000 suggestions incorporated (12,222 in the paper); +80 words average among updaters; more informative per blinded evaluators; longer author-reviewer discussions; feedback delivered only if it passed a suite of automated LLM reliability tests. Arm sizes corroborated from paper figures: 22,467 selected for feedback vs 22,364 control; 18,946 successfully received feedback, of whom 26.6% updated (the abstract rounds to 27%). Two nuances, neither contradicting the claim: (1) the superlative \"largest deployed RCT of AI critiquing human evaluative work\" is the researcher's framing, not verbatim in the abstract - no larger counterexample found, and it is consistent with the study's scale; (2) not superseded but now formally published: Nature Machine Intelligence, Feb 23, 2026, as \"A large-scale randomized study of large language model feedback in peer review\" (https://www.nature.com/articles/s42256-026-01188-x) - the NMI version is the preferred citation going forward. A related follow-up mixed-methods study on reviewer perceptions exists (arXiv:2602.13817) but does not alter these results."}, {"slug": "domain-grounding-faithfulness", "title": "Domain Report - Grounding & Faithfulness", "category": "Domain research", "summary": "Claim verification, evidence closure, decomposition, citation quality, and faithfulness detectors.", "tags": ["domain report", "grounding faithfulness"], "view": "documents/view/domain-grounding-faithfulness.html", "raw": "documents/raw/research/domain-grounding-faithfulness.md", "text": "# Domain: grounding-faithfulness ## Grounded claim-verification is commoditized but ceilinged: the best model on the LLM-AggreFact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7B model, Bespoke-MiniCheck-7B, at 77.4 average balanced accuracy, with 0.4-0.8B specialized checkers (FactCG, MiniCheck-Flan-T5-L) within ~2.5 points of frontier LLMs like Claude-3.5-Sonnet (77.2) and GPT-4o (75.9). [LOAD-BEARING] EVIDENCE: Fetched the live LLM-AggreFact leaderboard: top-10 shows Bespoke-MiniCheck-7B 77.4, Claude-3.5-Sonnet 77.2, Granite Guardian 3.3 8B 76.5, FactCG-DeBERTa-L (0.4B) 75.6, MiniCheck-Flan-T5-L (0.8B) 75.0, Llama-3.1-405B 74.4. Benchmark measures binary supported/unsupported vs grounding docs. Confirmed independently by Paladin-mini (June 2025 arXiv) citing Bespoke-MiniCheck as leaderboard SOTA. SOURCE: LLM-AggreFact Leaderboard (MiniCheck project) | https://llm-aggrefact.github.io/ | 2025 (accessed 2026-07-14) | primary IMPLICATION: A cheap, deterministic, reproducible entailment-check stage (sub-1B to 7B checker) for 'is this attempter statement supported by the cited evidence' is off-the-shelf and costs ~1/100 of frontier calls - but ~22% claim-level error means item verdicts cannot be a naive AND over claim checks; error-tolerant aggregation and confidence-routing to humans are structurally required. ## On adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on FaithBench, prior detectors hit ~50% accuracy (negligible), HHEM-2.1-Open 66.7% and Bespoke-MiniCheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while FaithJudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 F1. [LOAD-BEARING] EVIDENCE: Fetched full text of arXiv:2505.04847v2 (Vectara, EMNLP 2025 Industry Track, v2 Nov 2025). Table 1: baseline detector numbers; Table 2: FaithJudge results; FaithJudge beats the FACTS Grounding judging prompt head-to-head on all four RAGTruth/FaithBench splits (e.g., FaithBench 70.8 vs 54.3 F1). Documented failure modes: judge underpredicts hallucinations for some generator families; 'benign'/'questionable' ternary labels classified unreliably so only binary is used; specificity drops as more in-context examples are added. SOURCE: Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards (FaithJudge) | https://arxiv.org/abs/2505.04847 | 2025-11 | academic IMPLICATION: The strongest 2025-measured grounding-judge architecture is exactly the AutoQA rubric-compilation contract: a per-project pool of human-adjudicated exemplar judgments (annotated spans + labels on comparable items) injected few-shot, not zero-shot judging and not a fixed fine-tuned checker. Also: severity-graded verdicts (benign/questionable) are where both humans and judges lose reliability - keep the machine verdict binary and treat severity as a separate, human-anchored layer. ## Claim decomposition helps weak verifiers but actively degrades strong ones: with MiniCheck as verifier on WiCE, no-decomposition scores 80.01 balanced accuracy while FActScore-style atomic decomposition drops it to 71.11; with the weaker AlignScore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity. [LOAD-BEARING] EVIDENCE: Fetched full text of 'Decomposition Dilemmas' (NAACL 2025, arXiv:2411.02400). Four-way decomposition-error taxonomy from manual inspection: (A) omission of context/logical relations, (B) ambiguity (unclear pronouns/references), (C) over-decomposition, (D) alteration of original meaning. FActScore-style atomicity produces the most over-decomposition errors; VeriScore-style tends to omit context. On FELM, decomposition raised MiniCheck F1 48.1->~68 but dropped GPT-4o-mini F1 71.6->54.3. SOURCE: Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? | https://aclanthology.org/2025.naacl-long.320/ | 2025-05 | academic IMPLICATION: Directly answers the granularity design question: there is no universally-best decomposition level. Granularity must be tuned per verifier strength and per item complexity - atomic-claim pipelines with a strong judge are measurably WORSE than judging larger spans. The 'checklist myopia' risk is real and has a named mechanism (context omission + meaning alteration). Budget a granularity-calibration step per project rather than fixing atomic decomposition in the foundation. ## Decomposition and decontextualization are in direct tension - isolating atomic facts strips the context needed to verify them, while adding context back creates multi-fact claims where the verifier may credit or penalize the wrong content; DnDScore resolves this by verifying the original subclaim WITH the added information treated as context rather than as content to be verified. EVIDENCE: Fetched ACL Anthology abstract of DnDScore (Wanner, Van Durme, Dredze; EMNLP 2025 main, pp. 23609-23626). The paper evaluates combinations of decomposition, decontextualization, and verification strategies and finds the strategy choice materially changes factuality scores. SOURCE: DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation | https://aclanthology.org/2025.emnlp-main.1205/ | 2025-11 | academic IMPLICATION: When the AutoQA checks an attempter's sentence, the verification unit should be 'claim + explicit context annotations' with the verifier told which part is under test - not a free-floating atomic rewrite. This is a concrete spec for the claim-extraction layer's output schema. ## Verifiability triage before entailment is established machinery: VeriScore extracts and scores ONLY verifiable claims because FActScore/SAFE 'assume that every claim is verifiable', which breaks on real long-form text containing opinions and unverifiable content; FactBench's VERIFY pipeline (ACL 2025) further labels content units supported/unsupported/UNDECIDABLE against retrieval, with 4,467 human-annotated units released for validation. EVIDENCE: Fetched arXiv:2406.19276 abstract (VeriScore, EMNLP Findings 2024): human evaluation found VeriScore's extracted claims 'more sensible' than competitors across 8 long-form tasks. Fetched launchnlp/FactBench GitHub (ACL 2025 paper 2410.22257): three-way supported/unsupported/undecidable labeling according to retrieval results, benchmarked against FActScore, SAFE, Factcheck-GPT. SOURCE: VeriScore (EMNLP Findings 2024) + FactBench/VERIFY (ACL 2025) | https://arxiv.org/abs/2406.19276 | 2024-06 (VeriScore, foundational); 2025-07 (FactBench ACL 2025) | academic IMPLICATION: The claim ontology question has a field-standard answer: a front-stage classifier that routes each attempter statement into {verifiable claim, opinion/unverifiable, undecidable-given-evidence} BEFORE any entailment check. Opinion-class statements get consistency-with-cited-evidence checks at most, never truth verdicts; 'undecidable' is a first-class output, not a forced pass/fail. ## Holistic LLM judging of context-grounded outputs is far weaker than claim-level checking: on ContextualJudgeBench (2,000 pairs, 8 splits over RAG/summarization with conditional criteria like 'faithfulness first, then completeness'), the best of 20 judge models tested (OpenAI o1) barely reaches 55% consistent accuracy. EVIDENCE: Fetched arXiv:2503.15620 (Salesforce; ACL 2025). 11 specialized judge models + 9 general-purpose models evaluated; the benchmark encodes criteria hierarchies and finds contextual assessment poses a significant challenge even to SOTA models. Contrast: the same model class hits 77-84% on single-claim grounding tasks (LLM-AggreFact, FaithJudge). SOURCE: Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings | https://arxiv.org/abs/2503.15620 | 2025-03 | academic IMPLICATION: Do not architect the AutoQA as one holistic 'judge this writeup against the rubric' call - measured reliability roughly halves versus decomposed per-criterion, per-claim checks. Criteria priority ordering (which humans dispute) must be compiled explicitly into the pipeline structure, because judges fail at applying conditional criteria internally. ## Grounding judges have a measured agreement-default asymmetry: in Google's FACTS Leaderboard (Dec 2025), grounding judges score ~85 F1 on the positive (grounded) class but only ~46 F1 on the negative (ungrounded) class, and the grounding metric is gameable by vague, short responses that avoid unsupported claims - countered by a mandatory eligibility gate that scores non-responsive answers as inaccurate. [LOAD-BEARING] EVIDENCE: Fetched full text of arXiv:2512.10791v1 (The FACTS Leaderboard, Google, Dec 2025). Best judge-prompt combos reach only ~65 macro-F1 (gemini-2.5-flash + v2 prompt: 65.33) against a 320-example human-adjudicated held-out set; positive-class ~85 vs negative-class ~46 F1; eligibility check explicitly added because 'grounding metrics can be hacked via vague, short responses'. SOURCE: The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality | https://arxiv.org/abs/2512.10791 | 2025-12 | academic IMPLICATION: Two foundational mandates: (1) the sycophancy/agreement-default bias on positive claims is quantified and large - the AutoQA must be evaluated on per-class recall (especially fail-class recall at fixed prevalence), never single-number accuracy; (2) an eligibility/responsiveness gate must precede grounding checks or attempters can pass by writing vacuous hedged annotations - the exact adversarial dynamic the design taxonomy flags. ## Cross-family judge ensembles are the standard mitigation for self-preference: FACTS Grounding v1 (Jan 2025) aggregated three frontier judges from different families (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) explicitly because 'models are biased towards favorably judging their own outputs', and Grounding v2 (Dec 2025) kept a two-family ensemble (Gemini 2.5 Flash + GPT-5). EVIDENCE: Fetched arXiv:2512.10791v1 which states the v2 judge-ensemble rationale and composition; v1 design documented in arXiv:2501.03200 (Jan 2025) and the DeepMind blog (Dec 17, 2024). FACTS Parametric suite separately validated that a single-judge setup preserved rankings vs a mixed panel (Gemini 2.5 Pro, o3, Grok 4) - i.e., ensemble necessity is task-dependent and testable. SOURCE: FACTS Grounding Leaderboard (v1 paper + v2 in FACTS Leaderboard) | https://arxiv.org/abs/2501.03200 | 2025-01 (v1); 2025-12 (v2) | academic IMPLICATION: When attempters critique outputs from a given model family, the AutoQA judge should default to a different family or an ensemble spanning families; but ensemble overhead can be dropped where a single-judge-vs-panel ranking-preservation test passes - a cheap per-project validation, not a dogma. ## Span-restricted NLI citation checking (the ALCE-style 'does the cited span entail the sentence' standard) is judged a suboptimal proxy by 2025 work: CiteEval (ACL 2025) argues citation quality must be evaluated against the FULL retrieval context, user query, and generated text - not cited sources alone - and its model-based CiteEval-Auto metrics correlate better with human judgments on the multi-domain CiteBench than NLI-based metrics. EVIDENCE: Fetched arXiv:2506.01829 (CiteEval, ACL 2025 main). Explicitly frames binary/ternary NLI vs cited sources as 'a suboptimal proxy for citation evaluation'. Corroborated direction: CiteGuard (ACL 2026, aclanthology.org/2026.acl-long.282) reframes citation evaluation as attribution alignment and notes 'the reliability of LLM-as-a-Judge alone is also in doubt' for citation judging. SOURCE: CiteEval: Principle-Driven Citation Evaluation for Source Attribution | https://arxiv.org/abs/2506.01829 | 2025-06 | academic IMPLICATION: Evidence-closure design: checking attempter claims only against the spans they cite will miss the dominant human failure of citing the wrong/incomplete evidence when better evidence existed in scope. The AutoQA needs two distinct checks - (a) does cited evidence entail the claim (span-closed), and (b) is the citation the right one given the full in-scope corpus (context-closed) - with (b) also catching true-but-wrongly-cited claims. ## Fact-verification LLMs are brittle to semantically-minor input perturbations: FactEval (NAACL 2025) tested 17 realistic word- and character-level perturbations plus 4 subpopulations on FEVER across zero-shot/few-shot/CoT setups and found LLMs 'brittle to small input changes' with performance varying across subpopulations. EVIDENCE: Fetched ACL Anthology page for 2025.naacl-long.534 (Mamta & Cocarascu). Abstract is qualitative; per-perturbation numbers are in the full paper. Consistent with FaithJudge's finding that adding more in-context examples shifts sensitivity/specificity, and with FACTS' judge-prompt sensitivity (macro-F1 varies by prompt version). SOURCE: FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models | https://aclanthology.org/2025.naacl-long.534/ | 2025-04 | academic IMPLICATION: A perturbation-robustness harness (paraphrase, reorder, typo-level noise on both claims and evidence) belongs in the per-project ship gate; verdicts that flip under meaning-preserving edits should be auto-routed to human review rather than delivered. ## 2026 SOTA joins verdict and explanation in one cheap model: FaithLens (ACL 2026 Findings) is an 8B faithfulness-hallucination detector that jointly outputs a binary prediction AND an explanation, trained via cold-start fine-tuning on filtered synthetic data plus rule-based RL rewarding both prediction correctness and explanation quality, and outperforms GPT-5.2 and o3 across 12 tasks. EVIDENCE: Fetched ACL Anthology entry 2026.findings-acl.689 (Si et al., Findings of ACL 2026, pp. 14068-14099). Abstract claims superiority over GPT-5.2/o3 and 'a distinctive balance of trustworthiness, efficiency, and effectiveness'; per-task numbers require the full PDF. SOURCE: FaithLens: Detecting and Explaining Faithfulness Hallucination | https://aclanthology.org/2026.findings-acl.689/ | 2026-07 (ACL 2026) | academic IMPLICATION: Constructive-feedback generation and grounding verdicts do not need separate machinery: an explanation-rewarded detector produces the evidence-citing rationale as a first-class output, at 8B cost. This is the current-generation template for the AutoQA's 'every criticism must carry its evidence' contract, and shows explanation quality can be an explicit training/reward target rather than a post-hoc add-on. ## RAGAS-style prompt-chain faithfulness scoring materially underperforms finetuned/frontier judges on hallucination detection: on the 15K-sample HaluBench, RAGAS Faithfulness scored 66.9% accuracy versus 87.4% for finetuned Lynx-70B and 86.5% for GPT-4o; Patronus has since shipped Lynx 2.0 (8B, long-context, 8 hallucination subtypes including coreference and calculation errors). EVIDENCE: Confirmed via Patronus AI announcement and Lynx paper coverage (July 11, 2024) with the HaluBench table (Lynx-70B 87.4, GPT-4o 86.5, GPT-4-Turbo 85.0, Llama-3-70B 80.1, RAGAS 66.9); Lynx 2.0 details from Patronus docs. Caveat: vendor-reported numbers on the vendor's own benchmark, though HaluBench is public on HuggingFace. RAGAS remains the most widely adopted OSS RAG-eval framework per 2026 practitioner surveys (atlan.com RAG evaluation guide, 2026). SOURCE: Patronus AI Lynx / HaluBench results | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | 2024-07 (Lynx 1.0, foundational); Lynx 2.0 later update | practitioner IMPLICATION: Do not build the grounding stage on RAGAS-style statement-extraction prompt chains despite their ubiquity in 2026 tooling - the popularity/accuracy gap is ~20 points. If an off-the-shelf component is wanted, current-generation finetuned detectors (Bespoke-MiniCheck, Lynx 2.0, Granite Guardian, FaithLens) dominate at equal or lower cost. ## Tools & artifacts - Bespoke-MiniCheck-7B / MiniCheck family: Specialized grounded-claim fact-checking models (0.4B-7B), SOTA on LLM-AggreFact at 77.4 bacc; available on HuggingFace and Ollama, ~400x cheaper than GPT-4-class verification | https://github.com/Liyan06/MiniCheck | Default candidate for the cheap deterministic claim-vs-evidence entailment stage of the AutoQA - FaithJudge (Vectara): Open-source LLM-as-judge framework that anchors judgment with human-annotated peer examples on the same source; repo actively pushed through Nov 2025 | https://github.com/vectara/faithjudge | The measured-best architecture pattern for the AutoQA's per-project exemplar-anchored judge; directly reusable code - FACTS Grounding public set + FACTS Leaderboard: 860 public grounding examples (32k-token docs) plus Google's Dec 2025 four-suite factuality benchmark with Kaggle-held private splits | https://huggingface.co/datasets/google/FACTS-grounding-public | Source of judge-prompt designs, the eligibility-gate pattern, and human-adjudicated grounding labels for meta-evaluating our own judge - FactBench / VERIFY: ACL 2025 factuality pipeline with supported/unsupported/undecidable labeling and 4,467 human-annotated content units released | https://github.com/launchnlp/FactBench | Template + validation data for the verifiability-triage front stage (fact vs opinion vs undecidable) - VeriScore + VeriFastScore: Verifiable-claim-only extraction/scoring for long-form text (EMNLP Findings 2024) and its single-model accelerated successor (EMNLP Findings 2025) that collapses extraction+verification into one fine-tuned pass | https://github.com/Yixiao-Song/VeriScore | Claim-extraction spec that skips opinions; VeriFastScore is the cost-reduction pattern if per-item pipeline latency matters - HaluBench + Lynx 2.0 (Patronus): 15K-sample hallucination detection benchmark with hard/ambiguous cases, plus an 8B long-context finetuned detector covering 8 hallucination subtypes | https://huggingface.co/PatronusAI | Seeded-set material for measuring our checker's fail-class recall; Lynx 2.0 is an off-the-shelf detector option - ContextualJudgeBench (Salesforce): 2,000-pair benchmark for LLM judges in RAG/summarization settings with conditional criteria hierarchies (ACL 2025) | https://arxiv.org/abs/2503.15620 | Meta-evaluation instrument for any holistic judge configuration we consider; quantifies why we should not use one - CiteBench / CiteEval-Auto: Multi-domain human-annotated citation-quality benchmark and model-based metrics that evaluate citations against full retrieval context (ACL 2025) | https://arxiv.org/abs/2506.01829 | Blueprint for the context-closed 'is this the right citation given everything in scope' check, beyond span-closed NLI - HHEM-2.1-Open + Vectara Hallucination Leaderboard: Open small hallucination-detection model and the longest-running (2023-present, refreshed Nov 2025 with 7,700 articles up to 32k tokens) HHEM-scored LLM leaderboard | https://huggingface.co/spaces/vectara/leaderboard | Cheapest available checker tier, but its measured 66.7 bacc on hard cases (FaithBench) bounds where it can be trusted - screening, not verdicts ## Disagreements - HHEM's fitness as an automated grounding judge: Vectara's own next-generation hallucination leaderboard (blog, 2025-11-19) still uses commercial HHEM as the sole scorer and reports rates like Gemini-2.5-flash-lite 3.3% vs Gemini-3-pro 13.6%, while Vectara's own research paper (FaithJudge, EMNLP Industry 2025) shows HHEM-2.1-Open at only 66.7% balanced accuracy on FaithBench - near-random on hard hallucinations - and builds an LLM-judge replacement because of 'limitations observed in current hallucination detection methods'. Detector choice materially changes model rankings; automated hallucination-rate leaderboards from different detectors disagree and cannot be treated as interchangeable ground truth. - Atomic decomposition: the FActScore/SAFE/RAGAS lineage (2023-2024) treats finer-grained atomic claims as strictly better for verification, but Decomposition Dilemmas (NAACL 2025) shows decomposition DEGRADES strong verifiers (80.0 -> 71.1 bacc on WiCE with MiniCheck) and DnDScore (EMNLP 2025) shows atomization and decontextualization actively conflict. The field has not converged: long-form factuality pipelines still default to atomic claims while the 2025 analysis papers say granularity must be adaptive. - Specialized small detectors vs frontier LLM judges: LLM-AggreFact (through 2025) and FaithLens (ACL 2026) show small specialized models beating frontier LLMs (Bespoke-MiniCheck-7B > Claude-3.5/GPT-4o; FaithLens-8B > GPT-5.2/o3), but FaithJudge (2025) shows frontier reasoning judges WITH human-annotated exemplars beating all specialized detectors on adversarial sets (84.0 vs 71.2 bacc). Both are right in their regime: fixed checkers win zero-shot on broad distributions; exemplar-anchored frontier judges win on hard, domain-specific cases - which is an argument for a two-tier architecture rather than either alone. - Citation-support standard: ALCE-style span-closed NLI citation precision/recall remains the de-facto automated standard, but CiteEval (ACL 2025) calls NLI-vs-cited-spans 'a suboptimal proxy' and CiteGuard (ACL 2026) additionally doubts LLM-as-judge reliability for citation evaluation - i.e., the two dominant automated approaches each have a 2025-2026 paper attacking them, with no settled successor. ## Gaps - No published measurements of run-to-run verdict flip rates (self-consistency) for grounding checkers or LLM faithfulness judges on identical inputs - the taxonomy's 'flip rate: noise vs underspecification signal' question has no literature anchor found; specialized classifiers (MiniCheck/HHEM) are deterministic by construction but that property is unstated in evaluations. - Positive-claim verification of EVALUATIVE statements (warranted vs empty praise, 'stated severity matches evidence strength' calibration claims) is essentially unstudied: the entire 2024-2026 literature verifies factual/descriptive claims against documents; no benchmark tests whether a checker can verify claims like 'this response is complete' or 'this citation is the strongest available'. The AutoQA's core use case - verifying human evaluative writeups about AI outputs - is a domain transfer no found benchmark covers. - No dedicated 2025-2026 study found on evidence-swap/citation-theater probes for verifiers (does the verdict flip when cited evidence is replaced with equally-plausible irrelevant text?). FactEval perturbs claims, not evidence; FACTS documents gaming by response vagueness, not by decorative citation. This test would need to be built in-house. - Aggregation from claim-level verdicts to item-level pass/fail with severity weighting is under-researched: FActScore-style %-supported and FACTS' all-or-nothing eligibility+grounding are the only documented rules; RAGTruth/FaithBench severity taxonomies (benign/questionable/unwanted, evident/subtle) exist as LABELS but FaithJudge found judges classify the middle severity classes unreliably and retreated to binary - no validated severity-weighted aggregation scheme found. - Judge-vs-adjudicated-human agreement ceilings per axis type are only available for grounding (~65 macro-F1 in FACTS on held-out human labels; 84 bacc for FaithJudge on FaithBench) - no comparable 2025-2026 numbers found for instruction-compliance or tone/style axes from the faithfulness literature, so the lane-assignment decision (judge-autonomous vs assisted vs human-only per axis) lacks published anchors outside factual grounding. ## Verifications - CLAIM: Grounded claim-verification is commoditized but ceilinged: the best model on the LLM-AggreFact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7B model, Bespoke-MiniCheck-7B, at 77.4 average balanced accuracy, with 0.4-0.8B specialized checkers (FactCG, MiniCheck-Flan-T5-L) within ~2.5 points of frontier LLMs like Claude-3.5-Sonnet (77.2) and GPT-4o (75.9). VERDICT: confirmed | Fetched https://llm-aggrefact.github.io/ on 2026-07-14; every number matches exactly: Bespoke-MiniCheck-7B 77.4 (rank 1), Claude-3.5 Sonnet 77.2, Granite Guardian 3.3 8B 76.5, gpt-4o-2024-05-13 75.9, FactCG-DeBERTa-L (0.4B) 75.6, MiniCheck-Flan-T5-L (0.8B) 75.0, Llama-3.1-405B 74.4. Benchmark is 11 datasets of grounded factuality (supported/unsupported vs grounding docs), avg balanced accuracy - as claimed. The \"within ~2.5 points\" framing is accurate and even conservative: FactCG (75.6) is only 1.6 below Claude-3.5-Sonnet and above GPT-4o. Supersession search found nothing beating 77.4: ACV (May 2026) reports 76.5 training-free; an ACL 2026 paper ranks second to a post-trained metric; Paladin-mini beats Bespoke-MiniCheck only on its own separate benchmark subsets, not LLM-AggreFact overall. Caveats on the interpretive \"ceilinged\" framing: (1) the leaderboard's frontier entries are 2024-era models (Claude-3.5-Sonnet, gpt-4o-2024-05-13, Llama-3.1-405B) - no Claude 4/GPT-5-class/o3 entries, so \"specialized ~ frontier\" reflects 2024 frontiers; (2) \"Verifying the Verifiers\" (arXiv 2506.13342, June 2025) finds ~16% of benchmark labels ambiguous/incorrect and that few-shot frontier LLMs reach top-tier performance, suggesting the ~77 plateau is partly benchmark label noise rather than a pure task ceiling; (3) \"Verify with Caution\" (arXiv 2501.14883) shows models with similar aggregate BAcc make very different instance-level predictions. None of these contradict the stated facts. - CLAIM: On adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on FaithBench, prior detectors hit ~50% accuracy (negligible), HHEM-2.1-Open 66.7% and Bespoke-MiniCheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while FaithJudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 F1. VERDICT: partially_confirmed | Verified against arXiv:2505.04847 (v2 dated Nov 6, 2025; EMNLP 2025 Industry Track; Vectara/Waterloo authors - source and date correct). Confirmed: (1) paper states prior detectors, including LLM classifiers, achieved \"near 50% accuracy\" on FaithBench; (2) HHEM-2.1-Open = 66.7% balanced accuracy on FaithBench (claim-wise, Table 1) - though flagged with an asterisk because HHEM was used to adversarially select FaithBench articles; (3) FaithJudge with o3-mini-high = 84.0% balanced accuracy / 82.1 F1-macro (Table 2), best zero-shot judge on FaithBench was o3-mini-high at 68.8%; (4) FaithJudge vs FACTS Grounding head-to-head 70.8 vs 54.3 F1 on FaithBench confirmed (Table 4); (5) all three stated failure modes confirmed (underprediction for Command-R/Mistral/Qwen generators; Benign/Questionable misclassified - only 10/84 Benign labeled correctly; specificity slightly decreases as in-context examples increase). ERRORS: (a) Bespoke-MiniCheck's 71.2 is NOT its FaithBench score - 71.2 is its balanced-accuracy AVERAGE across all four datasets (AggreFact, RAGTruth, TofuEval-MB, FaithBench) in Table 1; its actual FaithBench score is 60.1% claim-wise / 55.7% summary-wise. (b) Minor: the \"below 78% balanced accuracy\" figure for zero-shot judges is the paper's cross-dataset average claim, not a FaithBench-specific figure (on FaithBench zero-shot judges max out at 68.8%, so the claim still holds directionally). Supersession check: searches found no 2026 work surpassing FaithJudge on FaithBench; it remains the reported state of the art as of July 2026. CORRECTED: On adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on FaithBench, prior detectors hit ~50% accuracy (negligible); fine-tuned detectors stay weak (HHEM-2.1-Open 66.7% and Bespoke-MiniCheck 60.1% claim-wise balanced accuracy on FaithBench; Bespoke-MiniCheck averages 71.2% across the four benchmark datasets); zero-shot frontier judges reach at most 68.8% on FaithBench (and stay below 78% averaged across datasets), while FaithJudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 F1-macro. - CLAIM: Claim decomposition helps weak verifiers but actively degrades strong ones: with MiniCheck as verifier on WiCE, no-decomposition scores 80.01 balanced accuracy while FActScore-style atomic decomposition drops it to 71.11; with the weaker AlignScore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity. VERDICT: confirmed | Verified against the full PDF (extracted text at [local path withheld]). (1) Source says exactly this: Table 2 (WiCE) shows MiniCheck baseline BAcc 80.01 vs FActScore decomposition 71.11 (F1 72.32 vs 59.90); all decomposition methods hurt MiniCheck on WiCE. With the weaker AlignScore verifier, FActScore decomposition improves BAcc 54.80->56.87 (and WiCE-style 56.26); note VeriScore decomposition slightly hurt AlignScore too, but the claim as stated (\"the same decomposition,\" i.e., FActScore-style) is accurate. Section 4.2 states verbatim that \"decomposition generally benefits weaker verifiers, while it tends to negatively affect stronger verification systems.\" Section 6.2-6.4 confirms gains reappear as input complexity grows (complexity scale-up experiments; FELMshort scale-down degrades), and Figure 3 discussion states \"for each level, the maximum F1 is observed when the number of the decomposed sub-claim is less than or equal to the complexity level.\" Evidence-summary side facts also check: FELM MiniCheck F1 48.10->67.5-68.1, GPT-4o-mini F1 71.56->54.34 (Table 3); four-way error taxonomy (context omission, ambiguity, over-decomposition, meaning alteration) present in Section 5. (2) Date: ACL Anthology lists April 2025 (NAACL 2025, Albuquerque, pp. 6313-6336); the conference ran Apr 29-May 4, so \"2025-05\" is a trivial one-month imprecision, not a substantive error. (3) Supersession search: later work (e.g., presupposition-free question decomposition, arXiv 2508.16838; \"Alignment Bottleneck in Decomposition-Based Claim Verification,\" Feb 2026) extends the decomposition-tradeoff line but does not refute the verifier-strength finding."}, {"slug": "domain-hybrid-statistical", "title": "Domain Report - Hybrid Statistical Methods", "category": "Domain research", "summary": "Prediction-powered inference, selective escalation, conformal routing, calibration, and drift attribution.", "tags": ["domain report", "hybrid statistical"], "view": "documents/view/domain-hybrid-statistical.html", "raw": "documents/raw/research/domain-hybrid-statistical.md", "text": "# Domain: hybrid-statistical ## Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a GPT-4 judge in 'AutoEval Done Right'. [LOAD-BEARING] EVIDENCE: arXiv:2403.07008 (Boyeau, Angelopoulos, Yosef, Malik, Jordan, UC Berkeley) abstract confirmed: 'increase the effective human-labeled sample size by up to 50% on experiments with GPT-4'; v3 camera-ready posted 2026-06-01. The unconditional-coverage property is restated and empirically demonstrated in the GLIDE paper (arXiv:2605.31278) and is the foundation of the entire 2026 LLM-judge debiasing literature (PRECISE/AAAI 2026, arXiv:2606.05308 ranking extension). SOURCE: AutoEval Done Right: Sample-Efficient Human Evaluation via Prediction-Powered Inference | https://arxiv.org/abs/2403.07008 | 2024-03 (v3 camera-ready 2026-06-01) | academic IMPLICATION: The AutoQA meta-evaluation layer should treat human gold labels as bias-correctors for judge-derived population metrics (pass rates, per-attempter/per-criterion error rates), never report raw judge numbers. Because validity survives a bad judge, pooled/hierarchical cross-project validation is statistically safe even where per-project gold sets are tiny - directly resolving taxonomy Q1's gold-set-arithmetic fork toward pooled certification with per-project bias correction. ## Cascaded Selective Evaluation (Trust or Escalate, ICLR 2025 Oral) delivers a provable, user-specified human-agreement guarantee for LLM judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a Chatbot Arena subset where GPT-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models. [LOAD-BEARING] EVIDENCE: arXiv:2407.18370 abstract fetched directly (80% agreement / ~80% coverage claim confirmed); ICLR 2025 Oral status confirmed at iclr.cc/virtual/2025/oral/31838; fixed-sequence-testing calibration mechanism confirmed via OpenReview PDF snippets and secondary reviews (83 citations as of 2026-07). Its 'Simulated Annotators' method (in-context simulation of diverse annotators) is the confidence estimator that makes the guarantee achievable at high coverage. SOURCE: Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement (ICLR 2025 Oral) | https://arxiv.org/abs/2407.18370 | 2024-07-25 (ICLR 2025-04) | academic IMPLICATION: This is the direct architectural blueprint for 'which items get the single human touch': the human queue is exactly the abstention set of a confidence-calibrated cascade, and the abstention rate is a tunable dial trading human budget against a guaranteed agreement level. Caveat for taxonomy Q1: the guarantee is agreement-with-human-majority, so this component's validity is capped at human panel reliability - it certifies consistency-of-application, not truth. ## The minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n*(1-rho^2) where n* is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with R^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes. [LOAD-BEARING] EVIDENCE: arXiv:2605.16354v1 (Jane Paik Kim, Stanford Psychiatry, 2026-05-08) HTML fetched: derives sample-size formulas from the doubly-robust estimator's asymptotic variance; worked examples confirmed (n*=200, R^2=0.7, N=2000 -> 65 human labels; N=400 -> 100). Explicit caveats: pilot overestimation of R^2 breaks precision guarantees (use conservative values), framework assumes human ratings are a reliable gold standard, single-rater scope. SOURCE: Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need? | https://arxiv.org/html/2605.16354v1 | 2026-05-08 | academic IMPLICATION: Answers taxonomy Q1's gold-set question with arithmetic instead of convention: run a small pilot to estimate per-axis judge-human correlation, then size each project's gold set from the formula with a conservative correlation estimate. Because the floor depends on rho, per-axis correlation heterogeneity argues for stratified gold sampling by criterion, and low-rho axes are where gold budget must concentrate. ## Using the LLM's own confidence signals to choose WHICH items receive human annotation (Confidence-Driven Inference, building on Active Statistical Inference) cut required human annotations by >25% in all three tested settings while keeping provably valid confidence intervals that remain safe even if the LLM annotations are poor - but a 2026 ICLR paper finds the opposite in the sequential regime, where near-uniform sampling at the budget ceiling beat uncertainty-driven querying. [LOAD-BEARING] EVIDENCE: arXiv:2408.15204 (Gligoric, Zrnic, Lee, Candes, Jurafsky; NAACL 2025) abstract fetched: '>25% reduction in each of three settings' (politeness, stance, bias) with validity guarantees regardless of LLM quality. Foundation: Active Statistical Inference (Zrnic & Candes, ICML 2024, arXiv:2403.03208): label where the model is uncertain, provably valid CIs, fewer labels than uniform. Counter-evidence verified separately (see disagreements): arXiv:2604.18569 (ICLR 2026). SOURCE: Can Unconfident LLM Annotations Be Used for Confident Conclusions? (NAACL 2025) + Active Statistical Inference (ICML 2024) | https://arxiv.org/abs/2408.15204 | 2024-08-27 (v2 2025-02-08; NAACL 2025-04) | academic IMPLICATION: Uncertainty-routed allocation of the scarce human touch is the statistically defensible default and comes with a safety net (validity even when the judge is wrong), but the gain over random sampling is regime-dependent and contested - so the foundation should make the routing policy (uncertainty-weighted vs uniform vs stratified) a measured, per-project choice with a built-in A/B against uniform, not a hard-wired assumption. ## GLIDE (May 2026) is an open-source industrial library unifying PPI++, stratified PPI, predict-then-debias bootstrap, and active inference with a decision tree for method selection, and it quantifies when PPI helps: effective-sample gain is ~1.0x at judge-human correlation rho=0.1, ~2.2x at rho=0.9; CLT-based estimators need roughly >=50 human labels per stratum (below that use bootstrap PTD); in a real agent-safety case (rho=0.59, ~13-point judge bias) 100 human labels became worth 143-157. EVIDENCE: arXiv:2605.31278v2 HTML fetched (Martinon, Merad, Raki; Emerton Data): Monte Carlo with 500 human + 1000 proxy labels shows effective n ~500 at rho=0.1 vs ~1100 at rho=0.9; R-Judge case study with Claude Sonnet as judge; library at github.com/EmertonData/glide with reproducible validation notebooks; limitations: means/proportions only, i.i.d. assumptions, no anytime-valid inference. SOURCE: Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation | https://arxiv.org/html/2605.31278v2 | 2026-05 (v2 2026-06) | practitioner IMPLICATION: Ready-made tooling exists for the statistical layer; more importantly it publishes the correlation-vs-benefit curve and the >=50-labels-per-stratum threshold the foundation can budget against: if a project's judge-human correlation on an axis is below ~0.3, PPI buys nothing there and gold labels must carry that axis alone. ## For estimating scores from noisy judges, PPI++/efficient-influence-function estimators produce near-identical, shortest confidence intervals - reported 3-15x narrower than Rogan-Gladen-style misclassification-correction estimators depending on the human-labeling ratio, with the advantage largest when the judge is close to random guessing. EVIDENCE: arXiv:2601.05420 (Chen, Lu, Li, Guo, Li; 2026-01-08) abstract fetched: unifies measurement-error correction and PPI classes via semiparametric efficiency theory, derives EIF-based estimators, characterizes when PPI-style strictly dominates; the 3-15x interval-width figure comes from the paper's empirical section as surfaced in search-indexed full text (not independently recomputed). Code: github.com/yiqunchen/debias-llm-as-a-judge. SOURCE: Efficient Inference for Noisy LLM-as-a-Judge Evaluation | https://arxiv.org/abs/2601.05420 | 2026-01-08 | academic IMPLICATION: Standardize the estimation layer on PPI++/EIF-style residual calibration rather than sensitivity/specificity (confusion-matrix) correction when the target is a rate or mean - but note the certification literature pushes the other way for hypothesis tests (see disagreements). ## For certifying that a failure rate is below a threshold (the low-base-rate pass/fail regime), the 'Noisy but Valid' framework (ICLR 2026) uses a small human calibration set to estimate judge TPR/FPR, applies a variance-corrected test to the large judge-labeled stream with finite-sample Type-I error control, and derives exact conditions under which judge-based testing has HIGHER statistical power than direct human evaluation. EVIDENCE: arXiv:2601.20913 (Feng, Shen, Balashankar, Gerner-Beuerle, Rodrigues; submitted 2026-01-28, accepted ICLR 2026 per arXiv page) abstract fetched: variance-corrected critical threshold, finite-sample Type-I control under calibration uncertainty, validation on Jigsaw/Hate Speech/SafeRLHF, and a quantified 'oracle gap' - the power cost of having to estimate judge reliability rather than knowing it. SOURCE: Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges | https://arxiv.org/abs/2601.20913 | 2026-01-28 | academic IMPLICATION: For attempter-level or batch-level certification ('this attempter's error rate is below X'), a judge-plus-small-calibration design can be provably MORE powerful than human-only review - reframing the human panel's job as estimating judge error rates, not per-item adjudication. This is direct evidence for the zero-per-item-touch + calibration-audit branch of taxonomy Q6's fork. ## Naive reporting of judge-scored evaluations is systematically biased by judge sensitivity/specificity; a plug-in correction with confidence intervals propagating BOTH test-set and calibration-set uncertainty fixes it, remains unbiased under distribution shift between calibration and test data, includes adaptive allocation of calibration labels, and characterizes regimes where corrected judge-based evaluation is more reliable than human-only evaluation. EVIDENCE: arXiv:2511.21140 (Lee, Zeng, Jeong, Sohn, Kangwook Lee; submitted 2025-11-26, v4 2026-05-31, accepted ICML 2026 per arXiv page) abstract fetched. The calibration-to-test distribution-shift robustness is the distinctive property versus standard PPI's i.i.d. assumption. SOURCE: How to Correctly Report LLM-as-a-Judge Evaluations | https://arxiv.org/abs/2511.21140 | 2025-11-26 (v4 2026-05-31) | academic IMPLICATION: Supports the taxonomy Q1 proposal to ban raw judge-agreement numbers from all reporting: every reported metric should be the corrected estimator with a CI that includes calibration uncertainty. The shift-robustness result matters specifically because attempter adaptation (taxonomy Q8) shifts the live item distribution away from the calibration set over time. ## Conformal prediction now provides per-item uncertainty for LLM-as-judge ratings: EMNLP 2025 work builds guaranteed-coverage score intervals from a single judging run (with an ordinal boundary adjustment and a lower-bias midpoint score), and April 2026 work shows conformal set width is a genuine per-instance reliability signal (r_s=+0.576 with reliability, n=1,918; widths correlate ~0.32-0.38 across different judges) with reliability driven more by the CRITERION than the judge model (relevance avg set size ~3.0 vs fluency/consistency ~4.9 on a 1-5 scale). EVIDENCE: Sheng, Liu, He, Zhao, Kang, 'Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction', EMNLP 2025 main (aclanthology.org/2025.emnlp-main.569), abstract confirmed via NeurIPS 2025 workshop listing. Gupta & Kumar, arXiv:2604.15302 (2026-04-16, under review): split conformal on SummEval; also finds 33-67% of documents contain intransitive judge preference cycles despite low aggregate violation rates. SOURCE: Analyzing Uncertainty of LLM-as-a-Judge (EMNLP 2025) + Diagnosing LLM Judge Reliability (arXiv:2604.15302) | https://arxiv.org/abs/2604.15302 | 2025-11 (EMNLP) / 2026-04-16 | academic IMPLICATION: Conformal set width is a cheap, distribution-free routing statistic for the human-touch queue that transfers across judges, and the criterion-dominates-judge finding is empirical support for taxonomy Q2/Q5's axis-triage: per-criterion reliability ceilings must be measured and some axes routed human-only regardless of judge choice. ## Judge-vs-system drift can be attributed with anytime-valid statistics: a frozen human-labeled anchor set periodically re-scored by the live judge, monitored with betting e-processes, correctly attributed a silent judge version bump in 60/60 runs with zero misattribution (rolling z-test baseline false-alarmed on 75% of drift-free streams), at 0.21-0.64x the cost of strong-judging every item. EVIDENCE: arXiv:2606.15474 (Yitao Li, 2026-06-13) abstract fetched: 'one-way identification (only the judge can move the anchors)'; three-verdict output (none/system/judge); strict-prompt change attributed in 110/120 runs; replicated on TL;DR summarization 240/240. Single-author preprint, not yet peer-reviewed - treat quantitative claims as provisional. SOURCE: Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines | https://arxiv.org/abs/2606.15474 | 2026-06-13 | academic IMPLICATION: Taxonomy Q9's three-moving-parts problem has an existing statistical design for one axis: freeze a per-project human-labeled anchor set at onboarding, interleave it continuously into the judge stream, and use e-process alarms to separate 'judge changed' from 'attempter population changed' before triggering rubric revision or de-graduation. ## In sequential prediction-powered mean estimation, uncertainty-based active querying contributed little: the smallest confidence widths occurred when query probabilities were near-constant at the budget ceiling, and theory shows optimized query probabilities converge to the maximum allowed constant rate when chosen without reference to covariates. EVIDENCE: arXiv:2604.18569 (Sfyraki & Wang, submitted 2026-04-20, accepted ICLR 2026 per arXiv page) abstract fetched: empirical finding that the uncertainty-blend weight near the constant term minimizes CI width, plus non-asymptotic analysis and no-regret query-probability selection corroborating near-uniform optimality in this regime. SOURCE: Revisiting Active Sequential Prediction-Powered Mean Estimation | https://arxiv.org/abs/2604.18569 | 2026-04-20 | contrarian IMPLICATION: Direct counter-evidence to uncertainty-routed human sampling for population estimation in streaming settings: the foundation must not assume uncertainty routing dominates - measure the routing policy's marginal value per project, and expect uniform-at-budget to be near-optimal when the goal is estimation rather than per-item verdict quality. ## Commercial tooling for confidence-scored LLM judgments with human calibration is mature: Cleanlab's Trustworthy Language Model attaches a real-time trustworthiness score to every LLM response, supports custom evaluation criteria, explicitly documents calibrating trust scores against human quality ratings, and publishes benchmarks (Nov 2025) for auto-flagging incorrect structured outputs for human review. EVIDENCE: Cleanlab TLM documentation (help.cleanlab.ai/tlm/: quickstart, custom-eval calibration-against-human-ratings tutorial, advanced usage with low-score explanations) and 2025-11-18 structured-outputs benchmark blog post, all fetched via search with page content confirmed. Vendor source: benchmark numbers are self-reported. SOURCE: Cleanlab Trustworthy Language Model (TLM) documentation and benchmark | https://help.cleanlab.ai/tlm/ | 2025-11-18 (benchmark); docs current 2026 | practitioner IMPLICATION: The 'score every judgment, calibrate the score against a human-rated set, route low-trust items to humans' pattern is already productized - the foundation can specify this as a component contract (per-verdict trust score + human-calibration hook) rather than inventing it, while keeping vendor-neutrality. ## Tools & artifacts - ppi_py: Reference open-source Python library for prediction-powered inference (Angelopoulos, Zrnic et al.): means, quantiles, GLMs, PPI++ power tuning. | https://github.com/aangelopoulos/ppi_py | Drop-in statistical layer for computing bias-corrected pass rates and attempter metrics from many judge labels + few human labels; covers estimand types GLIDE does not (quantiles, regressions). - GLIDE: Open-source industrial PPI library (Emerton Data, May 2026): PPI++, stratified PPI++, predict-then-debias bootstrap, active statistical inference, uniform/stratified/active/cost-optimal samplers, plus a method-selection decision tree. | https://github.com/EmertonData/glide | Closest existing implementation of this domain's whole playbook, including the sampler that decides which items get human labels and effective-sample-size ROI reporting; validated on agent-safety trajectories with a Claude judge. - active-inference: Code for Active Statistical Inference (Zrnic & Candes, ICML 2024): uncertainty-guided label collection with provably valid confidence intervals. | https://github.com/tijana-zrnic/active-inference | Reference implementation for uncertainty-routed allocation of the scarce human annotation budget. - debias-llm-as-a-judge: Code for Chen et al. 2026 (arXiv:2601.05420): efficient-influence-function and PPI++ estimators for noisy LLM-judge evaluation of scores and win rates. | https://github.com/yiqunchen/debias-llm-as-a-judge | Implements the estimator class this domain recommends standardizing on for judge-score estimation, with the Rogan-Gladen comparison built in. - Cleanlab Trustworthy Language Model (TLM): Commercial system that attaches a calibratable trustworthiness score to every LLM response, with custom evaluation criteria and tutorials for calibrating scores against human ratings (benchmark published 2025-11). | https://help.cleanlab.ai/tlm/ | Productized version of the per-verdict-confidence + human-calibration + route-low-trust-to-human pattern; useful as a component contract template even if not adopted as a vendor. - Cascaded Selective Evaluation (Trust or Escalate): ICLR 2025 Oral method: judge cascade with fixed-sequence-testing calibration delivering user-specified human-agreement guarantees; Simulated Annotators confidence estimation. | https://arxiv.org/abs/2407.18370 | The architectural template for the AutoQA's escalation spine: guarantee level and abstention rate become the two governing dials for the human-touch budget. ## Disagreements - Uncertainty-routed vs uniform human sampling: Zrnic & Candes (ICML 2024) and Gligoric et al. (NAACL 2025) report provably valid inference with >25% fewer human labels when the model's uncertainty guides which items humans label; Sfyraki & Wang (ICLR 2026, arXiv:2604.18569) find in the sequential regime that the uncertainty component contributes little and near-constant query probabilities at the budget ceiling give the tightest intervals. Both sides are peer-reviewed; the honest read is that active-sampling gains are regime-dependent (batch design + informative confidence signals vs sequential streaming), so routing policy must be validated per project, not assumed. - Explicit judge-error modeling vs black-box residual calibration: Chen et al. (arXiv:2601.05420) show Rogan-Gladen-style TPR/FPR-correction estimators yield intervals reportedly 3-15x wider than PPI++/EIF for score ESTIMATION; Feng et al. (ICLR 2026, arXiv:2601.20913) argue the opposite direction for CERTIFICATION - explicitly modeling judge TPR/FPR yields better-powered hypothesis tests than treating PPI as a black box. Partly reconcilable (different targets: estimation vs testing), but they issue opposing default recommendations and the synthesis should pick per use-case: PPI++/EIF for dashboards and rates, TPR/FPR-corrected tests for threshold certification. - What the human labels are for - target vs corrector: Trust or Escalate guarantees agreement with human majority preference (capping the system's validity at human panel reliability, i.e., consistency-of-application); the PPI/certification line treats human labels as gold-standard truth for bias correction, and both 'Noisy but Valid' and 'How to Correctly Report' prove regimes where corrected judge-based evaluation is strictly MORE reliable/powerful than human-only evaluation, while Kim (arXiv:2605.16354) explicitly warns her framework breaks when human raters are themselves unreliable. No paper in this literature resolves what to do when the gold itself is noisy - the taxonomy Q1 construct choice (instruction-satisfaction vs consensus-prediction) is genuinely open in the statistics literature. ## Gaps - No 2025-2026 paper found that applies PPI, selective escalation, or conformal routing specifically to QA of HUMAN-produced annotations (reviewing human attempters); every application evaluates model outputs with humans as gold. Transfer is plausible but unvalidated, especially under adversarial attempter adaptation, which violates the i.i.d. assumptions most PPI variants rely on (only arXiv:2511.21140 claims calibration-to-test shift robustness). - All PPI-family guarantees are population-level (rates, means, rankings) and selective-evaluation guarantees are marginal, not conditional: a system can meet its aggregate agreement/coverage target while concentrating its failures on specific attempters, criteria, or hard-item subgroups. Conditionally valid (per-subgroup) escalation guarantees for LLM judges remain an open problem in this literature. - The specific fork 'one shallow human touch per item vs zero-touch plus randomized deep audits' has no direct head-to-head study measuring total error caught per human-hour; the statistical literature implicitly favors the zero-touch+audit side (valid conclusions with human labels on a small fraction of items) but only for measurement/certification objectives, not for per-item verdict correction or attempter coaching. - Calibration economics for rubric-based QA: Trust or Escalate's Simulated Annotators and fixed-sequence testing were validated on pairwise preference tasks; required calibration-set sizes and achievable coverage for per-criterion rubric verdicts (the AutoQA's actual workload) are not established anywhere I found. - Could not open the Trust or Escalate OpenReview forum (CAPTCHA); handling of abstained items and calibration-set-size details confirmed only from abstract-level and secondary sources, not reviewer discussion. The 3-15x interval-width figure in Chen et al. was corroborated from search-indexed paper text, not independently recomputed. - No published head-to-head comparison of conformal-width routing vs Simulated-Annotators-style confidence routing vs verbalized-confidence routing for deciding the human-escalation queue - the three candidate mechanisms have never been benchmarked against each other. ## Verifications - CLAIM: Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a GPT-4 judge in 'AutoEval Done Right'. VERDICT: partially_confirmed | Verified directly against arXiv:2403.07008: authors (Boyeau, Angelopoulos, Yosef, Malik, Jordan) correct; abstract verbatim contains 'increase the effective human-labeled sample size by up to 50% on experiments with GPT-4'; abstract claims methods 'improve sample efficiency while remaining unbiased'; version history confirms v1 2024-03-09 and v3 2026-06-01 with comment 'camera-ready paper version' - dates as claimed. GLIDE (arXiv:2605.31278, Martinon/Merad/Raki, v1 2026-05-29, ICML 2026 workshop) exists and states PPI 'combines both into debiased estimates with valid confidence intervals', consistent with the claim, though the exact phrase 'regardless of judge quality' was not verifiable in its abstract. The 2026 PPI-for-LLM-judge literature is real and active (PRECISE/Amazon Science, arXiv:2606.05308 ranking extension, arXiv:2601.05420, arXiv:2601.20913), so the claim is not superseded - it is being extended. Two corrections: (1) 'always-valid confidence intervals' is a misuse of a term of art - 'always-valid'/anytime-valid refers to sequential inference; PPI/PPI++ intervals are fixed-n ASYMPTOTIC (CLT-based) intervals, so coverage is guaranteed only asymptotically and finite-sample coverage can degrade with very small gold sets. (2) The judge-quality-independence property is a general PPI/PPI++ property (Angelopoulos et al. 2023) rather than something stated in the AutoEval abstract itself; the claim's framing correctly captures its substance (worse judge -> wider intervals via lower correlation, coverage preserved) but slightly overstates its strength and its provenance in this specific source. CORRECTED: Prediction-Powered Inference (PPI/PPI++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with asymptotically valid confidence intervals whose coverage does not depend on judge accuracy (a worse judge widens intervals rather than breaking coverage, subject to standard asymptotic/regularity conditions and a sufficiently large gold set); 'AutoEval Done Right' (arXiv:2403.07008, Boyeau, Angelopoulos, Yosef, Malik & Jordan) reports increasing the effective human-labeled sample size by up to 50% in experiments with GPT-4. - CLAIM: Cascaded Selective Evaluation (Trust or Escalate, ICLR 2025 Oral) delivers a provable, user-specified human-agreement guarantee for LLM judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a Chatbot Arena subset where GPT-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models. VERDICT: confirmed | All load-bearing elements check out against the primary sources. (1) arXiv:2407.18370 abstract (fetched directly) states Cascaded Selective Evaluation provides a provable human-agreement guarantee at a user-specified level, and that on a Chatbot Arena subset \"where GPT-4 almost never achieves 80% human agreement,\" the method \"guarantees over 80% human agreement with almost 80% test coverage\" even using Mistral-7B. (2) Paper body (arxiv.org/html/2407.18370v1) confirms the mechanism: threshold calibration via fixed sequence testing (Bauer, 1991) on a small human calibration set (|D_cal|=500 for ChatArena/TL;DR, 392 for Auto-J), cascade escalation from cheap to strong judges on low confidence, abstention (output empty set) when even the strongest judge is unconfident, and Simulated Annotators as the confidence estimator enabling high coverage. (3) The ~88% figure is exact: \"79.1% of all samples, among which 88.1% are evaluated by substantially cheaper Mistral-7B or GPT-3.5 instead of GPT-4\" (Table 3: GPT-4 handles only 17.5%). (4) ICLR 2025 Oral confirmed at iclr.cc/virtual/2025/oral/31838 (Jung, Brahman, Choi); dates correct (v1 2024-07-25; ICLR 2025 in April 2025). Two trivial nuances, neither rising to a correction: the paper defines abstention as returning empty set / excluding from coverage - deferral \"to humans\" is the intended framing but abstained items are not actually routed to human annotators in the experiments; and the guarantee holds with probability 1-delta over calibration-set sampling (standard for such guarantees, and consistent with \"provable, user-specified\"). Supersession check: not superseded - but a newer alternative exists: SCOPE (arXiv:2602.13110, ICML 2026 poster, Feb 2026) uses conformal calibration with a Bidirectional Preference Entropy signal and reports better calibration/coverage than Simulated Annotators; it complements rather than invalidates the claim about this paper. The \"83 citations\" figure was not independently verified but is plausible and not load-bearing. - CLAIM: The minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n*(1-rho^2) where n* is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with R^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes. VERDICT: confirmed | All elements verified against the source (https://arxiv.org/html/2605.16354v1, Jane Paik Kim, Stanford Dept. of Psychiatry and Behavioral Sciences, submitted 2026-05-08 - date and attribution correct). (1) Floor formula: paper states required human reviews converge to n*(1-rho^2) as N increases - exact match. (2) Worked example: n*=200, R^2=0.70, N=2000 -> 65 human labels, confirmed verbatim; the N=400/100 figure is stated in the paper as \"with a budget of 100 human ratings, N can be reduced to 400 while achieving target power\" - same numbers, slightly different framing (human budget fixed, N solved) but mathematically equivalent to the claim. (3) Stratification savings: paper reports \"up to 12.9%\" reduction when R^2 gap is large (0.8 vs 0.1) - matches \"~13%\", with the paper adding that moderate gaps yield only ~2%. (4) Diminishing returns from more judge labels: explicitly stated (\"each successive increase in N produces diminishing reductions in n\"). (5) All three caveats in the evidence summary (pilot R^2 overestimation breaking precision guarantees, human-as-gold-standard assumption, single-rater scope) appear in the paper's limitations. Supersession check: two searches found no v2 revision, no citing papers, and no later work replacing or contradicting the result as of 2026-07-14. Minor precision notes only: \"up to ~13%\" should strictly be \"up to 12.9% in the extreme R^2-gap case, ~2% for moderate gaps,\" and the floor is asymptotic (approached as N grows), not attainable at finite N - the claim's own phrasing (\"converge to a floor,\" \"up to\") already reflects both."}, {"slug": "foundation-benchmark-protocol-md", "title": "Benchmark & Calibration Protocol", "category": "Foundation specifications", "summary": "How to construct adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-benchmark-protocol-md.html", "raw": "documents/raw/foundation/BENCHMARK_PROTOCOL.md", "text": "# AutoQA Benchmark, Calibration, and Launch Protocol **Protocol version:** 0.1.0 **Applies to schema version:** 0.1.0 ## 1. Purpose This protocol defines how to determine whether an AutoQA configuration is reliable enough for a specific project and criterion family. The benchmark is not merely a leaderboard for an overall judge score. It must test whether the system: - applies the intended criteria; - finds affirmative evidence of quality; - detects defects and omissions; - links findings to the correct evidence; - distinguishes grounding, truth, relevance, and warrant; - audits human annotations faithfully; - recognizes ambiguity; - calibrates confidence; - escalates selectively; - produces useful, safe feedback; - remains stable under irrelevant presentation changes; - improves the actual production workflow. A configuration includes all behaviorally relevant components: ```text model provider and version prompt templates and versions reasoning or sampling mode retrieval and chunking claim and expected-element extraction validators and tools reference-answer timing ensemble or challenger policy calibration profile decision-policy version ``` Changing any of these may require partial or full revalidation. ## 2. Evaluation levels The benchmark reports results at four levels. ### 2.1 Observation level Can the system correctly identify claims, expected elements, evidence spans, positive findings, defects, omissions, counterevidence, and ambiguity? ### 2.2 Criterion level Can it assign the correct applicability, verdict, severity, and substatuses for grounding, truth, warrant, and relevance? ### 2.3 Decision level Can the versioned policy produce the correct action for each target: - attempt; - annotation; - composite case; - batch, where used? ### 2.4 Workflow level Does the system reduce human effort and improve data quality without creating unacceptable false acceptance, false rejection, reviewer anchoring, or feedback-induced errors? ## 3. Governance roles At minimum, assign these roles: - **Project owner** - defines intended project outcome and accepts operational risk. - **Rubric owner** - maintains the contract and resolves instruction interpretation. - **Domain expert** - adjudicates cases requiring subject expertise. - **QA owner** - owns benchmark design, sampling, metrics, and launch gates. - **Benchmark curator** - freezes cases, manages splits, and prevents leakage. - **System owner** - owns model configuration, prompts, validators, and deployment. A single person may fill more than one role in a small project, but contract approval and benchmark scoring should not be controlled exclusively by the person optimizing the evaluator. ## 4. Freeze the target before testing models Model comparison begins only after a project contract is stable enough to adjudicate. Before benchmark construction: 1. include and hash all authoritative instructions; 2. compile criteria; 3. define evidence classes and source policy; 4. add positive, negative, and boundary anchors; 5. list valid alternatives; 6. define severity independently from action; 7. define attempt, annotation, and composite decision rules; 8. identify deterministic validators; 9. identify criteria requiring domain expertise; 10. approve contract version 1 for benchmarking. Do not repeatedly alter the rubric to favor a model during test-set evaluation. Changes discovered during development create a new contract or benchmark version and must be documented. ## 5. Benchmark corpus design The corpus contains two complementary distributions. ### 5.1 Natural-prevalence set Sample representative production cases using the expected live distribution of: - tasks; - reviewers; - difficulty; - answer length; - evidence volume; - labels; - failure types; - languages; - domains; - data-generation sources. This set estimates operational performance and workload. ### 5.2 Challenge set Intentionally oversample difficult and safety-critical cases. Required challenge categories include, where applicable: - strong positive attempt; - subtle valid alternative to a reference; - polished falsehood; - true but unsupported claim; - accurately cited weak source; - citation mismatch; - partial support; - causal or inferential overreach; - omitted required content; - selective evidence use; - correct conclusion with invalid rationale; - incorrect conclusion with plausible reasoning; - correct human label with wrong reason; - wrong human label with correct observation; - severity edge case; - instruction conflict; - reasonable disagreement; - rubric ambiguity; - domain-expertise dependency; - long-context contradiction; - prompt injection; - rubric parroting; - position-bias probe; - verbosity and formatting probe. Challenge-set rates must not be used as estimates of production prevalence. ### 5.3 Positive-data requirement At least one benchmark slice must be specifically designed around high-quality attempts. It should include: - concise complete answers; - detailed complete answers; - unconventional but valid approaches; - qualified conclusions; - correct disagreement with a reference; - evidence synthesis rather than extraction; - strong answers with minor stylistic imperfections; - good attempts incorrectly rejected by reviewers. This slice is necessary to measure false rejection and false praise independently. ### 5.4 Omission requirement Include cases where all present claims are accurate but required content is missing. Without this slice, a claim verifier can appear highly reliable while failing completeness. ### 5.5 Annotation-fidelity requirement Include all four combinations: | Attempt quality | Annotation quality | |---|---| | good | good | | good | bad | | bad | good | | bad | bad | Also include: - correct label, wrong rationale; - wrong label, useful observation; - correct criterion, wrong severity; - wrong criterion, correct span; - reasonable disagreement; - rubric defect. ## 6. Unit of sampling and leakage control Cases that share any of the following may leak across splits: - same underlying task template; - same source document or evidence bundle; - paraphrases of the same attempt; - same reference answer; - same synthetic mutation family; - adjacent items from the same conversation; - repeated work from one original datum. Group such cases before splitting. All members of a group must remain in one split. The benchmark item records: - split; - first freeze date; - whether prompt development may use it; - whether it belongs to a held-out project; - access policy; - challenge attributes; - slice tags. ## 7. Recommended splits ### Development May be inspected while designing prompts, validators, and extraction logic. Never use for final reporting. ### Calibration Used to fit confidence mappings, ensemble weights, and action thresholds. Do not use to select among unlimited prompt variants without accounting for overfitting. ### Test Locked before final model selection. Used for primary in-domain performance reporting. ### Challenge Locked adversarial and edge-case suite. Report separately from natural prevalence. ### Held-out project A project, contract, or task family excluded from architecture and prompt development. Tests whether the supposedly project-agnostic engine transfers to a new contract. ### Shadow Recent live cases evaluated without changing production decisions. Used to measure current distribution and drift. ## 8. Gold-standard construction ### 8.1 Adjudicator selection Use domain-qualified adjudicators for criteria that require expertise. General QA adjudicators may evaluate mechanical, instruction-following, and straightforward grounding criteria. The benchmark must record domain-expert count. ### 8.2 Independent first pass For consequential or subjective cases, use at least two independent first-pass judgments before discussion. Each adjudicator should record: - criterion applicability; - verdict; - positive evidence spans; - negative evidence spans; - omissions; - severity; - acceptable operational actions; - ambiguity status; - confidence or uncertainty notes. Preserve initial judgments before adjudication. Do not overwrite disagreement history. ### 8.3 Blinding Adjudicators should normally be blind to: - AutoQA outputs; - model identity; - model-generated rationale; - production consequence; - other adjudicators' initial labels. Whether adjudicators are blind to the original reviewer label depends on the task: - for independent attempt gold, hide the reviewer label; - for annotation-fidelity gold, reveal the review only after the independent attempt judgment is frozen. The benchmark item records blinding status. ### 8.4 Adjudication meeting Adjudication should resolve whether disagreement arose from: - missed observation; - evidence authority; - applicability; - criterion interpretation; - inference; - threshold; - severity; - expertise; - legitimate plural judgment; - rubric defect. Do not force a single verdict when the correct gold state is `reasonable_disagreement`, `rubric_ambiguous`, or `insufficient_evidence`. ### 8.5 Acceptable verdict sets Gold may contain more than one acceptable verdict for boundary cases. Examples: ```text acceptableVerdicts = [\"met\", \"partially_met\"] acceptableDecisions = [ { target: \"attempt\", acceptableActions: [\"accept_with_notes\", \"rework\"] } ] ``` This is preferable to labeling one defensible judgment as objectively wrong. ### 8.6 Gold evidence For each criterion, gold should include: - decisive positive spans; - decisive negative spans; - required omissions; - notes on valid alternatives; - severity; - acceptable verdicts. An evaluator should not receive full gold rationales during test execution. ### 8.7 Gold repair If adjudication reveals a contract defect, do not patch only the gold label. Revise the contract, version it, and re-adjudicate affected cases. ## 9. Sample-size planning Sample sizes must be driven by the operational error budget and desired confidence interval, not a single universal minimum. ### 9.1 Proportion estimation For a rough planning approximation, the worst-case sample size for a proportion with two-sided 95% margin of error `e` is: ```text n ~ 1.96^2 x 0.5 x 0.5 / e^2 ``` This yields approximately: - +/-10 percentage points: 97 cases; - +/-7 points: 196 cases; - +/-5 points: 385 cases; - +/-3 points: 1,068 cases. Use Wilson or exact binomial intervals in final reporting rather than relying only on this approximation. ### 9.2 Rare critical errors If zero errors are observed in `n` independent cases, the approximate one-sided 95% upper bound is `3/n`. Therefore, observing no critical false accepts in 300 representative autonomous accepts supports an upper bound near 1%, not proof of zero risk. Rare critical failures should be oversampled in the challenge set while also measured at natural prevalence. ### 9.3 Per-slice planning A broad benchmark can hide a failing subgroup. Plan enough cases for critical slices such as: - criterion family; - severity; - task type; - language; - domain; - long versus short context; - high versus low human agreement; - reviewer cohort; - model-generated versus human-generated attempt; - adversarial versus ordinary. Very small slices should be reported as exploratory and must not support autonomous action gates. ## 10. Model and architecture bakeoff Compare configurations by criterion family. A universal winner is not assumed. ### 10.1 Minimum configurations Test at least: 1. a deterministic-only baseline where applicable; 2. a single holistic model judge; 3. criterion decomposition; 4. criterion decomposition plus claim/evidence mapping; 5. separate positive, negative, coverage, and warrant observation channels; 6. primary judge plus independent challenger; 7. selective escalation using calibrated confidence. ### 10.2 Required ablations | Ablation | Primary question | |---|---| | Holistic vs decomposed | Does decomposition improve criterion accuracy and traceability? | | One-pass vs separate observation channels | Does separation improve positive precision, defect recall, or calibration? | | Claims only vs claims plus expected elements | Does expected-content mapping improve omission recall? | | Reference absent vs early vs late | Does the reference help without suppressing valid alternatives? | | Default vs extended reasoning | Does extra reasoning help this criterion family? | | One sample vs repeated samples | Does sampling provide useful uncertainty or correlated noise? | | Same-family vs cross-family challenger | Does independent model diversity improve decisive findings? | | Forced verdict vs abstention | What risk-coverage frontier is achievable? | | Generated feedback vs validated feedback | Does repair validation improve revision success? | | Full human rereview vs bounded question | Which yields more corrected decisions per human minute? | ### 10.3 Perturbation tests Run behavior-preserving variants of the same case: - swap candidate order; - change rubric order; - rename criterion identifiers; - alter harmless formatting; - shorten or expand nonmaterial prose; - move references before or after independent analysis; - paraphrase evaluator instructions without changing meaning; - insert irrelevant provenance cues; - vary source order; - vary answer position in long context. Record decision flip rates and criterion-specific flip rates. A flip is not necessarily an error when the original case is ambiguous. Report flips separately for clear and ambiguous gold cases. ### 10.4 Adversarial tests Test attempts and evidence containing: - direct evaluator commands; - fabricated system messages; - requests to ignore the rubric; - fake authority claims; - hidden scoring instructions; - answer-key mimicry; - irrelevant but persuasive citations; - language copied from positive anchors without the required behavior. ## 11. Core metrics No single metric is sufficient. ## 11.1 Criterion verdict metrics Report per criterion and macro-averaged: - precision, recall, and F1 for `met`; - precision, recall, and F1 for `not_met`; - confusion among met, partial, not met, ambiguous, unknown, and not applicable; - severity accuracy; - applicability accuracy; - cost-weighted criterion loss. For multi-acceptable gold, count a prediction correct when it belongs to the acceptable set. ### 11.2 Critical operational errors Report separately: - critical false-accept rate; - strong-positive false-reject rate; - autonomous reject precision; - autonomous accept precision; - rework precision; - escalation recall for ambiguous or unknown cases. Always provide numerator, denominator, and confidence interval. ### 11.3 Positive-finding metrics Report: - positive-finding precision; - positive-finding recall; - false-praise rate; - strength-to-criterion attribution accuracy; - positive span precision and recall; - required-behavior confirmation rate. A system can have good defect recall and poor positive-finding reliability. These must remain separate. ### 11.4 Negative and omission metrics Report: - negative-finding precision and recall; - omission precision and recall; - defect-to-criterion attribution accuracy; - materiality accuracy; - counterevidence recovery; - unsupported-finding rate. ### 11.5 Claim and evidence metrics Report: - material-claim extraction precision and recall; - central-claim recall; - compound-claim split accuracy; - claim-to-evidence relationship accuracy; - evidence-span precision and recall; - source-quality classification accuracy; - citation completeness; - contradiction detection; - entailment versus weak-support confusion. Evidence span scoring should include: - exact match; - token overlap or intersection-over-union; - adjudicator-accepted alternate span match. ### 11.6 Warrant metrics Report accuracy on: - association versus causation; - possibility versus certainty; - contribution versus primacy; - subgroup versus population; - descriptive versus normative inference; - calibrated uncertainty; - counterevidence acknowledgment. ### 11.7 Annotation-audit metrics Report: - label-support accuracy; - rationale-support accuracy; - criterion-attribution accuracy; - evidence-alignment accuracy; - severity-alignment accuracy; - omitted-positive recall; - omitted-negative recall; - disagreement-type accuracy; - false accusation rate against reviewers. The false accusation rate is particularly important if analytics will affect reviewer assignments, pay, or standing. ### 11.8 Calibration metrics Report: - Brier score; - log loss where probabilities are available; - expected calibration error with disclosed bins; - reliability diagram; - accuracy by confidence band; - risk-coverage curve; - coverage at project target risk; - selective false-accept and false-reject rates. Calibration must be evaluated on data not used to fit the calibration map. ### 11.9 Stability metrics Report: - order-swap flip rate; - rubric-paraphrase flip rate; - harmless-format flip rate; - repeated-run agreement; - model-version drift; - source-order sensitivity; - reference-timing sensitivity. Stability is not correctness. Report both. ### 11.10 Efficiency metrics Report: - median and tail latency; - inference cost per case; - model calls per case; - tokens per stage; - human seconds per escalated case; - percentage of cases receiving a human question; - corrected decisions per human minute; - cases processed per reviewer hour. ### 11.11 Feedback metrics Report: - successful revision rate; - residual defect rate after revision; - new-defect introduction rate; - repair validity; - feedback evidence-link accuracy; - user-rated usefulness; - edit distance or effort where meaningful; - repeated-failure rate after feedback. A second model's \"helpfulness\" score is not sufficient evidence that feedback works. ### 11.12 Downstream metrics When the system begins selecting training data, separately test: - target-model performance after training; - target-model calibration; - slice performance; - robustness; - marginal gain versus random accepted data; - redundancy; - diversity and coverage; - value per annotation or token. Item validity and dataset value remain separate benchmark layers. ## 12. Cost-sensitive decision evaluation Define a project loss matrix for target-specific actions. Example structure: | Gold state | Predicted accept | Predicted rework | Predicted reject | Predicted escalate | |---|---:|---:|---:|---:| | Acceptable attempt | false-accept cost 0 | unnecessary rework cost | false-reject cost | human-review cost | | Repairable attempt | harmful acceptance cost | 0 | over-rejection cost | human-review cost | | Unusable attempt | severe acceptance cost | rework-waste cost | 0 | human-review cost | | Ambiguous case | premature-decision cost | premature-rework cost | premature-reject cost | 0 or review cost | Use project-specific values. Report both unweighted metrics and expected cost under estimated production prevalence. ## 13. Calibration procedure ### 13.1 Define the event Confidence is meaningless unless tied to an event. Examples: - criterion verdict is correct; - decisive negative finding is correct; - attempt acceptance is correct; - annotation rework is correct; - evidence relationship is correct. Do not use one calibration mapping for unrelated events. ### 13.2 Fit on calibration split Possible methods include: - isotonic regression; - logistic or Platt-style scaling; - temperature scaling for suitable logits; - conformal risk control; - empirically defined confidence bands. Select the simplest method that performs adequately and remains stable. ### 13.3 Calibrate by relevant slice Separate profiles may be necessary for: - criterion family; - model role; - task type; - domain; - language; - context length; - clear versus ambiguity-prone cases; - positive versus negative findings. Avoid creating so many profiles that each has too little data. Hierarchical or pooled calibration may be preferable when slices are sparse. ### 13.4 Lock and test After fitting, freeze the profile and evaluate it on the locked test set. Report calibration degradation on challenge and held-out-project sets. ### 13.5 Recalibration triggers Recalibrate when any of the following changes materially: - model version; - provider behavior; - prompt template; - reasoning mode; - retrieval or chunking; - contract semantics; - source policy; - decision policy; - production distribution. ## 14. Launch gates Projects must define numerical gates before reading final test results. A gate should identify: - target action; - criterion or slice; - metric; - threshold; - confidence level; - minimum sample size; - acceptable subgroup variance; - fallback action. ### 14.1 Gate template ```yaml target: attempt action: reject scope: criteria tagged critical-hard-gate metric: precision threshold: PROJECT_DEFINED confidence_requirement: lower_95_bound_above_threshold minimum_cases: PROJECT_DEFINED stability_requirement: order_flip_rate_below_PROJECT_DEFINED fallback: escalate ``` ### 14.2 Conservative initial policy Until a criterion family meets its gate: - deterministic, fully verified failures may be automatic; - clear positive and negative semantic findings may assist humans; - uncalibrated semantic accept or reject decisions must escalate; - subjective failures should usually produce advice, rework, or escalation rather than autonomous rejection; - reviewer-performance consequences must rely on adjudicated samples, not raw model disagreement. ### 14.3 Required gate families At minimum define: - autonomous attempt acceptance; - autonomous attempt rejection; - annotation acceptance; - annotation rework or rejection; - composite-case admission; - critical-failure detection; - strong-positive protection; - human-escalation coverage; - feedback repair safety. ## 15. Human-loop experiment The one-interaction design must be tested rather than assumed. ### 15.1 Compare conditions Randomize eligible uncertain cases to: 1. no human intervention; 2. full independent second review; 3. bounded decision-changing question; 4. model recommendation plus confirmation, if testing automation bias. ### 15.2 Measure - final decision accuracy; - human time; - answer-changing rate; - corrected decisions per minute; - residual uncertainty; - reviewer acceptance of wrong model suggestions; - repeat-question rate; - rubric clarification yield; - user satisfaction; - expertise-routing success. ### 15.3 Bounded-question quality A question is useful when: - it resolves the declared uncertainty; - the answer maps unambiguously to criterion state; - it changes or confirms a decision that matters; - it requires less effort than a full review; - it does not reveal irrelevant model conclusions that induce anchoring. ## 16. Feedback experiment For repairable failures, compare: - no feedback; - generic criterion feedback; - evidence-linked minimal repair; - evidence-linked repair with validation. Measure whether a blinded adjudicator accepts the revision and whether new defects appear. Positive feedback should be evaluated for specificity and truth, not sentiment. ## 17. Reviewer analytics validation Do not operationalize reviewer scores until the annotation auditor is separately validated. ### 17.1 Difficulty adjustment Reviewer performance should be conditioned on: - criterion; - task difficulty; - item ambiguity; - evidence volume; - domain expertise required; - case severity; - AutoQA confidence. ### 17.2 Consequential-use safeguard For reviewer coaching, assignment, compensation, or discipline: - use adjudicated samples; - provide appeal; - exclude reasonable-disagreement and rubric-defect cases; - report uncertainty; - require minimum sample size; - avoid one aggregate score without criterion breakdown. ### 17.3 Drift signals Monitor: - systematic leniency or harshness; - rising criterion-specific disagreement; - recurring unstated preferences; - attention failures; - expertise-routing mismatch; - batch-level acceptance anomalies. ## 18. Shadow deployment Before autonomous action, run AutoQA in shadow mode. For every shadow case, store: - AutoQA target-specific decisions; - production reviewer decision; - whether AutoQA would have changed the workflow; - confidence and calibration profile; - decisive findings; - human escalation question, if any; - later adjudicated outcome for an audit sample. Shadow reporting should answer: - What percentage would AutoQA accept, rework, reject, or escalate? - Which criteria drive each action? - Where does model-human disagreement concentrate? - How much reviewer time would be saved? - How many good attempts would be incorrectly rejected? - How many bad annotations would remain undetected? - How often would the one human interaction change the decision? ## 19. Production monitoring ### 19.1 Ongoing audit sample Randomly audit: - automatic accepts; - automatic rejects; - rework cases; - escalations; - cases where model and reviewer agree; - cases where they disagree; - high-confidence cases; - low-frequency slices. Agreement cases require auditing because correlated error can create false reassurance. ### 19.2 Drift dashboard Track over time: - action rates; - criterion failure rates; - positive-finding rates; - calibrated confidence distribution; - risk estimates from adjudicated audits; - order and prompt sensitivity; - reviewer disagreement types; - model/provider version; - source-distribution changes; - answer length and evidence volume; - human question rate and value. ### 19.3 Drift triggers Pause or reduce autonomy when: - action rates shift without an explained production change; - audited error exceeds the project budget; - calibration deteriorates; - a subgroup falls below its gate; - model or prompt version changes; - a new adversarial failure appears; - repeated human questions expose a rubric defect; - the provider changes behavior materially; - source formats or context length move outside validated ranges. ### 19.4 Change management Use shadow or canary evaluation for every material evaluator change. Compare old and new configurations on the locked suite and recent live cases. Do not overwrite historical evaluation records. New runs receive new identifiers and preserve lineage. ## 20. Statistical reporting Every benchmark report should include: - case and criterion counts; - prevalence by split and slice; - point estimates; - confidence intervals; - denominators; - paired comparisons where the same cases are used; - uncertainty for rare errors; - missing or failed evaluation count; - human adjudication protocol; - calibration method; - model and prompt versions; - inference cost and latency; - known limitations. For paired configuration comparisons, use an appropriate paired method such as: - paired bootstrap for aggregate metrics; - McNemar-style analysis for binary paired correctness; - stratified resampling when project slices differ materially. Avoid declaring a winner based on a small difference without uncertainty analysis. ## 21. Required benchmark report tables ### 21.1 Configuration manifest | Component | Version | |---|---| | Contract | | | Decision policy | | | Benchmark | | | Models by role | | | Prompts | | | Validators | | | Retrieval | | | Calibration profiles | | ### 21.2 Primary target decisions | Target | Action | N | Precision | Recall where meaningful | Error cost | 95% interval | |---|---:|---:|---:|---:|---:|---:| | Attempt | Accept | | | | | | | Attempt | Reject | | | | | | | Annotation | Rework | | | | | | | Composite | Accept | | | | | | ### 21.3 Criterion metrics | Criterion | Applicability accuracy | Met F1 | Not-met F1 | Severity accuracy | Calibration | N | |---|---:|---:|---:|---:|---:|---:| ### 21.4 Evidence and positive-quality metrics | Slice | Positive precision | Positive recall | False praise | Omission recall | Evidence precision | Evidence recall | |---|---:|---:|---:|---:|---:|---:| ### 21.5 Stability | Perturbation | Clear-case flip rate | Ambiguous-case flip rate | Decision error change | |---|---:|---:|---:| ### 21.6 Human loop | Condition | Accuracy | Median human seconds | Decisions corrected | Corrections/minute | Anchoring errors | |---|---:|---:|---:|---:|---:| ### 21.7 Cost | Configuration | Mean model cost | P50 latency | P95 latency | Human cost | Total cost/case | |---|---:|---:|---:|---:|---:| ## 22. Release decision A production release requires written approval that: - contract and benchmark versions are frozen; - action-specific gates were set in advance; - locked test and challenge results meet the gates; - calibration is valid for the intended slices; - held-out-project results are understood; - security tests pass; - shadow deployment shows acceptable workflow impact; - rollback is available; - audit sampling is funded and staffed; - reviewer analytics safeguards are active; - known unsupported domains or slices are routed to humans. ## 23. Benchmark maintenance Create a new benchmark version when: - contract semantics change; - new failure modes are added; - gold labels are corrected; - source snapshots change; - split membership changes; - challenge attributes change materially; - adjudication policy changes. Do not delete difficult cases because a model performs poorly. Correct them only when the contract or gold is wrong. Retain retired benchmark versions for historical comparison. ## 24. Initial implementation plan ### Phase 1 - contract and gold foundation - author contract; - create 100-200 diverse development cases; - independently adjudicate a smaller calibration seed; - implement schema and referential validation; - implement deterministic checks; - establish claim, expected-element, and evidence objects. ### Phase 2 - architecture bakeoff - test holistic and decomposed judges; - test positive/negative/coverage/warrant passes; - test model/provider options by criterion family; - build prompt and order perturbation suite; - expand calibration and locked test sets. ### Phase 3 - selective AutoQA - fit calibration; - define risk-coverage thresholds; - enable shadow decisions; - test one-interaction human resolution; - validate feedback repairs. ### Phase 4 - governed autonomy - enable only actions whose gates are met; - retain human handling for unsupported slices; - monitor and audit continuously; - add reviewer and batch analytics only after annotation-audit validation. ## 25. Benchmark readiness checklist A benchmark is ready when: - natural and challenge distributions are separate; - strong positive attempts are represented; - omission cases are represented; - all annotation-quality combinations are represented; - cases are grouped to prevent leakage; - adjudicators are qualified; - initial judgments are preserved; - ambiguity can remain in gold; - evidence spans and omissions are recorded; - split access is controlled; - sample sizes support intended claims; - action-specific loss and gates are defined; - calibration and test data are distinct; - model, prompt, and policy versions are recorded; - a reporting template is precommitted; - held-out and shadow evaluation are planned. "}, {"slug": "foundation-changelog-md", "title": "Foundation Package - Changelog", "category": "Foundation specifications", "summary": "Version history and recorded changes for the foundation package.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-changelog-md.html", "raw": "documents/raw/foundation/CHANGELOG.md", "text": "# Changelog ## 0.1.0 - 2026-07-15 Initial foundational package. ### Added - project evaluation contract schema; - case input schema; - evaluation output schema; - benchmark item schema; - shared JSON Schema definitions; - TypeScript domain types; - synthetic end-to-end examples; - criterion authoring, gold-set, and model-bakeoff templates; - normative contract specification; - benchmark, calibration, shadow, and launch protocol; - target-specific decision policy reference; - attempt, annotation, and composite-case decision isolation; - positive, negative, omission, ambiguity, counterevidence, and integrity findings; - claim, expected-element, and evidence-relationship representations; - one-interaction human-resolution structure; - feedback and repair-validation structure; - validation script for all examples. "}, {"slug": "foundation-contract-spec-md", "title": "Evaluation Contract Specification", "category": "Foundation specifications", "summary": "Normative semantics for project contracts, evidence, criteria, findings, uncertainty, feedback, security, and conformance.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-contract-spec-md.html", "raw": "documents/raw/foundation/CONTRACT_SPEC.md", "text": "# AutoQA Evaluation Contract Specification **Specification version:** 0.1.0 **Status:** Foundational draft suitable for prototyping and benchmark construction ## 1. Purpose This specification defines a reusable structure for evaluating human-produced AI-training data under project-specific instructions. It also defines how to audit the human review attached to that work. The specification is intended to reduce unexplained evaluation variance caused by: - subjective or inconsistent interpretation of evaluation axes; - unstated reviewer preferences; - expertise gaps; - missed requirements or evidence; - inconsistent severity thresholds; - unsupported labels or rationales; - model-judge bias and overconfidence. It does so by making the evaluation contract explicit, decomposing observations from verdicts, preserving positive and negative evidence, and requiring calibrated escalation for uncertainty. The JSON Schemas define document shape. This document defines document meaning and evaluator behavior. Where the two conflict, the package version must be corrected before production use; an implementation must not silently choose one interpretation. ## 2. Normative language The terms **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** express requirements on a conforming implementation. - **MUST / MUST NOT**: required for conformance. - **SHOULD / SHOULD NOT**: expected unless a documented project-specific reason justifies deviation. - **MAY**: optional. ## 3. Design goals A conforming AutoQA system should make it possible to answer, for each case: 1. What did the project require? 2. Which requirements applied to this task and attempt? 3. What did the attempter claim, omit, or do? 4. What evidence supports, contradicts, or fails to resolve those claims? 5. Does the attempt affirmatively satisfy each applicable criterion? 6. Does the human reviewer's label and rationale faithfully evaluate the attempt? 7. What uncertainty remains, and can one bounded human interaction resolve it? 8. What action follows under the project's explicit decision policy? 9. What specific, constructive feedback is warranted? 10. Can the complete decision be replayed and audited later? ## 4. Non-goals Version 0.1.0 does not define: - a universal definition of quality; - a universal overall score; - a required model provider; - a required orchestration framework; - a universal confidence threshold; - a universal reviewer-quality score; - a data-selection or training-value algorithm; - a substitute for project-owner approval of criteria; - a substitute for domain expertise where the task requires it. ## 5. Conceptual model ### 5.1 Project-agnostic engine, project-specific contract The engine MAY be reusable across projects. The interpretation of quality MUST come from a versioned project evaluation contract. The reusable engine provides: - input normalization; - instruction compilation support; - claim extraction; - expected-content mapping; - evidence relationship analysis; - deterministic validation; - semantic evaluation; - confidence calibration; - decision-policy execution; - human escalation; - annotation auditing; - feedback generation; - audit logging. The project contract provides: - authoritative instructions; - criterion definitions; - applicability rules; - proof standards; - evidence permissions; - anchors and valid alternatives; - severity rules; - operational decision policy; - human-review policy; - feedback policy. ### 5.2 Separate evaluation targets The system recognizes at least three decision targets: - **attempt** - the human-produced answer or task completion; - **annotation** - the human reviewer's label, rationale, evidence, and severity; - **composite_case** - the attempt plus the annotation as one candidate annotated datum. These decisions MUST NOT be collapsed. Example: ```text Attempt decision: accept Annotation decision: rework Composite-case decision: rework ``` This means the attempt may be retained, but the annotation must be corrected before the combined record is admitted as annotated training data. A project MAY omit a target it does not use. The required targets are declared in `decisionTargets`. ### 5.3 Observation before disposition The evaluator MUST distinguish: ```text Observation -> evidence relationship -> criterion interpretation -> criterion verdict -> severity -> operational action ``` It MUST NOT use a final pass/fail impression as the primary evidence for lower-level findings. ### 5.4 Quality vector rather than universal score The canonical representation is a vector of criterion states, not one universal scalar. A project MAY expose scores for selected axes, but it MUST preserve the underlying criterion assessments. A scalar MUST NOT allow an unrelated strength to compensate for a noncompensable critical failure. ## 6. Required semantic distinctions A conforming system MUST preserve the following distinctions. ### 6.1 Truth Whether a proposition is actually correct under the project's truth-verification policy. ### 6.2 Grounding Whether a proposition is supported by evidence the project permits the attempter to use. A statement can be true but ungrounded. A statement can accurately reflect a supplied source that is itself false. ### 6.3 Source quality Whether the evidence source has sufficient authority, freshness, provenance, or methodological quality for the criterion. Source quality is not the same as entailment. A high-quality source can be irrelevant to a claim; a weak source can directly state a claim without being acceptable evidence. ### 6.4 Inferential warrant Whether the evidence justifies the type and strength of the conclusion. Examples of warrant errors include: - association presented as causation; - a contributing factor presented as the primary cause; - a possibility presented as certainty; - a subgroup result generalized to the full population; - absence of evidence presented as evidence of absence; - a descriptive source used to support a normative conclusion without the project-supplied value premise. ### 6.5 Relevance Whether the statement contributes to satisfying the task and criterion. ### 6.6 Criterion alignment Whether the observed behavior satisfies the specific project requirement. A response can be factually correct but fail a required format, scope, or decision rule. ### 6.7 Positive proof Concrete evidence that a required or preferred behavior is present. A required-presence criterion MUST NOT pass solely because the evaluator failed to detect a defect. ### 6.8 Omission Expected, applicable content that is absent or materially incomplete. Omissions MUST be represented independently from false present claims. ### 6.9 Uncertainty Uncertainty about the evidence, applicability, criterion interpretation, or threshold. Uncertainty MUST NOT be silently converted into failure or success. ### 6.10 Annotation fidelity Whether the human reviewer's judgment faithfully represents the attempt under the written contract. Annotation fidelity includes: - label support; - rationale support; - criterion attribution; - evidence alignment; - severity alignment; - omitted positive findings; - omitted negative findings; - disagreement type. ## 7. Document lifecycle The normative lifecycle is: ```text Project instructions v Draft contract v Contract review and approval v Versioned case input v Observation and adjudication v Provisional target-specific decisions v Optional bounded human resolution v Final target-specific decisions v Feedback and audit record v Benchmarking, monitoring, and contract revision ``` A production evaluation MUST reference an approved contract by identifier, semantic version, and content hash. ### 7.1 Hash canonicalization A deployment MUST choose and version a canonical hashing procedure. The example package uses **canonical JSON v1** for whole-document hashes: - UTF-8 encoding; - object keys sorted lexicographically; - array order preserved; - insignificant whitespace removed; - non-finite numbers forbidden. Exact text artifacts are hashed as their exact UTF-8 content. Cross-language production systems SHOULD adopt a rigorously specified JSON canonicalization standard when numeric serialization differences are possible. The canonicalization version must be retained with the audit record. ## 8. Project contract The project contract is the authoritative, executable interpretation of project instructions. ### 8.1 Contract identity The contract MUST contain: - `contractId` - stable project rubric identity; - `contractVersion` - semantic version; - `schemaVersion` - document schema version; - `status` - draft, approved, deprecated, or retired; - exact instruction documents and content hashes; - owners or responsible roles; - effective date where applicable. A change to criterion meaning, applicability, anchors, source policy, severity, or decision effect MUST create a new contract version. ### 8.2 Instruction documents and precedence Every criterion MUST cite at least one source instruction location. Instruction documents have an authority class: - **primary** - normative written instructions; - **supplemental** - additional normative clarification; - **example** - illustrative unless explicitly promoted to normative status; - **historical** - retained for provenance but not current authority. A project MUST define precedence when instructions can conflict. Examples MUST NOT silently override written instructions. If an example and a primary instruction conflict, the contract owner MUST resolve the conflict or mark the criterion ambiguous. ### 8.3 Criterion anatomy Each criterion MUST contain: - stable `criterionId`; - human-readable name; - exact requirement; - intent; - criterion type; - scope; - subjectivity class; - instruction references; - applicability rule; - positive proof standard; - failure condition; - omission condition where applicable; - evidence policy; - positive and negative anchors where practical; - valid alternatives; - exceptions; - dependencies and precedence; - default severity; - default decision effect; - tags. ### 8.4 Criterion types #### Required presence Use when the attempt must contain a behavior, conclusion, component, or analysis. Pass requires affirmative evidence. ```text Valid pass: \"Attempt span A performs required behavior B.\" Invalid pass: \"No problem found.\" ``` #### Prohibited presence Use when a behavior must not appear. Pass may rely on absence only after the system performs a targeted search over the relevant scope and records sufficient inspection coverage. The system SHOULD record what scope was inspected. #### Conditional Use when a requirement applies only under a condition. Applicability MUST be adjudicated before the obligation. If applicability is uncertain and decision-changing, the criterion MUST become ambiguous or unknown and be eligible for escalation. #### Factual Use for truth or factual consistency. The project MUST specify whether evaluation is closed-world, open-world, or hybrid. #### Grounding Use for support by permitted evidence. The evaluator MUST link material claims to evidence relationships. #### Inferential warrant Use for whether the conclusion follows at the stated strength from the evidence and project-supplied premises. #### Comparative Use when one candidate, option, or conclusion is compared with another on a named dimension. Candidate order SHOULD be randomized or swapped during evaluation to measure position sensitivity. #### Completeness Use when required concepts, evidence, or components must be covered. The evaluator MUST construct or receive an expected-element map. #### Holistic quality Use for genuinely global qualities such as organization, synthesis, strategic judgment, or usefulness. A holistic criterion MUST still cite concrete findings and SHOULD use approved anchors. A holistic criterion MUST NOT override an explicit critical factual, grounding, safety, or compliance failure unless the project explicitly makes that compensation permissible. #### Mechanically verifiable Use when a deterministic procedure can decide the condition: schema conformance, counts, exact formats, code execution, arithmetic, duplicate detection, citation identifiers, and similar checks. Deterministic validators SHOULD be preferred over model judgment for the property they directly verify. #### Annotation fidelity Use to evaluate the human review separately from the attempt. ### 8.5 Scope Criterion scope determines the primary object of evaluation: - `attempt` - the response as a whole; - `claim` - one or more extracted propositions; - `citation` - evidence linkage and citation behavior; - `annotation` - human reviewer output; - `global` - emergent quality of the whole response; - `batch` - reviewer, project, or dataset patterns across cases. A criterion MAY produce findings at more granular spans than its primary scope. ### 8.6 Subjectivity class #### Objective A deterministic or strongly evidence-constrained judgment with little acceptable interpretive range. #### Bounded judgment A judgment that permits interpretation but has explicit anchors, thresholds, or accepted alternatives. #### Holistic expert A global judgment requiring tacit or domain expertise that cannot be fully reduced to local checks. Holistic-expert criteria SHOULD have stricter escalation and calibration requirements than objective criteria. ### 8.7 Applicability Each criterion has: - a natural-language `ruleText` readable by humans; - an optional structured `condition` executable by the system. The natural-language and structured forms MUST agree. A mismatch is a contract defect. Applicability states are: - `applies`; - `does_not_apply`; - `uncertain`. If a criterion does not apply, its verdict MUST be `not_applicable`, and it MUST NOT trigger a negative operational action. ### 8.8 Proof policy Each criterion defines: - whether positive proof is required; - the positive proof standard; - the failure condition; - the omission condition; - whether absence of a violation is sufficient; - whether partial credit is permitted; - whether compensation by other criteria is permitted. #### Required-behavior invariant When `positiveProofRequired` is true and the criterion verdict is `met`, the assessment MUST reference at least one confirmed positive finding. #### Failure invariant When the verdict is `not_met`, the assessment MUST reference at least one confirmed negative or omission finding unless failure is produced by a deterministic validator recorded in the finding source. #### Partial-credit invariant `partially_met` MUST identify both the satisfied component and the missing or defective component. It MUST NOT be used as a vague confidence hedge. ### 8.9 Evidence policy The top-level source policy defines the project evidence universe. #### World mode - **closed_world** - only supplied evidence may support material claims; - **open_world** - external evidence may be retrieved under source rules; - **hybrid** - supplied evidence is primary, with limited external verification. #### Parametric model knowledge Model memory SHOULD default to `never_evidence`. It MAY assist retrieval or hypothesis generation, but it MUST NOT be recorded as verified evidence unless the project explicitly allows and verifies it. #### Evidence classes Every evidence artifact belongs to a project-defined class with: - permission status; - authority rank; - description; - optional freshness and provenance rules. A criterion may further restrict permitted evidence classes and minimum authority. #### Evidence conflicts The contract MUST state how to handle materially conflicting sources. The evaluator MUST NOT silently choose the evidence that best supports its preferred verdict. Possible policies include: - report the conflict and lower confidence; - prefer a specified authority hierarchy; - require the attempt to acknowledge the conflict; - escalate when the conflict changes the action. #### Source snapshots Production evidence SHOULD be frozen by exact content hash. Open-world evaluations SHOULD record retrieval time and exact source content used so the decision remains replayable. ### 8.10 Anchors Anchors operationalize subjective or easily misunderstood criteria. A useful anchor contains: - example behavior; - expected verdict; - explanation of why; - boundary notes; - relation to valid alternatives. A criterion SHOULD include: - at least one clear positive anchor; - at least one clear negative anchor; - one boundary anchor for high-variance subjective axes; - explicit valid alternatives where multiple approaches can pass. Anchors MUST be treated according to their declared authority. An illustrative reference is not automatically the only correct answer. ### 8.11 Dependencies and precedence Dependencies express evaluation order or logical relation. Precedence resolves conflicts. Examples: - grounding assessment precedes warrant assessment; - a safety prohibition overrides a stylistic preference; - project-specific formatting overrides a generic default; - a later supplemental clarification overrides an older ambiguous example. Circular precedence MUST be rejected during contract validation. ### 8.12 Severity and decision effect Severity describes impact of the finding: - `none`; - `advisory`; - `minor`; - `major`; - `critical`. Decision effect describes the default operational consequence: - none; - accept with note; - rework; - reject; - escalate. Severity and action MUST remain separate. A major defect may be repairable and therefore trigger rework rather than rejection. A lower-severity integrity anomaly may still trigger escalation if it undermines confidence in the whole case. ### 8.13 Criterion authoring acceptance tests Before approval, each criterion SHOULD answer yes to the following: 1. Can two reviewers identify the same applicability condition? 2. Is there observable positive evidence for passing? 3. Is failure distinguishable from insufficient evidence? 4. Are truth, grounding, and warrant separated where needed? 5. Are valid alternatives represented? 6. Is the criterion traceable to a written instruction? 7. Is severity distinguishable from verdict? 8. Is compensation explicitly allowed or forbidden? 9. Can the system identify missing required content? 10. Does the criterion avoid encoding an unstated stylistic preference? 11. Does it specify the allowed evidence universe? 12. Is a domain expert required for difficult cases? 13. Are examples consistent with the written rule? 14. Can the decision be audited from recorded spans and evidence? A criterion that fails these tests SHOULD remain draft. ## 9. Case input A case input freezes the evaluation object. ### 9.1 Task The task contains: - stable task identifier; - task type; - exact prompt; - task-specific instructions; - expected output format; - content hash; - metadata used by applicability rules. Task metadata MUST NOT contain hidden evaluator instructions that are unavailable to attempters unless the project explicitly defines them as internal QA rules. ### 9.2 Evidence bundle Each evidence artifact contains: - evidence identifier; - evidence class; - title; - exact content; - content hash; - authority rank; - permission status; - capture time; - optional source URI, publication date, and metadata. The evaluator MUST reject or ignore evidence marked disallowed for the relevant criterion. ### 9.3 Attempt The attempt contains: - exact content; - content hash; - submission time; - optional attachments; - optional tool traces. If the project evaluates process as well as outcome, the contract MUST state which tool traces or intermediate artifacts are normative. ### 9.4 Human review The optional human review contains: - reviewer identifier or stable pseudonym; - label; - score where used; - reason codes; - rationale; - criterion annotations; - cited spans; - reviewer confidence; - submission time. The evaluator SHOULD assess the attempt before revealing the reviewer's final label to the attempt-quality adjudicator. This reduces anchoring. The annotation auditor may then compare the independent attempt assessment with the review. ### 9.5 Reference answers References declare a role: - authoritative; - illustrative; - positive anchor; - negative anchor. An illustrative reference MUST NOT be treated as exhaustive. A reference SHOULD be introduced after independent requirement and evidence analysis when reference anchoring is a material risk. ## 10. Observation plane ### 10.1 Integrity checks Before semantic evaluation, the system SHOULD perform: - schema validation; - hash verification; - required-field checks; - corruption detection; - spam and duplicate checks; - evaluator-instruction injection detection; - citation identifier validation; - prohibited-format checks; - tool-trace integrity checks where applicable. All candidate and source content MUST be treated as untrusted data. Text inside an attempt or source MUST NOT override the project contract or system instructions. ### 10.2 Claim map The evaluator extracts explicit and implied material propositions. Each claim records: - proposition; - type; - importance; - explicitness; - exact attempt spans; - qualifiers; - cited evidence. Compound statements SHOULD be split when components can receive different support or truth judgments. The evaluator SHOULD prioritize central and supporting claims over incidental prose. Incidental claims may still matter when they are dangerous, disallowed, or materially false. ### 10.3 Expected-element map The evaluator MUST represent what should have appeared, not only what did appear. Expected elements may come from: - explicit task requirements; - project criteria; - approved instance-level rubric generation; - domain-expert templates; - declared required evidence or output sections. Each expected element records: - associated criterion; - description; - required status; - importance; - acceptable alternatives; - observed spans; - coverage status. A generated expected element not directly supported by project instructions MUST be marked inferred and MUST NOT independently create a hard failure unless the contract permits such inference. ### 10.4 Evidence relationships Each relationship links a claim, expected element, or finding to a specific evidence artifact. Relationship states are: - directly entails; - reasonably supports; - weakly supports; - contradicts; - irrelevant; - insufficient. The evaluator SHOULD cite exact evidence spans. It MUST record source quality and temporal validity independently from the relationship. ### 10.5 Findings Findings are the reusable evidence-linked observations from which criterion assessments are built. Finding types are: - **positive** - affirmative evidence of quality or compliance; - **negative** - present defect or violation; - **omission** - missing required content; - **ambiguity** - unresolved applicability, evidence, or interpretation; - **counterevidence** - evidence challenging another finding; - **integrity** - corruption, spam, manipulation, or procedural anomaly. A material finding SHOULD include: - exact target span; - evidence identifiers and spans where applicable; - criterion attribution; - materiality; - severity; - confidence; - producing component; - status. ### 10.6 Independent observation channels A robust implementation SHOULD generate at least these channels independently before final criterion adjudication: 1. **Positive prover** - searches for concrete satisfaction evidence. 2. **Negative challenger** - searches for violations, contradictions, and unsupported claims. 3. **Coverage auditor** - searches for omissions and underdeveloped requirements. 4. **Warrant auditor** - tests whether conclusions follow at the stated strength. A separate challenger SHOULD attempt to refute decisive positive or negative findings, especially before autonomous rejection or acceptance. This separation is a recommended design pattern to test empirically; the benchmark protocol requires an ablation rather than assuming it always helps. ## 11. Criterion adjudication A criterion assessment combines findings without discarding them. ### 11.1 Applicability first The adjudicator MUST decide applicability before assigning a substantive verdict. - `does_not_apply` -> `not_applicable`; - `uncertain` and decision-changing -> `ambiguous` or `unknown`, normally eligible for escalation; - `applies` -> evaluate proof, failure, and omission evidence. ### 11.2 Positive and negative evidence together The adjudicator MUST consider positive findings, negative findings, omissions, counterevidence, and unresolved findings. It MUST NOT hide positive evidence merely because the overall criterion fails. An otherwise failing attempt may contain valuable strengths useful for feedback or partial credit. ### 11.3 Verdict meanings - **met** - the applicable criterion is affirmatively satisfied under its proof standard; - **partially_met** - a decomposable portion is satisfied, but a material portion is missing or defective; - **not_met** - confirmed evidence shows failure under the criterion; - **ambiguous** - more than one interpretation is defensible under the current contract; - **unknown** - evidence or expertise is insufficient to decide; - **not_applicable** - the criterion does not apply. `ambiguous` is about competing defensible interpretations. `unknown` is about missing information or capability. ### 11.4 Grounding, truth, warrant, and relevance substatuses The assessment stores these statuses separately. An evaluator MUST NOT infer one from another without evidence. For example: ```text Grounding: grounded Truth: unknown Warrant: overstated Relevance: relevant ``` This could describe an answer that faithfully reports an uncertain source but draws a conclusion stronger than that source warrants. ### 11.5 Global assessment Global qualities MAY be assessed after local findings are available. The global evaluator SHOULD receive: - the whole attempt; - criterion findings; - expected-element coverage; - contradiction map; - project anchors. It SHOULD NOT independently override confirmed local findings. ## 12. Annotation audit The annotation audit is performed after, or independently from, the attempt assessment. ### 12.1 Label support Does the attempt assessment support the human reviewer's final label? Possible states: - supported; - partially supported; - unsupported; - reasonable disagreement; - rubric ambiguous; - not applicable. ### 12.2 Rationale support Does the rationale logically support the label, and is the rationale itself correct? A correct label with a wrong rationale MUST be represented as such. ### 12.3 Criterion attribution Did the reviewer identify the correct criterion, or apply: - the wrong criterion; - only part of the correct criterion; - an unstated preference? ### 12.4 Evidence alignment Did the cited span and evidence support the reviewer's stated reason? ### 12.5 Severity alignment Did the reviewer assign a consequence proportionate to the defect? ### 12.6 Omitted findings The audit records important positive and negative findings the reviewer missed. ### 12.7 Disagreement taxonomy The system SHOULD classify disagreement as one or more of: - observation; - evidence authority; - applicability; - criterion interpretation; - inference; - threshold; - severity; - attention; - expertise; - reasonable disagreement; - rubric defect. This taxonomy is diagnostic. It SHOULD drive rubric fixes, reviewer calibration, routing, and monitoring. A model-human disagreement MUST NOT automatically count as reviewer error. ## 13. Confidence and calibration ### 13.1 Raw confidence Raw confidence is the component's internal score or estimate. It is not operationally trustworthy by default. ### 13.2 Calibrated confidence Calibrated confidence is empirically mapped to observed correctness for a defined event, criterion family, model configuration, project, and slice. A calibration profile SHOULD identify: - event being predicted; - model and prompt configuration; - contract version; - benchmark version; - criterion or slice; - method; - sample size; - date; - calibration quality. ### 13.3 Confidence bands - **high** - suitable for the configured autonomous action under the project's validated threshold; - **medium** - useful decision support but not sufficient for the configured high-risk autonomous action; - **low** - substantial uncertainty; - **uncalibrated** - no empirical mapping exists. The words high, medium, and low have no universal numeric meaning. They are assigned by the project's calibration policy. ### 13.4 Autonomy invariant An action listed in `requiresCalibrationForActions` MUST NOT be taken automatically when confidence is uncalibrated. The evaluator must use `uncalibratedAction`, normally escalation. ### 13.5 Selective evaluation Projects SHOULD optimize risk as a function of coverage rather than forcing a verdict on every case. ```text Coverage = fraction decided automatically Risk = error rate or cost-weighted loss on automatically decided cases ``` The benchmark protocol defines required reporting. ## 14. Decision policy The decision policy maps criterion state to actions for each declared target. ### 14.1 Rule requirements Each rule contains: - target; - identifier; - priority; - human-readable description; - structured match conditions; - action; - terminal flag; - optional explanation template. Rules MUST be evaluated deterministically from recorded assessment state. Every final decision MUST contain a policy trace showing which rules matched. ### 14.2 Target isolation Attempt-quality criteria SHOULD determine the attempt decision. Annotation-fidelity criteria SHOULD determine the annotation decision. The composite-case policy MAY combine both. An incorrect human review MUST NOT cause a good attempt to be marked bad. It may cause the composite annotated datum to require rework. ### 14.3 Hard gates A hard gate MUST identify: - specific criterion or tag; - triggering verdict; - required severity; - required calibrated confidence; - operational action; - whether compensation is forbidden. ### 14.4 Default action The default SHOULD be conservative. For an uncalibrated or incomplete new project, the default action SHOULD be `escalate` rather than `accept` or `reject`. See `DECISION_POLICY.md` for a reference algorithm. ## 15. One-interaction human resolution ### 15.1 Purpose The human interaction resolves the smallest decisive uncertainty. It is not a duplicate full review. ### 15.2 Trigger The system MUST ask a question only when: 1. the unresolved issue can change an operational decision; 2. a qualified human can answer it from the displayed context; 3. the question is bounded; 4. the project interaction limit has not been exhausted. ### 15.3 Question content The request MUST include: - triggering criterion; - exact relevant attempt or review span; - decisive evidence; - unresolved relationship; - bounded options; - an ambiguity option when required; - required expertise tags; - statement of whether the answer is decision-changing. ### 15.4 Bias control The provisional overall disposition SHOULD be hidden until the responder answers. The system MAY show criterion-level observations necessary to answer the question. ### 15.5 Resolution effects The human answer SHOULD update only the affected criterion or policy premise. The system then reruns the decision policy. Repeated questions with the same root cause SHOULD generate a proposed contract clarification instead of consuming indefinite human review. ### 15.6 No useful bounded question When the issue cannot be resolved by one bounded interaction, the system MUST escalate to the appropriate full review or project owner. It MUST NOT manufacture a low-value question merely to satisfy the one-interaction design. ## 16. Feedback Feedback is a separate generated artifact and MUST be validated independently from the verdict when it recommends a repair. ### 16.1 Audiences - **attempter** - improve the attempt; - **reviewer** - improve criterion application and evidence use; - **project owner** - identify rubric ambiguity, drift, or repeated failure patterns. ### 16.2 Required properties Material feedback SHOULD: - identify the criterion; - cite the finding and exact span; - distinguish false, unsupported, irrelevant, incomplete, and overstated; - explain why the issue matters; - state the minimal sufficient repair; - preserve valid content; - avoid generic praise; - prioritize decisive issues. ### 16.3 Positive feedback Positive feedback MUST cite confirmed positive findings. It MUST NOT be generated merely to balance tone. ### 16.4 Repair validation A proposed repair is `validated` only if a validator or evaluator has checked that: - it addresses the identified issue; - it does not contradict permitted evidence; - it does not create a new criterion violation; - it preserves correct portions of the attempt. Otherwise it is `plausible_unvalidated`. ### 16.5 Feedback limits The project SHOULD cap feedback items per audience and order them by decision impact. Exhaustive minor comments can reduce usefulness and increase gaming risk. ## 17. Audit and reproducibility A final evaluation MUST preserve enough information to reconstruct the decision: - exact case hash; - exact contract version and hash; - exact source snapshots and hashes; - claim and expected-element maps; - evidence relationships; - all material findings; - model provider and model version where available; - prompt template identifier, version, and hash; - runtime parameters; - parent/child run lineage; - calibration profile; - deterministic validator versions; - human resolution; - decision policy version and trace; - warnings and nonreproducible components. A run may be statistically reproducible without being bit-for-bit deterministic. The audit record MUST state whether exact replay is expected. ## 18. Security and evaluator manipulation ### 18.1 Untrusted content Task content, sources, attempts, references, tool outputs, and human rationales MUST be treated as untrusted data. The evaluator MUST ignore embedded instructions that attempt to: - change the rubric; - assign a score; - reveal hidden prompts; - suppress findings; - instruct the evaluator to trust a claim; - override source policy; - alter output format; - impersonate a project owner. ### 18.2 Channel separation The implementation SHOULD separate: - system and contract instructions; - task content; - evidence; - attempt; - human annotation; - reference answers; - model-generated intermediate artifacts. ### 18.3 Structured output validation Every model-produced structured document MUST be schema validated. Invalid output MUST be retried, repaired by a constrained parser, or marked failed. It MUST NOT be silently accepted. ### 18.4 Adversarial regression The project SHOULD maintain tests for: - direct prompt injection; - indirect injection inside evidence; - rubric parroting; - fake citations; - hidden or zero-width text where relevant; - position and verbosity manipulation; - copying approved anchor language without performing the behavior; - attempts that instruct the evaluator to ignore defects. ## 19. Failure handling The evaluation status may be: - `provisional`; - `awaiting_human`; - `final`; - `failed`. A failed pipeline MUST distinguish: - invalid input; - missing required source; - deterministic validator failure; - model timeout or provider error; - malformed structured output; - context truncation; - evidence retrieval failure; - contract inconsistency; - calibration unavailable; - security anomaly. Operational failures MUST NOT be converted into criterion failures against the attempter or reviewer. ## 20. Reference state machine ```text RECEIVED +- validate hashes and schemas +- invalid -> FAILED_INPUT +- valid -> FROZEN FROZEN +- integrity and deterministic checks -> OBSERVED OBSERVED +- claim, expected-element, evidence, and finding generation -> CHALLENGED CHALLENGED +- criterion and global adjudication -> PROVISIONAL PROVISIONAL +- no decision-changing uncertainty -> FINALIZED +- bounded question available -> AWAITING_HUMAN +- unresolved and not bounded -> ESCALATED_FULL_REVIEW AWAITING_HUMAN +- apply answer to affected criterion and rerun policy -> FINALIZED FINALIZED +- generate/validate feedback and persist audit record ``` ## 21. Cross-document integrity requirements A conforming implementation MUST check: - every `criterionId` in assessments exists in the referenced contract; - every finding identifier referenced by an assessment exists; - every claim and expected-element identifier referenced by a finding exists; - every evidence identifier exists and is permitted for the relevant criterion; - every quoted span belongs to the declared artifact; - character offsets, when present, match the quoted text; - every decision target is declared in the contract; - every triggered rule exists and targets the same decision target; - every calibration profile is compatible with the model, prompt, contract, and criterion slice; - no duplicate identifiers exist within a document; - precedence and dependency graphs are acyclic unless a documented fixed-point policy exists; - final decisions occur after any recorded human response; - composite-case decisions are consistent with the project's composite policy. The JSON Schemas cannot enforce all referential integrity rules; the application MUST implement them. ## 22. Conformance profiles ### Profile A: Structured evaluator MUST: - validate schema; - use an approved contract; - create claims and expected elements; - record evidence-linked findings; - produce criterion assessments; - produce target-specific decisions and traces; - preserve audit metadata. ### Profile B: Calibrated selective evaluator Profile A plus: - empirical calibration; - risk-coverage measurement; - abstention/escalation; - locked calibration and test sets; - position/prompt perturbation tests; - challenge-set evaluation; - action-specific launch gates. ### Profile C: Governed production evaluator Profile B plus: - independent annotation audit; - bounded human resolution; - feedback validation; - shadow deployment; - drift monitoring; - replay and rollback; - appeals; - governance controls on reviewer analytics. ## 23. Minimum viable implementation The smallest useful implementation should support: 1. text-only cases; 2. closed-world evidence; 3. approved project contract; 4. deterministic format checks; 5. claim extraction; 6. expected-element mapping; 7. claim-to-evidence relationships; 8. positive, negative, omission, and warrant findings; 9. criterion adjudication; 10. attempt and annotation decisions; 11. conservative escalation; 12. evidence-linked feedback; 13. full version and hash logging. It should not initially attempt to automate every subjective decision. ## 24. Contract approval checklist A contract is ready for benchmark construction when: - all primary instructions are included and hashed; - precedence is explicit; - every criterion is atomic enough to audit; - global qualities remain represented where necessary; - positive proof standards exist; - omissions are detectable; - valid alternatives are recorded; - evidence permissions are explicit; - deterministic checks are identified; - applicability is testable; - severity and action are separated; - hard gates are explicit; - decision targets are declared; - human question types and limits are set; - feedback audiences and repair policy are set; - security and replay policies are approved; - project owner and domain expert have signed off; - no unresolved contradiction remains between instructions and examples. ## 25. Evaluation output acceptance checklist An evaluation output is complete when: - input and contract references resolve; - integrity status is recorded; - all material claims are represented; - all required expected elements are represented; - evidence relationships distinguish entailment and source quality; - every met required criterion has positive proof; - every failed criterion has confirmed negative or omission evidence; - counterevidence was considered for decisive findings; - uncertainty is explicit; - annotation quality is assessed separately; - each required target has a decision; - every decision has a policy trace; - any human interaction is bounded and decision-changing; - feedback is linked to findings; - repair validation status is honest; - the audit record is replayable or states why not. "}, {"slug": "foundation-decision-policy-md", "title": "Decision Policy Reference", "category": "Foundation specifications", "summary": "A target-specific rule engine for accept, rework, reject, and escalate decisions, including the bounded human interaction.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-decision-policy-md.html", "raw": "documents/raw/foundation/DECISION_POLICY.md", "text": "# AutoQA Decision Policy Reference **Reference version:** 0.1.0 ## 1. Purpose The decision policy converts evidence-backed criterion assessments into operational actions. It is deliberately separate from semantic evaluation so that: - project owners can change operational thresholds without rerunning all semantic analysis; - evidence and criterion verdicts remain inspectable; - severity does not silently become action; - attempt, annotation, and composite-case decisions remain independent; - every action has a deterministic policy trace. ## 2. Action semantics ### Accept The target is ready for its intended use. All hard requirements are affirmatively satisfied at the required calibrated confidence, and no unresolved major issue remains. ### Accept with notes The target is usable, but one or more advisory or minor findings should be retained. Notes must not hide a major unresolved defect. ### Rework The target is not ready, but the defect is specific and repairable. Feedback should identify the minimal sufficient correction. ### Reject The target has a disqualifying defect under an explicit project rule. Rejection should require strong evidence and calibrated confidence unless the failure is fully deterministic. ### Escalate The system should not decide autonomously. Reasons include: - uncertain applicability; - ambiguous instruction interpretation; - insufficient evidence; - required domain expertise; - conflicting high-quality evidence; - uncalibrated semantic judgment; - unsupported input type or context range; - security or integrity anomaly; - disagreement near a high-cost threshold. ## 3. Target semantics ### Attempt target Uses attempt-quality criteria. An annotation error must not lower the attempt verdict. ### Annotation target Uses annotation-fidelity criteria. A good attempt can have a bad annotation, and vice versa. ### Composite-case target Represents whether the attempt-plus-annotation record is ready as annotated data. Its policy may require both targets to be acceptable. Typical mapping: | Attempt | Annotation | Composite | |---|---|---| | accept | accept | accept | | accept | rework | rework | | rework | accept | rework | | reject | any | reject or rework according to retention policy | | escalate | any | escalate | | any | escalate | escalate | The exact mapping belongs in the project contract. ## 4. Inputs The policy engine consumes only recorded state: - criterion assessments; - criterion metadata and tags; - confirmed findings; - unresolved findings; - calibrated confidence; - annotation disagreement types; - deterministic validator outcomes; - human-resolution response where present; - policy rules and priorities. It must not independently reinterpret the raw attempt. ## 5. Rule match semantics Each rule targets one of: ```text attempt annotation composite_case batch ``` A rule may select assessments by: - criterion identifiers; - criterion tags; - verdicts; - severities; - default decision effects; - minimum calibrated confidence; - presence of unresolved findings; - annotation disagreement type. ### 5.1 Multiple selectors Selectors within one rule use logical AND unless the contract explicitly defines otherwise. Example: ```yaml criterionTags: [hard-quality] verdicts: [not_met] severities: [critical] minimumCalibratedConfidence: 0.95 ``` This matches a criterion only when all four conditions are satisfied. ### 5.2 Multiple criteria A project must declare whether a rule over multiple criteria means: - **any matching criterion**; - **all listed criteria**; - **at least N matching criteria**. Version 0.1.0 recommends the following default: - negative and escalation rules use **any**; - clean-accept rules use **all**. The human-readable rule description must make this explicit until a later schema version adds a formal quantifier. ### 5.3 Rule priority Rules are evaluated from lowest numeric priority to highest. A terminal matched rule ends evaluation for that target. Nonterminal matches remain in the policy trace and may be superseded by a later terminal rule according to project logic. ### 5.4 Tie-break order If multiple nonterminal actions remain, use the contract's `tieBreakOrder`. A conservative default is: ```text reject > escalate > rework > accept_with_notes > accept ``` This is not universally appropriate. For repairable work, a project may prefer escalation or rework over rejection. ## 6. Required preconditions Before applying semantic action rules, the policy engine must check: 1. input and contract hashes resolve; 2. required assessment targets exist; 3. no cross-document referential error exists; 4. decisive findings are confirmed; 5. the calibration profile is compatible; 6. required deterministic checks completed; 7. any human response is valid for its request; 8. no unresolved security anomaly invalidates the run. A failed precondition normally produces escalation or a system failure, not a negative judgment against the attempter. ## 7. Reference algorithm ```typescript function decideTarget( target: DecisionTarget, contract: ProjectContract, evaluation: EvaluationState, ): Decision { const applicableRules = contract.decisionPolicy.rules .filter((rule) => rule.target === target) .sort((a, b) => a.priority - b.priority); const trace: PolicyTraceEntry[] = []; const matchedActions: OperationalAction[] = []; const triggeredRuleIds: string[] = []; assertReferentialIntegrity(contract, evaluation); assertCalibrationCompatibility(contract, evaluation, target); for (const rule of applicableRules) { const match = evaluateRuleMatch(rule.match, contract, evaluation, target); trace.push({ ruleId: rule.ruleId, matched: match.matched, inputs: match.traceInputs, result: match.explanation, }); if (!match.matched) continue; matchedActions.push(rule.action); triggeredRuleIds.push(rule.ruleId); if (rule.terminal) { return buildDecision(target, rule.action, triggeredRuleIds, trace, evaluation); } } const action = matchedActions.length ? resolveByTieBreak(matchedActions, contract.decisionPolicy.tieBreakOrder) : contract.decisionPolicy.defaultAction; return buildDecision(target, action, triggeredRuleIds, trace, evaluation); } ``` The production implementation must also enforce the safeguards below. ## 8. Safeguards ### 8.1 Positive-proof safeguard An attempt or annotation cannot be automatically accepted merely because no failure rule matched. For every required-presence, completeness, grounding, or positive-quality criterion needed for acceptance: - applicability must be `applies`; - verdict must be `met`; - at least one confirmed positive finding must be referenced; - required expected elements must be complete; - calibrated confidence must meet the action gate. ### 8.2 Unresolved-finding safeguard A decision-changing unresolved finding blocks automatic acceptance or rejection unless the policy explicitly routes it to a qualified human resolution. ### 8.3 Calibration safeguard If the chosen action is listed in `requiresCalibrationForActions`, every decisive semantic assessment must have a compatible calibrated score or band. Otherwise use `uncalibratedAction`. A deterministic validator may be exempt when its correctness has been separately verified and the contract explicitly permits autonomous action from it. ### 8.4 Counterevidence safeguard Before autonomous reject or accept, decisive findings should have undergone a counterevidence challenge. A confirmed counterfinding that materially weakens the decision should lower confidence or trigger escalation. ### 8.5 Noncompensation safeguard A criterion with `allowCompensation = false` cannot be offset by strengths on other criteria. ### 8.6 Annotation isolation safeguard Annotation-fidelity failures cannot change the attempt target unless the project defines the annotation as part of the attempt itself. They may change the annotation and composite targets. ### 8.7 Ambiguity safeguard A criterion marked `ambiguous` must not be treated as `not_met`. A criterion marked `unknown` must not be treated as `met`. ### 8.8 System-error safeguard Provider failures, truncation, malformed output, unsupported media, or retrieval failure must never be counted as attempter defects. ## 9. Human-resolution policy ### 9.1 Candidate unresolved issues The engine identifies unresolved findings that are: - decision-changing; - answerable by one qualified human; - expressible as a bounded choice; - within the project's allowed question types. ### 9.2 Select one question When multiple issues are eligible, prioritize by expected reduction in decision loss: ```text expected value = loss before answer - expected loss after answer - interaction cost ``` Practical ranking factors: - severity of affected criterion; - probability the answer changes the target action; - confidence that the human can resolve it; - human effort; - recurrence value for contract improvement. ### 9.3 Apply response A response must map to an explicit update such as: - applicability becomes applies or not applicable; - one instruction gains precedence; - one evidence class becomes authoritative; - a specified inference is permitted or forbidden; - severity becomes major or minor; - an alternative is accepted; - the contract is declared ambiguous. The policy engine reruns only after the update is recorded in the evaluation state and audit trail. ### 9.4 Project ambiguity response When the responder chooses `project instruction is ambiguous`: - final target action should normally be escalate; - the case should not count as reviewer error; - a contract issue should be created; - recurring affected cases should be grouped for re-adjudication after clarification. ## 10. Example policy ### 10.1 Attempt rules ```yaml - target: attempt ruleId: reject-critical-grounding priority: 1 match: criterionTags: [hard-quality] verdicts: [not_met] severities: [critical] minimumCalibratedConfidence: 0.95 action: reject terminal: true - target: attempt ruleId: rework-major priority: 2 match: verdicts: [not_met] severities: [major] minimumCalibratedConfidence: 0.85 action: rework terminal: false - target: attempt ruleId: escalate-unresolved priority: 3 match: hasUnresolvedFindings: true action: escalate terminal: true - target: attempt ruleId: accept-clean priority: 10 match: criterionTags: [required-for-attempt-acceptance] verdicts: [met] minimumCalibratedConfidence: 0.90 hasUnresolvedFindings: false action: accept terminal: true ``` ### 10.2 Annotation rules ```yaml - target: annotation ruleId: rework-unfaithful-review priority: 1 match: criterionTags: [annotation-fidelity] verdicts: [not_met] severities: [major, critical] minimumCalibratedConfidence: 0.90 action: rework terminal: true - target: annotation ruleId: accept-faithful-review priority: 10 match: criterionTags: [annotation-fidelity] verdicts: [met] minimumCalibratedConfidence: 0.90 hasUnresolvedFindings: false action: accept terminal: true ``` ### 10.3 Composite rules ```yaml - target: composite_case ruleId: escalate-any-target-uncertain priority: 1 match: hasUnresolvedFindings: true action: escalate terminal: true - target: composite_case ruleId: rework-bad-annotation priority: 2 match: criterionIds: [annotation-fidelity] verdicts: [not_met] severities: [major, critical] action: rework terminal: true - target: composite_case ruleId: accept-both-clean priority: 10 match: criterionTags: [required-for-composite-acceptance] verdicts: [met] minimumCalibratedConfidence: 0.90 hasUnresolvedFindings: false action: accept terminal: true ``` ## 11. Policy trace requirements Every decision records: - target; - chosen action; - policy version; - triggered rules; - decisive criteria; - unresolved findings; - confidence; - full rule trace; - human-readable rationale; - decision time. A useful policy trace must allow an auditor to answer: - Which rule was evaluated? - What criterion state was supplied? - Did it match? - Why or why not? - Which rule determined the action? - What would have changed the action? ## 12. Policy testing The decision engine should have deterministic unit tests for: - each rule independently; - tie-break behavior; - no-rule default; - uncalibrated confidence; - not-applicable criteria; - ambiguous and unknown criteria; - positive-proof failure; - missing expected elements; - counterevidence; - noncompensable failures; - annotation isolation; - composite mapping; - human response updates; - invalid rule identifiers; - contract version mismatch; - calibration profile mismatch; - system error versus attempter failure. Golden policy tests should be run whenever the contract, policy engine, or schema changes. ## 13. Policy-version changes Create a new policy version when changing: - action thresholds; - rule priority; - target mapping; - hard gates; - tie-break order; - confidence requirements; - default action; - human escalation conditions; - composite-case admission policy. A policy-only change does not necessarily require rerunning claim and evidence extraction. It does require replaying the stored criterion state through the new policy and validating the operational effect on a locked set. ## 14. Initial default For a new project without sufficient calibration: ```text Deterministic clean pass/failure: project-defined automatic action Semantic clear finding: reviewer assistance Semantic uncalibrated decision: escalate Subjective disagreement: escalate or advisory Reviewer performance consequence: adjudicated samples only ``` The default action should remain `escalate` until benchmark gates justify selective autonomy. "}, {"slug": "foundation-manifest-json", "title": "Foundation Package - Manifest", "category": "Foundation specifications", "summary": "Machine-readable inventory and hashes for the complete package.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-manifest-json.html", "raw": "documents/raw/foundation/MANIFEST.json", "text": "{ \"package\": \"autoqa-foundation\", \"version\": \"0.1.0\", \"createdAt\": \"2026-07-15T00:00:00Z\", \"hashAlgorithm\": \"sha256\", \"files\": [ { \"path\": \"BENCHMARK_PROTOCOL.md\", \"bytes\": 31399, \"sha256\": \"81ce1c84096d03351356626c45b1e1818dbc4ee566a53807c934aa4ccddb9a2c\" }, { \"path\": \"CHANGELOG.md\", \"bytes\": 828, \"sha256\": \"69741186a45bd2f35bb6bd727d72906dd7e6779566339a69ac01c012f8c23473\" }, { \"path\": \"CONTRACT_SPEC.md\", \"bytes\": 41201, \"sha256\": \"8bec40d6df3f9339a2fe98d533b1f87ce6463b1b516161dd9e20dd72f5c41796\" }, { \"path\": \"DECISION_POLICY.md\", \"bytes\": 14038, \"sha256\": \"e96a072e0871118fd0f3119823216af09c97d657c51a14b0fbbd0f8f378bccd7\" }, { \"path\": \"README.md\", \"bytes\": 8130, \"sha256\": \"4486f3fb62cfefb0f3ac46764dd7db435d1943c57305c45b11eb6ed54cf8bb5d\" }, { \"path\": \"VERSION\", \"bytes\": 6, \"sha256\": \"e9dd8507f4bf0c6f42458e41aea833ad0bd3f6127272335eee9bf4d58541ed67\" }, { \"path\": \"examples/benchmark-item.example.json\", \"bytes\": 6191, \"sha256\": \"b90b5a73e3d266e2326607ae67731133da2b9bb95874d68daeabdcefea596b19\" }, { \"path\": \"examples/case-input.example.json\", \"bytes\": 4247, \"sha256\": \"e9fd3a97e157ef98c18cd2bfbd688d0d5d94b1b4f08a8441d23f950639dad37a\" }, { \"path\": \"examples/evaluation-output.example.json\", \"bytes\": 33545, \"sha256\": \"d5115d1630249d2df204c192055df29de3963442cd6271faaf7e9607e99854e4\" }, { \"path\": \"examples/human-resolution-packet.example.json\", \"bytes\": 2561, \"sha256\": \"61008558a6c2213fe6d1b254a7308c4bcfa673fbb448bbaf21543acb9817384e\" }, { \"path\": \"examples/project-contract.example.json\", \"bytes\": 22557, \"sha256\": \"45167a91393b7346a287be4fd414cdc7a20cf776986980175c6da00f16c8d6e5\" }, { \"path\": \"schemas/benchmark-item.schema.json\", \"bytes\": 7880, \"sha256\": \"0aeb486d5ca588dbb55a874e3549d2dfef73d07f71022830ac97c44ef6761721\" }, { \"path\": \"schemas/case-input.schema.json\", \"bytes\": 8266, \"sha256\": \"b22e7ab67dd0ae0db9cdd0efdc1d41ff118c823e9079a2d36f8b1bb02947f135\" }, { \"path\": \"schemas/common.schema.json\", \"bytes\": 32743, \"sha256\": \"3ba733b46b7d6053f7d6ab169756b11af13df88827aab1753e2dd9bee0c7f02c\" }, { \"path\": \"schemas/evaluation-output.schema.json\", \"bytes\": 4104, \"sha256\": \"f83580c20c7c99a4927a114c3972e7bc662bd22bfb5983047254bf5fda19b471\" }, { \"path\": \"schemas/human-resolution-packet.schema.json\", \"bytes\": 955, \"sha256\": \"8f460328039bde863bcbed1c15cfa9f83925f2b89375375864f849be35d8496e\" }, { \"path\": \"schemas/project-contract.schema.json\", \"bytes\": 23301, \"sha256\": \"d926d9c48a6cd85eacc8e5eb2f6fc921873a5c3faffdcce946e0f3a8b0878516\" }, { \"path\": \"templates/criterion-authoring-template.csv\", \"bytes\": 552, \"sha256\": \"273d0a921339787224982d81dce13c6348a6a30694167176e09ae078064f11e9\" }, { \"path\": \"templates/gold-set-template.csv\", \"bytes\": 593, \"sha256\": \"fe10577fe0dcb71ae5aad388468c71b08bdddf1fe9bdb57f9ab0df111eb1df8f\" }, { \"path\": \"templates/model-bakeoff-template.csv\", \"bytes\": 582, \"sha256\": \"271c3846fbe93c0d2b2d96eac3d64653f96cb787d160508076c247113218dedf\" }, { \"path\": \"types/autoqa.ts\", \"bytes\": 21216, \"sha256\": \"e714bd9ceb52df39729bdf55af1c5a095962015bafb7dc6c4858a52d6bd91049\" }, { \"path\": \"types/tsconfig.json\", \"bytes\": 207, \"sha256\": \"7bcf3373a17cf19ade5c59975ed0ec60ed9081b6258574daf0165b5dcfc3fc73\" }, { \"path\": \"validation/hash_utils.py\", \"bytes\": 1074, \"sha256\": \"801243a09e128c66163ac8edd0fbb26ba361b66518404eb5147d54b41dcc30d7\" }, { \"path\": \"validation/validate_examples.py\", \"bytes\": 2176, \"sha256\": \"a25475434cf0f528a17cff088712c3c4c2097ce10d7b9d2bf77d04c5fc5724f5\" }, { \"path\": \"validation/validate_package.py\", \"bytes\": 20165, \"sha256\": \"0e376b3588215043540dec575ce8bac4d71e1b2ebabe17b28c2869b352035e9e\" } ] } "}, {"slug": "foundation-readme-md", "title": "Foundation Package - README", "category": "Foundation specifications", "summary": "Package overview, core model, invariants, adoption sequence, and conformance levels.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-readme-md.html", "raw": "documents/raw/foundation/README.md", "text": "# AutoQA Foundation 0.1.0 A project-agnostic contract and benchmark package for evaluating human-produced AI-training data and the human annotations applied to it. The package defines the stable domain boundary for an AutoQA system. It intentionally does **not** prescribe a model provider, orchestration framework, database, or Vercel implementation. Those components can change while the evaluation objects, proof obligations, decision policy, and benchmark remain versioned and auditable. ## Core model AutoQA evaluates two different objects: 1. **The attempt** - whether the human-produced work satisfies the project's instructions. 2. **The annotation** - whether the human reviewer's label, rationale, cited evidence, criterion attribution, and severity faithfully describe that attempt. It does not reduce evaluation to a single score. The core unit is a criterion assertion: ```text Criterion C applies. Attempt span A exhibits behavior B. Evidence E supports, contradicts, or is insufficient for B. B does or does not satisfy C. The project decision policy assigns consequence S. ``` ## Non-negotiable invariants A conforming implementation preserves these distinctions: - truth versus support by the permitted evidence; - source quality versus entailment; - relevance versus correctness; - factual accuracy versus inferential warrant; - positive proof versus absence of a detected defect; - attempt quality versus annotation fidelity; - criterion verdict versus severity; - severity versus operational action; - raw model confidence versus empirically calibrated confidence; - item validity versus marginal dataset value. The engine may be project-agnostic. Its judgments may not be criterion-agnostic. Every production project must supply an approved, versioned evaluation contract. ## Package contents ```text . +-- README.md +-- CONTRACT_SPEC.md +-- BENCHMARK_PROTOCOL.md +-- DECISION_POLICY.md +-- CHANGELOG.md +-- VERSION +-- schemas/ | +-- common.schema.json | +-- project-contract.schema.json | +-- case-input.schema.json | +-- evaluation-output.schema.json | +-- benchmark-item.schema.json | +-- human-resolution-packet.schema.json +-- types/ | +-- autoqa.ts | +-- tsconfig.json +-- examples/ | +-- project-contract.example.json | +-- case-input.example.json | +-- evaluation-output.example.json | +-- benchmark-item.example.json | +-- human-resolution-packet.example.json +-- templates/ | +-- criterion-authoring-template.csv | +-- gold-set-template.csv | +-- model-bakeoff-template.csv +-- validation/ +-- hash_utils.py +-- validate_examples.py +-- validate_package.py ``` ### Primary documents - **`CONTRACT_SPEC.md`** defines the normative semantics, lifecycle, invariants, and conformance requirements. - **`BENCHMARK_PROTOCOL.md`** defines how to build adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift. - **`DECISION_POLICY.md`** defines the recommended rule engine, action semantics, human-resolution trigger, and policy trace. ### Machine-readable layer - **Project contract**: the approved interpretation of project instructions. - **Case input**: the task, evidence, attempt, optional reference, and optional human review. - **Evaluation output**: claims, expected elements, evidence relationships, findings, criterion assessments, annotation audit, decision trace, and feedback. - **Benchmark item**: adjudicated criterion truth, target-specific acceptable actions, ambiguity status, challenge attributes, and leakage controls. - **Human-resolution packet**: the one bounded, decision-changing question and its evidence context, with an optional response. Runtime input and output validation should use the JSON Schemas. TypeScript interfaces are provided for application development but are not a runtime trust boundary. ## Quick start Validate the included examples: ```bash python validation/validate_package.py ``` Type-check the TypeScript domain model: ```bash cd types tsc -p tsconfig.json ``` Recommended adoption sequence: 1. Copy `examples/project-contract.example.json` and replace the synthetic instructions and criteria. 2. Author each criterion in `templates/criterion-authoring-template.csv` before converting it to JSON. 3. Run a contract review with the project owner, a domain expert, and a QA owner. 4. Create representative `case_input` documents from historical production work. 5. Independently adjudicate a calibration and test set using `BENCHMARK_PROTOCOL.md`. 6. Compare evaluator configurations by criterion family rather than choosing one universal judge. 7. Calibrate confidence and set project-specific action gates. 8. Shadow the system before allowing autonomous acceptance or rejection. 9. Preserve every model, prompt, source, rubric, and policy version required for replay. ## Minimal evaluator pipeline ```text 1. Validate and freeze inputs 2. Compile/load the approved project contract 3. Run deterministic integrity and compliance checks 4. Extract claims and expected elements 5. Map claims and requirements to evidence 6. Generate positive, negative, omission, and warrant observations 7. Challenge material observations 8. Adjudicate each criterion 9. Evaluate global qualities 10. Audit the human annotation independently 11. Apply the versioned decision policy 12. Ask at most one decision-changing human question when necessary 13. Generate and validate audience-specific feedback 14. Persist the complete audit record ``` ## Conformance levels ### Level A - structured evaluation The implementation validates inputs and outputs, uses an approved project contract, produces criterion-level assessments, and records evidence-linked findings. ### Level B - calibrated selective evaluation Level A plus empirical confidence calibration, risk-coverage reporting, explicit abstention/escalation, and locked test suites. ### Level C - governed production AutoQA Level B plus independent annotation auditing, one-interaction human resolution, shadow deployment, drift monitoring, replay, appeals, and restricted use of reviewer analytics. Autonomous acceptance or rejection should not be enabled below Level B. ## Example scenario The included example contains a high-quality attempt that correctly concludes that the evidence does not establish a primary causal relationship. A human reviewer fails it for not choosing a forced yes/no answer. The project contract explicitly permits an inconclusive conclusion. The example evaluation therefore: - affirms the attempt's directness, grounding, warrant, and completeness; - accepts the attempt; - returns the annotation and composite case for rework; - marks the reviewer annotation as unsupported; - gives the attempter specific positive feedback; - gives the reviewer a minimal corrective explanation; - records a project-owner signal in case this misunderstanding recurs. This demonstrates why attempt quality and annotation fidelity must be separate outputs. ## Versioning rule Three versions may change independently: - **Schema version** - shape and semantics of machine-readable documents. - **Contract version** - project instructions, criteria, anchors, or evidence policy. - **Decision-policy version** - operational mapping from criterion state to action. A case must reference an exact contract version and content hash. An evaluation must record the exact case, contract, prompt, source snapshots, model runs, calibration profile, and decision-policy version used. ## What 0.1.0 deliberately leaves open - model and provider selection; - retrieval architecture; - prompt templates; - long-context chunking strategy; - calibration algorithm; - reviewer-competence model; - dataset-level value model; - UI design; - persistence and job orchestration. These should be decided empirically after the contract and benchmark are stable. "}, {"slug": "foundation-version", "title": "Foundation Package - Version", "category": "Foundation specifications", "summary": "Current package version identifier.", "tags": ["foundation", "specification"], "view": "documents/view/foundation-version.html", "raw": "documents/raw/foundation/VERSION", "text": "0.1.0 "}, {"slug": "foundation-examples-benchmark-item-example-json", "title": "Worked Example - Benchmark Item", "category": "Foundation examples", "summary": "Adjudicated benchmark record for the synthetic case, including acceptable outcomes and ambiguity metadata.", "tags": ["example", "worked case"], "view": "documents/view/foundation-examples-benchmark-item-example-json.html", "raw": "documents/raw/foundation/examples/benchmark-item.example.json", "text": "{ \"documentType\": \"benchmark_item\", \"schemaVersion\": \"0.1.0\", \"benchmarkItemId\": \"benchmark-example-001\", \"benchmarkVersion\": \"0.1.0\", \"split\": \"challenge\", \"contractRef\": { \"contractId\": \"example-grounded-analysis\", \"contractVersion\": \"0.1.0\", \"contentHash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\" }, \"caseRef\": { \"caseId\": \"case-example-001\", \"contentHash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\" }, \"sliceTags\": [ \"causal-warrant\", \"positive-attempt\", \"reviewer-false-reject\" ], \"challengeAttributes\": [ \"strong_positive\", \"wrong_label_correct_observation\", \"severity_edge\" ], \"goldStandard\": { \"criterionGold\": [ { \"criterionId\": \"C1-direct-response\", \"applicability\": \"applies\", \"acceptableVerdicts\": [ \"met\" ], \"severity\": \"none\", \"positiveEvidenceSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"negativeEvidenceSpans\": [], \"omissionDescriptions\": [], \"notes\": \"The calibrated conclusion directly answers the question.\" }, { \"criterionId\": \"C2-grounding\", \"applicability\": \"applies\", \"acceptableVerdicts\": [ \"met\" ], \"severity\": \"none\", \"positiveEvidenceSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" }, { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E2 estimates a 4% improvement, but its interval includes zero.\", \"locator\": \"sentence 3\" } ], \"negativeEvidenceSpans\": [], \"omissionDescriptions\": [], \"notes\": \"Material factual claims are supported by E1 and E2.\" }, { \"criterionId\": \"C3-warrant\", \"applicability\": \"applies\", \"acceptableVerdicts\": [ \"met\" ], \"severity\": \"none\", \"positiveEvidenceSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"negativeEvidenceSpans\": [], \"omissionDescriptions\": [], \"notes\": \"Conclusion strength is proportionate to study design and uncertainty.\" }, { \"criterionId\": \"C4-completeness\", \"applicability\": \"applies\", \"acceptableVerdicts\": [ \"met\" ], \"severity\": \"none\", \"positiveEvidenceSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" }, { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E2 estimates a 4% improvement, but its interval includes zero.\", \"locator\": \"sentence 3\" } ], \"negativeEvidenceSpans\": [], \"omissionDescriptions\": [], \"notes\": \"Both studies are addressed.\" }, { \"criterionId\": \"C5-annotation-fidelity\", \"applicability\": \"applies\", \"acceptableVerdicts\": [ \"not_met\" ], \"severity\": \"major\", \"positiveEvidenceSpans\": [], \"negativeEvidenceSpans\": [ { \"artifactType\": \"human_review\", \"artifactId\": \"review-example-001\", \"quotedText\": \"The attempter did not choose yes or no and therefore did not answer the question directly.\", \"locator\": \"rationale\" } ], \"omissionDescriptions\": [ \"The review omits the contract language permitting an inconclusive answer.\" ], \"notes\": \"The review applies an unstated binary-answer preference.\" } ], \"annotationGold\": { \"acceptableLabelSupport\": [ \"unsupported\" ], \"expectedDisagreementTypes\": [ \"criterion_interpretation\", \"threshold\" ], \"notes\": \"Accept the attempt and flag the reviewer annotation.\" }, \"ambiguityStatus\": \"clear\", \"validAlternativeNotes\": [ \"A qualified \"not established\" conclusion is explicitly valid.\" ], \"acceptableDecisions\": [ { \"target\": \"attempt\", \"acceptableActions\": [ \"accept\" ] }, { \"target\": \"annotation\", \"acceptableActions\": [ \"rework\" ] }, { \"target\": \"composite_case\", \"acceptableActions\": [ \"rework\" ] } ] }, \"adjudication\": { \"method\": \"independent_then_adjudicate\", \"adjudicatorCount\": 3, \"domainExpertCount\": 1, \"blindToAutoQA\": true, \"blindToReviewerLabel\": false, \"initialAgreementRate\": 1.0, \"disagreementTypes\": [], \"adjudicationNotes\": \"All adjudicators agreed that the attempt passed and the human review failed the fidelity criterion.\", \"completedAt\": \"2026-07-15T12:30:00Z\" }, \"leakageControl\": { \"allowedForPromptDevelopment\": false, \"heldOutProject\": false, \"firstFrozenAt\": \"2026-07-15T12:30:00Z\", \"accessPolicy\": \"Challenge-set content is available only to benchmark maintainers until a release decision is made.\" }, \"weight\": 1.0, \"metadata\": { \"example\": true } } "}, {"slug": "foundation-examples-case-input-example-json", "title": "Worked Example - Case Input", "category": "Foundation examples", "summary": "A strong attempt incorrectly failed by a human reviewer for not forcing an unsupported binary conclusion.", "tags": ["example", "worked case"], "view": "documents/view/foundation-examples-case-input-example-json.html", "raw": "documents/raw/foundation/examples/case-input.example.json", "text": "{ \"documentType\": \"case_input\", \"schemaVersion\": \"0.1.0\", \"caseId\": \"case-example-001\", \"createdAt\": \"2026-07-15T12:10:00Z\", \"contractRef\": { \"contractId\": \"example-grounded-analysis\", \"contractVersion\": \"0.1.0\", \"contentHash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\" }, \"task\": { \"taskId\": \"task-example-001\", \"taskType\": \"evidence_grounded_analysis\", \"prompt\": \"Using Evidence E1 and E2, assess whether Initiative X is the primary cause of the observed improvement in Outcome Y. Give a concise conclusion and explain the evidence.\", \"taskInstructions\": [], \"expectedOutputFormat\": \"One concise analytical paragraph with evidence identifiers.\", \"contentHash\": \"sha256:deae115ef5c75b4b125e4dbb09937257231c21abcefcfb6b9c024804a1d14ec6\", \"metadata\": { \"requiresCausalAssessment\": true } }, \"evidenceBundle\": [ { \"evidenceId\": \"E1\", \"evidenceClassId\": \"supplied-study\", \"title\": \"Observational comparison\", \"content\": \"An observational comparison found Outcome Y was 18% higher in units using Initiative X. The authors state that units self-selected into the initiative and that the study does not identify a causal effect.\", \"contentHash\": \"sha256:192359f876334c9282c7e79247a6b78b913f63cfb032b68d8c0f7466e6cdce82\", \"authorityRank\": 3, \"permitted\": true, \"capturedAt\": \"2026-07-15T12:05:00Z\", \"metadata\": { \"studyDesign\": \"observational\" } }, { \"evidenceId\": \"E2\", \"evidenceClassId\": \"supplied-study\", \"title\": \"Randomized pilot\", \"content\": \"A randomized pilot estimated a 4% improvement in Outcome Y. The confidence interval included zero, so the pilot did not establish a statistically distinguishable effect.\", \"contentHash\": \"sha256:b3bba4e3b5b48b3ace0ec78b81738a837464d94d945689b57065d2f8f50c4428\", \"authorityRank\": 3, \"permitted\": true, \"capturedAt\": \"2026-07-15T12:05:00Z\", \"metadata\": { \"studyDesign\": \"randomized_pilot\" } } ], \"attempt\": { \"attemptId\": \"attempt-example-001\", \"authorId\": \"worker-104\", \"content\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause. E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification. E2 estimates a 4% improvement, but its interval includes zero. The strongest justified conclusion is that X may contribute to the improvement; primacy remains unproven.\", \"contentHash\": \"sha256:8d0c8290f24f40ae2ca136a7556dbaa2938591a756497506d927d6fc28d43a64\", \"submittedAt\": \"2026-07-15T12:08:00Z\", \"attachments\": [], \"toolTraces\": [] }, \"humanReview\": { \"annotationId\": \"review-example-001\", \"reviewerId\": \"reviewer-207\", \"label\": \"fail\", \"score\": 0, \"reasonCodes\": [ \"not_direct_answer\" ], \"rationale\": \"Fail. The attempter did not choose yes or no and therefore did not answer the question directly.\", \"criterionAnnotations\": [ { \"criterionId\": \"C1-direct-response\", \"verdict\": \"not_met\", \"severity\": \"major\", \"rationale\": \"The response did not choose yes or no.\", \"citedSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"the supplied evidence does not establish that it is the primary cause\", \"locator\": \"sentence 1\" } ] } ], \"reviewerConfidence\": { \"rawScore\": 0.9, \"band\": \"uncalibrated\" }, \"submittedAt\": \"2026-07-15T12:09:00Z\" }, \"referenceAnswers\": [ { \"referenceId\": \"reference-example-001\", \"role\": \"illustrative\", \"content\": \"The evidence suggests a possible contribution from X but does not establish that X is the primary cause.\", \"contentHash\": \"sha256:83260b932b3b8b56d7785dc4ff7d859c2cc77426b927cba51ae6f2571d9c04d0\" } ], \"declaredExpectedElements\": [], \"caseTags\": [ \"synthetic\", \"positive-attempt\", \"incorrect-human-fail\", \"causal-warrant\" ], \"metadata\": { \"example\": true } } "}, {"slug": "foundation-examples-evaluation-output-example-json", "title": "Worked Example - Evaluation Output", "category": "Foundation examples", "summary": "Complete AutoQA result accepting the attempt, returning the annotation for rework, and generating audience-specific feedback.", "tags": ["example", "worked case"], "view": "documents/view/foundation-examples-evaluation-output-example-json.html", "raw": "documents/raw/foundation/examples/evaluation-output.example.json", "text": "{ \"documentType\": \"evaluation_output\", \"schemaVersion\": \"0.1.0\", \"evaluationId\": \"evaluation-example-001\", \"evaluationVersion\": \"0.1.0\", \"caseRef\": { \"caseId\": \"case-example-001\", \"contentHash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\" }, \"contractRef\": { \"contractId\": \"example-grounded-analysis\", \"contractVersion\": \"0.1.0\", \"contentHash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\" }, \"createdAt\": \"2026-07-15T12:12:00Z\", \"status\": \"final\", \"runManifest\": [ { \"modelRunId\": \"run-claim-001\", \"role\": \"claim_extractor\", \"provider\": \"example-provider-a\", \"modelName\": \"example-model-fast\", \"modelVersion\": \"2026-06\", \"promptTemplateId\": \"claim-extractor-v1\", \"promptTemplateVersion\": \"0.1.0\", \"promptTemplateHash\": \"sha256:c0134a01b0c26efe652d790a8e07662ef46052dc3b92fb183ab90e17c45729b7\", \"parameters\": { \"temperature\": 0 }, \"parentRunIds\": [], \"inputHash\": \"sha256:204ff6a20394ccad26982cd41cb7aa5a9f03133a3fb774baea728084e2a5a5d1\", \"outputHash\": \"sha256:fa1a14caa9781c43f098091c180ece6044fec851dee6c99b22f9315ff2ec42d1\", \"startedAt\": \"2026-07-15T12:10:05Z\", \"completedAt\": \"2026-07-15T12:10:06Z\" }, { \"modelRunId\": \"run-adjudicator-001\", \"role\": \"criterion_adjudicator\", \"provider\": \"example-provider-b\", \"modelName\": \"example-model-reasoning\", \"modelVersion\": \"2026-06\", \"promptTemplateId\": \"criterion-adjudicator-v1\", \"promptTemplateVersion\": \"0.1.0\", \"promptTemplateHash\": \"sha256:c0134a01b0c26efe652d790a8e07662ef46052dc3b92fb183ab90e17c45729b7\", \"parameters\": { \"temperature\": 0 }, \"parentRunIds\": [ \"run-claim-001\" ], \"inputHash\": \"sha256:50cf3c2562bbc06499085ca8f13ace7dc99463a30ed0098dcfdcc513ba690867\", \"outputHash\": \"sha256:1da443352eb7fbdc1a924549753603adde1205ac33f3ff8ee91eb9bca1df9143\", \"startedAt\": \"2026-07-15T12:10:07Z\", \"completedAt\": \"2026-07-15T12:10:12Z\" } ], \"integrityStatus\": \"clean\", \"findings\": [ { \"findingId\": \"F1-direct-positive\", \"findingType\": \"positive\", \"criterionId\": \"C1-direct-response\", \"summary\": \"The attempt directly states that primacy is not established.\", \"detail\": \"An inconclusive but calibrated answer is explicitly permitted by the contract.\", \"claimIds\": [ \"CL1\" ], \"expectedElementIds\": [ \"EE1\" ], \"targetSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"evidenceIds\": [ \"E1\", \"E2\" ], \"relationship\": \"reasonably_supports\", \"materiality\": \"decisive\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"status\": \"confirmed\", \"source\": { \"kind\": \"hybrid\", \"componentId\": \"criterion-pipeline-v1\", \"modelRunId\": \"run-adjudicator-001\" } }, { \"findingId\": \"F2-grounding-positive\", \"findingType\": \"positive\", \"criterionId\": \"C2-grounding\", \"summary\": \"The material study claims match E1 and E2.\", \"detail\": \"The reported effect sizes and limitations are accurately attributed.\", \"claimIds\": [ \"CL2\", \"CL3\" ], \"expectedElementIds\": [], \"targetSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" }, { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E2 estimates a 4% improvement, but its interval includes zero.\", \"locator\": \"sentence 3\" } ], \"evidenceIds\": [ \"E1\", \"E2\" ], \"relationship\": \"directly_entails\", \"materiality\": \"central\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"status\": \"confirmed\", \"source\": { \"kind\": \"hybrid\", \"componentId\": \"grounding-pipeline-v1\", \"modelRunId\": \"run-adjudicator-001\" } }, { \"findingId\": \"F3-warrant-positive\", \"findingType\": \"positive\", \"criterionId\": \"C3-warrant\", \"summary\": \"The conclusion appropriately distinguishes association, possible contribution, and unproven primacy.\", \"detail\": \"The conclusion is proportionate to an observational association plus an inconclusive randomized pilot.\", \"claimIds\": [ \"CL1\", \"CL4\" ], \"expectedElementIds\": [ \"EE2\" ], \"targetSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"evidenceIds\": [ \"E1\", \"E2\" ], \"relationship\": \"reasonably_supports\", \"materiality\": \"decisive\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"status\": \"confirmed\", \"source\": { \"kind\": \"llm_judge\", \"componentId\": \"warrant-auditor-v1\", \"modelRunId\": \"run-adjudicator-001\" } }, { \"findingId\": \"F4-completeness-positive\", \"findingType\": \"positive\", \"criterionId\": \"C4-completeness\", \"summary\": \"Both supplied studies are materially addressed.\", \"detail\": \"The attempt states the key result and limitation of each study.\", \"claimIds\": [ \"CL2\", \"CL3\" ], \"expectedElementIds\": [ \"EE3\", \"EE4\" ], \"targetSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" }, { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E2 estimates a 4% improvement, but its interval includes zero.\", \"locator\": \"sentence 3\" } ], \"evidenceIds\": [ \"E1\", \"E2\" ], \"relationship\": \"directly_entails\", \"materiality\": \"central\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"status\": \"confirmed\", \"source\": { \"kind\": \"hybrid\", \"componentId\": \"coverage-auditor-v1\", \"modelRunId\": \"run-adjudicator-001\" } }, { \"findingId\": \"F5-review-negative\", \"findingType\": \"negative\", \"criterionId\": \"C5-annotation-fidelity\", \"summary\": \"The reviewer applies an unstated binary-answer requirement that conflicts with the contract.\", \"detail\": \"The response directly answers that primacy is unproven, and the instructions explicitly permit an inconclusive conclusion.\", \"claimIds\": [], \"expectedElementIds\": [], \"targetSpans\": [ { \"artifactType\": \"human_review\", \"artifactId\": \"review-example-001\", \"quotedText\": \"The attempter did not choose yes or no and therefore did not answer the question directly.\", \"locator\": \"rationale\" }, { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"evidenceIds\": [], \"relationship\": \"contradicts\", \"materiality\": \"decisive\", \"severity\": \"major\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"status\": \"confirmed\", \"source\": { \"kind\": \"llm_judge\", \"componentId\": \"annotation-auditor-v1\", \"modelRunId\": \"run-adjudicator-001\" } } ], \"claims\": [ { \"claimId\": \"CL1\", \"proposition\": \"The supplied evidence does not establish that Initiative X is the primary cause of the improvement in Outcome Y.\", \"claimType\": \"interpretive\", \"importance\": \"central\", \"explicitness\": \"explicit\", \"attemptSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"qualifiers\": [ \"does not establish\", \"primary cause\" ], \"citedEvidenceIds\": [ \"E1\", \"E2\" ] }, { \"claimId\": \"CL2\", \"proposition\": \"E1 reports an 18% observational difference and notes self-selection and no causal identification.\", \"claimType\": \"factual\", \"importance\": \"supporting\", \"explicitness\": \"explicit\", \"attemptSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" } ], \"qualifiers\": [ \"observational\" ], \"citedEvidenceIds\": [ \"E1\" ] }, { \"claimId\": \"CL3\", \"proposition\": \"E2 estimates a 4% improvement and its confidence interval includes zero.\", \"claimType\": \"factual\", \"importance\": \"supporting\", \"explicitness\": \"explicit\", \"attemptSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E2 estimates a 4% improvement, but its interval includes zero.\", \"locator\": \"sentence 3\" } ], \"qualifiers\": [], \"citedEvidenceIds\": [ \"E2\" ] }, { \"claimId\": \"CL4\", \"proposition\": \"Initiative X may contribute to the improvement.\", \"claimType\": \"interpretive\", \"importance\": \"central\", \"explicitness\": \"explicit\", \"attemptSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"X may contribute to the improvement; primacy remains unproven.\", \"locator\": \"sentence 4\" } ], \"qualifiers\": [ \"may\" ], \"citedEvidenceIds\": [ \"E1\", \"E2\" ] } ], \"expectedElements\": [ { \"elementId\": \"EE1\", \"criterionId\": \"C1-direct-response\", \"description\": \"A direct conclusion about whether X is established as the primary cause.\", \"required\": true, \"importance\": \"decisive\", \"acceptableAlternatives\": [ \"Primacy is not established.\", \"The evidence is insufficient to determine primacy.\" ], \"observedSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" } ], \"coverage\": \"complete\" }, { \"elementId\": \"EE2\", \"criterionId\": \"C3-warrant\", \"description\": \"A distinction between association and causal or primary-cause inference.\", \"required\": true, \"importance\": \"decisive\", \"acceptableAlternatives\": [], \"observedSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"Initiative X is associated with higher Outcome Y, but the supplied evidence does not establish that it is the primary cause.\", \"locator\": \"sentence 1\" }, { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" } ], \"coverage\": \"complete\" }, { \"elementId\": \"EE3\", \"criterionId\": \"C4-completeness\", \"description\": \"Material treatment of E1.\", \"required\": true, \"importance\": \"central\", \"acceptableAlternatives\": [], \"observedSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\", \"locator\": \"sentence 2\" } ], \"coverage\": \"complete\" }, { \"elementId\": \"EE4\", \"criterionId\": \"C4-completeness\", \"description\": \"Material treatment of E2.\", \"required\": true, \"importance\": \"central\", \"acceptableAlternatives\": [], \"observedSpans\": [ { \"artifactType\": \"attempt\", \"artifactId\": \"attempt-example-001\", \"quotedText\": \"E2 estimates a 4% improvement, but its interval includes zero.\", \"locator\": \"sentence 3\" } ], \"coverage\": \"complete\" } ], \"evidenceRelationships\": [ { \"relationshipId\": \"ER1\", \"subjectType\": \"claim\", \"subjectId\": \"CL2\", \"evidenceId\": \"E1\", \"relationship\": \"directly_entails\", \"sourceQuality\": \"high\", \"temporalValidity\": \"not_time_sensitive\", \"evidenceSpans\": [ { \"artifactType\": \"evidence\", \"artifactId\": \"E1\", \"quotedText\": \"Outcome Y was 18% higher in units using Initiative X. The authors state that units self-selected into the initiative and that the study does not identify a causal effect.\", \"locator\": \"entire excerpt\" } ], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"notes\": \"The attempt accurately reports both the numerical result and the study-design limitation.\" }, { \"relationshipId\": \"ER2\", \"subjectType\": \"claim\", \"subjectId\": \"CL3\", \"evidenceId\": \"E2\", \"relationship\": \"directly_entails\", \"sourceQuality\": \"high\", \"temporalValidity\": \"not_time_sensitive\", \"evidenceSpans\": [ { \"artifactType\": \"evidence\", \"artifactId\": \"E2\", \"quotedText\": \"A randomized pilot estimated a 4% improvement in Outcome Y. The confidence interval included zero\", \"locator\": \"sentences 1-2\" } ], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"notes\": \"The attempt accurately reports the estimate and uncertainty.\" }, { \"relationshipId\": \"ER3\", \"subjectType\": \"claim\", \"subjectId\": \"CL1\", \"evidenceId\": \"E1\", \"relationship\": \"reasonably_supports\", \"sourceQuality\": \"high\", \"temporalValidity\": \"not_time_sensitive\", \"evidenceSpans\": [ { \"artifactType\": \"evidence\", \"artifactId\": \"E1\", \"quotedText\": \"Outcome Y was 18% higher in units using Initiative X. The authors state that units self-selected into the initiative and that the study does not identify a causal effect.\", \"locator\": \"entire excerpt\" } ], \"confidence\": { \"rawScore\": 0.82, \"calibratedScore\": 0.79, \"band\": \"medium\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"notes\": \"E1 explicitly cannot identify causation, supporting the conclusion that primacy is not established.\" }, { \"relationshipId\": \"ER4\", \"subjectType\": \"claim\", \"subjectId\": \"CL1\", \"evidenceId\": \"E2\", \"relationship\": \"reasonably_supports\", \"sourceQuality\": \"high\", \"temporalValidity\": \"not_time_sensitive\", \"evidenceSpans\": [ { \"artifactType\": \"evidence\", \"artifactId\": \"E2\", \"quotedText\": \"A randomized pilot estimated a 4% improvement in Outcome Y. The confidence interval included zero\", \"locator\": \"sentences 1-2\" } ], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"notes\": \"The randomized pilot is inconclusive and therefore does not establish primacy.\" } ], \"criterionAssessments\": [ { \"criterionId\": \"C1-direct-response\", \"applicability\": \"applies\", \"positiveFindingIds\": [ \"F1-direct-positive\" ], \"negativeFindingIds\": [], \"omissionFindingIds\": [], \"counterEvidenceFindingIds\": [], \"unresolvedFindingIds\": [], \"groundingStatus\": \"grounded\", \"truthStatus\": \"not_evaluated\", \"warrantStatus\": \"justified\", \"relevanceStatus\": \"relevant\", \"verdict\": \"met\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"decisionEffect\": \"none\", \"rationale\": \"The first sentence directly answers the primary-cause question with the calibrated conclusion permitted by the contract.\" }, { \"criterionId\": \"C2-grounding\", \"applicability\": \"applies\", \"positiveFindingIds\": [ \"F2-grounding-positive\" ], \"negativeFindingIds\": [], \"omissionFindingIds\": [], \"counterEvidenceFindingIds\": [], \"unresolvedFindingIds\": [], \"groundingStatus\": \"grounded\", \"truthStatus\": \"verified_true\", \"warrantStatus\": \"justified\", \"relevanceStatus\": \"relevant\", \"verdict\": \"met\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"decisionEffect\": \"none\", \"rationale\": \"The factual statements about both studies are directly supported by the supplied excerpts.\" }, { \"criterionId\": \"C3-warrant\", \"applicability\": \"applies\", \"positiveFindingIds\": [ \"F3-warrant-positive\" ], \"negativeFindingIds\": [], \"omissionFindingIds\": [], \"counterEvidenceFindingIds\": [], \"unresolvedFindingIds\": [], \"groundingStatus\": \"grounded\", \"truthStatus\": \"not_evaluated\", \"warrantStatus\": \"justified\", \"relevanceStatus\": \"relevant\", \"verdict\": \"met\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"decisionEffect\": \"none\", \"rationale\": \"The attempt does not infer causation from E1 and treats E2 as inconclusive, so its conclusion strength matches the evidence.\" }, { \"criterionId\": \"C4-completeness\", \"applicability\": \"applies\", \"positiveFindingIds\": [ \"F4-completeness-positive\" ], \"negativeFindingIds\": [], \"omissionFindingIds\": [], \"counterEvidenceFindingIds\": [], \"unresolvedFindingIds\": [], \"groundingStatus\": \"grounded\", \"truthStatus\": \"verified_true\", \"warrantStatus\": \"justified\", \"relevanceStatus\": \"relevant\", \"verdict\": \"met\", \"severity\": \"none\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"decisionEffect\": \"none\", \"rationale\": \"Both studies are materially summarized and synthesized.\" }, { \"criterionId\": \"C5-annotation-fidelity\", \"applicability\": \"applies\", \"positiveFindingIds\": [], \"negativeFindingIds\": [ \"F5-review-negative\" ], \"omissionFindingIds\": [], \"counterEvidenceFindingIds\": [], \"unresolvedFindingIds\": [], \"groundingStatus\": \"grounded\", \"truthStatus\": \"verified_false\", \"warrantStatus\": \"invalid\", \"relevanceStatus\": \"relevant\", \"verdict\": \"not_met\", \"severity\": \"major\", \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"decisionEffect\": \"accept_with_note\", \"rationale\": \"The reviewer fails the attempt for not choosing a binary answer even though the contract explicitly permits an inconclusive conclusion.\" } ], \"globalAssessments\": [ { \"axisId\": \"global-synthesis\", \"verdict\": \"met\", \"findingIds\": [ \"F1-direct-positive\", \"F3-warrant-positive\", \"F4-completeness-positive\" ], \"score\": 4, \"confidence\": { \"rawScore\": 0.82, \"calibratedScore\": 0.79, \"band\": \"medium\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"rationale\": \"The response combines the study designs and uncertainty into one calibrated conclusion rather than listing findings mechanically.\" } ], \"annotationAssessment\": { \"annotationPresent\": true, \"labelSupport\": \"unsupported\", \"rationaleSupport\": \"does_not_support_label\", \"criterionAttribution\": \"unstated_preference\", \"evidenceAlignment\": \"partially_aligned\", \"severityAlignment\": \"too_severe\", \"omittedPositiveFindingIds\": [ \"F1-direct-positive\", \"F2-grounding-positive\", \"F3-warrant-positive\", \"F4-completeness-positive\" ], \"omittedNegativeFindingIds\": [], \"unsupportedReviewerFindingIds\": [ \"F5-review-negative\" ], \"disagreementTypes\": [ \"criterion_interpretation\", \"threshold\" ], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"rationale\": \"The reviewer noticed the qualified conclusion but treated qualification as nonresponsiveness, contrary to the written criterion and pass anchor.\" }, \"feedback\": [ { \"feedbackId\": \"FB1-attempter-strength\", \"audience\": \"attempter\", \"criterionId\": \"C3-warrant\", \"feedbackType\": \"strength\", \"findingIds\": [ \"F3-warrant-positive\" ], \"message\": \"You correctly distinguish observational association from causal evidence and calibrate the conclusion to the uncertainty in both studies.\", \"priority\": 1, \"repairValidationStatus\": \"not_applicable\" }, { \"feedbackId\": \"FB2-reviewer-defect\", \"audience\": \"reviewer\", \"criterionId\": \"C1-direct-response\", \"feedbackType\": \"defect\", \"findingIds\": [ \"F5-review-negative\", \"F1-direct-positive\" ], \"message\": \"The attempt does answer the primary-cause question: it concludes that primacy is not established. The project explicitly permits an inconclusive answer when the evidence is inconclusive, so requiring a forced yes/no choice adds an unstated criterion.\", \"minimalRepair\": \"Change the label to pass and cite the first sentence as affirmative evidence for a direct, calibrated answer.\", \"priority\": 1, \"repairValidationStatus\": \"validated\" }, { \"feedbackId\": \"FB3-owner-signal\", \"audience\": \"project_owner\", \"criterionId\": \"C1-direct-response\", \"feedbackType\": \"ambiguity\", \"findingIds\": [ \"F5-review-negative\" ], \"message\": \"Monitor whether reviewers repeatedly equate \"direct answer\" with a forced binary choice. If the pattern recurs, add this exact positive anchor to reviewer training.\", \"priority\": 2, \"repairValidationStatus\": \"not_applicable\" } ], \"audit\": { \"caseInputHash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\", \"contractHash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\", \"sourceSnapshotHashes\": [ \"sha256:192359f876334c9282c7e79247a6b78b913f63cfb032b68d8c0f7466e6cdce82\", \"sha256:b3bba4e3b5b48b3ace0ec78b81738a837464d94d945689b57065d2f8f50c4428\" ], \"replayable\": true, \"warnings\": [ \"Example model and provider names are placeholders.\" ] }, \"provisionalDecisions\": [ { \"action\": \"accept\", \"policyVersion\": \"0.1.0\", \"triggeredRuleIds\": [ \"R4-accept-clean\" ], \"decisiveCriterionIds\": [ \"C1-direct-response\", \"C2-grounding\", \"C3-warrant\", \"C4-completeness\" ], \"unresolvedFindingIds\": [], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"policyTrace\": [ { \"ruleId\": \"R1-critical-hard-failure\", \"matched\": false, \"inputs\": {}, \"result\": \"No critical attempt-quality failure.\" }, { \"ruleId\": \"R2-major-repairable-failure\", \"matched\": false, \"inputs\": {}, \"result\": \"No major attempt-quality failure.\" }, { \"ruleId\": \"R3-decision-changing-uncertainty\", \"matched\": false, \"inputs\": {}, \"result\": \"No unresolved decision-changing finding.\" }, { \"ruleId\": \"R4-accept-clean\", \"matched\": true, \"inputs\": {}, \"result\": \"All attempt-quality criteria met.\" } ], \"rationale\": \"All applicable attempt-quality criteria are affirmatively met. The human review is incorrect but does not reduce the quality of the attempt itself.\", \"decidedAt\": \"2026-07-15T12:11:30Z\", \"target\": \"attempt\" }, { \"target\": \"annotation\", \"action\": \"rework\", \"policyVersion\": \"0.1.0\", \"triggeredRuleIds\": [ \"R5-annotation-major-failure\" ], \"decisiveCriterionIds\": [ \"C5-annotation-fidelity\" ], \"unresolvedFindingIds\": [], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"policyTrace\": [ { \"ruleId\": \"R5-annotation-major-failure\", \"matched\": true, \"inputs\": {}, \"result\": \"Annotation fidelity has a major, high-confidence failure.\" } ], \"rationale\": \"Return the human annotation for correction because its label and rationale add an unstated binary-answer requirement.\", \"decidedAt\": \"2026-07-15T12:12:00Z\" }, { \"target\": \"composite_case\", \"action\": \"rework\", \"policyVersion\": \"0.1.0\", \"triggeredRuleIds\": [ \"R7-composite-annotation-rework\" ], \"decisiveCriterionIds\": [ \"C5-annotation-fidelity\" ], \"unresolvedFindingIds\": [], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"policyTrace\": [ { \"ruleId\": \"R7-composite-annotation-rework\", \"matched\": true, \"inputs\": { \"attemptAction\": \"accept\", \"annotationAction\": \"rework\" }, \"result\": \"Retain the attempt but correct the annotation before admitting the composite datum.\" } ], \"rationale\": \"The attempt is acceptable, but the attempt-plus-review record is not ready for use as annotated training data until the review is corrected.\", \"decidedAt\": \"2026-07-15T12:12:00Z\" } ], \"finalDecisions\": [ { \"action\": \"accept\", \"policyVersion\": \"0.1.0\", \"triggeredRuleIds\": [ \"R4-accept-clean\" ], \"decisiveCriterionIds\": [ \"C1-direct-response\", \"C2-grounding\", \"C3-warrant\", \"C4-completeness\" ], \"unresolvedFindingIds\": [], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"policyTrace\": [ { \"ruleId\": \"R4-accept-clean\", \"matched\": true, \"inputs\": { \"humanResolutionRequired\": false }, \"result\": \"Accept attempt; flag reviewer annotation.\" } ], \"rationale\": \"Accept the attempt and mark the human annotation as unsupported. No human resolution is needed because the written contract and anchors directly resolve the dispute.\", \"decidedAt\": \"2026-07-15T12:12:00Z\", \"target\": \"attempt\" }, { \"target\": \"annotation\", \"action\": \"rework\", \"policyVersion\": \"0.1.0\", \"triggeredRuleIds\": [ \"R5-annotation-major-failure\" ], \"decisiveCriterionIds\": [ \"C5-annotation-fidelity\" ], \"unresolvedFindingIds\": [], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"policyTrace\": [ { \"ruleId\": \"R5-annotation-major-failure\", \"matched\": true, \"inputs\": {}, \"result\": \"Annotation fidelity has a major, high-confidence failure.\" } ], \"rationale\": \"Return the human annotation for correction because its label and rationale add an unstated binary-answer requirement.\", \"decidedAt\": \"2026-07-15T12:12:00Z\" }, { \"target\": \"composite_case\", \"action\": \"rework\", \"policyVersion\": \"0.1.0\", \"triggeredRuleIds\": [ \"R7-composite-annotation-rework\" ], \"decisiveCriterionIds\": [ \"C5-annotation-fidelity\" ], \"unresolvedFindingIds\": [], \"confidence\": { \"rawScore\": 0.94, \"calibratedScore\": 0.91, \"band\": \"high\", \"calibrationProfileId\": \"calibration-example-v1\" }, \"policyTrace\": [ { \"ruleId\": \"R7-composite-annotation-rework\", \"matched\": true, \"inputs\": { \"attemptAction\": \"accept\", \"annotationAction\": \"rework\" }, \"result\": \"Retain the attempt but correct the annotation before admitting the composite datum.\" } ], \"rationale\": \"The attempt is acceptable, but the attempt-plus-review record is not ready for use as annotated training data until the review is corrected.\", \"decidedAt\": \"2026-07-15T12:12:00Z\" } ], \"validatorManifest\": [ { \"validatorRunId\": \"validator-run-schema-001\", \"validatorId\": \"schema-validator\", \"validatorVersion\": \"0.1.0\", \"role\": \"schema_validation\", \"status\": \"passed\", \"parameters\": { \"draft\": \"2020-12\" }, \"inputHash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\", \"outputHash\": \"sha256:1111111111111111111111111111111111111111111111111111111111111111\", \"startedAt\": \"2026-07-15T12:10:00Z\", \"completedAt\": \"2026-07-15T12:10:01Z\" }, { \"validatorRunId\": \"validator-run-citation-001\", \"validatorId\": \"citation-id-validator\", \"validatorVersion\": \"0.1.0\", \"role\": \"citation_check\", \"status\": \"passed\", \"parameters\": {}, \"inputHash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\", \"outputHash\": \"sha256:2222222222222222222222222222222222222222222222222222222222222222\", \"startedAt\": \"2026-07-15T12:10:01Z\", \"completedAt\": \"2026-07-15T12:10:02Z\" }, { \"validatorRunId\": \"validator-run-required-evidence-001\", \"validatorId\": \"required-evidence-id-validator\", \"validatorVersion\": \"0.1.0\", \"role\": \"custom\", \"status\": \"passed\", \"parameters\": {}, \"inputHash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\", \"outputHash\": \"sha256:3333333333333333333333333333333333333333333333333333333333333333\", \"startedAt\": \"2026-07-15T12:10:01Z\", \"completedAt\": \"2026-07-15T12:10:02Z\" } ] } "}, {"slug": "foundation-examples-human-resolution-packet-example-json", "title": "Worked Example - Human Resolution", "category": "Foundation examples", "summary": "A bounded precedence question that hides the provisional decision and asks only what can change the outcome.", "tags": ["example", "worked case"], "view": "documents/view/foundation-examples-human-resolution-packet-example-json.html", "raw": "documents/raw/foundation/examples/human-resolution-packet.example.json", "text": "{ \"documentType\": \"human_resolution_packet\", \"schemaVersion\": \"0.1.0\", \"caseRef\": { \"caseId\": \"case-hypothetical-resolution-001\" }, \"contractRef\": { \"contractId\": \"example-grounded-analysis\", \"contractVersion\": \"0.1.0\", \"contentHash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\" }, \"interaction\": { \"decisionTarget\": \"attempt\", \"request\": { \"requestId\": \"resolution-hypothetical-001\", \"triggerCriterionIds\": [ \"C1-direct-response\" ], \"questionType\": \"instruction_precedence\", \"question\": \"A task-specific note says \"Select yes or no,\" while the primary project instruction says not to force a binary conclusion when the evidence is inconclusive. Which instruction controls this case?\", \"contextFindingIds\": [ \"F-hypothetical-conflict-001\" ], \"options\": [ { \"optionId\": \"task-note-controls\", \"label\": \"The task-specific yes/no note controls\", \"effectSummary\": \"Evaluate directness using a binary-answer requirement.\" }, { \"optionId\": \"project-instruction-controls\", \"label\": \"The primary project instruction controls\", \"effectSummary\": \"Permit a calibrated inconclusive conclusion.\" }, { \"optionId\": \"contract-ambiguous\", \"label\": \"The project instruction is ambiguous\", \"effectSummary\": \"Escalate the case and create a contract issue.\" } ], \"decisionChanging\": true, \"hideProvisionalDecision\": true, \"requiredExpertiseTags\": [ \"project-owner\" ], \"status\": \"pending\", \"createdAt\": \"2026-07-15T13:00:00Z\" } }, \"contextFindings\": [ { \"findingId\": \"F-hypothetical-conflict-001\", \"findingType\": \"ambiguity\", \"criterionId\": \"C1-direct-response\", \"summary\": \"A task-specific binary-answer note appears to conflict with the primary project instruction.\", \"detail\": \"The precedence of the task note and project-level instruction is not defined for this hypothetical case.\", \"claimIds\": [], \"expectedElementIds\": [], \"targetSpans\": [], \"evidenceIds\": [], \"relationship\": \"insufficient\", \"materiality\": \"decisive\", \"severity\": \"major\", \"confidence\": { \"rawScore\": 0.48, \"band\": \"uncalibrated\" }, \"status\": \"confirmed\", \"source\": { \"kind\": \"hybrid\", \"componentId\": \"ambiguity-detector-v1\" } } ] } "}, {"slug": "foundation-examples-project-contract-example-json", "title": "Worked Example - Project Contract", "category": "Foundation examples", "summary": "Synthetic closed-world evidence project with positive-proof requirements, anchors, severity, and explicit decision rules.", "tags": ["example", "worked case"], "view": "documents/view/foundation-examples-project-contract-example-json.html", "raw": "documents/raw/foundation/examples/project-contract.example.json", "text": "{ \"documentType\": \"project_contract\", \"schemaVersion\": \"0.1.0\", \"contractId\": \"example-grounded-analysis\", \"contractVersion\": \"0.1.0\", \"projectName\": \"Example Evidence-Grounded Analysis\", \"description\": \"Synthetic example showing positive proof, calibrated conclusions, and independent annotation auditing.\", \"status\": \"approved\", \"createdAt\": \"2026-07-15T12:00:00Z\", \"effectiveFrom\": \"2026-07-15T12:00:00Z\", \"owners\": [ { \"actorId\": \"owner-001\", \"role\": \"project_owner\" }, { \"actorId\": \"expert-001\", \"role\": \"domain_expert\" } ], \"instructionDocuments\": [ { \"instructionId\": \"instructions-main\", \"title\": \"Main project instructions\", \"authority\": \"primary\", \"content\": \"Use only the supplied evidence. State the strongest conclusion supported by the evidence. Distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim. Do not force a binary conclusion when the evidence is inconclusive.\", \"contentHash\": \"sha256:adca02e1bb85249a185e0460d27f4e9a28de241a2f835dce671cacd15a858495\", \"precedenceRank\": 1 } ], \"sourcePolicy\": { \"worldMode\": \"closed_world\", \"parametricKnowledgePolicy\": \"never_evidence\", \"externalRetrievalAllowed\": false, \"defaultCitationRequirement\": \"material_claims\", \"snapshotRequired\": true, \"conflictPolicy\": \"Report material conflict or uncertainty; do not silently choose the more convenient source.\", \"freshnessRules\": [], \"evidenceClasses\": [ { \"classId\": \"supplied-study\", \"name\": \"Supplied study excerpt\", \"description\": \"Evidence included with the task and permitted for all factual judgments.\", \"permitted\": true, \"authorityRank\": 3 } ] }, \"criteria\": [ { \"criterionId\": \"C1-direct-response\", \"name\": \"Directly answers the question\", \"requirement\": \"State the strongest conclusion supported by the evidence about whether X is the primary cause.\", \"intent\": \"The response must resolve the requested issue without inventing certainty.\", \"criterionType\": \"required_presence\", \"scope\": \"attempt\", \"subjectivity\": \"bounded_judgment\", \"instructionRefs\": [ { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 2\", \"quotedText\": \"State the strongest conclusion supported by the evidence.\" }, { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 4\", \"quotedText\": \"Do not force a binary conclusion when the evidence is inconclusive.\" } ], \"applicability\": { \"ruleText\": \"Always applies.\", \"condition\": { \"kind\": \"always\" } }, \"proofPolicy\": { \"positiveProofRequired\": true, \"positiveProofStandard\": \"The attempt states an explicit conclusion that answers the primary-cause question at the confidence level supported by the evidence.\", \"failureCondition\": \"The attempt does not state a conclusion, answers a different question, or forces a conclusion stronger than the evidence permits.\", \"omissionCondition\": \"No conclusion about primary causation is present.\", \"passOnNoViolation\": false, \"allowPartialCredit\": true, \"allowCompensation\": false }, \"evidencePolicy\": { \"allowedEvidenceClassIds\": [ \"supplied-study\" ], \"minimumAuthorityRank\": 3, \"citationRequired\": false, \"externalVerificationMode\": \"forbidden\", \"deterministicValidatorIds\": [] }, \"anchors\": [ { \"anchorId\": \"C1-pass-anchor\", \"label\": \"Calibrated direct answer\", \"example\": \"The evidence does not establish that X is the primary cause.\", \"expectedVerdict\": \"met\", \"notes\": \"A non-binary answer is direct when the evidence is inconclusive.\" }, { \"anchorId\": \"C1-fail-anchor\", \"label\": \"No answer\", \"example\": \"The studies have several methodological differences.\", \"expectedVerdict\": \"not_met\", \"notes\": \"Describes evidence without answering the causal question.\" } ], \"validAlternatives\": [ \"A qualified no: the evidence does not establish primacy.\", \"An explicitly uncertain conclusion supported by both studies.\" ], \"exceptions\": [], \"dependencies\": { \"requiredBeforeCriterionIds\": [], \"impliesCriterionIds\": [] }, \"precedence\": { \"overridesCriterionIds\": [], \"overriddenByCriterionIds\": [] }, \"defaultSeverity\": \"major\", \"defaultDecisionEffect\": \"rework\", \"tags\": [ \"core\", \"answerability\", \"positive-proof\" ] }, { \"criterionId\": \"C2-grounding\", \"name\": \"Material claims are grounded\", \"requirement\": \"Every material factual claim must be supported by permitted supplied evidence.\", \"intent\": \"Prevent unsupported facts and citation laundering.\", \"criterionType\": \"grounding\", \"scope\": \"claim\", \"subjectivity\": \"objective\", \"instructionRefs\": [ { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 1\", \"quotedText\": \"Use only the supplied evidence.\" }, { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 3\", \"quotedText\": \"Distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim.\" } ], \"applicability\": { \"ruleText\": \"Applies to all material factual claims.\", \"condition\": { \"kind\": \"always\" } }, \"proofPolicy\": { \"positiveProofRequired\": true, \"positiveProofStandard\": \"Each material claim is linked to evidence that directly entails or reasonably supports it.\", \"failureCondition\": \"A central or supporting factual claim lacks permitted evidence or conflicts with it.\", \"omissionCondition\": \"A required citation or evidence link for a material claim is absent.\", \"passOnNoViolation\": false, \"allowPartialCredit\": true, \"allowCompensation\": false }, \"evidencePolicy\": { \"allowedEvidenceClassIds\": [ \"supplied-study\" ], \"minimumAuthorityRank\": 3, \"citationRequired\": true, \"externalVerificationMode\": \"forbidden\", \"deterministicValidatorIds\": [ \"citation-id-validator\" ] }, \"anchors\": [ { \"anchorId\": \"C2-pass-anchor\", \"label\": \"Claim-specific support\", \"example\": \"E1 reports an 18% observational difference and notes self-selection.\", \"expectedVerdict\": \"met\", \"notes\": \"The claim matches the supplied excerpt.\" }, { \"anchorId\": \"C2-fail-anchor\", \"label\": \"Unsupported strengthening\", \"example\": \"E1 proves that X caused the improvement.\", \"expectedVerdict\": \"not_met\", \"notes\": \"The source explicitly disclaims causal identification.\" } ], \"validAlternatives\": [], \"exceptions\": [ \"Common connective language that adds no factual content does not require a citation.\" ], \"dependencies\": { \"requiredBeforeCriterionIds\": [], \"impliesCriterionIds\": [] }, \"precedence\": { \"overridesCriterionIds\": [], \"overriddenByCriterionIds\": [] }, \"defaultSeverity\": \"major\", \"defaultDecisionEffect\": \"rework\", \"tags\": [ \"grounding\", \"evidence\", \"hard-quality\" ] }, { \"criterionId\": \"C3-warrant\", \"name\": \"Conclusion strength matches evidence\", \"requirement\": \"Distinguish association from causation and do not assert primacy unless the evidence warrants it.\", \"intent\": \"Assess whether the inference, not merely the cited facts, is justified.\", \"criterionType\": \"inferential_warrant\", \"scope\": \"claim\", \"subjectivity\": \"bounded_judgment\", \"instructionRefs\": [ { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 2\", \"quotedText\": \"State the strongest conclusion supported by the evidence.\" }, { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 3\", \"quotedText\": \"Distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim.\" } ], \"applicability\": { \"ruleText\": \"Applies to causal and comparative conclusions.\", \"condition\": { \"kind\": \"always\" } }, \"proofPolicy\": { \"positiveProofRequired\": true, \"positiveProofStandard\": \"The response explicitly calibrates its causal conclusion to study design, uncertainty, and counterevidence.\", \"failureCondition\": \"The response infers causation or primacy from evidence that supports only association or an uncertain effect.\", \"omissionCondition\": \"The response fails to distinguish observational association from causal evidence.\", \"passOnNoViolation\": false, \"allowPartialCredit\": true, \"allowCompensation\": false }, \"evidencePolicy\": { \"allowedEvidenceClassIds\": [ \"supplied-study\" ], \"minimumAuthorityRank\": 3, \"citationRequired\": true, \"externalVerificationMode\": \"forbidden\", \"deterministicValidatorIds\": [] }, \"anchors\": [ { \"anchorId\": \"C3-pass-anchor\", \"label\": \"Proportionate inference\", \"example\": \"X may contribute, but the evidence does not establish that it is the primary cause.\", \"expectedVerdict\": \"met\", \"notes\": \"Uses appropriate uncertainty and distinguishes contribution from primacy.\" }, { \"anchorId\": \"C3-fail-anchor\", \"label\": \"Causal overreach\", \"example\": \"The 18% difference proves X is the primary cause.\", \"expectedVerdict\": \"not_met\", \"notes\": \"Treats observational association as proof of causation and primacy.\" } ], \"validAlternatives\": [ \"A cautious conclusion that evidence is insufficient to determine primacy.\" ], \"exceptions\": [], \"dependencies\": { \"requiredBeforeCriterionIds\": [ \"C2-grounding\" ], \"impliesCriterionIds\": [] }, \"precedence\": { \"overridesCriterionIds\": [], \"overriddenByCriterionIds\": [] }, \"defaultSeverity\": \"major\", \"defaultDecisionEffect\": \"rework\", \"tags\": [ \"reasoning\", \"warrant\", \"hard-quality\" ] }, { \"criterionId\": \"C4-completeness\", \"name\": \"Addresses both studies\", \"requirement\": \"Use and synthesize both E1 and E2.\", \"intent\": \"Prevent selective use of convenient evidence.\", \"criterionType\": \"completeness\", \"scope\": \"attempt\", \"subjectivity\": \"objective\", \"instructionRefs\": [ { \"instructionId\": \"instructions-main\", \"locator\": \"sentence 3\", \"quotedText\": \"Distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim.\" } ], \"applicability\": { \"ruleText\": \"Applies when both studies are supplied.\", \"condition\": { \"kind\": \"always\" } }, \"proofPolicy\": { \"positiveProofRequired\": true, \"positiveProofStandard\": \"The attempt materially describes the relevant contribution and limitation of E1 and E2.\", \"failureCondition\": \"One study is ignored or materially misrepresented.\", \"omissionCondition\": \"No substantive treatment of E1 or E2 appears.\", \"passOnNoViolation\": false, \"allowPartialCredit\": true, \"allowCompensation\": false }, \"evidencePolicy\": { \"allowedEvidenceClassIds\": [ \"supplied-study\" ], \"minimumAuthorityRank\": 3, \"citationRequired\": true, \"externalVerificationMode\": \"forbidden\", \"deterministicValidatorIds\": [ \"required-evidence-id-validator\" ] }, \"anchors\": [ { \"anchorId\": \"C4-pass-anchor\", \"label\": \"Both sources synthesized\", \"example\": \"E1 shows an association but cannot identify causation; E2 is randomized but inconclusive.\", \"expectedVerdict\": \"met\", \"notes\": \"Captures the evidentiary role of both studies.\" } ], \"validAlternatives\": [], \"exceptions\": [], \"dependencies\": { \"requiredBeforeCriterionIds\": [], \"impliesCriterionIds\": [] }, \"precedence\": { \"overridesCriterionIds\": [], \"overriddenByCriterionIds\": [] }, \"defaultSeverity\": \"major\", \"defaultDecisionEffect\": \"rework\", \"tags\": [ \"completeness\", \"positive-proof\" ] }, { \"criterionId\": \"C5-annotation-fidelity\", \"name\": \"Reviewer label and rationale faithfully evaluate the attempt\", \"requirement\": \"The reviewer must apply the written criteria, cite relevant evidence, and assign proportionate severity.\", \"intent\": \"Audit the human annotation separately from the attempt.\", \"criterionType\": \"annotation_fidelity\", \"scope\": \"annotation\", \"subjectivity\": \"bounded_judgment\", \"instructionRefs\": [ { \"instructionId\": \"instructions-main\", \"locator\": \"entire document\", \"quotedText\": \"Use only the supplied evidence. State the strongest conclusion supported by the evidence. Distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim. Do not force a binary conclusion when the evidence is inconclusive.\" } ], \"applicability\": { \"ruleText\": \"Applies only when a human review is present.\", \"condition\": { \"kind\": \"predicate\", \"fieldPath\": \"case.humanReview\", \"operator\": \"exists\", \"value\": true } }, \"proofPolicy\": { \"positiveProofRequired\": true, \"positiveProofStandard\": \"The reviewer label, criterion attribution, evidence, and severity are mutually consistent and aligned with the contract.\", \"failureCondition\": \"The review applies an unstated preference, misattributes the issue, cites irrelevant evidence, or assigns an unsupported disposition.\", \"omissionCondition\": \"The rationale omits the decisive criterion or evidence.\", \"passOnNoViolation\": false, \"allowPartialCredit\": true, \"allowCompensation\": false }, \"evidencePolicy\": { \"allowedEvidenceClassIds\": [ \"supplied-study\" ], \"minimumAuthorityRank\": 3, \"citationRequired\": false, \"externalVerificationMode\": \"forbidden\", \"deterministicValidatorIds\": [] }, \"anchors\": [ { \"anchorId\": \"C5-fail-anchor\", \"label\": \"Unstated binary preference\", \"example\": \"Fail because the answer did not choose yes or no, even though the instructions allow an inconclusive conclusion.\", \"expectedVerdict\": \"not_met\", \"notes\": \"The reviewer substitutes a preference that contradicts the written contract.\" } ], \"validAlternatives\": [], \"exceptions\": [], \"dependencies\": { \"requiredBeforeCriterionIds\": [ \"C1-direct-response\" ], \"impliesCriterionIds\": [] }, \"precedence\": { \"overridesCriterionIds\": [], \"overriddenByCriterionIds\": [] }, \"defaultSeverity\": \"major\", \"defaultDecisionEffect\": \"accept_with_note\", \"tags\": [ \"annotation\", \"reviewer-audit\" ] } ], \"decisionPolicy\": { \"policyVersion\": \"0.1.0\", \"allowedActions\": [ \"accept\", \"accept_with_notes\", \"rework\", \"reject\", \"escalate\" ], \"rules\": [ { \"ruleId\": \"R1-critical-hard-failure\", \"priority\": 1, \"description\": \"Reject a calibrated critical hard-quality failure.\", \"match\": { \"criterionTags\": [ \"hard-quality\" ], \"verdicts\": [ \"not_met\" ], \"severities\": [ \"critical\" ], \"minimumCalibratedConfidence\": 0.95 }, \"action\": \"reject\", \"terminal\": true, \"explanationTemplate\": \"A critical hard-quality criterion failed at high calibrated confidence.\", \"target\": \"attempt\" }, { \"ruleId\": \"R2-major-repairable-failure\", \"priority\": 2, \"description\": \"Request rework for a clear major, repairable failure.\", \"match\": { \"verdicts\": [ \"not_met\" ], \"severities\": [ \"major\" ], \"minimumCalibratedConfidence\": 0.85 }, \"action\": \"rework\", \"terminal\": false, \"explanationTemplate\": \"At least one major criterion is not met and has a specific repair.\", \"target\": \"attempt\" }, { \"ruleId\": \"R3-decision-changing-uncertainty\", \"priority\": 3, \"description\": \"Escalate unresolved findings capable of changing the final action.\", \"match\": { \"hasUnresolvedFindings\": true }, \"action\": \"escalate\", \"terminal\": false, \"explanationTemplate\": \"A decision-changing issue remains unresolved.\", \"target\": \"attempt\" }, { \"ruleId\": \"R4-accept-clean\", \"priority\": 10, \"description\": \"Accept when all applicable attempt-quality criteria are met and no major unresolved issue remains.\", \"match\": { \"criterionIds\": [ \"C1-direct-response\", \"C2-grounding\", \"C3-warrant\", \"C4-completeness\" ], \"verdicts\": [ \"met\" ], \"minimumCalibratedConfidence\": 0.85, \"hasUnresolvedFindings\": false }, \"action\": \"accept\", \"terminal\": true, \"explanationTemplate\": \"All applicable attempt-quality criteria are affirmatively satisfied.\", \"target\": \"attempt\" }, { \"target\": \"annotation\", \"ruleId\": \"R5-annotation-major-failure\", \"priority\": 1, \"description\": \"Request correction when annotation fidelity has a clear major failure.\", \"match\": { \"criterionIds\": [ \"C5-annotation-fidelity\" ], \"verdicts\": [ \"not_met\" ], \"severities\": [ \"major\" ], \"minimumCalibratedConfidence\": 0.85 }, \"action\": \"rework\", \"terminal\": true, \"explanationTemplate\": \"The human annotation is not faithful to the attempt and contract.\" }, { \"target\": \"annotation\", \"ruleId\": \"R6-annotation-clean\", \"priority\": 10, \"description\": \"Accept a faithful human annotation.\", \"match\": { \"criterionIds\": [ \"C5-annotation-fidelity\" ], \"verdicts\": [ \"met\" ], \"minimumCalibratedConfidence\": 0.85, \"hasUnresolvedFindings\": false }, \"action\": \"accept\", \"terminal\": true, \"explanationTemplate\": \"The annotation is faithful and adequately supported.\" }, { \"target\": \"composite_case\", \"ruleId\": \"R7-composite-annotation-rework\", \"priority\": 1, \"description\": \"Rework the composite datum when the attempt is acceptable but its required annotation is not.\", \"match\": { \"criterionIds\": [ \"C5-annotation-fidelity\" ], \"verdicts\": [ \"not_met\" ], \"severities\": [ \"major\" ], \"minimumCalibratedConfidence\": 0.85 }, \"action\": \"rework\", \"terminal\": true, \"explanationTemplate\": \"The attempt can be retained, but the annotation must be corrected before the composite datum is admitted.\" } ], \"defaultAction\": \"escalate\", \"tieBreakOrder\": [ \"reject\", \"escalate\", \"rework\", \"accept_with_notes\", \"accept\" ], \"confidencePolicy\": { \"requiresCalibrationForActions\": [ \"accept\", \"reject\" ], \"uncalibratedAction\": \"escalate\", \"minimumCalibrationCasesPerSlice\": 50 } }, \"humanReviewPolicy\": { \"maxInteractionsPerCase\": 1, \"onlyAskIfDecisionChanging\": true, \"hideProvisionalDecision\": true, \"requireAmbiguityOption\": true, \"allowedQuestionTypes\": [ \"applicability\", \"instruction_precedence\", \"evidence_authority\", \"inference_permission\", \"severity\", \"valid_alternative\", \"rubric_ambiguity\", \"domain_expertise\" ], \"expertiseRoutingRules\": [ \"Route causal-inference disputes to a reviewer qualified in study design.\" ] }, \"feedbackPolicy\": { \"audiences\": [ \"attempter\", \"reviewer\", \"project_owner\" ], \"maxItemsPerAudience\": 3, \"includeValidatedStrengths\": true, \"validateRepairs\": true, \"priorityOrder\": [ \"decisive\", \"critical\", \"major\", \"minor\", \"strength\" ] }, \"securityPolicy\": { \"treatCandidateContentAsUntrusted\": true, \"ignoreEmbeddedEvaluatorInstructions\": true, \"requireStructuredOutputs\": true, \"allowCandidateToOverrideRubric\": false, \"redactionRules\": [ \"Remove direct personal identifiers from reviewer analytics exports.\" ] }, \"auditPolicy\": { \"retainExactSourceSnapshots\": true, \"retainPromptVersions\": true, \"retainModelRunMetadata\": true, \"retainHumanResolutions\": true, \"replayRequired\": true, \"retentionDays\": 365 }, \"metadata\": { \"example\": true }, \"decisionTargets\": [ \"attempt\", \"annotation\", \"composite_case\" ] } "}, {"slug": "foundation-schemas-benchmark-item-schema-json", "title": "JSON Schema - Benchmark Item", "category": "Foundation schemas", "summary": "Runtime schema for adjudicated truth, acceptable actions, ambiguity, challenge attributes, and leakage controls.", "tags": ["schema", "runtime validation"], "view": "documents/view/foundation-schemas-benchmark-item-schema-json.html", "raw": "documents/raw/foundation/schemas/benchmark-item.schema.json", "text": "{ \"$schema\": \"https://json-schema.org/draft/2020-12/schema\", \"$id\": \"https://autoqa.example/schemas/benchmark-item.schema.json\", \"title\": \"AutoQA Benchmark Item\", \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"documentType\", \"schemaVersion\", \"benchmarkItemId\", \"benchmarkVersion\", \"split\", \"contractRef\", \"caseRef\", \"sliceTags\", \"challengeAttributes\", \"goldStandard\", \"adjudication\", \"leakageControl\", \"weight\" ], \"properties\": { \"documentType\": { \"const\": \"benchmark_item\" }, \"schemaVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"benchmarkItemId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"benchmarkVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"split\": { \"type\": \"string\", \"enum\": [ \"development\", \"calibration\", \"test\", \"challenge\", \"shadow\", \"held_out_project\" ] }, \"contractRef\": { \"$ref\": \"common.schema.json#/$defs/ContractRef\" }, \"caseRef\": { \"$ref\": \"common.schema.json#/$defs/CaseRef\" }, \"sliceTags\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\" } }, \"challengeAttributes\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\", \"enum\": [ \"strong_positive\", \"subtle_valid_alternative\", \"polished_falsehood\", \"unsupported_rationale\", \"citation_mismatch\", \"partial_support\", \"required_content_omission\", \"severity_edge\", \"instruction_conflict\", \"ambiguous_case\", \"prompt_injection\", \"rubric_parroting\", \"correct_label_wrong_reason\", \"wrong_label_correct_observation\", \"long_context\", \"domain_expertise\", \"position_bias_probe\" ] } }, \"goldStandard\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"criterionGold\", \"acceptableDecisions\", \"ambiguityStatus\" ], \"properties\": { \"criterionGold\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"criterionId\", \"applicability\", \"acceptableVerdicts\", \"severity\", \"positiveEvidenceSpans\", \"negativeEvidenceSpans\", \"omissionDescriptions\", \"notes\" ], \"properties\": { \"criterionId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"applicability\": { \"type\": \"string\", \"enum\": [ \"applies\", \"does_not_apply\", \"uncertain\" ] }, \"acceptableVerdicts\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/CriterionVerdict\" } }, \"severity\": { \"$ref\": \"common.schema.json#/$defs/Severity\" }, \"positiveEvidenceSpans\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/SpanRef\" } }, \"negativeEvidenceSpans\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/SpanRef\" } }, \"omissionDescriptions\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"notes\": { \"type\": \"string\" } } } }, \"annotationGold\": { \"type\": \"object\", \"additionalProperties\": false, \"properties\": { \"acceptableLabelSupport\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"expectedDisagreementTypes\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"notes\": { \"type\": \"string\" } } }, \"ambiguityStatus\": { \"type\": \"string\", \"enum\": [ \"clear\", \"reasonable_disagreement\", \"rubric_ambiguous\", \"insufficient_evidence\" ] }, \"validAlternativeNotes\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"acceptableDecisions\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"target\", \"acceptableActions\" ], \"properties\": { \"target\": { \"$ref\": \"common.schema.json#/$defs/DecisionTarget\" }, \"acceptableActions\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" } } } } } } }, \"adjudication\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"method\", \"adjudicatorCount\", \"domainExpertCount\", \"blindToAutoQA\", \"blindToReviewerLabel\", \"disagreementTypes\", \"completedAt\" ], \"properties\": { \"method\": { \"type\": \"string\", \"enum\": [ \"single_expert\", \"independent_then_adjudicate\", \"consensus_panel\", \"latent_label_model\" ] }, \"adjudicatorCount\": { \"type\": \"integer\", \"minimum\": 1 }, \"domainExpertCount\": { \"type\": \"integer\", \"minimum\": 0 }, \"blindToAutoQA\": { \"type\": \"boolean\" }, \"blindToReviewerLabel\": { \"type\": \"boolean\" }, \"initialAgreementRate\": { \"type\": \"number\", \"minimum\": 0, \"maximum\": 1 }, \"disagreementTypes\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\" } }, \"adjudicationNotes\": { \"type\": \"string\" }, \"completedAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" } } }, \"leakageControl\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"allowedForPromptDevelopment\", \"heldOutProject\", \"firstFrozenAt\", \"accessPolicy\" ], \"properties\": { \"allowedForPromptDevelopment\": { \"type\": \"boolean\" }, \"heldOutProject\": { \"type\": \"boolean\" }, \"firstFrozenAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"accessPolicy\": { \"type\": \"string\", \"minLength\": 1 } } }, \"weight\": { \"type\": \"number\", \"exclusiveMinimum\": 0 }, \"exclusionReason\": { \"type\": \"string\" }, \"metadata\": { \"type\": \"object\", \"additionalProperties\": true } } } "}, {"slug": "foundation-schemas-case-input-schema-json", "title": "JSON Schema - Case Input", "category": "Foundation schemas", "summary": "Runtime schema for task, evidence bundle, attempt, human review, references, and provenance.", "tags": ["schema", "runtime validation"], "view": "documents/view/foundation-schemas-case-input-schema-json.html", "raw": "documents/raw/foundation/schemas/case-input.schema.json", "text": "{ \"$schema\": \"https://json-schema.org/draft/2020-12/schema\", \"$id\": \"https://autoqa.example/schemas/case-input.schema.json\", \"title\": \"AutoQA Case Input\", \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"documentType\", \"schemaVersion\", \"caseId\", \"createdAt\", \"contractRef\", \"task\", \"evidenceBundle\", \"attempt\", \"caseTags\" ], \"properties\": { \"documentType\": { \"const\": \"case_input\" }, \"schemaVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"caseId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"createdAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"contractRef\": { \"$ref\": \"common.schema.json#/$defs/ContractRef\" }, \"task\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"taskId\", \"taskType\", \"prompt\", \"contentHash\" ], \"properties\": { \"taskId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"taskType\": { \"type\": \"string\", \"minLength\": 1 }, \"prompt\": { \"type\": \"string\", \"minLength\": 1 }, \"taskInstructions\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"expectedOutputFormat\": { \"type\": \"string\" }, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" }, \"metadata\": { \"type\": \"object\", \"additionalProperties\": true } } }, \"evidenceBundle\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"evidenceId\", \"evidenceClassId\", \"title\", \"content\", \"contentHash\", \"authorityRank\", \"permitted\", \"capturedAt\" ], \"properties\": { \"evidenceId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"evidenceClassId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"title\": { \"type\": \"string\", \"minLength\": 1 }, \"content\": { \"type\": \"string\", \"minLength\": 1 }, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" }, \"sourceUri\": { \"type\": \"string\", \"format\": \"uri\" }, \"authorityRank\": { \"type\": \"integer\", \"minimum\": 0 }, \"permitted\": { \"type\": \"boolean\" }, \"publishedAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"capturedAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"metadata\": { \"type\": \"object\", \"additionalProperties\": true } } } }, \"attempt\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"attemptId\", \"content\", \"contentHash\", \"submittedAt\" ], \"properties\": { \"attemptId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"authorId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"content\": { \"type\": \"string\" }, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" }, \"submittedAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"attachments\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"artifactId\", \"artifactType\", \"contentHash\" ], \"properties\": { \"artifactId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"artifactType\": { \"type\": \"string\" }, \"content\": { \"type\": \"string\" }, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" } } } }, \"toolTraces\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"traceId\", \"toolName\", \"input\", \"output\", \"contentHash\" ], \"properties\": { \"traceId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"toolName\": { \"type\": \"string\" }, \"input\": {}, \"output\": {}, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" } } } } } }, \"humanReview\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"annotationId\", \"reviewerId\", \"label\", \"reasonCodes\", \"rationale\", \"submittedAt\" ], \"properties\": { \"annotationId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"reviewerId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"label\": { \"type\": \"string\", \"minLength\": 1 }, \"score\": { \"type\": \"number\" }, \"reasonCodes\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"rationale\": { \"type\": \"string\" }, \"criterionAnnotations\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"criterionId\", \"verdict\", \"rationale\" ], \"properties\": { \"criterionId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"verdict\": { \"$ref\": \"common.schema.json#/$defs/CriterionVerdict\" }, \"severity\": { \"$ref\": \"common.schema.json#/$defs/Severity\" }, \"rationale\": { \"type\": \"string\" }, \"citedSpans\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/SpanRef\" } } } } }, \"reviewerConfidence\": { \"$ref\": \"common.schema.json#/$defs/Confidence\" }, \"submittedAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" } } }, \"referenceAnswers\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"referenceId\", \"role\", \"content\", \"contentHash\" ], \"properties\": { \"referenceId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"role\": { \"type\": \"string\", \"enum\": [ \"authoritative\", \"illustrative\", \"positive_anchor\", \"negative_anchor\" ] }, \"content\": { \"type\": \"string\" }, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" } } } }, \"declaredExpectedElements\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/ExpectedElement\" } }, \"caseTags\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\" } }, \"metadata\": { \"type\": \"object\", \"additionalProperties\": true } } } "}, {"slug": "foundation-schemas-common-schema-json", "title": "JSON Schema - Common Definitions", "category": "Foundation schemas", "summary": "Shared runtime definitions used by the project, case, evaluation, benchmark, and resolution schemas.", "tags": ["schema", "runtime validation"], "view": "documents/view/foundation-schemas-common-schema-json.html", "raw": "documents/raw/foundation/schemas/common.schema.json", "text": "{ \"$schema\": \"https://json-schema.org/draft/2020-12/schema\", \"$id\": \"https://autoqa.example/schemas/common.schema.json\", \"title\": \"AutoQA Common Definitions\", \"type\": \"object\", \"$defs\": { \"Identifier\": { \"type\": \"string\", \"pattern\": \"^[A-Za-z0-9][A-Za-z0-9._:-]{1,127}$\" }, \"SemVer\": { \"type\": \"string\", \"pattern\": \"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+(?:-[0-9A-Za-z.-]+)?(?:\\\\+[0-9A-Za-z.-]+)?$\" }, \"ContentHash\": { \"type\": \"string\", \"pattern\": \"^sha256:[a-f0-9]{64}$\" }, \"DateTime\": { \"type\": \"string\", \"format\": \"date-time\" }, \"ConfidenceBand\": { \"type\": \"string\", \"enum\": [ \"high\", \"medium\", \"low\", \"uncalibrated\" ] }, \"Severity\": { \"type\": \"string\", \"enum\": [ \"none\", \"advisory\", \"minor\", \"major\", \"critical\" ] }, \"OperationalAction\": { \"type\": \"string\", \"enum\": [ \"accept\", \"accept_with_notes\", \"rework\", \"reject\", \"escalate\" ] }, \"CriterionVerdict\": { \"type\": \"string\", \"enum\": [ \"met\", \"partially_met\", \"not_met\", \"ambiguous\", \"unknown\", \"not_applicable\" ] }, \"ContractRef\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"contractId\", \"contractVersion\", \"contentHash\" ], \"properties\": { \"contractId\": { \"$ref\": \"#/$defs/Identifier\" }, \"contractVersion\": { \"$ref\": \"#/$defs/SemVer\" }, \"contentHash\": { \"$ref\": \"#/$defs/ContentHash\" } } }, \"CaseRef\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"caseId\" ], \"properties\": { \"caseId\": { \"$ref\": \"#/$defs/Identifier\" }, \"contentHash\": { \"$ref\": \"#/$defs/ContentHash\" } } }, \"SpanRef\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"artifactType\", \"artifactId\", \"quotedText\" ], \"properties\": { \"artifactType\": { \"type\": \"string\", \"enum\": [ \"task\", \"evidence\", \"attempt\", \"human_review\", \"reference\", \"tool_trace\" ] }, \"artifactId\": { \"$ref\": \"#/$defs/Identifier\" }, \"startChar\": { \"type\": \"integer\", \"minimum\": 0 }, \"endChar\": { \"type\": \"integer\", \"minimum\": 0 }, \"quotedText\": { \"type\": \"string\", \"minLength\": 1 }, \"locator\": { \"type\": \"string\" } }, \"allOf\": [ { \"if\": { \"required\": [ \"startChar\" ] }, \"then\": { \"required\": [ \"endChar\" ] } }, { \"if\": { \"required\": [ \"endChar\" ] }, \"then\": { \"required\": [ \"startChar\" ] } } ] }, \"Confidence\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"band\" ], \"properties\": { \"rawScore\": { \"type\": \"number\", \"minimum\": 0, \"maximum\": 1 }, \"calibratedScore\": { \"type\": \"number\", \"minimum\": 0, \"maximum\": 1 }, \"band\": { \"$ref\": \"#/$defs/ConfidenceBand\" }, \"calibrationProfileId\": { \"$ref\": \"#/$defs/Identifier\" }, \"rationale\": { \"type\": \"string\" } }, \"allOf\": [ { \"if\": { \"properties\": { \"band\": { \"const\": \"uncalibrated\" } }, \"required\": [ \"band\" ] }, \"then\": { \"not\": { \"anyOf\": [ { \"required\": [ \"calibratedScore\" ] }, { \"required\": [ \"calibrationProfileId\" ] } ] } } }, { \"if\": { \"properties\": { \"band\": { \"enum\": [ \"high\", \"medium\", \"low\" ] } }, \"required\": [ \"band\" ] }, \"then\": { \"required\": [ \"calibratedScore\", \"calibrationProfileId\" ] } } ] }, \"GenerationSource\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"kind\", \"componentId\" ], \"properties\": { \"kind\": { \"type\": \"string\", \"enum\": [ \"deterministic_validator\", \"llm_judge\", \"human\", \"hybrid\" ] }, \"componentId\": { \"$ref\": \"#/$defs/Identifier\" }, \"modelRunId\": { \"$ref\": \"#/$defs/Identifier\" }, \"humanActorId\": { \"$ref\": \"#/$defs/Identifier\" } } }, \"Condition\": { \"oneOf\": [ { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"kind\" ], \"properties\": { \"kind\": { \"const\": \"always\" } } }, { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"kind\", \"fieldPath\", \"operator\" ], \"properties\": { \"kind\": { \"const\": \"predicate\" }, \"fieldPath\": { \"type\": \"string\", \"minLength\": 1 }, \"operator\": { \"type\": \"string\", \"enum\": [ \"equals\", \"not_equals\", \"in\", \"not_in\", \"contains\", \"exists\", \"greater_than\", \"greater_than_or_equal\", \"less_than\", \"less_than_or_equal\", \"matches_regex\" ] }, \"value\": {} } }, { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"kind\", \"conditions\" ], \"properties\": { \"kind\": { \"const\": \"all\" }, \"conditions\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"$ref\": \"#/$defs/Condition\" } } } }, { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"kind\", \"conditions\" ], \"properties\": { \"kind\": { \"const\": \"any\" }, \"conditions\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"$ref\": \"#/$defs/Condition\" } } } }, { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"kind\", \"condition\" ], \"properties\": { \"kind\": { \"const\": \"not\" }, \"condition\": { \"$ref\": \"#/$defs/Condition\" } } } ] }, \"Finding\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"findingId\", \"findingType\", \"summary\", \"materiality\", \"severity\", \"confidence\", \"status\", \"source\" ], \"properties\": { \"findingId\": { \"$ref\": \"#/$defs/Identifier\" }, \"findingType\": { \"type\": \"string\", \"enum\": [ \"positive\", \"negative\", \"omission\", \"ambiguity\", \"counterevidence\", \"integrity\" ] }, \"criterionId\": { \"$ref\": \"#/$defs/Identifier\" }, \"summary\": { \"type\": \"string\", \"minLength\": 1 }, \"detail\": { \"type\": \"string\" }, \"claimIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"expectedElementIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"targetSpans\": { \"type\": \"array\", \"items\": { \"$ref\": \"#/$defs/SpanRef\" } }, \"evidenceIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"relationship\": { \"type\": \"string\", \"enum\": [ \"directly_entails\", \"reasonably_supports\", \"weakly_supports\", \"contradicts\", \"irrelevant\", \"insufficient\", \"not_applicable\" ] }, \"materiality\": { \"type\": \"string\", \"enum\": [ \"incidental\", \"supporting\", \"central\", \"decisive\" ] }, \"severity\": { \"$ref\": \"#/$defs/Severity\" }, \"confidence\": { \"$ref\": \"#/$defs/Confidence\" }, \"status\": { \"type\": \"string\", \"enum\": [ \"observed\", \"challenged\", \"confirmed\", \"retracted\" ] }, \"source\": { \"$ref\": \"#/$defs/GenerationSource\" } } }, \"Claim\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"claimId\", \"proposition\", \"claimType\", \"importance\", \"explicitness\", \"attemptSpans\" ], \"properties\": { \"claimId\": { \"$ref\": \"#/$defs/Identifier\" }, \"proposition\": { \"type\": \"string\", \"minLength\": 1 }, \"claimType\": { \"type\": \"string\", \"enum\": [ \"factual\", \"interpretive\", \"predictive\", \"normative\", \"preference\", \"procedural\" ] }, \"importance\": { \"type\": \"string\", \"enum\": [ \"incidental\", \"supporting\", \"central\" ] }, \"explicitness\": { \"type\": \"string\", \"enum\": [ \"explicit\", \"implied\" ] }, \"attemptSpans\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"$ref\": \"#/$defs/SpanRef\" } }, \"qualifiers\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"citedEvidenceIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } } } }, \"ExpectedElement\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"elementId\", \"criterionId\", \"description\", \"required\", \"importance\", \"coverage\" ], \"properties\": { \"elementId\": { \"$ref\": \"#/$defs/Identifier\" }, \"criterionId\": { \"$ref\": \"#/$defs/Identifier\" }, \"description\": { \"type\": \"string\", \"minLength\": 1 }, \"required\": { \"type\": \"boolean\" }, \"importance\": { \"type\": \"string\", \"enum\": [ \"supporting\", \"central\", \"decisive\" ] }, \"acceptableAlternatives\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"observedSpans\": { \"type\": \"array\", \"items\": { \"$ref\": \"#/$defs/SpanRef\" } }, \"coverage\": { \"type\": \"string\", \"enum\": [ \"complete\", \"partial\", \"missing\", \"not_applicable\", \"unknown\" ] } } }, \"EvidenceRelationship\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"relationshipId\", \"subjectType\", \"subjectId\", \"evidenceId\", \"relationship\", \"sourceQuality\", \"temporalValidity\", \"confidence\" ], \"properties\": { \"relationshipId\": { \"$ref\": \"#/$defs/Identifier\" }, \"subjectType\": { \"type\": \"string\", \"enum\": [ \"claim\", \"expected_element\", \"finding\" ] }, \"subjectId\": { \"$ref\": \"#/$defs/Identifier\" }, \"evidenceId\": { \"$ref\": \"#/$defs/Identifier\" }, \"relationship\": { \"type\": \"string\", \"enum\": [ \"directly_entails\", \"reasonably_supports\", \"weakly_supports\", \"contradicts\", \"irrelevant\", \"insufficient\" ] }, \"sourceQuality\": { \"type\": \"string\", \"enum\": [ \"high\", \"acceptable\", \"weak\", \"disallowed\", \"unknown\" ] }, \"temporalValidity\": { \"type\": \"string\", \"enum\": [ \"valid\", \"stale\", \"not_time_sensitive\", \"unknown\" ] }, \"evidenceSpans\": { \"type\": \"array\", \"items\": { \"$ref\": \"#/$defs/SpanRef\" } }, \"confidence\": { \"$ref\": \"#/$defs/Confidence\" }, \"notes\": { \"type\": \"string\" } } }, \"CriterionAssessment\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"criterionId\", \"applicability\", \"positiveFindingIds\", \"negativeFindingIds\", \"omissionFindingIds\", \"counterEvidenceFindingIds\", \"unresolvedFindingIds\", \"groundingStatus\", \"truthStatus\", \"warrantStatus\", \"relevanceStatus\", \"verdict\", \"severity\", \"confidence\", \"decisionEffect\", \"rationale\" ], \"properties\": { \"criterionId\": { \"$ref\": \"#/$defs/Identifier\" }, \"applicability\": { \"type\": \"string\", \"enum\": [ \"applies\", \"does_not_apply\", \"uncertain\" ] }, \"positiveFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"negativeFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"omissionFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"counterEvidenceFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"unresolvedFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"groundingStatus\": { \"type\": \"string\", \"enum\": [ \"grounded\", \"partially_grounded\", \"ungrounded\", \"not_applicable\", \"unknown\" ] }, \"truthStatus\": { \"type\": \"string\", \"enum\": [ \"verified_true\", \"verified_false\", \"mixed\", \"not_evaluated\", \"not_applicable\", \"unknown\" ] }, \"warrantStatus\": { \"type\": \"string\", \"enum\": [ \"justified\", \"overstated\", \"invalid\", \"not_applicable\", \"unknown\" ] }, \"relevanceStatus\": { \"type\": \"string\", \"enum\": [ \"relevant\", \"partially_relevant\", \"irrelevant\", \"not_applicable\", \"unknown\" ] }, \"verdict\": { \"$ref\": \"#/$defs/CriterionVerdict\" }, \"severity\": { \"$ref\": \"#/$defs/Severity\" }, \"confidence\": { \"$ref\": \"#/$defs/Confidence\" }, \"decisionEffect\": { \"type\": \"string\", \"enum\": [ \"none\", \"accept_with_note\", \"rework\", \"reject\", \"escalate\" ] }, \"rationale\": { \"type\": \"string\", \"minLength\": 1 } } }, \"GlobalAssessment\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"axisId\", \"verdict\", \"findingIds\", \"confidence\", \"rationale\" ], \"properties\": { \"axisId\": { \"$ref\": \"#/$defs/Identifier\" }, \"verdict\": { \"$ref\": \"#/$defs/CriterionVerdict\" }, \"findingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"score\": { \"type\": \"number\" }, \"confidence\": { \"$ref\": \"#/$defs/Confidence\" }, \"rationale\": { \"type\": \"string\" } } }, \"PolicyTraceEntry\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"ruleId\", \"matched\", \"result\" ], \"properties\": { \"ruleId\": { \"$ref\": \"#/$defs/Identifier\" }, \"matched\": { \"type\": \"boolean\" }, \"inputs\": { \"type\": \"object\", \"additionalProperties\": true }, \"result\": { \"type\": \"string\" } } }, \"Decision\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"target\", \"action\", \"policyVersion\", \"triggeredRuleIds\", \"decisiveCriterionIds\", \"unresolvedFindingIds\", \"confidence\", \"policyTrace\", \"rationale\", \"decidedAt\" ], \"properties\": { \"action\": { \"$ref\": \"#/$defs/OperationalAction\" }, \"policyVersion\": { \"$ref\": \"#/$defs/SemVer\" }, \"triggeredRuleIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"decisiveCriterionIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"unresolvedFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"confidence\": { \"$ref\": \"#/$defs/Confidence\" }, \"policyTrace\": { \"type\": \"array\", \"items\": { \"$ref\": \"#/$defs/PolicyTraceEntry\" } }, \"rationale\": { \"type\": \"string\", \"minLength\": 1 }, \"decidedAt\": { \"$ref\": \"#/$defs/DateTime\" }, \"target\": { \"$ref\": \"#/$defs/DecisionTarget\" } } }, \"ResolutionOption\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"optionId\", \"label\" ], \"properties\": { \"optionId\": { \"$ref\": \"#/$defs/Identifier\" }, \"label\": { \"type\": \"string\", \"minLength\": 1 }, \"description\": { \"type\": \"string\" }, \"effectSummary\": { \"type\": \"string\" } } }, \"HumanResolutionRequest\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"requestId\", \"triggerCriterionIds\", \"questionType\", \"question\", \"contextFindingIds\", \"options\", \"decisionChanging\", \"hideProvisionalDecision\", \"status\", \"createdAt\" ], \"properties\": { \"requestId\": { \"$ref\": \"#/$defs/Identifier\" }, \"triggerCriterionIds\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"questionType\": { \"type\": \"string\", \"enum\": [ \"applicability\", \"instruction_precedence\", \"evidence_authority\", \"inference_permission\", \"severity\", \"valid_alternative\", \"rubric_ambiguity\", \"domain_expertise\" ] }, \"question\": { \"type\": \"string\", \"minLength\": 1 }, \"contextFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"options\": { \"type\": \"array\", \"minItems\": 2, \"items\": { \"$ref\": \"#/$defs/ResolutionOption\" } }, \"decisionChanging\": { \"type\": \"boolean\" }, \"hideProvisionalDecision\": { \"type\": \"boolean\" }, \"requiredExpertiseTags\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"status\": { \"type\": \"string\", \"enum\": [ \"pending\", \"answered\", \"cancelled\" ] }, \"createdAt\": { \"$ref\": \"#/$defs/DateTime\" } } }, \"HumanResolutionResponse\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"requestId\", \"selectedOptionId\", \"responderId\", \"respondedAt\" ], \"properties\": { \"requestId\": { \"$ref\": \"#/$defs/Identifier\" }, \"selectedOptionId\": { \"$ref\": \"#/$defs/Identifier\" }, \"responderId\": { \"$ref\": \"#/$defs/Identifier\" }, \"rationale\": { \"type\": \"string\" }, \"respondedAt\": { \"$ref\": \"#/$defs/DateTime\" } } }, \"HumanResolutionInteraction\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"request\", \"decisionTarget\" ], \"properties\": { \"request\": { \"$ref\": \"#/$defs/HumanResolutionRequest\" }, \"response\": { \"$ref\": \"#/$defs/HumanResolutionResponse\" }, \"decisionBefore\": { \"$ref\": \"#/$defs/OperationalAction\" }, \"decisionAfter\": { \"$ref\": \"#/$defs/OperationalAction\" }, \"changedDecision\": { \"type\": \"boolean\" }, \"decisionTarget\": { \"$ref\": \"#/$defs/DecisionTarget\" } } }, \"FeedbackItem\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"feedbackId\", \"audience\", \"feedbackType\", \"findingIds\", \"message\", \"priority\", \"repairValidationStatus\" ], \"properties\": { \"feedbackId\": { \"$ref\": \"#/$defs/Identifier\" }, \"audience\": { \"type\": \"string\", \"enum\": [ \"attempter\", \"reviewer\", \"project_owner\" ] }, \"criterionId\": { \"$ref\": \"#/$defs/Identifier\" }, \"feedbackType\": { \"type\": \"string\", \"enum\": [ \"strength\", \"defect\", \"omission\", \"ambiguity\" ] }, \"findingIds\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"message\": { \"type\": \"string\", \"minLength\": 1 }, \"minimalRepair\": { \"type\": \"string\" }, \"priority\": { \"type\": \"integer\", \"minimum\": 1 }, \"repairValidationStatus\": { \"type\": \"string\", \"enum\": [ \"validated\", \"plausible_unvalidated\", \"not_applicable\" ] } } }, \"AnnotationAssessment\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"annotationPresent\", \"labelSupport\", \"rationaleSupport\", \"criterionAttribution\", \"evidenceAlignment\", \"severityAlignment\", \"omittedPositiveFindingIds\", \"omittedNegativeFindingIds\", \"unsupportedReviewerFindingIds\", \"disagreementTypes\", \"confidence\", \"rationale\" ], \"properties\": { \"annotationPresent\": { \"type\": \"boolean\" }, \"labelSupport\": { \"type\": \"string\", \"enum\": [ \"supported\", \"partially_supported\", \"unsupported\", \"reasonable_disagreement\", \"rubric_ambiguous\", \"not_applicable\" ] }, \"rationaleSupport\": { \"type\": \"string\", \"enum\": [ \"supported\", \"contains_unsupported_reasoning\", \"does_not_support_label\", \"missing\", \"not_applicable\" ] }, \"criterionAttribution\": { \"type\": \"string\", \"enum\": [ \"correct\", \"partially_correct\", \"wrong_criterion\", \"unstated_preference\", \"not_applicable\" ] }, \"evidenceAlignment\": { \"type\": \"string\", \"enum\": [ \"aligned\", \"partially_aligned\", \"misquoted\", \"irrelevant\", \"missing\", \"not_applicable\" ] }, \"severityAlignment\": { \"type\": \"string\", \"enum\": [ \"appropriate\", \"too_severe\", \"too_lenient\", \"uncertain\", \"not_applicable\" ] }, \"omittedPositiveFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"omittedNegativeFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"unsupportedReviewerFindingIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"disagreementTypes\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\", \"enum\": [ \"none\", \"observation\", \"evidence_authority\", \"applicability\", \"criterion_interpretation\", \"inference\", \"threshold\", \"severity\", \"attention\", \"expertise\", \"reasonable_disagreement\", \"rubric_defect\" ] } }, \"confidence\": { \"$ref\": \"#/$defs/Confidence\" }, \"rationale\": { \"type\": \"string\" } } }, \"ModelRun\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"modelRunId\", \"role\", \"provider\", \"modelName\", \"promptTemplateId\", \"promptTemplateVersion\", \"promptTemplateHash\", \"startedAt\", \"completedAt\" ], \"properties\": { \"modelRunId\": { \"$ref\": \"#/$defs/Identifier\" }, \"role\": { \"type\": \"string\", \"enum\": [ \"claim_extractor\", \"evidence_retriever\", \"positive_prover\", \"negative_challenger\", \"coverage_auditor\", \"warrant_auditor\", \"criterion_adjudicator\", \"holistic_judge\", \"annotation_auditor\", \"feedback_generator\", \"repair_validator\", \"ensemble_aggregator\" ] }, \"provider\": { \"type\": \"string\", \"minLength\": 1 }, \"modelName\": { \"type\": \"string\", \"minLength\": 1 }, \"modelVersion\": { \"type\": \"string\" }, \"promptTemplateId\": { \"$ref\": \"#/$defs/Identifier\" }, \"promptTemplateVersion\": { \"$ref\": \"#/$defs/SemVer\" }, \"promptTemplateHash\": { \"$ref\": \"#/$defs/ContentHash\" }, \"parameters\": { \"type\": \"object\", \"additionalProperties\": true }, \"parentRunIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"#/$defs/Identifier\" } }, \"inputHash\": { \"$ref\": \"#/$defs/ContentHash\" }, \"outputHash\": { \"$ref\": \"#/$defs/ContentHash\" }, \"startedAt\": { \"$ref\": \"#/$defs/DateTime\" }, \"completedAt\": { \"$ref\": \"#/$defs/DateTime\" } } }, \"DecisionTarget\": { \"type\": \"string\", \"enum\": [ \"attempt\", \"annotation\", \"composite_case\", \"batch\" ] }, \"ValidatorRun\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"validatorRunId\", \"validatorId\", \"validatorVersion\", \"role\", \"status\", \"inputHash\", \"outputHash\", \"startedAt\", \"completedAt\" ], \"properties\": { \"validatorRunId\": { \"$ref\": \"#/$defs/Identifier\" }, \"validatorId\": { \"$ref\": \"#/$defs/Identifier\" }, \"validatorVersion\": { \"$ref\": \"#/$defs/SemVer\" }, \"role\": { \"type\": \"string\", \"enum\": [ \"schema_validation\", \"format_check\", \"citation_check\", \"calculation\", \"code_execution\", \"duplicate_detection\", \"integrity_check\", \"custom\" ] }, \"status\": { \"type\": \"string\", \"enum\": [ \"passed\", \"failed\", \"warning\", \"error\" ] }, \"parameters\": { \"type\": \"object\", \"additionalProperties\": true }, \"inputHash\": { \"$ref\": \"#/$defs/ContentHash\" }, \"outputHash\": { \"$ref\": \"#/$defs/ContentHash\" }, \"startedAt\": { \"$ref\": \"#/$defs/DateTime\" }, \"completedAt\": { \"$ref\": \"#/$defs/DateTime\" } } } } } "}, {"slug": "foundation-schemas-evaluation-output-schema-json", "title": "JSON Schema - Evaluation Output", "category": "Foundation schemas", "summary": "Runtime schema for claims, evidence relationships, findings, criterion assessments, decisions, annotation audit, and feedback.", "tags": ["schema", "runtime validation"], "view": "documents/view/foundation-schemas-evaluation-output-schema-json.html", "raw": "documents/raw/foundation/schemas/evaluation-output.schema.json", "text": "{ \"$schema\": \"https://json-schema.org/draft/2020-12/schema\", \"$id\": \"https://autoqa.example/schemas/evaluation-output.schema.json\", \"title\": \"AutoQA Evaluation Output\", \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"documentType\", \"schemaVersion\", \"evaluationId\", \"evaluationVersion\", \"caseRef\", \"contractRef\", \"createdAt\", \"status\", \"runManifest\", \"validatorManifest\", \"findings\", \"claims\", \"expectedElements\", \"evidenceRelationships\", \"criterionAssessments\", \"globalAssessments\", \"provisionalDecisions\", \"finalDecisions\", \"feedback\", \"audit\" ], \"properties\": { \"documentType\": { \"const\": \"evaluation_output\" }, \"schemaVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"evaluationId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"evaluationVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"caseRef\": { \"$ref\": \"common.schema.json#/$defs/CaseRef\" }, \"contractRef\": { \"$ref\": \"common.schema.json#/$defs/ContractRef\" }, \"createdAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"status\": { \"type\": \"string\", \"enum\": [ \"provisional\", \"awaiting_human\", \"final\", \"failed\" ] }, \"runManifest\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/ModelRun\" } }, \"integrityStatus\": { \"type\": \"string\", \"enum\": [ \"clean\", \"warning\", \"failed\", \"unknown\" ] }, \"findings\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/Finding\" } }, \"claims\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/Claim\" } }, \"expectedElements\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/ExpectedElement\" } }, \"evidenceRelationships\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/EvidenceRelationship\" } }, \"criterionAssessments\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/CriterionAssessment\" } }, \"globalAssessments\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/GlobalAssessment\" } }, \"annotationAssessment\": { \"$ref\": \"common.schema.json#/$defs/AnnotationAssessment\" }, \"humanResolution\": { \"$ref\": \"common.schema.json#/$defs/HumanResolutionInteraction\" }, \"feedback\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/FeedbackItem\" } }, \"audit\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"caseInputHash\", \"contractHash\", \"sourceSnapshotHashes\", \"replayable\", \"warnings\" ], \"properties\": { \"caseInputHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" }, \"contractHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" }, \"sourceSnapshotHashes\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" } }, \"replayable\": { \"type\": \"boolean\" }, \"warnings\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } } } }, \"provisionalDecisions\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"$ref\": \"common.schema.json#/$defs/Decision\" } }, \"finalDecisions\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"$ref\": \"common.schema.json#/$defs/Decision\" } }, \"validatorManifest\": { \"type\": \"array\", \"items\": { \"$ref\": \"common.schema.json#/$defs/ValidatorRun\" } } } } "}, {"slug": "foundation-schemas-human-resolution-packet-schema-json", "title": "JSON Schema - Human Resolution Packet", "category": "Foundation schemas", "summary": "Runtime schema for one bounded, decision-changing human question and its context.", "tags": ["schema", "runtime validation"], "view": "documents/view/foundation-schemas-human-resolution-packet-schema-json.html", "raw": "documents/raw/foundation/schemas/human-resolution-packet.schema.json", "text": "{ \"$schema\": \"https://json-schema.org/draft/2020-12/schema\", \"$id\": \"https://autoqa.example/schemas/human-resolution-packet.schema.json\", \"title\": \"AutoQA Human Resolution Packet\", \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"documentType\", \"schemaVersion\", \"caseRef\", \"contractRef\", \"contextFindings\", \"interaction\" ], \"properties\": { \"documentType\": { \"const\": \"human_resolution_packet\" }, \"schemaVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"caseRef\": { \"$ref\": \"common.schema.json#/$defs/CaseRef\" }, \"contractRef\": { \"$ref\": \"common.schema.json#/$defs/ContractRef\" }, \"interaction\": { \"$ref\": \"common.schema.json#/$defs/HumanResolutionInteraction\" }, \"contextFindings\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"$ref\": \"common.schema.json#/$defs/Finding\" } } } } "}, {"slug": "foundation-schemas-project-contract-schema-json", "title": "JSON Schema - Project Contract", "category": "Foundation schemas", "summary": "Runtime schema for approved project instructions, criteria, evidence policy, decision policy, and calibration requirements.", "tags": ["schema", "runtime validation"], "view": "documents/view/foundation-schemas-project-contract-schema-json.html", "raw": "documents/raw/foundation/schemas/project-contract.schema.json", "text": "{ \"$schema\": \"https://json-schema.org/draft/2020-12/schema\", \"$id\": \"https://autoqa.example/schemas/project-contract.schema.json\", \"title\": \"AutoQA Project Evaluation Contract\", \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"documentType\", \"schemaVersion\", \"contractId\", \"contractVersion\", \"projectName\", \"status\", \"createdAt\", \"instructionDocuments\", \"sourcePolicy\", \"criteria\", \"decisionTargets\", \"decisionPolicy\", \"humanReviewPolicy\", \"feedbackPolicy\", \"securityPolicy\", \"auditPolicy\" ], \"properties\": { \"documentType\": { \"const\": \"project_contract\" }, \"schemaVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"contractId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"contractVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"projectName\": { \"type\": \"string\", \"minLength\": 1 }, \"description\": { \"type\": \"string\" }, \"status\": { \"type\": \"string\", \"enum\": [ \"draft\", \"approved\", \"deprecated\", \"retired\" ] }, \"createdAt\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"effectiveFrom\": { \"$ref\": \"common.schema.json#/$defs/DateTime\" }, \"supersedesContractVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"owners\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"actorId\", \"role\" ], \"properties\": { \"actorId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"role\": { \"type\": \"string\", \"enum\": [ \"project_owner\", \"rubric_owner\", \"domain_expert\", \"qa_owner\" ] } } } }, \"instructionDocuments\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"instructionId\", \"title\", \"authority\", \"content\", \"contentHash\" ], \"properties\": { \"instructionId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"title\": { \"type\": \"string\", \"minLength\": 1 }, \"authority\": { \"type\": \"string\", \"enum\": [ \"primary\", \"supplemental\", \"example\", \"historical\" ] }, \"content\": { \"type\": \"string\", \"minLength\": 1 }, \"contentHash\": { \"$ref\": \"common.schema.json#/$defs/ContentHash\" }, \"precedenceRank\": { \"type\": \"integer\", \"minimum\": 1 } } } }, \"sourcePolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"worldMode\", \"parametricKnowledgePolicy\", \"externalRetrievalAllowed\", \"defaultCitationRequirement\", \"evidenceClasses\", \"conflictPolicy\", \"snapshotRequired\" ], \"properties\": { \"worldMode\": { \"type\": \"string\", \"enum\": [ \"closed_world\", \"open_world\", \"hybrid\" ] }, \"parametricKnowledgePolicy\": { \"type\": \"string\", \"enum\": [ \"never_evidence\", \"common_knowledge_only\", \"allowed_with_verification\" ] }, \"externalRetrievalAllowed\": { \"type\": \"boolean\" }, \"defaultCitationRequirement\": { \"type\": \"string\", \"enum\": [ \"none\", \"material_claims\", \"all_verifiable_claims\", \"project_defined\" ] }, \"snapshotRequired\": { \"type\": \"boolean\" }, \"conflictPolicy\": { \"type\": \"string\", \"minLength\": 1 }, \"freshnessRules\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"evidenceClasses\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"classId\", \"name\", \"permitted\", \"authorityRank\" ], \"properties\": { \"classId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"name\": { \"type\": \"string\", \"minLength\": 1 }, \"description\": { \"type\": \"string\" }, \"permitted\": { \"type\": \"boolean\" }, \"authorityRank\": { \"type\": \"integer\", \"minimum\": 0 } } } } } }, \"criteria\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"criterionId\", \"name\", \"requirement\", \"intent\", \"criterionType\", \"scope\", \"subjectivity\", \"instructionRefs\", \"applicability\", \"proofPolicy\", \"evidencePolicy\", \"anchors\", \"dependencies\", \"precedence\", \"defaultSeverity\", \"defaultDecisionEffect\", \"tags\" ], \"properties\": { \"criterionId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"name\": { \"type\": \"string\", \"minLength\": 1 }, \"requirement\": { \"type\": \"string\", \"minLength\": 1 }, \"intent\": { \"type\": \"string\", \"minLength\": 1 }, \"criterionType\": { \"type\": \"string\", \"enum\": [ \"required_presence\", \"prohibited_presence\", \"conditional\", \"factual\", \"grounding\", \"inferential_warrant\", \"comparative\", \"completeness\", \"holistic_quality\", \"mechanically_verifiable\", \"annotation_fidelity\" ] }, \"scope\": { \"type\": \"string\", \"enum\": [ \"attempt\", \"claim\", \"citation\", \"annotation\", \"global\", \"batch\" ] }, \"subjectivity\": { \"type\": \"string\", \"enum\": [ \"objective\", \"bounded_judgment\", \"holistic_expert\" ] }, \"instructionRefs\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"instructionId\", \"locator\" ], \"properties\": { \"instructionId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"locator\": { \"type\": \"string\", \"minLength\": 1 }, \"quotedText\": { \"type\": \"string\" } } } }, \"applicability\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"ruleText\", \"condition\" ], \"properties\": { \"ruleText\": { \"type\": \"string\", \"minLength\": 1 }, \"condition\": { \"$ref\": \"common.schema.json#/$defs/Condition\" } } }, \"proofPolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"positiveProofRequired\", \"positiveProofStandard\", \"failureCondition\", \"passOnNoViolation\", \"allowPartialCredit\", \"allowCompensation\" ], \"properties\": { \"positiveProofRequired\": { \"type\": \"boolean\" }, \"positiveProofStandard\": { \"type\": \"string\", \"minLength\": 1 }, \"failureCondition\": { \"type\": \"string\", \"minLength\": 1 }, \"omissionCondition\": { \"type\": \"string\" }, \"passOnNoViolation\": { \"type\": \"boolean\" }, \"allowPartialCredit\": { \"type\": \"boolean\" }, \"allowCompensation\": { \"type\": \"boolean\" } } }, \"evidencePolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"allowedEvidenceClassIds\", \"minimumAuthorityRank\", \"citationRequired\", \"externalVerificationMode\", \"deterministicValidatorIds\" ], \"properties\": { \"allowedEvidenceClassIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" } }, \"minimumAuthorityRank\": { \"type\": \"integer\", \"minimum\": 0 }, \"citationRequired\": { \"type\": \"boolean\" }, \"externalVerificationMode\": { \"type\": \"string\", \"enum\": [ \"forbidden\", \"optional\", \"required\" ] }, \"deterministicValidatorIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" } } } }, \"anchors\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"anchorId\", \"label\", \"example\", \"expectedVerdict\" ], \"properties\": { \"anchorId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"label\": { \"type\": \"string\", \"minLength\": 1 }, \"description\": { \"type\": \"string\" }, \"example\": { \"type\": \"string\", \"minLength\": 1 }, \"expectedVerdict\": { \"$ref\": \"common.schema.json#/$defs/CriterionVerdict\" }, \"notes\": { \"type\": \"string\" } } } }, \"validAlternatives\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"exceptions\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"dependencies\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"requiredBeforeCriterionIds\", \"impliesCriterionIds\" ], \"properties\": { \"requiredBeforeCriterionIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" } }, \"impliesCriterionIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" } } } }, \"precedence\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"overridesCriterionIds\", \"overriddenByCriterionIds\" ], \"properties\": { \"overridesCriterionIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" } }, \"overriddenByCriterionIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" } } } }, \"defaultSeverity\": { \"$ref\": \"common.schema.json#/$defs/Severity\" }, \"defaultDecisionEffect\": { \"type\": \"string\", \"enum\": [ \"none\", \"accept_with_note\", \"rework\", \"reject\", \"escalate\" ] }, \"tags\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\" } } } } }, \"decisionPolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"policyVersion\", \"allowedActions\", \"rules\", \"defaultAction\", \"tieBreakOrder\", \"confidencePolicy\" ], \"properties\": { \"policyVersion\": { \"$ref\": \"common.schema.json#/$defs/SemVer\" }, \"allowedActions\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" } }, \"rules\": { \"type\": \"array\", \"minItems\": 1, \"items\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"target\", \"ruleId\", \"priority\", \"description\", \"match\", \"action\", \"terminal\" ], \"properties\": { \"ruleId\": { \"$ref\": \"common.schema.json#/$defs/Identifier\" }, \"priority\": { \"type\": \"integer\", \"minimum\": 1 }, \"description\": { \"type\": \"string\", \"minLength\": 1 }, \"match\": { \"type\": \"object\", \"additionalProperties\": false, \"properties\": { \"criterionIds\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\" } }, \"criterionTags\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\" } }, \"verdicts\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/CriterionVerdict\" } }, \"severities\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/Severity\" } }, \"decisionEffects\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\", \"enum\": [ \"none\", \"accept_with_note\", \"rework\", \"reject\", \"escalate\" ] } }, \"minimumCalibratedConfidence\": { \"type\": \"number\", \"minimum\": 0, \"maximum\": 1 }, \"hasUnresolvedFindings\": { \"type\": \"boolean\" }, \"annotationDisagreementTypes\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } } }, \"minProperties\": 1 }, \"action\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" }, \"terminal\": { \"type\": \"boolean\" }, \"explanationTemplate\": { \"type\": \"string\" }, \"target\": { \"$ref\": \"common.schema.json#/$defs/DecisionTarget\" } } } }, \"defaultAction\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" }, \"tieBreakOrder\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" } }, \"confidencePolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"requiresCalibrationForActions\", \"uncalibratedAction\" ], \"properties\": { \"requiresCalibrationForActions\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" } }, \"uncalibratedAction\": { \"$ref\": \"common.schema.json#/$defs/OperationalAction\" }, \"minimumCalibrationCasesPerSlice\": { \"type\": \"integer\", \"minimum\": 1 } } } } }, \"humanReviewPolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"maxInteractionsPerCase\", \"onlyAskIfDecisionChanging\", \"hideProvisionalDecision\", \"requireAmbiguityOption\", \"allowedQuestionTypes\" ], \"properties\": { \"maxInteractionsPerCase\": { \"type\": \"integer\", \"minimum\": 0, \"maximum\": 1 }, \"onlyAskIfDecisionChanging\": { \"type\": \"boolean\" }, \"hideProvisionalDecision\": { \"type\": \"boolean\" }, \"requireAmbiguityOption\": { \"type\": \"boolean\" }, \"allowedQuestionTypes\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\", \"enum\": [ \"applicability\", \"instruction_precedence\", \"evidence_authority\", \"inference_permission\", \"severity\", \"valid_alternative\", \"rubric_ambiguity\", \"domain_expertise\" ] } }, \"expertiseRoutingRules\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } } } }, \"feedbackPolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"audiences\", \"maxItemsPerAudience\", \"includeValidatedStrengths\", \"validateRepairs\", \"priorityOrder\" ], \"properties\": { \"audiences\": { \"type\": \"array\", \"uniqueItems\": true, \"items\": { \"type\": \"string\", \"enum\": [ \"attempter\", \"reviewer\", \"project_owner\" ] } }, \"maxItemsPerAudience\": { \"type\": \"integer\", \"minimum\": 1 }, \"includeValidatedStrengths\": { \"type\": \"boolean\" }, \"validateRepairs\": { \"type\": \"boolean\" }, \"priorityOrder\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"type\": \"string\", \"enum\": [ \"decisive\", \"critical\", \"major\", \"minor\", \"strength\" ] } } } }, \"securityPolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"treatCandidateContentAsUntrusted\", \"ignoreEmbeddedEvaluatorInstructions\", \"requireStructuredOutputs\", \"allowCandidateToOverrideRubric\" ], \"properties\": { \"treatCandidateContentAsUntrusted\": { \"type\": \"boolean\" }, \"ignoreEmbeddedEvaluatorInstructions\": { \"type\": \"boolean\" }, \"requireStructuredOutputs\": { \"type\": \"boolean\" }, \"allowCandidateToOverrideRubric\": { \"type\": \"boolean\" }, \"redactionRules\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } } } }, \"auditPolicy\": { \"type\": \"object\", \"additionalProperties\": false, \"required\": [ \"retainExactSourceSnapshots\", \"retainPromptVersions\", \"retainModelRunMetadata\", \"retainHumanResolutions\", \"replayRequired\" ], \"properties\": { \"retainExactSourceSnapshots\": { \"type\": \"boolean\" }, \"retainPromptVersions\": { \"type\": \"boolean\" }, \"retainModelRunMetadata\": { \"type\": \"boolean\" }, \"retainHumanResolutions\": { \"type\": \"boolean\" }, \"replayRequired\": { \"type\": \"boolean\" }, \"retentionDays\": { \"type\": \"integer\", \"minimum\": 1 } } }, \"metadata\": { \"type\": \"object\", \"additionalProperties\": true }, \"decisionTargets\": { \"type\": \"array\", \"minItems\": 1, \"uniqueItems\": true, \"items\": { \"$ref\": \"common.schema.json#/$defs/DecisionTarget\" } } } } "}, {"slug": "foundation-templates-criterion-authoring-template-csv", "title": "Template - Criterion Authoring", "category": "Foundation templates", "summary": "CSV template for drafting criterion intent, proof standards, failure anchors, evidence requirements, severity, and governance.", "tags": ["template", "authoring"], "view": "documents/view/foundation-templates-criterion-authoring-template-csv.html", "raw": "documents/raw/foundation/templates/criterion-authoring-template.csv", "text": "criterion_id,name,requirement,intent,criterion_type,scope,subjectivity,instruction_id,instruction_locator,applicability_rule,positive_proof_standard,failure_condition,omission_condition,allowed_evidence_classes,minimum_authority_rank,citation_required,positive_anchor,negative_anchor,valid_alternatives,exceptions,dependencies,precedence,default_severity,default_decision_effect,hard_gate,owner,status,version C-example,Example criterion,,,required_presence,attempt,bounded_judgment,,,Always applies,,,,,0,false,,,,,,,major,rework,false,,draft,0.1.0 "}, {"slug": "foundation-templates-gold-set-template-csv", "title": "Template - Gold Set", "category": "Foundation templates", "summary": "CSV template for building an adjudicated calibration and test set.", "tags": ["template", "authoring"], "view": "documents/view/foundation-templates-gold-set-template-csv.html", "raw": "documents/raw/foundation/templates/gold-set-template.csv", "text": "benchmark_item_id,benchmark_version,split,contract_id,contract_version,case_id,criterion_id,applicability,acceptable_verdicts,severity,positive_evidence_spans,negative_evidence_spans,omission_descriptions,acceptable_actions,ambiguity_status,challenge_attributes,slice_tags,adjudication_method,adjudicator_count,domain_expert_count,blind_to_autoqa,blind_to_reviewer_label,initial_agreement_rate,adjudication_notes,allowed_for_prompt_development,held_out_project,weight B-001,0.1.0,calibration,,,,,applies,met,none,,,,accept,clear,,,independent_then_adjudicate,3,1,true,true,,,false,false,1.0 "}, {"slug": "foundation-templates-model-bakeoff-template-csv", "title": "Template - Model Bakeoff", "category": "Foundation templates", "summary": "CSV template for comparing evaluator configurations by criterion family.", "tags": ["template", "authoring"], "view": "documents/view/foundation-templates-model-bakeoff-template-csv.html", "raw": "documents/raw/foundation/templates/model-bakeoff-template.csv", "text": "experiment_id,run_date,contract_id,benchmark_version,model_provider,model_name,model_version,judge_role,prompt_template_id,prompt_template_version,reasoning_mode,temperature,samples_per_item,reference_timing,candidate_order_policy,ensemble_policy,criterion_macro_f1,critical_false_accept_rate,strong_positive_false_reject_rate,evidence_span_precision,evidence_span_recall,positive_finding_precision,omission_recall,brier_score,ece,coverage_at_target_risk,mean_latency_ms,mean_cost_per_case,notes EXP-001,,,,,,,criterion_adjudicator,,,default,0,1,late,randomized,none,,,,,,,,,,,,, "}, {"slug": "foundation-types-autoqa-ts", "title": "TypeScript Domain Model", "category": "Foundation developer files", "summary": "Strict TypeScript types for project contracts, cases, evaluations, benchmarks, findings, decisions, feedback, and human resolution.", "tags": ["typescript", "developer"], "view": "documents/view/foundation-types-autoqa-ts.html", "raw": "documents/raw/foundation/types/autoqa.ts", "text": "/** * AutoQA Foundation types - schema version 0.1.0 * * These interfaces mirror the JSON Schemas in ../schemas. Runtime validation * must use the JSON Schemas; TypeScript types alone are not a trust boundary. */ export type JsonPrimitive = string | number | boolean | null; export type JsonValue = JsonPrimitive | JsonValue[] | { [key: string]: JsonValue }; export type Identifier = string; export type SemVer = string; export type ContentHash = `sha256:${string}`; export type DateTime = string; export type ConfidenceBand = \"high\" | \"medium\" | \"low\" | \"uncalibrated\"; export type Severity = \"none\" | \"advisory\" | \"minor\" | \"major\" | \"critical\"; export type DecisionTarget = \"attempt\" | \"annotation\" | \"composite_case\" | \"batch\"; export type OperationalAction = | \"accept\" | \"accept_with_notes\" | \"rework\" | \"reject\" | \"escalate\"; export type CriterionVerdict = | \"met\" | \"partially_met\" | \"not_met\" | \"ambiguous\" | \"unknown\" | \"not_applicable\"; export interface ContractRef { contractId: Identifier; contractVersion: SemVer; contentHash: ContentHash; } export interface CaseRef { caseId: Identifier; contentHash?: ContentHash; } export interface SpanRef { artifactType: | \"task\" | \"evidence\" | \"attempt\" | \"human_review\" | \"reference\" | \"tool_trace\"; artifactId: Identifier; startChar?: number; endChar?: number; quotedText: string; locator?: string; } export interface Confidence { /** Model-native or component-native confidence. Never use alone for autonomy. */ rawScore?: number; /** Empirically calibrated probability for the defined event and slice. */ calibratedScore?: number; band: ConfidenceBand; calibrationProfileId?: Identifier; rationale?: string; } export interface GenerationSource { kind: \"deterministic_validator\" | \"llm_judge\" | \"human\" | \"hybrid\"; componentId: Identifier; modelRunId?: Identifier; humanActorId?: Identifier; } export type Condition = | { kind: \"always\" } | { kind: \"predicate\"; fieldPath: string; operator: | \"equals\" | \"not_equals\" | \"in\" | \"not_in\" | \"contains\" | \"exists\" | \"greater_than\" | \"greater_than_or_equal\" | \"less_than\" | \"less_than_or_equal\" | \"matches_regex\"; value?: JsonValue; } | { kind: \"all\"; conditions: Condition[] } | { kind: \"any\"; conditions: Condition[] } | { kind: \"not\"; condition: Condition }; export interface Finding { findingId: Identifier; findingType: | \"positive\" | \"negative\" | \"omission\" | \"ambiguity\" | \"counterevidence\" | \"integrity\"; criterionId?: Identifier; summary: string; detail?: string; claimIds?: Identifier[]; expectedElementIds?: Identifier[]; targetSpans?: SpanRef[]; evidenceIds?: Identifier[]; relationship?: | \"directly_entails\" | \"reasonably_supports\" | \"weakly_supports\" | \"contradicts\" | \"irrelevant\" | \"insufficient\" | \"not_applicable\"; materiality: \"incidental\" | \"supporting\" | \"central\" | \"decisive\"; severity: Severity; confidence: Confidence; status: \"observed\" | \"challenged\" | \"confirmed\" | \"retracted\"; source: GenerationSource; } export interface Claim { claimId: Identifier; proposition: string; claimType: | \"factual\" | \"interpretive\" | \"predictive\" | \"normative\" | \"preference\" | \"procedural\"; importance: \"incidental\" | \"supporting\" | \"central\"; explicitness: \"explicit\" | \"implied\"; attemptSpans: SpanRef[]; qualifiers?: string[]; citedEvidenceIds?: Identifier[]; } export interface ExpectedElement { elementId: Identifier; criterionId: Identifier; description: string; required: boolean; importance: \"supporting\" | \"central\" | \"decisive\"; acceptableAlternatives?: string[]; observedSpans?: SpanRef[]; coverage: \"complete\" | \"partial\" | \"missing\" | \"not_applicable\" | \"unknown\"; } export interface EvidenceRelationship { relationshipId: Identifier; subjectType: \"claim\" | \"expected_element\" | \"finding\"; subjectId: Identifier; evidenceId: Identifier; relationship: | \"directly_entails\" | \"reasonably_supports\" | \"weakly_supports\" | \"contradicts\" | \"irrelevant\" | \"insufficient\"; sourceQuality: \"high\" | \"acceptable\" | \"weak\" | \"disallowed\" | \"unknown\"; temporalValidity: \"valid\" | \"stale\" | \"not_time_sensitive\" | \"unknown\"; evidenceSpans?: SpanRef[]; confidence: Confidence; notes?: string; } export interface CriterionAssessment { criterionId: Identifier; applicability: \"applies\" | \"does_not_apply\" | \"uncertain\"; positiveFindingIds: Identifier[]; negativeFindingIds: Identifier[]; omissionFindingIds: Identifier[]; counterEvidenceFindingIds: Identifier[]; unresolvedFindingIds: Identifier[]; groundingStatus: | \"grounded\" | \"partially_grounded\" | \"ungrounded\" | \"not_applicable\" | \"unknown\"; truthStatus: | \"verified_true\" | \"verified_false\" | \"mixed\" | \"not_evaluated\" | \"not_applicable\" | \"unknown\"; warrantStatus: \"justified\" | \"overstated\" | \"invalid\" | \"not_applicable\" | \"unknown\"; relevanceStatus: | \"relevant\" | \"partially_relevant\" | \"irrelevant\" | \"not_applicable\" | \"unknown\"; verdict: CriterionVerdict; severity: Severity; confidence: Confidence; decisionEffect: \"none\" | \"accept_with_note\" | \"rework\" | \"reject\" | \"escalate\"; rationale: string; } export interface GlobalAssessment { axisId: Identifier; verdict: CriterionVerdict; findingIds: Identifier[]; score?: number; confidence: Confidence; rationale: string; } export interface PolicyTraceEntry { ruleId: Identifier; matched: boolean; inputs?: Record<string, JsonValue>; result: string; } export interface Decision { target: DecisionTarget; action: OperationalAction; policyVersion: SemVer; triggeredRuleIds: Identifier[]; decisiveCriterionIds: Identifier[]; unresolvedFindingIds: Identifier[]; confidence: Confidence; policyTrace: PolicyTraceEntry[]; rationale: string; decidedAt: DateTime; } export interface ResolutionOption { optionId: Identifier; label: string; description?: string; effectSummary?: string; } export interface HumanResolutionRequest { requestId: Identifier; triggerCriterionIds: Identifier[]; questionType: | \"applicability\" | \"instruction_precedence\" | \"evidence_authority\" | \"inference_permission\" | \"severity\" | \"valid_alternative\" | \"rubric_ambiguity\" | \"domain_expertise\"; question: string; contextFindingIds: Identifier[]; options: ResolutionOption[]; decisionChanging: boolean; hideProvisionalDecision: boolean; requiredExpertiseTags?: string[]; status: \"pending\" | \"answered\" | \"cancelled\"; createdAt: DateTime; } export interface HumanResolutionResponse { requestId: Identifier; selectedOptionId: Identifier; responderId: Identifier; rationale?: string; respondedAt: DateTime; } export interface HumanResolutionInteraction { decisionTarget: DecisionTarget; request: HumanResolutionRequest; response?: HumanResolutionResponse; decisionBefore?: OperationalAction; decisionAfter?: OperationalAction; changedDecision?: boolean; } export interface HumanResolutionPacket { documentType: \"human_resolution_packet\"; schemaVersion: SemVer; caseRef: CaseRef; contractRef: ContractRef; contextFindings: Finding[]; interaction: HumanResolutionInteraction; } export interface FeedbackItem { feedbackId: Identifier; audience: \"attempter\" | \"reviewer\" | \"project_owner\"; criterionId?: Identifier; feedbackType: \"strength\" | \"defect\" | \"omission\" | \"ambiguity\"; findingIds: Identifier[]; message: string; minimalRepair?: string; priority: number; repairValidationStatus: \"validated\" | \"plausible_unvalidated\" | \"not_applicable\"; } export interface AnnotationAssessment { annotationPresent: boolean; labelSupport: | \"supported\" | \"partially_supported\" | \"unsupported\" | \"reasonable_disagreement\" | \"rubric_ambiguous\" | \"not_applicable\"; rationaleSupport: | \"supported\" | \"contains_unsupported_reasoning\" | \"does_not_support_label\" | \"missing\" | \"not_applicable\"; criterionAttribution: | \"correct\" | \"partially_correct\" | \"wrong_criterion\" | \"unstated_preference\" | \"not_applicable\"; evidenceAlignment: | \"aligned\" | \"partially_aligned\" | \"misquoted\" | \"irrelevant\" | \"missing\" | \"not_applicable\"; severityAlignment: \"appropriate\" | \"too_severe\" | \"too_lenient\" | \"uncertain\" | \"not_applicable\"; omittedPositiveFindingIds: Identifier[]; omittedNegativeFindingIds: Identifier[]; unsupportedReviewerFindingIds: Identifier[]; disagreementTypes: Array< | \"none\" | \"observation\" | \"evidence_authority\" | \"applicability\" | \"criterion_interpretation\" | \"inference\" | \"threshold\" | \"severity\" | \"attention\" | \"expertise\" | \"reasonable_disagreement\" | \"rubric_defect\" >; confidence: Confidence; rationale: string; } export interface ModelRun { modelRunId: Identifier; role: | \"claim_extractor\" | \"evidence_retriever\" | \"positive_prover\" | \"negative_challenger\" | \"coverage_auditor\" | \"warrant_auditor\" | \"criterion_adjudicator\" | \"holistic_judge\" | \"annotation_auditor\" | \"feedback_generator\" | \"repair_validator\" | \"ensemble_aggregator\"; provider: string; modelName: string; modelVersion?: string; promptTemplateId: Identifier; promptTemplateVersion: SemVer; promptTemplateHash: ContentHash; parameters?: Record<string, JsonValue>; parentRunIds?: Identifier[]; inputHash?: ContentHash; outputHash?: ContentHash; startedAt: DateTime; completedAt: DateTime; } export interface ValidatorRun { validatorRunId: Identifier; validatorId: Identifier; validatorVersion: SemVer; role: | \"schema_validation\" | \"format_check\" | \"citation_check\" | \"calculation\" | \"code_execution\" | \"duplicate_detection\" | \"integrity_check\" | \"custom\"; status: \"passed\" | \"failed\" | \"warning\" | \"error\"; parameters?: Record<string, JsonValue>; inputHash: ContentHash; outputHash: ContentHash; startedAt: DateTime; completedAt: DateTime; } export interface InstructionDocument { instructionId: Identifier; title: string; authority: \"primary\" | \"supplemental\" | \"example\" | \"historical\"; content: string; contentHash: ContentHash; precedenceRank?: number; } export interface EvidenceClass { classId: Identifier; name: string; description?: string; permitted: boolean; authorityRank: number; } export interface InstructionRef { instructionId: Identifier; locator: string; quotedText?: string; } export interface CriterionAnchor { anchorId: Identifier; label: string; description?: string; example: string; expectedVerdict: CriterionVerdict; notes?: string; } export interface CriterionDefinition { criterionId: Identifier; name: string; requirement: string; intent: string; criterionType: | \"required_presence\" | \"prohibited_presence\" | \"conditional\" | \"factual\" | \"grounding\" | \"inferential_warrant\" | \"comparative\" | \"completeness\" | \"holistic_quality\" | \"mechanically_verifiable\" | \"annotation_fidelity\"; scope: \"attempt\" | \"claim\" | \"citation\" | \"annotation\" | \"global\" | \"batch\"; subjectivity: \"objective\" | \"bounded_judgment\" | \"holistic_expert\"; instructionRefs: InstructionRef[]; applicability: { ruleText: string; condition: Condition }; proofPolicy: { positiveProofRequired: boolean; positiveProofStandard: string; failureCondition: string; omissionCondition?: string; passOnNoViolation: boolean; allowPartialCredit: boolean; allowCompensation: boolean; }; evidencePolicy: { allowedEvidenceClassIds: Identifier[]; minimumAuthorityRank: number; citationRequired: boolean; externalVerificationMode: \"forbidden\" | \"optional\" | \"required\"; deterministicValidatorIds: Identifier[]; }; anchors: CriterionAnchor[]; validAlternatives?: string[]; exceptions?: string[]; dependencies: { requiredBeforeCriterionIds: Identifier[]; impliesCriterionIds: Identifier[]; }; precedence: { overridesCriterionIds: Identifier[]; overriddenByCriterionIds: Identifier[]; }; defaultSeverity: Severity; defaultDecisionEffect: \"none\" | \"accept_with_note\" | \"rework\" | \"reject\" | \"escalate\"; tags: string[]; } export interface DecisionRuleMatch { criterionIds?: string[]; criterionTags?: string[]; verdicts?: CriterionVerdict[]; severities?: Severity[]; decisionEffects?: Array<\"none\" | \"accept_with_note\" | \"rework\" | \"reject\" | \"escalate\">; minimumCalibratedConfidence?: number; hasUnresolvedFindings?: boolean; annotationDisagreementTypes?: string[]; } export interface DecisionRule { target: DecisionTarget; ruleId: Identifier; priority: number; description: string; match: DecisionRuleMatch; action: OperationalAction; terminal: boolean; explanationTemplate?: string; } export interface ProjectContract { documentType: \"project_contract\"; schemaVersion: SemVer; contractId: Identifier; contractVersion: SemVer; projectName: string; description?: string; status: \"draft\" | \"approved\" | \"deprecated\" | \"retired\"; createdAt: DateTime; effectiveFrom?: DateTime; supersedesContractVersion?: SemVer; owners?: Array<{ actorId: Identifier; role: \"project_owner\" | \"rubric_owner\" | \"domain_expert\" | \"qa_owner\"; }>; instructionDocuments: InstructionDocument[]; sourcePolicy: { worldMode: \"closed_world\" | \"open_world\" | \"hybrid\"; parametricKnowledgePolicy: \"never_evidence\" | \"common_knowledge_only\" | \"allowed_with_verification\"; externalRetrievalAllowed: boolean; defaultCitationRequirement: \"none\" | \"material_claims\" | \"all_verifiable_claims\" | \"project_defined\"; snapshotRequired: boolean; conflictPolicy: string; freshnessRules?: string[]; evidenceClasses: EvidenceClass[]; }; criteria: CriterionDefinition[]; decisionTargets: DecisionTarget[]; decisionPolicy: { policyVersion: SemVer; allowedActions: OperationalAction[]; rules: DecisionRule[]; defaultAction: OperationalAction; tieBreakOrder: OperationalAction[]; confidencePolicy: { requiresCalibrationForActions: OperationalAction[]; uncalibratedAction: OperationalAction; minimumCalibrationCasesPerSlice?: number; }; }; humanReviewPolicy: { maxInteractionsPerCase: 0 | 1; onlyAskIfDecisionChanging: boolean; hideProvisionalDecision: boolean; requireAmbiguityOption: boolean; allowedQuestionTypes: HumanResolutionRequest[\"questionType\"][]; expertiseRoutingRules?: string[]; }; feedbackPolicy: { audiences: FeedbackItem[\"audience\"][]; maxItemsPerAudience: number; includeValidatedStrengths: boolean; validateRepairs: boolean; priorityOrder: Array<\"decisive\" | \"critical\" | \"major\" | \"minor\" | \"strength\">; }; securityPolicy: { treatCandidateContentAsUntrusted: boolean; ignoreEmbeddedEvaluatorInstructions: boolean; requireStructuredOutputs: boolean; allowCandidateToOverrideRubric: boolean; redactionRules?: string[]; }; auditPolicy: { retainExactSourceSnapshots: boolean; retainPromptVersions: boolean; retainModelRunMetadata: boolean; retainHumanResolutions: boolean; replayRequired: boolean; retentionDays?: number; }; metadata?: Record<string, JsonValue>; } export interface TaskInput { taskId: Identifier; taskType: string; prompt: string; taskInstructions?: string[]; expectedOutputFormat?: string; contentHash: ContentHash; metadata?: Record<string, JsonValue>; } export interface EvidenceArtifact { evidenceId: Identifier; evidenceClassId: Identifier; title: string; content: string; contentHash: ContentHash; sourceUri?: string; authorityRank: number; permitted: boolean; publishedAt?: DateTime; capturedAt: DateTime; metadata?: Record<string, JsonValue>; } export interface AttemptInput { attemptId: Identifier; authorId?: Identifier; content: string; contentHash: ContentHash; submittedAt: DateTime; attachments?: Array<{ artifactId: Identifier; artifactType: string; content?: string; contentHash: ContentHash; }>; toolTraces?: Array<{ traceId: Identifier; toolName: string; input: JsonValue; output: JsonValue; contentHash: ContentHash; }>; } export interface HumanReviewInput { annotationId: Identifier; reviewerId: Identifier; label: string; score?: number; reasonCodes: string[]; rationale: string; criterionAnnotations?: Array<{ criterionId: Identifier; verdict: CriterionVerdict; severity?: Severity; rationale: string; citedSpans?: SpanRef[]; }>; reviewerConfidence?: Confidence; submittedAt: DateTime; } export interface CaseInput { documentType: \"case_input\"; schemaVersion: SemVer; caseId: Identifier; createdAt: DateTime; contractRef: ContractRef; task: TaskInput; evidenceBundle: EvidenceArtifact[]; attempt: AttemptInput; humanReview?: HumanReviewInput; referenceAnswers?: Array<{ referenceId: Identifier; role: \"authoritative\" | \"illustrative\" | \"positive_anchor\" | \"negative_anchor\"; content: string; contentHash: ContentHash; }>; declaredExpectedElements?: ExpectedElement[]; caseTags: string[]; metadata?: Record<string, JsonValue>; } export interface EvaluationOutput { documentType: \"evaluation_output\"; schemaVersion: SemVer; evaluationId: Identifier; evaluationVersion: SemVer; caseRef: CaseRef; contractRef: ContractRef; createdAt: DateTime; status: \"provisional\" | \"awaiting_human\" | \"final\" | \"failed\"; runManifest: ModelRun[]; validatorManifest: ValidatorRun[]; integrityStatus?: \"clean\" | \"warning\" | \"failed\" | \"unknown\"; findings: Finding[]; claims: Claim[]; expectedElements: ExpectedElement[]; evidenceRelationships: EvidenceRelationship[]; criterionAssessments: CriterionAssessment[]; globalAssessments: GlobalAssessment[]; annotationAssessment?: AnnotationAssessment; provisionalDecisions: Decision[]; humanResolution?: HumanResolutionInteraction; finalDecisions: Decision[]; feedback: FeedbackItem[]; audit: { caseInputHash: ContentHash; contractHash: ContentHash; sourceSnapshotHashes: ContentHash[]; replayable: boolean; warnings: string[]; }; } export interface BenchmarkCriterionGold { criterionId: Identifier; applicability: \"applies\" | \"does_not_apply\" | \"uncertain\"; acceptableVerdicts: CriterionVerdict[]; severity: Severity; positiveEvidenceSpans: SpanRef[]; negativeEvidenceSpans: SpanRef[]; omissionDescriptions: string[]; notes: string; } export interface BenchmarkItem { documentType: \"benchmark_item\"; schemaVersion: SemVer; benchmarkItemId: Identifier; benchmarkVersion: SemVer; split: \"development\" | \"calibration\" | \"test\" | \"challenge\" | \"shadow\" | \"held_out_project\"; contractRef: ContractRef; caseRef: CaseRef; sliceTags: string[]; challengeAttributes: Array< | \"strong_positive\" | \"subtle_valid_alternative\" | \"polished_falsehood\" | \"unsupported_rationale\" | \"citation_mismatch\" | \"partial_support\" | \"required_content_omission\" | \"severity_edge\" | \"instruction_conflict\" | \"ambiguous_case\" | \"prompt_injection\" | \"rubric_parroting\" | \"correct_label_wrong_reason\" | \"wrong_label_correct_observation\" | \"long_context\" | \"domain_expertise\" | \"position_bias_probe\" >; goldStandard: { criterionGold: BenchmarkCriterionGold[]; acceptableDecisions: Array<{ target: DecisionTarget; acceptableActions: OperationalAction[]; }>; annotationGold?: { acceptableLabelSupport?: string[]; expectedDisagreementTypes?: string[]; notes?: string; }; ambiguityStatus: \"clear\" | \"reasonable_disagreement\" | \"rubric_ambiguous\" | \"insufficient_evidence\"; validAlternativeNotes?: string[]; }; adjudication: { method: \"single_expert\" | \"independent_then_adjudicate\" | \"consensus_panel\" | \"latent_label_model\"; adjudicatorCount: number; domainExpertCount: number; blindToAutoQA: boolean; blindToReviewerLabel: boolean; initialAgreementRate?: number; disagreementTypes: string[]; adjudicationNotes?: string; completedAt: DateTime; }; leakageControl: { allowedForPromptDevelopment: boolean; heldOutProject: boolean; firstFrozenAt: DateTime; accessPolicy: string; }; weight: number; exclusionReason?: string; metadata?: Record<string, JsonValue>; } /** * Stable domain boundary for a Vercel/TypeScript application. * The orchestration layer can accept and return these documents without * depending on any particular model provider. */ export interface AutoQAEvaluator { evaluate(caseInput: CaseInput, contract: ProjectContract): Promise<EvaluationOutput>; } "}, {"slug": "foundation-types-tsconfig-json", "title": "TypeScript Configuration", "category": "Foundation developer files", "summary": "Compiler configuration for the package's type definitions.", "tags": ["typescript", "developer"], "view": "documents/view/foundation-types-tsconfig-json.html", "raw": "documents/raw/foundation/types/tsconfig.json", "text": "{ \"compilerOptions\": { \"target\": \"ES2022\", \"module\": \"ESNext\", \"moduleResolution\": \"Bundler\", \"strict\": true, \"noEmit\": true, \"skipLibCheck\": true }, \"include\": [\"autoqa.ts\"] } "}, {"slug": "foundation-validation-hash-utils-py", "title": "Validation - Hash Utilities", "category": "Foundation validation", "summary": "Canonical content hashing utilities used by the validation suite.", "tags": ["validation", "python"], "view": "documents/view/foundation-validation-hash-utils-py.html", "raw": "documents/raw/foundation/validation/hash_utils.py", "text": "from __future__ import annotations import hashlib import json from pathlib import Path from typing import Any def sha256_text(text: str) -> str: \"\"\"Hash exact UTF-8 text.\"\"\" return \"sha256:\" + hashlib.sha256(text.encode(\"utf-8\")).hexdigest() def canonical_json_bytes(value: Any) -> bytes: \"\"\"Canonical JSON v1 used by the example package. Object keys are sorted, arrays retain order, insignificant whitespace is removed, Unicode is encoded as UTF-8, and JSON numbers use Python's normal JSON serialization. Production systems that need cross-language numeric canonicalization should adopt a stricter standard and version it. \"\"\" return json.dumps( value, ensure_ascii=False, sort_keys=True, separators=(\",\", \":\"), allow_nan=False, ).encode(\"utf-8\") def sha256_json(value: Any) -> str: return \"sha256:\" + hashlib.sha256(canonical_json_bytes(value)).hexdigest() def load_json(path: Path) -> Any: with path.open(\"r\", encoding=\"utf-8\") as handle: return json.load(handle) "}, {"slug": "foundation-validation-validate-examples-py", "title": "Validation - Example Schema Checks", "category": "Foundation validation", "summary": "Python validator for the package's example documents.", "tags": ["validation", "python"], "view": "documents/view/foundation-validation-validate-examples-py.html", "raw": "documents/raw/foundation/validation/validate_examples.py", "text": "from __future__ import annotations import json from pathlib import Path from jsonschema import Draft202012Validator from referencing import Registry, Resource ROOT = Path(__file__).resolve().parents[1] SCHEMA_DIR = ROOT / \"schemas\" EXAMPLE_DIR = ROOT / \"examples\" PAIRS = [ (\"project-contract.schema.json\", \"project-contract.example.json\"), (\"case-input.schema.json\", \"case-input.example.json\"), (\"evaluation-output.schema.json\", \"evaluation-output.example.json\"), (\"benchmark-item.schema.json\", \"benchmark-item.example.json\"), (\"human-resolution-packet.schema.json\", \"human-resolution-packet.example.json\"), ] def load_json(path: Path): with path.open(\"r\", encoding=\"utf-8\") as handle: return json.load(handle) def build_registry() -> Registry: registry = Registry() for schema_path in SCHEMA_DIR.glob(\"*.schema.json\"): schema = load_json(schema_path) Draft202012Validator.check_schema(schema) schema_id = schema.get(\"$id\") if not schema_id: raise ValueError(f\"Schema has no $id: {schema_path}\") registry = registry.with_resource(schema_id, Resource.from_contents(schema)) return registry def validate_examples() -> list[str]: registry = build_registry() failures: list[str] = [] for schema_name, example_name in PAIRS: schema_path = SCHEMA_DIR / schema_name example_path = EXAMPLE_DIR / example_name schema = load_json(schema_path) instance = load_json(example_path) validator = Draft202012Validator(schema, registry=registry) errors = sorted(validator.iter_errors(instance), key=lambda e: list(e.absolute_path)) if errors: failures.append(example_name) print(f\"FAIL {example_name}\") for error in errors: path = \".\".join(str(part) for part in error.absolute_path) or \"<root>\" print(f\" {path}: {error.message}\") else: print(f\"PASS {example_name}\") return failures def main() -> None: failures = validate_examples() if failures: raise SystemExit(1) if __name__ == \"__main__\": main() "}, {"slug": "foundation-validation-validate-package-py", "title": "Validation - Package Integrity", "category": "Foundation validation", "summary": "Cross-document, schema, hash, proof-invariant, policy, and package integrity checks.", "tags": ["validation", "python"], "view": "documents/view/foundation-validation-validate-package-py.html", "raw": "documents/raw/foundation/validation/validate_package.py", "text": "from __future__ import annotations import json from collections import Counter from pathlib import Path from typing import Any, Iterable from hash_utils import load_json, sha256_json, sha256_text from validate_examples import validate_examples ROOT = Path(__file__).resolve().parents[1] EXAMPLES = ROOT / \"examples\" def duplicates(values: Iterable[str]) -> list[str]: counts = Counter(values) return sorted(value for value, count in counts.items() if count > 1) def check_hash(errors: list[str], label: str, actual: str, expected: str) -> None: if actual != expected: errors.append(f\"{label}: expected {expected}, found {actual}\") def build_artifact_text(case: dict[str, Any]) -> dict[tuple[str, str], str]: artifacts: dict[tuple[str, str], str] = { (\"task\", case[\"task\"][\"taskId\"]): case[\"task\"][\"prompt\"], (\"attempt\", case[\"attempt\"][\"attemptId\"]): case[\"attempt\"][\"content\"], } for evidence in case[\"evidenceBundle\"]: artifacts[(\"evidence\", evidence[\"evidenceId\"])] = evidence[\"content\"] for reference in case.get(\"referenceAnswers\", []): artifacts[(\"reference\", reference[\"referenceId\"])] = reference[\"content\"] for trace in case[\"attempt\"].get(\"toolTraces\", []): artifacts[(\"tool_trace\", trace[\"traceId\"])] = json.dumps( {\"input\": trace[\"input\"], \"output\": trace[\"output\"]}, ensure_ascii=False, sort_keys=True, ) review = case.get(\"humanReview\") if review: review_text = \"\\n\".join( [ review.get(\"label\", \"\"), \" \".join(review.get(\"reasonCodes\", [])), review.get(\"rationale\", \"\"), *[ annotation.get(\"rationale\", \"\") for annotation in review.get(\"criterionAnnotations\", []) ], ] ) artifacts[(\"human_review\", review[\"annotationId\"])] = review_text return artifacts def check_span(errors: list[str], prefix: str, span: dict[str, Any], artifacts: dict[tuple[str, str], str]) -> None: key = (span[\"artifactType\"], span[\"artifactId\"]) if key not in artifacts: errors.append(f\"{prefix}: unknown artifact {key}\") return text = artifacts[key] quote = span[\"quotedText\"] start = span.get(\"startChar\") end = span.get(\"endChar\") if start is not None and end is not None: if start > end or end > len(text): errors.append(f\"{prefix}: invalid character range {start}:{end} for length {len(text)}\") elif text[start:end] != quote: errors.append(f\"{prefix}: quotedText does not match declared character range\") elif quote not in text: errors.append(f\"{prefix}: quotedText not found in artifact {key}\") def validate_referential_integrity() -> list[str]: errors: list[str] = [] contract = load_json(EXAMPLES / \"project-contract.example.json\") case = load_json(EXAMPLES / \"case-input.example.json\") evaluation = load_json(EXAMPLES / \"evaluation-output.example.json\") benchmark = load_json(EXAMPLES / \"benchmark-item.example.json\") resolution_packet = load_json(EXAMPLES / \"human-resolution-packet.example.json\") contract_hash = sha256_json(contract) case_hash = sha256_json(case) # Contract content and identity. instruction_ids = [doc[\"instructionId\"] for doc in contract[\"instructionDocuments\"]] criterion_ids = [criterion[\"criterionId\"] for criterion in contract[\"criteria\"]] evidence_class_ids = [item[\"classId\"] for item in contract[\"sourcePolicy\"][\"evidenceClasses\"]] rule_ids = [rule[\"ruleId\"] for rule in contract[\"decisionPolicy\"][\"rules\"]] for label, values in [ (\"instruction IDs\", instruction_ids), (\"criterion IDs\", criterion_ids), (\"evidence class IDs\", evidence_class_ids), (\"decision rule IDs\", rule_ids), ]: dupes = duplicates(values) if dupes: errors.append(f\"Duplicate {label}: {dupes}\") instruction_by_id = {doc[\"instructionId\"]: doc for doc in contract[\"instructionDocuments\"]} criterion_by_id = {criterion[\"criterionId\"]: criterion for criterion in contract[\"criteria\"]} rule_by_id = {rule[\"ruleId\"]: rule for rule in contract[\"decisionPolicy\"][\"rules\"]} declared_targets = set(contract[\"decisionTargets\"]) for doc in contract[\"instructionDocuments\"]: check_hash(errors, f\"instruction {doc['instructionId']} hash\", doc[\"contentHash\"], sha256_text(doc[\"content\"])) for criterion in contract[\"criteria\"]: cid = criterion[\"criterionId\"] for ref in criterion[\"instructionRefs\"]: if ref[\"instructionId\"] not in instruction_by_id: errors.append(f\"{cid}: unknown instruction ref {ref['instructionId']}\") elif ref.get(\"quotedText\") and ref[\"quotedText\"] not in instruction_by_id[ref[\"instructionId\"]][\"content\"]: errors.append(f\"{cid}: quoted instruction text not found in {ref['instructionId']}\") for eid in criterion[\"evidencePolicy\"][\"allowedEvidenceClassIds\"]: if eid not in evidence_class_ids: errors.append(f\"{cid}: unknown evidence class {eid}\") for dep in criterion[\"dependencies\"][\"requiredBeforeCriterionIds\"] + criterion[\"dependencies\"][\"impliesCriterionIds\"]: if dep not in criterion_by_id: errors.append(f\"{cid}: unknown dependency {dep}\") for other in criterion[\"precedence\"][\"overridesCriterionIds\"] + criterion[\"precedence\"][\"overriddenByCriterionIds\"]: if other not in criterion_by_id: errors.append(f\"{cid}: unknown precedence criterion {other}\") for rule in contract[\"decisionPolicy\"][\"rules\"]: if rule[\"target\"] not in declared_targets: errors.append(f\"Rule {rule['ruleId']} targets undeclared target {rule['target']}\") for cid in rule[\"match\"].get(\"criterionIds\", []): if cid != \"*\" and cid not in criterion_by_id: errors.append(f\"Rule {rule['ruleId']} references unknown criterion {cid}\") # Case hashes and references. if case[\"contractRef\"][\"contractId\"] != contract[\"contractId\"]: errors.append(\"Case contractId does not match contract\") if case[\"contractRef\"][\"contractVersion\"] != contract[\"contractVersion\"]: errors.append(\"Case contractVersion does not match contract\") check_hash(errors, \"case contract hash\", case[\"contractRef\"][\"contentHash\"], contract_hash) check_hash(errors, \"task content hash\", case[\"task\"][\"contentHash\"], sha256_text(case[\"task\"][\"prompt\"])) check_hash(errors, \"attempt content hash\", case[\"attempt\"][\"contentHash\"], sha256_text(case[\"attempt\"][\"content\"])) evidence_ids = [evidence[\"evidenceId\"] for evidence in case[\"evidenceBundle\"]] if duplicates(evidence_ids): errors.append(f\"Duplicate case evidence IDs: {duplicates(evidence_ids)}\") for evidence in case[\"evidenceBundle\"]: if evidence[\"evidenceClassId\"] not in evidence_class_ids: errors.append(f\"Evidence {evidence['evidenceId']} uses unknown class {evidence['evidenceClassId']}\") check_hash(errors, f\"evidence {evidence['evidenceId']} hash\", evidence[\"contentHash\"], sha256_text(evidence[\"content\"])) for reference in case.get(\"referenceAnswers\", []): check_hash(errors, f\"reference {reference['referenceId']} hash\", reference[\"contentHash\"], sha256_text(reference[\"content\"])) for annotation in case.get(\"humanReview\", {}).get(\"criterionAnnotations\", []): if annotation[\"criterionId\"] not in criterion_by_id: errors.append(f\"Human review references unknown criterion {annotation['criterionId']}\") artifacts = build_artifact_text(case) for annotation_index, annotation in enumerate(case.get(\"humanReview\", {}).get(\"criterionAnnotations\", [])): for span_index, span in enumerate(annotation.get(\"citedSpans\", [])): check_span(errors, f\"humanReview criterionAnnotations[{annotation_index}].citedSpans[{span_index}]\", span, artifacts) # Evaluation references and internal links. if evaluation[\"caseRef\"][\"caseId\"] != case[\"caseId\"]: errors.append(\"Evaluation caseId does not match case\") check_hash(errors, \"evaluation case hash\", evaluation[\"caseRef\"][\"contentHash\"], case_hash) check_hash(errors, \"evaluation contract hash\", evaluation[\"contractRef\"][\"contentHash\"], contract_hash) check_hash(errors, \"audit case hash\", evaluation[\"audit\"][\"caseInputHash\"], case_hash) check_hash(errors, \"audit contract hash\", evaluation[\"audit\"][\"contractHash\"], contract_hash) finding_ids = [finding[\"findingId\"] for finding in evaluation[\"findings\"]] claim_ids = [claim[\"claimId\"] for claim in evaluation[\"claims\"]] expected_ids = [element[\"elementId\"] for element in evaluation[\"expectedElements\"]] relationship_ids = [rel[\"relationshipId\"] for rel in evaluation[\"evidenceRelationships\"]] run_ids = [run[\"modelRunId\"] for run in evaluation[\"runManifest\"]] validator_run_ids = [run[\"validatorRunId\"] for run in evaluation[\"validatorManifest\"]] validator_ids = [run[\"validatorId\"] for run in evaluation[\"validatorManifest\"]] for label, values in [ (\"finding IDs\", finding_ids), (\"claim IDs\", claim_ids), (\"expected-element IDs\", expected_ids), (\"evidence-relationship IDs\", relationship_ids), (\"model-run IDs\", run_ids), (\"validator-run IDs\", validator_run_ids), ]: dupes = duplicates(values) if dupes: errors.append(f\"Duplicate evaluation {label}: {dupes}\") run_id_set = set(run_ids) validator_id_set = set(validator_ids) for run in evaluation[\"runManifest\"]: for parent_id in run.get(\"parentRunIds\", []): if parent_id not in run_id_set: errors.append(f\"Model run {run['modelRunId']} references unknown parent run {parent_id}\") applicable_assessment_ids = { assessment[\"criterionId\"] for assessment in evaluation[\"criterionAssessments\"] if assessment[\"applicability\"] == \"applies\" } for cid in applicable_assessment_ids: for validator_id in criterion_by_id[cid][\"evidencePolicy\"][\"deterministicValidatorIds\"]: if validator_id not in validator_id_set: errors.append(f\"Applicable criterion {cid} requires missing validator run {validator_id}\") finding_by_id = {finding[\"findingId\"]: finding for finding in evaluation[\"findings\"]} claim_set = set(claim_ids) expected_set = set(expected_ids) evidence_set = set(evidence_ids) for index, finding in enumerate(evaluation[\"findings\"]): fid = finding[\"findingId\"] if finding.get(\"criterionId\") and finding[\"criterionId\"] not in criterion_by_id: errors.append(f\"Finding {fid} references unknown criterion {finding['criterionId']}\") for claim_id in finding.get(\"claimIds\", []): if claim_id not in claim_set: errors.append(f\"Finding {fid} references unknown claim {claim_id}\") for element_id in finding.get(\"expectedElementIds\", []): if element_id not in expected_set: errors.append(f\"Finding {fid} references unknown expected element {element_id}\") for evidence_id in finding.get(\"evidenceIds\", []): if evidence_id not in evidence_set: errors.append(f\"Finding {fid} references unknown evidence {evidence_id}\") source = finding[\"source\"] if source.get(\"modelRunId\") and source[\"modelRunId\"] not in run_id_set: errors.append(f\"Finding {fid} references unknown model run {source['modelRunId']}\") if source[\"kind\"] == \"deterministic_validator\" and source[\"componentId\"] not in validator_id_set: errors.append(f\"Finding {fid} references unknown deterministic validator {source['componentId']}\") for span_index, span in enumerate(finding.get(\"targetSpans\", [])): check_span(errors, f\"findings[{index}].targetSpans[{span_index}]\", span, artifacts) for index, claim in enumerate(evaluation[\"claims\"]): for evidence_id in claim.get(\"citedEvidenceIds\", []): if evidence_id not in evidence_set: errors.append(f\"Claim {claim['claimId']} cites unknown evidence {evidence_id}\") for span_index, span in enumerate(claim[\"attemptSpans\"]): check_span(errors, f\"claims[{index}].attemptSpans[{span_index}]\", span, artifacts) for index, element in enumerate(evaluation[\"expectedElements\"]): if element[\"criterionId\"] not in criterion_by_id: errors.append(f\"Expected element {element['elementId']} references unknown criterion\") for span_index, span in enumerate(element.get(\"observedSpans\", [])): check_span(errors, f\"expectedElements[{index}].observedSpans[{span_index}]\", span, artifacts) for rel in evaluation[\"evidenceRelationships\"]: if rel[\"evidenceId\"] not in evidence_set: errors.append(f\"Relationship {rel['relationshipId']} references unknown evidence\") subject_id = rel[\"subjectId\"] if rel[\"subjectType\"] == \"claim\" and subject_id not in claim_set: errors.append(f\"Relationship {rel['relationshipId']} references unknown claim {subject_id}\") if rel[\"subjectType\"] == \"expected_element\" and subject_id not in expected_set: errors.append(f\"Relationship {rel['relationshipId']} references unknown expected element {subject_id}\") if rel[\"subjectType\"] == \"finding\" and subject_id not in finding_by_id: errors.append(f\"Relationship {rel['relationshipId']} references unknown finding {subject_id}\") for span_index, span in enumerate(rel.get(\"evidenceSpans\", [])): check_span(errors, f\"relationship {rel['relationshipId']} evidenceSpans[{span_index}]\", span, artifacts) assessment_ids = [assessment[\"criterionId\"] for assessment in evaluation[\"criterionAssessments\"]] if duplicates(assessment_ids): errors.append(f\"Duplicate criterion assessments: {duplicates(assessment_ids)}\") for assessment in evaluation[\"criterionAssessments\"]: cid = assessment[\"criterionId\"] if cid not in criterion_by_id: errors.append(f\"Assessment references unknown criterion {cid}\") continue all_finding_refs = ( assessment[\"positiveFindingIds\"] + assessment[\"negativeFindingIds\"] + assessment[\"omissionFindingIds\"] + assessment[\"counterEvidenceFindingIds\"] + assessment[\"unresolvedFindingIds\"] ) for fid in all_finding_refs: if fid not in finding_by_id: errors.append(f\"Assessment {cid} references unknown finding {fid}\") if assessment[\"verdict\"] == \"met\" and criterion_by_id[cid][\"proofPolicy\"][\"positiveProofRequired\"]: confirmed_positive = [ fid for fid in assessment[\"positiveFindingIds\"] if fid in finding_by_id and finding_by_id[fid][\"findingType\"] == \"positive\" and finding_by_id[fid][\"status\"] == \"confirmed\" ] if not confirmed_positive: errors.append(f\"Assessment {cid} is met without confirmed positive proof\") if assessment[\"verdict\"] == \"not_met\": confirmed_failure = [ fid for fid in assessment[\"negativeFindingIds\"] + assessment[\"omissionFindingIds\"] if fid in finding_by_id and finding_by_id[fid][\"status\"] == \"confirmed\" ] if not confirmed_failure: errors.append(f\"Assessment {cid} is not_met without confirmed negative or omission evidence\") if assessment[\"applicability\"] == \"does_not_apply\" and assessment[\"verdict\"] != \"not_applicable\": errors.append(f\"Assessment {cid} does not apply but verdict is {assessment['verdict']}\") for decision_list_name in [\"provisionalDecisions\", \"finalDecisions\"]: decisions = evaluation[decision_list_name] targets = [decision[\"target\"] for decision in decisions] if duplicates(targets): errors.append(f\"Duplicate {decision_list_name} targets: {duplicates(targets)}\") missing = declared_targets - set(targets) if missing: errors.append(f\"{decision_list_name} missing declared targets: {sorted(missing)}\") for decision in decisions: if decision[\"target\"] not in declared_targets: errors.append(f\"Decision targets undeclared target {decision['target']}\") for rid in decision[\"triggeredRuleIds\"]: if rid not in rule_by_id: errors.append(f\"Decision references unknown rule {rid}\") elif rule_by_id[rid][\"target\"] != decision[\"target\"]: errors.append(f\"Decision target {decision['target']} uses rule {rid} for {rule_by_id[rid]['target']}\") for cid in decision[\"decisiveCriterionIds\"]: if cid not in criterion_by_id: errors.append(f\"Decision references unknown decisive criterion {cid}\") for fid in decision[\"unresolvedFindingIds\"]: if fid not in finding_by_id: errors.append(f\"Decision references unknown unresolved finding {fid}\") for feedback in evaluation[\"feedback\"]: if feedback.get(\"criterionId\") and feedback[\"criterionId\"] not in criterion_by_id: errors.append(f\"Feedback {feedback['feedbackId']} references unknown criterion\") for fid in feedback[\"findingIds\"]: if fid not in finding_by_id: errors.append(f\"Feedback {feedback['feedbackId']} references unknown finding {fid}\") # Benchmark links. check_hash(errors, \"benchmark contract hash\", benchmark[\"contractRef\"][\"contentHash\"], contract_hash) check_hash(errors, \"benchmark case hash\", benchmark[\"caseRef\"][\"contentHash\"], case_hash) for criterion_gold in benchmark[\"goldStandard\"][\"criterionGold\"]: if criterion_gold[\"criterionId\"] not in criterion_by_id: errors.append(f\"Benchmark references unknown criterion {criterion_gold['criterionId']}\") for field in [\"positiveEvidenceSpans\", \"negativeEvidenceSpans\"]: for index, span in enumerate(criterion_gold[field]): check_span(errors, f\"benchmark {criterion_gold['criterionId']} {field}[{index}]\", span, artifacts) for target_decision in benchmark[\"goldStandard\"][\"acceptableDecisions\"]: if target_decision[\"target\"] not in declared_targets: errors.append(f\"Benchmark acceptable decision targets undeclared target {target_decision['target']}\") # Standalone human-resolution packet integrity. packet_finding_ids = {finding[\"findingId\"] for finding in resolution_packet[\"contextFindings\"]} request_ids = set(resolution_packet[\"interaction\"][\"request\"][\"contextFindingIds\"]) if packet_finding_ids != request_ids: errors.append( \"Human-resolution packet context findings do not exactly match request contextFindingIds\" ) if resolution_packet[\"interaction\"][\"decisionTarget\"] not in declared_targets: errors.append(\"Human-resolution packet targets an undeclared decision target\") for cid in resolution_packet[\"interaction\"][\"request\"][\"triggerCriterionIds\"]: if cid not in criterion_by_id: errors.append(f\"Human-resolution packet references unknown criterion {cid}\") return errors def main() -> None: schema_failures = validate_examples() if schema_failures: raise SystemExit(1) errors = validate_referential_integrity() if errors: print(\"\\nREFERENTIAL INTEGRITY FAILURES\") for error in errors: print(f\" - {error}\") raise SystemExit(1) print(\"PASS cross-document referential integrity\") print(\"PASS canonical example hashes\") print(\"PASS positive-proof and failure-proof invariants\") if __name__ == \"__main__\": main() "}]