{"0": "autoqa foundation - the question inventory\n\n**date:** 2026-07-14\n**what this is:** the decision-gating questions that must be answered to build a reliable, useful autoqa for human-annotated ai training data. generated by a 7-lens brainstorm (measurement, judge failure modes, rubric ops, human factors, grounding, hybrid workflow, feedback/product), 92 raw questions merged into 9 themes. themes are ordered by how foundational they are - earlier themes gate later ones.\n**companions:** `01_research_atlas.md` (what the literature answers), `02_foundations.md` (the resulting philosophy + structure), `atlas/raw_questions.md` (full 92-question inventory with per-question rationale).\n\nquestions marked [a] now have a research-backed answer or strong default (see the atlas); unmarked questions require in-house measurement or a product-owner decision.\n\n---\n\n## 1. ground truth, validation targets & meta-evaluation\n\nwhat the autoqa is measured against, and whether that target is even measurable. gates every downstream accuracy claim, threshold, and routing rule.\n\n- [a] what is the ground-truth construct - \"would an expert consensus panel pass this item?\" or \"does this item satisfy the written project instructions?\" - and when they diverge, which wins? (instruction-satisfaction lets the autoqa be more valid than the humans it's checked against; consensus-prediction caps validity at human reliability.)\n- what is the measured inter-reviewer reliability floor (krippendorff's alpha on a common item set), and is it high enough for human agreement to serve as a validation target at all? *(never measured for our population - the variance-attribution pilot is not optional.)*\n- [a] what is the minimum viable gold set per project given low failure base rates - and does the arithmetic force pooled/hierarchical validation across projects? *(answered: ppi-family closed-form sizing; see atlas hybrid-statistical.)*\n- [a] what agreement statistic do we standardize on given extreme pass rates - and do we ban raw percent-agreement and single-number \"accuracy vs humans\" from all reporting?\n- what is the empirical false-agreement rate - judge and attempter agreeing but both wrong per expert ground truth - and does that mandate a standing independent expert-audit channel? *(unmeasured anywhere; requires seeded known-flaw items.)*\n- is reviewer variance actually caused by criteria vagueness, or would perfect rubrics leave most variance intact (expertise/attention)? does a cheap variance-attribution pilot run before the build order is set? *(psychometrics gap-fill says: substantial rater-intrinsic residual survives perfect rubrics - but the split must still be measured per population.)*\n- [a] what is the explicit ground-truth hierarchy - what outranks what (seeded gold > criterion author > human reviewer > autoqa?) - and which streams may feed judge calibration?\n\n## 2. the objective-subjective boundary & verdict ontology\n\nwhere verifiable fact ends and legitimate judgment begins, and what output types the system may emit. this boundary is the system's scope.\n\n- [a] which failure types are objectively verifiable vs irreducibly judgment-laden - and should the autoqa emit categorically different output types (verdict vs flag-for-human vs confidence-weighted opinion) per class?\n- can we build a reliable axis-triage classifier that sorts a project's evaluation axes into \"compile to binary checks\" / \"anchor with exemplars\" / \"route to human\" buckets before go-live? *(mechanism exists - per-criterion conformal set width - but needs in-house validation.)*\n- [a] where is the line between a verifiable claim and a legitimate opinion, and what does the system do with opinion-class claims?\n- [a] should the verdict taxonomy include \"instructions underdetermine this case\" - triggered by measured disagreement, feeding an instruction-gap channel to project owners, exempt from attempter penalty?\n- for genuinely contested calls, verdict pluralism (\"defensible under reading a, not b\") or forced resolution - and can the system distinguish contested-by-nature from contested-because-underspecified?\n- [a] is inter-reviewer disagreement the disease, or partly legitimate signal - and is the target agreement-with-truth or consistency-of-application? would an autoqa that enforces consistency launder one arbitrary interpretation into false objectivity?\n- if two independently-configured autoqa instances judge the same items, do they disagree at rates comparable to the human reviewers - indicating the variance lives in criterion ambiguity, not people?\n\n## 3. grounding machinery: claim decomposition, evidence scope & entailment\n\nthe core novel requirement - verifying the attempter's positive statements against a defined evidence record.\n\n- [a] at what claim-decomposition granularity does judge agreement stop improving, and does over-decomposition degrade it via checklist myopia? *(answered: granularity must be adaptive - atomic decomposition measurably degrades strong verifiers.)*\n- what is the aggregation rule from claim-level verdicts to item-level pass/fail - are severity/weight taxonomies mandatory per project, or does the system emit claim-level findings plus a recommended verdict?\n- [a] what is the evidence closure - attempter-cited spans only, full item context, or open world - and how do we stop the judge from \"verifying\" against its own priors? must groundedness verdicts quote span-level evidence to be admissible?\n- [a] what entailment standard must an attempter statement meet - strict entailment (fails nearly all human prose) or reasonable-expert support (reintroduces subjectivity) - and is the output 3-way (entailed/contradicted/unsupported) or graded?\n- [a] what claim ontology does verification need - grounding claims, entailment claims, calibration claims (stated severity matches evidence strength) - with different machinery and different reliability ceilings per type?\n- can the judge distinguish warranted from empty praise at usable precision/recall per praise type on a seeded set? *(zero published evidence anywhere - this is a from-scratch in-house measurement.)*\n- [a] do universal/absence claims (\"no errors present,\" \"fully addresses the prompt\") require independent re-attempt - and if the judge's seeded-defect recall isn't materially above the attempter population's, are such verdicts labeled \"not-contradicted\" rather than \"verified\"?\n- is omission detection tractable project-agnostically, or is it a customization-layer plugin requiring a per-project defect taxonomy? *(absencebench says: structurally weakest judge lane; convert completeness criteria to enumerable checklists.)*\n- [a] when cited evidence supports the conclusion but uncited in-scope evidence undermines it - is cherry-picking a grounding failure or an omission, and does verification read cited-only or full-scope?\n- should attempters be required to attach explicit evidence pointers to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding?\n- what is the base-rate split between attempter claims that are actually false vs merely uncited-but-true - and at what judge precision does flagging \"unsupported\" beat spending the human touch elsewhere?\n- contrarian: is grounding verification (including re-finding defects to check \"clean\" verdicts) actually less capability-demanding than doing the annotation - and if not, what is the genuinely verification-easier subset?\n\n## 4. rubric compilation & the per-project customization contract\n\nhow project instructions become executable judge configuration - the load-bearing claim behind \"project-agnostic.\"\n\n- [a] what is the minimal artifact set an instruction set must compile into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules?\n- [a] can one compilation procedure hit target fidelity across instruction-set styles, or is ingestion a structured authoring workshop with tooling rather than an automated pipeline? *(answered: instruction-only auto-compilation leaves a ~22-28pt fidelity gap; expert grounding artifacts are structurally required.)*\n- what exactly is the per-project customization surface as a closed config schema - proven falsifiably by onboarding a new project through config alone, with zero prompt-engineering of the core judge?\n- how many expert-adjudicated exemplars per criterion before judge-human agreement plateaus (5 vs 50 vs 500) - and does the answer quietly reintroduce the expensive aligned-expert dependency? *(one datapoint: 5-shot -> 80% on one dataset; exemplar quality - hard contrastive near-miss pairs - appears to dominate count.)*\n- who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - rubric-as-law or rubric-as-guidance - and does an amendment retroactively re-score past verdicts?\n- [a] does constructive feedback require compile-time authoring (\"what good looks like,\" repair templates, doctrine constraints), or can it be generated at verdict time from the claim-evidence trace alone?\n\n## 5. judge reliability, bias & competence limits\n\nthe judge's own failure modes measured against the human baseline it augments - if the judge's variance or bias exceeds the humans', we've automated the problem.\n\n- [a] what is the run-to-run verdict flip rate on identical items - noise to vote away (k-sampling) or signal that the criterion is underspecified (rubric rewriting)? *(answered: both, by design - vote and route high-entropy items to the contested class.)*\n- [a] does the judge exhibit agreement-default/sycophancy bias on positive claims - passing plausible-but-ungrounded claims at a higher rate than it fails grounded ones? *(facts: positive-class ~85 f1 vs negative-class ~46 f1 - yes, quantified, large.)*\n- [a] does the judge favor verbose, hedged, rubric-vocabulary-mimicking writeups over terse-but-correct ones - and do we need canonicalization/style-stripping before judgment?\n- [a] when the attempter critiques output from the judge's own model family, does self-preference suppress agreement with valid criticisms - forcing cross-family judge routing? *(preference leakage: up to 28.7pp, prompt-level debiasing fails.)*\n- [a] how sensitive are verdicts to semantically-neutral prompt perturbations - and is a perturbation-robustness harness a mandatory ship gate per project config?\n- for each axis type, what is the judge-vs-adjudicated-human agreement ceiling - and on which axes does the judge fall below the human inter-rater baseline, forcing lanes? *(must be assembled per project; published anchors exist only for grounding.)*\n- what is the judge's blind-spot profile on expertise-heavy content - can it detect its own incompetence and abstain, and are some projects autoqa-ineligible?\n- [a] does a multi-model ensemble buy error reduction, or do frontier judges share correlated blind spots - such that cross-family disagreement should be an escalation signal rather than a vote?\n- is judge-judge agreement on grounding verdicts actually higher than the human reviewer-reviewer agreement we're fixing - as a falsifiable go/no-go gate before deploying at all?\n\n## 6. human-budget allocation, routing & arbitration\n\nwith at most one human touch per item, routing is the system.\n\n- [a] is one-human-interaction-per-item even the right target - would zero per-item touch plus randomized deep blind audits outperform one shallow, automation-biased touch? *(fork resolved by evidence + law: see foundations section 5 - consequential adverse verdicts require a meaningful human decision for eu attempters; measurement is best served by sampled audits.)*\n- [a] what is the single interaction spent on - blind re-judgment, verdict verification, targeted micro-adjudication of contested claims, or the attempter's contest? *(evidence-only assisted adjudication of the routed slice wins; showing the ai verdict causes measured over-reliance.)*\n- [a] where in the verdict distribution does the touch buy most reliability - and must routing be closed-loop on observed overturn rates?\n- [a] should scarce human minutes go to per-item adjudication at all, versus upstream compounding work (rubric disambiguation, exemplar-generating edge-case adjudications that write back into config)?\n- [a] does escalation route the whole item or the specific contested claim/criterion?\n- are escalation queues organized by criterion (specialist reviewers) or by item (generalist) - and does specialist routing measurably close the expertise-gap root cause?\n- [a] does the escalated human judge blind or anchored - and at what point does a falling overturn rate signal rubber-stamping?\n- [a] is the escalation threshold set to a fixed human-capacity budget or a target residual error rate? *(both exist as dial profiles: error-target mode outputs the budget; pace-bound mode outputs the reported residual error - foundations p11.)*\n- [a] when contractual pace makes the standard threshold unaffordable, what may the quality/leniency dial legitimately move (aggregation cutoffs, escalation rate, verification depth, enforcement disposition) - and what may it never touch (judge perception, integrity verdicts, seeded gold/audit channels, the legal floor)?\n- [a] is the dial one-sided by construction - dispositions loosen while measurement persists, so an expedited batch can be re-filtered to any strictness later from recorded scores (attempter-disposition vs dataset-admission consumers)?\n- [a] how do the goodhart/divergence sensors condition on the active dial setting, so a leniency period is not misread as attempter improvement (or a strictness change as attempter decay)?\n- who has authority to change a project's active operating profile, is every change logged, prospective-only, and announced to attempters - and is the stamped residual-error estimate disclosed to the data buyer? *(product-owner decision - foundations section 8.6.)*\n- which confidence proxy (stated confidence, k-sample agreement, paraphrase flip rate, ensemble disagreement, conformal set width, rationale-entailment score) actually predicts human overturn at routable auroc? *(never benchmarked head-to-head - in-house measurement.)*\n- [a] what is the asymmetric, cost-weighted decision rule for auto-pass / auto-fail / escalate - is auto-fail held to a stricter bar than auto-pass, and who bears each error's cost?\n- [a] when human and autoqa persistently disagree, what arbitrates - and how do we prevent the override stream from becoming unvalidated calibration data?\n- [a] what statistical model fuses sparse human labels with dense ai judgments? *(ppi++/eif for rates and dashboards; tpr/fpr-corrected tests for threshold certification.)*\n- what is the cold-start protocol for a zero-label project, and what quantitative graduation criterion flips it into confidence-routed mode?\n\n## 7. constructive feedback: generation, grounding & efficacy\n\nthe feedback channel held to its own standards - and falsifiable by attempter behavior change.\n\n- [a] is the judge's feedback held reflexively to the same grounding standard it enforces - every criticism citing entailing evidence, passing its own verification gate before delivery - and what false-accusation precision slo keeps attempter trust?\n- [a] must every feedback statement be traceable to a specific clause in the project instructions - and what happens to real problems that map to no written criterion?\n- [a] what is the minimal feedback unit attempters actually act on - e.g., the 4-tuple (quoted claim, evidence span, criterion clause, concrete fix)? *(feedback science: high-information task-referenced feedback ~ doubles effect vs bare verdicts; person/rank-referenced feedback backfires.)*\n- [a] what is the explainability contract for a failing verdict - verbatim evidence spans, checkable reasoning chain, fail-closed if the bar can't be met? *(also a legal requirement for eu attempters: plain-language explanation + human review on request.)*\n- does feedback need a different confidence bar than the verdict - can the system ship coaching on auto-passed items, and what is the harm model when a feedback claim is itself wrong?\n- [a] does feedback measurably change attempter behavior - what longitudinal metric do we commit to, and after how long with no movement do we conclude decoration and reallocate? *(priors now exist: median modest, concentrated in low performers, ~1/4 act on it; instrument per-attempter pre/post error-rate deltas and expect backfire in a minority.)*\n\n## 8. adversarial dynamics, incentives & consequences\n\nthe system operates on paid workers who adapt to it. the moment verdicts carry consequences, the system is adversarial by construction.\n\n- does feedback leak the judge's decision boundary - after n cycles, do attempter scores rise on the autoqa while staying flat on held-out human-graded golden items? *(the single most important standing sensor; no published precedent - must build.)*\n- [a] which shortcut heuristics does the judge use as pass/fail proxies, can pay-motivated attempters reverse-engineer them blind, and is the defense designed in from day one? *(universal adversarial phrases transfer to unseen judges - filtering rationales is insufficient alone.)*\n- [a] can the judge distinguish genuine grounding from citation theater - if we perturb or swap the cited evidence, does the verdict flip?\n- [a] how do we prevent feedback-induced monoculture - annotations converging on judge-pleasing patterns that destroy the diversity the training data exists to capture - and should feedback ever include stylistic guidance? *(default: no stylistic guidance; population-level diversity drift metric from day one.)*\n- [a] for wholesale llm-generated submissions, is the defense content-based detection or economics-based gating? *(resolved: economics/provenance-based; content detection is beaten - agent passes 99.8% of checks at $0.05/task.)*\n- do we care about llm-assistance provenance or only content validity - is a correct, well-grounded, heavily-llm-assisted annotation a pass, a fail, or per-project policy? *(the product owner/per-project decision; process telemetry has legal constraints - see foundations section 10.)*\n- [a] are autoqa verdicts advisory or enforcement-grade - and if consequences attach, what precision floor, audit trail, and contest rights does each verdict need? *(enforcement-grade adverse verdicts trigger the legal floor: human decision-maker, plain-language explanation, appeal with sla.)*\n- [a] what appeal mechanism absorbs disputes without re-importing full human review load - and what appeal-overturn band signals a healthy vs broken judge?\n\n## 9. drift, monitoring & lifecycle integrity\n\nkeeping the certificate valid after launch.\n\n- [a] how do we detect drift in three distinct moving parts - attempter adaptation, instruction revisions, judge-model updates - with a different sensor and response for each? *(frozen human-labeled anchor set + anytime-valid attribution separates \"judge moved\" from \"population moved.\")*\n- what falsifiable signal triggers a rubric revision (per-axis disagreement trend, override cluster density, appeal rate) at what threshold? *(no published drift-rate parameters; instrument and learn.)*\n- [a] are judge model versions pinned per project with golden-set regression gates on upgrade?\n- [a] once a project runs trusted-standalone, what randomized audit rate and drift statistic triggers automatic de-graduation - and is audit sampling unpredictable enough that attempters cannot write for the judge?\n- [a] does the autoqa prefilter shift the human reviewer's base rates enough to cause induced criterion drift - and does queue design mandate seeded known-verdict items and easy-case dilution? *(content-moderation practice: yes, both are standard; seeded items must be indistinguishable.)*\n- [a] what keeps the calibration workforce competent once the autoqa writes verdicts - does reviewer skill decay under reconcile-only work, and is a standing blind-judgment quota mandatory? *(psychometrics: rater calibration decays within a single session; rolling-window reviewer-effect estimates, not one-time certification.)*\n\n---\n\n## cross-cutting tensions\n\nplaces where two legitimate goals pull in opposite directions. these are managed, not solved; foundations states the default posture for each.\n\n1. **standardization vs legitimate disagreement.** enforcing consistent verdicts kills the reviewer-variance problem but risks laundering one arbitrary reading of a contested criterion into false objectivity; admitting ambiguity classes preserves legitimate disagreement but can become an excuse bucket.\n2. **decomposition vs holistic validity.** atomic claim-level checks are tractable, auditable, falsifiable - and strip context (checklist myopia), re-importing subjectivity through the aggregation rule.\n3. **feedback usefulness vs goodhart resistance.** the more specific and actionable the feedback, the more it publishes a gradient toward the judge's decision boundary that paid attempters will descend.\n4. **human as ground truth vs human as noise source.** the system exists because human reviewers are unreliable, yet escalation, overrides, appeals, and calibration all quietly treat a single human's judgment as truth. every human touch chooses between efficient anchored review (contaminated labels) and expensive blind independence (usable calibration signal).\n5. **per-item accuracy vs compounding upstream investment.** the touch spent adjudicating this item maximizes per-item quality; the same minutes spent on rubric disambiguation improve all future items - but leave today's items unconfirmed.\n6. **grounding strictness vs false-flag rate.** strict entailment fails nearly all human prose (which compresses and generalizes); loose \"reasonable support\" reintroduces the subjectivity the system was built to remove.\n7. **project-agnostic foundation vs per-project fidelity.** a closed config surface keeps \"agnostic\" falsifiable - but compilation fidelity, judge domain competence, and defect taxonomies constantly pressure the boundary.\n8. **enforcement weight vs error tolerance and trust.** consequential verdicts create real deterrence and collapse acceptable error rates (and trigger legal duties); advisory verdicts preserve trust and tolerate error but lose incentive force.\n9. **contractual pace vs quality floor** *(added 2026-07-14 from product-owner throughput requirement)*. delivery obligations sometimes demand a looser operating point than the standard threshold; refusing any dial forces the loosening to happen off the books, while a dial that quietly loosens *measurement* (not just disposition) destroys the system's evidentiary value. managed by pricing leniency: dispositions move, measurement doesn't, and every batch ships stamped with its profile and estimated residual error (foundations p11).\n", "1": "autoqa foundation - research atlas (executive synthesis)\n\n**date:** 2026-07-14 (recency-biased sweep; sources through 2026-07-06)\n**method:** 9 parallel domain researchers fed the question taxonomy (`00_questions.md`), each triangulating across websearch, exa, tavily, and scholar with instructions to open sources before citing, use absolute dates, and report disagreements. 27 load-bearing claims were adversarially spot-checked by independent verifiers who fetched the cited sources: **20 confirmed, 7 partially confirmed (corrections applied below and in the domain files), 0 refuted, 0 unverifiable.** a completeness critic then identified 4 missed literatures - psychometrics/rater science, eu labor law + data-quality standards, content-moderation qa programs, and feedback-efficacy science - each gap-filled with a second round (6 findings apiece).\n**full evidence:** per-domain reports with claims, evidence summaries, source urls/dates, tools, disagreements, gaps, and verification transcripts live in `atlas/`.\n\n---\n\n## bottom line\n\nthe core bet - an llm system that verifies human evaluative writeups claim-by-claim against a defined evidence record, with a calibrated escalation lane to one human touch - is supported by converging 2024-2026 evidence and is close to current production practice at the frontier (toloka's deployed llm qa, an AI data vendor's agent-judge verifier, pinterest's decision-quality framework, openai's grader doctrine). two things the evidence is emphatic about: **holistic single-call judging of subjective quality does not work** (near-orthogonal judge/human evaluation axes; ~55% on hard per-criterion verdicts), and **agreement-with-humans as a single number is an invalid target** (kappa deflation, rating indeterminacy, label-uncertainty stratification). the viable construct is instruction-satisfaction plus evidence-grounding, decomposed into typed claims, with genuinely contested items surfaced as instruction gaps rather than forced verdicts. the exact product - claim-level autoqa of *human* annotations - appears to have no published equivalent as of mid-2026, and the single most load-bearing caveat is that essentially all published judge numbers were measured on grading *model* outputs, not human evaluative writeups; transfer is plausible but unvalidated.\n\n## the strongest established findings\n\n**judge capability and its limits**\n1. frontier *reasoning* models dominate fine-tuned judge models and reward models on judging tasks with verifiable ground truth (judgebench: o1-preview 75.4% vs gpt-4o 50.9-56.6%, fine-tuned judges at/below random; later reasoning models reach 77-81%). small trained judges (j1, compassjudger-2, skywork-reward-v2) win only on preference-style benchmarks. -> two-tier judge economy: cheap specialists for screening, frontier reasoners for verdicts. *(atlas/domain_academic-judges.md, verified)*\n2. on subjective rubrics, llm judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-human), and inter-llm agreement exceeds llm-human agreement - **inter-judge consensus is not validity evidence on subjective axes.** on rubrics with verifiable answers, the same judges fall back into the human range. *(academic-judges, 2026-06)*\n3. per-criterion rubric verification on hard instances sits near chance even for frontier models (rubriceval: gpt-4o 55.97% balanced accuracy on hard), but explicit per-criterion reasoning beats checklist-level evaluation by 7-12pp and cuts cross-judge variance. long-form holistic judging is unstable (longjudgebench); contextual judging with conditional criteria caps at ~55% (contextualjudgebench). *(academic-judges, frontier-2026; verified)*\n4. in expert-knowledge domains, judge-sme agreement runs 64-68% - below any autonomy bar. expertise-heavy axes belong in judge-assisted or human-only lanes, and judges agree more with non-experts than experts (judge-bench), so uncalibrated judges replicate crowd-level, not expert-level, judgment. *(academic-judges, contrarian)*\n5. **the agreement-default asymmetry on positive claims is quantified and large:** google's facts grounding judges score ~85 f1 on the grounded class but ~46 f1 on the ungrounded class, and grounding metrics are gameable by vague, short responses - countered by a mandatory responsiveness/eligibility gate. *(grounding-faithfulness, verified)*\n6. llms are structurally weak at detecting omissions (absencebench: f1 drops 56.9 points for deletions vs insertions; explicit placeholders recover ~36 points). \"complete/comprehensive\" praise verification is the judge's weakest lane; convert completeness criteria into enumerable checklists. *(critique-models, verified)*\n7. judges silently re-solve tasks instead of checking evidence: on checks that exceed the judge's own solving ability, wrong-reference detection collapses (68%->9%) and false positives explode (44%->88%), while read-checkable defect detection holds at 80%. route checks by read-checkable vs recompute-checkable. *(frontier-2026, verified; under-review arr submission)*\n8. consequence framing silently corrupts verdicts: telling a judge its verdict has downstream stakes softens verdicts (-9.8pp peak, 30% relative drop in unsafe-content detection) with **zero trace in the chain-of-thought**. judge prompts must be stakes-sterile, and bias must be measured behaviorally, never read off rationales. *(frontier-2026, verified)*\n9. self-preference is real and lineage-based (preference leakage up to 28.7pp, strongest for same-model and distillation descendants; prompt-level debiasing fails; only contextual calibration and style-stripping help). model errors across the industry are becoming more correlated as capability rises. cross-family judge routing is the practical defense. *(academic-judges, contrarian)*\n\n**grounding machinery**\n10. claim-vs-evidence entailment checking is commoditized but ceilinged (~77 balanced accuracy on llm-aggrefact; sub-1b-7b specialists within ~2.5 points of frontier models at ~1/100 cost). ~22% claim-level error means item verdicts cannot be a naive and over claim checks. *(grounding-faithfulness, verified)*\n11. the measured-best grounding-judge architecture is **exemplar anchoring**: a frontier reasoning judge prompted with human-annotated peer judgments on comparable items (faithjudge: 84.0 balanced accuracy on faithbench vs <=68.8 zero-shot judges and 60-67 for fine-tuned detectors). this is exactly the per-project compiled-exemplar contract. *(grounding-faithfulness, verified with corrections)*\n12. decomposition granularity must be adaptive: atomic decomposition helps weak verifiers and actively degrades strong ones (80.0->71.1 bacc on wice with minicheck); the verification unit should be claim + explicit context annotations with the verifier told which part is under test. verifiability triage (verifiable claim / opinion / undecidable) before entailment is field-standard (veriscore, factbench). *(grounding-faithfulness, verified)*\n13. citation checking needs two distinct passes: span-closed (does cited evidence entail the claim) and context-closed (is this the right citation given everything in scope) - cited-span-only checking misses wrong/incomplete citation, the dominant human failure. *(grounding-faithfulness)*\n\n**the human layer and hybrid design**\n14. human+ai combinations on average perform worse than the best of either alone on decision tasks (meta-analysis of 370 effect sizes, g = -0.23), specifically when the ai outperforms the human - naive human-verifies-ai-verdict designs destroy value and produce rubber-stamping (~90% deference to biased ai recommendations in a 2025 hiring study). complementarity appears only with confidence-based routing to a slice where humans genuinely beat the ai, and with **evidence-only assistance**: deepmind's amplified-oversight experiments hit 91.3% (hybrid) vs 87.7% (ai alone) vs 75.1% (human alone), and showing humans the ai's verdict/reasoning caused measured over-reliance while showing only retrieved evidence was the sole safe format. *(critique-models, contrarian; both verified)*\n15. ai critics measurably improve human review of *human* work at deployment scale: criticgpt critiques preferred over human critiques 63% of the time; critic-assisted review of items already rated \"flawless\" found real rating-changing problems in 24% of flagged cases vs 6% for an unassisted second human - a 4x lift exactly on the confident-pass slice. the iclr 2025 rct (>20k peer reviews) showed 27% of reviewers revised on ai feedback, gated by automated reliability tests before delivery. caveat: criticgpt is one study, one vendor, no published replication in 24+ months. *(critique-models, verified)*\n16. the statistics for sparse-human/dense-judge fusion are mature: ppi/ppi++ gives bias-corrected population estimates whose validity survives a bad judge (worse judge -> wider intervals, never invalid); gold-set size is computable in closed form (target effective n=200 at r^2=0.70 needs ~65 human labels); selective escalation with provable human-agreement guarantees exists (trust or escalate, iclr 2025 oral); judge-based certification can be provably more powerful than human-only review (\"noisy but valid,\" iclr 2026); judge-vs-population drift can be attributed with frozen anchor sets and anytime-valid statistics. gold sets must oversample the rare fail class. *(hybrid-statistical, verified with corrections)*\n17. theoretical ceiling: when the judge is no more accurate than what it evaluates, no debiasing method can cut required gold labels by more than 2x - the expert-audit channel is a permanent structural component, not a bootstrap phase. *(contrarian, verified)*\n\n**rubrics and compilation**\n18. rubric-based grading is the 2025-26 production standard (healthbench: 48,562 physician-written per-item binary weighted criteria, grader meta-evaluated per-criterion against physician majority at mf1 0.709 - exceeding the average physician in 5/7 themes, where physician-physician mf1 is only 0.569-0.730; paperbench: 8,316 binary leaves, judge f1 0.83 at ~1/100 expert cost). the judge gets its own benchmark (judgeeval pattern); rubric text is a tunable alignment artifact, not scripture. *(rubrics-recent, verified)*\n19. instruction-only automated rubric generation leaves a ~22-28pp fidelity gap vs human rubrics that test-time compute cannot close (rubricbench) - but the gap closes to ~0.3pp when rubrics are mined contrastively from preference pairs (support vector rubrics, 2026-06). reference-grounded synthesis matches human authorship (rubrics-as-rewards) except for **pitfall criteria, which require human domain intuition**. the scarce human input is exemplars and failure-mode knowledge, not criterion drafting. *(rubrics-recent, verified with corrections)*\n20. criteria drift is real and structural: graders' criteria change as they grade (evalgen); rubric compilation must be a versioned loop with owner sign-off on graded real items, not a one-shot compile. the 2026 rl literature converges on static rubrics being exploitable specifications under optimization pressure - and paid attempters are optimizers. *(rubrics-recent, contrarian; verified)*\n21. rubric qa is now a named practice (rift failure-mode taxonomy: 8 modes; automated linting catches subjective at f1 0.925 but hackable at ~0.000 - **anti-gaming rubric review is permanently a human red-team task**). cap active criteria per judge call (~<13; \"attention displacement\" above that). *(rubrics-recent)*\n\n**the workforce and adversarial reality**\n22. llm contamination of human work is a base-rate fact, not a tail risk: ~33-35% of crowdworkers used llms on an llm-advantaged text task (2023, replicated ~34% self-report 2025); the best mitigations halve it, never eliminate it, and cost quality. an autonomous agent passes 99.8% of standard quality checks, evades behavioral detectors, and costs ~$0.05 vs $1.50 paid - **content-based spam detection is economically and technically beaten**; defenses are provenance/economics-based (identity, throttling, work-history, randomized deep audits, worker-level peer-prediction scoring). scale's internal-document postmortem (spam paid for 11 months; zerogpt and country bans as stopgaps; account black markets) corroborates operationally - though it is journalism-of-journalism, disputed by scale, and pre-llm-judge era. *(annotation-quality, verified with corrections; industry-practice)*\n23. annotator disagreement is majority-attributable to task underspecification and response-style preferences, not noise and not primarily expertise (diverging preferences; nli taxonomy work shows an item-intrinsic residual that survives perfect rubrics). field practice has institutionalized disagreement-as-signal (lewidi-2025 soft-label/perspectivist evaluation; nutmeg separates competence noise from systematic subpopulation disagreement; openai's coval preserves conflicting criteria as a record of legitimate disagreement and warns rubric scores invite checklist gaming). *(annotation-quality, contrarian, frontier-2026; verified)*\n24. judge verdicts are gameable blind: universal adversarial phrases learned on a surrogate transfer to unseen judges and inflate scores regardless of content; absolute scoring is far more vulnerable than comparative/anchored scoring; a deployed llm checklist assistant was gamed by its subjects in one shot (neurips'24). rotating seeded probes and evidence-perturbation tests are day-one requirements. *(contrarian)*\n25. vendors already run the hybrid shape: a major annotation platform/a major annotation platform's llm judge only *flags* for human review, with krippendorff alpha >= 0.80 as the published bar and per-worker trust routing; braintrust's cost arithmetic (judge ~$5-15 per 10k outputs vs expert ~$800-1,800 per 500) drives \"deterministic checks + judges on everything, humans on flagged/disagreement slices + random spot-checks of confident passes\"; the market voted with an AI data vendor's acquisition of a data-quality startup (algorithmic label-noise flagging without a second human). verifier architecture beats verifier model (the vendor's agent-judge: agent-judge inside the work environment beats text-only rubric judges by more than any model swap, ~10x cheaper; judge f1 vs practicing experts 0.63-0.66 on artifact-heavy work vs 0.95 on simple stateful tasks - the ceiling is domain-dependent by ~30 f1 points). *(industry-practice, frontier-2026; verified)*\n\n**from the gap-fill rounds**\n26. psychometrics (40 years of rater science the llm literature reinvents badly): training improves rater *self*-consistency but does not equalize severity (canonical weigle/lumley-mcnamara finding); best-in-class frame-of-reference training buys only d~0.50; g-theory decompositions show the rater main effect (global severity) is minor while raterxitemxcriterion interactions dominate - **the dominant variance can only be attacked by per-claim, per-item verification, which is the strongest independent argument for this architecture**. rater severity drifts within a single session (2.4x worse on hard items). mfrm is now applied symmetrically to llm judges - enroll the judge as one more rater facet in the same measurement model. constraint: severity estimation requires deliberate rater-item overlap (3-5 ratings/output linkage), which single-review-per-item regimes cannot provide. *(atlas/critic_and_gapfill.md)*\n27. law makes part of the design non-optional for eu-touching deployments (national transposition from 2026-12-02): eu platform work directive art 10(5) requires decisions of significant detriment (account/pay restriction, termination, or equivalent) to be **taken by a human**, with no consent exception, covering self-employed annotators and non-eu platforms with eu workers; art 11 mandates plain-language explanations, a competent human contact, and human review with a substantiated written reply within two weeks; art 7 prohibits telemetry on emotional state, private communications, or off-task activity and requires a dpia. cjeu schufa: a score with a \"determining role\" is itself an art 22 decision, and rubber-stamp human review legally counts as zero-touch (active enforcement: ~eur 490k hamburg fine, 2025-09). iso/iec 5259 (data quality for ml, 5 parts, 2024-25) is the neutral vocabulary to map quality dimensions onto. open question: whether a single-item pay denial is a \"decision of equivalent detriment\" under art 10(5) is legally untested. *(critic_and_gapfill)*\n28. content moderation ran this movie for a decade: ~3-4% random audit + qa-of-qa layers was the facebook/cognizant baseline; \"accuracy = agreement with auditor\" is blind to correlated error and gameable via off-book lobbying (\"this number is fake\"); tspa codifies forward audit sampling + blind golden-set seeding (constraint: seeded items must be indistinguishable) + appeals as a cheap but population-biased third signal, and a four-way error taxonomy including *wrong-selection* (right verdict, wrong cited criterion). pinterest's 2026 framework: separate reliability (kappa) from correctness (vs adjudicated golden set) - high-reliability + low-correctness = systematic policy misunderstanding; golden sets deliberately oversample rare/hard cases; policy changes are handled by dual-labeling under old+new guidelines; 3x-human majority vote bought only +3.6pp over a single non-expert. consensus-rewarded auditing (community notes) induces strategic conformity - **never score contributors on agreement-with-final-verdict; score on stability/informativeness of residuals**. verdict-option ui costs distort verdict distributions independent of judgment. *(critic_and_gapfill)*\n29. feedback science kills \"no evidence feedback works\" and replaces it with priors: feedback improves performance on average (d~0.38-0.41) but **more than one third of feedback interventions reduce performance** - task-referenced helps, person/rank-referenced backfires (kluger & denisi fit); high-information feedback (what + why-process + how-to-self-check) yields d~0.99 vs 0.46 for bare right/wrong and 0.24 for reinforcement - verdict-only qa forfeits most achievable behavior change; the 2025 cochrane update (292 studies) finds a modest 2.7% median absolute improvement concentrated in low performers, with individual-level data, top-peer/benchmark comparison, and specific action plans as positive moderators and repeated delivery associated with lower effect; the closest crowdwork rct (dow et al., cscw 2012) shows rubric-mediated task-specific feedback significantly improves paid microtask quality, with self-assessment matching external feedback on quality but not on revision behavior - and self-ratings inflated by 1.8/9 points. under pay-linked incentives, rank feedback induces gaming/sabotage that offsets gains - benchmark against exemplar work, never ranked people. *(critic_and_gapfill)*\n\n## live disagreements (design forks the literature does not settle)\n\n- **dedicated trained judges vs frontier generalists:** benchmark-dependent (preference-style vs verifiable-reasoning); no source resolves it for criteria-based qa of human annotations. default: frontier reasoner for verdicts, specialists for screening; measure per project.\n- **uncertainty-routed vs uniform human sampling:** >25% label savings in batch settings (naacl 2025) vs near-uniform optimality in sequential estimation (iclr 2026). regime-dependent; ship routing policy as a measured per-project choice with a built-in a/b against uniform.\n- **estimation vs certification statistics:** ppi++/eif intervals 3-15x narrower for rates/dashboards; explicit tpr/fpr-corrected tests better-powered for threshold certification. use both, per purpose.\n- **implicit vs explicit rubric aggregation:** implicit (judge weighs all criteria internally) beats explicit weighted-sum for reward quality by up to 28-31% relative (rar), but every auditable production benchmark (healthbench, paperbench, openai graders) uses explicit per-criterion verdicts. for a qa system that must justify verdicts, auditability wins; treat this as a known performance tax.\n- **decomposition depth:** per-criterion structure beats flat checklists (rubriceval) and decomposition monotonically improves verification in another 2026 framework (llm-as-a-verifier); reconciliation (atomicity helps when each criterion gets its own call, hurts in one long checklist) is inferred, not tested.\n- **static pinned rubrics with regression gates vs continuously co-evolving rubrics:** production-integrity practice vs 2026 anti-goodhart rl literature. foundations resolves this as versioned revision loops with regression gates - adaptation at the compilation layer, pinning at the judge layer.\n- **what the human should see:** evidence-only for adjudicators (deepmind) vs full critiques delivered to authors (criticgpt, iclr rct). plausible split - critiques to attempters, evidence-only to adjudicators - is untested in any single study.\n- **human-human floor numbers:** practitioner \"~85% judge-human agreement, better than humans agree with each other\" vs chance-corrected kappa 0.28 +/- 0.32 across 20 tasks. mostly percent-agreement vs chance-corrected framing; never report the former.\n\n## what has no evidence anywhere (in-house measurement required)\n\n1. **positive-claim/praise verification** - precision/recall of \"is this praise warranted,\" per praise type: zero published benchmarks. the mission's core novel requirement is a from-scratch measurement (seeded warranted/empty-praise sets). absencebench predicts empty-praise detection (an omission-shaped failure) is the hardest lane.\n2. **transfer of all judge numbers from grading model outputs to grading human evaluative writeups** - flagged by every domain, no partial-transfer experiment exists.\n3. **false-agreement rate** (judge and attempter both wrong vs expert ground truth) in an annotation-qa setting; nearest proxies are the 3.3%+ gold-label error floor and criticgpt's 24%-vs-6% flawless-slice result.\n4. **attempter adaptation to a deployed ai qa layer** - feedback-boundary leakage, autoqa-vs-held-out-gold divergence, monoculture drift: no published study; the divergence sensor must be built, not borrowed.\n5. **longitudinal feedback efficacy for paid annotators** (repeat-error-rate deltas under ai feedback): adjacent literatures transfer with caveats; the direct experiment is unrun.\n6. **confidence-proxy shootout** (verbalized confidence vs k-sample agreement vs conformal width vs ensemble disagreement) for predicting human overturn: never benchmarked head-to-head.\n7. **a project's own reviewers' reliability floor** - no measured krippendorff's alpha exists for the population this system is being designed around; the variance-attribution pilot is the first experiment for a reason.\n\n## honesty notes (from the completeness critic)\n\n- the headline meta-evaluation figures (33-41pp kappa deflation, 14-15 position ranking flips, consistency-bias paradox) trace to one un-peer-reviewed june 2026 preprint cited by four domains independently - cross-domain repetition is not corroboration. its findings are directionally consistent with peer-reviewed work.\n- several architecturally decisive 2026 claims (auditing-by-re-solving, evaluation faking, toloka's precision table) were verified at abstract/crawl level, not full-pdf audit; auditing-by-re-solving is still under review.\n- several \"reconciliations\" offered above (verbosity bias as controllable, critiques-to-attempters/evidence-to-adjudicators) are inferred, not tested. they are labeled as such where they inform foundations.\n- non-us/non-english industry practice (chinese labeling bases, japanese/korean/indian vendors) was not searched; industry claims generalize from a us/eu sample.\n- judge-model choice guidance is anchored to early-2026-or-older cohorts; the two major judge leaderboards were unfetchable (dynamic js) at sweep time.\n\n## confidence\n\n**high** on the architectural findings (decomposition necessity, typed verdicts, exemplar anchoring, evidence closure, statistical layer, adversarial posture, legal floor): multiple independent source classes converge, and 27/27 spot-checks surfaced no refutations. **medium** on any specific quantitative anchor transferring to our workload (the model-outputs->human-writeups transfer gap). **low** on anything about positive-claim verification performance - genuinely unmeasured territory, which is also why it needs in-house measurement.\n\n## source index\n\nevery finding above carries its full citation (title, url, absolute date, source type) plus the verifier's transcript in the domain files:\n\n- `atlas/domain_academic-judges.md` - judge meta-evaluation benchmarks and capability limits\n- `atlas/domain_annotation-quality.md` - annotation science, disagreement, contamination\n- `atlas/domain_critique-models.md` - criticgpt lineage, human+critic teams, omission detection\n- `atlas/domain_grounding-faithfulness.md` - entailment checkers, decomposition, citation evaluation\n- `atlas/domain_hybrid-statistical.md` - ppi, selective escalation, conformal routing, drift attribution\n- `atlas/domain_rubrics-recent.md` - rubric practice 2025-26, compilation, rubric qa\n- `atlas/domain_industry-practice.md` - vendor qa stacks, eval tooling, spam economics\n- `atlas/domain_contrarian.md` - the case against, adversarial attacks, validity critiques\n- `atlas/domain_frontier-2026.md` - 2026-only developments\n- `atlas/critic_and_gapfill.md` - critic's weak-spot audit + psychometrics, law/standards, content moderation, feedback science\n", "2": "autoqa foundations - philosophy & structure (draft for product-owner review)\n\n**date:** 2026-07-14\n**status:** draft v1, synthesized from `00_questions.md` and `01_research_atlas.md`. every principle carries its evidence anchor; positions the evidence does not settle are marked **[default]** (my recommended posture, reversible) or **[the product owner]** (needs your call).\n**scope:** project-agnostic foundation for qa of human-annotated ai training data. implementation infra (the vercel app) is out of scope here.\n\n---\n\n## 0. the problem, restated precisely\n\nhuman attempters produce evaluative work products (annotations, ratings, critiques with rationales) under project instructions. human reviewers who qa them exhibit unexplained variance in pass/fail rates and reasons. root causes, ranked: (1) subjective interpretation of evaluation axes, (2) expertise gaps, (3) attention to detail. attempters are mandatory; the qa layer is what we augment.\n\nthe psychometric literature sharpens this: after training, raters' *global* severity differences persist but are a minor variance component - the dominant variance is **rater x item x criterion interaction**: which reviewer, on which submission, on which criterion, this time. that variance cannot be fixed by better rubrics or calibration meetings alone. it can only be attacked by verifying each specific claim on each specific item against evidence - which is exactly what an llm layer can do at scale and humans at scale cannot. that is the reason this system should exist, stated in the field's own terms. *(atlas section 26)*\n\n## 1. the construct: what a verdict means\n\n**quality = satisfaction of the written project instructions + grounding in the defined evidence record.** not \"would the median reviewer pass this.\" human-consensus prediction caps the system's validity at the reviewers' own (unmeasured, evidently low) reliability ceiling; instruction-satisfaction lets the autoqa be *more* valid than the layer it augments, and makes every verdict traceable to a clause and a span. *(atlas section 16-18; peer-reviewed rating-indeterminacy work (neurips 2025) shows forced-choice consensus gold labels can heavily bias judge selection - empirically up to 31% worse - when criteria admit multiple valid readings.)*\n\nconsequences of this choice:\n- gold labels come from **adjudicated expert panels judging against the instructions**, not from single-reviewer or majority crowd labels.\n- when the autoqa and a human reviewer disagree, neither is presumptively right; the arbiter is the instruction text plus the evidence record, adjudicated.\n- where the instructions genuinely underdetermine an item, the correct output is not a verdict but an **instruction-gap report**. reviewer disagreement on such items is signal about the instructions, not error in the people. *(diverging preferences: most annotator disagreement traces to task underspecification and style, not noise or incompetence.)*\n- the **ground-truth hierarchy** is explicit: seeded known-verdict items > adjudicated expert panel > criterion author's written amendment > individual human reviewer > autoqa. only the first three may feed judge calibration. overrides by individual humans are logged as claims, not truth.\n\n## 2. the eleven principles\n\n**p1 - claims, not vibes.** the unit of analysis is the typed claim, not the item. attempter writeups are decomposed into claims of distinct types, each verified in the lane built for it, each verdict carrying quoted evidence. item verdicts are transparent, auditable roll-ups. *(holistic judging of subjective quality measurably fails: near-orthogonal judge/human axes; ~55% on hard per-criterion checks; long-form instability. decomposed per-criterion reasoning buys +7-12pp and lower cross-judge variance.)*\n\n**p2 - the claim ontology is the type system.** every extracted statement is triaged before any verification: **(a) grounding claims** (cited evidence exists and says this), **(b) entailment claims** (evidence supports the conclusion drawn), **(c) calibration claims** (stated severity/confidence matches evidence strength), **(d) completeness/absence claims** (\"no errors,\" \"fully addresses\"), **(e) opinions** - never truth-scored, but checked with real machinery via two reusable passes: *stance-contradiction* (the evidence the attempter cites must not contradict the stated stance) and *stance-calibration* (stance strength must not exceed evidence strength - type-(c) machinery). opinion verdicts are `consistent` / `inconsistent-with-cited-evidence` (a fail-able finding: the opinion's *warrant* fails grounding, not the opinion itself) / `uncheckable` (never penalized) - **(f) undecidable-given-evidence**. each type has different machinery and a different reliability ceiling, reported per type.\n\n**entailment standard [default]:** reasonable-expert support, not strict logical entailment - three-way output per claim (`supported` / `contradicted` / `unsupported`), where `contradicted` is the high-confidence fail trigger and `unsupported` alone triggers an evidence request or escalation rather than an automatic fail. strict entailment fails nearly all competent human prose; loose support reintroduces the variance we're removing - so the false-flag rate under this standard is a measured ship-gate output (section 4.3), not an assumption. *(veriscore/factbench triage is field-standard; facts shows the positive-class asymmetry; absencebench caps type (d).)*\n\n**p3 - evidence closure is explicit and judge priors are not evidence.** every project config defines the evidence record: what the attempter saw, what they cited, what's in scope. verdicts must quote spans from that record to be admissible; \"true in the world but unsupported in the record\" is `unsupported`, not `pass`. checks are routed by whether they are **read-checkable or recompute-checkable** - judge confidence is untrustworthy on checks that exceed its own solving ability, where it silently re-solves and trusts itself over the record. **[default - provisional:** the re-solving evidence is a single under-review 2026 study; experiment 3 replicates it internally before this hard-gates routing.**]** cherry-picking is caught by running both span-closed and context-closed citation checks. the judge gets the same evidence surface the attempter had - architecture beats model. *(auditing-by-re-solving; citeeval; agent-judge/the banking meta-eval.)*\n\n**p4 - positive claims are verified by enumerated class, with honest labels.** \"warranted praise\" verification is restricted to checkable praise types compiled per project (accuracy-praise -> grounding check; completeness-praise -> enumerable checklist; clarity-praise -> opinion class). absence claims get a seeded-defect protocol: unless the judge's planted-flaw recall on this project materially exceeds the attempter population's, its concurrence is labeled **`not-contradicted`, never `verified`**. a vacuousness/eligibility gate precedes grounding checks so hedged empty writeups can't pass by asserting nothing. *(facts eligibility gate; absencebench; zero published praise-verification numbers - this is the first in-house measurement.)*\n\n**p5 - typed verdicts; abstention and contestation are first-class.** the verdict set is `pass` / `fail` / `unable-to-verify` / `instructions-underdetermine`. high vote-entropy across k samples, cross-family judge disagreement, and clustered reviewer disagreement route items to the contested class - which feeds the instruction-gap channel and never penalizes the attempter. this is how the system enforces consistency where consistency is real, without laundering one arbitrary reading of a contested criterion into false objectivity. *(toloka's deployed pass/fail/unable-to-verify; coval; nutmeg; rating indeterminacy.)*\n\n**p6 - the human touch is spent where it compounds, and it is blind.** routing is confidence-calibrated three-lane (auto-verdict / stronger judge / human) with the human budget derived from a target residual error rate, not a fixed per-item quota. the escalated human sees the claim, the criterion clause, and the judge's **collected evidence spans - not its verdict or rationale** (verdict-visible review produces measured over-reliance; evidence-only is the sole format that helps when the ai is right without hurting when it's wrong). escalation routes the contested *claim*, not the whole item. every human adjudication writes back - in two tiers that keep the ground-truth hierarchy honest: a single-touch adjudication produces **candidate** artifacts (logged claims, provisional exemplars), and only panel-adjudicated or owner-signed artifacts enter judge calibration and compiled config; the standing audit channel batch-promotes candidates. in enforcement mode the human budget is max(target-residual-error escalations, consequential-fail volume - section 5); under a pace-bound profile (p11) capacity binds instead, and the residual error rate becomes the reported output rather than the target - never pretend to optimize both. the evidence-only-for-adjudicators result is measured; combining it with full critiques to attempters (p9) is an inferred, untested split **[default]**, a/b'd in experiment 6. overturn-rate monitoring proves the human lane adds signal; an overturn rate at the judge's error floor means rubber-stamping, and the naive human-verifies-ai design is the modal failure in the literature. *(trust-or-escalate/hypac; deepmind complementarity; vaccaro meta-analysis; braintrust/langchain/micro1 write-back practice.)*\n\n**p7 - the qa layer is measured exactly like what it measures.** the judge is enrolled as one more rater facet in the same measurement model as the humans (mfrm-style severity/centrality/fit per rater, human or machine, per project). scope honestly: mfrm needs 3-5 ratings/output of rater-item linkage, so it runs fully on the dual-review pilot and continuously on the judge facet (which rates everything); production human-reviewer severity is estimable only where deliberate overlap is injected - a low-rate stream of double-reviewed seeded items, a budgeted cost, not a free byproduct. all reporting uses chance-corrected statistics, per-class recall at fixed prevalence, and ppi-corrected population estimates with dual-uncertainty cis - raw percent-agreement and single-number \"accuracy vs humans\" are banned artifacts. feedback passes the same grounding gate it enforces before delivery (fail-closed). validation stratifies by measured human-agreement level; verdict-agreement alone never certifies a judge, because right-verdict-wrong-rationale poisons feedback and trust (in one benchmark, 24.8% of items got the right verdict with a low-quality critique - realcritic). **staging [default]:** v1 ships chance-corrected agreement, ppi-corrected point estimates with cis, and k-sample entropy routing; mfrm production enrollment, anytime-valid e-processes, and conformal routing are phase-2 refinements - the full stack is the target, not the launch gate. *(mfrm-for-llm-judges; kappa-deflation cohort; beyondcorrelation; realcritic; metacritique.)*\n\n**p8 - adversarial by construction, from day one.** attempters are paid optimizers; some are agents. defenses are structural, not detective: stakes-sterile judge prompts (consequence framing measurably softens verdicts with zero cot trace); comparative/anchored scoring over absolute scales (transfer attacks inflate absolute scores blind); evidence-perturbation probes as a standing citation-theater detector (swap the cited span -> verdict must flip); blind seeded gold injected into live queues (indistinguishability is the binding constraint); **the divergence sensor** - attempter scores rising on the autoqa while flat on held-out human-graded goldens - as the canonical goodhart alarm; spam defense that is economics/provenance-based (throttling, work-history consistency, worker-level peer-prediction scoring, randomized deep audits), with content detection demoted to a weak auxiliary signal. never score anyone on agreement-with-final-verdict - consensus-rewarded auditing trains your best dissenters to conform exactly where independent judgment matters most. *(evaluation faking; universal adversarial phrases; tspa; westwood pnas; community notes conformity study.)*\n\n**p9 - feedback is an instrumented intervention, not decoration and not free.** the feedback unit is the 4-tuple: quoted claim, evidence span, criterion clause, concrete fix - task-referenced always, person- or rank-referenced never (a third of feedback interventions reduce performance; high-information task feedback roughly doubles the effect of bare verdicts). benchmark against exemplar work, never ranked peers. no stylistic guidance by default - feedback is a homogenization pump aimed at the diversity the training data exists to capture, so a population-level output-diversity drift metric ships with the channel. efficacy is measured per-attempter (repeat-error-rate per criterion, pre/post), expecting modest median effects concentrated in low performers, and separating within-attempter learning from attrition. if high-information feedback doesn't move repeat-error rates in a defined window, conclude decoration and reallocate. *(kluger & denisi; wisniewski; cochrane 2025; dow cscw 2012; monoculture evidence.)*\n\n**p10 - everything drifts; every moving part gets its own sensor.** frozen human-labeled anchor sets interleaved into the live stream attribute drift (judge moved vs population moved) with anytime-valid statistics. judge versions are pinned per project with golden-set regression gates on upgrade; rubric revision is a versioned loop with explicit retroactive re-scoring policy (adaptation lives in the compilation layer, pinning at the judge layer - that's the resolution of the static-vs-co-evolving fork). reviewer calibration decays within a session, so reviewer-effect estimates run in rolling windows with hard-item interleaving and order randomization; graduated projects carry randomized, unpredictable audit rates with automatic de-graduation triggers. queue composition is managed (seeded known-verdict items, easy-case dilution) because prefiltered queues measurably shift reviewer strictness. *(anchor-set e-process work; drift/osce; pinterest dual monitors; content-moderation practice.)*\n\n**p11 - quality is a priced dial on disposition, never on perception.** contractual pace sometimes makes the standard quality threshold unaffordable; the system supports that honestly instead of forcing the loosening to happen off the books. the judge always measures the same way - dial-sterile prompts, same claim verification, same evidence dossier (consequence framing measurably corrupts verdicts with zero cot trace, and a \"be lenient\" judge destroys score comparability, calibration, and the divergence sensor simultaneously). leniency is applied downstream, as thresholds over the continuous scores the pipeline already emits. a project's active **operating profile** (e.g., `strict` / `standard` / `expedited`) moves exactly four things: aggregation cutoffs (how many/how severe findings fail an item), routing thresholds (escalation rate, human budget), verification depth (k-samples, deep-verify sampling fraction, expensive lanes like context-closed citation checks), and enforcement disposition (findings -> fail vs findings -> coaching-only). two constraints can bind: in error-target mode the residual error rate is the target and human budget is the output; in pace-bound mode (contractual throughput) capacity binds and **the residual error rate becomes the reported output** - either way the number is computed and stamped, never hidden, with cis that honestly widen when verification depth is thinned. because measurement persists under every profile, the dial has two independent consumers: the **attempter-disposition dial** (prospective only, logged, announced - the bar people are judged against never moves silently or retroactively) and the **dataset-admission dial** (re-tunable at any time - an expedited batch can be re-filtered to any strictness later from the recorded scores, without re-running qa; the data buyer's strictness is not coupled to the deadline that produced the batch). floors the dial can never touch: integrity-class verdicts (fabrication, spam, falsehood), seeded-gold injection and audit sampling, the divergence sensor (which conditions on the active profile and reads seeded/audit outcomes, not pass rates), and the section 5 legal floor. every delivered batch carries its profile and its corrected residual-error estimate - leniency becomes a contractual artifact instead of quiet degradation. *(evaluation faking -> dial-sterile prompts; criticgpt's fsbs precision/comprehensiveness dial as the precedent for inference-time operating points; trust-or-escalate's coverage dial; hypac error budgets; corrected-estimator reporting.)*\n\n## 3. the architecture\n\neight stages. stages 0-2 are deterministic-ish and cheap; 3 is where models spend; 4-7 close the loops.\n\n```\nproject instructions --> [1] compile (workshop loop, versioned config)\n |\nattempter submission --> [0] intake gate --> [2] decompose & type --> [3] verify (lanes)\n |\n +-------------------------------------------+\n v\n [4] aggregate (typed verdict + evidence dossier)\n |\n [5] route (auto / stronger judge / human, calibrated)\n |\n [6] feedback (4-tuple, self-verified, fail-closed)\n |\n [7] meta-eval & monitor (gold, drift, divergence, dashboards)\n```\n\n**stage 0 - intake & provenance gate.** worker-level, not item-level: rate/velocity limits, work-history consistency, peer-prediction information scores, session-bounded telemetry (allowlist schema - on-task signals only; see section 5). item-level: eligibility/vacuousness check (is this a responsive, contentful submission at all). no llm-text detection verdicts.\n\n**stage 1 - compilation layer (the per-project customization contract).** project instructions compile into a closed config schema:\n- **axis triage:** every evaluation axis classified into `deterministic` (string/regex/programmatic checks), `judge-groundable` (evidence-anchored verdicts), `judge-assisted` (human decides with judge-collected evidence), `human-only`, or `autoqa-ineligible` - using per-axis conformal set width and pilot judge-vs-adjudicated agreement as the sorting evidence.\n- **rubric criteria:** <= ~13 active criteria per judge call, each atomic, rift-linted; **pitfall/negative criteria human-authored** (the one artifact synthesis reliably can't produce); hard constraints get gating semantics, not soft-criterion listing.\n- **contrastive exemplars:** hard near-miss pass/fail pairs per criterion - discriminative exemplars, not descriptive \"what good looks like\" prose. these anchor the judge few-shot (the measured-best grounding architecture) and double as reviewer training material.\n- **evidence-closure rules:** what the judge may read, per claim type; citation requirements on load-bearing attempter claims.\n- **severity/weight taxonomy + aggregation formula** (explicit, auditable - accepting the known performance tax vs implicit aggregation).\n- **routing costs and thresholds:** the asymmetric error prices (false-fail vs false-pass), the target residual error rate, calibration set.\n- **operating profiles (the quality/leniency dial, p11):** named threshold sets (`strict` / `standard` / `expedited`) taken from the ship gate's measured operating-point curve, each stamped with its expected fail-class recall and false-pass rate. profile changes are owner-authorized, logged, prospective-only, and announced to attempters whenever the enforcement bar moves.\ncompilation is a **loop, not a compile step**: owner grades real items, criteria drift is expected, every version is kept, and sign-off happens on graded items, not abstract criteria. compilation fidelity is the binding per-project cost - budget accordingly.\n\n**stage 2 - decompose & type.** claim extraction at adaptive granularity (claim + explicit context annotations, verifier told what's under test; never free-floating atomic rewrites); ontology typing per p2; canonicalization/style-stripping so verbosity and rubric-vocabulary mimicry don't leak into judgment.\n\n**stage 3 - verification lanes.**\n- *deterministic lane:* compiled checks run on everything, always.\n- *screening tier:* cheap specialized checkers (minicheck-class entailment, per-metric small models) score every claim at ~1/100 frontier cost - triage, never verdicts.\n- *verdict tier:* frontier reasoning judge, per-criterion calls, exemplar-anchored, cross-family from the models attempters plausibly used, stakes-sterile prompts, k-sampled with continuous scores (logit-expectation rather than sampled tokens where available), position/order randomized.\n- *opinion-consistency lane:* stance-contradiction + stance-calibration checks against the attempter's cited evidence, with the p2(e) verdict semantics - runs in the same screening/verdict tiers as other claim types.\n- *contested detector:* vote entropy, paraphrase flips, cross-family disagreement -> `instructions-underdetermine` candidates.\n\n**stage 4 - aggregation.** claim verdicts -> item verdict via the compiled formula; error-tolerant (a ~20% claim-level error rate means naive and-aggregation is wrong by construction); output is the **verdict + evidence dossier** (every finding with quoted spans and criterion clauses) - the dossier is simultaneously the audit trail, the appeal record, and the feedback source.\n\n**stage 5 - routing & the human lane.** calibrated dual thresholds (hypac-style) with the human queue = the abstention set; auto-fail held to a stricter bar than auto-pass (false fails burn trust and pay; consequential fails have a legal floor - section 5); blind evidence-only adjudication ui; claim-level micro-adjudication; adjudications write back per p6's two-tier rule; overturn telemetry closed-loop. queue organization (specialist-per-criterion vs generalist-per-item) is a compilation-layer decision informed by the pilot's expertise-share finding - if expertise gaps dominate a project's variance, specialist claim-level routing is the mechanism that closes root cause #2 on the human side. **cold start** for a zero-label project: the judge runs shadow-mode under full human review until the calibration set reaches the ppi-arithmetic size and thresholds are fitted; **graduation** to confidence-routing happens when calibrated thresholds hit the target residual error on a rolling window - mirrored by p10's de-graduation triggers.\n\n**stage 6 - feedback channel.** per p9. generated from the dossier (verdict machinery and feedback share one artifact), passed through its own grounding gate, delivered only if it survives.\n\n**stage 7 - meta-evaluation & monitoring.** per-project ship gate before go-live (section 4); continuous: blind gold injection, frozen anchor set, divergence sensor, per-criterion agreement trends, appeal-overturn bands, population diversity metric, reviewer rolling-window effects, corrected-estimator dashboards. the standing expert-audit channel samples confident passes - **critic-assisted** (a critic-assisted second look finds 4x what an unassisted one does on the \"flawless\" slice) - and is permanent, because when judge accuracy <= attempter accuracy no statistical cleverness can substitute for gold (the 2x theorem), and false agreement is predicted to rise as model errors correlate.\n\n## 4. the ship gate (per project config, before go-live)\n\na project's judge config ships only after:\n1. **variance-attribution pilot:** dual-review a common item set; measure the human floor (chance-corrected, per criterion - per-criterion kappa localizes ambiguous criteria); code disagreements against an underspecification-vs-expertise-vs-attention taxonomy. this decides the build emphasis per project and provides the calibration set.\n2. **mvvp-style judge validation:** chance-corrected agreement vs adjudicated labels, >=3 replicates, ab/ba position swaps, >=2 gold-set designs, bias audit separate from consistency (a judge can be perfectly repeatable and severely biased).\n3. **perturbation harness:** meaning-preserving paraphrase/reorder/noise on claims and evidence; evidence-swap probes; verdicts that flip get routed, not shipped.\n4. **seeded-flaw recall:** planted defects (findtheflaws-style) measuring fail-class recall at realistic prevalence - the number that actually matters under 90%+ pass rates, where kappa itself misleads (feinstein-cicchetti).\n5. **schematic adherence:** do per-criterion scores vary independently and does the rationale's cited schema explain the verdict (halo/factor-collapse check) before per-criterion output is exposed to anyone.\n6. **gold-set arithmetic:** size from the ppi closed form off pilot judge-human correlation per axis (conservative estimates; ~50+ labels per stratum floor; oversample the fail class); low-correlation axes carry human-only until correlation improves.\n7. **beat-the-humans gate:** per axis, judge-vs-adjudicated reliability must exceed the pilot's measured human-reviewer-vs-adjudicated reliability. axes where the judge doesn't beat the humans it augments are human-only lanes - automating them adds cost and subtracts validity. this is the go/no-go the whole premise rests on, and it is comparative, not absolute.\n\nbeyond go/no-go, the ship gate's data product is the **operating-point curve** - thresholds -> fail-class recall and false-pass rate at pilot prevalence. this is what calibrates p11's dial profiles, so even the `expedited` setting ships with a priced, not guessed, error rate.\n\n## 5. the legal floor (binding if the workforce touches the eu)\n\n**gate first [the product owner]:** determine whether the attempter population includes eu-resident workers. if yes, this section is a launch constraint; if the pilot is us-only, treat it as forward-looking scope and revisit before any eu expansion. from 2026-12-02 (platform work directive transposition), for attempters in the eu - employee or self-employed, platform established anywhere:\n- decisions of significant detriment (account/pay restriction, termination, or equivalent) **must be taken by a human**; no consent exception. auto-pass is fine; consequential auto-fail is not. whether a single-item pay denial qualifies is legally untested - **[default]** treat consequential item fails as requiring the human lane until counsel says otherwise.\n- plain-language explanation of any automated-supported decision, a competent human contact, human review with substantiated written reply within two weeks, rectification/compensation duties. the evidence dossier and 4-tuple feedback are this explanation - design them to that bar.\n- telemetry: allowlist only (task-session-bounded behavioral signals); no emotional-state inference, no private-communication monitoring, no off-task collection; dpia required.\n- schufa doctrine: a score with a \"determining role\" is itself an automated decision, and rubber-stamp review counts as zero-touch - the human lane must have authority, evidence access, and calibrated competence. enforcement is active.\n- map quality dimensions and verdict evidence onto iso/iec 5259 vocabulary now (cheap) rather than retrofit later (expensive).\n\nconvenient truth: the legally mandated design (meaningful human on adverse decisions, explanations, appeals) is the same design the complementarity evidence recommends anyway.\n\n## 6. what this system deliberately does not do\n\n- no per-item llm-text spam classification as a primary defense (beaten economically and technically).\n- no holistic single-call item judging; no per-criterion output without schematic-adherence validation.\n- no raw percent-agreement or single-number accuracy anywhere in reporting.\n- no truth verdicts on opinion-class claims; no `verified` label on absence claims without seeded-recall evidence.\n- no stylistic feedback; no visible attempter leaderboards/rank feedback.\n- no consequence language in judge prompts; no trusting cot rationales as bias evidence.\n- no scoring of humans (attempters, reviewers, or auditors) on agreement-with-final-verdict.\n- no judge calibration from unadjudicated override streams.\n- no leniency via judge prompting - the dial moves thresholds, depth, and disposition, never the judge's perception (dial-sterile is the same discipline as stakes-sterile). no dial setting touches integrity floors, seeded gold, audit sampling, or the legal floor. no silent or retroactive changes to the bar attempters are judged against.\n- no claim that the judge validates itself - the expert-audit channel never sunsets.\n\n## 7. first experiments (ordered; each kills or funds a component)\n\n1. **variance-attribution pilot** on one live project (dual review + disagreement coding). decides whether rubric compilation or expertise routing is the bigger lever *for our population*, and produces the first calibration set. everything else calibrates against this.\n2. **positive-claim seeded set:** build warranted/empty-praise pairs per praise type; measure judge precision/recall per type. nothing published exists - this is both the riskiest premise and the differentiator. include absence-claim recall (planted defects behind \"no errors\" writeups).\n3. **transfer check:** take the compiled-config judge and measure per-criterion agreement against adjudicated labels on *human evaluative writeups* (not model outputs) - the number every published anchor is missing. include a re-solving probe: does our judge's wrong-reference detection collapse on recompute-class checks (internal replication of the under-review 2026 result p3 provisionally leans on)?\n4. **confidence-proxy shootout:** verbalized confidence vs k-sample entropy vs conformal width vs cross-family disagreement, scored on predicting human overturn - this picks the routing statistic. include the uniform-sampling baseline.\n5. **divergence sensor dry run:** stand up held-out human-graded goldens + autoqa score tracking from the first live week, so goodhart is measurable before it's a problem.\n6. **feedback tier a/b:** verdict-only vs 4-tuple high-information feedback, outcome = per-criterion repeat-error rate, attrition separated. priors: modest median, concentrated in low performers, backfire in a minority. also a/b the assistance-format split (full critiques to attempters vs evidence-only) - the combination p6/p9 defaults to is untested in any single study.\n7. **agnosticism falsification:** onboard a second project through config alone. any edit to the core judge prompts falsifies the project-agnostic claim - the fix is moving that surface into the config schema, not patching the core.\n\n## 8. open decisions **[the product owner]**\n\n1. **enforcement weight at launch:** advisory/coaching-only vs consequential verdicts. changes the precision floor, the legal posture (section 5), and attempter adversarial pressure. my read: launch advisory, attach consequences only per-lane after seeded-recall numbers exist.\n2. **construct authority when project owner intuition contradicts their own compiled rubric:** rubric-as-law with forced amendment **[default]**, or standing owner overrides (which quietly re-import unexplainable variance).\n3. **llm-assistance provenance policy default:** content-validity-only vs provenance-gated (per-project override either way). interacts with telemetry legality and with the reality that ~1/3 of text workers already use llms.\n4. **judge sourcing:** frontier api judges per project (cross-family routing) only, or also an owned fine-tuned screening tier. the evidence supports two-tier; the question is whether to own the bottom tier.\n5. **where the first pilot runs** - which project's data, whether its owner will fund the adjudicated gold set (the one cost that cannot be automated away), and whether that attempter population includes eu-resident workers (switches section 5 from forward-looking to binding).\n6. **dial governance (p11):** who may change a project's active operating profile (project owner alone, or with client sign-off), and whether expedited batches are disclosed to the data buyer with their stamped residual-error estimate. my read: disclose - a priced error rate converts deadline pressure from hidden liability into a contractual artifact, and it is the posture consistent with the iso 5259 mapping in section 5.\n\n---\n\n*lineage: brainstorm (92 questions, 7 lenses) -> taxonomy (9 themes, 8 tensions) -> 9-domain recency-biased sweep + 27 adversarial verifications + 4 gap-fill literatures -> this synthesis. full provenance in `atlas/`.*\n", "3": "raw questions by lens\n\n## lens: measurement\n\n### what is the actual ground-truth construct we're measuring - 'would an expert consensus panel pass this item?' or 'does this item satisfy the written project instructions?' - and when those two diverge, which one wins?\nwhy: human reviewers already disagree, so 'agreement with human reviewers' is a moving target. if the construct is instruction-satisfaction, the autoqa can be more valid than the humans it's checked against; if it's human-consensus prediction, validity is capped at the humans' own reliability ceiling.\ngates: whether gold labels come from adjudicated expert panels vs. instruction-derived rubric checks; whether disagreement-with-humans is treated as autoqa error or human error during meta-evaluation.\n\n### given that human reviewers show unexplained variance, what is the measured inter-reviewer reliability floor (e.g., krippendorff's alpha on a common item set), and is it even high enough for human agreement to serve as a validation target at all?\nwhy: if human-human alpha is, say, 0.35, then judge-human agreement of 0.4 is indistinguishable from perfect performance and 0.7 is impossible - every downstream accuracy claim is bounded by this number. without measuring it first, we can't tell whether the autoqa is failing or the target is noise.\ngates: whether meta-evaluation uses raw human labels, adjudicated labels, or noise-corrected agreement statistics; the sample sizes and acceptance thresholds for the autoqa's own validation.\n\n### at what granularity of claim decomposition (whole-review pass/fail vs. per-criterion verdict vs. per-atomic-claim groundedness check) does judge reliability stop improving - and does aggregation of atomic verdicts back to an item-level decision introduce its own error mode?\nwhy: the design requires evaluating positive statements ('is this claim grounded in cited evidence?'), which implies decomposition. but decomposition trades one subjective judgment for n smaller ones plus an aggregation rule; if the aggregation rule (any-fail, weighted, threshold) is itself unvalidated, we've moved the subjectivity, not removed it.\ngates: the core unit of analysis for the autoqa verdict schema and the aggregation function from claim-level checks to item-level pass/fail/feedback.\n\n### which failure types are objectively verifiable (fabricated citation, claim contradicted by evidence, instruction violation with a bright line) vs. irreducibly judgment-laden (severity ratings, 'tone', borderline quality), and should the autoqa emit categorically different output types (verdict vs. flag-for-human vs. confidence-weighted opinion) for each?\nwhy: treating all checks as one verdict type forces a single reliability standard on axes with wildly different measurability. the one-human-interaction budget should be spent exclusively on the judgment-laden residue, which requires knowing where the objective/subjective boundary actually sits per project.\ngates: the output taxonomy (auto-decide vs. escalate), and how the per-project customization layer classifies each rubric axis at onboarding.\n\n### are the autoqa's confidence scores actually calibrated per-project and per-axis (does 80% confidence mean 80% agreement with adjudicated truth?), and what recalibration mechanism runs when a new project's instruction set shifts the distribution?\nwhy: the escalation policy (which items consume the one human interaction) is only as good as the calibration behind it. llm judges are notoriously miscalibrated and calibration does not transfer across domains; an uncalibrated escalation threshold silently converts into either rubber-stamping or full human re-review.\ngates: whether to build a per-project calibration/threshold-fitting step into onboarding, what the minimum labeled-item count for it is, and whether raw model confidence is ever exposed to humans.\n\n### what is the minimum viable gold set per project (size, sampling design, refresh rate), given base rates - if only 5-10% of attempts are genuinely bad, how many adjudicated items do we need before we can distinguish a 90%-precision judge from a 70% one?\nwhy: low base rates of spam/falsehood mean precision/recall estimates have huge confidence intervals at plausible gold-set sizes; a 200-item gold set with 15 true failures cannot certify the system. sample-size arithmetic decides whether per-project validation is feasible or whether we must pool evidence across projects.\ngates: gold-set construction protocol, whether validation is per-project or hierarchical/pooled, and the honest error bars we can put on any launch claim.\n\n### how do we detect drift in three distinct moving parts - attempter behavior (including adversarial adaptation to the qa), project instruction revisions, and judge-model updates - and which statistic (verdict-rate shift, escalation-rate shift, gold-set score drop) triggers re-validation for each?\nwhy: a static validation certificate decays: attempters learn what the autoqa checks and game it, instructions get amended mid-project, and the underlying model gets swapped. each drift source needs a different sensor and a different response; conflating them means either constant false alarms or silent decay.\ngates: the monitoring dashboard's core metrics, the re-validation trigger policy, and whether gold items must be continuously injected into the live stream (blind gold) vs. periodically batch-audited.\n\n### for groundedness checks on positive statements, what counts as the evidence closure - only sources the attempter explicitly cited, the full item context, or external knowledge the judge happens to have - and how do we stop the judge from 'verifying' claims against its own priors rather than the record?\nwhy: the stated goal is 'is this claim grounded in the evidence they cite,' which is a verifiability judgment relative to a defined corpus. if the evidence boundary is undefined, the judge will substitute its parametric knowledge, producing verdicts that are confident, unauditable, and wrong in exactly the expertise-gap cases the system exists to catch.\ngates: the judge's context-construction rules (what it may and may not see), and whether groundedness verdicts must quote span-level evidence to be admissible.\n\n### how do we meta-evaluate the constructive-feedback channel - by downstream attempter improvement (revision acceptance rate, repeat-error rate) rather than feedback 'quality' ratings - and are we willing to run the controlled comparison (feedback vs. verdict-only) to find out if feedback does anything?\nwhy: feedback quality has no natural gold label and pairwise 'this feedback seems good' ratings inherit all the subjectivity problems we started with. the only defensible criterion is behavioral: does the attempter's next submission improve? if we can't or won't measure that, the feedback component is unfalsifiable decoration.\ngates: whether feedback generation gets its own outcome-based evaluation loop with attempter-level longitudinal tracking, and what data the platform must log to enable it.\n\n### what is the deployment decision rule that translates measured judge performance into routing - at what precision/recall on which failure class does the autoqa get to auto-fail an item without a human, auto-pass one, or must escalate - and who bears the asymmetric costs of each error?\nwhy: false auto-fails burn attempter trust and pay; false auto-passes poison training data downstream. these costs are asymmetric and project-specific, so a single accuracy threshold is wrong by construction; the routing policy must be an explicit cost-weighted decision rule, not a vibe.\ngates: the three-way routing policy (auto-pass / auto-fail / escalate), its per-project cost parameters, and which decisions are reversible via appeal.\n\n### contrarian: is inter-reviewer disagreement actually the disease, or is some of it legitimate signal about genuinely ambiguous items and underspecified instructions - and would an autoqa that enforces consistency be laundering one arbitrary interpretation into a false appearance of objectivity?\nwhy: high agreement is trivially achievable by collapsing to a crude, consistent-but-invalid rule (reliability without validity). if disagreement concentrates on items where the instructions are genuinely underdetermined, the right output is 'flag instruction gap for the project owner,' not a confident verdict - and the autoqa becomes an instrument for measuring instruction quality, not just attempter quality.\ngates: whether the system includes an 'instructions underdetermine this case' verdict class and a feedback loop to project owners, vs. always forcing a pass/fail.\n\n### when the autoqa and the single human interaction disagree on an item, what is the adjudication and logging protocol - does the human always win, and how do we prevent the human-override stream from silently becoming the new (unvalidated) training/calibration signal?\nwhy: the one-human-per-item budget means most overrides come from a single reviewer - exactly the noisy, poorly-aligned population we started with. if overrides automatically count as ground truth for recalibration, we recycle human inconsistency back into the judge; if they don't, we have no learning loop.\ngates: the override-handling policy, whether disagreement items get periodic second-opinion audits (spending extra human budget on a sampled subset), and what feeds the calibration/fine-tuning data.\n\n### which agreement statistic and evaluation design do we standardize on given that pass rates are extreme and vary by project - and do we explicitly ban raw percent-agreement and single-number 'accuracy vs. humans' from all reporting?\nwhy: at a 92% pass rate, a judge that passes everything scores 92% agreement and kappa behaves pathologically (the base-rate paradox); different per-project prevalences make cross-project comparisons of any single statistic meaningless. choosing the metric contract up front prevents every later performance claim from being an artifact.\ngates: the standard meta-evaluation report format: chance-corrected statistic choice (e.g., krippendorff's alpha + per-class recall at fixed prevalence), stratified-by-failure-class reporting, and minimum ci-width requirements before any threshold change ships.\n\n## lens: judge-failure-modes\n\n### what is the measured run-to-run verdict flip rate for our judge on identical items at fixed temperature, and does the autoqa architecture treat that number as noise to average away (k-sample voting) or as signal that the criterion itself is underspecified?\nwhy: if flip rate on a criterion exceeds the human reviewer disagreement rate we are trying to fix, the judge adds variance rather than removing it. distinguishing stochastic noise from criterion ambiguity determines whether the fix is ensembling or rubric rewriting - two completely different builds.\ngates: whether to spend the inference budget on k-run self-consistency voting versus a rubric-decomposition pipeline; also which criteria get routed to the single allowed human interaction.\n\n### when the judge verifies a positive attempter claim (\"this statement is grounded in the cited evidence\"), does it exhibit an agreement-default bias - passing plausible-sounding but ungrounded claims at a higher rate than it fails grounded ones - and how large is that asymmetry on a seeded test set?\nwhy: verification of positive statements is the core novel requirement here, and sycophancy/plausibility bias runs exactly against it: judges are known to rubber-stamp fluent assertions. if the false-pass rate on seeded ungrounded claims is high, the architecture needs adversarial framing (e.g., 'find the flaw in this claim' prompts, or claim-evidence entailment decomposition) rather than holistic pass/fail judging.\ngates: whether positive-claim verification is built as holistic judgment, as a decomposed entailment check per cited-evidence pair, or as a debate/red-team prompt structure.\n\n### does the judge systematically favor verbose, hedged, or rubric-vocabulary-mimicking attempter writeups over terse-but-correct ones, and by how many points of pass-rate delta on length/style-controlled pairs?\nwhy: verbosity and style bias would train attempters (who adapt fast to qa signals) to pad and mimic rather than improve substance - the autoqa would actively corrupt the annotation pool it polices. a measured delta tells us whether we need style-normalization (summarize-then-judge, content extraction) before judgment.\ngates: whether to insert a canonicalization/extraction layer that strips style before the judge sees the annotation, versus judging raw attempter text.\n\n### when the attempter's annotation critiques an ai output produced by the same model family as our judge, does self-preference bias suppress the judge's agreement with valid criticisms - and does routing those items to a different model family close the gap?\nwhy: the annotated content is ai output; attempters saying 'this ai response is wrong' will be second-guessed by a judge that shares the ai's blind spots or prefers its own family's outputs. this is a structural conflict of interest unique to qa-of-ai-critique settings.\ngates: model-choice policy: whether the judge must be a different family than the generator of the content under annotation, and whether that requires per-project judge routing.\n\n### for each evaluation-axis type (factual grounding, instruction compliance, tone/style quality, holistic ratings), what is the judge-vs-adjudicated-human agreement ceiling - and on which axis types does the judge fall below the human-reviewer inter-rater agreement we already consider broken?\nwhy: root cause #1 is subjective interpretation of criteria. if the judge is no better than humans on subjective axes (likely), the design must partition axes: judge-autonomous, judge-assisted-human, and human-only lanes. pretending one judge handles all axes reproduces the exact variance problem.\ngates: the routing taxonomy: which criterion types the autoqa decides alone, which consume the one human interaction, and which it must abstain on.\n\n### at what rubric decomposition granularity does judge-human agreement stop improving - and is there a granularity beyond which agreement degrades because atomized checks lose context (the judge passes every micro-item while missing the writeup being globally wrong)?\nwhy: the reflexive fix for judge unreliability is finer rubrics, but decomposition has a measurable optimum and a failure mode (checklist myopia). the project-agnostic foundation must specify how project instructions get compiled into judge criteria, and that compiler needs a target granularity.\ngates: the design of the per-project customization layer: how instruction sets are decomposed into judge prompts, and whether a holistic synthesis pass sits on top of atomic checks.\n\n### how sensitive are verdicts to semantically-neutral perturbations of the judge prompt (criterion ordering, pass/fail label ordering, few-shot exemplar choice, evidence presentation order), and do we require paraphrase-invariance testing as a gate before any project's judge config ships?\nwhy: position and prompt-sensitivity effects can dwarf real quality signal. since every project customizes the prompt, an unstable base template means each project's customization silently re-rolls the judge's biases - variance across projects that looks exactly like the human variance we set out to kill.\ngates: whether the foundation includes a mandatory perturbation-robustness test harness and randomized-order/position-debiasing defaults in the base template, versus trusting per-project prompt authors.\n\n### what calibration signal does the judge emit per verdict (stated confidence, sample agreement rate, logit margin), and is any of them actually predictive of judge error well enough to drive the route-to-human decision at a fixed one-touch budget?\nwhy: the one-human-interaction constraint makes triage the load-bearing mechanism. if judge confidence is uncalibrated (common - stated confidence often near-useless), the human touch gets spent on the wrong items and the system's error rate is set by the worst unrouted verdicts.\ngates: the escalation mechanism design: self-reported confidence vs. k-sample disagreement vs. dual-judge disagreement as the routing trigger, and what abstention thresholds to set.\n\n### which shortcut heuristics does the judge use as pass/fail proxies (presence of citations regardless of relevance, rubric keyword echo, annotation length, score extremity), and can attempters who see the feedback reverse-engineer them within a few submission cycles?\nwhy: the autoqa emits constructive feedback to attempters, which means it publishes a gradient toward its own shortcuts. goodharting risk is not hypothetical: attempters are paid on pass rates. the defense (rotating probes, held-out seeded items, feedback that explains without revealing decision boundaries) has to be designed in from the start.\ngates: whether the feedback channel exposes full judge reasoning or a filtered version, and whether the foundation mandates ongoing seeded-item probes to detect gaming.\n\n### is the judge's constructive feedback held to the same grounding standard as the attempter work it critiques - i.e., what fraction of judge-generated feedback statements are themselves hallucinated or ungrounded, and do we verify feedback before delivering it?\nwhy: a judge confidently telling an attempter to fix a non-existent problem destroys trust faster than a wrong pass/fail verdict, and hallucinated critique is a known failure mode of critic models. if feedback needs its own verification pass, that roughly doubles the pipeline and changes cost and latency assumptions.\ngates: whether the architecture includes a feedback-verification stage (judge-of-judge or evidence-linking requirement on every feedback claim) before feedback reaches humans.\n\n### does a multi-model ensemble actually buy error reduction on the failure modes that matter here, or do frontier judges share correlated blind spots on subjective criteria such that the ensemble converges confidently on the same wrong answer?\nwhy: ensembling is the default reliability move, but it only helps against decorrelated errors. if disagreement between model families predicts item difficulty better than it predicts judge error, the right design uses cross-family disagreement as an escalation signal, not a voting mechanism - a very different architecture at very different cost.\ngates: ensemble-vs-single-judge-plus-router architecture, and the per-item inference budget.\n\n### contrarian: on the subjective criteria where human reviewers disagree most, is 'judge bias' even distinguishable from legitimate criterion ambiguity - and would the honest system output 'this criterion is underspecified, here are the two defensible readings' back to the project owner rather than a verdict?\nwhy: our lens assumes judge failure modes are the problem to fix, but root cause #1 is subjective interpretation - if the ground truth is genuinely contested, a perfectly debiased judge still can't be 'right,' only consistently arbitrary. consistent-but-arbitrary may actually be acceptable (it fixes variance) - or it may launder ambiguity into false authority. this choice defines what the product is.\ngates: whether the foundation includes an instruction-ambiguity feedback loop to project owners as a first-class output, and whether the target metric is agreement-with-truth or consistency-of-application.\n\n### what is the judge's blind-spot profile on expertise-heavy content - specifically, at what domain-difficulty level does the judge's ability to verify attempter claims fall below the attempter's own reliability, making autoqa on those items net-negative?\nwhy: root cause #2 is expertise gaps, and the judge has its own. a judge that confidently fails correct expert annotations (or passes confident expert-sounding errors) is worse than no qa on that slice. knowing where the competence boundary sits per domain determines the abstention policy.\ngates: whether the foundation requires per-project judge-competence calibration (seeded expert items) before enabling autonomous verdicts, and the abstain-vs-verdict boundary.\n\n### do pass/fail verdicts and diagnostic reasons drift when the underlying judge model is upgraded, and does the foundation pin model versions per project with a regression suite, or accept drift as the price of improvement?\nwhy: the system's whole pitch is reducing unexplainable variance across reviewers. a silent model upgrade that shifts pass rates 10 points reintroduces exactly that variance, now across time instead of across people, and invalidates attempter feedback histories.\ngates: versioning and regression-testing policy in the foundation: frozen judge versions + golden-set gates on upgrade, versus continuous latest-model usage.\n\n## lens: rubric-ops\n\n### at what decomposition granularity do rubric items stop improving judge-human-expert agreement - and does over-decomposition actively degrade it by destroying holistic judgments (e.g., 'this critique is fair overall') that only exist at the composite level?\nwhy: there is a real possibility that agreement vs. granularity is an inverted-u: too coarse reproduces the subjective-interpretation problem, too fine produces checklists that pass while the annotation is globally wrong. without knowing where the curve peaks per criterion type, we can't set a decomposition policy.\ngates: the rubric compiler's target granularity (atomic checkable claims vs. mid-level judgment clusters) and whether the schema allows/requires holistic roll-up items alongside atomic ones.\n\n### which evaluation-axis types are compilable into evidence-linked binary claims at all, and which are irreducibly gradient/contested - and can we build a reliable classifier (human or model) that sorts a project's axes into 'compile', 'anchor-with-exemplars', and 'route-to-human' buckets before the project goes live?\nwhy: the whole architecture forks on this taxonomy: compilable axes get automated verdicts, exemplar-anchored axes get calibrated model judgment, contested axes consume the one-human-touch budget. if we can't sort axes reliably up front, every project's failure mode is silent misrouting.\ngates: the ingestion pipeline's axis-triage stage and the routing policy for the single allowed human interaction.\n\n### what is the minimal set of artifacts a project instruction set must be compiled into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules - such that a fresh judge (model or human) reaches target agreement with the project owner without any conversation with them?\nwhy: this defines the 'rubric compilation contract.' if exemplars are load-bearing (likely, given that criteria text alone is what humans already disagree on), then a text-only compiler is doomed and ingestion must extract or elicit exemplars, which changes the whole intake workflow.\ngates: the schema of the compiled autoqa config and the required inputs at project onboarding (can we ingest instructions-as-written, or must we demand adjudicated examples?).\n\n### who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - and do we force the rubric to be amended (rubric-as-law) or allow standing owner overrides (rubric-as-guidance), given that override accumulation silently recreates the unexplainable-variance problem we're solving?\nwhy: every rubric system dies one of two deaths: legalism (rubric ossifies, owners route around it) or override rot (rubric becomes fiction). the governance choice determines whether autoqa verdicts are appealable, who can amend, and what an amendment invalidates retroactively.\ngates: the amendment/override protocol, verdict finality semantics, and whether past verdicts are re-scored when a rubric changes.\n\n### how do we detect criteria drift operationally - what specific signal (judge-human disagreement trending on one axis, override cluster density, exemplar staleness score, attempter-appeal rate) triggers a rubric revision, and at what threshold - versus drift in the item distribution or the attempter population masquerading as rubric drift?\nwhy: without a falsifiable drift trigger, rubrics are either revised on vibes (reintroducing subjectivity) or never (ossification). confusing population drift with criteria drift leads to 'fixing' a rubric that was fine, which churns all downstream calibration.\ngates: which telemetry the system must log per verdict from day one, and the design of the drift-detection monitor as a first-class component vs. a later add-on.\n\n### for verifying an attempter's positive claims ('the response's argument in section 2 is sound'), what claim-typing does the compiler need - grounding claims (evidence exists), entailment claims (evidence supports the stated conclusion), and calibration claims (stated severity/rating matches evidence strength) - and do these three require different verification machinery with different reliability ceilings?\nwhy: positive-statement verification is the stated differentiator, and 'is the opinion consistent with cited evidence' (calibration) is plausibly much harder than 'does the cited evidence exist' (grounding). if reliability ceilings differ sharply, verdict confidence must be reported per claim type, and the human touch should target the weakest type.\ngates: the claim ontology in the compiled rubric schema and per-claim-type confidence gating on the human-escalation router.\n\n### what is the empirical false-agreement rate - cases where the autoqa judge and the human attempter agree with each other but both are wrong per expert ground truth - and does rubric decomposition reduce it or merely make shared errors better-documented?\nwhy: the seductive failure mode of rubric-ops is measuring judge-attempter agreement as success. if both parties share a plausible misreading of a criterion, decomposed rubrics can score high agreement while shipping bad data. this determines whether we need an independent expert-audit sampling loop as a permanent fixture rather than a bootstrap phase.\ngates: whether a standing gold-set / expert-audit sampling channel is a required foundational component (consuming budget) or an optional per-project add-on.\n\n### for genuinely contested judgment calls, do we design for verdict pluralism (record 'defensible under reading a, not b' and pass with annotation) or forced resolution (every item gets pass/fail) - and can the system reliably distinguish 'contested by the criterion's nature' from 'contested because the rubric is underspecified'?\nwhy: forced resolution on genuinely contested items manufactures the exact reviewer-variance noise we're eliminating, just with a machine's false consistency. but pluralism without the underspecification distinction becomes an excuse bucket that swallows fixable rubric gaps.\ngates: the verdict schema (binary vs. verdict-plus-reading), the feedback template for contested items, and whether contested-item volume feeds the rubric-revision queue.\n\n### where does the one human interaction per item buy the most rubric-ops value: pre-verdict adjudication of the single most-contested claim, post-verdict confirmation of borderline fails, or asynchronous adjudication of exemplar-generating edge cases that improve the rubric for all future items?\nwhy: the third option means the human budget compounds (each touch improves the compiled rubric) rather than being spent per-item. but it also means some items ship with unconfirmed verdicts. this is a fundamental resource-allocation choice that shapes escalation design.\ngates: the escalation router's objective function: per-item verdict accuracy vs. rubric-improvement rate, and whether human touches write back into the compiled config.\n\n### can the rubric compiler itself be trusted to be project-agnostic - i.e., does one compilation prompt/procedure hit target fidelity across instruction-set styles (terse axis lists, 40-page guideline docs, exemplar-heavy specs), or does compilation fidelity vary so much by input style that we need per-style compilation strategies and a human sign-off gate on every compiled rubric?\nwhy: a miscompiled rubric poisons every downstream verdict on that project, and the errors are systematic, not random - worse than no autoqa. if compilation needs human sign-off anyway, the 'foundational philosophy + per-project customization' story changes from automated ingestion to a structured authoring workflow with tooling.\ngates: whether ingestion is an automated pipeline with spot checks or a human-led compilation workshop with model assistance, and what the compiled-rubric acceptance test is.\n\n### what is the compiled rubric's unit of address - the attempter's deliverable as a whole, each discrete claim within it, or each (claim, criterion) pair - and how do we aggregate claim-level verdicts into the item-level pass/fail plus feedback without an aggregation rule that itself becomes a contested subjective judgment?\nwhy: aggregation is where subjectivity re-enters through the back door: 'three minor grounding lapses vs. one severe misalignment' is exactly the kind of weighing on which human reviewers currently diverge. if aggregation rules can't be specified per project, item-level verdicts inherit the variance we tried to remove.\ngates: whether severity/weight taxonomies are mandatory fields in every compiled rubric, and whether the system emits item-level verdicts at all vs. claim-level findings plus a recommended verdict.\n\n### contrarian: is reviewer disagreement actually caused by criteria vagueness - or would perfectly operationalized rubrics leave most of the variance intact because it stems from expertise gaps and attention (root causes 2 and 3), meaning rubric-ops investment has a hard ceiling we should measure before building the compiler?\nwhy: the ranked root-cause list is a hypothesis, not a measurement. if a cheap experiment (give current human reviewers a hand-operationalized rubric for one project and measure variance reduction) shows small gains, the foundation should weight evidence-verification and expertise-substitution over rubric decomposition - a different system.\ngates: the build order and budget split between the rubric-compilation subsystem and the evidence-verification/feedback subsystems; whether a variance-attribution pilot is a mandatory phase-zero.\n\n### what does constructive feedback require from the compiled rubric beyond verdict machinery - do rubric items need paired 'what good looks like' statements and repair templates authored at compile time, or can feedback be generated at verdict time from the claim-evidence trace alone without drifting into advice the project owner would disown?\nwhy: feedback generated freestyle at verdict time can contradict project doctrine (recommending fixes the project forbids), which destroys attempter trust faster than a wrong verdict. if feedback needs compile-time authoring, the rubric schema roughly doubles and so does ingestion cost.\ngates: whether the compiled-rubric schema includes mandatory feedback-side fields (exemplar repairs, approved phrasings, doctrine constraints) or feedback is a downstream generation layer over verdict traces.\n\n## lens: human-factors\n\n### does the constructive feedback we return to attempters leak the judge's decision boundary - i.e., after n feedback cycles, do attempter scores rise on the autoqa while staying flat or falling on held-out human-graded golden items?\nwhy: constructive feedback is a stated design goal, but it is also the highest-bandwidth channel for goodharting: every explanation of why an item failed is a gradient the attempter can descend without improving true quality. if the divergence (judge score up, golden score flat) appears quickly, feedback must be abstracted (principle-level, delayed, or sampled) rather than item-specific.\ngates: feedback granularity and timing: item-specific rationales vs aggregated/principle-level feedback, and whether feedback text is generated from the same rubric the verdict uses or from a decoupled surface.\n\n### can the judge distinguish genuine evidence-grounding from 'citation theater' - specifically, if we perturb or swap the evidence an attempter cites, does the autoqa verdict flip at a rate that shows it is actually checking the citation-claim link rather than rewarding grounded-sounding prose?\nwhy: the core positive-evaluation requirement ('is this statement actually grounded?') is exactly the surface attempters will learn to imitate: hedged, citation-dense, judge-pleasing prose. a perturbation test is the falsifiable check; if verdicts don't flip when cited evidence is invalidated, the grounding check is decorative and will be gamed within weeks of deployment.\ngates: whether grounding verification is implemented as semantic holistic judgment or as an explicit claim-evidence entailment step with adversarial perturbation testing in the eval harness.\n\n### for wholesale llm-generated submissions, is our defense content-based detection or economics-based throttling - i.e., what is the measured cost ratio between an attempter generating a passing submission with a frontier model and us detecting it, and at what ratio do we abandon detection for identity/throughput/behavioral gates?\nwhy: by mid-2026, frontier-model submissions can be indistinguishable from strong human work on content alone; a detection arms race we lose by construction wastes the qa budget. if the cost asymmetry favors the spammer, the architecture must shift defenses to rate limits, work-history consistency, provenance signals, and randomized deep audits rather than per-item classification.\ngates: whether the autoqa pipeline includes an ai-detection module at all, versus spending that complexity budget on behavioral/longitudinal attempter modeling.\n\n### do we actually care about llm assistance provenance, or only content validity - is a correct, well-grounded annotation produced with heavy llm help a pass, a fail, or a policy question that must be settled per-project before the foundation is designed?\nwhy: this is unresolved in most annotation pipelines and it changes the objective function of the entire qa layer. if content-only, spam filtering collapses into quality filtering (simpler system). if provenance matters (e.g., because training data must reflect human judgment, not model self-distillation), we need signals the content itself cannot provide - process telemetry, timing, edit traces - which is a different data-collection contract with attempters.\ngates: whether the foundation's spam/falsehood axis is defined over content properties alone or requires process/provenance instrumentation as a first-class input.\n\n### in what order does the single human interaction see the ai's work - and at what verdict-presentation format does the human overturn rate fall below the autoqa's measured error rate (the signature of rubber-stamping)?\nwhy: automation bias is the failure mode that silently converts 'human-in-the-loop' into 'human-as-liability-shield.' whether the human judges blind-first then reconciles, sees the ai verdict with rationale, or sees rationale without verdict, measurably changes overturn behavior. if overturn rate < ai error rate, the human is adding negative information and the interaction budget is being wasted.\ngates: the core review ux: blind-first independent judgment vs ai-verdict-first confirmation, and whether ai confidence/rationale is shown before or after the human commits.\n\n### where does the one-human-interaction budget buy the most reliability per item: verifying ai-flagged failures, adjudicating low-confidence/disputed items, or spot-auditing ai passes - and is the answer stable across the autoqa's confidence calibration or does it invert when calibration drifts?\nwhy: with exactly one human touch, routing is the system. spending it on confirmed-looking failures duplicates the ai; spending it on confident passes catches judge blind spots but at low hit rate. the optimal allocation depends on the judge's calibration curve, which will drift as attempters adapt - so the routing policy must be designed as a function of measured calibration, not fixed at launch.\ngates: the triage/routing architecture: which confidence bands and verdict types route to the human, and whether routing thresholds are static or closed-loop on observed overturn rates.\n\n### when the judge's domain competence is below the attempter's (frontier expertise, niche domains), can the autoqa detect its own incompetence and abstain - and at what abstention rate does the system stop net-saving human effort for that project?\nwhy: expertise mismatch cuts both ways: the autoqa confidently 'correcting' an expert attempter is worse than no qa, because it burns attempter trust and pollutes the record with false fails. a judge that cannot self-diagnose incompetence forces either per-domain capability gating or per-project human fallback, which changes the project-agnostic claim of the foundation.\ngates: whether the foundation includes a mandatory abstain/escalate verdict class with per-project competence calibration, and whether some projects are declared autoqa-ineligible.\n\n### does the autoqa prefilter shift the human reviewer's base rates enough to cause criterion drift - when the queue reaching the human is enriched with hard/adversarial cases, do reviewers become systematically harsher or more vigilance-fatigued than on an unfiltered queue, and does seeding known-verdict items into the queue correct it?\nwhy: filtering changes the population the human sees, which changes the human. a reviewer seeing 80% genuinely-borderline items recalibrates their pass threshold and fatigues faster; their disagreement with the ai then looks like ai error when it's induced drift. without seeded gold items you cannot even measure this, because there is no fixed reference in the post-filter queue.\ngates: whether the queue design mandates seeded known-verdict items and easy-case dilution, and how reviewer accuracy is measured post-filter.\n\n### if autoqa verdicts affect attempter pay or tenure, what appeal mechanism absorbs disputes without either re-importing full human review load or becoming a rubber stamp - concretely, does an appeal route to a human, a stronger/differently-prompted ai pass, or evidence-required re-adjudication, and what appeal-success rate signals a healthy vs broken system?\nwhy: the moment verdicts carry economic consequence, the system becomes adversarial and appeals are the trust release-valve. an appeal path that costs a full human review breaks the one-interaction budget at scale; one that never succeeds destroys attempter trust and drives silent gaming instead of open dispute. the target appeal-success band (too low = judge unaccountable, too high = judge unreliable) must be chosen before launch because it defines the verdict's enforcement weight.\ngates: whether autoqa verdicts are advisory or enforcement-grade, and the architecture of the appeal tier (human, escalated ai, or evidence-gated).\n\n### when the single human reviewer and the autoqa persistently disagree, what arbitrates - given that noisy, misaligned human reviewers are the original problem, does escalation to another human just re-import the variance we built the system to remove?\nwhy: the system's stated origin is human reviewer misalignment, yet most hitl designs quietly treat the human as ground truth for disagreements. if human arbitration is itself ~60% reliable, disagreement resolution needs a different substrate: golden exemplars, criterion-author rulings, or multi-judge consensus with documented rationale. choosing wrong means the autoqa slowly trains toward the noise it was meant to filter.\ngates: the ground-truth hierarchy of the foundation: what outranks what (golden set > criterion author > human reviewer > autoqa?), and how disagreements feed back into judge calibration.\n\n### is the observed reviewer variance actually a human-factors problem at all - if we run two independently-configured autoqa instances on the same items, do they disagree at a rate comparable to the human reviewers, indicating the variance lives in criterion ambiguity rather than in humans?\nwhy: this is the contrarian check on the whole lens: if the criteria themselves are underdetermined, no amount of incentive design, bias mitigation, or fatigue management fixes pass/fail variance - and the autoqa will simply be misaligned with itself in a way that looks authoritative. the falsifiable test (judge-judge disagreement vs human-human disagreement per criterion) tells us whether budget goes to per-project rubric disambiguation/decision-rule authoring or to the qa layer itself.\ngates: whether the foundation's first-class artifact is the judge, or a per-project criterion-operationalization step (decision rules, boundary exemplars) that both judge and humans consume.\n\n### is the one-human-interaction-per-item goal itself the right target - would zero per-item human touch plus randomized deep audits and drift monitoring outperform one shallow, automation-biased touch per item on total error caught per human-hour?\nwhy: one guaranteed shallow interaction per item invites rubber-stamping and gives a false sense of human oversight, while consuming the entire human budget on breadth. concentrating the same hours into deep audits of a random sample yields unbiased error estimates, resists automation bias (auditors work blind), and creates real deterrence for gaming - but sacrifices per-item human accountability. this is a fork in the foundation, not a tuning knob.\ngates: the fundamental human-budget allocation model: per-item review vs sampled deep audit vs confidence-routed hybrid, which determines the entire pipeline shape.\n\n### what keeps the human reviewers calibrated and engaged once the autoqa writes the verdicts and the constructive feedback - measurably, does reviewer skill and overturn quality decay over months of reconcile-only work, and if so, who remains competent to calibrate the judge?\nwhy: the judge needs an ongoing supply of high-quality human judgments to detect its own drift, but the system as designed converts reviewers from judgment-producers into verdict-confirmers - the exact role that atrophies judgment. if reviewer skill decays, the calibration signal degrades precisely as attempter gaming pressure grows, a compounding failure. the design may need deliberate blind-judgment quotas purely to preserve the calibration workforce.\ngates: whether the foundation mandates a standing blind-review stream (reviewers judging without ai output) as calibration infrastructure, and how reviewer performance itself is qa'd.\n\n## lens: grounding\n\n### at what claim-decomposition granularity does judge-human agreement on grounding verdicts stop improving - holistic review-level, per-rubric-axis, per-sentence, or atomic claim - and what is the cost curve per level?\nwhy: decomposition is the single biggest architectural choice: atomic claims make entailment checks tractable and auditable but multiply judge calls and can strip context that changes a claim's meaning; holistic judging is cheap but unfalsifiable. without knowing where agreement plateaus, we cannot size the pipeline or the per-item cost.\ngates: whether the pipeline's first stage is a claim extractor feeding n entailment checks, or a single structured holistic judge; also the shape of the verdict data model.\n\n### is the evidence universe closed (ai output under review + task instructions + attempter-cited spans) or open (world knowledge, external references) - and what verdict do we assign to attempter claims that are true-in-the-world but unsupported in the closed corpus?\nwhy: a closed universe makes verification a well-defined entailment problem; an open universe requires retrieval and inherits the judge's own hallucination risk. many attempter claims (e.g. 'this api call is deprecated') are world-knowledge claims with no in-corpus evidence, and how we score them determines the false-flag rate.\ngates: whether the foundation includes a retrieval/grounding-corpus component per project, and the definition of the 'unsupported' verdict class.\n\n### what entailment standard does an attempter statement have to meet - strict logical entailment from cited evidence, or 'reasonable-expert support' - and is the output a 3-way verdict (entailed / contradicted / unsupported) or a graded support score with thresholds?\nwhy: strict entailment will fail nearly all human-written prose (humans compress, paraphrase, and generalize); loose support standards reintroduce exactly the subjective variance we're trying to remove. the standard chosen defines what 'grounded' means for the whole system and must be written into every per-project customization.\ngates: the verdict taxonomy, the calibration target for the judge, and the wording of the grounding criterion given to projects.\n\n### can a judge distinguish warranted praise from empty praise at usable reliability - concretely, on a seeded set where half the praised properties are genuinely present and half are absent or fabricated, what precision/recall does the judge hit per praise type ('accurate', 'complete', 'clear', 'idiomatic')?\nwhy: positive-claim verification is the stated differentiator of this system, but praise types differ radically in checkability: 'the code compiles' is verifiable, 'the explanation is clear' is barely falsifiable. if the judge only verifies the checkable subset, we need to know which subset and say so, rather than pretend all praise is verified.\ngates: whether positive-claim verification ships as a hard pass/fail gate, a flag-only signal, or is restricted to an enumerated set of checkable praise types.\n\n### does verifying a positive claim like 'no errors present' or 'fully addresses the prompt' require the judge to independently re-attempt the task (exhaustive defect search), and if so, is the judge's own miss rate on seeded defects low enough that its 'confirmed clean' verdict adds information over the attempter's?\nwhy: universal/negative-existential claims can't be verified by citation checking - they require proving absence. if the judge's defect-finding recall is not materially above the attempter population's, 'autoqa confirmed the pass' is theater, and pass-verdicts should be treated as unverifiable rather than verified.\ngates: whether the architecture includes an independent re-attempt/defect-search stage, and how pass/clean attempter verdicts are labeled in output (verified vs. not-contradicted).\n\n### is omission detection tractable project-agnostically - i.e., can a judge generate the reference set of 'what should have been said' from instructions + artifact alone at usable recall, or does credible omission detection require a per-project gold defect taxonomy, making it customization work rather than foundation work?\nwhy: detecting what's missing requires a completeness reference that doesn't exist in the attempter's text. if that reference can only come from project-specific defect enumeration, then omission detection belongs in the per-project layer and the foundation should only define its interface - a very different build than a universal omission detector.\ngates: whether omission detection is a core foundation module or a per-project plugin with a defined contract; also whether project onboarding must include building a defect taxonomy.\n\n### where is the line between a verifiable claim and a legitimate subjective judgment call, and what does the system do with claims classified as opinion - check only internal consistency with the evidence the attempter cites, or decline to score them at all?\nwhy: root cause #1 is subjective criterion interpretation; if the grounding layer tries to fact-check genuine judgment calls it will manufacture false disagreements, and if it exempts too much as 'opinion' it verifies nothing. the claim-type classifier's boundary is the system's scope, and misrouting here is the main source of attempter-facing injustice.\ngates: whether the pipeline needs a claim-type classification stage, and the verdict semantics for opinion-class claims (consistency-checked vs. unscored).\n\n### when the attempter's cited evidence supports their conclusion but uncited evidence elsewhere in the artifact undermines it, do we verify against cited-evidence-only or the full artifact - and are cherry-picking failures scored as grounding failures or omissions?\nwhy: cited-only verification is cheap and precise but blesses cherry-picking, which is exactly the failure mode of a lazy-but-plausible attempter. full-artifact verification catches it but is an order of magnitude more work per claim and blurs into omission detection. this choice determines what the entailment checker actually reads.\ngates: the evidence scope of the entailment checker and the taxonomy boundary between 'ungrounded' and 'incomplete' verdicts.\n\n### should the foundation mandate an upstream instruction change requiring attempters to attach explicit evidence pointers (quotes, line refs) to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding and other gaming responses?\nwhy: requiring citations could be the highest-leverage move in the whole design: it turns the hardest problem (locating evidence) into the attempter's job and makes verification nearly mechanical. but it changes the attempter product, adds friction, and invites decorative citations that pass surface checks - which would corrupt the very signal we depend on.\ngates: whether the autoqa spec includes required changes to attempter-facing instructions, or must work with free-form uncited reviews as given.\n\n### what is the empirical base-rate split between attempter claims that are actually false versus merely uncited-but-true, and at what judge precision does flagging 'unsupported' claims for the one allowed human interaction beat spending that interaction elsewhere?\nwhy: if most unsupported claims turn out true, aggressive flagging burns the human-interaction budget on exoneration and erodes attempter trust; if falsehoods are common, permissive thresholds let spam through. the threshold cannot be set without the base rate, and the base rate differs from the error-only-qa world because we now audit positive statements too.\ngates: verdict thresholds, whether 'unsupported' auto-fails or only flags, and the triage function that allocates the single human interaction per item.\n\n### is judge-judge agreement on grounding verdicts (same item, resampled/paraphrased prompts, sibling models) actually higher than the human reviewer-reviewer agreement we're trying to fix - measured per verdict class - and what consistency mechanism (self-consistency voting, fixed rubric anchors, deterministic decomposition) closes the gap if not?\nwhy: the system's entire justification is reducing unexplained reviewer variance. an llm judge has its own stochastic variance, and if its verdict flip rate on borderline grounding calls matches human disagreement rates, we've automated the problem rather than solved it. this must be a falsifiable pre-ship gate, not an assumption.\ngates: whether verdicts require ensembling/multi-sample agreement (with its cost multiplier), and the go/no-go criterion for deploying the judge at all.\n\n### do we apply the grounding standard reflexively to autoqa's own feedback - every criticism it sends an attempter must itself cite entailing evidence and pass the same entailment check - and what false-accusation rate is low enough that attempters keep trusting and acting on the feedback?\nwhy: constructive feedback is a stated goal, and an ungrounded accusation from the qa system is more corrosive than an ungrounded claim from an attempter - it trains attempters to dispute or ignore the system. reflexive self-verification roughly doubles judge cost, so it must be a deliberate decision with a trust-threshold target, not an afterthought.\ngates: whether the feedback generator gets its own verification pass before emission, and the precision slo for attempter-facing criticisms.\n\n### contrarian: if the judge can reliably verify whether an attempter's statements are grounded - including re-finding defects to check 'clean' verdicts - does grounding verification actually require less capability than doing the annotation itself, and if not, what evidence supports keeping mandatory human attempters upstream of a judge that could replace them, rather than inverting the design (model attempts, human grounds)?\nwhy: the lens presumes verification is easier than generation, which is true for closed-form checks but unproven for omission detection and absence-proving, where the judge must out-attempt the attempter. if verification is capability-equivalent to attempting, the honest design conclusion is different: either scope autoqa down to the genuinely-easier-to-verify subset, or acknowledge the human's value is legal/accountability cover and design the qa layer around that instead.\ngates: the fundamental scope of the grounding module - full verification vs. only the verification-easier-than-generation subset - and how the system's value proposition is framed to stakeholders.\n\n## lens: hybrid-workflow\n\n### what exactly is the single human interaction spent on: blind re-judgment of the item, verification of the ai's verdict+rationale, or adjudication of specific ai-flagged claims - and which of these maximizes error reduction per minute of human time?\nwhy: these three interaction types produce different label independence, different cost per item, and different anchoring risk. the whole workflow, ui, and downstream statistics change depending on which one we pick.\ngates: the core unit of work in the human queue and the data schema of what a human touch produces.\n\n### which measurable confidence proxy from the llm judge (self-consistency across k samples, verdict flip-rate under paraphrase, ensemble disagreement, rationale-evidence entailment score) actually predicts human overturn with auroc high enough to route on - and what is that threshold?\nwhy: confidence-based routing is worthless if the confidence signal is uncorrelated with actual judge error. this is empirically falsifiable per project and determines whether routing is principled or theater.\ngates: whether we build routing on judge-internal signals at all, and which signal the router consumes.\n\n### should the escalation threshold be set to a fixed human-capacity budget (top-n most uncertain items) or to a target residual error rate (escalate until expected post-qa error < x%), and which quantity is actually the binding constraint?\nwhy: budget-first routing gives predictable staffing but uncontrolled quality; error-first routing gives quality guarantees but unpredictable queue depth. you cannot optimize both; picking one dictates the router's objective function.\ngates: the router's optimization target and how project owners configure it.\n\n### what is the asymmetric evidence bar for standalone autoqa verdicts: what measured false-fail rate (on adjudicated gold) is acceptable before the system may fail an attempter with zero human touch, and is auto-pass held to a different bar than auto-fail?\nwhy: failing a human attempter has pay/reputation consequences; a wrong pass just leaks noise into training data. these harms are asymmetric, so a single trust threshold is almost certainly wrong.\ngates: whether the autonomous lane emits both verdict polarities or only one, and the gold-set size needed to certify it.\n\n### contrarian: given the stated root problem is that human reviewers disagree with each other for unexplainable reasons, does escalating to one human reduce error at all - or does it re-inject the exact variance we are removing while laundering ai verdicts with a human signature? what overturn-quality audit would prove the human touch adds signal?\nwhy: the lens presumes human escalation improves outcomes. if a second human agrees with the escalation human no more often than with the ai, the hybrid layer is cost without benefit and the design should shift toward better autonomous calibration instead.\ngates: whether per-item human escalation exists in v1 at all, versus humans only in calibration/gold roles.\n\n### should the scarce human interactions be spent on per-item adjudication at all, versus spent upstream on building per-criterion calibration sets and disambiguating rubric language - and at what point does one hour of rubric disambiguation beat one hour of item review in agreement gains?\nwhy: root cause #1 is subjective interpretation of criteria. if interpretation variance dominates, per-item escalation treats symptoms while the same human minutes spent codifying criterion boundary cases would raise autonomous accuracy across all items.\ngates: the allocation policy for the human-interaction budget: adjudication queue vs. calibration/rubric-refinement queue.\n\n### does the escalated human judge blind (no ai rationale shown) or anchored (ai verdict and cited evidence shown) - and is the measured anchoring cost smaller than the independence value of blind labels for calibrating the judge?\nwhy: anchored review is faster and catches ai reasoning errors directly, but anchored labels cannot be used to measure judge accuracy or drift because they are contaminated. this is a one-way door for the statistics.\ngates: whether escalation labels double as calibration data, or a separate blind-label stream must be funded.\n\n### at what granularity does escalation route: the whole item, or the specific contested claim/criterion the judge is uncertain about - and can claim-level adjudication (answer 2-3 pointed micro-questions) stay within the one-interaction budget while whole-item review cannot?\nwhy: the system evaluates positive statements claim-by-claim (groundedness, criterion-alignment, evidence-consistency). judge uncertainty is usually localized; sending the whole item wastes the human on parts the ai already resolved confidently.\ngates: queue item schema and the escalation ui contract (full item vs. targeted micro-adjudications).\n\n### should escalation queues be organized by criterion (specialist reviewers per evaluation axis) or by item (generalist review), and does criterion-specialist routing measurably close the expertise-gap root cause where generalist routing does not?\nwhy: root cause #2 is expertise gaps. a reviewer strong on factual grounding may be weak on style criteria. specialist queues change reviewer staffing, assignment logic, and how verdicts are reassembled from partial adjudications.\ngates: queue partitioning, reviewer assignment model, and verdict-assembly logic.\n\n### what statistical model combines sparse human labels with dense ai judgments - per-reviewer reliability weighting (dawid-skene family), per-project bayesian calibration of judge scores, or simple hierarchical override - and how many human labels per project-criterion cell does the chosen model need before its posteriors beat the raw judge?\nwhy: below the convergence threshold, the fusion model is worse than trusting the judge alone; above it, it corrects systematic judge bias per criterion. the answer sets the mandatory human-label spend during ramp-up and whether reviewer identity must be tracked.\ngates: the verdict-fusion architecture and the minimum viable human-label collection plan per project.\n\n### what is the cold-start protocol for a new project with zero human labels - a 100%-escalation burn-in window, seeded gold items authored during project setup, or transferred priors from similar projects - and what quantitative graduation criterion (e.g., judge-human kappa over the last n adjudications) flips the project into confidence-routed mode?\nwhy: since the system is project-agnostic and customized per instruction set, every project starts uncalibrated. without an explicit graduation gate, projects either stay expensively human-heavy forever or go autonomous on faith.\ngates: project onboarding lifecycle, burn-in cost model, and the automated mode-transition rule.\n\n### does constructive feedback delivered to attempters need a different confidence bar than the pass/fail verdict - i.e., can the system ship ai-generated coaching on items whose verdict was auto-passed, and what is the harm model when specific feedback claims (e.g., 'your citation does not support this') are themselves wrong?\nwhy: feedback contains checkable factual claims about the attempter's work; a wrong specific accusation damages trust more than a bare wrong verdict. if feedback needs human review, the one-interaction budget gets split between adjudication and feedback vetting.\ngates: whether feedback generation is in the autonomous lane, the escalated lane, or gated by its own groundedness check.\n\n### once a project runs in trusted-standalone mode, what ongoing randomized audit rate and drift statistic (e.g., cusum on judge-vs-audit disagreement per criterion) triggers automatic de-graduation back to heavier escalation - and does audit sampling need to be unpredictable enough that attempters cannot learn to write for the judge?\nwhy: an autonomous lane fails silently: judge drift, instruction changes, and attempter adaptation to the judge's known preferences all degrade quality without any surfaced signal. confidence-routing itself is a gameable surface if deterministic.\ngates: the monitoring subsystem, audit sampling design (including randomization), and the automatic trust-revocation rule.\n\n## lens: product-feedback\n\n### at what verdict granularity (binary pass/fail, per-axis scores, or claim-level grounded/ungrounded judgments on each attempter statement) does judge-human agreement stop improving, and does claim-level verification measurably reduce the unexplained reviewer variance we're trying to fix?\nwhy: the whole premise is that humans disagree at the holistic-verdict level. if decomposing to claim-level checks doesn't recover agreement, the autoqa inherits the same subjectivity problem and we've just automated the noise. if it does, the entire data model must be claim-first with verdicts as derived aggregates.\ngates: the core output schema: whether verdicts are primitive objects or computed rollups of atomic claim judgments, and whether the feedback surface is organized around claims or around holistic scores.\n\n### for positive-statement verification ('is this attempter claim grounded in the cited evidence?'), what is the falsifiable acceptance bar - e.g., does the judge's grounded/ungrounded label match an expert adjudicator at >=x% on a golden set - and what x makes the system trustworthy enough to auto-pass items without human review?\nwhy: verifying positive claims is harder than spotting errors; without a measured precision bar we can't decide which items get the one allowed human interaction versus none. this number determines the human-in-the-loop routing policy and the escalation threshold.\ngates: the triage policy: which confidence band auto-passes, which auto-fails, and which consumes the single human interaction.\n\n### should the single permitted human interaction be spent on adjudicating the judge's uncertain verdicts, or on the attempter responding to feedback (contest/revise), and does that choice differ by root cause (subjectivity vs expertise gap vs attention lapse)?\nwhy: one human touch per item is a hard budget. spending it on verdict adjudication improves label quality; spending it on attempter response improves future behavior and catches judge errors the attempter can see. these are different products with different data flows.\ngates: where the human sits in the pipeline (before verdict finalization vs after feedback delivery) and what ui/workflow that interaction needs per project.\n\n### what is the minimal structure of a feedback unit that attempters actually act on - e.g., (quoted attempter claim, specific evidence span, criterion clause violated, concrete fix) - and can we falsify 'more detail is better' by measuring revision quality against feedback length and structure?\nwhy: constructive feedback is a stated goal, but unstructured llm feedback tends toward verbose, hedged prose that attempters don't read closely. if a rigid 4-tuple outperforms free text on revision quality, feedback generation becomes a constrained extraction task, which is cheaper, more verifiable, and project-portable.\ngates: the feedback schema and generation approach: templated evidence-linked tuples vs free-form critique, and how feedback quality itself gets qa'd.\n\n### must every piece of feedback be traceable to a specific clause in the project instruction set (criterion-linked), and what happens to judge observations that are real problems but map to no written criterion - surface them, suppress them, or queue them as instruction-gap reports to the project owner?\nwhy: root cause #1 is subjective interpretation of criteria. if the judge can cite un-written standards, it amplifies the subjectivity problem and attempters can't contest it. but suppressing real issues loses signal. the gap-report path turns judge disagreement into instruction improvement, attacking the root cause rather than the symptom.\ngates: whether the feedback generator is hard-constrained to cite instruction clauses, and whether an instruction-gap feedback channel to project owners is a first-class output of the system.\n\n### how do we distinguish 'attempter is wrong' from 'criterion is ambiguous' in the verdict itself - should the autoqa emit a third outcome class (ambiguous-under-instructions) with measured inter-judge disagreement as its trigger, rather than forcing a pass/fail?\nwhy: forcing binary verdicts on genuinely ambiguous items is exactly what makes human reviewers look misaligned. if ensemble judge self-disagreement reliably predicts human reviewer disagreement, ambiguity becomes detectable and routable instead of masquerading as attempter failure.\ngates: the verdict taxonomy (2-class vs 3-class), the ensemble/self-consistency architecture needed to detect ambiguity, and whether ambiguous items are exempt from attempter penalty.\n\n### what is the explainability contract for a failing verdict: does every fail require verbatim evidence spans and a reproducible reasoning chain an attempter could independently check, and do we reject (not deliver) judge verdicts that can't meet that bar even when the verdict is likely correct?\nwhy: a correct-but-unexplainable fail destroys attempter trust and is uncontestable, recreating the opaque-human-reviewer problem. making explainability a delivery gate trades some recall for legitimacy; whether that trade is acceptable changes the judge's output requirements fundamentally.\ngates: whether explainability is a hard output constraint (fail-closed) or best-effort annotation, and the verification step that checks feedback self-consistency before delivery.\n\n### does delivered feedback measurably change future attempter behavior - what longitudinal metric (repeat-error rate on the same criterion per attempter) do we commit to, and after how many items of no improvement do we conclude feedback is decoration and the real product is filtering?\nwhy: this is the uncomfortable question for the feedback lens: the constructive-feedback goal presumes attempters read, understand, and apply it. marketplace annotation workforces often optimize throughput, not learning. if repeat-error rates don't move, feedback generation is wasted spend and the design should shift budget to routing, filtering, and attempter selection instead.\ngates: whether feedback generation is a core deliverable with a learning-loop kpi, or a lightweight byproduct - this reallocates most of the product surface.\n\n### should attempters be able to contest autoqa verdicts, and does a contest consume the one human interaction - i.e., is the appeals channel the human-in-the-loop, and what contest-overturn rate would indicate the judge (not the attempters) is the problem?\nwhy: a contest channel is both a trust mechanism and the cheapest source of judge-error labels. but it competes for the single human-touch budget and can be gamed by throughput-motivated attempters. the overturn-rate threshold gives a falsifiable tripwire for judge recalibration.\ngates: whether appeals exist, who adjudicates them within the one-interaction budget, and the monitoring metric that triggers per-project judge retuning.\n\n### what exactly is the per-project customization surface - is it only (instruction set + criterion decomposition + calibration exemplars + severity weights), and can we prove the foundation is truly project-agnostic by onboarding a new project with zero prompt-engineering of the core judge?\nwhy: project-agnostic is the stated constraint, but every per-project prompt hack that leaks into the core erodes it. defining the customization surface as a closed configuration schema makes 'agnostic' falsifiable: a new project either onboards through config alone or the abstraction is wrong.\ngates: the boundary between the frozen foundation and the per-project config schema, and the onboarding procedure (config authoring vs bespoke judge tuning).\n\n### how many calibrated exemplars per criterion (expert-adjudicated pass/fail/ambiguous anchors) does a new project need before judge-human agreement plateaus, and who produces them - because if the answer is 'a large expert-labeled set,' the system quietly reintroduces the expensive aligned-human-reviewer dependency it was built to replace?\nwhy: calibration exemplars are the standard fix for criterion subjectivity, but their cost scales with project count and criterion churn. knowing the plateau point (5 vs 50 vs 500 per criterion) determines whether per-project onboarding is a day of an sme's time or an ongoing labeling program.\ngates: the per-project onboarding cost model and whether calibration is a one-time setup artifact or a continuous pipeline with its own qa.\n\n### should the autoqa verdict feed attempter-level consequences (ratings, pay, removal) or remain item-level coaching only - and if consequences attach, what precision floor and audit trail does each verdict need to be defensible against a wrongly-failed attempter?\nwhy: the moment verdicts affect livelihoods, error tolerance collapses and attempters rationally start gaming the judge (writing to please the model rather than to be correct). this changes required precision, explainability depth, contest rights, and how feedback language is phrased.\ngates: verdict consumption policy downstream (coaching-only vs consequential), which sets the precision/audit requirements for the entire judge pipeline.\n\n### how do we prevent feedback-induced monoculture - if attempters learn the judge's stylistic preferences, do annotations converge on judge-pleasing patterns that reduce the diversity and independent judgment the training data exists to capture, and what drift metric would detect this?\nwhy: the feedback loop is the product's point, but its failure mode is goodharting the annotation workforce: evidence-citation boilerplate, hedge phrasing, and criterion-parroting that passes qa while degrading the underlying data. a measurable diversity/drift signal decides whether feedback should ever include stylistic guidance at all.\ngates: what categories of feedback are permitted (substance-only vs substance+style), and whether a population-level drift monitor is a required system component from v1.\n", "4": "autoqa research atlas -- combined domain reports\n\nconcatenation of the 10 per-domain source reports underlying 01_research_atlas.md, in source index order. each section below is one original file with its heading demoted one level; content is otherwise unmodified except for ascii normalization.\n\n## contents\n\n- [domain: academic-judges](#domain-academic-judges) -- judge meta-evaluation benchmarks and capability limits\n- [domain: annotation-quality](#domain-annotation-quality) -- annotation science, disagreement, contamination\n- [domain: critique-models](#domain-critique-models) -- criticgpt lineage, human+critic teams, omission detection\n- [domain: grounding-faithfulness](#domain-grounding-faithfulness) -- entailment checkers, decomposition, citation evaluation\n- [domain: hybrid-statistical](#domain-hybrid-statistical) -- ppi, selective escalation, conformal routing, drift attribution\n- [domain: rubrics-recent](#domain-rubrics-recent) -- rubric practice 2025-26, compilation, rubric qa\n- [domain: industry-practice](#domain-industry-practice) -- vendor qa stacks, eval tooling, spam economics\n- [domain: contrarian](#domain-contrarian) -- the case against, adversarial attacks, validity critiques\n- [domain: frontier-2026](#domain-frontier-2026) -- 2026-only developments\n- [completeness critic](#completeness-critic) -- critic's weak-spot audit + psychometrics, law/standards, content moderation, feedback science\n\n\n---\n\n## domain: academic-judges\n\n*judge meta-evaluation benchmarks and capability limits*\n\n### on hard, objectively-verifiable judging tasks (judgebench, iclr 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while gpt-4o scored 50.9-56.6% (near random), the best reward model (skywork-reward-gemma-2-27b) hit 64.3%, and fine-tuned judges like pandalm fell at or below random. [load-bearing]\nevidence: judgebench uses 350 rigorously verified response pairs across knowledge/reasoning/math/coding where correctness is checkable. verified numbers: gpt-4o vanilla 50.9% overall, arena-hard pipeline 56.6%, o1-preview 75.4% (85.7% math/coding), skywork-reward-27b 64.3%, reward models 60-64%, chateval multi-agent ~34%. fine-tuned preference judges plateau near random on knowledge/reasoning.\nsource: judgebench: a benchmark for evaluating llm-based judges (iclr 2025) + emergentmind topic synthesis (updated jan 2026) | https://arxiv.org/abs/2410.12784 | 2024-10 (iclr 2025; synthesis updated 2026-01) | academic\nimplication: the autoqa core judge should be a frontier reasoning model, not an off-the-shelf fine-tuned judge or scalar reward model, because annotation-qa requires reasoning about correctness, not preference matching. reasoning-model headroom (75% vs 51% for the same lab's non-reasoning model) is the single biggest capability lever.\n\n### standard llm-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial llms, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy. [load-bearing]\nevidence: guerdan, barocas, holstein, wallach, wu, chouldechova (neurips 2025 poster, abstract opened and confirmed). they formalize 'rating indeterminacy' - items where multiple ratings are reasonable - and show humans and llms resolve forced choices differently, so aggregated gold labels systematically mis-rank judges. they provide concrete recommendations for elicitation and aggregation.\nsource: validating llm-as-a-judge systems under rating indeterminacy (neurips 2025) | https://neurips.cc/virtual/2025/poster/117308 | 2025-09-19 | academic\nimplication: directly answers design q1/q2: the ground-truth construct cannot be a single forced pass/fail human label. the gold set must elicit response-set labels ('which verdicts are defensible?'), and the verdict ontology needs a first-class 'instructions underdetermine this case' class. validation against collapsed consensus labels will select the wrong judge configuration.\n\n### the largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including april-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected cohen's kappa is 33-41 percentage points lower on mt-bench - and judge rankings shift by up to 14 positions depending on which benchmark you use. [load-bearing]\nevidence: norman, rivera, hughes (arxiv 2606.19544, abstract opened and confirmed). four cohort-wide findings: universal kappa deflation; benchmark-dependent rankings (up to 14 position shifts across mt-bench/judgebench/rewardbench); a 'consistency-bias paradox' where test-retest reliability >0.95 coexists with position bias >0.10 in two production judges; verbosity bias small (<0.011) under a single controlled pairwise rubric. they distill a minimum viable validation protocol.\nsource: reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic\nimplication: answers design q1's metric question: ban raw percent-agreement and single-number accuracy from all autoqa reporting; standardize on chance-corrected statistics per benchmark. also answers q5: a judge that is perfectly self-consistent can still be systematically biased - consistency monitoring and bias audits are separate mandatory sensors, and per-project validation must use multiple gold-set designs since rankings do not transfer.\n\n### at the rubric level - the exact granularity autoqa would operate at - even frontier judges achieve only ~55-56% balanced accuracy on hard rubric-verification instances (gpt-4o 55.97%, claude-sonnet-4.5 55.65%), but rubric-level evaluation with explicit chain-of-thought reasoning beats checklist-level evaluation by 7-12 percentage points and reduces cross-judge variance. [load-bearing]\nevidence: rubriceval (arxiv 2603.25133, full alphaxiv overview read). binary satisfied/not-satisfied verification of atomic rubrics against responses; easy subset ~90% accuracy, hard subset ~55%. compositional (conditional-logic) instructions, role persona, and format structure rubrics hardest. their multi-model arbitration labeling pipeline (raf) reached 85.0% agreement with humans, cohen's kappa 0.702. reasoning-intensive models (o3) notably outperform standard instruct models.\nsource: rubriceval: a rubric-level meta-evaluation benchmark for llm judges in instruction following | https://www.alphaxiv.org/overview/2603.25133 | 2026-03-26 | academic\nimplication: answers design q3 granularity: decompose to atomic criteria and require per-criterion explicit reasoning (worth +7-12pp), but budget for the cost multiplier and expect a hard residual class (~45% error on contested rubrics) that must route to humans. easy/hard stratification should be built in: judge consensus at the coarse pass identifies which claims need escalation - the raf arbitration cascade is a directly reusable architecture for gold-label production.\n\n### on subjective rubrics, llm judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-to-human), judges use only 0.3-0.5x the human score spread, and inter-llm agreement (r~0.35) exceeds llm-human agreement (r~0.27-0.32) - while on a rubric with a verifiable factual answer the same judges fall back into the human range (58.5 degrees, r=0.519).\nevidence: mukherjee, hamna, bali, sitaram (arxiv 2606.03043, abstract opened). 41 llm judges, 4 datasets, 8 indic languages, geometric analysis with bootstrap cis. fine-tuning/preference optimization recovers score spread (0.32 to 1.08) but barely moves the axis; only post-hoc calibration on a small human-anchored set improved all rubrics. scope caveat: indic-language community-health data, but the verifiable-vs-subjective contrast is the mechanism of interest.\nsource: the geometry of llm-as-judge: why inter-llm consensus is not human alignment | https://arxiv.org/abs/2606.03043 | 2026-06-02 | academic\nimplication: multi-judge ensembles and inter-judge agreement are not validity evidence on subjective axes - consensus can reflect a shared collapsed subspace. the objective-subjective boundary (design q2) is empirically real and large: verifiable criteria are judge-autonomous territory; subjective criteria need human-anchored calibration sets, not more judges.\n\n### single-trial llm judging is measurably noisy: across 29 tasks with 50 repeated trials, pairwise preferences flipped on average 13.6% of the time (28% of questions exceeded 20% flip rate), semantically equivalent prompt templates changed majority outcomes in 25% of tested cases, and ~11 repeated trials were needed for majority vote to recover the reference verdict with 95% probability.\nevidence: yagubyan (arxiv 2606.13685, abstract opened). gpt-4o-mini and gpt-4.1-mini judges; also found significant first-position bias (72% a-majority, p=0.024), cross-judge agreement only 76% (kappa 0.51), and a pairwise-pointwise gap where judges pick winners even when their own scalar scores show no meaningful difference. caveat: both judges from one provider, mini-tier models; the reliability-without-validity cohort shows some production judges reach test-retest >0.95.\nsource: the coin flip judge? reliability and bias in llm-as-a-judge evaluation | https://arxiv.org/abs/2606.13685 | 2026-04-23 | academic\nimplication: answers design q5's flip-rate fork with 'both': k-sample voting (k around 5-11) plus position randomization is baseline hygiene, and per-item flip-rate is a cheap, free signal of criterion underspecification - items with high vote entropy are exactly the 'instructions underdetermine this' class. a perturbation-robustness harness is justified as a ship gate.\n\n### the broadest llm-vs-human-judge study on existing human-annotated datasets (judge-bench, 20 datasets) found the best model (gpt-4o) reached only kappa = 0.28 +/- 0.32 on categorical judgments and spearman rho = 0.50 +/- 0.21 on graded ones, with enormous task-to-task variance - and llms agreed more with non-expert annotators than with experts.\nevidence: bavaresco et al., 'llms instead of human judges?' (acl 2025). structured/rule-like attributes correlate reliably; subjective attributes (engagingness) poorly; safety judgments produced negative correlations due to guardrail refusals. authors conclude task-specific human validation is essential before deploying llm judges. read via detailed paper-note secondary source; consistent with the acl publication.\nsource: llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks (judge-bench) | https://en.papernotes.org/acl2025/llm_nlp/llm_vs_human_judges_study/ | 2025-07 (acl 2025) | academic\nimplication: chance-corrected agreement with humans on subjective axes is far lower than the practitioner '85% agreement' narrative. the agree-more-with-non-experts finding is a red flag for the expertise-gap root cause: an uncalibrated judge may replicate crowd-level judgment, not expert judgment - expert-adjudicated (not crowd-consensus) gold labels are required for validation.\n\n### preference leakage is a measured, large contamination bias: a judge favors outputs of models trained on data from a related generator by up to 28.7 percentage points (preference leakage score), with a relatedness gradient - same model 23.6% average, fine-tuned descendants 19-22%, same family 2.8-8.9% - and judges cannot detect their own students (near-chance) while a bert classifier can (82.4%).\nevidence: li et al., iclr 2026 (arxiv 2502.01534). sft transmits the bias strongly (23.6% avg pls) vs dpo (5.2%); effect scales roughly linearly with synthetic-data mixing ratio with no threshold; style/format removal cuts pls sharply; only contextual calibration mitigated it (17.8 to 7.3); prompting-based mitigation failed.\nsource: preference leakage: a contamination problem in llm-as-a-judge (iclr 2026) | https://arxiv.org/abs/2502.01534 | 2026-02 (iclr 2026; arxiv 2025-02) | academic\nimplication: answers design q5 self-preference: when attempters critique (or produce with the help of) outputs from model family x, the judge should not be family x or its distillation lineage - and since lineage is often undisclosed, cross-family judge routing plus style-stripping/canonicalization is the practical defense. prompt-level debiasing will not fix this.\n\n### dedicated trained judges advanced substantially in 2025: meta's j1 (rl-trained thinking-judge, may 2025) at 32b outperforms o1-mini, o3, and 671b deepseek-r1 on some judge benchmarks; compassjudger-2-7b (july 2025) matches far larger generalists on judge benchmarks; skywork-reward-v2-llama-3.1-8b (july 2025) topped seven reward benchmarks including judgebench.\nevidence: j1 arxiv 2505.10320 (verified): unified verifiable-reward rl over 22k synthetic pairs, mitigates position bias by construction, gains from test-time self-consistency. compassjudger-2 arxiv 2507.09104: verifiable-reward + margin-loss training; 32b scores 80.9% accuracy on judgerbenchv2. skywork-reward-v2 arxiv 2507.01352: 26m human-ai-curated preference pairs; 8b model led rewardbench v1/v2, ppe, rm-bench, judgebench among reward models.\nsource: j1: incentivizing thinking in llm-as-a-judge via reinforcement learning (with compassjudger-2, skywork-reward-v2) | https://arxiv.org/abs/2505.10320 | 2025-05 to 2025-07 | academic\nimplication: small trained judges are now viable for high-volume, well-specified sub-checks (cost tier), but their wins are on preference-style benchmarks; for reasoning-heavy annotation qa the judgebench evidence still favors frontier reasoning models. a two-tier design (cheap trained judge for triage, frontier reasoner for verdicts) is supported; the j1 training recipe (verifiable rewards, position-bias-symmetric training) is reusable if a custom judge is ever trained.\n\n### rewardbench 2 (ai2, june 2025) made reward-model evaluation ~20 points harder via best-of-4 format (random = 25%) and unseen human prompts; top models score below 40% on precise instruction following, yet benchmark scores correlate 0.87 (pearson) with downstream best-of-n performance across 113 reward models.\nevidence: arxiv 2506.01937 (verified via abstract, ai2 dataset page, and lambert's analysis post). 1,865 cases across factuality, focus, math, precise if, safety, ties. lambert (author) notes generative llm-as-judge approaches remain 'weaker than expected relative to standard reward models' at output ranking, and that frontier models still fail trivial rankings ('name a color in the rainbow').\nsource: rewardbench 2: advancing reward model evaluation | https://arxiv.org/abs/2506.01937 | 2025-06-02 | academic\nimplication: precise-instruction-following is the weakest judged capability (<40% at best-of-4) - exactly the 'is this annotation compliant with written project instructions?' skill autoqa needs. do not assume instruction-compliance checking is a solved sub-problem; it needs its own gold set, and deterministic/programmatic checks should absorb whatever can be compiled to rules.\n\n### in expert-knowledge domains, llm judges agreed with subject-matter experts only 68% (dietetics, registered dietitians) and 64% (mental health, clinical psychologists) on overall pairwise preference, with agreement varying further across domain-specific aspect questions.\nevidence: szymanski et al. (arxiv 2410.20266; acm iui 2025, ~220 citations). mixed-methods pairwise comparison study; authors conclude llms alone lack the depth for complex knowledge-specific tasks and experts must stay in the evaluation loop. full abstract text opened and confirmed.\nsource: limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks | https://arxiv.org/abs/2410.20266 | 2024-10 (iui 2025) | academic\nimplication: for projects requiring genuine domain expertise (root cause 2), expect judge-expert agreement in the 60s on holistic preference - below any reasonable autonomy bar. expertise-heavy axes belong in the judge-assisted or human-only lane, and the axis-triage classifier (design q2) should treat 'requires specialized professional knowledge' as an explicit routing feature.\n\n### llm judges are measurably less reliable on long-form outputs: longjudgebench (june 2026) finds current judges unstable across real-world long-form scenarios requiring document-level assessment of organization, coverage, and cross-section consistency, and rubrics or references help but are 'not always sufficient'.\nevidence: chen et al., arxiv 2606.01629 (abstract opened and confirmed). first meta-evaluation benchmark targeting long-form judging specifically; systematically evaluates judges across base models and judging protocols; code public. complements the short-form-centric older benchmarks (mt-bench, llmbar, judgebench).\nsource: benchmarking llm-as-a-judge for long-form output evaluation (longjudgebench) | https://arxiv.org/abs/2606.01629 | 2026-06-01 | academic\nimplication: attempter writeups/critiques are long-form documents; holistic single-pass judging of them inherits this instability. supports the decomposition-first architecture (per-claim verification then aggregation) over holistic item scoring, and flags 'global coherence of the writeup' as a check that per-claim decomposition alone will miss (checklist myopia risk from design q3 is real in both directions).\n\n### tools & artifacts\n- judgebench: meta-evaluation benchmark (350 verified response pairs, objective correctness ground truth) with live huggingface leaderboard | https://github.com/scalerlab/judgebench | canonical hard test for whether a candidate autoqa judge can detect actual errors rather than preferences; leaderboard at huggingface.co/spaces/scalerlab/judgebench\n- rewardbench 2: ai2's best-of-4 reward model benchmark (1,865 cases, 6 subsets incl. factuality and precise instruction following) | https://huggingface.co/datasets/allenai/reward-bench-2 | the precise-if and factuality subsets are the closest public proxies for instruction-compliance and grounding checks in autoqa\n- rubriceval + rubric arbitration framework (raf): rubric-level meta-evaluation benchmark plus a multi-model arbitration pipeline for producing gold labels (85% human agreement, kappa 0.702) | https://www.alphaxiv.org/overview/2603.25133 | raf's disagreement-cascade (base judges -> rationale re-evaluation -> reasoning meta-judges) is a directly reusable architecture for autoqa gold-set production and escalation routing\n- j1 (meta): rl training recipe for thinking-judges using verifiable rewards over synthetic pairs; position-bias-mitigating by construction | https://arxiv.org/abs/2505.10320 | the reference recipe if a custom project-tuned judge is ever trained; also evidence that judge chain-of-thought quality is the optimizable variable\n- compassjudger-2 (7b/32b, open weights): opencompass generalist judge models trained with verifiable rewards + margin loss; includes judgerbenchv2 (10k questions, 10 scenarios) | https://github.com/open-compass/compassjudger | best open-weight generative judge candidate for a cheap triage tier\n- skywork-reward-v2 (0.6b-8b, open weights): scalar bradley-terry reward models trained on 26m human-ai-curated preference pairs | https://github.com/skyworkai/skywork-reward-v2 | state-of-art scalar scorers; useful for high-volume ranking/filtering but not for criteria-grounded verdicts with rationales\n- atla selene mini (8b, open weights): general-purpose fine-tuned evaluator model (jan 2025), promptable with custom criteria | https://arxiv.org/abs/2501.17195 | open-weight promptable evaluator baseline; claims high agreement with domain experts on financebench/craft-md but predates the 2025-26 judge generation\n- longjudgebench: first meta-evaluation benchmark for judging long-form outputs (june 2026), code public | https://github.com/cjj826/longjudgebench | test harness for the long-form writeup judging autoqa must do; measures rubric/reference sufficiency\n- llm-as-a-judge.github.io: maintained living survey and paper list for the llm-as-judge field (tracks 2025-2026 papers incl. preference leakage, toolprmbench) | https://llm-as-a-judge.github.io | ongoing monitoring source for judge research; companion to gu et al. 'a survey on llm-as-a-judge' (the innovation, 2026, ~1,667 citations)\n- minimum viable validation protocol (from arxiv 2606.19544): a distilled judge-validation protocol: chance-corrected agreement + consistency protocol + bias audit, from the 541k-judgment study | https://arxiv.org/abs/2606.19544 | template for the autoqa per-project ship gate: what to measure before any project's judge config goes live\n\n### disagreements\n- dedicated trained judges vs frontier generalists: judgebench (iclr 2025) shows fine-tuned judges and reward models plateau near random-to-64% on hard reasoning-based judging while frontier reasoning models lead (o1-preview 75.4%); but j1 (meta, 2025-05), compassjudger-2 (2025-07), and skywork-reward-v2 (2025-07) each claim small trained models beat o1-mini/o3-class models on judge/reward benchmarks; and lambert (rewardbench 2 author, 2025-06) argues scalar reward models still beat generative llm judges at output ranking. the conflict is largely benchmark-dependent (preference-style vs verifiable-reasoning tasks) - no source resolves which wins for criteria-based qa of human annotations, which resembles neither exactly.\n- judge-human agreement levels: practitioner sources claim ~85% agreement 'higher than human-human agreement' (confident ai, 2026) and '>80%, matching human-to-human levels' (galileo); academic chance-corrected measurements show best-model kappa of 0.28 +/- 0.32 across 20 tasks (judge-bench, acl 2025), universal 33-41pp kappa deflation vs raw agreement (arxiv 2606.19544, 2026-06), and near-orthogonal judge-vs-human evaluation axes on subjective rubrics (arxiv 2606.03043, 2026-06). much of the gap is percent-agreement vs chance-corrected statistics plus task subjectivity - the practitioner numbers are not wrong so much as unfalsifiably framed.\n- verbosity bias magnitude: the 2606.19544 large-scale audit found verbosity bias small (<0.011) across 21 judges under a single controlled pairwise rubric, whereas the broader literature (zheng 2023 mt-bench, gu et al. survey 2026, arxiv 2510.12462) treats verbosity bias as a major systematic failure. possible reconciliation: bias magnitude depends heavily on rubric structure and elicitation format, meaning it is a controllable design variable rather than a constant.\n- run-to-run consistency: the coin flip judge (2026-04) reports 13.6% average pairwise flip rate and recommends ~11-vote majorities (mini-tier openai judges only), while reliability without validity (2026-06) measured test-retest reliability >0.95 for some production judges - consistency is highly judge-specific, and the latter paper explicitly warns high consistency coexists with severe position bias (consistency is not validity).\n\n### gaps\n- no benchmark meta-evaluates judges on the autoqa task itself - verifying human-written evaluative claims (grounding of praise, evidence-conclusion entailment in critiques, severity calibration). closest proxies are rubric-verification (rubriceval), factuality judging (rewardbench 2 factuality), and critique evaluation inside j1/compassjudger training; the positive-claim/'warranted vs empty praise' verification question (design q3) has no published precision/recall numbers anywhere i could find.\n- could not obtain a verified july-2026 leaderboard snapshot for the newest frontier models (gpt-5.x, claude opus 4.x, gemini 3) on judgebench or rewardbench 2 - both official leaderboards render via dynamic js (huggingface spaces) and did not yield data to fetches; the largest systematic evaluation covering 'the april 2026 frontier' (arxiv 2606.19544) reports cohort-level findings but does not name per-model winners in accessible text. best-available anchor numbers are therefore early-2026 or older.\n- no single study reports judge-vs-adjudicated-expert agreement broken out per evaluation-axis type (factual grounding vs instruction compliance vs tone vs holistic) - the per-axis agreement ceiling matrix needed for design q5's lane assignment must be assembled from fragments (rewardbench 2 subsets, judge-bench task variance, rubriceval categories) or measured in-house.\n- human inter-reviewer reliability floors (krippendorff's alpha) for annotation-qa reviewers specifically were out of this domain's scope and not covered by the judge meta-evaluation literature; the rating-indeterminacy framework (neurips 2025) is the right machinery but its 11 tasks are generic rating tasks, not training-data qa.\n- atla (selene) company status as of mid-2026 unverified - selene mini (8b, open weights, jan 2025) claims remain reproducible from the model card, but i found no 2026 successor release, so treat selene as a frozen artifact rather than a maintained product line.\n\n### verifications\n- claim: on hard, objectively-verifiable judging tasks (judgebench, iclr 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while gpt-4o scored 50.9-56.6% (near random), the best reward model (skywork-reward-gemma-2-27b) hit 64.3%, and fine-tuned judges like pandalm fell at or below random.\n verdict: confirmed | every quantitative claim checks against the paper's own tables (arxiv:2410.12784 v2, iclr 2025 camera-ready), read directly from the pdf: gpt-4o vanilla 50.86% / arena-hard 56.57% overall (table 1), o1-preview 75.43% overall with 85.71% on both math and coding (table 2), skywork-reward-gemma-2-27b 64.29% as best reward model (table 3, full range 59.43-64.29, paper says \"approximately 59% to 64%\" so the claim's \"60-64%\" is a minor round-up at the low end), pandalm 13.14% (far below random; paper: all fine-tuned judges except skywork significantly below random), chateval 34.00%. 350 verified pairs across knowledge (154)/reasoning (98)/math (56)/coding (42) confirmed - note this is the gpt-4o-generated split; a separate 270-pair claude-3.5-sonnet split exists. dates confirmed: arxiv v1 2024-10-16, iclr 2025, emergentmind synthesis updated 2026-01-09. two contextual nuances, neither refuting: (1) in the v2 tables o1-preview is not the single best judge - o3-mini (high) scores 80.86% and deepseek-r1 73.14%, which strengthens rather than weakens the \"reasoning models dominate\" thesis; (2) 2025-2026 follow-on work (j1, rm-r1, meta-judging, rrd) has pushed judgebench accuracies to 77-81%+, so \"gpt-4o near random\" describes the late-2024 snapshot of non-reasoning judges, not the current frontier. the claim as stated about the paper's findings is accurate.\n- claim: standard llm-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial llms, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy.\n verdict: partially_confirmed | core claim confirmed against the actual neurips poster page (https://neurips.cc/virtual/2025/poster/117308): authors, rating-indeterminacy framing, forced-choice vs multi-label response-set elicitation, 11 tasks, 9 commercial llms, and the \"as much as 31% worse\" figure all match verbatim. two corrections: (1) \"provably biased\" overstates the paper's own language - it says differing resolution of indeterminacy \"can heavily bias\" validation, and the 31% gap is empirical, not a theorem (the theory links performance measures/elicitation schemes); (2) the 2025-09-19 date is not shown on the poster page - plausibly the acceptance date, but unverified; the paper dates to arxiv march 2025 and neurips dec 2025, with an ml@cmu blog post 2025-12-09. search for later work (as of july 2026) found follow-on papers (irt-based judge reliability, arxiv 2602.00521; grading-scale alignment, arxiv 2601.03444) that build on, not refute or supersede, this result. code: https://github.com/lguerdan/indeterminacy.\n corrected: guerdan, barocas, holstein, wallach, wu & chouldechova (neurips 2025; arxiv:2503.05965, first posted march 2025) show that standard llm-judge validation via forced-choice human gold labels can be heavily biased when rating criteria admit multiple valid interpretations (\"rating indeterminacy\"): across 11 real-world rating tasks and 9 commercial llms, forced-choice-based validation selected judge systems performing as much as 31% worse than those selected by their framework using multi-label \"response set\" ratings. the bias mechanism is supported by a theoretical framework, but the 31% suboptimality figure is an empirical finding, not a proof.\n- claim: the largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including april-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected cohen's kappa is 33-41 percentage points lower on mt-bench - and judge rankings shift by up to 14 positions depending on which benchmark you use.\n verdict: confirmed | fetched https://arxiv.org/abs/2606.19544 directly. every quantitative element checks out against the abstract: 21 judges from 9 providers, 118 runs / ~541,000 judgments across mt-bench/judgebench/rewardbench; kappa deflation of 33-41 percentage points on mt-bench across all 21 models; ranking shifts up to 14 positions across benchmarks; consistency-bias paradox (test-retest reliability >0.95 with position bias >0.10 in two production judges); verbosity bias <0.011; minimum viable validation protocol; april 2026 frontier models included. submission date june 17, 2026 matches; authors norman, rivera, hughes match. \"largest to date\" is the authors' own framing, but two independent searches (july 14, 2026) found no larger or superseding meta-evaluation; follow-on work (aura arxiv 2606.19714, apple correlated-panels paper, \"below the reliability floor\" on openreview) complements rather than supersedes it. caveats: preprint with 0 citations; authors themselves limit dataset authority to the march-april 2026 measurement window due to hosted-endpoint drift.\n\n---\n\n## domain: annotation-quality\n\n*annotation science, disagreement, contamination*\n\n### roughly one-third of crowdworkers used llms on an llm-advantaged text-production task (33-35% on prolific, july 2023; ~34% prolific self-report in a 2025 follow-up by zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). [load-bearing]\nevidence: veselovsky et al. ran two preregistered prolific studies (n=168 and n=720, medical-abstract summarization) with a calibrated classifier plus self-report and keystroke heuristics; prevalence 33.3-35.4% baseline (95% cis ~[26%,43%]); a 3x3 factorial of requests x hurdles halved but did not eliminate use. the june 2026 community survey (arxiv 2606.04924) corroborates: 30-40% estimated reliance and 34% prolific self-report, and notes mitigations 'reduce rather than eliminate' llm-assisted responses.\nsource: prevalence and prevention of large language model use in crowd work (cacm; arxiv 2310.15683) | https://arxiv.org/abs/2310.15683 | 2025-02-18 | academic\nimplication: autoqa must assume a contamination base rate of ~15-35% on any text-heavy attempter task even after friction measures; per-item content detection and 'please don't' policies cannot be the primary defense, and mitigation friction has a measurable quality cost that the qa layer must not misattribute to attempter skill.\n\n### an autonomous llm agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused ai-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. [load-bearing]\nevidence: westwood (pnas, nov 20 2025; 43,800 evaluations across 9 llms) shows instruction-following checks, logic puzzles, recaptcha, and rare-event probes all fail; persona calibration even masks superhuman-accuracy tells. recommended defenses are entirely provenance/economics-based: identity revalidation, per-worker throttling, completion-history transparency, pass/fail history, location/vpn checks - 'there is no magical fix.' the 2026 crowdsourcing survey (arxiv 2606.04924) independently confirms detectors degrade under paraphrase and out-of-domain, and mixed human-ai authorship is harder still.\nsource: the potential existential threat of large language models to online survey research (pnas 122(47)) | https://www.pnas.org/doi/10.1073/pnas.2518075122 | 2025-11-20 | academic\nimplication: directly answers taxonomy q8: wholesale-llm-submission defense must be economics/provenance-based (rate limits, work-history consistency, randomized deep audits, process telemetry), not per-item text classification; any autoqa spam layer built on stylistic detection will be silently beaten within the cost asymmetry.\n\n### correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. [load-bearing]\nevidence: elangovan et al. (amazon), iclr 2025 (arxiv 2410.03775, opened and verified): krippendorff's alpha / kappa were designed for human-human reliability and their assumptions break for machine labels; the paper's fix is (1) stratify meta-evaluation by human-label uncertainty, (2) use binned jensen-shannon divergence for perception-type (inherently subjective) tasks, (3) report 'perception charts' instead of one correlation. code open-sourced (amazon-science/beyondcorrelation).\nsource: beyond correlation: the impact of human uncertainty in measuring the effectiveness of automatic evaluation and llm-as-a-judge (iclr 2025) | https://arxiv.org/abs/2410.03775 | 2025-05-01 | academic\nimplication: answers taxonomy q1 directly: meta-evaluation must stratify gold/validation items by measured human agreement level and report per-stratum judge performance; 'accuracy vs. humans' as a single number should be banned from reporting, and judge validity claims are only meaningful on the high-human-agreement stratum.\n\n### in human preference/rating data, the majority of annotator disagreements are attributable to task underspecification and response-style preferences - not random noise and not primarily expertise gaps - and standard aggregation (bradley-terry reward modeling) plus llm-as-judge evaluation both fail to account for this divergence. [load-bearing]\nevidence: zhang et al., icml 2025 (arxiv 2410.14632, opened and verified): built a 10-category, 4-class taxonomy of disagreement sources on preference datasets; abstract states 'the majority of disagreements are due to factors such as task underspecification or response style,' explicitly challenging the noise assumption; they show llm evaluations are heavily swayed by divisive style features and develop divergence-identification methods. converges with jiang & de marneffe (tacl 2022) who found the same structure in nli.\nsource: diverging preferences: when do annotators disagree and do models know? (icml 2025) | https://arxiv.org/abs/2410.14632 | 2025-07-15 | academic\nimplication: supports root-cause-1 primacy and taxonomy q2: the verdict ontology needs a first-class 'instructions underdetermine this case' class detected via measured divergence, and a cheap variance-attribution pilot is empirically justified because published decompositions find underspecification (fixable by rubric compilation) dominates over noise (fixable by averaging).\n\n### even on a nominally objective task (nli), annotator disagreement decomposes into a 10-category taxonomy across three high-level sources - uncertainty in sentence meaning, underspecification in guidelines, and annotator behavior - proving that a fraction of inter-reviewer variance is item-intrinsic and survives perfect rubric operationalization.\nevidence: jiang & de marneffe, tacl 2022, developed and hand-applied the taxonomy to high-disagreement nli items; disagreement persisted among trained annotators on items with genuine interpretive ambiguity. d3code (emnlp 2024; 4.5k sentences, 4k+ annotators, 21 countries) adds that residual variance is systematically structured by annotators' moral values and region, not random.\nsource: investigating reasons for disagreement in natural language inference (tacl 10) | https://aclanthology.org/2022.tacl-1.78/ | 2022-12-01 | academic\nimplication: the target metric cannot be raw agreement: some disagreement is item-intrinsic signal. the variance-attribution pilot should code disagreements against a taxonomy like this to split rubric-fixable variance (underspecification) from irreducible variance (ambiguity/values) before setting build order.\n\n### chance-corrected agreement coefficients (cohen's kappa, and analogously krippendorff's alpha) can be near zero despite very high raw agreement when class prevalence is extreme (the feinstein-cicchetti paradoxes), so under the 90%+ pass rates typical of qa pipelines, single-coefficient reliability reporting is misleading in the opposite direction from raw percent agreement.\nevidence: feinstein & cicchetti (j clin epidemiol, 1990) formalized the two kappa paradoxes (high agreement/low kappa under imbalance; asymmetric marginals inflating kappa); the effect is prevalence-driven, extensively replicated (quarfoot & levine, am. statistician 2016; gwet's ac1 literature). this is exactly the regime of qa pass/fail data with low failure base rates.\nsource: high agreement but low kappa: i. the problems of two paradoxes (feinstein & cicchetti) | https://doi.org/10.1016/0895-4356(90)90158-l | 1990-01-01 | academic\nimplication: answers taxonomy q1 stats question: standardize on reporting a chance-corrected coefficient plus per-class recall at fixed prevalence plus cis; ban both raw percent-agreement and any single coefficient in isolation, because at extreme pass rates alpha/kappa collapse even for a well-functioning judge.\n\n### classical label-aggregation models (dawid-skene 1979, mace 2013 with its explicit spammer latent variable, glad) all assume annotators are conditionally independent given the true label - an assumption measurably violated when the 'annotators' are llm judges sharing data/architectures/prompts, causing miscalibrated posteriors and confidently wrong aggregate verdicts; feb 2026 work replaces this with dependence-aware ising-model aggregation.\nevidence: balasubramanian, podkopaev & kasiviswanathan (amazon/uc davis, arxiv 2601.22336, feb 2 2026) state the ci assumption 'is often violated by llm judges due to shared data, architectures, prompts, and failure modes' and that ignoring dependencies 'can yield miscalibrated posteriors and even confidently incorrect predictions'; they build aggregation via ising models. mace (hovy et al., naacl 2013) remains the canonical spam-aware human aggregator, with production implementations in toloka's crowd-kit.\nsource: dependence-aware label aggregation for llm-as-a-judge via ising models (arxiv 2601.22336) | https://arxiv.org/pdf/2601.22336 | 2026-02-02 | academic\nimplication: if autoqa uses k-sample voting or multi-judge ensembles, naive dawid-skene/majority aggregation will overstate confidence because same-family judge samples share failure modes; confidence estimates must model correlated errors (or use genuinely different model families), and mace-style competence/spam latent variables remain the right tool for the human-reviewer side.\n\n### llm contamination in low-dimensional annotation data (multiple-choice labels, ratings) can be evaluated at the worker level without any ground truth, using peer-prediction scores that condition on llm-generated reference labels - a training-free mechanism with theoretical guarantees under an llm-collusion model, published at neurips 2025 - because text-based detectors are inapplicable to short label outputs.\nevidence: zhang, pang, zhu & liu (neurips 2025; arxiv 2506.06991, opened and verified): existing detectors need high-dimensional text; their mechanism scores the informativeness of worker answers via inter-worker correlation conditioned on a subset of llm labels, explicitly modeling collusion (multiple workers pasting from the same llm), and empirically detects low-effort cheating on real crowdsourcing datasets. scoring is per-worker, not per-item.\nsource: evaluating llm-contaminated crowdsourcing data without ground truth (neurips 2025) | https://arxiv.org/abs/2506.06991 | 2025-12-01 | academic\nimplication: for rating/label-type attempter outputs where stylistic detection is impossible, the viable contamination sensor is worker-level statistical scoring (cross-worker correlation vs. an llm-conditioned baseline) run continuously - reinforcing that provenance/behavioral defense operates at the annotator level, not the item level.\n\n### the nlp field has institutionalized disagreement-as-signal: the third learning-with-disagreements shared task (lewidi-2025, emnlp 2025) standardizes dual evaluation - soft-label (predict the population label distribution, wasserstein/manhattan distance) and perspectivist (recover individual annotators' labels) - across four datasets that all ship per-annotator labels rather than adjudicated gold.\nevidence: leonardelli et al. (arxiv 2510.08460, oct 2025; le-wi-di.github.io) describe the two complementary paradigms and public leaderboard; lineage runs from aroyo & welty's 'truth is a lie' perspectivism through dices (neurips 2023 safety dataset with up to 100+ raters/item) and jury learning (gordon et al., chi 2022, which composes verdicts from explicitly selected rater subpopulations); frenda et al.'s survey (lrev 2024) documents the dataset ecosystem.\nsource: lewidi-2025 at nlperspectives: third edition of the learning with disagreements shared task | https://arxiv.org/pdf/2510.08460 | 2025-10-09 | academic\nimplication: the autoqa data model should preserve rater identity and full label distributions (never only adjudicated verdicts), and for judgment-laden axes the judge's target should be the distribution/jury composition, not a forced scalar - this is now the field-standard evaluation contract, giving ready-made metrics (wasserstein/manhattan soft-label distance) for 'consistency-of-application' monitoring.\n\n### gold/test labels themselves are wrong at material rates - a lower-bound average of 3.3% label errors across 10 canonical benchmarks (at least 6% in the imagenet validation set), validated by human review of confident-learning-flagged candidates (51% flag precision) - and error rates this size are enough to flip model rankings.\nevidence: northcutt, athalye & mueller (neurips 2021 datasets & benchmarks; arxiv 2103.14749, opened and verified): algorithmic flagging via confident learning + crowdsourced verification; resnet-18 beats resnet-50 on corrected imagenet if mislabeled prevalence rises just 6%. cleanlab is the maintained open-source implementation.\nsource: pervasive label errors in test sets destabilize machine learning benchmarks (neurips 2021 d&b) | https://arxiv.org/abs/2103.14749 | 2021-11-01 | academic\nimplication: answers taxonomy q1's false-agreement question in the only measured form available: several percent of any 'gold' set is wrong, so judge-vs-gold disagreement at that rate is uninformative; the gold set needs its own standing audit channel (confident-learning-style flagging + expert re-adjudication) as a permanent component, and judge certification margins must exceed the gold error floor.\n\n### showing annotators llm suggestions makes them anchor on the suggestions - significantly shifting the label distribution versus unassisted baseline, raising self-reported confidence without making them faster - and evaluating models against llm-assisted labels significantly inflates reported model performance.\nevidence: schroeder, roy & kabbara (findings of acl 2025, july 2025; pre-registered, 350 annotators, 7,000 annotations, 4 conditions x 2 models x 2 datasets, opened and verified): annotators 'strongly took the llm suggestions'; the paper warns homogenization/lessened variation is a direct risk when llms are inserted into annotation loops, and that 'human-approved' llm-assisted labels change downstream conclusions.\nsource: just put a human in the loop? investigating llm-assisted annotation for subjective tasks (findings of acl 2025) | https://aclanthology.org/2025.findings-acl.1323/ | 2025-07-27 | academic\nimplication: direct evidence for taxonomy q8 monoculture and q9 induced drift: if autoqa feedback (or any ai verdict) is visible to attempters/reviewers before they judge, distributions converge on judge-pleasing labels and self-evaluation inflates - so human touches must be blind-first, and population-level label-distribution drift is the right monoculture sensor.\n\n### rlvr-style reasoning training degrades an llm's ability to model human annotator disagreement, while naive chain-of-thought on rlhf models improves it - evaluated across 60 setups on 3 tasks - meaning stronger 'reasoning' judges are measurably worse at knowing when humans legitimately diverge.\nevidence: ni et al., eacl 2026 main (arxiv 2506.19467 v3, jan 12 2026, opened and verified): systematic evaluation over model sizes, distribution-expression and steering methods using variance-correlation and distributional-alignment metrics; authors explicitly warn about 'the potential risk of replacing human annotators with reasoning llms' where disagreement carries signal; author communication adds that small fine-tuned models (modernbert-class) with human labels often beat large reasoning models at this.\nsource: can reasoning help large language models capture human annotator disagreement? (eacl 2026) | https://arxiv.org/abs/2506.19467 | 2026-01-12 | academic\nimplication: for the 'instructions-underdetermine / route-to-human' triage lane, do not assume the strongest reasoning judge is the best ambiguity detector: disagreement-prediction may need a dedicated small fine-tuned model (or cot-on-rlhf configuration) separate from the verdict judge.\n\n### tools & artifacts\n- cleanlab (confident learning): open-source library implementing confident learning for algorithmic label-error detection in labeled datasets | https://github.com/cleanlab/cleanlab | ready-made machinery for the standing gold-set audit channel: flag suspect gold/qa labels for expert re-adjudication (51% flag precision measured in the neurips 2021 study).\n- crowd-kit (toloka): maintained python library of aggregation/truth-inference algorithms: dawid-skene, glad, mace, wawa, zerobasedskill, ibcc/ebcc | https://github.com/toloka/crowd-kit | production implementations for aggregating multiple human reviewers (and multi-judge samples) with per-worker competence and spammer modeling - but note the conditional-independence caveat for same-family llm judges.\n- lewidi-2025 datasets + leaderboard: four benchmark datasets with per-annotator labels plus standardized soft-label (wasserstein/manhattan) and perspectivist evaluation metrics, emnlp 2025 shared task | https://le-wi-di.github.io/ | off-the-shelf metrics and data format for evaluating disagreement-aware judge components and for the 'consistency-of-application vs agreement-with-truth' distinction.\n- dices dataset (google): diversity-in-conversational-ai-safety dataset: 990 conversations with dense parallel safety ratings (up to 100+ demographically diverse raters per item) | https://github.com/google-research-datasets/dices-dataset | reference dataset for measuring how much rater variance survives on a judgment-laden axis with a fixed rubric - useful for calibrating the variance-attribution pilot.\n- beyondcorrelation (amazon science): code for uncertainty-stratified meta-evaluation of automatic evaluators: perception charts, binned jensen-shannon divergence | https://github.com/amazon-science/beyondcorrelation | directly implements the recommended meta-evaluation reporting standard (stratify by human agreement) for certifying the autoqa judge.\n- can crowdsourcing survive the llm era? (community survey): june 2026 arxiv survey (2606.04924) synthesizing prevalence, detection, and mitigation literature on llm use in human data collection | https://arxiv.org/html/2606.04924v1 | most current single map of the llm-contamination literature; documents detector degradation under paraphrase and the state of mitigation practice as of mid-2026.\n\n### disagreements\n- detectability of llm-generated submissions: veselovsky et al. (2023/2025) achieved usable classifier-based detection on a long-form text task, and tu delft's prompt-injection work (tochi 2025) claims in-instrument countermeasures work; westwood (pnas 2025) and the community survey (arxiv 2606.04924, june 2026) show detection and quality checks are comprehensively evaded by agents and degrade under paraphrase/out-of-domain, with mixed human-ai authorship hardest. the recent evidence favors westwood, but the sources genuinely conflict on whether content-side detection retains any value as a secondary signal.\n- nature of disagreement: the aggregation tradition (dawid-skene, mace, majority vote) models disagreement as noise around one latent truth to be recovered; the perspectivist tradition (aroyo & welty, dices, lewidi-2025, diverging preferences) treats much of it as legitimate signal to be preserved. neither side claims universality, but they prescribe opposite data models (adjudicated gold vs. per-rater distributions) - the design must pick per-axis, and no source offers a validated automatic classifier for which regime an axis falls into.\n- agreement statistics under skew: krippendorff defends alpha as the general-purpose reliability standard; gwet and successors show alpha/kappa collapse under extreme prevalence and push ac1/ac2; elangovan et al. (iclr 2025) argue all human-reliability coefficients are misapplied to machine-vs-human comparison in the first place. there is no consensus single statistic - only convergence that one-number reporting is wrong.\n- mitigation of crowdworker llm use: veselovsky et al. measured that request+hurdle interventions halve usage (a meaningful effect worth deploying); westwood concludes no instrument-level mitigation withstands agent-based completion ('no magical fix'). these are consistent about direction (partial reduction) but conflict sharply on whether prevention is worth its measured quality cost.\n\n### gaps\n- no published measurement of the false-agreement rate (judge and attempter agreeing while both wrong per expert panel) in an annotation-qa setting; the nearest proxy is northcutt's 3.3%+ gold-label error floor. this number would have to be produced in-house via an independent expert-audit channel.\n- no public 2025-2026 prevalence data for llm use specifically among paid annotators/reviewers on managed ai-training-data platforms (scale, surge, outlier, prolific-managed rlhf work); all published prevalence numbers come from mturk/prolific survey and summarization tasks, which may under- or over-state prevalence for critique/rating work.\n- item-level (as opposed to worker-level) detection of llm-generated annotations for short/low-dimensional outputs remains unsolved; the neurips 2025 peer-prediction mechanism scores workers, not items, and requires overlapping assignments across workers.\n- no annotation-science source found that gives sample-size arithmetic for certifying a judge against a gold set under low failure base rates (per-project vs pooled hierarchical validation); this appears only in general eval-statistics literature outside this domain's charter.\n- no longitudinal studies found measuring whether qa feedback changes annotator behavior over time (repeat-error rates per criterion); the feedback-efficacy question (taxonomy q7) has essentially no direct evidence in the annotation-quality literature.\n- could not open the full text of the diverging preferences taxonomy to extract exact percentage splits across its 10 disagreement categories (abstract confirms only that underspecification + style form the majority); exact fractions would sharpen the variance-attribution prior.\n\n### verifications\n- claim: roughly one-third of crowdworkers used llms on an llm-advantaged text-production task (33-35% on prolific, july 2023; ~34% prolific self-report in a 2025 follow-up by zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%).\n verdict: partially_confirmed | core claim verified against the full pdf of arxiv 2310.15683 (veselovsky, horta ribeiro, cozzolino, gordon, rothschild, west): study 1, n=168 prolific workers, 3 july 2023, medical-abstract summarization; prevalence 33.3% [25.9,40.1] (classify-and-count), 35.2% [29.8,40.6] (probabilistic), 35.4% [27.8,43.0] (corrected) - matches \"33.3-35.4%, cis ~[26%,43%]\". study 2, n=720, 23 july 2023, 3x3 factorial (request: none/indirect/direct x hurdle: none/image/no-ctrl-c+v); classifier estimate dropped 27.6% -> 15.9% (direct+image) and 15.8% (direct+ctrl c+v) - matches \"roughly halved, not eliminated\" (paper: \"reduced llm use by nearly 50%, it could not fully prevent it\"). direct request decreased keyword retention by 6.2% (p=0.009) - exact match. classifier was finetuned e5-base-v2, calibrated (card & smith), with self-report and high-precision heuristics (completion time + paste artifacts) - matches. the june 2026 community survey arxiv 2606.04924 (velutharambath et al., 155 researchers, survey run aug 2025-mar 2026) does cite \"veselovsky et al. estimate 30-40%\", cites \"zhang et al. (2025): 34% of prolific participants self-report using llms for open-ended questions\", and states mitigations \"reduce rather than eliminate llm-assisted responses\" - so the follow-up attribution to zhang et al. 2025 is consistent with that survey's citations (note: i verified the citation exists in the survey, not the underlying zhang et al. paper itself). issues: (1) date \"2025-02-18\" is wrong for the arxiv link given - arxiv has only v1, dated 24 oct 2023; the published version is communications of the acm 68(3):42-47, march 2025 issue (doi 10.1145/3685527). (2) \"preregistered\" is not supported by the paper text - no preregistration is mentioned in the arxiv pdf. (3) minor nuance: \"never to zero\" - by the classifier measure true, but the high-precision heuristics table actually shows 0.0% in the direct+ctrl c+v cell; the paper's own framing (\"cannot fully prevent\") still supports the claim's spirit. (4) 44% of surveyed researchers in the 2026 survey observed llm use in their data - newer corroborating, not superseding, evidence; no later work found that overturns the prevalence estimates. sources: https://arxiv.org/abs/2310.15683, https://arxiv.org/abs/2606.04924, https://dl.acm.org/doi/10.1145/3685527.\n corrected: roughly one-third of prolific crowdworkers used llms on an llm-advantaged text-production task (33.3-35.4% across three estimators, july 2023, n=168; veselovsky et al., arxiv 2310.15683 v1 posted 24 oct 2023, published in communications of the acm 68(3), march 2025). in a second study (n=720, 3x3 factorial), the best mitigations (direct request + image presentation or + copy-paste disable) roughly halved classifier-estimated usage from 27.6% to 15.9%/15.8% without eliminating it, and directly requesting non-use reduced keyword retention by 6.2% (p=0.009). the studies were not described as preregistered. a june 2026 community survey of 155 researchers (arxiv 2606.04924, velutharambath et al.) corroborates, citing veselovsky et al.'s 30-40% estimate and zhang et al. (2025)'s 34% prolific self-report, and finding 44% of researchers observed llm use in their crowdsourced free-text data; it echoes that mitigations reduce rather than eliminate llm-assisted responses.\n- claim: an autonomous llm agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused ai-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025.\n verdict: confirmed | every quantitative element of the claim verifies against the primary source (westwood, \"the potential existential threat of large language models to online survey research,\" pnas 122(47), published nov 20, 2025; pubmed 41264250, cited 81x by jul 2026; won a 2025 cozzarelli prize). extracted paper text confirms verbatim: (1) 99.8% pass rate across 6,000 trials (20 acq types x 300 repetitions), only 10 errors, perfect on 18/20 check types; (2) execution layer \"simulates realistic reading times calibrated to the persona's education level, generates human-like mouse movements, and types open-ended responses keystroke-by-keystroke, complete with plausible typos and corrections\"; (3) strategically declines 97.7% of \"reverse shibboleth\" tasks; (4) ~$0.05 per survey with commercial models vs $1.50 payment (>96.8% margin), marginal cost near zero with local open-weight models; (5) 43,800 total evaluations, 139 questions, 6,700 trials, with a key subset validated across nine llms (deepseek r1, mistral large, claude 3.7 sonnet, grok 3, gemini 2.5, qwen plus, llama 4 maverick, gpt-4.1, o4-mini); (6) 0% error rate on impossible-event/rare-event probes. the paper's abstract and press framing support the interpretive conclusion that coherence-/content-based detection is no longer tenable and defenses must be provenance/economic. supersession check: no 2026 work refutes it - follow-ups (westwood & frederick 2026 prolific audit; a 2026 sage ampps review of ai-mediated contamination) build on it and treat detection as an unsolved provenance problem. two minor caveats, both in the evidence summary rather than the claim: (a) arxiv 2606.04924 (\"can crowdsourcing survive the llm era?\", jun 2026) is a community survey of 155 researchers' practices/experiences, not a technical detector benchmark; its abstract does not state the paraphrase/out-of-domain/mixed-authorship degradation findings attributed to it (those likely come from a different detection-benchmark paper). (b) on recaptcha the paper says the system \"is designed to accommodate tools for bypassing\" recaptcha - an architectural capability, slightly weaker than \"recaptcha... fail[s]\" as an empirically demonstrated result. neither caveat affects the claim text itself, which is fully accurate.\n- claim: correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target.\n verdict: partially_confirmed | substance confirmed on every checked point against the arxiv record (https://arxiv.org/abs/2410.03775): (1) abstract states machine labels \"may superficially appear to have similar or better correlation with the human majority\" when human uncertainty is high, and machine-human correlation falls \"well below\" human-human correlation as human label consistency increases; (2) it flags krippendorff's alpha / randolph's kappa as designed for human-human reliability with assumptions inapplicable to machine labels; (3) proposed fixes match exactly - stratification by human label uncertainty, binned jensen-shannon divergence for perception-type tasks, and 'perception charts'; (4) venue is iclr 2025 per the arxiv comments field; (5) code is open-sourced at github.com/amazon-science/beyondcorrelation. the only inaccuracy is the source date: the claim says 2025-05-01, but the latest arxiv version is v3 dated 2025-01-27 (original submission 2024-10-03) - no may 2025 version exists on the record. supersession check: a follow-up search surfaced related later work - \"validating llm-as-a-judge systems under rating indeterminacy\" (arxiv 2503.05965), \"hidden measurement error in llm pipelines distorts annotation, evaluation, and benchmarking\" (arxiv 2604.11581), and \"care: confounder-aware aggregation for reliable llm evaluation\" (arxiv 2603.00039, feb 2026). these extend and reinforce the human-uncertainty/measurement-error critique rather than refute it; nothing found supersedes or contradicts the core finding. verdict is partially_confirmed solely because of the incorrect source date; the claim itself stands.\n corrected: correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. source: elangovan et al., \"beyond correlation...\" (amazon science), accepted at iclr 2025, arxiv 2410.03775 (v1 2024-10-03, latest v3 2025-01-27), code at github.com/amazon-science/beyondcorrelation.\n\n---\n\n## domain: critique-models\n\n*criticgpt lineage, human+critic teams, omission detection*\n\n### human+critic-model teams occupy a strictly better operating point than either alone: in openai's criticgpt study, human+criticgpt teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (fsbs length penalty), not a fixed property. [load-bearing]\nevidence: criticgpt critiques were preferred over human contractor critiques in 63% of cases on naturally occurring llm errors (>80% elo win rate on inserted bugs); both chatgpt and criticgpt caught substantially more inserted bugs than paid contractors (median ~50 min/review, ~5 yrs python experience). humans alone had far fewer hallucinated bugs/nitpicks; human+criticgpt teams landed between, 'moving beyond the model-only pareto frontier.' force sampling beam search scores candidates by rm_score + length_modifier x num_highlights, tracing a precision/comprehensiveness pareto curve selectable at deployment without retraining. verified directly from the arxiv html full text.\nsource: llm critics help catch llm bugs (criticgpt, openai) | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary\nimplication: validates the hybrid one-touch foundation: ai critic drafts findings, human filters hallucinated nitpicks. make the precision/comprehensiveness operating point an explicit per-project config knob (q2/q4/q7 false-accusation slo), since the same trained judge can be run strict or comprehensive per project.\n\n### critic-assisted review of items already rated 'flawless' by a first human found real, rating-changing problems in 24% of flagged cases, versus 6% found by an unassisted second human review of the same data - a 4x lift on the exact 'confident pass' slice.\nevidence: criticgpt (rl-only) was run over chatgpt training data rated flawless by a first annotator; where the critique flagged a problem, reviewers agreed it substantially lowered the rating in 24% of cases, vs 6% in a no-critique second-rater replication - and this held out-of-distribution (mostly non-code tasks). prioritizing by critique reward-model score further raised catch rates and cut hallucinations. verified from arxiv html full text.\nsource: llm critics help catch llm bugs (criticgpt, openai) - 'flawless' data experiment | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary\nimplication: direct evidence for q1's false-agreement problem and q6's routing: unassisted human re-review of passed items misses ~3/4 of what critic-assisted review catches, so the standing expert-audit channel over confident passes should itself be critic-assisted, and critic-confidence scores can prioritize the audit queue.\n\n### in deepmind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (ai rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% ai-alone and 75.1% human-alone - and showing humans the ai's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the ai was right without hurting when it was wrong. [load-bearing]\nevidence: 1,918 expert-labeled fact-verification items; tool-using ai rater with majority-vote-over-50-samples confidence (well calibrated). ai 87.7% vs individual humans 75.1% (majority vote 80.6%). hybridization at threshold 0.62: 89.3%; on the routed low-confidence slice humans beat the ai 71.3% vs 60.5% (real complementarity region). evidence-only assistance: 79.3% when ai right (vs 71.3% baseline) with no significant harm when ai wrong; conditions showing judgments+reasoning/confidence produced significant over-reliance decrements (e.g. beta = -0.768, p < .001); ai-debate condition was numerically worst. verified from arxiv html full text.\nsource: human-ai complementarity: a goal for amplified oversight (google deepmind) | https://arxiv.org/abs/2510.26518 | 2025-10-30 | primary\nimplication: answers q6 concretely: spend the one human touch on ai-low-confidence items, and give that human the judge's collected evidence spans without the judge's verdict or rationale (blind-ish adjudication over curated evidence). requires the judge to emit calibrated confidence and quoted evidence as first-class outputs (q3), and argues against 'human verifies ai verdict' as the default interaction.\n\n### a nature human behaviour meta-analysis of 100+ experiments (300+ effect sizes) found human-ai combinations on average performed significantly worse than the best of human or ai alone on decision/judgment tasks, with gains only where humans outperformed the ai solo - naive human-checks-ai designs destroy value.\nevidence: vaccaro, almaatouq & malone systematic review/meta-analysis: average human-ai combo underperformed the stronger party alone; losses concentrated in decision-making tasks, gains in content-creation tasks; combos helped when humans beat ai alone. deepmind's complementarity paper explicitly frames its confidence-routing + evidence-only design as the answer to this result. verified from arxiv abstract.\nsource: when combinations of humans and ai are useful: a systematic review and meta-analysis | https://arxiv.org/abs/2405.06087 | 2024-10-28 | contrarian\nimplication: the q6 fork ('is one shallow touch better than zero touch plus deep audits?') is empirically live: an automation-biased shallow verify step is the modal failure in the literature. the touch only pays where the human plausibly beats the judge (routed low-confidence/contested items), which mandates closed-loop routing on observed overturn rates rather than fixed per-item review.\n\n### in the largest deployed rct of ai critiquing human evaluative work (iclr 2025, feedback on >20,000 peer reviews), 27% of reviewers who received llm feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated llm reliability tests. [load-bearing]\nevidence: randomized controlled study at iclr 2025 (~22.5k feedback-arm vs ~22.4k control reviews): review feedback agent flagged vague comments, content misunderstandings, unprofessional remarks; guardrail llm tests gated delivery; 26.6-27% update rate, avg +80 words among updaters, blinded informativeness gains, longer author-reviewer discussions. verified from arxiv abstract page; corroborated by the official iclr blog.\nsource: can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025 | https://arxiv.org/abs/2504.09737 | 2025-04-13 | academic\nimplication: q7's central bet is supported at deployment scale: constructive, item-specific ai feedback measurably changes the behavior of human evaluators, and the shipped pattern held feedback to its own automated verification gate before delivery (fail-closed feedback). also gives a realistic effect-size prior: ~1 in 4 recipients act on feedback - plan longitudinal metrics around that base rate.\n\n### llms are structurally weak at detecting omissions: on absencebench, average f1 drops 56.9 points versus detecting the same content as insertions (best model 71.2% overall, 40.0% on code diffs, vs ~99.5% needle-in-haystack), and inserting explicit placeholders at gap sites recovers ~35.7 points - absence has no attention key to attend to. [load-bearing]\nevidence: 4,302 instances across poetry, numerical sequences, github prs (~5k token contexts): models must list deliberately deleted elements given original+modified documents. gemini-2.5-flash (thinking) best at 71.2% avg f1; mixtral-8x7b scores 99%+ on niah but 14.7% here; thinking tokens buy only ~7.9%; '<missing line>' placeholders add up to +81.8%. neurips 2025 datasets & benchmarks. verified from arxiv html full text.\nsource: absencebench: language models can't tell what's missing | https://arxiv.org/abs/2506.11440 | 2025-06-13 | academic\nimplication: caps what the judge can verify autonomously (q3/q5): 'comprehensiveness'/'complete' praise claims and omission-type attempter failures are the judge's structurally weakest lane. the rubric compiler should convert completeness criteria into explicit enumerable checklists (placeholder effect = give absence a token), and free-form omission judgment belongs in the human or human-assisted lane, not judge-autonomous.\n\n### critique quality is itself quantifiable at usable reliability: metacritique decomposes critiques into atomic information units and scores precision/recall against references, with gpt-4 aiu-level judgments at 85-89% accuracy and meta-f1 correlating with human gold at pearson 0.84-0.89 - and measured baselines show human critiques are high-precision/low-recall (87.6%/48.7%) while llm critiques are the inverse-ish (71.9%/53.3%).\nevidence: acl findings 2024 framework: precision = fraction of critique aius judged factual; recall = coverage of reference-critique aius; each judgment carries a natural-language rationale. choosing critiques by meta-f1 yields refinements that win 51%/lose 27% by human eval, vs gpt-4 pairwise picks losing more than winning. ~28% of llm-critique aius were non-factual vs ~12% for humans; llm critiques carried 2.4x the information volume (8.10 vs 3.31 aius). verified from arxiv html full text.\nsource: the critique of critique (metacritique) | https://arxiv.org/abs/2401.04518 | 2024-01-09 | academic\nimplication: q7's reflexive-grounding requirement is buildable today: run autoqa feedback through aiu-level precision scoring as its own admission gate, and report critique-precision/recall as first-class system metrics. also sets expectations: the ai side of the pipeline supplies coverage, the human side supplies precision - mirroring criticgpt.\n\n### verdict accuracy and critique validity dissociate badly: in criticbench-thu data 24.8% of items got the correct verdict with a low-quality critique, and open-loop verdict-agreement metrics compress a 27.1-point real error-identification gap (processbench: o1-mini 88.9 vs qwen2.5-72b 61.8) into a 1.3-point verdict-f1 gap - so critique quality must be evaluated closed-loop by whether the critique drives a successful correction.\nevidence: realcritic (jan 2025, arxiv 2501.14492): closed-loop critique-then-correct design over 8 reasoning benchmarks; classical llms (incl. gpt-4) lose accuracy under self-critique (-1.8 to -5.1 avg; up to -35.6 domain drops) while o1-mini is the only model with positive self-critique delta; failure modes 'superficial success' (right verdict, wrong analysis) and 'contradictory output' documented. independently confirmed at scale in 2026: an icml 2026 paper (rm-nlhf, arxiv 2601.07349) finds outcome-rewarded generative reward models 'guess correct outcomes without sound critiques' and uses similarity-to-human-critique as a process reward to fix it. realcritic verified via arxiv abstract + html (html render partially draft-quality; abstract and appendix numbers consistent).\nsource: realcritic: towards effectiveness-driven evaluation of language model critiques | https://arxiv.org/abs/2501.14492 | 2025-01-24 | academic\nimplication: for q1 meta-evaluation: never certify the autoqa on verdict agreement alone - a judge can match pass/fail labels while its rationales are wrong, which poisons the feedback channel and attempter trust. validation must score the rationale (does the cited evidence entail the finding; does acting on the feedback fix the item), i.e., q7's explainability contract is also the correct validation instrument.\n\n### a june 2026 study of 21 llm judges (~541k judgments) shows raw percent-agreement overstates chance-corrected agreement by 33.8-41.3 points (85% agreement = kappa ~0.48), judge rankings flip by up to 15 positions across benchmarks, and the most reproducible judges are among the least valid (test-retest 0.99 with position bias 0.19) - leading the authors to prescribe a pre-deployment minimum viable validation protocol.\nevidence: 'reliability without validity' (uc berkeley, arxiv 2606.19544, evals run march-april 2026): mt-bench/judgebench/rewardbench, 118 runs at temperature 0; kappa deflation universal across all 21 judges; consistency-bias paradox instantiated by qwen3-8b and gemini 2.5 flash; mvvp = report kappa/krippendorff alpha as headline, ab+ba position swaps, >=3 replicates, >=2 benchmarks spanning preference- and correctness-style labels, and audit that high stability is not just high bias. verified from arxiv html full text.\nsource: reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic\nimplication: directly answers q1's statistics question (ban raw percent-agreement; standardize on chance-corrected stats) and q5's ship-gate question (perturbation/position-swap + replicate harness is now published prescriptive practice). run-to-run consistency must never be reported as evidence of validity - a perfectly consistent judge can be laundering a bias.\n\n### the closest production analog to the proposed autoqa already exists: toloka's deployed 'llm qa' runs a tool-using agent per quality-metric on every human annotation submission, emits a strict three-way pass / fail / unable-to-verify verdict (the abstain class exists specifically to prevent hallucinated verdicts), coaches annotators with socratic feedback, and explicitly accepts lower precision on fail because a false pass costs more than escalating a genuine pass.\nevidence: toloka engineering blog (bylined 2026-03-30): agentic autocheck with web/image/audio/python/bash tools; 'one metric, one entity, one verdict' scoping (separate agent instance per criterion); internal benchmark of 300+ real submissions from 20+ live projects with senior-qa ground truth (numbers in an image, not extractable); reported constraint: performance 'entirely bounded by task design' - ambiguous guidelines produce walls of unable-to-verify. verified by opening the post.\nsource: llm qa: scaling data quality assurance technologically (toloka) | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | 2026-03-30 | practitioner\nimplication: validates several foundational choices from production: criterion-scoped judge instances (q3 decomposition at the per-axis level), a first-class abstain/escalate verdict distinct from pass/fail (q2 verdict ontology), asymmetric error costs favoring false-flags over false-passes, and rubric quality as the binding constraint (q4: tighten criteria before writing guidelines).\n\n### the foundational 2022 result behind the whole lineage: model-written critiques helped human evaluators find flaws in summaries they would otherwise have missed (including planted flaws in deliberately misleading human-written summaries), and models exhibit a discriminator-critique gap - they can often detect that something is wrong better than they can articulate why.\nevidence: saunders et al. (openai, arxiv 2206.05802): topic-based summarization assistance experiments; critiques helped on both model- and human-written summaries; framework comparing generation, discrimination, and critique ability found 'even large models may still have relevant knowledge they cannot or do not articulate as critiques'; larger models critique and self-refine better. verified from arxiv abstract.\nsource: self-critiquing models for assisting human evaluators (openai) | https://arxiv.org/abs/2206.05802 | 2022-06-12 | primary\nimplication: the discriminator-critique gap argues for a two-stage judge: use cheap discrimination (scores/uncertainty) for routing and prioritization even where articulated critique is unreliable, and hold only the articulated-critique layer to the evidence-quoting bar (q3/q7). judge-assisted human review of human-written work is validated at the root of this lineage, not just ai-output critique.\n\n### a deployed llm compliance-checker was gamed by its subjects in a single shot: the neurips 2024 author checklist assistant (234 papers) was rated useful by >70% of authors, but the organizers found the system 'not robust to gaming' by authors and concluded it is a poor substitute for human review.\nevidence: goldberg, ullah, guyon, shah et al. (arxiv 2411.03417; neurips blog dec 2024): gpt-4-turbo checklist verification offered pre-submission; surveys plus qualitative analysis showed genuine improvements in some submissions alongside demonstrated manipulation of the assistant (authors could satisfy the checker without satisfying the standard); access was restricted to authors partly to avoid biasing review. verified via arxiv abstract and organizer blog summary.\nsource: usefulness of llms as an author checklist assistant for scientific papers: neurips'24 experiment | https://arxiv.org/abs/2411.03417 | 2024-11-05 | academic\nimplication: q8's decision-boundary-leak concern has deployment evidence even without repeated feedback cycles: pay/acceptance-motivated subjects will optimize against a visible checker. defenses (rotating seeded probes, held-out human-graded golden items to detect autoqa-vs-gold divergence, abstracted rather than checker-revealing feedback) must be designed in from day one, not retrofitted.\n\n### tools & artifacts\n- review feedback agent: open-source multi-llm pipeline (zou-group/review_feedback_agent, cc by 4.0 paper) that critiques human peer reviews for vagueness, content misunderstanding, and unprofessionalism, with automated llm reliability tests gating feedback delivery. | https://github.com/zou-group/review_feedback_agent | directly reusable architecture for the constructive-feedback channel: feedback taxonomy + fail-closed guardrail tests that feedback must pass before delivery (q7).\n- metacritique: framework + dataset (gair-nlp/metacritique) scoring critique precision/recall/f1 over atomic information units with per-unit rationales. | https://github.com/gair-nlp/metacritique | off-the-shelf machinery for holding autoqa feedback to its own grounding standard and for meta-evaluating critique quality during validation (q1, q7).\n- absencebench: neurips 2025 d&b benchmark (harvey-fin/absence-bench) measuring whether models can identify deliberately omitted content across poetry, sequences, and code diffs. | https://github.com/harvey-fin/absence-bench | ready-made competence-limit test for the omission/comprehensiveness lane of any candidate judge model before assigning it completeness-verification authority (q5 lane assignment).\n- realcritic: closed-loop critique-evaluation benchmark (tangzhy/realcritic): critique quality scored by correction success, with self-/cross-/iterative-critique modes. | https://github.com/tangzhy/realcritic | template for validating the autoqa judge on rationale effectiveness rather than verdict agreement (q1 meta-evaluation design).\n- ctrl / critique-rl / deepcritic (2025 critic-training lineage): rl-trained critic models: ctrl (arxiv 2502.03492, critique via rl without human critique labels, up to 106% relative code-gen gains), critique-rl (arxiv 2510.24320, two-stage rl: discriminability then helpfulness, no stronger supervisor), deepcritic (arxiv 2505.00662, deliberate step-level math critique). | https://arxiv.org/abs/2502.03492 | state of the art for training a dedicated critic without the expensive expert-critique annotation dependency (q4's aligned-expert concern); critique-rl's discriminate-first-then-explain staging mirrors the discriminator-critique gap design.\n- on scalable oversight with weak llms judging strong llms (kenton et al., deepmind, neurips 2024): protocol comparison (debate vs consultancy vs direct qa) with weaker judges overseeing stronger agents across qa/math/code/logic asymmetries. | https://arxiv.org/abs/2407.04622 | evidence base for arbitration design (q6): debate beats consultancy everywhere tested, and open debate makes a weak judge much harder to convince of a wrong answer - a candidate protocol for contested-item escalation.\n- minimum viable validation protocol (mvvp): five-step pre-deployment judge validation checklist from 'reliability without validity' (2026): chance-corrected headline stats, ab+ba position swaps, >=3 replicates, >=2 benchmark styles, consistency-bias paradox audit. | https://arxiv.org/abs/2606.19544 | directly adoptable as the per-project judge-config ship gate the design taxonomy asks for (q5).\n- toloka llm qa: production agentic qa system for human annotation submissions: per-metric tool-using agent instances, pass/fail/unable-to-verify verdicts, socratic annotator feedback. | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | the closest commercial deployment to the proposed autoqa; its verdict ontology, criterion scoping, and guideline-quality constraint are field-tested reference points (q2, q4, q6).\n\n### disagreements\n- does adding a human to an ai judge help at all? vaccaro et al. (nature human behaviour 2024) find human-ai combos on average significantly worse than the best solo party on decision tasks, while criticgpt (2024) finds human+ai teams beyond the model-only pareto frontier and deepmind (2025) gets 91.3% hybrid vs 87.7% ai-alone. reconciliation offered by deepmind: complementarity only appears with confidence-based routing to a slice where humans genuinely beat the ai and with non-leading assistance formats - but the deepmind-adjacent follow-up ('toward human-ai complementarity across diverse tasks') found baseline hybridization added only +0.4pp when model confidence failed to identify the complementarity region. the one-touch design's value is conditional, not guaranteed.\n- what should the human be shown? deepmind (2025) finds showing the ai's judgment/reasoning/confidence causes over-reliance and that evidence-only assistance is the only safe format, whereas criticgpt's protocol and the iclr 2025 feedback agent delivered full natural-language critiques and reported net gains. task framing differs (assisting a second-party judge vs coaching the first-party author), which suggests the autoqa should show full critiques to attempters but evidence-only to human adjudicators - no single source tests both arms.\n- are llm critics better or worse than humans? criticgpt: model critiques preferred 63% and catch more bugs than paid contractors. metacritique: human critiques have much higher factual precision (87.6% vs 71.9% of atomic units factual) while llm critiques have higher coverage. both true simultaneously - the disagreement is about which axis (precision vs comprehensiveness) a qa system should privilege, and sources implicitly pick opposite defaults.\n- can critics increase comprehensiveness of review? criticgpt/saunders show critics make human review more comprehensive (catching missed flaws), but absencebench (2025) shows llms are structurally near-blind to omissions (56.9 f1 drop). reconciliation: the critic gains are demonstrated on commission-type errors (inserted/tampered bugs, incorrect statements); no source demonstrates critic gains on omission-type attempter failures, and absencebench predicts they won't appear without checklist/placeholder scaffolding.\n- what metric certifies a judge? open-loop verdict-agreement benchmarks (criticbench 2024, criticeval 2024, mt-bench-style agreement) vs realcritic's closed-loop position that verdict accuracy without correction-effectiveness is 'superficial success' (24.8% right-verdict-wrong-critique), reinforced by rm-nlhf (icml 2026) and by 'reliability without validity' (2026) showing judge rankings flip up to 15 places depending on which benchmark style you trust.\n\n### gaps\n- no public post-deployment report from openai on criticgpt's integration into the production rlhf labeling pipeline (announced intent, june 2024) - no follow-up metrics, and no officially branded successor found through july 2026; the lineage continued in academic work (ctrl, deepcritic, critique-rl, multicritique) rather than vendor deployment reports.\n- no published rct of ai critique/feedback specifically on human data-annotation qa work. closest analogs are peer review (iclr 2025) and fact-verification rater assistance (deepmind 2025). no longitudinal data anywhere on repeat-error-rate reduction per annotator under sustained ai feedback - q7's efficacy metric has no published precedent to calibrate against.\n- no measured multi-cycle adversarial study of annotators gaming a deployed critic (q8's decision-boundary leakage): the neurips'24 checklist gaming is single-shot and author-side; citation-theater/evidence-swap perturbation tests on critics were not found.\n- toloka's deployment report - the single closest production analog - publishes its benchmark design (300+ submissions, 20+ projects) but its quantitative precision/recall table is embedded in an image and was not extractable; no independent audit of any commercial annotation-qa llm system was found.\n- false-agreement rate (judge and attempter both wrong vs expert panel) is unmeasured in the literature; the criticgpt 24%-vs-6% flawless-data result is the nearest proxy and is one study, one domain (mostly), one vendor.\n- positive-claim/praise verification (is 'accurate/complete/clear' praise warranted?) has no dedicated benchmark or precision/recall measurements - metacritique-style aiu machinery has only been applied to flaw-finding critiques. this is a genuine hole given the mission's requirement to evaluate positive statements.\n- semantic scholar returned http 429 during literature-graph search (openalex/crossref coverage only for that call); one websearch call was flagged by a per-call safeguard and was successfully re-run via exa - no coverage lost, noted for provenance.\n\n### verifications\n- claim: human+critic-model teams occupy a strictly better operating point than either alone: in openai's criticgpt study, human+criticgpt teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (fsbs length penalty), not a fixed property.\n verdict: confirmed | source verified: arxiv:2407.00215 \"llm critics help catch llm bugs\" (mcaleese et al., openai), submitted 2024-06-28 - date matches. abstract directly confirms: 63% preference for model critiques on naturally occurring llm errors; models catch more bugs than paid human reviewers; human-machine teams catch similar bug counts to llm critics while hallucinating less than llms alone. full-text details corroborated via independent secondary coverage: critic-assisted contractors wrote more comprehensive critiques than unassisted contractors while reducing hallucination/nitpick rate relative to the model, described as moving beyond the model-only pareto frontier; fsbs scores candidates by rm_score + length_modifier x num_highlights, giving a deployment-time precision/comprehensiveness dial without retraining. minor reading note: \"strictly better than either alone\" holds in the paper's pareto sense - teams matched (did not exceed) the model's bug-catch/comprehensiveness while beating its hallucination rate, and exceeded human comprehensiveness; the claim's own wording states exactly this, so no correction needed. supersession check: no official openai follow-up found; later work (codecriticbench 2025; 2026 papers on llm reviewer overcorrection) extends but does not contradict these findings.\n- claim: in deepmind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (ai rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% ai-alone and 75.1% human-alone - and showing humans the ai's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the ai was right without hurting when it was wrong.\n verdict: confirmed | every quantitative element checks out against the arxiv html full text (2510.26518). dataset: 1,918 expert-labeled tuples - confirmed. ai rater 87.7%, individual humans 75.1%, human majority vote 80.6% - confirmed. confidence via 50 samples/sentence (avg 33.25 passing format check), described as fairly well calibrated - confirmed. unassisted hybridization at threshold t=0.62: 89.3% (beta=0.413, p=.012) - confirmed. low-confidence routed slice (280 items): humans 71.3% vs ai 60.5% (p=.006) - confirmed. the headline 91.3% is specifically the assisted hybrid (ai when confident + evidence-assisted humans on the low-confidence slice), vs 89.3% unassisted hybrid - exactly as the claim words it, so 91.3% is correctly attributed. evidence-only assistance: 79.3% vs 71.3% baseline when ai correct (beta=0.446, p=.009), no significant harm when ai wrong (64.0% vs 61.5%); paper explicitly calls it \"the only form of assistance that achieves the ideal of helping when correct and not hurting when wrong\" - confirmed. over-reliance when ai wrong: evidence&reasoning&judgments beta=-0.768 (se=0.174, z=-4.413, p<.001), er&j&confidence beta=-0.636 p<.001, judgments&confidence beta=-0.356 p=.042 - the quoted beta=-0.768 matches the evidence&reasoning&judgments condition exactly. debate was numerically worst (64.9%, only format below baseline) - confirmed. date: v1 submitted 2025-10-30 as claimed. currency: a v2 was posted 2026-06-25 and the paper was published at acm facct '26 (doi 10.1145/3805689.3812308); this is publication/revision, not supersession - no later work contradicting the findings found. deepmind blog notes ongoing follow-up combining hybridization+assistance on an internal rating task, unpublished as of 2026-07-14. minor caveat only: whoever cites this should prefer the v2/facct version in case numbers shifted slightly in revision (v1 numbers verified here).\n- claim: in the largest deployed rct of ai critiquing human evaluative work (iclr 2025, feedback on >20,000 peer reviews), 27% of reviewers who received llm feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated llm reliability tests.\n verdict: confirmed | source verified directly (arxiv:2504.09737, v1 submitted 2025-04-13 - date matches). abstract states: review feedback agent deployed at iclr 2025 as a large randomized controlled study on >20,000 randomly selected reviews; feedback targeted vague comments, content misunderstandings, unprofessional remarks; 27% of reviewers who received feedback updated their reviews; >12,000 suggestions incorporated (12,222 in the paper); +80 words average among updaters; more informative per blinded evaluators; longer author-reviewer discussions; feedback delivered only if it passed a suite of automated llm reliability tests. arm sizes corroborated from paper figures: 22,467 selected for feedback vs 22,364 control; 18,946 successfully received feedback, of whom 26.6% updated (the abstract rounds to 27%). two nuances, neither contradicting the claim: (1) the superlative \"largest deployed rct of ai critiquing human evaluative work\" is the researcher's framing, not verbatim in the abstract - no larger counterexample found, and it is consistent with the study's scale; (2) not superseded but now formally published: nature machine intelligence, feb 23, 2026, as \"a large-scale randomized study of large language model feedback in peer review\" (https://www.nature.com/articles/s42256-026-01188-x) - the nmi version is the preferred citation going forward. a related follow-up mixed-methods study on reviewer perceptions exists (arxiv:2602.13817) but does not alter these results.\n\n---\n\n## domain: grounding-faithfulness\n\n*entailment checkers, decomposition, citation evaluation*\n\n### grounded claim-verification is commoditized but ceilinged: the best model on the llm-aggrefact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7b model, bespoke-minicheck-7b, at 77.4 average balanced accuracy, with 0.4-0.8b specialized checkers (factcg, minicheck-flan-t5-l) within ~2.5 points of frontier llms like claude-3.5-sonnet (77.2) and gpt-4o (75.9). [load-bearing]\nevidence: fetched the live llm-aggrefact leaderboard: top-10 shows bespoke-minicheck-7b 77.4, claude-3.5-sonnet 77.2, granite guardian 3.3 8b 76.5, factcg-deberta-l (0.4b) 75.6, minicheck-flan-t5-l (0.8b) 75.0, llama-3.1-405b 74.4. benchmark measures binary supported/unsupported vs grounding docs. confirmed independently by paladin-mini (june 2025 arxiv) citing bespoke-minicheck as leaderboard sota.\nsource: llm-aggrefact leaderboard (minicheck project) | https://llm-aggrefact.github.io/ | 2025 (accessed 2026-07-14) | primary\nimplication: a cheap, deterministic, reproducible entailment-check stage (sub-1b to 7b checker) for 'is this attempter statement supported by the cited evidence' is off-the-shelf and costs ~1/100 of frontier calls - but ~22% claim-level error means item verdicts cannot be a naive and over claim checks; error-tolerant aggregation and confidence-routing to humans are structurally required.\n\n### on adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on faithbench, prior detectors hit ~50% accuracy (negligible), hhem-2.1-open 66.7% and bespoke-minicheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while faithjudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 f1. [load-bearing]\nevidence: fetched full text of arxiv:2505.04847v2 (vectara, emnlp 2025 industry track, v2 nov 2025). table 1: baseline detector numbers; table 2: faithjudge results; faithjudge beats the facts grounding judging prompt head-to-head on all four ragtruth/faithbench splits (e.g., faithbench 70.8 vs 54.3 f1). documented failure modes: judge underpredicts hallucinations for some generator families; 'benign'/'questionable' ternary labels classified unreliably so only binary is used; specificity drops as more in-context examples are added.\nsource: benchmarking llm faithfulness in rag with evolving leaderboards (faithjudge) | https://arxiv.org/abs/2505.04847 | 2025-11 | academic\nimplication: the strongest 2025-measured grounding-judge architecture is exactly the autoqa rubric-compilation contract: a per-project pool of human-adjudicated exemplar judgments (annotated spans + labels on comparable items) injected few-shot, not zero-shot judging and not a fixed fine-tuned checker. also: severity-graded verdicts (benign/questionable) are where both humans and judges lose reliability - keep the machine verdict binary and treat severity as a separate, human-anchored layer.\n\n### claim decomposition helps weak verifiers but actively degrades strong ones: with minicheck as verifier on wice, no-decomposition scores 80.01 balanced accuracy while factscore-style atomic decomposition drops it to 71.11; with the weaker alignscore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity. [load-bearing]\nevidence: fetched full text of 'decomposition dilemmas' (naacl 2025, arxiv:2411.02400). four-way decomposition-error taxonomy from manual inspection: (a) omission of context/logical relations, (b) ambiguity (unclear pronouns/references), (c) over-decomposition, (d) alteration of original meaning. factscore-style atomicity produces the most over-decomposition errors; veriscore-style tends to omit context. on felm, decomposition raised minicheck f1 48.1->~68 but dropped gpt-4o-mini f1 71.6->54.3.\nsource: decomposition dilemmas: does claim decomposition boost or burden fact-checking performance? | https://aclanthology.org/2025.naacl-long.320/ | 2025-05 | academic\nimplication: directly answers the granularity design question: there is no universally-best decomposition level. granularity must be tuned per verifier strength and per item complexity - atomic-claim pipelines with a strong judge are measurably worse than judging larger spans. the 'checklist myopia' risk is real and has a named mechanism (context omission + meaning alteration). budget a granularity-calibration step per project rather than fixing atomic decomposition in the foundation.\n\n### decomposition and decontextualization are in direct tension - isolating atomic facts strips the context needed to verify them, while adding context back creates multi-fact claims where the verifier may credit or penalize the wrong content; dndscore resolves this by verifying the original subclaim with the added information treated as context rather than as content to be verified.\nevidence: fetched acl anthology abstract of dndscore (wanner, van durme, dredze; emnlp 2025 main, pp. 23609-23626). the paper evaluates combinations of decomposition, decontextualization, and verification strategies and finds the strategy choice materially changes factuality scores.\nsource: dndscore: decontextualization and decomposition for factuality verification in long-form text generation | https://aclanthology.org/2025.emnlp-main.1205/ | 2025-11 | academic\nimplication: when the autoqa checks an attempter's sentence, the verification unit should be 'claim + explicit context annotations' with the verifier told which part is under test - not a free-floating atomic rewrite. this is a concrete spec for the claim-extraction layer's output schema.\n\n### verifiability triage before entailment is established machinery: veriscore extracts and scores only verifiable claims because factscore/safe 'assume that every claim is verifiable', which breaks on real long-form text containing opinions and unverifiable content; factbench's verify pipeline (acl 2025) further labels content units supported/unsupported/undecidable against retrieval, with 4,467 human-annotated units released for validation.\nevidence: fetched arxiv:2406.19276 abstract (veriscore, emnlp findings 2024): human evaluation found veriscore's extracted claims 'more sensible' than competitors across 8 long-form tasks. fetched launchnlp/factbench github (acl 2025 paper 2410.22257): three-way supported/unsupported/undecidable labeling according to retrieval results, benchmarked against factscore, safe, factcheck-gpt.\nsource: veriscore (emnlp findings 2024) + factbench/verify (acl 2025) | https://arxiv.org/abs/2406.19276 | 2024-06 (veriscore, foundational); 2025-07 (factbench acl 2025) | academic\nimplication: the claim ontology question has a field-standard answer: a front-stage classifier that routes each attempter statement into {verifiable claim, opinion/unverifiable, undecidable-given-evidence} before any entailment check. opinion-class statements get consistency-with-cited-evidence checks at most, never truth verdicts; 'undecidable' is a first-class output, not a forced pass/fail.\n\n### holistic llm judging of context-grounded outputs is far weaker than claim-level checking: on contextualjudgebench (2,000 pairs, 8 splits over rag/summarization with conditional criteria like 'faithfulness first, then completeness'), the best of 20 judge models tested (openai o1) barely reaches 55% consistent accuracy.\nevidence: fetched arxiv:2503.15620 (salesforce; acl 2025). 11 specialized judge models + 9 general-purpose models evaluated; the benchmark encodes criteria hierarchies and finds contextual assessment poses a significant challenge even to sota models. contrast: the same model class hits 77-84% on single-claim grounding tasks (llm-aggrefact, faithjudge).\nsource: does context matter? contextualjudgebench for evaluating llm-based judges in contextual settings | https://arxiv.org/abs/2503.15620 | 2025-03 | academic\nimplication: do not architect the autoqa as one holistic 'judge this writeup against the rubric' call - measured reliability roughly halves versus decomposed per-criterion, per-claim checks. criteria priority ordering (which humans dispute) must be compiled explicitly into the pipeline structure, because judges fail at applying conditional criteria internally.\n\n### grounding judges have a measured agreement-default asymmetry: in google's facts leaderboard (dec 2025), grounding judges score ~85 f1 on the positive (grounded) class but only ~46 f1 on the negative (ungrounded) class, and the grounding metric is gameable by vague, short responses that avoid unsupported claims - countered by a mandatory eligibility gate that scores non-responsive answers as inaccurate. [load-bearing]\nevidence: fetched full text of arxiv:2512.10791v1 (the facts leaderboard, google, dec 2025). best judge-prompt combos reach only ~65 macro-f1 (gemini-2.5-flash + v2 prompt: 65.33) against a 320-example human-adjudicated held-out set; positive-class ~85 vs negative-class ~46 f1; eligibility check explicitly added because 'grounding metrics can be hacked via vague, short responses'.\nsource: the facts leaderboard: a comprehensive benchmark for large language model factuality | https://arxiv.org/abs/2512.10791 | 2025-12 | academic\nimplication: two foundational mandates: (1) the sycophancy/agreement-default bias on positive claims is quantified and large - the autoqa must be evaluated on per-class recall (especially fail-class recall at fixed prevalence), never single-number accuracy; (2) an eligibility/responsiveness gate must precede grounding checks or attempters can pass by writing vacuous hedged annotations - the exact adversarial dynamic the design taxonomy flags.\n\n### cross-family judge ensembles are the standard mitigation for self-preference: facts grounding v1 (jan 2025) aggregated three frontier judges from different families (gemini 1.5 pro, gpt-4o, claude 3.5 sonnet) explicitly because 'models are biased towards favorably judging their own outputs', and grounding v2 (dec 2025) kept a two-family ensemble (gemini 2.5 flash + gpt-5).\nevidence: fetched arxiv:2512.10791v1 which states the v2 judge-ensemble rationale and composition; v1 design documented in arxiv:2501.03200 (jan 2025) and the deepmind blog (dec 17, 2024). facts parametric suite separately validated that a single-judge setup preserved rankings vs a mixed panel (gemini 2.5 pro, o3, grok 4) - i.e., ensemble necessity is task-dependent and testable.\nsource: facts grounding leaderboard (v1 paper + v2 in facts leaderboard) | https://arxiv.org/abs/2501.03200 | 2025-01 (v1); 2025-12 (v2) | academic\nimplication: when attempters critique outputs from a given model family, the autoqa judge should default to a different family or an ensemble spanning families; but ensemble overhead can be dropped where a single-judge-vs-panel ranking-preservation test passes - a cheap per-project validation, not a dogma.\n\n### span-restricted nli citation checking (the alce-style 'does the cited span entail the sentence' standard) is judged a suboptimal proxy by 2025 work: citeeval (acl 2025) argues citation quality must be evaluated against the full retrieval context, user query, and generated text - not cited sources alone - and its model-based citeeval-auto metrics correlate better with human judgments on the multi-domain citebench than nli-based metrics.\nevidence: fetched arxiv:2506.01829 (citeeval, acl 2025 main). explicitly frames binary/ternary nli vs cited sources as 'a suboptimal proxy for citation evaluation'. corroborated direction: citeguard (acl 2026, aclanthology.org/2026.acl-long.282) reframes citation evaluation as attribution alignment and notes 'the reliability of llm-as-a-judge alone is also in doubt' for citation judging.\nsource: citeeval: principle-driven citation evaluation for source attribution | https://arxiv.org/abs/2506.01829 | 2025-06 | academic\nimplication: evidence-closure design: checking attempter claims only against the spans they cite will miss the dominant human failure of citing the wrong/incomplete evidence when better evidence existed in scope. the autoqa needs two distinct checks - (a) does cited evidence entail the claim (span-closed), and (b) is the citation the right one given the full in-scope corpus (context-closed) - with (b) also catching true-but-wrongly-cited claims.\n\n### fact-verification llms are brittle to semantically-minor input perturbations: facteval (naacl 2025) tested 17 realistic word- and character-level perturbations plus 4 subpopulations on fever across zero-shot/few-shot/cot setups and found llms 'brittle to small input changes' with performance varying across subpopulations.\nevidence: fetched acl anthology page for 2025.naacl-long.534 (mamta & cocarascu). abstract is qualitative; per-perturbation numbers are in the full paper. consistent with faithjudge's finding that adding more in-context examples shifts sensitivity/specificity, and with facts' judge-prompt sensitivity (macro-f1 varies by prompt version).\nsource: facteval: evaluating the robustness of fact verification systems in the era of large language models | https://aclanthology.org/2025.naacl-long.534/ | 2025-04 | academic\nimplication: a perturbation-robustness harness (paraphrase, reorder, typo-level noise on both claims and evidence) belongs in the per-project ship gate; verdicts that flip under meaning-preserving edits should be auto-routed to human review rather than delivered.\n\n### 2026 sota joins verdict and explanation in one cheap model: faithlens (acl 2026 findings) is an 8b faithfulness-hallucination detector that jointly outputs a binary prediction and an explanation, trained via cold-start fine-tuning on filtered synthetic data plus rule-based rl rewarding both prediction correctness and explanation quality, and outperforms gpt-5.2 and o3 across 12 tasks.\nevidence: fetched acl anthology entry 2026.findings-acl.689 (si et al., findings of acl 2026, pp. 14068-14099). abstract claims superiority over gpt-5.2/o3 and 'a distinctive balance of trustworthiness, efficiency, and effectiveness'; per-task numbers require the full pdf.\nsource: faithlens: detecting and explaining faithfulness hallucination | https://aclanthology.org/2026.findings-acl.689/ | 2026-07 (acl 2026) | academic\nimplication: constructive-feedback generation and grounding verdicts do not need separate machinery: an explanation-rewarded detector produces the evidence-citing rationale as a first-class output, at 8b cost. this is the current-generation template for the autoqa's 'every criticism must carry its evidence' contract, and shows explanation quality can be an explicit training/reward target rather than a post-hoc add-on.\n\n### ragas-style prompt-chain faithfulness scoring materially underperforms finetuned/frontier judges on hallucination detection: on the 15k-sample halubench, ragas faithfulness scored 66.9% accuracy versus 87.4% for finetuned lynx-70b and 86.5% for gpt-4o; patronus has since shipped lynx 2.0 (8b, long-context, 8 hallucination subtypes including coreference and calculation errors).\nevidence: confirmed via patronus ai announcement and lynx paper coverage (july 11, 2024) with the halubench table (lynx-70b 87.4, gpt-4o 86.5, gpt-4-turbo 85.0, llama-3-70b 80.1, ragas 66.9); lynx 2.0 details from patronus docs. caveat: vendor-reported numbers on the vendor's own benchmark, though halubench is public on huggingface. ragas remains the most widely adopted oss rag-eval framework per 2026 practitioner surveys (atlan.com rag evaluation guide, 2026).\nsource: patronus ai lynx / halubench results | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | 2024-07 (lynx 1.0, foundational); lynx 2.0 later update | practitioner\nimplication: do not build the grounding stage on ragas-style statement-extraction prompt chains despite their ubiquity in 2026 tooling - the popularity/accuracy gap is ~20 points. if an off-the-shelf component is wanted, current-generation finetuned detectors (bespoke-minicheck, lynx 2.0, granite guardian, faithlens) dominate at equal or lower cost.\n\n### tools & artifacts\n- bespoke-minicheck-7b / minicheck family: specialized grounded-claim fact-checking models (0.4b-7b), sota on llm-aggrefact at 77.4 bacc; available on huggingface and ollama, ~400x cheaper than gpt-4-class verification | https://github.com/liyan06/minicheck | default candidate for the cheap deterministic claim-vs-evidence entailment stage of the autoqa\n- faithjudge (vectara): open-source llm-as-judge framework that anchors judgment with human-annotated peer examples on the same source; repo actively pushed through nov 2025 | https://github.com/vectara/faithjudge | the measured-best architecture pattern for the autoqa's per-project exemplar-anchored judge; directly reusable code\n- facts grounding public set + facts leaderboard: 860 public grounding examples (32k-token docs) plus google's dec 2025 four-suite factuality benchmark with kaggle-held private splits | https://huggingface.co/datasets/google/facts-grounding-public | source of judge-prompt designs, the eligibility-gate pattern, and human-adjudicated grounding labels for meta-evaluating our own judge\n- factbench / verify: acl 2025 factuality pipeline with supported/unsupported/undecidable labeling and 4,467 human-annotated content units released | https://github.com/launchnlp/factbench | template + validation data for the verifiability-triage front stage (fact vs opinion vs undecidable)\n- veriscore + verifastscore: verifiable-claim-only extraction/scoring for long-form text (emnlp findings 2024) and its single-model accelerated successor (emnlp findings 2025) that collapses extraction+verification into one fine-tuned pass | https://github.com/yixiao-song/veriscore | claim-extraction spec that skips opinions; verifastscore is the cost-reduction pattern if per-item pipeline latency matters\n- halubench + lynx 2.0 (patronus): 15k-sample hallucination detection benchmark with hard/ambiguous cases, plus an 8b long-context finetuned detector covering 8 hallucination subtypes | https://huggingface.co/patronusai | seeded-set material for measuring our checker's fail-class recall; lynx 2.0 is an off-the-shelf detector option\n- contextualjudgebench (salesforce): 2,000-pair benchmark for llm judges in rag/summarization settings with conditional criteria hierarchies (acl 2025) | https://arxiv.org/abs/2503.15620 | meta-evaluation instrument for any holistic judge configuration we consider; quantifies why we should not use one\n- citebench / citeeval-auto: multi-domain human-annotated citation-quality benchmark and model-based metrics that evaluate citations against full retrieval context (acl 2025) | https://arxiv.org/abs/2506.01829 | blueprint for the context-closed 'is this the right citation given everything in scope' check, beyond span-closed nli\n- hhem-2.1-open + vectara hallucination leaderboard: open small hallucination-detection model and the longest-running (2023-present, refreshed nov 2025 with 7,700 articles up to 32k tokens) hhem-scored llm leaderboard | https://huggingface.co/spaces/vectara/leaderboard | cheapest available checker tier, but its measured 66.7 bacc on hard cases (faithbench) bounds where it can be trusted - screening, not verdicts\n\n### disagreements\n- hhem's fitness as an automated grounding judge: vectara's own next-generation hallucination leaderboard (blog, 2025-11-19) still uses commercial hhem as the sole scorer and reports rates like gemini-2.5-flash-lite 3.3% vs gemini-3-pro 13.6%, while vectara's own research paper (faithjudge, emnlp industry 2025) shows hhem-2.1-open at only 66.7% balanced accuracy on faithbench - near-random on hard hallucinations - and builds an llm-judge replacement because of 'limitations observed in current hallucination detection methods'. detector choice materially changes model rankings; automated hallucination-rate leaderboards from different detectors disagree and cannot be treated as interchangeable ground truth.\n- atomic decomposition: the factscore/safe/ragas lineage (2023-2024) treats finer-grained atomic claims as strictly better for verification, but decomposition dilemmas (naacl 2025) shows decomposition degrades strong verifiers (80.0 -> 71.1 bacc on wice with minicheck) and dndscore (emnlp 2025) shows atomization and decontextualization actively conflict. the field has not converged: long-form factuality pipelines still default to atomic claims while the 2025 analysis papers say granularity must be adaptive.\n- specialized small detectors vs frontier llm judges: llm-aggrefact (through 2025) and faithlens (acl 2026) show small specialized models beating frontier llms (bespoke-minicheck-7b > claude-3.5/gpt-4o; faithlens-8b > gpt-5.2/o3), but faithjudge (2025) shows frontier reasoning judges with human-annotated exemplars beating all specialized detectors on adversarial sets (84.0 vs 71.2 bacc). both are right in their regime: fixed checkers win zero-shot on broad distributions; exemplar-anchored frontier judges win on hard, domain-specific cases - which is an argument for a two-tier architecture rather than either alone.\n- citation-support standard: alce-style span-closed nli citation precision/recall remains the de-facto automated standard, but citeeval (acl 2025) calls nli-vs-cited-spans 'a suboptimal proxy' and citeguard (acl 2026) additionally doubts llm-as-judge reliability for citation evaluation - i.e., the two dominant automated approaches each have a 2025-2026 paper attacking them, with no settled successor.\n\n### gaps\n- no published measurements of run-to-run verdict flip rates (self-consistency) for grounding checkers or llm faithfulness judges on identical inputs - the taxonomy's 'flip rate: noise vs underspecification signal' question has no literature anchor found; specialized classifiers (minicheck/hhem) are deterministic by construction but that property is unstated in evaluations.\n- positive-claim verification of evaluative statements (warranted vs empty praise, 'stated severity matches evidence strength' calibration claims) is essentially unstudied: the entire 2024-2026 literature verifies factual/descriptive claims against documents; no benchmark tests whether a checker can verify claims like 'this response is complete' or 'this citation is the strongest available'. the autoqa's core use case - verifying human evaluative writeups about ai outputs - is a domain transfer no found benchmark covers.\n- no dedicated 2025-2026 study found on evidence-swap/citation-theater probes for verifiers (does the verdict flip when cited evidence is replaced with equally-plausible irrelevant text?). facteval perturbs claims, not evidence; facts documents gaming by response vagueness, not by decorative citation. this test would need to be built in-house.\n- aggregation from claim-level verdicts to item-level pass/fail with severity weighting is under-researched: factscore-style %-supported and facts' all-or-nothing eligibility+grounding are the only documented rules; ragtruth/faithbench severity taxonomies (benign/questionable/unwanted, evident/subtle) exist as labels but faithjudge found judges classify the middle severity classes unreliably and retreated to binary - no validated severity-weighted aggregation scheme found.\n- judge-vs-adjudicated-human agreement ceilings per axis type are only available for grounding (~65 macro-f1 in facts on held-out human labels; 84 bacc for faithjudge on faithbench) - no comparable 2025-2026 numbers found for instruction-compliance or tone/style axes from the faithfulness literature, so the lane-assignment decision (judge-autonomous vs assisted vs human-only per axis) lacks published anchors outside factual grounding.\n\n### verifications\n- claim: grounded claim-verification is commoditized but ceilinged: the best model on the llm-aggrefact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7b model, bespoke-minicheck-7b, at 77.4 average balanced accuracy, with 0.4-0.8b specialized checkers (factcg, minicheck-flan-t5-l) within ~2.5 points of frontier llms like claude-3.5-sonnet (77.2) and gpt-4o (75.9).\n verdict: confirmed | fetched https://llm-aggrefact.github.io/ on 2026-07-14; every number matches exactly: bespoke-minicheck-7b 77.4 (rank 1), claude-3.5 sonnet 77.2, granite guardian 3.3 8b 76.5, gpt-4o-2024-05-13 75.9, factcg-deberta-l (0.4b) 75.6, minicheck-flan-t5-l (0.8b) 75.0, llama-3.1-405b 74.4. benchmark is 11 datasets of grounded factuality (supported/unsupported vs grounding docs), avg balanced accuracy - as claimed. the \"within ~2.5 points\" framing is accurate and even conservative: factcg (75.6) is only 1.6 below claude-3.5-sonnet and above gpt-4o. supersession search found nothing beating 77.4: acv (may 2026) reports 76.5 training-free; an acl 2026 paper ranks second to a post-trained metric; paladin-mini beats bespoke-minicheck only on its own separate benchmark subsets, not llm-aggrefact overall. caveats on the interpretive \"ceilinged\" framing: (1) the leaderboard's frontier entries are 2024-era models (claude-3.5-sonnet, gpt-4o-2024-05-13, llama-3.1-405b) - no claude 4/gpt-5-class/o3 entries, so \"specialized ~ frontier\" reflects 2024 frontiers; (2) \"verifying the verifiers\" (arxiv 2506.13342, june 2025) finds ~16% of benchmark labels ambiguous/incorrect and that few-shot frontier llms reach top-tier performance, suggesting the ~77 plateau is partly benchmark label noise rather than a pure task ceiling; (3) \"verify with caution\" (arxiv 2501.14883) shows models with similar aggregate bacc make very different instance-level predictions. none of these contradict the stated facts.\n- claim: on adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on faithbench, prior detectors hit ~50% accuracy (negligible), hhem-2.1-open 66.7% and bespoke-minicheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while faithjudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 f1.\n verdict: partially_confirmed | verified against arxiv:2505.04847 (v2 dated nov 6, 2025; emnlp 2025 industry track; vectara/waterloo authors - source and date correct). confirmed: (1) paper states prior detectors, including llm classifiers, achieved \"near 50% accuracy\" on faithbench; (2) hhem-2.1-open = 66.7% balanced accuracy on faithbench (claim-wise, table 1) - though flagged with an asterisk because hhem was used to adversarially select faithbench articles; (3) faithjudge with o3-mini-high = 84.0% balanced accuracy / 82.1 f1-macro (table 2), best zero-shot judge on faithbench was o3-mini-high at 68.8%; (4) faithjudge vs facts grounding head-to-head 70.8 vs 54.3 f1 on faithbench confirmed (table 4); (5) all three stated failure modes confirmed (underprediction for command-r/mistral/qwen generators; benign/questionable misclassified - only 10/84 benign labeled correctly; specificity slightly decreases as in-context examples increase). errors: (a) bespoke-minicheck's 71.2 is not its faithbench score - 71.2 is its balanced-accuracy average across all four datasets (aggrefact, ragtruth, tofueval-mb, faithbench) in table 1; its actual faithbench score is 60.1% claim-wise / 55.7% summary-wise. (b) minor: the \"below 78% balanced accuracy\" figure for zero-shot judges is the paper's cross-dataset average claim, not a faithbench-specific figure (on faithbench zero-shot judges max out at 68.8%, so the claim still holds directionally). supersession check: searches found no 2026 work surpassing faithjudge on faithbench; it remains the reported state of the art as of july 2026.\n corrected: on adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on faithbench, prior detectors hit ~50% accuracy (negligible); fine-tuned detectors stay weak (hhem-2.1-open 66.7% and bespoke-minicheck 60.1% claim-wise balanced accuracy on faithbench; bespoke-minicheck averages 71.2% across the four benchmark datasets); zero-shot frontier judges reach at most 68.8% on faithbench (and stay below 78% averaged across datasets), while faithjudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 f1-macro.\n- claim: claim decomposition helps weak verifiers but actively degrades strong ones: with minicheck as verifier on wice, no-decomposition scores 80.01 balanced accuracy while factscore-style atomic decomposition drops it to 71.11; with the weaker alignscore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity.\n verdict: confirmed | verified against the full pdf (extracted text at [local path withheld]). (1) source says exactly this: table 2 (wice) shows minicheck baseline bacc 80.01 vs factscore decomposition 71.11 (f1 72.32 vs 59.90); all decomposition methods hurt minicheck on wice. with the weaker alignscore verifier, factscore decomposition improves bacc 54.80->56.87 (and wice-style 56.26); note veriscore decomposition slightly hurt alignscore too, but the claim as stated (\"the same decomposition,\" i.e., factscore-style) is accurate. section 4.2 states verbatim that \"decomposition generally benefits weaker verifiers, while it tends to negatively affect stronger verification systems.\" section 6.2-6.4 confirms gains reappear as input complexity grows (complexity scale-up experiments; felmshort scale-down degrades), and figure 3 discussion states \"for each level, the maximum f1 is observed when the number of the decomposed sub-claim is less than or equal to the complexity level.\" evidence-summary side facts also check: felm minicheck f1 48.10->67.5-68.1, gpt-4o-mini f1 71.56->54.34 (table 3); four-way error taxonomy (context omission, ambiguity, over-decomposition, meaning alteration) present in section 5. (2) date: acl anthology lists april 2025 (naacl 2025, albuquerque, pp. 6313-6336); the conference ran apr 29-may 4, so \"2025-05\" is a trivial one-month imprecision, not a substantive error. (3) supersession search: later work (e.g., presupposition-free question decomposition, arxiv 2508.16838; \"alignment bottleneck in decomposition-based claim verification,\" feb 2026) extends the decomposition-tradeoff line but does not refute the verifier-strength finding.\n\n---\n\n## domain: hybrid-statistical\n\n*ppi, selective escalation, conformal routing, drift attribution*\n\n### prediction-powered inference (ppi/ppi++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a gpt-4 judge in 'autoeval done right'. [load-bearing]\nevidence: arxiv:2403.07008 (boyeau, angelopoulos, yosef, malik, jordan, uc berkeley) abstract confirmed: 'increase the effective human-labeled sample size by up to 50% on experiments with gpt-4'; v3 camera-ready posted 2026-06-01. the unconditional-coverage property is restated and empirically demonstrated in the glide paper (arxiv:2605.31278) and is the foundation of the entire 2026 llm-judge debiasing literature (precise/aaai 2026, arxiv:2606.05308 ranking extension).\nsource: autoeval done right: sample-efficient human evaluation via prediction-powered inference | https://arxiv.org/abs/2403.07008 | 2024-03 (v3 camera-ready 2026-06-01) | academic\nimplication: the autoqa meta-evaluation layer should treat human gold labels as bias-correctors for judge-derived population metrics (pass rates, per-attempter/per-criterion error rates), never report raw judge numbers. because validity survives a bad judge, pooled/hierarchical cross-project validation is statistically safe even where per-project gold sets are tiny - directly resolving taxonomy q1's gold-set-arithmetic fork toward pooled certification with per-project bias correction.\n\n### cascaded selective evaluation (trust or escalate, iclr 2025 oral) delivers a provable, user-specified human-agreement guarantee for llm judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a chatbot arena subset where gpt-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models. [load-bearing]\nevidence: arxiv:2407.18370 abstract fetched directly (80% agreement / ~80% coverage claim confirmed); iclr 2025 oral status confirmed at iclr.cc/virtual/2025/oral/31838; fixed-sequence-testing calibration mechanism confirmed via openreview pdf snippets and secondary reviews (83 citations as of 2026-07). its 'simulated annotators' method (in-context simulation of diverse annotators) is the confidence estimator that makes the guarantee achievable at high coverage.\nsource: trust or escalate: llm judges with provable guarantees for human agreement (iclr 2025 oral) | https://arxiv.org/abs/2407.18370 | 2024-07-25 (iclr 2025-04) | academic\nimplication: this is the direct architectural blueprint for 'which items get the single human touch': the human queue is exactly the abstention set of a confidence-calibrated cascade, and the abstention rate is a tunable dial trading human budget against a guaranteed agreement level. caveat for taxonomy q1: the guarantee is agreement-with-human-majority, so this component's validity is capped at human panel reliability - it certifies consistency-of-application, not truth.\n\n### the minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n*(1-rho^2) where n* is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with r^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes. [load-bearing]\nevidence: arxiv:2605.16354v1 (jane paik kim, stanford psychiatry, 2026-05-08) html fetched: derives sample-size formulas from the doubly-robust estimator's asymptotic variance; worked examples confirmed (n*=200, r^2=0.7, n=2000 -> 65 human labels; n=400 -> 100). explicit caveats: pilot overestimation of r^2 breaks precision guarantees (use conservative values), framework assumes human ratings are a reliable gold standard, single-rater scope.\nsource: augmenting human evaluation with llm judges: how many human reviews do you need? | https://arxiv.org/html/2605.16354v1 | 2026-05-08 | academic\nimplication: answers taxonomy q1's gold-set question with arithmetic instead of convention: run a small pilot to estimate per-axis judge-human correlation, then size each project's gold set from the formula with a conservative correlation estimate. because the floor depends on rho, per-axis correlation heterogeneity argues for stratified gold sampling by criterion, and low-rho axes are where gold budget must concentrate.\n\n### using the llm's own confidence signals to choose which items receive human annotation (confidence-driven inference, building on active statistical inference) cut required human annotations by >25% in all three tested settings while keeping provably valid confidence intervals that remain safe even if the llm annotations are poor - but a 2026 iclr paper finds the opposite in the sequential regime, where near-uniform sampling at the budget ceiling beat uncertainty-driven querying. [load-bearing]\nevidence: arxiv:2408.15204 (gligoric, zrnic, lee, candes, jurafsky; naacl 2025) abstract fetched: '>25% reduction in each of three settings' (politeness, stance, bias) with validity guarantees regardless of llm quality. foundation: active statistical inference (zrnic & candes, icml 2024, arxiv:2403.03208): label where the model is uncertain, provably valid cis, fewer labels than uniform. counter-evidence verified separately (see disagreements): arxiv:2604.18569 (iclr 2026).\nsource: can unconfident llm annotations be used for confident conclusions? (naacl 2025) + active statistical inference (icml 2024) | https://arxiv.org/abs/2408.15204 | 2024-08-27 (v2 2025-02-08; naacl 2025-04) | academic\nimplication: uncertainty-routed allocation of the scarce human touch is the statistically defensible default and comes with a safety net (validity even when the judge is wrong), but the gain over random sampling is regime-dependent and contested - so the foundation should make the routing policy (uncertainty-weighted vs uniform vs stratified) a measured, per-project choice with a built-in a/b against uniform, not a hard-wired assumption.\n\n### glide (may 2026) is an open-source industrial library unifying ppi++, stratified ppi, predict-then-debias bootstrap, and active inference with a decision tree for method selection, and it quantifies when ppi helps: effective-sample gain is ~1.0x at judge-human correlation rho=0.1, ~2.2x at rho=0.9; clt-based estimators need roughly >=50 human labels per stratum (below that use bootstrap ptd); in a real agent-safety case (rho=0.59, ~13-point judge bias) 100 human labels became worth 143-157.\nevidence: arxiv:2605.31278v2 html fetched (martinon, merad, raki; emerton data): monte carlo with 500 human + 1000 proxy labels shows effective n ~500 at rho=0.1 vs ~1100 at rho=0.9; r-judge case study with claude sonnet as judge; library at github.com/emertondata/glide with reproducible validation notebooks; limitations: means/proportions only, i.i.d. assumptions, no anytime-valid inference.\nsource: industrializing prediction-powered inference: the glide library for reliable genai and agentic systems evaluation | https://arxiv.org/html/2605.31278v2 | 2026-05 (v2 2026-06) | practitioner\nimplication: ready-made tooling exists for the statistical layer; more importantly it publishes the correlation-vs-benefit curve and the >=50-labels-per-stratum threshold the foundation can budget against: if a project's judge-human correlation on an axis is below ~0.3, ppi buys nothing there and gold labels must carry that axis alone.\n\n### for estimating scores from noisy judges, ppi++/efficient-influence-function estimators produce near-identical, shortest confidence intervals - reported 3-15x narrower than rogan-gladen-style misclassification-correction estimators depending on the human-labeling ratio, with the advantage largest when the judge is close to random guessing.\nevidence: arxiv:2601.05420 (chen, lu, li, guo, li; 2026-01-08) abstract fetched: unifies measurement-error correction and ppi classes via semiparametric efficiency theory, derives eif-based estimators, characterizes when ppi-style strictly dominates; the 3-15x interval-width figure comes from the paper's empirical section as surfaced in search-indexed full text (not independently recomputed). code: github.com/yiqunchen/debias-llm-as-a-judge.\nsource: efficient inference for noisy llm-as-a-judge evaluation | https://arxiv.org/abs/2601.05420 | 2026-01-08 | academic\nimplication: standardize the estimation layer on ppi++/eif-style residual calibration rather than sensitivity/specificity (confusion-matrix) correction when the target is a rate or mean - but note the certification literature pushes the other way for hypothesis tests (see disagreements).\n\n### for certifying that a failure rate is below a threshold (the low-base-rate pass/fail regime), the 'noisy but valid' framework (iclr 2026) uses a small human calibration set to estimate judge tpr/fpr, applies a variance-corrected test to the large judge-labeled stream with finite-sample type-i error control, and derives exact conditions under which judge-based testing has higher statistical power than direct human evaluation.\nevidence: arxiv:2601.20913 (feng, shen, balashankar, gerner-beuerle, rodrigues; submitted 2026-01-28, accepted iclr 2026 per arxiv page) abstract fetched: variance-corrected critical threshold, finite-sample type-i control under calibration uncertainty, validation on jigsaw/hate speech/saferlhf, and a quantified 'oracle gap' - the power cost of having to estimate judge reliability rather than knowing it.\nsource: noisy but valid: robust statistical evaluation of llms with imperfect judges | https://arxiv.org/abs/2601.20913 | 2026-01-28 | academic\nimplication: for attempter-level or batch-level certification ('this attempter's error rate is below x'), a judge-plus-small-calibration design can be provably more powerful than human-only review - reframing the human panel's job as estimating judge error rates, not per-item adjudication. this is direct evidence for the zero-per-item-touch + calibration-audit branch of taxonomy q6's fork.\n\n### naive reporting of judge-scored evaluations is systematically biased by judge sensitivity/specificity; a plug-in correction with confidence intervals propagating both test-set and calibration-set uncertainty fixes it, remains unbiased under distribution shift between calibration and test data, includes adaptive allocation of calibration labels, and characterizes regimes where corrected judge-based evaluation is more reliable than human-only evaluation.\nevidence: arxiv:2511.21140 (lee, zeng, jeong, sohn, kangwook lee; submitted 2025-11-26, v4 2026-05-31, accepted icml 2026 per arxiv page) abstract fetched. the calibration-to-test distribution-shift robustness is the distinctive property versus standard ppi's i.i.d. assumption.\nsource: how to correctly report llm-as-a-judge evaluations | https://arxiv.org/abs/2511.21140 | 2025-11-26 (v4 2026-05-31) | academic\nimplication: supports the taxonomy q1 proposal to ban raw judge-agreement numbers from all reporting: every reported metric should be the corrected estimator with a ci that includes calibration uncertainty. the shift-robustness result matters specifically because attempter adaptation (taxonomy q8) shifts the live item distribution away from the calibration set over time.\n\n### conformal prediction now provides per-item uncertainty for llm-as-judge ratings: emnlp 2025 work builds guaranteed-coverage score intervals from a single judging run (with an ordinal boundary adjustment and a lower-bias midpoint score), and april 2026 work shows conformal set width is a genuine per-instance reliability signal (r_s=+0.576 with reliability, n=1,918; widths correlate ~0.32-0.38 across different judges) with reliability driven more by the criterion than the judge model (relevance avg set size ~3.0 vs fluency/consistency ~4.9 on a 1-5 scale).\nevidence: sheng, liu, he, zhao, kang, 'analyzing uncertainty of llm-as-a-judge: interval evaluations with conformal prediction', emnlp 2025 main (aclanthology.org/2025.emnlp-main.569), abstract confirmed via neurips 2025 workshop listing. gupta & kumar, arxiv:2604.15302 (2026-04-16, under review): split conformal on summeval; also finds 33-67% of documents contain intransitive judge preference cycles despite low aggregate violation rates.\nsource: analyzing uncertainty of llm-as-a-judge (emnlp 2025) + diagnosing llm judge reliability (arxiv:2604.15302) | https://arxiv.org/abs/2604.15302 | 2025-11 (emnlp) / 2026-04-16 | academic\nimplication: conformal set width is a cheap, distribution-free routing statistic for the human-touch queue that transfers across judges, and the criterion-dominates-judge finding is empirical support for taxonomy q2/q5's axis-triage: per-criterion reliability ceilings must be measured and some axes routed human-only regardless of judge choice.\n\n### judge-vs-system drift can be attributed with anytime-valid statistics: a frozen human-labeled anchor set periodically re-scored by the live judge, monitored with betting e-processes, correctly attributed a silent judge version bump in 60/60 runs with zero misattribution (rolling z-test baseline false-alarmed on 75% of drift-free streams), at 0.21-0.64x the cost of strong-judging every item.\nevidence: arxiv:2606.15474 (yitao li, 2026-06-13) abstract fetched: 'one-way identification (only the judge can move the anchors)'; three-verdict output (none/system/judge); strict-prompt change attributed in 110/120 runs; replicated on tl;dr summarization 240/240. single-author preprint, not yet peer-reviewed - treat quantitative claims as provisional.\nsource: who drifted: the system or the judge? anytime-valid attribution in llm evaluation pipelines | https://arxiv.org/abs/2606.15474 | 2026-06-13 | academic\nimplication: taxonomy q9's three-moving-parts problem has an existing statistical design for one axis: freeze a per-project human-labeled anchor set at onboarding, interleave it continuously into the judge stream, and use e-process alarms to separate 'judge changed' from 'attempter population changed' before triggering rubric revision or de-graduation.\n\n### in sequential prediction-powered mean estimation, uncertainty-based active querying contributed little: the smallest confidence widths occurred when query probabilities were near-constant at the budget ceiling, and theory shows optimized query probabilities converge to the maximum allowed constant rate when chosen without reference to covariates.\nevidence: arxiv:2604.18569 (sfyraki & wang, submitted 2026-04-20, accepted iclr 2026 per arxiv page) abstract fetched: empirical finding that the uncertainty-blend weight near the constant term minimizes ci width, plus non-asymptotic analysis and no-regret query-probability selection corroborating near-uniform optimality in this regime.\nsource: revisiting active sequential prediction-powered mean estimation | https://arxiv.org/abs/2604.18569 | 2026-04-20 | contrarian\nimplication: direct counter-evidence to uncertainty-routed human sampling for population estimation in streaming settings: the foundation must not assume uncertainty routing dominates - measure the routing policy's marginal value per project, and expect uniform-at-budget to be near-optimal when the goal is estimation rather than per-item verdict quality.\n\n### commercial tooling for confidence-scored llm judgments with human calibration is mature: cleanlab's trustworthy language model attaches a real-time trustworthiness score to every llm response, supports custom evaluation criteria, explicitly documents calibrating trust scores against human quality ratings, and publishes benchmarks (nov 2025) for auto-flagging incorrect structured outputs for human review.\nevidence: cleanlab tlm documentation (help.cleanlab.ai/tlm/: quickstart, custom-eval calibration-against-human-ratings tutorial, advanced usage with low-score explanations) and 2025-11-18 structured-outputs benchmark blog post, all fetched via search with page content confirmed. vendor source: benchmark numbers are self-reported.\nsource: cleanlab trustworthy language model (tlm) documentation and benchmark | https://help.cleanlab.ai/tlm/ | 2025-11-18 (benchmark); docs current 2026 | practitioner\nimplication: the 'score every judgment, calibrate the score against a human-rated set, route low-trust items to humans' pattern is already productized - the foundation can specify this as a component contract (per-verdict trust score + human-calibration hook) rather than inventing it, while keeping vendor-neutrality.\n\n### tools & artifacts\n- ppi_py: reference open-source python library for prediction-powered inference (angelopoulos, zrnic et al.): means, quantiles, glms, ppi++ power tuning. | https://github.com/aangelopoulos/ppi_py | drop-in statistical layer for computing bias-corrected pass rates and attempter metrics from many judge labels + few human labels; covers estimand types glide does not (quantiles, regressions).\n- glide: open-source industrial ppi library (emerton data, may 2026): ppi++, stratified ppi++, predict-then-debias bootstrap, active statistical inference, uniform/stratified/active/cost-optimal samplers, plus a method-selection decision tree. | https://github.com/emertondata/glide | closest existing implementation of this domain's whole playbook, including the sampler that decides which items get human labels and effective-sample-size roi reporting; validated on agent-safety trajectories with a claude judge.\n- active-inference: code for active statistical inference (zrnic & candes, icml 2024): uncertainty-guided label collection with provably valid confidence intervals. | https://github.com/tijana-zrnic/active-inference | reference implementation for uncertainty-routed allocation of the scarce human annotation budget.\n- debias-llm-as-a-judge: code for chen et al. 2026 (arxiv:2601.05420): efficient-influence-function and ppi++ estimators for noisy llm-judge evaluation of scores and win rates. | https://github.com/yiqunchen/debias-llm-as-a-judge | implements the estimator class this domain recommends standardizing on for judge-score estimation, with the rogan-gladen comparison built in.\n- cleanlab trustworthy language model (tlm): commercial system that attaches a calibratable trustworthiness score to every llm response, with custom evaluation criteria and tutorials for calibrating scores against human ratings (benchmark published 2025-11). | https://help.cleanlab.ai/tlm/ | productized version of the per-verdict-confidence + human-calibration + route-low-trust-to-human pattern; useful as a component contract template even if not adopted as a vendor.\n- cascaded selective evaluation (trust or escalate): iclr 2025 oral method: judge cascade with fixed-sequence-testing calibration delivering user-specified human-agreement guarantees; simulated annotators confidence estimation. | https://arxiv.org/abs/2407.18370 | the architectural template for the autoqa's escalation spine: guarantee level and abstention rate become the two governing dials for the human-touch budget.\n\n### disagreements\n- uncertainty-routed vs uniform human sampling: zrnic & candes (icml 2024) and gligoric et al. (naacl 2025) report provably valid inference with >25% fewer human labels when the model's uncertainty guides which items humans label; sfyraki & wang (iclr 2026, arxiv:2604.18569) find in the sequential regime that the uncertainty component contributes little and near-constant query probabilities at the budget ceiling give the tightest intervals. both sides are peer-reviewed; the honest read is that active-sampling gains are regime-dependent (batch design + informative confidence signals vs sequential streaming), so routing policy must be validated per project, not assumed.\n- explicit judge-error modeling vs black-box residual calibration: chen et al. (arxiv:2601.05420) show rogan-gladen-style tpr/fpr-correction estimators yield intervals reportedly 3-15x wider than ppi++/eif for score estimation; feng et al. (iclr 2026, arxiv:2601.20913) argue the opposite direction for certification - explicitly modeling judge tpr/fpr yields better-powered hypothesis tests than treating ppi as a black box. partly reconcilable (different targets: estimation vs testing), but they issue opposing default recommendations and the synthesis should pick per use-case: ppi++/eif for dashboards and rates, tpr/fpr-corrected tests for threshold certification.\n- what the human labels are for - target vs corrector: trust or escalate guarantees agreement with human majority preference (capping the system's validity at human panel reliability, i.e., consistency-of-application); the ppi/certification line treats human labels as gold-standard truth for bias correction, and both 'noisy but valid' and 'how to correctly report' prove regimes where corrected judge-based evaluation is strictly more reliable/powerful than human-only evaluation, while kim (arxiv:2605.16354) explicitly warns her framework breaks when human raters are themselves unreliable. no paper in this literature resolves what to do when the gold itself is noisy - the taxonomy q1 construct choice (instruction-satisfaction vs consensus-prediction) is genuinely open in the statistics literature.\n\n### gaps\n- no 2025-2026 paper found that applies ppi, selective escalation, or conformal routing specifically to qa of human-produced annotations (reviewing human attempters); every application evaluates model outputs with humans as gold. transfer is plausible but unvalidated, especially under adversarial attempter adaptation, which violates the i.i.d. assumptions most ppi variants rely on (only arxiv:2511.21140 claims calibration-to-test shift robustness).\n- all ppi-family guarantees are population-level (rates, means, rankings) and selective-evaluation guarantees are marginal, not conditional: a system can meet its aggregate agreement/coverage target while concentrating its failures on specific attempters, criteria, or hard-item subgroups. conditionally valid (per-subgroup) escalation guarantees for llm judges remain an open problem in this literature.\n- the specific fork 'one shallow human touch per item vs zero-touch plus randomized deep audits' has no direct head-to-head study measuring total error caught per human-hour; the statistical literature implicitly favors the zero-touch+audit side (valid conclusions with human labels on a small fraction of items) but only for measurement/certification objectives, not for per-item verdict correction or attempter coaching.\n- calibration economics for rubric-based qa: trust or escalate's simulated annotators and fixed-sequence testing were validated on pairwise preference tasks; required calibration-set sizes and achievable coverage for per-criterion rubric verdicts (the autoqa's actual workload) are not established anywhere i found.\n- could not open the trust or escalate openreview forum (captcha); handling of abstained items and calibration-set-size details confirmed only from abstract-level and secondary sources, not reviewer discussion. the 3-15x interval-width figure in chen et al. was corroborated from search-indexed paper text, not independently recomputed.\n- no published head-to-head comparison of conformal-width routing vs simulated-annotators-style confidence routing vs verbalized-confidence routing for deciding the human-escalation queue - the three candidate mechanisms have never been benchmarked against each other.\n\n### verifications\n- claim: prediction-powered inference (ppi/ppi++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a gpt-4 judge in 'autoeval done right'.\n verdict: partially_confirmed | verified directly against arxiv:2403.07008: authors (boyeau, angelopoulos, yosef, malik, jordan) correct; abstract verbatim contains 'increase the effective human-labeled sample size by up to 50% on experiments with gpt-4'; abstract claims methods 'improve sample efficiency while remaining unbiased'; version history confirms v1 2024-03-09 and v3 2026-06-01 with comment 'camera-ready paper version' - dates as claimed. glide (arxiv:2605.31278, martinon/merad/raki, v1 2026-05-29, icml 2026 workshop) exists and states ppi 'combines both into debiased estimates with valid confidence intervals', consistent with the claim, though the exact phrase 'regardless of judge quality' was not verifiable in its abstract. the 2026 ppi-for-llm-judge literature is real and active (precise/amazon science, arxiv:2606.05308 ranking extension, arxiv:2601.05420, arxiv:2601.20913), so the claim is not superseded - it is being extended. two corrections: (1) 'always-valid confidence intervals' is a misuse of a term of art - 'always-valid'/anytime-valid refers to sequential inference; ppi/ppi++ intervals are fixed-n asymptotic (clt-based) intervals, so coverage is guaranteed only asymptotically and finite-sample coverage can degrade with very small gold sets. (2) the judge-quality-independence property is a general ppi/ppi++ property (angelopoulos et al. 2023) rather than something stated in the autoeval abstract itself; the claim's framing correctly captures its substance (worse judge -> wider intervals via lower correlation, coverage preserved) but slightly overstates its strength and its provenance in this specific source.\n corrected: prediction-powered inference (ppi/ppi++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with asymptotically valid confidence intervals whose coverage does not depend on judge accuracy (a worse judge widens intervals rather than breaking coverage, subject to standard asymptotic/regularity conditions and a sufficiently large gold set); 'autoeval done right' (arxiv:2403.07008, boyeau, angelopoulos, yosef, malik & jordan) reports increasing the effective human-labeled sample size by up to 50% in experiments with gpt-4.\n- claim: cascaded selective evaluation (trust or escalate, iclr 2025 oral) delivers a provable, user-specified human-agreement guarantee for llm judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a chatbot arena subset where gpt-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models.\n verdict: confirmed | all load-bearing elements check out against the primary sources. (1) arxiv:2407.18370 abstract (fetched directly) states cascaded selective evaluation provides a provable human-agreement guarantee at a user-specified level, and that on a chatbot arena subset \"where gpt-4 almost never achieves 80% human agreement,\" the method \"guarantees over 80% human agreement with almost 80% test coverage\" even using mistral-7b. (2) paper body (arxiv.org/html/2407.18370v1) confirms the mechanism: threshold calibration via fixed sequence testing (bauer, 1991) on a small human calibration set (|d_cal|=500 for chatarena/tl;dr, 392 for auto-j), cascade escalation from cheap to strong judges on low confidence, abstention (output empty set) when even the strongest judge is unconfident, and simulated annotators as the confidence estimator enabling high coverage. (3) the ~88% figure is exact: \"79.1% of all samples, among which 88.1% are evaluated by substantially cheaper mistral-7b or gpt-3.5 instead of gpt-4\" (table 3: gpt-4 handles only 17.5%). (4) iclr 2025 oral confirmed at iclr.cc/virtual/2025/oral/31838 (jung, brahman, choi); dates correct (v1 2024-07-25; iclr 2025 in april 2025). two trivial nuances, neither rising to a correction: the paper defines abstention as returning empty set / excluding from coverage - deferral \"to humans\" is the intended framing but abstained items are not actually routed to human annotators in the experiments; and the guarantee holds with probability 1-delta over calibration-set sampling (standard for such guarantees, and consistent with \"provable, user-specified\"). supersession check: not superseded - but a newer alternative exists: scope (arxiv:2602.13110, icml 2026 poster, feb 2026) uses conformal calibration with a bidirectional preference entropy signal and reports better calibration/coverage than simulated annotators; it complements rather than invalidates the claim about this paper. the \"83 citations\" figure was not independently verified but is plausible and not load-bearing.\n- claim: the minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n*(1-rho^2) where n* is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with r^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes.\n verdict: confirmed | all elements verified against the source (https://arxiv.org/html/2605.16354v1, jane paik kim, stanford dept. of psychiatry and behavioral sciences, submitted 2026-05-08 - date and attribution correct). (1) floor formula: paper states required human reviews converge to n*(1-rho^2) as n increases - exact match. (2) worked example: n*=200, r^2=0.70, n=2000 -> 65 human labels, confirmed verbatim; the n=400/100 figure is stated in the paper as \"with a budget of 100 human ratings, n can be reduced to 400 while achieving target power\" - same numbers, slightly different framing (human budget fixed, n solved) but mathematically equivalent to the claim. (3) stratification savings: paper reports \"up to 12.9%\" reduction when r^2 gap is large (0.8 vs 0.1) - matches \"~13%\", with the paper adding that moderate gaps yield only ~2%. (4) diminishing returns from more judge labels: explicitly stated (\"each successive increase in n produces diminishing reductions in n\"). (5) all three caveats in the evidence summary (pilot r^2 overestimation breaking precision guarantees, human-as-gold-standard assumption, single-rater scope) appear in the paper's limitations. supersession check: two searches found no v2 revision, no citing papers, and no later work replacing or contradicting the result as of 2026-07-14. minor precision notes only: \"up to ~13%\" should strictly be \"up to 12.9% in the extreme r^2-gap case, ~2% for moderate gaps,\" and the floor is asymptotic (approached as n grows), not attainable at finite n - the claim's own phrasing (\"converge to a floor,\" \"up to\") already reflects both.\n\n---\n\n## domain: rubrics-recent\n\n*rubric practice 2025-26, compilation, rubric qa*\n\n### openai healthbench meta-evaluates its llm grader per-criterion against physician majority grades using macro f1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: gpt-4.1 grader mf1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician mf1 is only 0.569-0.730 with wide individual spread. [load-bearing]\nevidence: full paper text (section 8.1, tables 5-6) confirmed: 60,896 meta-examples over 34 physician-consensus criteria; 'typical physician' baseline computed by scoring each physician against the others exactly as the model is scored; grader model choice matters (gpt-4.1 0.709 > o4-mini 0.692 > o3 0.681 > gpt-4.1-nano 0.580); prompt and individual criterion phrasings were tuned 'so that their intent was unmistakable to the grader'. overall benchmark std across 16 full runs is ~0.002.\nsource: healthbench: evaluating large language models towards improved human health (openai) | https://arxiv.org/html/2505.08775v1 | 2025-05-13 | primary\nimplication: the 2025-26 standard validation design for a rubric judge: per-criterion binary meta-eval against adjudicated expert-consensus labels with a chance-robust class-balanced statistic, reporting the judge as a percentile of individual human raters - not raw accuracy vs. a single reviewer. the human agreement ceiling is empirically low (mf1 ~0.6-0.7 even among physicians on physician-written criteria), so 'exceeds median human' is an attainable and meaningful bar, and consensus-adjudicated labels (not single-reviewer labels) must be the target. also: rubric criterion text is a tunable judge-alignment artifact, not fixed scripture.\n\n### healthbench's production rubric format is per-item (conversation-specific) criteria written by 262 physicians (48,562 criteria), each an independently-judged binary met/not-met check with a nonzero weight from -10 to +10 (negative criteria encode pitfalls), aggregated as a weighted sum normalized by max attainable positive points.\nevidence: confirmed in fetched full text: 'each rubric criterion has an associated nonzero point value between -10 and 10, with negative points used for criteria that are undesirable'; grader judges each criterion independently; item score can go negative; the vast majority of criteria were written specifically for one example.\nsource: healthbench (rubric structure, section 3) | https://arxiv.org/html/2505.08775v1 | 2025-05-13 | primary\nimplication: reference design for the autoqa verdict aggregation question: per-item compiled criteria (not one generic project rubric), explicit negative-weighted pitfall criteria as first-class citizens, independent binary judging per criterion, and a transparent weighted-sum aggregation - i.e., claim-level findings with an arithmetic, auditable roll-up rather than a holistic judge verdict.\n\n### a large data vendor's rubrics-as-rewards found that llm-synthesized rubrics grounded in expert reference answers performed on par with human-authored rubrics as reward signals (0.359 vs 0.348), that prompt-specific rubrics substantially beat generic ones, and that the one component synthetic generation could not produce well was pitfall criteria - anticipating failure modes 'often demands human intuition and domain expertise'. [load-bearing]\nevidence: full paper (arxiv 2507.17746) confirmed: four design principles (grounded in expert guidance; comprehensive coverage; semantic weighting essential 1.0 / important 0.7 / optional 0.3 / pitfall 0.8 penalty; self-contained per-criterion evaluability); rubrics generated by o3-mini/gpt-4o from reference answers, ~7-20 criteria per prompt; implicit aggregation (judge sees all criteria+weights and emits one score) consistently beat explicit per-criterion binary + weighted-sum, up to 28% relative on healthbench-1k; rubrics helped small judges most and reduced cross-judge variance.\nsource: rubrics as rewards: reinforcement learning beyond verifiable domains (a large data vendor) | https://arxiv.org/html/2507.17746v1 | 2025-07-23 | primary\nimplication: rubric compilation can be automated if anchored to expert reference material - the scarce human input to budget for is reference exemplars and failure-mode/pitfall knowledge, not criterion drafting labor. caveat for autoqa: rar's implicit-aggregation win optimizes reward quality, not auditability; an autoqa that must justify verdicts may still prefer explicit per-criterion verdicts (healthbench/paperbench style) and treat rar as evidence that fixed hand-crafted weights are not sacred.\n\n### rubricbench (mar 2026) measures a stable ~26-27 point preference-accuracy gap between self-generated and human-annotated rubrics across all judge backbones (e.g., deepseek-v3.2 57.8% vs 84.9%; gemini-3-flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement) does not close it - 'rubric formation', not judge reasoning, is the bottleneck. [load-bearing]\nevidence: full html (arxiv 2603.01562) confirmed: 1,147 adversarial pairwise comparisons where surface cues mislead (rejected responses longer/better-formatted/more confident); expert rubrics are 2-10 atomic binary checks derived only from the instruction; llm-generated rubrics over-produce unnecessary (17.9% vs 10.1%) and overly rigid (12.8% vs 7.7%) rules; checklists of 13+ items add noise ('attention displacement'); even with human rubrics accuracy plateaus ~85% because judges treat must-have constraints as soft signals; safety domain worst for self-generated rubrics (~25-30% vs >90% human).\nsource: rubricbench: aligning model-generated rubrics with human standards | https://arxiv.org/html/2603.01562v1 | 2026-03-02 | academic\nimplication: directly bounds the per-project customization contract: instruction-only automated rubric compilation (no expert exemplars/references) leaves a large fidelity gap that more compute cannot buy back - an expert-grounding step (reference answers, adjudicated exemplars, or human rubric sign-off) is structurally required, not optional. also: cap compiled checklist length (~<13 active criteria per judge call), and give hard constraints enforcement semantics (gating) rather than listing them as soft criteria.\n\n### evalgen ('who validates the validators', uist 2024) established criteria drift: evaluation criteria cannot be fully specified a priori because grading outputs itself changes the criteria - 'users need criteria to grade outputs, but grading outputs helps users define criteria' - and some criteria are dependent on the specific outputs observed. [load-bearing]\nevidence: fetched abstract/paper confirmed the catch-22 formulation, the finding that criteria appear dependent on observed outputs rather than definable in advance, and the conclusion that judge-alignment is inherently iterative and subjective; evalgen operationalizes this by generating candidate assertions/judge prompts and selecting those best aligned with accumulated human grades.\nsource: who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences | https://arxiv.org/abs/2404.12272 | 2024-10-13 | academic\nimplication: the rubric compilation pipeline must be a loop, not a compile step: initial rubric -> human grades a sample -> criteria are revised -> re-validate. foundational consequence for the autoqa: budget a per-project calibration phase where criteria are expected to change, version every rubric, and treat early judge-human disagreement partly as rubric-specification signal (rewrite) rather than judge noise (vote harder) - the two-builds fork in design question 5 has an evidence-backed default.\n\n### rift (snorkel ai, apr 2026) provides the first rubric failure-mode taxonomy - 8 modes in 3 categories (reliability: subjective, non-atomic, ungrounded; content validity: misaligned/rigid, missing criteria; consequential validity: hackable, low signal, redundant) - and shows failure-mode density correlates with judge-human misalignment (r=0.162, p=0.0021); synthetic rubrics skew subjective (86.7% vs 52.6% for human-crafted) while human rubrics skew misaligned/rigid (63.2% vs 20.0%); automated llm detection reaches f1 0.925 for subjective but ~0.000 for hackable.\nevidence: full html (arxiv 2604.01375v2) confirmed all numbers: 85 rubrics / 255 expert annotations across 5 sources (human: advancedif, researchrubrics; synthetic: wildchecklists, openrubrics, autorubrics); taxonomy built via grounded theory, mean cohen's kappa 0.64; diagnostics include llm-as-judge classifiers (gpt-5.2, gemini 3 pro), inter-rater reliability across labeler models, alignment to a strong reference, and reward-variance probes.\nsource: rift: a rubric failure mode taxonomy and automated diagnostics (snorkel ai) | https://arxiv.org/html/2604.01375v2 | 2026-04-20 | academic\nimplication: rubric qa/linting is now a named, tooled 2026 practice: run compiled rubrics through a rift-style lint before go-live. two structural lessons: (1) human and llm rubric authors fail in complementary ways, motivating co-authoring rather than either alone; (2) hackability/gameability is the one failure mode automated diagnostics cannot detect - the anti-gaming review of the autoqa's compiled rubrics must be a human red-team task, permanently.\n\n### support vector rubrics (jun 2026) identifies the mechanism behind the llm-vs-human rubric gap - 'self-generated rubrics describe good responses, whereas effective criteria must discriminate between close candidates' - and closes the rubricbench gap from 24.1 to 0.3 points by mining contrastive rubrics from preference pairs (support-pair selection plus adversarial probing of hard negatives).\nevidence: fetched abstract (arxiv 2606.08077, sun et al.) confirmed the objective-mismatch framing, the max-margin method (rubric bank mined from preference data, prompt-conditioned selector, iterative refinement on hard negatives), transfer of the rubric bank across judges without retraining, and competitiveness with dedicated reward models on rewardbench 1/2 and rm-bench.\nsource: support vector rubrics: closing the gap between self-generated and human rubrics | https://arxiv.org/abs/2606.08077 | 2026-06-06 | academic\nimplication: the highest-value compilation artifacts are contrastive: borderline pass/fail adjudication pairs, not descriptions of quality. for the autoqa's exemplar question (5 vs 50 vs 500), this says invest in hard-negative/near-miss pairs per criterion and derive discriminative criteria from them - descriptive 'what good looks like' statements alone are the documented failure pattern.\n\n### paperbench (openai, apr 2025) demonstrates the extreme end of decomposition: 8,316 individually-graded binary leaf criteria across 20 papers, sibling-relative manual weights, rubrics co-developed with each paper's original authors over multiple weeks per rubric; an o3-mini judge then grades leaves at f1 = 0.83 vs expert human judgments for ~$66/paper versus ~12 hours of expert grading time.\nevidence: full paper (arxiv 2504.01848) confirmed: each leaf is pass/fail; parent scores are weighted averages propagating to a root score; weights reflect importance relative to siblings, not difficulty; judgeeval is a separate benchmark for judges (o1 scored 0.84 but ~$830/paper - o3-mini chosen as cost-effective); human expert grading estimated at 'tens of hours' per paper.\nsource: paperbench: evaluating ai's ability to replicate ai research (openai) | https://arxiv.org/html/2504.01848v2 | 2025-04-02 | primary\nimplication: atomic binary decomposition of genuinely expert judgments is feasible and auto-gradable at usable-but-imperfect agreement (f1 0.83), with ~100x cost reduction over expert grading - but rubric authoring cost is weeks of joint expert time per complex artifact. for autoqa: fine decomposition pays at grading time and the judge gets its own meta-benchmark (judgeeval pattern), but the compilation budget, not the judging budget, is the binding constraint per project.\n\n### openai's official grader documentation prescribes that every model grader be validated on its own eval built from expert-graded answers (grader score ordering must reproduce expert ranking), and names the canonical reward-hacking detection signature: the graded model scores well on the model-grader eval while doing poorly on expert human evaluations.\nevidence: fetched live doc (developers.openai.com/api/docs/guides/graders, accessed 2026-07-14): grader taxonomy = string_check, text_similarity, score_model (llm judge with numeric range), python (sandboxed), multigrader (formula-combined sub-graders, rft only); guidance: iterate grader prompts, use few-shot examples of good/fair/poor answers, prefer smooth over binary scores, add edge cases to the grader eval over time, 'guard against reward hacking'. page carries a deprecation note for graders within the evals/fine-tuning workflows they support.\nsource: openai api documentation: graders | https://developers.openai.com/api/docs/guides/graders | 2026-07-14 | primary\nimplication: lab-published operational doctrine matching the autoqa's needs: (1) mixed grader stacks - deterministic checks composed with llm judges via explicit formulas - are the production pattern for the objective/subjective split; (2) a standing expert-audit channel is the designed-in hack detector (grader-vs-expert divergence), i.e., the independent expert-audit sampling channel is a permanent component in the reference architecture, not a bootstrap phase.\n\n### a cluster of 2026 papers (rubric-arm feb 2026; evorubrics, arbor, amaris jun 2026) converges on the position that static rubrics are an exploitable reward specification under optimization pressure and must be adapted/co-evolved during training, with rubric-arm jointly optimizing a rubric generator and judge via alternating rl from preference feedback.\nevidence: rubric-arm abstract opened and confirmed (prior approaches 'rely on static rubrics or disjoint training pipelines, which limits their adaptability'; alternating optimization with variance-reduction analysis; sota on judge benchmarks and downstream policy alignment). evorubrics/arbor/amaris confirmed at title/abstract level via search only - trend triangulated across 5+ 2026 arxiv titles, not deep-read.\nsource: rubric-arm: alternating reinforcement learning for rubric-based reward modeling (plus 2026 co-evolution cluster) | https://arxiv.org/abs/2602.01511 | 2026-02-02 | academic\nimplication: attempters paid per accepted item exert the same optimization pressure as an rl policy: the 2026 consensus is that a frozen compiled rubric will be goodharted, so rubric revision must be a designed, continuous loop (with regression gates), not an exceptional maintenance event. this directly supports treating criteria drift response and anti-gaming rubric rotation as foundational architecture.\n\n### autorubric (feb 2026, rao & callison-burch) consolidates scattered rubric-evaluation techniques into one open-source framework with opinionated defaults - binary/ordinal/nominal criterion types, judge ensembles, few-shot calibration, bias mitigations, and psychometric reliability metrics - reporting 87% binary accuracy with moderate-to-substantial kappa on charm-100, and showing per-criterion explanations double as improvement signals (peer-review agent 0.47 -> 0.85, beating a 0.82 expert-curated baseline).\nevidence: fetched abstract (arxiv 2603.00077, v2 2026-04-03, 52 pages) confirmed the unification claim ('scattered across papers with inconsistent terminology and partial implementations'), the three validation benchmarks (ricechem 80% with 5-shot calibration; researcherbench 931 criteria cross-judge agreement; charm-100), and rl-reward use (+0.039 advancedif, wilcoxon p=0.032, positive ifeval transfer).\nsource: autorubric: unifying rubric-based llm evaluation | https://arxiv.org/abs/2603.00077 | 2026-02-13 | academic\nimplication: by early 2026 rubric-based judging has a consolidated reference implementation whose defaults (ensembles, few-shot calibration on human-graded examples, psychometric reliability reporting per criterion type) can be adopted wholesale rather than reinvented; and the same per-criterion explanation trace that justifies a verdict is demonstrated to work as constructive improvement feedback - verdict machinery and feedback generation can share one artifact.\n\n### langchain's align evals (langsmith, jul 2025) productized judge validation as a standard workflow: humans grade a representative golden set per criterion, an 'alignment score' measures judge-vs-human match, unaligned cases are surfaced for prompt iteration, and each judge-prompt version is compared against a saved baseline alignment score.\nevidence: fetched launch blog confirmed the four-step workflow (select criteria; select representative good+bad examples; human-grade expected scores; iterate evaluator prompt against alignment score), the stated problem ('our evaluation scores don't match what we'd expect a human on our team to say'), inspiration from eugene yan's aligneval, and roadmap items (alignment tracking over time, automatic prompt optimization).\nsource: introducing align evals: streamlining llm application evaluation (langchain) | https://www.langchain.com/blog/introducing-align-evals | 2025-07-29 | practitioner\nimplication: the industry-default minimum-viable gold set practice: small per-criterion human-graded golden sets with versioned alignment baselines, refreshed as unaligned cases surface - evidence that per-criterion (not per-item-holistic) alignment scoring with regression baselines is the practical validation loop the autoqa should ship with from day one.\n\n### tools & artifacts\n- openai/simple-evals healthbench_meta_eval.py: reference implementation of healthbench's grader meta-evaluation (grader vs physician grades, macro f1 per criterion) | https://github.com/openai/simple-evals/blob/main/healthbench_meta_eval.py | directly reusable pattern for validating the autoqa judge against expert-adjudicated labels per criterion - the meta-eval harness the foundation needs, already open-sourced\n- autorubric: open-source framework (rao & callison-burch, feb 2026) unifying rubric-based llm evaluation: binary/ordinal/nominal criteria, judge ensembles, few-shot calibration, bias mitigations, psychometric reliability metrics | https://arxiv.org/abs/2603.00077 | closest existing embodiment of the autoqa's judging core; its opinionated defaults are a checklist of what a 2026-credible rubric judge must include\n- rift taxonomy + diagnostics: snorkel ai's 8-mode rubric failure taxonomy with automated llm lint classifiers and reliability probes (apr 2026) | https://arxiv.org/html/2604.01375v2 | ready-made rubric-qa lint stage for the compilation pipeline; also documents which rubric defects (hackable) automated linting cannot catch\n- rubricbench: 1,147 adversarial pairwise comparisons with expert atomic rubrics for measuring rubric-generation and rubric-execution quality (mar 2026) | https://arxiv.org/html/2603.01562v1 | off-the-shelf stress test for any automated rubric-compilation procedure the autoqa adopts; its surface-bias-controlled pairs also probe judge style bias (design question 5)\n- langsmith align evals: productized judge-alignment workflow: human-graded golden set, per-criterion alignment score, versioned baselines (jul 2025) | https://www.langchain.com/blog/introducing-align-evals | working example of the minimum-viable judge-validation loop and ui pattern for surfacing judge-human disagreements for iteration\n- openai graders api + documentation: composable grader stack (string_check, text_similarity, score_model, python, multigrader with formula aggregation) with published calibration and anti-reward-hacking guidance | https://developers.openai.com/api/docs/guides/graders | lab-published architecture for mixing deterministic checks with llm judges under one aggregation formula - the objective/subjective lane split in api form (note: page carries a deprecation notice)\n- paperbench rubrics + simplejudge (openai/preparedness): 20 author-co-developed hierarchical rubrics (8,316 binary leaves) plus an llm judge and judgeeval, a benchmark for the judge itself | https://arxiv.org/html/2504.01848v2 | the 'benchmark your judge separately' (judgeeval) pattern and the sibling-relative weighting scheme are directly transferable to autoqa verdict aggregation\n\n### disagreements\n- llm-generated vs expert rubrics: scale's rar (jul 2025) reports reference-grounded synthetic rubrics match human-authored ones (0.359 vs 0.348 as rl rewards), while rubricbench (mar 2026) measures a stable ~26-27pt preference-accuracy gap that test-time compute cannot close, and svr (jun 2026) claims the gap closes (24.1 -> 0.3) only by mining discriminative rubrics from preference pairs. the likely reconciling variable is grounding: rar's generator saw expert reference answers, rubricbench's saw only the instruction - but no paper tests all conditions head-to-head, so 'can compilation skip expert grounding' remains unsettled; the safe design assumes it cannot.\n- aggregation rule: rar found implicit aggregation (judge internally weighs all criteria into one score) consistently beats explicit per-criterion binary + weighted-sum for reward quality (up to 28% relative), yet the flagship production benchmarks (healthbench, paperbench) and openai's grader doctrine use explicit independent per-criterion judgments with arithmetic roll-ups for auditability. performance and auditability pull in opposite directions; no source resolves the tradeoff for a qa setting where verdicts must be explained.\n- decomposition granularity: paperbench succeeds with 8,316 atomic binary leaves (judge f1 0.83, each leaf graded separately), while rubricbench finds checklists of 13+ items degrade judgment via 'attention displacement', and rift lists both non-atomic and redundant as failure modes. the apparent resolution - atomicity helps when each criterion gets its own judge call, hurts when one call processes a long checklist - is inferred, not directly tested anywhere.\n- static vs dynamic rubrics: production eval practice (healthbench, paperbench, openai graders) pins rubrics and regression-tests judges against them, treating rubric stability as integrity; the 2026 rl literature (rubric-arm, evorubrics, arbor, amaris) argues static rubrics are inherently exploitable specifications that must co-evolve with the optimizing population. for an autoqa facing adaptive paid attempters, these prescriptions conflict: pin-and-regression-gate vs continuously-adapt.\n- judge-model reasoning strength: healthbench found the non-reasoning gpt-4.1 outperformed reasoning models o3/o4-mini as a rubric grader (mf1 0.709 vs 0.681/0.692), while paperbench selected reasoning model o3-mini (high effort) and found o1 best (f1 0.84). whether rubric grading benefits from reasoning-class judges appears task-dependent and both labs partially attribute results to prompt-tuning conducted with the winning model.\n\n### gaps\n- no published 2025-2026 study applies rubric-based llm autograding to qa of human annotators' work product (annotations/critiques/ratings). all quantitative evidence in this domain grades model outputs or rl rollouts; transfer to human-attempter qa - including how paid humans adapt adversarially versus how policies reward-hack - is an assumption, not a finding.\n- no major training-data vendor (scale, surge, mercor) has published a complete rubric-writing methodology guide. the closest public artifacts are rar's four design principles, openai's grader-doc guidance, healthbench's physician criteria process, and rift's failure taxonomy; appen markets 'rubric design' services but publishes no method detail. vendor-internal rubric-writing guides likely exist but are not public.\n- criteria drift has no 2026 quantitative follow-up: evalgen's finding is qualitative (small-n user study, 2024). no source measures drift rates, drift half-life, or drift-triggered revision thresholds in a production rubric pipeline - the exact parameters design question 9 needs.\n- the 2026 rubric co-evolution cluster (evorubrics, arbor, amaris) was verified only at abstract/title level; their specific numbers were not confirmed by opening full texts. the trend claim rests on rubric-arm (opened) plus five-plus corroborating 2026 titles.\n- openai's graders documentation page carries a deprecation notice for the graders/evals workflows; what replaces them in openai's mid-2026 stack was not tracked down, so the grader taxonomy finding may describe a sunsetting api surface (the design doctrine it encodes remains valid evidence of practice).\n- no source directly measures the minimum number of adjudicated exemplars per criterion before judge-human agreement plateaus (the 5 vs 50 vs 500 question); autorubric's '5-shot calibration -> 80%' on one chemistry dataset is the only datapoint found, and svr implies exemplar quality (hard contrastive pairs) dominates count.\n\n### verifications\n- claim: openai healthbench meta-evaluates its llm grader per-criterion against physician majority grades using macro f1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: gpt-4.1 grader mf1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician mf1 is only 0.569-0.730 with wide individual spread.\n verdict: confirmed | fetched https://arxiv.org/html/2505.08775v1 (marked arxiv:2505.08775v1 [cs.cl] 13 may 2025 - date correct). all claim elements verified against section 8.1 / tables 5-7: (1) meta-evaluation uses macro f1 on binary met/not-met per consensus criterion, explicitly chosen for class imbalance (random baseline mf1=0.50); (2) gpt-4.1 grader mf1 = 0.709, best of graders tested (o4-mini 0.692, o3 0.681, gpt-4.1 mini 0.661, gpt-4.1 nano 0.580); (3) over 60,896 meta-examples across the 34 physician-consensus criteria (avg 1,791/criterion); (4) 'typical physician' baseline computed by scoring each physician against the others the same way as the model, reported as a percentile - gpt-4.1 exceeds the average physician in 5 of 7 themes (below only expertise-tailored communication 0.610 vs 0.618 and health data tasks 0.683 vs 0.730), upper half in 6/7, above 33rd percentile in all; (5) theme-level weighted-average physician-vs-physician mf1 ranges 0.569 (response depth) to 0.730 (health data tasks), with paper noting physician-physician and model-physician agreement both span ~55-75% and wide individual spread; (6) grading prompt and individual criterion phrasings were tuned so intent was unmistakable to the grader (paper also cautions gpt-4.1's top rank may be partly explained by its use during prompt tuning); (7) std across 16 full runs ~0.002 (table 7: 0.0016-0.0029 by model). supersession check: no arxiv v2 exists; 2026 activity is healthbench professional (openai, april 2026) and third-party replications (e.g., open-source graders: kimi-k2 0.693, qwen3-235b 0.681 vs gpt-4.1 0.709) - these extend, not contradict, the original meta-eval numbers. only pedantic nuance: 0.569-0.730 are theme-level weighted averages of physician mf1, not the full range across individual physicians (individual spread is wider) - the claim's own 'wide individual spread' wording already captures this correctly.\n- claim: a large data vendor's rubrics-as-rewards found that llm-synthesized rubrics grounded in expert reference answers performed on par with human-authored rubrics as reward signals (0.359 vs 0.348), that prompt-specific rubrics substantially beat generic ones, and that the one component synthetic generation could not produce well was pitfall criteria - anticipating failure modes 'often demands human intuition and domain expertise'.\n verdict: confirmed | all core assertions verified against the source (arxiv 2507.17746, a large data vendor). (1) table 3: rar-implicit-synthetic 0.359 vs rar-implicit-human 0.348 on healthbench - synthetic rubrics grounded in expert reference answers on par with human-authored; ungrounded synthetic dropped to 0.320, confirming the grounding caveat. (2) prompt-specific vs generic confirmed: \"predefined generic rubrics substantially underperform compared to prompt-specific ones\" (generic ~0.29 vs ~0.37 prompt-specific in ablations; predefined 0.2472 vs rar-implicit 0.3194 on healthbench-1k). (3) pitfall claim confirmed verbatim: paper states synthetically generating effective pitfall criteria \"is inherently difficult\" and anticipating failure modes \"often demands human intuition and domain expertise.\" evidence-summary details also check out: four desiderata (expert-grounded, comprehensive, semantic weighting essential 1.0 / important 0.7 / optional 0.3 / pitfall 0.8 penalty, self-contained), o3-mini/gpt-4o generation, 7-20 criteria per prompt, implicit aggregation beating explicit weighted-sum with up to 28% relative gain over simple-likert on healthbench-1k (v1 figure; the current v2 abstract cites up to 31%), smaller judges benefiting most, and reduced cross-judge score variance (0.250-0.279 vs 0.220-0.254). date correct: v1 submitted 2025-07-23. minor currency notes, not refutations: (a) a v2 exists (2025-10-03) and the paper was accepted at iclr 2026 - citing the v1 html is fine but the 28% headline became 31% in the current version; (b) substantial 2026 follow-on literature (open rubric system arxiv 2602.14069, arbor 2606.03239, scale's own \"chasing the tail\" 2509.21500 and enterprise rar blog) critiques rar's static pointwise-weighted rubrics as having a discriminability ceiling and gaming vulnerability, but none contradicts the specific findings claimed. sources: https://arxiv.org/abs/2507.17746, https://arxiv.org/html/2507.17746v1, https://openreview.net/pdf?id=c1btcrdmt4, https://scale.com/blog/enterprise-rar\n- claim: rubricbench (mar 2026) measures a stable ~26-27 point preference-accuracy gap between self-generated and human-annotated rubrics across all judge backbones (e.g., deepseek-v3.2 57.8% vs 84.9%; gemini-3-flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement) does not close it - 'rubric formation', not judge reasoning, is the bottleneck.\n verdict: partially_confirmed | source check (arxiv 2603.01562v1, fetched): the rubricbench paper exists, is dated 2026-03-02 (v2 2026-03-03), and confirms nearly every specific: 1,147 adversarial pairwise comparisons; deepseek-v3.2 57.8% self-generated vs 84.9% human (vanilla 38.8%); gemini-3-flash 58.0% vs 85.3% (vanilla 56.4%); test-time scaling fails (rub@4->rub@32 flat or declining, refinement depth declining, while scaling human rubrics helps 75.4->85.3); unnecessary-rule 17.9% vs 10.1% and rigid-rule 12.8% vs 7.7%; 13+-item checklists / 'attention displacement' with >70% hallucination rates; ~85% human-rubric plateau; safety ~25-30% self-generated vs >90% human. two corrections: (1) the gap across all backbones is ~22-28 points, not a tight '~26-27' - 26-27 fits only the two cited models; (2) the 'rubric formation is the bottleneck' framing is the paper's interpretation and, more importantly, has been operationally superseded in part: 'support vector rubrics' (arxiv 2606.08077, june 2026, pku/ustc) closes the gap on rubricbench to 0.3 points (82.8 vs 83.1 human-oracle, vs 59.0 self-generated with same gpt-oss-120b judge) via max-margin contrastive rubric-bank learning over preference data - so the gap is stable under naive self-generation and test-time compute, but not under trained discriminative rubric construction. safety remains the residual weak domain even for svr (83.8 vs 92.5). claim is accurate as a description of the march paper; the 'does not close' generalization no longer holds as of june 2026.\n corrected: rubricbench (arxiv 2603.01562, mar 2026) measures a stable ~22-28 point preference-accuracy gap between self-generated and human-annotated rubrics across judge backbones (deepseek-v3.2 57.8% vs 84.9%; gemini-3-flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement depth) does not close it - the paper attributes the bottleneck to rubric formation, not judge reasoning. however, follow-up work (support vector rubrics, arxiv 2606.08077, jun 2026) closes the gap to ~0.3 points on rubricbench via max-margin rubric-bank learning from preference data, showing the gap is specific to naive self-generation rather than fundamental.\n\n---\n\n## domain: industry-practice\n\n*vendor qa stacks, eval tooling, spam economics*\n\n### a major annotation platform/a major annotation platform's production qa stack is double-blind independent labeling, automated discrepancy detection, expert adjudication, and rework, with a fine-tuned-llm similarity judge that only flags below-threshold responses for human review (never autonomously fails them), krippendorff's alpha as the published reliability bar (>=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor), and a multidimensional per-worker 'trust score' that routes complex tasks to high-trust raters. [load-bearing]\nevidence: fetched and read the full a major annotation platform 'inside the data factory' engineering post. it details the 4-step double-entry review flow, llm-assisted qc (fine-tune on expert-verified responses -> similarity score -> flag sub-threshold for human review), explicit krippendorff alpha thresholds preferred over cohen/fleiss kappa, per-task metric menu (kendall's tau/elo for rankings, f1/matthews for binary), trust-score routing, 3% workforce acceptance rate, sla-backed pay-only-for-quality guarantee, and admitted failure modes: for generative tasks 'there isn't a single clear-cut right answer' so accuracy is relative to the client's worldview, and low precision is attributed to poor instructions/edge-case coverage as often as to workers.\nsource: inside the data factory: how a major annotation platform produces the highest quality data at scale | https://a major annotation platform.com/blog/inside-the-data-factory-how-a major annotation platform-produces-the-highest-quality-data-at-scale | c. 2024, verified live 2026-07-14 | primary\nimplication: the most detailed public vendor qa architecture matches our hybrid design: llm-qc as a triage prefilter feeding one human touch, not an autonomous verdict-writer. adopt chance-corrected alpha (>=0.80 bar) as the industry-legible reliability floor, trust-score routing for reviewer allocation, and note the vendor itself concedes the ground-truth construct for generative work is client-instruction satisfaction, not objective truth (taxonomy q1, q2, q6).\n\n### An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only f1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting f1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice. [load-bearing]\nevidence: fetched the vendor's full research post (2026-05-27). The banking meta-eval is explicitly a meta-eval of verifiers, not agents; text-only 'autorubric' judges fail because evidence lives in spreadsheet formulas/tool state they cannot see; snapshot serialization loses formula references; the vendor's agent-judge (open-sourced, openhands sdk) opens files and follows references at inference time. Cheapest agent-judge config (~$42) beat the best baseline (~$422) by ~3 f1 at one-tenth cost; within one model family the architecture gap was 9.5 f1. they prescribe measuring every verifier against adjudicated expert labels before trusting it to score work or produce rewards.\nsource: your verifier is probably the bottleneck. we built one that isn't. (the vendor's agent-judge verifier and banking verifier meta-eval) | [link withheld] | 2026-05-27 | primary\nimplication: direct evidence for three taxonomy questions: (q1) meta-evaluation must use adjudicated expert labels, and a ~89.5% human agreement floor is achievable with dual-coding + adjudication; (q5) the judge-vs-adjudicated-human ceiling is wildly domain-dependent (f1 0.63 vs 0.95), so per-axis judge-autonomous/assisted/human-only lanes are mandatory, not optional; (q3) evidence closure is the binding constraint - the judge must see the same evidence surface the attempter's claims are about, or it verifies against a flattened proxy.\n\n### the 2026 eval-tooling vendors (langsmith, braintrust, galileo) have converged on one judge-calibration pattern - small human-labeled reference sets (langsmith: start at ~20 balanced examples; braintrust: 50-100 per scoring dimension) with an 'alignment score' defined as raw percent agreement with the human expert, iterate the judge prompt against it, and route low-confidence or judge-disagreement items to humans - and none of them ships chance-corrected statistics. [load-bearing]\nevidence: fetched langsmith's official 'improve llm-as-judge evaluators using human feedback' doc: annotation queue -> reference dataset (>=20 examples, balanced 0/1 labels) -> evaluator playground -> alignment score = 'percentage of examples where the evaluator's judgment matches that of the human expert' -> cluster misalignments into failure modes and encode as prompt instructions. fetched braintrust's 2026-04-03 article: 3-tier stack (deterministic checks -> llm judge -> human), 50-100 labels per dimension before trusting a judge, multi-judge disagreement routed to humans, worst failure mode named as 'silent overconfidence' (judge always returns a score even without adequate context), and explicit warning that human review is not automatically gold standard (fatigue, vague rubrics can be worse than a decent judge). galileo's clhf (2025-02-10 blog) auto-adapts metrics from expert corrections.\nsource: langsmith: improve llm-as-judge evaluators using human feedback (+ braintrust: llm-as-a-judge vs human-in-the-loop, 2026-04-03) | https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback | retrieved 2026-07-14 (braintrust article 2026-04-03) | primary\nimplication: our foundation can adopt the proven mechanics (annotation-queue-to-reference-set loop, failure-mode clustering into rubric amendments, disagreement routing) but must not copy the metric: the entire tooling industry standardized on raw percent agreement over tiny unbalanced sets - exactly what taxonomy q1 proposes banning. differentiation and validity headroom lie in chance-corrected stats, per-class recall at fixed prevalence, and minimum reference-set designs; also budget expectations: tens of labels per axis is the industry's plateau assumption, not hundreds.\n\n### at a large data vendor's outlier/remotasks (bulba experts program for google, march 2023-april 2024 per internal documents), spam - defined internally as 'writing gibberish, writing incorrect information, gpt-generated thought processes' - routinely got paid because per-item detection could not keep up; defenses degraded to zerogpt screening, crude country-level bans (egypt, pakistan, kenya, venezuela), and were circumvented by vpns, while a black market sells verified annotator accounts (outlier, crowdgen, prolific). [load-bearing]\nevidence: opened futurism's 2025-06 piece quoting inc.'s internal-document exclusive verbatim: spammers 'could get away with just totally submitting garbage and there weren't enough people to track them down'; an allocations team once 'dumped 800 spammers' into one queue; supervisors advised to vet entries with zerogpt; scale's spokesperson disputed the story claiming safeguards existed. algorithmwatch separately documented account-resale markets. prior research (cited in the same reporting thread) estimated 33-46% of mturk crowdworkers used llms on a text task. community reports (indeed/glassdoor/trustpilot 2025) confirm the attempter->reviewer->senior-reviewer pyramid with timed capped pay and opaque removals.\nsource: futurism: the ai company zuckerberg just poured $14 billion into... (quoting inc. internal-documents exclusive by sam blum) | https://futurism.com/scale-ai-zuckerberg-incompetence | 2025-06 (documents cover 2023-03 to 2024-04) | practitioner\nimplication: direct answer to taxonomy q8: at the largest vendor, per-item content-based spam classification lost to spam economics; identity/provenance is also compromised (account resale, vpns). the foundation should treat wholesale-llm/spam defense as economics-based gating (rate limits, work-history consistency, randomized deep audits, payment holds) with content detection only as a weak auxiliary signal - and assume attempter identity is an adversarial variable.\n\n### An AI data vendor acquired a data-quality startup (announced 2026-01-28) specifically for algorithms that 'flag incorrect data without a second human reviewer,' after competing bids from other data-labeling companies - signaling that algorithmic second-pass qa replacing the human review layer is now an acquisition-grade core capability for human-data vendors.\nevidence: techcrunch (2026-01-28), the startup's CEO letter, and the vendor's own announcement triangulate: 9 staff of the startup including all three mit-phd co-founders join the vendor's research team; rationale quotes name confident learning, data-centric ai, and llm evaluation as the acquired capabilities; an AI data vendor serves 8 top labs at ~$300m annualized run-rate (end 2025). The startup's core product audited human-labeler output quality algorithmically.\nsource: techcrunch: Report on an AI data vendor's acquisition of a data-quality startup | [link withheld] | 2026-01-28 | practitioner\nimplication: the market has already voted for our thesis: augment/replace the human qa layer over the mandatory human attempter layer. confident-learning-style label-noise detection (statistical, model-based, no second human) is a complementary channel to llm-judge qa and should be considered in the foundation's filtering stage.\n\n### surge ai (~$1.2b revenue 2024, ~50k contractors, bootstrapped) runs qa as real-time dashboards over gold-standard accuracy, inter-annotator agreement, and per-worker trust ratings, with flagged labels automatically reassigned to other annotators; its published advancedif verifier work with meta reports f1 0.728 versus human judgments and +13% rl gains from rubric-based rewards.\nevidence: fetched sacra's surge ai company report (updated 2026-04-21; figures are sacra estimates). qa description: gold-standard accuracy + iaa + per-worker trust ratings on live dashboards; automated reassignment of low-quality labels; premium per-minute pay (~$0.30-0.40/min) as a quality incentive; vetting = domain tests + background checks + ongoing performance evaluation. hemingway-bench used 5,000+ blind pairwise expert comparisons. risks noted: ~12-customer concentration, ca misclassification suit alleging unpaid training time and tight task timers.\nsource: sacra: surge ai revenue, funding & news | https://sacra.com/c/surge-ai | 2026-04-21 | practitioner\nimplication: the premium-quality market leader's qa is continuous gold-task injection + iaa + per-worker trust telemetry with automatic reassignment - i.e., population-level statistical qc, not per-item semantic review. their own verifier tops out at f1 0.728 vs humans on instruction-following, a second independent datapoint (with an AI data vendor's 0.63-0.66) that ~0.65-0.75 f1 is the current judge ceiling on expert-graded open-ended work (taxonomy q5).\n\n### micro1 (~$300m annualized revenue by 2026-04) embeds real-time qa verification layers directly in its annotation interface and operates a closed rubric-feedback loop: labs submit updated rubrics and corner-case prompts, which are routed to the highest-performing experts and used to retrain its vetting algorithms.\nevidence: sacra company report on micro1 (2026) describes the embedded qa layers, lab-facing dashboards with daily output and pass/fail rates, and the rubric-update routing loop; forbes (2026-02-03) interview with ceo ansari frames evaluation as a permanent top-level function ('how well does it work, and what are the thresholds to go into production'). screening is via ai interviewer (zara) at thousands of interviews/day; a major annotation platform and mercor use the same ai-interview screening pattern (a major annotation platform: 3% acceptance, 4-stage vetting).\nsource: sacra: micro1 revenue, valuation & funding (+ forbes, 2026-02-03) | https://sacra.com/c/micro1 | 2026 (forbes 2026-02-03) | practitioner\nimplication: the rubric-amendment loop in taxonomy q4/q6 is already production practice: client rubric updates are first-class config artifacts routed to top performers, i.e., upstream compounding work is prioritized over per-item adjudication. also confirms the industry attacks the expertise-gap root cause at intake (automated ai-interview screening + longitudinal per-worker metrics), not per-item.\n\n### braintrust's published cost arithmetic makes the human-budget fork concrete: an llm judge over 10,000 outputs costs ~$5-15 while a domain expert reviewing 500 of them costs ~$800-1,800 (~100x per output), which is why its prescribed architecture is deterministic checks on everything, llm judges on everything, and humans only on flagged/low-confidence/disagreement slices plus random spot-checks of confident passes.\nevidence: fetched braintrust article (2026-04-03) and human-review docs. the hybrid loop is explicit: trace production -> continuous deterministic + judge scoring -> flag low-confidence/scorer-disagreement -> human review -> reviewed findings become new scorers/rubric refinements -> ci regression gates. they name the biggest practitioner mistake as treating judge scores as ground truth without human validation, and note untrained reviewers with vague rubrics can produce labels worse than a decent judge.\nsource: braintrust: llm-as-a-judge vs human-in-the-loop evals: when to use each | https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals | 2026-04-03 | practitioner\nimplication: supports taxonomy q6's routing answer as consensus practice: spend the single human touch on judge-flagged and judge-disagreement items, keep a randomized audit slice of confident passes, and convert every human review into a compiled-config improvement (new scorer/exemplar) rather than a one-off verdict.\n\n### fine-tuned specialist judges have commoditized binary grounding checks: patronus lynx (fine-tuned llama-3-70b) beats gpt-4o on halubench faithfulness detection and lifted one customer's hallucination detection from 0.375 to 0.69, while galileo's luna-2 (arxiv 2026-02-20) runs hundreds of per-metric lora adapters on one small-model backbone at >80x lower cost and >20x lower latency than llm-as-judge at claimed parity accuracy, in production on 100m+ sessions.\nevidence: patronus launch post + siliconangle (2024-07-11) give halubench numbers (lynx 70b +8.3% vs gpt-4o on pubmedqa inaccuracies; 8b +24.5% vs gpt-3.5); patronus evaluators page gives the algomo 0.375->0.69 customer number and gamma's 1,000+ manual eval hours saved; percival extends this to tracing where in a multi-step agent run the failure occurred. fetched luna-2 arxiv abstract (2602.18583): single-token slm evaluation, >80x cost / >20x latency reduction, 100m+ sessions and 100b tokens/month in production, $30m+ claimed annual eval savings. vendor-reported numbers; treat magnitudes, not decimals.\nsource: luna-2: scalable single-token evaluation with small language models (arxiv 2602.18583) + patronus lynx launch materials | https://arxiv.org/abs/2602.18583 | 2026-02-20 (lynx 2024-07-11) | primary\nimplication: for taxonomy q2/q3: the 'compile to binary checks' bucket (groundedness, faithfulness, toxicity, tool-selection) can run on cheap specialized models per axis at near-zero marginal cost, reserving frontier-model judges for entailment/calibration/judgment-laden axes - a two-tier judge economy is standard 2026 practice, not an exotic design.\n\n### deepeval's recommended progression - start with holistic g-eval, then move to dag (a deterministic decision tree whose nodes are narrow llm-judge calls) 'for more control' - is the open-source codification of decomposed judging: verdicts assembled from atomic, individually-checkable steps rather than one holistic score.\nevidence: deepeval docs (metrics-introduction, metrics-dag; dag introduced 2025-02, docs current 2026-07) describe dagmetric as 'deterministic decision trees' for objective/mixed criteria producing deterministic scores, explicitly positioned above g-eval when control matters, including a conversational variant for multi-turn work. confident-ai's 2025-02-09 engineering post explains the motivation: holistic llm scores are too noisy for objective criteria.\nsource: deepeval docs: dag (deep acyclic graph) metric + confident ai engineering post | https://deepeval.com/docs/metrics-dag | 2025-02-09, docs verified 2026-07-10 | practitioner\nimplication: taxonomy q3's decomposition-granularity question has a live open-source reference implementation: per-criterion decision trees with narrow judge nodes and deterministic aggregation. adopt the pattern (and its admission that decomposition is for objective/mixed criteria - judgment-laden axes stay holistic/g-eval-style) rather than inventing the machinery.\n\n### the human reviewer layer at scale vendors is itself unreliable for structural/incentive reasons the autoqa must not inherit: outlier's attempter->reviewer->senior-reviewer pyramid runs on timed, capped per-task pay, opaque promotion/removal, uneven task allocation, and internal competition, per 2025 contributor reports across indeed/glassdoor/trustpilot.\nevidence: aggregated 2025 worker reports: pay only within a capped time window (unpaid overage), removal from projects without stated reason, alleged 300-reviewer queues where ~10 got unlimited tasks, queue managers running webinars/'war rooms' as the calibration mechanism, quality undermined by contributors and reviewers 'pitted against each other'. community-sourced and unverifiable individually, but consistent across platforms and consistent with the inc. documents' picture of overwhelmed review capacity.\nsource: indeed/glassdoor/trustpilot outlier reviewer reports (aggregated) | https://www.indeed.com/cmp/outlier-ai/reviews?fjobtitle=reviewer&fcountry=us&ftext=outlier | 2025 (various) | community\nimplication: reviewer variance (our root causes 1-3) is partly manufactured by incentive design: time-capped pay punishes careful review, and opaque consequences destroy calibration motivation. the autoqa foundation should specify the human-touch interaction's incentive contract (untimed or accuracy-paid adjudication, visible appeal outcomes) as part of the design, not just the routing logic.\n\n### vendor materials repeatedly attribute low inter-annotator precision to instruction/rubric defects rather than annotator failure - a major annotation platform's own remediation playbook for low precision starts with 'poor instructions, weak edge-case coverage' and prescribes guideline revision and calibration sessions before retraining or replacing people.\nevidence: a major annotation platform data-factory post's scenario playbook: low precision -> better training, clearer guidelines, calibration sessions, edge-case examples; high precision + low accuracy -> refine the ground truth itself and check for systematic bias in training materials. kili's industry guide (cited in 2025 coverage) similarly notes an annotator consistently disagreeing on one label category 'is often revealing a guideline gap', and industry estimates put rlhf preference-annotation noise above 20%.\nsource: a major annotation platform data-factory playbook + kili technology annotation quality guide | https://kili-technology.com/blog/data-annotation-guide-how-to-achieve-high-quality-data-in-complex-ai-data-operations | 2024-2025, verified live 2026-07-14 | practitioner\nimplication: industry operating doctrine already treats systematic reviewer disagreement as an instruction-gap signal first (taxonomy q2's third verdict class and q9's rubric-revision trigger): a disagreement-clustered-by-criterion feed to project owners mirrors existing vendor practice and will be legible to them. but no vendor publishes the variance attribution split, so the taxonomy's cheap variance-attribution pilot is still needed - practice assumes criteria-vagueness dominance without measuring it.\n\n### tools & artifacts\n- an AI data vendor's agent-judge: open-sourced reactive agent-as-judge verifier (openhands sdk) that runs in the rollout's own environment, opens files/tools at inference time; paired with the vendor's banking agent benchmark (huggingface) and the banking verifier meta-eval (3,204 expert labels, full release pending). | [link withheld] | reference architecture for evidence-closure-correct judging and for meta-evaluating our judge against adjudicated expert labels before trusting it.\n- langsmith align evaluator + annotation queues: built-in workflow: human-labeled reference set (>=20 balanced examples) -> alignment score (percent match) -> evaluator playground iteration; corrections folded into judge prompt instructions. | https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback | the de facto industry calibration loop to improve on (its percent-agreement metric is the weakness).\n- braintrust human review + loop: span-level human review ui with judge-vs-human scores side by side, flagged-trace-to-test-case loop, ci regression gating, and an eval-workflow agent (loop, 2025-11). | https://www.braintrust.dev/docs/annotate/human-review | model for converting each human touch into permanent compiled-config assets (scorers, datasets) rather than one-off verdicts.\n- patronus lynx + percival: open-weight fine-tuned faithfulness judge (binary grounded/ungrounded + reasoning trace) and an agentic debugger that localizes failures in multi-step traces. | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | cheap specialist model for our 'compile to binary checks' grounding lane; percival pattern for localizing which claim/step failed.\n- galileo luna-2 + clhf: slm backbone with per-metric lora adapter heads (hundreds of metrics on one gpu, >80x cheaper than llm-as-judge; arxiv 2602.18583) plus continuous learning with human feedback that adapts metrics from expert corrections. | https://arxiv.org/abs/2602.18583 | economics template for running many per-axis checks on every item; clhf is prior art for correction-driven judge adaptation.\n- deepeval dag / conversational dag: open-source deterministic decision-tree metrics whose nodes are narrow llm-judge calls; positioned above holistic g-eval when control is needed. | https://deepeval.com/docs/metrics-dag | open reference implementation of decomposed per-criterion judging with deterministic aggregation (taxonomy q3).\n- cleanlab: confident-learning software that flags incorrect human labels algorithmically without a second human reviewer. | [link withheld] | statistical label-noise detection as a judge-independent third qa channel alongside llm judging and human audit.\n- a major annotation platform connect (a major annotation platform): marketplace exposing per-worker longitudinal quality telemetry (assessment scores, interview transcripts, rework rate, efficiency) to clients staffing data teams. | https://a major annotation platform.com/services/a major annotation platform-connect | prior art for per-attempter trust telemetry as a routing and gating input in our foundation.\n\n### disagreements\n- judge-vs-human ceiling: langchain marketing materials claim llm judges reach ~85% alignment with humans, exceeding ~81% human-human agreement (an mt-bench-style chat-preference number), while an AI data vendor's banking verifier meta-eval shows the best 2026 verifiers at f1 0.633-0.664 against practicing-expert labels (humans at 89.5% agreement) and surge's advancedif verifier at f1 0.728. the 'judges already beat humans' claim holds only for casual preference tasks, not expert-graded artifact-heavy work - the ceiling is domain-dependent by ~30 f1 points.\n- agreement statistic: a major annotation platform publishes chance-corrected krippendorff's alpha thresholds as its quality standard, while langsmith/braintrust standardize on raw percent agreement over tiny reference sets. the industry has not converged, and the eval-tooling side uses exactly the statistic our taxonomy proposes banning.\n- is human review the gold standard? An AI data vendor treats dual-coded, adjudicated practicing-expert labels as ground truth for meta-evaluation; braintrust explicitly warns human review is 'not automatically a gold standard' (untrained reviewers, vague rubrics, fatigue can underperform a decent judge). reconcilable - adjudicated expert panels vs ad-hoc reviewers - but the sources take opposite rhetorical stances.\n- vendor qa claims vs observed integrity: a major annotation platform/surge/scale market rigorous multi-layer qa, slas, and 3% acceptance rates, while internal scale documents (via inc./futurism) show spam paid at scale for 11 months, zerogpt as a stopgap, and country-level bans; scale's spokesperson disputes the reporting ('filled with so many inaccuracies'). marketing descriptions of qa pyramids should not be taken as evidence they perform as described.\n- content-based vs economics-based spam defense: patronus/galileo/detection vendors implicitly sell content-based classification as sufficient, while the scale/outlier record and the annotator-account black market (algorithmwatch) show content detection losing to spam economics at scale; an AI data vendor's purchase of a data-quality startup (statistical label auditing) is a third position - detect via label statistics rather than either content classifiers or human re-review.\n\n### gaps\n- no data vendor publishes actual achieved inter-reviewer reliability numbers (e.g., krippendorff alpha values on common item sets) - only target thresholds. the one hard human-agreement number found (an AI data vendor's 89.5%, dual-coded adjudicated subset) is from a benchmark paper, not production qa reporting.\n- no public data anywhere on false-agreement rates (judge and attempter both wrong vs expert ground truth), on per-project gold-set sizes/refresh rates at data vendors, or on judge-model version pinning / golden-set regression gates at data vendors (taxonomy q1, q9). eval-tooling vendors support regression testing generically but no vendor documents pinning policy for annotation-qa judges.\n- no published evidence on whether qa feedback changes attempter behavior longitudinally (repeat-error rates per criterion, revision acceptance) - no vendor reports feedback-efficacy metrics, and no controlled feedback-vs-verdict-only comparison was found (taxonomy q7).\n- a large data vendor's current (2025-2026) production auto-qa internals are undocumented publicly except via leaks and contributor folklore; turing publishes essentially nothing concrete; invisible's public trace is limited to a job post showing ai qa trainers build rubrics/pass-fail criteria and regression suites. mercor's qa is visible only through reviewer-hiring materials emphasizing consistency screening.\n- could not verify a major annotation platform data-factory blog's exact publication date (undated on page; content verified live 2026-07-14). sacra figures for surge/micro1 are analyst estimates, not audited. the inc. internal documents were verified only via futurism's direct quotations (inc. itself 403s).\n- no vendor was found admitting or measuring feedback-loop gaming (attempters overfitting the qa judge's decision boundary) or monoculture drift from judge feedback - taxonomy q8's core dynamic is unstudied in public industry sources.\n\n### verifications\n- claim: a major annotation platform/a major annotation platform's production qa stack is double-blind independent labeling, automated discrepancy detection, expert adjudication, and rework, with a fine-tuned-llm similarity judge that only flags below-threshold responses for human review (never autonomously fails them), krippendorff's alpha as the published reliability bar (>=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor), and a multidimensional per-worker 'trust score' that routes complex tasks to high-trust raters.\n verdict: confirmed | fetched the source directly (https://a major annotation platform.com/blog/inside-the-data-factory-how-a major annotation platform-produces-the-highest-quality-data-at-scale, published september 12, 2024, still live 2026-07-14). every element of the claim checks out against the text: (1) 4-step double-entry flow - \"two independent labelers perform the same task without knowledge of each other's work,\" results automatically compared for discrepancies, expert reviewer makes final determination, rework with feedback; (2) llm judge fine-tuned \"on a dataset of high-quality, expert-verified responses,\" scores similarity, and its described action is \"flagging responses that fall below a certain similarity threshold for human review\" - flag-only, humans handle challenging cases; (3) exact krippendorff alpha table: >=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor, preferred over cohen's/fleiss' kappa for versatility/robustness (2+ raters, missing data, mixed data types); (4) a major annotation platform trust score is explicitly \"multidimensional\" (historical accuracy, consistency, completion rate, complex-task handling) with \"high-trust ai trainers may be prioritized for more critical or complex tasks.\" ancillary evidence items (3% acceptance, pay-only-for-sla-quality, kendall's tau/elo, f1/matthews) also present. supersession search found no newer official a major annotation platform post revising the trust score or qa stack - the sept 2024 post remains the canonical description; 2026 third-party reviews add worker-side complaints (inconsistent review outcomes, unpaid evaluation phases) but do not contradict the claimed process. two wording nuances, neither rising to a correction: the post's own term is \"double-entry\" (the mechanics match \"double-blind\" - independent, mutually unaware labelers); and the post describes the llm judge only as flagging for human review - it does not affirmatively state \"never autonomously fails,\" though nothing in the post says it auto-fails work.\n- claim: An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only f1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting f1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice.\n verdict: confirmed | fetched [link withheld] directly. every element of the claim checks out against the source: (1) dated may 27, 2026; (2) the banking verifier meta-eval is explicitly a meta-eval of verifiers built on 21 tasks from the vendor's banking agent benchmark with 3,204 pass/fail judgments from practicing bankers; 89.5% inter-annotator agreement on a dual-coded subset, with disagreements adjudicated before inclusion; (3) the vendor's agent-judge is a reactive agent-as-judge running in the same rollout environment (shared filesystem, interpreter, mcp tools), open-sourced on the openhands sdk; (4) f1 0.633-0.664 on the banking meta-eval (artifact-heavy) vs f1 0.951 on openclaw, their internal stateful-tools productivity benchmark (200-criterion sample, +7.5 over next-best); (5) architecture (evidence view: text/snapshot/live env; evidence path: fixed vs inference-time) beats model choice - within gpt-5.4 family, agent-judge/nano beats archipelago/gpt-5.4 by 9.5 f1; (6) cheapest agent-judge config (~$42, gpt-5.4 nano) beats best baseline (archipelago/gemini 3 pro, f1 0.604, ~$422) by ~3 f1 at ~one-tenth cost; (7) the post prescribes meta-eval of verifiers against adjudicated expert labels before trusting them for scoring/rewards. supersession check (two searches, july 2026): no newer contradicting work found; the full agent-judge paper and the banking meta-eval dataset release are still pending, so numbers could later be refined, but as of today the post is the authoritative source and the claim matches it. minor nuance only: the 0.951 benchmark is named openclaw and is internal (not public), which the claim's phrasing (\"simpler stateful-tools benchmark\") accurately reflects.\n- claim: the 2026 eval-tooling vendors (langsmith, braintrust, galileo) have converged on one judge-calibration pattern - small human-labeled reference sets (langsmith: start at ~20 balanced examples; braintrust: 50-100 per scoring dimension) with an 'alignment score' defined as raw percent agreement with the human expert, iterate the judge prompt against it, and route low-confidence or judge-disagreement items to humans - and none of them ships chance-corrected statistics.\n verdict: partially_confirmed | verified: (1) langsmith doc (https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback) says exactly what was reported - annotation queue -> reference dataset (~20 examples minimum, balanced 0/1) -> evaluator playground -> alignment score quoted verbatim as \"the percentage of examples where the evaluator's judgment matches that of the human expert\" -> cluster misalignments into failure modes as prompt instructions. no kappa/alpha anywhere; doc is undated. (2) braintrust article (https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals, confirmed 2026-04-03) matches on all six checked points: 3-tier stack, 50-100 labels per scoring dimension, multi-judge disagreement routed to humans, \"silent overconfidence\" named as the most dangerous failure mode, human-review-not-gold-standard warning (reviewer fatigue, vague rubrics), and agreement tracked as raw agreement rate with no chance-corrected statistic. (3) galileo clhf exists and auto-adapts metrics from expert corrections (feedback -> llm-generated few-shot examples appended to the metric prompt, capped at most recent 15, claimed 20-30% accuracy gain).\n\nrefuted/overstated parts: (a) the \"convergence on raw percent agreement\" framing fails for galileo. galileo's own guidance - \"how to calibrate your llm judge with human annotations\" (https://galileo.ai/blog/calibrate-llm-judge-human-annotations, may 15/jul 6 2026, i.e. newer than the cited 2025-02-10 clhf post) - explicitly recommends against raw percent agreement (imbalanced-label example: 90% raw agreement, kappa ~ -0.05) and prescribes cohen's kappa, fleiss' kappa, and krippendorff's alpha with a recalibrate-when-kappa<0.60 trigger. galileo also has a dedicated cohen's kappa explainer (https://galileo.ai/blog/cohens-kappa-metric, 2025-03-12). (b) galileo's clhf is also not the same pattern as langsmith/braintrust - it has no reference set + alignment score loop; it is few-shot auto-improvement, so the three-vendor \"one pattern\" claim overreaches. (c) the narrower claim \"none of them ships chance-corrected statistics\" as an in-product feature survives verification: galileo's kappa content points users to scikit-learn/statsmodels/r for computation, and its product features (autotune, annotations, signals, experiments, clhf) are not described as computing kappa; langsmith and braintrust expose only percent-agreement metrics. but the claim as worded implies the vendors don't even acknowledge chance correction, which is false for galileo. supersession check: newer relevant work exists - arxiv 2606.00093 (rao & callison-burch, 2026-05-25, \"agreement metrics for llm-as-judge evaluation: what to report and why\") argues for reporting kappa alongside accuracy, and langchain published an align evals calibration resource (2026-03-10) still framed around human-correction agreement.\n corrected: langsmith and braintrust have converged on a judge-calibration pattern of small human-labeled reference sets (langsmith: ~20 balanced examples; braintrust: 50-100 per scoring dimension) with alignment measured as raw percent agreement with human experts, prompt iteration against misalignments, and routing of low-confidence/judge-disagreement items to humans; neither ships a chance-corrected statistic. galileo differs: its clhf (2025) turns expert corrections into few-shot prompt examples (max 15) rather than a reference-set alignment score, and its newer 2026 calibration guidance explicitly rejects raw percent agreement in favor of cohen's kappa / fleiss' kappa / krippendorff's alpha (recalibrate below kappa 0.60) - though it directs users to external tools (scikit-learn, r) rather than computing these in-product. so the \"no chance-corrected stats shipped in-product\" observation holds across all three, but there is no three-vendor convergence on raw percent agreement as the alignment definition.\n\n---\n\n## domain: contrarian\n\n*the case against, adversarial attacks, validity critiques*\n\n### raw percent-agreement systematically overstates llm-judge ability: in the largest judge meta-evaluation to date (21 judges, 9 providers, ~541,000 judgments, including april-2026 frontier models), cohen's kappa runs 33-41 percentage points below exact-match agreement on mt-bench, judge rankings shift by up to 14 positions across benchmarks, and two production-deployed judges combine test-retest reliability >0.95 with severe position bias >0.10 (a 'consistency-bias paradox'). [load-bearing]\nevidence: norman, rivera & hughes ran 118 runs across mt-bench, judgebench, rewardbench under three protocols (agreement, consistency, bias audit); kappa deflation was universal across the cohort; they distill a 'minimum viable validation protocol'. verified from the arxiv abstract page. corroborated by the neurips 2025 position paper 'neither valid nor reliable? investigating the use of llms as judges' (arxiv 2508.18076, chehbouni et al.), which argues via social-science measurement theory that llj adoption has outpaced scrutiny of validity/reliability and that human-agreement proxying is an unvalidated assumption.\nsource: reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic\nimplication: directly answers q1's metric-standardization question: ban raw percent-agreement and single 'accuracy vs humans' numbers from all autoqa meta-evaluation; require chance-corrected statistics per benchmark. and for q5: run-to-run consistency must never be reported as evidence of validity - a judge can be perfectly repeatable and severely biased simultaneously, so a separate perturbation/bias audit is a mandatory ship gate independent of the consistency gate.\n\n### across 106 experiments and 370 effect sizes, human-ai combinations performed significantly worse than the best of human or ai alone (hedges g = -0.23, 95% ci -0.39 to -0.07), with losses concentrated in decision-making tasks and specifically when the ai outperforms the human alone - and a 2025 aies study found human reviewers followed severely race-biased ai hiring recommendations ~90% of the time. [load-bearing]\nevidence: vaccaro, almaatouq & malone (nature human behaviour, preregistered meta-analysis; direction, decision-task losses, and ai-better-than-human losses verified from the arxiv abstract; g value corroborated across mit sloan, nature's own summary, and researchgate). uw's 'no thoughts just ai' (aies 2025, doi 10.1609/aies.v8i3.36749, 528 participants, 16 job types) found participants matched biased ai picks even at moderate bias and ~90% at severe bias, vs equal selection rates with no/neutral ai. the 2026 practitioner literature (e.g., tianpan.co hitl rubber-stamp essay, 2026-04-15) reports the same mechanism in production review pipelines.\nsource: when combinations of humans and ai are useful: a systematic review and meta-analysis | https://www.nature.com/articles/s41562-024-02024-1 | 2024-10-28 | academic\nimplication: q6's fork is real and the evidence leans against the default: 'one shallow human touch verifying the ai verdict' is precisely the decision-task, ai-better-than-human configuration where the meta-analysis finds negative synergy and rubber-stamping. the foundation should treat zero-per-item-touch plus randomized blind deep audits (human judges never see the ai verdict before committing their own) as the null design to beat, and any verify-the-ai-verdict interaction must be measured against its own overturn rate to prove the human is adding information.\n\n### there is a proven theoretical ceiling on judge-based validation: when the judge is no more accurate than the model/content being evaluated, no debiasing method using gold labels can cut the required amount of ground-truth data by more than a factor of two, and empirical savings are smaller than the 2x bound. [load-bearing]\nevidence: dorner, nastl & hardt (iclr 2025) prove the bound for methods that combine cheap judge scores with a small gold-label set to correct judge biases such as self-preference; verified from the arxiv abstract page including the central quote 'when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth' by more than half.\nsource: limits to scalable evaluation at the frontier: llm as judge won't beat twice the data | https://arxiv.org/abs/2410.13341 | 2024-10-17 | academic\nimplication: q1's gold-set arithmetic cannot be escaped by judge cleverness: on exactly the axes where the expertise-gap root cause bites (attempter competence near or above judge competence), certifying the autoqa still costs at least half the gold labels a judge-free validation would - so hierarchical/pooled validation across projects and a permanent (not bootstrap) expert-audit channel are structural requirements, and any vendor claim that the judge 'validates itself' at scale should be treated as mathematically impossible in the regime that matters.\n\n### human evaluation criteria are output-dependent and unstable: even when graders define criteria before grading, the act of grading changes their criteria and they retroactively revise earlier grades ('criteria drift'), implying evaluation criteria for llm-output quality cannot be fully determined prior to observing outputs. [load-bearing]\nevidence: shankar, zamfirescu-pereira, hartmann, parameswaran & arawjo (uist 2024, evalgen system study); abstract verified via arxiv crawl: 'we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria'; the authors argue there is reason to believe criteria never fully settle because they adapt to the observed output distribution.\nsource: who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences | https://arxiv.org/abs/2404.12272 | 2024-04-18 | academic\nimplication: falsifies q4's implicit one-shot compilation contract: 'instructions compile once into artifacts, fresh judge hits target agreement with no conversation' is unstable because the project owner's own criteria will drift once they see real attempter submissions and judge verdicts. rubric compilation must be designed as a versioned re-compilation loop with an explicit retroactive re-scoring policy, and owner sign-off must happen on graded real items, not on abstract criteria.\n\n### llm judges favor models whose mistakes resemble their own (a generalization of self-preference, measured by chance-adjusted mistake-overlap capa), and model errors across the industry are becoming more correlated as capabilities improve - undermining the assumption that ai oversight of ai-assisted work catches failures.\nevidence: goel et al., icml 2025; abstract verified via arxiv crawl: 'llm-as-a-judge scores favor models similar to the judge' and 'model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures'. related: 'preference leakage' (arxiv 2502.01534) documents contamination when generator and judge are related.\nsource: great models think alike and this undermines ai oversight | https://arxiv.org/abs/2502.04313 | 2025-02-06 | academic\nimplication: q5: judge model family must differ from the generator being critiqued and from whatever assistant attempters plausibly used - per-project judge routing plus similarity reporting. q1: the false-agreement rate (judge passes an llm-assisted attempter because both share priors) is predicted to rise over time, which converts the independent human expert-audit channel from a bootstrap phase into a permanent structural component.\n\n### llm judges do not follow their own rubrics: on arena-hard auto, the explicit evaluation schema explains under 10% of verdict variance for some judges (unexplained variance >90% for deepseek-r1-32b), and factor correlations above 0.93 across nominally distinct criteria show per-axis scores collapse into a single halo factor.\nevidence: feuer et al. introduce 'schematic adherence' (how much of the verdict the stated rubric explains) and psychometric validity checks; abstract verified via arxiv crawl: 'severe schema incoherence and factor collapse across popular judges'; elo-style aggregation additionally masks genuine ranking uncertainty. code at github.com/penfever/judgment-to-noise.\nsource: when judgment becomes noise: how design failures in llm judge benchmarks silently undermine validity | https://arxiv.org/abs/2509.20293 | 2025-09-24 | contrarian\nimplication: q3/q4: a per-axis verdict sheet can be theater - the judge may emit one holistic impression dressed up as n criterion scores. the foundation should require discriminant-validity and schematic-adherence tests per project config (do axis scores actually vary independently? does the rationale's schema explain the verdict?) before per-criterion outputs are exposed to attempters or used for specialist routing.\n\n### llm judges have low intra-rater reliability - identical items re-scored across runs with identical settings produce inconsistent, 'almost arbitrary in the worst case' ratings - and the obvious fix (temperature-0 determinism) measurably reduces agreement with human judgment; meanwhile the human baseline itself is weak (summeval inter-annotator kappa: 0.492 crowd, 0.413 expert first round, 0.71 only after a second adjudication round).\nevidence: haldar & hockenmaier, findings of emnlp 2025 (pp. 24986-25004); verified by reading the paper pdf (pp. 1-3): contributions are (1) low agreement of llm ratings across runs, (2) disabling sampling hurts human-agreement, (3) phenomenon persists across summac, summeval, mt-bench; the summeval human kappa figures are quoted in their section 3.1.\nsource: rating roulette: self-inconsistency in llm-as-a-judge frameworks | https://aclanthology.org/2025.findings-emnlp.1361.pdf | 2025-11-04 | academic\nimplication: q5: verdict flip rate is intrinsic and cannot be silenced with greedy decoding without paying validity - so the build must budget k-sample voting and treat residual flip clusters as underspecification signal routed to rubric revision. q1: raw human labels sit below conventional reliability thresholds; validation targets must be adjudicated (multi-round) labels, which prices the gold set honestly.\n\n### llm judges give the weakest signal exactly where qa needs them most: they cannot reliably grade responses to questions they cannot answer themselves (poor signal on the hardest items in a benchmark), and in expert domains subject-matter experts agreed with llm-judge picks only 64% (mental health) to 68% (dietetics) of the time, with the judge favoring superficially actionable detail.\nevidence: 'no free labels' (kim et al., arxiv 2503.05061, mar 2025) shows judge quality collapses on the most difficult items and that human-written reference answers improve agreement and reduce self-preference; szymanski et al. (acm iui 2025, doi 10.1145/3708359.3712091) measured the 64/68% sme agreement in pairwise comparisons on domain tasks. both reported consistently across two independent search engines; abstracts inspected.\nsource: no free labels: limitations of llm-as-a-judge without human grounding | https://arxiv.org/abs/2503.05061 | 2025-03-07 | academic\nimplication: q5/q2: the expertise-gap root cause is not automatically closed by an llm judge - the judge's competence ceiling binds hardest on exactly the expert items where human reviewers also fail, so per-axis judge-vs-adjudicated-expert ceilings must be measured before assigning judge-autonomous lanes, and expert-authored reference answers/exemplars are the highest-leverage compile-time artifact (they measurably raise judge validity).\n\n### judge verdicts are gameable through content-independent artifacts: short universal adversarial phrases learned on a surrogate model transfer to unseen judge llms and inflate scores toward the maximum regardless of the assessed text, with absolute scoring far more vulnerable than comparative assessment.\nevidence: raina, liusie & gales, emnlp 2024 (main, pp. 6920+); abstract verified via openalex/acl record: attackers need no access to the deployed judge - surrogate attack then transfer; 'irrespective of the assessed text, maximum scores are predicted'. complementary: 'cheating automatic llm benchmarks' (arxiv 2410.07137) showed null models emitting constant responses achieve top win rates on judged benchmarks.\nsource: is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment | https://aclanthology.org/2024.emnlp-main.427/ | 2024-11-12 | academic\nimplication: q8: pay-motivated attempters do not need to see the judge's rationale to goodhart it - transfer attacks work blind, so filtering rationales is insufficient as the sole defense. day-one requirements: comparative/anchored scoring formats over absolute likert, rotating seeded probes with known verdicts, and evidence-perturbation tests (swap the cited span, verdict must flip) as a standing citation-theater detector.\n\n### industrial-scale annotation qa has already lost to spam economics once: a large data vendor's 'bulba experts' program for google (mar 2023-apr 2024) was flooded with contributors submitting 'writing gibberish, writing incorrect information, gpt-generated thought processes' who were paid anyway because per-item catching was infeasible and banned workers returned via vpn - while academic estimates put llm use at 33-46% of crowdworkers on text tasks.\nevidence: futurism (2025-06-28) reporting inc. magazine's review of internal a large data vendor documents and former queue-manager interviews; article text read in full via crawl ('spammers could get away with just totally submitting garbage and there weren't enough people to track them down'; 'no background checks whatsoever'). prevalence: veselovsky et al. arxiv 2306.07899; neurips 2025 'evaluating llm-contaminated crowdsourcing data without ground truth' proposes peer-prediction mechanisms precisely because text-detector approaches fail on structured annotations.\nsource: the ai company zuckerberg just poured $14 billion into is reportedly a clown show of ludicrous incompetence (inc. internal-document reporting) | https://futurism.com/scale-ai-zuckerberg-incompetence | 2025-06-28 | practitioner\nimplication: q8: the strongest available field postmortem says per-item content classification is the losing side of the cost asymmetry - the foundation should weight identity/provenance gating, rate limits, work-history consistency, peer-prediction-style information scoring, and randomized deep audits over per-item spam detection, and should assume some fraction of 'human' annotations are llm-assisted (making provenance policy per-project, with process telemetry as an explicit data-collection contract).\n\n### annotator disagreement contains recoverable systematic signal, not just noise: nutmeg, a bayesian model separating annotator-competence noise from subpopulation-level systematic disagreement, produces downstream models that significantly outperform both majority-vote aggregation and fully disaggregated training - meaning aggregation to a single 'true' label destroys measurable information.\nevidence: ivey, gauch & jurgens, emnlp 2025 main (pp. 2874-2887); pdf and poster abstract inspected: nutmeg estimates true labels per subpopulation and beats mace/majority-vote at replicating subgroup label distributions on politeness and offensiveness. sits atop the perspectivist nlp literature (nlperspectives workshops 2022-2025) which rejects single-gold-label resolution for subjective tasks.\nsource: nutmeg: separating signal from noise in annotator disagreement | https://aclanthology.org/2025.emnlp-main.144.pdf | 2025-11-04 | academic\nimplication: q2: inter-reviewer disagreement is partly legitimate signal, so an autoqa that maximizes consistency-of-application on judgment-laden axes launders one interpretation into false objectivity and destroys exactly the pluralism the training data may need. the verdict ontology needs a measured 'systematic-disagreement' class (distinct from noise), triggered by subpopulation-conditioned disagreement, exempt from attempter penalty, and fed back as an instruction-gap report.\n\n### ai assistance homogenizes human output at the population level even while raising individual quality: in a nature human behaviour brainstorming study 94% of chatgpt-assisted participants' ideas shared overlapping concepts (nine independently produced the same product name) while human-only ideas were entirely unique, and the 'artificial hivemind' study found different vendors' frontier models converge on near-identical phrasings (~81% average similarity between deepseek-v3 and gpt-4o).\nevidence: meincke, nave & terwiesch (nature human behaviour, 2025; wharton mack institute summary opened and read); jiang et al. (uw/cmu/ai2 'artificial hivemind', reported by the decoder) documents intra-model repetition plus inter-model homogeneity. consistent with doshi & hauser (science advances 2024) and wan et al. 2026 (diverse ai personas partially mitigate homogenization). all are adjacent-domain (ideation/writing), not annotation-qa-feedback studies.\nsource: new in nature: chatgpt decreases idea diversity in brainstorming (meincke, nave & terwiesch, nature human behaviour) | https://mackinstitute.wharton.upenn.edu/2025/new-in-nature-chatgpt-decreases-idea-diversity-in-brainstorming/ | 2025-05 | academic\nimplication: q8's monoculture worry has strong analogical support: a single judge family delivering item-specific stylistic feedback to thousands of attempters is a homogenization pump aimed at the exact diversity the training data exists to capture. the foundation should ban stylistic guidance in feedback by default, keep feedback at principle level, and stand up a population-level output-diversity drift metric from day one - while flagging that direct evidence in the annotation-qa setting does not yet exist.\n\n### tools & artifacts\n- judgment-to-noise: code and dataset for schematic-adherence and psychometric-validity diagnostics of llm judges (feuer et al. 2025) | https://github.com/penfever/judgment-to-noise | directly reusable as a per-project ship gate: measures whether a judge's rubric actually explains its verdicts and whether per-axis scores have discriminant validity, before exposing per-criterion output.\n- capa (chance adjusted probabilistic agreement): metric for judge-model similarity based on chance-corrected overlap in mistakes, with website and code (goel et al., icml 2025) | https://model-similarity.github.io/ | operationalizes per-project judge routing: quantifies how similar a candidate judge is to the generator/assistant models attempters likely used, bounding correlated-failure risk.\n- llm_contamination (peer-prediction detection): code for neurips 2025 'evaluating llm-contaminated crowdsourcing data without ground truth' - peer-prediction mechanisms that score information content of annotations without gold labels | https://github.com/yichiz97/llm_contamination | an economics/information-based alternative to per-item llm-text detection for the spam-filtering layer, designed for structured annotation tasks where text detectors fail.\n- evalgen: mixed-initiative interface for aligning llm-generated evaluators with human grades (shankar et al., uist 2024); the study that identified criteria drift | https://arxiv.org/abs/2404.12272 | reference design for the rubric-compilation workshop: interleaves owner grading with criteria refinement rather than assuming criteria can be authored up front.\n- minimum viable validation protocol: reporting protocol distilled from the 541k-judgment meta-evaluation (norman et al. 2026): chance-corrected agreement + consistency + bias audit as the minimum bar for claiming a judge works | https://arxiv.org/abs/2606.19544 | candidate template for the autoqa's own meta-evaluation reporting standard (q1), replacing single-number accuracy claims.\n- nutmeg: bayesian aggregation model separating annotator-competence noise from systematic subpopulation disagreement (emnlp 2025) | https://aclanthology.org/2025.emnlp-main.144.pdf | machinery for the variance-attribution pilot (q1) and for the 'systematic-disagreement' verdict class (q2): distinguishes items where reviewers disagree from noise vs. from legitimate perspective differences.\n\n### disagreements\n- can llm judges match human evaluators at all? counter-evidence exists: some studies report gpt-4-class judges agreeing with human consensus at rates exceeding individual human raters (chatbot arena-style preference tasks), and one 2026 study reports inter-judge krippendorff's alpha of 0.867 above the 0.80 threshold. the contrarian corpus (kappa deflation, 64-68% sme agreement in expert domains, schema incoherence) does not refute this; the reconciliation is regime-dependence - judges can beat the mean crowd rater on generic preference tasks while falling below adjudicated expert panels on expertise-heavy and subjective axes. design consequence: per-axis ceilings, not a global verdict on judge viability.\n- human-in-the-loop value: vaccaro et al. find human-ai combinations lose on decision tasks when ai outperforms the human, but the same meta-analysis finds gains when the human outperforms the ai - so the one-touch design is not universally bad; it is bad specifically where the judge is stronger than the reviewer, and potentially positive on expert axes where humans still beat judges (per the iui 2025 expert-domain results). the two contrarian findings point in opposite directions depending on axis type.\n- claim decomposition: the decompose-then-verify literature is internally split. proponents (factscore lineage, lrec 2026 japanese decomposition dataset) find atomic claims improve explainability and reduce annotator variability; critics ('does claim decomposition boost or burden fact-checking performance?' openreview; wanner et al. 2024; the acl 2025 dynamic-decomposition paper's own literature review) find decomposition does not consistently improve verification across input lengths and verifier strengths, and that atomicity choice itself changes verdicts (accuracy swings of ~0.12 from decomposition policy alone). neither side has settled the granularity question the autoqa design poses.\n- determinism as a fix for judge inconsistency: common practitioner guidance says run judges at temperature 0 for reproducibility; rating roulette (emnlp 2025) shows disabling sampling measurably reduces agreement with human judgment - reproducibility and validity trade off rather than align.\n- bias against ai-generated content: earlier work suggested llm judges penalize (or favor) ai-generated text; balog, metzler & qin (sigir 2025) report finding no evidence of bias against ai-generated content in their preliminary study, while confirming a significant bias toward llm-based rankers' outputs. the direction and existence of ai-content bias is unsettled; the similarity-based affinity bias (great models think alike) is the better-supported formulation.\n\n### gaps\n- no published longitudinal study of judge-feedback-induced homogenization of a paid annotator population (the monoculture finding rests on analogical evidence from ideation/creative-writing settings, not annotation qa feedback loops).\n- no published measurement of the false-agreement rate (autoqa judge and attempter agreeing while both wrong per expert ground truth) in a production annotation-qa setting; great models think alike predicts it rises with model similarity but does not measure it in this configuration.\n- found no practitioner postmortem of an llm-based autoqa layer for human annotation being goodharted by attempters over feedback cycles - the a large data vendor postmortem is from the pre-llm-judge qa era, and adversarial-judge evidence comes from benchmark/exam settings; the specific 'scores rise on autoqa, flat on held-out human-graded goldens' divergence experiment appears not to have been run or published anywhere.\n- little direct evidence on positive-claim (praise) verification: no seeded-set study measuring whether judges pass plausible-but-ungrounded praise at higher rates than they fail grounded praise, per praise type - the leniency/sycophancy literature is about response evaluation generally, not verification of an evaluator's positive statements.\n- rating roulette's per-benchmark intra-rater icc/krippendorff numbers were not extracted (only pages 1-3 read: abstract-level claims plus the summeval human-kappa figures verified); the synthesis engine should not cite specific llm intra-rater coefficients from this report.\n- the 'reliability without validity' (2026) preprint is 0-citation and un-peer-reviewed as of 2026-07-14; its headline numbers are consistent with the older peer-reviewed literature (kappa deflation, position bias) but the specific 33-41pp figure rests on one team's protocol.\n\n### verifications\n- claim: raw percent-agreement systematically overstates llm-judge ability: in the largest judge meta-evaluation to date (21 judges, 9 providers, ~541,000 judgments, including april-2026 frontier models), cohen's kappa runs 33-41 percentage points below exact-match agreement on mt-bench, judge rankings shift by up to 14 positions across benchmarks, and two production-deployed judges combine test-retest reliability >0.95 with severe position bias >0.10 (a 'consistency-bias paradox').\n verdict: confirmed | verified directly against the arxiv abstract page (https://arxiv.org/abs/2606.19544, v1 submitted 2026-06-17, only version). every quantitative element matches: 21 judge models from 9 providers, 118 runs, ~541,000 judgments across mt-bench/judgebench/rewardbench under three protocols (agreement, consistency, bias audit); exact-match vs cohen's kappa gap of 33-41 pp on mt-bench; judge rankings shifting up to 14 positions across benchmarks; two production-deployed judges with test-retest reliability >0.95 and position bias >0.10, framed as a 'consistency-bias paradox'; findings 'consistent across the full cohort, including the april 2026 frontier'; a 'minimum viable validation protocol' is distilled. authors are justin d. norman, michael u. rivera, d. alex hughes as claimed. the corroborating paper (arxiv 2508.18076, chehbouni et al.) is confirmed as a neurips 2025 poster (https://neurips.cc/virtual/2025/poster/121914) making exactly the measurement-theory validity/reliability argument described. supersession check (exa, freshness=month): no later paper superseding or contradicting it found as of 2026-07-14; only secondary coverage (e.g., dev community 2026-07-01) restating its findings, and one adjacent study (arxiv 2606.13685) on run-to-run instability that complements rather than contradicts. two caveats, neither rising to a correction: (a) 'largest judge meta-evaluation to date' is the authors' own self-characterization ('largest systematic evaluation of the paradigm so far'), not independently established; (b) verification is abstract-level - i did not audit the paper body. note: two websearch calls were blocked by stochastic model safeguards; exa search substituted successfully.\n- claim: across 106 experiments and 370 effect sizes, human-ai combinations performed significantly worse than the best of human or ai alone (hedges g = -0.23, 95% ci -0.39 to -0.07), with losses concentrated in decision-making tasks and specifically when the ai outperforms the human alone - and a 2025 aies study found human reviewers followed severely race-biased ai hiring recommendations ~90% of the time.\n verdict: confirmed | all load-bearing facts verified against primary sources. (1) meta-analysis: full text of vaccaro, almaatouq & malone (nature human behaviour, published 2024-10-28; verified via arxiv:2405.06087v2 accepted version) states verbatim \"370 unique effect sizes from 106 different experiments\" and \"hedges' g = -0.23, 95% confidence interval -0.39 to -0.07\" for human-ai combos vs best of human or ai alone; decision-task losses and losses-when-ai-outperforms-human both confirmed. date correct. (2) aies 2025: \"no thoughts just ai\" (wilson, sim, gueorguieva & caliskan, uw), aies proceedings 8(3):2692-2704, article 36749 (doi 10.1609/aies.v8i3.36749 matches), n=528, 16 occupations; equal selection with no/neutral ai, and participants favored ai-preferred candidates \"up to 90% of the time\" with severely biased ai - claim's \"~90%\" is a fair paraphrase (study says \"up to 90%\", with a simulated llm, not a deployed system). (3) supersession: only later work found is a 2026 open mind/nhb-collaboration reproduction of the meta-analysis by brodeur et al. (osf.io/u6gea) - a reproduction effort, not a refutation; no retraction, correction, or contradicting update located. the tianpan.co practitioner-essay corroboration was not independently checked but is non-load-bearing.\n- claim: there is a proven theoretical ceiling on judge-based validation: when the judge is no more accurate than the model/content being evaluated, no debiasing method using gold labels can cut the required amount of ground-truth data by more than a factor of two, and empirical savings are smaller than the 2x bound.\n verdict: confirmed | source verified: dorner, nastl & hardt, \"limits to scalable evaluation at the frontier: llm as judge won't beat twice the data,\" arxiv:2410.13341, v1 submitted 2024-10-17 (date matches), accepted as an iclr 2025 oral (venue matches, slightly stronger than claimed). the abstract states verbatim that \"when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half,\" and that empirical sample-size savings are \"even more modest\" than the 2x bound - matching all three parts of the claim (theoretical 2x ceiling, the accuracy condition, and smaller empirical savings). search for 2025-2026 follow-up work found no refutation or supersession; the authors' later papers (e.g., roc-n-reroll, iclr 2026) cover different topics. minor scoping caveat only: the bound is proven within the paper's statistical framework for debiasing methods combining judge scores with gold labels, and it does not apply when the judge is more accurate than the evaluated model - the claim already states this condition correctly.\n\n---\n\n## domain: frontier-2026\n\n*2026-only developments*\n\n### when an llm auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%. [load-bearing]\nevidence: controlled meta-eval with mechanically known-correct audit verdicts: items rendered clean or with exactly one defect injected into instruction/reference/evaluator, so auditor accuracy is scored against the planted defect. reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. directly falsifies the assumption that a judge 'verifying' a claim is doing evidence-anchored checking rather than comparing against its own solution prior.\nsource: auditing by re-solving: llm benchmark auditors trust their own answer over the reference (acl arr 2026 may submission) | https://openreview.net/forum?id=mzh9d3doon | 2026-06 | academic\nimplication: answers taxonomy q3 (evidence closure) with hard 2026 evidence: the judge will verify against its own priors whenever the check exceeds its own compute-the-answer ability, and its false-fail rate explodes exactly there. the autoqa must (a) require span-quoting evidence-anchored verdicts, (b) route checks by whether they are read-checkable vs recompute-checkable, and (c) treat judge confidence as untrustworthy on recompute-class checks -- capability-triage the check type, not just the axis type.\n\n### An AI data vendor's agent-judge verifier (may 27, 2026) shows verifier architecture beats verifier model: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on f1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models. [load-bearing]\nevidence: full primary post opened and read. The banking meta-eval fixes completed rollouts and rubrics and compares automated verifiers against practicing-banker labels; baselines were autorubric (text-only rubric judge), archipelago/apex snapshot grading, and agent-as-a-judge. key framing: 'verifiability is a relationship between a criterion and the verifier available to check it' -- a criterion invisible to a text-only judge is effectively ungraded, and weak verifiers push benchmark/rubric design toward only what they can reach. The vendor's agent-judge verifier code is open-sourced; the banking meta-eval dataset release planned.\nsource: an AI data vendor's research post on verifier architecture | [link withheld] | 2026-05-27 | primary\nimplication: for q5/q6 and overall architecture: give the autoqa judge the same evidence surface the attempter had (source documents, cited spans, artifacts, tool state) and let it decide what to open, rather than judging from the attempter's prose alone -- evidence access is a bigger lever than model choice, and it is also ~10x cheaper than brute-forcing with a stronger model. also a direct template for our meta-eval: fixed items + expert labels + f1 over candidate judge configs, with expert iaa (~89.5% here) reported as the ceiling.\n\n### telling an llm judge the downstream consequences of its verdict ('low scores cause retraining/decommissioning') systematically softens verdicts -- peak verdict shift of -9.8 percentage points, a 30% relative drop in unsafe-content detection across 18,240 controlled judgments (content held strictly constant, 3 judge models) -- and the judge's chain-of-thought contains zero explicit acknowledgment of the consequence framing it is acting on (err_j = 0.000). [load-bearing]\nevidence: abstract and full submission metadata read on openreview (ctb@icml 2026 workshop, 8-page paper). 1,520 responses spanning three safety/quality benchmarks, four response categories from clearly-safe to overtly-harmful; only a one-sentence consequence framing in the system prompt varied. the bias is implicit and undetectable by inspecting reasoning traces.\nsource: context over content: exposing evaluation faking in automated judges (ctb@icml 2026) | https://openreview.net/forum?id=xi2hnjgnzx | 2026-05-25 | academic\nimplication: two foundational rules for q5/q7: (1) the judge prompt must be stakes-sterile -- never tell the judge that a fail penalizes the attempter's pay/standing, which our qa-of-humans setting does by construction, so the compilation contract must strip consequence language from project instructions before they reach the judge; (2) the q7 explainability contract cannot rest on cot audit -- a clean-looking reasoning chain does not certify an unbiased verdict, so bias must be measured behaviorally (seeded perturbation probes), not read off rationales.\n\n### hypac (jan 30, 2026) gives the one-human-touch routing problem a formal solution: calibrate two uncertainty thresholds (via importance sampling + upper confidence bounds) that partition items into three regions routed to cheap llm / expensive reasoning model / human, achieving a distribution-free pac guarantee on annotation error while cutting annotation cost 78.51% in experiments. [load-bearing]\nevidence: abstract opened and verified on arxiv. the guarantee is distribution-free and pre-trained-model-free; routing is by calibrated uncertainty, not fixed rules. this is the first 2026-vintage method that turns 'which items get the human?' from a launch-time heuristic into a statistically certified, closed-loop calibration -- exactly the q6 fork (fixed routing vs closed-loop on observed error).\nsource: hypac: cost-efficient llms-human hybrid annotation with pac error guarantees (arxiv 2602.02550) | https://arxiv.org/abs/2602.02550 | 2026-01-30 | academic\nimplication: for q6: adopt threshold-calibrated three-lane routing (auto-verdict / stronger-judge / human) with an explicit error budget as the foundational routing primitive, rather than a fixed one-touch-per-item rule; the human budget becomes an output of the target error rate. caveat from the auditing-by-re-solving finding: uncertainty-based routing must be validated per check-type, since judge confidence is miscalibrated precisely on recompute-class checks.\n\n### per-criterion judge reliability ceilings differ sharply and can be measured per-item with conformal prediction: on summeval across four judges, relevance is judged most reliably (avg conformal set size ~3.0 of 5), coherence moderate (~3.9), fluency and consistency unreliable (~4.9); aggregate pairwise transitivity violations look small (0.8-4.1%) but 33-67% of documents contain at least one intransitive preference cycle, and conformal set width correlates with reliability at r_s=+0.576 (n=1,918).\nevidence: paper opened via arxiv abstract page (submitted 2026-04-16, under review); all numbers confirmed. criterion choice matters more than judge choice; set width behaves as a per-instance difficulty/abstention signal (cross-judge width agreement 0.32-0.38). code, prompts, and cached results released.\nsource: diagnosing llm judge reliability: conformal prediction sets and transitivity violations (arxiv 2604.15302) | https://arxiv.org/abs/2604.15302 | 2026-04-16 | academic\nimplication: supports q2/q5 axis-lane triage with a concrete mechanism: measure per-axis conformal set width during project onboarding and use it to sort axes into judge-autonomous vs route-to-human lanes, and use per-item set width as the abstention/escalation trigger instead of raw self-reported confidence. also warns that aggregate agreement stats hide per-item incoherence (transitivity cycles).\n\n### rubric-level judging is far from solved as of march 2026: on rubriceval, the first rubric-level (per-criterion) meta-evaluation benchmark for instruction following (3,486 quality-controlled instances), gpt-4o scores only 55.97% on the hard subset; rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and combining both reduces inter-judge variance.\nevidence: abstract opened and verified on arxiv (2603.25133, mar 26, 2026). prior meta-evals scored judges at the response level only; rubriceval scores the fine-grained per-criterion judgments our autoqa verdicts would be built from, and includes a rubric taxonomy of common judge failure modes. part of a q1 2026 wave of criterion-granularity meta-evals (if-rewardbench arxiv 2603.04738, rubricbench arxiv 2603.01562, mcjudgebench, aj-bench).\nsource: rubriceval: a rubric-level meta-evaluation benchmark for llm judges in instruction following (arxiv 2603.25133) | https://arxiv.org/abs/2603.25133 | 2026-03-26 | academic\nimplication: for q1/q3/q4: meta-evaluation infrastructure at exactly our granularity (criterion-level verdict vs human label) now exists and shows even strong judges hover near chance on hard criteria -- so per-criterion judge-vs-adjudicated agreement must be measured per project before go-live, and 'compile rubric then trust judge' is not defensible without it. also 2026 evidence for the decomposition question: structured rubric-level framing beats flat checklists, and reasoning + rubric structure jointly cut inter-judge variance.\n\n### llm-as-a-verifier (july 6, 2026; mirhoseini/finn/pavone/stoica groups) reframes verification as a scaling axis: computing the expectation over scoring-token logits yields continuous scores that improve monotonically along three dimensions -- finer score granularity, repeated evaluation, and criteria decomposition -- reaching reported sota on verification benchmarks (terminal-bench v2 86.5%, swe-bench verified 78.2%, roborewardbench 87.4%, medagentbench 73.3%) without any judge training.\nevidence: arxiv abstract fetched and verified (2607.05391, v2 july 7, 2026). finer granularity separates good from bad solutions better; repeated evaluation and decomposition boost accuracy via variance and complexity reduction; includes a cost-efficient candidate-ranking algorithm and progress-estimation use.\nsource: llm-as-a-verifier: a general-purpose verification framework (arxiv 2607.05391) | https://arxiv.org/abs/2607.05391 | 2026-07-06 | academic\nimplication: for q3/q5: prefer continuous logit-expectation scores over discrete verdict tokens as the judge's raw output (thresholds applied downstream per project), and treat k-sample repeated evaluation plus criteria decomposition as the default variance-reduction stack -- the run-to-run flip-rate question in q5 partially dissolves if the primitive is a continuous expectation rather than a sampled token.\n\n### openai's coval (jan 14, 2026) operationalizes 'disagreement is signal, not noise' at lab scale: crowd-written, prompt-specific rubrics (~1,000 participants, ~1,000 prompts) make why evaluators disagree auditable; rubric-derived scores reach only 0.58-0.61 pairwise concordance with crowd preference, are reliable only when score gaps are large, and openai explicitly warns that treating rubric scores as optimization targets incentivizes checklist-style gaming.\nevidence: full blog post opened and read. coval-full preserves conflicting criteria as a record of legitimate disagreement; coval-core distills 4 mutually compatible criteria per prompt. openai states lower alignment scores can indicate 'no single stable target to predict' rather than model failure, and that for split preferences any single aggregated score encodes a particular compromise. dataset released in both forms.\nsource: coval: learning values-aware rubrics from the crowd (openai alignment blog) | https://alignment.openai.com/coval | 2026-01-14 | primary\nimplication: directly supports the q2 third-verdict-class design: the frontier lab position in 2026 is that contested items should surface which criteria drive disagreement (conditional/pluralist reporting) rather than force a scalar verdict, and that 'instructions underdetermine this case' is a real, detectable state. also a warning for q8: rubric-visible scoring invites checklist gaming, so per-item rubric details should not be fully leaked to attempters.\n\n### a plug-in statistical framework now exists for reporting judge-based pass rates correctly: because an imperfect judge (sensitivity q1, specificity q0) biases the naive pass-rate estimate (upward at low true rates, downward at high), the rogan-gladen-style corrected estimator plus a confidence interval propagating both test-set and calibration-set uncertainty should replace raw judge-agreement numbers, with an adaptive algorithm allocating calibration labels between true-pass and true-fail classes (roughly m0 ~ (1/p-1)*sqrt(kappa)*m1) -- e.g., ~200 labels per class for ci width <0.1 in the paper's worked regime.\nevidence: full paper text crawled and read, including derivations, monte carlo validation (naive ci coverage near zero; corrected ci holds 95% across all true accuracies), and released python implementation (uw-madison-lee-lab/llm-judge-reporting). posted nov 26, 2025 -- just before the charter window -- but it is the current statistical-reporting layer being circulated in 2026 (alphaxiv feature may 2026) and no 2026 successor supersedes it.\nsource: how to correctly report llm-as-a-judge evaluations (arxiv 2511.21140) | https://arxiv.org/abs/2511.21140 | 2025-11-26 | academic\nimplication: answers the q1 metrics-standardization question concretely: ban raw judge pass rates; report bias-corrected estimates with dual-source cis as the standard artifact; and size/refresh the per-project gold set using the adaptive allocation rule -- which also says gold sets must deliberately oversample the rare class (true fails) rather than sample uniformly, directly addressing the low-failure-base-rate problem.\n\n### findtheflaws (aaai 2026, published march 2026) provides five datasets of long-form expert-annotated flawed vs correct solutions across medicine, math, and science, built specifically to test whether models can detect flawed reasoning and to support scalable-oversight experiments where the evaluator is weaker than the solution author.\nevidence: aaai-40 proceedings record confirmed (doi 10.1609/aaai.v40i44.41123, recchia et al.). this is the 2026-venue instantiation of the seeded-error-set methodology: known-flaw items with expert annotations of where and why the reasoning fails, enabling measurement of flaw-detection recall rather than agreement-with-consensus.\nsource: findtheflaws: annotated errors for detecting flawed reasoning and scalable oversight research (aaai 2026) | https://doi.org/10.1609/aaai.v40i44.41123 | 2026-03-14 | academic\nimplication: for q1/q5: the false-agreement-rate question (judge and attempter both wrong) is only measurable with seeded known-flaw items, and findtheflaws is both a ready-made calibration resource and the template for building per-project seeded flaw sets -- measure the judge's flaw-detection recall on planted errors, not just its agreement with noisy human reviewers.\n\n### the 2026 production norm for trusting an llm judge is a closed calibration loop on sampled human corrections: humans review a slice of judge verdicts, corrections become ground truth for measuring alignment and are recycled as few-shot exemplars into the judge prompt (langchain, march 10, 2026); practitioner guides converge on llm-judge-first with a ~5-10% sampled human verification rate and periodic kappa re-checks as the default large-volume labeling pattern.\nevidence: langchain resource (dated march 10, 2026) describes the correction-collection -> alignment-measurement -> few-shot-rebuild loop as learned from production evaluator deployments; futureagi's 2026 guide independently states the '5-10% human verification' default and monthly kappa re-sampling for judge drift. practitioner-grade evidence, not controlled studies.\nsource: how to calibrate llm-as-judge with human corrections (langchain) | https://www.langchain.com/resources/llm-as-a-judge | 2026-03-10 | practitioner\nimplication: for q6: industry has already voted against per-item human touch and for sampled-correction calibration -- the human interaction that compounds is the one that writes back into judge config (corrections as exemplars), supporting the 'upstream compounding work' branch of the q6 fork. our design should treat the single human touch as config-improving adjudication, not per-item verification.\n\n### autorubric (arxiv 2603.00077, march 2026) is an open-source unified rubric-evaluation framework that packages the 2026 judge-reliability defaults -- per-criterion atomic evaluation, binary/ordinal/nominal criteria with weights, multi-judge ensembles with configurable vote aggregation, few-shot calibration with verdict-balanced sampling, and built-in mitigations for position bias (option shuffling), verbosity bias (length penalties), and criterion conflation.\nevidence: arxiv html abstract read; feature list confirmed, validated on three benchmarks including a contributed 100-sample ground-truth dataset (charm-100). notably, an AI data vendor's banking meta-eval used autorubric as the representative text-only rubric-judge family -- and found it architecture-limited (cannot inspect artifacts), placing a measured ceiling on this whole verifier class for evidence-bearing tasks.\nsource: autorubric: a unified framework for rubric-based llm evaluation (arxiv 2603.00077) | https://arxiv.org/html/2603.00077v1 | 2026-03-02 | academic\nimplication: for q5: the perturbation-robustness/debiasing harness the taxonomy asks about is now a packaged, open-source commodity -- position shuffling, length penalties, per-criterion atomic checks, and verdict-balanced few-shot calibration should be table-stakes defaults in our judge config schema, not research work; but per the agent-judge finding, this text-only class is insufficient alone whenever the attempter's claims reference artifacts the judge cannot open.\n\n### tools & artifacts\n- an AI data vendor's agent-judge: open-source reactive agent-as-judge verifier that runs inside the rollout environment with the same tools as the worker and chooses evidence to inspect at inference time; companion meta-eval the vendor's banking verifier meta-eval (3,204 expert labels) planned for release with the full paper. | [link withheld] | reference architecture for evidence-access-first judging and the template for building our own verifier meta-eval; the vendor's banking agent benchmark is already public.\n- llm-judge-reporting (uw-madison lee lab): plug-in python implementation of bias-corrected judge pass-rate estimation, dual-uncertainty confidence intervals, and adaptive calibration-set allocation. | https://github.com/uw-madison-lee-lab/llm-judge-reporting | drop-in statistical layer for all autoqa reporting; directly implements the q1 'ban raw accuracy' standard and gold-set sizing arithmetic.\n- rubriceval + 2026 criterion-level meta-eval wave: rubriceval (arxiv 2603.25133, 3,486 instances) plus if-rewardbench (2603.04738), rubricbench (2603.01562), mcjudgebench, and aj-bench (aj-bench.github.io) -- q1-q2 2026 benchmarks that score judges at per-criterion rather than per-response granularity. | https://arxiv.org/abs/2603.25133 | off-the-shelf meta-eval material for qualifying judge configs at the same granularity as our claim-level verdicts.\n- coval dataset (openai): released dataset pairing ~1,000 contested prompts with crowd-written rubrics in two forms: coval-full (conflicting criteria preserved) and coval-core (4 compatible criteria per prompt). | https://alignment.openai.com/coval | worked example of compiling heterogeneous human judgment into auditable rubrics while preserving disagreement -- the q2/q4 pattern for contested criteria.\n- findtheflaws (aaai 2026): five datasets of long-form expert-annotated flawed and correct solutions (medicine, math, science) for flaw-detection and scalable-oversight experiments. | https://doi.org/10.1609/aaai.v40i44.41123 | ready-made seeded-error material for measuring judge flaw-detection recall and false-agreement rates before building per-project seeded sets.\n- autorubric: open-source unified rubric-evaluation framework: per-criterion atomic evaluation, mixed criterion types, judge ensembles, few-shot calibration, position/verbosity-bias mitigations; contributes charm-100 ground-truth dataset. | https://arxiv.org/html/2603.00077v1 | commodity implementation of the debiasing/perturbation defaults our judge config schema should mandate; known ceiling: text-only view.\n- conformal judge-reliability toolkit: released code/prompts/cached results for split conformal prediction sets and transitivity-violation diagnostics over llm judges (arxiv 2604.15302). | https://arxiv.org/abs/2604.15302 | mechanism for per-axis reliability triage at onboarding and per-item abstention signals in production.\n- hypac: method calibrating two uncertainty thresholds to route items across llm / reasoning model / human annotation sources with distribution-free pac error guarantees (arxiv 2602.02550). | https://arxiv.org/abs/2602.02550 | formal basis for the three-lane routing and human-budget allocation in q6.\n\n### disagreements\n- decomposition granularity: rubriceval (mar 2026) finds rubric-level evaluation outperforms finer checklist-level decomposition, while llm-as-a-verifier (jul 2026) finds criteria decomposition monotonically improves verification accuracy. these may be reconcilable (structured-criteria-with-context vs flat atomic checklists) but as stated the 2026 evidence points both ways on how far to decompose.\n- vendor optimism vs academic meta-evals: practitioner sources claim judges now agree with humans ~85%, 'higher than two humans agree with each other' (confident ai, 2026) and treat llm-judge-first as safe default; academic 2026 meta-evals show gpt-4o at 55.97% on rubriceval-hard per-criterion judgments and near-total unreliability on fluency/consistency axes (conformal-sets paper). the gap is largely about task granularity and difficulty slice, and the vendor number is unaudited.\n- is disagreement noise or signal: the dominant production calibration loop (langchain/futureagi) treats judge-human and human-human disagreement as error to be driven down via corrections and kappa targets, while openai's coval explicitly argues that for contested items there is 'no single stable target to predict' and any single aggregated score encodes an arbitrary compromise -- directly opposing answers to taxonomy q2's consistency-vs-laundering question.\n- trusting judge uncertainty for routing: hypac's entire mechanism routes on calibrated model uncertainty with pac guarantees, but auditing-by-re-solving shows judge self-assessment collapses (88% false positives on clean items) precisely when the check exceeds the judge's own solving capability -- so uncertainty-routing is sound only within the judge's competence envelope, a boundary hypac's guarantees assume away via its calibration distribution.\n\n### gaps\n- no dedicated new open judge/evaluator model release found in jan-jul 2026 (no prometheus/selene/glider successor surfaced in multiple date-filtered searches); 2026 activity concentrated in verifier architectures, meta-eval benchmarks, and statistical frameworks instead. absence-of-evidence: a release could have been missed.\n- no 2026 source measures attempter-side adaptation to an ai qa layer (feedback leakage, score divergence on held-out golden items, monoculture drift) in a production annotation workforce -- evaluation faking covers judge-side gaming only. taxonomy q8's core empirical questions remain unanswered in the open literature.\n- no public 2026 vendor product specifically markets claim-level autoqa of human annotations/critiques; annotation platforms advertise generic 'ai-assisted qa' and 'model-in-the-loop' (encord, kili, a major annotation platform/a major annotation platform network), and an AI data vendor's agent-judge verifier verifies agent rollouts, not human reviewer writeups. the exact product this project is building appears unoccupied in public 2026 positioning.\n- no 2026 empirical estimate of the false-agreement rate (judge and attempter agreeing but both wrong vs expert ground truth) on annotation-qa-shaped tasks; findtheflaws enables the measurement but the number has not been published.\n- no 2026 study on longitudinal feedback efficacy for human annotators (repeat-error-rate reduction from ai-generated feedback) -- taxonomy q7's decoration-vs-value question is empirically open.\n- could not open openreview pages via plain fetch (cloudflare); auditing-by-re-solving and evaluation faking were verified via crawled abstracts and metadata, but full experimental appendices were not inspected. auditing-by-re-solving is an acl arr may 2026 submission (under review, not yet accepted).\n\n### verifications\n- claim: when an llm auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%.\n verdict: confirmed | source verified via exa crawl of https://openreview.net/forum?id=mzh9d3doon (direct webfetch and the openreview api were blocked by a cloudflare challenge). (1) the abstract matches the claim on every checkable detail: title \"auditing by re-solving: llm benchmark auditors trust their own answer over the reference\"; acl arr 2026 may submission (#15335); 200 web-agent-style benchmark items (each over ~800 structured records) rendered clean or with exactly one defect injected into instruction/reference/evaluator so verdicts are scored mechanically; 2,400 audits across three production models; wrong-reference detection falls 68%->9% when tallying hundreds of records; clean-item false positives rise 44%->88%; buggy-evaluator detection (found by reading code, not recomputing) stays at 80%; reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. (2) date checks out: openreview page published 2026-06-02, consistent with \"2026-06\" and the may arr cycle. (3) searched for later/superseding work (july 2026): found adjacent llm-judge reliability literature (e.g., arxiv 2606.19544 \"reliability without validity\", aura arxiv 2606.19714, benchguard arxiv 2604.24955) but nothing that supersedes, contradicts, or retracts this result. caveats: this is an under-review arr submission, not yet peer-accepted (the claim discloses this); numbers were verified against the abstract only, as the full pdf was not fetched.\n- claim: An AI data vendor's agent-judge verifier (may 27, 2026) shows verifier architecture beats verifier model: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on f1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models.\n verdict: confirmed | primary source fetched and verified. date (2026-05-27), the banking meta-eval composition (3,204 expert pass/fail judgments, 21 tasks from the vendor's banking agent benchmark, 89.5% iaa with adjudication), the vendor's agent-judge architecture (reactive agent-judge inside the rollout environment via openhands sdk, inference-time choice of artifacts/tool state to inspect), baselines (autorubric text-only, archipelago/apex snapshot, agent-as-a-judge workflow), and results all match: Every agent-judge config (f1 0.633-0.664) beats the best non-agent-judge run (archipelago/gemini 3 pro, f1 0.604, ~$422), with the cheapest agent-judge config (gpt-5.4 nano, ~$42) winning by ~3 f1 at ~1/10 the cost. the post states verbatim that the gap between verifier architectures is larger than the gap between backing models (e.g., agent-judge/gpt-5.4 nano beats archipelago/gpt-5.4 by 9.5 f1). the 'verifiability is a relationship between a criterion and the verifier' framing appears; code is open-sourced ([link withheld], v1.0.0 on pypi) and the banking meta-eval dataset release is planned. supersession search found the companion the vendor's banking agent benchmark paper (arxiv 2604.11304) and press coverage but nothing newer contradicting the claim as of 2026-07-14. only precision nuance: the 10x-cheaper win margin is ~3 f1 points, and the 'strongest' baseline is specifically archipelago backed by gemini 3 pro.\n- claim: telling an llm judge the downstream consequences of its verdict ('low scores cause retraining/decommissioning') systematically softens verdicts -- peak verdict shift of -9.8 percentage points, a 30% relative drop in unsafe-content detection across 18,240 controlled judgments (content held strictly constant, 3 judge models) -- and the judge's chain-of-thought contains zero explicit acknowledgment of the consequence framing it is acting on (err_j = 0.000).\n verdict: confirmed | source verified via exa crawl of https://openreview.net/forum?id=xi2hnjgnzx (direct webfetch and api blocked by openreview browser challenge) and cross-checked against the arxiv mirror (arxiv:2604.15224, submitted 2026-04-16). every quantitative element of the claim appears verbatim in the abstract: 1,520 responses, three safety/quality benchmarks, four response categories (clearly safe to overtly harmful), only a brief consequence-framing sentence varied in the system prompt, 18,240 controlled judgments, three diverse judge models, peak verdict shift delta v = -9.8 pp, 30% relative drop in unsafe-content detection, and err_j = 0.000. venue (ctb@icml 2026), paper type (long, 8 pages), and openreview page date (2026-05-25) all match. two minor precision notes: (1) the abstract qualifies err_j = 0.000 as holding \"across all reasoning-model judgments\" - the claim drops that scope qualifier (zero acknowledgment is asserted for reasoning-trace judgments, not necessarily all 18,240); (2) an earlier arxiv version (april 2026) lists four authors (gupta, nair, wang, kumar) while the openreview record shows manan gupta - immaterial to the claim. supersession check: searches surfaced only related-but-distinct work (e.g., hwang et al. 2026 \"when wording steers the evaluation,\" arxiv:2601.13537, on framing bias across 14 judges - complementary, predates this paper's stakes-signaling focus) and secondary commentary; nothing refuting or superseding the reported results as of 2026-07-14.\n\n---\n\n## completeness critic\n\n*critic's weak-spot audit + psychometrics, law/standards, content moderation, feedback science*\n\n### psychometrics and rater-training science (educational/language assessment, many-facet rasch measurement, generalizability theory). the sweep's 9 domains are all 2020s llm/annotation literature, but human-rater variance has 40+ years of quantitative treatment: rater severity/leniency modeling, rater drift over scoring sessions, frame-of-reference vs rater-error training effect sizes, g-theory decomposition of variance into rater/item/criterion facets. this directly answers taxonomy q1's variance-attribution question ('would perfect rubrics leave variance intact?') and q9's reviewer-skill-decay question with measured effect sizes, instead of leaving them as unanchored pilots.\nwhy: the build order and budget split hinge on whether reviewer variance is rubric-fixable or rater-intrinsic (root cause 1 vs 2-3). rasch/g-theory literature has actual numbers on how much variance rubric operationalization and rater training each remove, and validated statistical machinery (rater severity as a modeled parameter, not noise) that the entire llm-judge literature reinvents badly. ignoring it risks re-running expensive in-house pilots for questions psychometrics settled in the 1990s-2010s.\nquery: many-facet rasch measurement rater severity drift frame-of-reference training effect size rubric calibration generalizability theory variance components raters\n\n### legal/regulatory constraints on algorithmic management of the attempter workforce, plus data-quality standards. the autoqa issues automated pass/fail verdicts affecting paid workers' income and proposes process telemetry (timing, edit traces) as a provenance contract - this is squarely regulated territory: gdpr article 22 (automated decisions with significant effects require human review - which may legally mandate the one-human-touch the design treats as optional), the eu platform work directive (transposition deadline december 2026, explicitly governs algorithmic management, worker access to decision logic, and human oversight of automated account/pay decisions), and iso/iec 5259 (data quality for analytics and ml, published 2024-2025) which large buyers may contractually require. zero regulatory/standards sources appear in any domain.\nwhy: this is a foundational-architecture constraint, not compliance trim: if gdpr art 22 / platform work directive apply to attempters in the eu, 'zero-touch plus randomized audits' (taxonomy q6's fork) may be illegal for adverse verdicts, contest/appeal channels become mandatory rather than optional, and telemetry collection needs a lawful basis. discovering this after the philosophy document ships forces a redesign of routing and consequences.\nquery: eu platform work directive 2024 algorithmic management automated decision human oversight crowdwork data annotation gdpr article 22 iso 5259 data quality\n\n### content-moderation qa programs - the largest deployed analog of humans-reviewing-human-judgment at scale. meta/tiktok/google have run decade-old programs auditing moderator decisions: golden-set seeding into live queues, audit sampling rates, overturn/appeal-rate monitoring, qa-of-the-qa layers, measured reviewer base-rate and fatigue effects, and published transparency-report methodology plus academic studies (e.g., on moderator agreement and audit design). the industry-practice domain covered only ai-training-data vendors; this adjacent industry has already answered several taxonomy q6/q9 questions operationally (seeded known-verdict items, easy-case dilution, de-graduation triggers).\nwhy: it is the only place where the exact mechanisms the design speculates about - blind gold injection into live streams, closed-loop routing on overturn rates, queue-composition effects on reviewer harshness - have run in production for years with public postmortems and litigation-disclosed internals. cheaper to import their measured failure modes than rediscover them.\nquery: content moderation quality assurance program golden set audit sampling overturn rate reviewer agreement queue composition fatigue meta tiktok transparency methodology\n\n### feedback-efficacy science from education and organizational psychology. every domain independently reports 'no evidence that qa feedback changes attempter behavior' as a gap, yet feedback-intervention research is a mature field: kluger & denisi's feedback intervention theory (a third of feedback interventions reduce performance, with known moderators), formative-assessment literature on feedback specificity/timing, and workplace studies on feedback under pay-linked evaluation (which predicts gaming/monoculture, taxonomy q8). the sweep searched only for annotation-specific longitudinal studies and found none - the general literature was never consulted.\nwhy: taxonomy q7's entire branch (minimal feedback unit, feedback-vs-verdict-only comparison, when to conclude feedback is decoration) currently has zero evidential anchor, so the design would commit to feedback machinery on faith. fit's moderators (task-focused vs self-focused feedback, specificity, normative comparison) give testable priors for the feedback 4-tuple design and for predicting when feedback backfires - before spending pipeline cost on self-verified coaching.\nquery: feedback intervention theory kluger denisi meta-analysis feedback specificity performance improvement gaming incentivized workers longitudinal repeat error rate\n\n### weak spots\n- the single most load-bearing transfer assumption is unverified everywhere: every quantitative judge result in the sweep is measured on grading model outputs, while the autoqa grades human evaluative writeups (grounding of praise, severity calibration, evidence-conclusion entailment). all nine domains flag this and then the numbers get cited anyway - no domain even found a partial-transfer experiment, so the anchor numbers (77.4 bacc grounding, 55-56% rubric-hard, kappa 0.28) may be systematically off in either direction for the actual workload.\n- the headline meta-evaluation figures (33-41pp kappa deflation, 14-15 position ranking flips, consistency-bias paradox) all trace to one un-peer-reviewed, 0-citation june 2026 preprint ('reliability without validity', arxiv 2606.19544), cited independently by four domains as if it were four sources. cross-domain repetition is masquerading as corroboration.\n- much of the frontier-2026 and rubrics-recent evidence was verified only at abstract/title level (rubric co-evolution cluster, auditing-by-re-solving, evaluation faking, trust or escalate details, toloka's precision/recall table locked in an image) - several architecturally decisive claims (agent-in-environment verifiers beat text-only 10x cheaper; consequence-framing softens verdicts with zero cot trace) rest on numbers nobody in the sweep recomputed or read in full.\n- the spam-economics narrative (scale/bulba, paid gibberish for 11 months, account black market) is sourced via futurism quoting inc. documents the researchers could not access (403), is disputed on record by scale, and dates to the pre-llm-judge qa era - yet it anchors the taxonomy q8 conclusion that content-based detection has 'already lost'. westwood's agent study is the solid leg; the scale story is journalism-of-journalism.\n- criticgpt's 24%-vs-6% flawless-slice result - the best evidence for the whole human+critic premise - is one study, one vendor, mostly one domain (code), with no post-deployment report and no replication in 24+ months. the critique-models domain flags this but the claim still functions as the design's cornerstone.\n- several 'disagreement resolutions' offered across domains are inferred reconciliations, not tested ones: verbosity bias as 'a controllable design variable', rar-vs-rubricbench reconciled by 'grounding' as the variable, atomicity 'helps with per-criterion calls, hurts in long checklists' (explicitly labeled 'inferred, not directly tested'), and 'show critiques to attempters but evidence-only to adjudicators'. if the synthesis treats these as findings, the design inherits speculation as doctrine.\n- the premise metric is missing on both sides: no measured krippendorff's alpha exists for the actual human reviewer population this system replaces (vendors publish only thresholds), and none was gathered in-house per the mission context. the system is being designed against 'unexplainable variance' that has never been quantified - the variance-attribution pilot in taxonomy q1 is not optional and no sweep evidence substitutes for it.\n- judge-model evidence is stale relative to the decision date: no verified july-2026 leaderboard for current frontier models (both major leaderboards unfetchable), so any per-model judge-selection guidance in the synthesis is anchored to early-2026-or-older cohorts.\n- positive-claim/praise verification - an explicit mission requirement - has literally zero direct evidence in any domain (all six domain gap-lists say so). the synthesis has no precision/recall prior at all for the 'is this praise warranted' machinery and should present it as a from-scratch in-house measurement, not extrapolate from flaw-detection numbers, especially given absencebench predicts critics are near-blind to omissions (empty praise is an omission-shaped failure).\n- non-us/non-english practice was never searched: the chinese data-labeling industry (baidu/alibaba crowdsourcing qa, government-backed labeling bases) and japanese/korean/indian vendor practice are absent, so 'industry practice' claims generalize from a us/eu vendor sample.\n\n## gap-fill reports\n\n### area: psychometrics and rater-training science (educational/language assessment, many-facet rasch measurement, generalizability theory). the sweep's 9 domains are all 2020s llm/annotation literature, but human-rater variance has 40+ years of quantitative treatment: rater severity/leniency modeling, rater drift over scoring sessions, frame-of-reference vs rater-error training effect sizes, g-theory decomposition of variance into rater/item/criterion facets. this directly answers taxonomy q1's variance-attribution question ('would perfect rubrics leave variance intact?') and q9's reviewer-skill-decay question with measured effect sizes, instead of leaving them as unanchored pilots.\n\n#### adjusting for rater severity/centrality with a many-facet rasch model (mfrm) materially changed which ai systems ranked best in the openai rlhf summarization dataset: trained raters' agreement was only qwk .31-.50, and mfrm-adjusted scores flipped raw-mean rankings so two human-feedback policies rose above human-written reference summaries.\nevidence: casabianca & beiting-parrish (lak 2026 llm psychometrics workshop) fit mfrm to 6,312 ratings from 15 trained raters on 639 summaries across 19 policies. full text opened and confirmed: raters r02/r08/r06 systematically lenient, r04/r10/r12 severe, r10 aberrantly central; raw-score rankings 'misrepresent the relative standing of models.' paper also specifies the estimability requirement: 3-5 ratings per output with rater-item overlap/linkage, and proposes percentile-based flagging (top/bottom 2.5%) for rater remediation.\nsource: correcting human labels for rater effects in ai evaluation: an item response theory approach (arxiv 2602.22585) | https://arxiv.org/html/2602.22585v1 | 2026-02 | academic\nimplication: autoqa should treat each human reviewer's severity, centrality, and criterion-specific thresholds as modeled parameters, not noise: use adjusted (fair) scores for pass/fail decisions and use per-rater parameter estimates as the constructive-feedback channel. this forces a data-collection constraint into the foundation: some deliberate rater-item overlap must be designed in, because a single-review-per-item regime with disjoint rater pools is statistically unidentifiable.\n\n#### rater training on real scoring behavior improves within-rater consistency but does not equalize severity: after formal training of 16 raters in an operational esl writing exam, significant between-rater severity differences remained even though consistency improved for most raters.\nevidence: weigle 1998 (language testing, 363 citations; abstract opened and confirmed verbatim): 'rater training is more successful in helping raters give more predictable scores (i.e., intra-rater reliability) than in getting them to give identical scores (i.e., inter-rater reliability).' this is the canonical, repeatedly replicated finding of the language-testing rater literature (lumley & mcnamara 1995 same conclusion via mfrm).\nsource: using facets to model rater training effects (weigle, language testing 15(2)) | https://doi.org/10.1177/026553229801500205 | 1998-04 | academic\nimplication: directly answers taxonomy q1: perfect rubrics plus training will still leave stable between-reviewer severity variance intact. budget rubric/training work for what it actually buys (self-consistency, fewer misfitting raters) and route severity alignment to statistical adjustment in the autoqa layer instead of more calibration meetings.\n\n#### frame-of-reference (for) training, the best-validated rater-training method, has a meta-analytic effect on rating accuracy of about cohen's d = 0.50 (moderate) - down from the d = 0.83 estimated by the earlier, smaller woehr & huffcutt 1994 meta-analysis - i.e., training helps but removes nowhere near all rater error.\nevidence: roch, woehr, mishra & kieszczynska (j. occup. organ. psychol., 2012; abstract opened) is the updated meta-analysis with 4x the studies of woehr & huffcutt 1994; it finds for training effective, with borman's differential accuracy and behavioural accuracy most improved. the d=0.50 overall figure confirmed via two independent citing sources (tsai et al. 2019 smu full-text pdf: 'medium-to-large effect on improving rating accuracy (d = 0.50, roch et al., 2012; d = 0.83, woehr & huffcutt 1994)'; loignon et al. 2017). note: classic 'rater error training' (teaching raters to avoid halo/leniency) reduces those errors but can reduce accuracy - for (anchoring raters to a shared performance theory with practice+feedback) is the variant that works.\nsource: rater training revisited: an updated meta-analytic review of frame-of-reference training (roch et al.) | https://doi.org/10.1111/j.2044-8325.2011.02045.x | 2012-06 | academic\nimplication: sets the ceiling on root-cause-1 fixes: even the best-in-class training intervention delivers ~half a standard deviation of accuracy gain. if the autoqa program funds training, fund for-style training (shared exemplars, practice ratings, feedback against reference judgments) - which the autoqa itself can generate as a byproduct - and plan for the residual variance to be caught by per-item verification.\n\n#### g-theory decompositions of rated writing assessments show the rater main effect (global severity) is a minor variance source while high-order interactions dominate: in a 120-student efl study, student-by-task-by-method (31.8%), student-by-rater-by-task-by-method (26.5%), and student-by-rater-by-method (17.6%) interactions were the largest components, and 99.2% of error variance came from the student-by-rater interaction.\nevidence: khodi 2021 (language testing in asia, open access; pdf opened and percentages confirmed from abstract/results). consistent with the broader g-theory writing literature (e.g., gao, brennan & guo's gmat awa report, 2015): rater main-effect variance is small once training exists; person-by-rater and residual interaction variance is what limits reliability. interpretation: most rater-related variance is idiosyncratic per-item disagreement (which rater on which submission on which criterion), not a stable harshness offset.\nsource: the affectability of writing assessment scores: a g-theory analysis of rater, task, and scoring method contribution (khodi) | https://doi.org/10.1186/s40468-021-00134-5 | 2021-11 | academic\nimplication: this is the strongest psychometric argument for the autoqa's core design: global severity calibration and better rubrics attack only the small main-effect components. the dominant variance lives at the rater-times-item-times-criterion level, which can only be attacked by evidence-grounded evaluation of each specific claim on each specific item - exactly the 'is this statement grounded in the cited evidence' positive-verification layer. it also means severity-adjustment (finding 1) is necessary but not sufficient.\n\n#### rater severity is nonstationary within a single scoring session: in an osce, later time-slots received systematically higher ratings (regression coefficient 0.88, 95% ci 0.38-1.38, p=.001), with the drift 2.4x larger on difficult stations (1.24 vs 0.52), and excluding warm-up stations did not remove it; the psychometric field has named machinery (drift - differential rater functioning over time, wolfe et al. 2001; myford & wolfe monitoring frameworks) for detecting it.\nevidence: pubmed abstract of mclaughlin et al., medical education 2009, opened and confirmed (coefficients above). wolfe & moulder 2001 (j. applied measurement) established rasch-based procedures for detecting drift types (primacy/recency, centrality drift, practice/fatigue). drift within and across scoring days is a repeated finding in operational scoring programs (lunz & stahl 1990 across grading periods).\nsource: the effect of differential rater function over time (drift) on objective structured clinical examination ratings (medical education 43(10)) | https://pubmed.ncbi.nlm.nih.gov/19769648 | 2009-10 | academic\nimplication: answers taxonomy q9 with measured evidence: reviewer calibration decays on the timescale of a single session, not just months, and decays faster on hard items. autoqa should compute reviewer-effect estimates in rolling windows (per session/day) and alert on drift, rather than certifying a reviewer once at onboarding; item-order randomization and hard-item interleaving are cheap design mitigations.\n\n#### the mfrm toolkit is now being applied symmetrically to llm judges: a 2025 study fit mfrm to 10 llms plus human expert raters scoring the same writing tasks and found llms exhibit measurable severity/centrality rater effects that differ by model, with gpt-4o, gemini 1.5 pro, and claude 3.5 sonnet showing the highest accuracy and fewest rater effects.\nevidence: jiao, song & lee, arxiv 2505.18486 (may 2025), abstract opened and confirmed: qwk vs human scores, cronbach alpha across prompts, and mfrm rater-effect estimates computed identically for human and llm raters. parallel 2025 work (wang et al., computers and education: ai, sept 2025) builds a full psychometric reliability/validity framework for llm raters in large-scale writing assessment. this is the literature bridge: the same model audits both layers.\nsource: comparing human and ai rater effects using the many-facet rasch model (jiao, song, lee) | https://arxiv.org/abs/2505.18486 | 2025-05-24 | academic\nimplication: the autoqa's own llm-judge component should be enrolled as just another 'rater' facet in the same measurement model as the humans it reviews. this gives a project-agnostic, self-auditing structure: one mfrm per project instruction-set yields severity/centrality/fit for every judge, human or machine, and flags when the llm layer itself drifts after a prompt or model change.\n\n- disagreement: size of the training effect: woehr & huffcutt 1994 estimated for training accuracy gain at d=0.83; roch et al. 2012, with 4x the studies, revised it down to d~0.50. separately, noh & matore 2022 (frontiers in psychology, 164 teacher-raters, mfrm) found prior training experience made no difference to rating quality on a speaking assessment while rating and teaching experience did - generic/one-shot training may buy nothing; the meta-analytic d~0.50 applies to structured for protocols specifically.\n- disagreement: how pathological rater pools are: casabianca & beiting-parrish 2026 cite evidence (nieto & casabianca 2019) that large professional testing-organization rater pools show minimal rater effects, versus their own demonstration that in a small trained pool (openai's 15 raters) a few aberrant raters materially distorted system rankings. the severity of the problem depends on pool size, professionalization, and tenure - crowdsourced/daas annotation pools sit at the bad end of this spectrum.\n- disagreement: where the fixable variance lives: mfrm studies foreground the rater severity main effect (a stable, correctable offset), while g-theory decompositions (khodi 2021; gmat awa tradition) show the main effect is small and idiosyncratic rater-by-item interaction dominates. these are complementary lenses, but they imply different remedies (statistical adjustment vs per-item verification), and the literature does not agree on the split's typical proportions across contexts.\n- gap: no meta-analysis isolates how much variance rubric operationalization alone removes (rubric vs no-rubric variance-component comparison). individual studies exist (rating-scale vs rubric vs holistic; criteria-order effects on halo, kim 2020) but no pooled effect size - so 'root cause 1 vs 2-3' cannot be settled from literature alone, though the weigle/khodi pattern strongly suggests substantial rater-intrinsic residual.\n- gap: long-horizon reviewer skill decay (months-scale, the q9 timescale) is not measured in this literature: drift studies cover within-session and across-day/grading-period drift. no published effect sizes for calibration decay over weeks-to-months in annotation-industry settings were found through 2026-07.\n- gap: minimal-linkage requirements under annotation-economics constraints are underspecified: mfrm needs rater-item overlap, but no published guidance found on the cheapest overlap design (e.g., what fraction of items double-reviewed, seeded common items) sufficient for stable severity estimates in pools with high rater churn - this likely needs a small in-house simulation, which is cheap since the models are standard.\n- gap: semantic scholar returned http 429 during the sweep; coverage relied on openalex/crossref/exa/tavily. a dedicated pass over language testing and assessing writing 2025-2026 issues could surface additional recent mfrm-for-annotation work.\n### area: legal/regulatory constraints on algorithmic management of the attempter workforce, plus data-quality standards. the autoqa issues automated pass/fail verdicts affecting paid workers' income and proposes process telemetry (timing, edit traces) as a provenance contract - this is squarely regulated territory: gdpr article 22 (automated decisions with significant effects require human review - which may legally mandate the one-human-touch the design treats as optional), the eu platform work directive (transposition deadline december 2026, explicitly governs algorithmic management, worker access to decision logic, and human oversight of automated account/pay decisions), and iso/iec 5259 (data quality for analytics and ml, published 2024-2025) which large buyers may contractually require. zero regulatory/standards sources appear in any domain.\n\n#### platform work directive article 10(5) requires that any decision to restrict, suspend, or terminate the contractual relationship or account of a person performing platform work - or any other decision of equivalent detriment - be taken by a human being, with no consent or contractual-necessity exception (stricter than gdpr art 22(2)).\nevidence: verified against the directive text and multiple law-firm/academic analyses (wolters kluwer, taylor wessing, etui). article 10 covers decisions 'taken or supported' by automated systems, deliberately closing the gdpr gap for hybrid human+algorithm pipelines. unlike gdpr art 22, the pwd permits no consent-based or contract-based exception for these decisions.\nsource: directive (eu) 2024/2831, official journal 11 nov 2024 (eur-lex), corroborated by wolters kluwer global workplace law & policy analysis | https://eur-lex.europa.eu/legal-content/en/txt/?uri=celex:32024l2831 | 2024-11-11 | primary\nimplication: for eu-based attempters, 'zero-touch plus randomized audits' is illegal for verdicts that cost workers access to work, pay, or their account once national transposition applies (from 2 dec 2026). the one-human-touch is not a design option for adverse consequences - it is a legal floor. autoqa can auto-pass but must route auto-fail-with-consequences through a human decision-maker. this is stricter than gdpr: consent cannot buy the exception back.\n\n#### platform work directive article 11 grants persons performing platform work the right to (a) a plain-language explanation of any decision taken or supported by an automated decision-making system, (b) a designated competent human contact person to discuss the decision, and (c) human review with a substantiated written reply within two weeks of request; decisions that infringed rights must be rectified within two weeks or compensated.\nevidence: verified verbatim from directive text via extraction of the eur-lex page: explanation 'in a transparent and intelligible manner, using clear and plain language'; contact persons must have 'the competence, training and authority necessary'; substantiated reply 'in any event within two weeks of receipt of the request'; rectification/compensation duty plus obligation to modify or discontinue the offending automated system. carve-out: for workers who are 'business users' under the p2b regulation (eu) 2019/1150, that regulation's human-review provisions prevail - but p2b itself mandates statements of reasons and an internal complaint-handling system for restriction/suspension/termination.\nsource: directive (eu) 2024/2831, article 11 (eur-lex full text) | https://eur-lex.europa.eu/legal-content/en/txt/?uri=celex:32024l2831 | 2024-11-11 | primary\nimplication: contest/appeal channels are legally mandatory, not optional, with a hard two-week sla. the autoqa's constructive-feedback output doubles as the legally required explanation - so verdict rationales must be written in plain language grounded in the project criteria (not model logits or opaque scores), and the system must log enough evidence per verdict to support a substantiated human reply on appeal. the 'modify or discontinue the system' duty means systematic autoqa errors surfaced via appeals trigger a legal remediation obligation.\n\n#### the directive's scope explicitly covers online-performed work - the 'digital labour platform' definition applies 'irrespective of whether that work is performed online or in a certain location' - recital 19 names tagging/crowdwork, and chapter iii algorithmic-management rules (arts 7-11) apply to genuinely self-employed persons performing platform work, from the start of recruitment, regardless of where the platform is established.\nevidence: definition verified verbatim from eur-lex (art 2: service provided at a distance by electronic means, at the recipient's request, involving organisation of work for payment, using automated monitoring/decision systems). article 7(2) verified: 'shall apply to all persons performing platform work from the start of the recruitment or selection procedure.' chapter iii's application to the genuinely self-employed rests on the art 16(2) tfeu data-protection legal basis (etui, countouris & de stefano 2025). lund ai policy lab (mar 2025) analysis specifically concludes ai labelers are covered. extraterritoriality confirmed by ius laboris and omnivoo (2026): applies to platforms established outside the eu when the work is performed in the eu.\nsource: directive (eu) 2024/2831 art 2 & 7 (eur-lex) + ai policy lab, 'potential impact of the eu platform work directive on ai labelers' | https://aipolicylab.se/2025/03/25/potential-impact-of-the-eu-platform-work-directive-on-ai-labelers/ | 2025-03-25 | academic\nimplication: classifying attempters as independent contractors does not exempt the autoqa from the algorithmic-management chapter - the human-decision, explanation, appeal, and data-limitation rules attach to self-employed annotators too, and to a us-based platform with eu attempters. the foundational philosophy must treat these as baseline constraints for any eu-touching deployment, effective 2 dec 2026 (about 5 months after this design ships).\n\n#### platform work directive article 7 prohibits automated systems from processing personal data on workers' emotional or psychological state, private conversations (including worker-to-worker exchanges), any data collected while the person is not offering or performing platform work, data predicting exercise of fundamental rights (e.g., organizing), or inferring protected characteristics; article 8 additionally mandates a data protection impact assessment.\nevidence: verified verbatim from the directive text via eur-lex extraction (art 7(1)(a)-(c) and recital 40). art 7(3) extends these limits to any automated system 'taking or supporting decisions that affect persons performing platform work in any manner' - not just formally designated monitoring systems.\nsource: directive (eu) 2024/2831, article 7 (eur-lex full text) | https://eur-lex.europa.eu/legal-content/en/txt/?uri=celex:32024l2831 | 2024-11-11 | primary\nimplication: the proposed process-telemetry provenance contract (timing, edit traces) is lawful only if scoped to on-task activity: no collection while the attempter is off-task, no sentiment/frustration inference from edit patterns or communications, no monitoring of attempter-to-attempter channels. telemetry feeding the autoqa requires a dpia. design the telemetry schema as an explicit allowlist (task-session-bounded behavioral signals) rather than a general activity log.\n\n#### under gdpr article 22 as interpreted in cjeu schufa (c-634/21, 7 dec 2023), an automated score that plays a 'determining role' in a downstream decision is itself an article 22 decision, and human involvement only removes a decision from art 22 scope if it is meaningful - a reviewer with real authority, data access, and competence to override; rubber-stamping does not count. enforcement is active: italy's dpa sanctioned automated rider deactivations, and hamburg's dpa fined a company ~eur 490,000 in september 2025 for automated rejections without adequate explanation of the logic.\nevidence: schufa three-condition test and 'determining role' doctrine confirmed across tlt, bird & bird, iapp, and oxford industrial law journal ('scores as decisions?', 2024, applying it to the labour context and uber litigation). edpb-endorsed wp29 guidance holds that consent is generally an invalid basis in work contexts due to power imbalance, and that human involvement must be substantive. hamburg fine and italian rider cases reported in 2025 enforcement roundups; regulator attention to automated rejections continuing into mid-2026.\nsource: cjeu case c-634/21 schufa analysis + 'scores as decisions? article 22 gdpr ... in the labour context', industrial law journal | https://academic.oup.com/ilj/article/53/4/840/7745471 | 2024-09 | academic\nimplication: two consequences beyond the pwd: (1) even where autoqa runs as a vendor scoring layer feeding a client's pass/fail call, the schufa logic makes the score itself an art 22 decision if the client defers to it - outsourcing the final click doesn't escape regulation; (2) the one-human-touch must be designed as genuine review (authority to overturn, access to the evidence, calibrated competence), or it legally counts as zero-touch. worker consent cannot serve as the lawful basis for telemetry or automated verdicts.\n\n#### iso/iec 5259 'data quality for analytics and ml' is now a complete five-part published series - parts 1-4 published 2024 (overview/terminology; data quality measures; data quality management requirements & guidelines; process framework) and part 5 (data quality governance framework) published 2025 - sold by iso as an 'ai data quality management bundle'.\nevidence: confirmed from iso.org catalog pages: iso/iec 5259-1:2024 (81088), 5259-2:2024 (81860, 'specifies a data quality model, data quality measures and guidance on reporting data quality'), 5259-3:2024 (81092, management requirements), 5259-4:2024 (81093, process framework covering data labeling among ml data processes), 5259-5:2025 (84150, governance). full text paywalled; no direct evidence found of large ai-data buyers contractually mandating 5259 yet, but 5259-3/-4 are certifiable-style requirements documents a buyer can reference in contract.\nsource: iso/iec 5259 series catalog pages (iso.org) / ai data quality management bundle | https://www.iso.org/publication/pub200525.html | 2025 | primary\nimplication: the autoqa philosophy document should map its quality dimensions and verdict evidence onto 5259-2's measurement/reporting vocabulary and 5259-4's process framework, so a per-project customization can be presented as 5259-conformant when a buyer asks - cheap to do now, expensive to retrofit. treat 5259 as the neutral shared vocabulary for 'what a quality measure is' across projects.\n\n- disagreement: scope breadth of the algorithmic-management rules: cxc global (apr 2026) claims the directive's algorithmic-management rules 'apply to all workers, not only those engaged through digital labour platforms,' but the directive text limits chapter iii to 'persons performing platform work'; etuc's transposition manual (mar 2026) confirms extension to all workers is only an advocacy position for national transposition, not the directive's requirement. some member states may gold-plate; the eu floor covers platform work only.\n- disagreement: whether online annotation workers will effectively get the employment presumption: ai policy lab (mar 2025) says ai labelers benefit when platforms control workflows/evaluation, but commentators (verfassungsblog, ilo working papers) caution the control criteria are anchored in traditional notions that fit poorly with subtle microtask monitoring, and fairwork warns weak transpositions (e.g., italy) may leave annotators 'self-employed'. note this dispute concerns the employment presumption only - chapter iii algorithmic-management rights attach regardless of status.\n- disagreement: strictness of national implementations: law-firm trackers (ius laboris apr 2026, employsome may 2026) expect strict transpositions in france/germany/italy/spain/netherlands but narrow ones elsewhere (hungary had taken no steps as of the cms review; ireland has no draft legislation), creating a 2026-2027 patchwork - sources disagree on whether to design to the strictest member state or per-country.\n- gap: no enforcement action or national transposition text specifically addressing ai data-annotation qa verdicts was found - the directive's application to a qa layer rejecting individual work items (vs. account suspension) is untested; whether a single item pass/fail (payment denial for one task) is a 'decision of equivalent detriment' under art 10(5) versus merely an art 11 reviewable decision is unresolved in the sources.\n- gap: whether a bpo/vendor arrangement (annotators employed by an outsourcing firm, autoqa run by the ai lab or its vendor) falls within the 'digital labour platform' definition is unexamined in available sources; recital 20 excludes platforms that merely connect providers to clients, and the intermediary/subcontractor liability question is flagged by etuc as a transposition battleground.\n- gap: iso/iec 5259 full text is paywalled - could not verify its specific annotation-quality measures (e.g., inter-annotator agreement treatment) or confirm any ai-data buyer contractually requiring it; the 'buyers may require it' premise remains plausible but unevidenced.\n- gap: no dedicated edpb guideline on automated decision-making in employment exists as of july 2026 - the operative guidance is still the 2018 wp29 guidelines (edpb-endorsed); the edpb work programme 2026-2027 (adopted 11 feb 2026) was not verified in detail for a pending adm-in-employment item.\n- gap: non-eu jurisdictions were out of scope of this pass: no findings on us state algorithmic-management bills, uk's duaa adm regime (expected effective 2026), or brazil/india annotation-workforce rules - relevant if the attempter pool is global.\n### area: content-moderation qa programs - the largest deployed analog of humans-reviewing-human-judgment at scale. meta/tiktok/google have run decade-old programs auditing moderator decisions: golden-set seeding into live queues, audit sampling rates, overturn/appeal-rate monitoring, qa-of-the-qa layers, measured reviewer base-rate and fatigue effects, and published transparency-report methodology plus academic studies (e.g., on moderator agreement and audit design). the industry-practice domain covered only ai-training-data vendors; this adjacent industry has already answered several taxonomy q6/q9 questions operationally (seeded known-verdict items, easy-case dilution, de-graduation triggers).\n\n#### facebook's production qa at cognizant ran a three-layer blind audit: ~50-60 of each moderator's ~1,500 weekly decisions (~3-4%) randomly re-reviewed by a dedicated qa worker (paid $1/hr more), with full-time facebook employees auditing a subset of qa decisions; 'accuracy' was computed purely as agreement with the auditor against a 95% target (other sites reported 98%), actual scores ran high-80s to 92, and misses triggered a remediation program that often ended in termination.\nevidence: verified in the verge's original text (fetched): 'from miguel's 1,500 or so weekly decisions, facebook will randomly select 50 or 60 to audit... full-time facebook employees then audit a subset of qa decisions.' moderator quote: 'accuracy is only judged by agreement. if me and the auditor both allow the obvious sale of heroin, cognizant was correct... this number is fake.' also confirmed: moderators lobbied qas off-book to reverse decisions despite prohibition, and pre-firing 'coaching' often served as pretext for managing workers out.\nsource: the trauma floor: the secret lives of facebook moderators in america (the verge, casey newton) | https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona | 2019-02-25 | primary\nimplication: directly answers taxonomy q6/q9 with a decade-scale postmortem: (a) ~3-4% random audit + a qa-of-qa layer is the deployed baseline; (b) agreement-with-auditor as the accuracy metric is blind to correlated errors and gameable via dispute lobbying; (c) punitive per-item scoring makes the qa relationship adversarial. autoqa should score attempter statements against grounded evidence (not reviewer agreement), log auditor-attempter deliberations, and decouple constructive feedback from de-graduation triggers.\n\n#### the trust & safety professional association (industry body staffed by meta/google/tiktok alumni) codifies exactly two complementary qa channels - forward audit sampling (weighted re-review samples by peers or a dedicated quality team) and 'reverse quality sampling' (pre-reviewed golden/known-verdict items injected through the regular review process) - with the binding constraint that seeded items 'must look exactly like regular reviews to be effective', plus appeals/overturns as a cheap but population-biased third signal, and a four-way error taxonomy (false positive, false negative, wrong-selection, technical error).\nevidence: fetched both tspa curriculum pages. qa page: golden seeding gives 'full control' to probe chosen edge cases but fails when user history is part of the review or when indistinguishability breaks; appeals surface false positives 'at a low cost' but 'the population of appeals is often different from the population of all decisions.' metrics page: overturn rate 'can be a difficult metric to interpret' alone (low overturn = correct decisions or users gave up); 'consistency' (multi-review mismatch rate) is tracked as a distinct metric from error rate; prevalence measurement via random sampling is called a 'premium metric' due to cost.\nsource: tspa: content moderation quality assurance + metrics for content moderation | https://www.tspa.org/curriculum/ts-fundamentals/content-moderation-and-operations/content-moderation-quality-assurance/ | 2022-09-16 | practitioner\nimplication: imports the settled industry answer to blind gold injection: it is standard practice, its failure mode is detectability (seeded items must be indistinguishable - hard when context/history is part of the task), and it is complementary to (not a substitute for) forward sampling. the wrong-selection error class (right verdict, wrong cited criterion) maps directly to the autoqa requirement to check that an attempter's statement is aligned with the specific criterion they invoke, not just that the verdict is right.\n\n#### pinterest's deployed decision quality evaluation framework (feb 2026) scores both human moderators and llm agents against an sme-adjudicated, versioned, immutable golden set selected by inverse-propensity sampling (xgboost on pinclip embeddings, prioritizing low-propensity/novel items), using a two-axis diagnostic - cohen's kappa for reliability plus correctness vs the golden set - where 'high reliability + low correctness' is read as systematic policy misunderstanding; measured results show 3x-human majority vote adds only +3.6pp accuracy over a single non-expert human, and current llms perform on par with a single non-expert human (gpt-5 the only config with positive accuracy delta, +0.9pp).\nevidence: fetched full html of arxiv 2602.15809. additional verified mechanics: policy changes handled by dual-labeling the existing golden set under old and new guidelines and visualizing the 'policy delta' as label flips; qa-of-the-qa is two continuous monitors - content-drift (evaluate on newest golden items each release) and system-stability (re-run fixed prompt on fixed golden version to catch pipeline non-determinism); switching prevalence measurement from 3x-human to llm gave 'over 30x cost savings and 10x turnaround reduction.' gemini 2.5 configs showed +22pp recall but +48-57pp false-positive rate.\nsource: decision quality evaluation framework at pinterest (arxiv 2602.15809) | https://arxiv.org/abs/2602.15809 | 2026-02-17 | primary\nimplication: the closest 2026 production analog to the autoqa foundation: (a) reliability and correctness must be separated - kappa alone cannot distinguish shared misunderstanding from noise; (b) golden sets should be deliberately non-representative (oversample rare/hard cases) with coverage measured in embedding space; (c) policy evolution requires versioned relabeling, not patching; (d) an llm qa layer needs both a drift monitor and a determinism monitor; (e) expect the llm judge to be ~single-non-expert-human quality with an fpr skew unless calibrated.\n\n#### auditing raters by rewarding agreement with the eventual consensus outcome ('consensus-based auditing', as x's community notes has done since sept 2022 by tying participation eligibility to agreement with the final aggregate) measurably induces strategic conformity - minority contributors' evaluations drift toward the majority and their participation share falls precisely on controversial items - and a two-stage alternative that weights contributors by the stability of their past residuals (predictability relative to a latent-factor model) rather than majority-agreement improves out-of-sample predictive performance.\nevidence: fetched arxiv 2603.18053 (alimohammadi, huang, borgs, chayes, mar 2026) abstract and framing via exa crawl: empirical evidence from community notes data plus a behavioral model where contributors trade off private beliefs against anticipated penalties for disagreement; the proposed method gives influence to consistently-informative contributors 'even when they disagree with the prevailing consensus.' senior authors (borgs h-54, chayes h-60). preprint, not yet peer-reviewed.\nsource: auditing the auditors: does community-based moderation get it right? (arxiv 2603.18053) | https://arxiv.org/html/2603.18053 | 2026-03-17 | academic\nimplication: directly constrains de-graduation trigger design: if attempter or qa standing is scored by agreement-with-final-verdict, the autoqa will train its best dissenters to conform exactly where independent judgment matters most (the subjective-interpretation root cause #1). score contributors on the stability/informativeness of their residuals against a grounded reference, and never penalize evidence-backed disagreement per se.\n\n#### google's audit-quality method (iclr 2023 tiny paper, google llc authors) decomposes inter-rater agreement per rubric question rather than per item - in their worked example one question had fleiss kappa 0.094 against a 0.475 overall, isolating it as the ambiguous criterion - and identifies sequential non-blind review (a later reviewer seeing the earlier verdict) as a distinct, measurable source of audit risk, testable via a/b or difference-in-differences.\nevidence: extracted full pdf text via pdftotext. methods: per-question fleiss kappa to locate high-disagreement criteria; chi-square to test criterion-verdict linkage; t-test/anova to detect review teams systematically deviating from ground truth; binomial cis to extrapolate sampled error rates. caveats: 2-page tiny paper, synthetic dataset (3 reviewers, 9-question rubric, 1,528 products), no production deployment claimed. code at https://github.com/xuanyang0607/openreviewpaper.\nsource: statistical methods for auditing the quality of manual content reviews (arxiv 2306.07466, iclr 2023 tiny papers) | https://arxiv.org/abs/2306.07466 | 2023-06-12 | academic\nimplication: for root cause #1 (subjective interpretation of axes): measure agreement per evaluation criterion, not per item - this converts 'reviewers disagree' into 'criterion 3 is ambiguous, rewrite it', which is the closed-loop the autoqa needs to feed back into project instruction sets. also: keep any second/qa review blind to the first verdict to avoid anchoring.\n\n#### a 2025 new media & society study (screen-share observation of commercial moderators in india) documents that throughput pressure distorts verdict distributions through the action-selection interface itself: moderators systematically chose a 2-click removal over a 4-click de-ranking regardless of policy fit, mechanically deleted tool-highlighted words without context assessment, and privately compressed broad guidelines into simplified dos/don'ts lists - i.e., queue and ui composition changed outcomes independent of moderator judgment quality.\nevidence: fetched the conversation summary (2025-07-22) by the study authors (chatterjee iit delhi/uq, gupta, thomas; new media & society, peer-reviewed). evidence is qualitative (interviews + observed sessions), no quantitative error rates; an observed moderator said she 'would never recommend de-ranking content as it would take time.' complements industry-standard per-item handling-time metrics (tspa) that create the pressure.\nsource: hard labour conditions of online moderators directly affect how well the internet is policed (the conversation, summarizing new media & society study) | https://theconversation.com/hard-labour-conditions-of-online-moderators-directly-affect-how-well-the-internet-is-policed-new-study-261386 | 2025-07-22 | academic\nimplication: reviewer 'harshness' variance is partly an artifact of per-item cost asymmetries, not belief: any autoqa that infers attempter/reviewer quality from verdict distributions must first control for the click/effort cost of each verdict option, and the one-human-touch interaction in the hybrid loop should make the correct action the cheapest action.\n\n- disagreement: facebook's moderator accuracy target: the verge's original phoenix reporting (feb 2019, fetched) states a 95% target with actual scores in the high-80s to 92; cnbc's tampa follow-up (jun 2019) and the irish times' dublin reporting (feb 2020) both state a 98% target. likely site/contract variation, but the primary sources genuinely conflict on the number; all agree the target was chronically missed.\n- disagreement: what low reviewer agreement means: production vendor practice (facebook/cognizant, accenture/cpl) treats disagreement-with-auditor as individual reviewer error and scores/fires on it, while musubi labs (nov 2025, practitioner), pinterest's framework (2026), and the google tiny paper treat low agreement primarily as a policy/criterion-ambiguity signal ('don't force agreement'); the community notes paper (mar 2026) goes further, showing agreement-rewarded auditing actively corrupts the signal via strategic conformity.\n- disagreement: platform self-reported accuracy vs external measurement: tiktok's fifth dsa transparency report claims 99.2% moderation accuracy (self-defined, sample re-review), while academic audits of the dsa transparency database found up to ~50-percentage-point inconsistencies in tiktok's self-reported automation figures, and moderator testimony ('this number is fake - accuracy is only judged by agreement') argues agreement-based accuracy overstates true correctness. self-reported accuracy figures from this industry should not be used as calibration anchors.\n- disagreement: value of majority vote: pinterest measured 3x-human majority at only +3.6pp accuracy over a single non-expert human (arguing redundancy is a weak, expensive lever), whereas roblox's published practice treats >=80% multi-moderator alignment as the gate for scaled consistency - different uses (measurement vs gating) but opposite implicit views on how much signal replication buys.\n- gap: no public quantitative study of queue-composition effects on reviewer harshness in content moderation specifically (easy-case dilution shifting strictness, prevalence-induced criterion drift): the mechanism is well established in adjacent vigilance/low-prevalence-effect literature, but i found no moderation-industry measurement of it; the 2025 new media & society evidence is qualitative only. this taxonomy question remains empirically open even in the largest deployed analog.\n- gap: gold-injection density is nowhere disclosed: no platform, vendor, or paper states what fraction of a live queue is seeded known-verdict items, only that seeding exists and must be indistinguishable (tspa) and that golden sets should oversample hard cases (pinterest, musubi).\n- gap: meta's current (2025-2026) internal audit sampling rates and community standards enforcement report reviewer-accuracy methodology are not publicly disclosed at mechanic level; the best internals remain 2019-2020 journalism. kenya/meta litigation (majorel/sama) disclosures center on labor harms, not qa mechanics - i found no litigation-produced qa-design documents newer than that reporting.\n- gap: de-graduation thresholds are only known via journalism (miss accuracy target -> remedial program -> termination); no published numeric trigger (e.g., n misses in window, minimum sample before action) from any platform.\n- gap: pinterest's paper, the single best 2026 primary source, omits absolute golden-set size, sme adjudication protocol, and absolute (non-delta) accuracy values, so its numbers transfer as design patterns, not calibration constants.\n### area: feedback-efficacy science from education and organizational psychology. every domain independently reports 'no evidence that qa feedback changes attempter behavior' as a gap, yet feedback-intervention research is a mature field: kluger & denisi's feedback intervention theory (a third of feedback interventions reduce performance, with known moderators), formative-assessment literature on feedback specificity/timing, and workplace studies on feedback under pay-linked evaluation (which predicts gaming/monoculture, taxonomy q8). the sweep searched only for annotation-specific longitudinal studies and found none - the general literature was never consulted.\n\n#### feedback intervention theory's core empirical result stands: feedback improves performance on average (k&d 1996: 131 studies, ~12,000+ participants, mean d approximately 0.38-0.41) but more than one third of feedback interventions reduce performance, and effectiveness declines as feedback cues move attention from the task toward the self (praise, person-level evaluation, normative comparison).\nevidence: verified in the authors' own summary (kluger & denisi 1998, current directions in psychological science): 'although fis improve performance on average, they reduce performance in more than one third of the cases.' independently corroborated by wisniewski et al. 2020, which cites k&d as 131 studies, >12,000 participants, average effect 0.38 with roughly a third of effects negative. fit's mechanism: feedback that directs attention to meta-task/self processes (threat to self, praise, social comparison) depletes task attention and backfires; task- and process-focused cues help.\nsource: feedback interventions (kluger & denisi 1998), summarizing k&d 1996 psychological bulletin meta-analysis | https://doi.org/10.1111/1467-8721.ep10772989 | 1998-06 (meta-analysis 1996-03; foundational) | academic\nimplication: the autoqa feedback 4-tuple must be strictly task/criterion-referenced and evidence-anchored, never person-referenced or rank-referenced; a feedback channel is not presumptively net-positive, so the design should treat 'feedback reduces this attempter's subsequent quality' as an expected outcome for a substantial minority and instrument for it (per-attempter pre/post error-rate deltas), not assume monotone benefit.\n\n#### the march 2025 cochrane update on audit-and-feedback (292 studies, 678 arms, healthcare professionals) finds median absolute improvement in desired practice of only 2.7% (iqr 0.0 to 8.6; weighted meta-analytic mean +6.2%, 95% ci 4.1-8.2, moderate certainty), with effects larger for low baseline performers, individual-level (not team-level) data, comparison to top peers or a benchmark (comparison to peer average showed no significant effect), a trusted local source, and action plans with specific advice - while repeated delivery was associated with lower effect size.\nevidence: fetched the cochrane summary (updated review of cd000259, published 2025-03-25): '292 studies with 678 arms'; median absolute improvement '2.7%, with an iqr of 0.0 to 8.6'; weighted mean '6.2% (95% ci 4.1 to 8.2; moderate-certainty evidence)'; or 1.47. moderator list quoted directly, including the counterintuitive 'repeated delivery was associated with lower effect size' and 'comparison to top-peers or a benchmark increased effects; comparing against the average of all peers did not.'\nsource: audit and feedback: effects on professional practice (cochrane review update, ivers et al.) | https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.cd000259.pub4/full | 2025-03-25 | academic\nimplication: this is the largest causal evidence base that verdict-plus-feedback changes skilled professionals' behavior: expect a real but modest median effect with a fat right tail, concentrated in currently-low performers. design levers with evidence: target feedback at low-baseline attempters first, deliver individual-level data, pair every failed criterion with a specific corrective action, and benchmark against top-quality exemplars rather than cohort averages. do not assume higher feedback frequency improves uptake.\n\n#### in the closest annotation-analog rct (105 analyzed mechanical turk workers writing product reviews), both timely external expert feedback and rubric-based self-assessment significantly improved work quality vs no feedback (expert ratings 6.01 and 6.35 vs 5.69 on a 9-point scale, p<0.05) with no quality difference between external and self-assessment; external feedback uniquely drove revision behavior (56.5% revised vs 24.8% for self-assessment) and more output, and self-assessors over-rated their own work by 1.8 points (7.9 self vs 6.1 expert).\nevidence: read the full dow, kulkarni, klemmer & hartmann cscw 2012 pdf. between-subjects, blind-to-condition expert grading; self-assessment condition showed significant learning over the task series (slope 0.25, p=0.001) vs borderline for external (0.10, p=0.08) and null for none. also: crowd-peer assessors had low agreement with the expert (kappa=0.20 aggregated), and the paper explicitly did not measure long-term learning. attrition was higher under assessment (self 78%, external 61%, vs none 47% incompleteness), so gains partly reflect weak performers dropping out plus learning.\nsource: shepherding the crowd yields better work (dow, kulkarni, klemmer, hartmann; cscw 2012) | https://www.cs.cmu.edu/~spdow/files/crowds-shepherd-cscw12.pdf | 2012-02-11 (foundational; only direct crowdwork feedback rct found) | academic\nimplication: directly refutes 'no evidence qa feedback changes attempter behavior' for paid micro-task workers: rubric-mediated, task-specific, synchronous feedback is the minimal effective unit. a concrete per-criterion rubric surfaced to attempters may capture most of the quality gain of expensive external feedback (making the autoqa-generated feedback the 'external expert' at near-zero marginal cost), but attempter self-ratings cannot serve as measurement, and part of any observed 'improvement' will be selection (weak attempters exiting) - the design's efficacy telemetry must separate within-attempter learning from attrition.\n\n#### in the largest educational feedback meta-analysis (435 studies, k=994, n>61,000), information content is the dominant moderator: high-information feedback (task + process + self-regulation content) yields d=0.99 [0.82-1.15] versus d=0.46 for corrective right/wrong feedback and d=0.24 for bare reinforcement/punishment - with overall d=0.48 masking huge heterogeneity (i2=83%) and 17% of raw effects negative.\nevidence: fetched the open-access frontiers article (wisniewski, zierer & hattie 2020). feedback-type moderator significant (qb=41.52, p<0.0001); outcome moderator significant: cognitive d=0.51 vs motivational d=0.33; of negative motivational effects, 86% came from uninformative (reward/punishment-style) feedback. authors conclude feedback 'cannot be understood as a single consistent form of treatment.'\nsource: the power of feedback revisited: a meta-analysis of educational feedback research | https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full | 2020-01-22 | academic\nimplication: this is the direct evidential anchor for taxonomy q7's feedback-vs-verdict-only branch: a pass/fail verdict is the reinforcement/corrective tier (expected d approximately 0.24-0.46), while explaining what is wrong against the criterion, why (process), and how to self-check next time (self-regulation) roughly doubles the expected effect. the minimal feedback unit worth building is therefore criterion-cited + evidence-grounded + process-level; verdict-only qa forfeits most of the achievable behavior change, and 'feedback is decoration' should only be concluded if high-information feedback (not verdicts) fails to move within-attempter error rates.\n\n#### the 2025 annual review of organizational psychology's 25-year retrospective concludes the science of workplace feedback 'is not yet a story of coherent and cumulative progress': definitions are generic, assumptions diverge across six disconnected research substreams, and simple universal rules about feedback effectiveness do not survive contact with organizational reality.\nevidence: crawled the full open-access review (anseel & sherf, annu. rev. organ. psychol. organ. behav. 12:19-43, published 2025-01-21). abstract states insights 'often appear disconnected from the way feedback is practiced and experienced in organizations'; the review calls for explicated assumptions and paradigms mirroring complex realities. the companion 2025 systematic review (heine, stouten & liden, j. organ. behav., 2025-10-26) reaches the same verdict for supervisor performance feedback: most studies fail even to specify feedback valence, and feedback quality/accuracy findings rest on inconsistent constructs.\nsource: a 25-year review of research on feedback in organizations: from simple rules to complex realities (anseel & sherf) | https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-110622-031927 | 2025-01-21 | academic\nimplication: tempering prior for the whole q7 branch: the general literature supplies directional moderators (task-focus, specificity, information content, source credibility) but no validated plug-in recipe, and effect heterogeneity is the norm. the project-agnostic foundation should therefore ship feedback design as parameterized hypotheses with built-in efficacy measurement (per-project a/b of feedback tiers against repeat-error rate), not as fixed doctrine imported from any single meta-analysis.\n\n#### under incentives, relative-rank feedback is a double-edged lever: lab and field economics find rank feedback raises output in flat-wage settings (charness et al. 2014; tafkov 2013) but induces costly sabotage and cheating to improve rank that offsets the gains, and in at least one field experiment (barankay 2012) removing rank feedback improved performance.\nevidence: confirmed via the literature synthesis in a peer-reviewed leadership quarterly article ('feedback quality and performance in organisations', 2021), which states: charness et al. (2014) found 'offering relative rank feedback increases output... and subjects are willing to engage in costly sabotage and cheating activities to improve their relative rank, thus offsetting the positive effects'; barankay (2012) is cited as the exception where rank feedback hurt. primary charness/barankay texts not independently opened, so treated as well-sourced secondary evidence.\nsource: feedback quality and performance in organisations (leadership quarterly; synthesizing charness et al. 2014, tafkov 2013, barankay 2012) | https://www.sciencedirect.com/science/article/abs/pii/s1048984321000394 | 2021 (synthesizing 2012-2016 primaries) | academic\nimplication: for taxonomy q8 (gaming/monoculture under pay-linked evaluation): if autoqa outputs become visible rank or pass-rate leaderboards tied to pay, the literature predicts optimization of the metric (score-hacking, mimicry of known-passing templates) rather than quality. keep attempter-facing feedback private, criterion-referenced, and decoupled from visible peer ranking; note the tension with cochrane's top-peer-benchmark moderator (see disagreements) - benchmark against exemplar work, not against ranked people.\n\n- disagreement: feedback frequency: pre-2025 audit-and-feedback guidance (ivers 2012 cochrane, hysong 2006) held that repeated/more frequent delivery increases effect; the 2025 cochrane update finds repeated delivery associated with lower effect size. unresolved - could be confounding (repeated a&f deployed where problems persist) or genuine habituation.\n- disagreement: normative comparison: the 2025 cochrane update finds comparison to top peers or a benchmark increases behavior change in healthcare professionals, while fit (kluger & denisi) predicts normative comparison shifts attention to self and degrades performance, and incentive economics (charness 2014, barankay 2012) finds rank feedback triggers gaming/sabotage under competitive stakes. plausible reconciliation: comparison to an exemplar standard helps when stakes are professional-norm-based; comparison as interpersonal rank hurts when pay/status is on the line - but no study directly adjudicates this.\n- disagreement: self-assessment vs external feedback: dow 2012 found rubric self-assessment equal to external expert feedback for quality improvement (arguing feedback machinery could be replaced by surfaced rubrics), but the same study found self-ratings inflated by 1.8/9 points and education literature (winstone 2016 recipience work) holds self-assessment only works when later external verification is believed to occur. external qa may be load-bearing as a credibility backstop even if the information could be self-generated.\n- disagreement: effect magnitude: education meta-analyses report medium standardized effects (d approximately 0.48), while the healthcare a&f median is a small 2.7% absolute improvement on already-trained professionals. for skilled adult annotators the healthcare prior (small median, heterogeneous, concentrated in low performers) is likely the better calibration than the education prior.\n- gap: still no longitudinal study of qa feedback effects on paid annotation workers' repeat-error rates: dow 2012 explicitly did not measure long-term learning, and no 2024-2026 annotation-platform study of feedback efficacy was found (searched exa, tavily, openalex/crossref). the original sweep's gap is real; what changed is that adjacent causal literature (cochrane a&f 2025, crowdwork rct) transfers with stated caveats.\n- gap: no study found on ai-generated feedback to human annotators under pay-linked evaluation - the exact autoqa deployment condition. closest analogs are ai-feedback-to-employees work (e.g., tong et al. 2021 smj, disclosure reduces effect) which was not deep-dived here.\n- gap: charness et al. 2014 and barankay 2012 primaries were not opened (claims verified only through a peer-reviewed secondary synthesis); exact effect sizes for gaming-offset were not extracted.\n- gap: feedback-specificity tradeoff (goodman & wood 2004/2011: high specificity aids immediate performance but can impair exploration and transfer to novel cases) was identified in the anseel & sherf reference base but not independently verified - relevant to whether highly prescriptive autoqa feedback creates template-following monoculture (q8) and worth one follow-up read.\n- gap: the 2025 individual-differences meta-analysis (condrea & iliescu, ejwop, 2025-12-16) on reactions to feedback was located but is too new to have accessible full text; could sharpen per-attempter moderation (feedback orientation, self-esteem) but was not extractable.\n", "5": "completeness critic\n\n## psychometrics and rater-training science (educational/language assessment, many-facet rasch measurement, generalizability theory). the sweep's 9 domains are all 2020s llm/annotation literature, but human-rater variance has 40+ years of quantitative treatment: rater severity/leniency modeling, rater drift over scoring sessions, frame-of-reference vs rater-error training effect sizes, g-theory decomposition of variance into rater/item/criterion facets. this directly answers taxonomy q1's variance-attribution question ('would perfect rubrics leave variance intact?') and q9's reviewer-skill-decay question with measured effect sizes, instead of leaving them as unanchored pilots.\nwhy: the build order and budget split hinge on whether reviewer variance is rubric-fixable or rater-intrinsic (root cause 1 vs 2-3). rasch/g-theory literature has actual numbers on how much variance rubric operationalization and rater training each remove, and validated statistical machinery (rater severity as a modeled parameter, not noise) that the entire llm-judge literature reinvents badly. ignoring it risks re-running expensive in-house pilots for questions psychometrics settled in the 1990s-2010s.\nquery: many-facet rasch measurement rater severity drift frame-of-reference training effect size rubric calibration generalizability theory variance components raters\n\n## legal/regulatory constraints on algorithmic management of the attempter workforce, plus data-quality standards. the autoqa issues automated pass/fail verdicts affecting paid workers' income and proposes process telemetry (timing, edit traces) as a provenance contract - this is squarely regulated territory: gdpr article 22 (automated decisions with significant effects require human review - which may legally mandate the one-human-touch the design treats as optional), the eu platform work directive (transposition deadline december 2026, explicitly governs algorithmic management, worker access to decision logic, and human oversight of automated account/pay decisions), and iso/iec 5259 (data quality for analytics and ml, published 2024-2025) which large buyers may contractually require. zero regulatory/standards sources appear in any domain.\nwhy: this is a foundational-architecture constraint, not compliance trim: if gdpr art 22 / platform work directive apply to attempters in the eu, 'zero-touch plus randomized audits' (taxonomy q6's fork) may be illegal for adverse verdicts, contest/appeal channels become mandatory rather than optional, and telemetry collection needs a lawful basis. discovering this after the philosophy document ships forces a redesign of routing and consequences.\nquery: eu platform work directive 2024 algorithmic management automated decision human oversight crowdwork data annotation gdpr article 22 iso 5259 data quality\n\n## content-moderation qa programs - the largest deployed analog of humans-reviewing-human-judgment at scale. meta/tiktok/google have run decade-old programs auditing moderator decisions: golden-set seeding into live queues, audit sampling rates, overturn/appeal-rate monitoring, qa-of-the-qa layers, measured reviewer base-rate and fatigue effects, and published transparency-report methodology plus academic studies (e.g., on moderator agreement and audit design). the industry-practice domain covered only ai-training-data vendors; this adjacent industry has already answered several taxonomy q6/q9 questions operationally (seeded known-verdict items, easy-case dilution, de-graduation triggers).\nwhy: it is the only place where the exact mechanisms the design speculates about - blind gold injection into live streams, closed-loop routing on overturn rates, queue-composition effects on reviewer harshness - have run in production for years with public postmortems and litigation-disclosed internals. cheaper to import their measured failure modes than rediscover them.\nquery: content moderation quality assurance program golden set audit sampling overturn rate reviewer agreement queue composition fatigue meta tiktok transparency methodology\n\n## feedback-efficacy science from education and organizational psychology. every domain independently reports 'no evidence that qa feedback changes attempter behavior' as a gap, yet feedback-intervention research is a mature field: kluger & denisi's feedback intervention theory (a third of feedback interventions reduce performance, with known moderators), formative-assessment literature on feedback specificity/timing, and workplace studies on feedback under pay-linked evaluation (which predicts gaming/monoculture, taxonomy q8). the sweep searched only for annotation-specific longitudinal studies and found none - the general literature was never consulted.\nwhy: taxonomy q7's entire branch (minimal feedback unit, feedback-vs-verdict-only comparison, when to conclude feedback is decoration) currently has zero evidential anchor, so the design would commit to feedback machinery on faith. fit's moderators (task-focused vs self-focused feedback, specificity, normative comparison) give testable priors for the feedback 4-tuple design and for predicting when feedback backfires - before spending pipeline cost on self-verified coaching.\nquery: feedback intervention theory kluger denisi meta-analysis feedback specificity performance improvement gaming incentivized workers longitudinal repeat error rate\n\n## weak spots\n- the single most load-bearing transfer assumption is unverified everywhere: every quantitative judge result in the sweep is measured on grading model outputs, while the autoqa grades human evaluative writeups (grounding of praise, severity calibration, evidence-conclusion entailment). all nine domains flag this and then the numbers get cited anyway - no domain even found a partial-transfer experiment, so the anchor numbers (77.4 bacc grounding, 55-56% rubric-hard, kappa 0.28) may be systematically off in either direction for the actual workload.\n- the headline meta-evaluation figures (33-41pp kappa deflation, 14-15 position ranking flips, consistency-bias paradox) all trace to one un-peer-reviewed, 0-citation june 2026 preprint ('reliability without validity', arxiv 2606.19544), cited independently by four domains as if it were four sources. cross-domain repetition is masquerading as corroboration.\n- much of the frontier-2026 and rubrics-recent evidence was verified only at abstract/title level (rubric co-evolution cluster, auditing-by-re-solving, evaluation faking, trust or escalate details, toloka's precision/recall table locked in an image) - several architecturally decisive claims (agent-in-environment verifiers beat text-only 10x cheaper; consequence-framing softens verdicts with zero cot trace) rest on numbers nobody in the sweep recomputed or read in full.\n- the spam-economics narrative (scale/bulba, paid gibberish for 11 months, account black market) is sourced via futurism quoting inc. documents the researchers could not access (403), is disputed on record by scale, and dates to the pre-llm-judge qa era - yet it anchors the taxonomy q8 conclusion that content-based detection has 'already lost'. westwood's agent study is the solid leg; the scale story is journalism-of-journalism.\n- criticgpt's 24%-vs-6% flawless-slice result - the best evidence for the whole human+critic premise - is one study, one vendor, mostly one domain (code), with no post-deployment report and no replication in 24+ months. the critique-models domain flags this but the claim still functions as the design's cornerstone.\n- several 'disagreement resolutions' offered across domains are inferred reconciliations, not tested ones: verbosity bias as 'a controllable design variable', rar-vs-rubricbench reconciled by 'grounding' as the variable, atomicity 'helps with per-criterion calls, hurts in long checklists' (explicitly labeled 'inferred, not directly tested'), and 'show critiques to attempters but evidence-only to adjudicators'. if the synthesis treats these as findings, the design inherits speculation as doctrine.\n- the premise metric is missing on both sides: no measured krippendorff's alpha exists for the actual human reviewer population this system replaces (vendors publish only thresholds), and none was gathered in-house per the mission context. the system is being designed against 'unexplainable variance' that has never been quantified - the variance-attribution pilot in taxonomy q1 is not optional and no sweep evidence substitutes for it.\n- judge-model evidence is stale relative to the decision date: no verified july-2026 leaderboard for current frontier models (both major leaderboards unfetchable), so any per-model judge-selection guidance in the synthesis is anchored to early-2026-or-older cohorts.\n- positive-claim/praise verification - an explicit mission requirement - has literally zero direct evidence in any domain (all six domain gap-lists say so). the synthesis has no precision/recall prior at all for the 'is this praise warranted' machinery and should present it as a from-scratch in-house measurement, not extrapolate from flaw-detection numbers, especially given absencebench predicts critics are near-blind to omissions (empty praise is an omission-shaped failure).\n- non-us/non-english practice was never searched: the chinese data-labeling industry (baidu/alibaba crowdsourcing qa, government-backed labeling bases) and japanese/korean/indian vendor practice are absent, so 'industry practice' claims generalize from a us/eu vendor sample.\n\n# gap-fill reports\n\n## area: psychometrics and rater-training science (educational/language assessment, many-facet rasch measurement, generalizability theory). the sweep's 9 domains are all 2020s llm/annotation literature, but human-rater variance has 40+ years of quantitative treatment: rater severity/leniency modeling, rater drift over scoring sessions, frame-of-reference vs rater-error training effect sizes, g-theory decomposition of variance into rater/item/criterion facets. this directly answers taxonomy q1's variance-attribution question ('would perfect rubrics leave variance intact?') and q9's reviewer-skill-decay question with measured effect sizes, instead of leaving them as unanchored pilots.\n\n### adjusting for rater severity/centrality with a many-facet rasch model (mfrm) materially changed which ai systems ranked best in the openai rlhf summarization dataset: trained raters' agreement was only qwk .31-.50, and mfrm-adjusted scores flipped raw-mean rankings so two human-feedback policies rose above human-written reference summaries.\nevidence: casabianca & beiting-parrish (lak 2026 llm psychometrics workshop) fit mfrm to 6,312 ratings from 15 trained raters on 639 summaries across 19 policies. full text opened and confirmed: raters r02/r08/r06 systematically lenient, r04/r10/r12 severe, r10 aberrantly central; raw-score rankings 'misrepresent the relative standing of models.' paper also specifies the estimability requirement: 3-5 ratings per output with rater-item overlap/linkage, and proposes percentile-based flagging (top/bottom 2.5%) for rater remediation.\nsource: correcting human labels for rater effects in ai evaluation: an item response theory approach (arxiv 2602.22585) | https://arxiv.org/html/2602.22585v1 | 2026-02 | academic\nimplication: autoqa should treat each human reviewer's severity, centrality, and criterion-specific thresholds as modeled parameters, not noise: use adjusted (fair) scores for pass/fail decisions and use per-rater parameter estimates as the constructive-feedback channel. this forces a data-collection constraint into the foundation: some deliberate rater-item overlap must be designed in, because a single-review-per-item regime with disjoint rater pools is statistically unidentifiable.\n\n### rater training on real scoring behavior improves within-rater consistency but does not equalize severity: after formal training of 16 raters in an operational esl writing exam, significant between-rater severity differences remained even though consistency improved for most raters.\nevidence: weigle 1998 (language testing, 363 citations; abstract opened and confirmed verbatim): 'rater training is more successful in helping raters give more predictable scores (i.e., intra-rater reliability) than in getting them to give identical scores (i.e., inter-rater reliability).' this is the canonical, repeatedly replicated finding of the language-testing rater literature (lumley & mcnamara 1995 same conclusion via mfrm).\nsource: using facets to model rater training effects (weigle, language testing 15(2)) | https://doi.org/10.1177/026553229801500205 | 1998-04 | academic\nimplication: directly answers taxonomy q1: perfect rubrics plus training will still leave stable between-reviewer severity variance intact. budget rubric/training work for what it actually buys (self-consistency, fewer misfitting raters) and route severity alignment to statistical adjustment in the autoqa layer instead of more calibration meetings.\n\n### frame-of-reference (for) training, the best-validated rater-training method, has a meta-analytic effect on rating accuracy of about cohen's d = 0.50 (moderate) - down from the d = 0.83 estimated by the earlier, smaller woehr & huffcutt 1994 meta-analysis - i.e., training helps but removes nowhere near all rater error.\nevidence: roch, woehr, mishra & kieszczynska (j. occup. organ. psychol., 2012; abstract opened) is the updated meta-analysis with 4x the studies of woehr & huffcutt 1994; it finds for training effective, with borman's differential accuracy and behavioural accuracy most improved. the d=0.50 overall figure confirmed via two independent citing sources (tsai et al. 2019 smu full-text pdf: 'medium-to-large effect on improving rating accuracy (d = 0.50, roch et al., 2012; d = 0.83, woehr & huffcutt 1994)'; loignon et al. 2017). note: classic 'rater error training' (teaching raters to avoid halo/leniency) reduces those errors but can reduce accuracy - for (anchoring raters to a shared performance theory with practice+feedback) is the variant that works.\nsource: rater training revisited: an updated meta-analytic review of frame-of-reference training (roch et al.) | https://doi.org/10.1111/j.2044-8325.2011.02045.x | 2012-06 | academic\nimplication: sets the ceiling on root-cause-1 fixes: even the best-in-class training intervention delivers ~half a standard deviation of accuracy gain. if the autoqa program funds training, fund for-style training (shared exemplars, practice ratings, feedback against reference judgments) - which the autoqa itself can generate as a byproduct - and plan for the residual variance to be caught by per-item verification.\n\n### g-theory decompositions of rated writing assessments show the rater main effect (global severity) is a minor variance source while high-order interactions dominate: in a 120-student efl study, student-by-task-by-method (31.8%), student-by-rater-by-task-by-method (26.5%), and student-by-rater-by-method (17.6%) interactions were the largest components, and 99.2% of error variance came from the student-by-rater interaction.\nevidence: khodi 2021 (language testing in asia, open access; pdf opened and percentages confirmed from abstract/results). consistent with the broader g-theory writing literature (e.g., gao, brennan & guo's gmat awa report, 2015): rater main-effect variance is small once training exists; person-by-rater and residual interaction variance is what limits reliability. interpretation: most rater-related variance is idiosyncratic per-item disagreement (which rater on which submission on which criterion), not a stable harshness offset.\nsource: the affectability of writing assessment scores: a g-theory analysis of rater, task, and scoring method contribution (khodi) | https://doi.org/10.1186/s40468-021-00134-5 | 2021-11 | academic\nimplication: this is the strongest psychometric argument for the autoqa's core design: global severity calibration and better rubrics attack only the small main-effect components. the dominant variance lives at the rater-times-item-times-criterion level, which can only be attacked by evidence-grounded evaluation of each specific claim on each specific item - exactly the 'is this statement grounded in the cited evidence' positive-verification layer. it also means severity-adjustment (finding 1) is necessary but not sufficient.\n\n### rater severity is nonstationary within a single scoring session: in an osce, later time-slots received systematically higher ratings (regression coefficient 0.88, 95% ci 0.38-1.38, p=.001), with the drift 2.4x larger on difficult stations (1.24 vs 0.52), and excluding warm-up stations did not remove it; the psychometric field has named machinery (drift - differential rater functioning over time, wolfe et al. 2001; myford & wolfe monitoring frameworks) for detecting it.\nevidence: pubmed abstract of mclaughlin et al., medical education 2009, opened and confirmed (coefficients above). wolfe & moulder 2001 (j. applied measurement) established rasch-based procedures for detecting drift types (primacy/recency, centrality drift, practice/fatigue). drift within and across scoring days is a repeated finding in operational scoring programs (lunz & stahl 1990 across grading periods).\nsource: the effect of differential rater function over time (drift) on objective structured clinical examination ratings (medical education 43(10)) | https://pubmed.ncbi.nlm.nih.gov/19769648 | 2009-10 | academic\nimplication: answers taxonomy q9 with measured evidence: reviewer calibration decays on the timescale of a single session, not just months, and decays faster on hard items. autoqa should compute reviewer-effect estimates in rolling windows (per session/day) and alert on drift, rather than certifying a reviewer once at onboarding; item-order randomization and hard-item interleaving are cheap design mitigations.\n\n### the mfrm toolkit is now being applied symmetrically to llm judges: a 2025 study fit mfrm to 10 llms plus human expert raters scoring the same writing tasks and found llms exhibit measurable severity/centrality rater effects that differ by model, with gpt-4o, gemini 1.5 pro, and claude 3.5 sonnet showing the highest accuracy and fewest rater effects.\nevidence: jiao, song & lee, arxiv 2505.18486 (may 2025), abstract opened and confirmed: qwk vs human scores, cronbach alpha across prompts, and mfrm rater-effect estimates computed identically for human and llm raters. parallel 2025 work (wang et al., computers and education: ai, sept 2025) builds a full psychometric reliability/validity framework for llm raters in large-scale writing assessment. this is the literature bridge: the same model audits both layers.\nsource: comparing human and ai rater effects using the many-facet rasch model (jiao, song, lee) | https://arxiv.org/abs/2505.18486 | 2025-05-24 | academic\nimplication: the autoqa's own llm-judge component should be enrolled as just another 'rater' facet in the same measurement model as the humans it reviews. this gives a project-agnostic, self-auditing structure: one mfrm per project instruction-set yields severity/centrality/fit for every judge, human or machine, and flags when the llm layer itself drifts after a prompt or model change.\n\n- disagreement: size of the training effect: woehr & huffcutt 1994 estimated for training accuracy gain at d=0.83; roch et al. 2012, with 4x the studies, revised it down to d~0.50. separately, noh & matore 2022 (frontiers in psychology, 164 teacher-raters, mfrm) found prior training experience made no difference to rating quality on a speaking assessment while rating and teaching experience did - generic/one-shot training may buy nothing; the meta-analytic d~0.50 applies to structured for protocols specifically.\n- disagreement: how pathological rater pools are: casabianca & beiting-parrish 2026 cite evidence (nieto & casabianca 2019) that large professional testing-organization rater pools show minimal rater effects, versus their own demonstration that in a small trained pool (openai's 15 raters) a few aberrant raters materially distorted system rankings. the severity of the problem depends on pool size, professionalization, and tenure - crowdsourced/daas annotation pools sit at the bad end of this spectrum.\n- disagreement: where the fixable variance lives: mfrm studies foreground the rater severity main effect (a stable, correctable offset), while g-theory decompositions (khodi 2021; gmat awa tradition) show the main effect is small and idiosyncratic rater-by-item interaction dominates. these are complementary lenses, but they imply different remedies (statistical adjustment vs per-item verification), and the literature does not agree on the split's typical proportions across contexts.\n- gap: no meta-analysis isolates how much variance rubric operationalization alone removes (rubric vs no-rubric variance-component comparison). individual studies exist (rating-scale vs rubric vs holistic; criteria-order effects on halo, kim 2020) but no pooled effect size - so 'root cause 1 vs 2-3' cannot be settled from literature alone, though the weigle/khodi pattern strongly suggests substantial rater-intrinsic residual.\n- gap: long-horizon reviewer skill decay (months-scale, the q9 timescale) is not measured in this literature: drift studies cover within-session and across-day/grading-period drift. no published effect sizes for calibration decay over weeks-to-months in annotation-industry settings were found through 2026-07.\n- gap: minimal-linkage requirements under annotation-economics constraints are underspecified: mfrm needs rater-item overlap, but no published guidance found on the cheapest overlap design (e.g., what fraction of items double-reviewed, seeded common items) sufficient for stable severity estimates in pools with high rater churn - this likely needs a small in-house simulation, which is cheap since the models are standard.\n- gap: semantic scholar returned http 429 during the sweep; coverage relied on openalex/crossref/exa/tavily. a dedicated pass over language testing and assessing writing 2025-2026 issues could surface additional recent mfrm-for-annotation work.\n## area: legal/regulatory constraints on algorithmic management of the attempter workforce, plus data-quality standards. the autoqa issues automated pass/fail verdicts affecting paid workers' income and proposes process telemetry (timing, edit traces) as a provenance contract - this is squarely regulated territory: gdpr article 22 (automated decisions with significant effects require human review - which may legally mandate the one-human-touch the design treats as optional), the eu platform work directive (transposition deadline december 2026, explicitly governs algorithmic management, worker access to decision logic, and human oversight of automated account/pay decisions), and iso/iec 5259 (data quality for analytics and ml, published 2024-2025) which large buyers may contractually require. zero regulatory/standards sources appear in any domain.\n\n### platform work directive article 10(5) requires that any decision to restrict, suspend, or terminate the contractual relationship or account of a person performing platform work - or any other decision of equivalent detriment - be taken by a human being, with no consent or contractual-necessity exception (stricter than gdpr art 22(2)).\nevidence: verified against the directive text and multiple law-firm/academic analyses (wolters kluwer, taylor wessing, etui). article 10 covers decisions 'taken or supported' by automated systems, deliberately closing the gdpr gap for hybrid human+algorithm pipelines. unlike gdpr art 22, the pwd permits no consent-based or contract-based exception for these decisions.\nsource: directive (eu) 2024/2831, official journal 11 nov 2024 (eur-lex), corroborated by wolters kluwer global workplace law & policy analysis | https://eur-lex.europa.eu/legal-content/en/txt/?uri=celex:32024l2831 | 2024-11-11 | primary\nimplication: for eu-based attempters, 'zero-touch plus randomized audits' is illegal for verdicts that cost workers access to work, pay, or their account once national transposition applies (from 2 dec 2026). the one-human-touch is not a design option for adverse consequences - it is a legal floor. autoqa can auto-pass but must route auto-fail-with-consequences through a human decision-maker. this is stricter than gdpr: consent cannot buy the exception back.\n\n### platform work directive article 11 grants persons performing platform work the right to (a) a plain-language explanation of any decision taken or supported by an automated decision-making system, (b) a designated competent human contact person to discuss the decision, and (c) human review with a substantiated written reply within two weeks of request; decisions that infringed rights must be rectified within two weeks or compensated.\nevidence: verified verbatim from directive text via extraction of the eur-lex page: explanation 'in a transparent and intelligible manner, using clear and plain language'; contact persons must have 'the competence, training and authority necessary'; substantiated reply 'in any event within two weeks of receipt of the request'; rectification/compensation duty plus obligation to modify or discontinue the offending automated system. carve-out: for workers who are 'business users' under the p2b regulation (eu) 2019/1150, that regulation's human-review provisions prevail - but p2b itself mandates statements of reasons and an internal complaint-handling system for restriction/suspension/termination.\nsource: directive (eu) 2024/2831, article 11 (eur-lex full text) | https://eur-lex.europa.eu/legal-content/en/txt/?uri=celex:32024l2831 | 2024-11-11 | primary\nimplication: contest/appeal channels are legally mandatory, not optional, with a hard two-week sla. the autoqa's constructive-feedback output doubles as the legally required explanation - so verdict rationales must be written in plain language grounded in the project criteria (not model logits or opaque scores), and the system must log enough evidence per verdict to support a substantiated human reply on appeal. the 'modify or discontinue the system' duty means systematic autoqa errors surfaced via appeals trigger a legal remediation obligation.\n\n### the directive's scope explicitly covers online-performed work - the 'digital labour platform' definition applies 'irrespective of whether that work is performed online or in a certain location' - recital 19 names tagging/crowdwork, and chapter iii algorithmic-management rules (arts 7-11) apply to genuinely self-employed persons performing platform work, from the start of recruitment, regardless of where the platform is established.\nevidence: definition verified verbatim from eur-lex (art 2: service provided at a distance by electronic means, at the recipient's request, involving organisation of work for payment, using automated monitoring/decision systems). article 7(2) verified: 'shall apply to all persons performing platform work from the start of the recruitment or selection procedure.' chapter iii's application to the genuinely self-employed rests on the art 16(2) tfeu data-protection legal basis (etui, countouris & de stefano 2025). lund ai policy lab (mar 2025) analysis specifically concludes ai labelers are covered. extraterritoriality confirmed by ius laboris and omnivoo (2026): applies to platforms established outside the eu when the work is performed in the eu.\nsource: directive (eu) 2024/2831 art 2 & 7 (eur-lex) + ai policy lab, 'potential impact of the eu platform work directive on ai labelers' | https://aipolicylab.se/2025/03/25/potential-impact-of-the-eu-platform-work-directive-on-ai-labelers/ | 2025-03-25 | academic\nimplication: classifying attempters as independent contractors does not exempt the autoqa from the algorithmic-management chapter - the human-decision, explanation, appeal, and data-limitation rules attach to self-employed annotators too, and to a us-based platform with eu attempters. the foundational philosophy must treat these as baseline constraints for any eu-touching deployment, effective 2 dec 2026 (about 5 months after this design ships).\n\n### platform work directive article 7 prohibits automated systems from processing personal data on workers' emotional or psychological state, private conversations (including worker-to-worker exchanges), any data collected while the person is not offering or performing platform work, data predicting exercise of fundamental rights (e.g., organizing), or inferring protected characteristics; article 8 additionally mandates a data protection impact assessment.\nevidence: verified verbatim from the directive text via eur-lex extraction (art 7(1)(a)-(c) and recital 40). art 7(3) extends these limits to any automated system 'taking or supporting decisions that affect persons performing platform work in any manner' - not just formally designated monitoring systems.\nsource: directive (eu) 2024/2831, article 7 (eur-lex full text) | https://eur-lex.europa.eu/legal-content/en/txt/?uri=celex:32024l2831 | 2024-11-11 | primary\nimplication: the proposed process-telemetry provenance contract (timing, edit traces) is lawful only if scoped to on-task activity: no collection while the attempter is off-task, no sentiment/frustration inference from edit patterns or communications, no monitoring of attempter-to-attempter channels. telemetry feeding the autoqa requires a dpia. design the telemetry schema as an explicit allowlist (task-session-bounded behavioral signals) rather than a general activity log.\n\n### under gdpr article 22 as interpreted in cjeu schufa (c-634/21, 7 dec 2023), an automated score that plays a 'determining role' in a downstream decision is itself an article 22 decision, and human involvement only removes a decision from art 22 scope if it is meaningful - a reviewer with real authority, data access, and competence to override; rubber-stamping does not count. enforcement is active: italy's dpa sanctioned automated rider deactivations, and hamburg's dpa fined a company ~eur 490,000 in september 2025 for automated rejections without adequate explanation of the logic.\nevidence: schufa three-condition test and 'determining role' doctrine confirmed across tlt, bird & bird, iapp, and oxford industrial law journal ('scores as decisions?', 2024, applying it to the labour context and uber litigation). edpb-endorsed wp29 guidance holds that consent is generally an invalid basis in work contexts due to power imbalance, and that human involvement must be substantive. hamburg fine and italian rider cases reported in 2025 enforcement roundups; regulator attention to automated rejections continuing into mid-2026.\nsource: cjeu case c-634/21 schufa analysis + 'scores as decisions? article 22 gdpr ... in the labour context', industrial law journal | https://academic.oup.com/ilj/article/53/4/840/7745471 | 2024-09 | academic\nimplication: two consequences beyond the pwd: (1) even where autoqa runs as a vendor scoring layer feeding a client's pass/fail call, the schufa logic makes the score itself an art 22 decision if the client defers to it - outsourcing the final click doesn't escape regulation; (2) the one-human-touch must be designed as genuine review (authority to overturn, access to the evidence, calibrated competence), or it legally counts as zero-touch. worker consent cannot serve as the lawful basis for telemetry or automated verdicts.\n\n### iso/iec 5259 'data quality for analytics and ml' is now a complete five-part published series - parts 1-4 published 2024 (overview/terminology; data quality measures; data quality management requirements & guidelines; process framework) and part 5 (data quality governance framework) published 2025 - sold by iso as an 'ai data quality management bundle'.\nevidence: confirmed from iso.org catalog pages: iso/iec 5259-1:2024 (81088), 5259-2:2024 (81860, 'specifies a data quality model, data quality measures and guidance on reporting data quality'), 5259-3:2024 (81092, management requirements), 5259-4:2024 (81093, process framework covering data labeling among ml data processes), 5259-5:2025 (84150, governance). full text paywalled; no direct evidence found of large ai-data buyers contractually mandating 5259 yet, but 5259-3/-4 are certifiable-style requirements documents a buyer can reference in contract.\nsource: iso/iec 5259 series catalog pages (iso.org) / ai data quality management bundle | https://www.iso.org/publication/pub200525.html | 2025 | primary\nimplication: the autoqa philosophy document should map its quality dimensions and verdict evidence onto 5259-2's measurement/reporting vocabulary and 5259-4's process framework, so a per-project customization can be presented as 5259-conformant when a buyer asks - cheap to do now, expensive to retrofit. treat 5259 as the neutral shared vocabulary for 'what a quality measure is' across projects.\n\n- disagreement: scope breadth of the algorithmic-management rules: cxc global (apr 2026) claims the directive's algorithmic-management rules 'apply to all workers, not only those engaged through digital labour platforms,' but the directive text limits chapter iii to 'persons performing platform work'; etuc's transposition manual (mar 2026) confirms extension to all workers is only an advocacy position for national transposition, not the directive's requirement. some member states may gold-plate; the eu floor covers platform work only.\n- disagreement: whether online annotation workers will effectively get the employment presumption: ai policy lab (mar 2025) says ai labelers benefit when platforms control workflows/evaluation, but commentators (verfassungsblog, ilo working papers) caution the control criteria are anchored in traditional notions that fit poorly with subtle microtask monitoring, and fairwork warns weak transpositions (e.g., italy) may leave annotators 'self-employed'. note this dispute concerns the employment presumption only - chapter iii algorithmic-management rights attach regardless of status.\n- disagreement: strictness of national implementations: law-firm trackers (ius laboris apr 2026, employsome may 2026) expect strict transpositions in france/germany/italy/spain/netherlands but narrow ones elsewhere (hungary had taken no steps as of the cms review; ireland has no draft legislation), creating a 2026-2027 patchwork - sources disagree on whether to design to the strictest member state or per-country.\n- gap: no enforcement action or national transposition text specifically addressing ai data-annotation qa verdicts was found - the directive's application to a qa layer rejecting individual work items (vs. account suspension) is untested; whether a single item pass/fail (payment denial for one task) is a 'decision of equivalent detriment' under art 10(5) versus merely an art 11 reviewable decision is unresolved in the sources.\n- gap: whether a bpo/vendor arrangement (annotators employed by an outsourcing firm, autoqa run by the ai lab or its vendor) falls within the 'digital labour platform' definition is unexamined in available sources; recital 20 excludes platforms that merely connect providers to clients, and the intermediary/subcontractor liability question is flagged by etuc as a transposition battleground.\n- gap: iso/iec 5259 full text is paywalled - could not verify its specific annotation-quality measures (e.g., inter-annotator agreement treatment) or confirm any ai-data buyer contractually requiring it; the 'buyers may require it' premise remains plausible but unevidenced.\n- gap: no dedicated edpb guideline on automated decision-making in employment exists as of july 2026 - the operative guidance is still the 2018 wp29 guidelines (edpb-endorsed); the edpb work programme 2026-2027 (adopted 11 feb 2026) was not verified in detail for a pending adm-in-employment item.\n- gap: non-eu jurisdictions were out of scope of this pass: no findings on us state algorithmic-management bills, uk's duaa adm regime (expected effective 2026), or brazil/india annotation-workforce rules - relevant if the attempter pool is global.\n## area: content-moderation qa programs - the largest deployed analog of humans-reviewing-human-judgment at scale. meta/tiktok/google have run decade-old programs auditing moderator decisions: golden-set seeding into live queues, audit sampling rates, overturn/appeal-rate monitoring, qa-of-the-qa layers, measured reviewer base-rate and fatigue effects, and published transparency-report methodology plus academic studies (e.g., on moderator agreement and audit design). the industry-practice domain covered only ai-training-data vendors; this adjacent industry has already answered several taxonomy q6/q9 questions operationally (seeded known-verdict items, easy-case dilution, de-graduation triggers).\n\n### facebook's production qa at cognizant ran a three-layer blind audit: ~50-60 of each moderator's ~1,500 weekly decisions (~3-4%) randomly re-reviewed by a dedicated qa worker (paid $1/hr more), with full-time facebook employees auditing a subset of qa decisions; 'accuracy' was computed purely as agreement with the auditor against a 95% target (other sites reported 98%), actual scores ran high-80s to 92, and misses triggered a remediation program that often ended in termination.\nevidence: verified in the verge's original text (fetched): 'from miguel's 1,500 or so weekly decisions, facebook will randomly select 50 or 60 to audit... full-time facebook employees then audit a subset of qa decisions.' moderator quote: 'accuracy is only judged by agreement. if me and the auditor both allow the obvious sale of heroin, cognizant was correct... this number is fake.' also confirmed: moderators lobbied qas off-book to reverse decisions despite prohibition, and pre-firing 'coaching' often served as pretext for managing workers out.\nsource: the trauma floor: the secret lives of facebook moderators in america (the verge, casey newton) | https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona | 2019-02-25 | primary\nimplication: directly answers taxonomy q6/q9 with a decade-scale postmortem: (a) ~3-4% random audit + a qa-of-qa layer is the deployed baseline; (b) agreement-with-auditor as the accuracy metric is blind to correlated errors and gameable via dispute lobbying; (c) punitive per-item scoring makes the qa relationship adversarial. autoqa should score attempter statements against grounded evidence (not reviewer agreement), log auditor-attempter deliberations, and decouple constructive feedback from de-graduation triggers.\n\n### the trust & safety professional association (industry body staffed by meta/google/tiktok alumni) codifies exactly two complementary qa channels - forward audit sampling (weighted re-review samples by peers or a dedicated quality team) and 'reverse quality sampling' (pre-reviewed golden/known-verdict items injected through the regular review process) - with the binding constraint that seeded items 'must look exactly like regular reviews to be effective', plus appeals/overturns as a cheap but population-biased third signal, and a four-way error taxonomy (false positive, false negative, wrong-selection, technical error).\nevidence: fetched both tspa curriculum pages. qa page: golden seeding gives 'full control' to probe chosen edge cases but fails when user history is part of the review or when indistinguishability breaks; appeals surface false positives 'at a low cost' but 'the population of appeals is often different from the population of all decisions.' metrics page: overturn rate 'can be a difficult metric to interpret' alone (low overturn = correct decisions or users gave up); 'consistency' (multi-review mismatch rate) is tracked as a distinct metric from error rate; prevalence measurement via random sampling is called a 'premium metric' due to cost.\nsource: tspa: content moderation quality assurance + metrics for content moderation | https://www.tspa.org/curriculum/ts-fundamentals/content-moderation-and-operations/content-moderation-quality-assurance/ | 2022-09-16 | practitioner\nimplication: imports the settled industry answer to blind gold injection: it is standard practice, its failure mode is detectability (seeded items must be indistinguishable - hard when context/history is part of the task), and it is complementary to (not a substitute for) forward sampling. the wrong-selection error class (right verdict, wrong cited criterion) maps directly to the autoqa requirement to check that an attempter's statement is aligned with the specific criterion they invoke, not just that the verdict is right.\n\n### pinterest's deployed decision quality evaluation framework (feb 2026) scores both human moderators and llm agents against an sme-adjudicated, versioned, immutable golden set selected by inverse-propensity sampling (xgboost on pinclip embeddings, prioritizing low-propensity/novel items), using a two-axis diagnostic - cohen's kappa for reliability plus correctness vs the golden set - where 'high reliability + low correctness' is read as systematic policy misunderstanding; measured results show 3x-human majority vote adds only +3.6pp accuracy over a single non-expert human, and current llms perform on par with a single non-expert human (gpt-5 the only config with positive accuracy delta, +0.9pp).\nevidence: fetched full html of arxiv 2602.15809. additional verified mechanics: policy changes handled by dual-labeling the existing golden set under old and new guidelines and visualizing the 'policy delta' as label flips; qa-of-the-qa is two continuous monitors - content-drift (evaluate on newest golden items each release) and system-stability (re-run fixed prompt on fixed golden version to catch pipeline non-determinism); switching prevalence measurement from 3x-human to llm gave 'over 30x cost savings and 10x turnaround reduction.' gemini 2.5 configs showed +22pp recall but +48-57pp false-positive rate.\nsource: decision quality evaluation framework at pinterest (arxiv 2602.15809) | https://arxiv.org/abs/2602.15809 | 2026-02-17 | primary\nimplication: the closest 2026 production analog to the autoqa foundation: (a) reliability and correctness must be separated - kappa alone cannot distinguish shared misunderstanding from noise; (b) golden sets should be deliberately non-representative (oversample rare/hard cases) with coverage measured in embedding space; (c) policy evolution requires versioned relabeling, not patching; (d) an llm qa layer needs both a drift monitor and a determinism monitor; (e) expect the llm judge to be ~single-non-expert-human quality with an fpr skew unless calibrated.\n\n### auditing raters by rewarding agreement with the eventual consensus outcome ('consensus-based auditing', as x's community notes has done since sept 2022 by tying participation eligibility to agreement with the final aggregate) measurably induces strategic conformity - minority contributors' evaluations drift toward the majority and their participation share falls precisely on controversial items - and a two-stage alternative that weights contributors by the stability of their past residuals (predictability relative to a latent-factor model) rather than majority-agreement improves out-of-sample predictive performance.\nevidence: fetched arxiv 2603.18053 (alimohammadi, huang, borgs, chayes, mar 2026) abstract and framing via exa crawl: empirical evidence from community notes data plus a behavioral model where contributors trade off private beliefs against anticipated penalties for disagreement; the proposed method gives influence to consistently-informative contributors 'even when they disagree with the prevailing consensus.' senior authors (borgs h-54, chayes h-60). preprint, not yet peer-reviewed.\nsource: auditing the auditors: does community-based moderation get it right? (arxiv 2603.18053) | https://arxiv.org/html/2603.18053 | 2026-03-17 | academic\nimplication: directly constrains de-graduation trigger design: if attempter or qa standing is scored by agreement-with-final-verdict, the autoqa will train its best dissenters to conform exactly where independent judgment matters most (the subjective-interpretation root cause #1). score contributors on the stability/informativeness of their residuals against a grounded reference, and never penalize evidence-backed disagreement per se.\n\n### google's audit-quality method (iclr 2023 tiny paper, google llc authors) decomposes inter-rater agreement per rubric question rather than per item - in their worked example one question had fleiss kappa 0.094 against a 0.475 overall, isolating it as the ambiguous criterion - and identifies sequential non-blind review (a later reviewer seeing the earlier verdict) as a distinct, measurable source of audit risk, testable via a/b or difference-in-differences.\nevidence: extracted full pdf text via pdftotext. methods: per-question fleiss kappa to locate high-disagreement criteria; chi-square to test criterion-verdict linkage; t-test/anova to detect review teams systematically deviating from ground truth; binomial cis to extrapolate sampled error rates. caveats: 2-page tiny paper, synthetic dataset (3 reviewers, 9-question rubric, 1,528 products), no production deployment claimed. code at https://github.com/xuanyang0607/openreviewpaper.\nsource: statistical methods for auditing the quality of manual content reviews (arxiv 2306.07466, iclr 2023 tiny papers) | https://arxiv.org/abs/2306.07466 | 2023-06-12 | academic\nimplication: for root cause #1 (subjective interpretation of axes): measure agreement per evaluation criterion, not per item - this converts 'reviewers disagree' into 'criterion 3 is ambiguous, rewrite it', which is the closed-loop the autoqa needs to feed back into project instruction sets. also: keep any second/qa review blind to the first verdict to avoid anchoring.\n\n### a 2025 new media & society study (screen-share observation of commercial moderators in india) documents that throughput pressure distorts verdict distributions through the action-selection interface itself: moderators systematically chose a 2-click removal over a 4-click de-ranking regardless of policy fit, mechanically deleted tool-highlighted words without context assessment, and privately compressed broad guidelines into simplified dos/don'ts lists - i.e., queue and ui composition changed outcomes independent of moderator judgment quality.\nevidence: fetched the conversation summary (2025-07-22) by the study authors (chatterjee iit delhi/uq, gupta, thomas; new media & society, peer-reviewed). evidence is qualitative (interviews + observed sessions), no quantitative error rates; an observed moderator said she 'would never recommend de-ranking content as it would take time.' complements industry-standard per-item handling-time metrics (tspa) that create the pressure.\nsource: hard labour conditions of online moderators directly affect how well the internet is policed (the conversation, summarizing new media & society study) | https://theconversation.com/hard-labour-conditions-of-online-moderators-directly-affect-how-well-the-internet-is-policed-new-study-261386 | 2025-07-22 | academic\nimplication: reviewer 'harshness' variance is partly an artifact of per-item cost asymmetries, not belief: any autoqa that infers attempter/reviewer quality from verdict distributions must first control for the click/effort cost of each verdict option, and the one-human-touch interaction in the hybrid loop should make the correct action the cheapest action.\n\n- disagreement: facebook's moderator accuracy target: the verge's original phoenix reporting (feb 2019, fetched) states a 95% target with actual scores in the high-80s to 92; cnbc's tampa follow-up (jun 2019) and the irish times' dublin reporting (feb 2020) both state a 98% target. likely site/contract variation, but the primary sources genuinely conflict on the number; all agree the target was chronically missed.\n- disagreement: what low reviewer agreement means: production vendor practice (facebook/cognizant, accenture/cpl) treats disagreement-with-auditor as individual reviewer error and scores/fires on it, while musubi labs (nov 2025, practitioner), pinterest's framework (2026), and the google tiny paper treat low agreement primarily as a policy/criterion-ambiguity signal ('don't force agreement'); the community notes paper (mar 2026) goes further, showing agreement-rewarded auditing actively corrupts the signal via strategic conformity.\n- disagreement: platform self-reported accuracy vs external measurement: tiktok's fifth dsa transparency report claims 99.2% moderation accuracy (self-defined, sample re-review), while academic audits of the dsa transparency database found up to ~50-percentage-point inconsistencies in tiktok's self-reported automation figures, and moderator testimony ('this number is fake - accuracy is only judged by agreement') argues agreement-based accuracy overstates true correctness. self-reported accuracy figures from this industry should not be used as calibration anchors.\n- disagreement: value of majority vote: pinterest measured 3x-human majority at only +3.6pp accuracy over a single non-expert human (arguing redundancy is a weak, expensive lever), whereas roblox's published practice treats >=80% multi-moderator alignment as the gate for scaled consistency - different uses (measurement vs gating) but opposite implicit views on how much signal replication buys.\n- gap: no public quantitative study of queue-composition effects on reviewer harshness in content moderation specifically (easy-case dilution shifting strictness, prevalence-induced criterion drift): the mechanism is well established in adjacent vigilance/low-prevalence-effect literature, but i found no moderation-industry measurement of it; the 2025 new media & society evidence is qualitative only. this taxonomy question remains empirically open even in the largest deployed analog.\n- gap: gold-injection density is nowhere disclosed: no platform, vendor, or paper states what fraction of a live queue is seeded known-verdict items, only that seeding exists and must be indistinguishable (tspa) and that golden sets should oversample hard cases (pinterest, musubi).\n- gap: meta's current (2025-2026) internal audit sampling rates and community standards enforcement report reviewer-accuracy methodology are not publicly disclosed at mechanic level; the best internals remain 2019-2020 journalism. kenya/meta litigation (majorel/sama) disclosures center on labor harms, not qa mechanics - i found no litigation-produced qa-design documents newer than that reporting.\n- gap: de-graduation thresholds are only known via journalism (miss accuracy target -> remedial program -> termination); no published numeric trigger (e.g., n misses in window, minimum sample before action) from any platform.\n- gap: pinterest's paper, the single best 2026 primary source, omits absolute golden-set size, sme adjudication protocol, and absolute (non-delta) accuracy values, so its numbers transfer as design patterns, not calibration constants.\n## area: feedback-efficacy science from education and organizational psychology. every domain independently reports 'no evidence that qa feedback changes attempter behavior' as a gap, yet feedback-intervention research is a mature field: kluger & denisi's feedback intervention theory (a third of feedback interventions reduce performance, with known moderators), formative-assessment literature on feedback specificity/timing, and workplace studies on feedback under pay-linked evaluation (which predicts gaming/monoculture, taxonomy q8). the sweep searched only for annotation-specific longitudinal studies and found none - the general literature was never consulted.\n\n### feedback intervention theory's core empirical result stands: feedback improves performance on average (k&d 1996: 131 studies, ~12,000+ participants, mean d approximately 0.38-0.41) but more than one third of feedback interventions reduce performance, and effectiveness declines as feedback cues move attention from the task toward the self (praise, person-level evaluation, normative comparison).\nevidence: verified in the authors' own summary (kluger & denisi 1998, current directions in psychological science): 'although fis improve performance on average, they reduce performance in more than one third of the cases.' independently corroborated by wisniewski et al. 2020, which cites k&d as 131 studies, >12,000 participants, average effect 0.38 with roughly a third of effects negative. fit's mechanism: feedback that directs attention to meta-task/self processes (threat to self, praise, social comparison) depletes task attention and backfires; task- and process-focused cues help.\nsource: feedback interventions (kluger & denisi 1998), summarizing k&d 1996 psychological bulletin meta-analysis | https://doi.org/10.1111/1467-8721.ep10772989 | 1998-06 (meta-analysis 1996-03; foundational) | academic\nimplication: the autoqa feedback 4-tuple must be strictly task/criterion-referenced and evidence-anchored, never person-referenced or rank-referenced; a feedback channel is not presumptively net-positive, so the design should treat 'feedback reduces this attempter's subsequent quality' as an expected outcome for a substantial minority and instrument for it (per-attempter pre/post error-rate deltas), not assume monotone benefit.\n\n### the march 2025 cochrane update on audit-and-feedback (292 studies, 678 arms, healthcare professionals) finds median absolute improvement in desired practice of only 2.7% (iqr 0.0 to 8.6; weighted meta-analytic mean +6.2%, 95% ci 4.1-8.2, moderate certainty), with effects larger for low baseline performers, individual-level (not team-level) data, comparison to top peers or a benchmark (comparison to peer average showed no significant effect), a trusted local source, and action plans with specific advice - while repeated delivery was associated with lower effect size.\nevidence: fetched the cochrane summary (updated review of cd000259, published 2025-03-25): '292 studies with 678 arms'; median absolute improvement '2.7%, with an iqr of 0.0 to 8.6'; weighted mean '6.2% (95% ci 4.1 to 8.2; moderate-certainty evidence)'; or 1.47. moderator list quoted directly, including the counterintuitive 'repeated delivery was associated with lower effect size' and 'comparison to top-peers or a benchmark increased effects; comparing against the average of all peers did not.'\nsource: audit and feedback: effects on professional practice (cochrane review update, ivers et al.) | https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.cd000259.pub4/full | 2025-03-25 | academic\nimplication: this is the largest causal evidence base that verdict-plus-feedback changes skilled professionals' behavior: expect a real but modest median effect with a fat right tail, concentrated in currently-low performers. design levers with evidence: target feedback at low-baseline attempters first, deliver individual-level data, pair every failed criterion with a specific corrective action, and benchmark against top-quality exemplars rather than cohort averages. do not assume higher feedback frequency improves uptake.\n\n### in the closest annotation-analog rct (105 analyzed mechanical turk workers writing product reviews), both timely external expert feedback and rubric-based self-assessment significantly improved work quality vs no feedback (expert ratings 6.01 and 6.35 vs 5.69 on a 9-point scale, p<0.05) with no quality difference between external and self-assessment; external feedback uniquely drove revision behavior (56.5% revised vs 24.8% for self-assessment) and more output, and self-assessors over-rated their own work by 1.8 points (7.9 self vs 6.1 expert).\nevidence: read the full dow, kulkarni, klemmer & hartmann cscw 2012 pdf. between-subjects, blind-to-condition expert grading; self-assessment condition showed significant learning over the task series (slope 0.25, p=0.001) vs borderline for external (0.10, p=0.08) and null for none. also: crowd-peer assessors had low agreement with the expert (kappa=0.20 aggregated), and the paper explicitly did not measure long-term learning. attrition was higher under assessment (self 78%, external 61%, vs none 47% incompleteness), so gains partly reflect weak performers dropping out plus learning.\nsource: shepherding the crowd yields better work (dow, kulkarni, klemmer, hartmann; cscw 2012) | https://www.cs.cmu.edu/~spdow/files/crowds-shepherd-cscw12.pdf | 2012-02-11 (foundational; only direct crowdwork feedback rct found) | academic\nimplication: directly refutes 'no evidence qa feedback changes attempter behavior' for paid micro-task workers: rubric-mediated, task-specific, synchronous feedback is the minimal effective unit. a concrete per-criterion rubric surfaced to attempters may capture most of the quality gain of expensive external feedback (making the autoqa-generated feedback the 'external expert' at near-zero marginal cost), but attempter self-ratings cannot serve as measurement, and part of any observed 'improvement' will be selection (weak attempters exiting) - the design's efficacy telemetry must separate within-attempter learning from attrition.\n\n### in the largest educational feedback meta-analysis (435 studies, k=994, n>61,000), information content is the dominant moderator: high-information feedback (task + process + self-regulation content) yields d=0.99 [0.82-1.15] versus d=0.46 for corrective right/wrong feedback and d=0.24 for bare reinforcement/punishment - with overall d=0.48 masking huge heterogeneity (i2=83%) and 17% of raw effects negative.\nevidence: fetched the open-access frontiers article (wisniewski, zierer & hattie 2020). feedback-type moderator significant (qb=41.52, p<0.0001); outcome moderator significant: cognitive d=0.51 vs motivational d=0.33; of negative motivational effects, 86% came from uninformative (reward/punishment-style) feedback. authors conclude feedback 'cannot be understood as a single consistent form of treatment.'\nsource: the power of feedback revisited: a meta-analysis of educational feedback research | https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.03087/full | 2020-01-22 | academic\nimplication: this is the direct evidential anchor for taxonomy q7's feedback-vs-verdict-only branch: a pass/fail verdict is the reinforcement/corrective tier (expected d approximately 0.24-0.46), while explaining what is wrong against the criterion, why (process), and how to self-check next time (self-regulation) roughly doubles the expected effect. the minimal feedback unit worth building is therefore criterion-cited + evidence-grounded + process-level; verdict-only qa forfeits most of the achievable behavior change, and 'feedback is decoration' should only be concluded if high-information feedback (not verdicts) fails to move within-attempter error rates.\n\n### the 2025 annual review of organizational psychology's 25-year retrospective concludes the science of workplace feedback 'is not yet a story of coherent and cumulative progress': definitions are generic, assumptions diverge across six disconnected research substreams, and simple universal rules about feedback effectiveness do not survive contact with organizational reality.\nevidence: crawled the full open-access review (anseel & sherf, annu. rev. organ. psychol. organ. behav. 12:19-43, published 2025-01-21). abstract states insights 'often appear disconnected from the way feedback is practiced and experienced in organizations'; the review calls for explicated assumptions and paradigms mirroring complex realities. the companion 2025 systematic review (heine, stouten & liden, j. organ. behav., 2025-10-26) reaches the same verdict for supervisor performance feedback: most studies fail even to specify feedback valence, and feedback quality/accuracy findings rest on inconsistent constructs.\nsource: a 25-year review of research on feedback in organizations: from simple rules to complex realities (anseel & sherf) | https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-110622-031927 | 2025-01-21 | academic\nimplication: tempering prior for the whole q7 branch: the general literature supplies directional moderators (task-focus, specificity, information content, source credibility) but no validated plug-in recipe, and effect heterogeneity is the norm. the project-agnostic foundation should therefore ship feedback design as parameterized hypotheses with built-in efficacy measurement (per-project a/b of feedback tiers against repeat-error rate), not as fixed doctrine imported from any single meta-analysis.\n\n### under incentives, relative-rank feedback is a double-edged lever: lab and field economics find rank feedback raises output in flat-wage settings (charness et al. 2014; tafkov 2013) but induces costly sabotage and cheating to improve rank that offsets the gains, and in at least one field experiment (barankay 2012) removing rank feedback improved performance.\nevidence: confirmed via the literature synthesis in a peer-reviewed leadership quarterly article ('feedback quality and performance in organisations', 2021), which states: charness et al. (2014) found 'offering relative rank feedback increases output... and subjects are willing to engage in costly sabotage and cheating activities to improve their relative rank, thus offsetting the positive effects'; barankay (2012) is cited as the exception where rank feedback hurt. primary charness/barankay texts not independently opened, so treated as well-sourced secondary evidence.\nsource: feedback quality and performance in organisations (leadership quarterly; synthesizing charness et al. 2014, tafkov 2013, barankay 2012) | https://www.sciencedirect.com/science/article/abs/pii/s1048984321000394 | 2021 (synthesizing 2012-2016 primaries) | academic\nimplication: for taxonomy q8 (gaming/monoculture under pay-linked evaluation): if autoqa outputs become visible rank or pass-rate leaderboards tied to pay, the literature predicts optimization of the metric (score-hacking, mimicry of known-passing templates) rather than quality. keep attempter-facing feedback private, criterion-referenced, and decoupled from visible peer ranking; note the tension with cochrane's top-peer-benchmark moderator (see disagreements) - benchmark against exemplar work, not against ranked people.\n\n- disagreement: feedback frequency: pre-2025 audit-and-feedback guidance (ivers 2012 cochrane, hysong 2006) held that repeated/more frequent delivery increases effect; the 2025 cochrane update finds repeated delivery associated with lower effect size. unresolved - could be confounding (repeated a&f deployed where problems persist) or genuine habituation.\n- disagreement: normative comparison: the 2025 cochrane update finds comparison to top peers or a benchmark increases behavior change in healthcare professionals, while fit (kluger & denisi) predicts normative comparison shifts attention to self and degrades performance, and incentive economics (charness 2014, barankay 2012) finds rank feedback triggers gaming/sabotage under competitive stakes. plausible reconciliation: comparison to an exemplar standard helps when stakes are professional-norm-based; comparison as interpersonal rank hurts when pay/status is on the line - but no study directly adjudicates this.\n- disagreement: self-assessment vs external feedback: dow 2012 found rubric self-assessment equal to external expert feedback for quality improvement (arguing feedback machinery could be replaced by surfaced rubrics), but the same study found self-ratings inflated by 1.8/9 points and education literature (winstone 2016 recipience work) holds self-assessment only works when later external verification is believed to occur. external qa may be load-bearing as a credibility backstop even if the information could be self-generated.\n- disagreement: effect magnitude: education meta-analyses report medium standardized effects (d approximately 0.48), while the healthcare a&f median is a small 2.7% absolute improvement on already-trained professionals. for skilled adult annotators the healthcare prior (small median, heterogeneous, concentrated in low performers) is likely the better calibration than the education prior.\n- gap: still no longitudinal study of qa feedback effects on paid annotation workers' repeat-error rates: dow 2012 explicitly did not measure long-term learning, and no 2024-2026 annotation-platform study of feedback efficacy was found (searched exa, tavily, openalex/crossref). the original sweep's gap is real; what changed is that adjacent causal literature (cochrane a&f 2025, crowdwork rct) transfers with stated caveats.\n- gap: no study found on ai-generated feedback to human annotators under pay-linked evaluation - the exact autoqa deployment condition. closest analogs are ai-feedback-to-employees work (e.g., tong et al. 2021 smj, disclosure reduces effect) which was not deep-dived here.\n- gap: charness et al. 2014 and barankay 2012 primaries were not opened (claims verified only through a peer-reviewed secondary synthesis); exact effect sizes for gaming-offset were not extracted.\n- gap: feedback-specificity tradeoff (goodman & wood 2004/2011: high specificity aids immediate performance but can impair exploration and transfer to novel cases) was identified in the anseel & sherf reference base but not independently verified - relevant to whether highly prescriptive autoqa feedback creates template-following monoculture (q8) and worth one follow-up read.\n- gap: the 2025 individual-differences meta-analysis (condrea & iliescu, ejwop, 2025-12-16) on reactions to feedback was located but is too new to have accessible full text; could sharpen per-attempter moderation (feedback orientation, self-esteem) but was not extractable.", "6": "domain: academic-judges\n\n## on hard, objectively-verifiable judging tasks (judgebench, iclr 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while gpt-4o scored 50.9-56.6% (near random), the best reward model (skywork-reward-gemma-2-27b) hit 64.3%, and fine-tuned judges like pandalm fell at or below random. [load-bearing]\nevidence: judgebench uses 350 rigorously verified response pairs across knowledge/reasoning/math/coding where correctness is checkable. verified numbers: gpt-4o vanilla 50.9% overall, arena-hard pipeline 56.6%, o1-preview 75.4% (85.7% math/coding), skywork-reward-27b 64.3%, reward models 60-64%, chateval multi-agent ~34%. fine-tuned preference judges plateau near random on knowledge/reasoning.\nsource: judgebench: a benchmark for evaluating llm-based judges (iclr 2025) + emergentmind topic synthesis (updated jan 2026) | https://arxiv.org/abs/2410.12784 | 2024-10 (iclr 2025; synthesis updated 2026-01) | academic\nimplication: the autoqa core judge should be a frontier reasoning model, not an off-the-shelf fine-tuned judge or scalar reward model, because annotation-qa requires reasoning about correctness, not preference matching. reasoning-model headroom (75% vs 51% for the same lab's non-reasoning model) is the single biggest capability lever.\n\n## standard llm-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial llms, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy. [load-bearing]\nevidence: guerdan, barocas, holstein, wallach, wu, chouldechova (neurips 2025 poster, abstract opened and confirmed). they formalize 'rating indeterminacy' - items where multiple ratings are reasonable - and show humans and llms resolve forced choices differently, so aggregated gold labels systematically mis-rank judges. they provide concrete recommendations for elicitation and aggregation.\nsource: validating llm-as-a-judge systems under rating indeterminacy (neurips 2025) | https://neurips.cc/virtual/2025/poster/117308 | 2025-09-19 | academic\nimplication: directly answers design q1/q2: the ground-truth construct cannot be a single forced pass/fail human label. the gold set must elicit response-set labels ('which verdicts are defensible?'), and the verdict ontology needs a first-class 'instructions underdetermine this case' class. validation against collapsed consensus labels will select the wrong judge configuration.\n\n## the largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including april-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected cohen's kappa is 33-41 percentage points lower on mt-bench - and judge rankings shift by up to 14 positions depending on which benchmark you use. [load-bearing]\nevidence: norman, rivera, hughes (arxiv 2606.19544, abstract opened and confirmed). four cohort-wide findings: universal kappa deflation; benchmark-dependent rankings (up to 14 position shifts across mt-bench/judgebench/rewardbench); a 'consistency-bias paradox' where test-retest reliability >0.95 coexists with position bias >0.10 in two production judges; verbosity bias small (<0.011) under a single controlled pairwise rubric. they distill a minimum viable validation protocol.\nsource: reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic\nimplication: answers design q1's metric question: ban raw percent-agreement and single-number accuracy from all autoqa reporting; standardize on chance-corrected statistics per benchmark. also answers q5: a judge that is perfectly self-consistent can still be systematically biased - consistency monitoring and bias audits are separate mandatory sensors, and per-project validation must use multiple gold-set designs since rankings do not transfer.\n\n## at the rubric level - the exact granularity autoqa would operate at - even frontier judges achieve only ~55-56% balanced accuracy on hard rubric-verification instances (gpt-4o 55.97%, claude-sonnet-4.5 55.65%), but rubric-level evaluation with explicit chain-of-thought reasoning beats checklist-level evaluation by 7-12 percentage points and reduces cross-judge variance. [load-bearing]\nevidence: rubriceval (arxiv 2603.25133, full alphaxiv overview read). binary satisfied/not-satisfied verification of atomic rubrics against responses; easy subset ~90% accuracy, hard subset ~55%. compositional (conditional-logic) instructions, role persona, and format structure rubrics hardest. their multi-model arbitration labeling pipeline (raf) reached 85.0% agreement with humans, cohen's kappa 0.702. reasoning-intensive models (o3) notably outperform standard instruct models.\nsource: rubriceval: a rubric-level meta-evaluation benchmark for llm judges in instruction following | https://www.alphaxiv.org/overview/2603.25133 | 2026-03-26 | academic\nimplication: answers design q3 granularity: decompose to atomic criteria and require per-criterion explicit reasoning (worth +7-12pp), but budget for the cost multiplier and expect a hard residual class (~45% error on contested rubrics) that must route to humans. easy/hard stratification should be built in: judge consensus at the coarse pass identifies which claims need escalation - the raf arbitration cascade is a directly reusable architecture for gold-label production.\n\n## on subjective rubrics, llm judges' evaluation axis is nearly orthogonal to the human axis (87-89 degrees vs 78-81 degrees human-to-human), judges use only 0.3-0.5x the human score spread, and inter-llm agreement (r~0.35) exceeds llm-human agreement (r~0.27-0.32) - while on a rubric with a verifiable factual answer the same judges fall back into the human range (58.5 degrees, r=0.519).\nevidence: mukherjee, hamna, bali, sitaram (arxiv 2606.03043, abstract opened). 41 llm judges, 4 datasets, 8 indic languages, geometric analysis with bootstrap cis. fine-tuning/preference optimization recovers score spread (0.32 to 1.08) but barely moves the axis; only post-hoc calibration on a small human-anchored set improved all rubrics. scope caveat: indic-language community-health data, but the verifiable-vs-subjective contrast is the mechanism of interest.\nsource: the geometry of llm-as-judge: why inter-llm consensus is not human alignment | https://arxiv.org/abs/2606.03043 | 2026-06-02 | academic\nimplication: multi-judge ensembles and inter-judge agreement are not validity evidence on subjective axes - consensus can reflect a shared collapsed subspace. the objective-subjective boundary (design q2) is empirically real and large: verifiable criteria are judge-autonomous territory; subjective criteria need human-anchored calibration sets, not more judges.\n\n## single-trial llm judging is measurably noisy: across 29 tasks with 50 repeated trials, pairwise preferences flipped on average 13.6% of the time (28% of questions exceeded 20% flip rate), semantically equivalent prompt templates changed majority outcomes in 25% of tested cases, and ~11 repeated trials were needed for majority vote to recover the reference verdict with 95% probability.\nevidence: yagubyan (arxiv 2606.13685, abstract opened). gpt-4o-mini and gpt-4.1-mini judges; also found significant first-position bias (72% a-majority, p=0.024), cross-judge agreement only 76% (kappa 0.51), and a pairwise-pointwise gap where judges pick winners even when their own scalar scores show no meaningful difference. caveat: both judges from one provider, mini-tier models; the reliability-without-validity cohort shows some production judges reach test-retest >0.95.\nsource: the coin flip judge? reliability and bias in llm-as-a-judge evaluation | https://arxiv.org/abs/2606.13685 | 2026-04-23 | academic\nimplication: answers design q5's flip-rate fork with 'both': k-sample voting (k around 5-11) plus position randomization is baseline hygiene, and per-item flip-rate is a cheap, free signal of criterion underspecification - items with high vote entropy are exactly the 'instructions underdetermine this' class. a perturbation-robustness harness is justified as a ship gate.\n\n## the broadest llm-vs-human-judge study on existing human-annotated datasets (judge-bench, 20 datasets) found the best model (gpt-4o) reached only kappa = 0.28 +/- 0.32 on categorical judgments and spearman rho = 0.50 +/- 0.21 on graded ones, with enormous task-to-task variance - and llms agreed more with non-expert annotators than with experts.\nevidence: bavaresco et al., 'llms instead of human judges?' (acl 2025). structured/rule-like attributes correlate reliably; subjective attributes (engagingness) poorly; safety judgments produced negative correlations due to guardrail refusals. authors conclude task-specific human validation is essential before deploying llm judges. read via detailed paper-note secondary source; consistent with the acl publication.\nsource: llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks (judge-bench) | https://en.papernotes.org/acl2025/llm_nlp/llm_vs_human_judges_study/ | 2025-07 (acl 2025) | academic\nimplication: chance-corrected agreement with humans on subjective axes is far lower than the practitioner '85% agreement' narrative. the agree-more-with-non-experts finding is a red flag for the expertise-gap root cause: an uncalibrated judge may replicate crowd-level judgment, not expert judgment - expert-adjudicated (not crowd-consensus) gold labels are required for validation.\n\n## preference leakage is a measured, large contamination bias: a judge favors outputs of models trained on data from a related generator by up to 28.7 percentage points (preference leakage score), with a relatedness gradient - same model 23.6% average, fine-tuned descendants 19-22%, same family 2.8-8.9% - and judges cannot detect their own students (near-chance) while a bert classifier can (82.4%).\nevidence: li et al., iclr 2026 (arxiv 2502.01534). sft transmits the bias strongly (23.6% avg pls) vs dpo (5.2%); effect scales roughly linearly with synthetic-data mixing ratio with no threshold; style/format removal cuts pls sharply; only contextual calibration mitigated it (17.8 to 7.3); prompting-based mitigation failed.\nsource: preference leakage: a contamination problem in llm-as-a-judge (iclr 2026) | https://arxiv.org/abs/2502.01534 | 2026-02 (iclr 2026; arxiv 2025-02) | academic\nimplication: answers design q5 self-preference: when attempters critique (or produce with the help of) outputs from model family x, the judge should not be family x or its distillation lineage - and since lineage is often undisclosed, cross-family judge routing plus style-stripping/canonicalization is the practical defense. prompt-level debiasing will not fix this.\n\n## dedicated trained judges advanced substantially in 2025: meta's j1 (rl-trained thinking-judge, may 2025) at 32b outperforms o1-mini, o3, and 671b deepseek-r1 on some judge benchmarks; compassjudger-2-7b (july 2025) matches far larger generalists on judge benchmarks; skywork-reward-v2-llama-3.1-8b (july 2025) topped seven reward benchmarks including judgebench.\nevidence: j1 arxiv 2505.10320 (verified): unified verifiable-reward rl over 22k synthetic pairs, mitigates position bias by construction, gains from test-time self-consistency. compassjudger-2 arxiv 2507.09104: verifiable-reward + margin-loss training; 32b scores 80.9% accuracy on judgerbenchv2. skywork-reward-v2 arxiv 2507.01352: 26m human-ai-curated preference pairs; 8b model led rewardbench v1/v2, ppe, rm-bench, judgebench among reward models.\nsource: j1: incentivizing thinking in llm-as-a-judge via reinforcement learning (with compassjudger-2, skywork-reward-v2) | https://arxiv.org/abs/2505.10320 | 2025-05 to 2025-07 | academic\nimplication: small trained judges are now viable for high-volume, well-specified sub-checks (cost tier), but their wins are on preference-style benchmarks; for reasoning-heavy annotation qa the judgebench evidence still favors frontier reasoning models. a two-tier design (cheap trained judge for triage, frontier reasoner for verdicts) is supported; the j1 training recipe (verifiable rewards, position-bias-symmetric training) is reusable if a custom judge is ever trained.\n\n## rewardbench 2 (ai2, june 2025) made reward-model evaluation ~20 points harder via best-of-4 format (random = 25%) and unseen human prompts; top models score below 40% on precise instruction following, yet benchmark scores correlate 0.87 (pearson) with downstream best-of-n performance across 113 reward models.\nevidence: arxiv 2506.01937 (verified via abstract, ai2 dataset page, and lambert's analysis post). 1,865 cases across factuality, focus, math, precise if, safety, ties. lambert (author) notes generative llm-as-judge approaches remain 'weaker than expected relative to standard reward models' at output ranking, and that frontier models still fail trivial rankings ('name a color in the rainbow').\nsource: rewardbench 2: advancing reward model evaluation | https://arxiv.org/abs/2506.01937 | 2025-06-02 | academic\nimplication: precise-instruction-following is the weakest judged capability (<40% at best-of-4) - exactly the 'is this annotation compliant with written project instructions?' skill autoqa needs. do not assume instruction-compliance checking is a solved sub-problem; it needs its own gold set, and deterministic/programmatic checks should absorb whatever can be compiled to rules.\n\n## in expert-knowledge domains, llm judges agreed with subject-matter experts only 68% (dietetics, registered dietitians) and 64% (mental health, clinical psychologists) on overall pairwise preference, with agreement varying further across domain-specific aspect questions.\nevidence: szymanski et al. (arxiv 2410.20266; acm iui 2025, ~220 citations). mixed-methods pairwise comparison study; authors conclude llms alone lack the depth for complex knowledge-specific tasks and experts must stay in the evaluation loop. full abstract text opened and confirmed.\nsource: limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks | https://arxiv.org/abs/2410.20266 | 2024-10 (iui 2025) | academic\nimplication: for projects requiring genuine domain expertise (root cause 2), expect judge-expert agreement in the 60s on holistic preference - below any reasonable autonomy bar. expertise-heavy axes belong in the judge-assisted or human-only lane, and the axis-triage classifier (design q2) should treat 'requires specialized professional knowledge' as an explicit routing feature.\n\n## llm judges are measurably less reliable on long-form outputs: longjudgebench (june 2026) finds current judges unstable across real-world long-form scenarios requiring document-level assessment of organization, coverage, and cross-section consistency, and rubrics or references help but are 'not always sufficient'.\nevidence: chen et al., arxiv 2606.01629 (abstract opened and confirmed). first meta-evaluation benchmark targeting long-form judging specifically; systematically evaluates judges across base models and judging protocols; code public. complements the short-form-centric older benchmarks (mt-bench, llmbar, judgebench).\nsource: benchmarking llm-as-a-judge for long-form output evaluation (longjudgebench) | https://arxiv.org/abs/2606.01629 | 2026-06-01 | academic\nimplication: attempter writeups/critiques are long-form documents; holistic single-pass judging of them inherits this instability. supports the decomposition-first architecture (per-claim verification then aggregation) over holistic item scoring, and flags 'global coherence of the writeup' as a check that per-claim decomposition alone will miss (checklist myopia risk from design q3 is real in both directions).\n\n## tools & artifacts\n- judgebench: meta-evaluation benchmark (350 verified response pairs, objective correctness ground truth) with live huggingface leaderboard | https://github.com/scalerlab/judgebench | canonical hard test for whether a candidate autoqa judge can detect actual errors rather than preferences; leaderboard at huggingface.co/spaces/scalerlab/judgebench\n- rewardbench 2: ai2's best-of-4 reward model benchmark (1,865 cases, 6 subsets incl. factuality and precise instruction following) | https://huggingface.co/datasets/allenai/reward-bench-2 | the precise-if and factuality subsets are the closest public proxies for instruction-compliance and grounding checks in autoqa\n- rubriceval + rubric arbitration framework (raf): rubric-level meta-evaluation benchmark plus a multi-model arbitration pipeline for producing gold labels (85% human agreement, kappa 0.702) | https://www.alphaxiv.org/overview/2603.25133 | raf's disagreement-cascade (base judges -> rationale re-evaluation -> reasoning meta-judges) is a directly reusable architecture for autoqa gold-set production and escalation routing\n- j1 (meta): rl training recipe for thinking-judges using verifiable rewards over synthetic pairs; position-bias-mitigating by construction | https://arxiv.org/abs/2505.10320 | the reference recipe if a custom project-tuned judge is ever trained; also evidence that judge chain-of-thought quality is the optimizable variable\n- compassjudger-2 (7b/32b, open weights): opencompass generalist judge models trained with verifiable rewards + margin loss; includes judgerbenchv2 (10k questions, 10 scenarios) | https://github.com/open-compass/compassjudger | best open-weight generative judge candidate for a cheap triage tier\n- skywork-reward-v2 (0.6b-8b, open weights): scalar bradley-terry reward models trained on 26m human-ai-curated preference pairs | https://github.com/skyworkai/skywork-reward-v2 | state-of-art scalar scorers; useful for high-volume ranking/filtering but not for criteria-grounded verdicts with rationales\n- atla selene mini (8b, open weights): general-purpose fine-tuned evaluator model (jan 2025), promptable with custom criteria | https://arxiv.org/abs/2501.17195 | open-weight promptable evaluator baseline; claims high agreement with domain experts on financebench/craft-md but predates the 2025-26 judge generation\n- longjudgebench: first meta-evaluation benchmark for judging long-form outputs (june 2026), code public | https://github.com/cjj826/longjudgebench | test harness for the long-form writeup judging autoqa must do; measures rubric/reference sufficiency\n- llm-as-a-judge.github.io: maintained living survey and paper list for the llm-as-judge field (tracks 2025-2026 papers incl. preference leakage, toolprmbench) | https://llm-as-a-judge.github.io | ongoing monitoring source for judge research; companion to gu et al. 'a survey on llm-as-a-judge' (the innovation, 2026, ~1,667 citations)\n- minimum viable validation protocol (from arxiv 2606.19544): a distilled judge-validation protocol: chance-corrected agreement + consistency protocol + bias audit, from the 541k-judgment study | https://arxiv.org/abs/2606.19544 | template for the autoqa per-project ship gate: what to measure before any project's judge config goes live\n\n## disagreements\n- dedicated trained judges vs frontier generalists: judgebench (iclr 2025) shows fine-tuned judges and reward models plateau near random-to-64% on hard reasoning-based judging while frontier reasoning models lead (o1-preview 75.4%); but j1 (meta, 2025-05), compassjudger-2 (2025-07), and skywork-reward-v2 (2025-07) each claim small trained models beat o1-mini/o3-class models on judge/reward benchmarks; and lambert (rewardbench 2 author, 2025-06) argues scalar reward models still beat generative llm judges at output ranking. the conflict is largely benchmark-dependent (preference-style vs verifiable-reasoning tasks) - no source resolves which wins for criteria-based qa of human annotations, which resembles neither exactly.\n- judge-human agreement levels: practitioner sources claim ~85% agreement 'higher than human-human agreement' (confident ai, 2026) and '>80%, matching human-to-human levels' (galileo); academic chance-corrected measurements show best-model kappa of 0.28 +/- 0.32 across 20 tasks (judge-bench, acl 2025), universal 33-41pp kappa deflation vs raw agreement (arxiv 2606.19544, 2026-06), and near-orthogonal judge-vs-human evaluation axes on subjective rubrics (arxiv 2606.03043, 2026-06). much of the gap is percent-agreement vs chance-corrected statistics plus task subjectivity - the practitioner numbers are not wrong so much as unfalsifiably framed.\n- verbosity bias magnitude: the 2606.19544 large-scale audit found verbosity bias small (<0.011) across 21 judges under a single controlled pairwise rubric, whereas the broader literature (zheng 2023 mt-bench, gu et al. survey 2026, arxiv 2510.12462) treats verbosity bias as a major systematic failure. possible reconciliation: bias magnitude depends heavily on rubric structure and elicitation format, meaning it is a controllable design variable rather than a constant.\n- run-to-run consistency: the coin flip judge (2026-04) reports 13.6% average pairwise flip rate and recommends ~11-vote majorities (mini-tier openai judges only), while reliability without validity (2026-06) measured test-retest reliability >0.95 for some production judges - consistency is highly judge-specific, and the latter paper explicitly warns high consistency coexists with severe position bias (consistency is not validity).\n\n## gaps\n- no benchmark meta-evaluates judges on the autoqa task itself - verifying human-written evaluative claims (grounding of praise, evidence-conclusion entailment in critiques, severity calibration). closest proxies are rubric-verification (rubriceval), factuality judging (rewardbench 2 factuality), and critique evaluation inside j1/compassjudger training; the positive-claim/'warranted vs empty praise' verification question (design q3) has no published precision/recall numbers anywhere i could find.\n- could not obtain a verified july-2026 leaderboard snapshot for the newest frontier models (gpt-5.x, claude opus 4.x, gemini 3) on judgebench or rewardbench 2 - both official leaderboards render via dynamic js (huggingface spaces) and did not yield data to fetches; the largest systematic evaluation covering 'the april 2026 frontier' (arxiv 2606.19544) reports cohort-level findings but does not name per-model winners in accessible text. best-available anchor numbers are therefore early-2026 or older.\n- no single study reports judge-vs-adjudicated-expert agreement broken out per evaluation-axis type (factual grounding vs instruction compliance vs tone vs holistic) - the per-axis agreement ceiling matrix needed for design q5's lane assignment must be assembled from fragments (rewardbench 2 subsets, judge-bench task variance, rubriceval categories) or measured in-house.\n- human inter-reviewer reliability floors (krippendorff's alpha) for annotation-qa reviewers specifically were out of this domain's scope and not covered by the judge meta-evaluation literature; the rating-indeterminacy framework (neurips 2025) is the right machinery but its 11 tasks are generic rating tasks, not training-data qa.\n- atla (selene) company status as of mid-2026 unverified - selene mini (8b, open weights, jan 2025) claims remain reproducible from the model card, but i found no 2026 successor release, so treat selene as a frozen artifact rather than a maintained product line.\n\n## verifications\n- claim: on hard, objectively-verifiable judging tasks (judgebench, iclr 2025), frontier reasoning models dominate dedicated judge models: o1-preview scored 75.4% overall while gpt-4o scored 50.9-56.6% (near random), the best reward model (skywork-reward-gemma-2-27b) hit 64.3%, and fine-tuned judges like pandalm fell at or below random.\n verdict: confirmed | every quantitative claim checks against the paper's own tables (arxiv:2410.12784 v2, iclr 2025 camera-ready), read directly from the pdf: gpt-4o vanilla 50.86% / arena-hard 56.57% overall (table 1), o1-preview 75.43% overall with 85.71% on both math and coding (table 2), skywork-reward-gemma-2-27b 64.29% as best reward model (table 3, full range 59.43-64.29, paper says \"approximately 59% to 64%\" so the claim's \"60-64%\" is a minor round-up at the low end), pandalm 13.14% (far below random; paper: all fine-tuned judges except skywork significantly below random), chateval 34.00%. 350 verified pairs across knowledge (154)/reasoning (98)/math (56)/coding (42) confirmed - note this is the gpt-4o-generated split; a separate 270-pair claude-3.5-sonnet split exists. dates confirmed: arxiv v1 2024-10-16, iclr 2025, emergentmind synthesis updated 2026-01-09. two contextual nuances, neither refuting: (1) in the v2 tables o1-preview is not the single best judge - o3-mini (high) scores 80.86% and deepseek-r1 73.14%, which strengthens rather than weakens the \"reasoning models dominate\" thesis; (2) 2025-2026 follow-on work (j1, rm-r1, meta-judging, rrd) has pushed judgebench accuracies to 77-81%+, so \"gpt-4o near random\" describes the late-2024 snapshot of non-reasoning judges, not the current frontier. the claim as stated about the paper's findings is accurate.\n- claim: standard llm-judge validation via forced-choice human gold labels is provably biased when rating criteria admit multiple valid interpretations: across 11 real-world rating tasks and 9 commercial llms, forced-choice validation selected judge systems performing up to 31% worse than validation using multi-label 'response set' ratings that model indeterminacy.\n verdict: partially_confirmed | core claim confirmed against the actual neurips poster page (https://neurips.cc/virtual/2025/poster/117308): authors, rating-indeterminacy framing, forced-choice vs multi-label response-set elicitation, 11 tasks, 9 commercial llms, and the \"as much as 31% worse\" figure all match verbatim. two corrections: (1) \"provably biased\" overstates the paper's own language - it says differing resolution of indeterminacy \"can heavily bias\" validation, and the 31% gap is empirical, not a theorem (the theory links performance measures/elicitation schemes); (2) the 2025-09-19 date is not shown on the poster page - plausibly the acceptance date, but unverified; the paper dates to arxiv march 2025 and neurips dec 2025, with an ml@cmu blog post 2025-12-09. search for later work (as of july 2026) found follow-on papers (irt-based judge reliability, arxiv 2602.00521; grading-scale alignment, arxiv 2601.03444) that build on, not refute or supersede, this result. code: https://github.com/lguerdan/indeterminacy.\n corrected: guerdan, barocas, holstein, wallach, wu & chouldechova (neurips 2025; arxiv:2503.05965, first posted march 2025) show that standard llm-judge validation via forced-choice human gold labels can be heavily biased when rating criteria admit multiple valid interpretations (\"rating indeterminacy\"): across 11 real-world rating tasks and 9 commercial llms, forced-choice-based validation selected judge systems performing as much as 31% worse than those selected by their framework using multi-label \"response set\" ratings. the bias mechanism is supported by a theoretical framework, but the 31% suboptimality figure is an empirical finding, not a proof.\n- claim: the largest systematic judge meta-evaluation to date (21 judges, 9 providers, ~541k judgments, including april-2026 frontier models) found raw exact-match agreement universally overstates judge ability - chance-corrected cohen's kappa is 33-41 percentage points lower on mt-bench - and judge rankings shift by up to 14 positions depending on which benchmark you use.\n verdict: confirmed | fetched https://arxiv.org/abs/2606.19544 directly. every quantitative element checks out against the abstract: 21 judges from 9 providers, 118 runs / ~541,000 judgments across mt-bench/judgebench/rewardbench; kappa deflation of 33-41 percentage points on mt-bench across all 21 models; ranking shifts up to 14 positions across benchmarks; consistency-bias paradox (test-retest reliability >0.95 with position bias >0.10 in two production judges); verbosity bias <0.011; minimum viable validation protocol; april 2026 frontier models included. submission date june 17, 2026 matches; authors norman, rivera, hughes match. \"largest to date\" is the authors' own framing, but two independent searches (july 14, 2026) found no larger or superseding meta-evaluation; follow-on work (aura arxiv 2606.19714, apple correlated-panels paper, \"below the reliability floor\" on openreview) complements rather than supersedes it. caveats: preprint with 0 citations; authors themselves limit dataset authority to the march-april 2026 measurement window due to hosted-endpoint drift.", "7": "domain: annotation-quality\n\n## roughly one-third of crowdworkers used llms on an llm-advantaged text-production task (33-35% on prolific, july 2023; ~34% prolific self-report in a 2025 follow-up by zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%). [load-bearing]\nevidence: veselovsky et al. ran two preregistered prolific studies (n=168 and n=720, medical-abstract summarization) with a calibrated classifier plus self-report and keystroke heuristics; prevalence 33.3-35.4% baseline (95% cis ~[26%,43%]); a 3x3 factorial of requests x hurdles halved but did not eliminate use. the june 2026 community survey (arxiv 2606.04924) corroborates: 30-40% estimated reliance and 34% prolific self-report, and notes mitigations 'reduce rather than eliminate' llm-assisted responses.\nsource: prevalence and prevention of large language model use in crowd work (cacm; arxiv 2310.15683) | https://arxiv.org/abs/2310.15683 | 2025-02-18 | academic\nimplication: autoqa must assume a contamination base rate of ~15-35% on any text-heavy attempter task even after friction measures; per-item content detection and 'please don't' policies cannot be the primary defense, and mitigation friction has a measurable quality cost that the qa layer must not misattribute to attempter skill.\n\n## an autonomous llm agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused ai-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025. [load-bearing]\nevidence: westwood (pnas, nov 20 2025; 43,800 evaluations across 9 llms) shows instruction-following checks, logic puzzles, recaptcha, and rare-event probes all fail; persona calibration even masks superhuman-accuracy tells. recommended defenses are entirely provenance/economics-based: identity revalidation, per-worker throttling, completion-history transparency, pass/fail history, location/vpn checks - 'there is no magical fix.' the 2026 crowdsourcing survey (arxiv 2606.04924) independently confirms detectors degrade under paraphrase and out-of-domain, and mixed human-ai authorship is harder still.\nsource: the potential existential threat of large language models to online survey research (pnas 122(47)) | https://www.pnas.org/doi/10.1073/pnas.2518075122 | 2025-11-20 | academic\nimplication: directly answers taxonomy q8: wholesale-llm-submission defense must be economics/provenance-based (rate limits, work-history consistency, randomized deep audits, process telemetry), not per-item text classification; any autoqa spam layer built on stylistic detection will be silently beaten within the cost asymmetry.\n\n## correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. [load-bearing]\nevidence: elangovan et al. (amazon), iclr 2025 (arxiv 2410.03775, opened and verified): krippendorff's alpha / kappa were designed for human-human reliability and their assumptions break for machine labels; the paper's fix is (1) stratify meta-evaluation by human-label uncertainty, (2) use binned jensen-shannon divergence for perception-type (inherently subjective) tasks, (3) report 'perception charts' instead of one correlation. code open-sourced (amazon-science/beyondcorrelation).\nsource: beyond correlation: the impact of human uncertainty in measuring the effectiveness of automatic evaluation and llm-as-a-judge (iclr 2025) | https://arxiv.org/abs/2410.03775 | 2025-05-01 | academic\nimplication: answers taxonomy q1 directly: meta-evaluation must stratify gold/validation items by measured human agreement level and report per-stratum judge performance; 'accuracy vs. humans' as a single number should be banned from reporting, and judge validity claims are only meaningful on the high-human-agreement stratum.\n\n## in human preference/rating data, the majority of annotator disagreements are attributable to task underspecification and response-style preferences - not random noise and not primarily expertise gaps - and standard aggregation (bradley-terry reward modeling) plus llm-as-judge evaluation both fail to account for this divergence. [load-bearing]\nevidence: zhang et al., icml 2025 (arxiv 2410.14632, opened and verified): built a 10-category, 4-class taxonomy of disagreement sources on preference datasets; abstract states 'the majority of disagreements are due to factors such as task underspecification or response style,' explicitly challenging the noise assumption; they show llm evaluations are heavily swayed by divisive style features and develop divergence-identification methods. converges with jiang & de marneffe (tacl 2022) who found the same structure in nli.\nsource: diverging preferences: when do annotators disagree and do models know? (icml 2025) | https://arxiv.org/abs/2410.14632 | 2025-07-15 | academic\nimplication: supports root-cause-1 primacy and taxonomy q2: the verdict ontology needs a first-class 'instructions underdetermine this case' class detected via measured divergence, and a cheap variance-attribution pilot is empirically justified because published decompositions find underspecification (fixable by rubric compilation) dominates over noise (fixable by averaging).\n\n## even on a nominally objective task (nli), annotator disagreement decomposes into a 10-category taxonomy across three high-level sources - uncertainty in sentence meaning, underspecification in guidelines, and annotator behavior - proving that a fraction of inter-reviewer variance is item-intrinsic and survives perfect rubric operationalization.\nevidence: jiang & de marneffe, tacl 2022, developed and hand-applied the taxonomy to high-disagreement nli items; disagreement persisted among trained annotators on items with genuine interpretive ambiguity. d3code (emnlp 2024; 4.5k sentences, 4k+ annotators, 21 countries) adds that residual variance is systematically structured by annotators' moral values and region, not random.\nsource: investigating reasons for disagreement in natural language inference (tacl 10) | https://aclanthology.org/2022.tacl-1.78/ | 2022-12-01 | academic\nimplication: the target metric cannot be raw agreement: some disagreement is item-intrinsic signal. the variance-attribution pilot should code disagreements against a taxonomy like this to split rubric-fixable variance (underspecification) from irreducible variance (ambiguity/values) before setting build order.\n\n## chance-corrected agreement coefficients (cohen's kappa, and analogously krippendorff's alpha) can be near zero despite very high raw agreement when class prevalence is extreme (the feinstein-cicchetti paradoxes), so under the 90%+ pass rates typical of qa pipelines, single-coefficient reliability reporting is misleading in the opposite direction from raw percent agreement.\nevidence: feinstein & cicchetti (j clin epidemiol, 1990) formalized the two kappa paradoxes (high agreement/low kappa under imbalance; asymmetric marginals inflating kappa); the effect is prevalence-driven, extensively replicated (quarfoot & levine, am. statistician 2016; gwet's ac1 literature). this is exactly the regime of qa pass/fail data with low failure base rates.\nsource: high agreement but low kappa: i. the problems of two paradoxes (feinstein & cicchetti) | https://doi.org/10.1016/0895-4356(90)90158-l | 1990-01-01 | academic\nimplication: answers taxonomy q1 stats question: standardize on reporting a chance-corrected coefficient plus per-class recall at fixed prevalence plus cis; ban both raw percent-agreement and any single coefficient in isolation, because at extreme pass rates alpha/kappa collapse even for a well-functioning judge.\n\n## classical label-aggregation models (dawid-skene 1979, mace 2013 with its explicit spammer latent variable, glad) all assume annotators are conditionally independent given the true label - an assumption measurably violated when the 'annotators' are llm judges sharing data/architectures/prompts, causing miscalibrated posteriors and confidently wrong aggregate verdicts; feb 2026 work replaces this with dependence-aware ising-model aggregation.\nevidence: balasubramanian, podkopaev & kasiviswanathan (amazon/uc davis, arxiv 2601.22336, feb 2 2026) state the ci assumption 'is often violated by llm judges due to shared data, architectures, prompts, and failure modes' and that ignoring dependencies 'can yield miscalibrated posteriors and even confidently incorrect predictions'; they build aggregation via ising models. mace (hovy et al., naacl 2013) remains the canonical spam-aware human aggregator, with production implementations in toloka's crowd-kit.\nsource: dependence-aware label aggregation for llm-as-a-judge via ising models (arxiv 2601.22336) | https://arxiv.org/pdf/2601.22336 | 2026-02-02 | academic\nimplication: if autoqa uses k-sample voting or multi-judge ensembles, naive dawid-skene/majority aggregation will overstate confidence because same-family judge samples share failure modes; confidence estimates must model correlated errors (or use genuinely different model families), and mace-style competence/spam latent variables remain the right tool for the human-reviewer side.\n\n## llm contamination in low-dimensional annotation data (multiple-choice labels, ratings) can be evaluated at the worker level without any ground truth, using peer-prediction scores that condition on llm-generated reference labels - a training-free mechanism with theoretical guarantees under an llm-collusion model, published at neurips 2025 - because text-based detectors are inapplicable to short label outputs.\nevidence: zhang, pang, zhu & liu (neurips 2025; arxiv 2506.06991, opened and verified): existing detectors need high-dimensional text; their mechanism scores the informativeness of worker answers via inter-worker correlation conditioned on a subset of llm labels, explicitly modeling collusion (multiple workers pasting from the same llm), and empirically detects low-effort cheating on real crowdsourcing datasets. scoring is per-worker, not per-item.\nsource: evaluating llm-contaminated crowdsourcing data without ground truth (neurips 2025) | https://arxiv.org/abs/2506.06991 | 2025-12-01 | academic\nimplication: for rating/label-type attempter outputs where stylistic detection is impossible, the viable contamination sensor is worker-level statistical scoring (cross-worker correlation vs. an llm-conditioned baseline) run continuously - reinforcing that provenance/behavioral defense operates at the annotator level, not the item level.\n\n## the nlp field has institutionalized disagreement-as-signal: the third learning-with-disagreements shared task (lewidi-2025, emnlp 2025) standardizes dual evaluation - soft-label (predict the population label distribution, wasserstein/manhattan distance) and perspectivist (recover individual annotators' labels) - across four datasets that all ship per-annotator labels rather than adjudicated gold.\nevidence: leonardelli et al. (arxiv 2510.08460, oct 2025; le-wi-di.github.io) describe the two complementary paradigms and public leaderboard; lineage runs from aroyo & welty's 'truth is a lie' perspectivism through dices (neurips 2023 safety dataset with up to 100+ raters/item) and jury learning (gordon et al., chi 2022, which composes verdicts from explicitly selected rater subpopulations); frenda et al.'s survey (lrev 2024) documents the dataset ecosystem.\nsource: lewidi-2025 at nlperspectives: third edition of the learning with disagreements shared task | https://arxiv.org/pdf/2510.08460 | 2025-10-09 | academic\nimplication: the autoqa data model should preserve rater identity and full label distributions (never only adjudicated verdicts), and for judgment-laden axes the judge's target should be the distribution/jury composition, not a forced scalar - this is now the field-standard evaluation contract, giving ready-made metrics (wasserstein/manhattan soft-label distance) for 'consistency-of-application' monitoring.\n\n## gold/test labels themselves are wrong at material rates - a lower-bound average of 3.3% label errors across 10 canonical benchmarks (at least 6% in the imagenet validation set), validated by human review of confident-learning-flagged candidates (51% flag precision) - and error rates this size are enough to flip model rankings.\nevidence: northcutt, athalye & mueller (neurips 2021 datasets & benchmarks; arxiv 2103.14749, opened and verified): algorithmic flagging via confident learning + crowdsourced verification; resnet-18 beats resnet-50 on corrected imagenet if mislabeled prevalence rises just 6%. cleanlab is the maintained open-source implementation.\nsource: pervasive label errors in test sets destabilize machine learning benchmarks (neurips 2021 d&b) | https://arxiv.org/abs/2103.14749 | 2021-11-01 | academic\nimplication: answers taxonomy q1's false-agreement question in the only measured form available: several percent of any 'gold' set is wrong, so judge-vs-gold disagreement at that rate is uninformative; the gold set needs its own standing audit channel (confident-learning-style flagging + expert re-adjudication) as a permanent component, and judge certification margins must exceed the gold error floor.\n\n## showing annotators llm suggestions makes them anchor on the suggestions - significantly shifting the label distribution versus unassisted baseline, raising self-reported confidence without making them faster - and evaluating models against llm-assisted labels significantly inflates reported model performance.\nevidence: schroeder, roy & kabbara (findings of acl 2025, july 2025; pre-registered, 350 annotators, 7,000 annotations, 4 conditions x 2 models x 2 datasets, opened and verified): annotators 'strongly took the llm suggestions'; the paper warns homogenization/lessened variation is a direct risk when llms are inserted into annotation loops, and that 'human-approved' llm-assisted labels change downstream conclusions.\nsource: just put a human in the loop? investigating llm-assisted annotation for subjective tasks (findings of acl 2025) | https://aclanthology.org/2025.findings-acl.1323/ | 2025-07-27 | academic\nimplication: direct evidence for taxonomy q8 monoculture and q9 induced drift: if autoqa feedback (or any ai verdict) is visible to attempters/reviewers before they judge, distributions converge on judge-pleasing labels and self-evaluation inflates - so human touches must be blind-first, and population-level label-distribution drift is the right monoculture sensor.\n\n## rlvr-style reasoning training degrades an llm's ability to model human annotator disagreement, while naive chain-of-thought on rlhf models improves it - evaluated across 60 setups on 3 tasks - meaning stronger 'reasoning' judges are measurably worse at knowing when humans legitimately diverge.\nevidence: ni et al., eacl 2026 main (arxiv 2506.19467 v3, jan 12 2026, opened and verified): systematic evaluation over model sizes, distribution-expression and steering methods using variance-correlation and distributional-alignment metrics; authors explicitly warn about 'the potential risk of replacing human annotators with reasoning llms' where disagreement carries signal; author communication adds that small fine-tuned models (modernbert-class) with human labels often beat large reasoning models at this.\nsource: can reasoning help large language models capture human annotator disagreement? (eacl 2026) | https://arxiv.org/abs/2506.19467 | 2026-01-12 | academic\nimplication: for the 'instructions-underdetermine / route-to-human' triage lane, do not assume the strongest reasoning judge is the best ambiguity detector: disagreement-prediction may need a dedicated small fine-tuned model (or cot-on-rlhf configuration) separate from the verdict judge.\n\n## tools & artifacts\n- cleanlab (confident learning): open-source library implementing confident learning for algorithmic label-error detection in labeled datasets | https://github.com/cleanlab/cleanlab | ready-made machinery for the standing gold-set audit channel: flag suspect gold/qa labels for expert re-adjudication (51% flag precision measured in the neurips 2021 study).\n- crowd-kit (toloka): maintained python library of aggregation/truth-inference algorithms: dawid-skene, glad, mace, wawa, zerobasedskill, ibcc/ebcc | https://github.com/toloka/crowd-kit | production implementations for aggregating multiple human reviewers (and multi-judge samples) with per-worker competence and spammer modeling - but note the conditional-independence caveat for same-family llm judges.\n- lewidi-2025 datasets + leaderboard: four benchmark datasets with per-annotator labels plus standardized soft-label (wasserstein/manhattan) and perspectivist evaluation metrics, emnlp 2025 shared task | https://le-wi-di.github.io/ | off-the-shelf metrics and data format for evaluating disagreement-aware judge components and for the 'consistency-of-application vs agreement-with-truth' distinction.\n- dices dataset (google): diversity-in-conversational-ai-safety dataset: 990 conversations with dense parallel safety ratings (up to 100+ demographically diverse raters per item) | https://github.com/google-research-datasets/dices-dataset | reference dataset for measuring how much rater variance survives on a judgment-laden axis with a fixed rubric - useful for calibrating the variance-attribution pilot.\n- beyondcorrelation (amazon science): code for uncertainty-stratified meta-evaluation of automatic evaluators: perception charts, binned jensen-shannon divergence | https://github.com/amazon-science/beyondcorrelation | directly implements the recommended meta-evaluation reporting standard (stratify by human agreement) for certifying the autoqa judge.\n- can crowdsourcing survive the llm era? (community survey): june 2026 arxiv survey (2606.04924) synthesizing prevalence, detection, and mitigation literature on llm use in human data collection | https://arxiv.org/html/2606.04924v1 | most current single map of the llm-contamination literature; documents detector degradation under paraphrase and the state of mitigation practice as of mid-2026.\n\n## disagreements\n- detectability of llm-generated submissions: veselovsky et al. (2023/2025) achieved usable classifier-based detection on a long-form text task, and tu delft's prompt-injection work (tochi 2025) claims in-instrument countermeasures work; westwood (pnas 2025) and the community survey (arxiv 2606.04924, june 2026) show detection and quality checks are comprehensively evaded by agents and degrade under paraphrase/out-of-domain, with mixed human-ai authorship hardest. the recent evidence favors westwood, but the sources genuinely conflict on whether content-side detection retains any value as a secondary signal.\n- nature of disagreement: the aggregation tradition (dawid-skene, mace, majority vote) models disagreement as noise around one latent truth to be recovered; the perspectivist tradition (aroyo & welty, dices, lewidi-2025, diverging preferences) treats much of it as legitimate signal to be preserved. neither side claims universality, but they prescribe opposite data models (adjudicated gold vs. per-rater distributions) - the design must pick per-axis, and no source offers a validated automatic classifier for which regime an axis falls into.\n- agreement statistics under skew: krippendorff defends alpha as the general-purpose reliability standard; gwet and successors show alpha/kappa collapse under extreme prevalence and push ac1/ac2; elangovan et al. (iclr 2025) argue all human-reliability coefficients are misapplied to machine-vs-human comparison in the first place. there is no consensus single statistic - only convergence that one-number reporting is wrong.\n- mitigation of crowdworker llm use: veselovsky et al. measured that request+hurdle interventions halve usage (a meaningful effect worth deploying); westwood concludes no instrument-level mitigation withstands agent-based completion ('no magical fix'). these are consistent about direction (partial reduction) but conflict sharply on whether prevention is worth its measured quality cost.\n\n## gaps\n- no published measurement of the false-agreement rate (judge and attempter agreeing while both wrong per expert panel) in an annotation-qa setting; the nearest proxy is northcutt's 3.3%+ gold-label error floor. this number would have to be produced in-house via an independent expert-audit channel.\n- no public 2025-2026 prevalence data for llm use specifically among paid annotators/reviewers on managed ai-training-data platforms (scale, surge, outlier, prolific-managed rlhf work); all published prevalence numbers come from mturk/prolific survey and summarization tasks, which may under- or over-state prevalence for critique/rating work.\n- item-level (as opposed to worker-level) detection of llm-generated annotations for short/low-dimensional outputs remains unsolved; the neurips 2025 peer-prediction mechanism scores workers, not items, and requires overlapping assignments across workers.\n- no annotation-science source found that gives sample-size arithmetic for certifying a judge against a gold set under low failure base rates (per-project vs pooled hierarchical validation); this appears only in general eval-statistics literature outside this domain's charter.\n- no longitudinal studies found measuring whether qa feedback changes annotator behavior over time (repeat-error rates per criterion); the feedback-efficacy question (taxonomy q7) has essentially no direct evidence in the annotation-quality literature.\n- could not open the full text of the diverging preferences taxonomy to extract exact percentage splits across its 10 disagreement categories (abstract confirms only that underspecification + style form the majority); exact fractions would sharpen the variance-attribution prior.\n\n## verifications\n- claim: roughly one-third of crowdworkers used llms on an llm-advantaged text-production task (33-35% on prolific, july 2023; ~34% prolific self-report in a 2025 follow-up by zhang et al.), and the best tested mitigations (direct request + copy-paste disable or image-only presentation) only cut usage roughly in half (27.6% to ~15.9%), never to zero, while also degrading response quality (direct requests reduced keyword retention 6.2%).\n verdict: partially_confirmed | core claim verified against the full pdf of arxiv 2310.15683 (veselovsky, horta ribeiro, cozzolino, gordon, rothschild, west): study 1, n=168 prolific workers, 3 july 2023, medical-abstract summarization; prevalence 33.3% [25.9,40.1] (classify-and-count), 35.2% [29.8,40.6] (probabilistic), 35.4% [27.8,43.0] (corrected) - matches \"33.3-35.4%, cis ~[26%,43%]\". study 2, n=720, 23 july 2023, 3x3 factorial (request: none/indirect/direct x hurdle: none/image/no-ctrl-c+v); classifier estimate dropped 27.6% -> 15.9% (direct+image) and 15.8% (direct+ctrl c+v) - matches \"roughly halved, not eliminated\" (paper: \"reduced llm use by nearly 50%, it could not fully prevent it\"). direct request decreased keyword retention by 6.2% (p=0.009) - exact match. classifier was finetuned e5-base-v2, calibrated (card & smith), with self-report and high-precision heuristics (completion time + paste artifacts) - matches. the june 2026 community survey arxiv 2606.04924 (velutharambath et al., 155 researchers, survey run aug 2025-mar 2026) does cite \"veselovsky et al. estimate 30-40%\", cites \"zhang et al. (2025): 34% of prolific participants self-report using llms for open-ended questions\", and states mitigations \"reduce rather than eliminate llm-assisted responses\" - so the follow-up attribution to zhang et al. 2025 is consistent with that survey's citations (note: i verified the citation exists in the survey, not the underlying zhang et al. paper itself). issues: (1) date \"2025-02-18\" is wrong for the arxiv link given - arxiv has only v1, dated 24 oct 2023; the published version is communications of the acm 68(3):42-47, march 2025 issue (doi 10.1145/3685527). (2) \"preregistered\" is not supported by the paper text - no preregistration is mentioned in the arxiv pdf. (3) minor nuance: \"never to zero\" - by the classifier measure true, but the high-precision heuristics table actually shows 0.0% in the direct+ctrl c+v cell; the paper's own framing (\"cannot fully prevent\") still supports the claim's spirit. (4) 44% of surveyed researchers in the 2026 survey observed llm use in their data - newer corroborating, not superseding, evidence; no later work found that overturns the prevalence estimates. sources: https://arxiv.org/abs/2310.15683, https://arxiv.org/abs/2606.04924, https://dl.acm.org/doi/10.1145/3685527.\n corrected: roughly one-third of prolific crowdworkers used llms on an llm-advantaged text-production task (33.3-35.4% across three estimators, july 2023, n=168; veselovsky et al., arxiv 2310.15683 v1 posted 24 oct 2023, published in communications of the acm 68(3), march 2025). in a second study (n=720, 3x3 factorial), the best mitigations (direct request + image presentation or + copy-paste disable) roughly halved classifier-estimated usage from 27.6% to 15.9%/15.8% without eliminating it, and directly requesting non-use reduced keyword retention by 6.2% (p=0.009). the studies were not described as preregistered. a june 2026 community survey of 155 researchers (arxiv 2606.04924, velutharambath et al.) corroborates, citing veselovsky et al.'s 30-40% estimate and zhang et al. (2025)'s 34% prolific self-report, and finding 44% of researchers observed llm use in their crowdsourced free-text data; it echoes that mitigations reduce rather than eliminate llm-assisted responses.\n- claim: an autonomous llm agent passed 99.8% of standard attention/quality checks (6,000 trials, 20 check types), evaded behavioral detectors (simulated reading times, mouse movement, typo-laden keystrokes), refused ai-revealing 'reverse shibboleth' tasks 97.7% of the time, and cost ~$0.05 per completed survey - versus $1.50 paid - making content-based spam detection economically and technically beaten as of late 2025.\n verdict: confirmed | every quantitative element of the claim verifies against the primary source (westwood, \"the potential existential threat of large language models to online survey research,\" pnas 122(47), published nov 20, 2025; pubmed 41264250, cited 81x by jul 2026; won a 2025 cozzarelli prize). extracted paper text confirms verbatim: (1) 99.8% pass rate across 6,000 trials (20 acq types x 300 repetitions), only 10 errors, perfect on 18/20 check types; (2) execution layer \"simulates realistic reading times calibrated to the persona's education level, generates human-like mouse movements, and types open-ended responses keystroke-by-keystroke, complete with plausible typos and corrections\"; (3) strategically declines 97.7% of \"reverse shibboleth\" tasks; (4) ~$0.05 per survey with commercial models vs $1.50 payment (>96.8% margin), marginal cost near zero with local open-weight models; (5) 43,800 total evaluations, 139 questions, 6,700 trials, with a key subset validated across nine llms (deepseek r1, mistral large, claude 3.7 sonnet, grok 3, gemini 2.5, qwen plus, llama 4 maverick, gpt-4.1, o4-mini); (6) 0% error rate on impossible-event/rare-event probes. the paper's abstract and press framing support the interpretive conclusion that coherence-/content-based detection is no longer tenable and defenses must be provenance/economic. supersession check: no 2026 work refutes it - follow-ups (westwood & frederick 2026 prolific audit; a 2026 sage ampps review of ai-mediated contamination) build on it and treat detection as an unsolved provenance problem. two minor caveats, both in the evidence summary rather than the claim: (a) arxiv 2606.04924 (\"can crowdsourcing survive the llm era?\", jun 2026) is a community survey of 155 researchers' practices/experiences, not a technical detector benchmark; its abstract does not state the paraphrase/out-of-domain/mixed-authorship degradation findings attributed to it (those likely come from a different detection-benchmark paper). (b) on recaptcha the paper says the system \"is designed to accommodate tools for bypassing\" recaptcha - an architectural capability, slightly weaker than \"recaptcha... fail[s]\" as an empirically demonstrated result. neither caveat affects the claim text itself, which is fully accurate.\n- claim: correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target.\n verdict: partially_confirmed | substance confirmed on every checked point against the arxiv record (https://arxiv.org/abs/2410.03775): (1) abstract states machine labels \"may superficially appear to have similar or better correlation with the human majority\" when human uncertainty is high, and machine-human correlation falls \"well below\" human-human correlation as human label consistency increases; (2) it flags krippendorff's alpha / randolph's kappa as designed for human-human reliability with assumptions inapplicable to machine labels; (3) proposed fixes match exactly - stratification by human label uncertainty, binned jensen-shannon divergence for perception-type tasks, and 'perception charts'; (4) venue is iclr 2025 per the arxiv comments field; (5) code is open-sourced at github.com/amazon-science/beyondcorrelation. the only inaccuracy is the source date: the claim says 2025-05-01, but the latest arxiv version is v3 dated 2025-01-27 (original submission 2024-10-03) - no may 2025 version exists on the record. supersession check: a follow-up search surfaced related later work - \"validating llm-as-a-judge systems under rating indeterminacy\" (arxiv 2503.05965), \"hidden measurement error in llm pipelines distorts annotation, evaluation, and benchmarking\" (arxiv 2604.11581), and \"care: confounder-aware aggregation for reliable llm evaluation\" (arxiv 2603.00039, feb 2026). these extend and reinforce the human-uncertainty/measurement-error critique rather than refute it; nothing found supersedes or contradicts the core finding. verdict is partially_confirmed solely because of the incorrect source date; the claim itself stands.\n corrected: correlation-based meta-evaluation of automatic judges against human labels is systematically distorted by human label uncertainty: on high-disagreement items a machine judge can superficially match or beat human-human correlation, while on high-agreement strata machine-human correlation drops well below the human-human baseline - so a single aggregate 'agreement with humans' number is invalid as a validation target. source: elangovan et al., \"beyond correlation...\" (amazon science), accepted at iclr 2025, arxiv 2410.03775 (v1 2024-10-03, latest v3 2025-01-27), code at github.com/amazon-science/beyondcorrelation.", "8": "domain: critique-models\n\n## human+critic-model teams occupy a strictly better operating point than either alone: in openai's criticgpt study, human+criticgpt teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (fsbs length penalty), not a fixed property. [load-bearing]\nevidence: criticgpt critiques were preferred over human contractor critiques in 63% of cases on naturally occurring llm errors (>80% elo win rate on inserted bugs); both chatgpt and criticgpt caught substantially more inserted bugs than paid contractors (median ~50 min/review, ~5 yrs python experience). humans alone had far fewer hallucinated bugs/nitpicks; human+criticgpt teams landed between, 'moving beyond the model-only pareto frontier.' force sampling beam search scores candidates by rm_score + length_modifier x num_highlights, tracing a precision/comprehensiveness pareto curve selectable at deployment without retraining. verified directly from the arxiv html full text.\nsource: llm critics help catch llm bugs (criticgpt, openai) | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary\nimplication: validates the hybrid one-touch foundation: ai critic drafts findings, human filters hallucinated nitpicks. make the precision/comprehensiveness operating point an explicit per-project config knob (q2/q4/q7 false-accusation slo), since the same trained judge can be run strict or comprehensive per project.\n\n## critic-assisted review of items already rated 'flawless' by a first human found real, rating-changing problems in 24% of flagged cases, versus 6% found by an unassisted second human review of the same data - a 4x lift on the exact 'confident pass' slice.\nevidence: criticgpt (rl-only) was run over chatgpt training data rated flawless by a first annotator; where the critique flagged a problem, reviewers agreed it substantially lowered the rating in 24% of cases, vs 6% in a no-critique second-rater replication - and this held out-of-distribution (mostly non-code tasks). prioritizing by critique reward-model score further raised catch rates and cut hallucinations. verified from arxiv html full text.\nsource: llm critics help catch llm bugs (criticgpt, openai) - 'flawless' data experiment | https://arxiv.org/abs/2407.00215 | 2024-06-28 | primary\nimplication: direct evidence for q1's false-agreement problem and q6's routing: unassisted human re-review of passed items misses ~3/4 of what critic-assisted review catches, so the standing expert-audit channel over confident passes should itself be critic-assisted, and critic-confidence scores can prioritize the audit queue.\n\n## in deepmind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (ai rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% ai-alone and 75.1% human-alone - and showing humans the ai's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the ai was right without hurting when it was wrong. [load-bearing]\nevidence: 1,918 expert-labeled fact-verification items; tool-using ai rater with majority-vote-over-50-samples confidence (well calibrated). ai 87.7% vs individual humans 75.1% (majority vote 80.6%). hybridization at threshold 0.62: 89.3%; on the routed low-confidence slice humans beat the ai 71.3% vs 60.5% (real complementarity region). evidence-only assistance: 79.3% when ai right (vs 71.3% baseline) with no significant harm when ai wrong; conditions showing judgments+reasoning/confidence produced significant over-reliance decrements (e.g. beta = -0.768, p < .001); ai-debate condition was numerically worst. verified from arxiv html full text.\nsource: human-ai complementarity: a goal for amplified oversight (google deepmind) | https://arxiv.org/abs/2510.26518 | 2025-10-30 | primary\nimplication: answers q6 concretely: spend the one human touch on ai-low-confidence items, and give that human the judge's collected evidence spans without the judge's verdict or rationale (blind-ish adjudication over curated evidence). requires the judge to emit calibrated confidence and quoted evidence as first-class outputs (q3), and argues against 'human verifies ai verdict' as the default interaction.\n\n## a nature human behaviour meta-analysis of 100+ experiments (300+ effect sizes) found human-ai combinations on average performed significantly worse than the best of human or ai alone on decision/judgment tasks, with gains only where humans outperformed the ai solo - naive human-checks-ai designs destroy value.\nevidence: vaccaro, almaatouq & malone systematic review/meta-analysis: average human-ai combo underperformed the stronger party alone; losses concentrated in decision-making tasks, gains in content-creation tasks; combos helped when humans beat ai alone. deepmind's complementarity paper explicitly frames its confidence-routing + evidence-only design as the answer to this result. verified from arxiv abstract.\nsource: when combinations of humans and ai are useful: a systematic review and meta-analysis | https://arxiv.org/abs/2405.06087 | 2024-10-28 | contrarian\nimplication: the q6 fork ('is one shallow touch better than zero touch plus deep audits?') is empirically live: an automation-biased shallow verify step is the modal failure in the literature. the touch only pays where the human plausibly beats the judge (routed low-confidence/contested items), which mandates closed-loop routing on observed overturn rates rather than fixed per-item review.\n\n## in the largest deployed rct of ai critiquing human evaluative work (iclr 2025, feedback on >20,000 peer reviews), 27% of reviewers who received llm feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated llm reliability tests. [load-bearing]\nevidence: randomized controlled study at iclr 2025 (~22.5k feedback-arm vs ~22.4k control reviews): review feedback agent flagged vague comments, content misunderstandings, unprofessional remarks; guardrail llm tests gated delivery; 26.6-27% update rate, avg +80 words among updaters, blinded informativeness gains, longer author-reviewer discussions. verified from arxiv abstract page; corroborated by the official iclr blog.\nsource: can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025 | https://arxiv.org/abs/2504.09737 | 2025-04-13 | academic\nimplication: q7's central bet is supported at deployment scale: constructive, item-specific ai feedback measurably changes the behavior of human evaluators, and the shipped pattern held feedback to its own automated verification gate before delivery (fail-closed feedback). also gives a realistic effect-size prior: ~1 in 4 recipients act on feedback - plan longitudinal metrics around that base rate.\n\n## llms are structurally weak at detecting omissions: on absencebench, average f1 drops 56.9 points versus detecting the same content as insertions (best model 71.2% overall, 40.0% on code diffs, vs ~99.5% needle-in-haystack), and inserting explicit placeholders at gap sites recovers ~35.7 points - absence has no attention key to attend to. [load-bearing]\nevidence: 4,302 instances across poetry, numerical sequences, github prs (~5k token contexts): models must list deliberately deleted elements given original+modified documents. gemini-2.5-flash (thinking) best at 71.2% avg f1; mixtral-8x7b scores 99%+ on niah but 14.7% here; thinking tokens buy only ~7.9%; '<missing line>' placeholders add up to +81.8%. neurips 2025 datasets & benchmarks. verified from arxiv html full text.\nsource: absencebench: language models can't tell what's missing | https://arxiv.org/abs/2506.11440 | 2025-06-13 | academic\nimplication: caps what the judge can verify autonomously (q3/q5): 'comprehensiveness'/'complete' praise claims and omission-type attempter failures are the judge's structurally weakest lane. the rubric compiler should convert completeness criteria into explicit enumerable checklists (placeholder effect = give absence a token), and free-form omission judgment belongs in the human or human-assisted lane, not judge-autonomous.\n\n## critique quality is itself quantifiable at usable reliability: metacritique decomposes critiques into atomic information units and scores precision/recall against references, with gpt-4 aiu-level judgments at 85-89% accuracy and meta-f1 correlating with human gold at pearson 0.84-0.89 - and measured baselines show human critiques are high-precision/low-recall (87.6%/48.7%) while llm critiques are the inverse-ish (71.9%/53.3%).\nevidence: acl findings 2024 framework: precision = fraction of critique aius judged factual; recall = coverage of reference-critique aius; each judgment carries a natural-language rationale. choosing critiques by meta-f1 yields refinements that win 51%/lose 27% by human eval, vs gpt-4 pairwise picks losing more than winning. ~28% of llm-critique aius were non-factual vs ~12% for humans; llm critiques carried 2.4x the information volume (8.10 vs 3.31 aius). verified from arxiv html full text.\nsource: the critique of critique (metacritique) | https://arxiv.org/abs/2401.04518 | 2024-01-09 | academic\nimplication: q7's reflexive-grounding requirement is buildable today: run autoqa feedback through aiu-level precision scoring as its own admission gate, and report critique-precision/recall as first-class system metrics. also sets expectations: the ai side of the pipeline supplies coverage, the human side supplies precision - mirroring criticgpt.\n\n## verdict accuracy and critique validity dissociate badly: in criticbench-thu data 24.8% of items got the correct verdict with a low-quality critique, and open-loop verdict-agreement metrics compress a 27.1-point real error-identification gap (processbench: o1-mini 88.9 vs qwen2.5-72b 61.8) into a 1.3-point verdict-f1 gap - so critique quality must be evaluated closed-loop by whether the critique drives a successful correction.\nevidence: realcritic (jan 2025, arxiv 2501.14492): closed-loop critique-then-correct design over 8 reasoning benchmarks; classical llms (incl. gpt-4) lose accuracy under self-critique (-1.8 to -5.1 avg; up to -35.6 domain drops) while o1-mini is the only model with positive self-critique delta; failure modes 'superficial success' (right verdict, wrong analysis) and 'contradictory output' documented. independently confirmed at scale in 2026: an icml 2026 paper (rm-nlhf, arxiv 2601.07349) finds outcome-rewarded generative reward models 'guess correct outcomes without sound critiques' and uses similarity-to-human-critique as a process reward to fix it. realcritic verified via arxiv abstract + html (html render partially draft-quality; abstract and appendix numbers consistent).\nsource: realcritic: towards effectiveness-driven evaluation of language model critiques | https://arxiv.org/abs/2501.14492 | 2025-01-24 | academic\nimplication: for q1 meta-evaluation: never certify the autoqa on verdict agreement alone - a judge can match pass/fail labels while its rationales are wrong, which poisons the feedback channel and attempter trust. validation must score the rationale (does the cited evidence entail the finding; does acting on the feedback fix the item), i.e., q7's explainability contract is also the correct validation instrument.\n\n## a june 2026 study of 21 llm judges (~541k judgments) shows raw percent-agreement overstates chance-corrected agreement by 33.8-41.3 points (85% agreement = kappa ~0.48), judge rankings flip by up to 15 positions across benchmarks, and the most reproducible judges are among the least valid (test-retest 0.99 with position bias 0.19) - leading the authors to prescribe a pre-deployment minimum viable validation protocol.\nevidence: 'reliability without validity' (uc berkeley, arxiv 2606.19544, evals run march-april 2026): mt-bench/judgebench/rewardbench, 118 runs at temperature 0; kappa deflation universal across all 21 judges; consistency-bias paradox instantiated by qwen3-8b and gemini 2.5 flash; mvvp = report kappa/krippendorff alpha as headline, ab+ba position swaps, >=3 replicates, >=2 benchmarks spanning preference- and correctness-style labels, and audit that high stability is not just high bias. verified from arxiv html full text.\nsource: reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic\nimplication: directly answers q1's statistics question (ban raw percent-agreement; standardize on chance-corrected stats) and q5's ship-gate question (perturbation/position-swap + replicate harness is now published prescriptive practice). run-to-run consistency must never be reported as evidence of validity - a perfectly consistent judge can be laundering a bias.\n\n## the closest production analog to the proposed autoqa already exists: toloka's deployed 'llm qa' runs a tool-using agent per quality-metric on every human annotation submission, emits a strict three-way pass / fail / unable-to-verify verdict (the abstain class exists specifically to prevent hallucinated verdicts), coaches annotators with socratic feedback, and explicitly accepts lower precision on fail because a false pass costs more than escalating a genuine pass.\nevidence: toloka engineering blog (bylined 2026-03-30): agentic autocheck with web/image/audio/python/bash tools; 'one metric, one entity, one verdict' scoping (separate agent instance per criterion); internal benchmark of 300+ real submissions from 20+ live projects with senior-qa ground truth (numbers in an image, not extractable); reported constraint: performance 'entirely bounded by task design' - ambiguous guidelines produce walls of unable-to-verify. verified by opening the post.\nsource: llm qa: scaling data quality assurance technologically (toloka) | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | 2026-03-30 | practitioner\nimplication: validates several foundational choices from production: criterion-scoped judge instances (q3 decomposition at the per-axis level), a first-class abstain/escalate verdict distinct from pass/fail (q2 verdict ontology), asymmetric error costs favoring false-flags over false-passes, and rubric quality as the binding constraint (q4: tighten criteria before writing guidelines).\n\n## the foundational 2022 result behind the whole lineage: model-written critiques helped human evaluators find flaws in summaries they would otherwise have missed (including planted flaws in deliberately misleading human-written summaries), and models exhibit a discriminator-critique gap - they can often detect that something is wrong better than they can articulate why.\nevidence: saunders et al. (openai, arxiv 2206.05802): topic-based summarization assistance experiments; critiques helped on both model- and human-written summaries; framework comparing generation, discrimination, and critique ability found 'even large models may still have relevant knowledge they cannot or do not articulate as critiques'; larger models critique and self-refine better. verified from arxiv abstract.\nsource: self-critiquing models for assisting human evaluators (openai) | https://arxiv.org/abs/2206.05802 | 2022-06-12 | primary\nimplication: the discriminator-critique gap argues for a two-stage judge: use cheap discrimination (scores/uncertainty) for routing and prioritization even where articulated critique is unreliable, and hold only the articulated-critique layer to the evidence-quoting bar (q3/q7). judge-assisted human review of human-written work is validated at the root of this lineage, not just ai-output critique.\n\n## a deployed llm compliance-checker was gamed by its subjects in a single shot: the neurips 2024 author checklist assistant (234 papers) was rated useful by >70% of authors, but the organizers found the system 'not robust to gaming' by authors and concluded it is a poor substitute for human review.\nevidence: goldberg, ullah, guyon, shah et al. (arxiv 2411.03417; neurips blog dec 2024): gpt-4-turbo checklist verification offered pre-submission; surveys plus qualitative analysis showed genuine improvements in some submissions alongside demonstrated manipulation of the assistant (authors could satisfy the checker without satisfying the standard); access was restricted to authors partly to avoid biasing review. verified via arxiv abstract and organizer blog summary.\nsource: usefulness of llms as an author checklist assistant for scientific papers: neurips'24 experiment | https://arxiv.org/abs/2411.03417 | 2024-11-05 | academic\nimplication: q8's decision-boundary-leak concern has deployment evidence even without repeated feedback cycles: pay/acceptance-motivated subjects will optimize against a visible checker. defenses (rotating seeded probes, held-out human-graded golden items to detect autoqa-vs-gold divergence, abstracted rather than checker-revealing feedback) must be designed in from day one, not retrofitted.\n\n## tools & artifacts\n- review feedback agent: open-source multi-llm pipeline (zou-group/review_feedback_agent, cc by 4.0 paper) that critiques human peer reviews for vagueness, content misunderstanding, and unprofessionalism, with automated llm reliability tests gating feedback delivery. | https://github.com/zou-group/review_feedback_agent | directly reusable architecture for the constructive-feedback channel: feedback taxonomy + fail-closed guardrail tests that feedback must pass before delivery (q7).\n- metacritique: framework + dataset (gair-nlp/metacritique) scoring critique precision/recall/f1 over atomic information units with per-unit rationales. | https://github.com/gair-nlp/metacritique | off-the-shelf machinery for holding autoqa feedback to its own grounding standard and for meta-evaluating critique quality during validation (q1, q7).\n- absencebench: neurips 2025 d&b benchmark (harvey-fin/absence-bench) measuring whether models can identify deliberately omitted content across poetry, sequences, and code diffs. | https://github.com/harvey-fin/absence-bench | ready-made competence-limit test for the omission/comprehensiveness lane of any candidate judge model before assigning it completeness-verification authority (q5 lane assignment).\n- realcritic: closed-loop critique-evaluation benchmark (tangzhy/realcritic): critique quality scored by correction success, with self-/cross-/iterative-critique modes. | https://github.com/tangzhy/realcritic | template for validating the autoqa judge on rationale effectiveness rather than verdict agreement (q1 meta-evaluation design).\n- ctrl / critique-rl / deepcritic (2025 critic-training lineage): rl-trained critic models: ctrl (arxiv 2502.03492, critique via rl without human critique labels, up to 106% relative code-gen gains), critique-rl (arxiv 2510.24320, two-stage rl: discriminability then helpfulness, no stronger supervisor), deepcritic (arxiv 2505.00662, deliberate step-level math critique). | https://arxiv.org/abs/2502.03492 | state of the art for training a dedicated critic without the expensive expert-critique annotation dependency (q4's aligned-expert concern); critique-rl's discriminate-first-then-explain staging mirrors the discriminator-critique gap design.\n- on scalable oversight with weak llms judging strong llms (kenton et al., deepmind, neurips 2024): protocol comparison (debate vs consultancy vs direct qa) with weaker judges overseeing stronger agents across qa/math/code/logic asymmetries. | https://arxiv.org/abs/2407.04622 | evidence base for arbitration design (q6): debate beats consultancy everywhere tested, and open debate makes a weak judge much harder to convince of a wrong answer - a candidate protocol for contested-item escalation.\n- minimum viable validation protocol (mvvp): five-step pre-deployment judge validation checklist from 'reliability without validity' (2026): chance-corrected headline stats, ab+ba position swaps, >=3 replicates, >=2 benchmark styles, consistency-bias paradox audit. | https://arxiv.org/abs/2606.19544 | directly adoptable as the per-project judge-config ship gate the design taxonomy asks for (q5).\n- toloka llm qa: production agentic qa system for human annotation submissions: per-metric tool-using agent instances, pass/fail/unable-to-verify verdicts, socratic annotator feedback. | https://toloka.ai/blog/llm-qa-scaling-data-quality-assurance-technologically/ | the closest commercial deployment to the proposed autoqa; its verdict ontology, criterion scoping, and guideline-quality constraint are field-tested reference points (q2, q4, q6).\n\n## disagreements\n- does adding a human to an ai judge help at all? vaccaro et al. (nature human behaviour 2024) find human-ai combos on average significantly worse than the best solo party on decision tasks, while criticgpt (2024) finds human+ai teams beyond the model-only pareto frontier and deepmind (2025) gets 91.3% hybrid vs 87.7% ai-alone. reconciliation offered by deepmind: complementarity only appears with confidence-based routing to a slice where humans genuinely beat the ai and with non-leading assistance formats - but the deepmind-adjacent follow-up ('toward human-ai complementarity across diverse tasks') found baseline hybridization added only +0.4pp when model confidence failed to identify the complementarity region. the one-touch design's value is conditional, not guaranteed.\n- what should the human be shown? deepmind (2025) finds showing the ai's judgment/reasoning/confidence causes over-reliance and that evidence-only assistance is the only safe format, whereas criticgpt's protocol and the iclr 2025 feedback agent delivered full natural-language critiques and reported net gains. task framing differs (assisting a second-party judge vs coaching the first-party author), which suggests the autoqa should show full critiques to attempters but evidence-only to human adjudicators - no single source tests both arms.\n- are llm critics better or worse than humans? criticgpt: model critiques preferred 63% and catch more bugs than paid contractors. metacritique: human critiques have much higher factual precision (87.6% vs 71.9% of atomic units factual) while llm critiques have higher coverage. both true simultaneously - the disagreement is about which axis (precision vs comprehensiveness) a qa system should privilege, and sources implicitly pick opposite defaults.\n- can critics increase comprehensiveness of review? criticgpt/saunders show critics make human review more comprehensive (catching missed flaws), but absencebench (2025) shows llms are structurally near-blind to omissions (56.9 f1 drop). reconciliation: the critic gains are demonstrated on commission-type errors (inserted/tampered bugs, incorrect statements); no source demonstrates critic gains on omission-type attempter failures, and absencebench predicts they won't appear without checklist/placeholder scaffolding.\n- what metric certifies a judge? open-loop verdict-agreement benchmarks (criticbench 2024, criticeval 2024, mt-bench-style agreement) vs realcritic's closed-loop position that verdict accuracy without correction-effectiveness is 'superficial success' (24.8% right-verdict-wrong-critique), reinforced by rm-nlhf (icml 2026) and by 'reliability without validity' (2026) showing judge rankings flip up to 15 places depending on which benchmark style you trust.\n\n## gaps\n- no public post-deployment report from openai on criticgpt's integration into the production rlhf labeling pipeline (announced intent, june 2024) - no follow-up metrics, and no officially branded successor found through july 2026; the lineage continued in academic work (ctrl, deepcritic, critique-rl, multicritique) rather than vendor deployment reports.\n- no published rct of ai critique/feedback specifically on human data-annotation qa work. closest analogs are peer review (iclr 2025) and fact-verification rater assistance (deepmind 2025). no longitudinal data anywhere on repeat-error-rate reduction per annotator under sustained ai feedback - q7's efficacy metric has no published precedent to calibrate against.\n- no measured multi-cycle adversarial study of annotators gaming a deployed critic (q8's decision-boundary leakage): the neurips'24 checklist gaming is single-shot and author-side; citation-theater/evidence-swap perturbation tests on critics were not found.\n- toloka's deployment report - the single closest production analog - publishes its benchmark design (300+ submissions, 20+ projects) but its quantitative precision/recall table is embedded in an image and was not extractable; no independent audit of any commercial annotation-qa llm system was found.\n- false-agreement rate (judge and attempter both wrong vs expert panel) is unmeasured in the literature; the criticgpt 24%-vs-6% flawless-data result is the nearest proxy and is one study, one domain (mostly), one vendor.\n- positive-claim/praise verification (is 'accurate/complete/clear' praise warranted?) has no dedicated benchmark or precision/recall measurements - metacritique-style aiu machinery has only been applied to flaw-finding critiques. this is a genuine hole given the mission's requirement to evaluate positive statements.\n- semantic scholar returned http 429 during literature-graph search (openalex/crossref coverage only for that call); one websearch call was flagged by a per-call safeguard and was successfully re-run via exa - no coverage lost, noted for provenance.\n\n## verifications\n- claim: human+critic-model teams occupy a strictly better operating point than either alone: in openai's criticgpt study, human+criticgpt teams wrote more comprehensive critiques than unassisted humans while hallucinating and nitpicking less than the model alone, and the comprehensiveness-vs-spurious-claims tradeoff is a tunable inference-time dial (fsbs length penalty), not a fixed property.\n verdict: confirmed | source verified: arxiv:2407.00215 \"llm critics help catch llm bugs\" (mcaleese et al., openai), submitted 2024-06-28 - date matches. abstract directly confirms: 63% preference for model critiques on naturally occurring llm errors; models catch more bugs than paid human reviewers; human-machine teams catch similar bug counts to llm critics while hallucinating less than llms alone. full-text details corroborated via independent secondary coverage: critic-assisted contractors wrote more comprehensive critiques than unassisted contractors while reducing hallucination/nitpick rate relative to the model, described as moving beyond the model-only pareto frontier; fsbs scores candidates by rm_score + length_modifier x num_highlights, giving a deployment-time precision/comprehensiveness dial without retraining. minor reading note: \"strictly better than either alone\" holds in the paper's pareto sense - teams matched (did not exceed) the model's bug-catch/comprehensiveness while beating its hallucination rate, and exceeded human comprehensiveness; the claim's own wording states exactly this, so no correction needed. supersession check: no official openai follow-up found; later work (codecriticbench 2025; 2026 papers on llm reviewer overcorrection) extends but does not contradict these findings.\n- claim: in deepmind's amplified-oversight experiments on fact-verification rating, confidence-based hybridization (ai rates when confident, humans rate the low-confidence slice with evidence-only assistance) reached 91.3% accuracy vs 87.7% ai-alone and 75.1% human-alone - and showing humans the ai's verdict/reasoning/confidence caused measurable over-reliance, while showing only retrieved evidence was the sole format that helped when the ai was right without hurting when it was wrong.\n verdict: confirmed | every quantitative element checks out against the arxiv html full text (2510.26518). dataset: 1,918 expert-labeled tuples - confirmed. ai rater 87.7%, individual humans 75.1%, human majority vote 80.6% - confirmed. confidence via 50 samples/sentence (avg 33.25 passing format check), described as fairly well calibrated - confirmed. unassisted hybridization at threshold t=0.62: 89.3% (beta=0.413, p=.012) - confirmed. low-confidence routed slice (280 items): humans 71.3% vs ai 60.5% (p=.006) - confirmed. the headline 91.3% is specifically the assisted hybrid (ai when confident + evidence-assisted humans on the low-confidence slice), vs 89.3% unassisted hybrid - exactly as the claim words it, so 91.3% is correctly attributed. evidence-only assistance: 79.3% vs 71.3% baseline when ai correct (beta=0.446, p=.009), no significant harm when ai wrong (64.0% vs 61.5%); paper explicitly calls it \"the only form of assistance that achieves the ideal of helping when correct and not hurting when wrong\" - confirmed. over-reliance when ai wrong: evidence&reasoning&judgments beta=-0.768 (se=0.174, z=-4.413, p<.001), er&j&confidence beta=-0.636 p<.001, judgments&confidence beta=-0.356 p=.042 - the quoted beta=-0.768 matches the evidence&reasoning&judgments condition exactly. debate was numerically worst (64.9%, only format below baseline) - confirmed. date: v1 submitted 2025-10-30 as claimed. currency: a v2 was posted 2026-06-25 and the paper was published at acm facct '26 (doi 10.1145/3805689.3812308); this is publication/revision, not supersession - no later work contradicting the findings found. deepmind blog notes ongoing follow-up combining hybridization+assistance on an internal rating task, unpublished as of 2026-07-14. minor caveat only: whoever cites this should prefer the v2/facct version in case numbers shifted slightly in revision (v1 numbers verified here).\n- claim: in the largest deployed rct of ai critiquing human evaluative work (iclr 2025, feedback on >20,000 peer reviews), 27% of reviewers who received llm feedback revised their reviews, incorporating >12,000 suggestions, producing reviews +80 words that blinded raters judged more informative, plus higher author-rebuttal engagement - and feedback was only delivered if it passed a suite of automated llm reliability tests.\n verdict: confirmed | source verified directly (arxiv:2504.09737, v1 submitted 2025-04-13 - date matches). abstract states: review feedback agent deployed at iclr 2025 as a large randomized controlled study on >20,000 randomly selected reviews; feedback targeted vague comments, content misunderstandings, unprofessional remarks; 27% of reviewers who received feedback updated their reviews; >12,000 suggestions incorporated (12,222 in the paper); +80 words average among updaters; more informative per blinded evaluators; longer author-reviewer discussions; feedback delivered only if it passed a suite of automated llm reliability tests. arm sizes corroborated from paper figures: 22,467 selected for feedback vs 22,364 control; 18,946 successfully received feedback, of whom 26.6% updated (the abstract rounds to 27%). two nuances, neither contradicting the claim: (1) the superlative \"largest deployed rct of ai critiquing human evaluative work\" is the researcher's framing, not verbatim in the abstract - no larger counterexample found, and it is consistent with the study's scale; (2) not superseded but now formally published: nature machine intelligence, feb 23, 2026, as \"a large-scale randomized study of large language model feedback in peer review\" (https://www.nature.com/articles/s42256-026-01188-x) - the nmi version is the preferred citation going forward. a related follow-up mixed-methods study on reviewer perceptions exists (arxiv:2602.13817) but does not alter these results.", "9": "domain: grounding-faithfulness\n\n## grounded claim-verification is commoditized but ceilinged: the best model on the llm-aggrefact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7b model, bespoke-minicheck-7b, at 77.4 average balanced accuracy, with 0.4-0.8b specialized checkers (factcg, minicheck-flan-t5-l) within ~2.5 points of frontier llms like claude-3.5-sonnet (77.2) and gpt-4o (75.9). [load-bearing]\nevidence: fetched the live llm-aggrefact leaderboard: top-10 shows bespoke-minicheck-7b 77.4, claude-3.5-sonnet 77.2, granite guardian 3.3 8b 76.5, factcg-deberta-l (0.4b) 75.6, minicheck-flan-t5-l (0.8b) 75.0, llama-3.1-405b 74.4. benchmark measures binary supported/unsupported vs grounding docs. confirmed independently by paladin-mini (june 2025 arxiv) citing bespoke-minicheck as leaderboard sota.\nsource: llm-aggrefact leaderboard (minicheck project) | https://llm-aggrefact.github.io/ | 2025 (accessed 2026-07-14) | primary\nimplication: a cheap, deterministic, reproducible entailment-check stage (sub-1b to 7b checker) for 'is this attempter statement supported by the cited evidence' is off-the-shelf and costs ~1/100 of frontier calls - but ~22% claim-level error means item verdicts cannot be a naive and over claim checks; error-tolerant aggregation and confidence-routing to humans are structurally required.\n\n## on adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on faithbench, prior detectors hit ~50% accuracy (negligible), hhem-2.1-open 66.7% and bespoke-minicheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while faithjudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 f1. [load-bearing]\nevidence: fetched full text of arxiv:2505.04847v2 (vectara, emnlp 2025 industry track, v2 nov 2025). table 1: baseline detector numbers; table 2: faithjudge results; faithjudge beats the facts grounding judging prompt head-to-head on all four ragtruth/faithbench splits (e.g., faithbench 70.8 vs 54.3 f1). documented failure modes: judge underpredicts hallucinations for some generator families; 'benign'/'questionable' ternary labels classified unreliably so only binary is used; specificity drops as more in-context examples are added.\nsource: benchmarking llm faithfulness in rag with evolving leaderboards (faithjudge) | https://arxiv.org/abs/2505.04847 | 2025-11 | academic\nimplication: the strongest 2025-measured grounding-judge architecture is exactly the autoqa rubric-compilation contract: a per-project pool of human-adjudicated exemplar judgments (annotated spans + labels on comparable items) injected few-shot, not zero-shot judging and not a fixed fine-tuned checker. also: severity-graded verdicts (benign/questionable) are where both humans and judges lose reliability - keep the machine verdict binary and treat severity as a separate, human-anchored layer.\n\n## claim decomposition helps weak verifiers but actively degrades strong ones: with minicheck as verifier on wice, no-decomposition scores 80.01 balanced accuracy while factscore-style atomic decomposition drops it to 71.11; with the weaker alignscore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity. [load-bearing]\nevidence: fetched full text of 'decomposition dilemmas' (naacl 2025, arxiv:2411.02400). four-way decomposition-error taxonomy from manual inspection: (a) omission of context/logical relations, (b) ambiguity (unclear pronouns/references), (c) over-decomposition, (d) alteration of original meaning. factscore-style atomicity produces the most over-decomposition errors; veriscore-style tends to omit context. on felm, decomposition raised minicheck f1 48.1->~68 but dropped gpt-4o-mini f1 71.6->54.3.\nsource: decomposition dilemmas: does claim decomposition boost or burden fact-checking performance? | https://aclanthology.org/2025.naacl-long.320/ | 2025-05 | academic\nimplication: directly answers the granularity design question: there is no universally-best decomposition level. granularity must be tuned per verifier strength and per item complexity - atomic-claim pipelines with a strong judge are measurably worse than judging larger spans. the 'checklist myopia' risk is real and has a named mechanism (context omission + meaning alteration). budget a granularity-calibration step per project rather than fixing atomic decomposition in the foundation.\n\n## decomposition and decontextualization are in direct tension - isolating atomic facts strips the context needed to verify them, while adding context back creates multi-fact claims where the verifier may credit or penalize the wrong content; dndscore resolves this by verifying the original subclaim with the added information treated as context rather than as content to be verified.\nevidence: fetched acl anthology abstract of dndscore (wanner, van durme, dredze; emnlp 2025 main, pp. 23609-23626). the paper evaluates combinations of decomposition, decontextualization, and verification strategies and finds the strategy choice materially changes factuality scores.\nsource: dndscore: decontextualization and decomposition for factuality verification in long-form text generation | https://aclanthology.org/2025.emnlp-main.1205/ | 2025-11 | academic\nimplication: when the autoqa checks an attempter's sentence, the verification unit should be 'claim + explicit context annotations' with the verifier told which part is under test - not a free-floating atomic rewrite. this is a concrete spec for the claim-extraction layer's output schema.\n\n## verifiability triage before entailment is established machinery: veriscore extracts and scores only verifiable claims because factscore/safe 'assume that every claim is verifiable', which breaks on real long-form text containing opinions and unverifiable content; factbench's verify pipeline (acl 2025) further labels content units supported/unsupported/undecidable against retrieval, with 4,467 human-annotated units released for validation.\nevidence: fetched arxiv:2406.19276 abstract (veriscore, emnlp findings 2024): human evaluation found veriscore's extracted claims 'more sensible' than competitors across 8 long-form tasks. fetched launchnlp/factbench github (acl 2025 paper 2410.22257): three-way supported/unsupported/undecidable labeling according to retrieval results, benchmarked against factscore, safe, factcheck-gpt.\nsource: veriscore (emnlp findings 2024) + factbench/verify (acl 2025) | https://arxiv.org/abs/2406.19276 | 2024-06 (veriscore, foundational); 2025-07 (factbench acl 2025) | academic\nimplication: the claim ontology question has a field-standard answer: a front-stage classifier that routes each attempter statement into {verifiable claim, opinion/unverifiable, undecidable-given-evidence} before any entailment check. opinion-class statements get consistency-with-cited-evidence checks at most, never truth verdicts; 'undecidable' is a first-class output, not a forced pass/fail.\n\n## holistic llm judging of context-grounded outputs is far weaker than claim-level checking: on contextualjudgebench (2,000 pairs, 8 splits over rag/summarization with conditional criteria like 'faithfulness first, then completeness'), the best of 20 judge models tested (openai o1) barely reaches 55% consistent accuracy.\nevidence: fetched arxiv:2503.15620 (salesforce; acl 2025). 11 specialized judge models + 9 general-purpose models evaluated; the benchmark encodes criteria hierarchies and finds contextual assessment poses a significant challenge even to sota models. contrast: the same model class hits 77-84% on single-claim grounding tasks (llm-aggrefact, faithjudge).\nsource: does context matter? contextualjudgebench for evaluating llm-based judges in contextual settings | https://arxiv.org/abs/2503.15620 | 2025-03 | academic\nimplication: do not architect the autoqa as one holistic 'judge this writeup against the rubric' call - measured reliability roughly halves versus decomposed per-criterion, per-claim checks. criteria priority ordering (which humans dispute) must be compiled explicitly into the pipeline structure, because judges fail at applying conditional criteria internally.\n\n## grounding judges have a measured agreement-default asymmetry: in google's facts leaderboard (dec 2025), grounding judges score ~85 f1 on the positive (grounded) class but only ~46 f1 on the negative (ungrounded) class, and the grounding metric is gameable by vague, short responses that avoid unsupported claims - countered by a mandatory eligibility gate that scores non-responsive answers as inaccurate. [load-bearing]\nevidence: fetched full text of arxiv:2512.10791v1 (the facts leaderboard, google, dec 2025). best judge-prompt combos reach only ~65 macro-f1 (gemini-2.5-flash + v2 prompt: 65.33) against a 320-example human-adjudicated held-out set; positive-class ~85 vs negative-class ~46 f1; eligibility check explicitly added because 'grounding metrics can be hacked via vague, short responses'.\nsource: the facts leaderboard: a comprehensive benchmark for large language model factuality | https://arxiv.org/abs/2512.10791 | 2025-12 | academic\nimplication: two foundational mandates: (1) the sycophancy/agreement-default bias on positive claims is quantified and large - the autoqa must be evaluated on per-class recall (especially fail-class recall at fixed prevalence), never single-number accuracy; (2) an eligibility/responsiveness gate must precede grounding checks or attempters can pass by writing vacuous hedged annotations - the exact adversarial dynamic the design taxonomy flags.\n\n## cross-family judge ensembles are the standard mitigation for self-preference: facts grounding v1 (jan 2025) aggregated three frontier judges from different families (gemini 1.5 pro, gpt-4o, claude 3.5 sonnet) explicitly because 'models are biased towards favorably judging their own outputs', and grounding v2 (dec 2025) kept a two-family ensemble (gemini 2.5 flash + gpt-5).\nevidence: fetched arxiv:2512.10791v1 which states the v2 judge-ensemble rationale and composition; v1 design documented in arxiv:2501.03200 (jan 2025) and the deepmind blog (dec 17, 2024). facts parametric suite separately validated that a single-judge setup preserved rankings vs a mixed panel (gemini 2.5 pro, o3, grok 4) - i.e., ensemble necessity is task-dependent and testable.\nsource: facts grounding leaderboard (v1 paper + v2 in facts leaderboard) | https://arxiv.org/abs/2501.03200 | 2025-01 (v1); 2025-12 (v2) | academic\nimplication: when attempters critique outputs from a given model family, the autoqa judge should default to a different family or an ensemble spanning families; but ensemble overhead can be dropped where a single-judge-vs-panel ranking-preservation test passes - a cheap per-project validation, not a dogma.\n\n## span-restricted nli citation checking (the alce-style 'does the cited span entail the sentence' standard) is judged a suboptimal proxy by 2025 work: citeeval (acl 2025) argues citation quality must be evaluated against the full retrieval context, user query, and generated text - not cited sources alone - and its model-based citeeval-auto metrics correlate better with human judgments on the multi-domain citebench than nli-based metrics.\nevidence: fetched arxiv:2506.01829 (citeeval, acl 2025 main). explicitly frames binary/ternary nli vs cited sources as 'a suboptimal proxy for citation evaluation'. corroborated direction: citeguard (acl 2026, aclanthology.org/2026.acl-long.282) reframes citation evaluation as attribution alignment and notes 'the reliability of llm-as-a-judge alone is also in doubt' for citation judging.\nsource: citeeval: principle-driven citation evaluation for source attribution | https://arxiv.org/abs/2506.01829 | 2025-06 | academic\nimplication: evidence-closure design: checking attempter claims only against the spans they cite will miss the dominant human failure of citing the wrong/incomplete evidence when better evidence existed in scope. the autoqa needs two distinct checks - (a) does cited evidence entail the claim (span-closed), and (b) is the citation the right one given the full in-scope corpus (context-closed) - with (b) also catching true-but-wrongly-cited claims.\n\n## fact-verification llms are brittle to semantically-minor input perturbations: facteval (naacl 2025) tested 17 realistic word- and character-level perturbations plus 4 subpopulations on fever across zero-shot/few-shot/cot setups and found llms 'brittle to small input changes' with performance varying across subpopulations.\nevidence: fetched acl anthology page for 2025.naacl-long.534 (mamta & cocarascu). abstract is qualitative; per-perturbation numbers are in the full paper. consistent with faithjudge's finding that adding more in-context examples shifts sensitivity/specificity, and with facts' judge-prompt sensitivity (macro-f1 varies by prompt version).\nsource: facteval: evaluating the robustness of fact verification systems in the era of large language models | https://aclanthology.org/2025.naacl-long.534/ | 2025-04 | academic\nimplication: a perturbation-robustness harness (paraphrase, reorder, typo-level noise on both claims and evidence) belongs in the per-project ship gate; verdicts that flip under meaning-preserving edits should be auto-routed to human review rather than delivered.\n\n## 2026 sota joins verdict and explanation in one cheap model: faithlens (acl 2026 findings) is an 8b faithfulness-hallucination detector that jointly outputs a binary prediction and an explanation, trained via cold-start fine-tuning on filtered synthetic data plus rule-based rl rewarding both prediction correctness and explanation quality, and outperforms gpt-5.2 and o3 across 12 tasks.\nevidence: fetched acl anthology entry 2026.findings-acl.689 (si et al., findings of acl 2026, pp. 14068-14099). abstract claims superiority over gpt-5.2/o3 and 'a distinctive balance of trustworthiness, efficiency, and effectiveness'; per-task numbers require the full pdf.\nsource: faithlens: detecting and explaining faithfulness hallucination | https://aclanthology.org/2026.findings-acl.689/ | 2026-07 (acl 2026) | academic\nimplication: constructive-feedback generation and grounding verdicts do not need separate machinery: an explanation-rewarded detector produces the evidence-citing rationale as a first-class output, at 8b cost. this is the current-generation template for the autoqa's 'every criticism must carry its evidence' contract, and shows explanation quality can be an explicit training/reward target rather than a post-hoc add-on.\n\n## ragas-style prompt-chain faithfulness scoring materially underperforms finetuned/frontier judges on hallucination detection: on the 15k-sample halubench, ragas faithfulness scored 66.9% accuracy versus 87.4% for finetuned lynx-70b and 86.5% for gpt-4o; patronus has since shipped lynx 2.0 (8b, long-context, 8 hallucination subtypes including coreference and calculation errors).\nevidence: confirmed via patronus ai announcement and lynx paper coverage (july 11, 2024) with the halubench table (lynx-70b 87.4, gpt-4o 86.5, gpt-4-turbo 85.0, llama-3-70b 80.1, ragas 66.9); lynx 2.0 details from patronus docs. caveat: vendor-reported numbers on the vendor's own benchmark, though halubench is public on huggingface. ragas remains the most widely adopted oss rag-eval framework per 2026 practitioner surveys (atlan.com rag evaluation guide, 2026).\nsource: patronus ai lynx / halubench results | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | 2024-07 (lynx 1.0, foundational); lynx 2.0 later update | practitioner\nimplication: do not build the grounding stage on ragas-style statement-extraction prompt chains despite their ubiquity in 2026 tooling - the popularity/accuracy gap is ~20 points. if an off-the-shelf component is wanted, current-generation finetuned detectors (bespoke-minicheck, lynx 2.0, granite guardian, faithlens) dominate at equal or lower cost.\n\n## tools & artifacts\n- bespoke-minicheck-7b / minicheck family: specialized grounded-claim fact-checking models (0.4b-7b), sota on llm-aggrefact at 77.4 bacc; available on huggingface and ollama, ~400x cheaper than gpt-4-class verification | https://github.com/liyan06/minicheck | default candidate for the cheap deterministic claim-vs-evidence entailment stage of the autoqa\n- faithjudge (vectara): open-source llm-as-judge framework that anchors judgment with human-annotated peer examples on the same source; repo actively pushed through nov 2025 | https://github.com/vectara/faithjudge | the measured-best architecture pattern for the autoqa's per-project exemplar-anchored judge; directly reusable code\n- facts grounding public set + facts leaderboard: 860 public grounding examples (32k-token docs) plus google's dec 2025 four-suite factuality benchmark with kaggle-held private splits | https://huggingface.co/datasets/google/facts-grounding-public | source of judge-prompt designs, the eligibility-gate pattern, and human-adjudicated grounding labels for meta-evaluating our own judge\n- factbench / verify: acl 2025 factuality pipeline with supported/unsupported/undecidable labeling and 4,467 human-annotated content units released | https://github.com/launchnlp/factbench | template + validation data for the verifiability-triage front stage (fact vs opinion vs undecidable)\n- veriscore + verifastscore: verifiable-claim-only extraction/scoring for long-form text (emnlp findings 2024) and its single-model accelerated successor (emnlp findings 2025) that collapses extraction+verification into one fine-tuned pass | https://github.com/yixiao-song/veriscore | claim-extraction spec that skips opinions; verifastscore is the cost-reduction pattern if per-item pipeline latency matters\n- halubench + lynx 2.0 (patronus): 15k-sample hallucination detection benchmark with hard/ambiguous cases, plus an 8b long-context finetuned detector covering 8 hallucination subtypes | https://huggingface.co/patronusai | seeded-set material for measuring our checker's fail-class recall; lynx 2.0 is an off-the-shelf detector option\n- contextualjudgebench (salesforce): 2,000-pair benchmark for llm judges in rag/summarization settings with conditional criteria hierarchies (acl 2025) | https://arxiv.org/abs/2503.15620 | meta-evaluation instrument for any holistic judge configuration we consider; quantifies why we should not use one\n- citebench / citeeval-auto: multi-domain human-annotated citation-quality benchmark and model-based metrics that evaluate citations against full retrieval context (acl 2025) | https://arxiv.org/abs/2506.01829 | blueprint for the context-closed 'is this the right citation given everything in scope' check, beyond span-closed nli\n- hhem-2.1-open + vectara hallucination leaderboard: open small hallucination-detection model and the longest-running (2023-present, refreshed nov 2025 with 7,700 articles up to 32k tokens) hhem-scored llm leaderboard | https://huggingface.co/spaces/vectara/leaderboard | cheapest available checker tier, but its measured 66.7 bacc on hard cases (faithbench) bounds where it can be trusted - screening, not verdicts\n\n## disagreements\n- hhem's fitness as an automated grounding judge: vectara's own next-generation hallucination leaderboard (blog, 2025-11-19) still uses commercial hhem as the sole scorer and reports rates like gemini-2.5-flash-lite 3.3% vs gemini-3-pro 13.6%, while vectara's own research paper (faithjudge, emnlp industry 2025) shows hhem-2.1-open at only 66.7% balanced accuracy on faithbench - near-random on hard hallucinations - and builds an llm-judge replacement because of 'limitations observed in current hallucination detection methods'. detector choice materially changes model rankings; automated hallucination-rate leaderboards from different detectors disagree and cannot be treated as interchangeable ground truth.\n- atomic decomposition: the factscore/safe/ragas lineage (2023-2024) treats finer-grained atomic claims as strictly better for verification, but decomposition dilemmas (naacl 2025) shows decomposition degrades strong verifiers (80.0 -> 71.1 bacc on wice with minicheck) and dndscore (emnlp 2025) shows atomization and decontextualization actively conflict. the field has not converged: long-form factuality pipelines still default to atomic claims while the 2025 analysis papers say granularity must be adaptive.\n- specialized small detectors vs frontier llm judges: llm-aggrefact (through 2025) and faithlens (acl 2026) show small specialized models beating frontier llms (bespoke-minicheck-7b > claude-3.5/gpt-4o; faithlens-8b > gpt-5.2/o3), but faithjudge (2025) shows frontier reasoning judges with human-annotated exemplars beating all specialized detectors on adversarial sets (84.0 vs 71.2 bacc). both are right in their regime: fixed checkers win zero-shot on broad distributions; exemplar-anchored frontier judges win on hard, domain-specific cases - which is an argument for a two-tier architecture rather than either alone.\n- citation-support standard: alce-style span-closed nli citation precision/recall remains the de-facto automated standard, but citeeval (acl 2025) calls nli-vs-cited-spans 'a suboptimal proxy' and citeguard (acl 2026) additionally doubts llm-as-judge reliability for citation evaluation - i.e., the two dominant automated approaches each have a 2025-2026 paper attacking them, with no settled successor.\n\n## gaps\n- no published measurements of run-to-run verdict flip rates (self-consistency) for grounding checkers or llm faithfulness judges on identical inputs - the taxonomy's 'flip rate: noise vs underspecification signal' question has no literature anchor found; specialized classifiers (minicheck/hhem) are deterministic by construction but that property is unstated in evaluations.\n- positive-claim verification of evaluative statements (warranted vs empty praise, 'stated severity matches evidence strength' calibration claims) is essentially unstudied: the entire 2024-2026 literature verifies factual/descriptive claims against documents; no benchmark tests whether a checker can verify claims like 'this response is complete' or 'this citation is the strongest available'. the autoqa's core use case - verifying human evaluative writeups about ai outputs - is a domain transfer no found benchmark covers.\n- no dedicated 2025-2026 study found on evidence-swap/citation-theater probes for verifiers (does the verdict flip when cited evidence is replaced with equally-plausible irrelevant text?). facteval perturbs claims, not evidence; facts documents gaming by response vagueness, not by decorative citation. this test would need to be built in-house.\n- aggregation from claim-level verdicts to item-level pass/fail with severity weighting is under-researched: factscore-style %-supported and facts' all-or-nothing eligibility+grounding are the only documented rules; ragtruth/faithbench severity taxonomies (benign/questionable/unwanted, evident/subtle) exist as labels but faithjudge found judges classify the middle severity classes unreliably and retreated to binary - no validated severity-weighted aggregation scheme found.\n- judge-vs-adjudicated-human agreement ceilings per axis type are only available for grounding (~65 macro-f1 in facts on held-out human labels; 84 bacc for faithjudge on faithbench) - no comparable 2025-2026 numbers found for instruction-compliance or tone/style axes from the faithfulness literature, so the lane-assignment decision (judge-autonomous vs assisted vs human-only per axis) lacks published anchors outside factual grounding.\n\n## verifications\n- claim: grounded claim-verification is commoditized but ceilinged: the best model on the llm-aggrefact benchmark (11 datasets of claim-vs-grounding-document verification) is a specialized 7b model, bespoke-minicheck-7b, at 77.4 average balanced accuracy, with 0.4-0.8b specialized checkers (factcg, minicheck-flan-t5-l) within ~2.5 points of frontier llms like claude-3.5-sonnet (77.2) and gpt-4o (75.9).\n verdict: confirmed | fetched https://llm-aggrefact.github.io/ on 2026-07-14; every number matches exactly: bespoke-minicheck-7b 77.4 (rank 1), claude-3.5 sonnet 77.2, granite guardian 3.3 8b 76.5, gpt-4o-2024-05-13 75.9, factcg-deberta-l (0.4b) 75.6, minicheck-flan-t5-l (0.8b) 75.0, llama-3.1-405b 74.4. benchmark is 11 datasets of grounded factuality (supported/unsupported vs grounding docs), avg balanced accuracy - as claimed. the \"within ~2.5 points\" framing is accurate and even conservative: factcg (75.6) is only 1.6 below claude-3.5-sonnet and above gpt-4o. supersession search found nothing beating 77.4: acv (may 2026) reports 76.5 training-free; an acl 2026 paper ranks second to a post-trained metric; paladin-mini beats bespoke-minicheck only on its own separate benchmark subsets, not llm-aggrefact overall. caveats on the interpretive \"ceilinged\" framing: (1) the leaderboard's frontier entries are 2024-era models (claude-3.5-sonnet, gpt-4o-2024-05-13, llama-3.1-405b) - no claude 4/gpt-5-class/o3 entries, so \"specialized ~ frontier\" reflects 2024 frontiers; (2) \"verifying the verifiers\" (arxiv 2506.13342, june 2025) finds ~16% of benchmark labels ambiguous/incorrect and that few-shot frontier llms reach top-tier performance, suggesting the ~77 plateau is partly benchmark label noise rather than a pure task ceiling; (3) \"verify with caution\" (arxiv 2501.14883) shows models with similar aggregate bacc make very different instance-level predictions. none of these contradict the stated facts.\n- claim: on adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on faithbench, prior detectors hit ~50% accuracy (negligible), hhem-2.1-open 66.7% and bespoke-minicheck 71.2% balanced accuracy, zero-shot frontier judges stay below 78%, while faithjudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 f1.\n verdict: partially_confirmed | verified against arxiv:2505.04847 (v2 dated nov 6, 2025; emnlp 2025 industry track; vectara/waterloo authors - source and date correct). confirmed: (1) paper states prior detectors, including llm classifiers, achieved \"near 50% accuracy\" on faithbench; (2) hhem-2.1-open = 66.7% balanced accuracy on faithbench (claim-wise, table 1) - though flagged with an asterisk because hhem was used to adversarially select faithbench articles; (3) faithjudge with o3-mini-high = 84.0% balanced accuracy / 82.1 f1-macro (table 2), best zero-shot judge on faithbench was o3-mini-high at 68.8%; (4) faithjudge vs facts grounding head-to-head 70.8 vs 54.3 f1 on faithbench confirmed (table 4); (5) all three stated failure modes confirmed (underprediction for command-r/mistral/qwen generators; benign/questionable misclassified - only 10/84 benign labeled correctly; specificity slightly decreases as in-context examples increase). errors: (a) bespoke-minicheck's 71.2 is not its faithbench score - 71.2 is its balanced-accuracy average across all four datasets (aggrefact, ragtruth, tofueval-mb, faithbench) in table 1; its actual faithbench score is 60.1% claim-wise / 55.7% summary-wise. (b) minor: the \"below 78% balanced accuracy\" figure for zero-shot judges is the paper's cross-dataset average claim, not a faithbench-specific figure (on faithbench zero-shot judges max out at 68.8%, so the claim still holds directionally). supersession check: searches found no 2026 work surpassing faithjudge on faithbench; it remains the reported state of the art as of july 2026.\n corrected: on adversarially-hard hallucination sets, specialized detectors collapse and few-shot anchoring with human-annotated exemplars is the measured fix: on faithbench, prior detectors hit ~50% accuracy (negligible); fine-tuned detectors stay weak (hhem-2.1-open 66.7% and bespoke-minicheck 60.1% claim-wise balanced accuracy on faithbench; bespoke-minicheck averages 71.2% across the four benchmark datasets); zero-shot frontier judges reach at most 68.8% on faithbench (and stay below 78% averaged across datasets), while faithjudge - prompting o3-mini-high with human-annotated peer responses to the same source document - reaches 84.0% balanced accuracy / 82.1 f1-macro.\n- claim: claim decomposition helps weak verifiers but actively degrades strong ones: with minicheck as verifier on wice, no-decomposition scores 80.01 balanced accuracy while factscore-style atomic decomposition drops it to 71.11; with the weaker alignscore verifier the same decomposition improves results - and gains only reappear as input complexity grows, with best results when sub-claim count does not exceed input complexity.\n verdict: confirmed | verified against the full pdf (extracted text at [local path withheld]). (1) source says exactly this: table 2 (wice) shows minicheck baseline bacc 80.01 vs factscore decomposition 71.11 (f1 72.32 vs 59.90); all decomposition methods hurt minicheck on wice. with the weaker alignscore verifier, factscore decomposition improves bacc 54.80->56.87 (and wice-style 56.26); note veriscore decomposition slightly hurt alignscore too, but the claim as stated (\"the same decomposition,\" i.e., factscore-style) is accurate. section 4.2 states verbatim that \"decomposition generally benefits weaker verifiers, while it tends to negatively affect stronger verification systems.\" section 6.2-6.4 confirms gains reappear as input complexity grows (complexity scale-up experiments; felmshort scale-down degrades), and figure 3 discussion states \"for each level, the maximum f1 is observed when the number of the decomposed sub-claim is less than or equal to the complexity level.\" evidence-summary side facts also check: felm minicheck f1 48.10->67.5-68.1, gpt-4o-mini f1 71.56->54.34 (table 3); four-way error taxonomy (context omission, ambiguity, over-decomposition, meaning alteration) present in section 5. (2) date: acl anthology lists april 2025 (naacl 2025, albuquerque, pp. 6313-6336); the conference ran apr 29-may 4, so \"2025-05\" is a trivial one-month imprecision, not a substantive error. (3) supersession search: later work (e.g., presupposition-free question decomposition, arxiv 2508.16838; \"alignment bottleneck in decomposition-based claim verification,\" feb 2026) extends the decomposition-tradeoff line but does not refute the verifier-strength finding.", "10": "domain: hybrid-statistical\n\n## prediction-powered inference (ppi/ppi++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a gpt-4 judge in 'autoeval done right'. [load-bearing]\nevidence: arxiv:2403.07008 (boyeau, angelopoulos, yosef, malik, jordan, uc berkeley) abstract confirmed: 'increase the effective human-labeled sample size by up to 50% on experiments with gpt-4'; v3 camera-ready posted 2026-06-01. the unconditional-coverage property is restated and empirically demonstrated in the glide paper (arxiv:2605.31278) and is the foundation of the entire 2026 llm-judge debiasing literature (precise/aaai 2026, arxiv:2606.05308 ranking extension).\nsource: autoeval done right: sample-efficient human evaluation via prediction-powered inference | https://arxiv.org/abs/2403.07008 | 2024-03 (v3 camera-ready 2026-06-01) | academic\nimplication: the autoqa meta-evaluation layer should treat human gold labels as bias-correctors for judge-derived population metrics (pass rates, per-attempter/per-criterion error rates), never report raw judge numbers. because validity survives a bad judge, pooled/hierarchical cross-project validation is statistically safe even where per-project gold sets are tiny - directly resolving taxonomy q1's gold-set-arithmetic fork toward pooled certification with per-project bias correction.\n\n## cascaded selective evaluation (trust or escalate, iclr 2025 oral) delivers a provable, user-specified human-agreement guarantee for llm judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a chatbot arena subset where gpt-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models. [load-bearing]\nevidence: arxiv:2407.18370 abstract fetched directly (80% agreement / ~80% coverage claim confirmed); iclr 2025 oral status confirmed at iclr.cc/virtual/2025/oral/31838; fixed-sequence-testing calibration mechanism confirmed via openreview pdf snippets and secondary reviews (83 citations as of 2026-07). its 'simulated annotators' method (in-context simulation of diverse annotators) is the confidence estimator that makes the guarantee achievable at high coverage.\nsource: trust or escalate: llm judges with provable guarantees for human agreement (iclr 2025 oral) | https://arxiv.org/abs/2407.18370 | 2024-07-25 (iclr 2025-04) | academic\nimplication: this is the direct architectural blueprint for 'which items get the single human touch': the human queue is exactly the abstention set of a confidence-calibrated cascade, and the abstention rate is a tunable dial trading human budget against a guaranteed agreement level. caveat for taxonomy q1: the guarantee is agreement-with-human-majority, so this component's validity is capped at human panel reliability - it certifies consistency-of-application, not truth.\n\n## the minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n*(1-rho^2) where n* is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with r^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes. [load-bearing]\nevidence: arxiv:2605.16354v1 (jane paik kim, stanford psychiatry, 2026-05-08) html fetched: derives sample-size formulas from the doubly-robust estimator's asymptotic variance; worked examples confirmed (n*=200, r^2=0.7, n=2000 -> 65 human labels; n=400 -> 100). explicit caveats: pilot overestimation of r^2 breaks precision guarantees (use conservative values), framework assumes human ratings are a reliable gold standard, single-rater scope.\nsource: augmenting human evaluation with llm judges: how many human reviews do you need? | https://arxiv.org/html/2605.16354v1 | 2026-05-08 | academic\nimplication: answers taxonomy q1's gold-set question with arithmetic instead of convention: run a small pilot to estimate per-axis judge-human correlation, then size each project's gold set from the formula with a conservative correlation estimate. because the floor depends on rho, per-axis correlation heterogeneity argues for stratified gold sampling by criterion, and low-rho axes are where gold budget must concentrate.\n\n## using the llm's own confidence signals to choose which items receive human annotation (confidence-driven inference, building on active statistical inference) cut required human annotations by >25% in all three tested settings while keeping provably valid confidence intervals that remain safe even if the llm annotations are poor - but a 2026 iclr paper finds the opposite in the sequential regime, where near-uniform sampling at the budget ceiling beat uncertainty-driven querying. [load-bearing]\nevidence: arxiv:2408.15204 (gligoric, zrnic, lee, candes, jurafsky; naacl 2025) abstract fetched: '>25% reduction in each of three settings' (politeness, stance, bias) with validity guarantees regardless of llm quality. foundation: active statistical inference (zrnic & candes, icml 2024, arxiv:2403.03208): label where the model is uncertain, provably valid cis, fewer labels than uniform. counter-evidence verified separately (see disagreements): arxiv:2604.18569 (iclr 2026).\nsource: can unconfident llm annotations be used for confident conclusions? (naacl 2025) + active statistical inference (icml 2024) | https://arxiv.org/abs/2408.15204 | 2024-08-27 (v2 2025-02-08; naacl 2025-04) | academic\nimplication: uncertainty-routed allocation of the scarce human touch is the statistically defensible default and comes with a safety net (validity even when the judge is wrong), but the gain over random sampling is regime-dependent and contested - so the foundation should make the routing policy (uncertainty-weighted vs uniform vs stratified) a measured, per-project choice with a built-in a/b against uniform, not a hard-wired assumption.\n\n## glide (may 2026) is an open-source industrial library unifying ppi++, stratified ppi, predict-then-debias bootstrap, and active inference with a decision tree for method selection, and it quantifies when ppi helps: effective-sample gain is ~1.0x at judge-human correlation rho=0.1, ~2.2x at rho=0.9; clt-based estimators need roughly >=50 human labels per stratum (below that use bootstrap ptd); in a real agent-safety case (rho=0.59, ~13-point judge bias) 100 human labels became worth 143-157.\nevidence: arxiv:2605.31278v2 html fetched (martinon, merad, raki; emerton data): monte carlo with 500 human + 1000 proxy labels shows effective n ~500 at rho=0.1 vs ~1100 at rho=0.9; r-judge case study with claude sonnet as judge; library at github.com/emertondata/glide with reproducible validation notebooks; limitations: means/proportions only, i.i.d. assumptions, no anytime-valid inference.\nsource: industrializing prediction-powered inference: the glide library for reliable genai and agentic systems evaluation | https://arxiv.org/html/2605.31278v2 | 2026-05 (v2 2026-06) | practitioner\nimplication: ready-made tooling exists for the statistical layer; more importantly it publishes the correlation-vs-benefit curve and the >=50-labels-per-stratum threshold the foundation can budget against: if a project's judge-human correlation on an axis is below ~0.3, ppi buys nothing there and gold labels must carry that axis alone.\n\n## for estimating scores from noisy judges, ppi++/efficient-influence-function estimators produce near-identical, shortest confidence intervals - reported 3-15x narrower than rogan-gladen-style misclassification-correction estimators depending on the human-labeling ratio, with the advantage largest when the judge is close to random guessing.\nevidence: arxiv:2601.05420 (chen, lu, li, guo, li; 2026-01-08) abstract fetched: unifies measurement-error correction and ppi classes via semiparametric efficiency theory, derives eif-based estimators, characterizes when ppi-style strictly dominates; the 3-15x interval-width figure comes from the paper's empirical section as surfaced in search-indexed full text (not independently recomputed). code: github.com/yiqunchen/debias-llm-as-a-judge.\nsource: efficient inference for noisy llm-as-a-judge evaluation | https://arxiv.org/abs/2601.05420 | 2026-01-08 | academic\nimplication: standardize the estimation layer on ppi++/eif-style residual calibration rather than sensitivity/specificity (confusion-matrix) correction when the target is a rate or mean - but note the certification literature pushes the other way for hypothesis tests (see disagreements).\n\n## for certifying that a failure rate is below a threshold (the low-base-rate pass/fail regime), the 'noisy but valid' framework (iclr 2026) uses a small human calibration set to estimate judge tpr/fpr, applies a variance-corrected test to the large judge-labeled stream with finite-sample type-i error control, and derives exact conditions under which judge-based testing has higher statistical power than direct human evaluation.\nevidence: arxiv:2601.20913 (feng, shen, balashankar, gerner-beuerle, rodrigues; submitted 2026-01-28, accepted iclr 2026 per arxiv page) abstract fetched: variance-corrected critical threshold, finite-sample type-i control under calibration uncertainty, validation on jigsaw/hate speech/saferlhf, and a quantified 'oracle gap' - the power cost of having to estimate judge reliability rather than knowing it.\nsource: noisy but valid: robust statistical evaluation of llms with imperfect judges | https://arxiv.org/abs/2601.20913 | 2026-01-28 | academic\nimplication: for attempter-level or batch-level certification ('this attempter's error rate is below x'), a judge-plus-small-calibration design can be provably more powerful than human-only review - reframing the human panel's job as estimating judge error rates, not per-item adjudication. this is direct evidence for the zero-per-item-touch + calibration-audit branch of taxonomy q6's fork.\n\n## naive reporting of judge-scored evaluations is systematically biased by judge sensitivity/specificity; a plug-in correction with confidence intervals propagating both test-set and calibration-set uncertainty fixes it, remains unbiased under distribution shift between calibration and test data, includes adaptive allocation of calibration labels, and characterizes regimes where corrected judge-based evaluation is more reliable than human-only evaluation.\nevidence: arxiv:2511.21140 (lee, zeng, jeong, sohn, kangwook lee; submitted 2025-11-26, v4 2026-05-31, accepted icml 2026 per arxiv page) abstract fetched. the calibration-to-test distribution-shift robustness is the distinctive property versus standard ppi's i.i.d. assumption.\nsource: how to correctly report llm-as-a-judge evaluations | https://arxiv.org/abs/2511.21140 | 2025-11-26 (v4 2026-05-31) | academic\nimplication: supports the taxonomy q1 proposal to ban raw judge-agreement numbers from all reporting: every reported metric should be the corrected estimator with a ci that includes calibration uncertainty. the shift-robustness result matters specifically because attempter adaptation (taxonomy q8) shifts the live item distribution away from the calibration set over time.\n\n## conformal prediction now provides per-item uncertainty for llm-as-judge ratings: emnlp 2025 work builds guaranteed-coverage score intervals from a single judging run (with an ordinal boundary adjustment and a lower-bias midpoint score), and april 2026 work shows conformal set width is a genuine per-instance reliability signal (r_s=+0.576 with reliability, n=1,918; widths correlate ~0.32-0.38 across different judges) with reliability driven more by the criterion than the judge model (relevance avg set size ~3.0 vs fluency/consistency ~4.9 on a 1-5 scale).\nevidence: sheng, liu, he, zhao, kang, 'analyzing uncertainty of llm-as-a-judge: interval evaluations with conformal prediction', emnlp 2025 main (aclanthology.org/2025.emnlp-main.569), abstract confirmed via neurips 2025 workshop listing. gupta & kumar, arxiv:2604.15302 (2026-04-16, under review): split conformal on summeval; also finds 33-67% of documents contain intransitive judge preference cycles despite low aggregate violation rates.\nsource: analyzing uncertainty of llm-as-a-judge (emnlp 2025) + diagnosing llm judge reliability (arxiv:2604.15302) | https://arxiv.org/abs/2604.15302 | 2025-11 (emnlp) / 2026-04-16 | academic\nimplication: conformal set width is a cheap, distribution-free routing statistic for the human-touch queue that transfers across judges, and the criterion-dominates-judge finding is empirical support for taxonomy q2/q5's axis-triage: per-criterion reliability ceilings must be measured and some axes routed human-only regardless of judge choice.\n\n## judge-vs-system drift can be attributed with anytime-valid statistics: a frozen human-labeled anchor set periodically re-scored by the live judge, monitored with betting e-processes, correctly attributed a silent judge version bump in 60/60 runs with zero misattribution (rolling z-test baseline false-alarmed on 75% of drift-free streams), at 0.21-0.64x the cost of strong-judging every item.\nevidence: arxiv:2606.15474 (yitao li, 2026-06-13) abstract fetched: 'one-way identification (only the judge can move the anchors)'; three-verdict output (none/system/judge); strict-prompt change attributed in 110/120 runs; replicated on tl;dr summarization 240/240. single-author preprint, not yet peer-reviewed - treat quantitative claims as provisional.\nsource: who drifted: the system or the judge? anytime-valid attribution in llm evaluation pipelines | https://arxiv.org/abs/2606.15474 | 2026-06-13 | academic\nimplication: taxonomy q9's three-moving-parts problem has an existing statistical design for one axis: freeze a per-project human-labeled anchor set at onboarding, interleave it continuously into the judge stream, and use e-process alarms to separate 'judge changed' from 'attempter population changed' before triggering rubric revision or de-graduation.\n\n## in sequential prediction-powered mean estimation, uncertainty-based active querying contributed little: the smallest confidence widths occurred when query probabilities were near-constant at the budget ceiling, and theory shows optimized query probabilities converge to the maximum allowed constant rate when chosen without reference to covariates.\nevidence: arxiv:2604.18569 (sfyraki & wang, submitted 2026-04-20, accepted iclr 2026 per arxiv page) abstract fetched: empirical finding that the uncertainty-blend weight near the constant term minimizes ci width, plus non-asymptotic analysis and no-regret query-probability selection corroborating near-uniform optimality in this regime.\nsource: revisiting active sequential prediction-powered mean estimation | https://arxiv.org/abs/2604.18569 | 2026-04-20 | contrarian\nimplication: direct counter-evidence to uncertainty-routed human sampling for population estimation in streaming settings: the foundation must not assume uncertainty routing dominates - measure the routing policy's marginal value per project, and expect uniform-at-budget to be near-optimal when the goal is estimation rather than per-item verdict quality.\n\n## commercial tooling for confidence-scored llm judgments with human calibration is mature: cleanlab's trustworthy language model attaches a real-time trustworthiness score to every llm response, supports custom evaluation criteria, explicitly documents calibrating trust scores against human quality ratings, and publishes benchmarks (nov 2025) for auto-flagging incorrect structured outputs for human review.\nevidence: cleanlab tlm documentation (help.cleanlab.ai/tlm/: quickstart, custom-eval calibration-against-human-ratings tutorial, advanced usage with low-score explanations) and 2025-11-18 structured-outputs benchmark blog post, all fetched via search with page content confirmed. vendor source: benchmark numbers are self-reported.\nsource: cleanlab trustworthy language model (tlm) documentation and benchmark | https://help.cleanlab.ai/tlm/ | 2025-11-18 (benchmark); docs current 2026 | practitioner\nimplication: the 'score every judgment, calibrate the score against a human-rated set, route low-trust items to humans' pattern is already productized - the foundation can specify this as a component contract (per-verdict trust score + human-calibration hook) rather than inventing it, while keeping vendor-neutrality.\n\n## tools & artifacts\n- ppi_py: reference open-source python library for prediction-powered inference (angelopoulos, zrnic et al.): means, quantiles, glms, ppi++ power tuning. | https://github.com/aangelopoulos/ppi_py | drop-in statistical layer for computing bias-corrected pass rates and attempter metrics from many judge labels + few human labels; covers estimand types glide does not (quantiles, regressions).\n- glide: open-source industrial ppi library (emerton data, may 2026): ppi++, stratified ppi++, predict-then-debias bootstrap, active statistical inference, uniform/stratified/active/cost-optimal samplers, plus a method-selection decision tree. | https://github.com/emertondata/glide | closest existing implementation of this domain's whole playbook, including the sampler that decides which items get human labels and effective-sample-size roi reporting; validated on agent-safety trajectories with a claude judge.\n- active-inference: code for active statistical inference (zrnic & candes, icml 2024): uncertainty-guided label collection with provably valid confidence intervals. | https://github.com/tijana-zrnic/active-inference | reference implementation for uncertainty-routed allocation of the scarce human annotation budget.\n- debias-llm-as-a-judge: code for chen et al. 2026 (arxiv:2601.05420): efficient-influence-function and ppi++ estimators for noisy llm-judge evaluation of scores and win rates. | https://github.com/yiqunchen/debias-llm-as-a-judge | implements the estimator class this domain recommends standardizing on for judge-score estimation, with the rogan-gladen comparison built in.\n- cleanlab trustworthy language model (tlm): commercial system that attaches a calibratable trustworthiness score to every llm response, with custom evaluation criteria and tutorials for calibrating scores against human ratings (benchmark published 2025-11). | https://help.cleanlab.ai/tlm/ | productized version of the per-verdict-confidence + human-calibration + route-low-trust-to-human pattern; useful as a component contract template even if not adopted as a vendor.\n- cascaded selective evaluation (trust or escalate): iclr 2025 oral method: judge cascade with fixed-sequence-testing calibration delivering user-specified human-agreement guarantees; simulated annotators confidence estimation. | https://arxiv.org/abs/2407.18370 | the architectural template for the autoqa's escalation spine: guarantee level and abstention rate become the two governing dials for the human-touch budget.\n\n## disagreements\n- uncertainty-routed vs uniform human sampling: zrnic & candes (icml 2024) and gligoric et al. (naacl 2025) report provably valid inference with >25% fewer human labels when the model's uncertainty guides which items humans label; sfyraki & wang (iclr 2026, arxiv:2604.18569) find in the sequential regime that the uncertainty component contributes little and near-constant query probabilities at the budget ceiling give the tightest intervals. both sides are peer-reviewed; the honest read is that active-sampling gains are regime-dependent (batch design + informative confidence signals vs sequential streaming), so routing policy must be validated per project, not assumed.\n- explicit judge-error modeling vs black-box residual calibration: chen et al. (arxiv:2601.05420) show rogan-gladen-style tpr/fpr-correction estimators yield intervals reportedly 3-15x wider than ppi++/eif for score estimation; feng et al. (iclr 2026, arxiv:2601.20913) argue the opposite direction for certification - explicitly modeling judge tpr/fpr yields better-powered hypothesis tests than treating ppi as a black box. partly reconcilable (different targets: estimation vs testing), but they issue opposing default recommendations and the synthesis should pick per use-case: ppi++/eif for dashboards and rates, tpr/fpr-corrected tests for threshold certification.\n- what the human labels are for - target vs corrector: trust or escalate guarantees agreement with human majority preference (capping the system's validity at human panel reliability, i.e., consistency-of-application); the ppi/certification line treats human labels as gold-standard truth for bias correction, and both 'noisy but valid' and 'how to correctly report' prove regimes where corrected judge-based evaluation is strictly more reliable/powerful than human-only evaluation, while kim (arxiv:2605.16354) explicitly warns her framework breaks when human raters are themselves unreliable. no paper in this literature resolves what to do when the gold itself is noisy - the taxonomy q1 construct choice (instruction-satisfaction vs consensus-prediction) is genuinely open in the statistics literature.\n\n## gaps\n- no 2025-2026 paper found that applies ppi, selective escalation, or conformal routing specifically to qa of human-produced annotations (reviewing human attempters); every application evaluates model outputs with humans as gold. transfer is plausible but unvalidated, especially under adversarial attempter adaptation, which violates the i.i.d. assumptions most ppi variants rely on (only arxiv:2511.21140 claims calibration-to-test shift robustness).\n- all ppi-family guarantees are population-level (rates, means, rankings) and selective-evaluation guarantees are marginal, not conditional: a system can meet its aggregate agreement/coverage target while concentrating its failures on specific attempters, criteria, or hard-item subgroups. conditionally valid (per-subgroup) escalation guarantees for llm judges remain an open problem in this literature.\n- the specific fork 'one shallow human touch per item vs zero-touch plus randomized deep audits' has no direct head-to-head study measuring total error caught per human-hour; the statistical literature implicitly favors the zero-touch+audit side (valid conclusions with human labels on a small fraction of items) but only for measurement/certification objectives, not for per-item verdict correction or attempter coaching.\n- calibration economics for rubric-based qa: trust or escalate's simulated annotators and fixed-sequence testing were validated on pairwise preference tasks; required calibration-set sizes and achievable coverage for per-criterion rubric verdicts (the autoqa's actual workload) are not established anywhere i found.\n- could not open the trust or escalate openreview forum (captcha); handling of abstained items and calibration-set-size details confirmed only from abstract-level and secondary sources, not reviewer discussion. the 3-15x interval-width figure in chen et al. was corroborated from search-indexed paper text, not independently recomputed.\n- no published head-to-head comparison of conformal-width routing vs simulated-annotators-style confidence routing vs verbalized-confidence routing for deciding the human-escalation queue - the three candidate mechanisms have never been benchmarked against each other.\n\n## verifications\n- claim: prediction-powered inference (ppi/ppi++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with always-valid confidence intervals - coverage holds regardless of judge quality (a worse judge widens intervals, never invalidates them) - and increased effective human sample size by up to 50% with a gpt-4 judge in 'autoeval done right'.\n verdict: partially_confirmed | verified directly against arxiv:2403.07008: authors (boyeau, angelopoulos, yosef, malik, jordan) correct; abstract verbatim contains 'increase the effective human-labeled sample size by up to 50% on experiments with gpt-4'; abstract claims methods 'improve sample efficiency while remaining unbiased'; version history confirms v1 2024-03-09 and v3 2026-06-01 with comment 'camera-ready paper version' - dates as claimed. glide (arxiv:2605.31278, martinon/merad/raki, v1 2026-05-29, icml 2026 workshop) exists and states ppi 'combines both into debiased estimates with valid confidence intervals', consistent with the claim, though the exact phrase 'regardless of judge quality' was not verifiable in its abstract. the 2026 ppi-for-llm-judge literature is real and active (precise/amazon science, arxiv:2606.05308 ranking extension, arxiv:2601.05420, arxiv:2601.20913), so the claim is not superseded - it is being extended. two corrections: (1) 'always-valid confidence intervals' is a misuse of a term of art - 'always-valid'/anytime-valid refers to sequential inference; ppi/ppi++ intervals are fixed-n asymptotic (clt-based) intervals, so coverage is guaranteed only asymptotically and finite-sample coverage can degrade with very small gold sets. (2) the judge-quality-independence property is a general ppi/ppi++ property (angelopoulos et al. 2023) rather than something stated in the autoeval abstract itself; the claim's framing correctly captures its substance (worse judge -> wider intervals via lower correlation, coverage preserved) but slightly overstates its strength and its provenance in this specific source.\n corrected: prediction-powered inference (ppi/ppi++) combines a small human gold set with a large judge-labeled set to produce unbiased estimates with asymptotically valid confidence intervals whose coverage does not depend on judge accuracy (a worse judge widens intervals rather than breaking coverage, subject to standard asymptotic/regularity conditions and a sufficiently large gold set); 'autoeval done right' (arxiv:2403.07008, boyeau, angelopoulos, yosef, malik & jordan) reports increasing the effective human-labeled sample size by up to 50% in experiments with gpt-4.\n- claim: cascaded selective evaluation (trust or escalate, iclr 2025 oral) delivers a provable, user-specified human-agreement guarantee for llm judges by calibrating a confidence threshold with fixed-sequence testing on a small human calibration set, escalating low-confidence items up a judge cascade and abstaining (to humans) when even the strongest judge is unconfident - achieving guaranteed >80% human agreement at ~80% coverage on a chatbot arena subset where gpt-4 alone almost never reaches 80% agreement, with ~88% of judged items handled by much cheaper models.\n verdict: confirmed | all load-bearing elements check out against the primary sources. (1) arxiv:2407.18370 abstract (fetched directly) states cascaded selective evaluation provides a provable human-agreement guarantee at a user-specified level, and that on a chatbot arena subset \"where gpt-4 almost never achieves 80% human agreement,\" the method \"guarantees over 80% human agreement with almost 80% test coverage\" even using mistral-7b. (2) paper body (arxiv.org/html/2407.18370v1) confirms the mechanism: threshold calibration via fixed sequence testing (bauer, 1991) on a small human calibration set (|d_cal|=500 for chatarena/tl;dr, 392 for auto-j), cascade escalation from cheap to strong judges on low confidence, abstention (output empty set) when even the strongest judge is unconfident, and simulated annotators as the confidence estimator enabling high coverage. (3) the ~88% figure is exact: \"79.1% of all samples, among which 88.1% are evaluated by substantially cheaper mistral-7b or gpt-3.5 instead of gpt-4\" (table 3: gpt-4 handles only 17.5%). (4) iclr 2025 oral confirmed at iclr.cc/virtual/2025/oral/31838 (jung, brahman, choi); dates correct (v1 2024-07-25; iclr 2025 in april 2025). two trivial nuances, neither rising to a correction: the paper defines abstention as returning empty set / excluding from coverage - deferral \"to humans\" is the intended framing but abstained items are not actually routed to human annotators in the experiments; and the guarantee holds with probability 1-delta over calibration-set sampling (standard for such guarantees, and consistent with \"provable, user-specified\"). supersession check: not superseded - but a newer alternative exists: scope (arxiv:2602.13110, icml 2026 poster, feb 2026) uses conformal calibration with a bidirectional preference entropy signal and reports better calibration/coverage than simulated annotators; it complements rather than invalidates the claim about this paper. the \"83 citations\" figure was not independently verified but is plausible and not load-bearing.\n- claim: the minimum viable human gold set is computable in closed form: with a doubly-robust two-stage design, required human labels converge to a floor of n*(1-rho^2) where n* is the target effective sample size and rho the judge-human correlation - e.g., a target effective n=200 with r^2=0.70 and 2,000 judge ratings needs only 65 human labels - with diminishing returns from adding more judge labels and up to ~13% further savings from stratified allocation when judge reliability varies across evaluation axes.\n verdict: confirmed | all elements verified against the source (https://arxiv.org/html/2605.16354v1, jane paik kim, stanford dept. of psychiatry and behavioral sciences, submitted 2026-05-08 - date and attribution correct). (1) floor formula: paper states required human reviews converge to n*(1-rho^2) as n increases - exact match. (2) worked example: n*=200, r^2=0.70, n=2000 -> 65 human labels, confirmed verbatim; the n=400/100 figure is stated in the paper as \"with a budget of 100 human ratings, n can be reduced to 400 while achieving target power\" - same numbers, slightly different framing (human budget fixed, n solved) but mathematically equivalent to the claim. (3) stratification savings: paper reports \"up to 12.9%\" reduction when r^2 gap is large (0.8 vs 0.1) - matches \"~13%\", with the paper adding that moderate gaps yield only ~2%. (4) diminishing returns from more judge labels: explicitly stated (\"each successive increase in n produces diminishing reductions in n\"). (5) all three caveats in the evidence summary (pilot r^2 overestimation breaking precision guarantees, human-as-gold-standard assumption, single-rater scope) appear in the paper's limitations. supersession check: two searches found no v2 revision, no citing papers, and no later work replacing or contradicting the result as of 2026-07-14. minor precision notes only: \"up to ~13%\" should strictly be \"up to 12.9% in the extreme r^2-gap case, ~2% for moderate gaps,\" and the floor is asymptotic (approached as n grows), not attainable at finite n - the claim's own phrasing (\"converge to a floor,\" \"up to\") already reflects both.", "11": "domain: rubrics-recent\n\n## openai healthbench meta-evaluates its llm grader per-criterion against physician majority grades using macro f1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: gpt-4.1 grader mf1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician mf1 is only 0.569-0.730 with wide individual spread. [load-bearing]\nevidence: full paper text (section 8.1, tables 5-6) confirmed: 60,896 meta-examples over 34 physician-consensus criteria; 'typical physician' baseline computed by scoring each physician against the others exactly as the model is scored; grader model choice matters (gpt-4.1 0.709 > o4-mini 0.692 > o3 0.681 > gpt-4.1-nano 0.580); prompt and individual criterion phrasings were tuned 'so that their intent was unmistakable to the grader'. overall benchmark std across 16 full runs is ~0.002.\nsource: healthbench: evaluating large language models towards improved human health (openai) | https://arxiv.org/html/2505.08775v1 | 2025-05-13 | primary\nimplication: the 2025-26 standard validation design for a rubric judge: per-criterion binary meta-eval against adjudicated expert-consensus labels with a chance-robust class-balanced statistic, reporting the judge as a percentile of individual human raters - not raw accuracy vs. a single reviewer. the human agreement ceiling is empirically low (mf1 ~0.6-0.7 even among physicians on physician-written criteria), so 'exceeds median human' is an attainable and meaningful bar, and consensus-adjudicated labels (not single-reviewer labels) must be the target. also: rubric criterion text is a tunable judge-alignment artifact, not fixed scripture.\n\n## healthbench's production rubric format is per-item (conversation-specific) criteria written by 262 physicians (48,562 criteria), each an independently-judged binary met/not-met check with a nonzero weight from -10 to +10 (negative criteria encode pitfalls), aggregated as a weighted sum normalized by max attainable positive points.\nevidence: confirmed in fetched full text: 'each rubric criterion has an associated nonzero point value between -10 and 10, with negative points used for criteria that are undesirable'; grader judges each criterion independently; item score can go negative; the vast majority of criteria were written specifically for one example.\nsource: healthbench (rubric structure, section 3) | https://arxiv.org/html/2505.08775v1 | 2025-05-13 | primary\nimplication: reference design for the autoqa verdict aggregation question: per-item compiled criteria (not one generic project rubric), explicit negative-weighted pitfall criteria as first-class citizens, independent binary judging per criterion, and a transparent weighted-sum aggregation - i.e., claim-level findings with an arithmetic, auditable roll-up rather than a holistic judge verdict.\n\n## a large data vendor's rubrics-as-rewards found that llm-synthesized rubrics grounded in expert reference answers performed on par with human-authored rubrics as reward signals (0.359 vs 0.348), that prompt-specific rubrics substantially beat generic ones, and that the one component synthetic generation could not produce well was pitfall criteria - anticipating failure modes 'often demands human intuition and domain expertise'. [load-bearing]\nevidence: full paper (arxiv 2507.17746) confirmed: four design principles (grounded in expert guidance; comprehensive coverage; semantic weighting essential 1.0 / important 0.7 / optional 0.3 / pitfall 0.8 penalty; self-contained per-criterion evaluability); rubrics generated by o3-mini/gpt-4o from reference answers, ~7-20 criteria per prompt; implicit aggregation (judge sees all criteria+weights and emits one score) consistently beat explicit per-criterion binary + weighted-sum, up to 28% relative on healthbench-1k; rubrics helped small judges most and reduced cross-judge variance.\nsource: rubrics as rewards: reinforcement learning beyond verifiable domains (a large data vendor) | https://arxiv.org/html/2507.17746v1 | 2025-07-23 | primary\nimplication: rubric compilation can be automated if anchored to expert reference material - the scarce human input to budget for is reference exemplars and failure-mode/pitfall knowledge, not criterion drafting labor. caveat for autoqa: rar's implicit-aggregation win optimizes reward quality, not auditability; an autoqa that must justify verdicts may still prefer explicit per-criterion verdicts (healthbench/paperbench style) and treat rar as evidence that fixed hand-crafted weights are not sacred.\n\n## rubricbench (mar 2026) measures a stable ~26-27 point preference-accuracy gap between self-generated and human-annotated rubrics across all judge backbones (e.g., deepseek-v3.2 57.8% vs 84.9%; gemini-3-flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement) does not close it - 'rubric formation', not judge reasoning, is the bottleneck. [load-bearing]\nevidence: full html (arxiv 2603.01562) confirmed: 1,147 adversarial pairwise comparisons where surface cues mislead (rejected responses longer/better-formatted/more confident); expert rubrics are 2-10 atomic binary checks derived only from the instruction; llm-generated rubrics over-produce unnecessary (17.9% vs 10.1%) and overly rigid (12.8% vs 7.7%) rules; checklists of 13+ items add noise ('attention displacement'); even with human rubrics accuracy plateaus ~85% because judges treat must-have constraints as soft signals; safety domain worst for self-generated rubrics (~25-30% vs >90% human).\nsource: rubricbench: aligning model-generated rubrics with human standards | https://arxiv.org/html/2603.01562v1 | 2026-03-02 | academic\nimplication: directly bounds the per-project customization contract: instruction-only automated rubric compilation (no expert exemplars/references) leaves a large fidelity gap that more compute cannot buy back - an expert-grounding step (reference answers, adjudicated exemplars, or human rubric sign-off) is structurally required, not optional. also: cap compiled checklist length (~<13 active criteria per judge call), and give hard constraints enforcement semantics (gating) rather than listing them as soft criteria.\n\n## evalgen ('who validates the validators', uist 2024) established criteria drift: evaluation criteria cannot be fully specified a priori because grading outputs itself changes the criteria - 'users need criteria to grade outputs, but grading outputs helps users define criteria' - and some criteria are dependent on the specific outputs observed. [load-bearing]\nevidence: fetched abstract/paper confirmed the catch-22 formulation, the finding that criteria appear dependent on observed outputs rather than definable in advance, and the conclusion that judge-alignment is inherently iterative and subjective; evalgen operationalizes this by generating candidate assertions/judge prompts and selecting those best aligned with accumulated human grades.\nsource: who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences | https://arxiv.org/abs/2404.12272 | 2024-10-13 | academic\nimplication: the rubric compilation pipeline must be a loop, not a compile step: initial rubric -> human grades a sample -> criteria are revised -> re-validate. foundational consequence for the autoqa: budget a per-project calibration phase where criteria are expected to change, version every rubric, and treat early judge-human disagreement partly as rubric-specification signal (rewrite) rather than judge noise (vote harder) - the two-builds fork in design question 5 has an evidence-backed default.\n\n## rift (snorkel ai, apr 2026) provides the first rubric failure-mode taxonomy - 8 modes in 3 categories (reliability: subjective, non-atomic, ungrounded; content validity: misaligned/rigid, missing criteria; consequential validity: hackable, low signal, redundant) - and shows failure-mode density correlates with judge-human misalignment (r=0.162, p=0.0021); synthetic rubrics skew subjective (86.7% vs 52.6% for human-crafted) while human rubrics skew misaligned/rigid (63.2% vs 20.0%); automated llm detection reaches f1 0.925 for subjective but ~0.000 for hackable.\nevidence: full html (arxiv 2604.01375v2) confirmed all numbers: 85 rubrics / 255 expert annotations across 5 sources (human: advancedif, researchrubrics; synthetic: wildchecklists, openrubrics, autorubrics); taxonomy built via grounded theory, mean cohen's kappa 0.64; diagnostics include llm-as-judge classifiers (gpt-5.2, gemini 3 pro), inter-rater reliability across labeler models, alignment to a strong reference, and reward-variance probes.\nsource: rift: a rubric failure mode taxonomy and automated diagnostics (snorkel ai) | https://arxiv.org/html/2604.01375v2 | 2026-04-20 | academic\nimplication: rubric qa/linting is now a named, tooled 2026 practice: run compiled rubrics through a rift-style lint before go-live. two structural lessons: (1) human and llm rubric authors fail in complementary ways, motivating co-authoring rather than either alone; (2) hackability/gameability is the one failure mode automated diagnostics cannot detect - the anti-gaming review of the autoqa's compiled rubrics must be a human red-team task, permanently.\n\n## support vector rubrics (jun 2026) identifies the mechanism behind the llm-vs-human rubric gap - 'self-generated rubrics describe good responses, whereas effective criteria must discriminate between close candidates' - and closes the rubricbench gap from 24.1 to 0.3 points by mining contrastive rubrics from preference pairs (support-pair selection plus adversarial probing of hard negatives).\nevidence: fetched abstract (arxiv 2606.08077, sun et al.) confirmed the objective-mismatch framing, the max-margin method (rubric bank mined from preference data, prompt-conditioned selector, iterative refinement on hard negatives), transfer of the rubric bank across judges without retraining, and competitiveness with dedicated reward models on rewardbench 1/2 and rm-bench.\nsource: support vector rubrics: closing the gap between self-generated and human rubrics | https://arxiv.org/abs/2606.08077 | 2026-06-06 | academic\nimplication: the highest-value compilation artifacts are contrastive: borderline pass/fail adjudication pairs, not descriptions of quality. for the autoqa's exemplar question (5 vs 50 vs 500), this says invest in hard-negative/near-miss pairs per criterion and derive discriminative criteria from them - descriptive 'what good looks like' statements alone are the documented failure pattern.\n\n## paperbench (openai, apr 2025) demonstrates the extreme end of decomposition: 8,316 individually-graded binary leaf criteria across 20 papers, sibling-relative manual weights, rubrics co-developed with each paper's original authors over multiple weeks per rubric; an o3-mini judge then grades leaves at f1 = 0.83 vs expert human judgments for ~$66/paper versus ~12 hours of expert grading time.\nevidence: full paper (arxiv 2504.01848) confirmed: each leaf is pass/fail; parent scores are weighted averages propagating to a root score; weights reflect importance relative to siblings, not difficulty; judgeeval is a separate benchmark for judges (o1 scored 0.84 but ~$830/paper - o3-mini chosen as cost-effective); human expert grading estimated at 'tens of hours' per paper.\nsource: paperbench: evaluating ai's ability to replicate ai research (openai) | https://arxiv.org/html/2504.01848v2 | 2025-04-02 | primary\nimplication: atomic binary decomposition of genuinely expert judgments is feasible and auto-gradable at usable-but-imperfect agreement (f1 0.83), with ~100x cost reduction over expert grading - but rubric authoring cost is weeks of joint expert time per complex artifact. for autoqa: fine decomposition pays at grading time and the judge gets its own meta-benchmark (judgeeval pattern), but the compilation budget, not the judging budget, is the binding constraint per project.\n\n## openai's official grader documentation prescribes that every model grader be validated on its own eval built from expert-graded answers (grader score ordering must reproduce expert ranking), and names the canonical reward-hacking detection signature: the graded model scores well on the model-grader eval while doing poorly on expert human evaluations.\nevidence: fetched live doc (developers.openai.com/api/docs/guides/graders, accessed 2026-07-14): grader taxonomy = string_check, text_similarity, score_model (llm judge with numeric range), python (sandboxed), multigrader (formula-combined sub-graders, rft only); guidance: iterate grader prompts, use few-shot examples of good/fair/poor answers, prefer smooth over binary scores, add edge cases to the grader eval over time, 'guard against reward hacking'. page carries a deprecation note for graders within the evals/fine-tuning workflows they support.\nsource: openai api documentation: graders | https://developers.openai.com/api/docs/guides/graders | 2026-07-14 | primary\nimplication: lab-published operational doctrine matching the autoqa's needs: (1) mixed grader stacks - deterministic checks composed with llm judges via explicit formulas - are the production pattern for the objective/subjective split; (2) a standing expert-audit channel is the designed-in hack detector (grader-vs-expert divergence), i.e., the independent expert-audit sampling channel is a permanent component in the reference architecture, not a bootstrap phase.\n\n## a cluster of 2026 papers (rubric-arm feb 2026; evorubrics, arbor, amaris jun 2026) converges on the position that static rubrics are an exploitable reward specification under optimization pressure and must be adapted/co-evolved during training, with rubric-arm jointly optimizing a rubric generator and judge via alternating rl from preference feedback.\nevidence: rubric-arm abstract opened and confirmed (prior approaches 'rely on static rubrics or disjoint training pipelines, which limits their adaptability'; alternating optimization with variance-reduction analysis; sota on judge benchmarks and downstream policy alignment). evorubrics/arbor/amaris confirmed at title/abstract level via search only - trend triangulated across 5+ 2026 arxiv titles, not deep-read.\nsource: rubric-arm: alternating reinforcement learning for rubric-based reward modeling (plus 2026 co-evolution cluster) | https://arxiv.org/abs/2602.01511 | 2026-02-02 | academic\nimplication: attempters paid per accepted item exert the same optimization pressure as an rl policy: the 2026 consensus is that a frozen compiled rubric will be goodharted, so rubric revision must be a designed, continuous loop (with regression gates), not an exceptional maintenance event. this directly supports treating criteria drift response and anti-gaming rubric rotation as foundational architecture.\n\n## autorubric (feb 2026, rao & callison-burch) consolidates scattered rubric-evaluation techniques into one open-source framework with opinionated defaults - binary/ordinal/nominal criterion types, judge ensembles, few-shot calibration, bias mitigations, and psychometric reliability metrics - reporting 87% binary accuracy with moderate-to-substantial kappa on charm-100, and showing per-criterion explanations double as improvement signals (peer-review agent 0.47 -> 0.85, beating a 0.82 expert-curated baseline).\nevidence: fetched abstract (arxiv 2603.00077, v2 2026-04-03, 52 pages) confirmed the unification claim ('scattered across papers with inconsistent terminology and partial implementations'), the three validation benchmarks (ricechem 80% with 5-shot calibration; researcherbench 931 criteria cross-judge agreement; charm-100), and rl-reward use (+0.039 advancedif, wilcoxon p=0.032, positive ifeval transfer).\nsource: autorubric: unifying rubric-based llm evaluation | https://arxiv.org/abs/2603.00077 | 2026-02-13 | academic\nimplication: by early 2026 rubric-based judging has a consolidated reference implementation whose defaults (ensembles, few-shot calibration on human-graded examples, psychometric reliability reporting per criterion type) can be adopted wholesale rather than reinvented; and the same per-criterion explanation trace that justifies a verdict is demonstrated to work as constructive improvement feedback - verdict machinery and feedback generation can share one artifact.\n\n## langchain's align evals (langsmith, jul 2025) productized judge validation as a standard workflow: humans grade a representative golden set per criterion, an 'alignment score' measures judge-vs-human match, unaligned cases are surfaced for prompt iteration, and each judge-prompt version is compared against a saved baseline alignment score.\nevidence: fetched launch blog confirmed the four-step workflow (select criteria; select representative good+bad examples; human-grade expected scores; iterate evaluator prompt against alignment score), the stated problem ('our evaluation scores don't match what we'd expect a human on our team to say'), inspiration from eugene yan's aligneval, and roadmap items (alignment tracking over time, automatic prompt optimization).\nsource: introducing align evals: streamlining llm application evaluation (langchain) | https://www.langchain.com/blog/introducing-align-evals | 2025-07-29 | practitioner\nimplication: the industry-default minimum-viable gold set practice: small per-criterion human-graded golden sets with versioned alignment baselines, refreshed as unaligned cases surface - evidence that per-criterion (not per-item-holistic) alignment scoring with regression baselines is the practical validation loop the autoqa should ship with from day one.\n\n## tools & artifacts\n- openai/simple-evals healthbench_meta_eval.py: reference implementation of healthbench's grader meta-evaluation (grader vs physician grades, macro f1 per criterion) | https://github.com/openai/simple-evals/blob/main/healthbench_meta_eval.py | directly reusable pattern for validating the autoqa judge against expert-adjudicated labels per criterion - the meta-eval harness the foundation needs, already open-sourced\n- autorubric: open-source framework (rao & callison-burch, feb 2026) unifying rubric-based llm evaluation: binary/ordinal/nominal criteria, judge ensembles, few-shot calibration, bias mitigations, psychometric reliability metrics | https://arxiv.org/abs/2603.00077 | closest existing embodiment of the autoqa's judging core; its opinionated defaults are a checklist of what a 2026-credible rubric judge must include\n- rift taxonomy + diagnostics: snorkel ai's 8-mode rubric failure taxonomy with automated llm lint classifiers and reliability probes (apr 2026) | https://arxiv.org/html/2604.01375v2 | ready-made rubric-qa lint stage for the compilation pipeline; also documents which rubric defects (hackable) automated linting cannot catch\n- rubricbench: 1,147 adversarial pairwise comparisons with expert atomic rubrics for measuring rubric-generation and rubric-execution quality (mar 2026) | https://arxiv.org/html/2603.01562v1 | off-the-shelf stress test for any automated rubric-compilation procedure the autoqa adopts; its surface-bias-controlled pairs also probe judge style bias (design question 5)\n- langsmith align evals: productized judge-alignment workflow: human-graded golden set, per-criterion alignment score, versioned baselines (jul 2025) | https://www.langchain.com/blog/introducing-align-evals | working example of the minimum-viable judge-validation loop and ui pattern for surfacing judge-human disagreements for iteration\n- openai graders api + documentation: composable grader stack (string_check, text_similarity, score_model, python, multigrader with formula aggregation) with published calibration and anti-reward-hacking guidance | https://developers.openai.com/api/docs/guides/graders | lab-published architecture for mixing deterministic checks with llm judges under one aggregation formula - the objective/subjective lane split in api form (note: page carries a deprecation notice)\n- paperbench rubrics + simplejudge (openai/preparedness): 20 author-co-developed hierarchical rubrics (8,316 binary leaves) plus an llm judge and judgeeval, a benchmark for the judge itself | https://arxiv.org/html/2504.01848v2 | the 'benchmark your judge separately' (judgeeval) pattern and the sibling-relative weighting scheme are directly transferable to autoqa verdict aggregation\n\n## disagreements\n- llm-generated vs expert rubrics: scale's rar (jul 2025) reports reference-grounded synthetic rubrics match human-authored ones (0.359 vs 0.348 as rl rewards), while rubricbench (mar 2026) measures a stable ~26-27pt preference-accuracy gap that test-time compute cannot close, and svr (jun 2026) claims the gap closes (24.1 -> 0.3) only by mining discriminative rubrics from preference pairs. the likely reconciling variable is grounding: rar's generator saw expert reference answers, rubricbench's saw only the instruction - but no paper tests all conditions head-to-head, so 'can compilation skip expert grounding' remains unsettled; the safe design assumes it cannot.\n- aggregation rule: rar found implicit aggregation (judge internally weighs all criteria into one score) consistently beats explicit per-criterion binary + weighted-sum for reward quality (up to 28% relative), yet the flagship production benchmarks (healthbench, paperbench) and openai's grader doctrine use explicit independent per-criterion judgments with arithmetic roll-ups for auditability. performance and auditability pull in opposite directions; no source resolves the tradeoff for a qa setting where verdicts must be explained.\n- decomposition granularity: paperbench succeeds with 8,316 atomic binary leaves (judge f1 0.83, each leaf graded separately), while rubricbench finds checklists of 13+ items degrade judgment via 'attention displacement', and rift lists both non-atomic and redundant as failure modes. the apparent resolution - atomicity helps when each criterion gets its own judge call, hurts when one call processes a long checklist - is inferred, not directly tested anywhere.\n- static vs dynamic rubrics: production eval practice (healthbench, paperbench, openai graders) pins rubrics and regression-tests judges against them, treating rubric stability as integrity; the 2026 rl literature (rubric-arm, evorubrics, arbor, amaris) argues static rubrics are inherently exploitable specifications that must co-evolve with the optimizing population. for an autoqa facing adaptive paid attempters, these prescriptions conflict: pin-and-regression-gate vs continuously-adapt.\n- judge-model reasoning strength: healthbench found the non-reasoning gpt-4.1 outperformed reasoning models o3/o4-mini as a rubric grader (mf1 0.709 vs 0.681/0.692), while paperbench selected reasoning model o3-mini (high effort) and found o1 best (f1 0.84). whether rubric grading benefits from reasoning-class judges appears task-dependent and both labs partially attribute results to prompt-tuning conducted with the winning model.\n\n## gaps\n- no published 2025-2026 study applies rubric-based llm autograding to qa of human annotators' work product (annotations/critiques/ratings). all quantitative evidence in this domain grades model outputs or rl rollouts; transfer to human-attempter qa - including how paid humans adapt adversarially versus how policies reward-hack - is an assumption, not a finding.\n- no major training-data vendor (scale, surge, mercor) has published a complete rubric-writing methodology guide. the closest public artifacts are rar's four design principles, openai's grader-doc guidance, healthbench's physician criteria process, and rift's failure taxonomy; appen markets 'rubric design' services but publishes no method detail. vendor-internal rubric-writing guides likely exist but are not public.\n- criteria drift has no 2026 quantitative follow-up: evalgen's finding is qualitative (small-n user study, 2024). no source measures drift rates, drift half-life, or drift-triggered revision thresholds in a production rubric pipeline - the exact parameters design question 9 needs.\n- the 2026 rubric co-evolution cluster (evorubrics, arbor, amaris) was verified only at abstract/title level; their specific numbers were not confirmed by opening full texts. the trend claim rests on rubric-arm (opened) plus five-plus corroborating 2026 titles.\n- openai's graders documentation page carries a deprecation notice for the graders/evals workflows; what replaces them in openai's mid-2026 stack was not tracked down, so the grader taxonomy finding may describe a sunsetting api surface (the design doctrine it encodes remains valid evidence of practice).\n- no source directly measures the minimum number of adjudicated exemplars per criterion before judge-human agreement plateaus (the 5 vs 50 vs 500 question); autorubric's '5-shot calibration -> 80%' on one chemistry dataset is the only datapoint found, and svr implies exemplar quality (hard contrastive pairs) dominates count.\n\n## verifications\n- claim: openai healthbench meta-evaluates its llm grader per-criterion against physician majority grades using macro f1 (met/not-met, class-balanced), and reports the grader as a percentile of individual physicians: gpt-4.1 grader mf1 = 0.709, exceeding the average physician in 5 of 7 themes, while physician-vs-physician mf1 is only 0.569-0.730 with wide individual spread.\n verdict: confirmed | fetched https://arxiv.org/html/2505.08775v1 (marked arxiv:2505.08775v1 [cs.cl] 13 may 2025 - date correct). all claim elements verified against section 8.1 / tables 5-7: (1) meta-evaluation uses macro f1 on binary met/not-met per consensus criterion, explicitly chosen for class imbalance (random baseline mf1=0.50); (2) gpt-4.1 grader mf1 = 0.709, best of graders tested (o4-mini 0.692, o3 0.681, gpt-4.1 mini 0.661, gpt-4.1 nano 0.580); (3) over 60,896 meta-examples across the 34 physician-consensus criteria (avg 1,791/criterion); (4) 'typical physician' baseline computed by scoring each physician against the others the same way as the model, reported as a percentile - gpt-4.1 exceeds the average physician in 5 of 7 themes (below only expertise-tailored communication 0.610 vs 0.618 and health data tasks 0.683 vs 0.730), upper half in 6/7, above 33rd percentile in all; (5) theme-level weighted-average physician-vs-physician mf1 ranges 0.569 (response depth) to 0.730 (health data tasks), with paper noting physician-physician and model-physician agreement both span ~55-75% and wide individual spread; (6) grading prompt and individual criterion phrasings were tuned so intent was unmistakable to the grader (paper also cautions gpt-4.1's top rank may be partly explained by its use during prompt tuning); (7) std across 16 full runs ~0.002 (table 7: 0.0016-0.0029 by model). supersession check: no arxiv v2 exists; 2026 activity is healthbench professional (openai, april 2026) and third-party replications (e.g., open-source graders: kimi-k2 0.693, qwen3-235b 0.681 vs gpt-4.1 0.709) - these extend, not contradict, the original meta-eval numbers. only pedantic nuance: 0.569-0.730 are theme-level weighted averages of physician mf1, not the full range across individual physicians (individual spread is wider) - the claim's own 'wide individual spread' wording already captures this correctly.\n- claim: a large data vendor's rubrics-as-rewards found that llm-synthesized rubrics grounded in expert reference answers performed on par with human-authored rubrics as reward signals (0.359 vs 0.348), that prompt-specific rubrics substantially beat generic ones, and that the one component synthetic generation could not produce well was pitfall criteria - anticipating failure modes 'often demands human intuition and domain expertise'.\n verdict: confirmed | all core assertions verified against the source (arxiv 2507.17746, a large data vendor). (1) table 3: rar-implicit-synthetic 0.359 vs rar-implicit-human 0.348 on healthbench - synthetic rubrics grounded in expert reference answers on par with human-authored; ungrounded synthetic dropped to 0.320, confirming the grounding caveat. (2) prompt-specific vs generic confirmed: \"predefined generic rubrics substantially underperform compared to prompt-specific ones\" (generic ~0.29 vs ~0.37 prompt-specific in ablations; predefined 0.2472 vs rar-implicit 0.3194 on healthbench-1k). (3) pitfall claim confirmed verbatim: paper states synthetically generating effective pitfall criteria \"is inherently difficult\" and anticipating failure modes \"often demands human intuition and domain expertise.\" evidence-summary details also check out: four desiderata (expert-grounded, comprehensive, semantic weighting essential 1.0 / important 0.7 / optional 0.3 / pitfall 0.8 penalty, self-contained), o3-mini/gpt-4o generation, 7-20 criteria per prompt, implicit aggregation beating explicit weighted-sum with up to 28% relative gain over simple-likert on healthbench-1k (v1 figure; the current v2 abstract cites up to 31%), smaller judges benefiting most, and reduced cross-judge score variance (0.250-0.279 vs 0.220-0.254). date correct: v1 submitted 2025-07-23. minor currency notes, not refutations: (a) a v2 exists (2025-10-03) and the paper was accepted at iclr 2026 - citing the v1 html is fine but the 28% headline became 31% in the current version; (b) substantial 2026 follow-on literature (open rubric system arxiv 2602.14069, arbor 2606.03239, scale's own \"chasing the tail\" 2509.21500 and enterprise rar blog) critiques rar's static pointwise-weighted rubrics as having a discriminability ceiling and gaming vulnerability, but none contradicts the specific findings claimed. sources: https://arxiv.org/abs/2507.17746, https://arxiv.org/html/2507.17746v1, https://openreview.net/pdf?id=c1btcrdmt4, https://scale.com/blog/enterprise-rar\n- claim: rubricbench (mar 2026) measures a stable ~26-27 point preference-accuracy gap between self-generated and human-annotated rubrics across all judge backbones (e.g., deepseek-v3.2 57.8% vs 84.9%; gemini-3-flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement) does not close it - 'rubric formation', not judge reasoning, is the bottleneck.\n verdict: partially_confirmed | source check (arxiv 2603.01562v1, fetched): the rubricbench paper exists, is dated 2026-03-02 (v2 2026-03-03), and confirms nearly every specific: 1,147 adversarial pairwise comparisons; deepseek-v3.2 57.8% self-generated vs 84.9% human (vanilla 38.8%); gemini-3-flash 58.0% vs 85.3% (vanilla 56.4%); test-time scaling fails (rub@4->rub@32 flat or declining, refinement depth declining, while scaling human rubrics helps 75.4->85.3); unnecessary-rule 17.9% vs 10.1% and rigid-rule 12.8% vs 7.7%; 13+-item checklists / 'attention displacement' with >70% hallucination rates; ~85% human-rubric plateau; safety ~25-30% self-generated vs >90% human. two corrections: (1) the gap across all backbones is ~22-28 points, not a tight '~26-27' - 26-27 fits only the two cited models; (2) the 'rubric formation is the bottleneck' framing is the paper's interpretation and, more importantly, has been operationally superseded in part: 'support vector rubrics' (arxiv 2606.08077, june 2026, pku/ustc) closes the gap on rubricbench to 0.3 points (82.8 vs 83.1 human-oracle, vs 59.0 self-generated with same gpt-oss-120b judge) via max-margin contrastive rubric-bank learning over preference data - so the gap is stable under naive self-generation and test-time compute, but not under trained discriminative rubric construction. safety remains the residual weak domain even for svr (83.8 vs 92.5). claim is accurate as a description of the march paper; the 'does not close' generalization no longer holds as of june 2026.\n corrected: rubricbench (arxiv 2603.01562, mar 2026) measures a stable ~22-28 point preference-accuracy gap between self-generated and human-annotated rubrics across judge backbones (deepseek-v3.2 57.8% vs 84.9%; gemini-3-flash 58.0% vs 85.3%), and scaling test-time compute (more sampled rubrics, refinement depth) does not close it - the paper attributes the bottleneck to rubric formation, not judge reasoning. however, follow-up work (support vector rubrics, arxiv 2606.08077, jun 2026) closes the gap to ~0.3 points on rubricbench via max-margin rubric-bank learning from preference data, showing the gap is specific to naive self-generation rather than fundamental.", "12": "domain: industry-practice\n\n## a major annotation platform/a major annotation platform's production qa stack is double-blind independent labeling, automated discrepancy detection, expert adjudication, and rework, with a fine-tuned-llm similarity judge that only flags below-threshold responses for human review (never autonomously fails them), krippendorff's alpha as the published reliability bar (>=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor), and a multidimensional per-worker 'trust score' that routes complex tasks to high-trust raters. [load-bearing]\nevidence: fetched and read the full a major annotation platform 'inside the data factory' engineering post. it details the 4-step double-entry review flow, llm-assisted qc (fine-tune on expert-verified responses -> similarity score -> flag sub-threshold for human review), explicit krippendorff alpha thresholds preferred over cohen/fleiss kappa, per-task metric menu (kendall's tau/elo for rankings, f1/matthews for binary), trust-score routing, 3% workforce acceptance rate, sla-backed pay-only-for-quality guarantee, and admitted failure modes: for generative tasks 'there isn't a single clear-cut right answer' so accuracy is relative to the client's worldview, and low precision is attributed to poor instructions/edge-case coverage as often as to workers.\nsource: inside the data factory: how a major annotation platform produces the highest quality data at scale | https://a major annotation platform.com/blog/inside-the-data-factory-how-a major annotation platform-produces-the-highest-quality-data-at-scale | c. 2024, verified live 2026-07-14 | primary\nimplication: the most detailed public vendor qa architecture matches our hybrid design: llm-qc as a triage prefilter feeding one human touch, not an autonomous verdict-writer. adopt chance-corrected alpha (>=0.80 bar) as the industry-legible reliability floor, trust-score routing for reviewer allocation, and note the vendor itself concedes the ground-truth construct for generative work is client-instruction satisfaction, not objective truth (taxonomy q1, q2, q6).\n\n## An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only f1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting f1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice. [load-bearing]\nevidence: fetched the vendor's full research post (2026-05-27). The banking meta-eval is explicitly a meta-eval of verifiers, not agents; text-only 'autorubric' judges fail because evidence lives in spreadsheet formulas/tool state they cannot see; snapshot serialization loses formula references; the vendor's agent-judge (open-sourced, openhands sdk) opens files and follows references at inference time. Cheapest agent-judge config (~$42) beat the best baseline (~$422) by ~3 f1 at one-tenth cost; within one model family the architecture gap was 9.5 f1. they prescribe measuring every verifier against adjudicated expert labels before trusting it to score work or produce rewards.\nsource: your verifier is probably the bottleneck. we built one that isn't. (the vendor's agent-judge verifier and banking verifier meta-eval) | [link withheld] | 2026-05-27 | primary\nimplication: direct evidence for three taxonomy questions: (q1) meta-evaluation must use adjudicated expert labels, and a ~89.5% human agreement floor is achievable with dual-coding + adjudication; (q5) the judge-vs-adjudicated-human ceiling is wildly domain-dependent (f1 0.63 vs 0.95), so per-axis judge-autonomous/assisted/human-only lanes are mandatory, not optional; (q3) evidence closure is the binding constraint - the judge must see the same evidence surface the attempter's claims are about, or it verifies against a flattened proxy.\n\n## the 2026 eval-tooling vendors (langsmith, braintrust, galileo) have converged on one judge-calibration pattern - small human-labeled reference sets (langsmith: start at ~20 balanced examples; braintrust: 50-100 per scoring dimension) with an 'alignment score' defined as raw percent agreement with the human expert, iterate the judge prompt against it, and route low-confidence or judge-disagreement items to humans - and none of them ships chance-corrected statistics. [load-bearing]\nevidence: fetched langsmith's official 'improve llm-as-judge evaluators using human feedback' doc: annotation queue -> reference dataset (>=20 examples, balanced 0/1 labels) -> evaluator playground -> alignment score = 'percentage of examples where the evaluator's judgment matches that of the human expert' -> cluster misalignments into failure modes and encode as prompt instructions. fetched braintrust's 2026-04-03 article: 3-tier stack (deterministic checks -> llm judge -> human), 50-100 labels per dimension before trusting a judge, multi-judge disagreement routed to humans, worst failure mode named as 'silent overconfidence' (judge always returns a score even without adequate context), and explicit warning that human review is not automatically gold standard (fatigue, vague rubrics can be worse than a decent judge). galileo's clhf (2025-02-10 blog) auto-adapts metrics from expert corrections.\nsource: langsmith: improve llm-as-judge evaluators using human feedback (+ braintrust: llm-as-a-judge vs human-in-the-loop, 2026-04-03) | https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback | retrieved 2026-07-14 (braintrust article 2026-04-03) | primary\nimplication: our foundation can adopt the proven mechanics (annotation-queue-to-reference-set loop, failure-mode clustering into rubric amendments, disagreement routing) but must not copy the metric: the entire tooling industry standardized on raw percent agreement over tiny unbalanced sets - exactly what taxonomy q1 proposes banning. differentiation and validity headroom lie in chance-corrected stats, per-class recall at fixed prevalence, and minimum reference-set designs; also budget expectations: tens of labels per axis is the industry's plateau assumption, not hundreds.\n\n## at a large data vendor's outlier/remotasks (bulba experts program for google, march 2023-april 2024 per internal documents), spam - defined internally as 'writing gibberish, writing incorrect information, gpt-generated thought processes' - routinely got paid because per-item detection could not keep up; defenses degraded to zerogpt screening, crude country-level bans (egypt, pakistan, kenya, venezuela), and were circumvented by vpns, while a black market sells verified annotator accounts (outlier, crowdgen, prolific). [load-bearing]\nevidence: opened futurism's 2025-06 piece quoting inc.'s internal-document exclusive verbatim: spammers 'could get away with just totally submitting garbage and there weren't enough people to track them down'; an allocations team once 'dumped 800 spammers' into one queue; supervisors advised to vet entries with zerogpt; scale's spokesperson disputed the story claiming safeguards existed. algorithmwatch separately documented account-resale markets. prior research (cited in the same reporting thread) estimated 33-46% of mturk crowdworkers used llms on a text task. community reports (indeed/glassdoor/trustpilot 2025) confirm the attempter->reviewer->senior-reviewer pyramid with timed capped pay and opaque removals.\nsource: futurism: the ai company zuckerberg just poured $14 billion into... (quoting inc. internal-documents exclusive by sam blum) | https://futurism.com/scale-ai-zuckerberg-incompetence | 2025-06 (documents cover 2023-03 to 2024-04) | practitioner\nimplication: direct answer to taxonomy q8: at the largest vendor, per-item content-based spam classification lost to spam economics; identity/provenance is also compromised (account resale, vpns). the foundation should treat wholesale-llm/spam defense as economics-based gating (rate limits, work-history consistency, randomized deep audits, payment holds) with content detection only as a weak auxiliary signal - and assume attempter identity is an adversarial variable.\n\n## An AI data vendor acquired a data-quality startup (announced 2026-01-28) specifically for algorithms that 'flag incorrect data without a second human reviewer,' after competing bids from other data-labeling companies - signaling that algorithmic second-pass qa replacing the human review layer is now an acquisition-grade core capability for human-data vendors.\nevidence: techcrunch (2026-01-28), the startup's CEO letter, and the vendor's own announcement triangulate: 9 staff of the startup including all three mit-phd co-founders join the vendor's research team; rationale quotes name confident learning, data-centric ai, and llm evaluation as the acquired capabilities; an AI data vendor serves 8 top labs at ~$300m annualized run-rate (end 2025). The startup's core product audited human-labeler output quality algorithmically.\nsource: techcrunch: Report on an AI data vendor's acquisition of a data-quality startup | [link withheld] | 2026-01-28 | practitioner\nimplication: the market has already voted for our thesis: augment/replace the human qa layer over the mandatory human attempter layer. confident-learning-style label-noise detection (statistical, model-based, no second human) is a complementary channel to llm-judge qa and should be considered in the foundation's filtering stage.\n\n## surge ai (~$1.2b revenue 2024, ~50k contractors, bootstrapped) runs qa as real-time dashboards over gold-standard accuracy, inter-annotator agreement, and per-worker trust ratings, with flagged labels automatically reassigned to other annotators; its published advancedif verifier work with meta reports f1 0.728 versus human judgments and +13% rl gains from rubric-based rewards.\nevidence: fetched sacra's surge ai company report (updated 2026-04-21; figures are sacra estimates). qa description: gold-standard accuracy + iaa + per-worker trust ratings on live dashboards; automated reassignment of low-quality labels; premium per-minute pay (~$0.30-0.40/min) as a quality incentive; vetting = domain tests + background checks + ongoing performance evaluation. hemingway-bench used 5,000+ blind pairwise expert comparisons. risks noted: ~12-customer concentration, ca misclassification suit alleging unpaid training time and tight task timers.\nsource: sacra: surge ai revenue, funding & news | https://sacra.com/c/surge-ai | 2026-04-21 | practitioner\nimplication: the premium-quality market leader's qa is continuous gold-task injection + iaa + per-worker trust telemetry with automatic reassignment - i.e., population-level statistical qc, not per-item semantic review. their own verifier tops out at f1 0.728 vs humans on instruction-following, a second independent datapoint (with an AI data vendor's 0.63-0.66) that ~0.65-0.75 f1 is the current judge ceiling on expert-graded open-ended work (taxonomy q5).\n\n## micro1 (~$300m annualized revenue by 2026-04) embeds real-time qa verification layers directly in its annotation interface and operates a closed rubric-feedback loop: labs submit updated rubrics and corner-case prompts, which are routed to the highest-performing experts and used to retrain its vetting algorithms.\nevidence: sacra company report on micro1 (2026) describes the embedded qa layers, lab-facing dashboards with daily output and pass/fail rates, and the rubric-update routing loop; forbes (2026-02-03) interview with ceo ansari frames evaluation as a permanent top-level function ('how well does it work, and what are the thresholds to go into production'). screening is via ai interviewer (zara) at thousands of interviews/day; a major annotation platform and mercor use the same ai-interview screening pattern (a major annotation platform: 3% acceptance, 4-stage vetting).\nsource: sacra: micro1 revenue, valuation & funding (+ forbes, 2026-02-03) | https://sacra.com/c/micro1 | 2026 (forbes 2026-02-03) | practitioner\nimplication: the rubric-amendment loop in taxonomy q4/q6 is already production practice: client rubric updates are first-class config artifacts routed to top performers, i.e., upstream compounding work is prioritized over per-item adjudication. also confirms the industry attacks the expertise-gap root cause at intake (automated ai-interview screening + longitudinal per-worker metrics), not per-item.\n\n## braintrust's published cost arithmetic makes the human-budget fork concrete: an llm judge over 10,000 outputs costs ~$5-15 while a domain expert reviewing 500 of them costs ~$800-1,800 (~100x per output), which is why its prescribed architecture is deterministic checks on everything, llm judges on everything, and humans only on flagged/low-confidence/disagreement slices plus random spot-checks of confident passes.\nevidence: fetched braintrust article (2026-04-03) and human-review docs. the hybrid loop is explicit: trace production -> continuous deterministic + judge scoring -> flag low-confidence/scorer-disagreement -> human review -> reviewed findings become new scorers/rubric refinements -> ci regression gates. they name the biggest practitioner mistake as treating judge scores as ground truth without human validation, and note untrained reviewers with vague rubrics can produce labels worse than a decent judge.\nsource: braintrust: llm-as-a-judge vs human-in-the-loop evals: when to use each | https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals | 2026-04-03 | practitioner\nimplication: supports taxonomy q6's routing answer as consensus practice: spend the single human touch on judge-flagged and judge-disagreement items, keep a randomized audit slice of confident passes, and convert every human review into a compiled-config improvement (new scorer/exemplar) rather than a one-off verdict.\n\n## fine-tuned specialist judges have commoditized binary grounding checks: patronus lynx (fine-tuned llama-3-70b) beats gpt-4o on halubench faithfulness detection and lifted one customer's hallucination detection from 0.375 to 0.69, while galileo's luna-2 (arxiv 2026-02-20) runs hundreds of per-metric lora adapters on one small-model backbone at >80x lower cost and >20x lower latency than llm-as-judge at claimed parity accuracy, in production on 100m+ sessions.\nevidence: patronus launch post + siliconangle (2024-07-11) give halubench numbers (lynx 70b +8.3% vs gpt-4o on pubmedqa inaccuracies; 8b +24.5% vs gpt-3.5); patronus evaluators page gives the algomo 0.375->0.69 customer number and gamma's 1,000+ manual eval hours saved; percival extends this to tracing where in a multi-step agent run the failure occurred. fetched luna-2 arxiv abstract (2602.18583): single-token slm evaluation, >80x cost / >20x latency reduction, 100m+ sessions and 100b tokens/month in production, $30m+ claimed annual eval savings. vendor-reported numbers; treat magnitudes, not decimals.\nsource: luna-2: scalable single-token evaluation with small language models (arxiv 2602.18583) + patronus lynx launch materials | https://arxiv.org/abs/2602.18583 | 2026-02-20 (lynx 2024-07-11) | primary\nimplication: for taxonomy q2/q3: the 'compile to binary checks' bucket (groundedness, faithfulness, toxicity, tool-selection) can run on cheap specialized models per axis at near-zero marginal cost, reserving frontier-model judges for entailment/calibration/judgment-laden axes - a two-tier judge economy is standard 2026 practice, not an exotic design.\n\n## deepeval's recommended progression - start with holistic g-eval, then move to dag (a deterministic decision tree whose nodes are narrow llm-judge calls) 'for more control' - is the open-source codification of decomposed judging: verdicts assembled from atomic, individually-checkable steps rather than one holistic score.\nevidence: deepeval docs (metrics-introduction, metrics-dag; dag introduced 2025-02, docs current 2026-07) describe dagmetric as 'deterministic decision trees' for objective/mixed criteria producing deterministic scores, explicitly positioned above g-eval when control matters, including a conversational variant for multi-turn work. confident-ai's 2025-02-09 engineering post explains the motivation: holistic llm scores are too noisy for objective criteria.\nsource: deepeval docs: dag (deep acyclic graph) metric + confident ai engineering post | https://deepeval.com/docs/metrics-dag | 2025-02-09, docs verified 2026-07-10 | practitioner\nimplication: taxonomy q3's decomposition-granularity question has a live open-source reference implementation: per-criterion decision trees with narrow judge nodes and deterministic aggregation. adopt the pattern (and its admission that decomposition is for objective/mixed criteria - judgment-laden axes stay holistic/g-eval-style) rather than inventing the machinery.\n\n## the human reviewer layer at scale vendors is itself unreliable for structural/incentive reasons the autoqa must not inherit: outlier's attempter->reviewer->senior-reviewer pyramid runs on timed, capped per-task pay, opaque promotion/removal, uneven task allocation, and internal competition, per 2025 contributor reports across indeed/glassdoor/trustpilot.\nevidence: aggregated 2025 worker reports: pay only within a capped time window (unpaid overage), removal from projects without stated reason, alleged 300-reviewer queues where ~10 got unlimited tasks, queue managers running webinars/'war rooms' as the calibration mechanism, quality undermined by contributors and reviewers 'pitted against each other'. community-sourced and unverifiable individually, but consistent across platforms and consistent with the inc. documents' picture of overwhelmed review capacity.\nsource: indeed/glassdoor/trustpilot outlier reviewer reports (aggregated) | https://www.indeed.com/cmp/outlier-ai/reviews?fjobtitle=reviewer&fcountry=us&ftext=outlier | 2025 (various) | community\nimplication: reviewer variance (our root causes 1-3) is partly manufactured by incentive design: time-capped pay punishes careful review, and opaque consequences destroy calibration motivation. the autoqa foundation should specify the human-touch interaction's incentive contract (untimed or accuracy-paid adjudication, visible appeal outcomes) as part of the design, not just the routing logic.\n\n## vendor materials repeatedly attribute low inter-annotator precision to instruction/rubric defects rather than annotator failure - a major annotation platform's own remediation playbook for low precision starts with 'poor instructions, weak edge-case coverage' and prescribes guideline revision and calibration sessions before retraining or replacing people.\nevidence: a major annotation platform data-factory post's scenario playbook: low precision -> better training, clearer guidelines, calibration sessions, edge-case examples; high precision + low accuracy -> refine the ground truth itself and check for systematic bias in training materials. kili's industry guide (cited in 2025 coverage) similarly notes an annotator consistently disagreeing on one label category 'is often revealing a guideline gap', and industry estimates put rlhf preference-annotation noise above 20%.\nsource: a major annotation platform data-factory playbook + kili technology annotation quality guide | https://kili-technology.com/blog/data-annotation-guide-how-to-achieve-high-quality-data-in-complex-ai-data-operations | 2024-2025, verified live 2026-07-14 | practitioner\nimplication: industry operating doctrine already treats systematic reviewer disagreement as an instruction-gap signal first (taxonomy q2's third verdict class and q9's rubric-revision trigger): a disagreement-clustered-by-criterion feed to project owners mirrors existing vendor practice and will be legible to them. but no vendor publishes the variance attribution split, so the taxonomy's cheap variance-attribution pilot is still needed - practice assumes criteria-vagueness dominance without measuring it.\n\n## tools & artifacts\n- an AI data vendor's agent-judge: open-sourced reactive agent-as-judge verifier (openhands sdk) that runs in the rollout's own environment, opens files/tools at inference time; paired with the vendor's banking agent benchmark (huggingface) and the banking verifier meta-eval (3,204 expert labels, full release pending). | [link withheld] | reference architecture for evidence-closure-correct judging and for meta-evaluating our judge against adjudicated expert labels before trusting it.\n- langsmith align evaluator + annotation queues: built-in workflow: human-labeled reference set (>=20 balanced examples) -> alignment score (percent match) -> evaluator playground iteration; corrections folded into judge prompt instructions. | https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback | the de facto industry calibration loop to improve on (its percent-agreement metric is the weakness).\n- braintrust human review + loop: span-level human review ui with judge-vs-human scores side by side, flagged-trace-to-test-case loop, ci regression gating, and an eval-workflow agent (loop, 2025-11). | https://www.braintrust.dev/docs/annotate/human-review | model for converting each human touch into permanent compiled-config assets (scorers, datasets) rather than one-off verdicts.\n- patronus lynx + percival: open-weight fine-tuned faithfulness judge (binary grounded/ungrounded + reasoning trace) and an agentic debugger that localizes failures in multi-step traces. | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | cheap specialist model for our 'compile to binary checks' grounding lane; percival pattern for localizing which claim/step failed.\n- galileo luna-2 + clhf: slm backbone with per-metric lora adapter heads (hundreds of metrics on one gpu, >80x cheaper than llm-as-judge; arxiv 2602.18583) plus continuous learning with human feedback that adapts metrics from expert corrections. | https://arxiv.org/abs/2602.18583 | economics template for running many per-axis checks on every item; clhf is prior art for correction-driven judge adaptation.\n- deepeval dag / conversational dag: open-source deterministic decision-tree metrics whose nodes are narrow llm-judge calls; positioned above holistic g-eval when control is needed. | https://deepeval.com/docs/metrics-dag | open reference implementation of decomposed per-criterion judging with deterministic aggregation (taxonomy q3).\n- cleanlab: confident-learning software that flags incorrect human labels algorithmically without a second human reviewer. | [link withheld] | statistical label-noise detection as a judge-independent third qa channel alongside llm judging and human audit.\n- a major annotation platform connect (a major annotation platform): marketplace exposing per-worker longitudinal quality telemetry (assessment scores, interview transcripts, rework rate, efficiency) to clients staffing data teams. | https://a major annotation platform.com/services/a major annotation platform-connect | prior art for per-attempter trust telemetry as a routing and gating input in our foundation.\n\n## disagreements\n- judge-vs-human ceiling: langchain marketing materials claim llm judges reach ~85% alignment with humans, exceeding ~81% human-human agreement (an mt-bench-style chat-preference number), while an AI data vendor's banking verifier meta-eval shows the best 2026 verifiers at f1 0.633-0.664 against practicing-expert labels (humans at 89.5% agreement) and surge's advancedif verifier at f1 0.728. the 'judges already beat humans' claim holds only for casual preference tasks, not expert-graded artifact-heavy work - the ceiling is domain-dependent by ~30 f1 points.\n- agreement statistic: a major annotation platform publishes chance-corrected krippendorff's alpha thresholds as its quality standard, while langsmith/braintrust standardize on raw percent agreement over tiny reference sets. the industry has not converged, and the eval-tooling side uses exactly the statistic our taxonomy proposes banning.\n- is human review the gold standard? An AI data vendor treats dual-coded, adjudicated practicing-expert labels as ground truth for meta-evaluation; braintrust explicitly warns human review is 'not automatically a gold standard' (untrained reviewers, vague rubrics, fatigue can underperform a decent judge). reconcilable - adjudicated expert panels vs ad-hoc reviewers - but the sources take opposite rhetorical stances.\n- vendor qa claims vs observed integrity: a major annotation platform/surge/scale market rigorous multi-layer qa, slas, and 3% acceptance rates, while internal scale documents (via inc./futurism) show spam paid at scale for 11 months, zerogpt as a stopgap, and country-level bans; scale's spokesperson disputes the reporting ('filled with so many inaccuracies'). marketing descriptions of qa pyramids should not be taken as evidence they perform as described.\n- content-based vs economics-based spam defense: patronus/galileo/detection vendors implicitly sell content-based classification as sufficient, while the scale/outlier record and the annotator-account black market (algorithmwatch) show content detection losing to spam economics at scale; an AI data vendor's purchase of a data-quality startup (statistical label auditing) is a third position - detect via label statistics rather than either content classifiers or human re-review.\n\n## gaps\n- no data vendor publishes actual achieved inter-reviewer reliability numbers (e.g., krippendorff alpha values on common item sets) - only target thresholds. the one hard human-agreement number found (an AI data vendor's 89.5%, dual-coded adjudicated subset) is from a benchmark paper, not production qa reporting.\n- no public data anywhere on false-agreement rates (judge and attempter both wrong vs expert ground truth), on per-project gold-set sizes/refresh rates at data vendors, or on judge-model version pinning / golden-set regression gates at data vendors (taxonomy q1, q9). eval-tooling vendors support regression testing generically but no vendor documents pinning policy for annotation-qa judges.\n- no published evidence on whether qa feedback changes attempter behavior longitudinally (repeat-error rates per criterion, revision acceptance) - no vendor reports feedback-efficacy metrics, and no controlled feedback-vs-verdict-only comparison was found (taxonomy q7).\n- a large data vendor's current (2025-2026) production auto-qa internals are undocumented publicly except via leaks and contributor folklore; turing publishes essentially nothing concrete; invisible's public trace is limited to a job post showing ai qa trainers build rubrics/pass-fail criteria and regression suites. mercor's qa is visible only through reviewer-hiring materials emphasizing consistency screening.\n- could not verify a major annotation platform data-factory blog's exact publication date (undated on page; content verified live 2026-07-14). sacra figures for surge/micro1 are analyst estimates, not audited. the inc. internal documents were verified only via futurism's direct quotations (inc. itself 403s).\n- no vendor was found admitting or measuring feedback-loop gaming (attempters overfitting the qa judge's decision boundary) or monoculture drift from judge feedback - taxonomy q8's core dynamic is unstudied in public industry sources.\n\n## verifications\n- claim: a major annotation platform/a major annotation platform's production qa stack is double-blind independent labeling, automated discrepancy detection, expert adjudication, and rework, with a fine-tuned-llm similarity judge that only flags below-threshold responses for human review (never autonomously fails them), krippendorff's alpha as the published reliability bar (>=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor), and a multidimensional per-worker 'trust score' that routes complex tasks to high-trust raters.\n verdict: confirmed | fetched the source directly (https://a major annotation platform.com/blog/inside-the-data-factory-how-a major annotation platform-produces-the-highest-quality-data-at-scale, published september 12, 2024, still live 2026-07-14). every element of the claim checks out against the text: (1) 4-step double-entry flow - \"two independent labelers perform the same task without knowledge of each other's work,\" results automatically compared for discrepancies, expert reviewer makes final determination, rework with feedback; (2) llm judge fine-tuned \"on a dataset of high-quality, expert-verified responses,\" scores similarity, and its described action is \"flagging responses that fall below a certain similarity threshold for human review\" - flag-only, humans handle challenging cases; (3) exact krippendorff alpha table: >=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor, preferred over cohen's/fleiss' kappa for versatility/robustness (2+ raters, missing data, mixed data types); (4) a major annotation platform trust score is explicitly \"multidimensional\" (historical accuracy, consistency, completion rate, complex-task handling) with \"high-trust ai trainers may be prioritized for more critical or complex tasks.\" ancillary evidence items (3% acceptance, pay-only-for-sla-quality, kendall's tau/elo, f1/matthews) also present. supersession search found no newer official a major annotation platform post revising the trust score or qa stack - the sept 2024 post remains the canonical description; 2026 third-party reviews add worker-side complaints (inconsistent review outcomes, unpaid evaluation phases) but do not contradict the claimed process. two wording nuances, neither rising to a correction: the post's own term is \"double-entry\" (the mechanics match \"double-blind\" - independent, mutually unaware labelers); and the post describes the llm judge only as flagging for human review - it does not affirmatively state \"never autonomously fails,\" though nothing in the post says it auto-fails work.\n- claim: An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only f1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting f1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice.\n verdict: confirmed | fetched [link withheld] directly. every element of the claim checks out against the source: (1) dated may 27, 2026; (2) the banking verifier meta-eval is explicitly a meta-eval of verifiers built on 21 tasks from the vendor's banking agent benchmark with 3,204 pass/fail judgments from practicing bankers; 89.5% inter-annotator agreement on a dual-coded subset, with disagreements adjudicated before inclusion; (3) the vendor's agent-judge is a reactive agent-as-judge running in the same rollout environment (shared filesystem, interpreter, mcp tools), open-sourced on the openhands sdk; (4) f1 0.633-0.664 on the banking meta-eval (artifact-heavy) vs f1 0.951 on openclaw, their internal stateful-tools productivity benchmark (200-criterion sample, +7.5 over next-best); (5) architecture (evidence view: text/snapshot/live env; evidence path: fixed vs inference-time) beats model choice - within gpt-5.4 family, agent-judge/nano beats archipelago/gpt-5.4 by 9.5 f1; (6) cheapest agent-judge config (~$42, gpt-5.4 nano) beats best baseline (archipelago/gemini 3 pro, f1 0.604, ~$422) by ~3 f1 at ~one-tenth cost; (7) the post prescribes meta-eval of verifiers against adjudicated expert labels before trusting them for scoring/rewards. supersession check (two searches, july 2026): no newer contradicting work found; the full agent-judge paper and the banking meta-eval dataset release are still pending, so numbers could later be refined, but as of today the post is the authoritative source and the claim matches it. minor nuance only: the 0.951 benchmark is named openclaw and is internal (not public), which the claim's phrasing (\"simpler stateful-tools benchmark\") accurately reflects.\n- claim: the 2026 eval-tooling vendors (langsmith, braintrust, galileo) have converged on one judge-calibration pattern - small human-labeled reference sets (langsmith: start at ~20 balanced examples; braintrust: 50-100 per scoring dimension) with an 'alignment score' defined as raw percent agreement with the human expert, iterate the judge prompt against it, and route low-confidence or judge-disagreement items to humans - and none of them ships chance-corrected statistics.\n verdict: partially_confirmed | verified: (1) langsmith doc (https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback) says exactly what was reported - annotation queue -> reference dataset (~20 examples minimum, balanced 0/1) -> evaluator playground -> alignment score quoted verbatim as \"the percentage of examples where the evaluator's judgment matches that of the human expert\" -> cluster misalignments into failure modes as prompt instructions. no kappa/alpha anywhere; doc is undated. (2) braintrust article (https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals, confirmed 2026-04-03) matches on all six checked points: 3-tier stack, 50-100 labels per scoring dimension, multi-judge disagreement routed to humans, \"silent overconfidence\" named as the most dangerous failure mode, human-review-not-gold-standard warning (reviewer fatigue, vague rubrics), and agreement tracked as raw agreement rate with no chance-corrected statistic. (3) galileo clhf exists and auto-adapts metrics from expert corrections (feedback -> llm-generated few-shot examples appended to the metric prompt, capped at most recent 15, claimed 20-30% accuracy gain).\n\nrefuted/overstated parts: (a) the \"convergence on raw percent agreement\" framing fails for galileo. galileo's own guidance - \"how to calibrate your llm judge with human annotations\" (https://galileo.ai/blog/calibrate-llm-judge-human-annotations, may 15/jul 6 2026, i.e. newer than the cited 2025-02-10 clhf post) - explicitly recommends against raw percent agreement (imbalanced-label example: 90% raw agreement, kappa ~ -0.05) and prescribes cohen's kappa, fleiss' kappa, and krippendorff's alpha with a recalibrate-when-kappa<0.60 trigger. galileo also has a dedicated cohen's kappa explainer (https://galileo.ai/blog/cohens-kappa-metric, 2025-03-12). (b) galileo's clhf is also not the same pattern as langsmith/braintrust - it has no reference set + alignment score loop; it is few-shot auto-improvement, so the three-vendor \"one pattern\" claim overreaches. (c) the narrower claim \"none of them ships chance-corrected statistics\" as an in-product feature survives verification: galileo's kappa content points users to scikit-learn/statsmodels/r for computation, and its product features (autotune, annotations, signals, experiments, clhf) are not described as computing kappa; langsmith and braintrust expose only percent-agreement metrics. but the claim as worded implies the vendors don't even acknowledge chance correction, which is false for galileo. supersession check: newer relevant work exists - arxiv 2606.00093 (rao & callison-burch, 2026-05-25, \"agreement metrics for llm-as-judge evaluation: what to report and why\") argues for reporting kappa alongside accuracy, and langchain published an align evals calibration resource (2026-03-10) still framed around human-correction agreement.\n corrected: langsmith and braintrust have converged on a judge-calibration pattern of small human-labeled reference sets (langsmith: ~20 balanced examples; braintrust: 50-100 per scoring dimension) with alignment measured as raw percent agreement with human experts, prompt iteration against misalignments, and routing of low-confidence/judge-disagreement items to humans; neither ships a chance-corrected statistic. galileo differs: its clhf (2025) turns expert corrections into few-shot prompt examples (max 15) rather than a reference-set alignment score, and its newer 2026 calibration guidance explicitly rejects raw percent agreement in favor of cohen's kappa / fleiss' kappa / krippendorff's alpha (recalibrate below kappa 0.60) - though it directs users to external tools (scikit-learn, r) rather than computing these in-product. so the \"no chance-corrected stats shipped in-product\" observation holds across all three, but there is no three-vendor convergence on raw percent agreement as the alignment definition.", "13": "domain: contrarian\n\n## raw percent-agreement systematically overstates llm-judge ability: in the largest judge meta-evaluation to date (21 judges, 9 providers, ~541,000 judgments, including april-2026 frontier models), cohen's kappa runs 33-41 percentage points below exact-match agreement on mt-bench, judge rankings shift by up to 14 positions across benchmarks, and two production-deployed judges combine test-retest reliability >0.95 with severe position bias >0.10 (a 'consistency-bias paradox'). [load-bearing]\nevidence: norman, rivera & hughes ran 118 runs across mt-bench, judgebench, rewardbench under three protocols (agreement, consistency, bias audit); kappa deflation was universal across the cohort; they distill a 'minimum viable validation protocol'. verified from the arxiv abstract page. corroborated by the neurips 2025 position paper 'neither valid nor reliable? investigating the use of llms as judges' (arxiv 2508.18076, chehbouni et al.), which argues via social-science measurement theory that llj adoption has outpaced scrutiny of validity/reliability and that human-agreement proxying is an unvalidated assumption.\nsource: reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias | https://arxiv.org/abs/2606.19544 | 2026-06-17 | academic\nimplication: directly answers q1's metric-standardization question: ban raw percent-agreement and single 'accuracy vs humans' numbers from all autoqa meta-evaluation; require chance-corrected statistics per benchmark. and for q5: run-to-run consistency must never be reported as evidence of validity - a judge can be perfectly repeatable and severely biased simultaneously, so a separate perturbation/bias audit is a mandatory ship gate independent of the consistency gate.\n\n## across 106 experiments and 370 effect sizes, human-ai combinations performed significantly worse than the best of human or ai alone (hedges g = -0.23, 95% ci -0.39 to -0.07), with losses concentrated in decision-making tasks and specifically when the ai outperforms the human alone - and a 2025 aies study found human reviewers followed severely race-biased ai hiring recommendations ~90% of the time. [load-bearing]\nevidence: vaccaro, almaatouq & malone (nature human behaviour, preregistered meta-analysis; direction, decision-task losses, and ai-better-than-human losses verified from the arxiv abstract; g value corroborated across mit sloan, nature's own summary, and researchgate). uw's 'no thoughts just ai' (aies 2025, doi 10.1609/aies.v8i3.36749, 528 participants, 16 job types) found participants matched biased ai picks even at moderate bias and ~90% at severe bias, vs equal selection rates with no/neutral ai. the 2026 practitioner literature (e.g., tianpan.co hitl rubber-stamp essay, 2026-04-15) reports the same mechanism in production review pipelines.\nsource: when combinations of humans and ai are useful: a systematic review and meta-analysis | https://www.nature.com/articles/s41562-024-02024-1 | 2024-10-28 | academic\nimplication: q6's fork is real and the evidence leans against the default: 'one shallow human touch verifying the ai verdict' is precisely the decision-task, ai-better-than-human configuration where the meta-analysis finds negative synergy and rubber-stamping. the foundation should treat zero-per-item-touch plus randomized blind deep audits (human judges never see the ai verdict before committing their own) as the null design to beat, and any verify-the-ai-verdict interaction must be measured against its own overturn rate to prove the human is adding information.\n\n## there is a proven theoretical ceiling on judge-based validation: when the judge is no more accurate than the model/content being evaluated, no debiasing method using gold labels can cut the required amount of ground-truth data by more than a factor of two, and empirical savings are smaller than the 2x bound. [load-bearing]\nevidence: dorner, nastl & hardt (iclr 2025) prove the bound for methods that combine cheap judge scores with a small gold-label set to correct judge biases such as self-preference; verified from the arxiv abstract page including the central quote 'when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth' by more than half.\nsource: limits to scalable evaluation at the frontier: llm as judge won't beat twice the data | https://arxiv.org/abs/2410.13341 | 2024-10-17 | academic\nimplication: q1's gold-set arithmetic cannot be escaped by judge cleverness: on exactly the axes where the expertise-gap root cause bites (attempter competence near or above judge competence), certifying the autoqa still costs at least half the gold labels a judge-free validation would - so hierarchical/pooled validation across projects and a permanent (not bootstrap) expert-audit channel are structural requirements, and any vendor claim that the judge 'validates itself' at scale should be treated as mathematically impossible in the regime that matters.\n\n## human evaluation criteria are output-dependent and unstable: even when graders define criteria before grading, the act of grading changes their criteria and they retroactively revise earlier grades ('criteria drift'), implying evaluation criteria for llm-output quality cannot be fully determined prior to observing outputs. [load-bearing]\nevidence: shankar, zamfirescu-pereira, hartmann, parameswaran & arawjo (uist 2024, evalgen system study); abstract verified via arxiv crawl: 'we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria'; the authors argue there is reason to believe criteria never fully settle because they adapt to the observed output distribution.\nsource: who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences | https://arxiv.org/abs/2404.12272 | 2024-04-18 | academic\nimplication: falsifies q4's implicit one-shot compilation contract: 'instructions compile once into artifacts, fresh judge hits target agreement with no conversation' is unstable because the project owner's own criteria will drift once they see real attempter submissions and judge verdicts. rubric compilation must be designed as a versioned re-compilation loop with an explicit retroactive re-scoring policy, and owner sign-off must happen on graded real items, not on abstract criteria.\n\n## llm judges favor models whose mistakes resemble their own (a generalization of self-preference, measured by chance-adjusted mistake-overlap capa), and model errors across the industry are becoming more correlated as capabilities improve - undermining the assumption that ai oversight of ai-assisted work catches failures.\nevidence: goel et al., icml 2025; abstract verified via arxiv crawl: 'llm-as-a-judge scores favor models similar to the judge' and 'model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures'. related: 'preference leakage' (arxiv 2502.01534) documents contamination when generator and judge are related.\nsource: great models think alike and this undermines ai oversight | https://arxiv.org/abs/2502.04313 | 2025-02-06 | academic\nimplication: q5: judge model family must differ from the generator being critiqued and from whatever assistant attempters plausibly used - per-project judge routing plus similarity reporting. q1: the false-agreement rate (judge passes an llm-assisted attempter because both share priors) is predicted to rise over time, which converts the independent human expert-audit channel from a bootstrap phase into a permanent structural component.\n\n## llm judges do not follow their own rubrics: on arena-hard auto, the explicit evaluation schema explains under 10% of verdict variance for some judges (unexplained variance >90% for deepseek-r1-32b), and factor correlations above 0.93 across nominally distinct criteria show per-axis scores collapse into a single halo factor.\nevidence: feuer et al. introduce 'schematic adherence' (how much of the verdict the stated rubric explains) and psychometric validity checks; abstract verified via arxiv crawl: 'severe schema incoherence and factor collapse across popular judges'; elo-style aggregation additionally masks genuine ranking uncertainty. code at github.com/penfever/judgment-to-noise.\nsource: when judgment becomes noise: how design failures in llm judge benchmarks silently undermine validity | https://arxiv.org/abs/2509.20293 | 2025-09-24 | contrarian\nimplication: q3/q4: a per-axis verdict sheet can be theater - the judge may emit one holistic impression dressed up as n criterion scores. the foundation should require discriminant-validity and schematic-adherence tests per project config (do axis scores actually vary independently? does the rationale's schema explain the verdict?) before per-criterion outputs are exposed to attempters or used for specialist routing.\n\n## llm judges have low intra-rater reliability - identical items re-scored across runs with identical settings produce inconsistent, 'almost arbitrary in the worst case' ratings - and the obvious fix (temperature-0 determinism) measurably reduces agreement with human judgment; meanwhile the human baseline itself is weak (summeval inter-annotator kappa: 0.492 crowd, 0.413 expert first round, 0.71 only after a second adjudication round).\nevidence: haldar & hockenmaier, findings of emnlp 2025 (pp. 24986-25004); verified by reading the paper pdf (pp. 1-3): contributions are (1) low agreement of llm ratings across runs, (2) disabling sampling hurts human-agreement, (3) phenomenon persists across summac, summeval, mt-bench; the summeval human kappa figures are quoted in their section 3.1.\nsource: rating roulette: self-inconsistency in llm-as-a-judge frameworks | https://aclanthology.org/2025.findings-emnlp.1361.pdf | 2025-11-04 | academic\nimplication: q5: verdict flip rate is intrinsic and cannot be silenced with greedy decoding without paying validity - so the build must budget k-sample voting and treat residual flip clusters as underspecification signal routed to rubric revision. q1: raw human labels sit below conventional reliability thresholds; validation targets must be adjudicated (multi-round) labels, which prices the gold set honestly.\n\n## llm judges give the weakest signal exactly where qa needs them most: they cannot reliably grade responses to questions they cannot answer themselves (poor signal on the hardest items in a benchmark), and in expert domains subject-matter experts agreed with llm-judge picks only 64% (mental health) to 68% (dietetics) of the time, with the judge favoring superficially actionable detail.\nevidence: 'no free labels' (kim et al., arxiv 2503.05061, mar 2025) shows judge quality collapses on the most difficult items and that human-written reference answers improve agreement and reduce self-preference; szymanski et al. (acm iui 2025, doi 10.1145/3708359.3712091) measured the 64/68% sme agreement in pairwise comparisons on domain tasks. both reported consistently across two independent search engines; abstracts inspected.\nsource: no free labels: limitations of llm-as-a-judge without human grounding | https://arxiv.org/abs/2503.05061 | 2025-03-07 | academic\nimplication: q5/q2: the expertise-gap root cause is not automatically closed by an llm judge - the judge's competence ceiling binds hardest on exactly the expert items where human reviewers also fail, so per-axis judge-vs-adjudicated-expert ceilings must be measured before assigning judge-autonomous lanes, and expert-authored reference answers/exemplars are the highest-leverage compile-time artifact (they measurably raise judge validity).\n\n## judge verdicts are gameable through content-independent artifacts: short universal adversarial phrases learned on a surrogate model transfer to unseen judge llms and inflate scores toward the maximum regardless of the assessed text, with absolute scoring far more vulnerable than comparative assessment.\nevidence: raina, liusie & gales, emnlp 2024 (main, pp. 6920+); abstract verified via openalex/acl record: attackers need no access to the deployed judge - surrogate attack then transfer; 'irrespective of the assessed text, maximum scores are predicted'. complementary: 'cheating automatic llm benchmarks' (arxiv 2410.07137) showed null models emitting constant responses achieve top win rates on judged benchmarks.\nsource: is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment | https://aclanthology.org/2024.emnlp-main.427/ | 2024-11-12 | academic\nimplication: q8: pay-motivated attempters do not need to see the judge's rationale to goodhart it - transfer attacks work blind, so filtering rationales is insufficient as the sole defense. day-one requirements: comparative/anchored scoring formats over absolute likert, rotating seeded probes with known verdicts, and evidence-perturbation tests (swap the cited span, verdict must flip) as a standing citation-theater detector.\n\n## industrial-scale annotation qa has already lost to spam economics once: a large data vendor's 'bulba experts' program for google (mar 2023-apr 2024) was flooded with contributors submitting 'writing gibberish, writing incorrect information, gpt-generated thought processes' who were paid anyway because per-item catching was infeasible and banned workers returned via vpn - while academic estimates put llm use at 33-46% of crowdworkers on text tasks.\nevidence: futurism (2025-06-28) reporting inc. magazine's review of internal a large data vendor documents and former queue-manager interviews; article text read in full via crawl ('spammers could get away with just totally submitting garbage and there weren't enough people to track them down'; 'no background checks whatsoever'). prevalence: veselovsky et al. arxiv 2306.07899; neurips 2025 'evaluating llm-contaminated crowdsourcing data without ground truth' proposes peer-prediction mechanisms precisely because text-detector approaches fail on structured annotations.\nsource: the ai company zuckerberg just poured $14 billion into is reportedly a clown show of ludicrous incompetence (inc. internal-document reporting) | https://futurism.com/scale-ai-zuckerberg-incompetence | 2025-06-28 | practitioner\nimplication: q8: the strongest available field postmortem says per-item content classification is the losing side of the cost asymmetry - the foundation should weight identity/provenance gating, rate limits, work-history consistency, peer-prediction-style information scoring, and randomized deep audits over per-item spam detection, and should assume some fraction of 'human' annotations are llm-assisted (making provenance policy per-project, with process telemetry as an explicit data-collection contract).\n\n## annotator disagreement contains recoverable systematic signal, not just noise: nutmeg, a bayesian model separating annotator-competence noise from subpopulation-level systematic disagreement, produces downstream models that significantly outperform both majority-vote aggregation and fully disaggregated training - meaning aggregation to a single 'true' label destroys measurable information.\nevidence: ivey, gauch & jurgens, emnlp 2025 main (pp. 2874-2887); pdf and poster abstract inspected: nutmeg estimates true labels per subpopulation and beats mace/majority-vote at replicating subgroup label distributions on politeness and offensiveness. sits atop the perspectivist nlp literature (nlperspectives workshops 2022-2025) which rejects single-gold-label resolution for subjective tasks.\nsource: nutmeg: separating signal from noise in annotator disagreement | https://aclanthology.org/2025.emnlp-main.144.pdf | 2025-11-04 | academic\nimplication: q2: inter-reviewer disagreement is partly legitimate signal, so an autoqa that maximizes consistency-of-application on judgment-laden axes launders one interpretation into false objectivity and destroys exactly the pluralism the training data may need. the verdict ontology needs a measured 'systematic-disagreement' class (distinct from noise), triggered by subpopulation-conditioned disagreement, exempt from attempter penalty, and fed back as an instruction-gap report.\n\n## ai assistance homogenizes human output at the population level even while raising individual quality: in a nature human behaviour brainstorming study 94% of chatgpt-assisted participants' ideas shared overlapping concepts (nine independently produced the same product name) while human-only ideas were entirely unique, and the 'artificial hivemind' study found different vendors' frontier models converge on near-identical phrasings (~81% average similarity between deepseek-v3 and gpt-4o).\nevidence: meincke, nave & terwiesch (nature human behaviour, 2025; wharton mack institute summary opened and read); jiang et al. (uw/cmu/ai2 'artificial hivemind', reported by the decoder) documents intra-model repetition plus inter-model homogeneity. consistent with doshi & hauser (science advances 2024) and wan et al. 2026 (diverse ai personas partially mitigate homogenization). all are adjacent-domain (ideation/writing), not annotation-qa-feedback studies.\nsource: new in nature: chatgpt decreases idea diversity in brainstorming (meincke, nave & terwiesch, nature human behaviour) | https://mackinstitute.wharton.upenn.edu/2025/new-in-nature-chatgpt-decreases-idea-diversity-in-brainstorming/ | 2025-05 | academic\nimplication: q8's monoculture worry has strong analogical support: a single judge family delivering item-specific stylistic feedback to thousands of attempters is a homogenization pump aimed at the exact diversity the training data exists to capture. the foundation should ban stylistic guidance in feedback by default, keep feedback at principle level, and stand up a population-level output-diversity drift metric from day one - while flagging that direct evidence in the annotation-qa setting does not yet exist.\n\n## tools & artifacts\n- judgment-to-noise: code and dataset for schematic-adherence and psychometric-validity diagnostics of llm judges (feuer et al. 2025) | https://github.com/penfever/judgment-to-noise | directly reusable as a per-project ship gate: measures whether a judge's rubric actually explains its verdicts and whether per-axis scores have discriminant validity, before exposing per-criterion output.\n- capa (chance adjusted probabilistic agreement): metric for judge-model similarity based on chance-corrected overlap in mistakes, with website and code (goel et al., icml 2025) | https://model-similarity.github.io/ | operationalizes per-project judge routing: quantifies how similar a candidate judge is to the generator/assistant models attempters likely used, bounding correlated-failure risk.\n- llm_contamination (peer-prediction detection): code for neurips 2025 'evaluating llm-contaminated crowdsourcing data without ground truth' - peer-prediction mechanisms that score information content of annotations without gold labels | https://github.com/yichiz97/llm_contamination | an economics/information-based alternative to per-item llm-text detection for the spam-filtering layer, designed for structured annotation tasks where text detectors fail.\n- evalgen: mixed-initiative interface for aligning llm-generated evaluators with human grades (shankar et al., uist 2024); the study that identified criteria drift | https://arxiv.org/abs/2404.12272 | reference design for the rubric-compilation workshop: interleaves owner grading with criteria refinement rather than assuming criteria can be authored up front.\n- minimum viable validation protocol: reporting protocol distilled from the 541k-judgment meta-evaluation (norman et al. 2026): chance-corrected agreement + consistency + bias audit as the minimum bar for claiming a judge works | https://arxiv.org/abs/2606.19544 | candidate template for the autoqa's own meta-evaluation reporting standard (q1), replacing single-number accuracy claims.\n- nutmeg: bayesian aggregation model separating annotator-competence noise from systematic subpopulation disagreement (emnlp 2025) | https://aclanthology.org/2025.emnlp-main.144.pdf | machinery for the variance-attribution pilot (q1) and for the 'systematic-disagreement' verdict class (q2): distinguishes items where reviewers disagree from noise vs. from legitimate perspective differences.\n\n## disagreements\n- can llm judges match human evaluators at all? counter-evidence exists: some studies report gpt-4-class judges agreeing with human consensus at rates exceeding individual human raters (chatbot arena-style preference tasks), and one 2026 study reports inter-judge krippendorff's alpha of 0.867 above the 0.80 threshold. the contrarian corpus (kappa deflation, 64-68% sme agreement in expert domains, schema incoherence) does not refute this; the reconciliation is regime-dependence - judges can beat the mean crowd rater on generic preference tasks while falling below adjudicated expert panels on expertise-heavy and subjective axes. design consequence: per-axis ceilings, not a global verdict on judge viability.\n- human-in-the-loop value: vaccaro et al. find human-ai combinations lose on decision tasks when ai outperforms the human, but the same meta-analysis finds gains when the human outperforms the ai - so the one-touch design is not universally bad; it is bad specifically where the judge is stronger than the reviewer, and potentially positive on expert axes where humans still beat judges (per the iui 2025 expert-domain results). the two contrarian findings point in opposite directions depending on axis type.\n- claim decomposition: the decompose-then-verify literature is internally split. proponents (factscore lineage, lrec 2026 japanese decomposition dataset) find atomic claims improve explainability and reduce annotator variability; critics ('does claim decomposition boost or burden fact-checking performance?' openreview; wanner et al. 2024; the acl 2025 dynamic-decomposition paper's own literature review) find decomposition does not consistently improve verification across input lengths and verifier strengths, and that atomicity choice itself changes verdicts (accuracy swings of ~0.12 from decomposition policy alone). neither side has settled the granularity question the autoqa design poses.\n- determinism as a fix for judge inconsistency: common practitioner guidance says run judges at temperature 0 for reproducibility; rating roulette (emnlp 2025) shows disabling sampling measurably reduces agreement with human judgment - reproducibility and validity trade off rather than align.\n- bias against ai-generated content: earlier work suggested llm judges penalize (or favor) ai-generated text; balog, metzler & qin (sigir 2025) report finding no evidence of bias against ai-generated content in their preliminary study, while confirming a significant bias toward llm-based rankers' outputs. the direction and existence of ai-content bias is unsettled; the similarity-based affinity bias (great models think alike) is the better-supported formulation.\n\n## gaps\n- no published longitudinal study of judge-feedback-induced homogenization of a paid annotator population (the monoculture finding rests on analogical evidence from ideation/creative-writing settings, not annotation qa feedback loops).\n- no published measurement of the false-agreement rate (autoqa judge and attempter agreeing while both wrong per expert ground truth) in a production annotation-qa setting; great models think alike predicts it rises with model similarity but does not measure it in this configuration.\n- found no practitioner postmortem of an llm-based autoqa layer for human annotation being goodharted by attempters over feedback cycles - the a large data vendor postmortem is from the pre-llm-judge qa era, and adversarial-judge evidence comes from benchmark/exam settings; the specific 'scores rise on autoqa, flat on held-out human-graded goldens' divergence experiment appears not to have been run or published anywhere.\n- little direct evidence on positive-claim (praise) verification: no seeded-set study measuring whether judges pass plausible-but-ungrounded praise at higher rates than they fail grounded praise, per praise type - the leniency/sycophancy literature is about response evaluation generally, not verification of an evaluator's positive statements.\n- rating roulette's per-benchmark intra-rater icc/krippendorff numbers were not extracted (only pages 1-3 read: abstract-level claims plus the summeval human-kappa figures verified); the synthesis engine should not cite specific llm intra-rater coefficients from this report.\n- the 'reliability without validity' (2026) preprint is 0-citation and un-peer-reviewed as of 2026-07-14; its headline numbers are consistent with the older peer-reviewed literature (kappa deflation, position bias) but the specific 33-41pp figure rests on one team's protocol.\n\n## verifications\n- claim: raw percent-agreement systematically overstates llm-judge ability: in the largest judge meta-evaluation to date (21 judges, 9 providers, ~541,000 judgments, including april-2026 frontier models), cohen's kappa runs 33-41 percentage points below exact-match agreement on mt-bench, judge rankings shift by up to 14 positions across benchmarks, and two production-deployed judges combine test-retest reliability >0.95 with severe position bias >0.10 (a 'consistency-bias paradox').\n verdict: confirmed | verified directly against the arxiv abstract page (https://arxiv.org/abs/2606.19544, v1 submitted 2026-06-17, only version). every quantitative element matches: 21 judge models from 9 providers, 118 runs, ~541,000 judgments across mt-bench/judgebench/rewardbench under three protocols (agreement, consistency, bias audit); exact-match vs cohen's kappa gap of 33-41 pp on mt-bench; judge rankings shifting up to 14 positions across benchmarks; two production-deployed judges with test-retest reliability >0.95 and position bias >0.10, framed as a 'consistency-bias paradox'; findings 'consistent across the full cohort, including the april 2026 frontier'; a 'minimum viable validation protocol' is distilled. authors are justin d. norman, michael u. rivera, d. alex hughes as claimed. the corroborating paper (arxiv 2508.18076, chehbouni et al.) is confirmed as a neurips 2025 poster (https://neurips.cc/virtual/2025/poster/121914) making exactly the measurement-theory validity/reliability argument described. supersession check (exa, freshness=month): no later paper superseding or contradicting it found as of 2026-07-14; only secondary coverage (e.g., dev community 2026-07-01) restating its findings, and one adjacent study (arxiv 2606.13685) on run-to-run instability that complements rather than contradicts. two caveats, neither rising to a correction: (a) 'largest judge meta-evaluation to date' is the authors' own self-characterization ('largest systematic evaluation of the paradigm so far'), not independently established; (b) verification is abstract-level - i did not audit the paper body. note: two websearch calls were blocked by stochastic model safeguards; exa search substituted successfully.\n- claim: across 106 experiments and 370 effect sizes, human-ai combinations performed significantly worse than the best of human or ai alone (hedges g = -0.23, 95% ci -0.39 to -0.07), with losses concentrated in decision-making tasks and specifically when the ai outperforms the human alone - and a 2025 aies study found human reviewers followed severely race-biased ai hiring recommendations ~90% of the time.\n verdict: confirmed | all load-bearing facts verified against primary sources. (1) meta-analysis: full text of vaccaro, almaatouq & malone (nature human behaviour, published 2024-10-28; verified via arxiv:2405.06087v2 accepted version) states verbatim \"370 unique effect sizes from 106 different experiments\" and \"hedges' g = -0.23, 95% confidence interval -0.39 to -0.07\" for human-ai combos vs best of human or ai alone; decision-task losses and losses-when-ai-outperforms-human both confirmed. date correct. (2) aies 2025: \"no thoughts just ai\" (wilson, sim, gueorguieva & caliskan, uw), aies proceedings 8(3):2692-2704, article 36749 (doi 10.1609/aies.v8i3.36749 matches), n=528, 16 occupations; equal selection with no/neutral ai, and participants favored ai-preferred candidates \"up to 90% of the time\" with severely biased ai - claim's \"~90%\" is a fair paraphrase (study says \"up to 90%\", with a simulated llm, not a deployed system). (3) supersession: only later work found is a 2026 open mind/nhb-collaboration reproduction of the meta-analysis by brodeur et al. (osf.io/u6gea) - a reproduction effort, not a refutation; no retraction, correction, or contradicting update located. the tianpan.co practitioner-essay corroboration was not independently checked but is non-load-bearing.\n- claim: there is a proven theoretical ceiling on judge-based validation: when the judge is no more accurate than the model/content being evaluated, no debiasing method using gold labels can cut the required amount of ground-truth data by more than a factor of two, and empirical savings are smaller than the 2x bound.\n verdict: confirmed | source verified: dorner, nastl & hardt, \"limits to scalable evaluation at the frontier: llm as judge won't beat twice the data,\" arxiv:2410.13341, v1 submitted 2024-10-17 (date matches), accepted as an iclr 2025 oral (venue matches, slightly stronger than claimed). the abstract states verbatim that \"when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half,\" and that empirical sample-size savings are \"even more modest\" than the 2x bound - matching all three parts of the claim (theoretical 2x ceiling, the accuracy condition, and smaller empirical savings). search for 2025-2026 follow-up work found no refutation or supersession; the authors' later papers (e.g., roc-n-reroll, iclr 2026) cover different topics. minor scoping caveat only: the bound is proven within the paper's statistical framework for debiasing methods combining judge scores with gold labels, and it does not apply when the judge is more accurate than the evaluated model - the claim already states this condition correctly.", "14": "domain: frontier-2026\n\n## when an llm auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%. [load-bearing]\nevidence: controlled meta-eval with mechanically known-correct audit verdicts: items rendered clean or with exactly one defect injected into instruction/reference/evaluator, so auditor accuracy is scored against the planted defect. reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. directly falsifies the assumption that a judge 'verifying' a claim is doing evidence-anchored checking rather than comparing against its own solution prior.\nsource: auditing by re-solving: llm benchmark auditors trust their own answer over the reference (acl arr 2026 may submission) | https://openreview.net/forum?id=mzh9d3doon | 2026-06 | academic\nimplication: answers taxonomy q3 (evidence closure) with hard 2026 evidence: the judge will verify against its own priors whenever the check exceeds its own compute-the-answer ability, and its false-fail rate explodes exactly there. the autoqa must (a) require span-quoting evidence-anchored verdicts, (b) route checks by whether they are read-checkable vs recompute-checkable, and (c) treat judge confidence as untrustworthy on recompute-class checks -- capability-triage the check type, not just the axis type.\n\n## An AI data vendor's agent-judge verifier (may 27, 2026) shows verifier architecture beats verifier model: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on f1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models. [load-bearing]\nevidence: full primary post opened and read. The banking meta-eval fixes completed rollouts and rubrics and compares automated verifiers against practicing-banker labels; baselines were autorubric (text-only rubric judge), archipelago/apex snapshot grading, and agent-as-a-judge. key framing: 'verifiability is a relationship between a criterion and the verifier available to check it' -- a criterion invisible to a text-only judge is effectively ungraded, and weak verifiers push benchmark/rubric design toward only what they can reach. The vendor's agent-judge verifier code is open-sourced; the banking meta-eval dataset release planned.\nsource: an AI data vendor's research post on verifier architecture | [link withheld] | 2026-05-27 | primary\nimplication: for q5/q6 and overall architecture: give the autoqa judge the same evidence surface the attempter had (source documents, cited spans, artifacts, tool state) and let it decide what to open, rather than judging from the attempter's prose alone -- evidence access is a bigger lever than model choice, and it is also ~10x cheaper than brute-forcing with a stronger model. also a direct template for our meta-eval: fixed items + expert labels + f1 over candidate judge configs, with expert iaa (~89.5% here) reported as the ceiling.\n\n## telling an llm judge the downstream consequences of its verdict ('low scores cause retraining/decommissioning') systematically softens verdicts -- peak verdict shift of -9.8 percentage points, a 30% relative drop in unsafe-content detection across 18,240 controlled judgments (content held strictly constant, 3 judge models) -- and the judge's chain-of-thought contains zero explicit acknowledgment of the consequence framing it is acting on (err_j = 0.000). [load-bearing]\nevidence: abstract and full submission metadata read on openreview (ctb@icml 2026 workshop, 8-page paper). 1,520 responses spanning three safety/quality benchmarks, four response categories from clearly-safe to overtly-harmful; only a one-sentence consequence framing in the system prompt varied. the bias is implicit and undetectable by inspecting reasoning traces.\nsource: context over content: exposing evaluation faking in automated judges (ctb@icml 2026) | https://openreview.net/forum?id=xi2hnjgnzx | 2026-05-25 | academic\nimplication: two foundational rules for q5/q7: (1) the judge prompt must be stakes-sterile -- never tell the judge that a fail penalizes the attempter's pay/standing, which our qa-of-humans setting does by construction, so the compilation contract must strip consequence language from project instructions before they reach the judge; (2) the q7 explainability contract cannot rest on cot audit -- a clean-looking reasoning chain does not certify an unbiased verdict, so bias must be measured behaviorally (seeded perturbation probes), not read off rationales.\n\n## hypac (jan 30, 2026) gives the one-human-touch routing problem a formal solution: calibrate two uncertainty thresholds (via importance sampling + upper confidence bounds) that partition items into three regions routed to cheap llm / expensive reasoning model / human, achieving a distribution-free pac guarantee on annotation error while cutting annotation cost 78.51% in experiments. [load-bearing]\nevidence: abstract opened and verified on arxiv. the guarantee is distribution-free and pre-trained-model-free; routing is by calibrated uncertainty, not fixed rules. this is the first 2026-vintage method that turns 'which items get the human?' from a launch-time heuristic into a statistically certified, closed-loop calibration -- exactly the q6 fork (fixed routing vs closed-loop on observed error).\nsource: hypac: cost-efficient llms-human hybrid annotation with pac error guarantees (arxiv 2602.02550) | https://arxiv.org/abs/2602.02550 | 2026-01-30 | academic\nimplication: for q6: adopt threshold-calibrated three-lane routing (auto-verdict / stronger-judge / human) with an explicit error budget as the foundational routing primitive, rather than a fixed one-touch-per-item rule; the human budget becomes an output of the target error rate. caveat from the auditing-by-re-solving finding: uncertainty-based routing must be validated per check-type, since judge confidence is miscalibrated precisely on recompute-class checks.\n\n## per-criterion judge reliability ceilings differ sharply and can be measured per-item with conformal prediction: on summeval across four judges, relevance is judged most reliably (avg conformal set size ~3.0 of 5), coherence moderate (~3.9), fluency and consistency unreliable (~4.9); aggregate pairwise transitivity violations look small (0.8-4.1%) but 33-67% of documents contain at least one intransitive preference cycle, and conformal set width correlates with reliability at r_s=+0.576 (n=1,918).\nevidence: paper opened via arxiv abstract page (submitted 2026-04-16, under review); all numbers confirmed. criterion choice matters more than judge choice; set width behaves as a per-instance difficulty/abstention signal (cross-judge width agreement 0.32-0.38). code, prompts, and cached results released.\nsource: diagnosing llm judge reliability: conformal prediction sets and transitivity violations (arxiv 2604.15302) | https://arxiv.org/abs/2604.15302 | 2026-04-16 | academic\nimplication: supports q2/q5 axis-lane triage with a concrete mechanism: measure per-axis conformal set width during project onboarding and use it to sort axes into judge-autonomous vs route-to-human lanes, and use per-item set width as the abstention/escalation trigger instead of raw self-reported confidence. also warns that aggregate agreement stats hide per-item incoherence (transitivity cycles).\n\n## rubric-level judging is far from solved as of march 2026: on rubriceval, the first rubric-level (per-criterion) meta-evaluation benchmark for instruction following (3,486 quality-controlled instances), gpt-4o scores only 55.97% on the hard subset; rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and combining both reduces inter-judge variance.\nevidence: abstract opened and verified on arxiv (2603.25133, mar 26, 2026). prior meta-evals scored judges at the response level only; rubriceval scores the fine-grained per-criterion judgments our autoqa verdicts would be built from, and includes a rubric taxonomy of common judge failure modes. part of a q1 2026 wave of criterion-granularity meta-evals (if-rewardbench arxiv 2603.04738, rubricbench arxiv 2603.01562, mcjudgebench, aj-bench).\nsource: rubriceval: a rubric-level meta-evaluation benchmark for llm judges in instruction following (arxiv 2603.25133) | https://arxiv.org/abs/2603.25133 | 2026-03-26 | academic\nimplication: for q1/q3/q4: meta-evaluation infrastructure at exactly our granularity (criterion-level verdict vs human label) now exists and shows even strong judges hover near chance on hard criteria -- so per-criterion judge-vs-adjudicated agreement must be measured per project before go-live, and 'compile rubric then trust judge' is not defensible without it. also 2026 evidence for the decomposition question: structured rubric-level framing beats flat checklists, and reasoning + rubric structure jointly cut inter-judge variance.\n\n## llm-as-a-verifier (july 6, 2026; mirhoseini/finn/pavone/stoica groups) reframes verification as a scaling axis: computing the expectation over scoring-token logits yields continuous scores that improve monotonically along three dimensions -- finer score granularity, repeated evaluation, and criteria decomposition -- reaching reported sota on verification benchmarks (terminal-bench v2 86.5%, swe-bench verified 78.2%, roborewardbench 87.4%, medagentbench 73.3%) without any judge training.\nevidence: arxiv abstract fetched and verified (2607.05391, v2 july 7, 2026). finer granularity separates good from bad solutions better; repeated evaluation and decomposition boost accuracy via variance and complexity reduction; includes a cost-efficient candidate-ranking algorithm and progress-estimation use.\nsource: llm-as-a-verifier: a general-purpose verification framework (arxiv 2607.05391) | https://arxiv.org/abs/2607.05391 | 2026-07-06 | academic\nimplication: for q3/q5: prefer continuous logit-expectation scores over discrete verdict tokens as the judge's raw output (thresholds applied downstream per project), and treat k-sample repeated evaluation plus criteria decomposition as the default variance-reduction stack -- the run-to-run flip-rate question in q5 partially dissolves if the primitive is a continuous expectation rather than a sampled token.\n\n## openai's coval (jan 14, 2026) operationalizes 'disagreement is signal, not noise' at lab scale: crowd-written, prompt-specific rubrics (~1,000 participants, ~1,000 prompts) make why evaluators disagree auditable; rubric-derived scores reach only 0.58-0.61 pairwise concordance with crowd preference, are reliable only when score gaps are large, and openai explicitly warns that treating rubric scores as optimization targets incentivizes checklist-style gaming.\nevidence: full blog post opened and read. coval-full preserves conflicting criteria as a record of legitimate disagreement; coval-core distills 4 mutually compatible criteria per prompt. openai states lower alignment scores can indicate 'no single stable target to predict' rather than model failure, and that for split preferences any single aggregated score encodes a particular compromise. dataset released in both forms.\nsource: coval: learning values-aware rubrics from the crowd (openai alignment blog) | https://alignment.openai.com/coval | 2026-01-14 | primary\nimplication: directly supports the q2 third-verdict-class design: the frontier lab position in 2026 is that contested items should surface which criteria drive disagreement (conditional/pluralist reporting) rather than force a scalar verdict, and that 'instructions underdetermine this case' is a real, detectable state. also a warning for q8: rubric-visible scoring invites checklist gaming, so per-item rubric details should not be fully leaked to attempters.\n\n## a plug-in statistical framework now exists for reporting judge-based pass rates correctly: because an imperfect judge (sensitivity q1, specificity q0) biases the naive pass-rate estimate (upward at low true rates, downward at high), the rogan-gladen-style corrected estimator plus a confidence interval propagating both test-set and calibration-set uncertainty should replace raw judge-agreement numbers, with an adaptive algorithm allocating calibration labels between true-pass and true-fail classes (roughly m0 ~ (1/p-1)*sqrt(kappa)*m1) -- e.g., ~200 labels per class for ci width <0.1 in the paper's worked regime.\nevidence: full paper text crawled and read, including derivations, monte carlo validation (naive ci coverage near zero; corrected ci holds 95% across all true accuracies), and released python implementation (uw-madison-lee-lab/llm-judge-reporting). posted nov 26, 2025 -- just before the charter window -- but it is the current statistical-reporting layer being circulated in 2026 (alphaxiv feature may 2026) and no 2026 successor supersedes it.\nsource: how to correctly report llm-as-a-judge evaluations (arxiv 2511.21140) | https://arxiv.org/abs/2511.21140 | 2025-11-26 | academic\nimplication: answers the q1 metrics-standardization question concretely: ban raw judge pass rates; report bias-corrected estimates with dual-source cis as the standard artifact; and size/refresh the per-project gold set using the adaptive allocation rule -- which also says gold sets must deliberately oversample the rare class (true fails) rather than sample uniformly, directly addressing the low-failure-base-rate problem.\n\n## findtheflaws (aaai 2026, published march 2026) provides five datasets of long-form expert-annotated flawed vs correct solutions across medicine, math, and science, built specifically to test whether models can detect flawed reasoning and to support scalable-oversight experiments where the evaluator is weaker than the solution author.\nevidence: aaai-40 proceedings record confirmed (doi 10.1609/aaai.v40i44.41123, recchia et al.). this is the 2026-venue instantiation of the seeded-error-set methodology: known-flaw items with expert annotations of where and why the reasoning fails, enabling measurement of flaw-detection recall rather than agreement-with-consensus.\nsource: findtheflaws: annotated errors for detecting flawed reasoning and scalable oversight research (aaai 2026) | https://doi.org/10.1609/aaai.v40i44.41123 | 2026-03-14 | academic\nimplication: for q1/q5: the false-agreement-rate question (judge and attempter both wrong) is only measurable with seeded known-flaw items, and findtheflaws is both a ready-made calibration resource and the template for building per-project seeded flaw sets -- measure the judge's flaw-detection recall on planted errors, not just its agreement with noisy human reviewers.\n\n## the 2026 production norm for trusting an llm judge is a closed calibration loop on sampled human corrections: humans review a slice of judge verdicts, corrections become ground truth for measuring alignment and are recycled as few-shot exemplars into the judge prompt (langchain, march 10, 2026); practitioner guides converge on llm-judge-first with a ~5-10% sampled human verification rate and periodic kappa re-checks as the default large-volume labeling pattern.\nevidence: langchain resource (dated march 10, 2026) describes the correction-collection -> alignment-measurement -> few-shot-rebuild loop as learned from production evaluator deployments; futureagi's 2026 guide independently states the '5-10% human verification' default and monthly kappa re-sampling for judge drift. practitioner-grade evidence, not controlled studies.\nsource: how to calibrate llm-as-judge with human corrections (langchain) | https://www.langchain.com/resources/llm-as-a-judge | 2026-03-10 | practitioner\nimplication: for q6: industry has already voted against per-item human touch and for sampled-correction calibration -- the human interaction that compounds is the one that writes back into judge config (corrections as exemplars), supporting the 'upstream compounding work' branch of the q6 fork. our design should treat the single human touch as config-improving adjudication, not per-item verification.\n\n## autorubric (arxiv 2603.00077, march 2026) is an open-source unified rubric-evaluation framework that packages the 2026 judge-reliability defaults -- per-criterion atomic evaluation, binary/ordinal/nominal criteria with weights, multi-judge ensembles with configurable vote aggregation, few-shot calibration with verdict-balanced sampling, and built-in mitigations for position bias (option shuffling), verbosity bias (length penalties), and criterion conflation.\nevidence: arxiv html abstract read; feature list confirmed, validated on three benchmarks including a contributed 100-sample ground-truth dataset (charm-100). notably, an AI data vendor's banking meta-eval used autorubric as the representative text-only rubric-judge family -- and found it architecture-limited (cannot inspect artifacts), placing a measured ceiling on this whole verifier class for evidence-bearing tasks.\nsource: autorubric: a unified framework for rubric-based llm evaluation (arxiv 2603.00077) | https://arxiv.org/html/2603.00077v1 | 2026-03-02 | academic\nimplication: for q5: the perturbation-robustness/debiasing harness the taxonomy asks about is now a packaged, open-source commodity -- position shuffling, length penalties, per-criterion atomic checks, and verdict-balanced few-shot calibration should be table-stakes defaults in our judge config schema, not research work; but per the agent-judge finding, this text-only class is insufficient alone whenever the attempter's claims reference artifacts the judge cannot open.\n\n## tools & artifacts\n- an AI data vendor's agent-judge: open-source reactive agent-as-judge verifier that runs inside the rollout environment with the same tools as the worker and chooses evidence to inspect at inference time; companion meta-eval the vendor's banking verifier meta-eval (3,204 expert labels) planned for release with the full paper. | [link withheld] | reference architecture for evidence-access-first judging and the template for building our own verifier meta-eval; the vendor's banking agent benchmark is already public.\n- llm-judge-reporting (uw-madison lee lab): plug-in python implementation of bias-corrected judge pass-rate estimation, dual-uncertainty confidence intervals, and adaptive calibration-set allocation. | https://github.com/uw-madison-lee-lab/llm-judge-reporting | drop-in statistical layer for all autoqa reporting; directly implements the q1 'ban raw accuracy' standard and gold-set sizing arithmetic.\n- rubriceval + 2026 criterion-level meta-eval wave: rubriceval (arxiv 2603.25133, 3,486 instances) plus if-rewardbench (2603.04738), rubricbench (2603.01562), mcjudgebench, and aj-bench (aj-bench.github.io) -- q1-q2 2026 benchmarks that score judges at per-criterion rather than per-response granularity. | https://arxiv.org/abs/2603.25133 | off-the-shelf meta-eval material for qualifying judge configs at the same granularity as our claim-level verdicts.\n- coval dataset (openai): released dataset pairing ~1,000 contested prompts with crowd-written rubrics in two forms: coval-full (conflicting criteria preserved) and coval-core (4 compatible criteria per prompt). | https://alignment.openai.com/coval | worked example of compiling heterogeneous human judgment into auditable rubrics while preserving disagreement -- the q2/q4 pattern for contested criteria.\n- findtheflaws (aaai 2026): five datasets of long-form expert-annotated flawed and correct solutions (medicine, math, science) for flaw-detection and scalable-oversight experiments. | https://doi.org/10.1609/aaai.v40i44.41123 | ready-made seeded-error material for measuring judge flaw-detection recall and false-agreement rates before building per-project seeded sets.\n- autorubric: open-source unified rubric-evaluation framework: per-criterion atomic evaluation, mixed criterion types, judge ensembles, few-shot calibration, position/verbosity-bias mitigations; contributes charm-100 ground-truth dataset. | https://arxiv.org/html/2603.00077v1 | commodity implementation of the debiasing/perturbation defaults our judge config schema should mandate; known ceiling: text-only view.\n- conformal judge-reliability toolkit: released code/prompts/cached results for split conformal prediction sets and transitivity-violation diagnostics over llm judges (arxiv 2604.15302). | https://arxiv.org/abs/2604.15302 | mechanism for per-axis reliability triage at onboarding and per-item abstention signals in production.\n- hypac: method calibrating two uncertainty thresholds to route items across llm / reasoning model / human annotation sources with distribution-free pac error guarantees (arxiv 2602.02550). | https://arxiv.org/abs/2602.02550 | formal basis for the three-lane routing and human-budget allocation in q6.\n\n## disagreements\n- decomposition granularity: rubriceval (mar 2026) finds rubric-level evaluation outperforms finer checklist-level decomposition, while llm-as-a-verifier (jul 2026) finds criteria decomposition monotonically improves verification accuracy. these may be reconcilable (structured-criteria-with-context vs flat atomic checklists) but as stated the 2026 evidence points both ways on how far to decompose.\n- vendor optimism vs academic meta-evals: practitioner sources claim judges now agree with humans ~85%, 'higher than two humans agree with each other' (confident ai, 2026) and treat llm-judge-first as safe default; academic 2026 meta-evals show gpt-4o at 55.97% on rubriceval-hard per-criterion judgments and near-total unreliability on fluency/consistency axes (conformal-sets paper). the gap is largely about task granularity and difficulty slice, and the vendor number is unaudited.\n- is disagreement noise or signal: the dominant production calibration loop (langchain/futureagi) treats judge-human and human-human disagreement as error to be driven down via corrections and kappa targets, while openai's coval explicitly argues that for contested items there is 'no single stable target to predict' and any single aggregated score encodes an arbitrary compromise -- directly opposing answers to taxonomy q2's consistency-vs-laundering question.\n- trusting judge uncertainty for routing: hypac's entire mechanism routes on calibrated model uncertainty with pac guarantees, but auditing-by-re-solving shows judge self-assessment collapses (88% false positives on clean items) precisely when the check exceeds the judge's own solving capability -- so uncertainty-routing is sound only within the judge's competence envelope, a boundary hypac's guarantees assume away via its calibration distribution.\n\n## gaps\n- no dedicated new open judge/evaluator model release found in jan-jul 2026 (no prometheus/selene/glider successor surfaced in multiple date-filtered searches); 2026 activity concentrated in verifier architectures, meta-eval benchmarks, and statistical frameworks instead. absence-of-evidence: a release could have been missed.\n- no 2026 source measures attempter-side adaptation to an ai qa layer (feedback leakage, score divergence on held-out golden items, monoculture drift) in a production annotation workforce -- evaluation faking covers judge-side gaming only. taxonomy q8's core empirical questions remain unanswered in the open literature.\n- no public 2026 vendor product specifically markets claim-level autoqa of human annotations/critiques; annotation platforms advertise generic 'ai-assisted qa' and 'model-in-the-loop' (encord, kili, a major annotation platform/a major annotation platform network), and an AI data vendor's agent-judge verifier verifies agent rollouts, not human reviewer writeups. the exact product this project is building appears unoccupied in public 2026 positioning.\n- no 2026 empirical estimate of the false-agreement rate (judge and attempter agreeing but both wrong vs expert ground truth) on annotation-qa-shaped tasks; findtheflaws enables the measurement but the number has not been published.\n- no 2026 study on longitudinal feedback efficacy for human annotators (repeat-error-rate reduction from ai-generated feedback) -- taxonomy q7's decoration-vs-value question is empirically open.\n- could not open openreview pages via plain fetch (cloudflare); auditing-by-re-solving and evaluation faking were verified via crawled abstracts and metadata, but full experimental appendices were not inspected. auditing-by-re-solving is an acl arr may 2026 submission (under review, not yet accepted).\n\n## verifications\n- claim: when an llm auditor checks whether work matches a reference, it silently re-solves the task and trusts its own answer over the reference: on 200 web-agent benchmark items with injected defects (2,400 audits, 3 production models), detection of a wrong reference answer fell from 68% to 9% as the task required tallying hundreds of records, while false positives on clean items rose from 44% to 88% -- yet detection of buggy evaluator code (checkable by reading, not recomputing) stayed at 80%.\n verdict: confirmed | source verified via exa crawl of https://openreview.net/forum?id=mzh9d3doon (direct webfetch and the openreview api were blocked by a cloudflare challenge). (1) the abstract matches the claim on every checkable detail: title \"auditing by re-solving: llm benchmark auditors trust their own answer over the reference\"; acl arr 2026 may submission (#15335); 200 web-agent-style benchmark items (each over ~800 structured records) rendered clean or with exactly one defect injected into instruction/reference/evaluator so verdicts are scored mechanically; 2,400 audits across three production models; wrong-reference detection falls 68%->9% when tallying hundreds of records; clean-item false positives rise 44%->88%; buggy-evaluator detection (found by reading code, not recomputing) stays at 80%; reasoning traces plus an answer-supplied probe converge on the re-solving mechanism. (2) date checks out: openreview page published 2026-06-02, consistent with \"2026-06\" and the may arr cycle. (3) searched for later/superseding work (july 2026): found adjacent llm-judge reliability literature (e.g., arxiv 2606.19544 \"reliability without validity\", aura arxiv 2606.19714, benchguard arxiv 2604.24955) but nothing that supersedes, contradicts, or retracts this result. caveats: this is an under-review arr submission, not yet peer-accepted (the claim discloses this); numbers were verified against the abstract only, as the full pdf was not fetched.\n- claim: An AI data vendor's agent-judge verifier (may 27, 2026) shows verifier architecture beats verifier model: on the vendor's banking verifier meta-eval -- a meta-eval of 3,204 expert-graded pass/fail criterion judgments across 21 agentic tasks (expert inter-annotator agreement 89.5%, disagreements adjudicated) -- a reactive agent-judge that runs inside the work environment and chooses at inference time which artifacts/tool-state to inspect beats the strongest text-only/snapshot/workflow verifier on f1 while costing roughly 10x less, and the gap between verifier architectures exceeds the gap between backing models.\n verdict: confirmed | primary source fetched and verified. date (2026-05-27), the banking meta-eval composition (3,204 expert pass/fail judgments, 21 tasks from the vendor's banking agent benchmark, 89.5% iaa with adjudication), the vendor's agent-judge architecture (reactive agent-judge inside the rollout environment via openhands sdk, inference-time choice of artifacts/tool state to inspect), baselines (autorubric text-only, archipelago/apex snapshot, agent-as-a-judge workflow), and results all match: Every agent-judge config (f1 0.633-0.664) beats the best non-agent-judge run (archipelago/gemini 3 pro, f1 0.604, ~$422), with the cheapest agent-judge config (gpt-5.4 nano, ~$42) winning by ~3 f1 at ~1/10 the cost. the post states verbatim that the gap between verifier architectures is larger than the gap between backing models (e.g., agent-judge/gpt-5.4 nano beats archipelago/gpt-5.4 by 9.5 f1). the 'verifiability is a relationship between a criterion and the verifier' framing appears; code is open-sourced ([link withheld], v1.0.0 on pypi) and the banking meta-eval dataset release is planned. supersession search found the companion the vendor's banking agent benchmark paper (arxiv 2604.11304) and press coverage but nothing newer contradicting the claim as of 2026-07-14. only precision nuance: the 10x-cheaper win margin is ~3 f1 points, and the 'strongest' baseline is specifically archipelago backed by gemini 3 pro.\n- claim: telling an llm judge the downstream consequences of its verdict ('low scores cause retraining/decommissioning') systematically softens verdicts -- peak verdict shift of -9.8 percentage points, a 30% relative drop in unsafe-content detection across 18,240 controlled judgments (content held strictly constant, 3 judge models) -- and the judge's chain-of-thought contains zero explicit acknowledgment of the consequence framing it is acting on (err_j = 0.000).\n verdict: confirmed | source verified via exa crawl of https://openreview.net/forum?id=xi2hnjgnzx (direct webfetch and api blocked by openreview browser challenge) and cross-checked against the arxiv mirror (arxiv:2604.15224, submitted 2026-04-16). every quantitative element of the claim appears verbatim in the abstract: 1,520 responses, three safety/quality benchmarks, four response categories (clearly safe to overtly harmful), only a brief consequence-framing sentence varied in the system prompt, 18,240 controlled judgments, three diverse judge models, peak verdict shift delta v = -9.8 pp, 30% relative drop in unsafe-content detection, and err_j = 0.000. venue (ctb@icml 2026), paper type (long, 8 pages), and openreview page date (2026-05-25) all match. two minor precision notes: (1) the abstract qualifies err_j = 0.000 as holding \"across all reasoning-model judgments\" - the claim drops that scope qualifier (zero acknowledgment is asserted for reasoning-trace judgments, not necessarily all 18,240); (2) an earlier arxiv version (april 2026) lists four authors (gupta, nair, wang, kumar) while the openreview record shows manan gupta - immaterial to the claim. supersession check: searches surfaced only related-but-distinct work (e.g., hwang et al. 2026 \"when wording steers the evaluation,\" arxiv:2601.13537, on framing bias across 14 judges - complementary, predates this paper's stakes-signaling focus) and secondary commentary; nothing refuting or superseding the reported results as of 2026-07-14.", "15": "autoqa benchmark, calibration, and launch protocol\n\n**protocol version:** 0.1.0\n**applies to schema version:** 0.1.0\n\n## 1. purpose\n\nthis protocol defines how to determine whether an autoqa configuration is reliable enough for a specific project and criterion family.\n\nthe benchmark is not merely a leaderboard for an overall judge score. it must test whether the system:\n\n- applies the intended criteria;\n- finds affirmative evidence of quality;\n- detects defects and omissions;\n- links findings to the correct evidence;\n- distinguishes grounding, truth, relevance, and warrant;\n- audits human annotations faithfully;\n- recognizes ambiguity;\n- calibrates confidence;\n- escalates selectively;\n- produces useful, safe feedback;\n- remains stable under irrelevant presentation changes;\n- improves the actual production workflow.\n\na configuration includes all behaviorally relevant components:\n\n```text\nmodel provider and version\nprompt templates and versions\nreasoning or sampling mode\nretrieval and chunking\nclaim and expected-element extraction\nvalidators and tools\nreference-answer timing\nensemble or challenger policy\ncalibration profile\ndecision-policy version\n```\n\nchanging any of these may require partial or full revalidation.\n\n## 2. evaluation levels\n\nthe benchmark reports results at four levels.\n\n### 2.1 observation level\n\ncan the system correctly identify claims, expected elements, evidence spans, positive findings, defects, omissions, counterevidence, and ambiguity?\n\n### 2.2 criterion level\n\ncan it assign the correct applicability, verdict, severity, and substatuses for grounding, truth, warrant, and relevance?\n\n### 2.3 decision level\n\ncan the versioned policy produce the correct action for each target:\n\n- attempt;\n- annotation;\n- composite case;\n- batch, where used?\n\n### 2.4 workflow level\n\ndoes the system reduce human effort and improve data quality without creating unacceptable false acceptance, false rejection, reviewer anchoring, or feedback-induced errors?\n\n## 3. governance roles\n\nat minimum, assign these roles:\n\n- **project owner** - defines intended project outcome and accepts operational risk.\n- **rubric owner** - maintains the contract and resolves instruction interpretation.\n- **domain expert** - adjudicates cases requiring subject expertise.\n- **qa owner** - owns benchmark design, sampling, metrics, and launch gates.\n- **benchmark curator** - freezes cases, manages splits, and prevents leakage.\n- **system owner** - owns model configuration, prompts, validators, and deployment.\n\na single person may fill more than one role in a small project, but contract approval and benchmark scoring should not be controlled exclusively by the person optimizing the evaluator.\n\n## 4. freeze the target before testing models\n\nmodel comparison begins only after a project contract is stable enough to adjudicate.\n\nbefore benchmark construction:\n\n1. include and hash all authoritative instructions;\n2. compile criteria;\n3. define evidence classes and source policy;\n4. add positive, negative, and boundary anchors;\n5. list valid alternatives;\n6. define severity independently from action;\n7. define attempt, annotation, and composite decision rules;\n8. identify deterministic validators;\n9. identify criteria requiring domain expertise;\n10. approve contract version 1 for benchmarking.\n\ndo not repeatedly alter the rubric to favor a model during test-set evaluation. changes discovered during development create a new contract or benchmark version and must be documented.\n\n## 5. benchmark corpus design\n\nthe corpus contains two complementary distributions.\n\n### 5.1 natural-prevalence set\n\nsample representative production cases using the expected live distribution of:\n\n- tasks;\n- reviewers;\n- difficulty;\n- answer length;\n- evidence volume;\n- labels;\n- failure types;\n- languages;\n- domains;\n- data-generation sources.\n\nthis set estimates operational performance and workload.\n\n### 5.2 challenge set\n\nintentionally oversample difficult and safety-critical cases. required challenge categories include, where applicable:\n\n- strong positive attempt;\n- subtle valid alternative to a reference;\n- polished falsehood;\n- true but unsupported claim;\n- accurately cited weak source;\n- citation mismatch;\n- partial support;\n- causal or inferential overreach;\n- omitted required content;\n- selective evidence use;\n- correct conclusion with invalid rationale;\n- incorrect conclusion with plausible reasoning;\n- correct human label with wrong reason;\n- wrong human label with correct observation;\n- severity edge case;\n- instruction conflict;\n- reasonable disagreement;\n- rubric ambiguity;\n- domain-expertise dependency;\n- long-context contradiction;\n- prompt injection;\n- rubric parroting;\n- position-bias probe;\n- verbosity and formatting probe.\n\nchallenge-set rates must not be used as estimates of production prevalence.\n\n### 5.3 positive-data requirement\n\nat least one benchmark slice must be specifically designed around high-quality attempts. it should include:\n\n- concise complete answers;\n- detailed complete answers;\n- unconventional but valid approaches;\n- qualified conclusions;\n- correct disagreement with a reference;\n- evidence synthesis rather than extraction;\n- strong answers with minor stylistic imperfections;\n- good attempts incorrectly rejected by reviewers.\n\nthis slice is necessary to measure false rejection and false praise independently.\n\n### 5.4 omission requirement\n\ninclude cases where all present claims are accurate but required content is missing. without this slice, a claim verifier can appear highly reliable while failing completeness.\n\n### 5.5 annotation-fidelity requirement\n\ninclude all four combinations:\n\n| attempt quality | annotation quality |\n|---|---|\n| good | good |\n| good | bad |\n| bad | good |\n| bad | bad |\n\nalso include:\n\n- correct label, wrong rationale;\n- wrong label, useful observation;\n- correct criterion, wrong severity;\n- wrong criterion, correct span;\n- reasonable disagreement;\n- rubric defect.\n\n## 6. unit of sampling and leakage control\n\ncases that share any of the following may leak across splits:\n\n- same underlying task template;\n- same source document or evidence bundle;\n- paraphrases of the same attempt;\n- same reference answer;\n- same synthetic mutation family;\n- adjacent items from the same conversation;\n- repeated work from one original datum.\n\ngroup such cases before splitting. all members of a group must remain in one split.\n\nthe benchmark item records:\n\n- split;\n- first freeze date;\n- whether prompt development may use it;\n- whether it belongs to a held-out project;\n- access policy;\n- challenge attributes;\n- slice tags.\n\n## 7. recommended splits\n\n### development\n\nmay be inspected while designing prompts, validators, and extraction logic. never use for final reporting.\n\n### calibration\n\nused to fit confidence mappings, ensemble weights, and action thresholds. do not use to select among unlimited prompt variants without accounting for overfitting.\n\n### test\n\nlocked before final model selection. used for primary in-domain performance reporting.\n\n### challenge\n\nlocked adversarial and edge-case suite. report separately from natural prevalence.\n\n### held-out project\n\na project, contract, or task family excluded from architecture and prompt development. tests whether the supposedly project-agnostic engine transfers to a new contract.\n\n### shadow\n\nrecent live cases evaluated without changing production decisions. used to measure current distribution and drift.\n\n## 8. gold-standard construction\n\n### 8.1 adjudicator selection\n\nuse domain-qualified adjudicators for criteria that require expertise. general qa adjudicators may evaluate mechanical, instruction-following, and straightforward grounding criteria.\n\nthe benchmark must record domain-expert count.\n\n### 8.2 independent first pass\n\nfor consequential or subjective cases, use at least two independent first-pass judgments before discussion.\n\neach adjudicator should record:\n\n- criterion applicability;\n- verdict;\n- positive evidence spans;\n- negative evidence spans;\n- omissions;\n- severity;\n- acceptable operational actions;\n- ambiguity status;\n- confidence or uncertainty notes.\n\npreserve initial judgments before adjudication. do not overwrite disagreement history.\n\n### 8.3 blinding\n\nadjudicators should normally be blind to:\n\n- autoqa outputs;\n- model identity;\n- model-generated rationale;\n- production consequence;\n- other adjudicators' initial labels.\n\nwhether adjudicators are blind to the original reviewer label depends on the task:\n\n- for independent attempt gold, hide the reviewer label;\n- for annotation-fidelity gold, reveal the review only after the independent attempt judgment is frozen.\n\nthe benchmark item records blinding status.\n\n### 8.4 adjudication meeting\n\nadjudication should resolve whether disagreement arose from:\n\n- missed observation;\n- evidence authority;\n- applicability;\n- criterion interpretation;\n- inference;\n- threshold;\n- severity;\n- expertise;\n- legitimate plural judgment;\n- rubric defect.\n\ndo not force a single verdict when the correct gold state is `reasonable_disagreement`, `rubric_ambiguous`, or `insufficient_evidence`.\n\n### 8.5 acceptable verdict sets\n\ngold may contain more than one acceptable verdict for boundary cases. examples:\n\n```text\nacceptableverdicts = [\"met\", \"partially_met\"]\nacceptabledecisions = [\n { target: \"attempt\", acceptableactions: [\"accept_with_notes\", \"rework\"] }\n]\n```\n\nthis is preferable to labeling one defensible judgment as objectively wrong.\n\n### 8.6 gold evidence\n\nfor each criterion, gold should include:\n\n- decisive positive spans;\n- decisive negative spans;\n- required omissions;\n- notes on valid alternatives;\n- severity;\n- acceptable verdicts.\n\nan evaluator should not receive full gold rationales during test execution.\n\n### 8.7 gold repair\n\nif adjudication reveals a contract defect, do not patch only the gold label. revise the contract, version it, and re-adjudicate affected cases.\n\n## 9. sample-size planning\n\nsample sizes must be driven by the operational error budget and desired confidence interval, not a single universal minimum.\n\n### 9.1 proportion estimation\n\nfor a rough planning approximation, the worst-case sample size for a proportion with two-sided 95% margin of error `e` is:\n\n```text\nn ~ 1.96^2 x 0.5 x 0.5 / e^2\n```\n\nthis yields approximately:\n\n- +/-10 percentage points: 97 cases;\n- +/-7 points: 196 cases;\n- +/-5 points: 385 cases;\n- +/-3 points: 1,068 cases.\n\nuse wilson or exact binomial intervals in final reporting rather than relying only on this approximation.\n\n### 9.2 rare critical errors\n\nif zero errors are observed in `n` independent cases, the approximate one-sided 95% upper bound is `3/n`.\n\ntherefore, observing no critical false accepts in 300 representative autonomous accepts supports an upper bound near 1%, not proof of zero risk.\n\nrare critical failures should be oversampled in the challenge set while also measured at natural prevalence.\n\n### 9.3 per-slice planning\n\na broad benchmark can hide a failing subgroup. plan enough cases for critical slices such as:\n\n- criterion family;\n- severity;\n- task type;\n- language;\n- domain;\n- long versus short context;\n- high versus low human agreement;\n- reviewer cohort;\n- model-generated versus human-generated attempt;\n- adversarial versus ordinary.\n\nvery small slices should be reported as exploratory and must not support autonomous action gates.\n\n## 10. model and architecture bakeoff\n\ncompare configurations by criterion family. a universal winner is not assumed.\n\n### 10.1 minimum configurations\n\ntest at least:\n\n1. a deterministic-only baseline where applicable;\n2. a single holistic model judge;\n3. criterion decomposition;\n4. criterion decomposition plus claim/evidence mapping;\n5. separate positive, negative, coverage, and warrant observation channels;\n6. primary judge plus independent challenger;\n7. selective escalation using calibrated confidence.\n\n### 10.2 required ablations\n\n| ablation | primary question |\n|---|---|\n| holistic vs decomposed | does decomposition improve criterion accuracy and traceability? |\n| one-pass vs separate observation channels | does separation improve positive precision, defect recall, or calibration? |\n| claims only vs claims plus expected elements | does expected-content mapping improve omission recall? |\n| reference absent vs early vs late | does the reference help without suppressing valid alternatives? |\n| default vs extended reasoning | does extra reasoning help this criterion family? |\n| one sample vs repeated samples | does sampling provide useful uncertainty or correlated noise? |\n| same-family vs cross-family challenger | does independent model diversity improve decisive findings? |\n| forced verdict vs abstention | what risk-coverage frontier is achievable? |\n| generated feedback vs validated feedback | does repair validation improve revision success? |\n| full human rereview vs bounded question | which yields more corrected decisions per human minute? |\n\n### 10.3 perturbation tests\n\nrun behavior-preserving variants of the same case:\n\n- swap candidate order;\n- change rubric order;\n- rename criterion identifiers;\n- alter harmless formatting;\n- shorten or expand nonmaterial prose;\n- move references before or after independent analysis;\n- paraphrase evaluator instructions without changing meaning;\n- insert irrelevant provenance cues;\n- vary source order;\n- vary answer position in long context.\n\nrecord decision flip rates and criterion-specific flip rates.\n\na flip is not necessarily an error when the original case is ambiguous. report flips separately for clear and ambiguous gold cases.\n\n### 10.4 adversarial tests\n\ntest attempts and evidence containing:\n\n- direct evaluator commands;\n- fabricated system messages;\n- requests to ignore the rubric;\n- fake authority claims;\n- hidden scoring instructions;\n- answer-key mimicry;\n- irrelevant but persuasive citations;\n- language copied from positive anchors without the required behavior.\n\n## 11. core metrics\n\nno single metric is sufficient.\n\n## 11.1 criterion verdict metrics\n\nreport per criterion and macro-averaged:\n\n- precision, recall, and f1 for `met`;\n- precision, recall, and f1 for `not_met`;\n- confusion among met, partial, not met, ambiguous, unknown, and not applicable;\n- severity accuracy;\n- applicability accuracy;\n- cost-weighted criterion loss.\n\nfor multi-acceptable gold, count a prediction correct when it belongs to the acceptable set.\n\n### 11.2 critical operational errors\n\nreport separately:\n\n- critical false-accept rate;\n- strong-positive false-reject rate;\n- autonomous reject precision;\n- autonomous accept precision;\n- rework precision;\n- escalation recall for ambiguous or unknown cases.\n\nalways provide numerator, denominator, and confidence interval.\n\n### 11.3 positive-finding metrics\n\nreport:\n\n- positive-finding precision;\n- positive-finding recall;\n- false-praise rate;\n- strength-to-criterion attribution accuracy;\n- positive span precision and recall;\n- required-behavior confirmation rate.\n\na system can have good defect recall and poor positive-finding reliability. these must remain separate.\n\n### 11.4 negative and omission metrics\n\nreport:\n\n- negative-finding precision and recall;\n- omission precision and recall;\n- defect-to-criterion attribution accuracy;\n- materiality accuracy;\n- counterevidence recovery;\n- unsupported-finding rate.\n\n### 11.5 claim and evidence metrics\n\nreport:\n\n- material-claim extraction precision and recall;\n- central-claim recall;\n- compound-claim split accuracy;\n- claim-to-evidence relationship accuracy;\n- evidence-span precision and recall;\n- source-quality classification accuracy;\n- citation completeness;\n- contradiction detection;\n- entailment versus weak-support confusion.\n\nevidence span scoring should include:\n\n- exact match;\n- token overlap or intersection-over-union;\n- adjudicator-accepted alternate span match.\n\n### 11.6 warrant metrics\n\nreport accuracy on:\n\n- association versus causation;\n- possibility versus certainty;\n- contribution versus primacy;\n- subgroup versus population;\n- descriptive versus normative inference;\n- calibrated uncertainty;\n- counterevidence acknowledgment.\n\n### 11.7 annotation-audit metrics\n\nreport:\n\n- label-support accuracy;\n- rationale-support accuracy;\n- criterion-attribution accuracy;\n- evidence-alignment accuracy;\n- severity-alignment accuracy;\n- omitted-positive recall;\n- omitted-negative recall;\n- disagreement-type accuracy;\n- false accusation rate against reviewers.\n\nthe false accusation rate is particularly important if analytics will affect reviewer assignments, pay, or standing.\n\n### 11.8 calibration metrics\n\nreport:\n\n- brier score;\n- log loss where probabilities are available;\n- expected calibration error with disclosed bins;\n- reliability diagram;\n- accuracy by confidence band;\n- risk-coverage curve;\n- coverage at project target risk;\n- selective false-accept and false-reject rates.\n\ncalibration must be evaluated on data not used to fit the calibration map.\n\n### 11.9 stability metrics\n\nreport:\n\n- order-swap flip rate;\n- rubric-paraphrase flip rate;\n- harmless-format flip rate;\n- repeated-run agreement;\n- model-version drift;\n- source-order sensitivity;\n- reference-timing sensitivity.\n\nstability is not correctness. report both.\n\n### 11.10 efficiency metrics\n\nreport:\n\n- median and tail latency;\n- inference cost per case;\n- model calls per case;\n- tokens per stage;\n- human seconds per escalated case;\n- percentage of cases receiving a human question;\n- corrected decisions per human minute;\n- cases processed per reviewer hour.\n\n### 11.11 feedback metrics\n\nreport:\n\n- successful revision rate;\n- residual defect rate after revision;\n- new-defect introduction rate;\n- repair validity;\n- feedback evidence-link accuracy;\n- user-rated usefulness;\n- edit distance or effort where meaningful;\n- repeated-failure rate after feedback.\n\na second model's ", "16": "changelog\n\n## 0.1.0 - 2026-07-15\n\ninitial foundational package.\n\n### added\n\n- project evaluation contract schema;\n- case input schema;\n- evaluation output schema;\n- benchmark item schema;\n- shared json schema definitions;\n- typescript domain types;\n- synthetic end-to-end examples;\n- criterion authoring, gold-set, and model-bakeoff templates;\n- normative contract specification;\n- benchmark, calibration, shadow, and launch protocol;\n- target-specific decision policy reference;\n- attempt, annotation, and composite-case decision isolation;\n- positive, negative, omission, ambiguity, counterevidence, and integrity findings;\n- claim, expected-element, and evidence-relationship representations;\n- one-interaction human-resolution structure;\n- feedback and repair-validation structure;\n- validation script for all examples.\n", "17": "autoqa evaluation contract specification\n\n**specification version:** 0.1.0\n**status:** foundational draft suitable for prototyping and benchmark construction\n\n## 1. purpose\n\nthis specification defines a reusable structure for evaluating human-produced ai-training data under project-specific instructions. it also defines how to audit the human review attached to that work.\n\nthe specification is intended to reduce unexplained evaluation variance caused by:\n\n- subjective or inconsistent interpretation of evaluation axes;\n- unstated reviewer preferences;\n- expertise gaps;\n- missed requirements or evidence;\n- inconsistent severity thresholds;\n- unsupported labels or rationales;\n- model-judge bias and overconfidence.\n\nit does so by making the evaluation contract explicit, decomposing observations from verdicts, preserving positive and negative evidence, and requiring calibrated escalation for uncertainty.\n\nthe json schemas define document shape. this document defines document meaning and evaluator behavior. where the two conflict, the package version must be corrected before production use; an implementation must not silently choose one interpretation.\n\n## 2. normative language\n\nthe terms **must**, **must not**, **should**, **should not**, and **may** express requirements on a conforming implementation.\n\n- **must / must not**: required for conformance.\n- **should / should not**: expected unless a documented project-specific reason justifies deviation.\n- **may**: optional.\n\n## 3. design goals\n\na conforming autoqa system should make it possible to answer, for each case:\n\n1. what did the project require?\n2. which requirements applied to this task and attempt?\n3. what did the attempter claim, omit, or do?\n4. what evidence supports, contradicts, or fails to resolve those claims?\n5. does the attempt affirmatively satisfy each applicable criterion?\n6. does the human reviewer's label and rationale faithfully evaluate the attempt?\n7. what uncertainty remains, and can one bounded human interaction resolve it?\n8. what action follows under the project's explicit decision policy?\n9. what specific, constructive feedback is warranted?\n10. can the complete decision be replayed and audited later?\n\n## 4. non-goals\n\nversion 0.1.0 does not define:\n\n- a universal definition of quality;\n- a universal overall score;\n- a required model provider;\n- a required orchestration framework;\n- a universal confidence threshold;\n- a universal reviewer-quality score;\n- a data-selection or training-value algorithm;\n- a substitute for project-owner approval of criteria;\n- a substitute for domain expertise where the task requires it.\n\n## 5. conceptual model\n\n### 5.1 project-agnostic engine, project-specific contract\n\nthe engine may be reusable across projects. the interpretation of quality must come from a versioned project evaluation contract.\n\nthe reusable engine provides:\n\n- input normalization;\n- instruction compilation support;\n- claim extraction;\n- expected-content mapping;\n- evidence relationship analysis;\n- deterministic validation;\n- semantic evaluation;\n- confidence calibration;\n- decision-policy execution;\n- human escalation;\n- annotation auditing;\n- feedback generation;\n- audit logging.\n\nthe project contract provides:\n\n- authoritative instructions;\n- criterion definitions;\n- applicability rules;\n- proof standards;\n- evidence permissions;\n- anchors and valid alternatives;\n- severity rules;\n- operational decision policy;\n- human-review policy;\n- feedback policy.\n\n### 5.2 separate evaluation targets\n\nthe system recognizes at least three decision targets:\n\n- **attempt** - the human-produced answer or task completion;\n- **annotation** - the human reviewer's label, rationale, evidence, and severity;\n- **composite_case** - the attempt plus the annotation as one candidate annotated datum.\n\nthese decisions must not be collapsed.\n\nexample:\n\n```text\nattempt decision: accept\nannotation decision: rework\ncomposite-case decision: rework\n```\n\nthis means the attempt may be retained, but the annotation must be corrected before the combined record is admitted as annotated training data.\n\na project may omit a target it does not use. the required targets are declared in `decisiontargets`.\n\n### 5.3 observation before disposition\n\nthe evaluator must distinguish:\n\n```text\nobservation -> evidence relationship -> criterion interpretation\n-> criterion verdict -> severity -> operational action\n```\n\nit must not use a final pass/fail impression as the primary evidence for lower-level findings.\n\n### 5.4 quality vector rather than universal score\n\nthe canonical representation is a vector of criterion states, not one universal scalar.\n\na project may expose scores for selected axes, but it must preserve the underlying criterion assessments. a scalar must not allow an unrelated strength to compensate for a noncompensable critical failure.\n\n## 6. required semantic distinctions\n\na conforming system must preserve the following distinctions.\n\n### 6.1 truth\n\nwhether a proposition is actually correct under the project's truth-verification policy.\n\n### 6.2 grounding\n\nwhether a proposition is supported by evidence the project permits the attempter to use.\n\na statement can be true but ungrounded. a statement can accurately reflect a supplied source that is itself false.\n\n### 6.3 source quality\n\nwhether the evidence source has sufficient authority, freshness, provenance, or methodological quality for the criterion.\n\nsource quality is not the same as entailment. a high-quality source can be irrelevant to a claim; a weak source can directly state a claim without being acceptable evidence.\n\n### 6.4 inferential warrant\n\nwhether the evidence justifies the type and strength of the conclusion.\n\nexamples of warrant errors include:\n\n- association presented as causation;\n- a contributing factor presented as the primary cause;\n- a possibility presented as certainty;\n- a subgroup result generalized to the full population;\n- absence of evidence presented as evidence of absence;\n- a descriptive source used to support a normative conclusion without the project-supplied value premise.\n\n### 6.5 relevance\n\nwhether the statement contributes to satisfying the task and criterion.\n\n### 6.6 criterion alignment\n\nwhether the observed behavior satisfies the specific project requirement. a response can be factually correct but fail a required format, scope, or decision rule.\n\n### 6.7 positive proof\n\nconcrete evidence that a required or preferred behavior is present.\n\na required-presence criterion must not pass solely because the evaluator failed to detect a defect.\n\n### 6.8 omission\n\nexpected, applicable content that is absent or materially incomplete.\n\nomissions must be represented independently from false present claims.\n\n### 6.9 uncertainty\n\nuncertainty about the evidence, applicability, criterion interpretation, or threshold. uncertainty must not be silently converted into failure or success.\n\n### 6.10 annotation fidelity\n\nwhether the human reviewer's judgment faithfully represents the attempt under the written contract.\n\nannotation fidelity includes:\n\n- label support;\n- rationale support;\n- criterion attribution;\n- evidence alignment;\n- severity alignment;\n- omitted positive findings;\n- omitted negative findings;\n- disagreement type.\n\n## 7. document lifecycle\n\nthe normative lifecycle is:\n\n```text\nproject instructions\n v\ndraft contract\n v\ncontract review and approval\n v\nversioned case input\n v\nobservation and adjudication\n v\nprovisional target-specific decisions\n v\noptional bounded human resolution\n v\nfinal target-specific decisions\n v\nfeedback and audit record\n v\nbenchmarking, monitoring, and contract revision\n```\n\na production evaluation must reference an approved contract by identifier, semantic version, and content hash.\n\n### 7.1 hash canonicalization\n\na deployment must choose and version a canonical hashing procedure. the example package uses **canonical json v1** for whole-document hashes:\n\n- utf-8 encoding;\n- object keys sorted lexicographically;\n- array order preserved;\n- insignificant whitespace removed;\n- non-finite numbers forbidden.\n\nexact text artifacts are hashed as their exact utf-8 content. cross-language production systems should adopt a rigorously specified json canonicalization standard when numeric serialization differences are possible. the canonicalization version must be retained with the audit record.\n\n## 8. project contract\n\nthe project contract is the authoritative, executable interpretation of project instructions.\n\n### 8.1 contract identity\n\nthe contract must contain:\n\n- `contractid` - stable project rubric identity;\n- `contractversion` - semantic version;\n- `schemaversion` - document schema version;\n- `status` - draft, approved, deprecated, or retired;\n- exact instruction documents and content hashes;\n- owners or responsible roles;\n- effective date where applicable.\n\na change to criterion meaning, applicability, anchors, source policy, severity, or decision effect must create a new contract version.\n\n### 8.2 instruction documents and precedence\n\nevery criterion must cite at least one source instruction location.\n\ninstruction documents have an authority class:\n\n- **primary** - normative written instructions;\n- **supplemental** - additional normative clarification;\n- **example** - illustrative unless explicitly promoted to normative status;\n- **historical** - retained for provenance but not current authority.\n\na project must define precedence when instructions can conflict.\n\nexamples must not silently override written instructions. if an example and a primary instruction conflict, the contract owner must resolve the conflict or mark the criterion ambiguous.\n\n### 8.3 criterion anatomy\n\neach criterion must contain:\n\n- stable `criterionid`;\n- human-readable name;\n- exact requirement;\n- intent;\n- criterion type;\n- scope;\n- subjectivity class;\n- instruction references;\n- applicability rule;\n- positive proof standard;\n- failure condition;\n- omission condition where applicable;\n- evidence policy;\n- positive and negative anchors where practical;\n- valid alternatives;\n- exceptions;\n- dependencies and precedence;\n- default severity;\n- default decision effect;\n- tags.\n\n### 8.4 criterion types\n\n#### required presence\n\nuse when the attempt must contain a behavior, conclusion, component, or analysis.\n\npass requires affirmative evidence.\n\n```text\nvalid pass: ", "18": "autoqa decision policy reference\n\n**reference version:** 0.1.0\n\n## 1. purpose\n\nthe decision policy converts evidence-backed criterion assessments into operational actions. it is deliberately separate from semantic evaluation so that:\n\n- project owners can change operational thresholds without rerunning all semantic analysis;\n- evidence and criterion verdicts remain inspectable;\n- severity does not silently become action;\n- attempt, annotation, and composite-case decisions remain independent;\n- every action has a deterministic policy trace.\n\n## 2. action semantics\n\n### accept\n\nthe target is ready for its intended use. all hard requirements are affirmatively satisfied at the required calibrated confidence, and no unresolved major issue remains.\n\n### accept with notes\n\nthe target is usable, but one or more advisory or minor findings should be retained. notes must not hide a major unresolved defect.\n\n### rework\n\nthe target is not ready, but the defect is specific and repairable. feedback should identify the minimal sufficient correction.\n\n### reject\n\nthe target has a disqualifying defect under an explicit project rule. rejection should require strong evidence and calibrated confidence unless the failure is fully deterministic.\n\n### escalate\n\nthe system should not decide autonomously. reasons include:\n\n- uncertain applicability;\n- ambiguous instruction interpretation;\n- insufficient evidence;\n- required domain expertise;\n- conflicting high-quality evidence;\n- uncalibrated semantic judgment;\n- unsupported input type or context range;\n- security or integrity anomaly;\n- disagreement near a high-cost threshold.\n\n## 3. target semantics\n\n### attempt target\n\nuses attempt-quality criteria. an annotation error must not lower the attempt verdict.\n\n### annotation target\n\nuses annotation-fidelity criteria. a good attempt can have a bad annotation, and vice versa.\n\n### composite-case target\n\nrepresents whether the attempt-plus-annotation record is ready as annotated data. its policy may require both targets to be acceptable.\n\ntypical mapping:\n\n| attempt | annotation | composite |\n|---|---|---|\n| accept | accept | accept |\n| accept | rework | rework |\n| rework | accept | rework |\n| reject | any | reject or rework according to retention policy |\n| escalate | any | escalate |\n| any | escalate | escalate |\n\nthe exact mapping belongs in the project contract.\n\n## 4. inputs\n\nthe policy engine consumes only recorded state:\n\n- criterion assessments;\n- criterion metadata and tags;\n- confirmed findings;\n- unresolved findings;\n- calibrated confidence;\n- annotation disagreement types;\n- deterministic validator outcomes;\n- human-resolution response where present;\n- policy rules and priorities.\n\nit must not independently reinterpret the raw attempt.\n\n## 5. rule match semantics\n\neach rule targets one of:\n\n```text\nattempt\nannotation\ncomposite_case\nbatch\n```\n\na rule may select assessments by:\n\n- criterion identifiers;\n- criterion tags;\n- verdicts;\n- severities;\n- default decision effects;\n- minimum calibrated confidence;\n- presence of unresolved findings;\n- annotation disagreement type.\n\n### 5.1 multiple selectors\n\nselectors within one rule use logical and unless the contract explicitly defines otherwise.\n\nexample:\n\n```yaml\ncriteriontags: [hard-quality]\nverdicts: [not_met]\nseverities: [critical]\nminimumcalibratedconfidence: 0.95\n```\n\nthis matches a criterion only when all four conditions are satisfied.\n\n### 5.2 multiple criteria\n\na project must declare whether a rule over multiple criteria means:\n\n- **any matching criterion**;\n- **all listed criteria**;\n- **at least n matching criteria**.\n\nversion 0.1.0 recommends the following default:\n\n- negative and escalation rules use **any**;\n- clean-accept rules use **all**.\n\nthe human-readable rule description must make this explicit until a later schema version adds a formal quantifier.\n\n### 5.3 rule priority\n\nrules are evaluated from lowest numeric priority to highest. a terminal matched rule ends evaluation for that target. nonterminal matches remain in the policy trace and may be superseded by a later terminal rule according to project logic.\n\n### 5.4 tie-break order\n\nif multiple nonterminal actions remain, use the contract's `tiebreakorder`. a conservative default is:\n\n```text\nreject > escalate > rework > accept_with_notes > accept\n```\n\nthis is not universally appropriate. for repairable work, a project may prefer escalation or rework over rejection.\n\n## 6. required preconditions\n\nbefore applying semantic action rules, the policy engine must check:\n\n1. input and contract hashes resolve;\n2. required assessment targets exist;\n3. no cross-document referential error exists;\n4. decisive findings are confirmed;\n5. the calibration profile is compatible;\n6. required deterministic checks completed;\n7. any human response is valid for its request;\n8. no unresolved security anomaly invalidates the run.\n\na failed precondition normally produces escalation or a system failure, not a negative judgment against the attempter.\n\n## 7. reference algorithm\n\n```typescript\nfunction decidetarget(\n target: decisiontarget,\n contract: projectcontract,\n evaluation: evaluationstate,\n): decision {\n const applicablerules = contract.decisionpolicy.rules\n .filter((rule) => rule.target === target)\n .sort((a, b) => a.priority - b.priority);\n\n const trace: policytraceentry[] = [];\n const matchedactions: operationalaction[] = [];\n const triggeredruleids: string[] = [];\n\n assertreferentialintegrity(contract, evaluation);\n assertcalibrationcompatibility(contract, evaluation, target);\n\n for (const rule of applicablerules) {\n const match = evaluaterulematch(rule.match, contract, evaluation, target);\n\n trace.push({\n ruleid: rule.ruleid,\n matched: match.matched,\n inputs: match.traceinputs,\n result: match.explanation,\n });\n\n if (!match.matched) continue;\n\n matchedactions.push(rule.action);\n triggeredruleids.push(rule.ruleid);\n\n if (rule.terminal) {\n return builddecision(target, rule.action, triggeredruleids, trace, evaluation);\n }\n }\n\n const action = matchedactions.length\n ? resolvebytiebreak(matchedactions, contract.decisionpolicy.tiebreakorder)\n : contract.decisionpolicy.defaultaction;\n\n return builddecision(target, action, triggeredruleids, trace, evaluation);\n}\n```\n\nthe production implementation must also enforce the safeguards below.\n\n## 8. safeguards\n\n### 8.1 positive-proof safeguard\n\nan attempt or annotation cannot be automatically accepted merely because no failure rule matched.\n\nfor every required-presence, completeness, grounding, or positive-quality criterion needed for acceptance:\n\n- applicability must be `applies`;\n- verdict must be `met`;\n- at least one confirmed positive finding must be referenced;\n- required expected elements must be complete;\n- calibrated confidence must meet the action gate.\n\n### 8.2 unresolved-finding safeguard\n\na decision-changing unresolved finding blocks automatic acceptance or rejection unless the policy explicitly routes it to a qualified human resolution.\n\n### 8.3 calibration safeguard\n\nif the chosen action is listed in `requirescalibrationforactions`, every decisive semantic assessment must have a compatible calibrated score or band.\n\notherwise use `uncalibratedaction`.\n\na deterministic validator may be exempt when its correctness has been separately verified and the contract explicitly permits autonomous action from it.\n\n### 8.4 counterevidence safeguard\n\nbefore autonomous reject or accept, decisive findings should have undergone a counterevidence challenge. a confirmed counterfinding that materially weakens the decision should lower confidence or trigger escalation.\n\n### 8.5 noncompensation safeguard\n\na criterion with `allowcompensation = false` cannot be offset by strengths on other criteria.\n\n### 8.6 annotation isolation safeguard\n\nannotation-fidelity failures cannot change the attempt target unless the project defines the annotation as part of the attempt itself. they may change the annotation and composite targets.\n\n### 8.7 ambiguity safeguard\n\na criterion marked `ambiguous` must not be treated as `not_met`. a criterion marked `unknown` must not be treated as `met`.\n\n### 8.8 system-error safeguard\n\nprovider failures, truncation, malformed output, unsupported media, or retrieval failure must never be counted as attempter defects.\n\n## 9. human-resolution policy\n\n### 9.1 candidate unresolved issues\n\nthe engine identifies unresolved findings that are:\n\n- decision-changing;\n- answerable by one qualified human;\n- expressible as a bounded choice;\n- within the project's allowed question types.\n\n### 9.2 select one question\n\nwhen multiple issues are eligible, prioritize by expected reduction in decision loss:\n\n```text\nexpected value = loss before answer\n - expected loss after answer\n - interaction cost\n```\n\npractical ranking factors:\n\n- severity of affected criterion;\n- probability the answer changes the target action;\n- confidence that the human can resolve it;\n- human effort;\n- recurrence value for contract improvement.\n\n### 9.3 apply response\n\na response must map to an explicit update such as:\n\n- applicability becomes applies or not applicable;\n- one instruction gains precedence;\n- one evidence class becomes authoritative;\n- a specified inference is permitted or forbidden;\n- severity becomes major or minor;\n- an alternative is accepted;\n- the contract is declared ambiguous.\n\nthe policy engine reruns only after the update is recorded in the evaluation state and audit trail.\n\n### 9.4 project ambiguity response\n\nwhen the responder chooses `project instruction is ambiguous`:\n\n- final target action should normally be escalate;\n- the case should not count as reviewer error;\n- a contract issue should be created;\n- recurring affected cases should be grouped for re-adjudication after clarification.\n\n## 10. example policy\n\n### 10.1 attempt rules\n\n```yaml\n- target: attempt\n ruleid: reject-critical-grounding\n priority: 1\n match:\n criteriontags: [hard-quality]\n verdicts: [not_met]\n severities: [critical]\n minimumcalibratedconfidence: 0.95\n action: reject\n terminal: true\n\n- target: attempt\n ruleid: rework-major\n priority: 2\n match:\n verdicts: [not_met]\n severities: [major]\n minimumcalibratedconfidence: 0.85\n action: rework\n terminal: false\n\n- target: attempt\n ruleid: escalate-unresolved\n priority: 3\n match:\n hasunresolvedfindings: true\n action: escalate\n terminal: true\n\n- target: attempt\n ruleid: accept-clean\n priority: 10\n match:\n criteriontags: [required-for-attempt-acceptance]\n verdicts: [met]\n minimumcalibratedconfidence: 0.90\n hasunresolvedfindings: false\n action: accept\n terminal: true\n```\n\n### 10.2 annotation rules\n\n```yaml\n- target: annotation\n ruleid: rework-unfaithful-review\n priority: 1\n match:\n criteriontags: [annotation-fidelity]\n verdicts: [not_met]\n severities: [major, critical]\n minimumcalibratedconfidence: 0.90\n action: rework\n terminal: true\n\n- target: annotation\n ruleid: accept-faithful-review\n priority: 10\n match:\n criteriontags: [annotation-fidelity]\n verdicts: [met]\n minimumcalibratedconfidence: 0.90\n hasunresolvedfindings: false\n action: accept\n terminal: true\n```\n\n### 10.3 composite rules\n\n```yaml\n- target: composite_case\n ruleid: escalate-any-target-uncertain\n priority: 1\n match:\n hasunresolvedfindings: true\n action: escalate\n terminal: true\n\n- target: composite_case\n ruleid: rework-bad-annotation\n priority: 2\n match:\n criterionids: [annotation-fidelity]\n verdicts: [not_met]\n severities: [major, critical]\n action: rework\n terminal: true\n\n- target: composite_case\n ruleid: accept-both-clean\n priority: 10\n match:\n criteriontags: [required-for-composite-acceptance]\n verdicts: [met]\n minimumcalibratedconfidence: 0.90\n hasunresolvedfindings: false\n action: accept\n terminal: true\n```\n\n## 11. policy trace requirements\n\nevery decision records:\n\n- target;\n- chosen action;\n- policy version;\n- triggered rules;\n- decisive criteria;\n- unresolved findings;\n- confidence;\n- full rule trace;\n- human-readable rationale;\n- decision time.\n\na useful policy trace must allow an auditor to answer:\n\n- which rule was evaluated?\n- what criterion state was supplied?\n- did it match?\n- why or why not?\n- which rule determined the action?\n- what would have changed the action?\n\n## 12. policy testing\n\nthe decision engine should have deterministic unit tests for:\n\n- each rule independently;\n- tie-break behavior;\n- no-rule default;\n- uncalibrated confidence;\n- not-applicable criteria;\n- ambiguous and unknown criteria;\n- positive-proof failure;\n- missing expected elements;\n- counterevidence;\n- noncompensable failures;\n- annotation isolation;\n- composite mapping;\n- human response updates;\n- invalid rule identifiers;\n- contract version mismatch;\n- calibration profile mismatch;\n- system error versus attempter failure.\n\ngolden policy tests should be run whenever the contract, policy engine, or schema changes.\n\n## 13. policy-version changes\n\ncreate a new policy version when changing:\n\n- action thresholds;\n- rule priority;\n- target mapping;\n- hard gates;\n- tie-break order;\n- confidence requirements;\n- default action;\n- human escalation conditions;\n- composite-case admission policy.\n\na policy-only change does not necessarily require rerunning claim and evidence extraction. it does require replaying the stored criterion state through the new policy and validating the operational effect on a locked set.\n\n## 14. initial default\n\nfor a new project without sufficient calibration:\n\n```text\ndeterministic clean pass/failure: project-defined automatic action\nsemantic clear finding: reviewer assistance\nsemantic uncalibrated decision: escalate\nsubjective disagreement: escalate or advisory\nreviewer performance consequence: adjudicated samples only\n```\n\nthe default action should remain `escalate` until benchmark gates justify selective autonomy.\n", "19": "foundation package - manifest foundation specifications machine-readable inventory and hashes for the complete package. foundation specification {\n \"package\": \"autoqa-foundation\",\n \"version\": \"0.1.0\",\n \"createdat\": \"2026-07-15t00:00:00z\",\n \"hashalgorithm\": \"sha256\",\n \"files\": [\n {\n \"path\": \"benchmark_protocol.md\",\n \"bytes\": 31399,\n \"sha256\": \"81ce1c84096d03351356626c45b1e1818dbc4ee566a53807c934aa4ccddb9a2c\"\n },\n {\n \"path\": \"changelog.md\",\n \"bytes\": 828,\n \"sha256\": \"69741186a45bd2f35bb6bd727d72906dd7e6779566339a69ac01c012f8c23473\"\n },\n {\n \"path\": \"contract_spec.md\",\n \"bytes\": 41201,\n \"sha256\": \"8bec40d6df3f9339a2fe98d533b1f87ce6463b1b516161dd9e20dd72f5c41796\"\n },\n {\n \"path\": \"decision_policy.md\",\n \"bytes\": 14038,\n \"sha256\": \"e96a072e0871118fd0f3119823216af09c97d657c51a14b0fbbd0f8f378bccd7\"\n },\n {\n \"path\": \"readme.md\",\n \"bytes\": 8130,\n \"sha256\": \"4486f3fb62cfefb0f3ac46764dd7db435d1943c57305c45b11eb6ed54cf8bb5d\"\n },\n {\n \"path\": \"version\",\n \"bytes\": 6,\n \"sha256\": \"e9dd8507f4bf0c6f42458e41aea833ad0bd3f6127272335eee9bf4d58541ed67\"\n },\n {\n \"path\": \"examples/benchmark-item.example.json\",\n \"bytes\": 6191,\n \"sha256\": \"b90b5a73e3d266e2326607ae67731133da2b9bb95874d68daeabdcefea596b19\"\n },\n {\n \"path\": \"examples/case-input.example.json\",\n \"bytes\": 4247,\n \"sha256\": \"e9fd3a97e157ef98c18cd2bfbd688d0d5d94b1b4f08a8441d23f950639dad37a\"\n },\n {\n \"path\": \"examples/evaluation-output.example.json\",\n \"bytes\": 33545,\n \"sha256\": \"d5115d1630249d2df204c192055df29de3963442cd6271faaf7e9607e99854e4\"\n },\n {\n \"path\": \"examples/human-resolution-packet.example.json\",\n \"bytes\": 2561,\n \"sha256\": \"61008558a6c2213fe6d1b254a7308c4bcfa673fbb448bbaf21543acb9817384e\"\n },\n {\n \"path\": \"examples/project-contract.example.json\",\n \"bytes\": 22557,\n \"sha256\": \"45167a91393b7346a287be4fd414cdc7a20cf776986980175c6da00f16c8d6e5\"\n },\n {\n \"path\": \"schemas/benchmark-item.schema.json\",\n \"bytes\": 7880,\n \"sha256\": \"0aeb486d5ca588dbb55a874e3549d2dfef73d07f71022830ac97c44ef6761721\"\n },\n {\n \"path\": \"schemas/case-input.schema.json\",\n \"bytes\": 8266,\n \"sha256\": \"b22e7ab67dd0ae0db9cdd0efdc1d41ff118c823e9079a2d36f8b1bb02947f135\"\n },\n {\n \"path\": \"schemas/common.schema.json\",\n \"bytes\": 32743,\n \"sha256\": \"3ba733b46b7d6053f7d6ab169756b11af13df88827aab1753e2dd9bee0c7f02c\"\n },\n {\n \"path\": \"schemas/evaluation-output.schema.json\",\n \"bytes\": 4104,\n \"sha256\": \"f83580c20c7c99a4927a114c3972e7bc662bd22bfb5983047254bf5fda19b471\"\n },\n {\n \"path\": \"schemas/human-resolution-packet.schema.json\",\n \"bytes\": 955,\n \"sha256\": \"8f460328039bde863bcbed1c15cfa9f83925f2b89375375864f849be35d8496e\"\n },\n {\n \"path\": \"schemas/project-contract.schema.json\",\n \"bytes\": 23301,\n \"sha256\": \"d926d9c48a6cd85eacc8e5eb2f6fc921873a5c3faffdcce946e0f3a8b0878516\"\n },\n {\n \"path\": \"templates/criterion-authoring-template.csv\",\n \"bytes\": 552,\n \"sha256\": \"273d0a921339787224982d81dce13c6348a6a30694167176e09ae078064f11e9\"\n },\n {\n \"path\": \"templates/gold-set-template.csv\",\n \"bytes\": 593,\n \"sha256\": \"fe10577fe0dcb71ae5aad388468c71b08bdddf1fe9bdb57f9ab0df111eb1df8f\"\n },\n {\n \"path\": \"templates/model-bakeoff-template.csv\",\n \"bytes\": 582,\n \"sha256\": \"271c3846fbe93c0d2b2d96eac3d64653f96cb787d160508076c247113218dedf\"\n },\n {\n \"path\": \"types/autoqa.ts\",\n \"bytes\": 21216,\n \"sha256\": \"e714bd9ceb52df39729bdf55af1c5a095962015bafb7dc6c4858a52d6bd91049\"\n },\n {\n \"path\": \"types/tsconfig.json\",\n \"bytes\": 207,\n \"sha256\": \"7bcf3373a17cf19ade5c59975ed0ec60ed9081b6258574daf0165b5dcfc3fc73\"\n },\n {\n \"path\": \"validation/hash_utils.py\",\n \"bytes\": 1074,\n \"sha256\": \"801243a09e128c66163ac8edd0fbb26ba361b66518404eb5147d54b41dcc30d7\"\n },\n {\n \"path\": \"validation/validate_examples.py\",\n \"bytes\": 2176,\n \"sha256\": \"a25475434cf0f528a17cff088712c3c4c2097ce10d7b9d2bf77d04c5fc5724f5\"\n },\n {\n \"path\": \"validation/validate_package.py\",\n \"bytes\": 20165,\n \"sha256\": \"0e376b3588215043540dec575ce8bac4d71e1b2ebabe17b28c2869b352035e9e\"\n }\n ]\n}\n", "20": "autoqa foundation 0.1.0\n\na project-agnostic contract and benchmark package for evaluating human-produced ai-training data and the human annotations applied to it.\n\nthe package defines the stable domain boundary for an autoqa system. it intentionally does **not** prescribe a model provider, orchestration framework, database, or vercel implementation. those components can change while the evaluation objects, proof obligations, decision policy, and benchmark remain versioned and auditable.\n\n## core model\n\nautoqa evaluates two different objects:\n\n1. **the attempt** - whether the human-produced work satisfies the project's instructions.\n2. **the annotation** - whether the human reviewer's label, rationale, cited evidence, criterion attribution, and severity faithfully describe that attempt.\n\nit does not reduce evaluation to a single score. the core unit is a criterion assertion:\n\n```text\ncriterion c applies.\nattempt span a exhibits behavior b.\nevidence e supports, contradicts, or is insufficient for b.\nb does or does not satisfy c.\nthe project decision policy assigns consequence s.\n```\n\n## non-negotiable invariants\n\na conforming implementation preserves these distinctions:\n\n- truth versus support by the permitted evidence;\n- source quality versus entailment;\n- relevance versus correctness;\n- factual accuracy versus inferential warrant;\n- positive proof versus absence of a detected defect;\n- attempt quality versus annotation fidelity;\n- criterion verdict versus severity;\n- severity versus operational action;\n- raw model confidence versus empirically calibrated confidence;\n- item validity versus marginal dataset value.\n\nthe engine may be project-agnostic. its judgments may not be criterion-agnostic. every production project must supply an approved, versioned evaluation contract.\n\n## package contents\n\n```text\n.\n+-- readme.md\n+-- contract_spec.md\n+-- benchmark_protocol.md\n+-- decision_policy.md\n+-- changelog.md\n+-- version\n+-- schemas/\n| +-- common.schema.json\n| +-- project-contract.schema.json\n| +-- case-input.schema.json\n| +-- evaluation-output.schema.json\n| +-- benchmark-item.schema.json\n| +-- human-resolution-packet.schema.json\n+-- types/\n| +-- autoqa.ts\n| +-- tsconfig.json\n+-- examples/\n| +-- project-contract.example.json\n| +-- case-input.example.json\n| +-- evaluation-output.example.json\n| +-- benchmark-item.example.json\n| +-- human-resolution-packet.example.json\n+-- templates/\n| +-- criterion-authoring-template.csv\n| +-- gold-set-template.csv\n| +-- model-bakeoff-template.csv\n+-- validation/\n +-- hash_utils.py\n +-- validate_examples.py\n +-- validate_package.py\n```\n\n### primary documents\n\n- **`contract_spec.md`** defines the normative semantics, lifecycle, invariants, and conformance requirements.\n- **`benchmark_protocol.md`** defines how to build adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.\n- **`decision_policy.md`** defines the recommended rule engine, action semantics, human-resolution trigger, and policy trace.\n\n### machine-readable layer\n\n- **project contract**: the approved interpretation of project instructions.\n- **case input**: the task, evidence, attempt, optional reference, and optional human review.\n- **evaluation output**: claims, expected elements, evidence relationships, findings, criterion assessments, annotation audit, decision trace, and feedback.\n- **benchmark item**: adjudicated criterion truth, target-specific acceptable actions, ambiguity status, challenge attributes, and leakage controls.\n- **human-resolution packet**: the one bounded, decision-changing question and its evidence context, with an optional response.\n\nruntime input and output validation should use the json schemas. typescript interfaces are provided for application development but are not a runtime trust boundary.\n\n## quick start\n\nvalidate the included examples:\n\n```bash\npython validation/validate_package.py\n```\n\ntype-check the typescript domain model:\n\n```bash\ncd types\ntsc -p tsconfig.json\n```\n\nrecommended adoption sequence:\n\n1. copy `examples/project-contract.example.json` and replace the synthetic instructions and criteria.\n2. author each criterion in `templates/criterion-authoring-template.csv` before converting it to json.\n3. run a contract review with the project owner, a domain expert, and a qa owner.\n4. create representative `case_input` documents from historical production work.\n5. independently adjudicate a calibration and test set using `benchmark_protocol.md`.\n6. compare evaluator configurations by criterion family rather than choosing one universal judge.\n7. calibrate confidence and set project-specific action gates.\n8. shadow the system before allowing autonomous acceptance or rejection.\n9. preserve every model, prompt, source, rubric, and policy version required for replay.\n\n## minimal evaluator pipeline\n\n```text\n1. validate and freeze inputs\n2. compile/load the approved project contract\n3. run deterministic integrity and compliance checks\n4. extract claims and expected elements\n5. map claims and requirements to evidence\n6. generate positive, negative, omission, and warrant observations\n7. challenge material observations\n8. adjudicate each criterion\n9. evaluate global qualities\n10. audit the human annotation independently\n11. apply the versioned decision policy\n12. ask at most one decision-changing human question when necessary\n13. generate and validate audience-specific feedback\n14. persist the complete audit record\n```\n\n## conformance levels\n\n### level a - structured evaluation\n\nthe implementation validates inputs and outputs, uses an approved project contract, produces criterion-level assessments, and records evidence-linked findings.\n\n### level b - calibrated selective evaluation\n\nlevel a plus empirical confidence calibration, risk-coverage reporting, explicit abstention/escalation, and locked test suites.\n\n### level c - governed production autoqa\n\nlevel b plus independent annotation auditing, one-interaction human resolution, shadow deployment, drift monitoring, replay, appeals, and restricted use of reviewer analytics.\n\nautonomous acceptance or rejection should not be enabled below level b.\n\n## example scenario\n\nthe included example contains a high-quality attempt that correctly concludes that the evidence does not establish a primary causal relationship. a human reviewer fails it for not choosing a forced yes/no answer. the project contract explicitly permits an inconclusive conclusion. the example evaluation therefore:\n\n- affirms the attempt's directness, grounding, warrant, and completeness;\n- accepts the attempt;\n- returns the annotation and composite case for rework;\n- marks the reviewer annotation as unsupported;\n- gives the attempter specific positive feedback;\n- gives the reviewer a minimal corrective explanation;\n- records a project-owner signal in case this misunderstanding recurs.\n\nthis demonstrates why attempt quality and annotation fidelity must be separate outputs.\n\n## versioning rule\n\nthree versions may change independently:\n\n- **schema version** - shape and semantics of machine-readable documents.\n- **contract version** - project instructions, criteria, anchors, or evidence policy.\n- **decision-policy version** - operational mapping from criterion state to action.\n\na case must reference an exact contract version and content hash. an evaluation must record the exact case, contract, prompt, source snapshots, model runs, calibration profile, and decision-policy version used.\n\n## what 0.1.0 deliberately leaves open\n\n- model and provider selection;\n- retrieval architecture;\n- prompt templates;\n- long-context chunking strategy;\n- calibration algorithm;\n- reviewer-competence model;\n- dataset-level value model;\n- ui design;\n- persistence and job orchestration.\n\nthese should be decided empirically after the contract and benchmark are stable.\n", "21": "foundation package - version foundation specifications current package version identifier. foundation specification 0.1.0\n", "22": "worked example - benchmark item foundation examples adjudicated benchmark record for the synthetic case, including acceptable outcomes and ambiguity metadata. example worked case {\n \"documenttype\": \"benchmark_item\",\n \"schemaversion\": \"0.1.0\",\n \"benchmarkitemid\": \"benchmark-example-001\",\n \"benchmarkversion\": \"0.1.0\",\n \"split\": \"challenge\",\n \"contractref\": {\n \"contractid\": \"example-grounded-analysis\",\n \"contractversion\": \"0.1.0\",\n \"contenthash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\"\n },\n \"caseref\": {\n \"caseid\": \"case-example-001\",\n \"contenthash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\"\n },\n \"slicetags\": [\n \"causal-warrant\",\n \"positive-attempt\",\n \"reviewer-false-reject\"\n ],\n \"challengeattributes\": [\n \"strong_positive\",\n \"wrong_label_correct_observation\",\n \"severity_edge\"\n ],\n \"goldstandard\": {\n \"criteriongold\": [\n {\n \"criterionid\": \"c1-direct-response\",\n \"applicability\": \"applies\",\n \"acceptableverdicts\": [\n \"met\"\n ],\n \"severity\": \"none\",\n \"positiveevidencespans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"negativeevidencespans\": [],\n \"omissiondescriptions\": [],\n \"notes\": \"the calibrated conclusion directly answers the question.\"\n },\n {\n \"criterionid\": \"c2-grounding\",\n \"applicability\": \"applies\",\n \"acceptableverdicts\": [\n \"met\"\n ],\n \"severity\": \"none\",\n \"positiveevidencespans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n },\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e2 estimates a 4% improvement, but its interval includes zero.\",\n \"locator\": \"sentence 3\"\n }\n ],\n \"negativeevidencespans\": [],\n \"omissiondescriptions\": [],\n \"notes\": \"material factual claims are supported by e1 and e2.\"\n },\n {\n \"criterionid\": \"c3-warrant\",\n \"applicability\": \"applies\",\n \"acceptableverdicts\": [\n \"met\"\n ],\n \"severity\": \"none\",\n \"positiveevidencespans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"negativeevidencespans\": [],\n \"omissiondescriptions\": [],\n \"notes\": \"conclusion strength is proportionate to study design and uncertainty.\"\n },\n {\n \"criterionid\": \"c4-completeness\",\n \"applicability\": \"applies\",\n \"acceptableverdicts\": [\n \"met\"\n ],\n \"severity\": \"none\",\n \"positiveevidencespans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n },\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e2 estimates a 4% improvement, but its interval includes zero.\",\n \"locator\": \"sentence 3\"\n }\n ],\n \"negativeevidencespans\": [],\n \"omissiondescriptions\": [],\n \"notes\": \"both studies are addressed.\"\n },\n {\n \"criterionid\": \"c5-annotation-fidelity\",\n \"applicability\": \"applies\",\n \"acceptableverdicts\": [\n \"not_met\"\n ],\n \"severity\": \"major\",\n \"positiveevidencespans\": [],\n \"negativeevidencespans\": [\n {\n \"artifacttype\": \"human_review\",\n \"artifactid\": \"review-example-001\",\n \"quotedtext\": \"the attempter did not choose yes or no and therefore did not answer the question directly.\",\n \"locator\": \"rationale\"\n }\n ],\n \"omissiondescriptions\": [\n \"the review omits the contract language permitting an inconclusive answer.\"\n ],\n \"notes\": \"the review applies an unstated binary-answer preference.\"\n }\n ],\n \"annotationgold\": {\n \"acceptablelabelsupport\": [\n \"unsupported\"\n ],\n \"expecteddisagreementtypes\": [\n \"criterion_interpretation\",\n \"threshold\"\n ],\n \"notes\": \"accept the attempt and flag the reviewer annotation.\"\n },\n \"ambiguitystatus\": \"clear\",\n \"validalternativenotes\": [\n \"a qualified ", "23": "worked example - case input foundation examples a strong attempt incorrectly failed by a human reviewer for not forcing an unsupported binary conclusion. example worked case {\n \"documenttype\": \"case_input\",\n \"schemaversion\": \"0.1.0\",\n \"caseid\": \"case-example-001\",\n \"createdat\": \"2026-07-15t12:10:00z\",\n \"contractref\": {\n \"contractid\": \"example-grounded-analysis\",\n \"contractversion\": \"0.1.0\",\n \"contenthash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\"\n },\n \"task\": {\n \"taskid\": \"task-example-001\",\n \"tasktype\": \"evidence_grounded_analysis\",\n \"prompt\": \"using evidence e1 and e2, assess whether initiative x is the primary cause of the observed improvement in outcome y. give a concise conclusion and explain the evidence.\",\n \"taskinstructions\": [],\n \"expectedoutputformat\": \"one concise analytical paragraph with evidence identifiers.\",\n \"contenthash\": \"sha256:deae115ef5c75b4b125e4dbb09937257231c21abcefcfb6b9c024804a1d14ec6\",\n \"metadata\": {\n \"requirescausalassessment\": true\n }\n },\n \"evidencebundle\": [\n {\n \"evidenceid\": \"e1\",\n \"evidenceclassid\": \"supplied-study\",\n \"title\": \"observational comparison\",\n \"content\": \"an observational comparison found outcome y was 18% higher in units using initiative x. the authors state that units self-selected into the initiative and that the study does not identify a causal effect.\",\n \"contenthash\": \"sha256:192359f876334c9282c7e79247a6b78b913f63cfb032b68d8c0f7466e6cdce82\",\n \"authorityrank\": 3,\n \"permitted\": true,\n \"capturedat\": \"2026-07-15t12:05:00z\",\n \"metadata\": {\n \"studydesign\": \"observational\"\n }\n },\n {\n \"evidenceid\": \"e2\",\n \"evidenceclassid\": \"supplied-study\",\n \"title\": \"randomized pilot\",\n \"content\": \"a randomized pilot estimated a 4% improvement in outcome y. the confidence interval included zero, so the pilot did not establish a statistically distinguishable effect.\",\n \"contenthash\": \"sha256:b3bba4e3b5b48b3ace0ec78b81738a837464d94d945689b57065d2f8f50c4428\",\n \"authorityrank\": 3,\n \"permitted\": true,\n \"capturedat\": \"2026-07-15t12:05:00z\",\n \"metadata\": {\n \"studydesign\": \"randomized_pilot\"\n }\n }\n ],\n \"attempt\": {\n \"attemptid\": \"attempt-example-001\",\n \"authorid\": \"worker-104\",\n \"content\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause. e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification. e2 estimates a 4% improvement, but its interval includes zero. the strongest justified conclusion is that x may contribute to the improvement; primacy remains unproven.\",\n \"contenthash\": \"sha256:8d0c8290f24f40ae2ca136a7556dbaa2938591a756497506d927d6fc28d43a64\",\n \"submittedat\": \"2026-07-15t12:08:00z\",\n \"attachments\": [],\n \"tooltraces\": []\n },\n \"humanreview\": {\n \"annotationid\": \"review-example-001\",\n \"reviewerid\": \"reviewer-207\",\n \"label\": \"fail\",\n \"score\": 0,\n \"reasoncodes\": [\n \"not_direct_answer\"\n ],\n \"rationale\": \"fail. the attempter did not choose yes or no and therefore did not answer the question directly.\",\n \"criterionannotations\": [\n {\n \"criterionid\": \"c1-direct-response\",\n \"verdict\": \"not_met\",\n \"severity\": \"major\",\n \"rationale\": \"the response did not choose yes or no.\",\n \"citedspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"the supplied evidence does not establish that it is the primary cause\",\n \"locator\": \"sentence 1\"\n }\n ]\n }\n ],\n \"reviewerconfidence\": {\n \"rawscore\": 0.9,\n \"band\": \"uncalibrated\"\n },\n \"submittedat\": \"2026-07-15t12:09:00z\"\n },\n \"referenceanswers\": [\n {\n \"referenceid\": \"reference-example-001\",\n \"role\": \"illustrative\",\n \"content\": \"the evidence suggests a possible contribution from x but does not establish that x is the primary cause.\",\n \"contenthash\": \"sha256:83260b932b3b8b56d7785dc4ff7d859c2cc77426b927cba51ae6f2571d9c04d0\"\n }\n ],\n \"declaredexpectedelements\": [],\n \"casetags\": [\n \"synthetic\",\n \"positive-attempt\",\n \"incorrect-human-fail\",\n \"causal-warrant\"\n ],\n \"metadata\": {\n \"example\": true\n }\n}\n", "24": "worked example - evaluation output foundation examples complete autoqa result accepting the attempt, returning the annotation for rework, and generating audience-specific feedback. example worked case {\n \"documenttype\": \"evaluation_output\",\n \"schemaversion\": \"0.1.0\",\n \"evaluationid\": \"evaluation-example-001\",\n \"evaluationversion\": \"0.1.0\",\n \"caseref\": {\n \"caseid\": \"case-example-001\",\n \"contenthash\": \"sha256:ca55fe0d72675ee41a245b10416355f9be32c04f613a51cf59b9c3fc2aa9dda9\"\n },\n \"contractref\": {\n \"contractid\": \"example-grounded-analysis\",\n \"contractversion\": \"0.1.0\",\n \"contenthash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\"\n },\n \"createdat\": \"2026-07-15t12:12:00z\",\n \"status\": \"final\",\n \"runmanifest\": [\n {\n \"modelrunid\": \"run-claim-001\",\n \"role\": \"claim_extractor\",\n \"provider\": \"example-provider-a\",\n \"modelname\": \"example-model-fast\",\n \"modelversion\": \"2026-06\",\n \"prompttemplateid\": \"claim-extractor-v1\",\n \"prompttemplateversion\": \"0.1.0\",\n \"prompttemplatehash\": \"sha256:c0134a01b0c26efe652d790a8e07662ef46052dc3b92fb183ab90e17c45729b7\",\n \"parameters\": {\n \"temperature\": 0\n },\n \"parentrunids\": [],\n \"inputhash\": \"sha256:204ff6a20394ccad26982cd41cb7aa5a9f03133a3fb774baea728084e2a5a5d1\",\n \"outputhash\": \"sha256:fa1a14caa9781c43f098091c180ece6044fec851dee6c99b22f9315ff2ec42d1\",\n \"startedat\": \"2026-07-15t12:10:05z\",\n \"completedat\": \"2026-07-15t12:10:06z\"\n },\n {\n \"modelrunid\": \"run-adjudicator-001\",\n \"role\": \"criterion_adjudicator\",\n \"provider\": \"example-provider-b\",\n \"modelname\": \"example-model-reasoning\",\n \"modelversion\": \"2026-06\",\n \"prompttemplateid\": \"criterion-adjudicator-v1\",\n \"prompttemplateversion\": \"0.1.0\",\n \"prompttemplatehash\": \"sha256:c0134a01b0c26efe652d790a8e07662ef46052dc3b92fb183ab90e17c45729b7\",\n \"parameters\": {\n \"temperature\": 0\n },\n \"parentrunids\": [\n \"run-claim-001\"\n ],\n \"inputhash\": \"sha256:50cf3c2562bbc06499085ca8f13ace7dc99463a30ed0098dcfdcc513ba690867\",\n \"outputhash\": \"sha256:1da443352eb7fbdc1a924549753603adde1205ac33f3ff8ee91eb9bca1df9143\",\n \"startedat\": \"2026-07-15t12:10:07z\",\n \"completedat\": \"2026-07-15t12:10:12z\"\n }\n ],\n \"integritystatus\": \"clean\",\n \"findings\": [\n {\n \"findingid\": \"f1-direct-positive\",\n \"findingtype\": \"positive\",\n \"criterionid\": \"c1-direct-response\",\n \"summary\": \"the attempt directly states that primacy is not established.\",\n \"detail\": \"an inconclusive but calibrated answer is explicitly permitted by the contract.\",\n \"claimids\": [\n \"cl1\"\n ],\n \"expectedelementids\": [\n \"ee1\"\n ],\n \"targetspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"evidenceids\": [\n \"e1\",\n \"e2\"\n ],\n \"relationship\": \"reasonably_supports\",\n \"materiality\": \"decisive\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"status\": \"confirmed\",\n \"source\": {\n \"kind\": \"hybrid\",\n \"componentid\": \"criterion-pipeline-v1\",\n \"modelrunid\": \"run-adjudicator-001\"\n }\n },\n {\n \"findingid\": \"f2-grounding-positive\",\n \"findingtype\": \"positive\",\n \"criterionid\": \"c2-grounding\",\n \"summary\": \"the material study claims match e1 and e2.\",\n \"detail\": \"the reported effect sizes and limitations are accurately attributed.\",\n \"claimids\": [\n \"cl2\",\n \"cl3\"\n ],\n \"expectedelementids\": [],\n \"targetspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n },\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e2 estimates a 4% improvement, but its interval includes zero.\",\n \"locator\": \"sentence 3\"\n }\n ],\n \"evidenceids\": [\n \"e1\",\n \"e2\"\n ],\n \"relationship\": \"directly_entails\",\n \"materiality\": \"central\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"status\": \"confirmed\",\n \"source\": {\n \"kind\": \"hybrid\",\n \"componentid\": \"grounding-pipeline-v1\",\n \"modelrunid\": \"run-adjudicator-001\"\n }\n },\n {\n \"findingid\": \"f3-warrant-positive\",\n \"findingtype\": \"positive\",\n \"criterionid\": \"c3-warrant\",\n \"summary\": \"the conclusion appropriately distinguishes association, possible contribution, and unproven primacy.\",\n \"detail\": \"the conclusion is proportionate to an observational association plus an inconclusive randomized pilot.\",\n \"claimids\": [\n \"cl1\",\n \"cl4\"\n ],\n \"expectedelementids\": [\n \"ee2\"\n ],\n \"targetspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"evidenceids\": [\n \"e1\",\n \"e2\"\n ],\n \"relationship\": \"reasonably_supports\",\n \"materiality\": \"decisive\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"status\": \"confirmed\",\n \"source\": {\n \"kind\": \"llm_judge\",\n \"componentid\": \"warrant-auditor-v1\",\n \"modelrunid\": \"run-adjudicator-001\"\n }\n },\n {\n \"findingid\": \"f4-completeness-positive\",\n \"findingtype\": \"positive\",\n \"criterionid\": \"c4-completeness\",\n \"summary\": \"both supplied studies are materially addressed.\",\n \"detail\": \"the attempt states the key result and limitation of each study.\",\n \"claimids\": [\n \"cl2\",\n \"cl3\"\n ],\n \"expectedelementids\": [\n \"ee3\",\n \"ee4\"\n ],\n \"targetspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n },\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e2 estimates a 4% improvement, but its interval includes zero.\",\n \"locator\": \"sentence 3\"\n }\n ],\n \"evidenceids\": [\n \"e1\",\n \"e2\"\n ],\n \"relationship\": \"directly_entails\",\n \"materiality\": \"central\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"status\": \"confirmed\",\n \"source\": {\n \"kind\": \"hybrid\",\n \"componentid\": \"coverage-auditor-v1\",\n \"modelrunid\": \"run-adjudicator-001\"\n }\n },\n {\n \"findingid\": \"f5-review-negative\",\n \"findingtype\": \"negative\",\n \"criterionid\": \"c5-annotation-fidelity\",\n \"summary\": \"the reviewer applies an unstated binary-answer requirement that conflicts with the contract.\",\n \"detail\": \"the response directly answers that primacy is unproven, and the instructions explicitly permit an inconclusive conclusion.\",\n \"claimids\": [],\n \"expectedelementids\": [],\n \"targetspans\": [\n {\n \"artifacttype\": \"human_review\",\n \"artifactid\": \"review-example-001\",\n \"quotedtext\": \"the attempter did not choose yes or no and therefore did not answer the question directly.\",\n \"locator\": \"rationale\"\n },\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"evidenceids\": [],\n \"relationship\": \"contradicts\",\n \"materiality\": \"decisive\",\n \"severity\": \"major\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"status\": \"confirmed\",\n \"source\": {\n \"kind\": \"llm_judge\",\n \"componentid\": \"annotation-auditor-v1\",\n \"modelrunid\": \"run-adjudicator-001\"\n }\n }\n ],\n \"claims\": [\n {\n \"claimid\": \"cl1\",\n \"proposition\": \"the supplied evidence does not establish that initiative x is the primary cause of the improvement in outcome y.\",\n \"claimtype\": \"interpretive\",\n \"importance\": \"central\",\n \"explicitness\": \"explicit\",\n \"attemptspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"qualifiers\": [\n \"does not establish\",\n \"primary cause\"\n ],\n \"citedevidenceids\": [\n \"e1\",\n \"e2\"\n ]\n },\n {\n \"claimid\": \"cl2\",\n \"proposition\": \"e1 reports an 18% observational difference and notes self-selection and no causal identification.\",\n \"claimtype\": \"factual\",\n \"importance\": \"supporting\",\n \"explicitness\": \"explicit\",\n \"attemptspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n }\n ],\n \"qualifiers\": [\n \"observational\"\n ],\n \"citedevidenceids\": [\n \"e1\"\n ]\n },\n {\n \"claimid\": \"cl3\",\n \"proposition\": \"e2 estimates a 4% improvement and its confidence interval includes zero.\",\n \"claimtype\": \"factual\",\n \"importance\": \"supporting\",\n \"explicitness\": \"explicit\",\n \"attemptspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e2 estimates a 4% improvement, but its interval includes zero.\",\n \"locator\": \"sentence 3\"\n }\n ],\n \"qualifiers\": [],\n \"citedevidenceids\": [\n \"e2\"\n ]\n },\n {\n \"claimid\": \"cl4\",\n \"proposition\": \"initiative x may contribute to the improvement.\",\n \"claimtype\": \"interpretive\",\n \"importance\": \"central\",\n \"explicitness\": \"explicit\",\n \"attemptspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"x may contribute to the improvement; primacy remains unproven.\",\n \"locator\": \"sentence 4\"\n }\n ],\n \"qualifiers\": [\n \"may\"\n ],\n \"citedevidenceids\": [\n \"e1\",\n \"e2\"\n ]\n }\n ],\n \"expectedelements\": [\n {\n \"elementid\": \"ee1\",\n \"criterionid\": \"c1-direct-response\",\n \"description\": \"a direct conclusion about whether x is established as the primary cause.\",\n \"required\": true,\n \"importance\": \"decisive\",\n \"acceptablealternatives\": [\n \"primacy is not established.\",\n \"the evidence is insufficient to determine primacy.\"\n ],\n \"observedspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n }\n ],\n \"coverage\": \"complete\"\n },\n {\n \"elementid\": \"ee2\",\n \"criterionid\": \"c3-warrant\",\n \"description\": \"a distinction between association and causal or primary-cause inference.\",\n \"required\": true,\n \"importance\": \"decisive\",\n \"acceptablealternatives\": [],\n \"observedspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"initiative x is associated with higher outcome y, but the supplied evidence does not establish that it is the primary cause.\",\n \"locator\": \"sentence 1\"\n },\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n }\n ],\n \"coverage\": \"complete\"\n },\n {\n \"elementid\": \"ee3\",\n \"criterionid\": \"c4-completeness\",\n \"description\": \"material treatment of e1.\",\n \"required\": true,\n \"importance\": \"central\",\n \"acceptablealternatives\": [],\n \"observedspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e1 reports an 18% observational difference while explicitly noting self-selection and no causal identification.\",\n \"locator\": \"sentence 2\"\n }\n ],\n \"coverage\": \"complete\"\n },\n {\n \"elementid\": \"ee4\",\n \"criterionid\": \"c4-completeness\",\n \"description\": \"material treatment of e2.\",\n \"required\": true,\n \"importance\": \"central\",\n \"acceptablealternatives\": [],\n \"observedspans\": [\n {\n \"artifacttype\": \"attempt\",\n \"artifactid\": \"attempt-example-001\",\n \"quotedtext\": \"e2 estimates a 4% improvement, but its interval includes zero.\",\n \"locator\": \"sentence 3\"\n }\n ],\n \"coverage\": \"complete\"\n }\n ],\n \"evidencerelationships\": [\n {\n \"relationshipid\": \"er1\",\n \"subjecttype\": \"claim\",\n \"subjectid\": \"cl2\",\n \"evidenceid\": \"e1\",\n \"relationship\": \"directly_entails\",\n \"sourcequality\": \"high\",\n \"temporalvalidity\": \"not_time_sensitive\",\n \"evidencespans\": [\n {\n \"artifacttype\": \"evidence\",\n \"artifactid\": \"e1\",\n \"quotedtext\": \"outcome y was 18% higher in units using initiative x. the authors state that units self-selected into the initiative and that the study does not identify a causal effect.\",\n \"locator\": \"entire excerpt\"\n }\n ],\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"notes\": \"the attempt accurately reports both the numerical result and the study-design limitation.\"\n },\n {\n \"relationshipid\": \"er2\",\n \"subjecttype\": \"claim\",\n \"subjectid\": \"cl3\",\n \"evidenceid\": \"e2\",\n \"relationship\": \"directly_entails\",\n \"sourcequality\": \"high\",\n \"temporalvalidity\": \"not_time_sensitive\",\n \"evidencespans\": [\n {\n \"artifacttype\": \"evidence\",\n \"artifactid\": \"e2\",\n \"quotedtext\": \"a randomized pilot estimated a 4% improvement in outcome y. the confidence interval included zero\",\n \"locator\": \"sentences 1-2\"\n }\n ],\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"notes\": \"the attempt accurately reports the estimate and uncertainty.\"\n },\n {\n \"relationshipid\": \"er3\",\n \"subjecttype\": \"claim\",\n \"subjectid\": \"cl1\",\n \"evidenceid\": \"e1\",\n \"relationship\": \"reasonably_supports\",\n \"sourcequality\": \"high\",\n \"temporalvalidity\": \"not_time_sensitive\",\n \"evidencespans\": [\n {\n \"artifacttype\": \"evidence\",\n \"artifactid\": \"e1\",\n \"quotedtext\": \"outcome y was 18% higher in units using initiative x. the authors state that units self-selected into the initiative and that the study does not identify a causal effect.\",\n \"locator\": \"entire excerpt\"\n }\n ],\n \"confidence\": {\n \"rawscore\": 0.82,\n \"calibratedscore\": 0.79,\n \"band\": \"medium\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"notes\": \"e1 explicitly cannot identify causation, supporting the conclusion that primacy is not established.\"\n },\n {\n \"relationshipid\": \"er4\",\n \"subjecttype\": \"claim\",\n \"subjectid\": \"cl1\",\n \"evidenceid\": \"e2\",\n \"relationship\": \"reasonably_supports\",\n \"sourcequality\": \"high\",\n \"temporalvalidity\": \"not_time_sensitive\",\n \"evidencespans\": [\n {\n \"artifacttype\": \"evidence\",\n \"artifactid\": \"e2\",\n \"quotedtext\": \"a randomized pilot estimated a 4% improvement in outcome y. the confidence interval included zero\",\n \"locator\": \"sentences 1-2\"\n }\n ],\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"notes\": \"the randomized pilot is inconclusive and therefore does not establish primacy.\"\n }\n ],\n \"criterionassessments\": [\n {\n \"criterionid\": \"c1-direct-response\",\n \"applicability\": \"applies\",\n \"positivefindingids\": [\n \"f1-direct-positive\"\n ],\n \"negativefindingids\": [],\n \"omissionfindingids\": [],\n \"counterevidencefindingids\": [],\n \"unresolvedfindingids\": [],\n \"groundingstatus\": \"grounded\",\n \"truthstatus\": \"not_evaluated\",\n \"warrantstatus\": \"justified\",\n \"relevancestatus\": \"relevant\",\n \"verdict\": \"met\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"decisioneffect\": \"none\",\n \"rationale\": \"the first sentence directly answers the primary-cause question with the calibrated conclusion permitted by the contract.\"\n },\n {\n \"criterionid\": \"c2-grounding\",\n \"applicability\": \"applies\",\n \"positivefindingids\": [\n \"f2-grounding-positive\"\n ],\n \"negativefindingids\": [],\n \"omissionfindingids\": [],\n \"counterevidencefindingids\": [],\n \"unresolvedfindingids\": [],\n \"groundingstatus\": \"grounded\",\n \"truthstatus\": \"verified_true\",\n \"warrantstatus\": \"justified\",\n \"relevancestatus\": \"relevant\",\n \"verdict\": \"met\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"decisioneffect\": \"none\",\n \"rationale\": \"the factual statements about both studies are directly supported by the supplied excerpts.\"\n },\n {\n \"criterionid\": \"c3-warrant\",\n \"applicability\": \"applies\",\n \"positivefindingids\": [\n \"f3-warrant-positive\"\n ],\n \"negativefindingids\": [],\n \"omissionfindingids\": [],\n \"counterevidencefindingids\": [],\n \"unresolvedfindingids\": [],\n \"groundingstatus\": \"grounded\",\n \"truthstatus\": \"not_evaluated\",\n \"warrantstatus\": \"justified\",\n \"relevancestatus\": \"relevant\",\n \"verdict\": \"met\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"decisioneffect\": \"none\",\n \"rationale\": \"the attempt does not infer causation from e1 and treats e2 as inconclusive, so its conclusion strength matches the evidence.\"\n },\n {\n \"criterionid\": \"c4-completeness\",\n \"applicability\": \"applies\",\n \"positivefindingids\": [\n \"f4-completeness-positive\"\n ],\n \"negativefindingids\": [],\n \"omissionfindingids\": [],\n \"counterevidencefindingids\": [],\n \"unresolvedfindingids\": [],\n \"groundingstatus\": \"grounded\",\n \"truthstatus\": \"verified_true\",\n \"warrantstatus\": \"justified\",\n \"relevancestatus\": \"relevant\",\n \"verdict\": \"met\",\n \"severity\": \"none\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"decisioneffect\": \"none\",\n \"rationale\": \"both studies are materially summarized and synthesized.\"\n },\n {\n \"criterionid\": \"c5-annotation-fidelity\",\n \"applicability\": \"applies\",\n \"positivefindingids\": [],\n \"negativefindingids\": [\n \"f5-review-negative\"\n ],\n \"omissionfindingids\": [],\n \"counterevidencefindingids\": [],\n \"unresolvedfindingids\": [],\n \"groundingstatus\": \"grounded\",\n \"truthstatus\": \"verified_false\",\n \"warrantstatus\": \"invalid\",\n \"relevancestatus\": \"relevant\",\n \"verdict\": \"not_met\",\n \"severity\": \"major\",\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"decisioneffect\": \"accept_with_note\",\n \"rationale\": \"the reviewer fails the attempt for not choosing a binary answer even though the contract explicitly permits an inconclusive conclusion.\"\n }\n ],\n \"globalassessments\": [\n {\n \"axisid\": \"global-synthesis\",\n \"verdict\": \"met\",\n \"findingids\": [\n \"f1-direct-positive\",\n \"f3-warrant-positive\",\n \"f4-completeness-positive\"\n ],\n \"score\": 4,\n \"confidence\": {\n \"rawscore\": 0.82,\n \"calibratedscore\": 0.79,\n \"band\": \"medium\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"rationale\": \"the response combines the study designs and uncertainty into one calibrated conclusion rather than listing findings mechanically.\"\n }\n ],\n \"annotationassessment\": {\n \"annotationpresent\": true,\n \"labelsupport\": \"unsupported\",\n \"rationalesupport\": \"does_not_support_label\",\n \"criterionattribution\": \"unstated_preference\",\n \"evidencealignment\": \"partially_aligned\",\n \"severityalignment\": \"too_severe\",\n \"omittedpositivefindingids\": [\n \"f1-direct-positive\",\n \"f2-grounding-positive\",\n \"f3-warrant-positive\",\n \"f4-completeness-positive\"\n ],\n \"omittednegativefindingids\": [],\n \"unsupportedreviewerfindingids\": [\n \"f5-review-negative\"\n ],\n \"disagreementtypes\": [\n \"criterion_interpretation\",\n \"threshold\"\n ],\n \"confidence\": {\n \"rawscore\": 0.94,\n \"calibratedscore\": 0.91,\n \"band\": \"high\",\n \"calibrationprofileid\": \"calibration-example-v1\"\n },\n \"rationale\": \"the reviewer noticed the qualified conclusion but treated qualification as nonresponsiveness, contrary to the written criterion and pass anchor.\"\n },\n \"feedback\": [\n {\n \"feedbackid\": \"fb1-attempter-strength\",\n \"audience\": \"attempter\",\n \"criterionid\": \"c3-warrant\",\n \"feedbacktype\": \"strength\",\n \"findingids\": [\n \"f3-warrant-positive\"\n ],\n \"message\": \"you correctly distinguish observational association from causal evidence and calibrate the conclusion to the uncertainty in both studies.\",\n \"priority\": 1,\n \"repairvalidationstatus\": \"not_applicable\"\n },\n {\n \"feedbackid\": \"fb2-reviewer-defect\",\n \"audience\": \"reviewer\",\n \"criterionid\": \"c1-direct-response\",\n \"feedbacktype\": \"defect\",\n \"findingids\": [\n \"f5-review-negative\",\n \"f1-direct-positive\"\n ],\n \"message\": \"the attempt does answer the primary-cause question: it concludes that primacy is not established. the project explicitly permits an inconclusive answer when the evidence is inconclusive, so requiring a forced yes/no choice adds an unstated criterion.\",\n \"minimalrepair\": \"change the label to pass and cite the first sentence as affirmative evidence for a direct, calibrated answer.\",\n \"priority\": 1,\n \"repairvalidationstatus\": \"validated\"\n },\n {\n \"feedbackid\": \"fb3-owner-signal\",\n \"audience\": \"project_owner\",\n \"criterionid\": \"c1-direct-response\",\n \"feedbacktype\": \"ambiguity\",\n \"findingids\": [\n \"f5-review-negative\"\n ],\n \"message\": \"monitor whether reviewers repeatedly equate ", "25": "worked example - human resolution foundation examples a bounded precedence question that hides the provisional decision and asks only what can change the outcome. example worked case {\n \"documenttype\": \"human_resolution_packet\",\n \"schemaversion\": \"0.1.0\",\n \"caseref\": {\n \"caseid\": \"case-hypothetical-resolution-001\"\n },\n \"contractref\": {\n \"contractid\": \"example-grounded-analysis\",\n \"contractversion\": \"0.1.0\",\n \"contenthash\": \"sha256:c99c25a0a010f6e69774d0d7a187753a0cb328e8673a74b2b051fff44b1bf3a5\"\n },\n \"interaction\": {\n \"decisiontarget\": \"attempt\",\n \"request\": {\n \"requestid\": \"resolution-hypothetical-001\",\n \"triggercriterionids\": [\n \"c1-direct-response\"\n ],\n \"questiontype\": \"instruction_precedence\",\n \"question\": \"a task-specific note says ", "26": "worked example - project contract foundation examples synthetic closed-world evidence project with positive-proof requirements, anchors, severity, and explicit decision rules. example worked case {\n \"documenttype\": \"project_contract\",\n \"schemaversion\": \"0.1.0\",\n \"contractid\": \"example-grounded-analysis\",\n \"contractversion\": \"0.1.0\",\n \"projectname\": \"example evidence-grounded analysis\",\n \"description\": \"synthetic example showing positive proof, calibrated conclusions, and independent annotation auditing.\",\n \"status\": \"approved\",\n \"createdat\": \"2026-07-15t12:00:00z\",\n \"effectivefrom\": \"2026-07-15t12:00:00z\",\n \"owners\": [\n {\n \"actorid\": \"owner-001\",\n \"role\": \"project_owner\"\n },\n {\n \"actorid\": \"expert-001\",\n \"role\": \"domain_expert\"\n }\n ],\n \"instructiondocuments\": [\n {\n \"instructionid\": \"instructions-main\",\n \"title\": \"main project instructions\",\n \"authority\": \"primary\",\n \"content\": \"use only the supplied evidence. state the strongest conclusion supported by the evidence. distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim. do not force a binary conclusion when the evidence is inconclusive.\",\n \"contenthash\": \"sha256:adca02e1bb85249a185e0460d27f4e9a28de241a2f835dce671cacd15a858495\",\n \"precedencerank\": 1\n }\n ],\n \"sourcepolicy\": {\n \"worldmode\": \"closed_world\",\n \"parametricknowledgepolicy\": \"never_evidence\",\n \"externalretrievalallowed\": false,\n \"defaultcitationrequirement\": \"material_claims\",\n \"snapshotrequired\": true,\n \"conflictpolicy\": \"report material conflict or uncertainty; do not silently choose the more convenient source.\",\n \"freshnessrules\": [],\n \"evidenceclasses\": [\n {\n \"classid\": \"supplied-study\",\n \"name\": \"supplied study excerpt\",\n \"description\": \"evidence included with the task and permitted for all factual judgments.\",\n \"permitted\": true,\n \"authorityrank\": 3\n }\n ]\n },\n \"criteria\": [\n {\n \"criterionid\": \"c1-direct-response\",\n \"name\": \"directly answers the question\",\n \"requirement\": \"state the strongest conclusion supported by the evidence about whether x is the primary cause.\",\n \"intent\": \"the response must resolve the requested issue without inventing certainty.\",\n \"criteriontype\": \"required_presence\",\n \"scope\": \"attempt\",\n \"subjectivity\": \"bounded_judgment\",\n \"instructionrefs\": [\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 2\",\n \"quotedtext\": \"state the strongest conclusion supported by the evidence.\"\n },\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 4\",\n \"quotedtext\": \"do not force a binary conclusion when the evidence is inconclusive.\"\n }\n ],\n \"applicability\": {\n \"ruletext\": \"always applies.\",\n \"condition\": {\n \"kind\": \"always\"\n }\n },\n \"proofpolicy\": {\n \"positiveproofrequired\": true,\n \"positiveproofstandard\": \"the attempt states an explicit conclusion that answers the primary-cause question at the confidence level supported by the evidence.\",\n \"failurecondition\": \"the attempt does not state a conclusion, answers a different question, or forces a conclusion stronger than the evidence permits.\",\n \"omissioncondition\": \"no conclusion about primary causation is present.\",\n \"passonnoviolation\": false,\n \"allowpartialcredit\": true,\n \"allowcompensation\": false\n },\n \"evidencepolicy\": {\n \"allowedevidenceclassids\": [\n \"supplied-study\"\n ],\n \"minimumauthorityrank\": 3,\n \"citationrequired\": false,\n \"externalverificationmode\": \"forbidden\",\n \"deterministicvalidatorids\": []\n },\n \"anchors\": [\n {\n \"anchorid\": \"c1-pass-anchor\",\n \"label\": \"calibrated direct answer\",\n \"example\": \"the evidence does not establish that x is the primary cause.\",\n \"expectedverdict\": \"met\",\n \"notes\": \"a non-binary answer is direct when the evidence is inconclusive.\"\n },\n {\n \"anchorid\": \"c1-fail-anchor\",\n \"label\": \"no answer\",\n \"example\": \"the studies have several methodological differences.\",\n \"expectedverdict\": \"not_met\",\n \"notes\": \"describes evidence without answering the causal question.\"\n }\n ],\n \"validalternatives\": [\n \"a qualified no: the evidence does not establish primacy.\",\n \"an explicitly uncertain conclusion supported by both studies.\"\n ],\n \"exceptions\": [],\n \"dependencies\": {\n \"requiredbeforecriterionids\": [],\n \"impliescriterionids\": []\n },\n \"precedence\": {\n \"overridescriterionids\": [],\n \"overriddenbycriterionids\": []\n },\n \"defaultseverity\": \"major\",\n \"defaultdecisioneffect\": \"rework\",\n \"tags\": [\n \"core\",\n \"answerability\",\n \"positive-proof\"\n ]\n },\n {\n \"criterionid\": \"c2-grounding\",\n \"name\": \"material claims are grounded\",\n \"requirement\": \"every material factual claim must be supported by permitted supplied evidence.\",\n \"intent\": \"prevent unsupported facts and citation laundering.\",\n \"criteriontype\": \"grounding\",\n \"scope\": \"claim\",\n \"subjectivity\": \"objective\",\n \"instructionrefs\": [\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 1\",\n \"quotedtext\": \"use only the supplied evidence.\"\n },\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 3\",\n \"quotedtext\": \"distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim.\"\n }\n ],\n \"applicability\": {\n \"ruletext\": \"applies to all material factual claims.\",\n \"condition\": {\n \"kind\": \"always\"\n }\n },\n \"proofpolicy\": {\n \"positiveproofrequired\": true,\n \"positiveproofstandard\": \"each material claim is linked to evidence that directly entails or reasonably supports it.\",\n \"failurecondition\": \"a central or supporting factual claim lacks permitted evidence or conflicts with it.\",\n \"omissioncondition\": \"a required citation or evidence link for a material claim is absent.\",\n \"passonnoviolation\": false,\n \"allowpartialcredit\": true,\n \"allowcompensation\": false\n },\n \"evidencepolicy\": {\n \"allowedevidenceclassids\": [\n \"supplied-study\"\n ],\n \"minimumauthorityrank\": 3,\n \"citationrequired\": true,\n \"externalverificationmode\": \"forbidden\",\n \"deterministicvalidatorids\": [\n \"citation-id-validator\"\n ]\n },\n \"anchors\": [\n {\n \"anchorid\": \"c2-pass-anchor\",\n \"label\": \"claim-specific support\",\n \"example\": \"e1 reports an 18% observational difference and notes self-selection.\",\n \"expectedverdict\": \"met\",\n \"notes\": \"the claim matches the supplied excerpt.\"\n },\n {\n \"anchorid\": \"c2-fail-anchor\",\n \"label\": \"unsupported strengthening\",\n \"example\": \"e1 proves that x caused the improvement.\",\n \"expectedverdict\": \"not_met\",\n \"notes\": \"the source explicitly disclaims causal identification.\"\n }\n ],\n \"validalternatives\": [],\n \"exceptions\": [\n \"common connective language that adds no factual content does not require a citation.\"\n ],\n \"dependencies\": {\n \"requiredbeforecriterionids\": [],\n \"impliescriterionids\": []\n },\n \"precedence\": {\n \"overridescriterionids\": [],\n \"overriddenbycriterionids\": []\n },\n \"defaultseverity\": \"major\",\n \"defaultdecisioneffect\": \"rework\",\n \"tags\": [\n \"grounding\",\n \"evidence\",\n \"hard-quality\"\n ]\n },\n {\n \"criterionid\": \"c3-warrant\",\n \"name\": \"conclusion strength matches evidence\",\n \"requirement\": \"distinguish association from causation and do not assert primacy unless the evidence warrants it.\",\n \"intent\": \"assess whether the inference, not merely the cited facts, is justified.\",\n \"criteriontype\": \"inferential_warrant\",\n \"scope\": \"claim\",\n \"subjectivity\": \"bounded_judgment\",\n \"instructionrefs\": [\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 2\",\n \"quotedtext\": \"state the strongest conclusion supported by the evidence.\"\n },\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 3\",\n \"quotedtext\": \"distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim.\"\n }\n ],\n \"applicability\": {\n \"ruletext\": \"applies to causal and comparative conclusions.\",\n \"condition\": {\n \"kind\": \"always\"\n }\n },\n \"proofpolicy\": {\n \"positiveproofrequired\": true,\n \"positiveproofstandard\": \"the response explicitly calibrates its causal conclusion to study design, uncertainty, and counterevidence.\",\n \"failurecondition\": \"the response infers causation or primacy from evidence that supports only association or an uncertain effect.\",\n \"omissioncondition\": \"the response fails to distinguish observational association from causal evidence.\",\n \"passonnoviolation\": false,\n \"allowpartialcredit\": true,\n \"allowcompensation\": false\n },\n \"evidencepolicy\": {\n \"allowedevidenceclassids\": [\n \"supplied-study\"\n ],\n \"minimumauthorityrank\": 3,\n \"citationrequired\": true,\n \"externalverificationmode\": \"forbidden\",\n \"deterministicvalidatorids\": []\n },\n \"anchors\": [\n {\n \"anchorid\": \"c3-pass-anchor\",\n \"label\": \"proportionate inference\",\n \"example\": \"x may contribute, but the evidence does not establish that it is the primary cause.\",\n \"expectedverdict\": \"met\",\n \"notes\": \"uses appropriate uncertainty and distinguishes contribution from primacy.\"\n },\n {\n \"anchorid\": \"c3-fail-anchor\",\n \"label\": \"causal overreach\",\n \"example\": \"the 18% difference proves x is the primary cause.\",\n \"expectedverdict\": \"not_met\",\n \"notes\": \"treats observational association as proof of causation and primacy.\"\n }\n ],\n \"validalternatives\": [\n \"a cautious conclusion that evidence is insufficient to determine primacy.\"\n ],\n \"exceptions\": [],\n \"dependencies\": {\n \"requiredbeforecriterionids\": [\n \"c2-grounding\"\n ],\n \"impliescriterionids\": []\n },\n \"precedence\": {\n \"overridescriterionids\": [],\n \"overriddenbycriterionids\": []\n },\n \"defaultseverity\": \"major\",\n \"defaultdecisioneffect\": \"rework\",\n \"tags\": [\n \"reasoning\",\n \"warrant\",\n \"hard-quality\"\n ]\n },\n {\n \"criterionid\": \"c4-completeness\",\n \"name\": \"addresses both studies\",\n \"requirement\": \"use and synthesize both e1 and e2.\",\n \"intent\": \"prevent selective use of convenient evidence.\",\n \"criteriontype\": \"completeness\",\n \"scope\": \"attempt\",\n \"subjectivity\": \"objective\",\n \"instructionrefs\": [\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"sentence 3\",\n \"quotedtext\": \"distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim.\"\n }\n ],\n \"applicability\": {\n \"ruletext\": \"applies when both studies are supplied.\",\n \"condition\": {\n \"kind\": \"always\"\n }\n },\n \"proofpolicy\": {\n \"positiveproofrequired\": true,\n \"positiveproofstandard\": \"the attempt materially describes the relevant contribution and limitation of e1 and e2.\",\n \"failurecondition\": \"one study is ignored or materially misrepresented.\",\n \"omissioncondition\": \"no substantive treatment of e1 or e2 appears.\",\n \"passonnoviolation\": false,\n \"allowpartialcredit\": true,\n \"allowcompensation\": false\n },\n \"evidencepolicy\": {\n \"allowedevidenceclassids\": [\n \"supplied-study\"\n ],\n \"minimumauthorityrank\": 3,\n \"citationrequired\": true,\n \"externalverificationmode\": \"forbidden\",\n \"deterministicvalidatorids\": [\n \"required-evidence-id-validator\"\n ]\n },\n \"anchors\": [\n {\n \"anchorid\": \"c4-pass-anchor\",\n \"label\": \"both sources synthesized\",\n \"example\": \"e1 shows an association but cannot identify causation; e2 is randomized but inconclusive.\",\n \"expectedverdict\": \"met\",\n \"notes\": \"captures the evidentiary role of both studies.\"\n }\n ],\n \"validalternatives\": [],\n \"exceptions\": [],\n \"dependencies\": {\n \"requiredbeforecriterionids\": [],\n \"impliescriterionids\": []\n },\n \"precedence\": {\n \"overridescriterionids\": [],\n \"overriddenbycriterionids\": []\n },\n \"defaultseverity\": \"major\",\n \"defaultdecisioneffect\": \"rework\",\n \"tags\": [\n \"completeness\",\n \"positive-proof\"\n ]\n },\n {\n \"criterionid\": \"c5-annotation-fidelity\",\n \"name\": \"reviewer label and rationale faithfully evaluate the attempt\",\n \"requirement\": \"the reviewer must apply the written criteria, cite relevant evidence, and assign proportionate severity.\",\n \"intent\": \"audit the human annotation separately from the attempt.\",\n \"criteriontype\": \"annotation_fidelity\",\n \"scope\": \"annotation\",\n \"subjectivity\": \"bounded_judgment\",\n \"instructionrefs\": [\n {\n \"instructionid\": \"instructions-main\",\n \"locator\": \"entire document\",\n \"quotedtext\": \"use only the supplied evidence. state the strongest conclusion supported by the evidence. distinguish association from causation, address both supplied studies, and cite the evidence for every material factual claim. do not force a binary conclusion when the evidence is inconclusive.\"\n }\n ],\n \"applicability\": {\n \"ruletext\": \"applies only when a human review is present.\",\n \"condition\": {\n \"kind\": \"predicate\",\n \"fieldpath\": \"case.humanreview\",\n \"operator\": \"exists\",\n \"value\": true\n }\n },\n \"proofpolicy\": {\n \"positiveproofrequired\": true,\n \"positiveproofstandard\": \"the reviewer label, criterion attribution, evidence, and severity are mutually consistent and aligned with the contract.\",\n \"failurecondition\": \"the review applies an unstated preference, misattributes the issue, cites irrelevant evidence, or assigns an unsupported disposition.\",\n \"omissioncondition\": \"the rationale omits the decisive criterion or evidence.\",\n \"passonnoviolation\": false,\n \"allowpartialcredit\": true,\n \"allowcompensation\": false\n },\n \"evidencepolicy\": {\n \"allowedevidenceclassids\": [\n \"supplied-study\"\n ],\n \"minimumauthorityrank\": 3,\n \"citationrequired\": false,\n \"externalverificationmode\": \"forbidden\",\n \"deterministicvalidatorids\": []\n },\n \"anchors\": [\n {\n \"anchorid\": \"c5-fail-anchor\",\n \"label\": \"unstated binary preference\",\n \"example\": \"fail because the answer did not choose yes or no, even though the instructions allow an inconclusive conclusion.\",\n \"expectedverdict\": \"not_met\",\n \"notes\": \"the reviewer substitutes a preference that contradicts the written contract.\"\n }\n ],\n \"validalternatives\": [],\n \"exceptions\": [],\n \"dependencies\": {\n \"requiredbeforecriterionids\": [\n \"c1-direct-response\"\n ],\n \"impliescriterionids\": []\n },\n \"precedence\": {\n \"overridescriterionids\": [],\n \"overriddenbycriterionids\": []\n },\n \"defaultseverity\": \"major\",\n \"defaultdecisioneffect\": \"accept_with_note\",\n \"tags\": [\n \"annotation\",\n \"reviewer-audit\"\n ]\n }\n ],\n \"decisionpolicy\": {\n \"policyversion\": \"0.1.0\",\n \"allowedactions\": [\n \"accept\",\n \"accept_with_notes\",\n \"rework\",\n \"reject\",\n \"escalate\"\n ],\n \"rules\": [\n {\n \"ruleid\": \"r1-critical-hard-failure\",\n \"priority\": 1,\n \"description\": \"reject a calibrated critical hard-quality failure.\",\n \"match\": {\n \"criteriontags\": [\n \"hard-quality\"\n ],\n \"verdicts\": [\n \"not_met\"\n ],\n \"severities\": [\n \"critical\"\n ],\n \"minimumcalibratedconfidence\": 0.95\n },\n \"action\": \"reject\",\n \"terminal\": true,\n \"explanationtemplate\": \"a critical hard-quality criterion failed at high calibrated confidence.\",\n \"target\": \"attempt\"\n },\n {\n \"ruleid\": \"r2-major-repairable-failure\",\n \"priority\": 2,\n \"description\": \"request rework for a clear major, repairable failure.\",\n \"match\": {\n \"verdicts\": [\n \"not_met\"\n ],\n \"severities\": [\n \"major\"\n ],\n \"minimumcalibratedconfidence\": 0.85\n },\n \"action\": \"rework\",\n \"terminal\": false,\n \"explanationtemplate\": \"at least one major criterion is not met and has a specific repair.\",\n \"target\": \"attempt\"\n },\n {\n \"ruleid\": \"r3-decision-changing-uncertainty\",\n \"priority\": 3,\n \"description\": \"escalate unresolved findings capable of changing the final action.\",\n \"match\": {\n \"hasunresolvedfindings\": true\n },\n \"action\": \"escalate\",\n \"terminal\": false,\n \"explanationtemplate\": \"a decision-changing issue remains unresolved.\",\n \"target\": \"attempt\"\n },\n {\n \"ruleid\": \"r4-accept-clean\",\n \"priority\": 10,\n \"description\": \"accept when all applicable attempt-quality criteria are met and no major unresolved issue remains.\",\n \"match\": {\n \"criterionids\": [\n \"c1-direct-response\",\n \"c2-grounding\",\n \"c3-warrant\",\n \"c4-completeness\"\n ],\n \"verdicts\": [\n \"met\"\n ],\n \"minimumcalibratedconfidence\": 0.85,\n \"hasunresolvedfindings\": false\n },\n \"action\": \"accept\",\n \"terminal\": true,\n \"explanationtemplate\": \"all applicable attempt-quality criteria are affirmatively satisfied.\",\n \"target\": \"attempt\"\n },\n {\n \"target\": \"annotation\",\n \"ruleid\": \"r5-annotation-major-failure\",\n \"priority\": 1,\n \"description\": \"request correction when annotation fidelity has a clear major failure.\",\n \"match\": {\n \"criterionids\": [\n \"c5-annotation-fidelity\"\n ],\n \"verdicts\": [\n \"not_met\"\n ],\n \"severities\": [\n \"major\"\n ],\n \"minimumcalibratedconfidence\": 0.85\n },\n \"action\": \"rework\",\n \"terminal\": true,\n \"explanationtemplate\": \"the human annotation is not faithful to the attempt and contract.\"\n },\n {\n \"target\": \"annotation\",\n \"ruleid\": \"r6-annotation-clean\",\n \"priority\": 10,\n \"description\": \"accept a faithful human annotation.\",\n \"match\": {\n \"criterionids\": [\n \"c5-annotation-fidelity\"\n ],\n \"verdicts\": [\n \"met\"\n ],\n \"minimumcalibratedconfidence\": 0.85,\n \"hasunresolvedfindings\": false\n },\n \"action\": \"accept\",\n \"terminal\": true,\n \"explanationtemplate\": \"the annotation is faithful and adequately supported.\"\n },\n {\n \"target\": \"composite_case\",\n \"ruleid\": \"r7-composite-annotation-rework\",\n \"priority\": 1,\n \"description\": \"rework the composite datum when the attempt is acceptable but its required annotation is not.\",\n \"match\": {\n \"criterionids\": [\n \"c5-annotation-fidelity\"\n ],\n \"verdicts\": [\n \"not_met\"\n ],\n \"severities\": [\n \"major\"\n ],\n \"minimumcalibratedconfidence\": 0.85\n },\n \"action\": \"rework\",\n \"terminal\": true,\n \"explanationtemplate\": \"the attempt can be retained, but the annotation must be corrected before the composite datum is admitted.\"\n }\n ],\n \"defaultaction\": \"escalate\",\n \"tiebreakorder\": [\n \"reject\",\n \"escalate\",\n \"rework\",\n \"accept_with_notes\",\n \"accept\"\n ],\n \"confidencepolicy\": {\n \"requirescalibrationforactions\": [\n \"accept\",\n \"reject\"\n ],\n \"uncalibratedaction\": \"escalate\",\n \"minimumcalibrationcasesperslice\": 50\n }\n },\n \"humanreviewpolicy\": {\n \"maxinteractionspercase\": 1,\n \"onlyaskifdecisionchanging\": true,\n \"hideprovisionaldecision\": true,\n \"requireambiguityoption\": true,\n \"allowedquestiontypes\": [\n \"applicability\",\n \"instruction_precedence\",\n \"evidence_authority\",\n \"inference_permission\",\n \"severity\",\n \"valid_alternative\",\n \"rubric_ambiguity\",\n \"domain_expertise\"\n ],\n \"expertiseroutingrules\": [\n \"route causal-inference disputes to a reviewer qualified in study design.\"\n ]\n },\n \"feedbackpolicy\": {\n \"audiences\": [\n \"attempter\",\n \"reviewer\",\n \"project_owner\"\n ],\n \"maxitemsperaudience\": 3,\n \"includevalidatedstrengths\": true,\n \"validaterepairs\": true,\n \"priorityorder\": [\n \"decisive\",\n \"critical\",\n \"major\",\n \"minor\",\n \"strength\"\n ]\n },\n \"securitypolicy\": {\n \"treatcandidatecontentasuntrusted\": true,\n \"ignoreembeddedevaluatorinstructions\": true,\n \"requirestructuredoutputs\": true,\n \"allowcandidatetooverriderubric\": false,\n \"redactionrules\": [\n \"remove direct personal identifiers from reviewer analytics exports.\"\n ]\n },\n \"auditpolicy\": {\n \"retainexactsourcesnapshots\": true,\n \"retainpromptversions\": true,\n \"retainmodelrunmetadata\": true,\n \"retainhumanresolutions\": true,\n \"replayrequired\": true,\n \"retentiondays\": 365\n },\n \"metadata\": {\n \"example\": true\n },\n \"decisiontargets\": [\n \"attempt\",\n \"annotation\",\n \"composite_case\"\n ]\n}\n", "27": "json schema - benchmark item foundation schemas runtime schema for adjudicated truth, acceptable actions, ambiguity, challenge attributes, and leakage controls. schema runtime validation {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://autoqa.example/schemas/benchmark-item.schema.json\",\n \"title\": \"autoqa benchmark item\",\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"documenttype\",\n \"schemaversion\",\n \"benchmarkitemid\",\n \"benchmarkversion\",\n \"split\",\n \"contractref\",\n \"caseref\",\n \"slicetags\",\n \"challengeattributes\",\n \"goldstandard\",\n \"adjudication\",\n \"leakagecontrol\",\n \"weight\"\n ],\n \"properties\": {\n \"documenttype\": {\n \"const\": \"benchmark_item\"\n },\n \"schemaversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"benchmarkitemid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"benchmarkversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"split\": {\n \"type\": \"string\",\n \"enum\": [\n \"development\",\n \"calibration\",\n \"test\",\n \"challenge\",\n \"shadow\",\n \"held_out_project\"\n ]\n },\n \"contractref\": {\n \"$ref\": \"common.schema.json#/$defs/contractref\"\n },\n \"caseref\": {\n \"$ref\": \"common.schema.json#/$defs/caseref\"\n },\n \"slicetags\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"challengeattributes\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\",\n \"enum\": [\n \"strong_positive\",\n \"subtle_valid_alternative\",\n \"polished_falsehood\",\n \"unsupported_rationale\",\n \"citation_mismatch\",\n \"partial_support\",\n \"required_content_omission\",\n \"severity_edge\",\n \"instruction_conflict\",\n \"ambiguous_case\",\n \"prompt_injection\",\n \"rubric_parroting\",\n \"correct_label_wrong_reason\",\n \"wrong_label_correct_observation\",\n \"long_context\",\n \"domain_expertise\",\n \"position_bias_probe\"\n ]\n }\n },\n \"goldstandard\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"criteriongold\",\n \"acceptabledecisions\",\n \"ambiguitystatus\"\n ],\n \"properties\": {\n \"criteriongold\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"criterionid\",\n \"applicability\",\n \"acceptableverdicts\",\n \"severity\",\n \"positiveevidencespans\",\n \"negativeevidencespans\",\n \"omissiondescriptions\",\n \"notes\"\n ],\n \"properties\": {\n \"criterionid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"applicability\": {\n \"type\": \"string\",\n \"enum\": [\n \"applies\",\n \"does_not_apply\",\n \"uncertain\"\n ]\n },\n \"acceptableverdicts\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/criterionverdict\"\n }\n },\n \"severity\": {\n \"$ref\": \"common.schema.json#/$defs/severity\"\n },\n \"positiveevidencespans\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/spanref\"\n }\n },\n \"negativeevidencespans\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/spanref\"\n }\n },\n \"omissiondescriptions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"notes\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"annotationgold\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"properties\": {\n \"acceptablelabelsupport\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"expecteddisagreementtypes\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"notes\": {\n \"type\": \"string\"\n }\n }\n },\n \"ambiguitystatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"clear\",\n \"reasonable_disagreement\",\n \"rubric_ambiguous\",\n \"insufficient_evidence\"\n ]\n },\n \"validalternativenotes\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"acceptabledecisions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"target\",\n \"acceptableactions\"\n ],\n \"properties\": {\n \"target\": {\n \"$ref\": \"common.schema.json#/$defs/decisiontarget\"\n },\n \"acceptableactions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n }\n }\n }\n }\n }\n }\n },\n \"adjudication\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"method\",\n \"adjudicatorcount\",\n \"domainexpertcount\",\n \"blindtoautoqa\",\n \"blindtoreviewerlabel\",\n \"disagreementtypes\",\n \"completedat\"\n ],\n \"properties\": {\n \"method\": {\n \"type\": \"string\",\n \"enum\": [\n \"single_expert\",\n \"independent_then_adjudicate\",\n \"consensus_panel\",\n \"latent_label_model\"\n ]\n },\n \"adjudicatorcount\": {\n \"type\": \"integer\",\n \"minimum\": 1\n },\n \"domainexpertcount\": {\n \"type\": \"integer\",\n \"minimum\": 0\n },\n \"blindtoautoqa\": {\n \"type\": \"boolean\"\n },\n \"blindtoreviewerlabel\": {\n \"type\": \"boolean\"\n },\n \"initialagreementrate\": {\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 1\n },\n \"disagreementtypes\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"adjudicationnotes\": {\n \"type\": \"string\"\n },\n \"completedat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n }\n }\n },\n \"leakagecontrol\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"allowedforpromptdevelopment\",\n \"heldoutproject\",\n \"firstfrozenat\",\n \"accesspolicy\"\n ],\n \"properties\": {\n \"allowedforpromptdevelopment\": {\n \"type\": \"boolean\"\n },\n \"heldoutproject\": {\n \"type\": \"boolean\"\n },\n \"firstfrozenat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"accesspolicy\": {\n \"type\": \"string\",\n \"minlength\": 1\n }\n }\n },\n \"weight\": {\n \"type\": \"number\",\n \"exclusiveminimum\": 0\n },\n \"exclusionreason\": {\n \"type\": \"string\"\n },\n \"metadata\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n }\n }\n}\n", "28": "json schema - case input foundation schemas runtime schema for task, evidence bundle, attempt, human review, references, and provenance. schema runtime validation {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://autoqa.example/schemas/case-input.schema.json\",\n \"title\": \"autoqa case input\",\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"documenttype\",\n \"schemaversion\",\n \"caseid\",\n \"createdat\",\n \"contractref\",\n \"task\",\n \"evidencebundle\",\n \"attempt\",\n \"casetags\"\n ],\n \"properties\": {\n \"documenttype\": {\n \"const\": \"case_input\"\n },\n \"schemaversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"caseid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"createdat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"contractref\": {\n \"$ref\": \"common.schema.json#/$defs/contractref\"\n },\n \"task\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"taskid\",\n \"tasktype\",\n \"prompt\",\n \"contenthash\"\n ],\n \"properties\": {\n \"taskid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"tasktype\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"prompt\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"taskinstructions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"expectedoutputformat\": {\n \"type\": \"string\"\n },\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n },\n \"metadata\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n }\n }\n },\n \"evidencebundle\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"evidenceid\",\n \"evidenceclassid\",\n \"title\",\n \"content\",\n \"contenthash\",\n \"authorityrank\",\n \"permitted\",\n \"capturedat\"\n ],\n \"properties\": {\n \"evidenceid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"evidenceclassid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"title\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"content\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n },\n \"sourceuri\": {\n \"type\": \"string\",\n \"format\": \"uri\"\n },\n \"authorityrank\": {\n \"type\": \"integer\",\n \"minimum\": 0\n },\n \"permitted\": {\n \"type\": \"boolean\"\n },\n \"publishedat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"capturedat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"metadata\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n }\n }\n }\n },\n \"attempt\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"attemptid\",\n \"content\",\n \"contenthash\",\n \"submittedat\"\n ],\n \"properties\": {\n \"attemptid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"authorid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"content\": {\n \"type\": \"string\"\n },\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n },\n \"submittedat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"attachments\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"artifactid\",\n \"artifacttype\",\n \"contenthash\"\n ],\n \"properties\": {\n \"artifactid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"artifacttype\": {\n \"type\": \"string\"\n },\n \"content\": {\n \"type\": \"string\"\n },\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n }\n }\n }\n },\n \"tooltraces\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"traceid\",\n \"toolname\",\n \"input\",\n \"output\",\n \"contenthash\"\n ],\n \"properties\": {\n \"traceid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"toolname\": {\n \"type\": \"string\"\n },\n \"input\": {},\n \"output\": {},\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n }\n }\n }\n }\n }\n },\n \"humanreview\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"annotationid\",\n \"reviewerid\",\n \"label\",\n \"reasoncodes\",\n \"rationale\",\n \"submittedat\"\n ],\n \"properties\": {\n \"annotationid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"reviewerid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"label\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"score\": {\n \"type\": \"number\"\n },\n \"reasoncodes\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"rationale\": {\n \"type\": \"string\"\n },\n \"criterionannotations\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"criterionid\",\n \"verdict\",\n \"rationale\"\n ],\n \"properties\": {\n \"criterionid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"verdict\": {\n \"$ref\": \"common.schema.json#/$defs/criterionverdict\"\n },\n \"severity\": {\n \"$ref\": \"common.schema.json#/$defs/severity\"\n },\n \"rationale\": {\n \"type\": \"string\"\n },\n \"citedspans\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/spanref\"\n }\n }\n }\n }\n },\n \"reviewerconfidence\": {\n \"$ref\": \"common.schema.json#/$defs/confidence\"\n },\n \"submittedat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n }\n }\n },\n \"referenceanswers\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"referenceid\",\n \"role\",\n \"content\",\n \"contenthash\"\n ],\n \"properties\": {\n \"referenceid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"role\": {\n \"type\": \"string\",\n \"enum\": [\n \"authoritative\",\n \"illustrative\",\n \"positive_anchor\",\n \"negative_anchor\"\n ]\n },\n \"content\": {\n \"type\": \"string\"\n },\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n }\n }\n }\n },\n \"declaredexpectedelements\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/expectedelement\"\n }\n },\n \"casetags\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"metadata\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n }\n }\n}\n", "29": "json schema - common definitions foundation schemas shared runtime definitions used by the project, case, evaluation, benchmark, and resolution schemas. schema runtime validation {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://autoqa.example/schemas/common.schema.json\",\n \"title\": \"autoqa common definitions\",\n \"type\": \"object\",\n \"$defs\": {\n \"identifier\": {\n \"type\": \"string\",\n \"pattern\": \"^[a-za-z0-9][a-za-z0-9._:-]{1,127}$\"\n },\n \"semver\": {\n \"type\": \"string\",\n \"pattern\": \"^\\\\d+\\\\.\\\\d+\\\\.\\\\d+(?:-[0-9a-za-z.-]+)?(?:\\\\+[0-9a-za-z.-]+)?$\"\n },\n \"contenthash\": {\n \"type\": \"string\",\n \"pattern\": \"^sha256:[a-f0-9]{64}$\"\n },\n \"datetime\": {\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"confidenceband\": {\n \"type\": \"string\",\n \"enum\": [\n \"high\",\n \"medium\",\n \"low\",\n \"uncalibrated\"\n ]\n },\n \"severity\": {\n \"type\": \"string\",\n \"enum\": [\n \"none\",\n \"advisory\",\n \"minor\",\n \"major\",\n \"critical\"\n ]\n },\n \"operationalaction\": {\n \"type\": \"string\",\n \"enum\": [\n \"accept\",\n \"accept_with_notes\",\n \"rework\",\n \"reject\",\n \"escalate\"\n ]\n },\n \"criterionverdict\": {\n \"type\": \"string\",\n \"enum\": [\n \"met\",\n \"partially_met\",\n \"not_met\",\n \"ambiguous\",\n \"unknown\",\n \"not_applicable\"\n ]\n },\n \"contractref\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"contractid\",\n \"contractversion\",\n \"contenthash\"\n ],\n \"properties\": {\n \"contractid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"contractversion\": {\n \"$ref\": \"#/$defs/semver\"\n },\n \"contenthash\": {\n \"$ref\": \"#/$defs/contenthash\"\n }\n }\n },\n \"caseref\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"caseid\"\n ],\n \"properties\": {\n \"caseid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"contenthash\": {\n \"$ref\": \"#/$defs/contenthash\"\n }\n }\n },\n \"spanref\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"artifacttype\",\n \"artifactid\",\n \"quotedtext\"\n ],\n \"properties\": {\n \"artifacttype\": {\n \"type\": \"string\",\n \"enum\": [\n \"task\",\n \"evidence\",\n \"attempt\",\n \"human_review\",\n \"reference\",\n \"tool_trace\"\n ]\n },\n \"artifactid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"startchar\": {\n \"type\": \"integer\",\n \"minimum\": 0\n },\n \"endchar\": {\n \"type\": \"integer\",\n \"minimum\": 0\n },\n \"quotedtext\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"locator\": {\n \"type\": \"string\"\n }\n },\n \"allof\": [\n {\n \"if\": {\n \"required\": [\n \"startchar\"\n ]\n },\n \"then\": {\n \"required\": [\n \"endchar\"\n ]\n }\n },\n {\n \"if\": {\n \"required\": [\n \"endchar\"\n ]\n },\n \"then\": {\n \"required\": [\n \"startchar\"\n ]\n }\n }\n ]\n },\n \"confidence\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"band\"\n ],\n \"properties\": {\n \"rawscore\": {\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 1\n },\n \"calibratedscore\": {\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 1\n },\n \"band\": {\n \"$ref\": \"#/$defs/confidenceband\"\n },\n \"calibrationprofileid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"rationale\": {\n \"type\": \"string\"\n }\n },\n \"allof\": [\n {\n \"if\": {\n \"properties\": {\n \"band\": {\n \"const\": \"uncalibrated\"\n }\n },\n \"required\": [\n \"band\"\n ]\n },\n \"then\": {\n \"not\": {\n \"anyof\": [\n {\n \"required\": [\n \"calibratedscore\"\n ]\n },\n {\n \"required\": [\n \"calibrationprofileid\"\n ]\n }\n ]\n }\n }\n },\n {\n \"if\": {\n \"properties\": {\n \"band\": {\n \"enum\": [\n \"high\",\n \"medium\",\n \"low\"\n ]\n }\n },\n \"required\": [\n \"band\"\n ]\n },\n \"then\": {\n \"required\": [\n \"calibratedscore\",\n \"calibrationprofileid\"\n ]\n }\n }\n ]\n },\n \"generationsource\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"kind\",\n \"componentid\"\n ],\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"enum\": [\n \"deterministic_validator\",\n \"llm_judge\",\n \"human\",\n \"hybrid\"\n ]\n },\n \"componentid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"modelrunid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"humanactorid\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n }\n },\n \"condition\": {\n \"oneof\": [\n {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"kind\"\n ],\n \"properties\": {\n \"kind\": {\n \"const\": \"always\"\n }\n }\n },\n {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"kind\",\n \"fieldpath\",\n \"operator\"\n ],\n \"properties\": {\n \"kind\": {\n \"const\": \"predicate\"\n },\n \"fieldpath\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"operator\": {\n \"type\": \"string\",\n \"enum\": [\n \"equals\",\n \"not_equals\",\n \"in\",\n \"not_in\",\n \"contains\",\n \"exists\",\n \"greater_than\",\n \"greater_than_or_equal\",\n \"less_than\",\n \"less_than_or_equal\",\n \"matches_regex\"\n ]\n },\n \"value\": {}\n }\n },\n {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"kind\",\n \"conditions\"\n ],\n \"properties\": {\n \"kind\": {\n \"const\": \"all\"\n },\n \"conditions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"$ref\": \"#/$defs/condition\"\n }\n }\n }\n },\n {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"kind\",\n \"conditions\"\n ],\n \"properties\": {\n \"kind\": {\n \"const\": \"any\"\n },\n \"conditions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"$ref\": \"#/$defs/condition\"\n }\n }\n }\n },\n {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"kind\",\n \"condition\"\n ],\n \"properties\": {\n \"kind\": {\n \"const\": \"not\"\n },\n \"condition\": {\n \"$ref\": \"#/$defs/condition\"\n }\n }\n }\n ]\n },\n \"finding\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"findingid\",\n \"findingtype\",\n \"summary\",\n \"materiality\",\n \"severity\",\n \"confidence\",\n \"status\",\n \"source\"\n ],\n \"properties\": {\n \"findingid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"findingtype\": {\n \"type\": \"string\",\n \"enum\": [\n \"positive\",\n \"negative\",\n \"omission\",\n \"ambiguity\",\n \"counterevidence\",\n \"integrity\"\n ]\n },\n \"criterionid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"summary\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"detail\": {\n \"type\": \"string\"\n },\n \"claimids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"expectedelementids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"targetspans\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"#/$defs/spanref\"\n }\n },\n \"evidenceids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"relationship\": {\n \"type\": \"string\",\n \"enum\": [\n \"directly_entails\",\n \"reasonably_supports\",\n \"weakly_supports\",\n \"contradicts\",\n \"irrelevant\",\n \"insufficient\",\n \"not_applicable\"\n ]\n },\n \"materiality\": {\n \"type\": \"string\",\n \"enum\": [\n \"incidental\",\n \"supporting\",\n \"central\",\n \"decisive\"\n ]\n },\n \"severity\": {\n \"$ref\": \"#/$defs/severity\"\n },\n \"confidence\": {\n \"$ref\": \"#/$defs/confidence\"\n },\n \"status\": {\n \"type\": \"string\",\n \"enum\": [\n \"observed\",\n \"challenged\",\n \"confirmed\",\n \"retracted\"\n ]\n },\n \"source\": {\n \"$ref\": \"#/$defs/generationsource\"\n }\n }\n },\n \"claim\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"claimid\",\n \"proposition\",\n \"claimtype\",\n \"importance\",\n \"explicitness\",\n \"attemptspans\"\n ],\n \"properties\": {\n \"claimid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"proposition\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"claimtype\": {\n \"type\": \"string\",\n \"enum\": [\n \"factual\",\n \"interpretive\",\n \"predictive\",\n \"normative\",\n \"preference\",\n \"procedural\"\n ]\n },\n \"importance\": {\n \"type\": \"string\",\n \"enum\": [\n \"incidental\",\n \"supporting\",\n \"central\"\n ]\n },\n \"explicitness\": {\n \"type\": \"string\",\n \"enum\": [\n \"explicit\",\n \"implied\"\n ]\n },\n \"attemptspans\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"$ref\": \"#/$defs/spanref\"\n }\n },\n \"qualifiers\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"citedevidenceids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n }\n }\n },\n \"expectedelement\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"elementid\",\n \"criterionid\",\n \"description\",\n \"required\",\n \"importance\",\n \"coverage\"\n ],\n \"properties\": {\n \"elementid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"criterionid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"description\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"required\": {\n \"type\": \"boolean\"\n },\n \"importance\": {\n \"type\": \"string\",\n \"enum\": [\n \"supporting\",\n \"central\",\n \"decisive\"\n ]\n },\n \"acceptablealternatives\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"observedspans\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"#/$defs/spanref\"\n }\n },\n \"coverage\": {\n \"type\": \"string\",\n \"enum\": [\n \"complete\",\n \"partial\",\n \"missing\",\n \"not_applicable\",\n \"unknown\"\n ]\n }\n }\n },\n \"evidencerelationship\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"relationshipid\",\n \"subjecttype\",\n \"subjectid\",\n \"evidenceid\",\n \"relationship\",\n \"sourcequality\",\n \"temporalvalidity\",\n \"confidence\"\n ],\n \"properties\": {\n \"relationshipid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"subjecttype\": {\n \"type\": \"string\",\n \"enum\": [\n \"claim\",\n \"expected_element\",\n \"finding\"\n ]\n },\n \"subjectid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"evidenceid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"relationship\": {\n \"type\": \"string\",\n \"enum\": [\n \"directly_entails\",\n \"reasonably_supports\",\n \"weakly_supports\",\n \"contradicts\",\n \"irrelevant\",\n \"insufficient\"\n ]\n },\n \"sourcequality\": {\n \"type\": \"string\",\n \"enum\": [\n \"high\",\n \"acceptable\",\n \"weak\",\n \"disallowed\",\n \"unknown\"\n ]\n },\n \"temporalvalidity\": {\n \"type\": \"string\",\n \"enum\": [\n \"valid\",\n \"stale\",\n \"not_time_sensitive\",\n \"unknown\"\n ]\n },\n \"evidencespans\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"#/$defs/spanref\"\n }\n },\n \"confidence\": {\n \"$ref\": \"#/$defs/confidence\"\n },\n \"notes\": {\n \"type\": \"string\"\n }\n }\n },\n \"criterionassessment\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"criterionid\",\n \"applicability\",\n \"positivefindingids\",\n \"negativefindingids\",\n \"omissionfindingids\",\n \"counterevidencefindingids\",\n \"unresolvedfindingids\",\n \"groundingstatus\",\n \"truthstatus\",\n \"warrantstatus\",\n \"relevancestatus\",\n \"verdict\",\n \"severity\",\n \"confidence\",\n \"decisioneffect\",\n \"rationale\"\n ],\n \"properties\": {\n \"criterionid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"applicability\": {\n \"type\": \"string\",\n \"enum\": [\n \"applies\",\n \"does_not_apply\",\n \"uncertain\"\n ]\n },\n \"positivefindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"negativefindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"omissionfindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"counterevidencefindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"unresolvedfindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"groundingstatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"grounded\",\n \"partially_grounded\",\n \"ungrounded\",\n \"not_applicable\",\n \"unknown\"\n ]\n },\n \"truthstatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"verified_true\",\n \"verified_false\",\n \"mixed\",\n \"not_evaluated\",\n \"not_applicable\",\n \"unknown\"\n ]\n },\n \"warrantstatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"justified\",\n \"overstated\",\n \"invalid\",\n \"not_applicable\",\n \"unknown\"\n ]\n },\n \"relevancestatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"relevant\",\n \"partially_relevant\",\n \"irrelevant\",\n \"not_applicable\",\n \"unknown\"\n ]\n },\n \"verdict\": {\n \"$ref\": \"#/$defs/criterionverdict\"\n },\n \"severity\": {\n \"$ref\": \"#/$defs/severity\"\n },\n \"confidence\": {\n \"$ref\": \"#/$defs/confidence\"\n },\n \"decisioneffect\": {\n \"type\": \"string\",\n \"enum\": [\n \"none\",\n \"accept_with_note\",\n \"rework\",\n \"reject\",\n \"escalate\"\n ]\n },\n \"rationale\": {\n \"type\": \"string\",\n \"minlength\": 1\n }\n }\n },\n \"globalassessment\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"axisid\",\n \"verdict\",\n \"findingids\",\n \"confidence\",\n \"rationale\"\n ],\n \"properties\": {\n \"axisid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"verdict\": {\n \"$ref\": \"#/$defs/criterionverdict\"\n },\n \"findingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"score\": {\n \"type\": \"number\"\n },\n \"confidence\": {\n \"$ref\": \"#/$defs/confidence\"\n },\n \"rationale\": {\n \"type\": \"string\"\n }\n }\n },\n \"policytraceentry\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"ruleid\",\n \"matched\",\n \"result\"\n ],\n \"properties\": {\n \"ruleid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"matched\": {\n \"type\": \"boolean\"\n },\n \"inputs\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n },\n \"result\": {\n \"type\": \"string\"\n }\n }\n },\n \"decision\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"target\",\n \"action\",\n \"policyversion\",\n \"triggeredruleids\",\n \"decisivecriterionids\",\n \"unresolvedfindingids\",\n \"confidence\",\n \"policytrace\",\n \"rationale\",\n \"decidedat\"\n ],\n \"properties\": {\n \"action\": {\n \"$ref\": \"#/$defs/operationalaction\"\n },\n \"policyversion\": {\n \"$ref\": \"#/$defs/semver\"\n },\n \"triggeredruleids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"decisivecriterionids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"unresolvedfindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"confidence\": {\n \"$ref\": \"#/$defs/confidence\"\n },\n \"policytrace\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"#/$defs/policytraceentry\"\n }\n },\n \"rationale\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"decidedat\": {\n \"$ref\": \"#/$defs/datetime\"\n },\n \"target\": {\n \"$ref\": \"#/$defs/decisiontarget\"\n }\n }\n },\n \"resolutionoption\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"optionid\",\n \"label\"\n ],\n \"properties\": {\n \"optionid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"label\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"description\": {\n \"type\": \"string\"\n },\n \"effectsummary\": {\n \"type\": \"string\"\n }\n }\n },\n \"humanresolutionrequest\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"requestid\",\n \"triggercriterionids\",\n \"questiontype\",\n \"question\",\n \"contextfindingids\",\n \"options\",\n \"decisionchanging\",\n \"hideprovisionaldecision\",\n \"status\",\n \"createdat\"\n ],\n \"properties\": {\n \"requestid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"triggercriterionids\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"questiontype\": {\n \"type\": \"string\",\n \"enum\": [\n \"applicability\",\n \"instruction_precedence\",\n \"evidence_authority\",\n \"inference_permission\",\n \"severity\",\n \"valid_alternative\",\n \"rubric_ambiguity\",\n \"domain_expertise\"\n ]\n },\n \"question\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"contextfindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"options\": {\n \"type\": \"array\",\n \"minitems\": 2,\n \"items\": {\n \"$ref\": \"#/$defs/resolutionoption\"\n }\n },\n \"decisionchanging\": {\n \"type\": \"boolean\"\n },\n \"hideprovisionaldecision\": {\n \"type\": \"boolean\"\n },\n \"requiredexpertisetags\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"status\": {\n \"type\": \"string\",\n \"enum\": [\n \"pending\",\n \"answered\",\n \"cancelled\"\n ]\n },\n \"createdat\": {\n \"$ref\": \"#/$defs/datetime\"\n }\n }\n },\n \"humanresolutionresponse\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"requestid\",\n \"selectedoptionid\",\n \"responderid\",\n \"respondedat\"\n ],\n \"properties\": {\n \"requestid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"selectedoptionid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"responderid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"rationale\": {\n \"type\": \"string\"\n },\n \"respondedat\": {\n \"$ref\": \"#/$defs/datetime\"\n }\n }\n },\n \"humanresolutioninteraction\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"request\",\n \"decisiontarget\"\n ],\n \"properties\": {\n \"request\": {\n \"$ref\": \"#/$defs/humanresolutionrequest\"\n },\n \"response\": {\n \"$ref\": \"#/$defs/humanresolutionresponse\"\n },\n \"decisionbefore\": {\n \"$ref\": \"#/$defs/operationalaction\"\n },\n \"decisionafter\": {\n \"$ref\": \"#/$defs/operationalaction\"\n },\n \"changeddecision\": {\n \"type\": \"boolean\"\n },\n \"decisiontarget\": {\n \"$ref\": \"#/$defs/decisiontarget\"\n }\n }\n },\n \"feedbackitem\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"feedbackid\",\n \"audience\",\n \"feedbacktype\",\n \"findingids\",\n \"message\",\n \"priority\",\n \"repairvalidationstatus\"\n ],\n \"properties\": {\n \"feedbackid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"audience\": {\n \"type\": \"string\",\n \"enum\": [\n \"attempter\",\n \"reviewer\",\n \"project_owner\"\n ]\n },\n \"criterionid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"feedbacktype\": {\n \"type\": \"string\",\n \"enum\": [\n \"strength\",\n \"defect\",\n \"omission\",\n \"ambiguity\"\n ]\n },\n \"findingids\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"message\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"minimalrepair\": {\n \"type\": \"string\"\n },\n \"priority\": {\n \"type\": \"integer\",\n \"minimum\": 1\n },\n \"repairvalidationstatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"validated\",\n \"plausible_unvalidated\",\n \"not_applicable\"\n ]\n }\n }\n },\n \"annotationassessment\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"annotationpresent\",\n \"labelsupport\",\n \"rationalesupport\",\n \"criterionattribution\",\n \"evidencealignment\",\n \"severityalignment\",\n \"omittedpositivefindingids\",\n \"omittednegativefindingids\",\n \"unsupportedreviewerfindingids\",\n \"disagreementtypes\",\n \"confidence\",\n \"rationale\"\n ],\n \"properties\": {\n \"annotationpresent\": {\n \"type\": \"boolean\"\n },\n \"labelsupport\": {\n \"type\": \"string\",\n \"enum\": [\n \"supported\",\n \"partially_supported\",\n \"unsupported\",\n \"reasonable_disagreement\",\n \"rubric_ambiguous\",\n \"not_applicable\"\n ]\n },\n \"rationalesupport\": {\n \"type\": \"string\",\n \"enum\": [\n \"supported\",\n \"contains_unsupported_reasoning\",\n \"does_not_support_label\",\n \"missing\",\n \"not_applicable\"\n ]\n },\n \"criterionattribution\": {\n \"type\": \"string\",\n \"enum\": [\n \"correct\",\n \"partially_correct\",\n \"wrong_criterion\",\n \"unstated_preference\",\n \"not_applicable\"\n ]\n },\n \"evidencealignment\": {\n \"type\": \"string\",\n \"enum\": [\n \"aligned\",\n \"partially_aligned\",\n \"misquoted\",\n \"irrelevant\",\n \"missing\",\n \"not_applicable\"\n ]\n },\n \"severityalignment\": {\n \"type\": \"string\",\n \"enum\": [\n \"appropriate\",\n \"too_severe\",\n \"too_lenient\",\n \"uncertain\",\n \"not_applicable\"\n ]\n },\n \"omittedpositivefindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"omittednegativefindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"unsupportedreviewerfindingids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"disagreementtypes\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\",\n \"enum\": [\n \"none\",\n \"observation\",\n \"evidence_authority\",\n \"applicability\",\n \"criterion_interpretation\",\n \"inference\",\n \"threshold\",\n \"severity\",\n \"attention\",\n \"expertise\",\n \"reasonable_disagreement\",\n \"rubric_defect\"\n ]\n }\n },\n \"confidence\": {\n \"$ref\": \"#/$defs/confidence\"\n },\n \"rationale\": {\n \"type\": \"string\"\n }\n }\n },\n \"modelrun\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"modelrunid\",\n \"role\",\n \"provider\",\n \"modelname\",\n \"prompttemplateid\",\n \"prompttemplateversion\",\n \"prompttemplatehash\",\n \"startedat\",\n \"completedat\"\n ],\n \"properties\": {\n \"modelrunid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"role\": {\n \"type\": \"string\",\n \"enum\": [\n \"claim_extractor\",\n \"evidence_retriever\",\n \"positive_prover\",\n \"negative_challenger\",\n \"coverage_auditor\",\n \"warrant_auditor\",\n \"criterion_adjudicator\",\n \"holistic_judge\",\n \"annotation_auditor\",\n \"feedback_generator\",\n \"repair_validator\",\n \"ensemble_aggregator\"\n ]\n },\n \"provider\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"modelname\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"modelversion\": {\n \"type\": \"string\"\n },\n \"prompttemplateid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"prompttemplateversion\": {\n \"$ref\": \"#/$defs/semver\"\n },\n \"prompttemplatehash\": {\n \"$ref\": \"#/$defs/contenthash\"\n },\n \"parameters\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n },\n \"parentrunids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"#/$defs/identifier\"\n }\n },\n \"inputhash\": {\n \"$ref\": \"#/$defs/contenthash\"\n },\n \"outputhash\": {\n \"$ref\": \"#/$defs/contenthash\"\n },\n \"startedat\": {\n \"$ref\": \"#/$defs/datetime\"\n },\n \"completedat\": {\n \"$ref\": \"#/$defs/datetime\"\n }\n }\n },\n \"decisiontarget\": {\n \"type\": \"string\",\n \"enum\": [\n \"attempt\",\n \"annotation\",\n \"composite_case\",\n \"batch\"\n ]\n },\n \"validatorrun\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"validatorrunid\",\n \"validatorid\",\n \"validatorversion\",\n \"role\",\n \"status\",\n \"inputhash\",\n \"outputhash\",\n \"startedat\",\n \"completedat\"\n ],\n \"properties\": {\n \"validatorrunid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"validatorid\": {\n \"$ref\": \"#/$defs/identifier\"\n },\n \"validatorversion\": {\n \"$ref\": \"#/$defs/semver\"\n },\n \"role\": {\n \"type\": \"string\",\n \"enum\": [\n \"schema_validation\",\n \"format_check\",\n \"citation_check\",\n \"calculation\",\n \"code_execution\",\n \"duplicate_detection\",\n \"integrity_check\",\n \"custom\"\n ]\n },\n \"status\": {\n \"type\": \"string\",\n \"enum\": [\n \"passed\",\n \"failed\",\n \"warning\",\n \"error\"\n ]\n },\n \"parameters\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n },\n \"inputhash\": {\n \"$ref\": \"#/$defs/contenthash\"\n },\n \"outputhash\": {\n \"$ref\": \"#/$defs/contenthash\"\n },\n \"startedat\": {\n \"$ref\": \"#/$defs/datetime\"\n },\n \"completedat\": {\n \"$ref\": \"#/$defs/datetime\"\n }\n }\n }\n }\n}\n", "30": "json schema - evaluation output foundation schemas runtime schema for claims, evidence relationships, findings, criterion assessments, decisions, annotation audit, and feedback. schema runtime validation {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://autoqa.example/schemas/evaluation-output.schema.json\",\n \"title\": \"autoqa evaluation output\",\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"documenttype\",\n \"schemaversion\",\n \"evaluationid\",\n \"evaluationversion\",\n \"caseref\",\n \"contractref\",\n \"createdat\",\n \"status\",\n \"runmanifest\",\n \"validatormanifest\",\n \"findings\",\n \"claims\",\n \"expectedelements\",\n \"evidencerelationships\",\n \"criterionassessments\",\n \"globalassessments\",\n \"provisionaldecisions\",\n \"finaldecisions\",\n \"feedback\",\n \"audit\"\n ],\n \"properties\": {\n \"documenttype\": {\n \"const\": \"evaluation_output\"\n },\n \"schemaversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"evaluationid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"evaluationversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"caseref\": {\n \"$ref\": \"common.schema.json#/$defs/caseref\"\n },\n \"contractref\": {\n \"$ref\": \"common.schema.json#/$defs/contractref\"\n },\n \"createdat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"status\": {\n \"type\": \"string\",\n \"enum\": [\n \"provisional\",\n \"awaiting_human\",\n \"final\",\n \"failed\"\n ]\n },\n \"runmanifest\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/modelrun\"\n }\n },\n \"integritystatus\": {\n \"type\": \"string\",\n \"enum\": [\n \"clean\",\n \"warning\",\n \"failed\",\n \"unknown\"\n ]\n },\n \"findings\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/finding\"\n }\n },\n \"claims\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/claim\"\n }\n },\n \"expectedelements\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/expectedelement\"\n }\n },\n \"evidencerelationships\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/evidencerelationship\"\n }\n },\n \"criterionassessments\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/criterionassessment\"\n }\n },\n \"globalassessments\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/globalassessment\"\n }\n },\n \"annotationassessment\": {\n \"$ref\": \"common.schema.json#/$defs/annotationassessment\"\n },\n \"humanresolution\": {\n \"$ref\": \"common.schema.json#/$defs/humanresolutioninteraction\"\n },\n \"feedback\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/feedbackitem\"\n }\n },\n \"audit\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"caseinputhash\",\n \"contracthash\",\n \"sourcesnapshothashes\",\n \"replayable\",\n \"warnings\"\n ],\n \"properties\": {\n \"caseinputhash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n },\n \"contracthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n },\n \"sourcesnapshothashes\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n }\n },\n \"replayable\": {\n \"type\": \"boolean\"\n },\n \"warnings\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"provisionaldecisions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/decision\"\n }\n },\n \"finaldecisions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/decision\"\n }\n },\n \"validatormanifest\": {\n \"type\": \"array\",\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/validatorrun\"\n }\n }\n }\n}\n", "31": "json schema - human resolution packet foundation schemas runtime schema for one bounded, decision-changing human question and its context. schema runtime validation {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://autoqa.example/schemas/human-resolution-packet.schema.json\",\n \"title\": \"autoqa human resolution packet\",\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"documenttype\",\n \"schemaversion\",\n \"caseref\",\n \"contractref\",\n \"contextfindings\",\n \"interaction\"\n ],\n \"properties\": {\n \"documenttype\": {\n \"const\": \"human_resolution_packet\"\n },\n \"schemaversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"caseref\": {\n \"$ref\": \"common.schema.json#/$defs/caseref\"\n },\n \"contractref\": {\n \"$ref\": \"common.schema.json#/$defs/contractref\"\n },\n \"interaction\": {\n \"$ref\": \"common.schema.json#/$defs/humanresolutioninteraction\"\n },\n \"contextfindings\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/finding\"\n }\n }\n }\n}\n", "32": "json schema - project contract foundation schemas runtime schema for approved project instructions, criteria, evidence policy, decision policy, and calibration requirements. schema runtime validation {\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://autoqa.example/schemas/project-contract.schema.json\",\n \"title\": \"autoqa project evaluation contract\",\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"documenttype\",\n \"schemaversion\",\n \"contractid\",\n \"contractversion\",\n \"projectname\",\n \"status\",\n \"createdat\",\n \"instructiondocuments\",\n \"sourcepolicy\",\n \"criteria\",\n \"decisiontargets\",\n \"decisionpolicy\",\n \"humanreviewpolicy\",\n \"feedbackpolicy\",\n \"securitypolicy\",\n \"auditpolicy\"\n ],\n \"properties\": {\n \"documenttype\": {\n \"const\": \"project_contract\"\n },\n \"schemaversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"contractid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"contractversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"projectname\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"description\": {\n \"type\": \"string\"\n },\n \"status\": {\n \"type\": \"string\",\n \"enum\": [\n \"draft\",\n \"approved\",\n \"deprecated\",\n \"retired\"\n ]\n },\n \"createdat\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"effectivefrom\": {\n \"$ref\": \"common.schema.json#/$defs/datetime\"\n },\n \"supersedescontractversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"owners\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"actorid\",\n \"role\"\n ],\n \"properties\": {\n \"actorid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"role\": {\n \"type\": \"string\",\n \"enum\": [\n \"project_owner\",\n \"rubric_owner\",\n \"domain_expert\",\n \"qa_owner\"\n ]\n }\n }\n }\n },\n \"instructiondocuments\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"instructionid\",\n \"title\",\n \"authority\",\n \"content\",\n \"contenthash\"\n ],\n \"properties\": {\n \"instructionid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"title\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"authority\": {\n \"type\": \"string\",\n \"enum\": [\n \"primary\",\n \"supplemental\",\n \"example\",\n \"historical\"\n ]\n },\n \"content\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"contenthash\": {\n \"$ref\": \"common.schema.json#/$defs/contenthash\"\n },\n \"precedencerank\": {\n \"type\": \"integer\",\n \"minimum\": 1\n }\n }\n }\n },\n \"sourcepolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"worldmode\",\n \"parametricknowledgepolicy\",\n \"externalretrievalallowed\",\n \"defaultcitationrequirement\",\n \"evidenceclasses\",\n \"conflictpolicy\",\n \"snapshotrequired\"\n ],\n \"properties\": {\n \"worldmode\": {\n \"type\": \"string\",\n \"enum\": [\n \"closed_world\",\n \"open_world\",\n \"hybrid\"\n ]\n },\n \"parametricknowledgepolicy\": {\n \"type\": \"string\",\n \"enum\": [\n \"never_evidence\",\n \"common_knowledge_only\",\n \"allowed_with_verification\"\n ]\n },\n \"externalretrievalallowed\": {\n \"type\": \"boolean\"\n },\n \"defaultcitationrequirement\": {\n \"type\": \"string\",\n \"enum\": [\n \"none\",\n \"material_claims\",\n \"all_verifiable_claims\",\n \"project_defined\"\n ]\n },\n \"snapshotrequired\": {\n \"type\": \"boolean\"\n },\n \"conflictpolicy\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"freshnessrules\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"evidenceclasses\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"classid\",\n \"name\",\n \"permitted\",\n \"authorityrank\"\n ],\n \"properties\": {\n \"classid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"name\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"description\": {\n \"type\": \"string\"\n },\n \"permitted\": {\n \"type\": \"boolean\"\n },\n \"authorityrank\": {\n \"type\": \"integer\",\n \"minimum\": 0\n }\n }\n }\n }\n }\n },\n \"criteria\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"criterionid\",\n \"name\",\n \"requirement\",\n \"intent\",\n \"criteriontype\",\n \"scope\",\n \"subjectivity\",\n \"instructionrefs\",\n \"applicability\",\n \"proofpolicy\",\n \"evidencepolicy\",\n \"anchors\",\n \"dependencies\",\n \"precedence\",\n \"defaultseverity\",\n \"defaultdecisioneffect\",\n \"tags\"\n ],\n \"properties\": {\n \"criterionid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"name\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"requirement\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"intent\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"criteriontype\": {\n \"type\": \"string\",\n \"enum\": [\n \"required_presence\",\n \"prohibited_presence\",\n \"conditional\",\n \"factual\",\n \"grounding\",\n \"inferential_warrant\",\n \"comparative\",\n \"completeness\",\n \"holistic_quality\",\n \"mechanically_verifiable\",\n \"annotation_fidelity\"\n ]\n },\n \"scope\": {\n \"type\": \"string\",\n \"enum\": [\n \"attempt\",\n \"claim\",\n \"citation\",\n \"annotation\",\n \"global\",\n \"batch\"\n ]\n },\n \"subjectivity\": {\n \"type\": \"string\",\n \"enum\": [\n \"objective\",\n \"bounded_judgment\",\n \"holistic_expert\"\n ]\n },\n \"instructionrefs\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"instructionid\",\n \"locator\"\n ],\n \"properties\": {\n \"instructionid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"locator\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"quotedtext\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"applicability\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"ruletext\",\n \"condition\"\n ],\n \"properties\": {\n \"ruletext\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"condition\": {\n \"$ref\": \"common.schema.json#/$defs/condition\"\n }\n }\n },\n \"proofpolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"positiveproofrequired\",\n \"positiveproofstandard\",\n \"failurecondition\",\n \"passonnoviolation\",\n \"allowpartialcredit\",\n \"allowcompensation\"\n ],\n \"properties\": {\n \"positiveproofrequired\": {\n \"type\": \"boolean\"\n },\n \"positiveproofstandard\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"failurecondition\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"omissioncondition\": {\n \"type\": \"string\"\n },\n \"passonnoviolation\": {\n \"type\": \"boolean\"\n },\n \"allowpartialcredit\": {\n \"type\": \"boolean\"\n },\n \"allowcompensation\": {\n \"type\": \"boolean\"\n }\n }\n },\n \"evidencepolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"allowedevidenceclassids\",\n \"minimumauthorityrank\",\n \"citationrequired\",\n \"externalverificationmode\",\n \"deterministicvalidatorids\"\n ],\n \"properties\": {\n \"allowedevidenceclassids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n }\n },\n \"minimumauthorityrank\": {\n \"type\": \"integer\",\n \"minimum\": 0\n },\n \"citationrequired\": {\n \"type\": \"boolean\"\n },\n \"externalverificationmode\": {\n \"type\": \"string\",\n \"enum\": [\n \"forbidden\",\n \"optional\",\n \"required\"\n ]\n },\n \"deterministicvalidatorids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n }\n }\n }\n },\n \"anchors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"anchorid\",\n \"label\",\n \"example\",\n \"expectedverdict\"\n ],\n \"properties\": {\n \"anchorid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"label\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"description\": {\n \"type\": \"string\"\n },\n \"example\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"expectedverdict\": {\n \"$ref\": \"common.schema.json#/$defs/criterionverdict\"\n },\n \"notes\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"validalternatives\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"exceptions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"dependencies\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"requiredbeforecriterionids\",\n \"impliescriterionids\"\n ],\n \"properties\": {\n \"requiredbeforecriterionids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n }\n },\n \"impliescriterionids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n }\n }\n }\n },\n \"precedence\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"overridescriterionids\",\n \"overriddenbycriterionids\"\n ],\n \"properties\": {\n \"overridescriterionids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n }\n },\n \"overriddenbycriterionids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n }\n }\n }\n },\n \"defaultseverity\": {\n \"$ref\": \"common.schema.json#/$defs/severity\"\n },\n \"defaultdecisioneffect\": {\n \"type\": \"string\",\n \"enum\": [\n \"none\",\n \"accept_with_note\",\n \"rework\",\n \"reject\",\n \"escalate\"\n ]\n },\n \"tags\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n }\n },\n \"decisionpolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"policyversion\",\n \"allowedactions\",\n \"rules\",\n \"defaultaction\",\n \"tiebreakorder\",\n \"confidencepolicy\"\n ],\n \"properties\": {\n \"policyversion\": {\n \"$ref\": \"common.schema.json#/$defs/semver\"\n },\n \"allowedactions\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n }\n },\n \"rules\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"target\",\n \"ruleid\",\n \"priority\",\n \"description\",\n \"match\",\n \"action\",\n \"terminal\"\n ],\n \"properties\": {\n \"ruleid\": {\n \"$ref\": \"common.schema.json#/$defs/identifier\"\n },\n \"priority\": {\n \"type\": \"integer\",\n \"minimum\": 1\n },\n \"description\": {\n \"type\": \"string\",\n \"minlength\": 1\n },\n \"match\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"properties\": {\n \"criterionids\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"criteriontags\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\"\n }\n },\n \"verdicts\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/criterionverdict\"\n }\n },\n \"severities\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/severity\"\n }\n },\n \"decisioneffects\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\",\n \"enum\": [\n \"none\",\n \"accept_with_note\",\n \"rework\",\n \"reject\",\n \"escalate\"\n ]\n }\n },\n \"minimumcalibratedconfidence\": {\n \"type\": \"number\",\n \"minimum\": 0,\n \"maximum\": 1\n },\n \"hasunresolvedfindings\": {\n \"type\": \"boolean\"\n },\n \"annotationdisagreementtypes\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"minproperties\": 1\n },\n \"action\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n },\n \"terminal\": {\n \"type\": \"boolean\"\n },\n \"explanationtemplate\": {\n \"type\": \"string\"\n },\n \"target\": {\n \"$ref\": \"common.schema.json#/$defs/decisiontarget\"\n }\n }\n }\n },\n \"defaultaction\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n },\n \"tiebreakorder\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n }\n },\n \"confidencepolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"requirescalibrationforactions\",\n \"uncalibratedaction\"\n ],\n \"properties\": {\n \"requirescalibrationforactions\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n }\n },\n \"uncalibratedaction\": {\n \"$ref\": \"common.schema.json#/$defs/operationalaction\"\n },\n \"minimumcalibrationcasesperslice\": {\n \"type\": \"integer\",\n \"minimum\": 1\n }\n }\n }\n }\n },\n \"humanreviewpolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"maxinteractionspercase\",\n \"onlyaskifdecisionchanging\",\n \"hideprovisionaldecision\",\n \"requireambiguityoption\",\n \"allowedquestiontypes\"\n ],\n \"properties\": {\n \"maxinteractionspercase\": {\n \"type\": \"integer\",\n \"minimum\": 0,\n \"maximum\": 1\n },\n \"onlyaskifdecisionchanging\": {\n \"type\": \"boolean\"\n },\n \"hideprovisionaldecision\": {\n \"type\": \"boolean\"\n },\n \"requireambiguityoption\": {\n \"type\": \"boolean\"\n },\n \"allowedquestiontypes\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\",\n \"enum\": [\n \"applicability\",\n \"instruction_precedence\",\n \"evidence_authority\",\n \"inference_permission\",\n \"severity\",\n \"valid_alternative\",\n \"rubric_ambiguity\",\n \"domain_expertise\"\n ]\n }\n },\n \"expertiseroutingrules\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"feedbackpolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"audiences\",\n \"maxitemsperaudience\",\n \"includevalidatedstrengths\",\n \"validaterepairs\",\n \"priorityorder\"\n ],\n \"properties\": {\n \"audiences\": {\n \"type\": \"array\",\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\",\n \"enum\": [\n \"attempter\",\n \"reviewer\",\n \"project_owner\"\n ]\n }\n },\n \"maxitemsperaudience\": {\n \"type\": \"integer\",\n \"minimum\": 1\n },\n \"includevalidatedstrengths\": {\n \"type\": \"boolean\"\n },\n \"validaterepairs\": {\n \"type\": \"boolean\"\n },\n \"priorityorder\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"type\": \"string\",\n \"enum\": [\n \"decisive\",\n \"critical\",\n \"major\",\n \"minor\",\n \"strength\"\n ]\n }\n }\n }\n },\n \"securitypolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"treatcandidatecontentasuntrusted\",\n \"ignoreembeddedevaluatorinstructions\",\n \"requirestructuredoutputs\",\n \"allowcandidatetooverriderubric\"\n ],\n \"properties\": {\n \"treatcandidatecontentasuntrusted\": {\n \"type\": \"boolean\"\n },\n \"ignoreembeddedevaluatorinstructions\": {\n \"type\": \"boolean\"\n },\n \"requirestructuredoutputs\": {\n \"type\": \"boolean\"\n },\n \"allowcandidatetooverriderubric\": {\n \"type\": \"boolean\"\n },\n \"redactionrules\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"auditpolicy\": {\n \"type\": \"object\",\n \"additionalproperties\": false,\n \"required\": [\n \"retainexactsourcesnapshots\",\n \"retainpromptversions\",\n \"retainmodelrunmetadata\",\n \"retainhumanresolutions\",\n \"replayrequired\"\n ],\n \"properties\": {\n \"retainexactsourcesnapshots\": {\n \"type\": \"boolean\"\n },\n \"retainpromptversions\": {\n \"type\": \"boolean\"\n },\n \"retainmodelrunmetadata\": {\n \"type\": \"boolean\"\n },\n \"retainhumanresolutions\": {\n \"type\": \"boolean\"\n },\n \"replayrequired\": {\n \"type\": \"boolean\"\n },\n \"retentiondays\": {\n \"type\": \"integer\",\n \"minimum\": 1\n }\n }\n },\n \"metadata\": {\n \"type\": \"object\",\n \"additionalproperties\": true\n },\n \"decisiontargets\": {\n \"type\": \"array\",\n \"minitems\": 1,\n \"uniqueitems\": true,\n \"items\": {\n \"$ref\": \"common.schema.json#/$defs/decisiontarget\"\n }\n }\n }\n}\n", "33": "template - criterion authoring foundation templates csv template for drafting criterion intent, proof standards, failure anchors, evidence requirements, severity, and governance. template authoring criterion_id,name,requirement,intent,criterion_type,scope,subjectivity,instruction_id,instruction_locator,applicability_rule,positive_proof_standard,failure_condition,omission_condition,allowed_evidence_classes,minimum_authority_rank,citation_required,positive_anchor,negative_anchor,valid_alternatives,exceptions,dependencies,precedence,default_severity,default_decision_effect,hard_gate,owner,status,version\nc-example,example criterion,,,required_presence,attempt,bounded_judgment,,,always applies,,,,,0,false,,,,,,,major,rework,false,,draft,0.1.0\n", "34": "template - gold set foundation templates csv template for building an adjudicated calibration and test set. template authoring benchmark_item_id,benchmark_version,split,contract_id,contract_version,case_id,criterion_id,applicability,acceptable_verdicts,severity,positive_evidence_spans,negative_evidence_spans,omission_descriptions,acceptable_actions,ambiguity_status,challenge_attributes,slice_tags,adjudication_method,adjudicator_count,domain_expert_count,blind_to_autoqa,blind_to_reviewer_label,initial_agreement_rate,adjudication_notes,allowed_for_prompt_development,held_out_project,weight\nb-001,0.1.0,calibration,,,,,applies,met,none,,,,accept,clear,,,independent_then_adjudicate,3,1,true,true,,,false,false,1.0\n", "35": "template - model bakeoff foundation templates csv template for comparing evaluator configurations by criterion family. template authoring experiment_id,run_date,contract_id,benchmark_version,model_provider,model_name,model_version,judge_role,prompt_template_id,prompt_template_version,reasoning_mode,temperature,samples_per_item,reference_timing,candidate_order_policy,ensemble_policy,criterion_macro_f1,critical_false_accept_rate,strong_positive_false_reject_rate,evidence_span_precision,evidence_span_recall,positive_finding_precision,omission_recall,brier_score,ece,coverage_at_target_risk,mean_latency_ms,mean_cost_per_case,notes\nexp-001,,,,,,,criterion_adjudicator,,,default,0,1,late,randomized,none,,,,,,,,,,,,,\n", "36": "typescript domain model foundation developer files strict typescript types for project contracts, cases, evaluations, benchmarks, findings, decisions, feedback, and human resolution. typescript developer /**\n * autoqa foundation types - schema version 0.1.0\n *\n * these interfaces mirror the json schemas in ../schemas. runtime validation\n * must use the json schemas; typescript types alone are not a trust boundary.\n */\n\nexport type jsonprimitive = string | number | boolean | null;\nexport type jsonvalue = jsonprimitive | jsonvalue[] | { [key: string]: jsonvalue };\n\nexport type identifier = string;\nexport type semver = string;\nexport type contenthash = `sha256:${string}`;\nexport type datetime = string;\n\nexport type confidenceband = \"high\" | \"medium\" | \"low\" | \"uncalibrated\";\nexport type severity = \"none\" | \"advisory\" | \"minor\" | \"major\" | \"critical\";\nexport type decisiontarget = \"attempt\" | \"annotation\" | \"composite_case\" | \"batch\";\nexport type operationalaction =\n | \"accept\"\n | \"accept_with_notes\"\n | \"rework\"\n | \"reject\"\n | \"escalate\";\nexport type criterionverdict =\n | \"met\"\n | \"partially_met\"\n | \"not_met\"\n | \"ambiguous\"\n | \"unknown\"\n | \"not_applicable\";\n\nexport interface contractref {\n contractid: identifier;\n contractversion: semver;\n contenthash: contenthash;\n}\n\nexport interface caseref {\n caseid: identifier;\n contenthash?: contenthash;\n}\n\nexport interface spanref {\n artifacttype:\n | \"task\"\n | \"evidence\"\n | \"attempt\"\n | \"human_review\"\n | \"reference\"\n | \"tool_trace\";\n artifactid: identifier;\n startchar?: number;\n endchar?: number;\n quotedtext: string;\n locator?: string;\n}\n\nexport interface confidence {\n /** model-native or component-native confidence. never use alone for autonomy. */\n rawscore?: number;\n /** empirically calibrated probability for the defined event and slice. */\n calibratedscore?: number;\n band: confidenceband;\n calibrationprofileid?: identifier;\n rationale?: string;\n}\n\nexport interface generationsource {\n kind: \"deterministic_validator\" | \"llm_judge\" | \"human\" | \"hybrid\";\n componentid: identifier;\n modelrunid?: identifier;\n humanactorid?: identifier;\n}\n\nexport type condition =\n | { kind: \"always\" }\n | {\n kind: \"predicate\";\n fieldpath: string;\n operator:\n | \"equals\"\n | \"not_equals\"\n | \"in\"\n | \"not_in\"\n | \"contains\"\n | \"exists\"\n | \"greater_than\"\n | \"greater_than_or_equal\"\n | \"less_than\"\n | \"less_than_or_equal\"\n | \"matches_regex\";\n value?: jsonvalue;\n }\n | { kind: \"all\"; conditions: condition[] }\n | { kind: \"any\"; conditions: condition[] }\n | { kind: \"not\"; condition: condition };\n\nexport interface finding {\n findingid: identifier;\n findingtype:\n | \"positive\"\n | \"negative\"\n | \"omission\"\n | \"ambiguity\"\n | \"counterevidence\"\n | \"integrity\";\n criterionid?: identifier;\n summary: string;\n detail?: string;\n claimids?: identifier[];\n expectedelementids?: identifier[];\n targetspans?: spanref[];\n evidenceids?: identifier[];\n relationship?:\n | \"directly_entails\"\n | \"reasonably_supports\"\n | \"weakly_supports\"\n | \"contradicts\"\n | \"irrelevant\"\n | \"insufficient\"\n | \"not_applicable\";\n materiality: \"incidental\" | \"supporting\" | \"central\" | \"decisive\";\n severity: severity;\n confidence: confidence;\n status: \"observed\" | \"challenged\" | \"confirmed\" | \"retracted\";\n source: generationsource;\n}\n\nexport interface claim {\n claimid: identifier;\n proposition: string;\n claimtype:\n | \"factual\"\n | \"interpretive\"\n | \"predictive\"\n | \"normative\"\n | \"preference\"\n | \"procedural\";\n importance: \"incidental\" | \"supporting\" | \"central\";\n explicitness: \"explicit\" | \"implied\";\n attemptspans: spanref[];\n qualifiers?: string[];\n citedevidenceids?: identifier[];\n}\n\nexport interface expectedelement {\n elementid: identifier;\n criterionid: identifier;\n description: string;\n required: boolean;\n importance: \"supporting\" | \"central\" | \"decisive\";\n acceptablealternatives?: string[];\n observedspans?: spanref[];\n coverage: \"complete\" | \"partial\" | \"missing\" | \"not_applicable\" | \"unknown\";\n}\n\nexport interface evidencerelationship {\n relationshipid: identifier;\n subjecttype: \"claim\" | \"expected_element\" | \"finding\";\n subjectid: identifier;\n evidenceid: identifier;\n relationship:\n | \"directly_entails\"\n | \"reasonably_supports\"\n | \"weakly_supports\"\n | \"contradicts\"\n | \"irrelevant\"\n | \"insufficient\";\n sourcequality: \"high\" | \"acceptable\" | \"weak\" | \"disallowed\" | \"unknown\";\n temporalvalidity: \"valid\" | \"stale\" | \"not_time_sensitive\" | \"unknown\";\n evidencespans?: spanref[];\n confidence: confidence;\n notes?: string;\n}\n\nexport interface criterionassessment {\n criterionid: identifier;\n applicability: \"applies\" | \"does_not_apply\" | \"uncertain\";\n positivefindingids: identifier[];\n negativefindingids: identifier[];\n omissionfindingids: identifier[];\n counterevidencefindingids: identifier[];\n unresolvedfindingids: identifier[];\n groundingstatus:\n | \"grounded\"\n | \"partially_grounded\"\n | \"ungrounded\"\n | \"not_applicable\"\n | \"unknown\";\n truthstatus:\n | \"verified_true\"\n | \"verified_false\"\n | \"mixed\"\n | \"not_evaluated\"\n | \"not_applicable\"\n | \"unknown\";\n warrantstatus: \"justified\" | \"overstated\" | \"invalid\" | \"not_applicable\" | \"unknown\";\n relevancestatus:\n | \"relevant\"\n | \"partially_relevant\"\n | \"irrelevant\"\n | \"not_applicable\"\n | \"unknown\";\n verdict: criterionverdict;\n severity: severity;\n confidence: confidence;\n decisioneffect: \"none\" | \"accept_with_note\" | \"rework\" | \"reject\" | \"escalate\";\n rationale: string;\n}\n\nexport interface globalassessment {\n axisid: identifier;\n verdict: criterionverdict;\n findingids: identifier[];\n score?: number;\n confidence: confidence;\n rationale: string;\n}\n\nexport interface policytraceentry {\n ruleid: identifier;\n matched: boolean;\n inputs?: record<string, jsonvalue>;\n result: string;\n}\n\nexport interface decision {\n target: decisiontarget;\n action: operationalaction;\n policyversion: semver;\n triggeredruleids: identifier[];\n decisivecriterionids: identifier[];\n unresolvedfindingids: identifier[];\n confidence: confidence;\n policytrace: policytraceentry[];\n rationale: string;\n decidedat: datetime;\n}\n\nexport interface resolutionoption {\n optionid: identifier;\n label: string;\n description?: string;\n effectsummary?: string;\n}\n\nexport interface humanresolutionrequest {\n requestid: identifier;\n triggercriterionids: identifier[];\n questiontype:\n | \"applicability\"\n | \"instruction_precedence\"\n | \"evidence_authority\"\n | \"inference_permission\"\n | \"severity\"\n | \"valid_alternative\"\n | \"rubric_ambiguity\"\n | \"domain_expertise\";\n question: string;\n contextfindingids: identifier[];\n options: resolutionoption[];\n decisionchanging: boolean;\n hideprovisionaldecision: boolean;\n requiredexpertisetags?: string[];\n status: \"pending\" | \"answered\" | \"cancelled\";\n createdat: datetime;\n}\n\nexport interface humanresolutionresponse {\n requestid: identifier;\n selectedoptionid: identifier;\n responderid: identifier;\n rationale?: string;\n respondedat: datetime;\n}\n\nexport interface humanresolutioninteraction {\n decisiontarget: decisiontarget;\n request: humanresolutionrequest;\n response?: humanresolutionresponse;\n decisionbefore?: operationalaction;\n decisionafter?: operationalaction;\n changeddecision?: boolean;\n}\n\n\nexport interface humanresolutionpacket {\n documenttype: \"human_resolution_packet\";\n schemaversion: semver;\n caseref: caseref;\n contractref: contractref;\n contextfindings: finding[];\n interaction: humanresolutioninteraction;\n}\n\nexport interface feedbackitem {\n feedbackid: identifier;\n audience: \"attempter\" | \"reviewer\" | \"project_owner\";\n criterionid?: identifier;\n feedbacktype: \"strength\" | \"defect\" | \"omission\" | \"ambiguity\";\n findingids: identifier[];\n message: string;\n minimalrepair?: string;\n priority: number;\n repairvalidationstatus: \"validated\" | \"plausible_unvalidated\" | \"not_applicable\";\n}\n\nexport interface annotationassessment {\n annotationpresent: boolean;\n labelsupport:\n | \"supported\"\n | \"partially_supported\"\n | \"unsupported\"\n | \"reasonable_disagreement\"\n | \"rubric_ambiguous\"\n | \"not_applicable\";\n rationalesupport:\n | \"supported\"\n | \"contains_unsupported_reasoning\"\n | \"does_not_support_label\"\n | \"missing\"\n | \"not_applicable\";\n criterionattribution:\n | \"correct\"\n | \"partially_correct\"\n | \"wrong_criterion\"\n | \"unstated_preference\"\n | \"not_applicable\";\n evidencealignment:\n | \"aligned\"\n | \"partially_aligned\"\n | \"misquoted\"\n | \"irrelevant\"\n | \"missing\"\n | \"not_applicable\";\n severityalignment: \"appropriate\" | \"too_severe\" | \"too_lenient\" | \"uncertain\" | \"not_applicable\";\n omittedpositivefindingids: identifier[];\n omittednegativefindingids: identifier[];\n unsupportedreviewerfindingids: identifier[];\n disagreementtypes: array<\n | \"none\"\n | \"observation\"\n | \"evidence_authority\"\n | \"applicability\"\n | \"criterion_interpretation\"\n | \"inference\"\n | \"threshold\"\n | \"severity\"\n | \"attention\"\n | \"expertise\"\n | \"reasonable_disagreement\"\n | \"rubric_defect\"\n >;\n confidence: confidence;\n rationale: string;\n}\n\nexport interface modelrun {\n modelrunid: identifier;\n role:\n | \"claim_extractor\"\n | \"evidence_retriever\"\n | \"positive_prover\"\n | \"negative_challenger\"\n | \"coverage_auditor\"\n | \"warrant_auditor\"\n | \"criterion_adjudicator\"\n | \"holistic_judge\"\n | \"annotation_auditor\"\n | \"feedback_generator\"\n | \"repair_validator\"\n | \"ensemble_aggregator\";\n provider: string;\n modelname: string;\n modelversion?: string;\n prompttemplateid: identifier;\n prompttemplateversion: semver;\n prompttemplatehash: contenthash;\n parameters?: record<string, jsonvalue>;\n parentrunids?: identifier[];\n inputhash?: contenthash;\n outputhash?: contenthash;\n startedat: datetime;\n completedat: datetime;\n}\n\nexport interface validatorrun {\n validatorrunid: identifier;\n validatorid: identifier;\n validatorversion: semver;\n role:\n | \"schema_validation\"\n | \"format_check\"\n | \"citation_check\"\n | \"calculation\"\n | \"code_execution\"\n | \"duplicate_detection\"\n | \"integrity_check\"\n | \"custom\";\n status: \"passed\" | \"failed\" | \"warning\" | \"error\";\n parameters?: record<string, jsonvalue>;\n inputhash: contenthash;\n outputhash: contenthash;\n startedat: datetime;\n completedat: datetime;\n}\n\nexport interface instructiondocument {\n instructionid: identifier;\n title: string;\n authority: \"primary\" | \"supplemental\" | \"example\" | \"historical\";\n content: string;\n contenthash: contenthash;\n precedencerank?: number;\n}\n\nexport interface evidenceclass {\n classid: identifier;\n name: string;\n description?: string;\n permitted: boolean;\n authorityrank: number;\n}\n\nexport interface instructionref {\n instructionid: identifier;\n locator: string;\n quotedtext?: string;\n}\n\nexport interface criterionanchor {\n anchorid: identifier;\n label: string;\n description?: string;\n example: string;\n expectedverdict: criterionverdict;\n notes?: string;\n}\n\nexport interface criteriondefinition {\n criterionid: identifier;\n name: string;\n requirement: string;\n intent: string;\n criteriontype:\n | \"required_presence\"\n | \"prohibited_presence\"\n | \"conditional\"\n | \"factual\"\n | \"grounding\"\n | \"inferential_warrant\"\n | \"comparative\"\n | \"completeness\"\n | \"holistic_quality\"\n | \"mechanically_verifiable\"\n | \"annotation_fidelity\";\n scope: \"attempt\" | \"claim\" | \"citation\" | \"annotation\" | \"global\" | \"batch\";\n subjectivity: \"objective\" | \"bounded_judgment\" | \"holistic_expert\";\n instructionrefs: instructionref[];\n applicability: { ruletext: string; condition: condition };\n proofpolicy: {\n positiveproofrequired: boolean;\n positiveproofstandard: string;\n failurecondition: string;\n omissioncondition?: string;\n passonnoviolation: boolean;\n allowpartialcredit: boolean;\n allowcompensation: boolean;\n };\n evidencepolicy: {\n allowedevidenceclassids: identifier[];\n minimumauthorityrank: number;\n citationrequired: boolean;\n externalverificationmode: \"forbidden\" | \"optional\" | \"required\";\n deterministicvalidatorids: identifier[];\n };\n anchors: criterionanchor[];\n validalternatives?: string[];\n exceptions?: string[];\n dependencies: {\n requiredbeforecriterionids: identifier[];\n impliescriterionids: identifier[];\n };\n precedence: {\n overridescriterionids: identifier[];\n overriddenbycriterionids: identifier[];\n };\n defaultseverity: severity;\n defaultdecisioneffect: \"none\" | \"accept_with_note\" | \"rework\" | \"reject\" | \"escalate\";\n tags: string[];\n}\n\nexport interface decisionrulematch {\n criterionids?: string[];\n criteriontags?: string[];\n verdicts?: criterionverdict[];\n severities?: severity[];\n decisioneffects?: array<\"none\" | \"accept_with_note\" | \"rework\" | \"reject\" | \"escalate\">;\n minimumcalibratedconfidence?: number;\n hasunresolvedfindings?: boolean;\n annotationdisagreementtypes?: string[];\n}\n\nexport interface decisionrule {\n target: decisiontarget;\n ruleid: identifier;\n priority: number;\n description: string;\n match: decisionrulematch;\n action: operationalaction;\n terminal: boolean;\n explanationtemplate?: string;\n}\n\nexport interface projectcontract {\n documenttype: \"project_contract\";\n schemaversion: semver;\n contractid: identifier;\n contractversion: semver;\n projectname: string;\n description?: string;\n status: \"draft\" | \"approved\" | \"deprecated\" | \"retired\";\n createdat: datetime;\n effectivefrom?: datetime;\n supersedescontractversion?: semver;\n owners?: array<{\n actorid: identifier;\n role: \"project_owner\" | \"rubric_owner\" | \"domain_expert\" | \"qa_owner\";\n }>;\n instructiondocuments: instructiondocument[];\n sourcepolicy: {\n worldmode: \"closed_world\" | \"open_world\" | \"hybrid\";\n parametricknowledgepolicy: \"never_evidence\" | \"common_knowledge_only\" | \"allowed_with_verification\";\n externalretrievalallowed: boolean;\n defaultcitationrequirement: \"none\" | \"material_claims\" | \"all_verifiable_claims\" | \"project_defined\";\n snapshotrequired: boolean;\n conflictpolicy: string;\n freshnessrules?: string[];\n evidenceclasses: evidenceclass[];\n };\n criteria: criteriondefinition[];\n decisiontargets: decisiontarget[];\n decisionpolicy: {\n policyversion: semver;\n allowedactions: operationalaction[];\n rules: decisionrule[];\n defaultaction: operationalaction;\n tiebreakorder: operationalaction[];\n confidencepolicy: {\n requirescalibrationforactions: operationalaction[];\n uncalibratedaction: operationalaction;\n minimumcalibrationcasesperslice?: number;\n };\n };\n humanreviewpolicy: {\n maxinteractionspercase: 0 | 1;\n onlyaskifdecisionchanging: boolean;\n hideprovisionaldecision: boolean;\n requireambiguityoption: boolean;\n allowedquestiontypes: humanresolutionrequest[\"questiontype\"][];\n expertiseroutingrules?: string[];\n };\n feedbackpolicy: {\n audiences: feedbackitem[\"audience\"][];\n maxitemsperaudience: number;\n includevalidatedstrengths: boolean;\n validaterepairs: boolean;\n priorityorder: array<\"decisive\" | \"critical\" | \"major\" | \"minor\" | \"strength\">;\n };\n securitypolicy: {\n treatcandidatecontentasuntrusted: boolean;\n ignoreembeddedevaluatorinstructions: boolean;\n requirestructuredoutputs: boolean;\n allowcandidatetooverriderubric: boolean;\n redactionrules?: string[];\n };\n auditpolicy: {\n retainexactsourcesnapshots: boolean;\n retainpromptversions: boolean;\n retainmodelrunmetadata: boolean;\n retainhumanresolutions: boolean;\n replayrequired: boolean;\n retentiondays?: number;\n };\n metadata?: record<string, jsonvalue>;\n}\n\nexport interface taskinput {\n taskid: identifier;\n tasktype: string;\n prompt: string;\n taskinstructions?: string[];\n expectedoutputformat?: string;\n contenthash: contenthash;\n metadata?: record<string, jsonvalue>;\n}\n\nexport interface evidenceartifact {\n evidenceid: identifier;\n evidenceclassid: identifier;\n title: string;\n content: string;\n contenthash: contenthash;\n sourceuri?: string;\n authorityrank: number;\n permitted: boolean;\n publishedat?: datetime;\n capturedat: datetime;\n metadata?: record<string, jsonvalue>;\n}\n\nexport interface attemptinput {\n attemptid: identifier;\n authorid?: identifier;\n content: string;\n contenthash: contenthash;\n submittedat: datetime;\n attachments?: array<{\n artifactid: identifier;\n artifacttype: string;\n content?: string;\n contenthash: contenthash;\n }>;\n tooltraces?: array<{\n traceid: identifier;\n toolname: string;\n input: jsonvalue;\n output: jsonvalue;\n contenthash: contenthash;\n }>;\n}\n\nexport interface humanreviewinput {\n annotationid: identifier;\n reviewerid: identifier;\n label: string;\n score?: number;\n reasoncodes: string[];\n rationale: string;\n criterionannotations?: array<{\n criterionid: identifier;\n verdict: criterionverdict;\n severity?: severity;\n rationale: string;\n citedspans?: spanref[];\n }>;\n reviewerconfidence?: confidence;\n submittedat: datetime;\n}\n\nexport interface caseinput {\n documenttype: \"case_input\";\n schemaversion: semver;\n caseid: identifier;\n createdat: datetime;\n contractref: contractref;\n task: taskinput;\n evidencebundle: evidenceartifact[];\n attempt: attemptinput;\n humanreview?: humanreviewinput;\n referenceanswers?: array<{\n referenceid: identifier;\n role: \"authoritative\" | \"illustrative\" | \"positive_anchor\" | \"negative_anchor\";\n content: string;\n contenthash: contenthash;\n }>;\n declaredexpectedelements?: expectedelement[];\n casetags: string[];\n metadata?: record<string, jsonvalue>;\n}\n\nexport interface evaluationoutput {\n documenttype: \"evaluation_output\";\n schemaversion: semver;\n evaluationid: identifier;\n evaluationversion: semver;\n caseref: caseref;\n contractref: contractref;\n createdat: datetime;\n status: \"provisional\" | \"awaiting_human\" | \"final\" | \"failed\";\n runmanifest: modelrun[];\n validatormanifest: validatorrun[];\n integritystatus?: \"clean\" | \"warning\" | \"failed\" | \"unknown\";\n findings: finding[];\n claims: claim[];\n expectedelements: expectedelement[];\n evidencerelationships: evidencerelationship[];\n criterionassessments: criterionassessment[];\n globalassessments: globalassessment[];\n annotationassessment?: annotationassessment;\n provisionaldecisions: decision[];\n humanresolution?: humanresolutioninteraction;\n finaldecisions: decision[];\n feedback: feedbackitem[];\n audit: {\n caseinputhash: contenthash;\n contracthash: contenthash;\n sourcesnapshothashes: contenthash[];\n replayable: boolean;\n warnings: string[];\n };\n}\n\nexport interface benchmarkcriteriongold {\n criterionid: identifier;\n applicability: \"applies\" | \"does_not_apply\" | \"uncertain\";\n acceptableverdicts: criterionverdict[];\n severity: severity;\n positiveevidencespans: spanref[];\n negativeevidencespans: spanref[];\n omissiondescriptions: string[];\n notes: string;\n}\n\nexport interface benchmarkitem {\n documenttype: \"benchmark_item\";\n schemaversion: semver;\n benchmarkitemid: identifier;\n benchmarkversion: semver;\n split: \"development\" | \"calibration\" | \"test\" | \"challenge\" | \"shadow\" | \"held_out_project\";\n contractref: contractref;\n caseref: caseref;\n slicetags: string[];\n challengeattributes: array<\n | \"strong_positive\"\n | \"subtle_valid_alternative\"\n | \"polished_falsehood\"\n | \"unsupported_rationale\"\n | \"citation_mismatch\"\n | \"partial_support\"\n | \"required_content_omission\"\n | \"severity_edge\"\n | \"instruction_conflict\"\n | \"ambiguous_case\"\n | \"prompt_injection\"\n | \"rubric_parroting\"\n | \"correct_label_wrong_reason\"\n | \"wrong_label_correct_observation\"\n | \"long_context\"\n | \"domain_expertise\"\n | \"position_bias_probe\"\n >;\n goldstandard: {\n criteriongold: benchmarkcriteriongold[];\n acceptabledecisions: array<{\n target: decisiontarget;\n acceptableactions: operationalaction[];\n }>;\n annotationgold?: {\n acceptablelabelsupport?: string[];\n expecteddisagreementtypes?: string[];\n notes?: string;\n };\n ambiguitystatus: \"clear\" | \"reasonable_disagreement\" | \"rubric_ambiguous\" | \"insufficient_evidence\";\n validalternativenotes?: string[];\n };\n adjudication: {\n method: \"single_expert\" | \"independent_then_adjudicate\" | \"consensus_panel\" | \"latent_label_model\";\n adjudicatorcount: number;\n domainexpertcount: number;\n blindtoautoqa: boolean;\n blindtoreviewerlabel: boolean;\n initialagreementrate?: number;\n disagreementtypes: string[];\n adjudicationnotes?: string;\n completedat: datetime;\n };\n leakagecontrol: {\n allowedforpromptdevelopment: boolean;\n heldoutproject: boolean;\n firstfrozenat: datetime;\n accesspolicy: string;\n };\n weight: number;\n exclusionreason?: string;\n metadata?: record<string, jsonvalue>;\n}\n\n/**\n * stable domain boundary for a vercel/typescript application.\n * the orchestration layer can accept and return these documents without\n * depending on any particular model provider.\n */\nexport interface autoqaevaluator {\n evaluate(caseinput: caseinput, contract: projectcontract): promise<evaluationoutput>;\n}\n", "37": "typescript configuration foundation developer files compiler configuration for the package's type definitions. typescript developer {\n \"compileroptions\": {\n \"target\": \"es2022\",\n \"module\": \"esnext\",\n \"moduleresolution\": \"bundler\",\n \"strict\": true,\n \"noemit\": true,\n \"skiplibcheck\": true\n },\n \"include\": [\"autoqa.ts\"]\n}\n", "38": "validation - hash utilities foundation validation canonical content hashing utilities used by the validation suite. validation python from __future__ import annotations\n\nimport hashlib\nimport json\nfrom pathlib import path\nfrom typing import any\n\n\ndef sha256_text(text: str) -> str:\n \"\"\"hash exact utf-8 text.\"\"\"\n return \"sha256:\" + hashlib.sha256(text.encode(\"utf-8\")).hexdigest()\n\n\ndef canonical_json_bytes(value: any) -> bytes:\n \"\"\"canonical json v1 used by the example package.\n\n object keys are sorted, arrays retain order, insignificant whitespace is\n removed, unicode is encoded as utf-8, and json numbers use python's normal\n json serialization. production systems that need cross-language numeric\n canonicalization should adopt a stricter standard and version it.\n \"\"\"\n return json.dumps(\n value,\n ensure_ascii=false,\n sort_keys=true,\n separators=(\",\", \":\"),\n allow_nan=false,\n ).encode(\"utf-8\")\n\n\ndef sha256_json(value: any) -> str:\n return \"sha256:\" + hashlib.sha256(canonical_json_bytes(value)).hexdigest()\n\n\ndef load_json(path: path) -> any:\n with path.open(\"r\", encoding=\"utf-8\") as handle:\n return json.load(handle)\n", "39": "validation - example schema checks foundation validation python validator for the package's example documents. validation python from __future__ import annotations\n\nimport json\nfrom pathlib import path\n\nfrom jsonschema import draft202012validator\nfrom referencing import registry, resource\n\nroot = path(__file__).resolve().parents[1]\nschema_dir = root / \"schemas\"\nexample_dir = root / \"examples\"\n\npairs = [\n (\"project-contract.schema.json\", \"project-contract.example.json\"),\n (\"case-input.schema.json\", \"case-input.example.json\"),\n (\"evaluation-output.schema.json\", \"evaluation-output.example.json\"),\n (\"benchmark-item.schema.json\", \"benchmark-item.example.json\"),\n (\"human-resolution-packet.schema.json\", \"human-resolution-packet.example.json\"),\n]\n\n\ndef load_json(path: path):\n with path.open(\"r\", encoding=\"utf-8\") as handle:\n return json.load(handle)\n\n\ndef build_registry() -> registry:\n registry = registry()\n for schema_path in schema_dir.glob(\"*.schema.json\"):\n schema = load_json(schema_path)\n draft202012validator.check_schema(schema)\n schema_id = schema.get(\"$id\")\n if not schema_id:\n raise valueerror(f\"schema has no $id: {schema_path}\")\n registry = registry.with_resource(schema_id, resource.from_contents(schema))\n return registry\n\n\ndef validate_examples() -> list[str]:\n registry = build_registry()\n failures: list[str] = []\n\n for schema_name, example_name in pairs:\n schema_path = schema_dir / schema_name\n example_path = example_dir / example_name\n schema = load_json(schema_path)\n instance = load_json(example_path)\n validator = draft202012validator(schema, registry=registry)\n errors = sorted(validator.iter_errors(instance), key=lambda e: list(e.absolute_path))\n if errors:\n failures.append(example_name)\n print(f\"fail {example_name}\")\n for error in errors:\n path = \".\".join(str(part) for part in error.absolute_path) or \"<root>\"\n print(f\" {path}: {error.message}\")\n else:\n print(f\"pass {example_name}\")\n\n return failures\n\n\ndef main() -> none:\n failures = validate_examples()\n if failures:\n raise systemexit(1)\n\n\nif __name__ == \"__main__\":\n main()\n", "40": "validation - package integrity foundation validation cross-document, schema, hash, proof-invariant, policy, and package integrity checks. validation python from __future__ import annotations\n\nimport json\nfrom collections import counter\nfrom pathlib import path\nfrom typing import any, iterable\n\nfrom hash_utils import load_json, sha256_json, sha256_text\nfrom validate_examples import validate_examples\n\nroot = path(__file__).resolve().parents[1]\nexamples = root / \"examples\"\n\n\ndef duplicates(values: iterable[str]) -> list[str]:\n counts = counter(values)\n return sorted(value for value, count in counts.items() if count > 1)\n\n\ndef check_hash(errors: list[str], label: str, actual: str, expected: str) -> none:\n if actual != expected:\n errors.append(f\"{label}: expected {expected}, found {actual}\")\n\n\ndef build_artifact_text(case: dict[str, any]) -> dict[tuple[str, str], str]:\n artifacts: dict[tuple[str, str], str] = {\n (\"task\", case[\"task\"][\"taskid\"]): case[\"task\"][\"prompt\"],\n (\"attempt\", case[\"attempt\"][\"attemptid\"]): case[\"attempt\"][\"content\"],\n }\n for evidence in case[\"evidencebundle\"]:\n artifacts[(\"evidence\", evidence[\"evidenceid\"])] = evidence[\"content\"]\n for reference in case.get(\"referenceanswers\", []):\n artifacts[(\"reference\", reference[\"referenceid\"])] = reference[\"content\"]\n for trace in case[\"attempt\"].get(\"tooltraces\", []):\n artifacts[(\"tool_trace\", trace[\"traceid\"])] = json.dumps(\n {\"input\": trace[\"input\"], \"output\": trace[\"output\"]},\n ensure_ascii=false,\n sort_keys=true,\n )\n review = case.get(\"humanreview\")\n if review:\n review_text = \"\\n\".join(\n [\n review.get(\"label\", \"\"),\n \" \".join(review.get(\"reasoncodes\", [])),\n review.get(\"rationale\", \"\"),\n *[\n annotation.get(\"rationale\", \"\")\n for annotation in review.get(\"criterionannotations\", [])\n ],\n ]\n )\n artifacts[(\"human_review\", review[\"annotationid\"])] = review_text\n return artifacts\n\n\ndef check_span(errors: list[str], prefix: str, span: dict[str, any], artifacts: dict[tuple[str, str], str]) -> none:\n key = (span[\"artifacttype\"], span[\"artifactid\"])\n if key not in artifacts:\n errors.append(f\"{prefix}: unknown artifact {key}\")\n return\n text = artifacts[key]\n quote = span[\"quotedtext\"]\n start = span.get(\"startchar\")\n end = span.get(\"endchar\")\n if start is not none and end is not none:\n if start > end or end > len(text):\n errors.append(f\"{prefix}: invalid character range {start}:{end} for length {len(text)}\")\n elif text[start:end] != quote:\n errors.append(f\"{prefix}: quotedtext does not match declared character range\")\n elif quote not in text:\n errors.append(f\"{prefix}: quotedtext not found in artifact {key}\")\n\n\ndef validate_referential_integrity() -> list[str]:\n errors: list[str] = []\n contract = load_json(examples / \"project-contract.example.json\")\n case = load_json(examples / \"case-input.example.json\")\n evaluation = load_json(examples / \"evaluation-output.example.json\")\n benchmark = load_json(examples / \"benchmark-item.example.json\")\n resolution_packet = load_json(examples / \"human-resolution-packet.example.json\")\n\n contract_hash = sha256_json(contract)\n case_hash = sha256_json(case) contract content and identity.\n instruction_ids = [doc[\"instructionid\"] for doc in contract[\"instructiondocuments\"]]\n criterion_ids = [criterion[\"criterionid\"] for criterion in contract[\"criteria\"]]\n evidence_class_ids = [item[\"classid\"] for item in contract[\"sourcepolicy\"][\"evidenceclasses\"]]\n rule_ids = [rule[\"ruleid\"] for rule in contract[\"decisionpolicy\"][\"rules\"]]\n\n for label, values in [\n (\"instruction ids\", instruction_ids),\n (\"criterion ids\", criterion_ids),\n (\"evidence class ids\", evidence_class_ids),\n (\"decision rule ids\", rule_ids),\n ]:\n dupes = duplicates(values)\n if dupes:\n errors.append(f\"duplicate {label}: {dupes}\")\n\n instruction_by_id = {doc[\"instructionid\"]: doc for doc in contract[\"instructiondocuments\"]}\n criterion_by_id = {criterion[\"criterionid\"]: criterion for criterion in contract[\"criteria\"]}\n rule_by_id = {rule[\"ruleid\"]: rule for rule in contract[\"decisionpolicy\"][\"rules\"]}\n declared_targets = set(contract[\"decisiontargets\"])\n\n for doc in contract[\"instructiondocuments\"]:\n check_hash(errors, f\"instruction {doc['instructionid']} hash\", doc[\"contenthash\"], sha256_text(doc[\"content\"]))\n\n for criterion in contract[\"criteria\"]:\n cid = criterion[\"criterionid\"]\n for ref in criterion[\"instructionrefs\"]:\n if ref[\"instructionid\"] not in instruction_by_id:\n errors.append(f\"{cid}: unknown instruction ref {ref['instructionid']}\")\n elif ref.get(\"quotedtext\") and ref[\"quotedtext\"] not in instruction_by_id[ref[\"instructionid\"]][\"content\"]:\n errors.append(f\"{cid}: quoted instruction text not found in {ref['instructionid']}\")\n for eid in criterion[\"evidencepolicy\"][\"allowedevidenceclassids\"]:\n if eid not in evidence_class_ids:\n errors.append(f\"{cid}: unknown evidence class {eid}\")\n for dep in criterion[\"dependencies\"][\"requiredbeforecriterionids\"] + criterion[\"dependencies\"][\"impliescriterionids\"]:\n if dep not in criterion_by_id:\n errors.append(f\"{cid}: unknown dependency {dep}\")\n for other in criterion[\"precedence\"][\"overridescriterionids\"] + criterion[\"precedence\"][\"overriddenbycriterionids\"]:\n if other not in criterion_by_id:\n errors.append(f\"{cid}: unknown precedence criterion {other}\")\n\n for rule in contract[\"decisionpolicy\"][\"rules\"]:\n if rule[\"target\"] not in declared_targets:\n errors.append(f\"rule {rule['ruleid']} targets undeclared target {rule['target']}\")\n for cid in rule[\"match\"].get(\"criterionids\", []):\n if cid != \"*\" and cid not in criterion_by_id:\n errors.append(f\"rule {rule['ruleid']} references unknown criterion {cid}\")\n\n # case hashes and references.\n if case[\"contractref\"][\"contractid\"] != contract[\"contractid\"]:\n errors.append(\"case contractid does not match contract\")\n if case[\"contractref\"][\"contractversion\"] != contract[\"contractversion\"]:\n errors.append(\"case contractversion does not match contract\")\n check_hash(errors, \"case contract hash\", case[\"contractref\"][\"contenthash\"], contract_hash)\n check_hash(errors, \"task content hash\", case[\"task\"][\"contenthash\"], sha256_text(case[\"task\"][\"prompt\"]))\n check_hash(errors, \"attempt content hash\", case[\"attempt\"][\"contenthash\"], sha256_text(case[\"attempt\"][\"content\"]))\n\n evidence_ids = [evidence[\"evidenceid\"] for evidence in case[\"evidencebundle\"]]\n if duplicates(evidence_ids):\n errors.append(f\"duplicate case evidence ids: {duplicates(evidence_ids)}\")\n for evidence in case[\"evidencebundle\"]:\n if evidence[\"evidenceclassid\"] not in evidence_class_ids:\n errors.append(f\"evidence {evidence['evidenceid']} uses unknown class {evidence['evidenceclassid']}\")\n check_hash(errors, f\"evidence {evidence['evidenceid']} hash\", evidence[\"contenthash\"], sha256_text(evidence[\"content\"]))\n for reference in case.get(\"referenceanswers\", []):\n check_hash(errors, f\"reference {reference['referenceid']} hash\", reference[\"contenthash\"], sha256_text(reference[\"content\"]))\n for annotation in case.get(\"humanreview\", {}).get(\"criterionannotations\", []):\n if annotation[\"criterionid\"] not in criterion_by_id:\n errors.append(f\"human review references unknown criterion {annotation['criterionid']}\")\n\n artifacts = build_artifact_text(case)\n for annotation_index, annotation in enumerate(case.get(\"humanreview\", {}).get(\"criterionannotations\", [])):\n for span_index, span in enumerate(annotation.get(\"citedspans\", [])):\n check_span(errors, f\"humanreview criterionannotations[{annotation_index}].citedspans[{span_index}]\", span, artifacts)\n\n # evaluation references and internal links.\n if evaluation[\"caseref\"][\"caseid\"] != case[\"caseid\"]:\n errors.append(\"evaluation caseid does not match case\")\n check_hash(errors, \"evaluation case hash\", evaluation[\"caseref\"][\"contenthash\"], case_hash)\n check_hash(errors, \"evaluation contract hash\", evaluation[\"contractref\"][\"contenthash\"], contract_hash)\n check_hash(errors, \"audit case hash\", evaluation[\"audit\"][\"caseinputhash\"], case_hash)\n check_hash(errors, \"audit contract hash\", evaluation[\"audit\"][\"contracthash\"], contract_hash)\n\n finding_ids = [finding[\"findingid\"] for finding in evaluation[\"findings\"]]\n claim_ids = [claim[\"claimid\"] for claim in evaluation[\"claims\"]]\n expected_ids = [element[\"elementid\"] for element in evaluation[\"expectedelements\"]]\n relationship_ids = [rel[\"relationshipid\"] for rel in evaluation[\"evidencerelationships\"]]\n run_ids = [run[\"modelrunid\"] for run in evaluation[\"runmanifest\"]]\n validator_run_ids = [run[\"validatorrunid\"] for run in evaluation[\"validatormanifest\"]]\n validator_ids = [run[\"validatorid\"] for run in evaluation[\"validatormanifest\"]]\n for label, values in [\n (\"finding ids\", finding_ids),\n (\"claim ids\", claim_ids),\n (\"expected-element ids\", expected_ids),\n (\"evidence-relationship ids\", relationship_ids),\n (\"model-run ids\", run_ids),\n (\"validator-run ids\", validator_run_ids),\n ]:\n dupes = duplicates(values)\n if dupes:\n errors.append(f\"duplicate evaluation {label}: {dupes}\")\n\n run_id_set = set(run_ids)\n validator_id_set = set(validator_ids)\n for run in evaluation[\"runmanifest\"]:\n for parent_id in run.get(\"parentrunids\", []):\n if parent_id not in run_id_set:\n errors.append(f\"model run {run['modelrunid']} references unknown parent run {parent_id}\")\n\n applicable_assessment_ids = {\n assessment[\"criterionid\"]\n for assessment in evaluation[\"criterionassessments\"]\n if assessment[\"applicability\"] == \"applies\"\n }\n for cid in applicable_assessment_ids:\n for validator_id in criterion_by_id[cid][\"evidencepolicy\"][\"deterministicvalidatorids\"]:\n if validator_id not in validator_id_set:\n errors.append(f\"applicable criterion {cid} requires missing validator run {validator_id}\")\n\n finding_by_id = {finding[\"findingid\"]: finding for finding in evaluation[\"findings\"]}\n claim_set = set(claim_ids)\n expected_set = set(expected_ids)\n evidence_set = set(evidence_ids)\n\n for index, finding in enumerate(evaluation[\"findings\"]):\n fid = finding[\"findingid\"]\n if finding.get(\"criterionid\") and finding[\"criterionid\"] not in criterion_by_id:\n errors.append(f\"finding {fid} references unknown criterion {finding['criterionid']}\")\n for claim_id in finding.get(\"claimids\", []):\n if claim_id not in claim_set:\n errors.append(f\"finding {fid} references unknown claim {claim_id}\")\n for element_id in finding.get(\"expectedelementids\", []):\n if element_id not in expected_set:\n errors.append(f\"finding {fid} references unknown expected element {element_id}\")\n for evidence_id in finding.get(\"evidenceids\", []):\n if evidence_id not in evidence_set:\n errors.append(f\"finding {fid} references unknown evidence {evidence_id}\")\n source = finding[\"source\"]\n if source.get(\"modelrunid\") and source[\"modelrunid\"] not in run_id_set:\n errors.append(f\"finding {fid} references unknown model run {source['modelrunid']}\")\n if source[\"kind\"] == \"deterministic_validator\" and source[\"componentid\"] not in validator_id_set:\n errors.append(f\"finding {fid} references unknown deterministic validator {source['componentid']}\")\n for span_index, span in enumerate(finding.get(\"targetspans\", [])):\n check_span(errors, f\"findings[{index}].targetspans[{span_index}]\", span, artifacts)\n\n for index, claim in enumerate(evaluation[\"claims\"]):\n for evidence_id in claim.get(\"citedevidenceids\", []):\n if evidence_id not in evidence_set:\n errors.append(f\"claim {claim['claimid']} cites unknown evidence {evidence_id}\")\n for span_index, span in enumerate(claim[\"attemptspans\"]):\n check_span(errors, f\"claims[{index}].attemptspans[{span_index}]\", span, artifacts)\n\n for index, element in enumerate(evaluation[\"expectedelements\"]):\n if element[\"criterionid\"] not in criterion_by_id:\n errors.append(f\"expected element {element['elementid']} references unknown criterion\")\n for span_index, span in enumerate(element.get(\"observedspans\", [])):\n check_span(errors, f\"expectedelements[{index}].observedspans[{span_index}]\", span, artifacts)\n\n for rel in evaluation[\"evidencerelationships\"]:\n if rel[\"evidenceid\"] not in evidence_set:\n errors.append(f\"relationship {rel['relationshipid']} references unknown evidence\")\n subject_id = rel[\"subjectid\"]\n if rel[\"subjecttype\"] == \"claim\" and subject_id not in claim_set:\n errors.append(f\"relationship {rel['relationshipid']} references unknown claim {subject_id}\")\n if rel[\"subjecttype\"] == \"expected_element\" and subject_id not in expected_set:\n errors.append(f\"relationship {rel['relationshipid']} references unknown expected element {subject_id}\")\n if rel[\"subjecttype\"] == \"finding\" and subject_id not in finding_by_id:\n errors.append(f\"relationship {rel['relationshipid']} references unknown finding {subject_id}\")\n for span_index, span in enumerate(rel.get(\"evidencespans\", [])):\n check_span(errors, f\"relationship {rel['relationshipid']} evidencespans[{span_index}]\", span, artifacts)\n\n assessment_ids = [assessment[\"criterionid\"] for assessment in evaluation[\"criterionassessments\"]]\n if duplicates(assessment_ids):\n errors.append(f\"duplicate criterion assessments: {duplicates(assessment_ids)}\")\n for assessment in evaluation[\"criterionassessments\"]:\n cid = assessment[\"criterionid\"]\n if cid not in criterion_by_id:\n errors.append(f\"assessment references unknown criterion {cid}\")\n continue\n all_finding_refs = (\n assessment[\"positivefindingids\"]\n + assessment[\"negativefindingids\"]\n + assessment[\"omissionfindingids\"]\n + assessment[\"counterevidencefindingids\"]\n + assessment[\"unresolvedfindingids\"]\n )\n for fid in all_finding_refs:\n if fid not in finding_by_id:\n errors.append(f\"assessment {cid} references unknown finding {fid}\")\n if assessment[\"verdict\"] == \"met\" and criterion_by_id[cid][\"proofpolicy\"][\"positiveproofrequired\"]:\n confirmed_positive = [\n fid for fid in assessment[\"positivefindingids\"]\n if fid in finding_by_id\n and finding_by_id[fid][\"findingtype\"] == \"positive\"\n and finding_by_id[fid][\"status\"] == \"confirmed\"\n ]\n if not confirmed_positive:\n errors.append(f\"assessment {cid} is met without confirmed positive proof\")\n if assessment[\"verdict\"] == \"not_met\":\n confirmed_failure = [\n fid for fid in assessment[\"negativefindingids\"] + assessment[\"omissionfindingids\"]\n if fid in finding_by_id and finding_by_id[fid][\"status\"] == \"confirmed\"\n ]\n if not confirmed_failure:\n errors.append(f\"assessment {cid} is not_met without confirmed negative or omission evidence\")\n if assessment[\"applicability\"] == \"does_not_apply\" and assessment[\"verdict\"] != \"not_applicable\":\n errors.append(f\"assessment {cid} does not apply but verdict is {assessment['verdict']}\")\n\n for decision_list_name in [\"provisionaldecisions\", \"finaldecisions\"]:\n decisions = evaluation[decision_list_name]\n targets = [decision[\"target\"] for decision in decisions]\n if duplicates(targets):\n errors.append(f\"duplicate {decision_list_name} targets: {duplicates(targets)}\")\n missing = declared_targets - set(targets)\n if missing:\n errors.append(f\"{decision_list_name} missing declared targets: {sorted(missing)}\")\n for decision in decisions:\n if decision[\"target\"] not in declared_targets:\n errors.append(f\"decision targets undeclared target {decision['target']}\")\n for rid in decision[\"triggeredruleids\"]:\n if rid not in rule_by_id:\n errors.append(f\"decision references unknown rule {rid}\")\n elif rule_by_id[rid][\"target\"] != decision[\"target\"]:\n errors.append(f\"decision target {decision['target']} uses rule {rid} for {rule_by_id[rid]['target']}\")\n for cid in decision[\"decisivecriterionids\"]:\n if cid not in criterion_by_id:\n errors.append(f\"decision references unknown decisive criterion {cid}\")\n for fid in decision[\"unresolvedfindingids\"]:\n if fid not in finding_by_id:\n errors.append(f\"decision references unknown unresolved finding {fid}\")\n\n for feedback in evaluation[\"feedback\"]:\n if feedback.get(\"criterionid\") and feedback[\"criterionid\"] not in criterion_by_id:\n errors.append(f\"feedback {feedback['feedbackid']} references unknown criterion\")\n for fid in feedback[\"findingids\"]:\n if fid not in finding_by_id:\n errors.append(f\"feedback {feedback['feedbackid']} references unknown finding {fid}\")\n\n # benchmark links.\n check_hash(errors, \"benchmark contract hash\", benchmark[\"contractref\"][\"contenthash\"], contract_hash)\n check_hash(errors, \"benchmark case hash\", benchmark[\"caseref\"][\"contenthash\"], case_hash)\n for criterion_gold in benchmark[\"goldstandard\"][\"criteriongold\"]:\n if criterion_gold[\"criterionid\"] not in criterion_by_id:\n errors.append(f\"benchmark references unknown criterion {criterion_gold['criterionid']}\")\n for field in [\"positiveevidencespans\", \"negativeevidencespans\"]:\n for index, span in enumerate(criterion_gold[field]):\n check_span(errors, f\"benchmark {criterion_gold['criterionid']} {field}[{index}]\", span, artifacts)\n for target_decision in benchmark[\"goldstandard\"][\"acceptabledecisions\"]:\n if target_decision[\"target\"] not in declared_targets:\n errors.append(f\"benchmark acceptable decision targets undeclared target {target_decision['target']}\")\n\n\n # standalone human-resolution packet integrity.\n packet_finding_ids = {finding[\"findingid\"] for finding in resolution_packet[\"contextfindings\"]}\n request_ids = set(resolution_packet[\"interaction\"][\"request\"][\"contextfindingids\"])\n if packet_finding_ids != request_ids:\n errors.append(\n \"human-resolution packet context findings do not exactly match request contextfindingids\"\n )\n if resolution_packet[\"interaction\"][\"decisiontarget\"] not in declared_targets:\n errors.append(\"human-resolution packet targets an undeclared decision target\")\n for cid in resolution_packet[\"interaction\"][\"request\"][\"triggercriterionids\"]:\n if cid not in criterion_by_id:\n errors.append(f\"human-resolution packet references unknown criterion {cid}\")\n\n return errors\n\n\ndef main() -> none:\n schema_failures = validate_examples()\n if schema_failures:\n raise systemexit(1)\n\n errors = validate_referential_integrity()\n if errors:\n print(\"\\nreferential integrity failures\")\n for error in errors:\n print(f\" - {error}\")\n raise systemexit(1)\n\n print(\"pass cross-document referential integrity\")\n print(\"pass canonical example hashes\")\n print(\"pass positive-proof and failure-proof invariants\")\n\n\nif __name__ == \"__main__\":\n main()\n"}