Evaluation Contract Specification
Normative semantics for project contracts, evidence, criteria, findings, uncertainty, feedback, security, and conformance.
A project-agnostic contract and benchmark package for evaluating human-produced AI-training data and the human annotations applied to it.
The package defines the stable domain boundary for an AutoQA system. It intentionally does not prescribe a model provider, orchestration framework, database, or Vercel implementation. Those components can change while the evaluation objects, proof obligations, decision policy, and benchmark remain versioned and auditable.
Core model
AutoQA evaluates two different objects:
- The attempt - whether the human-produced work satisfies the project's instructions.
- The annotation - whether the human reviewer's label, rationale, cited evidence, criterion attribution, and severity faithfully describe that attempt.
It does not reduce evaluation to a single score. The core unit is a criterion assertion:
Criterion C applies.
Attempt span A exhibits behavior B.
Evidence E supports, contradicts, or is insufficient for B.
B does or does not satisfy C.
The project decision policy assigns consequence S.
Non-negotiable invariants
A conforming implementation preserves these distinctions:
- truth versus support by the permitted evidence;
- source quality versus entailment;
- relevance versus correctness;
- factual accuracy versus inferential warrant;
- positive proof versus absence of a detected defect;
- attempt quality versus annotation fidelity;
- criterion verdict versus severity;
- severity versus operational action;
- raw model confidence versus empirically calibrated confidence;
- item validity versus marginal dataset value.
The engine may be project-agnostic. Its judgments may not be criterion-agnostic. Every production project must supply an approved, versioned evaluation contract.
Package contents
.
+-- README.md
+-- CONTRACT_SPEC.md
+-- BENCHMARK_PROTOCOL.md
+-- DECISION_POLICY.md
+-- CHANGELOG.md
+-- VERSION
+-- schemas/
| +-- common.schema.json
| +-- project-contract.schema.json
| +-- case-input.schema.json
| +-- evaluation-output.schema.json
| +-- benchmark-item.schema.json
| +-- human-resolution-packet.schema.json
+-- types/
| +-- autoqa.ts
| +-- tsconfig.json
+-- examples/
| +-- project-contract.example.json
| +-- case-input.example.json
| +-- evaluation-output.example.json
| +-- benchmark-item.example.json
| +-- human-resolution-packet.example.json
+-- templates/
| +-- criterion-authoring-template.csv
| +-- gold-set-template.csv
| +-- model-bakeoff-template.csv
+-- validation/
+-- hash_utils.py
+-- validate_examples.py
+-- validate_package.py
Primary documents
CONTRACT_SPEC.mddefines the normative semantics, lifecycle, invariants, and conformance requirements.BENCHMARK_PROTOCOL.mddefines how to build adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.DECISION_POLICY.mddefines the recommended rule engine, action semantics, human-resolution trigger, and policy trace.
Machine-readable layer
- Project contract: the approved interpretation of project instructions.
- Case input: the task, evidence, attempt, optional reference, and optional human review.
- Evaluation output: claims, expected elements, evidence relationships, findings, criterion assessments, annotation audit, decision trace, and feedback.
- Benchmark item: adjudicated criterion truth, target-specific acceptable actions, ambiguity status, challenge attributes, and leakage controls.
- Human-resolution packet: the one bounded, decision-changing question and its evidence context, with an optional response.
Runtime input and output validation should use the JSON Schemas. TypeScript interfaces are provided for application development but are not a runtime trust boundary.
Quick start
Validate the included examples:
python validation/validate_package.py
Type-check the TypeScript domain model:
cd types
tsc -p tsconfig.json
Recommended adoption sequence:
- Copy
examples/project-contract.example.jsonand replace the synthetic instructions and criteria. - Author each criterion in
templates/criterion-authoring-template.csvbefore converting it to JSON. - Run a contract review with the project owner, a domain expert, and a QA owner.
- Create representative
case_inputdocuments from historical production work. - Independently adjudicate a calibration and test set using
BENCHMARK_PROTOCOL.md. - Compare evaluator configurations by criterion family rather than choosing one universal judge.
- Calibrate confidence and set project-specific action gates.
- Shadow the system before allowing autonomous acceptance or rejection.
- Preserve every model, prompt, source, rubric, and policy version required for replay.
Minimal evaluator pipeline
1. Validate and freeze inputs
2. Compile/load the approved project contract
3. Run deterministic integrity and compliance checks
4. Extract claims and expected elements
5. Map claims and requirements to evidence
6. Generate positive, negative, omission, and warrant observations
7. Challenge material observations
8. Adjudicate each criterion
9. Evaluate global qualities
10. Audit the human annotation independently
11. Apply the versioned decision policy
12. Ask at most one decision-changing human question when necessary
13. Generate and validate audience-specific feedback
14. Persist the complete audit record
Conformance levels
Level A - structured evaluation
The implementation validates inputs and outputs, uses an approved project contract, produces criterion-level assessments, and records evidence-linked findings.
Level B - calibrated selective evaluation
Level A plus empirical confidence calibration, risk-coverage reporting, explicit abstention/escalation, and locked test suites.
Level C - governed production AutoQA
Level B plus independent annotation auditing, one-interaction human resolution, shadow deployment, drift monitoring, replay, appeals, and restricted use of reviewer analytics.
Autonomous acceptance or rejection should not be enabled below Level B.
Example scenario
The included example contains a high-quality attempt that correctly concludes that the evidence does not establish a primary causal relationship. A human reviewer fails it for not choosing a forced yes/no answer. The project contract explicitly permits an inconclusive conclusion. The example evaluation therefore:
- affirms the attempt's directness, grounding, warrant, and completeness;
- accepts the attempt;
- returns the annotation and composite case for rework;
- marks the reviewer annotation as unsupported;
- gives the attempter specific positive feedback;
- gives the reviewer a minimal corrective explanation;
- records a project-owner signal in case this misunderstanding recurs.
This demonstrates why attempt quality and annotation fidelity must be separate outputs.
Versioning rule
Three versions may change independently:
- Schema version - shape and semantics of machine-readable documents.
- Contract version - project instructions, criteria, anchors, or evidence policy.
- Decision-policy version - operational mapping from criterion state to action.
A case must reference an exact contract version and content hash. An evaluation must record the exact case, contract, prompt, source snapshots, model runs, calibration profile, and decision-policy version used.
What 0.1.0 deliberately leaves open
- model and provider selection;
- retrieval architecture;
- prompt templates;
- long-context chunking strategy;
- calibration algorithm;
- reviewer-competence model;
- dataset-level value model;
- UI design;
- persistence and job orchestration.
These should be decided empirically after the contract and benchmark are stable.
Download
| File | Format | Size | SHA-256 |
|---|---|---|---|
| autoqa-foundation-v0.1.0.zip | ZIP | 78.3 KB | e020f1e46f91 |
Foundation specifications
Decision Policy Reference
A target-specific rule engine for accept, rework, reject, and escalate decisions, including the bounded human interaction.
Benchmark & Calibration Protocol
How to construct adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.
Foundation schemas
JSON Schema - Benchmark Item
Runtime schema for adjudicated truth, acceptable actions, ambiguity, challenge attributes, and leakage controls.
JSON Schema - Case Input
Runtime schema for task, evidence bundle, attempt, human review, references, and provenance.
JSON Schema - Common Definitions
Shared runtime definitions used by the project, case, evaluation, benchmark, and resolution schemas.
JSON Schema - Evaluation Output
Runtime schema for claims, evidence relationships, findings, criterion assessments, decisions, annotation audit, and feedback.
JSON Schema - Human Resolution Packet
Runtime schema for one bounded, decision-changing human question and its context.
JSON Schema - Project Contract
Runtime schema for approved project instructions, criteria, evidence policy, decision policy, and calibration requirements.
Foundation examples
Worked Example - Benchmark Item
Adjudicated benchmark record for the synthetic case, including acceptable outcomes and ambiguity metadata.
Worked Example - Case Input
A strong attempt incorrectly failed by a human reviewer for not forcing an unsupported binary conclusion.
Worked Example - Evaluation Output
Complete AutoQA result accepting the attempt, returning the annotation for rework, and generating audience-specific feedback.
Worked Example - Human Resolution
A bounded precedence question that hides the provisional decision and asks only what can change the outcome.
Worked Example - Project Contract
Synthetic closed-world evidence project with positive-proof requirements, anchors, severity, and explicit decision rules.
Foundation templates
Template - Criterion Authoring
CSV template for drafting criterion intent, proof standards, failure anchors, evidence requirements, severity, and governance.
Template - Gold Set
CSV template for building an adjudicated calibration and test set.
Template - Model Bakeoff
CSV template for comparing evaluator configurations by criterion family.
Foundation developer files
TypeScript Domain Model
Strict TypeScript types for project contracts, cases, evaluations, benchmarks, findings, decisions, feedback, and human resolution.
TypeScript Configuration
Compiler configuration for the package's type definitions.
Foundation Package - Version
Current package version identifier.
Foundation Package - Manifest
Machine-readable inventory and hashes for the complete package.
Validation - Hash Utilities
Canonical content hashing utilities used by the validation suite.
Validation - Example Schema Checks
Python validator for the package's example documents.
Validation - Package Integrity
Cross-document, schema, hash, proof-invariant, policy, and package integrity checks.
On this page
- AutoQA Foundations - Philosophy & Structure (Draft)
- 0. The problem, restated precisely
- 1. The construct: what a verdict means
- 2. The eleven principles
- 3. The architecture
- 4. The ship gate (per project config, before go-live)
- 5. The legal floor (binding if the workforce touches the EU)
- 6. What this system deliberately does not do
- 7. First experiments (ordered; each kills or funds a component)
- 8. Decisions a team adopting this would need to make
Date: 2026-07-14 Status: Draft v1, synthesized from Question Inventory and Research Atlas. Every principle carries its evidence anchor; positions the evidence does not settle are marked [default] (my recommended posture, reversible) or [open decision] (needs your call). Scope: Project-agnostic foundation for QA of human-annotated AI training data.
0. The problem, restated precisely
Human attempters produce evaluative work products (annotations, ratings, critiques with rationales) under project instructions. Human reviewers who QA them exhibit unexplained variance in pass/fail rates and reasons. Root causes, ranked: (1) subjective interpretation of evaluation axes, (2) expertise gaps, (3) attention to detail. Attempters are mandatory; the QA layer is what we augment.
The psychometric literature sharpens this: after training, raters' global severity differences persist but are a minor variance component - the dominant variance is rater x item x criterion interaction: which reviewer, on which submission, on which criterion, this time. That variance cannot be fixed by better rubrics or calibration meetings alone. It can only be attacked by verifying each specific claim on each specific item against evidence - which is exactly what an LLM layer can do at scale and humans at scale cannot. That is the reason this system should exist, stated in the field's own terms. (Atlas Section 26)
1. The construct: what a verdict means
Quality = satisfaction of the written project instructions + grounding in the defined evidence record. Not "would the median reviewer pass this." Human-consensus prediction caps the system's validity at the reviewers' own (unmeasured, evidently low) reliability ceiling; instruction-satisfaction lets the AutoQA be more valid than the layer it augments, and makes every verdict traceable to a clause and a span. (Atlas Section 16-18; peer-reviewed rating-indeterminacy work (NeurIPS 2025) shows forced-choice consensus gold labels can heavily bias judge selection - empirically up to 31% worse [AJ-02] - when criteria admit multiple valid readings.)
Consequences of this choice:
- Gold labels come from adjudicated expert panels judging against the instructions, not from single-reviewer or majority crowd labels.
- When the AutoQA and a human reviewer disagree, neither is presumptively right; the arbiter is the instruction text plus the evidence record, adjudicated.
- Where the instructions genuinely underdetermine an item, the correct output is not a verdict but an instruction-gap report. Reviewer disagreement on such items is signal about the instructions, not error in the people. (Diverging Preferences: most annotator disagreement traces to task underspecification and style, not noise or incompetence.)
- The ground-truth hierarchy is explicit: seeded known-verdict items > adjudicated expert panel > criterion author's written amendment > individual human reviewer > AutoQA. Only the first three may feed judge calibration. Overrides by individual humans are logged as claims, not truth.
2. The eleven principles
P1 - Claims, not vibes. The unit of analysis is the typed claim, not the item. Attempter writeups are decomposed into claims of distinct types, each verified in the lane built for it, each verdict carrying quoted evidence. Item verdicts are transparent, auditable roll-ups. (Holistic judging of subjective quality measurably fails: near-orthogonal judge/human axes; ~55% on hard per-criterion checks [AJ-04]; long-form instability. Decomposed per-criterion reasoning buys +7-12pp [AJ-04] and lower cross-judge variance.)
P2 - The claim ontology is the type system. Every extracted statement is triaged before any verification: (a) grounding claims (cited evidence exists and says this), (b) entailment claims (evidence supports the conclusion drawn), (c) calibration claims (stated severity/confidence matches evidence strength), (d) completeness/absence claims ("no errors," "fully addresses"), (e) opinions - never truth-scored, but checked with real machinery via two reusable passes: stance-contradiction (the evidence the attempter cites must not contradict the stated stance) and stance-calibration (stance strength must not exceed evidence strength - type-(c) machinery). Opinion verdicts are consistent / inconsistent-with-cited-evidence (a fail-able finding: the opinion's warrant fails grounding, not the opinion itself) / uncheckable (never penalized) - (f) undecidable-given-evidence. Each type has different machinery and a different reliability ceiling, reported per type.
Entailment standard [default]: reasonable-expert support, not strict logical entailment - three-way output per claim (supported / contradicted / unsupported), where contradicted is the high-confidence fail trigger and unsupported alone triggers an evidence request or escalation rather than an automatic fail. Strict entailment fails nearly all competent human prose; loose support reintroduces the variance we're removing - so the false-flag rate under this standard is a measured ship-gate output (Section 4.3), not an assumption. (VeriScore/FactBench triage is field-standard; FACTS shows the positive-class asymmetry; AbsenceBench caps type (d).)
P3 - Evidence closure is explicit and judge priors are not evidence. Every project config defines the evidence record: what the attempter saw, what they cited, what's in scope. Verdicts must quote spans from that record to be admissible; "true in the world but unsupported in the record" is unsupported, not pass. Checks are routed by whether they are read-checkable or recompute-checkable - judge confidence is untrustworthy on checks that exceed its own solving ability, where it silently re-solves and trusts itself over the record. [default - provisional: the re-solving evidence is a single under-review 2026 study; experiment 3 replicates it internally before this hard-gates routing.] Cherry-picking is caught by running both span-closed and context-closed citation checks. The judge gets the same evidence surface the attempter had - architecture beats model. (Auditing-by-Re-Solving; CiteEval; agent-judge/the banking meta-eval.)
P4 - Positive claims are verified by enumerated class, with honest labels. "Warranted praise" verification is restricted to checkable praise types compiled per project (accuracy-praise -> grounding check; completeness-praise -> enumerable checklist; clarity-praise -> opinion class). Absence claims get a seeded-defect protocol: unless the judge's planted-flaw recall on this project materially exceeds the attempter population's, its concurrence is labeled not-contradicted, never verified. A vacuousness/eligibility gate precedes grounding checks so hedged empty writeups can't pass by asserting nothing. (FACTS eligibility gate; AbsenceBench; zero published praise-verification numbers - this is the first in-house measurement.)
P5 - Typed verdicts; abstention and contestation are first-class. The verdict set is pass / fail / unable-to-verify / instructions-underdetermine. High vote-entropy across k samples, cross-family judge disagreement, and clustered reviewer disagreement route items to the contested class - which feeds the instruction-gap channel and never penalizes the attempter. This is how the system enforces consistency where consistency is real, without laundering one arbitrary reading of a contested criterion into false objectivity. (Toloka's deployed Pass/Fail/Unable-to-verify; CoVal; NUTMEG; rating indeterminacy.)
P6 - The human touch is spent where it compounds, and it is blind. Routing is confidence-calibrated three-lane (auto-verdict / stronger judge / human) with the human budget derived from a target residual error rate, not a fixed per-item quota. The escalated human sees the claim, the criterion clause, and the judge's collected evidence spans - not its verdict or rationale (verdict-visible review produces measured over-reliance; evidence-only is the sole format that helps when the AI is right without hurting when it's wrong). Escalation routes the contested claim, not the whole item. Every human adjudication writes back - in two tiers that keep the ground-truth hierarchy honest: a single-touch adjudication produces candidate artifacts (logged claims, provisional exemplars), and only panel-adjudicated or owner-signed artifacts enter judge calibration and compiled config; the standing audit channel batch-promotes candidates. In enforcement mode the human budget is max(target-residual-error escalations, consequential-fail volume - Section 5); under a pace-bound profile (P11) capacity binds instead, and the residual error rate becomes the reported output rather than the target - never pretend to optimize both. The evidence-only-for-adjudicators result is measured; combining it with full critiques to attempters (P9) is an inferred, untested split [default], A/B'd in experiment 6. Overturn-rate monitoring proves the human lane adds signal; an overturn rate at the judge's error floor means rubber-stamping, and the naive human-verifies-AI design is the modal failure in the literature. (Trust-or-Escalate/HyPAC; DeepMind complementarity; Vaccaro meta-analysis; Braintrust/LangChain/micro1 write-back practice.)
P7 - The QA layer is measured exactly like what it measures. The judge is enrolled as one more rater facet in the same measurement model as the humans (MFRM-style severity/centrality/fit per rater, human or machine, per project). Scope honestly: MFRM needs 3-5 ratings/output of rater-item linkage, so it runs fully on the dual-review pilot and continuously on the judge facet (which rates everything); production human-reviewer severity is estimable only where deliberate overlap is injected - a low-rate stream of double-reviewed seeded items, a budgeted cost, not a free byproduct. All reporting uses chance-corrected statistics, per-class recall at fixed prevalence, and PPI-corrected population estimates with dual-uncertainty CIs - raw percent-agreement and single-number "accuracy vs humans" are banned artifacts. Feedback passes the same grounding gate it enforces before delivery (fail-closed). Validation stratifies by measured human-agreement level; verdict-agreement alone never certifies a judge, because right-verdict-wrong-rationale poisons feedback and trust (in one benchmark, 24.8% of items got the right verdict with a low-quality critique - RealCritic [CM-08]). Staging [default]: v1 ships chance-corrected agreement, PPI-corrected point estimates with CIs, and k-sample entropy routing; MFRM production enrollment, anytime-valid e-processes, and conformal routing are phase-2 refinements - the full stack is the target, not the launch gate. (MFRM-for-LLM-judges; kappa-deflation cohort; BeyondCorrelation; RealCritic; MetaCritique.)
P8 - Adversarial by construction, from day one. Attempters are paid optimizers; some are agents. Defenses are structural, not detective: stakes-sterile judge prompts (consequence framing measurably softens verdicts with zero CoT trace); comparative/anchored scoring over absolute scales (transfer attacks inflate absolute scores blind); evidence-perturbation probes as a standing citation-theater detector (swap the cited span -> verdict must flip); blind seeded gold injected into live queues (indistinguishability is the binding constraint); the divergence sensor - attempter scores rising on the AutoQA while flat on held-out human-graded goldens - as the canonical Goodhart alarm; spam defense that is economics/provenance-based (throttling, work-history consistency, worker-level peer-prediction scoring, randomized deep audits), with content detection demoted to a weak auxiliary signal. Never score anyone on agreement-with-final-verdict - consensus-rewarded auditing trains your best dissenters to conform exactly where independent judgment matters most. (Evaluation Faking; universal adversarial phrases; TSPA; Westwood PNAS; Community Notes conformity study.)
P9 - Feedback is an instrumented intervention, not decoration and not free. The feedback unit is the 4-tuple: quoted claim, evidence span, criterion clause, concrete fix - task-referenced always, person- or rank-referenced never (a third of feedback interventions reduce performance [CG-19]; high-information task feedback roughly doubles the effect of bare verdicts [CG-22]). Benchmark against exemplar work, never ranked peers. No stylistic guidance by default - feedback is a homogenization pump aimed at the diversity the training data exists to capture, so a population-level output-diversity drift metric ships with the channel. Efficacy is measured per-attempter (repeat-error-rate per criterion, pre/post), expecting modest median effects concentrated in low performers, and separating within-attempter learning from attrition. If high-information feedback doesn't move repeat-error rates in a defined window, conclude decoration and reallocate. (Kluger & DeNisi; Wisniewski; Cochrane 2025; Dow CSCW 2012; monoculture evidence.)
P10 - Everything drifts; every moving part gets its own sensor. Frozen human-labeled anchor sets interleaved into the live stream attribute drift (judge moved vs population moved) with anytime-valid statistics. Judge versions are pinned per project with golden-set regression gates on upgrade; rubric revision is a versioned loop with explicit retroactive re-scoring policy (adaptation lives in the compilation layer, pinning at the judge layer - that's the resolution of the static-vs-co-evolving fork). Reviewer calibration decays within a session, so reviewer-effect estimates run in rolling windows with hard-item interleaving and order randomization; graduated projects carry randomized, unpredictable audit rates with automatic de-graduation triggers. Queue composition is managed (seeded known-verdict items, easy-case dilution) because prefiltered queues measurably shift reviewer strictness. (Anchor-set e-process work; DRIFT/OSCE; Pinterest dual monitors; content-moderation practice.)
P11 - Quality is a priced dial on disposition, never on perception. Contractual pace sometimes makes the standard quality threshold unaffordable; the system supports that honestly instead of forcing the loosening to happen off the books. The judge always measures the same way - dial-sterile prompts, same claim verification, same evidence dossier (consequence framing measurably corrupts verdicts with zero CoT trace, and a "be lenient" judge destroys score comparability, calibration, and the divergence sensor simultaneously). Leniency is applied downstream, as thresholds over the continuous scores the pipeline already emits. A project's active operating profile (e.g., strict / standard / expedited) moves exactly four things: aggregation cutoffs (how many/how severe findings fail an item), routing thresholds (escalation rate, human budget), verification depth (k-samples, deep-verify sampling fraction, expensive lanes like context-closed citation checks), and enforcement disposition (findings -> fail vs findings -> coaching-only). Two constraints can bind: in error-target mode the residual error rate is the target and human budget is the output; in pace-bound mode (contractual throughput) capacity binds and the residual error rate becomes the reported output - either way the number is computed and stamped, never hidden, with CIs that honestly widen when verification depth is thinned. Because measurement persists under every profile, the dial has two independent consumers: the attempter-disposition dial (prospective only, logged, announced - the bar people are judged against never moves silently or retroactively) and the dataset-admission dial (re-tunable at any time - an expedited batch can be re-filtered to any strictness later from the recorded scores, without re-running QA; the data buyer's strictness is not coupled to the deadline that produced the batch). Floors the dial can never touch: integrity-class verdicts (fabrication, spam, falsehood), seeded-gold injection and audit sampling, the divergence sensor (which conditions on the active profile and reads seeded/audit outcomes, not pass rates), and the Section 5 legal floor. Every delivered batch carries its profile and its corrected residual-error estimate - leniency becomes a contractual artifact instead of quiet degradation. (Evaluation Faking -> dial-sterile prompts; CriticGPT's FSBS precision/comprehensiveness dial as the precedent for inference-time operating points; Trust-or-Escalate's coverage dial; HyPAC error budgets; corrected-estimator reporting.)
3. The architecture
This section is published once, on System.
4. The ship gate (per project config, before go-live)
A project's judge config ships only after:
- Variance-attribution pilot: dual-review a common item set; measure the human floor (chance-corrected, per criterion - per-criterion kappa localizes ambiguous criteria); code disagreements against an underspecification-vs-expertise-vs-attention taxonomy. This decides the build emphasis per project and provides the calibration set.
- MVVP-style judge validation: chance-corrected agreement vs adjudicated labels, >=3 replicates, AB/BA position swaps, >=2 gold-set designs, bias audit separate from consistency (a judge can be perfectly repeatable and severely biased).
- Perturbation harness: meaning-preserving paraphrase/reorder/noise on claims AND evidence; evidence-swap probes; verdicts that flip get routed, not shipped.
- Seeded-flaw recall: planted defects (FindTheFlaws-style) measuring fail-class recall at realistic prevalence - the number that actually matters under 90%+ pass rates [AQ-06], where kappa itself misleads (Feinstein-Cicchetti).
- Schematic adherence: do per-criterion scores vary independently and does the rationale's cited schema explain the verdict (halo/factor-collapse check) before per-criterion output is exposed to anyone.
- Gold-set arithmetic: size from the PPI closed form off pilot judge-human correlation per axis (conservative estimates; ~50+ labels per stratum floor; oversample the fail class); low-correlation axes carry human-only until correlation improves.
- Beat-the-humans gate: per axis, judge-vs-adjudicated reliability must exceed the pilot's measured human-reviewer-vs-adjudicated reliability. Axes where the judge doesn't beat the humans it augments are human-only lanes - automating them adds cost and subtracts validity. This is the go/no-go the whole premise rests on, and it is comparative, not absolute.
Beyond go/no-go, the ship gate's data product is the operating-point curve - thresholds -> fail-class recall and false-pass rate at pilot prevalence. This is what calibrates P11's dial profiles, so even the expedited setting ships with a priced, not guessed, error rate.
5. The legal floor (binding if the workforce touches the EU)
These are design notes on how the cited law could constrain a QA system, not legal advice.
Gate first [open decision]: determine whether the attempter population includes EU-resident workers. If yes, this section is a launch constraint; if the pilot is US-only, treat it as forward-looking scope and revisit before any EU expansion. From 2026-12-02 (Platform Work Directive transposition), for attempters in the EU - employee or self-employed, platform established anywhere:
- Decisions of significant detriment (account/pay restriction, termination, or equivalent) must be taken by a human; no consent exception. Auto-pass is fine; consequential auto-fail is not. Whether a single-item pay denial qualifies is legally untested - [default] treat consequential item fails as requiring the human lane until counsel says otherwise.
- Plain-language explanation of any automated-supported decision, a competent human contact, human review with substantiated written reply within two weeks, rectification/compensation duties. The evidence dossier and 4-tuple feedback ARE this explanation - design them to that bar.
- Telemetry: allowlist only (task-session-bounded behavioral signals); no emotional-state inference, no private-communication monitoring, no off-task collection; DPIA required.
- SCHUFA doctrine: a score with a "determining role" is itself an automated decision, and rubber-stamp review counts as zero-touch - the human lane must have authority, evidence access, and calibrated competence. Enforcement is active.
- Map quality dimensions and verdict evidence onto ISO/IEC 5259 vocabulary now (cheap) rather than retrofit later (expensive).
Convenient truth: the legally mandated design (meaningful human on adverse decisions, explanations, appeals) is the same design the complementarity evidence recommends anyway.
6. What this system deliberately does not do
- No per-item LLM-text spam classification as a primary defense (beaten economically and technically).
- No holistic single-call item judging; no per-criterion output without schematic-adherence validation.
- No raw percent-agreement or single-number accuracy anywhere in reporting.
- No truth verdicts on opinion-class claims; no
verifiedlabel on absence claims without seeded-recall evidence. - No stylistic feedback; no visible attempter leaderboards/rank feedback.
- No consequence language in judge prompts; no trusting CoT rationales as bias evidence.
- No scoring of humans (attempters, reviewers, or auditors) on agreement-with-final-verdict.
- No judge calibration from unadjudicated override streams.
- No leniency via judge prompting - the dial moves thresholds, depth, and disposition, never the judge's perception (dial-sterile is the same discipline as stakes-sterile). No dial setting touches integrity floors, seeded gold, audit sampling, or the legal floor. No silent or retroactive changes to the bar attempters are judged against.
- No claim that the judge validates itself - the expert-audit channel never sunsets.
7. First experiments (ordered; each kills or funds a component)
- Variance-attribution pilot on one live project (dual review + disagreement coding). Decides whether rubric compilation or expertise routing is the bigger lever for the target population, and produces the first calibration set. Everything else calibrates against this.
- Positive-claim seeded set: build warranted/empty-praise pairs per praise type; measure judge precision/recall per type. Nothing published exists - this is both the riskiest premise and the differentiator. Include absence-claim recall (planted defects behind "no errors" writeups).
- Transfer check: take the compiled-config judge and measure per-criterion agreement against adjudicated labels on human evaluative writeups (not model outputs) - the number every published anchor is missing. Include a re-solving probe: does our judge's wrong-reference detection collapse on recompute-class checks (internal replication of the under-review 2026 result P3 provisionally leans on)?
- Confidence-proxy shootout: verbalized confidence vs k-sample entropy vs conformal width vs cross-family disagreement, scored on predicting human overturn - this picks the routing statistic. Include the uniform-sampling baseline.
- Divergence sensor dry run: stand up held-out human-graded goldens + AutoQA score tracking from the first live week, so Goodhart is measurable before it's a problem.
- Feedback tier A/B: verdict-only vs 4-tuple high-information feedback, outcome = per-criterion repeat-error rate, attrition separated. Priors: modest median, concentrated in low performers, backfire in a minority. Also A/B the assistance-format split (full critiques to attempters vs evidence-only) - the combination P6/P9 defaults to is untested in any single study.
- Agnosticism falsification: onboard a second project through config alone. Any edit to the core judge prompts falsifies the project-agnostic claim - the fix is moving that surface into the config schema, not patching the core.
8. Decisions a team adopting this would need to make
- Enforcement weight at launch: advisory/coaching-only vs consequential verdicts. Changes the precision floor, the legal posture (Section 5), and attempter adversarial pressure. My read: launch advisory, attach consequences only per-lane after seeded-recall numbers exist.
- Construct authority when project owner intuition contradicts their own compiled rubric: rubric-as-law with forced amendment [default], or standing owner overrides (which quietly re-import unexplainable variance).
- LLM-assistance provenance policy default: content-validity-only vs provenance-gated (per-project override either way). Interacts with telemetry legality and with the reality that ~1/3 of text workers already use LLMs.
- Judge sourcing: frontier API judges per project (cross-family routing) only, or also an owned fine-tuned screening tier. The evidence supports two-tier; the question is whether to own the bottom tier.
- Where the first pilot runs - which project's data, whether its owner will fund the adjudicated gold set (the one cost that cannot be automated away), and whether that attempter population includes EU-resident workers (switches Section 5 from forward-looking to binding).
- Dial governance (P11): who may change a project's active operating profile (project owner alone, or with client sign-off), and whether expedited batches are disclosed to the data buyer with their stamped residual-error estimate. My read: disclose - a priced error rate converts deadline pressure from hidden liability into a contractual artifact, and it is the posture consistent with the ISO 5259 mapping in Section 5.
Lineage: brainstorm (92 questions, 7 lenses) -> taxonomy (9 themes, 8 tensions) -> 9-domain recency-biased sweep + 27 adversarial verifications + 4 gap-fill literatures -> this synthesis. Full provenance in domain reports.
Page record
Source: documents/view/foundation-readme-md.html