System | AutoQA | Learning Voyage

A versioned contract defines the system, and the models inside it can be swapped. Every decision has to be reconstructible from one auditable unit. Nothing in the pipeline recovers a truth the record doesn't hold.

The auditable unit

Criterion C applies.
Attempt span A exhibits behavior B.
Evidence E supports, contradicts, or is insufficient for B.
B does or does not satisfy C.
Project decision policy assigns consequence S - only if authorized for that lane.

Pipeline, in order

Intake and integrity. Provenance, completeness, eligibility. A gap in the record routes to record repair - never held against the worker.
Compile the contract. Instructions become versioned criteria, proof standards, examples, evidence rules - with owner sign-off on graded cases.
Decompose and type. Writeup -> claims typed as grounding, warrant, calibration, completeness, or opinion. Type picks machinery. Uncertifiable checks are marked, not guessed.
Verify in lanes. Deterministic checks always; cheap screeners triage; models judge only with the evidence the worker had; contested items surface instead of being forced.
Adjudicate criteria. Positive evidence, defects, omissions, and open questions recorded separately; rolled up by the project's versioned formula.
Route. Clear cases decide within authority map; borderline get deeper verification; only decision-changing ambiguity reaches a human as one bounded question.
Feedback. Task-referenced only - claim, evidence, clause, fix - and it must pass the same grounding gate it enforces.
Monitor. Blind gold (including auto-accepts and agreements), drift sensors, challenge sets, appeals that write back into the contract. Throughput KPIs must not suppress abstention rates.

Different claims need different proof

Claim type	Question	Treatment
Grounding	Does the cited evidence say this?	Span-level entailment. Fail class measured directly - never assumed.
Warrant	Does the evidence justify this strength of conclusion?	Reasoning confined to the record; discovery separate from judgment.
Completeness / absence	Is anything required missing?	Explicit checklists; "no errors" is not-contradicted, never verified, until planted-defect recall is measured.
Opinion	Is the stance consistent with its cited support?	Never truth-scored. Check contradiction and proportion only.
Undecidable	Do the inputs decide this at all?	Route to owner as instructions problem - no worker penalty. Forced choice on ambiguous items corrupts evaluation.

Disagreement is a first-class output

When two reviewers (or reviewer and AutoQA) split, the system does not average them into a single fake truth. It tags the best-supported cause class:

Instruction -> contract queue
Error -> evidence packet; human confirm before adverse action
Expertise -> specialist queue
Legitimate split -> record; optional rubric change; not a forced winner

Using AutoQA solely as "the second reviewer who breaks ties" is an anti-pattern under volume pressure.

Human routing

See guide: human loop. Packet design: criterion, spans, one question, finite options, hide provisional decision for gold and consequence. Escalation rate and escalation-change rate are operating metrics - if humans never change outcomes, the loop is decoration.

Human routing (one screen)

Escalate only when the answer changes the outcome. Packet: criterion + spans + one question. Hide model Pass/Fail for gold and consequence. Queues: contract owner · expert · consequence owner · blind audit (includes agreements).

Worked case, briefly

Task: does the evidence establish Initiative X as the primary cause of an improvement? Worker correctly says the studies do not establish primacy. Reviewer fails the item: "did not choose yes or no." Project instructions permit inconclusive answers.

System output: work claim supported; review claim contradicted by the instruction clause; no worker fault for following the contract; optional instruction-clarity note if the task note conflicts with the project rule - route to owner, not automatic reject of either party without policy.

Claim lane register

lane_id,claim_text,unit_of_analysis,admitted_record,permitted_external_sources,outcome_contract_version,authority,false_positive_consequence,false_negative_consequence,ambiguity_owner,accountable_owner,status
L-001,,,,,,shadow,,,,,draft

Raw file

Contract and decision policy

Evaluation Contract Specification

Normative semantics for project contracts, evidence, criteria, findings, uncertainty, feedback, security, and conformance.

Foundation specificationsMD1272 lines40.2 KBfoundationspecification

Read Raw

Decision Policy Reference

A target-specific rule engine for accept, rework, reject, and escalate decisions, including the bounded human interaction.

Foundation specificationsMD499 lines13.7 KBfoundationspecification

Read Raw

3. The architecture

Eight stages. Stages 0-2 are deterministic-ish and cheap; 3 is where models spend; 4-7 close the loops.

project instructions --> [1] COMPILE (workshop loop, versioned config)
 |
attempter submission --> [0] INTAKE GATE --> [2] DECOMPOSE & TYPE --> [3] VERIFY (lanes)
 |
 +-------------------------------------------+
 v
 [4] AGGREGATE (typed verdict + evidence dossier)
 |
 [5] ROUTE (auto / stronger judge / human, calibrated)
 |
 [6] FEEDBACK (4-tuple, self-verified, fail-closed)
 |
 [7] META-EVAL & MONITOR (gold, drift, divergence, dashboards)

Stage 0 - Intake & provenance gate. Worker-level, not item-level: rate/velocity limits, work-history consistency, peer-prediction information scores, session-bounded telemetry (allowlist schema - on-task signals only; see Section 5). Item-level: eligibility/vacuousness check (is this a responsive, contentful submission at all). No LLM-text detection verdicts.

Stage 1 - Compilation layer (the per-project customization contract). Project instructions compile into a closed config schema:

Axis triage: every evaluation axis classified into deterministic (string/regex/programmatic checks), judge-groundable (evidence-anchored verdicts), judge-assisted (human decides with judge-collected evidence), human-only, or AutoQA-ineligible - using per-axis conformal set width and pilot judge-vs-adjudicated agreement as the sorting evidence.
Rubric criteria: <= ~13 active criteria per judge call, each atomic, RIFT-linted; pitfall/negative criteria human-authored (the one artifact synthesis reliably can't produce); hard constraints get gating semantics, not soft-criterion listing.
Contrastive exemplars: hard near-miss pass/fail pairs per criterion - discriminative exemplars, not descriptive "what good looks like" prose. These anchor the judge few-shot (the measured-best grounding architecture) and double as reviewer training material.
Evidence-closure rules: what the judge may read, per claim type; citation requirements on load-bearing attempter claims.
Severity/weight taxonomy + aggregation formula (explicit, auditable - accepting the known performance tax vs implicit aggregation).
Routing costs and thresholds: the asymmetric error prices (false-fail vs false-pass), the target residual error rate, calibration set.
Operating profiles (the quality/leniency dial, P11): named threshold sets (strict / standard / expedited) taken from the ship gate's measured operating-point curve, each stamped with its expected fail-class recall and false-pass rate. Profile changes are owner-authorized, logged, prospective-only, and announced to attempters whenever the enforcement bar moves. Compilation is a loop, not a compile step: owner grades real items, criteria drift is expected, every version is kept, and sign-off happens on graded items, not abstract criteria. Compilation fidelity is the binding per-project cost - budget accordingly.

Stage 2 - Decompose & type. Claim extraction at adaptive granularity (claim + explicit context annotations, verifier told what's under test; never free-floating atomic rewrites); ontology typing per P2; canonicalization/style-stripping so verbosity and rubric-vocabulary mimicry don't leak into judgment.

Stage 3 - Verification lanes.

Deterministic lane: compiled checks run on everything, always.
Screening tier: cheap specialized checkers (MiniCheck-class entailment, per-metric small models) score every claim at ~1/100 frontier cost - triage, never verdicts.
Verdict tier: frontier reasoning judge, per-criterion calls, exemplar-anchored, cross-family from the models attempters plausibly used, stakes-sterile prompts, k-sampled with continuous scores (logit-expectation rather than sampled tokens where available), position/order randomized.
Opinion-consistency lane: stance-contradiction + stance-calibration checks against the attempter's cited evidence, with the P2(e) verdict semantics - runs in the same screening/verdict tiers as other claim types.
Contested detector: vote entropy, paraphrase flips, cross-family disagreement -> instructions-underdetermine candidates.

Stage 4 - Aggregation. Claim verdicts -> item verdict via the compiled formula; error-tolerant (a ~20% claim-level error rate means naive AND-aggregation is wrong by construction); output is the verdict + evidence dossier (every finding with quoted spans and criterion clauses) - the dossier is simultaneously the audit trail, the appeal record, and the feedback source.

Stage 5 - Routing & the human lane. Calibrated dual thresholds (HyPAC-style) with the human queue = the abstention set; auto-FAIL held to a stricter bar than auto-PASS (false fails burn trust and pay; consequential fails have a legal floor - Section 5); blind evidence-only adjudication UI; claim-level micro-adjudication; adjudications write back per P6's two-tier rule; overturn telemetry closed-loop. Queue organization (specialist-per-criterion vs generalist-per-item) is a compilation-layer decision informed by the pilot's expertise-share finding - if expertise gaps dominate a project's variance, specialist claim-level routing is the mechanism that closes root cause #2 on the human side. Cold start for a zero-label project: the judge runs shadow-mode under full human review until the calibration set reaches the PPI-arithmetic size and thresholds are fitted; graduation to confidence-routing happens when calibrated thresholds hit the target residual error on a rolling window - mirrored by P10's de-graduation triggers.

Stage 6 - Feedback channel. Per P9. Generated from the dossier (verdict machinery and feedback share one artifact), passed through its own grounding gate, delivered only if it survives.

Stage 7 - Meta-evaluation & monitoring. Per-project ship gate before go-live (Section 4); continuous: blind gold injection, frozen anchor set, divergence sensor, per-criterion agreement trends, appeal-overturn bands, population diversity metric, reviewer rolling-window effects, corrected-estimator dashboards. The standing expert-audit channel samples confident passes - critic-assisted (a critic-assisted second look finds 4x what an unassisted one does on the "flawless" slice [CM-02]) - and is permanent, because when judge accuracy <= attempter accuracy no statistical cleverness can substitute for gold (the 2x theorem) [CT-03], and false agreement is predicted to rise as model errors correlate.