AutoQA: when automated review can and can't support a QA decision

A case that shows the problem

A contributor is asked whether Initiative X caused an improvement. The evidence is one observational study and one inconclusive trial, so they answer that primacy isn't established. A reviewer fails them for "not choosing yes or no." The project instructions explicitly allow an inconclusive answer.

An agreement score sees one disagreement. What actually happened is three different things: the work was right, the review was wrong, and a task note may conflict with the project rule. Each needs a different fix, and none of them is "make the reviewers agree."

The approach

Reviewers disagree for four reasons: unclear instructions, reviewer error, missing expertise, and genuine judgment calls. A single agreement score blends them together, so pushing it up confuses error with ambiguity.

AutoQA checks each claim in a review against the written instructions and the evidence the reviewer had, and returns one of four results:

Supported: the evidence backs the claim.
Contradicted: the evidence shows the claim is wrong.
Insufficient evidence: the instructions are clear, but the record isn't enough. That isn't proof of error.
Instructions don't decide: the written rules don't pick an answer, so it goes to the person who owns them. It doesn't count against the contributor.

The work and the review are judged separately, because correct work can still receive a wrong review.

Where it still needs a person

Model judges are useful for checking claims against evidence at scale. They are unreliable in a few predictable places, and each one has a human job attached:

Re-solving instead of reading. On hard checks, judges tend to solve the problem themselves and trust their own answer over the record. Human role: route recompute-heavy checks to qualified reviewers. Evidence: F26-01 is provisional. It rests on one study whose source couldn't be retrieved at re-check. See also principle P3.
"No errors" claims. Models are weak at confirming that something is absent. Human role: seed known defects and measure recall before trusting a pass. Limit: L4.
Right verdict, wrong reason. A judge can reach the correct label with a poor rationale, which corrupts the feedback contributors receive. Human role: audit rationales, not just verdicts. Evidence: CM-08.
Automation bias. People defer to a model's verdict once they see it. Human role: adjudicate from the evidence with the model's answer hidden. Evidence: AQ-11.
Questions the record can't answer. Intent, "what a competent expert would have done," and unclear instructions can't be verified from the record at all. Human role: own those calls and fix the instructions. Limits: L2, L3.

Status

A system design, a synthetic demonstration, and a research base of 132 sourced claims. Each claim was graded for how much decision weight it can bear. A claim whose source couldn't be retrieved stays provisional. Nothing here estimates accuracy on real work or authorizes automated enforcement.

The rest of the record

Limits: what no model generation can settle, and an eight-item checklist for spotting a deployment that has gone past them.
System: how a claim moves through the pipeline.
Pilot: how to measure before trusting it.
Research and Evidence: the literature, with every claim linked to its source.
Proof: a synthetic demo you can download and run.
Kernel: the definitions, facts, presumptions and owned commitments the design rests on, with each consequence labeled and every change logged.
Foundation: the contract, decision policy, benchmark protocol and schemas as a downloadable package with validators, plus the design synthesis.
Library: every source document and file, searchable, each with a rendered view and the raw download.
Provenance: how the site and its evidence base were built, checked and audited.

Glossary

Older and archived documents on this site use other names for the same ideas. This is how they map to the labels used here.

attempter, worker: contributor.
cannot verify, unsupported, insufficient: Insufficient evidence.
instructions insufficient, contract-ambiguous, undecidable: Instructions don't decide.
pass, fail, unable-to-verify, instructions-underdetermine: item verdicts, the layer that claim results roll up into.

Problem

Non-consensus has four causes - unclear instructions, reviewer error, missing expertise, legitimate split. One agreement score mixes them. Driving kappa as the goal confuses error with ambiguity.

Throughput pressure plus "make them agree" is when unmeasured automation becomes a casting vote. That path is refused here.

Method in brief

Each claim in the review is supported, contradicted, short of evidence, or left open because the instructions don't decide. Unclear instructions count against the owner, not the contributor. People own the contract, any consequence, and the audit. Detail: system · KERNEL.

Not claimed

Universal judge; automated rejection on subjective criteria without measurement.
Praise verification without in-house gold.
Removal of expert audit cost.
Paper numbers substituting for measurement on human writeups.
Agreement rising only because people imitate the model.

Evidence posture

Architecture case is durable. Specific performance on this workload is open (transfer gap). Research · Evidence ledger.

Guide

Theory, the literature, the pipeline, the demo, and the archive.

Need	Go here
Theory (L0-L8, long records)	Limits
Literature map	Research · Evidence ledger
Pipeline and claim types	System · KERNEL.md
Human packet schema	Resolution example · Decision policy
Measurement pilot (optional later)	Pilot protocol
Synthetic demo	Proof · miniqa zip
Full archive	Library

AutoQA operating model

Status: Decision-ready operating model
Updated: 2026-07-23

Purpose

AutoQA checks whether human review decisions are supported by the project instructions and the evidence available for the work.

It exists because reviewer non-consensus is expensive and opaque - not because a model should become a casting vote. It exists under real pressure for throughput. That pressure is exactly when unmeasured automation becomes a weapon against data quality and workers.

AutoQA is not approved to reject work, change pay, restrict accounts, or make other consequential decisions until a specific lane meets release criteria and a separate decision says so.

The problem

Human reviewers disagree for several reasons:

The instructions are unclear.
A reviewer made an error.
The decision requires expertise the reviewer did not have.
More than one interpretation is reasonable.

An agreement score does not distinguish among these cases. AutoQA is intended to say whether a review decision is supported, contradicted, short of evidence, or open because the instructions don't decide.

Consensus is not the objective function. Explaining disagreement, fixing instructions, catching checkable error, and preserving legitimate splits is. Agreement may rise as a side effect. If it only rises because people imitate AutoQA, that is monoculture, not quality.

What AutoQA evaluates

Two objects, kept separate:

The work: Did the submitted work satisfy the project instructions?
The review: Did the reviewer's label, rationale, cited evidence, criterion, and severity accurately describe that work?

A correct work product can receive an incorrect review. Collapsing them turns review error into fake work-quality error.

The project contract

Each project needs an approved, versioned contract before evaluation:

project instructions;
criteria used to evaluate work;
evidence AutoQA may use;
examples of acceptable and unacceptable outcomes;
how severity is assigned;
how findings map to actions;
who owns instruction changes and exceptions.

Models change. The contract remains the source of truth.

How an evaluation works

Validate the record. Instructions, evidence, work, and review present and same project version.
Identify relevant criteria.
Break the review into claims.
Find supporting evidence for each material claim.
Evaluate each claim: supported, contradicted, insufficient evidence, or instructions that don't decide.
Evaluate work and review separately.
Apply project decision policy (advisory until a lane graduates).
Route unresolved cases - only decision-changing uncertainty to a qualified human.
Store the audit record - inputs, versions, evidence links, findings, model config, human decisions, policy trace.

Result types

Supported - evidence supports the claim.
Contradicted - evidence shows the claim is wrong.
Insufficient evidence - the record isn't enough, and that isn't proof of error.
Instructions don't decide - the rules don't pick an answer, so it goes to the owner, not the contributor.

Decision rules

Use project instructions and permitted evidence - not model preference or reviewer consensus as ground truth.
Require evidence for material findings.
Keep quality and consequence separate.
Represent uncertainty directly; abstain when record, instructions, or measured capability is insufficient.
Treat model confidence as uncalibrated until measured.
Keep consequential decisions human until a specific lane meets release criteria.
Preserve replayability.
Throughput, coverage, and automation-rate KPIs must not disable abstention or force pass/fail.
Do not use AutoQA solely as a casting vote to manufacture consensus on undecidable work.

Human responsibilities

People remain responsible for:

approving the project contract;
resolving unclear or conflicting instructions;
creating and adjudicating benchmark data;
reviewing consequential or unresolved cases (one decision-changing question, evidence first, hide provisional model answer for gold and consequence);
handling appeals;
approving any change in system authority;
monitoring reliability after deployment - including blind audit of auto-accepts and model-human agreements.

When AutoQA escalates, the human receives criterion, relevant evidence, and unresolved question - not a steered essay toward the model's preferred conclusion.

Measurement and release

Measure current human-review baseline and disagreement causes.
Build adjudicated development and held-out data for one project.
Evaluate AutoQA by criterion and case type in shadow mode.
Recommend: stop, narrow, remain assist-only, or approve a specific lane for further graduation work.

A lane is not eligible for autonomous use merely because it looks useful. It needs pre-defined accuracy, false-positive risk, stability, calibration, auditability, and monitoring on untouched test data.

Any approval applies only to tested project, contract version, criterion family, model configuration, and action. Expansion requires a separate decision.

Known risks

missed evidence in long records;
weak omission and unsupported-praise detection;
correlated errors among workers, reviewers, and models;
prompt injection / manipulation;
rubric and instruction drift;
over-reliance and automation bias when humans see model scores;
project and criterion transfer failure;
unrepresentative benchmarks;
privacy, worker-impact, and legal requirements;
KPI override of abstention under volume pressure;
monoculture if AutoQA becomes the agreement target.

Controls: held-out data, shadow deployment, human review, appeals, recurring audits that include agreement cases. Architecture alone does not solve these.