The bounded measurement pilot

AutoQA benchmark and pilot protocol

Version: 1.0-grounded
Checked: 2026-07-22

1. Purpose

Estimate the performance and consequences of one pinned AutoQA configuration on one named task and population. The protocol does not produce a general certificate.

2. Registration

Before accessing the locked test set, record:

• project, accountable owner, and decision;
• claim lanes and units of analysis;
• admitted evidence and retrieval sources;
• instruction, schema, model, prompt, parser, tool, index, and post-processing versions;
• outcome contract and ambiguity process;
• primary and secondary outcomes;
• representative, rare-case, challenge, and temporal sampling plans;
• exclusion and missing-data rules;
• stopping rules; and
• planned statistical analysis.

3. Reference construction

1. Sample from the intended population and oversample prespecified consequential strata for error analysis.
2. Obtain independent blinded ratings before adjudication.
3. Store each rater's outcome, evidence spans, reason, and ambiguity flag.
4. Preserve disagreement. Use response sets where more than one rating can reasonably apply.
5. Adjudicate under a documented rule. Report both pre-adjudication disagreement and adjudicated outcomes.
6. Keep development and test examples separate.

4. Data splits

5. Pipeline measures

Measure ingestion completeness, parse errors, candidate recall, retrieval recall, judgment performance conditional on adequate evidence, abstention/coverage, and end-to-end performance. If the pipeline is candidate-gated, end-to-end recall cannot exceed candidate recall (P05).

For each lane report:

• denominator, prevalence, and missing records;
• full confusion matrix;
• class-specific precision, recall, specificity, and negative predictive value where defined and decision-relevant;
• coverage and risk by abstention threshold;
• calibration and proper scoring rules only when outputs are interpreted probabilistically;
• uncertainty intervals compatible with the sampling design;
• cost, latency, and operational failures; and
• results for prespecified relevant strata, with privacy and sample-size caveats.

6. Human-loop experiment

Recommended arms:

1. human-only baseline;
2. evidence and provenance only; and
3. evidence plus AutoQA's draft verdict/rationale.

Randomize at a unit that limits contamination. Blind final adjudicators to arm where possible. Account for repeated observations by reviewer and item. The primary outcome should be reference-correct decisions. Secondary outcomes may include completion time, correction rate, reliance when the suggestion is wrong, overrides, perceived usefulness, and performance over repeated exposure.

7. Statistical planning

Choose sample sizes for the consequential error and required interval precision. For zero observed independent representative Bernoulli failures, the exact one-sided upper bound is 1 - alpha^(1/n) (S01). Do not use this illustration when the sample is not representative, events are dependent, prevalence is unknown, or the outcome is not binary.

8. Security and manipulation

Challenge-test fabricated citations, prompt injection, hidden text, malformed documents, contradictory instructions, evidence hiding, strategic over-documentation, and unsupported justifications. Preserve raw inputs and outputs needed for incident reconstruction, subject to the approved retention rule.

9. Stop conditions

Immediate stop: unauthorized disclosure, unauthorized write-back, unreconstructable configuration, automated adverse action outside authorization, or confirmed material compromise.

Review stop: consequential error above the prespecified threshold, unstable reference process, major distribution shift, or a configuration change that invalidates the frozen comparison.

10. Decision record

Decide separately for every lane: stop, revise/retest, shadow, advisory, or authority review. Record evidence, limitations, residual uncertainty, owner, monitoring, appeal path, expiration/review date, and change triggers.