Benchmark & Calibration Protocol

How to construct adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.

AutoQA Benchmark, Calibration, and Launch Protocol

Protocol version: 0.1.0
Applies to schema version: 0.1.0

Part of Foundation

1. Purpose

This protocol defines how to determine whether an AutoQA configuration is reliable enough for a specific project and criterion family.

The benchmark is not merely a leaderboard for an overall judge score. It must test whether the system:

• applies the intended criteria;
• finds affirmative evidence of quality;
• detects defects and omissions;
• links findings to the correct evidence;
• distinguishes grounding, truth, relevance, and warrant;
• audits human annotations faithfully;
• recognizes ambiguity;
• calibrates confidence;
• escalates selectively;
• produces useful, safe feedback;
• remains stable under irrelevant presentation changes;
• improves the actual production workflow.

A configuration includes all behaviorally relevant components:

model provider and version
prompt templates and versions
reasoning or sampling mode
retrieval and chunking
claim and expected-element extraction
validators and tools
reference-answer timing
ensemble or challenger policy
calibration profile
decision-policy version

Changing any of these may require partial or full revalidation.

2. Evaluation levels

The benchmark reports results at four levels.

2.1 Observation level

Can the system correctly identify claims, expected elements, evidence spans, positive findings, defects, omissions, counterevidence, and ambiguity?

2.2 Criterion level

Can it assign the correct applicability, verdict, severity, and substatuses for grounding, truth, warrant, and relevance?

2.3 Decision level

Can the versioned policy produce the correct action for each target:

• attempt;
• annotation;
• composite case;
• batch, where used?

2.4 Workflow level

Does the system reduce human effort and improve data quality without creating unacceptable false acceptance, false rejection, reviewer anchoring, or feedback-induced errors?

3. Governance roles

At minimum, assign these roles:

• Project owner - defines intended project outcome and accepts operational risk.
• Rubric owner - maintains the contract and resolves instruction interpretation.
• Domain expert - adjudicates cases requiring subject expertise.
• QA owner - owns benchmark design, sampling, metrics, and launch gates.
• Benchmark curator - freezes cases, manages splits, and prevents leakage.
• System owner - owns model configuration, prompts, validators, and deployment.

A single person may fill more than one role in a small project, but contract approval and benchmark scoring should not be controlled exclusively by the person optimizing the evaluator.

4. Freeze the target before testing models

Model comparison begins only after a project contract is stable enough to adjudicate.

Before benchmark construction:

1. include and hash all authoritative instructions;
2. compile criteria;
3. define evidence classes and source policy;
4. add positive, negative, and boundary anchors;
5. list valid alternatives;
6. define severity independently from action;
7. define attempt, annotation, and composite decision rules;
8. identify deterministic validators;
9. identify criteria requiring domain expertise;
10. approve contract version 1 for benchmarking.

Do not repeatedly alter the rubric to favor a model during test-set evaluation. Changes discovered during development create a new contract or benchmark version and must be documented.

5. Benchmark corpus design

The corpus contains two complementary distributions.

5.1 Natural-prevalence set

Sample representative production cases using the expected live distribution of:

• tasks;
• reviewers;
• difficulty;
• answer length;
• evidence volume;
• labels;
• failure types;
• languages;
• domains;
• data-generation sources.

This set estimates operational performance and workload.

5.2 Challenge set

Intentionally oversample difficult and safety-critical cases. Required challenge categories include, where applicable:

• strong positive attempt;
• subtle valid alternative to a reference;
• polished falsehood;
• true but unsupported claim;
• accurately cited weak source;
• citation mismatch;
• partial support;
• causal or inferential overreach;
• omitted required content;
• selective evidence use;
• correct conclusion with invalid rationale;
• incorrect conclusion with plausible reasoning;
• correct human label with wrong reason;
• wrong human label with correct observation;
• severity edge case;
• instruction conflict;
• reasonable disagreement;
• rubric ambiguity;
• domain-expertise dependency;
• long-context contradiction;
• prompt injection;
• rubric parroting;
• position-bias probe;
• verbosity and formatting probe.

Challenge-set rates must not be used as estimates of production prevalence.

5.3 Positive-data requirement

At least one benchmark slice must be specifically designed around high-quality attempts. It should include:

• concise complete answers;
• detailed complete answers;
• unconventional but valid approaches;
• qualified conclusions;
• correct disagreement with a reference;
• evidence synthesis rather than extraction;
• strong answers with minor stylistic imperfections;
• good attempts incorrectly rejected by reviewers.

This slice is necessary to measure false rejection and false praise independently.

5.4 Omission requirement

Include cases where all present claims are accurate but required content is missing. Without this slice, a claim verifier can appear highly reliable while failing completeness.

5.5 Annotation-fidelity requirement

Include all four combinations:

Also include:

• correct label, wrong rationale;
• wrong label, useful observation;
• correct criterion, wrong severity;
• wrong criterion, correct span;
• reasonable disagreement;
• rubric defect.

6. Unit of sampling and leakage control

Cases that share any of the following may leak across splits:

• same underlying task template;
• same source document or evidence bundle;
• paraphrases of the same attempt;
• same reference answer;
• same synthetic mutation family;
• adjacent items from the same conversation;
• repeated work from one original datum.

Group such cases before splitting. All members of a group must remain in one split.

The benchmark item records:

• split;
• first freeze date;
• whether prompt development may use it;
• whether it belongs to a held-out project;
• access policy;
• challenge attributes;
• slice tags.

7. Recommended splits

Development

May be inspected while designing prompts, validators, and extraction logic. Never use for final reporting.

Calibration

Used to fit confidence mappings, ensemble weights, and action thresholds. Do not use to select among unlimited prompt variants without accounting for overfitting.

Test

Locked before final model selection. Used for primary in-domain performance reporting.

Challenge

Locked adversarial and edge-case suite. Report separately from natural prevalence.

Held-out project

A project, contract, or task family excluded from architecture and prompt development. Tests whether the supposedly project-agnostic engine transfers to a new contract.

Shadow

Recent live cases evaluated without changing production decisions. Used to measure current distribution and drift.

8. Gold-standard construction

8.1 Adjudicator selection

Use domain-qualified adjudicators for criteria that require expertise. General QA adjudicators may evaluate mechanical, instruction-following, and straightforward grounding criteria.

The benchmark must record domain-expert count.

8.2 Independent first pass

For consequential or subjective cases, use at least two independent first-pass judgments before discussion.

Each adjudicator should record:

• criterion applicability;
• verdict;
• positive evidence spans;
• negative evidence spans;
• omissions;
• severity;
• acceptable operational actions;
• ambiguity status;
• confidence or uncertainty notes.

Preserve initial judgments before adjudication. Do not overwrite disagreement history.

8.3 Blinding

Adjudicators should normally be blind to:

• AutoQA outputs;
• model identity;
• model-generated rationale;
• production consequence;
• other adjudicators' initial labels.

Whether adjudicators are blind to the original reviewer label depends on the task:

• for independent attempt gold, hide the reviewer label;
• for annotation-fidelity gold, reveal the review only after the independent attempt judgment is frozen.

The benchmark item records blinding status.

8.4 Adjudication meeting

Adjudication should resolve whether disagreement arose from:

• missed observation;
• evidence authority;
• applicability;
• criterion interpretation;
• inference;
• threshold;
• severity;
• expertise;
• legitimate plural judgment;
• rubric defect.

Do not force a single verdict when the correct gold state is reasonable_disagreement, rubric_ambiguous, or insufficient_evidence.

8.5 Acceptable verdict sets

Gold may contain more than one acceptable verdict for boundary cases. Examples:

acceptableVerdicts = ["met", "partially_met"]
acceptableDecisions = [
 { target: "attempt", acceptableActions: ["accept_with_notes", "rework"] }
]

This is preferable to labeling one defensible judgment as objectively wrong.

8.6 Gold evidence

For each criterion, gold should include:

• decisive positive spans;
• decisive negative spans;
• required omissions;
• notes on valid alternatives;
• severity;
• acceptable verdicts.

An evaluator should not receive full gold rationales during test execution.

8.7 Gold repair

If adjudication reveals a contract defect, do not patch only the gold label. Revise the contract, version it, and re-adjudicate affected cases.

9. Sample-size planning

Sample sizes must be driven by the operational error budget and desired confidence interval, not a single universal minimum.

9.1 Proportion estimation

For a rough planning approximation, the worst-case sample size for a proportion with two-sided 95% margin of error e is:

n ~ 1.96^2 x 0.5 x 0.5 / e^2

This yields approximately:

• +/-10 percentage points: 97 cases;
• +/-7 points: 196 cases;
• +/-5 points: 385 cases;
• +/-3 points: 1,068 cases.

Use Wilson or exact binomial intervals in final reporting rather than relying only on this approximation.

9.2 Rare critical errors

If zero errors are observed in n independent cases, the approximate one-sided 95% upper bound is 3/n.

Therefore, observing no critical false accepts in 300 representative autonomous accepts supports an upper bound near 1%, not proof of zero risk.

Rare critical failures should be oversampled in the challenge set while also measured at natural prevalence.

9.3 Per-slice planning

A broad benchmark can hide a failing subgroup. Plan enough cases for critical slices such as:

• criterion family;
• severity;
• task type;
• language;
• domain;
• long versus short context;
• high versus low human agreement;
• reviewer cohort;
• model-generated versus human-generated attempt;
• adversarial versus ordinary.

Very small slices should be reported as exploratory and must not support autonomous action gates.

10. Model and architecture bakeoff

Compare configurations by criterion family. A universal winner is not assumed.

10.1 Minimum configurations

Test at least:

1. a deterministic-only baseline where applicable;
2. a single holistic model judge;
3. criterion decomposition;
4. criterion decomposition plus claim/evidence mapping;
5. separate positive, negative, coverage, and warrant observation channels;
6. primary judge plus independent challenger;
7. selective escalation using calibrated confidence.

10.2 Required ablations

10.3 Perturbation tests

Run behavior-preserving variants of the same case:

• swap candidate order;
• change rubric order;
• rename criterion identifiers;
• alter harmless formatting;
• shorten or expand nonmaterial prose;
• move references before or after independent analysis;
• paraphrase evaluator instructions without changing meaning;
• insert irrelevant provenance cues;
• vary source order;
• vary answer position in long context.

Record decision flip rates and criterion-specific flip rates.

A flip is not necessarily an error when the original case is ambiguous. Report flips separately for clear and ambiguous gold cases.

10.4 Adversarial tests

Test attempts and evidence containing:

• direct evaluator commands;
• fabricated system messages;
• requests to ignore the rubric;
• fake authority claims;
• hidden scoring instructions;
• answer-key mimicry;
• irrelevant but persuasive citations;
• language copied from positive anchors without the required behavior.

11. Core metrics

No single metric is sufficient.

11.1 Criterion verdict metrics

Report per criterion and macro-averaged:

• precision, recall, and F1 for met;
• precision, recall, and F1 for not_met;
• confusion among met, partial, not met, ambiguous, unknown, and not applicable;
• severity accuracy;
• applicability accuracy;
• cost-weighted criterion loss.

For multi-acceptable gold, count a prediction correct when it belongs to the acceptable set.

11.2 Critical operational errors

Report separately:

• critical false-accept rate;
• strong-positive false-reject rate;
• autonomous reject precision;
• autonomous accept precision;
• rework precision;
• escalation recall for ambiguous or unknown cases.

Always provide numerator, denominator, and confidence interval.

11.3 Positive-finding metrics

Report:

• positive-finding precision;
• positive-finding recall;
• false-praise rate;
• strength-to-criterion attribution accuracy;
• positive span precision and recall;
• required-behavior confirmation rate.

A system can have good defect recall and poor positive-finding reliability. These must remain separate.

11.4 Negative and omission metrics

Report:

• negative-finding precision and recall;
• omission precision and recall;
• defect-to-criterion attribution accuracy;
• materiality accuracy;
• counterevidence recovery;
• unsupported-finding rate.

11.5 Claim and evidence metrics

Report:

• material-claim extraction precision and recall;
• central-claim recall;
• compound-claim split accuracy;
• claim-to-evidence relationship accuracy;
• evidence-span precision and recall;
• source-quality classification accuracy;
• citation completeness;
• contradiction detection;
• entailment versus weak-support confusion.

Evidence span scoring should include:

• exact match;
• token overlap or intersection-over-union;
• adjudicator-accepted alternate span match.

11.6 Warrant metrics

Report accuracy on:

• association versus causation;
• possibility versus certainty;
• contribution versus primacy;
• subgroup versus population;
• descriptive versus normative inference;
• calibrated uncertainty;
• counterevidence acknowledgment.

11.7 Annotation-audit metrics

Report:

• label-support accuracy;
• rationale-support accuracy;
• criterion-attribution accuracy;
• evidence-alignment accuracy;
• severity-alignment accuracy;
• omitted-positive recall;
• omitted-negative recall;
• disagreement-type accuracy;
• false accusation rate against reviewers.

The false accusation rate is particularly important if analytics will affect reviewer assignments, pay, or standing.

11.8 Calibration metrics

Report:

• Brier score;
• log loss where probabilities are available;
• expected calibration error with disclosed bins;
• reliability diagram;
• accuracy by confidence band;
• risk-coverage curve;
• coverage at project target risk;
• selective false-accept and false-reject rates.

Calibration must be evaluated on data not used to fit the calibration map.

11.9 Stability metrics

Report:

• order-swap flip rate;
• rubric-paraphrase flip rate;
• harmless-format flip rate;
• repeated-run agreement;
• model-version drift;
• source-order sensitivity;
• reference-timing sensitivity.

Stability is not correctness. Report both.

11.10 Efficiency metrics

Report:

• median and tail latency;
• inference cost per case;
• model calls per case;
• tokens per stage;
• human seconds per escalated case;
• percentage of cases receiving a human question;
• corrected decisions per human minute;
• cases processed per reviewer hour.

11.11 Feedback metrics

Report:

• successful revision rate;
• residual defect rate after revision;
• new-defect introduction rate;
• repair validity;
• feedback evidence-link accuracy;
• user-rated usefulness;
• edit distance or effort where meaningful;
• repeated-failure rate after feedback.

A second model's "helpfulness" score is not sufficient evidence that feedback works.

11.12 Downstream metrics

When the system begins selecting training data, separately test:

• target-model performance after training;
• target-model calibration;
• slice performance;
• robustness;
• marginal gain versus random accepted data;
• redundancy;
• diversity and coverage;
• value per annotation or token.

Item validity and dataset value remain separate benchmark layers.

12. Cost-sensitive decision evaluation

Define a project loss matrix for target-specific actions.

Example structure:

Use project-specific values. Report both unweighted metrics and expected cost under estimated production prevalence.

13. Calibration procedure

13.1 Define the event

Confidence is meaningless unless tied to an event. Examples:

• criterion verdict is correct;
• decisive negative finding is correct;
• attempt acceptance is correct;
• annotation rework is correct;
• evidence relationship is correct.

Do not use one calibration mapping for unrelated events.

13.2 Fit on calibration split

Possible methods include:

• isotonic regression;
• logistic or Platt-style scaling;
• temperature scaling for suitable logits;
• conformal risk control;
• empirically defined confidence bands.

Select the simplest method that performs adequately and remains stable.

13.3 Calibrate by relevant slice

Separate profiles may be necessary for:

• criterion family;
• model role;
• task type;
• domain;
• language;
• context length;
• clear versus ambiguity-prone cases;
• positive versus negative findings.

Avoid creating so many profiles that each has too little data. Hierarchical or pooled calibration may be preferable when slices are sparse.

13.4 Lock and test

After fitting, freeze the profile and evaluate it on the locked test set. Report calibration degradation on challenge and held-out-project sets.

13.5 Recalibration triggers

Recalibrate when any of the following changes materially:

• model version;
• provider behavior;
• prompt template;
• reasoning mode;
• retrieval or chunking;
• contract semantics;
• source policy;
• decision policy;
• production distribution.

14. Launch gates

Projects must define numerical gates before reading final test results.

A gate should identify:

• target action;
• criterion or slice;
• metric;
• threshold;
• confidence level;
• minimum sample size;
• acceptable subgroup variance;
• fallback action.

14.1 Gate template

target: attempt
action: reject
scope: criteria tagged critical-hard-gate
metric: precision
threshold: PROJECT_DEFINED
confidence_requirement: lower_95_bound_above_threshold
minimum_cases: PROJECT_DEFINED
stability_requirement: order_flip_rate_below_PROJECT_DEFINED
fallback: escalate

14.2 Conservative initial policy

Until a criterion family meets its gate:

• deterministic, fully verified failures may be automatic;
• clear positive and negative semantic findings may assist humans;
• uncalibrated semantic accept or reject decisions must escalate;
• subjective failures should usually produce advice, rework, or escalation rather than autonomous rejection;
• reviewer-performance consequences must rely on adjudicated samples, not raw model disagreement.

14.3 Required gate families

At minimum define:

• autonomous attempt acceptance;
• autonomous attempt rejection;
• annotation acceptance;
• annotation rework or rejection;
• composite-case admission;
• critical-failure detection;
• strong-positive protection;
• human-escalation coverage;
• feedback repair safety.

15. Human-loop experiment

The one-interaction design must be tested rather than assumed.

15.1 Compare conditions

Randomize eligible uncertain cases to:

1. no human intervention;
2. full independent second review;
3. bounded decision-changing question;
4. model recommendation plus confirmation, if testing automation bias.

15.2 Measure

• final decision accuracy;
• human time;
• answer-changing rate;
• corrected decisions per minute;
• residual uncertainty;
• reviewer acceptance of wrong model suggestions;
• repeat-question rate;
• rubric clarification yield;
• user satisfaction;
• expertise-routing success.

15.3 Bounded-question quality

A question is useful when:

• it resolves the declared uncertainty;
• the answer maps unambiguously to criterion state;
• it changes or confirms a decision that matters;
• it requires less effort than a full review;
• it does not reveal irrelevant model conclusions that induce anchoring.

16. Feedback experiment

For repairable failures, compare:

• no feedback;
• generic criterion feedback;
• evidence-linked minimal repair;
• evidence-linked repair with validation.

Measure whether a blinded adjudicator accepts the revision and whether new defects appear.

Positive feedback should be evaluated for specificity and truth, not sentiment.

17. Reviewer analytics validation

Do not operationalize reviewer scores until the annotation auditor is separately validated.

17.1 Difficulty adjustment

Reviewer performance should be conditioned on:

• criterion;
• task difficulty;
• item ambiguity;
• evidence volume;
• domain expertise required;
• case severity;
• AutoQA confidence.

17.2 Consequential-use safeguard

For reviewer coaching, assignment, compensation, or discipline:

• use adjudicated samples;
• provide appeal;
• exclude reasonable-disagreement and rubric-defect cases;
• report uncertainty;
• require minimum sample size;
• avoid one aggregate score without criterion breakdown.

17.3 Drift signals

Monitor:

• systematic leniency or harshness;
• rising criterion-specific disagreement;
• recurring unstated preferences;
• attention failures;
• expertise-routing mismatch;
• batch-level acceptance anomalies.

18. Shadow deployment

Before autonomous action, run AutoQA in shadow mode.

For every shadow case, store:

• AutoQA target-specific decisions;
• production reviewer decision;
• whether AutoQA would have changed the workflow;
• confidence and calibration profile;
• decisive findings;
• human escalation question, if any;
• later adjudicated outcome for an audit sample.

Shadow reporting should answer:

• What percentage would AutoQA accept, rework, reject, or escalate?
• Which criteria drive each action?
• Where does model-human disagreement concentrate?
• How much reviewer time would be saved?
• How many good attempts would be incorrectly rejected?
• How many bad annotations would remain undetected?
• How often would the one human interaction change the decision?

19. Production monitoring

19.1 Ongoing audit sample

Randomly audit:

• automatic accepts;
• automatic rejects;
• rework cases;
• escalations;
• cases where model and reviewer agree;
• cases where they disagree;
• high-confidence cases;
• low-frequency slices.

Agreement cases require auditing because correlated error can create false reassurance.

19.2 Drift dashboard

Track over time:

• action rates;
• criterion failure rates;
• positive-finding rates;
• calibrated confidence distribution;
• risk estimates from adjudicated audits;
• order and prompt sensitivity;
• reviewer disagreement types;
• model/provider version;
• source-distribution changes;
• answer length and evidence volume;
• human question rate and value.

19.3 Drift triggers

Pause or reduce autonomy when:

• action rates shift without an explained production change;
• audited error exceeds the project budget;
• calibration deteriorates;
• a subgroup falls below its gate;
• model or prompt version changes;
• a new adversarial failure appears;
• repeated human questions expose a rubric defect;
• the provider changes behavior materially;
• source formats or context length move outside validated ranges.

19.4 Change management

Use shadow or canary evaluation for every material evaluator change. Compare old and new configurations on the locked suite and recent live cases.

Do not overwrite historical evaluation records. New runs receive new identifiers and preserve lineage.

20. Statistical reporting

Every benchmark report should include:

• case and criterion counts;
• prevalence by split and slice;
• point estimates;
• confidence intervals;
• denominators;
• paired comparisons where the same cases are used;
• uncertainty for rare errors;
• missing or failed evaluation count;
• human adjudication protocol;
• calibration method;
• model and prompt versions;
• inference cost and latency;
• known limitations.

For paired configuration comparisons, use an appropriate paired method such as:

• paired bootstrap for aggregate metrics;
• McNemar-style analysis for binary paired correctness;
• stratified resampling when project slices differ materially.

Avoid declaring a winner based on a small difference without uncertainty analysis.

21. Required benchmark report tables

21.1 Configuration manifest

21.2 Primary target decisions

21.3 Criterion metrics

21.4 Evidence and positive-quality metrics

21.5 Stability

21.6 Human loop

21.7 Cost

22. Release decision

A production release requires written approval that:

• contract and benchmark versions are frozen;
• action-specific gates were set in advance;
• locked test and challenge results meet the gates;
• calibration is valid for the intended slices;
• held-out-project results are understood;
• security tests pass;
• shadow deployment shows acceptable workflow impact;
• rollback is available;
• audit sampling is funded and staffed;
• reviewer analytics safeguards are active;
• known unsupported domains or slices are routed to humans.

23. Benchmark maintenance

Create a new benchmark version when:

• contract semantics change;
• new failure modes are added;
• gold labels are corrected;
• source snapshots change;
• split membership changes;
• challenge attributes change materially;
• adjudication policy changes.

Do not delete difficult cases because a model performs poorly. Correct them only when the contract or gold is wrong.

Retain retired benchmark versions for historical comparison.

24. Initial implementation plan

Phase 1 - contract and gold foundation

• author contract;
• create 100-200 diverse development cases;
• independently adjudicate a smaller calibration seed;
• implement schema and referential validation;
• implement deterministic checks;
• establish claim, expected-element, and evidence objects.

Phase 2 - architecture bakeoff

• test holistic and decomposed judges;
• test positive/negative/coverage/warrant passes;
• test model/provider options by criterion family;
• build prompt and order perturbation suite;
• expand calibration and locked test sets.

Phase 3 - selective AutoQA

• fit calibration;
• define risk-coverage thresholds;
• enable shadow decisions;
• test one-interaction human resolution;
• validate feedback repairs.

Phase 4 - governed autonomy

• enable only actions whose gates are met;
• retain human handling for unsupported slices;
• monitor and audit continuously;
• add reviewer and batch analytics only after annotation-audit validation.

25. Benchmark readiness checklist

A benchmark is ready when:

• natural and challenge distributions are separate;
• strong positive attempts are represented;
• omission cases are represented;
• all annotation-quality combinations are represented;
• cases are grouped to prevent leakage;
• adjudicators are qualified;
• initial judgments are preserved;
• ambiguity can remain in gold;
• evidence spans and omissions are recorded;
• split access is controlled;
• sample sizes support intended claims;
• action-specific loss and gates are defined;
• calibration and test data are distinct;
• model, prompt, and policy versions are recorded;
• a reporting template is precommitted;
• held-out and shadow evaluation are planned.

Linked from

• Foundation, Benchmark & Calibration Protocol
• Pilot, Benchmark & Calibration Protocol
• Library, Benchmark & Calibration Protocol