The AutoQA Kernel

What this document learned to be

Three adversarial reviews converged on one lesson, and this version finally obeys it: the facts in this domain are limitative, and the architecture is chosen. Mathematics tells you what cannot be presumed - not what to build. Every attempt (v0.1, v0.2) to make the design fall out of the facts as theorems was found, correctly, to be smuggling. So v0.3 is smaller where it claims necessity and more explicit where it chooses: a thin layer of genuinely mathematical facts; presumptions the domain warrants, stated as presumptions with measurement obligations; commitments owned as commitments, each with a named owner and a version; and consequences labeled by what they actually are. This is not a retreat from rigor. For a system whose product is honest verdicts, the foundation's rigor is its refusal to claim more than it can show.

Scope and the migration principle

The kernel governs evaluative claims about open-ended human work for which no sound, complete certificate system exists - "this critique is well-grounded," "this severity matches the evidence," "nothing important is missing" - judged against natural-language instructions and a bounded evidence record. Where a criterion admits a total, sound decision procedure, it is not the kernel's problem: it migrates to the deterministic lane and ordinary software-correctness discipline. As capability and formalization advance, checks may graduate out (and mis-formalized checks may migrate back). The kernel's claims concern the residue; the residue's boundary is re-surveyed every generation (I1).

Tags

• [TELOS] - the stipulated purpose.
• [DEF] - thin lexical definitions; vocabulary only.
• [FACT] - mathematical/logical constraints, stated minimally. All are limitative: they say what may not be presumed or claimed, never what must be built.
• [COND] - warranted presumptions of this domain: defaults the program adopts pending measurement, each tied to the interface that measures it. Not facts; not pretended to be.
• [CHOSEN] - normative commitments, owned and versioned. Motivations are given as motivations; where a motivation contains a causal claim, it is a belief held, not a theorem proved.
• [LAW] - the thin legal-floor rule; current statutes and doctrines live on the versioned legal interface (I7), because law is external and changes.

Parts II and VI are meta-analysis and carry no tags by design. Commentary is marked and non-load-bearing.

The discipline. A design element is legitimate only as: (a) [DERIVED] - entailed by named elements; (b) [UNPACKED] - a consequence of a CHOSEN commitment under named pressure; (c) [DESIGN RESPONSE] - one admissible architecture satisfying kernel constraints, adopted and replaceable; (d) an operationalization filling a declared interface slot with a measured-error obligation; (e) a governed choice at a declared Part V choice-point - with a named owner, a version, and an effective date. Anything else is an explicit kernel amendment, or it does not ship.

Part I - The kernel

K0 [TELOS]. The enterprise exists to increase the validity of training data about genuine human judgment: to make the recorded signal about what humans actually judged, and why, more truthful, more traceable, and more usable - not to make workers agree, not to make reviews uniform, and not to replace the judgment the data exists to capture.

Commentary - the motivating problem, named. This enterprise is being considered because human reviews of human-labeled data are not reaching consensus. K0 takes a position on that problem rather than dodging it: non-consensus is a symptom with four distinct causes - ambiguous instructions, reviewer error, missing expertise, and legitimate difference of judgment - and a system that targets consensus itself treats all four identically, by erasure. That cure kills the patient twice: it caps the system's validity at the reviewers' own unmeasured reliability, and it manufactures agreement exactly where the recorded signal should show a defensible plurality. What the organization needs from its non-consensus problem is explanation, pricing, and reduction at the source: instruction ambiguity routed to the owner and fixed, error caught claim by claim, expertise gaps routed to competence, and genuine plurality recorded as such. Consensus in application then rises as a byproduct of removing its illegitimate causes - it is the one thing the system must never pursue directly. (This is commentary; the operative content lives in K0, D1, C1, and K14's underdetermine disposition.)

Definitions

K1 [DEF]. "Project" names a triple (instructions, evidence policy, owner) - the owner being the party holding amendment authority over the first two. (The program takes on work only after it is in this form: a scope condition.)

K2 [DEF]. "Quality" names whatever construct a project-version's contract fixes as the target of evaluation. Which construct - see choice-point CP1.

K3 [DEF]. "Verdict" names a claim about quality issued for a project-version. What verdicts must carry is protocol (K14).

K4 [DEF]. "Evaluator" names anything that issues verdicts - human, model, or hybrid. Kernel requirements attach to verdicts, never to substrates.

Facts (limitative)

F1 [FACT] - Certificates, and their absence. A sound, checkable certificate lets a verifier certify work independently of its own generative competence. Absent such a certificate, no certificate-based entailment exists: a reliability advantage of evaluator over producer is then not a logical consequence of anything - it is an empirical property, if it is anything. (No ceiling is asserted: decomposition, tooling, repetition, and discriminative reading can outperform single-attempt generation. The corresponding protocol rule - unmeasured advantage never enters routing or reporting - lives in K14 and I5, where norms belong.)

F2 [FACT] - The witness asymmetry, stated narrowly. For an explicitly existential claim with a valid witness relation, one verified witness suffices. A universal/absence claim ("no defects," "nothing missing") is warranted only to the extent of a sound completeness argument - and absent one (the kernel's domain, by scope), a negative search result warrants only a claim about its disclosed procedure and scope.

F3 [FACT] - Small-signal arithmetic. (i) Among accusations, the false share is a function of prevalence, sensitivity, and specificity - as illustration only: at 3% prevalence, 95% sensitivity, 99% specificity, roughly a quarter of fail-verdicts are false. (ii) Absent formal assurance, error rates are known through adjudicated observation, whose precision depends on sample size, dependence structure, and the estimation target - and the adjudication is itself fallible, so reference error is part of the accounting (CP3). (iii) Calibration and coverage guarantees are conditional on assumptions linking calibration data to deployment (exchangeability-class); nothing in the mathematics discharges those assumptions. (Whether they hold in a given deployment - including deployments that change their own population - is measured transport, I4/I6.)

F4 [FACT] - Agreement does not self-certify. Marginal accuracies plus agreement do not identify ensemble correctness without assumptions or bounds on the joint error distribution; in the limit of perfectly shared errors, additional agreeing evaluators add no evidence. Whether given evaluators' errors are dependent is an empirical property of the ensemble at hand. (The rule that it is measured or structurally justified, never assumed away, is P3's presumption discipline and K14's protocol - not mathematics.)

F5 [FACT] - No capability supplies missing premises. Where the instructions-plus-record underdetermine a verdict, the missing information is not in the inputs, and no evaluator capability manufactures it; a better evaluator converges on better detection of underdetermination. (Whether and how often material underdetermination occurs in a given project is not asserted here - it is presumed by P1 and measured at I6.)

Presumptions of the domain

Each is a default the program adopts, with the interface that must keep it honest. None is capability-invariant truth; all are re-examined per project and generation.

P1 [COND] - Underdetermination occurs materially. Natural-language instructions over open-ended evaluative work are presumed to leave a material share of admissible cases underdetermined. Measured at I6 (the variance-attribution baseline exists to measure exactly this); a project where it measures near zero has little kernel-governed residue.

P2 [COND] - The population is adaptive. The evaluated population is paid, informed, persistent, and tool-equipped; consequential legible criteria are presumed to create non-negligible incentive and information for boundary-seeking, and incentives on measured dimensions are presumed to divert effort from unmeasured ones (the multitask mechanism). Magnitudes and actual behavioral response are measured at I6, never assumed.

P3 [COND] - Dependence is the default presumption among substrate-sharing evaluators. Where evaluators (machine or human) materially share training, tooling, or process, error dependence is presumed until measured otherwise (F4 sets the stakes of the presumption). The ecology census - who shares what substrate, including whether producers and evaluators are resource-symmetric - is itself an interface measurement (I9), re-taken each generation. No trend about the future supply of independence is asserted.

P4 [COND] - Frontier mass is presumed nonzero until measured. Some share of incoming work is presumed to sit where the evaluator can demonstrate no reliability advantage (F1) - from shared substrate (P3), producer assistance, or task difficulty. That share ("frontier mass") is a measured deployment quantity (I1/I6), per project and generation; assisted and escalation lanes persist while measured frontier mass exceeds declared thresholds - in either direction. The v0.1-v0.2 claim that this band permanently "does not empty" is retracted as unproven. (The standing obligation to measure frontier mass, and the routing capability that acts on it, are K14-protocol and D2-architecture respectively - a COND carries presumptions, not obligations.)

P5 [COND] - Reading long records is unreliable evidence discovery, in three distinct ways. Within this domain, an evaluator misses in-record evidence through (i) attention dilution - the passage is lost in length; (ii) recognition failure - the evaluator reads the relevant passage and does not connect it to a claim phrased in functionally equivalent but semantically distant terms; and (iii) compositional distribution - the evidence is not a passage at all but a property of the trajectory, assembled from state across distant parts of the record, so no local reading can see it. All three failures are silent. Chunking addresses only (i); similarity-based retrieval addresses none of them and is structurally worst at (ii). A per-chunk coverage ledger over failures (ii) and (iii) is false confidence: a complete-looking record of an examination that could not have seen the evidence. Presumed binding until measured (I10) with plants that specifically include semantically distant and cross-chunk distributed evidence - never lexical needles alone - per record-length and claim-type bucket, re-measured each generation.

Commitments

Owner of K-commitments: program governance (the kernel's own amendment authority), version-stamped with the kernel.

K13 [CHOSEN] - Owner authority and the citable bar. Prospective interpretation may be delegated; settling a contested or novel meaning is reserved to the project owner and takes effect as a versioned amendment; the bar workers are judged against never changes silently, and no later bar is applied retroactively to earlier work. Motivation: some authority structure must be chosen for evaluation to be reproducible; this one keeps every verdict traceable to a citable version.

K14 [CHOSEN] - The verdict protocol. An admissible verdict cites the instruction clauses it applies and the evidence it rests on; carries calibrated confidence; is issued under a declared evidence-integrity regime (provenance and completeness checked before grounding; record gaps disclosed and routed to record repair, not to worker detriment); and no synthetic verdict is issued below the project's declared admissibility confidence - the evaluator abstains or escalates instead. The verdict set includes an explicit underdetermine disposition. One further clause, added at v0.3.3: a verdict's grounds include coverage accounting - which parts of the record were examined against which claims, by what procedure - and any claim of record-level support or non-contradiction extends no further than the coverage actually achieved. This is F2's disclosed-scope rule applied inside the record, where P5 says a single pass cannot be presumed to have looked everywhere. Reflexivity: verdicts, feedback, and adjudications are themselves evaluative work products and owe this protocol.

K15 [CHOSEN] - Contestability, everywhere. Any verdict that binds a person must be reconstructible from its grounds by that person and contestable through a channel that can change the outcome - for all workers, everywhere, not merely where law compels it. Motivation, held as belief: unaccountable verdicts corrupt the K0 record silently; and the scope beyond legal compulsion is a worker-dignity commitment this program elects by name.

K16 [CHOSEN] - Measurement integrity. The system never knowingly reports a biased number as unbiased, and disposition pressure acts downstream of the recorded measurement or not at all. Numbers estimated under assumptions are reported as such.

K17 [CHOSEN] - Specification-risk allocation. The cost of ambiguity the worker did not create sits with the specification's owner. Workers may be contractually obligated to flag ambiguity; they are never penalized for its existence. This protection is categorical: it is not tradeable against any error price (CP4). Motivation, held as belief: assigning ambiguity cost to the party with amendment power generates correction; assigning it to workers generates fear and hiding.

K18 [CHOSEN] - Separation of measurement, attribution, and sanction - constitutive. "This work has a defect," "this person is responsible," and "this consequence is warranted" are, in this program, three determinations with three evidence standards by construction. No automation of the first constitutes the second or third; the second and third have their own proceedings and cannot be collapsed by any owner's error-price declaration (CP4 is subordinate to this).

K19 [CHOSEN] - Symmetric threat model. The program treats every actor with power over the loop - workers, reviewers, adjudicators, project owners, operators, buyers - as within the threat model, by policy extending to them the same presumption of incentive exposure that P2 warrants for the workforce. Integrity controls (audits, logs, gold custody, disclosure) are owed upward and sideways, not only downward; the concrete control matrix per actor is a design obligation (D5) with acceptance criteria at I6.

K20 [CHOSEN] - Construct-validity duty (detection and escalation). Commentary (the motivating possibility, not part of the commitment): instructions can mis-capture the judgment the data exists to serve. The commitment: where evidence of that accumulates (clustered legitimate disagreement, instruction-gap patterns, divergence between what instructions reward and the project's stated purpose), the program reports it to the project owner and, where contracted, the buyer - through the named escalation channel in the decision register. This duty is detection and escalation, not defense: the kernel cannot force a bad project to stop (CP5 governs what happens next, above this document).

K21 [LAW] - The legal floor, thin. Jurisdictional law binding on a project's workforce constrains the program as a floor that no internal choice may undercut. Current statutes, doctrines, scope determinations, and counsel opinions live on the versioned legal interface (I7) and are censused per project, two-sided (EU and US at minimum) - they are load-bearing there, and perishable there, by design.

These are design notes on how the cited law could constrain a QA system, not legal advice.

K22 [CHOSEN] - The strictest-regime rider, made unambiguous. This program adopts globally, by choice: decisions of significant detriment to a worker are taken by a human with real authority and competence - everywhere, not only where a directive compels it. (This resolves what v0.2 left ambiguous: the human-decision floor is global by election; I7 tracks where it is also legally mandatory.)

K23 [CHOSEN] - Fairness commitment. For any binding use, group-specific error burdens (above all: false-accusation rates) are measured and governed. Which classifications, which disparity measures, and what bounds are acceptable is a governed choice at CP6 with legal floors from I7 (adverse-impact doctrine where applicable); that the measurement and governance exist at all is committed here and is not optional per project.

K24 [CHOSEN] - Exculpatory reopening. Correcting the record in a worker's favor is always permitted; applying a later, stricter bar to earlier work never is (K13). Reopening a past leniency under the original bar is permitted only through the K18 proceedings, never by re-scoring alone.

Part II - Dependency and rationale map

Meta-analysis; untagged by design (the tagging rule governs kernel statements, not this audit of them). This is a map of currently stated rationales - removing an element removes its stated ground, which is not a proof that no other ground could be constructed.

Review trail