The AutoQA Kernel

Part I — The kernel

The purpose

K0 [TELOS]. The enterprise exists to increase the validity of training data about genuine human judgment: to make the recorded signal about what humans actually judged, and why, more truthful, more traceable, and more usable — not to make workers agree, not to make reviews uniform, and not to replace the judgment the data exists to capture.

K0 is stipulated, not argued. It is the one commitment everything else serves, and the first test of any proposal is whether it increases or destroys the information K0 names. Two immediate consequences are worth stating up front because they are so often gotten wrong: a QA layer that homogenizes worker output destroys the very signal being purchased, and a QA layer that launders one arbitrary reading of an ambiguous instruction into "consistency" records a fiction. Both fail K0 before any capability question is reached.

Definitions

K1 [DEF] — Project. A project is a triple ⟨instructions, evidence policy, owner⟩: a written statement of intent; a rule fixing what counts as admissible evidence for claims about work done under that intent; and an authority entitled to amend both.

K2 [DEF] — Quality. The quality of a work product is the degree to which it satisfies the project's instructions, as grounded in the project's admissible evidence. Quality is thus a three-place relation — work, instructions, evidence record — not a property of the work alone, and not a prediction of any reviewer's impression.

K3 [DEF] — Verdict. A verdict is a claim about quality that carries its own grounds: the instruction clauses it applies, the evidence spans it rests on, and a confidence. A statement about quality without grounds is an opinion, not a verdict, whoever utters it — human or machine.

K4 [DEF] — Evaluator. An evaluator is anything that emits verdicts. The definition is deliberately substrate-neutral: everything the kernel requires of verdicts binds human reviewers, model judges, and hybrid pipelines identically. There is no separate standard for machines; there is a standard for verdicts.

K3 contains the kernel's reflexive move, and it does most of the work people expect a "philosophy of AI evaluation" to do: the QA layer's outputs are instances of the thing it evaluates. A judge's verdict is itself a claim requiring grounding in admissible evidence; feedback is itself an evaluative work product; the judge is itself a rater whose reliability is an empirical question. Nothing about being the evaluator exempts anything from the definition of quality. Whatever the system demands of an attempter's claims, it owes for its own.

Permanent constraints

Each is stated as narrowly as its grounds support. These are facts about the task, not about 2026 models; that is what qualifies them for the kernel.

K5 [FACT] — Underdetermination. No finite instruction text decides every case that will arise under it, and an evidence record can be insufficient for a claim without being silent about it. For some items, the correct output of a perfect evaluator is that the question has no determinate answer from the admissible inputs. This is a property of rules and records, not of evaluator skill; more capability converges on better detection of underdetermination, never on resolution the inputs do not contain.

K6 [FACT] — The existential/universal asymmetry. Verifying that support exists requires one witness; verifying that nothing required is missing, or that no defect is present, quantifies over an open space and can only be exhausted relative to an explicit, finite enumeration. Under bounded computation this asymmetry does not close — better search improves the constant, not the logic. Positive certification of absence is only ever as strong as the enumeration it checked.

K7 [FACT] — The verification-difficulty relation. For some claims, checking against the record is reading; for others, checking requires re-deriving the result, and there the verifier's reliability is bounded by its own competence at the underlying task. Two consequences are permanent. First, an evaluator cannot certify work beyond its own frontier — its concurrence there is an educated guess, whatever its confidence reports say. Second, because producers use the same technological substrate as evaluators, the producer-evaluator gap does not close the way the machine-versus-unaided-human gap does; there is always a band of work near the shared frontier, and adversarial producers can deliberately locate work inside it.

K8 [FACT] — Small-signal statistics. Three arithmetic facts govern any consequential evaluator. (i) When true failures are rare, the share of false accusations among accusations is set by prevalence and specificity, not by intentions: at 3% prevalence, 99% specificity yields roughly one false fail for every three true ones. (ii) Knowing an error rate precisely enough to bear consequences requires adjudicated samples whose size scales inversely with the rate being bounded — certainty about rare errors is bought, not assumed. (iii) Every calibration guarantee is conditional on the future resembling the calibration data; a deployed evaluator changes the population that produced it, by design, so calibration is a maintained property requiring standing measurement, never a shipped one.

K9 [FACT] — Incentive reflexivity. When stakes attach to a legible criterion, the criterion becomes an optimization target for the population it binds; the more specific the feedback, the better the gradient it publishes; and incentives concentrated on measured dimensions divert effort from unmeasured ones. This is not cynicism about workers; it is what optimization under a known objective is. No evaluator improvement removes it, because it is a property of the evaluated population's rationality, and that population is equipped with the same tooling as the evaluator.

K10 [FACT] — Non-independence of similar evaluators. Evaluators produced by similar processes on similar data share failure modes; agreement among them is therefore bounded evidence. The scarce resource in evaluation is not accuracy but independence, and it grows scarcer as human experts adopt the same tools. An architecture that needs k independent opinions cannot obtain them from k copies of the same epistemic substrate.

K11 [FACT] — Exteriority of meaning. What instructions mean in an unanticipated case is fixed, when it is fixed at all, by the owner's amendment — not by any evaluator's best interpolation, however good. Interpretation can be delegated; the authority to settle meaning cannot be, because settling meaning is amending the instructions (K1). Equally, the standing to impose detriment on a person is conferred by institutions, not by accuracy; no error rate, however low, self-certifies the right to punish.

Normative commitments

Three, and only three. Each is chosen; the defense given is motivation, not proof.

K12 [CHOSEN] — Contestability. Any verdict that binds a person must be reconstructible from its grounds by the person it binds, and contestable through a channel that can actually change the outcome. Motivation: K0 requires truthful records of judgment, and unaccountable verdicts corrupt the record silently; K11 already denies verdicts intrinsic authority; and where the workforce is in scope of platform-work law, most of K12 is independently mandated. But the kernel adopts it for all workers, everywhere, as a commitment — not merely where compelled.

K13 [CHOSEN] — Measurement integrity. The system never knowingly reports a biased number as if unbiased, never silently or retroactively moves a bar people are judged against, and never lets what is done about a measurement alter what is measured. Motivation: under K9, any channel by which disposition pressure can bend measurement will be found and used; a measurement layer that bends is not a lenient instrument but a broken one, and everything downstream of it — prices, trust, the K0 record — is quietly falsified.

K14 [CHOSEN] — Persons as sources, not targets. Workers are evaluated against the standard, never against each other or against the evaluator's own outputs; conformity to the machine is not a virtue the system may reward. Motivation: K0 — the data's value is the genuine, diverse judgment of the workers; a system that scores agreement manufactures the consensus it then records, consuming the asset it was built to protect. (K9 supplies the mechanism by which such scoring would propagate; K14 is the commitment not to build it.)

Legal givens

K15 [LAW]. Where workers are in scope of the EU Platform Work Directive: decisions of significant detriment must be taken by a human with real authority and competence; automated-supported decisions require plain-language explanation, a competent human contact, and substantiated review with a deadline; monitoring is allowlist-bounded. Where automated scores play a determining role in consequential decisions, rubber-stamp review legally counts as no review (SCHUFA doctrine). Where US workers are affected by automated employment decision tools, audit and validation obligations attach in enacted jurisdictions (e.g., NYC Local Law 144) and selection-procedure validation standards apply. Scope is a fact to census per project, not a preference; the kernel treats the strictest applicable regime as the design target because K12 independently demands most of it.

Part II — Kernel tests

Minimality. Remove K5 and the underdetermine verdict class loses its ground; remove K6 and absence-capping becomes a style choice; remove K7 and full automation of frontier work becomes arguable; remove K8 and consequences may precede measurement; remove K9 and static rubrics with published feedback become safe; remove K10 and model ensembles substitute for audit; remove K11 and the owner loop is optional; remove any of K12–K14 and a real design element loses its only justification (respectively: appeals; the dial's one-way glass; the ban on agreement-scoring). Nothing in the kernel is idle.

Closure. Part III derives the system's load-bearing commitments from Parts I alone. Where a downstream element needs a premise the kernel does not supply, Part III says so and the element is demoted to interface or hole. No derivation may cite a benchmark, a model name, or a dated capability result — that is the enforcement of capability-invariance.

Invariance. Every K-element remains true, chosen, or binding if models become arbitrarily capable. K7 and K10 deserve the explicit check, because they sound like capability claims: K7 asserts a relation (verifier reliability bounded by verifier competence on recompute-class checks, and co-movement of producer and evaluator frontiers), not a level; K10 asserts that independence, not skill, is what multiplies evidence. Arbitrary capability changes where the frontier band sits and how much agreement is worth — never whether the band exists or whether shared substrates share blind spots.

Part III — Derivations

Each derivation names its premises. These are the system's principles and structures as consequences. Where the historical philosophy document (P1–P11) asserted the same content on empirical grounds, the kernel replaces the perishable justification with the permanent one; the empirical literature then corroborates rather than carries.

T1. The construct: instruction-satisfaction plus evidence-grounding — from K0, K2, K3, K11, K12. Quality must be defined relative to inspectable referents, because K12 requires verdicts reconstructible by the bound party and K3 requires verdicts to carry their grounds; private impressions cannot be grounds anyone can reconstruct. Among inspectable referents, the instructions-plus-record is the only pair the owner controls (K1) and the only one whose meaning the owner can amend when K5 bites (K11). Defining quality as predicted-reviewer-consensus fails three ways at once: it caps validity at the reviewers' own reliability, it has no owner-amendable referent, and where K5-ambiguity exists it manufactures an answer where the inputs contain none — recording fiction, against K0. The construct is therefore not a preference among options; it is the only definition consistent with the kernel.

T2. Typed verdicts with a first-class underdetermine class — from K5, K0, K13. Since some items have no determinate answer from admissible inputs (K5), an evaluator that must always answer pass/fail is forced to fabricate on exactly those items, and under K13 the fabrication may not be laundered into reported consistency. So the verdict type system must include "the instructions underdetermine this case," it must never penalize the attempter (their work did not create the ambiguity), and it must route to the one party who can resolve it — the owner (K11), as an instruction amendment. Disagreement on such items is information about the instructions, which is precisely the K0 signal.

T3. Claims, not items, as the unit of verification — from K3, K6, K7, K12. A verdict must cite clause and span (K3, K12); clauses and spans attach to specific claims, not to gestalt impressions; and the checkability that K7 routing needs is a property of individual claims (this one is read-checkable, that one requires re-derivation), invisible at item grain. Decomposition is therefore constitutive of verdicts-with-grounds, not an accuracy tactic — with the corollary that item-level dispositions are transparent aggregations of claim-level findings, because that is the only way the grounds survive the roll-up. (How much decomposition, and at what granularity accuracy peaks, is an interface question — see Part IV.)

T4. The absence discipline — from K6, K13. "No errors," "fully addresses," "complete" quantify over open spaces. By K6, the evaluator can certify such claims only relative to an explicit enumeration; by K13 it may not report more than it checked. Hence: completeness criteria compile to enumerable checklists; absence concurrence is labeled "not-contradicted" rather than "verified" unless recall against planted instances has been measured on the relevant class; and praise of completeness is treated as the highest-risk claim type in the ontology — not because current models are weak, but because the logic of the claim is the hard direction.

T5. Routing by checkability, and the permanent frontier band — from K7, K10. Checks divide by whether the record decides them by reading or by re-derivation. On the second class the evaluator's verdict is bounded by its own competence (K7), and adding similar evaluators adds little (K10). Therefore: the compiled config must route by checkability class; the evaluator must be able to say "this exceeds what I can certify"; and there is a standing band of work — moving, never vanishing — where certification requires either genuinely independent competence (human expertise, tool-based recomputation) or honest abstention. Any roadmap that promises the band away is claiming K7 is false, not that models improved.

T6. Consequences gated by measured operating points — from K8, K12, K13, K15. K8(i) makes the false-accusation share an arithmetic function of prevalence and specificity; K8(ii) prices the knowledge of those rates in adjudicated samples; K12/K15 make false accusations the costliest error class. Therefore autonomous adverse action is legitimate only per lane, only after the lane's error rates are bounded on untouched adjudicated data at precommitted ceilings, and only with the human/appeal channel K12 and K15 require. Advisory-first is not caution; it is what the arithmetic licenses before the gold is bought. And the gold budget is a permanent price of stakes: whoever wants consequences funds the sample sizes that bound the relevant error, forever, at every requalification.

T7. Measurement/disposition separation — the dial — from K9, K13 (and K8(iii)). Suppose leniency pressure could alter what the evaluator perceives. Under K9 that channel becomes the cheapest way to hit throughput targets; under K13 the resulting numbers are knowingly false-as-stated; under K8(iii) even honest recalibration then chases a population the system itself bent. So the only coherent design is: perception fixed and stakes-sterile; all leniency implemented downstream as explicit thresholds over unchanged measurements; every batch stamped with its operating profile and corrected error estimate; bar changes prospective and announced. The dial's one-way glass is a theorem of the kernel, not a governance flourish.

T8. The adversarial posture — from K9, K10, K6. The evaluated population optimizes against any legible target (K9); content-level detection of that optimization is itself an evaluator subject to K7/K10 limits; and gaming that produces genuinely instruction-satisfying work is not gaming at all under T1. Therefore defenses are structural: unpredictable, indistinguishable seeded gold (the one probe K9 cannot price, because it is not legible); randomized deep audits; evidence-perturbation probes (a verdict that survives swapped evidence was never grounded, by K3); economics and provenance at the worker level; and a standing divergence sensor comparing scores on live work against held-out human-graded gold — the canonical Goodhart alarm, required because K9 guarantees the pressure it detects.

T9. The audit channel is constitutive, permanent, and human-anchored — from K11, K10, K7, K3. The construct is defined by owner-amendable instructions (T1); therefore the calibration referent must be adjudication against the instructions, produced by a process with authority to consult the owner (K11). No stream of evaluator outputs can substitute, because evaluators sharing substrates share errors (K10) and cannot certify their own frontier (K7). The channel never sunsets: not as humility, but because it is the thing that gives the numbers their meaning. Its outputs are themselves verdicts under K3 — so adjudicators are measured, blinded where anchoring threatens (see T10), and never scored on agreement (K14).

T10. The human lane's honest job description — from K11, K12, K15, K10, minus one interface question. What humans are for, kernel-derivably: settling meaning (K11), satisfying contestability and law (K12, K15), and supplying the independence that model ensembles cannot (K10). Whether escalated humans also out-judge the evaluator on the median routed item is not derivable — it is an empirical interface question, and the design must not depend on the answer. Hence evidence-only presentation (protecting the independence that is the human's kernel-derived value; an anchored human is a copy, K10), claim-level micro-questions (spending scarce independence where it changes decisions), and write-back into calibration only through adjudicated tiers (T9).

T11. Feedback as instrumented intervention — from K9, K14, K0, K3. Feedback specific enough to help is specific enough to train against (K9): the same 4-tuple that satisfies K3-grounds is a gradient. The kernel cannot remove this tension (it is K9 itself); it can only bound it: task-referenced always, person/rank-referenced never (K14); no stylistic guidance, because style convergence destroys K0's diversity signal; efficacy measured as an intervention with the divergence sensor (T8) standing guard. Feedback that cannot pass the system's own grounding gate does not ship — K3 reflexivity, again.

T12. The evaluator is enrolled in its own measurement model — from K3, K4, K10. Since evaluators are defined by their verdicts (K4) and verdicts carry measurable grounds (K3), the judge is one more rater: same reliability questions, same bias audits, same enrollment in whatever measurement model governs humans, with its verdict-consistency never accepted as validity (K10 — self-agreement is the cheapest correlated opinion of all). The ship gate is therefore a validity argument about an instrument, not a leaderboard: scoring evidence, generalization evidence, extrapolation evidence, and decision evidence, per lane.

T13. Version discipline everywhere — from K5, K11, K13. Meaning is settled by amendment (K11), amendments will recur indefinitely (K5), and bars may not move silently or retroactively (K13). Therefore instructions, compiled configs, evaluator versions, policies, and calibration profiles are all versioned, pinned, and replayable, with explicit retroactivity policy per amendment. The system's memory is part of its integrity, not an ops convenience.

Part IV — Interfaces

Declared slots where perishable, measured facts enter. The kernel defines the slot, the obligation, and who may fill it; measurement fills it. Literature never does — published numbers inform experiment design and priors, and are barred from being thresholds (that bar is K13: reporting a borrowed number as if it measured this system is reporting a number known to be biased-as-stated).

• I1. Lane boundaries. Which criterion families are deterministic / evaluator-groundable / evaluator-assisted / human-only, per project, per evaluator generation. Filled by: the ship gate's per-lane comparisons on adjudicated data. Re-filled at every model generation, both directions open (T5 forbids promising the band away; nothing forbids the band moving favorably).
• I2. Thresholds and budgets. Confidence cutoffs, escalation rates, k-samples, verification depth, audit rates, gold-set sizes. Filled by: operating-point curves plus K8 arithmetic at the project's error prices. The acceptance-sampling tradition (switching rules, outgoing-quality guarantees) is the standing template — a solved discipline, imported rather than reinvented.
• I3. Decomposition granularity and exemplar counts. T3 fixes that decomposition exists; the accuracy-optimal grain and the exemplar curve are measured per project.
• I4. The entailment standard's false-flag price. "Reasonable-expert support" is an operationalization (Part V, H2); its cost is a measured number the owner accepts or tightens, per lane.
• I5. The complementarity question. Whether routed humans outperform the evaluator anywhere beyond their kernel-derived roles (T10), and on what slice. An open empirical question the design is built not to need answered in advance.
• I6. The moat question. Whether warranted-praise and planted-defect recall are strong at the current generation — the single interface measurement that moves the most decisions (enforcement timeline, differentiation claim, lane map), in either direction.

The interface rule, stated once: capability is rented, walls are owned. Every I-slot is re-measured on the deployment-tier models at each generation; no I-value is ever promoted into Part I–III.

Part V — Holes

Where the kernel is honestly silent. Each hole is an operationalization obligation, not an embarrassment; pretending these are settled is how foundations rot.

H1. Aggregation. No principled function maps claim verdicts to item dispositions; any formula re-imports judgment at the aggregation layer. The kernel requires only that the formula be explicit, versioned, owner-signed, and error-tolerant under K8's claim-level error reality (T3 corollary). Which formula, per project, is a governed choice.

H2. The entailment standard. Between strict entailment (fails competent prose) and loose support (readmits the variance the system removes) there is no theoretically privileged point. The standard is chosen, its false-flag rate measured (I4), and its choice owned. What the kernel forbids is presenting any point as derived.

H3. Construct authority in the breach. When the owner's case-intuitions contradict their compiled rubric, rubric-as-law-with-forced-amendment versus standing override is a governance choice. The kernel constrains it (K13: no silent motion; K11: the owner may amend; T13: versioned either way) but does not decide it.

H4. The telos boundary. K0 takes the project's instructions as the given expression of what the data should capture. Whether the instructions themselves are good — whether the project should exist, whether the buyer's construct is the right one — is outside this kernel entirely. The system inherits the legitimacy of its projects; it cannot manufacture it.

H5. Moral status of the dial's floor. K13 and T7 make leniency explicit and priced; the kernel does not say how much residual error is acceptable to sell at any price. That is a commercial-ethical judgment for the owner and buyer, made with open eyes because the number is stamped on the batch — which is all a measurement layer can force.

Coda — why this kernel survives model progress

Nothing in Parts I–III mentions a model, a benchmark, or a year. If evaluator capability doubles, what changes is the location of every boundary the kernel declares — the frontier band moves (K7), lanes re-derive (I1), thresholds re-fit (I2), the moat question re-answers (I6) — and what does not change is a single sentence of the kernel itself. That is the design intent: a system whose architecture is aligned with the walls, whose contact with capability is confined to declared interfaces, and whose values are stated as choices signed by their owners rather than theorems no one proved.

The kernel is amendable (it is itself governed by T13). What it is not, is silently amendable.