Working brief - 2026-07-15
Summary. A one-project pilot that measures whether a model-based system can check human review work against written instructions and evidence, reliably enough to matter. Fund the adjudicated gold data that measurement requires. No enforcement authority - no automated rejections, pay decisions, or account actions. That authority has to be earned per lane with error rates we do not yet have, and in part it is reserved to humans by law.
How to read this. Every factual claim on these pages carries a bracketed reference. Clicking it opens the evidence ledger entry: the source, what we could verify against it this week, a verbatim quote, and a grade saying whether the finding still holds under the models we would actually deploy. Nothing here asks to be taken on trust, including our own summary sentences.
The reason this proposal exists: human reviews of human-labeled data are not reaching consensus, and nobody can currently say why, or what that costs, on any given item. Non-consensus is not one problem. It has four distinct causes - instructions that do not decide the case, reviewer error, missing expertise, and legitimate difference of judgment - and each needs a different fix. Decades of rater research add a hard constraint: the largest share of the variance is not the stable, trainable kind; it lives in the interaction of a particular reviewer with a particular item on a particular criterion, which training and calibration meetings measurably do not remove %s %s. The only lever that reaches it is checking each specific claim on each specific item against the evidence. A model can do that at scale for the claim classes whose evidence can be pinned down - quoted spans, and factual events queryable from a structured record; how much of real review writeups falls in those classes, versus claims about whole long trajectories that no current model reliably verifies, is one of the first things the pilot measures rather than assumes.
One warning shapes the whole design. The obvious fix - drive reviewers toward agreement - treats all four causes by erasure, and it is the one cure that kills the patient: it caps the system's accuracy at the reviewers' own reliability, manufactures agreement where the honest record shows a defensible split, and teaches a workforce whose judgment is the product to imitate a machine %s. So this system never targets consensus directly. It explains disagreement - fixes the instruction ambiguity at the source, catches the errors, routes the expertise gaps, and records the genuine splits as signal. Consensus in application then rises as a byproduct of removing its illegitimate causes, which is the version of consensus that is actually worth buying.
""" % (R("CG-02"), R("CG-04"), R("CG-16"))) at("index.html", "proposal", "Overview - the proposal") parts.append("""A system that reads a reviewer's writeup, breaks it into individual claims, and checks each claim against the project's instructions and the evidence the reviewer had. It returns one of four answers per item: pass, fail, cannot verify, or - critically - "the instructions do not decide this case," which goes back to the project owner as a fix to the instructions rather than a mark against the worker. Every verdict cites the instruction clause and the evidence span it rests on, so a worker can check it, contest it, and win when it is wrong.
Humans stay in three places, on purpose: they own what the instructions mean, they take every decision that significantly harms a worker (our choice everywhere; the law's requirement for EU workers %s), and they audit the system permanently through blind gold items it cannot distinguish from live work. The complete design, with the reasoning behind each part, is in the system description and the kernel.
""" % R("CG-07")) at("index.html", "evidence", "Overview - what the evidence supports") parts.append("""We reviewed 132 findings across the research corpus, then re-verified every one of them against its primary source this week: 82 confirmed, 43 confirmed with caveats we list per claim, 7 unreachable behind paywalls, and none contradicted. Then we asked a harder question than "is it true": is it still decision-grade under the models we would actually deploy? That second question matters because most published numbers in this field were measured on systems one or two generations old.
The honest position: the case for the architecture is strong and durable - it rests on law, statistics, and forty years of rater science that do not move when models improve. The case for any specific performance level is not: no published study tests current frontier models on this exact task, in either direction. Yesterday's weak results are not today's ceiling, and yesterday's strong results are not today's promise. So this proposal leans only on the durable evidence, and treats every capability number as a hypothesis the pilot itself must measure %s. Three examples of what that discipline looks like:
One structural point deserves a sentence of its own. The layers of this system that answer the company's actual problem - measuring the reviewer baseline, decomposing disagreement into its causes, and repairing ambiguous instructions at the source - do not depend on long-record verification at all. The capability limits that remain genuinely open are confined to one lane: machine verification of claims about whole long trajectories. If that lane measures badly, it stays assisted or human; the rest of the system's value does not move.
The full picture, organized by how much weight each finding can bear, is in the research review.
""" % (R("AJ-03"), R("AJ-04"), R("F26-06"), R("GF-07"), G("gap-01"))) at("index.html", "decision", "Overview - five items") parts.append("""Each ask, with its options, evidence, reversibility, and the specific result that would change the answer, is one section of the decision register. That page is the working surface; this one is the summary.
No universal judge. No automated rejections on subjective criteria. No claim that praise verification works until our own benchmark says so. No removal of expert audit cost, ever. And no pretense that a system this new can be evaluated by reading papers: the pilot exists because the decisive numbers do not exist yet.
""" % (R("AJ-02"), R("HS-01"), R("CG-02"), R("CG-07"), R("CG-11"), R("CT-03"))) write_page("index.html", "Overview", "The AutoQA decision brief: the summary, the problem, the proposal, and what the evidence supports.", "home", "".join(parts)) def build_brief(): at("brief.html", "top", "Summary - opening") parts = [] parts.append("""The whole case, shortest honest version - 2026-07-15
The problem. Human reviews of human-labeled data are not reaching consensus, and that non-consensus is unexplained, unpriced variance in the product we sell. It has four causes - ambiguous instructions, error, missing expertise, legitimate difference - and the dominant variance component is one that training provably does not remove; it can only be attacked by checking claims against evidence, item by item - at scale for pinnable claims, with the trajectory-global remainder sized by measurement and routed honestly. Forcing agreement would erase all four causes indiscriminately and consume the judgment signal the data exists to capture; this design explains disagreement instead, and lets real consensus emerge from fixing its illegitimate causes.
""") at("brief.html", "measurement", "Summary - measurement point") parts.append("""Why measurement discipline is the core. In the largest judge study to date, raw agreement made systems look 33 to 41 points better than chance-corrected agreement showed them to be %s. Numbers without their correction sell; corrected numbers decide. Everything in this program reports the corrected kind, and every number on these pages links to its source and to what we verified about it.
""" % R("AJ-03")) at("brief.html", "hybrid", "Summary - human role") parts.append("""The human role. People stay where they add what machines cannot: owning what instructions mean, taking consequential decisions, and providing independent audit. The one careful study of assisted review found people help most when shown the evidence and not the machine's verdict - anchoring is real %s. We design for that, and we do not assume humans out-judge the system on routine escalations; that gets measured too.
""" % R("CM-03")) at("brief.html", "moat", "Summary - the unmeasured capability") parts.append("""The honest unknown. Whether a model can verify that praise is warranted - "complete," "well grounded," "appropriately calibrated" - has never been measured by anyone %s. It is the capability this product most depends on. A seeded two-week test answers it under the models we would actually deploy, before any serious commitment.
Summary. One measurement pilot, one project, gold data funded, no enforcement authority. Four outcomes are acceptable: stop, narrow, assist-only, or selective automation earned lane by lane. A credible "stop" is a success of the process - it would mean we learned the truth cheaply.
The decision surface with evidence attached is the decision register. The claim-by-claim receipts are the evidence ledger.
""" % G("gap-01")) write_page("brief.html", "Summary", "Three-minute summary of the AutoQA proposal and ask.", "summary", "".join(parts)) def build_research(): parts = [] at("research.html", "top", "Research - how to read") parts.append("""132 findings reviewed and re-verified against primary sources - 2026-07-15
Two cautions frame everything below. First, almost every published result grades model outputs; this program grades human writeups about model outputs. The architecture lessons transfer; the accuracy numbers do not transfer automatically. Second, most numbers were measured on models one or two generations behind what we would deploy, and no published study tests the current tier on this task at all. So the findings are grouped below by the only question that matters for a decision: how much weight can this carry today?
""") at("research.html", "takeaways", "Research - what survives review") parts.append("""Check claims, not overall impressions. Give the checker the same evidence the worker had - access beats model size. Use humans selectively, blind to the machine's opinion. Ground rubrics in expert examples, because instructions alone compile badly. Treat calibration as permanent, not a launch task. And treat positive judgment - verifying that required things are present and praised things are real - as the hard, unmeasured frontier. Those six conclusions held up across independent research lines and our own source-by-source re-verification; nothing we checked contradicted them.
""") at("research.html", "research-cards", "Research - findings by decision weight") parts.append("""Law sets a floor. For EU platform workers, decisions of significant detriment must be taken by a human, with explanation and review rights - from December 2026, with no consent exception %s %s. US-side, automated employment decision tools carry bias-audit and notice obligations in enacted jurisdictions, and selection procedures with adverse impact carry validation duties %s %s. This is not compliance trim; it fixes the human lane's minimum job description.
The statistics do not bend. At realistic failure rates, even an excellent checker produces a meaningful share of false accusations - at 3 percent prevalence and 99 percent specificity, roughly one accusation in four is wrong - and demonstrating better requires adjudicated samples that scale with the rarity of the error %s. Separately, there is a proven ceiling: when a judge is no more accurate than what it judges, nothing cuts the need for ground truth by more than half %s. Audit cost is structural.
Rater science explains our problem. Training makes reviewers more self-consistent but does not equalize their severity %s, and variance decompositions show the reviewer-item-criterion interaction dominating stable harshness %s. This is the forty-year-old result the whole proposal stands on: only per-item verification touches the dominant variance.
Incentives are physics here. A workforce paid against a legible criterion optimizes the criterion; the sharper the feedback, the better the map it provides for gaming %s. An autonomous agent already passes 99.8 percent of standard quality checks at around five cents a task, so content-detection is a lost race; defenses are economic and procedural %s. And rewarding agreement with consensus provably teaches your best dissenters to conform %s.
Agreement can flatter. Across 21 judges and roughly 541,000 judgments, raw agreement exceeded chance-corrected agreement by 33 to 41 points, and judge rankings reshuffled by up to 14 positions across benchmarks %s. Measured in the 2026 window; the reporting discipline it forces is permanent.
Structured beats holistic. Criterion-level checking with explicit reasoning outperformed flat checklist grading by 7 to 12 points in the 2026 benchmark closest to our setting %s %s - though the same study found frontier models still near chance on its hardest criteria, which is why lanes and escalation exist.
Confirming is easier than catching. Grounding judges scored about 85 F1 confirming supported claims but about 46 F1 catching unsupported ones, and vague answers gamed the metric until an eligibility gate was added %s. Flash-tier graders; the asymmetry's direction is what we bank on, not its size.
Architecture beat model choice in production. The one vendor evaluation on expert-graded human work found that giving the verifier the worker's environment mattered more than which model verified, with F1 around 0.63 to 0.66 on artifact-heavy tasks against practicing experts %s %s.
Feedback moves people. In a randomized deployment across more than 20,000 peer reviews, 27 percent of reviewers revised after model feedback that had passed reliability gates %s. Peer reviewers are not paid annotators; we treat this as a direction with a measurement plan, not a forecast.
Long records defeat single-pass reading. Judges are measurably unstable on long-form work %s, and checking a citation against only its cited span misses the wrong-citation failures that need record-wide search %s. This is why the design separates evidence discovery from judgment, records coverage with every verdict, and measures discovery recall with planted evidence rather than assuming the model read everything.
These findings shaped the design but their magnitudes are one to two model generations old: the human-plus-AI review lift on already-passed items (24 percent versus 6 percent, GPT-4 era) %s; the omission-detection gap (a 57-point F1 drop on deletions, mid-2025 flash tier) %s; expert-domain agreement in the 60s (2024) %s; and the entailment-checker ceiling near 77 balanced accuracy (2024-25 models) %s. Each mechanism is plausible under current models; none of these numbers is allowed to set a current threshold in either direction. The pilot re-measures every one of them on the deployment tier before anything binds.
""" % (R("CG-07"), R("CG-08"), R("DA-01"), R("DA-02"), R("HS-03"), R("CT-03"), R("CG-02"), R("CG-04"), R("DA-03"), R("AQ-02"), R("CG-16"), R("AJ-03"), R("AJ-04"), R("F26-06"), R("GF-07"), R("F26-02"), R("IP-02"), R("CM-05"), R("AJ-12"), R("GF-09"), R("CM-02"), R("CM-06"), R("AJ-11"), R("GF-01"))) at("research.html", "robustness", "Research - robustness check") parts.append("""A robustness check on the whole case. Suppose you struck every LLM experimental result measured on pre-2025 models - sixteen findings gone outright, seven more reduced to their current-model halves. Nine of the eleven decisions in the register are untouched; the other two lose one illustration each and still stand on law, mathematics, or current-generation results. The kernel loses nothing, because it was built on the sixty-three findings that are law, statistics, or human science. The proposal does not depend on old model results in either direction - which is the point of grading evidence for currency instead of merely verifying it. One deliberate exception: old adversarial results are kept as floors, since a successful 2024 attack still bounds today's stronger attackers from below.
Seven questions have no published answer anywhere in the reviewed corpus, and they are exactly the ones the pilot is sequenced to answer: whether praise verification works at usable precision %s; whether any published judge result transfers from model outputs to human writeups %s; the rate at which judge and worker are wrong together %s; how workers adapt once a judge is deployed %s; whether feedback durably changes paid annotators %s; which confidence signal best predicts when a human would overturn the machine %s; and our own reviewers' baseline reliability %s. Treating these as open is not a weakness of the proposal - it is the proposal. Each is assigned to a specific experiment in the decision register.
The complete corpus - every domain report, disagreement, and verification transcript - is in the library, and every claim above can be audited in the evidence ledger.
""" % (G("gap-01"), G("gap-02"), G("gap-03"), G("gap-04"), G("gap-05"), G("gap-06"), G("gap-07"))) write_page("research.html", "Research", "What the research supports, organized by how much weight each finding can bear under current models.", "research", "".join(parts))