Raw Questions by Lens

Raw questions by lens

Lens: measurement

What is the actual ground-truth construct we're measuring - 'would an expert consensus panel pass this item?' or 'does this item satisfy the written project instructions?' - and when those two diverge, which one wins?

WHY: Human reviewers already disagree, so 'agreement with human reviewers' is a moving target. If the construct is instruction-satisfaction, the AutoQA can be MORE valid than the humans it's checked against; if it's human-consensus prediction, validity is capped at the humans' own reliability ceiling. GATES: Whether gold labels come from adjudicated expert panels vs. instruction-derived rubric checks; whether disagreement-with-humans is treated as AutoQA error or human error during meta-evaluation.

Given that human reviewers show unexplained variance, what is the measured inter-reviewer reliability floor (e.g., Krippendorff's alpha on a common item set), and is it even high enough for human agreement to serve as a validation target at all?

WHY: If human-human alpha is, say, 0.35, then judge-human agreement of 0.4 is indistinguishable from perfect performance and 0.7 is impossible - every downstream accuracy claim is bounded by this number. Without measuring it first, we can't tell whether the AutoQA is failing or the target is noise. GATES: Whether meta-evaluation uses raw human labels, adjudicated labels, or noise-corrected agreement statistics; the sample sizes and acceptance thresholds for the AutoQA's own validation.

At what granularity of claim decomposition (whole-review pass/fail vs. per-criterion verdict vs. per-atomic-claim groundedness check) does judge reliability stop improving - and does aggregation of atomic verdicts back to an item-level decision introduce its own error mode?

WHY: The design requires evaluating positive statements ('is this claim grounded in cited evidence?'), which implies decomposition. But decomposition trades one subjective judgment for N smaller ones plus an aggregation rule; if the aggregation rule (any-fail, weighted, threshold) is itself unvalidated, we've moved the subjectivity, not removed it. GATES: The core unit of analysis for the AutoQA verdict schema and the aggregation function from claim-level checks to item-level pass/fail/feedback.

Which failure types are objectively verifiable (fabricated citation, claim contradicted by evidence, instruction violation with a bright line) vs. irreducibly judgment-laden (severity ratings, 'tone', borderline quality), and should the AutoQA emit categorically different output types (verdict vs. flag-for-human vs. confidence-weighted opinion) for each?

WHY: Treating all checks as one verdict type forces a single reliability standard on axes with wildly different measurability. The one-human-interaction budget should be spent exclusively on the judgment-laden residue, which requires knowing where the objective/subjective boundary actually sits per project. GATES: The output taxonomy (auto-decide vs. escalate), and how the per-project customization layer classifies each rubric axis at onboarding.

Are the AutoQA's confidence scores actually calibrated per-project and per-axis (does 80% confidence mean 80% agreement with adjudicated truth?), and what recalibration mechanism runs when a new project's instruction set shifts the distribution?

WHY: The escalation policy (which items consume the one human interaction) is only as good as the calibration behind it. LLM judges are notoriously miscalibrated and calibration does not transfer across domains; an uncalibrated escalation threshold silently converts into either rubber-stamping or full human re-review. GATES: Whether to build a per-project calibration/threshold-fitting step into onboarding, what the minimum labeled-item count for it is, and whether raw model confidence is ever exposed to humans.

What is the minimum viable gold set per project (size, sampling design, refresh rate), given base rates - if only 5-10% of attempts are genuinely bad, how many adjudicated items do we need before we can distinguish a 90%-precision judge from a 70% one?

WHY: Low base rates of spam/falsehood mean precision/recall estimates have huge confidence intervals at plausible gold-set sizes; a 200-item gold set with 15 true failures cannot certify the system. Sample-size arithmetic decides whether per-project validation is feasible or whether we must pool evidence across projects. GATES: Gold-set construction protocol, whether validation is per-project or hierarchical/pooled, and the honest error bars we can put on any launch claim.

How do we detect drift in three distinct moving parts - attempter behavior (including adversarial adaptation to the QA), project instruction revisions, and judge-model updates - and which statistic (verdict-rate shift, escalation-rate shift, gold-set score drop) triggers re-validation for each?

WHY: A static validation certificate decays: attempters learn what the AutoQA checks and game it, instructions get amended mid-project, and the underlying model gets swapped. Each drift source needs a different sensor and a different response; conflating them means either constant false alarms or silent decay. GATES: The monitoring dashboard's core metrics, the re-validation trigger policy, and whether gold items must be continuously injected into the live stream (blind gold) vs. periodically batch-audited.

For groundedness checks on positive statements, what counts as the evidence closure - only sources the attempter explicitly cited, the full item context, or external knowledge the judge happens to have - and how do we stop the judge from 'verifying' claims against its own priors rather than the record?

WHY: The stated goal is 'is this claim grounded in the evidence they cite,' which is a verifiability judgment relative to a defined corpus. If the evidence boundary is undefined, the judge will substitute its parametric knowledge, producing verdicts that are confident, unauditable, and wrong in exactly the expertise-gap cases the system exists to catch. GATES: The judge's context-construction rules (what it may and may not see), and whether groundedness verdicts must quote span-level evidence to be admissible.

How do we meta-evaluate the constructive-feedback channel - by downstream attempter improvement (revision acceptance rate, repeat-error rate) rather than feedback 'quality' ratings - and are we willing to run the controlled comparison (feedback vs. verdict-only) to find out if feedback does anything?

WHY: Feedback quality has no natural gold label and pairwise 'this feedback seems good' ratings inherit all the subjectivity problems we started with. The only defensible criterion is behavioral: does the attempter's next submission improve? If we can't or won't measure that, the feedback component is unfalsifiable decoration. GATES: Whether feedback generation gets its own outcome-based evaluation loop with attempter-level longitudinal tracking, and what data the platform must log to enable it.

What is the deployment decision rule that translates measured judge performance into routing - at what precision/recall on which failure class does the AutoQA get to auto-fail an item without a human, auto-pass one, or must escalate - and who bears the asymmetric costs of each error?

WHY: False auto-fails burn attempter trust and pay; false auto-passes poison training data downstream. These costs are asymmetric and project-specific, so a single accuracy threshold is wrong by construction; the routing policy must be an explicit cost-weighted decision rule, not a vibe. GATES: The three-way routing policy (auto-pass / auto-fail / escalate), its per-project cost parameters, and which decisions are reversible via appeal.

Contrarian: is inter-reviewer disagreement actually the disease, or is some of it legitimate signal about genuinely ambiguous items and underspecified instructions - and would an AutoQA that enforces consistency be laundering one arbitrary interpretation into a false appearance of objectivity?

WHY: High agreement is trivially achievable by collapsing to a crude, consistent-but-invalid rule (reliability without validity). If disagreement concentrates on items where the instructions are genuinely underdetermined, the right output is 'flag instruction gap for the project owner,' not a confident verdict - and the AutoQA becomes an instrument for measuring instruction quality, not just attempter quality. GATES: Whether the system includes an 'instructions underdetermine this case' verdict class and a feedback loop to project owners, vs. always forcing a pass/fail.

When the AutoQA and the single human interaction disagree on an item, what is the adjudication and logging protocol - does the human always win, and how do we prevent the human-override stream from silently becoming the new (unvalidated) training/calibration signal?

WHY: The one-human-per-item budget means most overrides come from a single reviewer - exactly the noisy, poorly-aligned population we started with. If overrides automatically count as ground truth for recalibration, we recycle human inconsistency back into the judge; if they don't, we have no learning loop. GATES: The override-handling policy, whether disagreement items get periodic second-opinion audits (spending extra human budget on a sampled subset), and what feeds the calibration/fine-tuning data.

Which agreement statistic and evaluation design do we standardize on given that pass rates are extreme and vary by project - and do we explicitly ban raw percent-agreement and single-number 'accuracy vs. humans' from all reporting?

WHY: At a 92% pass rate, a judge that passes everything scores 92% agreement and kappa behaves pathologically (the base-rate paradox); different per-project prevalences make cross-project comparisons of any single statistic meaningless. Choosing the metric contract up front prevents every later performance claim from being an artifact. GATES: The standard meta-evaluation report format: chance-corrected statistic choice (e.g., Krippendorff's alpha + per-class recall at fixed prevalence), stratified-by-failure-class reporting, and minimum CI-width requirements before any threshold change ships.

Lens: judge-failure-modes

What is the measured run-to-run verdict flip rate for our judge on identical items at fixed temperature, and does the AutoQA architecture treat that number as noise to average away (k-sample voting) or as signal that the criterion itself is underspecified?

WHY: If flip rate on a criterion exceeds the human reviewer disagreement rate we are trying to fix, the judge adds variance rather than removing it. Distinguishing stochastic noise from criterion ambiguity determines whether the fix is ensembling or rubric rewriting - two completely different builds. GATES: Whether to spend the inference budget on k-run self-consistency voting versus a rubric-decomposition pipeline; also which criteria get routed to the single allowed human interaction.

When the judge verifies a POSITIVE attempter claim ("this statement is grounded in the cited evidence"), does it exhibit an agreement-default bias - passing plausible-sounding but ungrounded claims at a higher rate than it fails grounded ones - and how large is that asymmetry on a seeded test set?

WHY: Verification of positive statements is the core novel requirement here, and sycophancy/plausibility bias runs exactly against it: judges are known to rubber-stamp fluent assertions. If the false-pass rate on seeded ungrounded claims is high, the architecture needs adversarial framing (e.g., 'find the flaw in this claim' prompts, or claim-evidence entailment decomposition) rather than holistic pass/fail judging. GATES: Whether positive-claim verification is built as holistic judgment, as a decomposed entailment check per cited-evidence pair, or as a debate/red-team prompt structure.

Does the judge systematically favor verbose, hedged, or rubric-vocabulary-mimicking attempter writeups over terse-but-correct ones, and by how many points of pass-rate delta on length/style-controlled pairs?

WHY: Verbosity and style bias would train attempters (who adapt fast to QA signals) to pad and mimic rather than improve substance - the AutoQA would actively corrupt the annotation pool it polices. A measured delta tells us whether we need style-normalization (summarize-then-judge, content extraction) before judgment. GATES: Whether to insert a canonicalization/extraction layer that strips style before the judge sees the annotation, versus judging raw attempter text.

When the attempter's annotation critiques an AI output produced by the same model family as our judge, does self-preference bias suppress the judge's agreement with valid criticisms - and does routing those items to a different model family close the gap?

WHY: The annotated content is AI output; attempters saying 'this AI response is wrong' will be second-guessed by a judge that shares the AI's blind spots or prefers its own family's outputs. This is a structural conflict of interest unique to QA-of-AI-critique settings. GATES: Model-choice policy: whether the judge must be a different family than the generator of the content under annotation, and whether that requires per-project judge routing.

For each evaluation-axis type (factual grounding, instruction compliance, tone/style quality, holistic ratings), what is the judge-vs-adjudicated-human agreement ceiling - and on which axis types does the judge fall below the human-reviewer inter-rater agreement we already consider broken?

WHY: Root cause #1 is subjective interpretation of criteria. If the judge is no better than humans on subjective axes (likely), the design must partition axes: judge-autonomous, judge-assisted-human, and human-only lanes. Pretending one judge handles all axes reproduces the exact variance problem. GATES: The routing taxonomy: which criterion types the AutoQA decides alone, which consume the one human interaction, and which it must abstain on.

At what rubric decomposition granularity does judge-human agreement stop improving - and is there a granularity beyond which agreement degrades because atomized checks lose context (the judge passes every micro-item while missing the writeup being globally wrong)?

WHY: The reflexive fix for judge unreliability is finer rubrics, but decomposition has a measurable optimum and a failure mode (checklist myopia). The project-agnostic foundation must specify how project instructions get compiled into judge criteria, and that compiler needs a target granularity. GATES: The design of the per-project customization layer: how instruction sets are decomposed into judge prompts, and whether a holistic synthesis pass sits on top of atomic checks.

How sensitive are verdicts to semantically-neutral perturbations of the judge prompt (criterion ordering, pass/fail label ordering, few-shot exemplar choice, evidence presentation order), and do we require paraphrase-invariance testing as a gate before any project's judge config ships?

WHY: Position and prompt-sensitivity effects can dwarf real quality signal. Since every project customizes the prompt, an unstable base template means each project's customization silently re-rolls the judge's biases - variance across projects that looks exactly like the human variance we set out to kill. GATES: Whether the foundation includes a mandatory perturbation-robustness test harness and randomized-order/position-debiasing defaults in the base template, versus trusting per-project prompt authors.

What calibration signal does the judge emit per verdict (stated confidence, sample agreement rate, logit margin), and is any of them actually predictive of judge error well enough to drive the route-to-human decision at a fixed one-touch budget?

WHY: The one-human-interaction constraint makes triage the load-bearing mechanism. If judge confidence is uncalibrated (common - stated confidence often near-useless), the human touch gets spent on the wrong items and the system's error rate is set by the worst unrouted verdicts. GATES: The escalation mechanism design: self-reported confidence vs. k-sample disagreement vs. dual-judge disagreement as the routing trigger, and what abstention thresholds to set.

Which shortcut heuristics does the judge use as pass/fail proxies (presence of citations regardless of relevance, rubric keyword echo, annotation length, score extremity), and can attempters who see the feedback reverse-engineer them within a few submission cycles?

WHY: The AutoQA emits constructive feedback to attempters, which means it publishes a gradient toward its own shortcuts. Goodharting risk is not hypothetical: attempters are paid on pass rates. The defense (rotating probes, held-out seeded items, feedback that explains without revealing decision boundaries) has to be designed in from the start. GATES: Whether the feedback channel exposes full judge reasoning or a filtered version, and whether the foundation mandates ongoing seeded-item probes to detect gaming.

Is the judge's constructive feedback held to the same grounding standard as the attempter work it critiques - i.e., what fraction of judge-generated feedback statements are themselves hallucinated or ungrounded, and do we verify feedback before delivering it?

WHY: A judge confidently telling an attempter to fix a non-existent problem destroys trust faster than a wrong pass/fail verdict, and hallucinated critique is a known failure mode of critic models. If feedback needs its own verification pass, that roughly doubles the pipeline and changes cost and latency assumptions. GATES: Whether the architecture includes a feedback-verification stage (judge-of-judge or evidence-linking requirement on every feedback claim) before feedback reaches humans.

Does a multi-model ensemble actually buy error reduction on the failure modes that matter here, or do frontier judges share correlated blind spots on subjective criteria such that the ensemble converges confidently on the same wrong answer?

WHY: Ensembling is the default reliability move, but it only helps against decorrelated errors. If disagreement between model families predicts item difficulty better than it predicts judge error, the right design uses cross-family disagreement as an escalation signal, not a voting mechanism - a very different architecture at very different cost. GATES: Ensemble-vs-single-judge-plus-router architecture, and the per-item inference budget.

Contrarian: on the subjective criteria where human reviewers disagree most, is 'judge bias' even distinguishable from legitimate criterion ambiguity - and would the honest system output 'this criterion is underspecified, here are the two defensible readings' back to the project owner rather than a verdict?

WHY: Our lens assumes judge failure modes are the problem to fix, but root cause #1 is subjective interpretation - if the ground truth is genuinely contested, a perfectly debiased judge still can't be 'right,' only consistently arbitrary. Consistent-but-arbitrary may actually be acceptable (it fixes variance) - or it may launder ambiguity into false authority. This choice defines what the product is. GATES: Whether the foundation includes an instruction-ambiguity feedback loop to project owners as a first-class output, and whether the target metric is agreement-with-truth or consistency-of-application.

What is the judge's blind-spot profile on expertise-heavy content - specifically, at what domain-difficulty level does the judge's ability to verify attempter claims fall below the attempter's own reliability, making AutoQA on those items net-negative?

WHY: Root cause #2 is expertise gaps, and the judge has its own. A judge that confidently fails correct expert annotations (or passes confident expert-sounding errors) is worse than no QA on that slice. Knowing where the competence boundary sits per domain determines the abstention policy. GATES: Whether the foundation requires per-project judge-competence calibration (seeded expert items) before enabling autonomous verdicts, and the abstain-vs-verdict boundary.

Do pass/fail verdicts and diagnostic reasons drift when the underlying judge model is upgraded, and does the foundation pin model versions per project with a regression suite, or accept drift as the price of improvement?

WHY: The system's whole pitch is reducing unexplainable variance across reviewers. A silent model upgrade that shifts pass rates 10 points reintroduces exactly that variance, now across time instead of across people, and invalidates attempter feedback histories. GATES: Versioning and regression-testing policy in the foundation: frozen judge versions + golden-set gates on upgrade, versus continuous latest-model usage.

Lens: rubric-ops

At what decomposition granularity do rubric items stop improving judge-human-expert agreement - and does over-decomposition actively degrade it by destroying holistic judgments (e.g., 'this critique is fair overall') that only exist at the composite level?

WHY: There is a real possibility that agreement vs. granularity is an inverted-U: too coarse reproduces the subjective-interpretation problem, too fine produces checklists that pass while the annotation is globally wrong. Without knowing where the curve peaks per criterion type, we can't set a decomposition policy. GATES: The rubric compiler's target granularity (atomic checkable claims vs. mid-level judgment clusters) and whether the schema allows/requires holistic roll-up items alongside atomic ones.

Which evaluation-axis types are compilable into evidence-linked binary claims at all, and which are irreducibly gradient/contested - and can we build a reliable classifier (human or model) that sorts a project's axes into 'compile', 'anchor-with-exemplars', and 'route-to-human' buckets before the project goes live?

WHY: The whole architecture forks on this taxonomy: compilable axes get automated verdicts, exemplar-anchored axes get calibrated model judgment, contested axes consume the one-human-touch budget. If we can't sort axes reliably up front, every project's failure mode is silent misrouting. GATES: The ingestion pipeline's axis-triage stage and the routing policy for the single allowed human interaction.

What is the minimal set of artifacts a project instruction set must be compiled into - checkable claims, positive/negative exemplars, counter-exemplars, edge-case adjudications, forbidden-inference rules - such that a fresh judge (model or human) reaches target agreement with the project owner without any conversation with them?

WHY: This defines the 'rubric compilation contract.' If exemplars are load-bearing (likely, given that criteria text alone is what humans already disagree on), then a text-only compiler is doomed and ingestion must extract or elicit exemplars, which changes the whole intake workflow. GATES: The schema of the compiled AutoQA config and the required inputs at project onboarding (can we ingest instructions-as-written, or must we demand adjudicated examples?).

Who is the authority when the compiled rubric and the project owner's case-by-case intuitions diverge - and do we force the rubric to be amended (rubric-as-law) or allow standing owner overrides (rubric-as-guidance), given that override accumulation silently recreates the unexplainable-variance problem we're solving?

WHY: Every rubric system dies one of two deaths: legalism (rubric ossifies, owners route around it) or override rot (rubric becomes fiction). The governance choice determines whether AutoQA verdicts are appealable, who can amend, and what an amendment invalidates retroactively. GATES: The amendment/override protocol, verdict finality semantics, and whether past verdicts are re-scored when a rubric changes.

How do we detect criteria drift operationally - what specific signal (judge-human disagreement trending on one axis, override cluster density, exemplar staleness score, attempter-appeal rate) triggers a rubric revision, and at what threshold - versus drift in the item distribution or the attempter population masquerading as rubric drift?

WHY: Without a falsifiable drift trigger, rubrics are either revised on vibes (reintroducing subjectivity) or never (ossification). Confusing population drift with criteria drift leads to 'fixing' a rubric that was fine, which churns all downstream calibration. GATES: Which telemetry the system must log per verdict from day one, and the design of the drift-detection monitor as a first-class component vs. a later add-on.

For verifying an attempter's POSITIVE claims ('the response's argument in section 2 is sound'), what claim-typing does the compiler need - grounding claims (evidence exists), entailment claims (evidence supports the stated conclusion), and calibration claims (stated severity/rating matches evidence strength) - and do these three require different verification machinery with different reliability ceilings?

WHY: Positive-statement verification is the stated differentiator, and 'is the opinion consistent with cited evidence' (calibration) is plausibly much harder than 'does the cited evidence exist' (grounding). If reliability ceilings differ sharply, verdict confidence must be reported per claim type, and the human touch should target the weakest type. GATES: The claim ontology in the compiled rubric schema and per-claim-type confidence gating on the human-escalation router.

What is the empirical false-agreement rate - cases where the AutoQA judge and the human attempter agree with each other but both are wrong per expert ground truth - and does rubric decomposition reduce it or merely make shared errors better-documented?

WHY: The seductive failure mode of rubric-ops is measuring judge-attempter agreement as success. If both parties share a plausible misreading of a criterion, decomposed rubrics can score high agreement while shipping bad data. This determines whether we need an independent expert-audit sampling loop as a permanent fixture rather than a bootstrap phase. GATES: Whether a standing gold-set / expert-audit sampling channel is a required foundational component (consuming budget) or an optional per-project add-on.

For genuinely contested judgment calls, do we design for verdict pluralism (record 'defensible under reading A, not B' and pass with annotation) or forced resolution (every item gets pass/fail) - and can the system reliably distinguish 'contested by the criterion's nature' from 'contested because the rubric is underspecified'?

WHY: Forced resolution on genuinely contested items manufactures the exact reviewer-variance noise we're eliminating, just with a machine's false consistency. But pluralism without the underspecification distinction becomes an excuse bucket that swallows fixable rubric gaps. GATES: The verdict schema (binary vs. verdict-plus-reading), the feedback template for contested items, and whether contested-item volume feeds the rubric-revision queue.

Where does the one human interaction per item buy the most rubric-ops value: pre-verdict adjudication of the single most-contested claim, post-verdict confirmation of borderline fails, or asynchronous adjudication of exemplar-generating edge cases that improve the rubric for all future items?

WHY: The third option means the human budget compounds (each touch improves the compiled rubric) rather than being spent per-item. But it also means some items ship with unconfirmed verdicts. This is a fundamental resource-allocation choice that shapes escalation design. GATES: The escalation router's objective function: per-item verdict accuracy vs. rubric-improvement rate, and whether human touches write back into the compiled config.

Can the rubric compiler itself be trusted to be project-agnostic - i.e., does one compilation prompt/procedure hit target fidelity across instruction-set styles (terse axis lists, 40-page guideline docs, exemplar-heavy specs), or does compilation fidelity vary so much by input style that we need per-style compilation strategies and a human sign-off gate on every compiled rubric?

WHY: A miscompiled rubric poisons every downstream verdict on that project, and the errors are systematic, not random - worse than no AutoQA. If compilation needs human sign-off anyway, the 'foundational philosophy + per-project customization' story changes from automated ingestion to a structured authoring workflow with tooling. GATES: Whether ingestion is an automated pipeline with spot checks or a human-led compilation workshop with model assistance, and what the compiled-rubric acceptance test is.

What is the compiled rubric's unit of address - the attempter's deliverable as a whole, each discrete claim within it, or each (claim, criterion) pair - and how do we aggregate claim-level verdicts into the item-level pass/fail plus feedback without an aggregation rule that itself becomes a contested subjective judgment?

WHY: Aggregation is where subjectivity re-enters through the back door: 'three minor grounding lapses vs. one severe misalignment' is exactly the kind of weighing on which human reviewers currently diverge. If aggregation rules can't be specified per project, item-level verdicts inherit the variance we tried to remove. GATES: Whether severity/weight taxonomies are mandatory fields in every compiled rubric, and whether the system emits item-level verdicts at all vs. claim-level findings plus a recommended verdict.

Contrarian: is reviewer disagreement actually caused by criteria vagueness - or would perfectly operationalized rubrics leave most of the variance intact because it stems from expertise gaps and attention (root causes 2 and 3), meaning rubric-ops investment has a hard ceiling we should measure before building the compiler?

WHY: The ranked root-cause list is a hypothesis, not a measurement. If a cheap experiment (give current human reviewers a hand-operationalized rubric for one project and measure variance reduction) shows small gains, the foundation should weight evidence-verification and expertise-substitution over rubric decomposition - a different system. GATES: The build order and budget split between the rubric-compilation subsystem and the evidence-verification/feedback subsystems; whether a variance-attribution pilot is a mandatory phase-zero.

What does constructive feedback require from the compiled rubric beyond verdict machinery - do rubric items need paired 'what good looks like' statements and repair templates authored at compile time, or can feedback be generated at verdict time from the claim-evidence trace alone without drifting into advice the project owner would disown?

WHY: Feedback generated freestyle at verdict time can contradict project doctrine (recommending fixes the project forbids), which destroys attempter trust faster than a wrong verdict. If feedback needs compile-time authoring, the rubric schema roughly doubles and so does ingestion cost. GATES: Whether the compiled-rubric schema includes mandatory feedback-side fields (exemplar repairs, approved phrasings, doctrine constraints) or feedback is a downstream generation layer over verdict traces.

Lens: human-factors

Does the constructive feedback we return to attempters leak the judge's decision boundary - i.e., after N feedback cycles, do attempter scores rise on the AutoQA while staying flat or falling on held-out human-graded golden items?

WHY: Constructive feedback is a stated design goal, but it is also the highest-bandwidth channel for Goodharting: every explanation of why an item failed is a gradient the attempter can descend without improving true quality. If the divergence (judge score up, golden score flat) appears quickly, feedback must be abstracted (principle-level, delayed, or sampled) rather than item-specific. GATES: Feedback granularity and timing: item-specific rationales vs aggregated/principle-level feedback, and whether feedback text is generated from the same rubric the verdict uses or from a decoupled surface.

Can the judge distinguish genuine evidence-grounding from 'citation theater' - specifically, if we perturb or swap the evidence an attempter cites, does the AutoQA verdict flip at a rate that shows it is actually checking the citation-claim link rather than rewarding grounded-sounding prose?

WHY: The core positive-evaluation requirement ('is this statement actually grounded?') is exactly the surface attempters will learn to imitate: hedged, citation-dense, judge-pleasing prose. A perturbation test is the falsifiable check; if verdicts don't flip when cited evidence is invalidated, the grounding check is decorative and will be gamed within weeks of deployment. GATES: Whether grounding verification is implemented as semantic holistic judgment or as an explicit claim-evidence entailment step with adversarial perturbation testing in the eval harness.

For wholesale LLM-generated submissions, is our defense content-based detection or economics-based throttling - i.e., what is the measured cost ratio between an attempter generating a passing submission with a frontier model and us detecting it, and at what ratio do we abandon detection for identity/throughput/behavioral gates?

WHY: By mid-2026, frontier-model submissions can be indistinguishable from strong human work on content alone; a detection arms race we lose by construction wastes the QA budget. If the cost asymmetry favors the spammer, the architecture must shift defenses to rate limits, work-history consistency, provenance signals, and randomized deep audits rather than per-item classification. GATES: Whether the AutoQA pipeline includes an AI-detection module at all, versus spending that complexity budget on behavioral/longitudinal attempter modeling.

Do we actually care about LLM assistance provenance, or only content validity - is a correct, well-grounded annotation produced with heavy LLM help a pass, a fail, or a policy question that must be settled per-project before the foundation is designed?

WHY: This is unresolved in most annotation pipelines and it changes the objective function of the entire QA layer. If content-only, spam filtering collapses into quality filtering (simpler system). If provenance matters (e.g., because training data must reflect human judgment, not model self-distillation), we need signals the content itself cannot provide - process telemetry, timing, edit traces - which is a different data-collection contract with attempters. GATES: Whether the foundation's spam/falsehood axis is defined over content properties alone or requires process/provenance instrumentation as a first-class input.

In what order does the single human interaction see the AI's work - and at what verdict-presentation format does the human overturn rate fall below the AutoQA's measured error rate (the signature of rubber-stamping)?

WHY: Automation bias is the failure mode that silently converts 'human-in-the-loop' into 'human-as-liability-shield.' Whether the human judges blind-first then reconciles, sees the AI verdict with rationale, or sees rationale without verdict, measurably changes overturn behavior. If overturn rate < AI error rate, the human is adding negative information and the interaction budget is being wasted. GATES: The core review UX: blind-first independent judgment vs AI-verdict-first confirmation, and whether AI confidence/rationale is shown before or after the human commits.

Where does the one-human-interaction budget buy the most reliability per item: verifying AI-flagged failures, adjudicating low-confidence/disputed items, or spot-auditing AI passes - and is the answer stable across the AutoQA's confidence calibration or does it invert when calibration drifts?

WHY: With exactly one human touch, routing IS the system. Spending it on confirmed-looking failures duplicates the AI; spending it on confident passes catches judge blind spots but at low hit rate. The optimal allocation depends on the judge's calibration curve, which will drift as attempters adapt - so the routing policy must be designed as a function of measured calibration, not fixed at launch. GATES: The triage/routing architecture: which confidence bands and verdict types route to the human, and whether routing thresholds are static or closed-loop on observed overturn rates.

When the judge's domain competence is below the attempter's (frontier expertise, niche domains), can the AutoQA detect its own incompetence and abstain - and at what abstention rate does the system stop net-saving human effort for that project?

WHY: Expertise mismatch cuts both ways: the AutoQA confidently 'correcting' an expert attempter is worse than no QA, because it burns attempter trust and pollutes the record with false fails. A judge that cannot self-diagnose incompetence forces either per-domain capability gating or per-project human fallback, which changes the project-agnostic claim of the foundation. GATES: Whether the foundation includes a mandatory abstain/escalate verdict class with per-project competence calibration, and whether some projects are declared AutoQA-ineligible.

Does the AutoQA prefilter shift the human reviewer's base rates enough to cause criterion drift - when the queue reaching the human is enriched with hard/adversarial cases, do reviewers become systematically harsher or more vigilance-fatigued than on an unfiltered queue, and does seeding known-verdict items into the queue correct it?

WHY: Filtering changes the population the human sees, which changes the human. A reviewer seeing 80% genuinely-borderline items recalibrates their pass threshold and fatigues faster; their disagreement with the AI then looks like AI error when it's induced drift. Without seeded gold items you cannot even measure this, because there is no fixed reference in the post-filter queue. GATES: Whether the queue design mandates seeded known-verdict items and easy-case dilution, and how reviewer accuracy is measured post-filter.

If AutoQA verdicts affect attempter pay or tenure, what appeal mechanism absorbs disputes without either re-importing full human review load or becoming a rubber stamp - concretely, does an appeal route to a human, a stronger/differently-prompted AI pass, or evidence-required re-adjudication, and what appeal-success rate signals a healthy vs broken system?

WHY: The moment verdicts carry economic consequence, the system becomes adversarial and appeals are the trust release-valve. An appeal path that costs a full human review breaks the one-interaction budget at scale; one that never succeeds destroys attempter trust and drives silent gaming instead of open dispute. The target appeal-success band (too low = judge unaccountable, too high = judge unreliable) must be chosen before launch because it defines the verdict's enforcement weight. GATES: Whether AutoQA verdicts are advisory or enforcement-grade, and the architecture of the appeal tier (human, escalated AI, or evidence-gated).

When the single human reviewer and the AutoQA persistently disagree, what arbitrates - given that noisy, misaligned human reviewers are the original problem, does escalation to another human just re-import the variance we built the system to remove?

WHY: The system's stated origin is human reviewer misalignment, yet most HITL designs quietly treat the human as ground truth for disagreements. If human arbitration is itself ~60% reliable, disagreement resolution needs a different substrate: golden exemplars, criterion-author rulings, or multi-judge consensus with documented rationale. Choosing wrong means the AutoQA slowly trains toward the noise it was meant to filter. GATES: The ground-truth hierarchy of the foundation: what outranks what (golden set > criterion author > human reviewer > AutoQA?), and how disagreements feed back into judge calibration.

Is the observed reviewer variance actually a human-factors problem at all - if we run two independently-configured AutoQA instances on the same items, do they disagree at a rate comparable to the human reviewers, indicating the variance lives in criterion ambiguity rather than in humans?

WHY: This is the contrarian check on the whole lens: if the criteria themselves are underdetermined, no amount of incentive design, bias mitigation, or fatigue management fixes pass/fail variance - and the AutoQA will simply be misaligned with itself in a way that looks authoritative. The falsifiable test (judge-judge disagreement vs human-human disagreement per criterion) tells us whether budget goes to per-project rubric disambiguation/decision-rule authoring or to the QA layer itself. GATES: Whether the foundation's first-class artifact is the judge, or a per-project criterion-operationalization step (decision rules, boundary exemplars) that both judge and humans consume.

Is the one-human-interaction-per-item goal itself the right target - would zero per-item human touch plus randomized deep audits and drift monitoring outperform one shallow, automation-biased touch per item on total error caught per human-hour?

WHY: One guaranteed shallow interaction per item invites rubber-stamping and gives a false sense of human oversight, while consuming the entire human budget on breadth. Concentrating the same hours into deep audits of a random sample yields unbiased error estimates, resists automation bias (auditors work blind), and creates real deterrence for gaming - but sacrifices per-item human accountability. This is a fork in the foundation, not a tuning knob. GATES: The fundamental human-budget allocation model: per-item review vs sampled deep audit vs confidence-routed hybrid, which determines the entire pipeline shape.

What keeps the human reviewers calibrated and engaged once the AutoQA writes the verdicts and the constructive feedback - measurably, does reviewer skill and overturn quality decay over months of reconcile-only work, and if so, who remains competent to calibrate the judge?

WHY: The judge needs an ongoing supply of high-quality human judgments to detect its own drift, but the system as designed converts reviewers from judgment-producers into verdict-confirmers - the exact role that atrophies judgment. If reviewer skill decays, the calibration signal degrades precisely as attempter gaming pressure grows, a compounding failure. The design may need deliberate blind-judgment quotas purely to preserve the calibration workforce. GATES: Whether the foundation mandates a standing blind-review stream (reviewers judging without AI output) as calibration infrastructure, and how reviewer performance itself is QA'd.

Lens: grounding

At what claim-decomposition granularity does judge-human agreement on grounding verdicts stop improving - holistic review-level, per-rubric-axis, per-sentence, or atomic claim - and what is the cost curve per level?

WHY: Decomposition is the single biggest architectural choice: atomic claims make entailment checks tractable and auditable but multiply judge calls and can strip context that changes a claim's meaning; holistic judging is cheap but unfalsifiable. Without knowing where agreement plateaus, we cannot size the pipeline or the per-item cost. GATES: Whether the pipeline's first stage is a claim extractor feeding N entailment checks, or a single structured holistic judge; also the shape of the verdict data model.

Is the evidence universe closed (AI output under review + task instructions + attempter-cited spans) or open (world knowledge, external references) - and what verdict do we assign to attempter claims that are true-in-the-world but unsupported in the closed corpus?

WHY: A closed universe makes verification a well-defined entailment problem; an open universe requires retrieval and inherits the judge's own hallucination risk. Many attempter claims (e.g. 'this API call is deprecated') are world-knowledge claims with no in-corpus evidence, and how we score them determines the false-flag rate. GATES: Whether the foundation includes a retrieval/grounding-corpus component per project, and the definition of the 'unsupported' verdict class.

What entailment standard does an attempter statement have to meet - strict logical entailment from cited evidence, or 'reasonable-expert support' - and is the output a 3-way verdict (entailed / contradicted / unsupported) or a graded support score with thresholds?

WHY: Strict entailment will fail nearly all human-written prose (humans compress, paraphrase, and generalize); loose support standards reintroduce exactly the subjective variance we're trying to remove. The standard chosen defines what 'grounded' means for the whole system and must be written into every per-project customization. GATES: The verdict taxonomy, the calibration target for the judge, and the wording of the grounding criterion given to projects.

Can a judge distinguish warranted praise from empty praise at usable reliability - concretely, on a seeded set where half the praised properties are genuinely present and half are absent or fabricated, what precision/recall does the judge hit per praise type ('accurate', 'complete', 'clear', 'idiomatic')?

WHY: Positive-claim verification is the stated differentiator of this system, but praise types differ radically in checkability: 'the code compiles' is verifiable, 'the explanation is clear' is barely falsifiable. If the judge only verifies the checkable subset, we need to know which subset and say so, rather than pretend all praise is verified. GATES: Whether positive-claim verification ships as a hard pass/fail gate, a flag-only signal, or is restricted to an enumerated set of checkable praise types.

Does verifying a positive claim like 'no errors present' or 'fully addresses the prompt' require the judge to independently re-attempt the task (exhaustive defect search), and if so, is the judge's own miss rate on seeded defects low enough that its 'confirmed clean' verdict adds information over the attempter's?

WHY: Universal/negative-existential claims can't be verified by citation checking - they require proving absence. If the judge's defect-finding recall is not materially above the attempter population's, 'AutoQA confirmed the pass' is theater, and pass-verdicts should be treated as unverifiable rather than verified. GATES: Whether the architecture includes an independent re-attempt/defect-search stage, and how pass/clean attempter verdicts are labeled in output (verified vs. not-contradicted).

Is omission detection tractable project-agnostically - i.e., can a judge generate the reference set of 'what should have been said' from instructions + artifact alone at usable recall, or does credible omission detection require a per-project gold defect taxonomy, making it customization work rather than foundation work?

WHY: Detecting what's missing requires a completeness reference that doesn't exist in the attempter's text. If that reference can only come from project-specific defect enumeration, then omission detection belongs in the per-project layer and the foundation should only define its interface - a very different build than a universal omission detector. GATES: Whether omission detection is a core foundation module or a per-project plugin with a defined contract; also whether project onboarding must include building a defect taxonomy.

Where is the line between a verifiable claim and a legitimate subjective judgment call, and what does the system do with claims classified as opinion - check only internal consistency with the evidence the attempter cites, or decline to score them at all?

WHY: Root cause #1 is subjective criterion interpretation; if the grounding layer tries to fact-check genuine judgment calls it will manufacture false disagreements, and if it exempts too much as 'opinion' it verifies nothing. The claim-type classifier's boundary IS the system's scope, and misrouting here is the main source of attempter-facing injustice. GATES: Whether the pipeline needs a claim-type classification stage, and the verdict semantics for opinion-class claims (consistency-checked vs. unscored).

When the attempter's cited evidence supports their conclusion but uncited evidence elsewhere in the artifact undermines it, do we verify against cited-evidence-only or the full artifact - and are cherry-picking failures scored as grounding failures or omissions?

WHY: Cited-only verification is cheap and precise but blesses cherry-picking, which is exactly the failure mode of a lazy-but-plausible attempter. Full-artifact verification catches it but is an order of magnitude more work per claim and blurs into omission detection. This choice determines what the entailment checker actually reads. GATES: The evidence scope of the entailment checker and the taxonomy boundary between 'ungrounded' and 'incomplete' verdicts.

Should the foundation mandate an upstream instruction change requiring attempters to attach explicit evidence pointers (quotes, line refs) to every load-bearing claim - converting open verification into closed entailment - and what does piloting show about citation-padding and other gaming responses?

WHY: Requiring citations could be the highest-leverage move in the whole design: it turns the hardest problem (locating evidence) into the attempter's job and makes verification nearly mechanical. But it changes the attempter product, adds friction, and invites decorative citations that pass surface checks - which would corrupt the very signal we depend on. GATES: Whether the AutoQA spec includes required changes to attempter-facing instructions, or must work with free-form uncited reviews as given.

What is the empirical base-rate split between attempter claims that are actually false versus merely uncited-but-true, and at what judge precision does flagging 'unsupported' claims for the one allowed human interaction beat spending that interaction elsewhere?

WHY: If most unsupported claims turn out true, aggressive flagging burns the human-interaction budget on exoneration and erodes attempter trust; if falsehoods are common, permissive thresholds let spam through. The threshold cannot be set without the base rate, and the base rate differs from the error-only-QA world because we now audit positive statements too. GATES: Verdict thresholds, whether 'unsupported' auto-fails or only flags, and the triage function that allocates the single human interaction per item.

Is judge-judge agreement on grounding verdicts (same item, resampled/paraphrased prompts, sibling models) actually higher than the human reviewer-reviewer agreement we're trying to fix - measured per verdict class - and what consistency mechanism (self-consistency voting, fixed rubric anchors, deterministic decomposition) closes the gap if not?

WHY: The system's entire justification is reducing unexplained reviewer variance. An LLM judge has its own stochastic variance, and if its verdict flip rate on borderline grounding calls matches human disagreement rates, we've automated the problem rather than solved it. This must be a falsifiable pre-ship gate, not an assumption. GATES: Whether verdicts require ensembling/multi-sample agreement (with its cost multiplier), and the go/no-go criterion for deploying the judge at all.

Do we apply the grounding standard reflexively to AutoQA's own feedback - every criticism it sends an attempter must itself cite entailing evidence and pass the same entailment check - and what false-accusation rate is low enough that attempters keep trusting and acting on the feedback?

WHY: Constructive feedback is a stated goal, and an ungrounded accusation from the QA system is more corrosive than an ungrounded claim from an attempter - it trains attempters to dispute or ignore the system. Reflexive self-verification roughly doubles judge cost, so it must be a deliberate decision with a trust-threshold target, not an afterthought. GATES: Whether the feedback generator gets its own verification pass before emission, and the precision SLO for attempter-facing criticisms.

Contrarian: if the judge can reliably verify whether an attempter's statements are grounded - including re-finding defects to check 'clean' verdicts - does grounding verification actually require less capability than doing the annotation itself, and if not, what evidence supports keeping mandatory human attempters upstream of a judge that could replace them, rather than inverting the design (model attempts, human grounds)?

WHY: The lens presumes verification is easier than generation, which is true for closed-form checks but unproven for omission detection and absence-proving, where the judge must out-attempt the attempter. If verification is capability-equivalent to attempting, the honest design conclusion is different: either scope AutoQA down to the genuinely-easier-to-verify subset, or acknowledge the human's value is legal/accountability cover and design the QA layer around that instead. GATES: The fundamental scope of the grounding module - full verification vs. only the verification-easier-than-generation subset - and how the system's value proposition is framed to stakeholders.

Lens: hybrid-workflow

What exactly is the single human interaction spent on: blind re-judgment of the item, verification of the AI's verdict+rationale, or adjudication of specific AI-flagged claims - and which of these maximizes error reduction per minute of human time?

WHY: These three interaction types produce different label independence, different cost per item, and different anchoring risk. The whole workflow, UI, and downstream statistics change depending on which one we pick. GATES: The core unit of work in the human queue and the data schema of what a human touch produces.

Which measurable confidence proxy from the LLM judge (self-consistency across k samples, verdict flip-rate under paraphrase, ensemble disagreement, rationale-evidence entailment score) actually predicts human overturn with AUROC high enough to route on - and what is that threshold?

WHY: Confidence-based routing is worthless if the confidence signal is uncorrelated with actual judge error. This is empirically falsifiable per project and determines whether routing is principled or theater. GATES: Whether we build routing on judge-internal signals at all, and which signal the router consumes.

Should the escalation threshold be set to a fixed human-capacity budget (top-N most uncertain items) or to a target residual error rate (escalate until expected post-QA error < X%), and which quantity is actually the binding constraint?

WHY: Budget-first routing gives predictable staffing but uncontrolled quality; error-first routing gives quality guarantees but unpredictable queue depth. You cannot optimize both; picking one dictates the router's objective function. GATES: The router's optimization target and how project owners configure it.

What is the asymmetric evidence bar for standalone AutoQA verdicts: what measured false-FAIL rate (on adjudicated gold) is acceptable before the system may fail an attempter with zero human touch, and is auto-PASS held to a different bar than auto-FAIL?

WHY: Failing a human attempter has pay/reputation consequences; a wrong pass just leaks noise into training data. These harms are asymmetric, so a single trust threshold is almost certainly wrong. GATES: Whether the autonomous lane emits both verdict polarities or only one, and the gold-set size needed to certify it.

Contrarian: given the stated root problem is that human reviewers disagree with each other for unexplainable reasons, does escalating to one human reduce error at all - or does it re-inject the exact variance we are removing while laundering AI verdicts with a human signature? What overturn-quality audit would prove the human touch adds signal?

WHY: The lens presumes human escalation improves outcomes. If a second human agrees with the escalation human no more often than with the AI, the hybrid layer is cost without benefit and the design should shift toward better autonomous calibration instead. GATES: Whether per-item human escalation exists in v1 at all, versus humans only in calibration/gold roles.

Should the scarce human interactions be spent on per-item adjudication at all, versus spent upstream on building per-criterion calibration sets and disambiguating rubric language - and at what point does one hour of rubric disambiguation beat one hour of item review in agreement gains?

WHY: Root cause #1 is subjective interpretation of criteria. If interpretation variance dominates, per-item escalation treats symptoms while the same human minutes spent codifying criterion boundary cases would raise autonomous accuracy across all items. GATES: The allocation policy for the human-interaction budget: adjudication queue vs. calibration/rubric-refinement queue.

Does the escalated human judge blind (no AI rationale shown) or anchored (AI verdict and cited evidence shown) - and is the measured anchoring cost smaller than the independence value of blind labels for calibrating the judge?

WHY: Anchored review is faster and catches AI reasoning errors directly, but anchored labels cannot be used to measure judge accuracy or drift because they are contaminated. This is a one-way door for the statistics. GATES: Whether escalation labels double as calibration data, or a separate blind-label stream must be funded.

At what granularity does escalation route: the whole item, or the specific contested claim/criterion the judge is uncertain about - and can claim-level adjudication (answer 2-3 pointed micro-questions) stay within the one-interaction budget while whole-item review cannot?

WHY: The system evaluates positive statements claim-by-claim (groundedness, criterion-alignment, evidence-consistency). Judge uncertainty is usually localized; sending the whole item wastes the human on parts the AI already resolved confidently. GATES: Queue item schema and the escalation UI contract (full item vs. targeted micro-adjudications).

Should escalation queues be organized by criterion (specialist reviewers per evaluation axis) or by item (generalist review), and does criterion-specialist routing measurably close the expertise-gap root cause where generalist routing does not?

WHY: Root cause #2 is expertise gaps. A reviewer strong on factual grounding may be weak on style criteria. Specialist queues change reviewer staffing, assignment logic, and how verdicts are reassembled from partial adjudications. GATES: Queue partitioning, reviewer assignment model, and verdict-assembly logic.

What statistical model combines sparse human labels with dense AI judgments - per-reviewer reliability weighting (Dawid-Skene family), per-project Bayesian calibration of judge scores, or simple hierarchical override - and how many human labels per project-criterion cell does the chosen model need before its posteriors beat the raw judge?

WHY: Below the convergence threshold, the fusion model is worse than trusting the judge alone; above it, it corrects systematic judge bias per criterion. The answer sets the mandatory human-label spend during ramp-up and whether reviewer identity must be tracked. GATES: The verdict-fusion architecture and the minimum viable human-label collection plan per project.

What is the cold-start protocol for a new project with zero human labels - a 100%-escalation burn-in window, seeded gold items authored during project setup, or transferred priors from similar projects - and what quantitative graduation criterion (e.g., judge-human kappa over the last N adjudications) flips the project into confidence-routed mode?

WHY: Since the system is project-agnostic and customized per instruction set, every project starts uncalibrated. Without an explicit graduation gate, projects either stay expensively human-heavy forever or go autonomous on faith. GATES: Project onboarding lifecycle, burn-in cost model, and the automated mode-transition rule.

Does constructive feedback delivered to attempters need a different confidence bar than the pass/fail verdict - i.e., can the system ship AI-generated coaching on items whose verdict was auto-passed, and what is the harm model when specific feedback claims (e.g., 'your citation does not support this') are themselves wrong?

WHY: Feedback contains checkable factual claims about the attempter's work; a wrong specific accusation damages trust more than a bare wrong verdict. If feedback needs human review, the one-interaction budget gets split between adjudication and feedback vetting. GATES: Whether feedback generation is in the autonomous lane, the escalated lane, or gated by its own groundedness check.

Once a project runs in trusted-standalone mode, what ongoing randomized audit rate and drift statistic (e.g., CUSUM on judge-vs-audit disagreement per criterion) triggers automatic de-graduation back to heavier escalation - and does audit sampling need to be unpredictable enough that attempters cannot learn to write for the judge?

WHY: An autonomous lane fails silently: judge drift, instruction changes, and attempter adaptation to the judge's known preferences all degrade quality without any surfaced signal. Confidence-routing itself is a gameable surface if deterministic. GATES: The monitoring subsystem, audit sampling design (including randomization), and the automatic trust-revocation rule.

Lens: product-feedback

At what verdict granularity (binary pass/fail, per-axis scores, or claim-level grounded/ungrounded judgments on each attempter statement) does judge-human agreement stop improving, and does claim-level verification measurably reduce the unexplained reviewer variance we're trying to fix?

WHY: The whole premise is that humans disagree at the holistic-verdict level. If decomposing to claim-level checks doesn't recover agreement, the AutoQA inherits the same subjectivity problem and we've just automated the noise. If it does, the entire data model must be claim-first with verdicts as derived aggregates. GATES: The core output schema: whether verdicts are primitive objects or computed rollups of atomic claim judgments, and whether the feedback surface is organized around claims or around holistic scores.

For positive-statement verification ('is this attempter claim grounded in the cited evidence?'), what is the falsifiable acceptance bar - e.g., does the judge's grounded/ungrounded label match an expert adjudicator at >=X% on a golden set - and what X makes the system trustworthy enough to auto-pass items without human review?

WHY: Verifying positive claims is harder than spotting errors; without a measured precision bar we can't decide which items get the one allowed human interaction versus none. This number determines the human-in-the-loop routing policy and the escalation threshold. GATES: The triage policy: which confidence band auto-passes, which auto-fails, and which consumes the single human interaction.

Should the single permitted human interaction be spent on adjudicating the judge's uncertain verdicts, or on the attempter responding to feedback (contest/revise), and does that choice differ by root cause (subjectivity vs expertise gap vs attention lapse)?

WHY: One human touch per item is a hard budget. Spending it on verdict adjudication improves label quality; spending it on attempter response improves future behavior and catches judge errors the attempter can see. These are different products with different data flows. GATES: Where the human sits in the pipeline (before verdict finalization vs after feedback delivery) and what UI/workflow that interaction needs per project.

What is the minimal structure of a feedback unit that attempters actually act on - e.g., (quoted attempter claim, specific evidence span, criterion clause violated, concrete fix) - and can we falsify 'more detail is better' by measuring revision quality against feedback length and structure?

WHY: Constructive feedback is a stated goal, but unstructured LLM feedback tends toward verbose, hedged prose that attempters don't read closely. If a rigid 4-tuple outperforms free text on revision quality, feedback generation becomes a constrained extraction task, which is cheaper, more verifiable, and project-portable. GATES: The feedback schema and generation approach: templated evidence-linked tuples vs free-form critique, and how feedback quality itself gets QA'd.

Must every piece of feedback be traceable to a specific clause in the project instruction set (criterion-linked), and what happens to judge observations that are real problems but map to no written criterion - surface them, suppress them, or queue them as instruction-gap reports to the project owner?

WHY: Root cause #1 is subjective interpretation of criteria. If the judge can cite un-written standards, it amplifies the subjectivity problem and attempters can't contest it. But suppressing real issues loses signal. The gap-report path turns judge disagreement into instruction improvement, attacking the root cause rather than the symptom. GATES: Whether the feedback generator is hard-constrained to cite instruction clauses, and whether an instruction-gap feedback channel to project owners is a first-class output of the system.

How do we distinguish 'attempter is wrong' from 'criterion is ambiguous' in the verdict itself - should the AutoQA emit a third outcome class (ambiguous-under-instructions) with measured inter-judge disagreement as its trigger, rather than forcing a pass/fail?

WHY: Forcing binary verdicts on genuinely ambiguous items is exactly what makes human reviewers look misaligned. If ensemble judge self-disagreement reliably predicts human reviewer disagreement, ambiguity becomes detectable and routable instead of masquerading as attempter failure. GATES: The verdict taxonomy (2-class vs 3-class), the ensemble/self-consistency architecture needed to detect ambiguity, and whether ambiguous items are exempt from attempter penalty.

What is the explainability contract for a failing verdict: does every fail require verbatim evidence spans and a reproducible reasoning chain an attempter could independently check, and do we reject (not deliver) judge verdicts that can't meet that bar even when the verdict is likely correct?

WHY: A correct-but-unexplainable fail destroys attempter trust and is uncontestable, recreating the opaque-human-reviewer problem. Making explainability a delivery gate trades some recall for legitimacy; whether that trade is acceptable changes the judge's output requirements fundamentally. GATES: Whether explainability is a hard output constraint (fail-closed) or best-effort annotation, and the verification step that checks feedback self-consistency before delivery.

Does delivered feedback measurably change future attempter behavior - what longitudinal metric (repeat-error rate on the same criterion per attempter) do we commit to, and after how many items of no improvement do we conclude feedback is decoration and the real product is filtering?

WHY: This is the uncomfortable question for the feedback lens: the constructive-feedback goal presumes attempters read, understand, and apply it. Marketplace annotation workforces often optimize throughput, not learning. If repeat-error rates don't move, feedback generation is wasted spend and the design should shift budget to routing, filtering, and attempter selection instead. GATES: Whether feedback generation is a core deliverable with a learning-loop KPI, or a lightweight byproduct - this reallocates most of the product surface.

Should attempters be able to contest AutoQA verdicts, and does a contest consume the one human interaction - i.e., is the appeals channel the human-in-the-loop, and what contest-overturn rate would indicate the judge (not the attempters) is the problem?

WHY: A contest channel is both a trust mechanism and the cheapest source of judge-error labels. But it competes for the single human-touch budget and can be gamed by throughput-motivated attempters. The overturn-rate threshold gives a falsifiable tripwire for judge recalibration. GATES: Whether appeals exist, who adjudicates them within the one-interaction budget, and the monitoring metric that triggers per-project judge retuning.

What exactly is the per-project customization surface - is it only (instruction set + criterion decomposition + calibration exemplars + severity weights), and can we prove the foundation is truly project-agnostic by onboarding a new project with zero prompt-engineering of the core judge?

WHY: Project-agnostic is the stated constraint, but every per-project prompt hack that leaks into the core erodes it. Defining the customization surface as a closed configuration schema makes 'agnostic' falsifiable: a new project either onboards through config alone or the abstraction is wrong. GATES: The boundary between the frozen foundation and the per-project config schema, and the onboarding procedure (config authoring vs bespoke judge tuning).

How many calibrated exemplars per criterion (expert-adjudicated pass/fail/ambiguous anchors) does a new project need before judge-human agreement plateaus, and who produces them - because if the answer is 'a large expert-labeled set,' the system quietly reintroduces the expensive aligned-human-reviewer dependency it was built to replace?

WHY: Calibration exemplars are the standard fix for criterion subjectivity, but their cost scales with project count and criterion churn. Knowing the plateau point (5 vs 50 vs 500 per criterion) determines whether per-project onboarding is a day of an SME's time or an ongoing labeling program. GATES: The per-project onboarding cost model and whether calibration is a one-time setup artifact or a continuous pipeline with its own QA.

Should the AutoQA verdict feed attempter-level consequences (ratings, pay, removal) or remain item-level coaching only - and if consequences attach, what precision floor and audit trail does each verdict need to be defensible against a wrongly-failed attempter?

WHY: The moment verdicts affect livelihoods, error tolerance collapses and attempters rationally start gaming the judge (writing to please the model rather than to be correct). This changes required precision, explainability depth, contest rights, and how feedback language is phrased. GATES: Verdict consumption policy downstream (coaching-only vs consequential), which sets the precision/audit requirements for the entire judge pipeline.

How do we prevent feedback-induced monoculture - if attempters learn the judge's stylistic preferences, do annotations converge on judge-pleasing patterns that reduce the diversity and independent judgment the training data exists to capture, and what drift metric would detect this?

WHY: The feedback loop is the product's point, but its failure mode is Goodharting the annotation workforce: evidence-citation boilerplate, hedge phrasing, and criterion-parroting that passes QA while degrading the underlying data. A measurable diversity/drift signal decides whether feedback should ever include stylistic guidance at all. GATES: What categories of feedback are permitted (substance-only vs substance+style), and whether a population-level drift monitor is a required system component from v1.