AutoQA Foundation - The Question Inventory
The decision-gating questions that define what the system must measure, automate, escalate, and govern.
These are working drafts. Names and file paths in them may be out of date.
Published judge studies mostly score model output, not a person's written evaluation. Positive praise still has to be checked on your own work.
AutoQA Foundation - Research Atlas
Executive synthesis of the recency-biased research sweep, including strongest findings, disagreements, caveats, and gaps.
AutoQA Foundations - Philosophy & Structure
The eleven principles, eight-stage architecture, operating model, legal floor, and unresolved decisions.
Raw Questions by Lens
The full 92-question brainstorm across measurement, judge failure modes, rubric operations, human factors, grounding, workflow, and feedback.
Combined Domain Reports
All domain reports concatenated into one source atlas, with evidence summaries, tools, disagreements, gaps, and verification notes.
Completeness Critic & Gap-Fill Reports
Adversarial audit of the original sweep, plus psychometrics, legal, content moderation, and feedback-science gap fills.
Domain Report - Academic Judges
Academic judge benchmarks, task-dependent capability limits, bias, and model-selection evidence.
Domain Report - Annotation Quality
Annotation disagreement, rater uncertainty, LLM contamination, label error, and human-assistance effects.
Domain Report - Critique Models
CriticGPT lineage, human-plus-critic evidence, omission limits, and feedback validation.
Domain Report - Grounding & Faithfulness
Claim verification, evidence closure, decomposition, citation quality, and faithfulness detectors.
Domain Report - Hybrid Statistical Methods
Prediction-powered inference, selective escalation, conformal routing, calibration, and drift attribution.
Domain Report - Rubrics 2025-2026
Rubric compilation, criterion-level grading, rubric failure modes, and expert-grounded exemplars.
Domain Report - Industry Practice
Production QA stacks, reviewer trust telemetry, vendor calibration patterns, and verifier economics.
Domain Report - Contrarian Evidence
The strongest case against naive AutoQA: invalidity, gaming, correlated errors, automation bias, and limits.
Domain Report - Frontier 2026
The newest verifier, routing, rubric, and judge findings through early July 2026.
Benchmark & Calibration Protocol
How to construct adjudicated gold data, calibrate evaluators, run ablations, set launch gates, and monitor drift.
Foundation Package - Changelog
Version history and recorded changes for the foundation package.
Evaluation Contract Specification
Normative semantics for project contracts, evidence, criteria, findings, uncertainty, feedback, security, and conformance.
Decision Policy Reference
A target-specific rule engine for accept, rework, reject, and escalate decisions, including the bounded human interaction.
Foundation Package - Manifest
Machine-readable inventory and hashes for the complete package.
Foundation Package - README
Package overview, core model, invariants, adoption sequence, and conformance levels.
Foundation Package - Version
Current package version identifier.
Worked Example - Benchmark Item
Adjudicated benchmark record for the synthetic case, including acceptable outcomes and ambiguity metadata.
Worked Example - Case Input
A strong attempt incorrectly failed by a human reviewer for not forcing an unsupported binary conclusion.
Worked Example - Evaluation Output
Complete AutoQA result accepting the attempt, returning the annotation for rework, and generating audience-specific feedback.
Worked Example - Human Resolution
A bounded precedence question that hides the provisional decision and asks only what can change the outcome.
Worked Example - Project Contract
Synthetic closed-world evidence project with positive-proof requirements, anchors, severity, and explicit decision rules.
JSON Schema - Benchmark Item
Runtime schema for adjudicated truth, acceptable actions, ambiguity, challenge attributes, and leakage controls.
JSON Schema - Case Input
Runtime schema for task, evidence bundle, attempt, human review, references, and provenance.
JSON Schema - Common Definitions
Shared runtime definitions used by the project, case, evaluation, benchmark, and resolution schemas.
JSON Schema - Evaluation Output
Runtime schema for claims, evidence relationships, findings, criterion assessments, decisions, annotation audit, and feedback.
JSON Schema - Human Resolution Packet
Runtime schema for one bounded, decision-changing human question and its context.
JSON Schema - Project Contract
Runtime schema for approved project instructions, criteria, evidence policy, decision policy, and calibration requirements.
Template - Criterion Authoring
CSV template for drafting criterion intent, proof standards, failure anchors, evidence requirements, severity, and governance.
Template - Gold Set
CSV template for building an adjudicated calibration and test set.
Template - Model Bakeoff
CSV template for comparing evaluator configurations by criterion family.
TypeScript Domain Model
Strict TypeScript types for project contracts, cases, evaluations, benchmarks, findings, decisions, feedback, and human resolution.
TypeScript Configuration
Compiler configuration for the package's type definitions.
Validation - Hash Utilities
Canonical content hashing utilities used by the validation suite.
Validation - Example Schema Checks
Python validator for the package's example documents.
Validation - Package Integrity
Cross-document, schema, hash, proof-invariant, policy, and package integrity checks.
Downloads
| File | Format | Size | SHA-256 |
|---|---|---|---|
| autoqa-foundation-v0.1.0.zip | ZIP | 78.3 KB | e020f1e46f91 |
| miniqa-0.2.0-20260717.zip | ZIP | 29.2 KB | 790381cacf9b |
Machine-readable indexes
| File | Format | Size | SHA-256 |
|---|---|---|---|
| assets/library-fulltext.json | JSON | 1.0 MB | df3ce0d313f7 |
| assets/search-index.json | JSON | 718.8 KB | 7abf9be3f146 |
Page record
Source: site/library.html