Domain Report — Industry Practice

Domain: industry-practice

a major labeling platform's production QA stack is double-blind independent labeling, automated discrepancy detection, expert adjudication, and rework, with a fine-tuned-LLM similarity judge that only FLAGS below-threshold responses for human review (never autonomously fails them), Krippendorff's alpha as the published reliability bar (>=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor), and a multidimensional per-worker 'trust score' that routes complex tasks to high-trust raters. Load-bearing

EVIDENCE: Fetched and read the full a major labeling platform 'Inside the data factory' engineering post. It details the 4-step double-entry review flow, LLM-assisted QC (fine-tune on expert-verified responses -> similarity score -> flag sub-threshold for human review), explicit Krippendorff alpha thresholds preferred over Cohen/Fleiss kappa, per-task metric menu (Kendall's tau/ELO for rankings, F1/Matthews for binary), trust-score routing, 3% workforce acceptance rate, SLA-backed pay-only-for-quality guarantee, and admitted failure modes: for generative tasks 'there isn't a single clear-cut right answer' so accuracy is relative to the client's worldview, and low precision is attributed to poor instructions/edge-case coverage as often as to workers. SOURCE: Inside the data factory: How a major labeling platform produces the highest quality data at scale | https://a major labeling platform.com/blog/inside-the-data-factory-how-a major labeling platform-produces-the-highest-quality-data-at-scale | c. 2024, verified live 2026-07-14 | primary IMPLICATION: The most detailed public vendor QA architecture matches our hybrid design: LLM-QC as a triage prefilter feeding one human touch, not an autonomous verdict-writer. Adopt chance-corrected alpha (>=0.80 bar) as the industry-legible reliability floor, trust-score routing for reviewer allocation, and note the vendor itself concedes the ground-truth construct for generative work is client-instruction satisfaction, not objective truth (taxonomy Q1, Q2, Q6).

An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only F1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting F1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice. Load-bearing

EVIDENCE: Fetched the vendor's full research post (2026-05-27). The banking meta-eval is explicitly a meta-eval of verifiers, not agents; text-only 'autorubric' judges fail because evidence lives in spreadsheet formulas/tool state they cannot see; snapshot serialization loses formula references; the vendor's agent-judge (open-sourced, OpenHands SDK) opens files and follows references at inference time. Cheapest agent-judge config (~$42) beat the best baseline (~$422) by ~3 F1 at one-tenth cost; within one model family the architecture gap was 9.5 F1. They prescribe measuring every verifier against adjudicated expert labels before trusting it to score work or produce rewards. SOURCE: Your verifier is probably the bottleneck. We built one that isn't. (the vendor's agent-judge verifier and banking verifier meta-eval) | [link withheld] [link withheld] | 2026-05-27 | primary IMPLICATION: Direct evidence for three taxonomy questions: (Q1) meta-evaluation must use adjudicated expert labels, and a ~89.5% human agreement floor is achievable with dual-coding + adjudication; (Q5) the judge-vs-adjudicated-human ceiling is wildly domain-dependent (F1 0.63 vs 0.95), so per-axis judge-autonomous/assisted/human-only lanes are mandatory, not optional; (Q3) evidence closure is the binding constraint - the judge must see the same evidence surface the attempter's claims are about, or it verifies against a flattened proxy.

The 2026 eval-tooling vendors (LangSmith, Braintrust, Galileo) have converged on one judge-calibration pattern - small human-labeled reference sets (LangSmith: start at ~20 balanced examples; Braintrust: 50-100 per scoring dimension) with an 'alignment score' defined as RAW PERCENT AGREEMENT with the human expert, iterate the judge prompt against it, and route low-confidence or judge-disagreement items to humans - and none of them ships chance-corrected statistics. Load-bearing

EVIDENCE: Fetched LangSmith's official 'Improve LLM-as-judge evaluators using human feedback' doc: annotation queue -> reference dataset (>=20 examples, balanced 0/1 labels) -> Evaluator Playground -> alignment score = 'percentage of examples where the evaluator's judgment matches that of the human expert' -> cluster misalignments into failure modes and encode as prompt instructions. Fetched Braintrust's 2026-04-03 article: 3-tier stack (deterministic checks -> LLM judge -> human), 50-100 labels per dimension before trusting a judge, multi-judge disagreement routed to humans, worst failure mode named as 'silent overconfidence' (judge always returns a score even without adequate context), and explicit warning that human review is not automatically gold standard (fatigue, vague rubrics can be worse than a decent judge). Galileo's CLHF (2025-02-10 blog) auto-adapts metrics from expert corrections. SOURCE: LangSmith: Improve LLM-as-judge evaluators using human feedback (+ Braintrust: LLM-as-a-judge vs human-in-the-loop, 2026-04-03) | https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback | retrieved 2026-07-14 (Braintrust article 2026-04-03) | primary IMPLICATION: Our foundation can adopt the proven mechanics (annotation-queue-to-reference-set loop, failure-mode clustering into rubric amendments, disagreement routing) but must NOT copy the metric: the entire tooling industry standardized on raw percent agreement over tiny unbalanced sets - exactly what taxonomy Q1 proposes banning. Differentiation and validity headroom lie in chance-corrected stats, per-class recall at fixed prevalence, and minimum reference-set designs; also budget expectations: tens of labels per axis is the industry's plateau assumption, not hundreds.

At Scale AI's Outlier/Remotasks (Bulba Experts program for Google, March 2023-April 2024 per internal documents), spam - defined internally as 'writing gibberish, writing incorrect information, GPT-generated thought processes' - routinely got paid because per-item detection could not keep up; defenses degraded to ZeroGPT screening, crude country-level bans (Egypt, Pakistan, Kenya, Venezuela), and were circumvented by VPNs, while a black market sells verified annotator accounts (Outlier, CrowdGen, Prolific). Load-bearing

EVIDENCE: Opened Futurism's 2025-06 piece quoting Inc.'s internal-document exclusive verbatim: spammers 'could get away with just totally submitting garbage and there weren't enough people to track them down'; an allocations team once 'dumped 800 spammers' into one queue; supervisors advised to vet entries with ZeroGPT; Scale's spokesperson disputed the story claiming safeguards existed. AlgorithmWatch separately documented account-resale markets. Prior research (cited in the same reporting thread) estimated 33-46% of MTurk crowdworkers used LLMs on a text task. Community reports (Indeed/Glassdoor/Trustpilot 2025) confirm the attempter->reviewer->senior-reviewer pyramid with timed capped pay and opaque removals. SOURCE: Futurism: The AI Company Zuckerberg Just Poured $14 Billion Into... (quoting Inc. internal-documents exclusive by Sam Blum) | https://futurism.com/scale-ai-zuckerberg-incompetence | 2025-06 (documents cover 2023-03 to 2024-04) | practitioner IMPLICATION: Direct answer to taxonomy Q8: at the largest vendor, per-item content-based spam classification LOST to spam economics; identity/provenance is also compromised (account resale, VPNs). The foundation should treat wholesale-LLM/spam defense as economics-based gating (rate limits, work-history consistency, randomized deep audits, payment holds) with content detection only as a weak auxiliary signal - and assume attempter identity is an adversarial variable.

An AI data vendor acquired a data-quality startup (announced 2026-01-28) specifically for algorithms that 'flag incorrect data without a second human reviewer,' after competing bids from other data-labeling companies - signaling that algorithmic second-pass QA replacing the human review layer is now an acquisition-grade core capability for human-data vendors.

EVIDENCE: TechCrunch (2026-01-28), the startup's CEO letter, and the vendor's own announcement triangulate: 9 staff of the startup including all three MIT-PhD co-founders join the vendor's research team; rationale quotes name confident learning, data-centric AI, and LLM evaluation as the acquired capabilities; an AI data vendor serves 8 top labs at ~$300M annualized run-rate (end 2025). The startup's core product audited human-labeler output quality algorithmically. SOURCE: TechCrunch: Report on an AI data vendor's acquisition of a data-quality startup | [link withheld] [link withheld] | 2026-01-28 | practitioner IMPLICATION: The market has already voted for our thesis: augment/replace the human QA layer over the mandatory human attempter layer. Confident-learning-style label-noise detection (statistical, model-based, no second human) is a complementary channel to LLM-judge QA and should be considered in the foundation's filtering stage.

Surge AI (~$1.2B revenue 2024, ~50k contractors, bootstrapped) runs QA as real-time dashboards over gold-standard accuracy, inter-annotator agreement, and per-worker trust ratings, with flagged labels automatically reassigned to other annotators; its published AdvancedIF verifier work with Meta reports F1 0.728 versus human judgments and +13% RL gains from rubric-based rewards.

EVIDENCE: Fetched Sacra's Surge AI company report (updated 2026-04-21; figures are Sacra estimates). QA description: gold-standard accuracy + IAA + per-worker trust ratings on live dashboards; automated reassignment of low-quality labels; premium per-minute pay (~$0.30-0.40/min) as a quality incentive; vetting = domain tests + background checks + ongoing performance evaluation. Hemingway-bench used 5,000+ blind pairwise expert comparisons. Risks noted: ~12-customer concentration, CA misclassification suit alleging unpaid training time and tight task timers. SOURCE: Sacra: Surge AI revenue, funding & news | https://sacra.com/c/surge-ai | 2026-04-21 | practitioner IMPLICATION: The premium-quality market leader's QA is continuous gold-task injection + IAA + per-worker trust telemetry with automatic reassignment - i.e., population-level statistical QC, not per-item semantic review. Their own verifier tops out at F1 0.728 vs humans on instruction-following, a second independent datapoint (with an AI data vendor's 0.63-0.66) that ~0.65-0.75 F1 is the current judge ceiling on expert-graded open-ended work (taxonomy Q5).

micro1 (~$300M annualized revenue by 2026-04) embeds real-time QA verification layers directly in its annotation interface and operates a closed rubric-feedback loop: labs submit updated rubrics and corner-case prompts, which are routed to the highest-performing experts and used to retrain its vetting algorithms.

EVIDENCE: Sacra company report on micro1 (2026) describes the embedded QA layers, lab-facing dashboards with daily output and pass/fail rates, and the rubric-update routing loop; Forbes (2026-02-03) interview with CEO Ansari frames evaluation as a permanent top-level function ('how well does it work, and what are the thresholds to go into production'). Screening is via AI interviewer (Zara) at thousands of interviews/day; a trainer network and Mercor use the same AI-interview screening pattern (a trainer network: 3% acceptance, 4-stage vetting). SOURCE: Sacra: micro1 revenue, valuation & funding (+ Forbes, 2026-02-03) | https://sacra.com/c/micro1 | 2026 (Forbes 2026-02-03) | practitioner IMPLICATION: The rubric-amendment loop in taxonomy Q4/Q6 is already production practice: client rubric updates are first-class config artifacts routed to top performers, i.e., upstream compounding work is prioritized over per-item adjudication. Also confirms the industry attacks the expertise-gap root cause at intake (automated AI-interview screening + longitudinal per-worker metrics), not per-item.

Braintrust's published cost arithmetic makes the human-budget fork concrete: an LLM judge over 10,000 outputs costs ~$5-15 while a domain expert reviewing 500 of them costs ~$800-1,800 (~100x per output), which is why its prescribed architecture is deterministic checks on everything, LLM judges on everything, and humans only on flagged/low-confidence/disagreement slices plus random spot-checks of confident passes.

EVIDENCE: Fetched Braintrust article (2026-04-03) and human-review docs. The hybrid loop is explicit: trace production -> continuous deterministic + judge scoring -> flag low-confidence/scorer-disagreement -> human review -> reviewed findings become new scorers/rubric refinements -> CI regression gates. They name the biggest practitioner mistake as treating judge scores as ground truth without human validation, and note untrained reviewers with vague rubrics can produce labels worse than a decent judge. SOURCE: Braintrust: LLM-as-a-judge vs human-in-the-loop evals: When to use each | https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals | 2026-04-03 | practitioner IMPLICATION: Supports taxonomy Q6's routing answer as consensus practice: spend the single human touch on judge-flagged and judge-disagreement items, keep a randomized audit slice of confident passes, and convert every human review into a compiled-config improvement (new scorer/exemplar) rather than a one-off verdict.

Fine-tuned specialist judges have commoditized binary grounding checks: Patronus Lynx (fine-tuned Llama-3-70B) beats GPT-4o on HaluBench faithfulness detection and lifted one customer's hallucination detection from 0.375 to 0.69, while Galileo's Luna-2 (arXiv 2026-02-20) runs hundreds of per-metric LoRA adapters on one small-model backbone at >80x lower cost and >20x lower latency than LLM-as-judge at claimed parity accuracy, in production on 100M+ sessions.

EVIDENCE: Patronus launch post + SiliconANGLE (2024-07-11) give HaluBench numbers (Lynx 70B +8.3% vs GPT-4o on PubMedQA inaccuracies; 8B +24.5% vs GPT-3.5); Patronus evaluators page gives the Algomo 0.375->0.69 customer number and Gamma's 1,000+ manual eval hours saved; Percival extends this to tracing where in a multi-step agent run the failure occurred. Fetched Luna-2 arXiv abstract (2602.18583): single-token SLM evaluation, >80x cost / >20x latency reduction, 100M+ sessions and 100B tokens/month in production, $30M+ claimed annual eval savings. Vendor-reported numbers; treat magnitudes, not decimals. SOURCE: Luna-2: Scalable Single-Token Evaluation with Small Language Models (arXiv 2602.18583) + Patronus Lynx launch materials | https://arxiv.org/abs/2602.18583 | 2026-02-20 (Lynx 2024-07-11) | primary IMPLICATION: For taxonomy Q2/Q3: the 'compile to binary checks' bucket (groundedness, faithfulness, toxicity, tool-selection) can run on cheap specialized models per axis at near-zero marginal cost, reserving frontier-model judges for entailment/calibration/judgment-laden axes - a two-tier judge economy is standard 2026 practice, not an exotic design.

DeepEval's recommended progression - start with holistic G-Eval, then move to DAG (a deterministic decision tree whose nodes are narrow LLM-judge calls) 'for more control' - is the open-source codification of decomposed judging: verdicts assembled from atomic, individually-checkable steps rather than one holistic score.

EVIDENCE: DeepEval docs (metrics-introduction, metrics-dag; DAG introduced 2025-02, docs current 2026-07) describe DAGMetric as 'deterministic decision trees' for objective/mixed criteria producing deterministic scores, explicitly positioned above G-Eval when control matters, including a conversational variant for multi-turn work. Confident-AI's 2025-02-09 engineering post explains the motivation: holistic LLM scores are too noisy for objective criteria. SOURCE: DeepEval docs: DAG (Deep Acyclic Graph) metric + Confident AI engineering post | https://deepeval.com/docs/metrics-dag | 2025-02-09, docs verified 2026-07-10 | practitioner IMPLICATION: Taxonomy Q3's decomposition-granularity question has a live open-source reference implementation: per-criterion decision trees with narrow judge nodes and deterministic aggregation. Adopt the pattern (and its admission that decomposition is for objective/mixed criteria - judgment-laden axes stay holistic/G-Eval-style) rather than inventing the machinery.

The human reviewer layer at scale vendors is itself unreliable for structural/incentive reasons the AutoQA must not inherit: Outlier's attempter->reviewer->senior-reviewer pyramid runs on timed, capped per-task pay, opaque promotion/removal, uneven task allocation, and internal competition, per 2025 contributor reports across Indeed/Glassdoor/Trustpilot.

EVIDENCE: Aggregated 2025 worker reports: pay only within a capped time window (unpaid overage), removal from projects without stated reason, alleged 300-reviewer queues where ~10 got unlimited tasks, queue managers running webinars/'war rooms' as the calibration mechanism, quality undermined by contributors and reviewers 'pitted against each other'. Community-sourced and unverifiable individually, but consistent across platforms and consistent with the Inc. documents' picture of overwhelmed review capacity. SOURCE: Indeed/Glassdoor/Trustpilot Outlier reviewer reports (aggregated) | https://www.indeed.com/cmp/Outlier-Ai/reviews?fjobtitle=Reviewer&fcountry=US&ftext=outlier | 2025 (various) | community IMPLICATION: Reviewer variance (our root causes 1-3) is partly manufactured by incentive design: time-capped pay punishes careful review, and opaque consequences destroy calibration motivation. The AutoQA foundation should specify the human-touch interaction's incentive contract (untimed or accuracy-paid adjudication, visible appeal outcomes) as part of the design, not just the routing logic.

Vendor materials repeatedly attribute low inter-annotator precision to instruction/rubric defects rather than annotator failure - a major labeling platform's own remediation playbook for low precision starts with 'poor instructions, weak edge-case coverage' and prescribes guideline revision and calibration sessions before retraining or replacing people.

EVIDENCE: a major labeling platform data-factory post's scenario playbook: low precision -> better training, clearer guidelines, calibration sessions, edge-case examples; high precision + low accuracy -> refine the ground truth itself and check for systematic bias in training materials. Kili's industry guide (cited in 2025 coverage) similarly notes an annotator consistently disagreeing on one label category 'is often revealing a guideline gap', and industry estimates put RLHF preference-annotation noise above 20%. SOURCE: a major labeling platform data-factory playbook + Kili Technology annotation quality guide | https://kili-technology.com/blog/data-annotation-guide-how-to-achieve-high-quality-data-in-complex-ai-data-operations | 2024-2025, verified live 2026-07-14 | practitioner IMPLICATION: Industry operating doctrine already treats systematic reviewer disagreement as an instruction-gap signal first (taxonomy Q2's third verdict class and Q9's rubric-revision trigger): a disagreement-clustered-by-criterion feed to project owners mirrors existing vendor practice and will be legible to them. But no vendor publishes the variance attribution split, so the taxonomy's cheap variance-attribution pilot is still needed - practice assumes criteria-vagueness dominance without measuring it.

Tools & artifacts

• An AI data vendor's agent-judge: Open-sourced reactive agent-as-judge verifier (OpenHands SDK) that runs in the rollout's own environment, opens files/tools at inference time; paired with the vendor's banking agent benchmark (HuggingFace) and the banking verifier meta-eval (3,204 expert labels, full release pending). | [link withheld] [link withheld] | Reference architecture for evidence-closure-correct judging and for meta-evaluating our judge against adjudicated expert labels before trusting it.
• LangSmith Align Evaluator + annotation queues: Built-in workflow: human-labeled reference set (>=20 balanced examples) -> alignment score (percent match) -> evaluator playground iteration; corrections folded into judge prompt instructions. | https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback | The de facto industry calibration loop to improve on (its percent-agreement metric is the weakness).
• Braintrust human review + Loop: Span-level human review UI with judge-vs-human scores side by side, flagged-trace-to-test-case loop, CI regression gating, and an eval-workflow agent (Loop, 2025-11). | https://www.braintrust.dev/docs/annotate/human-review | Model for converting each human touch into permanent compiled-config assets (scorers, datasets) rather than one-off verdicts.
• Patronus Lynx + Percival: Open-weight fine-tuned faithfulness judge (binary grounded/ungrounded + reasoning trace) and an agentic debugger that localizes failures in multi-step traces. | https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model | Cheap specialist model for our 'compile to binary checks' grounding lane; Percival pattern for localizing which claim/step failed.
• Galileo Luna-2 + CLHF: SLM backbone with per-metric LoRA adapter heads (hundreds of metrics on one GPU, >80x cheaper than LLM-as-judge; arXiv 2602.18583) plus Continuous Learning with Human Feedback that adapts metrics from expert corrections. | https://arxiv.org/abs/2602.18583 | Economics template for running many per-axis checks on every item; CLHF is prior art for correction-driven judge adaptation.
• DeepEval DAG / Conversational DAG: Open-source deterministic decision-tree metrics whose nodes are narrow LLM-judge calls; positioned above holistic G-Eval when control is needed. | https://deepeval.com/docs/metrics-dag | Open reference implementation of decomposed per-criterion judging with deterministic aggregation (taxonomy Q3).
• Cleanlab: Confident-learning software that flags incorrect human labels algorithmically without a second human reviewer. | [link withheld] [link withheld] | Statistical label-noise detection as a judge-independent third QA channel alongside LLM judging and human audit.
• a trainer marketplace: Marketplace exposing per-worker longitudinal quality telemetry (assessment scores, interview transcripts, rework rate, efficiency) to clients staffing data teams. | https://a major labeling platform.com/services/a trainer network-connect | Prior art for per-attempter trust telemetry as a routing and gating input in our foundation.

Disagreements

• Judge-vs-human ceiling: LangChain marketing materials claim LLM judges reach ~85% alignment with humans, exceeding ~81% human-human agreement (an MT-Bench-style chat-preference number), while an AI data vendor's banking verifier meta-eval shows the best 2026 verifiers at F1 0.633-0.664 against practicing-expert labels (humans at 89.5% agreement) and Surge's AdvancedIF verifier at F1 0.728. The 'judges already beat humans' claim holds only for casual preference tasks, not expert-graded artifact-heavy work - the ceiling is domain-dependent by ~30 F1 points.
• Agreement statistic: a major labeling platform publishes chance-corrected Krippendorff's alpha thresholds as its quality standard, while LangSmith/Braintrust standardize on raw percent agreement over tiny reference sets. The industry has not converged, and the eval-tooling side uses exactly the statistic our taxonomy proposes banning.
• Is human review the gold standard? An AI data vendor treats dual-coded, adjudicated practicing-expert labels as ground truth for meta-evaluation; Braintrust explicitly warns human review is 'not automatically a gold standard' (untrained reviewers, vague rubrics, fatigue can underperform a decent judge). Reconcilable - adjudicated expert panels vs ad-hoc reviewers - but the sources take opposite rhetorical stances.
• Vendor QA claims vs observed integrity: a major labeling platform/Surge/Scale market rigorous multi-layer QA, SLAs, and 3% acceptance rates, while internal Scale documents (via Inc./Futurism) show spam paid at scale for 11 months, ZeroGPT as a stopgap, and country-level bans; Scale's spokesperson disputes the reporting ('filled with so many inaccuracies'). Marketing descriptions of QA pyramids should not be taken as evidence they perform as described.
• Content-based vs economics-based spam defense: Patronus/Galileo/detection vendors implicitly sell content-based classification as sufficient, while the Scale/Outlier record and the annotator-account black market (AlgorithmWatch) show content detection losing to spam economics at scale; an AI data vendor's purchase of a data-quality startup (statistical label auditing) is a third position - detect via label statistics rather than either content classifiers or human re-review.

Gaps

• No data vendor publishes actual achieved inter-reviewer reliability numbers (e.g., Krippendorff alpha values on common item sets) - only target thresholds. The one hard human-agreement number found (an AI data vendor's 89.5%, dual-coded adjudicated subset) is from a benchmark paper, not production QA reporting.
• No public data anywhere on false-agreement rates (judge and attempter both wrong vs expert ground truth), on per-project gold-set sizes/refresh rates at data vendors, or on judge-model version pinning / golden-set regression gates at data vendors (taxonomy Q1, Q9). Eval-tooling vendors support regression testing generically but no vendor documents pinning policy for annotation-QA judges.
• No published evidence on whether QA feedback changes attempter behavior longitudinally (repeat-error rates per criterion, revision acceptance) - no vendor reports feedback-efficacy metrics, and no controlled feedback-vs-verdict-only comparison was found (taxonomy Q7).
• Scale AI's current (2025-2026) production auto-QA internals are undocumented publicly except via leaks and contributor folklore; Turing publishes essentially nothing concrete; Invisible's public trace is limited to a job post showing AI QA Trainers build rubrics/pass-fail criteria and regression suites. Mercor's QA is visible only through reviewer-hiring materials emphasizing consistency screening.
• Could not verify a major labeling platform data-factory blog's exact publication date (undated on page; content verified live 2026-07-14). Sacra figures for Surge/micro1 are analyst estimates, not audited. The Inc. internal documents were verified only via Futurism's direct quotations (Inc. itself 403s).
• No vendor was found admitting or measuring feedback-loop gaming (attempters overfitting the QA judge's decision boundary) or monoculture drift from judge feedback - taxonomy Q8's core dynamic is unstudied in public industry sources.

Verifications

• CLAIM: a major labeling platform's production QA stack is double-blind independent labeling, automated discrepancy detection, expert adjudication, and rework, with a fine-tuned-LLM similarity judge that only FLAGS below-threshold responses for human review (never autonomously fails them), Krippendorff's alpha as the published reliability bar (>=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor), and a multidimensional per-worker 'trust score' that routes complex tasks to high-trust raters. VERDICT: confirmed | Fetched the source directly (https://a major labeling platform.com/blog/inside-the-data-factory-how-a major labeling platform-produces-the-highest-quality-data-at-scale, published September 12, 2024, still live 2026-07-14). Every element of the claim checks out against the text: (1) 4-step double-entry flow - "Two independent labelers perform the same task without knowledge of each other's work," results automatically compared for discrepancies, expert reviewer makes final determination, rework with feedback; (2) LLM judge fine-tuned "on a dataset of high-quality, expert-verified responses," scores similarity, and its described action is "Flagging responses that fall below a certain similarity threshold for human review" - flag-only, humans handle challenging cases; (3) exact Krippendorff alpha table: >=0.80 satisfactory, 0.67-0.79 moderate, <0.67 poor, preferred over Cohen's/Fleiss' kappa for versatility/robustness (2+ raters, missing data, mixed data types); (4) a trainer network trust score is explicitly "multidimensional" (historical accuracy, consistency, completion rate, complex-task handling) with "High-trust AI trainers may be prioritized for more critical or complex tasks." Ancillary evidence items (3% acceptance, pay-only-for-SLA-quality, Kendall's tau/ELO, F1/Matthews) also present. Supersession search found no newer official a major labeling platform post revising the trust score or QA stack - the Sept 2024 post remains the canonical description; 2026 third-party reviews add worker-side complaints (inconsistent review outcomes, unpaid evaluation phases) but do not contradict the claimed process. Two wording nuances, neither rising to a correction: the post's own term is "double-entry" (the mechanics match "double-blind" - independent, mutually unaware labelers); and the post describes the LLM judge only as flagging for human review - it does not affirmatively state "never autonomously fails," though nothing in the post says it auto-fails work.
• CLAIM: An AI data vendor's banking verifier meta-eval (3,204 practicing-banker pass/fail labels, 89.5% inter-annotator agreement on a dual-coded adjudicated subset) shows the best automated verifier they could build (the vendor's agent-judge, a reactive agent-judge running in the same environment as the work) reaches only F1 0.633-0.664 against expert labels on artifact-heavy tasks, while hitting F1 0.951 on a simpler stateful-tools benchmark, and that verifier architecture (evidence view + evidence path) matters more than judge model choice. VERDICT: confirmed | Fetched [link withheld] [link withheld] directly. Every element of the claim checks out against the source: (1) dated May 27, 2026; (2) the banking verifier meta-eval is explicitly a meta-eval of verifiers built on 21 tasks from the vendor's banking agent benchmark with 3,204 pass/fail judgments from practicing bankers; 89.5% inter-annotator agreement on a dual-coded subset, with disagreements adjudicated before inclusion; (3) the vendor's agent-judge is a reactive agent-as-judge running in the same rollout environment (shared filesystem, interpreter, MCP tools), open-sourced on the OpenHands SDK; (4) F1 0.633-0.664 on the banking meta-eval (artifact-heavy) vs F1 0.951 on OpenClaw, their internal stateful-tools productivity benchmark (200-criterion sample, +7.5 over next-best); (5) architecture (evidence view: text/snapshot/live env; evidence path: fixed vs inference-time) beats model choice - within GPT-5.4 family, agent-judge/Nano beats Archipelago/GPT-5.4 by 9.5 F1; (6) cheapest agent-judge config (~$42, GPT-5.4 Nano) beats best baseline (Archipelago/Gemini 3 Pro, F1 0.604, ~$422) by ~3 F1 at ~one-tenth cost; (7) the post prescribes meta-eval of verifiers against adjudicated expert labels before trusting them for scoring/rewards. Supersession check (two searches, July 2026): no newer contradicting work found; the full agent-judge paper and the banking meta-eval dataset release are still pending, so numbers could later be refined, but as of today the post is the authoritative source and the claim matches it. Minor nuance only: the 0.951 benchmark is named OpenClaw and is internal (not public), which the claim's phrasing ("simpler stateful-tools benchmark") accurately reflects.
• CLAIM: The 2026 eval-tooling vendors (LangSmith, Braintrust, Galileo) have converged on one judge-calibration pattern - small human-labeled reference sets (LangSmith: start at ~20 balanced examples; Braintrust: 50-100 per scoring dimension) with an 'alignment score' defined as RAW PERCENT AGREEMENT with the human expert, iterate the judge prompt against it, and route low-confidence or judge-disagreement items to humans - and none of them ships chance-corrected statistics. VERDICT: partially_confirmed | VERIFIED: (1) LangSmith doc (https://docs.langchain.com/langsmith/improve-judge-evaluator-feedback) says exactly what was reported - annotation queue -> reference dataset (~20 examples minimum, balanced 0/1) -> Evaluator Playground -> alignment score quoted verbatim as "the percentage of examples where the evaluator's judgment matches that of the human expert" -> cluster misalignments into failure modes as prompt instructions. No kappa/alpha anywhere; doc is undated. (2) Braintrust article (https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals, confirmed 2026-04-03) matches on all six checked points: 3-tier stack, 50-100 labels per scoring dimension, multi-judge disagreement routed to humans, "silent overconfidence" named as the most dangerous failure mode, human-review-not-gold-standard warning (reviewer fatigue, vague rubrics), and agreement tracked as raw agreement rate with no chance-corrected statistic. (3) Galileo CLHF exists and auto-adapts metrics from expert corrections (feedback -> LLM-generated few-shot examples appended to the metric prompt, capped at most recent 15, claimed 20-30% accuracy gain).

REFUTED/OVERSTATED PARTS: (a) The "convergence on raw percent agreement" framing fails for Galileo. Galileo's own guidance - "How to Calibrate Your LLM Judge With Human Annotations" (https://galileo.ai/blog/calibrate-llm-judge-human-annotations, May 15/Jul 6 2026, i.e. NEWER than the cited 2025-02-10 CLHF post) - explicitly recommends AGAINST raw percent agreement (imbalanced-label example: 90% raw agreement, kappa ~ -0.05) and prescribes Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha with a recalibrate-when-kappa<0.60 trigger. Galileo also has a dedicated Cohen's Kappa explainer (https://galileo.ai/blog/cohens-kappa-metric, 2025-03-12). (b) Galileo's CLHF is also not the same pattern as LangSmith/Braintrust - it has no reference set + alignment score loop; it is few-shot auto-improvement, so the three-vendor "one pattern" claim overreaches. (c) The narrower claim "none of them SHIPS chance-corrected statistics" as an in-product feature survives verification: Galileo's kappa content points users to scikit-learn/statsmodels/R for computation, and its product features (Autotune, Annotations, Signals, Experiments, CLHF) are not described as computing kappa; LangSmith and Braintrust expose only percent-agreement metrics. But the claim as worded implies the vendors don't even acknowledge chance correction, which is false for Galileo. SUPERSESSION CHECK: newer relevant work exists - arXiv 2606.00093 (Rao & Callison-Burch, 2026-05-25, "Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why") argues for reporting kappa alongside accuracy, and LangChain published an Align Evals calibration resource (2026-03-10) still framed around human-correction agreement. CORRECTED: LangSmith and Braintrust have converged on a judge-calibration pattern of small human-labeled reference sets (LangSmith: ~20 balanced examples; Braintrust: 50-100 per scoring dimension) with alignment measured as raw percent agreement with human experts, prompt iteration against misalignments, and routing of low-confidence/judge-disagreement items to humans; neither ships a chance-corrected statistic. Galileo differs: its CLHF (2025) turns expert corrections into few-shot prompt examples (max 15) rather than a reference-set alignment score, and its newer 2026 calibration guidance explicitly rejects raw percent agreement in favor of Cohen's kappa / Fleiss' kappa / Krippendorff's alpha (recalibrate below kappa 0.60) - though it directs users to external tools (scikit-learn, R) rather than computing these in-product. So the "no chance-corrected stats shipped in-product" observation holds across all three, but there is no three-vendor convergence on raw percent agreement as the alignment definition.