How CDF scores documentation quality — the complete explanation

Audience: anyone who needs to publicize, defend, or understand the CDF benchmark numbers — marketing, sales engineering, and technical reviewers. This is the one document that puts the whole scoring story together; the other three files (METHODOLOGY.html, RUBRIC.html, FABRICATION-DETECTION.html) are the detailed sources it draws on.

TL;DR — what the number actually means

CDF turns COBOL programs into Program Logic Manuals (PLMs) — maintenance documentation — and reports the result as two numbers, side by side:

98.7 average quality (0–100) · 0 fabricated entities

Those numbers are meaningful because of how they are produced and graded, not because a model was asked to rate itself. Three properties do the work, and every claim in this repo traces back to one of them:

1. Grounded generation — the factual content (layouts, offsets, control flow, call graph) is generated deterministically and transcribed verbatim; the AI writes prose around fixed facts, it does not invent them.
2. Independent grading — a different AI vendor grades the output against the source, so no vendor grades its own work.
3. Repeated, not cherry-picked — every program is generated multiple times and graded multiple times; the reported number is built from per-document medians, and every low individual score is inspected on the record.

If someone asks "why should I trust 98.7?", the answer is those three properties — not the number.

1. Two numbers, never one

Quality and fabrication are different kinds of claim, so they are reported separately and never blended:

• Quality (0–100) — how well the documentation supports real maintenance work (§3–§4).
• Fabrications (a count) — how many named entities in the documentation don't exist in the source. This is a hard pass/fail gate, not a scored dimension (§5).

Folding them into one average would let a fluent-but-wrong document hide a hallucination behind a high score. Regulated buyers care most about exactly the thing an average would bury, so it stays its own number.

2. Why the score is meaningful: grounded generation

The pipeline has two stages:

COBOL source ──► Stage 1: deterministic extraction ──► STB (fact inventory)
                                                         │
                          ┌──────────────────────────────┘
                          ▼
        deterministic "spine" (fixed facts as tables)
                          │
                          ▼
   Stage 2: AI writes narrative AROUND the fixed facts ──► PLM

• Stage 1 (no AI): independent parsers decompose each program into a Structured Technical Baseline (STB) — a machine-readable inventory of every verifiable fact (data structures, control flow, I/O, CICS commands, JCL, call graph). Because it is deterministic, every STB fact is reproducible and auditable back to specific source lines — and, importantly, it is not assumed infallible: the adversarial grader (§3) checks the documentation against the original source, so if an extraction gap ever produces a wrong or missing fact, it surfaces as a caught, classified defect and is fixed in the extractor, for every program — never patched in one document.
• The spine: from the STB, CDF builds a deterministic fact scaffold — record and COMMAREA layouts with byte offsets and per-field read/write direction, control-flow structure, the inbound/outbound call graph — as fixed tables.
• Stage 2 (AI): the model transcribes those tables verbatim and writes only the surrounding narrative (business purpose, flow, maintenance guidance), grounded in the scaffold and in specific source lines.

Why this matters for the score: the numbers and layouts in a PLM are transcribed deterministic values, not figures a model reasoned its way to. So when the grader checks them, it is verifying transcription fidelity, not adjudicating the model's arithmetic — a wrong byte offset is a caught defect, not a matter of opinion. This is the foundation the "grounded" and "zero-fabrication" claims rest on. (Detail: METHODOLOGY.html §3.)

3. Why the score is credible: independent, adversarial grading

The generator and the grader are different AI vendors on purpose:

• Generation — an Anthropic Claude model (Opus class).
• Grading — an OpenAI GPT-5.5 model, in a separate harness.

This is the single most important control against the classic way an AI benchmark inflates itself: a model (or its sibling) grading output that shares its own blind spots. CDF tested this rather than assuming it — two same-vendor graders agreed only moderately (Cohen's κ ≈ 0.48), and in at least one real case three same-vendor graders passed a document that contained a genuine contradiction, which a cross-vendor grader caught (~8–9 points of false confidence avoided).

The grader gets the PLM, the original source, the STB, and the deterministic fact scaffold (spine) — the full basis the document claims to be grounded in — and is run adversarially: its job is to find contradictions and unsupported claims, not to be charitable. (Detail: METHODOLOGY.html §4.1–4.2.)

4. What "quality" measures: the archetype rubric

Different program types need different documentation, so scoring is archetype-specific. Each program is classified into one of eight archetypes, then scored on six maintenance-scenario tasks — each a real question a programmer brings to the doc ("trace what this job updated"; "which PF-key does what"). Each task is scored coverage × quality × 100 (a doc must be both complete and correct to score high), and the six are combined by archetype-specific weights.

The weights encode what actually matters for that program type — for example:

The full weight tables for all eight archetypes are in RUBRIC.html. Two design choices there are worth repeating to a skeptic: appropriately-qualified inferences and source-true statements of absence are scored as correct (humility is not penalized), and a separate diagnostic (grounding_coverage) measures citation density but never feeds the score.

5. The fabrication hard-gate

Before any quality task is scored, every named entity in the PLM — field, paragraph, file, copybook, CICS command/map, CALL'd program — is resolved against the source and STB. If any name doesn't exist there, the document is rejected and re-processed, not patched. Plausible-but-wrong (writing ACCT-BALANCE for ACCT-BAL) is treated as a hard failure, because in maintenance that is the dangerous case. Legitimate aliases for the same resource (a file's JCL DDNAME vs its CICS dataset name) are not fabrications. (Detail: FABRICATION-DETECTION.html.)

This gate is why "zero fabrications" is a structural property of what ships, not a lucky statistic.

6. From one document to a corpus number: repeated, audited

Neither generation nor grading is treated as deterministic, so neither runs once:

• Multiple generations — each program is documented independently 3 times, so the number reflects the pipeline's typical output, not its best attempt. Each generation is stamped with the exact pipeline version that produced it, and the run manifest records that stamp per generation — so any version differences between generations are disclosed, never averaged away silently.
• Multiple evaluations — each document is graded independently 3 times; the score of record is the median of the three (robust to one anomalous grading roll).
• No hidden failures — every individual grading roll below 90 on any task is individually triaged, even when the program's median passes, and classified on the record as a real defect (fixed) or a grading artifact (documented). A good median is never allowed to bury a one-in-three failure.

Aggregation is transparent: per-generation median-of-3 gradings → per-program figure = the mean of the three generation medians (per-generation medians recorded in the run ledger) → corpus mean, with no program dropped to improve the total. (Detail: METHODOLOGY.html §4.3.)

7. How to talk about it (defensible phrasings)

Say:

• "Two independent facts: 98.7 average quality and zero fabricated entities."
• "Graded by a different AI vendor than the one that wrote it — no model grades its own work."
• "The facts are extracted deterministically; the AI writes explanation around them, it doesn't invent them."
• "Each program graded three times; we report the median and hand-audit every low roll."
• "Scored against public, archetype-specific rubrics — the weights are published."

Don't say:

• "Zero hallucinations." Say "zero fabricated entities" — that is the gate's exact, provable scope; unqualified "zero hallucinations" over-claims.
• "Human experts graded it" — unless METHODOLOGY.html §4.5 is resolved to disclose a real human-SME step. The quality score is an independent AI grader (with deterministic fabrication/entity checks underneath).
• "100%" or "perfect" about the corpus — the rubric is designed so an honest corpus aggregate lands ~98–99, not 100. (Individual programs do legitimately score a 100 median — simple programs documented completely — so per-program 100s in the published table are expected, not suspicious; it is a corpus-wide perfect score that would be a red flag.)

8. What a third party can and cannot reproduce

• Public: the corpus (AWS CardDemo, vendored at application version 1.0.0), the eight archetypes, and the rubric weights (RUBRIC.html). A reviewer can re-run the published rubric with their own grader.
• Proprietary: the CDF pipeline (Stage-1 parsers, the spine) and the specific model versions. So a third party can verify what was measured and how it was scored, but cannot reproduce the pipeline itself. This limit is stated plainly rather than hidden.

9. Before this ships — open confirmations

All pre-publication confirmations are now resolved against the final full-42 publication run (recorded in the ratification package, 2026-07-29).

Resolved (filled in the source docs):

• Archetype taxonomy & mixed-behavior rule — the eight definitions and the dominant-behavior, single-label classifier are stated (METHODOLOGY.html §2).
• Spine disclosure depth — the conceptual minimum only (deterministic tables transcribed verbatim, narrative grounded in them); the mechanism stays proprietary.
• Models — Anthropic Claude Opus (generation) and OpenAI GPT-5.5 (grading, selected as the strongest adversarial judge over Claude Opus and Gemini 3.1 Pro).
• Grader-agreement evidence — the same-vendor κ = 0.48 (decorrelation study, 2026-05-09/10) and the "three same-vendor graders missed a control-flow contradiction the cross-vendor grader caught" case both trace to recorded runs.
• Human review — early informal human-SME spot-checks are disclosed; the absence of a recent, systematic SME calibration is flagged as a tracked open item (a stratified sample with published κ / α), not claimed as done.
• Aggregation of record — a flat mean of per-program scores (each program's score = the mean of its per-generation medians), reported with its standard deviation; explicitly not archetype-weighted.
• Published bar — aggregate ≥ 95 with no task below 90 on the score of record; not published until every program either meets that floor or carries an individually-audited, recorded grading-artifact exception (a documented grader error on a correct document — never a document defect; see §6, "no hidden failures").
• Corpus source — public CardDemo, vendored (version 1.0.0), stable; no commit pin required.

Resolved against the publication run:

1. The headline number — RESOLVED: 98.7 (std dev 0.9), the flat mean of the 42 per-program scores from the full-42 run under the §4.3 protocol (ratification package, 2026-07-29).
2. Full-42 coverage — RESOLVED: the final wave scores all 42 CardDemo programs (a roster-census guard in the generation driver verifies the full 42-program roster, including COUSR01C, before any run counts as full-corpus).
3. Model provenance — RESOLVED: --model opus is not a single point-version, so it is recorded per generation from the run-session transcripts (attached to the run manifest). Generation spans claude-opus-4-7, claude-opus-4-8, and claude-opus-5 across the vintage chain, disclosed as a mix rather than a single version (see METHODOLOGY.html §3).
4. Module provenance (CardDemo sub-projects) — RESOLVED: the corpus composition is disclosed (29 core app/cbl programs + 13 optional extension-module programs, e.g. COACCT01 from app/app-vsam-mq/; recorded per program in the run manifest — see METHODOLOGY.html §1). Say "the CardDemo repository's full COBOL program set," not "every core-application program."

Sources: METHODOLOGY.html (how it's produced and graded) · RUBRIC.html (what's scored, all weights) · FABRICATION-DETECTION.html (the hard-gate).