Methodology

This document describes exactly how the CDF CardDemo benchmark was produced, so that the results can be understood, critiqued, and — as far as the public corpus allows — reproduced.

Reproducibility note. CardDemo is public; CDF itself is a commercial pipeline. This document is written so a reader can verify what was measured and how it was scored even though they cannot run the CDF pipeline themselves.

1. The corpus

• Source: AWS CardDemo, public main branch.
• Source stability: the corpus is the public CardDemo COBOL source (upstream application version 1.0.0), vendored into the CDF repository so the exact bytes scored are frozen. Because the public source is stable, a reader can clone the public repository and obtain the same programs; no commit pin is required for reproducibility.
• Scope: the full CardDemo COBOL program set — 42 programs: the core application's COBOL programs (app/cbl, 29) plus the optional extension modules shipped in the same repository (e.g. the VSAM/MQ and DB2 sample suites, 13), inventoried per program in the run manifest — spanning the eight archetypes in §2. No programs are excluded; non-COBOL artifacts (JCL, BMS, copybooks) are inputs to extraction, not separately scored units. The per-archetype breakdown is: Batch File Maintenance 11, CICS Inquiry 8, CICS Data Entry 7, CICS Menu 6, CICS Other 4, Batch Report 2, Called Subprogram 3, Batch Utility 1. (Archetype counts follow the rubric applied at scoring; one program, PAUDBUNL, was classified batch-report by the pipeline classifier but scored under the batch-file-maintenance rubric — a recorded assignment mismatch, noted in RESULTS.html.) The published score comes from the full-42 run under the §4.3 protocol (ratification package, 2026-07-29); no partial-corpus figure is published as the 42-program result.

2. Archetypes

Each program was classified into one of eight archetypes before documentation, because different program types have different maintenance patterns and are scored against archetype-specific expectations (see RUBRIC.html).

A program with mixed behavior receives a single archetype by dominant behavior: the classifier scores the program against each candidate archetype's signal profile (weighted signals, with a violated hard constraint disqualifying that candidate) and assigns the highest-scoring one. Where the top two scores are close, the case is flagged for review but still receives one label — e.g. a CICS program that both displays and updates data lands wherever its signals weigh heaviest, not in two archetypes at once.

3. The CDF pipeline

Stage 1 — Deterministic fact extraction (no AI)

Independent parser layers decompose each COBOL program into a Structured Technical Baseline (STB): a machine-readable JSON inventory of every verifiable fact in the source, including:

• Data structures and copybook hierarchies
• Control-flow graphs
• I/O operations and file definitions
• CICS commands and BMS screen layouts
• JCL job configurations
• Rule-based business-logic facts

Because this stage is deterministic, every STB fact is reproducible and auditable back to specific source lines. It is not assumed infallible, though: the adversarial grader (§4) checks the documentation against the original source, so an extraction gap that ever produced a wrong or missing fact surfaces as a caught, classified defect and is fixed in the extractor, for every program — never patched in a single document. That is what lets the STB serve as the reference Stage 2 is graded against. Extraction is a three-layer deterministic chain — (1) a base COBOL parse, (2) type-aware augmentation for BMS/DB2/IMS/JCL artifacts, and (3) a unified merge that adds cross-program lineage — implemented by roughly fifty specialized parsers, with every fact auditable to specific source lines.

Illustrative STB fragment:

{
  "program": "COTRN01C",
  "archetype": "cics_inquiry",
  "paragraphs": 9,
  "cics_commands": ["SEND", "RECEIVE", "READ", "RETURN"]
}

Stage 2 — Constrained AI synthesis (grounded in a deterministic fact scaffold)

Between the STB and the AI sits a deterministic fact scaffold (internally, the "spine"). The structured facts a maintenance programmer relies on as tables — record and COMMAREA layouts with byte offsets and per-field read/write direction, control-flow structure, and the inbound/outbound call graph — are generated deterministically from the STB, not by the AI. The model transcribes those tables verbatim and writes only the surrounding narrative (business-logic explanation, flow description, maintenance guidance), grounded in the scaffold and in specific source lines.

The distinction matters for how the scores should be read: the model authors prose around fixed facts; it does not compute or invent the numbers, layouts, or field directions. A byte offset or a field-direction label in a PLM is a deterministic value that was transcribed, not a figure the model reasoned its way to — which is why an error in one is a caught defect (see §4.2), not a judgment call. Archetype-aware specifications determine the structure and emphasis of each PLM.

Outputs per program:

• Narrative business-logic explanation
• Step-by-step program-flow description
• Maintenance guidance and impact analysis
• Line-number citations for every claim
• Archetype-specific documentation structure
• A production-ready Program Logic Manual (PLM)

The models are specific and deliberately chosen, not interchangeable. Generation uses an Anthropic Claude Opus-class model. Grading uses OpenAI GPT-5.5, adopted as the adversarial grader after cross-vendor testing showed it the strongest adversarial judge among the candidates evaluated — including Anthropic Claude (Opus) and Google Gemini 3.1 Pro: in documented cases GPT-5.5 caught genuine defects (e.g. a fabricated field reference) that both Claude adversarial and Gemini 3.1 Pro scored clean. The generation model is requested as the Opus alias rather than a pinned point-version, so the exact model that ran is recorded per generation from the run-session transcripts (attached to the run manifest). Across the vintage chain that feeds the published scores, generation spans claude-opus-4-7, claude-opus-4-8, and claude-opus-5 — disclosed as the mix that actually ran, the same per-generation provenance discipline applied to the per-generation vintage stamps, rather than approximated to a single version.

The hallucination hard-gate

Every named reference in the output — field names, paragraph names, CICS commands, file names, dataset names — is cross-checked against the source and the STB. An output containing a name that does not appear in the source is rejected, not patched. This is the mechanism behind the "zero fabricated entities" result. Full definition in FABRICATION-DETECTION.html.

4. Scoring: who grades, how, and what keeps it honest

The credibility of a benchmark rests almost entirely on this section, so it is written to be over-disclosed rather than under-disclosed. Three design choices carry the weight: the grader is a different AI vendor than the generator (no vendor grades its own work), every claim is scored against deterministic source facts rather than against the grader's own judgment, and each number is the product of repeated independent generation and repeated independent scoring — not a single lucky run.

4.1 Grader independence — the generator and the grader are different vendors

The two stages of the pipeline are run by models from different vendors on purpose:

• Generation (writing the PLM) is done by an Anthropic Claude model (Opus class), grounded in the STB and source as described in §3.
• Scoring (grading the PLM) is done by an OpenAI GPT-5.5 model, run through a separate scoring harness with no connection to the generation run.

This is a deliberate control against the single most common way an LLM benchmark inflates itself: a model — or a sibling model from the same family — grading output that shares its own blind spots. We tested this rather than assumed it. In an internal decorrelation study, the same generator-family model scored under two prompt framings agreed only moderately — Cohen's κ = 0.48 (decorrelation study, 2026-05-09/10). Consistent with that moderate agreement, in a separately documented case three same-vendor Claude graders all credited a PLM containing a genuine control-flow contradiction that only the cross-vendor grader caught — a gap of roughly 8–9 points of false confidence that would otherwise have reached a customer unflagged. Cross-vendor grading is therefore not a formality; it is the layer that caught a real defect the same-vendor layer missed.

The grader receives, for every program: the PLM under test, the original COBOL source, the STB (the deterministic fact inventory from §3), and the deterministic fact scaffold (spine) — the full basis the document claims to be grounded in. It is instructed to cross-check every substantive claim against the source/STB and is run under an adversarial framing — its job is to find contradictions, over-claims, and unsupported inferences, not to be charitable. The same per-archetype rubric (see RUBRIC.html) is applied regardless of which grader vendor runs it; the only thing that changes between graders is the vendor, so any grader-vendor bias is isolable rather than baked in.

4.2 The grade is anchored to source facts, not to the grader's opinion

Scoring is grounded, meaning the rubric rewards claims that are verifiable against the source and penalizes claims that are not. Because Stage 1 (§3) is deterministic and every STB fact is auditable back to source — and re-checked against the source by the grader rather than assumed infallible (§3) — the grader's role is closer to an auditor checking a document against a system of record than to a critic offering a subjective impression. Because the PLM's structured facts are transcribed from the deterministic scaffold (§3, Stage 2) rather than authored by the model, the grader is checking transcription fidelity and narrative grounding — not adjudicating the model's arithmetic. A wrong byte offset or a mislabeled field direction is therefore a caught defect, not a matter of the grader's opinion. A PLM that qualifies an unverifiable statement as an inference scores higher than one that asserts it as fact — appropriate epistemic humility is scored as correctness, which is also why well-formed PLMs land around 98–99 rather than a suspicious 100 (see RUBRIC.html, "Why perfect scores are rare").

Underneath the LLM grader sit deterministic, non-AI checks that do not depend on any model's judgment at all: an entity/fabrication checker that resolves each named field, paragraph, file, and CICS command back to the source, and a provenance checker that hard-flags any claim contradicting the STB enumeration it cites. These are constructed checks — no model, no circularity — and they back the zero-fabrication claim independently of the quality score (§4.4).

4.3 Repeated generation and repeated scoring — the number is a median, not a cherry-pick

Neither generation nor scoring is treated as deterministic, so neither is run only once.

• Multi-generation. Each program is documented independently multiple times from the same inputs (the published run uses 3 independent generations per program), so the reported number reflects the pipeline's typical output, not its best single attempt. A program's score is the mean of its three per-generation medians; generation variance across the runs is recorded per program in the run ledger.
• Multi-evaluation. Each generated PLM is then scored independently 3 times by the cross-vendor grader. A program's score of record is the median of the three scores, which is robust to a single anomalous grading roll in either direction.
• No hidden failures. Consistency is not allowed to mask a one-off failure: every individual scoring roll that falls below 90 on any rubric dimension is individually triaged on the record, even when the program's median is comfortably passing. A strong median cannot bury a one-in-three crater — each such roll is classified as either a real defect (returned to the pipeline) or a grading artifact (recorded, never silently discarded).

Aggregation is transparent and every program's median is carried into it — no program is dropped to improve the total. The corpus figure is a flat mean of the per-program scores (each program weighted equally; not an archetype-mean → overall-mean rollup, which would over-weight the small archetypes), reported with its standard deviation so dispersion is visible. For the published full-42 run this is a mean of 98.7 with a standard deviation of 0.9 (n = 42).

4.4 What is reported, and separately

The corpus result is deliberately reported as two independent facts, never folded together:

1. a quality score (0–100, aggregated as in §4.3), and
2. a fabrication count, tracked as a hard pass/fail gate before scoring (§4.2, RUBRIC.html).

Folding fabrication into the quality score would let a fluent-but-wrong PLM average away a hallucination. Keeping them separate is why the headline is stated as "98.7 quality and zero fabrications," not a single blended number.

4.5 Threats to this approach, stated plainly

• The grader is still an AI, not a panel of human COBOL SMEs. Cross-vendor + adversarial + source-anchored grading substantially decorrelates it from the generator, and the deterministic checks (§4.2) are model-independent — but the quality score is an LLM-as-judge score, and we state that rather than implying human review at corpus scale. Human-SME spot-checks were performed early in development — informally, on a small number of PLMs (the review that drove the "zero entity fabrications" rather than "zero hallucinations" scoping) — and again later as quality visibly improved. No recent, systematic human-SME calibration has been run against the current-vintage corpus, and closing that gap is a known open item: a stratified SME sample with a published inter-rater agreement (Cohen's κ / Krippendorff's α) against the AI grader. It is tracked, not yet a completed control, and we do not claim it as one.
• Judge consistency is measured, not assumed. The multi-roll median + any-roll-below-90 audit (§4.3) is our control for grader variance; measured run-to-run grading variance on this corpus is a ~2.3-point median same-document spread across rolls, well inside the reported dispersion.
• Judge acuity vs. document size. Grader consistency and recall were measured on this corpus as a whole, not stratified by context size — a handful of programs produce much larger grading contexts than the median, and per-size recall parity has not yet been separately established (it is a tracked follow-up). The fabrication/grounding claims do not rest on this: they are corroborated by the deterministic, model-independent checks (§4.2) regardless of grader context size.
• Provenance of the headline number. The published corpus score must be the full-42 run under exactly the §4.3 protocol (3 generations × 3 evaluation rolls, per-generation median-of-3), and the run's identity is recorded with the number: the published figure is the full-42 run selected by the vintage-chain convention recorded in the ratification package (RATIFICATION_PACKAGE_20260729, dated 2026-07-29) — newest clean vintage per program×gen, scored under the full §4.3 protocol (3 generations × 3 evaluation rolls, per-generation median-of-3). No number scored under a lighter protocol is published as if it met this one.
• Reproducibility ceiling. The corpus and rubric are public; the pipeline and the specific model versions are not. A third party can re-derive what was scored and how, and can re-run the public rubric with their own grader, but cannot reproduce the CDF pipeline itself.

5. Threats to validity

A benchmark that does not name its own limits is marketing. These are the limits of this one.

Single corpus. The published score comes from one application. CardDemo is a well-formed, representative commercial workload spanning batch, CICS online, VSAM and DB2 sample suites, but it is one codebase, and these numbers are not a claim about every COBOL dialect, vintage or in-house coding style. The pipeline is corpus-agnostic and has been run beyond CardDemo, including against IBM's public GenApp insurance application and additional DB2 and VSAM sample programs, producing gold PLMs in both. Those runs are generalization evidence and are deliberately excluded from the scored corpus reported here. In commercial use the same pipeline runs per engagement against a client's own estate, typically 10× to 100× the size of this corpus, under the same eval, triage and gap-fix loop.

CardDemo is cleaner than production code. It is a reference application: consistently named, structurally regular, and without the accumulated irregularity of a system modified by many hands over thirty years. We expect real estates to score lower rather than higher, for two reasons. Messier code carries more constructs the parsers have not previously encountered, which the gap detector surfaces rather than guesses at, and it carries more operational context that cannot be derived from source and must therefore be qualified as inference. Neither effect is measured here and no figure is claimed for it. Read 98.7 as a reference point obtained on well-formed code, not as an expected result on an arbitrary estate.

Small archetype samples. Three of the eight archetypes carry very few programs: Batch Utility (n = 1), Batch Report (n = 2) and Called Subprogram (n = 3). Their archetype means are reported for completeness but are not robust, and a single program moves them materially. The overall figure is a flat mean over all 42 programs precisely so that these small groups are not over-weighted. Archetype means should not be quoted as claims about archetypes.

Grader independence is a control, not a guarantee. Section 4 documents the cross-vendor design in full, including the decorrelation study (Cohen's κ = 0.48) and the recorded case in which three same-vendor graders credited a PLM containing a control-flow contradiction that only the cross-vendor grader caught. The residual limitation is that both the generator and the grader are large language models. Cross-vendor grading removes same-family blind spots; it does not remove the possibility of failure modes common to the current generation of models. The mitigations against that are structural rather than statistical: grading is anchored to deterministic source facts rather than to the grader's opinion, and entity grounding is enforced by a non-AI checker (see FABRICATION-DETECTION.md).

Fabrication detection covers entities, not semantics. The zero-fabrication result is a precise claim with a precise scope. The hard gate verifies that every named reference in a document resolves to a real entity in the source or its Structured Technical Baseline. It does not, and is not intended to, verify that a correctly-named entity is described as doing the right thing. Semantic accuracy is what the rubric score measures, and it is reported separately for exactly this reason. The two numbers answer two different questions and should be read that way.

Reproducibility ceiling. The corpus is public and stable and the rubric and methodology are published, so a third party can re-derive what was scored and how, and can re-run the published rubric against the same programs with a grader of their own. The CDF pipeline and the specific model versions are not open, so the generation step cannot be independently reproduced. Every published number in RESULTS.md is recomputable from results/scores.csv.