How accurate is automated documentation for legacy COBOL — and how would you even measure that?
This repository publishes the methodology, scoring rubric, and results from evaluating the Brevidy COBOL Documentation Factory (CDF) against AWS CardDemo, the open-source mainframe reference application widely used to benchmark COBOL modernization tools.
Our goal in publishing this is not marketing. It is to put a reproducible, transparent measuring stick into a space that doesn't have one. Today, if a maintenance lead or a modernization architect asks "how accurate is AI-generated COBOL documentation?", there is no public, methodical answer. This is our attempt at one, with the rubric and the process open so others can critique, replicate, or improve on it.
| Archetype | Programs | Mean score |
|---|---|---|
| CICS Other | 4 | 99.4 |
| Batch File Maintenance | 11 | 99.1 |
| Batch Report | 2 | 99.1 |
| CICS Inquiry | 8 | 98.6 |
| CICS Menu | 6 | 98.4 |
| CICS Data Entry | 7 | 98.3 |
| Batch Utility | 1 | 97.7 |
| Called Subprogram | 3 | 97.7 |
| Overall | 42 | 98.7 |
Machine-readable scores:
results/scores.csv
Three facts make COBOL documentation quality unusually hard — and unusually important — to measure:
The stakes are high and the tolerance for error is low. COBOL runs core banking, insurance, and government systems. In these environments, documentation that is confidently wrong is more dangerous than no documentation at all — a maintenance programmer who trusts a fabricated field name or an invented control-flow path can cause a production incident.
Generic AI tools hallucinate. Large language models pointed at COBOL will readily invent field names, paragraph references, and plausible-but-wrong business-logic descriptions. Any honest benchmark therefore has to measure fabrication, not just fluency.
There was no shared yardstick. CardDemo is the natural corpus — it is open source, it is the reference app AWS itself uses for modernization patterns, and it contains a realistic mix of CICS online programs, batch jobs, called subprograms, copybooks, and JCL. But nobody had published a documentation-quality rubric against it. So we wrote one.
AWS CardDemo is a comprehensive COBOL credit-card management application (accounts, cards, transactions, bill payment) built to showcase mainframe modernization patterns. It is a good benchmark corpus precisely because it is representative: online CICS transactions with BMS maps, batch file maintenance and reporting, called subprograms, copybook hierarchies, and JCL job streams — the same archetypes a real mainframe estate contains.
We used the public main branch as the corpus. The exact
commit hash used for this run is recorded in METHODOLOGY.html for
reproducibility.
CDF is a two-stage pipeline. Stage 1 is
deterministic: independent parser layers break each
COBOL program into a machine-readable Structured Technical Baseline
(STB) — every data structure, control-flow edge, I/O operation,
CICS command, BMS screen, and JCL configuration, as a verified fact
inventory with no AI involved. Stage 2 is
constrained AI synthesis: the model reads the source
independently and uses the STB as a cross-reference, grounding every
statement in specific source lines. A hallucination
hard-gate then cross-checks every named entity in the output
against the source; any name that does not appear in the code causes the
PLM to be rejected, not patched. Full detail in METHODOLOGY.html.
| File | What's in it |
|---|---|
METHODOLOGY.html |
The full pipeline, the corpus commit, and how a score is produced end to end |
RUBRIC.html |
The scoring rubric — dimensions, weights, and what earns/loses points |
RESULTS.html |
Full results, per-archetype and per-program tables, and where the 1.3 points go |
FABRICATION-DETECTION.html |
How the hard-gate defines and detects a fabricated entity |
results/scores.csv |
Machine-readable scores |
samples/ |
Example PLM deliverables from the CardDemo run |
CITATION.cff |
Citation metadata (enables GitHub's "Cite this repository") |
If you reference this benchmark, please cite it via the metadata in
CITATION.cff, or link to this
repository directly. We welcome scrutiny of the rubric and the
methodology — open an issue.
The COBOL Documentation Factory is a service from Brevidy. It turns legacy COBOL into production-ready, line-referenced Program Logic Manuals for compliance, maintenance, and modernization. Learn more at https://brevidy.com.
The documentation, rubric, and methodology in this repository are
released under CC BY 4.0 — you may share and adapt them
with attribution. See LICENSE. AWS
CardDemo is the property of Amazon Web Services under its own license;
this repository does not redistribute CardDemo source.