Scoring rubric

Every Program Logic Manual (PLM) is scored 0–100 so the corpus result is interpretable, not a bare marketing number. The rubric is archetype-specific: a program is classified into one of eight archetypes first (see METHODOLOGY.html §2), and each archetype is scored on six maintenance-scenario tasks with archetype-appropriate weights. The tasks and weights below are the live rubric weights used to score this corpus — extracted from the scoring harness, not illustrative.

How a score is computed

1. Classify the program into one archetype.
2. Score six tasks, each 0–100. A task is a concrete maintenance question a programmer would bring to the PLM (e.g. "A master record was corrupted after this job ran — trace what updates were made"). Each task score is multiplicative — coverage × quality × 100 — so a PLM that covers a scenario broadly but shallowly, or narrowly but well, cannot max the task; it must be both complete and correct.
3. Weighted average the six task scores using that archetype's weights (each column below sums to 100%) → the program's 0–100 score.
4. grounding_coverage (how much of each claim traces to a citation) is measured and reported but is diagnostic only — it never feeds the score.

Two things sit outside this weighted average and are handled separately: the fabrication hard-gate (below) and critical-error penalties (a direct source contradiction can cap a task regardless of coverage).

The six tasks and weights, per archetype

Every archetype carries Business Purpose (what/why) and Error Handling (failure paths); the middle four tasks are what distinguishes the archetype.

CICS Inquiry — online read/display

CICS Data Entry — online create/update

CICS Menu — navigation

CICS Other — other online CICS

Batch File Maintenance — master-file update

Batch Report — report production

Batch Utility — support / utility

Called Subprogram — reusable CALL'd module

The weights encode what a maintainer of that kind of program actually needs: a called subprogram's value is its interface contract (30%), a menu's is its routing (30%), an inquiry's is the file→screen data flow (25%). Business Purpose and Error Handling are held light (10–25% / 5%) so the discriminating middle tasks drive the score.

Fabrication: a separate hard gate (checked before any task scoring)

Fabrication is not a scored dimension that can be averaged against good writing. It is a pass/fail pre-check run before the six tasks are scored: every named entity in the PLM — field, paragraph, file/dataset, copybook, CICS command/map, CALL'd program — is resolved against the source and its Structured Technical Baseline. Any entity not found → the PLM is rejected and re-processed, not patched (total_fabrications > 0 ⇒ verdict UNACCEPTABLE, task scoring is skipped). File-name variants that denote the same resource (JCL DDNAME, CICS dataset name, WS literal, FD name) are all treated as grounded — the gate does not trip on a legitimate alias. Full definition: FABRICATION-DETECTION.html.

This is why the corpus result is reported as two distinct facts — a quality score and zero fabrications — never folded into one.

Not-applicable ≠ absent (why humility is scored as correctness)

A required element that genuinely does not apply to a program (e.g. a JCL trigger for a called subprogram) is marked NOT_APPLICABLE and is not penalized. A source-true statement of an absence — "no in-program trigger; invoked by an external scheduler not in the corpus" — counts as VERIFIED, not a gap. ABSENT is reserved for a required fact the PLM simply failed to provide. The rubric therefore rewards a PLM that says "this cannot be determined from source" over one that confidently asserts an unverifiable fact.

Scoring bands

The production bar is aggregate ≥ 95 with no individual task scoring below 90 on the score of record (the median of the three grading rolls, METHODOLOGY.html §4.3; every sub-90 roll individually triaged). The 95–100 top band matches this internal bar. The benchmark is not published until every program either clears the no-task-below-90 floor or carries an individually-audited, recorded grading-artifact exception — a documented grader error on a correct document, never a document defect — so the published claim matches the audit trail rather than resting on an unqualified "universally." Judge consistency (not inter-rater human agreement) is the relevant control here — handled by the multi-roll median in METHODOLOGY.html §4.3.

Why perfect scores are rare (and that's correct)

The rubric intentionally does not award points for confidently stating things that cannot be mechanically verified from source — e.g. who consumes a report or when a batch job is scheduled in production. A PLM that qualifies such statements as inferences scores higher than one that asserts them as fact. This — together with the multiplicative coverage × quality per task — is what keeps scores honest.

The "rare" is a statement about the corpus aggregate, not individual programs. Many individual programs legitimately earn a median of 100 — a program with no non-derivable operational context to qualify (a self-contained subprogram, a program whose every claim resolves to source) can and does score a clean 100 across all three rolls. What is rare is the whole-corpus average sitting at 100: across programs of every archetype, some carry inherently non-derivable context (report consumers, production schedules) that a correct PLM must hedge rather than assert, which pulls the aggregate to ~98–99. So a per-program 100 is expected and correct; a corpus-wide 100 would be the red flag. In regulated environments, appropriate epistemic humility is part of correctness, not a deduction from it.