ci: run the conformance suites that cargo test never executed #93

Merged

dcharlot merged 1 commit from ci/run-conformance-verifiers into main 2026-09-17 06:58:33 -04:00

Conversation 0 Commits 1 Files changed 1 +34 -5

2026-09-15

Transaction Science

ci: run the conformance suites that cargo test never executed ...

Five standards ship a conformance runner whose suite lives in a binary's
`main()`, not in `#[test]`. `cargo test` therefore never executes them, and
`--all-targets` compiles the binary without running it — so the suites reported
nothing and the matrix looked green.

Verified rather than inferred:

    cargo test -p aac-conformance   ->  0 tests
    cargo run  -p aac-conformance   -> 40 checks
    cargo test -p map-conformance   ->  1 test
    cargo run  -p map-conformance   -> 15 checks

Same shape for jcp, proof and sandbox. Every one of these suites includes
negative vectors (`C-neg`, `Cneg`) — the checks that catch a verifier ACCEPTING
what it must reject, which no amount of positive-path testing finds.

This is the `main()` instance of the feature-gated-test bug already recorded for
this repo: code that compiles, reports nothing, and reads as covered. Compiling
a suite is not running it.

aac-conformance's own doc comment says "Exit code is the number of failed checks
(capped), so CI can gate on a clean 0" — it was written to be gated on, and
nothing ever gated on it. All five exit non-zero on failure (aac by failure
count, the rest by `exit(1)`), so the step has teeth.

Two things this step deliberately does NOT claim:

  * No path argument. These binaries take none — they check the implementation
    against in-code expectations. An earlier draft of this step passed
    `-- ../conformance/v1`, which they silently ignore; a tampered copy of
    map/conformance/v1 still passed 15/15, which is how the mistake surfaced.
  * It does not validate the published `conformance/v1` files on disk. Nothing
    in CI does that yet. Standards whose packs ARE read by a `#[test]` (jcp
    action-receipt, joule-ui, jouleclaw, joulecontract, sandbox, sense) are
    already covered by the Test step; joule-code has its own workflow. The
    remaining on-disk-pack gap is real and stays open rather than being papered
    over by a step that looks like it closes it.

Co-Authored-By: Claude Opus 5 <[email protected]>

2951ed150b

• View at this point in history

2026-09-15 11:11:47 -04:00