ci: run the conformance suites that cargo test never executed #93
Loading…
Reference in a new issue
No description provided.
Delete branch "ci/run-conformance-verifiers"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Five standards ship a conformance runner whose suite lives in a binary's
main(), not in#[test].cargo testnever executes them, and--all-targetscompiles the binary without running it — so the suites reported nothing and the matrix looked green.Verified, not inferred
Same shape for
jcp,proof,sandbox. Every one of these suites includes negative vectors (C-neg,Cneg) — the checks that catch a verifier accepting what it must reject, which no amount of positive-path testing finds.This is the
main()instance of the feature-gated-test bug already recorded for this repo: code that compiles, reports nothing, and reads as covered. Compiling a suite is not running it.aac-conformance's own doc comment says "Exit code is the number of failed checks (capped), so CI can gate on a clean0" — it was written to be gated on, and nothing ever gated on it. All five exit non-zero on failure (aac by failure count, the rest viaexit(1)), so the step has teeth.Two things this step deliberately does NOT claim
-- ../conformance/v1, which they silently ignore; a tampered copy ofmap/conformance/v1still passed 15/15, which is how the mistake surfaced.conformance/v1files on disk. Nothing in CI does that yet. Standards whose packs are read by a#[test](jcp action-receipt, joule-ui, jouleclaw, joulecontract, sandbox, sense) are already covered by the Test step, and joule-code has its own workflow. The remaining on-disk-pack gap is real and stays open rather than being papered over by a step that looks like it closes it.🤖 Generated with Claude Code
Five standards ship a conformance runner whose suite lives in a binary's `main()`, not in `#[test]`. `cargo test` therefore never executes them, and `--all-targets` compiles the binary without running it — so the suites reported nothing and the matrix looked green. Verified rather than inferred: cargo test -p aac-conformance -> 0 tests cargo run -p aac-conformance -> 40 checks cargo test -p map-conformance -> 1 test cargo run -p map-conformance -> 15 checks Same shape for jcp, proof and sandbox. Every one of these suites includes negative vectors (`C-neg`, `Cneg`) — the checks that catch a verifier ACCEPTING what it must reject, which no amount of positive-path testing finds. This is the `main()` instance of the feature-gated-test bug already recorded for this repo: code that compiles, reports nothing, and reads as covered. Compiling a suite is not running it. aac-conformance's own doc comment says "Exit code is the number of failed checks (capped), so CI can gate on a clean 0" — it was written to be gated on, and nothing ever gated on it. All five exit non-zero on failure (aac by failure count, the rest by `exit(1)`), so the step has teeth. Two things this step deliberately does NOT claim: * No path argument. These binaries take none — they check the implementation against in-code expectations. An earlier draft of this step passed `-- ../conformance/v1`, which they silently ignore; a tampered copy of map/conformance/v1 still passed 15/15, which is how the mistake surfaced. * It does not validate the published `conformance/v1` files on disk. Nothing in CI does that yet. Standards whose packs ARE read by a `#[test]` (jcp action-receipt, joule-ui, jouleclaw, joulecontract, sandbox, sense) are already covered by the Test step; joule-code has its own workflow. The remaining on-disk-pack gap is real and stays open rather than being papered over by a step that looks like it closes it. Co-Authored-By: Claude Opus 5 <[email protected]>