From 2951ed150b0618cb5dac9cff1b426e0af92a05ff Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Tue, 15 Sep 2026 11:11:47 -0400 Subject: [PATCH] ci: run the conformance suites that cargo test never executed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five standards ship a conformance runner whose suite lives in a binary's `main()`, not in `#[test]`. `cargo test` therefore never executes them, and `--all-targets` compiles the binary without running it — so the suites reported nothing and the matrix looked green. Verified rather than inferred: cargo test -p aac-conformance -> 0 tests cargo run -p aac-conformance -> 40 checks cargo test -p map-conformance -> 1 test cargo run -p map-conformance -> 15 checks Same shape for jcp, proof and sandbox. Every one of these suites includes negative vectors (`C-neg`, `Cneg`) — the checks that catch a verifier ACCEPTING what it must reject, which no amount of positive-path testing finds. This is the `main()` instance of the feature-gated-test bug already recorded for this repo: code that compiles, reports nothing, and reads as covered. Compiling a suite is not running it. aac-conformance's own doc comment says "Exit code is the number of failed checks (capped), so CI can gate on a clean 0" — it was written to be gated on, and nothing ever gated on it. All five exit non-zero on failure (aac by failure count, the rest by `exit(1)`), so the step has teeth. Two things this step deliberately does NOT claim: * No path argument. These binaries take none — they check the implementation against in-code expectations. An earlier draft of this step passed `-- ../conformance/v1`, which they silently ignore; a tampered copy of map/conformance/v1 still passed 15/15, which is how the mistake surfaced. * It does not validate the published `conformance/v1` files on disk. Nothing in CI does that yet. Standards whose packs ARE read by a `#[test]` (jcp action-receipt, joule-ui, jouleclaw, joulecontract, sandbox, sense) are already covered by the Test step; joule-code has its own workflow. The remaining on-disk-pack gap is real and stays open rather than being papered over by a step that looks like it closes it. Co-Authored-By: Claude Opus 5 --- .github/workflows/standards-build.yml | 39 +++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index 8a13bc820..f3ec8bd85 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -50,18 +50,18 @@ jobs: fail-fast: false matrix: include: - - { name: jcp-rs, dir: jcp/jcp-rs, runner: linux } - - { name: sandbox-rs, dir: sandbox/sandbox-rs, runner: linux } + - { name: jcp-rs, dir: jcp/jcp-rs, runner: linux, conformance: jcp-conformance } + - { name: sandbox-rs, dir: sandbox/sandbox-rs, runner: linux, conformance: sandbox-conformance } - { name: smart-byte-rs, dir: smart-byte/smart-byte-rs, runner: linux } - { name: openpay, dir: openpay, runner: linux-arm64 } - { name: joule-code-rs, dir: joule-code/joule-code-rs, runner: linux-arm64 } - - { name: map-rs, dir: map/map-rs, runner: linux-arm64 } - - { name: aac-rs, dir: aac/aac-rs, runner: linux-arm64 } + - { name: map-rs, dir: map/map-rs, runner: linux-arm64, conformance: map-conformance } + - { name: aac-rs, dir: aac/aac-rs, runner: linux-arm64, conformance: aac-conformance } - { name: joulecontract-rs, dir: joulecontract/joulecontract-rs, runner: linux-arm64, z3: true } - { name: jouledesk-rs, dir: jouledesk/jouledesk-rs, runner: macos } - { name: arl-rs, dir: arl/arl-rs, runner: macos } - { name: sense-rs, dir: sense/sense-rs, runner: macos } - - { name: proof-rs, dir: proof/proof-rs, runner: linux, z3: true } + - { name: proof-rs, dir: proof/proof-rs, runner: linux, z3: true, conformance: proof-conformance } - { name: joulehook-rs, dir: joulehook/joulehook-rs, runner: macos } - { name: eoc-rs, dir: eoc/eoc-rs, runner: linux } - { name: mesh-rs, dir: mesh/mesh-rs, runner: linux-arm64 } @@ -150,3 +150,32 @@ jobs: env: RUSTFLAGS: ${{ matrix.rustflags }} run: cargo test --workspace + + # Run each standard's conformance suite. + # + # These suites live in a binary's `main()`, not in `#[test]`, so + # `cargo test` never executes them — verified, not assumed: + # `cargo test -p aac-conformance` runs **0 tests** while + # `cargo run -p aac-conformance` runs **40 checks**; map is 1 test vs 15 + # checks. Same shape for jcp, proof and sandbox. The suites include + # negative vectors (`C-neg`, `Cneg`), which are the ones that catch a + # verifier ACCEPTING what it must reject. + # + # This is the `main()` instance of the same bug as a feature-gated test: + # the code compiles under `--all-targets`, reports nothing, and looks + # covered. Compiling a suite is not running it. + # + # `--bin` is explicit because these crates also ship a `gen-vectors` + # binary; `cargo run -p ` is ambiguous with two binaries and fails + # rather than choosing — which is how this step would silently never run. + # + # No path argument: these binaries take none. They check the + # implementation against in-code expectations, which is a different claim + # from validating the published `conformance/v1` files on disk — nothing + # in CI does that yet, and this step does not pretend to. + - name: Conformance suite + if: matrix.conformance + working-directory: ${{ matrix.dir }} + env: + RUSTFLAGS: ${{ matrix.rustflags }} + run: cargo run -q -p ${{ matrix.conformance }} --bin ${{ matrix.conformance }} -- 2.54.0 (Apple Git-157)