joule-code: close the two gaps its conformance gate cannot see #210

Merged

dcharlot merged 2 commits from dcharlot/joule-code-pack-test into main 2026-10-13 02:40:37 -04:00 AGit

Conversation 0 Commits 2 Files changed 2 +176 -2

2026-10-10

David Charlot

ci: the comment named a mechanism that is a convention, not a rule ...

The commit before this one (revived from #97) added a note explaining why a
missing `conformance:` key does not mean a standard's pack is unchecked. Its
reasoning is right and its mechanism was wrong: it said the checking is done
"by a `tests/pack.rs` in each conformance crate". Several crates do use that
name, but it is a convention — some read the pack from a test named something
else, and some from another crate entirely.

That matters because the wrong mechanism invites the wrong audit. Counting
files named `tests/pack.rs` and finding crates without one looks like a
coverage gap and is not one; I did exactly that earlier today and was about to
report three standards as unchecked before a note from the last time it
happened stopped me. So the comment now names the question — which tests read
the published bytes — and the grep that answers it.

No count in the text on purpose: any "N of M conformance crates" would be
stale by the next crate added, and a stale number in a comment is how the
first wrong conclusion got its confidence.

Co-Authored-By: Claude Opus 5 <[email protected]>

37f5bda4ba

• View at this point in history

2026-10-10 17:22:44 -04:00

Transaction Science

joule-code: close the two gaps its conformance gate cannot see ...

`joule-code-conformance` ships a complete pack verifier — `verify_pack` reads
all six files of `joule-code/conformance/v1`, recomputes every derived field
(each `op_id` is BLAKE3 over canonical CBOR of its `CodingOp`, each binding and
receipt signature a real Ed25519 signature over the spec's preimage) and returns
a FAIL count that `main()` turns into a non-zero exit. A dedicated workflow,
`joule-code-conformance.yml`, already runs it. The pack is not unchecked, and
this commit is not what makes it checked.

It closes the two gaps that gate cannot see.

**It cannot distinguish a verified pack from a skipped one.** The gate is
`grep -Eq '0 failed'` over the summary line, and `verify_pack` is documented to
let SKIPs pass. So `0 passed, 0 failed, 24 skipped` satisfies that grep exactly
as well as `24 passed, 0 failed, 0 skipped` — verified by running the three
summary strings through the gate's own expression. A pack that shrank to one
vector passes it too. This pins the count and requires `skip == 0`, which is
the distinction the grep cannot draw. The count is pinned rather than bounded
because `>= 24` would let a case be deleted as long as another was added.

**It is `paths:`-filtered to `joule-code/**`, and this crate reaches outside
it.** `joule-code-conformance` depends on `jcp-schema`, `jcp-grant` and
`jcp-receipt` by relative path across workspaces, and `joule-code-joulecontract`
on `joulecontract-schema`. A change in `jcp` to canonical CBOR or the receipt
preimage would change the `op_id`s and signatures this pack pins, and would not
touch `joule-code/**` — so that workflow would not run. `standards-build.yml`
carries a deliberate "NO `paths:` filter" and runs `cargo test --workspace`,
confirmed by running that exact command rather than reasoning about it.

The second test is the load-bearing one. A verifier reporting PASS on a pack it
never read is worse than none, because it manufactures confidence — a tampered
copy of `map`'s pack once still reported 15/15. So the tamper case does not
trust the report: it flips one hex digit of a signature in a scratch copy and
requires the verifier to notice. The offset is found by scanning to the value's
opening quote, since the pack is column-aligned and the run of spaces is not
something a test should depend on.

Falsified, each assertion separately: corrupting a published signature fails
the verify test; claiming 25 vectors fails the count; neutering the flip to a
no-op fails the tamper test with "the verifier is not checking what it
publishes". The pack was restored afterwards and is byte-identical to main.

One honest limit: I could not force a SKIP to confirm that assertion fires, so
unlike the other three it is reasoned rather than falsified.

Corrects the workflow comment claiming nothing in CI validates the published
`conformance/v1` files on disk. That stopped being true with the `tests/pack.rs`
suites, and a missing `conformance:` key is not evidence a pack is unchecked —
`joule-code` and `sandbox` are both covered without one.

Co-Authored-By: Claude Opus 5 <[email protected]>

692d769f2e

• View at this point in history

2026-10-10 17:20:38 -04:00