wai breaking and wai conform: the compatibility gate on every WAI pull request, and conformance runs with signed results #507

Merged

dcharlot merged 11 commits from wai/breaking-conform into main 2026-10-28 05:08:42 -04:00

Conversation 0 Commits 11 Files changed 38 +6380 -15

dcharlot commented 2026-10-27 05:23:27 -04:00

Owner

Copy link

What this is

Two operations of the WAI toolchain, and the CI gate WAI's libraries are published behind.

• wai breaking --against <git-ref> (operation compat.check): what the working tree breaks against the merge base of a ref and HEAD, in three cumulative categories, strictest by default (--category wire|record|api, --only/--skip <surface>).

  • wire. registry: the registry export (wai/registry/capabilities.json, held to REGISTRY.md by registry export --check). A capability, identifier or wire identifier (keyed by kind and name: wai1 is both a codecs prefix and a 4CC) that is gone breaks; so does a capability whose kind, media, role, tier, licence, replicate class, edition or operator set changed, one that left a set or joined the floor, one demoted to candidate, and a wire identifier whose status fell back. corpora: every wai/*-conformance/ vector and wai-rs/tests/vectors/ file that is gone or whose bytes changed; a JSON file is diffed structurally (entries keyed by name/file/id/…, prose members not compared), so a corpus that only gained vectors is compatible.
  • record. domains: every literal constant named …DOMAIN…, …MAGIC… or …VERSION… in wai/*/src (binaries and test modules left out): a changed value or a removed one breaks (a move to another file with the same name and value does not). corpora: the record corpora (claim, signer, jwp, pq, oer, pin, edge, live). schemas: wai/**/*.schema.json, a document the old schema accepted that the new one refuses.
  • api. rust: the public API of wai and wai-quantum, from rustdoc's JSON (cargo rustdoc -- -Z unstable-options --output-format json under RUSTC_BOOTSTRAP=1), both sides documented with the toolchain rust-toolchain.toml pins and every feature but those needing system libraries or downloads. The base is extracted with git archive to a scratch directory. Every public path is walked from the crate root through modules and re-exports: functions (signature with ids dropped and paths made canonical; parameter names not compared), structs and fields, enums and variants, traits and their items, inherent methods, trait implementations (auto traits in, blanket ones out), constants, statics, aliases, macros; plus the semver rules for exhaustive structs and enums and required trait items. A crate is documented only when something under it or its path dependencies changed. c: wai-rs/include/wai.h parsed (functions by return and parameter types, struct layouts, enums, macro values, preprocessor conditions). openapi: the room service's openapi.json (on PR #500's branch; read from joule-code/.../joule-moq-room/openapi.json once it lands, and wai/**/openapi*.json): routes, operationIds, parameters, required bodies, success responses and their properties.
  • Each finding has an id <category>:<surface>:<item>, what changed and why it breaks; additions are compatible. A break passes only when a line this change adds to wai/CHANGELOG.md, in an entry that says "breaking", names the id in backticks. Exit 1 otherwise; the run prints the line to add.

• .github/workflows/wai-breaking.yml: on every pull request touching wai/** (and the room OpenAPI, the toolchain pin, this file): job breaking runs wai breaking --against origin/main; job runner runs the tests below and then the corpora's independent Python verifiers (claim_verify.py, jwp_verify.py, signer_verify.py, python3 -I) and the reference packager (wai-pack policy-verdict) through wai conform, and wai verify on every record. Concurrency group per ref; the relevant gate reads the paths from the file (scripts/pr-touches-paths.py), as the other WAI workflows do.

• wai conform <kind> --suite <suite> --impl <command or URL> (operation conformance.run): one of WAI's corpora run against another implementation. Kinds and suites:

  kind	suite	corpus	cases	passes when
  verifier	claim	claim-conformance/vectors.json	each vector	the declared verdict (claim README grammar)
  verifier	jwp	jwp-conformance/vectors.json	each vector	the declared verdict
  verifier	signer	signer-conformance/vectors.json	each vector	the declared verdict line
  decoder	score	score-conformance/	each WSCR	rendered i16le PCM has the declared pcm_sha256
  packager	policy	pack-conformance/policy + refusals/policy	each policy file	ok <digest> as declared, or the declared refusal reason
  origin	http	http-conformance/objects/	each object	§5.3 headers (ETag, Content-Type, Cache-Control, Accept-Ranges, CORS, Timing-Allow-Origin, no Vary/Content-Encoding), a byte range (206 + Content-Range or 200 whole), HEAD, the §6.2 head proof, the preflight

  Adapters: a command ({corpus} whole-corpus name<TAB>verdict, {case}, {in}, {out}, else stdin/stdout; per-case --timeout), and HTTP (POST per case, a JSON verdict member or the first line; an origin is fetched from). The result is a signed record conform-result/1 (<suite>.wai-conform.json): corpus digest (SHA-256 over sha256 path lines of every file read), implementation (command + SHA-256 of each file it names, or URL; --name/--version), each case's outcome, summary, conformant, the standard's version (CHANGELOG heading) and toolchain version; Ed25519 over wai:conform-result\x01 ‖ JCS(record without sig), by a run key or --key <seed file>. wai verify now also checks *.wai-conform.json: the signature (strict), and that the summary and conformant are what the cases say.

• Both follow the agent rules: envelope/1 JSON with --json or off a terminal (--text keeps text), exit statuses 0/1/2/4/5/6/7, --dry-run for conform, --confirm before replacing a record. Registered as entries of dx::VERBS (breaking, conform) until the operation catalogue lands; breaking::run(&Options) -> Report and Report::to_json() are what an MCP tool compat_check calls.

• conform-result/1 in the standard (extensions/conformance-results.md): the record's members and corpus digest (§2, §2.1), its signing input (wai:conform-result and 0x01, then the record without sig in RFC 8785 form) and Ed25519 signature under SPEC §7's strict rule (§3), what a verifier checks and in what order (§4: it reads; its version; its members; its signature; its summary; its verdict) and the codes (§5). Registered as a document kind (registry export and REGISTRY.md regenerated, registry export --check up to date); a SPEC §7 bullet points to it.

  • Reference: the wai-rs module conform_result (feature conform_result, now in quickstart); wai conform signs and wai verify checks through it. It reads records with the crate's strict JSON reader (a member named twice, an unpaired surrogate, a number written other than as a plain integer are refused) and verifies with wai::sig::verify_strict.
  • Corpus conform-result-conformance/: accepted records (conformant; not conformant; no cases; HTTP adapter with a supplied key) and refused ones (a case flipped after signing, the corpus digest changed, the summary alone changed, a bad signature, S + L, a small-order key, each summary count wrong alone and two at once, a verdict that disagrees, an unknown version, a member missing, added or named twice, a fractional count, an upper-case digest, bytes that are not JSON). Every outcome is stated in the generator from the text, and the generator asserts the reference reaches it before writing; wai_conform_result_vectors gen|verify|verdicts.
  • Second implementation conform_result_verify.py (standard library, python3 -I, WAI's strict Ed25519 rule from claim-conformance/_ed25519.py, its own JSON reading and RFC 8785 serialisation), which reaches every outcome, reads §5's table, and with --skip <check> fails the corpus for each check of §4 left out. CI step in wai-conformance.yml's independent-verifiers job (feature added to its union and to features-alone; the extension added to its paths). wai conform gains the verifier suite conform-result, and the runner job runs the Python verifier through it.

• Measurements are not compared: wai breaking leaves out an explicit list (breaking/corpora.rs MEASUREMENTS, one entry a file, each with its reason) and reports each such change as a note: wai/format-probe-conformance/provenance.json (the external decodes measured on one build host; no verifier reads it) and wai/live-conformance/metering/relay-record.wmh and relay-ledger.wml (one live loopback run's recording; the vectors built around it in vectors.json stay compared). A bin test requires every listed file to exist, once, with a reason. The corpora are otherwise deterministic by design (their READMEs state the same bytes on every machine), so the list is short.

• A command wai conform runs is killed alone at its timeout, and its output is read for at most two seconds after it exits, so a child that keeps the output open does not hold the run. (An earlier commit killed the command's process group with kill -KILL -<pgid>; in the CI image that call signals the caller's own group, which killed the runner job with exit 137. Reproduced in that image and removed; the step peaks at 4.6 GB built cold under a 16 GB cap, so memory was not the cause.)

• tests/int_kernels_vectors.rs (every corpus has a decoded-output sidecar or a stated exclusion) lists conform-result-conformance with its reason: signed records, no decoded output.

• Docs: /docs/breaking/ and /docs/conform/ on the WAI site's docs-quickstart branch (head 0d217b2), each with a .md twin and an /llms.txt entry; the conform page links the record's specification. Not deployed.

Design decisions

• Merge base, not the ref's tip: a branch is not charged with removing what main gained after it forked.
• Acknowledgement only from added changelog lines, so an old entry never re-licenses a break; the entry must say "breaking" and the id must be backticked. Corpus findings are per file, so one entry acknowledges a regenerated corpus file.
• Record preimages are checked twice over: statically (domain/magic/version constants), and through the corpora, which the conformance workflow already holds byte-identical to the generating code. A preimage change in code reaches a corpus file or fails that workflow; wai breaking reads the file.
• Rust API from rustdoc JSON read as plain JSON (no rustdoc-types version pin), both sides from one toolchain; a format-version mismatch refuses to compare.
• Additive registry members (a capability gaining an ops declaration it never had) are compatible; changed or removed ones break.
• Plain HTTP only for the HTTP adapter in this build (no TLS client in quickstart).

Checks

check	command	host	result
tests, both commands	cargo test --locked --no-default-features --features quickstart,score,key_discovery --bin wai --test cli_breaking --test cli_conform	Studio	all pass, five runs in a loop
existing dx tests	cargo test --locked --no-default-features --features quickstart --bin wai --test floor --test cli_dx; cargo test --no-default-features --bin wai	Studio	pass
clippy	cargo clippy --locked --no-default-features [--features quickstart | quickstart,score,key_discovery] --bin wai --tests	Studio	no note in the new files (the pre-existing notes in wai.rs, http*.rs are untouched)
build, no feature	cargo build --locked --no-default-features --bin wai	Framework (Linux)	ok
gate against main	wai breaking --against <main> --text (all surfaces, wai documented twice)	Framework, Studio	no finding at all: the same public surface on both sides
wai-quantum documented	a comment appended to wai-quantum/src/lib.rs, --only rust	Studio	both crates documented, no finding, 61 s
conform, independent verifiers	wai conform verifier --suite {claim,jwp,signer} --impl "python3.11 -I wai/<s>-conformance/<s>_verify.py verdicts {corpus}"	Studio	every case passed, each conformant
conform, reference packager	wai conform packager --suite policy --impl "wai-pack policy-verdict {in}"	Studio	every case passed
conform, reference origin / renderer / Rust verifiers / HTTP verifier	in cli_conform	Studio	conformant
record corpus and verifiers	cargo test --features conform_result --lib conform_result:: and --test conform_result; wai_conform_result_vectors verify ../conform-result-conformance; python3 -I conform-result-conformance/conform_result_verify.py	Framework (Linux), local	all pass; every outcome reached by both
registry	cargo test --lib codecs::registry:: (wire identifiers in their documents); wai registry export --check	Framework	pass; up to date
tests after the additions	cargo test --no-default-features --features quickstart,score,key_discovery --bin wai --test cli_breaking --test cli_conform --test cli_dx --test conform_result	Framework (Linux)	all pass
gate on this branch	wai breaking --against <main> --text, every surface	Framework	no unacknowledged break; the new module, corpus and registry row compatible
site after the link	npm run build; pages.e2e /docs/conform/	Studio	ok; page checks all passed
repo checks	check-tests-in-ci.py, check-workflow-concurrency.sh, check-spec-refs.py, check-wai-section-refs.py, check-no-vanity-metrics.sh, test-public-text-checks.sh, check-comparative-prose.sh, check-orphan-sources.py, YAML parse	local	all pass
public names	check-public-names.sh on every changed file; commit messages and this body with grep -E	local	clean
site	npm run build (routes, public text with the denylist, names); node e2e/pages.e2e.mjs /docs/breaking/ /docs/conform/	Studio	build ok; 307 page checks passed

Falsifications

On this repository against main (each: exit 1 without an acknowledgement, exit 0 with a - **Breaking:** \`` line added):

1. A registered code removed: wai.audio.flac dropped from the registry export → wire:capability:wai.audio.flac; the magic WCT1 dropped → wire:wire-identifier:magic/WCT1.
2. A signing preimage changed: live_events.rs SIGN_DOMAIN \x01→\x02 → record:domain:wai/wai-rs/src/live_events.rs::SIGN_DOMAIN. And in code: access_services::signing_bytes appends a byte, the corpus regenerated with wai_access_services_vectors gen → record:corpus:wai/live-conformance/access/vectors.json; access_services_verify.py refuses the regenerated corpus (exit 1), and passes again once restored.
3. A public Rust function removed: energy_order::is_learned made private → api:rust:wai::energy_order::is_learned ("the function fn is_learned(&str) -> bool is gone").
4. A C function's signature changed: wai_envelope_detect_version's len: usize → u32 in ffi.rs, the header regenerated by build.rs → api:c:wai_envelope_detect_version (uintptr_t → uint32_t).
5. An OpenAPI route removed (on PR #500's branch, which holds the document): GET /room removed → api:openapi:joule-moq-room:GET /room; a required query parameter and a changed operationId → two more findings. Against main, every route of the document reads as compatible.

wai conform: a verifier that changes one verdict fails exactly that case (exit 1; Python claim verifier and Rust signer reference); a packager that changes one digest digit fails every admitted policy and nothing else; a renderer that adds one to one output byte fails every score case; an origin serving one object with one byte flipped fails that object ("the body is not the object"). A record with a failed case turned to a pass and its counts made to agree, one with a corpus-digest byte changed, and one with only its summary changed are each refused by wai verify (exit 1).

Guards mutated in the code, each caught by its test: an entry acknowledging without the word "breaking"; the record's signature not checked; C parameter types not compared.

conform-result/1's verifiers (on Linux):

• wai verify on each vector's record alone: every refused vector exits 1 at its stated step and reason, every accepted one exits 0; no mismatch.
• The Rust module with one check removed, each in turn (the signature; the summary's case count, its passed count, its failed count; the verdict; the members; the format; a byte added to the signing input): wai_conform_result_vectors verify fails every time. The first run found the case-count check unguarded (every summary vector had two counts wrong); three vectors with one count wrong each were added, and the rerun caught it.
• conform_result_verify.py --skip <check> for each check of §4 (duplicates, format, members, signature, summary, verdict): the corpus fails each time, naming the vectors that guard it.

The measurement list: a changed provenance.json is a note and no finding (exit 0); with the list emptied, the same change is wire:corpus:wai/format-probe-conformance/provenance.json (exit 1). cli_breaking tests a listed file not compared and an unlisted vector beside it still breaking.

Not done

• Kinds not covered: relay (the MOQT interop cases live in joule-moq-interop; no runner adapter yet) and player (browser sink). Decoder suites other than score (audio needs its sources beside the scene; int-payload, neural, film and the others need their output forms named per capability).
• HTTPS for the HTTP adapter; --format junit|html; --level, --independent, wai conformance gen; breaking.allow in wai.json.
• API surfaces the catalogue will add (catalogue IDs and flags, CLI JSON fields, MCP tool schemas); the JS, Swift and Kotlin APIs.
• The operation catalogue (wai ops) and the MCP tools compat_check/conformance_run: registration waits for those branches; both ops are separable modules.

🤖 Generated with Claude Code

## What this is Two operations of the WAI toolchain, and the CI gate WAI's libraries are published behind. - **`wai breaking --against <git-ref>`** (operation `compat.check`): what the working tree breaks against the merge base of a ref and `HEAD`, in three cumulative categories, strictest by default (`--category wire|record|api`, `--only`/`--skip <surface>`). - **wire.** `registry`: the registry export (`wai/registry/capabilities.json`, held to REGISTRY.md by `registry export --check`). A capability, identifier or wire identifier (keyed by kind and name: `wai1` is both a `codecs` prefix and a 4CC) that is gone breaks; so does a capability whose kind, media, role, tier, licence, replicate class, edition or operator set changed, one that left a set or joined the floor, one demoted to candidate, and a wire identifier whose status fell back. `corpora`: every `wai/*-conformance/` vector and `wai-rs/tests/vectors/` file that is gone or whose bytes changed; a JSON file is diffed structurally (entries keyed by `name`/`file`/`id`/…, prose members not compared), so a corpus that only gained vectors is compatible. - **record.** `domains`: every literal constant named `…DOMAIN…`, `…MAGIC…` or `…VERSION…` in `wai/*/src` (binaries and test modules left out): a changed value or a removed one breaks (a move to another file with the same name and value does not). `corpora`: the record corpora (claim, signer, jwp, pq, oer, pin, edge, live). `schemas`: `wai/**/*.schema.json`, a document the old schema accepted that the new one refuses. - **api.** `rust`: the public API of `wai` and `wai-quantum`, from rustdoc's JSON (`cargo rustdoc -- -Z unstable-options --output-format json` under `RUSTC_BOOTSTRAP=1`), both sides documented with the toolchain `rust-toolchain.toml` pins and every feature but those needing system libraries or downloads. The base is extracted with `git archive` to a scratch directory. Every public path is walked from the crate root through modules and re-exports: functions (signature with ids dropped and paths made canonical; parameter names not compared), structs and fields, enums and variants, traits and their items, inherent methods, trait implementations (auto traits in, blanket ones out), constants, statics, aliases, macros; plus the semver rules for exhaustive structs and enums and required trait items. A crate is documented only when something under it or its path dependencies changed. `c`: `wai-rs/include/wai.h` parsed (functions by return and parameter types, struct layouts, enums, macro values, preprocessor conditions). `openapi`: the room service's `openapi.json` (on PR #500's branch; read from `joule-code/.../joule-moq-room/openapi.json` once it lands, and `wai/**/openapi*.json`): routes, operationIds, parameters, required bodies, success responses and their properties. - Each finding has an id `<category>:<surface>:<item>`, what changed and why it breaks; additions are compatible. **A break passes only when a line this change adds to `wai/CHANGELOG.md`, in an entry that says "breaking", names the id in backticks.** Exit 1 otherwise; the run prints the line to add. - **`.github/workflows/wai-breaking.yml`**: on every pull request touching `wai/**` (and the room OpenAPI, the toolchain pin, this file): job `breaking` runs `wai breaking --against origin/main`; job `runner` runs the tests below and then the corpora's independent Python verifiers (`claim_verify.py`, `jwp_verify.py`, `signer_verify.py`, `python3 -I`) and the reference packager (`wai-pack policy-verdict`) through `wai conform`, and `wai verify` on every record. Concurrency group per ref; the `relevant` gate reads the paths from the file (`scripts/pr-touches-paths.py`), as the other WAI workflows do. - **`wai conform <kind> --suite <suite> --impl <command or URL>`** (operation `conformance.run`): one of WAI's corpora run against another implementation. Kinds and suites: | kind | suite | corpus | cases | passes when | |---|---|---|---|---| | verifier | `claim` | `claim-conformance/vectors.json` | each vector | the declared verdict (claim README grammar) | | verifier | `jwp` | `jwp-conformance/vectors.json` | each vector | the declared verdict | | verifier | `signer` | `signer-conformance/vectors.json` | each vector | the declared verdict line | | decoder | `score` | `score-conformance/` | each `WSCR` | rendered `i16le` PCM has the declared `pcm_sha256` | | packager | `policy` | `pack-conformance/policy` + `refusals/policy` | each policy file | `ok <digest>` as declared, or the declared refusal reason | | origin | `http` | `http-conformance/objects/` | each object | §5.3 headers (ETag, Content-Type, Cache-Control, Accept-Ranges, CORS, Timing-Allow-Origin, no Vary/Content-Encoding), a byte range (206 + Content-Range or 200 whole), HEAD, the §6.2 head proof, the preflight | Adapters: a command (`{corpus}` whole-corpus `name<TAB>verdict`, `{case}`, `{in}`, `{out}`, else stdin/stdout; per-case `--timeout`), and HTTP (`POST` per case, a JSON `verdict` member or the first line; an origin is fetched from). The result is a signed record `conform-result/1` (`<suite>.wai-conform.json`): corpus digest (SHA-256 over `sha256 path` lines of every file read), implementation (command + SHA-256 of each file it names, or URL; `--name`/`--version`), each case's outcome, summary, `conformant`, the standard's version (CHANGELOG heading) and toolchain version; Ed25519 over `wai:conform-result\x01` ‖ JCS(record without `sig`), by a run key or `--key <seed file>`. **`wai verify`** now also checks `*.wai-conform.json`: the signature (strict), and that the summary and `conformant` are what the cases say. - Both follow the agent rules: `envelope/1` JSON with `--json` or off a terminal (`--text` keeps text), exit statuses 0/1/2/4/5/6/7, `--dry-run` for conform, `--confirm` before replacing a record. Registered as entries of `dx::VERBS` (`breaking`, `conform`) until the operation catalogue lands; `breaking::run(&Options) -> Report` and `Report::to_json()` are what an MCP tool `compat_check` calls. - **`conform-result/1` in the standard** ([`extensions/conformance-results.md`](wai/extensions/conformance-results.md)): the record's members and corpus digest (§2, §2.1), its signing input (`wai:conform-result` and `0x01`, then the record without `sig` in RFC 8785 form) and Ed25519 signature under SPEC §7's strict rule (§3), what a verifier checks and in what order (§4: it reads; its version; its members; its signature; its summary; its verdict) and the codes (§5). Registered as a document kind (registry export and REGISTRY.md regenerated, `registry export --check` up to date); a SPEC §7 bullet points to it. - Reference: the `wai-rs` module `conform_result` (feature `conform_result`, now in `quickstart`); `wai conform` signs and `wai verify` checks through it. It reads records with the crate's strict JSON reader (a member named twice, an unpaired surrogate, a number written other than as a plain integer are refused) and verifies with `wai::sig::verify_strict`. - Corpus [`conform-result-conformance/`](wai/conform-result-conformance/): accepted records (conformant; not conformant; no cases; HTTP adapter with a supplied key) and refused ones (a case flipped after signing, the corpus digest changed, the summary alone changed, a bad signature, `S + L`, a small-order key, each summary count wrong alone and two at once, a verdict that disagrees, an unknown version, a member missing, added or named twice, a fractional count, an upper-case digest, bytes that are not JSON). Every outcome is stated in the generator from the text, and the generator asserts the reference reaches it before writing; `wai_conform_result_vectors gen|verify|verdicts`. - Second implementation `conform_result_verify.py` (standard library, `python3 -I`, WAI's strict Ed25519 rule from `claim-conformance/_ed25519.py`, its own JSON reading and RFC 8785 serialisation), which reaches every outcome, reads §5's table, and with `--skip <check>` fails the corpus for each check of §4 left out. CI step in `wai-conformance.yml`'s independent-verifiers job (feature added to its union and to features-alone; the extension added to its paths). `wai conform` gains the verifier suite `conform-result`, and the runner job runs the Python verifier through it. - **Measurements are not compared**: `wai breaking` leaves out an explicit list (`breaking/corpora.rs` `MEASUREMENTS`, one entry a file, each with its reason) and reports each such change as a note: `wai/format-probe-conformance/provenance.json` (the external decodes measured on one build host; no verifier reads it) and `wai/live-conformance/metering/relay-record.wmh` and `relay-ledger.wml` (one live loopback run's recording; the vectors built around it in `vectors.json` stay compared). A bin test requires every listed file to exist, once, with a reason. The corpora are otherwise deterministic by design (their READMEs state the same bytes on every machine), so the list is short. - A command `wai conform` runs is killed alone at its timeout, and its output is read for at most two seconds after it exits, so a child that keeps the output open does not hold the run. (An earlier commit killed the command's process group with `kill -KILL -<pgid>`; in the CI image that call signals the caller's own group, which killed the runner job with exit 137. Reproduced in that image and removed; the step peaks at 4.6 GB built cold under a 16 GB cap, so memory was not the cause.) - `tests/int_kernels_vectors.rs` (every corpus has a decoded-output sidecar or a stated exclusion) lists `conform-result-conformance` with its reason: signed records, no decoded output. - Docs: `/docs/breaking/` and `/docs/conform/` on the WAI site's `docs-quickstart` branch (head `0d217b2`), each with a `.md` twin and an `/llms.txt` entry; the conform page links the record's specification. Not deployed. ## Design decisions - **Merge base, not the ref's tip**: a branch is not charged with removing what main gained after it forked. - **Acknowledgement only from added changelog lines**, so an old entry never re-licenses a break; the entry must say "breaking" and the id must be backticked. Corpus findings are per file, so one entry acknowledges a regenerated corpus file. - **Record preimages are checked twice over**: statically (domain/magic/version constants), and through the corpora, which the conformance workflow already holds byte-identical to the generating code. A preimage change in code reaches a corpus file or fails that workflow; `wai breaking` reads the file. - **Rust API from rustdoc JSON read as plain JSON** (no `rustdoc-types` version pin), both sides from one toolchain; a format-version mismatch refuses to compare. - **Additive registry members** (a capability gaining an `ops` declaration it never had) are compatible; changed or removed ones break. - **Plain HTTP only** for the HTTP adapter in this build (no TLS client in `quickstart`). ## Checks | check | command | host | result | |---|---|---|---| | tests, both commands | `cargo test --locked --no-default-features --features quickstart,score,key_discovery --bin wai --test cli_breaking --test cli_conform` | Studio | all pass, five runs in a loop | | existing dx tests | `cargo test --locked --no-default-features --features quickstart --bin wai --test floor --test cli_dx`; `cargo test --no-default-features --bin wai` | Studio | pass | | clippy | `cargo clippy --locked --no-default-features [--features quickstart \| quickstart,score,key_discovery] --bin wai --tests` | Studio | no note in the new files (the pre-existing notes in `wai.rs`, `http*.rs` are untouched) | | build, no feature | `cargo build --locked --no-default-features --bin wai` | Framework (Linux) | ok | | gate against main | `wai breaking --against <main> --text` (all surfaces, wai documented twice) | Framework, Studio | no finding at all: the same public surface on both sides | | wai-quantum documented | a comment appended to `wai-quantum/src/lib.rs`, `--only rust` | Studio | both crates documented, no finding, 61 s | | conform, independent verifiers | `wai conform verifier --suite {claim,jwp,signer} --impl "python3.11 -I wai/<s>-conformance/<s>_verify.py verdicts {corpus}"` | Studio | every case passed, each conformant | | conform, reference packager | `wai conform packager --suite policy --impl "wai-pack policy-verdict {in}"` | Studio | every case passed | | conform, reference origin / renderer / Rust verifiers / HTTP verifier | in `cli_conform` | Studio | conformant | | record corpus and verifiers | `cargo test --features conform_result --lib conform_result::` and `--test conform_result`; `wai_conform_result_vectors verify ../conform-result-conformance`; `python3 -I conform-result-conformance/conform_result_verify.py` | Framework (Linux), local | all pass; every outcome reached by both | | registry | `cargo test --lib codecs::registry::` (wire identifiers in their documents); `wai registry export --check` | Framework | pass; up to date | | tests after the additions | `cargo test --no-default-features --features quickstart,score,key_discovery --bin wai --test cli_breaking --test cli_conform --test cli_dx --test conform_result` | Framework (Linux) | all pass | | gate on this branch | `wai breaking --against <main> --text`, every surface | Framework | no unacknowledged break; the new module, corpus and registry row compatible | | site after the link | `npm run build`; `pages.e2e /docs/conform/` | Studio | ok; page checks all passed | | repo checks | `check-tests-in-ci.py`, `check-workflow-concurrency.sh`, `check-spec-refs.py`, `check-wai-section-refs.py`, `check-no-vanity-metrics.sh`, `test-public-text-checks.sh`, `check-comparative-prose.sh`, `check-orphan-sources.py`, YAML parse | local | all pass | | public names | `check-public-names.sh` on every changed file; commit messages and this body with `grep -E` | local | clean | | site | `npm run build` (routes, public text with the denylist, names); `node e2e/pages.e2e.mjs /docs/breaking/ /docs/conform/` | Studio | build ok; 307 page checks passed | ## Falsifications On this repository against main (each: exit 1 without an acknowledgement, exit 0 with a `- **Breaking:** \`<id>\`` line added): 1. **A registered code removed**: `wai.audio.flac` dropped from the registry export → `wire:capability:wai.audio.flac`; the magic `WCT1` dropped → `wire:wire-identifier:magic/WCT1`. 2. **A signing preimage changed**: `live_events.rs` `SIGN_DOMAIN` `\x01`→`\x02` → `record:domain:wai/wai-rs/src/live_events.rs::SIGN_DOMAIN`. And in code: `access_services::signing_bytes` appends a byte, the corpus regenerated with `wai_access_services_vectors gen` → `record:corpus:wai/live-conformance/access/vectors.json`; `access_services_verify.py` refuses the regenerated corpus (exit 1), and passes again once restored. 3. **A public Rust function removed**: `energy_order::is_learned` made private → `api:rust:wai::energy_order::is_learned` ("the function fn is_learned(&str) -> bool is gone"). 4. **A C function's signature changed**: `wai_envelope_detect_version`'s `len: usize` → `u32` in `ffi.rs`, the header regenerated by `build.rs` → `api:c:wai_envelope_detect_version` (`uintptr_t` → `uint32_t`). 5. **An OpenAPI route removed** (on PR #500's branch, which holds the document): `GET /room` removed → `api:openapi:joule-moq-room:GET /room`; a required query parameter and a changed operationId → two more findings. Against main, every route of the document reads as compatible. `wai conform`: a verifier that changes one verdict fails exactly that case (exit 1; Python claim verifier and Rust signer reference); a packager that changes one digest digit fails every admitted policy and nothing else; a renderer that adds one to one output byte fails every score case; an origin serving one object with one byte flipped fails that object ("the body is not the object"). A record with a failed case turned to a pass and its counts made to agree, one with a corpus-digest byte changed, and one with only its summary changed are each refused by `wai verify` (exit 1). Guards mutated in the code, each caught by its test: an entry acknowledging without the word "breaking"; the record's signature not checked; C parameter types not compared. **`conform-result/1`'s verifiers** (on Linux): - `wai verify` on each vector's record alone: every refused vector exits 1 at its stated step and reason, every accepted one exits 0; no mismatch. - The Rust module with one check removed, each in turn (the signature; the summary's case count, its passed count, its failed count; the verdict; the members; the format; a byte added to the signing input): `wai_conform_result_vectors verify` fails every time. The first run found the case-count check unguarded (every summary vector had two counts wrong); three vectors with one count wrong each were added, and the rerun caught it. - `conform_result_verify.py --skip <check>` for each check of §4 (duplicates, format, members, signature, summary, verdict): the corpus fails each time, naming the vectors that guard it. **The measurement list**: a changed `provenance.json` is a note and no finding (exit 0); with the list emptied, the same change is `wire:corpus:wai/format-probe-conformance/provenance.json` (exit 1). `cli_breaking` tests a listed file not compared and an unlisted vector beside it still breaking. ## Not done - **Kinds not covered**: `relay` (the MOQT interop cases live in `joule-moq-interop`; no runner adapter yet) and `player` (browser sink). Decoder suites other than `score` (audio needs its sources beside the scene; int-payload, neural, film and the others need their output forms named per capability). - HTTPS for the HTTP adapter; `--format junit|html`; `--level`, `--independent`, `wai conformance gen`; `breaking.allow` in `wai.json`. - API surfaces the catalogue will add (catalogue IDs and flags, CLI JSON fields, MCP tool schemas); the JS, Swift and Kotlin APIs. - The operation catalogue (`wai ops`) and the MCP tools `compat_check`/`conformance_run`: registration waits for those branches; both ops are separable modules. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 4 commits 2026-10-27 05:23:27 -04:00

wai breaking and wai conform: the compatibility gate, and conformance anyone can run ... 482244fdce

`wai breaking --against <git-ref>` (operation compat.check) reports what the working
tree breaks against the merge base of a ref, in three cumulative categories: wire (the
registry export, the media and delivery corpora), record (the signing domains, magics
and versions in the WAI crates' sources, the record corpora, the published JSON
Schemas) and API (the Rust public API of wai and wai-quantum from rustdoc's JSON, the
C header, the room service's OpenAPI document). Each finding names its item, category,
what changed and why it breaks; additions are compatible. A break passes only when a
line the change adds to wai/CHANGELOG.md, in an entry that says Breaking, names its id.

`wai conform <kind> --suite <suite> --impl <command or URL>` (operation
conformance.run) runs a corpus against another implementation through a command-line
or an HTTP adapter: verifiers (claim, jwp, signer), a decoder (score), a packager
(policy) and an origin (http). The result is a signed conform-result/1 record that
`wai verify` checks. Both follow the agent rules: the envelope/1 JSON output, the exit
statuses, --dry-run.

.github/workflows/wai-breaking.yml runs the gate on every pull request that touches
WAI, the tests of both commands, and the corpora's independent Python verifiers and
the reference packager through wai conform.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai breaking: a registry string reads as itself in a finding; CI builds the tools unoptimised ... 18de7d357e

Co-Authored-By: Claude Opus 5.5 <[email protected]>

CI: wai breaking reads the checkout as its owner and fetches main's tip into origin/main ... b90e9aaca2

Co-Authored-By: Claude Opus 5.5 <[email protected]>

tests/cli_conform.rs: the origin picks its own port and says which ... 63481e6917

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request from a commit 2026-10-27 05:40:33 -04:00

Merge TC-B's wai/breaking-conform (#507) into the catalogue: one agent-rules module

dcharlot added 2 commits 2026-10-27 05:48:47 -04:00

conformance-results: conform-result/1 in the standard, its corpus and a second verifier ... 92ab88a375

extensions/conformance-results.md specifies the record a conformance run signs: its
members (§2) and corpus digest (§2.1), its signing input (the domain
`wai:conform-result` and 0x01, then the record without `sig` in RFC 8785 form) and
Ed25519 signature under SPEC §7's strict rule (§3), what a verifier checks and in what
order (§4: it reads, its version, its members, its signature, its summary, its verdict)
and the codes of each refusal (§5). conform-result/1 is registered as a document kind
(registry export and REGISTRY.md regenerated); SPEC §7 points to it.

wai-rs gains the module conform_result (feature conform_result, in quickstart): sign,
check and verdict, the corpus digest, and the corpus of §7, whose outcomes are stated
from the text and asserted before anything is written. `wai conform` signs with it and
`wai verify` checks with it. conform-result-conformance/ holds accepted records and
refused ones (a case flipped, the corpus digest changed, the summary alone changed, a
bad signature, S past L, a small-order key, a summary or verdict that disagrees though
signed, an unknown version, members missing, added or named twice, a fractional count,
a digest out of form, bytes that are not JSON); wai_conform_result_vectors generates
and verifies it, and conform_result_verify.py, written from the text and sharing no code
with wai-rs, reaches every outcome and fails the corpus with each check of §4 left out.
`wai conform` gains the verifier suite conform-result.

`wai breaking` leaves out an explicit list of corpus files that record a measured run
(the format probes' external decodes, the relay's recorded metering run), each with
its reason, and says so in a note; the vectors built from them stay compared.

A command `wai conform` runs is put in a process group of its own, so a timed-out
command's children are stopped too.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

conform-result-conformance: each summary count wrong alone, its other counts right ... 20e94861ee

The falsification of the reference found that leaving out the check of summary.cases
alone left the corpus passing: every summary vector had passed and failed wrong too.
Three vectors now carry each count wrong alone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-10-27 06:08:00 -04:00

wai: one operation catalogue (noun.verb) for every door, wai ops, the agent rules on every command, and the wai-* service binaries #506

dcharlot added 1 commit 2026-10-27 08:24:11 -04:00

wai conform: no process-group kill; the sidecar test names the record corpus ... 7e4be9405c

The runner job died with exit 137 the moment the adapter test's 200 ms timeout fired.
In the CI image (Ubuntu 22.04's procps kill), `kill -KILL -<pgid>` signals the
caller's own process group, not the one named: the test, cargo and the step's shell
were killed, and the timed-out `sh` and `sleep` survived. Reproduced in that image
with a small program; the same call with no such group also killed the caller. It was
not memory: the step, built cold and run in that image under a 16 GB cap, peaks at
4.6 GB. The adapter now kills the shell alone, as before, and reads output for at most
two seconds after it exits, so a child that keeps the output open no longer holds the
run; the command stays in the caller's process group.

tests/int_kernels_vectors.rs walks every wai/*-conformance/ directory and requires a
sha256.json sidecar of decoded outputs, or an exclusion with its reason.
conform-result-conformance holds signed records and no decoded output, so it is listed
with that reason, as the other record corpora are.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot force-pushed wai/breaking-conform from 7e4be9405c to 4484518afd 2026-10-27 11:36:14 -04:00 Compare

dcharlot added 2 commits 2026-10-27 14:35:53 -04:00

wai breaking: the base holds every standard its crate reaches by path ... 59fa15f254

The Rust surface extracted only wai/ from the base. Once wai depended by
path on joule-ui's crates (wai-app), documenting the base failed: the
dependency's directory was missing, and a crate in another workspace also
needs that workspace's root manifest for the fields it inherits. Each
top-level directory in the base crate's path closure is now extracted
whole, once.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge remote-tracking branch 'origin/main' into wai/breaking-conform 2b1439c4c2

dcharlot added 1 commit 2026-10-28 01:57:14 -04:00

Merge remote-tracking branch 'origin/main' into wai/breaking-conform ... 55e1cf5446

# Conflicts:
#	.github/workflows/wai-conformance.yml
#	wai/CHANGELOG.md
#	wai/wai-rs/Cargo.toml
#	wai/wai-rs/src/bin/wai_cli/dx/mod.rs
#	wai/wai-rs/src/bin/wai_cli/dx/verify.rs
#	wai/wai-rs/src/lib.rs

dcharlot added 1 commit 2026-10-28 03:21:22 -04:00

Merge remote-tracking branch 'origin/main' into wai/breaking-conform 0ccd848ca1

dcharlot merged commit 25d38c4803 into main 2026-10-28 05:08:42 -04:00

dcharlot referenced this pull request from a commit 2026-10-28 05:08:43 -04:00

Merge pull request 'wai breaking and wai conform: the compatibility gate on every WAI pull request, and conformance runs with signed results' (#507) from wai/breaking-conform into main

dcharlot referenced this pull request from a commit 2026-10-28 05:49:51 -04:00

Merge origin/main into wai/ops-catalogue: #507, #518 and #521 as they landed

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!507

No description provided.