WLK1 for the link record, wai identify, the carousel schedule and holdings (replaces #604 and #605) #611

Merged

dcharlot merged 8 commits from wai/inspect-carousel-holdings-b into main 2026-11-04 07:18:17 -05:00

Conversation 0 Commits 8 Files changed 638 +44442 -171

dcharlot commented 2026-11-04 05:21:33 -05:00

Owner

Copy link

This one pull request replaces #604 and #605. Each of them was green apart from one check that this change does not touch: #604 failed the GPU energy test (it fails when other processes on the shared runner host drop GPU clients) and #605 failed the co-watch browser scenario (a room page bug); both are fixed on a separate branch whose PR follows. They could not both merge, because each adds an entry at the head of wai/CHANGELOG.md, so whichever landed second would conflict. Here they sit in one linear branch on current main, in order: #604's two commits, then #605's six, unchanged.

How this branch was built. git cherry-pick of #604's commits (2729806f6) and then #605's (c303cf55d) onto main c5c418e33. The only conflicts were at the head of wai/CHANGELOG.md, resolved by keeping every entry. Every file other than wai/CHANGELOG.md is byte-identical to git merge-tree of main, #604 and #605, and the CHANGELOG only gains lines (+193, none removed), each entry once. check-wai-section-refs.py, check-spec-refs.py and check-tests-in-ci.py pass on the combined tree.

Part 1, from #604: live-contribution: the link record's magic is WLK1, no longer the clip record's WCL1

The contribution link record and the live-recording clip record both began with the 4-byte magic WCL1. The clip record had it first (live-recording §5.1, 2026-10-17, and the only WCL1 row in the registry); the link record took the same bytes when live-contribution was added (2026-10-25). A reader that tells records apart by their magic could not tell one from the other.

This gives the later one, the link record, its own magic, WLK1. WCL1 stays the clip record's. WLK1 collides with no magic in REGISTRY.md, in the code or texts of origin/main, or in any open PR's branch.

Wire change. A link record written before this change is refused as malformed by a reader of this revision (live-contribution §6.5 step 2), and a new one by a reader of the earlier revision. The signature's domain (wai:contrib-link 0x01) and every other field are unchanged.

What changed:

• wai/extensions/live-contribution.md: the header and §6.2 name WLK1. §6.5 step 2 says the clip record's WCL1 is one of the other magics refused as malformed. SPEC.md's summary names WLK1.
• wai/wai-rs/src/codecs/registry.rs: a wire row WLK1, magic, live-contribution §6.2. REGISTRY.md and wai/registry/ are regenerated from it by wai registry export.
• joule-moq-ingest: linkrec::MAGIC is WLK1, and the doc comments, README and example config say so. The generator writes .wlk1 files and adds clip_record_magic, a part that bears the clip record's magic and is refused malformed.
• wai/live-conformance/links/: regenerated byte for byte by joule-moq-ingest-link-vectors --out on the build host. The .wcl1 files are replaced by .wlk1 ones, and the README is updated.
• wai/live-conformance/live_contribution_verify.py: reads WLK1. It also mutates each accepted part to the clip record's magic and requires that copy to be refused.
• CHANGELOGs: wai/CHANGELOG.md has an entry under Unreleased that calls this a wire change and acknowledges each corpus finding wai breaking reports. joule-code/CHANGELOG.md has a matching entry for joule-moq-ingest.

The reverse direction (second commit): a record with the link record's magic is not a clip record.

• wai/wai-rs/src/live_record/corpus.rs: the live-recording corpus gains the clip link_record_magic, chunk 4's clip record beginning WLK1 and signed under it, so the magic is its one defect. It is refused malformed at the clip. wai/live-conformance/recordings/vectors.json was regenerated byte for byte by wai_live_record_vectors gen on the build host; the only change is that vector, appended.
• wai/extensions/live-recording.md §5.2 names WLK1 among the other magics a clip reader refuses as malformed. The recordings README lists the new vector.
• wai/live-conformance/live_record_verify.py: reads the clip record's magic from a constant, and requires that a copy of each accepted clip (9 of them) given WLK1 is refused malformed at the clip, so a verifier that reads past the magic to the signature fails.
• wai/CHANGELOG.md: a bullet in the WLK1 entry.

A falsification run on the build host used a copy of live_record_verify.py whose clip reader skips the magic. It failed, exit 1: it accepted link_record_magic and failed the new check.

Run on the macOS build host at the first commit, each exiting 0:

• cargo run --locked -q -p joule-moq-ingest --bin joule-moq-ingest-link-vectors -- --check ../../wai/live-conformance/links (102 files, byte for byte)
• cargo test --locked -p joule-moq-ingest --lib --test link_corpus
• python3 -I live-conformance/live_contribution_verify.py live-conformance/links
• cargo run -q --no-default-features --bin wai -- registry export --check
• cargo test -q --no-default-features --lib codecs::registry
• wai breaking --against origin/main --text --all: no unacknowledged breaking change

Run on the macOS build host at the second commit (2729806f6), each exiting 0:

• cargo run --locked -q --no-default-features --features live_record --bin wai_live_record_vectors -- verify ../live-conformance/recordings (regenerates byte for byte; every vector gets its outcome)
• cargo test --locked -q --no-default-features --features live_record --lib -- live_record:: output_tree:: (15 passed) and --test live_record (5 passed)
• python3 -I live-conformance/live_record_verify.py live-conformance/recordings (27 recordings, 35 clips)
• python3 -I live-conformance/live_moderation_verify.py live-conformance/moderation, which reads clips with live_record_verify.py
• wai breaking --against 6c8016b61 --text --all: no unacknowledged breaking change

Run locally at both commits (text only): scripts/check-public-names.sh, check-wai-section-refs.py, check-spec-refs.py, check-comparative-prose.sh and check-no-vanity-metrics.sh, all clean.

Part 2, from #605: wai identify, the carousel schedule and holdings

This replaces #589, #592 and #595, closed because main moved under them: wai/CHANGELOG.md's Unreleased head conflicted with main's newer entries (from #591's tariff version 6 on), and a stacked branch can neither be force-pushed nor carry a merge commit. The six commits here are theirs, cherry-picked in stack order onto main at 6c8016b61, with messages unchanged and no merge commit: #589's three, #592's carousel commit, the carousel §11 Patents commit (f64d8fb53: the IETF disclosures on RaptorQ and Raptor, and no-code as the default), which was #592's last push, about seven minutes before it closed; its CI there never finished, so this pull request's CI is its first complete run, and #595's holdings commit. Three commits on the stacked branches carried #589's second and third commits again under the same patch IDs (9680914bb, 716dfd41d, f356702ad), so they were skipped.

Conflicts. Only one: wai/CHANGELOG.md, when #589's first commit was picked. This branch's entries go first under Unreleased, newest first (holdings, carousel, identify), above main's entries, which are kept as they are. Main had not changed the shared lists since the stack's base (WAI_FEATURES, the union build's --test and --bin lists, the feature-alone loops, codecs/registry.rs, REGISTRY.md, the registry and vocab exports, catalogue/ops.json), so they took the branch's entries in their stacking positions with no conflict. wai-rs/Cargo.toml merged cleanly: main's receipt_ledger line keeps exact_upscale and exact_tonemap beside the new identify, carousel and holdings features. No generated file needed regenerating: both exports check as current at the head.

Re-run at the head (c303cf55d). These ran on the macOS build host, exactly as the CI steps write them unless a note says otherwise, and all passed:

• wai registry export --check and wai ops export --check ../catalogue/ops.json.
• The carriage step, with cli_registry and corpus_provenance.
• The library unit tests step (902 tests, among them codecs::registry:: and the vocab tests).
• Each feature alone: cargo check --lib --features identify, its wasm32 build, and cargo check --tests with carousel and with holdings.
• Media identification: lib identify:: (17), tests/identify (6) and wai_identify_vectors verify, then identify_verify.py: 429 vectors, 0 failures, with every mutated copy failing.
• cli_mcp, cli_setup_agent, the binary's own tests and lib catalogue::.
• int_kernels_vectors (8 tests; not the CI step's full test list).
• Carousel: lib carousel:: claim_receipt::schedule:: (14) and tests/carousel (7). carousel_verify.py gave 99 cases with 0 failures, 10 of 10 mutated copies failed, and 14 of 14 weakened verifiers were caught. claim_verify.py carousel checked 134 claims with 0 failures.
• Holdings: lib holdings:: (5) and tests/holdings (3). holdings_verify.py gave 39 cases with 0 failures, 9 of 9 mutated copies failed, and 10 of 10 weakened verifiers were caught.
• Bearer receipts, since claim_verify.py and bearer-receipts.md changed: tests/bearer_receipts, bearer_verify.py, and claim_verify.py bearer (72 claims).
• claim_verify.py on its own corpus (636 vectors) and in consumption mode (27 claims).
• http_verify.py, since http_receipts.py changed: 1057 of 1057 cases.
• scripts/check-lockfiles.sh: 60 lockfiles current.

These ran locally and also passed:

• A step-by-step comparison of the workflow against main and the three source branches (no step body new to this branch).
• ruby -ryaml and yaml.safe_load of wai-conformance.yml.
• check-public-names.sh, check-wai-section-refs.py, check-tests-in-ci.py, check-workflow-concurrency.sh and check-wai-ops-doors.py.

wai identify and media.inspect (was #589)

wai inspect reads envelopes. Most files a toolchain meets are not envelopes, and until now the answer to "what is this file?" came from a file's extension or from a third-party prober. This adds wai identify <file> (catalogue operation media.inspect, agent tool media_inspect): what a file is, from its bytes alone.

What it says

• The formats the bytes fit. Each candidate has the offsets that say so and how far its structure was followed: signature, structure or parameters. There is no confidence figure, because the strength says what was checked.
• A container's tracks: each with its codec, size and protection, as the container states them.
• The registered capabilities those codecs map to (media-identify §5.1).
• Signatures embedded inside the file (§7).
• When no format is recognised, measures that still hold: how its byte values are spread (chi-square and uniform windows) and the lags at which it repeats (§8).
• verdict: identified, ambiguous, signature_only or unknown. A walk that stops says why: overrun, invalid, crc, limit or unknown_size.

Formats it walks: ISO media, Matroska, transport streams (with CRC-32), Ogg, FLAC, RIFF, MPEG audio, ADTS, PNG, JPEG, GIF, IVF, YUV4MPEG2, FLV, AV1 OBU, Annex B streams, and HLS and DASH text. It knows many more by signature, and the §6 and §7 tables are generated from the code into the spec.

Limits:

• Nothing is decoded and no protected track is opened.
• Cost is bounded for any file size: it reads 1 MiB at the head and scans at most 16 MiB, in at most 4,096 steps.
• The agent door marks results untrusted_content, because a file's author chooses the strings in it.

How it is checked

• Corpus: wai/identify-conformance/, 429 vectors.

  • About 290 cases are generated (including files repeated past the head and the scan window).
  • Files come from real muxers and encoders (tools/make_external.sh).
  • Each file has what an independent prober read of it (prober.json).
  • Negative controls are included.
  • wai_identify_vectors verify and tests/identify.rs require the corpus to regenerate byte for byte, with every result.

• Second implementation: identify_verify.py.

  • It is written from the extension's text and shares no code with wai-rs.
  • It reads the codec tables, bitrates, ADTS rates and H.264 profiles from the spec.
  • It gives every file its whole result, holds ext/ to the prober's reading, and requires every mutated copy to fail.
  • Result: 429 vectors, 429 files, 0 failures.

• Hostile bytes: prefix, flip and fuzz tests run through every walk. This found the AVS3 probe panic fixed in #586, which is identical here.

• Mutation testing: three rounds against the Rust walks. The last round caught 1,999 of 2,185 mutants. The survivors are mostly equivalent mutants: slice bounds that cannot be reached, caps the head or scan limit makes unreachable, and the repeat measure's initial values.

• Tests:

  • lib identify::: 17;
  • tests/identify: 6, covering the corpus, the §6 and §7 tables, every reason and verdict reached, the CLI equal to the library, and a directory or missing file being an error;
  • cli_ops: 10;
  • format_probe: 7.

  Clippy is clean, check-wai-section-refs and the public-name check pass, and the ops export is current.

• CI:

  • The conformance workflow checks the identify feature alone and builds it for wasm32.
  • It adds the test and vector binary to the union build.
  • It runs identify_verify.py.
  • A change to media-identify.md triggers it.

The carousel schedule (was #592)

On a bearer with no return path (broadcast, satellite, one-way multicast), a receiver can't ask for what it missed. It holds an object only if the passes it heard carried enough encoding symbols. WAI could declare such a transmission (bearer-receipts) and reconcile it afterwards, but it had no signed statement of the plan in between: when each object passes, and with what forward error correction. Without one, a one-way bearer can't promise a deadline that anyone can check, and a receiver can't plan when to listen.

This adds wai.bearer.carousel (extensions/carousel.md) with the jwp-receipts claim class schedule (§2.4.16).

The claim

• What it is: a publisher's signed plan, beside the group receipt and the instance's transmission claim, naming the group by its link hash.
• What each object's entry states:
  • its code: RaptorQ (RFC 6330 with its verified erratum 5548) or Compact No-Code (RFC 5445);
  • that code's object transmission information;
  • the repair symbols each pass adds;
  • whether later passes repeat their ESIs or send new ones;
  • the first pass's instant, the period, the duration and the count of passes.
• Step 4 holds each code's parameters to its bounds:
  • F up to 942,574,504,275;
  • T a multiple of Al, Z from 1 to 255, and N up to T/Al;
  • at most 56,403 source symbols per block, with RFC 6330's Partition[];
  • RFC 5052 §9.1 blocking within RFC 5445's 16-bit fields;
  • ESIs within their fields, passes that don't overlap, and at most 2^53 − 1 octets sent.
• Step 6 checks each entry against the group: subject, signer, range, first pass not before group_start, content hash, and length where the bytes are held.

What a verifier derives

• The pass after which an object is held, for a loss bound in parts per million. The answer holds for every pattern of loss within the bound.
  • A RaptorQ block counts as held at K′ + h distinct symbols, the count RFC 6330 §5.8 states its 10⁻² / 10⁻⁴ / 10⁻⁶ decoding-failure bounds at.
  • A symbol that a repeating pass missed is counted as missed again, so repeat and Compact No-Code never count on a second pass.
  • This is the deadline a one-way bearer can promise, checked with no receiver.
• Statements that an object or a whole group is held by an instant: holds, availability_after_deadline, availability_not_within_schedule or object_not_scheduled.
• Schedules set aside when they contradict each other, or when their instance is undeclared or two-way.
• A receiver's listening plan: the passes that add symbols it is certain to keep, when it holds the object, and how long its radio must be on.

How it is checked

• Corpus (live-conformance/carousel/):
  • 33 verdict cases;
  • 50 claim cases, covering every reachable bound of both codes, both met and passed;
  • 16 plans.
• tests/carousel.rs, against an independent RFC 6330 implementation (the Apache-2.0 raptorq crate, added as a dev-dependency only):
  • K′ and Partition[] agree for every block size from 1 to 56,403;
  • every RaptorQ object a case holds available decodes from what a receiver within the case's loss bound holds by the end of that pass. In each pass the first ⌊L·m/10⁶⌋ symbols of each block are dropped, which in the first pass are source symbols, so the decoder must really decode.
• Second implementation: carousel_verify.py, written from the text.
  • It agrees on all 99 cases.
  • It catches 10 of 10 mutated copies of the corpus.
  • All 14 versions of it with one check removed are caught by the corpus.
• Claims: claim_verify.py carousel agrees with wai-rs on all 134 claims through steps 1–4. Its own corpus (636 vectors) and the bearer corpus (72 claims) still pass.
• Independent expectations: the corpus states its expected passes by hand and computes the octets sent by a different formula from the module's, never copying either from the module.
• HTTP delivery: a receipt index now checks a schedule claim against its group and listed objects, instead of accepting it after step 5. The rule is in http-delivery's text, in wai-rs and in http_receipts.py. The HTTP corpus has no schedule claim yet, so that branch is exercised only by compiling it.

Breaking

New variants in ClaimClass, ClaimBody and Reason; each is named in CHANGELOG.

Holdings (was #595)

Neighbours (a receiver that stores what a broadcast carried, a neighbourhood cache, a peer) can serve objects over a short link, but nothing said who holds what. This adds wai.delivery.holdings (extensions/holdings.md).

The record

WHD1 carries, for each group a node holds objects of:

• the group's link hash;
• its leaf count;
• either whole or a bitmap of the objects held, encoded as a reconciliation's bitmap.

The record also states an instant it holds at and one it expires at, and the holder signs it under SPEC §7 Signatures. Each holding has exactly one encoding.

It uses groups and bitmaps rather than a filter over hashes, for three reasons:

• WAI names objects by position under group receipts, and nodes hold objects as groups arrive, so holdings are dense within groups.
• A bitmap is exact: no neighbour is sent after an object its holder lacks.
• It costs a bit an object, and 41 bytes for a group held whole.

What a verifier derives

• Instant: a record before as_of or at its expiry is refused.
• Group receipts: a record that states a leaf count differently from a group receipt the verifier holds is refused whole. An entry for a group with no receipt held is kept.
• One holder's records: the latest stands. Two at one instant that differ are both set aside.
• Queries: held, object_not_held, object_out_of_range or group_not_advertised.
• An asker's plan: each wanted object's holders, in the asker's order, and the objects no neighbour holds, which come from the origin.

An object a neighbour serves is verified under its receipts like any other object; a record is never evidence of bytes. A node advertises only what reached it over a broadcast bearer, and may sign each record with a fresh key.

Checks

• Corpus: 21 records (a control for each rule and for the signature, including a small-order key), 11 verdicts, 6 queries, and a plan over three neighbours.
• Second implementation: holdings_verify.py, written from the text, agrees on all 39 cases, catches 9 of 9 mutated copies, and catches 10 of 10 weakened verifiers.
• wai-rs:
  • the full-feature build compiles;
  • unit tests pass (53 across holdings, carousel and claims), and the holdings and carousel tests pass;
  • clippy is clean on the new files;
  • the registry export is up to date;
  • wai breaking: no new breaking change.

🤖 Generated with Claude Code

This one pull request replaces #604 and #605. Each of them was green apart from one check that this change does not touch: #604 failed the GPU energy test (it fails when other processes on the shared runner host drop GPU clients) and #605 failed the co-watch browser scenario (a room page bug); both are fixed on a separate branch whose PR follows. They could not both merge, because each adds an entry at the head of `wai/CHANGELOG.md`, so whichever landed second would conflict. Here they sit in one linear branch on current main, in order: #604's two commits, then #605's six, unchanged. **How this branch was built.** `git cherry-pick` of #604's commits (2729806f6) and then #605's (c303cf55d) onto main c5c418e33. The only conflicts were at the head of `wai/CHANGELOG.md`, resolved by keeping every entry. Every file other than `wai/CHANGELOG.md` is byte-identical to `git merge-tree` of main, #604 and #605, and the CHANGELOG only gains lines (+193, none removed), each entry once. `check-wai-section-refs.py`, `check-spec-refs.py` and `check-tests-in-ci.py` pass on the combined tree. ## Part 1, from #604: live-contribution: the link record's magic is WLK1, no longer the clip record's WCL1 The contribution link record and the live-recording clip record both began with the 4-byte magic `WCL1`. The clip record had it first (live-recording §5.1, 2026-10-17, and the only `WCL1` row in the registry); the link record took the same bytes when live-contribution was added (2026-10-25). A reader that tells records apart by their magic could not tell one from the other. This gives the later one, the link record, its own magic, `WLK1`. `WCL1` stays the clip record's. `WLK1` collides with no magic in REGISTRY.md, in the code or texts of origin/main, or in any open PR's branch. **Wire change.** A link record written before this change is refused as `malformed` by a reader of this revision (live-contribution §6.5 step 2), and a new one by a reader of the earlier revision. The signature's domain (`wai:contrib-link` 0x01) and every other field are unchanged. What changed: - `wai/extensions/live-contribution.md`: the header and §6.2 name `WLK1`. §6.5 step 2 says the clip record's `WCL1` is one of the other magics refused as `malformed`. SPEC.md's summary names `WLK1`. - `wai/wai-rs/src/codecs/registry.rs`: a wire row `WLK1`, magic, live-contribution §6.2. REGISTRY.md and `wai/registry/` are regenerated from it by `wai registry export`. - `joule-moq-ingest`: `linkrec::MAGIC` is `WLK1`, and the doc comments, README and example config say so. The generator writes `.wlk1` files and adds `clip_record_magic`, a part that bears the clip record's magic and is refused `malformed`. - `wai/live-conformance/links/`: regenerated byte for byte by `joule-moq-ingest-link-vectors --out` on the build host. The `.wcl1` files are replaced by `.wlk1` ones, and the README is updated. - `wai/live-conformance/live_contribution_verify.py`: reads `WLK1`. It also mutates each accepted part to the clip record's magic and requires that copy to be refused. - CHANGELOGs: `wai/CHANGELOG.md` has an entry under Unreleased that calls this a wire change and acknowledges each corpus finding `wai breaking` reports. `joule-code/CHANGELOG.md` has a matching entry for `joule-moq-ingest`. **The reverse direction** (second commit): a record with the link record's magic is not a clip record. - `wai/wai-rs/src/live_record/corpus.rs`: the live-recording corpus gains the clip `link_record_magic`, chunk 4's clip record beginning `WLK1` and signed under it, so the magic is its one defect. It is refused `malformed` at the clip. `wai/live-conformance/recordings/vectors.json` was regenerated byte for byte by `wai_live_record_vectors gen` on the build host; the only change is that vector, appended. - `wai/extensions/live-recording.md` §5.2 names `WLK1` among the other magics a clip reader refuses as `malformed`. The recordings README lists the new vector. - `wai/live-conformance/live_record_verify.py`: reads the clip record's magic from a constant, and requires that a copy of each accepted clip (9 of them) given `WLK1` is refused `malformed` at the clip, so a verifier that reads past the magic to the signature fails. - `wai/CHANGELOG.md`: a bullet in the `WLK1` entry. A falsification run on the build host used a copy of `live_record_verify.py` whose clip reader skips the magic. It failed, exit 1: it accepted `link_record_magic` and failed the new check. Run on the macOS build host at the first commit, each exiting 0: - `cargo run --locked -q -p joule-moq-ingest --bin joule-moq-ingest-link-vectors -- --check ../../wai/live-conformance/links` (102 files, byte for byte) - `cargo test --locked -p joule-moq-ingest --lib --test link_corpus` - `python3 -I live-conformance/live_contribution_verify.py live-conformance/links` - `cargo run -q --no-default-features --bin wai -- registry export --check` - `cargo test -q --no-default-features --lib codecs::registry` - `wai breaking --against origin/main --text --all`: no unacknowledged breaking change Run on the macOS build host at the second commit (2729806f6), each exiting 0: - `cargo run --locked -q --no-default-features --features live_record --bin wai_live_record_vectors -- verify ../live-conformance/recordings` (regenerates byte for byte; every vector gets its outcome) - `cargo test --locked -q --no-default-features --features live_record --lib -- live_record:: output_tree::` (15 passed) and `--test live_record` (5 passed) - `python3 -I live-conformance/live_record_verify.py live-conformance/recordings` (27 recordings, 35 clips) - `python3 -I live-conformance/live_moderation_verify.py live-conformance/moderation`, which reads clips with `live_record_verify.py` - `wai breaking --against 6c8016b61 --text --all`: no unacknowledged breaking change Run locally at both commits (text only): `scripts/check-public-names.sh`, `check-wai-section-refs.py`, `check-spec-refs.py`, `check-comparative-prose.sh` and `check-no-vanity-metrics.sh`, all clean. ## Part 2, from #605: wai identify, the carousel schedule and holdings This replaces #589, #592 and #595, closed because main moved under them: wai/CHANGELOG.md's Unreleased head conflicted with main's newer entries (from #591's tariff version 6 on), and a stacked branch can neither be force-pushed nor carry a merge commit. The six commits here are theirs, cherry-picked in stack order onto main at 6c8016b61, with messages unchanged and no merge commit: #589's three, #592's carousel commit, the carousel §11 Patents commit (f64d8fb53: the IETF disclosures on RaptorQ and Raptor, and no-code as the default), which was #592's last push, about seven minutes before it closed; its CI there never finished, so this pull request's CI is its first complete run, and #595's holdings commit. Three commits on the stacked branches carried #589's second and third commits again under the same patch IDs (9680914bb, 716dfd41d, f356702ad), so they were skipped. **Conflicts.** Only one: `wai/CHANGELOG.md`, when #589's first commit was picked. This branch's entries go first under Unreleased, newest first (holdings, carousel, identify), above main's entries, which are kept as they are. Main had not changed the shared lists since the stack's base (`WAI_FEATURES`, the union build's `--test` and `--bin` lists, the feature-alone loops, `codecs/registry.rs`, `REGISTRY.md`, the registry and vocab exports, `catalogue/ops.json`), so they took the branch's entries in their stacking positions with no conflict. `wai-rs/Cargo.toml` merged cleanly: main's `receipt_ledger` line keeps `exact_upscale` and `exact_tonemap` beside the new `identify`, `carousel` and `holdings` features. No generated file needed regenerating: both exports check as current at the head. **Re-run at the head (c303cf55d).** These ran on the macOS build host, exactly as the CI steps write them unless a note says otherwise, and all passed: - `wai registry export --check` and `wai ops export --check ../catalogue/ops.json`. - The carriage step, with `cli_registry` and `corpus_provenance`. - The library unit tests step (902 tests, among them `codecs::registry::` and the vocab tests). - Each feature alone: `cargo check --lib --features identify`, its wasm32 build, and `cargo check --tests` with `carousel` and with `holdings`. - Media identification: `lib identify::` (17), `tests/identify` (6) and `wai_identify_vectors verify`, then `identify_verify.py`: 429 vectors, 0 failures, with every mutated copy failing. - `cli_mcp`, `cli_setup_agent`, the binary's own tests and `lib catalogue::`. - `int_kernels_vectors` (8 tests; not the CI step's full test list). - Carousel: `lib carousel:: claim_receipt::schedule::` (14) and `tests/carousel` (7). `carousel_verify.py` gave 99 cases with 0 failures, 10 of 10 mutated copies failed, and 14 of 14 weakened verifiers were caught. `claim_verify.py carousel` checked 134 claims with 0 failures. - Holdings: `lib holdings::` (5) and `tests/holdings` (3). `holdings_verify.py` gave 39 cases with 0 failures, 9 of 9 mutated copies failed, and 10 of 10 weakened verifiers were caught. - Bearer receipts, since `claim_verify.py` and bearer-receipts.md changed: `tests/bearer_receipts`, `bearer_verify.py`, and `claim_verify.py bearer` (72 claims). - `claim_verify.py` on its own corpus (636 vectors) and in `consumption` mode (27 claims). - `http_verify.py`, since `http_receipts.py` changed: 1057 of 1057 cases. - `scripts/check-lockfiles.sh`: 60 lockfiles current. These ran locally and also passed: - A step-by-step comparison of the workflow against main and the three source branches (no step body new to this branch). - `ruby -ryaml` and `yaml.safe_load` of `wai-conformance.yml`. - `check-public-names.sh`, `check-wai-section-refs.py`, `check-tests-in-ci.py`, `check-workflow-concurrency.sh` and `check-wai-ops-doors.py`. ### `wai identify` and `media.inspect` (was #589) `wai inspect` reads envelopes. Most files a toolchain meets are not envelopes, and until now the answer to "what is this file?" came from a file's extension or from a third-party prober. This adds `wai identify <file>` (catalogue operation `media.inspect`, agent tool `media_inspect`): what a file is, from its bytes alone. #### What it says - **The formats the bytes fit.** Each candidate has the offsets that say so and how far its structure was followed: `signature`, `structure` or `parameters`. There is no confidence figure, because the strength says what was checked. - **A container's tracks:** each with its codec, size and protection, as the container states them. - **The registered capabilities** those codecs map to (media-identify §5.1). - **Signatures embedded inside the file** (§7). - **When no format is recognised,** measures that still hold: how its byte values are spread (chi-square and uniform windows) and the lags at which it repeats (§8). - **`verdict`:** `identified`, `ambiguous`, `signature_only` or `unknown`. A walk that stops says why: `overrun`, `invalid`, `crc`, `limit` or `unknown_size`. **Formats it walks:** ISO media, Matroska, transport streams (with CRC-32), Ogg, FLAC, RIFF, MPEG audio, ADTS, PNG, JPEG, GIF, IVF, YUV4MPEG2, FLV, AV1 OBU, Annex B streams, and HLS and DASH text. It knows many more by signature, and the §6 and §7 tables are generated from the code into the spec. **Limits:** - Nothing is decoded and no protected track is opened. - Cost is bounded for any file size: it reads 1 MiB at the head and scans at most 16 MiB, in at most 4,096 steps. - The agent door marks results `untrusted_content`, because a file's author chooses the strings in it. #### How it is checked - **Corpus: `wai/identify-conformance/`, 429 vectors.** - About 290 cases are generated (including files repeated past the head and the scan window). - Files come from real muxers and encoders (`tools/make_external.sh`). - Each file has what an independent prober read of it (`prober.json`). - Negative controls are included. - `wai_identify_vectors verify` and `tests/identify.rs` require the corpus to regenerate byte for byte, with every result. - **Second implementation:** `identify_verify.py`. - It is written from the extension's text and shares no code with wai-rs. - It reads the codec tables, bitrates, ADTS rates and H.264 profiles from the spec. - It gives every file its whole result, holds `ext/` to the prober's reading, and requires every mutated copy to fail. - Result: 429 vectors, 429 files, 0 failures. - **Hostile bytes:** prefix, flip and fuzz tests run through every walk. This found the AVS3 probe panic fixed in #586, which is identical here. - **Mutation testing:** three rounds against the Rust walks. The last round caught 1,999 of 2,185 mutants. The survivors are mostly equivalent mutants: slice bounds that cannot be reached, caps the head or scan limit makes unreachable, and the repeat measure's initial values. - **Tests:** - `lib identify::`: 17; - `tests/identify`: 6, covering the corpus, the §6 and §7 tables, every reason and verdict reached, the CLI equal to the library, and a directory or missing file being an error; - `cli_ops`: 10; - `format_probe`: 7. Clippy is clean, `check-wai-section-refs` and the public-name check pass, and the ops export is current. - **CI:** - The conformance workflow checks the `identify` feature alone and builds it for wasm32. - It adds the test and vector binary to the union build. - It runs `identify_verify.py`. - A change to `media-identify.md` triggers it. ### The carousel schedule (was #592) On a bearer with no return path (broadcast, satellite, one-way multicast), a receiver can't ask for what it missed. It holds an object only if the passes it heard carried enough encoding symbols. WAI could declare such a transmission (bearer-receipts) and reconcile it afterwards, but it had no signed statement of the plan in between: when each object passes, and with what forward error correction. Without one, a one-way bearer can't promise a deadline that anyone can check, and a receiver can't plan when to listen. This adds `wai.bearer.carousel` (extensions/carousel.md) with the jwp-receipts claim class `schedule` (§2.4.16). #### The claim - **What it is:** a publisher's signed plan, beside the group receipt and the instance's transmission claim, naming the group by its link hash. - **What each object's entry states:** - its code: RaptorQ (RFC 6330 with its verified erratum 5548) or Compact No-Code (RFC 5445); - that code's object transmission information; - the repair symbols each pass adds; - whether later passes repeat their ESIs or send new ones; - the first pass's instant, the period, the duration and the count of passes. - **Step 4** holds each code's parameters to its bounds: - F up to 942,574,504,275; - T a multiple of Al, Z from 1 to 255, and N up to T/Al; - at most 56,403 source symbols per block, with RFC 6330's Partition[]; - RFC 5052 §9.1 blocking within RFC 5445's 16-bit fields; - ESIs within their fields, passes that don't overlap, and at most 2^53 − 1 octets sent. - **Step 6** checks each entry against the group: subject, signer, range, first pass not before `group_start`, content hash, and length where the bytes are held. #### What a verifier derives - **The pass after which an object is held,** for a loss bound in parts per million. The answer holds for every pattern of loss within the bound. - A RaptorQ block counts as held at K′ + h distinct symbols, the count RFC 6330 §5.8 states its 10⁻² / 10⁻⁴ / 10⁻⁶ decoding-failure bounds at. - A symbol that a repeating pass missed is counted as missed again, so `repeat` and Compact No-Code never count on a second pass. - This is the deadline a one-way bearer can promise, checked with no receiver. - **Statements** that an object or a whole group is held by an instant: holds, `availability_after_deadline`, `availability_not_within_schedule` or `object_not_scheduled`. - **Schedules set aside** when they contradict each other, or when their instance is undeclared or two-way. - **A receiver's listening plan:** the passes that add symbols it is certain to keep, when it holds the object, and how long its radio must be on. #### How it is checked - **Corpus** (`live-conformance/carousel/`): - 33 verdict cases; - 50 claim cases, covering every reachable bound of both codes, both met and passed; - 16 plans. - **`tests/carousel.rs`**, against an independent RFC 6330 implementation (the Apache-2.0 `raptorq` crate, added as a dev-dependency only): - K′ and Partition[] agree for every block size from 1 to 56,403; - every RaptorQ object a case holds available decodes from what a receiver within the case's loss bound holds by the end of that pass. In each pass the first ⌊L·m/10⁶⌋ symbols of each block are dropped, which in the first pass are source symbols, so the decoder must really decode. - **Second implementation:** `carousel_verify.py`, written from the text. - It agrees on all 99 cases. - It catches 10 of 10 mutated copies of the corpus. - All 14 versions of it with one check removed are caught by the corpus. - **Claims:** `claim_verify.py carousel` agrees with wai-rs on all 134 claims through steps 1–4. Its own corpus (636 vectors) and the bearer corpus (72 claims) still pass. - **Independent expectations:** the corpus states its expected passes by hand and computes the octets sent by a different formula from the module's, never copying either from the module. - **HTTP delivery:** a receipt index now checks a schedule claim against its group and listed objects, instead of accepting it after step 5. The rule is in http-delivery's text, in wai-rs and in `http_receipts.py`. The HTTP corpus has no schedule claim yet, so that branch is exercised only by compiling it. #### Breaking New variants in `ClaimClass`, `ClaimBody` and `Reason`; each is named in CHANGELOG. ### Holdings (was #595) Neighbours (a receiver that stores what a broadcast carried, a neighbourhood cache, a peer) can serve objects over a short link, but nothing said who holds what. This adds `wai.delivery.holdings` (extensions/holdings.md). #### The record `WHD1` carries, for each group a node holds objects of: - the group's link hash; - its leaf count; - either `whole` or a bitmap of the objects held, encoded as a reconciliation's bitmap. The record also states an instant it holds at and one it expires at, and the holder signs it under SPEC §7 *Signatures*. Each holding has exactly one encoding. It uses groups and bitmaps rather than a filter over hashes, for three reasons: - WAI names objects by position under group receipts, and nodes hold objects as groups arrive, so holdings are dense within groups. - A bitmap is exact: no neighbour is sent after an object its holder lacks. - It costs a bit an object, and 41 bytes for a group held whole. #### What a verifier derives - **Instant:** a record before `as_of` or at its expiry is refused. - **Group receipts:** a record that states a leaf count differently from a group receipt the verifier holds is refused whole. An entry for a group with no receipt held is kept. - **One holder's records:** the latest stands. Two at one instant that differ are both set aside. - **Queries:** held, `object_not_held`, `object_out_of_range` or `group_not_advertised`. - **An asker's plan:** each wanted object's holders, in the asker's order, and the objects no neighbour holds, which come from the origin. An object a neighbour serves is verified under its receipts like any other object; a record is never evidence of bytes. A node advertises only what reached it over a broadcast bearer, and may sign each record with a fresh key. #### Checks - **Corpus:** 21 records (a control for each rule and for the signature, including a small-order key), 11 verdicts, 6 queries, and a plan over three neighbours. - **Second implementation:** `holdings_verify.py`, written from the text, agrees on all 39 cases, catches 9 of 9 mutated copies, and catches 10 of 10 weakened verifiers. - **wai-rs:** - the full-feature build compiles; - unit tests pass (53 across holdings, carousel and claims), and the holdings and carousel tests pass; - clippy is clean on the new files; - the registry export is up to date; - `wai breaking`: no new breaking change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 8 commits 2026-11-04 05:21:33 -05:00

live-contribution: the link record's magic is WLK1, no longer the clip record's WCL1 ... 57e426c940

The contribution link record and the live-recording clip record both began WCL1. The clip
record had it first (live-recording §5.1, the registry's only WCL1 row); the link record took
the same four bytes when it was added, so a reader that tells records apart by their magic
could not tell one from the other.

The link record is now WLK1, a magic nothing else in the registry, the code or an open branch
uses. live-contribution §6.2 and SPEC name it, §6.5 step 2 says the clip record's WCL1 is
refused as malformed, and codecs/registry.rs registers it (REGISTRY.md and registry/ exported
from it). joule-moq-ingest writes and reads it. The corpus live-conformance/links/ is
regenerated with it, its files now .wlk1, plus clip_record_magic, a part with the clip
record's magic, refused. live_contribution_verify.py reads WLK1 and also refuses each accepted
part mutated to WCL1.

A wire change: a link record written before this one is malformed to a reader of this
revision. The CHANGELOG says so and acknowledges every corpus file that changed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

live-recording: a record with the link record's magic, WLK1, is not a clip record ... 673554abf0

The link corpus refuses a part that bears the clip record's magic; this adds the reverse.
The live-recording corpus gains the clip link_record_magic: chunk 4's clip record beginning
WLK1 and signed under it, so the magic is its one defect, refused malformed at the clip
(§5.2, which now names WLK1 among the other magics). wai_live_record_vectors writes it;
the README lists it.

live_record_verify.py requires the same refusal, malformed at the clip, of a copy of each
accepted clip given WLK1, so a verifier that reads past the magic to the signature fails.
The CHANGELOG entry for WLK1 says so.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai identify: what any file is, from its bytes alone ... 2fd26925d3

`wai inspect` reads envelopes; most files a toolchain meets are not envelopes, and the
first question about one is what it is. `wai identify <file>` (operation media.inspect,
extensions/media-identify.md) answers it for any file:

- the formats the bytes fit, each with the offsets of the bytes that say so, how far its
  structure was followed and where and why that stopped;
- a container's tracks as it states them: codec, size, channels, rate, protection;
- the registered capabilities they map to, and what a format probe reads of them;
- signatures found inside the file;
- where no format is recognised, measures that still hold: how the byte values are
  spread, uniform windows, and the lags at which the bytes repeat.

The verdict is identified, ambiguous, signature_only or unknown, and there is no
confidence figure: a candidate's strength says what was checked. Nothing is decoded, no
protected track is opened, and unknown stays unknown. Walks cover ISO media, Matroska,
transport streams, Ogg, FLAC, RIFF, MPEG audio, ADTS, PNG, JPEG, GIF, IVF, YUV4MPEG2,
FLV, AV1 OBU and Annex B streams, and HLS and DASH text; 35 more formats are known by
their signature. Every walk is bounded, reads a file where its lengths lead (a large file
is never held whole), and has been given hostile bytes; it builds for wasm32.

The agent door has it as media_inspect, its results marked untrusted_content: a file's
author chooses the strings in it.

identify-conformance/ holds files written by real muxers and encoders from test patterns
and tones, what an independent media prober reads of them (prober.json), and generated
cases sitting on each rule's edge. identify_verify.py is a second implementation written
from the text; it reads the extension's tables, gives every file its whole result, and
holds ext/ to the prober's reading. tests/identify.rs holds the reference's tables to
the extension's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

int_kernels_vectors: the identify corpus decodes nothing ... 03575317a2

every_decoded_output_states_a_sha256 requires each conformance directory to state its decoded
outputs' SHA-256 or say why it has none. identify-conformance decodes nothing: each vector states
what a file's bytes are read as, and the file's SHA-256 in vectors.json.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai mcp: media_inspect is in the core toolset's list ... 3d42de258e

The core toolset serves media_inspect after envelope_inspect; tests/cli_mcp.rs and the
README's toolset table now name it in that place.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the carousel schedule — when each object passes on a bearer with no return path, and when a receiver holds it ... e2a7bfb87f

On a bearer with no return path a receiver cannot ask for what it missed: it holds an object
only if the passes it heard carried enough encoding symbols. bearer-receipts declares the
transmission and reconciles it afterwards; nothing stated the plan in between.

extensions/carousel.md registers `wai.bearer.carousel` and the jwp-receipts claim class
`schedule` (§2.4.16): a publisher's signed plan, beside the group receipt and the instance's
transmission claim, naming for each object its forward error correction (RaptorQ, RFC 6330 with
its verified erratum 5548; Compact No-Code, RFC 5445) and that code's object transmission
information, the repair symbols each pass adds, whether later passes repeat their ESIs or send
new ones, and the passes' times and count. Step 4 holds each code's parameters to its bounds
and division into source blocks; step 6 holds the entries to the group, its objects' content
hashes and, where the bytes are held, their lengths.

From it a verifier derives, for a loss bound in parts per million, the pass after which a
receiver whose losses stay within it holds each object, for every pattern of loss within the
bound: a RaptorQ block at K' + h distinct symbols, the count RFC 6330 §5.8 states its decoding
failure bound at, and a symbol a repeating pass missed counted as missed again. It checks a
statement that an object or a group is held by an instant, sets aside schedules that contradict
each other or whose instance is undeclared or two-way, and a receiver derives the passes it
must listen to and how long its radio must be on.

The reference is the `carousel` module of wai-rs with the class's body in
`claim_receipt::schedule`; the corpus is live-conformance/carousel/ (33 verdicts, 50 claims, 16
plans) with carousel_verify.py as its second implementation and claim_verify.py's new
`carousel` mode for steps 1-4. tests/carousel.rs checks K' and Partition[] against an
independent RFC 6330 implementation (a dev-dependency) for every block size, and decodes each
RaptorQ object a case holds available from what a receiver within the bound holds. A receipt
index served over HTTP now checks a schedule claim against its group and listed objects.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

carousel §11: the patents disclosed on RaptorQ and Raptor, and no-code as the default ... d93ca3327a

The IETF holds disclosures on RaptorQ (RFC 6330; #2554 current, updating #1958) and on Raptor
(RFC 5053; #1957 current, with #1176 the holder's general licence statement). Both current
ones give two terms by product: a licence within the holder's standard terms for a device that
implements a wireless wide-area standard, and a non-assertion, conditional on full
implementation and suspended against a party that asserts, for one that implements none. No
disclosure relates to RFC 5445 or RFC 5052.

§11 cites them by number and states each licence class as disclosed. Since neither code is
royalty-free for every implementer, no-code is the default a tool that writes schedules uses,
as SPEC §5 keeps a capability that is not royalty-free from being a default. The reference
implementation encodes and decodes no symbol; its RFC 6330 decoder is a development dependency.
Security and privacy considerations move to §12; nothing cites them by number.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: holdings — which objects of which groups a node holds now, for its neighbours ... 36f5074b4c

A receiver that stores what a broadcast carried, a neighbourhood cache or a peer can serve those
objects over a short link; a neighbour that knows who holds what fetches from them before it spends
the backhaul. Nothing said who holds what.

extensions/holdings.md registers `wai.delivery.holdings`: a record (WHD1) of, per group a node holds
objects of, the group's link hash, its leaf count and either `whole` or a bitmap (as a
reconciliation's), in one canonical form, with the instant it states them at and the one it expires
at, signed by the holder under SPEC §7 Signatures. Groups and bitmaps rather than a filter over
hashes: WAI delivers everything under group receipts that name objects by position, a node holds
objects as groups arrive, and a bitmap is exact (no neighbour sent after an object its holder
lacks) at a bit an object, 41 bytes for a group held whole.

A verifier holds a record to its instant and to the group receipts it has (a misstated leaf count
refuses it whole; a group with no receipt held is kept), keeps a holder's latest record and sets
aside two of one instant that differ, answers whether a record holds an object, and derives an
asker's plan: each wanted object's holders in the asker's order, and the objects no neighbour holds,
which come from the origin. An object a neighbour serves is verified under its receipts like any
other; a record is never evidence of bytes. A node advertises only what reached it over a broadcast
bearer, and may sign each record with a fresh key.

The reference is the `holdings` module of wai-rs; the corpus is live-conformance/holdings/ (21
records with a control for each rule, 11 verdicts, 6 queries, a plan over three neighbours), with
holdings_verify.py as its second implementation: 39 cases agree, 9 of 9 mutated copies and 10 of 10
weakened verifiers caught.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-11-04 05:21:47 -05:00

live-contribution: the link record's magic is WLK1, no longer the clip record's WCL1 #604

dcharlot referenced this pull request 2026-11-04 05:22:00 -05:00

wai identify, the carousel schedule and holdings: what a file is, when each object passes on a one-way bearer, and which neighbour holds it #605

dcharlot merged commit 69f88e62f9 into main 2026-11-04 07:18:17 -05:00

dcharlot referenced this pull request from a commit 2026-11-04 07:18:19 -05:00

Merge pull request 'WLK1 for the link record, wai identify, the carousel schedule and holdings (replaces #604 and #605)' (#611) from wai/inspect-carousel-holdings-b into main

dcharlot referenced this pull request 2026-11-04 11:29:13 -05:00

Timing under load: the GPU span test, co-watch re-entry, and the SRT link tests #614

dcharlot referenced this pull request 2026-11-04 12:36:17 -05:00

Wave A: public-name check, WebAssembly determinism in CI, decode-path reporting, and pin by default #616

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!611

No description provided.