wai: parameter sets — where a set is fetched, a successor coded against its predecessor, when a set is worth holding, who may read it, and a set verified piece by piece #638

Merged

dcharlot merged 1 commit from wai/prior-carriage-spec into main 2026-11-05 15:07:24 -05:00

Conversation 0 Commits 1 Files changed 43 +11106 -28

dcharlot commented 2026-11-05 04:25:59 -05:00

Owner

Copy link

What this is

Rules for carrying a parameter set (a prior) to a sink, in http-delivery and prior-carriage:

• where a set is fetched;
• how a successor is coded against its predecessor;
• when a set is worth holding;
• who may read a set, and what a sink discloses of the sets it holds;
• a form of a set that is verified piece by piece from any peers.

Merge order. This branch sits on #632's commit, 946f3beda, which moves changelog entries into wai/changes/ and the conformance workflow's lists into step files. Merge #632 first; until it merges, this pull request's diff also shows #632's change.

What it carries. One commit, 6c8eb07af: the change of wai/prior-carriage-spec (98192fe63..26834bb4c) in #632's layout. Its message lists the source commits. Compared with the source branch:

• its changelog entry is a file of wai/changes/;
• its edits to wai-conformance.yml are a new step file, .github/wai-conformance/verifiers/prior-carriage.sh, at order 335, with the feature prior_carriage;
• its wai/.gitattributes line sits just before the integer-payload block, and its row of the corpus table in wai/live-conformance/README.md sits beside the upscale row, instead of at the end of each.

Every other file is as the source branch has it.

What it defines

Set roots (http-delivery §8.5, §8.7). A set is served at <root><sha256> below a set root, an absolute https URI whose path ends in /. §8.5 named the place <base>/<sha256> and never said what <base> was; the word also collided with §2's term Base. An origin serves its sets below /.well-known/wai-sets/, so a sink that holds an object can always name where to ask.

Location hints. A wai_priors entry's member locations (moq-streaming-format §4.2) lists roots to ask, read in order, at most eight. So does a WAI multivariant playlist's one EXT-X-SESSION-DATA line of DATA-ID science.transaction.wai.sets. An element that holds no root is passed over (set_location_malformed) and never refuses a track. A sink asks, each once:

1. the roots of its own configuration;
2. then the hint's;
3. then the origin's well-known root.

It holds a response's bytes only when their SHA-256 is the digest it asked for, and otherwise asks the next root. The pin stays identity-only: no member of a pin is read as a location. prior-carriage §15 says where a set is fetched and why the pin carries no location.

Sets as compression dictionaries (http-delivery §8.8, prior-carriage §16). RFC 9842's Available-Dictionary names a dictionary by its SHA-256, which is the digest a pin states for a set in a one-file form, so a held set is a dictionary under its own name. A server may offer a set as one, and answer a request that names a held set with the requested set coded against it, dcz or dcb.

A client checks, in order:

• that it asked for the coding (dictionary_unrequested);
• the coding's header (dictionary_coding_malformed);
• the dictionary's SHA-256 in it (dictionary_mismatch);
• the window RFC 9842 has it decode (dictionary_window_too_large);
• the decoding within its bound.

It then holds the decoded bytes only by the pin's digest. A refused coded response sends it to the same root again, without a dictionary. The coded bytes are a transfer and never a form: no pin names them. Caches store a coded response under its Vary and never code a set themselves. The reference decodes dcz.

When a set is worth holding (prior-carriage §17). The break-even rule that exact-upscale §5 stated for per-title priors is now prior-carriage's, for every set, and exact-upscale §5 points to it. A generation's saving to a sink, over the titles it is expected to fetch within a stated span, must exceed its cost to that sink. That cost is nothing when the sink holds the set, the coded bytes of §16 when it holds the predecessor, and the set's bytes otherwise. The cost is charged per sink, never divided among the sinks a publisher serves. A sink holds at most a stated number of generations, the largest net first and, at an equal net, the later. Every other title takes its classical rendition. A re-tuned set begins the next catalog epoch as a new generation whose pin names its predecessor in supersedes. A plan that does not read is refused (plan_malformed).

Usage, access and leakage (prior-carriage §18).

• A set's usage statement (wai-set-usage) gives an SPDX licence expression, a terms URI, whether the set was fitted to content, whether a holder may serve it to others, and an ai-usage statement. It is bound to a set only inside the bytes its pin covers.
• Sealed sets, wai.prior.sealed (WSK1), are registered: a one-file set under AES-128 in counter mode, its key named by a 16-byte identifier and released by its origin. The pin covers the sealed bytes, so any server may serve them. A sink without the key does not hold the set. Opened, the plaintext is checked against the SHA-256 the header states.
• Leakage. A content-derived set can carry features of the frames it was fitted to, and should be sealed when that content is under access control.

What a sink discloses of the sets it holds (http-delivery §12.5). Sets are kept per top-level site. held_sets is off unless it is turned on for a site, and then lists only the held sets that site's content pins, never a revoked one. Nothing is sent or named across sites, Available-Dictionary included. Clearing a site's data clears its sets. SECURITY.md no longer says these rules are not written.

The chunk-tree form, wai.prior.tree (WTR1, prior-carriage §19), registered. A head names a one-file set by its length, SHA-256 and BLAKE3 hash. It lists the BLAKE3 chaining value of each piece of 1024 · 2^g bytes, each a subtree of BLAKE3's tree over the set. The pin covers the head. The set is served under its own SHA-256 and fetched by range from any peers. Each piece is checked as it arrives; a changed piece is refused at that piece (tree_piece_mismatch) and fetched elsewhere. The whole set is then checked against its SHA-256 (tree_digest_mismatch). Its inner form may be a sealed set.

The registry lists the new wire identifiers of http-delivery §8.7 and prior-carriage §18 and §19, and wai.prior.sealed and wai.prior.tree as established exact.

The reference and the independent verifier

The reference. The prior_carriage module of wai-rs. Its unit tests cover:

• set roots, and the hints of a catalog entry and of a playlist;
• a fetch past a tampered mirror;
• the RFC 9842 field values, a set coded against another, and each check of a coded response;
• a plan's break-even, its successor and its refusals;
• SPDX licence expressions and usage statements;
• counter mode against NIST SP 800-38A F.5.1, and a set sealed and opened;
• held sets per site;
• a chunk tree's root equal to the BLAKE3 hash for every piece size and length tried, and a flipped piece refused at that piece.

tests/prior_carriage.rs checks that:

• the committed corpus regenerates byte for byte from the sets other corpora commit;
• every case gets its stated outcome;
• http-delivery §13 lists exactly the reasons the reference reports for §8.7 and §8.8;
• prior-carriage §20 lists its own.

The corpus. live-conformance/prior-carriage/ reads the priors of the prior-forms corpus and a listing of the pin corpus. It holds:

• set roots, hints from catalog entries and playlists, and fetches over stated responses;
• the upscale corpus's per-title prior coded against generic, fetches over it, field values, and the check a stock client makes;
• worked and refused plans, among them a net of exactly 0, and sinks that together would save more than a set's size while each saves less;
• usage statements, title and generic sealed under a published test key with openings of each kind, and held sets per site;
• chunk-tree heads, and fetches from two peers, among them a changed piece at one peer.

The independent verifier. live-conformance/prior_carriage_verify.py is written from the texts and shares no code with wai-rs. It decodes Zstandard with its own decoder written from RFC 8878 (_rfc8878_zstd.py), and opens sealed sets with its own AES-128 written from FIPS 197 (_fips197_aes.py). It reads ai-usage statements with ai_usage_verify.py, and hashes with the corpora's shared BLAKE3. It runs the stock-tool check where the Zstandard tool is on the PATH. It requires every mutated copy of the corpus to fail, and drops each of its own checks in turn. It uses the standard library only and runs isolated (-I).

How it was verified

This commit, 6c8eb07af.

• Locally (git and Python checks only): scripts/wai-conformance-lists.py selftest and check, scripts/wai-conformance-lists-selftest.sh, scripts/check-changelog-fragment.py against main, scripts/changelog-assemble.py --check and scripts/changelog-fragments-selftest.sh all pass.
• The layout changes only the layout. A comparison of trees shows that the change is the source branch's change except for the differences listed above. The entry's text is identical to what the source branch added to wai/CHANGELOG.md. By #632's dry-run, the step file gives the workflow what the source branch's workflow edits gave it. The .gitattributes and README changes are the same lines, moved. That comparison was made on the conversion's first base, #632's earlier tip 14f1abb7c; rebasing onto 946f3beda left every file's change the same.
• On a Linux x86_64 build host: in wai/wai-rs, cargo check --locked --no-default-features --tests with each of prior_carriage, prior_wire, prior_wire,neural_int and prior_carriage,prior_wire. All pass.
• With the other converted branches. This branch merges without conflict with each of the other branches converted to #632's layout (by git merge-tree). Merging all of them in forward, reverse and a mixed order gives one tree. On the build host, that tree passes these runs: cargo check --tests under the derived WAI_FEATURES and under WAI_PQ_FEATURES; cargo check --tests of quickstart,app; the catalogue's library tests; and the wai binary's unit tests.

The module's tests, the corpus's regeneration and the Python verifier did not run on this commit here; their results come from this pull request's CI, in the new step.

On the CI hosts. Results on the CI runners come from this pull request's own CI run. None of the runs above is a CI result.

🤖 Generated with Claude Code

## What this is Rules for carrying a parameter set (a prior) to a sink, in http-delivery and prior-carriage: - where a set is fetched; - how a successor is coded against its predecessor; - when a set is worth holding; - who may read a set, and what a sink discloses of the sets it holds; - a form of a set that is verified piece by piece from any peers. **Merge order.** This branch sits on #632's commit, 946f3beda, which moves changelog entries into `wai/changes/` and the conformance workflow's lists into step files. Merge #632 first; until it merges, this pull request's diff also shows #632's change. **What it carries.** One commit, 6c8eb07af: the change of `wai/prior-carriage-spec` (98192fe63..26834bb4c) in #632's layout. Its message lists the source commits. Compared with the source branch: - its changelog entry is a file of `wai/changes/`; - its edits to `wai-conformance.yml` are a new step file, `.github/wai-conformance/verifiers/prior-carriage.sh`, at order 335, with the feature `prior_carriage`; - its `wai/.gitattributes` line sits just before the integer-payload block, and its row of the corpus table in `wai/live-conformance/README.md` sits beside the upscale row, instead of at the end of each. Every other file is as the source branch has it. ## What it defines **Set roots** (http-delivery §8.5, §8.7). A set is served at `<root><sha256>` below a set root, an absolute `https` URI whose path ends in `/`. §8.5 named the place `<base>/<sha256>` and never said what `<base>` was; the word also collided with §2's term **Base**. An origin serves its sets below `/.well-known/wai-sets/`, so a sink that holds an object can always name where to ask. **Location hints.** A `wai_priors` entry's member `locations` (moq-streaming-format §4.2) lists roots to ask, read in order, at most eight. So does a WAI multivariant playlist's one `EXT-X-SESSION-DATA` line of `DATA-ID` `science.transaction.wai.sets`. An element that holds no root is passed over (`set_location_malformed`) and never refuses a track. A sink asks, each once: 1. the roots of its own configuration; 2. then the hint's; 3. then the origin's well-known root. It holds a response's bytes only when their SHA-256 is the digest it asked for, and otherwise asks the next root. The pin stays identity-only: no member of a pin is read as a location. prior-carriage §15 says where a set is fetched and why the pin carries no location. **Sets as compression dictionaries** (http-delivery §8.8, prior-carriage §16). RFC 9842's `Available-Dictionary` names a dictionary by its SHA-256, which is the digest a pin states for a set in a one-file form, so a held set is a dictionary under its own name. A server may offer a set as one, and answer a request that names a held set with the requested set coded against it, `dcz` or `dcb`. A client checks, in order: - that it asked for the coding (`dictionary_unrequested`); - the coding's header (`dictionary_coding_malformed`); - the dictionary's SHA-256 in it (`dictionary_mismatch`); - the window RFC 9842 has it decode (`dictionary_window_too_large`); - the decoding within its bound. It then holds the decoded bytes only by the pin's digest. A refused coded response sends it to the same root again, without a dictionary. The coded bytes are a transfer and never a form: no pin names them. Caches store a coded response under its `Vary` and never code a set themselves. The reference decodes `dcz`. **When a set is worth holding** (prior-carriage §17). The break-even rule that exact-upscale §5 stated for per-title priors is now prior-carriage's, for every set, and exact-upscale §5 points to it. A generation's saving to a sink, over the titles it is expected to fetch within a stated span, must exceed its cost to that sink. That cost is nothing when the sink holds the set, the coded bytes of §16 when it holds the predecessor, and the set's bytes otherwise. The cost is charged per sink, never divided among the sinks a publisher serves. A sink holds at most a stated number of generations, the largest net first and, at an equal net, the later. Every other title takes its classical rendition. A re-tuned set begins the next catalog epoch as a new generation whose pin names its predecessor in `supersedes`. A plan that does not read is refused (`plan_malformed`). **Usage, access and leakage** (prior-carriage §18). - A set's **usage statement** (`wai-set-usage`) gives an SPDX licence expression, a terms URI, whether the set was fitted to content, whether a holder may serve it to others, and an ai-usage statement. It is bound to a set only inside the bytes its pin covers. - **Sealed sets**, `wai.prior.sealed` (`WSK1`), are registered: a one-file set under AES-128 in counter mode, its key named by a 16-byte identifier and released by its origin. The pin covers the sealed bytes, so any server may serve them. A sink without the key does not hold the set. Opened, the plaintext is checked against the SHA-256 the header states. - **Leakage.** A content-derived set can carry features of the frames it was fitted to, and should be sealed when that content is under access control. **What a sink discloses of the sets it holds** (http-delivery §12.5). Sets are kept per top-level site. `held_sets` is off unless it is turned on for a site, and then lists only the held sets that site's content pins, never a revoked one. Nothing is sent or named across sites, `Available-Dictionary` included. Clearing a site's data clears its sets. `SECURITY.md` no longer says these rules are not written. **The chunk-tree form**, `wai.prior.tree` (`WTR1`, prior-carriage §19), registered. A head names a one-file set by its length, SHA-256 and BLAKE3 hash. It lists the BLAKE3 chaining value of each piece of 1024 · 2^g bytes, each a subtree of BLAKE3's tree over the set. The pin covers the head. The set is served under its own SHA-256 and fetched by range from any peers. Each piece is checked as it arrives; a changed piece is refused at that piece (`tree_piece_mismatch`) and fetched elsewhere. The whole set is then checked against its SHA-256 (`tree_digest_mismatch`). Its inner form may be a sealed set. **The registry** lists the new wire identifiers of http-delivery §8.7 and prior-carriage §18 and §19, and `wai.prior.sealed` and `wai.prior.tree` as established exact. ## The reference and the independent verifier **The reference.** The `prior_carriage` module of `wai-rs`. Its unit tests cover: - set roots, and the hints of a catalog entry and of a playlist; - a fetch past a tampered mirror; - the RFC 9842 field values, a set coded against another, and each check of a coded response; - a plan's break-even, its successor and its refusals; - SPDX licence expressions and usage statements; - counter mode against NIST SP 800-38A F.5.1, and a set sealed and opened; - held sets per site; - a chunk tree's root equal to the BLAKE3 hash for every piece size and length tried, and a flipped piece refused at that piece. `tests/prior_carriage.rs` checks that: - the committed corpus regenerates byte for byte from the sets other corpora commit; - every case gets its stated outcome; - http-delivery §13 lists exactly the reasons the reference reports for §8.7 and §8.8; - prior-carriage §20 lists its own. **The corpus.** `live-conformance/prior-carriage/` reads the priors of the prior-forms corpus and a listing of the pin corpus. It holds: - set roots, hints from catalog entries and playlists, and fetches over stated responses; - the upscale corpus's per-title prior coded against `generic`, fetches over it, field values, and the check a stock client makes; - worked and refused plans, among them a net of exactly 0, and sinks that together would save more than a set's size while each saves less; - usage statements, `title` and `generic` sealed under a published test key with openings of each kind, and held sets per site; - chunk-tree heads, and fetches from two peers, among them a changed piece at one peer. **The independent verifier.** `live-conformance/prior_carriage_verify.py` is written from the texts and shares no code with `wai-rs`. It decodes Zstandard with its own decoder written from RFC 8878 (`_rfc8878_zstd.py`), and opens sealed sets with its own AES-128 written from FIPS 197 (`_fips197_aes.py`). It reads ai-usage statements with `ai_usage_verify.py`, and hashes with the corpora's shared BLAKE3. It runs the stock-tool check where the Zstandard tool is on the PATH. It requires every mutated copy of the corpus to fail, and drops each of its own checks in turn. It uses the standard library only and runs isolated (`-I`). ## How it was verified **This commit, 6c8eb07af.** - **Locally** (git and Python checks only): `scripts/wai-conformance-lists.py selftest` and `check`, `scripts/wai-conformance-lists-selftest.sh`, `scripts/check-changelog-fragment.py` against main, `scripts/changelog-assemble.py --check` and `scripts/changelog-fragments-selftest.sh` all pass. - **The layout changes only the layout.** A comparison of trees shows that the change is the source branch's change except for the differences listed above. The entry's text is identical to what the source branch added to `wai/CHANGELOG.md`. By #632's `dry-run`, the step file gives the workflow what the source branch's workflow edits gave it. The `.gitattributes` and README changes are the same lines, moved. That comparison was made on the conversion's first base, #632's earlier tip 14f1abb7c; rebasing onto 946f3beda left every file's change the same. - **On a Linux x86_64 build host:** in `wai/wai-rs`, `cargo check --locked --no-default-features --tests` with each of `prior_carriage`, `prior_wire`, `prior_wire,neural_int` and `prior_carriage,prior_wire`. All pass. - **With the other converted branches.** This branch merges without conflict with each of the other branches converted to #632's layout (by `git merge-tree`). Merging all of them in forward, reverse and a mixed order gives one tree. On the build host, that tree passes these runs: `cargo check --tests` under the derived `WAI_FEATURES` and under `WAI_PQ_FEATURES`; `cargo check --tests` of `quickstart,app`; the catalogue's library tests; and the `wai` binary's unit tests. The module's tests, the corpus's regeneration and the Python verifier did not run on this commit here; their results come from this pull request's CI, in the new step. **On the CI hosts.** Results on the CI runners come from this pull request's own CI run. None of the runs above is a CI result. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-11-05 04:25:59 -05:00

wai/prior-carriage-spec, converted to the layout of the fix: no shared append point ... 6c8eb07afe

The change of wai/prior-carriage-spec (98192fe63..26834bb4c, 6 commits), on the fix branch: its
changelog entries as files of wai/changes/, its wai-conformance.yml edits as step files,
corpus files and test tags, and every other file as the branch has it.

- e5db9b313 wai: where a parameter set is fetched — set roots, location hints, and <base> defined
- 29fe43ea2 wai: a parameter set as an RFC 9842 compression dictionary, with a vector a stock client checks
- faaeb6c73 wai: when a parameter set is worth holding — amortisation charged per sink, bootstrap and generations
- 9a820db4a wai: a parameter set's usage statement, sealed sets released by key, leakage, and the privacy of held sets
- 89a9f9924 wai: the chunk-tree prior form, wai.prior.tree — a set verified piece by piece from any peers
- 26834bb4c wai: one playlist line of set roots, inner forms as the layouts list them, and a listing's usage statement read or the set refused

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit a9ae256857 into main 2026-11-05 15:07:24 -05:00

dcharlot referenced this pull request from a commit 2026-11-05 15:07:26 -05:00

Merge pull request 'wai: parameter sets — where a set is fetched, a successor coded against its predecessor, when a set is worth holding, who may read it, and a set verified piece by piece' (#638) from wai/prior-carriage-spec into main

dcharlot referenced this pull request from a commit 2026-11-05 16:15:40 -05:00

Merge origin/main (#638, #652: parameter sets, and the relay loop test's ports) into wai/quantum-cloud-tasks

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!638

No description provided.