Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions 7

wai: parameter sets — where a set is fetched, a successor coded against its predecessor, when a set is worth holding, who may read it, and a set verified piece by piece #638

Merged
dcharlot merged 1 commit from wai/prior-carriage-spec into main 2026-11-05 15:07:24 -05:00
Conversation 0 Commits 1 Files changed 43 +11106 -28
43 changed files with 11106 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

24
.github/wai-conformance/verifiers/prior-carriage.sh vendored Normal file
View file

@ -0,0 +1,24 @@
# step: prior carriage · wai-rs and prior_carriage_verify.py
# order: 335
# features: prior_carriage
#
# Prior carriage (extensions/prior-carriage.md §15 to §19; extensions/http-delivery.md §8.7,
# §8.8, §12.5): the module's unit tests (set roots, the hints of a catalog entry and of a
# playlist, a fetch past a tampered mirror, the RFC 9842 field values, a set coded against
# another and each check of a coded response, a plan's break-even, successor and refusals,
# SPDX licence expressions and usage statements, counter mode against NIST SP 800-38A F.5.1,
# a set sealed and opened, held sets per site, a chunk tree's root equal to the BLAKE3 hash
# for every piece size and length tried, a flipped piece refused at that piece) and
# tests/prior_carriage.rs (the committed corpus regenerates byte for byte from the sets other
# corpora commit, every case gets its stated outcome, http-delivery §13 lists exactly the
# reasons the reference reports for §8.7 and §8.8, and prior-carriage §20 lists its own);
# then the corpus's second implementation, prior_carriage_verify.py, written from
# the texts and sharing no code with wai-rs, which decodes Zstandard with its own RFC 8878
# decoder (_rfc8878_zstd.py), opens sealed sets with its own AES-128 (_fips197_aes.py),
# reads ai-usage statements with ai_usage_verify.py, hashes with the shared _blake3.py, runs the stock-tool check where the Zstandard tool is on the
# PATH, requires every mutated copy of the corpus to fail and drops each of its own checks
# in turn; standard library only, run isolated (-I).
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --lib -- prior_carriage::)
(cd wai-rs && cargo test --no-default-features --features "$WAI_FEATURES" --test prior_carriage)
python3 -I live-conformance/prior_carriage_verify.py live-conformance/prior-carriage

3
wai/.gitattributes vendored
View file

@ -1,6 +1,9 @@
# REGISTRY.md carries a table `wai registry export` generates, which a library test
# compares byte for byte (as registry/ and vocab/v2.* are): no line-ending conversion.
REGISTRY.md -text
# The prior-carriage corpus (extensions/prior-carriage.md §15 on): coded bodies and set
# files pinned by SHA-256 and regenerated byte for byte, so never converted.
live-conformance/prior-carriage/** -text
# The binary payloads of the integer codecs (extensions/integer-payloads.md) beside
# their JSON sources: pinned by SHA-256, so never converted.
*.wiv1 -text

8
wai/REGISTRY.md
View file

@ -278,7 +278,9 @@ and a role are fixed once registered.
| `wai.prior.gguf` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Prior artifacts, general tensors and derivation operations |
| `wai.prior.nnc` | payload | registered | binary | standalone | patent-pool | standard-unclassified | — | §5 Prior artifacts, general tensors and derivation operations |
| `wai.prior.safetensors` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Prior artifacts, general tensors and derivation operations |
| `wai.prior.sealed` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Prior artifacts, general tensors and derivation operations |
| `wai.prior.set` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Prior artifacts, general tensors and derivation operations |
| `wai.prior.tree` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Prior artifacts, general tensors and derivation operations |
| `wai.quantum.calibration` | record | registered | — | — | — | — | — | §5 Interactive, temporal & computed content |
| `wai.quantum.circuit` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Interactive, temporal & computed content |
| `wai.quantum.circuit2` | payload | registered | binary | standalone | royalty-free | standard-defined | — | §5 Interactive, temporal & computed content |
@ -357,6 +359,7 @@ and a role are fixed once registered.
| `/.well-known/wai-key-status` | well-known | defined | http-delivery §8.4 |
| `/.well-known/wai-keys` | well-known | defined | jwp-receipts §7.1 |
| `/.well-known/wai-mark/` | well-known | proposed | synthesis §7.7 |
| `/.well-known/wai-sets/` | well-known | defined | http-delivery §8.7 |
| `EXT_gaussian_splatting_compression_wai` | gltf-extension | proposed | splat-exact §3.2 |
| `GRANT_CHALLENGE` | setup-option | proposed | live-rooms §10.5.2 |
| `GRANT_PROOF` | setup-option | proposed | live-rooms §10.5.2 |
@ -447,11 +450,13 @@ and a role are fixed once registered.
| `WSD1` | magic | defined | energy-steering §7.1 |
| `WSI1` | magic | defined | stitched-presentation §4.1 |
| `WSIX` | magic | defined | packaging §6 |
| `WSK1` | magic | defined | prior-carriage §18.2 |
| `WSL1` | magic | defined | stream-termination §3.1 |
| `WSP1` | magic | defined | energy-steering §3.1 |
| `WSR1` | magic | defined | exact-upscale §3.1 |
| `WSS1` | magic | defined | settlement §7.1 |
| `WTM1` | magic | defined | capture-binding §7.1 |
| `WTR1` | magic | defined | prior-carriage §19 |
| `WTX1` | magic | defined | timed-text §4.2 |
| `WUR1` | magic | defined | capture-binding §6.2 |
| `WWD1` | magic | defined | forensic-watermark §8.1 |
@ -495,6 +500,7 @@ and a role are fixed once registered.
| `science.transaction.wai.class` | data-id | defined | packaging §8 |
| `science.transaction.wai.event` | playlist-attribute | defined | live-events §8.3 |
| `science.transaction.wai.multivariant` | data-id | defined | http-delivery §11.1 |
| `science.transaction.wai.sets` | data-id | defined | http-delivery §8.7 |
| `science.transaction.wai.stages` | data-id | defined | packaging §8 |
| `science.transaction.wai.timed-text` | data-id | defined | timed-text §10.5 |
| `tag:transaction.science,2026:wai/access` | scheme | defined | access-services §5.5 |
@ -566,6 +572,7 @@ and a role are fixed once registered.
| `wai-scene/1` | scheme | defined | scene-carriage §2.3 |
| `wai-sequence` | document-kind | defined | direction §6 |
| `wai-session` | document-kind | defined | contribution §3.1 |
| `wai-set-usage` | document-kind | defined | prior-carriage §18.1 |
| `wai-signing` | document-kind | defined | access-services §6.1 |
| `wai-signing-review` | document-kind | defined | access-services §6.4 |
| `wai-splat-commitment` | document-kind | defined | splat-exact §6.1 |
@ -584,6 +591,7 @@ and a role are fixed once registered.
| `wai-terms` | document-kind | defined | influence §3.2 |
| `wai-tonemap` | document-kind | defined | exact-tonemap §7.1 |
| `wai-upscale` | document-kind | defined | exact-upscale §4.2 |
| `wai-usage.json` | path | defined | prior-carriage §18.1 |
| `wai1` | codecs-prefix | defined | http-delivery §4.1 |
| `wai1` | fourcc | provisional | http-delivery Appendix B |
| `wai2` | codecs-prefix | defined | http-delivery §4.1 |

19
wai/SECURITY.md
View file

@ -250,12 +250,19 @@ sink holds. Which sets a sink holds depends on what it has fetched before, so th
list can identify a sink across the sites it is sent to, much as a cookie does. A
shared compression dictionary named by its hash raises the same concern (RFC 9842).
Privacy rules for the descriptor and for parameter-set fetches are not written yet.
They are to be added to http-delivery: the list kept per top-level site, clearable by
the user, disclosed only as far as a choice of rendition needs it, never sent across
sites, and off by default. Until they land, send `held_sets` only to the origin that
needs it to choose a rendition, and never across sites. No code in this tree builds
or sends the descriptor.
The privacy rules for the descriptor and for parameter-set fetches are
[http-delivery](extensions/http-delivery.md) §12.5: the sets a sink holds are kept per
top-level site, `held_sets` is off unless turned on for a site, it lists only the held
sets the content being chosen among pins, nothing is sent or named across sites,
clearing a site's data clears its sets, and a report of which sets sinks hold is an
aggregate, never one sink's list. `prior_carriage::privacy` builds the member under
them (feature `prior_carriage`); no code in this tree sends the descriptor.
A parameter set fitted to a catalog's frames can carry features of those frames, and its
digest names the title it was fitted to. [prior-carriage](extensions/prior-carriage.md)
§18 states the usage statement a set carries inside the bytes its pin covers, the sealed
form `wai.prior.sealed`, whose set no sink reads until its origin releases the key, and
what a set can disclose.
## Protected content

2
wai/SPEC.md
View file

@ -876,6 +876,8 @@ reconstruction, with the acquisition class and coverage C2PA 2.4's
| `wai.prior.set` | the listing of a parameter set of one or more files: each file's SHA-256 and path (§3.1) | a registered wire form a pin MAY declare (`prior.form`), and the form in which a set of several files is pinned: the pin covers the listing, and each line covers one file. Reading a set through its listing compares digests and hands the decoder each file's bytes unchanged, so the registry lists it as established exact, and a `replicate` MAY rest on a pin in it. WAI-native, royalty-free; as a payload its output is its own bytes (`StandardDefined`) · [prior-carriage](extensions/prior-carriage.md) |
| `wai.prior.safetensors` | a parameter set as one safetensors file: an 8-byte header length, a JSON header naming each tensor's type, shape and bytes, then the tensors' bytes | a registered wire form a pin MAY declare (`prior.form`); its digest covers the whole file, which a reader requires canonical, so one set of tensors has one digest. Reading it hands the decoder each tensor's bytes as the file holds them, so the registry lists it as established exact. A prior in the integer-quantised profile `wai-int/1` loads, bit for bit, into the native artifact an integer kernel reads. Royalty-free (the layout is published with its reference implementation under Apache-2.0); as a payload `StandardDefined` · [prior-carriage](extensions/prior-carriage.md) |
| `wai.prior.gguf` | a parameter set as one GGUF version 3 file: key-value metadata, tensor descriptions, aligned tensor data | as `wai.prior.safetensors`: a registered wire form, its digest over the whole canonical file, established exact, and in the profile `wai-int/1` a prior that loads bit for bit into the integer kernels. Royalty-free (published under MIT); as a payload `StandardDefined` · [prior-carriage](extensions/prior-carriage.md) |
| `wai.prior.sealed` | a one-file set encrypted under a key its origin releases (`WSK1`): the inner form, the key's identifier, the counter block, the plaintext's SHA-256, the set's usage statement, then the set under AES-128 in counter mode | a registered wire form a pin MAY declare (`prior.form`); its digest covers the sealed bytes, so any server may serve them and a sink checks them before it holds the key. A sink without the key does not hold the set (`prior_not_held`, `set_key_absent`); opened, the plaintext is checked against the SHA-256 the header states, and is a set in an admitted one-file form, so the registry lists it as established exact. WAI-native, royalty-free; as a payload `StandardDefined` · [prior-carriage](extensions/prior-carriage.md) §18.2 |
| `wai.prior.tree` | the head of a chunk tree over a one-file set (`WTR1`): the inner form, the set's length, SHA-256 and BLAKE3 hash, and the BLAKE3 chaining value of each piece of 1024 · 2^g bytes | a registered wire form a pin MAY declare (`prior.form`); its digest covers the head, so a sink fetches the set by range from any peers and checks each piece as it arrives, refusing a changed piece at that piece, then checks the whole set against the SHA-256 the head states. It hands over the set it names, so the registry lists it as established exact. WAI-native, royalty-free; as a payload `StandardDefined` · [prior-carriage](extensions/prior-carriage.md) §19 |
| `wai.tensor.nnc` | general tensorial data (split-inference feature maps, 3D/4D scene parameters) coded with NNC | **candidate**: general-tensor coding is in an unpublished amendment, so the payload format is not fixed and MUST NOT be emitted until pinned (*Candidate capabilities*, below) · [prior-carriage](extensions/prior-carriage.md) |
The ten derivation operations of ISO/IEC 23001-16:2021 (derived visual tracks)

108
wai/changes/2026-11-04-parameter-sets-where-set-fetched-successor.md Normal file
View file

@ -0,0 +1,108 @@
### Parameter sets: where a set is fetched, a successor coded against its predecessor, when a set is worth holding, who may read it, and a set verified piece by piece
- **Set roots** ([http-delivery](extensions/http-delivery.md) §8.5, §8.7): a set is served at
`<root><sha256>` below a set root, an absolute `https` URI whose path ends in `/`. §8.5 named the
place `<base>/<sha256>` and never said what `<base>` was; the word also collided with §2's term
**Base**. An origin serves its sets below `/.well-known/wai-sets/`, so a sink that holds an
object can always name where to ask.
- **Location hints**: a `wai_priors` entry's member `locations`
([moq-streaming-format](extensions/moq-streaming-format.md) §4.2) and a WAI multivariant
playlist's one `EXT-X-SESSION-DATA` line of `DATA-ID` `science.transaction.wai.sets`, its roots
separated by spaces (RFC 8216 §4.3.4.4 admits one such line with no `LANGUAGE`, and a reader
ignores any later one), list roots to ask, read in order, at most eight; an element that holds no
root is passed over (`set_location_malformed`) and never refuses a track. A sink asks the roots of
its own configuration, then the hint's, then the origin's well-known root, each once; it holds a
response's bytes only when their SHA-256 is the digest it asked for, and otherwise asks the next
root (`set_location_unavailable`, `content_coding_present`, `set_too_large`,
`set_location_mismatch`). A listing's files are asked first where the listing was found. The pin
stays identity-only: no member of a pin is read as a location.
- **prior-carriage §15** says where a set is fetched and why the pin carries no location.
- **Sets as compression dictionaries** ([http-delivery](extensions/http-delivery.md) §8.8,
[prior-carriage](extensions/prior-carriage.md) §16): RFC 9842's `Available-Dictionary` names a
dictionary by its SHA-256, which is the digest a pin states for a set in a one-file form, so a
held set is a dictionary under its own name. A server MAY offer a set as one
(`Use-As-Dictionary: match="<root path>*"`) and answer a request that names a held set with the
requested set coded against it, `dcz` or `dcb`, with `Vary: accept-encoding,
available-dictionary`. A client checks, in order, that it asked for the coding
(`dictionary_unrequested`), the coding's header (`dictionary_coding_malformed`), the dictionary's
SHA-256 in it (`dictionary_mismatch`), the window RFC 9842 has it decode
(`dictionary_window_too_large`) and the decoding within its bound, then holds the decoded bytes
only by the pin's digest; a refused coded response sends it to the same root again without a
dictionary. A client that writes the field itself names the set the pin `supersedes`. The coded
bytes are a transfer and never a form: no pin names them. Caches store a coded response under
its `Vary` and never code a set themselves; a range of a set is never coded. The reference
decodes `dcz` only, through the Zstandard binding the neural decoders use.
- **When a set is worth holding** ([prior-carriage](extensions/prior-carriage.md) §17): the
break-even rule that exact-upscale §5 stated for per-title priors is now prior-carriage's, for
every set, and exact-upscale §5 points to it. A generation's saving to a sink, over the titles it
is expected to fetch within a stated span, must exceed its cost to that sink: nothing when the
sink holds the set, the coded bytes of §16 when it holds the predecessor, the set's bytes
otherwise. The cost is charged per sink, never divided among the sinks a publisher serves. A
sink holds at most a stated number of generations, the largest net first and, at an equal net,
the later; every other title takes its classical rendition. A sink that has disclosed no held set
counts as one that holds none. An origin freezes a set for a catalog epoch, and a re-tuned set
begins the next as a new generation whose pin names its predecessor in `supersedes`; older
titles keep their pins and decode where their generation is kept, or are re-encoded. A plan that
does not read is refused (`plan_malformed`, prior-carriage §20).
- **Usage, access and leakage** ([prior-carriage](extensions/prior-carriage.md) §18): a set's
**usage statement** (`wai-set-usage`: an SPDX licence expression, a terms URI, whether the set
was fitted to content, whether a holder may serve it to others, and an ai-usage statement for
uses of its bytes besides decoding) is bound to a set only inside the bytes its pin covers, as
the top-level path `wai-usage.json` of a listing or in a sealed set's header; a bound statement
that does not read leaves the set not held (`set_usage_malformed`); a holder of a set that may
not be redistributed serves it to no one but as a cache for its origin. **Sealed sets**,
`wai.prior.sealed` (`WSK1`), registered: a one-file set under AES-128 in counter mode, its key
named by a 16-byte identifier as a content key is and released by its origin; the pin covers
the sealed bytes, so any server may serve them, and a sink without the key does not hold the set
(`prior_not_held`, `set_key_absent`); opened, the plaintext is checked against the SHA-256 the
header states (`set_key_mismatch`), and both digests are the set's identities for revocation.
The registry lists it as established exact. **Leakage**: a content-derived set can carry
features of the frames it was fitted to, and SHOULD be sealed when that content is under access
control; a per-title prior's digest names its title.
- **What a sink discloses of the sets it holds** ([http-delivery](extensions/http-delivery.md)
§12.5): sets are kept per top-level site; `held_sets` is off unless turned on for a site and
then lists only the held sets that site's content pins, a revoked set never; nothing is sent or
named across sites, `Available-Dictionary` included; clearing a site's data clears its sets; a
report of held sets is an aggregate. SECURITY.md no longer says these rules are not written.
- **The chunk-tree form**, `wai.prior.tree` (`WTR1`, [prior-carriage](extensions/prior-carriage.md)
§19), registered: a head names a one-file set by its length, SHA-256 and BLAKE3 hash and lists
the BLAKE3 chaining value of each piece of 1024 · 2^g bytes, each a subtree of BLAKE3's tree over
the set, whose values merge to the set's BLAKE3 hash. The pin covers the head; the set is served
under its own SHA-256 and fetched by range from any peers, each piece checked as it arrives and a
changed piece refused at that piece (`tree_piece_mismatch`) and fetched elsewhere; the whole set
is then checked against its SHA-256 (`tree_digest_mismatch`). Its inner form may be a sealed set.
The registry lists it as established exact.
- **The registry** lists `/.well-known/wai-sets/` and `science.transaction.wai.sets` as wire
identifiers of http-delivery §8.7, `wai.prior.sealed` and `wai.prior.tree`, and `WSK1`,
`wai-set-usage`, `wai-usage.json` and `WTR1` of prior-carriage §18 and §19.
- **The corpus** [`live-conformance/prior-carriage/`](live-conformance/prior-carriage/) reads the
priors of the prior-forms corpus and a listing of the pin corpus. For §8.7: 19 strings tested as
set roots, 14 hints from catalog entries and playlists (among them playlists with a second line of
the hint's `DATA-ID` and with one that carries `LANGUAGE`, each ignored), and 14 fetches over
stated responses, among them a set held from the second root of its hint after the first served
other bytes, a listing and its files held from two roots, a listed file no root holds
(`part_mismatch`), a pin whose own member names a location, which no fetch asks, and listings with
a file named `wai-usage.json`: at the top-level path, held with its terms; there and not reading
(`set_usage_malformed`); and below the top, which binds nothing. For §8.8: `title`, the upscale
corpus's per-title prior, fine-tuned from `generic`, coded against `generic`
(`files/title-from-generic.dcz`), 9 fetches over it (the successor held through its predecessor,
each check of a coded response refusing, a decoding that yields another set, one past the bound),
9 field values, and the check a stock client makes, which the Zstandard tool passes: the body
after its 40-byte header, decoded against `generic`, has `title`'s SHA-256. For §17: 11 worked
plans, among them a net of exactly 0 (not worth holding), a successor costing its coded bytes
where its predecessor is held, a sink holding two generations when a third is worth fetching (with
a cap of two it evicts the one of smallest net; with three it keeps all), and three sinks that
together would save more than the set's size and each save less, none of which fetches it; and 10
plans refused. For §18 and §12.5: 16 usage statements, `title` and `generic` sealed under a
published test key and 13 openings (with the key, without it, under another key, a changed
ciphertext byte, and headers broken one way each, among them inner forms of a listing, a sealed
set and a tree), and 9 cases of held sets per site. For §19: 13 heads (of `generic` in pieces of
one and two chunks, of the sealed `title`, and broken one way each, among them inner forms of a
listing, a tree and a string no form registers) and 9 fetches from two peers (a byte changed in
piece 3 at one peer, refused there and fetched from the other; a peer that does not answer; both
peers wrong at one piece, which leaves the set not held; a content coding; a peer whose bytes end
early; the sealed `title`; a head that misstates its set's digest). `prior_carriage_verify.py` is
a second implementation, with a Zstandard decoder of its own written from RFC 8878
(`_rfc8878_zstd.py`) and an AES-128 written from FIPS 197 (`_fips197_aes.py`), and the BLAKE3 the
corpora's Python verifiers share; it catches 29 of 29 mutated copies of the corpus and 44 of 44
weakened copies of itself.

10
wai/extensions/exact-upscale.md
View file

@ -232,10 +232,12 @@ their digest, are what is reproducible. The reference tool, `tools/wai_int_sr_tr
in floating point from fixed seeds, and its training is not claimed to regenerate the same bytes
on another machine.
An origin **SHOULD** send a per-title prior only where the bytes it saves over the title, at the
quality the origin delivers (its residuals' `q` and the bounds it states), exceed the prior's own
bytes plus the bytes of the records that name it. A prior that saves less costs every sink that
fetches it more than it returns.
An origin **SHOULD** send a per-title prior only where it is worth holding for the sinks it
reaches, as [prior-carriage](prior-carriage.md) §17 states for every set: the bytes it saves a sink
over the title, at the quality the origin delivers (its residuals' `q` and the bounds it states),
must exceed what holding it costs that sink, the records that name it counted with the pinned
renditions, and the prior's bytes charged to every sink that fetches it. A prior that saves less
costs every sink that fetches it more than it returns.
## 6. Accounting for sink-made pictures

202
wai/extensions/http-delivery.md
View file

@ -616,10 +616,10 @@ as the receipt carrying it binds it.
### 8.5 Parameter sets
A parameter set MAY be served as an immutable resource named by the digest its pin
states (`<base>/<sha256>`), under §5.3's cache headers with `application/octet-stream`;
a set in the form `wai.prior.set` is served as its listing under that digest and each
file the listing names under its own digest. The name locates the bytes and proves
nothing. A sink holds a fetched set only as [SPEC](../SPEC.md) §3.1's checks hold it over
states, at `<root><sha256>` below a **set root** (§8.7), under §5.3's cache headers with
`application/octet-stream`; a set in the form `wai.prior.set` is served as its listing
under that digest and each file the listing names under its own digest, below the same
root. The name locates the bytes and proves nothing. A sink holds a fetched set only as [SPEC](../SPEC.md) §3.1's checks hold it over
the bytes it received, the digest among them, check 5 being
[jwp-receipts](jwp-receipts.md) §7.9 with the statuses and grants it fetched as §8.4
states. Bytes served under a set's name that are not the set are not held
@ -687,6 +687,161 @@ that lists another cid for a URL or a media sequence number the chain bound, or
where it bound an object, is refused (`playlist_listing_rebound`). A refused revision
leaves the chain as it was.
### 8.7 Set roots and location hints
A **set root** is an absolute URI (RFC 3986 §4.3) that begins `https://`, with a host,
no userinfo, no query and no fragment, whose path ends in `/`, at most 2048 bytes long. A
set is at `<root><sha256>`: the root followed by the set's digest as 64 lowercase
hexadecimal digits, and a file a `wai.prior.set` listing names at the root followed by that file's
digest (§8.5). An origin that serves the sets its objects pin serves them below its
**well-known root**, `https://<host>[:<port>]/.well-known/wai-sets/`, its own host and
port, so a sink that holds an object can always name where to ask. Any other root is a
mirror, a cache, a peer or a node of the sink's own: what it serves is held by its
digest, as what the origin serves is.
**Location hints.** A list of set roots MAY be stated where a sink reads, before it
fetches, which sets content pins:
- in a MoQ catalog's `wai_priors` entry, as its member `locations`
([moq-streaming-format](moq-streaming-format.md) §4.2), for that entry's set;
- in a WAI multivariant playlist (§11.1), in one line,
`#EXT-X-SESSION-DATA:DATA-ID="science.transaction.wai.sets",VALUE="<root> <root>"`, for
every set an object of the presentation pins: the roots in order, one space (0x20)
between each two, as many as the hint has. The line carries `VALUE` and neither `URI`
nor `LANGUAGE`. RFC 8216 §4.3.4.4 does not admit two `EXT-X-SESSION-DATA` tags with the
same `DATA-ID` and the same `LANGUAGE`, so a playlist carries this line once; a set root
holds no space, so its roots read back as they were written.
A hint is read in its order. In a catalog, a `locations` that is not an array is ignored
whole, and each element that is not a string holding a set root is passed over
(`set_location_malformed`). In a playlist, the hint's line is the first line whose
attribute list reads as RFC 8216 §4.2 has one, whose `DATA-ID` is that quoted string and
that carries no `LANGUAGE`; a later such line, which RFC 8216 does not admit, is ignored.
The line's `VALUE`, split at every space, gives the hint's elements, each read as a
catalog's is: an empty element (two spaces together, or a space at either end) holds no
root and is passed over the same way. A hint line whose `VALUE` is absent or not a quoted
string, or that carries `URI`, is ignored whole. A root equal to one kept before it is
passed over, and the first 8 roots kept are the hint: the rest are ignored. A hint never
makes a catalog, a playlist or an object malformed, and it is never evidence: it names
where to ask.
**The pin stays identity-only.** No member of a pin locates its set
([SPEC](../SPEC.md) §3.1), and a sink MUST NOT read a location from a pin: a member a pin
carries that SPEC §3.1 does not define is ignored, whatever its name and value. A
location in a pin would be signed into every envelope that carries it, and would outlive
the server it names.
**Fetching a set.** A sink that is to hold a set it lacks (SPEC §3.1 check 6) asks the
roots in this order, each at most once for that set: the roots of its own configuration
it asks first (a node of its household, a cache it is enrolled with), the hint the set's
catalog entry or the presentation's playlist states, and, for content that came over
HTTP, the well-known root of the origin it fetched the playlist or the object from. At
each root:
1. It requests `GET <root><sha256>`, following redirects as its HTTP client does.
2. The response is `200` (`set_location_unavailable` otherwise) with no
`Content-Encoding` other than none (`content_coding_present`), or with a dictionary
coding that passes §8.8's checks, which then run in place of this step's.
3. It reads at most the bound it sets on the bytes of one set or file, and stops past it
(`set_too_large`).
4. The bytes' SHA-256 is the digest it asked for. Bytes that are not are discarded and
never held (`set_location_mismatch`).
When a step fails, the sink asks the next root. For a set in the form `wai.prior.set`,
the listing is fetched first and then each file it names, each by the same steps under
its own digest, asking first the root that served the listing and then the others in
order. For a set in the form `wai.prior.tree`, the head is fetched by these steps and the set
it names by range, piece by piece, from any of the roots ([prior-carriage](prior-carriage.md)
§19). When no root yields the set, the pin is refused as SPEC §3.1 states
(`prior_not_held`, or `digest_superseded`); a listing that is not one is
`set_malformed`, a file of it that no root yields is `part_mismatch`, and a listing whose
file at the path `wai-usage.json` does not read as a usage statement leaves the set not held
(`set_usage_malformed`, [prior-carriage](prior-carriage.md) §18.1). Dispatch then
continues as SPEC §4 and §7.1 state. A set fetched this way is held only as §8.5 holds
one. The order of the roots decides which server is asked, and never which bytes are
held.
Every request names the digest it asks for to the server it reaches and to every cache
on the way, which is what a sink's held sets disclose (§12.5).
Reference implementation: `prior_carriage::locations` (feature `prior_carriage`) reads a
catalog's and a playlist's hints (`read_catalog_hint`, `read_playlist_hint`), checks a
root (`is_set_root`), names a set's URL (`set_url`, `well_known_root`) and the order of
the roots, and runs the steps above over the responses its caller supplies
(`fetch_set`); it opens no connection. The corpus is
[`live-conformance/prior-carriage/`](../live-conformance/prior-carriage/), with
`live-conformance/prior_carriage_verify.py` its second implementation
([prior-carriage](prior-carriage.md) §15).
### 8.8 Parameter sets as compression dictionaries
A set's bytes, as §8.7 serves them, are a dictionary in the sense of RFC 9842 (*Compression
Dictionary Transport*): the SHA-256 that RFC's `Available-Dictionary` field carries for a
dictionary is the digest a pin states for a set in a one-file form, and the digest a listing
gives a file of a `wai.prior.set`. So a sink that holds a set can receive another, most often
its successor, coded against it by an HTTP stack that implements that RFC, and it holds the
result only when the result's SHA-256 is the digest it asked for, as §8.7 holds any set. The
coded bytes are a transfer and never a form: no pin names them
([prior-carriage](prior-carriage.md) §16).
- **Offering.** A server MAY send `Use-As-Dictionary: match="<path>*"`, `<path>` being the
path of the set root, with a set's or a listed file's `200` response, and no `match-dest`,
`id` or `type` (RFC 9842 §2.1): every set below the root then matches it, and
`Available-Dictionary` names it by the digest it is served under.
- **Asking.** A client sends `Available-Dictionary` only over `https` (RFC 9842 §8), naming
one set it holds. One whose HTTP stack implements RFC 9842 lets the stack choose (RFC 9842
§2.2.3: the longest `match`, then the most recently fetched). One that writes the field
itself names the set that the pin it fetches for `supersedes`, where it holds that set, and
otherwise sends none; for a listed file it MAY name the file at the same path of the
superseded set's listing. It lists `dcz`, and `dcb` if it decodes it, in
`Accept-Encoding` only when it sends `Available-Dictionary` (RFC 9842 §6.1). The value is
read as a Structured Field Byte Sequence (RFC 9651 §3.3.5) of exactly 32 bytes: padded
base64 between colons, its two pad bits zero, with spaces around it discarded.
- **Answering.** A server that holds the set a request names MAY answer `<root><sha256>`
with the requested set coded against it: `Content-Encoding: dcz` (Zstandard, RFC 8878,
with the named set as a raw dictionary, RFC 9842 §5) or `dcb` (Brotli with the named set
as its dictionary, RFC 9842 §4), with `Vary: accept-encoding, available-dictionary` (RFC
9842 §6.2). These two fields are the only exceptions §5.3 has for a set; an object is never
dictionary-coded. Otherwise it answers with the set as it is.
- **Checking.** For a response whose `Content-Encoding` is `dcz` or `dcb`, a client checks, in
order:
1. the request named a dictionary and listed that coding (`dictionary_unrequested`);
2. the body begins with the coding's header: for `dcz` the 8 bytes
`5e 2a 4d 18 20 00 00 00`, for `dcb` the 4 bytes `ff 44 43 42`
(`dictionary_coding_malformed`);
3. the 32 bytes after them are the SHA-256 of the set the request named
(`dictionary_mismatch`);
4. the window the coded stream states is one RFC 9842 has a client decode: for `dcz`, the
Zstandard frame's window (RFC 8878 §3.1.1.1.2, or its content size for a single
segment) is at most 8 MB or 1.25 times the dictionary's length, whichever is larger,
and never more than 128 MB; for `dcb`, at most 16 MB; MB read as 2^20 bytes
(`dictionary_window_too_large`);
5. the stream decodes against that set, as one whole frame and nothing after it
(`dictionary_coding_malformed`), to no more bytes than §8.7 step 3's bound
(`set_too_large`).
§8.7 step 4 then runs over the decoded bytes. A response refused by any check is
discarded. A client that named a dictionary and is refused by any of them but the bound
asks the same root once more without `Available-Dictionary` before it goes to the next.
- **Caches.** §9 holds for a set's own representation. A dictionary-coded response is the
server's: a cache stores it under its `Vary` (RFC 9111 §4.1) and serves it unchanged, and
never codes a set itself (§9 rule 1). Rule 4 does not apply to it.
- **No range of a coded set.** A client that asks for a range of a set sends no
`Available-Dictionary`, and refuses a coded response to it (`content_coding_present`): a
range of a coded body is not a range of the set.
- **What it discloses.** `Available-Dictionary` tells the server one set the client holds,
as `held_sets` does (§12.1). RFC 9842 has a client keep dictionaries as it keeps cookies
(§10 there), and §12.5 states the rules for sets.
Reference implementation: `prior_carriage::dictionary` writes and reads the two fields'
values (`available_dictionary`, `parse_available_dictionary`, `use_as_dictionary`), runs the
checks above and decodes `dcz` (`check_coded`, through the Zstandard library its neural
decoders use, the dictionary given as raw content), and codes a set against another as a
server does (`encode_dcz`). It decodes no `dcb`, so a sink built on it lists only `dcz`, and
`prior_carriage::locations::fetch_set` names a dictionary only for a set in a one-file form.
The corpus states the command with which a stock Zstandard tool checks its `dcz` body
([prior-carriage](prior-carriage.md) §16).
## 9. Cache rules
An HTTP cache that stores and serves objects under this extension is a transparent
@ -1049,6 +1204,41 @@ output had the published digest (an exact tier), or decoded without error (any o
tier). A report states |C|, the window, how C was sampled and the descriptor version. It
is a share of sessions, not of devices or people.
### 12.5 What a sink discloses of the sets it holds
Which sets a sink holds depends on what it has fetched before, so a list of them can identify
the sink across the sites it is sent to, much as a cookie does, and the digest of a per-title
prior names the title it was fitted to ([prior-carriage](prior-carriage.md) §18.3). Three
things disclose a held set: the descriptor's `held_sets`, `Available-Dictionary` (§8.8), and
the absence of a fetch where a sink that lacked the set would have fetched it. A sink keeps
these rules:
1. **Per top-level site.** A sink keeps the sets it holds by the top-level site (the scheme
and registrable domain of the top-level document, or of the application that embeds the
sink) it fetched each for. A set held for one site is, for another, not held: it is not
listed or named there, and a sink that is not to disclose its other sites fetches the set
again for that site rather than reading it from its store.
2. **Off unless turned on.** A sink sends no `held_sets` to a site unless its user, or the
deployer of a sink with no user, has turned it on for that site.
3. **Only what a choice needs.** `held_sets`, where it is sent, lists only the sets the sink
holds for that site that the content it is choosing among pins, as its heads or its catalog
state them; never the rest of what the sink holds. A set a key log the sink holds revokes is
never listed.
4. **Never across sites.** A descriptor built for one site is not sent to another, and
`Available-Dictionary` names only a set held for the site the request is made for.
5. **Clearable.** Clearing a site's data clears the sets held for it, as it clears the
dictionaries RFC 9842 has a client keep for it (§10 there).
6. **Counts, not lists.** A report of which sets sinks hold is an aggregate over sinks, never
one sink's list. [telemetry-aggregation](telemetry-aggregation.md) is how WAI aggregates a
measure; this revision defines no measure of held sets.
A fetch by digest names the set to the server and every cache on the way (§8.7): a sink that
fetches a per-title prior from a mirror tells that mirror which title it is about to decode.
Reference implementation: `prior_carriage::privacy::held_sets_for` builds the member under
rules 1 to 4 from a store keyed by site, and `prior_carriage::privacy::may_name_dictionary`
decides rule 4 for `Available-Dictionary`. No code in this tree sends the descriptor.
## 13. Reasons
| reason | raised at |
@ -1065,6 +1255,9 @@ is a share of sessions, not of devices or people.
| `receipt_not_served`, `receipt_index_malformed`, `receipt_index_too_large`, `receipt_too_large`, `receipt_legacy_group`, `receipt_fetch_budget_exhausted`, `receipt_fetch_bytes_exhausted` | §8.1, §8.2 |
| `context_not_held` | §8.2 (jwp-receipts §3.1 step 6) |
| `key_log_not_from_origin`, `key_log_too_large`, `grants_file_malformed` | §8.4 |
| `set_location_malformed` | §8.7 (a hint's root passed over, never a refusal) |
| `set_location_unavailable`, `set_too_large`, `set_location_mismatch` | §8.7 (the next root is asked) |
| `dictionary_unrequested`, `dictionary_coding_malformed`, `dictionary_mismatch`, `dictionary_window_too_large` | §8.8 (the root is asked again without a dictionary) |
| `codecs_malformed`, `codecs_not_media`, `codecs_candidate`, `codecs_no_common_capability`, `codecs_undeclared`, `codecs_layer_unbound` | §4 |
| `base_unanswered` | §10.1 |
| `not_a_layer`, `layer_tag_malformed` | §10.4 |
@ -1232,6 +1425,7 @@ request ends at `covered_len`. Without `h`, the first request is 16,369 bytes.
| RFC 9110 §7.7, §8.4, §8.8.3, §13.1.5, §14.1–§14.6, §15.3.7, §15.5.17 | read |
| RFC 9111 §5.2.2.6 · RFC 8246 §2 · RFC 6381 §3.2 · RFC 6838 §3.1, §3.2, §5.6 | read |
| RFC 9530 §2, §3 · RFC 9218 §4.1, §4.2, §5 | read |
| RFC 9842 §2.1, §2.2, §4, §5, §6, §8, §9.3, §10 | read, for §8.8 |
| RFC 8216 §3.1, §3.3, §4.3.4.2, §4.3.4.4 · the HLS second-edition draft (its date only) | read |
| RFC 8006 `MI.*` objects | read |
| The HLS second-edition draft (draft 22): `EXT-X-PART` (its placement before the parent segment's URI, and the media segment tags before its first part), `EXT-X-PART-INF`, `EXT-X-SERVER-CONTROL` (`PART-HOLD-BACK` required with parts, at least twice the part target), `EXT-X-PRELOAD-HINT` | read |

3
wai/extensions/moq-streaming-format.md
View file

@ -377,6 +377,7 @@ needs, and which tier each is offered at:
| `sha256` | REQUIRED, string | A pin's `sha256`: 64 lowercase hexadecimal digits. |
| `form` | OPTIONAL, string | The pin's `form`. Absent, or `null`, where the pin declares none. |
| `tier` | OPTIONAL, string | The tier the publisher offers the set at, from SPEC §3.1's vocabulary. Advisory (below). |
| `locations` | OPTIONAL, array of strings | Set roots to fetch the set from, in order: a location hint of [http-delivery](http-delivery.md) §8.7: where to ask, never evidence. |
An entry **lists** a pin for a capability when its `capability` is that capability, its
`sha256` is the pin's, and its `form` is the pin's: both absent, or the same string. A
@ -387,6 +388,8 @@ member of an entry that this revision does not define is ignored.
and `form`, and each `capability` one the track's `wai_capabilities` lists. A sink that
reads anything else refuses the track (`catalog_priors_malformed`): it does not
subscribe to it. A `tier` this revision does not register is read as `none` (SPEC §3.1).
`locations` is read as http-delivery §8.7 reads a hint: a value that does not read as
one is passed over there, and never refuses the track or the entry.
- **Exhaustive.** Where a track carries `wai_priors`, the publisher **MUST** list in it
every pin its objects carry. An object **breaks the catalog** when:
1. it carries a pin that no entry lists, either for its `capability` or for its

438
wai/extensions/prior-carriage.md
View file

@ -15,7 +15,13 @@
> `codecs::{CAP_PRIOR_SET, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_GGUF, CAP_PRIOR_NNC,
> CAP_TENSOR_NNC}`, and `prior_wire` (feature `prior_wire`) for §9 to §14; corpus
> [`live-conformance/prior-forms/`](../live-conformance/prior-forms/), second
> implementation `live-conformance/prior_forms_verify.py`.
> implementation `live-conformance/prior_forms_verify.py`. §15 to §19 say where a set is
> fetched and how it travels: set roots and location hints, a successor coded against its
> predecessor as an RFC 9842 dictionary, when a set is worth holding, the usage statement, and
> two more registered forms, `wai.prior.sealed` (a set released by key) and `wai.prior.tree` (a
> set verified per piece); reference `prior_carriage` (feature `prior_carriage`), corpus
> [`live-conformance/prior-carriage/`](../live-conformance/prior-carriage/), second
> implementation `live-conformance/prior_carriage_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
## 1. A prior has a digest, and had no form
@ -78,6 +84,8 @@ operand needs a form both ends can name.
| `wai.prior.set` | the listing of a parameter set of one or more files (SPEC §3.1) | registered; WAI-native; established exact (§5) |
| `wai.prior.safetensors` | a parameter set as one safetensors file: an 8-byte header length, a JSON header naming each tensor's type, shape and bytes, the tensors' bytes (§9.1) | registered; established exact (§5) |
| `wai.prior.gguf` | a parameter set as one GGUF version 3 file: key-value metadata, tensor descriptions, aligned tensor data (§9.2) | registered; established exact (§5) |
| `wai.prior.sealed` | a one-file set encrypted under a key its origin releases: the inner form, the key's identifier, the counter block, the plaintext's SHA-256, the set's usage statement, then the set under AES-128 in counter mode (§18.2) | registered; WAI-native; established exact (§5) |
| `wai.prior.tree` | the head of a chunk tree over a one-file set: the inner form, the set's length, SHA-256 and BLAKE3 hash, and the BLAKE3 chaining value of each piece (§19) | registered; WAI-native; established exact (§5) |
| `wai.prior.nnc` | the parameters of a trained network as a bitstream conforming to **ISO/IEC 15938-17:2024** (NNC, 2nd edition: parameters and their incremental updates, DeepCABAC entropy coding) | registered; a coded form |
| `wai.tensor.nnc` | general tensorial data — split-inference feature maps, 3D/4D scene parameters — coded with NNC | **candidate**: payload format not fixed |
@ -102,6 +110,13 @@ Registry classification (`codecs::capability_*`):
tensor's bytes as the file holds them, and §9 leaves no choice of which bytes
those are; licence royalty-free (the safetensors layout is published with its
reference implementation under Apache-2.0, GGUF under MIT).
- **`wai.prior.sealed`**: fidelity tier lossless and replicate class `StandardDefined` —
opened under its key, it hands over the inner set's bytes, checked against the SHA-256 its
header states (§18.2); licence royalty-free (WAI-native, over AES-128, FIPS 197, in counter
mode, NIST SP 800-38A).
- **`wai.prior.tree`**: fidelity tier lossless and replicate class `StandardDefined` — it hands
over the set it names, every piece checked against the head and the whole against the SHA-256
the head states (§19); licence royalty-free (WAI-native, over BLAKE3).
For the NNC entries:
@ -220,13 +235,18 @@ reasons:
published editions and for any later one.
- Therefore an envelope with `intent = replicate` MUST NOT rest on a prior whose
declared form is not established exact. The registry lists the forms that are
established (`codecs::prior_form::form_reconstruction_is_exact`). It lists three.
established (`codecs::prior_form::form_reconstruction_is_exact`). It lists five.
`wai.prior.set`: reading a set through its listing compares digests and hands
the decoder each file's bytes unchanged, so every sink that holds the set reads
the same parameters. `wai.prior.safetensors` and `wai.prior.gguf`: a reader that
follows §9 finds each tensor's bytes where the file's header puts them and hands
them over unchanged, and §9 refuses every file whose header could be read two
ways. No coded form is listed: within the subset of §12 the integers an NNC reader
ways. `wai.prior.sealed`: opening it under its key is a function of its bytes and the key,
the plaintext is checked against the SHA-256 the header states, and its inner form is a
capability's one file, `wai.prior.safetensors` or `wai.prior.gguf` (§18.2). `wai.prior.tree`:
every piece is checked against the head and the whole set against the SHA-256 the head
states, and its inner form is one of those three or a sealed set
(§19). No coded form is listed: within the subset of §12 the integers an NNC reader
obtains are a function of the bits (§12.4), but the form also covers quantised
payloads, whose reconstruction is the open question above. In the reference implementation the rule is part of
`Prior::is_cross_hardware_exact`, which requires `Prior::form_preserves_exactness`
@ -802,6 +822,418 @@ regenerates it and loads every prior in every form into the integer kernels, and
`live-conformance/prior_forms_verify.py`, written from this text and sharing no code
with `wai-rs`, is its second implementation.
## 15. Where a set is fetched
A pin names a set by its digest and says nothing of where the set is (SPEC §3.1). Over HTTP
a set is served at `<root><sha256>` below a **set root**, an origin's own being
`/.well-known/wai-sets/`, and a sink that lacks a set asks, in order, the roots of its own
configuration, the location hints that a MoQ catalog's `wai_priors` entry or a WAI
multivariant playlist states, and the origin's well-known root
([http-delivery](http-delivery.md) §8.5, §8.7). Any server may serve a set. Whatever served
the bytes, a sink holds them only when their digest is the pin's, and then under SPEC §3.1's
checks like any other set; a server that serves other bytes under the set's name is passed
over for the next root. So a mirror, a cache, a peer or a node of the sink's household that
holds a set serves it as well as the origin does, and no central server is on the path.
The pin carries no location. A location in a pin would be signed into every envelope that
carries it and outlive the server it names; a hint beside the pin can change without
touching an envelope, and is never evidence.
A file of a `wai.prior.set` is a resource of its own under its own digest, so a sink that
holds a set's predecessor fetches only the files the two do not share (SPEC §3.1,
*Updates*).
**Reference implementation and corpus.** `prior_carriage` (feature `prior_carriage`) is the
reference implementation, `prior_carriage::locations` for this section. The corpus
[`live-conformance/prior-carriage/`](../live-conformance/prior-carriage/) holds, for this
section, hints read from catalog entries and playlists, and fetches over the responses each
case states: a set held from the second root its hint lists after the first served other
bytes, a root that does not answer and one that answers with a content coding, a set past a
sink's bound, a listing and its files held from two roots, a file no root holds, and a pin
with a member that names a location, which no fetch reads. `wai_prior_carriage_vectors
gen|verify` writes and checks it, `tests/prior_carriage.rs` requires it to regenerate byte
for byte, and `live-conformance/prior_carriage_verify.py`, written from this text and
http-delivery's and sharing no code with `wai-rs`, is its second implementation.
## 16. A successor coded against its predecessor
A re-tuned set is a new set under a new digest (SPEC §3.1, *Updates*), and most of its bytes
are often its predecessor's. RFC 9842 (*Compression Dictionary Transport*) lets an HTTP stack
send a response coded against a dictionary the client already holds, named by its SHA-256 in
`Available-Dictionary`. A set's digest in a one-file form is the SHA-256 of that one file, so a
held set is a dictionary under the very name its pin gives it, with nothing to map: a sink that
holds a predecessor names it, a server that holds both answers with the successor coded
against it (`dcz` or `dcb`), and the sink decodes the response and holds the successor only
when the decoded bytes' SHA-256 is the successor's pin ([http-delivery](http-delivery.md)
§8.8). A server that codes nothing, or codes against the wrong set, costs bytes and never
correctness.
- **The coded bytes are a transfer.** No pin names them, and no form is registered for them,
as none is for an NNC incremental update (§2): what a sink holds, and what a pin covers, is
the self-contained successor the decoding yields. A delta that would need a third set to
decode cannot arise: the dictionary is the one set the request named, checked by the digest
the coded body repeats.
- **A set of several files** is coded file by file. Each file of a `wai.prior.set` is served
under its own digest, so a client MAY name, for a file of the successor, the file at the same
path in its predecessor's listing; a file the two share is not fetched at all (§15).
- **Which set is named.** A client that writes the field itself names the set the successor's
pin `supersedes`, when it holds it. A client whose HTTP stack keeps dictionaries itself names
what that stack chooses; the result is checked the same way.
- **Privacy.** Naming a held set tells the server that the sink holds it, as the Sink
Capability Descriptor's `held_sets` does. http-delivery §12.5 and RFC 9842 §10 say what that
discloses and what a sink may name.
**Reference implementation and corpus.** `prior_carriage::dictionary`; it decodes `dcz` only.
The corpus holds `title`, the upscale corpus's per-title `WSR1` prior, which was fine-tuned
from its generic prior `generic`, coded against `generic` (`files/title-from-generic.dcz`), and
the fetches of
http-delivery §8.8 over it: the successor held through its predecessor; a coded response no
dictionary was named for, and one in a coding the sink did not list; a header naming another
dictionary, a body that is not a `dcz` header, a frame whose window passes RFC 9842's limit
(`files/title-window-16m.dcz`), a truncated frame, each followed by the same root asked without
a dictionary; a decoding that yields another set; and one past the sink's bound. Its `stock`
entry states the check a stock Zstandard tool makes: the body after its 40-byte header, decoded
against `generic`, has `title`'s SHA-256.
## 17. Amortisation, bootstrap and generations
A set costs bytes before it saves any: a sink fetches it once and then decodes the titles
pinned to it in fewer bytes than their classical renditions take. This section states when
that is worth it, charged to each sink, and how a publisher re-tunes a set without stranding
the titles coded against the one before.
### 17.1 Terms
- A **generation** is a set and the titles an origin codes against it.
- A **catalog epoch** is the period in which an origin codes new titles against one
generation. An origin freezes a set for its epoch: every title it codes in the epoch pins
that set, and a re-tuned set begins the next epoch, as a new generation whose pin
**SHOULD** name the previous epoch's set in `supersedes` (SPEC §3.1).
- A title's **saving** to a sink is the bytes of the rendition the sink fetches when it does
not decode against the set (the classical rendition, at the quality the origin delivers),
less the bytes of the pinned rendition and of the records that name the set and that the
sink fetches with it ([exact-upscale](exact-upscale.md) §4's upscale records among them).
Both are in the heads the sink fetches: a `WAI2` rendition table states every component's
length (SPEC §7.1).
- A generation's **cost** to a sink is 0 when the sink holds its set; the coded bytes of §16
when the sink holds the set this one is coded against and the origin offers that coding;
and the set's bytes otherwise (for a `wai.prior.set`, its listing and every file the sink
does not hold).
- The **span** is the period over which a sink's fetches are counted, from one instant to a
later one. Whoever applies this section states the span with the outcome, and two outcomes
over different spans are not compared.
- The **cap** is the most generations the origin plans a sink to hold at once, at least 1.
### 17.2 The rule, charged per sink
For one sink, and the titles it is expected to fetch within the span, each with the number of
times it is fetched:
1. A generation's **saving** is the sum, over its titles the sink fetches whose saving is
above 0, of the number of fetches times the saving. A title whose saving is 0 or less takes
its classical rendition whatever the sink holds.
2. Its **net** is its saving less its cost to that sink. It is **worth holding** when the net
is above 0: the bytes the set saves must exceed what holding it costs.
3. The sink holds the generations worth holding, at most the cap, the largest net first and,
at an equal net, the later generation first. When fewer than the cap are kept, it keeps
the generations it already holds, the later first, until the cap is reached; any other it
holds may be evicted. Costs are those at the time the rule is applied, so a successor
coded against a predecessor this step evicts is fetched before the predecessor goes.
4. A title is decoded against its set when its generation is kept and its saving is above 0,
and takes its classical rendition otherwise.
The cost is charged per sink. A set's bytes are counted in full for every sink that fetches
it, never divided among the sinks a publisher serves: every one of them fetches the whole set.
So a set that saves each of a thousand sinks less than its own size is not worth sending to
any of them, however much the thousand save together.
**An origin SHOULD** encode a title's pinned rendition, and list its set for sinks to fetch
(§15), only where the set's generation is kept for the sinks the content reaches by the
origin's own estimate of their fetches; **a sink SHOULD** fetch a set only where its
generation is kept over the fetches the sink expects.
### 17.3 Bootstrap
- **First contact.** A sink that has disclosed no held set counts as one that holds none:
every cost is the set's bytes. The Sink Capability Descriptor's `held_sets` is how a sink
says what it holds, and it is sent only under http-delivery §12.5's rules: off unless turned
on for the site, and then only what the site's content pins.
- **The classical rendition.** An encoder whose content reaches sinks it does not control
carries a classical rendition beside each pinned one (SPEC §4), so a sink for which a set is
not worth holding takes the classical rendition and fetches nothing.
- **Fetching.** A sink for which a generation is kept fetches its set as §15 states, before it
decodes a title pinned to it, and holds it under SPEC §3.1.
### 17.4 Generations
Older titles keep the digests they were coded against, so a re-tuned set leaves the back
catalog pinned to the generations before it. An origin either keeps serving those titles,
which a sink then decodes against their own generation where step 3 keeps it and classically
otherwise, or re-encodes a title against a later set: a new envelope with a new pin, while
the old envelope still decodes wherever its set is held. Freezing a set per epoch bounds the
generations a back catalog spans, the cap bounds the generations a sink holds, and §16 makes
a successor cost a sink only the coded bytes when the sink holds its predecessor.
**Reference implementation and corpus.** `prior_carriage::amortise::plan` applies §17.2 to a
stated plan: a span, a cap, the generations in epoch order (each with its set's bytes and,
where it is coded against an earlier one, that generation and the coded bytes), the titles
(each with its generation and its pinned and classical bytes) and one or more sinks (each with
the generations it holds and its fetches per title). For each sink it gives each generation's
saving, cost and net, the generations kept, fetched and evicted, each title's rendition, and
the bytes the sink receives beside the bytes it would receive with no set. The corpus's
worked plans: a set worth holding and one that is not; a net of exactly 0, which is not worth
holding; a held set, which costs nothing; a title whose pinned rendition is larger; a
successor costing its coded bytes where its predecessor is held and its whole set where it is
not; a sink holding two generations when a third is worth fetching, with a cap of two, which
evicts the one with the smallest net, and with a cap of three; an equal net, where the later
generation is kept; and three sinks that would together save more than the set's size and
each save less, none of which fetches it. Plans that do not read are refused
(`plan_malformed`).
## 18. Usage, access and leakage
A set fitted to a catalog's frames is the rights holder's as much as the frames are. This
section states the terms that travel with a set, a form that keeps a set's bytes unreadable to
any sink its origin has not released a key to, and what a set can disclose of the frames it
was fitted to.
### 18.1 The usage statement
A **usage statement** is a JSON object with exactly these members, read from JSON text in which
no object names a member twice, at most 65 536 bytes long:
| member | value |
|---|---|
| `kind` | `"wai-set-usage"` |
| `licence` | the licence the set's bytes are under, as an SPDX licence expression (*SPDX Specification* 2.3, Annex D), or `null` where the statement names none |
| `terms` | an `https` URI of the terms under which the set may be held and decoded, 8 to 2048 bytes of `!` to `~`, or `null` |
| `content_derived` | `true` when the set's parameters were fitted to frames of content (a per-title or a per-catalog prior), `false` otherwise |
| `redistribute` | `true` when a holder may serve the set to others, `false` when only its origin, and caches that serve for the origin, may |
| `ai_usage` | an [ai-usage](ai-usage.md) §3 statement, read as ai-usage §3.6 reads one, governing uses of the set's bytes other than decoding the content pinned to it (training on them, extracting from them), or `null` |
A licence expression is read by the grammar of SPDX 2.3 Annex D: simple expressions (a licence
identifier, with `+` or not, or `LicenseRef-` and an identifier string, with `DocumentRef-`, an
identifier string and `:` before it or not), each with `WITH` and an exception identifier or
not, joined by `AND` and `OR` and grouped by parentheses; the operators upper case; tokens
separated by single spaces; at most 1024 bytes. An identifier is checked for its characters
(letters, digits, `-` and `.`), and is not looked up in the SPDX licence list. A statement that
breaks a rule of this section is refused (`set_usage_malformed`), and so is one whose
`ai_usage` ai-usage refuses.
- **Where it is bound.** A statement is bound to a set only as part of the bytes the set's pin
covers: in a `wai.prior.set`, as the file of the top-level path `wai-usage.json` of its
listing, listed like any other file; in a `wai.prior.sealed`, in its header (§18.2). A file
of that name at any other path of a listing is a file of the set and no statement, and a
statement found anywhere else (served beside a set, stated in a catalog, quoted in a
playlist) governs nothing. Since the pin covers it, a statement cannot be dropped or changed
without the set failing its pin.
- **A bound statement that does not read.** A sink that holds a listing and every file it names
(SPEC §3.1) reads the file of the path `wai-usage.json`, where the listing has one, as a
statement. One that breaks a rule of this section leaves the set not held
(`set_usage_malformed`), as §18.2 step 3 does for a sealed set: a sink holds a set with the
terms bound to it, or does not hold it.
- **Redistribution.** A holder of a set whose bound statement says `redistribute: false`
**MUST NOT** serve it to any other party, nor name a root of its own for it in a hint (§15),
other than as an HTTP cache that stores and serves it for its origin under
[http-delivery](http-delivery.md) §9.
- **No statement.** A set with no bound statement states no terms: nothing in this extension
grants or withholds a use of it.
- **What a statement is not.** A statement says what the rights holder permits. It keeps the set
from no one: §18.2 does that.
### 18.2 Sealed sets: `wai.prior.sealed`
A **sealed set** is a set in a one-file form encrypted under a 16-byte AES-128 key that its
origin releases to the sinks it entitles. Its pin names the sealed bytes, so a mirror, a cache
or a peer that holds them serves them as it serves any set (§15), and a sink checks them by
digest before it has a key. All integers are little-endian.
| offset | size | field |
|---|---|---|
| 0 | 4 | magic `WSK1` |
| 4 | 1 | `k`, the inner form's length, 0 to 64 |
| 5 | `k` | the inner form, bytes `!` to `~`: empty for a capability's one file (a form-less pin), or `wai.prior.safetensors` or `wai.prior.gguf` |
| 5 + `k` | 16 | `kid`, the key's identifier |
| 21 + `k` | 16 | `iv`, the initial counter block |
| 37 + `k` | 32 | `inner`, the SHA-256 of the plaintext: the set's digest in its inner form |
| 69 + `k` | 4 | `u`, the usage statement's length, 0 to 65 536 |
| 73 + `k` | `u` | the usage statement (§18.1), or nothing |
| 73 + `k` + `u` | 8 | `L`, the plaintext's length, 1 to 2^40 |
| 81 + `k` + `u` | `L` | the ciphertext |
The file is exactly `81 + k + u + L` bytes. The ciphertext is the plaintext under AES-128
(FIPS 197) in counter mode (NIST SP 800-38A §6.5), the counter block starting at `iv` and
incremented as one 128-bit big-endian integer for each 16-byte block; decryption is the same
operation.
A sink reads a sealed set in this order, the first check that fails being the reason:
1. the layout above (`sealed_malformed`);
2. the inner form, which is admitted (`sealed_inner_form`): none, `wai.prior.safetensors` or
`wai.prior.gguf`, as the table above lists, and no other string;
3. the usage statement, where `u` is not 0 (`set_usage_malformed`);
4. the key for `kid`: a sink that holds none has bytes it cannot read, which SPEC §3.1 check 6
does not hold (`prior_not_held`, with `set_key_absent`);
5. the plaintext's SHA-256 is `inner` (`set_key_mismatch`: another key, or a header that
misstates its set; the set is not held).
Opened, the plaintext is the set in its inner form, read as that form is read (§9, or the
capability's one file). The set has two identities, the pin's digest over the sealed bytes and
`inner`, and SPEC §3.1 check 5 runs over both, so a set revoked under one is not reached under
the other.
- **The key.** `kid` names the key as ISO/IEC 23001-7 names a content key, by a 16-byte
identifier, and the key is a 16-byte AES-128 key. An origin releases it as it releases a
content key, by a key delivery that carries keys by their identifier (W3C Clear Key, or a
content protection system: [key-release](key-release.md) §3.2). This revision registers no
scheme of key-release §3.1 for it, so key-release's proxy form names none.
- **Sealing.** An encoder **MUST NOT** seal two different plaintexts under one key with counter
ranges that overlap, and **SHOULD** use a fresh key for each sealed set.
- **Ranges.** A large sealed set is fetched by range from several peers as a chunk tree whose
inner form is `wai.prior.sealed` (§19), whose ranges verify before the key is used.
### 18.3 Leakage
- **A content-derived set carries its content.** Parameters fitted to a title's frames can hold
features of those frames, and a decode through them can reproduce them; how much is not
bounded by anything in this extension, and WAI states no measure of it. A set whose bound
statement says `content_derived: true`, fitted to content that is under access control,
**SHOULD** be sealed (§18.2), its key released only to sinks entitled to that content, and
**SHOULD** state `redistribute: false`.
- **A digest names a title.** The digest of a per-title prior names the title it was fitted to,
so a list of the sets a sink holds says which titles it decoded. The Sink Capability
Descriptor's `held_sets`, `Available-Dictionary` and a fetch by digest each disclose a held
set, and [http-delivery](http-delivery.md) §12.5 states what a sink may disclose.
- **A sealed header names its plaintext.** `inner` is the plaintext's SHA-256: a party that
holds the plaintext can tell which sealed set holds it, and learns nothing else of it. `kid`
is shared by every set sealed under one key, so a fresh key for each set keeps sets from being
linked by their key.
- **Receipts.** A receipt that names the set a decode read names a title for a per-title prior,
and is disclosed as the receipt is.
**Reference implementation and corpus.** `prior_carriage::usage` reads a statement;
`prior_carriage::sealed` seals and opens a set (`seal`, `read_header`, `open`; counter mode
through the AES-128 that the `hpke` feature already links); `prior_carriage::privacy` builds
`held_sets` (http-delivery §12.5). The reference pin verifier reads no sealed set
(`pin::READABLE_FORMS` holds `wai.prior.set`), so the reference sink refuses a pin in this form
with `form_unreadable`; a sink that holds the key calls `prior_carriage::sealed::open` and then
holds the plaintext under its inner form. `prior_carriage::locations::fetch_set` reads the
statement a listing binds once it holds the listing's files. The corpus holds usage
statements, `title` sealed with a content-derived statement (`files/title.wsk1`) and `generic`
sealed as a capability's one file (`files/generic.wsk1`), each under the corpus's published
test key, which protects nothing: opened with the key; without it, refused `set_key_absent`;
under another key and with a ciphertext byte changed, refused `set_key_mismatch`; and headers
broken one way each, among them the inner forms of a coded form, a listing, a sealed set and a
tree. It holds three fetches of a listing with a file named `wai-usage.json`: one at the
top-level path, held with the terms it states; one there that does not read, which leaves the
set not held (`set_usage_malformed`); and one at the path `a/wai-usage.json`, a file of the set
that binds nothing.
## 19. The chunk-tree form `wai.prior.tree`
A set's SHA-256 can be checked only once every byte of the set is held, so a large set fetched
from peers that may serve other bytes is verified only at its end, and a changed byte costs the
whole fetch. An envelope does not have that problem: its content hash is a BLAKE3 tree whose
parts verify on their own ([http-delivery](http-delivery.md) §6). The chunk-tree form gives a set
the same: a small **tree head** names the set by its length, its SHA-256 and its BLAKE3 hash, and
lists a BLAKE3 chaining value for each **piece**, a run of 2^`g` of BLAKE3's 1024-byte chunks. A
pin in the form `wai.prior.tree` names the head's SHA-256, so the head is checked as any set is
(§15), and every piece then verifies as it arrives, from any peer.
### 19.1 The head
All integers are little-endian.
| offset | size | field |
|---|---|---|
| 0 | 4 | magic `WTR1` |
| 4 | 1 | `g`, 0 to 16: a piece is 1024 · 2^`g` bytes (`P`) |
| 5 | 1 | `k`, the inner form's length, 0 to 64 |
| 6 | `k` | the inner form, bytes `!` to `~`: a form §18.2 admits inside a sealed set, or `wai.prior.sealed` |
| 6 + `k` | 8 | `L`, the set's length in bytes, at least 1 |
| 14 + `k` | 32 | `inner`, the SHA-256 of the set's bytes: its digest in its inner form |
| 46 + `k` | 32 | `root`, the BLAKE3 hash of the set's bytes |
| 78 + `k` | 4 | `n`, the number of pieces: ⌈`L` / `P`⌉, 2 to 65 536 |
| 82 + `k` | 32 · `n` | the chaining value of each piece, in order |
The head is exactly `82 + k + 32·n` bytes. Piece `i` is bytes `i·P` to `min((i+1)·P, L)` of the
set; its **value** is the non-root chaining value of those bytes as the subtree of BLAKE3's tree
over the whole set whose first chunk counter is `i · 2^g` (the BLAKE3 specification §2.1, in
hash mode). Every piece but the last is 2^`g` whole chunks, so each piece is a
subtree of that tree, and the values merge to its root: over `m` values, with `m` at least 2, the
left child is the merge of the first `2^⌊log₂(m−1)⌋` and the right child of the rest, a parent
node without the ROOT flag below the top and with it at the top. That root is `root`, which is
the BLAKE3 hash of the set's bytes: hashing a held set with any BLAKE3 implementation checks it
against its head.
A sink reads a head in this order, the first check that fails being the reason:
1. the layout above (`tree_head_malformed`): the magic, `g`, `k`, `L`, `n` equal to
⌈`L` / `P`⌉ and from 2 to 65 536, and the length;
2. the inner form, which is admitted (`tree_inner_form`): none, `wai.prior.safetensors`,
`wai.prior.gguf` or `wai.prior.sealed`, as the table above lists, and no other string;
3. the root the values merge to is `root` (`tree_root_mismatch`).
A set that fits in one piece is pinned in its inner form: its head would add nothing.
### 19.2 Fetching by range
The set's bytes are served as a one-file set under `inner`, at `<root><inner>` below any set
root (§15), so every server that holds the set serves it whole or by range. A sink that holds
the head:
1. requests pieces with `Range: bytes=<i·P>-<min((i+1)·P, L) − 1>`, one piece or several whole
pieces in a request, from as many sources as it likes at once, and checks each response as
[http-delivery](http-delivery.md) §7.1 checks a range response (`content_coding_present`,
`content_range_mismatch`, and the others there), a server that does not answer being
`set_location_unavailable`;
2. checks each piece against its value as it arrives, and discards one that does not match
(`tree_piece_mismatch`), fetching that piece from another source: a changed byte costs one
piece, and a verified piece is never fetched again;
3. once every piece is held, requires the set's SHA-256 to be `inner`
(`tree_digest_mismatch`: a head that misstates its set, which is not held).
A piece that no source yields leaves the set not held (SPEC §3.1 check 6, `prior_not_held`). The
held set is then the inner form's set, whose digest is `inner`: it satisfies a pin in its inner
form too. The set has two identities, the head's digest and `inner`, and SPEC §3.1 check 5 runs
over both. A sink sends no `Available-Dictionary` with a range request (http-delivery §8.8).
The order of the sources is the sink's; the reference fetches piece `i` first from source
`i mod m` of its `m` sources, then from each source after it in turn.
**Reference implementation and corpus.** `prior_carriage::tree` writes and reads a head
(`write_head`, `read_head`, `root_of`), checks a piece (`piece_verifies`) and fetches every piece
from several sources over the responses its caller supplies (`fetch_pieces`), through the BLAKE3
hazmat API the HTTP head proof uses; its unit tests require `root` to be the BLAKE3 hash for
every piece size and set length they try. The reference pin verifier reads no tree head
(`pin::READABLE_FORMS`), so the reference sink refuses a pin in this form with
`form_unreadable`. The corpus holds the heads of `generic` in pieces of one chunk and of two
(`files/generic.wtr1`, `files/generic-g1.wtr1`) and of the sealed `title`
(`files/title-sealed.wtr1`, inner form `wai.prior.sealed`), heads broken one way each (among
them the inner forms of a coded form, a listing, a tree and a string no form registers), and
fetches from two peers: both whole; one with a byte changed in piece 3, refused at that piece,
which the other peer then yields; one that does not answer; both changed at one piece, which
leaves the set not held; one with a content coding; one whose bytes end early; the sealed
`title`; and a head whose `inner` misstates its set.
## 20. Refusals of §15 to §19
The reasons of http-delivery §8.7 and §8.8 are listed in its §13. This section's are:
| code | where |
|---|---|
| `plan_malformed` | §17: a plan with a span that does not run forward, a cap of 0, generations out of epoch order or named twice, a coding against a generation that is not an earlier one, a title of no stated generation, a sink that holds or fetches what the plan does not state, or a member of another type |
| `set_usage_malformed` | §18.1: a statement that does not read, and a listing whose bound statement does not read, which leaves the set not held; §18.2 step 3 |
| `sealed_malformed` | §18.2 step 1 |
| `sealed_inner_form` | §18.2 step 2 |
| `set_key_absent` | §18.2 step 4: the set is not held (`prior_not_held`) |
| `set_key_mismatch` | §18.2 step 5: the set is not held |
| `tree_head_malformed` | §19.1 step 1 |
| `tree_inner_form` | §19.1 step 2 |
| `tree_root_mismatch` | §19.1 step 3 |
| `tree_piece_mismatch` | §19.2 step 2: that piece is fetched from another source |
| `tree_digest_mismatch` | §19.2 step 3: the set is not held |
## Appendix A — The integer payload's arithmetic coding (normative)
The integers of an `NNR_PT_INT` payload are binarised and coded with context-adaptive

1
wai/live-conformance/README.md
View file

@ -41,6 +41,7 @@ and the independent verifier of each, beside it.
| [`worlds/`](worlds/) | [live-worlds](../extensions/live-worlds.md) (`wai.live.world`) | `wai_live_world_vectors gen` | `live_worlds_verify.py` |
| [`synthesis-mark/`](synthesis-mark/) | [synthesis](../extensions/synthesis.md) §7.4 to §7.8 (the watermark scheme `wai-mark:tent64`) | `wai_synthesis_mark_vectors gen` | `synthesis_mark_verify.py` |
| [`upscale/`](upscale/) | [exact-upscale](../extensions/exact-upscale.md) (`wai.media.upscale`) | `wai_upscale_vectors gen` | `upscale_verify.py` |
| [`prior-carriage/`](prior-carriage/) | [prior-carriage](../extensions/prior-carriage.md) §15 and [http-delivery](../extensions/http-delivery.md) §8.7 (where a parameter set is fetched) | `wai_prior_carriage_vectors gen` | `prior_carriage_verify.py` |
| [`dred/`](dred/) | [opus-dred](../extensions/opus-dred.md) (`wai.audio.opus.dred.int`) | `wai_dred_vectors gen` (with the draft's weights file, `dred/fetch.sh`) | `dred_verify.py` |
| [`aggregate/`](aggregate/) | [telemetry-aggregation](../extensions/telemetry-aggregation.md) (`wai.telemetry.aggregate`) | `wai_telemetry_aggregate_vectors gen` | `telemetry_aggregate_verify.py` |
| [`tonemap/`](tonemap/) | [exact-tonemap](../extensions/exact-tonemap.md) (`wai.media.tonemap`, `wai.meta.gainmap`, `wai.meta.st2094_50`) | `wai_tonemap_vectors gen` | `tonemap_verify.py` |

100
wai/live-conformance/_fips197_aes.py Normal file
View file

@ -0,0 +1,100 @@
"""AES-128 encryption of one block, written from FIPS 197, and counter mode from NIST SP 800-38A
§6.5, in pure Python, for the prior-carriage verifier's sealed sets (prior-carriage §18.2).
It shares no code with wai-rs, which uses a Rust AES implementation. Only the forward cipher is
needed: counter mode encrypts and decrypts with it alike.
"""
def _xtime(a):
a <<= 1
return (a ^ 0x11B) & 0xFF if a & 0x100 else a
def _mul(a, b):
r = 0
while b:
if b & 1:
r ^= a
a = _xtime(a)
b >>= 1
return r
def _sbox():
"""FIPS 197 §5.1.1: the multiplicative inverse in GF(2^8), then the affine transformation."""
inv = [0] * 256
for a in range(1, 256):
for b in range(1, 256):
if _mul(a, b) == 1:
inv[a] = b
break
box = []
for a in range(256):
b = inv[a]
s = b
for k in range(1, 5):
s ^= ((b << k) | (b >> (8 - k))) & 0xFF
box.append(s ^ 0x63)
return box
SBOX = _sbox()
def expand_key(key):
"""FIPS 197 §5.2 for a 16-byte key: 11 round keys of 16 bytes."""
assert len(key) == 16
w = [list(key[4 * i:4 * i + 4]) for i in range(4)]
rcon = 1
for i in range(4, 44):
t = list(w[i - 1])
if i % 4 == 0:
t = [SBOX[x] for x in t[1:] + t[:1]]
t[0] ^= rcon
rcon = _xtime(rcon)
w.append([a ^ b for a, b in zip(w[i - 4], t)])
return [sum(w[4 * r:4 * r + 4], []) for r in range(11)]
def encrypt_block(round_keys, block):
"""FIPS 197 §5.1: the state in column order, ten rounds."""
s = [b ^ k for b, k in zip(block, round_keys[0])]
for r in range(1, 11):
s = [SBOX[x] for x in s]
# ShiftRows: byte (row, column) at index 4 * column + row moves left by its row.
s = [s[4 * ((c + row) % 4) + row] for c in range(4) for row in range(4)]
if r != 10:
out = []
for c in range(4):
a = s[4 * c:4 * c + 4]
out += [_mul(a[0], 2) ^ _mul(a[1], 3) ^ a[2] ^ a[3],
a[0] ^ _mul(a[1], 2) ^ _mul(a[2], 3) ^ a[3],
a[0] ^ a[1] ^ _mul(a[2], 2) ^ _mul(a[3], 3),
_mul(a[0], 3) ^ a[1] ^ a[2] ^ _mul(a[3], 2)]
s = out
s = [b ^ k for b, k in zip(s, round_keys[r])]
return bytes(s)
def ctr(key, iv, data):
"""Counter mode: the counter block from `iv`, incremented as one 128-bit big-endian integer."""
rk = expand_key(key)
counter = int.from_bytes(iv, "big")
out = bytearray()
for i in range(0, len(data), 16):
ks = encrypt_block(rk, counter.to_bytes(16, "big"))
out += bytes(a ^ b for a, b in zip(data[i:i + 16], ks))
counter = (counter + 1) % (1 << 128)
return bytes(out)
def self_test():
"""FIPS 197 Appendix C.1 and NIST SP 800-38A F.5.1's first two blocks."""
rk = expand_key(bytes.fromhex("000102030405060708090a0b0c0d0e0f"))
if encrypt_block(rk, bytes.fromhex("00112233445566778899aabbccddeeff")).hex() != "69c4e0d86a7b0430d8cdb78070b4c55a":
return False
key = bytes.fromhex("2b7e151628aed2a6abf7158809cf4f3c")
iv = bytes.fromhex("f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff")
plain = bytes.fromhex("6bc1bee22e409f96e93d7e117393172aae2d8a571e03ac9c9eb76fac45af8e51")
return ctr(key, iv, plain).hex() == "874d6191b620e3261bef6864990db6ce9806f66b7970fdff8617187bb9fffdff"

526
wai/live-conformance/_rfc8878_zstd.py Normal file
View file

@ -0,0 +1,526 @@
"""A Zstandard decoder written from RFC 8878, in pure Python, for the prior-carriage verifier.
It decodes one frame against a raw-content dictionary (RFC 8878 §5: the dictionary's bytes are
history before the frame's first byte, and the repeat offsets start at 1, 4 and 8), checks the
frame's content size and its XXH64 checksum when they are stated, and refuses anything it does
not read with `ZstdError`. It shares no code with wai-rs, which decodes with the Zstandard
library. It is written for the corpus's small frames, not for speed.
"""
class ZstdError(Exception):
pass
MAGIC = 0xFD2FB528
MAX_BLOCK = 128 * 1024
def frame_window(frame):
"""(Window_Size, header length, content size or None, checksum flag) of a frame header."""
if len(frame) < 6 or int.from_bytes(frame[:4], "little") != MAGIC:
raise ZstdError("no frame magic")
fhd = frame[4]
if fhd & 0x08:
raise ZstdError("reserved bit set")
single = bool(fhd & 0x20)
did = (0, 1, 2, 4)[fhd & 3]
fcs = {0: 1 if single else 0, 1: 2, 2: 4, 3: 8}[fhd >> 6]
at = 5
window = None
if not single:
wd = frame[at]
at += 1
base = 1 << (10 + (wd >> 3))
window = base + (base // 8) * (wd & 7)
at += did
if len(frame) < at + fcs:
raise ZstdError("header truncated")
size = int.from_bytes(frame[at:at + fcs], "little") if fcs else None
if fcs == 2:
size += 256
at += fcs
if window is None:
window = size
return window, at, size, bool(fhd & 0x04)
# --- bit readers --------------------------------------------------------------------------------
class Forward:
"""Bits read from the least significant bit of each byte, first byte first (§4.1.1)."""
def __init__(self, data):
self.v = int.from_bytes(data, "little")
self.n = len(data) * 8
self.pos = 0
def read(self, k):
if self.pos + k > self.n:
raise ZstdError("table description past its end")
x = (self.v >> self.pos) & ((1 << k) - 1)
self.pos += k
return x
def peek(self, k):
return (self.v >> self.pos) & ((1 << k) - 1)
class Backward:
"""A bitstream read from its end (§4.1): the highest set bit of the last byte ends it."""
def __init__(self, data):
if not data or data[-1] == 0:
raise ZstdError("bitstream without its end mark")
self.v = int.from_bytes(data, "little")
self.left = self.v.bit_length() - 1
def read(self, k):
if k == 0:
return 0
self.left -= k
if self.left >= 0:
return (self.v >> self.left) & ((1 << k) - 1)
return (self.v << -self.left) & ((1 << k) - 1)
def peek(self, k):
if self.left >= k:
return (self.v >> (self.left - k)) & ((1 << k) - 1)
return (self.v << (k - self.left)) & ((1 << k) - 1)
def overflowed(self):
return self.left < 0
# --- FSE (§4.1) ---------------------------------------------------------------------------------
def read_distribution(br, max_symbol, max_log):
"""An FSE table description: (accuracy log, normalised counts)."""
log = br.read(4) + 5
if log > max_log:
raise ZstdError("accuracy log too large")
remaining = (1 << log) + 1
threshold = 1 << log
bits = log + 1
counts = []
previous_zero = False
while remaining > 1 and len(counts) <= max_symbol:
if previous_zero:
while True:
flag = br.read(2)
counts.extend([0] * flag)
if flag != 3:
break
if len(counts) > max_symbol:
break
top = (2 * threshold - 1) - remaining
low = br.peek(bits - 1)
if low < top:
value = low
br.read(bits - 1)
else:
value = br.peek(bits)
if value >= threshold:
value -= top
br.read(bits)
count = value - 1
remaining -= -count if count < 0 else count
counts.append(count)
previous_zero = count == 0
while remaining < threshold:
bits -= 1
threshold >>= 1
if remaining != 1 or len(counts) > max_symbol + 1:
raise ZstdError("distribution does not sum")
return log, counts
def build_table(log, counts):
"""The decoding table: per state, (symbol, bits, baseline)."""
size = 1 << log
symbol = [0] * size
high = size - 1
for s, c in enumerate(counts):
if c == -1:
symbol[high] = s
high -= 1
step = (size >> 1) + (size >> 3) + 3
pos = 0
for s, c in enumerate(counts):
for _ in range(max(c, 0)):
symbol[pos] = s
pos = (pos + step) & (size - 1)
while pos > high:
pos = (pos + step) & (size - 1)
if pos != 0:
raise ZstdError("distribution does not spread")
nxt = [1 if c == -1 else c for c in counts]
table = []
for u in range(size):
s = symbol[u]
x = nxt[s]
nxt[s] += 1
nb = log - (x.bit_length() - 1)
table.append((s, nb, (x << nb) - size))
return table
# --- Huffman literals (§4.2) --------------------------------------------------------------------
def huffman_weights(data):
"""(table, max bits, bytes read) from a Huffman tree description."""
if not data:
raise ZstdError("no tree description")
head = data[0]
if head < 128:
stream = data[1:1 + head]
if len(stream) != head:
raise ZstdError("tree description truncated")
fb = Forward(stream)
log, counts = read_distribution(fb, 255, 6)
table = build_table(log, counts)
used = (fb.pos + 7) // 8
bb = Backward(stream[used:])
s1, s2 = bb.read(log), bb.read(log)
weights = []
# Two states, alternating, until a state update reads past the stream's start; the
# other state then gives the last weight (§4.2.1.2).
while True:
sym, nb, base = table[s1]
weights.append(sym)
s1 = base + bb.read(nb)
if bb.overflowed():
weights.append(table[s2][0])
break
sym, nb, base = table[s2]
weights.append(sym)
s2 = base + bb.read(nb)
if bb.overflowed():
weights.append(table[s1][0])
break
if len(weights) > 255:
raise ZstdError("too many weights")
used = 1 + head
else:
n = head - 127
used = 1 + (n + 1) // 2
if len(data) < used:
raise ZstdError("tree description truncated")
weights = []
for i in range(n):
byte = data[1 + i // 2]
weights.append(byte >> 4 if i % 2 == 0 else byte & 15)
if any(w > 11 for w in weights):
raise ZstdError("weight too large")
total = sum(1 << (w - 1) for w in weights if w)
if total == 0:
raise ZstdError("no weights")
maxbits = total.bit_length()
rest = (1 << maxbits) - total
if rest & (rest - 1):
raise ZstdError("weights do not complete a tree")
weights.append(rest.bit_length())
if maxbits > 11:
raise ZstdError("code too long")
table = [None] * (1 << maxbits)
pos = 0
for w in range(1, 12):
for s, ws in enumerate(weights):
if ws == w:
nb = maxbits + 1 - w
for _ in range(1 << (w - 1)):
table[pos] = (s, nb)
pos += 1
if pos != len(table):
raise ZstdError("weights do not fill the table")
return (table, maxbits), used
def huffman_stream(data, tree, count):
table, maxbits = tree
br = Backward(data)
out = bytearray()
for _ in range(count):
s, nb = table[br.peek(maxbits)]
br.read(nb)
out.append(s)
if br.left != 0:
raise ZstdError("Huffman stream not consumed exactly")
return bytes(out)
def literals(block, state):
"""(literal bytes, bytes read) of a Literals_Section (§3.1.1.3.1)."""
b0 = block[0]
kind, fmt = b0 & 3, (b0 >> 2) & 3
if kind in (0, 1):
if fmt in (0, 2):
hl, size = 1, b0 >> 3
elif fmt == 1:
hl, size = 2, int.from_bytes(block[:2], "little") >> 4
else:
hl, size = 3, int.from_bytes(block[:3], "little") >> 4
if kind == 0:
if len(block) < hl + size:
raise ZstdError("raw literals truncated")
return block[hl:hl + size], hl + size
if len(block) < hl + 1:
raise ZstdError("RLE literals truncated")
return bytes([block[hl]]) * size, hl + 1
lhc = int.from_bytes(block[:5].ljust(5, b"\0"), "little")
if fmt in (0, 1):
hl, size, csize, streams = 3, (lhc >> 4) & 0x3FF, (lhc >> 14) & 0x3FF, 1 if fmt == 0 else 4
elif fmt == 2:
hl, size, csize, streams = 4, (lhc >> 4) & 0x3FFF, (lhc >> 18) & 0x3FFF, 4
else:
hl, size, csize, streams = 5, (lhc >> 4) & 0x3FFFF, (lhc >> 22) & 0x3FFFF, 4
body = block[hl:hl + csize]
if len(body) != csize:
raise ZstdError("compressed literals truncated")
at = 0
if kind == 2:
state["huffman"], at = huffman_weights(body)
elif state.get("huffman") is None:
raise ZstdError("treeless literals with no previous tree")
tree = state["huffman"]
if streams == 1:
out = huffman_stream(body[at:], tree, size)
else:
if len(body) < at + 6:
raise ZstdError("jump table truncated")
s1, s2, s3 = (int.from_bytes(body[at + 2 * i:at + 2 * i + 2], "little") for i in range(3))
at += 6
seg = (size + 3) // 4
parts = []
for i, n in enumerate((s1, s2, s3, len(body) - at - s1 - s2 - s3)):
if n < 0:
raise ZstdError("jump table past the literals")
parts.append(huffman_stream(body[at:at + n], tree, seg if i < 3 else size - 3 * seg))
at += n
out = b"".join(parts)
return out, hl + csize
# --- sequences (§3.1.1.3.2) ---------------------------------------------------------------------
LL_BASE = list(range(16)) + [16, 18, 20, 22, 24, 28, 32, 40, 48, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384,
32768, 65536]
LL_BITS = [0] * 16 + [1, 1, 1, 1, 2, 2, 3, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
ML_BASE = [n + 3 for n in range(32)] + [35, 37, 39, 41, 43, 47, 51, 59, 67, 83, 99, 131, 259, 515, 1027, 2051,
4099, 8195, 16387, 32771, 65539]
ML_BITS = [0] * 32 + [1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
LL_DEFAULT = (6, [4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1,
-1, -1, -1, -1])
ML_DEFAULT = (6, [1, 4, 3, 2, 2, 2, 2, 2, 2] + [1] * 37 + [-1] * 7)
OF_DEFAULT = (5, [1, 1, 1, 1, 1, 1, 2, 2, 2] + [1] * 15 + [-1] * 5)
KINDS = (("ll", 35, 9, LL_DEFAULT), ("of", 31, 8, OF_DEFAULT), ("ml", 52, 9, ML_DEFAULT))
def sequences(block, state):
"""The sequences of a Sequences_Section: [(literals length, offset value, match length)]."""
if not block:
raise ZstdError("sequences section missing")
b0 = block[0]
if b0 == 0:
if len(block) != 1:
raise ZstdError("bytes after an empty sequences section")
return []
if b0 < 128:
n, at = b0, 1
elif b0 < 255:
n, at = ((b0 - 128) << 8) + block[1], 2
else:
n, at = block[1] + (block[2] << 8) + 0x7F00, 3
modes = block[at]
at += 1
if modes & 3:
raise ZstdError("reserved bits in the modes")
tables = {}
for (name, maxsym, maxlog, default), shift in zip(KINDS, (6, 4, 2)):
mode = (modes >> shift) & 3
if mode == 0:
tables[name] = build_table(*default)
elif mode == 1:
if block[at] > maxsym:
raise ZstdError("RLE symbol out of range")
tables[name] = [(block[at], 0, 0)]
at += 1
elif mode == 2:
fb = Forward(block[at:])
log, counts = read_distribution(fb, maxsym, maxlog)
tables[name] = build_table(log, counts)
at += (fb.pos + 7) // 8
else:
if state.get(name) is None:
raise ZstdError("repeat mode with no previous table")
tables[name] = state[name]
state[name] = tables[name]
br = Backward(block[at:])
def log_of(t):
return len(t).bit_length() - 1
ll, of, ml = (br.read(log_of(tables[k])) for k in ("ll", "of", "ml"))
out = []
for i in range(n):
ofc = tables["of"][of][0]
mlc = tables["ml"][ml][0]
llc = tables["ll"][ll][0]
if ofc > 31 or mlc > 52 or llc > 35:
raise ZstdError("code out of range")
offset = (1 << ofc) + br.read(ofc)
match = ML_BASE[mlc] + br.read(ML_BITS[mlc])
lit = LL_BASE[llc] + br.read(LL_BITS[llc])
out.append((lit, offset, match))
if i + 1 < n:
for k in ("ll", "ml", "of"):
s, nb, base = tables[k][{"ll": ll, "ml": ml, "of": of}[k]]
v = base + br.read(nb)
if k == "ll":
ll = v
elif k == "ml":
ml = v
else:
of = v
if br.left != 0:
raise ZstdError("sequence bitstream not consumed exactly")
return out
def execute(seqs, lits, history, reps):
"""Appends the block's output to `history` (dictionary and frame so far)."""
at = 0
for lit, value, match in seqs:
if at + lit > len(lits):
raise ZstdError("literals overrun")
history += lits[at:at + lit]
at += lit
if value > 3:
offset = value - 3
reps[:] = [offset, reps[0], reps[1]]
else:
idx = value + (1 if lit == 0 else 0)
if idx == 1:
offset = reps[0]
elif idx == 2:
offset = reps[1]
reps[:] = [offset, reps[0], reps[2]]
elif idx == 3:
offset = reps[2]
reps[:] = [offset, reps[0], reps[1]]
else:
offset = reps[0] - 1
reps[:] = [offset, reps[0], reps[1]]
if offset <= 0 or offset > len(history):
raise ZstdError("offset past the history")
start = len(history) - offset
for j in range(match):
history.append(history[start + j])
history += lits[at:]
# --- XXH64 --------------------------------------------------------------------------------------
M64 = (1 << 64) - 1
P1, P2, P3, P4, P5 = (0x9E3779B185EBCA87, 0xC2B2AE3D27D4EB4F, 0x165667B19E3779F9, 0x85EBCA77C2B2AE63,
0x27D4EB2F165667C5)
def _rotl(x, r):
return ((x << r) | (x >> (64 - r))) & M64
def _round(acc, lane):
return (_rotl((acc + lane * P2) & M64, 31) * P1) & M64
def xxh64(data, seed=0):
n = len(data)
i = 0
if n >= 32:
v = [(seed + P1 + P2) & M64, (seed + P2) & M64, seed, (seed - P1) & M64]
while i + 32 <= n:
for k in range(4):
v[k] = _round(v[k], int.from_bytes(data[i + 8 * k:i + 8 * k + 8], "little"))
i += 32
h = (_rotl(v[0], 1) + _rotl(v[1], 7) + _rotl(v[2], 12) + _rotl(v[3], 18)) & M64
for k in range(4):
h = ((h ^ _round(0, v[k])) * P1 + P4) & M64
else:
h = (seed + P5) & M64
h = (h + n) & M64
while i + 8 <= n:
h ^= _round(0, int.from_bytes(data[i:i + 8], "little"))
h = (_rotl(h, 27) * P1 + P4) & M64
i += 8
if i + 4 <= n:
h ^= (int.from_bytes(data[i:i + 4], "little") * P1) & M64
h = (_rotl(h, 23) * P2 + P3) & M64
i += 4
while i < n:
h ^= (data[i] * P5) & M64
h = (_rotl(h, 11) * P1) & M64
i += 1
h ^= h >> 33
h = (h * P2) & M64
h ^= h >> 29
h = (h * P3) & M64
h ^= h >> 32
return h
# --- a frame ------------------------------------------------------------------------------------
def decode(frame, dictionary=b"", limit=None):
"""The content of one whole frame, decoded against `dictionary` as raw content; nothing may
follow the frame. With `limit`, a frame whose content would pass it raises `TooLarge`."""
window, at, size, checksum = frame_window(frame)
history = bytearray(dictionary)
base = len(history)
reps = [1, 4, 8]
state = {}
while True:
if len(frame) < at + 3:
raise ZstdError("block header truncated")
h = int.from_bytes(frame[at:at + 3], "little")
at += 3
last, kind, bsize = h & 1, (h >> 1) & 3, h >> 3
if bsize > min(window, MAX_BLOCK) and kind != 1:
raise ZstdError("block too large")
if kind == 0:
if len(frame) < at + bsize:
raise ZstdError("raw block truncated")
history += frame[at:at + bsize]
at += bsize
elif kind == 1:
if len(frame) < at + 1:
raise ZstdError("RLE block truncated")
history += bytes([frame[at]]) * bsize
at += 1
elif kind == 2:
block = frame[at:at + bsize]
if len(block) != bsize:
raise ZstdError("compressed block truncated")
lits, used = literals(block, state)
execute(sequences(block[used:], state), lits, history, reps)
at += bsize
else:
raise ZstdError("reserved block type")
if limit is not None and len(history) - base > limit:
raise TooLarge()
if last:
break
content = bytes(history[base:])
if size is not None and len(content) != size:
raise ZstdError("content size not as stated")
if checksum:
if frame[at:at + 4] != (xxh64(content) & 0xFFFFFFFF).to_bytes(4, "little"):
raise ZstdError("checksum mismatch")
at += 4
if at != len(frame):
raise ZstdError("bytes after the frame")
return content
class TooLarge(Exception):
pass

92
wai/live-conformance/prior-carriage/README.md Normal file
View file

@ -0,0 +1,92 @@
# live-conformance/prior-carriage — where a parameter set is fetched, how it is delivered, when it is worth holding, who may read it, and how it verifies by range
The corpus of [`extensions/prior-carriage.md`](../../extensions/prior-carriage.md) §15 to §19
and [`extensions/http-delivery.md`](../../extensions/http-delivery.md) §8.7, §8.8 and §12.5:
`vectors.json` and the files under `files/`.
**Provenance.** The parameter sets are files other corpora commit, named by their paths from
this directory and pinned by their SHA-256: the trained `WSR1` priors `generic` and `title` in
the safetensors form of [`../prior-forms/`](../prior-forms/) (`title` was fine-tuned from
`generic`, `seeded` is a seeded prior of the same shape), and the listing `demo` of
[`../../pin-conformance/`](../../pin-conformance/) with the three stand-in files it names. The
hosts are `.example` names (RFC 2606); nothing here was fetched from a server.
`vectors.json` holds:
- `sets`: each set the cases read, with its path, SHA-256 and length.
- `roots`: strings, and whether each is a set root (§8.7: an absolute `https` URI with a host,
no userinfo, query or fragment, whose path ends in `/`, at most 2048 bytes).
- `hints`: a `wai_priors` entry (moq-streaming-format §4.2) or a WAI multivariant playlist, and
the hint read from it: the roots kept, in order, and the positions passed over as
`set_location_malformed` (in a playlist, among the elements of the hint line's `VALUE` split
at every space). Some playlists also carry lines a reader ignores: a second line of the
hint's `DATA-ID`, one with `LANGUAGE`, and one whose attribute list does not read.
- `fetches`: a pin, the `wai_priors` entry that states its hint, the roots of the sink's own
configuration, the origin, the sink's bound on one response's bytes, and the responses the
servers give, keyed by URL (a URL with no response is a server that does not answer); and
the outcome: the roots in the order the sink asks them, each URL asked with what came of it
(`held` or a reason of §8.7 or §8.8), the result, `held` or the reason SPEC §3.1 refuses the pin
with (or `set_usage_malformed`, prior-carriage §18.1), and `usage`, the usage statement a held
listing binds as its file `wai-usage.json` (its licence, terms, the two booleans and the digest
of its ai-usage statement), `null` where it binds none. A body is a set's bytes (`set`),
those bytes with one byte's lowest bit flipped (`flip`), a file of `files/` (`file`, with
`flip` or `truncate`, the first bytes kept), or bytes given in hexadecimal (`hex`). A response is keyed by its URL and the
`Available-Dictionary` value of the request it answers (`null` for a request that names no
dictionary); a fetch's `dictionary` is the set the sink holds and names (§8.8).
- `fields`: `Available-Dictionary` values and the digest each names (`null` where it does not
read as one), and the `Use-As-Dictionary` value for a root's path.
- `stock`: the check a stock client makes of §8.8: `files/title-from-generic.dcz` after its
40-byte header, decoded by the Zstandard tool against `generic`, has `title`'s SHA-256:
tail -c +41 files/title-from-generic.dcz | zstd -d -D ../prior-forms/files/generic.safetensors | sha256sum
- `plans`: §17's worked plans, each a plan (a span, a cap, the generations in epoch order with
their sets' bytes and any coding against an earlier one, the titles with their pinned and
classical bytes, and one or more sinks with what each holds and its fetches per title) and
the outcome worked by hand: per sink, each generation's saving, cost (`held`, `delta` or
`set`) and net, the generations kept, fetched and evicted, each title's rendition, and the
bytes received beside the bytes with no set; or `plan_malformed`. The sizes are worked
numbers, not measurements.
- `usage`: §18.1's usage statements as JSON text, and what reading each gives (its licence,
terms, the two booleans and the digest of its ai-usage statement) or `set_usage_malformed`.
- `sealed`: §18.2's sealed sets (`files/title.wsk1`, `files/generic.wsk1`, or a header written
in hexadecimal), the keys the sink holds by their identifiers, and what opening gives (the
inner form, the plaintext's digest and the usage statement) or the reason. The corpus's key
(16 bytes of `0x6b`, identifier 16 bytes of `0x4b`, counter block 16 bytes of `0x69`) is
published here and protects nothing.
- `held_sets`: http-delivery §12.5's `held_sets` from a store of sets held per site, the site
it is sent to, the pins the site's content carries, and whether it is turned on; and
`dictionary_naming`, whether a sink may name a held set in `Available-Dictionary` for a site.
- `tree_heads`: §19's heads (`files/generic.wtr1` in pieces of one chunk, `files/generic-g1.wtr1`
in pieces of two, `files/title-sealed.wtr1` over `files/title.wsk1`, or a head written in
hexadecimal) and what reading each gives (the inner form, the set's length, the piece exponent,
the number of pieces, the set's SHA-256 and its BLAKE3 hash) or the reason.
- `tree_fetches`: a head, sources (each named, with the bytes it serves, whether it answers, and
any content coding), and the outcome of fetching every piece by range: each piece asked of
which source with what came of it, piece `i` first from source `i mod m` and then from the
sources after it; then `held` and the set's SHA-256, or the reason.
`files/title-from-generic.dcz` is `title` coded against `generic` by the Zstandard library at
level 19, its content size and checksum stated, under the `dcz` header naming `generic`.
`files/title-window-16m.dcz` holds `title` in one raw block of a frame that states a 16 MB
window and no content size, under the same header: a frame RFC 9842 has no client decode
against a dictionary of `generic`'s size.
A verifier conforms when it reaches every case's outcome, field for field.
**Verifiers.**
- `wai_prior_carriage_vectors verify live-conformance/prior-carriage` (wai-rs, feature
`prior_carriage`): every file regenerates byte for byte, no other file is here, and the
module reaches every outcome. `wai_prior_carriage_vectors gen` writes the corpus.
- `python3 -I live-conformance/prior_carriage_verify.py live-conformance/prior-carriage`,
written from the texts and sharing no code with wai-rs. It decodes Zstandard with
`_rfc8878_zstd.py`, written from RFC 8878 beside it, hashes with the shared
`../../claim-conformance/_blake3.py`, written from the BLAKE3 specification, opens sealed sets with
`_fips197_aes.py`, written from FIPS 197 and NIST SP 800-38A, reads ai-usage statements with
the ai-usage corpus's second implementation (`ai_usage_verify.py`), and, where the Zstandard tool is on the
PATH, runs the `stock` check with that tool too. It also mutates copies of the
corpus, one defect each, and requires every copy to fail, and drops each of its own checks
in turn and requires the corpus to catch the weakened verifier.

BIN
wai/live-conformance/prior-carriage/files/generic-g1.wtr1 Normal file
View file

Binary file not shown.

BIN
wai/live-conformance/prior-carriage/files/generic.wsk1 Normal file
View file

Binary file not shown.

BIN
wai/live-conformance/prior-carriage/files/generic.wtr1 Normal file
View file

Binary file not shown.

BIN
wai/live-conformance/prior-carriage/files/title-from-generic.dcz Normal file
View file

Binary file not shown.

BIN
wai/live-conformance/prior-carriage/files/title-sealed.wtr1 Normal file
View file

Binary file not shown.

BIN
wai/live-conformance/prior-carriage/files/title-window-16m.dcz Normal file
View file

Binary file not shown.

BIN
wai/live-conformance/prior-carriage/files/title.wsk1 Normal file
View file

Binary file not shown.

5324
wai/live-conformance/prior-carriage/vectors.json Normal file
View file

File diff suppressed because it is too large Load diff

1054
wai/live-conformance/prior_carriage_verify.py Normal file
View file

File diff suppressed because it is too large Load diff

70
wai/registry/capabilities.json
View file

@ -1740,6 +1740,20 @@
"edition": null,
"registered": "2026-10-22"
},
{
"capability": "wai.prior.sealed",
"kind": "payload",
"status": "registered",
"media": "binary",
"role": "standalone",
"tier": "lossless",
"licence": "royalty-free",
"replicate": "standard-defined",
"sets": [],
"spec_section": "§5 Prior artifacts, general tensors and derivation operations",
"edition": null,
"registered": "2026-11-04"
},
{
"capability": "wai.prior.set",
"kind": "payload",
@ -1754,6 +1768,20 @@
"edition": null,
"registered": "2026-09-30"
},
{
"capability": "wai.prior.tree",
"kind": "payload",
"status": "registered",
"media": "binary",
"role": "standalone",
"tier": "lossless",
"licence": "royalty-free",
"replicate": "standard-defined",
"sets": [],
"spec_section": "§5 Prior artifacts, general tensors and derivation operations",
"edition": null,
"registered": "2026-11-04"
},
{
"capability": "wai.quantum.calibration",
"kind": "record",
@ -2632,6 +2660,13 @@
"section": "§7.7",
"status": "proposed"
},
{
"name": "/.well-known/wai-sets/",
"kind": "well-known",
"document": "extensions/http-delivery.md",
"section": "§8.7",
"status": "defined"
},
{
"name": "EXT_gaussian_splatting_compression_wai",
"kind": "gltf-extension",
@ -3262,6 +3297,13 @@
"section": "§6",
"status": "defined"
},
{
"name": "WSK1",
"kind": "magic",
"document": "extensions/prior-carriage.md",
"section": "§18.2",
"status": "defined"
},
{
"name": "WSL1",
"kind": "magic",
@ -3297,6 +3339,13 @@
"section": "§7.1",
"status": "defined"
},
{
"name": "WTR1",
"kind": "magic",
"document": "extensions/prior-carriage.md",
"section": "§19",
"status": "defined"
},
{
"name": "WTX1",
"kind": "magic",
@ -3598,6 +3647,13 @@
"section": "§11.1",
"status": "defined"
},
{
"name": "science.transaction.wai.sets",
"kind": "data-id",
"document": "extensions/http-delivery.md",
"section": "§8.7",
"status": "defined"
},
{
"name": "science.transaction.wai.stages",
"kind": "data-id",
@ -4095,6 +4151,13 @@
"section": "§3.1",
"status": "defined"
},
{
"name": "wai-set-usage",
"kind": "document-kind",
"document": "extensions/prior-carriage.md",
"section": "§18.1",
"status": "defined"
},
{
"name": "wai-signing",
"kind": "document-kind",
@ -4221,6 +4284,13 @@
"section": "§4.2",
"status": "defined"
},
{
"name": "wai-usage.json",
"kind": "path",
"document": "extensions/prior-carriage.md",
"section": "§18.1",
"status": "defined"
},
{
"name": "wai1",
"kind": "codecs-prefix",

8
wai/vocab/v2.shapes.ttl
View file

@ -78,7 +78,9 @@ wai:ManifestShape a sh:NodeShape ;
wai:wai.prior.gguf
wai:wai.prior.nnc
wai:wai.prior.safetensors
wai:wai.prior.sealed
wai:wai.prior.set
wai:wai.prior.tree
wai:wai.quantum.circuit
wai:wai.quantum.circuit2
wai:wai.semantic.clip
@ -167,7 +169,9 @@ wai:ManifestShape a sh:NodeShape ;
wai:wai.prior.gguf
wai:wai.prior.nnc
wai:wai.prior.safetensors
wai:wai.prior.sealed
wai:wai.prior.set
wai:wai.prior.tree
wai:wai.quantum.circuit
wai:wai.quantum.circuit2
wai:wai.semantic.clip
@ -275,7 +279,9 @@ wai:ComponentShape a sh:NodeShape ;
wai:wai.prior.gguf
wai:wai.prior.nnc
wai:wai.prior.safetensors
wai:wai.prior.sealed
wai:wai.prior.set
wai:wai.prior.tree
wai:wai.quantum.circuit
wai:wai.quantum.circuit2
wai:wai.semantic.clip
@ -364,7 +370,9 @@ wai:ComponentShape a sh:NodeShape ;
wai:wai.prior.gguf
wai:wai.prior.nnc
wai:wai.prior.safetensors
wai:wai.prior.sealed
wai:wai.prior.set
wai:wai.prior.tree
wai:wai.quantum.circuit
wai:wai.quantum.circuit2
wai:wai.semantic.clip

24
wai/vocab/v2.ttl
View file

@ -1262,6 +1262,18 @@ wai:wai.prior.safetensors a wai:Capability ;
wai:specSection "§5 Prior artifacts, general tensors and derivation operations" ;
wai:registeredOn "2026-10-22"^^xsd:date .
wai:wai.prior.sealed a wai:Capability ;
rdfs:label "wai.prior.sealed" ;
wai:kind wai:kind-payload ;
wai:status wai:status-registered ;
wai:media wai:media-binary ;
wai:role wai:role-standalone ;
wai:tier wai:tier-lossless ;
wai:licence wai:licence-royalty-free ;
wai:replicateClass wai:replicate-standard-defined ;
wai:specSection "§5 Prior artifacts, general tensors and derivation operations" ;
wai:registeredOn "2026-11-04"^^xsd:date .
wai:wai.prior.set a wai:Capability ;
rdfs:label "wai.prior.set" ;
wai:kind wai:kind-payload ;
@ -1274,6 +1286,18 @@ wai:wai.prior.set a wai:Capability ;
wai:specSection "§5 Prior artifacts, general tensors and derivation operations" ;
wai:registeredOn "2026-09-30"^^xsd:date .
wai:wai.prior.tree a wai:Capability ;
rdfs:label "wai.prior.tree" ;
wai:kind wai:kind-payload ;
wai:status wai:status-registered ;
wai:media wai:media-binary ;
wai:role wai:role-standalone ;
wai:tier wai:tier-lossless ;
wai:licence wai:licence-royalty-free ;
wai:replicateClass wai:replicate-standard-defined ;
wai:specSection "§5 Prior artifacts, general tensors and derivation operations" ;
wai:registeredOn "2026-11-04"^^xsd:date .
wai:wai.quantum.calibration a wai:Capability ;
rdfs:label "wai.quantum.calibration" ;
wai:kind wai:kind-record ;

21
wai/wai-rs/Cargo.toml
View file

@ -653,6 +653,17 @@ exact_tonemap = ["receipt_claims", "ed25519", "png_still", "video_int"]
# prior) and `whm1/1` (the WHM1 model), through which a prior in any of the three loads, bit for
# bit, into the native artifact an integer kernel reads. SHA-256 only; no new dependency.
prior_wire = []
# Prior carriage (extensions/prior-carriage.md §15, §16; extensions/http-delivery.md §8.7, §8.8):
# where a parameter set is fetched — set roots, the location hints of a MoQ catalog entry and of
# a WAI multivariant playlist — and a fetch over responses the caller supplies, holding bytes only
# by their digest; a set as an RFC 9842 compression dictionary, a successor decoded from `dcz`
# (Zstandard with the held set as a raw dictionary) through the zstd binding the neural decoders
# already use; when a set is worth holding, charged per sink; a set's usage statement (with an
# ai-usage statement inside it), a set sealed under AES-128 in counter mode through the AES the
# hpke feature already links, and the held_sets a sink discloses; and the chunk-tree form, a set's
# pieces verified by range against BLAKE3 chaining values through the hazmat API the HTTP head
# proof uses. No I/O; no new dependency.
prior_carriage = ["dep:zstd-safe", "dep:aes-gcm", "dep:blake3", "ai_usage"]
# Scene carriage (extensions/scene-carriage.md): WAI worlds, audio scenes, mesh assets and timed
# media mapped onto ISO/IEC 23090-14 scene description (glTF 2.0 with the MPEG extensions) and
# onto a USD layer (USD Core Specification v1.0.1, text format), both ways: the export writes a
@ -1404,6 +1415,11 @@ name = "wai_prior_forms_vectors"
path = "src/bin/wai_prior_forms_vectors.rs"
required-features = ["prior_wire"]
[[bin]]
name = "wai_prior_carriage_vectors"
path = "src/bin/wai_prior_carriage_vectors.rs"
required-features = ["prior_carriage"]
[[bin]]
name = "wai_scene_vectors"
path = "src/bin/wai_scene_vectors.rs"
@ -2481,6 +2497,11 @@ name = "prior_wire"
path = "tests/prior_wire.rs"
required-features = ["prior_wire", "neural_int"]
[[test]]
name = "prior_carriage"
path = "tests/prior_carriage.rs"
required-features = ["prior_carriage"]
[[test]]
name = "scene_map"
path = "tests/scene_map.rs"

81
wai/wai-rs/src/bin/wai_prior_carriage_vectors.rs Normal file
View file

@ -0,0 +1,81 @@
//! Conformance corpus for prior carriage (`live-conformance/prior-carriage/`,
//! extensions/prior-carriage.md §15; extensions/http-delivery.md §8.7).
//!
//! wai_prior_carriage_vectors gen <corpus-dir> write the corpus, from the sets other
//! corpora commit and the cases written here
//! wai_prior_carriage_vectors verify <corpus-dir> every file regenerates byte for byte, no
//! other file but README.md is there, and
//! every case gets the outcome it states
//!
//! The generator and this verifier are one implementation. The corpus's second implementation
//! is `live-conformance/prior_carriage_verify.py`, written from the texts and sharing no code
//! with this binary.
use std::path::Path;
use wai::prior_carriage::corpus;
fn main() {
let a: Vec<String> = std::env::args().collect();
match (a.get(1).map(String::as_str), a.get(2)) {
(Some("gen"), Some(d)) => std::process::exit(write_corpus(Path::new(d))),
(Some("verify"), Some(d)) => std::process::exit(verify(Path::new(d))),
_ => {
eprintln!("usage: wai_prior_carriage_vectors gen|verify <corpus-dir>");
std::process::exit(2);
}
}
}
fn write_corpus(dir: &Path) -> i32 {
// The sets are read by their paths from this directory, so it exists first.
if let Err(e) = std::fs::create_dir_all(dir) {
eprintln!("create {}: {e}", dir.display());
return 1;
}
let files = match corpus::generate(dir) {
Ok(f) => f,
Err(e) => {
eprintln!("{e}");
return 1;
}
};
for (name, body) in files {
let p = dir.join(&name);
if let Some(parent) = p.parent() {
if let Err(e) = std::fs::create_dir_all(parent) {
eprintln!("create {}: {e}", parent.display());
return 1;
}
}
if let Err(e) = std::fs::write(&p, body) {
eprintln!("write {}: {e}", p.display());
return 1;
}
}
println!("wrote {}", dir.display());
0
}
fn verify(dir: &Path) -> i32 {
let files = match corpus::committed(dir) {
Ok(f) => f,
Err(e) => {
eprintln!("{e}");
return 1;
}
};
let same = match corpus::generate(dir) {
Ok(g) => g == files,
Err(e) => {
eprintln!("{e}");
false
}
};
println!("{} every file regenerates byte for byte, and no other file is there", if same { "PASS" } else { "FAIL" });
let fails = corpus::verify(dir, &files);
for f in &fails {
println!("FAIL {f}");
}
println!("{} every case gets the outcome the corpus states", if fails.is_empty() { "PASS" } else { "FAIL" });
(!same || !fails.is_empty()) as i32
}

2
wai/wai-rs/src/codecs/glossary_check.rs
View file

@ -130,6 +130,8 @@ const GLOSSARY_NO_ENTRY: &[(&str, &str)] = &[
(CAP_PRIOR_SET, WAI_NATIVE),
(CAP_PRIOR_SAFETENSORS, "a tensor-file layout for parameters, not a media codec"),
(CAP_PRIOR_GGUF, "a tensor-file layout for parameters, not a media codec"),
(CAP_PRIOR_SEALED, WAI_NATIVE),
(CAP_PRIOR_TREE, WAI_NATIVE),
(CAP_TENSOR_NNC, "a candidate with no payload format pinned; neural-network coding codes tensors, not media"),
(CAP_SPLAT_SOG, "a candidate with no payload format pinned"),
(CAP_AUDIO_OPUS_DRED, "the glossary has no entry for the Opus deep redundancy extension yet"),

33
wai/wai-rs/src/codecs/mod.rs
View file

@ -363,6 +363,19 @@ pub const CAP_PRIOR_SET: &str = "wai.prior.set";
// artifact an integer kernel reads. Read and written by `crate::prior_wire`.
pub const CAP_PRIOR_SAFETENSORS: &str = "wai.prior.safetensors";
pub const CAP_PRIOR_GGUF: &str = "wai.prior.gguf";
// wai.prior.sealed: a one-file set encrypted under a key its origin releases
// (extensions/prior-carriage.md §18.2), `WSK1`: the inner form, the key's identifier, the
// counter block, the plaintext's SHA-256, the set's usage statement, then the set under
// AES-128 in counter mode. The pin covers the sealed bytes, so any server may serve them;
// opened with its key, it hands the decoder the inner set's bytes unchanged, so the registry
// lists it as established exact. Read and sealed by `crate::prior_carriage::sealed`.
pub const CAP_PRIOR_SEALED: &str = "wai.prior.sealed";
// wai.prior.tree: a chunk-tree head over a one-file set (extensions/prior-carriage.md §19),
// `WTR1`: the inner form, the set's length, SHA-256 and BLAKE3 hash, and the BLAKE3 chaining
// value of each piece of 1024·2^g bytes. The pin covers the head, so each piece fetched by range
// from any peer verifies on arrival; the set it names is handed over unchanged, so the registry
// lists it as established exact. Read and written by `crate::prior_carriage::tree`.
pub const CAP_PRIOR_TREE: &str = "wai.prior.tree";
// wai.tensor.nnc: general tensorial data (split-inference feature maps, 3D/4D
// scene parameters) coded with NNC. A CANDIDATE (is_candidate; SPEC §5, Candidate
// capabilities): general-tensor coding is in an NNC amendment at committee-draft
@ -719,6 +732,10 @@ pub fn known_capabilities() -> &'static [&'static str] {
CAP_PRIOR_SET,
// a parameter set as one tensor file
CAP_PRIOR_SAFETENSORS, CAP_PRIOR_GGUF,
// a one-file set encrypted under a key its origin releases
CAP_PRIOR_SEALED,
// a chunk-tree head over a one-file set, for verified fetch by range
CAP_PRIOR_TREE,
// immersive audio (IAMF royalty-free; AC-4 / IVAS license-flagged)
CAP_AUDIO_IAMF, CAP_AUDIO_AC4, CAP_AUDIO_IVAS,
// cinema mastering (JPEG 2000 / DCP)
@ -944,8 +961,9 @@ pub fn capability_tier(cap: &str) -> Option<FidelityTier> {
// The projection descriptor reproduces exactly: it IS its 40 bytes.
| CAP_META_PROJECTION
// A parameter-set listing is its own output, byte for byte; a tensor file hands
// its reader the tensors' bytes as it holds them.
| CAP_PRIOR_SET | CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF => Lossless,
// its reader the tensors' bytes as it holds them; a sealed set, opened, hands over
// the inner set's, and a chunk tree the set it names.
| CAP_PRIOR_SET | CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF | CAP_PRIOR_SEALED | CAP_PRIOR_TREE => Lossless,
// Interactive worlds: replay-equivalence is EXACT state
// reproduction (identical BLAKE3 state hash at every checkpoint,
// no tolerance parameter) — the simulation analog of lossless.
@ -1129,6 +1147,9 @@ pub fn capability_license(cap: &str) -> Option<LicenseClass> {
// Tensor-file formats published with their reference implementations under
// permissive licences (the safetensors layout under Apache-2.0, GGUF under MIT).
| CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF
// WAI-native, over AES-128 (FIPS 197) in counter mode (NIST SP 800-38A); a chunk tree
// over BLAKE3, published with its reference implementation under CC0 and Apache-2.0.
| CAP_PRIOR_SEALED | CAP_PRIOR_TREE
// Open asset formats: glTF (Khronos), SPZ (open source).
| CAP_MESH_GLB | CAP_SPLAT_SPZ
// JPEG 2000 Part 1 is fee-free under ITU-T policy 2.1 (extensions /
@ -1492,8 +1513,10 @@ pub fn capability_replicate_class(cap: &str) -> Option<ReplicateClass> {
| CAP_QUANTUM_CIRCUIT | CAP_QUANTUM_CIRCUIT2 | CAP_TERNARY_TT
// A parameter-set listing: SPEC §3.1 fixes its bytes exactly, and they are
// its own output. A tensor file: its format fixes where each tensor's bytes are,
// and reading hands them over unchanged (prior-carriage §9).
| CAP_PRIOR_SET | CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF
// and reading hands them over unchanged (prior-carriage §9). A sealed set: opening
// it under its key is a function of its bytes and the key (prior-carriage §18.2). A
// chunk tree: every piece is checked against the head, the whole by its SHA-256 (§19).
| CAP_PRIOR_SET | CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF | CAP_PRIOR_SEALED | CAP_PRIOR_TREE
// A residual refinement: integer prediction, residual and clamp
// (staged-delivery §6.1), exact given the stage below; it composes its base's
// decoded output (`composes_decoded_sources`), so it is as exact as its base.
@ -2717,6 +2740,8 @@ mod tests {
(CAP_PRIOR_SET, "SPEC.md §3.1: a listing's bytes are fixed exactly and are their own output"),
(CAP_PRIOR_SAFETENSORS, "extensions/prior-carriage.md §9.1: the tensors' bytes are the file's, fixed by its header"),
(CAP_PRIOR_GGUF, "extensions/prior-carriage.md §9.2: the tensors' bytes are the file's, fixed by its tensor descriptions"),
(CAP_PRIOR_SEALED, "extensions/prior-carriage.md §18.2: opened under its key, the inner set's bytes, checked by their SHA-256"),
(CAP_PRIOR_TREE, "extensions/prior-carriage.md §19: the set's bytes, each piece checked against the head and the whole by its SHA-256"),
(CAP_IMAGE_INT_REFINE, "extensions/staged-delivery.md §6.1: integer predictor, residual and clamp; staged-conformance"),
(CAP_VIDEO_INT_REFINE, "extensions/staged-delivery.md §6.1: integer predictor, residual and clamp; staged-conformance"),
(CAP_AUDIO_INT_REFINE, "extensions/staged-delivery.md §6.1: integer predictor, residual and clamp; staged-conformance"),

19
wai/wai-rs/src/codecs/prior_form.rs
View file

@ -8,11 +8,11 @@
//! name, and a sink decides whether it can read a form by the same dispatch it
//! uses for payloads.
use super::{CAP_PRIOR_GGUF, CAP_PRIOR_NNC, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SET};
use super::{CAP_PRIOR_GGUF, CAP_PRIOR_NNC, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SEALED, CAP_PRIOR_SET, CAP_PRIOR_TREE};
/// Every capability string a `Prior` may declare as its artifact's form.
pub fn prior_forms() -> &'static [&'static str] {
&[CAP_PRIOR_GGUF, CAP_PRIOR_NNC, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SET]
&[CAP_PRIOR_GGUF, CAP_PRIOR_NNC, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SEALED, CAP_PRIOR_SET, CAP_PRIOR_TREE]
}
/// Whether `cap` is a registered prior form.
@ -23,13 +23,18 @@ pub fn is_prior_form(cap: &str) -> bool {
/// Prior forms whose decode to the tensors a network consumes is **established**
/// to give identical tensors at every conforming reader.
///
/// Three. `wai.prior.set`: reading a set through its listing compares digests and
/// Five. `wai.prior.set`: reading a set through its listing compares digests and
/// hands the decoder each file's bytes unchanged, so every sink that holds the set
/// reads the same parameters. `wai.prior.safetensors` and `wai.prior.gguf`: a reader that
/// follows prior-carriage §9 finds each tensor's bytes where the file's header puts them
/// and hands them over unchanged, refusing a file whose header could be read two ways (a
/// name twice, overlapping or uncovered bytes), so every such reader reads the same
/// tensors from the same file.
/// tensors from the same file. `wai.prior.sealed`: opening it under its key is a function of
/// its bytes and the key, its plaintext is checked against the SHA-256 it states, and its inner
/// form is one of the two tensor files or a capability's one file (prior-carriage §18.2), so it
/// hands over the bytes the inner form would. `wai.prior.tree`: every piece is checked against
/// its head and the whole set against the SHA-256 the head states, and its inner form is one of
/// these or a sealed set (prior-carriage §19).
///
/// No coded form is listed. ISO/IEC 15938-17 already has payload types that carry
/// values without quantisation: its 2022 edition lists decoding methods for an
@ -43,7 +48,7 @@ pub fn is_prior_form(cap: &str) -> bool {
/// lossless mode added by an amendment would not settle it by itself. Until a coded
/// form is listed, a `replicate` claim cannot rest on a pin in it (see
/// [`Prior::form_preserves_exactness`](crate::container::Prior::form_preserves_exactness)).
const EXACT_RECONSTRUCTION_FORMS: &[&str] = &[CAP_PRIOR_GGUF, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SET];
const EXACT_RECONSTRUCTION_FORMS: &[&str] = &[CAP_PRIOR_GGUF, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SEALED, CAP_PRIOR_SET, CAP_PRIOR_TREE];
/// Whether reading an artifact out of `form` is established to reproduce the
/// same tensors at every conforming reader. `false` for an unregistered form.
@ -66,7 +71,7 @@ mod tests {
#[test]
fn prior_forms_are_registered_capabilities() {
assert_eq!(prior_forms(), [CAP_PRIOR_GGUF, CAP_PRIOR_NNC, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SET]);
assert_eq!(prior_forms(), [CAP_PRIOR_GGUF, CAP_PRIOR_NNC, CAP_PRIOR_SAFETENSORS, CAP_PRIOR_SEALED, CAP_PRIOR_SET, CAP_PRIOR_TREE]);
for &f in prior_forms() {
assert!(is_known_capability(f), "{f} must be a registered capability");
}
@ -82,6 +87,8 @@ mod tests {
assert!(form_reconstruction_is_exact(CAP_PRIOR_SET));
assert!(form_reconstruction_is_exact(CAP_PRIOR_SAFETENSORS));
assert!(form_reconstruction_is_exact(CAP_PRIOR_GGUF));
assert!(form_reconstruction_is_exact(CAP_PRIOR_SEALED));
assert!(form_reconstruction_is_exact(CAP_PRIOR_TREE));
assert!(!form_reconstruction_is_exact(CAP_PRIOR_NNC));
assert!(!form_reconstruction_is_exact("wai.prior.nope"));
}

10
wai/wai-rs/src/codecs/registry.rs
View file

@ -318,6 +318,8 @@ const ROWS: &[(&str, &str, Option<&str>, &str)] = &[
(CAP_PRIOR_SET, S_PRIOR, None, "2026-09-30"),
(CAP_PRIOR_SAFETENSORS, S_PRIOR, None, "2026-10-22"),
(CAP_PRIOR_GGUF, S_PRIOR, Some("GGUF version 3"), "2026-10-22"),
(CAP_PRIOR_SEALED, S_PRIOR, None, "2026-11-04"),
(CAP_PRIOR_TREE, S_PRIOR, None, "2026-11-04"),
(CAP_TENSOR_NNC, S_PRIOR, None, "2026-09-28"),
(CAP_EDGE_DERIVATION, S_PRIOR, None, "2026-10-21"),
(CAP_WORLD_SCENE, S_INTERACTIVE, None, "2026-06-13"),
@ -726,6 +728,8 @@ pub const WIRE_IDENTIFIERS: &[WireIdentifier] = &[
wire("/.well-known/wai-grants/", WireKind::WellKnown, HTTP_DELIVERY, "§8.4", WireStatus::Defined),
wire("wai-module-grants", WireKind::DocumentKind, HTTP_DELIVERY, "§8.4", WireStatus::Defined),
wire("mr/", WireKind::Path, HTTP_DELIVERY, "§8.6", WireStatus::Defined),
wire("/.well-known/wai-sets/", WireKind::WellKnown, HTTP_DELIVERY, "§8.7", WireStatus::Defined),
wire("science.transaction.wai.sets", WireKind::DataId, HTTP_DELIVERY, "§8.7", WireStatus::Defined),
wire("#WAI-LAYER", WireKind::PlaylistTag, HTTP_DELIVERY, "§10.4", WireStatus::Defined),
wire("science.transaction.wai.multivariant", WireKind::DataId, HTTP_DELIVERY, "§11.1", WireStatus::Defined),
wire("X-WAI-FLOOR", WireKind::PlaylistAttribute, HTTP_DELIVERY, "§11.1", WireStatus::Defined),
@ -835,6 +839,10 @@ pub const WIRE_IDENTIFIERS: &[WireIdentifier] = &[
wire("wai-mark:conformance", WireKind::Scheme, SYNTHESIS, "§7.4", WireStatus::Defined),
wire("/.well-known/wai-mark/", WireKind::WellKnown, SYNTHESIS, "§7.7", WireStatus::Proposed),
wire("WSR1", WireKind::Magic, EXACT_UPSCALE, "§3.1", WireStatus::Defined),
wire("WSK1", WireKind::Magic, PRIOR_CARRIAGE, "§18.2", WireStatus::Defined),
wire("wai-set-usage", WireKind::DocumentKind, PRIOR_CARRIAGE, "§18.1", WireStatus::Defined),
wire("wai-usage.json", WireKind::Path, PRIOR_CARRIAGE, "§18.1", WireStatus::Defined),
wire("WTR1", WireKind::Magic, PRIOR_CARRIAGE, "§19", WireStatus::Defined),
wire("wai-upscale", WireKind::DocumentKind, EXACT_UPSCALE, "§4.2", WireStatus::Defined),
wire("wai-presentation", WireKind::DocumentKind, EXACT_UPSCALE, "§6.2", WireStatus::Defined),
wire("wai-int/1", WireKind::Scheme, PRIOR_CARRIAGE, "§10", WireStatus::Defined),
@ -1150,7 +1158,7 @@ pub fn capability_media(cap: &str) -> Option<Media> {
| CAP_FILM_LINEAR | CAP_FILM_REEL => Video,
CAP_WORLD_SCENE | CAP_WORLD_REPLAY | CAP_WORLD_LIVE | CAP_WORLD_MODEL
| CAP_FEED_STATE | CAP_AVATAR_POSE | CAP_AVATAR_FACE => Interactive,
CAP_PRIOR_NNC | CAP_PRIOR_SET | CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF | CAP_TENSOR_NNC | CAP_QUANTUM_CIRCUIT | CAP_QUANTUM_CIRCUIT2
CAP_PRIOR_NNC | CAP_PRIOR_SET | CAP_PRIOR_SAFETENSORS | CAP_PRIOR_GGUF | CAP_PRIOR_SEALED | CAP_PRIOR_TREE | CAP_TENSOR_NNC | CAP_QUANTUM_CIRCUIT | CAP_QUANTUM_CIRCUIT2
| CAP_TERNARY_TT => Binary,
c if c.starts_with("wai.image.") => Image,
c if c.starts_with("wai.audio.") => Audio,

5
wai/wai-rs/src/lib.rs
View file

@ -826,6 +826,11 @@ pub mod influence;
/// cbindgen:ignore
#[cfg(feature = "prior_wire")]
pub mod prior_wire;
// Prior carriage (extensions/prior-carriage.md §15; extensions/http-delivery.md §8.7): set roots,
// location hints, and a set fetched by its digest over responses the caller supplies. Rust-only.
/// cbindgen:ignore
#[cfg(feature = "prior_carriage")]
pub mod prior_carriage;
// Scene carriage (extensions/scene-carriage.md): WAI worlds, audio scenes, mesh assets and timed
// media onto ISO/IEC 23090-14 scene description and a USD layer, both ways. Rust-only.

75
wai/wai-rs/src/prior_carriage.rs Normal file
View file

@ -0,0 +1,75 @@
//! Where a parameter set is fetched, and the rules around holding one
//! (extensions/prior-carriage.md §15 to §19; extensions/http-delivery.md §8.5, §8.7, §8.8,
//! §12.5).
//!
//! A pin names a set by its SHA-256 and nothing else (SPEC §3.1). This module is what a sink
//! does to obtain a set it lacks: the set roots it asks, in their order, the location hints a
//! MoQ catalog entry or a WAI multivariant playlist states, and the checks that hold a
//! response's bytes only when their digest is the pin's. It opens no connection: the caller
//! supplies each response.
//!
//! - [`locations`]: set roots, hints, a set's URL, and a fetch over supplied responses.
//! - [`dictionary`]: a set as an RFC 9842 dictionary: the fields' values, the checks of a
//! coded response, and `dcz` decoding and coding.
//! - [`amortise`]: when a set is worth holding, charged per sink, with the generations a sink
//! holds.
//! - [`usage`], [`sealed`], [`privacy`]: a set's usage statement, a set sealed under a key its
//! origin releases, and what a sink discloses of the sets it holds.
//! - [`tree`]: the chunk-tree form, a set fetched by range from several peers and verified per
//! piece.
//!
//! The conformance corpus is `live-conformance/prior-carriage/` ([`corpus`]), and its second
//! implementation `live-conformance/prior_carriage_verify.py`.
pub mod amortise;
pub mod corpus;
pub mod dictionary;
pub mod locations;
pub mod privacy;
pub mod sealed;
pub mod tree;
pub mod usage;
/// The reasons http-delivery §13 lists for §8.7 and §8.8, in the order its rows give them.
pub const HTTP_CODES: &[&str] = &[
"set_location_malformed",
"set_location_unavailable",
"set_too_large",
"set_location_mismatch",
"dictionary_unrequested",
"dictionary_coding_malformed",
"dictionary_mismatch",
"dictionary_window_too_large",
];
/// The reasons prior-carriage §20 lists, in its order.
pub const CODES: &[&str] = &[
amortise::PLAN_MALFORMED,
usage::USAGE_MALFORMED,
sealed::SEALED_MALFORMED,
sealed::SEALED_INNER_FORM,
sealed::SET_KEY_ABSENT,
sealed::SET_KEY_MISMATCH,
tree::TREE_HEAD_MALFORMED,
tree::TREE_INNER_FORM,
tree::TREE_ROOT_MISMATCH,
tree::TREE_PIECE_MISMATCH,
tree::TREE_DIGEST_MISMATCH,
];
/// 64 lowercase hexadecimal digits.
pub(crate) fn hex(b: &[u8]) -> String {
b.iter().map(|x| format!("{x:02x}")).collect()
}
/// A digest written as 64 lowercase hexadecimal digits, or `None`.
pub(crate) fn digest_of_hex(s: &str) -> Option<[u8; 32]> {
if s.len() != 64 || !s.bytes().all(|c| c.is_ascii_digit() || (b'a'..=b'f').contains(&c)) {
return None;
}
let mut d = [0u8; 32];
for (i, o) in d.iter_mut().enumerate() {
*o = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
}
Some(d)
}

249
wai/wai-rs/src/prior_carriage/amortise.rs Normal file
View file

@ -0,0 +1,249 @@
//! Amortisation, bootstrap and generations (extensions/prior-carriage.md §17).
//!
//! A set is worth holding for a sink when what it saves that sink over the titles the sink is
//! expected to fetch within a declared span exceeds what holding it costs that sink. The cost is
//! charged per sink: nothing when the sink holds the set, the coded bytes when it holds the set
//! a successor is coded against (§16), the set's bytes otherwise. A sink holds at most a stated
//! number of generations; [`plan`] keeps those whose net saving is largest and gives every
//! other title its classical rendition. Integer arithmetic only.
use serde_json::{json, Map, Value};
use std::collections::BTreeMap;
/// The reason a plan that does not read is refused with.
pub const PLAN_MALFORMED: &str = "plan_malformed";
const MAX_INT: u64 = (1 << 53) - 1;
fn uint(v: &Value) -> Option<u64> {
v.as_u64().filter(|n| *n <= MAX_INT)
}
/// `YYYY-MM-DDTHH:MM:SSZ`, digits where the pattern has them.
fn is_instant(s: &str) -> bool {
let b = s.as_bytes();
b.len() == 20
&& b.iter().enumerate().all(|(i, c)| match i {
4 | 7 => *c == b'-',
10 => *c == b'T',
13 | 16 => *c == b':',
19 => *c == b'Z',
_ => c.is_ascii_digit(),
})
}
/// A generation as a plan states it.
#[derive(Debug, Clone)]
struct Generation {
id: String,
epoch: u64,
set_bytes: u64,
/// The generation this one's set is coded against, and the coded bytes.
delta: Option<(String, u64)>,
}
#[derive(Debug, Clone)]
struct Title {
generation: String,
pinned: u64,
classical: u64,
}
fn ids_unique<'a>(ids: impl Iterator<Item = &'a str>) -> bool {
let mut seen = std::collections::BTreeSet::new();
ids.into_iter().all(|i| !i.is_empty() && seen.insert(i))
}
fn read(doc: &Value) -> Option<(u64, Vec<Generation>, BTreeMap<String, Title>, Vec<&Value>)> {
let span = doc.get("span")?;
let (from, to) = (span.get("from")?.as_str()?, span.get("to")?.as_str()?);
if !(is_instant(from) && is_instant(to) && from < to) {
return None;
}
let cap = uint(doc.get("cap")?).filter(|c| *c >= 1)?;
let mut gens: Vec<Generation> = Vec::new();
for g in doc.get("generations")?.as_array()? {
let id = g.get("id")?.as_str()?.to_string();
let epoch = uint(g.get("epoch")?)?;
let set_bytes = uint(g.get("set_bytes")?).filter(|n| *n >= 1)?;
let delta = match g.get("delta")? {
Value::Null => None,
d => {
let from = d.get("from")?.as_str()?.to_string();
let bytes = uint(d.get("bytes")?).filter(|n| *n >= 1)?;
// Coded against an earlier generation only.
gens.iter().find(|e| e.id == from)?;
Some((from, bytes))
}
};
if gens.last().is_some_and(|l| l.epoch >= epoch) {
return None;
}
gens.push(Generation { id, epoch, set_bytes, delta });
}
if gens.is_empty() || !ids_unique(gens.iter().map(|g| g.id.as_str())) {
return None;
}
let mut titles = BTreeMap::new();
for t in doc.get("titles")?.as_array()? {
let id = t.get("id")?.as_str()?.to_string();
let generation = t.get("generation")?.as_str()?.to_string();
gens.iter().find(|g| g.id == generation)?;
let title = Title { generation, pinned: uint(t.get("pinned_bytes")?)?, classical: uint(t.get("classical_bytes")?)? };
if id.is_empty() || titles.insert(id, title).is_some() {
return None;
}
}
let sinks: Vec<&Value> = doc.get("sinks")?.as_array()?.iter().collect();
for s in &sinks {
let holds = s.get("holds")?.as_array()?;
let held: Vec<&str> = holds.iter().map(Value::as_str).collect::<Option<_>>()?;
if !ids_unique(held.iter().copied()) || !held.iter().all(|h| gens.iter().any(|g| g.id == *h)) {
return None;
}
for (t, n) in s.get("demand")?.as_object()? {
titles.get(t)?;
uint(n)?;
}
}
if sinks.is_empty() {
return None;
}
Some((cap, gens, titles, sinks))
}
/// The plan for every sink a document states (§17), or [`PLAN_MALFORMED`].
pub fn plan(doc: &Value) -> Result<Value, &'static str> {
let (cap, gens, titles, sinks) = read(doc).ok_or(PLAN_MALFORMED)?;
let mut out = Vec::new();
for s in sinks {
let held: Vec<&str> = s["holds"].as_array().into_iter().flatten().filter_map(Value::as_str).collect();
let demand: BTreeMap<&str, u64> =
s["demand"].as_object().into_iter().flatten().map(|(k, v)| (k.as_str(), v.as_u64().unwrap_or(0))).collect();
// Each generation the demand touches, or the sink holds: saving, cost, net.
let mut rows: Vec<(usize, u128, u128, &'static str, i128)> = Vec::new();
for (i, g) in gens.iter().enumerate() {
let touched = demand.iter().any(|(t, n)| *n > 0 && titles[*t].generation == g.id);
let is_held = held.contains(&g.id.as_str());
if !touched && !is_held {
continue;
}
let saving: u128 = demand
.iter()
.filter(|(t, n)| **n > 0 && titles[**t].generation == g.id && titles[**t].classical > titles[**t].pinned)
.map(|(t, n)| *n as u128 * (titles[*t].classical - titles[*t].pinned) as u128)
.sum();
let (cost, via) = if is_held {
(0, "held")
} else {
match &g.delta {
Some((from, bytes)) if held.contains(&from.as_str()) => (*bytes as u128, "delta"),
_ => (g.set_bytes as u128, "set"),
}
};
rows.push((i, saving, cost, via, saving as i128 - cost as i128));
}
// Keep those worth holding, at most `cap`, the largest net first and then the later
// generation; then, with room left, generations already held, the later first.
let mut worth: Vec<&(usize, u128, u128, &str, i128)> = rows.iter().filter(|r| r.4 > 0).collect();
worth.sort_by(|a, b| b.4.cmp(&a.4).then(b.0.cmp(&a.0)));
let mut kept: Vec<usize> = worth.iter().take(cap as usize).map(|r| r.0).collect();
let mut rest: Vec<&(usize, u128, u128, &str, i128)> = rows.iter().filter(|r| r.3 == "held" && !kept.contains(&r.0)).collect();
rest.sort_by(|a, b| b.0.cmp(&a.0));
for r in rest {
if kept.len() < cap as usize {
kept.push(r.0);
}
}
let mut fetch = Vec::new();
let mut bytes: u128 = 0;
for r in &rows {
if kept.contains(&r.0) && r.3 != "held" {
fetch.push(json!([gens[r.0].id, r.3, r.2 as u64]));
bytes += r.2;
}
}
let mut per_title = Map::new();
let mut classical_bytes: u128 = 0;
for (t, n) in &demand {
if *n == 0 {
continue;
}
let title = &titles[*t];
let g = gens.iter().position(|g| g.id == title.generation).unwrap_or(usize::MAX);
let pinned = kept.contains(&g) && title.classical > title.pinned;
per_title.insert(t.to_string(), json!(if pinned { "pinned" } else { "classical" }));
bytes += *n as u128 * (if pinned { title.pinned } else { title.classical }) as u128;
classical_bytes += *n as u128 * title.classical as u128;
}
out.push(json!({
"generations": rows.iter().map(|r| json!({
"id": gens[r.0].id, "saving": r.1 as u64, "cost": r.2 as u64, "via": r.3, "net": r.4 as i64,
})).collect::<Vec<_>>(),
"hold": kept.iter().map(|i| gens[*i].id.clone()).collect::<Vec<_>>(),
"fetch": fetch,
"evict": held.iter().filter(|h| !kept.iter().any(|k| gens[*k].id == **h)).collect::<Vec<_>>(),
"titles": per_title,
"bytes": bytes as u64,
"classical_bytes": classical_bytes as u64,
}));
}
Ok(Value::Array(out))
}
#[cfg(test)]
mod tests {
use super::*;
fn doc(cap: u64, holds: &[&str], demand: Value) -> Value {
json!({
"span": { "from": "2026-11-01T00:00:00Z", "to": "2026-12-01T00:00:00Z" }, "cap": cap,
"generations": [
{ "id": "g1", "epoch": 1, "set_bytes": 6000, "delta": null },
{ "id": "g2", "epoch": 2, "set_bytes": 6000, "delta": { "from": "g1", "bytes": 1500 } },
],
"titles": [
{ "id": "a", "generation": "g1", "pinned_bytes": 1000, "classical_bytes": 3500 },
{ "id": "b", "generation": "g2", "pinned_bytes": 1000, "classical_bytes": 3000 },
],
"sinks": [{ "holds": holds, "demand": demand }],
})
}
#[test]
fn worth_holding_is_strictly_more_than_the_cost() {
let p = plan(&doc(2, &[], json!({ "a": 3 }))).expect("reads");
assert_eq!(p[0]["fetch"], json!([["g1", "set", 6000]]));
assert_eq!(p[0]["bytes"], json!(9000));
// 2500 × 2 = 5000 < 6000: classical.
let p = plan(&doc(2, &[], json!({ "a": 2 }))).expect("reads");
assert_eq!(p[0]["titles"]["a"], json!("classical"));
assert_eq!(p[0]["hold"], json!([]));
}
#[test]
fn a_successor_costs_its_delta_where_the_predecessor_is_held() {
let p = plan(&doc(2, &["g1"], json!({ "b": 1 }))).expect("reads");
assert_eq!(p[0]["fetch"], json!([["g2", "delta", 1500]]));
assert_eq!(p[0]["hold"], json!(["g2", "g1"]));
let p = plan(&doc(1, &["g1"], json!({ "b": 1 }))).expect("reads");
assert_eq!(p[0]["evict"], json!(["g1"]));
let p = plan(&doc(2, &[], json!({ "b": 1 }))).expect("reads");
assert_eq!(p[0]["titles"]["b"], json!("classical"));
}
#[test]
fn malformed_plans() {
let mut d = doc(2, &[], json!({ "a": 1 }));
d["cap"] = json!(0);
assert_eq!(plan(&d), Err(PLAN_MALFORMED));
let mut d = doc(2, &[], json!({ "z": 1 }));
assert_eq!(plan(&d), Err(PLAN_MALFORMED));
d = doc(2, &[], json!({ "a": 1 }));
d["span"]["to"] = json!("2026-10-01T00:00:00Z");
assert_eq!(plan(&d), Err(PLAN_MALFORMED));
d = doc(2, &[], json!({ "a": 1 }));
d["generations"][0]["delta"] = json!({ "from": "g2", "bytes": 10 });
assert_eq!(plan(&d), Err(PLAN_MALFORMED));
}
}

1088
wai/wai-rs/src/prior_carriage/corpus.rs Normal file
View file

File diff suppressed because it is too large Load diff

272
wai/wai-rs/src/prior_carriage/dictionary.rs Normal file
View file

@ -0,0 +1,272 @@
//! A parameter set as an RFC 9842 compression dictionary (extensions/http-delivery.md §8.8;
//! extensions/prior-carriage.md §16).
//!
//! The SHA-256 that RFC 9842's `Available-Dictionary` field carries is the digest a pin states
//! for a set in a one-file form. A sink that holds a set names it in that field; a server that
//! holds it MAY answer with the requested set coded against it, `Content-Encoding: dcz`
//! (Zstandard with the set as a raw dictionary, RFC 9842 §5). [`check_coded`] runs §8.8's
//! checks over such a response and decodes it; [`encode_dcz`] codes a set against another, as
//! an origin does. The reference decodes `dcz` only, and a sink built on it lists only `dcz`.
use crate::pin::sha256;
/// The `dcz` header's first 8 bytes: a Zstandard skippable frame of 32 bytes (RFC 9842 §5).
pub const DCZ_MAGIC: [u8; 8] = [0x5e, 0x2a, 0x4d, 0x18, 0x20, 0x00, 0x00, 0x00];
/// The `dcb` header's first 4 bytes (RFC 9842 §4).
pub const DCB_MAGIC: [u8; 4] = [0xff, 0x44, 0x43, 0x42];
/// The `dcz` window every client decodes: 8 MB, read as 2^23 bytes.
pub const DCZ_WINDOW_FLOOR: u64 = 1 << 23;
/// The largest `dcz` window: 128 MB, read as 2^27 bytes.
pub const DCZ_WINDOW_CEILING: u64 = 1 << 27;
/// The Zstandard frame magic (RFC 8878 §3.1.1).
const ZSTD_MAGIC: [u8; 4] = [0x28, 0xb5, 0x2f, 0xfd];
const B64: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
fn base64(b: &[u8]) -> String {
let mut s = String::new();
for c in b.chunks(3) {
let n = (c[0] as u32) << 16 | (*c.get(1).unwrap_or(&0) as u32) << 8 | *c.get(2).unwrap_or(&0) as u32;
for i in 0..4 {
if i <= c.len() {
s.push(B64[(n >> (18 - 6 * i) & 63) as usize] as char);
} else {
s.push('=');
}
}
}
s
}
/// The value of `Available-Dictionary` naming the set of digest `d`: a Structured Field Byte
/// Sequence (RFC 9651 §3.3.5), `:` base64 `:`.
pub fn available_dictionary(d: &[u8; 32]) -> String {
format!(":{}:", base64(d))
}
/// The digest an `Available-Dictionary` value names: one Byte Sequence of 32 bytes, padded
/// base64 between colons, with spaces around it discarded; `None` otherwise.
pub fn parse_available_dictionary(v: &str) -> Option<[u8; 32]> {
let inner = v.trim_matches(' ').strip_prefix(':')?.strip_suffix(':')?;
if inner.len() != 44 || !inner.ends_with('=') || inner[..43].contains('=') {
return None;
}
let mut bits: u32 = 0;
let mut n = 0;
let mut out = Vec::with_capacity(33);
for c in inner[..43].bytes() {
let x = B64.iter().position(|&b| b == c)? as u32;
bits = bits << 6 | x;
n += 6;
if n >= 8 {
n -= 8;
out.push((bits >> n) as u8);
bits &= (1 << n) - 1;
}
}
// 43 characters carry 258 bits: 32 bytes and 2 pad bits, which RFC 4648 §3.5 has zero.
(out.len() == 32 && bits == 0).then(|| out.try_into().ok()).flatten()
}
/// The value of `Use-As-Dictionary` a server sends with a set served below the root whose path
/// is `root_path`: every set below the root matches it (RFC 9842 §2.1).
pub fn use_as_dictionary(root_path: &str) -> String {
format!("match=\"{root_path}*\"")
}
/// The window a Zstandard frame states (RFC 8878 §3.1.1.1): its Window_Size, or its
/// Frame_Content_Size when it is a single segment. `None` where the header does not read.
pub fn zstd_window(frame: &[u8]) -> Option<u64> {
if frame.get(..4)? != ZSTD_MAGIC {
return None;
}
let fhd = *frame.get(4)?;
if fhd & 0x08 != 0 {
return None;
}
let single = fhd & 0x20 != 0;
let did = [0usize, 1, 2, 4][(fhd & 3) as usize];
let fcs = match fhd >> 6 {
0 if single => 1,
0 => 0,
1 => 2,
2 => 4,
_ => 8,
};
let mut at = 5;
let window = if single {
None
} else {
let wd = *frame.get(at)?;
at += 1;
let log = 10 + (wd >> 3) as u32;
let base = 1u64 << log;
Some(base + (base / 8) * (wd & 7) as u64)
};
at += did;
let size = frame.get(at..at + fcs)?;
let mut v = 0u64;
for (i, b) in size.iter().enumerate() {
v |= (*b as u64) << (8 * i);
}
if fcs == 2 {
v += 256;
}
Some(window.unwrap_or(v))
}
/// Whether a `dcz` frame's window is one RFC 9842 §5 has a client decode against a dictionary
/// of `dict_len` bytes: at most 8 MB or 1.25 times the dictionary's length, whichever is
/// larger, and never above 128 MB.
pub fn dcz_window_admitted(window: u64, dict_len: u64) -> bool {
window <= DCZ_WINDOW_CEILING && (window <= DCZ_WINDOW_FLOOR || window.saturating_mul(4) <= dict_len.saturating_mul(5))
}
/// Decodes a Zstandard frame against `dictionary` as raw content (RFC 8878 §5), reading at
/// most `limit` bytes of output. `Err("set_too_large")` past it, `Err("dictionary_coding_malformed")`
/// where it does not decode to exactly one whole frame.
fn zstd_decode(frame: &[u8], dictionary: &[u8], limit: u64) -> Result<Vec<u8>, &'static str> {
use zstd_safe::{DCtx, DParameter, InBuffer, OutBuffer};
const MALFORMED: &str = "dictionary_coding_malformed";
let mut d = DCtx::create();
d.set_parameter(DParameter::WindowLogMax(27)).map_err(|_| MALFORMED)?;
d.ref_prefix(dictionary).map_err(|_| MALFORMED)?;
let mut input = InBuffer::around(frame);
let mut out = Vec::new();
let mut chunk = vec![0u8; 1 << 16];
loop {
let mut o = OutBuffer::around(&mut chunk[..]);
let hint = d.decompress_stream(&mut o, &mut input).map_err(|_| MALFORMED)?;
let n = o.pos();
out.extend_from_slice(&chunk[..n]);
if out.len() as u64 > limit {
return Err("set_too_large");
}
if hint == 0 {
// A frame ended: nothing may follow it.
return if input.pos() == frame.len() { Ok(out) } else { Err(MALFORMED) };
}
if input.pos() == frame.len() && n == 0 {
return Err(MALFORMED);
}
}
}
/// Codes `target` against `dictionary` as a `dcz` body: the header, then a Zstandard frame at
/// level 19 with the dictionary as raw content, its content size and checksum stated.
pub fn encode_dcz(target: &[u8], dictionary: &[u8]) -> Result<Vec<u8>, String> {
use zstd_safe::{CCtx, CParameter};
let mut c = CCtx::create();
for p in [CParameter::CompressionLevel(19), CParameter::ChecksumFlag(true), CParameter::ContentSizeFlag(true)] {
c.set_parameter(p).map_err(|e| format!("zstd parameter: {e}"))?;
}
c.ref_prefix(dictionary).map_err(|e| format!("zstd prefix: {e}"))?;
let mut frame = Vec::with_capacity(zstd_safe::compress_bound(target.len()));
c.compress2(&mut frame, target).map_err(|e| format!("zstd: {e}"))?;
let mut out = DCZ_MAGIC.to_vec();
out.extend_from_slice(&sha256(dictionary));
out.extend_from_slice(&frame);
Ok(out)
}
/// A `dcz` body holding `target` in one raw block of a frame that states the window `2^log`
/// and no content size: a frame whose window a client checks before it decodes anything.
pub fn dcz_raw_with_window(target: &[u8], dictionary: &[u8], log: u8) -> Vec<u8> {
assert!((10..=41).contains(&log) && target.len() < 1 << 17);
let mut out = DCZ_MAGIC.to_vec();
out.extend_from_slice(&sha256(dictionary));
out.extend_from_slice(&ZSTD_MAGIC);
out.push(0x00); // no content size, multi-segment, no checksum, no dictionary ID
out.push((log - 10) << 3);
let h = 1u32 | (target.len() as u32) << 3; // last block, raw
out.extend_from_slice(&h.to_le_bytes()[..3]);
out.extend_from_slice(target);
out
}
/// What a sink sent with a request: the dictionary it named, if any. It lists `dcz` in
/// `Accept-Encoding` exactly when it names one.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Asked<'a> {
pub dictionary: Option<(&'a [u8; 32], &'a [u8])>,
}
/// §8.8's checks over a response whose `Content-Encoding` is `coding`, in order, and its
/// decoding: the decoded bytes, or the reason. A coding other than `dcz` and `dcb` is not a
/// dictionary coding and is refused by §8.7 step 2, not here.
pub fn check_coded(coding: &str, body: &[u8], asked: Asked<'_>, limit: u64) -> Result<Vec<u8>, &'static str> {
// 1. The request named a dictionary and listed this coding (the reference lists dcz only).
let Some((named, dictionary)) = asked.dictionary.filter(|_| coding == "dcz") else {
return Err("dictionary_unrequested");
};
// 2. The header: the magic, then the SHA-256 of the dictionary the request named.
if body.len() < 40 || body[..8] != DCZ_MAGIC {
return Err("dictionary_coding_malformed");
}
if body[8..40] != named[..] || sha256(dictionary) != *named {
return Err("dictionary_mismatch");
}
// 3. The frame's window.
let frame = &body[40..];
let window = zstd_window(frame).ok_or("dictionary_coding_malformed")?;
if !dcz_window_admitted(window, dictionary.len() as u64) {
return Err("dictionary_window_too_large");
}
// 4. The decoding, within the bound.
zstd_decode(frame, dictionary, limit)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_field_values() {
let d = [0xa5u8; 32];
let v = available_dictionary(&d);
assert_eq!(v.len(), 46);
assert_eq!(parse_available_dictionary(&v), Some(d));
assert_eq!(parse_available_dictionary(&format!(" {v} ")), Some(d));
// RFC 9842 §2.2's example value.
assert!(parse_available_dictionary(":pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG4=:").is_some());
assert_eq!(parse_available_dictionary(&v[1..]), None);
assert_eq!(parse_available_dictionary(":pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG5=:"), None);
assert_eq!(parse_available_dictionary(":pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG4:"), None);
assert_eq!(use_as_dictionary("/.well-known/wai-sets/"), "match=\"/.well-known/wai-sets/*\"");
}
#[test]
fn windows() {
assert!(dcz_window_admitted(1 << 23, 10));
assert!(!dcz_window_admitted((1 << 23) + 1, 10));
assert!(dcz_window_admitted(10 << 20, 8 << 20));
assert!(!dcz_window_admitted((10 << 20) + 1, 8 << 20));
assert!(!dcz_window_admitted((1 << 27) + 1, 1 << 30));
let raw = dcz_raw_with_window(b"abc", b"dict", 24);
assert_eq!(zstd_window(&raw[40..]), Some(1 << 24));
}
#[test]
fn a_set_against_another_round_trips() {
let dict: Vec<u8> = (0..4000u32).map(|i| (i * 7 % 251) as u8).collect();
let mut target = dict.clone();
target[100] ^= 0xff;
target.extend_from_slice(b"tail");
let body = encode_dcz(&target, &dict).expect("codes");
assert!(body.len() < target.len() / 4, "{}", body.len());
let named = sha256(&dict);
let asked = Asked { dictionary: Some((&named, &dict)) };
assert_eq!(check_coded("dcz", &body, asked, 1 << 20).as_deref(), Ok(&target[..]));
assert_eq!(check_coded("dcb", &body, asked, 1 << 20), Err("dictionary_unrequested"));
assert_eq!(check_coded("dcz", &body, Asked { dictionary: None }, 1 << 20), Err("dictionary_unrequested"));
assert_eq!(check_coded("dcz", &body, asked, 100), Err("set_too_large"));
assert_eq!(check_coded("dcz", &body[..body.len() - 3], asked, 1 << 20), Err("dictionary_coding_malformed"));
let other = vec![0u8; 4000];
let other_named = sha256(&other);
assert_eq!(check_coded("dcz", &body, Asked { dictionary: Some((&other_named, &other)) }, 1 << 20), Err("dictionary_mismatch"));
let wide = dcz_raw_with_window(&target, &dict, 24);
assert_eq!(check_coded("dcz", &wide, asked, 1 << 20), Err("dictionary_window_too_large"));
let fits = dcz_raw_with_window(&target, &dict, 13);
assert_eq!(check_coded("dcz", &fits, asked, 1 << 20).as_deref(), Ok(&target[..]));
}
}

529
wai/wai-rs/src/prior_carriage/locations.rs Normal file
View file

@ -0,0 +1,529 @@
//! Set roots, location hints and a set fetched by its digest (extensions/http-delivery.md §8.5,
//! §8.7; extensions/prior-carriage.md §15).
//!
//! A **set root** is an `https` URI whose path ends in `/`; a set is at the root followed by its
//! digest. A hint is a list of roots a MoQ catalog's `wai_priors` entry (`locations`) or a WAI
//! multivariant playlist (the one `EXT-X-SESSION-DATA` line of [`DATA_ID`], its roots separated
//! by spaces) states; it names where to ask and proves nothing. [`fetch_set`] runs §8.7's steps over the responses its caller
//! supplies and holds bytes only when their SHA-256 is the digest asked for.
use super::dictionary::{check_coded, Asked};
use super::usage::{self, Usage, USAGE_FILE};
use super::{digest_of_hex, hex};
use crate::codecs::CAP_PRIOR_SET;
use crate::pin::{parse_listing, sha256};
use serde_json::Value;
/// The path below an origin's host and port under which it serves its sets (§8.7).
pub const WELL_KNOWN: &str = "/.well-known/wai-sets/";
/// The `DATA-ID` of a playlist's location-hint line (§8.7).
pub const DATA_ID: &str = "science.transaction.wai.sets";
/// The longest set root, in bytes.
pub const MAX_ROOT_BYTES: usize = 2048;
/// The most roots a hint keeps.
pub const MAX_ROOTS: usize = 8;
fn unreserved(c: u8) -> bool {
c.is_ascii_alphanumeric() || b"-._~".contains(&c)
}
fn sub_delim(c: u8) -> bool {
b"!$&'()*+,;=".contains(&c)
}
/// Every byte is unreserved, a sub-delimiter, one of `extra`, or starts a `%` and two
/// hexadecimal digits (RFC 3986 §2.1).
fn uri_chars(s: &[u8], extra: &[u8]) -> bool {
let mut i = 0;
while i < s.len() {
let c = s[i];
if c == b'%' {
if !(s.get(i + 1).is_some_and(u8::is_ascii_hexdigit) && s.get(i + 2).is_some_and(u8::is_ascii_hexdigit)) {
return false;
}
i += 3;
continue;
}
if !(unreserved(c) || sub_delim(c) || extra.contains(&c)) {
return false;
}
i += 1;
}
true
}
/// Whether `s` is a set root: an absolute URI (RFC 3986 §4.3) that begins `https://`, with a
/// non-empty host, no userinfo, no query and no fragment, whose path ends in `/`, at most
/// [`MAX_ROOT_BYTES`] long (http-delivery §8.7).
pub fn is_set_root(s: &str) -> bool {
if s.len() > MAX_ROOT_BYTES {
return false;
}
let Some(rest) = s.strip_prefix("https://") else { return false };
if rest.contains('?') || rest.contains('#') {
return false;
}
let Some(slash) = rest.find('/') else { return false };
let (authority, path) = rest.split_at(slash);
if authority.contains('@') {
return false;
}
let (host, port) = if let Some(inner) = authority.strip_prefix('[') {
let Some(close) = inner.find(']') else { return false };
let literal = &inner[..close];
if literal.is_empty() || !uri_chars(literal.as_bytes(), b":") || literal.contains('%') {
return false;
}
let after = &inner[close + 1..];
match after.strip_prefix(':') {
Some(p) => ("[]", Some(p)),
None if after.is_empty() => ("[]", None),
None => return false,
}
} else {
match authority.rfind(':') {
Some(i) => (&authority[..i], Some(&authority[i + 1..])),
None => (authority, None),
}
};
if host.is_empty() || (host != "[]" && !uri_chars(host.as_bytes(), b"")) {
return false;
}
if let Some(p) = port {
if p.is_empty() || !p.bytes().all(|c| c.is_ascii_digit()) {
return false;
}
}
path.ends_with('/') && uri_chars(path.as_bytes(), b":@/")
}
/// A hint as read: the roots it keeps, in order, and the positions of the elements passed over
/// as not holding a root (`set_location_malformed`). A playlist's elements are its hint line's
/// `VALUE` split at every space.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct Hint {
pub roots: Vec<String>,
pub malformed: Vec<usize>,
}
impl Hint {
/// Takes the element at `i`; `false` once the hint is full and the rest are ignored.
fn take(&mut self, i: usize, root: Option<&str>) -> bool {
if self.roots.len() == MAX_ROOTS {
return false;
}
match root {
Some(r) if is_set_root(r) => {
if !self.roots.iter().any(|k| k == r) {
self.roots.push(r.to_string());
}
}
_ => self.malformed.push(i),
}
true
}
}
/// The hint of a `wai_priors` entry: its `locations` member (moq-streaming-format §4.2). A
/// member that is absent, or not an array, gives none.
pub fn read_catalog_hint(entry: &Value) -> Hint {
let mut h = Hint::default();
if let Some(a) = entry.get("locations").and_then(Value::as_array) {
for (i, e) in a.iter().enumerate() {
if !h.take(i, e.as_str()) {
break;
}
}
}
h
}
/// An attribute value of an HLS attribute list: quoted, or not.
#[derive(Debug, PartialEq, Eq)]
enum AttrValue<'a> {
Quoted(&'a str),
Bare(&'a str),
}
/// An HLS attribute list (RFC 8216 §4.2), or `None` where it does not read as one: names of
/// `A`–`Z`, `0`–`9` and `-`, each once; a quoted string with no `"`, CR or LF inside.
fn attribute_list(s: &str) -> Option<Vec<(&str, AttrValue<'_>)>> {
let mut out: Vec<(&str, AttrValue<'_>)> = Vec::new();
let mut rest = s;
loop {
let eq = rest.find('=')?;
let name = &rest[..eq];
if name.is_empty() || !name.bytes().all(|c| c.is_ascii_uppercase() || c.is_ascii_digit() || c == b'-') {
return None;
}
if out.iter().any(|(n, _)| *n == name) {
return None;
}
rest = &rest[eq + 1..];
let value = if let Some(q) = rest.strip_prefix('"') {
let end = q.find('"')?;
let v = &q[..end];
if v.contains('\r') || v.contains('\n') {
return None;
}
rest = &q[end + 1..];
AttrValue::Quoted(v)
} else {
let end = rest.find(',').unwrap_or(rest.len());
let v = &rest[..end];
if v.is_empty() || v.contains('"') || v.bytes().any(|c| c.is_ascii_whitespace()) {
return None;
}
rest = &rest[end..];
AttrValue::Bare(v)
};
out.push((name, value));
if rest.is_empty() {
return Some(out);
}
rest = rest.strip_prefix(',')?;
}
}
/// The hint of a WAI multivariant playlist: the first `EXT-X-SESSION-DATA` line whose attribute
/// list reads, whose `DATA-ID` is [`DATA_ID`] and that carries no `LANGUAGE`, its `VALUE` split
/// at every space into elements. A later such line is ignored (RFC 8216 §4.3.4.4 admits one);
/// a hint line whose `VALUE` is not a quoted string, or that carries `URI`, gives none.
pub fn read_playlist_hint(text: &str) -> Hint {
let mut h = Hint::default();
for line in text.split('\n') {
let line = line.strip_suffix('\r').unwrap_or(line);
let Some(attrs) = line.strip_prefix("#EXT-X-SESSION-DATA:") else { continue };
let Some(list) = attribute_list(attrs) else { continue };
let get = |n: &str| list.iter().find(|(k, _)| *k == n).map(|(_, v)| v);
if get("DATA-ID") != Some(&AttrValue::Quoted(DATA_ID)) || get("LANGUAGE").is_some() {
continue;
}
if let (Some(AttrValue::Quoted(v)), None) = (get("VALUE"), get("URI")) {
for (i, e) in v.split(' ').enumerate() {
if !h.take(i, Some(e)) {
break;
}
}
}
break;
}
h
}
/// Where a set, or a listed file, of digest `sha256` is below `root`.
pub fn set_url(root: &str, sha256: &[u8; 32]) -> String {
format!("{root}{}", hex(sha256))
}
/// An origin's well-known root, from the origin `https://<host>[:<port>]`; `None` where that
/// does not make a set root.
pub fn well_known_root(origin: &str) -> Option<String> {
let r = format!("{origin}{WELL_KNOWN}");
(origin.strip_prefix("https://").is_some_and(|a| !a.contains('/')) && is_set_root(&r)).then_some(r)
}
/// The roots a sink asks, in order, each once: those of its own configuration, the hint's,
/// then the origin's well-known root.
pub fn roots_in_order(own: &[String], hint: &[String], origin_root: Option<&str>) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
for r in own.iter().map(String::as_str).chain(hint.iter().map(String::as_str)).chain(origin_root) {
if !out.iter().any(|k| k == r) {
out.push(r.to_string());
}
}
out
}
/// A request a sink sends: the URL, and the set it names in `Available-Dictionary` (with `dcz`
/// in `Accept-Encoding`), if any (http-delivery §8.8).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Request<'a> {
pub url: &'a str,
pub available_dictionary: Option<[u8; 32]>,
}
/// A response a caller supplies for one request.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Response {
pub status: u16,
/// The `Content-Encoding` field's value, `None` where absent.
pub content_encoding: Option<String>,
pub body: Vec<u8>,
}
/// One request sent, and what came of it: `held`, or a reason of §8.7 or §8.8.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Attempt {
pub url: String,
pub outcome: &'static str,
}
/// A set fetched and held: its bytes in the pin's form (the listing, for `wai.prior.set`) and,
/// for a listing, each file it names with its path and the usage statement it binds as its file
/// [`USAGE_FILE`] (prior-carriage §18.1), if any.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct HeldSet {
pub bytes: Vec<u8>,
pub files: Vec<(String, Vec<u8>)>,
pub usage: Option<Usage>,
}
/// The requests sent, in order, and the set held or the reason SPEC §3.1 refuses the pin with
/// (`prior_not_held`, `set_malformed`, `part_mismatch`), or `set_usage_malformed` for a listing
/// whose bound statement does not read (prior-carriage §18.1).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Fetch {
pub attempts: Vec<Attempt>,
pub result: Result<HeldSet, &'static str>,
}
/// A set a sink holds and names as a dictionary: its digest and bytes.
pub type Dictionary<'a> = (&'a [u8; 32], &'a [u8]);
/// What one response yields for a resource of digest `digest`: its bytes, or the reason; and
/// whether the reason is a check of §8.8 after which the same root is asked without the
/// dictionary.
fn outcome_of(
digest: &[u8; 32],
r: Option<Response>,
dictionary: Option<Dictionary<'_>>,
limit: u64,
) -> (Result<Vec<u8>, &'static str>, bool) {
let Some(r) = r else { return (Err("set_location_unavailable"), false) };
if r.status != 200 {
return (Err("set_location_unavailable"), false);
}
let bytes = match r.content_encoding.as_deref() {
None => {
if r.body.len() as u64 > limit {
return (Err("set_too_large"), false);
}
r.body
}
Some(c @ ("dcz" | "dcb")) => match check_coded(c, &r.body, Asked { dictionary }, limit) {
Ok(b) => b,
Err("set_too_large") => return (Err("set_too_large"), false),
Err(code) => return (Err(code), dictionary.is_some()),
},
Some(_) => return (Err("content_coding_present"), false),
};
if sha256(&bytes) != *digest {
return (Err("set_location_mismatch"), false);
}
(Ok(bytes), false)
}
/// The steps of §8.7 for one resource, with §8.8's when the sink names a dictionary: the first
/// root, in `roots`' order, whose response holds it, with that root's position.
fn fetch_one(
digest: &[u8; 32],
roots: &[&str],
limit: u64,
dictionary: Option<Dictionary<'_>>,
get: &mut dyn FnMut(&Request<'_>) -> Option<Response>,
attempts: &mut Vec<Attempt>,
) -> Option<(usize, Vec<u8>)> {
for (i, root) in roots.iter().enumerate() {
let url = set_url(root, digest);
let mut named = dictionary;
loop {
let r = get(&Request { url: &url, available_dictionary: named.map(|(d, _)| *d) });
let (got, again) = outcome_of(digest, r, named, limit);
attempts.push(Attempt { url: url.clone(), outcome: *got.as_ref().err().unwrap_or(&"held") });
match got {
Ok(b) => return Some((i, b)),
// A coded response refused by §8.8: the same root, asked without the dictionary.
Err(_) if again => named = None,
Err(_) => break,
}
}
}
None
}
/// Fetches the set a pin of digest `sha256_digest` and form `form` names, asking `roots` in
/// order with `get`, reading at most `limit` bytes of each response (http-delivery §8.7). With
/// `dictionary`, a set the sink holds, it names that set in `Available-Dictionary` when it asks
/// for a set in a one-file form (§8.8); it names none for a listing or a listed file. A listing's
/// file at the path [`USAGE_FILE`] is read as its usage statement once every file is held.
pub fn fetch_set(
sha256_digest: &[u8; 32],
form: Option<&str>,
roots: &[String],
limit: u64,
dictionary: Option<Dictionary<'_>>,
get: &mut dyn FnMut(&Request<'_>) -> Option<Response>,
) -> Fetch {
let mut attempts = Vec::new();
let order: Vec<&str> = roots.iter().map(String::as_str).collect();
let listing = form == Some(CAP_PRIOR_SET);
let named = if listing { None } else { dictionary };
let Some((at, bytes)) = fetch_one(sha256_digest, &order, limit, named, get, &mut attempts) else {
return Fetch { attempts, result: Err("prior_not_held") };
};
if !listing {
return Fetch { attempts, result: Ok(HeldSet { bytes, files: Vec::new(), usage: None }) };
}
let Ok(lines) = parse_listing(&bytes) else {
return Fetch { attempts, result: Err("set_malformed") };
};
// Each file first at the root that served the listing, then at the others in order.
let mut file_order = vec![order[at]];
file_order.extend(order.iter().enumerate().filter(|(i, _)| *i != at).map(|(_, r)| *r));
let mut files = Vec::new();
for (path, d) in lines {
match fetch_one(&d, &file_order, limit, None, get, &mut attempts) {
Some((_, b)) => files.push((path, b)),
None => return Fetch { attempts, result: Err("part_mismatch") },
}
}
// The statement the listing binds, at its top-level path only (prior-carriage §18.1).
let usage = match files.iter().find(|(p, _)| p == USAGE_FILE) {
Some((_, b)) => match usage::read(b) {
Ok(u) => Some(u),
Err(code) => return Fetch { attempts, result: Err(code) },
},
None => None,
};
Fetch { attempts, result: Ok(HeldSet { bytes, files, usage }) }
}
/// The digest and form a pin's JSON states, read for a fetch: `sha256` as 64 lowercase
/// hexadecimal digits and `form` a string or absent. No other member is read, so a member
/// SPEC §3.1 does not define, whatever it names, never reaches a fetch.
pub fn pin_target(pin: &Value) -> Option<([u8; 32], Option<String>)> {
let d = digest_of_hex(pin.get("sha256")?.as_str()?)?;
let form = match pin.get("form") {
None | Some(Value::Null) => None,
Some(Value::String(s)) => Some(s.clone()),
Some(_) => return None,
};
Some((d, form))
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn set_roots() {
for ok in [
"https://origin.example/.well-known/wai-sets/",
"https://cdn.example:8443/sets/",
"https://[2001:db8::1]/s/",
"https://h.example/a%20b/",
"https://h.example/",
] {
assert!(is_set_root(ok), "{ok}");
}
for bad in [
"http://origin.example/sets/",
"HTTPS://origin.example/sets/",
"https://origin.example/sets",
"https://origin.example/sets/?x=1",
"https://origin.example/sets/#f",
"https://[email protected]/sets/",
"https:///sets/",
"https://origin.example:/sets/",
"https://origin.example:84a3/sets/",
"https://origin.example/se ts/",
"https://origin.example/%zz/",
"https://origin.example",
"",
] {
assert!(!is_set_root(bad), "{bad}");
}
let long = format!("https://h.example/{}/", "a".repeat(MAX_ROOT_BYTES));
assert!(!is_set_root(&long));
let edge = format!("https://h.example/{}/", "a".repeat(MAX_ROOT_BYTES - 19));
assert_eq!(edge.len(), MAX_ROOT_BYTES);
assert!(is_set_root(&edge));
}
#[test]
fn catalog_hints() {
let e = json!({"locations": ["https://a.example/s/", 7, "http://b.example/s/", "https://a.example/s/", "https://c.example/s/"]});
let h = read_catalog_hint(&e);
assert_eq!(h.roots, ["https://a.example/s/", "https://c.example/s/"]);
assert_eq!(h.malformed, [1, 2]);
assert_eq!(read_catalog_hint(&json!({"locations": "https://a.example/s/"})), Hint::default());
assert_eq!(read_catalog_hint(&json!({})), Hint::default());
let many: Vec<String> = (0..12).map(|i| format!("https://m{i}.example/s/")).collect();
let h = read_catalog_hint(&json!({ "locations": many }));
assert_eq!(h.roots.len(), MAX_ROOTS);
assert_eq!(h.roots[7], "https://m7.example/s/");
}
#[test]
fn playlist_hints() {
let p = "#EXTM3U\r\n#EXT-X-SESSION-DATA:DATA-ID=\"other\",VALUE=\"https://x.example/s/\"\n\
#EXT-X-SESSION-DATA:DATA-ID=\"science.transaction.wai.sets\",VALUE=\"https://l.example/s/\",LANGUAGE=\"en\"\n\
#EXT-X-SESSION-DATA:DATA-ID=\"science.transaction.wai.sets\",VALUE=\"https://a.example/s/ http://b.example/s/ https://c.example/s/\"\r\n\
#EXT-X-SESSION-DATA:DATA-ID=\"science.transaction.wai.sets\",VALUE=\"https://d.example/s/\"\n";
let h = read_playlist_hint(p);
assert_eq!(h.roots, ["https://a.example/s/", "https://c.example/s/"]);
assert_eq!(h.malformed, [1, 2]);
let uri = "#EXT-X-SESSION-DATA:DATA-ID=\"science.transaction.wai.sets\",VALUE=\"https://a.example/s/\",URI=\"https://b.example/s.json\"\n\
#EXT-X-SESSION-DATA:DATA-ID=\"science.transaction.wai.sets\",VALUE=\"https://c.example/s/\"\n";
assert_eq!(read_playlist_hint(uri), Hint::default());
assert!(attribute_list("A=\"x\",A=\"y\"").is_none());
assert!(attribute_list("a=1").is_none());
}
#[test]
fn well_known() {
assert_eq!(well_known_root("https://origin.example").as_deref(), Some("https://origin.example/.well-known/wai-sets/"));
assert_eq!(well_known_root("https://origin.example:8443").as_deref(), Some("https://origin.example:8443/.well-known/wai-sets/"));
assert_eq!(well_known_root("http://origin.example"), None);
assert_eq!(well_known_root("https://origin.example/x"), None);
}
#[test]
fn a_tampered_mirror_is_passed_over() {
let set = b"a parameter set".to_vec();
let d = sha256(&set);
let roots = vec!["https://m1.example/s/".to_string(), "https://m2.example/s/".to_string()];
let mut bad = set.clone();
bad[0] ^= 1;
let mut get = |r: &Request<'_>| {
let body = if r.url.starts_with("https://m1.") { bad.clone() } else { set.clone() };
Some(Response { status: 200, content_encoding: None, body })
};
let f = fetch_set(&d, None, &roots, 1 << 20, None, &mut get);
assert_eq!(f.attempts.iter().map(|a| a.outcome).collect::<Vec<_>>(), ["set_location_mismatch", "held"]);
assert_eq!(f.result.expect("held").bytes, set);
let f = fetch_set(&d, None, &roots, 4, None, &mut get);
assert_eq!(f.result, Err("prior_not_held"));
}
#[test]
fn a_listing_binds_its_statement_at_its_top_level_path() {
let good: &[u8] = br#"{"kind":"wai-set-usage","licence":"Apache-2.0","terms":null,"content_derived":false,"redistribute":false,"ai_usage":null}"#;
let bad: &[u8] = br#"{"kind":"wai-set-usage"}"#;
let run = |path: &str, statement: &[u8]| {
let mut files = vec![(path.to_string(), statement.to_vec()), ("w.bin".to_string(), b"weights".to_vec())];
files.sort();
let listing: String = files.iter().map(|(p, b)| format!("{} {p}\n", hex(&sha256(b)))).collect();
let mut bodies: Vec<Vec<u8>> = files.into_iter().map(|(_, b)| b).collect();
bodies.push(listing.clone().into_bytes());
let mut get = |r: &Request<'_>| {
bodies.iter().find(|b| r.url.ends_with(&hex(&sha256(b)))).map(|b| Response { status: 200, content_encoding: None, body: b.clone() })
};
let roots = vec!["https://m.example/s/".to_string()];
fetch_set(&sha256(listing.as_bytes()), Some(CAP_PRIOR_SET), &roots, 1 << 20, None, &mut get).result
};
assert!(run(USAGE_FILE, good).expect("held").usage.is_some_and(|u| !u.redistribute));
assert_eq!(run(USAGE_FILE, bad), Err(usage::USAGE_MALFORMED));
assert_eq!(run("a/wai-usage.json", bad).expect("held").usage, None);
}
#[test]
fn a_pin_target_reads_two_members() {
let p = json!({"id": "x", "sha256": "ab".repeat(32), "determinism": "d", "locations": ["https://x.example/s/"]});
assert_eq!(pin_target(&p), Some(([0xab; 32], None)));
assert_eq!(pin_target(&json!({"sha256": "AB".repeat(32)})), None);
assert_eq!(pin_target(&json!({"sha256": "ab".repeat(32), "form": 3})), None);
}
}

66
wai/wai-rs/src/prior_carriage/privacy.rs Normal file
View file

@ -0,0 +1,66 @@
//! The privacy of held sets (extensions/http-delivery.md §12.5).
//!
//! A sink keeps the sets it holds per top-level site. The Sink Capability Descriptor's
//! `held_sets` member it sends to a site is off unless turned on for that site, and then lists
//! only sets held for that site that the site's content pins and that are not revoked.
use serde_json::{json, Map, Value};
/// A set a sink holds: the top-level site it was fetched for, the capability, its digest
/// (64 lowercase hexadecimal digits), and whether a key log the sink holds revokes it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Held {
pub site: String,
pub capability: String,
pub sha256: String,
pub revoked: bool,
}
/// The `held_sets` member a sink sends to `site`: `None` (no member) unless `enabled` for that
/// site; otherwise, per capability in the order `wanted` first names it, the digests of `wanted`
/// (the pins the site's content carries) that the sink holds for `site` and that no key log it
/// holds revokes, in `wanted`'s order, each once.
pub fn held_sets_for(store: &[Held], site: &str, wanted: &[(String, String)], enabled: bool) -> Option<Value> {
if !enabled {
return None;
}
let mut m = Map::new();
for (cap, d) in wanted {
let held = store.iter().any(|h| h.site == site && h.capability == *cap && h.sha256 == *d && !h.revoked);
if !held {
continue;
}
let list = m.entry(cap.clone()).or_insert_with(|| json!([]));
if let Some(a) = list.as_array_mut() {
if !a.iter().any(|x| x == d) {
a.push(json!(d));
}
}
}
Some(Value::Object(m))
}
/// Whether a sink may name `sha256` in `Available-Dictionary` on a request it sends for `site`
/// (http-delivery §8.8, §12.5): it holds the set for that site and no key log it holds revokes it.
pub fn may_name_dictionary(store: &[Held], site: &str, sha256: &str) -> bool {
store.iter().any(|h| h.site == site && h.sha256 == sha256 && !h.revoked)
}
#[cfg(test)]
mod tests {
use super::*;
fn h(site: &str, cap: &str, d: &str, revoked: bool) -> Held {
Held { site: site.into(), capability: cap.into(), sha256: d.into(), revoked }
}
#[test]
fn per_site_and_only_what_is_pinned() {
let store = [h("a.example", "c1", "11", false), h("b.example", "c1", "22", false), h("a.example", "c1", "33", true)];
let wanted = [("c1".to_string(), "11".to_string()), ("c1".to_string(), "22".to_string()), ("c1".to_string(), "33".to_string())];
assert_eq!(held_sets_for(&store, "a.example", &wanted, false), None);
assert_eq!(held_sets_for(&store, "a.example", &wanted, true), Some(json!({ "c1": ["11"] })));
assert_eq!(held_sets_for(&store, "b.example", &wanted[..1], true), Some(json!({})));
assert!(may_name_dictionary(&store, "a.example", "11") && !may_name_dictionary(&store, "b.example", "11"));
}
}

158
wai/wai-rs/src/prior_carriage/sealed.rs Normal file
View file

@ -0,0 +1,158 @@
//! Sealed sets: `wai.prior.sealed` (extensions/prior-carriage.md §18.2).
//!
//! A one-file set encrypted under a 16-byte AES-128 key that its origin releases to the sinks it
//! entitles. The pin covers the sealed bytes, so any server may serve them and a sink checks them
//! by digest before it holds a key; opened, the plaintext is checked against the SHA-256 the
//! header states. A sink without the key holds bytes it cannot read, which SPEC §3.1 check 6
//! does not hold (`prior_not_held`), and reports why (`set_key_absent`).
use super::usage::{self, Usage};
use crate::codecs::{CAP_PRIOR_GGUF, CAP_PRIOR_SAFETENSORS};
use crate::pin::sha256;
use aes_gcm::aes::cipher::{generic_array::GenericArray, BlockEncrypt, KeyInit};
use aes_gcm::aes::Aes128;
/// The magic a sealed set begins with.
pub const MAGIC: [u8; 4] = *b"WSK1";
/// The longest plaintext: 2^40 bytes.
pub const MAX_PLAINTEXT: u64 = 1 << 40;
pub const SEALED_MALFORMED: &str = "sealed_malformed";
pub const SEALED_INNER_FORM: &str = "sealed_inner_form";
pub const SET_KEY_ABSENT: &str = "set_key_absent";
pub const SET_KEY_MISMATCH: &str = "set_key_mismatch";
/// The inner forms a sealed set admits: a capability's one file (no form), and the two
/// tensor-file forms, each established exact (§5).
pub fn inner_form_admitted(form: &str) -> bool {
form.is_empty() || form == CAP_PRIOR_SAFETENSORS || form == CAP_PRIOR_GGUF
}
/// A sealed set's header, read without its key.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Header {
/// The inner form; empty for a capability's one file.
pub inner_form: String,
pub kid: [u8; 16],
pub iv: [u8; 16],
/// The SHA-256 of the plaintext: the set's digest in its inner form.
pub inner: [u8; 32],
pub usage: Option<Usage>,
/// Where the ciphertext begins, and its length.
pub at: usize,
pub len: u64,
}
/// AES-128 in counter mode (NIST SP 800-38A §6.5): the counter block `iv`, incremented as one
/// 128-bit big-endian integer per 16-byte block. Encryption and decryption are the same.
pub fn ctr(key: &[u8; 16], iv: &[u8; 16], data: &[u8]) -> Vec<u8> {
let cipher = Aes128::new(GenericArray::from_slice(key));
let mut counter = u128::from_be_bytes(*iv);
let mut out = Vec::with_capacity(data.len());
for chunk in data.chunks(16) {
let mut block = GenericArray::clone_from_slice(&counter.to_be_bytes());
cipher.encrypt_block(&mut block);
out.extend(chunk.iter().zip(block.iter()).map(|(a, b)| a ^ b));
counter = counter.wrapping_add(1);
}
out
}
fn le(b: &[u8]) -> u64 {
b.iter().rev().fold(0u64, |a, x| a << 8 | *x as u64)
}
/// §18.2's header checks, in order: the layout (`sealed_malformed`), the inner form
/// (`sealed_inner_form`), the usage statement (`set_usage_malformed`).
pub fn read_header(b: &[u8]) -> Result<Header, &'static str> {
let bad = SEALED_MALFORMED;
if b.len() < 5 || b[..4] != MAGIC {
return Err(bad);
}
let k = b[4] as usize;
if k > 64 || b.len() < 73 + k {
return Err(bad);
}
let form = &b[5..5 + k];
if !form.iter().all(|c| (0x21..=0x7e).contains(c)) {
return Err(bad);
}
let inner_form = String::from_utf8(form.to_vec()).map_err(|_| bad)?;
let kid: [u8; 16] = b[5 + k..21 + k].try_into().map_err(|_| bad)?;
let iv: [u8; 16] = b[21 + k..37 + k].try_into().map_err(|_| bad)?;
let inner: [u8; 32] = b[37 + k..69 + k].try_into().map_err(|_| bad)?;
let u = le(&b[69 + k..73 + k]) as usize;
if u > usage::MAX_USAGE || b.len() < 81 + k + u {
return Err(bad);
}
let len = le(&b[73 + k + u..81 + k + u]);
let at = 81 + k + u;
if len == 0 || len > MAX_PLAINTEXT || (b.len() - at) as u64 != len {
return Err(bad);
}
if !inner_form_admitted(&inner_form) {
return Err(SEALED_INNER_FORM);
}
let usage = if u == 0 { None } else { Some(usage::read(&b[73 + k..73 + k + u])?) };
Ok(Header { inner_form, kid, iv, inner, usage, at, len })
}
/// Opens a sealed set with the key a sink holds for its `kid`, if any: the header and the
/// plaintext, checked against the header's SHA-256; or the reason (`set_key_absent`,
/// `set_key_mismatch`, or a header's).
pub fn open(b: &[u8], key_for: &dyn Fn(&[u8; 16]) -> Option<[u8; 16]>) -> Result<(Header, Vec<u8>), &'static str> {
let h = read_header(b)?;
let key = key_for(&h.kid).ok_or(SET_KEY_ABSENT)?;
let plain = ctr(&key, &h.iv, &b[h.at..]);
if sha256(&plain) != h.inner {
return Err(SET_KEY_MISMATCH);
}
Ok((h, plain))
}
/// Seals `plaintext`, a set in `inner_form`, under `key` (named `kid`) from the counter block
/// `iv`, with the usage statement `usage` (its JSON text) or none.
pub fn seal(plaintext: &[u8], inner_form: &str, kid: &[u8; 16], key: &[u8; 16], iv: &[u8; 16], usage: Option<&[u8]>) -> Vec<u8> {
let u = usage.unwrap_or(&[]);
let mut out = MAGIC.to_vec();
out.push(inner_form.len() as u8);
out.extend_from_slice(inner_form.as_bytes());
out.extend_from_slice(kid);
out.extend_from_slice(iv);
out.extend_from_slice(&sha256(plaintext));
out.extend_from_slice(&(u.len() as u32).to_le_bytes());
out.extend_from_slice(u);
out.extend_from_slice(&(plaintext.len() as u64).to_le_bytes());
out.extend_from_slice(&ctr(key, iv, plaintext));
out
}
#[cfg(test)]
mod tests {
use super::*;
/// NIST SP 800-38A F.5.1, CTR-AES128.Encrypt, its first two blocks.
#[test]
fn counter_mode_matches_sp_800_38a() {
let h = |s: &str| (0..s.len()).step_by(2).map(|i| u8::from_str_radix(&s[i..i + 2], 16).expect("hex")).collect::<Vec<u8>>();
let key: [u8; 16] = h("2b7e151628aed2a6abf7158809cf4f3c").try_into().expect("16");
let iv: [u8; 16] = h("f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff").try_into().expect("16");
let plain = h("6bc1bee22e409f96e93d7e117393172aae2d8a571e03ac9c9eb76fac45af8e51");
assert_eq!(ctr(&key, &iv, &plain), h("874d6191b620e3261bef6864990db6ce9806f66b7970fdff8617187bb9fffdff"));
}
#[test]
fn sealed_and_opened() {
let plain = b"a one-file parameter set".to_vec();
let (kid, key, iv) = ([1u8; 16], [2u8; 16], [3u8; 16]);
let s = seal(&plain, "", &kid, &key, &iv, None);
let (h, got) = open(&s, &|k| (*k == kid).then_some(key)).expect("opens");
assert_eq!((got, h.inner_form.as_str()), (plain.clone(), ""));
assert_eq!(open(&s, &|_| None).map(|_| ()), Err(SET_KEY_ABSENT));
assert_eq!(open(&s, &|_| Some([9u8; 16])).map(|_| ()), Err(SET_KEY_MISMATCH));
for inner in ["wai.prior.nnc", "wai.prior.set", "wai.prior.sealed", "wai.prior.tree", "x.unregistered"] {
assert_eq!(read_header(&seal(&plain, inner, &kid, &key, &iv, None)), Err(SEALED_INNER_FORM), "{inner}");
}
assert_eq!(read_header(&s[..s.len() - 1]), Err(SEALED_MALFORMED));
}
}

253
wai/wai-rs/src/prior_carriage/tree.rs Normal file
View file

@ -0,0 +1,253 @@
//! The chunk-tree form `wai.prior.tree` (extensions/prior-carriage.md §19).
//!
//! A tree head (`WTR1`) names a one-file set by its length, its SHA-256 and its BLAKE3 hash,
//! and lists the BLAKE3 chaining value of each of its pieces: runs of 2^g of BLAKE3's 1024-byte
//! chunks, each a subtree of BLAKE3's tree over the whole set. The pin covers the head, so a
//! sink verifies each piece as it arrives, from any peer, and refuses a changed piece at that
//! piece; the pieces' values give the root, which is the set's BLAKE3 hash.
use super::sealed;
use crate::codecs::CAP_PRIOR_SEALED;
use crate::pin::sha256;
use blake3::hazmat::{merge_subtrees_non_root, merge_subtrees_root, ChainingValue, HasherExt, Mode};
/// The magic a tree head begins with.
pub const MAGIC: [u8; 4] = *b"WTR1";
/// The largest piece exponent: a piece of 64 MiB.
pub const MAX_G: u8 = 16;
/// The most pieces a head lists.
pub const MAX_PIECES: u64 = 65536;
pub const TREE_HEAD_MALFORMED: &str = "tree_head_malformed";
pub const TREE_INNER_FORM: &str = "tree_inner_form";
pub const TREE_ROOT_MISMATCH: &str = "tree_root_mismatch";
pub const TREE_PIECE_MISMATCH: &str = "tree_piece_mismatch";
pub const TREE_DIGEST_MISMATCH: &str = "tree_digest_mismatch";
/// The inner forms a tree admits: a sealed set's, and a sealed set.
pub fn inner_form_admitted(form: &str) -> bool {
sealed::inner_form_admitted(form) || form == CAP_PRIOR_SEALED
}
/// A tree head, read.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Head {
pub g: u8,
pub inner_form: String,
pub len: u64,
/// The SHA-256 of the bytes: the set's digest in its inner form.
pub inner: [u8; 32],
/// The BLAKE3 hash of the bytes.
pub root: [u8; 32],
pub values: Vec<[u8; 32]>,
}
impl Head {
/// The length of a piece.
pub fn piece_len(&self) -> u64 {
1024u64 << self.g
}
/// Piece `i`'s byte range, `[start, end)`.
pub fn range(&self, i: usize) -> (u64, u64) {
let p = self.piece_len();
(i as u64 * p, ((i as u64 + 1) * p).min(self.len))
}
}
/// The chaining value of `bytes` as the subtree of BLAKE3's tree that begins at byte `start`.
fn subtree_cv(bytes: &[u8], start: u64) -> ChainingValue {
let mut h = blake3::Hasher::new();
h.set_input_offset(start);
h.update(bytes);
h.finalize_non_root()
}
/// The chaining value of pieces `v[..]` merged as BLAKE3 merges subtrees: the left subtree the
/// largest power of two of pieces below their count.
fn merged(v: &[[u8; 32]]) -> ChainingValue {
if v.len() == 1 {
return v[0];
}
let l = 1usize << (usize::BITS - 1 - (v.len() - 1).leading_zeros());
merge_subtrees_non_root(&merged(&v[..l]), &merged(&v[l..]), Mode::Hash)
}
/// The root the values give: BLAKE3's root node over them, at least two.
pub fn root_of(v: &[[u8; 32]]) -> [u8; 32] {
let l = 1usize << (usize::BITS - 1 - (v.len() - 1).leading_zeros());
*merge_subtrees_root(&merged(&v[..l]), &merged(&v[l..]), Mode::Hash).as_bytes()
}
fn le(b: &[u8]) -> u64 {
b.iter().rev().fold(0u64, |a, x| a << 8 | *x as u64)
}
/// §19's head checks, in order: the layout (`tree_head_malformed`), the inner form
/// (`tree_inner_form`), and the root the values give (`tree_root_mismatch`).
pub fn read_head(b: &[u8]) -> Result<Head, &'static str> {
let bad = TREE_HEAD_MALFORMED;
if b.len() < 6 || b[..4] != MAGIC {
return Err(bad);
}
let (g, k) = (b[4], b[5] as usize);
if g > MAX_G || k > 64 || b.len() < 82 + k {
return Err(bad);
}
let form = &b[6..6 + k];
if !form.iter().all(|c| (0x21..=0x7e).contains(c)) {
return Err(bad);
}
let inner_form = String::from_utf8(form.to_vec()).map_err(|_| bad)?;
let len = le(&b[6 + k..14 + k]);
let inner: [u8; 32] = b[14 + k..46 + k].try_into().map_err(|_| bad)?;
let root: [u8; 32] = b[46 + k..78 + k].try_into().map_err(|_| bad)?;
let n = le(&b[78 + k..82 + k]);
let p = 1024u64 << g;
if len == 0 || n != len.div_ceil(p) || !(2..=MAX_PIECES).contains(&n) || b.len() as u64 != 82 + k as u64 + 32 * n {
return Err(bad);
}
if !inner_form_admitted(&inner_form) {
return Err(TREE_INNER_FORM);
}
let values: Vec<[u8; 32]> = b[82 + k..].chunks(32).map(|c| c.try_into().expect("32 bytes")).collect();
if root_of(&values) != root {
return Err(TREE_ROOT_MISMATCH);
}
Ok(Head { g, inner_form, len, inner, root, values })
}
/// The head of `bytes`, a set in `inner_form`, with pieces of 1024·2^`g` bytes.
pub fn write_head(bytes: &[u8], inner_form: &str, g: u8) -> Vec<u8> {
let p = 1024usize << g;
let values: Vec<[u8; 32]> = bytes.chunks(p).enumerate().map(|(i, c)| subtree_cv(c, (i * p) as u64)).collect();
let mut out = MAGIC.to_vec();
out.push(g);
out.push(inner_form.len() as u8);
out.extend_from_slice(inner_form.as_bytes());
out.extend_from_slice(&(bytes.len() as u64).to_le_bytes());
out.extend_from_slice(&sha256(bytes));
out.extend_from_slice(blake3::hash(bytes).as_bytes());
out.extend_from_slice(&(values.len() as u32).to_le_bytes());
for v in &values {
out.extend_from_slice(v);
}
out
}
/// Whether `piece` is piece `i` of the set `head` names.
pub fn piece_verifies(head: &Head, i: usize, piece: &[u8]) -> bool {
let (s, e) = head.range(i);
piece.len() as u64 == e - s && subtree_cv(piece, s) == head.values[i]
}
/// A source's answer to a request for the bytes `[start, end)`: its content coding, if any,
/// and the bytes it sends.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Range {
pub content_encoding: Option<String>,
pub body: Vec<u8>,
}
/// A source: its name, and what it answers for a range (`None`: no answer).
pub type Source<'a> = (&'a str, &'a dyn Fn(u64, u64) -> Option<Range>);
/// Each piece asked, from which source, and what came of it; then the set's bytes or the
/// reason it is not held.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TreeFetch {
pub attempts: Vec<(String, usize, &'static str)>,
pub result: Result<Vec<u8>, &'static str>,
}
/// Fetches every piece the head names, piece `i` first from source `i mod m` and then from the
/// sources after it, verifying each as it arrives (§19); then requires the whole set's SHA-256
/// to be the head's `inner`.
pub fn fetch_pieces(head: &Head, sources: &[Source<'_>]) -> TreeFetch {
let mut attempts = Vec::new();
let mut bytes = Vec::with_capacity(head.len as usize);
for i in 0..head.values.len() {
let (s, e) = head.range(i);
let mut got = None;
for j in 0..sources.len() {
let (name, serve) = sources[(i + j) % sources.len()];
let outcome = match serve(s, e) {
None => "set_location_unavailable",
Some(r) if r.content_encoding.is_some() => "content_coding_present",
Some(r) if r.body.len() as u64 != e - s => "content_range_mismatch",
Some(r) if !piece_verifies(head, i, &r.body) => TREE_PIECE_MISMATCH,
Some(r) => {
got = Some(r.body);
"held"
}
};
attempts.push((name.to_string(), i, outcome));
if got.is_some() {
break;
}
}
match got {
Some(b) => bytes.extend_from_slice(&b),
None => return TreeFetch { attempts, result: Err("prior_not_held") },
}
}
let result = if sha256(&bytes) == head.inner { Ok(bytes) } else { Err(TREE_DIGEST_MISMATCH) };
TreeFetch { attempts, result }
}
#[cfg(test)]
mod tests {
use super::*;
fn data(n: usize) -> Vec<u8> {
(0..n as u32).map(|i| (i.wrapping_mul(2654435761) >> 13) as u8).collect()
}
#[test]
fn the_root_is_the_blake3_hash_for_every_length_and_piece_size() {
for g in 0..3u8 {
for n in [1025usize, 2048, 2049, 3 * 1024 + 7, 5 * 1024, 9 * 1024 + 1, 33 * 1024 - 1] {
if n.div_ceil(1024 << g) < 2 {
continue;
}
let d = data(n);
let h = read_head(&write_head(&d, "", g)).expect("reads");
assert_eq!(h.root, *blake3::hash(&d).as_bytes(), "{g} {n}");
for i in 0..h.values.len() {
let (s, e) = h.range(i);
assert!(piece_verifies(&h, i, &d[s as usize..e as usize]));
}
}
}
}
#[test]
fn a_flipped_piece_is_refused_at_that_piece() {
let d = data(5000);
let head = read_head(&write_head(&d, "", 0)).expect("reads");
let mut bad = d.clone();
bad[3 * 1024 + 5] ^= 1;
let good = |s: u64, e: u64| Some(Range { content_encoding: None, body: d[s as usize..e as usize].to_vec() });
let flipped = |s: u64, e: u64| Some(Range { content_encoding: None, body: bad[s as usize..e as usize].to_vec() });
let f = fetch_pieces(&head, &[("a", &good), ("b", &flipped)]);
assert_eq!(f.result.as_deref(), Ok(&d[..]));
let refused: Vec<_> = f.attempts.iter().filter(|a| a.2 != "held").collect();
assert_eq!(refused, [&("b".to_string(), 3, TREE_PIECE_MISMATCH)]);
}
#[test]
fn heads_that_do_not_read() {
let d = data(5000);
let mut h = write_head(&d, "", 0);
assert!(read_head(&write_head(&data(900), "", 0)).is_err());
for inner in ["wai.prior.nnc", "wai.prior.set", "wai.prior.tree", "x.unregistered"] {
assert_eq!(read_head(&write_head(&d, inner, 0)), Err(TREE_INNER_FORM), "{inner}");
}
for inner in ["", "wai.prior.safetensors", "wai.prior.gguf", "wai.prior.sealed"] {
assert!(read_head(&write_head(&d, inner, 0)).is_ok(), "{inner}");
}
let last = h.len() - 1;
h[last] ^= 1;
assert_eq!(read_head(&h), Err(TREE_ROOT_MISMATCH));
assert_eq!(read_head(&h[..last]), Err(TREE_HEAD_MALFORMED));
}
}

197
wai/wai-rs/src/prior_carriage/usage.rs Normal file
View file

@ -0,0 +1,197 @@
//! A parameter set's usage statement (extensions/prior-carriage.md §18.1).
//!
//! A JSON object of exactly the members `kind` (`wai-set-usage`), `licence` (an SPDX licence
//! expression or `null`), `terms` (an `https` URI or `null`), `content_derived`,
//! `redistribute` and `ai_usage` (an [ai-usage](crate::ai_usage) statement or `null`). It is
//! bound to a set only as part of the bytes the set's pin covers: the file `wai-usage.json` of a
//! `wai.prior.set`, or the statement in a sealed set's header.
use crate::ai_usage::Statement as AiUsage;
use serde_json::Value;
/// The reason a statement that does not read is refused with.
pub const USAGE_MALFORMED: &str = "set_usage_malformed";
/// The file of a `wai.prior.set` that holds its statement.
pub const USAGE_FILE: &str = "wai-usage.json";
/// The longest statement, in bytes.
pub const MAX_USAGE: usize = 65536;
/// A statement as read.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Usage {
pub licence: Option<String>,
pub terms: Option<String>,
pub content_derived: bool,
pub redistribute: bool,
/// The ai-usage statement's digest (ai-usage §3.6: SHA-256 over its canonical form).
pub ai_usage: Option<[u8; 32]>,
}
/// One token of an SPDX licence expression.
fn tokens(s: &str) -> Option<Vec<&str>> {
let mut out = Vec::new();
for word in s.split(' ') {
if word.is_empty() {
return None;
}
let mut rest = word;
while let Some(r) = rest.strip_prefix('(') {
out.push("(");
rest = r;
}
let mut closes = 0;
while let Some(r) = rest.strip_suffix(')') {
closes += 1;
rest = r;
}
if !rest.is_empty() {
out.push(rest);
}
out.extend(std::iter::repeat_n(")", closes));
}
Some(out)
}
fn idstring(s: &str) -> bool {
!s.is_empty() && s.bytes().all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'.')
}
/// A simple expression: a licence identifier, with `+` or not, or a `LicenseRef-`, with a
/// `DocumentRef-…:` before it or not.
fn simple(t: &str) -> bool {
if ["AND", "OR", "WITH", "(", ")"].contains(&t) {
return false;
}
let t = match t.split_once(':') {
Some((doc, r)) => match doc.strip_prefix("DocumentRef-") {
Some(d) if idstring(d) => {
return r.strip_prefix("LicenseRef-").is_some_and(idstring);
}
_ => return false,
},
None => t,
};
if let Some(r) = t.strip_prefix("LicenseRef-") {
return idstring(r);
}
idstring(t.strip_suffix('+').unwrap_or(t))
}
/// Whether `s` is an SPDX licence expression (SPDX 2.3 Annex D): simple expressions joined by
/// `AND` and `OR`, each with `WITH` and an exception identifier or not, in parentheses or not,
/// tokens separated by single spaces. Identifiers are checked for their characters only.
pub fn is_licence_expression(s: &str) -> bool {
fn expr(t: &[&str], i: &mut usize, depth: usize) -> bool {
if depth > 32 {
return false;
}
loop {
match t.get(*i) {
Some(&"(") => {
*i += 1;
if !expr(t, i, depth + 1) || t.get(*i) != Some(&")") {
return false;
}
*i += 1;
}
Some(x) if simple(x) => {
*i += 1;
if t.get(*i) == Some(&"WITH") {
*i += 1;
match t.get(*i) {
Some(e) if idstring(e) && !["AND", "OR", "WITH"].contains(e) => *i += 1,
_ => return false,
}
}
}
_ => return false,
}
match t.get(*i) {
Some(&"AND") | Some(&"OR") => *i += 1,
_ => return true,
}
}
}
if s.is_empty() || s.len() > 1024 {
return false;
}
let Some(t) = tokens(s) else { return false };
let mut i = 0;
expr(&t, &mut i, 0) && i == t.len()
}
fn terms_ok(s: &str) -> bool {
(8..=2048).contains(&s.len()) && s.starts_with("https://") && s.bytes().all(|b| (0x21..=0x7e).contains(&b))
}
/// §18.1's statement from its JSON text: no member named twice, exactly the members above, each
/// of its type and within its range; or [`USAGE_MALFORMED`].
pub fn read(bytes: &[u8]) -> Result<Usage, &'static str> {
let bad = USAGE_MALFORMED;
if bytes.len() > MAX_USAGE {
return Err(bad);
}
let text = std::str::from_utf8(bytes).map_err(|_| bad)?;
let v = crate::claim_text::read(text).map_err(|_| bad)?;
let m = v.as_object().ok_or(bad)?;
let names = ["kind", "licence", "terms", "content_derived", "redistribute", "ai_usage"];
if m.len() != names.len() || !names.iter().all(|n| m.contains_key(*n)) {
return Err(bad);
}
if m["kind"] != Value::String("wai-set-usage".into()) {
return Err(bad);
}
let licence = match &m["licence"] {
Value::Null => None,
Value::String(s) if is_licence_expression(s) => Some(s.clone()),
_ => return Err(bad),
};
let terms = match &m["terms"] {
Value::Null => None,
Value::String(s) if terms_ok(s) => Some(s.clone()),
_ => return Err(bad),
};
let (Some(content_derived), Some(redistribute)) = (m["content_derived"].as_bool(), m["redistribute"].as_bool()) else {
return Err(bad);
};
let ai_usage = match &m["ai_usage"] {
Value::Null => None,
a => Some(AiUsage::read(a).map_err(|_| bad)?.digest()),
};
Ok(Usage { licence, terms, content_derived, redistribute, ai_usage })
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn licence_expressions() {
for ok in [
"Apache-2.0",
"MIT OR Apache-2.0",
"(MIT OR Apache-2.0) AND CC-BY-4.0",
"GPL-2.0-or-later WITH Classpath-exception-2.0",
"LicenseRef-studio-catalog-1",
"DocumentRef-terms:LicenseRef-x",
"GPL-2.0+",
"((MIT))",
] {
assert!(is_licence_expression(ok), "{ok}");
}
for bad in ["", "MIT OR", "MIT OR Apache-2.0", "(MIT", "MIT)", "MIT and Apache-2.0", "WITH", "MIT WITH", "a/b", "MIT WITH x+"] {
assert!(!is_licence_expression(bad), "{bad}");
}
}
#[test]
fn statements() {
let s = br#"{"kind":"wai-set-usage","licence":"Apache-2.0","terms":null,"content_derived":false,"redistribute":true,"ai_usage":null}"#;
let u = read(s).expect("reads");
assert!(u.redistribute && !u.content_derived);
let twice = br#"{"kind":"wai-set-usage","kind":"wai-set-usage","licence":null,"terms":null,"content_derived":false,"redistribute":true,"ai_usage":null}"#;
assert_eq!(read(twice), Err(USAGE_MALFORMED));
let extra = br#"{"kind":"wai-set-usage","licence":null,"terms":null,"content_derived":false,"redistribute":true,"ai_usage":null,"x":1}"#;
assert_eq!(read(extra), Err(USAGE_MALFORMED));
}
}

60
wai/wai-rs/tests/prior_carriage.rs Normal file
View file

@ -0,0 +1,60 @@
//! Prior carriage (extensions/prior-carriage.md §15, §16; extensions/http-delivery.md §8.7, §8.8): the
//! committed corpus `live-conformance/prior-carriage/` regenerates byte for byte from the sets
//! other corpora commit, holds no other file, every case in it gets the outcome it states, and
//! http-delivery §13 lists every reason the reference reports for §8.7 and §8.8.
#![cfg(feature = "prior_carriage")]
use std::path::PathBuf;
use wai::prior_carriage::{corpus, CODES, HTTP_CODES};
fn wai_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("..")
}
fn corpus_dir() -> PathBuf {
wai_dir().join("live-conformance/prior-carriage")
}
#[test]
fn the_committed_corpus_regenerates() {
let files = corpus::committed(&corpus_dir()).expect("the committed corpus");
let generated = corpus::generate(&corpus_dir()).expect("the corpus generates");
let extra: Vec<&String> = files.keys().filter(|k| !generated.contains_key(*k)).collect();
let missing: Vec<&String> = generated.keys().filter(|k| !files.contains_key(*k)).collect();
let differ: Vec<&String> = generated.iter().filter(|(k, v)| files.get(*k).is_some_and(|f| f != *v)).map(|(k, _)| k).collect();
assert!(extra.is_empty() && missing.is_empty() && differ.is_empty(), "extra {extra:?}, missing {missing:?}, differ {differ:?}: run `wai_prior_carriage_vectors gen`");
}
#[test]
fn every_case_gets_its_stated_outcome() {
let files = corpus::committed(&corpus_dir()).expect("the committed corpus");
let fails = corpus::verify(&corpus_dir(), &files);
assert!(fails.is_empty(), "{}", fails.join("\n"));
}
/// The rows http-delivery §13 adds for §8.7 and §8.8 name exactly `HTTP_CODES`, in order.
#[test]
fn http_delivery_lists_the_reasons() {
let text = std::fs::read_to_string(wai_dir().join("extensions/http-delivery.md")).expect("the extension");
let at = text.find("\n## 13. Reasons").expect("a §13");
let end = text[at + 1..].find("\n## ").map_or(text.len(), |e| at + 1 + e);
let listed: Vec<String> = text[at..end]
.lines()
.filter(|l| l.contains("| §8.7") || l.contains("| §8.8"))
.flat_map(|l| l.split('|').nth(1).unwrap_or("").split('`').skip(1).step_by(2).map(String::from).collect::<Vec<_>>())
.collect();
let want: Vec<String> = HTTP_CODES.iter().map(|c| c.to_string()).collect();
assert_eq!(listed, want, "§13's rows for §8.7 and §8.8 and HTTP_CODES differ");
}
/// prior-carriage §20 lists exactly `CODES`, each once, in order.
#[test]
fn prior_carriage_lists_the_reasons() {
let text = std::fs::read_to_string(wai_dir().join("extensions/prior-carriage.md")).expect("the extension");
let at = text.find("\n## 20. Refusals").expect("a §20");
let end = text[at + 1..].find("\n## ").map_or(text.len(), |e| at + 1 + e);
let listed: Vec<String> =
text[at..end].lines().filter(|l| l.starts_with("| `")).filter_map(|l| l.split('`').nth(1).map(String::from)).collect();
let want: Vec<String> = CODES.iter().map(|c| c.to_string()).collect();
assert_eq!(listed, want, "§20's table and CODES differ");
}