open-standards/wai/live-conformance/prior-carriage/README.md

# live-conformance/prior-carriage — where a parameter set is fetched, how it is delivered, when it is worth holding, who may read it, and how it verifies by range

The corpus of [`extensions/prior-carriage.md`](../../extensions/prior-carriage.md) §15 to §19
and [`extensions/http-delivery.md`](../../extensions/http-delivery.md) §8.7, §8.8 and §12.5:
`vectors.json` and the files under `files/`.

**Provenance.** The parameter sets are files other corpora commit, named by their paths from
this directory and pinned by their SHA-256: the trained `WSR1` priors `generic` and `title` in
the safetensors form of [`../prior-forms/`](../prior-forms/) (`title` was fine-tuned from
`generic`, `seeded` is a seeded prior of the same shape), and the listing `demo` of
[`../../pin-conformance/`](../../pin-conformance/) with the three stand-in files it names. The
hosts are `.example` names (RFC 2606); nothing here was fetched from a server.

`vectors.json` holds:

- `sets`: each set the cases read, with its path, SHA-256 and length.
- `roots`: strings, and whether each is a set root (§8.7: an absolute `https` URI with a host,
  no userinfo, query or fragment, whose path ends in `/`, at most 2048 bytes).
- `hints`: a `wai_priors` entry (moq-streaming-format §4.2) or a WAI multivariant playlist, and
  the hint read from it: the roots kept, in order, and the positions passed over as
  `set_location_malformed` (in a playlist, among the elements of the hint line's `VALUE` split
  at every space). Some playlists also carry lines a reader ignores: a second line of the
  hint's `DATA-ID`, one with `LANGUAGE`, and one whose attribute list does not read.
- `fetches`: a pin, the `wai_priors` entry that states its hint, the roots of the sink's own
  configuration, the origin, the sink's bound on one response's bytes, and the responses the
  servers give, keyed by URL (a URL with no response is a server that does not answer); and
  the outcome: the roots in the order the sink asks them, each URL asked with what came of it
  (`held` or a reason of §8.7 or §8.8), the result, `held` or the reason SPEC §3.1 refuses the pin
  with (or `set_usage_malformed`, prior-carriage §18.1), and `usage`, the usage statement a held
  listing binds as its file `wai-usage.json` (its licence, terms, the two booleans and the digest
  of its ai-usage statement), `null` where it binds none. A body is a set's bytes (`set`),
  those bytes with one byte's lowest bit flipped (`flip`), a file of `files/` (`file`, with
  `flip` or `truncate`, the first bytes kept), or bytes given in hexadecimal (`hex`). A response is keyed by its URL and the
  `Available-Dictionary` value of the request it answers (`null` for a request that names no
  dictionary); a fetch's `dictionary` is the set the sink holds and names (§8.8).
- `fields`: `Available-Dictionary` values and the digest each names (`null` where it does not
  read as one), and the `Use-As-Dictionary` value for a root's path.
- `stock`: the check a stock client makes of §8.8: `files/title-from-generic.dcz` after its
  40-byte header, decoded by the Zstandard tool against `generic`, has `title`'s SHA-256:

      tail -c +41 files/title-from-generic.dcz | zstd -d -D ../prior-forms/files/generic.safetensors | sha256sum

- `plans`: §17's worked plans, each a plan (a span, a cap, the generations in epoch order with
  their sets' bytes and any coding against an earlier one, the titles with their pinned and
  classical bytes, and one or more sinks with what each holds and its fetches per title) and
  the outcome worked by hand: per sink, each generation's saving, cost (`held`, `delta` or
  `set`) and net, the generations kept, fetched and evicted, each title's rendition, and the
  bytes received beside the bytes with no set; or `plan_malformed`. The sizes are worked
  numbers, not measurements.

- `usage`: §18.1's usage statements as JSON text, and what reading each gives (its licence,
  terms, the two booleans and the digest of its ai-usage statement) or `set_usage_malformed`.
- `sealed`: §18.2's sealed sets (`files/title.wsk1`, `files/generic.wsk1`, or a header written
  in hexadecimal), the keys the sink holds by their identifiers, and what opening gives (the
  inner form, the plaintext's digest and the usage statement) or the reason. The corpus's key
  (16 bytes of `0x6b`, identifier 16 bytes of `0x4b`, counter block 16 bytes of `0x69`) is
  published here and protects nothing.
- `held_sets`: http-delivery §12.5's `held_sets` from a store of sets held per site, the site
  it is sent to, the pins the site's content carries, and whether it is turned on; and
  `dictionary_naming`, whether a sink may name a held set in `Available-Dictionary` for a site.

- `tree_heads`: §19's heads (`files/generic.wtr1` in pieces of one chunk, `files/generic-g1.wtr1`
  in pieces of two, `files/title-sealed.wtr1` over `files/title.wsk1`, or a head written in
  hexadecimal) and what reading each gives (the inner form, the set's length, the piece exponent,
  the number of pieces, the set's SHA-256 and its BLAKE3 hash) or the reason.
- `tree_fetches`: a head, sources (each named, with the bytes it serves, whether it answers, and
  any content coding), and the outcome of fetching every piece by range: each piece asked of
  which source with what came of it, piece `i` first from source `i mod m` and then from the
  sources after it; then `held` and the set's SHA-256, or the reason.

`files/title-from-generic.dcz` is `title` coded against `generic` by the Zstandard library at
level 19, its content size and checksum stated, under the `dcz` header naming `generic`.
`files/title-window-16m.dcz` holds `title` in one raw block of a frame that states a 16 MB
window and no content size, under the same header: a frame RFC 9842 has no client decode
against a dictionary of `generic`'s size.

A verifier conforms when it reaches every case's outcome, field for field.

**Verifiers.**

- `wai_prior_carriage_vectors verify live-conformance/prior-carriage` (wai-rs, feature
  `prior_carriage`): every file regenerates byte for byte, no other file is here, and the
  module reaches every outcome. `wai_prior_carriage_vectors gen` writes the corpus.
- `python3 -I live-conformance/prior_carriage_verify.py live-conformance/prior-carriage`,
  written from the texts and sharing no code with wai-rs. It decodes Zstandard with
  `_rfc8878_zstd.py`, written from RFC 8878 beside it, hashes with the shared
  `../../claim-conformance/_blake3.py`, written from the BLAKE3 specification, opens sealed sets with
  `_fips197_aes.py`, written from FIPS 197 and NIST SP 800-38A, reads ai-usage statements with
  the ai-usage corpus's second implementation (`ai_usage_verify.py`), and, where the Zstandard tool is on the
  PATH, runs the `stock` check with that tool too. It also mutates copies of the
  corpus, one defect each, and requires every copy to fail, and drops each of its own checks
  in turn and requires the corpus to catch the weakened verifier.