wai-pack: staged packages — refinement layers, stills, a piece's ladder, live windows and simulated delivery (W4-4) #199

Merged

dcharlot merged 27 commits from wai/w4-4-staged-packages into main 2026-10-11 16:58:55 -04:00

Conversation 0 Commits 27 Files changed 279 +10588 -668

2026-10-11

Transaction Science

Merge wai/d1-record-true (the wasip2 link fix, with #215, #216, #217 and main) into wai/w4-4-staged-packages ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

02cb302e33

• View at this point in history

2026-10-11 08:38:05 -04:00

Transaction Science

Merge wai/d1-record-true (the record leg runs the record's own demo scripts) into wai/w4-4-staged-packages ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

d4eb15e7ce

• View at this point in history

2026-10-11 03:48:06 -04:00

2026-10-10

Transaction Science

Merge wai/d1-record-true (with #205, #197 and main) into wai/w4-4-staged-packages ...

Brings in #201, #205 and #197 and, through them, main as of #200 and #206.

- .github/workflows/wai-conformance.yml: both sides added a step after
  sim_verify.py. Kept both: this branch's sim_diff.py differential, then
  main's quantum referee.py. wai-pack.yml merged without conflict: this
  branch's test-count floors (--lib 34, --test=staged 10) and differential
  steps, and main's cargo deny step for the wai binary's TLS client.
- wai/CHANGELOG.md: both sides added entries at the top of v0.4.0. Kept
  every entry, newest first: this branch's "Packaging: refinement layers,
  stills, ...", then #201's, #205's and #197's, then main's unchanged.
- wai/http-conformance/cases.json: regenerated with wai_http_vectors gen;
  the generator's output equals the merged text (1051 cases, with
  union_O10_repeated_layer_after_first and _after_second), and both
  wai_http_vectors verify and http_verify.py give every outcome.
- falsify/union-no-layer.patch, regenerated on this branch, applies to the
  merged codecs.rs unchanged and names the two repeated-layer cases among
  its must-fail cases; it is patches 0/3 of the sharded runner. This
  branch adds no http_verify.py mutation, so mutations.txt is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

e1acae7881

• View at this point in history

2026-10-10 17:40:02 -04:00

Transaction Science

http falsification: union-no-layer against this branch's described() ...

The union-no-layer patch was written against the described() that this branch
rewrote in round 2 (a repeated object described once, a layer joining every
occurrence of its base), so CI could not apply it ("patch failed: codecs.rs:326").

It is regenerated from the branch's codecs.rs, still a plain patch file: the
layer joins no alt. Its must-fail list adds the two cases that pin the
repeated-object rule: union_O10_repeated_layer_after_first and
union_O10_repeated_layer_after_second. With the patch applied, verify fails all
three and two multivariant playlists.

The receipt-surrogates mutation is left to #200's sharded runner, which owns it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

409b1c1fb2

• View at this point in history

2026-10-10 01:21:31 -04:00

2026-10-09

Transaction Science

wai-pack: the inflater's bit reader aligns, by name (clippy's conversion rule) ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

9ab59f2c96

• View at this point in history

2026-10-09 16:03:42 -04:00

Transaction Science

Merge origin/main into wai/w4-4-staged-packages ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

5072ce77af

• View at this point in history

2026-10-09 15:41:51 -04:00

Transaction Science

changelog: W4-4 round 3 — zlib's code-set rules, the longest piece, manifests in both verifiers, live bookkeeping ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

af0572a73b

• View at this point in history

2026-10-09 15:40:48 -04:00

Transaction Science

ci: wai-pack's test-count floors at its counts (lib 34, staged 10) ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

42bfce677f

• View at this point in history

2026-10-09 15:39:55 -04:00

Transaction Science

wai-pack live keeps no record per file: listing lines on disk, open objects only ...

LiveOut held every object's path, hex SHA-256 and first piece for the whole
session. It now holds:

- no listing record per file. Each file's listing line is appended to a hidden
  .LISTING.lines beside the package as the file is written. At the end the lines
  are read back, sorted into LISTING, and the lines file is removed; that is the
  one moment it holds a line per file.
- no set of written objects. An object already written is known by its file on
  disk.
- for the abandon path, only the objects the unsealed pieces published, as a
  piece number and a 32-byte content hash. Those of a window are dropped when
  the window is sealed.

packaging §9.2 and the CLI help now say exactly what remains:

- what no window has sealed yet: those pieces' entry fields and their objects'
  hashes, which with --window 0 means every piece's until the end;
- the frames of the piece being cut;
- eight bytes per piece for the plan.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

e6a16c0408

• View at this point in history

2026-10-09 15:28:03 -04:00

Transaction Science

wai-pack: both verifiers compare manifests with §5.2/§5.4 (object-manifest); an unreadable still is prefix-digest ...

The Rust verifier never compared a base's manifest with §5.2, while
pack_verify.py did, under a refusal that named no §11.3 reason. A still whose
manifest had one extra target member was ok in Rust and refused in Python.
Python's refusal of a base it cannot decode named no reason either.

§11.3 now states one order, and both verifiers follow it. They take every base
and refinement in entry order:

- a base decodes by its capability, or it is prefix-digest;
- its manifest is §5.2's for its piece's frames and frame size
  (object-manifest, a new §11.3 reason);
- a refinement's manifest is §5.4's (object-manifest);
- each layer applies to every frame of the stage below, or it is prefix-digest;
- a base that does not decode, or a layer that does not apply, ends the pass;
- then the decode-equivalent prefixes are compared with their digests.

Rust's prefix_digests was restructured into this pass.

The new staged mutation cases are:

- still-manifest-target (object-manifest);
- refinement-manifest-target (object-manifest);
- still-incomplete-dist: the still re-coded with a lone two-bit distance code
  (prefix-digest).

The reviewer's still-png-variant and still-manifest-variant packages now get
those reasons from both verifiers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

d913dcb622

• View at this point in history

2026-10-09 14:45:44 -04:00

Transaction Science

packaging §4.4: a piece is at most ten minutes and 2^24 units; floors counted before decoding ...

The new max_ms check bounded nothing, because a policy could set max_ms up to
2^53 − 1. A still's piece could then state 2^32 − 1 frames. Rust hashed once
per stated frame (about 29 minutes), and Python built a list of n references
(about 34 GB).

- **§3.4 item 5 and §4.4.** max_ms is at most 600000, and at most 2^24 units on
  every component's timebase. Both are policy-piece-bounds, in both readers.
  §4.4 gives the reason: a verifier takes a piece whole, so the longest piece
  bounds its work. Ten minutes is far past any piece a delivery cuts, and 2^24
  units bounds the work on any timebase, which may have up to 2^32 − 1 units a
  second.
- **§11.3, before any frame is decoded.** Each piece's floor object, where its
  framing reads, must hold one unit per frame of its piece
  (floor-object-mismatch). Both verifiers make this check first.
- **Held stills.** Python holds a still's frame and its count (Held) and hashes
  the digest a block at a time. Rust's held_digest also feeds a block at a time.
- **Corpus.**
  - refusals: max_ms 600001; and the default max_ms on a timebase of 2^32 − 1
    units a second.
  - policy/max-piece.toml: accepted at both bounds.
  - staged-mutations max-ms-past-cap (policy-piece-bounds).
  - staged-mutations still-at-cap: the still given the longest piece the policy
    allows, its manifest naming the frames, refused by the floor count in both
    verifiers.
- **policy_diff.** Its random policies now reach both bounds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

e3375530f6

• View at this point in history

2026-10-09 14:13:41 -04:00

Transaction Science

wai-pack: a still's reader inflates as zlib does; png_diff runs both readers ...

The reader used miniz_oxide, which accepts a dynamic block whose lone distance
code is longer than one bit. zlib's inflate, and so pack_verify.py, refuses
that block. A golden resealed with such a still verified in wai-pack and was
refused by pack_verify. Both readers now refuse incomplete code sets exactly as
zlib does: the only incomplete literal/length or distance code allowed is one
whose longest code is one bit.

The reader now uses its own inflater (png::inflate_zlib), written from RFC
1950/1951 to zlib's inflate rules:

- the header's check bits, method, window and preset-dictionary flag;
- the stored-block complement;
- the symbol counts;
- a complete code-length code;
- repeats with a length before them and within the code lengths;
- a code for the end of the block;
- over-subscribed and incomplete codes;
- a code with no symbol, or a distance past the bytes written, refused where it
  is read;
- the Adler-32, with nothing after it.

miniz_oxide is kept only as a dev-dependency: the writer's round-trip test reads
its stills back with an inflater independent of the reader's.

packaging §5.7 now states these rules instead of "any PNG reader", and says
that a still the verifier does not read is prefix-digest.

stills/ gains the same dynamic block twice: a lone one-bit distance code
(read) and a lone two-bit one (refused, still-incomplete-dist).

png_diff.py drives wai-pack's reader (a new stage-json op, read-png) against
pack_verify.py's. The inputs are:

- zlib's own streams at every level, strategy and window;
- the packager's own stills;
- dynamic blocks built on each code-set rule (22 kinds, each checked against
  the verdict the rules give it);
- mutations of all of these.

wai-pack.yml runs it in debug and release.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

07a34c7e36

• View at this point in history

2026-10-09 13:42:54 -04:00

Transaction Science

Merge remote-tracking branch 'origin/main' into wai/w4-4-staged-packages

1aa1b191a1

• View at this point in history

2026-10-09 11:07:16 -04:00

2026-10-08

Transaction Science

Merge origin/main into wai/w4-4-staged-packages ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

6a299aed56

• View at this point in history

2026-10-08 20:05:53 -04:00

Transaction Science

changelog: W4-4 round 2 — live writes as it goes, PNG rules, repeated objects over HTTP, one stall bound ...

Co-Authored-By: Claude Opus 5.5 <[email protected]>

35b80d6e62

• View at this point in history

2026-10-08 20:05:32 -04:00

Transaction Science

staged-measure: the delivery simulator differentially, in CI ...

sim_verify.py reproduced the committed simulator cases, but nothing ran the two
simulators side by side on inputs outside the corpus. Those inputs include a
refinement with its own delivery timeout, which the corpus has few of.

sim_diff.py generates seeded random models and plans. A model has groups of a
base and up to three layers, each layer one object or one per unit; a
refinement either has its own timeout (zero, small, large or null) or takes the
plan's. A plan varies:

- live or on demand;
- fifo or strict priority;
- subgroup or datagram;
- base and refinement timeouts;
- a layer cap;
- one or two workers;
- the decode order;
- what happens to late refinements.

Each case runs through `wai_stage_measure sim` and through sim_verify.py. The
two must agree on the refusal, or on the whole result, the stall bound and the
longest stall. Where the bound applies, every stall must be within it.

It started as a review probe; the hard-coded paths are removed. wai-conformance
now runs it with two fixed seeds next to sim_verify.py.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

01730c68d3

• View at this point in history

2026-10-08 20:04:33 -04:00

Transaction Science

stage_diff: wide stills, so every deflate distance code is exercised ...

stage_diff's stills were at most 48 columns wide, so the still writer's matches
never reached past distance code 22. Every sixth still is now a picture 1024 to
10923 columns wide whose rows repeat, some with a few bytes changed. The match to
the row above, one filtered row back, then uses distance codes 23 to 29. At
10923 columns that row is past the 32768-byte window, so the writer must fall
back to nearer matches or literals.

A run with nine or more wide stills fails if any of codes 23 to 29 goes unused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

510eaca0a2

• View at this point in history

2026-10-08 19:58:07 -04:00

Transaction Science

pack-conformance: a live-window corpus, checked by both implementations ...

The corpus had no live windows, so pack_verify.py never read a live-window
receipt, a non-zero `prev_index_digest` or a `first_entry` past zero.

live/ holds the staged golden's source packed live with its key, in windows of
two pieces: each window's index and receipt, and expected.json with each
window's place in the chain, index digest and receipt hash.

pack_verify.py reads each window with its own index reader and §6.3 rules, and
verifies the receipt (§9.1, strict Ed25519). It then checks the chain:

- the parent receipt (`live-window`);
- `prev_index_digest`;
- `first_piece` and `first_entry`;
- the timeline;
- the signer is the golden's key.

It also checks that each window's pieces, entries (but flags and window-relative
`refines`), class plan and plan digest are the golden's.

wai-pack's conformance check writes the windows again, compares their bytes, and
verifies the same chain.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

b6858ea485

• View at this point in history

2026-10-08 19:52:14 -04:00

Transaction Science

wai-pack live writes as it goes: each object when published, each window when sealed ...

`wai-pack live` used to gather every object and window in memory and write
them all after the end of the stream. It now writes:

- each object when it is published;
- each window's index, then its receipt, when the window is sealed;
- the plan file and LISTING at the end.

Each file is written to a hidden temporary file beside it, then renamed into
place.

If the source fails mid-stream (a short frame or a frame of another size):

- every sealed window and the objects it lists are kept;
- the objects only unsealed pieces published are removed;
- no plan file and no LISTING are written;
- it exits 1, naming what it kept.

Once an object is written, its bytes are no longer held. A coded piece's stages
now carry their entry lengths (`pack::Lens`), so window_index builds an entry
without the envelope. The live packer moves each envelope into its publication.
The packer hands out sealed windows with `take_sealed()` and keeps none. With
--window 0, the packer still holds every piece's entry fields until the end of
the stream. The module doc, the CLI help and packaging §9.2 now say exactly what
is held.

wai::pin::listing_from_sha256 builds the listing from per-file digests (the
same lines as listing_from), so LISTING needs no file's bytes.

Tests:

- A CLI test feeds the stream through a pipe and checks that windows 0 and 1
  and their objects are on disk while the stream is still open.
- A second CLI test checks that a stream cut inside a frame keeps window 0 and
  its objects and nothing else.
- A unit test checks that no open piece holds an envelope.
- A wai-rs test checks that the listing from digests equals the listing from
  bytes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

c6cbb73843

• View at this point in history

2026-10-08 19:47:05 -04:00

Transaction Science

wai-pack: one PNG reader's rules in both verifiers, with a corpus of stills ...

§5.7 says a still is any PNG of 8-bit RGB without interlace, and the two
readers disagreed on what that admits. Rust refused a suggested palette, which
an RGB picture may carry; Python accepted an interlace, bytes after IEND and a
short row. Both now read by PNG's chunk rules: IHDR first; an optional PLTE
before the image data; consecutive IDATs; ancillary chunks read past; unknown
critical chunks refused; IEND last and empty; a whole zlib stream with a window
of at most 32 KiB, its Adler-32 checked and nothing after it; exact rows.

Rust now inflates with miniz_oxide's core decompressor, so it sees how much
input the stream used. It refuses bytes after the stream, which it used to
read past whenever their last four bytes were the checksum. It also refuses
the window PNG forbids, which a non-wrapping buffer would otherwise let
through.

pack-conformance/stills/ holds the files both readers give one verdict on.
wai-pack's conformance check and pack_verify.py check each file against its
verdict and, when it is read, the SHA-256 of its samples.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

7598dde8bc

• View at this point in history

2026-10-08 15:42:44 -04:00

Transaction Science

wai-pack: a still held for its piece is hashed, never repeated; no piece past max_ms ...

The verifier built a still's prefix as the frame repeated for the piece's
duration (`vec![rgb; n]`), so a package stating a long duration made it
allocate in proportion to that number.

- verify.rs: a still's prefix digest is its frame's digest fed n times
  (`pack::held_digest`); no frame is repeated in memory.
- §4.4 bounds every piece by max_ms; the verifier holds an index to it first
  of every per-piece rule, so nothing is decoded for a piece past it
  (index-policy-mismatch). pack_verify.py does the same, and hashes a held
  still once.
- staged-mutations: the last piece, and the still as the last piece of a cut
  package, each lasting 2^32 − 1 frames with their rules, moments and class
  plan what that span gives, are refused, not aborted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

979c9e002c

• View at this point in history

2026-10-08 15:30:00 -04:00

Transaction Science

wai: a repeated object is one object of a sequence; pack_verify reads the playlists with http_verify ...

http_verify.py joined a layer only to the first occurrence of its base, so the
staged golden's tv playlist, which lists the bumper's base twice, gave a floor
without +video.int_refine while wai-rs, which reads each object once, gave it
with: the two implementations of http-delivery §4.2 disagreed.

- http-delivery §4.2: an object a sequence lists more than once is one object
  of it, described once with every layer it has, and a layer joins every object
  of the sequence that is its base.
- wai-rs (`codecs::described`) and http_verify.py both: each object once, each
  layer joined to every object that is its base.
- http-conformance: two vectors, a base listed twice with its layer beside the
  first and beside the second occurrence; the union and floor keep the layer.
- pack_verify.py runs http_verify.py's adjacency, union and floor over every
  golden's class playlists and MPDs, so CI sees a disagreement; wai-pack.yml's
  push filter lists http_verify.py.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

bbcab8269f

• View at this point in history

2026-10-08 15:17:02 -04:00

Transaction Science

wai: staged_resources counts each thread's allocations, not the process's ...

The counters were process-wide, so libtest's main thread, which allocates a
594-byte message when a test passes 60 seconds, landed in the parse window on
a loaded host: 1,310,720 + 594 = 1,311,314 bytes against 1,310,721 declared.

- The allocator keeps per-thread counts (`thread_local!` `const` `Cell`s, no
  destructor, reached with `try_with`) of bytes allocated, allocations, bytes
  live (signed: a thread may free what another allocated) and their peak, and
  every window reads the measuring thread's.
- A new check: a megabyte another thread allocates inside a window does not
  count, and one this thread allocates does.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2e8d380e26

• View at this point in history

2026-10-08 15:09:34 -04:00

Transaction Science

wai: one stall bound, of a model — wai-web prints the bound the native tool prints ...

wai-web's simulated() printed the plan-only bound while wai_stage_measure prints
the model's, so a per-track case (refinements with timeouts of their own) gave
35000 against the corpus's 65000 and measure::tests failed.

- wai-web: `stall_bound_for(model, &p)`.
- stage_measure::sim: the plan-only bound is gone from the API; §10.4's bound
  is `stall_bound_for(model, plan)` alone, its documentation the full rule.
  The simulator's tests keep a plan-only helper over `bound_with`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

c31ea0556e

• View at this point in history

2026-10-08 15:09:34 -04:00

Transaction Science

ci, changelog: wai-pack's staged packages ...

- wai-pack.yml: the push filter lists the staged-delivery and
  staged-measurement verifiers pack_verify.py now imports; the test-count
  floor covers tests/staged.rs and the library's new tests; stage_diff.py
  runs against the debug and the release build; the media job reads both
  goldens' floors.
- CHANGELOG: the staged packages, the MoQ stage set, live windows, simulated
  delivery and the simulator's per-object timeout.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

103ef76156

• View at this point in history

2026-10-08 14:21:54 -04:00

Transaction Science

wai-pack: staged packages — refinement layers, stills, a piece's ladder, live windows and simulated delivery (W4-4) ...

The packer writes refinement layers, chooses each piece's representation by
its ladder, packs live, and simulates delivery over MoQ; the corpus gains a
staged golden that the independent verifier re-derives byte for byte.

- Refinements (packaging §5.4): for each refinement rung of a piece's ladder,
  a wai.video.int_refine object over the int_motion base, written by wai-rs's
  refinement encoder (staged::wir1::encode) with the manifest `wai stage
  refine` writes; every layer applied to the decoded base by the reference
  sink before it is kept. The index states each prefix's tier, digest, SSE
  and deadline.
- A piece's ladder and stills (§5.7, plan::piece_ladder): a rule's
  then.ladder, or the component's first base rung, its refinement chain and
  its floors. §3.4 item 10 now requires a rule's ladder to name exactly one
  base, every refinement over a rung it names and every floor. A
  wai.image.png base is a still, the piece's first frame held for the piece,
  written as a PNG whose every byte is stated (png.rs; the verifier's reader
  inflates with miniz_oxide, the one new dependency).
- One coder (pack::Coder) for the on-demand and the live packer; the
  verifier holds each piece to its ladder and its plan's deadline, and
  decodes every prefix (a still held, each layer through the sink).
- Catalogs: the MoQ catalog is a stage set of moq-streaming-format §4.1,
  written by staged::moq (wai_staged, a receipts track and an alias per
  media track, depends on a track of every layer below); `priority` (MOQT's,
  §8.5) and `wai_layer` leave it, and the registry drops `wai_layer`. HTTP
  views carry the component's default base capability only: a still's piece
  plays from the floor there (http-delivery §4.2), and its stage view and
  MoQ tracks list it.
- Live (§9.2, live.rs, `wai-pack live`): frames pushed one at a time, pieces
  coded as the segmenter closes them, windows sealed by version-2 receipts
  chained live-window; its plan, bases and refinements are the on-demand
  package's byte for byte (`wai-pack listing --primary`).
- Delivery (§8.9, delivery.rs, `wai-pack sim`): a thin consumer of
  stage_measure::sim — per-track timeouts, §8.5's priorities, the
  subscriber's cap, group receipts on each track's chain; delivered objects
  accepted by Merkle proof, dropped ones held by inclusion only.
- Corpus: the golden video-staged-01 (fixture, policy, supplied floor and
  supplied_floor.py, which states how it was written), staged-mutations/,
  delivery/; video-base-01 keeps its objects, index and receipt (its MoQ
  catalog and LISTING change).
- pack_verify.py re-derives every layer and still from the source (its own
  rANS coder and deflate), decodes every prefix (staged_verify.py's WIR1 and
  PNG readers), checks the MoQ catalog with staged_verify.py's §4.1 check,
  renders every catalog, and re-derives each delivery case through
  sim_verify.py's simulator. stage_diff.py runs layers, stills, piece plans
  and deliveries differentially.
- Tests 5, 7, 12 and 13 over the staged golden (tests/staged.rs).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

abc75ec98c

• View at this point in history

2026-10-08 14:21:54 -04:00

Transaction Science

wai: a refinement's own delivery timeout in the staged delivery simulator ...

A MoQ track sets one DELIVERY_TIMEOUT for every object on it, and a packager
gives each refinement layer and deadline its own track (packaging §8.6), so
one refinement timeout per plan cannot model a package's delivery.

- stage_measure::sim: a model object may carry `timeout_us`, a refinement's
  own timeout in place of the plan's for its layer; a base carrying one is not
  a well-formed model. `timeout_us` is written only where an object has one,
  so every existing model reads and writes back unchanged.
- §10.4's bound under fifo takes T as the longest timeout a refinement takes,
  its own or the plan's (`stall_bound_for`); `stall_bound` keeps its meaning
  for a model with none, and `check_stall` uses the model's.
- staged-measurement §10.4 and Appendix B state both; the corpus adds the
  `per-track` model and three cases (a fifo plan with no plan timeout is
  bounded by the objects' own); sim_verify.py follows, independently.
- Tests: a hand-computed case, and the bound over targeted draws whose
  refinements carry timeouts of their own.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

bdf52d511b

• View at this point in history

2026-10-08 14:21:27 -04:00