wai: staged delivery — a base first, refinements after #149

Merged

dcharlot merged 37 commits from wai/land-17-staged-delivery into main 2026-10-05 04:55:35 -04:00

Conversation 0 Commits 37 Files changed 439 +23777 -198

dcharlot commented 2026-10-04 21:28:49 -04:00

Owner

Copy link

Staged delivery: a sink starts from a base and improves it as refinements arrive, each a separate object linked to its base by hash. Every prefix carries a stated determinism tier, so a partly refined picture says what it reproduces, and a missing or refused refinement leaves the base intact.

• The extension. extensions/staged-delivery.md defines the refinement-only role, how a layer names its base and the stages before it (refines, with per-unit prev runs), and the order in which a sink admits, holds, applies or refuses each layer (§4). Layers that arrive out of order are held within a bound. Two different layers for the same unit freeze that unit at the output it already shows. A sink that holds a key log or receipts counts a layer as delivered only against verified evidence.
• WIR1. The residual refinement payload (Appendix A), with identity, up-sampling and rate predictors. Like the integer payloads, it has a parser that is total, caps, a closed list of refusals in a stated order, a cost a sink reads before decoding, refusal at the first missing symbol, and canonical stream ends. It reads no parameter set.
• What a sink holds. A layer is parsed once and dropped once no unit holds it. A chain budget bounds the whole chain, and a layer past it is refused with chain-over-limit before any stream is decoded. Re-decoding a base keeps its state. §14 states the sink's own work bound.
• The registry and SPEC. Base admission is one table (codecs::BASE_ADMISSION), with the existing enhancement layer folded in. The registry gains the refinement kind. SPEC §3, §4, §5, §7.1 and §8 follow. A refinement-only capability is never dispatched. A manifest that repeats a member name is refused (SPEC §2).
• Multi-rendition envelopes. Refinement components sit on the existing component identity. Exploding an entry gives that entry's own WAI1, and packing back gives the same bytes; pack checks its own output before returning.
• Tools and conformance.
  • wai stage refine|apply|inspect|pack|explode.
  • The corpus staged-conformance/ covers the image, video, audio and splat bases, every refusal, the ordering, batch, freeze and budget rules, and multi-rendition cases.
  • staged_verify.py is an independent verifier written from the extension's text, with a strict envelope reader.
  • There are cross-platform goldens and Kani proofs of the staged arithmetic, predictors and indices.

The sink's work grows linearly with the payload. A test with a counting allocator checks this in both CI kernel steps.

Checked locally on macOS arm64:

• the link gate, including the Kani harnesses;
• the release fast-kernel step;
• the staged corpus through both verifiers;
• wai registry export --check;
• the section-reference guard;
• the public-text checks.

Two adversarial reviews' findings are fixed, each with a test that was shown to fail first. Over 400,000 parser, payload and step mutations, the Rust sink and the independent verifier agree. The forge CI runs on this pull request.

🤖 Generated with Claude Code

Staged delivery: a sink starts from a base and improves it as refinements arrive, each a separate object linked to its base by hash. Every prefix carries a stated determinism tier, so a partly refined picture says what it reproduces, and a missing or refused refinement leaves the base intact. - **The extension.** `extensions/staged-delivery.md` defines the refinement-only role, how a layer names its base and the stages before it (`refines`, with per-unit `prev` runs), and the order in which a sink admits, holds, applies or refuses each layer (§4). Layers that arrive out of order are held within a bound. Two different layers for the same unit freeze that unit at the output it already shows. A sink that holds a key log or receipts counts a layer as delivered only against verified evidence. - **WIR1.** The residual refinement payload (Appendix A), with identity, up-sampling and rate predictors. Like the integer payloads, it has a parser that is total, caps, a closed list of refusals in a stated order, a cost a sink reads before decoding, refusal at the first missing symbol, and canonical stream ends. It reads no parameter set. - **What a sink holds.** A layer is parsed once and dropped once no unit holds it. A chain budget bounds the whole chain, and a layer past it is refused with `chain-over-limit` before any stream is decoded. Re-decoding a base keeps its state. §14 states the sink's own work bound. - **The registry and SPEC.** Base admission is one table (`codecs::BASE_ADMISSION`), with the existing enhancement layer folded in. The registry gains the refinement kind. SPEC §3, §4, §5, §7.1 and §8 follow. A refinement-only capability is never dispatched. A manifest that repeats a member name is refused (SPEC §2). - **Multi-rendition envelopes.** Refinement components sit on the existing component identity. Exploding an entry gives that entry's own WAI1, and packing back gives the same bytes; `pack` checks its own output before returning. - **Tools and conformance.** - `wai stage refine|apply|inspect|pack|explode`. - The corpus `staged-conformance/` covers the image, video, audio and splat bases, every refusal, the ordering, batch, freeze and budget rules, and multi-rendition cases. - `staged_verify.py` is an independent verifier written from the extension's text, with a strict envelope reader. - There are cross-platform goldens and Kani proofs of the staged arithmetic, predictors and indices. The sink's work grows linearly with the payload. A test with a counting allocator checks this in both CI kernel steps. **Checked locally on macOS arm64:** - the link gate, including the Kani harnesses; - the release fast-kernel step; - the staged corpus through both verifiers; - `wai registry export --check`; - the section-reference guard; - the public-text checks. Two adversarial reviews' findings are fixed, each with a test that was shown to fail first. Over 400,000 parser, payload and step mutations, the Rust sink and the independent verifier agree. The forge CI runs on this pull request. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 24 commits 2026-10-04 21:28:49 -04:00

wai: the rANS encoder writes the bypass escape; an integer CDF from a histogram ... 2496c95410

RansEncoder::encode now codes a value outside its table's span through the
bypass escape, exactly as RansDecoder reads it: the escape symbol, the count
of 4-bit groups as a run of 15s closed by a group below 15, then the raw
value's groups least significant first. It returns false only where no
stream can code the value: a table shorter than three entries or longer than
the slice, a frequency that is not positive, or value - offset outside i32.
Every stream it writes still ends canonically.

int_entropy::hist_to_cdf builds the encoder's table from a histogram in
integer arithmetic alone: the escape is the last bin with frequency 1, every
symbol gets at least 1, the total is exactly 2^precision, and the leftover
goes by largest remainder with ties to the lower index.

ternary::rans_encode_trits called encode inside debug_assert!, so a release
build coded no symbol; the call now runs in every build.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: WIR1, the residual refinement payload, and its predictors ... ac5a0b2bf4

staged::wir1 parses, decodes and encodes WIR1: one rANS stream per unit, and
for each element clamp(pred + r*q) in the element's range. The parser checks
in a stated order and refuses with the first rule that fails (magic, version,
header, reserved field, header fields, the unit kind's rules, caps, tables,
stream lengths, streams, trailing bytes), with the codes the integer payloads
use for the same faults. A unit's decode refuses the wrong shape below before
any symbol, a stream at the first symbol it runs out on, and an end that is
not canonical. check_streams decides whether a payload is well formed without
the stage below. The header declares the layer's output, symbols and working
bytes before anything is decoded.

staged::predict holds the predictors: identity, up2 (the half-pel arithmetic
of int_video, odd sizes included) and rate (linear interpolation rounded half
away from zero, channels interleaved). staged::wir1_arith holds the one
arithmetic step. Both are self-contained so the model checker and the
byte-equality crate can include them.

Feature `staged` = neural_int + jwp_hash.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the refinement-only role, base admission as one table, and refines ... 7fb55e8c17

SPEC registers the residual refinement capabilities wai.image.int_refine,
wai.video.int_refine, wai.audio.int_refine and wai.splat.int_refine, each
refinement-only: applied in a declared order to the output of the stage
below it, bound to that output's producer by content hash, and never decoded
on its own. Section 4 never takes step 3 or 5 with one; section 8 adds the
role and lists the envelopes its registration makes non-conforming.

SPEC section 5's Base admission table holds the bases of every capability
applied to a base's output, the refinements and the LCEVC enhancement layer.
codecs::BASE_ADMISSION is that table; enhancement_bases, companion_applies_to
and admits read it, and no other admission logic remains.

The registry gains the refinement kind and role. A refinement is not
dispatchable, never selected, never a companion or a companion's base, and
not in the video media class; its class is StandardDefined and, like the
audio scene, it composes another envelope's decoded output, so it is as
exact as the base it refines. The registry export is regenerated.

A WAI1 manifest gains the optional member refines, kept as JSON so a
malformed value still parses; container::Refines reads it and reports
refines-malformed. Wai::check_component_roles reports a refinement without
refines, with a fallback, as a fallback, and refines on any other capability.
The emit paths parse a refinement's WIR1 payload.

extensions/staged-delivery.md defines stage sets, the refines member, the
per-unit application algorithm with its closed list of reasons, holds and
batches, tiers and stage hashes, the residual form, deferral, live deadlines,
attestation, byte counting, and the WIR1 payload with its check order, caps,
declared cost and refusal codes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the stage chain applies layers per unit, in the extension's order ... 9aa98b03ce

staged::StageChain holds one base's units, each with its stage, producer,
tier and current output, and runs staged-delivery 4.1 for every layer it is
offered: steps 2-6 decide for the envelope (base present, refines against
the base's units, not via a fallback, admitted, attested, payload well
formed and within the cost limit), steps 7-10 for each unit (conflict,
readiness, geometry, deadline). A unit is applied whole or not at all.
Layers a unit is not yet ready for are held to the bound and released when
its stage advances; the pairs a release makes ready run as one batch, so two
conflicting layers released together are both refused and the unit frozen,
whichever arrived first. A layer that conflicts with one that already passed
freezes the unit at the stage it has.

staged::StageSink holds layers whose base it has not decoded and releases
them together when the base arrives.

staged::tier composes a unit's tier as a pure function: the weakest step
wins and entropy-consistency does not pass through a layer; a stage hash is
given only at decode-equivalence. staged::schedule holds the live schedule:
presentation times, the lead rule, the layers a throughput affords, and what
a sink presents for a trace of arrivals.

The extension's step order puts every envelope-scoped step before the unit
steps, and the conflict check before readiness, so a layer that differs from
one already applied is a conflict rather than a mismatch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the stage refusal list stays out of the C header ... 52440061a7

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the staged-delivery corpus and its independent verifier ... ce9207ac3c

staged-conformance/ holds stage sets and the steps a sink takes with them:
image layers over a pinned integer image base (identity, up2 to even and odd
sizes) and over a PNG base; video layers per frame and per GOP, with prev
runs, over integer motion bases and over a pinned neural-keyframe clip; audio
layers over a pinned integer audio base, and a stereo rate ladder over a FLAC
base; a splat attribute layer; and one case per refusal and order rule
(another base, a wrong producer, a missing layer, conflicts held and after an
application, out of order, a fallback base, geometry, truncation, a malformed
table, q of 0, a word after the last symbol, a stream that runs out, clamped
and bypassed residuals, malformed refines, a capability that is not a
refinement, a base not admitted, inclusion-only evidence, a cost limit, a
missed deadline, the hold bound). Each case's case.json states what a sink
reports at every step and holds at the end; schedule.json holds arrival
traces and what a sink presents.

wai_staged_vectors gen writes the corpus; verify regenerates it byte for
byte, replays every case through StageSink, decodes the integer bases with
their pins against stage0.bin and checks their tiers. staged_verify.py is a
standard-library reimplementation of the envelope, refines, WIR1, the sink's
algorithm, tiers and stage hashes, and the PNG, FLAC and WIV1 keyframe-0
decodes; it checks every layer application in every case.

The pure-Python BLAKE3 moves from claim_verify.py to _blake3.py, which both
verifiers import. A unit held and decided within one step is reported by its
decision alone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: WAI2 refinement components, explode and pack, on the component identity ... 24fcf4db3c

A WAI2 entry with role "refinement" carries refines whose base and producers
are the ids of entries in the same envelope, and is placed after every
selectable entry. check_component_roles holds it to SPEC 7.1's rules in a
stated order: refinement-only capability, id and sha256 on it and on what it
names, well-formed refines, a selectable base, admission, producers that are
selectable or earlier refinements, and placement; refines on any other entry,
and a refinement-only capability in any other role, are reported. A
refinement is never selectable; refinements_of gives a picked entry's
refinements whose payloads match their sha256.

WaiMulti::explode_all and explode write each component's WAI1 envelope: the
entry's envelope_manifest verbatim when it has one, or a manifest built from
the WAI2 manifest's members as written, read as raw text so nothing is
re-serialised, with the ids of refines replaced by the content hashes of the
named entries' exploded envelopes. So an exploded refinement is the object a
separately carried layer is. WaiMulti::pack packs WAI1 envelopes with their
manifests verbatim, and they explode back to the same bytes;
check_envelope_manifests reports an envelope_manifest that does not match its
entry. No separate component digest is defined: the component identity is
the entry's id, sha256 and prior.

SPEC 7.1 states exploding, envelope_manifest and refinement components; SPEC
8 lists the new optional members and states that a malformed one does not
make the manifest unreadable. serde_json gains raw_value.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the corpus checks WAI2 refinement components and exploding ... 880099fec0

staged-conformance/wai2/ holds WAI2 envelopes with refinement components and
an index of what a sink decides about each: the role check, whether each
envelope_manifest matches its entry, the content hash of every exploded
component, and entry 0's refinements. A live-style chain packed with its
manifests verbatim explodes back to its sources byte for byte. Both verifiers
check every file; staged_verify.py explodes components by reading the
manifest's members as the text they are.

Explode now decides entry by entry, so an entry whose refines cannot be
translated does not stop the others, and the envelope_manifest check reads
only what each entry names. RoleError gains a code. SPEC 7.1 states that a
member written as null is absent when a component is exploded.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: wai stage writes, applies, inspects, packs and explodes layers ... e370e7840d

wai stage refine writes a residual layer toward a target over the stage the
given layers bring a base to, with prev naming the producers its span holds
(a string, or runs); apply offers layers to a base in order and prints each
report and the units' stages, producers, tiers and stage hashes; inspect
prints a layer's refines, role check, WIR1 header, declared cost and stream
check; pack and explode convert between WAI1 envelopes and a WAI2. A base of
an integer codec is decoded with its parameter set and its pin; any other
base is read from its stage-0 buffers. The subcommand builds with staged and
the integer codecs' features.

wai wrap refuses a refinement capability, since it writes no refines, and
points to wai stage refine; wai inspect prints a WAI1's refines.
staged::units_from_decoded gives a base's units from its integer decode.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the carriage tests hold a refinement to its own role ... 091d1c8a6d

A refinement-only capability is neither standalone media nor a companion: the
carriage tests carry it as a WAI1 with refines, byte-exact, and report it
carried as a companion as a refinement-only capability in another role.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: cross-architecture goldens for staged delivery ... 48b41f21d3

byte-exact-conformance includes the WIR1 parser, predictors and arithmetic
step and pins every stage of three corpus sets: a WIV1 GOP with per-frame
and per-GOP layers, up2 to odd sizes, and a stereo rate ladder. The last
stage of each must also be the clip hash its case states.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge main into wai/land-17-staged-delivery ... b2b9294590

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ci(wai): staged delivery in the conformance, byte-equality and proof jobs ... 5bd1cb9470

The cargo-class build and the library tests add staged and the integer
media features; the fast-kernel steps (overflow checks on and off) add
staged and run the wai stage tests; staged is checked alone and for wasm32;
the corpus loop verifies staged-conformance on every leg; staged_verify.py
runs beside the other independent verifiers. The byte-equality job fires on
the staged sources it includes and on the corpus, and the proof job on the
two staged sources its harnesses cover.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: Kani proofs of the staged arithmetic, predictors and indices ... ea174e7a6f

wai-verify includes staged/predict.rs and staged/wir1_arith.rs, whose
harnesses prove: step_in_range (for every residual the rANS decoder returns,
every step q and both element ranges, pred + r*q fits i64 and the element is
that sum clamped), up2_bounded (the half-pel prediction lies between the
neighbours it reads, odd output sizes included), rdiv_matches_spec (rdiv is
the rounding half away from zero, stated by inequalities rather than a second
division) and rate_index_in_bounds (every output sample's indices are source
samples, for every channel). The last two are nonlinear and hold over the
bounded windows their docs state; each harness's cover is reached.

rdiv and the rate indices move from i128/u128 to checked 64-bit arithmetic,
which is total for every input and gives the same values for every shape a
WIR1 payload allows; the corpus and the goldens are unchanged.

The CHANGELOG gains the staged-delivery subsection.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the sidecar coverage test names where the staged corpus states its digests ... e4b46cbd4c

every_decoded_output_states_a_sha256 requires every conformance directory to
carry a sha256.json sidecar or a stated reason; the staged corpus states the
SHA-256 of every stage-0 buffer and every unit's stage in its case.json.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the stage chain parses a layer once, keeps a chain budget, drops what no unit holds ... 3d4f6ca03d

- A layer's WIR1 payload is parsed once, at §4.1 step 6. Wir1 keeps its tables
  flat and its streams as end offsets beside the payload, so a parse holds its
  declared working bytes (which now count 8 bytes per unit for the stream
  index), and a unit's output is allocated at its exact size.
- Step 7 compares a unit's key with the producers of its stages (a unit's stage
  rises by one per application) and a batch is grouped by unit; held units are
  kept per unit and counted per layer; prev's runs are indexed once and
  bisected. An offer's work and memory grow linearly with the layer's units
  (tests/staged_resources.rs, a counting allocator, in both CI profiles).
- The chain budget (§4.5; chain-over-limit at step 6, before any stream is
  decoded) bounds the units' output, 32 bytes per stage, and every retained
  layer's payload, declared output and declared working bytes. A layer no unit
  is held for is dropped at the end of the offer, release or expiry.
- Decoding again a base the sink holds keeps its chain and reports nothing;
  StageSink::remove_base drops a chain.
- Shape products are checked: one past u64 is geometry-mismatch.
- A layer whose manifest names a fallback is refused at step 1.
- A layer refused when its base arrives has one report.
- BaseInfo::first_unit places a base's units on the clock a set split across
  bases shares.
- A layer held for its base and offered again with stronger receipt evidence
  keeps it; over an attested base it counts as arrived when its receipt was
  verified.
- A payload whose unit count is not the span's is payload-malformed with the
  code field_range; A.6 states how step 6 and step 9 split A.4.
- span: null is absent; SPEC §2 states what a JSON integer is.
- base_never_waits plays random arrival traces through a live StageSink.

The corpus is regenerated through its generator (the config's budget, each
base's first_unit); every case keeps its outcome.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: WAI2 explodes entry by entry, pack checks its own round trip, names are unique ... 8fa68207f6

- WaiMulti::explode(i) returns entry i's own result (explode_each is public):
  an entry that cannot be exploded leaves every entry that does not name it
  intact. The corpus states each entry's hash, null only where that entry
  fails.
- pack refuses a WAI1 with bytes around its manifest's JSON value or after its
  payload, and before returning parses its output, explodes it back to the
  sources byte for byte and runs check_envelope_manifests; a manifest nested
  near the depth limit, which a WAI1 reader takes, is refused because inside a
  WAI2 entry it passes the limit. SPEC §7.1 states the rule.
- A refinement-only entry's envelope_manifest names no fallback
  (component-manifest-mismatch).
- A manifest that names a member twice in one object, at any depth, is
  malformed (SPEC §2), in Wai::from_bytes and WaiMulti::from_bytes.
- New WAI2 corpus files, one per refinement rule of SPEC §7.1 not covered
  before (not refinement-only, identity missing, base missing, base not
  admitted, refines on a non-refinement, a refinement-only capability as a
  plain component, malformed refines) and a refinement envelope_manifest with
  a fallback.
- Test 20 models the earlier selection with its digest and pin skips.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the staged corpus covers each rule it was missing ... adc05fdb88

New cases, written by wai_staged_vectors gen and replayed by its verify:
- §3 and step 1: no refines; a fallback; a layer 1 naming another producer;
  prev as one run; adjacent runs with one producer; runs not summing to the
  span; -0, 1.0, 1e0 and -0.0 in integer members; span null as absent.
- Step 2: a span past the base; runs past the base's units.
- Step 6: a unit count not the span's (field_range); bad_magic, version,
  reserved_nonzero, an RGB8 element of 2 bytes, too_large, trailing; the
  symbol and working-byte cost limits; chain-over-limit; a retained held layer
  counted in the budget until it expires.
- Steps 7 and 8: a freeze refusing a held layer; a conflict inside a release
  batch; layer-gap on arrival at the bound; a held layer turned prev-mismatch.
- Steps 9 and 10: up2 and rate geometry; rate downsampling; a held layer late
  on release; a clip split into two bases on one live clock.
- A base decoded again; a layer refused on release with one report; receipt
  evidence upgraded while held, and its effect on a deadline.
- wai2/index.json lists pack attempts: bytes after a manifest, before it,
  after a payload, and a manifest too deep inside a WAI2 are refused.

The CHANGELOG states what the corpus covers in place of "one case per refusal
and order rule".

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: staged_verify.py implements §4 from the text, with a strict envelope reader ... c1ec1e9d64

The Python verifier is rewritten as a direct implementation of the
extension, not of the reference sink's structure:
- a strict reader of envelope JSON (SPEC §2): UTF-8 with no byte-order mark, no
  member named twice, no NaN or Infinity, no number past the float range, no
  lone surrogate, at most 127 nested containers, and -0, a fraction or an
  exponent as no integer; the members SPEC §3 and §7.1 require, with their
  types. A WAI1 or WAI2 it cannot read is the step's error, never a crash;
- each envelope goes through §4.1's numbered steps in turn: step 1, a hold for
  an absent base with the strongest evidence kept, steps 2 to 6 with the chain
  budget, then steps 7 and 8 per unit against a literal record of what passed
  step 8; batches as §4.4 forms them; holds, their bound and expiry; frozen
  units; dropping what no unit holds;
- live deadlines on one clock across bases (first_unit), saturating at 2^64-1,
  and over an attested base from when the layer's receipt was verified;
- a step's reports are compared as a set, one per envelope;
- WAI2: per-entry explode, the fallback rule for a refinement's
  envelope_manifest, the WAI1 schema of an envelope_manifest, and what a packer
  refuses;
- a base whose payload is a committed integer-codec vector has its stage 0
  checked against that vector's sha256.json.

The README defines the evidence tokens and the report fields, and states
which digests check each pinned base's stage 0: the vectors' sidecars, and the
cross-architecture goldens for the int_hyper image.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: is_enhancement_layer reads the admission table; a test reads §4.2 and A.6 ... e4cf48c3e0

- codecs::is_enhancement_layer is read from BASE_ADMISSION (the capabilities
  applied to a base as an enhancement) rather than naming one capability.
- staged::tests::staged_delivery_text_lists_the_codes reads the extension and
  requires §4.2's table to list StageRefusal's reasons and A.6 to list
  Wir1Refusal's codes, each in its order; the CI paths comment for
  staged-delivery.md now names that test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: wai stage refine takes the layer's shape from --out-dims ... 3b769ee4be

--dims is the stage-0 shape a --stage0 base is read at; with --stage0 the
layer's own shape could not be given. --out-dims gives it in every case, and
tests/cli_stage.rs refines a --stage0 base by up2 to an odd 31x31.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a staged case of envelopes a strict reader refuses ... 9b61af63ed

The corpus replay records an offer that is not a WAI1 the sink reads as the
step's error (not-wai1, encrypted). The new case unreadable_envelopes offers
a member named twice (in refines and at the top), NaN, a number past the
float range, a lone surrogate, a byte-order mark, a manifest nested 128 deep,
a manifest with no media and an encrypted layer, each refused before §4.1;
then a layer nested 127 deep, which is read and applies. staged_verify.py
compares an error step as it is.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the CHANGELOG states the staged fix round's rules and interfaces ... 854a339950

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a retained layer reserves the stage each covered unit gains ... 77df55cca8

The chain budget counted a retained layer's payload, declared output and
declared working bytes, but not the 32 bytes of the stage each covered unit
gains when the layer applies; for units smaller than that, applying a layer
admitted at the budget could take the chain past it. The reserve now counts 32
bytes per covered unit, in the sink, the spec (§4.5), staged_verify.py, the
generator and the tests; chain_budget applies a layer over 64 one-pixel frames
at its budget and requires the state to stay within it. The two budget cases
are regenerated with their outcomes unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 9 commits 2026-10-04 23:29:59 -04:00

wai: a report's held units are filtered by binary search, not by scans ... 9d1006e456

OfferReport::sort dropped from `held` each unit also applied or refused with
a linear scan of those lists per held unit, so a layer whose report mixed held
units with applied or refused ones took time quadratic in its units. The
lists are sorted first, and membership is a binary search. A unit's held
layers are kept in id order, so is_held and an expiry find a layer by binary
search too.

tests/staged_resources.rs adds the two mixed shapes (half held and half
applied; half held and half prev-mismatch) to its linearity checks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: WIR1's table capacity is formed in u64 ... d6a6d7164e

parse sized its table vector by n_tables·cdf_len in usize, which passes 2^32
for a header of one table per unit, 2^20 units and tables of 4097 entries: a
32-bit debug build aborted on a 44-byte payload. The product is formed in u64,
saturating, and capped by the bytes the payload holds. No other usize product
in wir1.rs or staged/mod.rs is formed from header values: the predictors'
indices are bounded by the stage below, whose length was checked.

wir1_header_products_fit_any_usize parses such headers and requires
stream_length; it also passes under wasm32-wasip1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: an applied unit takes its share out of its layer's reserve ... cc44ec3030

A retained layer reserved one unit's output and 32 bytes for every unit it
covers, and an application added the unit's new output and stage on top, so
while a layer was applied to some units and held for others, state_bytes()
counted those units twice and could pass the budget. Each application now
releases the unit's share from the layer's reserve: the state rises only when
a layer is admitted, and §4.5 states it. staged_verify.py counts the same.

chain_budget offers a layer applied to half its units and held for the rest
at a budget with no byte to spare and requires the state within it; the new
case budget_partly_applied needs the released share to admit a further layer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a base's receipt verified after it was decoded makes it attested ... f29536afc7

add_base on a base the sink held ignored the new BaseInfo, so a sink that
verified the base's receipt after decoding it never set attested, and a
layer with no receipt still applied. Decoding the base again with
attested=true now marks it attested, and never the reverse;
StageSink::mark_attested and StageChain::mark_attested do the same without a
decode. Step 5 applies to every envelope that arrives after; those admitted
before keep their decision (§4.4, §10). staged_verify.py does the same, and
the corpus case attested_after_decode decodes, attests, decodes unattested
again and offers layers between.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a dropped base keeps each unit's record; its held layers are base-absent ... 360ebd6178

remove_base followed by add_base started the base afresh, so a layer that had
conflicted, or one that differed from the producers a unit had, applied on
the second decode; and the layers held for the dropped chain vanished with no
reason reported.

- StageSink::remove_base discards the layers held for the chain, reporting
  each base-absent, and keeps each unit's record: the producers of the stages
  it reached, the envelope that passed step 8 at its last stage without being
  applied, and its frozen flag (32 bytes a recorded stage, one more producer
  and a flag).
- The base decoded again starts each unit at stage 0 with its record
  (StageChain::with_records). Step 7 compares a layer with the recorded
  producers, so a unit advances only along them; other content under a
  recorded key conflicts, and a frozen unit stays frozen. The record's stages
  count in the chain's state; replaying one adds nothing.
- StageSink::forget_base drops a record; §14 states what that gives up.
- §4.1 step 7, §4.4, §4.5 and §14 state it; staged_verify.py keeps the same
  record, and the corpus gains a remove step and three cases (replay, frozen,
  diverging producer).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a test that a unit frozen with nothing recorded stays frozen when its base is decoded again ... 9b5e19f83e

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a staged case of a unit frozen in a batch keeping its flag across a dropped chain ... ab0e198182

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the browser writers refuse a manifest that names a member twice ... 10c5497351

wai-web's pack and packMulti parsed the manifest with serde_json, which keeps
the last of two equal names outside a struct's own fields, and then wrote the
text as given, so they could emit an envelope every reader refuses (SPEC §2).
Both now read the manifest through check_unique_members first. A native test
covers a repeat at the top, in target, in model_requirement, nested with an
escape, and in a WAI2 entry; verify_pack.mjs checks the built wasm writers
refuse it; the CI native test count is 9.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: verify_pack.mjs names its failure for both rules ... fa1b60fa2d

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 4 commits 2026-10-05 00:50:52 -04:00

wai: attestation belongs to the base hash and survives a dropped chain ... 61edd17971

remove_base kept unit records but not that the base was attested, so a base
dropped and decoded again by a decode that carried no receipt came back
unattested and layers with no receipt applied; and mark_attested only reached
a held chain.

The sink now keeps the attestation of every base it does not hold (dropped
while attested, or marked before it was decoded). mark_attested(hash) records
it whether the base is held, dropped or not yet decoded; add_base attests a
base it marked before, whatever the decode says; forget_base discards it with
the record. §4.4 and §10 state it. staged_verify.py does the same; the corpus
gains an attest step and the cases attested_across_drop and
attested_before_decode.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a frozen unit of a base decoded again replays its record in order ... 8d89785c34

After a dropped chain and a re-decode, a frozen unit sat at stage 0 and
presented its base, while §14 said a frozen unit stays at its stage. A frozen
unit may now replay its own recorded producers, in order, up to its recorded
stage and never past it; every other layer is refused (layer-conflict) and
the unit stays frozen. It presents its base or a recorded producer's output,
never any other. §4.1 step 7, §4.4 and §14 say so; staged_verify.py does the
same. remove_readd_frozen now replays layer 1a and refuses a layer 2 before
and after; removed_base_keeps_its_record checks order, the frozen stage
presented again, and nothing past the record.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: a sink keeps only non-empty records, within a declared record budget ... 83e7fe6b20

remove_base kept a record for every unit of every dropped base, outside any
budget: twenty dropped bases of 4096 units with no layer held 5 MB.

- A unit record that holds nothing (no stage, nothing passed step 8, not
  frozen) is not kept, nor a base all of whose records are empty and that is
  not attested.
- ChainConfig::record_budget bounds what the sink keeps of the bases it does
  not hold, records and attestations together, counted as §4.4 states (32
  bytes a base; 8 a kept record, 32 a recorded stage, 32 an envelope that
  passed step 8 unapplied). Past it, the bases written longest ago are
  forgotten whole, with forget_base's meaning: as never decoded and never
  attested. StageSink::kept_bytes reports the count.
- §4.4 and §14 state the budget and the eviction; staged_verify.py keeps the
  same records and evicts the same bases; every case states its record
  budget, and the case record_budget evicts one base and keeps another.
- records_within_their_budget and tests/staged_resources.rs check that bases
  with nothing recorded keep nothing, that distinct dropped bases stay within
  the budget, and that an evicted base decoded again starts with no history.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: the record budget is counted alike by every sink ... 5f991e87e8

staged-delivery §4.4: the counted bytes are the interoperable measure, so
two sinks with the same record budget keep and forget the same bases; what an
implementation allocates per counted byte is its own, stated beside its
budget. The reference states its ratio (at most four allocated bytes per
counted byte), which tests/staged_resources.rs asserts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 7b2b1a7551 into main 2026-10-05 04:55:35 -04:00

dcharlot referenced this pull request from a commit 2026-10-05 04:55:37 -04:00

Merge pull request 'wai: staged delivery — a base first, refinements after' (#149) from wai/land-17-staged-delivery into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!149

No description provided.