# WAI Extension: Staged Delivery

> Status: Draft. A base first, refinements after, each its own object, bound to what
> it refines by content hash. Every stage is presentable, every prefix of a stage set
> states its determinism tier, and a refinement that never arrives leaves the stages
> below it intact. Reference impl: `wai-rs` — `staged::{StageChain, StageSink}`,
> `staged::{wir1, wcp1, side, predict, wir1_arith, tier, schedule}`, `staged::claim::StageObserver`,
> `container::Refines`, `codecs::BASE_ADMISSION`; the `wai stage` tool; the corpus
> [`staged-conformance/`](../staged-conformance/). The level form (§6.2) is
> `int_field::FieldStages`, with its corpus in
> [`volumetric-conformance/fields/`](../volumetric-conformance/fields/). Keywords **MUST**, **MUST NOT**,
> **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope and model

A **stage set** is a **base** and **layers**. The base is any envelope a sink decodes
under SPEC §4 or §7.1. A layer is its own `WAI1` envelope whose capability is
**refinement-only** (SPEC §5, *Refinement capabilities*) and which names, by content
hash, the base and the envelopes it refines.

A sink holds, for every **unit** of the base's output, a **stage** (0 after the base
decodes) and a **producer**: the content hash of the envelope that produced the
unit's current output, the base at stage 0. Layers apply per unit, and every stage is
presentable. A layer of the conditioned form (§6.3) also reads **side planes**: pictures
the sink holds exactly besides the base, named in the layer's `refines` (§3.1). A base never names its layers, so a layer MAY be produced after its
base, by the base's publisher or by a party the sink accepts (§10.1).

## 2. Terms

- **Content hash**: [jwp-receipts](jwp-receipts.md) §2.1,
  `BLAKE3("moq-jwp:object\x01" ‖ envelope_bytes)` over the envelope bytes exactly as
  received, written as 64 lowercase hexadecimal digits.
- **Unit**: the smallest presentable piece of a base's output, fixed by the base's
  capability: one frame of video; the whole picture of an image; the whole clip of
  audio; the whole attribute tensor of a splat cloud; the whole field of an integer neural
  field.
- **Span**: a contiguous run of a base's units.
- **Producer**: for a unit, the content hash of the envelope that produced its current
  output.
- **Refinement-only capability**: a capability registered with that role (SPEC §5,
  §8).
- **Side plane**: an RGB8 picture a layer of the conditioned form reads besides the
  unit's current output (§3.1): the output of an object the sink decoded, or a frame the
  sink rendered itself.

## 3. The `refines` member

A `WAI1` manifest whose `model_requirement.capability` is refinement-only **MUST**
carry `refines`, and its `model_requirement.fallback` **MUST** be `null`. A manifest
whose capability is not refinement-only **MUST NOT** carry it. A sink routes an
envelope by its capability: one whose capability is refinement-only is a layer, which
§4 processes; any other is a base, dispatched by SPEC §4, and a `refines` it carries is
ignored. Likewise a `WAI2` rendition is a layer only when its capability is
refinement-only (SPEC §7.1).

```json
"refines": { "base": "<64 hex>",
             "prev": "<64 hex>" | [ {"producer": "<64 hex>", "count": 4}, ... ],
             "layer": 2,
             "span": { "from": 0, "count": 4 } }
```

| member | presence | meaning |
|---|---|---|
| `base` | REQUIRED | Content hash of the stage set's base. |
| `prev` | REQUIRED | The producer each unit the layer covers is expected to have. A string means every such unit. An array of runs gives, in unit order, `count` consecutive units whose producer is `producer`. |
| `layer` | REQUIRED | Integer 1–255: the stage this layer brings each unit to. |
| `span` | OPTIONAL | `from` (integer ≥ 0) and `count` (integer ≥ 1): the units the layer covers. Absent, or `null` (which is the same as absent, as SPEC §3.1 reads an OPTIONAL member): every unit of the base. |
| `side` | OPTIONAL | The side planes a layer of the conditioned form reads (§3.1). Absent, or `null`: none. |

Rules:

- Hashes **MUST** be 64 lowercase hexadecimal digits (in a `WAI2` entry they are
  component ids; SPEC §7.1, *Refinement components*).
- When `layer` is 1, every producer `prev` names **MUST** be `base`.
- A run array **MUST NOT** contain two adjacent runs with the same producer. A layer
  whose units have one producer **MUST** write `prev` as a string, so an array of one
  run is malformed.
- With a `span`, the run counts **MUST** sum to its `count`. The span **MUST** lie
  within the base's units, and with no span the run counts **MUST** sum to the base's
  unit count; a sink checks these once it holds the base (§4.1 step 2).
- `from + count` **MUST** fit in 32 bits, and so **MUST** each `count`.

Any other JSON type (an integer is written as SPEC §2 defines one), or a value outside
these rules, makes `refines` malformed (`refines-malformed`); the envelope still
parses. A member of `refines` this
revision does not define is ignored (SPEC §8).

### 3.1 Side planes

`side` is read for a capability whose registration says its layers read side planes:
in this revision `wai.image.int_cond_refine` (§6.3). For any other capability the
member is ignored. Present and not `null`, it is an array of 1 to 4 entries, in the order
the layer's payload reads them, each one of:

- a content hash (64 lowercase hexadecimal digits): the output of the object whose
  content hash it is, an envelope the sink decoded under SPEC §4 and holds. A G-buffer
  of rendered content (depth, normals, albedo, coded into the three channels as its
  producer chooses), or any other picture both sides hold, is carried this way as an
  exact component of its own;
- an object of exactly the members `object` (a content hash), `renderer` (the string
  `wai-splat-render:i1`) and `pose` (a pose of [splat-exact](splat-exact.md) §2): the
  frame that renderer gives at that pose (splat-exact §5) for the splat set the sink
  decoded from `object` through `wai.splat.int_codec`. The sink renders it itself, so the
  plane costs no bytes on the wire.

No entry appears twice: two content hashes are the same entry when they are equal, two
render entries when their `object` is equal and their poses are the same pose
(splat-exact §2). Any other value makes `refines` malformed (`refines-malformed`).

A sink **holds** an entry's plane when it holds the object an entry names, decoded,
and, for a render entry, the object was decoded through `wai.splat.int_codec` and the
sink has rendered the frame at the pose; a render the renderer refuses (splat-exact
§10) is no plane. A plane is **exact** when the object it comes from was decoded
through its manifest's `model_requirement.capability`, not a fallback, and the object's
base step has the tier `decode-equivalence` (§5): the bytes of the plane are then the
same on every conforming sink. A plane is read as one RGB8 picture: the object's output
when it is one picture, the frame of a render.

A layer that reads side planes travels as a `WAI1` object. A `WAI2` refinement component
carries no `side` (SPEC §7.1, *Refinement components*), and a packer refuses to pack a
`WAI1` layer that carries one: its side planes are objects of their own, not components
of the base's envelope.

## 4. Applying layers

### 4.1 Algorithm

A sink that implements this extension **MUST** process each received envelope *R*
whose capability is refinement-only as follows. A sink routes by capability (§3), so an
envelope whose capability is not refinement-only reaches step 1 only where a caller
offers it to this algorithm as a layer; step 1 refuses it, as it refuses a
refinement-only envelope whose fallback is not `null` (`capability-not-refinement`). Steps 1–6 decide for the whole
envelope; steps 7–10 decide for each unit *u* the layer covers. The first step that
fails is the reason for the envelope (1–6) or for the unit (7–10).

1. `refines` is present and well formed (§3) (`refines-malformed`). *R*'s capability
   is refinement-only, and its `model_requirement.fallback` is `null`
   (`capability-not-refinement`).
2. The sink has decoded a base whose content hash is `refines.base`. Otherwise it
   **MAY** hold *R* until its bound (§4.3), and then discards it (`base-absent`).
   With the base, `refines` is checked against the base's unit count (§3)
   (`refines-malformed`).
3. The base was decoded through its manifest's `model_requirement.capability`, or
   through the `WAI2` component whose exploded envelope (SPEC §7.1) has the content
   hash `refines.base`, and not through a fallback (`base-via-fallback`).
4. The capability the base was decoded through is admitted by *R*'s capability (SPEC
   §5, *Base admission*), at the tier that table requires of it (`base-not-admitted`).
   Then, where *R*'s capability reads side planes, each entry of `side` in order
   (§3.1): the sink holds its plane (`side-absent`), and the plane is exact
   (`side-not-exact`).
5. Where the sink verified a delivery receipt for the base, it has verified one for
   *R* as §10.1 requires (`refinement-unattested`).
6. The payload is well formed for *R*'s capability, in this order: it parses in its
   staged form (Appendix A; Appendix B for the conditioned form) (`payload-malformed`);
   its unit count is the number of units the layer covers, and, for the conditioned
   form, its `n_side` the number of entries `side` names (`payload-malformed`); its
   declared cost (Appendix A.5, B.5) is within the limit the sink applies, if it applies
   one (`cost-over-limit`); retaining *R* keeps the chain within its budget (§4.5)
   (`chain-over-limit`); and every unit's stream decodes to its last symbol and ends
   canonically (`payload-malformed`). A payload
   malformed for one unit is malformed for all. No unit's shape is read before step
   9. A payload of the level form (§6.2) parses as [neural-fields](neural-fields.md)
   §7.2 states, is one unit, and has no stream and no declared cost.
7. Let *P(u)* be the producer `prev` gives for *u*. *u* is **conflicted** when it is
   frozen, or when an envelope with different content, the same `base` and the same
   `layer`, whose producer for *u* is also *P(u)*, has passed step 8 for *u*, earlier
   (before the base's chain was dropped included, §4.4) or in the same batch (§4.4)
   (`layer-conflict`). A conflicted unit is **frozen**:
   the sink **MUST** apply no further layer to it, except the replay of its own record
   after its base's chain is dropped and the base decoded again (§4.4); it keeps the
   output it had (a layer applied to it before the conflict stays applied), and the
   sink **MUST** report the conflict.
8. *u* is **ready** when its stage is `layer − 1` and its producer is *P(u)*. A unit
   whose stage is below `layer − 1` keeps *R* held for it, and is evaluated again when
   its stage advances; at the bound *R* is discarded for that unit (`layer-gap`). Any
   other unit is not ready (`prev-mismatch`).
9. *R*'s payload reads *u*'s current output, and each side plane step 4 found, in the
   shape its staged form needs (`geometry-mismatch`).
10. Under a live schedule, *R* arrived at least `lead_ms` before *u*'s presentation
    time (§8) (`deadline-missed`).

The sink then computes, for every unit that passed, the next stage by *R*'s staged
form (§6), and sets the unit's stage to `layer` and its producer to *R*'s content
hash. Application to one unit is atomic. A sink **MUST NOT** present a unit's output
from a layer that failed for that unit, and **MUST NOT** present a partly applied
unit.

### 4.2 Reasons

| step | reason | scope | effect |
|---|---|---|---|
| 1 | `refines-malformed`, `capability-not-refinement` | envelope | discarded |
| 2 | `base-absent` | envelope | held, then discarded |
| 2 | `refines-malformed` | envelope | discarded |
| 3 | `base-via-fallback` | envelope | discarded |
| 4 | `base-not-admitted` | envelope | discarded |
| 4 | `side-absent`, `side-not-exact` | envelope | discarded |
| 5 | `refinement-unattested` | envelope | discarded |
| 6 | `payload-malformed`, `cost-over-limit`, `chain-over-limit` | envelope | discarded |
| §4.5 | `chain-over-limit`, for a base | the base's envelope | the base is not added; its layers stay held |
| 7 | `layer-conflict` | unit | not applied; the unit is frozen |
| 8 | `prev-mismatch` | unit | not applied to the unit |
| 8 | `layer-gap` | unit | held for the unit, then not applied |
| 9 | `geometry-mismatch` | unit | not applied to the unit |
| 10 | `deadline-missed` | unit | not applied to the unit |

The list is closed. Two conforming sinks that receive the same envelopes in the same
order, at the same times, with the same receipts, hold bound, cost limit, chain budget
and schedule, report the same reasons per envelope and per unit and reach the same
stages and producers. The order of arrival changes the outcome only through the chain
budget at step 6 (§4.5), step 7, the hold bound at step 8 (§4.3: what is refused on
arrival and what expires) and step 10.

### 4.3 Holding

A sink **MUST** declare the bound to which it holds envelopes it cannot yet apply, as
a count of envelopes and a duration from arrival. An envelope held past the duration
is discarded with its hold reason (`base-absent`, or `layer-gap` for each unit it is
held for). An envelope that would exceed the count is discarded on arrival with the
reason it would have been held for. A unit refined by a held envelope that is
released within the bound, and that the chain budget admits, reaches the same output
as one refined in order.

The bound holds envelopes for their base only. A layer whose side plane the sink does
not hold is discarded (§4.1 step 4, `side-absent`), so a sink acquires a layer's side
planes before it offers the layer: it decodes the object, or renders the frame, first, and
a layer it receives again once it holds them is processed as any arrival.

An envelope held for its base and received again is held once. If it comes with
stronger evidence of its delivery receipt (§10.1: a receipt verified over its bytes is
stronger than one verified by inclusion alone, which is stronger than none), the sink
keeps the stronger evidence for steps 5 and 10; its hold still runs from its first
arrival.

### 4.4 Batches

When the base an envelope names is decoded, the envelopes held for it are released
together; when a unit's stage advances, the envelopes held for that unit are released
together. Each envelope released, or offered on arrival, passes steps 1–6, and for
each unit step 7 against the envelopes that passed step 8 earlier and step 8, alone.
The (envelope, unit) pairs that are ready then form one **batch**: step 7 compares
every pair of the batch with the others, and only then do steps 9 and 10 and
application run. So two conflicting layers released together are both refused and
the unit frozen, whichever arrived first; a layer that conflicts with one that passed
step 8 earlier is refused, and the unit frozen at the stage it has. When a unit is
frozen, every envelope held for it is refused for it (`layer-conflict`). Applying a
batch can release another, which is evaluated the same way.

A sink that decodes again a base whose chain it holds keeps the chain: re-decoding a
base does not reset its units' stages, producers or frozen state, and releases and
reports nothing. If the sink has verified the base's delivery receipt since, the base
is attested from then on (§10.1). Attestation belongs to the base's content hash: a base
once attested stays attested, held or dropped, and a decode that carries no receipt
does not undo it; a receipt the sink verifies before it decodes the base makes the base
attested when it is decoded. Only forgetting the base (§14) ends it. A sink **MAY** drop a base's chain when it no longer presents the
base. Every envelope held for the chain's units is then discarded (`base-absent`,
reported once for each), and an envelope for that base is held as for any base the
sink has not decoded. The sink keeps each unit's **record**: the producers of the
stages it reached, the envelope that passed step 8 at its last stage without being
applied, and whether it is frozen. A record is empty when the unit reached no stage,
nothing passed step 8 for it and it is not frozen; the sink keeps only the others, and
nothing of a base all of whose records are empty and that is not attested. When it
decodes the base again, each unit
starts at stage 0 with its record. Step 7 compares a layer with the recorded
producers, so a unit advances again only along them: a layer whose key a recorded
stage holds with other content conflicts, and the unit freezes. A frozen unit stays
frozen: it may replay its own recorded producers, in order, up to its recorded stage
and never past it, and every other layer is refused for it (`layer-conflict`). So it
presents its base's output or the output of one of its recorded producers, never any
other.

A sink **MUST** declare a **record budget**, in bytes, for what it keeps of the bases it
does not hold: their records, and the attestation of a base dropped while attested or
attested before it was decoded (§10.1). A kept base counts 32 bytes, and each record it
keeps 8 bytes, 32 for each recorded stage and 32 for an envelope that passed step 8
without being applied. When what it keeps passes the budget, the sink forgets whole
bases, the one whose record or attestation it wrote longest ago first, until it is
within it. A forgotten base is as one the sink never decoded and never attested
(§14).

These counts are the interoperable measure: every conforming sink counts the same
bytes, so two sinks with the same record budget keep and forget the same bases. What an
implementation allocates for each counted byte is its own, and it states that ratio
beside its record budget.

### 4.5 Chain budget

A sink **MUST** declare a budget, in bytes, for the chain of each base it decodes. The
chain's **state** is the sum of:

- for every unit, the bytes of its current output (1 per u8 element, 2 per i16
  sample), and 32 bytes for each stage its record holds (the producers step 7
  compares: its stages above 0, or, for a base decoded again, the stages it reached
  before, §4.4);
- for every layer the chain **retains**, its payload's bytes and its declared working
  bytes (Appendix A.5), and, for each unit it covers and has not been applied to, one
  unit's output (its declared output over its units) and 32 bytes (the stage it would
  add).

A chain retains a layer from the moment it passes step 6 to the end of the offer,
release or expiry after which no unit is held for it; the sink then drops it. At step
6, after the cost limit, a layer whose payload, declared output and working bytes and
32 bytes for each unit it covers, added to the state, pass the budget is refused
(`chain-over-limit`) before any stream is decoded. An envelope the chain still
retains, received again, adds nothing. Applying a layer to a unit replaces its output
with one, and adds a stage, that the layer's retention counted, and that unit's share
leaves the layer's: an application lowers the state by the unit's old output (and by
32 bytes more where it replays a recorded stage). The
state rises only when a layer is admitted, so it never passes the budget, and the
bytes a chain holds never pass the state. A side plane (§3.1) is held by the sink, not
by a chain, and is no part of a chain's state; a sink bounds the objects it holds for
side planes as it bounds any object it decodes.

A chain starts with its base. A base whose units' output, with 32 bytes for each stage
the records the sink keeps of it hold (§4.4), passes the budget is not added: the sink
reports `chain-over-limit` for the base's envelope, keeps no chain for it, leaves the
records it keeps of it as they were, and holds the layers that name it as for a base it
has not decoded (§4.3). A receipt verified with the base is recorded all the same
(§10.1). The base's own decode is bounded before it starts: where its capability declares
a cost before decoding (`extensions/integer-payloads.md` §7), a sink applies its cost
limit to that declared cost first, and SPEC §4 treats a base past it as one this sink
does not decode (`cost_over_limit`), so dispatch moves on to the fallback.

## 5. The tier of a stage, and the stage hash

- **The base step**: `decode-equivalence` when the capability the base was decoded
  through has the registry class `StandardDefined`, or `NeuralIntegerExact` with a pin
  that resolved (SPEC §3.1) and declares `decode-equivalence`, or is `wai.video.av1` over a
  grain-free stream whose frames are the reference decode's or a platform decoder's whose
  probe matched ([platform-decode](platform-decode.md) §6.2);
  `entropy-consistency` when the class is `NeuralEntropyExact`; otherwise none.
- **A layer step**: the tier its registration gives (§6); `decode-equivalence` for the
  residual form and the level form, and for the conditioned form, whose side planes are
  exact (§4.1 step 4).

A unit's tier is the weakest step of its chain, composed as
[derived-tracks](derived-tracks.md) §3 rule 1. A layer acts on reconstructed output,
so `entropy-consistency` does not pass through it (derived-tracks §3 rule 2): a unit
over such a base holds no tier once a layer applies.

The **stage hash** of a unit is the SHA-256 of its output in the canonical buffer of
SPEC §7 *Digests*: RGB8 for a picture or a frame; the samples as signed 16-bit
little-endian integers, channels interleaved, for audio; the attribute tensor's bytes,
plane after plane, for a splat; for a field, the RGB8 frame of its render at its own view
([neural-fields](neural-fields.md) §7.4). The stage hash of several units is the SHA-256 of
their 32-byte stage hashes concatenated in unit order, as that section defines a clip
digest. A stage hash **MUST NOT** be published, compared as a conformance value, or
sealed in a receipt unless every unit it covers holds `decode-equivalence`.

## 6. Staged forms

A refinement-only capability's registration fixes its staged form, payload format,
admitted bases (SPEC §5, *Base admission*), tier and truncation points.

### 6.1 Residual (`WIR1`)

`wai.image.int_refine`, `wai.video.int_refine`, `wai.audio.int_refine` and
`wai.splat.int_refine` carry a `WIR1` payload (Appendix A): one rANS stream per
unit. For each element of a unit, the next stage is `clamp(pred + r · q)`. `pred` is
the payload's predictor applied to the unit's current output; `r` is the decoded
residual; `q` is the payload's step; and the clamp is to the element type's range.
Every operation is on integers of at most 64 bits.

| capability | unit | element | predictors |
|---|---|---|---|
| `wai.image.int_refine` | the picture, RGB8 | u8 | identity, up2 |
| `wai.video.int_refine` | one frame, RGB8 (over a `wai.video.av1` base, the frame converted by [platform-decode](platform-decode.md) §6.3) | u8 | identity, up2 |
| `wai.audio.int_refine` | the clip, `i16` PCM, channels interleaved | i16 | identity, rate |
| `wai.splat.int_refine` | the attribute tensor | u8 | identity |

A residual layer reads no parameter set: everything its decode reads is in its
payload and in the unit's current output. The base's parameter set was checked when
the base was decoded (SPEC §3.1, check 5 among them), and nothing a layer reads needs
the check again. Tier: `decode-equivalence`. Every layer boundary is a truncation
point.

### 6.2 Level (`WIL1`)

`wai.field.int_refine` carries a `WIL1` payload ([neural-fields](neural-fields.md) §7.2): one
grid level of an integer neural field, over the base `wai.field.int_grid`. The unit is the whole
field. The next stage is the field with that level applied as its next level (neural-fields
§7.3): with the field holding levels 0 to `h − 1`, the payload's level is `h`, of the features,
table size and side the field declares for it, or §4.1 step 9 fails (`geometry-mismatch`). Layer
`k` of a field whose payload carries `K` levels carries level `K + k − 1`. A level reads no
parameter set: everything its decode reads is in its payload and in the field below. Tier:
`decode-equivalence`. Every layer boundary is a truncation point.

### 6.3 Conditioned residual (`WCP1`)

`wai.image.int_cond_refine` carries a `WCP1` payload (Appendix B): an integer network and
a residual. Its unit is the picture, RGB8, and the next stage is three steps: the **base
prediction**, the unit's current output (identity) or its up2 at twice the resolution, as
§6.1's up2; the **network's prediction**, the base prediction plus a correction an
integer network computes from it and the side planes the layer names (§3.1), clamped to
`[0, 255]`; and the residual, decoded over the network's prediction as §6.1 decodes a
unit with the identity predictor. Every operation is on integers of at most 64 bits.

One mechanism carries two uses. Over rendered content, the side planes are the G-buffers
both sides hold: the exact components the origin sends once, or the frame the sink
renders itself from a splat set at a pinned pose, and the network turns them and the
base into a prediction the residual corrects. Over a base at a lower resolution, the
layer is the detail stream of a dual stream: the up2 base prediction is the reference,
the network the detail conditioned on it, the residual what remains. A layer with no
`side` reads no side plane.

The sink composes the stage from objects of their own: the base, each side object and
the layer, each named by its content hash and covered by its own receipt (§10.1). It
computes the stage itself, and the stage hash (§5) is the digest a `stage` claim states
(§10.2) and a sink that presents the picture checks.

Its network is carried in its payload and it reads no parameter set; the network runs
through one graph of the integer operator set (Appendix B.6), and every payload that
parses meets `acc64`. Its admitted bases are `wai.image.int_refine`'s (SPEC §5, *Base
admission*). Tier: `decode-equivalence`, given exact side planes (§4.1 step 4). Every
layer boundary is a truncation point.

### 6.4 Other staged forms

This revision registers the residual form (§6.1), the level form (§6.2) and the
conditioned residual (§6.3). A refinement-only capability of
another staged form (the continuation of a world's state, an enhancement layer
carried as its own object, a byte prefix of a progressive codestream, the further
codebooks of a residual-vector-quantised code) is registered by a revision that fixes
its form, payload, admitted bases, tier and truncation points. Where the form decodes
a prefix of a base codec's own codestream, that revision cites the clause of the
codec's standard that defines the decode of such a prefix, and its reference sink
decodes one.

### 6.5 Truncation points

A refinement capability declares truncation points only where its staged form defines
the decode of a prefix. For the residual form, the level form and the conditioned residual
that is every layer boundary.

## 7. Deferral

A layer **MAY** be published after its base, served only on request, or never. A sink
**MUST** present the current stage of a unit without waiting for a layer it has not
been told to expect. A later revision of a refinement is a new chain from the same
`base`, distinguished by `prev` from layer 2 on. Every layer 1 names the base as its
producer, so `prev` cannot distinguish two revisions of layer 1: they are two
envelopes of the same `base`, `layer` and producer, and conflict (§4.1 step 7).

## 8. Live deadlines

Under a live schedule, every unit has a presentation time, given by the base's timing
and the target latency the sink applies, and every layer has a lead, `lead_ms`: the
time before a unit's presentation by which the layer must have arrived, and been
verified under §10.1, to be applied to it. A carriage may declare both: on MoQ, the
tracks' target latency and each layer track's `lead_ms`
([moq-streaming-format](moq-streaming-format.md) §4.1). A sink **MUST NOT** delay presenting a unit's
current stage to wait for any layer. A layer that arrived less than `lead_ms` before a
unit's presentation time is not applied to that unit (`deadline-missed`). It **MAY**
be applied to units of its span that are not yet due. A still image **MAY** be
refined after it is presented. A sink **MUST** count every unit presented below the
stage its expected layers would have reached on time. Verifying a layer's receipt
(§10.1) adds the time to receive and verify that receipt to the layer's arrival, which
sets the least `lead_ms` that can be met: over a base whose receipt the sink verified,
a layer counts as arrived when the sink verified its receipt (§4.3).

A stage set split across bases (a base for each group of frames) has one clock: a base
whose units are the set's units from position `k` on presents its unit `i` at the
set's position `k + i`, and a sink gives each base its first position.

## 9. Carriage

A layer is a `WAI1` envelope, carried and stored as any envelope is, or a `WAI2`
component with `role: "refinement"` (SPEC §7.1, *Refinement components*). A component
and its exploded envelope (SPEC §7.1) are the same object for every rule that names
an object by content hash, so a layer reaches the same stages whichever way it
travelled. Two bindings carry each layer as an object of its own:
[moq-streaming-format](moq-streaming-format.md) §4.1, a track for each layer index,
and [http-delivery](http-delivery.md) §10.4, a resource for each layer.

## 10. Receipts and claims

### 10.1 Attestation of layers

Each layer is an object under [jwp-receipts](jwp-receipts.md). A sink that verified a
delivery receipt for a base **MUST NOT** apply a layer unless it has verified a
delivery receipt for the layer, through that extension's §3 step 6, over the layer's
envelope bytes as received (`delivered-verified`). That receipt must be signed by the
base's signer, or by a key the sink holds with the authority `delivery` under the same
origin (jwp-receipts §7.5). A receipt verified without the layer's envelope bytes is
`inclusion-verified` (jwp-receipts §3, *Verified by inclusion*): it shows that a
publisher signed an object with that content hash, and it never authorizes an
application. A layer on a track of its own has its receipts on a chain of their own,
one per publisher key and track (jwp-receipts §3 step 3), so a sink verifies the layers
it subscribes to without the receipts of the ones it does not. A content hash in
`refines` binds a layer to a base, not to its publisher. A sink that verifies a base's delivery receipt after
it decoded the base applies step 5 to every envelope that arrives after it; envelopes
it admitted before keep the decision they had. The verification belongs to the base's
content hash and holds across the base's chain being dropped and the base decoded
again (§4.4).

### 10.2 Claims

A party that observed a stage set **MAY** sign a `stage` claim about its base
(jwp-receipts §2.4.10): the stage it asked for, the stage it was promised, the stage the
objects it verified over their bytes reach, the stages it presented on time, the tier
and stage hash of what it presented, and every layer object with what became of it and
what it verified of the object's receipt. A layer object it never received but whose
receipt it verified by inclusion is listed as `dropped` with the evidence
`inclusion-verified`; one that arrived after its units were due is `late`. A claim
covers what its signer observed since it last forgot the base: a base the sink forgot
(§4.4, §14) is, decoded again, one it never decoded and never attested, so its
observation starts over, and nothing applied to it before is part of the claim. An
envelope the sink still holds for the base when it forgets it stays observed, with no
decision, and is stated as the sink then decides it. A claim never spans two such
epochs. The claim records what its signer observed, not that content
was delivered.

A relay that did not forward a layer object **MAY** sign a `delivery-gap` claim about it
(jwp-receipts §2.4.11), beside the object's receipts, which it forwards all the same
(jwp-receipts §4.1).

## 11. Counting bytes

A figure of delivered bytes for a stage set:

- **MUST** count the bytes of the `WAI1` and `WAI2` envelopes as delivered (SPEC §2:
  magic, length prefixes, manifest and payload);
- **MUST** state separately the receipt, index and transport framing bytes, and the
  parameter-set bytes a sink fetched;
- **MUST NOT** count a text re-encoding of a payload (JSON, base64) in place of the
  payload's registered binary format.

A figure over a payload in any other form is not a staged-delivery byte figure.

## 12. Conformance

A conforming sink of this extension implements §4, §5, §8 and §10.1, and every staged
form whose capabilities it advertises. The corpus
[`staged-conformance/`](../staged-conformance/) holds the cases: for each, a
conforming sink reports the stated reason per envelope and unit, and reaches the
stated stage, producer, tier and stage hash per unit. Its `README.md` states the
cases, what each base's stage 0 is checked against, and the independent verifier
`staged_verify.py`. The cases of the conditioned form, with the side objects a sink holds
and the frames it renders, are its `cond/`, which a sink that advertises
`wai.image.int_cond_refine` reproduces.

## 13. Earlier layered carriage

A prototype MoQ binding in this repository carries a layer index in an object
header (`mrl_layer`, with `mrl_total_layers` and a subscriber's `max_mrl_layer`). For
an object that is a `WAI1` envelope with `refines`, a sink reads the layer from
`refines.layer`, which sits inside the envelope and so inside its content hash, and
ignores the header. On a track of packaging `wai`, a subscriber caps its stage by the
layer tracks it subscribes to, and a relay reads no header or envelope to cap it
([moq-streaming-format](moq-streaming-format.md) §4.1).

## 14. Security considerations

- A layer is not authenticated by its `refines`; §10.1 authenticates it.
- Deferral lets anyone mint a chain for a public base hash. §10.1 and §4.1 step 7 bound
  what a sink presents. A conflict between two attested chains is evidence that an
  authorized signer signed both, and the sink reports it; a `stage` claim (§10.2)
  records it as `conflicts`.
- A hold bound (§4.3), a cost limit (Appendix A.5) and a chain budget (§4.5) are
  resource limits. A layer's header declares its output, its symbols and its working
  bytes before any symbol is decoded, so a sink refuses a layer past its limit or its
  budget without decoding it. A base of an integer codec declares its cost the same way
  (`extensions/integer-payloads.md` §7), so a sink refuses a base past its limit before
  decoding it, and one whose output passes the chain budget before keeping it (§4.5):
  a few bytes of envelope can declare a large output, and only the declared cost bounds
  the work. A sink's own work for a layer is linear in the
  layer's payload bytes and declared symbols, up to a factor logarithmic in its
  units: it parses the payload once, at step 6; decides each unit the layer covers
  once for each offer or release of the layer; finds a unit's producer in `prev` by
  bisection over its runs; and compares a unit's key at step 7 with the producers of
  that unit's stages. Beyond the chain's state, it keeps a fixed amount per unit of
  the base and per unit held.
- A side plane is named by the content hash of the object it comes from, so a layer
  reads the bytes it was made against or none (§3.1). A plane that is not exact is
  refused (`side-not-exact`), so a sink never presents a stage whose prediction rests on
  an output another sink computes differently. A render entry costs the sink a render
  (splat-exact §5) at a pose of at most 2^20 pixels; a sink renders only for a layer it
  accepts to process. A layer's declared cost counts its network's working bytes and
  multiply-accumulates (Appendix B.5), so a sink refuses a network past its limit before
  it runs.
- An encrypted layer ([confidentiality](confidentiality.md)) binds to the content hash
  of its base's envelope as delivered, ciphertext included, and is decrypted before
  §4.1 step 6.
- A sink keeps the record of a base whose chain it dropped (§4.4), so a base decoded
  again cannot be refined along other producers than it had; the record counts in the
  chain's state when the base is decoded again (§4.5), and, while the base is not
  held, in the sink's record budget (§4.4). A sink that forgets a base, by
  `StageSink::forget_base` in the reference or because its record budget is full,
  treats it as never decoded and never attested: a layer that conflicted with the
  forgotten history can apply, and step 5 no longer requires a receipt for it. A sink
  sizes its record budget for the bases it may decode again.
- Where the base is attested (§10.1), a frozen unit presents only its base's output or
  the output of one of the producers it recorded, replayed in order (§4.4): a party
  that can sign conflicting layers can stop a unit's refinement, and cannot make it
  present anything else. Over a base whose receipt the sink did not verify, anyone who learns the
  base's hash can mint a layer, and the first one ready for a unit applies to it.

## Appendix A. `WIR1`

### A.1 Layout

Every multi-byte field is little-endian; offsets are in bytes.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIR1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `elem` | `1` = u8, `2` = i16 |
| 6 | 1 | `predictor` | `0` identity, `1` up2, `2` rate |
| 7 | 1 | `table_mode` | `0` one table for every unit, `1` one per unit |
| 8 | 4 | `n_units` | ≥ 1 |
| 12 | 4 | `d0` | ≥ 1: rows (picture), samples per channel (audio), planes (splat) |
| 16 | 4 | `d1` | ≥ 1: columns, `1`, the planes' width |
| 20 | 4 | `d2` | ≥ 1: channels (`3` for RGB8), channels, the planes' height |
| 24 | 4 | `rate_num` | predictor 2: the output rate, ≥ 1; otherwise `0` |
| 28 | 4 | `rate_den` | predictor 2: the rate of the stage below, ≥ 1; otherwise `0` |
| 32 | 4 | `q` | 1..=65535 |
| 36 | 4 | `offset` | `i32`: the symbol offset of every table |
| 40 | 2 | `cdf_len` | 3..=4097 |
| 42 | 2 | reserved | `0` |
| 44 | T·`cdf_len`·4 | the tables | T = 1 or `n_units`; each a table of [integer-payloads](integer-payloads.md) §2 |
| … | `n_units`·4 | stream lengths | `u32` each, a multiple of 4, ≥ 8 |
| … | Σ | the streams | one per unit, in unit order; nothing follows |

Each stream is decoded as [integer-payloads](integer-payloads.md) §2 decodes a rANS
stream (precision 16, the bypass escape of 4-bit groups), against the unit's table at
`offset`.

### A.2 The unit kind

A capability fixes what its units hold (§6.1), and so the fields a payload may carry:

- RGB8 (`wai.image.int_refine`, `wai.video.int_refine`): `elem` 1, `predictor` 0 or
  1, `d2` = 3;
- PCM (`wai.audio.int_refine`): `elem` 2, `predictor` 0 or 2, `d1` = 1;
- attribute tensor (`wai.splat.int_refine`): `elem` 1, `predictor` 0.

### A.3 Order of the checks

A parser checks, in this order, and refuses with the first that fails:

1. the magic (`bad_magic`);
2. the version byte: present (`stream_length`), and 1 (`version`);
3. the 44-byte header: present (`stream_length`);
4. the reserved field (`reserved_nonzero`);
5. the header's fields, in the order `elem`, `predictor`, `table_mode`, `n_units`,
   `d0`/`d1`/`d2`, the rate fields, `q`, `cdf_len`; then the unit kind's rules
   (A.2) (`field_range`);
6. the caps, in this order (`too_large`): a unit's `d0·d1·d2 ≤ 2^26`;
   `n_units ≤ 2^20`; `n_units·d0·d1·d2 ≤ 2^30`;
7. the tables, one at a time, each present (`stream_length`) then checked
   (`cdf_malformed`); the stream lengths, present (`stream_length`), each a multiple
   of 4 and at least 8 (`stream_length`); the streams, present (`stream_length`);
8. nothing after the last stream (`trailing`).

Every size is formed in 64-bit arithmetic with overflow checked, and a size past 64
bits is past the caps, so a 32-bit sink refuses exactly what a 64-bit one does. A
parser reads tables and lengths from the bytes the payload holds and allocates
nothing a payload's bytes do not justify.

### A.4 Decode

For each unit *u* the layer is applied to, with the unit's current output of shape
`(e0, e1, e2)`:

1. The shape must be the one the predictor reads (`geometry_mismatch`), before any
   symbol is decoded: `(d0, d1, d2)` for identity; `(⌈d0/2⌉, ⌈d1/2⌉, d2)` for up2;
   `(n′, 1, d2)` with `⌊n′·rate_num/rate_den⌋ = d0` for rate.
2. For element *i* in raster order (`i = (a·d1 + b)·d2 + c`), decode the residual `r`
   from *u*'s stream. A stream that runs out during a symbol's decode is refused at
   that symbol (`stream_exhausted`), and nothing after it is decoded. The element is
   `clamp(pred(i) + r·q)`, to `[0, 255]` for u8 and `[−32768, 32767]` for i16. `r` lies
   in `[−2^31, 2^31 − 1]` and `|r·q| < 2^47`, so every step fits `i64`.
3. After the unit's last element, the stream ends canonically
   ([integer-payloads](integer-payloads.md) §2) (`stream_noncanonical`).

**Predictors**, with `prev` the unit's current output:

- **identity**: `pred(i) = prev(i)`.
- **up2**: `prev` is `h′ × w′ × c` with `h′ = ⌈d0/2⌉`, `w′ = ⌈d1/2⌉` and `c = d2`,
  so odd sizes are allowed. For output `(y, x, ch)`: `iy = y >> 1`, `fy = y & 1`,
  `ix = x >> 1`, `fx = x & 1`, and `p(Y, X) = prev[min(Y, h′ − 1), min(X, w′ − 1), ch]`.
  The prediction is `p(iy, ix)` when both are 0;
  `(p(iy, ix) + p(iy, ix + 1) + 1) >> 1` when only `fx` is 1;
  `(p(iy, ix) + p(iy + 1, ix) + 1) >> 1` when only `fy` is 1; and
  `(p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2` when
  both are. This is the half-pel arithmetic of [integer-payloads](integer-payloads.md)
  §3.
- **rate**: `prev` holds `n′` samples of `c = d2` interleaved channels. For output
  sample *i* and channel *ch*: `s = i·rate_den`, `lo = s div rate_num`,
  `ρ = s mod rate_num`, `hi = min(lo + 1, n′ − 1)`, `a = prev[lo·c + ch]`,
  `b = prev[hi·c + ch]`, and the prediction is `rdiv(a·rate_num + (b − a)·ρ, rate_num)`,
  where `rdiv(n, d) = sign(n)·⌊(2|n| + d) / 2d⌋` rounds half away from zero. This is
  `a + (b − a)·ρ/rate_num` rounded half away from zero, the linear interpolation at
  source position `i·rate_den/rate_num`, in exact integer arithmetic.

This decode is the stage operators of [integer-operators](integer-operators.md) §5E,
with no graph (§13.5 there): each unit's stream is E1 `rans64_decode` with `t4097`,
`lenient` and `canonical`; the up2 prediction is E11 `up2_predict` and the rate
prediction E12 `rate_predict`; and the element is E9 `residual_step` on the element
type's range. On both ranges E9's 32-bit form holds. The reference decoder decodes
through them.

### A.5 Declared cost

A payload's header declares, before any symbol is decoded: its output,
`n_units·d0·d1·d2` elements of 1 byte (u8) or 2 bytes (i16); its symbols,
`n_units·d0·d1·d2`, one entropy-decode step each; and the most bytes its decode holds
at once beyond the payload and the stages below, its parsed tables (`4·T·cdf_len`),
the index of its streams (`8·n_units`) and one unit's output while it is built. It
runs no network. A sink that applies a limit compares this cost with it after the
parse and before any stream is decoded (§4.1 step 6, `cost-over-limit`), and counts
it in the chain's state while it retains the layer (§4.5).

### A.6 Refusal codes

The list is closed, and its codes are those [integer-payloads](integer-payloads.md) §8
uses for the same faults: `bad_magic`, `version`, `reserved_nonzero`, `field_range`,
`too_large`, `cdf_malformed`, `stream_length`, `trailing`, `stream_exhausted`,
`stream_noncanonical` and `geometry_mismatch`. A parser reports the first that
applies, in the order of A.3. §4.1 runs A.4 in two parts: step 6 decodes every unit's
stream to its end (A.4 steps 2 and 3, which read no prediction), so
`stream_exhausted` and `stream_noncanonical` are the envelope's `payload-malformed`,
reported before any unit's shape is read; step 9 then checks each unit's shape (A.4
step 1), so `geometry_mismatch` is that unit's `geometry-mismatch`. A payload whose
`n_units` is not the number of units the layer covers is the envelope's
`payload-malformed` with the code `field_range`. Every other code is the envelope's
`payload-malformed`.

## Appendix B. `WCP1`

The payload of `wai.image.int_cond_refine` (§6.2). Its refusal codes are Appendix A.6's,
and §4.1 reads them as it reads a `WIR1` payload's.

### B.1 Layout

Every multi-byte field is little-endian; offsets are in bytes.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WCP1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `predictor` | `0` identity, `1` up2: the base prediction |
| 6 | 1 | `n_side` | 0 to 4: the side planes the network reads |
| 7 | 1 | `n_layers` | 1 to 8 |
| 8 | 2 | `act_hi` | 1 to 32767: the ceiling of every hidden activation |
| 10 | 2 | reserved | `0` |
| 12 | … | the layers | `n_layers` layers, one after another |
| … | … | the residual | a `WIR1` payload (Appendix A); nothing follows it |

A layer is the layer of [exact-upscale](exact-upscale.md) §3.1:

| size | field | rule |
|---|---|---|
| 1 | `k` | 1, 3 or 5: the kernel's side |
| 1 | `shift` | 0 to 31 |
| 2 | `cout` | 1 to 64; the last layer's is 3 |
| `cout` · `cin` · `k` · `k` | weights | signed bytes, in the order `[cout][cin][k][k]` |
| 4 · `cout` | biases | signed 32-bit integers |
| 2 · `cout` | multipliers | unsigned 16-bit integers, each at least 1 |

`cin` is `3 · (1 + n_side)` for the first layer, the base prediction's three channels and
each side plane's, and the previous layer's `cout` for every other. The residual is of the
RGB8 unit kind (A.2) with the identity predictor; its `d0` and `d1` are the output's rows
and columns, and its `n_units` the units the layer covers.

### B.2 Side planes

The network reads, after the base prediction, the planes of the entries of `refines.side`
(§3.1), in their order, each an RGB8 picture of `d0` rows and `d1` columns.

### B.3 Order of the checks

A parser checks, in this order, and refuses with the first that fails:

1. the magic (`bad_magic`);
2. the version byte: present (`stream_length`), and 1 (`version`);
3. the 12-byte header: present (`stream_length`);
4. the reserved field (`reserved_nonzero`);
5. the header's fields, in the order `predictor`, `n_side`, `n_layers`, `act_hi`
   (`field_range`);
6. each layer in turn: its first four bytes present (`stream_length`); its `k`, `shift`
   and `cout` (`field_range`); its weights, biases and multipliers present
   (`stream_length`); each multiplier at least 1 (`field_range`);
7. the residual: at least one byte present (`stream_length`); then Appendix A.3 over the
   bytes that remain, for the RGB8 unit kind, with its codes in its order; then its
   `predictor` identity (`field_range`);
8. the network's caps (`too_large`): its widest tensor, `max(3 · (1 + n_side), the largest
   cout) · d0 · d1` elements, at most 2^26.

### B.4 Decode

For each unit *u* the layer is applied to, with the unit's current output `prev` and the
side planes `S_1 … S_n` (B.2):

1. The shapes must be the ones the payload reads (`geometry_mismatch`), before any symbol
   is decoded and before the network runs: `prev` is `(d0, d1, 3)` for identity and
   `(⌈d0/2⌉, ⌈d1/2⌉, 3)` for up2; each side plane is `(d0, d1, 3)`.
2. **The base prediction** `b`: `prev` for identity; for up2, A.4's up2 of `prev` at
   `d0 × d1`.
3. **Centring.** `x₀[c][y][x] = b(x, y)[c] − 128` for `c` < 3, and
   `x₀[3j + c][y][x] = S_j(x, y)[c] − 128` for the *j*-th side plane (*j* from 1).
4. **The layers**, as [exact-upscale](exact-upscale.md) §3.2 step 3 states them, at the
   output's size: for each layer *i*, `a[o][y][x] = Σ W[o][ci][ky][kx] · x̃ᵢ[ci][y + ky −
   ⌊k/2⌋][x + kx − ⌊k/2⌋]`, with `x̃ᵢ` zero outside the picture; then `v =
   R_half_up((a + bias[o]) · mult[o], shift)` (integer-operators §2); `xᵢ₊₁ = clamp(v, 0,
   act_hi)` for every layer but the last, whose `v`, of 3 channels, is the correction
   `d`.
5. **The network's prediction** `p(x, y)[c] = clamp(b(x, y)[c] + d[c][y][x], 0, 255)`.
6. **The residual**: A.4 steps 2 and 3 for unit *u* of the residual, with `p` as the
   unit's prediction: each element `clamp(p(i) + r · q, 0, 255)`, then the stream's
   canonical end (`stream_exhausted`, `stream_noncanonical`).

A network whose last layer's weights and biases are all 0 gives `p = b`. Every value the
network forms fits `i64`: an input lies in `[−128, 127]` or `[0, act_hi]`, a weight in a
byte, a sum has at most `64 · 25` terms, a bias 32 bits and a multiplier 16 bits.

### B.5 Declared cost

A payload's header declares, before any symbol is decoded: its output and its symbols,
the residual's (A.5); its working bytes, the residual's (A.5) and `3 · d0 · d1` for the
base prediction and `8 · d0 · d1 · max(cin + cout)` over its layers, eight bytes for each
element of the widest pair of consecutive network tensors; and its multiply-accumulates,
`n_units · d0 · d1 · Σ cout · cin · k²` over its layers. A sink that applies a limit
compares this cost with it after the parse and before any stream is decoded (§4.1 step 6,
`cost-over-limit`, naming the figure past it: output bytes, symbols, working bytes, then
multiply-accumulates), and counts the output and working bytes in the chain's state while
it retains the layer (§4.5).

### B.6 Through the integer operator set

The network is one `WHG1` graph ([integer-operators](integer-operators.md) §10). The
lowering `wcp1/1` writes, for a payload:

- `1 + n_side` data inputs, sizes set per decode: tensor 0, the base prediction
  `[3, d0, d1]`, then each side plane `[3, d0, d1]` in order, each with the policy `check`
  over [0, 255];
- then the operators, each one's output taking the next tensor index, every tensor
  `derived` with no exponent: with side planes, `concat` of the inputs in order; `rescale`
  with no multiplier, shift 0 and a `pre` bias of −128 on each channel; for each layer,
  `conv` with its `cout`, kernel `k` × `k`, stride 1, zero padding ⌊`k`/2⌋ on each side,
  one group, its weights at one byte, and an `rq` with one multiplier per channel at four
  bytes, its one `shift`, a `pre` bias per channel at four bytes and the rounding
  `half_up`; after each layer but the last, `clamp(0, act_hi)`; then `add` of tensor 0 and
  the correction; `clamp(0, 255)`, the graph's one output.

The driver forms the base prediction with E11 for up2, supplies the inputs, reads the
output back as RGB8, and decodes the residual with E1 and E9 as a `WIR1` layer's. A sink
loads the graph from its canonical bytes under `acc64` with its bounds certificate
(integer-operators §8): every payload B.3 admits meets `acc64` (B.4). The capability's
registry `ops` member names the set, the profile `acc64`, the lowering `wcp1/1` and E1's
modes (integer-operators §11).