open-standards/wai/extensions/integer-payloads.md

# WAI Extension: Integer Payloads

> Status: Draft. Pins the payload formats of the WAI-native integer codecs that
> [SPEC.md §5](../SPEC.md) registers without one (*Payloads of the WAI-native integer
> codecs*): `WIV1` for `wai.video.int_motion` and `wai.video.int_hyper`, `WIH1` for
> `wai.neural.int_hyper`, `WIA1` for `wai.audio.int_codec` and `WIS1` for
> `wai.splat.int_codec`. Defines each layout, its decode, the checks a sink makes and
> the closed list of refusal codes; names each capability's parameter-set members;
> and makes the JSON the encode and export tools write a converter input, never a
> payload. Writes down the payload formats and parameter sets that SPEC §5 keeps for
> `wai.neural.int_synth` (§13) and `wai.neural.int_mlicpp` (`WIM1` and `WMB1`, §14).
> Reference impl: `wai-rs` — `int_payload::{wire, decode_int, resolve}`,
> `int_payload::{wiv1_from_json, wih1_from_json, wia1_from_json, wis1_from_json}`,
> `codecs::PINNED_INT_PAYLOADS`, `container::check_int_payload_framing`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope

| capability | payload | parameter set (§9) |
|---|---|---|
| `wai.video.int_motion` | `WIV1` with keyframe 0 (§3) | none |
| `wai.video.int_hyper` | `WIV1` with keyframe 1, a nested `WIH1` (§3) | several files: `entropy.json`, `g_s.whm`, `h_s.whm` |
| `wai.neural.int_hyper` | `WIH1` (§4) | several files: `entropy.json`, `g_s.whm`, `h_s.whm` |
| `wai.audio.int_codec` | `WIA1` (§5) | one file: `model.wam` |
| `wai.splat.int_codec` | `WIS1` (§6) | one file: `g_s.whm` |

**Why this pin is permitted.** Until this revision, these capabilities' SPEC §5 rows
named the decoder that read their payload, and no byte layout. The rows of
`wai.video.int_motion`, `wai.audio.int_codec` and `wai.splat.int_codec` also named the
encode or export tool that wrote a bitstream, a JSON document; those of
`wai.neural.int_hyper` and `wai.video.int_hyper` named their decoders only. SPEC §5
called every row's payload the bytes its codec normally emits, which for these rows
was the tools' JSON. So this revision changes what these strings carry. SPEC §8
permits it under *Completing a registration*: a revision may complete the
registration of a WAI-native string whose row named a decoder and fixed no byte layout
by pinning one, and it lists the envelopes that become non-conforming (§11; SPEC §5).
SPEC §8's rule that a registered string's payload format is fixed forever applies to
these strings from this revision on. The encode and export tools still write JSON
that carries the same fields; it is a converter input (§10).

`wai.neural.int_synth` and `wai.neural.int_mlicpp` keep the payload formats SPEC §5
gives them, and §13 and §14 write those formats down. A sink decodes them and the five
strings above (`wai.video.int_motion`, `wai.video.int_hyper`, `wai.neural.int_hyper`,
`wai.audio.int_codec`, `wai.splat.int_codec`) through one path
(`int_payload::decode_int`), and §8 names the codes it reports for those two.

## 2. Common rules

**Encoding.** Every multi-byte field is little-endian. Offsets are in bytes.

**Framing.** A payload starts with its four-byte magic and a version byte. This
revision defines version 1 for every format. Every reserved field is zero. Nothing
follows the last section.

**rANS streams.** A stream is a `u32` length `L` followed by `L` bytes. `L` is a
multiple of 4 and at least 8. It is decoded as follows (precision 16, with a bypass
escape of 4-bit groups):

- The state `x` starts as `(w1 << 32) | w0`, from the stream's first two `u32`
  words. *Renormalising*: if `x < 2^31`, then `x = (x << 32) | w`, where `w` is the
  next unread word; if no word is left, the stream has **run out** and `x` is left
  as it is.
- `bits(n)`: `v = x mod 2^n`, `x = x >> n`, renormalise, return `v`.
- Decoding a symbol against a table `cdf` of length `n` with offset `o`: let
  `cum = x mod 2^16` and `s` be the largest index with `cdf[s] ≤ cum`
  (`0 ≤ s ≤ n − 2`); then `x = (cdf[s+1] − cdf[s]) · (x >> 16) + cum − cdf[s]`, and
  renormalise. If `s < n − 2`, the value is `s`. If `s = n − 2` (the escape): let
  `v = bits(4)` and `count = v`; then, at most 8 times, while `v = 15` and the stream
  has not run out, `v = bits(4)` and `count = count + v`; then `count = min(count, 8)`
  and `raw = Σ_{j < count} bits(4) << 4j`. The value is `−(raw >> 1) − 1` if `raw`
  is odd and `(raw >> 1) + (n − 2)` if it is even, saturated to `i32`. The symbol
  is `value + o`, saturated to `i32`.
- A decode stops at the first symbol during whose decode its stream runs out, and
  refuses the payload (`stream_exhausted`): the value that symbol would give is not
  used, and nothing after it is decoded. A stream that never runs out decodes every
  symbol, so a conforming payload is unaffected.
- After the last symbol a decode reads from a stream, the stream ends **canonically**:
  every word of it has been read, and the state is `2^31`. Otherwise the payload is
  refused (`stream_noncanonical`). An encoder starts its state at `2^31` and writes
  the words it flushes and no others, so its streams end this way. A word after the
  last symbol's, or a state the encoder did not start from, codes the same symbols in
  other bytes; the rule closes both, so neither can give one decode several content
  hashes. A stream that codes no symbol (the residual stream of a one-frame `WIV1`
  with keyframe 1) is the state `2^31` alone: the words `2^31` and `0`.

The decode these rules define is [integer-operators](integer-operators.md) E1,
`rans64_decode`, with the modes `t4097`, `lenient` and `canonical`.

**One clip, several payloads.** These rules fix how a payload is read, not the
only payload of a decoded output: tables that differ in entries the stream never
selects (a symbol the clip does not use, say), or a different escape layout, code
the same output in other bytes. A payload's hash therefore identifies the payload,
not what it decodes to. Where one output must have one identity, it is identified by
the digest of its decoded output: for a clip, its frame-state-equivalence digests
(SPEC §7, a video receipt's `clip_hash`); for the other formats, their capabilities'
equivalence digests.

**CDF tables.** A table is `n` entries of `u32` with `3 ≤ n ≤ 4097`, `cdf[0] = 0`,
`cdf[n − 1] = 65536` and every entry greater than the one before. These rules are
what keep the symbol search and the state update above in range.

**Order of the checks.** A parser checks, in this order, and refuses with the first
that fails:

1. the magic: the capability's (§1), or else `format_capability_mismatch` if it is
   another of the four magics, or `bad_magic`;
2. the version byte: present (`stream_length`), and 1 (`version`);
3. the fixed header: present (`stream_length`);
4. the reserved fields (`reserved_nonzero`);
5. the header's fields, in the order the format's section lists their rules
   (`field_range`), then, for `WIV1`, the keyframe kind (`keyframe_capability_mismatch`);
6. the caps of §7 on the sizes the header declares (`too_large`), in the order the
   format's section lists them;
7. each later section in order: present (`stream_length`), then its rules
   (`field_range`, `cdf_malformed`). A section of one table per channel (`WIA1`,
   `WIS1`) is read table by table: each table present, then checked, before the next;
8. nothing after the last section (`trailing`, or `nested_length` for `WIV1`'s
   nested keyframe).

A decode then makes the checks of §7, and refuses a stream that runs out or does not
end canonically (§2). Two conforming sinks refuse the same bytes with the same code.

## 3. `WIV1` — integer inter-frame video

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIV1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `mv_precision` | `1` (whole-pel) or `2` (half-pel); `1` when `n_frames` is 1 |
| 6 | 1 | `keyframe` | `0` or `1` |
| 7 | 1 | reserved | `0` |
| 8 | 4 | `n_frames` | ≥ 1 |
| 12 | 4 | `h` | ≥ 1 |
| 16 | 4 | `w` | ≥ 1 |
| 20 | 4 | `block` | ≥ 1; `1` when `n_frames` is 1 |
| 24 | 4 | `q` | 1..=65535 |
| 28 | 4 | `offset` | `i32` |
| 32 | 2 | `cdf_len` | 3..=4097 |
| 34 | 2 | reserved | `0` |
| 36 | `cdf_len`·4 | the residual table | §2 |
| … | (`n_frames` − 1)·`nb`·4 | motion vectors | `i16 dy, i16 dx` |
| … | 4 + `L` | the residual stream | §2 |
| … | 4 + `K` | keyframe 1 only: `u32 K`, then a `WIH1` payload | `K` is exactly the bytes that remain |

The header's field rules are checked in the order `mv_precision`, `keyframe`,
`n_frames`, `h` and `w`, `block`, `q`, `cdf_len`, then, for a one-frame clip,
`mv_precision` 1 and `block` 1: such a clip codes no motion vector, so neither field
would change its decode, and they are fixed so that it has one header. `keyframe` 0 is
`wai.video.int_motion`'s and 1 is `wai.video.int_hyper`'s; the other is
`keyframe_capability_mismatch`. The frame `h·w·3`, the clip `n_frames·h·w·3`, the frame
count `n_frames` and the motion-vector field `2·nb·(n_frames − 1)` are capped (§7), in
that order.

**Motion vectors.** `nb = ⌈h/block⌉·⌈w/block⌉` and `nbx = ⌈w/block⌉`. The vectors
are frames 1 to `n_frames − 1` in order, each its block grid row-major: the vector of
pixel `(y, x)` in frame `f` is entry `(f − 1)·nb + (y div block)·nbx + (x div block)`,
in units of `1/mv_precision` pel.

**Decode.** One decoder reads the residual stream for the whole clip, against the
residual table at `offset`. For each frame in order, for each pixel in raster order
`(y, x)`, for each channel `c` of R, G, B: decode the symbol `r`, and the sample is
`clamp(pred + r·q, 0, 255)`. Since `|r| < 2^31` and `q < 2^16`, every step fits `i64`.

- Frame 0 with keyframe 0: `pred = 128`.
- Frame 0 with keyframe 1: the frame is the nested `WIH1`'s decode (§4), which reads
  no symbol of the residual stream. It must be `h × w` (`geometry_mismatch`).
- Frame `f ≥ 1`: `pred` reads the previous **decoded** frame `R` with the pixel's
  vector `(dy, dx)`, where `p(Y, X)` is `R` at row `clamp(Y, 0, h − 1)`, column
  `clamp(X, 0, w − 1)`, channel `c`:
  - whole-pel: `pred = p(y + dy, x + dx)`;
  - half-pel: `iy = y + (dy >> 1)`, `fy = dy & 1`, `ix = x + (dx >> 1)`,
    `fx = dx & 1` (arithmetic shift); then `p(iy, ix)` when both are 0,
    `(p(iy, ix) + p(iy, ix + 1) + 1) >> 1` when only `fx` is 1,
    `(p(iy, ix) + p(iy + 1, ix) + 1) >> 1` when only `fy` is 1, and
    `(p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2`
    when both are.

The output is `n_frames` RGB8 frames of `h × w` (rows top to bottom, pixels left to
right, R, G, B bytes): the buffers of the capability's frame-state-equivalence digests
(SPEC §7 *Digests*).

This decode is the stage operators of [integer-operators](integer-operators.md) §5E,
with no graph (§13.5 there): the residual stream is E1 `rans64_decode` with `t4097`,
`lenient` and `canonical`, the prediction of frames `f ≥ 1` is E10 `mc_predict`, and the
sample is E9 `residual_step` on `[0, 255]`. The reference decoder decodes through them.

## 4. `WIH1` — the integer conditional image codec

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIH1` | |
| 4 | 1 | version | `1` |
| 5 | 3 | reserved | `0` |
| 8 | 4 | `m` | ≥ 1: latent channels |
| 12 | 4 | `sh` | ≥ 1: latent height |
| 16 | 4 | `sw` | ≥ 1: latent width |
| 20 | 4 | `n_z` | ≥ 1: hyper-latent channels |
| 24 | 4 | `zh` | ≥ 1: hyper-latent height |
| 28 | 4 | `zw` | ≥ 1: hyper-latent width |
| 32 | 4 + `Lz` | the hyper-latent stream | §2 |
| … | 4 + `Ly` | the latent stream | §2 |

The caps (§7) apply to `n_z·zh·zw` and to `2·m·sh·sw`.

**Decode,** against the set of §9 (`g_s.whm` and `h_s.whm`, two integer networks at
one fixed-point scale `2^afrac`; `entropy.json`, the tables):

1. For each hyper-latent channel `k` and each of its `zh·zw` elements, decode a
   symbol `s` from the hyper-latent stream against channel `k`'s table and offset;
   the element is `s·2^afrac + median_k`.
2. `h_s` on the hyper-latent `[n_z, zh, zw]` gives `[2·m, sh, sw]`: its first `m`
   channels are the scales `σ`, the rest the means `μ`. `h_s` decodes through its
   lowering `whm1/1` ([integer-operators](integer-operators.md) §13.2); the split of its
   output into `σ` and `μ` is a step of this decode, after the shape check, not part of
   the lowered graph (integer-operators §13.5).
3. For each latent element `p` in order (channel, then row, then column), its table
   is `b`, where, with the scale edges `e_0 … e_{N−1}` and `σ' = max(σ_p, e_0)`,
   `b = N − 1 − #{ i < N − 1 : σ' ≤ e_i }`. Decode a symbol `r` from the latent
   stream against table `b` and its offset; the element is `r·2^afrac + μ_p`.
4. `g_s` on the latent `[m, sh, sw]` gives three channels; each value `v` is clamped
   to `[0, 2^afrac]` and becomes `(v·255 + 2^(afrac−1)) >> afrac`, as RGB8. `g_s`
   decodes through its lowering `whm1/1` (integer-operators §13.2), and this is the RGB8
   step that ends its lowered graph (§13.0).

Every value is formed in `i64` and must fit it (§7).

## 5. `WIA1` — the integer learned audio codec

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIA1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `afrac` | 1..=30; equal to the model's (§7) |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `sr` | ≥ 1: the sample rate the PCM is presented at, in hertz |
| 12 | 4 | `n_ch` | ≥ 1: latent channels |
| 16 | 4 | `lat_len` | ≥ 1: latent length |
| 20 | 4 | `t_out` | ≥ 1: samples presented |
| 24 | 4 | `offset` | `i32`: the symbol offset of every channel's table |
| 28 | 2 | `cdf_len` | 3..=4097 |
| 30 | 2 | reserved | `0` |
| 32 | `n_ch`·8 | medians | `i64` each, `|median| ≤ 2^47` |
| … | `n_ch`·`cdf_len`·4 | one table per channel | §2 |
| … | 4 + `L` | the latent stream | §2 |

The header's field rules are checked in the order `afrac`, then `sr`, `n_ch`,
`lat_len` and `t_out`, then `cdf_len`. The caps (§7) apply to `n_ch·lat_len` and to
`t_out`.

**Decode,** against the set of §9 (`model.wam`, the integer synthesis):

1. For each channel `k` and each of its `lat_len` elements, decode a symbol `s`
   against channel `k`'s table at `offset`; the element is `s·2^afrac + median_k`.
2. The synthesis (one-dimensional integer transposed convolutions and integer
   inverse GDN) on `[n_ch, lat_len]` gives one channel of `t ≥ t_out` samples. It
   decodes through its lowering `wam1/1` ([integer-operators](integer-operators.md)
   §13.3), with `t_out` as its scalar.
3. The first `t_out` samples `v` become `(v·32767 + 2^(afrac−1)) >> afrac`, clamped
   to `[−32768, 32767]`: the PCM step that ends the lowered graph (integer-operators
   §13.3). A sample for which `v·32767 + 2^(afrac−1)` leaves `i64` is refused
   (`field_range`).

The output is `t_out` samples of `i16` PCM, one channel, at `sr` hertz: the buffer of
the capability's sample-equivalence digest (signed 16-bit little-endian samples).

## 6. `WIS1` — the integer learned Gaussian-splat attribute codec

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIS1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `afrac` | 1..=30; equal to the model's (§7) |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `c` | ≥ 1: latent channels |
| 12 | 4 | `sh` | ≥ 1: latent height |
| 16 | 4 | `sw` | ≥ 1: latent width |
| 20 | 4 | `s` | ≥ 1: the attribute planes' side |
| 24 | 4 | `attr` | ≥ 1: attribute planes |
| 28 | 4 | `n_gaussians` | 1..=`s·s` |
| 32 | `attr`·8 | ranges | per plane, `f32 lo` then `f32 hi` (IEEE 754 binary32): finite, `lo ≤ hi` |
| … | `c`·2 | table lengths | `u16` each, 3..=4097 |
| … | `c`·4 | symbol offsets | `i32` each |
| … | `c`·8 | medians | `i64` each, `|median| ≤ 2^47` |
| … | the lengths' sum ·4 | one table per channel | §2 |
| … | 4 + `L` | the latent stream | §2 |

The header's field rules are checked in the order `afrac`, the dimensions,
`n_gaussians`. The caps (§7) apply to `c·sh·sw` and to `attr·s·s`.

**Decode,** against the set of §9 (`g_s.whm`, the integer synthesis):

1. For each channel `k` and each of its `sh·sw` elements, decode a symbol `s`
   against channel `k`'s table and offset; the element is `s·2^afrac + median_k`.
2. `g_s` on `[c, sh, sw]` gives `[attr, s, s]` (`geometry_mismatch` otherwise).
   `g_s` decodes through its lowering `whm1/1`
   ([integer-operators](integer-operators.md) §13.2).
3. Each value `v` is clamped to `[0, 2^afrac]` and becomes
   `(v·255 + 2^(afrac−1)) >> afrac`, a byte: the RGB8 step that ends the lowered
   graph (integer-operators §13.0).

The output, plane after plane, is the buffer of the capability's splat-equivalence
digest. The first `n_gaussians` cells of the planes, in raster order, hold one
Gaussian each. The ranges are presentation data: a sink presents a plane's byte `u`
at about `lo + (hi − lo)·u/255`. They are outside the digest, and this extension does
not fix presentation.

## 7. Caps and the checks a decode makes

**Caps, the same on every target.** No tensor a payload sizes may exceed `2^26`
elements: a frame `h·w·3`, a `WIV1` motion-vector field `2·nb·(n_frames − 1)`, a
latent, a hyper-latent, a network's output, an attribute tensor, a clip's `t_out`. A
`WIV1` clip may not exceed `2^30` bytes, nor `2^20` frames.

**Planning the networks.** Before a decode allocates a latent, it plans every network
on the payload's shapes from the layers' geometry and parameter lengths alone. The
network's input must be within the cap (`too_large`); then, layer by layer in order:

1. the layer fits the running channel count (`model_mismatch`): a convolution's output
   channels, kernel and stride are non-zero, it has `c·out·k·k` weights (`c·out·k` for
   the audio synthesis) and `out` multipliers and biases, and its requantisation shift
   is under 128; an inverse GDN has `c·c` and `c` parameters and shifts under 128; a
   leaky ReLU's shift is under 64;
2. every size the layer forms is within `2^26` (`too_large`): a convolution's padded
   input side `n + 2·pad`, a transposed convolution's uncropped side
   `(n − 1)·stride + k + opad` and uncropped plane, and the output;
3. the layer leaves an output (`model_mismatch`): a kernel no larger than the padded
   input, a crop `2·pad` smaller than the uncropped side.

Every size is formed in 64-bit arithmetic with overflow checked, and a size past 64
bits is past the cap, so a 32-bit sink refuses exactly what a 64-bit one does, with
the same code; once a network is planned, every size it forms fits a 32-bit index.

**Memory and work.** A payload's header declares its decode's output and the work
that output takes, and a sink can read both before it decodes anything. A stream does
not bound them: a table can make one symbol cost a small fraction of a bit, so a few
kilobytes of stream can code a clip at the `2^30`-byte cap, as any codec can describe
a large output in a small file. The bound is the header's, within the caps:

- the output: a `WIV1` clip's `n_frames·h·w·3` bytes (one per sample), a `WIH1`
  image's `3·h·w` bytes at its synthesis network's size, a `WIA1` clip's `2·t_out`
  bytes (`i16` samples), a `WIS1` attribute tensor's `attr·s·s` bytes;
- the entropy decode: one step per symbol, `n_frames·h·w·3` for a keyframe-0 `WIV1`
  (`h·w·3` fewer, plus its keyframe's, for keyframe 1), `n_z·zh·zw + m·sh·sw` for a
  `WIH1`, `n_ch·lat_len` for a `WIA1`, `c·sh·sw` for a `WIS1`;
- the buffers: latents and network activations of 8 bytes per element (`i64`); while
  a network layer runs, its input, its output and, for a transposed convolution, the
  uncropped plane it accumulates into, each at most `2^26` elements (*Planning the
  networks*); a `WIV1` decode's frames (one byte per sample) and its motion vectors
  (4 bytes each);
- each network's work, its multiply-accumulates layer by layer: a convolution's
  every output times `c·k·k` (every tap of its padded input), a transposed
  convolution's every input times `out·k·k` (`out·k` for the audio synthesis), an
  inverse GDN's every element times `c`; a `WIV1` with keyframe 0 runs no network. A
  kernel may skip a zero term, so it performs at most this many.

A `WIA1`'s latent is the one its `t_out` needs and no longer: the synthesis of
`lat_len` samples gives at least `t_out` samples and that of `lat_len − 1` fewer
(`geometry_mismatch`), so a payload cannot declare a large latent to present a few
samples.

A decode refuses a stream that runs out at its first missing symbol (§2), so a short
stream stops early, but a stream that carries its symbols is decoded in full: a sink
that will not spend what a header declares refuses before decoding, by its own size
policy, applied to the declared cost (`int_payload::declared_cost` and
`int_payload::check_cost`, with the set; `neural::declared_cost` over the set a
registry resolved; the C ABI's `wai_int_payload_declared`, given the set's path;
`wai-web`'s `intPayloadCost`; the header alone through `container::declared_size`),
`cost_over_limit` (§8). The reference neural sink applies one when it is given one
(`neural::ModelRegistry::with_cost_limit`); its C ABI does not decode integer video,
PCM or splat attributes, which it has no output kind for. This extension fixes the
caps; a sink's policy may be stricter.

*Informative.* A sink may keep a parameter set lowered and loaded between decodes,
keyed by the SHA-256 of its bytes ([integer-operators](integer-operators.md) §8.6),
so that only the first decode through it pays for the lowering and the bounds pass.
What it keeps is the set's, the same for every payload, and is not part of any
decode's declared working bytes; the sink's own budget for kept sets bounds it. The
reference implementation's store (`int_set_cache::LoadedSets`) starts with a budget of
256 MiB, evicts the least recently used set first, keeps no set larger than the whole
budget, and keeps nothing with a budget of 0 (`LoadedSets::set_budget`; `wai-web`'s
`setLoadedSetBudget`). A decode that finds its set kept makes every other check of this
section, in the order below, and gives the same output and the same refusals.

**The parameter set must fit the payload** (`model_mismatch`):

- every network parses (the `WHM1` and `WAM1` forms, §9), and the networks of one set
  share one fixed-point scale `afrac` in 1..=30;
- a `WIA1` or `WIS1` payload's `afrac` is the model's;
- the entropy tables of `entropy.json` are consistent (a table, length and offset per
  scale edge, at least one edge; a table, length, offset and median per hyper-latent
  channel) and every table is a CDF of §2 over its length; a payload has no more
  hyper-latent channels than the tables;
- each layer's parameters have the lengths the running channel count needs, and its
  geometry leaves an output (*Planning the networks*);
- the audio synthesis gives one channel; the conditional codec's `g_s` gives three;
- a network that decodes through its lowering lowers, and its graph loads under
  `acc64` ([integer-operators](integer-operators.md) §13.2 and §13.3: for `WIH1` and a
  `WIV1`'s keyframe, `h_s` and `g_s`; for `WIS1`, `g_s`; for `WIA1`, the synthesis).

**The shapes the payload declares must be the ones the decode gives**
(`geometry_mismatch`): `h_s`'s output and `[2·m, sh, sw]`; a `WIV1` keyframe and
`h × w`; the audio synthesis's length and `t_out` (it may be longer, by less than one
latent sample's worth: the synthesis of `lat_len − 1` samples gives fewer than
`t_out`); `g_s`'s output and `[attr, s, s]`.

**The order of a decode's checks.** After the parse (§2), a decode checks in this
order and refuses with the first that fails:

1. the set's members, in the order of §9's list, each present (`set_incomplete`) and
   parsed (`model_mismatch`);
2. the parameter set against the payload (`model_mismatch`): one fixed-point scale,
   the payload's `afrac`, the entropy tables, the hyper-latent channels;
3. every network planned, in decode order (`h_s` then `g_s`; the audio synthesis;
   the splat `g_s`), each layer by the three rules above (`model_mismatch`,
   `too_large`): the caps come before any shape is compared;
4. what the networks give against the payload: for `WIH1`, `h_s`'s shape
   (`geometry_mismatch`), then `g_s`'s three channels (`model_mismatch`), then `h_s`
   and `g_s` lowered and loaded, in that order (`model_mismatch`); for `WIV1` with
   keyframe 1, its keyframe's shape and channels as for `WIH1`, then the keyframe's size
   against `h × w` (`geometry_mismatch`), then the keyframe's networks lowered and loaded
   (`model_mismatch`); for `WIA1`,
   one channel (`model_mismatch`), then at least `t_out` samples, then a latent no
   longer than that needs (both `geometry_mismatch`), then the synthesis lowered and
   loaded (`model_mismatch`); for `WIS1`, `g_s`'s shape
   (`geometry_mismatch`), then `g_s` lowered and loaded (`model_mismatch`);
5. each stream in decode order (`WIH1`: the hyper-latent's, then, after `h_s` runs,
   the latent's; `WIV1` with keyframe 1: the keyframe's streams, then the residual
   stream), symbol by symbol: a
   symbol that runs its stream out (`stream_exhausted`) before its value's range
   (`field_range`); after its last symbol, its canonical end (`stream_noncanonical`);
6. each network as it runs, layer by layer (`field_range`, below).

So a run-out is reported before the value it would have given, and a size past the
caps before a shape that disagrees. A sink that applies a cost limit (*Memory and
work*) compares the declared cost with it after step 4, once everything the cost reads
has been checked, and before any symbol is decoded (`cost_over_limit`, §8).

**Exactness.** A latent rebuilt as `s·2^afrac + base` must fit `i64`; before each
network layer runs, the decode checks on the values present that no product, partial
sum, requantised value or inverse-GDN term can leave the type its arithmetic is
formed in. A stream whose values fail either check is refused (`field_range`). So a
decode with overflow checks and one without compute the same integers, and neither
panics. A network that decodes through its lowering
([integer-operators](integer-operators.md) §13; for `WIH1`, a `WIV1`'s keyframe and
`WIS1`, §13.2; for `WIA1`, §13.3) makes the second check as the `check` ranges of its
graph: a value outside one is
refused (`field_range`), and the refusals are the same. So does `WIA1`'s PCM step
(§5 step 3), whose range is its graph's last `check` range.

## 8. Refusal codes

The list is closed: a sink reports exactly one of these, the first that applies in the
order of §2 and §7.

A refusal of the payload or of the parameter set against it, `bad_magic` through
`model_mismatch` in the table below (and `payload_malformed`), is final: the sink does
not decode the payload by another path, and SPEC §4's fallback is not tried, since it
would decode the same bytes. The other codes say the sink cannot decode through the
capability at all, and route as SPEC §4 routes such a capability:

- `set_incomplete` is a pin that did not resolve (SPEC §3.1): a `WAI1` continues at
  SPEC §4 step 4 (or reports it at step 6), and a `WAI2` moves on to its next entry
  (SPEC §7.1);
- `not_dispatchable` and `unsupported_capability` name a capability the sink does not
  decode: SPEC §4 does not take step 3 (or 5) with it, so a `WAI1` continues at step 4
  and a `WAI2` at its next entry, as for a capability the sink does not advertise;
- `cost_over_limit` is the sink's own policy, not a fault of the payload: a payload
  whose declared cost exceeds the limit the sink applies (§7, *Memory and work*) is
  refused before any symbol is decoded, and routes as `unsupported_capability` does,
  so a `WAI1` continues at step 4 and a `WAI2` at its next entry.

| code | when |
|---|---|
| `bad_magic` | the payload does not start with any of the four magics: a tool's JSON form lands here |
| `format_capability_mismatch` | the payload is another of the four formats than the capability's |
| `keyframe_capability_mismatch` | a `WIV1` keyframe kind that is not the capability's |
| `version` | a version this revision does not define |
| `reserved_nonzero` | a reserved field that is not zero |
| `field_range` | a field of any section outside its range (a header field, a `WIS1` range or table length, a median); or a value the stream decodes that leaves the exact integer range (§7) |
| `too_large` | a size past the caps of §7 |
| `cdf_malformed` | a table that breaks the rules of §2 |
| `stream_length` | the payload ends before a field or section it declares; or a stream length that is not a multiple of 4, or is under 8 |
| `nested_length` | a `WIV1` keyframe length `K` that is not the rest of the payload |
| `trailing` | bytes after the last section |
| `stream_exhausted` | a stream that ran out before its last symbol |
| `stream_noncanonical` | a stream that does not end canonically after its last symbol: a word left unread, or a state other than `2^31` (§2) |
| `geometry_mismatch` | a shape the payload declares that the decode does not give (§7) |
| `model_mismatch` | a parameter set that does not fit the payload (§7) |
| `set_incomplete` | a parameter set without a member the decode reads (§9); a pinned decode reports SPEC §3.1's `set_incomplete` |
| `not_dispatchable` | a capability SPEC §4 does not dispatch |
| `unsupported_capability` | a capability the sink has no integer decoder for |
| `payload_malformed` | a `wai.neural.int_synth` or `wai.neural.int_mlicpp` payload its decoder refuses |
| `cost_over_limit` | a payload whose declared cost (§7) exceeds the limit the sink applies before decoding |

## 9. Parameter sets

A capability that decodes against a parameter set reads these members, by the names a
set's listing gives them ([prior-carriage](prior-carriage.md) §7):

| capability | shape | members |
|---|---|---|
| `wai.neural.int_hyper`, `wai.video.int_hyper` | several files | `g_s.whm` and `h_s.whm` (networks in `WHM1`), `entropy.json` (the tables) |
| `wai.audio.int_codec` | one file | `model.wam` (the synthesis in `WAM1`) |
| `wai.splat.int_codec` | one file | `g_s.whm` (the synthesis in `WHM1`) |
| `wai.video.int_motion` | none | — |

A set of several files is pinned through its listing (`wai.prior.set`); a one-file set
MAY be pinned form-less, by its file's SHA-256 (SPEC §3.1). `wai.audio.int_codec` and
`wai.splat.int_codec` were registered with sets of several files, because no one-file
format had been named for them; this revision registers each as one file, the file its
decoder reads, which holds every parameter its output depends on. That widens their
registrations, as SPEC §8 allows a MINOR revision to; a set pinned through its listing
still verifies.

**`WHM1`** (`int_hyper_synth::IntModel::from_bytes`): magic, then `afrac`, `gf`,
`shift` and the layer count (`u32` each), then per layer a tag byte: 0 a transposed
convolution and 1 a convolution (input and output channels, kernel, stride, padding and,
for 0, output padding as `u32`; weights as `i16`; per-output-channel multipliers and
biases as `i64`), 2 an inverse GDN (channels as `u32`; `gamma` as `i32`, `beta` as
`i64`), 3 a ReLU, 4 a leaky ReLU (slope `i64`, its shift `u32`). **`WAM1`**
(`int_audio::AudioModel::from_bytes`): the same header, then layers of tag 0, a
one-dimensional transposed convolution, and tag 2, an inverse GDN. In both forms
nothing follows the last layer: a file with bytes after it is refused
(`model_mismatch`), so one network has one file and one digest. The layers' integer
arithmetic is [integer-operators](integer-operators.md) §13.2 (`WHM1`) and §13.3
(`WAM1`).

**`entropy.json`**, the tables of `wai.neural.int_hyper` and `wai.video.int_hyper`: a
JSON object with these members. Other members are ignored. An integer is a JSON number
with no fraction and no exponent, and `-0` is zero. A file that is not a JSON text
(RFC 8259, UTF-8, no member named twice, at most 64 levels, nothing after the value) is
refused (`model_mismatch`). The reference decode reads the text with no float parser.

| member | value |
|---|---|
| `scale_edges` | `N ≥ 1` integers, each fitting `i64`: the edges `e_0 … e_{N−1}` of the `ceil` bucket rule ([integer-operators](integer-operators.md) E3) |
| `cdfs` | `N` arrays of integers, each fitting `i32`: the latent tables, one per bucket |
| `cdf_lengths` | `N` non-negative integers: table `b`'s length `n_b` |
| `offsets` | `N` integers, each fitting `i32`: table `b`'s offset |
| `z_cdfs` | `K ≥ 1` arrays of integers, each fitting `i32`: the hyper-latent tables, one per channel |
| `z_cdf_lengths` | `K` non-negative integers |
| `z_offsets` | `K` integers, each fitting `i32` |
| `z_median_q` | `K` integers, each fitting `i64`: channel `k`'s median at `2^afrac` |

Each table, over its first `n_b` entries, is a table of §2: `3 ≤ n_b ≤ 4097`, `n_b` no
more than its array's length, `cdf[0] = 0`, `cdf[n_b − 1] = 65536`, strictly
increasing. Entries after `n_b` are not read. A set whose tables break any of this is
refused (`model_mismatch`). A payload has no more hyper-latent channels than `K` (§7).
Hyper-latent channel `k` decodes against `z_cdfs[k]`, `z_cdf_lengths[k]` and
`z_offsets[k]`, and its elements are `s·2^afrac + z_median_q[k]` (§4 step 1). A latent
element decodes against table `b`, the `ceil` bucket of its `σ` over `scale_edges`
(§4 step 3).

## 10. The tools' JSON forms

The encode and export tools under `tools/` write a JSON bitstream. It is a converter input and a
debugging aid, never a payload. An encoder MUST NOT carry it as the payload of any
capability of §1, and a sink refuses it (`bad_magic`). The converters
(`int_payload::{wiv1,wih1,wia1,wis1}_from_json`, and the independent
`int-payload-conformance/int_payload_verify.py`) make explicit what the JSON leaves
implicit:

- `WIV1`: an absent `mv_precision` is 1, and only 1 and 2 are converted; a one-frame
  clip is written with `mv_precision` 1 and `block` 1, whatever the JSON states; `mvs` has one
  list per frame, and the first is empty; a frame's list shorter than its block grid is
  padded with `(0, 0)`, the vector the JSON decoder read for a missing block; a list
  longer than the grid, or a vector outside `i16`, is refused; `rmax` is not carried.
  The keyframe of `wai.video.int_hyper` is the `WIH1` converted from its own JSON.
- `WIH1`: the fields of `cond_bitstream.json`, the streams as they are.
- `WIA1`: `sr` is the export's `sample_rate` (in `audio.json`, which the bitstream
  does not carry); `lo` is the symbol offset; every channel's table must have the first
  table's length; `afrac` is carried and must be the model's.
- `WIS1`: the ranges `lo` and `hi` must each be a binary32 value, carried exactly. A
  number spelled as an integer (no fraction, no exponent) is that integer: `2^60 + 1` is
  refused, not rounded to `2^60`, and `-0` is zero, so `+0`. Any other spelling denotes
  the binary64 value nearest it (ties to even; one that rounds to zero is a zero of its
  sign, and one past the binary64 range is no value), and that value must be a binary32
  value: the export writes each range as the shortest decimal of a binary32 value
  widened to binary64. The reference converter reads the JSON text in integer
  arithmetic, so no float parser decides a range. `afrac` is carried and must be the
  model's.

A payload and the JSON it is converted from decode to the same bytes; the conformance
vectors check that for every committed JSON bitstream.

## 11. Envelopes this revision makes non-conforming

This revision completes these strings' registrations with a payload format they did
not have (SPEC §8, *Completing a registration*). It makes non-conforming, and they
still parse:

- a `WAI1` envelope whose `capability` or `fallback` is a string of §1 and whose
  payload is not that string's format: a tool's JSON form among them;
- a `WAI2` envelope with a component of a string of §1 whose bytes are not that
  string's format.

A sink refuses such a payload when it decodes it (§8), with no fallback attempt. An
encoder MUST NOT write one; the reference implementation's emit paths (`wai wrap`, the
C packers and the browser writers) parse the payload as a sink does and refuse to. An
encrypted envelope's payload (SPEC §6) is ciphertext until a sink decrypts it, so the
emit paths do not check it; the sink checks the plaintext when it decodes it (SPEC
§6.4).
No committed envelope or conformance corpus carried such a payload when this revision
was made.

## 12. Reference implementation and conformance

- `int_payload::wire` parses and writes the four formats, with no dependency on the
  rest of the crate; `int_payload::decode_int` decodes every integer capability from
  parameter files held in memory, with no ONNX runtime and no filesystem, for native
  and `wasm32` sinks; `int_payload::resolve` verifies a pin over sets held in memory
  (`pin::verify_bytes_with`) and runs SPEC §3.1's check 5, for a pinned decode and over
  the files an unpinned decode reads (a listing the set carries must match them, and
  the digests include the listing of the files read); it is the only way to build the
  parameters a decode reads (`IntParams`). The neural sink (`neural::decode_payload`)
  delegates every integer capability to `decode_int`, and its registry reads an
  unpinned integer set once and runs check 5 over those bytes.
- `codecs::PINNED_INT_PAYLOADS` is the one table of strings and magics;
  `container::check_int_payload_framing` parses a payload with `int_payload::wire` and
  the capability's keyframe kind, which needs no integer decoder and no feature, so
  every emit path runs it: `Wai::to_emit_bytes` and `WaiMulti::to_emit_bytes` (and so
  the C packers), `wai-web`'s `pack` and `packMulti`, and `wai wrap`; `wai inspect`
  reports it.
- `int_video::decode_typed` and `decode_pframes_typed` decode `WIV1` clips through
  `wai-ops`' stage operators (`int_video::ops`); the direct decode they replaced is
  test-only (`legacy_int`), the reference the differential holds them to.
- `video_receipt::VideoReceipt::seal_decode_wiv1` seals a `wai.video.int_motion`
  receipt whose identity is BLAKE3 over the `WIV1` payload.
- `wai-web` decodes the four formats in the browser (`decodeWiv1`, `decodeWih1`,
  `decodeWia1`, `decodeWis1`).
- Conformance: the payloads beside their JSON sources in `wai-rs/tests/vectors/`,
  and the refusal fixtures and index in
  [`int-payload-conformance/`](../int-payload-conformance/), written by
  `wai_int_payload_vectors gen` and checked by `wai_int_payload_vectors verify`; the
  same corpus checked by `int_payload_verify.py`, which shares no code with wai-rs and
  decodes the `WIV1` keyframe-0 payloads in full; and the cross-architecture goldens of
  `byte-exact-conformance`, which parse and decode the `WIV1`, `WIH1`, `WIA1` and `WIS1`
  payloads on every platform of its matrix.

## 13. The `wai.neural.int_synth` payload and set

SPEC §5 keeps this capability's payload format; this section writes it down.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | `H` | ≥ 1: the image's height |
| 4 | 4 | `W` | ≥ 1: the image's width |
| 8 | 2 | `C` | ≥ 1: latent channels |
| 10 | 1 | `S_log2` | the latent's side is `S = 2^S_log2` |
| 11 | 1 | reserved | `0` |
| 12 | `C·S·S` | the latents | one signed 8-bit value each, `[C, S, S]` in CHW order |

The format has no magic and no version. The payload is exactly `12 + C·S·S` bytes, and
`C·S·S ≤ 2^26`.

The set is one file, `int_synth.json`: a JSON object whose member `layers` is a
non-empty array of layer objects. A layer has `in_ch`, `out_ch`, `k`, `stride` and
`pad`, each a non-negative integer below `2^32`, with `out_ch`, `k` and `stride` at
least 1; `out_pad`, a non-negative integer below `2^32`, 0 when absent; `weight`,
`in_ch·out_ch·k·k` integers each fitting `i32`, in the layout `[in_ch][out_ch][k][k]`;
`bias`, `out_ch` integers each fitting `i32`; `shift`, in `0..=127`, 0 when absent; and
`relu`, a boolean, false when absent. Other members are ignored.

The decode is [integer-operators](integer-operators.md) §13.1's graph on the latents.
Its output must be `[3, H, W]`, and it is presented as row-major HWC bytes. Every
refusal of the payload, the set or the decode is `payload_malformed` (§8).

The reference decoder (`int_synth::decode_bytes`) decodes through the lowering. It
checks the payload first: its header, a reserved byte of 0, `C ≥ 1`, `C·S·S ≤ 2^26` and
its length. It then lowers the set and loads the graph under `acc64`, which makes the
set's checks ([integer-operators](integer-operators.md) §13.1: at least one layer, the
layers chain, the lengths, fields below `2^32`, `out_ch`, `k` and `stride` at least 1,
a shift of at most 127, the caps and the certificate), and runs it. A shift from 64 to
127 has its exact meaning. Several of these checks are new refusals of input no
committed corpus contains ([integer-operators](integer-operators.md) §15.3, SPEC §8
*Planned breaks*).

## 14. `WIM1` and `WMB1` — `wai.neural.int_mlicpp`

SPEC §5 keeps this capability's payload format and its one-file parameter set, the
`WMB1` bundle (read from a set as `model.wmb`); this section writes both down.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIM1` | |
| 4 | 4 | `H` | ≥ 1 |
| 8 | 4 | `W` | ≥ 1 |
| 12 | 2 | `Cz` | ≥ 1: hyper-latent channels |
| 14 | 2 | `Zh` | ≥ 1 |
| 16 | 2 | `Zw` | ≥ 1 |
| 18 | 2 | `Cy` | ≥ 1: latent channels, a multiple of `K` |
| 20 | 2 | `Yh` | ≥ 1 |
| 22 | 2 | `Yw` | ≥ 1 |
| 24 | 2 | `K` | ≥ 1: channel slices |
| 26 | 2 | reserved | `0` |
| 28 | 4 | `Lz` | the `z` stream's length in bytes |
| 32 | 4 | `Ly` | the `y` stream's length in bytes |
| 36 | `Lz` | the `z` stream | §2, with no length prefix |
| 36 + `Lz` | `Ly` | the `y` stream | §2, with no length prefix |

A parser checks, in order: at least 36 bytes; the magic; the reserved field; no
dimension 0; `Cy` a multiple of `K`; `Cz·Zh·Zw`, `3·Cy·Yh·Yw` and `3·H·W` each at most
`2^26`; and the payload exactly `36 + Lz + Ly` bytes. Each stream's length is checked (a
multiple of 4, at least 8) when the decode reaches it.

The set is one file, a `WMB1` bundle: the magic `WMB1`, then four sections, each a
`u32` length followed by that many bytes, in the order `h_s`, `entropy`, `g_s` (each a
`WHM1` network, §9) and the tables. The tables are a JSON object with the members of
`entropy.json` (§9), and `afrac`, read as §9 reads `entropy.json`. Nothing follows the
fourth section. The bundle's rules: the three networks' `afrac` are equal, lie in
`1..=30`, and equal the tables' `afrac`; `1 ≤ N ≤ 65536`; and every table, over its
length, follows [integer-operators](integer-operators.md) E1's `t2` rule (`2 ≤ n_b`,
`n_b` no more than its array's length, `cdf[0] = 0`, `cdf[n_b − 1] = 65536`, strictly
increasing), in place of §2's rule.

The decode:

1. **Plan**, before any stream: `Cz` is at most the number of `z` tables; `h_s` maps
   `[Cz, Zh, Zw]` to `[2·Cy, Yh, Yw]`; `entropy` maps `[3·Cy, Yh, Yw]` to
   `[2·Cy, Yh, Yw]`; `g_s` maps `[Cy, Yh, Yw]` to `[3, H, W]`; and no tensor any of them
   forms exceeds `2^26` elements.
2. **`z`.** For each channel `k`, and each of its `Zh·Zw` elements in raster order,
   decode a symbol `s` against `z` table `k`, and the element is
   `dequantize(s, afrac, z_median_q[k])` (integer-operators E2). After the last `z`
   symbol, the stream must not have run out.
3. `hyper = h_s(z)`.
4. **`y`.** Let `y` be `[Cy, Yh, Yw]`, all 0. For each slice `j = 0 … K − 1`, and for
   each pass (the anchors, `(row + col)` even, then the rest):
   `σ‖μ = entropy(concat(y, hyper))`, where every element of `y` not yet decoded is
   still 0. Then, for each channel of the slice (`j·Cy/K` to `(j + 1)·Cy/K − 1`), and
   each of its positions in raster order that belongs to the pass: `b` is the `ceil`
   bucket of `σ` over `scale_edges` (integer-operators E3); decode `r` against table
   `b`; the element is `dequantize(r, afrac, μ)`. One `y` stream carries every slice's
   two passes, in that order. After the last `y` symbol, the stream must not have run
   out.
5. `g_s(y)`, then the RGB8 output step (integer-operators §13.0), as row-major HWC
   bytes.

The streams decode with `t2`, `lenient` and `final_only`
([integer-operators](integer-operators.md) E1). A refusal of the payload, of either
stream or of a value the decode forms is `payload_malformed`; a bundle that breaks its
rules, or that does not fit the payload (step 1, and the load checks below), is
`model_mismatch` (§8).

The reference decoder (`int_mlicpp::decode`) runs the three networks through their
lowering, `wmb1/1` ([integer-operators](integer-operators.md) §13.4: each network by
`whm1/1`, `g_s` with the RGB8 step), each graph loaded under `acc64`. Step 1 checks the
header and plans the networks from their layer geometry, then lowers and loads them,
which makes the set's checks of integer-operators §13.2 (a layer of tag 0, 1 or 2 in
each network; each layer fits the channels the layer before it gives; header shifts
under 128; a leaky shift under 64; no bias of magnitude `2^63 − 1` or more; the
certificate), then plans each graph on the payload's shapes (integer-operators §4: its
input's channel count, `Cz`, `3·Cy` or `Cy`, and the caps, a convolution's padded side
included), all before either stream. A value outside a graph's `check` range is refused
(`payload_malformed`): the ranges refuse exactly the values a layer's arithmetic could
leave `i64` on. Some of these checks are new refusals of input no committed corpus
contains, and some now come before the streams where an earlier decoder made them when
the network ran ([integer-operators](integer-operators.md) §15.3, SPEC §8 *Planned
breaks*).

## 15. Receipts of `wai.video.int_motion`

Two signed receipts describe a `WIV1` payload of `wai.video.int_motion` (keyframe 0, §3).
The **decode receipt** (§15.1) is a sink's statement of what the payload reconstructs to;
the **encode receipt** (§15.3) is an encoder's statement of the payload it wrote, the
source it wrote it from, its parameters and its error. Each binds a measured energy
figure and its acquisition class ([energy-measurement](energy-measurement.md) §2) beside
the figures anyone can recompute, and keeps the two apart. The reference is
`wai-rs`'s `video_receipt` (`VideoReceipt`, `VideoEncodeReceipt`; feature
`video_receipt`).

In this section, integers in a signing payload are big-endian and those hashed into a
frame or clip commitment little-endian, as the reference writes them; BLAKE3 is the
hash of every commitment.

### 15.1 The decode receipt

| field | what it is |
|---|---|
| `codestream_hash` | BLAKE3 over the payload's bytes |
| `h`, `w` | the payload's geometry |
| `frames` | for each decoded frame `i`, from 0: BLAKE3 over `wai:video-rframe` (ASCII), 0x01, `i` (u64) and the frame's RGB8 bytes |
| `clip_hash` | BLAKE3 over `wai:video-clip`, 0x01, `h` (u32), `w` (u32), the number of frames (u64), then each frame's `i` (u64) and hash |
| `merkle_root` | the [jwp-receipts](jwp-receipts.md) §2.2 Merkle root over the frame hashes, in order |
| `work_pixel_ops` | frames × `h` × `w` × 3, saturating at 2⁶⁴ − 1: the decoded subpixels |
| `joules_micro`, the class | the energy the sink spent reconstructing, in microjoules, and its acquisition class; `0` and no class is the unmetered marker |
| `parent_receipt_hash` | the receipt hash of the receipt before it, or none |
| `signer_pubkey`, `signer_id`, `sig` | the signer's Ed25519 key, its id, and the signature |

The **signing payload** is `wai:video-receipt` and 0x01, then `codestream_hash`,
`clip_hash`, `h` (u32), `w` (u32), `merkle_root`, the number of frames (u64),
`work_pixel_ops` (u64), `joules_micro` (u64), the class, the parent (0x00, or 0x01 and
the hash), each frame's `i` (u64) and hash, `signer_pubkey`, and `signer_id`'s length
(u32) and UTF-8 bytes. The class is one tag byte and its evidence: 0 the unmetered
marker; 1 `HwShunt` (legacy), 2 `ModelBased`, 3 `Estimator`, with none; 4
`OnChipCounter`, then `relative_ppm` (u32), `window_us` (u64) and the filtering state
(u8: 0 `Unknown`, 1 `Off`, 2 `On`); 5 `CalibratedInstrument`, then `relative_ppm` (u32),
`window_us` (u64), and the certificate reference's length (u32) and UTF-8 bytes. `sig`
is Ed25519 over the payload, verified strictly ([SPEC](../SPEC.md) §7 *Signatures*). The
**receipt hash** is BLAKE3 over `wai:video-receipt-id`, 0x01, the payload and `sig`.

A receipt is **consistent** when `merkle_root`, `clip_hash` and `work_pixel_ops` are
what its frames give, its frames are numbered 0 upwards, its figure and class cohere (0
with no class, or a figure above 0 with a class whose evidence energy-measurement §2.3
requires and whose declared `relative_ppm` × `joules_micro` reaches the resolution floor
of 288676), and its signature verifies. It is **re-derived** from a payload when the
payload's BLAKE3 is its `codestream_hash`, the payload decodes (§3), and the decode has
its geometry and, frame by frame, its frame hashes; and then it is consistent. The figure
is attested by the signature and by nothing else: another sink measures another figure
for the same decode.

### 15.2 The JSON form

A decode receipt's JSON form is one object of these members, in this order:
`profile` (`"wai.video.int_motion/receipt"`), `codestream_hash`, `clip_hash`, `h`, `w`,
`frames` (each frame's hash), `merkle_root`, `work_pixel_ops`, `joules_micro`,
`uj_per_kilopixel_op`, `energy_provenance`, the class's evidence (for `OnChipCounter`,
`energy_uncertainty` `{"relative_ppm", "window_us"}` and `energy_counter_filtering`; for
`CalibratedInstrument`, `energy_uncertainty` and `energy_calibration_ref`; for the
others, none), `parent_receipt_hash`, `signer_pubkey`, `signer_id`, `sig` and
`receipt_hash`. Hashes, keys and signatures are lowercase hexadecimal; `h`, `w` and the
figures are integers in plain decimal; `energy_provenance` is the class's label, or
`null` when unmetered, and `parent_receipt_hash` `null` when there is none.
`uj_per_kilopixel_op` is derived: `null` when the receipt is unmetered or its work is 0,
and otherwise `joules_micro` × 1000 / `work_pixel_ops`, computed in IEEE 754 binary64
(the product, then the quotient) and written with six digits after the point, correctly
rounded.

A reader reads the text strictly (RFC 8259, no member named twice), requires exactly the
members the receipt's class writes, each of its form, reads the class and its evidence
as energy-measurement §2.7 reads them, and refuses a `uj_per_kilopixel_op` that is not,
spelled as above, the receipt's own, and a `receipt_hash` that is not its receipt hash.
The members' order and the whitespace between them are not held. A receipt read so is
then consistent or not, and re-derived from its payload or not, as §15.1 says.

### 15.3 The encode receipt

An encode receipt (`wai.video.int_motion/encode-receipt`) states the payload an encoder
wrote and the source it wrote it from:

| field | what it is |
|---|---|
| `codestream_hash` | BLAKE3 over the payload's bytes |
| `h`, `w` | the source's geometry |
| `source_frames` | for each source frame `i`, from 0: BLAKE3 over `wai:video-sframe` (ASCII), 0x01, `i` (u64) and the frame's RGB8 bytes |
| `source_hash` | BLAKE3 over `wai:video-source`, 0x01, `h` (u32), `w` (u32), the number of frames (u64), then each frame's `i` (u64) and hash |
| `block`, `search`, `mv_precision`, `q` | the encoder's parameters: the payload's `block`, `mv_precision` and `q` (§3), and the motion search's radius, which the payload does not carry |
| `sse` | Σ (source − decode)² over every sample of every frame (u64): the error the payload decodes with |
| `work_pixel_ops` | frames × `h` × `w` × 3, saturating at 2⁶⁴ − 1: the encoded subpixels |
| `joules_micro`, the class | the energy the encoder spent, in microjoules, and its acquisition class; `0` and no class is the unmetered marker |
| `parent_receipt_hash`, `signer_pubkey`, `signer_id`, `sig` | as §15.1's |

Its **signing payload** is `wai:video-encode-receipt` and 0x01, then `codestream_hash`,
`source_hash`, `h` (u32), `w` (u32), the number of frames (u64), `block` (u32), `search`
(u32), `mv_precision` (u8), `q` (u32), `sse` (u64), `work_pixel_ops` (u64),
`joules_micro` (u64), the class (§15.1's encoding), the parent, each source frame's `i`
(u64) and hash, `signer_pubkey`, and `signer_id`'s length (u32) and bytes. Its **receipt
hash** is BLAKE3 over `wai:video-encode-receipt-id`, 0x01, the payload and `sig`. It is
a profile defined after the hardware class was refined (energy-measurement §2.6, *New
profiles*): its class is never the legacy `HwShunt`.

An encode receipt is **consistent** as §15.1 says of a decode receipt, with `source_hash`
for `clip_hash` and no Merkle root, and the new-profile rule for its class. It is
**re-derived** from a payload and a source when the payload's BLAKE3 is its
`codestream_hash`; the payload reads as `WIV1` keyframe 0 (§3) of its `h`, `w` and
number of frames, with its `block`, `mv_precision` and `q`; each source frame hashes to
its own; the payload decodes; and the decode against the source gives its `sse`; and
then it is consistent. Re-deriving it proves the payload decodes to frames within `sse`
of the stated source. It does not prove which encoder wrote it, or that `search` was the
radius used.

Its JSON form is one object of `profile` (`"wai.video.int_motion/encode-receipt"`),
`codestream_hash`, `source_hash`, `h`, `w`, `source_frames`, `block`, `search`,
`mv_precision`, `q`, `sse`, `work_pixel_ops`, `joules_micro`, `energy_provenance`, the
class's evidence (as §15.2), `parent_receipt_hash`, `signer_pubkey`, `signer_id`, `sig`
and `receipt_hash`, read as §15.2 reads a decode receipt's.