open-standards/wai/extensions/integer-payloads.md

# WAI Extension: Integer Payloads

> Status: Draft. Pins the payload formats of the WAI-native integer codecs that
> [SPEC.md §5](../SPEC.md) registers without one (*Payloads of the WAI-native integer
> codecs*): `WIV1` for `wai.video.int_motion` and `wai.video.int_hyper`, `WIH1` for
> `wai.neural.int_hyper`, `WIA1` for `wai.audio.int_codec` and `WIS1` for
> `wai.splat.int_codec`. Defines each layout, its decode, the checks a sink makes and
> the closed list of refusal codes; names each capability's parameter-set members;
> and makes the JSON the encode and export tools write a converter input, never a
> payload. Writes down the payload formats and parameter sets that SPEC §5 keeps for
> `wai.neural.int_synth` (§13) and `wai.neural.int_mlicpp` (`WIM1` and `WMB1`, §14).
> Reference impl: `wai-rs` — `int_payload::{wire, decode_int, resolve}`,
> `int_payload::{wiv1_from_json, wih1_from_json, wia1_from_json, wis1_from_json}`,
> `codecs::PINNED_INT_PAYLOADS`, `container::check_int_payload_framing`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope

| capability | payload | parameter set (§9) |
|---|---|---|
| `wai.video.int_motion` | `WIV1` with keyframe 0 (§3) | none |
| `wai.video.int_hyper` | `WIV1` with keyframe 1, a nested `WIH1` (§3) | several files: `entropy.json`, `g_s.whm`, `h_s.whm` |
| `wai.neural.int_hyper` | `WIH1` (§4) | several files: `entropy.json`, `g_s.whm`, `h_s.whm` |
| `wai.audio.int_codec` | `WIA1` (§5) | one file: `model.wam` |
| `wai.splat.int_codec` | `WIS1` (§6) | one file: `g_s.whm` |

**Why this pin is permitted.** Until this revision, these capabilities' SPEC §5 rows
named the decoder that read their payload, and no byte layout. The rows of
`wai.video.int_motion`, `wai.audio.int_codec` and `wai.splat.int_codec` also named the
encode or export tool that wrote a bitstream, a JSON document; those of
`wai.neural.int_hyper` and `wai.video.int_hyper` named their decoders only. SPEC §5
called every row's payload the bytes its codec normally emits, which for these rows
was the tools' JSON. So this revision changes what these strings carry. SPEC §8
permits it under *Completing a registration*: a revision may complete the
registration of a WAI-native string whose row named a decoder and fixed no byte layout
by pinning one, and it lists the envelopes that become non-conforming (§11; SPEC §5).
SPEC §8's rule that a registered string's payload format is fixed forever applies to
these strings from this revision on. The encode and export tools still write JSON
that carries the same fields; it is a converter input (§10).

`wai.neural.int_synth` and `wai.neural.int_mlicpp` keep the payload formats SPEC §5
gives them, and §13 and §14 write those formats down. A sink decodes them and the five
strings above (`wai.video.int_motion`, `wai.video.int_hyper`, `wai.neural.int_hyper`,
`wai.audio.int_codec`, `wai.splat.int_codec`) through one path
(`int_payload::decode_int`), and §8 names the codes it reports for those two.

## 2. Common rules

**Encoding.** Every multi-byte field is little-endian. Offsets are in bytes.

**Framing.** A payload starts with its four-byte magic and a version byte. This
revision defines version 1 for every format. Every reserved field is zero. Nothing
follows the last section.

**rANS streams.** A stream is a `u32` length `L` followed by `L` bytes. `L` is a
multiple of 4 and at least 8. It is decoded as follows (precision 16, with a bypass
escape of 4-bit groups):

- The state `x` starts as `(w1 << 32) | w0`, from the stream's first two `u32`
  words. *Renormalising*: if `x < 2^31`, then `x = (x << 32) | w`, where `w` is the
  next unread word; if no word is left, the stream has **run out** and `x` is left
  as it is.
- `bits(n)`: `v = x mod 2^n`, `x = x >> n`, renormalise, return `v`.
- Decoding a symbol against a table `cdf` of length `n` with offset `o`: let
  `cum = x mod 2^16` and `s` be the largest index with `cdf[s] ≤ cum`
  (`0 ≤ s ≤ n − 2`); then `x = (cdf[s+1] − cdf[s]) · (x >> 16) + cum − cdf[s]`, and
  renormalise. If `s < n − 2`, the value is `s`. If `s = n − 2` (the escape): let
  `v = bits(4)` and `count = v`; then, at most 8 times, while `v = 15` and the stream
  has not run out, `v = bits(4)` and `count = count + v`; then `count = min(count, 8)`
  and `raw = Σ_{j < count} bits(4) << 4j`. The value is `−(raw >> 1) − 1` if `raw`
  is odd and `(raw >> 1) + (n − 2)` if it is even, saturated to `i32`. The symbol
  is `value + o`, saturated to `i32`.
- A decode stops at the first symbol during whose decode its stream runs out, and
  refuses the payload (`stream_exhausted`): the value that symbol would give is not
  used, and nothing after it is decoded. A stream that never runs out decodes every
  symbol, so a conforming payload is unaffected.
- After the last symbol a decode reads from a stream, the stream ends **canonically**:
  every word of it has been read, and the state is `2^31`. Otherwise the payload is
  refused (`stream_noncanonical`). An encoder starts its state at `2^31` and writes
  the words it flushes and no others, so its streams end this way. A word after the
  last symbol's, or a state the encoder did not start from, codes the same symbols in
  other bytes; the rule closes both, so neither can give one decode several content
  hashes. A stream that codes no symbol (the residual stream of a one-frame `WIV1`
  with keyframe 1) is the state `2^31` alone: the words `2^31` and `0`.

**One clip, several payloads.** These rules fix how a payload is read, not the
only payload of a decoded output: tables that differ in entries the stream never
selects (a symbol the clip does not use, say), or a different escape layout, code
the same output in other bytes. A payload's hash therefore identifies the payload,
not what it decodes to. Where one output must have one identity, it is identified by
the digest of its decoded output: for a clip, its frame-state-equivalence digests
(SPEC §7, a video receipt's `clip_hash`); for the other formats, their capabilities'
equivalence digests.

**CDF tables.** A table is `n` entries of `u32` with `3 ≤ n ≤ 4097`, `cdf[0] = 0`,
`cdf[n − 1] = 65536` and every entry greater than the one before. These rules are
what keep the symbol search and the state update above in range.

**Order of the checks.** A parser checks, in this order, and refuses with the first
that fails:

1. the magic: the capability's (§1), or else `format_capability_mismatch` if it is
   another of the four magics, or `bad_magic`;
2. the version byte: present (`stream_length`), and 1 (`version`);
3. the fixed header: present (`stream_length`);
4. the reserved fields (`reserved_nonzero`);
5. the header's fields, in the order the format's section lists their rules
   (`field_range`), then, for `WIV1`, the keyframe kind (`keyframe_capability_mismatch`);
6. the caps of §7 on the sizes the header declares (`too_large`), in the order the
   format's section lists them;
7. each later section in order: present (`stream_length`), then its rules
   (`field_range`, `cdf_malformed`). A section of one table per channel (`WIA1`,
   `WIS1`) is read table by table: each table present, then checked, before the next;
8. nothing after the last section (`trailing`, or `nested_length` for `WIV1`'s
   nested keyframe).

A decode then makes the checks of §7, and refuses a stream that runs out or does not
end canonically (§2). Two conforming sinks refuse the same bytes with the same code.

## 3. `WIV1` — integer inter-frame video

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIV1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `mv_precision` | `1` (whole-pel) or `2` (half-pel); `1` when `n_frames` is 1 |
| 6 | 1 | `keyframe` | `0` or `1` |
| 7 | 1 | reserved | `0` |
| 8 | 4 | `n_frames` | ≥ 1 |
| 12 | 4 | `h` | ≥ 1 |
| 16 | 4 | `w` | ≥ 1 |
| 20 | 4 | `block` | ≥ 1; `1` when `n_frames` is 1 |
| 24 | 4 | `q` | 1..=65535 |
| 28 | 4 | `offset` | `i32` |
| 32 | 2 | `cdf_len` | 3..=4097 |
| 34 | 2 | reserved | `0` |
| 36 | `cdf_len`·4 | the residual table | §2 |
| … | (`n_frames` − 1)·`nb`·4 | motion vectors | `i16 dy, i16 dx` |
| … | 4 + `L` | the residual stream | §2 |
| … | 4 + `K` | keyframe 1 only: `u32 K`, then a `WIH1` payload | `K` is exactly the bytes that remain |

The header's field rules are checked in the order `mv_precision`, `keyframe`,
`n_frames`, `h` and `w`, `block`, `q`, `cdf_len`, then, for a one-frame clip,
`mv_precision` 1 and `block` 1: such a clip codes no motion vector, so neither field
would change its decode, and they are fixed so that it has one header. `keyframe` 0 is
`wai.video.int_motion`'s and 1 is `wai.video.int_hyper`'s; the other is
`keyframe_capability_mismatch`. The frame `h·w·3`, the clip `n_frames·h·w·3`, the frame
count `n_frames` and the motion-vector field `2·nb·(n_frames − 1)` are capped (§7), in
that order.

**Motion vectors.** `nb = ⌈h/block⌉·⌈w/block⌉` and `nbx = ⌈w/block⌉`. The vectors
are frames 1 to `n_frames − 1` in order, each its block grid row-major: the vector of
pixel `(y, x)` in frame `f` is entry `(f − 1)·nb + (y div block)·nbx + (x div block)`,
in units of `1/mv_precision` pel.

**Decode.** One decoder reads the residual stream for the whole clip, against the
residual table at `offset`. For each frame in order, for each pixel in raster order
`(y, x)`, for each channel `c` of R, G, B: decode the symbol `r`, and the sample is
`clamp(pred + r·q, 0, 255)`. Since `|r| < 2^31` and `q < 2^16`, every step fits `i64`.

- Frame 0 with keyframe 0: `pred = 128`.
- Frame 0 with keyframe 1: the frame is the nested `WIH1`'s decode (§4), which reads
  no symbol of the residual stream. It must be `h × w` (`geometry_mismatch`).
- Frame `f ≥ 1`: `pred` reads the previous **decoded** frame `R` with the pixel's
  vector `(dy, dx)`, where `p(Y, X)` is `R` at row `clamp(Y, 0, h − 1)`, column
  `clamp(X, 0, w − 1)`, channel `c`:
  - whole-pel: `pred = p(y + dy, x + dx)`;
  - half-pel: `iy = y + (dy >> 1)`, `fy = dy & 1`, `ix = x + (dx >> 1)`,
    `fx = dx & 1` (arithmetic shift); then `p(iy, ix)` when both are 0,
    `(p(iy, ix) + p(iy, ix + 1) + 1) >> 1` when only `fx` is 1,
    `(p(iy, ix) + p(iy + 1, ix) + 1) >> 1` when only `fy` is 1, and
    `(p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2`
    when both are.

The output is `n_frames` RGB8 frames of `h × w` (rows top to bottom, pixels left to
right, R, G, B bytes): the buffers of the capability's frame-state-equivalence digests
(SPEC §7 *Digests*).

## 4. `WIH1` — the integer conditional image codec

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIH1` | |
| 4 | 1 | version | `1` |
| 5 | 3 | reserved | `0` |
| 8 | 4 | `m` | ≥ 1: latent channels |
| 12 | 4 | `sh` | ≥ 1: latent height |
| 16 | 4 | `sw` | ≥ 1: latent width |
| 20 | 4 | `n_z` | ≥ 1: hyper-latent channels |
| 24 | 4 | `zh` | ≥ 1: hyper-latent height |
| 28 | 4 | `zw` | ≥ 1: hyper-latent width |
| 32 | 4 + `Lz` | the hyper-latent stream | §2 |
| … | 4 + `Ly` | the latent stream | §2 |

The caps (§7) apply to `n_z·zh·zw` and to `2·m·sh·sw`.

**Decode,** against the set of §9 (`g_s.whm` and `h_s.whm`, two integer networks at
one fixed-point scale `2^afrac`; `entropy.json`, the tables):

1. For each hyper-latent channel `k` and each of its `zh·zw` elements, decode a
   symbol `s` from the hyper-latent stream against channel `k`'s table and offset;
   the element is `s·2^afrac + median_k`.
2. `h_s` on the hyper-latent `[n_z, zh, zw]` gives `[2·m, sh, sw]`: its first `m`
   channels are the scales `σ`, the rest the means `μ`.
3. For each latent element `p` in order (channel, then row, then column), its table
   is `b`, where, with the scale edges `e_0 … e_{N−1}` and `σ' = max(σ_p, e_0)`,
   `b = N − 1 − #{ i < N − 1 : σ' ≤ e_i }`. Decode a symbol `r` from the latent
   stream against table `b` and its offset; the element is `r·2^afrac + μ_p`.
4. `g_s` on the latent `[m, sh, sw]` gives three channels; each value `v` is clamped
   to `[0, 2^afrac]` and becomes `(v·255 + 2^(afrac−1)) >> afrac`, as RGB8.

Every value is formed in `i64` and must fit it (§7).

## 5. `WIA1` — the integer learned audio codec

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIA1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `afrac` | 1..=30; equal to the model's (§7) |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `sr` | ≥ 1: the sample rate the PCM is presented at, in hertz |
| 12 | 4 | `n_ch` | ≥ 1: latent channels |
| 16 | 4 | `lat_len` | ≥ 1: latent length |
| 20 | 4 | `t_out` | ≥ 1: samples presented |
| 24 | 4 | `offset` | `i32`: the symbol offset of every channel's table |
| 28 | 2 | `cdf_len` | 3..=4097 |
| 30 | 2 | reserved | `0` |
| 32 | `n_ch`·8 | medians | `i64` each, `|median| ≤ 2^47` |
| … | `n_ch`·`cdf_len`·4 | one table per channel | §2 |
| … | 4 + `L` | the latent stream | §2 |

The header's field rules are checked in the order `afrac`, then `sr`, `n_ch`,
`lat_len` and `t_out`, then `cdf_len`. The caps (§7) apply to `n_ch·lat_len` and to
`t_out`.

**Decode,** against the set of §9 (`model.wam`, the integer synthesis):

1. For each channel `k` and each of its `lat_len` elements, decode a symbol `s`
   against channel `k`'s table at `offset`; the element is `s·2^afrac + median_k`.
2. The synthesis (one-dimensional integer transposed convolutions and integer
   inverse GDN) on `[n_ch, lat_len]` gives one channel of `t ≥ t_out` samples.
3. The first `t_out` samples `v` become `(v·32767 + 2^(afrac−1)) >> afrac`, clamped
   to `[−32768, 32767]`.

The output is `t_out` samples of `i16` PCM, one channel, at `sr` hertz: the buffer of
the capability's sample-equivalence digest (signed 16-bit little-endian samples).

## 6. `WIS1` — the integer learned Gaussian-splat attribute codec

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIS1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `afrac` | 1..=30; equal to the model's (§7) |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `c` | ≥ 1: latent channels |
| 12 | 4 | `sh` | ≥ 1: latent height |
| 16 | 4 | `sw` | ≥ 1: latent width |
| 20 | 4 | `s` | ≥ 1: the attribute planes' side |
| 24 | 4 | `attr` | ≥ 1: attribute planes |
| 28 | 4 | `n_gaussians` | 1..=`s·s` |
| 32 | `attr`·8 | ranges | per plane, `f32 lo` then `f32 hi` (IEEE 754 binary32): finite, `lo ≤ hi` |
| … | `c`·2 | table lengths | `u16` each, 3..=4097 |
| … | `c`·4 | symbol offsets | `i32` each |
| … | `c`·8 | medians | `i64` each, `|median| ≤ 2^47` |
| … | the lengths' sum ·4 | one table per channel | §2 |
| … | 4 + `L` | the latent stream | §2 |

The header's field rules are checked in the order `afrac`, the dimensions,
`n_gaussians`. The caps (§7) apply to `c·sh·sw` and to `attr·s·s`.

**Decode,** against the set of §9 (`g_s.whm`, the integer synthesis):

1. For each channel `k` and each of its `sh·sw` elements, decode a symbol `s`
   against channel `k`'s table and offset; the element is `s·2^afrac + median_k`.
2. `g_s` on `[c, sh, sw]` gives `[attr, s, s]` (`geometry_mismatch` otherwise).
3. Each value `v` is clamped to `[0, 2^afrac]` and becomes
   `(v·255 + 2^(afrac−1)) >> afrac`, a byte.

The output, plane after plane, is the buffer of the capability's splat-equivalence
digest. The first `n_gaussians` cells of the planes, in raster order, hold one
Gaussian each. The ranges are presentation data: a sink presents a plane's byte `u`
at about `lo + (hi − lo)·u/255`. They are outside the digest, and this extension does
not fix presentation.

## 7. Caps and the checks a decode makes

**Caps, the same on every target.** No tensor a payload sizes may exceed `2^26`
elements: a frame `h·w·3`, a `WIV1` motion-vector field `2·nb·(n_frames − 1)`, a
latent, a hyper-latent, a network's output, an attribute tensor, a clip's `t_out`. A
`WIV1` clip may not exceed `2^30` bytes, nor `2^20` frames.

**Planning the networks.** Before a decode allocates a latent, it plans every network
on the payload's shapes from the layers' geometry and parameter lengths alone. The
network's input must be within the cap (`too_large`); then, layer by layer in order:

1. the layer fits the running channel count (`model_mismatch`): a convolution's output
   channels, kernel and stride are non-zero, it has `c·out·k·k` weights (`c·out·k` for
   the audio synthesis) and `out` multipliers and biases, and its requantisation shift
   is under 128; an inverse GDN has `c·c` and `c` parameters and shifts under 128; a
   leaky ReLU's shift is under 64;
2. every size the layer forms is within `2^26` (`too_large`): a convolution's padded
   input side `n + 2·pad`, a transposed convolution's uncropped side
   `(n − 1)·stride + k + opad` and uncropped plane, and the output;
3. the layer leaves an output (`model_mismatch`): a kernel no larger than the padded
   input, a crop `2·pad` smaller than the uncropped side.

Every size is formed in 64-bit arithmetic with overflow checked, and a size past 64
bits is past the cap, so a 32-bit sink refuses exactly what a 64-bit one does, with
the same code; once a network is planned, every size it forms fits a 32-bit index.

**Memory and work.** A payload's header declares its decode's output and the work
that output takes, and a sink can read both before it decodes anything. A stream does
not bound them: a table can make one symbol cost a small fraction of a bit, so a few
kilobytes of stream can code a clip at the `2^30`-byte cap, as any codec can describe
a large output in a small file. The bound is the header's, within the caps:

- the output: a `WIV1` clip's `n_frames·h·w·3` bytes (one per sample), a `WIH1`
  image's `3·h·w` bytes at its synthesis network's size, a `WIA1` clip's `2·t_out`
  bytes (`i16` samples), a `WIS1` attribute tensor's `attr·s·s` bytes;
- the entropy decode: one step per symbol, `n_frames·h·w·3` for a keyframe-0 `WIV1`
  (`h·w·3` fewer, plus its keyframe's, for keyframe 1), `n_z·zh·zw + m·sh·sw` for a
  `WIH1`, `n_ch·lat_len` for a `WIA1`, `c·sh·sw` for a `WIS1`;
- the buffers: latents and network activations of 8 bytes per element (`i64`); while
  a network layer runs, its input, its output and, for a transposed convolution, the
  uncropped plane it accumulates into, each at most `2^26` elements (*Planning the
  networks*); a `WIV1` decode's frames (one byte per sample) and its motion vectors
  (4 bytes each);
- each network's work, its multiply-accumulates layer by layer: a convolution's
  every output times `c·k·k` (every tap of its padded input), a transposed
  convolution's every input times `out·k·k` (`out·k` for the audio synthesis), an
  inverse GDN's every element times `c`; a `WIV1` with keyframe 0 runs no network. A
  kernel may skip a zero term, so it performs at most this many.

A `WIA1`'s latent is the one its `t_out` needs and no longer: the synthesis of
`lat_len` samples gives at least `t_out` samples and that of `lat_len − 1` fewer
(`geometry_mismatch`), so a payload cannot declare a large latent to present a few
samples.

A decode refuses a stream that runs out at its first missing symbol (§2), so a short
stream stops early, but a stream that carries its symbols is decoded in full: a sink
that will not spend what a header declares refuses before decoding, by its own size
policy, applied to the declared cost (`int_payload::declared_cost` and
`int_payload::check_cost`, with the set; `neural::declared_cost` over the set a
registry resolved; the C ABI's `wai_int_payload_declared`, given the set's path;
`wai-web`'s `intPayloadCost`; the header alone through `container::declared_size`),
`cost_over_limit` (§8). The reference neural sink applies one when it is given one
(`neural::ModelRegistry::with_cost_limit`); its C ABI does not decode integer video,
PCM or splat attributes, which it has no output kind for. This extension fixes the
caps; a sink's policy may be stricter.

**The parameter set must fit the payload** (`model_mismatch`):

- every network parses (the `WHM1` and `WAM1` forms, §9), and the networks of one set
  share one fixed-point scale `afrac` in 1..=30;
- a `WIA1` or `WIS1` payload's `afrac` is the model's;
- the entropy tables of `entropy.json` are consistent (a table, length and offset per
  scale edge, at least one edge; a table, length, offset and median per hyper-latent
  channel) and every table is a CDF of §2 over its length; a payload has no more
  hyper-latent channels than the tables;
- each layer's parameters have the lengths the running channel count needs, and its
  geometry leaves an output (*Planning the networks*);
- the audio synthesis gives one channel; the conditional codec's `g_s` gives three.

**The shapes the payload declares must be the ones the decode gives**
(`geometry_mismatch`): `h_s`'s output and `[2·m, sh, sw]`; a `WIV1` keyframe and
`h × w`; the audio synthesis's length and `t_out` (it may be longer, by less than one
latent sample's worth: the synthesis of `lat_len − 1` samples gives fewer than
`t_out`); `g_s`'s output and `[attr, s, s]`.

**The order of a decode's checks.** After the parse (§2), a decode checks in this
order and refuses with the first that fails:

1. the set's members, in the order of §9's list, each present (`set_incomplete`) and
   parsed (`model_mismatch`);
2. the parameter set against the payload (`model_mismatch`): one fixed-point scale,
   the payload's `afrac`, the entropy tables, the hyper-latent channels;
3. every network planned, in decode order (`h_s` then `g_s`; the audio synthesis;
   the splat `g_s`), each layer by the three rules above (`model_mismatch`,
   `too_large`): the caps come before any shape is compared;
4. what the networks give against the payload: for `WIH1`, `h_s`'s shape
   (`geometry_mismatch`), then `g_s`'s three channels (`model_mismatch`); for `WIV1`
   with keyframe 1, its keyframe's as for `WIH1`, then the keyframe's size against
   `h × w` (`geometry_mismatch`); for `WIA1`,
   one channel (`model_mismatch`), then at least `t_out` samples, then a latent no
   longer than that needs (both `geometry_mismatch`); for `WIS1`, `g_s`'s shape
   (`geometry_mismatch`);
5. each stream in decode order (`WIH1`: the hyper-latent's, then, after `h_s` runs,
   the latent's; `WIV1` with keyframe 1: the keyframe's streams, then the residual
   stream), symbol by symbol: a
   symbol that runs its stream out (`stream_exhausted`) before its value's range
   (`field_range`); after its last symbol, its canonical end (`stream_noncanonical`);
6. each network as it runs, layer by layer (`field_range`, below).

So a run-out is reported before the value it would have given, and a size past the
caps before a shape that disagrees. A sink that applies a cost limit (*Memory and
work*) compares the declared cost with it after step 4, once everything the cost reads
has been checked, and before any symbol is decoded (`cost_over_limit`, §8).

**Exactness.** A latent rebuilt as `s·2^afrac + base` must fit `i64`; before each
network layer runs, the decode checks on the values present that no product, partial
sum, requantised value or inverse-GDN term can leave the type its arithmetic is
formed in. A stream whose values fail either check is refused (`field_range`). So a
decode with overflow checks and one without compute the same integers, and neither
panics.

## 8. Refusal codes

The list is closed: a sink reports exactly one of these, the first that applies in the
order of §2 and §7.

A refusal of the payload or of the parameter set against it, `bad_magic` through
`model_mismatch` in the table below (and `payload_malformed`), is final: the sink does
not decode the payload by another path, and SPEC §4's fallback is not tried, since it
would decode the same bytes. The other codes say the sink cannot decode through the
capability at all, and route as SPEC §4 routes such a capability:

- `set_incomplete` is a pin that did not resolve (SPEC §3.1): a `WAI1` continues at
  SPEC §4 step 4 (or reports it at step 6), and a `WAI2` moves on to its next entry
  (SPEC §7.1);
- `not_dispatchable` and `unsupported_capability` name a capability the sink does not
  decode: SPEC §4 does not take step 3 (or 5) with it, so a `WAI1` continues at step 4
  and a `WAI2` at its next entry, as for a capability the sink does not advertise;
- `cost_over_limit` is the sink's own policy, not a fault of the payload: a payload
  whose declared cost exceeds the limit the sink applies (§7, *Memory and work*) is
  refused before any symbol is decoded, and routes as `unsupported_capability` does,
  so a `WAI1` continues at step 4 and a `WAI2` at its next entry.

| code | when |
|---|---|
| `bad_magic` | the payload does not start with any of the four magics: a tool's JSON form lands here |
| `format_capability_mismatch` | the payload is another of the four formats than the capability's |
| `keyframe_capability_mismatch` | a `WIV1` keyframe kind that is not the capability's |
| `version` | a version this revision does not define |
| `reserved_nonzero` | a reserved field that is not zero |
| `field_range` | a field of any section outside its range (a header field, a `WIS1` range or table length, a median); or a value the stream decodes that leaves the exact integer range (§7) |
| `too_large` | a size past the caps of §7 |
| `cdf_malformed` | a table that breaks the rules of §2 |
| `stream_length` | the payload ends before a field or section it declares; or a stream length that is not a multiple of 4, or is under 8 |
| `nested_length` | a `WIV1` keyframe length `K` that is not the rest of the payload |
| `trailing` | bytes after the last section |
| `stream_exhausted` | a stream that ran out before its last symbol |
| `stream_noncanonical` | a stream that does not end canonically after its last symbol: a word left unread, or a state other than `2^31` (§2) |
| `geometry_mismatch` | a shape the payload declares that the decode does not give (§7) |
| `model_mismatch` | a parameter set that does not fit the payload (§7) |
| `set_incomplete` | a parameter set without a member the decode reads (§9); a pinned decode reports SPEC §3.1's `set_incomplete` |
| `not_dispatchable` | a capability SPEC §4 does not dispatch |
| `unsupported_capability` | a capability the sink has no integer decoder for |
| `payload_malformed` | a `wai.neural.int_synth` or `wai.neural.int_mlicpp` payload its decoder refuses |
| `cost_over_limit` | a payload whose declared cost (§7) exceeds the limit the sink applies before decoding |

## 9. Parameter sets

A capability that decodes against a parameter set reads these members, by the names a
set's listing gives them ([prior-carriage](prior-carriage.md) §7):

| capability | shape | members |
|---|---|---|
| `wai.neural.int_hyper`, `wai.video.int_hyper` | several files | `g_s.whm` and `h_s.whm` (networks in `WHM1`), `entropy.json` (the tables) |
| `wai.audio.int_codec` | one file | `model.wam` (the synthesis in `WAM1`) |
| `wai.splat.int_codec` | one file | `g_s.whm` (the synthesis in `WHM1`) |
| `wai.video.int_motion` | none | — |

A set of several files is pinned through its listing (`wai.prior.set`); a one-file set
MAY be pinned form-less, by its file's SHA-256 (SPEC §3.1). `wai.audio.int_codec` and
`wai.splat.int_codec` were registered with sets of several files, because no one-file
format had been named for them; this revision registers each as one file, the file its
decoder reads, which holds every parameter its output depends on. That widens their
registrations, as SPEC §8 allows a MINOR revision to; a set pinned through its listing
still verifies.

**`WHM1`** (`int_hyper_synth::IntModel::from_bytes`): magic, then `afrac`, `gf`,
`shift` and the layer count (`u32` each), then per layer a tag byte: 0 a transposed
convolution and 1 a convolution (input and output channels, kernel, stride, padding and,
for 0, output padding as `u32`; weights as `i16`; per-output-channel multipliers and
biases as `i64`), 2 an inverse GDN (channels as `u32`; `gamma` as `i32`, `beta` as
`i64`), 3 a ReLU, 4 a leaky ReLU (slope `i64`, its shift `u32`). **`WAM1`**
(`int_audio::AudioModel::from_bytes`): the same header, then layers of tag 0, a
one-dimensional transposed convolution, and tag 2, an inverse GDN. In both forms
nothing follows the last layer: a file with bytes after it is refused
(`model_mismatch`), so one network has one file and one digest. The layers' integer
arithmetic is [integer-operators](integer-operators.md) §13.2 (`WHM1`) and §13.3
(`WAM1`).

**`entropy.json`**, the tables of `wai.neural.int_hyper` and `wai.video.int_hyper`: a
JSON object with these members. Other members are ignored. An integer is a JSON number
with no fraction and no exponent.

| member | value |
|---|---|
| `scale_edges` | `N ≥ 1` integers, each fitting `i64`: the edges `e_0 … e_{N−1}` of the `ceil` bucket rule ([integer-operators](integer-operators.md) E3) |
| `cdfs` | `N` arrays of integers, each fitting `i32`: the latent tables, one per bucket |
| `cdf_lengths` | `N` non-negative integers: table `b`'s length `n_b` |
| `offsets` | `N` integers, each fitting `i32`: table `b`'s offset |
| `z_cdfs` | `K ≥ 1` arrays of integers, each fitting `i32`: the hyper-latent tables, one per channel |
| `z_cdf_lengths` | `K` non-negative integers |
| `z_offsets` | `K` integers, each fitting `i32` |
| `z_median_q` | `K` integers, each fitting `i64`: channel `k`'s median at `2^afrac` |

Each table, over its first `n_b` entries, is a table of §2: `3 ≤ n_b ≤ 4097`, `n_b` no
more than its array's length, `cdf[0] = 0`, `cdf[n_b − 1] = 65536`, strictly
increasing. Entries after `n_b` are not read. A set whose tables break any of this is
refused (`model_mismatch`). A payload has no more hyper-latent channels than `K` (§7).
Hyper-latent channel `k` decodes against `z_cdfs[k]`, `z_cdf_lengths[k]` and
`z_offsets[k]`, and its elements are `s·2^afrac + z_median_q[k]` (§4 step 1). A latent
element decodes against table `b`, the `ceil` bucket of its `σ` over `scale_edges`
(§4 step 3).

## 10. The tools' JSON forms

The encode and export tools under `tools/` write a JSON bitstream. It is a converter input and a
debugging aid, never a payload. An encoder MUST NOT carry it as the payload of any
capability of §1, and a sink refuses it (`bad_magic`). The converters
(`int_payload::{wiv1,wih1,wia1,wis1}_from_json`, and the independent
`int-payload-conformance/int_payload_verify.py`) make explicit what the JSON leaves
implicit:

- `WIV1`: an absent `mv_precision` is 1, and only 1 and 2 are converted; a one-frame
  clip is written with `mv_precision` 1 and `block` 1, whatever the JSON states; `mvs` has one
  list per frame, and the first is empty; a frame's list shorter than its block grid is
  padded with `(0, 0)`, the vector the JSON decoder read for a missing block; a list
  longer than the grid, or a vector outside `i16`, is refused; `rmax` is not carried.
  The keyframe of `wai.video.int_hyper` is the `WIH1` converted from its own JSON.
- `WIH1`: the fields of `cond_bitstream.json`, the streams as they are.
- `WIA1`: `sr` is the export's `sample_rate` (in `audio.json`, which the bitstream
  does not carry); `lo` is the symbol offset; every channel's table must have the first
  table's length; `afrac` is carried and must be the model's.
- `WIS1`: the ranges `lo` and `hi` must each be a binary32 value, carried exactly (a
  JSON integer included: `2^60 + 1` is refused, not rounded to `2^60`);
  `afrac` is carried and must be the model's.

A payload and the JSON it is converted from decode to the same bytes; the conformance
vectors check that for every committed JSON bitstream.

## 11. Envelopes this revision makes non-conforming

This revision completes these strings' registrations with a payload format they did
not have (SPEC §8, *Completing a registration*). It makes non-conforming, and they
still parse:

- a `WAI1` envelope whose `capability` or `fallback` is a string of §1 and whose
  payload is not that string's format: a tool's JSON form among them;
- a `WAI2` envelope with a component of a string of §1 whose bytes are not that
  string's format.

A sink refuses such a payload when it decodes it (§8), with no fallback attempt. An
encoder MUST NOT write one; the reference implementation's emit paths (`wai wrap`, the
C packers and the browser writers) parse the payload as a sink does and refuse to. An
encrypted envelope's payload (SPEC §6) is ciphertext until a sink decrypts it, so the
emit paths do not check it; the sink checks the plaintext when it decodes it (SPEC
§6.4).
No committed envelope or conformance corpus carried such a payload when this revision
was made.

## 12. Reference implementation and conformance

- `int_payload::wire` parses and writes the four formats, with no dependency on the
  rest of the crate; `int_payload::decode_int` decodes every integer capability from
  parameter files held in memory, with no ONNX runtime and no filesystem, for native
  and `wasm32` sinks; `int_payload::resolve` verifies a pin over sets held in memory
  (`pin::verify_bytes_with`) and runs SPEC §3.1's check 5, for a pinned decode and over
  the files an unpinned decode reads (a listing the set carries must match them, and
  the digests include the listing of the files read); it is the only way to build the
  parameters a decode reads (`IntParams`). The neural sink (`neural::decode_payload`)
  delegates every integer capability to `decode_int`, and its registry reads an
  unpinned integer set once and runs check 5 over those bytes.
- `codecs::PINNED_INT_PAYLOADS` is the one table of strings and magics;
  `container::check_int_payload_framing` parses a payload with `int_payload::wire` and
  the capability's keyframe kind, which needs no integer decoder and no feature, so
  every emit path runs it: `Wai::to_emit_bytes` and `WaiMulti::to_emit_bytes` (and so
  the C packers), `wai-web`'s `pack` and `packMulti`, and `wai wrap`; `wai inspect`
  reports it.
- `video_receipt::VideoReceipt::seal_decode_wiv1` seals a `wai.video.int_motion`
  receipt whose identity is BLAKE3 over the `WIV1` payload.
- `wai-web` decodes the four formats in the browser (`decodeWiv1`, `decodeWih1`,
  `decodeWia1`, `decodeWis1`).
- Conformance: the payloads beside their JSON sources in `wai-rs/tests/vectors/`,
  and the refusal fixtures and index in
  [`int-payload-conformance/`](../int-payload-conformance/), written by
  `wai_int_payload_vectors gen` and checked by `wai_int_payload_vectors verify`; the
  same corpus checked by `int_payload_verify.py`, which shares no code with wai-rs and
  decodes the `WIV1` keyframe-0 payloads in full; and the cross-architecture goldens of
  `byte-exact-conformance`, which parse and decode the `WIV1` and `WIH1` payloads on
  every platform of its matrix.

## 13. The `wai.neural.int_synth` payload and set

SPEC §5 keeps this capability's payload format; this section writes it down.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | `H` | ≥ 1: the image's height |
| 4 | 4 | `W` | ≥ 1: the image's width |
| 8 | 2 | `C` | ≥ 1: latent channels |
| 10 | 1 | `S_log2` | the latent's side is `S = 2^S_log2` |
| 11 | 1 | reserved | `0` |
| 12 | `C·S·S` | the latents | one signed 8-bit value each, `[C, S, S]` in CHW order |

The format has no magic and no version. The payload is exactly `12 + C·S·S` bytes, and
`C·S·S ≤ 2^26`.

The set is one file, `int_synth.json`: a JSON object whose member `layers` is an array
of layer objects. A layer has `in_ch`, `out_ch`, `k`, `stride` and `pad`, each a
non-negative integer; `out_pad`, a non-negative integer, 0 when absent; `weight`,
`in_ch·out_ch·k·k` integers each fitting `i32`, in the layout `[in_ch][out_ch][k][k]`;
`bias`, `out_ch` integers each fitting `i32`; `shift`, in `0..=127`, 0 when absent; and
`relu`, a boolean, false when absent. Other members are ignored.

The decode is [integer-operators](integer-operators.md) §13.1's graph on the latents.
Its output must be `[3, H, W]`, and it is presented as row-major HWC bytes. Every
refusal of the payload, the set or the decode is `payload_malformed` (§8).

**Checks that wait for the lowering.** The reference decoder
(`int_synth::decode_bytes`) makes every check of this section except these, which it
makes once the capability decodes through its lowering
([integer-operators](integer-operators.md) §15.3): a reserved byte of 0; `C ≥ 1`; the
caps; `k ≥ 1`, `stride ≥ 1` and a shift of at most 127; and the certificate. Until
then it decodes a payload or set that fails one of them where it can, and with a shift
from 64 to 127 its result depends on how it was built.

## 14. `WIM1` and `WMB1` — `wai.neural.int_mlicpp`

SPEC §5 keeps this capability's payload format and its one-file parameter set, the
`WMB1` bundle (read from a set as `model.wmb`); this section writes both down.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIM1` | |
| 4 | 4 | `H` | ≥ 1 |
| 8 | 4 | `W` | ≥ 1 |
| 12 | 2 | `Cz` | ≥ 1: hyper-latent channels |
| 14 | 2 | `Zh` | ≥ 1 |
| 16 | 2 | `Zw` | ≥ 1 |
| 18 | 2 | `Cy` | ≥ 1: latent channels, a multiple of `K` |
| 20 | 2 | `Yh` | ≥ 1 |
| 22 | 2 | `Yw` | ≥ 1 |
| 24 | 2 | `K` | ≥ 1: channel slices |
| 26 | 2 | reserved | `0` |
| 28 | 4 | `Lz` | the `z` stream's length in bytes |
| 32 | 4 | `Ly` | the `y` stream's length in bytes |
| 36 | `Lz` | the `z` stream | §2, with no length prefix |
| 36 + `Lz` | `Ly` | the `y` stream | §2, with no length prefix |

A parser checks, in order: at least 36 bytes; the magic; the reserved field; no
dimension 0; `Cy` a multiple of `K`; `Cz·Zh·Zw`, `3·Cy·Yh·Yw` and `3·H·W` each at most
`2^26`; and the payload exactly `36 + Lz + Ly` bytes. Each stream's length is checked (a
multiple of 4, at least 8) when the decode reaches it.

The set is one file, a `WMB1` bundle: the magic `WMB1`, then four sections, each a
`u32` length followed by that many bytes, in the order `h_s`, `entropy`, `g_s` (each a
`WHM1` network, §9) and the tables. The tables are a JSON object with the members of
`entropy.json` (§9), and `afrac`. Nothing follows the fourth section. The bundle's
rules: the three networks' `afrac` are equal, lie in `1..=30`, and equal the tables'
`afrac`; `1 ≤ N ≤ 65536`; and every table, over its length, follows
[integer-operators](integer-operators.md) E1's `t2` rule (`2 ≤ n_b`, `n_b` no more than
its array's length, `cdf[0] = 0`, `cdf[n_b − 1] = 65536`, strictly increasing), in
place of §2's rule.

The decode:

1. **Plan**, before any stream: `Cz` is at most the number of `z` tables; `h_s` maps
   `[Cz, Zh, Zw]` to `[2·Cy, Yh, Yw]`; `entropy` maps `[3·Cy, Yh, Yw]` to
   `[2·Cy, Yh, Yw]`; `g_s` maps `[Cy, Yh, Yw]` to `[3, H, W]`; and no tensor any of them
   forms exceeds `2^26` elements.
2. **`z`.** For each channel `k`, and each of its `Zh·Zw` elements in raster order,
   decode a symbol `s` against `z` table `k`, and the element is
   `dequantize(s, afrac, z_median_q[k])` (integer-operators E2). After the last `z`
   symbol, the stream must not have run out.
3. `hyper = h_s(z)`.
4. **`y`.** Let `y` be `[Cy, Yh, Yw]`, all 0. For each slice `j = 0 … K − 1`, and for
   each pass (the anchors, `(row + col)` even, then the rest):
   `σ‖μ = entropy(concat(y, hyper))`, where every element of `y` not yet decoded is
   still 0. Then, for each channel of the slice (`j·Cy/K` to `(j + 1)·Cy/K − 1`), and
   each of its positions in raster order that belongs to the pass: `b` is the `ceil`
   bucket of `σ` over `scale_edges` (integer-operators E3); decode `r` against table
   `b`; the element is `dequantize(r, afrac, μ)`. One `y` stream carries every slice's
   two passes, in that order. After the last `y` symbol, the stream must not have run
   out.
5. `g_s(y)`, then the RGB8 output step (integer-operators §13.0), as row-major HWC
   bytes.

The streams decode with `t2`, `lenient` and `final_only`
([integer-operators](integer-operators.md) E1). Every refusal of the payload, the
bundle or the decode is `payload_malformed` (§8).