open-standards/wai/extensions/frame-stream.md

# WAI Extension: Frame Streams

> Status: Draft. Carries an inter-frame video capability **one frame an object**: each
> object is one whole `WAI1` envelope holding one frame; a **group** opens with an intra
> frame, which resets the decoder state; and each later frame of the group names its
> predecessor by content hash, so that a sink refuses a lost, reordered or altered frame
> instead of decoding past it against the wrong reference. Registers four frame
> capabilities (§1), the manifest member `frame` (§2), the neural frame payload `WNF1` (§4)
> beside the integer one, `WIF1` ([integer-payloads](integer-payloads.md) §16), and the
> checks a sink makes (§5). The MoQ catalog member that declares a frame stream on a track,
> `wai_frame_stream`, is [moq-streaming-format](moq-streaming-format.md) §4.3. §9 maps a
> neural frame stream to and from a track of the neural video packaging of the MOQT
> Streaming Format (NMSF, `draft-herz-moq-nmsf-01`), byte for byte where both sides state a
> frame. No transport is defined or changed: a frame stream rides any carrier that delivers
> objects in order.
> Reference implementation: `wai-rs` — `frame_stream::{FrameStream, IntFrames, ParseOnly,
> split_wiv1, split_dcvc, join_dcvc, catalog, nmsf}`, `neural::DcvcFrames`. Conformance:
> [`int-payload-conformance/frame-stream/`](../int-payload-conformance/frame-stream/).
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope

An inter-frame capability's registered payload holds a whole clip: `WIV1` codes every frame
of a clip in one residual stream (integer-payloads §3), and the reference neural
inter-frame payload holds every frame's bitstreams behind one header (§6). A live sender
has one frame at a time, and a carrier that moves objects (MoQ, SPEC §1) moves each frame
as it is made. Each **frame capability** below carries one frame of its **base**'s
bitstream as its payload:

| frame capability | base | payload | parameter set | registry class |
|---|---|---|---|---|
| `wai.video.int_motion.frame` | `wai.video.int_motion` | `WIF1`, keyframe 0 (integer-payloads §16) | none | `StandardDefined` |
| `wai.video.int_hyper.frame` | `wai.video.int_hyper` | `WIF1`, keyframe 1 (integer-payloads §16) | the base's: `entropy.json`, `g_s.whm`, `h_s.whm` | `NeuralIntegerExact` |
| `wai.video.int_cond.frame` | `wai.video.int_cond` | `WCF1` (integer-payloads §17) | the base's: `model.wcb` | `NeuralIntegerExact` |
| `wai.neural.dcvc_rt.frame` | `wai.neural.dcvc_rt` | `WNF1` (§4) | the base's, sink-supplied | `NeuralFloat` |
| `wai.neural.dcvc_fm.frame` | `wai.neural.dcvc_fm` | `WNF1` (§4) | the base's, sink-supplied | `NeuralFloat` |

A frame capability is a capability of its own because it is a different framing of its
base's bitstream, and a registered string's payload format is fixed (SPEC §8);
`wai.audio.opus.packet` is the precedent, a capability of one Opus packet whose decoder
state carries from packet to packet. Each is a payload capability, media class `video`,
with its base's fidelity tier and licence class, and decodes against its base's parameter
set, of its base's shape: a pin on a frame object names the set its base's pin would name.

## 2. Groups and the `frame` member

A **frame stream** is a sequence of objects of one frame capability, in the order a sink
receives them. It is made of **groups**: a group is one intra frame, then zero or more
inter frames, in order. An intra frame's decode reads nothing but its own payload (and the
parameter set); an inter frame's reads the state the frames before it in its group left
(for `WIF1`, the previous decoded frame and its intra frame's table; for `WNF1` and
`WCF1`, the reference feature).

Every envelope of a frame capability carries the manifest member `frame`, an object:

| member | type | meaning |
|---|---|---|
| `index` | integer, 0 to 2^53 − 1 | the frame's position in its group: 0 for its intra frame, the frame before it plus 1 for every other |
| `prev` | string, or `null` | on an inter frame, the content hash ([jwp-receipts](jwp-receipts.md) §2.1) of the envelope of the frame before it in its group, as 64 lowercase hexadecimal digits; `null` on an intra frame |
| `number` | OPTIONAL, integer, 0 to 2^53 − 1 | the frame's sequence number in its stream, as its publisher counts it: on every frame of a group whose intra frame carries one, each the one before it plus 1; on none of a group whose intra frame carries none |
| `capture_ms` | OPTIONAL, integer, 1 to 2^53 − 1 | the wallclock at which the frame was captured, in milliseconds since 1970-01-01T00:00:00Z, as its publisher states it; absent where it states none |

```json
"frame": { "index": 3, "prev": "<64 lowercase hexadecimal digits>", "number": 1203, "capture_ms": 1793491239699 }
```

- A member of `frame` this revision does not define is ignored. `index`, `number` and
  `capture_ms` are integers written as one (SPEC §2).
- `number` and `capture_ms` are inside the envelope, so the content hash, the next frame's
  `prev` and the object's receipt cover them with the frame. A sink holds a group's
  `number`s to their sequence (§5) and reads `capture_ms` as its publisher's statement.
- `prev` names the predecessor's whole envelope, its manifest included, and so its own
  `prev`: the hash of a group's last frame commits to every frame of the group, in order,
  as a receipt chain does.
- The content hash is over the envelope's bytes as they travel, the bytes an object
  receipt names (jwp-receipts §2.1). A sink that receives an encrypted envelope (SPEC §6)
  takes the hash of what it received, before it opens the payload.
- An encoder writes `fallback` `null`: no other capability reads a frame's payload.
- Every frame of a group carries the pin its intra frame carries: both have no
  `conditioning.prior`, or both have the same value.

`frame` is an OPTIONAL manifest member under SPEC §8's rule for a MINOR revision: an
envelope that does not carry it keeps its bytes and its meaning. It is carried exactly
when the capability is a frame capability. A reader of an earlier revision does not
advertise a frame capability, so it finds such an envelope's capability unsupported and
the envelope inert (SPEC §4).

## 3. The integer frame forms

`wai.video.int_motion.frame` and `wai.video.int_hyper.frame` carry `WIF1`
(integer-payloads §16). A group's intra frame carries the group's residual table, its `q`
and `offset`, and frame 0: with keyframe 0 its residual symbols, predicted from grey; with
keyframe 1 the base's nested `WIH1`, the learned keyframe. Each inter frame carries its
motion vectors, its `mv_precision` and `block`, and its residual symbols, in a stream of its
own. The decode of each frame is `WIV1`'s frame step (integer-payloads §3), through the
same stage operators ([integer-operators](integer-operators.md) §13.5; the keyframe through
`whm1/1`, §13.2), so:

- a frame decodes to the same bytes on every conforming sink, given its chain: each frame
  capability has its base's registry class, and its frames' frame-state-equivalence digests
  (SPEC §7 *Digests*) are portable;
- a stream split from a clip (§6) decodes, frame by frame, to the clip's frames, byte for
  byte, and to the digests the clip's decode gives.

`wai.video.int_cond.frame` carries `WCF1` (integer-payloads §17): one frame of conditional
inter-frame coding, whose latent is coded given the feature the frame before it in its group
left, and which leaves its own feature to the frame after it. Its intra frame reads the zero
feature. Each frame is decoded by integer-payloads §17.4 through its pinned set, so the same
two properties hold: its frames reproduce on every conforming sink given their chain, and a
group split from a `WIC1` clip (integer-payloads §17.2, a copy of the clip's frames) decodes
to the clip's frames, byte for byte.

## 4. `WNF1` — one frame of the reference neural inter-frame payload

`wai.neural.dcvc_rt.frame` and `wai.neural.dcvc_fm.frame` carry `WNF1`: one frame's two
bitstreams, behind the clip header every frame of its group repeats.

| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WNF1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `kind` | `0` intra, `1` inter |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `h` | ≥ 1 |
| 12 | 4 | `w` | ≥ 1 |
| 16 | 4 | `fps_milli` | frames per second × 1000 |
| 20 | 2 | `cy` | ≥ 1: the latent's channels |
| 22 | 2 | `cz` | ≥ 1: the hyper-latent's channels |
| 24 | 1 | `sy_log2` | ≤ 15: the latent's side is `sy = 2^sy_log2` |
| 25 | 1 | `sz_log2` | ≤ 15: the hyper-latent's side is `sz = 2^sz_log2` |
| 26 | 2 | `q` | the quality index the entropy tables are read at |
| 28 | 2 | `cf` | ≥ 1: the reference feature's channels |
| 30 | 2 | reserved | `0` |
| 32 | 4 | `lz` | the hyper-latent bitstream's length |
| 36 | 4 | `ly` | the latent bitstream's length |
| 40 | `lz` | the hyper-latent bitstream | |
| … | `ly` | the latent bitstream | |

Nothing follows the latent bitstream. Every multi-byte field is little-endian.

**Checks**, in this order, each with its code from integer-payloads §8: the magic
(`bad_magic`); the version byte, present (`stream_length`) and 1 (`version`); the fixed
header of 40 bytes (`stream_length`); the reserved fields (`reserved_nonzero`); `kind`, then
`h`, `w`, `cy`, `cz` and `cf` at least 1, then `sy_log2` and `sz_log2` at most 15
(`field_range`); the frame `h·w·3`, the latent `cy·sy·sy`, the hyper-latent `cz·sz·sz` and
the reference feature `cf·sy·sy`, each at most `2^26` elements (`too_large`); the two
bitstreams (`stream_length`); nothing after them (`trailing`).

**Decode.** The bitstreams are the capability's own, as its reference decoder reads them
(`neural/video_dcvc.rs`): the latents are entropy-decoded from the two bitstreams against
the base's parameter set at `q`, and the base's temporal network maps the latents and the
reference feature to the frame and the next reference feature. An intra frame starts the
reference feature at zero; an inter frame reads the one the frame before it left, and
carries its intra frame's header, every field from `h` to `cf` (`geometry_mismatch`). A
decode the capability's decoder cannot complete is refused `payload_malformed`, and a pin
that does not resolve with SPEC §3.1's code. The decode runs floating point: its frames
reproduce on one build and not across hardware (`NeuralFloat`, SPEC §7), so these strings
never back `intent = replicate`. The chain of §2 still holds every frame to its
predecessor: two sinks may present different pixels, never different frames.

## 5. What a sink checks

A sink holds, for each frame stream it decodes, its **reference**: the last frame it
decoded, its content hash, its `index` and its pin, and the decoder state the frame left. It
checks each object it receives, in this order, and refuses it at the first rule that fails:

1. The envelope reads (SPEC §2): `envelope_malformed`.
2. Its capability is the stream's: `frame_capability_mismatch`.
3. It carries `frame`, an object: `frame_link_missing`.
4. `frame`'s `index` is an integer from 0 to 2^53 − 1, its `prev` is `null` or 64
   lowercase hexadecimal digits, its `number`, where present, is an integer from 0 to
   2^53 − 1, and its `capture_ms`, where present, is an integer from 1 to 2^53 − 1:
   `frame_link_malformed`.
5. The payload parses alone: integer-payloads §16 for `WIF1`, §17.1 for `WCF1`, §4 for
   `WNF1`, each with its
   code.
6. `frame` agrees with the payload's kind: an intra frame has `index` 0 and `prev` `null`,
   an inter frame an `index` of 1 or more and a `prev`: `frame_link_mismatch`.

An **intra frame** then drops the reference, is decoded (SPEC §3.1's pin checks, then the
decode's codes), and becomes the reference. An **inter frame** is checked against the
reference first:

7. The sink holds a reference: `frame_no_reference`.
8. Its `prev` is the reference's content hash: `frame_gap` when its `index` is past the one
   after the reference's (frames are missing), `frame_prev_mismatch` otherwise.
9. Its `index` is the reference's plus 1: `frame_index_mismatch`. Its `number` is the
   reference's plus 1 where the reference carries one, and absent where the reference
   carries none: `frame_number_mismatch`.
10. Its `conditioning.prior` is the reference's, compared as JSON values (both absent
    counts as the same): `frame_prior_changed`.
11. It is decoded against the reference: for `WIF1`, its `h × w` is its group's
    (`geometry_mismatch`), then the decode's codes; for `WCF1`, its header, `h` to `cf`, is
    its group's (`geometry_mismatch`), then the decode's codes (integer-payloads §17.4); for
    `WNF1`, its header is its group's (`geometry_mismatch`), then the decode's.

It then becomes the reference. **After any refusal the sink holds no reference**: every
inter frame it receives is refused (`frame_no_reference`) until an intra frame opens a group.
So a sink never decodes an inter frame against a reference that is not its predecessor's
decode: a lost frame, a frame delivered twice or out of order, and an altered `prev` are
each refused at the first frame they would corrupt, and the stream resumes at the next
group. A sink that joins a stream joins at an intra frame.

| code | when |
|---|---|
| `envelope_malformed` | the object is not a `WAI1` envelope that reads |
| `frame_capability_mismatch` | the object's capability is not the stream's |
| `frame_link_missing` | no `frame` member, or one that is not an object |
| `frame_link_malformed` | an `index` or `prev` outside its type or range |
| `frame_link_mismatch` | an intra frame with a `prev` or an `index` other than 0, or an inter frame with no `prev` or `index` 0 |
| `frame_no_reference` | an inter frame while the sink holds no reference |
| `frame_gap` | an inter frame whose predecessor is not the reference, with frames missing between |
| `frame_prev_mismatch` | an inter frame whose `prev` is not the reference's content hash |
| `frame_index_mismatch` | an inter frame that names the reference but whose `index` is not the next |
| `frame_number_mismatch` | an inter frame whose `number` is not the one after the reference's, or that carries one where the reference carries none, or none where it carries one |
| `frame_prior_changed` | an inter frame whose pin is not its group's |

A payload's refusals are integer-payloads §8's codes; a pin's are SPEC §3.1's.

## 6. Splitting a clip

**`WIV1`.** A `WIV1` clip is one group of `WIF1` frames, frame for frame. Frame 0 is the
intra frame: the clip's table, `q` and `offset`, and, with keyframe 0, frame 0's residual
symbols; with keyframe 1, the clip's nested `WIH1`. Each frame `f ≥ 1` is an inter frame:
frame `f`'s motion vectors, the clip's `mv_precision` and `block`, and frame `f`'s residual
symbols. Each frame's symbols are the ones the clip's residual stream decodes to for that
frame, coded into a stream of their own by the rANS coder integer-payloads §2 decodes (its
state starting at `2^31`, so each stream ends canonically). The group decodes, frame by
frame, to the clip's frames, byte for byte (§3). The split refuses a clip whose residual
stream does not decode: `stream_exhausted`, `stream_noncanonical`, and the parse's codes.
A stream of several groups is the split of several clips, one after another.

**The reference neural payload.** The reference decoder of `wai.neural.dcvc_rt` and
`wai.neural.dcvc_fm` reads a whole clip as: `h`, `w`, `n_frames`, `fps_milli` (`u32`),
`cy`, `cz` (`u16`), `sy_log2`, `sz_log2` (`u8`), `q`, `cf` (`u16`), 26 bytes; then
`n_frames` pairs of `u32` lengths `(lz, ly)`; then each frame's hyper-latent and latent
bitstreams, in order. Its split is one group of `WNF1` frames, each with the clip's header
and its own two bitstreams, copied; frame 0 intra, every other inter. The split reads the
clip strictly: its header as `WNF1`'s checks read it, at least one frame (`field_range`),
every bitstream inside it (`stream_length`) and nothing after the last (`trailing`). The
**join** is its inverse: one group of `WNF1` frames, the first intra and the others inter
(`field_range`), each with the first's header (`geometry_mismatch`), back to the clip. The
split and the join give each other's input back, byte for byte, and the decode of the
frames is the decode of the clip: the reference decoder's whole-clip decode runs the same
step a frame (`neural/video_dcvc.rs`, `Temporal::step`).

## 7. Carriage

A frame stream rides any carrier that delivers its objects in order; this extension
defines no transport. Each object keeps every rule of the carrier for a `WAI1` object, and
its receipt is an object receipt like any other ([jwp-receipts](jwp-receipts.md)).

- **MoQ.** One track carries one frame stream, its catalog entry declaring it
  (`wai_frame_stream`, [moq-streaming-format](moq-streaming-format.md) §4.3). Each MOQT group
  of the track is one group of the stream, and the object of Object ID *k* in it is the frame
  of `index` *k*. A subscriber joins at a group's first object (MOQT §3.5). A frame a relay
  drops is refused as a gap at the frame after it, never decoded past; the group's receipts
  still account for it (moq-streaming-format §5). A frame stream of a neural frame
  capability maps to and from a track of packaging `nvc` (§9).
- **HTTP and files.** A sequence of envelopes, in order: a playlist's objects
  ([http-delivery](http-delivery.md)), or a file of envelopes one after another.

A frame object is complete when its frame is coded, so a live sender publishes each frame
as it is made, and a sink presents it when it is decoded.

## 8. Reference implementation and conformance

- `wai-rs` (feature `video_int`): `frame_stream::FrameStream` holds a stream's reference and
  makes §5's checks in order, decoding through a `FrameCodec`: `IntFrames`, the decoder of
  the integer frame forms (`int_video::decode_unit_typed`, the clip decode's frame step, and,
  for a keyframe, `int_payload::decode_keyframe` through the capability's set as its pin
  resolves it); and `ParseOnly`, which parses each frame and checks its geometry against its
  group's intra frame without decoding. `frame_stream::split_wiv1`, `split_dcvc` and
  `join_dcvc` are §6's; `frame_stream::envelope` and `wrap_group` write frame objects;
  `frame_stream::catalog` reads `wai_frame_stream` and checks an object against it;
  `container::check_int_payload_framing` parses `WIF1`, `WCF1` and `WNF1` on every emit
  path. `int_cond_video::CondFrames` decodes `wai.video.int_cond.frame`, keeping the reference
  feature from frame to frame (integer-payloads §17.5); its corpus is
  [`int-payload-conformance/int-cond/`](../int-payload-conformance/int-cond/).
  `neural::DcvcFrames` (features `neural` and `video_int`) decodes the neural frame forms
  through the reference decoder's `Temporal::step`. A native sink that decodes in its own
  process takes the integer frame forms through `sink_decode::frames` (feature `sink_decode`),
  each frame's networks on the device's GPU or its CPU ([native-sink](native-sink.md) §13.6).
- [`int-payload-conformance/frame-stream/`](../int-payload-conformance/frame-stream/), written
  by `wai_int_payload_vectors gen` and checked by `wai_int_payload_vectors verify`: the
  committed `WIV1` clips `int_video`, `video_real` (half-pel) and `video_nk` (a learned
  keyframe, pinned) split into frame objects, each frame decoding to its clip's frame byte
  for byte; two clips as one stream of two groups; a stream with each of §5's faults planted;
  `WIF1` and `WNF1` payloads each changed in one place; a synthetic neural clip, its split,
  its join and their refusals; and catalog entries with objects checked against them. The
  neural clip's bitstreams are generated bytes, not an encode: the corpus holds its framing
  and its chain, and no decode, which needs the sink-supplied networks.
- `int_payload_verify.py`, which shares no code with wai-rs, checks the same corpus from
  this text: it splits each clip again with its own rANS coder and requires the committed
  payloads, decodes every keyframe-0 frame and requires each frame's SHA-256, makes §5's
  checks over every stream with its own BLAKE3, and checks every payload, split, join and
  catalog case. It does not decode a `WIH1` keyframe; those streams it checks to §5 and the
  frames' parse, and wai-rs checks their decode.

## 9. NMSF tracks

`draft-herz-moq-nmsf-01` (NMSF) packages the frames of a neural inter-frame codec for the
MOQT Streaming Format as packaging `nvc`: one frame an object, a group of pictures a MOQT
group that opens with a neural keyframe (NMSF §3.5), the model's weights installed on both
sides and named, not sent (NMSF §8). A frame stream of `wai.neural.dcvc_rt.frame` or
`wai.neural.dcvc_fm.frame` carries the same bitstreams, so a publisher maps one to the
other: an NMSF frame becomes one `WNF1` frame object (a **rewrap**, §9.4), and a frame object
becomes one NMSF object, or two (an **unwrap**, §9.5). Neither changes a bitstream; an
unwrap of a rewrap gives the NMSF objects back byte for byte; and a relay forwards both
tracks as it forwards any track. A section number of NMSF here is one of that revision's;
where a later revision renumbers or changes what this section cites, this section is what
changes.

### 9.1 What maps

An NMSF object is a 26-byte header and a payload (NMSF §3.2). This section reads the header
from the field definitions, which give each field's size and sum to the 26 bytes the section
states; the figure before them draws `pts_ms` one byte short. Every field is big-endian:

| off | size | field |
|---|---|---|
| 0 | 1 | `frame_type`: `0x00` Intra, `0x01` Inter, `0x02` to `0xFF` reserved (NMSF §3.3) |
| 1 | 1 | `qp`: 0 to 63; 64 to 255 reserved |
| 2 | 4 | `frame_number` |
| 6 | 8 | `pts_ms` |
| 14 | 4 | `width` |
| 18 | 4 | `height` |
| 22 | 4 | `payload_len` |
| 26 | `payload_len` | the payload |

The payload is **components** (NMSF §3.6), each `channels`, `height`, `width` and
`data_len` (`u32`, big-endian) and then `data_len` bytes of an entropy-coded tensor. In
single-track mode an object's payload is the hyperprior's component and then the latent's;
in two-track mode the hyperprior track's object holds the hyperprior's component and the
latent track's object the latent's, the two objects of a frame with the same header (NMSF
§3.4). A frame object holds the same frame as:

| NMSF | frame object | where it has no form |
|---|---|---|
| `codec` `dcvc-rt`, `dcvc-fm` (NMSF §5) | the capability `wai.neural.dcvc_rt.frame`, `wai.neural.dcvc_fm.frame`; `conditioning.kind` `dcvc_rt`, `dcvc_fm` | any other codec: `nmsf_codec_unmapped`. `dcvc-dc` is SPEC §5's candidate `wai.neural.dcvc_dc`, which pins no bitstream and has no frame form; `ssf`, `fvc`, `rlvc` and `elfvc` WAI does not register |
| a MOQT group: an Intra frame, then Inter frames | a group of the stream (§2); the object of Object ID *k* is the frame of `index` *k* (moq-streaming-format §4.3) | |
| `frame_type` Intra, Inter | `WNF1` `kind` 0 (`index` 0, `prev` `null`), 1 (`index` 1 or more, `prev` the frame before) | a reserved type: `nmsf_reserved` |
| `qp` | `WNF1` `q` | a reserved `qp`: `nmsf_reserved`; a `q` past 63 (§9.5): `nmsf_value_unmappable` |
| `frame_number` | `frame.number` (§2) | a frame with no `number`, or one past 2^32 − 1 (§9.5): `nmsf_value_unmappable` |
| `pts_ms`; 0 where it states none | `frame.capture_ms` (§2); absent where it states none | a `pts_ms` past 2^53 − 1: `nmsf_value_unmappable` |
| `width`, `height` | `WNF1` `w`, `h` | |
| the hyperprior's component | `WNF1` `cz`, `sz_log2` (the side is `2^sz_log2`) and the hyper-latent bitstream | a tensor that is not square, whose side is not a power of two at most 2^15, or of channels outside 1 to 65 535 (§4): `nmsf_shape_unmappable` |
| the latent's component | `WNF1` `cy`, `sy_log2` and the latent bitstream | as the hyperprior's |
| the catalog's `framerate` | `WNF1` `fps_milli` | a `framerate` that is not a whole number of thousandths (§9.2), or a frame of another `fps_milli` (§9.5): `nmsf_value_unmappable` |
| — | `WNF1` `cf`, the reference feature's channels, from the model binding (§9.3) | |
| `codec` and `nvc.modelVersion`, the model by name | `conditioning.prior` and `wai_priors`, its parameter set by SHA-256 (SPEC §3.1), through the model binding | |

### 9.2 The catalog member `wai_nmsf`

A WAI track whose frames map to an NMSF track keeps that track's catalog entries in the
member `wai_nmsf`: an array of one entry (single-track mode) or two (two-track mode, a
`hyperprior` and a `latent` track, NMSF §3.1), each a JSON object as NMSF §3.7 and §3.8 state
it, in the order given. The frame objects do not carry `colorspace`, `gopSize`, `priority`,
`nvcRole`, `depends` or the `nvc` object, and `wai_nmsf` keeps them, so the entries an unwrap
publishes are the entries the rewrap read, member for member. A rewrap publishes the entry:

```json
{ "name": "video/wai", "packaging": "wai", "isLive": true, "role": "video",
  "width": 64, "height": 64, "framerate": 30,
  "wai_capabilities": ["wai.neural.dcvc_rt.frame"],
  "wai_frame_stream": { "capability": "wai.neural.dcvc_rt.frame" },
  "wai_priors": [ { "capability": "wai.neural.dcvc_rt.frame",
                    "sha256": "<64 lowercase hexadecimal digits>",
                    "form": "wai.prior.set", "tier": "none" } ],
  "wai_nmsf": [ { "name": "video", "packaging": "nvc", "isLive": true, "codec": "dcvc-rt",
                  "width": 64, "height": 64, "framerate": 30, "colorspace": "ycbcr-bt709",
                  "gopSize": 4, "nvc": { "modelVersion": "cvpr2025" } } ] }
```

Its `name` is the publisher's; `isLive`, `width`, `height` and `framerate` are the first NMSF
entry's; it declares the frame stream (moq-streaming-format §4.3); `wai_priors` (§4.2) lists
the binding's pin, with its `form` and `tier` where the pin states them, and is absent where
the binding has no pin; `wai_nmsf` holds the NMSF entries.

**Reading NMSF entries.** A publisher reads them, in this order, and refuses the track at
the first rule that fails:

1. Each entry is an object with `name` (a non-empty string), `packaging` `"nvc"`, `isLive`
   (a boolean), `codec` and `colorspace` (strings), `width`, `height` and `gopSize`
   (integers of at least 1) and `framerate` (a number above 0); `priority`, where present,
   a number; `nvcRole` and `depends`, where present, strings; and `nvc`, where present, an
   object whose `modelVersion`, where present, is a string. There is one entry, with no
   `nvcRole`, or two: one of `nvcRole` `"hyperprior"` and one of `"latent"`, whose `depends`
   is the hyperprior entry's `name`, the two names different. Otherwise
   `nmsf_catalog_malformed`.
2. Two entries agree on `codec`, `isLive`, `width`, `height`, `framerate` (as numbers),
   `colorspace`, `gopSize` and `nvc.modelVersion` (both absent counts as the same):
   `nmsf_track_mismatch`.
3. The codec has a frame capability (§9.1): `nmsf_codec_unmapped`.
4. `fps_milli` is the integer *m* nearest `framerate` × 1000, from 1 to 2^32 − 1, where *m*
   ÷ 1000 is `framerate` in binary64 arithmetic: 29.97 is 29 970, and 30 000 ÷ 1 001 has
   none (`nmsf_value_unmappable`).

**Reading a WAI entry.** An unwrap reads the entry's frame stream (moq-streaming-format
§4.3, `catalog_frame_stream_malformed`; a track that declares none has no NMSF form,
`nmsf_no_form`); its `wai_nmsf` (absent or `null`: `nmsf_no_form`; not an array:
`nmsf_catalog_malformed`), whose entries read as above and whose codec's frame capability is
the declared one (`nmsf_catalog_malformed`); and the pin `wai_priors` lists for that
capability: each of its entries an object, and one for the capability with a `sha256` of 64
lowercase hexadecimal digits and a `form` that is absent, `null` or a string
(`nmsf_catalog_malformed`). It lists none, and the frames carry no pin; or one, and they
carry it; or several, and the track has no NMSF form, which names one model
(`nmsf_no_form`). A publisher that unwraps a frame stream it wrote itself states its NMSF
entries in `wai_nmsf`: the colorspace and GOP size its frames do not carry are its to state.

### 9.3 The model binding

NMSF names a model by `codec` and `nvc.modelVersion` and leaves the distribution of its
weights to the deployment (NMSF §8). A frame object names its parameter set by SHA-256 in its
pin, which a sink checks before it decodes (SPEC §3.1). A **model binding** is the
publisher's statement of the set a name stands for, and of what `WNF1` needs of the model
that an NMSF object does not carry:

```json
{ "codec": "dcvc-rt", "modelVersion": "cvpr2025", "cf": 64,
  "prior": { "id": "dcvc-rt/cvpr2025", "sha256": "<64 lowercase hexadecimal digits>",
             "determinism": "float", "form": "wai.prior.set", "tier": "none" } }
```

| member | presence | meaning |
|---|---|---|
| `codec` | REQUIRED, string | the NMSF codec identifier |
| `modelVersion` | string, `null` or absent | the `nvc.modelVersion` it binds; `null` or absent binds a track that names none |
| `cf` | REQUIRED, integer, 1 to 65 535 | the channels of the model's reference feature, `WNF1`'s `cf` (§4) |
| `prior` | a pin (SPEC §3.1), `null` or absent | the parameter set every frame object carries as its pin; `null` or absent, the frames carry none |

A binding that does not read so is `nmsf_binding_malformed`. A rewrap maps a track through
the first binding whose `codec` and `modelVersion` are the track's: none is
`nmsf_model_unbound`. The neural frame forms are `NeuralFloat`, so a binding's pin claims no
tier above `none` (SPEC §3.1). A binding is the publisher's configuration and is not carried:
in the catalog a rewrap publishes, its name is in `wai_nmsf` and its pin in `wai_priors`, so an
unwrap reads both from there.

### 9.4 Rewrap: NMSF to frame objects

A publisher maps an NMSF track's frames in the order it receives them: each frame one object,
or in two-track mode the hyperprior track's object and the latent track's object of the
frame (NMSF §4.2), with the MOQT Object ID where the carrier gives one. It holds the
**group** it is mapping: its last frame's `index`, `frame_number`, `WNF1` header and frame
object's content hash. It checks each frame in this order, and refuses it at the first rule
that fails:

1. The frame is one object in single-track mode and two in two-track mode:
   `nmsf_track_mismatch`.
2. Each object, in order, is at least 26 bytes and its `payload_len` is the length of the
   bytes after its header (`nmsf_object_malformed`), and its `frame_type` and `qp` are not
   reserved (`nmsf_reserved`).
3. The two objects of a frame have the same header, `payload_len` aside:
   `nmsf_track_mismatch`.
4. Components fill each payload: two in single-track mode, one in each object in two-track
   mode: `nmsf_object_malformed`.
5. An Intra frame drops the group, and is at Object ID 0 where it has one. An Inter frame
   follows a group, its `frame_number` the group's last plus 1 and its Object ID, where it
   has one, the group's last `index` plus 1. Otherwise `nmsf_group_order`.
6. Its `pts_ms` is at most 2^53 − 1: `nmsf_value_unmappable`.
7. Each component has a `WNF1` form (§9.1), and the `WNF1` header the frame gives passes §4's
   checks: `nmsf_shape_unmappable`.
8. An Inter frame's `WNF1` header is its group's, so its `qp`, `width`, `height` and tensor
   shapes are its Intra frame's: `nmsf_group_header_changed`.

It then writes the frame object: a `WAI1` envelope of the codec's frame capability, `media`
`video`, `intent` `perceptual`, `conditioning.kind` the codec's (§9.1) and `conditioning.prior`
the binding's pin, with no fallback (§2). Its payload is the `WNF1` frame of `kind`
`frame_type`, `h` `height`, `w` `width`, `fps_milli` the track's (§9.2), `cy`, `cz`,
`sy_log2` and `sz_log2` the components', `q` `qp`, `cf` the binding's, and the components'
data as its two bitstreams. Its `frame` is `index`, `prev`, `number` `frame_number` and
`capture_ms` `pts_ms`, absent where `pts_ms` is 0. The frame object becomes the group's
last.

After any refusal the publisher holds no group: it refuses every Inter frame
(`nmsf_group_order`) until an Intra frame opens one. A lost or damaged NMSF object therefore
yields no frame object for any frame its loss would have changed. A single-track frame and the
two objects of the same frame in two-track mode give the same frame object, which a sink
admits as any frame (§5).

### 9.5 Unwrap: frame objects to NMSF

A publisher whose WAI entry reads (§9.2) maps each frame object in order, and refuses it at
the first rule that fails:

1. It is admitted as a sink admits a frame (§5), its payload parsed and held to its group's
   header as `ParseOnly` holds it: §5's codes and §4's.
2. An intra frame's pin is the one the entry lists: the same `sha256` and `form`, or no pin
   where the entry lists none: `nmsf_model_unbound`. §5 holds each inter frame's pin to its
   group's.
3. Its `frame.number` is present and at most 2^32 − 1: `nmsf_value_unmappable`.
4. Its `WNF1` `q` is at most 63 and its `fps_milli` is the track's: `nmsf_value_unmappable`.

It then writes the NMSF object, or the two: the header `frame_type` `kind`, `qp` `q`,
`frame_number` `number`, `pts_ms` `capture_ms` (0 where absent), `width` `w`, `height` `h`;
the hyperprior's component (`cz` channels, a side of `2^sz_log2` both ways, the hyper-latent
bitstream) and the latent's (`cy`, `2^sy_log2`, the latent bitstream). In single-track mode
one object holds both components; in two-track mode the hyperprior track's object holds the
first and the latent track's object the second, the same header on both. The frame's `cf` is
not carried. After any refusal the publisher holds no reference (§5), and refuses every
inter frame until an intra frame opens a group.

### 9.6 What each side carries, and the codes

A frame object carries what an NMSF object does not:

- the link to the frame before it (§2): a sink refuses a lost, reordered or altered frame at
  the first frame it would change, and resumes at the next group (§5);
- a receipt per object ([jwp-receipts](jwp-receipts.md)), which covers the frame with its
  `number` and `capture_ms`;
- the parameter set by SHA-256 (SPEC §3.1), which a sink checks against the set it holds;
- a catalog from which a sink chooses, before it subscribes, a track it can decode: a frame
  stream, or beside it a track of a floor or recommended video capability (SPEC §5) for a
  sink that holds no networks;
- the integer frame forms (§3), whose frames reproduce on every machine; NMSF registers no
  codec for them, so a frame stream of `wai.video.int_motion.frame` or
  `wai.video.int_hyper.frame` has no NMSF form.

An NMSF track carries what a frame stream does not:

- two-track delivery, the hyperprior on a track of its own at a priority of its own (NMSF
  §3.1, §3.8). A frame object holds both bitstreams; an unwrap to two-track mode gives the
  NMSF subscriber both tracks;
- `colorspace`, `gopSize`, `priority` and the `nvc` object, which `wai_nmsf` keeps (§9.2);
- a `qp` that changes inside a group, which `WNF1` holds once a group
  (`nmsf_group_header_changed`); tensors of any shape, where `WNF1` holds square tensors whose
  side is a power of two, as the reference decoder takes them (§4, `nmsf_shape_unmappable`);
  and frame rates that are not whole thousandths (`nmsf_value_unmappable`).

| code | when |
|---|---|
| `nmsf_catalog_malformed` | an NMSF entry, or `wai_nmsf` or `wai_priors` in a WAI entry, does not read (§9.2) |
| `nmsf_track_mismatch` | two NMSF entries disagree (§9.2), or the two objects of a frame are not two, or differ in their headers (§9.4) |
| `nmsf_codec_unmapped` | an NMSF codec with no frame capability |
| `nmsf_model_unbound` | no model binding for the track's codec and model version (§9.3), or an intra frame whose pin is not the one the WAI entry lists (§9.5) |
| `nmsf_binding_malformed` | a model binding that does not read (§9.3) |
| `nmsf_object_malformed` | an object shorter than its header, whose `payload_len` is not its payload's length, or whose payload is not its components (§9.4) |
| `nmsf_reserved` | a reserved `frame_type` or `qp` |
| `nmsf_group_order` | an Intra frame past Object ID 0, an Inter frame with no group, or not the next `frame_number` or Object ID |
| `nmsf_value_unmappable` | a `pts_ms`, `framerate`, `number`, `q` or `fps_milli` the other side cannot state |
| `nmsf_shape_unmappable` | a tensor or frame `WNF1` cannot state (§9.1) |
| `nmsf_group_header_changed` | an Inter frame whose `qp`, size or tensor shapes are not its group's |
| `nmsf_no_form` | a WAI track with no frame stream, no `wai_nmsf`, or several pins |

### 9.7 Reference implementation and conformance

- `wai-rs` (feature `video_int`): `frame_stream::nmsf` — `parse_object`, `write_object`,
  `parse_components` and `write_component` (§9.1); `read_nmsf`, `Track::bind`,
  `Track::wai_entry` and `read_wai` (§9.2); `Binding` (§9.3); `Rewrap` (§9.4) and `Unwrap`
  (§9.5), which runs `FrameStream` over `ParseOnly`. `frame_stream::Stamp` reads `number` and
  `capture_ms` (§2), and `FrameStream` makes §5's checks of them.
- [`int-payload-conformance/frame-stream/nmsf/`](../int-payload-conformance/frame-stream/nmsf/),
  written by `wai_int_payload_vectors gen` and checked by `wai_int_payload_vectors verify`: a
  64 × 64 stream of two groups as NMSF objects in single-track and two-track mode and the
  frame objects each maps to, the same frame objects for both modes; NMSF objects changed in
  one place each, for every rule of §9.4; NMSF entries and WAI entries for every rule of §9.2;
  model bindings that do not read; frame objects an unwrap refuses, among them the reference
  neural stream of §8, whose frames carry no `number`; and streams whose `number` or
  `capture_ms` a sink refuses (§5). The objects' bitstreams are generated bytes, not an
  encode: the corpus holds the framing and the mapping, and no decode. Every stream's frame
  objects unwrap back to its NMSF objects byte for byte, and every code of §9.6 is given by
  a case.
- `int_payload_verify.py`, which shares no code with wai-rs, checks the same corpus from
  this text: it reads every NMSF object and entry with its own parser, maps every stream
  and requires each committed frame object's payload and manifest, admits the frame objects
  with its own §5 checks and BLAKE3, unwraps them and requires the NMSF objects byte for
  byte, and checks every catalog, binding, unwrap and links case.