open-standards/wai/extensions/frame-stream.md

WAI Extension: Frame Streams

Status: Draft. Carries an inter-frame video capability one frame an object: each object is one whole WAI1 envelope holding one frame; a group opens with an intra frame, which resets the decoder state; and each later frame of the group names its predecessor by content hash, so that a sink refuses a lost, reordered or altered frame instead of decoding past it against the wrong reference. Registers four frame capabilities (§1), the manifest member frame (§2), the neural frame payload WNF1 (§4) beside the integer one, WIF1 (integer-payloads §16), and the checks a sink makes (§5). The MoQ catalog member that declares a frame stream on a track, wai_frame_stream, is moq-streaming-format §4.3. §9 maps a neural frame stream to and from a track of the neural video packaging of the MOQT Streaming Format (NMSF, draft-herz-moq-nmsf-01), byte for byte where both sides state a frame. No transport is defined or changed: a frame stream rides any carrier that delivers objects in order. Reference implementation: wai-rs — frame_stream::{FrameStream, IntFrames, ParseOnly, split_wiv1, split_dcvc, join_dcvc, catalog, nmsf}, neural::DcvcFrames. Conformance: int-payload-conformance/frame-stream/. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

An inter-frame capability's registered payload holds a whole clip: WIV1 codes every frame of a clip in one residual stream (integer-payloads §3), and the reference neural inter-frame payload holds every frame's bitstreams behind one header (§6). A live sender has one frame at a time, and a carrier that moves objects (MoQ, SPEC §1) moves each frame as it is made. Each frame capability below carries one frame of its base's bitstream as its payload:

frame capability	base	payload	parameter set	registry class
wai.video.int_motion.frame	wai.video.int_motion	WIF1, keyframe 0 (integer-payloads §16)	none	StandardDefined
wai.video.int_hyper.frame	wai.video.int_hyper	WIF1, keyframe 1 (integer-payloads §16)	the base's: entropy.json, g_s.whm, h_s.whm	NeuralIntegerExact
wai.video.int_cond.frame	wai.video.int_cond	WCF1 (integer-payloads §17)	the base's: model.wcb	NeuralIntegerExact
wai.neural.dcvc_rt.frame	wai.neural.dcvc_rt	WNF1 (§4)	the base's, sink-supplied	NeuralFloat
wai.neural.dcvc_fm.frame	wai.neural.dcvc_fm	WNF1 (§4)	the base's, sink-supplied	NeuralFloat

A frame capability is a capability of its own because it is a different framing of its base's bitstream, and a registered string's payload format is fixed (SPEC §8); wai.audio.opus.packet is the precedent, a capability of one Opus packet whose decoder state carries from packet to packet. Each is a payload capability, media class video, with its base's fidelity tier and licence class, and decodes against its base's parameter set, of its base's shape: a pin on a frame object names the set its base's pin would name.

2. Groups and the frame member

A frame stream is a sequence of objects of one frame capability, in the order a sink receives them. It is made of groups: a group is one intra frame, then zero or more inter frames, in order. An intra frame's decode reads nothing but its own payload (and the parameter set); an inter frame's reads the state the frames before it in its group left (for WIF1, the previous decoded frame and its intra frame's table; for WNF1 and WCF1, the reference feature).

Every envelope of a frame capability carries the manifest member frame, an object:

member	type	meaning
index	integer, 0 to 2^53 − 1	the frame's position in its group: 0 for its intra frame, the frame before it plus 1 for every other
prev	string, or null	on an inter frame, the content hash (jwp-receipts §2.1) of the envelope of the frame before it in its group, as 64 lowercase hexadecimal digits; null on an intra frame
number	OPTIONAL, integer, 0 to 2^53 − 1	the frame's sequence number in its stream, as its publisher counts it: on every frame of a group whose intra frame carries one, each the one before it plus 1; on none of a group whose intra frame carries none
capture_ms	OPTIONAL, integer, 1 to 2^53 − 1	the wallclock at which the frame was captured, in milliseconds since 1970-01-01T00:00:00Z, as its publisher states it; absent where it states none

"frame": { "index": 3, "prev": "<64 lowercase hexadecimal digits>", "number": 1203, "capture_ms": 1793491239699 }

• A member of frame this revision does not define is ignored. index, number and capture_ms are integers written as one (SPEC §2).
• number and capture_ms are inside the envelope, so the content hash, the next frame's prev and the object's receipt cover them with the frame. A sink holds a group's numbers to their sequence (§5) and reads capture_ms as its publisher's statement.
• prev names the predecessor's whole envelope, its manifest included, and so its own prev: the hash of a group's last frame commits to every frame of the group, in order, as a receipt chain does.
• The content hash is over the envelope's bytes as they travel, the bytes an object receipt names (jwp-receipts §2.1). A sink that receives an encrypted envelope (SPEC §6) takes the hash of what it received, before it opens the payload.
• An encoder writes fallback null: no other capability reads a frame's payload.
• Every frame of a group carries the pin its intra frame carries: both have no conditioning.prior, or both have the same value.

frame is an OPTIONAL manifest member under SPEC §8's rule for a MINOR revision: an envelope that does not carry it keeps its bytes and its meaning. It is carried exactly when the capability is a frame capability. A reader of an earlier revision does not advertise a frame capability, so it finds such an envelope's capability unsupported and the envelope inert (SPEC §4).

3. The integer frame forms

wai.video.int_motion.frame and wai.video.int_hyper.frame carry WIF1 (integer-payloads §16). A group's intra frame carries the group's residual table, its q and offset, and frame 0: with keyframe 0 its residual symbols, predicted from grey; with keyframe 1 the base's nested WIH1, the learned keyframe. Each inter frame carries its motion vectors, its mv_precision and block, and its residual symbols, in a stream of its own. The decode of each frame is WIV1's frame step (integer-payloads §3), through the same stage operators (integer-operators §13.5; the keyframe through whm1/1, §13.2), so:

• a frame decodes to the same bytes on every conforming sink, given its chain: each frame capability has its base's registry class, and its frames' frame-state-equivalence digests (SPEC §7 Digests) are portable;
• a stream split from a clip (§6) decodes, frame by frame, to the clip's frames, byte for byte, and to the digests the clip's decode gives.

wai.video.int_cond.frame carries WCF1 (integer-payloads §17): one frame of conditional inter-frame coding, whose latent is coded given the feature the frame before it in its group left, and which leaves its own feature to the frame after it. Its intra frame reads the zero feature. Each frame is decoded by integer-payloads §17.4 through its pinned set, so the same two properties hold: its frames reproduce on every conforming sink given their chain, and a group split from a WIC1 clip (integer-payloads §17.2, a copy of the clip's frames) decodes to the clip's frames, byte for byte.

4. WNF1 — one frame of the reference neural inter-frame payload

wai.neural.dcvc_rt.frame and wai.neural.dcvc_fm.frame carry WNF1: one frame's two bitstreams, behind the clip header every frame of its group repeats.

off	size	field	rule
0	4	magic WNF1
4	1	version	1
5	1	kind	0 intra, 1 inter
6	2	reserved	0
8	4	h	≥ 1
12	4	w	≥ 1
16	4	fps_milli	frames per second × 1000
20	2	cy	≥ 1: the latent's channels
22	2	cz	≥ 1: the hyper-latent's channels
24	1	sy_log2	≤ 15: the latent's side is sy = 2^sy_log2
25	1	sz_log2	≤ 15: the hyper-latent's side is sz = 2^sz_log2
26	2	q	the quality index the entropy tables are read at
28	2	cf	≥ 1: the reference feature's channels
30	2	reserved	0
32	4	lz	the hyper-latent bitstream's length
36	4	ly	the latent bitstream's length
40	lz	the hyper-latent bitstream
…	ly	the latent bitstream

Nothing follows the latent bitstream. Every multi-byte field is little-endian.

Checks, in this order, each with its code from integer-payloads §8: the magic (bad_magic); the version byte, present (stream_length) and 1 (version); the fixed header of 40 bytes (stream_length); the reserved fields (reserved_nonzero); kind, then h, w, cy, cz and cf at least 1, then sy_log2 and sz_log2 at most 15 (field_range); the frame h·w·3, the latent cy·sy·sy, the hyper-latent cz·sz·sz and the reference feature cf·sy·sy, each at most 2^26 elements (too_large); the two bitstreams (stream_length); nothing after them (trailing).

Decode. The bitstreams are the capability's own, as its reference decoder reads them (neural/video_dcvc.rs): the latents are entropy-decoded from the two bitstreams against the base's parameter set at q, and the base's temporal network maps the latents and the reference feature to the frame and the next reference feature. An intra frame starts the reference feature at zero; an inter frame reads the one the frame before it left, and carries its intra frame's header, every field from h to cf (geometry_mismatch). A decode the capability's decoder cannot complete is refused payload_malformed, and a pin that does not resolve with SPEC §3.1's code. The decode runs floating point: its frames reproduce on one build and not across hardware (NeuralFloat, SPEC §7), so these strings never back intent = replicate. The chain of §2 still holds every frame to its predecessor: two sinks may present different pixels, never different frames.

5. What a sink checks

A sink holds, for each frame stream it decodes, its reference: the last frame it decoded, its content hash, its index and its pin, and the decoder state the frame left. It checks each object it receives, in this order, and refuses it at the first rule that fails:

1. The envelope reads (SPEC §2): envelope_malformed.
2. Its capability is the stream's: frame_capability_mismatch.
3. It carries frame, an object: frame_link_missing.
4. frame's index is an integer from 0 to 2^53 − 1, its prev is null or 64 lowercase hexadecimal digits, its number, where present, is an integer from 0 to 2^53 − 1, and its capture_ms, where present, is an integer from 1 to 2^53 − 1: frame_link_malformed.
5. The payload parses alone: integer-payloads §16 for WIF1, §17.1 for WCF1, §4 for WNF1, each with its code.
6. frame agrees with the payload's kind: an intra frame has index 0 and prev null, an inter frame an index of 1 or more and a prev: frame_link_mismatch.

An intra frame then drops the reference, is decoded (SPEC §3.1's pin checks, then the decode's codes), and becomes the reference. An inter frame is checked against the reference first:

7. The sink holds a reference: frame_no_reference.
8. Its prev is the reference's content hash: frame_gap when its index is past the one after the reference's (frames are missing), frame_prev_mismatch otherwise.
9. Its index is the reference's plus 1: frame_index_mismatch. Its number is the reference's plus 1 where the reference carries one, and absent where the reference carries none: frame_number_mismatch.
10. Its conditioning.prior is the reference's, compared as JSON values (both absent counts as the same): frame_prior_changed.
11. It is decoded against the reference: for WIF1, its h × w is its group's (geometry_mismatch), then the decode's codes; for WCF1, its header, h to cf, is its group's (geometry_mismatch), then the decode's codes (integer-payloads §17.4); for WNF1, its header is its group's (geometry_mismatch), then the decode's.

It then becomes the reference. After any refusal the sink holds no reference: every inter frame it receives is refused (frame_no_reference) until an intra frame opens a group. So a sink never decodes an inter frame against a reference that is not its predecessor's decode: a lost frame, a frame delivered twice or out of order, and an altered prev are each refused at the first frame they would corrupt, and the stream resumes at the next group. A sink that joins a stream joins at an intra frame.

code	when
envelope_malformed	the object is not a WAI1 envelope that reads
frame_capability_mismatch	the object's capability is not the stream's
frame_link_missing	no frame member, or one that is not an object
frame_link_malformed	an index or prev outside its type or range
frame_link_mismatch	an intra frame with a prev or an index other than 0, or an inter frame with no prev or index 0
frame_no_reference	an inter frame while the sink holds no reference
frame_gap	an inter frame whose predecessor is not the reference, with frames missing between
frame_prev_mismatch	an inter frame whose prev is not the reference's content hash
frame_index_mismatch	an inter frame that names the reference but whose index is not the next
frame_number_mismatch	an inter frame whose number is not the one after the reference's, or that carries one where the reference carries none, or none where it carries one
frame_prior_changed	an inter frame whose pin is not its group's

A payload's refusals are integer-payloads §8's codes; a pin's are SPEC §3.1's.

6. Splitting a clip

WIV1. A WIV1 clip is one group of WIF1 frames, frame for frame. Frame 0 is the intra frame: the clip's table, q and offset, and, with keyframe 0, frame 0's residual symbols; with keyframe 1, the clip's nested WIH1. Each frame f ≥ 1 is an inter frame: frame f's motion vectors, the clip's mv_precision and block, and frame f's residual symbols. Each frame's symbols are the ones the clip's residual stream decodes to for that frame, coded into a stream of their own by the rANS coder integer-payloads §2 decodes (its state starting at 2^31, so each stream ends canonically). The group decodes, frame by frame, to the clip's frames, byte for byte (§3). The split refuses a clip whose residual stream does not decode: stream_exhausted, stream_noncanonical, and the parse's codes. A stream of several groups is the split of several clips, one after another.

The reference neural payload. The reference decoder of wai.neural.dcvc_rt and wai.neural.dcvc_fm reads a whole clip as: h, w, n_frames, fps_milli (u32), cy, cz (u16), sy_log2, sz_log2 (u8), q, cf (u16), 26 bytes; then n_frames pairs of u32 lengths (lz, ly); then each frame's hyper-latent and latent bitstreams, in order. Its split is one group of WNF1 frames, each with the clip's header and its own two bitstreams, copied; frame 0 intra, every other inter. The split reads the clip strictly: its header as WNF1's checks read it, at least one frame (field_range), every bitstream inside it (stream_length) and nothing after the last (trailing). The join is its inverse: one group of WNF1 frames, the first intra and the others inter (field_range), each with the first's header (geometry_mismatch), back to the clip. The split and the join give each other's input back, byte for byte, and the decode of the frames is the decode of the clip: the reference decoder's whole-clip decode runs the same step a frame (neural/video_dcvc.rs, Temporal::step).

7. Carriage

A frame stream rides any carrier that delivers its objects in order; this extension defines no transport. Each object keeps every rule of the carrier for a WAI1 object, and its receipt is an object receipt like any other (jwp-receipts).

• MoQ. One track carries one frame stream, its catalog entry declaring it (wai_frame_stream, moq-streaming-format §4.3). Each MOQT group of the track is one group of the stream, and the object of Object ID k in it is the frame of index k. A subscriber joins at a group's first object (MOQT §3.5). A frame a relay drops is refused as a gap at the frame after it, never decoded past; the group's receipts still account for it (moq-streaming-format §5). A frame stream of a neural frame capability maps to and from a track of packaging nvc (§9).
• HTTP and files. A sequence of envelopes, in order: a playlist's objects (http-delivery), or a file of envelopes one after another.

A frame object is complete when its frame is coded, so a live sender publishes each frame as it is made, and a sink presents it when it is decoded.

8. Reference implementation and conformance

• wai-rs (feature video_int): frame_stream::FrameStream holds a stream's reference and makes §5's checks in order, decoding through a FrameCodec: IntFrames, the decoder of the integer frame forms (int_video::decode_unit_typed, the clip decode's frame step, and, for a keyframe, int_payload::decode_keyframe through the capability's set as its pin resolves it); and ParseOnly, which parses each frame and checks its geometry against its group's intra frame without decoding. frame_stream::split_wiv1, split_dcvc and join_dcvc are §6's; frame_stream::envelope and wrap_group write frame objects; frame_stream::catalog reads wai_frame_stream and checks an object against it; container::check_int_payload_framing parses WIF1, WCF1 and WNF1 on every emit path. int_cond_video::CondFrames decodes wai.video.int_cond.frame, keeping the reference feature from frame to frame (integer-payloads §17.5); its corpus is int-payload-conformance/int-cond/. neural::DcvcFrames (features neural and video_int) decodes the neural frame forms through the reference decoder's Temporal::step.
• int-payload-conformance/frame-stream/, written by wai_int_payload_vectors gen and checked by wai_int_payload_vectors verify: the committed WIV1 clips int_video, video_real (half-pel) and video_nk (a learned keyframe, pinned) split into frame objects, each frame decoding to its clip's frame byte for byte; two clips as one stream of two groups; a stream with each of §5's faults planted; WIF1 and WNF1 payloads each changed in one place; a synthetic neural clip, its split, its join and their refusals; and catalog entries with objects checked against them. The neural clip's bitstreams are generated bytes, not an encode: the corpus holds its framing and its chain, and no decode, which needs the sink-supplied networks.
• int_payload_verify.py, which shares no code with wai-rs, checks the same corpus from this text: it splits each clip again with its own rANS coder and requires the committed payloads, decodes every keyframe-0 frame and requires each frame's SHA-256, makes §5's checks over every stream with its own BLAKE3, and checks every payload, split, join and catalog case. It does not decode a WIH1 keyframe; those streams it checks to §5 and the frames' parse, and wai-rs checks their decode.

9. NMSF tracks

draft-herz-moq-nmsf-01 (NMSF) packages the frames of a neural inter-frame codec for the MOQT Streaming Format as packaging nvc: one frame an object, a group of pictures a MOQT group that opens with a neural keyframe (NMSF §3.5), the model's weights installed on both sides and named, not sent (NMSF §8). A frame stream of wai.neural.dcvc_rt.frame or wai.neural.dcvc_fm.frame carries the same bitstreams, so a publisher maps one to the other: an NMSF frame becomes one WNF1 frame object (a rewrap, §9.4), and a frame object becomes one NMSF object, or two (an unwrap, §9.5). Neither changes a bitstream; an unwrap of a rewrap gives the NMSF objects back byte for byte; and a relay forwards both tracks as it forwards any track. A section number of NMSF here is one of that revision's; where a later revision renumbers or changes what this section cites, this section is what changes.

9.1 What maps

An NMSF object is a 26-byte header and a payload (NMSF §3.2). This section reads the header from the field definitions, which give each field's size and sum to the 26 bytes the section states; the figure before them draws pts_ms one byte short. Every field is big-endian:

off	size	field
0	1	frame_type: 0x00 Intra, 0x01 Inter, 0x02 to 0xFF reserved (NMSF §3.3)
1	1	qp: 0 to 63; 64 to 255 reserved
2	4	frame_number
6	8	pts_ms
14	4	width
18	4	height
22	4	payload_len
26	payload_len	the payload

The payload is components (NMSF §3.6), each channels, height, width and data_len (u32, big-endian) and then data_len bytes of an entropy-coded tensor. In single-track mode an object's payload is the hyperprior's component and then the latent's; in two-track mode the hyperprior track's object holds the hyperprior's component and the latent track's object the latent's, the two objects of a frame with the same header (NMSF §3.4). A frame object holds the same frame as:

NMSF	frame object	where it has no form
codec dcvc-rt, dcvc-fm (NMSF §5)	the capability wai.neural.dcvc_rt.frame, wai.neural.dcvc_fm.frame; conditioning.kind dcvc_rt, dcvc_fm	any other codec: nmsf_codec_unmapped. dcvc-dc is SPEC §5's candidate wai.neural.dcvc_dc, which pins no bitstream and has no frame form; ssf, fvc, rlvc and elfvc WAI does not register
a MOQT group: an Intra frame, then Inter frames	a group of the stream (§2); the object of Object ID k is the frame of index k (moq-streaming-format §4.3)
frame_type Intra, Inter	WNF1 kind 0 (index 0, prev null), 1 (index 1 or more, prev the frame before)	a reserved type: nmsf_reserved
qp	WNF1 q	a reserved qp: nmsf_reserved; a q past 63 (§9.5): nmsf_value_unmappable
frame_number	frame.number (§2)	a frame with no number, or one past 2^32 − 1 (§9.5): nmsf_value_unmappable
pts_ms; 0 where it states none	frame.capture_ms (§2); absent where it states none	a pts_ms past 2^53 − 1: nmsf_value_unmappable
width, height	WNF1 w, h
the hyperprior's component	WNF1 cz, sz_log2 (the side is 2^sz_log2) and the hyper-latent bitstream	a tensor that is not square, whose side is not a power of two at most 2^15, or of channels outside 1 to 65 535 (§4): nmsf_shape_unmappable
the latent's component	WNF1 cy, sy_log2 and the latent bitstream	as the hyperprior's
the catalog's framerate	WNF1 fps_milli	a framerate that is not a whole number of thousandths (§9.2), or a frame of another fps_milli (§9.5): nmsf_value_unmappable
—	WNF1 cf, the reference feature's channels, from the model binding (§9.3)
codec and nvc.modelVersion, the model by name	conditioning.prior and wai_priors, its parameter set by SHA-256 (SPEC §3.1), through the model binding

9.2 The catalog member wai_nmsf

A WAI track whose frames map to an NMSF track keeps that track's catalog entries in the member wai_nmsf: an array of one entry (single-track mode) or two (two-track mode, a hyperprior and a latent track, NMSF §3.1), each a JSON object as NMSF §3.7 and §3.8 state it, in the order given. The frame objects do not carry colorspace, gopSize, priority, nvcRole, depends or the nvc object, and wai_nmsf keeps them, so the entries an unwrap publishes are the entries the rewrap read, member for member. A rewrap publishes the entry:

{ "name": "video/wai", "packaging": "wai", "isLive": true, "role": "video",
  "width": 64, "height": 64, "framerate": 30,
  "wai_capabilities": ["wai.neural.dcvc_rt.frame"],
  "wai_frame_stream": { "capability": "wai.neural.dcvc_rt.frame" },
  "wai_priors": [ { "capability": "wai.neural.dcvc_rt.frame",
                    "sha256": "<64 lowercase hexadecimal digits>",
                    "form": "wai.prior.set", "tier": "none" } ],
  "wai_nmsf": [ { "name": "video", "packaging": "nvc", "isLive": true, "codec": "dcvc-rt",
                  "width": 64, "height": 64, "framerate": 30, "colorspace": "ycbcr-bt709",
                  "gopSize": 4, "nvc": { "modelVersion": "cvpr2025" } } ] }

Its name is the publisher's; isLive, width, height and framerate are the first NMSF entry's; it declares the frame stream (moq-streaming-format §4.3); wai_priors (§4.2) lists the binding's pin, with its form and tier where the pin states them, and is absent where the binding has no pin; wai_nmsf holds the NMSF entries.

Reading NMSF entries. A publisher reads them, in this order, and refuses the track at the first rule that fails:

1. Each entry is an object with name (a non-empty string), packaging "nvc", isLive (a boolean), codec and colorspace (strings), width, height and gopSize (integers of at least 1) and framerate (a number above 0); priority, where present, a number; nvcRole and depends, where present, strings; and nvc, where present, an object whose modelVersion, where present, is a string. There is one entry, with no nvcRole, or two: one of nvcRole "hyperprior" and one of "latent", whose depends is the hyperprior entry's name, the two names different. Otherwise nmsf_catalog_malformed.
2. Two entries agree on codec, isLive, width, height, framerate (as numbers), colorspace, gopSize and nvc.modelVersion (both absent counts as the same): nmsf_track_mismatch.
3. The codec has a frame capability (§9.1): nmsf_codec_unmapped.
4. fps_milli is the integer m nearest framerate × 1000, from 1 to 2^32 − 1, where m ÷ 1000 is framerate in binary64 arithmetic: 29.97 is 29 970, and 30 000 ÷ 1 001 has none (nmsf_value_unmappable).

Reading a WAI entry. An unwrap reads the entry's frame stream (moq-streaming-format §4.3, catalog_frame_stream_malformed; a track that declares none has no NMSF form, nmsf_no_form); its wai_nmsf (absent or null: nmsf_no_form; not an array: nmsf_catalog_malformed), whose entries read as above and whose codec's frame capability is the declared one (nmsf_catalog_malformed); and the pin wai_priors lists for that capability: each of its entries an object, and one for the capability with a sha256 of 64 lowercase hexadecimal digits and a form that is absent, null or a string (nmsf_catalog_malformed). It lists none, and the frames carry no pin; or one, and they carry it; or several, and the track has no NMSF form, which names one model (nmsf_no_form). A publisher that unwraps a frame stream it wrote itself states its NMSF entries in wai_nmsf: the colorspace and GOP size its frames do not carry are its to state.

9.3 The model binding

NMSF names a model by codec and nvc.modelVersion and leaves the distribution of its weights to the deployment (NMSF §8). A frame object names its parameter set by SHA-256 in its pin, which a sink checks before it decodes (SPEC §3.1). A model binding is the publisher's statement of the set a name stands for, and of what WNF1 needs of the model that an NMSF object does not carry:

{ "codec": "dcvc-rt", "modelVersion": "cvpr2025", "cf": 64,
  "prior": { "id": "dcvc-rt/cvpr2025", "sha256": "<64 lowercase hexadecimal digits>",
             "determinism": "float", "form": "wai.prior.set", "tier": "none" } }

member	presence	meaning
codec	REQUIRED, string	the NMSF codec identifier
modelVersion	string, null or absent	the nvc.modelVersion it binds; null or absent binds a track that names none
cf	REQUIRED, integer, 1 to 65 535	the channels of the model's reference feature, WNF1's cf (§4)
prior	a pin (SPEC §3.1), null or absent	the parameter set every frame object carries as its pin; null or absent, the frames carry none

A binding that does not read so is nmsf_binding_malformed. A rewrap maps a track through the first binding whose codec and modelVersion are the track's: none is nmsf_model_unbound. The neural frame forms are NeuralFloat, so a binding's pin claims no tier above none (SPEC §3.1). A binding is the publisher's configuration and is not carried: in the catalog a rewrap publishes, its name is in wai_nmsf and its pin in wai_priors, so an unwrap reads both from there.

9.4 Rewrap: NMSF to frame objects

A publisher maps an NMSF track's frames in the order it receives them: each frame one object, or in two-track mode the hyperprior track's object and the latent track's object of the frame (NMSF §4.2), with the MOQT Object ID where the carrier gives one. It holds the group it is mapping: its last frame's index, frame_number, WNF1 header and frame object's content hash. It checks each frame in this order, and refuses it at the first rule that fails:

1. The frame is one object in single-track mode and two in two-track mode: nmsf_track_mismatch.
2. Each object, in order, is at least 26 bytes and its payload_len is the length of the bytes after its header (nmsf_object_malformed), and its frame_type and qp are not reserved (nmsf_reserved).
3. The two objects of a frame have the same header, payload_len aside: nmsf_track_mismatch.
4. Components fill each payload: two in single-track mode, one in each object in two-track mode: nmsf_object_malformed.
5. An Intra frame drops the group, and is at Object ID 0 where it has one. An Inter frame follows a group, its frame_number the group's last plus 1 and its Object ID, where it has one, the group's last index plus 1. Otherwise nmsf_group_order.
6. Its pts_ms is at most 2^53 − 1: nmsf_value_unmappable.
7. Each component has a WNF1 form (§9.1), and the WNF1 header the frame gives passes §4's checks: nmsf_shape_unmappable.
8. An Inter frame's WNF1 header is its group's, so its qp, width, height and tensor shapes are its Intra frame's: nmsf_group_header_changed.

It then writes the frame object: a WAI1 envelope of the codec's frame capability, media video, intent perceptual, conditioning.kind the codec's (§9.1) and conditioning.prior the binding's pin, with no fallback (§2). Its payload is the WNF1 frame of kind frame_type, h height, w width, fps_milli the track's (§9.2), cy, cz, sy_log2 and sz_log2 the components', q qp, cf the binding's, and the components' data as its two bitstreams. Its frame is index, prev, number frame_number and capture_ms pts_ms, absent where pts_ms is 0. The frame object becomes the group's last.

After any refusal the publisher holds no group: it refuses every Inter frame (nmsf_group_order) until an Intra frame opens one. A lost or damaged NMSF object therefore yields no frame object for any frame its loss would have changed. A single-track frame and the two objects of the same frame in two-track mode give the same frame object, which a sink admits as any frame (§5).

9.5 Unwrap: frame objects to NMSF

A publisher whose WAI entry reads (§9.2) maps each frame object in order, and refuses it at the first rule that fails:

1. It is admitted as a sink admits a frame (§5), its payload parsed and held to its group's header as ParseOnly holds it: §5's codes and §4's.
2. An intra frame's pin is the one the entry lists: the same sha256 and form, or no pin where the entry lists none: nmsf_model_unbound. §5 holds each inter frame's pin to its group's.
3. Its frame.number is present and at most 2^32 − 1: nmsf_value_unmappable.
4. Its WNF1 q is at most 63 and its fps_milli is the track's: nmsf_value_unmappable.

It then writes the NMSF object, or the two: the header frame_type kind, qp q, frame_number number, pts_ms capture_ms (0 where absent), width w, height h; the hyperprior's component (cz channels, a side of 2^sz_log2 both ways, the hyper-latent bitstream) and the latent's (cy, 2^sy_log2, the latent bitstream). In single-track mode one object holds both components; in two-track mode the hyperprior track's object holds the first and the latent track's object the second, the same header on both. The frame's cf is not carried. After any refusal the publisher holds no reference (§5), and refuses every inter frame until an intra frame opens a group.

9.6 What each side carries, and the codes

A frame object carries what an NMSF object does not:

• the link to the frame before it (§2): a sink refuses a lost, reordered or altered frame at the first frame it would change, and resumes at the next group (§5);
• a receipt per object (jwp-receipts), which covers the frame with its number and capture_ms;
• the parameter set by SHA-256 (SPEC §3.1), which a sink checks against the set it holds;
• a catalog from which a sink chooses, before it subscribes, a track it can decode: a frame stream, or beside it a track of a floor or recommended video capability (SPEC §5) for a sink that holds no networks;
• the integer frame forms (§3), whose frames reproduce on every machine; NMSF registers no codec for them, so a frame stream of wai.video.int_motion.frame or wai.video.int_hyper.frame has no NMSF form.

An NMSF track carries what a frame stream does not:

• two-track delivery, the hyperprior on a track of its own at a priority of its own (NMSF §3.1, §3.8). A frame object holds both bitstreams; an unwrap to two-track mode gives the NMSF subscriber both tracks;
• colorspace, gopSize, priority and the nvc object, which wai_nmsf keeps (§9.2);
• a qp that changes inside a group, which WNF1 holds once a group (nmsf_group_header_changed); tensors of any shape, where WNF1 holds square tensors whose side is a power of two, as the reference decoder takes them (§4, nmsf_shape_unmappable); and frame rates that are not whole thousandths (nmsf_value_unmappable).

code	when
nmsf_catalog_malformed	an NMSF entry, or wai_nmsf or wai_priors in a WAI entry, does not read (§9.2)
nmsf_track_mismatch	two NMSF entries disagree (§9.2), or the two objects of a frame are not two, or differ in their headers (§9.4)
nmsf_codec_unmapped	an NMSF codec with no frame capability
nmsf_model_unbound	no model binding for the track's codec and model version (§9.3), or an intra frame whose pin is not the one the WAI entry lists (§9.5)
nmsf_binding_malformed	a model binding that does not read (§9.3)
nmsf_object_malformed	an object shorter than its header, whose payload_len is not its payload's length, or whose payload is not its components (§9.4)
nmsf_reserved	a reserved frame_type or qp
nmsf_group_order	an Intra frame past Object ID 0, an Inter frame with no group, or not the next frame_number or Object ID
nmsf_value_unmappable	a pts_ms, framerate, number, q or fps_milli the other side cannot state
nmsf_shape_unmappable	a tensor or frame WNF1 cannot state (§9.1)
nmsf_group_header_changed	an Inter frame whose qp, size or tensor shapes are not its group's
nmsf_no_form	a WAI track with no frame stream, no wai_nmsf, or several pins

9.7 Reference implementation and conformance

• wai-rs (feature video_int): frame_stream::nmsf — parse_object, write_object, parse_components and write_component (§9.1); read_nmsf, Track::bind, Track::wai_entry and read_wai (§9.2); Binding (§9.3); Rewrap (§9.4) and Unwrap (§9.5), which runs FrameStream over ParseOnly. frame_stream::Stamp reads number and capture_ms (§2), and FrameStream makes §5's checks of them.
• int-payload-conformance/frame-stream/nmsf/, written by wai_int_payload_vectors gen and checked by wai_int_payload_vectors verify: a 64 × 64 stream of two groups as NMSF objects in single-track and two-track mode and the frame objects each maps to, the same frame objects for both modes; NMSF objects changed in one place each, for every rule of §9.4; NMSF entries and WAI entries for every rule of §9.2; model bindings that do not read; frame objects an unwrap refuses, among them the reference neural stream of §8, whose frames carry no number; and streams whose number or capture_ms a sink refuses (§5). The objects' bitstreams are generated bytes, not an encode: the corpus holds the framing and the mapping, and no decode. Every stream's frame objects unwrap back to its NMSF objects byte for byte, and every code of §9.6 is given by a case.
• int_payload_verify.py, which shares no code with wai-rs, checks the same corpus from this text: it reads every NMSF object and entry with its own parser, maps every stream and requires each committed frame object's payload and manifest, admits the frame objects with its own §5 checks and BLAKE3, unwraps them and requires the NMSF objects byte for byte, and checks every catalog, binding, unwrap and links case.