open-standards/wai/extensions/integer-payloads.md

WAI Extension: Integer Payloads

Status: Draft. Pins the payload formats of the WAI-native integer codecs that SPEC.md §5 registers without one (Payloads of the WAI-native integer codecs): WIV1 for wai.video.int_motion and wai.video.int_hyper, WIH1 for wai.neural.int_hyper, WIA1 for wai.audio.int_codec and WIS1 for wai.splat.int_codec. Defines each layout, its decode, the checks a sink makes and the closed list of refusal codes; names each capability's parameter-set members; and makes the JSON the encode and export tools write a converter input, never a payload. Writes down the payload formats and parameter sets that SPEC §5 keeps for wai.neural.int_synth (§13) and wai.neural.int_mlicpp (WIM1 and WMB1, §14). Reference impl: wai-rs — int_payload::{wire, decode_int, resolve}, int_payload::{wiv1_from_json, wih1_from_json, wia1_from_json, wis1_from_json}, codecs::PINNED_INT_PAYLOADS, container::check_int_payload_framing. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

capability	payload	parameter set (§9)
wai.video.int_motion	WIV1 with keyframe 0 (§3)	none
wai.video.int_hyper	WIV1 with keyframe 1, a nested WIH1 (§3)	several files: entropy.json, g_s.whm, h_s.whm
wai.neural.int_hyper	WIH1 (§4)	several files: entropy.json, g_s.whm, h_s.whm
wai.audio.int_codec	WIA1 (§5)	one file: model.wam
wai.splat.int_codec	WIS1 (§6)	one file: g_s.whm

Why this pin is permitted. Until this revision, these capabilities' SPEC §5 rows named the decoder that read their payload, and no byte layout. The rows of wai.video.int_motion, wai.audio.int_codec and wai.splat.int_codec also named the encode or export tool that wrote a bitstream, a JSON document; those of wai.neural.int_hyper and wai.video.int_hyper named their decoders only. SPEC §5 called every row's payload the bytes its codec normally emits, which for these rows was the tools' JSON. So this revision changes what these strings carry. SPEC §8 permits it under Completing a registration: a revision may complete the registration of a WAI-native string whose row named a decoder and fixed no byte layout by pinning one, and it lists the envelopes that become non-conforming (§11; SPEC §5). SPEC §8's rule that a registered string's payload format is fixed forever applies to these strings from this revision on. The encode and export tools still write JSON that carries the same fields; it is a converter input (§10).

wai.neural.int_synth and wai.neural.int_mlicpp keep the payload formats SPEC §5 gives them, and §13 and §14 write those formats down. A sink decodes them and the five strings above (wai.video.int_motion, wai.video.int_hyper, wai.neural.int_hyper, wai.audio.int_codec, wai.splat.int_codec) through one path (int_payload::decode_int), and §8 names the codes it reports for those two.

2. Common rules

Encoding. Every multi-byte field is little-endian. Offsets are in bytes.

Framing. A payload starts with its four-byte magic and a version byte. This revision defines version 1 for every format. Every reserved field is zero. Nothing follows the last section.

rANS streams. A stream is a u32 length L followed by L bytes. L is a multiple of 4 and at least 8. It is decoded as follows (precision 16, with a bypass escape of 4-bit groups):

• The state x starts as (w1 << 32) | w0, from the stream's first two u32 words. Renormalising: if x < 2^31, then x = (x << 32) | w, where w is the next unread word; if no word is left, the stream has run out and x is left as it is.
• bits(n): v = x mod 2^n, x = x >> n, renormalise, return v.
• Decoding a symbol against a table cdf of length n with offset o: let cum = x mod 2^16 and s be the largest index with cdf[s] ≤ cum (0 ≤ s ≤ n − 2); then x = (cdf[s+1] − cdf[s]) · (x >> 16) + cum − cdf[s], and renormalise. If s < n − 2, the value is s. If s = n − 2 (the escape): let v = bits(4) and count = v; then, at most 8 times, while v = 15 and the stream has not run out, v = bits(4) and count = count + v; then count = min(count, 8) and raw = Σ_{j < count} bits(4) << 4j. The value is −(raw >> 1) − 1 if raw is odd and (raw >> 1) + (n − 2) if it is even, saturated to i32. The symbol is value + o, saturated to i32.
• A decode stops at the first symbol during whose decode its stream runs out, and refuses the payload (stream_exhausted): the value that symbol would give is not used, and nothing after it is decoded. A stream that never runs out decodes every symbol, so a conforming payload is unaffected.
• After the last symbol a decode reads from a stream, the stream ends canonically: every word of it has been read, and the state is 2^31. Otherwise the payload is refused (stream_noncanonical). An encoder starts its state at 2^31 and writes the words it flushes and no others, so its streams end this way. A word after the last symbol's, or a state the encoder did not start from, codes the same symbols in other bytes; the rule closes both, so neither can give one decode several content hashes. A stream that codes no symbol (the residual stream of a one-frame WIV1 with keyframe 1) is the state 2^31 alone: the words 2^31 and 0.

The decode these rules define is integer-operators E1, rans64_decode, with the modes t4097, lenient and canonical.

One clip, several payloads. These rules fix how a payload is read, not the only payload of a decoded output: tables that differ in entries the stream never selects (a symbol the clip does not use, say), or a different escape layout, code the same output in other bytes. A payload's hash therefore identifies the payload, not what it decodes to. Where one output must have one identity, it is identified by the digest of its decoded output: for a clip, its frame-state-equivalence digests (SPEC §7, a video receipt's clip_hash); for the other formats, their capabilities' equivalence digests.

CDF tables. A table is n entries of u32 with 3 ≤ n ≤ 4097, cdf[0] = 0, cdf[n − 1] = 65536 and every entry greater than the one before. These rules are what keep the symbol search and the state update above in range.

Order of the checks. A parser checks, in this order, and refuses with the first that fails:

1. the magic: the capability's (§1), or else format_capability_mismatch if it is another of the four magics, or bad_magic;
2. the version byte: present (stream_length), and 1 (version);
3. the fixed header: present (stream_length);
4. the reserved fields (reserved_nonzero);
5. the header's fields, in the order the format's section lists their rules (field_range), then, for WIV1, the keyframe kind (keyframe_capability_mismatch);
6. the caps of §7 on the sizes the header declares (too_large), in the order the format's section lists them;
7. each later section in order: present (stream_length), then its rules (field_range, cdf_malformed). A section of one table per channel (WIA1, WIS1) is read table by table: each table present, then checked, before the next;
8. nothing after the last section (trailing, or nested_length for WIV1's nested keyframe).

A decode then makes the checks of §7, and refuses a stream that runs out or does not end canonically (§2). Two conforming sinks refuse the same bytes with the same code.

3. WIV1 — integer inter-frame video

off	size	field	rule
0	4	magic WIV1
4	1	version	1
5	1	mv_precision	1 (whole-pel) or 2 (half-pel); 1 when n_frames is 1
6	1	keyframe	0 or 1
7	1	reserved	0
8	4	n_frames	≥ 1
12	4	h	≥ 1
16	4	w	≥ 1
20	4	block	≥ 1; 1 when n_frames is 1
24	4	q	1..=65535
28	4	offset	i32
32	2	cdf_len	3..=4097
34	2	reserved	0
36	cdf_len·4	the residual table	§2
…	(n_frames − 1)·nb·4	motion vectors	i16 dy, i16 dx
…	4 + L	the residual stream	§2
…	4 + K	keyframe 1 only: u32 K, then a WIH1 payload	K is exactly the bytes that remain

The header's field rules are checked in the order mv_precision, keyframe, n_frames, h and w, block, q, cdf_len, then, for a one-frame clip, mv_precision 1 and block 1: such a clip codes no motion vector, so neither field would change its decode, and they are fixed so that it has one header. keyframe 0 is wai.video.int_motion's and 1 is wai.video.int_hyper's; the other is keyframe_capability_mismatch. The frame h·w·3, the clip n_frames·h·w·3, the frame count n_frames and the motion-vector field 2·nb·(n_frames − 1) are capped (§7), in that order.

Motion vectors. nb = ⌈h/block⌉·⌈w/block⌉ and nbx = ⌈w/block⌉. The vectors are frames 1 to n_frames − 1 in order, each its block grid row-major: the vector of pixel (y, x) in frame f is entry (f − 1)·nb + (y div block)·nbx + (x div block), in units of 1/mv_precision pel.

Decode. One decoder reads the residual stream for the whole clip, against the residual table at offset. For each frame in order, for each pixel in raster order (y, x), for each channel c of R, G, B: decode the symbol r, and the sample is clamp(pred + r·q, 0, 255). Since |r| < 2^31 and q < 2^16, every step fits i64.

• Frame 0 with keyframe 0: pred = 128.
• Frame 0 with keyframe 1: the frame is the nested WIH1's decode (§4), which reads no symbol of the residual stream. It must be h × w (geometry_mismatch).
• Frame f ≥ 1: pred reads the previous decoded frame R with the pixel's vector (dy, dx), where p(Y, X) is R at row clamp(Y, 0, h − 1), column clamp(X, 0, w − 1), channel c:
  • whole-pel: pred = p(y + dy, x + dx);
  • half-pel: iy = y + (dy >> 1), fy = dy & 1, ix = x + (dx >> 1), fx = dx & 1 (arithmetic shift); then p(iy, ix) when both are 0, (p(iy, ix) + p(iy, ix + 1) + 1) >> 1 when only fx is 1, (p(iy, ix) + p(iy + 1, ix) + 1) >> 1 when only fy is 1, and (p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2 when both are.

The output is n_frames RGB8 frames of h × w (rows top to bottom, pixels left to right, R, G, B bytes): the buffers of the capability's frame-state-equivalence digests (SPEC §7 Digests).

This decode is the stage operators of integer-operators §5E, with no graph (§13.5 there): the residual stream is E1 rans64_decode with t4097, lenient and canonical, the prediction of frames f ≥ 1 is E10 mc_predict, and the sample is E9 residual_step on [0, 255]. The reference decoder decodes through them.

4. WIH1 — the integer conditional image codec

off	size	field	rule
0	4	magic WIH1
4	1	version	1
5	3	reserved	0
8	4	m	≥ 1: latent channels
12	4	sh	≥ 1: latent height
16	4	sw	≥ 1: latent width
20	4	n_z	≥ 1: hyper-latent channels
24	4	zh	≥ 1: hyper-latent height
28	4	zw	≥ 1: hyper-latent width
32	4 + Lz	the hyper-latent stream	§2
…	4 + Ly	the latent stream	§2

The caps (§7) apply to n_z·zh·zw and to 2·m·sh·sw.

Decode, against the set of §9 (g_s.whm and h_s.whm, two integer networks at one fixed-point scale 2^afrac; entropy.json, the tables):

1. For each hyper-latent channel k and each of its zh·zw elements, decode a symbol s from the hyper-latent stream against channel k's table and offset; the element is s·2^afrac + median_k.
2. h_s on the hyper-latent [n_z, zh, zw] gives [2·m, sh, sw]: its first m channels are the scales σ, the rest the means μ. h_s decodes through its lowering whm1/1 (integer-operators §13.2); the split of its output into σ and μ is a step of this decode, after the shape check, not part of the lowered graph (integer-operators §13.5).
3. For each latent element p in order (channel, then row, then column), its table is b, where, with the scale edges e_0 … e_{N−1} and σ' = max(σ_p, e_0), b = N − 1 − #{ i < N − 1 : σ' ≤ e_i }. Decode a symbol r from the latent stream against table b and its offset; the element is r·2^afrac + μ_p.
4. g_s on the latent [m, sh, sw] gives three channels; each value v is clamped to [0, 2^afrac] and becomes (v·255 + 2^(afrac−1)) >> afrac, as RGB8. g_s decodes through its lowering whm1/1 (integer-operators §13.2), and this is the RGB8 step that ends its lowered graph (§13.0).

Every value is formed in i64 and must fit it (§7).

5. WIA1 — the integer learned audio codec

off	size	field	rule
0	4	magic WIA1
4	1	version	1
5	1	afrac	1..=30; equal to the model's (§7)
6	2	reserved	0
8	4	sr	≥ 1: the sample rate the PCM is presented at, in hertz
12	4	n_ch	≥ 1: latent channels
16	4	lat_len	≥ 1: latent length
20	4	t_out	≥ 1: samples presented
24	4	offset	i32: the symbol offset of every channel's table
28	2	cdf_len	3..=4097
30	2	reserved	0
32	n_ch·8	medians	i64 each, `
…	n_ch·cdf_len·4	one table per channel	§2
…	4 + L	the latent stream	§2

The header's field rules are checked in the order afrac, then sr, n_ch, lat_len and t_out, then cdf_len. The caps (§7) apply to n_ch·lat_len and to t_out.

Decode, against the set of §9 (model.wam, the integer synthesis):

1. For each channel k and each of its lat_len elements, decode a symbol s against channel k's table at offset; the element is s·2^afrac + median_k.
2. The synthesis (one-dimensional integer transposed convolutions and integer inverse GDN) on [n_ch, lat_len] gives one channel of t ≥ t_out samples. It decodes through its lowering wam1/1 (integer-operators §13.3), with t_out as its scalar.
3. The first t_out samples v become (v·32767 + 2^(afrac−1)) >> afrac, clamped to [−32768, 32767]: the PCM step that ends the lowered graph (integer-operators §13.3). A sample for which v·32767 + 2^(afrac−1) leaves i64 is refused (field_range).

The output is t_out samples of i16 PCM, one channel, at sr hertz: the buffer of the capability's sample-equivalence digest (signed 16-bit little-endian samples).

6. WIS1 — the integer learned Gaussian-splat attribute codec

off	size	field	rule
0	4	magic WIS1
4	1	version	1
5	1	afrac	1..=30; equal to the model's (§7)
6	2	reserved	0
8	4	c	≥ 1: latent channels
12	4	sh	≥ 1: latent height
16	4	sw	≥ 1: latent width
20	4	s	≥ 1: the attribute planes' side
24	4	attr	≥ 1: attribute planes
28	4	n_gaussians	1..=s·s
32	attr·8	ranges	per plane, f32 lo then f32 hi (IEEE 754 binary32): finite, lo ≤ hi
…	c·2	table lengths	u16 each, 3..=4097
…	c·4	symbol offsets	i32 each
…	c·8	medians	i64 each, `
…	the lengths' sum ·4	one table per channel	§2
…	4 + L	the latent stream	§2

The header's field rules are checked in the order afrac, the dimensions, n_gaussians. The caps (§7) apply to c·sh·sw and to attr·s·s.

Decode, against the set of §9 (g_s.whm, the integer synthesis):

1. For each channel k and each of its sh·sw elements, decode a symbol s against channel k's table and offset; the element is s·2^afrac + median_k.
2. g_s on [c, sh, sw] gives [attr, s, s] (geometry_mismatch otherwise). g_s decodes through its lowering whm1/1 (integer-operators §13.2).
3. Each value v is clamped to [0, 2^afrac] and becomes (v·255 + 2^(afrac−1)) >> afrac, a byte: the RGB8 step that ends the lowered graph (integer-operators §13.0).

The output, plane after plane, is the buffer of the capability's splat-equivalence digest. The first n_gaussians cells of the planes, in raster order, hold one Gaussian each. The ranges are presentation data: a sink presents a plane's byte u at about lo + (hi − lo)·u/255. They are outside the digest, and this extension does not fix presentation.

7. Caps and the checks a decode makes

Caps, the same on every target. No tensor a payload sizes may exceed 2^26 elements: a frame h·w·3, a WIV1 motion-vector field 2·nb·(n_frames − 1), a latent, a hyper-latent, a network's output, an attribute tensor, a clip's t_out. A WIV1 clip may not exceed 2^30 bytes, nor 2^20 frames.

Planning the networks. Before a decode allocates a latent, it plans every network on the payload's shapes from the layers' geometry and parameter lengths alone. The network's input must be within the cap (too_large); then, layer by layer in order:

1. the layer fits the running channel count (model_mismatch): a convolution's output channels, kernel and stride are non-zero, it has c·out·k·k weights (c·out·k for the audio synthesis) and out multipliers and biases, and its requantisation shift is under 128; an inverse GDN has c·c and c parameters and shifts under 128; a leaky ReLU's shift is under 64;
2. every size the layer forms is within 2^26 (too_large): a convolution's padded input side n + 2·pad, a transposed convolution's uncropped side (n − 1)·stride + k + opad and uncropped plane, and the output;
3. the layer leaves an output (model_mismatch): a kernel no larger than the padded input, a crop 2·pad smaller than the uncropped side.

Every size is formed in 64-bit arithmetic with overflow checked, and a size past 64 bits is past the cap, so a 32-bit sink refuses exactly what a 64-bit one does, with the same code; once a network is planned, every size it forms fits a 32-bit index.

Memory and work. A payload's header declares its decode's output and the work that output takes, and a sink can read both before it decodes anything. A stream does not bound them: a table can make one symbol cost a small fraction of a bit, so a few kilobytes of stream can code a clip at the 2^30-byte cap, as any codec can describe a large output in a small file. The bound is the header's, within the caps:

• the output: a WIV1 clip's n_frames·h·w·3 bytes (one per sample), a WIH1 image's 3·h·w bytes at its synthesis network's size, a WIA1 clip's 2·t_out bytes (i16 samples), a WIS1 attribute tensor's attr·s·s bytes;
• the entropy decode: one step per symbol, n_frames·h·w·3 for a keyframe-0 WIV1 (h·w·3 fewer, plus its keyframe's, for keyframe 1), n_z·zh·zw + m·sh·sw for a WIH1, n_ch·lat_len for a WIA1, c·sh·sw for a WIS1;
• the buffers: latents and network activations of 8 bytes per element (i64); while a network layer runs, its input, its output and, for a transposed convolution, the uncropped plane it accumulates into, each at most 2^26 elements (Planning the networks); a WIV1 decode's frames (one byte per sample) and its motion vectors (4 bytes each);
• each network's work, its multiply-accumulates layer by layer: a convolution's every output times c·k·k (every tap of its padded input), a transposed convolution's every input times out·k·k (out·k for the audio synthesis), an inverse GDN's every element times c; a WIV1 with keyframe 0 runs no network. A kernel may skip a zero term, so it performs at most this many.

A WIA1's latent is the one its t_out needs and no longer: the synthesis of lat_len samples gives at least t_out samples and that of lat_len − 1 fewer (geometry_mismatch), so a payload cannot declare a large latent to present a few samples.

A decode refuses a stream that runs out at its first missing symbol (§2), so a short stream stops early, but a stream that carries its symbols is decoded in full: a sink that will not spend what a header declares refuses before decoding, by its own size policy, applied to the declared cost (int_payload::declared_cost and int_payload::check_cost, with the set; neural::declared_cost over the set a registry resolved; the C ABI's wai_int_payload_declared, given the set's path; wai-web's intPayloadCost; the header alone through container::declared_size), cost_over_limit (§8). The reference neural sink applies one when it is given one (neural::ModelRegistry::with_cost_limit); its C ABI does not decode integer video, PCM or splat attributes, which it has no output kind for. This extension fixes the caps; a sink's policy may be stricter.

Informative. A sink may keep a parameter set lowered and loaded between decodes, keyed by the SHA-256 of its bytes (integer-operators §8.6), so that only the first decode through it pays for the lowering and the bounds pass. What it keeps is the set's, the same for every payload, and is not part of any decode's declared working bytes; the sink's own budget for kept sets bounds it. The reference implementation's store (int_set_cache::LoadedSets) starts with a budget of 256 MiB, evicts the least recently used set first, keeps no set larger than the whole budget, and keeps nothing with a budget of 0 (LoadedSets::set_budget; wai-web's setLoadedSetBudget). A decode that finds its set kept makes every other check of this section, in the order below, and gives the same output and the same refusals.

The parameter set must fit the payload (model_mismatch):

• every network parses (the WHM1 and WAM1 forms, §9), and the networks of one set share one fixed-point scale afrac in 1..=30;
• a WIA1 or WIS1 payload's afrac is the model's;
• the entropy tables of entropy.json are consistent (a table, length and offset per scale edge, at least one edge; a table, length, offset and median per hyper-latent channel) and every table is a CDF of §2 over its length; a payload has no more hyper-latent channels than the tables;
• each layer's parameters have the lengths the running channel count needs, and its geometry leaves an output (Planning the networks);
• the audio synthesis gives one channel; the conditional codec's g_s gives three;
• a network that decodes through its lowering lowers, and its graph loads under acc64 (integer-operators §13.2 and §13.3: for WIH1 and a WIV1's keyframe, h_s and g_s; for WIS1, g_s; for WIA1, the synthesis).

The shapes the payload declares must be the ones the decode gives (geometry_mismatch): h_s's output and [2·m, sh, sw]; a WIV1 keyframe and h × w; the audio synthesis's length and t_out (it may be longer, by less than one latent sample's worth: the synthesis of lat_len − 1 samples gives fewer than t_out); g_s's output and [attr, s, s].

The order of a decode's checks. After the parse (§2), a decode checks in this order and refuses with the first that fails:

1. the set's members, in the order of §9's list, each present (set_incomplete) and parsed (model_mismatch);
2. the parameter set against the payload (model_mismatch): one fixed-point scale, the payload's afrac, the entropy tables, the hyper-latent channels;
3. every network planned, in decode order (h_s then g_s; the audio synthesis; the splat g_s), each layer by the three rules above (model_mismatch, too_large): the caps come before any shape is compared;
4. what the networks give against the payload: for WIH1, h_s's shape (geometry_mismatch), then g_s's three channels (model_mismatch), then h_s and g_s lowered and loaded, in that order (model_mismatch); for WIV1 with keyframe 1, its keyframe's shape and channels as for WIH1, then the keyframe's size against h × w (geometry_mismatch), then the keyframe's networks lowered and loaded (model_mismatch); for WIA1, one channel (model_mismatch), then at least t_out samples, then a latent no longer than that needs (both geometry_mismatch), then the synthesis lowered and loaded (model_mismatch); for WIS1, g_s's shape (geometry_mismatch), then g_s lowered and loaded (model_mismatch);
5. each stream in decode order (WIH1: the hyper-latent's, then, after h_s runs, the latent's; WIV1 with keyframe 1: the keyframe's streams, then the residual stream), symbol by symbol: a symbol that runs its stream out (stream_exhausted) before its value's range (field_range); after its last symbol, its canonical end (stream_noncanonical);
6. each network as it runs, layer by layer (field_range, below).

So a run-out is reported before the value it would have given, and a size past the caps before a shape that disagrees. A sink that applies a cost limit (Memory and work) compares the declared cost with it after step 4, once everything the cost reads has been checked, and before any symbol is decoded (cost_over_limit, §8).

Exactness. A latent rebuilt as s·2^afrac + base must fit i64; before each network layer runs, the decode checks on the values present that no product, partial sum, requantised value or inverse-GDN term can leave the type its arithmetic is formed in. A stream whose values fail either check is refused (field_range). So a decode with overflow checks and one without compute the same integers, and neither panics. A network that decodes through its lowering (integer-operators §13; for WIH1, a WIV1's keyframe and WIS1, §13.2; for WIA1, §13.3) makes the second check as the check ranges of its graph: a value outside one is refused (field_range), and the refusals are the same. So does WIA1's PCM step (§5 step 3), whose range is its graph's last check range.

8. Refusal codes

The list is closed: a sink reports exactly one of these, the first that applies in the order of §2 and §7.

A refusal of the payload or of the parameter set against it, bad_magic through model_mismatch in the table below (and payload_malformed), is final: the sink does not decode the payload by another path, and SPEC §4's fallback is not tried, since it would decode the same bytes. The other codes say the sink cannot decode through the capability at all, and route as SPEC §4 routes such a capability:

• set_incomplete is a pin that did not resolve (SPEC §3.1): a WAI1 continues at SPEC §4 step 4 (or reports it at step 6), and a WAI2 moves on to its next entry (SPEC §7.1);
• not_dispatchable and unsupported_capability name a capability the sink does not decode: SPEC §4 does not take step 3 (or 5) with it, so a WAI1 continues at step 4 and a WAI2 at its next entry, as for a capability the sink does not advertise;
• cost_over_limit is the sink's own policy, not a fault of the payload: a payload whose declared cost exceeds the limit the sink applies (§7, Memory and work) is refused before any symbol is decoded, and routes as unsupported_capability does, so a WAI1 continues at step 4 and a WAI2 at its next entry.

code	when
bad_magic	the payload does not start with any of the four magics: a tool's JSON form lands here
format_capability_mismatch	the payload is another of the four formats than the capability's
keyframe_capability_mismatch	a WIV1 keyframe kind that is not the capability's
version	a version this revision does not define
reserved_nonzero	a reserved field that is not zero
field_range	a field of any section outside its range (a header field, a WIS1 range or table length, a median); or a value the stream decodes that leaves the exact integer range (§7)
too_large	a size past the caps of §7
cdf_malformed	a table that breaks the rules of §2
stream_length	the payload ends before a field or section it declares; or a stream length that is not a multiple of 4, or is under 8
nested_length	a WIV1 keyframe length K that is not the rest of the payload
trailing	bytes after the last section
stream_exhausted	a stream that ran out before its last symbol
stream_noncanonical	a stream that does not end canonically after its last symbol: a word left unread, or a state other than 2^31 (§2)
geometry_mismatch	a shape the payload declares that the decode does not give (§7)
model_mismatch	a parameter set that does not fit the payload (§7)
set_incomplete	a parameter set without a member the decode reads (§9); a pinned decode reports SPEC §3.1's set_incomplete
not_dispatchable	a capability SPEC §4 does not dispatch
unsupported_capability	a capability the sink has no integer decoder for
payload_malformed	a wai.neural.int_synth or wai.neural.int_mlicpp payload its decoder refuses
cost_over_limit	a payload whose declared cost (§7) exceeds the limit the sink applies before decoding

9. Parameter sets

A capability that decodes against a parameter set reads these members, by the names a set's listing gives them (prior-carriage §7):

capability	shape	members
wai.neural.int_hyper, wai.video.int_hyper	several files	g_s.whm and h_s.whm (networks in WHM1), entropy.json (the tables)
wai.audio.int_codec	one file	model.wam (the synthesis in WAM1)
wai.splat.int_codec	one file	g_s.whm (the synthesis in WHM1)
wai.video.int_motion	none	—

A set of several files is pinned through its listing (wai.prior.set); a one-file set MAY be pinned form-less, by its file's SHA-256 (SPEC §3.1). wai.audio.int_codec and wai.splat.int_codec were registered with sets of several files, because no one-file format had been named for them; this revision registers each as one file, the file its decoder reads, which holds every parameter its output depends on. That widens their registrations, as SPEC §8 allows a MINOR revision to; a set pinned through its listing still verifies.

WHM1 (int_hyper_synth::IntModel::from_bytes): magic, then afrac, gf, shift and the layer count (u32 each), then per layer a tag byte: 0 a transposed convolution and 1 a convolution (input and output channels, kernel, stride, padding and, for 0, output padding as u32; weights as i16; per-output-channel multipliers and biases as i64), 2 an inverse GDN (channels as u32; gamma as i32, beta as i64), 3 a ReLU, 4 a leaky ReLU (slope i64, its shift u32). WAM1 (int_audio::AudioModel::from_bytes): the same header, then layers of tag 0, a one-dimensional transposed convolution, and tag 2, an inverse GDN. In both forms nothing follows the last layer: a file with bytes after it is refused (model_mismatch), so one network has one file and one digest. The layers' integer arithmetic is integer-operators §13.2 (WHM1) and §13.3 (WAM1).

entropy.json, the tables of wai.neural.int_hyper and wai.video.int_hyper: a JSON object with these members. Other members are ignored. An integer is a JSON number with no fraction and no exponent.

member	value
scale_edges	N ≥ 1 integers, each fitting i64: the edges e_0 … e_{N−1} of the ceil bucket rule (integer-operators E3)
cdfs	N arrays of integers, each fitting i32: the latent tables, one per bucket
cdf_lengths	N non-negative integers: table b's length n_b
offsets	N integers, each fitting i32: table b's offset
z_cdfs	K ≥ 1 arrays of integers, each fitting i32: the hyper-latent tables, one per channel
z_cdf_lengths	K non-negative integers
z_offsets	K integers, each fitting i32
z_median_q	K integers, each fitting i64: channel k's median at 2^afrac

Each table, over its first n_b entries, is a table of §2: 3 ≤ n_b ≤ 4097, n_b no more than its array's length, cdf[0] = 0, cdf[n_b − 1] = 65536, strictly increasing. Entries after n_b are not read. A set whose tables break any of this is refused (model_mismatch). A payload has no more hyper-latent channels than K (§7). Hyper-latent channel k decodes against z_cdfs[k], z_cdf_lengths[k] and z_offsets[k], and its elements are s·2^afrac + z_median_q[k] (§4 step 1). A latent element decodes against table b, the ceil bucket of its σ over scale_edges (§4 step 3).

10. The tools' JSON forms

The encode and export tools under tools/ write a JSON bitstream. It is a converter input and a debugging aid, never a payload. An encoder MUST NOT carry it as the payload of any capability of §1, and a sink refuses it (bad_magic). The converters (int_payload::{wiv1,wih1,wia1,wis1}_from_json, and the independent int-payload-conformance/int_payload_verify.py) make explicit what the JSON leaves implicit:

• WIV1: an absent mv_precision is 1, and only 1 and 2 are converted; a one-frame clip is written with mv_precision 1 and block 1, whatever the JSON states; mvs has one list per frame, and the first is empty; a frame's list shorter than its block grid is padded with (0, 0), the vector the JSON decoder read for a missing block; a list longer than the grid, or a vector outside i16, is refused; rmax is not carried. The keyframe of wai.video.int_hyper is the WIH1 converted from its own JSON.
• WIH1: the fields of cond_bitstream.json, the streams as they are.
• WIA1: sr is the export's sample_rate (in audio.json, which the bitstream does not carry); lo is the symbol offset; every channel's table must have the first table's length; afrac is carried and must be the model's.
• WIS1: the ranges lo and hi must each be a binary32 value, carried exactly (a JSON integer included: 2^60 + 1 is refused, not rounded to 2^60); afrac is carried and must be the model's.

A payload and the JSON it is converted from decode to the same bytes; the conformance vectors check that for every committed JSON bitstream.

11. Envelopes this revision makes non-conforming

This revision completes these strings' registrations with a payload format they did not have (SPEC §8, Completing a registration). It makes non-conforming, and they still parse:

• a WAI1 envelope whose capability or fallback is a string of §1 and whose payload is not that string's format: a tool's JSON form among them;
• a WAI2 envelope with a component of a string of §1 whose bytes are not that string's format.

A sink refuses such a payload when it decodes it (§8), with no fallback attempt. An encoder MUST NOT write one; the reference implementation's emit paths (wai wrap, the C packers and the browser writers) parse the payload as a sink does and refuse to. An encrypted envelope's payload (SPEC §6) is ciphertext until a sink decrypts it, so the emit paths do not check it; the sink checks the plaintext when it decodes it (SPEC §6.4). No committed envelope or conformance corpus carried such a payload when this revision was made.

12. Reference implementation and conformance

• int_payload::wire parses and writes the four formats, with no dependency on the rest of the crate; int_payload::decode_int decodes every integer capability from parameter files held in memory, with no ONNX runtime and no filesystem, for native and wasm32 sinks; int_payload::resolve verifies a pin over sets held in memory (pin::verify_bytes_with) and runs SPEC §3.1's check 5, for a pinned decode and over the files an unpinned decode reads (a listing the set carries must match them, and the digests include the listing of the files read); it is the only way to build the parameters a decode reads (IntParams). The neural sink (neural::decode_payload) delegates every integer capability to decode_int, and its registry reads an unpinned integer set once and runs check 5 over those bytes.
• codecs::PINNED_INT_PAYLOADS is the one table of strings and magics; container::check_int_payload_framing parses a payload with int_payload::wire and the capability's keyframe kind, which needs no integer decoder and no feature, so every emit path runs it: Wai::to_emit_bytes and WaiMulti::to_emit_bytes (and so the C packers), wai-web's pack and packMulti, and wai wrap; wai inspect reports it.
• int_video::decode_typed and decode_pframes_typed decode WIV1 clips through wai-ops' stage operators (int_video::ops); the direct decode they replaced is test-only (legacy_int), the reference the differential holds them to.
• video_receipt::VideoReceipt::seal_decode_wiv1 seals a wai.video.int_motion receipt whose identity is BLAKE3 over the WIV1 payload.
• wai-web decodes the four formats in the browser (decodeWiv1, decodeWih1, decodeWia1, decodeWis1).
• Conformance: the payloads beside their JSON sources in wai-rs/tests/vectors/, and the refusal fixtures and index in int-payload-conformance/, written by wai_int_payload_vectors gen and checked by wai_int_payload_vectors verify; the same corpus checked by int_payload_verify.py, which shares no code with wai-rs and decodes the WIV1 keyframe-0 payloads in full; and the cross-architecture goldens of byte-exact-conformance, which parse and decode the WIV1, WIH1, WIA1 and WIS1 payloads on every platform of its matrix.

13. The wai.neural.int_synth payload and set

SPEC §5 keeps this capability's payload format; this section writes it down.

off	size	field	rule
0	4	H	≥ 1: the image's height
4	4	W	≥ 1: the image's width
8	2	C	≥ 1: latent channels
10	1	S_log2	the latent's side is S = 2^S_log2
11	1	reserved	0
12	C·S·S	the latents	one signed 8-bit value each, [C, S, S] in CHW order

The format has no magic and no version. The payload is exactly 12 + C·S·S bytes, and C·S·S ≤ 2^26.

The set is one file, int_synth.json: a JSON object whose member layers is a non-empty array of layer objects. A layer has in_ch, out_ch, k, stride and pad, each a non-negative integer below 2^32, with out_ch, k and stride at least 1; out_pad, a non-negative integer below 2^32, 0 when absent; weight, in_ch·out_ch·k·k integers each fitting i32, in the layout [in_ch][out_ch][k][k]; bias, out_ch integers each fitting i32; shift, in 0..=127, 0 when absent; and relu, a boolean, false when absent. Other members are ignored.

The decode is integer-operators §13.1's graph on the latents. Its output must be [3, H, W], and it is presented as row-major HWC bytes. Every refusal of the payload, the set or the decode is payload_malformed (§8).

The reference decoder (int_synth::decode_bytes) decodes through the lowering. It checks the payload first: its header, a reserved byte of 0, C ≥ 1, C·S·S ≤ 2^26 and its length. It then lowers the set and loads the graph under acc64, which makes the set's checks (integer-operators §13.1: at least one layer, the layers chain, the lengths, fields below 2^32, out_ch, k and stride at least 1, a shift of at most 127, the caps and the certificate), and runs it. A shift from 64 to 127 has its exact meaning. Several of these checks are new refusals of input no committed corpus contains (integer-operators §15.3, SPEC §8 Planned breaks).

14. WIM1 and WMB1 — wai.neural.int_mlicpp

SPEC §5 keeps this capability's payload format and its one-file parameter set, the WMB1 bundle (read from a set as model.wmb); this section writes both down.

off	size	field	rule
0	4	magic WIM1
4	4	H	≥ 1
8	4	W	≥ 1
12	2	Cz	≥ 1: hyper-latent channels
14	2	Zh	≥ 1
16	2	Zw	≥ 1
18	2	Cy	≥ 1: latent channels, a multiple of K
20	2	Yh	≥ 1
22	2	Yw	≥ 1
24	2	K	≥ 1: channel slices
26	2	reserved	0
28	4	Lz	the z stream's length in bytes
32	4	Ly	the y stream's length in bytes
36	Lz	the z stream	§2, with no length prefix
36 + Lz	Ly	the y stream	§2, with no length prefix

A parser checks, in order: at least 36 bytes; the magic; the reserved field; no dimension 0; Cy a multiple of K; Cz·Zh·Zw, 3·Cy·Yh·Yw and 3·H·W each at most 2^26; and the payload exactly 36 + Lz + Ly bytes. Each stream's length is checked (a multiple of 4, at least 8) when the decode reaches it.

The set is one file, a WMB1 bundle: the magic WMB1, then four sections, each a u32 length followed by that many bytes, in the order h_s, entropy, g_s (each a WHM1 network, §9) and the tables. The tables are a JSON object with the members of entropy.json (§9), and afrac. Nothing follows the fourth section. The bundle's rules: the three networks' afrac are equal, lie in 1..=30, and equal the tables' afrac; 1 ≤ N ≤ 65536; and every table, over its length, follows integer-operators E1's t2 rule (2 ≤ n_b, n_b no more than its array's length, cdf[0] = 0, cdf[n_b − 1] = 65536, strictly increasing), in place of §2's rule.

The decode:

1. Plan, before any stream: Cz is at most the number of z tables; h_s maps [Cz, Zh, Zw] to [2·Cy, Yh, Yw]; entropy maps [3·Cy, Yh, Yw] to [2·Cy, Yh, Yw]; g_s maps [Cy, Yh, Yw] to [3, H, W]; and no tensor any of them forms exceeds 2^26 elements.
2. z. For each channel k, and each of its Zh·Zw elements in raster order, decode a symbol s against z table k, and the element is dequantize(s, afrac, z_median_q[k]) (integer-operators E2). After the last z symbol, the stream must not have run out.
3. hyper = h_s(z).
4. y. Let y be [Cy, Yh, Yw], all 0. For each slice j = 0 … K − 1, and for each pass (the anchors, (row + col) even, then the rest): σ‖μ = entropy(concat(y, hyper)), where every element of y not yet decoded is still 0. Then, for each channel of the slice (j·Cy/K to (j + 1)·Cy/K − 1), and each of its positions in raster order that belongs to the pass: b is the ceil bucket of σ over scale_edges (integer-operators E3); decode r against table b; the element is dequantize(r, afrac, μ). One y stream carries every slice's two passes, in that order. After the last y symbol, the stream must not have run out.
5. g_s(y), then the RGB8 output step (integer-operators §13.0), as row-major HWC bytes.

The streams decode with t2, lenient and final_only (integer-operators E1). A refusal of the payload, of either stream or of a value the decode forms is payload_malformed; a bundle that breaks its rules, or that does not fit the payload (step 1, and the load checks below), is model_mismatch (§8).

The reference decoder (int_mlicpp::decode) runs the three networks through their lowering, wmb1/1 (integer-operators §13.4: each network by whm1/1, g_s with the RGB8 step), each graph loaded under acc64. Step 1 checks the header and plans the networks from their layer geometry, then lowers and loads them, which makes the set's checks of integer-operators §13.2 (a layer of tag 0, 1 or 2 in each network; each layer fits the channels the layer before it gives; header shifts under 128; a leaky shift under 64; no bias of magnitude 2^63 − 1 or more; the certificate), then plans each graph on the payload's shapes (integer-operators §4: its input's channel count, Cz, 3·Cy or Cy, and the caps, a convolution's padded side included), all before either stream. A value outside a graph's check range is refused (payload_malformed): the ranges refuse exactly the values a layer's arithmetic could leave i64 on. Some of these checks are new refusals of input no committed corpus contains, and some now come before the streams where an earlier decoder made them when the network ran (integer-operators §15.3, SPEC §8 Planned breaks).

15. Receipts of wai.video.int_motion

Two signed receipts describe a WIV1 payload of wai.video.int_motion (keyframe 0, §3). The decode receipt (§15.1) is a sink's statement of what the payload reconstructs to; the encode receipt (§15.3) is an encoder's statement of the payload it wrote, the source it wrote it from, its parameters and its error. Each binds a measured energy figure and its acquisition class (energy-measurement §2) beside the figures anyone can recompute, and keeps the two apart. The reference is wai-rs's video_receipt (VideoReceipt, VideoEncodeReceipt; feature video_receipt).

In this section, integers in a signing payload are big-endian and those hashed into a frame or clip commitment little-endian, as the reference writes them; BLAKE3 is the hash of every commitment.

15.1 The decode receipt

field	what it is
codestream_hash	BLAKE3 over the payload's bytes
h, w	the payload's geometry
frames	for each decoded frame i, from 0: BLAKE3 over wai:video-rframe (ASCII), 0x01, i (u64) and the frame's RGB8 bytes
clip_hash	BLAKE3 over wai:video-clip, 0x01, h (u32), w (u32), the number of frames (u64), then each frame's i (u64) and hash
merkle_root	the jwp-receipts §2.2 Merkle root over the frame hashes, in order
work_pixel_ops	frames × h × w × 3, saturating at 2⁶⁴ − 1: the decoded subpixels
joules_micro, the class	the energy the sink spent reconstructing, in microjoules, and its acquisition class; 0 and no class is the unmetered marker
parent_receipt_hash	the receipt hash of the receipt before it, or none
signer_pubkey, signer_id, sig	the signer's Ed25519 key, its id, and the signature

The signing payload is wai:video-receipt and 0x01, then codestream_hash, clip_hash, h (u32), w (u32), merkle_root, the number of frames (u64), work_pixel_ops (u64), joules_micro (u64), the class, the parent (0x00, or 0x01 and the hash), each frame's i (u64) and hash, signer_pubkey, and signer_id's length (u32) and UTF-8 bytes. The class is one tag byte and its evidence: 0 the unmetered marker; 1 HwShunt (legacy), 2 ModelBased, 3 Estimator, with none; 4 OnChipCounter, then relative_ppm (u32), window_us (u64) and the filtering state (u8: 0 Unknown, 1 Off, 2 On); 5 CalibratedInstrument, then relative_ppm (u32), window_us (u64), and the certificate reference's length (u32) and UTF-8 bytes. sig is Ed25519 over the payload, verified strictly (SPEC §7 Signatures). The receipt hash is BLAKE3 over wai:video-receipt-id, 0x01, the payload and sig.

A receipt is consistent when merkle_root, clip_hash and work_pixel_ops are what its frames give, its frames are numbered 0 upwards, its figure and class cohere (0 with no class, or a figure above 0 with a class whose evidence energy-measurement §2.3 requires and whose declared relative_ppm × joules_micro reaches the resolution floor of 288676), and its signature verifies. It is re-derived from a payload when the payload's BLAKE3 is its codestream_hash, the payload decodes (§3), and the decode has its geometry and, frame by frame, its frame hashes; and then it is consistent. The figure is attested by the signature and by nothing else: another sink measures another figure for the same decode.

15.2 The JSON form

A decode receipt's JSON form is one object of these members, in this order: profile ("wai.video.int_motion/receipt"), codestream_hash, clip_hash, h, w, frames (each frame's hash), merkle_root, work_pixel_ops, joules_micro, uj_per_kilopixel_op, energy_provenance, the class's evidence (for OnChipCounter, energy_uncertainty {"relative_ppm", "window_us"} and energy_counter_filtering; for CalibratedInstrument, energy_uncertainty and energy_calibration_ref; for the others, none), parent_receipt_hash, signer_pubkey, signer_id, sig and receipt_hash. Hashes, keys and signatures are lowercase hexadecimal; h, w and the figures are integers in plain decimal; energy_provenance is the class's label, or null when unmetered, and parent_receipt_hash null when there is none. uj_per_kilopixel_op is derived: null when the receipt is unmetered or its work is 0, and otherwise joules_micro × 1000 / work_pixel_ops, computed in IEEE 754 binary64 (the product, then the quotient) and written with six digits after the point, correctly rounded.

A reader reads the text strictly (RFC 8259, no member named twice), requires exactly the members the receipt's class writes, each of its form, reads the class and its evidence as energy-measurement §2.7 reads them, and refuses a uj_per_kilopixel_op that is not, spelled as above, the receipt's own, and a receipt_hash that is not its receipt hash. The members' order and the whitespace between them are not held. A receipt read so is then consistent or not, and re-derived from its payload or not, as §15.1 says.

15.3 The encode receipt

An encode receipt (wai.video.int_motion/encode-receipt) states the payload an encoder wrote and the source it wrote it from:

field	what it is
codestream_hash	BLAKE3 over the payload's bytes
h, w	the source's geometry
source_frames	for each source frame i, from 0: BLAKE3 over wai:video-sframe (ASCII), 0x01, i (u64) and the frame's RGB8 bytes
source_hash	BLAKE3 over wai:video-source, 0x01, h (u32), w (u32), the number of frames (u64), then each frame's i (u64) and hash
block, search, mv_precision, q	the encoder's parameters: the payload's block, mv_precision and q (§3), and the motion search's radius, which the payload does not carry
sse	Σ (source − decode)² over every sample of every frame (u64): the error the payload decodes with
work_pixel_ops	frames × h × w × 3, saturating at 2⁶⁴ − 1: the encoded subpixels
joules_micro, the class	the energy the encoder spent, in microjoules, and its acquisition class; 0 and no class is the unmetered marker
parent_receipt_hash, signer_pubkey, signer_id, sig	as §15.1's

Its signing payload is wai:video-encode-receipt and 0x01, then codestream_hash, source_hash, h (u32), w (u32), the number of frames (u64), block (u32), search (u32), mv_precision (u8), q (u32), sse (u64), work_pixel_ops (u64), joules_micro (u64), the class (§15.1's encoding), the parent, each source frame's i (u64) and hash, signer_pubkey, and signer_id's length (u32) and bytes. Its receipt hash is BLAKE3 over wai:video-encode-receipt-id, 0x01, the payload and sig. It is a profile defined after the hardware class was refined (energy-measurement §2.6, New profiles): its class is never the legacy HwShunt.

An encode receipt is consistent as §15.1 says of a decode receipt, with source_hash for clip_hash and no Merkle root, and the new-profile rule for its class. It is re-derived from a payload and a source when the payload's BLAKE3 is its codestream_hash; the payload reads as WIV1 keyframe 0 (§3) of its h, w and number of frames, with its block, mv_precision and q; each source frame hashes to its own; the payload decodes; and the decode against the source gives its sse; and then it is consistent. Re-deriving it proves the payload decodes to frames within sse of the stated source. It does not prove which encoder wrote it, or that search was the radius used.

Its JSON form is one object of profile ("wai.video.int_motion/encode-receipt"), codestream_hash, source_hash, h, w, source_frames, block, search, mv_precision, q, sse, work_pixel_ops, joules_micro, energy_provenance, the class's evidence (as §15.2), parent_receipt_hash, signer_pubkey, signer_id, sig and receipt_hash, read as §15.2 reads a decode receipt's.