open-standards/wai/extensions/moq-streaming-format.md

WAI Extension: MoQ Streaming Format

Status: Draft, and deliberately tracking rather than pinning — MOQT is pre-Working-Group-Last-Call and its text still moves. Declares WAI as a streaming format over Media over QUIC Transport, so a WAI envelope is a first-class MOQT object rather than an opaque payload inside someone else's container. No new transport: WAI defines none by charter, and this extension does not change that. Field semantics and rules over them only — never MOQT's byte encoding. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Drafts read for this revision: draft-ietf-moq-transport-22 (MOQT), draft-ietf-moq-msf-01 (MSF) and draft-ietf-moq-cmsf-01 (CMSF), all Internet-Drafts of the IETF MoQ working group, none past Working Group Last Call. A section number of MOQT, MSF or CMSF in this text is one of those revisions; §6 says what happens when they move.

1. Why a streaming format

MoQ separates two things that other stacks fuse:

• MOQT (draft-ietf-moq-transport) moves objects, in groups, on tracks, and orders them by priority. It does not know what an object means.
• MSF (draft-ietf-moq-msf) says what objects mean: it fragments a stream into objects, maps each object to a stream of its own, and declares what a publisher emits in a catalog.

WAI already references MoQ for confidentiality (MLS + SFrame; see confidentiality). That binding treats MoQ as a pipe. This one treats it as what it is — a two-layer design with an open slot at the upper layer, which is precisely where a container that carries capability-dispatched objects belongs.

The alignment is structural, not aspirational. MSF fragments a stream into objects in groups; WAI envelopes are already objects, and jwp-receipts already commits per object and signs per group. The two layers agree on the unit without either being changed.

2. Precedent: the slot is open, and cheaply

MSF's catalog names a format in its packaging field, over a value set MSF's own text fixes (MSF §5.2.4: loc, mediatimeline, eventtimeline, moqlog, moqmetrics; the catalog itself is the track named catalog, MSF §5). There is no central streaming-format registry. Read alone, that is closed.

draft-ietf-moq-cmsf shows it is not. CMSF is a sibling Internet-Draft that declares packaging value cmaf, stating that it extends the allowed packaging values defined in MSF; inherits MSF wholesale — all of the specifications, requirements and terminology defined in MSF apply to implementations of this extension; adds two track-level catalog fields; and registers nothing: this document has no IANA actions. LOCMAF is a second instance of the same move.

So the path is a sibling draft with no registry action and no permission sought. This extension is the WAI-side statement of what such a draft would say.

3. The packaging value

A track carrying WAI envelopes MUST declare packaging value "wai".

• Every object on such a track MUST be exactly one WAI envelope (WAI1 magic + manifest + payload), whole. An object carrying a fragment of an envelope, or more than one, is not this packaging.
• The object boundary MUST be the envelope boundary. This is what makes the receipt's per-object commitment (jwp-receipts §2.1) and MOQT's object identity the same unit; if they diverge, every guarantee that composes them is void.
• A sink that does not recognise "wai" MUST NOT attempt to interpret the objects. MSF's tolerant-parser rule covers unknown fields; it does not license guessing at unknown packaging.

4. Catalog declaration

A WAI track's catalog entry MUST declare, in addition to MSF's required fields, the capabilities a sink needs to dispatch what the track carries:

field	type	meaning
wai_capabilities	array of string	Capability strings (WAI SPEC.md §4) that objects on this track may dispatch to.
wai_determinism	string	The weakest tier any object on the track claims — see determinism-tiers. Absent means no tier is claimed.
wai_receipts	boolean	Whether objects carry receipts per §5.
wai_receipt_alias	integer	The track_alias the publisher signs in this track's receipts (jwp-receipts §2.4.1), from 0 to 2^53 − 1.
wai_receipts_track	string	The name, in this track's namespace, of the track that carries this track's receipts (§4.1).

• wai_receipt_alias names the track's receipt chain, which a verifier keeps per publisher key and track_alias (jwp-receipts §3 step 3). MOQT's own Track Alias is chosen for each session by the publisher of that session, a relay included (MOQT §3.1.3), so past the first hop it is not the number the receipts sign; a sink finds a track's chain by this member. It is an integer written as one (SPEC §2: not 8.0, not "8"), compared as a number; a catalog that carries any other value for it, on any track, carries no stage set with receipts (§4.1, check 6). Two tracks of one catalog MUST NOT carry the same value: each track that declares receipts has a chain of its own (§4.1). §4.1 requires it, and wai_receipts_track, on the tracks of a stage set that declare receipts; on another track both are OPTIONAL.
• wai_capabilities MUST be exhaustive for the track. A publisher that dispatches to a capability it did not declare has broken the catalog's purpose, which is letting a sink decide before subscribing whether it can play what it is about to receive.
• A sink MUST NOT subscribe on the assumption that an undeclared capability will not appear, and MUST treat an undeclared capability at dispatch as a protocol error rather than falling back silently.
• wai_determinism is the weakest tier on the track, not the strongest. Declaring the strongest would let a track advertise a guarantee most of its objects do not meet.

These are custom catalog fields under MSF's extension rule (MSF §5: custom field names "MUST NOT collide" with MSF's, and a parser ignores a field it does not understand); they are namespaced by their wai_ prefix and MUST NOT collide with MSF-defined names.

4.1 Staged tracks

A stage set (staged-delivery) is carried as one track per layer: a base track, which carries the set's bases, and a layer track for each layer index, which carries the layers of that index. Every track keeps §3: each object is one whole WAI1 envelope. A subscriber chooses its stage by the tracks it subscribes to; a relay forwards tracks and never reads an envelope.

The catalog. Each track of a set MUST carry, in its catalog entry,

"wai_staged": { "set": "cam1", "layer": 1, "lead_ms": 40 }

member	type	meaning
set	non-empty string	The set's name: the same on every track of the set.
layer	integer, 0–255	0 on a base track; k on a track that carries layer k.
lead_ms	integer from 0 to 2^53 − 1, or null	On a layer track of a live set, the lead of its layer (staged-delivery §8): the time before a unit's presentation by which the layer must arrive to be applied to it. null on a base track. A sink reads it only while the set is live: when isLive is false it ignores it, as MSF §5.2.8 has a sink ignore targetLatency.

A set's extent is the tracks present: its layers are 0 to the highest layer a track of the set states, each with at least one track. The catalog states no count of layers, so an MSF delta update (MSF §5.1.6, §5.3), which adds and removes tracks and changes none it declared before, changes a set's extent by adding or removing a layer track alone. More than one track MAY carry one layer, as renditions of it. Each rendition is a track of its own, with a receipt chain of its own: a clone (MSF §5.1.6) inherits every member of its parent but its name, so a clone of a track that declares receipts MUST override wai_receipt_alias and wai_receipts_track (Receipts, below).

A layer track's wai_capabilities lists the refinement capabilities its envelopes carry; a sink that does not implement staged delivery finds them unsupported (SPEC §4) and has no reason to subscribe. A member of wai_staged this revision does not define is ignored, an earlier revision's layers among them. A publisher puts on a track of layer k only envelopes whose refines.layer is k, and on a base track only envelopes with no refines; a sink applies an envelope by its own refines (staged-delivery §4.1, §13), whichever track it came on.

Names and types. A track is named by its namespace and its name (MSF §5.2.2, §5.2.3); one that declares no namespace inherits the catalog track's (MSF §5.2.2). Two tracks of a catalog MUST NOT share one namespace|name pair, and the tracks of a set MUST share one namespace. The check below needs the catalog track's namespace, which a sink knows, since it subscribed to the catalog track: a checker that does not know it compares a track that inherits only with tracks that inherit, and refuses a set that mixes declared and inherited namespaces (staged-malformed). On a track of a set, renderGroup, where present, MUST be an integer that fits 64 unsigned bits, written as one (SPEC §2), and is compared as such: true, 1.0 and "1" are not 1. depends, where present, MUST be an array of strings.

MSF fields. The tracks of a set MUST have the same isLive (MSF §5.2.7) and MUST share one renderGroup (MSF §5.2.11), which is present on each. On a live set, MSF already requires one targetLatency across a render group (MSF §5.2.8); it is the target latency staged-delivery §8 adds to a unit's timing, unless the sink applies its own. Every entry of a track's depends (MSF §5.2.14) MUST name a track of the catalog in the track's own namespace, where MSF resolves it. The depends of a track of layer k MUST name at least one track of the set at each layer below k, and MUST NOT name a track of the set at k or above; a base track's MUST NOT name a track of the set. An altGroup (MSF §5.2.12) MAY group the base tracks of alternative sets, and MUST NOT be set on a layer track: a subscriber picks one track of an alternate group, and a layer is never an alternative to its base or to another layer. A layer track MUST NOT carry temporalId or spatialId (MSF §5.2.16, §5.2.17), which number a codec's own layers; a layer's index is wai_staged.layer here, and refines.layer in the envelope.

Receipts. Every track of a set that declares wai_receipts true MUST carry wai_receipt_alias (§4) and wai_receipts_track: the name, in its namespace, of its receipts track, of packaging "wai-receipts". Each media track has a receipts track of its own, so each layer has a receipt chain of its own and a subscriber verifies the tracks it subscribes to without the receipts of those it does not (jwp-receipts §6.1); and a subscriber subscribes to the receipts tracks of the media tracks it subscribes to and no others, so the tracks it chooses cap the receipts it receives as they cap its stage. A track MUST NOT carry an alias another track of the catalog carries, or name a receipts track another track names: a rendition of a layer carries an alias and names a receipts track of its own.

• Group g of a receipts track carries one object: a JSON array of the group receipt and the object receipts of group g of its media track, in the JSON form of jwp-receipts §2.4.1 and §2.4.2. A publisher publishes it no later than that group's first object.
• A receipt is about one MOQT object, and MUST say which: the group receipt in group g of a receipts track has group_id g, the MOQT Group ID of the media track's group, and each object receipt in it has that group_id and, as object_id, the MOQT Object ID of its object. Its signer_pubkey is the publisher's key and its track_alias the media track's wai_receipt_alias. A subscriber refuses a receipt that breaks this (moqt_mismatch, jwp-receipts §3.3) before it verifies the receipt's signature, and holds nothing of it.
• A receipts track has no delivery timeout of either kind, and its publisher priority is its media track's. A subscriber sets no delivery timeout on its subscription to a receipts track (MOQT §9.20.3, §9.20.4); a receipt it loses anyway it recovers as below.
• Its objects are receipts, not envelopes, and §3 does not apply to them. The WAI_RECEIPT property (§5) names an object's group receipt but does not carry it.

Verifying, recovering and joining. A subscriber verifies each group receipt as jwp-receipts §3 does, on the chain of the publisher key and the track's wai_receipt_alias, and admits a group only after its parent (§3 step 3). Each object of a receipts track travels on a stream of its own (MSF §6), so receipts can arrive out of order, and a relay can drop one at its discretion (MOQT §5.2). It holds no receipt of the track's object until it verifies it, and until then applies no layer it carries (staged-delivery §10.1).

• A subscriber that subscribed to a receipts track from its first group, and that has received every object of it, holds the receipt of every object of its media track, delivered or not, and checks one it did not receive by inclusion (jwp-receipts §3, Verified by inclusion). Where a receipts-track object is missing, the receipts after it on that track wait for it, within the bound below.
• A receipt whose parent the subscriber lacks is held (parent_not_held). The subscriber recovers the parent by a FETCH (MOQT §3.2) of the receipts track's groups from its chain's head to the held group: group g of the receipts track holds the receipts of group g of its media track.
• A subscriber MUST state a bound on what it holds for missing parents (jwp-receipts §3 step 3, Holding), each count a finite number: a count of receipts held live for each chain, and a time from each one's arrival only beside it, never alone; a count of receipts held as history for each chain; and a count across all its chains. Past the live bound it refuses the layers whose receipts it holds, their receipts refused (parent_not_recovered) and the layers refused for want of them (staged-delivery §10.1), and re-anchors the chain at the next validly signed group of it that it receives (Re-anchoring). The layers of the groups the anchor skips are refused for want of their receipts, unless history later fills them.
• After a re-anchor, a FETCH of the gap (MOQT §3.2) can deliver its groups in any order, ascending among them: the verifier admits as history each group that links forward from the last group admitted before the gap or back from the anchor (jwp-receipts §3 step 3, History), never as a fork, and holds as history, under the history count, one that links to neither yet.
• A subscriber that joins a live track after its first group (MOQT §3.5) declares a join: its verifier anchors the chain at the first validly signed group of it that it receives, and holds the history before it as unverified (jwp-receipts §3 step 3, Joining). Joining at an earlier group (MOQT §3.5: a SUBSCRIBE whose FILL_PARAMETERS select it, its objects delivered on a fill fetch stream, §3.4; earlier revisions of MOQT called this a Joining FETCH) retrieves earlier receipts. The fill fetch stream delivers them in ascending order, so each waits for the one after it: the verifier holds them as history, under the history count and outside the live one, and admits them, back from the anchor, where their signed links lead to it.

A MOQT relay forwards a receipts track as it forwards any track, its objects opaque payload (MOQT §7.8): it neither reads nor verifies the receipts in it. It is therefore not a relay that conforms to jwp-receipts §4, which verifies a group's signature before fanning its objects out; the subscriber verifies every receipt itself (jwp-receipts §4.1, Receipts carried as payload).

Checking a catalog. A sink checks each set's entries in this order, and stops at the first that fails:

1. wai_staged is an object with the three members above, of their types and in their ranges; the track's packaging is "wai" and its isLive a boolean; its name is a string, its namespace absent or a string, and the pair is no other track's; its renderGroup, if present, an integer that fits 64 unsigned bits; its depends, if present, an array of strings; and, where the checker does not know the catalog track's namespace, either every track of the set declares a namespace or none does (staged-malformed). An entry whose set cannot be read belongs to no set, and is reported for the empty name.
2. The set has a track at each layer from 0 to the highest a track of it states (staged-set-incomplete).
3. Every entry of every track's depends names a track of the catalog in the track's namespace; a track of layer k names a track of the set at each layer below k, and none at k or above (staged-depends).
4. The tracks have one namespace, one isLive and one renderGroup, present on each, and no layer track carries altGroup, temporalId or spatialId (staged-grouping).
5. On a live set, lead_ms is an integer on every layer track and null on every base track (staged-lead). A set that is not live is not checked here.
6. Every wai_receipt_alias of the catalog, on any track, is an integer from 0 to 2^53 − 1; every track of the set that declares wai_receipts true carries an alias and names, in its namespace, a receipts track of packaging "wai-receipts"; no other track of the catalog carries its alias; and no other track names its receipts track (staged-receipts).

A sink treats the layer tracks of a set that fails as absent. Its base tracks are tracks of packaging wai like any other, and the sink MAY subscribe to them.

Groups and objects.

• Group and object ids of different tracks need not align. refines names what a layer refines, and a sink checks it (staged-delivery §4.1); nothing is read from where a group falls on another track.
• On a live track each group MUST carry exactly one object, which travels in a subgroup of its own (MOQT §2.2): MSF maps every object to a new stream (MSF §6), so none travels as a datagram. A group of one object is complete when its object is, so its group receipt is signed before the object is sent, and the latency the receipt adds to a layer's arrival (staged-delivery §8) is one signature and its carriage, not a group's duration.

Priority. MOQT schedules first by subscriber priority, the priority a subscriber gives its request (MOQT §5.1.2, §9.20.7), and only between objects of equal subscriber priority by publisher priority, the lower number first, 0 the highest (MOQT §5.1.1, §5.1.2). The base track's publisher priority MUST be a number strictly below every layer track's; the priority of a track of layer k MUST be no greater than that of any track of a layer above k; and a receipts track's is its media track's. These hold for each track's DEFAULT_PUBLISHER_PRIORITY (MOQT §10.4) and for every subgroup that sets its own (MOQT §5.1.1). A subscriber's own priorities can still put a layer before its base: one that wants the publisher's order SHOULD give its requests for a set's tracks one subscriber priority.

Delivery timeouts.

• A publisher SHOULD set a live layer object's OBJECT_DELIVERY_TIMEOUT (MOQT §5.2, §10.2) to no more than its slack at publication: the time from publication to the presentation of the last unit the object covers, less the layer's lead_ms, less the publisher's estimate of the path. MOQT reads a timeout set as an object property on the first object of a subgroup (MOQT §5.2, §10.2); here that object is its subgroup's only one, so the publisher MAY set each object's timeout on the object, from its own slack. A slack of zero or less cannot be set, since a timeout of 0 is no timeout (MOQT §5.2): such an object cannot arrive in time for any unit it covers, and a publisher MAY withhold it.
• A publisher MUST NOT set on a base track a delivery timeout of either kind shorter than the time from a base's publication to the presentation of its last unit.
• OBJECT_DELIVERY_TIMEOUT runs at each hop, from the time the object's last header byte reaches that hop, and is checked when the hop passes the object to its transport (MOQT §5.2). SUBGROUP_DELIVERY_TIMEOUT starts instead when the subgroup's FIN arrives, and runs until all its data is acknowledged (MOQT §5.2): on a subgroup of one object, it bounds that object's delivery after it was sent. Where both the publisher and the subscriber set a value, MOQT uses the smaller (MOQT §5.2, §9.20.3, §9.20.4), so a sink whose own target latency is longer than the one a publisher assumed cannot extend the publisher's timeout. Either timeout releases, hop by hop, bandwidth that a late layer would waste; neither is an end-to-end deadline: an object can pass every hop within its timeouts and still arrive late. The rule of staged-delivery §8 is the only rule that decides whether a layer is late.

Capping. A subscriber caps its stage by subscribing to the base tracks and to the tracks of layers 1 to n, and to their receipts tracks, and to no others. A relay MUST NOT read an envelope to cap layers: it treats every payload as opaque (MOQT §7.8), and forwards the tracks it is asked for. On a track of packaging wai the prototype header fields of staged-delivery §13 are not read.

Example. The entry of the first layer track of a live set, with its base track cam1/base, and the entry of its receipts track:

{ "namespace": "example/cam1", "name": "cam1/L1", "packaging": "wai",
  "isLive": true, "targetLatency": 300, "role": "video", "renderGroup": 1,
  "depends": ["cam1/base"],
  "wai_capabilities": ["wai.video.int_refine"], "wai_determinism": "decode-equivalence",
  "wai_receipts": true, "wai_receipt_alias": 8, "wai_receipts_track": "cam1/L1/receipts",
  "wai_staged": { "set": "cam1", "layer": 1, "lead_ms": 20 } }
{ "namespace": "example/cam1", "name": "cam1/L1/receipts", "packaging": "wai-receipts",
  "isLive": true }

5. The per-object property

MOQT carries metadata a relay can see as Properties of a track or of an object (MOQT §8.4), registered in a table MOQT creates, with ranges set aside for an application's own use. MSF §12 defines its properties this way; the one it defines, MSF_COMPRESSION, is at both scopes. CMSF adds track-level catalog fields and defines no per-object property. None of them carries a receipt.

A WAI receipt is already per-object. This extension therefore defines one object-scope property, WAI_RECEIPT, whose value is the object's receipt commitment:

field	type	meaning
content_hash	32 B	jwp-receipts §2.1 BLAKE3 over the envelope bytes.
joules_micro	u64	Microjoules producing this object; 0 is the unmetered marker, never 0 J (energy-measurement §2).
acquisition	string	Acquisition class per energy-measurement §2; empty when joules_micro is 0.
acquisition_evidence	bytes	For OnChipCounter and CalibratedInstrument, the evidence that follows the tag byte in energy-measurement.md §2.7 (declared uncertainty, then filtering state or calibration reference). Empty for every other class.
group_receipt	32 B	The link hash of the group receipt covering this object (jwp-receipts §3 step 3).

• A non-zero joules_micro MUST be accompanied by acquisition. A joule figure without its acquisition class is exactly what energy-measurement.md §2 forbids, and putting it on the wire does not make it acceptable.

• A zero joules_micro is the unmetered marker: acquisition MUST be empty and acquisition_evidence empty, since a class beside a zero figure claims a measurement that did not occur (energy-measurement.md §2). A sink MUST NOT present the marker as a figure of zero joules, and MUST discard a property that breaks this bullet or the one above (property_invalid).

• acquisition_evidence MUST be present and non-empty for a refined hardware class (OnChipCounter, CalibratedInstrument), and empty otherwise — including for legacy HwShunt, which carries no evidence. A refined class without its declared uncertainty is refused by energy-measurement.md §2 on the same terms as a figure without a class. A property written before this field existed has no acquisition_evidence; a reader treats its absence as empty, which is valid only for the classes that carry none.

• No signature covers this property. A sink MUST NOT report acquisition or acquisition_evidence as signed unless they and joules_micro equal a signed statement it holds of the same object, which is one of:

  • the label in the object's Merkle leaf, once the object receipt has been accepted under jwp-receipts §3 against the group receipt group_receipt names — group_receipt must then name that group; or
  • an energy claim about the object that it has accepted under jwp-receipts §3.1. A claim names no group, so group_receipt is outside what it states.

  Where such a statement gives another figure, class, evidence or group, the sink MUST discard the property (property_disagrees). Where it holds none, the class is outside every signature, and the sink reads the figure as unlabelled.

• A class read as signed is the statement of the key that signed it: the group receipt's signer_pubkey for a leaf, the claim's signer_pubkey for an energy claim. A sink MUST report that key with the class, and MUST NOT report a class a claim signs as the publisher's unless the claim's signer is a key it holds as the publisher's. Anyone can sign an energy claim, a relay that rewrites this property included.

• A relay MUST preserve this property unchanged, per jwp-receipts §4.1. A relay that recomputes it is asserting something it did not measure.

• A relay that does not forward an object, whether because a delivery timeout passed or at its discretion (MOQT §5.2), MUST still forward that object's receipts to every subscriber that requested receipts for its track (jwp-receipts §4.1). This property rides on the object and is lost with it; the receipts ride the track's receipts track (§4.1), which the relay forwards as it forwards any track, so a subscriber of that receipts track that receives the receipt accounts for the object as inclusion-verified (jwp-receipts §4.4). A receipts-track object the relay drops in turn is recovered as §4.1 says. A relay that signs a delivery-gap claim (jwp-receipts §2.4.11) does not add it to the publisher's receipts track, whose objects it forwards unaltered (MOQT §7.8); this extension defines no carriage for such a claim.

• Telemetry about objects on a WAI track is delivery-telemetry's. It is unsigned and advisory, and it MUST NOT restate this property's figure, class or evidence.

• Absence of the property is not evidence of absence of a receipt; receipts MAY ride elsewhere (§1 of jwp-receipts lists sidecar and trailer carriage). A sink MUST NOT infer that an object without the property is unattested.

6. What this extension does not do

Stated so it is not re-argued:

• It does not define, profile, or restate MOQT's wire encoding — not the object header, not the byte layout of properties, not the control messages. Those are MOQT's, and the property that makes this binding cheap is that WAI rides a transport instead of defining one. §4.1 paraphrases MOQT's scheduling order, its delivery timeouts and its joining (MOQT §5.1, §5.2, §3.5) only to say how a stage set uses them and which values it sets; it defines none of them, and where a paraphrase and MOQT differ, MOQT holds.
• It does not replace CMSF or LOC. A publisher with CMAF-packaged media should use CMSF; this packaging is for tracks whose objects are WAI envelopes.
• It does not make MoQ WAI's transport. WAI remains carrier-agnostic; this is one binding among the several jwp-receipts §1 anticipates.
• It is not the only binding. http-delivery binds the same objects to HTTP, admits WAI2 envelopes with verified component ranges, carries WAI_RECEIPT in a sidecar index, and maps this extension in its Appendix C.
• It does not pin a MOQT draft revision. MOQT is pre-Last-Call; an extension that pinned a moving target would be wrong within a meeting cycle. Where MOQT and this text disagree, MOQT wins and this text is the thing that must change. The revisions named in the status note are the ones this text was last read against: where a later revision renumbers or renames what it cites, the citation is what changes.

7. Framing

The delivery industry is converging on MoQ, and the layer above it is where formats declare themselves. A container that carries instructions, dispatches by capability, and commits per object has a natural home there — not because it competes with the transport, but because the transport deliberately left the question of meaning to someone else.