open-standards/wai/extensions/live-ingest.md

WAI Extension: Live Ingest and Egress

Status: Draft. Binds the contribution and playback protocols of standard encoders and players to the tracks of a room (live-rooms): WHIP (RFC 9725) into a room's tracks, WHEP (draft-ietf-wish-whep-05) out of them, Enhanced RTMP v2 into them, single-track and multitrack, and SRT, which live-contribution profiles with RIST. An ingest gateway admits a publisher by its room grant, takes its media frame by frame, packages each frame as one WAI envelope on a MoQ track (moq-streaming-format), maps its media time onto the room clock with a stated bound, and signs every group's receipts as the first hop (jwp-receipts). An egress gateway verifies every object against its receipt before it hands the frame to a player. No new transport and no new token format: WAI defines none by charter. The reference implementation of WHIP, WHEP and Enhanced RTMP is joule-moq-ingest in joule-code/joule-code-rs/crates/, on the joule-moq codec and the joule-moq-relay relay; its Enhanced RTMP corpus is live-conformance/ertmp/, and that corpus's second implementation is live-conformance/ertmp_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: RFC 9725 (WHIP); draft-ietf-wish-whep-05 (WHEP), an Internet-Draft not past Working Group Last Call; RFC 6750 §2.1 and §3; RFC 8840 (the trickle ICE fragment); RFC 9110 §8.8.3 and §13.1.1; RFC 6585 §3; RFC 9457; RFC 6184 (H.264 over RTP); RFC 7741 (VP8); RFC 9628 (VP9); the AV1 RTP payload format v1.0.0; RFC 7587 and RFC 6716 (Opus); draft-ietf-moq-transport-22 (MOQT) and draft-ietf-moq-msf-01 (MSF); Enhanced RTMP v2, document version v2-2026-01-31-r2 (E-RTMP), with the legacy RTMP 1.0, FLV 10.1 and AMF0 specifications it references as [RTMP], [FLV] and [AMF0]; ISO/IEC 14496-15 for the AVC and HEVC configuration records, ISO/IEC 14496-3 and ISO/IEC 13818-7 for AAC's AudioSpecificConfig and ADTS, RFC 7845 §5.1 for the Opus ID header, RFC 9639 for FLAC, and the AV1 and VP codec ISOBMFF bindings for their configuration records; and SRT 1.5.7, its stream ID guidelines and its rejection codes. A section number of one of those texts here is that revision's.

1. Scope

In scope: how a WHIP request is admitted under a room grant (§3); the tracks an ingest publishes and its catalog (§4); one frame as one object, the payload framing of each codec, and how frames form groups (§5); the live member of an object's manifest (§6); the mapping from RTP time to the room clock and its bound (§7); the receipts the gateway signs (§8); WHEP egress and what it verifies (§9); the bounds a gateway holds a session to (§10); trickle ICE and ICE restarts (§11); Enhanced RTMP v2 ingest (§12) and the mapping of SRT (§13); the refusals and their statuses (§14); and how an ingest relates to capture binding and to metering (§15, §16).

Out of scope: a transport (WebRTC, MoQ, RTMP and SRT are the transports); how a principal obtains a grant (live-rooms §1); transcoding: a gateway carries the frames the publisher sent, as the publisher encoded them; slots and their takeover and failover (live-rooms §7), which an ingest into a slot follows and this revision does not map; and simulcast or SVC within one WHIP session (RFC 9725 §4.8).

Why an extension of its own. moq-streaming-format says what a WAI track is on MoQ. This text binds four other protocols onto those tracks, under the rooms and grants of the wai.live family; written into that extension it would make a MoQ format describe RTMP and SRT. It sits beside live-rooms, live-events and capture-binding, and changes none of them.

Relationship to the core spec: no record and no capability is registered here. The payload capabilities a live track carries are SPEC §5's, among them wai.video.vp8 and wai.audio.opus.packet, registered for this binding. The manifest member live (§6) is defined here; a reader that does not know it ignores it (SPEC §3).

2. Terms

• Room, room clock, principal, grant, room authority: as in live-events §2 and live-rooms §2.
• Gateway: an endpoint that speaks one of the protocols of this text on one side and MOQT to a relay on the other. An ingest gateway takes media from a publisher; an egress gateway hands media to a player. One process may be both.
• First hop: the gateway an ingest's media reaches first. It is the MoQ publisher of the tracks (MOQT §7.6) and the signer of their receipts (§8).
• Owner: the principal whose tracks they are: the grant's principal (live-rooms §6.1).
• Frame: one access unit of video, or one packet of audio, as the RTP payload format of its codec delivers it whole (§5.1).
• µs: microseconds. Integers are little-endian where bytes are written.

3. Admission

3.1 The bearer token

A WHIP or WHEP request is authenticated by an HTTP Authorization header with a bearer token (RFC 9725 §4.7.1; WHEP §4.9.1; RFC 6750 §2.1). The token is a room grant: the WRG1 record of live-rooms §4.1, whole, in the base64url alphabet without padding (RFC 4648 §5). The room it names is the room the request is for. WAI defines no second token format; a deployment that issues tokens of another format maps them onto a grant as live-rooms §10 says, and admits by the grant.

A grant is a public record, so presenting it proves nothing about who presents it. Every request that carries a grant also carries a proof of possession of the grant's key (live-rooms §10.5): a WRP1 record, in base64url, in the WAI-Room-Proof field, over a challenge the gateway issued and the request's context (§3.2 step 7). The gateway issues single-use challenges in the WAI-Room-Challenge field: on a 401 that refuses for want of a good proof, and on every response to a request that carried a grant, the one a 201 carries being scoped to the session it opened (live-rooms §10.5.3).

3.2 The checks

A gateway holds the signed descriptor of every room it serves. It admits a token by these checks, in order, and refuses at the first that fails with the reason it names:

1. The token decodes, and the room its grant names (bytes 5 to 36) is a room the gateway holds: otherwise room_unknown, or the reading refusal of live-rooms §4.3 step 1 (truncated, malformed, unknown_version).
2. live-rooms §4.3 steps 1, 2 and 4 to 8, against that room's descriptor: the record reads; its signature verifies under the room's authority; it names the room; its fields keep §4.3's rules; it is at most fourteen days long; it is within its role and within the room's ceiling. Each refusal is §4.3's reason. Step 3, an exchanged grant's link to the grant it replaces, needs the earlier grant, which a token does not bring: the gateway checks instead that the room clock has reached the exchanged grant's effective tick.
3. The grant is good at the gateway's clock (live-rooms §4.3): otherwise outside_grant_window.
4. Neither its digest nor its viewer's tag is among the revocations the gateway holds for the room's authority's key log (live-rooms §5): otherwise grant_revoked.
5. Where the grant lists origins, the gateway's own origin is one of them: otherwise origin_not_granted.
6. Where the grant's once is not zero, no session of this gateway was opened under it: otherwise grant_reused. The nonce is spent when a session opens.
7. The request's proof passes live-rooms §10.5.4 for the grant, as this gateway reads it (its digest and its principal), with the challenge it holds for the request's scope (the endpoint, or the session its URL names), the challenges it consumed before, and the context METHOD //authority/path of the request it received: otherwise that section's reason (pop_missing, pop_malformed, pop_wrong_grant, pop_wrong_key, pop_bad_signature, pop_wrong_challenge, pop_replayed, pop_wrong_context). A proof that passes consumes its challenge: it is good for one request.

A token missing or not a bearer token is answered 401 with WWW-Authenticate: Bearer realm=… and no error (RFC 6750 §3.1). A token refused by these checks is answered 401 with error="invalid_token" and the reason as error_description.

3.3 The powers

The grant must give what the request asks for (live-rooms §3.3). A grant that passes §3.2 and lacks it is answered 403 with WWW-Authenticate: Bearer error="insufficient_scope" (RFC 6750 §3.1):

• WHIP: the grant's publish bit for each media kind the offer carries, audio (1) and video (2): otherwise publish_not_permitted. A screen is published as media kind screen (3) only through a mapping that names it, which this revision does not.
• WHEP: subscribe, and a max_layer at or above the layer asked for: otherwise subscribe_not_permitted or layer_not_permitted. The room in the resource's path is the grant's: otherwise wrong_room.

Where the grant names a publish profile for a media kind (live-rooms §4.1 profiles), the session accepts only the codecs of that profile's codecs that the gateway carries, by RFC 6381 family (avc1, vp8, vp09, av01, opus), and states the profile's layer-0 max_kbps as the media section's bandwidth (§10).

3.4 The session's resource

Every request to a session's resource (PATCH, DELETE, GET, HEAD) carries the token the session was opened with, and a proof of possession over a challenge scoped to that session (§3.2 step 7); another token, however good, is answered 403 (not_the_session_token). A session's URL carries at least 128 random bits (RFC 9725 §5).

4. Tracks and the catalog

4.1 Names

A WHIP session publishes, for each media kind its offer carries, one media track and its receipts track, and, for the owner, one catalog track and its receipts track:

track	Track Namespace (live-rooms §6.1)	Track Name
media	(room id, owner, audio or video)	0
its receipts	the media track's	0_receipts
catalog	(room id, owner)	catalog
its receipts	the catalog's	catalog_receipts

A WHIP session sends one encoding of each kind, so its media tracks are layer 0 (live-rooms §6.3, single). A receipts track is named as live-rooms §6.1 names one, the media track's name and _receipts, and is of packaging wai-receipts. The catalog is the owner's MSF catalog (MSF §5): the namespace prefix (room id, owner) reaches it and every track it declares.

4.2 Publication

The gateway publishes every track at the relay with PUBLISH (MOQT §9.8) before it answers the WHIP request, so a subscriber can subscribe from the first group. The tracks' default publisher priority is 0 for audio and the catalog and 1 for video (MOQT §5.1.1): a subscriber that falls behind keeps the sound. A receipts track's priority is its media track's (moq-streaming-format §4.1). No track carries a delivery timeout.

When the session ends the gateway ends each open group, publishes its receipts, waits for every stream to be acknowledged, and sends PUBLISH_DONE with status TRACK_ENDED on each track (MOQT §9.9). The relay then ends its subscriptions: a subscriber sees the track end.

4.3 The catalog

The catalog is RFC 8785 canonical JSON, {"tracks": [...], "version": 1}, one object a group. Each media track's entry carries MSF's namespace (the MOQT §8.8 rendering of its namespace, live-rooms §6.2), name, packaging "wai", isLive true, role, renderGroup 1, and, where the bitstream states them, codec (RFC 6381), width and height, or samplerate and channelConfig; and moq-streaming-format §4's wai_capabilities (the one capability its objects dispatch to), wai_receipts true, wai_receipt_alias (§8.2) and wai_receipts_track. Each is followed by its receipts track's entry.

The gateway publishes the catalog once every media track has carried its first frame, or a second after the first did, whichever comes first, and publishes it again, whole, in a new group, when a track's codec or picture size changes. Its group IDs follow §5.3's rule. The catalog is signed as the media is (§8): each catalog group is one object, receipted on catalog_receipts. A relay that checks every track of a room at its door forwards it, and a player verifies what the catalog declares before it relies on it (§9.1).

5. Objects and groups

5.1 One frame, one object

Each frame is one MOQT object, and each object is exactly one WAI1 envelope (moq-streaming-format §3). The frame is what the codec's RTP payload format reassembles, whole:

codec	RTP payload format	capability (SPEC §5)	the object's payload
H.264	RFC 6184	wai.video.avc	one access unit, Annex B byte stream: each NAL unit after a four-byte start code
VP8	RFC 7741	wai.video.vp8	one frame in the IVF frame container, FourCC VP80: the 32-byte file header (the frame's size, a time base of 1/90 000, one frame), the 12-byte frame header (its length, its time in 90 kHz ticks), the frame
VP9	RFC 9628	wai.video.vp9	as VP8, FourCC VP90
AV1	AV1 RTP payload format v1.0.0	wai.video.av1	one temporal unit in SPEC §5's AV1 framing: n = 1, the largest frame's height and width from the sequence header, a frame rate of 90 000 over the ticks since the track's previous frame, then the temporal unit's length and the unit: a temporal delimiter OBU (which RTP does not carry, and which the gateway restores), then every OBU with its size field
Opus	RFC 7587	wai.audio.opus.packet	one Opus packet (RFC 6716 §3)

A frame of a codec this table does not list is not carried. A frame larger than the gateway's bound is dropped and counted (§10).

A live track's objects are one stream. A frame is a unit of its codec's stream, not a picture or a sound decodable alone. The objects of a media track, in object order from a group's first object and in group order, are consecutive units of one stream of the codec: a sink decodes them in that order with one decoder state, from a group's first object, and resets that state where a group follows one it did not receive. The capability of every object says how its payload is read; this rule says how consecutive payloads relate. A sink that joins a track joins at a group's first object.

5.2 Groups

• Video: a group begins at each keyframe (an IDR access unit for H.264; a key frame for VP8 and VP9; a temporal unit with a sequence header for AV1) and holds it and every frame up to the next. A frame that arrives while no group is open (before the first keyframe) is dropped, counted, and answered with a keyframe request (a PLI, RFC 4585 §6.3.1). A group that runs longer than max_video_group_ms (2 000 by default) on the room clock makes the gateway ask for a keyframe, at most once a keyframe_request_interval_ms; the group ends when the keyframe comes.
• Audio: a group holds the frames whose room-clock time (§7) falls in one window of audio_group_ms (200 by default). The windows are laid end to end from the track's first frame; after a pause longer than a window the next window starts at the next frame.

A group's objects travel on one subgroup stream (MOQT §10.3, Subgroup ID 0), written as the frames arrive, and the stream ends with the group: its END_OF_GROUP bit is set and its FIN follows the group's last object.

5.3 Group IDs

A track's group IDs ascend by one. The first group's ID is the room-clock time, in µs, of its first frame (§7). A publisher that reconnects, or a new session of the same owner, therefore continues above every ID the track used before, so a relay never takes a new group for a copy of an old one (MOQT §7.3). A catalog group's ID follows the same rule, from the room-clock time it is published.

6. The live manifest member

An object's manifest carries SPEC §3's members and the member live, an object:

member	type	meaning
t	integer	the frame's room-clock time, µs since the room's t0 (§7)
rtp	integer	the frame's RTP timestamp, extended past its 32 bits by the WebRTC stack; for an Enhanced RTMP track, its decode time in µs (§12)
rate	integer	its RTP clock rate: 90 000 for video, 48 000 for Opus; 1 000 000 for an Enhanced RTMP track
key	boolean	video only: whether the frame is a keyframe, the first of its group
cto	integer	Enhanced RTMP video only, and only when not 0: the composition time offset in µs; the frame is presented at t plus it (§12)
grant	object	on the first object of each group only: the grant reference of live-rooms §4.5, grant_hash, capability (wai.live.room), joule_ceiling_micro and funds_ceiling (null for none)

The receipt's content hash covers the envelope's bytes (jwp-receipts §2.1), so the group receipt that covers a group's first object signs the grant reference for the whole group: this is how every object of an ingest names the grant it was carried under (live-rooms §4.5), at the cost of one reference a group. The manifest's members are written in a fixed order with no insignificant whitespace (SPEC §2).

A sink that reads t reads the gateway's statement of the frame's time, bounded as §7 says; it is not the publisher's.

7. Time

7.1 The mapping

The gateway's clock gives Unix time; the room clock is that time less the room's t0 (live-rooms §3.1). For each media track, the first frame anchors the mapping: its room-clock time is its arrival at the gateway, A, and its extended RTP timestamp is R0. A later frame whose timestamp is R maps to

t = A + ⌊(R − R0) · 10^6 / rate⌋ µs.

A frame whose mapped time lies more than max_skew_ms (250 by default) from its own arrival, or whose timestamp is below R0, re-anchors the mapping at itself: its time is its arrival, and the anchor is (its arrival, its timestamp). Mapped times never decrease on a track: a frame that would map before the last one takes the last one's time.

7.2 The bound

So every frame's t lies within max_skew_ms of its arrival at the gateway. A frame arrives after it was captured, by the path's one-way delay, so t is no earlier than its capture less max_skew_ms, and no later than its capture plus its one-way delay plus max_skew_ms, both on the gateway's clock. Two frames of one session, of either track, are placed relative to each other within their arrivals' difference plus twice max_skew_ms. Between re-anchors, the publisher's media clock is followed exactly: t advances as the RTP timestamps do.

The bound is to the gateway's clock. Its error against the room clock, as the room's clock key keeps it, is the gateway host's clock error, which this revision neither measures nor states. A gateway counts its re-anchors in its session record (§16).

7.3 Across the room

A marker or a command set on the room clock lands on an ingest's media within §7.2's bound. An event anchored to an object of an ingest track (live-events §3.1) names the object, and the object's live.t gives that object's time; a sink that maps an anchored event takes t from the object, not from the event.

8. Receipts

8.1 What the gateway signs

When a group ends, the gateway signs its group receipt (jwp-receipts §2.4.1) over every object of the group and publishes it with the object receipts as group g of the track's receipts track (moq-streaming-format §4.1). Every object receipt is labelled (jwp-receipts §2.4.2): its figure is the unmetered marker, 0 with class tag 0 (energy-measurement §2), since no meter backs the gateway; it is never a figure of zero joules. Its origin label is the gateway's. The group receipt is signed in the labelled form, its class composed from its objects', and chains to the track's previous group by parent_receipt_hash. Its signer_id is the gateway's (jwp-receipts §7.1): origin-bound where the gateway's origin publishes its key log, its key's did:key otherwise.

What the receipts state is the first hop's: that the gateway received these bytes, under this grant, at these room-clock times. A WebRTC publisher signs nothing, so they are not the publisher's statement. A publisher that speaks WAI publishes its tracks over MoQ itself and signs its own receipts.

8.2 The chain

A track's wai_receipt_alias names its receipt chain (moq-streaming-format §4): the first eight bytes, little-endian, of BLAKE3 over wai:ingest-receipt-alias (ASCII), 0x01, the session's 16-byte id, the rendered namespace, one zero byte, and the Track Name, reduced below 2^53. Each session's tracks so sign chains of their own: a publisher that reconnects starts a new chain, and never forks the old one. The catalog that follows the reconnection names the new alias.

8.3 When the receipts are published

A group's receipts go out after its last object, since the group receipt's root covers every object of the group. This differs from moq-streaming-format §4.1's rule for a stage set, where a live group holds one object and its receipt precedes it: an ingest's groups hold many frames, and a frame is published as it arrives. A subscriber that does not verify receives each frame at once; one that verifies holds a group until its receipts arrive, at the group's end. A relay with a door on the room's namespace (jwp-receipts §4.1) holds each object until its group's receipts arrive, so its hold must exceed the longest group, max_video_group_ms plus the time a keyframe takes to come.

9. WHEP egress

9.1 A session

A player POSTs its offer to the resource of one owner's tracks, …/whep/<room id>/<owner>[?layer=<n>], the room id and the owner as 64 lowercase hexadecimal digits each, and n the layer, 0 by default (WHEP §4.3). Admission is §3's. The gateway:

1. subscribes at the relay to the owner's catalog and its receipts track, and reads the latest catalog whose receipt verifies (§9.2) under a key it trusts as a first hop. Where none arrives within its wait, the request is answered 409 with Retry-After (WHEP §4.3.9): the owner is not live. Where the catalog declares no track of the layer, 404 (layer_unknown);
2. answers the offer with the codecs of the declared tracks only, so a section of a kind the owner does not publish, or of no common codec, is rejected (WHEP §4.3.4). An offer none of whose sections is accepted is 422 (no_common_codec);
3. subscribes to each accepted kind's media track and its receipts track from the start of the latest group: the live part from the Next Object, and a fill of the latest group from its first object (MOQT §3.5, §9.20.9). A player so starts at a group's first object, a keyframe for video.

A track's end at the relay (PUBLISH_DONE) ends the session, after every frame that verified is handed on.

9.2 Verification

For each group of each track, before any of its frames reaches the player:

1. its receipts object reads as a bundle: otherwise refused_bundle;
2. its receipts name this MOQT group, each object receipt its Object ID, and the track's wai_receipt_alias (moq-streaming-format §4.1, moqt_mismatch): otherwise refused_moqt. The catalog's own chain is not declared anywhere, so a catalog receipt is held to its group and Object ID alone;
3. its group receipt verifies under a trusted first hop's key (jwp-receipts §3): otherwise refused_signature;
4. where the session holds the receipt of the group before it, its parent_receipt_hash is that receipt's link hash: otherwise refused_chain;
5. each object matches its object receipt: its content hash, and its Merkle proof to the signed root (jwp-receipts §3): otherwise refused_inclusion;
6. each object opens as a WAI1 envelope of the track's capability with a live member, and its payload as one frame of its framing (§5.1): otherwise unopenable.

What fails is dropped and counted, and never reaches the player. A group whose receipts do not arrive within hold_ms (5 000 by default) of its last object is dropped and counted (receipt_missing); an object a verified group's receipts name that did not arrive is counted as receipted and not delivered (object_missing). Groups are handed on in group order, their frames in object order. A frame's RTP time is its live.t at the codec's clock rate, and its wallclock is placed from the session's first frame by t, so the tracks of a session keep the timing §7 gave them.

9.3 What a WHEP player cannot see

WebRTC carries media, not receipts: a WHEP player receives frames and nothing that proves where they came from or that they are whole. The egress gateway is the verifier on its behalf, and its session record (§16) is its account of what it verified, dropped and why. A WAI-aware player gets the receipts itself: it subscribes to the same media tracks and their receipts tracks over MoQ, through the relay's WebTransport or WebSocket paths, and verifies each object as §9.2 does, with no gateway between. Both read the same tracks; only the second holds the proof.

9.4 Latency

A frame reaches a WHEP player when its group's receipts have verified, at the group's end: up to a video group's length after it reached the relay, plus the relay's and the network's delay. A deployment that wants a shorter path shortens its groups (§5.2), at the cost of keyframes, or plays over MoQ unverified and verifies afterwards.

10. Bounds

A gateway states each bound in its configuration, and refuses what passes it:

bound	refused with
WHIP sessions at once; WHEP sessions at once	503 with Retry-After (RFC 9725 §4.5), session_limit
media sections in one offer (max_tracks_per_session, 2 by default); more than one of a kind	422 (RFC 9725 §4.4.2), too_many_tracks, one_track_per_kind
the media a WHIP session sends, in kilobits per second over a window	stated per media section in the answer as b=AS (RFC 8866 §5.8); a session that passes it over bitrate_window_ms is ended, bitrate_exceeded
the largest frame	dropped and counted
the bytes waiting for the relay	the session is ended, relay_backpressure
proof-of-possession challenges held at once, issued and consumed each (max_challenges), and how long one stays good (challenge_ttl_s)	the oldest is dropped; a proof over a challenge past its time is refused pop_wrong_challenge

11. ICE

A session supports trickle ICE and ICE restarts (RFC 9725 §4.3; WHEP §4.5), with entity-tags: the 201 carries the ICE session's ETag, and a PATCH names the ICE session it is for in If-Match. A PATCH without If-Match is 428; one whose tag is not the current one's, or *, is 412 (RFC 9110 §13.1.1; RFC 6585 §3).

• Trickle: a fragment whose credentials are the current ones, or absent, adds its candidates; one the gateway cannot use is discarded. 204, no ETag.
• Restart: a fragment with new ice-ufrag and ice-pwd restarts ICE. The gateway applies to its WebRTC stack the client's offer with those credentials and the fragment's candidates in place of the old ones, which is the offer an ICE restart would have sent (RFC 8839 §4.4.1.1.1), and answers 200 with an application/trickle-ice-sdpfrag body: its ice-lite and ice-options attributes, the BUNDLE group, the first media section's m= line and mid, its new credentials, its candidates, a=end-of-candidates; and a new ETag. A restart it cannot apply keeps the ICE session it has and is 422 (ice_restart_refused, RFC 9725 §4.3.3).

The gateway is an ICE lite agent where every client reaches its address (RFC 9725 §4.4.5), and a full agent otherwise. A session whose ICE stays disconnected past a grace period is ended (ice_disconnected).

12. Enhanced RTMP v2

This section binds Enhanced RTMP v2 (v2-2026-01-31-r2), single-track and multitrack, over TCP or TLS, onto room tracks. The gateway is the RTMP server.

12.1 Admission: a stream key

An encoder is given a server URL and a stream key, and has no way to answer a challenge, so the grant's proof of possession (§3.1) is made over HTTP before it connects:

1. The publisher POSTs to the gateway's …/rtmp resource with the bearer token of §3.1 and a proof of possession, admitted as §3.2 admits a WHIP request (the request opens a session: a single-use grant's nonce is spent). A grant that publishes neither audio nor video is answered 403 (publish_not_permitted).
2. The gateway answers 201 with application/json, Cache-Control: no-store: {"expires_in": <s>, "stream_key": <key>, "url": <server URL>}. The key is 128 random bits as 32 lowercase hexadecimal digits, good for one publish within expires_in seconds (60 by default).
3. The encoder connects and publishes with that key as its stream name. The gateway takes the key for its one use (stream_key_unknown where it holds no such key, for one used already among them; stream_key_expired past its time), and admits the grant it was issued under again at that instant (§3.2 steps 2 to 5; a grant whose window has closed is outside_grant_window). A refusal is an onStatus with level error, code NetStream.Publish.BadName, and the reason as description; then the connection is closed. A second publish on a connection is already_publishing.

A stream key is a bearer secret for its time: it travels in the clear over RTMP and under TLS over RTMPS, and a deployment serves the /rtmp resource over HTTPS (§18).

12.2 The connection

• Handshake and chunks. The handshake of [RTMP] §5.2: C0 and S0 are version 3 (another is handshake_version); S1 is a time, four zero bytes and random bytes; S2 echoes C1. The chunk stream of [RTMP] §5.3, with the extended timestamp on every chunk of a chunk stream whose header carries one, and the delta of a type-3 chunk that begins a message the last delta of its chunk stream, where a fmt-0 header's delta is its timestamp field. A header of fmt 1, 2 or 3 on a chunk stream with no header before it is chunk_no_header; a header of fmt 0, 1 or 2 while a message of its chunk stream is incomplete is chunk_interleaved; a message longer than the gateway's bound (8 MiB by default) is message_too_large; a Set Chunk Size of 0, of more than 2^31 − 1, or past that bound is chunk_size; more than 64 chunk streams with a message incomplete is chunk_streams. Each ends the connection.
• Protocol control. Set Chunk Size, Abort, Window Acknowledgement Size (the gateway acknowledges each window the client set) and a User Control ping, answered; the gateway sends Window Acknowledgement Size and Set Peer Bandwidth of 2 500 000, then its own chunk size (4 096 by default).
• connect. Its _result's properties declare capsEx with Reconnect (0x01), Multitrack (0x02), ModEx (0x04) and TimestampNanoOffset (0x08), and the FourCCs of §12.4 in videoFourCcInfoMap and audioFourCcInfoMap, each CanForward (0x04) (E-RTMP Enhancing NetConnection connect Command); its information object is NetConnection.Connect.Success with objectEncoding 0. The client's capsEx is kept for the reconnect request (§12.8).
• Commands. createStream is answered with stream 1; releaseStream, FCPublish, getStreamLength and _checkbw with an empty _result; publish is §12.1's, answered with User Control Stream Begin and onStatus NetStream.Publish.Start; FCUnpublish, deleteStream and closeStream end the session (deleted). A command that does not read as AMF0 is command_malformed. AMF3 command and data messages are read past their format byte; AMF3 values are not read.
• Data. @setDataFrame onMetaData, or onMetaData: its argument, an object or an ECMA array, states the tracks (E-RTMP Enhancing onMetaData): width, height, framerate, videocodecid, audiosamplerate, audiochannels or stereo, audiocodecid for track 0 of each media type, and videoTrackIdInfoMap and audioTrackIdInfoMap for the others, by track id. A later call replaces it.
• Aggregate messages ([RTMP] §7.1.6) are FLV tags, each 11 bytes of header, its body and a 4-byte back pointer; each tag's timestamp is offset to the message's.

12.3 Tracks

The tracks are published at the relay when publish is admitted, before NetStream.Publish.Start (§4.2): the owner's catalog and its receipts track; audio layer 0 where the grant publishes audio; and, where it publishes video, a video track per layer of the grant's video publish profile, or layer 0 where it names none. Each is named as §4.1 names a layer, its layer the Track Name (live-rooms §6.1, §6.3).

Each (media type, track id) of the RTMP stream (0 for a message that is not multitrack) is bound to a room track at its first packet, and keeps that binding:

1. its FourCC is one of §12.4's (codec_not_carried); the grant publishes its media kind (publish_not_permitted); and the grant's publish profile for the kind, where it names one, lists the codec's RFC 6381 family (codec_not_permitted);
2. audio: track 0 is layer 0, and another track is not carried (audio_track_not_default);
3. video: where the grant names a video profile and the track's onMetaData entry states its width, height and frame rate, its layer is the profile layer of that width and height and that frame rate rounded to the nearest integer (track_no_layer where no layer is); otherwise track 0 is layer 0, and another track is not carried (track_no_layer). Track ids imply no order (E-RTMP Multitrack Media Message Guidelines), so a layer is never inferred from an id alone;
4. a layer another track already holds is not carried (track_layer_taken).

What is not carried is counted, by reason, in the session's record (§16).

12.4 FourCCs and payloads

A message's body is read as E-RTMP's Enhanced Video and Enhanced Audio tables read it, legacy or enhanced, through every ModEx and every track of a multitrack body; a body that does not read is counted and carries nothing: a ModEx TimestampOffsetNano of fewer than 3 bytes or above 999 999 (tag_modex), a multitrack header naming multitrack again or a reserved multitrack type (tag_multitrack), a packet type the text reserves (tag_packet_type), a body that ends early (tag_truncated). Each track's sequence start is its configuration (sequence_start_unreadable where it does not read); a frame before it is no_sequence_start; a frame that does not read under it is frame_unreadable; a frame the payload cannot hold is frame_not_carried. A sequence end ends the track's open group; a video Metadata packet, an MPEG2TSSequenceStart, an audio MultichannelConfig, a command frame, an empty (silence) audio message, and a coded-frames packet of no bytes (as an encoder sends at the end of a stream) carry nothing (packet_not_carried). A frame of a FourCC other than its track's sequence start's is codec_changed_without_sequence_start.

FourCC	capability	configuration	the object's payload
avc1 (and legacy codec id 7)	wai.video.avc	AVCDecoderConfigurationRecord (ISO/IEC 14496-15 §5.3.3.1)	Annex B: each NAL unit after a four-byte start code; the record's SPS and PPS first in an access unit with a key frame type or an IDR picture that carries none
hvc1	wai.video.hevc	HEVCDecoderConfigurationRecord (§8.3.3.1 there)	Annex B; the record's NAL units first in a key access unit that carries no VPS, SPS or PPS
av01	wai.video.av1	AV1CodecConfigurationRecord	§5.1's AV1 framing: the temporal unit's OBUs, a temporal delimiter it carries dropped (the framing restores one), the record's sequence header OBU first in a key temporal unit that carries none
vp09, vp08	wai.video.vp9, wai.video.vp8	VPCodecConfigurationRecord (with or without its box's version and flags)	§5.1's IVF framing, the size from the frame's header
Opus	wai.audio.opus.packet	the ID header (RFC 7845 §5.1), or none	the packet; a body of more than one stream (N > 1) is frame_not_carried
fLaC	wai.audio.flac	fLaC and a STREAMINFO block; or STREAMINFO's 34 bytes alone, as an encoder sends it	a FLAC stream: fLaC, the STREAMINFO block marked the last metadata block, the frames
mp4a (and legacy sound format 10)	wai.audio.aac	AudioSpecificConfig	one ADTS frame (ISO/IEC 13818-7 §6.2): protection_absent 1, the profile of the core audio object type (1 to 4, through explicit SBR signalling), the sampling frequency index (0 to 12) and the channel configuration (1 to 7) of the configuration, a frame length within 13 bits

vvc1 (wai.video.vvc) is mapped by this table's rule for hvc1, with the VVCDecoderConfigurationRecord's NAL units, and is not carried in this revision (codec_not_carried): no encoder available to its tests sends it. ac-3, ec-3, .mp3, the other legacy codec ids and sound formats are not carried. The catalog's codec is written from the configuration: avc1.PPCCLL from the AVC record's profile, compatibility and level; hvc1. and ISO/IEC 14496-15 Annex E's fields from the HEVC record's profile space, profile, compatibility flags (bit-reversed, in hexadecimal), tier, level and constraint bytes (trailing zero bytes left out); av01.P.LLT.DD from the AV1 record; vp09.PP.LL.DD from the VP record; mp4a.40.<object type>; opus; flac. Its width and height are the bitstream's where §5.1's framing reads them, and the track's onMetaData entry's otherwise; samplerate and channelConfig are the configuration's.

12.5 Time

A frame's media time is its message's timestamp, extended past its 32 bits (a step back of more than 2^31 ms is a wrap), in µs, plus its message's TimestampOffsetNano in whole µs: the frame's decode time. §7's mapping applies with it as R and a rate of 1 000 000, anchored at the track's first frame's arrival, so t follows the decode times and never decreases. The object's live.rtp is the media time and live.rate 1 000 000. A video frame's composition time offset (the SI24 of CodedFrames and of legacy AVC, 0 for CodedFramesX) is written as live.cto in µs where it is not 0: the frame is presented at t plus it. RTMP runs over TCP, so no frame is lost and none reordered; a stall shows as frames that arrive late together, which §7 bounds by re-anchoring.

12.6 Groups and receipts

A video group begins at a key frame of its track (§5.2): a frame of frame type 1, or, for AVC and HEVC, an access unit with an IDR or IRAP picture; audio windows are §5.2's. RTMP has no keyframe request, so a group that runs past max_video_group_ms is counted (group_long), not answered. Receipts and the catalog are §8's and §4.3's.

12.7 Bounds

A session that sends more than the gateway's max_bitrate_kbps over every track (40 000 by default) over a window ends (bitrate_exceeded); one that carries no media for its idle bound ends (idle); a connection that has not published within its connect bound is closed (connect_timeout); the gateway holds at most its max_sessions connections, and closes one more at accept.

12.8 The reconnect request

When the gateway shuts down it sends onStatus with code NetConnection.Connect.ReconnectRequest, level status, and its configured tcUrl, if any, to each publishing client that declared Reconnect in capsEx (E-RTMP Reconnect Request), and the client's stream key is good once more for its time, for the reconnection. The client reconnects at its next keyframe; the gateway keeps the session for reconnect_grace_ms (5 000 by default) and ends it then or when the client leaves (reconnect_requested). A client that did not declare Reconnect is ended (gateway_stopped). The new session's first group ID is its first frame's room-clock time (§5.3), above every ID the old session used; its receipt chain is new (§8.2), and its catalog names it. A key is held by the gateway that issued it: a reconnection to another gateway is admitted there only where that gateway was given the key, which this revision does not define.

13. SRT (mapping)

This section maps SRT 1.5.7 caller-listener connections onto room tracks. live-contribution profiles it as the reference gateway implements it (its §3.2 gives the listener's checks in order, its §4.1 the transport's settings, its §5 the stream onto the tracks), adds RIST, and defines the link record the gateway signs.

• The stream ID and the grant. A stream ID holds at most 512 characters (SRT stream ID guidelines). A grant's fixed fields alone take 315 bytes, 420 characters in base64url, and with the room, the owner and the mode beside them a stream ID would pass 512 before the grant names a profile, an origin or an attribute. A caller therefore first obtains a session ID: it POSTs to the gateway's …/srt resource with the bearer token of §3.1, admitted as §3 admits WHIP and WHEP, and receives a one-time session ID, valid for 60 seconds and invalidated by its first use, and the passphrase for the connection. Its stream ID is then #!::r=<room id>/<owner>,m=publish,s=<session ID> to publish and m=request to play, in the guidelines' key-value syntax with their standard keys (r the resource, m the mode, s the session ID, which the guidelines define for exactly this case: details negotiated over a separate connection first). The gateway's listener callback finds the session, checks that its grant gives the mode (m=publish the publish power, m=request the subscribe power) for that room and owner, and sets the connection's passphrase, so the SRT handshake fails for a caller without it.
• Refusals. The listener callback rejects with the SRT rejection codes of the HTTP statuses §14 gives: 1401 (SRT_REJX_UNAUTHORIZED) for a session or grant that is not good, 1403 (SRT_REJX_FORBIDDEN) for a power the grant does not give, 1404 (SRT_REJX_NOTFOUND) for a room the gateway does not hold, 1405 (SRT_REJX_BAD_MODE) for a mode other than publish and request, 1406 (SRT_REJX_UNACCEPTABLE) for a payload it cannot read, 1409 (SRT_REJX_CONFLICT) for an owner already publishing that kind through another session, and 1402 (SRT_REJX_OVERLOAD) past a bound of §10.
• Payload. SRT carries any payload; in live mode a contribution encoder sends an MPEG transport stream (ISO/IEC 13818-1), seven 188-byte packets a datagram. The gateway demultiplexes it by its program map table: stream type 0x1B is wai.video.avc (each PES packet's access unit, already Annex B), 0x24 wai.video.hevc, 0x0F wai.audio.aac (one ADTS frame an object). Other stream types are not carried in this revision. Each carried elementary stream of a kind is a room track as §4.1 names it; a second of one kind is not carried.
• Time. A PES packet's presentation time stamp, 33 bits at 90 kHz, extended past its 33 bits, is the media time of §7's mapping, with a rate of 90 000. SRT delivers each packet at its sender's time plus the connection's latency (its timestamp-based packet delivery), so frames arrive with that latency added and evenly; §7's bound applies to the variation.
• Latency. The connection's latency is the larger of the caller's and the listener's (SRTO_LATENCY; 120 ms unless set). It bounds how long SRT retransmits a lost packet, and a retransmission takes at least one round trip, so a gateway sets its listener's latency to several round-trip times of the paths it serves. The gateway states its latency in its configuration; a grant's profile states none.
• Groups and receipts are §5 and §8's.

14. Reasons

The refusals of this extension, each with its HTTP status and where it is raised. A reader of the WHIP and WHEP statuses keeps RFC 9725 §4.1's rule: a client falls back to a status's generic class for a reason it does not know. Each refusal's body is a problem details object (RFC 9457) whose reason member is the code.

reason	status	raised at
token_missing	401	§3.2
token_malformed	401	§3.2 step 1
room_unknown	401	§3.2 step 1
truncated, malformed, unknown_version, bad_key, bad_signature, wrong_room, grant_invalid, grant_too_long, caps_exceed_role, grant_exceeds_room	401	§3.2 step 2 (live-rooms §4.3, §11)
outside_grant_window	401	§3.2 steps 2 and 3
grant_revoked	401	§3.2 step 4
origin_not_granted	401	§3.2 step 5
grant_reused	401	§3.2 step 6
pop_missing, pop_malformed, pop_wrong_grant, pop_wrong_key, pop_bad_signature, pop_wrong_challenge, pop_replayed, pop_wrong_context	401	§3.2 step 7 (live-rooms §10.5.4, §11)
publish_not_permitted	403	§3.3
subscribe_not_permitted, layer_not_permitted	403	§3.3
wrong_room	403	§3.3, a WHEP resource of another room
not_the_session_token	403	§3.4
not_sdp, not_sdpfrag	415	an offer that is not application/sdp; a PATCH that is not a fragment
offer_malformed, sdpfrag_malformed	400	the body does not read
section_unsupported, direction, no_media, too_many_tracks, one_track_per_kind, one_media_stream	422	§10; RFC 9725 §4.2, §4.4.2; WHEP §4.3.6
no_common_codec, offer_refused	422	§3.3, §9.1; RFC 9725 §4.4.3
if_match_missing	428	§11
if_match_stale	412	§11
ice_restart_refused	422	§11
not_live	409, with Retry-After	§9.1
layer_unknown, owner_unknown, session_unknown	404	§9.1; a session that has ended
body_too_large	413	a request body past the gateway's bound
body_timeout	408	a request body that does not arrive in time
not_found	404	a path that is no resource of the gateway
session_limit	503, with Retry-After	§10
relay_unreachable, relay_refused, media_socket	503	the gateway cannot reach its relay, or bind a socket

The refusals of Enhanced RTMP (§12), each an onStatus NetStream.Publish.BadName whose description is the code, the closing of the connection, or a count in the session's record:

reason	raised at
stream_key_unknown, stream_key_expired, already_publishing	§12.1: NetStream.Publish.BadName, the connection closed
handshake_version, handshake_closed, chunk_no_header, chunk_interleaved, message_too_large, chunk_size, chunk_streams, command_malformed	§12.2: the connection closed
codec_not_carried, codec_not_permitted, audio_track_not_default, track_no_layer, track_layer_taken	§12.3: counted
tag_truncated, tag_modex, tag_multitrack, tag_packet_type, sequence_start_unreadable, no_sequence_start, frame_unreadable, frame_not_carried, packet_not_carried, codec_changed_without_sequence_start	§12.4: counted
dropped_before_keyframe, dropped_too_large, dropped_unreadable, group_long	§5.2, §10, §12.6: counted

A session that ends states why in its record: deleted, connect_timeout, idle, ice_disconnected, bitrate_exceeded, relay_backpressure, relay_lost, or, for WHEP, track_ended; for Enhanced RTMP also closed (the client left), refused:<reason>, reconnect_requested and gateway_stopped.

15. Ingest and capture binding

An ingest is not a capture binding. A WHIP session's tracks are the live tier: what the publisher's encoder sent, receipted by the first hop on arrival. A take (capture-binding) is the master tier: the device's own recording, committed by the device's key into the room's events while it is captured. The two relate through the room clock (capture-binding §10): the ingest's objects carry their room-clock times (§6, §7), and the take's commitments carry theirs.

A WHIP publisher's take can still be committed by the device, if the device speaks capture binding: it joins the room under a grant that binds it to the principal it captures for (capture-binding §3.1), records its master, and sends its take_commit events on its own events track over MoQ, beside its WHIP session. Its WHIP media and its master then share the room clock: a marker set on the live call at room time t lands at t in the master (capture-binding §10), within the master's bound there and this extension's §7.2 here. A device that speaks WHIP alone has a live tier and no take; nothing in this extension makes its live media a take, and a single-take claim is never made of an ingest's tracks.

16. Metering

A gateway is a hop of every room session it carries, in the role of an ingest gateway. When each of its sessions ends it signs a record of what the session carried: per track and direction the objects and bytes, and the drops by reason; the subscriber sessions served and their time by layer class; the grant the session ran under, with its ceilings and the bytes in and out under it; and its own energy, the unmetered marker. These are the counts of the per-hop session record that WAI's live metering defines, kept in that record's shape. Until that record is defined, the reference gateway signs them as JSON (joule-moq-ingest/session/1); a later revision has the gateway sign the per-hop record itself, under the key-log authority live metering gives an ingest gateway's key.

17. Status and conformance

mapping	status	where
WHIP (§3 to §11)	implemented	joule-moq-ingest: admission, the tracks and catalog, framing for H.264, VP8, VP9, AV1 and Opus, groups, time, receipts, ICE, bounds
WHEP (§9)	implemented	joule-moq-ingest: the catalog read and verified, joining at a group, per-object verification, egress
Enhanced RTMP v2 (§12)	implemented	joule-moq-ingest rtmp: the stream key, the handshake and chunk stream, AMF0 commands and onMetaData, legacy AVC and AAC, the FourCCs of §12.4 but vvc1, ModEx, multitrack layers by the grant's profile, aggregate messages, RTMPS, the reconnect request
SRT (§13)	implemented, as live-contribution profiles it	joule-contrib (the transport), joule-moq-ingest (the session, the stream onto the tracks, the link record)

The reference implementation's tests run on loopback: a WHIP publisher of H.264 and Opus whose frames a relay subscriber receives whole, every receipt verified (by the subscriber and by the relay's door); a WHEP player that gets the same frames from the group it joined; VP8, VP9 and AV1 each through WHIP; every refusal of §3 with its status; trickle ICE and an ICE restart with media across it; and DELETE ending the tracks at the relay. Its reading of room descriptors and grants is held to the corpus of live-rooms (live-conformance/rooms/): every room id and grant digest there, the registered profiles' records, and every refusal of a descriptor or a grant but those of an exchange's link (§3.2 step 2) and of the key log (§3.2 step 4), which admission does not make.

The Enhanced RTMP tests (tests/ertmp.rs) run a publisher of the reference's own over TCP and TLS: legacy AVC and AAC; enhanced HEVC with composition offsets and Opus; AV1 with FLAC and VP9 with enhanced AAC, with a ModEx nanosecond offset; multitrack layers bound by a grant's profile; every refusal of a stream key; and the reconnect request at shutdown. Each frame reaches a relay subscriber in §12.4's payload, at §12.5's time, every receipt verified by the subscriber and by the relay's door. tests/ertmp_ffmpeg.rs, run where a local encoder is present, publishes from that encoder as a test client and decodes what the relay carries in a conformant decoder. The corpus (live-conformance/ertmp/) holds RTMP message sequences with their arrivals and the objects each room track carries; ertmp_verify.py, written from this text and E-RTMP's, sharing no code with the reference, agrees with it on every object.

Not implemented in this revision, beside SRT's m=request (§13) and vvc1: a WHEP counter-offer (WHEP §4.3.2); a WHEP player's keyframe request carried upstream (MOQT's NEW_GROUP_REQUEST); publishing into a slot (live-rooms §7); simulcast within one WHIP session; STUN and TURN servers in Link headers (RFC 9725 §4.6); and reading the key log itself (§3.2 step 4 holds the revocations a deployment gives it).

18. Security and privacy considerations

• A token is a bearer capability (live-rooms §13): whoever holds it may publish as its principal through a gateway. Its window is fourteen days at most, a single-use grant opens one session, and a gateway's TLS front keeps it off the wire in clear (RFC 9725 §5).
• The first hop's receipts attest what the gateway received, not what the publisher's camera saw: a gateway that lies signs its lie with its own key, and a player that trusts its receipts trusts that key. A deployment lists the gateways it trusts as first hops, and a player that needs more verifies a take (§15).
• The receipts are signed after a group ends: a relay that withholds a group's receipts holds back its frames from every verifying sink, which notices the gap by the chain (§9.2 step 4), not by a missing frame alone.
• An egress gateway that verifies hands a player only frames that verified, so a tampered relay cannot reach a WHEP player with altered media; it can still withhold media, which the session record counts.
• A session's URL is unguessable (§3.4), and its resource answers only its own token, so a third party cannot end or restart another's session.
• Every bound of §10 limits what one session, and one grant, can make a gateway hold.
• An Enhanced RTMP stream key (§12.1) is a bearer secret for one publish within its time: it is issued only to a request that proved the grant's key, and it crosses the network in the clear over RTMP. A deployment offers RTMPS, and serves the /rtmp resource over HTTPS, as it serves WHIP's.