open-standards/joule-code/joule-code-rs/crates/joule-moq-relay

joule-moq-relay

A relay for MOQT, draft-ietf-moq-transport-22, which also speaks -18 and -16 to the sessions that choose them. Publishers send it tracks, subscribers ask it for them, and it fans each object out to every subscription, in the order the draft's priorities give, dropping what passes a subscriber's delivery timeout at this hop. Whatever it drops, the object's receipts still reach the subscriber, on the track's receipts track.

It is a library (Relay, RelayServer, client::Client) and a binary, wai-relay (the crate keeps its name; the binary is named for WAI). The wire codec is joule-moq's.

RelayServer::shutdown sends GOAWAY (below) and returns once every session's task has ended, its summary signed, and every port the relay listened on is closed: each UDP socket of its endpoint (quinn closes one when its endpoint's driver ends), the WebSocket listener and the control listener. A relay can start again on the same ports at once, in the same process. RelayServer::start refused part way (a port another socket holds, a file in the control socket's place) is shut down so before its error returns: every port it bound is free, and the next start can bind them at once (Control::start likewise closes what it began).

The drafts it speaks

draft-ietf-moq-transport-22, and -18 and -16 for a peer that offers no later one. Drafts do not interoperate, so a session speaks one from its first byte. The draft is chosen before any MOQT byte, the relay preferring -22, then -18, then -16:

• by ALPN on native QUIC: moqt-22, moqt-18, moqt-16 (-22 §6.2.2);
• by WT-Available-Protocols on WebTransport, which the relay answers with the WT-Protocol it chose (§6.2.1; draft-ietf-webtrans-http3-16 §3.3). A client that sends no WT-Available-Protocols is taken to speak -22; one that offers only drafts the relay does not speak is refused;
• by the subprotocol moqt-22 on the WebSocket fallback (below), which is -22 only.

The relay is written once, against -22. A session at -18 or -16 runs behind transport::bridge, which translates each message and object at the session's edge with joule-moq's draft18::bridge and draft16::bridge: at -18 stream by stream (the drafts share their shape); at -16 by giving each request on -16's one control stream a stream of its own, its Request IDs renumbered into -16's sequence, MAX_REQUEST_ID granted ahead, a cancelled stream becoming UNSUBSCRIBE, FETCH_CANCEL, PUBLISH_NAMESPACE_DONE or PUBLISH_NAMESPACE_CANCEL. What a session's draft cannot say is refused (a request, with REQUEST_ERROR) or left out, by name, and counted in the session's notes. Two rules differ at the edge: at -18 a requester's FIN on PUBLISH_NAMESPACE, SUBSCRIBE_NAMESPACE or SUBSCRIBE_TRACKS cancels it (-18 §6.1, §6.2), where at -22 a FIN is not a cancellation (§6.4.2.2); and at -16 a new Request ID may arrive ahead of one still in flight on another stream (SUBSCRIBE_NAMESPACE rides its own), which is admitted, where -16 §9.1 asks for the next in sequence. Each message in and out is logged at debug level. joule-moq's README tables every difference. Receipts ride objects' Immutable Properties and receipts tracks, which are the same on every draft, so they cross the bridge unchanged.

The relay sends its SETUP as soon as a session opens, without waiting for the client's (§6.3 lets either end send first): it declares MOQT_IMPLEMENTATION, MAX_REQUEST_UPDATES, MAX_AUTH_TOKEN_CACHE_SIZE, joule-moq's receipt extension (Setup Option 0x3A46, its value the extension version; on for the session when the client declares it too), and GRANT_CHALLENGE (0x3A47): 32 bytes drawn for this session, which a client presenting a room grant signs (below, Tokens and room grants).

Transports

Path	Listener	Selected by
Native QUIC	UDP quic_listen	ALPN moqt-22, moqt-18 or moqt-16
WebTransport over HTTP/3	the same UDP port	ALPN h3, extended CONNECT to path, WT-Available-Protocols
WebSocket fallback	TCP websocket_listen	upgrade at path with subprotocol moqt-22

Native QUIC and WebTransport share one quinn endpoint: the relay reads the ALPN of each connection and either runs MOQT on it directly or hands it to the WebTransport layer.

The WebTransport implementation is wtransport 0.7.2 (MIT OR Apache-2.0), vendored (below). It was chosen because it runs on the same quinn 0.11 as joule-moq, so one UDP endpoint serves both ALPNs; it gives the server the CONNECT request's headers and lets it answer with its own, which is what WT-Available-Protocols and WT-Protocol need; and its client checks a certificate by hash under the rules a browser applies to serverCertificateHashes (ECDSA P-256, a validity of fourteen days at most), so the relay's tests connect the way a page does. Stream bytes are read and written on the quinn streams underneath; application error codes on WebTransport streams are mapped into the HTTP/3 error space as draft-ietf-webtrans-http3-16 §4.4 gives.

Closing a WebTransport session. A MOQT session over WebTransport is closed by closing the WebTransport session (draft-ietf-moq-transport-22 §6.6). The relay sends a WT_CLOSE_SESSION capsule (0x2843) on the session's CONNECT stream, then a FIN (draft-ietf-webtrans-http3-16 §6): the MOQT session termination code (§12.2) is the capsule's 32-bit application error code as it stands, not mapped into the HTTP/3 space as stream codes are, and the reason, at most 1,024 bytes of UTF-8, is its message. The QUIC connection is closed with H3_NO_ERROR once the peer has closed its side of the CONNECT stream, or two seconds after the capsule, so the connection's close cannot overtake it. Nothing else of the session overtakes it either: a stream the relay lets go once the session is closing (the client's control stream of a refused session, say) is held until the QUIC connection has closed, rather than ended at once with a FIN or STOP_SENDING that would reach the page before the capsule; a page whose stream was stopped just ahead of the close was seen to lose its renderer. A page reads the code and reason from WebTransport.closed (closeCode, reason), as it reads them from the WebSocket binding's WT_CLOSE_SESSION. A session the client closes is read the same way: the relay's closed() gives the client's code and reason, and the session's summary records the code as the peer's. A QUIC connection that closes with the session still open gives no MOQT code; its code is an HTTP/3 error code.

The release of wtransport reads WT_CLOSE_SESSION but cannot send it, so the joule-code workspace vendors 0.7.2 with a session close added (vendor/wtransport, patched in with [patch.crates-io]; vendor/wtransport/VENDORED.md lists what changed and why).

How a browser connects

WebTransport, in development. Start the relay with --dev (or with no certificate files in the configuration). It makes a certificate a browser accepts by hash and logs its SHA-256 three ways, including a ready serverCertificateHashes entry:

const transport = new WebTransport("https://127.0.0.1:4443/moq", {
  serverCertificateHashes: [{ algorithm: "sha-256", value: new Uint8Array([/* printed */]) }],
  protocols: ["moqt-22"],
});
await transport.ready;          // transport.protocol === "moqt-22"

The certificate is valid fourteen days from the relay's start; restart the relay to make a new one. In production give it a certificate chain and key from files, and a page connects without serverCertificateHashes.

WebSocket, where there is no UDP path. A site behind a front that carries HTTP over TCP only reaches no UDP endpoint, so neither QUIC path is open to its pages. Such a page uses the fallback:

const ws = new WebSocket("wss://relay.example/moq", "moqt-22");
ws.binaryType = "arraybuffer";  // each message is one capsule, below

The WebSocket listener speaks plain HTTP; TLS is the front's.

The WebSocket binding (non-normative)

This binding is this crate's, not a standard's. It carries what a WebTransport session over HTTP/2 carries (draft-ietf-webtrans-http2-15), on a WebSocket instead of an extended CONNECT stream, so that a client written for one is most of the way to the other.

1. Opening. The client opens a WebSocket (RFC 6455) to the relay's path, offering the subprotocol moqt-22 in Sec-WebSocket-Protocol. The relay answers with that subprotocol, or refuses the upgrade (400 without it, 404 at another path).

2. Messages. Every message is a binary message holding exactly one capsule: Type (i) Length (i) Value, with QUIC variable-length integers (RFC 9000 §16), not MOQT's. A text message carries nothing and is ignored. A binary message that is not one whole capsule closes the session.

3. Capsules. Those of draft-ietf-webtrans-http2-15 §6, with their type codes:

   Capsule	Type	Value	Use here
   WT_STREAM	0x190B4D3C, 0x190B4D3B with FIN	Stream ID, data	stream data; a stream opens with its first WT_STREAM
   WT_RESET_STREAM	0x190B4D39	Stream ID, code, Reliable Size	reset a stream (Reliable Size 0)
   WT_STOP_SENDING	0x190B4D3A	Stream ID, code	ask the peer to stop
   WT_MAX_STREAMS	0x190B4D3F bidi, 0x190B4D40 uni	Maximum Streams	stream-count credit
   WT_STREAMS_BLOCKED	0x190B4D43, 0x190B4D44	Maximum Streams	sent when out of credit; informational
   DATAGRAM	0x00	payload	one MOQT datagram, reliable and ordered
   WT_CLOSE_SESSION	0x2843	code (32 bits), message (UTF-8, at most 1,024 bytes)	close with a MOQT session termination code
   WT_DRAIN_SESSION	0x78AE	empty	read and ignored; MOQT's GOAWAY is the drain
   others			read and ignored, as unknown capsules are

4. Streams. Stream IDs are numbered as QUIC numbers them: bit 0 the initiator (0 the client), bit 1 the direction (0 bidirectional), and the nth stream of a kind is 4n + kind. MOQT's control streams, request streams and data streams ride them byte for byte as they would ride QUIC streams: the client's control stream is its first unidirectional stream (ID 2), beginning with SETUP; a request is a bidirectional stream; a subgroup, fetch or padding stream is a unidirectional one.

5. Stream credit. Each end first sends its WT_MAX_STREAMS for both kinds; until the peer's arrive, an end opens nothing. Credit is cumulative, as in QUIC. A stream's credit comes back once both of its halves are done, and is granted in batches. Opening past the limit closes the session with code 0x045D4487. Because MOQT -22 bounds requests in flight by stream limits alone (§6.4.2), the relay's bidirectional limit (max_requests) is its bound on requests in flight on this binding, as MAX_STREAMS is on QUIC.

6. Data flow control. None at the capsule level (WT_MAX_DATA, WT_MAX_STREAM_DATA and their BLOCKED capsules are read and ignored): TCP's backpressure bounds a sender, and a receiver that holds more than websocket_buffer_bytes unread closes the session with 0x045D4487.

7. Closing. WT_CLOSE_SESSION with a MOQT session termination code (§12.2), then the WebSocket's close. A WebSocket that closes without one is a session closed with code 0.

8. Keeping alive. Each end the relay runs, accepting or dialling, sends a WebSocket ping (RFC 6455 §5.5.2) every 30 seconds, and the peer's WebSocket answers each with a pong. A front between the two that closes a connection it sees idle (a tunnel, a proxy) then keeps a session that has nothing to carry, such as a relay's link to its upstream while no one watches.

The first capsules a relay with the default limits sends are its credit:

99 0b 4d 3f 02 41 00     WT_MAX_STREAMS (bidirectional), 256
99 0b 4d 40 02 44 00     WT_MAX_STREAMS (unidirectional), 1024

and a session it closes at the end of a GOAWAY's timeout ends with:

68 43 12 00 00 00 10 47 4f 41 57 41 59 20 74 69 6d 65 6f 75 74
                         WT_CLOSE_SESSION, GOAWAY_TIMEOUT (0x10), "GOAWAY timeout"

What the relay does

-22	Here
§6.3 session initialization	Each end's control stream opens with SETUP; data streams that come first wait for it. A client's SETUP is held to §9.1's rules for a client (no PATH or AUTHORITY over WebTransport; no DELETE or USE_ALIAS token). The relay's control stream carries only its SETUP and a GOAWAY; the end of the peer's closes the session with PROTOCOL_VIOLATION, once nothing of the peer's publishing is left (each track it published ended, every data stream of its read) or after three seconds, so a publisher that closes its control stream as it leaves loses no object it sent.
§6.4.2 requests	A request stream must begin with one of the seven requests. A Request ID of the wrong parity or seen before closes the session with INVALID_REQUEST_ID. MAX_REQUEST_UPDATES is enforced per stream; requests in flight are bounded by the stream limit (max_requests).
§8.9, §9.1.3, §9.1.4 tokens	A receiver's alias cache per session: REGISTER, DELETE, USE_ALIAS, USE_VALUE; AUTH_TOKEN_CACHE_OVERFLOW, DUPLICATE_AUTH_TOKEN_ALIAS, UNKNOWN_AUTH_TOKEN_ALIAS; a registration in SETUP past the cache reads as USE_VALUE. Grants name a namespace prefix, rights to publish and subscribe, the SHA-256 of a token of type 0, and whether it is a room grant honoured only beside a proof of possession (Tokens and room grants).
§4.1, §4.2 namespaces	PUBLISH_NAMESPACE is accepted (REQUEST_OK) and stands until it is cancelled or its session ends. SUBSCRIBE_NAMESPACE answers with NAMESPACE for each published namespace under the prefix, then NAMESPACE and NAMESPACE_DONE as they change, each carrying the suffix after the prefix; an overlapping prefix in one session is PREFIX_OVERLAP.
§7.6, §9.8 PUBLISH	Accepted with REQUEST_OK; its Track Alias is the publisher's for its data. A second track under one alias in one session closes it with DUPLICATE_TRACK_ALIAS. PUBLISH_DONE ends the track's downstream subscriptions with its status, once the relay has read as many of the publisher's data streams as its Stream Count says it opened (they and the request stream are not ordered with each other, §9.9), or after three seconds, or at once when the count is 0 or unknown or the publisher's session is gone. A publisher's session that ends ends its tracks from their own request streams: with the status of a PUBLISH_DONE already read, else GOING_AWAY. A track its publisher ended with TRACK_ENDED and another then publishes is a new instance: what the relay held of the ended one (its cache, its Largest Location, which upstream each subgroup came from) goes, unless the DVR store keeps the track.
§7.4, §7.6, §9.6 SUBSCRIBE	SUBSCRIBE_OK only once an upstream subscription is Established (§7.4): one the relay holds, or a SUBSCRIBE it sends to every publisher of a matching namespace. The relay's own upstream SUBSCRIBE asks for every object at priority 128 with no timeout, copying no subscriber's priority, group order or timeout (§5.1.3, §9.20.3, §9.20.4). RENDEZVOUS_TIMEOUT holds a SUBSCRIBE for a publisher, up to max_rendezvous_ms; without it a track with no publisher is DOES_NOT_EXIST. Location Filters set where the subscription starts and ends. A publisher of a namespace that arrives later is sent a SUBSCRIBE for every subscription the relay holds under it. When the last subscriber leaves, the relay cancels its own upstream SUBSCRIBE.
§9.5 REQUEST_UPDATE	Subscriber priority (re-applied to what is queued), FORWARD, the Location Filter and both delivery timeouts.
§3.2, §3.4, §7.1, §9.11 FETCH and fills	A fetch the cache covers is served from it, groups in the requested order (coverage, below); otherwise from the cache and the DVR store together where they cover it (The DVR store, below); otherwise it goes to a publisher of the track or its namespace and its fetch stream is piped through, past the Fill Timeout a Timed-Out range. A range that starts after the Largest Object is INVALID_RANGE; one that starts below the live edge needs the seek power (Tokens and room grants). A SUBSCRIBE with FILL_PARAMETERS gets its fill range the same way on a fill fetch stream named by its Request ID; a fill it cannot serve is a fetch stream reset after its header (§3.4.1).
§9.13 TRACK_STATUS	Answered from what the relay holds: the Largest Object and the track's properties, from the DVR store when the track's publisher is gone (or before a restart).
§7.3 several publishers	A track may have several upstreams; each (group, subgroup) is taken from the first that sends it, and the rest are counted as deduplicated.
§5.1 priorities, §5.2 delivery timeouts	Below.
§6.6.1, §9.2 GOAWAY	On shutdown every session gets GOAWAY with goaway_uri and the timeout; new requests are refused with GOING_AWAY; a session still open at the timeout is closed with GOAWAY_TIMEOUT.
§6.5 .session	No session-level track is defined: a request for one is DOES_NOT_EXIST.

What the cache covers

The cache answers a fetch alone only where it holds every object of the range, so that a gap in its answer means an object that does not exist (§3.2.2). It keeps what it knows of each group's arrival. A group is whole when its start arrived (Object ID 0, or a Prior Object ID Gap saying the objects before do not exist, §10.9), none of its subgroup streams is open or was reset, and its end is known: its streams ended with a FIN, or an End of Group or End of Track status, or a datagram with END_OF_GROUP, ended it. A fetch from a group is covered when every group from it up to the group of the Largest Object is whole or known not to exist (a Prior Group ID Gap, §10.8), that last group has its start, and nothing in between was evicted. Coverage is worked out from the groups held each time it is asked, so a group whose streams finish after a later group's still counts once it is whole; a group ID never seen is a gap the cache does not answer across. The relay cannot tell a subgroup it has not yet seen from one that does not exist, so a group whose subgroups have all closed counts as whole.

A fetch that is not covered only because groups are still arriving waits for them: while the track's publisher is still sending and nothing from the fetch's start was evicted, the relay checks again every 20 ms, for up to the fetch's fill budget or one second, whichever is less, before it asks upstream. A track of one group per object, each on its own stream (a page's audio is a group every 20 ms), always has a stream or two in flight, and a publisher by PUBLISH need not answer a FETCH at all.

A media group and its receipts group leave the cache together, since a group served without its receipts could not be verified. Receipts are cached just before their media, so the cache's budget reaches them first. Evicting either moves the other track's mark to that group as well, even when the other's group has not arrived yet, so a late member of the pair is never served from the cache alone.

Priorities and delivery timeouts

Each downstream session has a scheduler. Every data stream it will send (a subgroup, a fetch or fill, a datagram) is a lane whose items go out in order and one at a time, so no stream's objects are reordered (§7.4). Among the lanes with something to send it takes the item §5.1.2 puts first: the lower subscriber priority number; then the lower publisher priority number; then, within a subscription, the group its Group Order puts first (and in one group a fill before its subscription); then the lower Subgroup ID or, for a datagram, Object ID, a datagram before a subgroup object. At most write_concurrency writes are in flight, and a QUIC session's send window is quic_send_window_bytes: what waits, waits in the scheduler, where it can still be ordered and dropped (§5.2: "minimize the amount of data buffered at the underlying transport layer"). Each stream is also given a transport priority from its subscriber and publisher priorities.

An object's deadline is its OBJECT_DELIVERY_TIMEOUT from when it reached the relay: the smaller of the publisher's (the first object of its subgroup, else the track's) and the subscriber's, where both are set. It is checked before the object goes to the transport and bounds both opening its stream and writing it. Past it the stream is reset with DELIVERY_TIMEOUT and nothing more of that subgroup goes to that subscriber; a datagram is dropped. SUBGROUP_DELIVERY_TIMEOUT runs from the subgroup's FIN until all its data is acknowledged, and resets the stream if it is not. On the WebSocket binding a FIN counts as acknowledged once written, as webtrans-http2 §5.2 makes the transitions an acknowledgement would make at once.

A subscriber that does not read

QUIC's connection-level flow control is shared by every stream of a session (RFC 9000 §4.1), and bytes a subscriber has received but not read hold it. So yes: a subscriber that stops reading stalls its own high-priority track along with everything else, until it reads again. And a subscriber that reads its streams one at a time, in the order they arrive, can stall itself while still reading: the stream it waits on may have its bytes held back for credit that the unread streams behind it are holding. A subscriber reads its streams concurrently.

The relay cannot release credit the subscriber holds, and does not try to by dropping what has no deadline. What it does:

• An object with no delivery timeout (a base track, a receipts track) is never dropped. It waits, in the scheduler or the transport, and is delivered once the subscriber reads.
• What can still wait, waits in the relay rather than in the transport (§5.2: "minimize the amount of data buffered at the underlying transport layer"). Writes in flight are bounded by write_concurrency and the transport's unacknowledged bytes by quic_send_window_bytes. So an object whose deadline passes while it waits is dropped before it takes any credit.
• An object that passes its deadline is reset. A reset stream's final size settles its account (RFC 9000 §4.5), and a receiver can credit its unread bytes back to the connection at once, as quinn does. The credit a dropped layer held then comes back without the subscriber reading it.
• When credit frees, the highest-priority bytes go first: each stream carries its subscriber and publisher priority as its transport priority.

MOQT lets a publisher end a subscription whose subscriber falls too far behind, with PUBLISH_DONE TOO_FAR_BEHIND (§5.2); this relay does not yet.

Receipts

A media track's receipts ride its receipts track (WAI extensions/moq-streaming-format.md §4.1): group g of it is one object holding the group receipt and the object receipts of media group g. The relay recognises a receipts track by its name: the media track's name followed by receipts_track_suffix (_receipts unless configured, the name WAI extensions/live-rooms.md §6.1 gives a room's receipts tracks: 0_receipts for layer 0). Each object of it is the group's receipts in the binary form of WAI extensions/jwp-receipts.md §2.7.3 (WJR1): the group receipt as its canonical bytes, its link hash computed over them as received, then the object receipts. The door reads that form, and refuses a receipts object that does not read (§2.7.4) as it refuses one that does not verify.

• The receipts of dropped objects are forwarded. A receipts track has no delivery timeout, and the relay applies none to it, even one a subscriber asks for. So when an object of a media track passes its deadline, its receipt still reaches every subscriber of that track's receipts track, which can account for the object by inclusion (WAI extensions/jwp-receipts.md §4.1, §4.4). A receipt that rides on the object itself, as a property, is lost with it.
• Elsewhere receipts are opaque. Outside the namespaces a door rule names, the relay forwards receipts tracks as it forwards any track and reads nothing in them (§7.8).
• At the door. In a namespace a [[receipts.door]] rule names, the relay reads the receipts it forwards, and takes on jwp-receipts §4.1's duties: each group receipt's signature is verified, under the rule's trusted keys or any key, before the first object of its group is fanned out, and a group that does not verify is dropped with a structured error in the log; each media object must match the receipt of its position (its group, its Object ID, its content hash, its Merkle proof to the signed root), and an object that carries its receipt as a property must carry that same one. An object waits for its group's receipts up to hold_ms, and its stream is not read meanwhile; a publisher whose receipts follow its group's last object can finish the group only while the whole group fits the stream's flow-control window, limits.quic_stream_window_bytes (16 MiB by default). What bounds the bytes one session leaves unread is not that window but the session's, limits.quic_connection_window_bytes, over all its streams together (64 MiB by default, the budget websocket_buffer_bytes gives a WebSocket session, and never below the stream window): a peer that opens many streams and finishes none of their groups stalls there, however many streams it opens. Both windows hold as well for the links the relay dials to its upstreams, over native QUIC or WebTransport (tests/upstream_window.rs). Receipt bytes are forwarded unaltered and nothing is re-signed. A refused object ends its downstream streams rather than leaving a hole in them.

Load

examples/fanout_load.rs measures the fan-out: one publisher, N subscribers of one track over native QUIC, and a bystander on a track of its own, whose lateness is what the audience costs everyone else on the relay. It reports the relay's resident memory and CPU, how many objects arrived, how late, and the packets the kernel dropped at each end. A fan-out delivers from a snapshot of its track's subscribers, outside the relay's lock, so a large audience holds up no other track. On Linux the QUIC listener binds limits.quic_sockets UDP sockets on its one port (one per core up to 16 by default), each its own endpoint, and the kernel steers each packet by its connection ID, so a viewer whose address changes keeps its connection. A deployment sets net.core.rmem_max to at least limits.quic_socket_buffer_bytes (4 MiB by default): the relay asks for that, Linux quietly gives no more than its limit, and the relay logs a warning. An object is held once from when it is read until the last subscriber's QUIC stream has it acknowledged: the cache, each session's queue and each QUIC send buffer share one allocation. A WebTransport or WebSocket session, and a session bridged to an earlier draft, has its own copy in its transport.

Tokens and room grants

[[auth.grants]] rules name a token by the SHA-256 of its value (Token Type 0), a namespace prefix, and the rights it gives there. With auth.required, a request is served only where a token of its own or of the session's SETUP matches a rule.

A rule with proof = true names a room grant: the token's value is a WRG1 record (WAI live-rooms §4.1). Every participant of a room holds every grant of it, to verify the room, so the record proves nothing about who presents it, and the relay honours it only beside a proof of possession of the grant's principal's key (live-rooms §10.5.2):

1. the relay's SETUP carries GRANT_CHALLENGE, drawn for the session;
2. the client waits for it, then sends its SETUP with the grant as an AUTHORIZATION TOKEN and GRANT_PROOF (0x3A49): a WRP1 record naming the grant's digest and the principal's key, over the challenge and the context SETUP //<authority><path> of the URI it addressed, signed by the principal;
3. the relay checks it (live-rooms §10.5.4: the grant named, the key the grant's principal — read from the record at its fixed place, since the rule names these exact bytes — the signature under WAI's strict Ed25519 rule, the session's own challenge, and the target it received: a WebTransport CONNECT's :authority and :path, a WebSocket upgrade's Host and path, native QUIC's AUTHORITY and PATH) before it serves any request;
4. a SETUP that presents such a grant without a proof that passes is closed with UNAUTHORIZED (§12.2), its reason (pop_missing, pop_wrong_key, …) in the log; a grant first presented in a request, never proved in SETUP, gives that request nothing (REQUEST_ERROR UNAUTHORIZED).

The relay keeps no consumed challenges: one is drawn for each session, so a proof made for another session is refused as pop_wrong_challenge. A rule without proof is a bearer token, for a key the relay's operator gives a party directly (a room's clock, an ingest gateway's link).

A room's authority. A room whose grants are issued as visitors arrive (a room service such as joule-moq-room) cannot be listed rule by rule. [[auth.authorities]] names the authority's key instead (and, optionally, the room ids it is honoured for): a token of type 0 that is a WRG1 record whose signature verifies strictly under that key (live-rooms §4.2; joule-moq's GrantHead) is a room grant, honoured only beside a proof as above and only within its window (nbf to exp), and mapped onto rights as live-rooms §10.2 maps it: PUBLISH under [R, P, m] for each media kind m its publish bits allow, of [R, P] (its catalog) when it may publish any, under [R, P, events] when its policy permits an event kind, and under [R, P, world] when its attribute world is act (its world track, WAI live-worlds §5.2); SUBSCRIBE, FETCH and the rest under [R] when it may subscribe. The relay's rights are per namespace, so §10.2's track matches (layers up to max_layer) are not narrowed. A session ends, EXPIRED_AUTH_TOKEN, when the earliest such grant it proved expires.

Revocation (live-rooms §5). [[auth.revocations]] holds an authority's revocation list, as its key log's head lists it in revoked_modules: grant digests (§4.2) and viewer tags (SHA-256 over wai:live-viewer, 0x01 and the grant's 16-byte viewer), 64 hexadecimal digits each. A grant whose digest or whose viewer's tag its authority's list holds is no grant: a SETUP that presents it is closed UNAUTHORIZED, its reason live-rooms §5: grant_revoked, before its proof is read. The relay reads no key log itself; the operator gives it the lists, in the configuration, the grants file or on the operator's surface (below), where a revocation also ends the sessions that presented the grant, in their SETUP or in a request, proved or not. It refuses every use: a rule that names the grant gives it nothing, a request that presents it is refused (REQUEST_ERROR UNAUTHORIZED, its reason live-rooms §5: grant_revoked), on a relay that requires no authorization too, its session standing under what else it holds, and a list holds when its authority is dropped. So that a revocation reaches a session by its requests, the relay keeps every token of theirs a rule names, and 64 KiB of records no rule names (relay::PRESENTED_BYTES); a request past that is refused UNAUTHORIZED. A list only grows (§5: "a revocation stays"): a reload, or a set that does not repeat it, leaves it on the list, and control.revocations_path carries it to the next start.

An authority's grant whose funds ceiling and currency break live-rooms §4.3 step 5's rule (funds_ceiling is 2^64 − 1 exactly when currency is three zero bytes, and otherwise three capitals) is no grant either: a SETUP that presents it is closed UNAUTHORIZED, its reason live-rooms §4.3: grant_invalid.

Seeking back (live-rooms §10.6). The live edge of a track is the group of its Largest Object. A FETCH whose range starts below it, or a SUBSCRIBE whose fill does, seeks back, and with auth.required it is served only to a session holding the seek power: an authority's grant whose powers hold bit 4 (§3.3), proved as above, or a [[auth.grants]] rule with subscribe and seek. Any other is refused REQUEST_ERROR UNAUTHORIZED with the reason phrase seek_not_granted. A FETCH that starts at the live edge, as a page joining the group in progress does, needs nothing more. The relay's seek_refused counter counts the refusals.

Age ratings (WAI age-assurance §3.6, §5.5, §5.7). A rating claim reaches the relay as the property WAI_RATING (0x3A4D, its value the claim's text): a Track Property of a PUBLISH or of an upstream SUBSCRIBE_OK, or an Object Property of an object it forwards. From the instant it reads one, the relay holds it where the extension lets it honour it: a claim about a room signed by an authority it trusts for that room; a claim about a track (from a group on, or a span of groups) signed by such an authority or by the track's owner, the principal its namespace names, and carried on that track, or carried anywhere in the room by the authority; an object's claim it passes over. It keeps at most 64 claims a room or track and 65 536 in all, forgetting the claims of rooms it no longer carries first. A session's threshold in a room is the greatest age attribute (§5.4) of the room's grants its SETUP proved; 0 otherwise. A group's age is the greatest age of the claims that cover it, and the relay serves the group to a session only when the threshold reaches it: a live object of a group above it is not forwarded; when a newly held claim raises a group above a subscriber's threshold, the streams of the groups it covers are ended for that subscriber (CANCELLED) and nothing more of them is sent; a SUBSCRIBE that forwards objects, a FETCH, or a fill whose first group is above it is refused, REQUEST_ERROR UNAUTHORIZED with the reason phrase below_threshold; and one whose range reaches such a group later is answered up to the group before it. The counters age_refused, age_withheld and ratings_held count the refusals, the objects not forwarded, and the claims held.

The room's own service is not held to the rule (§5.5): a session whose SETUP carries a GRANT_PROOF naming no grant (32 bytes of zero) by a room key, over its challenge, is served every object of that room. The relay checks the proof as a grant's proof (live-rooms §10.5.4, the zero digest for the grant and the proof's key for the principal) and closes the session UNAUTHORIZED for one that does not pass. A key it holds as a room key is an authority's of [[auth.authorities]], for the rooms that authority is honoured for, or one [floor] room_keys or [dvr] room_keys names, for every room; a proof that passes by any other key gives nothing. The client presents such proofs with ClientOptions::room_keys; the room service's clock does, so it follows and logs a rated room's tracks.

Patterns. A field * in a rule's prefix, or a door rule's, matches any one field: a room's clock token can be given ["*", "<clock>", "events"], its events track in every room. A door rule's trust_field names the namespace field holding the one key it accepts there: prefix = ["*", "*", "video"] with trust_field = 1 holds each publisher's video in a room to its own key (live-rooms §6.1), whoever joins; ["*", "*", "world"] does the same for each actor's world track (WAI live-worlds §4.3).

Limits per session. limits.max_session_ms closes a session at that age (NO_ERROR, the limit as its reason); limits.max_ingress_kbps closes one that sends more object bytes than that rate allows over ingress_burst_ms (INTERNAL_ERROR). A session whose SETUP presents a token whose rule says unlimited = true is exempt from both: a deployment's own long-lived sessions, such as a room service's clock. limits.max_resident_mb makes the binary exit with status 70 past that resident memory, for its supervisor to start it again.

Grants that change while the relay runs

The grants in force ([auth] and [[metering.grants]]) change while the relay runs, and a change ends no session but those that presented a grant it withdraws:

• The operator's surface ([control], src/control.rs): a Unix socket (socket), made mode 0600, on which the relay serves a peer of its own user or root; or a TCP address on loopback (listen), on which it serves a peer on loopback. A peer of another user, or one off loopback, is answered with a refusal, and a listen address off loopback stops the relay at start. On a host with no Unix sockets, listen is the surface, and socket stops the relay at start. A request is one JSON object a line, its answer one a line; a line that does not read as a request is answered and ends the connection:

  op	fields
  show		the grants in force, and the grants this hop holds past their joule ceiling
  set	auth (required, grants, authorities, revocations?), metered	these grants in place of those in force. It states what it replaces: one that leaves out any of them but revocations, which only add, does not read and changes nothing
  grant	grant	add an [[auth.grants]] rule
  withdraw	token_sha256	withdraw every rule and metered grant naming the token, ending the sessions that presented it
  authority / drop_authority	authority / key	honour an authority's grants, or stop
  revoke	authority, revoked	add to an authority's revocation list
  meter	grant	add a [[metering.grants]] entry
  reload		read the grants file again
  flush		close the window open now (below), wait flush_grace_ms, seal it
  record	room, t0_unix_us, window?	the room's per-hop record over a sealed window, or over the sessions that ended

• The grants file (control.grants_path): [[grants]], [[authorities]], [[revocations]] and [[metered]] in the forms of [[auth.grants]], [[auth.authorities]], [[auth.revocations]] and [[metering.grants]]. The relay reads it at start, its entries beside the configuration's, and again on SIGHUP or a reload: the rules, authorities and metered grants in force are then the configuration's and the file's, in place of every change made since. Whether authorization is required stays as it stands, and every revocation stays. A file that does not read, or a change that does not check, changes nothing.

• The revocations file (control.revocations_path): the relay appends each revocation it takes while it runs to this file, in [[revocations]] form, before it is in force, and reads the file at start; a file that does not read stops the start. Where the file cannot be written the revocation is put in force all the same, and the answer says it will not outlive this run. With none named, a revocation given while the relay runs lasts until it stops.

• The library: Relay::set_grants, add_grant, withdraw_token, add_authority, remove_authority, revoke, add_metered_grant and reload_grants, and auth_config and metered_grants for the grants in force (Relay::config is the configuration the relay started with).

A change is checked whole and put in force at once: a new session and a new request are held to it, and so is every session open then. One that presented a grant the change no longer honours (in its SETUP, or a rule's token in a request) or revokes, one whose SETUP token no metered grant maps any longer, or one that holds a subscription or a publication its tokens no longer give it (a rule moved to another prefix, a right removed), is closed UNAUTHORIZED, its reason beginning grant_revoked, and every other session goes on, in every room, with no GOAWAY. A session that loses a right it does not use goes on. A request admitted before a change and placed after it, as a subscription that waited for its upstream is, is held to the grants in force as it takes its place. A session that goes on runs, from the change, under the grant its SETUP token maps to then. The limits a session was given at SETUP stand, and the floor and the DVR store hold the room keys they started with.

An authority's grants metered with no configuration. A session that proved an authority's grant and whose token no metered grant names runs under that grant, read from the grant itself: its digest, its joules_ceiling, funds_ceiling and currency (live-rooms §4.1, joule-moq's GrantHead), which its summary and the room's ledger name.

A session's grant in each room. A session that holds grants for several rooms runs in each under that room's. An authority's grant states its room, and so does a token that is a room grant's record and that a metered grant maps, with no authority configured: the metered grant names exactly those bytes, so it vouches for the room they state (a relay that lists each grant by a rule with proof and a metered grant). A token that is no record states no room: a session runs under it in each room none of its grants names, and of two such tokens under the first. A room's ledger therefore counts a session against that room's grant, or against a grant of no room, and never against another room's.

Ceilings during the run (live-metering §4.6). A meter holds a use to its ceilings through the room's own records, and this surface gives it what it needs while the sessions run: flush and record state each grant's bytes and time in the room's ledger over a window, and revoke or withdraw end a use. A funds ceiling is held by the meter that prices those records: the relay holds no tariff and prices nothing, so it states each grant's funds_ceiling and currency in the summary and the room's ledger, beside the bytes and time they are priced on, refuses a grant whose two break live-rooms §4.3, and ends a use when the meter withdraws or revokes its grant. With control.hold_joule_ceilings the relay also holds the joule ceiling itself, on its own reading: at each seal it shares its figure for a room it served alone among the room's grants by the bytes each moved, and a grant whose shares here pass its ceiling has its sessions closed UNAUTHORIZED, reason joule_ceiling_passed, and its next SETUP refused the same. A hop's share is part of what §4.6 sums over every hop, so a grant past its ceiling on it is past it. Nothing seals by itself but what control.flush_interval_ms asks for, so a relay told to hold ceilings without that interval does not start. A window with no figure is shared nothing, the shares count from the relay's start and are held in memory, so a restart counts them from zero, and show lists the grants held. It ends the hop's service to the grant; the room's records say when the principal leaves.

Cascading: one room across relays

A relay can carry a room's tracks to and from other relays, so a room spans relays in different places: the home relay behind an HTTP tunnel that carries WebSocket alone, say, and an edge relay with public UDP. draft-ietf-moq-transport-22 leaves how relays interconnect out of scope (§7) and says namespace publication "does not protect against loops" (§4.1); what the relay does is below, and WAI live-rooms.md §10.7 is what a room needs of it. The module is cascade.

Links. Each [[upstream]] entry is a relay this one dials, as a client:

[[upstream]]
name = "home"
url = "wss://home.example/moq"      # moqt://host:4433/moq (QUIC), https://host:4433/moq (WebTransport)
transport = "websocket"             # "quic", "webtransport" or "websocket"
prefixes = [["*"]]                  # the namespaces routed there; a field "*" matches any one field
native_roots = true                 # or ca_path = "<PEM file>", or cert_sha256 = "<hex>" (pinned)
token_sha256 = "<hex>"              # the token presented in SETUP, named by its SHA-256 ...
token_path = "/etc/joule-moq-relay/home.token"   # ... and read from this file
relay_key = "<hex>"                 # optional: the relay id the upstream must declare

A pinned certificate is trusted by the SHA-256 of its DER alone, whatever its validity or names, its handshake signature still verified with its key. The token's file holds its value (trailing white space removed), and the relay does not start when it does not hash to token_sha256. The session is a MOQT session at -22 in which each end publishes and subscribes; it is redialed after it ends, waiting limits.upstream_reconnect_ms (1 s) and doubling to 30 s. Either relay can dial the other: an edge relay dials the home relay's wss:// endpoint through its tunnel, or the home relay dials an edge's UDP port over QUIC or WebTransport.

Relay ids. Every relay's SETUP declares the Setup Option RELAY_ID (0x3A4F): its id, the public key it signs its session summaries and per-hop records with. A relay that dials proves it: its SETUP carries a GRANT_PROOF naming no grant, by that key, over the other's GRANT_CHALLENGE and the target it addressed, which the other checks as it checks a room key's proof (Tokens and room grants); a declared id without its proof closes the session UNAUTHORIZED (relay_id_unproved), and a relay's session to itself PROTOCOL_VIOLATION (relay_loop). Every other client ignores the option (§9.1). A session with both ids is a hop link, at -22 alone (the drafts the bridge speaks have no form for what follows).

Routes. A SUBSCRIBE, FETCH or TRACK_STATUS for a track that no session of this relay publishes (by PUBLISH, or under a namespace published by PUBLISH_NAMESPACE) is forwarded on the first link, in the configuration's order, that routes the track's namespace and whose session is open and not going away. It is forwarded once: the subscription it makes is the relay's own upstream subscription of the track (§7.4), cached and fanned out to every local subscriber as a publisher's is, and a RENDEZVOUS_TIMEOUT goes upstream with the time left of it. A link that refuses (TIMEOUT, INTERNAL_ERROR, GOING_AWAY, EXCESSIVE_LOAD) or does not answer is passed over for the next; when every link that routes the track is down, the request waits for one within its rendezvous, and is then refused REQUEST_ERROR TIMEOUT with the reason phrase no_upstream. A FETCH of a track the relay knows nothing of asks the upstream for its live edge first (TRACK_STATUS), so the range it forwards is the request's own and the seek power is held here (below).

Announcements. Every namespace published to this relay under a link's prefixes, by PUBLISH_NAMESPACE or as the namespace of a track a publisher sent by PUBLISH, is published on the link (PUBLISH_NAMESPACE), so the upstream's subscribers are sent here (§7.6), and is withdrawn when it goes. A SUBSCRIBE_NAMESPACE whose prefix overlaps a link's is forwarded on it, as each link's session opens, and the upstream's NAMESPACE and NAMESPACE_DONE, of the namespaces the link routes, are merged with this relay's own: one NAMESPACE when a namespace is first seen from either, one NAMESPACE_DONE when the last source of it goes.

No loops. On a hop link the Message Parameter RELAY_PATH (0x3A51) is negotiated (-22 §9.20: "negotiated via Setup Options"): every request one relay sends another carries the relays it has passed through, the sender last. A relay never forwards a request back on the session it came from, nor to a relay on its path, nor past limits.max_relay_hops (8); a namespace is published on no link to a relay its path holds. A request that arrives naming the relay that receives it is refused relay_loop, one past the bound relay_hops_exceeded (UNINTERESTED for a PUBLISH_NAMESPACE, DOES_NOT_EXIST otherwise), and one that could only be sent toward a relay on its path is refused DOES_NOT_EXIST relay_loop at once. RELAY_PATH on a session that did not negotiate it, or whose last relay is not the peer, closes the session PROTOCOL_VIOLATION. A relay-id path rather than a hop limit alone: the path stops a request at the first relay it would repeat and names the relay that closed the loop, and its length is the hop limit.

GOAWAY. An upstream's GOAWAY ends nothing downstream: the relay sends that session no new request (§9.2), opens a session to the GOAWAY's New Session URI (one of the link's own scheme; else the link's URL), subscribes there to every track it held through the old session (§7.6.1: what arrives on both is deduplicated, §7.3), publishes its namespaces there, and closes the old session. It does not pass the GOAWAY on to its own sessions. A GOAWAY from a relay that dialed this one stops new requests to it. The relay's own GOAWAY on shutdown names no URI on a session it dialed (a client's, §9.2).

What each relay holds to its own rules. The door checks what a peer forwards as it checks a publisher's: a tampered object one relay forwarded is refused at the next relay's door. A link runs under its token's rule: the home relay names the edge's token in [[auth.grants]] with publish, subscribe and seek over the routed prefixes, and unlimited; the edge grants its upstream the right to subscribe to what it routes there. Each relay holds its own sessions to their grants (rights, the seek power, a rated track's threshold); a hop link proves no room grant, so its threshold is 0 and a rated room is not carried across one. The floor and audience inputs are decided at the relay the performer or the author publishes to: an object or an input that arrived over a hop link is forwarded as that relay decided, and its events are read for their schedules, handovers, windows and commitments alone (WAI co-performance.md §8). The counters forwarded_upstream, no_upstream and relay_loops count the requests forwarded and Established upstream, those no upstream answered, and those refused for their path; each link keeps its dials, sessions, GOAWAYs, moved subscriptions and announcements (Link::stats).

The session summary

When a session ends the relay signs a summary of it with its own Ed25519 key (from summary.key_path, or made at start and its public half logged), keeps it (Relay::summaries), and appends it as one JSON line to summary.out_path if set. It holds, for the session:

• as a subscriber: objects and their bytes forwarded, every byte written to its data streams, objects dropped for their delivery timeout and how many of those were on a track whose receipts track the session also held, subgroups reset after their FIN, objects served by fetch or fill, objects still queued at the end, and the same per track, with each track's fetched bytes, objects still queued at the end and objects of Object ID above 0;
• the spans of its layer class in each WAI room it subscribed to: the highest layer of the room's media tracks it held, or 8 when it held none of them but other tracks of the room;
• as a publisher: objects and bytes received, objects refused at the door, objects deduplicated, and the same per track;
• relay_energy: {"acquisition": "unmetered", "joules_micro": 0}. A session's own share of the relay's energy is not read: the counter covers the whole relay process, in which sessions run together (a room's record carries the relay's figure where it can, below). The 0 is the unmetered marker of WAI extensions/energy-measurement.md §2, never a measurement of zero joules, and must not be read or shown as one;
• publisher_signed: where the door verified an object's receipt, the publisher's signed figure for producing it, summed under the acquisition class its leaf was signed with. These are the publisher's statements about the objects, not the relay's energy;
• room_grant, when the token the session presented in SETUP is one a metered grant names, or an authority's grant it proved: the room grant's digest and its ceilings;
• room_grants, by room, the grant it ran under there: an authority's grant it proved for that room, or a metered grant whose token is a record of that room;
• peer, when the session is a hop link: the other relay's id, hex.

The summary is relay-scoped telemetry (jwp-receipts §4.2), not a receipt of the publisher's. Its format is joule-moq-relay/session-summary/2; the signature covers joule-moq-relay:session-summary\x01 followed by the summary's JSON bytes, and SignedSessionSummary::verify checks it.

The per-hop session record

Relay::hop_record(room, t0_unix_us) signs WAI's per-hop session record, WMH1 (wai/extensions/live-metering.md §3), of the room whose id is room, over the sessions that ended, on the room clock of the room's t0. A track is the room's when its namespace's first field is the room id in hexadecimal (WAI live-rooms §6.1); it is a media track when its namespace ends audio, video or screen and its name is a layer 0 to 7, an events track when its namespace ends events and its name is 0, and another track of the room otherwise. The record holds:

• per media track and direction, and for the events tracks and the other tracks together: the objects and bytes received and sent, the objects dropped for their delivery timeout, refused at the door, deduplicated and still queued at the end, and those of the dropped whose receipts the relay forwarded;
• its subscriber sessions, their microseconds of each layer class, and the events it delivered to them;
• the root of its ledger, one entry for each room grant its sessions ran under (or none, for sessions with no grant): the sessions, the bytes in and out and the subscriber time;
• its energy over the span (below).

Its energy. The relay reads a counter the host keeps, every 100 ms while it runs, and keeps four hours of samples (src/energy.rs):

• on macOS, the relay process's own CPU energy, ri_energy_nj from proc_pid_rusage(getpid(), RUSAGE_INFO_V6), unprivileged: scope task, every thread of the relay and nothing else (not other processes, the GPU, the network interface, or work a system service does for the relay);
• on Linux, the CPU packages' RAPL counters through powercap, scope package: the whole package, every process on it. It is read only with [metering] dedicated_host = true, the operator's statement that the host runs the relay alone, since it is otherwise not the relay's energy;
• elsewhere, or with [metering] energy = "off", none.

A record carries the counter's advance over its span as OnChipCounter, filtering state Unknown, with a declared uncertainty (the interpolation at each end of the span, the counter's lag at each end, for a package counter the noise a filtered counter's vendor states for the window, and the rounding to whole microjoules) over the span as its window, when three things hold: the counter was live across the span (WAI energy-measurement §3.1: it advanced in two consecutive intervals, in at least five, and in four of every five, over at least 300 ms); the relay served that room alone in the span (every session that overlapped it has ended and carried tracks of that room and of no other); and the figure is at least one microjoule. Otherwise the record carries the unmetered marker, and the relay logs why. A figure of the task scope covers the relay's CPU and nothing past it; a figure of the package scope covers the whole package of a host the operator stated was the relay's alone.

As a session closes, the relay ends its subscriptions one at a time, each when its request stream fails; the moments between are classed by what the session still holds, so a session whose media subscriptions end before its receipts or events ones has a span of class 8, typically microseconds, at its end. HopRecord::to_bytes is the record and HopRecord::ledger_bytes the ledger beside it. A hop link (Cascading) is counted in the track entries, its objects in at one end and out at the other, and is neither a subscriber session nor in the ledger (live-metering §3.10); every other session of the relay that subscribes is a subscriber session. Nothing in the record is per object or per viewer, so its size depends on the room's published tracks and not on its audience. tests/room_metering.rs checks that the record recomputes from the summaries.

Windows, with sessions open (live-metering §3.8). Relay::flush closes the window open now at that instant and opens the next, with no GOAWAY and no session ended, and from a relay's first flush each session's tally counts by window beside its whole. Until then there is one window, and its counts are the sessions' own, so a relay never flushed holds no second copy of them; the first flush puts everything counted so far in window 0. control.flush_interval_ms has the relay flush and seal by itself; without it a window closes only when asked. An object is counted in the window that was open when its group's first object reached the relay, so each track is cut at a group boundary: the rest of a group that began before the flush, arriving after it, is counted in the window it began in. Time on a room's tracks is cut at the flush's instant. Relay::seal takes every session's counts in a closed window, those of the sessions still open among them, a grace after the flush (flush_and_seal, and the surface's flush, wait control.flush_grace_ms); an object of a sealed window's group counted after that goes to the window open then, and an object served by FETCH or a fill is counted in the window open when it is sent (live-metering §3.8). What becomes of an object counted (its refusal at the door, its deduplication) stands in the window that counted it, sealed or not, so an entry's drops stay among its objects. Each count goes to the whole and to one window, so a session's windows sum to its summary. A relay with a DVR store states in a window's record the objects and bytes it has written since the window before, as far as the window's in entries count (§3.6, §3.9): what a window cannot state waits for the next with objects in. Relay::window_record(room, t0, window), and the surface's record, sign the room's record over a sealed window: its span is the window's, so records over successive windows do not overlap, and its energy is the relay's own over the window where it served that room alone in it, read from its own meter. The relay keeps its latest 64 sealed windows, and an ended session's counts in a window only until that window is sealed.

The DVR store

With [dvr] configured the relay keeps, on disk, a window of the tracks under each [[dvr.window]] prefix with their receipts, and serves it to FETCH below the live edge (WAI extensions/live-recording.md §8). Its module is dvr.

What it keeps. Every track a window's prefix matches (the rule with the longest prefix keeps it): media tracks, their receipts tracks, events tracks, catalogs, takes. A group is stored once the cache holds it whole (What the cache covers), with every field a FETCH carries. For a track the store keeps, the cache does not evict a group while a stream of it is still arriving, past its budget by that group if it must, and it takes a group whose streams finish after a later group was evicted (coverage is unchanged: nothing at or below an evicted group is covered), so the store gets every group whatever order its streams finish in.

What it never keeps. A room whose objects are sealed (WAI confidentiality.md: an end-to-end encrypted room) is never recorded, and a relay keeps nothing of its tracks beyond delivery (§9; live-recording §8.2). The store knows such a room by its first object whose properties carry the secure-objects Key ID (SECURE_OBJECT_KEY_ID, read without a key), or whose properties do not read. That object's group is never written. From then on the store writes nothing of the room and deletes what it held of it: the room's events and receipts tracks are in the clear and can be whole before its first sealed group. It also drops the room's decisions from decisions.wdl, while still honouring them for what the cache serves. The room is recorded in sealed.wdl (room(32) check(32), check the SHA-256 of room), so a restart keeps refusing it; rooms_sealed counts them.

A room in which a minor publishes (WAI age-assurance.md §8.4) is never kept either. The relay knows one by a minor's publishing grant: a session whose SETUP proved a grant of a configured authority whose publish is not 0 and whose attribute minor is true. From then on the store treats the room as a sealed one: it writes nothing of it, deletes what it held and its decisions, and records it in sealed.wdl, which so holds every room never kept; rooms_sealed counts these too. A minor's grant that publishes nothing changes nothing.

The window. Each [[dvr.window]] has a max_age_s, from when a group's first object reached the relay, of 1 to 86 400 (24 hours, the start-over buffer a live-to-VOD harvest clips from), and a max_bytes of payload over its tracks together. Past either the oldest group goes first, whole, and group g of a media track goes with group g of its receipts track. A group that becomes whole after the window has moved past its number is not stored (counted as groups_late).

On disk. Under path, one directory per track, tracks/<the first 16 bytes of the SHA-256 of its rendered name, hex>/, holding TRACK (namespace, name, properties, and a SHA-256 check) and the track's segments, s<n, 20 digits>.wds: files each group's record is appended to as it is stored, a new one begun every ten seconds or 16 MiB. A segment is a sequence of frames, len(u32) state(u8) record, the state 1 for a record that stands and 0 for one a later record replaced (a group written again: an object erased, or a subgroup that came late), whose bytes are then overwritten with zeros. A record is

header   "WDG1" 1 track(32) group(u64) arrived_us(u64) count(u32) erased(u32) check(32)
object   id(u64) subgroup(u64, 2^64-1 for a datagram) priority(u8) status(u8)
         props_len(u32) payload_len(u32) digest(32) check(32) props payload
erased   id(u64) digest(32) why(u8: 1 redacted, 2 did not verify)

Integers are little-endian. track is the SHA-256 of the rendered track; the header's check is SHA-256 over the header before it; an object's digest is the content hash its receipt names it by (jwp-receipts §2.1), and its check SHA-256 over its fixed fields before check, its properties and its payload. A segment is deleted once no group's standing record is in it, so on disk a window holds up to a segment of each track more than its budget. decisions.wdl beside tracks/ holds the moderation decisions the relay honoured, each a length, the WEV1 record and its SHA-256.

Durability. The writer syncs every segment it wrote, and the directories it made files in, at least once a second and before it answers a flush or stops; a shutdown flushes. A crash loses at most the last second's groups. (One sync per group could not keep up with a room's audio, a group every 20 ms per publisher.) On start the relay reads every segment again:

• a frame a crash cut short, at a segment's end, is cut off (torn_writes);
• a record whose header does not check is dropped;
• every object is checked: its check and digest; on a media track whose receipts track is stored, against group g's receipts (the group receipt's signature under the door's trust for the namespace, or any key that verifies where no door rule covers it; the object's receipt at its position; its Merkle proof to the signed root; its content hash); on an events track, every record under its principal. One that does not verify is dropped, counted (objects_dropped, and the WMH1 record's dvr_dropped), and its group written again with it among the erased. It is never served.

What is below the first group the store holds of a track after a restart is unknown to it. Each object's digest is checked again as it is read.

Answering a FETCH. A FETCH the cache covers is answered from it, as before. Otherwise, where every group of the range is held whole by the cache or the store, known not to exist, or evicted from the store, the relay answers from the cache and the store together, groups in the requested order, reading the store's groups as the fetch stream drains (about a mebibyte at a time). Otherwise it asks upstream. A FETCH of the receipts track brings each group's receipts the same way, so a viewer checks a recorded object exactly as a live one.

What is no longer there. Evicted groups are marked with an End of Unknown Range (0x10C, MOQT §11.4.1.2) through the last of them: a range partly evicted gets what remains and the marker, one wholly evicted the marker alone. Not an unmarked gap and not an End of Non-Existent Range: an evicted object existed, and an unmarked gap would say it did not (MOQT §3.2.2). An End of Range moves to a later group than the entry before it (its Group ID is delta-coded, as joule-moq reads -22), so an object of a group that cannot be served (dropped, withheld) is marked by one End of Unknown Range from the group's start through the last such object, and the group's objects after it follow.

Moderation. The relay reads the decision events (WAI live-events §4.11) of the events tracks it forwards in a kept room. It honours one whose record verifies, whose principal owns the track ([R, P, events]), and whose principal is a room key (dvr.room_keys, or a key of [[auth.authorities]]) or proved a grant with the moderate power on its publishing session. The last honoured decision on a target stands. Deny and redact withhold the target from every FETCH and fill, from the store and from the cache, for every session but a moderator's; redact also erases it from its group's file; allow restores what deny withheld. A media object is named by its receipt's digest, an event by its record's digest: a withheld event is left out of the events-track object that carries it, the other records byte for byte, and an object left with none is withheld whole. Live subscriptions are not filtered here (Not yet implemented).

Metering. A relay with a store signs WMH1 version 2 (live-metering §3.9): the objects and bytes its sessions were served from the store (dvr_objects, dvr_bytes in each summary's tracks), and from the store's own series, kept a point a second, the objects and bytes it wrote for the room, the bytes it held integrated over the span in byte-seconds, and the objects it dropped. Dvr::counts gives the store's counters, Dvr::figures a room's figures over a span.

Configuration.

[dvr]
path = "/var/lib/joule-moq-relay/dvr"
room_keys = []          # a room's authority and clock, hex; the authorities' keys count too

[[dvr.window]]
prefix = ["*"]          # a field "*" matches any one field
max_age_s = 900
max_bytes = 4294967296

The store must be kept where the room's region pin allows (live-rooms §3.7).

Configuration

wai-relay --config relay.toml, or wai-relay --dev for loopback with a development certificate. relay.example.toml shows every field: the listen addresses and path, the certificate files (or the names for a development certificate), the cache's byte budget, the limits (sessions, requests and streams in flight, MAX_REQUEST_UPDATES, the token cache, the rendezvous, upstream and fill timeouts, write concurrency, the QUIC send window, the GOAWAY timeout, the session, ingress and memory limits, the relay-path bound and the upstream redial wait), the receipts suffix and door rules, the summary's key and file, the grants, the room authorities and their revocation lists, the DVR store and its windows, the upstream links, and the operator's surface with its grants file and its flush interval. The configuration holds no secret: keys are files, and tokens are named by their SHA-256. JOULE_MOQ_RELAY_LOG sets the log level.

Not yet implemented

• SUBSCRIBE_TRACKS (§3.6): refused with NOT_SUPPORTED.
• Range Filters (§3.3.2): the relay declares MAX_FILTER_RANGES 0, so a request that carries one is INVALID_FILTER, and it evaluates no Track or Object Property filter.
• WebTransport over HTTP/2 (draft-ietf-webtrans-http2-15), the path a browser takes when HTTP/3 is unavailable. The WebSocket binding carries the same capsules and covers the same case for now.
• Parts of draft-ietf-webtrans-http3-16 the library does not speak: it negotiates with the settings deployed browsers send rather than -16's SETTINGS_WT_ENABLED, has no per-session flow-control capsules, and closes a session by closing its QUIC connection with the code rather than with WT_CLOSE_SESSION; and quinn 0.11 has no RESET_STREAM_AT, so a reset WebTransport stream is not guaranteed to deliver its stream header first.
• 0-RTT is not accepted.
• Migration as a client: the relay moves an upstream link's subscriptions after its GOAWAY (Cascading), but follows no REDIRECT, and a publisher's session that receives the relay's GOAWAY is not moved by the relay.
• Cascading: the per-hop record names no peer hop (a link is counted in the track entries, live-metering §3.10), so the chain is read by track, not by link; a rated room is not carried across a hop link (its threshold is 0); a FETCH piped through from upstream is checked against its receipts by the subscriber, not at this relay's door, as on any upstream fetch; routes are the configured prefixes, and a namespace an upstream names in NAMESPACE routes nothing by itself.
• Pausing upstream when every downstream subscriber is paused (§7.2), and forwarding NEW_GROUP_REQUEST upstream. EXPIRES is neither issued nor acted on.
• TOO_FAR_BEHIND: a subscription whose subscriber does not read is held, not ended (§5.2 allows ending it with PUBLISH_DONE TOO_FAR_BEHIND).
• A cancellation the relay cannot see: a subscriber that sends FIN on its direction and later only STOP_SENDING is noticed when the relay next writes on the request stream, or when the session ends.
• Receipts: the receipts track of a media track is found by the configured suffix, not by reading the catalog's wai_receipts_track; the relay signs no delivery-gap claim; datagrams at the door are checked against their group's receipts, not a property.
• Energy: a room's record carries the relay's own energy only where the relay served that room alone across the record's span; dividing the counter among rooms or sessions that ran together is not done here, and a session's summary always carries the unmetered marker. No figure is read on a host whose counter is neither the macOS task counter nor the RAPL package counter of a host stated to run the relay alone.
• The DVR store: decisions withhold from FETCH and fills, not yet from live subscriptions (a reviewed item's held state, live-moderation §6.3, needs the review policy); objects are checked against their receipts at a restart, and by the door as they arrive where a door rule covers them, not again as they are stored; no HLS or DASH presentation of the window is written (live-recording §8.6 is a mapping).
• Re-validation (moqt-reval, live-rooms §10.2): a grant is checked at SETUP and on each request, its expiry ends the session, and a revocation the operator gives ends its sessions (Grants that change while the relay runs); the relay reads no key log itself (§5).

Tests

All on loopback, every wait bounded:

Test	What it shows
relay_quic, relay_webtransport, relay_websocket	Two publishers (one by PUBLISH_NAMESPACE and the relay's SUBSCRIBE, one by PUBLISH) and three subscribers over each transport; every object verifies against its publisher's receipts, which the door also checks; SUBSCRIBE_NAMESPACE; every summary verifies and labels the relay's energy unmetered.
slow_subscriber	A base track at priority 0 and a layer at 200 with a 100 ms delivery timeout. A subscriber that reads nothing for 1.5 s, then reads all its streams concurrently, loses layer objects to the timeout (their streams reset with DELIVERY_TIMEOUT), keeps the whole base track, and receives every receipt of both tracks although it asked for a timeout on its receipts tracks; each dropped object verifies by inclusion; the summary counts each layer object as forwarded or dropped.
late_joiner	TRACK_STATUS; FETCH of a track and its receipts from the cache; a range in descending order; INVALID_RANGE; a fill on SUBSCRIBE; a group that reaches the relay after a later one, the FETCH from it still served from the cache; a FETCH forwarded upstream when the cache holds nothing.
goaway	GOAWAY with its URI and timeout to every session; GOING_AWAY for a new request; GOAWAY_TIMEOUT for the session that stays.
shutdown_ports	From a task on a runtime of one worker, the relay shut down and started again on the same ports many times, a session open over QUIC, WebTransport and WebSocket each time: when the shutdown returns its UDP, WebSocket and control ports bind at once, no session is open and every summary is in.
start_ports	From a task on a runtime of one worker, a start on fixed ports refused part way (the WebSocket port held, the control port held, a file where the control socket goes): when it returns, every other port binds at once, and a start with nothing held comes up on all of them; the control surface alone, refused at its socket, frees the TCP port it had bound.
auth_tokens	Tokens in SETUP, by value and by alias; UNAUTHORIZED; UNKNOWN_AUTH_TOKEN_ALIAS.
grant_proof	A room grant with its proof over each transport gives its rights, each session with a challenge of its own; the grant as a bare token, a proof by another key, one for another relay and one over an earlier session's challenge each close the session with UNAUTHORIZED; a grant presented only in a request is refused it.
request_ids	INVALID_REQUEST_ID for the wrong parity and for a repeat; requests in flight bounded by the stream limit.
relay_energy	On a host with a counter (macOS's task counter; on Linux, the package counter with dedicated_host): a room's record, after a publisher sends for over a second, carries OnChipCounter with its evidence, reads back, and verifies; with a second room's session in the span it carries the unmetered marker; with energy = "off" it carries the marker. A record over a window of sessions still open carries the relay's figure for the window, a session that counted nothing in it sharing nothing, and with a second room's objects in the window the marker. With hold_joule_ceilings and a flush_interval_ms, a grant whose ceiling its share passes has its session closed at the relay's own seal, with nobody flushing it, and its next SETUP refused, and a grant with no ceiling goes on; without it, nobody ends. Where the host has no counter the test says SKIPPED, and with JOULE_MOQ_RELAY_ENERGY_REQUIRE=1 it fails instead. The figure's arithmetic, its liveness rule and its uncertainty are src/energy.rs's unit tests, which run on every host.
dvr	A long publish crosses the window: the oldest groups go, whole, each media group with its receipts group, within the byte budget; a window by age empties as its groups pass it. FETCH inside the window (ascending and descending), across it (an End of Unknown Range through the last evicted group, then every object, from the store and the cache together) and before it (the marker alone); every object verifies against the fetched receipts. A restart keeps the store, drops and counts a flipped byte and bytes changed with their checks made again (only the receipt finds those), removes a cut-short write, and never serves what it dropped. A room key's redaction erases its target and withholds it, a deny withholds until an allow, a redacted event leaves the object that carries it, a decision by no moderator changes nothing, and the decisions hold across a restart. Without the seek power a FETCH and a fill below the live edge are refused seek_not_granted, and a FETCH at the live edge is served; with it, the window. The record of the room is WMH1 version 2 with the store's figures. A room whose objects are sealed is never kept: its first sealed group is not written, its events and receipts already held and its decisions are deleted, nothing more of it is written, and so after a restart, while a room in the clear beside it keeps its window. A room in which a session proves a minor's publishing grant is never kept from then on, likewise, while a minor's grant that publishes nothing changes nothing. A window past 24 hours is refused.
age_serving	A room rated 13 by its authority, the claim on the publisher's PUBLISH, and a stranger's rating ignored: a session at no threshold is refused the subscription below_threshold; the performer rates groups 5 to 7 of its track 18 on group 5's object, after group 4 went out, and from then the viewer at 16 is forwarded none of them while the viewer at 18 is forwarded all; a FETCH across the span is answered up to group 4 and one from inside it is refused; a SUBSCRIBE at a live edge rated 18 is refused.
age_room_service	A room rated 18 by its authority, with the floor held ([floor] room_keys the clock): sessions proving the clock's key and the authority's (a GRANT_PROOF naming no grant) are served every group, verified; the clock's token with no proof, and a proof by a key that is no room key, are refused the subscription below_threshold; a proof naming the clock's key under another key's signature, and the clock's proof over another session's challenge, close the session UNAUTHORIZED.
age_rising	A rating that rises while a subscriber's objects wait at the relay: a viewer at 18 that reads nothing, with small QUIC windows and the relay a small send window, holds groups 0 to 9 queued when the performer rates its whole track 21 on group 10's object; once the relay holds the claim the viewer is given no more of what waited (those streams ended, CANCELLED) and nothing of group 10, while a viewer at 21 is given every group.
priority_fan_out	Two publishers, four tracks, three subscribers with their own subscriber priorities and group orders, their transports held shut while the load arrives: each receives every object in §5.1.2's order.
drafts	The two-publisher scenario with every session at -18 (over QUIC and over WebTransport), at -16, and with publishers at -16 and -18 and subscribers at -22, -18 and -16: every object verifies against its publisher's receipts at every subscriber. FETCH from the cache at -18 and -16 (the drafts' exclusive End Locations, -16's absolute fetch IDs). A SUBSCRIBE with a Range Filter, refused at an -18 client by name. An -18 session and a -16 session written from those drafts' own messages, not through this crate's client: SETUP each way, a request answered, a Joining FETCH refused with NOT_SUPPORTED, an unknown Fetch Type closing the session with PROTOCOL_VIOLATION; a -16 Request ID ahead of one in flight on another stream admitted and one seen before closing the session with INVALID_REQUEST_ID. An -18 publisher's FIN on PUBLISH_NAMESPACE withdrawing the namespace, a -22 FIN not; an -18 publisher that closes its control stream before its last object and PUBLISH_DONE losing neither.
cascade	An edge that dials a home relay over native QUIC, WebTransport (both trusting a pinned certificate) and the WebSocket binding: a publisher on each, a subscriber of the other's track on each, every object verified at the subscriber against its publisher's receipts and by both doors; each namespace published both ways and discovered by SUBSCRIBE_NAMESPACE from either relay; each relay's summary of the link naming the other. Three relays in a chain, a room's track each way: a namespace's path; a per-hop record at each relay, each signed by its own key with its own unmetered figure, the in and out entries of each track chaining hop to hop, only the end viewers subscriber sessions, the middle relay's ledger empty. A home relay with no door forwarding a tampered object that the edge's door refuses. A loop of three relays: an announcement never returns to its origin, and a SUBSCRIBE for a track nobody publishes refused relay_loop at once. A link to an upstream that is down passed over for the next, and with none up TIMEOUT no_upstream, with and without a rendezvous. An upstream's GOAWAY naming another relay: the edge moves its subscriptions there, its subscriber sees no GOAWAY and loses no group. The home relay's DVR window fetched from the edge with the seek power, and refused seek_not_granted at the edge without it, by FETCH and by fill. A member's votes, sent out of order on the edge, decided once there in chain order, a watcher on the home relay given exactly what one on the edge is; and an object a relay with no floor forwarded not held to the floor again over the link.
live_control	Two rooms, a publisher and a subscriber in each under a token of its own: the first room's token withdrawn on the control socket closes its two sessions UNAUTHORIZED, grant_revoked, while the second room delivers every later group with no GOAWAY; a token granted at run time is served and the withdrawn one refused; a metered grant added at run time names the grant of sessions already open. A metered grant withdrawn, or left out of a set, ends the sessions under it with no rule naming its token, and a set that maps a token to another grant moves its session there. A session subscribed by a token its requests carry ends when that token is withdrawn, and goes on through a change that leaves it alone. A change that moves a rule to another prefix, or takes a right, ends the sessions holding what it no longer covers, and none that loses a right it does not use. A control listen address off loopback stops the relay at start; the socket is mode 0600; a peer of another user or off loopback is refused (a peer stated as another user on the socket on every host, and with root a peer that dropped to another user, said SKIPPED otherwise), and a connection from a non-loopback address of the host to the loopback listener is answered with a refusal (said SKIPPED where the host has no such address); an HTTP request on the listener ends its connection, its body unread. A set that leaves out required or metered changes nothing, and a reload leaves required as it stands. An authority's grant revoked by its digest or its viewer's tag is refused at SETUP before its proof; one revoked while its session runs ends that session alone; an authority's grant is metered from its own ceilings; one whose funds ceiling and currency disagree is refused grant_invalid. A revocation ends a session that proved nothing on a relay requiring no authorization, one a rule admits as well and one whose request presented the grant, with a rule naming it or on a relay requiring no authorization; a request presenting a revoked grant is refused on both, its session standing and left alone by a later change; and the list holds once its authority is dropped. A session's requests are refused past the bound on the records they may present. An authority added at run time is honoured, and one dropped ends the sessions that proved its grants alone. A session holding grants for two rooms is in each room's ledger under that room's grant, and under none in a third room's, over a window and over the sessions ended; on a relay holding no authority, by rules with proof and metered grants, a token that is no record names the third room. A revocation outlives a reload, a set that does not repeat it and a restart. A flush inside a group: the group's later subgroup, after the cut, counted in the window it began in and the next group in the next; the record signed on the socket over the first window, with every session open, read by this crate and by wai/live-conformance/live_metering_verify.py (said SKIPPED where the host has no Python 3.8; under CI, or with JOULE_MOQ_RELAY_VERIFY_REQUIRE, it fails instead); each session's windows summing to its summary, and the windows' records to the whole's. A relay never flushed holds no second copy of its sessions' counts, and its first flush puts them in window 0, an ended session's whole among them. An interval seals with nobody asking, and a relay told to hold a ceiling without one does not start. With a DVR store, groups cut by a flush and sealed before or after the store wrote them: each window's record reads, its stored figures within its in entries. SIGHUP reads the grants file again: the dropped token's session ends, the added one is served.
publish_done_streams	A PUBLISH_DONE that arrives before the data stream it counts, at -22, -18 (QUIC and WebTransport) and -16: the object still reaches the subscriber, then PUBLISH_DONE; a count no stream meets ends the track after the wait; 0 and the unknown count at once. A close right after PUBLISH_DONE ends the track with the publisher's status, not GOING_AWAY, and without the wait. A track published again after TRACK_ENDED is a new instance.

Licence

Apache-2.0.