open-standards/joule-code/joule-code-rs/crates/joule-moq-relay

joule-moq-relay

A relay for MOQT, draft-ietf-moq-transport-22. Publishers send it tracks, subscribers ask it for them, and it fans each object out to every subscription, in the order the draft's priorities give, dropping what passes a subscriber's delivery timeout at this hop. Whatever it drops, the object's receipts still reach the subscriber, on the track's receipts track.

It is a library (Relay, RelayServer, client::Client) and a binary, joule-moq-relay. The wire codec is joule-moq's.

The draft it speaks

draft-ietf-moq-transport-22, and no other. Drafts do not interoperate, so a session speaks one from its first byte. The draft is chosen before any MOQT byte:

• by the ALPN moqt-22 on native QUIC (§6.2.2);
• by WT-Available-Protocols: "moqt-22" on WebTransport, which the relay answers with WT-Protocol: "moqt-22" (§6.2.1; draft-ietf-webtrans-http3-16 §3.3). A client that sends no WT-Available-Protocols is taken to speak -22; one that offers other protocols only is refused;
• by the subprotocol moqt-22 on the WebSocket fallback (below).

The relay sends its SETUP as soon as a session opens, without waiting for the client's (§6.3 lets either end send first): it declares MOQT_IMPLEMENTATION, MAX_REQUEST_UPDATES, MAX_AUTH_TOKEN_CACHE_SIZE, joule-moq's receipt extension (Setup Option 0x3A46, its value the extension version; on for the session when the client declares it too), and GRANT_CHALLENGE (0x3A47): 32 bytes drawn for this session, which a client presenting a room grant signs (below, Tokens and room grants).

Transports

Path	Listener	Selected by
Native QUIC	UDP quic_listen	ALPN moqt-22
WebTransport over HTTP/3	the same UDP port	ALPN h3, extended CONNECT to path, WT-Available-Protocols
WebSocket fallback	TCP websocket_listen	upgrade at path with subprotocol moqt-22

Native QUIC and WebTransport share one quinn endpoint: the relay reads the ALPN of each connection and either runs MOQT on it directly or hands it to the WebTransport layer.

The WebTransport implementation is wtransport 0.7 (MIT OR Apache-2.0). It was chosen because it runs on the same quinn 0.11 as joule-moq, so one UDP endpoint serves both ALPNs; it gives the server the CONNECT request's headers and lets it answer with its own, which is what WT-Available-Protocols and WT-Protocol need; and its client checks a certificate by hash under the rules a browser applies to serverCertificateHashes (ECDSA P-256, a validity of fourteen days at most), so the relay's tests connect the way a page does. Stream bytes are read and written on the quinn streams underneath; application error codes on WebTransport streams are mapped into the HTTP/3 error space as draft-ietf-webtrans-http3-16 §4.4 gives.

How a browser connects

WebTransport, in development. Start the relay with --dev (or with no certificate files in the configuration). It makes a certificate a browser accepts by hash and logs its SHA-256 three ways, including a ready serverCertificateHashes entry:

const transport = new WebTransport("https://127.0.0.1:4443/moq", {
  serverCertificateHashes: [{ algorithm: "sha-256", value: new Uint8Array([/* printed */]) }],
  protocols: ["moqt-22"],
});
await transport.ready;          // transport.protocol === "moqt-22"

The certificate is valid fourteen days from the relay's start; restart the relay to make a new one. In production give it a certificate chain and key from files, and a page connects without serverCertificateHashes.

WebSocket, where there is no UDP path. A site behind a front that carries HTTP over TCP only reaches no UDP endpoint, so neither QUIC path is open to its pages. Such a page uses the fallback:

const ws = new WebSocket("wss://relay.example/moq", "moqt-22");
ws.binaryType = "arraybuffer";  // each message is one capsule, below

The WebSocket listener speaks plain HTTP; TLS is the front's.

The WebSocket binding (non-normative)

This binding is this crate's, not a standard's. It carries what a WebTransport session over HTTP/2 carries (draft-ietf-webtrans-http2-15), on a WebSocket instead of an extended CONNECT stream, so that a client written for one is most of the way to the other.

1. Opening. The client opens a WebSocket (RFC 6455) to the relay's path, offering the subprotocol moqt-22 in Sec-WebSocket-Protocol. The relay answers with that subprotocol, or refuses the upgrade (400 without it, 404 at another path).

2. Messages. Every message is a binary message holding exactly one capsule: Type (i) Length (i) Value, with QUIC variable-length integers (RFC 9000 §16), not MOQT's. A text message carries nothing and is ignored. A binary message that is not one whole capsule closes the session.

3. Capsules. Those of draft-ietf-webtrans-http2-15 §6, with their type codes:

   Capsule	Type	Value	Use here
   WT_STREAM	0x190B4D3C, 0x190B4D3B with FIN	Stream ID, data	stream data; a stream opens with its first WT_STREAM
   WT_RESET_STREAM	0x190B4D39	Stream ID, code, Reliable Size	reset a stream (Reliable Size 0)
   WT_STOP_SENDING	0x190B4D3A	Stream ID, code	ask the peer to stop
   WT_MAX_STREAMS	0x190B4D3F bidi, 0x190B4D40 uni	Maximum Streams	stream-count credit
   WT_STREAMS_BLOCKED	0x190B4D43, 0x190B4D44	Maximum Streams	sent when out of credit; informational
   DATAGRAM	0x00	payload	one MOQT datagram, reliable and ordered
   WT_CLOSE_SESSION	0x2843	code (32 bits), message (UTF-8, at most 1,024 bytes)	close with a MOQT session termination code
   WT_DRAIN_SESSION	0x78AE	empty	read and ignored; MOQT's GOAWAY is the drain
   others			read and ignored, as unknown capsules are

4. Streams. Stream IDs are numbered as QUIC numbers them: bit 0 the initiator (0 the client), bit 1 the direction (0 bidirectional), and the nth stream of a kind is 4n + kind. MOQT's control streams, request streams and data streams ride them byte for byte as they would ride QUIC streams: the client's control stream is its first unidirectional stream (ID 2), beginning with SETUP; a request is a bidirectional stream; a subgroup, fetch or padding stream is a unidirectional one.

5. Stream credit. Each end first sends its WT_MAX_STREAMS for both kinds; until the peer's arrive, an end opens nothing. Credit is cumulative, as in QUIC. A stream's credit comes back once both of its halves are done, and is granted in batches. Opening past the limit closes the session with code 0x045D4487. Because MOQT -22 bounds requests in flight by stream limits alone (§6.4.2), the relay's bidirectional limit (max_requests) is its bound on requests in flight on this binding, as MAX_STREAMS is on QUIC.

6. Data flow control. None at the capsule level (WT_MAX_DATA, WT_MAX_STREAM_DATA and their BLOCKED capsules are read and ignored): TCP's backpressure bounds a sender, and a receiver that holds more than websocket_buffer_bytes unread closes the session with 0x045D4487.

7. Closing. WT_CLOSE_SESSION with a MOQT session termination code (§12.2), then the WebSocket's close. A WebSocket that closes without one is a session closed with code 0.

The first capsules a relay with the default limits sends are its credit:

99 0b 4d 3f 02 41 00     WT_MAX_STREAMS (bidirectional), 256
99 0b 4d 40 02 44 00     WT_MAX_STREAMS (unidirectional), 1024

and a session it closes at the end of a GOAWAY's timeout ends with:

68 43 12 00 00 00 10 47 4f 41 57 41 59 20 74 69 6d 65 6f 75 74
                         WT_CLOSE_SESSION, GOAWAY_TIMEOUT (0x10), "GOAWAY timeout"

What the relay does

-22	Here
§6.3 session initialization	Each end's control stream opens with SETUP; data streams that come first wait for it. A client's SETUP is held to §9.1's rules for a client (no PATH or AUTHORITY over WebTransport; no DELETE or USE_ALIAS token). The relay's control stream carries only its SETUP and a GOAWAY; the end of the peer's closes the session with PROTOCOL_VIOLATION.
§6.4.2 requests	A request stream must begin with one of the seven requests. A Request ID of the wrong parity or seen before closes the session with INVALID_REQUEST_ID. MAX_REQUEST_UPDATES is enforced per stream; requests in flight are bounded by the stream limit (max_requests).
§8.9, §9.1.3, §9.1.4 tokens	A receiver's alias cache per session: REGISTER, DELETE, USE_ALIAS, USE_VALUE; AUTH_TOKEN_CACHE_OVERFLOW, DUPLICATE_AUTH_TOKEN_ALIAS, UNKNOWN_AUTH_TOKEN_ALIAS; a registration in SETUP past the cache reads as USE_VALUE. Grants name a namespace prefix, rights to publish and subscribe, the SHA-256 of a token of type 0, and whether it is a room grant honoured only beside a proof of possession (Tokens and room grants).
§4.1, §4.2 namespaces	PUBLISH_NAMESPACE is accepted (REQUEST_OK) and stands until it is cancelled or its session ends. SUBSCRIBE_NAMESPACE answers with NAMESPACE for each published namespace under the prefix, then NAMESPACE and NAMESPACE_DONE as they change, each carrying the suffix after the prefix; an overlapping prefix in one session is PREFIX_OVERLAP.
§7.6, §9.8 PUBLISH	Accepted with REQUEST_OK; its Track Alias is the publisher's for its data. A second track under one alias in one session closes it with DUPLICATE_TRACK_ALIAS. PUBLISH_DONE ends the track's downstream subscriptions.
§7.4, §7.6, §9.6 SUBSCRIBE	SUBSCRIBE_OK only once an upstream subscription is Established (§7.4): one the relay holds, or a SUBSCRIBE it sends to every publisher of a matching namespace. The relay's own upstream SUBSCRIBE asks for every object at priority 128 with no timeout, copying no subscriber's priority, group order or timeout (§5.1.3, §9.20.3, §9.20.4). RENDEZVOUS_TIMEOUT holds a SUBSCRIBE for a publisher, up to max_rendezvous_ms; without it a track with no publisher is DOES_NOT_EXIST. Location Filters set where the subscription starts and ends. A publisher of a namespace that arrives later is sent a SUBSCRIBE for every subscription the relay holds under it. When the last subscriber leaves, the relay cancels its own upstream SUBSCRIBE.
§9.5 REQUEST_UPDATE	Subscriber priority (re-applied to what is queued), FORWARD, the Location Filter and both delivery timeouts.
§3.2, §3.4, §7.1, §9.11 FETCH and fills	A fetch the cache covers is served from it, groups in the requested order (coverage, below); otherwise from the cache and the DVR store together where they cover it (The DVR store, below); otherwise it goes to a publisher of the track or its namespace and its fetch stream is piped through, past the Fill Timeout a Timed-Out range. A range that starts after the Largest Object is INVALID_RANGE; one that starts below the live edge needs the seek power (Tokens and room grants). A SUBSCRIBE with FILL_PARAMETERS gets its fill range the same way on a fill fetch stream named by its Request ID; a fill it cannot serve is a fetch stream reset after its header (§3.4.1).
§9.13 TRACK_STATUS	Answered from what the relay holds: the Largest Object and the track's properties, from the DVR store when the track's publisher is gone (or before a restart).
§7.3 several publishers	A track may have several upstreams; each (group, subgroup) is taken from the first that sends it, and the rest are counted as deduplicated.
§5.1 priorities, §5.2 delivery timeouts	Below.
§6.6.1, §9.2 GOAWAY	On shutdown every session gets GOAWAY with goaway_uri and the timeout; new requests are refused with GOING_AWAY; a session still open at the timeout is closed with GOAWAY_TIMEOUT.
§6.5 .session	No session-level track is defined: a request for one is DOES_NOT_EXIST.

What the cache covers

The cache answers a fetch alone only where it holds every object of the range, so that a gap in its answer means an object that does not exist (§3.2.2). It keeps what it knows of each group's arrival. A group is whole when its start arrived (Object ID 0, or a Prior Object ID Gap saying the objects before do not exist, §10.9), none of its subgroup streams is open or was reset, and its end is known: its streams ended with a FIN, or an End of Group or End of Track status, or a datagram with END_OF_GROUP, ended it. A fetch from a group is covered when every group from it up to the group of the Largest Object is whole or known not to exist (a Prior Group ID Gap, §10.8), that last group has its start, and nothing in between was evicted. Coverage is worked out from the groups held each time it is asked, so a group whose streams finish after a later group's still counts once it is whole; a group ID never seen is a gap the cache does not answer across. The relay cannot tell a subgroup it has not yet seen from one that does not exist, so a group whose subgroups have all closed counts as whole.

A fetch that is not covered only because groups are still arriving waits for them: while the track's publisher is still sending and nothing from the fetch's start was evicted, the relay checks again every 20 ms, for up to the fetch's fill budget or one second, whichever is less, before it asks upstream. A track of one group per object, each on its own stream (a page's audio is a group every 20 ms), always has a stream or two in flight, and a publisher by PUBLISH need not answer a FETCH at all.

A media group and its receipts group leave the cache together, since a group served without its receipts could not be verified. Receipts are cached just before their media, so the cache's budget reaches them first. Evicting either moves the other track's mark to that group as well, even when the other's group has not arrived yet, so a late member of the pair is never served from the cache alone.

Priorities and delivery timeouts

Each downstream session has a scheduler. Every data stream it will send (a subgroup, a fetch or fill, a datagram) is a lane whose items go out in order and one at a time, so no stream's objects are reordered (§7.4). Among the lanes with something to send it takes the item §5.1.2 puts first: the lower subscriber priority number; then the lower publisher priority number; then, within a subscription, the group its Group Order puts first (and in one group a fill before its subscription); then the lower Subgroup ID or, for a datagram, Object ID, a datagram before a subgroup object. At most write_concurrency writes are in flight, and a QUIC session's send window is quic_send_window_bytes: what waits, waits in the scheduler, where it can still be ordered and dropped (§5.2: "minimize the amount of data buffered at the underlying transport layer"). Each stream is also given a transport priority from its subscriber and publisher priorities.

An object's deadline is its OBJECT_DELIVERY_TIMEOUT from when it reached the relay: the smaller of the publisher's (the first object of its subgroup, else the track's) and the subscriber's, where both are set. It is checked before the object goes to the transport and bounds both opening its stream and writing it. Past it the stream is reset with DELIVERY_TIMEOUT and nothing more of that subgroup goes to that subscriber; a datagram is dropped. SUBGROUP_DELIVERY_TIMEOUT runs from the subgroup's FIN until all its data is acknowledged, and resets the stream if it is not. On the WebSocket binding a FIN counts as acknowledged once written, as webtrans-http2 §5.2 makes the transitions an acknowledgement would make at once.

A subscriber that does not read

QUIC's connection-level flow control is shared by every stream of a session (RFC 9000 §4.1), and bytes a subscriber has received but not read hold it. So yes: a subscriber that stops reading stalls its own high-priority track along with everything else, until it reads again. And a subscriber that reads its streams one at a time, in the order they arrive, can stall itself while still reading: the stream it waits on may have its bytes held back for credit that the unread streams behind it are holding. A subscriber reads its streams concurrently.

The relay cannot release credit the subscriber holds, and does not try to by dropping what has no deadline. What it does:

• An object with no delivery timeout (a base track, a receipts track) is never dropped. It waits, in the scheduler or the transport, and is delivered once the subscriber reads.
• What can still wait, waits in the relay rather than in the transport (§5.2: "minimize the amount of data buffered at the underlying transport layer"). Writes in flight are bounded by write_concurrency and the transport's unacknowledged bytes by quic_send_window_bytes. So an object whose deadline passes while it waits is dropped before it takes any credit.
• An object that passes its deadline is reset. A reset stream's final size settles its account (RFC 9000 §4.5), and a receiver can credit its unread bytes back to the connection at once, as quinn does. The credit a dropped layer held then comes back without the subscriber reading it.
• When credit frees, the highest-priority bytes go first: each stream carries its subscriber and publisher priority as its transport priority.

MOQT lets a publisher end a subscription whose subscriber falls too far behind, with PUBLISH_DONE TOO_FAR_BEHIND (§5.2); this relay does not yet.

Receipts

A media track's receipts ride its receipts track (WAI extensions/moq-streaming-format.md §4.1): group g of it is one object holding the group receipt and the object receipts of media group g. The relay recognises a receipts track by its name: the media track's name followed by receipts_track_suffix (_receipts unless configured, the name WAI extensions/live-rooms.md §6.1 gives a room's receipts tracks: 0_receipts for layer 0). Each object of it is the group's receipts in the binary form of WAI extensions/jwp-receipts.md §2.7.3 (WJR1): the group receipt as its canonical bytes, its link hash computed over them as received, then the object receipts. The door reads that form, and refuses a receipts object that does not read (§2.7.4) as it refuses one that does not verify.

• The receipts of dropped objects are forwarded. A receipts track has no delivery timeout, and the relay applies none to it, even one a subscriber asks for. So when an object of a media track passes its deadline, its receipt still reaches every subscriber of that track's receipts track, which can account for the object by inclusion (WAI extensions/jwp-receipts.md §4.1, §4.4). A receipt that rides on the object itself, as a property, is lost with it.
• Elsewhere receipts are opaque. Outside the namespaces a door rule names, the relay forwards receipts tracks as it forwards any track and reads nothing in them (§7.8).
• At the door. In a namespace a [[receipts.door]] rule names, the relay reads the receipts it forwards, and takes on jwp-receipts §4.1's duties: each group receipt's signature is verified, under the rule's trusted keys or any key, before the first object of its group is fanned out, and a group that does not verify is dropped with a structured error in the log; each media object must match the receipt of its position (its group, its Object ID, its content hash, its Merkle proof to the signed root), and an object that carries its receipt as a property must carry that same one. An object waits for its group's receipts up to hold_ms. Receipt bytes are forwarded unaltered and nothing is re-signed. A refused object ends its downstream streams rather than leaving a hole in them.

Tokens and room grants

[[auth.grants]] rules name a token by the SHA-256 of its value (Token Type 0), a namespace prefix, and the rights it gives there. With auth.required, a request is served only where a token of its own or of the session's SETUP matches a rule.

A rule with proof = true names a room grant: the token's value is a WRG1 record (WAI live-rooms §4.1). Every participant of a room holds every grant of it, to verify the room, so the record proves nothing about who presents it, and the relay honours it only beside a proof of possession of the grant's principal's key (live-rooms §10.5.2):

1. the relay's SETUP carries GRANT_CHALLENGE, drawn for the session;
2. the client waits for it, then sends its SETUP with the grant as an AUTHORIZATION TOKEN and GRANT_PROOF (0x3A49): a WRP1 record naming the grant's digest and the principal's key, over the challenge and the context SETUP //<authority><path> of the URI it addressed, signed by the principal;
3. the relay checks it (live-rooms §10.5.4: the grant named, the key the grant's principal — read from the record at its fixed place, since the rule names these exact bytes — the signature under WAI's strict Ed25519 rule, the session's own challenge, and the target it received: a WebTransport CONNECT's :authority and :path, a WebSocket upgrade's Host and path, native QUIC's AUTHORITY and PATH) before it serves any request;
4. a SETUP that presents such a grant without a proof that passes is closed with UNAUTHORIZED (§12.2), its reason (pop_missing, pop_wrong_key, …) in the log; a grant first presented in a request, never proved in SETUP, gives that request nothing (REQUEST_ERROR UNAUTHORIZED).

The relay keeps no consumed challenges: one is drawn for each session, so a proof made for another session is refused as pop_wrong_challenge. A rule without proof is a bearer token, for a key the relay's operator gives a party directly (a room's clock, an ingest gateway's link).

A room's authority. A room whose grants are issued as visitors arrive (a room service such as joule-moq-room) cannot be listed rule by rule. [[auth.authorities]] names the authority's key instead (and, optionally, the room ids it is honoured for): a token of type 0 that is a WRG1 record whose signature verifies strictly under that key (live-rooms §4.2; joule-moq's GrantHead) is a room grant, honoured only beside a proof as above and only within its window (nbf to exp), and mapped onto rights as live-rooms §10.2 maps it: PUBLISH under [R, P, m] for each media kind m its publish bits allow, of [R, P] (its catalog) when it may publish any, and under [R, P, events] when its policy permits an event kind; SUBSCRIBE, FETCH and the rest under [R] when it may subscribe. The relay's rights are per namespace, so §10.2's track matches (layers up to max_layer) are not narrowed. A session ends, EXPIRED_AUTH_TOKEN, when the earliest such grant it proved expires.

Seeking back (live-rooms §10.6). The live edge of a track is the group of its Largest Object. A FETCH whose range starts below it, or a SUBSCRIBE whose fill does, seeks back, and with auth.required it is served only to a session holding the seek power: an authority's grant whose powers hold bit 4 (§3.3), proved as above, or a [[auth.grants]] rule with subscribe and seek. Any other is refused REQUEST_ERROR UNAUTHORIZED with the reason phrase seek_not_granted. A FETCH that starts at the live edge, as a page joining the group in progress does, needs nothing more. The relay's seek_refused counter counts the refusals.

Patterns. A field * in a rule's prefix, or a door rule's, matches any one field: a room's clock token can be given ["*", "<clock>", "events"], its events track in every room. A door rule's trust_field names the namespace field holding the one key it accepts there: prefix = ["*", "*", "video"] with trust_field = 1 holds each publisher's video in a room to its own key (live-rooms §6.1), whoever joins.

Limits per session. limits.max_session_ms closes a session at that age (NO_ERROR, the limit as its reason); limits.max_ingress_kbps closes one that sends more object bytes than that rate allows over ingress_burst_ms (INTERNAL_ERROR). A session whose SETUP presents a token whose rule says unlimited = true is exempt from both: a deployment's own long-lived sessions, such as a room service's clock. limits.max_resident_mb makes the binary exit with status 70 past that resident memory, for its supervisor to start it again.

The session summary

When a session ends the relay signs a summary of it with its own Ed25519 key (from summary.key_path, or made at start and its public half logged), keeps it (Relay::summaries), and appends it as one JSON line to summary.out_path if set. It holds, for the session:

• as a subscriber: objects and their bytes forwarded, every byte written to its data streams, objects dropped for their delivery timeout and how many of those were on a track whose receipts track the session also held, subgroups reset after their FIN, objects served by fetch or fill, objects still queued at the end, and the same per track, with each track's fetched bytes, objects still queued at the end and objects of Object ID above 0;
• the spans of its layer class in each WAI room it subscribed to: the highest layer of the room's media tracks it held, or 8 when it held none of them but other tracks of the room;
• as a publisher: objects and bytes received, objects refused at the door, objects deduplicated, and the same per track;
• relay_energy: {"acquisition": "unmetered", "joules_micro": 0}. No meter backs this relay. The 0 is the unmetered marker of WAI extensions/energy-measurement.md §2, never a measurement of zero joules, and must not be read or shown as one;
• publisher_signed: where the door verified an object's receipt, the publisher's signed figure for producing it, summed under the acquisition class its leaf was signed with. These are the publisher's statements about the objects, not the relay's energy;
• room_grant, when the token the session presented in SETUP is one [[metering.grants]] names: the room grant's digest and its ceilings.

The summary is relay-scoped telemetry (jwp-receipts §4.2), not a receipt of the publisher's. Its format is joule-moq-relay/session-summary/2; the signature covers joule-moq-relay:session-summary\x01 followed by the summary's JSON bytes, and SignedSessionSummary::verify checks it.

The per-hop session record

Relay::hop_record(room, t0_unix_us) signs WAI's per-hop session record, WMH1 (wai/extensions/live-metering.md §3), of the room whose id is room, over the sessions that ended, on the room clock of the room's t0. A track is the room's when its namespace's first field is the room id in hexadecimal (WAI live-rooms §6.1); it is a media track when its namespace ends audio, video or screen and its name is a layer 0 to 7, an events track when its namespace ends events and its name is 0, and another track of the room otherwise. The record holds:

• per media track and direction, and for the events tracks and the other tracks together: the objects and bytes received and sent, the objects dropped for their delivery timeout, refused at the door, deduplicated and still queued at the end, and those of the dropped whose receipts the relay forwarded;
• its subscriber sessions, their microseconds of each layer class, and the events it delivered to them;
• the root of its ledger, one entry for each room grant its sessions ran under (or none, for sessions with no grant): the sessions, the bytes in and out and the subscriber time;
• its energy, the unmetered marker.

As a session closes, the relay ends its subscriptions one at a time, each when its request stream fails; the moments between are classed by what the session still holds, so a session whose media subscriptions end before its receipts or events ones has a span of class 8, typically microseconds, at its end. HopRecord::to_bytes is the record and HopRecord::ledger_bytes the ledger beside it. The relay connects to no other hop, so every session of it that subscribes is a subscriber session. Nothing in the record is per object or per viewer, so its size depends on the room's published tracks and not on its audience. tests/room_metering.rs checks that the record recomputes from the summaries.

The DVR store

With [dvr] configured the relay keeps, on disk, a window of the tracks under each [[dvr.window]] prefix with their receipts, and serves it to FETCH below the live edge (WAI extensions/live-recording.md §8). Its module is dvr.

What it keeps. Every track a window's prefix matches (the rule with the longest prefix keeps it): media tracks, their receipts tracks, events tracks, catalogs, takes. A group is stored once the cache holds it whole (What the cache covers), with every field a FETCH carries. For a track the store keeps, the cache does not evict a group while a stream of it is still arriving, past its budget by that group if it must, and it takes a group whose streams finish after a later group was evicted (coverage is unchanged: nothing at or below an evicted group is covered), so the store gets every group whatever order its streams finish in.

What it never keeps. A room whose objects are sealed (WAI confidentiality.md: an end-to-end encrypted room) is never recorded, and a relay keeps nothing of its tracks beyond delivery (§9; live-recording §8.2). The store knows such a room by its first object whose properties carry the secure-objects Key ID (SECURE_OBJECT_KEY_ID, read without a key), or whose properties do not read. That object's group is never written. From then on the store writes nothing of the room and deletes what it held of it: the room's events and receipts tracks are in the clear and can be whole before its first sealed group. It also drops the room's decisions from decisions.wdl, while still honouring them for what the cache serves. The room is recorded in sealed.wdl (room(32) check(32), check the SHA-256 of room), so a restart keeps refusing it; rooms_sealed counts them.

The window. Each [[dvr.window]] has a max_age_s, from when a group's first object reached the relay, of 1 to 86 400 (24 hours, the start-over buffer a live-to-VOD harvest clips from), and a max_bytes of payload over its tracks together. Past either the oldest group goes first, whole, and group g of a media track goes with group g of its receipts track. A group that becomes whole after the window has moved past its number is not stored (counted as groups_late).

On disk. Under path, one directory per track, tracks/<the first 16 bytes of the SHA-256 of its rendered name, hex>/, holding TRACK (namespace, name, properties, and a SHA-256 check) and the track's segments, s<n, 20 digits>.wds: files each group's record is appended to as it is stored, a new one begun every ten seconds or 16 MiB. A segment is a sequence of frames, len(u32) state(u8) record, the state 1 for a record that stands and 0 for one a later record replaced (a group written again: an object erased, or a subgroup that came late), whose bytes are then overwritten with zeros. A record is

header   "WDG1" 1 track(32) group(u64) arrived_us(u64) count(u32) erased(u32) check(32)
object   id(u64) subgroup(u64, 2^64-1 for a datagram) priority(u8) status(u8)
         props_len(u32) payload_len(u32) digest(32) check(32) props payload
erased   id(u64) digest(32) why(u8: 1 redacted, 2 did not verify)

Integers are little-endian. track is the SHA-256 of the rendered track; the header's check is SHA-256 over the header before it; an object's digest is the content hash its receipt names it by (jwp-receipts §2.1), and its check SHA-256 over its fixed fields before check, its properties and its payload. A segment is deleted once no group's standing record is in it, so on disk a window holds up to a segment of each track more than its budget. decisions.wdl beside tracks/ holds the moderation decisions the relay honoured, each a length, the WEV1 record and its SHA-256.

Durability. The writer syncs every segment it wrote, and the directories it made files in, at least once a second and before it answers a flush or stops; a shutdown flushes. A crash loses at most the last second's groups. (One sync per group could not keep up with a room's audio, a group every 20 ms per publisher.) On start the relay reads every segment again:

• a frame a crash cut short, at a segment's end, is cut off (torn_writes);
• a record whose header does not check is dropped;
• every object is checked: its check and digest; on a media track whose receipts track is stored, against group g's receipts (the group receipt's signature under the door's trust for the namespace, or any key that verifies where no door rule covers it; the object's receipt at its position; its Merkle proof to the signed root; its content hash); on an events track, every record under its principal. One that does not verify is dropped, counted (objects_dropped, and the WMH1 record's dvr_dropped), and its group written again with it among the erased. It is never served.

What is below the first group the store holds of a track after a restart is unknown to it. Each object's digest is checked again as it is read.

Answering a FETCH. A FETCH the cache covers is answered from it, as before. Otherwise, where every group of the range is held whole by the cache or the store, known not to exist, or evicted from the store, the relay answers from the cache and the store together, groups in the requested order, reading the store's groups as the fetch stream drains (about a mebibyte at a time). Otherwise it asks upstream. A FETCH of the receipts track brings each group's receipts the same way, so a viewer checks a recorded object exactly as a live one.

What is no longer there. Evicted groups are marked with an End of Unknown Range (0x10C, MOQT §11.4.1.2) through the last of them: a range partly evicted gets what remains and the marker, one wholly evicted the marker alone. Not an unmarked gap and not an End of Non-Existent Range: an evicted object existed, and an unmarked gap would say it did not (MOQT §3.2.2). An End of Range moves to a later group than the entry before it (its Group ID is delta-coded, as joule-moq reads -22), so an object of a group that cannot be served (dropped, withheld) is marked by one End of Unknown Range from the group's start through the last such object, and the group's objects after it follow.

Moderation. The relay reads the decision events (WAI live-events §4.11) of the events tracks it forwards in a kept room. It honours one whose record verifies, whose principal owns the track ([R, P, events]), and whose principal is a room key (dvr.room_keys, or a key of [[auth.authorities]]) or proved a grant with the moderate power on its publishing session. The last honoured decision on a target stands. Deny and redact withhold the target from every FETCH and fill, from the store and from the cache, for every session but a moderator's; redact also erases it from its group's file; allow restores what deny withheld. A media object is named by its receipt's digest, an event by its record's digest: a withheld event is left out of the events-track object that carries it, the other records byte for byte, and an object left with none is withheld whole. Live subscriptions are not filtered here (Not yet implemented).

Metering. A relay with a store signs WMH1 version 2 (live-metering §3.9): the objects and bytes its sessions were served from the store (dvr_objects, dvr_bytes in each summary's tracks), and from the store's own series, kept a point a second, the objects and bytes it wrote for the room, the bytes it held integrated over the span in byte-seconds, and the objects it dropped. Dvr::counts gives the store's counters, Dvr::figures a room's figures over a span.

Configuration.

[dvr]
path = "/var/lib/joule-moq-relay/dvr"
room_keys = []          # a room's authority and clock, hex; the authorities' keys count too

[[dvr.window]]
prefix = ["*"]          # a field "*" matches any one field
max_age_s = 900
max_bytes = 4294967296

The store must be kept where the room's region pin allows (live-rooms §3.7).

Configuration

joule-moq-relay --config relay.toml, or joule-moq-relay --dev for loopback with a development certificate. relay.example.toml shows every field: the listen addresses and path, the certificate files (or the names for a development certificate), the cache's byte budget, the limits (sessions, requests and streams in flight, MAX_REQUEST_UPDATES, the token cache, the rendezvous, upstream and fill timeouts, write concurrency, the QUIC send window, the GOAWAY timeout, the session, ingress and memory limits), the receipts suffix and door rules, the summary's key and file, the grants and the room authorities, and the DVR store and its windows. The configuration holds no secret: keys are files, and tokens are named by their SHA-256. JOULE_MOQ_RELAY_LOG sets the log level.

Not yet implemented

• SUBSCRIBE_TRACKS (§3.6): refused with NOT_SUPPORTED.
• Range Filters (§3.3.2): the relay declares MAX_FILTER_RANGES 0, so a request that carries one is INVALID_FILTER, and it evaluates no Track or Object Property filter.
• WebTransport over HTTP/2 (draft-ietf-webtrans-http2-15), the path a browser takes when HTTP/3 is unavailable. The WebSocket binding carries the same capsules and covers the same case for now.
• Parts of draft-ietf-webtrans-http3-16 the library does not speak: it negotiates with the settings deployed browsers send rather than -16's SETTINGS_WT_ENABLED, has no per-session flow-control capsules, and closes a session by closing its QUIC connection with the code rather than with WT_CLOSE_SESSION; and quinn 0.11 has no RESET_STREAM_AT, so a reset WebTransport stream is not guaranteed to deliver its stream header first.
• 0-RTT is not accepted.
• Migration as a client: the relay sends GOAWAY to its sessions, but does not itself move its upstream subscriptions to a new relay, or follow a REDIRECT.
• Pausing upstream when every downstream subscriber is paused (§7.2), and forwarding NEW_GROUP_REQUEST upstream. EXPIRES is neither issued nor acted on.
• TOO_FAR_BEHIND: a subscription whose subscriber does not read is held, not ended (§5.2 allows ending it with PUBLISH_DONE TOO_FAR_BEHIND).
• A cancellation the relay cannot see: a subscriber that sends FIN on its direction and later only STOP_SENDING is noticed when the relay next writes on the request stream, or when the session ends.
• Receipts: the receipts track of a media track is found by the configured suffix, not by reading the catalog's wai_receipts_track; the relay signs no delivery-gap claim; datagrams at the door are checked against their group's receipts, not a property.
• Energy: no meter backs the relay, so its own figure is always the unmetered marker. Attributing a metered host's energy to sessions is not done here.
• The DVR store: decisions withhold from FETCH and fills, not yet from live subscriptions (a reviewed item's held state, live-moderation §6.3, needs the review policy); objects are checked against their receipts at a restart, and by the door as they arrive where a door rule covers them, not again as they are stored; no HLS or DASH presentation of the window is written (live-recording §8.6 is a mapping).
• Re-validation (moqt-reval, live-rooms §10.2): a grant is checked at SETUP and on each request, and its expiry ends the session; revocation through the key log (§5) is not read.

Tests

All on loopback, every wait bounded:

Test	What it shows
relay_quic, relay_webtransport, relay_websocket	Two publishers (one by PUBLISH_NAMESPACE and the relay's SUBSCRIBE, one by PUBLISH) and three subscribers over each transport; every object verifies against its publisher's receipts, which the door also checks; SUBSCRIBE_NAMESPACE; every summary verifies and labels the relay's energy unmetered.
slow_subscriber	A base track at priority 0 and a layer at 200 with a 100 ms delivery timeout. A subscriber that reads nothing for 1.5 s, then reads all its streams concurrently, loses layer objects to the timeout (their streams reset with DELIVERY_TIMEOUT), keeps the whole base track, and receives every receipt of both tracks although it asked for a timeout on its receipts tracks; each dropped object verifies by inclusion; the summary counts each layer object as forwarded or dropped.
late_joiner	TRACK_STATUS; FETCH of a track and its receipts from the cache; a range in descending order; INVALID_RANGE; a fill on SUBSCRIBE; a group that reaches the relay after a later one, the FETCH from it still served from the cache; a FETCH forwarded upstream when the cache holds nothing.
goaway	GOAWAY with its URI and timeout to every session; GOING_AWAY for a new request; GOAWAY_TIMEOUT for the session that stays.
auth_tokens	Tokens in SETUP, by value and by alias; UNAUTHORIZED; UNKNOWN_AUTH_TOKEN_ALIAS.
grant_proof	A room grant with its proof over each transport gives its rights, each session with a challenge of its own; the grant as a bare token, a proof by another key, one for another relay and one over an earlier session's challenge each close the session with UNAUTHORIZED; a grant presented only in a request is refused it.
request_ids	INVALID_REQUEST_ID for the wrong parity and for a repeat; requests in flight bounded by the stream limit.
dvr	A long publish crosses the window: the oldest groups go, whole, each media group with its receipts group, within the byte budget; a window by age empties as its groups pass it. FETCH inside the window (ascending and descending), across it (an End of Unknown Range through the last evicted group, then every object, from the store and the cache together) and before it (the marker alone); every object verifies against the fetched receipts. A restart keeps the store, drops and counts a flipped byte and bytes changed with their checks made again (only the receipt finds those), removes a cut-short write, and never serves what it dropped. A room key's redaction erases its target and withholds it, a deny withholds until an allow, a redacted event leaves the object that carries it, a decision by no moderator changes nothing, and the decisions hold across a restart. Without the seek power a FETCH and a fill below the live edge are refused seek_not_granted, and a FETCH at the live edge is served; with it, the window. The record of the room is WMH1 version 2 with the store's figures. A room whose objects are sealed is never kept: its first sealed group is not written, its events and receipts already held and its decisions are deleted, nothing more of it is written, and so after a restart, while a room in the clear beside it keeps its window. A window past 24 hours is refused.
priority_fan_out	Two publishers, four tracks, three subscribers with their own subscriber priorities and group orders, their transports held shut while the load arrives: each receives every object in §5.1.2's order.

Licence

Apache-2.0.