open-standards/joule-code/joule-code-rs/crates/joule-moq-relay/README.md
Transaction Science cd6d0dfdd8 Merge T1 #401 (MOQT drafts -18 and -16, interoperability) into train/live-room
By meaning: the relay keeps PP's rating functions (take_ratings, gate,
group_age) and T1's track_restarts side by side. A track restarted after
TRACK_ENDED drops its cache, owners and Largest Location as T1 has it; the
rating claims held for it stay, since a claim only raises an age and the
publisher carries its claims again on the new PUBLISH. CHANGELOG kept both.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-22 03:46:09 -04:00

660 lines
51 KiB
Markdown

# joule-moq-relay
A relay for MOQT, `draft-ietf-moq-transport-22`, which also speaks `-18` and `-16` to
the sessions that choose them. Publishers send it tracks, subscribers
ask it for them, and it fans each object out to every subscription, in the order the
draft's priorities give, dropping what passes a subscriber's delivery timeout at this hop.
Whatever it drops, the object's receipts still reach the subscriber, on the track's
receipts track.
It is a library (`Relay`, `RelayServer`, `client::Client`) and a binary,
`joule-moq-relay`. The wire codec is `joule-moq`'s.
## The drafts it speaks
`draft-ietf-moq-transport-22`, and `-18` and `-16` for a peer that offers no later one.
Drafts do not interoperate, so a session speaks one from its first byte. The draft is
chosen before any MOQT byte, the relay preferring -22, then -18, then -16:
- by ALPN on native QUIC: `moqt-22`, `moqt-18`, `moqt-16` (-22 §6.2.2);
- by `WT-Available-Protocols` on WebTransport, which the relay answers with the
`WT-Protocol` it chose (§6.2.1; `draft-ietf-webtrans-http3-16` §3.3). A client that
sends no `WT-Available-Protocols` is taken to speak -22; one that offers only drafts
the relay does not speak is refused;
- by the subprotocol `moqt-22` on the WebSocket fallback (below), which is -22 only.
The relay is written once, against -22. A session at -18 or -16 runs behind
`transport::bridge`, which translates each message and object at the session's edge with
`joule-moq`'s `draft18::bridge` and `draft16::bridge`: at -18 stream by stream (the
drafts share their shape); at -16 by giving each request on -16's one control stream a
stream of its own, its Request IDs renumbered into -16's sequence, MAX_REQUEST_ID granted
ahead, a cancelled stream becoming UNSUBSCRIBE, FETCH_CANCEL, PUBLISH_NAMESPACE_DONE or
PUBLISH_NAMESPACE_CANCEL. What a session's draft cannot say is refused (a request, with
REQUEST_ERROR) or left out, by name, and counted in the session's notes. Two rules differ
at the edge: at -18 a requester's FIN on PUBLISH_NAMESPACE, SUBSCRIBE_NAMESPACE or
SUBSCRIBE_TRACKS cancels it (-18 §6.1, §6.2), where at -22 a FIN is not a cancellation
(§6.4.2.2); and at -16 a new Request ID may arrive ahead of one still in flight on another
stream (SUBSCRIBE_NAMESPACE rides its own), which is admitted, where -16 §9.1 asks for the
next in sequence. Each message in and out is logged at debug level. `joule-moq`'s
README tables every difference. Receipts ride objects' Immutable Properties and receipts
tracks, which are the same on every draft, so they cross the bridge unchanged.
The relay sends its SETUP as soon as a session opens, without waiting for the client's
(§6.3 lets either end send first): it declares `MOQT_IMPLEMENTATION`, `MAX_REQUEST_UPDATES`,
`MAX_AUTH_TOKEN_CACHE_SIZE`, `joule-moq`'s receipt extension (Setup Option `0x3A46`, its
value the extension version; on for the session when the client declares it too), and
`GRANT_CHALLENGE` (`0x3A47`): 32 bytes drawn for this session, which a client presenting
a room grant signs (below, *Tokens and room grants*).
## Transports
| Path | Listener | Selected by |
|---|---|---|
| Native QUIC | UDP `quic_listen` | ALPN `moqt-22`, `moqt-18` or `moqt-16` |
| WebTransport over HTTP/3 | the same UDP port | ALPN `h3`, extended CONNECT to `path`, `WT-Available-Protocols` |
| WebSocket fallback | TCP `websocket_listen` | upgrade at `path` with subprotocol `moqt-22` |
Native QUIC and WebTransport share one quinn endpoint: the relay reads the ALPN of each
connection and either runs MOQT on it directly or hands it to the WebTransport layer.
**The WebTransport implementation** is `wtransport` 0.7.2 (MIT OR Apache-2.0), vendored
(below). It was chosen because it runs on the same quinn 0.11 as `joule-moq`, so one UDP
endpoint serves both ALPNs; it gives the server the CONNECT request's headers and lets it answer with its
own, which is what `WT-Available-Protocols` and `WT-Protocol` need; and its client checks
a certificate by hash under the rules a browser applies to `serverCertificateHashes`
(ECDSA P-256, a validity of fourteen days at most), so the relay's tests connect the way a
page does. Stream bytes are read and written on the quinn streams underneath; application
error codes on WebTransport streams are mapped into the HTTP/3 error space as
`draft-ietf-webtrans-http3-16` §4.4 gives.
**Closing a WebTransport session.** A MOQT session over WebTransport is closed by closing
the WebTransport session (`draft-ietf-moq-transport-22` §6.6). The relay sends a
`WT_CLOSE_SESSION` capsule (`0x2843`) on the session's CONNECT stream, then a FIN
(`draft-ietf-webtrans-http3-16` §6): the MOQT session termination code (§12.2) is the
capsule's 32-bit application error code as it stands, not mapped into the HTTP/3 space as
stream codes are, and the reason, at most 1,024 bytes of UTF-8, is its message. The QUIC
connection is closed with `H3_NO_ERROR` once the peer has closed its side of the CONNECT
stream, or two seconds after the capsule, so the connection's close cannot overtake it.
Nothing else of the session overtakes it either: a stream the relay lets go once the session
is closing (the client's control stream of a refused session, say) is held until the QUIC
connection has closed, rather than ended at once with a FIN or STOP_SENDING that would reach
the page before the capsule; a page whose stream was stopped just ahead of the close was
seen to lose its renderer. A page reads the code and reason from `WebTransport.closed`
(`closeCode`, `reason`), as it reads them from the WebSocket binding's `WT_CLOSE_SESSION`. A session the client closes is
read the same way: the relay's `closed()` gives the client's code and reason, and the
session's summary records the code as the peer's. A QUIC connection that closes with the
session still open gives no MOQT code; its code is an HTTP/3 error code.
The release of `wtransport` reads `WT_CLOSE_SESSION` but cannot send it, so the joule-code
workspace vendors 0.7.2 with a session close added (`vendor/wtransport`, patched in with
`[patch.crates-io]`; `vendor/wtransport/VENDORED.md` lists what changed and why).
## How a browser connects
**WebTransport, in development.** Start the relay with `--dev` (or with no certificate
files in the configuration). It makes a certificate a browser accepts by hash and logs its
SHA-256 three ways, including a ready `serverCertificateHashes` entry:
```js
const transport = new WebTransport("https://127.0.0.1:4443/moq", {
serverCertificateHashes: [{ algorithm: "sha-256", value: new Uint8Array([/* printed */]) }],
protocols: ["moqt-22"],
});
await transport.ready; // transport.protocol === "moqt-22"
```
The certificate is valid fourteen days from the relay's start; restart the relay to make a
new one. In production give it a certificate chain and key from files, and a page connects
without `serverCertificateHashes`.
**WebSocket, where there is no UDP path.** A site behind a front that carries HTTP over
TCP only reaches no UDP endpoint, so neither QUIC path is open to its pages. Such a page
uses the fallback:
```js
const ws = new WebSocket("wss://relay.example/moq", "moqt-22");
ws.binaryType = "arraybuffer"; // each message is one capsule, below
```
The WebSocket listener speaks plain HTTP; TLS is the front's.
## The WebSocket binding (non-normative)
This binding is this crate's, not a standard's. It carries what a WebTransport session over
HTTP/2 carries (`draft-ietf-webtrans-http2-15`), on a WebSocket instead of an extended
CONNECT stream, so that a client written for one is most of the way to the other.
1. **Opening.** The client opens a WebSocket (RFC 6455) to the relay's `path`, offering
the subprotocol `moqt-22` in `Sec-WebSocket-Protocol`. The relay answers with that
subprotocol, or refuses the upgrade (`400` without it, `404` at another path).
2. **Messages.** Every message is a binary message holding exactly one capsule: `Type (i)
Length (i) Value`, with QUIC variable-length integers (RFC 9000 §16), not MOQT's. A
text message carries nothing and is ignored. A binary message that is not one whole
capsule closes the session.
3. **Capsules.** Those of `draft-ietf-webtrans-http2-15` §6, with their type codes:
| Capsule | Type | Value | Use here |
|---|---|---|---|
| `WT_STREAM` | `0x190B4D3C`, `0x190B4D3B` with FIN | Stream ID, data | stream data; a stream opens with its first `WT_STREAM` |
| `WT_RESET_STREAM` | `0x190B4D39` | Stream ID, code, Reliable Size | reset a stream (Reliable Size 0) |
| `WT_STOP_SENDING` | `0x190B4D3A` | Stream ID, code | ask the peer to stop |
| `WT_MAX_STREAMS` | `0x190B4D3F` bidi, `0x190B4D40` uni | Maximum Streams | stream-count credit |
| `WT_STREAMS_BLOCKED` | `0x190B4D43`, `0x190B4D44` | Maximum Streams | sent when out of credit; informational |
| `DATAGRAM` | `0x00` | payload | one MOQT datagram, reliable and ordered |
| `WT_CLOSE_SESSION` | `0x2843` | code (32 bits), message (UTF-8, at most 1,024 bytes) | close with a MOQT session termination code |
| `WT_DRAIN_SESSION` | `0x78AE` | empty | read and ignored; MOQT's GOAWAY is the drain |
| others | | | read and ignored, as unknown capsules are |
4. **Streams.** Stream IDs are numbered as QUIC numbers them: bit 0 the initiator (0 the
client), bit 1 the direction (0 bidirectional), and the `n`th stream of a kind is
`4n + kind`. MOQT's control streams, request streams and data streams ride them
byte for byte as they would ride QUIC streams: the client's control stream is its
first unidirectional stream (ID 2), beginning with SETUP; a request is a bidirectional
stream; a subgroup, fetch or padding stream is a unidirectional one.
5. **Stream credit.** Each end first sends its `WT_MAX_STREAMS` for both kinds; until the
peer's arrive, an end opens nothing. Credit is cumulative, as in QUIC. A stream's credit
comes back once both of its halves are done, and is granted in batches. Opening past
the limit closes the session with code `0x045D4487`. Because MOQT -22 bounds requests in
flight by stream limits alone (§6.4.2), the relay's bidirectional limit
(`max_requests`) is its bound on requests in flight on this binding, as MAX_STREAMS is
on QUIC.
6. **Data flow control.** None at the capsule level (`WT_MAX_DATA`, `WT_MAX_STREAM_DATA`
and their `BLOCKED` capsules are read and ignored): TCP's backpressure bounds a sender,
and a receiver that holds more than `websocket_buffer_bytes` unread closes the session
with `0x045D4487`.
7. **Closing.** `WT_CLOSE_SESSION` with a MOQT session termination code (§12.2), then the
WebSocket's close. A WebSocket that closes without one is a session closed with code 0.
The first capsules a relay with the default limits sends are its credit:
```text
99 0b 4d 3f 02 41 00 WT_MAX_STREAMS (bidirectional), 256
99 0b 4d 40 02 44 00 WT_MAX_STREAMS (unidirectional), 1024
```
and a session it closes at the end of a GOAWAY's timeout ends with:
```text
68 43 12 00 00 00 10 47 4f 41 57 41 59 20 74 69 6d 65 6f 75 74
WT_CLOSE_SESSION, GOAWAY_TIMEOUT (0x10), "GOAWAY timeout"
```
## What the relay does
| -22 | Here |
|---|---|
| §6.3 session initialization | Each end's control stream opens with SETUP; data streams that come first wait for it. A client's SETUP is held to §9.1's rules for a client (no PATH or AUTHORITY over WebTransport; no DELETE or USE_ALIAS token). The relay's control stream carries only its SETUP and a GOAWAY; the end of the peer's closes the session with PROTOCOL_VIOLATION, once nothing of the peer's publishing is left (each track it published ended, every data stream of its read) or after three seconds, so a publisher that closes its control stream as it leaves loses no object it sent. |
| §6.4.2 requests | A request stream must begin with one of the seven requests. A Request ID of the wrong parity or seen before closes the session with INVALID_REQUEST_ID. MAX_REQUEST_UPDATES is enforced per stream; requests in flight are bounded by the stream limit (`max_requests`). |
| §8.9, §9.1.3, §9.1.4 tokens | A receiver's alias cache per session: REGISTER, DELETE, USE_ALIAS, USE_VALUE; AUTH_TOKEN_CACHE_OVERFLOW, DUPLICATE_AUTH_TOKEN_ALIAS, UNKNOWN_AUTH_TOKEN_ALIAS; a registration in SETUP past the cache reads as USE_VALUE. Grants name a namespace prefix, rights to publish and subscribe, the SHA-256 of a token of type 0, and whether it is a room grant honoured only beside a proof of possession (*Tokens and room grants*). |
| §4.1, §4.2 namespaces | PUBLISH_NAMESPACE is accepted (REQUEST_OK) and stands until it is cancelled or its session ends. SUBSCRIBE_NAMESPACE answers with NAMESPACE for each published namespace under the prefix, then NAMESPACE and NAMESPACE_DONE as they change, each carrying the suffix after the prefix; an overlapping prefix in one session is PREFIX_OVERLAP. |
| §7.6, §9.8 PUBLISH | Accepted with REQUEST_OK; its Track Alias is the publisher's for its data. A second track under one alias in one session closes it with DUPLICATE_TRACK_ALIAS. PUBLISH_DONE ends the track's downstream subscriptions with its status, once the relay has read as many of the publisher's data streams as its Stream Count says it opened (they and the request stream are not ordered with each other, §9.9), or after three seconds, or at once when the count is 0 or unknown or the publisher's session is gone. A publisher's session that ends ends its tracks from their own request streams: with the status of a PUBLISH_DONE already read, else GOING_AWAY. A track its publisher ended with TRACK_ENDED and another then publishes is a new instance: what the relay held of the ended one (its cache, its Largest Location, which upstream each subgroup came from) goes, unless the DVR store keeps the track. |
| §7.4, §7.6, §9.6 SUBSCRIBE | SUBSCRIBE_OK only once an upstream subscription is Established (§7.4): one the relay holds, or a SUBSCRIBE it sends to every publisher of a matching namespace. The relay's own upstream SUBSCRIBE asks for every object at priority 128 with no timeout, copying no subscriber's priority, group order or timeout (§5.1.3, §9.20.3, §9.20.4). RENDEZVOUS_TIMEOUT holds a SUBSCRIBE for a publisher, up to `max_rendezvous_ms`; without it a track with no publisher is DOES_NOT_EXIST. Location Filters set where the subscription starts and ends. A publisher of a namespace that arrives later is sent a SUBSCRIBE for every subscription the relay holds under it. When the last subscriber leaves, the relay cancels its own upstream SUBSCRIBE. |
| §9.5 REQUEST_UPDATE | Subscriber priority (re-applied to what is queued), FORWARD, the Location Filter and both delivery timeouts. |
| §3.2, §3.4, §7.1, §9.11 FETCH and fills | A fetch the cache covers is served from it, groups in the requested order (coverage, below); otherwise from the cache and the DVR store together where they cover it (*The DVR store*, below); otherwise it goes to a publisher of the track or its namespace and its fetch stream is piped through, past the Fill Timeout a Timed-Out range. A range that starts after the Largest Object is INVALID_RANGE; one that starts below the live edge needs the seek power (*Tokens and room grants*). A SUBSCRIBE with FILL_PARAMETERS gets its fill range the same way on a fill fetch stream named by its Request ID; a fill it cannot serve is a fetch stream reset after its header (§3.4.1). |
| §9.13 TRACK_STATUS | Answered from what the relay holds: the Largest Object and the track's properties, from the DVR store when the track's publisher is gone (or before a restart). |
| §7.3 several publishers | A track may have several upstreams; each (group, subgroup) is taken from the first that sends it, and the rest are counted as deduplicated. |
| §5.1 priorities, §5.2 delivery timeouts | Below. |
| §6.6.1, §9.2 GOAWAY | On shutdown every session gets GOAWAY with `goaway_uri` and the timeout; new requests are refused with GOING_AWAY; a session still open at the timeout is closed with GOAWAY_TIMEOUT. |
| §6.5 `.session` | No session-level track is defined: a request for one is DOES_NOT_EXIST. |
### What the cache covers
The cache answers a fetch alone only where it holds every object of the range, so that a
gap in its answer means an object that does not exist (§3.2.2). It keeps what it knows of
each group's arrival. A group is whole when its start arrived (Object ID 0, or a Prior
Object ID Gap saying the objects before do not exist, §10.9), none of its subgroup streams
is open or was reset, and its end is known: its streams ended with a FIN, or an End of
Group or End of Track status, or a datagram with END_OF_GROUP, ended it. A fetch from a
group is covered when every group from it up to the group of the Largest Object is whole
or known not to exist (a Prior Group ID Gap, §10.8), that last group has its start, and
nothing in between was evicted. Coverage is worked out from the groups held each time it
is asked, so a group whose streams finish after a later group's still counts once it is
whole; a group ID never seen is a gap the cache does not answer across. The relay cannot
tell a subgroup it has not yet seen from one that does not exist, so a group whose
subgroups have all closed counts as whole.
A fetch that is not covered only because groups are still arriving waits for them: while
the track's publisher is still sending and nothing from the fetch's start was evicted, the
relay checks again every 20 ms, for up to the fetch's fill budget or one second, whichever
is less, before it asks upstream. A track of one group per object, each on its own stream
(a page's audio is a group every 20 ms), always has a stream or two in flight, and a
publisher by PUBLISH need not answer a FETCH at all.
A media group and its receipts group leave the cache together, since a group served without
its receipts could not be verified. Receipts are cached just before their media, so the
cache's budget reaches them first. Evicting either moves the other track's mark to that group
as well, even when the other's group has not arrived yet, so a late member of the pair is
never served from the cache alone.
### Priorities and delivery timeouts
Each downstream session has a scheduler. Every data stream it will send (a subgroup, a
fetch or fill, a datagram) is a lane whose items go out in order and one at a time, so no
stream's objects are reordered (§7.4). Among the lanes with something to send it takes the
item §5.1.2 puts first: the lower subscriber priority number; then the lower publisher
priority number; then, within a subscription, the group its Group Order puts first (and in
one group a fill before its subscription); then the lower Subgroup ID or, for a datagram,
Object ID, a datagram before a subgroup object. At most `write_concurrency` writes are in
flight, and a QUIC session's send window is `quic_send_window_bytes`: what waits, waits in
the scheduler, where it can still be ordered and dropped (§5.2: "minimize the amount of
data buffered at the underlying transport layer"). Each stream is also given a transport
priority from its subscriber and publisher priorities.
An object's deadline is its OBJECT_DELIVERY_TIMEOUT from when it reached the relay: the
smaller of the publisher's (the first object of its subgroup, else the track's) and the
subscriber's, where both are set. It is checked before the object goes to the transport and
bounds both opening its stream and writing it. Past it the stream is reset with
DELIVERY_TIMEOUT and nothing more of that subgroup goes to that subscriber; a datagram is
dropped. SUBGROUP_DELIVERY_TIMEOUT runs from the subgroup's FIN until all its data is
acknowledged, and resets the stream if it is not. On the WebSocket binding a FIN counts as
acknowledged once written, as webtrans-http2 §5.2 makes the transitions an acknowledgement
would make at once.
### A subscriber that does not read
QUIC's connection-level flow control is shared by every stream of a session (RFC 9000
§4.1), and bytes a subscriber has received but not read hold it. So yes: a subscriber that
stops reading stalls its own high-priority track along with everything else, until it reads
again. And a subscriber that reads its streams one at a time, in the order they arrive, can
stall itself while still reading: the stream it waits on may have its bytes held back for
credit that the unread streams behind it are holding. A subscriber reads its streams
concurrently.
The relay cannot release credit the subscriber holds, and does not try to by dropping
what has no deadline. What it does:
- An object with no delivery timeout (a base track, a receipts track) is never dropped. It
waits, in the scheduler or the transport, and is delivered once the subscriber reads.
- What can still wait, waits in the relay rather than in the transport (§5.2: "minimize the
amount of data buffered at the underlying transport layer"). Writes in flight are
bounded by `write_concurrency` and the transport's unacknowledged bytes by
`quic_send_window_bytes`. So an object whose deadline passes while it waits is dropped
before it takes any credit.
- An object that passes its deadline is reset. A reset stream's final size settles its
account (RFC 9000 §4.5), and a receiver can credit its unread bytes back to the
connection at once, as quinn does. The credit a dropped layer held then comes back
without the subscriber reading it.
- When credit frees, the highest-priority bytes go first: each stream carries its
subscriber and publisher priority as its transport priority.
MOQT lets a publisher end a subscription whose subscriber falls too far behind, with
PUBLISH_DONE TOO_FAR_BEHIND (§5.2); this relay does not yet.
### Receipts
A media track's receipts ride its **receipts track** (WAI `extensions/moq-streaming-format.md`
§4.1): group *g* of it is one object holding the group receipt and the object receipts of
media group *g*. The relay recognises a receipts track by its name: the media track's name
followed by `receipts_track_suffix` (`_receipts` unless configured, the name WAI
`extensions/live-rooms.md` §6.1 gives a room's receipts tracks: `0_receipts` for layer 0).
Each object of it is the group's receipts in the binary form of WAI
`extensions/jwp-receipts.md` §2.7.3 (`WJR1`): the group receipt as its canonical bytes, its
link hash computed over them as received, then the object receipts. The door reads that
form, and refuses a receipts object that does not read (§2.7.4) as it refuses one that does
not verify.
- **The receipts of dropped objects are forwarded.** A receipts track has no delivery
timeout, and the relay applies none to it, even one a subscriber asks for. So when an
object of a media track passes its deadline, its receipt still reaches every subscriber of
that track's receipts track, which can account for the object by inclusion (WAI
`extensions/jwp-receipts.md` §4.1, §4.4). A receipt that rides on the object itself, as a
property, is lost with it.
- **Elsewhere receipts are opaque.** Outside the namespaces a door rule names, the relay
forwards receipts tracks as it forwards any track and reads nothing in them (§7.8).
- **At the door.** In a namespace a `[[receipts.door]]` rule names, the relay reads the
receipts it forwards, and takes on jwp-receipts §4.1's duties: each group receipt's
signature is verified, under the rule's trusted keys or any key, before the first object
of its group is fanned out, and a group that does not verify is dropped with a structured
error in the log; each media object must match the receipt of its position (its group,
its Object ID, its content hash, its Merkle proof to the signed root), and an object that
carries its receipt as a property must carry that same one. An object waits for its
group's receipts up to `hold_ms`. Receipt bytes are forwarded unaltered and nothing is
re-signed. A refused object ends its downstream streams rather than leaving a hole in
them.
## Tokens and room grants
`[[auth.grants]]` rules name a token by the SHA-256 of its value (Token Type 0), a
namespace prefix, and the rights it gives there. With `auth.required`, a request is served
only where a token of its own or of the session's SETUP matches a rule.
A rule with `proof = true` names a **room grant**: the token's value is a `WRG1` record (WAI
live-rooms §4.1). Every participant of a room holds every grant of it, to verify the room,
so the record proves nothing about who presents it, and the relay honours it only beside a
proof of possession of the grant's principal's key (live-rooms §10.5.2):
1. the relay's SETUP carries `GRANT_CHALLENGE`, drawn for the session;
2. the client waits for it, then sends its SETUP with the grant as an AUTHORIZATION TOKEN
and `GRANT_PROOF` (`0x3A49`): a `WRP1` record naming the grant's digest and the
principal's key, over the challenge and the context `SETUP //<authority><path>` of the
URI it addressed, signed by the principal;
3. the relay checks it (live-rooms §10.5.4: the grant named, the key the grant's principal
— read from the record at its fixed place, since the rule names these exact bytes — the
signature under WAI's strict Ed25519 rule, the session's own challenge, and the target
it received: a WebTransport CONNECT's `:authority` and `:path`, a WebSocket upgrade's
`Host` and path, native QUIC's AUTHORITY and PATH) before it serves any request;
4. a SETUP that presents such a grant without a proof that passes is closed with
`UNAUTHORIZED` (§12.2), its reason (`pop_missing`, `pop_wrong_key`, …) in the log; a
grant first presented in a request, never proved in SETUP, gives that request nothing
(`REQUEST_ERROR` `UNAUTHORIZED`).
The relay keeps no consumed challenges: one is drawn for each session, so a proof made for
another session is refused as `pop_wrong_challenge`. A rule without `proof` is a bearer
token, for a key the relay's operator gives a party directly (a room's clock, an ingest
gateway's link).
**A room's authority.** A room whose grants are issued as visitors arrive (a room service
such as `joule-moq-room`) cannot be listed rule by rule. `[[auth.authorities]]` names the
authority's key instead (and, optionally, the room ids it is honoured for): a token of type
0 that is a `WRG1` record whose signature verifies strictly under that key (live-rooms §4.2;
`joule-moq`'s `GrantHead`) is a room grant, honoured only beside a proof as above and only
within its window (`nbf` to `exp`), and mapped onto rights as live-rooms §10.2 maps it:
PUBLISH under `[R, P, m]` for each media kind `m` its `publish` bits allow, of `[R, P]` (its
catalog) when it may publish any, and under `[R, P, events]` when its policy permits an
event kind; SUBSCRIBE, FETCH and the rest under `[R]` when it may subscribe. The relay's
rights are per namespace, so §10.2's track matches (layers up to `max_layer`) are not
narrowed. A session ends, `EXPIRED_AUTH_TOKEN`, when the earliest such grant it proved
expires.
**Seeking back** (live-rooms §10.6). The **live edge** of a track is the group of its Largest
Object. A FETCH whose range starts below it, or a SUBSCRIBE whose fill does, seeks back, and
with `auth.required` it is served only to a session holding the **seek** power: an
authority's grant whose `powers` hold bit 4 (§3.3), proved as above, or a `[[auth.grants]]`
rule with `subscribe` and `seek`. Any other is refused `REQUEST_ERROR` `UNAUTHORIZED` with
the reason phrase `seek_not_granted`. A FETCH that starts at the live edge, as a page joining
the group in progress does, needs nothing more. The relay's `seek_refused` counter counts the
refusals.
**Age ratings** (WAI age-assurance §3.6, §5.5, §5.7). A rating claim reaches the relay as the
property `WAI_RATING` (`0x3A4D`, its value the claim's text): a Track Property of a PUBLISH or
of an upstream SUBSCRIBE_OK, or an Object Property of an object it forwards. From the instant it
reads one, the relay holds it where the extension lets it honour it: a claim about a room signed
by an authority it trusts for that room; a claim about a track (from a group on, or a span of
groups) signed by such an authority or by the track's owner, the principal its namespace names,
and carried on that track, or carried anywhere in the room by the authority; an object's claim
it passes over. It keeps at most 64 claims a room or track and 65 536 in all, forgetting the
claims of rooms it no longer carries first. A session's **threshold** in a room is the greatest
`age` attribute (§5.4) of the room's grants its SETUP proved; 0 otherwise. A group's age is the
greatest age of the claims that cover it, and the relay serves the group to a session only when
the threshold reaches it: a live object of a group above it is not forwarded; when a newly held
claim raises a group above a subscriber's threshold, the streams of the groups it covers are
ended for that subscriber (`CANCELLED`) and nothing more of them is sent; a SUBSCRIBE that
forwards objects, a FETCH, or a fill whose first group is above it is refused, `REQUEST_ERROR`
`UNAUTHORIZED` with the reason phrase `below_threshold`; and one whose range reaches such a group
later is answered up to the group before it. The counters `age_refused`, `age_withheld` and
`ratings_held` count the refusals, the objects not forwarded, and the claims held.
**Patterns.** A field `*` in a rule's prefix, or a door rule's, matches any one field: a
room's clock token can be given `["*", "<clock>", "events"]`, its events track in every
room. A door rule's `trust_field` names the namespace field holding the one key it accepts
there: `prefix = ["*", "*", "video"]` with `trust_field = 1` holds each publisher's video in
a room to its own key (live-rooms §6.1), whoever joins.
**Limits per session.** `limits.max_session_ms` closes a session at that age (`NO_ERROR`,
the limit as its reason); `limits.max_ingress_kbps` closes one that sends more object bytes
than that rate allows over `ingress_burst_ms` (`INTERNAL_ERROR`). A session whose SETUP
presents a token whose rule says `unlimited = true` is exempt from both: a deployment's own
long-lived sessions, such as a room service's clock. `limits.max_resident_mb` makes the
binary exit with status 70 past that resident memory, for its supervisor to start it again.
## The session summary
When a session ends the relay signs a summary of it with its own Ed25519 key (from
`summary.key_path`, or made at start and its public half logged), keeps it
(`Relay::summaries`), and appends it as one JSON line to `summary.out_path` if set. It
holds, for the session:
- as a subscriber: objects and their bytes forwarded, every byte written to its data
streams, objects dropped for their delivery timeout and how many of those were on a
track whose receipts track the session also held, subgroups reset after their FIN,
objects served by fetch or fill, objects still queued at the end, and the same per
track, with each track's fetched bytes, objects still queued at the end and objects of
Object ID above 0;
- the spans of its layer class in each WAI room it subscribed to: the highest layer of the
room's media tracks it held, or 8 when it held none of them but other tracks of the room;
- as a publisher: objects and bytes received, objects refused at the door, objects
deduplicated, and the same per track;
- `relay_energy`: `{"acquisition": "unmetered", "joules_micro": 0}`. No meter backs this
relay. The 0 is the unmetered marker of WAI `extensions/energy-measurement.md` §2, never
a measurement of zero joules, and must not be read or shown as one;
- `publisher_signed`: where the door verified an object's receipt, the publisher's signed
figure for producing it, summed under the acquisition class its leaf was signed with.
These are the publisher's statements about the objects, not the relay's energy;
- `room_grant`, when the token the session presented in SETUP is one `[[metering.grants]]`
names: the room grant's digest and its ceilings.
The summary is relay-scoped telemetry (jwp-receipts §4.2), not a receipt of the publisher's.
Its format is `joule-moq-relay/session-summary/2`; the signature covers
`joule-moq-relay:session-summary\x01` followed by the summary's JSON bytes, and
`SignedSessionSummary::verify` checks it.
## The per-hop session record
`Relay::hop_record(room, t0_unix_us)` signs WAI's per-hop session record, `WMH1`
(`wai/extensions/live-metering.md` §3), of the room whose id is `room`, over the sessions
that ended, on the room clock of the room's `t0`. A track is the room's when its
namespace's first field is the room id in hexadecimal (WAI live-rooms §6.1); it is a media
track when its namespace ends `audio`, `video` or `screen` and its name is a layer `0` to
`7`, an events track when its namespace ends `events` and its name is `0`, and another
track of the room otherwise. The record holds:
- per media track and direction, and for the events tracks and the other tracks together:
the objects and bytes received and sent, the objects dropped for their delivery timeout,
refused at the door, deduplicated and still queued at the end, and those of the dropped
whose receipts the relay forwarded;
- its subscriber sessions, their microseconds of each layer class, and the events it
delivered to them;
- the root of its ledger, one entry for each room grant its sessions ran under (or none, for
sessions with no grant): the sessions, the bytes in and out and the subscriber time;
- its energy, the unmetered marker.
As a session closes, the relay ends its subscriptions one at a time, each when its request
stream fails; the moments between are classed by what the session still holds, so a
session whose media subscriptions end before its receipts or events ones has a span of
class 8, typically microseconds, at its end. `HopRecord::to_bytes` is the record and
`HopRecord::ledger_bytes` the ledger beside it. The
relay connects to no other hop, so every session of it that subscribes is a subscriber
session. Nothing in the record is per object or per viewer, so its size depends on the
room's published tracks and not on its audience. `tests/room_metering.rs` checks that the
record recomputes from the summaries.
## The DVR store
With `[dvr]` configured the relay keeps, on disk, a window of the tracks under each
`[[dvr.window]]` prefix with their receipts, and serves it to FETCH below the live edge (WAI
`extensions/live-recording.md` §8). Its module is [`dvr`](src/dvr.rs).
**What it keeps.** Every track a window's prefix matches (the rule with the longest prefix
keeps it): media tracks, their receipts tracks, events tracks, catalogs, takes. A group is
stored once the cache holds it whole (*What the cache covers*), with every field a FETCH
carries. For a track the store keeps, the cache does not evict a group while a stream of it is
still arriving, past its budget by that group if it must, and it takes a group whose streams
finish after a later group was evicted (coverage is unchanged: nothing at or below an evicted
group is covered), so the store gets every group whatever order its streams finish in.
**What it never keeps.** A room whose objects are sealed (WAI `confidentiality.md`: an
end-to-end encrypted room) is never recorded, and a relay keeps nothing of its tracks beyond
delivery (§9; live-recording §8.2). The store knows such a room by its first object whose
properties carry the secure-objects Key ID (`SECURE_OBJECT_KEY_ID`, read without a key), or
whose properties do not read. That object's group is never written. From then on the store
writes nothing of the room and deletes what it held of it: the room's events and receipts
tracks are in the clear and can be whole before its first sealed group. It also drops the
room's decisions from `decisions.wdl`, while still honouring them for what the cache serves.
The room is recorded in `sealed.wdl` (`room(32) check(32)`, `check` the SHA-256 of `room`), so
a restart keeps refusing it; `rooms_sealed` counts them.
A room in which a minor publishes (WAI `age-assurance.md` §8.4) is never kept either. The relay
knows one by a minor's publishing grant: a session whose SETUP proved a grant of a configured
authority whose `publish` is not 0 and whose attribute `minor` is `true`. From then on the store
treats the room as a sealed one: it writes nothing of it, deletes what it held and its
decisions, and records it in `sealed.wdl`, which so holds every room never kept;
`rooms_sealed` counts these too. A minor's grant that publishes nothing changes nothing.
**The window.** Each `[[dvr.window]]` has a `max_age_s`, from when a group's first object
reached the relay, of 1 to 86 400 (24 hours, the start-over buffer a live-to-VOD harvest clips
from), and a `max_bytes` of payload over its tracks together. Past either the oldest group goes
first, whole, and group *g* of a media track goes with group *g* of its receipts track. A group
that becomes whole after the window has moved past its number is not stored (counted as
`groups_late`).
**On disk.** Under `path`, one directory per track, `tracks/<the first 16 bytes of the SHA-256
of its rendered name, hex>/`, holding `TRACK` (namespace, name, properties, and a SHA-256
check) and the track's **segments**, `s<n, 20 digits>.wds`: files each group's record is
appended to as it is stored, a new one begun every ten seconds or 16 MiB. A segment is a
sequence of frames, `len(u32) state(u8) record`, the state 1 for a record that stands and 0 for
one a later record replaced (a group written again: an object erased, or a subgroup that came
late), whose bytes are then overwritten with zeros. A record is
```text
header "WDG1" 1 track(32) group(u64) arrived_us(u64) count(u32) erased(u32) check(32)
object id(u64) subgroup(u64, 2^64-1 for a datagram) priority(u8) status(u8)
props_len(u32) payload_len(u32) digest(32) check(32) props payload
erased id(u64) digest(32) why(u8: 1 redacted, 2 did not verify)
```
Integers are little-endian. `track` is the SHA-256 of the rendered track; the header's `check`
is SHA-256 over the header before it; an object's `digest` is the content hash its receipt
names it by (jwp-receipts §2.1), and its `check` SHA-256 over its fixed fields before `check`,
its properties and its payload. A segment is deleted once no group's standing record is in it,
so on disk a window holds up to a segment of each track more than its budget. `decisions.wdl`
beside `tracks/` holds the moderation decisions the relay honoured, each a length, the `WEV1`
record and its SHA-256.
**Durability.** The writer syncs every segment it wrote, and the directories it made files in,
at least once a second and before it answers a flush or stops; a shutdown flushes. A crash
loses at most the last second's groups. (One sync per group could not keep up with a room's
audio, a group every 20 ms per publisher.) On start the relay reads every segment again:
- a frame a crash cut short, at a segment's end, is cut off (`torn_writes`);
- a record whose header does not check is dropped;
- every object is checked: its `check` and `digest`; on a media track whose receipts track is
stored, against group *g*'s receipts (the group receipt's signature under the door's trust
for the namespace, or any key that verifies where no door rule covers it; the object's
receipt at its position; its Merkle proof to the signed root; its content hash); on an
events track, every record under its principal. One that does not verify is dropped, counted
(`objects_dropped`, and the `WMH1` record's `dvr_dropped`), and its group written again with
it among the erased. It is never served.
What is below the first group the store holds of a track after a restart is unknown to it.
Each object's digest is checked again as it is read.
**Answering a FETCH.** A FETCH the cache covers is answered from it, as before. Otherwise,
where every group of the range is held whole by the cache or the store, known not to exist, or
evicted from the store, the relay answers from the cache and the store together, groups in the
requested order, reading the store's groups as the fetch stream drains (about a mebibyte at a
time). Otherwise it asks upstream. A FETCH of the receipts track brings each group's receipts
the same way, so a viewer checks a recorded object exactly as a live one.
**What is no longer there.** Evicted groups are marked with an **End of Unknown Range**
(`0x10C`, MOQT §11.4.1.2) through the last of them: a range partly evicted gets what remains
and the marker, one wholly evicted the marker alone. Not an unmarked gap and not an End of
Non-Existent Range: an evicted object existed, and an unmarked gap would say it did not (MOQT
§3.2.2). An End of Range moves to a later group than the entry before it (its Group ID is
delta-coded, as `joule-moq` reads -22), so an object of a group that cannot be served (dropped,
withheld) is marked by one End of Unknown Range from the group's start through the last such
object, and the group's objects after it follow.
**Moderation.** The relay reads the `decision` events (WAI live-events §4.11) of the events
tracks it forwards in a kept room. It honours one whose record verifies, whose principal owns
the track (`[R, P, events]`), and whose principal is a room key (`dvr.room_keys`, or a key of
`[[auth.authorities]]`) or proved a grant with the moderate power on its publishing session.
The last honoured decision on a target stands. Deny and redact withhold the target from every
FETCH and fill, from the store and from the cache, for every session but a moderator's; redact
also erases it from its group's file; allow restores what deny withheld. A media object is
named by its receipt's digest, an event by its record's digest: a withheld event is left out of
the events-track object that carries it, the other records byte for byte, and an object left
with none is withheld whole. Live subscriptions are not filtered here (*Not yet implemented*).
**Metering.** A relay with a store signs `WMH1` version 2 (live-metering §3.9): the objects and
bytes its sessions were served from the store (`dvr_objects`, `dvr_bytes` in each summary's
tracks), and from the store's own series, kept a point a second, the objects and bytes it wrote
for the room, the bytes it held integrated over the span in byte-seconds, and the objects it
dropped. `Dvr::counts` gives the store's counters, `Dvr::figures` a room's figures over a span.
**Configuration.**
```toml
[dvr]
path = "/var/lib/joule-moq-relay/dvr"
room_keys = [] # a room's authority and clock, hex; the authorities' keys count too
[[dvr.window]]
prefix = ["*"] # a field "*" matches any one field
max_age_s = 900
max_bytes = 4294967296
```
The store must be kept where the room's region pin allows (live-rooms §3.7).
## Configuration
`joule-moq-relay --config relay.toml`, or `joule-moq-relay --dev` for loopback with a
development certificate. `relay.example.toml` shows every field: the listen addresses and
path, the certificate files (or the names for a development certificate), the cache's byte
budget, the limits (sessions, requests and streams in flight, MAX_REQUEST_UPDATES, the token
cache, the rendezvous, upstream and fill timeouts, write concurrency, the QUIC send window,
the GOAWAY timeout, the session, ingress and memory limits), the receipts suffix and door
rules, the summary's key and file, the grants and the room authorities, and the DVR store and its
windows. The configuration holds no secret: keys are files, and tokens are named by their
SHA-256. `JOULE_MOQ_RELAY_LOG` sets the log level.
## Not yet implemented
- **SUBSCRIBE_TRACKS** (§3.6): refused with NOT_SUPPORTED.
- **Range Filters** (§3.3.2): the relay declares MAX_FILTER_RANGES 0, so a request that
carries one is INVALID_FILTER, and it evaluates no Track or Object Property filter.
- **WebTransport over HTTP/2** (`draft-ietf-webtrans-http2-15`), the path a browser takes
when HTTP/3 is unavailable. The WebSocket binding carries the same capsules and covers
the same case for now.
- **Parts of `draft-ietf-webtrans-http3-16` the library does not speak:** it negotiates
with the settings deployed browsers send rather than -16's `SETTINGS_WT_ENABLED`, has no
per-session flow-control capsules, and closes a session by closing its QUIC connection
with the code rather than with `WT_CLOSE_SESSION`; and quinn 0.11 has no RESET_STREAM_AT,
so a reset WebTransport stream is not guaranteed to deliver its stream header first.
- **0-RTT** is not accepted.
- **Migration as a client:** the relay sends GOAWAY to its sessions, but does not itself
move its upstream subscriptions to a new relay, or follow a REDIRECT.
- **Pausing upstream** when every downstream subscriber is paused (§7.2), and forwarding
NEW_GROUP_REQUEST upstream. EXPIRES is neither issued nor acted on.
- **TOO_FAR_BEHIND:** a subscription whose subscriber does not read is held, not ended
(§5.2 allows ending it with PUBLISH_DONE TOO_FAR_BEHIND).
- **A cancellation the relay cannot see:** a subscriber that sends FIN on its direction and
later only STOP_SENDING is noticed when the relay next writes on the request stream, or
when the session ends.
- **Receipts:** the receipts track of a media track is found by the configured suffix, not
by reading the catalog's `wai_receipts_track`; the relay signs no `delivery-gap` claim;
datagrams at the door are checked against their group's receipts, not a property.
- **Energy:** no meter backs the relay, so its own figure is always the unmetered marker.
Attributing a metered host's energy to sessions is not done here.
- **The DVR store:** decisions withhold from FETCH and fills, not yet from live subscriptions
(a reviewed item's `held` state, live-moderation §6.3, needs the review policy); objects
are checked against their receipts at a restart, and by the door as they arrive where a door
rule covers them, not again as they are stored; no HLS or DASH presentation of the window is
written (live-recording §8.6 is a mapping).
- **Re-validation** (`moqt-reval`, live-rooms §10.2): a grant is checked at SETUP and on
each request, and its expiry ends the session; revocation through the key log (§5) is not
read.
## Tests
All on loopback, every wait bounded:
| Test | What it shows |
|---|---|
| `relay_quic`, `relay_webtransport`, `relay_websocket` | Two publishers (one by PUBLISH_NAMESPACE and the relay's SUBSCRIBE, one by PUBLISH) and three subscribers over each transport; every object verifies against its publisher's receipts, which the door also checks; SUBSCRIBE_NAMESPACE; every summary verifies and labels the relay's energy unmetered. |
| `slow_subscriber` | A base track at priority 0 and a layer at 200 with a 100 ms delivery timeout. A subscriber that reads nothing for 1.5 s, then reads all its streams concurrently, loses layer objects to the timeout (their streams reset with DELIVERY_TIMEOUT), keeps the whole base track, and receives every receipt of both tracks although it asked for a timeout on its receipts tracks; each dropped object verifies by inclusion; the summary counts each layer object as forwarded or dropped. |
| `late_joiner` | TRACK_STATUS; FETCH of a track and its receipts from the cache; a range in descending order; INVALID_RANGE; a fill on SUBSCRIBE; a group that reaches the relay after a later one, the FETCH from it still served from the cache; a FETCH forwarded upstream when the cache holds nothing. |
| `goaway` | GOAWAY with its URI and timeout to every session; GOING_AWAY for a new request; GOAWAY_TIMEOUT for the session that stays. |
| `auth_tokens` | Tokens in SETUP, by value and by alias; UNAUTHORIZED; UNKNOWN_AUTH_TOKEN_ALIAS. |
| `grant_proof` | A room grant with its proof over each transport gives its rights, each session with a challenge of its own; the grant as a bare token, a proof by another key, one for another relay and one over an earlier session's challenge each close the session with UNAUTHORIZED; a grant presented only in a request is refused it. |
| `request_ids` | INVALID_REQUEST_ID for the wrong parity and for a repeat; requests in flight bounded by the stream limit. |
| `dvr` | A long publish crosses the window: the oldest groups go, whole, each media group with its receipts group, within the byte budget; a window by age empties as its groups pass it. FETCH inside the window (ascending and descending), across it (an End of Unknown Range through the last evicted group, then every object, from the store and the cache together) and before it (the marker alone); every object verifies against the fetched receipts. A restart keeps the store, drops and counts a flipped byte and bytes changed with their checks made again (only the receipt finds those), removes a cut-short write, and never serves what it dropped. A room key's redaction erases its target and withholds it, a deny withholds until an allow, a redacted event leaves the object that carries it, a decision by no moderator changes nothing, and the decisions hold across a restart. Without the seek power a FETCH and a fill below the live edge are refused `seek_not_granted`, and a FETCH at the live edge is served; with it, the window. The record of the room is `WMH1` version 2 with the store's figures. A room whose objects are sealed is never kept: its first sealed group is not written, its events and receipts already held and its decisions are deleted, nothing more of it is written, and so after a restart, while a room in the clear beside it keeps its window. A room in which a session proves a minor's publishing grant is never kept from then on, likewise, while a minor's grant that publishes nothing changes nothing. A window past 24 hours is refused. |
| `age_serving` | A room rated 13 by its authority, the claim on the publisher's PUBLISH, and a stranger's rating ignored: a session at no threshold is refused the subscription `below_threshold`; the performer rates groups 5 to 7 of its track 18 on group 5's object, after group 4 went out, and from then the viewer at 16 is forwarded none of them while the viewer at 18 is forwarded all; a FETCH across the span is answered up to group 4 and one from inside it is refused; a SUBSCRIBE at a live edge rated 18 is refused. |
| `age_rising` | A rating that rises while a subscriber's objects wait at the relay: a viewer at 18 that reads nothing, with small QUIC windows and the relay a small send window, holds groups 0 to 9 queued when the performer rates its whole track 21 on group 10's object; once the relay holds the claim the viewer is given no more of what waited (those streams ended, `CANCELLED`) and nothing of group 10, while a viewer at 21 is given every group. |
| `priority_fan_out` | Two publishers, four tracks, three subscribers with their own subscriber priorities and group orders, their transports held shut while the load arrives: each receives every object in §5.1.2's order. |
| `drafts` | The two-publisher scenario with every session at -18 (over QUIC and over WebTransport), at -16, and with publishers at -16 and -18 and subscribers at -22, -18 and -16: every object verifies against its publisher's receipts at every subscriber. FETCH from the cache at -18 and -16 (the drafts' exclusive End Locations, -16's absolute fetch IDs). A SUBSCRIBE with a Range Filter, refused at an -18 client by name. An -18 session and a -16 session written from those drafts' own messages, not through this crate's client: SETUP each way, a request answered, a Joining FETCH refused with NOT_SUPPORTED, an unknown Fetch Type closing the session with PROTOCOL_VIOLATION; a -16 Request ID ahead of one in flight on another stream admitted and one seen before closing the session with INVALID_REQUEST_ID. An -18 publisher's FIN on PUBLISH_NAMESPACE withdrawing the namespace, a -22 FIN not; an -18 publisher that closes its control stream before its last object and PUBLISH_DONE losing neither. |
| `publish_done_streams` | A PUBLISH_DONE that arrives before the data stream it counts, at -22, -18 (QUIC and WebTransport) and -16: the object still reaches the subscriber, then PUBLISH_DONE; a count no stream meets ends the track after the wait; 0 and the unknown count at once. A close right after PUBLISH_DONE ends the track with the publisher's status, not GOING_AWAY, and without the wait. A track published again after TRACK_ENDED is a new instance. |
## Licence
Apache-2.0.