Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions

joule-moq-relay: a MOQT draft-22 relay over QUIC, WebTransport and a WebSocket fallback #293

Merged
dcharlot merged 19 commits from joule-moq/relay into main 2026-10-18 12:37:19 -04:00
Conversation 0 Commits 19 Files changed 39 +9435 -37
39 changed files with 9435 additions and 37 deletions
Download patch file Download diff file Expand all files Collapse all files

34
.github/workflows/joule-code-conformance.yml vendored
View file

@ -91,11 +91,13 @@ jobs:
$SUDO apt-get install -y -qq z3
z3 --version
# joule-moq-relay runs in the job of its own below, on linux-arm64: it needs
# nothing x86-specific, and this runner is the one there is least of.
- name: Reference library tests
working-directory: joule-code/joule-code-rs
env:
JOULE_FIX_EXPECT_Z3: '1'
run: cargo test --workspace
run: cargo test --workspace --exclude joule-moq-relay
# joule-moq's QUIC transport and its example compile only with `quic`. Named
# here so they stay built and tested whatever the workspace's other members
@ -138,3 +140,33 @@ jobs:
- name: L3 JouleClaw binding example builds + tests
working-directory: joule-code/examples/l3-jouleclaw
run: cargo test
# The MOQT draft-22 relay. Every test is on loopback: two publishers and three
# subscribers over native QUIC, WebTransport over HTTP/3 and the WebSocket fallback;
# a slow subscriber whose late layers time out while every receipt still arrives; a
# late joiner served by FETCH and a fill from the cache, and a FETCH forwarded
# upstream; GOAWAY on shutdown; tokens by alias; Request IDs and the stream limit;
# fan-out order under load. `--all-targets` runs the unit tests and the binary's
# build as well.
moq-relay:
name: joule-moq-relay, every target
runs-on: linux-arm64
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Rust 1.98.0
shell: bash
run: |
export PATH="$HOME/.cargo/bin:$PATH"
command -v rustup >/dev/null 2>&1 || \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
rustup toolchain install 1.98.0 --profile minimal
rustup default 1.98.0
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
- name: joule-moq-relay, every target
working-directory: joule-code/joule-code-rs
shell: bash
run: cargo test --locked -p joule-moq-relay --all-targets

27
joule-code/CHANGELOG.md
View file

@ -57,6 +57,33 @@ object framing. The crate's version moves to 0.3.0 because its public API change
- `tests/vectors/draft-22/`: a corpus of -22 vectors, good and negative, generated
by `tests/draft22_vectors.rs`, and an independent decoder that checks it in CI.
### Added — `joule-moq-relay` 0.1.0: a `draft-ietf-moq-transport-22` relay
A new crate, library and binary, on `joule-moq`'s codec. Publishers send it tracks,
subscribers ask it for them, and it fans each object out to every subscription in the
order -22's priorities give (§5.1.2), dropping what passes a subscriber's delivery timeout
at this hop (§5.2). Whatever it drops, the object's receipts still reach the subscriber:
a track's receipts track is never given a delivery timeout.
- **Three transports, one draft:** native QUIC (ALPN `moqt-22`) and WebTransport over
HTTP/3 (`WT-Available-Protocols` / `WT-Protocol`, `draft-ietf-webtrans-http3-16`) on one
UDP endpoint, and a WebSocket fallback that carries `draft-ietf-webtrans-http2-15`
capsules, one per binary message, under the subprotocol `moqt-22`. The fallback is a
non-normative binding, written out in the crate's README with example bytes.
- **The relay protocol:** session setup; namespace publish and subscribe routing;
PUBLISH; SUBSCRIBE served from an upstream the relay holds or opens; FETCH and fills
from its cache or forwarded upstream; TRACK_STATUS; REQUEST_UPDATE; GOAWAY on shutdown;
Request-ID parity and duplicate checks; the authorization-token alias cache; several
publishers of one track, deduplicated by (group, subgroup).
- **What the cache covers** is worked out from the groups it holds whole, each time it is
asked, so a group whose streams finish after a later group's is still served from the
cache once it is whole; a group never seen is a gap it does not answer across.
- **A signed session summary** per session. The relay's own energy is the unmetered
marker, since no meter backs it.
- **Not yet:** SUBSCRIBE_TRACKS and Range Filters, WebTransport over HTTP/2, 0-RTT, the
relay moving its own upstream subscriptions, and finding a receipts track from the
catalog rather than by its name; the README lists each.
### Changed — `joule-moq` 0.2.0: WAI's labelled JWP forms, and a strict verifier
`joule-moq` reads and writes the forms of WAI's `extensions/jwp-receipts.md`

719
joule-code/joule-code-rs/Cargo.lock generated
View file

File diff suppressed because it is too large Load diff

1
joule-code/joule-code-rs/Cargo.toml
View file

@ -17,6 +17,7 @@ members = [
"crates/joule-code-identity",
"crates/jwp",
"crates/joule-moq",
"crates/joule-moq-relay",
"crates/lockstep-quic",
]

43
joule-code/joule-code-rs/crates/joule-moq-relay/Cargo.toml Normal file
View file

@ -0,0 +1,43 @@
[package]
name = "joule-moq-relay"
version = "0.1.0"
description = "A MOQT relay for draft-ietf-moq-transport-22: native QUIC and WebTransport over HTTP/3 on one UDP port, a WebSocket fallback, priority-ordered fan-out with per-subscriber delivery timeouts, and every object's receipts forwarded whether the object was delivered or dropped."
edition = "2024"
rust-version = "1.97"
license = "Apache-2.0"
repository = "https://github.com/transaction-science/open-standards"
[lib]
path = "src/lib.rs"
[[bin]]
name = "joule-moq-relay"
path = "src/main.rs"
[dependencies]
joule-moq = { path = "../joule-moq" }
quinn = "0.11"
rustls = { version = "0.23", default-features = false, features = ["ring", "std"] }
ring = "0.17"
# WebTransport over HTTP/3 (MIT OR Apache-2.0). Its `quinn` feature lets one quinn
# endpoint serve both the native MOQT ALPN and `h3`.
wtransport = { version = "0.7.2", default-features = false, features = ["ring", "self-signed", "quinn"] }
tokio-tungstenite = "0.29"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "time", "net", "io-util", "signal"] }
bytes = "1"
futures = "0.3"
async-trait = "0.1"
tracing = "0.1"
tracing-subscriber = "0.3"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
toml = "0.9"
ed25519-dalek = { version = "2", features = ["rand_core"] }
rand_core_06 = { package = "rand_core", version = "0.6", features = ["getrandom"] }
blake3 = "1"
hex = "0.4"
base64 = "0.22"
thiserror = "2"
[dev-dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

336
joule-code/joule-code-rs/crates/joule-moq-relay/README.md Normal file
View file

@ -0,0 +1,336 @@
# joule-moq-relay
A relay for MOQT, `draft-ietf-moq-transport-22`. Publishers send it tracks, subscribers
ask it for them, and it fans each object out to every subscription, in the order the
draft's priorities give, dropping what passes a subscriber's delivery timeout at this hop.
Whatever it drops, the object's receipts still reach the subscriber, on the track's
receipts track.
It is a library (`Relay`, `RelayServer`, `client::Client`) and a binary,
`joule-moq-relay`. The wire codec is `joule-moq`'s.
## The draft it speaks
`draft-ietf-moq-transport-22`, and no other. Drafts do not interoperate, so a session
speaks one from its first byte. The draft is chosen before any MOQT byte:
- by the ALPN `moqt-22` on native QUIC (§6.2.2);
- by `WT-Available-Protocols: "moqt-22"` on WebTransport, which the relay answers with
`WT-Protocol: "moqt-22"` (§6.2.1; `draft-ietf-webtrans-http3-16` §3.3). A client that
sends no `WT-Available-Protocols` is taken to speak -22; one that offers other
protocols only is refused;
- by the subprotocol `moqt-22` on the WebSocket fallback (below).
In its SETUP the relay declares `MOQT_IMPLEMENTATION`, `MAX_REQUEST_UPDATES` and
`MAX_AUTH_TOKEN_CACHE_SIZE`. Where the client declares `joule-moq`'s receipt extension
(Setup Option `0x3A46`, its value the extension version), the relay declares it back.
## Transports
| Path | Listener | Selected by |
|---|---|---|
| Native QUIC | UDP `quic_listen` | ALPN `moqt-22` |
| WebTransport over HTTP/3 | the same UDP port | ALPN `h3`, extended CONNECT to `path`, `WT-Available-Protocols` |
| WebSocket fallback | TCP `websocket_listen` | upgrade at `path` with subprotocol `moqt-22` |
Native QUIC and WebTransport share one quinn endpoint: the relay reads the ALPN of each
connection and either runs MOQT on it directly or hands it to the WebTransport layer.
**The WebTransport implementation** is `wtransport` 0.7 (MIT OR Apache-2.0). It was
chosen because it runs on the same quinn 0.11 as `joule-moq`, so one UDP endpoint serves
both ALPNs; it gives the server the CONNECT request's headers and lets it answer with its
own, which is what `WT-Available-Protocols` and `WT-Protocol` need; and its client checks
a certificate by hash under the rules a browser applies to `serverCertificateHashes`
(ECDSA P-256, a validity of fourteen days at most), so the relay's tests connect the way a
page does. Stream bytes are read and written on the quinn streams underneath; application
error codes on WebTransport streams are mapped into the HTTP/3 error space as
`draft-ietf-webtrans-http3-16` §4.4 gives.
## How a browser connects
**WebTransport, in development.** Start the relay with `--dev` (or with no certificate
files in the configuration). It makes a certificate a browser accepts by hash and logs its
SHA-256 three ways, including a ready `serverCertificateHashes` entry:
```js
const transport = new WebTransport("https://127.0.0.1:4443/moq", {
serverCertificateHashes: [{ algorithm: "sha-256", value: new Uint8Array([/* printed */]) }],
protocols: ["moqt-22"],
});
await transport.ready; // transport.protocol === "moqt-22"
```
The certificate is valid fourteen days from the relay's start; restart the relay to make a
new one. In production give it a certificate chain and key from files, and a page connects
without `serverCertificateHashes`.
**WebSocket, where there is no UDP path.** A site behind a front that carries HTTP over
TCP only reaches no UDP endpoint, so neither QUIC path is open to its pages. Such a page
uses the fallback:
```js
const ws = new WebSocket("wss://relay.example/moq", "moqt-22");
ws.binaryType = "arraybuffer"; // each message is one capsule, below
```
The WebSocket listener speaks plain HTTP; TLS is the front's.
## The WebSocket binding (non-normative)
This binding is this crate's, not a standard's. It carries what a WebTransport session over
HTTP/2 carries (`draft-ietf-webtrans-http2-15`), on a WebSocket instead of an extended
CONNECT stream, so that a client written for one is most of the way to the other.
1. **Opening.** The client opens a WebSocket (RFC 6455) to the relay's `path`, offering
the subprotocol `moqt-22` in `Sec-WebSocket-Protocol`. The relay answers with that
subprotocol, or refuses the upgrade (`400` without it, `404` at another path).
2. **Messages.** Every message is a binary message holding exactly one capsule: `Type (i)
Length (i) Value`, with QUIC variable-length integers (RFC 9000 §16), not MOQT's. A
text message carries nothing and is ignored. A binary message that is not one whole
capsule closes the session.
3. **Capsules.** Those of `draft-ietf-webtrans-http2-15` §6, with their type codes:
| Capsule | Type | Value | Use here |
|---|---|---|---|
| `WT_STREAM` | `0x190B4D3C`, `0x190B4D3B` with FIN | Stream ID, data | stream data; a stream opens with its first `WT_STREAM` |
| `WT_RESET_STREAM` | `0x190B4D39` | Stream ID, code, Reliable Size | reset a stream (Reliable Size 0) |
| `WT_STOP_SENDING` | `0x190B4D3A` | Stream ID, code | ask the peer to stop |
| `WT_MAX_STREAMS` | `0x190B4D3F` bidi, `0x190B4D40` uni | Maximum Streams | stream-count credit |
| `WT_STREAMS_BLOCKED` | `0x190B4D43`, `0x190B4D44` | Maximum Streams | sent when out of credit; informational |
| `DATAGRAM` | `0x00` | payload | one MOQT datagram, reliable and ordered |
| `WT_CLOSE_SESSION` | `0x2843` | code (32 bits), message (UTF-8, at most 1,024 bytes) | close with a MOQT session termination code |
| `WT_DRAIN_SESSION` | `0x78AE` | empty | read and ignored; MOQT's GOAWAY is the drain |
| others | | | read and ignored, as unknown capsules are |
4. **Streams.** Stream IDs are numbered as QUIC numbers them: bit 0 the initiator (0 the
client), bit 1 the direction (0 bidirectional), and the `n`th stream of a kind is
`4n + kind`. MOQT's control streams, request streams and data streams ride them
byte for byte as they would ride QUIC streams: the client's control stream is its
first unidirectional stream (ID 2), beginning with SETUP; a request is a bidirectional
stream; a subgroup, fetch or padding stream is a unidirectional one.
5. **Stream credit.** Each end first sends its `WT_MAX_STREAMS` for both kinds; until the
peer's arrive, an end opens nothing. Credit is cumulative, as in QUIC. A stream's credit
comes back once both of its halves are done, and is granted in batches. Opening past
the limit closes the session with code `0x045D4487`. Because MOQT -22 bounds requests in
flight by stream limits alone (§6.4.2), the relay's bidirectional limit
(`max_requests`) is its bound on requests in flight on this binding, as MAX_STREAMS is
on QUIC.
6. **Data flow control.** None at the capsule level (`WT_MAX_DATA`, `WT_MAX_STREAM_DATA`
and their `BLOCKED` capsules are read and ignored): TCP's backpressure bounds a sender,
and a receiver that holds more than `websocket_buffer_bytes` unread closes the session
with `0x045D4487`.
7. **Closing.** `WT_CLOSE_SESSION` with a MOQT session termination code (§12.2), then the
WebSocket's close. A WebSocket that closes without one is a session closed with code 0.
The first capsules a relay with the default limits sends are its credit:
```text
99 0b 4d 3f 02 41 00 WT_MAX_STREAMS (bidirectional), 256
99 0b 4d 40 02 44 00 WT_MAX_STREAMS (unidirectional), 1024
```
and a session it closes at the end of a GOAWAY's timeout ends with:
```text
68 43 12 00 00 00 10 47 4f 41 57 41 59 20 74 69 6d 65 6f 75 74
WT_CLOSE_SESSION, GOAWAY_TIMEOUT (0x10), "GOAWAY timeout"
```
## What the relay does
| -22 | Here |
|---|---|
| §6.3 session initialization | Each end's control stream opens with SETUP; data streams that come first wait for it. A client's SETUP is held to §9.1's rules for a client (no PATH or AUTHORITY over WebTransport; no DELETE or USE_ALIAS token). The relay's control stream carries only its SETUP and a GOAWAY; the end of the peer's closes the session with PROTOCOL_VIOLATION. |
| §6.4.2 requests | A request stream must begin with one of the seven requests. A Request ID of the wrong parity or seen before closes the session with INVALID_REQUEST_ID. MAX_REQUEST_UPDATES is enforced per stream; requests in flight are bounded by the stream limit (`max_requests`). |
| §8.9, §9.1.3, §9.1.4 tokens | A receiver's alias cache per session: REGISTER, DELETE, USE_ALIAS, USE_VALUE; AUTH_TOKEN_CACHE_OVERFLOW, DUPLICATE_AUTH_TOKEN_ALIAS, UNKNOWN_AUTH_TOKEN_ALIAS; a registration in SETUP past the cache reads as USE_VALUE. Grants name a namespace prefix, rights to publish and subscribe, and the SHA-256 of a token of type 0. |
| §4.1, §4.2 namespaces | PUBLISH_NAMESPACE is accepted (REQUEST_OK) and stands until it is cancelled or its session ends. SUBSCRIBE_NAMESPACE answers with NAMESPACE for each published namespace under the prefix, then NAMESPACE and NAMESPACE_DONE as they change, each carrying the suffix after the prefix; an overlapping prefix in one session is PREFIX_OVERLAP. |
| §7.6, §9.8 PUBLISH | Accepted with REQUEST_OK; its Track Alias is the publisher's for its data. A second track under one alias in one session closes it with DUPLICATE_TRACK_ALIAS. PUBLISH_DONE ends the track's downstream subscriptions. |
| §7.4, §7.6, §9.6 SUBSCRIBE | SUBSCRIBE_OK only once an upstream subscription is Established (§7.4): one the relay holds, or a SUBSCRIBE it sends to every publisher of a matching namespace. The relay's own upstream SUBSCRIBE asks for every object at priority 128 with no timeout, copying no subscriber's priority, group order or timeout (§5.1.3, §9.20.3, §9.20.4). RENDEZVOUS_TIMEOUT holds a SUBSCRIBE for a publisher, up to `max_rendezvous_ms`; without it a track with no publisher is DOES_NOT_EXIST. Location Filters set where the subscription starts and ends. A publisher of a namespace that arrives later is sent a SUBSCRIBE for every subscription the relay holds under it. When the last subscriber leaves, the relay cancels its own upstream SUBSCRIBE. |
| §9.5 REQUEST_UPDATE | Subscriber priority (re-applied to what is queued), FORWARD, the Location Filter and both delivery timeouts. |
| §3.2, §3.4, §7.1, §9.11 FETCH and fills | A fetch the cache covers is served from it, groups in the requested order (coverage, below); otherwise it goes to a publisher of the track or its namespace and its fetch stream is piped through, past the Fill Timeout a Timed-Out range. A range that starts after the Largest Object is INVALID_RANGE. A SUBSCRIBE with FILL_PARAMETERS gets its fill range the same way on a fill fetch stream named by its Request ID; a fill it cannot serve is a fetch stream reset after its header (§3.4.1). |
| §9.13 TRACK_STATUS | Answered from what the relay holds: the Largest Object and the track's properties. |
| §7.3 several publishers | A track may have several upstreams; each (group, subgroup) is taken from the first that sends it, and the rest are counted as deduplicated. |
| §5.1 priorities, §5.2 delivery timeouts | Below. |
| §6.6.1, §9.2 GOAWAY | On shutdown every session gets GOAWAY with `goaway_uri` and the timeout; new requests are refused with GOING_AWAY; a session still open at the timeout is closed with GOAWAY_TIMEOUT. |
| §6.5 `.session` | No session-level track is defined: a request for one is DOES_NOT_EXIST. |
### What the cache covers
The cache answers a fetch alone only where it holds every object of the range, so that a
gap in its answer means an object that does not exist (§3.2.2). It keeps what it knows of
each group's arrival. A group is whole when its start arrived (Object ID 0, or a Prior
Object ID Gap saying the objects before do not exist, §10.9), none of its subgroup streams
is open or was reset, and its end is known: its streams ended with a FIN, or an End of
Group or End of Track status, or a datagram with END_OF_GROUP, ended it. A fetch from a
group is covered when every group from it up to the group of the Largest Object is whole
or known not to exist (a Prior Group ID Gap, §10.8), that last group has its start, and
nothing in between was evicted. Coverage is worked out from the groups held each time it
is asked, so a group whose streams finish after a later group's still counts once it is
whole; a group ID never seen is a gap the cache does not answer across. The relay cannot
tell a subgroup it has not yet seen from one that does not exist, so a group whose
subgroups have all closed counts as whole.
### Priorities and delivery timeouts
Each downstream session has a scheduler. Every data stream it will send (a subgroup, a
fetch or fill, a datagram) is a lane whose items go out in order and one at a time, so no
stream's objects are reordered (§7.4). Among the lanes with something to send it takes the
item §5.1.2 puts first: the lower subscriber priority number; then the lower publisher
priority number; then, within a subscription, the group its Group Order puts first (and in
one group a fill before its subscription); then the lower Subgroup ID or, for a datagram,
Object ID, a datagram before a subgroup object. At most `write_concurrency` writes are in
flight, and a QUIC session's send window is `quic_send_window_bytes`: what waits, waits in
the scheduler, where it can still be ordered and dropped (§5.2: "minimize the amount of
data buffered at the underlying transport layer"). Each stream is also given a transport
priority from its subscriber and publisher priorities.
An object's deadline is its OBJECT_DELIVERY_TIMEOUT from when it reached the relay: the
smaller of the publisher's (the first object of its subgroup, else the track's) and the
subscriber's, where both are set. It is checked before the object goes to the transport and
bounds both opening its stream and writing it. Past it the stream is reset with
DELIVERY_TIMEOUT and nothing more of that subgroup goes to that subscriber; a datagram is
dropped. SUBGROUP_DELIVERY_TIMEOUT runs from the subgroup's FIN until all its data is
acknowledged, and resets the stream if it is not. On the WebSocket binding a FIN counts as
acknowledged once written, as webtrans-http2 §5.2 makes the transitions an acknowledgement
would make at once.
### A subscriber that does not read
QUIC's connection-level flow control is shared by every stream of a session (RFC 9000
§4.1), and bytes a subscriber has received but not read hold it. So yes: a subscriber that
stops reading stalls its own high-priority track along with everything else, until it reads
again. And a subscriber that reads its streams one at a time, in the order they arrive, can
stall itself while still reading: the stream it waits on may have its bytes held back for
credit that the unread streams behind it are holding. A subscriber reads its streams
concurrently.
The relay cannot release credit the subscriber holds, and does not try to by dropping
what has no deadline. What it does:
- An object with no delivery timeout (a base track, a receipts track) is never dropped. It
waits, in the scheduler or the transport, and is delivered once the subscriber reads.
- What can still wait, waits in the relay rather than in the transport (§5.2: "minimize the
amount of data buffered at the underlying transport layer"). Writes in flight are
bounded by `write_concurrency` and the transport's unacknowledged bytes by
`quic_send_window_bytes`. So an object whose deadline passes while it waits is dropped
before it takes any credit.
- An object that passes its deadline is reset. A reset stream's final size settles its
account (RFC 9000 §4.5), and a receiver can credit its unread bytes back to the
connection at once, as quinn does. The credit a dropped layer held then comes back
without the subscriber reading it.
- When credit frees, the highest-priority bytes go first: each stream carries its
subscriber and publisher priority as its transport priority.
MOQT lets a publisher end a subscription whose subscriber falls too far behind, with
PUBLISH_DONE TOO_FAR_BEHIND (§5.2); this relay does not yet.
### Receipts
A media track's receipts ride its **receipts track** (WAI `extensions/moq-streaming-format.md`
§4.1): group *g* of it is one object holding the group receipt and the object receipts of
media group *g*. The relay recognises a receipts track by its name: the media track's name
followed by `receipts_track_suffix` (`/receipts` unless configured).
- **The receipts of dropped objects are forwarded.** A receipts track has no delivery
timeout, and the relay applies none to it, even one a subscriber asks for. So when an
object of a media track passes its deadline, its receipt still reaches every subscriber of
that track's receipts track, which can account for the object by inclusion (WAI
`extensions/jwp-receipts.md` §4.1, §4.4). A receipt that rides on the object itself, as a
property, is lost with it.
- **Elsewhere receipts are opaque.** Outside the namespaces a door rule names, the relay
forwards receipts tracks as it forwards any track and reads nothing in them (§7.8).
- **At the door.** In a namespace a `[[receipts.door]]` rule names, the relay reads the
receipts it forwards, and takes on jwp-receipts §4.1's duties: each group receipt's
signature is verified, under the rule's trusted keys or any key, before the first object
of its group is fanned out, and a group that does not verify is dropped with a structured
error in the log; each media object must match the receipt of its position (its group,
its Object ID, its content hash, its Merkle proof to the signed root), and an object that
carries its receipt as a property must carry that same one. An object waits for its
group's receipts up to `hold_ms`. Receipt bytes are forwarded unaltered and nothing is
re-signed. A refused object ends its downstream streams rather than leaving a hole in
them.
## The session summary
When a session ends the relay signs a summary of it with its own Ed25519 key (from
`summary.key_path`, or made at start and its public half logged), keeps it
(`Relay::summaries`), and appends it as one JSON line to `summary.out_path` if set. It
holds, for the session:
- as a subscriber: objects and their bytes forwarded, every byte written to its data
streams, objects dropped for their delivery timeout and how many of those were on a
track whose receipts track the session also held, subgroups reset after their FIN,
objects served by fetch or fill, objects still queued at the end, and the same per
track;
- as a publisher: objects and bytes received, objects refused at the door, objects
deduplicated;
- `relay_energy`: `{"acquisition": "unmetered", "joules_micro": 0}`. No meter backs this
relay. The 0 is the unmetered marker of WAI `extensions/energy-measurement.md` §2, never
a measurement of zero joules, and must not be read or shown as one;
- `publisher_signed`: where the door verified an object's receipt, the publisher's signed
figure for producing it, summed under the acquisition class its leaf was signed with.
These are the publisher's statements about the objects, not the relay's energy.
The summary is relay-scoped telemetry (jwp-receipts §4.2), not a receipt of the publisher's.
Its format is `joule-moq-relay/session-summary/1`; the signature covers
`joule-moq-relay:session-summary\x01` followed by the summary's JSON bytes, and
`SignedSessionSummary::verify` checks it.
## Configuration
`joule-moq-relay --config relay.toml`, or `joule-moq-relay --dev` for loopback with a
development certificate. `relay.example.toml` shows every field: the listen addresses and
path, the certificate files (or the names for a development certificate), the cache's byte
budget, the limits (sessions, requests and streams in flight, MAX_REQUEST_UPDATES, the token
cache, the rendezvous, upstream and fill timeouts, write concurrency, the QUIC send window,
the GOAWAY timeout), the receipts suffix and door rules, the summary's key and file, and the
grants. The configuration holds no secret: keys are files, and tokens are named by their
SHA-256. `JOULE_MOQ_RELAY_LOG` sets the log level.
## Not yet implemented
- **SUBSCRIBE_TRACKS** (§3.6): refused with NOT_SUPPORTED.
- **Range Filters** (§3.3.2): the relay declares MAX_FILTER_RANGES 0, so a request that
carries one is INVALID_FILTER, and it evaluates no Track or Object Property filter.
- **WebTransport over HTTP/2** (`draft-ietf-webtrans-http2-15`), the path a browser takes
when HTTP/3 is unavailable. The WebSocket binding carries the same capsules and covers
the same case for now.
- **Parts of `draft-ietf-webtrans-http3-16` the library does not speak:** it negotiates
with the settings deployed browsers send rather than -16's `SETTINGS_WT_ENABLED`, has no
per-session flow-control capsules, and closes a session by closing its QUIC connection
with the code rather than with `WT_CLOSE_SESSION`; and quinn 0.11 has no RESET_STREAM_AT,
so a reset WebTransport stream is not guaranteed to deliver its stream header first.
- **0-RTT** is not accepted.
- **Migration as a client:** the relay sends GOAWAY to its sessions, but does not itself
move its upstream subscriptions to a new relay, or follow a REDIRECT.
- **Pausing upstream** when every downstream subscriber is paused (§7.2), and forwarding
NEW_GROUP_REQUEST upstream. EXPIRES is neither issued nor acted on.
- **TOO_FAR_BEHIND:** a subscription whose subscriber does not read is held, not ended
(§5.2 allows ending it with PUBLISH_DONE TOO_FAR_BEHIND).
- **A cancellation the relay cannot see:** a subscriber that sends FIN on its direction and
later only STOP_SENDING is noticed when the relay next writes on the request stream, or
when the session ends.
- **Receipts:** the receipts track of a media track is found by the configured suffix, not
by reading the catalog's `wai_receipts_track`; the relay signs no `delivery-gap` claim;
datagrams at the door are checked against their group's receipts, not a property.
- **Energy:** no meter backs the relay, so its own figure is always the unmetered marker.
Attributing a metered host's energy to sessions is not done here.
- **The browser client** and a page that uses it.
## Tests
All on loopback, every wait bounded:
| Test | What it shows |
|---|---|
| `relay_quic`, `relay_webtransport`, `relay_websocket` | Two publishers (one by PUBLISH_NAMESPACE and the relay's SUBSCRIBE, one by PUBLISH) and three subscribers over each transport; every object verifies against its publisher's receipts, which the door also checks; SUBSCRIBE_NAMESPACE; every summary verifies and labels the relay's energy unmetered. |
| `slow_subscriber` | A base track at priority 0 and a layer at 200 with a 100 ms delivery timeout. A subscriber that reads nothing for 1.5 s, then reads all its streams concurrently, loses layer objects to the timeout (their streams reset with DELIVERY_TIMEOUT), keeps the whole base track, and receives every receipt of both tracks although it asked for a timeout on its receipts tracks; each dropped object verifies by inclusion; the summary counts each layer object as forwarded or dropped. |
| `late_joiner` | TRACK_STATUS; FETCH of a track and its receipts from the cache; a range in descending order; INVALID_RANGE; a fill on SUBSCRIBE; a group that reaches the relay after a later one, the FETCH from it still served from the cache; a FETCH forwarded upstream when the cache holds nothing. |
| `goaway` | GOAWAY with its URI and timeout to every session; GOING_AWAY for a new request; GOAWAY_TIMEOUT for the session that stays. |
| `auth_tokens` | Tokens in SETUP, by value and by alias; UNAUTHORIZED; UNKNOWN_AUTH_TOKEN_ALIAS. |
| `request_ids` | INVALID_REQUEST_ID for the wrong parity and for a repeat; requests in flight bounded by the stream limit. |
| `priority_fan_out` | Two publishers, four tracks, three subscribers with their own subscriber priorities and group orders, their transports held shut while the load arrives: each receives every object in §5.1.2's order. |
## Licence
Apache-2.0.

62
joule-code/joule-code-rs/crates/joule-moq-relay/relay.example.toml Normal file
View file

@ -0,0 +1,62 @@
# joule-moq-relay: an example configuration. Every field is optional; these are the
# defaults unless a comment says otherwise. Nothing secret belongs here: keys are files
# the operator provides, and tokens are named by their SHA-256.
# Native QUIC (ALPN moqt-22) and WebTransport over HTTP/3 (ALPN h3), one UDP port.
quic_listen = "127.0.0.1:4443"
# The WebSocket fallback (plain HTTP upgrade; put TLS in front of it).
websocket_listen = "127.0.0.1:4480"
# The path of the WebTransport CONNECT and of the WebSocket upgrade.
path = "/moq"
# The New Session URI a shutdown GOAWAY names; empty reuses the current one.
goaway_uri = ""
[tls]
# cert_path = "/etc/joule-moq-relay/fullchain.pem"
# key_path = "/etc/joule-moq-relay/key.pem"
# Without files: a development certificate for these names, its SHA-256 printed.
dev_names = ["localhost", "127.0.0.1", "::1"]
[cache]
max_bytes = 268435456
[limits]
max_sessions = 4096
max_requests = 256
max_uni_streams = 1024
max_request_updates = 16
max_auth_token_cache_size = 16384
max_rendezvous_ms = 30000
upstream_timeout_ms = 5000
fill_timeout_ms = 2000
write_concurrency = 8
max_object_bytes = 16777216
unknown_alias_wait_ms = 1000
goaway_timeout_ms = 5000
websocket_buffer_bytes = 67108864
idle_timeout_ms = 30000
quic_send_window_bytes = 1048576
[receipts]
receipts_track_suffix = "/receipts"
# Check objects in this namespace against their receipts at the door (example; not a
# default). An empty trust list accepts any key whose signature verifies.
[[receipts.door]]
prefix = ["example", "live"]
trust = []
hold_ms = 2000
[summary]
# key_path = "/etc/joule-moq-relay/summary-seed"
# out_path = "/var/log/joule-moq-relay/sessions.jsonl"
[auth]
required = false
# An example grant; the hash below is of the placeholder string "replace-me".
[[auth.grants]]
prefix = ["example"]
token_sha256 = "1df4b07d05b9b3b1115b4b6b22af319a2f99564a2fa2573a34896063c7513c10"
publish = true
subscribe = true

197
joule-code/joule-code-rs/crates/joule-moq-relay/src/auth.rs Normal file
View file

@ -0,0 +1,197 @@
//! Authorization tokens: the alias cache a receiver keeps for each session
//! (`draft-ietf-moq-transport-22` §8.9, §9.1.3, §9.1.4), and the relay's grants.
//!
//! A grant names a namespace prefix, the rights it gives (publish, subscribe), and the
//! SHA-256 of the token value that carries it. Tokens are of type 0, agreed out of band
//! (§8.9: "Type 0 is reserved to indicate that the type is not defined in the table and
//! is negotiated out-of-band").
use crate::config::AuthConfig;
use joule_moq::draft22::names::prefix_matches;
use joule_moq::draft22::{SessionError, Token};
use joule_moq::TrackNamespace;
use std::collections::HashMap;
/// A token's size against MAX_AUTH_TOKEN_CACHE_SIZE: 16 bytes and its value (§9.1.3).
fn size_of(value: &[u8]) -> u64 {
16 + value.len() as u64
}
/// One token as a message resolves it: its type and value.
pub type Resolved = (u64, Vec<u8>);
/// The aliases one peer has registered with this endpoint.
#[derive(Debug)]
pub struct TokenCache {
max: u64,
used: u64,
entries: HashMap<u64, Resolved>,
}
impl TokenCache {
/// A cache bounded by the MAX_AUTH_TOKEN_CACHE_SIZE this endpoint declared.
pub fn new(max: u64) -> Self {
TokenCache { max, used: 0, entries: HashMap::new() }
}
pub fn used(&self) -> u64 {
self.used
}
/// The tokens of the peer's SETUP. A REGISTER past the limit is read as USE_VALUE,
/// never a failure (§9.1.4); a duplicate alias still closes the session. A client's
/// SETUP carries no DELETE or USE_ALIAS (`Setup::check_from` refuses them first).
pub fn setup(&mut self, tokens: &[Token]) -> Result<Vec<Resolved>, SessionError> {
let mut out = Vec::new();
for t in tokens {
match t {
Token::Register { alias, token_type, value } => {
if self.entries.contains_key(alias) {
return Err(SessionError::DUPLICATE_AUTH_TOKEN_ALIAS);
}
if self.used + size_of(value) <= self.max {
self.used += size_of(value);
self.entries.insert(*alias, (*token_type, value.clone()));
}
out.push((*token_type, value.clone()));
}
Token::UseValue { token_type, value } => out.push((*token_type, value.clone())),
Token::Delete { .. } | Token::UseAlias { .. } => return Err(SessionError::PROTOCOL_VIOLATION),
}
}
Ok(out)
}
/// The tokens of one request: registered, retired or resolved. A REGISTER that would
/// pass the limit closes the session with AUTH_TOKEN_CACHE_OVERFLOW, and a duplicate
/// alias with DUPLICATE_AUTH_TOKEN_ALIAS; a reference to an alias not registered is
/// UNKNOWN_AUTH_TOKEN_ALIAS (§8.9). A REGISTER is kept even when the request it rides
/// fails for another reason.
pub fn resolve(&mut self, tokens: &[Token]) -> Result<Vec<Resolved>, SessionError> {
let mut out = Vec::new();
for t in tokens {
match t {
Token::Register { alias, token_type, value } => {
if self.entries.contains_key(alias) {
return Err(SessionError::DUPLICATE_AUTH_TOKEN_ALIAS);
}
if self.used + size_of(value) > self.max {
return Err(SessionError::AUTH_TOKEN_CACHE_OVERFLOW);
}
self.used += size_of(value);
self.entries.insert(*alias, (*token_type, value.clone()));
out.push((*token_type, value.clone()));
}
Token::Delete { alias } => match self.entries.remove(alias) {
Some((_, v)) => self.used -= size_of(&v),
None => return Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS),
},
Token::UseAlias { alias } => match self.entries.get(alias) {
Some(r) => out.push(r.clone()),
None => return Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS),
},
Token::UseValue { token_type, value } => out.push((*token_type, value.clone())),
}
}
Ok(out)
}
}
/// What a request asks to do.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Right {
Publish,
Subscribe,
}
struct GrantRule {
prefix: TrackNamespace,
token_sha256: [u8; 32],
publish: bool,
subscribe: bool,
}
/// The relay's grants.
pub struct Authorizer {
required: bool,
grants: Vec<GrantRule>,
}
pub fn sha256(b: &[u8]) -> [u8; 32] {
crate::cert::sha256(b)
}
impl Authorizer {
pub fn from_config(c: &AuthConfig) -> Result<Self, String> {
let mut grants = Vec::new();
for g in &c.grants {
let h: [u8; 32] = hex::decode(&g.token_sha256)
.map_err(|e| format!("a grant's token_sha256: {e}"))?
.try_into()
.map_err(|_| "a grant's token_sha256 is not 32 bytes".to_string())?;
grants.push(GrantRule { prefix: TrackNamespace::from_strs(g.prefix.iter().map(String::as_str)), token_sha256: h, publish: g.publish, subscribe: g.subscribe });
}
Ok(Authorizer { required: c.required, grants })
}
pub fn open() -> Self {
Authorizer { required: false, grants: Vec::new() }
}
/// Whether `tokens` (the request's, and the session's from SETUP) give `right` over
/// `ns`.
pub fn allows(&self, ns: &TrackNamespace, right: Right, tokens: &[Resolved]) -> bool {
if !self.required {
return true;
}
tokens.iter().any(|(ty, value)| {
let h = sha256(value);
*ty == 0
&& self.grants.iter().any(|g| {
g.token_sha256 == h && prefix_matches(&g.prefix, ns) && match right {
Right::Publish => g.publish,
Right::Subscribe => g.subscribe,
}
})
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_cache_follows_section_8_9() {
let mut c = TokenCache::new(16 + 4 + 16 + 3);
let reg = |a, v: &[u8]| Token::Register { alias: a, token_type: 0, value: v.to_vec() };
assert_eq!(c.resolve(&[reg(1, b"abcd")]).unwrap(), vec![(0, b"abcd".to_vec())]);
assert_eq!(c.resolve(&[reg(1, b"x")]), Err(SessionError::DUPLICATE_AUTH_TOKEN_ALIAS));
assert_eq!(c.resolve(&[reg(2, b"wxyz")]), Err(SessionError::AUTH_TOKEN_CACHE_OVERFLOW));
assert_eq!(c.resolve(&[reg(2, b"xyz")]).unwrap().len(), 1);
assert_eq!(c.resolve(&[Token::UseAlias { alias: 1 }]).unwrap(), vec![(0, b"abcd".to_vec())]);
assert_eq!(c.resolve(&[Token::UseAlias { alias: 9 }]), Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS));
c.resolve(&[Token::Delete { alias: 1 }]).unwrap();
assert_eq!(c.used(), 16 + 3);
assert_eq!(c.resolve(&[Token::Delete { alias: 1 }]), Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS));
// In SETUP a registration past the limit reads as USE_VALUE.
let mut s = TokenCache::new(0);
assert_eq!(s.setup(&[reg(5, b"big")]).unwrap(), vec![(0, b"big".to_vec())]);
assert_eq!(s.resolve(&[Token::UseAlias { alias: 5 }]), Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS));
}
#[test]
fn grants_cover_a_prefix_and_a_right() {
let cfg = AuthConfig {
required: true,
grants: vec![crate::config::Grant { prefix: vec!["live".into()], token_sha256: hex::encode(sha256(b"t")), publish: false, subscribe: true }],
};
let a = Authorizer::from_config(&cfg).unwrap();
let ns = TrackNamespace::from_strs(["live", "a"]);
assert!(a.allows(&ns, Right::Subscribe, &[(0, b"t".to_vec())]));
assert!(!a.allows(&ns, Right::Publish, &[(0, b"t".to_vec())]));
assert!(!a.allows(&ns, Right::Subscribe, &[(0, b"u".to_vec())]));
assert!(!a.allows(&TrackNamespace::from_strs(["other"]), Right::Subscribe, &[(0, b"t".to_vec())]));
assert!(Authorizer::open().allows(&ns, Right::Publish, &[]));
}
}

421
joule-code/joule-code-rs/crates/joule-moq-relay/src/cache.rs Normal file
View file

@ -0,0 +1,421 @@
//! The relay's cache of objects (`draft-ietf-moq-transport-22` §7.1), from which it
//! serves FETCH and fills.
//!
//! An object is kept by its Full Track Name, Group ID and Object ID with every field §2.1.1
//! defines: its delivery mode (a Subgroup ID, or none for a datagram), its publisher
//! priority, its properties, its status and its payload. A duplicate is not stored again
//! (§7.1: "Relays MAY ignore subsequently received Objects"). A byte budget across all
//! tracks evicts whole groups, oldest first.
//!
//! **Coverage.** The cache serves a fetch alone only where it holds every object of the
//! range, so that a gap in its answer means an object that does not exist, as an unmarked
//! gap in a fetch does (§3.2.2). It keeps, for each group, what it knows of the group's
//! arrival ([`GroupState`]), and a group is **whole** when:
//!
//! - its start arrived: Object ID 0, or an object whose Prior Object ID Gap says every
//! object before it does not exist (§10.9);
//! - no subgroup stream of it is still open, and none was reset or cut short;
//! - and its end is known: each of its subgroup streams ended with a FIN, or an End of
//! Group or End of Track status, or a datagram with END_OF_GROUP, ended it.
//!
//! A fetch from `start` is covered when the group of the Largest Object has its start and
//! lost nothing, and every group from `start` up to it is whole or known not to exist (a
//! Prior Group ID Gap, §10.8), and nothing in that span was evicted. The cache computes this
//! from the groups it holds when asked, so a group that arrives after a later one, as
//! streams finish out of order, extends coverage back once it is whole; a group ID it has
//! never seen is a gap it does not cover across.
//!
//! The relay cannot tell a subgroup it has not yet seen from one that does not exist: a
//! group whose subgroups are all closed counts as whole, though another subgroup of it could
//! still open.
use bytes::Bytes;
use joule_moq::draft22::data::ObjectStatus;
use joule_moq::draft22::kvp::{prop, Properties};
use joule_moq::draft22::Location;
use std::collections::{BTreeMap, VecDeque};
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct CachedObject {
/// `None` for an object whose delivery mode was Datagram.
pub subgroup: Option<u64>,
pub publisher_priority: u8,
pub properties: Properties,
pub status: ObjectStatus,
pub payload: Bytes,
}
impl CachedObject {
fn bytes(&self) -> usize {
self.payload.len() + 64
}
}
/// What the cache knows of one group's arrival.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct GroupState {
/// Its first object arrived, or the objects before the first that did are known not
/// to exist.
pub start: bool,
/// Its subgroup streams still open upstream.
pub open: u32,
/// Its subgroup streams that ended with a FIN.
pub finished: u32,
/// A stream of it was reset or cut short: objects of it may be missing.
pub broken: bool,
/// An End of Group or End of Track status, or a datagram with END_OF_GROUP, ended it.
pub ended: bool,
}
impl GroupState {
/// Every object of the group is held (see the module documentation).
pub fn whole(&self) -> bool {
self.start && self.open == 0 && !self.broken && (self.finished > 0 || self.ended)
}
}
/// One track's objects.
#[derive(Debug, Default)]
pub struct TrackCache {
pub objects: BTreeMap<(u64, u64), CachedObject>,
/// What is known of each group held.
pub groups: BTreeMap<u64, GroupState>,
/// Groups known not to exist, as ranges `lo ..= hi`, keyed by `lo`.
pub absent: BTreeMap<u64, u64>,
/// Groups at or below this were evicted.
pub evicted_through: Option<u64>,
/// The track ended (an End of Track status), at this location.
pub end_of_track: Option<Location>,
}
/// How an object entering the cache relates to what it holds.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Insert {
New,
/// The same object again: ignored.
Duplicate,
/// An object at that location that differs in its delivery mode, Subgroup ID,
/// priority or payload: the track is malformed (§7.1).
Conflict,
/// In a group already evicted: not kept.
Evicted,
}
impl TrackCache {
pub fn insert(&mut self, group: u64, object: u64, o: CachedObject) -> Insert {
if self.evicted_through.is_some_and(|e| group <= e) {
return Insert::Evicted;
}
if let Some(prev) = self.objects.get(&(group, object)) {
let same = prev.subgroup == o.subgroup && prev.publisher_priority == o.publisher_priority && prev.payload == o.payload && prev.status == o.status;
return if same { Insert::Duplicate } else { Insert::Conflict };
}
let gap = |ty| o.properties.find(ty).ok().flatten().and_then(|k| k.as_varint());
let prior_objects = gap(prop::PRIOR_OBJECT_ID_GAP);
let prior_groups = gap(prop::PRIOR_GROUP_ID_GAP).filter(|n| *n > 0 && *n <= group);
let st = self.groups.entry(group).or_default();
if object == 0 || prior_objects == Some(object) {
st.start = true;
}
match o.status {
ObjectStatus::EndOfGroup => st.ended = true,
ObjectStatus::EndOfTrack => {
st.ended = true;
self.end_of_track = Some(Location::new(group, object));
}
ObjectStatus::Normal => {}
}
if let Some(n) = prior_groups {
self.absent.insert(group - n, group - 1);
}
self.objects.insert((group, object), o);
Insert::New
}
fn group(&mut self, group: u64) -> Option<&mut GroupState> {
if self.evicted_through.is_some_and(|e| group <= e) {
return None;
}
Some(self.groups.entry(group).or_default())
}
/// A subgroup stream of `group` began upstream; `starts_group` when it began with the
/// group's first object.
pub fn stream_opened(&mut self, group: u64, starts_group: bool) {
if let Some(st) = self.group(group) {
st.open += 1;
st.start |= starts_group;
}
}
/// A subgroup stream of `group` ended: with a FIN (`clean`), or reset or cut short.
pub fn stream_ended(&mut self, group: u64, clean: bool) {
if let Some(st) = self.group(group) {
st.open = st.open.saturating_sub(1);
if clean {
st.finished += 1;
} else {
st.broken = true;
}
}
}
/// Objects of `group` that arrived could not be kept (a malformed track).
pub fn group_broken(&mut self, group: u64) {
if let Some(st) = self.group(group) {
st.broken = true;
}
}
/// A datagram with END_OF_GROUP ended `group` (§11.2.1).
pub fn group_ended(&mut self, group: u64) {
if let Some(st) = self.group(group) {
st.ended = true;
}
}
/// Whether `group` is known not to exist.
fn absent_range(&self, group: u64) -> Option<u64> {
self.absent.range(..=group).next_back().filter(|(_, hi)| **hi >= group).map(|(lo, _)| *lo)
}
/// Whether a fetch starting at `start` can be served from the cache alone (see the
/// module documentation).
pub fn covers(&self, start: Location) -> bool {
if self.evicted_through.is_some_and(|e| start.group <= e) {
return false;
}
let Some((&last, st)) = self.groups.iter().next_back() else { return false };
if start.group > last || !st.start || st.broken {
return false;
}
let mut g = last;
while g > start.group {
g -= 1;
match self.groups.get(&g) {
Some(s) if s.whole() => {}
Some(_) => return false,
None => match self.absent_range(g) {
Some(lo) if lo <= start.group => return true,
Some(lo) => g = lo,
None => return false,
},
}
}
true
}
/// The objects from `start` to `end` inclusive, groups in `descending` order or not,
/// objects within a group ascending (§3.2.1).
pub fn range(&self, start: Location, end: Location, descending: bool) -> Vec<((u64, u64), CachedObject)> {
let mut groups: Vec<u64> = self.objects.range((start.group, 0)..=(end.group, u64::MAX)).map(|((g, _), _)| *g).collect();
groups.dedup();
if descending {
groups.reverse();
}
let mut out = Vec::new();
for g in groups {
for ((gg, oo), o) in self.objects.range((g, 0)..=(g, u64::MAX)) {
let loc = Location::new(*gg, *oo);
if loc >= start && loc <= end {
out.push(((*gg, *oo), o.clone()));
}
}
}
out
}
/// Drop group `g`; the bytes freed.
fn evict_group(&mut self, g: u64) -> usize {
let keys: Vec<(u64, u64)> = self.objects.range((g, 0)..=(g, u64::MAX)).map(|(k, _)| *k).collect();
let mut freed = 0;
for k in keys {
if let Some(o) = self.objects.remove(&k) {
freed += o.bytes();
}
}
let through = self.evicted_through.map_or(g, |e| e.max(g));
self.evicted_through = Some(through);
self.groups = self.groups.split_off(&through.saturating_add(1));
self.absent.retain(|_, hi| *hi > through);
freed
}
}
/// The byte budget across tracks: groups in the order they were first cached.
#[derive(Debug)]
pub struct Budget {
pub max: usize,
pub used: usize,
order: VecDeque<(u64, u64, usize)>,
}
impl Budget {
pub fn new(max: usize) -> Self {
Budget { max, used: 0, order: VecDeque::new() }
}
/// Account `n` bytes of track `track`'s group `group`. Returns the (track, group)
/// pairs to evict, oldest first, until the budget holds.
pub fn add(&mut self, track: u64, group: u64, n: usize) -> Vec<(u64, u64)> {
self.used += n;
match self.order.back_mut() {
Some((t, g, b)) if *t == track && *g == group => *b += n,
_ => match self.order.iter_mut().rev().find(|(t, g, _)| *t == track && *g == group) {
Some((_, _, b)) => *b += n,
None => self.order.push_back((track, group, n)),
},
}
let mut evict = Vec::new();
while self.used > self.max {
let Some((t, g, b)) = self.order.pop_front() else { break };
self.used = self.used.saturating_sub(b);
evict.push((t, g));
}
evict
}
/// Forget the accounting of a track that is gone.
pub fn forget(&mut self, track: u64) {
let mut freed = 0;
self.order.retain(|(t, _, b)| {
if *t == track {
freed += b;
false
} else {
true
}
});
self.used = self.used.saturating_sub(freed);
}
}
/// Evict `group` from `cache`, for the budget.
pub fn evict(cache: &mut TrackCache, group: u64) -> usize {
cache.evict_group(group)
}
/// The size an object counts for.
pub fn size_of(o: &CachedObject) -> usize {
o.bytes()
}
#[cfg(test)]
mod tests {
use super::*;
use joule_moq::draft22::kvp::KeyValuePair;
fn obj(p: &'static [u8]) -> CachedObject {
CachedObject { subgroup: Some(0), publisher_priority: 1, properties: Properties::new(), status: ObjectStatus::Normal, payload: Bytes::from_static(p) }
}
fn whole(c: &mut TrackCache, g: u64) {
c.stream_opened(g, true);
assert_eq!(c.insert(g, 0, obj(b"x")), Insert::New);
c.stream_ended(g, true);
}
#[test]
fn ranges_and_conflicts() {
let mut c = TrackCache::default();
for g in 4..=5 {
whole(&mut c, g);
}
assert_eq!(c.insert(4, 1, obj(b"b")), Insert::New);
assert_eq!(c.insert(4, 1, obj(b"b")), Insert::Duplicate);
assert_eq!(c.insert(4, 1, obj(b"y")), Insert::Conflict);
let asc: Vec<_> = c.range(Location::new(4, 0), Location::new(5, 0), false).into_iter().map(|(k, _)| k).collect();
assert_eq!(asc, vec![(4, 0), (4, 1), (5, 0)]);
let desc: Vec<_> = c.range(Location::new(4, 1), Location::new(5, 9), true).into_iter().map(|(k, _)| k).collect();
assert_eq!(desc, vec![(5, 0), (4, 1)]);
}
#[test]
fn coverage_is_the_run_of_whole_groups() {
let mut c = TrackCache::default();
assert!(!c.covers(Location::new(0, 0)), "nothing held");
// Group 1 whole before group 0 arrives.
whole(&mut c, 1);
assert!(c.covers(Location::new(1, 0)));
assert!(!c.covers(Location::new(0, 0)), "group 0 not seen: a gap");
// Group 0 arrives: covered once its stream ends.
c.stream_opened(0, true);
c.insert(0, 0, obj(b"a"));
assert!(!c.covers(Location::new(0, 0)), "its stream is still open");
c.stream_ended(0, true);
assert!(c.covers(Location::new(0, 0)), "a late group extends coverage back once whole");
// The newest group need only have started.
c.stream_opened(2, true);
c.insert(2, 0, obj(b"c"));
assert!(c.covers(Location::new(0, 0)) && c.covers(Location::new(2, 0)));
// A group never seen between: no coverage across it.
c.stream_ended(2, true);
whole(&mut c, 4);
assert!(c.covers(Location::new(4, 0)) && !c.covers(Location::new(2, 0)), "group 3 unseen");
}
#[test]
fn a_prior_group_id_gap_bridges_groups_that_do_not_exist() {
// Group 4's first object says the one group before it does not exist (§10.8).
let mut c = TrackCache::default();
whole(&mut c, 2);
let mut p = Properties::new();
p.push(KeyValuePair::varint(prop::PRIOR_GROUP_ID_GAP, 1));
c.stream_opened(4, true);
c.insert(4, 0, CachedObject { properties: p, ..obj(b"e") });
c.stream_ended(4, true);
assert!(c.covers(Location::new(2, 0)), "group 3 known absent");
assert!(c.covers(Location::new(3, 0)), "a fetch may start in a group that does not exist");
assert!(!c.covers(Location::new(1, 0)), "group 1 unseen");
}
#[test]
fn what_breaks_coverage() {
// A reset stream: the group may be missing objects.
let mut c = TrackCache::default();
c.stream_opened(0, true);
c.insert(0, 0, obj(b"a"));
c.stream_ended(0, false);
whole(&mut c, 1);
assert!(!c.covers(Location::new(0, 0)) && c.covers(Location::new(1, 0)));
// Joined mid-group: group 7 began at object 3, so its start is unknown.
let mut c = TrackCache::default();
c.stream_opened(7, false);
c.insert(7, 3, obj(b"m"));
c.stream_ended(7, true);
whole(&mut c, 8);
assert!(!c.covers(Location::new(7, 0)) && c.covers(Location::new(8, 0)));
// Unless a Prior Object ID Gap says objects 0 to 2 do not exist (§10.9).
let mut p = Properties::new();
p.push(KeyValuePair::varint(prop::PRIOR_OBJECT_ID_GAP, 3));
let mut c = TrackCache::default();
c.stream_opened(7, false);
c.insert(7, 3, CachedObject { properties: p, ..obj(b"m") });
c.stream_ended(7, true);
whole(&mut c, 8);
assert!(c.covers(Location::new(7, 0)));
// Datagrams: whole with the start and an END_OF_GROUP.
let mut c = TrackCache::default();
c.insert(0, 0, obj(b"d0"));
c.insert(0, 1, obj(b"d1"));
whole(&mut c, 1);
assert!(!c.covers(Location::new(0, 0)));
c.group_ended(0);
assert!(c.covers(Location::new(0, 0)));
// Eviction: nothing at or below an evicted group is covered.
evict(&mut c, 0);
assert!(!c.covers(Location::new(0, 0)) && c.covers(Location::new(1, 0)));
assert_eq!(c.insert(0, 2, obj(b"late")), Insert::Evicted);
}
#[test]
fn the_budget_evicts_oldest_groups_first() {
let mut b = Budget::new(100);
assert!(b.add(1, 0, 60).is_empty());
assert!(b.add(2, 0, 30).is_empty());
assert_eq!(b.add(1, 1, 20), vec![(1, 0)]);
assert_eq!(b.used, 50);
b.forget(2);
assert_eq!(b.used, 20);
}
}

135
joule-code/joule-code-rs/crates/joule-moq-relay/src/cert.rs Normal file
View file

@ -0,0 +1,135 @@
//! The relay's TLS identity, and the TLS configuration of its UDP endpoint.
//!
//! In production the certificate chain and key come from PEM files named in the
//! configuration. In development the relay makes its own: ECDSA P-256 and valid for
//! fourteen days from now, which are the constraints a browser puts on a certificate it
//! trusts by hash (W3C WebTransport, `serverCertificateHashes`: "the total length of the
//! validity period MUST NOT exceed two weeks", and ECDSA with secp256r1 an allowed key).
//! The relay prints the certificate's SHA-256 so a page can pass it to the browser.
use base64::Engine;
use rustls::pki_types::{CertificateDer, PrivateKeyDer, PrivatePkcs8KeyDer};
use std::sync::{Arc, Once};
static PROVIDER: Once = Once::new();
/// Install the ring crypto provider as rustls's process default (once).
pub fn install_provider() {
PROVIDER.call_once(|| {
let _ = rustls::crypto::ring::default_provider().install_default();
});
}
/// A certificate chain and its private key.
pub struct RelayIdentity {
pub chain: Vec<CertificateDer<'static>>,
pub key: PrivateKeyDer<'static>,
/// SHA-256 of the leaf certificate's DER.
pub leaf_sha256: [u8; 32],
}
impl Clone for RelayIdentity {
fn clone(&self) -> Self {
RelayIdentity { chain: self.chain.clone(), key: self.key.clone_key(), leaf_sha256: self.leaf_sha256 }
}
}
/// SHA-256 of a certificate's DER: the value `serverCertificateHashes` compares.
pub fn sha256(der: &[u8]) -> [u8; 32] {
let d = ring::digest::digest(&ring::digest::SHA256, der);
let mut out = [0u8; 32];
out.copy_from_slice(d.as_ref());
out
}
impl RelayIdentity {
/// A self-signed identity for development: ECDSA P-256, valid fourteen days from now,
/// for `names` (DNS names or IP addresses).
pub fn dev_self_signed(names: &[String]) -> anyhow_like::Result<Self> {
let id = wtransport::Identity::self_signed_builder()
.subject_alt_names(names.iter().map(String::as_str))
.from_now_utc()
.validity_days(14)
.build()
.map_err(|e| anyhow_like::Error(format!("self-signed certificate: {e}")))?;
Self::from_wtransport(&id)
}
/// Load a chain and key from PEM files.
pub async fn from_pem_files(cert: &std::path::Path, key: &std::path::Path) -> anyhow_like::Result<Self> {
let id = wtransport::Identity::load_pemfiles(cert, key)
.await
.map_err(|e| anyhow_like::Error(format!("loading {} and {}: {e}", cert.display(), key.display())))?;
Self::from_wtransport(&id)
}
fn from_wtransport(id: &wtransport::Identity) -> anyhow_like::Result<Self> {
let chain: Vec<CertificateDer<'static>> = id.certificate_chain().as_slice().iter().map(|c| CertificateDer::from(c.der().to_vec())).collect();
let leaf = chain.first().ok_or_else(|| anyhow_like::Error("an empty certificate chain".into()))?;
let leaf_sha256 = sha256(leaf.as_ref());
let key = PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(id.private_key().secret_der().to_vec()));
Ok(RelayIdentity { chain, key, leaf_sha256 })
}
/// The leaf hash in the forms a page needs: hex, base64, and the byte array a
/// `serverCertificateHashes` entry takes (`{algorithm: "sha-256", value}`).
pub fn hash_lines(&self) -> Vec<String> {
let b = &self.leaf_sha256;
vec![
format!("certificate sha-256 (hex): {}", hex::encode(b)),
format!("certificate sha-256 (base64): {}", base64::engine::general_purpose::STANDARD.encode(b)),
format!(
"serverCertificateHashes: [{{ algorithm: \"sha-256\", value: new Uint8Array([{}]) }}]",
b.iter().map(|x| x.to_string()).collect::<Vec<_>>().join(", ")
),
]
}
/// The TLS configuration of the relay's UDP endpoint: TLS 1.3, offering the native
/// MOQT ALPN first and `h3` (for WebTransport) second.
pub fn server_tls(&self) -> anyhow_like::Result<rustls::ServerConfig> {
install_provider();
let provider = Arc::new(rustls::crypto::ring::default_provider());
let mut tls = rustls::ServerConfig::builder_with_provider(provider)
.with_protocol_versions(&[&rustls::version::TLS13])
.map_err(|e| anyhow_like::Error(e.to_string()))?
.with_no_client_auth()
.with_single_cert(self.chain.clone(), self.key.clone_key())
.map_err(|e| anyhow_like::Error(format!("the certificate and key: {e}")))?;
tls.alpn_protocols = vec![joule_moq::draft22::ALPN.to_vec(), b"h3".to_vec()];
tls.max_early_data_size = 0;
Ok(tls)
}
}
/// A small error type for setup paths, so the crate needs no general error dependency.
pub mod anyhow_like {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Error(pub String);
impl std::fmt::Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.0)
}
}
impl std::error::Error for Error {}
pub type Result<T> = std::result::Result<T, Error>;
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_dev_certificate_is_one_a_browser_takes_by_hash() {
let id = RelayIdentity::dev_self_signed(&["localhost".into(), "127.0.0.1".into()]).unwrap();
assert_eq!(id.chain.len(), 1);
assert_ne!(id.leaf_sha256, [0; 32]);
// The hash is SHA-256 of the DER, the value a page passes; the WebTransport
// library's own digest of the same certificate agrees.
let theirs = wtransport::tls::Certificate::from_der(id.chain[0].as_ref().to_vec()).unwrap().hash();
assert_eq!(theirs.as_ref(), &id.leaf_sha256);
let tls = id.server_tls().unwrap();
assert_eq!(tls.alpn_protocols, vec![b"moqt-22".to_vec(), b"h3".to_vec()]);
assert!(id.hash_lines()[0].contains(&hex::encode(id.leaf_sha256)));
}
}

324
joule-code/joule-code-rs/crates/joule-moq-relay/src/client.rs Normal file
View file

@ -0,0 +1,324 @@
//! A small MOQT -22 client over any of the relay's transports: enough to publish and
//! subscribe through the relay, for its tests and for tools. Every wait in it is bounded.
use crate::transport::{BoxSend, SharedConnection};
use crate::wire::{open_request, responder, MessageReader, ReqRx, ReqTx, UniReader};
use joule_moq::carriage::SETUP_OPTION_JWP_RECEIPTS;
use joule_moq::draft22::data::{FetchEntry, SubgroupCodec, SubgroupHeader, SubgroupObject};
use joule_moq::draft22::kvp::{opt, KeyValuePair, Properties, SetupOptions};
use joule_moq::draft22::{
ControlMessage, FetchOk, Goaway, GroupOrder, NamespaceRequest, Parameters, Perspective, Publish, RequestIds, Setup, StreamHeader, SubscribeOk, Token,
TrackRequest, UniEvent,
};
use joule_moq::{TrackName, TrackNamespace, JWP_MOQ_EXTENSION_VERSION};
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::sync::{mpsc, oneshot, watch};
use tokio::task::JoinHandle;
use tokio::time::timeout;
/// How long any one step of the client waits.
pub const STEP: Duration = Duration::from_secs(10);
#[derive(Clone, Debug)]
pub struct ClientOptions {
pub jwp: bool,
pub tokens: Vec<Token>,
pub max_request_updates: u64,
pub max_object_bytes: u64,
/// Read nothing of a data stream until [`Client::next_data`] asks for it, so unread
/// streams hold the transport's flow control (a subscriber that reads slowly). Fetch
/// streams are then not routed to [`Client::fetch`].
pub lazy_data: bool,
}
impl Default for ClientOptions {
fn default() -> Self {
ClientOptions { jwp: true, tokens: Vec::new(), max_request_updates: 0, max_object_bytes: 16 << 20, lazy_data: false }
}
}
/// A request the relay opened to this client (a SUBSCRIBE or FETCH for a namespace it
/// published), its first message, and its stream.
pub type Incoming = (ReqTx, ReqRx, ControlMessage);
pub struct Client {
pub conn: SharedConnection,
ids: Mutex<RequestIds>,
pub peer_setup: Setup,
goaway: watch::Receiver<Option<Goaway>>,
_control: tokio::sync::Mutex<BoxSend>,
data: tokio::sync::Mutex<mpsc::UnboundedReceiver<UniReader>>,
requests: tokio::sync::Mutex<mpsc::UnboundedReceiver<Incoming>>,
pending_fetch: Arc<Mutex<HashMap<u64, oneshot::Sender<UniReader>>>>,
my_updates: u64,
tasks: Vec<JoinHandle<()>>,
/// A stream [`Client::next_data`] took and gave up waiting on before its header came:
/// offered first on the next call, so no stream is lost to a timeout.
held: tokio::sync::Mutex<Option<UniReader>>,
/// Each time `next_data` gave up waiting on a header: how many streams were queued
/// behind the one it waited on.
pub waited_out: Mutex<Vec<usize>>,
}
impl Drop for Client {
fn drop(&mut self) {
for t in &self.tasks {
t.abort();
}
}
}
fn s<E: std::fmt::Display>(e: E) -> String {
e.to_string()
}
impl Client {
/// Open this end's control stream with SETUP, and read the relay's (§6.3).
pub async fn connect(conn: SharedConnection, opts: ClientOptions) -> Result<Client, String> {
let mut o = SetupOptions::new();
o.push(KeyValuePair::bytes(opt::MOQT_IMPLEMENTATION, format!("joule-moq-relay-client/{}", env!("CARGO_PKG_VERSION")).into_bytes()));
if opts.max_request_updates != 0 {
o.push(KeyValuePair::varint(opt::MAX_REQUEST_UPDATES, opts.max_request_updates));
}
for t in &opts.tokens {
o.push(KeyValuePair::bytes(opt::AUTHORIZATION_TOKEN, t.to_bytes()));
}
if opts.jwp {
o.push(KeyValuePair::varint(SETUP_OPTION_JWP_RECEIPTS, JWP_MOQ_EXTENSION_VERSION as u64));
}
let mut control = timeout(STEP, conn.open_uni()).await.map_err(s)?.map_err(s)?;
control.write_all(&ControlMessage::Setup(Setup { options: o }).encode().map_err(s)?).await.map_err(s)?;
let (data_tx, data_rx) = mpsc::unbounded_channel();
let pending_fetch: Arc<Mutex<HashMap<u64, oneshot::Sender<UniReader>>>> = Arc::default();
// The relay's control stream; data streams that come first are kept.
let (peer_setup, mut ctl) = timeout(STEP, async {
loop {
let recv = conn.accept_uni().await.map_err(s)?;
let mut r = UniReader::new(recv, opts.max_object_bytes);
match r.start().await.map_err(s)? {
StreamHeader::Control => match r.next().await.map_err(s)? {
Some(UniEvent::Control(ControlMessage::Setup(st))) => return Ok::<_, String>((st, r)),
_ => return Err("the relay's control stream does not begin with SETUP".to_string()),
},
_ => {
let _ = data_tx.send(r);
}
}
}
})
.await
.map_err(|_| "no SETUP from the relay".to_string())??;
peer_setup.check_from(Perspective::Server, false).map_err(|c| format!("the relay's SETUP: {c}"))?;
let (gtx, grx) = watch::channel(None);
let mut tasks = Vec::new();
tasks.push(tokio::spawn(async move {
while let Ok(Some(ev)) = ctl.next().await {
if let UniEvent::Control(ControlMessage::Goaway(g)) = ev {
let _ = gtx.send(Some(g));
}
}
}));
let (c2, pf, max, lazy) = (conn.clone(), pending_fetch.clone(), opts.max_object_bytes, opts.lazy_data);
tasks.push(tokio::spawn(async move {
while let Ok(recv) = c2.accept_uni().await {
let mut r = UniReader::new(recv, max);
if lazy {
let _ = data_tx.send(r);
continue;
}
let (pf, data_tx) = (pf.clone(), data_tx.clone());
tokio::spawn(async move {
if let Ok(Ok(StreamHeader::Fetch(fh))) = timeout(STEP, r.start()).await {
let waiting = pf.lock().unwrap().remove(&fh.request_id);
if let Some(tx) = waiting {
let _ = tx.send(r);
return;
}
}
let _ = data_tx.send(r);
});
}
}));
let (req_tx, req_rx) = mpsc::unbounded_channel();
let (c3, my, peer) = (conn.clone(), opts.max_request_updates, peer_setup.options.max_request_updates());
tasks.push(tokio::spawn(async move {
while let Ok((send, recv)) = c3.accept_bi().await {
let mut reader = MessageReader::new(recv);
let req_tx = req_tx.clone();
tokio::spawn(async move {
if let Ok(Ok(Some(first))) = timeout(STEP, reader.next()).await
&& let Ok((tx, rx)) = responder(send, reader, &first, my, peer)
{
let _ = req_tx.send((tx, rx, first));
}
});
}
}));
Ok(Client {
conn,
ids: Mutex::new(RequestIds::new(Perspective::Client)),
peer_setup,
goaway: grx,
_control: tokio::sync::Mutex::new(control),
data: tokio::sync::Mutex::new(data_rx),
requests: tokio::sync::Mutex::new(req_rx),
pending_fetch,
my_updates: opts.max_request_updates,
tasks,
held: tokio::sync::Mutex::new(None),
waited_out: Mutex::new(Vec::new()),
})
}
pub fn next_request_id(&self) -> u64 {
self.ids.lock().unwrap().next().expect("Request IDs left")
}
/// Open a request stream with `m` as its first message.
pub async fn request(&self, m: &ControlMessage) -> Result<(ReqTx, ReqRx), String> {
timeout(STEP, open_request(&*self.conn, m, self.my_updates, self.peer_setup.options.max_request_updates())).await.map_err(s)?.map_err(s)
}
/// The first response on a request stream.
pub async fn response(rx: &mut ReqRx) -> Result<ControlMessage, String> {
match timeout(STEP, rx.next()).await {
Ok(Ok(Some(m))) => Ok(m),
Ok(Ok(None)) => Err("the stream ended before its response".into()),
Ok(Err(e)) => Err(e.to_string()),
Err(_) => Err("no response in time".into()),
}
}
fn refusal(m: ControlMessage) -> String {
match m {
ControlMessage::RequestError(e) => format!("REQUEST_ERROR {}: {}", e.code, String::from_utf8_lossy(&e.reason)),
other => format!("an unexpected {}", other.name()),
}
}
pub async fn publish_namespace(&self, prefix: TrackNamespace, params: Parameters) -> Result<(ReqTx, ReqRx), String> {
let m = ControlMessage::PublishNamespace(NamespaceRequest { request_id: self.next_request_id(), prefix, params });
let (tx, mut rx) = self.request(&m).await?;
match Self::response(&mut rx).await? {
ControlMessage::RequestOk(_) => Ok((tx, rx)),
other => Err(Self::refusal(other)),
}
}
pub async fn publish(&self, namespace: TrackNamespace, name: TrackName, track_alias: u64, params: Parameters, track_properties: Properties) -> Result<(ReqTx, ReqRx), String> {
let m = ControlMessage::Publish(Publish { request_id: self.next_request_id(), namespace, name, track_alias, params, track_properties });
let (tx, mut rx) = self.request(&m).await?;
match Self::response(&mut rx).await? {
ControlMessage::RequestOk(_) => Ok((tx, rx)),
other => Err(Self::refusal(other)),
}
}
pub async fn subscribe(&self, namespace: TrackNamespace, name: TrackName, params: Parameters) -> Result<(SubscribeOk, ReqTx, ReqRx), String> {
let m = ControlMessage::Subscribe(TrackRequest { request_id: self.next_request_id(), namespace, name, params });
let (tx, mut rx) = self.request(&m).await?;
match Self::response(&mut rx).await? {
ControlMessage::SubscribeOk(ok) => Ok((ok, tx, rx)),
other => Err(Self::refusal(other)),
}
}
pub async fn subscribe_namespace(&self, prefix: TrackNamespace) -> Result<(ReqTx, ReqRx), String> {
let m = ControlMessage::SubscribeNamespace(NamespaceRequest { request_id: self.next_request_id(), prefix, params: Parameters::new() });
let (tx, mut rx) = self.request(&m).await?;
match Self::response(&mut rx).await? {
ControlMessage::RequestOk(_) => Ok((tx, rx)),
other => Err(Self::refusal(other)),
}
}
/// FETCH, and read its whole fetch stream (each read bounded by [`STEP`]).
pub async fn fetch(&self, namespace: TrackNamespace, name: TrackName, params: Parameters) -> Result<(FetchOk, Vec<FetchEntry>), String> {
let order = params.group_order().unwrap_or(GroupOrder::Ascending);
let id = self.next_request_id();
let (otx, orx) = oneshot::channel();
self.pending_fetch.lock().unwrap().insert(id, otx);
let m = ControlMessage::Fetch(TrackRequest { request_id: id, namespace, name, params });
let (_tx, mut rx) = self.request(&m).await?;
let ok = match Self::response(&mut rx).await? {
ControlMessage::FetchOk(ok) => ok,
other => {
self.pending_fetch.lock().unwrap().remove(&id);
return Err(Self::refusal(other));
}
};
let mut r = timeout(STEP, orx).await.map_err(|_| "no fetch stream in time".to_string())?.map_err(s)?;
r.set_group_order(order);
let mut out = Vec::new();
loop {
match timeout(STEP, r.next()).await.map_err(|_| "a fetch stream stalled".to_string())?.map_err(s)? {
Some(UniEvent::FetchEntry(e)) => out.push(e),
Some(_) => return Err("a fetch stream carried something else".into()),
None => break,
}
}
Ok((ok, out))
}
/// The next data stream the relay opened, its header read. A stream reset before its
/// header is passed over.
pub async fn next_data(&self, wait: Duration) -> Option<UniReader> {
let deadline = tokio::time::Instant::now() + wait;
let mut held = self.held.lock().await;
let mut rx = self.data.lock().await;
loop {
let mut r = match held.take() {
Some(r) => r,
None => tokio::time::timeout_at(deadline, rx.recv()).await.ok().flatten()?,
};
match tokio::time::timeout_at(deadline, r.start()).await {
Ok(Ok(_)) => return Some(r),
// Reset before its header: passed over.
Ok(Err(_)) => continue,
Err(_) => {
self.waited_out.lock().unwrap().push(rx.len());
*held = Some(r);
return None;
}
}
}
}
/// The next data stream the relay opened, unread: its header not yet waited for.
pub async fn next_stream(&self, wait: Duration) -> Option<UniReader> {
if let Some(r) = self.held.lock().await.take() {
return Some(r);
}
timeout(wait, async { self.data.lock().await.recv().await }).await.ok().flatten()
}
/// The next request the relay opened to this client.
pub async fn next_request(&self, wait: Duration) -> Option<Incoming> {
timeout(wait, async { self.requests.lock().await.recv().await }).await.ok().flatten()
}
pub fn goaway(&self) -> watch::Receiver<Option<Goaway>> {
self.goaway.clone()
}
/// One subgroup stream: its header, its objects, a FIN.
pub async fn send_subgroup(&self, header: SubgroupHeader, objects: &[SubgroupObject]) -> Result<(), String> {
let mut b = Vec::new();
header.encode(&mut b);
let mut c = SubgroupCodec::new(header);
for o in objects {
c.encode_object(o, &mut b).map_err(s)?;
}
let mut st = timeout(STEP, self.conn.open_uni()).await.map_err(s)?.map_err(s)?;
timeout(STEP, st.write_all(&b)).await.map_err(s)?.map_err(s)?;
st.finish().map_err(s)
}
pub fn close(&self, code: u64, reason: &str) {
self.conn.close(code, reason.as_bytes());
}
}

325
joule-code/joule-code-rs/crates/joule-moq-relay/src/config.rs Normal file
View file

@ -0,0 +1,325 @@
//! The relay's configuration, read from TOML.
//!
//! Nothing secret lives in a configuration file: the TLS key and the summary-signing key
//! are paths to files the operator provides, and authorization grants name tokens by
//! their SHA-256, never by value. `relay.example.toml` beside this crate shows every
//! field.
use serde::Deserialize;
use std::net::SocketAddr;
use std::path::PathBuf;
#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RelayConfig {
/// UDP address for native QUIC (ALPN `moqt-22`) and WebTransport (ALPN `h3`), one
/// port for both. Absent: no UDP endpoint.
#[serde(default)]
pub quic_listen: Option<SocketAddr>,
/// TCP address of the WebSocket fallback. Absent: none.
#[serde(default)]
pub websocket_listen: Option<SocketAddr>,
/// The path a WebTransport CONNECT and a WebSocket upgrade use.
#[serde(default = "default_path")]
pub path: String,
#[serde(default)]
pub tls: TlsConfig,
#[serde(default)]
pub cache: CacheConfig,
#[serde(default)]
pub limits: LimitsConfig,
#[serde(default)]
pub receipts: ReceiptsConfig,
#[serde(default)]
pub summary: SummaryConfig,
#[serde(default)]
pub auth: AuthConfig,
/// The New Session URI a GOAWAY names (§9.2); empty: reuse the current one.
#[serde(default)]
pub goaway_uri: String,
}
fn default_path() -> String {
"/moq".into()
}
#[derive(Clone, Debug, Default, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct TlsConfig {
/// PEM certificate chain and PKCS#8 key. Both or neither.
pub cert_path: Option<PathBuf>,
pub key_path: Option<PathBuf>,
/// Without files: make a development certificate for these names (ECDSA P-256, valid
/// fourteen days) and print its SHA-256 for `serverCertificateHashes`.
#[serde(default = "default_dev_names")]
pub dev_names: Vec<String>,
}
fn default_dev_names() -> Vec<String> {
vec!["localhost".into(), "127.0.0.1".into(), "::1".into()]
}
#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct CacheConfig {
/// Bytes of object payload the relay keeps across all tracks; past it the oldest
/// groups go first.
#[serde(default = "d_cache_bytes")]
pub max_bytes: usize,
}
fn d_cache_bytes() -> usize {
256 << 20
}
impl Default for CacheConfig {
fn default() -> Self {
CacheConfig { max_bytes: d_cache_bytes() }
}
}
#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct LimitsConfig {
#[serde(default = "d_sessions")]
pub max_sessions: usize,
/// Requests a peer may have open at once: the bidirectional stream limit
/// (MAX_STREAMS on QUIC, WT_MAX_STREAMS on the WebSocket binding). -22 bounds
/// requests in flight by stream limits alone (§6.4.2).
#[serde(default = "d_requests")]
pub max_requests: u64,
/// Unidirectional streams a peer may have open at once (its data streams).
#[serde(default = "d_uni")]
pub max_uni_streams: u64,
/// MAX_REQUEST_UPDATES the relay declares (§9.1.7); 0 is no limit.
#[serde(default = "d_updates")]
pub max_request_updates: u64,
/// MAX_AUTH_TOKEN_CACHE_SIZE the relay declares (§9.1.3); 0 forbids aliases.
#[serde(default = "d_token_cache")]
pub max_auth_token_cache_size: u64,
/// The longest RENDEZVOUS_TIMEOUT the relay honours, in milliseconds (§9.20.6).
#[serde(default = "d_rendezvous")]
pub max_rendezvous_ms: u64,
/// How long an upstream SUBSCRIBE or FETCH may take to be answered.
#[serde(default = "d_upstream")]
pub upstream_timeout_ms: u64,
/// The relay's FILL_TIMEOUT budget when a request names none, and its ceiling when
/// one does (§9.20.5).
#[serde(default = "d_fill")]
pub fill_timeout_ms: u64,
/// Writes in flight per downstream session: the scheduler hands the transport the
/// highest-priority object each time one finishes.
#[serde(default = "d_writes")]
pub write_concurrency: usize,
#[serde(default = "d_object")]
pub max_object_bytes: u64,
/// How long a data stream whose Track Alias is not yet known is held (§3.1.3.1).
#[serde(default = "d_alias_wait")]
pub unknown_alias_wait_ms: u64,
/// The Timeout a shutdown GOAWAY gives (§9.2).
#[serde(default = "d_goaway")]
pub goaway_timeout_ms: u64,
/// Bytes received and unread before a WebSocket session closes.
#[serde(default = "d_ws_buffer")]
pub websocket_buffer_bytes: usize,
/// QUIC idle timeout.
#[serde(default = "d_idle")]
pub idle_timeout_ms: u64,
/// Bytes a QUIC or WebTransport session may have sent and unacknowledged, or
/// buffered, before a write waits. Kept near the path's bandwidth-delay product, so
/// what waits waits in the relay's scheduler, where priority and delivery timeouts
/// still apply (§5.2), rather than in the transport's buffers.
#[serde(default = "d_send_window")]
pub quic_send_window_bytes: u64,
}
fn d_sessions() -> usize {
4096
}
fn d_requests() -> u64 {
256
}
fn d_uni() -> u64 {
1024
}
fn d_updates() -> u64 {
16
}
fn d_token_cache() -> u64 {
16 * 1024
}
fn d_rendezvous() -> u64 {
30_000
}
fn d_upstream() -> u64 {
5_000
}
fn d_fill() -> u64 {
2_000
}
fn d_writes() -> usize {
8
}
fn d_object() -> u64 {
16 << 20
}
fn d_alias_wait() -> u64 {
1_000
}
fn d_goaway() -> u64 {
5_000
}
fn d_ws_buffer() -> usize {
64 << 20
}
fn d_idle() -> u64 {
30_000
}
fn d_send_window() -> u64 {
1 << 20
}
impl Default for LimitsConfig {
fn default() -> Self {
LimitsConfig {
max_sessions: d_sessions(),
max_requests: d_requests(),
max_uni_streams: d_uni(),
max_request_updates: d_updates(),
max_auth_token_cache_size: d_token_cache(),
max_rendezvous_ms: d_rendezvous(),
upstream_timeout_ms: d_upstream(),
fill_timeout_ms: d_fill(),
write_concurrency: d_writes(),
max_object_bytes: d_object(),
unknown_alias_wait_ms: d_alias_wait(),
goaway_timeout_ms: d_goaway(),
websocket_buffer_bytes: d_ws_buffer(),
idle_timeout_ms: d_idle(),
quic_send_window_bytes: d_send_window(),
}
}
}
#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ReceiptsConfig {
/// A track whose name ends with this is the receipts track of the track named
/// without it, in the same namespace (WAI `moq-streaming-format.md` §4.1 names a
/// media track's receipts track in its catalog; this is the relay's local rule for
/// recognising one without reading the catalog).
#[serde(default = "d_suffix")]
pub receipts_track_suffix: String,
/// Namespaces whose objects are checked against their receipts before they are
/// forwarded. Elsewhere the relay forwards receipts tracks as opaque objects.
#[serde(default)]
pub door: Vec<DoorRule>,
}
fn d_suffix() -> String {
"/receipts".into()
}
impl Default for ReceiptsConfig {
fn default() -> Self {
ReceiptsConfig { receipts_track_suffix: d_suffix(), door: Vec::new() }
}
}
#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DoorRule {
/// The Track Namespace prefix, one entry per field.
pub prefix: Vec<String>,
/// Publisher keys (Ed25519, hex) whose group receipts the door accepts. Empty: any key
/// whose signature verifies, the subscriber binding trust itself.
#[serde(default)]
pub trust: Vec<String>,
/// How long an object waits at the door for its group's receipts.
#[serde(default = "d_hold")]
pub hold_ms: u64,
}
fn d_hold() -> u64 {
2_000
}
#[derive(Clone, Debug, Default, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SummaryConfig {
/// A file holding the relay's Ed25519 signing seed (32 bytes, raw or hex). Absent: a
/// key made at start, its public half logged.
pub key_path: Option<PathBuf>,
/// A file the signed session summaries are appended to, one JSON object a line.
pub out_path: Option<PathBuf>,
}
#[derive(Clone, Debug, Default, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct AuthConfig {
/// When true, every PUBLISH_NAMESPACE, PUBLISH, SUBSCRIBE_NAMESPACE, SUBSCRIBE, FETCH
/// and TRACK_STATUS must carry an AUTHORIZATION TOKEN (or rely on one from SETUP) that
/// a grant covers.
#[serde(default)]
pub required: bool,
#[serde(default)]
pub grants: Vec<Grant>,
}
#[derive(Clone, Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Grant {
/// The namespace prefix the grant covers, one entry per field.
pub prefix: Vec<String>,
/// SHA-256 of the token value, hex. The token itself is never in the configuration.
pub token_sha256: String,
#[serde(default)]
pub publish: bool,
#[serde(default)]
pub subscribe: bool,
}
impl RelayConfig {
pub fn from_toml(s: &str) -> Result<Self, String> {
toml::from_str(s).map_err(|e| e.to_string())
}
/// Development defaults: both endpoints on loopback, a self-signed certificate.
pub fn dev() -> Self {
RelayConfig {
quic_listen: Some("127.0.0.1:4443".parse().expect("an address")),
websocket_listen: Some("127.0.0.1:4480".parse().expect("an address")),
path: default_path(),
tls: TlsConfig { cert_path: None, key_path: None, dev_names: default_dev_names() },
cache: CacheConfig::default(),
limits: LimitsConfig::default(),
receipts: ReceiptsConfig::default(),
summary: SummaryConfig::default(),
auth: AuthConfig::default(),
goaway_uri: String::new(),
}
}
/// Ephemeral loopback ports, for tests.
pub fn loopback() -> Self {
let mut c = Self::dev();
c.quic_listen = Some("127.0.0.1:0".parse().expect("an address"));
c.websocket_listen = Some("127.0.0.1:0".parse().expect("an address"));
c
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_example_configuration_reads() {
let s = include_str!("../relay.example.toml");
let c = RelayConfig::from_toml(s).expect("the example reads");
assert!(c.quic_listen.is_some() && c.websocket_listen.is_some());
assert_eq!(c.receipts.door.len(), 1);
assert_eq!(c.auth.grants.len(), 1);
assert!(RelayConfig::from_toml("unknown_field = 1").is_err());
}
}

268
joule-code/joule-code-rs/crates/joule-moq-relay/src/door.rs Normal file
View file

@ -0,0 +1,268 @@
//! Checking objects against their receipts before they are forwarded, in the namespaces
//! the configuration names.
//!
//! Elsewhere the relay forwards a receipts track as it forwards any track, its objects
//! opaque (WAI `moq-streaming-format.md` §4.1; MOQT §7.8), and every subscriber verifies
//! for itself. In a namespace with a door rule the relay is a relay that reads the receipts
//! it forwards, and takes on the duties of WAI `jwp-receipts.md` §4.1:
//!
//! - each group receipt, from the bundle that group *g* of a receipts track carries, has
//! its Ed25519 signature verified (under the rule's trusted keys, or any key) before the
//! first object of media group *g* is fanned out. A group that does not verify is
//! dropped, its bundle and its objects, and a structured error is logged for the
//! upstream operator;
//! - each media object is checked against the receipt of its own position in the bundle:
//! the receipt names this group and Object ID (`moqt_mismatch` otherwise), its content
//! hash is the payload's, and its Merkle proof reaches the signed root. An object that
//! carries its receipt as a property must carry that same receipt;
//! - an object whose bundle has not arrived waits for it, up to the rule's `hold_ms`
//! (the publisher publishes the bundle no later than the group's first object), and is
//! refused after that;
//! - receipt bytes are forwarded unaltered; the relay re-signs nothing.
//!
//! The door reuses `joule-moq`'s verifier ([`ReceiptVerifier`]) and bundle reader.
use crate::config::DoorRule;
use joule_moq::carriage::ReceiptBundle;
use joule_moq::draft22::names::prefix_matches;
use joule_moq::receipt::{ObjectReceipt, ReceiptVerifier, VerifiedGroup};
use joule_moq::{GroupId, ObjectId, TrackName, TrackNamespace};
use std::collections::{HashMap, VecDeque};
use std::sync::Mutex;
use std::time::Duration;
use tokio::sync::Notify;
/// How many groups of each track the door remembers.
const GROUPS_HELD: usize = 4096;
struct Rule {
prefix: TrackNamespace,
verifier: ReceiptVerifier,
hold: Duration,
}
enum Entry {
Verified { group: VerifiedGroup, objects: HashMap<u64, ObjectReceipt> },
Refused(String),
}
type MediaKey = (TrackNamespace, TrackName);
#[derive(Default)]
struct Groups {
entries: HashMap<(MediaKey, u64), Entry>,
order: VecDeque<(MediaKey, u64)>,
}
pub struct Door {
rules: Vec<Rule>,
suffix: Vec<u8>,
groups: Mutex<Groups>,
arrived: Notify,
}
/// What the door decided about one object.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Verdict {
/// No rule covers the namespace.
Unchecked,
/// The object's receipt verified: its signed figure and leaf class.
Pass { joules_micro: u64, class: Option<Vec<u8>> },
Refuse(String),
}
impl Door {
pub fn new(rules: &[DoorRule], suffix: &str) -> Result<Self, String> {
let mut out = Vec::new();
for r in rules {
let mut anchors = Vec::new();
for k in &r.trust {
let b: [u8; 32] = hex::decode(k).map_err(|e| format!("a door key: {e}"))?.try_into().map_err(|_| "a door key is not 32 bytes".to_string())?;
anchors.push(ed25519_dalek::VerifyingKey::from_bytes(&b).map_err(|e| format!("a door key: {e}"))?);
}
let verifier = if anchors.is_empty() { ReceiptVerifier::new() } else { ReceiptVerifier::with_anchors(anchors) };
out.push(Rule { prefix: TrackNamespace::from_strs(r.prefix.iter().map(String::as_str)), verifier, hold: Duration::from_millis(r.hold_ms) });
}
Ok(Door { rules: out, suffix: suffix.as_bytes().to_vec(), groups: Mutex::new(Groups::default()), arrived: Notify::new() })
}
fn rule(&self, ns: &TrackNamespace) -> Option<&Rule> {
self.rules.iter().filter(|r| prefix_matches(&r.prefix, ns)).max_by_key(|r| r.prefix.0.len())
}
/// Whether `ns` is checked at the door.
pub fn checks(&self, ns: &TrackNamespace) -> bool {
self.rule(ns).is_some()
}
/// The media track a receipts track carries the receipts of, by the configured suffix.
pub fn media_of(&self, name: &TrackName) -> Option<TrackName> {
if self.suffix.is_empty() || name.0.len() <= self.suffix.len() || !name.0.ends_with(&self.suffix) {
return None;
}
Some(TrackName(name.0[..name.0.len() - self.suffix.len()].to_vec()))
}
/// The receipts track of a media track.
pub fn receipts_of(&self, name: &TrackName) -> TrackName {
let mut v = name.0.clone();
v.extend_from_slice(&self.suffix);
TrackName(v)
}
fn remember(&self, key: (MediaKey, u64), e: Entry) {
let mut g = self.groups.lock().unwrap();
if g.entries.insert(key.clone(), e).is_none() {
g.order.push_back(key);
while g.order.len() > GROUPS_HELD * self.rules.len().max(1) {
if let Some(old) = g.order.pop_front() {
g.entries.remove(&old);
}
}
}
drop(g);
self.arrived.notify_waiters();
}
/// Group `group` of the receipts track of `media` arrived, with this payload. `Ok` to
/// forward it, `Err` with why the group is dropped.
pub fn bundle(&self, ns: &TrackNamespace, media: &TrackName, group: u64, payload: &[u8]) -> Result<(), String> {
let Some(rule) = self.rule(ns) else { return Ok(()) };
let key = ((ns.clone(), media.clone()), group);
let checked = (|| {
let b = ReceiptBundle::from_payload(payload).map_err(|e| e.to_string())?;
if b.group.group_id != GroupId(group) {
return Err("moqt_mismatch: the group receipt names another group".to_string());
}
let vg = rule.verifier.verify_group(&b.group).map_err(|e| e.to_string())?;
let mut objects = HashMap::new();
for o in b.objects {
o.check_moqt(b.group.track_alias, GroupId(group), o.object_id).map_err(|_| "moqt_mismatch: an object receipt of another track or group".to_string())?;
rule.verifier.verify_object(&o, &vg).map_err(|e| format!("an object receipt: {e}"))?;
objects.insert(o.object_id.0, o);
}
Ok((vg, objects))
})();
match checked {
Ok((group, objects)) => {
self.remember(key, Entry::Verified { group, objects });
Ok(())
}
Err(why) => {
self.remember(key, Entry::Refused(why.clone()));
Err(why)
}
}
}
/// Check one media object. `property` is the receipt the object carries as its
/// JWP_OBJECT_RECEIPT property, if any.
pub async fn object(&self, ns: &TrackNamespace, media: &TrackName, group: u64, object: u64, payload: &[u8], property: Option<&ObjectReceipt>) -> Verdict {
let Some(rule) = self.rule(ns) else { return Verdict::Unchecked };
let key = ((ns.clone(), media.clone()), group);
let deadline = tokio::time::Instant::now() + rule.hold;
loop {
let notified = self.arrived.notified();
tokio::pin!(notified);
notified.as_mut().enable();
{
let g = self.groups.lock().unwrap();
match g.entries.get(&key) {
Some(Entry::Refused(why)) => return Verdict::Refuse(format!("its group was refused: {why}")),
Some(Entry::Verified { group: vg, objects }) => {
let Some(r) = objects.get(&object) else {
return Verdict::Refuse("no receipt for its position in the group".into());
};
if r.object_id != ObjectId(object) {
return Verdict::Refuse("moqt_mismatch".into());
}
if let Some(p) = property
&& p != r
{
return Verdict::Refuse("its receipt property is not the receipt its group signed".into());
}
return match rule.verifier.verify_payload(payload, r, vg) {
Ok(()) => Verdict::Pass { joules_micro: r.joules_micro, class: r.energy_class.clone() },
Err(e) => Verdict::Refuse(format!("its payload does not verify: {e}")),
};
}
None => {}
}
}
if tokio::time::timeout_at(deadline, notified).await.is_err() {
return Verdict::Refuse("its group's receipts did not arrive in time".into());
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use joule_moq::receipt::ReceiptBuilder;
use joule_moq::TrackAlias;
fn bundle(key: &ed25519_dalek::SigningKey, group: u64, payloads: &[&[u8]]) -> ReceiptBundle {
let mut b = ReceiptBuilder::new(TrackAlias(8), GroupId(group));
for p in payloads {
b.add_labelled_object(ObjectReceipt::hash_payload(p), 0, "o", "t", vec![0]).unwrap();
}
let objects = (0..payloads.len())
.map(|i| ObjectReceipt {
track_alias: TrackAlias(8),
group_id: GroupId(group),
object_id: ObjectId(i as u64),
content_hash: ObjectReceipt::hash_payload(payloads[i]),
joules_micro: 0,
origin: "o".into(),
merkle_proof: b.proof_for(i).unwrap(),
energy_class: Some(vec![0]),
})
.collect();
let group = b.finalize_labelled(key, "p", "t", None).unwrap();
ReceiptBundle { group, objects }
}
#[tokio::test]
async fn the_door_passes_what_its_group_signed() {
let key = ed25519_dalek::SigningKey::from_bytes(&[5; 32]);
let rule = DoorRule { prefix: vec!["live".into()], trust: vec![hex::encode(key.verifying_key().to_bytes())], hold_ms: 100 };
let door = Door::new(&[rule], "/receipts").unwrap();
let ns = TrackNamespace::from_strs(["live", "a"]);
let media = TrackName::from_str("cam");
assert_eq!(door.media_of(&TrackName::from_str("cam/receipts")), Some(media.clone()));
assert_eq!(door.media_of(&media), None);
assert!(!door.checks(&TrackNamespace::from_strs(["other"])));
let b = bundle(&key, 3, &[b"x", b"y"]);
door.bundle(&ns, &media, 3, &b.to_payload().unwrap()).unwrap();
assert!(matches!(door.object(&ns, &media, 3, 1, b"y", None).await, Verdict::Pass { .. }));
assert!(matches!(door.object(&ns, &media, 3, 1, b"tampered", None).await, Verdict::Refuse(_)));
assert!(matches!(door.object(&ns, &media, 3, 0, b"x", Some(&b.objects[1])).await, Verdict::Refuse(_)), "a property that is not its receipt");
assert!(matches!(door.object(&ns, &media, 4, 0, b"x", None).await, Verdict::Refuse(_)), "no bundle within the hold");
// A bundle signed by a key the rule does not trust drops its group.
let stranger = ed25519_dalek::SigningKey::from_bytes(&[6; 32]);
let bad = bundle(&stranger, 5, &[b"z"]);
assert!(door.bundle(&ns, &media, 5, &bad.to_payload().unwrap()).is_err());
assert!(matches!(door.object(&ns, &media, 5, 0, b"z", None).await, Verdict::Refuse(_)));
// A bundle under the wrong MOQT group is refused before its signature is read.
assert!(door.bundle(&ns, &media, 9, &bundle(&key, 8, &[b"q"]).to_payload().unwrap()).is_err());
}
#[tokio::test]
async fn an_object_waits_for_its_bundle() {
let key = ed25519_dalek::SigningKey::from_bytes(&[5; 32]);
let rule = DoorRule { prefix: vec![], trust: vec![], hold_ms: 2_000 };
let door = std::sync::Arc::new(Door::new(&[rule], "/receipts").unwrap());
let ns = TrackNamespace::from_strs(["any"]);
let media = TrackName::from_str("m");
let d2 = door.clone();
let (ns2, m2) = (ns.clone(), media.clone());
let waiter = tokio::spawn(async move { d2.object(&ns2, &m2, 0, 0, b"late", None).await });
tokio::time::sleep(Duration::from_millis(50)).await;
door.bundle(&ns, &media, 0, &bundle(&key, 0, &[b"late"]).to_payload().unwrap()).unwrap();
let v = tokio::time::timeout(Duration::from_secs(5), waiter).await.unwrap().unwrap();
assert!(matches!(v, Verdict::Pass { .. }), "{v:?}");
}
}

44
joule-code/joule-code-rs/crates/joule-moq-relay/src/lib.rs Normal file
View file

@ -0,0 +1,44 @@
//! **joule-moq-relay**: a MOQT relay for `draft-ietf-moq-transport-22`.
//!
//! - **Transports** ([`transport`]): native QUIC with ALPN `moqt-22` and WebTransport over
//! HTTP/3 (ALPN `h3`, the draft chosen by `WT-Available-Protocols`) on one UDP port;
//! and a WebSocket fallback that carries the same MOQT streams as
//! `draft-ietf-webtrans-http2-15` capsules, one to each message, for paths with no UDP.
//! - **Relay behaviour** ([`relay`], [`session`]): SETUP, namespace publication and
//! discovery (PUBLISH_NAMESPACE, SUBSCRIBE_NAMESPACE with NAMESPACE and
//! NAMESPACE_DONE), PUBLISH, SUBSCRIBE served from an upstream subscription the relay
//! holds or makes, RENDEZVOUS_TIMEOUT, FETCH and fills served from the cache or
//! upstream, TRACK_STATUS, REQUEST_UPDATE, GOAWAY on shutdown, Request ID rules and
//! MAX_REQUEST_UPDATES, request concurrency by stream limits, the authorization-token
//! alias cache ([`auth`]).
//! - **Fan-out** ([`sched`]): each downstream session's objects go out in §5.1.2's order
//! (subscriber priority, publisher priority, group order), with each subscriber's
//! delivery timeouts applied at this hop. Receipts tracks carry no timeout, so the
//! receipts of every dropped object still reach a subscriber of its receipts track.
//! - **Receipts at the door** ([`door`]): optional per namespace, with `joule-moq`'s
//! verifier.
//! - **Session summaries** ([`summary`]): bytes and objects per subscriber, signed by the
//! relay at session end; its own energy is reported as unmetered, since no meter backs
//! it.
//!
//! [`server::RelayServer`] binds the listeners; [`client::Client`] is a small client for
//! the tests and for tools.
pub mod auth;
pub mod cache;
pub mod cert;
pub mod client;
pub mod config;
pub mod door;
pub mod relay;
pub mod sched;
pub mod server;
pub mod session;
pub mod summary;
pub mod transport;
pub mod wire;
pub use config::RelayConfig;
pub use relay::{Relay, RelayStats};
pub use server::RelayServer;
pub use summary::{SessionSummary, SignedSessionSummary};

58
joule-code/joule-code-rs/crates/joule-moq-relay/src/main.rs Normal file
View file

@ -0,0 +1,58 @@
//! `joule-moq-relay`: run the relay.
//!
//! ```text
//! joule-moq-relay --config relay.toml
//! joule-moq-relay --dev # loopback, a development certificate
//! ```
//!
//! It logs where it listens, the certificate's SHA-256 (for a page's
//! `serverCertificateHashes`), and the key it signs session summaries with. On Ctrl-C it
//! sends GOAWAY to every session and waits for them to leave.
use joule_moq_relay::{RelayConfig, RelayServer};
use std::process::ExitCode;
fn usage() -> ExitCode {
eprintln!("usage: joule-moq-relay (--config <file.toml> | --dev)");
ExitCode::from(2)
}
#[tokio::main]
async fn main() -> ExitCode {
let level = std::env::var("JOULE_MOQ_RELAY_LOG").ok().and_then(|l| l.parse().ok()).unwrap_or(tracing::Level::INFO);
tracing_subscriber::fmt().with_max_level(level).with_target(false).init();
let args: Vec<String> = std::env::args().skip(1).collect();
let cfg = match args.iter().map(String::as_str).collect::<Vec<_>>().as_slice() {
["--dev"] => RelayConfig::dev(),
["--config", path] => match std::fs::read_to_string(path).map_err(|e| e.to_string()).and_then(|s| RelayConfig::from_toml(&s)) {
Ok(c) => c,
Err(e) => {
eprintln!("joule-moq-relay: {path}: {e}");
return ExitCode::FAILURE;
}
},
_ => return usage(),
};
let server = match RelayServer::start(cfg).await {
Ok(s) => s,
Err(e) => {
eprintln!("joule-moq-relay: {e}");
return ExitCode::FAILURE;
}
};
if let Some(a) = server.quic_addr {
tracing::info!("native QUIC (moqt-22) and WebTransport (h3) on udp {a}, path {}", server.relay().config().path);
}
if let Some(a) = server.websocket_addr {
tracing::info!("WebSocket fallback (subprotocol moqt-22) on tcp {a}, path {}", server.relay().config().path);
}
for line in server.identity.hash_lines() {
tracing::info!("{line}");
}
tracing::info!("session summaries signed by {}", hex::encode(server.relay().summary_key().to_bytes()));
let _ = tokio::signal::ctrl_c().await;
tracing::info!("shutting down: GOAWAY to every session");
let closed = server.shutdown().await;
tracing::info!("stopped; {closed} session(s) closed at the GOAWAY timeout");
ExitCode::SUCCESS
}

730
joule-code/joule-code-rs/crates/joule-moq-relay/src/relay.rs Normal file
View file

@ -0,0 +1,730 @@
//! The relay's shared state: sessions, published namespaces, tracks with their upstream
//! and downstream subscriptions and their caches; and the fan-out of each object
//! (`draft-ietf-moq-transport-22` §7).
//!
//! Requests are handled in [`crate::session`]; this module holds what they share.
use crate::auth::{Authorizer, Resolved, TokenCache};
use crate::cache::{self, Budget, CachedObject, Insert, TrackCache};
use crate::config::RelayConfig;
use crate::door::Door;
use crate::sched::{Item, LaneKey, LaneKind, ObjMeta, Op, Prio, SchedHandle};
use crate::summary::{class_label, SignedSessionSummary, Tally};
use crate::transport::{BoxSend, SharedConnection, TransportKind};
use crate::wire::{ReqTx, UniReader};
use bytes::Bytes;
use ed25519_dalek::SigningKey;
use joule_moq::draft22::data::{ObjectDatagram, SubgroupHeader, SubgroupId, SubgroupObject};
use joule_moq::draft22::kvp::{prop, Properties};
use joule_moq::draft22::names::{prefix_matches, render};
use joule_moq::draft22::{GroupOrder, Location, PeerRequestIds, PublishDoneCode, RequestIds};
use joule_moq::{TrackName, TrackNamespace};
use std::collections::{HashMap, HashSet};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::sync::{mpsc, oneshot, watch, Notify};
use tokio::time::Instant;
pub type TrackKey = (TrackNamespace, TrackName);
/// The default subscriber and publisher priority (§5.1.3, §9.20.7: 128).
pub const DEFAULT_PRIORITY: u8 = 128;
/// One transport session at the relay, whichever role its peer plays.
pub struct SessionCtx {
pub id: u64,
pub conn: SharedConnection,
pub kind: TransportKind,
pub remote: String,
pub sched: SchedHandle,
pub tally: Arc<Mutex<Tally>>,
pub peer_ids: Mutex<PeerRequestIds>,
pub my_ids: Mutex<RequestIds>,
pub tokens: Mutex<TokenCache>,
pub setup_tokens: Vec<Resolved>,
/// MAX_REQUEST_UPDATES the peer declared, and the relay's.
pub peer_max_updates: u64,
pub my_max_updates: u64,
/// The relay's control stream to the peer, after its SETUP.
pub control: tokio::sync::Mutex<Option<BoxSend>>,
pub goaway_sent: AtomicBool,
/// Fetch streams awaited from this peer, by the relay's Request ID.
pub pending_fetch: Mutex<HashMap<u64, oneshot::Sender<UniReader>>>,
pub opened_unix_ms: u64,
pub close_code: Mutex<Option<u64>>,
pub jwp: bool,
}
impl SessionCtx {
pub fn next_request_id(&self) -> Option<u64> {
self.my_ids.lock().unwrap().next()
}
/// Close the session with a termination code (§12.2), recorded for the summary.
pub fn close(&self, code: u64, reason: &str) {
self.close_code.lock().unwrap().get_or_insert(code);
tracing::debug!(session = self.id, code, reason, "closing a session");
self.conn.close(code, reason.as_bytes());
}
}
/// What a downstream subscription asked for, as REQUEST_UPDATE may change it.
#[derive(Clone, Debug)]
pub struct SubParams {
pub subscriber_priority: u8,
pub order: GroupOrder,
pub forward: bool,
pub start: Location,
pub end: Option<Location>,
/// OBJECT_DELIVERY_TIMEOUT and SUBGROUP_DELIVERY_TIMEOUT the subscriber set; 0 none.
pub object_timeout: u64,
pub subgroup_timeout: u64,
}
impl SubParams {
pub fn passes(&self, loc: Location) -> bool {
self.forward && loc >= self.start && self.end.is_none_or(|e| loc <= e)
}
}
/// A downstream subscription.
pub struct DownSub {
pub id: u64,
pub session: u64,
pub request_id: u64,
pub alias: u64,
pub track: TrackKey,
pub render: Arc<str>,
pub sched: SchedHandle,
pub params: Mutex<SubParams>,
/// A subscription to a receipts track: no delivery timeout applies to it.
pub receipts_track: bool,
pub tx: ReqTx,
pub ended: watch::Sender<bool>,
}
#[derive(Clone)]
pub enum UpstreamKind {
/// The publisher sent PUBLISH.
Publish,
/// The relay sent SUBSCRIBE, on this stream.
Subscribe(ReqTx),
}
#[derive(Clone)]
pub struct Upstream {
pub session: u64,
pub alias: u64,
pub kind: UpstreamKind,
}
pub struct Track {
pub id: u64,
pub key: TrackKey,
pub render: Arc<str>,
pub props: Properties,
pub largest: Option<Location>,
pub upstreams: Vec<Upstream>,
pub downs: Vec<Arc<DownSub>>,
pub cache: TrackCache,
pub pub_object_timeout: u64,
pub pub_subgroup_timeout: u64,
pub default_priority: u8,
pub default_order: GroupOrder,
/// Which upstream session a (group, subgroup) is taken from (§7.3: deduplication).
pub owners: HashMap<(u64, u64), u64>,
pub ended: Option<PublishDoneCode>,
}
impl Track {
fn new(id: u64, key: TrackKey) -> Self {
let render: Arc<str> = render(&key.0, Some(&key.1)).into();
Track {
id,
key,
render,
props: Properties::new(),
largest: None,
upstreams: Vec::new(),
downs: Vec::new(),
cache: TrackCache::default(),
pub_object_timeout: 0,
pub_subgroup_timeout: 0,
default_priority: DEFAULT_PRIORITY,
default_order: GroupOrder::Ascending,
owners: HashMap::new(),
ended: None,
}
}
/// Take the properties a PUBLISH or SUBSCRIBE_OK carried (§10).
pub fn set_props(&mut self, props: Properties) {
let v = |ty| props.find(ty).ok().flatten().and_then(|k| k.as_varint());
self.pub_object_timeout = v(prop::OBJECT_DELIVERY_TIMEOUT).unwrap_or(0);
self.pub_subgroup_timeout = v(prop::SUBGROUP_DELIVERY_TIMEOUT).unwrap_or(0);
self.default_priority = v(prop::DEFAULT_PUBLISHER_PRIORITY).and_then(|p| u8::try_from(p).ok()).unwrap_or(DEFAULT_PRIORITY);
self.default_order = match v(prop::DEFAULT_PUBLISHER_GROUP_ORDER) {
Some(2) => GroupOrder::Descending,
_ => GroupOrder::Ascending,
};
self.props = props;
}
pub fn note_largest(&mut self, l: Location) {
if self.largest.is_none_or(|x| l > x) {
self.largest = Some(l);
}
}
}
pub struct PublishedNs {
pub namespace: TrackNamespace,
pub session: u64,
pub id: u64,
}
pub enum NsEvent {
Add(TrackNamespace),
Remove(TrackNamespace),
}
pub struct NsSub {
pub prefix: TrackNamespace,
pub session: u64,
pub id: u64,
pub tx: mpsc::UnboundedSender<NsEvent>,
}
pub type UpstreamWait = watch::Receiver<Option<Result<(), (u64, String)>>>;
#[derive(Default)]
pub struct State {
pub sessions: HashMap<u64, Arc<SessionCtx>>,
pub namespaces: Vec<PublishedNs>,
pub ns_subs: Vec<NsSub>,
pub tracks: HashMap<TrackKey, Track>,
pub track_ids: HashMap<u64, TrackKey>,
pub aliases: HashMap<(u64, u64), TrackKey>,
pub pending_upstream: HashMap<TrackKey, UpstreamWait>,
}
/// Counters across the relay.
#[derive(Default)]
pub struct Stats {
pub sessions_opened: AtomicU64,
/// Objects received from upstream, counted on arrival: before the door, the
/// deduplication and the cache. Whether the cache holds a track is
/// [`Relay::cache_covers`]'s to say.
pub objects_in: AtomicU64,
pub fetch_served_from_cache: AtomicU64,
pub fetch_served_upstream: AtomicU64,
pub refused_at_door: AtomicU64,
pub groups_dropped_at_door: AtomicU64,
}
/// A snapshot of [`Stats`].
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct RelayStats {
pub sessions_opened: u64,
pub sessions_open: u64,
pub objects_in: u64,
pub fetch_served_from_cache: u64,
pub fetch_served_upstream: u64,
pub refused_at_door: u64,
pub groups_dropped_at_door: u64,
/// Items pushed to sessions' schedulers and not yet dispatched.
pub queued: u64,
}
pub struct Inner {
pub cfg: RelayConfig,
pub auth: Authorizer,
pub door: Door,
pub key: SigningKey,
pub state: Mutex<State>,
pub budget: Mutex<Budget>,
pub ids: AtomicU64,
pub summaries: Mutex<Vec<SignedSessionSummary>>,
pub going_away: AtomicBool,
pub stats: Stats,
/// Woken when a publisher or a session appears or leaves.
pub change: Notify,
pub sessions_gone: Notify,
}
/// A MOQT relay. Cheap to clone; every clone is the same relay.
#[derive(Clone)]
pub struct Relay {
pub(crate) inner: Arc<Inner>,
}
/// The smaller of two delivery timeouts where both are set, either where one is; 0 none
/// (§5.2).
pub fn effective_timeout(publisher_ms: u64, subscriber_ms: u64) -> u64 {
match (publisher_ms, subscriber_ms) {
(0, s) => s,
(p, 0) => p,
(p, s) => p.min(s),
}
}
pub fn now_unix_ms() -> u64 {
std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0)
}
fn load_key(path: &std::path::Path) -> Result<SigningKey, String> {
let raw = std::fs::read(path).map_err(|e| format!("the summary key {}: {e}", path.display()))?;
let seed: [u8; 32] = if raw.len() == 32 {
raw.try_into().expect("32 bytes")
} else {
let s = String::from_utf8_lossy(&raw);
hex::decode(s.trim()).map_err(|e| format!("the summary key is neither 32 bytes nor hex: {e}"))?.try_into().map_err(|_| "the summary key is not 32 bytes".to_string())?
};
Ok(SigningKey::from_bytes(&seed))
}
/// Where one upstream object goes: the fields the fan-out reads.
pub struct ObjectIn<'a> {
pub upstream: u64,
pub header: &'a SubgroupHeader,
pub subgroup_id: u64,
pub first_in_stream: Option<u64>,
pub object: &'a SubgroupObject,
pub arrived: Instant,
pub object_timeout: u64,
pub signed: Option<(u64, &'static str)>,
}
impl Relay {
pub fn new(cfg: RelayConfig) -> Result<Self, String> {
let auth = Authorizer::from_config(&cfg.auth)?;
let door = Door::new(&cfg.receipts.door, &cfg.receipts.receipts_track_suffix)?;
let key = match &cfg.summary.key_path {
Some(p) => load_key(p)?,
None => SigningKey::generate(&mut rand_core_06::OsRng),
};
let budget = Budget::new(cfg.cache.max_bytes);
Ok(Relay {
inner: Arc::new(Inner {
cfg,
auth,
door,
key,
state: Mutex::new(State::default()),
budget: Mutex::new(budget),
ids: AtomicU64::new(1),
summaries: Mutex::new(Vec::new()),
going_away: AtomicBool::new(false),
stats: Stats::default(),
change: Notify::new(),
sessions_gone: Notify::new(),
}),
})
}
pub fn config(&self) -> &RelayConfig {
&self.inner.cfg
}
/// The public key the relay signs session summaries with.
pub fn summary_key(&self) -> ed25519_dalek::VerifyingKey {
self.inner.key.verifying_key()
}
pub fn next_id(&self) -> u64 {
self.inner.ids.fetch_add(1, Ordering::Relaxed)
}
/// The signed summaries of the sessions that have ended.
pub fn summaries(&self) -> Vec<SignedSessionSummary> {
self.inner.summaries.lock().unwrap().clone()
}
pub fn stats(&self) -> RelayStats {
let s = &self.inner.stats;
let st = self.inner.state.lock().unwrap();
RelayStats {
sessions_opened: s.sessions_opened.load(Ordering::Relaxed),
sessions_open: st.sessions.len() as u64,
objects_in: s.objects_in.load(Ordering::Relaxed),
fetch_served_from_cache: s.fetch_served_from_cache.load(Ordering::Relaxed),
fetch_served_upstream: s.fetch_served_upstream.load(Ordering::Relaxed),
refused_at_door: s.refused_at_door.load(Ordering::Relaxed),
groups_dropped_at_door: s.groups_dropped_at_door.load(Ordering::Relaxed),
queued: st.sessions.values().map(|c| c.sched.queued()).sum(),
}
}
/// Whether a FETCH starting at `start` would be served from the cache alone.
pub fn cache_covers(&self, key: &TrackKey, start: Location) -> bool {
self.inner.state.lock().unwrap().tracks.get(key).is_some_and(|t| t.cache.covers(start))
}
/// The Largest Object the relay knows of a track (§3.1.4).
pub fn largest(&self, key: &TrackKey) -> Option<Location> {
self.inner.state.lock().unwrap().tracks.get(key).and_then(|t| t.largest)
}
/// Each session: its id, its peer, its transport's counters, and the items its
/// scheduler holds, for diagnosis.
pub fn session_diagnostics(&self) -> Vec<(u64, String, Option<String>, u64)> {
let st = self.inner.state.lock().unwrap();
st.sessions.values().map(|c| (c.id, c.remote.clone(), c.conn.diagnostics(), c.sched.queued())).collect()
}
pub fn going_away(&self) -> bool {
self.inner.going_away.load(Ordering::Acquire)
}
/// Wait until the alias `(session, alias)` names a track, up to `wait` (§3.1.3.1: data
/// can arrive before the control message that establishes its alias).
pub async fn resolve_alias(&self, session: u64, alias: u64, wait: Duration) -> Option<TrackKey> {
let deadline = Instant::now() + wait;
loop {
let notified = self.inner.change.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if let Some(k) = self.inner.state.lock().unwrap().aliases.get(&(session, alias)) {
return Some(k.clone());
}
if tokio::time::timeout_at(deadline, notified).await.is_err() {
return None;
}
}
}
/// The receipts track of a media track, and the media track of a receipts track.
pub fn receipts_of(&self, key: &TrackKey) -> TrackKey {
(key.0.clone(), self.inner.door.receipts_of(&key.1))
}
pub fn is_receipts_track(&self, name: &TrackName) -> bool {
self.inner.door.media_of(name).is_some()
}
/// Account bytes to the cache budget, evicting the oldest groups past it.
fn charge_cache(&self, st: &mut State, track_id: u64, group: u64, n: usize) {
let evict = self.inner.budget.lock().unwrap().add(track_id, group, n);
for (t, g) in evict {
if let Some(k) = st.track_ids.get(&t).cloned()
&& let Some(tr) = st.tracks.get_mut(&k)
{
cache::evict(&mut tr.cache, g);
}
}
}
/// Create a track's state if it has none.
pub fn track_entry<'a>(&self, st: &'a mut State, key: &TrackKey) -> &'a mut Track {
if !st.tracks.contains_key(key) {
let id = self.next_id();
st.track_ids.insert(id, key.clone());
st.tracks.insert(key.clone(), Track::new(id, key.clone()));
}
st.tracks.get_mut(key).expect("the track")
}
/// One object of an upstream subgroup stream: cached, then queued to every downstream
/// subscription its filter passes, each with its own deadline and priority. `started`
/// collects the subscriptions this stream has been queued to. Returns whether the
/// stream is the one the relay takes its subgroup from (§7.3): the cache counts it
/// open from its first object, and [`Relay::cache_stream_ended`] closes the count.
pub fn fan_out_object(&self, key: &TrackKey, o: ObjectIn<'_>, started: &mut HashSet<u64>) -> bool {
let mut st = self.inner.state.lock().unwrap();
let receipts_key = self.receipts_of(key);
let is_receipts = self.is_receipts_track(&key.1);
let rsessions: HashSet<u64> = if is_receipts {
HashSet::new()
} else {
st.tracks.get(&receipts_key).map(|t| t.downs.iter().map(|d| d.session).collect()).unwrap_or_default()
};
let session_tally = st.sessions.get(&o.upstream).map(|c| c.tally.clone());
let Some(track) = st.tracks.get_mut(key) else { return false };
let group = o.header.group_id;
let oid = o.object.object_id;
match track.owners.get(&(group, o.subgroup_id)) {
Some(owner) if *owner != o.upstream => {
if let Some(t) = session_tally {
t.lock().unwrap().publisher.deduplicated += 1;
}
return false;
}
Some(_) => {}
None => {
track.owners.insert((group, o.subgroup_id), o.upstream);
}
}
if o.first_in_stream == Some(oid) {
track.cache.stream_opened(group, oid == 0);
}
let loc = Location::new(group, oid);
track.note_largest(loc);
let priority = o.header.publisher_priority.unwrap_or(track.default_priority);
let cached = CachedObject {
subgroup: Some(o.subgroup_id),
publisher_priority: priority,
properties: o.object.properties.clone(),
status: o.object.status,
payload: Bytes::from(o.object.payload.clone()),
};
let size = cache::size_of(&cached);
let ins = track.cache.insert(group, oid, cached);
if ins == Insert::Conflict {
tracing::warn!(track = %track.render, group, object = oid, "a duplicate object that differs: the track is malformed (§7.1)");
track.cache.group_broken(group);
return true;
}
let track_id = track.id;
for sub in &track.downs {
let p = sub.params.lock().unwrap().clone();
if !p.passes(loc) {
continue;
}
let first_for_sub = started.insert(sub.id);
let starts_stream = first_for_sub && o.first_in_stream == Some(oid);
let header = SubgroupHeader {
track_alias: sub.alias,
group_id: group,
subgroup_id: if starts_stream { o.header.subgroup_id } else { SubgroupId::Explicit(o.subgroup_id) },
publisher_priority: o.header.publisher_priority,
properties: o.header.properties,
end_of_group: o.header.end_of_group,
first_object: o.header.first_object && starts_stream,
};
let timeout = if sub.receipts_track { 0 } else { effective_timeout(o.object_timeout, p.object_timeout) };
let deadline = (timeout != 0).then(|| o.arrived + Duration::from_millis(timeout));
let prio = Prio { subscriber: p.subscriber_priority, publisher: priority, group_rank: Prio::group_rank(p.order, group), fill_rank: 1, mode_rank: 1, within: o.subgroup_id };
let meta = ObjMeta {
track: sub.render.clone(),
payload_len: o.object.payload.len() as u64,
receipts_subscribed: rsessions.contains(&sub.session),
fetched: false,
signed: o.signed,
};
sub.sched.push(
LaneKey { sub: sub.id, kind: LaneKind::Subgroup { group, subgroup: o.subgroup_id } },
Item { op: Op::Object { header, object: o.object.clone() }, prio, deadline, meta: Some(meta) },
);
}
if ins == Insert::New {
self.charge_cache(&mut st, track_id, group, size);
}
true
}
/// An upstream subgroup stream the cache counted ended: with a FIN (`clean`), or
/// reset, cut short or refused.
pub fn cache_stream_ended(&self, key: &TrackKey, group: u64, clean: bool) {
if let Some(t) = self.inner.state.lock().unwrap().tracks.get_mut(key) {
t.cache.stream_ended(group, clean);
}
}
/// An upstream subgroup stream ended: with a FIN, or reset with `reset`.
pub fn end_subgroup(&self, key: &TrackKey, group: u64, subgroup: u64, subgroup_timeout_pub: u64, started: &HashSet<u64>, reset: Option<u64>) {
let st = self.inner.state.lock().unwrap();
let Some(track) = st.tracks.get(key) else { return };
for sub in track.downs.iter().filter(|d| started.contains(&d.id)) {
let lane = LaneKey { sub: sub.id, kind: LaneKind::Subgroup { group, subgroup } };
let p = sub.params.lock().unwrap().clone();
let op = match reset {
Some(code) => Op::Reset { code },
None => {
let t = if sub.receipts_track { 0 } else { effective_timeout(subgroup_timeout_pub, p.subgroup_timeout) };
Op::Fin { subgroup_timeout: (t != 0).then(|| Duration::from_millis(t)) }
}
};
let prio = Prio { subscriber: p.subscriber_priority, publisher: track.default_priority, group_rank: Prio::group_rank(p.order, group), fill_rank: 1, mode_rank: 1, within: subgroup };
sub.sched.push(lane, Item { op, prio, deadline: None, meta: None });
}
}
/// One object datagram from upstream (§11.2).
pub fn fan_out_datagram(&self, key: &TrackKey, upstream: u64, d: &ObjectDatagram, arrived: Instant, signed: Option<(u64, &'static str)>) {
use joule_moq::draft22::data::DatagramContent;
let mut st = self.inner.state.lock().unwrap();
let receipts_key = self.receipts_of(key);
let rsessions: HashSet<u64> = st.tracks.get(&receipts_key).map(|t| t.downs.iter().map(|d| d.session).collect()).unwrap_or_default();
let Some(track) = st.tracks.get_mut(key) else { return };
let loc = Location::new(d.group_id, d.object_id);
let _ = upstream;
track.note_largest(loc);
let priority = d.publisher_priority.unwrap_or(track.default_priority);
let (status, payload) = match &d.content {
DatagramContent::Payload(p) => (joule_moq::draft22::data::ObjectStatus::Normal, p.clone()),
DatagramContent::Status(s) => (*s, Vec::new()),
};
let cached = CachedObject { subgroup: None, publisher_priority: priority, properties: d.properties.clone().unwrap_or_default(), status, payload: Bytes::from(payload.clone()) };
let size = cache::size_of(&cached);
let ins = track.cache.insert(d.group_id, d.object_id, cached);
if d.end_of_group && ins != Insert::Evicted {
track.cache.group_ended(d.group_id);
}
if ins != Insert::New {
// A datagram already forwarded (another publisher's, or a repeat): once only.
return;
}
let track_id = track.id;
let object_timeout = effective_timeout(track.pub_object_timeout, track.pub_subgroup_timeout);
for sub in &track.downs {
let p = sub.params.lock().unwrap().clone();
if !p.passes(loc) {
continue;
}
// For a datagram SUBGROUP_DELIVERY_TIMEOUT acts as OBJECT_DELIVERY_TIMEOUT (§5.2).
let t = effective_timeout(object_timeout, effective_timeout(p.object_timeout, p.subgroup_timeout));
let deadline = (t != 0 && !sub.receipts_track).then(|| arrived + Duration::from_millis(t));
let mut out = d.clone();
out.track_alias = sub.alias;
let prio = Prio { subscriber: p.subscriber_priority, publisher: priority, group_rank: Prio::group_rank(p.order, d.group_id), fill_rank: 1, mode_rank: 0, within: d.object_id };
let meta = ObjMeta { track: sub.render.clone(), payload_len: payload.len() as u64, receipts_subscribed: rsessions.contains(&sub.session), fetched: false, signed };
sub.sched.push(LaneKey { sub: sub.id, kind: LaneKind::Datagram { group: d.group_id, object: d.object_id } }, Item { op: Op::Datagram { datagram: out }, prio, deadline, meta: Some(meta) });
}
self.charge_cache(&mut st, track_id, d.group_id, size);
}
/// An upstream of `key` is gone. When it was the last, every downstream
/// subscription ends with PUBLISH_DONE once what it has queued is sent (§9.9).
pub fn upstream_gone(&self, session: u64, key: &TrackKey, status: PublishDoneCode) {
let ends = {
let mut st = self.inner.state.lock().unwrap();
let mut gone_aliases = Vec::new();
let ends = match st.tracks.get_mut(key) {
None => Vec::new(),
Some(track) => {
track.upstreams.retain(|u| {
if u.session == session {
gone_aliases.push(u.alias);
false
} else {
true
}
});
if track.upstreams.is_empty() {
track.ended = Some(status);
std::mem::take(&mut track.downs)
} else {
Vec::new()
}
}
};
for a in gone_aliases {
st.aliases.remove(&(session, a));
}
ends
};
for sub in ends {
end_down(sub, status);
}
self.inner.change.notify_waiters();
}
/// Remove a downstream subscription (its subscriber cancelled it, or its session
/// ended), and cancel the relay's own upstream SUBSCRIBE when nobody is left.
pub fn remove_down(&self, sub: &DownSub) {
let cancel = {
let mut st = self.inner.state.lock().unwrap();
match st.tracks.get_mut(&sub.track) {
None => Vec::new(),
Some(track) => {
track.downs.retain(|d| d.id != sub.id);
if track.downs.is_empty() {
let mut c = Vec::new();
track.upstreams.retain(|u| match &u.kind {
UpstreamKind::Subscribe(tx) => {
c.push((u.session, u.alias, tx.clone()));
false
}
UpstreamKind::Publish => true,
});
for (s, a, _) in &c {
st.aliases.remove(&(*s, *a));
}
c
} else {
Vec::new()
}
}
}
};
for (_, _, tx) in cancel {
tokio::spawn(async move {
tx.reset(joule_moq::draft22::StreamResetCode::CANCELLED.0).await;
});
}
}
/// Wait for every session to end, up to `until`.
pub async fn wait_sessions_gone(&self, until: Instant) -> bool {
loop {
let notified = self.inner.sessions_gone.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if self.inner.state.lock().unwrap().sessions.is_empty() {
return true;
}
if tokio::time::timeout_at(until, notified).await.is_err() {
return self.inner.state.lock().unwrap().sessions.is_empty();
}
}
}
/// GOAWAY to every session (§9.2), then wait up to `timeout` for them to leave, and
/// close the rest with GOAWAY_TIMEOUT (§6.6.1). New requests meanwhile are refused
/// with GOING_AWAY. Returns how many sessions had to be closed.
pub async fn shutdown(&self, timeout: Duration) -> usize {
self.inner.going_away.store(true, Ordering::Release);
let sessions: Vec<Arc<SessionCtx>> = self.inner.state.lock().unwrap().sessions.values().cloned().collect();
let uri = self.inner.cfg.goaway_uri.clone().into_bytes();
for ctx in &sessions {
crate::session::send_goaway(ctx, &uri, timeout.as_millis() as u64).await;
}
let deadline = Instant::now() + timeout;
if self.wait_sessions_gone(deadline).await {
return 0;
}
let left: Vec<Arc<SessionCtx>> = self.inner.state.lock().unwrap().sessions.values().cloned().collect();
for ctx in &left {
ctx.close(joule_moq::draft22::SessionError::GOAWAY_TIMEOUT.0, "GOAWAY timeout");
}
let _ = self.wait_sessions_gone(Instant::now() + Duration::from_secs(5)).await;
left.len()
}
/// Namespaces published to the relay that `prefix` matches, other than `except`'s.
pub fn published_under(&self, st: &State, prefix: &TrackNamespace, except: u64) -> Vec<TrackNamespace> {
let mut out: Vec<TrackNamespace> = st.namespaces.iter().filter(|n| n.session != except && prefix_matches(prefix, &n.namespace)).map(|n| n.namespace.clone()).collect();
out.sort();
out.dedup();
out
}
/// The sessions that published a namespace `ns` falls under, other than `except`.
pub fn publishers_of(&self, st: &State, ns: &TrackNamespace, except: u64) -> Vec<Arc<SessionCtx>> {
let mut ids: Vec<u64> = st.namespaces.iter().filter(|n| n.session != except && prefix_matches(&n.namespace, ns)).map(|n| n.session).collect();
ids.sort();
ids.dedup();
ids.iter().filter_map(|i| st.sessions.get(i).cloned()).collect()
}
}
/// End a downstream subscription: after its queue drains, PUBLISH_DONE with the streams
/// it opened, then a FIN (§6.4.2.2, §9.9).
pub fn end_down(sub: Arc<DownSub>, status: PublishDoneCode) {
let _ = sub.ended.send(true);
tokio::spawn(async move {
let n = sub.sched.drained(sub.id).await;
let done = joule_moq::draft22::ControlMessage::PublishDone(joule_moq::draft22::PublishDone { status, stream_count: n, reason: Vec::new() });
let _ = sub.tx.send(&done).await;
sub.tx.fin().await;
});
}
/// The class label and figure of a door verdict, for the tally.
pub fn signed_of(v: &crate::door::Verdict) -> Option<(u64, &'static str)> {
match v {
crate::door::Verdict::Pass { joules_micro, class } => Some((*joules_micro, class_label(class.as_deref()))),
_ => None,
}
}

766
joule-code/joule-code-rs/crates/joule-moq-relay/src/sched.rs Normal file
View file

@ -0,0 +1,766 @@
//! The scheduler of one downstream session: what the relay sends it, in what order, and
//! what it drops (`draft-ietf-moq-transport-22` §5.1, §5.2, §7.4).
//!
//! Everything the relay sends a subscriber passes through here: objects of its
//! subscriptions on subgroup streams and datagrams, and objects of its fetches and fills on
//! fetch streams. Each data stream is a **lane**: a queue whose items go out in order, one
//! at a time, so a relay never reorders the objects of a stream (§7.4). Among the lanes
//! with something to send, the scheduler picks by §5.1.2:
//!
//! 1. the lower subscriber priority number;
//! 2. then the lower publisher priority number;
//! 3. then, within a subscription, the group its Group Order puts first (and, between a
//! fill and its subscription in one group, the fill);
//! 4. then the lower Subgroup ID, or Object ID for a datagram (a datagram before a
//! subgroup object);
//!
//! and first come, first served after that. At most `write_concurrency` writes are in
//! flight, so what waits, waits here, where it can still be dropped, and not in the
//! transport's buffers (§5.2: "minimize the amount of data buffered at the underlying
//! transport layer"). Each stream is also given a transport priority from rules 1 and 2,
//! so bytes the transport already holds are sent in the same order.
//!
//! **Delivery timeouts.** An object carries the deadline its OBJECT_DELIVERY_TIMEOUT
//! sets, counted from when it reached the relay. It is checked before the object goes to
//! the transport, and bounds the write itself; past it the subgroup's stream is reset
//! with DELIVERY_TIMEOUT and nothing more of that subgroup is sent (§5.2), and a datagram
//! is dropped. SUBGROUP_DELIVERY_TIMEOUT starts at the subgroup's FIN; if the stream's
//! data is not all acknowledged by then, the stream is reset.
//!
//! Receipts tracks carry no deadline (WAI moq-streaming-format §4.1), so the receipts of
//! an object dropped here still reach a subscriber of its receipts track.
use crate::summary::Tally;
use crate::transport::{BoxSend, SharedConnection, TransportError};
use joule_moq::draft22::data::{FetchCodec, FetchEntry, FetchHeader, ObjectDatagram, SubgroupCodec, SubgroupHeader, SubgroupObject};
use joule_moq::draft22::{GroupOrder, StreamResetCode};
use std::collections::{BTreeSet, HashMap, VecDeque};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::sync::{mpsc, oneshot};
use tokio::time::Instant;
/// An item's place in §5.1.2's order: lower first, compared field by field.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Prio {
pub subscriber: u8,
pub publisher: u8,
/// The group's rank in its request's Group Order: the Group ID ascending, its
/// complement descending.
pub group_rank: u64,
/// 0 for a fill-delivered object, 1 for a subscription-delivered one.
pub fill_rank: u8,
/// 0 for a datagram, 1 for a subgroup object.
pub mode_rank: u8,
/// The Subgroup ID, or the Object ID of a datagram.
pub within: u64,
}
impl Prio {
pub fn group_rank(order: GroupOrder, group: u64) -> u64 {
match order {
GroupOrder::Ascending => group,
GroupOrder::Descending => u64::MAX - group,
}
}
/// The transport's priority for a stream of this item: higher first.
fn transport_priority(&self) -> i32 {
-(((self.subscriber as i32) << 8) | self.publisher as i32)
}
}
/// A data stream (or one datagram) of one downstream request.
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum LaneKind {
Subgroup { group: u64, subgroup: u64 },
Datagram { group: u64, object: u64 },
Fetch { request_id: u64 },
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct LaneKey {
/// The relay's id of the downstream subscription or fetch.
pub sub: u64,
pub kind: LaneKind,
}
/// What an object item counts toward.
#[derive(Clone, Debug)]
pub struct ObjMeta {
/// The track, rendered (§8.8).
pub track: Arc<str>,
pub payload_len: u64,
/// The session also subscribes to this track's receipts track.
pub receipts_subscribed: bool,
/// A fetch or fill object.
pub fetched: bool,
/// The publisher's signed figure for the object and its class label, where the door
/// verified its receipt.
pub signed: Option<(u64, &'static str)>,
}
pub enum Op {
Object { header: SubgroupHeader, object: SubgroupObject },
/// The subgroup is complete upstream: FIN the stream, then hold it to
/// SUBGROUP_DELIVERY_TIMEOUT.
Fin { subgroup_timeout: Option<Duration> },
/// The upstream stream was reset: reset this one.
Reset { code: u64 },
Datagram { datagram: ObjectDatagram },
FetchEntry { request_id: u64, order: GroupOrder, entry: FetchEntry },
FetchFin { request_id: u64, order: GroupOrder },
/// End a fetch stream abruptly, opening it first if it is not (a fill that fails,
/// §3.4.1).
FetchReset { request_id: u64, order: GroupOrder, code: u64 },
}
pub struct Item {
pub op: Op,
pub prio: Prio,
pub deadline: Option<Instant>,
pub meta: Option<ObjMeta>,
}
// A push carries its object by value: the scheduler is the one place it is held.
#[allow(clippy::large_enum_variant)]
enum Cmd {
Push(LaneKey, Item),
Cancel { sub: u64, code: u64 },
Priority { sub: u64, subscriber: u8 },
Drain { sub: u64, tx: oneshot::Sender<u64> },
Shutdown(Option<oneshot::Sender<()>>),
}
/// A handle on a session's scheduler.
#[derive(Clone)]
pub struct SchedHandle {
tx: mpsc::UnboundedSender<Cmd>,
queued: Arc<AtomicU64>,
}
impl SchedHandle {
pub fn push(&self, lane: LaneKey, item: Item) {
self.queued.fetch_add(1, Ordering::AcqRel);
if self.tx.send(Cmd::Push(lane, item)).is_err() {
self.queued.fetch_sub(1, Ordering::AcqRel);
}
}
/// End every stream of a request and send nothing more for it (§3.1.2: "It MUST
/// reset any open streams associated with the SUBSCRIBE").
pub fn cancel(&self, sub: u64, code: u64) {
let _ = self.tx.send(Cmd::Cancel { sub, code });
}
/// A REQUEST_UPDATE changed a request's subscriber priority: applied to what is
/// queued, best effort (§5.1.1).
pub fn set_priority(&self, sub: u64, subscriber: u8) {
let _ = self.tx.send(Cmd::Priority { sub, subscriber });
}
/// Resolves when nothing of `sub` is queued or being written, with the number of
/// streams opened for it: what PUBLISH_DONE may then say (§9.9).
pub async fn drained(&self, sub: u64) -> u64 {
let (tx, rx) = oneshot::channel();
if self.tx.send(Cmd::Drain { sub, tx }).is_err() {
return u64::MAX;
}
rx.await.unwrap_or(u64::MAX)
}
pub fn shutdown(&self) {
let _ = self.tx.send(Cmd::Shutdown(None));
}
/// Stop, and wait until what was left queued is counted.
pub async fn shutdown_and_wait(&self) {
let (tx, rx) = oneshot::channel();
if self.tx.send(Cmd::Shutdown(Some(tx))).is_ok() {
let _ = rx.await;
}
}
/// Items pushed and not yet dispatched.
pub fn queued(&self) -> u64 {
self.queued.load(Ordering::Acquire)
}
}
#[derive(Default)]
struct SubInfo {
/// Items queued or being written.
pending: u64,
streams_opened: u64,
waiters: Vec<oneshot::Sender<u64>>,
cancelled: bool,
}
#[derive(Default)]
struct Lane {
queue: VecDeque<(u64, Item)>,
send: Option<BoxSend>,
opened: bool,
busy: bool,
dead: bool,
subgroup: Option<SubgroupCodec>,
fetch: Option<FetchCodec>,
index: Option<(Prio, u64)>,
/// Why the lane was ended early: its reset code.
dead_code: Option<u64>,
/// The track of its objects, for the tally.
track: Option<Arc<str>>,
}
enum Then {
Nothing,
Finish(Option<Duration>),
Reset(u64),
}
struct Job {
lane: LaneKey,
send: Option<BoxSend>,
bytes: Vec<u8>,
then: Then,
deadline: Option<Instant>,
priority: i32,
meta: Option<ObjMeta>,
track: Option<Arc<str>>,
}
enum Done {
Write { lane: LaneKey, send: Option<BoxSend>, result: Result<(), TransportError>, meta: Option<ObjMeta>, bytes: u64, ended: bool },
/// A subgroup was reset at its SUBGROUP_DELIVERY_TIMEOUT, after its FIN.
SubgroupTimedOut { track: Option<Arc<str>> },
}
struct Written {
lane: LaneKey,
send: Option<BoxSend>,
result: Result<(), TransportError>,
meta: Option<ObjMeta>,
bytes: u64,
ended: bool,
}
struct Sched {
conn: SharedConnection,
lanes: HashMap<LaneKey, Lane>,
ready: BTreeSet<(Prio, u64, LaneKey)>,
in_flight: usize,
max_in_flight: usize,
seq: u64,
subs: HashMap<u64, SubInfo>,
tally: Arc<Mutex<Tally>>,
queued: Arc<AtomicU64>,
done_tx: mpsc::UnboundedSender<Done>,
}
/// Start a session's scheduler. It runs until [`SchedHandle::shutdown`] or until the
/// connection fails.
pub fn spawn(conn: SharedConnection, write_concurrency: usize, tally: Arc<Mutex<Tally>>) -> SchedHandle {
let (tx, mut rx) = mpsc::unbounded_channel();
let (done_tx, mut done_rx) = mpsc::unbounded_channel();
let queued = Arc::new(AtomicU64::new(0));
let mut s = Sched {
conn,
lanes: HashMap::new(),
ready: BTreeSet::new(),
in_flight: 0,
max_in_flight: write_concurrency.max(1),
seq: 0,
subs: HashMap::new(),
tally,
queued: queued.clone(),
done_tx,
};
tokio::spawn(async move {
let mut closed: Option<oneshot::Sender<()>> = None;
loop {
while s.in_flight < s.max_in_flight {
let Some((_, _, key)) = s.ready.pop_first() else { break };
s.dispatch(key);
}
tokio::select! {
biased;
d = done_rx.recv() => match d {
Some(d) => s.on_done(d),
None => break,
},
c = rx.recv() => match c {
Some(Cmd::Push(k, i)) => s.push(k, i),
Some(Cmd::Cancel { sub, code }) => s.cancel(sub, code),
Some(Cmd::Priority { sub, subscriber }) => s.reprioritise(sub, subscriber),
Some(Cmd::Drain { sub, tx }) => s.drain(sub, tx),
Some(Cmd::Shutdown(tx)) => {
closed = tx;
break;
}
None => break,
},
}
}
s.close();
if let Some(tx) = closed {
let _ = tx.send(());
}
});
SchedHandle { tx, queued }
}
fn expired(deadline: Option<Instant>) -> bool {
deadline.is_some_and(|d| Instant::now() >= d)
}
impl Sched {
fn sub(&mut self, sub: u64) -> &mut SubInfo {
self.subs.entry(sub).or_default()
}
fn settle(&mut self, sub: u64, n: u64) {
let info = self.sub(sub);
info.pending = info.pending.saturating_sub(n);
if info.pending == 0 {
let opened = info.streams_opened;
for w in info.waiters.drain(..) {
let _ = w.send(opened);
}
}
}
fn push(&mut self, key: LaneKey, item: Item) {
self.queued.fetch_sub(1, Ordering::AcqRel);
if self.sub(key.sub).cancelled {
return;
}
self.sub(key.sub).pending += 1;
self.seq += 1;
let seq = self.seq;
let lane = self.lanes.entry(key).or_default();
lane.queue.push_back((seq, item));
self.refresh(key);
}
/// Put the lane in the ready index exactly when it has something to send and no
/// write in flight.
fn refresh(&mut self, key: LaneKey) {
let Some(lane) = self.lanes.get_mut(&key) else { return };
if let Some((p, s)) = lane.index.take() {
self.ready.remove(&(p, s, key));
}
if lane.busy {
return;
}
if let Some((seq, item)) = lane.queue.front() {
lane.index = Some((item.prio, *seq));
self.ready.insert((item.prio, *seq, key));
}
}
fn count_dropped(&mut self, meta: &ObjMeta) {
let mut t = self.tally.lock().unwrap();
t.subscriber.dropped_delivery_timeout += 1;
let tr = t.track(&meta.track);
tr.dropped_delivery_timeout += 1;
if meta.receipts_subscribed {
tr.dropped_receipts_track_subscribed += 1;
t.subscriber.dropped_receipts_track_subscribed += 1;
}
}
/// The lane's stream is reset for a delivery timeout: what it still holds is dropped
/// with it, until its subgroup ends upstream.
fn kill(&mut self, key: LaneKey, code: u64) {
let Some(lane) = self.lanes.get_mut(&key) else { return };
lane.dead = true;
lane.dead_code.get_or_insert(code);
if let Some(s) = lane.send.as_mut() {
s.reset(code);
}
let mut dropped = Vec::new();
let mut keep = VecDeque::new();
for (seq, item) in lane.queue.drain(..) {
match item.op {
Op::Fin { .. } | Op::Reset { .. } | Op::FetchFin { .. } | Op::FetchReset { .. } => keep.push_back((seq, item)),
_ => dropped.push(item),
}
}
lane.queue = keep;
let n = dropped.len() as u64;
for item in dropped {
if let Some(m) = &item.meta
&& code == StreamResetCode::DELIVERY_TIMEOUT.0
{
self.count_dropped(m);
}
}
self.settle(key.sub, n);
self.refresh(key);
}
fn remove_lane(&mut self, key: LaneKey) {
if let Some(mut lane) = self.lanes.remove(&key) {
if let Some((p, s)) = lane.index.take() {
self.ready.remove(&(p, s, key));
}
let n = lane.queue.len() as u64;
self.settle(key.sub, n);
}
}
fn spawn_job(&mut self, job: Job) {
self.in_flight += 1;
let conn = self.conn.clone();
let done = self.done_tx.clone();
tokio::spawn(async move {
let Job { lane, send, bytes, then, deadline, priority, meta, track } = job;
let mut send = send;
let n = bytes.len() as u64;
// Opening the stream (which can wait for the peer's stream credit) and the
// write are both bounded by the object's deadline.
let work = async {
if send.is_none() {
let mut s = conn.open_uni().await?;
s.set_priority(priority);
send = Some(s);
}
let s = send.as_mut().expect("a stream");
if !bytes.is_empty() {
s.write_all(&bytes).await?;
}
Ok::<(), TransportError>(())
};
let result = match deadline {
Some(d) => {
// The write future (which borrows the stream) ends with this statement.
let timed = tokio::time::timeout_at(d, work).await;
match timed {
Ok(r) => r,
Err(_) => {
if let Some(s) = send.as_mut() {
s.reset(StreamResetCode::DELIVERY_TIMEOUT.0);
}
Err(TransportError::Deadline)
}
}
}
None => work.await,
};
let mut ended = false;
if result.is_ok() {
match then {
Then::Nothing => {}
Then::Reset(code) => {
if let Some(s) = send.as_mut() {
s.reset(code);
}
ended = true;
}
Then::Finish(sgt) => {
ended = true;
if let Some(mut s) = send.take() {
let _ = s.finish();
if let Some(t) = sgt {
// Held to SUBGROUP_DELIVERY_TIMEOUT: reset unless all of
// it is acknowledged first.
let done2 = done.clone();
tokio::spawn(async move {
if tokio::time::timeout(t, s.delivered()).await.is_err() {
s.reset(StreamResetCode::DELIVERY_TIMEOUT.0);
let _ = done2.send(Done::SubgroupTimedOut { track });
}
});
}
}
}
}
}
let _ = done.send(Done::Write { lane, send, result, meta, bytes: n, ended });
});
}
fn dispatch(&mut self, key: LaneKey) {
let Some(lane) = self.lanes.get_mut(&key) else { return };
lane.index = None;
let Some((_, item)) = lane.queue.pop_front() else { return };
let Item { op, prio, deadline, meta } = item;
if lane.dead {
// A subgroup already reset: only its end is still awaited.
let ends = matches!(op, Op::Fin { .. } | Op::Reset { .. } | Op::FetchFin { .. } | Op::FetchReset { .. });
let timed_out = lane.dead_code == Some(StreamResetCode::DELIVERY_TIMEOUT.0);
if let (Some(m), true) = (&meta, timed_out) {
self.count_dropped(m);
}
self.settle(key.sub, 1);
if ends {
self.remove_lane(key);
} else {
self.refresh(key);
}
return;
}
let priority = prio.transport_priority();
match op {
Op::Object { header, object } => {
if expired(deadline) {
if let Some(m) = &meta {
self.count_dropped(m);
}
self.settle(key.sub, 1);
self.kill(key, StreamResetCode::DELIVERY_TIMEOUT.0);
return;
}
let mut bytes = Vec::new();
let lane = self.lanes.get_mut(&key).expect("the lane");
if lane.track.is_none() {
lane.track = meta.as_ref().map(|m| m.track.clone());
}
if !lane.opened {
header.encode(&mut bytes);
lane.subgroup = Some(SubgroupCodec::new(header));
lane.opened = true;
self.subs.entry(key.sub).or_default().streams_opened += 1;
}
let lane = self.lanes.get_mut(&key).expect("the lane");
let enc = lane.subgroup.as_mut().expect("a subgroup codec").encode_object(&object, &mut bytes);
if enc.is_err() {
self.settle(key.sub, 1);
self.kill(key, StreamResetCode::INTERNAL_ERROR.0);
return;
}
lane.busy = true;
let send = lane.send.take();
self.spawn_job(Job { lane: key, send, bytes, then: Then::Nothing, deadline, priority, meta, track: None });
}
Op::Fin { subgroup_timeout } => {
if !lane.opened {
self.settle(key.sub, 1);
self.remove_lane(key);
return;
}
lane.busy = true;
let send = lane.send.take();
let track = lane.track.clone();
self.spawn_job(Job { lane: key, send, bytes: vec![], then: Then::Finish(subgroup_timeout), deadline: None, priority, meta: None, track });
}
Op::Reset { code } => {
if let Some(s) = lane.send.as_mut() {
s.reset(code);
}
self.settle(key.sub, 1);
self.remove_lane(key);
}
Op::Datagram { datagram } => {
self.settle(key.sub, 1);
if expired(deadline) {
if let Some(m) = &meta {
self.count_dropped(m);
}
} else if let Ok(b) = datagram.encode() {
let n = b.len() as u64;
if self.conn.send_datagram(b.into()).is_ok()
&& let Some(m) = &meta
{
self.count_forwarded(m, n);
}
}
self.remove_lane(key);
}
Op::FetchEntry { request_id, order, entry } => {
let mut bytes = Vec::new();
if !lane.opened {
FetchHeader { request_id }.encode(&mut bytes);
lane.fetch = Some(FetchCodec::new(order));
lane.opened = true;
self.subs.entry(key.sub).or_default().streams_opened += 1;
}
let lane = self.lanes.get_mut(&key).expect("the lane");
if lane.fetch.as_mut().expect("a fetch codec").encode(&entry, &mut bytes).is_err() {
self.settle(key.sub, 1);
self.kill(key, StreamResetCode::INTERNAL_ERROR.0);
return;
}
lane.busy = true;
let send = lane.send.take();
self.spawn_job(Job { lane: key, send, bytes, then: Then::Nothing, deadline: None, priority, meta, track: None });
}
Op::FetchFin { request_id, order } | Op::FetchReset { request_id, order, .. } => {
let then = match op {
Op::FetchReset { code, .. } => Then::Reset(code),
_ => Then::Finish(None),
};
let mut bytes = Vec::new();
if !lane.opened {
// A fetch with nothing in range still opens its stream and sends its
// header (§3.2.1); a failing fill resets right after it (§3.4.1).
FetchHeader { request_id }.encode(&mut bytes);
lane.fetch = Some(FetchCodec::new(order));
lane.opened = true;
self.subs.entry(key.sub).or_default().streams_opened += 1;
}
let lane = self.lanes.get_mut(&key).expect("the lane");
lane.busy = true;
let send = lane.send.take();
self.spawn_job(Job { lane: key, send, bytes, then, deadline: None, priority, meta: None, track: None });
}
}
}
fn count_forwarded(&mut self, m: &ObjMeta, wire: u64) {
let mut t = self.tally.lock().unwrap();
t.subscriber.wire_bytes_sent += wire;
if m.fetched {
t.subscriber.fetch_objects += 1;
t.track(&m.track).fetch_objects += 1;
} else {
t.subscriber.objects_forwarded += 1;
t.subscriber.object_bytes_forwarded += m.payload_len;
let tr = t.track(&m.track);
tr.objects_forwarded += 1;
tr.object_bytes_forwarded += m.payload_len;
}
if let Some((j, class)) = m.signed {
let f = t.publisher_signed.entry(class.to_string()).or_default();
f.objects += 1;
f.joules_micro = f.joules_micro.saturating_add(j);
}
}
fn on_done(&mut self, d: Done) {
let (lane, send, result, meta, bytes, ended) = match d {
Done::SubgroupTimedOut { track } => {
if let Some(name) = track {
self.tally.lock().unwrap().track(&name).subgroups_reset_after_fin += 1;
}
return;
}
Done::Write { lane, send, result, meta, bytes, ended } => (lane, send, result, meta, bytes, ended),
};
let d = Written { lane, send, result, meta, bytes, ended };
self.in_flight = self.in_flight.saturating_sub(1);
self.settle(d.lane.sub, 1);
match d.result {
Ok(()) => {
if let Some(m) = &d.meta {
self.count_forwarded(m, d.bytes);
} else if d.bytes > 0 {
self.tally.lock().unwrap().subscriber.wire_bytes_sent += d.bytes;
}
if d.ended {
self.remove_lane(d.lane);
return;
}
if let Some(lane) = self.lanes.get_mut(&d.lane) {
lane.busy = false;
lane.send = d.send;
}
self.refresh(d.lane);
}
Err(TransportError::Deadline) => {
if let Some(m) = &d.meta {
self.count_dropped(m);
}
if let Some(lane) = self.lanes.get_mut(&d.lane) {
lane.busy = false;
lane.send = d.send;
}
self.kill(d.lane, StreamResetCode::DELIVERY_TIMEOUT.0);
}
Err(TransportError::Stopped(_)) | Err(TransportError::Reset(_)) | Err(TransportError::Other(_)) => {
// The subscriber stopped this stream: nothing more goes on it.
if let Some(lane) = self.lanes.get_mut(&d.lane) {
lane.busy = false;
lane.send = d.send;
}
self.kill(d.lane, StreamResetCode::CANCELLED.0);
}
Err(TransportError::Closed { .. }) => {
if let Some(lane) = self.lanes.get_mut(&d.lane) {
lane.busy = false;
lane.dead = true;
}
self.refresh(d.lane);
}
}
}
fn cancel(&mut self, sub: u64, code: u64) {
let keys: Vec<LaneKey> = self.lanes.keys().filter(|k| k.sub == sub).copied().collect();
for k in keys {
if let Some(lane) = self.lanes.get_mut(&k) {
lane.dead = true;
if let Some(s) = lane.send.as_mut() {
s.reset(code);
}
if !lane.busy {
self.remove_lane(k);
} else {
let n = lane.queue.len() as u64;
lane.queue.clear();
self.settle(sub, n);
}
}
}
self.sub(sub).cancelled = true;
self.settle(sub, 0);
}
fn reprioritise(&mut self, sub: u64, subscriber: u8) {
let keys: Vec<LaneKey> = self.lanes.keys().filter(|k| k.sub == sub).copied().collect();
for k in keys {
if let Some(lane) = self.lanes.get_mut(&k) {
for (_, item) in lane.queue.iter_mut() {
item.prio.subscriber = subscriber;
}
}
self.refresh(k);
}
}
fn drain(&mut self, sub: u64, tx: oneshot::Sender<u64>) {
let info = self.sub(sub);
if info.pending == 0 {
let _ = tx.send(info.streams_opened);
} else {
info.waiters.push(tx);
}
}
fn close(&mut self) {
let left: u64 = self.lanes.values().map(|l| l.queue.iter().filter(|(_, i)| i.meta.is_some()).count() as u64).sum();
self.tally.lock().unwrap().subscriber.undelivered_at_close += left;
for lane in self.lanes.values_mut() {
if let Some(s) = lane.send.as_mut() {
s.reset(StreamResetCode::SESSION_CLOSED.0);
}
}
self.lanes.clear();
for info in self.subs.values_mut() {
for w in info.waiters.drain(..) {
let _ = w.send(info.streams_opened);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_order_is_section_5_1_2s() {
let p = |s, pp, g| Prio { subscriber: s, publisher: pp, group_rank: g, fill_rank: 1, mode_rank: 1, within: 0 };
assert!(p(10, 200, 9) < p(11, 0, 0), "subscriber priority first");
assert!(p(10, 0, 9) < p(10, 1, 0), "then publisher priority");
assert!(p(10, 0, 1) < p(10, 0, 2), "then the group order");
assert!(Prio::group_rank(GroupOrder::Descending, 5) < Prio::group_rank(GroupOrder::Descending, 4));
let fill = Prio { fill_rank: 0, ..p(1, 1, 3) };
assert!(fill < p(1, 1, 3), "a fill before its subscription in one group");
let dg = Prio { mode_rank: 0, within: 9, ..p(1, 1, 3) };
assert!(dg < Prio { within: 0, ..p(1, 1, 3) }, "a datagram before a subgroup object");
assert!(p(0, 0, 0).transport_priority() > p(0, 1, 0).transport_priority());
assert!(p(0, 255, 0).transport_priority() > p(1, 0, 0).transport_priority());
}
}

179
joule-code/joule-code-rs/crates/joule-moq-relay/src/server.rs Normal file
View file

@ -0,0 +1,179 @@
//! The relay's listeners: one UDP endpoint for native QUIC and WebTransport, and one TCP
//! listener for the WebSocket fallback.
use crate::cert::RelayIdentity;
use crate::config::RelayConfig;
use crate::relay::Relay;
use crate::transport::mux::MuxConfig;
use crate::transport::quic::QuicConnection;
use crate::transport::webtransport::{select_protocol, WtConnection};
use crate::transport::websocket;
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use tokio::task::JoinHandle;
use tokio::time::timeout;
const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);
/// A running relay and its listeners.
pub struct RelayServer {
relay: Relay,
pub identity: RelayIdentity,
pub quic_addr: Option<SocketAddr>,
pub websocket_addr: Option<SocketAddr>,
endpoint: Option<quinn::Endpoint>,
quic_task: Option<JoinHandle<()>>,
websocket_task: Option<JoinHandle<()>>,
}
fn transport_config(cfg: &RelayConfig) -> quinn::TransportConfig {
let mut t = quinn::TransportConfig::default();
t.max_concurrent_bidi_streams(quinn::VarInt::from_u64(cfg.limits.max_requests).unwrap_or(quinn::VarInt::MAX));
t.max_concurrent_uni_streams(quinn::VarInt::from_u64(cfg.limits.max_uni_streams).unwrap_or(quinn::VarInt::MAX));
if let Ok(idle) = quinn::IdleTimeout::try_from(Duration::from_millis(cfg.limits.idle_timeout_ms)) {
t.max_idle_timeout(Some(idle));
}
t.keep_alive_interval(Some(Duration::from_millis((cfg.limits.idle_timeout_ms / 3).max(1_000))));
t.send_window(cfg.limits.quic_send_window_bytes.max(16 * 1024));
t
}
impl RelayServer {
/// Bind the configured listeners and start serving.
pub async fn start(cfg: RelayConfig) -> Result<Self, String> {
let identity = match (&cfg.tls.cert_path, &cfg.tls.key_path) {
(Some(c), Some(k)) => RelayIdentity::from_pem_files(c, k).await.map_err(|e| e.to_string())?,
(None, None) => RelayIdentity::dev_self_signed(&cfg.tls.dev_names).map_err(|e| e.to_string())?,
_ => return Err("tls: cert_path and key_path go together".into()),
};
let relay = Relay::new(cfg.clone())?;
let mut endpoint = None;
let mut quic_task = None;
let mut websocket_task = None;
let mut quic_addr = None;
if let Some(addr) = cfg.quic_listen {
let tls = identity.server_tls().map_err(|e| e.to_string())?;
let crypto = quinn::crypto::rustls::QuicServerConfig::try_from(tls).map_err(|e| e.to_string())?;
let mut sc = quinn::ServerConfig::with_crypto(Arc::new(crypto));
sc.transport_config(Arc::new(transport_config(&cfg)));
let ep = quinn::Endpoint::server(sc, addr).map_err(|e| format!("binding {addr}: {e}"))?;
quic_addr = Some(ep.local_addr().map_err(|e| e.to_string())?);
quic_task = Some(tokio::spawn(quic_accept(relay.clone(), ep.clone(), cfg.path.clone())));
endpoint = Some(ep);
}
let mut websocket_addr = None;
if let Some(addr) = cfg.websocket_listen {
let l = tokio::net::TcpListener::bind(addr).await.map_err(|e| format!("binding {addr}: {e}"))?;
websocket_addr = Some(l.local_addr().map_err(|e| e.to_string())?);
let mux = MuxConfig { max_bidi: cfg.limits.max_requests, max_uni: cfg.limits.max_uni_streams, max_buffered: cfg.limits.websocket_buffer_bytes, ..MuxConfig::default() };
websocket_task = Some(tokio::spawn(websocket_accept(relay.clone(), l, cfg.path.clone(), mux)));
}
Ok(RelayServer { relay, identity, quic_addr, websocket_addr, endpoint, quic_task, websocket_task })
}
pub fn relay(&self) -> &Relay {
&self.relay
}
/// Stop accepting, GOAWAY every session, wait up to the configured GOAWAY timeout,
/// close what is left with GOAWAY_TIMEOUT. Returns how many sessions that closed.
pub async fn shutdown(self) -> usize {
if let Some(ep) = &self.endpoint {
ep.set_server_config(None);
}
// The WebSocket listener stops taking upgrades; the sessions it made go on.
if let Some(t) = &self.websocket_task {
t.abort();
}
let t = Duration::from_millis(self.relay.config().limits.goaway_timeout_ms);
let closed = self.relay.shutdown(t).await;
if let Some(ep) = &self.endpoint {
ep.close(quinn::VarInt::from_u32(0), b"the relay stopped");
let _ = timeout(Duration::from_secs(5), ep.wait_idle()).await;
}
if let Some(t) = &self.quic_task {
t.abort();
}
closed
}
}
async fn quic_accept(relay: Relay, ep: quinn::Endpoint, path: String) {
while let Some(incoming) = ep.accept().await {
let (relay, path) = (relay.clone(), path.clone());
tokio::spawn(async move {
let Ok(mut connecting) = incoming.accept() else { return };
let alpn = match timeout(HANDSHAKE_TIMEOUT, connecting.handshake_data()).await {
Ok(Ok(hd)) => hd.downcast::<quinn::crypto::rustls::HandshakeData>().ok().and_then(|h| h.protocol.clone()),
_ => return,
};
match alpn.as_deref() {
Some(a) if a == joule_moq::draft22::ALPN => {
if let Ok(Ok(c)) = timeout(HANDSHAKE_TIMEOUT, connecting).await {
crate::session::serve(relay, Arc::new(QuicConnection::new(c))).await;
}
}
Some(b"h3") => {
let req = match timeout(HANDSHAKE_TIMEOUT, wtransport::endpoint::IncomingSessionFuture::with_quic_connecting(connecting)).await {
Ok(Ok(r)) => r,
_ => return,
};
if req.path().split('?').next() != Some(path.as_str()) {
req.not_found().await;
return;
}
// -22 §6.2.1: the draft is chosen by WT-Available-Protocols. A client that
// sends the header must offer moqt-22; one that sends none is taken to
// speak it.
let offered = req.headers().get("wt-available-protocols").map(String::as_str);
let chosen = match (offered, select_protocol(offered)) {
(None, _) => None,
(Some(_), Some(p)) => Some(p),
(Some(_), None) => {
req.forbidden().await;
return;
}
};
let headers: Vec<(String, String)> = chosen.map(|p| vec![("wt-protocol".to_string(), format!("\"{p}\""))]).unwrap_or_default();
if let Ok(Ok(conn)) = timeout(HANDSHAKE_TIMEOUT, req.accept_with_headers(headers)).await {
crate::session::serve(relay, Arc::new(WtConnection::new(conn))).await;
}
}
_ => {}
}
});
}
}
async fn websocket_accept(relay: Relay, l: tokio::net::TcpListener, path: String, mux: MuxConfig) {
use tokio_tungstenite::tungstenite::handshake::server::{ErrorResponse, Request, Response};
use tokio_tungstenite::tungstenite::http;
loop {
let Ok((tcp, peer)) = l.accept().await else {
tokio::time::sleep(Duration::from_millis(50)).await;
continue;
};
let _ = tcp.set_nodelay(true);
let (relay, path) = (relay.clone(), path.clone());
tokio::spawn(async move {
// The handshake library's callback type; its error is the refusal response.
#[allow(clippy::result_large_err)]
let check = move |req: &Request, mut resp: Response| -> Result<Response, ErrorResponse> {
let refuse = |status: u16, why: &str| http::Response::builder().status(status).body(Some(why.to_string())).expect("a response");
if req.uri().path() != path {
return Err(refuse(404, "no MOQT endpoint at this path"));
}
let offered = req.headers().get("sec-websocket-protocol").and_then(|v| v.to_str().ok());
if !websocket::offers_subprotocol(offered) {
return Err(refuse(400, "offer the moqt-22 subprotocol"));
}
resp.headers_mut().insert("sec-websocket-protocol", http::HeaderValue::from_static("moqt-22"));
Ok(resp)
};
if let Ok(Ok(ws)) = timeout(HANDSHAKE_TIMEOUT, tokio_tungstenite::accept_hdr_async(tcp, check)).await {
crate::session::serve(relay, Arc::new(websocket::session(ws, false, peer.to_string(), mux))).await;
}
});
}
}

1421
joule-code/joule-code-rs/crates/joule-moq-relay/src/session.rs Normal file
View file

File diff suppressed because it is too large Load diff

198
joule-code/joule-code-rs/crates/joule-moq-relay/src/summary.rs Normal file
View file

@ -0,0 +1,198 @@
//! What the relay did for each session, counted as it happens and signed when the
//! session ends.
//!
//! The summary is relay-scoped telemetry (WAI `jwp-receipts.md` §4.2: "local to the relay
//! and MUST NOT be conflated with publisher-issued receipts"), signed by the relay's own
//! key. It counts bytes and objects. It states energy only as its acquisition class
//! allows: no meter backs this relay, so its own figure is the unmetered marker, 0 with
//! the class `unmetered`, which a reader must not present as zero joules (WAI
//! `energy-measurement.md` §2). Where the door verified an object's receipt, the
//! publisher's signed figure for producing that object is summed under the class it was
//! signed with, and labelled as the publisher's: it is not the relay's energy.
use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey};
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
/// The format name a summary carries.
pub const FORMAT: &str = "joule-moq-relay/session-summary/1";
/// The domain the relay's signature covers, before the summary's bytes.
pub const SIGNING_DOMAIN: &[u8] = b"joule-moq-relay:session-summary\x01";
/// One track as one downstream session received it.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct TrackTally {
pub objects_forwarded: u64,
pub object_bytes_forwarded: u64,
/// Objects not sent because their delivery timeout passed at this hop (§5.2), the
/// rest of a subgroup reset for that reason among them.
pub dropped_delivery_timeout: u64,
/// Of those, the ones on a track whose receipts track this session also subscribed
/// to: their receipts travel there, with no delivery timeout.
pub dropped_receipts_track_subscribed: u64,
/// Subgroups reset because SUBGROUP_DELIVERY_TIMEOUT passed before all their data
/// was acknowledged.
pub subgroups_reset_after_fin: u64,
/// Objects served on a fetch or fill stream.
pub fetch_objects: u64,
}
/// What a session received as a subscriber.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct SubscriberTally {
pub objects_forwarded: u64,
pub object_bytes_forwarded: u64,
/// Every byte written to the session's data streams and datagrams.
pub wire_bytes_sent: u64,
pub dropped_delivery_timeout: u64,
pub dropped_receipts_track_subscribed: u64,
pub fetch_objects: u64,
/// Objects still queued when the session ended.
pub undelivered_at_close: u64,
pub tracks: BTreeMap<String, TrackTally>,
}
/// What a session sent as a publisher.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct PublisherTally {
pub objects_received: u64,
pub object_bytes_received: u64,
/// Objects the door refused (WAI jwp-receipts §4.1), with the groups it dropped.
pub refused_at_door: u64,
/// Objects already received from another publisher of the track (§7.3).
pub deduplicated: u64,
}
/// Publisher-signed production figures of objects delivered to the session, summed
/// under the class their leaves were signed with. Only objects whose receipts the door
/// verified are counted.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedFigure {
pub objects: u64,
pub joules_micro: u64,
}
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct Tally {
pub subscriber: SubscriberTally,
pub publisher: PublisherTally,
/// Keyed by acquisition class label (`unmetered` for the marker).
pub publisher_signed: BTreeMap<String, SignedFigure>,
}
impl Tally {
pub fn track(&mut self, name: &str) -> &mut TrackTally {
self.subscriber.tracks.entry(name.to_string()).or_default()
}
}
/// The relay's own energy for the session, as its acquisition class allows.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RelayEnergy {
/// `unmetered`: no meter backs the figure.
pub acquisition: String,
/// 0 is the unmetered marker, never a measurement of zero joules.
pub joules_micro: u64,
}
impl RelayEnergy {
pub fn unmetered() -> Self {
RelayEnergy { acquisition: "unmetered".into(), joules_micro: 0 }
}
}
/// One session's summary, as the relay signs it.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct SessionSummary {
pub format: String,
/// The relay's Ed25519 public key, hex.
pub relay_key: String,
pub session: u64,
pub transport: String,
pub remote: String,
pub draft: String,
pub opened_unix_ms: u64,
pub closed_unix_ms: u64,
/// The session termination code it closed with (§12.2), when known.
pub close_code: Option<u64>,
pub relay_energy: RelayEnergy,
pub tally: Tally,
}
/// A summary and the relay's signature over [`SIGNING_DOMAIN`] and its JSON bytes.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedSessionSummary {
pub summary: SessionSummary,
/// Ed25519, hex.
pub signature: String,
}
fn signed_bytes(s: &SessionSummary) -> Vec<u8> {
let mut b = SIGNING_DOMAIN.to_vec();
b.extend_from_slice(&serde_json::to_vec(s).expect("a summary serialises"));
b
}
impl SignedSessionSummary {
pub fn sign(summary: SessionSummary, key: &SigningKey) -> Self {
let sig = key.sign(&signed_bytes(&summary));
SignedSessionSummary { summary, signature: hex::encode(sig.to_bytes()) }
}
/// Check the signature under the key the summary names.
pub fn verify(&self) -> Result<(), String> {
let k: [u8; 32] = hex::decode(&self.summary.relay_key).map_err(|e| e.to_string())?.try_into().map_err(|_| "a relay key of 32 bytes".to_string())?;
let key = VerifyingKey::from_bytes(&k).map_err(|e| e.to_string())?;
let s: [u8; 64] = hex::decode(&self.signature).map_err(|e| e.to_string())?.try_into().map_err(|_| "a signature of 64 bytes".to_string())?;
key.verify(&signed_bytes(&self.summary), &Signature::from_bytes(&s)).map_err(|e| e.to_string())
}
}
/// The class label of a receipt leaf's class bytes (WAI energy-measurement §2.7 tags).
pub fn class_label(class: Option<&[u8]>) -> &'static str {
match class.and_then(|c| c.first()) {
None => "unlabelled",
Some(0) => "unmetered",
Some(1) => "HwShunt",
Some(2) => "ModelBased",
Some(3) => "Estimator",
Some(4) => "OnChipCounter",
Some(5) => "CalibratedInstrument",
Some(_) => "unknown",
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_summary_verifies_and_a_change_does_not() {
let key = SigningKey::from_bytes(&[3; 32]);
let mut tally = Tally::default();
tally.subscriber.objects_forwarded = 4;
tally.track("a/b").dropped_delivery_timeout = 1;
let s = SessionSummary {
format: FORMAT.into(),
relay_key: hex::encode(key.verifying_key().to_bytes()),
session: 7,
transport: "quic".into(),
remote: "127.0.0.1:1".into(),
draft: "moqt-22".into(),
opened_unix_ms: 1,
closed_unix_ms: 2,
close_code: Some(0),
relay_energy: RelayEnergy::unmetered(),
tally,
};
let signed = SignedSessionSummary::sign(s, &key);
signed.verify().unwrap();
let json = serde_json::to_string(&signed).unwrap();
let back: SignedSessionSummary = serde_json::from_str(&json).unwrap();
back.verify().unwrap();
let mut forged = back.clone();
forged.summary.tally.subscriber.objects_forwarded = 5;
assert!(forged.verify().is_err());
assert_eq!(back.summary.relay_energy.acquisition, "unmetered");
}
}

272
joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/capsule.rs Normal file
View file

@ -0,0 +1,272 @@
//! The capsules the WebSocket binding carries: those of WebTransport over HTTP/2
//! (`draft-ietf-webtrans-http2-15` §6), one capsule to each WebSocket binary message.
//!
//! A capsule is `Type (i) Length (i) Value`, with QUIC's variable-length integers
//! (RFC 9000 §16), not MOQT's: the integers here frame the binding, and the MOQT bytes a
//! stream carries ride inside `WT_STREAM` unchanged.
/// Capsule types (webtrans-http2 §6, webtrans-http3 §5 and §4.6).
pub mod ty {
pub const DATAGRAM: u64 = 0x00;
pub const CLOSE_SESSION: u64 = 0x2843;
pub const DRAIN_SESSION: u64 = 0x78ae;
pub const PADDING: u64 = 0x190b_4d38;
pub const RESET_STREAM: u64 = 0x190b_4d39;
pub const STOP_SENDING: u64 = 0x190b_4d3a;
/// WT_STREAM with the FIN bit.
pub const STREAM_FIN: u64 = 0x190b_4d3b;
/// WT_STREAM without it.
pub const STREAM: u64 = 0x190b_4d3c;
pub const MAX_DATA: u64 = 0x190b_4d3d;
pub const MAX_STREAM_DATA: u64 = 0x190b_4d3e;
pub const MAX_STREAMS_BIDI: u64 = 0x190b_4d3f;
pub const MAX_STREAMS_UNI: u64 = 0x190b_4d40;
pub const DATA_BLOCKED: u64 = 0x190b_4d41;
pub const STREAM_DATA_BLOCKED: u64 = 0x190b_4d42;
pub const STREAMS_BLOCKED_BIDI: u64 = 0x190b_4d43;
pub const STREAMS_BLOCKED_UNI: u64 = 0x190b_4d44;
}
/// The largest QUIC varint, 2^62-1.
pub const QUIC_VARINT_MAX: u64 = (1 << 62) - 1;
/// The largest Maximum Streams a WT_MAX_STREAMS may carry (webtrans-http2 §6.7).
pub const MAX_STREAMS_LIMIT: u64 = 1 << 60;
/// The longest close message (webtrans-http2 §6.12).
pub const MAX_CLOSE_MESSAGE: usize = 1024;
/// Append a QUIC varint; values above 2^62-1 are clamped, which no caller here sends.
pub fn put_qvarint(out: &mut Vec<u8>, v: u64) {
let v = v.min(QUIC_VARINT_MAX);
if v < 1 << 6 {
out.push(v as u8);
} else if v < 1 << 14 {
out.extend_from_slice(&((v as u16) | 0x4000).to_be_bytes());
} else if v < 1 << 30 {
out.extend_from_slice(&((v as u32) | 0x8000_0000).to_be_bytes());
} else {
out.extend_from_slice(&(v | 0xc000_0000_0000_0000).to_be_bytes());
}
}
/// Read a QUIC varint from the front of `b`: the value and its length.
pub fn get_qvarint(b: &[u8]) -> Option<(u64, usize)> {
let first = *b.first()?;
let len = 1usize << (first >> 6);
if b.len() < len {
return None;
}
let mut v = (first & 0x3f) as u64;
for x in &b[1..len] {
v = (v << 8) | *x as u64;
}
Some((v, len))
}
/// One capsule of the binding.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Capsule {
Stream { id: u64, data: Vec<u8>, fin: bool },
ResetStream { id: u64, code: u64, reliable_size: u64 },
StopSending { id: u64, code: u64 },
MaxStreams { bidi: bool, max: u64 },
StreamsBlocked { bidi: bool, max: u64 },
Datagram(Vec<u8>),
CloseSession { code: u32, message: Vec<u8> },
DrainSession,
/// A capsule the binding does not act on (PADDING, the data flow-control capsules,
/// or a type it does not know): HTTP capsules of unknown type are skipped.
Ignored(u64),
}
/// Why a message is not a capsule of the binding.
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum CapsuleError {
#[error("the message is not one whole capsule")]
Framing,
#[error("a {0} capsule whose value does not parse")]
Value(&'static str),
}
struct Cursor<'a>(&'a [u8]);
impl<'a> Cursor<'a> {
fn v(&mut self, what: &'static str) -> Result<u64, CapsuleError> {
let (v, n) = get_qvarint(self.0).ok_or(CapsuleError::Value(what))?;
self.0 = &self.0[n..];
Ok(v)
}
fn end(&self, what: &'static str) -> Result<(), CapsuleError> {
if self.0.is_empty() { Ok(()) } else { Err(CapsuleError::Value(what)) }
}
}
impl Capsule {
pub fn encode(&self) -> Vec<u8> {
let mut v = Vec::new();
let t = match self {
Capsule::Stream { id, data, fin } => {
put_qvarint(&mut v, *id);
v.extend_from_slice(data);
if *fin { ty::STREAM_FIN } else { ty::STREAM }
}
Capsule::ResetStream { id, code, reliable_size } => {
put_qvarint(&mut v, *id);
put_qvarint(&mut v, *code);
put_qvarint(&mut v, *reliable_size);
ty::RESET_STREAM
}
Capsule::StopSending { id, code } => {
put_qvarint(&mut v, *id);
put_qvarint(&mut v, *code);
ty::STOP_SENDING
}
Capsule::MaxStreams { bidi, max } => {
put_qvarint(&mut v, *max);
if *bidi { ty::MAX_STREAMS_BIDI } else { ty::MAX_STREAMS_UNI }
}
Capsule::StreamsBlocked { bidi, max } => {
put_qvarint(&mut v, *max);
if *bidi { ty::STREAMS_BLOCKED_BIDI } else { ty::STREAMS_BLOCKED_UNI }
}
Capsule::Datagram(d) => {
v.extend_from_slice(d);
ty::DATAGRAM
}
Capsule::CloseSession { code, message } => {
v.extend_from_slice(&code.to_be_bytes());
v.extend_from_slice(&message[..message.len().min(MAX_CLOSE_MESSAGE)]);
ty::CLOSE_SESSION
}
Capsule::DrainSession => ty::DRAIN_SESSION,
Capsule::Ignored(t) => *t,
};
let mut out = Vec::with_capacity(v.len() + 16);
put_qvarint(&mut out, t);
put_qvarint(&mut out, v.len() as u64);
out.extend_from_slice(&v);
out
}
/// Read a message that holds exactly one capsule.
pub fn decode(msg: &[u8]) -> Result<Capsule, CapsuleError> {
let mut c = Cursor(msg);
let t = c.v("type").map_err(|_| CapsuleError::Framing)?;
let len = c.v("length").map_err(|_| CapsuleError::Framing)?;
if c.0.len() as u64 != len {
return Err(CapsuleError::Framing);
}
let mut c = Cursor(c.0);
Ok(match t {
ty::STREAM | ty::STREAM_FIN => {
let id = c.v("WT_STREAM")?;
Capsule::Stream { id, data: c.0.to_vec(), fin: t == ty::STREAM_FIN }
}
ty::RESET_STREAM => {
let id = c.v("WT_RESET_STREAM")?;
let code = c.v("WT_RESET_STREAM")?;
let reliable_size = c.v("WT_RESET_STREAM")?;
c.end("WT_RESET_STREAM")?;
Capsule::ResetStream { id, code, reliable_size }
}
ty::STOP_SENDING => {
let id = c.v("WT_STOP_SENDING")?;
let code = c.v("WT_STOP_SENDING")?;
c.end("WT_STOP_SENDING")?;
Capsule::StopSending { id, code }
}
ty::MAX_STREAMS_BIDI | ty::MAX_STREAMS_UNI => {
let max = c.v("WT_MAX_STREAMS")?;
c.end("WT_MAX_STREAMS")?;
Capsule::MaxStreams { bidi: t == ty::MAX_STREAMS_BIDI, max }
}
ty::STREAMS_BLOCKED_BIDI | ty::STREAMS_BLOCKED_UNI => {
let max = c.v("WT_STREAMS_BLOCKED")?;
c.end("WT_STREAMS_BLOCKED")?;
Capsule::StreamsBlocked { bidi: t == ty::STREAMS_BLOCKED_BIDI, max }
}
ty::DATAGRAM => Capsule::Datagram(c.0.to_vec()),
ty::CLOSE_SESSION => {
if c.0.len() < 4 || c.0.len() > 4 + MAX_CLOSE_MESSAGE {
return Err(CapsuleError::Value("WT_CLOSE_SESSION"));
}
let code = u32::from_be_bytes([c.0[0], c.0[1], c.0[2], c.0[3]]);
let message = c.0[4..].to_vec();
if std::str::from_utf8(&message).is_err() {
return Err(CapsuleError::Value("WT_CLOSE_SESSION"));
}
Capsule::CloseSession { code, message }
}
ty::DRAIN_SESSION => {
c.end("WT_DRAIN_SESSION")?;
Capsule::DrainSession
}
other => Capsule::Ignored(other),
})
}
}
/// Stream IDs as QUIC numbers them (RFC 9000 §2.1; webtrans-http2 §5.2): bit 0 the
/// initiator (0 the client), bit 1 the direction (0 bidirectional).
pub mod stream_id {
pub fn client_initiated(id: u64) -> bool {
id & 1 == 0
}
pub fn bidi(id: u64) -> bool {
id & 2 == 0
}
/// The `n`th stream (from 0) of a kind.
pub fn nth(n: u64, client: bool, bidi: bool) -> u64 {
(n << 2) | (!client as u64) | ((!bidi as u64) << 1)
}
/// Which stream of its kind `id` is (from 0).
pub fn index(id: u64) -> u64 {
id >> 2
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn quic_varints() {
for v in [0u64, 63, 64, 16383, 16384, (1 << 30) - 1, 1 << 30, QUIC_VARINT_MAX] {
let mut b = Vec::new();
put_qvarint(&mut b, v);
assert_eq!(get_qvarint(&b), Some((v, b.len())), "{v}");
}
assert_eq!(get_qvarint(&[0x40]), None);
}
#[test]
fn every_capsule_round_trips() {
for c in [
Capsule::Stream { id: 0, data: b"moqt".to_vec(), fin: false },
Capsule::Stream { id: 7, data: vec![], fin: true },
Capsule::ResetStream { id: 3, code: 2, reliable_size: 0 },
Capsule::StopSending { id: 4, code: 1 },
Capsule::MaxStreams { bidi: true, max: 100 },
Capsule::MaxStreams { bidi: false, max: 1 << 20 },
Capsule::StreamsBlocked { bidi: true, max: 5 },
Capsule::Datagram(vec![1, 2, 3]),
Capsule::CloseSession { code: 0x10, message: b"going away".to_vec() },
Capsule::DrainSession,
] {
assert_eq!(Capsule::decode(&c.encode()).unwrap(), c);
}
assert_eq!(Capsule::decode(&Capsule::Ignored(ty::PADDING).encode()).unwrap(), Capsule::Ignored(ty::PADDING));
let mut b = Capsule::Datagram(vec![1]).encode();
b.push(0);
assert_eq!(Capsule::decode(&b), Err(CapsuleError::Framing));
}
#[test]
fn stream_ids() {
assert_eq!(stream_id::nth(0, true, true), 0);
assert_eq!(stream_id::nth(0, false, true), 1);
assert_eq!(stream_id::nth(0, true, false), 2);
assert_eq!(stream_id::nth(3, false, false), 15);
assert!(stream_id::client_initiated(4) && stream_id::bidi(4) && !stream_id::bidi(6));
assert_eq!(stream_id::index(15), 3);
}
}

131
joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mod.rs Normal file
View file

@ -0,0 +1,131 @@
//! The transports a MOQT session runs on, behind one interface.
//!
//! MOQT needs streams, datagrams and a way to close the session
//! (`draft-ietf-moq-transport-22` §6.2: "Both provide streams and datagrams with
//! similar semantics"). [`Connection`], [`SendStream`] and [`RecvStream`] are that
//! surface, and the relay is written once against them. There are four
//! implementations:
//!
//! - [`quic`]: native QUIC on quinn, ALPN `moqt-22` (§6.2.2);
//! - [`webtransport`]: WebTransport over HTTP/3 (`draft-ietf-webtrans-http3-16`) on
//! wtransport, negotiated by `WT-Available-Protocols` (§6.2.1);
//! - [`websocket`]: the fallback for paths that carry no UDP. Each WebSocket binary
//! message is one capsule of `draft-ietf-webtrans-http2-15` §6 ([`capsule`]), so a
//! WebSocket carries what a WebTransport session over HTTP/2 carries, multiplexed by
//! [`mux`];
//! - [`mux::memory_pair`]: the same multiplexer over an in-process pipe, for tests and
//! for embedding.
pub mod capsule;
pub mod mux;
pub mod quic;
pub mod webtransport;
pub mod websocket;
use async_trait::async_trait;
use bytes::Bytes;
use std::sync::Arc;
/// Which transport a session runs on.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum TransportKind {
Quic,
WebTransport,
WebSocket,
Memory,
}
impl TransportKind {
pub fn label(self) -> &'static str {
match self {
TransportKind::Quic => "quic",
TransportKind::WebTransport => "webtransport",
TransportKind::WebSocket => "websocket",
TransportKind::Memory => "memory",
}
}
}
/// Why a transport operation did not complete.
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum TransportError {
/// The session is closed. `code` is the application error code it closed with
/// (MOQT's session termination codes, §12.2) when one is known.
#[error("the session closed (code {code:#x}, by the peer: {by_peer}): {reason}")]
Closed { code: u64, reason: String, by_peer: bool },
/// The peer reset the stream this side reads.
#[error("the peer reset the stream with code {0:#x}")]
Reset(u64),
/// The peer asked this side to stop sending on the stream.
#[error("the peer stopped the stream with code {0:#x}")]
Stopped(u64),
/// A write did not finish before its object's delivery deadline.
#[error("the delivery deadline passed during the write")]
Deadline,
#[error("{0}")]
Other(String),
}
impl TransportError {
pub fn other(e: impl std::fmt::Display) -> Self {
TransportError::Other(e.to_string())
}
}
pub type TResult<T> = Result<T, TransportError>;
/// The sending half of a stream.
#[async_trait]
pub trait SendStream: Send + Sync {
/// Write all of `buf`, waiting for flow control as the transport requires.
async fn write_all(&mut self, buf: &[u8]) -> TResult<()>;
/// End the stream gracefully (a FIN).
fn finish(&mut self) -> TResult<()>;
/// End the stream abruptly with an application code (RESET_STREAM).
fn reset(&mut self, code: u64);
/// The transport's priority for this stream's buffered bytes: higher first.
fn set_priority(&mut self, priority: i32);
/// After [`finish`](Self::finish): `Ok(None)` once the peer has acknowledged every
/// byte ("all data committed", the point SUBGROUP_DELIVERY_TIMEOUT waits for,
/// §5.2), `Ok(Some(code))` when the peer stopped the stream first.
async fn delivered(&mut self) -> TResult<Option<u64>>;
}
/// The receiving half of a stream.
#[async_trait]
pub trait RecvStream: Send + Sync {
/// The next bytes, `None` at the FIN, [`TransportError::Reset`] when the peer reset it.
async fn read_chunk(&mut self) -> TResult<Option<Bytes>>;
/// Ask the peer to stop sending (STOP_SENDING).
fn stop(&mut self, code: u64);
}
pub type BoxSend = Box<dyn SendStream>;
pub type BoxRecv = Box<dyn RecvStream>;
/// One transport session: streams both ways, datagrams, and its close.
#[async_trait]
pub trait Connection: Send + Sync {
fn kind(&self) -> TransportKind;
/// The peer, for logs and the session summary.
fn remote(&self) -> String;
async fn open_uni(&self) -> TResult<BoxSend>;
async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)>;
async fn accept_uni(&self) -> TResult<BoxRecv>;
async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)>;
/// Send one datagram. Unreliable on QUIC and WebTransport over HTTP/3; reliable and
/// ordered on the WebSocket binding, as on WebTransport over HTTP/2.
fn send_datagram(&self, payload: Bytes) -> TResult<()>;
async fn recv_datagram(&self) -> TResult<Bytes>;
/// Close the session with a MOQT session termination code (§6.6, §12.2).
fn close(&self, code: u64, reason: &[u8]);
/// Resolves when the session is closed, with why.
async fn closed(&self) -> TransportError;
/// The transport's own counters, for diagnosis, where it keeps any.
fn diagnostics(&self) -> Option<String> {
None
}
}
pub type SharedConnection = Arc<dyn Connection>;

818
joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mux.rs Normal file
View file

@ -0,0 +1,818 @@
//! Streams and datagrams multiplexed over a pipe of whole messages, each message one
//! capsule of `draft-ietf-webtrans-http2-15` §6 ([`super::capsule`]).
//!
//! This is what a WebTransport session over HTTP/2 does over its CONNECT stream, put on
//! a pipe that already frames messages: a WebSocket ([`super::websocket`]), or an
//! in-process channel ([`memory_pair`]). What it keeps of webtrans-http2:
//!
//! - stream IDs numbered as QUIC numbers them, opened implicitly by their first
//! `WT_STREAM` (§5.2, §6.4), with the FIN bit in the capsule type;
//! - `WT_RESET_STREAM` and `WT_STOP_SENDING` (§6.2, §6.3);
//! - stream-count flow control by `WT_MAX_STREAMS` (§6.7): each end sends its limits
//! first, and until it has, the other opens nothing. A stream's credit returns when
//! both of its halves are done, and is granted back in batches. This bounds the
//! requests in flight, which MOQT -22 leaves to the transport's stream limits;
//! - `DATAGRAM` capsules, reliable and ordered as on HTTP/2 (§6.11);
//! - `WT_CLOSE_SESSION` with a 32-bit code (§6.12), and `WT_DRAIN_SESSION` (§6.13).
//!
//! What it does not do: the data flow-control capsules (`WT_MAX_DATA`,
//! `WT_MAX_STREAM_DATA` and their BLOCKED forms). The pipe's own backpressure (TCP under
//! a WebSocket) bounds the sender, and the receiver bounds what it buffers: past
//! `max_buffered` bytes not yet read it closes the session. Capsules of those types are
//! read and ignored.
use super::capsule::{stream_id, Capsule};
use super::{BoxRecv, BoxSend, Connection, RecvStream, SendStream, TResult, TransportError, TransportKind};
use async_trait::async_trait;
use bytes::Bytes;
use std::collections::{BTreeSet, HashMap};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use tokio::sync::{mpsc, watch};
/// The sending end of a message pipe.
#[async_trait]
pub trait PipeTx: Send + 'static {
async fn send(&mut self, msg: Vec<u8>) -> TResult<()>;
async fn close(&mut self);
}
/// The receiving end of a message pipe: `None` once it is closed.
#[async_trait]
pub trait PipeRx: Send + 'static {
async fn recv(&mut self) -> TResult<Option<Vec<u8>>>;
}
/// Limits of one multiplexed session.
#[derive(Clone, Copy, Debug)]
pub struct MuxConfig {
/// Bidirectional streams the peer may have open at once (each a MOQT request).
pub max_bidi: u64,
/// Unidirectional streams the peer may have open at once.
pub max_uni: u64,
/// Bytes received and not yet read, across all streams, before the session closes.
pub max_buffered: usize,
/// Datagrams received and not yet read; more are dropped.
pub datagram_queue: usize,
/// Bytes queued for the pipe before a writer waits.
pub send_high_water: usize,
}
impl Default for MuxConfig {
fn default() -> Self {
MuxConfig { max_bidi: 256, max_uni: 1024, max_buffered: 64 << 20, datagram_queue: 1024, send_high_water: 1 << 20 }
}
}
/// The largest stream data one capsule carries.
const CHUNK: usize = 16 * 1024;
/// The session code a receiver closes with when its buffer bound is passed: the value of
/// WT_FLOW_CONTROL_ERROR (webtrans-http3 §9.4), as a 32-bit application code.
pub const FLOW_CONTROL_ERROR: u64 = 0x045d_4487;
/// How far ahead of the lowest unseen one the peer's stream IDs may arrive.
const SEEN_WINDOW: usize = 1 << 16;
enum RecvEvent {
Data(Bytes),
Fin,
Reset(u64),
}
#[derive(Default)]
struct SendShared {
stopped: Mutex<Option<u64>>,
}
/// The peer's stream indices of one kind: which are new.
#[derive(Default)]
struct Seen {
floor: u64,
above: BTreeSet<u64>,
}
impl Seen {
/// Admit index `i`: `true` when it was not seen before.
fn admit(&mut self, i: u64) -> Option<bool> {
if i < self.floor || self.above.contains(&i) {
return Some(false);
}
if self.above.len() >= SEEN_WINDOW {
return None;
}
self.above.insert(i);
while self.above.remove(&self.floor) {
self.floor += 1;
}
Some(true)
}
}
struct State {
next_bidi: u64,
next_uni: u64,
peer_max_bidi: u64,
peer_max_uni: u64,
local_max_bidi: u64,
local_max_uni: u64,
pending_bidi: u64,
pending_uni: u64,
seen_bidi: Seen,
seen_uni: Seen,
recv: HashMap<u64, mpsc::UnboundedSender<RecvEvent>>,
send: HashMap<u64, Arc<SendShared>>,
accept_uni: Option<mpsc::UnboundedSender<BoxRecv>>,
accept_bi: Option<mpsc::UnboundedSender<(BoxSend, BoxRecv)>>,
datagrams: Option<mpsc::Sender<Bytes>>,
blocked_sent: (u64, u64),
}
struct Inner {
client: bool,
kind: TransportKind,
remote: String,
cfg: MuxConfig,
out: mpsc::UnboundedSender<Vec<u8>>,
queued: AtomicUsize,
space: watch::Sender<u64>,
state: Mutex<State>,
/// Bumped whenever stream credit changes or the session closes.
generation: watch::Sender<u64>,
closed: watch::Sender<Option<TransportError>>,
buffered: AtomicUsize,
accept_uni_rx: tokio::sync::Mutex<mpsc::UnboundedReceiver<BoxRecv>>,
accept_bi_rx: tokio::sync::Mutex<mpsc::UnboundedReceiver<(BoxSend, BoxRecv)>>,
datagram_rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
}
/// A stream's credit: returned to the peer when the last half holding it is dropped.
struct Credit {
inner: std::sync::Weak<Inner>,
bidi: bool,
}
impl Drop for Credit {
fn drop(&mut self) {
if let Some(inner) = self.inner.upgrade() {
inner.return_credit(self.bidi);
}
}
}
impl Inner {
fn is_closed(&self) -> Option<TransportError> {
self.closed.borrow().clone()
}
fn closed_err(&self) -> TransportError {
self.is_closed().unwrap_or(TransportError::Closed { code: 0, reason: "the session closed".into(), by_peer: false })
}
fn push(&self, c: &Capsule) {
let b = c.encode();
self.queued.fetch_add(b.len(), Ordering::AcqRel);
let _ = self.out.send(b);
}
fn bump(&self) {
self.generation.send_modify(|g| *g += 1);
}
/// Mark the session closed with `why` (the first reason wins) and wake everyone.
fn set_closed(&self, why: TransportError) {
let mut first = false;
self.closed.send_if_modified(|c| {
if c.is_none() {
*c = Some(why.clone());
first = true;
true
} else {
false
}
});
if first {
let dropped = {
let mut st = self.state.lock().unwrap();
st.accept_uni = None;
st.accept_bi = None;
st.datagrams = None;
std::mem::take(&mut st.recv)
};
drop(dropped);
// Streams nobody accepted hold this session; drop them, unlocked, so it can go.
if let Ok(mut rx) = self.accept_uni_rx.try_lock() {
while rx.try_recv().is_ok() {}
}
if let Ok(mut rx) = self.accept_bi_rx.try_lock() {
while rx.try_recv().is_ok() {}
}
}
self.bump();
}
fn return_credit(&self, bidi: bool) {
let grant = {
let mut guard = self.state.lock().unwrap();
let st = &mut *guard;
let (pending, max, initial) = if bidi {
(&mut st.pending_bidi, &mut st.local_max_bidi, self.cfg.max_bidi)
} else {
(&mut st.pending_uni, &mut st.local_max_uni, self.cfg.max_uni)
};
*pending += 1;
if *pending >= (initial / 8).max(1) {
*max += *pending;
*pending = 0;
Some(*max)
} else {
None
}
};
if let Some(max) = grant
&& self.is_closed().is_none()
{
self.push(&Capsule::MaxStreams { bidi, max });
}
}
fn close_with(&self, code: u64, reason: &[u8]) {
if self.is_closed().is_some() {
return;
}
let code32 = u32::try_from(code).unwrap_or(u32::MAX);
let mut msg = reason.to_vec();
msg.truncate(super::capsule::MAX_CLOSE_MESSAGE);
while std::str::from_utf8(&msg).is_err() {
msg.pop();
}
self.push(&Capsule::CloseSession { code: code32, message: msg });
self.set_closed(TransportError::Closed { code, reason: String::from_utf8_lossy(reason).into_owned(), by_peer: false });
}
/// One capsule from the peer.
fn on_capsule(self: &Arc<Self>, c: Capsule) -> Result<(), (u64, &'static str)> {
match c {
Capsule::Stream { id, data, fin } => self.on_stream(id, Some(data), fin, None),
Capsule::ResetStream { id, code, .. } => self.on_stream(id, None, false, Some(code)),
Capsule::StopSending { id, code } => {
let st = self.state.lock().unwrap();
if let Some(s) = st.send.get(&id) {
*s.stopped.lock().unwrap() = Some(code);
}
Ok(())
}
Capsule::MaxStreams { bidi, max } => {
if max > super::capsule::MAX_STREAMS_LIMIT {
return Err((FLOW_CONTROL_ERROR, "a WT_MAX_STREAMS above 2^60"));
}
let mut st = self.state.lock().unwrap();
let cur = if bidi { &mut st.peer_max_bidi } else { &mut st.peer_max_uni };
if max < *cur {
return Err((FLOW_CONTROL_ERROR, "a WT_MAX_STREAMS that lowers the limit"));
}
*cur = max;
drop(st);
self.bump();
Ok(())
}
Capsule::Datagram(d) => {
let st = self.state.lock().unwrap();
if let Some(tx) = &st.datagrams {
let _ = tx.try_send(Bytes::from(d));
}
Ok(())
}
Capsule::CloseSession { code, message } => {
self.set_closed(TransportError::Closed { code: code as u64, reason: String::from_utf8_lossy(&message).into_owned(), by_peer: true });
Ok(())
}
Capsule::DrainSession | Capsule::StreamsBlocked { .. } | Capsule::Ignored(_) => Ok(()),
}
}
fn on_stream(self: &Arc<Self>, id: u64, data: Option<Vec<u8>>, fin: bool, reset: Option<u64>) -> Result<(), (u64, &'static str)> {
let peer_initiated = stream_id::client_initiated(id) != self.client;
let bidi = stream_id::bidi(id);
let mut st = self.state.lock().unwrap();
if !st.recv.contains_key(&id) {
if !peer_initiated {
if !bidi {
return Err((FLOW_CONTROL_ERROR, "data on a unidirectional stream this end opened"));
}
// The peer's half of a stream this end opened and has stopped reading.
return Ok(());
}
let idx = stream_id::index(id);
let limit = if bidi { st.local_max_bidi } else { st.local_max_uni };
if idx >= limit {
return Err((FLOW_CONTROL_ERROR, "a stream past WT_MAX_STREAMS"));
}
let seen = if bidi { &mut st.seen_bidi } else { &mut st.seen_uni };
match seen.admit(idx) {
None => return Err((FLOW_CONTROL_ERROR, "stream IDs too far out of order")),
Some(false) => return Ok(()),
Some(true) => {}
}
let (tx, rx) = mpsc::unbounded_channel();
st.recv.insert(id, tx);
let shared = bidi.then(|| {
let s = Arc::new(SendShared::default());
st.send.insert(id, s.clone());
s
});
let (accept_bi, accept_uni) = (st.accept_bi.clone(), st.accept_uni.clone());
drop(st);
// Built and handed over with the state unlocked: a stream half dropped here
// (nobody accepting) takes the lock in its own Drop.
let credit = Arc::new(Credit { inner: Arc::downgrade(self), bidi });
let recv = MuxRecv { id, rx, inner: self.clone(), done: false, _credit: Some(credit.clone()) };
match (shared, accept_bi, accept_uni) {
(Some(shared), Some(a), _) => {
let send = MuxSend { id, inner: self.clone(), shared, ended: false, _credit: Some(credit) };
let _ = a.send((Box::new(send), Box::new(recv)));
}
(None, _, Some(a)) => {
let _ = a.send(Box::new(recv));
}
_ => {}
}
st = self.state.lock().unwrap();
}
let tx = st.recv.get(&id).cloned();
if fin || reset.is_some() {
st.recv.remove(&id);
}
drop(st);
let Some(tx) = tx else { return Ok(()) };
if let Some(d) = data.filter(|d| !d.is_empty()) {
let n = self.buffered.fetch_add(d.len(), Ordering::AcqRel) + d.len();
if n > self.cfg.max_buffered {
return Err((FLOW_CONTROL_ERROR, "more received and unread than the session buffers"));
}
let _ = tx.send(RecvEvent::Data(Bytes::from(d)));
}
if let Some(code) = reset {
let _ = tx.send(RecvEvent::Reset(code));
} else if fin {
let _ = tx.send(RecvEvent::Fin);
}
Ok(())
}
async fn open(self: &Arc<Self>, bidi: bool) -> TResult<u64> {
let mut generation = self.generation.subscribe();
loop {
if let Some(e) = self.is_closed() {
return Err(e);
}
{
let mut st = self.state.lock().unwrap();
let (next, max) = if bidi { (st.next_bidi, st.peer_max_bidi) } else { (st.next_uni, st.peer_max_uni) };
if next < max {
if bidi { st.next_bidi += 1 } else { st.next_uni += 1 }
return Ok(stream_id::nth(next, self.client, bidi));
}
let blocked = if bidi { &mut st.blocked_sent.0 } else { &mut st.blocked_sent.1 };
let send_blocked = *blocked != max;
*blocked = max;
drop(st);
if send_blocked {
self.push(&Capsule::StreamsBlocked { bidi, max });
}
}
if generation.changed().await.is_err() {
return Err(self.closed_err());
}
}
}
async fn wait_space(&self) -> TResult<()> {
let mut rx = self.space.subscribe();
while self.queued.load(Ordering::Acquire) > self.cfg.send_high_water {
if let Some(e) = self.is_closed() {
return Err(e);
}
if rx.changed().await.is_err() {
return Err(self.closed_err());
}
}
Ok(())
}
}
pub struct MuxSend {
id: u64,
inner: Arc<Inner>,
shared: Arc<SendShared>,
ended: bool,
_credit: Option<Arc<Credit>>,
}
pub struct MuxRecv {
id: u64,
rx: mpsc::UnboundedReceiver<RecvEvent>,
inner: Arc<Inner>,
done: bool,
_credit: Option<Arc<Credit>>,
}
impl Drop for MuxSend {
fn drop(&mut self) {
if !self.ended && self.inner.is_closed().is_none() {
self.inner.push(&Capsule::Stream { id: self.id, data: vec![], fin: true });
}
self.inner.state.lock().unwrap().send.remove(&self.id);
}
}
impl Drop for MuxRecv {
fn drop(&mut self) {
if !self.done {
self.inner.state.lock().unwrap().recv.remove(&self.id);
if self.inner.is_closed().is_none() {
self.inner.push(&Capsule::StopSending { id: self.id, code: 0 });
}
}
while let Ok(ev) = self.rx.try_recv() {
if let RecvEvent::Data(b) = ev {
self.inner.buffered.fetch_sub(b.len(), Ordering::AcqRel);
}
}
}
}
#[async_trait]
impl SendStream for MuxSend {
async fn write_all(&mut self, buf: &[u8]) -> TResult<()> {
for piece in buf.chunks(CHUNK) {
if let Some(code) = *self.shared.stopped.lock().unwrap() {
return Err(TransportError::Stopped(code));
}
if let Some(e) = self.inner.is_closed() {
return Err(e);
}
if self.ended {
return Err(TransportError::Other("a write after the stream ended".into()));
}
self.inner.wait_space().await?;
self.inner.push(&Capsule::Stream { id: self.id, data: piece.to_vec(), fin: false });
}
Ok(())
}
fn finish(&mut self) -> TResult<()> {
if self.ended {
return Err(TransportError::Other("the stream already ended".into()));
}
self.ended = true;
self.inner.push(&Capsule::Stream { id: self.id, data: vec![], fin: true });
Ok(())
}
fn reset(&mut self, code: u64) {
if !self.ended {
self.ended = true;
self.inner.push(&Capsule::ResetStream { id: self.id, code, reliable_size: 0 });
}
}
fn set_priority(&mut self, _priority: i32) {}
async fn delivered(&mut self) -> TResult<Option<u64>> {
// webtrans-http2 §5.2: with no acknowledgements of its own, the binding makes the
// transitions QUIC makes on an acknowledgement at once.
Ok(*self.shared.stopped.lock().unwrap())
}
}
#[async_trait]
impl RecvStream for MuxRecv {
async fn read_chunk(&mut self) -> TResult<Option<Bytes>> {
if self.done {
return Ok(None);
}
match self.rx.recv().await {
Some(RecvEvent::Data(b)) => {
self.inner.buffered.fetch_sub(b.len(), Ordering::AcqRel);
Ok(Some(b))
}
Some(RecvEvent::Fin) => {
self.done = true;
Ok(None)
}
Some(RecvEvent::Reset(code)) => {
self.done = true;
Err(TransportError::Reset(code))
}
None => {
self.done = true;
Err(self.inner.closed_err())
}
}
}
fn stop(&mut self, code: u64) {
if !self.done {
self.done = true;
self.inner.state.lock().unwrap().recv.remove(&self.id);
self.inner.push(&Capsule::StopSending { id: self.id, code });
}
}
}
/// One multiplexed session.
pub struct MuxConnection {
inner: Arc<Inner>,
}
impl MuxConnection {
/// Run a session over a pipe. `client` is this end's role, which numbers its streams.
/// The tasks it spawns end when the session closes.
pub fn new(client: bool, kind: TransportKind, remote: String, cfg: MuxConfig, mut tx: Box<dyn PipeTx>, mut rx: Box<dyn PipeRx>) -> Self {
let (out, mut out_rx) = mpsc::unbounded_channel::<Vec<u8>>();
let (uni_tx, uni_rx) = mpsc::unbounded_channel();
let (bi_tx, bi_rx) = mpsc::unbounded_channel();
let (dg_tx, dg_rx) = mpsc::channel(cfg.datagram_queue.max(1));
let inner = Arc::new(Inner {
client,
kind,
remote,
cfg,
out,
queued: AtomicUsize::new(0),
space: watch::channel(0).0,
state: Mutex::new(State {
next_bidi: 0,
next_uni: 0,
peer_max_bidi: 0,
peer_max_uni: 0,
local_max_bidi: cfg.max_bidi,
local_max_uni: cfg.max_uni,
pending_bidi: 0,
pending_uni: 0,
seen_bidi: Seen::default(),
seen_uni: Seen::default(),
recv: HashMap::new(),
send: HashMap::new(),
accept_uni: Some(uni_tx),
accept_bi: Some(bi_tx),
datagrams: Some(dg_tx),
blocked_sent: (u64::MAX, u64::MAX),
}),
generation: watch::channel(0).0,
closed: watch::channel(None).0,
buffered: AtomicUsize::new(0),
accept_uni_rx: tokio::sync::Mutex::new(uni_rx),
accept_bi_rx: tokio::sync::Mutex::new(bi_rx),
datagram_rx: tokio::sync::Mutex::new(dg_rx),
});
// Each end's limits go first: until the peer's arrive, it opens nothing.
inner.push(&Capsule::MaxStreams { bidi: true, max: cfg.max_bidi });
inner.push(&Capsule::MaxStreams { bidi: false, max: cfg.max_uni });
let w = inner.clone();
tokio::spawn(async move {
let mut closed = w.closed.subscribe();
loop {
tokio::select! {
m = out_rx.recv() => {
let Some(m) = m else { break };
let n = m.len();
let r = tx.send(m).await;
w.queued.fetch_sub(n, Ordering::AcqRel);
w.space.send_modify(|g| *g += 1);
if let Err(e) = r {
w.set_closed(e);
break;
}
}
_ = closed.changed() => {
// Flush what was queued before the close (the WT_CLOSE_SESSION among it).
while let Ok(m) = out_rx.try_recv() {
if tx.send(m).await.is_err() {
break;
}
}
break;
}
}
}
tx.close().await;
});
let r = inner.clone();
tokio::spawn(async move {
let mut closed = r.closed.subscribe();
loop {
let msg = tokio::select! {
m = rx.recv() => m,
_ = closed.changed() => break,
};
match msg {
Ok(Some(m)) => match Capsule::decode(&m) {
Ok(c) => {
if let Err((code, why)) = r.on_capsule(c) {
r.close_with(code, why.as_bytes());
break;
}
}
Err(_) => {
r.close_with(FLOW_CONTROL_ERROR, b"a message that is not one capsule");
break;
}
},
Ok(None) => {
r.set_closed(TransportError::Closed { code: 0, reason: "the pipe closed".into(), by_peer: true });
break;
}
Err(e) => {
r.set_closed(e);
break;
}
}
}
});
MuxConnection { inner }
}
}
#[async_trait]
impl Connection for MuxConnection {
fn kind(&self) -> TransportKind {
self.inner.kind
}
fn remote(&self) -> String {
self.inner.remote.clone()
}
async fn open_uni(&self) -> TResult<BoxSend> {
let id = self.inner.open(false).await?;
let shared = Arc::new(SendShared::default());
self.inner.state.lock().unwrap().send.insert(id, shared.clone());
Ok(Box::new(MuxSend { id, inner: self.inner.clone(), shared, ended: false, _credit: None }))
}
async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)> {
let id = self.inner.open(true).await?;
let shared = Arc::new(SendShared::default());
let (tx, rx) = mpsc::unbounded_channel();
{
let mut st = self.inner.state.lock().unwrap();
st.send.insert(id, shared.clone());
st.recv.insert(id, tx);
}
Ok((
Box::new(MuxSend { id, inner: self.inner.clone(), shared, ended: false, _credit: None }),
Box::new(MuxRecv { id, rx, inner: self.inner.clone(), done: false, _credit: None }),
))
}
async fn accept_uni(&self) -> TResult<BoxRecv> {
match self.inner.accept_uni_rx.lock().await.recv().await {
Some(r) => Ok(r),
None => Err(self.inner.closed_err()),
}
}
async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)> {
match self.inner.accept_bi_rx.lock().await.recv().await {
Some(p) => Ok(p),
None => Err(self.inner.closed_err()),
}
}
fn send_datagram(&self, payload: Bytes) -> TResult<()> {
if let Some(e) = self.inner.is_closed() {
return Err(e);
}
self.inner.push(&Capsule::Datagram(payload.to_vec()));
Ok(())
}
async fn recv_datagram(&self) -> TResult<Bytes> {
match self.inner.datagram_rx.lock().await.recv().await {
Some(d) => Ok(d),
None => Err(self.inner.closed_err()),
}
}
fn close(&self, code: u64, reason: &[u8]) {
self.inner.close_with(code, reason);
}
async fn closed(&self) -> TransportError {
let mut rx = self.inner.closed.subscribe();
loop {
if let Some(e) = rx.borrow_and_update().clone() {
return e;
}
if rx.changed().await.is_err() {
return self.inner.closed_err();
}
}
}
}
// ── The in-process pipe ─────────────────────────────────────────────────────
struct MemTx {
tx: mpsc::Sender<Vec<u8>>,
gate: Option<watch::Receiver<bool>>,
}
struct MemRx(mpsc::Receiver<Vec<u8>>);
#[async_trait]
impl PipeTx for MemTx {
async fn send(&mut self, msg: Vec<u8>) -> TResult<()> {
if let Some(g) = &mut self.gate {
while !*g.borrow_and_update() {
if g.changed().await.is_err() {
break;
}
}
}
self.tx.send(msg).await.map_err(|_| TransportError::Closed { code: 0, reason: "the pipe closed".into(), by_peer: true })
}
async fn close(&mut self) {}
}
#[async_trait]
impl PipeRx for MemRx {
async fn recv(&mut self) -> TResult<Option<Vec<u8>>> {
Ok(self.0.recv().await)
}
}
/// A gate on what one end of a [`memory_pair`] sends: closed, it holds every message;
/// opened, it lets them through in order.
pub struct Gate(watch::Sender<bool>);
impl Gate {
pub fn open(&self) {
let _ = self.0.send(true);
}
pub fn close(&self) {
let _ = self.0.send(false);
}
}
/// Two ends of an in-process session: `(client, server)`. With `gated`, what the server
/// sends waits behind the returned [`Gate`], which starts closed.
pub fn memory_pair(cfg: MuxConfig, gated: bool) -> (MuxConnection, MuxConnection, Option<Gate>) {
let (c2s_tx, c2s_rx) = mpsc::channel(64);
let (s2c_tx, s2c_rx) = mpsc::channel(64);
let (gate, gate_rx) = if gated {
let (t, r) = watch::channel(false);
(Some(Gate(t)), Some(r))
} else {
(None, None)
};
let client = MuxConnection::new(true, TransportKind::Memory, "memory:server".into(), cfg, Box::new(MemTx { tx: c2s_tx, gate: None }), Box::new(MemRx(s2c_rx)));
let server = MuxConnection::new(false, TransportKind::Memory, "memory:client".into(), cfg, Box::new(MemTx { tx: s2c_tx, gate: gate_rx }), Box::new(MemRx(c2s_rx)));
(client, server, gate)
}
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
async fn read_all(r: &mut BoxRecv) -> TResult<Vec<u8>> {
let mut v = Vec::new();
while let Some(b) = tokio::time::timeout(Duration::from_secs(5), r.read_chunk()).await.expect("a read in time")? {
v.extend_from_slice(&b);
}
Ok(v)
}
#[tokio::test]
async fn streams_both_ways_and_a_reset() {
let (c, s, _) = memory_pair(MuxConfig::default(), false);
let (mut cs, mut cr) = c.open_bi().await.unwrap();
cs.write_all(b"request").await.unwrap();
cs.finish().unwrap();
let (mut ss, mut sr) = tokio::time::timeout(Duration::from_secs(5), s.accept_bi()).await.unwrap().unwrap();
assert_eq!(read_all(&mut sr).await.unwrap(), b"request");
ss.write_all(&vec![7u8; 40_000]).await.unwrap();
ss.finish().unwrap();
assert_eq!(read_all(&mut cr).await.unwrap().len(), 40_000);
let mut u = s.open_uni().await.unwrap();
u.write_all(b"half").await.unwrap();
u.reset(2);
let mut ur = tokio::time::timeout(Duration::from_secs(5), c.accept_uni()).await.unwrap().unwrap();
assert_eq!(ur.read_chunk().await.unwrap().as_deref(), Some(&b"half"[..]));
assert_eq!(ur.read_chunk().await, Err(TransportError::Reset(2)));
c.send_datagram(Bytes::from_static(b"dg")).unwrap();
assert_eq!(tokio::time::timeout(Duration::from_secs(5), s.recv_datagram()).await.unwrap().unwrap(), Bytes::from_static(b"dg"));
s.close(0x10, b"bye");
match tokio::time::timeout(Duration::from_secs(5), c.closed()).await.unwrap() {
TransportError::Closed { code, by_peer, .. } => assert_eq!((code, by_peer), (0x10, true)),
other => panic!("{other:?}"),
}
}
#[tokio::test]
async fn the_stream_limit_holds_until_credit_returns() {
let cfg = MuxConfig { max_bidi: 1, ..MuxConfig::default() };
let (c, s, _) = memory_pair(cfg, false);
let (mut a, _ar) = c.open_bi().await.unwrap();
a.write_all(b"one").await.unwrap();
// A second request waits for credit.
let second = tokio::time::timeout(Duration::from_millis(200), c.open_bi()).await;
assert!(second.is_err(), "a second stream opened past WT_MAX_STREAMS");
let (ss, sr) = tokio::time::timeout(Duration::from_secs(5), s.accept_bi()).await.unwrap().unwrap();
drop((ss, sr));
let opened = tokio::time::timeout(Duration::from_secs(5), c.open_bi()).await;
assert!(opened.is_ok(), "credit came back when the first stream ended");
}
}

208
joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/quic.rs Normal file
View file

@ -0,0 +1,208 @@
//! Native QUIC (`draft-ietf-moq-transport-22` §6.2.2) on quinn.
use super::{BoxRecv, BoxSend, Connection, RecvStream, SendStream, TResult, TransportError, TransportKind};
use async_trait::async_trait;
use bytes::Bytes;
use quinn::VarInt;
/// A quinn connection whose ALPN selected `moqt-22`.
pub struct QuicConnection {
conn: quinn::Connection,
}
impl QuicConnection {
pub fn new(conn: quinn::Connection) -> Self {
QuicConnection { conn }
}
pub fn inner(&self) -> &quinn::Connection {
&self.conn
}
}
/// A MOQT code as a QUIC application error code (62 bits).
pub(crate) fn varint(code: u64) -> VarInt {
VarInt::from_u64(code).unwrap_or(VarInt::MAX)
}
/// Why a quinn connection ended, as this crate reports it.
pub(crate) fn conn_err(e: &quinn::ConnectionError) -> TransportError {
match e {
quinn::ConnectionError::ApplicationClosed(c) => TransportError::Closed {
code: c.error_code.into_inner(),
reason: String::from_utf8_lossy(&c.reason).into_owned(),
by_peer: true,
},
quinn::ConnectionError::LocallyClosed => TransportError::Closed { code: 0, reason: "closed by this endpoint".into(), by_peer: false },
other => TransportError::Closed { code: u64::MAX, reason: other.to_string(), by_peer: true },
}
}
pub(crate) fn write_err(e: quinn::WriteError) -> TransportError {
match e {
quinn::WriteError::Stopped(code) => TransportError::Stopped(code.into_inner()),
quinn::WriteError::ConnectionLost(c) => conn_err(&c),
other => TransportError::other(other),
}
}
pub(crate) fn read_err(e: quinn::ReadError) -> TransportError {
match e {
quinn::ReadError::Reset(code) => TransportError::Reset(code.into_inner()),
quinn::ReadError::ConnectionLost(c) => conn_err(&c),
other => TransportError::other(other),
}
}
/// The largest chunk a read hands back.
pub(crate) const READ_CHUNK: usize = 64 * 1024;
pub struct QuicSend(pub quinn::SendStream);
pub struct QuicRecv(pub quinn::RecvStream);
#[async_trait]
impl SendStream for QuicSend {
async fn write_all(&mut self, buf: &[u8]) -> TResult<()> {
self.0.write_all(buf).await.map_err(write_err)
}
fn finish(&mut self) -> TResult<()> {
self.0.finish().map_err(TransportError::other)
}
fn reset(&mut self, code: u64) {
let _ = self.0.reset(varint(code));
}
fn set_priority(&mut self, priority: i32) {
let _ = self.0.set_priority(priority);
}
async fn delivered(&mut self) -> TResult<Option<u64>> {
match self.0.stopped().await {
Ok(None) => Ok(None),
Ok(Some(code)) => Ok(Some(code.into_inner())),
Err(quinn::StoppedError::ConnectionLost(c)) => Err(conn_err(&c)),
Err(e) => Err(TransportError::other(e)),
}
}
}
#[async_trait]
impl RecvStream for QuicRecv {
async fn read_chunk(&mut self) -> TResult<Option<Bytes>> {
match self.0.read_chunk(READ_CHUNK, true).await {
Ok(Some(c)) => Ok(Some(c.bytes)),
Ok(None) => Ok(None),
Err(e) => Err(read_err(e)),
}
}
fn stop(&mut self, code: u64) {
let _ = self.0.stop(varint(code));
}
}
#[async_trait]
impl Connection for QuicConnection {
fn kind(&self) -> TransportKind {
TransportKind::Quic
}
fn remote(&self) -> String {
self.conn.remote_address().to_string()
}
async fn open_uni(&self) -> TResult<BoxSend> {
let s = self.conn.open_uni().await.map_err(|e| conn_err(&e))?;
Ok(Box::new(QuicSend(s)))
}
async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)> {
let (s, r) = self.conn.open_bi().await.map_err(|e| conn_err(&e))?;
Ok((Box::new(QuicSend(s)), Box::new(QuicRecv(r))))
}
async fn accept_uni(&self) -> TResult<BoxRecv> {
let r = self.conn.accept_uni().await.map_err(|e| conn_err(&e))?;
Ok(Box::new(QuicRecv(r)))
}
async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)> {
let (s, r) = self.conn.accept_bi().await.map_err(|e| conn_err(&e))?;
Ok((Box::new(QuicSend(s)), Box::new(QuicRecv(r))))
}
fn send_datagram(&self, payload: Bytes) -> TResult<()> {
self.conn.send_datagram(payload).map_err(TransportError::other)
}
async fn recv_datagram(&self) -> TResult<Bytes> {
self.conn.read_datagram().await.map_err(|e| conn_err(&e))
}
fn close(&self, code: u64, reason: &[u8]) {
self.conn.close(varint(code), reason);
}
async fn closed(&self) -> TransportError {
conn_err(&self.conn.closed().await)
}
fn diagnostics(&self) -> Option<String> {
Some(quic_diagnostics(&self.conn))
}
}
/// The flow-control and loss counters of a quinn connection, one line.
pub fn quic_diagnostics(c: &quinn::Connection) -> String {
let s = c.stats();
format!(
"rx DATA_BLOCKED {} STREAM_DATA_BLOCKED {} STREAMS_BLOCKED_UNI {} MAX_DATA {} RESET_STREAM {} STOP_SENDING {}; tx DATA_BLOCKED {} STREAM_DATA_BLOCKED {} STREAMS_BLOCKED_UNI {} MAX_DATA {} MAX_STREAM_DATA {} MAX_STREAMS_UNI {} RESET_STREAM {} STOP_SENDING {}; lost packets {} cwnd {}",
s.frame_rx.data_blocked,
s.frame_rx.stream_data_blocked,
s.frame_rx.streams_blocked_uni,
s.frame_rx.max_data,
s.frame_rx.reset_stream,
s.frame_rx.stop_sending,
s.frame_tx.data_blocked,
s.frame_tx.stream_data_blocked,
s.frame_tx.streams_blocked_uni,
s.frame_tx.max_data,
s.frame_tx.max_stream_data,
s.frame_tx.max_streams_uni,
s.frame_tx.reset_stream,
s.frame_tx.stop_sending,
s.path.lost_packets,
s.path.cwnd,
)
}
/// A client endpoint that offers `alpns` and trusts `roots`, with an optional
/// transport configuration (a subscriber that limits its receive windows, say).
pub fn client_endpoint(
bind: std::net::SocketAddr,
roots: Vec<rustls::pki_types::CertificateDer<'static>>,
alpns: Vec<Vec<u8>>,
transport: Option<quinn::TransportConfig>,
) -> TResult<quinn::Endpoint> {
crate::cert::install_provider();
let mut store = rustls::RootCertStore::empty();
for c in roots {
store.add(c).map_err(TransportError::other)?;
}
let provider = std::sync::Arc::new(rustls::crypto::ring::default_provider());
let mut tls = rustls::ClientConfig::builder_with_provider(provider)
.with_protocol_versions(&[&rustls::version::TLS13])
.map_err(TransportError::other)?
.with_root_certificates(store)
.with_no_client_auth();
tls.alpn_protocols = alpns;
let quic = quinn::crypto::rustls::QuicClientConfig::try_from(tls).map_err(TransportError::other)?;
let mut cc = quinn::ClientConfig::new(std::sync::Arc::new(quic));
if let Some(t) = transport {
cc.transport_config(std::sync::Arc::new(t));
}
let mut ep = quinn::Endpoint::client(bind).map_err(TransportError::other)?;
ep.set_default_client_config(cc);
Ok(ep)
}
/// Dial `addr` as `server_name` and return the connection if its ALPN is `moqt-22`.
pub async fn connect(ep: &quinn::Endpoint, addr: std::net::SocketAddr, server_name: &str) -> TResult<QuicConnection> {
let conn = ep.connect(addr, server_name).map_err(TransportError::other)?.await.map_err(|e| conn_err(&e))?;
let alpn = conn
.handshake_data()
.and_then(|h| h.downcast::<quinn::crypto::rustls::HandshakeData>().ok())
.and_then(|h| h.protocol.clone());
if alpn.as_deref() != Some(joule_moq::draft22::ALPN) {
conn.close(varint(0), b"no MOQT ALPN");
return Err(TransportError::Other("the server selected no moqt-22 ALPN".into()));
}
Ok(QuicConnection::new(conn))
}

82
joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/websocket.rs Normal file
View file

@ -0,0 +1,82 @@
//! The WebSocket fallback: a non-normative binding of MOQT -22 for paths that carry no
//! UDP, where neither native QUIC nor WebTransport over HTTP/3 can reach the relay.
//!
//! The binding (the crate README states it in full):
//!
//! - the client opens a WebSocket (RFC 6455) to the relay's path with the subprotocol
//! `moqt-22` in `Sec-WebSocket-Protocol`; the relay answers with that subprotocol, or
//! refuses the upgrade. The subprotocol selects the draft, as the ALPN does on QUIC;
//! - every binary message is exactly one capsule of `draft-ietf-webtrans-http2-15` §6, and
//! the session is what a WebTransport session over HTTP/2 would carry ([`super::mux`]);
//! - MOQT's control, request and data streams ride those streams unchanged: the same
//! bytes a QUIC stream would carry.
use super::mux::{MuxConfig, MuxConnection, PipeRx, PipeTx};
use super::{TResult, TransportError, TransportKind};
use async_trait::async_trait;
use futures::stream::{SplitSink, SplitStream};
use futures::{SinkExt, StreamExt};
use tokio::io::{AsyncRead, AsyncWrite};
use tokio_tungstenite::tungstenite::Message;
use tokio_tungstenite::WebSocketStream;
/// The subprotocol that selects this draft.
pub fn subprotocol() -> &'static str {
std::str::from_utf8(joule_moq::draft22::ALPN).expect("the ALPN is ASCII")
}
struct WsTx<S>(SplitSink<WebSocketStream<S>, Message>);
struct WsRx<S>(SplitStream<WebSocketStream<S>>);
#[async_trait]
impl<S: AsyncRead + AsyncWrite + Unpin + Send + 'static> PipeTx for WsTx<S> {
async fn send(&mut self, msg: Vec<u8>) -> TResult<()> {
self.0.send(Message::Binary(msg.into())).await.map_err(TransportError::other)
}
async fn close(&mut self) {
let _ = self.0.close().await;
}
}
#[async_trait]
impl<S: AsyncRead + AsyncWrite + Unpin + Send + 'static> PipeRx for WsRx<S> {
async fn recv(&mut self) -> TResult<Option<Vec<u8>>> {
loop {
match self.0.next().await {
None => return Ok(None),
Some(Err(e)) => return Err(TransportError::other(e)),
Some(Ok(Message::Binary(b))) => return Ok(Some(b.to_vec())),
Some(Ok(Message::Close(_))) => return Ok(None),
// Text carries nothing in this binding; pings are answered by the library.
Some(Ok(_)) => continue,
}
}
}
}
/// A MOQT session over an upgraded WebSocket.
pub fn session<S: AsyncRead + AsyncWrite + Unpin + Send + 'static>(ws: WebSocketStream<S>, client: bool, remote: String, cfg: MuxConfig) -> MuxConnection {
let (tx, rx) = ws.split();
MuxConnection::new(client, TransportKind::WebSocket, remote, cfg, Box::new(WsTx(tx)), Box::new(WsRx(rx)))
}
/// Whether a `Sec-WebSocket-Protocol` header offers this draft.
pub fn offers_subprotocol(header: Option<&str>) -> bool {
header.is_some_and(|h| h.split(',').any(|p| p.trim() == subprotocol()))
}
/// Connect to a relay's WebSocket endpoint (`ws://` or `wss://`), offering `moqt-22`.
pub async fn connect(url: &str, cfg: MuxConfig) -> TResult<MuxConnection> {
use tokio_tungstenite::tungstenite::client::IntoClientRequest;
let mut req = url.into_client_request().map_err(TransportError::other)?;
req.headers_mut().insert(
"sec-websocket-protocol",
subprotocol().parse().map_err(TransportError::other)?,
);
let (ws, resp) = tokio_tungstenite::connect_async(req).await.map_err(TransportError::other)?;
let chosen = resp.headers().get("sec-websocket-protocol").and_then(|v| v.to_str().ok());
if chosen != Some(subprotocol()) {
return Err(TransportError::Other("the relay did not select the moqt-22 subprotocol".into()));
}
Ok(session(ws, true, url.to_string(), cfg))
}

222
joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/webtransport.rs Normal file
View file

@ -0,0 +1,222 @@
//! WebTransport over HTTP/3 (`draft-ietf-webtrans-http3-16`; `draft-ietf-moq-transport-22`
//! §6.2.1) on wtransport.
//!
//! The relay's quinn endpoint offers both `moqt-22` and `h3`; a connection whose ALPN is
//! `h3` is handed to wtransport, which runs the HTTP/3 layer and the extended CONNECT.
//! The draft is negotiated by `WT-Available-Protocols` and `WT-Protocol` (webtrans-http3
//! §3.3), both Structured Fields of String items.
//!
//! Once a WebTransport stream is open it is a QUIC stream behind a short header, so its
//! bytes are read and written on the quinn stream. Application error codes on WebTransport
//! streams are 32-bit and ride RESET_STREAM and STOP_SENDING mapped into the HTTP/3 error
//! space (webtrans-http3 §4.4); [`wt_to_h3`] and [`h3_to_wt`] are that mapping.
use super::quic::{conn_err, READ_CHUNK};
use super::{BoxRecv, BoxSend, Connection, RecvStream, SendStream, TResult, TransportError, TransportKind};
use async_trait::async_trait;
use bytes::Bytes;
use wtransport::VarInt;
/// The first HTTP/3 error code of the WebTransport application range.
const WT_FIRST: u64 = 0x52e4_a40f_a8db;
/// The last one.
const WT_LAST: u64 = 0x52e5_ac98_3162;
/// A WebTransport application error code as an HTTP/3 error code (webtrans-http3 §4.4).
pub fn wt_to_h3(n: u32) -> u64 {
let n = n as u64;
WT_FIRST + n + n / 0x1e
}
/// The inverse of [`wt_to_h3`]: `None` for a code outside the range or on one of its
/// reserved points.
pub fn h3_to_wt(h: u64) -> Option<u32> {
if !(WT_FIRST..=WT_LAST).contains(&h) || (h - 0x21).is_multiple_of(0x1f) {
return None;
}
let shifted = h - WT_FIRST;
u32::try_from(shifted - shifted / 0x1f).ok()
}
fn moqt_code_to_h3(code: u64) -> quinn::VarInt {
let n = u32::try_from(code).unwrap_or(u32::MAX);
quinn::VarInt::from_u64(wt_to_h3(n)).unwrap_or(quinn::VarInt::MAX)
}
fn h3_to_moqt(h: u64) -> u64 {
h3_to_wt(h).map_or(h, u64::from)
}
/// The protocol a client offered in `WT-Available-Protocols` that this relay speaks:
/// `"moqt-22"`. The header is a Structured Field List; only String items count, and
/// their parameters are ignored (webtrans-http3 §3.3).
pub fn select_protocol(header: Option<&str>) -> Option<&'static str> {
let want = std::str::from_utf8(joule_moq::draft22::ALPN).ok()?;
let h = header?;
for member in h.split(',') {
let item = member.trim().split(';').next().unwrap_or("").trim();
if let Some(s) = item.strip_prefix('"').and_then(|s| s.strip_suffix('"'))
&& s == want
{
return Some(want);
}
}
None
}
/// A WebTransport session that negotiated `moqt-22`.
pub struct WtConnection {
conn: wtransport::Connection,
}
impl WtConnection {
pub fn new(conn: wtransport::Connection) -> Self {
WtConnection { conn }
}
}
fn wt_conn_err(e: &wtransport::error::ConnectionError) -> TransportError {
use wtransport::error::ConnectionError as E;
match e {
E::ApplicationClosed(c) => TransportError::Closed {
code: c.code().into_inner(),
reason: String::from_utf8_lossy(c.reason()).into_owned(),
by_peer: true,
},
E::LocallyClosed => TransportError::Closed { code: 0, reason: "closed by this endpoint".into(), by_peer: false },
other => TransportError::Closed { code: u64::MAX, reason: other.to_string(), by_peer: true },
}
}
pub struct WtSend(wtransport::SendStream);
pub struct WtRecv(wtransport::RecvStream);
#[async_trait]
impl SendStream for WtSend {
async fn write_all(&mut self, buf: &[u8]) -> TResult<()> {
match self.0.quic_stream_mut().write_all(buf).await {
Ok(()) => Ok(()),
Err(quinn::WriteError::Stopped(code)) => Err(TransportError::Stopped(h3_to_moqt(code.into_inner()))),
Err(quinn::WriteError::ConnectionLost(c)) => Err(conn_err(&c)),
Err(e) => Err(TransportError::other(e)),
}
}
fn finish(&mut self) -> TResult<()> {
self.0.quic_stream_mut().finish().map_err(TransportError::other)
}
fn reset(&mut self, code: u64) {
let _ = self.0.quic_stream_mut().reset(moqt_code_to_h3(code));
}
fn set_priority(&mut self, priority: i32) {
self.0.set_priority(priority);
}
async fn delivered(&mut self) -> TResult<Option<u64>> {
match self.0.quic_stream_mut().stopped().await {
Ok(None) => Ok(None),
Ok(Some(code)) => Ok(Some(h3_to_moqt(code.into_inner()))),
Err(quinn::StoppedError::ConnectionLost(c)) => Err(conn_err(&c)),
Err(e) => Err(TransportError::other(e)),
}
}
}
#[async_trait]
impl RecvStream for WtRecv {
async fn read_chunk(&mut self) -> TResult<Option<Bytes>> {
match self.0.quic_stream_mut().read_chunk(READ_CHUNK, true).await {
Ok(Some(c)) => Ok(Some(c.bytes)),
Ok(None) => Ok(None),
Err(quinn::ReadError::Reset(code)) => Err(TransportError::Reset(h3_to_moqt(code.into_inner()))),
Err(quinn::ReadError::ConnectionLost(c)) => Err(conn_err(&c)),
Err(e) => Err(TransportError::other(e)),
}
}
fn stop(&mut self, code: u64) {
let _ = self.0.quic_stream_mut().stop(moqt_code_to_h3(code));
}
}
#[async_trait]
impl Connection for WtConnection {
fn kind(&self) -> TransportKind {
TransportKind::WebTransport
}
fn remote(&self) -> String {
self.conn.remote_address().to_string()
}
async fn open_uni(&self) -> TResult<BoxSend> {
let opening = self.conn.open_uni().await.map_err(|e| wt_conn_err(&e))?;
let s = opening.await.map_err(TransportError::other)?;
Ok(Box::new(WtSend(s)))
}
async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)> {
let opening = self.conn.open_bi().await.map_err(|e| wt_conn_err(&e))?;
let (s, r) = opening.await.map_err(TransportError::other)?;
Ok((Box::new(WtSend(s)), Box::new(WtRecv(r))))
}
async fn accept_uni(&self) -> TResult<BoxRecv> {
let r = self.conn.accept_uni().await.map_err(|e| wt_conn_err(&e))?;
Ok(Box::new(WtRecv(r)))
}
async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)> {
let (s, r) = self.conn.accept_bi().await.map_err(|e| wt_conn_err(&e))?;
Ok((Box::new(WtSend(s)), Box::new(WtRecv(r))))
}
fn send_datagram(&self, payload: Bytes) -> TResult<()> {
self.conn.send_datagram(&payload).map_err(TransportError::other)
}
async fn recv_datagram(&self) -> TResult<Bytes> {
let d = self.conn.receive_datagram().await.map_err(|e| wt_conn_err(&e))?;
Ok(d.payload())
}
fn close(&self, code: u64, reason: &[u8]) {
let code = VarInt::try_from_u64(code).unwrap_or(VarInt::MAX);
self.conn.close(code, reason);
}
async fn closed(&self) -> TransportError {
wt_conn_err(&self.conn.closed().await)
}
}
/// A WebTransport client for tests and tools. With `cert_hash` it trusts exactly the
/// server certificate of that SHA-256 hash, under the rules a browser applies to
/// `serverCertificateHashes` (ECDSA P-256, a validity of two weeks at most); without it
/// the platform's roots.
pub async fn connect(url: &str, cert_hash: Option<[u8; 32]>, bind: std::net::SocketAddr) -> TResult<WtConnection> {
crate::cert::install_provider();
let b = wtransport::ClientConfig::builder().with_bind_address(bind);
let cfg = match cert_hash {
Some(h) => b.with_server_certificate_hashes([wtransport::tls::Sha256Digest::new(h)]).build(),
None => b.with_native_certs().build(),
};
let ep = wtransport::Endpoint::client(cfg).map_err(TransportError::other)?;
let want = std::str::from_utf8(joule_moq::draft22::ALPN).map_err(TransportError::other)?;
let opts = wtransport::endpoint::ConnectOptions::builder(url).add_header("wt-available-protocols", format!("\"{want}\"")).build();
let conn = ep.connect(opts).await.map_err(TransportError::other)?;
Ok(WtConnection::new(conn))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_error_code_mapping_round_trips() {
assert_eq!(wt_to_h3(0), WT_FIRST);
assert_eq!(wt_to_h3(u32::MAX), WT_LAST);
for n in [0u32, 1, 2, 0x1d, 0x1e, 0x1f, 0x3c, 1000, 0xffff, u32::MAX] {
assert_eq!(h3_to_wt(wt_to_h3(n)), Some(n), "{n}");
}
assert_eq!(h3_to_wt(WT_FIRST - 1), None);
assert_eq!(h3_to_wt(0x2), None);
}
#[test]
fn the_protocol_is_chosen_from_a_structured_list() {
assert_eq!(select_protocol(Some("\"moqt-22\"")), Some("moqt-22"));
assert_eq!(select_protocol(Some("\"moqt-18\", \"moqt-22\";q=1")), Some("moqt-22"));
assert_eq!(select_protocol(Some("moqt-22")), None, "a token, not a String item");
assert_eq!(select_protocol(Some("\"moqt-18\"")), None);
assert_eq!(select_protocol(None), None);
}
}

202
joule-code/joule-code-rs/crates/joule-moq-relay/src/wire.rs Normal file
View file

@ -0,0 +1,202 @@
//! Reading MOQT from the transport's streams: framed control messages on control and
//! request streams, and the items of data streams as their bytes arrive.
use crate::transport::{BoxRecv, BoxSend, TransportError};
use joule_moq::draft22::data::{Limits, UniStreamDecoder};
use joule_moq::draft22::{ControlMessage, GroupOrder, RequestKind, RequestStream, Side, StreamHeader, UniEvent, WireError};
use std::sync::{Arc, Mutex};
/// Why reading or writing a stream stopped.
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum StreamError {
#[error("transport: {0}")]
Transport(#[from] TransportError),
#[error("moqt: {0}")]
Wire(#[from] WireError),
}
/// Framed control messages (§9: Type, a 16-bit Length, the body) read from a stream.
pub struct MessageReader {
recv: BoxRecv,
buf: Vec<u8>,
fin: bool,
}
impl MessageReader {
pub fn new(recv: BoxRecv) -> Self {
MessageReader { recv, buf: Vec::new(), fin: false }
}
/// The length of the whole message at the front of `b`, once its header is there.
fn framed_len(b: &[u8]) -> Option<usize> {
let first = *b.first()?;
let tlen = joule_moq::draft22::varint::len_from_first(first);
if b.len() < tlen + 2 {
return None;
}
let len = u16::from_be_bytes([b[tlen], b[tlen + 1]]) as usize;
Some(tlen + 2 + len)
}
/// The next message; `None` at a FIN on a message boundary.
pub async fn next(&mut self) -> Result<Option<ControlMessage>, StreamError> {
loop {
if let Some(n) = Self::framed_len(&self.buf).filter(|n| *n <= self.buf.len()) {
let m = ControlMessage::decode_exact(&self.buf[..n])?;
self.buf.drain(..n);
return Ok(Some(m));
}
if self.fin {
if self.buf.is_empty() {
return Ok(None);
}
return Err(WireError::Truncated { what: "a stream ends inside a control message" }.into());
}
match self.recv.read_chunk().await? {
Some(b) => self.buf.extend_from_slice(&b),
None => self.fin = true,
}
}
}
pub fn stop(&mut self, code: u64) {
self.recv.stop(code);
}
}
/// The items of a unidirectional stream, each yielded once all of it has arrived.
pub struct UniReader {
recv: BoxRecv,
dec: UniStreamDecoder,
fin: bool,
/// What the stream began with, once read.
pub header: Option<StreamHeader>,
}
impl UniReader {
pub fn new(recv: BoxRecv, max_payload: u64) -> Self {
let limits = Limits { max_payload, ..Limits::default() };
UniReader { recv, dec: UniStreamDecoder::new(GroupOrder::Ascending, limits), fin: false, header: None }
}
/// Read up to the stream's header, and return it.
pub async fn start(&mut self) -> Result<StreamHeader, StreamError> {
if let Some(h) = &self.header {
return Ok(h.clone());
}
match self.next().await? {
Some(UniEvent::Header(h)) => Ok(h),
_ => Err(WireError::Truncated { what: "a unidirectional stream ends before its type" }.into()),
}
}
/// The Group Order of the request a fetch stream answers.
pub fn set_group_order(&mut self, order: GroupOrder) {
self.dec.set_group_order(order);
}
/// The next item; `None` at the FIN.
pub async fn next(&mut self) -> Result<Option<UniEvent>, StreamError> {
loop {
if let Some(e) = self.dec.next()? {
if let UniEvent::Header(h) = &e {
self.header = Some(h.clone());
}
return Ok(Some(e));
}
if self.fin {
self.dec.finish()?;
return Ok(None);
}
match self.recv.read_chunk().await? {
Some(b) => self.dec.push(&b),
None => self.fin = true,
}
}
}
pub fn stop(&mut self, code: u64) {
self.recv.stop(code);
}
}
/// The sending direction of a request stream, shareable between tasks: every message is
/// checked against the stream's rules before it goes ([`RequestStream::on_send`]).
#[derive(Clone)]
pub struct ReqTx {
send: Arc<tokio::sync::Mutex<BoxSend>>,
pub state: Arc<Mutex<RequestStream>>,
}
impl ReqTx {
pub async fn send(&self, m: &ControlMessage) -> Result<(), StreamError> {
self.state.lock().unwrap().on_send(m)?;
let b = m.encode()?;
self.send.lock().await.write_all(&b).await?;
Ok(())
}
/// A FIN on this direction (§6.4.2.2).
pub async fn fin(&self) {
let _ = self.send.lock().await.finish();
}
/// Cancel this direction (§6.4.2.3).
pub async fn reset(&self, code: u64) {
self.send.lock().await.reset(code);
}
}
/// The receiving direction of a request stream: each message admitted by the stream's
/// rules ([`RequestStream::on_receive`]).
pub struct ReqRx {
pub reader: MessageReader,
pub state: Arc<Mutex<RequestStream>>,
}
impl ReqRx {
pub async fn next(&mut self) -> Result<Option<ControlMessage>, StreamError> {
let m = self.reader.next().await?;
if let Some(m) = &m {
self.state.lock().unwrap().on_receive(m)?;
}
Ok(m)
}
}
impl std::fmt::Debug for ReqTx {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ReqTx").field("kind", &self.state.lock().unwrap().kind()).finish_non_exhaustive()
}
}
impl std::fmt::Debug for ReqRx {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ReqRx").field("kind", &self.state.lock().unwrap().kind()).finish_non_exhaustive()
}
}
/// A request stream this endpoint opened, its request already sent.
pub fn requester(send: BoxSend, recv: BoxRecv, kind: RequestKind, my_limit: u64, peer_limit: u64) -> (ReqTx, ReqRx) {
let state = Arc::new(Mutex::new(RequestStream::new(kind, Side::Requester, my_limit, peer_limit)));
(ReqTx { send: Arc::new(tokio::sync::Mutex::new(send)), state: state.clone() }, ReqRx { reader: MessageReader::new(recv), state })
}
/// A request stream the peer opened with `first`.
pub fn responder(send: BoxSend, reader: MessageReader, first: &ControlMessage, my_limit: u64, peer_limit: u64) -> Result<(ReqTx, ReqRx), WireError> {
let state = Arc::new(Mutex::new(RequestStream::opened_by_peer(first, my_limit, peer_limit)?));
Ok((ReqTx { send: Arc::new(tokio::sync::Mutex::new(send)), state: state.clone() }, ReqRx { reader, state }))
}
/// Open a request stream on `conn` with `first` (§6.3).
pub async fn open_request(
conn: &dyn crate::transport::Connection,
first: &ControlMessage,
my_limit: u64,
peer_limit: u64,
) -> Result<(ReqTx, ReqRx), StreamError> {
let kind = first.opens_request().ok_or(WireError::UnexpectedMessage("not a request"))?;
let (mut send, recv) = conn.open_bi().await?;
send.write_all(&first.encode()?).await?;
Ok(requester(send, recv, kind, my_limit, peer_limit))
}

62
joule-code/joule-code-rs/crates/joule-moq-relay/tests/auth_tokens.rs Normal file
View file

@ -0,0 +1,62 @@
//! Authorization tokens and their alias cache (`draft-ietf-moq-transport-22` §8.9, §9.1.3,
//! §9.1.4), over the WebSocket fallback: a token registered in SETUP authorizes the
//! session; one registered in a request is used by alias in the next; a request without a
//! token is refused; a reference to an alias never registered closes the session.
mod common;
use common::*;
use joule_moq::draft22::{Parameter, Parameters, RequestErrorCode, SessionError, Token};
use joule_moq_relay::client::ClientOptions;
use joule_moq_relay::config::Grant;
use joule_moq_relay::transport::TransportError;
use std::time::Duration;
use tokio::time::timeout;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn tokens_by_value_and_by_alias() {
timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time");
}
async fn run() {
// The token is made here; only its hash is in the relay's configuration.
let token: Vec<u8> = (0..24u8).map(|i| i.wrapping_mul(37).wrapping_add(11)).collect();
let hash = hex::encode(joule_moq_relay::auth::sha256(&token));
let server = relay(|c| {
c.auth.required = true;
c.auth.grants = vec![Grant { prefix: vec!["live".into()], token_sha256: hash, publish: true, subscribe: true }];
c.limits.max_auth_token_cache_size = 1024;
})
.await;
let t = Tracks::new(ns(&["live", "t"]), "cam", 10, 5, 128);
// A publisher whose SETUP registers the token (§9.1.4).
let setup_token = Token::Register { alias: 1, token_type: 0, value: token.clone() };
let (p, _) = connect(&server, Via::WebSocket, ClientOptions { tokens: vec![setup_token], ..ClientOptions::default() }).await;
let _pub = publish_tracks(&p, &t, None).await;
// A subscriber with no token is refused.
let (s, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let refused = s.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.expect_err("no token");
assert!(refused.contains(&RequestErrorCode::UNAUTHORIZED.to_string()), "{refused}");
// It registers the token in one request, then uses it by alias in the next (§8.9).
let reg = Parameters::new().with(Parameter::AuthorizationToken(Token::Register { alias: 7, token_type: 0, value: token.clone() }));
s.subscribe(t.ns.clone(), t.media.clone(), reg).await.expect("authorized by the token it registers");
let by_alias = Parameters::new().with(Parameter::AuthorizationToken(Token::UseAlias { alias: 7 }));
s.subscribe(t.ns.clone(), t.receipts.clone(), by_alias).await.expect("authorized by alias");
// A wrong token is refused even when it is well formed.
let wrong = Parameters::new().with(Parameter::AuthorizationToken(Token::UseValue { token_type: 0, value: b"not-it".to_vec() }));
assert!(s.subscribe(t.ns.clone(), t.media.clone(), wrong).await.is_err());
// A reference to an alias never registered closes the session (§8.9).
let (x, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let unknown = Parameters::new().with(Parameter::AuthorizationToken(Token::UseAlias { alias: 99 }));
assert!(x.subscribe(t.ns.clone(), t.media.clone(), unknown).await.is_err());
match timeout(STEP, x.conn.closed()).await.expect("closed in time") {
TransportError::Closed { code, .. } => assert_eq!(code, SessionError::UNKNOWN_AUTH_TOKEN_ALIAS.0),
other => panic!("{other:?}"),
}
s.close(0, "done");
p.close(0, "done");
server.shutdown().await;
}

444
joule-code/joule-code-rs/crates/joule-moq-relay/tests/common/mod.rs Normal file
View file

@ -0,0 +1,444 @@
//! What the relay's integration tests share: a relay on loopback, clients over each
//! transport, publishers that sign their groups, and subscribers that read and verify.
//!
//! Every network address is loopback, and every wait here and in the tests has a hard
//! bound, so no loop can outlive its test.
#![allow(dead_code)]
use ed25519_dalek::SigningKey;
use joule_moq::carriage::{object_properties, ObjectExtras, ReceiptBundle};
use joule_moq::draft22::data::{ObjectStatus as WireStatus, SubgroupHeader, SubgroupId, SubgroupObject};
use joule_moq::draft22::kvp::{prop, KeyValuePair, Properties};
use joule_moq::draft22::{ControlMessage, Parameters, StreamHeader, SubscribeOk, UniEvent};
use joule_moq::messages::ObjectHeader;
use joule_moq::receipt::{ObjectReceipt, ReceiptBuilder, ReceiptVerifier, VerifiedGroup};
use joule_moq::{GroupId, ObjectId, ObjectStatus, TrackAlias, TrackName, TrackNamespace};
use joule_moq_relay::client::{Client, ClientOptions};
use joule_moq_relay::transport::mux::MuxConfig;
use joule_moq_relay::transport::{quic, webtransport, websocket, TransportError};
use joule_moq_relay::wire::{ReqRx, ReqTx, StreamError, UniReader};
use joule_moq_relay::{RelayConfig, RelayServer};
use std::collections::{BTreeMap, HashMap};
use std::sync::Arc;
use std::time::Duration;
use tokio::time::{timeout, Instant};
/// The bound on any single wait in a test.
pub const STEP: Duration = Duration::from_secs(20);
pub fn loopback() -> std::net::SocketAddr {
"127.0.0.1:0".parse().unwrap()
}
/// A relay on loopback ports, the configuration adjusted by `f`.
pub async fn relay(f: impl FnOnce(&mut RelayConfig)) -> RelayServer {
let mut c = RelayConfig::loopback();
f(&mut c);
RelayServer::start(c).await.expect("the relay starts")
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Via {
Quic,
WebTransport,
WebSocket,
}
/// A client of `server` over `via`. A QUIC client keeps its endpoint, returned with it.
pub async fn connect(server: &RelayServer, via: Via, opts: ClientOptions) -> (Client, Option<quinn::Endpoint>) {
connect_with(server, via, opts, None).await
}
pub async fn connect_with(server: &RelayServer, via: Via, opts: ClientOptions, transport: Option<quinn::TransportConfig>) -> (Client, Option<quinn::Endpoint>) {
let quic_addr = server.quic_addr.expect("a UDP endpoint");
let path = server.relay().config().path.clone();
match via {
Via::Quic => {
let ep = quic::client_endpoint(loopback(), vec![server.identity.chain[0].clone()], vec![joule_moq::draft22::ALPN.to_vec()], transport).unwrap();
let conn = timeout(STEP, quic::connect(&ep, quic_addr, "localhost")).await.expect("in time").expect("a QUIC session");
let c = Client::connect(Arc::new(conn), opts).await.expect("SETUP over QUIC");
(c, Some(ep))
}
Via::WebTransport => {
let url = format!("https://127.0.0.1:{}{}", quic_addr.port(), path);
let conn = timeout(STEP, webtransport::connect(&url, Some(server.identity.leaf_sha256), loopback())).await.expect("in time").expect("a WebTransport session");
let c = Client::connect(Arc::new(conn), opts).await.expect("SETUP over WebTransport");
(c, None)
}
Via::WebSocket => {
let url = format!("ws://{}{}", server.websocket_addr.expect("a WebSocket listener"), path);
let conn = timeout(STEP, websocket::connect(&url, MuxConfig::default())).await.expect("in time").expect("a WebSocket session");
let c = Client::connect(Arc::new(conn), opts).await.expect("SETUP over the WebSocket binding");
(c, None)
}
}
}
pub fn ns(fields: &[&str]) -> TrackNamespace {
TrackNamespace::from_strs(fields.iter().copied())
}
pub fn name(s: &str) -> TrackName {
TrackName::from_str(s)
}
/// Track properties: a default publisher priority, and an OBJECT_DELIVERY_TIMEOUT.
pub fn track_props(priority: u8, object_timeout_ms: Option<u64>) -> Properties {
let mut p = Properties::new();
p.push(KeyValuePair::varint(prop::DEFAULT_PUBLISHER_PRIORITY, priority as u64));
if let Some(t) = object_timeout_ms {
p.push(KeyValuePair::varint(prop::OBJECT_DELIVERY_TIMEOUT, t));
}
p
}
/// One group of a media track, one object, and its receipts as group `g` of the receipts
/// track carries them (WAI moq-streaming-format §4.1).
pub fn signed_group(key: &SigningKey, receipt_alias: u64, g: u64, payload: &[u8]) -> (ObjectReceipt, ReceiptBundle) {
let hash = ObjectReceipt::hash_payload(payload);
let mut b = ReceiptBuilder::new(TrackAlias(receipt_alias), GroupId(g));
b.add_labelled_object(hash, 0, "test-publisher", "2026-10-17T00:00:00Z", vec![0]).unwrap();
let receipt = ObjectReceipt {
track_alias: TrackAlias(receipt_alias),
group_id: GroupId(g),
object_id: ObjectId(0),
content_hash: hash,
joules_micro: 0,
origin: "test-publisher".into(),
merkle_proof: b.proof_for(0).unwrap(),
energy_class: Some(vec![0]),
};
let group = b.finalize_labelled(key, "test-publisher", "2026-10-17T00:00:01Z", None).unwrap();
(receipt.clone(), ReceiptBundle { group, objects: vec![receipt] })
}
/// A media track and its receipts track, as one publisher sends them.
#[derive(Clone)]
pub struct Tracks {
pub ns: TrackNamespace,
pub media: TrackName,
pub receipts: TrackName,
pub media_alias: u64,
pub receipts_alias: u64,
pub receipt_alias: u64,
pub priority: u8,
}
impl Tracks {
pub fn new(ns: TrackNamespace, media: &str, media_alias: u64, receipt_alias: u64, priority: u8) -> Self {
Tracks { ns, media: name(media), receipts: name(&format!("{media}/receipts")), media_alias, receipts_alias: media_alias + 1, receipt_alias, priority }
}
}
/// Publish group `g` of `t`: the receipts first (no later than the group's first object,
/// moq-streaming-format §4.1), then the object, which carries its receipt as a property.
pub async fn publish_group(c: &Client, key: &SigningKey, t: &Tracks, g: u64, payload: &[u8]) -> ObjectReceipt {
let (receipt, bundle) = signed_group(key, t.receipt_alias, g, payload);
let rh = SubgroupHeader { track_alias: t.receipts_alias, group_id: g, subgroup_id: SubgroupId::Zero, publisher_priority: Some(t.priority), properties: false, end_of_group: true, first_object: true };
c.send_subgroup(rh, &[SubgroupObject::normal(0, bundle.to_payload().unwrap())]).await.expect("the receipts");
let header = ObjectHeader {
track_alias: TrackAlias(t.media_alias),
group_id: GroupId(g),
object_id: ObjectId(0),
publisher_priority: t.priority,
status: ObjectStatus::Normal,
payload_len: payload.len() as u64,
mrl_layer: 0,
mrl_total_layers: 0,
wai_envelope: false,
};
let props = object_properties(&header, &ObjectExtras { receipt: Some(receipt.clone()), wai_receipt: None, delivery_timeout_ms: None }).unwrap();
let mh = SubgroupHeader { track_alias: t.media_alias, group_id: g, subgroup_id: SubgroupId::Zero, publisher_priority: Some(t.priority), properties: true, end_of_group: true, first_object: true };
c.send_subgroup(mh, &[SubgroupObject { object_id: 0, properties: props, status: WireStatus::Normal, payload: payload.to_vec() }]).await.expect("the object");
receipt
}
/// PUBLISH a media track and its receipts track (§9.8).
pub async fn publish_tracks(c: &Client, t: &Tracks, object_timeout_ms: Option<u64>) -> Vec<(ReqTx, ReqRx)> {
let a = c.publish(t.ns.clone(), t.media.clone(), t.media_alias, Parameters::new(), track_props(t.priority, object_timeout_ms)).await.expect("PUBLISH the media track");
let b = c.publish(t.ns.clone(), t.receipts.clone(), t.receipts_alias, Parameters::new(), track_props(t.priority, None)).await.expect("PUBLISH the receipts track");
vec![a, b]
}
/// Answer the relay's SUBSCRIBEs for `t`'s tracks (the namespace route, §7.6) until both
/// are Established; the streams are returned to keep the subscriptions open.
pub async fn answer_subscribes(c: &Client, t: &Tracks) -> Vec<(ReqTx, ReqRx)> {
let mut held = Vec::new();
let mut seen = (false, false);
let deadline = Instant::now() + STEP;
while !(seen.0 && seen.1) {
let left = deadline.saturating_duration_since(Instant::now());
let (tx, rx, first) = c.next_request(left).await.expect("the relay's SUBSCRIBE in time");
let ControlMessage::Subscribe(r) = first else { panic!("the relay sent {}", first.name()) };
let (alias, props) = if r.name == t.media {
seen.0 = true;
(t.media_alias, track_props(t.priority, None))
} else {
assert_eq!(r.name, t.receipts, "the relay subscribes only to the tracks asked for");
seen.1 = true;
(t.receipts_alias, track_props(t.priority, None))
};
tx.send(&ControlMessage::SubscribeOk(SubscribeOk { track_alias: alias, params: Parameters::new(), track_properties: props })).await.unwrap();
held.push((tx, rx));
}
held
}
/// What a subscriber read from one data stream.
#[derive(Debug)]
pub struct Got {
pub header: SubgroupHeader,
pub objects: Vec<SubgroupObject>,
/// The stream was reset, with this code.
pub reset: Option<u64>,
pub arrived: Instant,
}
/// Read one subgroup stream to its end, each read bounded.
pub async fn read_stream(mut r: UniReader) -> Option<Got> {
let arrived = Instant::now();
let StreamHeader::Subgroup(header) = r.header.clone()? else { return None };
let mut objects = Vec::new();
loop {
match timeout(STEP, r.next()).await {
Ok(Ok(Some(UniEvent::SubgroupObject(o)))) => objects.push(o),
Ok(Ok(Some(_))) => return None,
Ok(Ok(None)) => return Some(Got { header, objects, reset: None, arrived }),
Ok(Err(StreamError::Transport(TransportError::Reset(code)))) => return Some(Got { header, objects, reset: Some(code), arrived }),
Ok(Err(e)) => panic!("a data stream: {e}"),
Err(_) => panic!("a data stream stalled"),
}
}
}
/// Read data streams until `done` holds of what was read, or `limit` passes.
pub async fn collect(c: &Client, limit: Duration, mut done: impl FnMut(&[Got]) -> bool) -> Vec<Got> {
let deadline = Instant::now() + limit;
let mut out = Vec::new();
while !done(&out) {
let left = deadline.saturating_duration_since(Instant::now());
if left.is_zero() {
break;
}
match c.next_data(left).await {
Some(r) => {
if let Some(g) = read_stream(r).await {
out.push(g);
}
}
None => break,
}
}
out
}
/// Read every data stream at once, each in a task of its own, as a subscriber does that
/// reads its streams concurrently: until `done` holds of what was read and `quiet` has
/// passed with no new stream or object, or `limit` passes. Each read is bounded.
///
/// Reading one stream at a time in the order they arrive can stall: QUIC's
/// connection-level flow control is shared by a session's streams, so the bytes of the
/// stream being waited on can wait for credit that unread later streams hold.
pub async fn collect_concurrent(c: &Client, limit: Duration, quiet: Duration, mut done: impl FnMut(&[Got]) -> bool) -> Vec<Got> {
let deadline = Instant::now() + limit;
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<Got>();
let mut out = Vec::new();
let mut readers = Vec::new();
let mut last = Instant::now();
loop {
let now = Instant::now();
if now >= deadline || (done(&out) && now.duration_since(last) >= quiet) {
break;
}
let tick = deadline.min(now + quiet).saturating_duration_since(now);
tokio::select! {
s = c.next_stream(tick) => {
if let Some(mut r) = s {
last = Instant::now();
let tx = tx.clone();
readers.push(tokio::spawn(async move {
if let Ok(Ok(_)) = timeout(STEP, r.start()).await
&& let Some(g) = read_stream(r).await
{
let _ = tx.send(g);
}
}));
}
}
g = rx.recv() => {
if let Some(g) = g {
last = Instant::now();
out.push(g);
}
}
}
}
for r in readers {
r.abort();
}
while let Ok(g) = rx.try_recv() {
out.push(g);
}
out
}
/// A subscriber's view of one publisher's tracks: the downstream aliases the relay gave.
pub struct Subscribed {
pub media_alias: u64,
pub receipts_alias: u64,
pub held: Vec<(ReqTx, ReqRx)>,
}
pub async fn subscribe_both(c: &Client, t: &Tracks, media_params: Parameters) -> Subscribed {
subscribe_both_with(c, t, media_params, Parameters::new()).await
}
pub async fn subscribe_both_with(c: &Client, t: &Tracks, media_params: Parameters, receipts_params: Parameters) -> Subscribed {
let (m, mtx, mrx) = c.subscribe(t.ns.clone(), t.media.clone(), media_params).await.expect("SUBSCRIBE the media track");
let (r, rtx, rrx) = c.subscribe(t.ns.clone(), t.receipts.clone(), receipts_params).await.expect("SUBSCRIBE the receipts track");
Subscribed { media_alias: m.track_alias, receipts_alias: r.track_alias, held: vec![(mtx, mrx), (rtx, rrx)] }
}
/// Every media object of `sub` verified against the bundles of its receipts track, under
/// the publisher's key alone. Returns (objects verified, bundles verified, the verified
/// groups by Group ID, the object receipts by Group ID).
pub fn verify(got: &[Got], sub: &Subscribed, publisher: &SigningKey) -> (usize, usize, BTreeMap<u64, VerifiedGroup>, HashMap<u64, ObjectReceipt>) {
let verifier = ReceiptVerifier::with_anchors(vec![publisher.verifying_key()]);
let mut groups = BTreeMap::new();
let mut receipts = HashMap::new();
for g in got.iter().filter(|g| g.header.track_alias == sub.receipts_alias && g.reset.is_none()) {
let b = ReceiptBundle::from_payload(&g.objects[0].payload).expect("a bundle");
let vg = verifier.verify_group(&b.group).expect("the group receipt verifies under the publisher's key");
assert_eq!(b.group.group_id, GroupId(g.header.group_id), "the receipt names its MOQT group");
groups.insert(g.header.group_id, vg);
receipts.insert(g.header.group_id, b.objects[0].clone());
}
let mut objects = 0;
for g in got.iter().filter(|g| g.header.track_alias == sub.media_alias && g.reset.is_none()) {
for o in &g.objects {
let vg = groups.get(&g.header.group_id).expect("the receipts of every delivered group arrived");
let r = &receipts[&g.header.group_id];
verifier.verify_payload(&o.payload, r, vg).expect("the payload verifies against its signed root");
objects += 1;
}
}
(objects, groups.len(), groups, receipts)
}
pub fn payload(tag: &str, g: u64, len: usize) -> Vec<u8> {
let mut v = format!("{tag}:{g}:").into_bytes();
v.resize(len.max(v.len()), (g % 251) as u8);
v
}
/// Wait, bounded, until `f` holds.
pub async fn until(limit: Duration, mut f: impl FnMut() -> bool) -> bool {
let deadline = Instant::now() + limit;
while !f() {
if Instant::now() >= deadline {
return false;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
true
}
/// The scenario every transport runs: two publishers, one reached by the namespace route
/// (PUBLISH_NAMESPACE, the relay's SUBSCRIBE) and one by PUBLISH, and three subscribers
/// of both, all over `via`. Every object reaches every subscriber and verifies against its
/// publisher's signed receipts, which the relay also checks at the door; one subscriber
/// discovers the first namespace by SUBSCRIBE_NAMESPACE; every session's summary is signed.
pub async fn two_publishers_three_subscribers(via: Via) {
let key_a = SigningKey::from_bytes(&[0xa1; 32]);
let key_b = SigningKey::from_bytes(&[0xb2; 32]);
let (ka, kb) = (hex::encode(key_a.verifying_key().to_bytes()), hex::encode(key_b.verifying_key().to_bytes()));
let server = relay(|c| {
c.receipts.door = vec![joule_moq_relay::config::DoorRule { prefix: vec!["live".into()], trust: vec![ka, kb], hold_ms: 5_000 }];
})
.await;
const GROUPS: u64 = 12;
let ta = Tracks::new(ns(&["live", "a"]), "cam", 10, 8, 64);
let tb = Tracks::new(ns(&["live", "b"]), "cam", 20, 9, 96);
let (pa, _ea) = connect(&server, via, ClientOptions::default()).await;
let (pb, _eb) = connect(&server, via, ClientOptions::default()).await;
let _ns_a = pa.publish_namespace(ta.ns.clone(), Parameters::new()).await.expect("PUBLISH_NAMESPACE");
let _pub_b = publish_tracks(&pb, &tb, None).await;
// One subscriber discovers live/a by prefix (§4.2).
let (s0, _e0) = connect(&server, via, ClientOptions::default()).await;
let (_nstx, mut nsrx) = s0.subscribe_namespace(ns(&["live"])).await.expect("SUBSCRIBE_NAMESPACE");
match timeout(STEP, nsrx.next()).await.expect("NAMESPACE in time").expect("a message") {
Some(ControlMessage::Namespace(suffix)) => assert_eq!(suffix, ns(&["a"]), "NAMESPACE carries the suffix after the prefix"),
other => panic!("expected NAMESPACE, got {other:?}"),
}
let mut subs = vec![s0];
let mut eps = Vec::new();
for _ in 0..2 {
let (s, e) = connect(&server, via, ClientOptions::default()).await;
subs.push(s);
eps.push(e);
}
// The first SUBSCRIBE to live/a makes the relay subscribe upstream; publisher A answers.
let answering = {
let ta = ta.clone();
let pa_ref = &pa;
async move { answer_subscribes(pa_ref, &ta).await }
};
let s_first = &subs[0];
let (held_a, first_sub) = tokio::join!(answering, subscribe_both(s_first, &ta, Parameters::new()));
let mut views: Vec<(Subscribed, Subscribed)> = vec![(first_sub, subscribe_both(&subs[0], &tb, Parameters::new()).await)];
for s in &subs[1..] {
views.push((subscribe_both(s, &ta, Parameters::new()).await, subscribe_both(s, &tb, Parameters::new()).await));
}
// Both publish, concurrently.
let pub_a = async {
for g in 0..GROUPS {
publish_group(&pa, &key_a, &ta, g, &payload("a", g, 900)).await;
}
};
let pub_b = async {
for g in 0..GROUPS {
publish_group(&pb, &key_b, &tb, g, &payload("b", g, 1400)).await;
}
};
let reads = futures::future::join_all(subs.iter().map(|s| collect(s, STEP * 2, |got| got.len() as u64 >= 4 * GROUPS)));
let ((), (), got) = tokio::join!(pub_a, pub_b, reads);
for (i, (g, (va, vb))) in got.iter().zip(&views).enumerate() {
let (oa, ba, _, _) = verify(g, va, &key_a);
let (ob, bb, _, _) = verify(g, vb, &key_b);
assert_eq!((oa, ba, ob, bb), (GROUPS as usize, GROUPS as usize, GROUPS as usize, GROUPS as usize), "subscriber {i}: every object and every receipt of both publishers");
for x in g.iter().filter(|x| x.header.track_alias == va.media_alias) {
assert_eq!(x.objects[0].payload, payload("a", x.header.group_id, 900));
}
}
assert_eq!(server.relay().stats().refused_at_door, 0);
drop(held_a);
// Every session ends with a signed summary.
for s in &subs {
s.close(0, "done");
}
pa.close(0, "done");
pb.close(0, "done");
assert!(until(STEP, || server.relay().summaries().len() >= 5).await, "five summaries");
for sum in server.relay().summaries() {
sum.verify().expect("the summary verifies under the relay's key");
assert_eq!(sum.summary.relay_energy.acquisition, "unmetered");
assert_eq!(sum.summary.relay_energy.joules_micro, 0, "0 is the unmetered marker");
assert_eq!(sum.summary.transport, match via { Via::Quic => "quic", Via::WebTransport => "webtransport", Via::WebSocket => "websocket" });
}
let as_subscriber: Vec<_> = server.relay().summaries().into_iter().filter(|s| s.summary.tally.subscriber.objects_forwarded > 0).collect();
assert_eq!(as_subscriber.len(), 3);
for s in &as_subscriber {
assert_eq!(s.summary.tally.subscriber.objects_forwarded, 4 * GROUPS, "media and receipts of both publishers");
assert_eq!(s.summary.tally.subscriber.dropped_delivery_timeout, 0);
let signed = s.summary.tally.publisher_signed.get("unmetered").expect("the door verified the media objects' receipts");
assert_eq!(signed.objects, 2 * GROUPS, "the publishers' signed figures, unmetered, for the media objects");
}
server.shutdown().await;
}

77
joule-code/joule-code-rs/crates/joule-moq-relay/tests/goaway.rs Normal file
View file

@ -0,0 +1,77 @@
//! GOAWAY on shutdown (`draft-ietf-moq-transport-22` §6.6.1, §9.2): every session is told,
//! with the New Session URI and the Timeout; new requests are refused with GOING_AWAY; a
//! client that leaves is let go, and one that stays is closed with GOAWAY_TIMEOUT.
mod common;
use common::*;
use joule_moq::draft22::{Parameters, RequestErrorCode, SessionError};
use joule_moq_relay::client::{Client, ClientOptions};
use joule_moq_relay::transport::TransportError;
use std::time::Duration;
use tokio::time::timeout;
const URI: &str = "moqt://127.0.0.1:4999/moq";
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn goaway_on_shutdown() {
timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time");
}
async fn goaway_of(c: &Client) -> joule_moq::draft22::Goaway {
let mut w = c.goaway();
timeout(STEP, async {
loop {
if let Some(g) = w.borrow_and_update().clone() {
return g;
}
w.changed().await.expect("the session lives until its GOAWAY");
}
})
.await
.expect("a GOAWAY in time")
}
async fn run() {
let server = relay(|c| {
c.limits.goaway_timeout_ms = 1500;
c.goaway_uri = URI.into();
})
.await;
let t = Tracks::new(ns(&["live", "g"]), "cam", 10, 4, 128);
let (p, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await;
let _pub = publish_tracks(&p, &t, None).await;
let (a, _ea) = connect(&server, Via::Quic, ClientOptions::default()).await;
let (b, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let _sa = a.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.unwrap();
let _sb = b.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.unwrap();
let relay = server.relay().clone();
let shutdown = tokio::spawn(async move { server.shutdown().await });
for c in [&p, &a, &b] {
let g = goaway_of(c).await;
assert_eq!(g.new_session_uri, URI.as_bytes(), "the New Session URI");
assert_eq!(g.timeout_ms, 1500, "the Timeout the relay will wait");
}
// A new request after GOAWAY is refused (§9.2: "MAY reject new requests").
let refused = b.subscribe(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.expect_err("a new SUBSCRIBE after GOAWAY");
assert!(refused.contains(&RequestErrorCode::GOING_AWAY.to_string()), "{refused}");
// The publisher and one subscriber migrate and close with NO_ERROR; the other stays.
a.close(SessionError::NO_ERROR.0, "migrated");
p.close(SessionError::NO_ERROR.0, "migrated");
match timeout(STEP, b.conn.closed()).await.expect("closed in time") {
TransportError::Closed { code, by_peer, .. } => {
assert_eq!(code, SessionError::GOAWAY_TIMEOUT.0, "the relay closes a session that stayed with GOAWAY_TIMEOUT");
assert!(by_peer);
}
other => panic!("{other:?}"),
}
let closed = timeout(STEP, shutdown).await.expect("the shutdown ends").unwrap();
assert_eq!(closed, 1, "one session outlived the GOAWAY timeout");
assert!(until(STEP, || relay.summaries().len() == 3).await);
let codes: Vec<Option<u64>> = relay.summaries().iter().map(|s| s.summary.close_code).collect();
assert_eq!(codes.iter().filter(|c| **c == Some(SessionError::GOAWAY_TIMEOUT.0)).count(), 1, "{codes:?}");
assert_eq!(codes.iter().filter(|c| **c == Some(SessionError::NO_ERROR.0)).count(), 2, "{codes:?}");
}

210
joule-code/joule-code-rs/crates/joule-moq-relay/tests/late_joiner.rs Normal file
View file

@ -0,0 +1,210 @@
//! A late joiner served from the relay's cache (`draft-ietf-moq-transport-22` §3.2, §3.4,
//! §7.1), and a FETCH the relay forwards upstream when its cache holds nothing (§3.2.3).
mod common;
use common::*;
use joule_moq::carriage::ReceiptBundle;
use joule_moq::draft22::data::{FetchCodec, FetchEntry, FetchHeader, FetchObject};
use joule_moq::draft22::{
ControlMessage, FetchOk, GroupOrder, Location, LocationFilter, Parameter, Parameters, RequestErrorCode, StreamHeader, TrackRequest, UniEvent,
};
use joule_moq::receipt::ReceiptVerifier;
use joule_moq_relay::client::{Client, ClientOptions};
use std::time::Duration;
use tokio::time::timeout;
const GROUPS: u64 = 10;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_late_joiner_is_served_from_the_cache() {
timeout(Duration::from_secs(120), from_cache()).await.expect("the test finishes in time");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_group_that_arrives_after_a_later_one_is_still_served_from_the_cache() {
timeout(Duration::from_secs(120), reordered()).await.expect("the test finishes in time");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_fetch_the_cache_cannot_serve_goes_upstream() {
timeout(Duration::from_secs(120), from_upstream()).await.expect("the test finishes in time");
}
fn objects(entries: &[FetchEntry]) -> Vec<&FetchObject> {
entries
.iter()
.filter_map(|e| match e {
FetchEntry::Object(o) => Some(o),
_ => None,
})
.collect()
}
async fn from_cache() {
let server = relay(|_| {}).await;
let key = ed25519_dalek::SigningKey::from_bytes(&[0x1a; 32]);
let t = Tracks::new(ns(&["vod", "show"]), "ep1", 10, 3, 128);
let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await;
let _held = publish_tracks(&publisher, &t, None).await;
for g in 0..GROUPS {
publish_group(&publisher, &key, &t, g, &payload("ep1", g, 2000)).await;
}
// The cache, not the count of objects received, says when it holds the track: an
// object counted on arrival may not be cached yet, and streams finish out of order.
let relay = server.relay().clone();
let (mk, rk) = ((t.ns.clone(), t.media.clone()), (t.ns.clone(), t.receipts.clone()));
assert!(
until(STEP, || relay.largest(&mk) == Some(Location::new(GROUPS - 1, 0)) && relay.cache_covers(&mk, Location::new(0, 0)) && relay.cache_covers(&rk, Location::new(0, 0))).await,
"the cache holds the track and its receipts from group 0"
);
// The late joiner comes over the WebSocket fallback.
let (late, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
// TRACK_STATUS names the Largest Object (§9.13).
let m = ControlMessage::TrackStatus(TrackRequest { request_id: late.next_request_id(), namespace: t.ns.clone(), name: t.media.clone(), params: Parameters::new() });
let (_tx, mut rx) = late.request(&m).await.unwrap();
match Client::response(&mut rx).await.unwrap() {
ControlMessage::RequestOk(ok) => assert_eq!(ok.params.largest_object(), Some(Location::new(GROUPS - 1, 0))),
other => panic!("{other:?}"),
}
// The whole track and its receipts, by FETCH, from the cache.
let (ok, media) = late.fetch(t.ns.clone(), t.media.clone(), Parameters::new()).await.expect("FETCH the media track");
assert_eq!(ok.end_location, Location::new(GROUPS - 1, 0));
let (_, rec) = late.fetch(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.expect("FETCH the receipts track");
let media = objects(&media);
let rec = objects(&rec);
assert_eq!((media.len(), rec.len()), (GROUPS as usize, GROUPS as usize));
let verifier = ReceiptVerifier::with_anchors(vec![key.verifying_key()]);
for (m, r) in media.iter().zip(&rec) {
assert_eq!((m.group_id, r.group_id), (r.group_id, m.group_id));
let b = ReceiptBundle::from_payload(&r.payload).unwrap();
let vg = verifier.verify_group(&b.group).expect("a receipt from the cache verifies");
verifier.verify_payload(&m.payload, &b.objects[0], &vg).expect("an object from the cache verifies");
assert_eq!(m.payload, payload("ep1", m.group_id, 2000));
assert_eq!(m.publisher_priority, 128);
}
assert_eq!(media.iter().map(|o| o.group_id).collect::<Vec<_>>(), (0..GROUPS).collect::<Vec<_>>(), "ascending by default (§9.20.8)");
// A range, groups descending (§3.2.1).
let range = Parameters::new()
.with(Parameter::LocationFilter(LocationFilter::AbsoluteRange { start: Location::new(3, 0), end_group_delta: 4, end_object: 0 }))
.with(Parameter::GroupOrder(GroupOrder::Descending));
let (ok, part) = late.fetch(t.ns.clone(), t.media.clone(), range).await.expect("FETCH a range");
assert_eq!(ok.end_location, Location::new(7, 0));
assert_eq!(objects(&part).iter().map(|o| o.group_id).collect::<Vec<_>>(), vec![7, 6, 5, 4, 3]);
// A fetch that starts after the Largest Object is refused (§3.2: INVALID_RANGE).
let after = Parameters::new().with(Parameter::LocationFilter(LocationFilter::AbsoluteStart { start: Location::new(GROUPS + 5, 0) }));
let refused = late.fetch(t.ns.clone(), t.media.clone(), after).await.expect_err("a range past the Largest Object");
assert!(refused.contains(&RequestErrorCode::INVALID_RANGE.to_string()), "{refused}");
// Joining at a past group: SUBSCRIBE from the Next Object with a fill of the last three
// groups (§3.5), delivered on a fill fetch stream named by the SUBSCRIBE's Request ID.
let fill = Parameters::new().with(Parameter::LocationFilter(LocationFilter::RelativeStart { start_group: 3 }));
let id_before = late.next_request_id();
let params = Parameters::new().with(Parameter::LocationFilter(LocationFilter::NextObject)).with(Parameter::FillParameters(fill));
let (ok, _stx, _srx) = late.subscribe(t.ns.clone(), t.media.clone(), params).await.expect("SUBSCRIBE with a fill");
assert_eq!(ok.params.largest_object(), Some(Location::new(GROUPS - 1, 0)));
let subscribe_id = id_before + 2;
let mut r = late.next_data(STEP).await.expect("the fill fetch stream");
match r.header.clone() {
Some(StreamHeader::Fetch(FetchHeader { request_id })) => assert_eq!(request_id, subscribe_id),
other => panic!("expected a fill fetch stream, got {other:?}"),
}
let mut filled = Vec::new();
while let Some(UniEvent::FetchEntry(e)) = timeout(STEP, r.next()).await.expect("in time").expect("a fill entry") {
filled.push(e);
}
assert_eq!(objects(&filled).iter().map(|o| o.group_id).collect::<Vec<_>>(), vec![GROUPS - 3, GROUPS - 2, GROUPS - 1]);
// All of it from the cache: the publisher was asked for nothing.
assert!(relay.stats().fetch_served_from_cache >= 4);
assert_eq!(relay.stats().fetch_served_upstream, 0);
assert!(publisher.next_request(Duration::from_millis(300)).await.is_none(), "the relay sent the publisher no request");
late.close(0, "done");
publisher.close(0, "done");
server.shutdown().await;
}
/// Group 1 reaches the relay whole before group 0 does, as streams finish out of order on a
/// loaded host. The cache still serves a FETCH from group 0: coverage is the run of whole
/// groups, not wherever the first object kept happened to fall.
async fn reordered() {
let server = relay(|_| {}).await;
let key = ed25519_dalek::SigningKey::from_bytes(&[0x2b; 32]);
let t = Tracks::new(ns(&["vod", "reorder"]), "ep", 10, 7, 128);
let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await;
let _held = publish_tracks(&publisher, &t, None).await;
let relay = server.relay().clone();
let (mk, rk) = ((t.ns.clone(), t.media.clone()), (t.ns.clone(), t.receipts.clone()));
// Group 1 first: its receipts stream and its object's stream, each to its FIN.
publish_group(&publisher, &key, &t, 1, &payload("ep", 1, 1500)).await;
assert!(until(STEP, || relay.largest(&mk) == Some(Location::new(1, 0))).await, "the relay has group 1");
// Then group 0, then the rest in order.
for g in [0, 2, 3, 4, 5] {
publish_group(&publisher, &key, &t, g, &payload("ep", g, 1500)).await;
}
assert!(
until(STEP, || relay.largest(&mk) == Some(Location::new(5, 0)) && relay.cache_covers(&mk, Location::new(0, 0)) && relay.cache_covers(&rk, Location::new(0, 0))).await,
"the cache covers the track from group 0, though group 0 came after group 1"
);
let (late, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let (ok, media) = late.fetch(t.ns.clone(), t.media.clone(), Parameters::new()).await.expect("FETCH from group 0");
assert_eq!(ok.end_location, Location::new(5, 0));
let (_, rec) = late.fetch(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.expect("FETCH its receipts");
let (media, rec) = (objects(&media), objects(&rec));
assert_eq!(media.iter().map(|o| o.group_id).collect::<Vec<_>>(), (0..=5).collect::<Vec<_>>());
let verifier = ReceiptVerifier::with_anchors(vec![key.verifying_key()]);
for (m, r) in media.iter().zip(&rec) {
let b = ReceiptBundle::from_payload(&r.payload).unwrap();
let vg = verifier.verify_group(&b.group).unwrap();
verifier.verify_payload(&m.payload, &b.objects[0], &vg).expect("each object verifies");
}
assert_eq!(relay.stats().fetch_served_upstream, 0, "nothing went upstream");
assert!(relay.stats().fetch_served_from_cache >= 2);
assert!(publisher.next_request(Duration::from_millis(300)).await.is_none(), "the relay sent the publisher no request");
late.close(0, "done");
publisher.close(0, "done");
server.shutdown().await;
}
async fn from_upstream() {
let server = relay(|_| {}).await;
let space = ns(&["archive"]);
let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await;
let _ns = publisher.publish_namespace(space.clone(), Parameters::new()).await.unwrap();
let (late, _el) = connect(&server, Via::Quic, ClientOptions::default()).await;
let asking = late.fetch(space.clone(), name("old"), Parameters::new());
let answering = async {
let (tx, _rx, first) = publisher.next_request(STEP).await.expect("the relay's upstream FETCH");
let ControlMessage::Fetch(r) = first else { panic!("expected FETCH, got {}", first.name()) };
assert_eq!(r.name, name("old"));
let end = Location::new(2, 0);
tx.send(&ControlMessage::FetchOk(FetchOk { end_of_track: true, end_location: end, params: Parameters::new(), track_properties: Default::default() })).await.unwrap();
let mut b = Vec::new();
FetchHeader { request_id: r.request_id }.encode(&mut b);
let mut codec = FetchCodec::new(r.params.group_order().unwrap_or(GroupOrder::Ascending));
for g in 0..=2u64 {
codec.encode(&FetchEntry::Object(FetchObject { group_id: g, object_id: 0, subgroup_id: Some(0), publisher_priority: 7, properties: None, payload: payload("old", g, 300) }), &mut b).unwrap();
}
let mut s = publisher.conn.open_uni().await.unwrap();
s.write_all(&b).await.unwrap();
s.finish().unwrap();
tx
};
let (got, _tx) = tokio::join!(asking, answering);
let (ok, entries) = got.expect("the FETCH answered through the relay");
assert!(ok.end_of_track);
let objs = objects(&entries);
assert_eq!(objs.iter().map(|o| (o.group_id, o.payload.clone())).collect::<Vec<_>>(), (0..=2u64).map(|g| (g, payload("old", g, 300))).collect::<Vec<_>>());
assert_eq!(server.relay().stats().fetch_served_upstream, 1);
late.close(0, "done");
publisher.close(0, "done");
server.shutdown().await;
}

146
joule-code/joule-code-rs/crates/joule-moq-relay/tests/priority_fan_out.rs Normal file
View file

@ -0,0 +1,146 @@
//! Fan-out order under load (`draft-ietf-moq-transport-22` §5.1.2): two publishers send
//! four tracks of different publisher priorities to three subscribers, each subscribing
//! with its own subscriber priorities and group orders, while the subscribers' transports
//! are held shut. When they open, every object reaches every subscriber in the order
//! §5.1.2 gives: subscriber priority, then publisher priority, then group order.
//!
//! The sessions run over the in-process multiplexer, the WebSocket binding's own, with a
//! gate on what the relay sends each subscriber; the relay writes one object at a time.
//! Each loop here is bounded by a hard timeout.
mod common;
use common::*;
use joule_moq::draft22::data::{SubgroupHeader, SubgroupId, SubgroupObject};
use joule_moq::draft22::{GroupOrder, Parameter, Parameters};
use joule_moq_relay::client::{Client, ClientOptions};
use joule_moq_relay::transport::mux::{memory_pair, Gate, MuxConfig};
use joule_moq_relay::{Relay, RelayConfig};
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use tokio::time::timeout;
const GROUPS: u64 = 25;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn fan_out_follows_subscriber_then_publisher_priority_then_group_order() {
timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time");
}
async fn attach(relay: &Relay, gated: bool, opts: ClientOptions) -> (Client, Option<Gate>) {
let mux = MuxConfig { send_high_water: 1, ..MuxConfig::default() };
let (c, s, gate) = memory_pair(mux, gated);
if let Some(g) = &gate {
g.open();
}
tokio::spawn(joule_moq_relay::session::serve(relay.clone(), Arc::new(s)));
let client = timeout(STEP, Client::connect(Arc::new(c), opts)).await.expect("in time").expect("SETUP in memory");
(client, gate)
}
/// A track: its name, its publisher's alias for it, its publisher priority.
struct T {
name: &'static str,
alias: u64,
priority: u8,
}
const TRACKS: [T; 4] = [
T { name: "t1", alias: 1, priority: 0 },
T { name: "t2", alias: 2, priority: 50 },
T { name: "t3", alias: 3, priority: 0 },
T { name: "t4", alias: 4, priority: 200 },
];
/// Each subscriber's (subscriber priority, group order) for t1..t4.
fn plan(i: usize) -> [(u8, GroupOrder); 4] {
use GroupOrder::*;
match i {
0 => [(10, Ascending), (10, Ascending), (200, Descending), (5, Ascending)],
1 => [(128, Descending), (128, Ascending), (128, Ascending), (128, Descending)],
_ => [(100, Ascending), (100, Descending), (0, Descending), (100, Ascending)],
}
}
/// Every group of the tracks `js`, one object to a group, each send bounded.
async fn send(c: &Client, js: &[usize]) {
for g in 0..GROUPS {
for &j in js {
let t = &TRACKS[j];
let h = SubgroupHeader { track_alias: t.alias, group_id: g, subgroup_id: SubgroupId::Zero, publisher_priority: Some(t.priority), properties: false, end_of_group: true, first_object: true };
timeout(STEP, c.send_subgroup(h, &[SubgroupObject::normal(0, payload(t.name, g, 200))])).await.expect("a send in time").expect("a send");
}
}
}
async fn run() {
let mut cfg = RelayConfig::loopback();
cfg.quic_listen = None;
cfg.websocket_listen = None;
cfg.limits.write_concurrency = 1;
let relay = Relay::new(cfg).unwrap();
let space = ns(&["load"]);
let (p1, _) = attach(&relay, false, ClientOptions::default()).await;
let (p2, _) = attach(&relay, false, ClientOptions::default()).await;
let pubs = [&p1, &p1, &p2, &p2];
let mut held = Vec::new();
for (t, p) in TRACKS.iter().zip(pubs) {
held.push(p.publish(space.clone(), name(t.name), t.alias, Parameters::new(), track_props(t.priority, None)).await.expect("PUBLISH"));
}
// Three subscribers; each read in arrival order (lazily, so nothing reorders them).
let mut subs = Vec::new();
for i in 0..3 {
let (c, gate) = attach(&relay, true, ClientOptions { lazy_data: true, ..ClientOptions::default() }).await;
let mut keys: HashMap<u64, (u8, u8, GroupOrder)> = HashMap::new();
for (j, t) in TRACKS.iter().enumerate() {
let (sp, order) = plan(i)[j];
let params = Parameters::new().with(Parameter::SubscriberPriority(sp)).with(Parameter::GroupOrder(order));
let (ok, tx, rx) = c.subscribe(space.clone(), name(t.name), params).await.expect("SUBSCRIBE");
held.push((tx, rx));
keys.insert(ok.track_alias, (sp, t.priority, order));
}
subs.push((c, gate.expect("a gate"), keys));
}
for (_, g, _) in &subs {
g.close();
}
// Load: both publishers send every group of their tracks, concurrently.
tokio::join!(send(&p1, &[0, 1]), send(&p2, &[2, 3]));
let total = GROUPS * TRACKS.len() as u64;
assert!(until(STEP, || relay.stats().objects_in >= total).await, "the relay received every object");
// Settle: everything received has reached the subscribers' schedulers.
let mut stable = 0;
assert!(
until(STEP, || {
stable = if relay.stats().queued == 0 { stable + 1 } else { 0 };
stable >= 5
})
.await
);
for (_, g, _) in &subs {
g.open();
}
let reads = futures::future::join_all(subs.iter().map(|(c, _, _)| collect(c, STEP, |got| got.len() as u64 >= total))).await;
for (i, (got, (_, _, keys))) in reads.iter().zip(&subs).enumerate() {
assert_eq!(got.len() as u64, total, "subscriber {i}: every object");
let key = |x: &Got| {
let (sp, pp, order) = keys[&x.header.track_alias];
let rank = match order {
GroupOrder::Ascending => x.header.group_id,
GroupOrder::Descending => u64::MAX - x.header.group_id,
};
(sp, pp, rank)
};
// At most two objects were handed over before the gate shut: one in the pipe and
// one write in flight. Everything after them waited in the scheduler.
let ordered: Vec<_> = got.iter().skip(2).map(key).collect();
let out_of_order = ordered.windows(2).filter(|w| w[0] > w[1]).count();
assert_eq!(out_of_order, 0, "subscriber {i}: fan-out in §5.1.2 order: {:?}", got.iter().map(|x| (x.header.track_alias, x.header.group_id)).collect::<Vec<_>>());
}
drop(held);
}

10
joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_quic.rs Normal file
View file

@ -0,0 +1,10 @@
//! Two publishers and three subscribers through the relay over native QUIC (ALPN moqt-22).
mod common;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn two_publishers_three_subscribers_over_quic() {
tokio::time::timeout(std::time::Duration::from_secs(120), common::two_publishers_three_subscribers(common::Via::Quic))
.await
.expect("the scenario finishes in time");
}

11
joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_websocket.rs Normal file
View file

@ -0,0 +1,11 @@
//! Two publishers and three subscribers through the relay over the WebSocket fallback
//! (`draft-ietf-webtrans-http2-15` capsules, one to each message).
mod common;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn two_publishers_three_subscribers_over_websocket() {
tokio::time::timeout(std::time::Duration::from_secs(120), common::two_publishers_three_subscribers(common::Via::WebSocket))
.await
.expect("the scenario finishes in time");
}

12
joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_webtransport.rs Normal file
View file

@ -0,0 +1,12 @@
//! Two publishers and three subscribers through the relay over WebTransport over HTTP/3,
//! the client trusting the relay's certificate by its SHA-256 as a browser does with
//! `serverCertificateHashes`.
mod common;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn two_publishers_three_subscribers_over_webtransport() {
tokio::time::timeout(std::time::Duration::from_secs(120), common::two_publishers_three_subscribers(common::Via::WebTransport))
.await
.expect("the scenario finishes in time");
}

62
joule-code/joule-code-rs/crates/joule-moq-relay/tests/request_ids.rs Normal file
View file

@ -0,0 +1,62 @@
//! Request IDs and requests in flight (`draft-ietf-moq-transport-22` §6.4.2): a Request ID
//! of the server's parity, or one seen before, closes the session with INVALID_REQUEST_ID;
//! and the requests a client may have open at once are bounded by the relay's stream limit
//! (here WT_MAX_STREAMS on the WebSocket binding), credit returning as requests end.
mod common;
use common::*;
use joule_moq::draft22::{ControlMessage, Parameters, SessionError, StreamResetCode, TrackRequest};
use joule_moq_relay::client::ClientOptions;
use joule_moq_relay::transport::TransportError;
use std::time::Duration;
use tokio::time::timeout;
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn request_ids_and_requests_in_flight() {
timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time");
}
fn track_status(id: u64) -> ControlMessage {
ControlMessage::TrackStatus(TrackRequest { request_id: id, namespace: ns(&["x"]), name: name("y"), params: Parameters::new() })
}
async fn closed_with(c: &joule_moq_relay::client::Client) -> u64 {
match timeout(STEP, c.conn.closed()).await.expect("closed in time") {
TransportError::Closed { code, .. } => code,
other => panic!("{other:?}"),
}
}
async fn run() {
let server = relay(|c| c.limits.max_requests = 2).await;
// A Request ID with the server's parity (odd).
let (a, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let _r = a.request(&track_status(1)).await.unwrap();
assert_eq!(closed_with(&a).await, SessionError::INVALID_REQUEST_ID.0);
// A Request ID used twice.
let (b, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let (_t1, mut r1) = b.request(&track_status(0)).await.unwrap();
let _ = joule_moq_relay::client::Client::response(&mut r1).await;
let _r2 = b.request(&track_status(0)).await.unwrap();
assert_eq!(closed_with(&b).await, SessionError::INVALID_REQUEST_ID.0);
// Two requests open at once; a third waits for credit until one of them ends.
let t = Tracks::new(ns(&["live", "r"]), "cam", 10, 6, 128);
let (p, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await;
let _pub = publish_tracks(&p, &t, None).await;
let (c, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await;
let (_ok1, tx1, _rx1) = c.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.unwrap();
let (_ok2, _tx2, _rx2) = c.subscribe(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.unwrap();
assert!(timeout(Duration::from_millis(400), c.conn.open_bi()).await.is_err(), "a third request opened past the stream limit");
// Cancelling the first returns its credit (§6.4.2.3).
tx1.reset(StreamResetCode::CANCELLED.0).await;
let third = timeout(STEP, c.conn.open_bi()).await;
assert!(matches!(third, Ok(Ok(_))), "the third request opened once the first ended");
c.close(0, "done");
p.close(0, "done");
server.shutdown().await;
}

190
joule-code/joule-code-rs/crates/joule-moq-relay/tests/slow_subscriber.rs Normal file
View file

@ -0,0 +1,190 @@
//! A slow subscriber: its low-priority objects time out at the relay, while every receipt
//! still reaches it and its high-priority track is unaffected (`draft-ietf-moq-transport-22`
//! §5.1, §5.2; WAI jwp-receipts §4.1, §4.4).
//!
//! One publisher sends a staged pair over native QUIC: a base track at publisher priority
//! 0 with no delivery timeout, and a layer track at priority 200 whose
//! OBJECT_DELIVERY_TIMEOUT is 100 ms, each with its receipts track. A fast subscriber reads
//! as it goes. A slow one gives the relay small receive windows and reads nothing until the
//! publisher has sent everything and a second and a half more has passed; the relay is
//! given a small send window, so what cannot be sent waits in its scheduler, where the
//! layer's objects pass their deadline. Then it reads all its streams concurrently.
mod common;
use common::*;
use joule_moq::draft22::{Parameter, Parameters, StreamResetCode};
use joule_moq::receipt::ReceiptVerifier;
use joule_moq_relay::client::ClientOptions;
use std::time::Duration;
const GROUPS: u64 = 40;
const STALL: Duration = Duration::from_millis(1500);
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_slow_subscriber_loses_late_layers_and_keeps_every_receipt() {
tokio::time::timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time");
}
async fn run() {
let server = relay(|c| {
c.limits.quic_send_window_bytes = 64 * 1024;
})
.await;
let key = ed25519_dalek::SigningKey::from_bytes(&[0x51; 32]);
let space = ns(&["stage", "cam"]);
let base = Tracks::new(space.clone(), "base", 10, 1, 0);
let layer = Tracks::new(space.clone(), "layer", 20, 2, 200);
let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await;
let _b = publish_tracks(&publisher, &base, None).await;
let _l = publish_tracks(&publisher, &layer, Some(100)).await;
let (fast, _ef) = connect(&server, Via::Quic, ClientOptions::default()).await;
let mut small = quinn::TransportConfig::default();
small.stream_receive_window(quinn::VarInt::from_u32(32 * 1024));
small.receive_window(quinn::VarInt::from_u32(96 * 1024));
let (slow, _es) = connect_with(&server, Via::Quic, ClientOptions { lazy_data: true, ..ClientOptions::default() }, Some(small)).await;
let next = || Parameters::new().with(Parameter::LocationFilter(joule_moq::draft22::LocationFilter::NextObject));
let fast_base = subscribe_both(&fast, &base, next()).await;
let fast_layer = subscribe_both(&fast, &layer, next()).await;
// The slow subscriber also asks for a 50 ms delivery timeout on its receipts
// subscriptions, which it should not (moq-streaming-format §4.1): a receipts track has
// none, and the relay applies none to it.
let receipts_timeout = || Parameters::new().with(Parameter::ObjectDeliveryTimeout(50));
let slow_base = subscribe_both_with(&slow, &base, next(), receipts_timeout()).await;
let slow_layer = subscribe_both_with(&slow, &layer, next(), receipts_timeout()).await;
let (published_tx, published_rx) = tokio::sync::oneshot::channel::<()>();
let publish = async {
for g in 0..GROUPS {
publish_group(&publisher, &key, &base, g, &payload("base", g, 1024)).await;
publish_group(&publisher, &key, &layer, g, &payload("layer", g, 24 * 1024)).await;
tokio::time::sleep(Duration::from_millis(15)).await;
}
let _ = published_tx.send(());
};
// Both subscribers read their streams concurrently, as a subscriber does: one that read
// them one at a time could wait on a stream whose bytes wait for connection credit
// held by the unread streams behind it (see the README, "A subscriber that does not
// read"). Each reads until the base track and every receipt are whole, then until a
// second passes with nothing new, so every layer stream is read or seen reset.
let whole = |media: u64, receipts: u64, layer_receipts: u64| {
move |got: &[Got]| {
let n = |a: u64| got.iter().filter(|x| x.header.track_alias == a && x.reset.is_none()).count() as u64;
n(media) == GROUPS && n(receipts) == GROUPS && n(layer_receipts) == GROUPS
}
};
let fast_reads = collect_concurrent(&fast, STEP * 2, Duration::from_secs(1), whole(fast_base.media_alias, fast_base.receipts_alias, fast_layer.receipts_alias));
let slow_reads = async {
// Nothing is read while the load arrives, however long a loaded host takes to send
// it, nor for the stall after: every layer object waiting then is past its deadline.
let _ = tokio::time::timeout(STEP * 2, published_rx).await;
tokio::time::sleep(STALL).await;
collect_concurrent(&slow, STEP * 2, Duration::from_secs(1), whole(slow_base.media_alias, slow_base.receipts_alias, slow_layer.receipts_alias)).await
};
let ((), fast_got, slow_got) = tokio::join!(publish, fast_reads, slow_reads);
// The fast subscriber: every base object and every receipt.
let (fo, fb, _, _) = verify(&fast_got, &fast_base, &key);
assert_eq!((fo, fb), (GROUPS as usize, GROUPS as usize));
let (_, flb, _, _) = verify(&fast_got, &fast_layer, &key);
assert_eq!(flb, GROUPS as usize, "every layer receipt reached the fast subscriber");
// The slow subscriber's base track is whole: each group once, none reset. (Objects on
// different streams may arrive in any order; the order the relay hands them to the
// transport is checked in priority_fan_out.rs.)
let (bo, bb, _, _) = verify(&slow_got, &slow_base, &key);
if (bo, bb) != (GROUPS as usize, GROUPS as usize) {
// Where the missing base objects were: what the subscriber saw, what its transport
// counted, and what the relay says it handed to the transport.
let seen: Vec<(u64, Option<u64>)> = slow_got.iter().filter(|x| x.header.track_alias == slow_base.media_alias).map(|x| (x.header.group_id, x.reset)).collect();
let missing: Vec<u64> = (0..GROUPS).filter(|g| !seen.iter().any(|(b, r)| b == g && r.is_none())).collect();
let waited = slow.waited_out.lock().unwrap().clone();
let client_quic = slow.conn.diagnostics();
let relay_sessions = server.relay().session_diagnostics();
// Then read every stream still queued, all at once, as a subscriber reading its
// streams concurrently would: base objects held only by flow control now arrive.
let mut reads = Vec::new();
for _ in 0..8 * GROUPS {
let Some(mut r) = slow.next_stream(Duration::from_millis(500)).await else { break };
reads.push(tokio::spawn(async move {
tokio::time::timeout(Duration::from_secs(10), r.start()).await.ok()?.ok()?;
read_stream(r).await
}));
}
let mut late_base = Vec::new();
for t in reads {
if let Ok(Some(g)) = t.await
&& g.header.track_alias == slow_base.media_alias
&& g.reset.is_none()
{
late_base.push(g.header.group_id);
}
}
late_base.sort();
slow.close(0, "diagnosis");
let relay = server.relay().clone();
until(STEP, || relay.summaries().iter().any(|s| s.summary.tally.subscriber.tracks.len() >= 4)).await;
let base_name = joule_moq::draft22::names::render(&space, Some(&base.media));
let tallies: Vec<String> = relay
.summaries()
.iter()
.filter_map(|s| s.summary.tally.subscriber.tracks.get(&base_name).map(|t| format!("session {}: {:?}, undelivered at close {}", s.summary.session, t, s.summary.tally.subscriber.undelivered_at_close)))
.collect();
panic!(
"the high-priority track is unaffected: {bo} base objects and {bb} base receipts of {GROUPS}\n\
missing base groups {missing:?}; base streams seen (group, reset) {seen:?}\n\
next_data gave up waiting on a header {} time(s), streams queued behind each: {waited:?}\n\
base groups that arrived once the queued streams were read concurrently: {late_base:?}\n\
subscriber QUIC: {client_quic:?}\nrelay sessions: {relay_sessions:?}\nrelay's base tallies: {tallies:?}",
waited.len()
);
}
let mut base_groups: Vec<u64> = slow_got.iter().filter(|x| x.header.track_alias == slow_base.media_alias).map(|x| x.header.group_id).collect();
base_groups.sort();
assert_eq!(base_groups, (0..GROUPS).collect::<Vec<_>>());
// Its layer track lost objects to the delivery timeout; every layer receipt arrived
// anyway, and each dropped object is accounted for by inclusion (jwp-receipts §4.4).
let (lo, lb, groups, receipts) = verify(&slow_got, &slow_layer, &key);
assert_eq!(lb, GROUPS as usize, "every layer receipt reached the slow subscriber");
assert!(lo < GROUPS as usize, "some layer objects passed their deadline ({lo} of {GROUPS} delivered)");
let delivered: std::collections::HashSet<u64> =
slow_got.iter().filter(|x| x.header.track_alias == slow_layer.media_alias && x.reset.is_none()).map(|x| x.header.group_id).collect();
let verifier = ReceiptVerifier::with_anchors(vec![key.verifying_key()]);
let mut inclusion = 0;
for g in 0..GROUPS {
if !delivered.contains(&g) {
verifier.verify_object(&receipts[&g], &groups[&g]).expect("a dropped object's receipt verifies by inclusion");
inclusion += 1;
}
}
assert_eq!(inclusion as usize, GROUPS as usize - lo);
for x in slow_got.iter().filter(|x| x.header.track_alias == slow_layer.media_alias) {
if let Some(code) = x.reset {
assert_eq!(code, StreamResetCode::DELIVERY_TIMEOUT.0, "a layer stream reset for its deadline");
}
}
// The relay's summary of the slow session says the same.
slow.close(0, "done");
let relay = server.relay().clone();
assert!(until(STEP, || relay.summaries().iter().any(|s| s.summary.tally.subscriber.dropped_delivery_timeout > 0)).await);
let s = relay.summaries().into_iter().find(|s| s.summary.tally.subscriber.dropped_delivery_timeout > 0).unwrap();
s.verify().unwrap();
let t = &s.summary.tally.subscriber.tracks;
let layer_name = joule_moq::draft22::names::render(&space, Some(&layer.media));
let base_name = joule_moq::draft22::names::render(&space, Some(&base.media));
let lt = &t[&layer_name];
assert_eq!(lt.objects_forwarded as usize, lo, "the relay forwarded exactly what arrived whole");
assert_eq!(lt.objects_forwarded + lt.dropped_delivery_timeout, GROUPS, "each layer object forwarded or dropped");
assert_eq!(lt.dropped_receipts_track_subscribed, lt.dropped_delivery_timeout, "every drop on a track whose receipts track the session holds");
assert_eq!(t[&base_name].dropped_delivery_timeout, 0);
assert_eq!(t[&base_name].objects_forwarded, GROUPS);
fast.close(0, "done");
publisher.close(0, "done");
server.shutdown().await;
}

11
joule-code/joule-code-rs/crates/joule-moq/README.md
View file

@ -98,13 +98,14 @@ and a peer that does not know it ignores it (§9.1).
## Not yet implemented
- WebTransport (MOQT over HTTP/3, §6.2.1), a relay with priority fan-out and
per-object receipts, and a browser client: the next slice builds them on this codec.
- The authorization token cache (§8.9: aliases, MAX_AUTH_TOKEN_CACHE_SIZE,
retirement): the codec reads and writes tokens, and the session keeps no cache.
per-object receipts, and the authorization token cache (§8.9) are in
`joule-moq-relay`, built on this codec; this crate's own session keeps no token
cache. A browser client is not built yet.
- Evaluating filters at a publisher (§3.3), and holding a peer to its
MAX_FILTER_RANGES: the codec reads and writes every filter and counts their ranges.
- GOAWAY migration, delivery-timeout enforcement on real streams, and §5.1's
scheduling in the QUIC transport; the mock relay applies delivery timeouts.
- GOAWAY, delivery-timeout enforcement on real streams and §5.1's scheduling are
`joule-moq-relay`'s; this crate's QUIC transport has none of them, and its mock
relay applies delivery timeouts on a caller-driven clock.
- `draft-ietf-moq-loc` and `draft-ietf-moq-msf` semantics (catalogs, LOC
properties): their properties ride as any property does.
- The receipts-track payload in the JSON form of WAI `extensions/jwp-receipts.md`

14
joule-code/joule-code-rs/crates/joule-moq/src/lib.rs
View file

@ -44,13 +44,13 @@
//!
//! ## What this crate intentionally does not ship (yet)
//!
//! - WebTransport (MOQT over HTTP/3), a relay with priority fan-out, and a
//! browser client: the next slice builds them on this codec.
//! - The authorization token cache (§8.9: aliases, MAX_AUTH_TOKEN_CACHE_SIZE)
//! and the subscription filters' evaluation at a publisher: the codec reads
//! and writes them, and the session does not yet act on them.
//! - The relay binary (`joule-moq-relay`) and publisher SDK (`joule-moq-publisher`)
//! live in their own crates and depend on this one.
//! - WebTransport (MOQT over HTTP/3), a relay with priority fan-out, and the
//! authorization token cache (§8.9): `joule-moq-relay`, its own crate, builds
//! them on this codec. A browser client is not built yet.
//! - The subscription filters' evaluation at a publisher: the codec reads and
//! writes them, and the session does not yet act on them.
//! - A publisher SDK (`joule-moq-publisher`) would live in its own crate and depend
//! on this one.
//!
//! ## Standards alignment
//!