joule-moq-relay: a MOQT draft-22 relay over QUIC, WebTransport and a WebSocket fallback #293
Loading…
Reference in a new issue
No description provided.
Delete branch "joule-moq/relay"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
joule-moq-relay: a MOQT draft-22 relay over QUIC, WebTransport and a WebSocket fallback
A new crate in the joule-code workspace,
joule-code/joule-code-rs/crates/joule-moq-relay, library and binary. It is a relay fordraft-ietf-moq-transport-22, built onjoule-moq's codec. It serves:moqt-22);h3, with the draft chosen byWT-Available-Protocols) on the same UDP port;It fans objects out in §5.1.2's priority order and applies each subscriber's delivery timeouts at this hop. Every object's receipts reach the subscriber, including the receipts of objects it drops.
What is in it
Transports (
src/transport/), behind oneConnection/SendStream/RecvStreaminterface.moqt-22andh3. A connection with ALPNmoqt-22runs MOQT directly. One with ALPNh3is handed to the WebTransport layer, which:WT-Available-Protocols;WT-Protocol: "moqt-22";draft-ietf-webtrans-http3-16§4.4.draft-ietf-webtrans-http2-15§6:WT_STREAM,WT_RESET_STREAM,WT_STOP_SENDING,WT_MAX_STREAMS,DATAGRAMandWT_CLOSE_SESSION;moqt-22selects the draft;The WebTransport library is
wtransport0.7.2 (MIT OR Apache-2.0). I chose it for three reasons:joule-moqdoes. With itsquinnfeature, one UDP endpoint serves both ALPNs.WT-Available-ProtocolsandWT-Protocolneed.serverCertificateHashes: ECDSA P-256, valid two weeks at most. So the tests connect the way a page does.The other permissive candidate had three releases in the two weeks before this slice.
wtransport0.7.2 has been out since August.Relay behaviour (
src/relay.rs,src/session.rs):joule-moq's receipt Setup Option when the client declares it.goaway_uriand the timeout.Fan-out (
src/sched.rs). Each downstream session has a scheduler.write_concurrencywrites are in flight, and QUIC's send window is configurable. What waits therefore waits in the scheduler, where it can still be dropped (§5.2).Receipts (
src/door.rs):moq-streaming-format.md§4.1) gets no delivery timeout from the relay, even one a subscriber asks for. The receipts of every dropped object therefore still reach a subscriber of that receipts track (WAIjwp-receipts.md§4.1, §4.4).joule-moq's verifier:hold_ms;Session summaries (
src/summary.rs). When a session ends, the relay signs a summary with its own Ed25519 key. It counts, per subscriber and per track:relay_energyis{"acquisition": "unmetered", "joules_micro": 0}: no meter backs the relay, and the 0 is WAI's unmetered marker, not a figure. Where the door verified an object's receipt, the summary also sums the publisher's signed figures under their class, labelled as the publisher's.Configuration is TOML (
relay.example.toml);--devgives loopback defaults. Without certificate files, the relay makes a development certificate (ECDSA P-256, valid fourteen days) and logs its SHA-256 as hex, as base64 and as a readyserverCertificateHashesentry. No secret is in the repository: keys are files, and tokens are named by their hash.Tests
All tests run on loopback, and every wait has a hard bound:
relay_quic,relay_webtransport,relay_websocketslow_subscriberlate_joinergoawayauth_tokensrequest_idspriority_fan_outThere are also 20 unit tests: the capsule codec, the multiplexer (streams, resets, datagrams, close, stream credit), WebTransport code mapping and protocol selection, the development certificate, the door, the token cache and grants, the cache's coverage rules and its budget, the scheduler's order, and the signed summary.
Three falsification runs (each on a committed tree, then reverted):
priority_fan_outfails: subscriber 0 receives in arrival order.slow_subscriberfails: "the receipts of every delivered group arrived" does not hold.late_joinerfails: "the cache covers the track from group 0, though group 0 came after group 1" does not hold.The cache's coverage: a defect the first CI run found, now fixed
The first CI run passed the dedicated relay job. But
standards workspaces build + test / joule-code-rs (linux-arm64)(job 32029) failed inlate_joiner,a_late_joiner_is_served_from_the_cache, withREQUEST_ERROR INTERNAL_ERROR: the publisher did not answer FETCH. The relay had sent a late joiner upstream for objects it held.Cause. The cache latched its coverage on the first object it kept. The test publishes each group on its own subgroup stream. On a loaded host, group 1's stream can be handled before group 0's; the latch then sat on group 1, a FETCH from group 0 was never "covered", and the relay forwarded it to a publisher that answers no FETCH. This was a relay defect, not only a flaky test. I reproduced it with a new test that sends group 1 whole, waits until the relay has it, then sends group 0: against the latch it failed (rb exit 101).
Fix (
src/cache.rs, with the relay counting streams insrc/relay.rsandsrc/session.rs). The cache now keeps what it knows of each group's arrival. A group is whole when:A FETCH from
startis covered when:startup to the Largest Object's group is whole, or known not to exist (a Prior Group ID Gap, §10.8);Coverage is worked out from the groups held each time it is asked. A group that arrives after a later one therefore extends coverage back once it is whole. A group ID never seen is still a gap the cache does not answer across. The relay counts each upstream subgroup stream from its first kept object and closes the count at its FIN; any other end leaves the group not whole. The relay cannot tell a subgroup it has not yet seen from one that does not exist, so a group whose subgroups have all closed counts as whole. The module documentation and README say so.
objects_in. I kept its meaning: objects received from upstream, counted on arrival, before the door, deduplication and the cache. Its documentation now says so. The late-joiner test no longer waits on it. It waits onRelay::cache_coversfor the track and its receipts track from group 0, a query added for this, and onRelay::largest.Stability. In one rb run, the
late_joinerbinary (its three tests, the reordering test among them) ran 32 times, as four concurrent loops of eight, on the build host under its CI load (load average about 43 to 73). It passed 32 of 32 times.The slow subscriber's base track: not a relay defect, now tested as claimed
The second CI run, at
4a8a6e34, failedslow_subscriberonce instandards workspaces build + test / joule-code-rs (linux-arm64)(job 33622): 39 of 40 base objects arrived, after 42.6 s. That time is the 1.5 s stall, then the first read loop running to its 40 s limit, then the drain.Reproduction. I ran the test unchanged on the build host, with CI running there too. It failed 3 times in 64 runs (eight concurrent loops), then 8 times in 144 (24 concurrent loops). Before rerunning, I added a report that runs only on failure. It records:
The evidence, the same in all eight failures of the second batch:
So it is (b). QUIC's connection-level flow control is shared by a session's streams. Bytes the subscriber had received but not read, in the streams queued behind the one it waited on, held the window. The stream it waited on could not get its bytes, and the reader read nothing else. quinn credits a reset stream's unread bytes back to the connection at once, so the layer streams the relay resets for their deadline were not what held the credit.
The test now waits on what it claims:
A first round of stress on that change found one more timing dependency. With concurrent reading, the drops rested on the stall alone. At a load average near 200, a slow publisher was still sending when the stall ended, and 3 of 144 runs dropped no layer object at all. The subscriber now reads nothing until the publisher has sent everything and the stall has passed after that. After that change, 144 of 144 runs passed at a load average of 204 to 244.
The test client no longer drops a stream it timed out waiting on.
next_datakeeps it and offers it again, records each such wait, andnext_streamhands streams out unread.For diagnosis:
Connection::diagnosticsgives a QUIC session's flow-control counters, andRelay::session_diagnosticsgives each session's.The README has a new section, "A subscriber that does not read". A subscriber that does not read stalls its own high-priority track through connection flow control, and one that reads its streams one at a time can stall itself. The relay cannot release credit the subscriber holds, and does not drop what has no deadline. What it does do (§5.2):
write_concurrencyandquic_send_window_bytes, so an object that expires there takes no credit;It does not yet end such a subscription with PUBLISH_DONE TOO_FAR_BEHIND. That is listed under "Not yet implemented".
Commands run on the build host
Every build ran through
rb. The final runs are on the branch head,159d7ad9, after mergingorigin/main(6abb063d, with #303 and #308) in. Each exit status below isrb's, which is the command's own.rb <worktree> -- 'cd joule-code/joule-code-rs && cargo clippy --locked -p joule-moq-relay --all-targets > $RB_ROOT/tmp/moq-relay-b-clippy2.log 2>&1'joule-moq's andjwp's, which predate this slice.rb <worktree> -- 'cd joule-code/joule-code-rs && cargo test --locked -p joule-moq-relay --all-targets > $RB_ROOT/tmp/moq-relay-b-test2.log 2>&1'slow_subscriberandlate_joinerbinaries, 16 concurrent loops of four runs each, all 32 loops at once, at a load average of about 207 to 220; the command fails if any run didrb <worktree> -- './scripts/check-lockfiles.sh joule-code/joule-code-rs'(no dependency has changed since)--test late_joineragainst the latch)--test priority_fan_out,--test slow_subscriber,--test late_joiner, each with its property removed)Locally, these pass:
check-tests-in-ci.py,check-workflow-concurrency.shand its self-test,check-spec-refs.py,check-wai-section-refs.py,check-no-vanity-metrics.sh,check-comparative-prose.sh,check-orphan-sources.py, andtest-public-text-checks.sh.CI
joule-code-conformance.ymlgains amoq-relayjob onlinux-arm64that runscargo test --locked -p joule-moq-relay --all-targets. The crate needs nothing x86-specific. The x86 workspace step now excludes the crate, so it does not run twice on the scarcer runner. Thestandards workspaces build + test / joule-code-rs (linux-arm64)leg also builds and tests the whole joule-code workspace, the relay included.On the first CI run, at
beb2a883, the dedicated job and the x86 conformance gate passed. The standards-build leg failed on the coverage defect described above, which is fixed at4b8f08ba. On the second, at4a8a6e34, every check but that leg passed. That leg failed on the slow subscriber's read order, also described above, and the test is fixed at159d7ad9.Not yet implemented
The crate README lists these too. They fall into four groups.
Left for later in this crate:
wai_receipts_track.delivery-gapclaim is signed.Library limits:
draft-ietf-webtrans-http3-16that the library does not speak: it negotiates with the settings deployed browsers send, notSETTINGS_WT_ENABLED; it has no session flow-control capsules; and it closes a session by closing its QUIC connection, not withWT_CLOSE_SESSION.Deferred to L5c:
/room/page, with end-to-end tests against real media.0x3A46, also an owner decision.Deployment: none. Nothing was started outside the build host's tests, and no port, firewall, proxy or tunnel was touched.
🤖 Generated with Claude Code