diff --git a/.github/workflows/joule-code-conformance.yml b/.github/workflows/joule-code-conformance.yml index 0b76bc505..9e34f5905 100644 --- a/.github/workflows/joule-code-conformance.yml +++ b/.github/workflows/joule-code-conformance.yml @@ -91,11 +91,13 @@ jobs: $SUDO apt-get install -y -qq z3 z3 --version + # joule-moq-relay runs in the job of its own below, on linux-arm64: it needs + # nothing x86-specific, and this runner is the one there is least of. - name: Reference library tests working-directory: joule-code/joule-code-rs env: JOULE_FIX_EXPECT_Z3: '1' - run: cargo test --workspace + run: cargo test --workspace --exclude joule-moq-relay # joule-moq's QUIC transport and its example compile only with `quic`. Named # here so they stay built and tested whatever the workspace's other members @@ -138,3 +140,33 @@ jobs: - name: L3 JouleClaw binding example builds + tests working-directory: joule-code/examples/l3-jouleclaw run: cargo test + + # The MOQT draft-22 relay. Every test is on loopback: two publishers and three + # subscribers over native QUIC, WebTransport over HTTP/3 and the WebSocket fallback; + # a slow subscriber whose late layers time out while every receipt still arrives; a + # late joiner served by FETCH and a fill from the cache, and a FETCH forwarded + # upstream; GOAWAY on shutdown; tokens by alias; Request IDs and the stream limit; + # fan-out order under load. `--all-targets` runs the unit tests and the binary's + # build as well. + moq-relay: + name: joule-moq-relay, every target + runs-on: linux-arm64 + timeout-minutes: 45 + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install Rust 1.98.0 + shell: bash + run: | + export PATH="$HOME/.cargo/bin:$PATH" + command -v rustup >/dev/null 2>&1 || \ + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0 + rustup toolchain install 1.98.0 --profile minimal + rustup default 1.98.0 + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + + - name: joule-moq-relay, every target + working-directory: joule-code/joule-code-rs + shell: bash + run: cargo test --locked -p joule-moq-relay --all-targets diff --git a/joule-code/CHANGELOG.md b/joule-code/CHANGELOG.md index b6aaf84b4..4f10b9966 100644 --- a/joule-code/CHANGELOG.md +++ b/joule-code/CHANGELOG.md @@ -57,6 +57,33 @@ object framing. The crate's version moves to 0.3.0 because its public API change - `tests/vectors/draft-22/`: a corpus of -22 vectors, good and negative, generated by `tests/draft22_vectors.rs`, and an independent decoder that checks it in CI. +### Added — `joule-moq-relay` 0.1.0: a `draft-ietf-moq-transport-22` relay + +A new crate, library and binary, on `joule-moq`'s codec. Publishers send it tracks, +subscribers ask it for them, and it fans each object out to every subscription in the +order -22's priorities give (§5.1.2), dropping what passes a subscriber's delivery timeout +at this hop (§5.2). Whatever it drops, the object's receipts still reach the subscriber: +a track's receipts track is never given a delivery timeout. + +- **Three transports, one draft:** native QUIC (ALPN `moqt-22`) and WebTransport over + HTTP/3 (`WT-Available-Protocols` / `WT-Protocol`, `draft-ietf-webtrans-http3-16`) on one + UDP endpoint, and a WebSocket fallback that carries `draft-ietf-webtrans-http2-15` + capsules, one per binary message, under the subprotocol `moqt-22`. The fallback is a + non-normative binding, written out in the crate's README with example bytes. +- **The relay protocol:** session setup; namespace publish and subscribe routing; + PUBLISH; SUBSCRIBE served from an upstream the relay holds or opens; FETCH and fills + from its cache or forwarded upstream; TRACK_STATUS; REQUEST_UPDATE; GOAWAY on shutdown; + Request-ID parity and duplicate checks; the authorization-token alias cache; several + publishers of one track, deduplicated by (group, subgroup). +- **What the cache covers** is worked out from the groups it holds whole, each time it is + asked, so a group whose streams finish after a later group's is still served from the + cache once it is whole; a group never seen is a gap it does not answer across. +- **A signed session summary** per session. The relay's own energy is the unmetered + marker, since no meter backs it. +- **Not yet:** SUBSCRIBE_TRACKS and Range Filters, WebTransport over HTTP/2, 0-RTT, the + relay moving its own upstream subscriptions, and finding a receipts track from the + catalog rather than by its name; the README lists each. + ### Changed — `joule-moq` 0.2.0: WAI's labelled JWP forms, and a strict verifier `joule-moq` reads and writes the forms of WAI's `extensions/jwp-receipts.md` diff --git a/joule-code/joule-code-rs/Cargo.lock b/joule-code/joule-code-rs/Cargo.lock index 66ffbcc23..e719cf582 100644 --- a/joule-code/joule-code-rs/Cargo.lock +++ b/joule-code/joule-code-rs/Cargo.lock @@ -8,7 +8,7 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" dependencies = [ - "crypto-common", + "crypto-common 0.1.7", "generic-array", ] @@ -77,7 +77,7 @@ dependencies = [ "proc-macro2", "quote", "syn 2.0.119", - "synstructure", + "synstructure 0.13.2", ] [[package]] @@ -195,6 +195,15 @@ dependencies = [ "generic-array", ] +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + [[package]] name = "bs58" version = "0.5.1" @@ -328,7 +337,7 @@ version = "0.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" dependencies = [ - "crypto-common", + "crypto-common 0.1.7", "inout", "zeroize", ] @@ -373,6 +382,12 @@ version = "0.9.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + [[package]] name = "constant_time_eq" version = "0.4.2" @@ -473,6 +488,15 @@ dependencies = [ "typenum", ] +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + [[package]] name = "ctr" version = "0.9.2" @@ -491,7 +515,7 @@ dependencies = [ "cfg-if", "cpufeatures 0.2.17", "curve25519-dalek-derive", - "digest", + "digest 0.10.7", "fiat-crypto", "rand_core 0.6.4", "rustc_version", @@ -528,7 +552,7 @@ version = "0.7.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" dependencies = [ - "const-oid", + "const-oid 0.9.6", "pem-rfc7468", "zeroize", ] @@ -570,12 +594,23 @@ version = "0.10.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ - "block-buffer", - "const-oid", - "crypto-common", + "block-buffer 0.10.4", + "const-oid 0.9.6", + "crypto-common 0.1.7", "subtle", ] +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer 0.12.1", + "const-oid 0.10.2", + "crypto-common 0.2.2", +] + [[package]] name = "displaydoc" version = "0.2.7" @@ -609,7 +644,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" dependencies = [ "der", - "digest", + "digest 0.10.7", "elliptic-curve", "rfc6979", "signature", @@ -637,7 +672,7 @@ dependencies = [ "ed25519", "rand_core 0.6.4", "serde", - "sha2", + "sha2 0.10.9", "subtle", "zeroize", ] @@ -656,7 +691,7 @@ checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" dependencies = [ "base16ct", "crypto-bigint", - "digest", + "digest 0.10.7", "ff", "generic-array", "group", @@ -681,6 +716,22 @@ version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "edd0f118536f44f5ccd48bcb8b111bdc3de888b58c74639dfb034a357d0f206d" +[[package]] +name = "equivalent" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "00d174d5400e5e8fd687ad1049e2f578285fa914201b1af7e8b112a4546bd826" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + [[package]] name = "fastbloom" version = "0.17.0" @@ -721,6 +772,15 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + [[package]] name = "frost-core" version = "2.2.0" @@ -754,7 +814,7 @@ dependencies = [ "frost-core", "frost-rerandomized", "rand_core 0.6.4", - "sha2", + "sha2 0.10.9", ] [[package]] @@ -888,6 +948,18 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + [[package]] name = "getrandom" version = "0.4.3" @@ -897,7 +969,7 @@ dependencies = [ "cfg-if", "js-sys", "libc", - "r-efi", + "r-efi 6.0.0", "rand_core 0.10.1", "wasm-bindgen", ] @@ -943,6 +1015,12 @@ dependencies = [ "byteorder", ] +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + [[package]] name = "heapless" version = "0.7.17" @@ -978,7 +1056,7 @@ version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" dependencies = [ - "digest", + "digest 0.10.7", ] [[package]] @@ -1015,12 +1093,43 @@ dependencies = [ "hpke-rs-crypto", "p256", "p384", - "rand_chacha", + "rand_chacha 0.3.1", "rand_core 0.6.4", - "sha2", + "sha2 0.10.9", "x25519-dalek", ] +[[package]] +name = "httlib-huffman" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a9fcbcc408c5526c3ab80d534e5c86e7967c1fb7aa0a8c76abd1edc27deb877" + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "hybrid-array" +version = "0.4.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17" +dependencies = [ + "typenum", +] + [[package]] name = "iana-time-zone" version = "0.1.65" @@ -1045,6 +1154,120 @@ dependencies = [ "cc", ] +[[package]] +name = "icu_collections" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0" + +[[package]] +name = "icu_properties" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148" +dependencies = [ + "displaydoc", + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa" + +[[package]] +name = "icu_provider" +version = "2.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d27bbb9d3abbefac45d55f647c9de1d44aafcd1186eb91879afef17c396c3e73" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "indexmap" +version = "2.14.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855" +dependencies = [ + "equivalent", + "hashbrown", +] + [[package]] name = "inout" version = "0.1.4" @@ -1389,6 +1612,33 @@ dependencies = [ "uuid", ] +[[package]] +name = "joule-moq-relay" +version = "0.1.0" +dependencies = [ + "async-trait", + "base64", + "blake3", + "bytes", + "ed25519-dalek", + "futures", + "hex", + "joule-moq", + "quinn", + "rand_core 0.6.4", + "ring", + "rustls", + "serde", + "serde_json", + "thiserror 2.0.20", + "tokio", + "tokio-tungstenite", + "toml", + "tracing", + "tracing-subscriber", + "wtransport", +] + [[package]] name = "joulecontract-amend" version = "0.6.0" @@ -1428,7 +1678,7 @@ dependencies = [ "hex", "serde", "serde_json", - "sha2", + "sha2 0.10.9", "thiserror 2.0.20", ] @@ -1516,6 +1766,12 @@ version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" +[[package]] +name = "litemap" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae" + [[package]] name = "litrs" version = "1.0.0" @@ -1600,6 +1856,15 @@ dependencies = [ "minimal-lexical", ] +[[package]] +name = "nu-ansi-term" +version = "0.50.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "num-bigint" version = "0.4.8" @@ -1634,6 +1899,12 @@ dependencies = [ "autocfg", ] +[[package]] +name = "octets" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d59d30d3ad7f7e1c9a66e51c9cb71ae1c794b28ffe7053fd785defb0d180069" + [[package]] name = "oid-registry" version = "0.8.1" @@ -1714,9 +1985,9 @@ dependencies = [ "openmls_traits", "p256", "rand 0.8.8", - "rand_chacha", + "rand_chacha 0.3.1", "serde", - "sha2", + "sha2 0.10.9", "thiserror 1.0.69", "tls_codec", ] @@ -1746,7 +2017,7 @@ dependencies = [ "ecdsa", "elliptic-curve", "primeorder", - "sha2", + "sha2 0.10.9", ] [[package]] @@ -1778,6 +2049,12 @@ dependencies = [ "base64ct", ] +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + [[package]] name = "pin-project-lite" version = "0.2.17" @@ -1842,6 +2119,15 @@ dependencies = [ "serde", ] +[[package]] +name = "potential_utf" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661" +dependencies = [ + "zerovec", +] + [[package]] name = "powerfmt" version = "0.2.0" @@ -1970,6 +2256,12 @@ dependencies = [ "proc-macro2", ] +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + [[package]] name = "r-efi" version = "6.0.0" @@ -1983,10 +2275,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e058c7de0b26af77780c769414d6257830bb240f3c38477dbc2c16e5f54d6d4c" dependencies = [ "libc", - "rand_chacha", + "rand_chacha 0.3.1", "rand_core 0.6.4", ] +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha 0.9.0", + "rand_core 0.9.5", +] + [[package]] name = "rand" version = "0.10.2" @@ -2008,6 +2310,16 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + [[package]] name = "rand_core" version = "0.6.4" @@ -2017,6 +2329,15 @@ dependencies = [ "getrandom 0.2.17", ] +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + [[package]] name = "rand_core" version = "0.10.1" @@ -2330,6 +2651,15 @@ dependencies = [ "zmij", ] +[[package]] +name = "serde_spanned" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7523beb55eece201a2356bee0bbca0d1ab466c14c07703b2e0ee6d42cb0c2c" +dependencies = [ + "serde_core", +] + [[package]] name = "serdect" version = "0.2.0" @@ -2340,6 +2670,17 @@ dependencies = [ "serde", ] +[[package]] +name = "sha1" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest 0.10.7", +] + [[package]] name = "sha1_smol" version = "1.0.1" @@ -2354,7 +2695,27 @@ checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" dependencies = [ "cfg-if", "cpufeatures 0.2.17", - "digest", + "digest 0.10.7", +] + +[[package]] +name = "sha2" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47d7069beb7d6ac7b9acd1039986e73443f24234f41074da099d6f994ac9ad19" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.3", +] + +[[package]] +name = "sharded-slab" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +dependencies = [ + "lazy_static", ] [[package]] @@ -2363,13 +2724,23 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "signature" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" dependencies = [ - "digest", + "digest 0.10.7", "rand_core 0.6.4", ] @@ -2401,6 +2772,12 @@ version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" +[[package]] +name = "smallvec" +version = "1.16.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b" + [[package]] name = "socket2" version = "0.6.5" @@ -2475,6 +2852,17 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "synstructure" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.4", +] + [[package]] name = "thiserror" version = "1.0.69" @@ -2515,6 +2903,15 @@ dependencies = [ "syn 3.0.4", ] +[[package]] +name = "thread_local" +version = "1.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070" +dependencies = [ + "cfg-if", +] + [[package]] name = "time" version = "0.3.55" @@ -2545,6 +2942,16 @@ dependencies = [ "time-core", ] +[[package]] +name = "tinystr" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643" +dependencies = [ + "displaydoc", + "zerovec", +] + [[package]] name = "tinyvec" version = "1.12.0" @@ -2592,6 +2999,7 @@ dependencies = [ "libc", "mio", "pin-project-lite", + "signal-hook-registry", "socket2", "tokio-macros", "windows-sys 0.61.2", @@ -2608,6 +3016,18 @@ dependencies = [ "syn 3.0.4", ] +[[package]] +name = "tokio-tungstenite" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f72a05e828585856dacd553fba484c242c46e391fb0e58917c942ee9202915c" +dependencies = [ + "futures-util", + "log", + "tokio", + "tungstenite", +] + [[package]] name = "tokio-util" version = "0.7.19" @@ -2622,6 +3042,45 @@ dependencies = [ "tokio", ] +[[package]] +name = "toml" +version = "0.9.12+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf92845e79fc2e2def6a5d828f0801e29a2f8acc037becc5ab08595c7d5e9863" +dependencies = [ + "indexmap", + "serde_core", + "serde_spanned", + "toml_datetime", + "toml_parser", + "toml_writer", + "winnow 0.7.15", +] + +[[package]] +name = "toml_datetime" +version = "0.7.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_parser" +version = "1.1.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" +dependencies = [ + "winnow 1.0.4", +] + +[[package]] +name = "toml_writer" +version = "1.1.3+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06bdbd8cfc056b8d2e2e85f29b56a3bdbecb527cef81eb39e3e7b98af4652770" + [[package]] name = "tracing" version = "0.1.44" @@ -2652,6 +3111,48 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" dependencies = [ "once_cell", + "valuable", +] + +[[package]] +name = "tracing-log" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" +dependencies = [ + "log", + "once_cell", + "tracing-core", +] + +[[package]] +name = "tracing-subscriber" +version = "0.3.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319" +dependencies = [ + "nu-ansi-term", + "sharded-slab", + "smallvec", + "thread_local", + "tracing-core", + "tracing-log", +] + +[[package]] +name = "tungstenite" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c01152af293afb9c7c2a57e4b559c5620b421f6d133261c60dd2d0cdb38e6b8" +dependencies = [ + "bytes", + "data-encoding", + "http", + "httparse", + "log", + "rand 0.9.5", + "sha1", + "thiserror 2.0.20", ] [[package]] @@ -2678,7 +3179,7 @@ version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" dependencies = [ - "crypto-common", + "crypto-common 0.1.7", "subtle", ] @@ -2688,6 +3189,24 @@ version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + [[package]] name = "uuid" version = "1.25.0" @@ -2701,6 +3220,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "valuable" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" + [[package]] name = "version_check" version = "0.9.5" @@ -2734,6 +3259,15 @@ version = "0.11.1+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + [[package]] name = "wasm-bindgen" version = "0.2.127" @@ -2948,6 +3482,66 @@ version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" +[[package]] +name = "winnow" +version = "0.7.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" + +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "writeable" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc" + +[[package]] +name = "wtransport" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4273ce3157a3262a68665f8d3f20a0ac0c5b8a69ffd67f05ae986832ebec036" +dependencies = [ + "bytes", + "pem", + "quinn", + "rcgen", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "sha2 0.11.1", + "socket2", + "thiserror 2.0.20", + "time", + "tokio", + "tracing", + "url", + "wtransport-proto", + "x509-parser", +] + +[[package]] +name = "wtransport-proto" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aad9059572c7dbd6901ccef37f3b7321678cd708dcf58a64b1921dbeab7bfede" +dependencies = [ + "httlib-huffman", + "octets", + "thiserror 2.0.20", + "url", +] + [[package]] name = "x25519-dalek" version = "2.0.1" @@ -2988,6 +3582,29 @@ dependencies = [ "time", ] +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.4", + "synstructure 0.14.0", +] + [[package]] name = "zerocopy" version = "0.8.56" @@ -3008,6 +3625,27 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f75b4683f6c7f45248d4d64056a24298c6281e0993356d7d1b4a1a962ef10d4a" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.4", + "synstructure 0.14.0", +] + [[package]] name = "zeroize" version = "1.9.0" @@ -3028,6 +3666,39 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "zerotrie" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0464e17806c1d976d5cba29399c7f08e516e279e2ba493f63123b5fca67dd8" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.4", +] + [[package]] name = "zmij" version = "1.0.23" diff --git a/joule-code/joule-code-rs/Cargo.toml b/joule-code/joule-code-rs/Cargo.toml index d6452fc20..7fdd965e7 100644 --- a/joule-code/joule-code-rs/Cargo.toml +++ b/joule-code/joule-code-rs/Cargo.toml @@ -17,6 +17,7 @@ members = [ "crates/joule-code-identity", "crates/jwp", "crates/joule-moq", + "crates/joule-moq-relay", "crates/lockstep-quic", ] diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/Cargo.toml b/joule-code/joule-code-rs/crates/joule-moq-relay/Cargo.toml new file mode 100644 index 000000000..7bc70bcc8 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/Cargo.toml @@ -0,0 +1,43 @@ +[package] +name = "joule-moq-relay" +version = "0.1.0" +description = "A MOQT relay for draft-ietf-moq-transport-22: native QUIC and WebTransport over HTTP/3 on one UDP port, a WebSocket fallback, priority-ordered fan-out with per-subscriber delivery timeouts, and every object's receipts forwarded whether the object was delivered or dropped." +edition = "2024" +rust-version = "1.97" +license = "Apache-2.0" +repository = "https://github.com/transaction-science/open-standards" + +[lib] +path = "src/lib.rs" + +[[bin]] +name = "joule-moq-relay" +path = "src/main.rs" + +[dependencies] +joule-moq = { path = "../joule-moq" } +quinn = "0.11" +rustls = { version = "0.23", default-features = false, features = ["ring", "std"] } +ring = "0.17" +# WebTransport over HTTP/3 (MIT OR Apache-2.0). Its `quinn` feature lets one quinn +# endpoint serve both the native MOQT ALPN and `h3`. +wtransport = { version = "0.7.2", default-features = false, features = ["ring", "self-signed", "quinn"] } +tokio-tungstenite = "0.29" +tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "time", "net", "io-util", "signal"] } +bytes = "1" +futures = "0.3" +async-trait = "0.1" +tracing = "0.1" +tracing-subscriber = "0.3" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +toml = "0.9" +ed25519-dalek = { version = "2", features = ["rand_core"] } +rand_core_06 = { package = "rand_core", version = "0.6", features = ["getrandom"] } +blake3 = "1" +hex = "0.4" +base64 = "0.22" +thiserror = "2" + +[dev-dependencies] +tokio = { version = "1", features = ["rt-multi-thread", "macros"] } diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/README.md b/joule-code/joule-code-rs/crates/joule-moq-relay/README.md new file mode 100644 index 000000000..75e705330 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/README.md @@ -0,0 +1,336 @@ +# joule-moq-relay + +A relay for MOQT, `draft-ietf-moq-transport-22`. Publishers send it tracks, subscribers +ask it for them, and it fans each object out to every subscription, in the order the +draft's priorities give, dropping what passes a subscriber's delivery timeout at this hop. +Whatever it drops, the object's receipts still reach the subscriber, on the track's +receipts track. + +It is a library (`Relay`, `RelayServer`, `client::Client`) and a binary, +`joule-moq-relay`. The wire codec is `joule-moq`'s. + +## The draft it speaks + +`draft-ietf-moq-transport-22`, and no other. Drafts do not interoperate, so a session +speaks one from its first byte. The draft is chosen before any MOQT byte: + +- by the ALPN `moqt-22` on native QUIC (§6.2.2); +- by `WT-Available-Protocols: "moqt-22"` on WebTransport, which the relay answers with + `WT-Protocol: "moqt-22"` (§6.2.1; `draft-ietf-webtrans-http3-16` §3.3). A client that + sends no `WT-Available-Protocols` is taken to speak -22; one that offers other + protocols only is refused; +- by the subprotocol `moqt-22` on the WebSocket fallback (below). + +In its SETUP the relay declares `MOQT_IMPLEMENTATION`, `MAX_REQUEST_UPDATES` and +`MAX_AUTH_TOKEN_CACHE_SIZE`. Where the client declares `joule-moq`'s receipt extension +(Setup Option `0x3A46`, its value the extension version), the relay declares it back. + +## Transports + +| Path | Listener | Selected by | +|---|---|---| +| Native QUIC | UDP `quic_listen` | ALPN `moqt-22` | +| WebTransport over HTTP/3 | the same UDP port | ALPN `h3`, extended CONNECT to `path`, `WT-Available-Protocols` | +| WebSocket fallback | TCP `websocket_listen` | upgrade at `path` with subprotocol `moqt-22` | + +Native QUIC and WebTransport share one quinn endpoint: the relay reads the ALPN of each +connection and either runs MOQT on it directly or hands it to the WebTransport layer. + +**The WebTransport implementation** is `wtransport` 0.7 (MIT OR Apache-2.0). It was +chosen because it runs on the same quinn 0.11 as `joule-moq`, so one UDP endpoint serves +both ALPNs; it gives the server the CONNECT request's headers and lets it answer with its +own, which is what `WT-Available-Protocols` and `WT-Protocol` need; and its client checks +a certificate by hash under the rules a browser applies to `serverCertificateHashes` +(ECDSA P-256, a validity of fourteen days at most), so the relay's tests connect the way a +page does. Stream bytes are read and written on the quinn streams underneath; application +error codes on WebTransport streams are mapped into the HTTP/3 error space as +`draft-ietf-webtrans-http3-16` §4.4 gives. + +## How a browser connects + +**WebTransport, in development.** Start the relay with `--dev` (or with no certificate +files in the configuration). It makes a certificate a browser accepts by hash and logs its +SHA-256 three ways, including a ready `serverCertificateHashes` entry: + +```js +const transport = new WebTransport("https://127.0.0.1:4443/moq", { + serverCertificateHashes: [{ algorithm: "sha-256", value: new Uint8Array([/* printed */]) }], + protocols: ["moqt-22"], +}); +await transport.ready; // transport.protocol === "moqt-22" +``` + +The certificate is valid fourteen days from the relay's start; restart the relay to make a +new one. In production give it a certificate chain and key from files, and a page connects +without `serverCertificateHashes`. + +**WebSocket, where there is no UDP path.** A site behind a front that carries HTTP over +TCP only reaches no UDP endpoint, so neither QUIC path is open to its pages. Such a page +uses the fallback: + +```js +const ws = new WebSocket("wss://relay.example/moq", "moqt-22"); +ws.binaryType = "arraybuffer"; // each message is one capsule, below +``` + +The WebSocket listener speaks plain HTTP; TLS is the front's. + +## The WebSocket binding (non-normative) + +This binding is this crate's, not a standard's. It carries what a WebTransport session over +HTTP/2 carries (`draft-ietf-webtrans-http2-15`), on a WebSocket instead of an extended +CONNECT stream, so that a client written for one is most of the way to the other. + +1. **Opening.** The client opens a WebSocket (RFC 6455) to the relay's `path`, offering + the subprotocol `moqt-22` in `Sec-WebSocket-Protocol`. The relay answers with that + subprotocol, or refuses the upgrade (`400` without it, `404` at another path). +2. **Messages.** Every message is a binary message holding exactly one capsule: `Type (i) + Length (i) Value`, with QUIC variable-length integers (RFC 9000 §16), not MOQT's. A + text message carries nothing and is ignored. A binary message that is not one whole + capsule closes the session. +3. **Capsules.** Those of `draft-ietf-webtrans-http2-15` §6, with their type codes: + + | Capsule | Type | Value | Use here | + |---|---|---|---| + | `WT_STREAM` | `0x190B4D3C`, `0x190B4D3B` with FIN | Stream ID, data | stream data; a stream opens with its first `WT_STREAM` | + | `WT_RESET_STREAM` | `0x190B4D39` | Stream ID, code, Reliable Size | reset a stream (Reliable Size 0) | + | `WT_STOP_SENDING` | `0x190B4D3A` | Stream ID, code | ask the peer to stop | + | `WT_MAX_STREAMS` | `0x190B4D3F` bidi, `0x190B4D40` uni | Maximum Streams | stream-count credit | + | `WT_STREAMS_BLOCKED` | `0x190B4D43`, `0x190B4D44` | Maximum Streams | sent when out of credit; informational | + | `DATAGRAM` | `0x00` | payload | one MOQT datagram, reliable and ordered | + | `WT_CLOSE_SESSION` | `0x2843` | code (32 bits), message (UTF-8, at most 1,024 bytes) | close with a MOQT session termination code | + | `WT_DRAIN_SESSION` | `0x78AE` | empty | read and ignored; MOQT's GOAWAY is the drain | + | others | | | read and ignored, as unknown capsules are | + +4. **Streams.** Stream IDs are numbered as QUIC numbers them: bit 0 the initiator (0 the + client), bit 1 the direction (0 bidirectional), and the `n`th stream of a kind is + `4n + kind`. MOQT's control streams, request streams and data streams ride them + byte for byte as they would ride QUIC streams: the client's control stream is its + first unidirectional stream (ID 2), beginning with SETUP; a request is a bidirectional + stream; a subgroup, fetch or padding stream is a unidirectional one. +5. **Stream credit.** Each end first sends its `WT_MAX_STREAMS` for both kinds; until the + peer's arrive, an end opens nothing. Credit is cumulative, as in QUIC. A stream's credit + comes back once both of its halves are done, and is granted in batches. Opening past + the limit closes the session with code `0x045D4487`. Because MOQT -22 bounds requests in + flight by stream limits alone (§6.4.2), the relay's bidirectional limit + (`max_requests`) is its bound on requests in flight on this binding, as MAX_STREAMS is + on QUIC. +6. **Data flow control.** None at the capsule level (`WT_MAX_DATA`, `WT_MAX_STREAM_DATA` + and their `BLOCKED` capsules are read and ignored): TCP's backpressure bounds a sender, + and a receiver that holds more than `websocket_buffer_bytes` unread closes the session + with `0x045D4487`. +7. **Closing.** `WT_CLOSE_SESSION` with a MOQT session termination code (§12.2), then the + WebSocket's close. A WebSocket that closes without one is a session closed with code 0. + +The first capsules a relay with the default limits sends are its credit: + +```text +99 0b 4d 3f 02 41 00 WT_MAX_STREAMS (bidirectional), 256 +99 0b 4d 40 02 44 00 WT_MAX_STREAMS (unidirectional), 1024 +``` + +and a session it closes at the end of a GOAWAY's timeout ends with: + +```text +68 43 12 00 00 00 10 47 4f 41 57 41 59 20 74 69 6d 65 6f 75 74 + WT_CLOSE_SESSION, GOAWAY_TIMEOUT (0x10), "GOAWAY timeout" +``` + +## What the relay does + +| -22 | Here | +|---|---| +| §6.3 session initialization | Each end's control stream opens with SETUP; data streams that come first wait for it. A client's SETUP is held to §9.1's rules for a client (no PATH or AUTHORITY over WebTransport; no DELETE or USE_ALIAS token). The relay's control stream carries only its SETUP and a GOAWAY; the end of the peer's closes the session with PROTOCOL_VIOLATION. | +| §6.4.2 requests | A request stream must begin with one of the seven requests. A Request ID of the wrong parity or seen before closes the session with INVALID_REQUEST_ID. MAX_REQUEST_UPDATES is enforced per stream; requests in flight are bounded by the stream limit (`max_requests`). | +| §8.9, §9.1.3, §9.1.4 tokens | A receiver's alias cache per session: REGISTER, DELETE, USE_ALIAS, USE_VALUE; AUTH_TOKEN_CACHE_OVERFLOW, DUPLICATE_AUTH_TOKEN_ALIAS, UNKNOWN_AUTH_TOKEN_ALIAS; a registration in SETUP past the cache reads as USE_VALUE. Grants name a namespace prefix, rights to publish and subscribe, and the SHA-256 of a token of type 0. | +| §4.1, §4.2 namespaces | PUBLISH_NAMESPACE is accepted (REQUEST_OK) and stands until it is cancelled or its session ends. SUBSCRIBE_NAMESPACE answers with NAMESPACE for each published namespace under the prefix, then NAMESPACE and NAMESPACE_DONE as they change, each carrying the suffix after the prefix; an overlapping prefix in one session is PREFIX_OVERLAP. | +| §7.6, §9.8 PUBLISH | Accepted with REQUEST_OK; its Track Alias is the publisher's for its data. A second track under one alias in one session closes it with DUPLICATE_TRACK_ALIAS. PUBLISH_DONE ends the track's downstream subscriptions. | +| §7.4, §7.6, §9.6 SUBSCRIBE | SUBSCRIBE_OK only once an upstream subscription is Established (§7.4): one the relay holds, or a SUBSCRIBE it sends to every publisher of a matching namespace. The relay's own upstream SUBSCRIBE asks for every object at priority 128 with no timeout, copying no subscriber's priority, group order or timeout (§5.1.3, §9.20.3, §9.20.4). RENDEZVOUS_TIMEOUT holds a SUBSCRIBE for a publisher, up to `max_rendezvous_ms`; without it a track with no publisher is DOES_NOT_EXIST. Location Filters set where the subscription starts and ends. A publisher of a namespace that arrives later is sent a SUBSCRIBE for every subscription the relay holds under it. When the last subscriber leaves, the relay cancels its own upstream SUBSCRIBE. | +| §9.5 REQUEST_UPDATE | Subscriber priority (re-applied to what is queued), FORWARD, the Location Filter and both delivery timeouts. | +| §3.2, §3.4, §7.1, §9.11 FETCH and fills | A fetch the cache covers is served from it, groups in the requested order (coverage, below); otherwise it goes to a publisher of the track or its namespace and its fetch stream is piped through, past the Fill Timeout a Timed-Out range. A range that starts after the Largest Object is INVALID_RANGE. A SUBSCRIBE with FILL_PARAMETERS gets its fill range the same way on a fill fetch stream named by its Request ID; a fill it cannot serve is a fetch stream reset after its header (§3.4.1). | +| §9.13 TRACK_STATUS | Answered from what the relay holds: the Largest Object and the track's properties. | +| §7.3 several publishers | A track may have several upstreams; each (group, subgroup) is taken from the first that sends it, and the rest are counted as deduplicated. | +| §5.1 priorities, §5.2 delivery timeouts | Below. | +| §6.6.1, §9.2 GOAWAY | On shutdown every session gets GOAWAY with `goaway_uri` and the timeout; new requests are refused with GOING_AWAY; a session still open at the timeout is closed with GOAWAY_TIMEOUT. | +| §6.5 `.session` | No session-level track is defined: a request for one is DOES_NOT_EXIST. | + +### What the cache covers + +The cache answers a fetch alone only where it holds every object of the range, so that a +gap in its answer means an object that does not exist (§3.2.2). It keeps what it knows of +each group's arrival. A group is whole when its start arrived (Object ID 0, or a Prior +Object ID Gap saying the objects before do not exist, §10.9), none of its subgroup streams +is open or was reset, and its end is known: its streams ended with a FIN, or an End of +Group or End of Track status, or a datagram with END_OF_GROUP, ended it. A fetch from a +group is covered when every group from it up to the group of the Largest Object is whole +or known not to exist (a Prior Group ID Gap, §10.8), that last group has its start, and +nothing in between was evicted. Coverage is worked out from the groups held each time it +is asked, so a group whose streams finish after a later group's still counts once it is +whole; a group ID never seen is a gap the cache does not answer across. The relay cannot +tell a subgroup it has not yet seen from one that does not exist, so a group whose +subgroups have all closed counts as whole. + +### Priorities and delivery timeouts + +Each downstream session has a scheduler. Every data stream it will send (a subgroup, a +fetch or fill, a datagram) is a lane whose items go out in order and one at a time, so no +stream's objects are reordered (§7.4). Among the lanes with something to send it takes the +item §5.1.2 puts first: the lower subscriber priority number; then the lower publisher +priority number; then, within a subscription, the group its Group Order puts first (and in +one group a fill before its subscription); then the lower Subgroup ID or, for a datagram, +Object ID, a datagram before a subgroup object. At most `write_concurrency` writes are in +flight, and a QUIC session's send window is `quic_send_window_bytes`: what waits, waits in +the scheduler, where it can still be ordered and dropped (§5.2: "minimize the amount of +data buffered at the underlying transport layer"). Each stream is also given a transport +priority from its subscriber and publisher priorities. + +An object's deadline is its OBJECT_DELIVERY_TIMEOUT from when it reached the relay: the +smaller of the publisher's (the first object of its subgroup, else the track's) and the +subscriber's, where both are set. It is checked before the object goes to the transport and +bounds both opening its stream and writing it. Past it the stream is reset with +DELIVERY_TIMEOUT and nothing more of that subgroup goes to that subscriber; a datagram is +dropped. SUBGROUP_DELIVERY_TIMEOUT runs from the subgroup's FIN until all its data is +acknowledged, and resets the stream if it is not. On the WebSocket binding a FIN counts as +acknowledged once written, as webtrans-http2 §5.2 makes the transitions an acknowledgement +would make at once. + +### A subscriber that does not read + +QUIC's connection-level flow control is shared by every stream of a session (RFC 9000 +§4.1), and bytes a subscriber has received but not read hold it. So yes: a subscriber that +stops reading stalls its own high-priority track along with everything else, until it reads +again. And a subscriber that reads its streams one at a time, in the order they arrive, can +stall itself while still reading: the stream it waits on may have its bytes held back for +credit that the unread streams behind it are holding. A subscriber reads its streams +concurrently. + +The relay cannot release credit the subscriber holds, and does not try to by dropping +what has no deadline. What it does: + +- An object with no delivery timeout (a base track, a receipts track) is never dropped. It + waits, in the scheduler or the transport, and is delivered once the subscriber reads. +- What can still wait, waits in the relay rather than in the transport (§5.2: "minimize the + amount of data buffered at the underlying transport layer"). Writes in flight are + bounded by `write_concurrency` and the transport's unacknowledged bytes by + `quic_send_window_bytes`. So an object whose deadline passes while it waits is dropped + before it takes any credit. +- An object that passes its deadline is reset. A reset stream's final size settles its + account (RFC 9000 §4.5), and a receiver can credit its unread bytes back to the + connection at once, as quinn does. The credit a dropped layer held then comes back + without the subscriber reading it. +- When credit frees, the highest-priority bytes go first: each stream carries its + subscriber and publisher priority as its transport priority. + +MOQT lets a publisher end a subscription whose subscriber falls too far behind, with +PUBLISH_DONE TOO_FAR_BEHIND (§5.2); this relay does not yet. + +### Receipts + +A media track's receipts ride its **receipts track** (WAI `extensions/moq-streaming-format.md` +§4.1): group *g* of it is one object holding the group receipt and the object receipts of +media group *g*. The relay recognises a receipts track by its name: the media track's name +followed by `receipts_track_suffix` (`/receipts` unless configured). + +- **The receipts of dropped objects are forwarded.** A receipts track has no delivery + timeout, and the relay applies none to it, even one a subscriber asks for. So when an + object of a media track passes its deadline, its receipt still reaches every subscriber of + that track's receipts track, which can account for the object by inclusion (WAI + `extensions/jwp-receipts.md` §4.1, §4.4). A receipt that rides on the object itself, as a + property, is lost with it. +- **Elsewhere receipts are opaque.** Outside the namespaces a door rule names, the relay + forwards receipts tracks as it forwards any track and reads nothing in them (§7.8). +- **At the door.** In a namespace a `[[receipts.door]]` rule names, the relay reads the + receipts it forwards, and takes on jwp-receipts §4.1's duties: each group receipt's + signature is verified, under the rule's trusted keys or any key, before the first object + of its group is fanned out, and a group that does not verify is dropped with a structured + error in the log; each media object must match the receipt of its position (its group, + its Object ID, its content hash, its Merkle proof to the signed root), and an object that + carries its receipt as a property must carry that same one. An object waits for its + group's receipts up to `hold_ms`. Receipt bytes are forwarded unaltered and nothing is + re-signed. A refused object ends its downstream streams rather than leaving a hole in + them. + +## The session summary + +When a session ends the relay signs a summary of it with its own Ed25519 key (from +`summary.key_path`, or made at start and its public half logged), keeps it +(`Relay::summaries`), and appends it as one JSON line to `summary.out_path` if set. It +holds, for the session: + +- as a subscriber: objects and their bytes forwarded, every byte written to its data + streams, objects dropped for their delivery timeout and how many of those were on a + track whose receipts track the session also held, subgroups reset after their FIN, + objects served by fetch or fill, objects still queued at the end, and the same per + track; +- as a publisher: objects and bytes received, objects refused at the door, objects + deduplicated; +- `relay_energy`: `{"acquisition": "unmetered", "joules_micro": 0}`. No meter backs this + relay. The 0 is the unmetered marker of WAI `extensions/energy-measurement.md` §2, never + a measurement of zero joules, and must not be read or shown as one; +- `publisher_signed`: where the door verified an object's receipt, the publisher's signed + figure for producing it, summed under the acquisition class its leaf was signed with. + These are the publisher's statements about the objects, not the relay's energy. + +The summary is relay-scoped telemetry (jwp-receipts §4.2), not a receipt of the publisher's. +Its format is `joule-moq-relay/session-summary/1`; the signature covers +`joule-moq-relay:session-summary\x01` followed by the summary's JSON bytes, and +`SignedSessionSummary::verify` checks it. + +## Configuration + +`joule-moq-relay --config relay.toml`, or `joule-moq-relay --dev` for loopback with a +development certificate. `relay.example.toml` shows every field: the listen addresses and +path, the certificate files (or the names for a development certificate), the cache's byte +budget, the limits (sessions, requests and streams in flight, MAX_REQUEST_UPDATES, the token +cache, the rendezvous, upstream and fill timeouts, write concurrency, the QUIC send window, +the GOAWAY timeout), the receipts suffix and door rules, the summary's key and file, and the +grants. The configuration holds no secret: keys are files, and tokens are named by their +SHA-256. `JOULE_MOQ_RELAY_LOG` sets the log level. + +## Not yet implemented + +- **SUBSCRIBE_TRACKS** (§3.6): refused with NOT_SUPPORTED. +- **Range Filters** (§3.3.2): the relay declares MAX_FILTER_RANGES 0, so a request that + carries one is INVALID_FILTER, and it evaluates no Track or Object Property filter. +- **WebTransport over HTTP/2** (`draft-ietf-webtrans-http2-15`), the path a browser takes + when HTTP/3 is unavailable. The WebSocket binding carries the same capsules and covers + the same case for now. +- **Parts of `draft-ietf-webtrans-http3-16` the library does not speak:** it negotiates + with the settings deployed browsers send rather than -16's `SETTINGS_WT_ENABLED`, has no + per-session flow-control capsules, and closes a session by closing its QUIC connection + with the code rather than with `WT_CLOSE_SESSION`; and quinn 0.11 has no RESET_STREAM_AT, + so a reset WebTransport stream is not guaranteed to deliver its stream header first. +- **0-RTT** is not accepted. +- **Migration as a client:** the relay sends GOAWAY to its sessions, but does not itself + move its upstream subscriptions to a new relay, or follow a REDIRECT. +- **Pausing upstream** when every downstream subscriber is paused (§7.2), and forwarding + NEW_GROUP_REQUEST upstream. EXPIRES is neither issued nor acted on. +- **TOO_FAR_BEHIND:** a subscription whose subscriber does not read is held, not ended + (§5.2 allows ending it with PUBLISH_DONE TOO_FAR_BEHIND). +- **A cancellation the relay cannot see:** a subscriber that sends FIN on its direction and + later only STOP_SENDING is noticed when the relay next writes on the request stream, or + when the session ends. +- **Receipts:** the receipts track of a media track is found by the configured suffix, not + by reading the catalog's `wai_receipts_track`; the relay signs no `delivery-gap` claim; + datagrams at the door are checked against their group's receipts, not a property. +- **Energy:** no meter backs the relay, so its own figure is always the unmetered marker. + Attributing a metered host's energy to sessions is not done here. +- **The browser client** and a page that uses it. + +## Tests + +All on loopback, every wait bounded: + +| Test | What it shows | +|---|---| +| `relay_quic`, `relay_webtransport`, `relay_websocket` | Two publishers (one by PUBLISH_NAMESPACE and the relay's SUBSCRIBE, one by PUBLISH) and three subscribers over each transport; every object verifies against its publisher's receipts, which the door also checks; SUBSCRIBE_NAMESPACE; every summary verifies and labels the relay's energy unmetered. | +| `slow_subscriber` | A base track at priority 0 and a layer at 200 with a 100 ms delivery timeout. A subscriber that reads nothing for 1.5 s, then reads all its streams concurrently, loses layer objects to the timeout (their streams reset with DELIVERY_TIMEOUT), keeps the whole base track, and receives every receipt of both tracks although it asked for a timeout on its receipts tracks; each dropped object verifies by inclusion; the summary counts each layer object as forwarded or dropped. | +| `late_joiner` | TRACK_STATUS; FETCH of a track and its receipts from the cache; a range in descending order; INVALID_RANGE; a fill on SUBSCRIBE; a group that reaches the relay after a later one, the FETCH from it still served from the cache; a FETCH forwarded upstream when the cache holds nothing. | +| `goaway` | GOAWAY with its URI and timeout to every session; GOING_AWAY for a new request; GOAWAY_TIMEOUT for the session that stays. | +| `auth_tokens` | Tokens in SETUP, by value and by alias; UNAUTHORIZED; UNKNOWN_AUTH_TOKEN_ALIAS. | +| `request_ids` | INVALID_REQUEST_ID for the wrong parity and for a repeat; requests in flight bounded by the stream limit. | +| `priority_fan_out` | Two publishers, four tracks, three subscribers with their own subscriber priorities and group orders, their transports held shut while the load arrives: each receives every object in §5.1.2's order. | + +## Licence + +Apache-2.0. diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/relay.example.toml b/joule-code/joule-code-rs/crates/joule-moq-relay/relay.example.toml new file mode 100644 index 000000000..74ee7e42f --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/relay.example.toml @@ -0,0 +1,62 @@ +# joule-moq-relay: an example configuration. Every field is optional; these are the +# defaults unless a comment says otherwise. Nothing secret belongs here: keys are files +# the operator provides, and tokens are named by their SHA-256. + +# Native QUIC (ALPN moqt-22) and WebTransport over HTTP/3 (ALPN h3), one UDP port. +quic_listen = "127.0.0.1:4443" +# The WebSocket fallback (plain HTTP upgrade; put TLS in front of it). +websocket_listen = "127.0.0.1:4480" +# The path of the WebTransport CONNECT and of the WebSocket upgrade. +path = "/moq" +# The New Session URI a shutdown GOAWAY names; empty reuses the current one. +goaway_uri = "" + +[tls] +# cert_path = "/etc/joule-moq-relay/fullchain.pem" +# key_path = "/etc/joule-moq-relay/key.pem" +# Without files: a development certificate for these names, its SHA-256 printed. +dev_names = ["localhost", "127.0.0.1", "::1"] + +[cache] +max_bytes = 268435456 + +[limits] +max_sessions = 4096 +max_requests = 256 +max_uni_streams = 1024 +max_request_updates = 16 +max_auth_token_cache_size = 16384 +max_rendezvous_ms = 30000 +upstream_timeout_ms = 5000 +fill_timeout_ms = 2000 +write_concurrency = 8 +max_object_bytes = 16777216 +unknown_alias_wait_ms = 1000 +goaway_timeout_ms = 5000 +websocket_buffer_bytes = 67108864 +idle_timeout_ms = 30000 +quic_send_window_bytes = 1048576 + +[receipts] +receipts_track_suffix = "/receipts" + +# Check objects in this namespace against their receipts at the door (example; not a +# default). An empty trust list accepts any key whose signature verifies. +[[receipts.door]] +prefix = ["example", "live"] +trust = [] +hold_ms = 2000 + +[summary] +# key_path = "/etc/joule-moq-relay/summary-seed" +# out_path = "/var/log/joule-moq-relay/sessions.jsonl" + +[auth] +required = false + +# An example grant; the hash below is of the placeholder string "replace-me". +[[auth.grants]] +prefix = ["example"] +token_sha256 = "1df4b07d05b9b3b1115b4b6b22af319a2f99564a2fa2573a34896063c7513c10" +publish = true +subscribe = true diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/auth.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/auth.rs new file mode 100644 index 000000000..8b5f07225 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/auth.rs @@ -0,0 +1,197 @@ +//! Authorization tokens: the alias cache a receiver keeps for each session +//! (`draft-ietf-moq-transport-22` §8.9, §9.1.3, §9.1.4), and the relay's grants. +//! +//! A grant names a namespace prefix, the rights it gives (publish, subscribe), and the +//! SHA-256 of the token value that carries it. Tokens are of type 0, agreed out of band +//! (§8.9: "Type 0 is reserved to indicate that the type is not defined in the table and +//! is negotiated out-of-band"). + +use crate::config::AuthConfig; +use joule_moq::draft22::names::prefix_matches; +use joule_moq::draft22::{SessionError, Token}; +use joule_moq::TrackNamespace; +use std::collections::HashMap; + +/// A token's size against MAX_AUTH_TOKEN_CACHE_SIZE: 16 bytes and its value (§9.1.3). +fn size_of(value: &[u8]) -> u64 { + 16 + value.len() as u64 +} + +/// One token as a message resolves it: its type and value. +pub type Resolved = (u64, Vec); + +/// The aliases one peer has registered with this endpoint. +#[derive(Debug)] +pub struct TokenCache { + max: u64, + used: u64, + entries: HashMap, +} + +impl TokenCache { + /// A cache bounded by the MAX_AUTH_TOKEN_CACHE_SIZE this endpoint declared. + pub fn new(max: u64) -> Self { + TokenCache { max, used: 0, entries: HashMap::new() } + } + + pub fn used(&self) -> u64 { + self.used + } + + /// The tokens of the peer's SETUP. A REGISTER past the limit is read as USE_VALUE, + /// never a failure (§9.1.4); a duplicate alias still closes the session. A client's + /// SETUP carries no DELETE or USE_ALIAS (`Setup::check_from` refuses them first). + pub fn setup(&mut self, tokens: &[Token]) -> Result, SessionError> { + let mut out = Vec::new(); + for t in tokens { + match t { + Token::Register { alias, token_type, value } => { + if self.entries.contains_key(alias) { + return Err(SessionError::DUPLICATE_AUTH_TOKEN_ALIAS); + } + if self.used + size_of(value) <= self.max { + self.used += size_of(value); + self.entries.insert(*alias, (*token_type, value.clone())); + } + out.push((*token_type, value.clone())); + } + Token::UseValue { token_type, value } => out.push((*token_type, value.clone())), + Token::Delete { .. } | Token::UseAlias { .. } => return Err(SessionError::PROTOCOL_VIOLATION), + } + } + Ok(out) + } + + /// The tokens of one request: registered, retired or resolved. A REGISTER that would + /// pass the limit closes the session with AUTH_TOKEN_CACHE_OVERFLOW, and a duplicate + /// alias with DUPLICATE_AUTH_TOKEN_ALIAS; a reference to an alias not registered is + /// UNKNOWN_AUTH_TOKEN_ALIAS (§8.9). A REGISTER is kept even when the request it rides + /// fails for another reason. + pub fn resolve(&mut self, tokens: &[Token]) -> Result, SessionError> { + let mut out = Vec::new(); + for t in tokens { + match t { + Token::Register { alias, token_type, value } => { + if self.entries.contains_key(alias) { + return Err(SessionError::DUPLICATE_AUTH_TOKEN_ALIAS); + } + if self.used + size_of(value) > self.max { + return Err(SessionError::AUTH_TOKEN_CACHE_OVERFLOW); + } + self.used += size_of(value); + self.entries.insert(*alias, (*token_type, value.clone())); + out.push((*token_type, value.clone())); + } + Token::Delete { alias } => match self.entries.remove(alias) { + Some((_, v)) => self.used -= size_of(&v), + None => return Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS), + }, + Token::UseAlias { alias } => match self.entries.get(alias) { + Some(r) => out.push(r.clone()), + None => return Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS), + }, + Token::UseValue { token_type, value } => out.push((*token_type, value.clone())), + } + } + Ok(out) + } +} + +/// What a request asks to do. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Right { + Publish, + Subscribe, +} + +struct GrantRule { + prefix: TrackNamespace, + token_sha256: [u8; 32], + publish: bool, + subscribe: bool, +} + +/// The relay's grants. +pub struct Authorizer { + required: bool, + grants: Vec, +} + +pub fn sha256(b: &[u8]) -> [u8; 32] { + crate::cert::sha256(b) +} + +impl Authorizer { + pub fn from_config(c: &AuthConfig) -> Result { + let mut grants = Vec::new(); + for g in &c.grants { + let h: [u8; 32] = hex::decode(&g.token_sha256) + .map_err(|e| format!("a grant's token_sha256: {e}"))? + .try_into() + .map_err(|_| "a grant's token_sha256 is not 32 bytes".to_string())?; + grants.push(GrantRule { prefix: TrackNamespace::from_strs(g.prefix.iter().map(String::as_str)), token_sha256: h, publish: g.publish, subscribe: g.subscribe }); + } + Ok(Authorizer { required: c.required, grants }) + } + + pub fn open() -> Self { + Authorizer { required: false, grants: Vec::new() } + } + + /// Whether `tokens` (the request's, and the session's from SETUP) give `right` over + /// `ns`. + pub fn allows(&self, ns: &TrackNamespace, right: Right, tokens: &[Resolved]) -> bool { + if !self.required { + return true; + } + tokens.iter().any(|(ty, value)| { + let h = sha256(value); + *ty == 0 + && self.grants.iter().any(|g| { + g.token_sha256 == h && prefix_matches(&g.prefix, ns) && match right { + Right::Publish => g.publish, + Right::Subscribe => g.subscribe, + } + }) + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_cache_follows_section_8_9() { + let mut c = TokenCache::new(16 + 4 + 16 + 3); + let reg = |a, v: &[u8]| Token::Register { alias: a, token_type: 0, value: v.to_vec() }; + assert_eq!(c.resolve(&[reg(1, b"abcd")]).unwrap(), vec![(0, b"abcd".to_vec())]); + assert_eq!(c.resolve(&[reg(1, b"x")]), Err(SessionError::DUPLICATE_AUTH_TOKEN_ALIAS)); + assert_eq!(c.resolve(&[reg(2, b"wxyz")]), Err(SessionError::AUTH_TOKEN_CACHE_OVERFLOW)); + assert_eq!(c.resolve(&[reg(2, b"xyz")]).unwrap().len(), 1); + assert_eq!(c.resolve(&[Token::UseAlias { alias: 1 }]).unwrap(), vec![(0, b"abcd".to_vec())]); + assert_eq!(c.resolve(&[Token::UseAlias { alias: 9 }]), Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS)); + c.resolve(&[Token::Delete { alias: 1 }]).unwrap(); + assert_eq!(c.used(), 16 + 3); + assert_eq!(c.resolve(&[Token::Delete { alias: 1 }]), Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS)); + // In SETUP a registration past the limit reads as USE_VALUE. + let mut s = TokenCache::new(0); + assert_eq!(s.setup(&[reg(5, b"big")]).unwrap(), vec![(0, b"big".to_vec())]); + assert_eq!(s.resolve(&[Token::UseAlias { alias: 5 }]), Err(SessionError::UNKNOWN_AUTH_TOKEN_ALIAS)); + } + + #[test] + fn grants_cover_a_prefix_and_a_right() { + let cfg = AuthConfig { + required: true, + grants: vec![crate::config::Grant { prefix: vec!["live".into()], token_sha256: hex::encode(sha256(b"t")), publish: false, subscribe: true }], + }; + let a = Authorizer::from_config(&cfg).unwrap(); + let ns = TrackNamespace::from_strs(["live", "a"]); + assert!(a.allows(&ns, Right::Subscribe, &[(0, b"t".to_vec())])); + assert!(!a.allows(&ns, Right::Publish, &[(0, b"t".to_vec())])); + assert!(!a.allows(&ns, Right::Subscribe, &[(0, b"u".to_vec())])); + assert!(!a.allows(&TrackNamespace::from_strs(["other"]), Right::Subscribe, &[(0, b"t".to_vec())])); + assert!(Authorizer::open().allows(&ns, Right::Publish, &[])); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/cache.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/cache.rs new file mode 100644 index 000000000..a26420178 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/cache.rs @@ -0,0 +1,421 @@ +//! The relay's cache of objects (`draft-ietf-moq-transport-22` §7.1), from which it +//! serves FETCH and fills. +//! +//! An object is kept by its Full Track Name, Group ID and Object ID with every field §2.1.1 +//! defines: its delivery mode (a Subgroup ID, or none for a datagram), its publisher +//! priority, its properties, its status and its payload. A duplicate is not stored again +//! (§7.1: "Relays MAY ignore subsequently received Objects"). A byte budget across all +//! tracks evicts whole groups, oldest first. +//! +//! **Coverage.** The cache serves a fetch alone only where it holds every object of the +//! range, so that a gap in its answer means an object that does not exist, as an unmarked +//! gap in a fetch does (§3.2.2). It keeps, for each group, what it knows of the group's +//! arrival ([`GroupState`]), and a group is **whole** when: +//! +//! - its start arrived: Object ID 0, or an object whose Prior Object ID Gap says every +//! object before it does not exist (§10.9); +//! - no subgroup stream of it is still open, and none was reset or cut short; +//! - and its end is known: each of its subgroup streams ended with a FIN, or an End of +//! Group or End of Track status, or a datagram with END_OF_GROUP, ended it. +//! +//! A fetch from `start` is covered when the group of the Largest Object has its start and +//! lost nothing, and every group from `start` up to it is whole or known not to exist (a +//! Prior Group ID Gap, §10.8), and nothing in that span was evicted. The cache computes this +//! from the groups it holds when asked, so a group that arrives after a later one, as +//! streams finish out of order, extends coverage back once it is whole; a group ID it has +//! never seen is a gap it does not cover across. +//! +//! The relay cannot tell a subgroup it has not yet seen from one that does not exist: a +//! group whose subgroups are all closed counts as whole, though another subgroup of it could +//! still open. + +use bytes::Bytes; +use joule_moq::draft22::data::ObjectStatus; +use joule_moq::draft22::kvp::{prop, Properties}; +use joule_moq::draft22::Location; +use std::collections::{BTreeMap, VecDeque}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CachedObject { + /// `None` for an object whose delivery mode was Datagram. + pub subgroup: Option, + pub publisher_priority: u8, + pub properties: Properties, + pub status: ObjectStatus, + pub payload: Bytes, +} + +impl CachedObject { + fn bytes(&self) -> usize { + self.payload.len() + 64 + } +} + +/// What the cache knows of one group's arrival. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct GroupState { + /// Its first object arrived, or the objects before the first that did are known not + /// to exist. + pub start: bool, + /// Its subgroup streams still open upstream. + pub open: u32, + /// Its subgroup streams that ended with a FIN. + pub finished: u32, + /// A stream of it was reset or cut short: objects of it may be missing. + pub broken: bool, + /// An End of Group or End of Track status, or a datagram with END_OF_GROUP, ended it. + pub ended: bool, +} + +impl GroupState { + /// Every object of the group is held (see the module documentation). + pub fn whole(&self) -> bool { + self.start && self.open == 0 && !self.broken && (self.finished > 0 || self.ended) + } +} + +/// One track's objects. +#[derive(Debug, Default)] +pub struct TrackCache { + pub objects: BTreeMap<(u64, u64), CachedObject>, + /// What is known of each group held. + pub groups: BTreeMap, + /// Groups known not to exist, as ranges `lo ..= hi`, keyed by `lo`. + pub absent: BTreeMap, + /// Groups at or below this were evicted. + pub evicted_through: Option, + /// The track ended (an End of Track status), at this location. + pub end_of_track: Option, +} + +/// How an object entering the cache relates to what it holds. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Insert { + New, + /// The same object again: ignored. + Duplicate, + /// An object at that location that differs in its delivery mode, Subgroup ID, + /// priority or payload: the track is malformed (§7.1). + Conflict, + /// In a group already evicted: not kept. + Evicted, +} + +impl TrackCache { + pub fn insert(&mut self, group: u64, object: u64, o: CachedObject) -> Insert { + if self.evicted_through.is_some_and(|e| group <= e) { + return Insert::Evicted; + } + if let Some(prev) = self.objects.get(&(group, object)) { + let same = prev.subgroup == o.subgroup && prev.publisher_priority == o.publisher_priority && prev.payload == o.payload && prev.status == o.status; + return if same { Insert::Duplicate } else { Insert::Conflict }; + } + let gap = |ty| o.properties.find(ty).ok().flatten().and_then(|k| k.as_varint()); + let prior_objects = gap(prop::PRIOR_OBJECT_ID_GAP); + let prior_groups = gap(prop::PRIOR_GROUP_ID_GAP).filter(|n| *n > 0 && *n <= group); + let st = self.groups.entry(group).or_default(); + if object == 0 || prior_objects == Some(object) { + st.start = true; + } + match o.status { + ObjectStatus::EndOfGroup => st.ended = true, + ObjectStatus::EndOfTrack => { + st.ended = true; + self.end_of_track = Some(Location::new(group, object)); + } + ObjectStatus::Normal => {} + } + if let Some(n) = prior_groups { + self.absent.insert(group - n, group - 1); + } + self.objects.insert((group, object), o); + Insert::New + } + + fn group(&mut self, group: u64) -> Option<&mut GroupState> { + if self.evicted_through.is_some_and(|e| group <= e) { + return None; + } + Some(self.groups.entry(group).or_default()) + } + + /// A subgroup stream of `group` began upstream; `starts_group` when it began with the + /// group's first object. + pub fn stream_opened(&mut self, group: u64, starts_group: bool) { + if let Some(st) = self.group(group) { + st.open += 1; + st.start |= starts_group; + } + } + + /// A subgroup stream of `group` ended: with a FIN (`clean`), or reset or cut short. + pub fn stream_ended(&mut self, group: u64, clean: bool) { + if let Some(st) = self.group(group) { + st.open = st.open.saturating_sub(1); + if clean { + st.finished += 1; + } else { + st.broken = true; + } + } + } + + /// Objects of `group` that arrived could not be kept (a malformed track). + pub fn group_broken(&mut self, group: u64) { + if let Some(st) = self.group(group) { + st.broken = true; + } + } + + /// A datagram with END_OF_GROUP ended `group` (§11.2.1). + pub fn group_ended(&mut self, group: u64) { + if let Some(st) = self.group(group) { + st.ended = true; + } + } + + /// Whether `group` is known not to exist. + fn absent_range(&self, group: u64) -> Option { + self.absent.range(..=group).next_back().filter(|(_, hi)| **hi >= group).map(|(lo, _)| *lo) + } + + /// Whether a fetch starting at `start` can be served from the cache alone (see the + /// module documentation). + pub fn covers(&self, start: Location) -> bool { + if self.evicted_through.is_some_and(|e| start.group <= e) { + return false; + } + let Some((&last, st)) = self.groups.iter().next_back() else { return false }; + if start.group > last || !st.start || st.broken { + return false; + } + let mut g = last; + while g > start.group { + g -= 1; + match self.groups.get(&g) { + Some(s) if s.whole() => {} + Some(_) => return false, + None => match self.absent_range(g) { + Some(lo) if lo <= start.group => return true, + Some(lo) => g = lo, + None => return false, + }, + } + } + true + } + + /// The objects from `start` to `end` inclusive, groups in `descending` order or not, + /// objects within a group ascending (§3.2.1). + pub fn range(&self, start: Location, end: Location, descending: bool) -> Vec<((u64, u64), CachedObject)> { + let mut groups: Vec = self.objects.range((start.group, 0)..=(end.group, u64::MAX)).map(|((g, _), _)| *g).collect(); + groups.dedup(); + if descending { + groups.reverse(); + } + let mut out = Vec::new(); + for g in groups { + for ((gg, oo), o) in self.objects.range((g, 0)..=(g, u64::MAX)) { + let loc = Location::new(*gg, *oo); + if loc >= start && loc <= end { + out.push(((*gg, *oo), o.clone())); + } + } + } + out + } + + /// Drop group `g`; the bytes freed. + fn evict_group(&mut self, g: u64) -> usize { + let keys: Vec<(u64, u64)> = self.objects.range((g, 0)..=(g, u64::MAX)).map(|(k, _)| *k).collect(); + let mut freed = 0; + for k in keys { + if let Some(o) = self.objects.remove(&k) { + freed += o.bytes(); + } + } + let through = self.evicted_through.map_or(g, |e| e.max(g)); + self.evicted_through = Some(through); + self.groups = self.groups.split_off(&through.saturating_add(1)); + self.absent.retain(|_, hi| *hi > through); + freed + } +} + +/// The byte budget across tracks: groups in the order they were first cached. +#[derive(Debug)] +pub struct Budget { + pub max: usize, + pub used: usize, + order: VecDeque<(u64, u64, usize)>, +} + +impl Budget { + pub fn new(max: usize) -> Self { + Budget { max, used: 0, order: VecDeque::new() } + } + + /// Account `n` bytes of track `track`'s group `group`. Returns the (track, group) + /// pairs to evict, oldest first, until the budget holds. + pub fn add(&mut self, track: u64, group: u64, n: usize) -> Vec<(u64, u64)> { + self.used += n; + match self.order.back_mut() { + Some((t, g, b)) if *t == track && *g == group => *b += n, + _ => match self.order.iter_mut().rev().find(|(t, g, _)| *t == track && *g == group) { + Some((_, _, b)) => *b += n, + None => self.order.push_back((track, group, n)), + }, + } + let mut evict = Vec::new(); + while self.used > self.max { + let Some((t, g, b)) = self.order.pop_front() else { break }; + self.used = self.used.saturating_sub(b); + evict.push((t, g)); + } + evict + } + + /// Forget the accounting of a track that is gone. + pub fn forget(&mut self, track: u64) { + let mut freed = 0; + self.order.retain(|(t, _, b)| { + if *t == track { + freed += b; + false + } else { + true + } + }); + self.used = self.used.saturating_sub(freed); + } +} + +/// Evict `group` from `cache`, for the budget. +pub fn evict(cache: &mut TrackCache, group: u64) -> usize { + cache.evict_group(group) +} + +/// The size an object counts for. +pub fn size_of(o: &CachedObject) -> usize { + o.bytes() +} + +#[cfg(test)] +mod tests { + use super::*; + use joule_moq::draft22::kvp::KeyValuePair; + + fn obj(p: &'static [u8]) -> CachedObject { + CachedObject { subgroup: Some(0), publisher_priority: 1, properties: Properties::new(), status: ObjectStatus::Normal, payload: Bytes::from_static(p) } + } + + fn whole(c: &mut TrackCache, g: u64) { + c.stream_opened(g, true); + assert_eq!(c.insert(g, 0, obj(b"x")), Insert::New); + c.stream_ended(g, true); + } + + #[test] + fn ranges_and_conflicts() { + let mut c = TrackCache::default(); + for g in 4..=5 { + whole(&mut c, g); + } + assert_eq!(c.insert(4, 1, obj(b"b")), Insert::New); + assert_eq!(c.insert(4, 1, obj(b"b")), Insert::Duplicate); + assert_eq!(c.insert(4, 1, obj(b"y")), Insert::Conflict); + let asc: Vec<_> = c.range(Location::new(4, 0), Location::new(5, 0), false).into_iter().map(|(k, _)| k).collect(); + assert_eq!(asc, vec![(4, 0), (4, 1), (5, 0)]); + let desc: Vec<_> = c.range(Location::new(4, 1), Location::new(5, 9), true).into_iter().map(|(k, _)| k).collect(); + assert_eq!(desc, vec![(5, 0), (4, 1)]); + } + + #[test] + fn coverage_is_the_run_of_whole_groups() { + let mut c = TrackCache::default(); + assert!(!c.covers(Location::new(0, 0)), "nothing held"); + // Group 1 whole before group 0 arrives. + whole(&mut c, 1); + assert!(c.covers(Location::new(1, 0))); + assert!(!c.covers(Location::new(0, 0)), "group 0 not seen: a gap"); + // Group 0 arrives: covered once its stream ends. + c.stream_opened(0, true); + c.insert(0, 0, obj(b"a")); + assert!(!c.covers(Location::new(0, 0)), "its stream is still open"); + c.stream_ended(0, true); + assert!(c.covers(Location::new(0, 0)), "a late group extends coverage back once whole"); + // The newest group need only have started. + c.stream_opened(2, true); + c.insert(2, 0, obj(b"c")); + assert!(c.covers(Location::new(0, 0)) && c.covers(Location::new(2, 0))); + // A group never seen between: no coverage across it. + c.stream_ended(2, true); + whole(&mut c, 4); + assert!(c.covers(Location::new(4, 0)) && !c.covers(Location::new(2, 0)), "group 3 unseen"); + } + + #[test] + fn a_prior_group_id_gap_bridges_groups_that_do_not_exist() { + // Group 4's first object says the one group before it does not exist (§10.8). + let mut c = TrackCache::default(); + whole(&mut c, 2); + let mut p = Properties::new(); + p.push(KeyValuePair::varint(prop::PRIOR_GROUP_ID_GAP, 1)); + c.stream_opened(4, true); + c.insert(4, 0, CachedObject { properties: p, ..obj(b"e") }); + c.stream_ended(4, true); + assert!(c.covers(Location::new(2, 0)), "group 3 known absent"); + assert!(c.covers(Location::new(3, 0)), "a fetch may start in a group that does not exist"); + assert!(!c.covers(Location::new(1, 0)), "group 1 unseen"); + } + + #[test] + fn what_breaks_coverage() { + // A reset stream: the group may be missing objects. + let mut c = TrackCache::default(); + c.stream_opened(0, true); + c.insert(0, 0, obj(b"a")); + c.stream_ended(0, false); + whole(&mut c, 1); + assert!(!c.covers(Location::new(0, 0)) && c.covers(Location::new(1, 0))); + // Joined mid-group: group 7 began at object 3, so its start is unknown. + let mut c = TrackCache::default(); + c.stream_opened(7, false); + c.insert(7, 3, obj(b"m")); + c.stream_ended(7, true); + whole(&mut c, 8); + assert!(!c.covers(Location::new(7, 0)) && c.covers(Location::new(8, 0))); + // Unless a Prior Object ID Gap says objects 0 to 2 do not exist (§10.9). + let mut p = Properties::new(); + p.push(KeyValuePair::varint(prop::PRIOR_OBJECT_ID_GAP, 3)); + let mut c = TrackCache::default(); + c.stream_opened(7, false); + c.insert(7, 3, CachedObject { properties: p, ..obj(b"m") }); + c.stream_ended(7, true); + whole(&mut c, 8); + assert!(c.covers(Location::new(7, 0))); + // Datagrams: whole with the start and an END_OF_GROUP. + let mut c = TrackCache::default(); + c.insert(0, 0, obj(b"d0")); + c.insert(0, 1, obj(b"d1")); + whole(&mut c, 1); + assert!(!c.covers(Location::new(0, 0))); + c.group_ended(0); + assert!(c.covers(Location::new(0, 0))); + // Eviction: nothing at or below an evicted group is covered. + evict(&mut c, 0); + assert!(!c.covers(Location::new(0, 0)) && c.covers(Location::new(1, 0))); + assert_eq!(c.insert(0, 2, obj(b"late")), Insert::Evicted); + } + + #[test] + fn the_budget_evicts_oldest_groups_first() { + let mut b = Budget::new(100); + assert!(b.add(1, 0, 60).is_empty()); + assert!(b.add(2, 0, 30).is_empty()); + assert_eq!(b.add(1, 1, 20), vec![(1, 0)]); + assert_eq!(b.used, 50); + b.forget(2); + assert_eq!(b.used, 20); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/cert.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/cert.rs new file mode 100644 index 000000000..08bd4c4d2 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/cert.rs @@ -0,0 +1,135 @@ +//! The relay's TLS identity, and the TLS configuration of its UDP endpoint. +//! +//! In production the certificate chain and key come from PEM files named in the +//! configuration. In development the relay makes its own: ECDSA P-256 and valid for +//! fourteen days from now, which are the constraints a browser puts on a certificate it +//! trusts by hash (W3C WebTransport, `serverCertificateHashes`: "the total length of the +//! validity period MUST NOT exceed two weeks", and ECDSA with secp256r1 an allowed key). +//! The relay prints the certificate's SHA-256 so a page can pass it to the browser. + +use base64::Engine; +use rustls::pki_types::{CertificateDer, PrivateKeyDer, PrivatePkcs8KeyDer}; +use std::sync::{Arc, Once}; + +static PROVIDER: Once = Once::new(); + +/// Install the ring crypto provider as rustls's process default (once). +pub fn install_provider() { + PROVIDER.call_once(|| { + let _ = rustls::crypto::ring::default_provider().install_default(); + }); +} + +/// A certificate chain and its private key. +pub struct RelayIdentity { + pub chain: Vec>, + pub key: PrivateKeyDer<'static>, + /// SHA-256 of the leaf certificate's DER. + pub leaf_sha256: [u8; 32], +} + +impl Clone for RelayIdentity { + fn clone(&self) -> Self { + RelayIdentity { chain: self.chain.clone(), key: self.key.clone_key(), leaf_sha256: self.leaf_sha256 } + } +} + +/// SHA-256 of a certificate's DER: the value `serverCertificateHashes` compares. +pub fn sha256(der: &[u8]) -> [u8; 32] { + let d = ring::digest::digest(&ring::digest::SHA256, der); + let mut out = [0u8; 32]; + out.copy_from_slice(d.as_ref()); + out +} + +impl RelayIdentity { + /// A self-signed identity for development: ECDSA P-256, valid fourteen days from now, + /// for `names` (DNS names or IP addresses). + pub fn dev_self_signed(names: &[String]) -> anyhow_like::Result { + let id = wtransport::Identity::self_signed_builder() + .subject_alt_names(names.iter().map(String::as_str)) + .from_now_utc() + .validity_days(14) + .build() + .map_err(|e| anyhow_like::Error(format!("self-signed certificate: {e}")))?; + Self::from_wtransport(&id) + } + + /// Load a chain and key from PEM files. + pub async fn from_pem_files(cert: &std::path::Path, key: &std::path::Path) -> anyhow_like::Result { + let id = wtransport::Identity::load_pemfiles(cert, key) + .await + .map_err(|e| anyhow_like::Error(format!("loading {} and {}: {e}", cert.display(), key.display())))?; + Self::from_wtransport(&id) + } + + fn from_wtransport(id: &wtransport::Identity) -> anyhow_like::Result { + let chain: Vec> = id.certificate_chain().as_slice().iter().map(|c| CertificateDer::from(c.der().to_vec())).collect(); + let leaf = chain.first().ok_or_else(|| anyhow_like::Error("an empty certificate chain".into()))?; + let leaf_sha256 = sha256(leaf.as_ref()); + let key = PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(id.private_key().secret_der().to_vec())); + Ok(RelayIdentity { chain, key, leaf_sha256 }) + } + + /// The leaf hash in the forms a page needs: hex, base64, and the byte array a + /// `serverCertificateHashes` entry takes (`{algorithm: "sha-256", value}`). + pub fn hash_lines(&self) -> Vec { + let b = &self.leaf_sha256; + vec![ + format!("certificate sha-256 (hex): {}", hex::encode(b)), + format!("certificate sha-256 (base64): {}", base64::engine::general_purpose::STANDARD.encode(b)), + format!( + "serverCertificateHashes: [{{ algorithm: \"sha-256\", value: new Uint8Array([{}]) }}]", + b.iter().map(|x| x.to_string()).collect::>().join(", ") + ), + ] + } + + /// The TLS configuration of the relay's UDP endpoint: TLS 1.3, offering the native + /// MOQT ALPN first and `h3` (for WebTransport) second. + pub fn server_tls(&self) -> anyhow_like::Result { + install_provider(); + let provider = Arc::new(rustls::crypto::ring::default_provider()); + let mut tls = rustls::ServerConfig::builder_with_provider(provider) + .with_protocol_versions(&[&rustls::version::TLS13]) + .map_err(|e| anyhow_like::Error(e.to_string()))? + .with_no_client_auth() + .with_single_cert(self.chain.clone(), self.key.clone_key()) + .map_err(|e| anyhow_like::Error(format!("the certificate and key: {e}")))?; + tls.alpn_protocols = vec![joule_moq::draft22::ALPN.to_vec(), b"h3".to_vec()]; + tls.max_early_data_size = 0; + Ok(tls) + } +} + +/// A small error type for setup paths, so the crate needs no general error dependency. +pub mod anyhow_like { + #[derive(Debug, Clone, PartialEq, Eq)] + pub struct Error(pub String); + impl std::fmt::Display for Error { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(&self.0) + } + } + impl std::error::Error for Error {} + pub type Result = std::result::Result; +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_dev_certificate_is_one_a_browser_takes_by_hash() { + let id = RelayIdentity::dev_self_signed(&["localhost".into(), "127.0.0.1".into()]).unwrap(); + assert_eq!(id.chain.len(), 1); + assert_ne!(id.leaf_sha256, [0; 32]); + // The hash is SHA-256 of the DER, the value a page passes; the WebTransport + // library's own digest of the same certificate agrees. + let theirs = wtransport::tls::Certificate::from_der(id.chain[0].as_ref().to_vec()).unwrap().hash(); + assert_eq!(theirs.as_ref(), &id.leaf_sha256); + let tls = id.server_tls().unwrap(); + assert_eq!(tls.alpn_protocols, vec![b"moqt-22".to_vec(), b"h3".to_vec()]); + assert!(id.hash_lines()[0].contains(&hex::encode(id.leaf_sha256))); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/client.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/client.rs new file mode 100644 index 000000000..c1dbaf5f1 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/client.rs @@ -0,0 +1,324 @@ +//! A small MOQT -22 client over any of the relay's transports: enough to publish and +//! subscribe through the relay, for its tests and for tools. Every wait in it is bounded. + +use crate::transport::{BoxSend, SharedConnection}; +use crate::wire::{open_request, responder, MessageReader, ReqRx, ReqTx, UniReader}; +use joule_moq::carriage::SETUP_OPTION_JWP_RECEIPTS; +use joule_moq::draft22::data::{FetchEntry, SubgroupCodec, SubgroupHeader, SubgroupObject}; +use joule_moq::draft22::kvp::{opt, KeyValuePair, Properties, SetupOptions}; +use joule_moq::draft22::{ + ControlMessage, FetchOk, Goaway, GroupOrder, NamespaceRequest, Parameters, Perspective, Publish, RequestIds, Setup, StreamHeader, SubscribeOk, Token, + TrackRequest, UniEvent, +}; +use joule_moq::{TrackName, TrackNamespace, JWP_MOQ_EXTENSION_VERSION}; +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; +use std::time::Duration; +use tokio::sync::{mpsc, oneshot, watch}; +use tokio::task::JoinHandle; +use tokio::time::timeout; + +/// How long any one step of the client waits. +pub const STEP: Duration = Duration::from_secs(10); + +#[derive(Clone, Debug)] +pub struct ClientOptions { + pub jwp: bool, + pub tokens: Vec, + pub max_request_updates: u64, + pub max_object_bytes: u64, + /// Read nothing of a data stream until [`Client::next_data`] asks for it, so unread + /// streams hold the transport's flow control (a subscriber that reads slowly). Fetch + /// streams are then not routed to [`Client::fetch`]. + pub lazy_data: bool, +} + +impl Default for ClientOptions { + fn default() -> Self { + ClientOptions { jwp: true, tokens: Vec::new(), max_request_updates: 0, max_object_bytes: 16 << 20, lazy_data: false } + } +} + +/// A request the relay opened to this client (a SUBSCRIBE or FETCH for a namespace it +/// published), its first message, and its stream. +pub type Incoming = (ReqTx, ReqRx, ControlMessage); + +pub struct Client { + pub conn: SharedConnection, + ids: Mutex, + pub peer_setup: Setup, + goaway: watch::Receiver>, + _control: tokio::sync::Mutex, + data: tokio::sync::Mutex>, + requests: tokio::sync::Mutex>, + pending_fetch: Arc>>>, + my_updates: u64, + tasks: Vec>, + /// A stream [`Client::next_data`] took and gave up waiting on before its header came: + /// offered first on the next call, so no stream is lost to a timeout. + held: tokio::sync::Mutex>, + /// Each time `next_data` gave up waiting on a header: how many streams were queued + /// behind the one it waited on. + pub waited_out: Mutex>, +} + +impl Drop for Client { + fn drop(&mut self) { + for t in &self.tasks { + t.abort(); + } + } +} + +fn s(e: E) -> String { + e.to_string() +} + +impl Client { + /// Open this end's control stream with SETUP, and read the relay's (§6.3). + pub async fn connect(conn: SharedConnection, opts: ClientOptions) -> Result { + let mut o = SetupOptions::new(); + o.push(KeyValuePair::bytes(opt::MOQT_IMPLEMENTATION, format!("joule-moq-relay-client/{}", env!("CARGO_PKG_VERSION")).into_bytes())); + if opts.max_request_updates != 0 { + o.push(KeyValuePair::varint(opt::MAX_REQUEST_UPDATES, opts.max_request_updates)); + } + for t in &opts.tokens { + o.push(KeyValuePair::bytes(opt::AUTHORIZATION_TOKEN, t.to_bytes())); + } + if opts.jwp { + o.push(KeyValuePair::varint(SETUP_OPTION_JWP_RECEIPTS, JWP_MOQ_EXTENSION_VERSION as u64)); + } + let mut control = timeout(STEP, conn.open_uni()).await.map_err(s)?.map_err(s)?; + control.write_all(&ControlMessage::Setup(Setup { options: o }).encode().map_err(s)?).await.map_err(s)?; + + let (data_tx, data_rx) = mpsc::unbounded_channel(); + let pending_fetch: Arc>>> = Arc::default(); + // The relay's control stream; data streams that come first are kept. + let (peer_setup, mut ctl) = timeout(STEP, async { + loop { + let recv = conn.accept_uni().await.map_err(s)?; + let mut r = UniReader::new(recv, opts.max_object_bytes); + match r.start().await.map_err(s)? { + StreamHeader::Control => match r.next().await.map_err(s)? { + Some(UniEvent::Control(ControlMessage::Setup(st))) => return Ok::<_, String>((st, r)), + _ => return Err("the relay's control stream does not begin with SETUP".to_string()), + }, + _ => { + let _ = data_tx.send(r); + } + } + } + }) + .await + .map_err(|_| "no SETUP from the relay".to_string())??; + peer_setup.check_from(Perspective::Server, false).map_err(|c| format!("the relay's SETUP: {c}"))?; + + let (gtx, grx) = watch::channel(None); + let mut tasks = Vec::new(); + tasks.push(tokio::spawn(async move { + while let Ok(Some(ev)) = ctl.next().await { + if let UniEvent::Control(ControlMessage::Goaway(g)) = ev { + let _ = gtx.send(Some(g)); + } + } + })); + let (c2, pf, max, lazy) = (conn.clone(), pending_fetch.clone(), opts.max_object_bytes, opts.lazy_data); + tasks.push(tokio::spawn(async move { + while let Ok(recv) = c2.accept_uni().await { + let mut r = UniReader::new(recv, max); + if lazy { + let _ = data_tx.send(r); + continue; + } + let (pf, data_tx) = (pf.clone(), data_tx.clone()); + tokio::spawn(async move { + if let Ok(Ok(StreamHeader::Fetch(fh))) = timeout(STEP, r.start()).await { + let waiting = pf.lock().unwrap().remove(&fh.request_id); + if let Some(tx) = waiting { + let _ = tx.send(r); + return; + } + } + let _ = data_tx.send(r); + }); + } + })); + let (req_tx, req_rx) = mpsc::unbounded_channel(); + let (c3, my, peer) = (conn.clone(), opts.max_request_updates, peer_setup.options.max_request_updates()); + tasks.push(tokio::spawn(async move { + while let Ok((send, recv)) = c3.accept_bi().await { + let mut reader = MessageReader::new(recv); + let req_tx = req_tx.clone(); + tokio::spawn(async move { + if let Ok(Ok(Some(first))) = timeout(STEP, reader.next()).await + && let Ok((tx, rx)) = responder(send, reader, &first, my, peer) + { + let _ = req_tx.send((tx, rx, first)); + } + }); + } + })); + Ok(Client { + conn, + ids: Mutex::new(RequestIds::new(Perspective::Client)), + peer_setup, + goaway: grx, + _control: tokio::sync::Mutex::new(control), + data: tokio::sync::Mutex::new(data_rx), + requests: tokio::sync::Mutex::new(req_rx), + pending_fetch, + my_updates: opts.max_request_updates, + tasks, + held: tokio::sync::Mutex::new(None), + waited_out: Mutex::new(Vec::new()), + }) + } + + pub fn next_request_id(&self) -> u64 { + self.ids.lock().unwrap().next().expect("Request IDs left") + } + + /// Open a request stream with `m` as its first message. + pub async fn request(&self, m: &ControlMessage) -> Result<(ReqTx, ReqRx), String> { + timeout(STEP, open_request(&*self.conn, m, self.my_updates, self.peer_setup.options.max_request_updates())).await.map_err(s)?.map_err(s) + } + + /// The first response on a request stream. + pub async fn response(rx: &mut ReqRx) -> Result { + match timeout(STEP, rx.next()).await { + Ok(Ok(Some(m))) => Ok(m), + Ok(Ok(None)) => Err("the stream ended before its response".into()), + Ok(Err(e)) => Err(e.to_string()), + Err(_) => Err("no response in time".into()), + } + } + + fn refusal(m: ControlMessage) -> String { + match m { + ControlMessage::RequestError(e) => format!("REQUEST_ERROR {}: {}", e.code, String::from_utf8_lossy(&e.reason)), + other => format!("an unexpected {}", other.name()), + } + } + + pub async fn publish_namespace(&self, prefix: TrackNamespace, params: Parameters) -> Result<(ReqTx, ReqRx), String> { + let m = ControlMessage::PublishNamespace(NamespaceRequest { request_id: self.next_request_id(), prefix, params }); + let (tx, mut rx) = self.request(&m).await?; + match Self::response(&mut rx).await? { + ControlMessage::RequestOk(_) => Ok((tx, rx)), + other => Err(Self::refusal(other)), + } + } + + pub async fn publish(&self, namespace: TrackNamespace, name: TrackName, track_alias: u64, params: Parameters, track_properties: Properties) -> Result<(ReqTx, ReqRx), String> { + let m = ControlMessage::Publish(Publish { request_id: self.next_request_id(), namespace, name, track_alias, params, track_properties }); + let (tx, mut rx) = self.request(&m).await?; + match Self::response(&mut rx).await? { + ControlMessage::RequestOk(_) => Ok((tx, rx)), + other => Err(Self::refusal(other)), + } + } + + pub async fn subscribe(&self, namespace: TrackNamespace, name: TrackName, params: Parameters) -> Result<(SubscribeOk, ReqTx, ReqRx), String> { + let m = ControlMessage::Subscribe(TrackRequest { request_id: self.next_request_id(), namespace, name, params }); + let (tx, mut rx) = self.request(&m).await?; + match Self::response(&mut rx).await? { + ControlMessage::SubscribeOk(ok) => Ok((ok, tx, rx)), + other => Err(Self::refusal(other)), + } + } + + pub async fn subscribe_namespace(&self, prefix: TrackNamespace) -> Result<(ReqTx, ReqRx), String> { + let m = ControlMessage::SubscribeNamespace(NamespaceRequest { request_id: self.next_request_id(), prefix, params: Parameters::new() }); + let (tx, mut rx) = self.request(&m).await?; + match Self::response(&mut rx).await? { + ControlMessage::RequestOk(_) => Ok((tx, rx)), + other => Err(Self::refusal(other)), + } + } + + /// FETCH, and read its whole fetch stream (each read bounded by [`STEP`]). + pub async fn fetch(&self, namespace: TrackNamespace, name: TrackName, params: Parameters) -> Result<(FetchOk, Vec), String> { + let order = params.group_order().unwrap_or(GroupOrder::Ascending); + let id = self.next_request_id(); + let (otx, orx) = oneshot::channel(); + self.pending_fetch.lock().unwrap().insert(id, otx); + let m = ControlMessage::Fetch(TrackRequest { request_id: id, namespace, name, params }); + let (_tx, mut rx) = self.request(&m).await?; + let ok = match Self::response(&mut rx).await? { + ControlMessage::FetchOk(ok) => ok, + other => { + self.pending_fetch.lock().unwrap().remove(&id); + return Err(Self::refusal(other)); + } + }; + let mut r = timeout(STEP, orx).await.map_err(|_| "no fetch stream in time".to_string())?.map_err(s)?; + r.set_group_order(order); + let mut out = Vec::new(); + loop { + match timeout(STEP, r.next()).await.map_err(|_| "a fetch stream stalled".to_string())?.map_err(s)? { + Some(UniEvent::FetchEntry(e)) => out.push(e), + Some(_) => return Err("a fetch stream carried something else".into()), + None => break, + } + } + Ok((ok, out)) + } + + /// The next data stream the relay opened, its header read. A stream reset before its + /// header is passed over. + pub async fn next_data(&self, wait: Duration) -> Option { + let deadline = tokio::time::Instant::now() + wait; + let mut held = self.held.lock().await; + let mut rx = self.data.lock().await; + loop { + let mut r = match held.take() { + Some(r) => r, + None => tokio::time::timeout_at(deadline, rx.recv()).await.ok().flatten()?, + }; + match tokio::time::timeout_at(deadline, r.start()).await { + Ok(Ok(_)) => return Some(r), + // Reset before its header: passed over. + Ok(Err(_)) => continue, + Err(_) => { + self.waited_out.lock().unwrap().push(rx.len()); + *held = Some(r); + return None; + } + } + } + } + + /// The next data stream the relay opened, unread: its header not yet waited for. + pub async fn next_stream(&self, wait: Duration) -> Option { + if let Some(r) = self.held.lock().await.take() { + return Some(r); + } + timeout(wait, async { self.data.lock().await.recv().await }).await.ok().flatten() + } + + /// The next request the relay opened to this client. + pub async fn next_request(&self, wait: Duration) -> Option { + timeout(wait, async { self.requests.lock().await.recv().await }).await.ok().flatten() + } + + pub fn goaway(&self) -> watch::Receiver> { + self.goaway.clone() + } + + /// One subgroup stream: its header, its objects, a FIN. + pub async fn send_subgroup(&self, header: SubgroupHeader, objects: &[SubgroupObject]) -> Result<(), String> { + let mut b = Vec::new(); + header.encode(&mut b); + let mut c = SubgroupCodec::new(header); + for o in objects { + c.encode_object(o, &mut b).map_err(s)?; + } + let mut st = timeout(STEP, self.conn.open_uni()).await.map_err(s)?.map_err(s)?; + timeout(STEP, st.write_all(&b)).await.map_err(s)?.map_err(s)?; + st.finish().map_err(s) + } + + pub fn close(&self, code: u64, reason: &str) { + self.conn.close(code, reason.as_bytes()); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/config.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/config.rs new file mode 100644 index 000000000..283821e58 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/config.rs @@ -0,0 +1,325 @@ +//! The relay's configuration, read from TOML. +//! +//! Nothing secret lives in a configuration file: the TLS key and the summary-signing key +//! are paths to files the operator provides, and authorization grants name tokens by +//! their SHA-256, never by value. `relay.example.toml` beside this crate shows every +//! field. + +use serde::Deserialize; +use std::net::SocketAddr; +use std::path::PathBuf; + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RelayConfig { + /// UDP address for native QUIC (ALPN `moqt-22`) and WebTransport (ALPN `h3`), one + /// port for both. Absent: no UDP endpoint. + #[serde(default)] + pub quic_listen: Option, + /// TCP address of the WebSocket fallback. Absent: none. + #[serde(default)] + pub websocket_listen: Option, + /// The path a WebTransport CONNECT and a WebSocket upgrade use. + #[serde(default = "default_path")] + pub path: String, + #[serde(default)] + pub tls: TlsConfig, + #[serde(default)] + pub cache: CacheConfig, + #[serde(default)] + pub limits: LimitsConfig, + #[serde(default)] + pub receipts: ReceiptsConfig, + #[serde(default)] + pub summary: SummaryConfig, + #[serde(default)] + pub auth: AuthConfig, + /// The New Session URI a GOAWAY names (§9.2); empty: reuse the current one. + #[serde(default)] + pub goaway_uri: String, +} + +fn default_path() -> String { + "/moq".into() +} + +#[derive(Clone, Debug, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct TlsConfig { + /// PEM certificate chain and PKCS#8 key. Both or neither. + pub cert_path: Option, + pub key_path: Option, + /// Without files: make a development certificate for these names (ECDSA P-256, valid + /// fourteen days) and print its SHA-256 for `serverCertificateHashes`. + #[serde(default = "default_dev_names")] + pub dev_names: Vec, +} + +fn default_dev_names() -> Vec { + vec!["localhost".into(), "127.0.0.1".into(), "::1".into()] +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CacheConfig { + /// Bytes of object payload the relay keeps across all tracks; past it the oldest + /// groups go first. + #[serde(default = "d_cache_bytes")] + pub max_bytes: usize, +} + +fn d_cache_bytes() -> usize { + 256 << 20 +} + +impl Default for CacheConfig { + fn default() -> Self { + CacheConfig { max_bytes: d_cache_bytes() } + } +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct LimitsConfig { + #[serde(default = "d_sessions")] + pub max_sessions: usize, + /// Requests a peer may have open at once: the bidirectional stream limit + /// (MAX_STREAMS on QUIC, WT_MAX_STREAMS on the WebSocket binding). -22 bounds + /// requests in flight by stream limits alone (§6.4.2). + #[serde(default = "d_requests")] + pub max_requests: u64, + /// Unidirectional streams a peer may have open at once (its data streams). + #[serde(default = "d_uni")] + pub max_uni_streams: u64, + /// MAX_REQUEST_UPDATES the relay declares (§9.1.7); 0 is no limit. + #[serde(default = "d_updates")] + pub max_request_updates: u64, + /// MAX_AUTH_TOKEN_CACHE_SIZE the relay declares (§9.1.3); 0 forbids aliases. + #[serde(default = "d_token_cache")] + pub max_auth_token_cache_size: u64, + /// The longest RENDEZVOUS_TIMEOUT the relay honours, in milliseconds (§9.20.6). + #[serde(default = "d_rendezvous")] + pub max_rendezvous_ms: u64, + /// How long an upstream SUBSCRIBE or FETCH may take to be answered. + #[serde(default = "d_upstream")] + pub upstream_timeout_ms: u64, + /// The relay's FILL_TIMEOUT budget when a request names none, and its ceiling when + /// one does (§9.20.5). + #[serde(default = "d_fill")] + pub fill_timeout_ms: u64, + /// Writes in flight per downstream session: the scheduler hands the transport the + /// highest-priority object each time one finishes. + #[serde(default = "d_writes")] + pub write_concurrency: usize, + #[serde(default = "d_object")] + pub max_object_bytes: u64, + /// How long a data stream whose Track Alias is not yet known is held (§3.1.3.1). + #[serde(default = "d_alias_wait")] + pub unknown_alias_wait_ms: u64, + /// The Timeout a shutdown GOAWAY gives (§9.2). + #[serde(default = "d_goaway")] + pub goaway_timeout_ms: u64, + /// Bytes received and unread before a WebSocket session closes. + #[serde(default = "d_ws_buffer")] + pub websocket_buffer_bytes: usize, + /// QUIC idle timeout. + #[serde(default = "d_idle")] + pub idle_timeout_ms: u64, + /// Bytes a QUIC or WebTransport session may have sent and unacknowledged, or + /// buffered, before a write waits. Kept near the path's bandwidth-delay product, so + /// what waits waits in the relay's scheduler, where priority and delivery timeouts + /// still apply (§5.2), rather than in the transport's buffers. + #[serde(default = "d_send_window")] + pub quic_send_window_bytes: u64, +} + +fn d_sessions() -> usize { + 4096 +} +fn d_requests() -> u64 { + 256 +} +fn d_uni() -> u64 { + 1024 +} +fn d_updates() -> u64 { + 16 +} +fn d_token_cache() -> u64 { + 16 * 1024 +} +fn d_rendezvous() -> u64 { + 30_000 +} +fn d_upstream() -> u64 { + 5_000 +} +fn d_fill() -> u64 { + 2_000 +} +fn d_writes() -> usize { + 8 +} +fn d_object() -> u64 { + 16 << 20 +} +fn d_alias_wait() -> u64 { + 1_000 +} +fn d_goaway() -> u64 { + 5_000 +} +fn d_ws_buffer() -> usize { + 64 << 20 +} +fn d_idle() -> u64 { + 30_000 +} +fn d_send_window() -> u64 { + 1 << 20 +} + +impl Default for LimitsConfig { + fn default() -> Self { + LimitsConfig { + max_sessions: d_sessions(), + max_requests: d_requests(), + max_uni_streams: d_uni(), + max_request_updates: d_updates(), + max_auth_token_cache_size: d_token_cache(), + max_rendezvous_ms: d_rendezvous(), + upstream_timeout_ms: d_upstream(), + fill_timeout_ms: d_fill(), + write_concurrency: d_writes(), + max_object_bytes: d_object(), + unknown_alias_wait_ms: d_alias_wait(), + goaway_timeout_ms: d_goaway(), + websocket_buffer_bytes: d_ws_buffer(), + idle_timeout_ms: d_idle(), + quic_send_window_bytes: d_send_window(), + } + } +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ReceiptsConfig { + /// A track whose name ends with this is the receipts track of the track named + /// without it, in the same namespace (WAI `moq-streaming-format.md` §4.1 names a + /// media track's receipts track in its catalog; this is the relay's local rule for + /// recognising one without reading the catalog). + #[serde(default = "d_suffix")] + pub receipts_track_suffix: String, + /// Namespaces whose objects are checked against their receipts before they are + /// forwarded. Elsewhere the relay forwards receipts tracks as opaque objects. + #[serde(default)] + pub door: Vec, +} + +fn d_suffix() -> String { + "/receipts".into() +} + +impl Default for ReceiptsConfig { + fn default() -> Self { + ReceiptsConfig { receipts_track_suffix: d_suffix(), door: Vec::new() } + } +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DoorRule { + /// The Track Namespace prefix, one entry per field. + pub prefix: Vec, + /// Publisher keys (Ed25519, hex) whose group receipts the door accepts. Empty: any key + /// whose signature verifies, the subscriber binding trust itself. + #[serde(default)] + pub trust: Vec, + /// How long an object waits at the door for its group's receipts. + #[serde(default = "d_hold")] + pub hold_ms: u64, +} + +fn d_hold() -> u64 { + 2_000 +} + +#[derive(Clone, Debug, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct SummaryConfig { + /// A file holding the relay's Ed25519 signing seed (32 bytes, raw or hex). Absent: a + /// key made at start, its public half logged. + pub key_path: Option, + /// A file the signed session summaries are appended to, one JSON object a line. + pub out_path: Option, +} + +#[derive(Clone, Debug, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct AuthConfig { + /// When true, every PUBLISH_NAMESPACE, PUBLISH, SUBSCRIBE_NAMESPACE, SUBSCRIBE, FETCH + /// and TRACK_STATUS must carry an AUTHORIZATION TOKEN (or rely on one from SETUP) that + /// a grant covers. + #[serde(default)] + pub required: bool, + #[serde(default)] + pub grants: Vec, +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Grant { + /// The namespace prefix the grant covers, one entry per field. + pub prefix: Vec, + /// SHA-256 of the token value, hex. The token itself is never in the configuration. + pub token_sha256: String, + #[serde(default)] + pub publish: bool, + #[serde(default)] + pub subscribe: bool, +} + +impl RelayConfig { + pub fn from_toml(s: &str) -> Result { + toml::from_str(s).map_err(|e| e.to_string()) + } + + /// Development defaults: both endpoints on loopback, a self-signed certificate. + pub fn dev() -> Self { + RelayConfig { + quic_listen: Some("127.0.0.1:4443".parse().expect("an address")), + websocket_listen: Some("127.0.0.1:4480".parse().expect("an address")), + path: default_path(), + tls: TlsConfig { cert_path: None, key_path: None, dev_names: default_dev_names() }, + cache: CacheConfig::default(), + limits: LimitsConfig::default(), + receipts: ReceiptsConfig::default(), + summary: SummaryConfig::default(), + auth: AuthConfig::default(), + goaway_uri: String::new(), + } + } + + /// Ephemeral loopback ports, for tests. + pub fn loopback() -> Self { + let mut c = Self::dev(); + c.quic_listen = Some("127.0.0.1:0".parse().expect("an address")); + c.websocket_listen = Some("127.0.0.1:0".parse().expect("an address")); + c + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_example_configuration_reads() { + let s = include_str!("../relay.example.toml"); + let c = RelayConfig::from_toml(s).expect("the example reads"); + assert!(c.quic_listen.is_some() && c.websocket_listen.is_some()); + assert_eq!(c.receipts.door.len(), 1); + assert_eq!(c.auth.grants.len(), 1); + assert!(RelayConfig::from_toml("unknown_field = 1").is_err()); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/door.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/door.rs new file mode 100644 index 000000000..81cd999e0 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/door.rs @@ -0,0 +1,268 @@ +//! Checking objects against their receipts before they are forwarded, in the namespaces +//! the configuration names. +//! +//! Elsewhere the relay forwards a receipts track as it forwards any track, its objects +//! opaque (WAI `moq-streaming-format.md` §4.1; MOQT §7.8), and every subscriber verifies +//! for itself. In a namespace with a door rule the relay is a relay that reads the receipts +//! it forwards, and takes on the duties of WAI `jwp-receipts.md` §4.1: +//! +//! - each group receipt, from the bundle that group *g* of a receipts track carries, has +//! its Ed25519 signature verified (under the rule's trusted keys, or any key) before the +//! first object of media group *g* is fanned out. A group that does not verify is +//! dropped, its bundle and its objects, and a structured error is logged for the +//! upstream operator; +//! - each media object is checked against the receipt of its own position in the bundle: +//! the receipt names this group and Object ID (`moqt_mismatch` otherwise), its content +//! hash is the payload's, and its Merkle proof reaches the signed root. An object that +//! carries its receipt as a property must carry that same receipt; +//! - an object whose bundle has not arrived waits for it, up to the rule's `hold_ms` +//! (the publisher publishes the bundle no later than the group's first object), and is +//! refused after that; +//! - receipt bytes are forwarded unaltered; the relay re-signs nothing. +//! +//! The door reuses `joule-moq`'s verifier ([`ReceiptVerifier`]) and bundle reader. + +use crate::config::DoorRule; +use joule_moq::carriage::ReceiptBundle; +use joule_moq::draft22::names::prefix_matches; +use joule_moq::receipt::{ObjectReceipt, ReceiptVerifier, VerifiedGroup}; +use joule_moq::{GroupId, ObjectId, TrackName, TrackNamespace}; +use std::collections::{HashMap, VecDeque}; +use std::sync::Mutex; +use std::time::Duration; +use tokio::sync::Notify; + +/// How many groups of each track the door remembers. +const GROUPS_HELD: usize = 4096; + +struct Rule { + prefix: TrackNamespace, + verifier: ReceiptVerifier, + hold: Duration, +} + +enum Entry { + Verified { group: VerifiedGroup, objects: HashMap }, + Refused(String), +} + +type MediaKey = (TrackNamespace, TrackName); + +#[derive(Default)] +struct Groups { + entries: HashMap<(MediaKey, u64), Entry>, + order: VecDeque<(MediaKey, u64)>, +} + +pub struct Door { + rules: Vec, + suffix: Vec, + groups: Mutex, + arrived: Notify, +} + +/// What the door decided about one object. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Verdict { + /// No rule covers the namespace. + Unchecked, + /// The object's receipt verified: its signed figure and leaf class. + Pass { joules_micro: u64, class: Option> }, + Refuse(String), +} + +impl Door { + pub fn new(rules: &[DoorRule], suffix: &str) -> Result { + let mut out = Vec::new(); + for r in rules { + let mut anchors = Vec::new(); + for k in &r.trust { + let b: [u8; 32] = hex::decode(k).map_err(|e| format!("a door key: {e}"))?.try_into().map_err(|_| "a door key is not 32 bytes".to_string())?; + anchors.push(ed25519_dalek::VerifyingKey::from_bytes(&b).map_err(|e| format!("a door key: {e}"))?); + } + let verifier = if anchors.is_empty() { ReceiptVerifier::new() } else { ReceiptVerifier::with_anchors(anchors) }; + out.push(Rule { prefix: TrackNamespace::from_strs(r.prefix.iter().map(String::as_str)), verifier, hold: Duration::from_millis(r.hold_ms) }); + } + Ok(Door { rules: out, suffix: suffix.as_bytes().to_vec(), groups: Mutex::new(Groups::default()), arrived: Notify::new() }) + } + + fn rule(&self, ns: &TrackNamespace) -> Option<&Rule> { + self.rules.iter().filter(|r| prefix_matches(&r.prefix, ns)).max_by_key(|r| r.prefix.0.len()) + } + + /// Whether `ns` is checked at the door. + pub fn checks(&self, ns: &TrackNamespace) -> bool { + self.rule(ns).is_some() + } + + /// The media track a receipts track carries the receipts of, by the configured suffix. + pub fn media_of(&self, name: &TrackName) -> Option { + if self.suffix.is_empty() || name.0.len() <= self.suffix.len() || !name.0.ends_with(&self.suffix) { + return None; + } + Some(TrackName(name.0[..name.0.len() - self.suffix.len()].to_vec())) + } + + /// The receipts track of a media track. + pub fn receipts_of(&self, name: &TrackName) -> TrackName { + let mut v = name.0.clone(); + v.extend_from_slice(&self.suffix); + TrackName(v) + } + + fn remember(&self, key: (MediaKey, u64), e: Entry) { + let mut g = self.groups.lock().unwrap(); + if g.entries.insert(key.clone(), e).is_none() { + g.order.push_back(key); + while g.order.len() > GROUPS_HELD * self.rules.len().max(1) { + if let Some(old) = g.order.pop_front() { + g.entries.remove(&old); + } + } + } + drop(g); + self.arrived.notify_waiters(); + } + + /// Group `group` of the receipts track of `media` arrived, with this payload. `Ok` to + /// forward it, `Err` with why the group is dropped. + pub fn bundle(&self, ns: &TrackNamespace, media: &TrackName, group: u64, payload: &[u8]) -> Result<(), String> { + let Some(rule) = self.rule(ns) else { return Ok(()) }; + let key = ((ns.clone(), media.clone()), group); + let checked = (|| { + let b = ReceiptBundle::from_payload(payload).map_err(|e| e.to_string())?; + if b.group.group_id != GroupId(group) { + return Err("moqt_mismatch: the group receipt names another group".to_string()); + } + let vg = rule.verifier.verify_group(&b.group).map_err(|e| e.to_string())?; + let mut objects = HashMap::new(); + for o in b.objects { + o.check_moqt(b.group.track_alias, GroupId(group), o.object_id).map_err(|_| "moqt_mismatch: an object receipt of another track or group".to_string())?; + rule.verifier.verify_object(&o, &vg).map_err(|e| format!("an object receipt: {e}"))?; + objects.insert(o.object_id.0, o); + } + Ok((vg, objects)) + })(); + match checked { + Ok((group, objects)) => { + self.remember(key, Entry::Verified { group, objects }); + Ok(()) + } + Err(why) => { + self.remember(key, Entry::Refused(why.clone())); + Err(why) + } + } + } + + /// Check one media object. `property` is the receipt the object carries as its + /// JWP_OBJECT_RECEIPT property, if any. + pub async fn object(&self, ns: &TrackNamespace, media: &TrackName, group: u64, object: u64, payload: &[u8], property: Option<&ObjectReceipt>) -> Verdict { + let Some(rule) = self.rule(ns) else { return Verdict::Unchecked }; + let key = ((ns.clone(), media.clone()), group); + let deadline = tokio::time::Instant::now() + rule.hold; + loop { + let notified = self.arrived.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + { + let g = self.groups.lock().unwrap(); + match g.entries.get(&key) { + Some(Entry::Refused(why)) => return Verdict::Refuse(format!("its group was refused: {why}")), + Some(Entry::Verified { group: vg, objects }) => { + let Some(r) = objects.get(&object) else { + return Verdict::Refuse("no receipt for its position in the group".into()); + }; + if r.object_id != ObjectId(object) { + return Verdict::Refuse("moqt_mismatch".into()); + } + if let Some(p) = property + && p != r + { + return Verdict::Refuse("its receipt property is not the receipt its group signed".into()); + } + return match rule.verifier.verify_payload(payload, r, vg) { + Ok(()) => Verdict::Pass { joules_micro: r.joules_micro, class: r.energy_class.clone() }, + Err(e) => Verdict::Refuse(format!("its payload does not verify: {e}")), + }; + } + None => {} + } + } + if tokio::time::timeout_at(deadline, notified).await.is_err() { + return Verdict::Refuse("its group's receipts did not arrive in time".into()); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use joule_moq::receipt::ReceiptBuilder; + use joule_moq::TrackAlias; + + fn bundle(key: &ed25519_dalek::SigningKey, group: u64, payloads: &[&[u8]]) -> ReceiptBundle { + let mut b = ReceiptBuilder::new(TrackAlias(8), GroupId(group)); + for p in payloads { + b.add_labelled_object(ObjectReceipt::hash_payload(p), 0, "o", "t", vec![0]).unwrap(); + } + let objects = (0..payloads.len()) + .map(|i| ObjectReceipt { + track_alias: TrackAlias(8), + group_id: GroupId(group), + object_id: ObjectId(i as u64), + content_hash: ObjectReceipt::hash_payload(payloads[i]), + joules_micro: 0, + origin: "o".into(), + merkle_proof: b.proof_for(i).unwrap(), + energy_class: Some(vec![0]), + }) + .collect(); + let group = b.finalize_labelled(key, "p", "t", None).unwrap(); + ReceiptBundle { group, objects } + } + + #[tokio::test] + async fn the_door_passes_what_its_group_signed() { + let key = ed25519_dalek::SigningKey::from_bytes(&[5; 32]); + let rule = DoorRule { prefix: vec!["live".into()], trust: vec![hex::encode(key.verifying_key().to_bytes())], hold_ms: 100 }; + let door = Door::new(&[rule], "/receipts").unwrap(); + let ns = TrackNamespace::from_strs(["live", "a"]); + let media = TrackName::from_str("cam"); + assert_eq!(door.media_of(&TrackName::from_str("cam/receipts")), Some(media.clone())); + assert_eq!(door.media_of(&media), None); + assert!(!door.checks(&TrackNamespace::from_strs(["other"]))); + + let b = bundle(&key, 3, &[b"x", b"y"]); + door.bundle(&ns, &media, 3, &b.to_payload().unwrap()).unwrap(); + assert!(matches!(door.object(&ns, &media, 3, 1, b"y", None).await, Verdict::Pass { .. })); + assert!(matches!(door.object(&ns, &media, 3, 1, b"tampered", None).await, Verdict::Refuse(_))); + assert!(matches!(door.object(&ns, &media, 3, 0, b"x", Some(&b.objects[1])).await, Verdict::Refuse(_)), "a property that is not its receipt"); + assert!(matches!(door.object(&ns, &media, 4, 0, b"x", None).await, Verdict::Refuse(_)), "no bundle within the hold"); + + // A bundle signed by a key the rule does not trust drops its group. + let stranger = ed25519_dalek::SigningKey::from_bytes(&[6; 32]); + let bad = bundle(&stranger, 5, &[b"z"]); + assert!(door.bundle(&ns, &media, 5, &bad.to_payload().unwrap()).is_err()); + assert!(matches!(door.object(&ns, &media, 5, 0, b"z", None).await, Verdict::Refuse(_))); + // A bundle under the wrong MOQT group is refused before its signature is read. + assert!(door.bundle(&ns, &media, 9, &bundle(&key, 8, &[b"q"]).to_payload().unwrap()).is_err()); + } + + #[tokio::test] + async fn an_object_waits_for_its_bundle() { + let key = ed25519_dalek::SigningKey::from_bytes(&[5; 32]); + let rule = DoorRule { prefix: vec![], trust: vec![], hold_ms: 2_000 }; + let door = std::sync::Arc::new(Door::new(&[rule], "/receipts").unwrap()); + let ns = TrackNamespace::from_strs(["any"]); + let media = TrackName::from_str("m"); + let d2 = door.clone(); + let (ns2, m2) = (ns.clone(), media.clone()); + let waiter = tokio::spawn(async move { d2.object(&ns2, &m2, 0, 0, b"late", None).await }); + tokio::time::sleep(Duration::from_millis(50)).await; + door.bundle(&ns, &media, 0, &bundle(&key, 0, &[b"late"]).to_payload().unwrap()).unwrap(); + let v = tokio::time::timeout(Duration::from_secs(5), waiter).await.unwrap().unwrap(); + assert!(matches!(v, Verdict::Pass { .. }), "{v:?}"); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/lib.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/lib.rs new file mode 100644 index 000000000..bca4896b4 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/lib.rs @@ -0,0 +1,44 @@ +//! **joule-moq-relay**: a MOQT relay for `draft-ietf-moq-transport-22`. +//! +//! - **Transports** ([`transport`]): native QUIC with ALPN `moqt-22` and WebTransport over +//! HTTP/3 (ALPN `h3`, the draft chosen by `WT-Available-Protocols`) on one UDP port; +//! and a WebSocket fallback that carries the same MOQT streams as +//! `draft-ietf-webtrans-http2-15` capsules, one to each message, for paths with no UDP. +//! - **Relay behaviour** ([`relay`], [`session`]): SETUP, namespace publication and +//! discovery (PUBLISH_NAMESPACE, SUBSCRIBE_NAMESPACE with NAMESPACE and +//! NAMESPACE_DONE), PUBLISH, SUBSCRIBE served from an upstream subscription the relay +//! holds or makes, RENDEZVOUS_TIMEOUT, FETCH and fills served from the cache or +//! upstream, TRACK_STATUS, REQUEST_UPDATE, GOAWAY on shutdown, Request ID rules and +//! MAX_REQUEST_UPDATES, request concurrency by stream limits, the authorization-token +//! alias cache ([`auth`]). +//! - **Fan-out** ([`sched`]): each downstream session's objects go out in §5.1.2's order +//! (subscriber priority, publisher priority, group order), with each subscriber's +//! delivery timeouts applied at this hop. Receipts tracks carry no timeout, so the +//! receipts of every dropped object still reach a subscriber of its receipts track. +//! - **Receipts at the door** ([`door`]): optional per namespace, with `joule-moq`'s +//! verifier. +//! - **Session summaries** ([`summary`]): bytes and objects per subscriber, signed by the +//! relay at session end; its own energy is reported as unmetered, since no meter backs +//! it. +//! +//! [`server::RelayServer`] binds the listeners; [`client::Client`] is a small client for +//! the tests and for tools. + +pub mod auth; +pub mod cache; +pub mod cert; +pub mod client; +pub mod config; +pub mod door; +pub mod relay; +pub mod sched; +pub mod server; +pub mod session; +pub mod summary; +pub mod transport; +pub mod wire; + +pub use config::RelayConfig; +pub use relay::{Relay, RelayStats}; +pub use server::RelayServer; +pub use summary::{SessionSummary, SignedSessionSummary}; diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/main.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/main.rs new file mode 100644 index 000000000..a324f04da --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/main.rs @@ -0,0 +1,58 @@ +//! `joule-moq-relay`: run the relay. +//! +//! ```text +//! joule-moq-relay --config relay.toml +//! joule-moq-relay --dev # loopback, a development certificate +//! ``` +//! +//! It logs where it listens, the certificate's SHA-256 (for a page's +//! `serverCertificateHashes`), and the key it signs session summaries with. On Ctrl-C it +//! sends GOAWAY to every session and waits for them to leave. + +use joule_moq_relay::{RelayConfig, RelayServer}; +use std::process::ExitCode; + +fn usage() -> ExitCode { + eprintln!("usage: joule-moq-relay (--config | --dev)"); + ExitCode::from(2) +} + +#[tokio::main] +async fn main() -> ExitCode { + let level = std::env::var("JOULE_MOQ_RELAY_LOG").ok().and_then(|l| l.parse().ok()).unwrap_or(tracing::Level::INFO); + tracing_subscriber::fmt().with_max_level(level).with_target(false).init(); + let args: Vec = std::env::args().skip(1).collect(); + let cfg = match args.iter().map(String::as_str).collect::>().as_slice() { + ["--dev"] => RelayConfig::dev(), + ["--config", path] => match std::fs::read_to_string(path).map_err(|e| e.to_string()).and_then(|s| RelayConfig::from_toml(&s)) { + Ok(c) => c, + Err(e) => { + eprintln!("joule-moq-relay: {path}: {e}"); + return ExitCode::FAILURE; + } + }, + _ => return usage(), + }; + let server = match RelayServer::start(cfg).await { + Ok(s) => s, + Err(e) => { + eprintln!("joule-moq-relay: {e}"); + return ExitCode::FAILURE; + } + }; + if let Some(a) = server.quic_addr { + tracing::info!("native QUIC (moqt-22) and WebTransport (h3) on udp {a}, path {}", server.relay().config().path); + } + if let Some(a) = server.websocket_addr { + tracing::info!("WebSocket fallback (subprotocol moqt-22) on tcp {a}, path {}", server.relay().config().path); + } + for line in server.identity.hash_lines() { + tracing::info!("{line}"); + } + tracing::info!("session summaries signed by {}", hex::encode(server.relay().summary_key().to_bytes())); + let _ = tokio::signal::ctrl_c().await; + tracing::info!("shutting down: GOAWAY to every session"); + let closed = server.shutdown().await; + tracing::info!("stopped; {closed} session(s) closed at the GOAWAY timeout"); + ExitCode::SUCCESS +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/relay.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/relay.rs new file mode 100644 index 000000000..16076c58b --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/relay.rs @@ -0,0 +1,730 @@ +//! The relay's shared state: sessions, published namespaces, tracks with their upstream +//! and downstream subscriptions and their caches; and the fan-out of each object +//! (`draft-ietf-moq-transport-22` §7). +//! +//! Requests are handled in [`crate::session`]; this module holds what they share. + +use crate::auth::{Authorizer, Resolved, TokenCache}; +use crate::cache::{self, Budget, CachedObject, Insert, TrackCache}; +use crate::config::RelayConfig; +use crate::door::Door; +use crate::sched::{Item, LaneKey, LaneKind, ObjMeta, Op, Prio, SchedHandle}; +use crate::summary::{class_label, SignedSessionSummary, Tally}; +use crate::transport::{BoxSend, SharedConnection, TransportKind}; +use crate::wire::{ReqTx, UniReader}; +use bytes::Bytes; +use ed25519_dalek::SigningKey; +use joule_moq::draft22::data::{ObjectDatagram, SubgroupHeader, SubgroupId, SubgroupObject}; +use joule_moq::draft22::kvp::{prop, Properties}; +use joule_moq::draft22::names::{prefix_matches, render}; +use joule_moq::draft22::{GroupOrder, Location, PeerRequestIds, PublishDoneCode, RequestIds}; +use joule_moq::{TrackName, TrackNamespace}; +use std::collections::{HashMap, HashSet}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; +use tokio::sync::{mpsc, oneshot, watch, Notify}; +use tokio::time::Instant; + +pub type TrackKey = (TrackNamespace, TrackName); + +/// The default subscriber and publisher priority (§5.1.3, §9.20.7: 128). +pub const DEFAULT_PRIORITY: u8 = 128; + +/// One transport session at the relay, whichever role its peer plays. +pub struct SessionCtx { + pub id: u64, + pub conn: SharedConnection, + pub kind: TransportKind, + pub remote: String, + pub sched: SchedHandle, + pub tally: Arc>, + pub peer_ids: Mutex, + pub my_ids: Mutex, + pub tokens: Mutex, + pub setup_tokens: Vec, + /// MAX_REQUEST_UPDATES the peer declared, and the relay's. + pub peer_max_updates: u64, + pub my_max_updates: u64, + /// The relay's control stream to the peer, after its SETUP. + pub control: tokio::sync::Mutex>, + pub goaway_sent: AtomicBool, + /// Fetch streams awaited from this peer, by the relay's Request ID. + pub pending_fetch: Mutex>>, + pub opened_unix_ms: u64, + pub close_code: Mutex>, + pub jwp: bool, +} + +impl SessionCtx { + pub fn next_request_id(&self) -> Option { + self.my_ids.lock().unwrap().next() + } + + /// Close the session with a termination code (§12.2), recorded for the summary. + pub fn close(&self, code: u64, reason: &str) { + self.close_code.lock().unwrap().get_or_insert(code); + tracing::debug!(session = self.id, code, reason, "closing a session"); + self.conn.close(code, reason.as_bytes()); + } +} + +/// What a downstream subscription asked for, as REQUEST_UPDATE may change it. +#[derive(Clone, Debug)] +pub struct SubParams { + pub subscriber_priority: u8, + pub order: GroupOrder, + pub forward: bool, + pub start: Location, + pub end: Option, + /// OBJECT_DELIVERY_TIMEOUT and SUBGROUP_DELIVERY_TIMEOUT the subscriber set; 0 none. + pub object_timeout: u64, + pub subgroup_timeout: u64, +} + +impl SubParams { + pub fn passes(&self, loc: Location) -> bool { + self.forward && loc >= self.start && self.end.is_none_or(|e| loc <= e) + } +} + +/// A downstream subscription. +pub struct DownSub { + pub id: u64, + pub session: u64, + pub request_id: u64, + pub alias: u64, + pub track: TrackKey, + pub render: Arc, + pub sched: SchedHandle, + pub params: Mutex, + /// A subscription to a receipts track: no delivery timeout applies to it. + pub receipts_track: bool, + pub tx: ReqTx, + pub ended: watch::Sender, +} + +#[derive(Clone)] +pub enum UpstreamKind { + /// The publisher sent PUBLISH. + Publish, + /// The relay sent SUBSCRIBE, on this stream. + Subscribe(ReqTx), +} + +#[derive(Clone)] +pub struct Upstream { + pub session: u64, + pub alias: u64, + pub kind: UpstreamKind, +} + +pub struct Track { + pub id: u64, + pub key: TrackKey, + pub render: Arc, + pub props: Properties, + pub largest: Option, + pub upstreams: Vec, + pub downs: Vec>, + pub cache: TrackCache, + pub pub_object_timeout: u64, + pub pub_subgroup_timeout: u64, + pub default_priority: u8, + pub default_order: GroupOrder, + /// Which upstream session a (group, subgroup) is taken from (§7.3: deduplication). + pub owners: HashMap<(u64, u64), u64>, + pub ended: Option, +} + +impl Track { + fn new(id: u64, key: TrackKey) -> Self { + let render: Arc = render(&key.0, Some(&key.1)).into(); + Track { + id, + key, + render, + props: Properties::new(), + largest: None, + upstreams: Vec::new(), + downs: Vec::new(), + cache: TrackCache::default(), + pub_object_timeout: 0, + pub_subgroup_timeout: 0, + default_priority: DEFAULT_PRIORITY, + default_order: GroupOrder::Ascending, + owners: HashMap::new(), + ended: None, + } + } + + /// Take the properties a PUBLISH or SUBSCRIBE_OK carried (§10). + pub fn set_props(&mut self, props: Properties) { + let v = |ty| props.find(ty).ok().flatten().and_then(|k| k.as_varint()); + self.pub_object_timeout = v(prop::OBJECT_DELIVERY_TIMEOUT).unwrap_or(0); + self.pub_subgroup_timeout = v(prop::SUBGROUP_DELIVERY_TIMEOUT).unwrap_or(0); + self.default_priority = v(prop::DEFAULT_PUBLISHER_PRIORITY).and_then(|p| u8::try_from(p).ok()).unwrap_or(DEFAULT_PRIORITY); + self.default_order = match v(prop::DEFAULT_PUBLISHER_GROUP_ORDER) { + Some(2) => GroupOrder::Descending, + _ => GroupOrder::Ascending, + }; + self.props = props; + } + + pub fn note_largest(&mut self, l: Location) { + if self.largest.is_none_or(|x| l > x) { + self.largest = Some(l); + } + } +} + +pub struct PublishedNs { + pub namespace: TrackNamespace, + pub session: u64, + pub id: u64, +} + +pub enum NsEvent { + Add(TrackNamespace), + Remove(TrackNamespace), +} + +pub struct NsSub { + pub prefix: TrackNamespace, + pub session: u64, + pub id: u64, + pub tx: mpsc::UnboundedSender, +} + +pub type UpstreamWait = watch::Receiver>>; + +#[derive(Default)] +pub struct State { + pub sessions: HashMap>, + pub namespaces: Vec, + pub ns_subs: Vec, + pub tracks: HashMap, + pub track_ids: HashMap, + pub aliases: HashMap<(u64, u64), TrackKey>, + pub pending_upstream: HashMap, +} + +/// Counters across the relay. +#[derive(Default)] +pub struct Stats { + pub sessions_opened: AtomicU64, + /// Objects received from upstream, counted on arrival: before the door, the + /// deduplication and the cache. Whether the cache holds a track is + /// [`Relay::cache_covers`]'s to say. + pub objects_in: AtomicU64, + pub fetch_served_from_cache: AtomicU64, + pub fetch_served_upstream: AtomicU64, + pub refused_at_door: AtomicU64, + pub groups_dropped_at_door: AtomicU64, +} + +/// A snapshot of [`Stats`]. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct RelayStats { + pub sessions_opened: u64, + pub sessions_open: u64, + pub objects_in: u64, + pub fetch_served_from_cache: u64, + pub fetch_served_upstream: u64, + pub refused_at_door: u64, + pub groups_dropped_at_door: u64, + /// Items pushed to sessions' schedulers and not yet dispatched. + pub queued: u64, +} + +pub struct Inner { + pub cfg: RelayConfig, + pub auth: Authorizer, + pub door: Door, + pub key: SigningKey, + pub state: Mutex, + pub budget: Mutex, + pub ids: AtomicU64, + pub summaries: Mutex>, + pub going_away: AtomicBool, + pub stats: Stats, + /// Woken when a publisher or a session appears or leaves. + pub change: Notify, + pub sessions_gone: Notify, +} + +/// A MOQT relay. Cheap to clone; every clone is the same relay. +#[derive(Clone)] +pub struct Relay { + pub(crate) inner: Arc, +} + +/// The smaller of two delivery timeouts where both are set, either where one is; 0 none +/// (§5.2). +pub fn effective_timeout(publisher_ms: u64, subscriber_ms: u64) -> u64 { + match (publisher_ms, subscriber_ms) { + (0, s) => s, + (p, 0) => p, + (p, s) => p.min(s), + } +} + +pub fn now_unix_ms() -> u64 { + std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0) +} + +fn load_key(path: &std::path::Path) -> Result { + let raw = std::fs::read(path).map_err(|e| format!("the summary key {}: {e}", path.display()))?; + let seed: [u8; 32] = if raw.len() == 32 { + raw.try_into().expect("32 bytes") + } else { + let s = String::from_utf8_lossy(&raw); + hex::decode(s.trim()).map_err(|e| format!("the summary key is neither 32 bytes nor hex: {e}"))?.try_into().map_err(|_| "the summary key is not 32 bytes".to_string())? + }; + Ok(SigningKey::from_bytes(&seed)) +} + +/// Where one upstream object goes: the fields the fan-out reads. +pub struct ObjectIn<'a> { + pub upstream: u64, + pub header: &'a SubgroupHeader, + pub subgroup_id: u64, + pub first_in_stream: Option, + pub object: &'a SubgroupObject, + pub arrived: Instant, + pub object_timeout: u64, + pub signed: Option<(u64, &'static str)>, +} + +impl Relay { + pub fn new(cfg: RelayConfig) -> Result { + let auth = Authorizer::from_config(&cfg.auth)?; + let door = Door::new(&cfg.receipts.door, &cfg.receipts.receipts_track_suffix)?; + let key = match &cfg.summary.key_path { + Some(p) => load_key(p)?, + None => SigningKey::generate(&mut rand_core_06::OsRng), + }; + let budget = Budget::new(cfg.cache.max_bytes); + Ok(Relay { + inner: Arc::new(Inner { + cfg, + auth, + door, + key, + state: Mutex::new(State::default()), + budget: Mutex::new(budget), + ids: AtomicU64::new(1), + summaries: Mutex::new(Vec::new()), + going_away: AtomicBool::new(false), + stats: Stats::default(), + change: Notify::new(), + sessions_gone: Notify::new(), + }), + }) + } + + pub fn config(&self) -> &RelayConfig { + &self.inner.cfg + } + + /// The public key the relay signs session summaries with. + pub fn summary_key(&self) -> ed25519_dalek::VerifyingKey { + self.inner.key.verifying_key() + } + + pub fn next_id(&self) -> u64 { + self.inner.ids.fetch_add(1, Ordering::Relaxed) + } + + /// The signed summaries of the sessions that have ended. + pub fn summaries(&self) -> Vec { + self.inner.summaries.lock().unwrap().clone() + } + + pub fn stats(&self) -> RelayStats { + let s = &self.inner.stats; + let st = self.inner.state.lock().unwrap(); + RelayStats { + sessions_opened: s.sessions_opened.load(Ordering::Relaxed), + sessions_open: st.sessions.len() as u64, + objects_in: s.objects_in.load(Ordering::Relaxed), + fetch_served_from_cache: s.fetch_served_from_cache.load(Ordering::Relaxed), + fetch_served_upstream: s.fetch_served_upstream.load(Ordering::Relaxed), + refused_at_door: s.refused_at_door.load(Ordering::Relaxed), + groups_dropped_at_door: s.groups_dropped_at_door.load(Ordering::Relaxed), + queued: st.sessions.values().map(|c| c.sched.queued()).sum(), + } + } + + /// Whether a FETCH starting at `start` would be served from the cache alone. + pub fn cache_covers(&self, key: &TrackKey, start: Location) -> bool { + self.inner.state.lock().unwrap().tracks.get(key).is_some_and(|t| t.cache.covers(start)) + } + + /// The Largest Object the relay knows of a track (§3.1.4). + pub fn largest(&self, key: &TrackKey) -> Option { + self.inner.state.lock().unwrap().tracks.get(key).and_then(|t| t.largest) + } + + /// Each session: its id, its peer, its transport's counters, and the items its + /// scheduler holds, for diagnosis. + pub fn session_diagnostics(&self) -> Vec<(u64, String, Option, u64)> { + let st = self.inner.state.lock().unwrap(); + st.sessions.values().map(|c| (c.id, c.remote.clone(), c.conn.diagnostics(), c.sched.queued())).collect() + } + + pub fn going_away(&self) -> bool { + self.inner.going_away.load(Ordering::Acquire) + } + + /// Wait until the alias `(session, alias)` names a track, up to `wait` (§3.1.3.1: data + /// can arrive before the control message that establishes its alias). + pub async fn resolve_alias(&self, session: u64, alias: u64, wait: Duration) -> Option { + let deadline = Instant::now() + wait; + loop { + let notified = self.inner.change.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + if let Some(k) = self.inner.state.lock().unwrap().aliases.get(&(session, alias)) { + return Some(k.clone()); + } + if tokio::time::timeout_at(deadline, notified).await.is_err() { + return None; + } + } + } + + /// The receipts track of a media track, and the media track of a receipts track. + pub fn receipts_of(&self, key: &TrackKey) -> TrackKey { + (key.0.clone(), self.inner.door.receipts_of(&key.1)) + } + + pub fn is_receipts_track(&self, name: &TrackName) -> bool { + self.inner.door.media_of(name).is_some() + } + + /// Account bytes to the cache budget, evicting the oldest groups past it. + fn charge_cache(&self, st: &mut State, track_id: u64, group: u64, n: usize) { + let evict = self.inner.budget.lock().unwrap().add(track_id, group, n); + for (t, g) in evict { + if let Some(k) = st.track_ids.get(&t).cloned() + && let Some(tr) = st.tracks.get_mut(&k) + { + cache::evict(&mut tr.cache, g); + } + } + } + + /// Create a track's state if it has none. + pub fn track_entry<'a>(&self, st: &'a mut State, key: &TrackKey) -> &'a mut Track { + if !st.tracks.contains_key(key) { + let id = self.next_id(); + st.track_ids.insert(id, key.clone()); + st.tracks.insert(key.clone(), Track::new(id, key.clone())); + } + st.tracks.get_mut(key).expect("the track") + } + + /// One object of an upstream subgroup stream: cached, then queued to every downstream + /// subscription its filter passes, each with its own deadline and priority. `started` + /// collects the subscriptions this stream has been queued to. Returns whether the + /// stream is the one the relay takes its subgroup from (§7.3): the cache counts it + /// open from its first object, and [`Relay::cache_stream_ended`] closes the count. + pub fn fan_out_object(&self, key: &TrackKey, o: ObjectIn<'_>, started: &mut HashSet) -> bool { + let mut st = self.inner.state.lock().unwrap(); + let receipts_key = self.receipts_of(key); + let is_receipts = self.is_receipts_track(&key.1); + let rsessions: HashSet = if is_receipts { + HashSet::new() + } else { + st.tracks.get(&receipts_key).map(|t| t.downs.iter().map(|d| d.session).collect()).unwrap_or_default() + }; + let session_tally = st.sessions.get(&o.upstream).map(|c| c.tally.clone()); + let Some(track) = st.tracks.get_mut(key) else { return false }; + let group = o.header.group_id; + let oid = o.object.object_id; + match track.owners.get(&(group, o.subgroup_id)) { + Some(owner) if *owner != o.upstream => { + if let Some(t) = session_tally { + t.lock().unwrap().publisher.deduplicated += 1; + } + return false; + } + Some(_) => {} + None => { + track.owners.insert((group, o.subgroup_id), o.upstream); + } + } + if o.first_in_stream == Some(oid) { + track.cache.stream_opened(group, oid == 0); + } + let loc = Location::new(group, oid); + track.note_largest(loc); + let priority = o.header.publisher_priority.unwrap_or(track.default_priority); + let cached = CachedObject { + subgroup: Some(o.subgroup_id), + publisher_priority: priority, + properties: o.object.properties.clone(), + status: o.object.status, + payload: Bytes::from(o.object.payload.clone()), + }; + let size = cache::size_of(&cached); + let ins = track.cache.insert(group, oid, cached); + if ins == Insert::Conflict { + tracing::warn!(track = %track.render, group, object = oid, "a duplicate object that differs: the track is malformed (§7.1)"); + track.cache.group_broken(group); + return true; + } + let track_id = track.id; + for sub in &track.downs { + let p = sub.params.lock().unwrap().clone(); + if !p.passes(loc) { + continue; + } + let first_for_sub = started.insert(sub.id); + let starts_stream = first_for_sub && o.first_in_stream == Some(oid); + let header = SubgroupHeader { + track_alias: sub.alias, + group_id: group, + subgroup_id: if starts_stream { o.header.subgroup_id } else { SubgroupId::Explicit(o.subgroup_id) }, + publisher_priority: o.header.publisher_priority, + properties: o.header.properties, + end_of_group: o.header.end_of_group, + first_object: o.header.first_object && starts_stream, + }; + let timeout = if sub.receipts_track { 0 } else { effective_timeout(o.object_timeout, p.object_timeout) }; + let deadline = (timeout != 0).then(|| o.arrived + Duration::from_millis(timeout)); + let prio = Prio { subscriber: p.subscriber_priority, publisher: priority, group_rank: Prio::group_rank(p.order, group), fill_rank: 1, mode_rank: 1, within: o.subgroup_id }; + let meta = ObjMeta { + track: sub.render.clone(), + payload_len: o.object.payload.len() as u64, + receipts_subscribed: rsessions.contains(&sub.session), + fetched: false, + signed: o.signed, + }; + sub.sched.push( + LaneKey { sub: sub.id, kind: LaneKind::Subgroup { group, subgroup: o.subgroup_id } }, + Item { op: Op::Object { header, object: o.object.clone() }, prio, deadline, meta: Some(meta) }, + ); + } + if ins == Insert::New { + self.charge_cache(&mut st, track_id, group, size); + } + true + } + + /// An upstream subgroup stream the cache counted ended: with a FIN (`clean`), or + /// reset, cut short or refused. + pub fn cache_stream_ended(&self, key: &TrackKey, group: u64, clean: bool) { + if let Some(t) = self.inner.state.lock().unwrap().tracks.get_mut(key) { + t.cache.stream_ended(group, clean); + } + } + + /// An upstream subgroup stream ended: with a FIN, or reset with `reset`. + pub fn end_subgroup(&self, key: &TrackKey, group: u64, subgroup: u64, subgroup_timeout_pub: u64, started: &HashSet, reset: Option) { + let st = self.inner.state.lock().unwrap(); + let Some(track) = st.tracks.get(key) else { return }; + for sub in track.downs.iter().filter(|d| started.contains(&d.id)) { + let lane = LaneKey { sub: sub.id, kind: LaneKind::Subgroup { group, subgroup } }; + let p = sub.params.lock().unwrap().clone(); + let op = match reset { + Some(code) => Op::Reset { code }, + None => { + let t = if sub.receipts_track { 0 } else { effective_timeout(subgroup_timeout_pub, p.subgroup_timeout) }; + Op::Fin { subgroup_timeout: (t != 0).then(|| Duration::from_millis(t)) } + } + }; + let prio = Prio { subscriber: p.subscriber_priority, publisher: track.default_priority, group_rank: Prio::group_rank(p.order, group), fill_rank: 1, mode_rank: 1, within: subgroup }; + sub.sched.push(lane, Item { op, prio, deadline: None, meta: None }); + } + } + + /// One object datagram from upstream (§11.2). + pub fn fan_out_datagram(&self, key: &TrackKey, upstream: u64, d: &ObjectDatagram, arrived: Instant, signed: Option<(u64, &'static str)>) { + use joule_moq::draft22::data::DatagramContent; + let mut st = self.inner.state.lock().unwrap(); + let receipts_key = self.receipts_of(key); + let rsessions: HashSet = st.tracks.get(&receipts_key).map(|t| t.downs.iter().map(|d| d.session).collect()).unwrap_or_default(); + let Some(track) = st.tracks.get_mut(key) else { return }; + let loc = Location::new(d.group_id, d.object_id); + let _ = upstream; + track.note_largest(loc); + let priority = d.publisher_priority.unwrap_or(track.default_priority); + let (status, payload) = match &d.content { + DatagramContent::Payload(p) => (joule_moq::draft22::data::ObjectStatus::Normal, p.clone()), + DatagramContent::Status(s) => (*s, Vec::new()), + }; + let cached = CachedObject { subgroup: None, publisher_priority: priority, properties: d.properties.clone().unwrap_or_default(), status, payload: Bytes::from(payload.clone()) }; + let size = cache::size_of(&cached); + let ins = track.cache.insert(d.group_id, d.object_id, cached); + if d.end_of_group && ins != Insert::Evicted { + track.cache.group_ended(d.group_id); + } + if ins != Insert::New { + // A datagram already forwarded (another publisher's, or a repeat): once only. + return; + } + let track_id = track.id; + let object_timeout = effective_timeout(track.pub_object_timeout, track.pub_subgroup_timeout); + for sub in &track.downs { + let p = sub.params.lock().unwrap().clone(); + if !p.passes(loc) { + continue; + } + // For a datagram SUBGROUP_DELIVERY_TIMEOUT acts as OBJECT_DELIVERY_TIMEOUT (§5.2). + let t = effective_timeout(object_timeout, effective_timeout(p.object_timeout, p.subgroup_timeout)); + let deadline = (t != 0 && !sub.receipts_track).then(|| arrived + Duration::from_millis(t)); + let mut out = d.clone(); + out.track_alias = sub.alias; + let prio = Prio { subscriber: p.subscriber_priority, publisher: priority, group_rank: Prio::group_rank(p.order, d.group_id), fill_rank: 1, mode_rank: 0, within: d.object_id }; + let meta = ObjMeta { track: sub.render.clone(), payload_len: payload.len() as u64, receipts_subscribed: rsessions.contains(&sub.session), fetched: false, signed }; + sub.sched.push(LaneKey { sub: sub.id, kind: LaneKind::Datagram { group: d.group_id, object: d.object_id } }, Item { op: Op::Datagram { datagram: out }, prio, deadline, meta: Some(meta) }); + } + self.charge_cache(&mut st, track_id, d.group_id, size); + } + + /// An upstream of `key` is gone. When it was the last, every downstream + /// subscription ends with PUBLISH_DONE once what it has queued is sent (§9.9). + pub fn upstream_gone(&self, session: u64, key: &TrackKey, status: PublishDoneCode) { + let ends = { + let mut st = self.inner.state.lock().unwrap(); + let mut gone_aliases = Vec::new(); + let ends = match st.tracks.get_mut(key) { + None => Vec::new(), + Some(track) => { + track.upstreams.retain(|u| { + if u.session == session { + gone_aliases.push(u.alias); + false + } else { + true + } + }); + if track.upstreams.is_empty() { + track.ended = Some(status); + std::mem::take(&mut track.downs) + } else { + Vec::new() + } + } + }; + for a in gone_aliases { + st.aliases.remove(&(session, a)); + } + ends + }; + for sub in ends { + end_down(sub, status); + } + self.inner.change.notify_waiters(); + } + + /// Remove a downstream subscription (its subscriber cancelled it, or its session + /// ended), and cancel the relay's own upstream SUBSCRIBE when nobody is left. + pub fn remove_down(&self, sub: &DownSub) { + let cancel = { + let mut st = self.inner.state.lock().unwrap(); + match st.tracks.get_mut(&sub.track) { + None => Vec::new(), + Some(track) => { + track.downs.retain(|d| d.id != sub.id); + if track.downs.is_empty() { + let mut c = Vec::new(); + track.upstreams.retain(|u| match &u.kind { + UpstreamKind::Subscribe(tx) => { + c.push((u.session, u.alias, tx.clone())); + false + } + UpstreamKind::Publish => true, + }); + for (s, a, _) in &c { + st.aliases.remove(&(*s, *a)); + } + c + } else { + Vec::new() + } + } + } + }; + for (_, _, tx) in cancel { + tokio::spawn(async move { + tx.reset(joule_moq::draft22::StreamResetCode::CANCELLED.0).await; + }); + } + } + + /// Wait for every session to end, up to `until`. + pub async fn wait_sessions_gone(&self, until: Instant) -> bool { + loop { + let notified = self.inner.sessions_gone.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + if self.inner.state.lock().unwrap().sessions.is_empty() { + return true; + } + if tokio::time::timeout_at(until, notified).await.is_err() { + return self.inner.state.lock().unwrap().sessions.is_empty(); + } + } + } + + /// GOAWAY to every session (§9.2), then wait up to `timeout` for them to leave, and + /// close the rest with GOAWAY_TIMEOUT (§6.6.1). New requests meanwhile are refused + /// with GOING_AWAY. Returns how many sessions had to be closed. + pub async fn shutdown(&self, timeout: Duration) -> usize { + self.inner.going_away.store(true, Ordering::Release); + let sessions: Vec> = self.inner.state.lock().unwrap().sessions.values().cloned().collect(); + let uri = self.inner.cfg.goaway_uri.clone().into_bytes(); + for ctx in &sessions { + crate::session::send_goaway(ctx, &uri, timeout.as_millis() as u64).await; + } + let deadline = Instant::now() + timeout; + if self.wait_sessions_gone(deadline).await { + return 0; + } + let left: Vec> = self.inner.state.lock().unwrap().sessions.values().cloned().collect(); + for ctx in &left { + ctx.close(joule_moq::draft22::SessionError::GOAWAY_TIMEOUT.0, "GOAWAY timeout"); + } + let _ = self.wait_sessions_gone(Instant::now() + Duration::from_secs(5)).await; + left.len() + } + + /// Namespaces published to the relay that `prefix` matches, other than `except`'s. + pub fn published_under(&self, st: &State, prefix: &TrackNamespace, except: u64) -> Vec { + let mut out: Vec = st.namespaces.iter().filter(|n| n.session != except && prefix_matches(prefix, &n.namespace)).map(|n| n.namespace.clone()).collect(); + out.sort(); + out.dedup(); + out + } + + /// The sessions that published a namespace `ns` falls under, other than `except`. + pub fn publishers_of(&self, st: &State, ns: &TrackNamespace, except: u64) -> Vec> { + let mut ids: Vec = st.namespaces.iter().filter(|n| n.session != except && prefix_matches(&n.namespace, ns)).map(|n| n.session).collect(); + ids.sort(); + ids.dedup(); + ids.iter().filter_map(|i| st.sessions.get(i).cloned()).collect() + } +} + +/// End a downstream subscription: after its queue drains, PUBLISH_DONE with the streams +/// it opened, then a FIN (§6.4.2.2, §9.9). +pub fn end_down(sub: Arc, status: PublishDoneCode) { + let _ = sub.ended.send(true); + tokio::spawn(async move { + let n = sub.sched.drained(sub.id).await; + let done = joule_moq::draft22::ControlMessage::PublishDone(joule_moq::draft22::PublishDone { status, stream_count: n, reason: Vec::new() }); + let _ = sub.tx.send(&done).await; + sub.tx.fin().await; + }); +} + +/// The class label and figure of a door verdict, for the tally. +pub fn signed_of(v: &crate::door::Verdict) -> Option<(u64, &'static str)> { + match v { + crate::door::Verdict::Pass { joules_micro, class } => Some((*joules_micro, class_label(class.as_deref()))), + _ => None, + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/sched.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/sched.rs new file mode 100644 index 000000000..49eca1ea7 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/sched.rs @@ -0,0 +1,766 @@ +//! The scheduler of one downstream session: what the relay sends it, in what order, and +//! what it drops (`draft-ietf-moq-transport-22` §5.1, §5.2, §7.4). +//! +//! Everything the relay sends a subscriber passes through here: objects of its +//! subscriptions on subgroup streams and datagrams, and objects of its fetches and fills on +//! fetch streams. Each data stream is a **lane**: a queue whose items go out in order, one +//! at a time, so a relay never reorders the objects of a stream (§7.4). Among the lanes +//! with something to send, the scheduler picks by §5.1.2: +//! +//! 1. the lower subscriber priority number; +//! 2. then the lower publisher priority number; +//! 3. then, within a subscription, the group its Group Order puts first (and, between a +//! fill and its subscription in one group, the fill); +//! 4. then the lower Subgroup ID, or Object ID for a datagram (a datagram before a +//! subgroup object); +//! +//! and first come, first served after that. At most `write_concurrency` writes are in +//! flight, so what waits, waits here, where it can still be dropped, and not in the +//! transport's buffers (§5.2: "minimize the amount of data buffered at the underlying +//! transport layer"). Each stream is also given a transport priority from rules 1 and 2, +//! so bytes the transport already holds are sent in the same order. +//! +//! **Delivery timeouts.** An object carries the deadline its OBJECT_DELIVERY_TIMEOUT +//! sets, counted from when it reached the relay. It is checked before the object goes to +//! the transport, and bounds the write itself; past it the subgroup's stream is reset +//! with DELIVERY_TIMEOUT and nothing more of that subgroup is sent (§5.2), and a datagram +//! is dropped. SUBGROUP_DELIVERY_TIMEOUT starts at the subgroup's FIN; if the stream's +//! data is not all acknowledged by then, the stream is reset. +//! +//! Receipts tracks carry no deadline (WAI moq-streaming-format §4.1), so the receipts of +//! an object dropped here still reach a subscriber of its receipts track. + +use crate::summary::Tally; +use crate::transport::{BoxSend, SharedConnection, TransportError}; +use joule_moq::draft22::data::{FetchCodec, FetchEntry, FetchHeader, ObjectDatagram, SubgroupCodec, SubgroupHeader, SubgroupObject}; +use joule_moq::draft22::{GroupOrder, StreamResetCode}; +use std::collections::{BTreeSet, HashMap, VecDeque}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; +use tokio::sync::{mpsc, oneshot}; +use tokio::time::Instant; + +/// An item's place in §5.1.2's order: lower first, compared field by field. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Prio { + pub subscriber: u8, + pub publisher: u8, + /// The group's rank in its request's Group Order: the Group ID ascending, its + /// complement descending. + pub group_rank: u64, + /// 0 for a fill-delivered object, 1 for a subscription-delivered one. + pub fill_rank: u8, + /// 0 for a datagram, 1 for a subgroup object. + pub mode_rank: u8, + /// The Subgroup ID, or the Object ID of a datagram. + pub within: u64, +} + +impl Prio { + pub fn group_rank(order: GroupOrder, group: u64) -> u64 { + match order { + GroupOrder::Ascending => group, + GroupOrder::Descending => u64::MAX - group, + } + } + + /// The transport's priority for a stream of this item: higher first. + fn transport_priority(&self) -> i32 { + -(((self.subscriber as i32) << 8) | self.publisher as i32) + } +} + +/// A data stream (or one datagram) of one downstream request. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum LaneKind { + Subgroup { group: u64, subgroup: u64 }, + Datagram { group: u64, object: u64 }, + Fetch { request_id: u64 }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct LaneKey { + /// The relay's id of the downstream subscription or fetch. + pub sub: u64, + pub kind: LaneKind, +} + +/// What an object item counts toward. +#[derive(Clone, Debug)] +pub struct ObjMeta { + /// The track, rendered (§8.8). + pub track: Arc, + pub payload_len: u64, + /// The session also subscribes to this track's receipts track. + pub receipts_subscribed: bool, + /// A fetch or fill object. + pub fetched: bool, + /// The publisher's signed figure for the object and its class label, where the door + /// verified its receipt. + pub signed: Option<(u64, &'static str)>, +} + +pub enum Op { + Object { header: SubgroupHeader, object: SubgroupObject }, + /// The subgroup is complete upstream: FIN the stream, then hold it to + /// SUBGROUP_DELIVERY_TIMEOUT. + Fin { subgroup_timeout: Option }, + /// The upstream stream was reset: reset this one. + Reset { code: u64 }, + Datagram { datagram: ObjectDatagram }, + FetchEntry { request_id: u64, order: GroupOrder, entry: FetchEntry }, + FetchFin { request_id: u64, order: GroupOrder }, + /// End a fetch stream abruptly, opening it first if it is not (a fill that fails, + /// §3.4.1). + FetchReset { request_id: u64, order: GroupOrder, code: u64 }, +} + +pub struct Item { + pub op: Op, + pub prio: Prio, + pub deadline: Option, + pub meta: Option, +} + +// A push carries its object by value: the scheduler is the one place it is held. +#[allow(clippy::large_enum_variant)] +enum Cmd { + Push(LaneKey, Item), + Cancel { sub: u64, code: u64 }, + Priority { sub: u64, subscriber: u8 }, + Drain { sub: u64, tx: oneshot::Sender }, + Shutdown(Option>), +} + +/// A handle on a session's scheduler. +#[derive(Clone)] +pub struct SchedHandle { + tx: mpsc::UnboundedSender, + queued: Arc, +} + +impl SchedHandle { + pub fn push(&self, lane: LaneKey, item: Item) { + self.queued.fetch_add(1, Ordering::AcqRel); + if self.tx.send(Cmd::Push(lane, item)).is_err() { + self.queued.fetch_sub(1, Ordering::AcqRel); + } + } + + /// End every stream of a request and send nothing more for it (§3.1.2: "It MUST + /// reset any open streams associated with the SUBSCRIBE"). + pub fn cancel(&self, sub: u64, code: u64) { + let _ = self.tx.send(Cmd::Cancel { sub, code }); + } + + /// A REQUEST_UPDATE changed a request's subscriber priority: applied to what is + /// queued, best effort (§5.1.1). + pub fn set_priority(&self, sub: u64, subscriber: u8) { + let _ = self.tx.send(Cmd::Priority { sub, subscriber }); + } + + /// Resolves when nothing of `sub` is queued or being written, with the number of + /// streams opened for it: what PUBLISH_DONE may then say (§9.9). + pub async fn drained(&self, sub: u64) -> u64 { + let (tx, rx) = oneshot::channel(); + if self.tx.send(Cmd::Drain { sub, tx }).is_err() { + return u64::MAX; + } + rx.await.unwrap_or(u64::MAX) + } + + pub fn shutdown(&self) { + let _ = self.tx.send(Cmd::Shutdown(None)); + } + + /// Stop, and wait until what was left queued is counted. + pub async fn shutdown_and_wait(&self) { + let (tx, rx) = oneshot::channel(); + if self.tx.send(Cmd::Shutdown(Some(tx))).is_ok() { + let _ = rx.await; + } + } + + /// Items pushed and not yet dispatched. + pub fn queued(&self) -> u64 { + self.queued.load(Ordering::Acquire) + } +} + +#[derive(Default)] +struct SubInfo { + /// Items queued or being written. + pending: u64, + streams_opened: u64, + waiters: Vec>, + cancelled: bool, +} + +#[derive(Default)] +struct Lane { + queue: VecDeque<(u64, Item)>, + send: Option, + opened: bool, + busy: bool, + dead: bool, + subgroup: Option, + fetch: Option, + index: Option<(Prio, u64)>, + /// Why the lane was ended early: its reset code. + dead_code: Option, + /// The track of its objects, for the tally. + track: Option>, +} + +enum Then { + Nothing, + Finish(Option), + Reset(u64), +} + +struct Job { + lane: LaneKey, + send: Option, + bytes: Vec, + then: Then, + deadline: Option, + priority: i32, + meta: Option, + track: Option>, +} + +enum Done { + Write { lane: LaneKey, send: Option, result: Result<(), TransportError>, meta: Option, bytes: u64, ended: bool }, + /// A subgroup was reset at its SUBGROUP_DELIVERY_TIMEOUT, after its FIN. + SubgroupTimedOut { track: Option> }, +} + +struct Written { + lane: LaneKey, + send: Option, + result: Result<(), TransportError>, + meta: Option, + bytes: u64, + ended: bool, +} + +struct Sched { + conn: SharedConnection, + lanes: HashMap, + ready: BTreeSet<(Prio, u64, LaneKey)>, + in_flight: usize, + max_in_flight: usize, + seq: u64, + subs: HashMap, + tally: Arc>, + queued: Arc, + done_tx: mpsc::UnboundedSender, +} + +/// Start a session's scheduler. It runs until [`SchedHandle::shutdown`] or until the +/// connection fails. +pub fn spawn(conn: SharedConnection, write_concurrency: usize, tally: Arc>) -> SchedHandle { + let (tx, mut rx) = mpsc::unbounded_channel(); + let (done_tx, mut done_rx) = mpsc::unbounded_channel(); + let queued = Arc::new(AtomicU64::new(0)); + let mut s = Sched { + conn, + lanes: HashMap::new(), + ready: BTreeSet::new(), + in_flight: 0, + max_in_flight: write_concurrency.max(1), + seq: 0, + subs: HashMap::new(), + tally, + queued: queued.clone(), + done_tx, + }; + tokio::spawn(async move { + let mut closed: Option> = None; + loop { + while s.in_flight < s.max_in_flight { + let Some((_, _, key)) = s.ready.pop_first() else { break }; + s.dispatch(key); + } + tokio::select! { + biased; + d = done_rx.recv() => match d { + Some(d) => s.on_done(d), + None => break, + }, + c = rx.recv() => match c { + Some(Cmd::Push(k, i)) => s.push(k, i), + Some(Cmd::Cancel { sub, code }) => s.cancel(sub, code), + Some(Cmd::Priority { sub, subscriber }) => s.reprioritise(sub, subscriber), + Some(Cmd::Drain { sub, tx }) => s.drain(sub, tx), + Some(Cmd::Shutdown(tx)) => { + closed = tx; + break; + } + None => break, + }, + } + } + s.close(); + if let Some(tx) = closed { + let _ = tx.send(()); + } + }); + SchedHandle { tx, queued } +} + +fn expired(deadline: Option) -> bool { + deadline.is_some_and(|d| Instant::now() >= d) +} + +impl Sched { + fn sub(&mut self, sub: u64) -> &mut SubInfo { + self.subs.entry(sub).or_default() + } + + fn settle(&mut self, sub: u64, n: u64) { + let info = self.sub(sub); + info.pending = info.pending.saturating_sub(n); + if info.pending == 0 { + let opened = info.streams_opened; + for w in info.waiters.drain(..) { + let _ = w.send(opened); + } + } + } + + fn push(&mut self, key: LaneKey, item: Item) { + self.queued.fetch_sub(1, Ordering::AcqRel); + if self.sub(key.sub).cancelled { + return; + } + self.sub(key.sub).pending += 1; + self.seq += 1; + let seq = self.seq; + let lane = self.lanes.entry(key).or_default(); + lane.queue.push_back((seq, item)); + self.refresh(key); + } + + /// Put the lane in the ready index exactly when it has something to send and no + /// write in flight. + fn refresh(&mut self, key: LaneKey) { + let Some(lane) = self.lanes.get_mut(&key) else { return }; + if let Some((p, s)) = lane.index.take() { + self.ready.remove(&(p, s, key)); + } + if lane.busy { + return; + } + if let Some((seq, item)) = lane.queue.front() { + lane.index = Some((item.prio, *seq)); + self.ready.insert((item.prio, *seq, key)); + } + } + + fn count_dropped(&mut self, meta: &ObjMeta) { + let mut t = self.tally.lock().unwrap(); + t.subscriber.dropped_delivery_timeout += 1; + let tr = t.track(&meta.track); + tr.dropped_delivery_timeout += 1; + if meta.receipts_subscribed { + tr.dropped_receipts_track_subscribed += 1; + t.subscriber.dropped_receipts_track_subscribed += 1; + } + } + + /// The lane's stream is reset for a delivery timeout: what it still holds is dropped + /// with it, until its subgroup ends upstream. + fn kill(&mut self, key: LaneKey, code: u64) { + let Some(lane) = self.lanes.get_mut(&key) else { return }; + lane.dead = true; + lane.dead_code.get_or_insert(code); + if let Some(s) = lane.send.as_mut() { + s.reset(code); + } + let mut dropped = Vec::new(); + let mut keep = VecDeque::new(); + for (seq, item) in lane.queue.drain(..) { + match item.op { + Op::Fin { .. } | Op::Reset { .. } | Op::FetchFin { .. } | Op::FetchReset { .. } => keep.push_back((seq, item)), + _ => dropped.push(item), + } + } + lane.queue = keep; + let n = dropped.len() as u64; + for item in dropped { + if let Some(m) = &item.meta + && code == StreamResetCode::DELIVERY_TIMEOUT.0 + { + self.count_dropped(m); + } + } + self.settle(key.sub, n); + self.refresh(key); + } + + fn remove_lane(&mut self, key: LaneKey) { + if let Some(mut lane) = self.lanes.remove(&key) { + if let Some((p, s)) = lane.index.take() { + self.ready.remove(&(p, s, key)); + } + let n = lane.queue.len() as u64; + self.settle(key.sub, n); + } + } + + fn spawn_job(&mut self, job: Job) { + self.in_flight += 1; + let conn = self.conn.clone(); + let done = self.done_tx.clone(); + tokio::spawn(async move { + let Job { lane, send, bytes, then, deadline, priority, meta, track } = job; + let mut send = send; + let n = bytes.len() as u64; + // Opening the stream (which can wait for the peer's stream credit) and the + // write are both bounded by the object's deadline. + let work = async { + if send.is_none() { + let mut s = conn.open_uni().await?; + s.set_priority(priority); + send = Some(s); + } + let s = send.as_mut().expect("a stream"); + if !bytes.is_empty() { + s.write_all(&bytes).await?; + } + Ok::<(), TransportError>(()) + }; + let result = match deadline { + Some(d) => { + // The write future (which borrows the stream) ends with this statement. + let timed = tokio::time::timeout_at(d, work).await; + match timed { + Ok(r) => r, + Err(_) => { + if let Some(s) = send.as_mut() { + s.reset(StreamResetCode::DELIVERY_TIMEOUT.0); + } + Err(TransportError::Deadline) + } + } + } + None => work.await, + }; + let mut ended = false; + if result.is_ok() { + match then { + Then::Nothing => {} + Then::Reset(code) => { + if let Some(s) = send.as_mut() { + s.reset(code); + } + ended = true; + } + Then::Finish(sgt) => { + ended = true; + if let Some(mut s) = send.take() { + let _ = s.finish(); + if let Some(t) = sgt { + // Held to SUBGROUP_DELIVERY_TIMEOUT: reset unless all of + // it is acknowledged first. + let done2 = done.clone(); + tokio::spawn(async move { + if tokio::time::timeout(t, s.delivered()).await.is_err() { + s.reset(StreamResetCode::DELIVERY_TIMEOUT.0); + let _ = done2.send(Done::SubgroupTimedOut { track }); + } + }); + } + } + } + } + } + let _ = done.send(Done::Write { lane, send, result, meta, bytes: n, ended }); + }); + } + + fn dispatch(&mut self, key: LaneKey) { + let Some(lane) = self.lanes.get_mut(&key) else { return }; + lane.index = None; + let Some((_, item)) = lane.queue.pop_front() else { return }; + let Item { op, prio, deadline, meta } = item; + if lane.dead { + // A subgroup already reset: only its end is still awaited. + let ends = matches!(op, Op::Fin { .. } | Op::Reset { .. } | Op::FetchFin { .. } | Op::FetchReset { .. }); + let timed_out = lane.dead_code == Some(StreamResetCode::DELIVERY_TIMEOUT.0); + if let (Some(m), true) = (&meta, timed_out) { + self.count_dropped(m); + } + self.settle(key.sub, 1); + if ends { + self.remove_lane(key); + } else { + self.refresh(key); + } + return; + } + let priority = prio.transport_priority(); + match op { + Op::Object { header, object } => { + if expired(deadline) { + if let Some(m) = &meta { + self.count_dropped(m); + } + self.settle(key.sub, 1); + self.kill(key, StreamResetCode::DELIVERY_TIMEOUT.0); + return; + } + let mut bytes = Vec::new(); + let lane = self.lanes.get_mut(&key).expect("the lane"); + if lane.track.is_none() { + lane.track = meta.as_ref().map(|m| m.track.clone()); + } + if !lane.opened { + header.encode(&mut bytes); + lane.subgroup = Some(SubgroupCodec::new(header)); + lane.opened = true; + self.subs.entry(key.sub).or_default().streams_opened += 1; + } + let lane = self.lanes.get_mut(&key).expect("the lane"); + let enc = lane.subgroup.as_mut().expect("a subgroup codec").encode_object(&object, &mut bytes); + if enc.is_err() { + self.settle(key.sub, 1); + self.kill(key, StreamResetCode::INTERNAL_ERROR.0); + return; + } + lane.busy = true; + let send = lane.send.take(); + self.spawn_job(Job { lane: key, send, bytes, then: Then::Nothing, deadline, priority, meta, track: None }); + } + Op::Fin { subgroup_timeout } => { + if !lane.opened { + self.settle(key.sub, 1); + self.remove_lane(key); + return; + } + lane.busy = true; + let send = lane.send.take(); + let track = lane.track.clone(); + self.spawn_job(Job { lane: key, send, bytes: vec![], then: Then::Finish(subgroup_timeout), deadline: None, priority, meta: None, track }); + } + Op::Reset { code } => { + if let Some(s) = lane.send.as_mut() { + s.reset(code); + } + self.settle(key.sub, 1); + self.remove_lane(key); + } + Op::Datagram { datagram } => { + self.settle(key.sub, 1); + if expired(deadline) { + if let Some(m) = &meta { + self.count_dropped(m); + } + } else if let Ok(b) = datagram.encode() { + let n = b.len() as u64; + if self.conn.send_datagram(b.into()).is_ok() + && let Some(m) = &meta + { + self.count_forwarded(m, n); + } + } + self.remove_lane(key); + } + Op::FetchEntry { request_id, order, entry } => { + let mut bytes = Vec::new(); + if !lane.opened { + FetchHeader { request_id }.encode(&mut bytes); + lane.fetch = Some(FetchCodec::new(order)); + lane.opened = true; + self.subs.entry(key.sub).or_default().streams_opened += 1; + } + let lane = self.lanes.get_mut(&key).expect("the lane"); + if lane.fetch.as_mut().expect("a fetch codec").encode(&entry, &mut bytes).is_err() { + self.settle(key.sub, 1); + self.kill(key, StreamResetCode::INTERNAL_ERROR.0); + return; + } + lane.busy = true; + let send = lane.send.take(); + self.spawn_job(Job { lane: key, send, bytes, then: Then::Nothing, deadline: None, priority, meta, track: None }); + } + Op::FetchFin { request_id, order } | Op::FetchReset { request_id, order, .. } => { + let then = match op { + Op::FetchReset { code, .. } => Then::Reset(code), + _ => Then::Finish(None), + }; + let mut bytes = Vec::new(); + if !lane.opened { + // A fetch with nothing in range still opens its stream and sends its + // header (§3.2.1); a failing fill resets right after it (§3.4.1). + FetchHeader { request_id }.encode(&mut bytes); + lane.fetch = Some(FetchCodec::new(order)); + lane.opened = true; + self.subs.entry(key.sub).or_default().streams_opened += 1; + } + let lane = self.lanes.get_mut(&key).expect("the lane"); + lane.busy = true; + let send = lane.send.take(); + self.spawn_job(Job { lane: key, send, bytes, then, deadline: None, priority, meta: None, track: None }); + } + } + } + + fn count_forwarded(&mut self, m: &ObjMeta, wire: u64) { + let mut t = self.tally.lock().unwrap(); + t.subscriber.wire_bytes_sent += wire; + if m.fetched { + t.subscriber.fetch_objects += 1; + t.track(&m.track).fetch_objects += 1; + } else { + t.subscriber.objects_forwarded += 1; + t.subscriber.object_bytes_forwarded += m.payload_len; + let tr = t.track(&m.track); + tr.objects_forwarded += 1; + tr.object_bytes_forwarded += m.payload_len; + } + if let Some((j, class)) = m.signed { + let f = t.publisher_signed.entry(class.to_string()).or_default(); + f.objects += 1; + f.joules_micro = f.joules_micro.saturating_add(j); + } + } + + fn on_done(&mut self, d: Done) { + let (lane, send, result, meta, bytes, ended) = match d { + Done::SubgroupTimedOut { track } => { + if let Some(name) = track { + self.tally.lock().unwrap().track(&name).subgroups_reset_after_fin += 1; + } + return; + } + Done::Write { lane, send, result, meta, bytes, ended } => (lane, send, result, meta, bytes, ended), + }; + let d = Written { lane, send, result, meta, bytes, ended }; + self.in_flight = self.in_flight.saturating_sub(1); + self.settle(d.lane.sub, 1); + match d.result { + Ok(()) => { + if let Some(m) = &d.meta { + self.count_forwarded(m, d.bytes); + } else if d.bytes > 0 { + self.tally.lock().unwrap().subscriber.wire_bytes_sent += d.bytes; + } + if d.ended { + self.remove_lane(d.lane); + return; + } + if let Some(lane) = self.lanes.get_mut(&d.lane) { + lane.busy = false; + lane.send = d.send; + } + self.refresh(d.lane); + } + Err(TransportError::Deadline) => { + if let Some(m) = &d.meta { + self.count_dropped(m); + } + if let Some(lane) = self.lanes.get_mut(&d.lane) { + lane.busy = false; + lane.send = d.send; + } + self.kill(d.lane, StreamResetCode::DELIVERY_TIMEOUT.0); + } + Err(TransportError::Stopped(_)) | Err(TransportError::Reset(_)) | Err(TransportError::Other(_)) => { + // The subscriber stopped this stream: nothing more goes on it. + if let Some(lane) = self.lanes.get_mut(&d.lane) { + lane.busy = false; + lane.send = d.send; + } + self.kill(d.lane, StreamResetCode::CANCELLED.0); + } + Err(TransportError::Closed { .. }) => { + if let Some(lane) = self.lanes.get_mut(&d.lane) { + lane.busy = false; + lane.dead = true; + } + self.refresh(d.lane); + } + } + } + + fn cancel(&mut self, sub: u64, code: u64) { + let keys: Vec = self.lanes.keys().filter(|k| k.sub == sub).copied().collect(); + for k in keys { + if let Some(lane) = self.lanes.get_mut(&k) { + lane.dead = true; + if let Some(s) = lane.send.as_mut() { + s.reset(code); + } + if !lane.busy { + self.remove_lane(k); + } else { + let n = lane.queue.len() as u64; + lane.queue.clear(); + self.settle(sub, n); + } + } + } + self.sub(sub).cancelled = true; + self.settle(sub, 0); + } + + fn reprioritise(&mut self, sub: u64, subscriber: u8) { + let keys: Vec = self.lanes.keys().filter(|k| k.sub == sub).copied().collect(); + for k in keys { + if let Some(lane) = self.lanes.get_mut(&k) { + for (_, item) in lane.queue.iter_mut() { + item.prio.subscriber = subscriber; + } + } + self.refresh(k); + } + } + + fn drain(&mut self, sub: u64, tx: oneshot::Sender) { + let info = self.sub(sub); + if info.pending == 0 { + let _ = tx.send(info.streams_opened); + } else { + info.waiters.push(tx); + } + } + + fn close(&mut self) { + let left: u64 = self.lanes.values().map(|l| l.queue.iter().filter(|(_, i)| i.meta.is_some()).count() as u64).sum(); + self.tally.lock().unwrap().subscriber.undelivered_at_close += left; + for lane in self.lanes.values_mut() { + if let Some(s) = lane.send.as_mut() { + s.reset(StreamResetCode::SESSION_CLOSED.0); + } + } + self.lanes.clear(); + for info in self.subs.values_mut() { + for w in info.waiters.drain(..) { + let _ = w.send(info.streams_opened); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_order_is_section_5_1_2s() { + let p = |s, pp, g| Prio { subscriber: s, publisher: pp, group_rank: g, fill_rank: 1, mode_rank: 1, within: 0 }; + assert!(p(10, 200, 9) < p(11, 0, 0), "subscriber priority first"); + assert!(p(10, 0, 9) < p(10, 1, 0), "then publisher priority"); + assert!(p(10, 0, 1) < p(10, 0, 2), "then the group order"); + assert!(Prio::group_rank(GroupOrder::Descending, 5) < Prio::group_rank(GroupOrder::Descending, 4)); + let fill = Prio { fill_rank: 0, ..p(1, 1, 3) }; + assert!(fill < p(1, 1, 3), "a fill before its subscription in one group"); + let dg = Prio { mode_rank: 0, within: 9, ..p(1, 1, 3) }; + assert!(dg < Prio { within: 0, ..p(1, 1, 3) }, "a datagram before a subgroup object"); + assert!(p(0, 0, 0).transport_priority() > p(0, 1, 0).transport_priority()); + assert!(p(0, 255, 0).transport_priority() > p(1, 0, 0).transport_priority()); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/server.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/server.rs new file mode 100644 index 000000000..e473c6f1d --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/server.rs @@ -0,0 +1,179 @@ +//! The relay's listeners: one UDP endpoint for native QUIC and WebTransport, and one TCP +//! listener for the WebSocket fallback. + +use crate::cert::RelayIdentity; +use crate::config::RelayConfig; +use crate::relay::Relay; +use crate::transport::mux::MuxConfig; +use crate::transport::quic::QuicConnection; +use crate::transport::webtransport::{select_protocol, WtConnection}; +use crate::transport::websocket; +use std::net::SocketAddr; +use std::sync::Arc; +use std::time::Duration; +use tokio::task::JoinHandle; +use tokio::time::timeout; + +const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10); + +/// A running relay and its listeners. +pub struct RelayServer { + relay: Relay, + pub identity: RelayIdentity, + pub quic_addr: Option, + pub websocket_addr: Option, + endpoint: Option, + quic_task: Option>, + websocket_task: Option>, +} + +fn transport_config(cfg: &RelayConfig) -> quinn::TransportConfig { + let mut t = quinn::TransportConfig::default(); + t.max_concurrent_bidi_streams(quinn::VarInt::from_u64(cfg.limits.max_requests).unwrap_or(quinn::VarInt::MAX)); + t.max_concurrent_uni_streams(quinn::VarInt::from_u64(cfg.limits.max_uni_streams).unwrap_or(quinn::VarInt::MAX)); + if let Ok(idle) = quinn::IdleTimeout::try_from(Duration::from_millis(cfg.limits.idle_timeout_ms)) { + t.max_idle_timeout(Some(idle)); + } + t.keep_alive_interval(Some(Duration::from_millis((cfg.limits.idle_timeout_ms / 3).max(1_000)))); + t.send_window(cfg.limits.quic_send_window_bytes.max(16 * 1024)); + t +} + +impl RelayServer { + /// Bind the configured listeners and start serving. + pub async fn start(cfg: RelayConfig) -> Result { + let identity = match (&cfg.tls.cert_path, &cfg.tls.key_path) { + (Some(c), Some(k)) => RelayIdentity::from_pem_files(c, k).await.map_err(|e| e.to_string())?, + (None, None) => RelayIdentity::dev_self_signed(&cfg.tls.dev_names).map_err(|e| e.to_string())?, + _ => return Err("tls: cert_path and key_path go together".into()), + }; + let relay = Relay::new(cfg.clone())?; + let mut endpoint = None; + let mut quic_task = None; + let mut websocket_task = None; + let mut quic_addr = None; + if let Some(addr) = cfg.quic_listen { + let tls = identity.server_tls().map_err(|e| e.to_string())?; + let crypto = quinn::crypto::rustls::QuicServerConfig::try_from(tls).map_err(|e| e.to_string())?; + let mut sc = quinn::ServerConfig::with_crypto(Arc::new(crypto)); + sc.transport_config(Arc::new(transport_config(&cfg))); + let ep = quinn::Endpoint::server(sc, addr).map_err(|e| format!("binding {addr}: {e}"))?; + quic_addr = Some(ep.local_addr().map_err(|e| e.to_string())?); + quic_task = Some(tokio::spawn(quic_accept(relay.clone(), ep.clone(), cfg.path.clone()))); + endpoint = Some(ep); + } + let mut websocket_addr = None; + if let Some(addr) = cfg.websocket_listen { + let l = tokio::net::TcpListener::bind(addr).await.map_err(|e| format!("binding {addr}: {e}"))?; + websocket_addr = Some(l.local_addr().map_err(|e| e.to_string())?); + let mux = MuxConfig { max_bidi: cfg.limits.max_requests, max_uni: cfg.limits.max_uni_streams, max_buffered: cfg.limits.websocket_buffer_bytes, ..MuxConfig::default() }; + websocket_task = Some(tokio::spawn(websocket_accept(relay.clone(), l, cfg.path.clone(), mux))); + } + Ok(RelayServer { relay, identity, quic_addr, websocket_addr, endpoint, quic_task, websocket_task }) + } + + pub fn relay(&self) -> &Relay { + &self.relay + } + + /// Stop accepting, GOAWAY every session, wait up to the configured GOAWAY timeout, + /// close what is left with GOAWAY_TIMEOUT. Returns how many sessions that closed. + pub async fn shutdown(self) -> usize { + if let Some(ep) = &self.endpoint { + ep.set_server_config(None); + } + // The WebSocket listener stops taking upgrades; the sessions it made go on. + if let Some(t) = &self.websocket_task { + t.abort(); + } + let t = Duration::from_millis(self.relay.config().limits.goaway_timeout_ms); + let closed = self.relay.shutdown(t).await; + if let Some(ep) = &self.endpoint { + ep.close(quinn::VarInt::from_u32(0), b"the relay stopped"); + let _ = timeout(Duration::from_secs(5), ep.wait_idle()).await; + } + if let Some(t) = &self.quic_task { + t.abort(); + } + closed + } +} + +async fn quic_accept(relay: Relay, ep: quinn::Endpoint, path: String) { + while let Some(incoming) = ep.accept().await { + let (relay, path) = (relay.clone(), path.clone()); + tokio::spawn(async move { + let Ok(mut connecting) = incoming.accept() else { return }; + let alpn = match timeout(HANDSHAKE_TIMEOUT, connecting.handshake_data()).await { + Ok(Ok(hd)) => hd.downcast::().ok().and_then(|h| h.protocol.clone()), + _ => return, + }; + match alpn.as_deref() { + Some(a) if a == joule_moq::draft22::ALPN => { + if let Ok(Ok(c)) = timeout(HANDSHAKE_TIMEOUT, connecting).await { + crate::session::serve(relay, Arc::new(QuicConnection::new(c))).await; + } + } + Some(b"h3") => { + let req = match timeout(HANDSHAKE_TIMEOUT, wtransport::endpoint::IncomingSessionFuture::with_quic_connecting(connecting)).await { + Ok(Ok(r)) => r, + _ => return, + }; + if req.path().split('?').next() != Some(path.as_str()) { + req.not_found().await; + return; + } + // -22 §6.2.1: the draft is chosen by WT-Available-Protocols. A client that + // sends the header must offer moqt-22; one that sends none is taken to + // speak it. + let offered = req.headers().get("wt-available-protocols").map(String::as_str); + let chosen = match (offered, select_protocol(offered)) { + (None, _) => None, + (Some(_), Some(p)) => Some(p), + (Some(_), None) => { + req.forbidden().await; + return; + } + }; + let headers: Vec<(String, String)> = chosen.map(|p| vec![("wt-protocol".to_string(), format!("\"{p}\""))]).unwrap_or_default(); + if let Ok(Ok(conn)) = timeout(HANDSHAKE_TIMEOUT, req.accept_with_headers(headers)).await { + crate::session::serve(relay, Arc::new(WtConnection::new(conn))).await; + } + } + _ => {} + } + }); + } +} + +async fn websocket_accept(relay: Relay, l: tokio::net::TcpListener, path: String, mux: MuxConfig) { + use tokio_tungstenite::tungstenite::handshake::server::{ErrorResponse, Request, Response}; + use tokio_tungstenite::tungstenite::http; + loop { + let Ok((tcp, peer)) = l.accept().await else { + tokio::time::sleep(Duration::from_millis(50)).await; + continue; + }; + let _ = tcp.set_nodelay(true); + let (relay, path) = (relay.clone(), path.clone()); + tokio::spawn(async move { + // The handshake library's callback type; its error is the refusal response. + #[allow(clippy::result_large_err)] + let check = move |req: &Request, mut resp: Response| -> Result { + let refuse = |status: u16, why: &str| http::Response::builder().status(status).body(Some(why.to_string())).expect("a response"); + if req.uri().path() != path { + return Err(refuse(404, "no MOQT endpoint at this path")); + } + let offered = req.headers().get("sec-websocket-protocol").and_then(|v| v.to_str().ok()); + if !websocket::offers_subprotocol(offered) { + return Err(refuse(400, "offer the moqt-22 subprotocol")); + } + resp.headers_mut().insert("sec-websocket-protocol", http::HeaderValue::from_static("moqt-22")); + Ok(resp) + }; + if let Ok(Ok(ws)) = timeout(HANDSHAKE_TIMEOUT, tokio_tungstenite::accept_hdr_async(tcp, check)).await { + crate::session::serve(relay, Arc::new(websocket::session(ws, false, peer.to_string(), mux))).await; + } + }); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/session.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/session.rs new file mode 100644 index 000000000..8294d6f16 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/session.rs @@ -0,0 +1,1421 @@ +//! One session at the relay, from SETUP to its summary, and every request it makes +//! (`draft-ietf-moq-transport-22` §6, §7, §9). + +use crate::auth::{Resolved, Right, TokenCache}; +use crate::cache::CachedObject; +use crate::door::Verdict; +use crate::relay::{ + now_unix_ms, signed_of, DownSub, NsEvent, NsSub, ObjectIn, PublishedNs, Relay, SessionCtx, SubParams, TrackKey, Upstream, + UpstreamKind, DEFAULT_PRIORITY, +}; +use crate::sched::{self, Item, LaneKey, LaneKind, ObjMeta, Op, Prio, SchedHandle}; +use crate::summary::{RelayEnergy, SessionSummary, SignedSessionSummary, Tally, FORMAT}; +use crate::transport::{BoxRecv, BoxSend, SharedConnection, TransportError, TransportKind}; +use crate::wire::{open_request, responder, MessageReader, ReqRx, ReqTx, StreamError, UniReader}; +use joule_moq::carriage::{read_object_properties, SETUP_OPTION_JWP_RECEIPTS}; +use joule_moq::draft22::data::{Datagram, FetchEntry, FetchObject, ObjectStatus, RangeEnd}; +use joule_moq::draft22::kvp::{opt, prop, KeyValuePair, SetupOptions}; +use joule_moq::draft22::names::{is_reserved, prefix_matches, prefixes_overlap, SESSION_NAMESPACE}; +use joule_moq::draft22::params::param; +use joule_moq::draft22::{ + Consequence, ControlMessage, FetchOk, Goaway, GroupOrder, Location, LocationFilter, NamespaceRequest, Parameter, Parameters, PeerRequestIds, + Perspective, Publish, PublishDoneCode, RequestError, RequestErrorCode, RequestIds, RequestKind, RequestOk, SessionError, Setup, StreamHeader, + StreamResetCode, SubscribeOk, Token, TrackRequest, UniEvent, +}; +use joule_moq::messages::ObjectHeader; +use joule_moq::session::MoqSession; +use joule_moq::{GroupId, ObjectId, TrackAlias, TrackNamespace, JWP_MOQ_EXTENSION_VERSION}; +use std::collections::HashSet; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; +use tokio::sync::{mpsc, oneshot, watch}; +use tokio::time::{timeout, Instant}; + +/// How long a peer has to open its control stream, and a request stream to state its +/// request (§6.3). +const SETUP_TIMEOUT: Duration = Duration::from_secs(10); +/// Data streams that arrive before the control stream, held until SETUP (§6.3). +const EARLY_STREAMS: usize = 64; + +/// The implementation the relay names in SETUP (§9.1.5). +pub fn implementation() -> String { + format!("joule-moq-relay/{}", env!("CARGO_PKG_VERSION")) +} + +fn params_of(m: &ControlMessage) -> Option<&Parameters> { + match m { + ControlMessage::Subscribe(r) | ControlMessage::Fetch(r) | ControlMessage::TrackStatus(r) => Some(&r.params), + ControlMessage::Publish(p) => Some(&p.params), + ControlMessage::PublishNamespace(r) | ControlMessage::SubscribeNamespace(r) | ControlMessage::SubscribeTracks(r) => Some(&r.params), + _ => None, + } +} + +fn tokens_in(p: &Parameters) -> Vec { + p.0.iter() + .filter_map(|x| match x { + Parameter::AuthorizationToken(t) => Some(t.clone()), + _ => None, + }) + .collect() +} + +fn varint_param(p: &Parameters, ty: u64) -> Option { + p.0.iter().find_map(|x| match (x.type_code() == ty, x) { + (true, Parameter::RendezvousTimeout(v)) | (true, Parameter::SubgroupDeliveryTimeout(v)) | (true, Parameter::FillTimeout(v)) => Some(*v), + _ => None, + }) +} + +fn is_session_namespace(ns: &TrackNamespace) -> bool { + ns.0.first().is_some_and(|f| f.as_slice() == SESSION_NAMESPACE) +} + +/// What a wire error means for the session. +fn fail(ctx: &SessionCtx, e: &StreamError) { + if let StreamError::Wire(w) = e { + match w.consequence() { + Consequence::CloseSession(code) => ctx.close(code.0, &w.to_string()), + _ => ctx.close(SessionError::PROTOCOL_VIOLATION.0, &w.to_string()), + } + } +} + +async fn reject(tx: &ReqTx, code: RequestErrorCode, reason: &str) { + let e = ControlMessage::RequestError(RequestError { code, retry_interval: 0, reason: reason.as_bytes().to_vec(), redirect: None }); + let _ = tx.send(&e).await; + tx.fin().await; +} + +/// A Subscription's start and end from its Location Filter, given the Largest Object +/// (§3.3.1, §9.20.9). +pub fn subscription_range(f: Option, largest: Option) -> (Location, Option) { + match f.unwrap_or(LocationFilter::None) { + LocationFilter::None => (Location::new(0, 0), None), + LocationFilter::NextObject => (largest.map_or(Location::new(0, 0), |l| Location::new(l.group, l.object.saturating_add(1))), None), + LocationFilter::RelativeStart { start_group } => (Location::new(LocationFilter::relative_start(start_group, largest.map(|l| l.group)), 0), None), + LocationFilter::AbsoluteStart { start } => (start, None), + LocationFilter::AbsoluteStartGroupEnd { start, end_group_delta } => (start, Some(Location::new(start.group.saturating_add(end_group_delta), u64::MAX))), + LocationFilter::AbsoluteRange { start, end_group_delta, end_object } => (start, Some(Location::new(start.group.saturating_add(end_group_delta), end_object))), + } +} + +/// A fetch's range from its Location Filter (§3.2, §9.20.9): never past the Largest +/// Object. `None` when it starts after it (INVALID_RANGE). +pub fn fetch_range(f: Option, largest: Location) -> Option<(Location, Location)> { + let (start, end) = match f.unwrap_or(LocationFilter::None) { + LocationFilter::None => (Location::new(0, 0), largest), + LocationFilter::NextObject => return None, + LocationFilter::RelativeStart { start_group } => (Location::new(LocationFilter::relative_start(start_group, Some(largest.group)), 0), largest), + LocationFilter::AbsoluteStart { start } => (start, largest), + LocationFilter::AbsoluteStartGroupEnd { start, end_group_delta } => (start, Location::new(start.group.saturating_add(end_group_delta), u64::MAX).min(largest)), + LocationFilter::AbsoluteRange { start, end_group_delta, end_object } => (start, Location::new(start.group.saturating_add(end_group_delta), end_object).min(largest)), + }; + (start <= largest && start <= end).then_some((start, end)) +} + +// ── The session ──────────────────────────────────────────────────────────── + +async fn read_setup(conn: &SharedConnection, max_payload: u64, early: &mut Vec) -> Result<(Setup, UniReader), String> { + loop { + let recv = conn.accept_uni().await.map_err(|e| e.to_string())?; + let mut r = UniReader::new(recv, max_payload); + match r.start().await.map_err(|e| e.to_string())? { + StreamHeader::Control => { + return match r.next().await.map_err(|e| e.to_string())? { + Some(UniEvent::Control(ControlMessage::Setup(s))) => Ok((s, r)), + _ => Err("the control stream does not begin with SETUP".into()), + }; + } + _ => { + if early.len() >= EARLY_STREAMS { + return Err("too many data streams before SETUP".into()); + } + early.push(r); + } + } + } +} + +/// Run one session to its end. +pub async fn serve(relay: Relay, conn: SharedConnection) { + let cfg = relay.config().clone(); + let id = relay.next_id(); + relay.inner.stats.sessions_opened.fetch_add(1, Ordering::Relaxed); + let opened = now_unix_ms(); + if relay.inner.state.lock().unwrap().sessions.len() >= cfg.limits.max_sessions { + conn.close(SessionError::INTERNAL_ERROR.0, b"the relay is at its session limit"); + return; + } + let mut early = Vec::new(); + let (setup, ctl_rx) = match timeout(SETUP_TIMEOUT, read_setup(&conn, cfg.limits.max_object_bytes, &mut early)).await { + Ok(Ok(x)) => x, + Ok(Err(why)) => { + conn.close(SessionError::PROTOCOL_VIOLATION.0, why.as_bytes()); + return; + } + Err(_) => { + conn.close(SessionError::CONTROL_MESSAGE_TIMEOUT.0, b"no SETUP"); + return; + } + }; + let over_wt = conn.kind() == TransportKind::WebTransport; + if let Err(code) = setup.check_from(Perspective::Client, over_wt) { + conn.close(code.0, b"the client's SETUP breaks a rule for its end"); + return; + } + let mut tokens = TokenCache::new(cfg.limits.max_auth_token_cache_size); + let setup_tokens = match tokens.setup(&setup.options.authorization_tokens()) { + Ok(t) => t, + Err(code) => { + conn.close(code.0, b"an authorization token in SETUP"); + return; + } + }; + let jwp = MoqSession::declares_jwp(&setup); + let mut o = SetupOptions::new(); + o.push(KeyValuePair::bytes(opt::MOQT_IMPLEMENTATION, implementation().into_bytes())); + if cfg.limits.max_request_updates != 0 { + o.push(KeyValuePair::varint(opt::MAX_REQUEST_UPDATES, cfg.limits.max_request_updates)); + } + if cfg.limits.max_auth_token_cache_size != 0 { + o.push(KeyValuePair::varint(opt::MAX_AUTH_TOKEN_CACHE_SIZE, cfg.limits.max_auth_token_cache_size)); + } + if jwp { + o.push(KeyValuePair::varint(SETUP_OPTION_JWP_RECEIPTS, JWP_MOQ_EXTENSION_VERSION as u64)); + } + let ours = match ControlMessage::Setup(Setup { options: o }).encode() { + Ok(b) => b, + Err(_) => return, + }; + let control = match conn.open_uni().await { + Ok(mut s) => { + s.set_priority(i32::MAX); + if s.write_all(&ours).await.is_err() { + return; + } + s + } + Err(_) => return, + }; + let tally = Arc::new(Mutex::new(Tally::default())); + let sched = sched::spawn(conn.clone(), cfg.limits.write_concurrency, tally.clone()); + let ctx = Arc::new(SessionCtx { + id, + conn: conn.clone(), + kind: conn.kind(), + remote: conn.remote(), + sched, + tally, + peer_ids: Mutex::new(PeerRequestIds::new(Perspective::Client)), + my_ids: Mutex::new(RequestIds::new(Perspective::Server)), + tokens: Mutex::new(tokens), + setup_tokens, + peer_max_updates: setup.options.max_request_updates(), + my_max_updates: cfg.limits.max_request_updates, + control: tokio::sync::Mutex::new(Some(control)), + goaway_sent: AtomicBool::new(false), + pending_fetch: Mutex::new(Default::default()), + opened_unix_ms: opened, + close_code: Mutex::new(None), + jwp, + }); + relay.inner.state.lock().unwrap().sessions.insert(id, ctx.clone()); + relay.inner.change.notify_waiters(); + if relay.going_away() { + send_goaway(&ctx, cfg.goaway_uri.as_bytes(), cfg.limits.goaway_timeout_ms).await; + } + + let tasks = vec![ + tokio::spawn(control_reader(ctx.clone(), ctl_rx)), + tokio::spawn(bi_loop(relay.clone(), ctx.clone())), + tokio::spawn(uni_loop(relay.clone(), ctx.clone(), early)), + tokio::spawn(datagram_loop(relay.clone(), ctx.clone())), + ]; + let why = conn.closed().await; + for t in tasks { + t.abort(); + } + cleanup(&relay, &ctx, why).await; +} + +/// The rest of the peer's control stream: at most a GOAWAY. Its end is a protocol +/// violation (§6.3). +async fn control_reader(ctx: Arc, mut r: UniReader) { + loop { + match r.next().await { + Ok(Some(UniEvent::Control(ControlMessage::Goaway(g)))) => { + if g.check_from(Perspective::Client).is_err() { + ctx.close(SessionError::PROTOCOL_VIOLATION.0, "a client's GOAWAY names a URI"); + return; + } + tracing::debug!(session = ctx.id, "the client sent GOAWAY"); + } + Ok(Some(_)) => {} + Ok(None) => { + ctx.close(SessionError::PROTOCOL_VIOLATION.0, "the control stream closed"); + return; + } + Err(StreamError::Wire(e)) => { + fail(&ctx, &StreamError::Wire(e)); + return; + } + Err(StreamError::Transport(_)) => return, + } + } +} + +/// Send GOAWAY on a session's control stream, once (§9.2). +pub async fn send_goaway(ctx: &SessionCtx, uri: &[u8], timeout_ms: u64) { + if ctx.goaway_sent.swap(true, Ordering::AcqRel) { + return; + } + let m = ControlMessage::Goaway(Goaway { new_session_uri: uri.to_vec(), timeout_ms }); + if let Ok(b) = m.encode() + && let Some(s) = ctx.control.lock().await.as_mut() + { + let _ = s.write_all(&b).await; + } +} + +async fn bi_loop(relay: Relay, ctx: Arc) { + while let Ok((send, recv)) = ctx.conn.accept_bi().await { + tokio::spawn(request(relay.clone(), ctx.clone(), send, recv)); + } +} + +async fn uni_loop(relay: Relay, ctx: Arc, early: Vec) { + for r in early { + tokio::spawn(data_stream(relay.clone(), ctx.clone(), r)); + } + let max = relay.config().limits.max_object_bytes; + while let Ok(recv) = ctx.conn.accept_uni().await { + tokio::spawn(data_stream(relay.clone(), ctx.clone(), UniReader::new(recv, max))); + } +} + +async fn datagram_loop(relay: Relay, ctx: Arc) { + while let Ok(b) = ctx.conn.recv_datagram().await { + let arrived = Instant::now(); + let d = match Datagram::decode(&b) { + Ok(Datagram::Object(d)) => d, + Ok(Datagram::Padding(_)) => continue, + Err(e) => { + fail(&ctx, &StreamError::Wire(e)); + return; + } + }; + let key = match relay.inner.state.lock().unwrap().aliases.get(&(ctx.id, d.track_alias)) { + Some(k) => k.clone(), + // §3.1.3.1: a datagram whose alias is not yet known MAY be dropped. + None => continue, + }; + { + let mut t = ctx.tally.lock().unwrap(); + t.publisher.objects_received += 1; + } + relay.inner.stats.objects_in.fetch_add(1, Ordering::Relaxed); + if relay.inner.door.checks(&key.0) && relay.inner.door.media_of(&key.1).is_none() { + let relay = relay.clone(); + let ctx = ctx.clone(); + tokio::spawn(async move { + let payload = match &d.content { + joule_moq::draft22::data::DatagramContent::Payload(p) => p.clone(), + _ => Vec::new(), + }; + let v = relay.inner.door.object(&key.0, &key.1, d.group_id, d.object_id, &payload, None).await; + if let Verdict::Refuse(why) = &v { + refused(&relay, &ctx, &key, d.group_id, d.object_id, why); + return; + } + relay.fan_out_datagram(&key, ctx.id, &d, arrived, signed_of(&v)); + }); + } else { + relay.fan_out_datagram(&key, ctx.id, &d, arrived, None); + } + } +} + +fn refused(relay: &Relay, ctx: &SessionCtx, key: &TrackKey, group: u64, object: u64, why: &str) { + relay.inner.stats.refused_at_door.fetch_add(1, Ordering::Relaxed); + ctx.tally.lock().unwrap().publisher.refused_at_door += 1; + tracing::warn!( + target: "joule_moq_relay::door", + session = ctx.id, + track = %joule_moq::draft22::names::render(&key.0, Some(&key.1)), + group, + object, + reason = why, + "refused at the door" + ); +} + +/// One unidirectional stream from the peer. +async fn data_stream(relay: Relay, ctx: Arc, mut r: UniReader) { + let header = match timeout(SETUP_TIMEOUT, r.start()).await { + Ok(Ok(h)) => h, + Ok(Err(e)) => { + fail(&ctx, &e); + return; + } + Err(_) => { + r.stop(StreamResetCode::CANCELLED.0); + return; + } + }; + match header { + StreamHeader::Subgroup(h) => upstream_subgroup(relay, ctx, h, r).await, + StreamHeader::Fetch(fh) => { + let waiting = ctx.pending_fetch.lock().unwrap().remove(&fh.request_id); + match waiting { + Some(tx) => { + let _ = tx.send(r); + } + None => r.stop(StreamResetCode::CANCELLED.0), + } + } + StreamHeader::Padding => { + // Read and discarded (§11.5.1), each read bounded. + while let Ok(Ok(Some(_))) = timeout(SETUP_TIMEOUT, r.next()).await {} + } + StreamHeader::Control => ctx.close(SessionError::PROTOCOL_VIOLATION.0, "a second control stream"), + } +} + +/// Closes the cache's count of an upstream subgroup stream however the stream's reading +/// ends: cleanly only at its FIN. +struct StreamCount<'a> { + relay: &'a Relay, + key: &'a TrackKey, + group: u64, + counted: bool, + clean: bool, +} + +impl Drop for StreamCount<'_> { + fn drop(&mut self) { + if self.counted { + self.relay.cache_stream_ended(self.key, self.group, self.clean); + } + } +} + +async fn upstream_subgroup(relay: Relay, ctx: Arc, h: joule_moq::draft22::SubgroupHeader, mut r: UniReader) { + let wait = Duration::from_millis(relay.config().limits.unknown_alias_wait_ms); + let Some(key) = relay.resolve_alias(ctx.id, h.track_alias, wait).await else { + r.stop(StreamResetCode::CANCELLED.0); + return; + }; + let door_on = relay.inner.door.checks(&key.0); + let media_of = relay.inner.door.media_of(&key.1); + let (mut object_timeout, mut subgroup_timeout) = { + let st = relay.inner.state.lock().unwrap(); + st.tracks.get(&key).map_or((0, 0), |t| (t.pub_object_timeout, t.pub_subgroup_timeout)) + }; + let mut started = HashSet::new(); + let mut first: Option = None; + let mut subgroup_id: Option = None; + let mut count = StreamCount { relay: &relay, key: &key, group: h.group_id, counted: false, clean: false }; + loop { + match r.next().await { + Ok(Some(UniEvent::SubgroupObject(o))) => { + let arrived = Instant::now(); + if first.is_none() { + first = Some(o.object_id); + subgroup_id = Some(h.resolve_subgroup_id(o.object_id)); + // Either timeout set on the subgroup's first object overrides the + // track's for this subgroup (§5.2). + let v = |ty| o.properties.find(ty).ok().flatten().and_then(|k| k.as_varint()); + if let Some(t) = v(prop::OBJECT_DELIVERY_TIMEOUT) { + object_timeout = t; + } + if let Some(t) = v(prop::SUBGROUP_DELIVERY_TIMEOUT) { + subgroup_timeout = t; + } + } + let sid = subgroup_id.unwrap_or(0); + { + let mut t = ctx.tally.lock().unwrap(); + t.publisher.objects_received += 1; + t.publisher.object_bytes_received += o.payload.len() as u64; + } + relay.inner.stats.objects_in.fetch_add(1, Ordering::Relaxed); + let mut signed = None; + if door_on && o.status == ObjectStatus::Normal { + if let Some(media) = &media_of { + if let Err(why) = relay.inner.door.bundle(&key.0, media, h.group_id, &o.payload) { + relay.inner.stats.groups_dropped_at_door.fetch_add(1, Ordering::Relaxed); + refused(&relay, &ctx, &key, h.group_id, o.object_id, &why); + relay.end_subgroup(&key, h.group_id, sid, 0, &started, Some(StreamResetCode::MALFORMED_TRACK.0)); + r.stop(StreamResetCode::CANCELLED.0); + return; + } + } else { + let mut hh = ObjectHeader { + track_alias: TrackAlias(h.track_alias), + group_id: GroupId(h.group_id), + object_id: ObjectId(o.object_id), + publisher_priority: 0, + status: joule_moq::ObjectStatus::Normal, + payload_len: 0, + mrl_layer: 0, + mrl_total_layers: 0, + wai_envelope: false, + }; + let property = read_object_properties(&o.properties, &mut hh).ok().and_then(|x| x.receipt); + let v = relay.inner.door.object(&key.0, &key.1, h.group_id, o.object_id, &o.payload, property.as_ref()).await; + if let Verdict::Refuse(why) = &v { + // The rest of a multi-object stream is not forwarded around a + // hole: its downstream streams are reset (§7.4). + refused(&relay, &ctx, &key, h.group_id, o.object_id, why); + relay.end_subgroup(&key, h.group_id, sid, 0, &started, Some(StreamResetCode::MALFORMED_TRACK.0)); + r.stop(StreamResetCode::CANCELLED.0); + return; + } + signed = signed_of(&v); + } + } + let is_first = first == Some(o.object_id); + let taken = relay.fan_out_object( + &key, + ObjectIn { upstream: ctx.id, header: &h, subgroup_id: sid, first_in_stream: first, object: &o, arrived, object_timeout, signed }, + &mut started, + ); + if is_first { + count.counted = taken; + } + } + Ok(Some(_)) => { + ctx.close(SessionError::PROTOCOL_VIOLATION.0, "a subgroup stream carried something other than an object"); + return; + } + Ok(None) => { + if let Some(sid) = subgroup_id { + relay.end_subgroup(&key, h.group_id, sid, subgroup_timeout, &started, None); + } + count.clean = true; + return; + } + Err(StreamError::Transport(TransportError::Reset(code))) => { + if let Some(sid) = subgroup_id { + relay.end_subgroup(&key, h.group_id, sid, 0, &started, Some(code)); + } + return; + } + Err(StreamError::Wire(e)) => { + if let Some(sid) = subgroup_id { + relay.end_subgroup(&key, h.group_id, sid, 0, &started, Some(StreamResetCode::MALFORMED_TRACK.0)); + } + if let Consequence::CloseSession(code) = e.consequence() { + ctx.close(code.0, &e.to_string()); + } else { + r.stop(StreamResetCode::MALFORMED_TRACK.0); + } + return; + } + Err(StreamError::Transport(_)) => { + if let Some(sid) = subgroup_id { + relay.end_subgroup(&key, h.group_id, sid, 0, &started, Some(StreamResetCode::SESSION_CLOSED.0)); + } + return; + } + } + } +} + +// ── Requests ─────────────────────────────────────────────────────────────── + +async fn request(relay: Relay, ctx: Arc, mut send: BoxSend, recv: BoxRecv) { + send.set_priority(i32::MAX - 1); + let mut reader = MessageReader::new(recv); + let first = match timeout(SETUP_TIMEOUT, reader.next()).await { + Ok(Ok(Some(m))) => m, + Ok(Err(e)) => { + fail(&ctx, &e); + return; + } + _ => return, + }; + if first.opens_request().is_none() { + ctx.close(SessionError::PROTOCOL_VIOLATION.0, "a request stream that does not begin with a request"); + return; + } + if let Some(rid) = first.request_id() + && ctx.peer_ids.lock().unwrap().accept(rid).is_err() + { + ctx.close(SessionError::INVALID_REQUEST_ID.0, "a Request ID of the wrong parity or seen before"); + return; + } + let (tx, rx) = match responder(send, reader, &first, ctx.my_max_updates, ctx.peer_max_updates) { + Ok(p) => p, + Err(e) => { + fail(&ctx, &StreamError::Wire(e)); + return; + } + }; + let mut tokens = ctx.setup_tokens.clone(); + if let Some(p) = params_of(&first) { + match ctx.tokens.lock().unwrap().resolve(&tokens_in(p)) { + Ok(t) => tokens.extend(t), + Err(code) => { + ctx.close(code.0, "an authorization token"); + return; + } + } + } + if relay.going_away() { + reject(&tx, RequestErrorCode::GOING_AWAY, "the relay is going away").await; + return; + } + match first { + ControlMessage::Subscribe(r) => subscribe(relay, ctx, r, tx, rx, tokens).await, + ControlMessage::Publish(p) => publish(relay, ctx, p, tx, rx, tokens).await, + ControlMessage::Fetch(r) => fetch(relay, ctx, r, tx, rx, tokens).await, + ControlMessage::TrackStatus(r) => track_status(relay, ctx, r, tx, tokens).await, + ControlMessage::PublishNamespace(r) => publish_namespace(relay, ctx, r, tx, rx, tokens).await, + ControlMessage::SubscribeNamespace(r) => subscribe_namespace(relay, ctx, r, tx, rx, tokens).await, + ControlMessage::SubscribeTracks(_) => reject(&tx, RequestErrorCode::NOT_SUPPORTED, "SUBSCRIBE_TRACKS is not implemented by this relay").await, + _ => {} + } +} + +/// Make sure `key` has an Established upstream subscription (§7.4: "The relay MUST have +/// an Established upstream subscription before sending SUBSCRIBE_OK"), subscribing to +/// every publisher of a namespace it falls under, or waiting up to `rendezvous_ms` for +/// one to appear (§9.20.6). +async fn ensure_upstream(relay: &Relay, key: &TrackKey, requester: u64, rendezvous_ms: u64) -> Result<(), (RequestErrorCode, String)> { + let deadline = Instant::now() + Duration::from_millis(rendezvous_ms); + let upstream_timeout = Duration::from_millis(relay.config().limits.upstream_timeout_ms); + loop { + let notified = relay.inner.change.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + enum Next { + Ready, + Wait(crate::relay::UpstreamWait), + Subscribe(Vec>, watch::Sender>>), + Nobody, + } + let next = { + let mut st = relay.inner.state.lock().unwrap(); + if st.tracks.get(key).is_some_and(|t| !t.upstreams.is_empty()) { + Next::Ready + } else if let Some(w) = st.pending_upstream.get(key) { + Next::Wait(w.clone()) + } else { + let pubs = relay.publishers_of(&st, &key.0, requester); + if pubs.is_empty() { + Next::Nobody + } else { + let (wtx, wrx) = watch::channel(None); + st.pending_upstream.insert(key.clone(), wrx); + Next::Subscribe(pubs, wtx) + } + } + }; + match next { + Next::Ready => return Ok(()), + Next::Wait(mut w) => { + let r = timeout(upstream_timeout, async { + loop { + if let Some(r) = w.borrow_and_update().clone() { + return r; + } + if w.changed().await.is_err() { + return Err((RequestErrorCode::INTERNAL_ERROR.0, "the upstream request ended".into())); + } + } + }) + .await; + match r { + Ok(Ok(())) => continue, + Ok(Err((c, why))) => return Err((RequestErrorCode(c), why)), + Err(_) => return Err((RequestErrorCode::TIMEOUT, "the upstream did not answer".into())), + } + } + Next::Subscribe(pubs, wtx) => { + let r = subscribe_upstream_any(relay, key, pubs).await; + relay.inner.state.lock().unwrap().pending_upstream.remove(key); + let _ = wtx.send(Some(r.clone())); + return r.map_err(|(c, w)| (RequestErrorCode(c), w)); + } + Next::Nobody => { + if rendezvous_ms == 0 { + return Err((RequestErrorCode::DOES_NOT_EXIST, "no publisher of the track".into())); + } + if tokio::time::timeout_at(deadline, notified).await.is_err() { + return Err((RequestErrorCode::TIMEOUT, "no publisher within RENDEZVOUS_TIMEOUT".into())); + } + } + } + } +} + +/// SUBSCRIBE upstream to every publisher; done when the first is Established (the rest +/// join the track as they answer). +async fn subscribe_upstream_any(relay: &Relay, key: &TrackKey, pubs: Vec>) -> Result<(), (u64, String)> { + let (rtx, mut rrx) = mpsc::unbounded_channel(); + let n = pubs.len(); + for p in pubs { + let (relay, key, rtx) = (relay.clone(), key.clone(), rtx.clone()); + tokio::spawn(async move { + let _ = rtx.send(upstream_subscribe(&relay, p, &key).await); + }); + } + let mut first_err = None; + for _ in 0..n { + match rrx.recv().await { + Some(Ok(())) => return Ok(()), + Some(Err(e)) => { + first_err.get_or_insert(e); + } + None => break, + } + } + Err(first_err.unwrap_or((RequestErrorCode::INTERNAL_ERROR.0, "no upstream answered".into()))) +} + +/// The relay's own SUBSCRIBE to a publisher. It asks for every object (no filter), at the +/// default priority and with no delivery timeout: §5.1.3 has a relay not copy a +/// subscriber's priority or Group Order upstream, and §9.20.3, §9.20.4 its timeouts. +pub async fn upstream_subscribe(relay: &Relay, pubc: Arc, key: &TrackKey) -> Result<(), (u64, String)> { + let Some(id) = pubc.next_request_id() else { return Err((RequestErrorCode::INTERNAL_ERROR.0, "Request IDs spent".into())) }; + let msg = ControlMessage::Subscribe(TrackRequest { + request_id: id, + namespace: key.0.clone(), + name: key.1.clone(), + params: Parameters::new().with(Parameter::SubscriberPriority(DEFAULT_PRIORITY)).with(Parameter::Forward(true)), + }); + let (utx, mut urx) = open_request(&*pubc.conn, &msg, pubc.my_max_updates, pubc.peer_max_updates).await.map_err(|e| (RequestErrorCode::INTERNAL_ERROR.0, e.to_string()))?; + let t = Duration::from_millis(relay.config().limits.upstream_timeout_ms); + match timeout(t, urx.next()).await { + Ok(Ok(Some(ControlMessage::SubscribeOk(ok)))) => { + { + let mut st = relay.inner.state.lock().unwrap(); + if let Some(other) = st.aliases.get(&(pubc.id, ok.track_alias)) + && other != key + { + drop(st); + pubc.close(SessionError::DUPLICATE_TRACK_ALIAS.0, "a Track Alias already in use"); + return Err((RequestErrorCode::INTERNAL_ERROR.0, "a duplicate Track Alias".into())); + } + let track = relay.track_entry(&mut st, key); + if track.upstreams.is_empty() { + track.set_props(ok.track_properties.clone()); + } + if let Some(l) = ok.params.largest_object() { + track.note_largest(l); + } + track.ended = None; + track.upstreams.push(Upstream { session: pubc.id, alias: ok.track_alias, kind: UpstreamKind::Subscribe(utx.clone()) }); + st.aliases.insert((pubc.id, ok.track_alias), key.clone()); + } + relay.inner.change.notify_waiters(); + let (relay, key, sid) = (relay.clone(), key.clone(), pubc.id); + tokio::spawn(async move { + let status = loop { + match urx.next().await { + Ok(Some(ControlMessage::PublishDone(d))) => break d.status, + Ok(Some(_)) => continue, + Ok(None) | Err(_) => break PublishDoneCode::GOING_AWAY, + } + }; + relay.upstream_gone(sid, &key, status); + }); + Ok(()) + } + Ok(Ok(Some(ControlMessage::RequestError(e)))) => Err((e.code.0, String::from_utf8_lossy(&e.reason).into_owned())), + Ok(Err(e)) => { + fail(&pubc, &e); + Err((RequestErrorCode::INTERNAL_ERROR.0, e.to_string())) + } + _ => { + utx.reset(StreamResetCode::CANCELLED.0).await; + Err((RequestErrorCode::TIMEOUT.0, "the publisher did not answer SUBSCRIBE".into())) + } + } +} + +async fn subscribe(relay: Relay, ctx: Arc, r: TrackRequest, tx: ReqTx, mut rx: ReqRx, tokens: Vec) { + let key: TrackKey = (r.namespace.clone(), r.name.clone()); + if is_session_namespace(&key.0) { + return reject(&tx, RequestErrorCode::DOES_NOT_EXIST, "no session-level track is defined here").await; + } + if !relay.inner.auth.allows(&key.0, Right::Subscribe, &tokens) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "no grant covers this subscription").await; + } + if r.params.range_filters().next().is_some() { + return reject(&tx, RequestErrorCode::INVALID_FILTER, "Range Filters need MAX_FILTER_RANGES, which this relay declares 0").await; + } + let rendezvous = varint_param(&r.params, param::RENDEZVOUS_TIMEOUT).unwrap_or(0).min(relay.config().limits.max_rendezvous_ms); + if let Err((code, why)) = ensure_upstream(&relay, &key, ctx.id, rendezvous).await { + return reject(&tx, code, &why).await; + } + let is_receipts = relay.is_receipts_track(&key.1); + let made = { + let mut st = relay.inner.state.lock().unwrap(); + match st.tracks.get_mut(&key) { + Some(track) if !track.upstreams.is_empty() => { + let (start, end) = subscription_range(r.params.location_filter(), track.largest); + let p = SubParams { + subscriber_priority: r.params.subscriber_priority().unwrap_or(DEFAULT_PRIORITY), + order: r.params.group_order().unwrap_or(track.default_order), + forward: r.params.forward().unwrap_or(true), + start, + end, + object_timeout: r.params.object_delivery_timeout().unwrap_or(0), + subgroup_timeout: varint_param(&r.params, param::SUBGROUP_DELIVERY_TIMEOUT).unwrap_or(0), + }; + let id = relay.next_id(); + let sub = Arc::new(DownSub { + id, + session: ctx.id, + request_id: r.request_id, + alias: id, + track: key.clone(), + render: track.render.clone(), + sched: ctx.sched.clone(), + params: Mutex::new(p), + receipts_track: is_receipts, + tx: tx.clone(), + ended: watch::channel(false).0, + }); + track.downs.push(sub.clone()); + let mut params = Parameters::new(); + if let Some(l) = track.largest { + params.push(Parameter::LargestObject(l)); + } + Some((sub, SubscribeOk { track_alias: id, params, track_properties: track.props.clone() })) + } + _ => None, + } + }; + let Some((sub, ok)) = made else { + return reject(&tx, RequestErrorCode::DOES_NOT_EXIST, "the publisher left").await; + }; + let mut ended = sub.ended.subscribe(); + if tx.send(&ControlMessage::SubscribeOk(ok)).await.is_err() { + relay.remove_down(&sub); + ctx.sched.cancel(sub.id, StreamResetCode::CANCELLED.0); + return; + } + if let Some(Parameter::FillParameters(fill)) = r.params.get(param::FILL_PARAMETERS) { + let p = sub.params.lock().unwrap().clone(); + if p.forward { + let fill = fill.clone(); + let filter = fill.location_filter().or(r.params.location_filter()); + let order = fill.group_order().unwrap_or(p.order); + let prio = fill.subscriber_priority().unwrap_or(p.subscriber_priority); + let budget = varint_param(&fill, param::FILL_TIMEOUT); + let (relay, ctx, key, request_id, lane_sub) = (relay.clone(), ctx.clone(), key.clone(), r.request_id, sub.id); + tokio::spawn(async move { + serve_fill(&relay, &ctx, &key, filter, order, prio, request_id, lane_sub, budget).await; + }); + } + } + let mut reading = true; + loop { + tokio::select! { + _ = ended.changed() => return, + m = rx.next(), if reading => match m { + Ok(Some(ControlMessage::RequestUpdate(u))) => { + if u.check_for(RequestKind::Subscribe).is_err() { + let e = ControlMessage::RequestError(RequestError { code: RequestErrorCode::INVALID_FILTER, retry_interval: 0, reason: b"a parameter a subscription update does not take".to_vec(), redirect: None }); + let _ = tx.send(&e).await; + continue; + } + let largest = relay.inner.state.lock().unwrap().tracks.get(&key).and_then(|t| t.largest); + { + let mut p = sub.params.lock().unwrap(); + if let Some(v) = u.params.subscriber_priority() { + p.subscriber_priority = v; + ctx.sched.set_priority(sub.id, v); + } + if let Some(f) = u.params.forward() { + p.forward = f; + } + if let Some(f) = u.params.location_filter() { + let (s, e) = subscription_range(Some(f), largest); + p.start = s; + p.end = e; + } + if let Some(t) = u.params.object_delivery_timeout() { + p.object_timeout = t; + } + if let Some(t) = varint_param(&u.params, param::SUBGROUP_DELIVERY_TIMEOUT) { + p.subgroup_timeout = t; + } + } + let mut params = Parameters::new(); + if let Some(l) = largest { + params.push(Parameter::LargestObject(l)); + } + let _ = tx.send(&ControlMessage::RequestOk(RequestOk { params, track_properties: Default::default() })).await; + } + Ok(Some(_)) => {} + // A FIN says no more updates; the subscription goes on (§6.4.2.2). + Ok(None) => reading = false, + Err(StreamError::Wire(e)) => { + fail(&ctx, &StreamError::Wire(e)); + return; + } + // The subscriber cancelled (§6.4.2.3). + Err(StreamError::Transport(_)) => { + relay.remove_down(&sub); + ctx.sched.cancel(sub.id, StreamResetCode::CANCELLED.0); + tx.reset(StreamResetCode::CANCELLED.0).await; + return; + } + } + } + } +} + +async fn publish(relay: Relay, ctx: Arc, p: Publish, tx: ReqTx, mut rx: ReqRx, tokens: Vec) { + let key: TrackKey = (p.namespace.clone(), p.name.clone()); + if is_reserved(&key.0) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "a reserved namespace").await; + } + if !relay.inner.auth.allows(&key.0, Right::Publish, &tokens) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "no grant covers this track").await; + } + { + let mut st = relay.inner.state.lock().unwrap(); + if st.aliases.get(&(ctx.id, p.track_alias)).is_some_and(|k| *k != key) { + drop(st); + ctx.close(SessionError::DUPLICATE_TRACK_ALIAS.0, "a Track Alias already in use"); + return; + } + let track = relay.track_entry(&mut st, &key); + if track.upstreams.is_empty() { + track.set_props(p.track_properties.clone()); + } + if let Some(l) = p.params.largest_object() { + track.note_largest(l); + } + track.ended = None; + track.upstreams.push(Upstream { session: ctx.id, alias: p.track_alias, kind: UpstreamKind::Publish }); + st.aliases.insert((ctx.id, p.track_alias), key.clone()); + } + relay.inner.change.notify_waiters(); + if tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await.is_err() { + relay.upstream_gone(ctx.id, &key, PublishDoneCode::INTERNAL_ERROR); + return; + } + let status = loop { + match rx.next().await { + Ok(Some(ControlMessage::PublishDone(d))) => break d.status, + Ok(Some(ControlMessage::RequestUpdate(_))) => { + let _ = tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await; + } + Ok(Some(_)) => {} + // A FIN before PUBLISH_DONE: the request failed (§6.4.2.2). + Ok(None) => break PublishDoneCode::INTERNAL_ERROR, + Err(StreamError::Wire(e)) => { + fail(&ctx, &StreamError::Wire(e)); + break PublishDoneCode::INTERNAL_ERROR; + } + Err(StreamError::Transport(_)) => break PublishDoneCode::GOING_AWAY, + } + }; + relay.upstream_gone(ctx.id, &key, status); + tx.fin().await; +} + +/// Push the objects from the cache in `[start, end]` onto a fetch or fill lane, then its +/// end. Returns how many objects. +#[allow(clippy::too_many_arguments)] +fn queue_from_cache(relay: &Relay, sched: &SchedHandle, key: &TrackKey, start: Location, end: Location, order: GroupOrder, prio: u8, request_id: u64, lane_sub: u64, fill: bool) -> usize { + let (objects, render) = { + let st = relay.inner.state.lock().unwrap(); + match st.tracks.get(key) { + Some(t) => (t.cache.range(start, end, order == GroupOrder::Descending), t.render.clone()), + None => (Vec::new(), Arc::from("")), + } + }; + let lane = LaneKey { sub: lane_sub, kind: LaneKind::Fetch { request_id } }; + let mut n = 0; + for ((g, o), c) in objects { + if c.status != ObjectStatus::Normal { + // A fetch carries no status (§11.4.1); a gap says an object does not exist. + continue; + } + let CachedObject { subgroup, publisher_priority, properties, payload, .. } = c; + let entry = FetchEntry::Object(FetchObject { + group_id: g, + object_id: o, + subgroup_id: subgroup, + publisher_priority, + properties: (!properties.is_empty()).then_some(properties), + payload: payload.to_vec(), + }); + let p = Prio { subscriber: prio, publisher: publisher_priority, group_rank: Prio::group_rank(order, g), fill_rank: if fill { 0 } else { 1 }, mode_rank: 1, within: o }; + let meta = ObjMeta { track: render.clone(), payload_len: payload.len() as u64, receipts_subscribed: false, fetched: true, signed: None }; + sched.push(lane, Item { op: Op::FetchEntry { request_id, order, entry }, prio: p, deadline: None, meta: Some(meta) }); + n += 1; + } + let p = Prio { subscriber: prio, publisher: u8::MAX, group_rank: u64::MAX, fill_rank: 1, mode_rank: 1, within: u64::MAX }; + sched.push(lane, Item { op: Op::FetchFin { request_id, order }, prio: p, deadline: None, meta: None }); + n +} + +/// The session of an upstream that can answer a FETCH for `key`: one of the track's +/// upstreams, else a publisher of its namespace (§7.6: the same matching as SUBSCRIBE). +fn fetch_upstream_of(relay: &Relay, key: &TrackKey, except: u64) -> Option> { + let st = relay.inner.state.lock().unwrap(); + if let Some(t) = st.tracks.get(key) + && let Some(u) = t.upstreams.iter().find(|u| u.session != except) + && let Some(c) = st.sessions.get(&u.session) + { + return Some(c.clone()); + } + relay.publishers_of(&st, &key.0, except).into_iter().next() +} + +/// A FETCH upstream whose objects go onto a downstream lane as they arrive (§3.2.3). +/// Returns the publisher's FETCH_OK, or its refusal. +#[allow(clippy::too_many_arguments)] +async fn upstream_fetch( + relay: &Relay, + pubc: Arc, + sched: SchedHandle, + key: &TrackKey, + start: Location, + end: Option, + order: GroupOrder, + prio: u8, + request_id: u64, + lane_sub: u64, + fill: bool, + budget: Duration, +) -> Result { + let Some(id) = pubc.next_request_id() else { return Err((RequestErrorCode::INTERNAL_ERROR, "Request IDs spent".into())) }; + let (otx, orx) = oneshot::channel(); + pubc.pending_fetch.lock().unwrap().insert(id, otx); + let filter = match end { + Some(e) => LocationFilter::AbsoluteRange { start, end_group_delta: e.group - start.group, end_object: e.object }, + None => LocationFilter::AbsoluteStart { start }, + }; + let msg = ControlMessage::Fetch(TrackRequest { + request_id: id, + namespace: key.0.clone(), + name: key.1.clone(), + params: Parameters::new().with(Parameter::LocationFilter(filter)).with(Parameter::GroupOrder(order)).with(Parameter::SubscriberPriority(DEFAULT_PRIORITY)), + }); + let fail_up = |why: String| { + pubc.pending_fetch.lock().unwrap().remove(&id); + Err((RequestErrorCode::INTERNAL_ERROR, why)) + }; + let (utx, mut urx) = match open_request(&*pubc.conn, &msg, pubc.my_max_updates, pubc.peer_max_updates).await { + Ok(p) => p, + Err(e) => return fail_up(e.to_string()), + }; + let t = Duration::from_millis(relay.config().limits.upstream_timeout_ms); + let ok = match timeout(t, urx.next()).await { + Ok(Ok(Some(ControlMessage::FetchOk(ok)))) => ok, + Ok(Ok(Some(ControlMessage::RequestError(e)))) => { + pubc.pending_fetch.lock().unwrap().remove(&id); + return Err((e.code, String::from_utf8_lossy(&e.reason).into_owned())); + } + _ => { + utx.reset(StreamResetCode::CANCELLED.0).await; + return fail_up("the publisher did not answer FETCH".into()); + } + }; + relay.inner.stats.fetch_served_upstream.fetch_add(1, Ordering::Relaxed); + let render: Arc = joule_moq::draft22::names::render(&key.0, Some(&key.1)).into(); + let end_location = ok.end_location; + tokio::spawn(async move { + let lane = LaneKey { sub: lane_sub, kind: LaneKind::Fetch { request_id } }; + let fin = |op| Item { op, prio: Prio { subscriber: prio, publisher: u8::MAX, group_rank: u64::MAX, fill_rank: 1, mode_rank: 1, within: u64::MAX }, deadline: None, meta: None }; + let reader = timeout(budget, orx).await; + let Ok(Ok(mut reader)) = reader else { + // §3.2.3: past the Fill Timeout what is not there is a Timed-Out gap. + sched.push(lane, fin(Op::FetchEntry { request_id, order, entry: FetchEntry::EndOfRange { kind: RangeEnd::TimedOut, location: end_location } })); + sched.push(lane, fin(Op::FetchFin { request_id, order })); + return; + }; + reader.set_group_order(order); + loop { + match timeout(budget, reader.next()).await { + Ok(Ok(Some(UniEvent::FetchEntry(entry)))) => { + let (pp, g, o, len) = match &entry { + FetchEntry::Object(f) => (f.publisher_priority, f.group_id, f.object_id, f.payload.len() as u64), + FetchEntry::EndOfRange { location, .. } => (u8::MAX, location.group, location.object, 0), + }; + let meta = matches!(entry, FetchEntry::Object(_)).then(|| ObjMeta { track: render.clone(), payload_len: len, receipts_subscribed: false, fetched: true, signed: None }); + let p = Prio { subscriber: prio, publisher: pp, group_rank: Prio::group_rank(order, g), fill_rank: if fill { 0 } else { 1 }, mode_rank: 1, within: o }; + sched.push(lane, Item { op: Op::FetchEntry { request_id, order, entry }, prio: p, deadline: None, meta }); + } + Ok(Ok(None)) => { + sched.push(lane, fin(Op::FetchFin { request_id, order })); + return; + } + Ok(Err(StreamError::Transport(TransportError::Reset(code)))) => { + sched.push(lane, fin(Op::FetchReset { request_id, order, code })); + return; + } + Ok(Ok(Some(_))) | Ok(Err(_)) => { + sched.push(lane, fin(Op::FetchReset { request_id, order, code: StreamResetCode::INTERNAL_ERROR.0 })); + return; + } + Err(_) => { + sched.push(lane, fin(Op::FetchEntry { request_id, order, entry: FetchEntry::EndOfRange { kind: RangeEnd::TimedOut, location: end_location } })); + sched.push(lane, fin(Op::FetchFin { request_id, order })); + return; + } + } + } + }); + Ok(ok) +} + +async fn fetch(relay: Relay, ctx: Arc, r: TrackRequest, tx: ReqTx, mut rx: ReqRx, tokens: Vec) { + let key: TrackKey = (r.namespace.clone(), r.name.clone()); + if is_session_namespace(&key.0) { + return reject(&tx, RequestErrorCode::DOES_NOT_EXIST, "no session-level track is defined here").await; + } + if !relay.inner.auth.allows(&key.0, Right::Subscribe, &tokens) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "no grant covers this fetch").await; + } + if r.params.range_filters().next().is_some() { + return reject(&tx, RequestErrorCode::INVALID_FILTER, "Range Filters need MAX_FILTER_RANGES, which this relay declares 0").await; + } + let order = r.params.group_order().unwrap_or(GroupOrder::Ascending); + let prio = r.params.subscriber_priority().unwrap_or(DEFAULT_PRIORITY); + let budget_ms = varint_param(&r.params, param::FILL_TIMEOUT).unwrap_or(relay.config().limits.fill_timeout_ms).min(relay.config().limits.fill_timeout_ms); + let lane_sub = relay.next_id(); + enum Plan { + Cache { start: Location, end: Location, eot: bool, props: joule_moq::draft22::kvp::Properties }, + Upstream { start: Location, end: Option }, + Refuse(RequestErrorCode, &'static str), + } + let plan = { + let st = relay.inner.state.lock().unwrap(); + match st.tracks.get(&key) { + None => Plan::Upstream { start: Location::new(0, 0), end: None }, + Some(t) => match t.largest { + None => Plan::Refuse(RequestErrorCode::INVALID_RANGE, "no object has been published on the track"), + Some(largest) => match fetch_range(r.params.location_filter(), largest) { + None => Plan::Refuse(RequestErrorCode::INVALID_RANGE, "the range starts after the Largest Object"), + Some((start, end)) if t.cache.covers(start) => Plan::Cache { start, end, eot: t.cache.end_of_track == Some(end), props: t.props.clone() }, + Some((start, end)) => Plan::Upstream { start, end: Some(end) }, + }, + }, + } + }; + match plan { + Plan::Refuse(code, why) => reject(&tx, code, why).await, + Plan::Cache { start, end, eot, props } => { + relay.inner.stats.fetch_served_from_cache.fetch_add(1, Ordering::Relaxed); + let ok = FetchOk { end_of_track: eot, end_location: end, params: Parameters::new(), track_properties: props }; + if tx.send(&ControlMessage::FetchOk(ok)).await.is_err() { + return; + } + queue_from_cache(&relay, &ctx.sched, &key, start, end, order, prio, r.request_id, lane_sub, false); + finish_fetch(&ctx, lane_sub, &tx, &mut rx).await; + } + Plan::Upstream { start, end } => { + let Some(pubc) = fetch_upstream_of(&relay, &key, ctx.id) else { + return reject(&tx, RequestErrorCode::DOES_NOT_EXIST, "the relay holds no such objects and has no publisher to ask").await; + }; + // With no Location Filter the request's own range goes upstream unchanged. + let (start, end) = match (r.params.location_filter(), end) { + (Some(_), _) | (None, Some(_)) => (start, end), + (None, None) => (Location::new(0, 0), None), + }; + match upstream_fetch(&relay, pubc, ctx.sched.clone(), &key, start, end, order, prio, r.request_id, lane_sub, false, Duration::from_millis(budget_ms)).await { + Ok(ok) => { + let ok = FetchOk { end_of_track: ok.end_of_track, end_location: ok.end_location, params: Parameters::new(), track_properties: ok.track_properties }; + if tx.send(&ControlMessage::FetchOk(ok)).await.is_err() { + ctx.sched.cancel(lane_sub, StreamResetCode::CANCELLED.0); + return; + } + finish_fetch(&ctx, lane_sub, &tx, &mut rx).await; + } + Err((code, why)) => reject(&tx, code, &why).await, + } + } + } +} + +/// After FETCH_OK: answer priority updates until the fetch stream is sent, then FIN; a +/// cancelled request cancels the stream (§3.2.4). +async fn finish_fetch(ctx: &SessionCtx, lane_sub: u64, tx: &ReqTx, rx: &mut ReqRx) { + let sched = ctx.sched.clone(); + let drained = sched.drained(lane_sub); + tokio::pin!(drained); + let mut reading = true; + loop { + tokio::select! { + _ = &mut drained => break, + m = rx.next(), if reading => match m { + Ok(Some(ControlMessage::RequestUpdate(u))) => { + if let Some(p) = u.params.subscriber_priority() { + ctx.sched.set_priority(lane_sub, p); + } + let _ = tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await; + } + Ok(Some(_)) => {} + Ok(None) => reading = false, + Err(_) => { + ctx.sched.cancel(lane_sub, StreamResetCode::CANCELLED.0); + tx.reset(StreamResetCode::CANCELLED.0).await; + return; + } + } + } + } + tx.fin().await; +} + +/// A subscription's fill (§3.4): the fill range from the cache, or upstream, on a fill +/// fetch stream whose header names the SUBSCRIBE's Request ID. A fill the relay cannot +/// serve is a fetch stream reset right after its header (§3.4.1). +#[allow(clippy::too_many_arguments)] +async fn serve_fill(relay: &Relay, ctx: &Arc, key: &TrackKey, filter: Option, order: GroupOrder, prio: u8, request_id: u64, lane_sub: u64, budget: Option) { + let range = { + let st = relay.inner.state.lock().unwrap(); + st.tracks.get(key).and_then(|t| t.largest.and_then(|l| fetch_range(filter, l)).map(|(s, e)| (s, e, t.cache.covers(s)))) + }; + let Some((start, end, covered)) = range else { return }; + if covered { + relay.inner.stats.fetch_served_from_cache.fetch_add(1, Ordering::Relaxed); + queue_from_cache(relay, &ctx.sched, key, start, end, order, prio, request_id, lane_sub, true); + return; + } + let budget = Duration::from_millis(budget.unwrap_or(relay.config().limits.fill_timeout_ms).min(relay.config().limits.fill_timeout_ms)); + let fail_lane = |sched: &SchedHandle| { + let lane = LaneKey { sub: lane_sub, kind: LaneKind::Fetch { request_id } }; + let p = Prio { subscriber: prio, publisher: u8::MAX, group_rank: 0, fill_rank: 0, mode_rank: 1, within: 0 }; + sched.push(lane, Item { op: Op::FetchReset { request_id, order, code: StreamResetCode::UNKNOWN_OBJECT_STATUS.0 }, prio: p, deadline: None, meta: None }); + }; + match fetch_upstream_of(relay, key, ctx.id) { + Some(pubc) => { + if upstream_fetch(relay, pubc, ctx.sched.clone(), key, start, Some(end), order, prio, request_id, lane_sub, true, budget).await.is_err() { + fail_lane(&ctx.sched); + } + } + None => fail_lane(&ctx.sched), + } +} + +async fn track_status(relay: Relay, _ctx: Arc, r: TrackRequest, tx: ReqTx, tokens: Vec) { + let key: TrackKey = (r.namespace.clone(), r.name.clone()); + if !relay.inner.auth.allows(&key.0, Right::Subscribe, &tokens) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "no grant covers this track").await; + } + let found = relay.inner.state.lock().unwrap().tracks.get(&key).map(|t| (t.largest, t.props.clone())); + match found { + Some((largest, props)) => { + let mut params = Parameters::new(); + if let Some(l) = largest { + params.push(Parameter::LargestObject(l)); + } + let _ = tx.send(&ControlMessage::RequestOk(RequestOk { params, track_properties: props })).await; + tx.fin().await; + } + None => reject(&tx, RequestErrorCode::DOES_NOT_EXIST, "the relay knows no such track").await, + } +} + +/// Withdraw a published namespace, telling namespace subscribers when nobody publishes +/// it any longer. +fn withdraw(relay: &Relay, session: u64, id: u64) { + let mut st = relay.inner.state.lock().unwrap(); + let Some(pos) = st.namespaces.iter().position(|n| n.id == id) else { return }; + let gone = st.namespaces.remove(pos); + let still = st.namespaces.iter().any(|n| n.namespace == gone.namespace); + if !still { + for s in st.ns_subs.iter().filter(|s| s.session != session && prefix_matches(&s.prefix, &gone.namespace)) { + let _ = s.tx.send(NsEvent::Remove(gone.namespace.clone())); + } + } +} + +async fn publish_namespace(relay: Relay, ctx: Arc, r: NamespaceRequest, tx: ReqTx, mut rx: ReqRx, tokens: Vec) { + if is_reserved(&r.prefix) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "a reserved namespace").await; + } + if !relay.inner.auth.allows(&r.prefix, Right::Publish, &tokens) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "no grant covers this namespace").await; + } + let id = relay.next_id(); + let to_subscribe = { + let mut st = relay.inner.state.lock().unwrap(); + let known = st.namespaces.iter().any(|n| n.namespace == r.prefix); + st.namespaces.push(PublishedNs { namespace: r.prefix.clone(), session: ctx.id, id }); + if !known { + for s in st.ns_subs.iter().filter(|s| s.session != ctx.id && prefix_matches(&s.prefix, &r.prefix)) { + let _ = s.tx.send(NsEvent::Add(r.prefix.clone())); + } + } + // §7.6: a new publisher of a namespace gets a SUBSCRIBE for every subscription + // the relay holds under it. + st.tracks + .values() + .filter(|t| !t.downs.is_empty() && prefix_matches(&r.prefix, &t.key.0) && !t.upstreams.iter().any(|u| u.session == ctx.id)) + .map(|t| t.key.clone()) + .collect::>() + }; + if tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await.is_err() { + withdraw(&relay, ctx.id, id); + return; + } + relay.inner.change.notify_waiters(); + for key in to_subscribe { + let (relay, ctx) = (relay.clone(), ctx.clone()); + tokio::spawn(async move { + let _ = upstream_subscribe(&relay, ctx, &key).await; + }); + } + loop { + match rx.next().await { + Ok(Some(ControlMessage::RequestUpdate(_))) => { + let _ = tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await; + } + Ok(Some(_)) => {} + // A FIN is not a withdrawal; the namespace stands until the request is + // cancelled or the session ends (§4.1). + Ok(None) => { + let _ = ctx.conn.closed().await; + return; + } + Err(_) => { + withdraw(&relay, ctx.id, id); + return; + } + } + } +} + +fn suffix(ns: &TrackNamespace, prefix: &TrackNamespace) -> TrackNamespace { + TrackNamespace(ns.0[prefix.0.len().min(ns.0.len())..].to_vec()) +} + +async fn subscribe_namespace(relay: Relay, ctx: Arc, r: NamespaceRequest, tx: ReqTx, mut rx: ReqRx, tokens: Vec) { + if !relay.inner.auth.allows(&r.prefix, Right::Subscribe, &tokens) { + return reject(&tx, RequestErrorCode::UNAUTHORIZED, "no grant covers this namespace").await; + } + let (etx, mut erx) = mpsc::unbounded_channel(); + let id = relay.next_id(); + let initial = { + let mut st = relay.inner.state.lock().unwrap(); + if st.ns_subs.iter().any(|s| s.session == ctx.id && prefixes_overlap(&s.prefix, &r.prefix)) { + None + } else { + st.ns_subs.push(NsSub { prefix: r.prefix.clone(), session: ctx.id, id, tx: etx }); + Some(relay.published_under(&st, &r.prefix, ctx.id)) + } + }; + let Some(initial) = initial else { + return reject(&tx, RequestErrorCode::PREFIX_OVERLAP, "it overlaps another namespace subscription of this session").await; + }; + let remove = |relay: &Relay| relay.inner.state.lock().unwrap().ns_subs.retain(|s| s.id != id); + if tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await.is_err() { + remove(&relay); + return; + } + for ns in initial { + if tx.send(&ControlMessage::Namespace(suffix(&ns, &r.prefix))).await.is_err() { + remove(&relay); + return; + } + } + let mut reading = true; + loop { + tokio::select! { + ev = erx.recv() => { + let m = match ev { + Some(NsEvent::Add(ns)) => ControlMessage::Namespace(suffix(&ns, &r.prefix)), + Some(NsEvent::Remove(ns)) => ControlMessage::NamespaceDone(suffix(&ns, &r.prefix)), + None => break, + }; + if tx.send(&m).await.is_err() { + break; + } + } + m = rx.next(), if reading => match m { + Ok(Some(ControlMessage::RequestUpdate(_))) => { + let _ = tx.send(&ControlMessage::RequestOk(RequestOk { params: Parameters::new(), track_properties: Default::default() })).await; + } + Ok(Some(_)) => {} + Ok(None) => reading = false, + Err(_) => break, + } + } + } + remove(&relay); +} + +// ── The end of a session ─────────────────────────────────────────────────── + +async fn cleanup(relay: &Relay, ctx: &Arc, why: TransportError) { + let (downs, ups, nss) = { + let mut st = relay.inner.state.lock().unwrap(); + st.sessions.remove(&ctx.id); + let mut downs = Vec::new(); + let mut ups = Vec::new(); + for t in st.tracks.values_mut() { + t.downs.retain(|d| { + if d.session == ctx.id { + downs.push(d.clone()); + false + } else { + true + } + }); + if t.upstreams.iter().any(|u| u.session == ctx.id) { + ups.push(t.key.clone()); + } + } + let nss: Vec = st.namespaces.iter().filter(|n| n.session == ctx.id).map(|n| n.id).collect(); + st.ns_subs.retain(|s| s.session != ctx.id); + (downs, ups, nss) + }; + ctx.pending_fetch.lock().unwrap().clear(); + for d in &downs { + let _ = d.ended.send(true); + relay.remove_down(d); + } + // The publisher's session ended: its tracks end with GOING_AWAY (§12.4). + for key in ups { + relay.upstream_gone(ctx.id, &key, PublishDoneCode::GOING_AWAY); + } + for id in nss { + withdraw(relay, ctx.id, id); + } + ctx.sched.shutdown_and_wait().await; + let close_code = match &why { + TransportError::Closed { code, by_peer: true, .. } if *code != u64::MAX => Some(*code), + _ => *ctx.close_code.lock().unwrap(), + }; + let tally = ctx.tally.lock().unwrap().clone(); + let summary = SessionSummary { + format: FORMAT.into(), + relay_key: hex::encode(relay.summary_key().to_bytes()), + session: ctx.id, + transport: ctx.kind.label().into(), + remote: ctx.remote.clone(), + draft: String::from_utf8_lossy(joule_moq::draft22::ALPN).into_owned(), + opened_unix_ms: ctx.opened_unix_ms, + closed_unix_ms: now_unix_ms(), + close_code, + relay_energy: RelayEnergy::unmetered(), + tally, + }; + let signed = SignedSessionSummary::sign(summary, &relay.inner.key); + if let Some(path) = &relay.config().summary.out_path + && let Ok(line) = serde_json::to_string(&signed) + { + use std::io::Write; + if let Ok(mut f) = std::fs::OpenOptions::new().create(true).append(true).open(path) { + let _ = writeln!(f, "{line}"); + } + } + tracing::info!(session = ctx.id, transport = ctx.kind.label(), "session ended"); + relay.inner.summaries.lock().unwrap().push(signed); + relay.inner.sessions_gone.notify_waiters(); + relay.inner.change.notify_waiters(); +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/summary.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/summary.rs new file mode 100644 index 000000000..98bbf08d8 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/summary.rs @@ -0,0 +1,198 @@ +//! What the relay did for each session, counted as it happens and signed when the +//! session ends. +//! +//! The summary is relay-scoped telemetry (WAI `jwp-receipts.md` §4.2: "local to the relay +//! and MUST NOT be conflated with publisher-issued receipts"), signed by the relay's own +//! key. It counts bytes and objects. It states energy only as its acquisition class +//! allows: no meter backs this relay, so its own figure is the unmetered marker, 0 with +//! the class `unmetered`, which a reader must not present as zero joules (WAI +//! `energy-measurement.md` §2). Where the door verified an object's receipt, the +//! publisher's signed figure for producing that object is summed under the class it was +//! signed with, and labelled as the publisher's: it is not the relay's energy. + +use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey}; +use serde::{Deserialize, Serialize}; +use std::collections::BTreeMap; + +/// The format name a summary carries. +pub const FORMAT: &str = "joule-moq-relay/session-summary/1"; +/// The domain the relay's signature covers, before the summary's bytes. +pub const SIGNING_DOMAIN: &[u8] = b"joule-moq-relay:session-summary\x01"; + +/// One track as one downstream session received it. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct TrackTally { + pub objects_forwarded: u64, + pub object_bytes_forwarded: u64, + /// Objects not sent because their delivery timeout passed at this hop (§5.2), the + /// rest of a subgroup reset for that reason among them. + pub dropped_delivery_timeout: u64, + /// Of those, the ones on a track whose receipts track this session also subscribed + /// to: their receipts travel there, with no delivery timeout. + pub dropped_receipts_track_subscribed: u64, + /// Subgroups reset because SUBGROUP_DELIVERY_TIMEOUT passed before all their data + /// was acknowledged. + pub subgroups_reset_after_fin: u64, + /// Objects served on a fetch or fill stream. + pub fetch_objects: u64, +} + +/// What a session received as a subscriber. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct SubscriberTally { + pub objects_forwarded: u64, + pub object_bytes_forwarded: u64, + /// Every byte written to the session's data streams and datagrams. + pub wire_bytes_sent: u64, + pub dropped_delivery_timeout: u64, + pub dropped_receipts_track_subscribed: u64, + pub fetch_objects: u64, + /// Objects still queued when the session ended. + pub undelivered_at_close: u64, + pub tracks: BTreeMap, +} + +/// What a session sent as a publisher. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct PublisherTally { + pub objects_received: u64, + pub object_bytes_received: u64, + /// Objects the door refused (WAI jwp-receipts §4.1), with the groups it dropped. + pub refused_at_door: u64, + /// Objects already received from another publisher of the track (§7.3). + pub deduplicated: u64, +} + +/// Publisher-signed production figures of objects delivered to the session, summed +/// under the class their leaves were signed with. Only objects whose receipts the door +/// verified are counted. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct SignedFigure { + pub objects: u64, + pub joules_micro: u64, +} + +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct Tally { + pub subscriber: SubscriberTally, + pub publisher: PublisherTally, + /// Keyed by acquisition class label (`unmetered` for the marker). + pub publisher_signed: BTreeMap, +} + +impl Tally { + pub fn track(&mut self, name: &str) -> &mut TrackTally { + self.subscriber.tracks.entry(name.to_string()).or_default() + } +} + +/// The relay's own energy for the session, as its acquisition class allows. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct RelayEnergy { + /// `unmetered`: no meter backs the figure. + pub acquisition: String, + /// 0 is the unmetered marker, never a measurement of zero joules. + pub joules_micro: u64, +} + +impl RelayEnergy { + pub fn unmetered() -> Self { + RelayEnergy { acquisition: "unmetered".into(), joules_micro: 0 } + } +} + +/// One session's summary, as the relay signs it. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct SessionSummary { + pub format: String, + /// The relay's Ed25519 public key, hex. + pub relay_key: String, + pub session: u64, + pub transport: String, + pub remote: String, + pub draft: String, + pub opened_unix_ms: u64, + pub closed_unix_ms: u64, + /// The session termination code it closed with (§12.2), when known. + pub close_code: Option, + pub relay_energy: RelayEnergy, + pub tally: Tally, +} + +/// A summary and the relay's signature over [`SIGNING_DOMAIN`] and its JSON bytes. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct SignedSessionSummary { + pub summary: SessionSummary, + /// Ed25519, hex. + pub signature: String, +} + +fn signed_bytes(s: &SessionSummary) -> Vec { + let mut b = SIGNING_DOMAIN.to_vec(); + b.extend_from_slice(&serde_json::to_vec(s).expect("a summary serialises")); + b +} + +impl SignedSessionSummary { + pub fn sign(summary: SessionSummary, key: &SigningKey) -> Self { + let sig = key.sign(&signed_bytes(&summary)); + SignedSessionSummary { summary, signature: hex::encode(sig.to_bytes()) } + } + + /// Check the signature under the key the summary names. + pub fn verify(&self) -> Result<(), String> { + let k: [u8; 32] = hex::decode(&self.summary.relay_key).map_err(|e| e.to_string())?.try_into().map_err(|_| "a relay key of 32 bytes".to_string())?; + let key = VerifyingKey::from_bytes(&k).map_err(|e| e.to_string())?; + let s: [u8; 64] = hex::decode(&self.signature).map_err(|e| e.to_string())?.try_into().map_err(|_| "a signature of 64 bytes".to_string())?; + key.verify(&signed_bytes(&self.summary), &Signature::from_bytes(&s)).map_err(|e| e.to_string()) + } +} + +/// The class label of a receipt leaf's class bytes (WAI energy-measurement §2.7 tags). +pub fn class_label(class: Option<&[u8]>) -> &'static str { + match class.and_then(|c| c.first()) { + None => "unlabelled", + Some(0) => "unmetered", + Some(1) => "HwShunt", + Some(2) => "ModelBased", + Some(3) => "Estimator", + Some(4) => "OnChipCounter", + Some(5) => "CalibratedInstrument", + Some(_) => "unknown", + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_summary_verifies_and_a_change_does_not() { + let key = SigningKey::from_bytes(&[3; 32]); + let mut tally = Tally::default(); + tally.subscriber.objects_forwarded = 4; + tally.track("a/b").dropped_delivery_timeout = 1; + let s = SessionSummary { + format: FORMAT.into(), + relay_key: hex::encode(key.verifying_key().to_bytes()), + session: 7, + transport: "quic".into(), + remote: "127.0.0.1:1".into(), + draft: "moqt-22".into(), + opened_unix_ms: 1, + closed_unix_ms: 2, + close_code: Some(0), + relay_energy: RelayEnergy::unmetered(), + tally, + }; + let signed = SignedSessionSummary::sign(s, &key); + signed.verify().unwrap(); + let json = serde_json::to_string(&signed).unwrap(); + let back: SignedSessionSummary = serde_json::from_str(&json).unwrap(); + back.verify().unwrap(); + let mut forged = back.clone(); + forged.summary.tally.subscriber.objects_forwarded = 5; + assert!(forged.verify().is_err()); + assert_eq!(back.summary.relay_energy.acquisition, "unmetered"); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/capsule.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/capsule.rs new file mode 100644 index 000000000..799ac00f2 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/capsule.rs @@ -0,0 +1,272 @@ +//! The capsules the WebSocket binding carries: those of WebTransport over HTTP/2 +//! (`draft-ietf-webtrans-http2-15` §6), one capsule to each WebSocket binary message. +//! +//! A capsule is `Type (i) Length (i) Value`, with QUIC's variable-length integers +//! (RFC 9000 §16), not MOQT's: the integers here frame the binding, and the MOQT bytes a +//! stream carries ride inside `WT_STREAM` unchanged. + +/// Capsule types (webtrans-http2 §6, webtrans-http3 §5 and §4.6). +pub mod ty { + pub const DATAGRAM: u64 = 0x00; + pub const CLOSE_SESSION: u64 = 0x2843; + pub const DRAIN_SESSION: u64 = 0x78ae; + pub const PADDING: u64 = 0x190b_4d38; + pub const RESET_STREAM: u64 = 0x190b_4d39; + pub const STOP_SENDING: u64 = 0x190b_4d3a; + /// WT_STREAM with the FIN bit. + pub const STREAM_FIN: u64 = 0x190b_4d3b; + /// WT_STREAM without it. + pub const STREAM: u64 = 0x190b_4d3c; + pub const MAX_DATA: u64 = 0x190b_4d3d; + pub const MAX_STREAM_DATA: u64 = 0x190b_4d3e; + pub const MAX_STREAMS_BIDI: u64 = 0x190b_4d3f; + pub const MAX_STREAMS_UNI: u64 = 0x190b_4d40; + pub const DATA_BLOCKED: u64 = 0x190b_4d41; + pub const STREAM_DATA_BLOCKED: u64 = 0x190b_4d42; + pub const STREAMS_BLOCKED_BIDI: u64 = 0x190b_4d43; + pub const STREAMS_BLOCKED_UNI: u64 = 0x190b_4d44; +} + +/// The largest QUIC varint, 2^62-1. +pub const QUIC_VARINT_MAX: u64 = (1 << 62) - 1; +/// The largest Maximum Streams a WT_MAX_STREAMS may carry (webtrans-http2 §6.7). +pub const MAX_STREAMS_LIMIT: u64 = 1 << 60; +/// The longest close message (webtrans-http2 §6.12). +pub const MAX_CLOSE_MESSAGE: usize = 1024; + +/// Append a QUIC varint; values above 2^62-1 are clamped, which no caller here sends. +pub fn put_qvarint(out: &mut Vec, v: u64) { + let v = v.min(QUIC_VARINT_MAX); + if v < 1 << 6 { + out.push(v as u8); + } else if v < 1 << 14 { + out.extend_from_slice(&((v as u16) | 0x4000).to_be_bytes()); + } else if v < 1 << 30 { + out.extend_from_slice(&((v as u32) | 0x8000_0000).to_be_bytes()); + } else { + out.extend_from_slice(&(v | 0xc000_0000_0000_0000).to_be_bytes()); + } +} + +/// Read a QUIC varint from the front of `b`: the value and its length. +pub fn get_qvarint(b: &[u8]) -> Option<(u64, usize)> { + let first = *b.first()?; + let len = 1usize << (first >> 6); + if b.len() < len { + return None; + } + let mut v = (first & 0x3f) as u64; + for x in &b[1..len] { + v = (v << 8) | *x as u64; + } + Some((v, len)) +} + +/// One capsule of the binding. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Capsule { + Stream { id: u64, data: Vec, fin: bool }, + ResetStream { id: u64, code: u64, reliable_size: u64 }, + StopSending { id: u64, code: u64 }, + MaxStreams { bidi: bool, max: u64 }, + StreamsBlocked { bidi: bool, max: u64 }, + Datagram(Vec), + CloseSession { code: u32, message: Vec }, + DrainSession, + /// A capsule the binding does not act on (PADDING, the data flow-control capsules, + /// or a type it does not know): HTTP capsules of unknown type are skipped. + Ignored(u64), +} + +/// Why a message is not a capsule of the binding. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum CapsuleError { + #[error("the message is not one whole capsule")] + Framing, + #[error("a {0} capsule whose value does not parse")] + Value(&'static str), +} + +struct Cursor<'a>(&'a [u8]); + +impl<'a> Cursor<'a> { + fn v(&mut self, what: &'static str) -> Result { + let (v, n) = get_qvarint(self.0).ok_or(CapsuleError::Value(what))?; + self.0 = &self.0[n..]; + Ok(v) + } + fn end(&self, what: &'static str) -> Result<(), CapsuleError> { + if self.0.is_empty() { Ok(()) } else { Err(CapsuleError::Value(what)) } + } +} + +impl Capsule { + pub fn encode(&self) -> Vec { + let mut v = Vec::new(); + let t = match self { + Capsule::Stream { id, data, fin } => { + put_qvarint(&mut v, *id); + v.extend_from_slice(data); + if *fin { ty::STREAM_FIN } else { ty::STREAM } + } + Capsule::ResetStream { id, code, reliable_size } => { + put_qvarint(&mut v, *id); + put_qvarint(&mut v, *code); + put_qvarint(&mut v, *reliable_size); + ty::RESET_STREAM + } + Capsule::StopSending { id, code } => { + put_qvarint(&mut v, *id); + put_qvarint(&mut v, *code); + ty::STOP_SENDING + } + Capsule::MaxStreams { bidi, max } => { + put_qvarint(&mut v, *max); + if *bidi { ty::MAX_STREAMS_BIDI } else { ty::MAX_STREAMS_UNI } + } + Capsule::StreamsBlocked { bidi, max } => { + put_qvarint(&mut v, *max); + if *bidi { ty::STREAMS_BLOCKED_BIDI } else { ty::STREAMS_BLOCKED_UNI } + } + Capsule::Datagram(d) => { + v.extend_from_slice(d); + ty::DATAGRAM + } + Capsule::CloseSession { code, message } => { + v.extend_from_slice(&code.to_be_bytes()); + v.extend_from_slice(&message[..message.len().min(MAX_CLOSE_MESSAGE)]); + ty::CLOSE_SESSION + } + Capsule::DrainSession => ty::DRAIN_SESSION, + Capsule::Ignored(t) => *t, + }; + let mut out = Vec::with_capacity(v.len() + 16); + put_qvarint(&mut out, t); + put_qvarint(&mut out, v.len() as u64); + out.extend_from_slice(&v); + out + } + + /// Read a message that holds exactly one capsule. + pub fn decode(msg: &[u8]) -> Result { + let mut c = Cursor(msg); + let t = c.v("type").map_err(|_| CapsuleError::Framing)?; + let len = c.v("length").map_err(|_| CapsuleError::Framing)?; + if c.0.len() as u64 != len { + return Err(CapsuleError::Framing); + } + let mut c = Cursor(c.0); + Ok(match t { + ty::STREAM | ty::STREAM_FIN => { + let id = c.v("WT_STREAM")?; + Capsule::Stream { id, data: c.0.to_vec(), fin: t == ty::STREAM_FIN } + } + ty::RESET_STREAM => { + let id = c.v("WT_RESET_STREAM")?; + let code = c.v("WT_RESET_STREAM")?; + let reliable_size = c.v("WT_RESET_STREAM")?; + c.end("WT_RESET_STREAM")?; + Capsule::ResetStream { id, code, reliable_size } + } + ty::STOP_SENDING => { + let id = c.v("WT_STOP_SENDING")?; + let code = c.v("WT_STOP_SENDING")?; + c.end("WT_STOP_SENDING")?; + Capsule::StopSending { id, code } + } + ty::MAX_STREAMS_BIDI | ty::MAX_STREAMS_UNI => { + let max = c.v("WT_MAX_STREAMS")?; + c.end("WT_MAX_STREAMS")?; + Capsule::MaxStreams { bidi: t == ty::MAX_STREAMS_BIDI, max } + } + ty::STREAMS_BLOCKED_BIDI | ty::STREAMS_BLOCKED_UNI => { + let max = c.v("WT_STREAMS_BLOCKED")?; + c.end("WT_STREAMS_BLOCKED")?; + Capsule::StreamsBlocked { bidi: t == ty::STREAMS_BLOCKED_BIDI, max } + } + ty::DATAGRAM => Capsule::Datagram(c.0.to_vec()), + ty::CLOSE_SESSION => { + if c.0.len() < 4 || c.0.len() > 4 + MAX_CLOSE_MESSAGE { + return Err(CapsuleError::Value("WT_CLOSE_SESSION")); + } + let code = u32::from_be_bytes([c.0[0], c.0[1], c.0[2], c.0[3]]); + let message = c.0[4..].to_vec(); + if std::str::from_utf8(&message).is_err() { + return Err(CapsuleError::Value("WT_CLOSE_SESSION")); + } + Capsule::CloseSession { code, message } + } + ty::DRAIN_SESSION => { + c.end("WT_DRAIN_SESSION")?; + Capsule::DrainSession + } + other => Capsule::Ignored(other), + }) + } +} + +/// Stream IDs as QUIC numbers them (RFC 9000 §2.1; webtrans-http2 §5.2): bit 0 the +/// initiator (0 the client), bit 1 the direction (0 bidirectional). +pub mod stream_id { + pub fn client_initiated(id: u64) -> bool { + id & 1 == 0 + } + pub fn bidi(id: u64) -> bool { + id & 2 == 0 + } + /// The `n`th stream (from 0) of a kind. + pub fn nth(n: u64, client: bool, bidi: bool) -> u64 { + (n << 2) | (!client as u64) | ((!bidi as u64) << 1) + } + /// Which stream of its kind `id` is (from 0). + pub fn index(id: u64) -> u64 { + id >> 2 + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn quic_varints() { + for v in [0u64, 63, 64, 16383, 16384, (1 << 30) - 1, 1 << 30, QUIC_VARINT_MAX] { + let mut b = Vec::new(); + put_qvarint(&mut b, v); + assert_eq!(get_qvarint(&b), Some((v, b.len())), "{v}"); + } + assert_eq!(get_qvarint(&[0x40]), None); + } + + #[test] + fn every_capsule_round_trips() { + for c in [ + Capsule::Stream { id: 0, data: b"moqt".to_vec(), fin: false }, + Capsule::Stream { id: 7, data: vec![], fin: true }, + Capsule::ResetStream { id: 3, code: 2, reliable_size: 0 }, + Capsule::StopSending { id: 4, code: 1 }, + Capsule::MaxStreams { bidi: true, max: 100 }, + Capsule::MaxStreams { bidi: false, max: 1 << 20 }, + Capsule::StreamsBlocked { bidi: true, max: 5 }, + Capsule::Datagram(vec![1, 2, 3]), + Capsule::CloseSession { code: 0x10, message: b"going away".to_vec() }, + Capsule::DrainSession, + ] { + assert_eq!(Capsule::decode(&c.encode()).unwrap(), c); + } + assert_eq!(Capsule::decode(&Capsule::Ignored(ty::PADDING).encode()).unwrap(), Capsule::Ignored(ty::PADDING)); + let mut b = Capsule::Datagram(vec![1]).encode(); + b.push(0); + assert_eq!(Capsule::decode(&b), Err(CapsuleError::Framing)); + } + + #[test] + fn stream_ids() { + assert_eq!(stream_id::nth(0, true, true), 0); + assert_eq!(stream_id::nth(0, false, true), 1); + assert_eq!(stream_id::nth(0, true, false), 2); + assert_eq!(stream_id::nth(3, false, false), 15); + assert!(stream_id::client_initiated(4) && stream_id::bidi(4) && !stream_id::bidi(6)); + assert_eq!(stream_id::index(15), 3); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mod.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mod.rs new file mode 100644 index 000000000..ac989f95c --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mod.rs @@ -0,0 +1,131 @@ +//! The transports a MOQT session runs on, behind one interface. +//! +//! MOQT needs streams, datagrams and a way to close the session +//! (`draft-ietf-moq-transport-22` §6.2: "Both provide streams and datagrams with +//! similar semantics"). [`Connection`], [`SendStream`] and [`RecvStream`] are that +//! surface, and the relay is written once against them. There are four +//! implementations: +//! +//! - [`quic`]: native QUIC on quinn, ALPN `moqt-22` (§6.2.2); +//! - [`webtransport`]: WebTransport over HTTP/3 (`draft-ietf-webtrans-http3-16`) on +//! wtransport, negotiated by `WT-Available-Protocols` (§6.2.1); +//! - [`websocket`]: the fallback for paths that carry no UDP. Each WebSocket binary +//! message is one capsule of `draft-ietf-webtrans-http2-15` §6 ([`capsule`]), so a +//! WebSocket carries what a WebTransport session over HTTP/2 carries, multiplexed by +//! [`mux`]; +//! - [`mux::memory_pair`]: the same multiplexer over an in-process pipe, for tests and +//! for embedding. + +pub mod capsule; +pub mod mux; +pub mod quic; +pub mod webtransport; +pub mod websocket; + +use async_trait::async_trait; +use bytes::Bytes; +use std::sync::Arc; + +/// Which transport a session runs on. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum TransportKind { + Quic, + WebTransport, + WebSocket, + Memory, +} + +impl TransportKind { + pub fn label(self) -> &'static str { + match self { + TransportKind::Quic => "quic", + TransportKind::WebTransport => "webtransport", + TransportKind::WebSocket => "websocket", + TransportKind::Memory => "memory", + } + } +} + +/// Why a transport operation did not complete. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum TransportError { + /// The session is closed. `code` is the application error code it closed with + /// (MOQT's session termination codes, §12.2) when one is known. + #[error("the session closed (code {code:#x}, by the peer: {by_peer}): {reason}")] + Closed { code: u64, reason: String, by_peer: bool }, + /// The peer reset the stream this side reads. + #[error("the peer reset the stream with code {0:#x}")] + Reset(u64), + /// The peer asked this side to stop sending on the stream. + #[error("the peer stopped the stream with code {0:#x}")] + Stopped(u64), + /// A write did not finish before its object's delivery deadline. + #[error("the delivery deadline passed during the write")] + Deadline, + #[error("{0}")] + Other(String), +} + +impl TransportError { + pub fn other(e: impl std::fmt::Display) -> Self { + TransportError::Other(e.to_string()) + } +} + +pub type TResult = Result; + +/// The sending half of a stream. +#[async_trait] +pub trait SendStream: Send + Sync { + /// Write all of `buf`, waiting for flow control as the transport requires. + async fn write_all(&mut self, buf: &[u8]) -> TResult<()>; + /// End the stream gracefully (a FIN). + fn finish(&mut self) -> TResult<()>; + /// End the stream abruptly with an application code (RESET_STREAM). + fn reset(&mut self, code: u64); + /// The transport's priority for this stream's buffered bytes: higher first. + fn set_priority(&mut self, priority: i32); + /// After [`finish`](Self::finish): `Ok(None)` once the peer has acknowledged every + /// byte ("all data committed", the point SUBGROUP_DELIVERY_TIMEOUT waits for, + /// §5.2), `Ok(Some(code))` when the peer stopped the stream first. + async fn delivered(&mut self) -> TResult>; +} + +/// The receiving half of a stream. +#[async_trait] +pub trait RecvStream: Send + Sync { + /// The next bytes, `None` at the FIN, [`TransportError::Reset`] when the peer reset it. + async fn read_chunk(&mut self) -> TResult>; + /// Ask the peer to stop sending (STOP_SENDING). + fn stop(&mut self, code: u64); +} + +pub type BoxSend = Box; +pub type BoxRecv = Box; + +/// One transport session: streams both ways, datagrams, and its close. +#[async_trait] +pub trait Connection: Send + Sync { + fn kind(&self) -> TransportKind; + /// The peer, for logs and the session summary. + fn remote(&self) -> String; + async fn open_uni(&self) -> TResult; + async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)>; + async fn accept_uni(&self) -> TResult; + async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)>; + /// Send one datagram. Unreliable on QUIC and WebTransport over HTTP/3; reliable and + /// ordered on the WebSocket binding, as on WebTransport over HTTP/2. + fn send_datagram(&self, payload: Bytes) -> TResult<()>; + async fn recv_datagram(&self) -> TResult; + /// Close the session with a MOQT session termination code (§6.6, §12.2). + fn close(&self, code: u64, reason: &[u8]); + /// Resolves when the session is closed, with why. + async fn closed(&self) -> TransportError; + /// The transport's own counters, for diagnosis, where it keeps any. + fn diagnostics(&self) -> Option { + None + } +} + +pub type SharedConnection = Arc; diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mux.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mux.rs new file mode 100644 index 000000000..489969f21 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/mux.rs @@ -0,0 +1,818 @@ +//! Streams and datagrams multiplexed over a pipe of whole messages, each message one +//! capsule of `draft-ietf-webtrans-http2-15` §6 ([`super::capsule`]). +//! +//! This is what a WebTransport session over HTTP/2 does over its CONNECT stream, put on +//! a pipe that already frames messages: a WebSocket ([`super::websocket`]), or an +//! in-process channel ([`memory_pair`]). What it keeps of webtrans-http2: +//! +//! - stream IDs numbered as QUIC numbers them, opened implicitly by their first +//! `WT_STREAM` (§5.2, §6.4), with the FIN bit in the capsule type; +//! - `WT_RESET_STREAM` and `WT_STOP_SENDING` (§6.2, §6.3); +//! - stream-count flow control by `WT_MAX_STREAMS` (§6.7): each end sends its limits +//! first, and until it has, the other opens nothing. A stream's credit returns when +//! both of its halves are done, and is granted back in batches. This bounds the +//! requests in flight, which MOQT -22 leaves to the transport's stream limits; +//! - `DATAGRAM` capsules, reliable and ordered as on HTTP/2 (§6.11); +//! - `WT_CLOSE_SESSION` with a 32-bit code (§6.12), and `WT_DRAIN_SESSION` (§6.13). +//! +//! What it does not do: the data flow-control capsules (`WT_MAX_DATA`, +//! `WT_MAX_STREAM_DATA` and their BLOCKED forms). The pipe's own backpressure (TCP under +//! a WebSocket) bounds the sender, and the receiver bounds what it buffers: past +//! `max_buffered` bytes not yet read it closes the session. Capsules of those types are +//! read and ignored. + +use super::capsule::{stream_id, Capsule}; +use super::{BoxRecv, BoxSend, Connection, RecvStream, SendStream, TResult, TransportError, TransportKind}; +use async_trait::async_trait; +use bytes::Bytes; +use std::collections::{BTreeSet, HashMap}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use tokio::sync::{mpsc, watch}; + +/// The sending end of a message pipe. +#[async_trait] +pub trait PipeTx: Send + 'static { + async fn send(&mut self, msg: Vec) -> TResult<()>; + async fn close(&mut self); +} + +/// The receiving end of a message pipe: `None` once it is closed. +#[async_trait] +pub trait PipeRx: Send + 'static { + async fn recv(&mut self) -> TResult>>; +} + +/// Limits of one multiplexed session. +#[derive(Clone, Copy, Debug)] +pub struct MuxConfig { + /// Bidirectional streams the peer may have open at once (each a MOQT request). + pub max_bidi: u64, + /// Unidirectional streams the peer may have open at once. + pub max_uni: u64, + /// Bytes received and not yet read, across all streams, before the session closes. + pub max_buffered: usize, + /// Datagrams received and not yet read; more are dropped. + pub datagram_queue: usize, + /// Bytes queued for the pipe before a writer waits. + pub send_high_water: usize, +} + +impl Default for MuxConfig { + fn default() -> Self { + MuxConfig { max_bidi: 256, max_uni: 1024, max_buffered: 64 << 20, datagram_queue: 1024, send_high_water: 1 << 20 } + } +} + +/// The largest stream data one capsule carries. +const CHUNK: usize = 16 * 1024; +/// The session code a receiver closes with when its buffer bound is passed: the value of +/// WT_FLOW_CONTROL_ERROR (webtrans-http3 §9.4), as a 32-bit application code. +pub const FLOW_CONTROL_ERROR: u64 = 0x045d_4487; +/// How far ahead of the lowest unseen one the peer's stream IDs may arrive. +const SEEN_WINDOW: usize = 1 << 16; + +enum RecvEvent { + Data(Bytes), + Fin, + Reset(u64), +} + +#[derive(Default)] +struct SendShared { + stopped: Mutex>, +} + +/// The peer's stream indices of one kind: which are new. +#[derive(Default)] +struct Seen { + floor: u64, + above: BTreeSet, +} + +impl Seen { + /// Admit index `i`: `true` when it was not seen before. + fn admit(&mut self, i: u64) -> Option { + if i < self.floor || self.above.contains(&i) { + return Some(false); + } + if self.above.len() >= SEEN_WINDOW { + return None; + } + self.above.insert(i); + while self.above.remove(&self.floor) { + self.floor += 1; + } + Some(true) + } +} + +struct State { + next_bidi: u64, + next_uni: u64, + peer_max_bidi: u64, + peer_max_uni: u64, + local_max_bidi: u64, + local_max_uni: u64, + pending_bidi: u64, + pending_uni: u64, + seen_bidi: Seen, + seen_uni: Seen, + recv: HashMap>, + send: HashMap>, + accept_uni: Option>, + accept_bi: Option>, + datagrams: Option>, + blocked_sent: (u64, u64), +} + +struct Inner { + client: bool, + kind: TransportKind, + remote: String, + cfg: MuxConfig, + out: mpsc::UnboundedSender>, + queued: AtomicUsize, + space: watch::Sender, + state: Mutex, + /// Bumped whenever stream credit changes or the session closes. + generation: watch::Sender, + closed: watch::Sender>, + buffered: AtomicUsize, + accept_uni_rx: tokio::sync::Mutex>, + accept_bi_rx: tokio::sync::Mutex>, + datagram_rx: tokio::sync::Mutex>, +} + +/// A stream's credit: returned to the peer when the last half holding it is dropped. +struct Credit { + inner: std::sync::Weak, + bidi: bool, +} + +impl Drop for Credit { + fn drop(&mut self) { + if let Some(inner) = self.inner.upgrade() { + inner.return_credit(self.bidi); + } + } +} + +impl Inner { + fn is_closed(&self) -> Option { + self.closed.borrow().clone() + } + + fn closed_err(&self) -> TransportError { + self.is_closed().unwrap_or(TransportError::Closed { code: 0, reason: "the session closed".into(), by_peer: false }) + } + + fn push(&self, c: &Capsule) { + let b = c.encode(); + self.queued.fetch_add(b.len(), Ordering::AcqRel); + let _ = self.out.send(b); + } + + fn bump(&self) { + self.generation.send_modify(|g| *g += 1); + } + + /// Mark the session closed with `why` (the first reason wins) and wake everyone. + fn set_closed(&self, why: TransportError) { + let mut first = false; + self.closed.send_if_modified(|c| { + if c.is_none() { + *c = Some(why.clone()); + first = true; + true + } else { + false + } + }); + if first { + let dropped = { + let mut st = self.state.lock().unwrap(); + st.accept_uni = None; + st.accept_bi = None; + st.datagrams = None; + std::mem::take(&mut st.recv) + }; + drop(dropped); + // Streams nobody accepted hold this session; drop them, unlocked, so it can go. + if let Ok(mut rx) = self.accept_uni_rx.try_lock() { + while rx.try_recv().is_ok() {} + } + if let Ok(mut rx) = self.accept_bi_rx.try_lock() { + while rx.try_recv().is_ok() {} + } + } + self.bump(); + } + + fn return_credit(&self, bidi: bool) { + let grant = { + let mut guard = self.state.lock().unwrap(); + let st = &mut *guard; + let (pending, max, initial) = if bidi { + (&mut st.pending_bidi, &mut st.local_max_bidi, self.cfg.max_bidi) + } else { + (&mut st.pending_uni, &mut st.local_max_uni, self.cfg.max_uni) + }; + *pending += 1; + if *pending >= (initial / 8).max(1) { + *max += *pending; + *pending = 0; + Some(*max) + } else { + None + } + }; + if let Some(max) = grant + && self.is_closed().is_none() + { + self.push(&Capsule::MaxStreams { bidi, max }); + } + } + + fn close_with(&self, code: u64, reason: &[u8]) { + if self.is_closed().is_some() { + return; + } + let code32 = u32::try_from(code).unwrap_or(u32::MAX); + let mut msg = reason.to_vec(); + msg.truncate(super::capsule::MAX_CLOSE_MESSAGE); + while std::str::from_utf8(&msg).is_err() { + msg.pop(); + } + self.push(&Capsule::CloseSession { code: code32, message: msg }); + self.set_closed(TransportError::Closed { code, reason: String::from_utf8_lossy(reason).into_owned(), by_peer: false }); + } + + /// One capsule from the peer. + fn on_capsule(self: &Arc, c: Capsule) -> Result<(), (u64, &'static str)> { + match c { + Capsule::Stream { id, data, fin } => self.on_stream(id, Some(data), fin, None), + Capsule::ResetStream { id, code, .. } => self.on_stream(id, None, false, Some(code)), + Capsule::StopSending { id, code } => { + let st = self.state.lock().unwrap(); + if let Some(s) = st.send.get(&id) { + *s.stopped.lock().unwrap() = Some(code); + } + Ok(()) + } + Capsule::MaxStreams { bidi, max } => { + if max > super::capsule::MAX_STREAMS_LIMIT { + return Err((FLOW_CONTROL_ERROR, "a WT_MAX_STREAMS above 2^60")); + } + let mut st = self.state.lock().unwrap(); + let cur = if bidi { &mut st.peer_max_bidi } else { &mut st.peer_max_uni }; + if max < *cur { + return Err((FLOW_CONTROL_ERROR, "a WT_MAX_STREAMS that lowers the limit")); + } + *cur = max; + drop(st); + self.bump(); + Ok(()) + } + Capsule::Datagram(d) => { + let st = self.state.lock().unwrap(); + if let Some(tx) = &st.datagrams { + let _ = tx.try_send(Bytes::from(d)); + } + Ok(()) + } + Capsule::CloseSession { code, message } => { + self.set_closed(TransportError::Closed { code: code as u64, reason: String::from_utf8_lossy(&message).into_owned(), by_peer: true }); + Ok(()) + } + Capsule::DrainSession | Capsule::StreamsBlocked { .. } | Capsule::Ignored(_) => Ok(()), + } + } + + fn on_stream(self: &Arc, id: u64, data: Option>, fin: bool, reset: Option) -> Result<(), (u64, &'static str)> { + let peer_initiated = stream_id::client_initiated(id) != self.client; + let bidi = stream_id::bidi(id); + let mut st = self.state.lock().unwrap(); + if !st.recv.contains_key(&id) { + if !peer_initiated { + if !bidi { + return Err((FLOW_CONTROL_ERROR, "data on a unidirectional stream this end opened")); + } + // The peer's half of a stream this end opened and has stopped reading. + return Ok(()); + } + let idx = stream_id::index(id); + let limit = if bidi { st.local_max_bidi } else { st.local_max_uni }; + if idx >= limit { + return Err((FLOW_CONTROL_ERROR, "a stream past WT_MAX_STREAMS")); + } + let seen = if bidi { &mut st.seen_bidi } else { &mut st.seen_uni }; + match seen.admit(idx) { + None => return Err((FLOW_CONTROL_ERROR, "stream IDs too far out of order")), + Some(false) => return Ok(()), + Some(true) => {} + } + let (tx, rx) = mpsc::unbounded_channel(); + st.recv.insert(id, tx); + let shared = bidi.then(|| { + let s = Arc::new(SendShared::default()); + st.send.insert(id, s.clone()); + s + }); + let (accept_bi, accept_uni) = (st.accept_bi.clone(), st.accept_uni.clone()); + drop(st); + // Built and handed over with the state unlocked: a stream half dropped here + // (nobody accepting) takes the lock in its own Drop. + let credit = Arc::new(Credit { inner: Arc::downgrade(self), bidi }); + let recv = MuxRecv { id, rx, inner: self.clone(), done: false, _credit: Some(credit.clone()) }; + match (shared, accept_bi, accept_uni) { + (Some(shared), Some(a), _) => { + let send = MuxSend { id, inner: self.clone(), shared, ended: false, _credit: Some(credit) }; + let _ = a.send((Box::new(send), Box::new(recv))); + } + (None, _, Some(a)) => { + let _ = a.send(Box::new(recv)); + } + _ => {} + } + st = self.state.lock().unwrap(); + } + let tx = st.recv.get(&id).cloned(); + if fin || reset.is_some() { + st.recv.remove(&id); + } + drop(st); + let Some(tx) = tx else { return Ok(()) }; + if let Some(d) = data.filter(|d| !d.is_empty()) { + let n = self.buffered.fetch_add(d.len(), Ordering::AcqRel) + d.len(); + if n > self.cfg.max_buffered { + return Err((FLOW_CONTROL_ERROR, "more received and unread than the session buffers")); + } + let _ = tx.send(RecvEvent::Data(Bytes::from(d))); + } + if let Some(code) = reset { + let _ = tx.send(RecvEvent::Reset(code)); + } else if fin { + let _ = tx.send(RecvEvent::Fin); + } + Ok(()) + } + + async fn open(self: &Arc, bidi: bool) -> TResult { + let mut generation = self.generation.subscribe(); + loop { + if let Some(e) = self.is_closed() { + return Err(e); + } + { + let mut st = self.state.lock().unwrap(); + let (next, max) = if bidi { (st.next_bidi, st.peer_max_bidi) } else { (st.next_uni, st.peer_max_uni) }; + if next < max { + if bidi { st.next_bidi += 1 } else { st.next_uni += 1 } + return Ok(stream_id::nth(next, self.client, bidi)); + } + let blocked = if bidi { &mut st.blocked_sent.0 } else { &mut st.blocked_sent.1 }; + let send_blocked = *blocked != max; + *blocked = max; + drop(st); + if send_blocked { + self.push(&Capsule::StreamsBlocked { bidi, max }); + } + } + if generation.changed().await.is_err() { + return Err(self.closed_err()); + } + } + } + + async fn wait_space(&self) -> TResult<()> { + let mut rx = self.space.subscribe(); + while self.queued.load(Ordering::Acquire) > self.cfg.send_high_water { + if let Some(e) = self.is_closed() { + return Err(e); + } + if rx.changed().await.is_err() { + return Err(self.closed_err()); + } + } + Ok(()) + } +} + +pub struct MuxSend { + id: u64, + inner: Arc, + shared: Arc, + ended: bool, + _credit: Option>, +} + +pub struct MuxRecv { + id: u64, + rx: mpsc::UnboundedReceiver, + inner: Arc, + done: bool, + _credit: Option>, +} + +impl Drop for MuxSend { + fn drop(&mut self) { + if !self.ended && self.inner.is_closed().is_none() { + self.inner.push(&Capsule::Stream { id: self.id, data: vec![], fin: true }); + } + self.inner.state.lock().unwrap().send.remove(&self.id); + } +} + +impl Drop for MuxRecv { + fn drop(&mut self) { + if !self.done { + self.inner.state.lock().unwrap().recv.remove(&self.id); + if self.inner.is_closed().is_none() { + self.inner.push(&Capsule::StopSending { id: self.id, code: 0 }); + } + } + while let Ok(ev) = self.rx.try_recv() { + if let RecvEvent::Data(b) = ev { + self.inner.buffered.fetch_sub(b.len(), Ordering::AcqRel); + } + } + } +} + +#[async_trait] +impl SendStream for MuxSend { + async fn write_all(&mut self, buf: &[u8]) -> TResult<()> { + for piece in buf.chunks(CHUNK) { + if let Some(code) = *self.shared.stopped.lock().unwrap() { + return Err(TransportError::Stopped(code)); + } + if let Some(e) = self.inner.is_closed() { + return Err(e); + } + if self.ended { + return Err(TransportError::Other("a write after the stream ended".into())); + } + self.inner.wait_space().await?; + self.inner.push(&Capsule::Stream { id: self.id, data: piece.to_vec(), fin: false }); + } + Ok(()) + } + fn finish(&mut self) -> TResult<()> { + if self.ended { + return Err(TransportError::Other("the stream already ended".into())); + } + self.ended = true; + self.inner.push(&Capsule::Stream { id: self.id, data: vec![], fin: true }); + Ok(()) + } + fn reset(&mut self, code: u64) { + if !self.ended { + self.ended = true; + self.inner.push(&Capsule::ResetStream { id: self.id, code, reliable_size: 0 }); + } + } + fn set_priority(&mut self, _priority: i32) {} + async fn delivered(&mut self) -> TResult> { + // webtrans-http2 §5.2: with no acknowledgements of its own, the binding makes the + // transitions QUIC makes on an acknowledgement at once. + Ok(*self.shared.stopped.lock().unwrap()) + } +} + +#[async_trait] +impl RecvStream for MuxRecv { + async fn read_chunk(&mut self) -> TResult> { + if self.done { + return Ok(None); + } + match self.rx.recv().await { + Some(RecvEvent::Data(b)) => { + self.inner.buffered.fetch_sub(b.len(), Ordering::AcqRel); + Ok(Some(b)) + } + Some(RecvEvent::Fin) => { + self.done = true; + Ok(None) + } + Some(RecvEvent::Reset(code)) => { + self.done = true; + Err(TransportError::Reset(code)) + } + None => { + self.done = true; + Err(self.inner.closed_err()) + } + } + } + fn stop(&mut self, code: u64) { + if !self.done { + self.done = true; + self.inner.state.lock().unwrap().recv.remove(&self.id); + self.inner.push(&Capsule::StopSending { id: self.id, code }); + } + } +} + +/// One multiplexed session. +pub struct MuxConnection { + inner: Arc, +} + +impl MuxConnection { + /// Run a session over a pipe. `client` is this end's role, which numbers its streams. + /// The tasks it spawns end when the session closes. + pub fn new(client: bool, kind: TransportKind, remote: String, cfg: MuxConfig, mut tx: Box, mut rx: Box) -> Self { + let (out, mut out_rx) = mpsc::unbounded_channel::>(); + let (uni_tx, uni_rx) = mpsc::unbounded_channel(); + let (bi_tx, bi_rx) = mpsc::unbounded_channel(); + let (dg_tx, dg_rx) = mpsc::channel(cfg.datagram_queue.max(1)); + let inner = Arc::new(Inner { + client, + kind, + remote, + cfg, + out, + queued: AtomicUsize::new(0), + space: watch::channel(0).0, + state: Mutex::new(State { + next_bidi: 0, + next_uni: 0, + peer_max_bidi: 0, + peer_max_uni: 0, + local_max_bidi: cfg.max_bidi, + local_max_uni: cfg.max_uni, + pending_bidi: 0, + pending_uni: 0, + seen_bidi: Seen::default(), + seen_uni: Seen::default(), + recv: HashMap::new(), + send: HashMap::new(), + accept_uni: Some(uni_tx), + accept_bi: Some(bi_tx), + datagrams: Some(dg_tx), + blocked_sent: (u64::MAX, u64::MAX), + }), + generation: watch::channel(0).0, + closed: watch::channel(None).0, + buffered: AtomicUsize::new(0), + accept_uni_rx: tokio::sync::Mutex::new(uni_rx), + accept_bi_rx: tokio::sync::Mutex::new(bi_rx), + datagram_rx: tokio::sync::Mutex::new(dg_rx), + }); + // Each end's limits go first: until the peer's arrive, it opens nothing. + inner.push(&Capsule::MaxStreams { bidi: true, max: cfg.max_bidi }); + inner.push(&Capsule::MaxStreams { bidi: false, max: cfg.max_uni }); + + let w = inner.clone(); + tokio::spawn(async move { + let mut closed = w.closed.subscribe(); + loop { + tokio::select! { + m = out_rx.recv() => { + let Some(m) = m else { break }; + let n = m.len(); + let r = tx.send(m).await; + w.queued.fetch_sub(n, Ordering::AcqRel); + w.space.send_modify(|g| *g += 1); + if let Err(e) = r { + w.set_closed(e); + break; + } + } + _ = closed.changed() => { + // Flush what was queued before the close (the WT_CLOSE_SESSION among it). + while let Ok(m) = out_rx.try_recv() { + if tx.send(m).await.is_err() { + break; + } + } + break; + } + } + } + tx.close().await; + }); + + let r = inner.clone(); + tokio::spawn(async move { + let mut closed = r.closed.subscribe(); + loop { + let msg = tokio::select! { + m = rx.recv() => m, + _ = closed.changed() => break, + }; + match msg { + Ok(Some(m)) => match Capsule::decode(&m) { + Ok(c) => { + if let Err((code, why)) = r.on_capsule(c) { + r.close_with(code, why.as_bytes()); + break; + } + } + Err(_) => { + r.close_with(FLOW_CONTROL_ERROR, b"a message that is not one capsule"); + break; + } + }, + Ok(None) => { + r.set_closed(TransportError::Closed { code: 0, reason: "the pipe closed".into(), by_peer: true }); + break; + } + Err(e) => { + r.set_closed(e); + break; + } + } + } + }); + MuxConnection { inner } + } +} + +#[async_trait] +impl Connection for MuxConnection { + fn kind(&self) -> TransportKind { + self.inner.kind + } + fn remote(&self) -> String { + self.inner.remote.clone() + } + async fn open_uni(&self) -> TResult { + let id = self.inner.open(false).await?; + let shared = Arc::new(SendShared::default()); + self.inner.state.lock().unwrap().send.insert(id, shared.clone()); + Ok(Box::new(MuxSend { id, inner: self.inner.clone(), shared, ended: false, _credit: None })) + } + async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)> { + let id = self.inner.open(true).await?; + let shared = Arc::new(SendShared::default()); + let (tx, rx) = mpsc::unbounded_channel(); + { + let mut st = self.inner.state.lock().unwrap(); + st.send.insert(id, shared.clone()); + st.recv.insert(id, tx); + } + Ok(( + Box::new(MuxSend { id, inner: self.inner.clone(), shared, ended: false, _credit: None }), + Box::new(MuxRecv { id, rx, inner: self.inner.clone(), done: false, _credit: None }), + )) + } + async fn accept_uni(&self) -> TResult { + match self.inner.accept_uni_rx.lock().await.recv().await { + Some(r) => Ok(r), + None => Err(self.inner.closed_err()), + } + } + async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)> { + match self.inner.accept_bi_rx.lock().await.recv().await { + Some(p) => Ok(p), + None => Err(self.inner.closed_err()), + } + } + fn send_datagram(&self, payload: Bytes) -> TResult<()> { + if let Some(e) = self.inner.is_closed() { + return Err(e); + } + self.inner.push(&Capsule::Datagram(payload.to_vec())); + Ok(()) + } + async fn recv_datagram(&self) -> TResult { + match self.inner.datagram_rx.lock().await.recv().await { + Some(d) => Ok(d), + None => Err(self.inner.closed_err()), + } + } + fn close(&self, code: u64, reason: &[u8]) { + self.inner.close_with(code, reason); + } + async fn closed(&self) -> TransportError { + let mut rx = self.inner.closed.subscribe(); + loop { + if let Some(e) = rx.borrow_and_update().clone() { + return e; + } + if rx.changed().await.is_err() { + return self.inner.closed_err(); + } + } + } +} + +// ── The in-process pipe ───────────────────────────────────────────────────── + +struct MemTx { + tx: mpsc::Sender>, + gate: Option>, +} + +struct MemRx(mpsc::Receiver>); + +#[async_trait] +impl PipeTx for MemTx { + async fn send(&mut self, msg: Vec) -> TResult<()> { + if let Some(g) = &mut self.gate { + while !*g.borrow_and_update() { + if g.changed().await.is_err() { + break; + } + } + } + self.tx.send(msg).await.map_err(|_| TransportError::Closed { code: 0, reason: "the pipe closed".into(), by_peer: true }) + } + async fn close(&mut self) {} +} + +#[async_trait] +impl PipeRx for MemRx { + async fn recv(&mut self) -> TResult>> { + Ok(self.0.recv().await) + } +} + +/// A gate on what one end of a [`memory_pair`] sends: closed, it holds every message; +/// opened, it lets them through in order. +pub struct Gate(watch::Sender); + +impl Gate { + pub fn open(&self) { + let _ = self.0.send(true); + } + pub fn close(&self) { + let _ = self.0.send(false); + } +} + +/// Two ends of an in-process session: `(client, server)`. With `gated`, what the server +/// sends waits behind the returned [`Gate`], which starts closed. +pub fn memory_pair(cfg: MuxConfig, gated: bool) -> (MuxConnection, MuxConnection, Option) { + let (c2s_tx, c2s_rx) = mpsc::channel(64); + let (s2c_tx, s2c_rx) = mpsc::channel(64); + let (gate, gate_rx) = if gated { + let (t, r) = watch::channel(false); + (Some(Gate(t)), Some(r)) + } else { + (None, None) + }; + let client = MuxConnection::new(true, TransportKind::Memory, "memory:server".into(), cfg, Box::new(MemTx { tx: c2s_tx, gate: None }), Box::new(MemRx(s2c_rx))); + let server = MuxConnection::new(false, TransportKind::Memory, "memory:client".into(), cfg, Box::new(MemTx { tx: s2c_tx, gate: gate_rx }), Box::new(MemRx(c2s_rx))); + (client, server, gate) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::time::Duration; + + async fn read_all(r: &mut BoxRecv) -> TResult> { + let mut v = Vec::new(); + while let Some(b) = tokio::time::timeout(Duration::from_secs(5), r.read_chunk()).await.expect("a read in time")? { + v.extend_from_slice(&b); + } + Ok(v) + } + + #[tokio::test] + async fn streams_both_ways_and_a_reset() { + let (c, s, _) = memory_pair(MuxConfig::default(), false); + let (mut cs, mut cr) = c.open_bi().await.unwrap(); + cs.write_all(b"request").await.unwrap(); + cs.finish().unwrap(); + let (mut ss, mut sr) = tokio::time::timeout(Duration::from_secs(5), s.accept_bi()).await.unwrap().unwrap(); + assert_eq!(read_all(&mut sr).await.unwrap(), b"request"); + ss.write_all(&vec![7u8; 40_000]).await.unwrap(); + ss.finish().unwrap(); + assert_eq!(read_all(&mut cr).await.unwrap().len(), 40_000); + + let mut u = s.open_uni().await.unwrap(); + u.write_all(b"half").await.unwrap(); + u.reset(2); + let mut ur = tokio::time::timeout(Duration::from_secs(5), c.accept_uni()).await.unwrap().unwrap(); + assert_eq!(ur.read_chunk().await.unwrap().as_deref(), Some(&b"half"[..])); + assert_eq!(ur.read_chunk().await, Err(TransportError::Reset(2))); + + c.send_datagram(Bytes::from_static(b"dg")).unwrap(); + assert_eq!(tokio::time::timeout(Duration::from_secs(5), s.recv_datagram()).await.unwrap().unwrap(), Bytes::from_static(b"dg")); + + s.close(0x10, b"bye"); + match tokio::time::timeout(Duration::from_secs(5), c.closed()).await.unwrap() { + TransportError::Closed { code, by_peer, .. } => assert_eq!((code, by_peer), (0x10, true)), + other => panic!("{other:?}"), + } + } + + #[tokio::test] + async fn the_stream_limit_holds_until_credit_returns() { + let cfg = MuxConfig { max_bidi: 1, ..MuxConfig::default() }; + let (c, s, _) = memory_pair(cfg, false); + let (mut a, _ar) = c.open_bi().await.unwrap(); + a.write_all(b"one").await.unwrap(); + // A second request waits for credit. + let second = tokio::time::timeout(Duration::from_millis(200), c.open_bi()).await; + assert!(second.is_err(), "a second stream opened past WT_MAX_STREAMS"); + let (ss, sr) = tokio::time::timeout(Duration::from_secs(5), s.accept_bi()).await.unwrap().unwrap(); + drop((ss, sr)); + let opened = tokio::time::timeout(Duration::from_secs(5), c.open_bi()).await; + assert!(opened.is_ok(), "credit came back when the first stream ended"); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/quic.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/quic.rs new file mode 100644 index 000000000..97d5b8fbe --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/quic.rs @@ -0,0 +1,208 @@ +//! Native QUIC (`draft-ietf-moq-transport-22` §6.2.2) on quinn. + +use super::{BoxRecv, BoxSend, Connection, RecvStream, SendStream, TResult, TransportError, TransportKind}; +use async_trait::async_trait; +use bytes::Bytes; +use quinn::VarInt; + +/// A quinn connection whose ALPN selected `moqt-22`. +pub struct QuicConnection { + conn: quinn::Connection, +} + +impl QuicConnection { + pub fn new(conn: quinn::Connection) -> Self { + QuicConnection { conn } + } + + pub fn inner(&self) -> &quinn::Connection { + &self.conn + } +} + +/// A MOQT code as a QUIC application error code (62 bits). +pub(crate) fn varint(code: u64) -> VarInt { + VarInt::from_u64(code).unwrap_or(VarInt::MAX) +} + +/// Why a quinn connection ended, as this crate reports it. +pub(crate) fn conn_err(e: &quinn::ConnectionError) -> TransportError { + match e { + quinn::ConnectionError::ApplicationClosed(c) => TransportError::Closed { + code: c.error_code.into_inner(), + reason: String::from_utf8_lossy(&c.reason).into_owned(), + by_peer: true, + }, + quinn::ConnectionError::LocallyClosed => TransportError::Closed { code: 0, reason: "closed by this endpoint".into(), by_peer: false }, + other => TransportError::Closed { code: u64::MAX, reason: other.to_string(), by_peer: true }, + } +} + +pub(crate) fn write_err(e: quinn::WriteError) -> TransportError { + match e { + quinn::WriteError::Stopped(code) => TransportError::Stopped(code.into_inner()), + quinn::WriteError::ConnectionLost(c) => conn_err(&c), + other => TransportError::other(other), + } +} + +pub(crate) fn read_err(e: quinn::ReadError) -> TransportError { + match e { + quinn::ReadError::Reset(code) => TransportError::Reset(code.into_inner()), + quinn::ReadError::ConnectionLost(c) => conn_err(&c), + other => TransportError::other(other), + } +} + +/// The largest chunk a read hands back. +pub(crate) const READ_CHUNK: usize = 64 * 1024; + +pub struct QuicSend(pub quinn::SendStream); +pub struct QuicRecv(pub quinn::RecvStream); + +#[async_trait] +impl SendStream for QuicSend { + async fn write_all(&mut self, buf: &[u8]) -> TResult<()> { + self.0.write_all(buf).await.map_err(write_err) + } + fn finish(&mut self) -> TResult<()> { + self.0.finish().map_err(TransportError::other) + } + fn reset(&mut self, code: u64) { + let _ = self.0.reset(varint(code)); + } + fn set_priority(&mut self, priority: i32) { + let _ = self.0.set_priority(priority); + } + async fn delivered(&mut self) -> TResult> { + match self.0.stopped().await { + Ok(None) => Ok(None), + Ok(Some(code)) => Ok(Some(code.into_inner())), + Err(quinn::StoppedError::ConnectionLost(c)) => Err(conn_err(&c)), + Err(e) => Err(TransportError::other(e)), + } + } +} + +#[async_trait] +impl RecvStream for QuicRecv { + async fn read_chunk(&mut self) -> TResult> { + match self.0.read_chunk(READ_CHUNK, true).await { + Ok(Some(c)) => Ok(Some(c.bytes)), + Ok(None) => Ok(None), + Err(e) => Err(read_err(e)), + } + } + fn stop(&mut self, code: u64) { + let _ = self.0.stop(varint(code)); + } +} + +#[async_trait] +impl Connection for QuicConnection { + fn kind(&self) -> TransportKind { + TransportKind::Quic + } + fn remote(&self) -> String { + self.conn.remote_address().to_string() + } + async fn open_uni(&self) -> TResult { + let s = self.conn.open_uni().await.map_err(|e| conn_err(&e))?; + Ok(Box::new(QuicSend(s))) + } + async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)> { + let (s, r) = self.conn.open_bi().await.map_err(|e| conn_err(&e))?; + Ok((Box::new(QuicSend(s)), Box::new(QuicRecv(r)))) + } + async fn accept_uni(&self) -> TResult { + let r = self.conn.accept_uni().await.map_err(|e| conn_err(&e))?; + Ok(Box::new(QuicRecv(r))) + } + async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)> { + let (s, r) = self.conn.accept_bi().await.map_err(|e| conn_err(&e))?; + Ok((Box::new(QuicSend(s)), Box::new(QuicRecv(r)))) + } + fn send_datagram(&self, payload: Bytes) -> TResult<()> { + self.conn.send_datagram(payload).map_err(TransportError::other) + } + async fn recv_datagram(&self) -> TResult { + self.conn.read_datagram().await.map_err(|e| conn_err(&e)) + } + fn close(&self, code: u64, reason: &[u8]) { + self.conn.close(varint(code), reason); + } + async fn closed(&self) -> TransportError { + conn_err(&self.conn.closed().await) + } + fn diagnostics(&self) -> Option { + Some(quic_diagnostics(&self.conn)) + } +} + +/// The flow-control and loss counters of a quinn connection, one line. +pub fn quic_diagnostics(c: &quinn::Connection) -> String { + let s = c.stats(); + format!( + "rx DATA_BLOCKED {} STREAM_DATA_BLOCKED {} STREAMS_BLOCKED_UNI {} MAX_DATA {} RESET_STREAM {} STOP_SENDING {}; tx DATA_BLOCKED {} STREAM_DATA_BLOCKED {} STREAMS_BLOCKED_UNI {} MAX_DATA {} MAX_STREAM_DATA {} MAX_STREAMS_UNI {} RESET_STREAM {} STOP_SENDING {}; lost packets {} cwnd {}", + s.frame_rx.data_blocked, + s.frame_rx.stream_data_blocked, + s.frame_rx.streams_blocked_uni, + s.frame_rx.max_data, + s.frame_rx.reset_stream, + s.frame_rx.stop_sending, + s.frame_tx.data_blocked, + s.frame_tx.stream_data_blocked, + s.frame_tx.streams_blocked_uni, + s.frame_tx.max_data, + s.frame_tx.max_stream_data, + s.frame_tx.max_streams_uni, + s.frame_tx.reset_stream, + s.frame_tx.stop_sending, + s.path.lost_packets, + s.path.cwnd, + ) +} + +/// A client endpoint that offers `alpns` and trusts `roots`, with an optional +/// transport configuration (a subscriber that limits its receive windows, say). +pub fn client_endpoint( + bind: std::net::SocketAddr, + roots: Vec>, + alpns: Vec>, + transport: Option, +) -> TResult { + crate::cert::install_provider(); + let mut store = rustls::RootCertStore::empty(); + for c in roots { + store.add(c).map_err(TransportError::other)?; + } + let provider = std::sync::Arc::new(rustls::crypto::ring::default_provider()); + let mut tls = rustls::ClientConfig::builder_with_provider(provider) + .with_protocol_versions(&[&rustls::version::TLS13]) + .map_err(TransportError::other)? + .with_root_certificates(store) + .with_no_client_auth(); + tls.alpn_protocols = alpns; + let quic = quinn::crypto::rustls::QuicClientConfig::try_from(tls).map_err(TransportError::other)?; + let mut cc = quinn::ClientConfig::new(std::sync::Arc::new(quic)); + if let Some(t) = transport { + cc.transport_config(std::sync::Arc::new(t)); + } + let mut ep = quinn::Endpoint::client(bind).map_err(TransportError::other)?; + ep.set_default_client_config(cc); + Ok(ep) +} + +/// Dial `addr` as `server_name` and return the connection if its ALPN is `moqt-22`. +pub async fn connect(ep: &quinn::Endpoint, addr: std::net::SocketAddr, server_name: &str) -> TResult { + let conn = ep.connect(addr, server_name).map_err(TransportError::other)?.await.map_err(|e| conn_err(&e))?; + let alpn = conn + .handshake_data() + .and_then(|h| h.downcast::().ok()) + .and_then(|h| h.protocol.clone()); + if alpn.as_deref() != Some(joule_moq::draft22::ALPN) { + conn.close(varint(0), b"no MOQT ALPN"); + return Err(TransportError::Other("the server selected no moqt-22 ALPN".into())); + } + Ok(QuicConnection::new(conn)) +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/websocket.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/websocket.rs new file mode 100644 index 000000000..56d4e257c --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/websocket.rs @@ -0,0 +1,82 @@ +//! The WebSocket fallback: a non-normative binding of MOQT -22 for paths that carry no +//! UDP, where neither native QUIC nor WebTransport over HTTP/3 can reach the relay. +//! +//! The binding (the crate README states it in full): +//! +//! - the client opens a WebSocket (RFC 6455) to the relay's path with the subprotocol +//! `moqt-22` in `Sec-WebSocket-Protocol`; the relay answers with that subprotocol, or +//! refuses the upgrade. The subprotocol selects the draft, as the ALPN does on QUIC; +//! - every binary message is exactly one capsule of `draft-ietf-webtrans-http2-15` §6, and +//! the session is what a WebTransport session over HTTP/2 would carry ([`super::mux`]); +//! - MOQT's control, request and data streams ride those streams unchanged: the same +//! bytes a QUIC stream would carry. + +use super::mux::{MuxConfig, MuxConnection, PipeRx, PipeTx}; +use super::{TResult, TransportError, TransportKind}; +use async_trait::async_trait; +use futures::stream::{SplitSink, SplitStream}; +use futures::{SinkExt, StreamExt}; +use tokio::io::{AsyncRead, AsyncWrite}; +use tokio_tungstenite::tungstenite::Message; +use tokio_tungstenite::WebSocketStream; + +/// The subprotocol that selects this draft. +pub fn subprotocol() -> &'static str { + std::str::from_utf8(joule_moq::draft22::ALPN).expect("the ALPN is ASCII") +} + +struct WsTx(SplitSink, Message>); +struct WsRx(SplitStream>); + +#[async_trait] +impl PipeTx for WsTx { + async fn send(&mut self, msg: Vec) -> TResult<()> { + self.0.send(Message::Binary(msg.into())).await.map_err(TransportError::other) + } + async fn close(&mut self) { + let _ = self.0.close().await; + } +} + +#[async_trait] +impl PipeRx for WsRx { + async fn recv(&mut self) -> TResult>> { + loop { + match self.0.next().await { + None => return Ok(None), + Some(Err(e)) => return Err(TransportError::other(e)), + Some(Ok(Message::Binary(b))) => return Ok(Some(b.to_vec())), + Some(Ok(Message::Close(_))) => return Ok(None), + // Text carries nothing in this binding; pings are answered by the library. + Some(Ok(_)) => continue, + } + } + } +} + +/// A MOQT session over an upgraded WebSocket. +pub fn session(ws: WebSocketStream, client: bool, remote: String, cfg: MuxConfig) -> MuxConnection { + let (tx, rx) = ws.split(); + MuxConnection::new(client, TransportKind::WebSocket, remote, cfg, Box::new(WsTx(tx)), Box::new(WsRx(rx))) +} + +/// Whether a `Sec-WebSocket-Protocol` header offers this draft. +pub fn offers_subprotocol(header: Option<&str>) -> bool { + header.is_some_and(|h| h.split(',').any(|p| p.trim() == subprotocol())) +} + +/// Connect to a relay's WebSocket endpoint (`ws://` or `wss://`), offering `moqt-22`. +pub async fn connect(url: &str, cfg: MuxConfig) -> TResult { + use tokio_tungstenite::tungstenite::client::IntoClientRequest; + let mut req = url.into_client_request().map_err(TransportError::other)?; + req.headers_mut().insert( + "sec-websocket-protocol", + subprotocol().parse().map_err(TransportError::other)?, + ); + let (ws, resp) = tokio_tungstenite::connect_async(req).await.map_err(TransportError::other)?; + let chosen = resp.headers().get("sec-websocket-protocol").and_then(|v| v.to_str().ok()); + if chosen != Some(subprotocol()) { + return Err(TransportError::Other("the relay did not select the moqt-22 subprotocol".into())); + } + Ok(session(ws, true, url.to_string(), cfg)) +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/webtransport.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/webtransport.rs new file mode 100644 index 000000000..da14a17c4 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/transport/webtransport.rs @@ -0,0 +1,222 @@ +//! WebTransport over HTTP/3 (`draft-ietf-webtrans-http3-16`; `draft-ietf-moq-transport-22` +//! §6.2.1) on wtransport. +//! +//! The relay's quinn endpoint offers both `moqt-22` and `h3`; a connection whose ALPN is +//! `h3` is handed to wtransport, which runs the HTTP/3 layer and the extended CONNECT. +//! The draft is negotiated by `WT-Available-Protocols` and `WT-Protocol` (webtrans-http3 +//! §3.3), both Structured Fields of String items. +//! +//! Once a WebTransport stream is open it is a QUIC stream behind a short header, so its +//! bytes are read and written on the quinn stream. Application error codes on WebTransport +//! streams are 32-bit and ride RESET_STREAM and STOP_SENDING mapped into the HTTP/3 error +//! space (webtrans-http3 §4.4); [`wt_to_h3`] and [`h3_to_wt`] are that mapping. + +use super::quic::{conn_err, READ_CHUNK}; +use super::{BoxRecv, BoxSend, Connection, RecvStream, SendStream, TResult, TransportError, TransportKind}; +use async_trait::async_trait; +use bytes::Bytes; +use wtransport::VarInt; + +/// The first HTTP/3 error code of the WebTransport application range. +const WT_FIRST: u64 = 0x52e4_a40f_a8db; +/// The last one. +const WT_LAST: u64 = 0x52e5_ac98_3162; + +/// A WebTransport application error code as an HTTP/3 error code (webtrans-http3 §4.4). +pub fn wt_to_h3(n: u32) -> u64 { + let n = n as u64; + WT_FIRST + n + n / 0x1e +} + +/// The inverse of [`wt_to_h3`]: `None` for a code outside the range or on one of its +/// reserved points. +pub fn h3_to_wt(h: u64) -> Option { + if !(WT_FIRST..=WT_LAST).contains(&h) || (h - 0x21).is_multiple_of(0x1f) { + return None; + } + let shifted = h - WT_FIRST; + u32::try_from(shifted - shifted / 0x1f).ok() +} + +fn moqt_code_to_h3(code: u64) -> quinn::VarInt { + let n = u32::try_from(code).unwrap_or(u32::MAX); + quinn::VarInt::from_u64(wt_to_h3(n)).unwrap_or(quinn::VarInt::MAX) +} + +fn h3_to_moqt(h: u64) -> u64 { + h3_to_wt(h).map_or(h, u64::from) +} + +/// The protocol a client offered in `WT-Available-Protocols` that this relay speaks: +/// `"moqt-22"`. The header is a Structured Field List; only String items count, and +/// their parameters are ignored (webtrans-http3 §3.3). +pub fn select_protocol(header: Option<&str>) -> Option<&'static str> { + let want = std::str::from_utf8(joule_moq::draft22::ALPN).ok()?; + let h = header?; + for member in h.split(',') { + let item = member.trim().split(';').next().unwrap_or("").trim(); + if let Some(s) = item.strip_prefix('"').and_then(|s| s.strip_suffix('"')) + && s == want + { + return Some(want); + } + } + None +} + +/// A WebTransport session that negotiated `moqt-22`. +pub struct WtConnection { + conn: wtransport::Connection, +} + +impl WtConnection { + pub fn new(conn: wtransport::Connection) -> Self { + WtConnection { conn } + } +} + +fn wt_conn_err(e: &wtransport::error::ConnectionError) -> TransportError { + use wtransport::error::ConnectionError as E; + match e { + E::ApplicationClosed(c) => TransportError::Closed { + code: c.code().into_inner(), + reason: String::from_utf8_lossy(c.reason()).into_owned(), + by_peer: true, + }, + E::LocallyClosed => TransportError::Closed { code: 0, reason: "closed by this endpoint".into(), by_peer: false }, + other => TransportError::Closed { code: u64::MAX, reason: other.to_string(), by_peer: true }, + } +} + +pub struct WtSend(wtransport::SendStream); +pub struct WtRecv(wtransport::RecvStream); + +#[async_trait] +impl SendStream for WtSend { + async fn write_all(&mut self, buf: &[u8]) -> TResult<()> { + match self.0.quic_stream_mut().write_all(buf).await { + Ok(()) => Ok(()), + Err(quinn::WriteError::Stopped(code)) => Err(TransportError::Stopped(h3_to_moqt(code.into_inner()))), + Err(quinn::WriteError::ConnectionLost(c)) => Err(conn_err(&c)), + Err(e) => Err(TransportError::other(e)), + } + } + fn finish(&mut self) -> TResult<()> { + self.0.quic_stream_mut().finish().map_err(TransportError::other) + } + fn reset(&mut self, code: u64) { + let _ = self.0.quic_stream_mut().reset(moqt_code_to_h3(code)); + } + fn set_priority(&mut self, priority: i32) { + self.0.set_priority(priority); + } + async fn delivered(&mut self) -> TResult> { + match self.0.quic_stream_mut().stopped().await { + Ok(None) => Ok(None), + Ok(Some(code)) => Ok(Some(h3_to_moqt(code.into_inner()))), + Err(quinn::StoppedError::ConnectionLost(c)) => Err(conn_err(&c)), + Err(e) => Err(TransportError::other(e)), + } + } +} + +#[async_trait] +impl RecvStream for WtRecv { + async fn read_chunk(&mut self) -> TResult> { + match self.0.quic_stream_mut().read_chunk(READ_CHUNK, true).await { + Ok(Some(c)) => Ok(Some(c.bytes)), + Ok(None) => Ok(None), + Err(quinn::ReadError::Reset(code)) => Err(TransportError::Reset(h3_to_moqt(code.into_inner()))), + Err(quinn::ReadError::ConnectionLost(c)) => Err(conn_err(&c)), + Err(e) => Err(TransportError::other(e)), + } + } + fn stop(&mut self, code: u64) { + let _ = self.0.quic_stream_mut().stop(moqt_code_to_h3(code)); + } +} + +#[async_trait] +impl Connection for WtConnection { + fn kind(&self) -> TransportKind { + TransportKind::WebTransport + } + fn remote(&self) -> String { + self.conn.remote_address().to_string() + } + async fn open_uni(&self) -> TResult { + let opening = self.conn.open_uni().await.map_err(|e| wt_conn_err(&e))?; + let s = opening.await.map_err(TransportError::other)?; + Ok(Box::new(WtSend(s))) + } + async fn open_bi(&self) -> TResult<(BoxSend, BoxRecv)> { + let opening = self.conn.open_bi().await.map_err(|e| wt_conn_err(&e))?; + let (s, r) = opening.await.map_err(TransportError::other)?; + Ok((Box::new(WtSend(s)), Box::new(WtRecv(r)))) + } + async fn accept_uni(&self) -> TResult { + let r = self.conn.accept_uni().await.map_err(|e| wt_conn_err(&e))?; + Ok(Box::new(WtRecv(r))) + } + async fn accept_bi(&self) -> TResult<(BoxSend, BoxRecv)> { + let (s, r) = self.conn.accept_bi().await.map_err(|e| wt_conn_err(&e))?; + Ok((Box::new(WtSend(s)), Box::new(WtRecv(r)))) + } + fn send_datagram(&self, payload: Bytes) -> TResult<()> { + self.conn.send_datagram(&payload).map_err(TransportError::other) + } + async fn recv_datagram(&self) -> TResult { + let d = self.conn.receive_datagram().await.map_err(|e| wt_conn_err(&e))?; + Ok(d.payload()) + } + fn close(&self, code: u64, reason: &[u8]) { + let code = VarInt::try_from_u64(code).unwrap_or(VarInt::MAX); + self.conn.close(code, reason); + } + async fn closed(&self) -> TransportError { + wt_conn_err(&self.conn.closed().await) + } +} + +/// A WebTransport client for tests and tools. With `cert_hash` it trusts exactly the +/// server certificate of that SHA-256 hash, under the rules a browser applies to +/// `serverCertificateHashes` (ECDSA P-256, a validity of two weeks at most); without it +/// the platform's roots. +pub async fn connect(url: &str, cert_hash: Option<[u8; 32]>, bind: std::net::SocketAddr) -> TResult { + crate::cert::install_provider(); + let b = wtransport::ClientConfig::builder().with_bind_address(bind); + let cfg = match cert_hash { + Some(h) => b.with_server_certificate_hashes([wtransport::tls::Sha256Digest::new(h)]).build(), + None => b.with_native_certs().build(), + }; + let ep = wtransport::Endpoint::client(cfg).map_err(TransportError::other)?; + let want = std::str::from_utf8(joule_moq::draft22::ALPN).map_err(TransportError::other)?; + let opts = wtransport::endpoint::ConnectOptions::builder(url).add_header("wt-available-protocols", format!("\"{want}\"")).build(); + let conn = ep.connect(opts).await.map_err(TransportError::other)?; + Ok(WtConnection::new(conn)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_error_code_mapping_round_trips() { + assert_eq!(wt_to_h3(0), WT_FIRST); + assert_eq!(wt_to_h3(u32::MAX), WT_LAST); + for n in [0u32, 1, 2, 0x1d, 0x1e, 0x1f, 0x3c, 1000, 0xffff, u32::MAX] { + assert_eq!(h3_to_wt(wt_to_h3(n)), Some(n), "{n}"); + } + assert_eq!(h3_to_wt(WT_FIRST - 1), None); + assert_eq!(h3_to_wt(0x2), None); + } + + #[test] + fn the_protocol_is_chosen_from_a_structured_list() { + assert_eq!(select_protocol(Some("\"moqt-22\"")), Some("moqt-22")); + assert_eq!(select_protocol(Some("\"moqt-18\", \"moqt-22\";q=1")), Some("moqt-22")); + assert_eq!(select_protocol(Some("moqt-22")), None, "a token, not a String item"); + assert_eq!(select_protocol(Some("\"moqt-18\"")), None); + assert_eq!(select_protocol(None), None); + } +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/src/wire.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/src/wire.rs new file mode 100644 index 000000000..4a89bcfa5 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/src/wire.rs @@ -0,0 +1,202 @@ +//! Reading MOQT from the transport's streams: framed control messages on control and +//! request streams, and the items of data streams as their bytes arrive. + +use crate::transport::{BoxRecv, BoxSend, TransportError}; +use joule_moq::draft22::data::{Limits, UniStreamDecoder}; +use joule_moq::draft22::{ControlMessage, GroupOrder, RequestKind, RequestStream, Side, StreamHeader, UniEvent, WireError}; +use std::sync::{Arc, Mutex}; + +/// Why reading or writing a stream stopped. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum StreamError { + #[error("transport: {0}")] + Transport(#[from] TransportError), + #[error("moqt: {0}")] + Wire(#[from] WireError), +} + +/// Framed control messages (§9: Type, a 16-bit Length, the body) read from a stream. +pub struct MessageReader { + recv: BoxRecv, + buf: Vec, + fin: bool, +} + +impl MessageReader { + pub fn new(recv: BoxRecv) -> Self { + MessageReader { recv, buf: Vec::new(), fin: false } + } + + /// The length of the whole message at the front of `b`, once its header is there. + fn framed_len(b: &[u8]) -> Option { + let first = *b.first()?; + let tlen = joule_moq::draft22::varint::len_from_first(first); + if b.len() < tlen + 2 { + return None; + } + let len = u16::from_be_bytes([b[tlen], b[tlen + 1]]) as usize; + Some(tlen + 2 + len) + } + + /// The next message; `None` at a FIN on a message boundary. + pub async fn next(&mut self) -> Result, StreamError> { + loop { + if let Some(n) = Self::framed_len(&self.buf).filter(|n| *n <= self.buf.len()) { + let m = ControlMessage::decode_exact(&self.buf[..n])?; + self.buf.drain(..n); + return Ok(Some(m)); + } + if self.fin { + if self.buf.is_empty() { + return Ok(None); + } + return Err(WireError::Truncated { what: "a stream ends inside a control message" }.into()); + } + match self.recv.read_chunk().await? { + Some(b) => self.buf.extend_from_slice(&b), + None => self.fin = true, + } + } + } + + pub fn stop(&mut self, code: u64) { + self.recv.stop(code); + } +} + +/// The items of a unidirectional stream, each yielded once all of it has arrived. +pub struct UniReader { + recv: BoxRecv, + dec: UniStreamDecoder, + fin: bool, + /// What the stream began with, once read. + pub header: Option, +} + +impl UniReader { + pub fn new(recv: BoxRecv, max_payload: u64) -> Self { + let limits = Limits { max_payload, ..Limits::default() }; + UniReader { recv, dec: UniStreamDecoder::new(GroupOrder::Ascending, limits), fin: false, header: None } + } + + /// Read up to the stream's header, and return it. + pub async fn start(&mut self) -> Result { + if let Some(h) = &self.header { + return Ok(h.clone()); + } + match self.next().await? { + Some(UniEvent::Header(h)) => Ok(h), + _ => Err(WireError::Truncated { what: "a unidirectional stream ends before its type" }.into()), + } + } + + /// The Group Order of the request a fetch stream answers. + pub fn set_group_order(&mut self, order: GroupOrder) { + self.dec.set_group_order(order); + } + + /// The next item; `None` at the FIN. + pub async fn next(&mut self) -> Result, StreamError> { + loop { + if let Some(e) = self.dec.next()? { + if let UniEvent::Header(h) = &e { + self.header = Some(h.clone()); + } + return Ok(Some(e)); + } + if self.fin { + self.dec.finish()?; + return Ok(None); + } + match self.recv.read_chunk().await? { + Some(b) => self.dec.push(&b), + None => self.fin = true, + } + } + } + + pub fn stop(&mut self, code: u64) { + self.recv.stop(code); + } +} + +/// The sending direction of a request stream, shareable between tasks: every message is +/// checked against the stream's rules before it goes ([`RequestStream::on_send`]). +#[derive(Clone)] +pub struct ReqTx { + send: Arc>, + pub state: Arc>, +} + +impl ReqTx { + pub async fn send(&self, m: &ControlMessage) -> Result<(), StreamError> { + self.state.lock().unwrap().on_send(m)?; + let b = m.encode()?; + self.send.lock().await.write_all(&b).await?; + Ok(()) + } + + /// A FIN on this direction (§6.4.2.2). + pub async fn fin(&self) { + let _ = self.send.lock().await.finish(); + } + + /// Cancel this direction (§6.4.2.3). + pub async fn reset(&self, code: u64) { + self.send.lock().await.reset(code); + } +} + +/// The receiving direction of a request stream: each message admitted by the stream's +/// rules ([`RequestStream::on_receive`]). +pub struct ReqRx { + pub reader: MessageReader, + pub state: Arc>, +} + +impl ReqRx { + pub async fn next(&mut self) -> Result, StreamError> { + let m = self.reader.next().await?; + if let Some(m) = &m { + self.state.lock().unwrap().on_receive(m)?; + } + Ok(m) + } +} + +impl std::fmt::Debug for ReqTx { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ReqTx").field("kind", &self.state.lock().unwrap().kind()).finish_non_exhaustive() + } +} + +impl std::fmt::Debug for ReqRx { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ReqRx").field("kind", &self.state.lock().unwrap().kind()).finish_non_exhaustive() + } +} + +/// A request stream this endpoint opened, its request already sent. +pub fn requester(send: BoxSend, recv: BoxRecv, kind: RequestKind, my_limit: u64, peer_limit: u64) -> (ReqTx, ReqRx) { + let state = Arc::new(Mutex::new(RequestStream::new(kind, Side::Requester, my_limit, peer_limit))); + (ReqTx { send: Arc::new(tokio::sync::Mutex::new(send)), state: state.clone() }, ReqRx { reader: MessageReader::new(recv), state }) +} + +/// A request stream the peer opened with `first`. +pub fn responder(send: BoxSend, reader: MessageReader, first: &ControlMessage, my_limit: u64, peer_limit: u64) -> Result<(ReqTx, ReqRx), WireError> { + let state = Arc::new(Mutex::new(RequestStream::opened_by_peer(first, my_limit, peer_limit)?)); + Ok((ReqTx { send: Arc::new(tokio::sync::Mutex::new(send)), state: state.clone() }, ReqRx { reader, state })) +} + +/// Open a request stream on `conn` with `first` (§6.3). +pub async fn open_request( + conn: &dyn crate::transport::Connection, + first: &ControlMessage, + my_limit: u64, + peer_limit: u64, +) -> Result<(ReqTx, ReqRx), StreamError> { + let kind = first.opens_request().ok_or(WireError::UnexpectedMessage("not a request"))?; + let (mut send, recv) = conn.open_bi().await?; + send.write_all(&first.encode()?).await?; + Ok(requester(send, recv, kind, my_limit, peer_limit)) +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/auth_tokens.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/auth_tokens.rs new file mode 100644 index 000000000..e2e0222bc --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/auth_tokens.rs @@ -0,0 +1,62 @@ +//! Authorization tokens and their alias cache (`draft-ietf-moq-transport-22` §8.9, §9.1.3, +//! §9.1.4), over the WebSocket fallback: a token registered in SETUP authorizes the +//! session; one registered in a request is used by alias in the next; a request without a +//! token is refused; a reference to an alias never registered closes the session. + +mod common; + +use common::*; +use joule_moq::draft22::{Parameter, Parameters, RequestErrorCode, SessionError, Token}; +use joule_moq_relay::client::ClientOptions; +use joule_moq_relay::config::Grant; +use joule_moq_relay::transport::TransportError; +use std::time::Duration; +use tokio::time::timeout; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn tokens_by_value_and_by_alias() { + timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time"); +} + +async fn run() { + // The token is made here; only its hash is in the relay's configuration. + let token: Vec = (0..24u8).map(|i| i.wrapping_mul(37).wrapping_add(11)).collect(); + let hash = hex::encode(joule_moq_relay::auth::sha256(&token)); + let server = relay(|c| { + c.auth.required = true; + c.auth.grants = vec![Grant { prefix: vec!["live".into()], token_sha256: hash, publish: true, subscribe: true }]; + c.limits.max_auth_token_cache_size = 1024; + }) + .await; + let t = Tracks::new(ns(&["live", "t"]), "cam", 10, 5, 128); + + // A publisher whose SETUP registers the token (§9.1.4). + let setup_token = Token::Register { alias: 1, token_type: 0, value: token.clone() }; + let (p, _) = connect(&server, Via::WebSocket, ClientOptions { tokens: vec![setup_token], ..ClientOptions::default() }).await; + let _pub = publish_tracks(&p, &t, None).await; + + // A subscriber with no token is refused. + let (s, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let refused = s.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.expect_err("no token"); + assert!(refused.contains(&RequestErrorCode::UNAUTHORIZED.to_string()), "{refused}"); + // It registers the token in one request, then uses it by alias in the next (§8.9). + let reg = Parameters::new().with(Parameter::AuthorizationToken(Token::Register { alias: 7, token_type: 0, value: token.clone() })); + s.subscribe(t.ns.clone(), t.media.clone(), reg).await.expect("authorized by the token it registers"); + let by_alias = Parameters::new().with(Parameter::AuthorizationToken(Token::UseAlias { alias: 7 })); + s.subscribe(t.ns.clone(), t.receipts.clone(), by_alias).await.expect("authorized by alias"); + // A wrong token is refused even when it is well formed. + let wrong = Parameters::new().with(Parameter::AuthorizationToken(Token::UseValue { token_type: 0, value: b"not-it".to_vec() })); + assert!(s.subscribe(t.ns.clone(), t.media.clone(), wrong).await.is_err()); + + // A reference to an alias never registered closes the session (§8.9). + let (x, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let unknown = Parameters::new().with(Parameter::AuthorizationToken(Token::UseAlias { alias: 99 })); + assert!(x.subscribe(t.ns.clone(), t.media.clone(), unknown).await.is_err()); + match timeout(STEP, x.conn.closed()).await.expect("closed in time") { + TransportError::Closed { code, .. } => assert_eq!(code, SessionError::UNKNOWN_AUTH_TOKEN_ALIAS.0), + other => panic!("{other:?}"), + } + s.close(0, "done"); + p.close(0, "done"); + server.shutdown().await; +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/common/mod.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/common/mod.rs new file mode 100644 index 000000000..0b0b340ee --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/common/mod.rs @@ -0,0 +1,444 @@ +//! What the relay's integration tests share: a relay on loopback, clients over each +//! transport, publishers that sign their groups, and subscribers that read and verify. +//! +//! Every network address is loopback, and every wait here and in the tests has a hard +//! bound, so no loop can outlive its test. + +#![allow(dead_code)] + +use ed25519_dalek::SigningKey; +use joule_moq::carriage::{object_properties, ObjectExtras, ReceiptBundle}; +use joule_moq::draft22::data::{ObjectStatus as WireStatus, SubgroupHeader, SubgroupId, SubgroupObject}; +use joule_moq::draft22::kvp::{prop, KeyValuePair, Properties}; +use joule_moq::draft22::{ControlMessage, Parameters, StreamHeader, SubscribeOk, UniEvent}; +use joule_moq::messages::ObjectHeader; +use joule_moq::receipt::{ObjectReceipt, ReceiptBuilder, ReceiptVerifier, VerifiedGroup}; +use joule_moq::{GroupId, ObjectId, ObjectStatus, TrackAlias, TrackName, TrackNamespace}; +use joule_moq_relay::client::{Client, ClientOptions}; +use joule_moq_relay::transport::mux::MuxConfig; +use joule_moq_relay::transport::{quic, webtransport, websocket, TransportError}; +use joule_moq_relay::wire::{ReqRx, ReqTx, StreamError, UniReader}; +use joule_moq_relay::{RelayConfig, RelayServer}; +use std::collections::{BTreeMap, HashMap}; +use std::sync::Arc; +use std::time::Duration; +use tokio::time::{timeout, Instant}; + +/// The bound on any single wait in a test. +pub const STEP: Duration = Duration::from_secs(20); + +pub fn loopback() -> std::net::SocketAddr { + "127.0.0.1:0".parse().unwrap() +} + +/// A relay on loopback ports, the configuration adjusted by `f`. +pub async fn relay(f: impl FnOnce(&mut RelayConfig)) -> RelayServer { + let mut c = RelayConfig::loopback(); + f(&mut c); + RelayServer::start(c).await.expect("the relay starts") +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Via { + Quic, + WebTransport, + WebSocket, +} + +/// A client of `server` over `via`. A QUIC client keeps its endpoint, returned with it. +pub async fn connect(server: &RelayServer, via: Via, opts: ClientOptions) -> (Client, Option) { + connect_with(server, via, opts, None).await +} + +pub async fn connect_with(server: &RelayServer, via: Via, opts: ClientOptions, transport: Option) -> (Client, Option) { + let quic_addr = server.quic_addr.expect("a UDP endpoint"); + let path = server.relay().config().path.clone(); + match via { + Via::Quic => { + let ep = quic::client_endpoint(loopback(), vec![server.identity.chain[0].clone()], vec![joule_moq::draft22::ALPN.to_vec()], transport).unwrap(); + let conn = timeout(STEP, quic::connect(&ep, quic_addr, "localhost")).await.expect("in time").expect("a QUIC session"); + let c = Client::connect(Arc::new(conn), opts).await.expect("SETUP over QUIC"); + (c, Some(ep)) + } + Via::WebTransport => { + let url = format!("https://127.0.0.1:{}{}", quic_addr.port(), path); + let conn = timeout(STEP, webtransport::connect(&url, Some(server.identity.leaf_sha256), loopback())).await.expect("in time").expect("a WebTransport session"); + let c = Client::connect(Arc::new(conn), opts).await.expect("SETUP over WebTransport"); + (c, None) + } + Via::WebSocket => { + let url = format!("ws://{}{}", server.websocket_addr.expect("a WebSocket listener"), path); + let conn = timeout(STEP, websocket::connect(&url, MuxConfig::default())).await.expect("in time").expect("a WebSocket session"); + let c = Client::connect(Arc::new(conn), opts).await.expect("SETUP over the WebSocket binding"); + (c, None) + } + } +} + +pub fn ns(fields: &[&str]) -> TrackNamespace { + TrackNamespace::from_strs(fields.iter().copied()) +} + +pub fn name(s: &str) -> TrackName { + TrackName::from_str(s) +} + +/// Track properties: a default publisher priority, and an OBJECT_DELIVERY_TIMEOUT. +pub fn track_props(priority: u8, object_timeout_ms: Option) -> Properties { + let mut p = Properties::new(); + p.push(KeyValuePair::varint(prop::DEFAULT_PUBLISHER_PRIORITY, priority as u64)); + if let Some(t) = object_timeout_ms { + p.push(KeyValuePair::varint(prop::OBJECT_DELIVERY_TIMEOUT, t)); + } + p +} + +/// One group of a media track, one object, and its receipts as group `g` of the receipts +/// track carries them (WAI moq-streaming-format §4.1). +pub fn signed_group(key: &SigningKey, receipt_alias: u64, g: u64, payload: &[u8]) -> (ObjectReceipt, ReceiptBundle) { + let hash = ObjectReceipt::hash_payload(payload); + let mut b = ReceiptBuilder::new(TrackAlias(receipt_alias), GroupId(g)); + b.add_labelled_object(hash, 0, "test-publisher", "2026-10-17T00:00:00Z", vec![0]).unwrap(); + let receipt = ObjectReceipt { + track_alias: TrackAlias(receipt_alias), + group_id: GroupId(g), + object_id: ObjectId(0), + content_hash: hash, + joules_micro: 0, + origin: "test-publisher".into(), + merkle_proof: b.proof_for(0).unwrap(), + energy_class: Some(vec![0]), + }; + let group = b.finalize_labelled(key, "test-publisher", "2026-10-17T00:00:01Z", None).unwrap(); + (receipt.clone(), ReceiptBundle { group, objects: vec![receipt] }) +} + +/// A media track and its receipts track, as one publisher sends them. +#[derive(Clone)] +pub struct Tracks { + pub ns: TrackNamespace, + pub media: TrackName, + pub receipts: TrackName, + pub media_alias: u64, + pub receipts_alias: u64, + pub receipt_alias: u64, + pub priority: u8, +} + +impl Tracks { + pub fn new(ns: TrackNamespace, media: &str, media_alias: u64, receipt_alias: u64, priority: u8) -> Self { + Tracks { ns, media: name(media), receipts: name(&format!("{media}/receipts")), media_alias, receipts_alias: media_alias + 1, receipt_alias, priority } + } +} + +/// Publish group `g` of `t`: the receipts first (no later than the group's first object, +/// moq-streaming-format §4.1), then the object, which carries its receipt as a property. +pub async fn publish_group(c: &Client, key: &SigningKey, t: &Tracks, g: u64, payload: &[u8]) -> ObjectReceipt { + let (receipt, bundle) = signed_group(key, t.receipt_alias, g, payload); + let rh = SubgroupHeader { track_alias: t.receipts_alias, group_id: g, subgroup_id: SubgroupId::Zero, publisher_priority: Some(t.priority), properties: false, end_of_group: true, first_object: true }; + c.send_subgroup(rh, &[SubgroupObject::normal(0, bundle.to_payload().unwrap())]).await.expect("the receipts"); + let header = ObjectHeader { + track_alias: TrackAlias(t.media_alias), + group_id: GroupId(g), + object_id: ObjectId(0), + publisher_priority: t.priority, + status: ObjectStatus::Normal, + payload_len: payload.len() as u64, + mrl_layer: 0, + mrl_total_layers: 0, + wai_envelope: false, + }; + let props = object_properties(&header, &ObjectExtras { receipt: Some(receipt.clone()), wai_receipt: None, delivery_timeout_ms: None }).unwrap(); + let mh = SubgroupHeader { track_alias: t.media_alias, group_id: g, subgroup_id: SubgroupId::Zero, publisher_priority: Some(t.priority), properties: true, end_of_group: true, first_object: true }; + c.send_subgroup(mh, &[SubgroupObject { object_id: 0, properties: props, status: WireStatus::Normal, payload: payload.to_vec() }]).await.expect("the object"); + receipt +} + +/// PUBLISH a media track and its receipts track (§9.8). +pub async fn publish_tracks(c: &Client, t: &Tracks, object_timeout_ms: Option) -> Vec<(ReqTx, ReqRx)> { + let a = c.publish(t.ns.clone(), t.media.clone(), t.media_alias, Parameters::new(), track_props(t.priority, object_timeout_ms)).await.expect("PUBLISH the media track"); + let b = c.publish(t.ns.clone(), t.receipts.clone(), t.receipts_alias, Parameters::new(), track_props(t.priority, None)).await.expect("PUBLISH the receipts track"); + vec![a, b] +} + +/// Answer the relay's SUBSCRIBEs for `t`'s tracks (the namespace route, §7.6) until both +/// are Established; the streams are returned to keep the subscriptions open. +pub async fn answer_subscribes(c: &Client, t: &Tracks) -> Vec<(ReqTx, ReqRx)> { + let mut held = Vec::new(); + let mut seen = (false, false); + let deadline = Instant::now() + STEP; + while !(seen.0 && seen.1) { + let left = deadline.saturating_duration_since(Instant::now()); + let (tx, rx, first) = c.next_request(left).await.expect("the relay's SUBSCRIBE in time"); + let ControlMessage::Subscribe(r) = first else { panic!("the relay sent {}", first.name()) }; + let (alias, props) = if r.name == t.media { + seen.0 = true; + (t.media_alias, track_props(t.priority, None)) + } else { + assert_eq!(r.name, t.receipts, "the relay subscribes only to the tracks asked for"); + seen.1 = true; + (t.receipts_alias, track_props(t.priority, None)) + }; + tx.send(&ControlMessage::SubscribeOk(SubscribeOk { track_alias: alias, params: Parameters::new(), track_properties: props })).await.unwrap(); + held.push((tx, rx)); + } + held +} + +/// What a subscriber read from one data stream. +#[derive(Debug)] +pub struct Got { + pub header: SubgroupHeader, + pub objects: Vec, + /// The stream was reset, with this code. + pub reset: Option, + pub arrived: Instant, +} + +/// Read one subgroup stream to its end, each read bounded. +pub async fn read_stream(mut r: UniReader) -> Option { + let arrived = Instant::now(); + let StreamHeader::Subgroup(header) = r.header.clone()? else { return None }; + let mut objects = Vec::new(); + loop { + match timeout(STEP, r.next()).await { + Ok(Ok(Some(UniEvent::SubgroupObject(o)))) => objects.push(o), + Ok(Ok(Some(_))) => return None, + Ok(Ok(None)) => return Some(Got { header, objects, reset: None, arrived }), + Ok(Err(StreamError::Transport(TransportError::Reset(code)))) => return Some(Got { header, objects, reset: Some(code), arrived }), + Ok(Err(e)) => panic!("a data stream: {e}"), + Err(_) => panic!("a data stream stalled"), + } + } +} + +/// Read data streams until `done` holds of what was read, or `limit` passes. +pub async fn collect(c: &Client, limit: Duration, mut done: impl FnMut(&[Got]) -> bool) -> Vec { + let deadline = Instant::now() + limit; + let mut out = Vec::new(); + while !done(&out) { + let left = deadline.saturating_duration_since(Instant::now()); + if left.is_zero() { + break; + } + match c.next_data(left).await { + Some(r) => { + if let Some(g) = read_stream(r).await { + out.push(g); + } + } + None => break, + } + } + out +} + +/// Read every data stream at once, each in a task of its own, as a subscriber does that +/// reads its streams concurrently: until `done` holds of what was read and `quiet` has +/// passed with no new stream or object, or `limit` passes. Each read is bounded. +/// +/// Reading one stream at a time in the order they arrive can stall: QUIC's +/// connection-level flow control is shared by a session's streams, so the bytes of the +/// stream being waited on can wait for credit that unread later streams hold. +pub async fn collect_concurrent(c: &Client, limit: Duration, quiet: Duration, mut done: impl FnMut(&[Got]) -> bool) -> Vec { + let deadline = Instant::now() + limit; + let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::(); + let mut out = Vec::new(); + let mut readers = Vec::new(); + let mut last = Instant::now(); + loop { + let now = Instant::now(); + if now >= deadline || (done(&out) && now.duration_since(last) >= quiet) { + break; + } + let tick = deadline.min(now + quiet).saturating_duration_since(now); + tokio::select! { + s = c.next_stream(tick) => { + if let Some(mut r) = s { + last = Instant::now(); + let tx = tx.clone(); + readers.push(tokio::spawn(async move { + if let Ok(Ok(_)) = timeout(STEP, r.start()).await + && let Some(g) = read_stream(r).await + { + let _ = tx.send(g); + } + })); + } + } + g = rx.recv() => { + if let Some(g) = g { + last = Instant::now(); + out.push(g); + } + } + } + } + for r in readers { + r.abort(); + } + while let Ok(g) = rx.try_recv() { + out.push(g); + } + out +} + +/// A subscriber's view of one publisher's tracks: the downstream aliases the relay gave. +pub struct Subscribed { + pub media_alias: u64, + pub receipts_alias: u64, + pub held: Vec<(ReqTx, ReqRx)>, +} + +pub async fn subscribe_both(c: &Client, t: &Tracks, media_params: Parameters) -> Subscribed { + subscribe_both_with(c, t, media_params, Parameters::new()).await +} + +pub async fn subscribe_both_with(c: &Client, t: &Tracks, media_params: Parameters, receipts_params: Parameters) -> Subscribed { + let (m, mtx, mrx) = c.subscribe(t.ns.clone(), t.media.clone(), media_params).await.expect("SUBSCRIBE the media track"); + let (r, rtx, rrx) = c.subscribe(t.ns.clone(), t.receipts.clone(), receipts_params).await.expect("SUBSCRIBE the receipts track"); + Subscribed { media_alias: m.track_alias, receipts_alias: r.track_alias, held: vec![(mtx, mrx), (rtx, rrx)] } +} + +/// Every media object of `sub` verified against the bundles of its receipts track, under +/// the publisher's key alone. Returns (objects verified, bundles verified, the verified +/// groups by Group ID, the object receipts by Group ID). +pub fn verify(got: &[Got], sub: &Subscribed, publisher: &SigningKey) -> (usize, usize, BTreeMap, HashMap) { + let verifier = ReceiptVerifier::with_anchors(vec![publisher.verifying_key()]); + let mut groups = BTreeMap::new(); + let mut receipts = HashMap::new(); + for g in got.iter().filter(|g| g.header.track_alias == sub.receipts_alias && g.reset.is_none()) { + let b = ReceiptBundle::from_payload(&g.objects[0].payload).expect("a bundle"); + let vg = verifier.verify_group(&b.group).expect("the group receipt verifies under the publisher's key"); + assert_eq!(b.group.group_id, GroupId(g.header.group_id), "the receipt names its MOQT group"); + groups.insert(g.header.group_id, vg); + receipts.insert(g.header.group_id, b.objects[0].clone()); + } + let mut objects = 0; + for g in got.iter().filter(|g| g.header.track_alias == sub.media_alias && g.reset.is_none()) { + for o in &g.objects { + let vg = groups.get(&g.header.group_id).expect("the receipts of every delivered group arrived"); + let r = &receipts[&g.header.group_id]; + verifier.verify_payload(&o.payload, r, vg).expect("the payload verifies against its signed root"); + objects += 1; + } + } + (objects, groups.len(), groups, receipts) +} + +pub fn payload(tag: &str, g: u64, len: usize) -> Vec { + let mut v = format!("{tag}:{g}:").into_bytes(); + v.resize(len.max(v.len()), (g % 251) as u8); + v +} + +/// Wait, bounded, until `f` holds. +pub async fn until(limit: Duration, mut f: impl FnMut() -> bool) -> bool { + let deadline = Instant::now() + limit; + while !f() { + if Instant::now() >= deadline { + return false; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + true +} + +/// The scenario every transport runs: two publishers, one reached by the namespace route +/// (PUBLISH_NAMESPACE, the relay's SUBSCRIBE) and one by PUBLISH, and three subscribers +/// of both, all over `via`. Every object reaches every subscriber and verifies against its +/// publisher's signed receipts, which the relay also checks at the door; one subscriber +/// discovers the first namespace by SUBSCRIBE_NAMESPACE; every session's summary is signed. +pub async fn two_publishers_three_subscribers(via: Via) { + let key_a = SigningKey::from_bytes(&[0xa1; 32]); + let key_b = SigningKey::from_bytes(&[0xb2; 32]); + let (ka, kb) = (hex::encode(key_a.verifying_key().to_bytes()), hex::encode(key_b.verifying_key().to_bytes())); + let server = relay(|c| { + c.receipts.door = vec![joule_moq_relay::config::DoorRule { prefix: vec!["live".into()], trust: vec![ka, kb], hold_ms: 5_000 }]; + }) + .await; + const GROUPS: u64 = 12; + let ta = Tracks::new(ns(&["live", "a"]), "cam", 10, 8, 64); + let tb = Tracks::new(ns(&["live", "b"]), "cam", 20, 9, 96); + + let (pa, _ea) = connect(&server, via, ClientOptions::default()).await; + let (pb, _eb) = connect(&server, via, ClientOptions::default()).await; + let _ns_a = pa.publish_namespace(ta.ns.clone(), Parameters::new()).await.expect("PUBLISH_NAMESPACE"); + let _pub_b = publish_tracks(&pb, &tb, None).await; + + // One subscriber discovers live/a by prefix (§4.2). + let (s0, _e0) = connect(&server, via, ClientOptions::default()).await; + let (_nstx, mut nsrx) = s0.subscribe_namespace(ns(&["live"])).await.expect("SUBSCRIBE_NAMESPACE"); + match timeout(STEP, nsrx.next()).await.expect("NAMESPACE in time").expect("a message") { + Some(ControlMessage::Namespace(suffix)) => assert_eq!(suffix, ns(&["a"]), "NAMESPACE carries the suffix after the prefix"), + other => panic!("expected NAMESPACE, got {other:?}"), + } + + let mut subs = vec![s0]; + let mut eps = Vec::new(); + for _ in 0..2 { + let (s, e) = connect(&server, via, ClientOptions::default()).await; + subs.push(s); + eps.push(e); + } + // The first SUBSCRIBE to live/a makes the relay subscribe upstream; publisher A answers. + let answering = { + let ta = ta.clone(); + let pa_ref = &pa; + async move { answer_subscribes(pa_ref, &ta).await } + }; + let s_first = &subs[0]; + let (held_a, first_sub) = tokio::join!(answering, subscribe_both(s_first, &ta, Parameters::new())); + let mut views: Vec<(Subscribed, Subscribed)> = vec![(first_sub, subscribe_both(&subs[0], &tb, Parameters::new()).await)]; + for s in &subs[1..] { + views.push((subscribe_both(s, &ta, Parameters::new()).await, subscribe_both(s, &tb, Parameters::new()).await)); + } + + // Both publish, concurrently. + let pub_a = async { + for g in 0..GROUPS { + publish_group(&pa, &key_a, &ta, g, &payload("a", g, 900)).await; + } + }; + let pub_b = async { + for g in 0..GROUPS { + publish_group(&pb, &key_b, &tb, g, &payload("b", g, 1400)).await; + } + }; + let reads = futures::future::join_all(subs.iter().map(|s| collect(s, STEP * 2, |got| got.len() as u64 >= 4 * GROUPS))); + let ((), (), got) = tokio::join!(pub_a, pub_b, reads); + + for (i, (g, (va, vb))) in got.iter().zip(&views).enumerate() { + let (oa, ba, _, _) = verify(g, va, &key_a); + let (ob, bb, _, _) = verify(g, vb, &key_b); + assert_eq!((oa, ba, ob, bb), (GROUPS as usize, GROUPS as usize, GROUPS as usize, GROUPS as usize), "subscriber {i}: every object and every receipt of both publishers"); + for x in g.iter().filter(|x| x.header.track_alias == va.media_alias) { + assert_eq!(x.objects[0].payload, payload("a", x.header.group_id, 900)); + } + } + assert_eq!(server.relay().stats().refused_at_door, 0); + drop(held_a); + + // Every session ends with a signed summary. + for s in &subs { + s.close(0, "done"); + } + pa.close(0, "done"); + pb.close(0, "done"); + assert!(until(STEP, || server.relay().summaries().len() >= 5).await, "five summaries"); + for sum in server.relay().summaries() { + sum.verify().expect("the summary verifies under the relay's key"); + assert_eq!(sum.summary.relay_energy.acquisition, "unmetered"); + assert_eq!(sum.summary.relay_energy.joules_micro, 0, "0 is the unmetered marker"); + assert_eq!(sum.summary.transport, match via { Via::Quic => "quic", Via::WebTransport => "webtransport", Via::WebSocket => "websocket" }); + } + let as_subscriber: Vec<_> = server.relay().summaries().into_iter().filter(|s| s.summary.tally.subscriber.objects_forwarded > 0).collect(); + assert_eq!(as_subscriber.len(), 3); + for s in &as_subscriber { + assert_eq!(s.summary.tally.subscriber.objects_forwarded, 4 * GROUPS, "media and receipts of both publishers"); + assert_eq!(s.summary.tally.subscriber.dropped_delivery_timeout, 0); + let signed = s.summary.tally.publisher_signed.get("unmetered").expect("the door verified the media objects' receipts"); + assert_eq!(signed.objects, 2 * GROUPS, "the publishers' signed figures, unmetered, for the media objects"); + } + server.shutdown().await; +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/goaway.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/goaway.rs new file mode 100644 index 000000000..df12f6eda --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/goaway.rs @@ -0,0 +1,77 @@ +//! GOAWAY on shutdown (`draft-ietf-moq-transport-22` §6.6.1, §9.2): every session is told, +//! with the New Session URI and the Timeout; new requests are refused with GOING_AWAY; a +//! client that leaves is let go, and one that stays is closed with GOAWAY_TIMEOUT. + +mod common; + +use common::*; +use joule_moq::draft22::{Parameters, RequestErrorCode, SessionError}; +use joule_moq_relay::client::{Client, ClientOptions}; +use joule_moq_relay::transport::TransportError; +use std::time::Duration; +use tokio::time::timeout; + +const URI: &str = "moqt://127.0.0.1:4999/moq"; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn goaway_on_shutdown() { + timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time"); +} + +async fn goaway_of(c: &Client) -> joule_moq::draft22::Goaway { + let mut w = c.goaway(); + timeout(STEP, async { + loop { + if let Some(g) = w.borrow_and_update().clone() { + return g; + } + w.changed().await.expect("the session lives until its GOAWAY"); + } + }) + .await + .expect("a GOAWAY in time") +} + +async fn run() { + let server = relay(|c| { + c.limits.goaway_timeout_ms = 1500; + c.goaway_uri = URI.into(); + }) + .await; + let t = Tracks::new(ns(&["live", "g"]), "cam", 10, 4, 128); + let (p, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await; + let _pub = publish_tracks(&p, &t, None).await; + let (a, _ea) = connect(&server, Via::Quic, ClientOptions::default()).await; + let (b, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let _sa = a.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.unwrap(); + let _sb = b.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.unwrap(); + + let relay = server.relay().clone(); + let shutdown = tokio::spawn(async move { server.shutdown().await }); + + for c in [&p, &a, &b] { + let g = goaway_of(c).await; + assert_eq!(g.new_session_uri, URI.as_bytes(), "the New Session URI"); + assert_eq!(g.timeout_ms, 1500, "the Timeout the relay will wait"); + } + // A new request after GOAWAY is refused (§9.2: "MAY reject new requests"). + let refused = b.subscribe(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.expect_err("a new SUBSCRIBE after GOAWAY"); + assert!(refused.contains(&RequestErrorCode::GOING_AWAY.to_string()), "{refused}"); + + // The publisher and one subscriber migrate and close with NO_ERROR; the other stays. + a.close(SessionError::NO_ERROR.0, "migrated"); + p.close(SessionError::NO_ERROR.0, "migrated"); + match timeout(STEP, b.conn.closed()).await.expect("closed in time") { + TransportError::Closed { code, by_peer, .. } => { + assert_eq!(code, SessionError::GOAWAY_TIMEOUT.0, "the relay closes a session that stayed with GOAWAY_TIMEOUT"); + assert!(by_peer); + } + other => panic!("{other:?}"), + } + let closed = timeout(STEP, shutdown).await.expect("the shutdown ends").unwrap(); + assert_eq!(closed, 1, "one session outlived the GOAWAY timeout"); + assert!(until(STEP, || relay.summaries().len() == 3).await); + let codes: Vec> = relay.summaries().iter().map(|s| s.summary.close_code).collect(); + assert_eq!(codes.iter().filter(|c| **c == Some(SessionError::GOAWAY_TIMEOUT.0)).count(), 1, "{codes:?}"); + assert_eq!(codes.iter().filter(|c| **c == Some(SessionError::NO_ERROR.0)).count(), 2, "{codes:?}"); +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/late_joiner.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/late_joiner.rs new file mode 100644 index 000000000..1b96dd95c --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/late_joiner.rs @@ -0,0 +1,210 @@ +//! A late joiner served from the relay's cache (`draft-ietf-moq-transport-22` §3.2, §3.4, +//! §7.1), and a FETCH the relay forwards upstream when its cache holds nothing (§3.2.3). + +mod common; + +use common::*; +use joule_moq::carriage::ReceiptBundle; +use joule_moq::draft22::data::{FetchCodec, FetchEntry, FetchHeader, FetchObject}; +use joule_moq::draft22::{ + ControlMessage, FetchOk, GroupOrder, Location, LocationFilter, Parameter, Parameters, RequestErrorCode, StreamHeader, TrackRequest, UniEvent, +}; +use joule_moq::receipt::ReceiptVerifier; +use joule_moq_relay::client::{Client, ClientOptions}; +use std::time::Duration; +use tokio::time::timeout; + +const GROUPS: u64 = 10; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_late_joiner_is_served_from_the_cache() { + timeout(Duration::from_secs(120), from_cache()).await.expect("the test finishes in time"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_group_that_arrives_after_a_later_one_is_still_served_from_the_cache() { + timeout(Duration::from_secs(120), reordered()).await.expect("the test finishes in time"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_fetch_the_cache_cannot_serve_goes_upstream() { + timeout(Duration::from_secs(120), from_upstream()).await.expect("the test finishes in time"); +} + +fn objects(entries: &[FetchEntry]) -> Vec<&FetchObject> { + entries + .iter() + .filter_map(|e| match e { + FetchEntry::Object(o) => Some(o), + _ => None, + }) + .collect() +} + +async fn from_cache() { + let server = relay(|_| {}).await; + let key = ed25519_dalek::SigningKey::from_bytes(&[0x1a; 32]); + let t = Tracks::new(ns(&["vod", "show"]), "ep1", 10, 3, 128); + let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await; + let _held = publish_tracks(&publisher, &t, None).await; + for g in 0..GROUPS { + publish_group(&publisher, &key, &t, g, &payload("ep1", g, 2000)).await; + } + // The cache, not the count of objects received, says when it holds the track: an + // object counted on arrival may not be cached yet, and streams finish out of order. + let relay = server.relay().clone(); + let (mk, rk) = ((t.ns.clone(), t.media.clone()), (t.ns.clone(), t.receipts.clone())); + assert!( + until(STEP, || relay.largest(&mk) == Some(Location::new(GROUPS - 1, 0)) && relay.cache_covers(&mk, Location::new(0, 0)) && relay.cache_covers(&rk, Location::new(0, 0))).await, + "the cache holds the track and its receipts from group 0" + ); + + // The late joiner comes over the WebSocket fallback. + let (late, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + + // TRACK_STATUS names the Largest Object (§9.13). + let m = ControlMessage::TrackStatus(TrackRequest { request_id: late.next_request_id(), namespace: t.ns.clone(), name: t.media.clone(), params: Parameters::new() }); + let (_tx, mut rx) = late.request(&m).await.unwrap(); + match Client::response(&mut rx).await.unwrap() { + ControlMessage::RequestOk(ok) => assert_eq!(ok.params.largest_object(), Some(Location::new(GROUPS - 1, 0))), + other => panic!("{other:?}"), + } + + // The whole track and its receipts, by FETCH, from the cache. + let (ok, media) = late.fetch(t.ns.clone(), t.media.clone(), Parameters::new()).await.expect("FETCH the media track"); + assert_eq!(ok.end_location, Location::new(GROUPS - 1, 0)); + let (_, rec) = late.fetch(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.expect("FETCH the receipts track"); + let media = objects(&media); + let rec = objects(&rec); + assert_eq!((media.len(), rec.len()), (GROUPS as usize, GROUPS as usize)); + let verifier = ReceiptVerifier::with_anchors(vec![key.verifying_key()]); + for (m, r) in media.iter().zip(&rec) { + assert_eq!((m.group_id, r.group_id), (r.group_id, m.group_id)); + let b = ReceiptBundle::from_payload(&r.payload).unwrap(); + let vg = verifier.verify_group(&b.group).expect("a receipt from the cache verifies"); + verifier.verify_payload(&m.payload, &b.objects[0], &vg).expect("an object from the cache verifies"); + assert_eq!(m.payload, payload("ep1", m.group_id, 2000)); + assert_eq!(m.publisher_priority, 128); + } + assert_eq!(media.iter().map(|o| o.group_id).collect::>(), (0..GROUPS).collect::>(), "ascending by default (§9.20.8)"); + + // A range, groups descending (§3.2.1). + let range = Parameters::new() + .with(Parameter::LocationFilter(LocationFilter::AbsoluteRange { start: Location::new(3, 0), end_group_delta: 4, end_object: 0 })) + .with(Parameter::GroupOrder(GroupOrder::Descending)); + let (ok, part) = late.fetch(t.ns.clone(), t.media.clone(), range).await.expect("FETCH a range"); + assert_eq!(ok.end_location, Location::new(7, 0)); + assert_eq!(objects(&part).iter().map(|o| o.group_id).collect::>(), vec![7, 6, 5, 4, 3]); + + // A fetch that starts after the Largest Object is refused (§3.2: INVALID_RANGE). + let after = Parameters::new().with(Parameter::LocationFilter(LocationFilter::AbsoluteStart { start: Location::new(GROUPS + 5, 0) })); + let refused = late.fetch(t.ns.clone(), t.media.clone(), after).await.expect_err("a range past the Largest Object"); + assert!(refused.contains(&RequestErrorCode::INVALID_RANGE.to_string()), "{refused}"); + + // Joining at a past group: SUBSCRIBE from the Next Object with a fill of the last three + // groups (§3.5), delivered on a fill fetch stream named by the SUBSCRIBE's Request ID. + let fill = Parameters::new().with(Parameter::LocationFilter(LocationFilter::RelativeStart { start_group: 3 })); + let id_before = late.next_request_id(); + let params = Parameters::new().with(Parameter::LocationFilter(LocationFilter::NextObject)).with(Parameter::FillParameters(fill)); + let (ok, _stx, _srx) = late.subscribe(t.ns.clone(), t.media.clone(), params).await.expect("SUBSCRIBE with a fill"); + assert_eq!(ok.params.largest_object(), Some(Location::new(GROUPS - 1, 0))); + let subscribe_id = id_before + 2; + let mut r = late.next_data(STEP).await.expect("the fill fetch stream"); + match r.header.clone() { + Some(StreamHeader::Fetch(FetchHeader { request_id })) => assert_eq!(request_id, subscribe_id), + other => panic!("expected a fill fetch stream, got {other:?}"), + } + let mut filled = Vec::new(); + while let Some(UniEvent::FetchEntry(e)) = timeout(STEP, r.next()).await.expect("in time").expect("a fill entry") { + filled.push(e); + } + assert_eq!(objects(&filled).iter().map(|o| o.group_id).collect::>(), vec![GROUPS - 3, GROUPS - 2, GROUPS - 1]); + + // All of it from the cache: the publisher was asked for nothing. + assert!(relay.stats().fetch_served_from_cache >= 4); + assert_eq!(relay.stats().fetch_served_upstream, 0); + assert!(publisher.next_request(Duration::from_millis(300)).await.is_none(), "the relay sent the publisher no request"); + late.close(0, "done"); + publisher.close(0, "done"); + server.shutdown().await; +} + +/// Group 1 reaches the relay whole before group 0 does, as streams finish out of order on a +/// loaded host. The cache still serves a FETCH from group 0: coverage is the run of whole +/// groups, not wherever the first object kept happened to fall. +async fn reordered() { + let server = relay(|_| {}).await; + let key = ed25519_dalek::SigningKey::from_bytes(&[0x2b; 32]); + let t = Tracks::new(ns(&["vod", "reorder"]), "ep", 10, 7, 128); + let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await; + let _held = publish_tracks(&publisher, &t, None).await; + let relay = server.relay().clone(); + let (mk, rk) = ((t.ns.clone(), t.media.clone()), (t.ns.clone(), t.receipts.clone())); + + // Group 1 first: its receipts stream and its object's stream, each to its FIN. + publish_group(&publisher, &key, &t, 1, &payload("ep", 1, 1500)).await; + assert!(until(STEP, || relay.largest(&mk) == Some(Location::new(1, 0))).await, "the relay has group 1"); + // Then group 0, then the rest in order. + for g in [0, 2, 3, 4, 5] { + publish_group(&publisher, &key, &t, g, &payload("ep", g, 1500)).await; + } + assert!( + until(STEP, || relay.largest(&mk) == Some(Location::new(5, 0)) && relay.cache_covers(&mk, Location::new(0, 0)) && relay.cache_covers(&rk, Location::new(0, 0))).await, + "the cache covers the track from group 0, though group 0 came after group 1" + ); + + let (late, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let (ok, media) = late.fetch(t.ns.clone(), t.media.clone(), Parameters::new()).await.expect("FETCH from group 0"); + assert_eq!(ok.end_location, Location::new(5, 0)); + let (_, rec) = late.fetch(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.expect("FETCH its receipts"); + let (media, rec) = (objects(&media), objects(&rec)); + assert_eq!(media.iter().map(|o| o.group_id).collect::>(), (0..=5).collect::>()); + let verifier = ReceiptVerifier::with_anchors(vec![key.verifying_key()]); + for (m, r) in media.iter().zip(&rec) { + let b = ReceiptBundle::from_payload(&r.payload).unwrap(); + let vg = verifier.verify_group(&b.group).unwrap(); + verifier.verify_payload(&m.payload, &b.objects[0], &vg).expect("each object verifies"); + } + assert_eq!(relay.stats().fetch_served_upstream, 0, "nothing went upstream"); + assert!(relay.stats().fetch_served_from_cache >= 2); + assert!(publisher.next_request(Duration::from_millis(300)).await.is_none(), "the relay sent the publisher no request"); + late.close(0, "done"); + publisher.close(0, "done"); + server.shutdown().await; +} + +async fn from_upstream() { + let server = relay(|_| {}).await; + let space = ns(&["archive"]); + let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await; + let _ns = publisher.publish_namespace(space.clone(), Parameters::new()).await.unwrap(); + let (late, _el) = connect(&server, Via::Quic, ClientOptions::default()).await; + + let asking = late.fetch(space.clone(), name("old"), Parameters::new()); + let answering = async { + let (tx, _rx, first) = publisher.next_request(STEP).await.expect("the relay's upstream FETCH"); + let ControlMessage::Fetch(r) = first else { panic!("expected FETCH, got {}", first.name()) }; + assert_eq!(r.name, name("old")); + let end = Location::new(2, 0); + tx.send(&ControlMessage::FetchOk(FetchOk { end_of_track: true, end_location: end, params: Parameters::new(), track_properties: Default::default() })).await.unwrap(); + let mut b = Vec::new(); + FetchHeader { request_id: r.request_id }.encode(&mut b); + let mut codec = FetchCodec::new(r.params.group_order().unwrap_or(GroupOrder::Ascending)); + for g in 0..=2u64 { + codec.encode(&FetchEntry::Object(FetchObject { group_id: g, object_id: 0, subgroup_id: Some(0), publisher_priority: 7, properties: None, payload: payload("old", g, 300) }), &mut b).unwrap(); + } + let mut s = publisher.conn.open_uni().await.unwrap(); + s.write_all(&b).await.unwrap(); + s.finish().unwrap(); + tx + }; + let (got, _tx) = tokio::join!(asking, answering); + let (ok, entries) = got.expect("the FETCH answered through the relay"); + assert!(ok.end_of_track); + let objs = objects(&entries); + assert_eq!(objs.iter().map(|o| (o.group_id, o.payload.clone())).collect::>(), (0..=2u64).map(|g| (g, payload("old", g, 300))).collect::>()); + assert_eq!(server.relay().stats().fetch_served_upstream, 1); + late.close(0, "done"); + publisher.close(0, "done"); + server.shutdown().await; +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/priority_fan_out.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/priority_fan_out.rs new file mode 100644 index 000000000..af1ea3c6e --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/priority_fan_out.rs @@ -0,0 +1,146 @@ +//! Fan-out order under load (`draft-ietf-moq-transport-22` §5.1.2): two publishers send +//! four tracks of different publisher priorities to three subscribers, each subscribing +//! with its own subscriber priorities and group orders, while the subscribers' transports +//! are held shut. When they open, every object reaches every subscriber in the order +//! §5.1.2 gives: subscriber priority, then publisher priority, then group order. +//! +//! The sessions run over the in-process multiplexer, the WebSocket binding's own, with a +//! gate on what the relay sends each subscriber; the relay writes one object at a time. +//! Each loop here is bounded by a hard timeout. + +mod common; + +use common::*; +use joule_moq::draft22::data::{SubgroupHeader, SubgroupId, SubgroupObject}; +use joule_moq::draft22::{GroupOrder, Parameter, Parameters}; +use joule_moq_relay::client::{Client, ClientOptions}; +use joule_moq_relay::transport::mux::{memory_pair, Gate, MuxConfig}; +use joule_moq_relay::{Relay, RelayConfig}; +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; +use tokio::time::timeout; + +const GROUPS: u64 = 25; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fan_out_follows_subscriber_then_publisher_priority_then_group_order() { + timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time"); +} + +async fn attach(relay: &Relay, gated: bool, opts: ClientOptions) -> (Client, Option) { + let mux = MuxConfig { send_high_water: 1, ..MuxConfig::default() }; + let (c, s, gate) = memory_pair(mux, gated); + if let Some(g) = &gate { + g.open(); + } + tokio::spawn(joule_moq_relay::session::serve(relay.clone(), Arc::new(s))); + let client = timeout(STEP, Client::connect(Arc::new(c), opts)).await.expect("in time").expect("SETUP in memory"); + (client, gate) +} + +/// A track: its name, its publisher's alias for it, its publisher priority. +struct T { + name: &'static str, + alias: u64, + priority: u8, +} + +const TRACKS: [T; 4] = [ + T { name: "t1", alias: 1, priority: 0 }, + T { name: "t2", alias: 2, priority: 50 }, + T { name: "t3", alias: 3, priority: 0 }, + T { name: "t4", alias: 4, priority: 200 }, +]; + +/// Each subscriber's (subscriber priority, group order) for t1..t4. +fn plan(i: usize) -> [(u8, GroupOrder); 4] { + use GroupOrder::*; + match i { + 0 => [(10, Ascending), (10, Ascending), (200, Descending), (5, Ascending)], + 1 => [(128, Descending), (128, Ascending), (128, Ascending), (128, Descending)], + _ => [(100, Ascending), (100, Descending), (0, Descending), (100, Ascending)], + } +} + +/// Every group of the tracks `js`, one object to a group, each send bounded. +async fn send(c: &Client, js: &[usize]) { + for g in 0..GROUPS { + for &j in js { + let t = &TRACKS[j]; + let h = SubgroupHeader { track_alias: t.alias, group_id: g, subgroup_id: SubgroupId::Zero, publisher_priority: Some(t.priority), properties: false, end_of_group: true, first_object: true }; + timeout(STEP, c.send_subgroup(h, &[SubgroupObject::normal(0, payload(t.name, g, 200))])).await.expect("a send in time").expect("a send"); + } + } +} + +async fn run() { + let mut cfg = RelayConfig::loopback(); + cfg.quic_listen = None; + cfg.websocket_listen = None; + cfg.limits.write_concurrency = 1; + let relay = Relay::new(cfg).unwrap(); + let space = ns(&["load"]); + + let (p1, _) = attach(&relay, false, ClientOptions::default()).await; + let (p2, _) = attach(&relay, false, ClientOptions::default()).await; + let pubs = [&p1, &p1, &p2, &p2]; + let mut held = Vec::new(); + for (t, p) in TRACKS.iter().zip(pubs) { + held.push(p.publish(space.clone(), name(t.name), t.alias, Parameters::new(), track_props(t.priority, None)).await.expect("PUBLISH")); + } + + // Three subscribers; each read in arrival order (lazily, so nothing reorders them). + let mut subs = Vec::new(); + for i in 0..3 { + let (c, gate) = attach(&relay, true, ClientOptions { lazy_data: true, ..ClientOptions::default() }).await; + let mut keys: HashMap = HashMap::new(); + for (j, t) in TRACKS.iter().enumerate() { + let (sp, order) = plan(i)[j]; + let params = Parameters::new().with(Parameter::SubscriberPriority(sp)).with(Parameter::GroupOrder(order)); + let (ok, tx, rx) = c.subscribe(space.clone(), name(t.name), params).await.expect("SUBSCRIBE"); + held.push((tx, rx)); + keys.insert(ok.track_alias, (sp, t.priority, order)); + } + subs.push((c, gate.expect("a gate"), keys)); + } + for (_, g, _) in &subs { + g.close(); + } + + // Load: both publishers send every group of their tracks, concurrently. + tokio::join!(send(&p1, &[0, 1]), send(&p2, &[2, 3])); + let total = GROUPS * TRACKS.len() as u64; + assert!(until(STEP, || relay.stats().objects_in >= total).await, "the relay received every object"); + // Settle: everything received has reached the subscribers' schedulers. + let mut stable = 0; + assert!( + until(STEP, || { + stable = if relay.stats().queued == 0 { stable + 1 } else { 0 }; + stable >= 5 + }) + .await + ); + + for (_, g, _) in &subs { + g.open(); + } + let reads = futures::future::join_all(subs.iter().map(|(c, _, _)| collect(c, STEP, |got| got.len() as u64 >= total))).await; + for (i, (got, (_, _, keys))) in reads.iter().zip(&subs).enumerate() { + assert_eq!(got.len() as u64, total, "subscriber {i}: every object"); + let key = |x: &Got| { + let (sp, pp, order) = keys[&x.header.track_alias]; + let rank = match order { + GroupOrder::Ascending => x.header.group_id, + GroupOrder::Descending => u64::MAX - x.header.group_id, + }; + (sp, pp, rank) + }; + // At most two objects were handed over before the gate shut: one in the pipe and + // one write in flight. Everything after them waited in the scheduler. + let ordered: Vec<_> = got.iter().skip(2).map(key).collect(); + let out_of_order = ordered.windows(2).filter(|w| w[0] > w[1]).count(); + assert_eq!(out_of_order, 0, "subscriber {i}: fan-out in §5.1.2 order: {:?}", got.iter().map(|x| (x.header.track_alias, x.header.group_id)).collect::>()); + } + drop(held); +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_quic.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_quic.rs new file mode 100644 index 000000000..11b5ede2c --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_quic.rs @@ -0,0 +1,10 @@ +//! Two publishers and three subscribers through the relay over native QUIC (ALPN moqt-22). + +mod common; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn two_publishers_three_subscribers_over_quic() { + tokio::time::timeout(std::time::Duration::from_secs(120), common::two_publishers_three_subscribers(common::Via::Quic)) + .await + .expect("the scenario finishes in time"); +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_websocket.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_websocket.rs new file mode 100644 index 000000000..0e463b211 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_websocket.rs @@ -0,0 +1,11 @@ +//! Two publishers and three subscribers through the relay over the WebSocket fallback +//! (`draft-ietf-webtrans-http2-15` capsules, one to each message). + +mod common; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn two_publishers_three_subscribers_over_websocket() { + tokio::time::timeout(std::time::Duration::from_secs(120), common::two_publishers_three_subscribers(common::Via::WebSocket)) + .await + .expect("the scenario finishes in time"); +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_webtransport.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_webtransport.rs new file mode 100644 index 000000000..cae316a19 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/relay_webtransport.rs @@ -0,0 +1,12 @@ +//! Two publishers and three subscribers through the relay over WebTransport over HTTP/3, +//! the client trusting the relay's certificate by its SHA-256 as a browser does with +//! `serverCertificateHashes`. + +mod common; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn two_publishers_three_subscribers_over_webtransport() { + tokio::time::timeout(std::time::Duration::from_secs(120), common::two_publishers_three_subscribers(common::Via::WebTransport)) + .await + .expect("the scenario finishes in time"); +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/request_ids.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/request_ids.rs new file mode 100644 index 000000000..debe671be --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/request_ids.rs @@ -0,0 +1,62 @@ +//! Request IDs and requests in flight (`draft-ietf-moq-transport-22` §6.4.2): a Request ID +//! of the server's parity, or one seen before, closes the session with INVALID_REQUEST_ID; +//! and the requests a client may have open at once are bounded by the relay's stream limit +//! (here WT_MAX_STREAMS on the WebSocket binding), credit returning as requests end. + +mod common; + +use common::*; +use joule_moq::draft22::{ControlMessage, Parameters, SessionError, StreamResetCode, TrackRequest}; +use joule_moq_relay::client::ClientOptions; +use joule_moq_relay::transport::TransportError; +use std::time::Duration; +use tokio::time::timeout; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn request_ids_and_requests_in_flight() { + timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time"); +} + +fn track_status(id: u64) -> ControlMessage { + ControlMessage::TrackStatus(TrackRequest { request_id: id, namespace: ns(&["x"]), name: name("y"), params: Parameters::new() }) +} + +async fn closed_with(c: &joule_moq_relay::client::Client) -> u64 { + match timeout(STEP, c.conn.closed()).await.expect("closed in time") { + TransportError::Closed { code, .. } => code, + other => panic!("{other:?}"), + } +} + +async fn run() { + let server = relay(|c| c.limits.max_requests = 2).await; + + // A Request ID with the server's parity (odd). + let (a, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let _r = a.request(&track_status(1)).await.unwrap(); + assert_eq!(closed_with(&a).await, SessionError::INVALID_REQUEST_ID.0); + + // A Request ID used twice. + let (b, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let (_t1, mut r1) = b.request(&track_status(0)).await.unwrap(); + let _ = joule_moq_relay::client::Client::response(&mut r1).await; + let _r2 = b.request(&track_status(0)).await.unwrap(); + assert_eq!(closed_with(&b).await, SessionError::INVALID_REQUEST_ID.0); + + // Two requests open at once; a third waits for credit until one of them ends. + let t = Tracks::new(ns(&["live", "r"]), "cam", 10, 6, 128); + let (p, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await; + let _pub = publish_tracks(&p, &t, None).await; + let (c, _) = connect(&server, Via::WebSocket, ClientOptions::default()).await; + let (_ok1, tx1, _rx1) = c.subscribe(t.ns.clone(), t.media.clone(), Parameters::new()).await.unwrap(); + let (_ok2, _tx2, _rx2) = c.subscribe(t.ns.clone(), t.receipts.clone(), Parameters::new()).await.unwrap(); + assert!(timeout(Duration::from_millis(400), c.conn.open_bi()).await.is_err(), "a third request opened past the stream limit"); + // Cancelling the first returns its credit (§6.4.2.3). + tx1.reset(StreamResetCode::CANCELLED.0).await; + let third = timeout(STEP, c.conn.open_bi()).await; + assert!(matches!(third, Ok(Ok(_))), "the third request opened once the first ended"); + + c.close(0, "done"); + p.close(0, "done"); + server.shutdown().await; +} diff --git a/joule-code/joule-code-rs/crates/joule-moq-relay/tests/slow_subscriber.rs b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/slow_subscriber.rs new file mode 100644 index 000000000..22d1633e1 --- /dev/null +++ b/joule-code/joule-code-rs/crates/joule-moq-relay/tests/slow_subscriber.rs @@ -0,0 +1,190 @@ +//! A slow subscriber: its low-priority objects time out at the relay, while every receipt +//! still reaches it and its high-priority track is unaffected (`draft-ietf-moq-transport-22` +//! §5.1, §5.2; WAI jwp-receipts §4.1, §4.4). +//! +//! One publisher sends a staged pair over native QUIC: a base track at publisher priority +//! 0 with no delivery timeout, and a layer track at priority 200 whose +//! OBJECT_DELIVERY_TIMEOUT is 100 ms, each with its receipts track. A fast subscriber reads +//! as it goes. A slow one gives the relay small receive windows and reads nothing until the +//! publisher has sent everything and a second and a half more has passed; the relay is +//! given a small send window, so what cannot be sent waits in its scheduler, where the +//! layer's objects pass their deadline. Then it reads all its streams concurrently. + +mod common; + +use common::*; +use joule_moq::draft22::{Parameter, Parameters, StreamResetCode}; +use joule_moq::receipt::ReceiptVerifier; +use joule_moq_relay::client::ClientOptions; +use std::time::Duration; + +const GROUPS: u64 = 40; +const STALL: Duration = Duration::from_millis(1500); + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_slow_subscriber_loses_late_layers_and_keeps_every_receipt() { + tokio::time::timeout(Duration::from_secs(120), run()).await.expect("the test finishes in time"); +} + +async fn run() { + let server = relay(|c| { + c.limits.quic_send_window_bytes = 64 * 1024; + }) + .await; + let key = ed25519_dalek::SigningKey::from_bytes(&[0x51; 32]); + let space = ns(&["stage", "cam"]); + let base = Tracks::new(space.clone(), "base", 10, 1, 0); + let layer = Tracks::new(space.clone(), "layer", 20, 2, 200); + + let (publisher, _ep) = connect(&server, Via::Quic, ClientOptions::default()).await; + let _b = publish_tracks(&publisher, &base, None).await; + let _l = publish_tracks(&publisher, &layer, Some(100)).await; + + let (fast, _ef) = connect(&server, Via::Quic, ClientOptions::default()).await; + let mut small = quinn::TransportConfig::default(); + small.stream_receive_window(quinn::VarInt::from_u32(32 * 1024)); + small.receive_window(quinn::VarInt::from_u32(96 * 1024)); + let (slow, _es) = connect_with(&server, Via::Quic, ClientOptions { lazy_data: true, ..ClientOptions::default() }, Some(small)).await; + + let next = || Parameters::new().with(Parameter::LocationFilter(joule_moq::draft22::LocationFilter::NextObject)); + let fast_base = subscribe_both(&fast, &base, next()).await; + let fast_layer = subscribe_both(&fast, &layer, next()).await; + // The slow subscriber also asks for a 50 ms delivery timeout on its receipts + // subscriptions, which it should not (moq-streaming-format §4.1): a receipts track has + // none, and the relay applies none to it. + let receipts_timeout = || Parameters::new().with(Parameter::ObjectDeliveryTimeout(50)); + let slow_base = subscribe_both_with(&slow, &base, next(), receipts_timeout()).await; + let slow_layer = subscribe_both_with(&slow, &layer, next(), receipts_timeout()).await; + + let (published_tx, published_rx) = tokio::sync::oneshot::channel::<()>(); + let publish = async { + for g in 0..GROUPS { + publish_group(&publisher, &key, &base, g, &payload("base", g, 1024)).await; + publish_group(&publisher, &key, &layer, g, &payload("layer", g, 24 * 1024)).await; + tokio::time::sleep(Duration::from_millis(15)).await; + } + let _ = published_tx.send(()); + }; + // Both subscribers read their streams concurrently, as a subscriber does: one that read + // them one at a time could wait on a stream whose bytes wait for connection credit + // held by the unread streams behind it (see the README, "A subscriber that does not + // read"). Each reads until the base track and every receipt are whole, then until a + // second passes with nothing new, so every layer stream is read or seen reset. + let whole = |media: u64, receipts: u64, layer_receipts: u64| { + move |got: &[Got]| { + let n = |a: u64| got.iter().filter(|x| x.header.track_alias == a && x.reset.is_none()).count() as u64; + n(media) == GROUPS && n(receipts) == GROUPS && n(layer_receipts) == GROUPS + } + }; + let fast_reads = collect_concurrent(&fast, STEP * 2, Duration::from_secs(1), whole(fast_base.media_alias, fast_base.receipts_alias, fast_layer.receipts_alias)); + let slow_reads = async { + // Nothing is read while the load arrives, however long a loaded host takes to send + // it, nor for the stall after: every layer object waiting then is past its deadline. + let _ = tokio::time::timeout(STEP * 2, published_rx).await; + tokio::time::sleep(STALL).await; + collect_concurrent(&slow, STEP * 2, Duration::from_secs(1), whole(slow_base.media_alias, slow_base.receipts_alias, slow_layer.receipts_alias)).await + }; + let ((), fast_got, slow_got) = tokio::join!(publish, fast_reads, slow_reads); + + // The fast subscriber: every base object and every receipt. + let (fo, fb, _, _) = verify(&fast_got, &fast_base, &key); + assert_eq!((fo, fb), (GROUPS as usize, GROUPS as usize)); + let (_, flb, _, _) = verify(&fast_got, &fast_layer, &key); + assert_eq!(flb, GROUPS as usize, "every layer receipt reached the fast subscriber"); + + // The slow subscriber's base track is whole: each group once, none reset. (Objects on + // different streams may arrive in any order; the order the relay hands them to the + // transport is checked in priority_fan_out.rs.) + let (bo, bb, _, _) = verify(&slow_got, &slow_base, &key); + if (bo, bb) != (GROUPS as usize, GROUPS as usize) { + // Where the missing base objects were: what the subscriber saw, what its transport + // counted, and what the relay says it handed to the transport. + let seen: Vec<(u64, Option)> = slow_got.iter().filter(|x| x.header.track_alias == slow_base.media_alias).map(|x| (x.header.group_id, x.reset)).collect(); + let missing: Vec = (0..GROUPS).filter(|g| !seen.iter().any(|(b, r)| b == g && r.is_none())).collect(); + let waited = slow.waited_out.lock().unwrap().clone(); + let client_quic = slow.conn.diagnostics(); + let relay_sessions = server.relay().session_diagnostics(); + // Then read every stream still queued, all at once, as a subscriber reading its + // streams concurrently would: base objects held only by flow control now arrive. + let mut reads = Vec::new(); + for _ in 0..8 * GROUPS { + let Some(mut r) = slow.next_stream(Duration::from_millis(500)).await else { break }; + reads.push(tokio::spawn(async move { + tokio::time::timeout(Duration::from_secs(10), r.start()).await.ok()?.ok()?; + read_stream(r).await + })); + } + let mut late_base = Vec::new(); + for t in reads { + if let Ok(Some(g)) = t.await + && g.header.track_alias == slow_base.media_alias + && g.reset.is_none() + { + late_base.push(g.header.group_id); + } + } + late_base.sort(); + slow.close(0, "diagnosis"); + let relay = server.relay().clone(); + until(STEP, || relay.summaries().iter().any(|s| s.summary.tally.subscriber.tracks.len() >= 4)).await; + let base_name = joule_moq::draft22::names::render(&space, Some(&base.media)); + let tallies: Vec = relay + .summaries() + .iter() + .filter_map(|s| s.summary.tally.subscriber.tracks.get(&base_name).map(|t| format!("session {}: {:?}, undelivered at close {}", s.summary.session, t, s.summary.tally.subscriber.undelivered_at_close))) + .collect(); + panic!( + "the high-priority track is unaffected: {bo} base objects and {bb} base receipts of {GROUPS}\n\ + missing base groups {missing:?}; base streams seen (group, reset) {seen:?}\n\ + next_data gave up waiting on a header {} time(s), streams queued behind each: {waited:?}\n\ + base groups that arrived once the queued streams were read concurrently: {late_base:?}\n\ + subscriber QUIC: {client_quic:?}\nrelay sessions: {relay_sessions:?}\nrelay's base tallies: {tallies:?}", + waited.len() + ); + } + let mut base_groups: Vec = slow_got.iter().filter(|x| x.header.track_alias == slow_base.media_alias).map(|x| x.header.group_id).collect(); + base_groups.sort(); + assert_eq!(base_groups, (0..GROUPS).collect::>()); + + // Its layer track lost objects to the delivery timeout; every layer receipt arrived + // anyway, and each dropped object is accounted for by inclusion (jwp-receipts §4.4). + let (lo, lb, groups, receipts) = verify(&slow_got, &slow_layer, &key); + assert_eq!(lb, GROUPS as usize, "every layer receipt reached the slow subscriber"); + assert!(lo < GROUPS as usize, "some layer objects passed their deadline ({lo} of {GROUPS} delivered)"); + let delivered: std::collections::HashSet = + slow_got.iter().filter(|x| x.header.track_alias == slow_layer.media_alias && x.reset.is_none()).map(|x| x.header.group_id).collect(); + let verifier = ReceiptVerifier::with_anchors(vec![key.verifying_key()]); + let mut inclusion = 0; + for g in 0..GROUPS { + if !delivered.contains(&g) { + verifier.verify_object(&receipts[&g], &groups[&g]).expect("a dropped object's receipt verifies by inclusion"); + inclusion += 1; + } + } + assert_eq!(inclusion as usize, GROUPS as usize - lo); + for x in slow_got.iter().filter(|x| x.header.track_alias == slow_layer.media_alias) { + if let Some(code) = x.reset { + assert_eq!(code, StreamResetCode::DELIVERY_TIMEOUT.0, "a layer stream reset for its deadline"); + } + } + + // The relay's summary of the slow session says the same. + slow.close(0, "done"); + let relay = server.relay().clone(); + assert!(until(STEP, || relay.summaries().iter().any(|s| s.summary.tally.subscriber.dropped_delivery_timeout > 0)).await); + let s = relay.summaries().into_iter().find(|s| s.summary.tally.subscriber.dropped_delivery_timeout > 0).unwrap(); + s.verify().unwrap(); + let t = &s.summary.tally.subscriber.tracks; + let layer_name = joule_moq::draft22::names::render(&space, Some(&layer.media)); + let base_name = joule_moq::draft22::names::render(&space, Some(&base.media)); + let lt = &t[&layer_name]; + assert_eq!(lt.objects_forwarded as usize, lo, "the relay forwarded exactly what arrived whole"); + assert_eq!(lt.objects_forwarded + lt.dropped_delivery_timeout, GROUPS, "each layer object forwarded or dropped"); + assert_eq!(lt.dropped_receipts_track_subscribed, lt.dropped_delivery_timeout, "every drop on a track whose receipts track the session holds"); + assert_eq!(t[&base_name].dropped_delivery_timeout, 0); + assert_eq!(t[&base_name].objects_forwarded, GROUPS); + + fast.close(0, "done"); + publisher.close(0, "done"); + server.shutdown().await; +} diff --git a/joule-code/joule-code-rs/crates/joule-moq/README.md b/joule-code/joule-code-rs/crates/joule-moq/README.md index 55db94679..fb8623c62 100644 --- a/joule-code/joule-code-rs/crates/joule-moq/README.md +++ b/joule-code/joule-code-rs/crates/joule-moq/README.md @@ -98,13 +98,14 @@ and a peer that does not know it ignores it (§9.1). ## Not yet implemented - WebTransport (MOQT over HTTP/3, §6.2.1), a relay with priority fan-out and - per-object receipts, and a browser client: the next slice builds them on this codec. -- The authorization token cache (§8.9: aliases, MAX_AUTH_TOKEN_CACHE_SIZE, - retirement): the codec reads and writes tokens, and the session keeps no cache. + per-object receipts, and the authorization token cache (§8.9) are in + `joule-moq-relay`, built on this codec; this crate's own session keeps no token + cache. A browser client is not built yet. - Evaluating filters at a publisher (§3.3), and holding a peer to its MAX_FILTER_RANGES: the codec reads and writes every filter and counts their ranges. -- GOAWAY migration, delivery-timeout enforcement on real streams, and §5.1's - scheduling in the QUIC transport; the mock relay applies delivery timeouts. +- GOAWAY, delivery-timeout enforcement on real streams and §5.1's scheduling are + `joule-moq-relay`'s; this crate's QUIC transport has none of them, and its mock + relay applies delivery timeouts on a caller-driven clock. - `draft-ietf-moq-loc` and `draft-ietf-moq-msf` semantics (catalogs, LOC properties): their properties ride as any property does. - The receipts-track payload in the JSON form of WAI `extensions/jwp-receipts.md` diff --git a/joule-code/joule-code-rs/crates/joule-moq/src/lib.rs b/joule-code/joule-code-rs/crates/joule-moq/src/lib.rs index 97604259b..da365e3b0 100644 --- a/joule-code/joule-code-rs/crates/joule-moq/src/lib.rs +++ b/joule-code/joule-code-rs/crates/joule-moq/src/lib.rs @@ -44,13 +44,13 @@ //! //! ## What this crate intentionally does not ship (yet) //! -//! - WebTransport (MOQT over HTTP/3), a relay with priority fan-out, and a -//! browser client: the next slice builds them on this codec. -//! - The authorization token cache (§8.9: aliases, MAX_AUTH_TOKEN_CACHE_SIZE) -//! and the subscription filters' evaluation at a publisher: the codec reads -//! and writes them, and the session does not yet act on them. -//! - The relay binary (`joule-moq-relay`) and publisher SDK (`joule-moq-publisher`) -//! live in their own crates and depend on this one. +//! - WebTransport (MOQT over HTTP/3), a relay with priority fan-out, and the +//! authorization token cache (§8.9): `joule-moq-relay`, its own crate, builds +//! them on this codec. A browser client is not built yet. +//! - The subscription filters' evaluation at a publisher: the codec reads and +//! writes them, and the session does not yet act on them. +//! - A publisher SDK (`joule-moq-publisher`) would live in its own crate and depend +//! on this one. //! //! ## Standards alignment //!