open-standards/joule-code/joule-code-rs/crates/joule-moq-remote

joule-moq-remote

A host agent for WAI remote desktop rooms (wai.live.remote, wai/extensions/remote-desktop.md) over joule-moq-relay, and a viewer with no display.

The host captures its screen, encodes it with the platform's encoder, and publishes it as the room's screen track, one frame a group, each with its receipts (WAI moq-streaming-format §4.1, jwp-receipts). It signs a screen statement (WDS1) naming what it captures, the frame size every input's coordinates are in, the actions it injects and the encoder that made the frames. It reads every participant's events, decides each input statement (WDI1) by the rules of §5.3 at its place in the room's order, with wai-rs's own remote_desktop::decide, injects what the rules accept, and signs an outcome (WDO1) for every input: applied, anchored on the first frame it captured after the injection, or refused with the rule's number. A room that keeps a floor passes it by floor statements (WDF1): the host gives it on request, or when told.

The host makes a sealing key for each session (an X25519 key pair) and states its public half in its screen statement (§5.5): controllers seal their key, text and clipboard actions to it, so the relay, the room service and every viewer hold only ciphertext and commitments, and the host opens each sealed input before it injects it (wai-rs's decide_opened, all nine rules of §5.3). The opening key never leaves the session unless the host is asked for it: --opening-key <file> writes it (mode 0600, never over a file) for a verifier the host chooses, which checks the session at the opened level (remote_desktop_verify.py --room <session> --opening-key <hex>).

A room a room service keeps

A room service (joule-moq-room with [remote]) keeps the room: it grants the host, each controller and each viewer for a key the visitor makes, and its clock records who is present and takes every principal's records in order. The host makes a key once, and runs in the room:

wai-remote-host key ./host.key                       # mode 0600; prints the public key
wai-remote-host run --room https://room.example/room-api --key ./host.key \
    [--invitation <hex from wai-room invite>] --relay wss://relay.example/moq \
    [--ca ./private-ca.pem] --floor floor --give-on-request

The service names the host's key in its configuration, or takes the first key that presents an invitation its authority signed. The host asks for its grant, publishes its screen, and states it once the service's clock has recorded it joining; it learns each participant from the clock's join, reads the room's grants again for one it does not hold, and stamps its records on the service's clock, by the skew the service's answer states. It hands control over by asking the service for a grant exchange (agent::Command::Control), its grant and a proof of its key over the service's challenge. The service is reached over https:// (or http:// on loopback) and the relay over wss:// (or ws://). Over TLS every certificate is verified: its chain against the system's roots and any certificate a PEM file names (--ca, a private deployment's authority), and its name against the host the URL names; nothing turns that off (tls). A host on another machine than the service needs no front of its own. tls::front is the other end, a TLS front for a plain backend, which the tests put before the service and the relay. A page enters such a room through the room page of joule-moq-web, under a key it makes; viewer::Viewer::join_service is the headless one.

Either way the host takes each participant's records in its chain's order, as the service's clock does: under the policy of its grant in effect (live-events §6), after its join, each statement read and held to who may sign it; a floor statement §7.2 refuses is taken and changes nothing, as a verifier has it. A participant whose record is out of that order is cut there: the host takes nothing more of it, and decides none of its inputs, so the host and the clock hold the same records. In a service's room the host also reads the service's bound per principal (GET /room's remote.max_records_per_principal) and cuts a participant whose records pass it at the bound, the record at which the service's clock cuts and removes it.

A room the host keeps

wai-remote-host room <dir> makes a desk, a room the host keeps itself: its descriptor and grants, signed by an authority whose key is derived from a seed, the host's grant marked remote_host, and a grant for each participant with its control classes in remote_control (1 pointer, 2 keys, 4 touch, 8 clipboard; 0 for a viewer). It writes room.json (public), host-keys.json (the host's and the clock's keys, mode 0600), participant-<name>.json for each participant (the room and that participant's key, mode 0600, for the headless viewer), and relay.toml: the lines the relay's configuration takes, the authority's key and the clock's token by its SHA-256. The host is its room's clock: a participant joins when its events track appears on the relay, and leaves when it ends.

wai-remote-host room ./desk --size 1280x720 --fps 30 --participant alice:3 --participant bob:0
# append ./desk/relay.toml to the relay's configuration, then
wai-remote-host run ./desk --relay wss://relay.example/moq --floor floor --give-on-request

run takes --source display|pattern, --encoder platform|platform-hevc|ipcm, --input platform|record, --floor open|floor, --max-age-ms, --redact x,y,w,h (any number) and --seconds. wai-remote-host check says what the machine offers: the platform encoder and whether it runs on the media engine, and whether the operating system lets the process capture the screen and inject input. Both need the user's permission once, in the system's privacy settings; until then run --source display refuses to start, and states why.

What each platform gives

	capture	encode	input
macOS	the display stream of the window server, BGRA (adapter_macos_capture)	the media framework's encoder on the hardware media engine, AVC High 4.0 or HEVC Main, real time, no reordering (adapter_macos_encode)	the window server's event services: pointer, buttons and drags, wheel in pixels, keys by HID usage, composed text; clipboard through the system's pasteboard tool (adapter_macos_inject)
Linux	the X Window System's root window through its client library, loaded at run time (adapter_linux_x11)	the pure-function encoder; a platform encoder adapter plugs in behind the same trait	the kernel's user-level input device: an absolute pointer, buttons, wheels and keys; clipboard through the display's clipboard tool when one is installed (adapter_linux_uinput)
anywhere	a pattern source that draws what was injected (capture::Pattern)	encode::Ipcm: H.264 with every macroblock I_PCM, every picture an IDR picture, a pure function of its input that every conforming decoder decodes	inject::Recorder, which draws on the pattern

A screen statement's accepts states what the injector injects, so a controller sees, before it sends, that touch is not taken on a desktop and text not on the kernel's input device. The encoder sits behind encode::Encoder: an adapter of the platform-encode interfaces plugs in as another implementation.

The viewer

viewer::Viewer joins a room under its participant file, fetches what the clock and the host said before it arrived, verifies every frame against its receipt, sends input and floor statements, its key, text and clipboard actions sealed to the host's sealing key, and signs latency records of source 3 (no display: the instant it held the frame verified). It keeps the records it receives, and follows another principal's events when told to. The browser viewer is the room page of joule-moq-web (web/room/remote.js).

Tests

• cargo test --locked --all-targets -- --test-threads=1: the units (the bit writer's Exp-Golomb codes, emulation prevention, an I_PCM unit's parameter sets, redactions, the pattern and the recorder, the room's grants, the envelope, timestamps, a service's URL, and the gate's order: a record that arrives before the one it follows waits, one stamped before it cuts its author, one of a principal not yet joined waits for its join and is out of order once the clock's chain is held past it), and tests/loopback.rs: a session over a relay on loopback that requires grants with their proofs. Alice's click under the floor is applied, bob (no control power) is refused input_not_permitted, carol off the floor input_no_floor; alice types a string and composes a text, sealed, and the host injects them as typed while bob, following her events, receives no trace of them, and her text in the clear is refused input_unsealed; the floor is handed to carol, and alice is refused while carol's move is applied; the host pauses and carol is refused input_screen_paused. Every viewer verifies every frame; alice and carol sign latency records; and the session's chains, as the host holds them, verify whole by wai-rs, at the commitment level and at the opened level under the host's opening key, every outcome the rules', each latency with its legs. tests/tls.rs: the service over https:// and the relay over wss:// through a TLS front, trusted by a PEM file; refused when only the system's roots are trusted, and refused for a certificate that names another host. On macOS the session runs a second time with the platform's hardware encoder, and a third test holds its units to the Annex B form wai.video.avc carries.
• The same host in a room the room service keeps is joule-moq-room's tests/remote_room.rs.