open-standards/joule-code/joule-code-rs/crates/joule-moq-remote/README.md

# joule-moq-remote

A host agent for WAI remote desktop rooms (`wai.live.remote`,
[`wai/extensions/remote-desktop.md`](../../../../wai/extensions/remote-desktop.md)) over
`joule-moq-relay`, and a viewer with no display.

The host captures its screen, encodes it with the platform's encoder, and publishes it as the
room's `screen` track, one frame a group, each with its receipts (WAI moq-streaming-format §4.1,
jwp-receipts). It signs a screen statement (`WDS1`) naming what it captures, the frame size every
input's coordinates are in, the actions it injects and the encoder that made the frames. It reads
every participant's events, decides each input statement (`WDI1`) by the rules of §5.3 at its
place in the room's order, with `wai-rs`'s own `remote_desktop::decide`, injects what the rules
accept, and signs an outcome (`WDO1`) for every input: applied, anchored on the first frame it
captured after the injection, or refused with the rule's number. A room that keeps a floor passes
it by floor statements (`WDF1`): the host gives it on request, or when told.

The host makes a sealing key for each session (an X25519 key pair) and states its public half in
its screen statement (§5.5): controllers seal their key, text and clipboard actions to it, so the
relay, the room service and every viewer hold only ciphertext and commitments, and the host opens
each sealed input before it injects it (`wai-rs`'s `decide_opened`, all nine rules of §5.3). The
opening key never leaves the session unless the host is asked for it: `--opening-key <file>`
writes it (mode 0600, never over a file) for a verifier the host chooses, which checks the session
at the opened level (`remote_desktop_verify.py --room <session> --opening-key <hex>`).

## A room a room service keeps

A room service (`joule-moq-room` with `[remote]`) keeps the room: it grants the host, each
controller and each viewer for a key the visitor makes, and its clock records who is present and
takes every principal's records in order. The host makes a key once, and runs in the room:

```
wai-remote-host key ./host.key                       # mode 0600; prints the public key
wai-remote-host run --room https://room.example/room-api --key ./host.key \
    [--invitation <hex from wai-room invite>] --relay wss://relay.example/moq \
    [--ca ./private-ca.pem] --floor floor --give-on-request
```

The service names the host's key in its configuration, or takes the first key that presents an
invitation its authority signed. The host asks for its grant, publishes its screen, and states it
once the service's clock has recorded it joining; it learns each participant from the clock's
`join`, reads the room's grants again for one it does not hold, and stamps its records on the
service's clock, by the skew the service's answer states. It hands control over by asking the
service for a grant exchange (`agent::Command::Control`), its grant and a proof of its key over
the service's challenge. The service is reached over `https://` (or `http://` on loopback) and the
relay over `wss://` (or `ws://`). Over TLS every certificate is verified: its chain against the
system's roots and any certificate a PEM file names (`--ca`, a private deployment's authority),
and its name against the host the URL names; nothing turns that off (`tls`). A host on another
machine than the service needs no front of its own. `tls::front` is the other end, a TLS front for
a plain backend, which the tests put before the service and the relay. A page enters such a room
through the room page of `joule-moq-web`, under a key it makes; `viewer::Viewer::join_service` is
the headless one.

Either way the host takes each participant's records in its chain's order, as the service's clock
does: under the policy of its grant in effect (live-events §6), after its `join`, each statement
read and held to who may sign it; a floor statement §7.2 refuses is taken and changes nothing,
as a verifier has it. A participant whose record is out of that order is cut there: the host takes nothing more of it, and decides none of
its inputs, so the host and the clock hold the same records. In a service's room the host also reads the service's
bound per principal (`GET /room`'s `remote.max_records_per_principal`) and cuts a participant whose records pass it at
the bound, the record at which the service's clock cuts and removes it.

## A room the host keeps

`wai-remote-host room <dir>` makes a desk, a room the host keeps itself: its descriptor and
grants, signed by an authority whose key is derived from a seed, the host's grant marked
`remote_host`, and a grant for each participant with its control classes in `remote_control` (1
pointer, 2 keys, 4 touch, 8 clipboard; 0 for a viewer). It writes `room.json` (public),
`host-keys.json` (the host's and the clock's keys, mode 0600), `participant-<name>.json` for each
participant (the room and that participant's key, mode 0600, for the headless viewer), and
`relay.toml`: the lines the relay's configuration takes, the authority's key and the clock's token
by its SHA-256. The host is its room's clock: a participant joins when its events track appears on
the relay, and leaves when it ends.

```
wai-remote-host room ./desk --size 1280x720 --fps 30 --participant alice:3 --participant bob:0
# append ./desk/relay.toml to the relay's configuration, then
wai-remote-host run ./desk --relay wss://relay.example/moq --floor floor --give-on-request
```

`run` takes `--source display|pattern`, `--encoder platform|platform-hevc|ipcm`,
`--input platform|record`, `--floor open|floor`, `--max-age-ms`, `--redact x,y,w,h` (any number)
and `--seconds`. `wai-remote-host check` says what the machine offers: the platform encoder and
whether it runs on the media engine, and whether the operating system lets the process capture the
screen and inject input. Both need the user's permission once, in the system's privacy settings;
until then `run --source display` refuses to start, and states why.

## What each platform gives

| | capture | encode | input |
|---|---|---|---|
| macOS | the display stream of the window server, BGRA (`adapter_macos_capture`) | the media framework's encoder on the hardware media engine, AVC High 4.0 or HEVC Main, real time, no reordering (`adapter_macos_encode`) | the window server's event services: pointer, buttons and drags, wheel in pixels, keys by HID usage, composed text; clipboard through the system's pasteboard tool (`adapter_macos_inject`) |
| Linux | the X Window System's root window through its client library, loaded at run time (`adapter_linux_x11`) | the pure-function encoder; a platform encoder adapter plugs in behind the same trait | the kernel's user-level input device: an absolute pointer, buttons, wheels and keys; clipboard through the display's clipboard tool when one is installed (`adapter_linux_uinput`) |
| anywhere | a pattern source that draws what was injected (`capture::Pattern`) | `encode::Ipcm`: H.264 with every macroblock I_PCM, every picture an IDR picture, a pure function of its input that every conforming decoder decodes | `inject::Recorder`, which draws on the pattern |

A screen statement's `accepts` states what the injector injects, so a controller sees, before it
sends, that touch is not taken on a desktop and text not on the kernel's input device. The
encoder sits behind `encode::Encoder`: an adapter of the platform-encode interfaces plugs in as
another implementation.

## The viewer

`viewer::Viewer` joins a room under its participant file, fetches what the clock and the host
said before it arrived, verifies every frame against its receipt, sends input and floor
statements, its key, text and clipboard actions sealed to the host's sealing key, and signs
latency records of `source` 3 (no display: the instant it held the frame verified). It keeps the
records it receives, and follows another principal's events when told to. The browser viewer is the room page of `joule-moq-web` (`web/room/remote.js`).

## Tests

- `cargo test --locked --all-targets -- --test-threads=1`: the units (the bit writer's
  Exp-Golomb codes, emulation prevention, an I_PCM unit's parameter sets, redactions, the
  pattern and the recorder, the room's grants, the envelope, timestamps, a service's URL, and the
  gate's order: a record that arrives before the one it follows waits, one stamped before it cuts
  its author, one of a principal not yet joined waits for its `join` and is out of order once the
  clock's chain is held past it), and
  `tests/loopback.rs`: a session over a relay on loopback that requires grants with their proofs.
  Alice's click under the floor is applied, bob (no control power) is refused
  `input_not_permitted`, carol off the floor `input_no_floor`; alice types a string and composes
  a text, sealed, and the host injects them as typed while bob, following her events, receives no
  trace of them, and her text in the clear is refused `input_unsealed`; the floor is handed to
  carol, and alice is refused while carol's move is applied; the host pauses and carol is refused
  `input_screen_paused`. Every viewer verifies every frame; alice and carol sign latency records;
  and the session's chains, as the host holds them, verify whole by `wai-rs`, at the commitment
  level and at the opened level under the host's opening key, every outcome the rules', each
  latency with its legs. `tests/tls.rs`: the service over `https://` and the relay over `wss://`
  through a TLS front, trusted by a PEM file; refused when only the system's roots are trusted,
  and refused for a certificate that names another host. On macOS the session runs a second time with the platform's
  hardware encoder, and a third test holds its units to the Annex B form `wai.video.avc`
  carries.
- The same host in a room the room service keeps is `joule-moq-room`'s `tests/remote_room.rs`.