open-standards/wai/extensions/remote-desktop.md

WAI Extension: Remote Desktop

Status: Draft. Registers the record wai.live.remote: one principal's screen shared into a room and controlled from it. The host publishes its screen as the room's screen track, each frame an object with its WAI receipts, and signs a screen statement, WDS1: what it captures, the frame size that is the coordinate space of every input, the actions it injects, and the encoder that made the frames, named as packaging §7 (its paragraph 7.5) requires of an encoder that is not a pure function of its input, and read by no step that accepts or refuses anything. A principal whose grant carries the control power, a grant attribute, sends input statements, WDI1: pointer, key, text, touch and clipboard actions in the frame's coordinates. The host injects an input only when every rule of §5.3 holds at the input's instant, and signs an outcome, WDO1, for every input it receives: applied, naming the first frame it captured after it injected the actions, or refused, with the rule that refused it. Control passes by the room's grants alone, or, in a room that keeps a floor, by floor statements, WDF1, that request, give, release and take it back. A viewer that presents the frame an outcome names signs a latency record, WDL1: the instant of its action and the instant its platform reports the frame presented, both on its own estimate of the room clock, with the round trip that estimate rests on. Everything rides in the room's events, as meta payloads their authors sign; a verifier recomputes every outcome from the rules and refuses a host that applied what the rules refuse. A host that states a sealing key takes key, text and clipboard actions sealed to it (§5.5): their fields travel in one HPKE ciphertext a statement, which the host alone opens, beside a commitment to each action, while pointer, wheel and touch actions stay in the clear, as the screen shows them anyway. A viewer verifies every outcome without the plaintext; a verifier given the host's opening key also opens every seal and checks every commitment (§11.4). No new transport. The reference implementation is the remote_desktop module of wai-rs (feature remote_desktop); the reference host agent and its browser viewer are joule-moq-remote and the room page of joule-moq-web; the corpus is live-conformance/remote/, and its second implementation is live-conformance/remote_desktop_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: the live family this extension builds on, as merged (live-events, live-rooms, moq-streaming-format, live-metering, live-recording, confidentiality and co-watch), packaging §7 (its paragraph 7.5) and jwp-receipts; RFC 8032, FIPS 180-4 and RFC 6381; RFC 9180 (§4, §5.1, §6.1, §7.1) and RFC 7748 §5, for sealed actions; HID Usage Tables 1.5 §10, the Keyboard/Keypad page (0x07); and the W3C UI Events specifications for pointer buttons, wheel deltas in pixels and keyboard code values (uievents, uievents-code). No other outside text is pinned: the rules of this extension are WAI's own.

1. Scope

In scope: the grant attributes this extension reads (§3.1); the host's screen track and its screen statement, what the host captures and the encoder it names (§3); how the statements ride in a room's events, how they are read, and who signs each (§4); input statements, their actions, the host's rules and how it applies an input (§5), and key, text and clipboard actions sealed to the host (§5.5); outcomes (§6); handover of control, by grant change and by the floor (§7); measured input-to-photon latency (§8); session records and metering (§9); where each rule is enforced (§10); and verification at two levels (§11).

Out of scope: a transport; how a host captures its screen or injects an action, which is its platform's; how a codec encodes a frame (§3.6); how a sink decodes and draws one; the host's clipboard and files sent to viewers, which this revision does not define; audio from the host, which is a room audio track like any other (live-rooms §6.1); and a recording of the session, which the room's recording policy governs (live-rooms §3.7, live-recording).

Relationship to the core spec: wai.live.remote is a record (SPEC §5): signed records a WAI tool emits and verifies. It names no payload format and is not dispatchable. The frames are payloads of capabilities SPEC §5 already registers (wai.video.avc, wai.video.hevc, wai.video.av1, wai.video.vp9), one access unit an object.

Relationship to live-events: this extension changes nothing in it. Its statements are payloads of the meta kind (§4.2 there), which live-events leaves to applications, as co-watch's are, so every statement is signed by its author, chained, stamped with the room clock and rate-bounded by its author's policy (live-events §5, §6). An outcome that names a frame anchors it (live-events §3.1).

Relationship to live-rooms: the descriptor's and the grant's layouts are unchanged. The screen is the room's screen media kind (live-rooms §6.1, publish bit 2), and its tracks are named as live-rooms §6.1 names every track. The host mark and the control power are grant attributes (§3.1), which live-rooms §4.4 signs with the grant and leaves to the extensions that read them: a new power bit would be refused by every room verifier of the current revision (live-rooms §3.4: no power above bit 4). The direct power (live-rooms §3.3, bit 1) is the power to give and take back the floor (§7.2).

Relationship to packaging §7 (its paragraph 7.5): a frame is made by an encoder that is not a pure function of its input, so the host names that encoder and its version. The name is a statement about where the bytes came from; the bytes are bound by their digests in their receipts, and nothing recomputes them (§3.6).

2. Terms

• Room, room clock, principal, event, chain, grant, grant in effect, room key, merged order, present: as in live-events §2 and live-rooms §2, §9.2 and §9.3. A room key is the room authority's key or the clock's.
• Statement: the payload of a meta event whose first four bytes are one of the magics of §4.1. It is named by the digest of its event (live-events §3.2).
• Host: a principal whose grant in effect carries remote_host with the value true (§3.1). Its screen is what it captures and publishes (§3.5).
• Controller: a principal whose grant in effect carries the control power (§3.1).
• Frame: one object of the host's screen track: one access unit of the capability its screen statement names, in a WAI1 envelope (moq-streaming-format §3), with its receipts (§3.2).
• Action: one pointer, key, text, touch or clipboard input (§5.2). Its class is the bit of the control power that allows it.
• Sealing key: an X25519 public key (RFC 7748 §5) a host states in its screen statement (§3.3). Its opening key is the matching private key, which the host holds.
• Sealed action: a key, text or clipboard action whose fields are sealed to the host's sealing key (§5.2, §5.5). Its opening is a salt and its fields; its commitment is the digest that binds them (§5.5). A statement's seal holds the openings of its sealed actions.
• Estimate: a principal's estimate of the room clock, on which it stamps its events' t.
• A short field is a u8 length and that many bytes; a token is 1 or more bytes of ! to ~ (0x21–0x7E), as in live-rooms §2.

Integers are little-endian; a signed integer is two's complement. A digest is SHA-256 (FIPS 180-4), 32 bytes. Ed25519 is RFC 8032's, verified strictly (SPEC §7 Signatures): a key that fails step 1 or 2 is bad_key, and a signature that fails a later step is bad_signature. µs is microseconds of room clock.

3. The host and its screen

3.1 The attributes

This extension reads two grant attributes (live-rooms §4.4):

key	value	meaning
remote_host	true or false	whether the principal may host: sign screen statements and outcomes (§4.3) and receive input
remote_control	a decimal integer from 1 to 15	the control power: the classes of action the principal may send. Bit 0 pointer (types 1 to 3 of §5.2), bit 1 keys (types 4 and 5), bit 2 touch (type 6), bit 3 clipboard (type 7)

A decimal integer here is 1 or 2 of the digits 0–9, the first not 0. A grant whose remote_host is neither true nor false, or whose remote_control is not of that form or is above 15, is refused rd_attr_malformed. A grant without remote_host does not host, and one without remote_control carries no control power. A host's grant MUST also let it publish the screen media kind (live-rooms §3.3, publish bit 2); one that does not cannot sign a screen statement (§4.3).

The authority states who hosts and who controls in the grants it signs, and changes either by exchanging a grant (live-rooms §4.6, §7.1 here). The value of remote_control is the classes: 1 pointer alone, 3 pointer and keys, 15 everything.

3.2 The screen track

The host publishes its screen as its own track of the screen kind (live-rooms §6.1): the Track Namespace of the room id, its key and screen, the Track Name its layer, 0 for a single layer. The track's catalog entry declares the capability its frames carry (moq-streaming-format §4, wai_capabilities), its codecs value (codec, RFC 6381), its size, wai_receipts true and its receipts track (moq-streaming-format §4.1), and a sink verifies each frame against its receipt before it decodes it, as it verifies every object of a room. Each frame is one group of one object, so a group receipt covers one frame; a frame that is an intra-coded picture (an IDR access unit, or an AV1 or VP9 key frame) carries key true in its manifest's target.

The track digest of the screen is the anchor track of its layer 0 (live-events §3.1): SHA-256 over the namespace's rendering <room>-<host>-screen (the room id and the host's key as 64 lowercase hexadecimal digits each), one zero byte, and 0.

A host SHOULD send an intra-coded frame at least every two seconds, and at once when a principal joins, so that a viewer that joins decodes within that time.

3.3 The screen statement

field	size	meaning
magic, version	5	WDS1, 1
prev	32	the digest of the event of the host's screen statement this one replaces; 32 zero bytes for its first
state	1	1 live: the host captures and accepts input; 2 paused: it captures nothing and accepts no input (§3.5)
track	32	the track digest of the host's screen (§3.2)
width, height	2 each	u16: the frames' size in pixels, which is the coordinate space of every action (§5.2); each above 0
fps	2	u16: the most frames a second it captures, above 0
scope	1	what it captures: 1 a whole display, 2 one window, 3 a region of a display
region	8	for scope 3, the region on the display, in the display's pixels: x, y, w, h (u16 each), w and h above 0; for scope 1 and 2, eight zero bytes
redactions	1 + 8 × n	a u8 count, at most 16, then each rectangle the host blanks in every frame before it encodes it: x, y, w, h (u16 each) in frame pixels, w and h above 0, within the frame
pointer	1	1 the frames show the host's pointer, 0 they do not
accepts	1	the action types the host injects: bit k − 1 for type k of §5.2; no bit above bit 6
clip_max	2	u16: the most bytes of text one clipboard action may carry; above 0 exactly when accepts holds type 7
max_age_ms	4	u32: the oldest an action may be when the host decides on its input (§5.3 step 7), above 0
floor	1	1 open: every controller may send input; 2 floor: only the controller holding the floor (§7)
capability	short	the frames' payload capability (SPEC §5), a token
codec	short	the frames' codecs value (RFC 6381) as the track's catalog entry states it, a token
encoder	short	the encoder's name and version (§3.6): a token holding exactly one @, with a byte before it and one after
api	1	the encoder's interface, by role (§3.6): 1 to 5
hardware	1	where the encoder runs (§3.6): 0 general-purpose processors, 1 a fixed-function media engine, 2 not stated
seal	0 or 32	the host's sealing key (§5.5): an X25519 public key (RFC 7748 §5); absent when the host takes key, text and clipboard actions in the clear. The statement ends after hardware, or after these 32 bytes

A screen statement is a meta payload of at most 5 + 32 + 1 + 32 + 2 × 3 + 1 + 8 + 1 + 16 × 8

• 1 + 1 + 2 + 4 + 1 + 3 × 256 + 2 + 32 bytes, within the default max_payload of live-events §5.

3.4 Rules and the chain

A screen statement whose fields break §3.3's table is rd_statement_malformed. Then:

1. its track is the track digest of its author's screen (§3.2): otherwise screen_wrong_track;
2. its prev is the digest of the event of its author's screen statement before it in the merged order, or 32 zero bytes when it has none: otherwise screen_unlinked.

The host's screen in effect at a point of the merged order is its last screen statement the room accepted before that point. A statement is replaced, never amended: a new size, a new scope, a pause, a resume or a new encoder is a new statement, which takes effect at its place in the merged order.

3.5 What is captured

What a host captures is the host's choice, and the screen statement states it: whether the frames hold a whole display, one window or a region of a display (scope, region); the rectangles it blanks before it encodes (redactions); whether its pointer is drawn (pointer); which actions it injects (accepts) and how much clipboard text it takes (clip_max); whether it takes key, text and clipboard actions sealed (seal, §5.5); and whether it captures at all (state). A host MUST capture nothing outside what its screen in effect states, MUST blank every redaction in every frame before the frame reaches its encoder, and MUST NOT publish a frame while its screen in effect is paused. A sink SHOULD show its viewer the scope, the redactions and the state.

A host that stops sharing for a moment pauses (a statement with state 2) rather than leaving; one that resumes signs a live statement, and its next frame is intra-coded.

3.6 The encoder

encoder names the encoder that made the frames, and its version, as packaging §7 (paragraph 7.5) has a floor name an encoder that is not a pure function of its input: the encoder's name, @, and its version as the encoder reports it. api names the interface the host drove it through, by role:

api	the interface
1	the operating system's media framework
2	a video acceleration interface whose driver drives a GPU's encode engine
3	the encode interface a GPU driver exports from its own library
4	a software encoder library
5	an encoder that is a pure function of its input

hardware states where it ran, as the interface reports it: 0 general-purpose processors, 1 a fixed-function media engine, 2 not stated. A host that cannot tell states 2, never 1.

The three fields describe how the frames were made, and nothing else. No step of this extension reads them to accept or refuse a frame, an input, an outcome or a latency record; no digest a verifier recomputes covers them, beyond the statement's own event; and no verifier re-encodes a frame. Each frame is bound by the content hash its receipt states (jwp-receipts), and a viewer that holds a frame and its receipt holds the bytes the host published, whatever made them. A host that changes its encoder signs a new screen statement.

4. Statements

4.1 Carriage

magic	statement	section	who signs it
WDS1	screen statement	§3.3	a host
WDI1	input statement	§5.1	any principal that is not a room key
WDO1	outcome	§6.1	the host its input names
WDF1	floor statement	§7.2	as §7.2 states for its action
WDL1	latency record	§8.1	any principal that is not a room key

Each statement is the payload of a meta event: its magic, a version byte of 1, and the fields its section lists. A meta payload whose first four bytes are none of these magics is not this extension's, and a sink that does not implement it shows none of them: they are timed metadata for its application (live-events §4.2). Every statement is bounded by its author's policy (live-events §5, max_payload): an authority that gives a principal the clipboard class gives its grant a max_payload that holds the clipboard text it allows.

4.2 Reading

A statement that ends before a field, that holds bytes after its last field, whose version is not 1, or whose field holds a value its section does not allow, is rd_statement_malformed.

4.3 Who signs

A screen statement whose author's grant in effect at its t does not carry remote_host with the value true or does not let it publish the screen kind, an outcome whose author is not the host of the input it names, and an input statement or a latency record by a room key, are rd_not_permitted. A floor statement's author is checked by §7.2. A grant's powers and attributes are those of the author's grant in effect at its event's t (live-rooms §9.2 step 4). Since a statement is a meta event, the room verifier has already held it to its author's grant: a principal whose policy does not permit meta cannot send one (live-rooms §9.2 step 4, kind_not_permitted).

5. Input

5.1 The input statement

field	size	meaning
magic, version	5	WDI1, 1
host	32	the key of the host the actions are for
screen	32	the digest of the event of the host's screen statement whose frame the actions' coordinates are in
count	1	the number of actions, 1 to 32
actions	…	each action: dt_us (u32), type (u8), and the fields of its type (§5.2)
seal	…	present exactly when an action is sealed (type 8 of §5.2): enc, 32 bytes, then the ciphertext, to the statement's end (§5.5)

dt_us is how long before its event's t the action happened, so the action's instant is t − dt_us; it is at most t. The actions are in the order they happened: each one's dt_us is at most the one's before it. A statement that breaks this, or whose count is outside 1 to 32, is rd_statement_malformed. So is one with a sealed action whose ciphertext is not exactly 16 bytes longer than the openings of its sealed actions (§5.5), and one with no sealed action that holds any byte after its last action. A controller batches the actions of a short interval (a pointer moving over one display refresh, a key and its release) into one statement, so that the rate its grant's policy bounds is a rate of statements, not of actions.

5.2 Actions

type	name	class	fields	meaning
1	move	pointer	x, y (u16 each)	the pointer moved to (x, y)
2	button	pointer	button (u8), down (u8), x, y (u16 each)	a button went down (down 1) or up (0) at (x, y); button 0 primary, 1 auxiliary, 2 secondary, 3 fourth, 4 fifth, as UI Events numbers a mouse event's button
3	wheel	pointer	dx, dy (i16 each), x, y (u16 each)	the wheel turned by dx, dy pixels with the pointer at (x, y), as UI Events states a wheel event's deltas in pixels; not both 0
4	key	keys	usage (u16), down (u8), modifiers (u8)	a key went down (1) or up (0): usage its usage ID on the HID Keyboard/Keypad page (0x07), 0x04 to 0xE7; modifiers the modifiers held: bit 0 shift, bit 1 control, bit 2 alt, bit 3 meta, no other bit
5	text	keys	text: a u8 length, 1 to 64, and that many bytes of UTF-8	text composed on the controller's side, as an input method composes it, typed as it is
6	touch	touch	id (u8), phase (u8), x, y (u16 each)	touch point id began (phase 1), moved (2), ended (3) or was cancelled (4) at (x, y)
7	clipboard	clipboard	text: a u16 length and that many bytes of UTF-8	text the host puts on its clipboard
8	sealed	the class of of	of (u8), length (u16), commit (32 bytes)	a key, text or clipboard action sealed to the host's sealing key (§5.5): of the type it seals, 4, 5 or 7; length the clipboard text's length in bytes for 7, and 0 for 4 and 5; commit its commitment. Its fields are in the statement's seal

Coordinates are in the frame of the screen statement the input names, from (0, 0) at its top left, x below width and y below height; the host maps them to what it captures (its scope, its region and any scaling). UTF-8 is RFC 3629's: no overlong form, no surrogate, nothing above U+10FFFF. A field outside its row is rd_statement_malformed; a coordinate outside the frame and clipboard text longer than clip_max are not malformed but out of bounds (§5.3 step 6), since they depend on the screen. A controller in a page maps a keyboard event's code (uievents-code) to its HID usage. A sealed action is of the type it seals wherever a rule reads an action's type (accepts, §3.3 and §5.3 step 5), and of that type's class; accepts has no bit for type 8.

5.3 The host's rules

A host decides on an input statement when it receives it. With t the statement's event's t, P its author and n the instant the host decides (the t of the outcome it signs, §6), the host refuses it at the first of these that holds, with the reason and its number:

1. its host is not the host's key, or its screen is not the digest of the host's screen in effect at the input's place in the merged order: input_wrong_screen (1);
2. P's grant in effect at t does not carry the control power with the class of every action: input_not_permitted (2);
3. the host's screen in effect keeps a floor (floor 2) and P does not hold it at the input's place in the merged order (§7.3): input_no_floor (3);
4. the screen in effect is paused: input_screen_paused (4);
5. an action's type is not among the screen's accepts: input_unsupported (5);
6. an action's coordinate is outside the frame, or a clipboard action's text (a sealed clipboard action's length) is longer than the screen's clip_max bytes: input_out_of_bounds (6);
7. n less the instant of its first action is more than the screen's max_age_ms × 1000: input_stale (7);
8. the screen in effect states a sealing key, and an action is a key, text or clipboard action in the clear (type 4, 5 or 7): input_unsealed (8);
9. it carries a seal, and the screen in effect states no sealing key, or the seal does not open under the host's opening key, does not read, or does not keep its commitments (§5.5): input_bad_seal (9).

Otherwise it applies the input (§5.4). It applies all of a statement's actions or none of them. A host MUST NOT apply an input these rules refuse, and MUST sign one outcome for every input statement addressed to it that it receives (§6).

The rules are made at the input's place in the merged order: the grant in effect, the screen in effect and the floor are those every party holding the room computes there, so a verifier recomputes the host's decision (§11.2). Rule 9 reads what a seal holds, which the opening key alone opens: a verification at the commitment level makes it only where the screen states no sealing key, and one at the opened level makes it in full (§11.4). A host decides as soon as it holds the input and every record before it that the rules read: its own screen statements, the clock's lifecycle, and the floor statements. An input that crosses a grant change, a new screen or a floor change on the wire is decided by its place, and refused if the rules then refuse it; its author sends it again under what is in effect.

5.4 Applying

A host applies an input by injecting its actions into what it captures, in order, each sealed action as the action it seals (§5.5), each at its coordinates mapped from the frame. Text and clipboard text are taken as given; the host injects no key or button state the actions do not state, and SHOULD release, when a controller loses control (§7), every key and button that controller left down.

5.5 Sealed actions

Keys typed and text composed or pasted can be passwords. A host that states a sealing key in its screen statement (seal, §3.3) takes key, text and clipboard actions sealed to it, so that no party but the host reads them: not a relay, not the room service, not a viewer. Pointer, wheel and touch actions stay in the clear: the host's screen shows where the pointer goes and what it does, and the screen is the room's. A sealed action keeps in the clear what rules 1 to 8 of §5.3 read: the type it seals, which gives its class (rule 2) and is held to accepts (rule 5); its dt_us (rule 7, and the latency records of §8); and a clipboard text's length (rule 6). A text's length is not in the clear: its opening pads it to 64 bytes.

The opening of a sealed action is 16 bytes of salt, then the fields of the type it seals: for a key, its usage (u16), down and modifiers (u8 each), as §5.2 lays them out; for a text, its length (u8, 1 to 64), then 64 bytes, the text and zero bytes after it; for a clipboard text, its length bytes of UTF-8, with no length before them. A sender draws each salt afresh from a cryptographically secure random source.

The commitment of a sealed action at index k among its statement's actions, from 0, is SHA-256 over the 18 bytes wai:remote-commit (ASCII) and 0x01, the room id, the author's key, the statement's host and screen, k (u8), the action's dt_us (u32), of (u8) and its opening. It binds the input (its fields, and the room, the host and the screen statement it is sent to), its order (its index among the statement's actions and its instant) and its author. The salt keeps it from being guessed: a key's fields take a few thousand values, and a commitment over them alone would name the key to anyone who tried each.

The seal is HPKE's single-shot SealBase (RFC 9180 §6.1), base mode, with DHKEM(X25519, HKDF-SHA256), HKDF-SHA256 and AES-128-GCM (RFC 9180 §7.1: KEM 0x0020, KDF 0x0001, AEAD 0x0001), and:

• pkR: the sealing key of the screen statement the input names (screen);
• info: the 16 bytes wai:remote-seal (ASCII) and 0x01, the room id and the host's key;
• aad: the author's key, then the statement from magic through its last action;
• pt: the openings of its sealed actions, in their order among its actions.

The statement holds the seal after its last action: enc, 32 bytes, then the ciphertext, which is 16 bytes longer than pt. A sender draws a new ephemeral key for every statement (RFC 9180 §4.1: GenerateKeyPair, or DeriveKeyPair of fresh random bytes).

Opening. The host opens a seal with OpenBase under its opening key, the same info and aad, and reads each sealed action's opening from what it opens to, in order, each as long as its of and length make it. Rule 9 of §5.3 holds, for the first of these reasons, which a verification at the opened level states (§11.4; §12's fourth table):

1. the screen statement the input names states no sealing key: seal_no_key;
2. OpenBase fails: the seal was made to another key, by another author or in another room, or was altered: seal_not_opened;
3. what it opens to does not read as the openings: a key's fields outside §5.2's row, a text's length outside 1 to 64, a padding byte that is not zero, text that is not UTF-8, or bytes left over: seal_unreadable;
4. a sealed action's commitment is not the commitment of its opening: seal_commitment.

The aad binds the seal to its statement and its author: a principal that copies another's sealed input into a statement of its own sends a seal that does not open.

A controller MUST seal every key, text and clipboard action it sends to a host whose screen in effect states a sealing key, to that key, and MUST NOT send one in the clear: the host refuses it (rule 8). A host MUST open a sealed input before it injects it, and inject only what rule 9 accepts. A host SHOULD make a sealing key for each session and discard its opening key when the session ends, so that a later compromise of its own keys opens none of the session's inputs; it MAY give the opening key to a verifier it wants to check the session at the opened level (§11.4).

6. Outcomes

6.1 Layout

field	size	meaning
magic, version	5	WDO1, 1
input	32	the digest of the input statement's event
outcome	1	1 applied, 2 refused
reason	1	0 when applied; when refused, the number of the rule of §5.3 that refused it, 1 to 9
applied_at	8	u64: when applied, the instant on the host's estimate it injected the last action; 0 when refused
captured_at	8	u64: when applied, the instant it captured the frame the event's anchor names; 0 when refused
content	32	when applied, the content hash of that frame's object as its receipt states it (jwp-receipts); 32 zero bytes when refused
rtt_us	4	u32: the round trip of the clock measurement the host's estimate rests on (§8.4); 2^32 − 1 when it states none

An applied outcome's event carries an anchor (live-events §3.1): the screen track, the group and the object of the first frame the host captured after it injected the last action, which is the first frame that can show what the actions did. A host signs it once that frame is published. A refused outcome's event carries no anchor, and the host signs it at once.

6.2 Rules

An outcome is checked, in this order:

1. its input is the digest of an input statement the room accepted: otherwise outcome_unknown_input;
2. its author is that input's host (§4.3): otherwise rd_not_permitted;
3. no outcome before it in the merged order names the same input: otherwise outcome_duplicate;
4. its fields agree with its outcome and with one another: an applied outcome has reason 0, an anchor whose track is the track digest of the host's screen and whose object is 0, and a content not all zero, and the input's t ≤ applied_at ≤ captured_at ≤ its own t; a refused one has a reason from 1 to 9, no anchor, applied_at, captured_at and content zero, and a t at or after the input's. Otherwise outcome_invalid;
5. the rules of §5.3, made at the input's place with n the outcome's t, give what it states: applied when none refuses, and refused with the reason of the first that does. At the commitment level (§11.4), rule 9 is made only for an input whose screen states no sealing key: a sealed input to a screen that states one, which rules 1 to 8 accept, takes an outcome that states applied or refused with reason 9. Otherwise outcome_mismatch.

An input statement with no outcome is unanswered. A verifier states it; it is not a refusal, since a host may never have received it.

7. Handover of control

7.1 Two ways

Control passes in one of two ways, as the host's screen in effect states.

• By grant (floor 1, open). Every controller may send input, and the classes it may send are its grant's. Control is handed over by exchanging grants (live-rooms §4.6): the authority exchanges the grant of the principal giving up control for one without remote_control, and that of the principal taking it for one with it, each effective at a tick of the room clock. From that tick the rules of §5.3 refuse the first and accept the second.
• By the floor (floor 2). Controllers may send input only while they hold the floor, which floor statements pass (§7.2). The grant still bounds what the holder may send: the floor selects one controller, and its grant's classes are what it may do.

7.2 The floor statement

field	size	meaning
magic, version	5	WDF1, 1
host	32	the host whose floor it is
action	1	1 request, 2 give, 3 release, 4 take
to	32	for give, the principal given the floor; otherwise 32 zero bytes

A floor statement whose action is outside 1 to 4, or whose to is not zero exactly when its action is give, is rd_statement_malformed. Its author, with t its event's t:

• request: a controller at t; it changes nothing, and tells the host and the holder that the author asks for the floor;
• give: the host, a room key, a principal whose grant in effect holds the direct power, or the floor's holder; to is a present controller at t, and not the host: otherwise floor_not_eligible;
• release: the floor's holder;
• take: the host, a room key or a principal whose grant in effect holds the direct power.

An author who may not send its action is floor_not_permitted, but a release by a principal that does not hold the floor is floor_not_holder. A floor statement whose host is not a principal that has signed a screen statement before it in the merged order is floor_not_permitted.

A floor statement these rules refuse at its place is not a refusal of the room. It is recorded: it stays in its author's chain, held to the room's rules as every event is (live-rooms §9.2), and it changes nothing: the floor in effect after it is the floor in effect before it. A verifier states it with its reason (§11.3) and computes the floor without it, and a host gives it no effect. So two floor statements that cross on the wire, such as a holder's release that takes its place after the host has taken the floor back, leave the floor where the statements these rules accept put it, and the room verifies.

7.3 The floor in effect

A host's floor starts held by no one. In the merged order, a give that §7.2 accepts passes it to to, and a release or a take that §7.2 accepts leaves it held by no one; a floor statement §7.2 refuses changes nothing. The holder also loses it, and it is held by no one, at the leave of the holder, at a grant_exchange that makes the holder's grant in effect one without the control power, and at the leave of the host. The floor in effect at a point of the merged order is the floor after every statement and lifecycle event before that point.

A floor is kept whatever the host's screen in effect states: a host that moves from open to floor and back keeps its floor's holder, and its rules (§5.3 step 3) read the floor only while its screen keeps one.

8. Input-to-photon latency

8.1 The latency record

field	size	meaning
magic, version	5	WDL1, 1
room	32	the room id
viewer	32	the key that signs it, its author's
host	32	the host
input	32	the digest of the event of the viewer's input statement
action	1	the index in that statement of the action measured, from 0
outcome	32	the digest of the event of the host's outcome of that input
track	32	the frame's track, as the outcome's anchor names it
group, object	8 each	u64: the frame's group and object
content	32	the frame's content hash, as its receipt states it
input_at	8	u64: the action's instant (§5.1)
presented_at	8	u64: the instant, on the viewer's estimate, the frame was presented
source	1	what reported presented_at: 1 the platform's frame-timing interface, 2 a light sensor watching the display, 3 no display: the instant the viewer held the frame verified against its receipt
latency_us	4	u32: presented_at − input_at
rtt_us	4	u32: the round trip of the clock measurement the viewer's estimate rests on (§8.4); 2^32 − 1 when it states none
sig	64	Ed25519 by viewer over the 19 bytes wai:remote-latency (ASCII) and 0x01, followed by the record from magic through rtt_us

A record whose source is outside 1 to 3 is rd_statement_malformed. Its digest, where a party names it apart from its event, is SHA-256 over the record from magic through rtt_us. It carries its own signature, beside its event's, so that it stands on its own: a viewer that shows a third party the latency it measured presents its records and its grant, with no other event of the room.

8.2 Rules

A latency record is checked, in this order:

1. its room is the room id: otherwise wrong_room;
2. its viewer is its event's principal: otherwise latency_wrong_key;
3. its signature verifies under viewer (§8.1): bad_key, bad_signature;
4. its input is the digest of an input statement the room accepted, by viewer, whose host is the record's host, with an action at index action: otherwise latency_unknown_input;
5. its outcome is the digest of the outcome of that input, which the room accepted, and which states it applied; its track, group and object are that outcome's anchor's, and its content that outcome's content; its input_at is the input's t less the action's dt_us: otherwise latency_mismatch;
6. its presented_at is at most its event's t: a record states what was presented, never what will be. Otherwise latency_ahead;
7. its presented_at is at least its input_at, and its latency_us is the difference: otherwise latency_mismatch.

8.3 What it measures

A latency record measures from an action to the presentation of the first frame that can show it. Both instants are on the viewer's own estimate, so the room clock's offset cancels from the difference; what remains is the estimate's drift over the interval, which the viewer keeps small by measuring its clock against a room key's as co-watch §7.3 has a player do. The first frame that can show an input is the first the host captured after injecting it (§6.1): the host's operating system and the application under the pointer may take longer to draw the effect, so the record is a lower bound on the time until the effect is seen, and is exact when they draw it in time for that frame.

A viewer SHOULD state as presented_at the instant its platform's frame-timing interface reports for the frame it drew: the presentation time of a video frame callback, or the start of the display refresh after the one in which the frame was drawn, never the instant it was decoded. A light sensor (source 2) watching the display measures the photon itself, and is the record a laboratory makes. A viewer with no display, such as an agent that reads the screen or a test harness, states source 3 and the instant it held the frame verified against its receipt: no photon is involved, and the record measures the path to the viewer alone.

A viewer measures the actions it chooses: every input, or one in so many. A record is a sample, and a verifier states each one; it states no percentile over a room, since which actions were measured is the viewer's choice.

8.4 The legs and their uncertainty

From a latency record and the outcome it names, a verifier states three legs:

• up: the outcome's applied_at less the record's input_at: from the action to its injection;
• host: the outcome's captured_at less its applied_at: from the injection to the capture of the first frame after it;
• down: the record's presented_at less the outcome's captured_at: from the capture to the presentation.

They sum to latency_us. The host leg is on the host's estimate alone and the total on the viewer's alone. The up and down legs cross from one estimate to the other, and each is stated with an uncertainty of half the sum of the two rtt_us, rounded down: the most a clock measured over a round trip of that length can be off, when the path's two directions take equal time. When either estimate states no round trip, the uncertainty is not stated, and the crossing legs are stated without it. A leg that crosses estimates may come out negative by up to its uncertainty; it is stated as it is.

9. Session records and metering

A remote desktop session is the room's session, and its records are the room's:

• the session: from the host's join to its leave, the clock's lifecycle (live-rooms §7.2), and in it every principal's presence, its grant in effect, and each grant exchange that handed control over;
• what happened: the host's screen statements, every input, its outcome and the floor, in the chains of their authors, which a verifier replays (§11) and states;
• what it cost: the host's screen track is a track of the room, and every hop that carried it signs its per-hop session record, as it does for every track; the room-session receipt totals them, its publisher-seconds and subscriber-seconds included (live-metering §3, §4). The input statements, the outcomes and the latency records are events of their authors, which live-metering counts as events sent and delivered. An action costs its share of an event's bytes; a frame its object's;
• what was kept: a recording of the session, when the room's recording policy records it (live-rooms §3.7), is a recording of the room (live-recording); its manifest holds the screen track and the events, so a recording of a session verifies as the session does.

This extension adds no record for any of these.

10. Where each rule is enforced

• The host applies only what §5.3 accepts, signs an outcome for every input it receives (§6), captures only what its screen statement states (§3.5), and names its encoder (§3.6). It is the rules' enforcement point: it is the one party whose machine an action reaches, and the one that holds the opening key of its sealing key (§5.5).
• A relay or a room service that carries the room's events carries each principal's events in its chain order, as it carries every event, and a sealed action as it is: it holds no opening key.
• A room service that keeps a session's records (§9, §11.1) SHOULD bound how many it keeps, and SHOULD bound them per principal first. A bound per principal falls on the principal that reaches it: the service keeps none of its records past the bound, cuts its chain there and removes it (live-rooms §7.2, leave reason 2), and the session goes on for everyone else; a host that knows the bound takes none of that principal's records past it either, so the host and the service hold the same records and the session verifies. The host itself is held to no bound of its own, since its records answer the others' inputs and grow with the room. A bound on the whole session is a safety net, set well above the bound per principal times a room's usual number of active principals, so that the bound per principal is the one any single principal meets first: when the session's bound is reached the service ends the session, every principal's leave signed, the host's last. Either way the records kept are whole chains, and a verifier accepts them. It MAY decline to forward an input statement to the host whose author carries no control power at its t (§5.3 step 2), which it can tell from the grants it holds; it then never reaches the host, and stays unanswered.
• The room authority signs the grants that mark the host and give the control power (§3.1), and exchanges them to hand control over (§7.1).
• Controllers send actions in the coordinates of the screen in effect, and send again what was refused once what refused it has changed.
• Viewers verify each frame against its receipt before they decode it (§3.2), and state latency as §8.3 has them.

After the fact, a verifier (§11) checks every one of these against the room: an input a host applied that §5.3 refuses, an outcome that states another reason than the rule that refuses, and a latency record that names another frame than the host's outcome, are each found out. A floor statement by a principal that may not send it is stated, and passes nothing (§7.2).

11. Verification

11.1 Inputs

A remote desktop verifier is given what a room verifier is given (live-rooms §9.1: a descriptor, grants, a key-log head or none, and the chains), and at the opened level the opening keys of the hosts' sealing keys (§11.4). Every statement this extension defines is in the chains.

11.2 Steps

It runs these steps and stops at the first refusal, which it reports with the reason and where:

1. The room (live-rooms §9): a refusal, at room with the room verifier's own place. A gap is not a refusal; the steps go on, and the checks that look up another statement (§6.2 steps 1 and 5, §8.2 steps 4 and 5) are not made for a statement whose lookup fails: the statement looked for may be in the gap.
2. The grants' attributes (§3.1), in the grants' order: at the grant's index.
3. The statements, in the merged order: each meta event the room accepted whose payload begins with a magic of §4.1: §4.2, then §4.3, then the rules of its section in the order the section gives, with the screen in effect, the floor in effect and each grant in effect at its place. A floor statement §7.2 refuses is not a refusal: it is stated (§11.3) and changes nothing. The checks of an outcome and of a latency record that look up another statement (§6.2, §8.2) are made after every statement has been read, since the statement they name may follow them in the merged order. At the record.
4. The outcome (§11.3).

11.3 Outcomes and what a verification states

• refuse: the reason and where (§11.2).
• gap: no refusal, and the room has a gap: the gaps, under the code delivery_gap, and what accept states, computed from what the verifier holds, with each outcome and latency record whose lookup failed stated unchecked.
• accept: no refusal and no gap.

With accept or gap, the verifier states:

• each screen statement, in the merged order: its digest, host, state, size, scope, floor, accepts, capability, codec, encoder, api and hardware;
• each input statement, in the merged order: its digest, author, host, the classes of its actions, the number of its sealed actions when it has any, and its outcome: applied (with the frame its outcome names), refused with the reason, or unanswered; and at the opened level, for a sealed input, what its seal gave (§11.4);
• the floor: each change of each host's floor in effect, in the merged order, with the instant and the holder, or none;
• each floor statement §7.2 refuses, in the merged order: its digest, author, host, t, action and reason (§12's third table);
• each latency record, in the merged order: its digest, viewer, source, latency_us, and its legs (§8.4) with their uncertainty, or unchecked;
• for each host, the count of inputs applied, refused for each reason of rules 1 to 7, and for rules 8 and 9 where that count is not zero, and unanswered.

11.4 Two levels

A remote desktop verification is made at one of two levels.

• The commitment level is what every party that holds the room verifies, with no key beyond the room's. It makes every step of §11.2 and reads no seal's plaintext: it makes rule 9 of §5.3 only where the screen statement a sealed input names states no sealing key (§6.2 step 5). It checks every statement's form, every outcome against rules 1 to 8, the floor, every latency record and every chain. A host that applies a key, text or clipboard action sent in the clear to a screen with a sealing key, or a sealed input to a screen with none, is found out at this level; one that applies a seal that does not open, or refuses one that does, is not.
• The opened level is a verification also given an opening key for each sealing key the room's screen statements state. For each sealed input it opens the seal with the opening key of the sealing key of the screen statement the input names, makes rule 9 in full, and requires every outcome to state what all nine rules give. It states what each sealed input's seal gave: opened, with the input statement as it reads opened (each sealed action replaced by the action it seals, and no seal); the reason rule 9 holds (§12's fourth table); or unopened, when it was given no opening key for that sealing key, and then that input is checked as at the commitment level.

The opened level is optional: the opening key is the host's to give, and a room verifies without it. A room the opened level accepts, the commitment level accepts.

12. Reasons

The codes this extension adds that refuse a room, each with where it is raised:

reason	raised at
rd_attr_malformed	§3.1, §11.2 step 2
rd_statement_malformed	§3.3, §4.2, §5.1, §5.2, §7.2, §8.1
rd_not_permitted	§4.3, §6.2 step 2
screen_wrong_track	§3.4 step 1
screen_unlinked	§3.4 step 2
outcome_unknown_input	§6.2 step 1
outcome_duplicate	§6.2 step 3
outcome_invalid	§6.2 step 4
outcome_mismatch	§6.2 step 5
latency_wrong_key	§8.2 step 2
latency_unknown_input	§8.2 step 4
latency_mismatch	§8.2 steps 5 and 7
latency_ahead	§8.2 step 6

The reasons a host states in a refused outcome, each with its number and the rule of §5.3 it names; a verifier refuses none of them, and states each input refused with them:

number	reason	rule
1	input_wrong_screen	§5.3 step 1
2	input_not_permitted	§5.3 step 2
3	input_no_floor	§5.3 step 3
4	input_screen_paused	§5.3 step 4
5	input_unsupported	§5.3 step 5
6	input_out_of_bounds	§5.3 step 6
7	input_stale	§5.3 step 7
8	input_unsealed	§5.3 step 8
9	input_bad_seal	§5.3 step 9

The reasons a floor statement is refused at its place (§7.2); a verifier refuses none of them, states each floor statement refused with them, and computes the floor without it (§7.3):

reason	raised at
floor_not_permitted	§7.2
floor_not_eligible	§7.2
floor_not_holder	§7.2

The reasons rule 9 of §5.3 holds for a seal (§5.5), which a verification at the opened level states of each sealed input it does not open (§11.4); a verifier refuses none of them:

reason	raised at
seal_no_key	§5.5
seal_not_opened	§5.5
seal_unreadable	§5.5
seal_commitment	§5.5

A remote desktop verifier also raises every code a room verifier raises (live-rooms §11 and the live-events codes it lists), at the room; wrong_room, bad_key and bad_signature for a latency record (§8.2); and delivery_gap for a gap, which is not a refusal.

13. Conformance

live-conformance/remote/ holds the corpus, in one file, vectors.json, written from fixed seeds by wai_remote_desktop_vectors gen. It has rooms, each a room (descriptor, grants, key-log head or none, chains) and the outcome §11.3 gives it with everything it states.

• Accepted: a session of a host that keeps a floor, a controller of pointer, keys and touch, a controller of every class but touch, a viewer with no control power, and a lead with the direct power: the host gives the floor and the holder's click and key are applied; the viewer's input is refused input_not_permitted and the second controller's input_no_floor; the second controller requests the floor, the host gives it, and its clipboard text is applied, while the first's input is refused input_no_floor; the holder releases it and the lead gives it back; the host resizes its screen, and an input in the old coordinates is refused input_wrong_screen; a touch it does not inject is refused input_unsupported, a pointer off the frame input_out_of_bounds, and an input decided too late input_stale; an input the host never answers is unanswered; it pauses, and an input is refused input_screen_paused; it resumes, and a wheel turn and composed text are applied; it takes the floor back; and latency records from both controllers, with their legs. And an open session in which control passes by grant exchange: the first controller's input before the exchange is applied and its input after it refused input_not_permitted, and the second controller's refused before it and applied after it.
• Accepted, with a floor statement §7.2 refuses: each reason of §12's third table at least once, each stated and changing nothing: the floor given by the viewer and given to the viewer, released by a controller that does not hold it, requested by the viewer, taken by a controller, and stated for a principal that is no host; and three whole floor sessions, each with the holder after the refused statement the one before it: a controller that does not hold the floor gives it to itself, a controller releases the floor another holds, and the holder's release takes its place after the host took the floor back, a race on the wire.
• Gap: the floor session with one record of a controller not delivered.
• Refused: each reason of §12's first table at least once, each vector a session of one defect: among them a host that applied an input that each rule of §5.3 refuses, a host that refused a good input or stated another rule than the one that refuses, an outcome that names no frame, and latency records naming another frame, signed by another key, or presented after they were signed.
• Sealed (§5.5), after every vector above, each stating what the opened level gives it under the corpus's opening key (opening_key) beside what the commitment level does: a session of a host that states a sealing key, in which keys, a composed text and a clipboard text are applied sealed, a text in the clear is refused input_unsealed, a text sealed to the viewer's key, one whose commitment is another text's, one whose opening does not read and another controller's copy of a sealed input are each refused input_bad_seal, and, on a screen with no sealing key, a sealed text is refused input_bad_seal and keys in the clear are applied; three sessions the commitment level accepts and the opened level refuses (the host applies the text sealed to the viewer's key, applies the text whose commitment is another's, refuses a seal that opens); and vectors both levels refuse (the host applies a text in the clear to a screen with a sealing key, applies a sealed text to a screen with none, refuses keys in the clear by rule 9; a ciphertext one byte short, a sealed action that seals a touch, a sealing key of 31 bytes). Every other vector holds no seal, and the opened level gives it what the commitment level does.

A conforming verifier gives every room the outcome, the reason, the place, and everything else the corpus states, at both levels. wai_remote_desktop_vectors verify and the test tests/remote_desktop.rs check that the file regenerates byte for byte, that wai-rs gives every room its outcome at both levels, that every reason of the four tables of §12 appears, and that §12 lists exactly the reference's codes. live-conformance/remote_desktop_verify.py is the second implementation, written from this text: it shares no code with wai-rs; it verifies each room with the live-rooms corpus's second implementation (live_rooms_verify.py), every signature with WAI's strict rule (claim-conformance/_ed25519.py), and every seal with the HPKE, X25519, HKDF and AES-128-GCM the aggregation corpus's second implementation writes from their texts (telemetry_aggregate_verify.py); and it mutates copies of the corpus and requires every copy to fail. The browser viewer's module (web/room/remote.js of joule-moq-web) writes the corpus's input statements, sealed ones among them, floor statements and latency records byte for byte, and gives every input of the corpus the outcome the corpus states.

14. Security and privacy considerations

• Input reaches a machine. An applied action types and clicks on the host as its user would. The host is the enforcement point (§10): it applies only what the grant in effect allows, at the input's place, and states every decision, so a host that applies more than the grant allows is found out by any verifier holding the room (§6.2 step 5). The authority decides who controls, in the grants it signs; the host decides what it injects at all (accepts) and how old an action may be (max_age_ms). A room key cannot send input (§4.3): the room's services never act on a host's machine.
• Every statement is signed by its author and rides in its author's chain. A relay can withhold an input but alter none, and a withheld input is a gap in its author's chain and is unanswered. A replayed input is a duplicate (live-events §6.2) and is decided once.
• What is captured is the host's. The screen statement states the scope, the redactions, whether the pointer is drawn and whether capture is paused (§3.5), and a sink shows them. A frame can hold whatever the host's screen holds at that instant, including personal data and credentials; a host that shares a window rather than a display, or that redacts, shares less. A room that needs its frames unreadable by its relays is confidential (confidentiality): its objects and events are sealed end to end, and the rules here are unchanged. A recording of the room keeps its events as they are, a sealed action sealed: the room's recording policy decides (live-rooms §3.7).
• What a controller types is sealed to a host that states a sealing key (§5.5): a relay, the room service, every viewer and a recording of the room hold a ciphertext and a commitment for each key, text and clipboard action, and only the holder of the opening key reads them. A commitment is salted, so it cannot be guessed from the few values a key takes, and it binds the input, its order and its author, so a host that opens a seal can show a third party one action's opening without the others, and that party checks it against the record. What stays in the clear is what rules 1 to 8 read and what the screen shows: that an input holds key, text or clipboard actions and how many, their instants, a clipboard text's length, and every pointer, wheel and touch action. The instants of keystrokes can say something of what was typed; a controller that needs them hidden sends text as one composed text action, whose length is hidden too. The host reads what it injects, as it must: a host is trusted with what is typed into it.
• The encoder's name is a statement. It says what made the frames, as the host reports it, and is not evidence of it; nothing is accepted or refused on it (§3.6). The receipts bind the bytes.
• A latency record is a viewer's claim: it proves that the holder of the viewer's key stated it, not that it is true, as a presentation statement of co-watch §14 is. A modified viewer can state any instant. The record names the frame by its content hash and the host's outcome, so it cannot claim a frame the host did not name, and its presentation can be no later than its own event (§8.2 step 6). A light sensor's record is as good as the sensor.
• Keys and timing. A controller's statements say when and where it acted, which is personal data, and to a host with no sealing key what it typed too. A principal that needs its sessions unlinkable across rooms uses a key per room, as live-events §11 has every principal do. Taking part in a room under a grant is taking part in its mechanics, and this extension adds no consent step to them. Where a regime asks the operator for a lawful basis for processing what a room carries, the operator records it as a lawful-basis claim (jwp-receipts §2.4.6).