Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions

wai: live-compose, a room's composition as an integer function (L3) #296

Merged
dcharlot merged 10 commits from wai/live-compose into main 2026-10-17 16:04:02 -04:00
Conversation 0 Commits 10 Files changed 21 +24099 -17
21 changed files with 24099 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

29
.github/workflows/wai-conformance.yml vendored
View file

@ -75,6 +75,9 @@ on:
# tests/capture_take.rs requires this extension's §12 to list the codes the reference
# reports, and the registry's wire identifiers test reads its sections.
- 'wai/extensions/capture-binding.md'
# tests/live_compose.rs requires this extension's §10 to list the codes the reference
# reports, and the registry's wire identifiers test reads its sections.
- 'wai/extensions/live-compose.md'
- 'wai/wai-web/**'
# The audio provenance check below reads the corpora above with this tool.
- 'wai/tools/wai_audio_provenance_check.py'
@ -121,6 +124,9 @@ on:
# tests/capture_take.rs requires this extension's §12 to list the codes the reference
# reports, and the registry's wire identifiers test reads its sections.
- 'wai/extensions/capture-binding.md'
# tests/live_compose.rs requires this extension's §10 to list the codes the reference
# reports, and the registry's wire identifiers test reads its sections.
- 'wai/extensions/live-compose.md'
- 'wai/wai-web/**'
# The audio provenance check below reads the corpora above with this tool.
- 'wai/tools/wai_audio_provenance_check.py'
@ -773,7 +779,8 @@ jobs:
# staged_verify.py (the staged-delivery corpus), and http_verify.py (the HTTP delivery
# corpus), and quantum-conformance/tools/referee.py (both quantum-sim capabilities), and
# live_events_verify.py (the live-events corpus), and live_rooms_verify.py (the
# live-rooms corpus), and capture_take_verify.py (the capture-binding corpus).
# live-rooms corpus), and capture_take_verify.py (the capture-binding corpus), and
# live_compose_verify.py (the live-compose corpus).
# Each exits non-zero on any failure or any difference, and the step fails with it.
# One platform: what they check is the reading of the text, not the architecture.
independent-verifiers:
@ -944,6 +951,26 @@ jobs:
(cd wai-rs && cargo test --no-default-features --features capture_take --lib -- capture_take:: output_tree::)
(cd wai-rs && cargo test --no-default-features --features capture_take --test capture_take)
python3 -I live-conformance/capture_take_verify.py live-conformance/takes
# wai.live.compose (extensions/live-compose.md): the reference module's unit tests (the
# layout over every grid of 1 to 64 tiles stays on its canvas with no tile overlapping
# another, the fill modes, the attributes, the mixer) and tests/live_compose.rs (the
# committed corpus regenerates byte for byte from its fixed seeds, wai-rs gives every
# case its stated outcome, every reason of §10 refuses a case, and §10 lists exactly
# the reference's codes); then the corpus's second implementation,
# live_compose_verify.py, written from the extension's text and sharing no code with
# wai-rs, which verifies each room with live_rooms_verify.py, steps through its
# lifecycle replay to each instant itself, gives every layout, mix, descriptor,
# composition and receipt the same outcome rectangle for rectangle and sample for
# sample, and requires every mutated copy of the corpus to fail; standard library
# only, run isolated (-I).
- name: live compose · wai-rs and live_compose_verify.py
working-directory: wai
shell: bash
run: |
set -euo pipefail
(cd wai-rs && cargo test --no-default-features --features live_compose --lib live_compose::)
(cd wai-rs && cargo test --no-default-features --features live_compose --test live_compose)
python3 -I live-conformance/live_compose_verify.py live-conformance/compose
# The quantum corpus's second implementation (quantum-sim extension), integers only:
# it derives the circuit/1 phase table by its recurrence and the circuit2 table by an
# interval-certified procedure, and requires both literal tables and their digests,

39
wai/CHANGELOG.md
View file

@ -6,6 +6,45 @@ wire format may change and such changes are recorded here.
## v0.4.0 — current
### `wai.live.compose`: a room's composition as an integer function
- **The extension** ([live-compose](extensions/live-compose.md)) registers the record
`wai.live.compose`. A `WCD1` composition descriptor, signed by the room authority or by a
principal whose grant has the compose power, fixes a canvas (each edge even, 16 to 1920
pixels, at most 1920 × 1080 pixels, 1 to 60 frames a second), an sRGB background, the
highest layer composed, a layout, the grant attributes it reads, and its limits on tiles
and voices.
- **The participant set** at a room-clock instant is the room's lifecycle replay up to it:
every present publisher but the compositor, its video source (own video, own screen, or
its live place in a slot) and audio source, each with the dimensions of the composed
layer, the attributes of its grant in effect, and its join order. A gap in a room key's
chain is `lifecycle_incomplete`; a gap elsewhere changes nothing. The set has a digest.
- **The layout** (§5) is integer arithmetic throughout, with every rounding stated: a grid
chosen by the largest tile of the reference aspect among every column count, its rows
centred; tile aspects auto, 16:9, 1:1 and 3:4; a featured participant above the others;
fill, cover and contain, by default cover for grid tiles and contain for the featured
one; stopped video shown or omitted; order by attribute, then join order, then key; and
picture in picture, the PiP participant's window over the grid in any corner, of a set or
derived size, static or dynamic (alone, it is laid out as though it had no PiP
attribute). The layout is computed again at every lifecycle event and grant exchange.
- **The mix** (§6): per-voice gain and pan as integers in units of 1/4096, over signed
16-bit PCM, summed exactly in 64 bits and saturated once, so it is byte-exact over the
decoded PCM; a decode and a lossy re-encode are not, and the receipt binds objects.
- **Where it runs** (§7): on each sink by default, with no server encode; a server
compositor must be the descriptor's compositor, hold the compose power and sign a `WCR1`
receipt for each object of a composed track, naming the descriptor, the set's digest at
`t`, the output object and the input objects of exactly the tracks the layout reads.
- **Live rooms.** §1, §3.3 and §4.4 point at the extension for the compose power and the
attributes `featured`, `order` and `pip` (and `gain` and `pan`, which it adds).
- **Reference and corpus.** `wai-rs` feature `live_compose` (`src/live_compose.rs`; the
layout and the mixer in `src/live_compose/layout.rs` use nothing beyond `core`, `Vec` and
`String`); `live-conformance/compose/`, written from fixed seeds by
`wai_live_compose_vectors`, and checked by `tests/live_compose.rs` and by
`live-conformance/live_compose_verify.py`, a second implementation that shares no code
with `wai-rs`, steps through the room's lifecycle replay itself and requires every
mutated copy of the corpus to fail. The registry lists the record and its magics (`WCD1`,
`WCR1`).
### `wai.capture.take`: a device's master recording bound to a live room
- **The extension** ([capture-binding](extensions/capture-binding.md)) registers the

9
wai/REGISTRY.md
View file

@ -40,7 +40,7 @@ Every registered string has one kind.
- **record** — a signed record a WAI tool emits and verifies (a receipt kind:
`wai.asset.provenance`, the quantum-operations receipts, the delivery records
`wai.video.manifest` and its siblings; the live records `wai.live.events`,
`wai.live.room` and `wai.capture.take`).
`wai.live.room`, `wai.capture.take` and `wai.live.compose`).
It names no payload format.
A derivation or record string is not a payload capability, so it is not
@ -62,8 +62,8 @@ bindings ([http-delivery](extensions/http-delivery.md),
[moq-streaming-format](extensions/moq-streaming-format.md)), the key discovery
they serve ([jwp-receipts](extensions/jwp-receipts.md) §7),
[packaging](extensions/packaging.md), [live-events](extensions/live-events.md),
[live-rooms](extensions/live-rooms.md) and [capture-binding](extensions/capture-binding.md)
put on the wire outside a
[live-rooms](extensions/live-rooms.md), [capture-binding](extensions/capture-binding.md)
and [live-compose](extensions/live-compose.md) put on the wire outside a
manifest — magic numbers, a media type, `codecs` prefixes, paths, playlist tags and
attributes, document kinds and catalog fields — each with its kind, the section that
defines it, and whether it is defined, provisional (test material only until
@ -192,6 +192,7 @@ and a role are fixed once registered.
| `wai.image.jpegai` | payload | registered | image | standalone | royalty-free | neural-entropy-exact | — | §5 Neural capabilities |
| `wai.image.jxl` | payload | registered | image | standalone | royalty-free | standard-tolerance | recommended | §5 Recommended modern set |
| `wai.image.png` | payload | registered | image | standalone | royalty-free | standard-defined | floor | §5 Mandatory floor |
| `wai.live.compose` | record | registered | — | — | — | — | — | §5 Interactive, temporal & computed content |
| `wai.live.events` | record | registered | — | — | — | — | — | §5 Interactive, temporal & computed content |
| `wai.live.room` | record | registered | — | — | — | — | — | §5 Interactive, temporal & computed content |
| `wai.mesh.draco` | payload | registered | volumetric | standalone | royalty-free | standard-unclassified | — | §5 Immersive audio, cinema, volumetric, haptics, HDR |
@ -288,7 +289,9 @@ and a role are fixed once registered.
| `WAI1` | magic | defined | SPEC §2 |
| `WAI2` | magic | defined | SPEC §7.1 |
| `WAI_RECEIPT` | catalog | defined | moq-streaming-format §5 |
| `WCD1` | magic | defined | live-compose §3.1 |
| `WCN1` | magic | defined | capture-binding §8.1 |
| `WCR1` | magic | defined | live-compose §8.1 |
| `WEP1` | magic | defined | live-events §5 |
| `WEV1` | magic | defined | live-events §3.1 |
| `WHP1` | magic | defined | http-delivery §6.2 |

19
wai/SPEC.md
View file

@ -592,7 +592,8 @@ kind:
`wai.asset.provenance`, `wai.quantum.calibration`, `wai.quantum.job`, and the
delivery records `wai.video.manifest`, `wai.video.ssai`, `wai.video.keys`,
`wai.video.package` and `wai.video.channel`; and the live records `wai.live.events`,
`wai.live.room` and `wai.capture.take`), which names no payload format.
`wai.live.room`, `wai.capture.take` and `wai.live.compose`), which names no payload
format.
Only payload, companion and refinement capabilities have a media class, a
fidelity tier, a replicate class and a role. A derivation or record string is not a payload
@ -894,6 +895,22 @@ at its index), `aligned` (capture began at the count-in's instant), `complete`,
`publishable` for each purpose. The take maps onto C2PA live video output:
[`extensions/capture-binding.md`](extensions/capture-binding.md).
And for the **composition** of a room — `wai.live.compose`: a signed composition
descriptor, by the room authority or a principal whose grant has the compose power, that
fixes a canvas (at most 1920 pixels an edge and 1920 × 1080 pixels, at 1 to 60 frames a
second), a background, and a layout: a grid, with a featured participant above the
others, a gap, a tile aspect, a fill mode (fill, cover or contain), an order by
attribute and stopped video shown or omitted; or the same grid under a picture-in-picture
window in a corner, of a set or derived size, static or dynamic. The participant set at
each instant follows from the room's lifecycle replay: who publishes, each one's source
and its dimensions, the attributes of its grant in effect and its join order. The layout
is a pure integer function of the descriptor and the set, laid out again at every
lifecycle event and grant exchange, and the mix a saturating integer function of decoded
PCM, so every sink composes the same rectangles locally; a server compositor under the
compose power signs a receipt naming the descriptor, the set and the input and output
objects, which anyone checks against the same function:
[`extensions/live-compose.md`](extensions/live-compose.md).
### Payloads of the WAI-native integer codecs
This revision pins a payload format to each WAI-native integer codec whose row named

592
wai/extensions/live-compose.md Normal file
View file

@ -0,0 +1,592 @@
# WAI Extension: Live Composition
> Status: Draft. Registers the record `wai.live.compose`: the deterministic composition
> of a live room's participants. A signed **composition descriptor** fixes a canvas, a
> background, a layout (a grid, with a featured tile, or a grid under a
> picture-in-picture window), the grant attributes it reads and its limits. The
> **participant set** at a room-clock instant is derived from the room's lifecycle
> replay ([live-rooms](live-rooms.md) §9): who publishes, each one's video and audio
> source and its dimensions, the attributes of its grant in effect, and its join order.
> The layout is then a pure integer function of the descriptor and the set, and the mix
> a pure integer function of decoded PCM, so every sink composes the same rectangles
> locally, with no server encode, and a server compositor that publishes a composed track
> signs a **composition receipt** that anyone can check against the same function. No new
> transport: a composed track is a track of the room, and the receipt names the objects it
> was made from. The reference implementation is the `live_compose` module of `wai-rs`
> (feature `live_compose`); the corpus is
> [`live-conformance/compose/`](../live-conformance/compose/), and its second
> implementation is `live-conformance/live_compose_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision: `draft-ietf-moq-transport-22` (MOQT), an Internet-Draft
> not past Working Group Last Call, for the Full Track Name (§2.4.1) and the Location
> (§8.2); RFC 6716 §6; RFC 8032; FIPS 180-4; and IEC 61966-2-1 (sRGB). A section number of
> that draft in this text is that revision's.
## 1. Scope
In scope: the composition descriptor, its digest and signature, its limits and who may
sign it (§3); the participant set at an instant, the attributes the descriptor reads, the
order, and the set's digest (§4); the layout function (§5); the audio mix (§6); where
composition runs (§7); the composition receipt and its verification (§8, §9).
Out of scope: a transport; how a renderer scales and filters a source into its rectangle,
and how it converts the background's colour into its output's (the layout fixes
rectangles, not pixels); decoding, resampling and encoding; and the browser compositor
that renders a layout in a page, which a later slice builds from this function.
Relationship to the core spec: `wai.live.compose` is a **record** ([SPEC](../SPEC.md) §5):
a signed record a WAI tool emits and verifies. It names no payload format and is not
dispatchable.
## 2. Terms
- **Room**, **room clock**, **principal**, **event**, **chain**, **grant**, **grant in
effect**, **room key**, **merged order**, **slot**, **publish profile**: as in
live-events §2 and live-rooms §2, §8, §9.2.
- **Room authority**: the key that signs the room's descriptor (live-rooms §2).
- **Compose power**: bit 2 of a capability set's `powers` (live-rooms §3.3).
- **Canvas**: the picture a composition makes, `width` × `height` pixels.
- **Source**: the track a participant is composed from (§4.2).
- **Compositor**: the principal that publishes a composed track (§7).
- **Rectangle**: (`x`, `y`, `w`, `h`): its left edge and top edge, in pixels from the top
left of the canvas (or of a source), and its width and height.
- A **short** field is a u8 length and that many bytes. Integers are little-endian. A
digest is SHA-256 (FIPS 180-4), 32 bytes.
## 3. The composition descriptor
### 3.1 Layout
| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WCD1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id (live-rooms §3.2) |
| `signer` | 32 | the Ed25519 key that signs the descriptor (§3.5) |
| `effective` | 8 | u64: the room-clock instant from which the descriptor applies |
| `compositor` | 32 | the principal that may publish the composed tracks (§7); 32 zero bytes for none |
| `width`, `height` | 2 each | u16: the canvas, in pixels (§3.4) |
| `fps` | 1 | the canvas's frame rate, in frames per second (§3.4) |
| `background` | 3 | the canvas's colour: red, green and blue, 8 bits each, in sRGB (IEC 61966-2-1) |
| `layer` | 1 | the highest layer of a source composed (§4.2) |
| `layout` | 1 | 1 grid, 2 picture in picture (§5) |
| `gap` | 2 | u16: the pixels between two grid tiles |
| `aspect` | 1 | the grid tiles' aspect: 0 auto, 1 video (16:9), 2 square (1:1), 3 portrait (3:4) (§5.3) |
| `fill` | 1 | 0 default, 1 fill, 2 cover, 3 contain (§5.6) |
| `featured_attr` | short | the attribute key that marks the featured participant; empty for none (§4.3) |
| `order_attr` | short | the attribute key that orders participants; empty for none |
| `omit_stopped` | 1 | 1: a participant whose video is stopped has no tile; 0: it has one (§5.2) |
| `pip_attr` | short | the attribute key that marks the PiP participant; empty for none |
| `corner` | 1 | the PiP window's corner: 1 top left, 2 top right, 3 bottom left, 4 bottom right |
| `pip_width`, `pip_height` | 2 each | u16: the PiP window's size; 0 derives it (§5.5) |
| `offset` | 2 | u16: the window's distance, in pixels, from both edges of its corner |
| `behaviour` | 1 | what the PiP participant does when it is the only one composed: 1 static, 2 dynamic (§5.5) |
| `channels` | 1 | the mix's channels: 1 mono, 2 stereo (§6) |
| `sample_rate` | 4 | u32: the rate, in Hz, every voice is decoded to before it is mixed |
| `gain_attr` | short | the attribute key of a participant's gain; empty for none (§6) |
| `pan_attr` | short | the attribute key of a participant's pan; empty for none |
| `max_tiles` | 1 | the most tiles (§5.2) |
| `max_voices` | 1 | the most voices of the mix (§6) |
| `sig` | 64 | §3.2 |
A grid's `pip_attr` is empty and its `corner`, `pip_width`, `pip_height`, `offset` and
`behaviour` are 0 (§3.3).
### 3.2 Digest and signature
The **digest** of a descriptor is SHA-256 over its record from `magic` through
`max_voices`: every byte but `sig`. The **signature** is Ed25519 (RFC 8032) by `signer`
over the 17 bytes `wai:live-compose` (ASCII) and 0x01, followed by the bytes the digest
covers. It is verified strictly, as [SPEC](../SPEC.md) §7 *Signatures* requires: a key
that fails step 1 or 2 is `bad_key`, and a signature that fails a later step is
`bad_signature`.
### 3.3 Reading and field rules
A reader reads exactly the bytes it is given as one descriptor, in the order of §3.1, and
refuses at the first of these that holds:
1. Fewer than 4 bytes: `truncated`. A magic other than `WCD1`: `malformed`. No byte after
the magic: `truncated`. A version other than 1: `unknown_version`.
2. The bytes end before a field the layout requires: `truncated`.
3. Bytes after `sig`: `malformed`.
It then holds the canvas to §3.4, and refuses one beyond its limits as
`canvas_out_of_range`. It then refuses, as `compose_invalid`, a descriptor that breaks any
of these rules:
- `layout` is 1 or 2, `aspect` at most 3, `fill` at most 3, and `omit_stopped` 0 or 1.
- Each attribute key is empty or a slot name of live-rooms §2 (1 to 32 bytes of `a`–`z`,
`0`–`9` and `_`), which is what a grant attribute's key is (live-rooms §4.4).
- A grid has an empty `pip_attr`, and `corner`, `pip_width`, `pip_height`, `offset` and
`behaviour` 0. A picture in picture has a `corner` of 1 to 4 and a `behaviour` of 1 or 2,
an `offset` below both `width` and `height`, `pip_width` + `offset` at most `width`,
and `pip_height` + `offset` at most `height`.
- `channels` is 1 or 2, and `sample_rate` 8 000 to 192 000.
- `max_tiles` and `max_voices` are each 1 to 64.
- With *k* the least integer whose square is at least `max_tiles`: `gap` is below
`height`; `width` is at least *k* + (*k* − 1) × `gap`; and the band under a featured
tile (§5.4), `height` − `gap` − ⌊(`height` − `gap`) × 3 / 4⌋, is at least *k* +
(*k* − 1) × `gap`. Every grid the layout makes then has a candidate whose cells are at
least a pixel each way (§5.3).
Then the signature (§3.2): `bad_key` or `bad_signature`.
### 3.4 The canvas
`width` and `height` are each even, and each 16 to 1920; `width` × `height` is at most
2 073 600, the pixels of 1920 × 1080; and `fps` is 1 to 60. The edges are even so that a
picture subsampled 4:2:0 has whole chroma samples. A portrait 1080 × 1920 and a square
1440 × 1440 canvas are within these limits.
### 3.5 Who may sign
Against a room, a descriptor that reads (§3.3) is refused as `wrong_room` when its `room`
is not the room id; and as `compose_not_permitted` unless its `signer` is the room
authority, or a principal present at `effective` whose grant in effect then (§4.1) has the
compose power.
The power is checked at `effective`, when the descriptor takes effect: a descriptor does
not lapse when its signer leaves or later loses the power. A composition has one
descriptor; a room that changes its layout signs another, and each composed track and
each receipt names the one it follows (§8).
## 4. The participant set
### 4.1 The room at an instant
A composition reads a room that verifies (live-rooms §9): one refused is refused here
with the room's reason, at `room`. A room whose outcome is a gap is composed only when no
gap is in a room key's chain: the lifecycle is then whole, and a gap in another
principal's chain (a chat not delivered) changes no room state. A gap in a room key's
chain is `lifecycle_incomplete`, at `room`: the state at no instant after it is known.
The **room at** *t* is the room state (live-rooms §9.3) replayed from the lifecycle events
whose `t` is at most *t*, in the merged order: equally, the room each chain gives when it
is cut before its first record whose `t` is above *t*. Events at *t* itself are applied,
so a principal that joins at *t* is in the set at *t*.
The **join order** of a principal present at *t* is the number of the `join` that made it
present (its last), where the room's accepted `join` events up to *t* are numbered from 0
in the merged order, a duplicate (live-events §6.2 step 1) counted once.
### 4.2 Participants and their sources
The **participants** at *t* are the principals present at *t*, other than the
descriptor's `compositor`, that publish at *t*: each has an own track of any media kind
published, or is the live publisher of a slot (live-rooms §7.1). A standby is not
composed for its place in a slot, since its media is not delivered.
Each participant has:
- its **video source**: the first of its own `video` track; its own `screen` track; and
its live place in a slot of media kind `video` or `screen`, the first such slot in the
room descriptor's order. With none, its video is **stopped**.
- its **audio source**: its own `audio` track; or else its live place in the first slot of
media kind `audio`. With none, it has no voice.
- the attributes of its grant in effect at *t* (live-rooms §4.4).
A source's **profile** is the profile its track was published under (the `publish` or
`takeover` that made it, live-rooms §7.2), and its **layers** the number of layers
published. Its **composed layer** *L* is the lower of `layers` − 1 and the descriptor's
`layer`, and its width and height are those of layer *L* of the profile (live-rooms §8.1);
an audio source's are 0. A descriptor whose `layer` is below a publisher's top layer
composes from a layer the compositor's grant can subscribe to, and every sink lays the
room out in that layer's pixels.
### 4.3 Attributes
For each participant, each attribute key the descriptor names (not empty) is looked up
among the attributes of its grant in effect, and read:
| key | value | when absent |
|---|---|---|
| `featured_attr`, `pip_attr` | `true` or `false` | `false` |
| `order_attr` | a decimal integer from −2 147 483 648 to 2 147 483 647 | no order |
| `gain_attr` | a decimal integer from 0 to 16 384 | 4096 |
| `pan_attr` | a decimal integer from −4096 to 4096 | 0 |
A **decimal integer** is `0`, or an optional `-` followed by 1 to 10 of the digits `0`–`9`
of which the first is not `0`: no `+`, no space, no `-0`. Every attribute the descriptor
names is read for every participant, whether or not it has the source the attribute is
about; one whose value is not of its row is `attr_malformed`, and the composition at *t*
is refused, at `set`. The keys `featured`, `order` and `pip` that live-rooms §4.4 names
are these attributes under the keys a descriptor gives them; `gain` and `pan` are the
mix's.
### 4.4 Order
The participants are ordered: those with an order first, by order ascending, then those
with none; between two of one order (or none), by join order ascending; and between two
of one join order, by key, as unsigned bytes. This order is **S**.
### 4.5 The set's digest
The digest of the participant set at *t* is SHA-256 over the 21 bytes
`wai:live-compose-set` (ASCII) and 0x01, the room id, *t* (u64), the number of
participants (u32), and then, for each participant in ascending order of key:
| field | size | meaning |
|---|---|---|
| `principal` | 32 | its key |
| `grant` | 32 | the digest of its grant in effect |
| `join` | 4 | u32: its join order |
| `video`, `audio` | 1 or 9 + *s* each | its source: one byte 0 for none; otherwise `media` (u8, the media kind of live-rooms §6.1), `slot` (short: empty for an own track), `profile` (u8, the profile's index in the room descriptor), `layers` (u8), the composed layer *L* (u8), `width` and `height` (u16 each) |
The grant's digest fixes its attributes, so the set's digest fixes everything §5 and §6
read of the room at *t*.
## 5. The layout
### 5.1 Integers
Every quantity of this section is an integer, and none is negative. ⌊*a* / *b*⌋, for *b*
above 0, rounds down. **fit**(*bw*, *bh*, *a*, *b*), the largest rectangle of aspect
*a*:*b* within *bw* × *bh*, is (*bw*, max(1, ⌊*bw* × *b* / *a*⌋)) when *bw* × *b* ≤ *bh*
× *a*, and (max(1, ⌊*bh* × *a* / *b*⌋), *bh*) otherwise. A rectangle of *w* × *h*
**centred** in (*x*, *y*, *W*, *H*) is (*x* + ⌊(*W* − *w*) / 2⌋, *y* + ⌊(*H* − *h*) / 2⌋,
*w*, *h*). No product here exceeds 2^63.
### 5.2 Which participants have tiles
From S (§4.4):
1. **V** is S, less every participant whose video is stopped when `omit_stopped` is 1.
2. For a picture in picture, the **PiP participant** P is the first of V whose `pip_attr`
attribute is `true`; with none there is no PiP window. For a grid there is none.
3. **G** is V without P, in order. The **featured participant** F is the first of G whose
`featured_attr` attribute is `true`; it is moved to the front of G.
4. G is cut to its first `max_tiles` members, less one when there is a P. F, being first,
is kept; when G is then empty there is no F.
### 5.3 The grid
A grid of *n* ≥ 1 tiles over a region (*X*, *Y*, *W*, *H*) is chosen by its **reference
aspect** *a*:*b*: 16:9 for auto and video, 1:1 for square, 3:4 for portrait.
- **Candidates.** For each number of columns *c* from 1 to *n*, with *r* = ⌈*n* / *c*⌉
rows: *c* is a candidate when (*c* − 1) × `gap` is below *W*, (*r* − 1) × `gap` below
*H*, and the cells, *cw* = ⌊(*W* − (*c* − 1) × `gap`) / *c*⌋ wide and *ch* = ⌊(*H* −
(*r* − 1) × `gap`) / *r*⌋ high, are each at least 1. Its **area** is *tw* × *th*, where
(*tw*, *th*) = fit(*cw*, *ch*, *a*, *b*).
- **The choice.** The candidate of the largest area; of several, the one with the most
columns. §3.3's rule on `gap` makes *c* = ⌈√*n*⌉ a candidate for every *n* a
descriptor allows.
- **Placement.** The rows are centred: the first row's top is *y*0 = *Y* + ⌊(*H* − (*r* ×
*ch* + (*r* − 1) × `gap`)) / 2⌋. Tile *k*, from 0, is in row *i* = ⌊*k* / *c*⌋ and column
*j* = *k* mod *c*. Every row holds *c* tiles but the last, which holds *n* − (*r* − 1) ×
*c*; a row of *m* tiles is centred, beginning at *x*0 = *X* + ⌊(*W* − (*m* × *cw* + (*m* −
1) × `gap`)) / 2⌋. Cell *k* is (*x*0 + *j* × (*cw* + `gap`), *y*0 + *i* × (*ch* + `gap`),
*cw*, *ch*).
- **The tile.** Under auto, tile *k* is its cell; otherwise it is fit(*cw*, *ch*, *a*, *b*)
centred in its cell.
With no tile the canvas is its background.
### 5.4 The featured tile
With F and *n* ≥ 2 members of G, the **featured tile** is (0, 0, `width`, *mh*), with *mh*
= ⌊(`height` − `gap`) × 3 / 4⌋, and the other *n* − 1 members are a grid (§5.3) over the
band below it, (0, *mh* + `gap`, `width`, `height` − *mh* − `gap`). The featured tile is its
whole region: `aspect` is the grid tiles'. With F alone the featured tile is the whole
canvas. With no F, the members of G are a grid over the whole canvas.
### 5.5 Picture in picture
With P and a G that is not empty, G is laid out over the canvas as §5.3 and §5.4 lay it out,
and P's **window** is drawn over it. With P and an empty G:
- **static**: P's window, and the background;
- **dynamic**: P is laid out as though it had no `pip_attr` attribute: a grid of P alone,
or, when P is featured, the featured tile, the whole canvas.
The window is `pip_width` × `pip_height` when both are above 0. Otherwise, with *a*:*b* the
width and height of P's video source, or the canvas's when P's video is stopped, it is
fit(*bw*, *bh*, *a*, *b*), where *bw* is `pip_width` when that is above 0, `width` −
`offset` when only `pip_height` is, and the lower of ⌊`width` / 4⌋ and `width` − `offset`
when neither is; and *bh* is `pip_height` when that is above 0 and `height` − `offset`
otherwise. With *o* the `offset` and *pw* × *ph* the window, it is at (*o*, *o*) in the top
left corner, (`width` − *o* − *pw*, *o*) in the top right, (*o*, `height` − *o* − *ph*) in
the bottom left, and (`width` − *o* − *pw*, `height` − *o* − *ph*) in the bottom right.
### 5.6 Fill
A tile (*tx*, *ty*, *tw*, *th*) shows a source of *sw* × *sh* by its fill mode. The source
is **wider** when *sw* × *th* ≥ *sh* × *tw*.
- **Fill**: the whole source, (0, 0, *sw*, *sh*), stretched over the whole tile.
- **Cover**: the source cropped to the tile's aspect, over the whole tile. A wider source's
crop is *cw* = max(1, ⌊*sh* × *tw* / *th*⌋) wide, at (⌊(*sw* − *cw*) / 2⌋, 0, *cw*, *sh*);
any other's is *ch* = max(1, ⌊*sw* × *th* / *tw*⌋) high, at (0, ⌊(*sh* − *ch*) / 2⌋, *sw*,
*ch*).
- **Contain**: the whole source, letterboxed in the tile. A wider source is drawn into
(*tx*, *ty* + ⌊(*th* − *dh*) / 2⌋, *tw*, *dh*) with *dh* = max(1, ⌊*tw* × *sh* / *sw*⌋);
any other into (*tx* + ⌊(*tw* − *dw*) / 2⌋, *ty*, *dw*, *th*) with *dw* = max(1, ⌊*th* ×
*sw* / *sh*⌋).
- **Default**: cover for a grid tile and the PiP window, contain for the featured tile.
Each mode gives a **source rectangle** and a **destination rectangle**: the part of the
source drawn, and where on the canvas it is drawn. A tile of stopped video has neither, and
shows the background, as does the part of a tile outside its destination. How the source
rectangle is scaled into the destination is the renderer's; a sink that holds a source at a
layer below its composed one scales it to the composed layer's width and height first.
### 5.7 What the layout gives
The layout of a descriptor and a participant set gives:
- **the order**: S, every participant, by key;
- **the tiles**, in drawing order: the featured tile, then the grid's in G's order, then
the PiP window. Each has its participant, its **place** (`grid`, `featured` or `pip`),
its **z** (0, or 1 for the PiP window, drawn over the others), the tile, and, unless its
video is stopped, its source and destination rectangles;
- **the voices** (§6).
A renderer paints the background over the canvas, then the tiles in order. No two tiles
of z 0 overlap.
### 5.8 Re-layout
The participant set changes only at the `t` of a lifecycle event: a `join`, `leave`,
`publish`, `unpublish`, `grant_exchange`, `takeover` or `failover` (an exchange always
changes it, since the set names each participant's grant). At every such instant at or
after `effective` the layout is computed again, and it changes when the sources, the
attributes or the order do; between two such instants it does not change. The layout at *t*
is the function of this section over the descriptor and the set at *t*, and of nothing a
sink sees otherwise, so a sink that joins late lays the room out as one that was there
throughout, and a grant exchange that changes an attribute re-lays the room out at the
exchange's tick, with no participant leaving.
## 6. The audio mix
The **voices** are the members of S (before any is omitted for its video) that have an
audio source, in S's order, the first `max_voices` of them. Each has its **gain** *g*,
from 0 to 16 384 in units of 1/4096 (4096 is unity), and its **pan** *p*, from −4096
(left) to 4096 (right) (§4.3).
The mixer mixes one block of *N* frames. Each voice's input is its decoded audio for the
block: *N* frames of signed 16-bit samples, of 1 or 2 channels (interleaved, left first),
at the descriptor's `sample_rate`. Each output sample is a sum over the voices in signed
64-bit integers, one term per voice:
- **mono output** (`channels` 1): *x* × 4096 × *g* for a mono voice's sample *x*; (*x*L +
*x*R) × 2048 × *g* for a stereo voice's samples *x*L and *x*R.
- **stereo output** (`channels` 2), with *a*L = 4096 − max(*p*, 0) and *a*R = 4096 +
min(*p*, 0): to the left channel *x* × *g* × *a*L and to the right *x* × *g* × *a*R for
a mono voice; *x*L × *g* × *a*L and *x*R × *g* × *a*R for a stereo one.
The output sample is ⌊(*s* + 2^23) / 2^24⌋ of the sum *s*, the arithmetic shift right by
24 bits of *s* + 2^23, which rounds half up; then saturated once to −32 768 … 32 767. A
term is at most 2^41 in magnitude and a mix has at most 64 voices, so the sum is exact,
the order of its terms changes nothing, and saturation applies to the whole once. The pan
is a balance: at the centre both channels are at the voice's gain, and panning attenuates
one channel only. A single voice at unity gain and the centre is passed through
unchanged. A mixer that holds no decoded block of a voice for the output mixes it as
silence.
A mixer is refused as `mix_input_malformed` when an output has other than 1 or 2
channels, or an input has other than 1 or 2 channels, has other than *N* frames, or a
gain or pan out of range.
The mix is byte-exact over the decoded PCM: two mixers given the same blocks give the
same samples. It is not byte-exact over what precedes and follows it. A decoder of a lossy
codec need not reproduce another decoder's PCM bit for bit (RFC 6716 §6 holds a decoder to
a quality measure, not to equality), and a lossy encode of the mix is not
byte-reproducible: two encoders, or one encoder under two settings, give different bytes.
The receipt (§8) therefore binds the input objects and the output object, not samples.
## 7. Where composition runs
**On each sink, by default.** A sink composes the room locally from the tracks it
subscribes to, by §5 and §6, with no server decode or encode. It needs no power: the
descriptor states how the room is laid out, and laying it out is reading it.
**On a server, under the compose power.** A principal that publishes a composed track (its
own `video` track for a layout, its own `audio` track for a mix, live-rooms §6.1) **MUST**
be the descriptor's `compositor`, **MUST** hold the compose power in its grant in effect,
and **MUST** sign a receipt (§8) for each object of a composed track it publishes. It is
not a participant of its own composition (§4.2), so a composed track is never composed
again. A subscriber subscribes to a composed track as to any other; the descriptor and the
receipts let it check that the track is the room's composition.
## 8. The composition receipt
### 8.1 Layout
| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WCR1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id |
| `compositor` | 32 | the compositor's key, which signs the receipt |
| `descriptor` | 32 | the digest of the descriptor it follows (§3.2) |
| `t` | 8 | u64: the room-clock instant of the output object |
| `state` | 32 | the digest of the participant set at `t` (§4.5) |
| `media` | 1 | 1 audio (a mix), 2 video (a layout) |
| `output` | 80 | the output object |
| `inputs` | 1 + 112 × *n* | a u8 count *n*, then each input: `principal` (32), the participant it is read from, and an object (80) |
| `sig` | 64 | Ed25519 by `compositor` |
An **object** is 80 bytes: `track` (32), the digest live-events §3.1 gives an anchor's
`track`, over the track's namespace as live-rooms §6.2 renders it, one zero byte, and its
name; `group` and `object` (u64 each), its MOQT Location (MOQT §8.2); and `digest` (32),
the SHA-256 of the object's payload. An input whose `digest` is 32 zero bytes states that
the compositor held no object of that track for the output: it drew the tile with no
video, or mixed the voice as silence.
The **digest** of a receipt is SHA-256 over its bytes before `sig`; its **signature** is
Ed25519 by `compositor` over the 25 bytes `wai:live-compose-receipt` (ASCII) and 0x01,
followed by the bytes the digest covers, verified strictly (SPEC §7 *Signatures*). A
reader reads exactly the bytes it is given, as §3.3 steps 1 to 3 read a descriptor with
`WCR1` for `WCD1` (`truncated`, `malformed`, `unknown_version`); refuses as
`receipt_invalid` a receipt whose `media` is not 1 or 2, or that names two inputs of one
participant and one track; and then verifies the signature (`bad_key`, `bad_signature`).
### 8.2 The inputs
The inputs of an output at *t* are, in this order:
- **video** (`media` 2): for each tile of the layout at *t* that shows video, in drawing
order (§5.7), the tracks of its participant's video source;
- **audio** (`media` 1): for each voice, in order (§6), the tracks of its audio source.
The **tracks** of a source are named as live-rooms §6.1 names them: the namespace is the
room, the owner (the participant's key, or `_` and the slot's name for a slot) and the
media kind; the names follow the profile's `layering` (live-rooms §6.3) at the composed
layer *L*: track `0` for `single` and `svc` (an `svc` track carries every layer), track
*L* for `simulcast`, and tracks `0` to *L* for `staged`, since a refinement is applied to
the layers below it ([staged-delivery](staged-delivery.md)). The receipt names, for each,
the object it read.
### 8.3 Verifying a receipt
A receipt verifier is given a room (as live-rooms §9.1), a descriptor and a receipt. In
this order, the first that fails names the refusal and where:
1. **The room** (§4.1): its refusal, at `room`.
2. **The descriptor** (§3.3, §3.5): at `descriptor`.
3. **The receipt** reads (§8.1), at `receipt`; and its `room` is the room id, otherwise
`wrong_room`.
4. Its `descriptor` is the descriptor's digest: otherwise `descriptor_mismatch`.
5. Its `t` is at or after the descriptor's `effective`: otherwise `not_yet_effective`.
6. The descriptor names a compositor, the receipt's `compositor` is it, and that principal
is present at `t` with a grant in effect that has the compose power: otherwise
`compositor_not_permitted`.
7. Its `output`'s `track` is a track of the compositor's own track of the receipt's media
kind, published at `t`, at a layer from 0 to its `layers` − 1: otherwise
`output_not_published`.
8. **The set and the layout** at `t` (§4, §5): their refusal, at `set`.
9. Its `state` is the set's digest: otherwise `state_mismatch`.
10. Its inputs, each its `principal` and its object's `track`, are §8.2's, in order:
otherwise `inputs_mismatch`.
A receipt that passes is **accepted**: its output object was composed under this
descriptor, from this set, out of objects of exactly the tracks the layout reads. Anyone
holding the room's events and the descriptor recomputes the layout and checks that the
inputs were the right ones; a party holding the objects checks their digests. The receipt
does not fix the output's pixels or samples (§6).
## 9. Verification
A **composition verifier** is given a room, a descriptor and a list of instants. It runs
these steps and stops at the first refusal, which it reports with the reason and where:
1. **The room** (§4.1): at `room`, with the room's own place (live-rooms §9.1) when the
room is refused.
2. **The descriptor** (§3.3, §3.5): at `descriptor`.
3. **Each instant**, in the list's order: the instant is at or after `effective`
(otherwise `not_yet_effective`), and the participants' attributes read (§4.3,
otherwise `attr_malformed`): at `set`, with the instant.
Its outcome is **refuse**, with the reason and the place; or the **composition**: the
descriptor's digest and, for each instant, the participants with their sources and
tracks, the set's digest, the order, the tiles and the voices. A receipt verifier (§8.3)
reports its refusals at the same places, and at `receipt`.
## 10. Reasons
The codes this extension adds, each with where it is raised:
| reason | raised at |
|---|---|
| `canvas_out_of_range` | §3.3, §3.4 |
| `compose_invalid` | §3.3 |
| `compose_not_permitted` | §3.5 |
| `not_yet_effective` | §8.3 step 5, §9 step 3 |
| `lifecycle_incomplete` | §4.1 |
| `attr_malformed` | §4.3 |
| `mix_input_malformed` | §6 |
| `receipt_invalid` | §8.1 |
| `descriptor_mismatch` | §8.3 step 4 |
| `compositor_not_permitted` | §7, §8.3 step 6 |
| `output_not_published` | §8.3 step 7 |
| `state_mismatch` | §8.3 step 9 |
| `inputs_mismatch` | §8.3 step 10 |
A verifier of this extension also raises `truncated`, `malformed` and `unknown_version`
for a descriptor or a receipt that does not read; `bad_key` and `bad_signature` for their
signatures; `wrong_room` for one of another room (§3.5, §8.3); and every code a room
verifier raises (live-rooms §11 and the live-events codes it lists), at `room`.
## 11. Conformance
[`live-conformance/compose/`](../live-conformance/compose/) holds the corpus, in one file,
`vectors.json`, written from fixed seeds by `wai_live_compose_vectors gen`. It has:
- **layouts**: the layout function alone (§5, §6's voices), each case its parameters and
its participants with their join orders, video sizes, audio and attributes, and the
order, tiles and voices it gives, or its refusal. Grids of 0 to 12 tiles on a 16:9
canvas, and on portrait, square and auto ones; every fill mode over sources of four
shapes and of extreme aspects; a featured participant present, absent, kept past the
tile limit, and first in order of two; ties in order broken by join order and by key;
stopped video shown and omitted; picture in picture in every corner under both
behaviours, alone static and alone dynamic, with derived sizes and stopped video; and
attributes that do not read.
- **mixes**: blocks of PCM and the mix of them (§6): passing through, both widths into
both, saturation high and low and only at the end, pans and gains at their limits,
rounding at ties, silence, and inputs refused.
- **descriptors**: records read (§3), with their digest and fields, or refused with each
reason a descriptor alone can have.
- **rooms**: a studio room under the live-rooms descriptor, grants and chains that its
vectors name, and four variants of it: a grant whose order does not read, a lifecycle
record not delivered, a viewer's chat not delivered, and a viewer's record refused.
- **compositions**: a room, a descriptor and instants (§9), with the composition or the
refusal. The grid over the whole session re-lays the room out at each lifecycle
instant, among them a grant exchange that reorders it; the picture in picture composes
a lower layer.
- **receipts**: a room, a descriptor and a receipt (§8.3), accepted with its digest and
the set's, or refused with each reason a receipt can have.
A conforming verifier gives every case the outcome, the reason, the place and the data the
corpus states. `wai_live_compose_vectors verify` and the test `tests/live_compose.rs`
check that the file regenerates byte for byte, that `wai-rs` gives every case its
outcome, that every reason of §10 refuses a case, and that §10 lists exactly the
reference's codes. `live-conformance/live_compose_verify.py` is the second implementation,
written from this text: it shares no code with `wai-rs`, verifies signatures with WAI's
strict rule written from RFC 8032 (`claim-conformance/_ed25519.py`), verifies each room
with the live-rooms corpus's own second implementation (`live_rooms_verify.py`) and
replays its lifecycle to each instant itself, and gives every case the same outcome,
rectangle for rectangle and sample for sample; it also mutates copies of the corpus and
requires every copy to fail.
## 12. Security and privacy considerations
- The layout is computed from what the room authority signed: the descriptor, or a
principal's under the power the authority granted, and the attributes of grants the
authority signed. A participant cannot place itself; a relay that drops or reorders
lifecycle events is seen as a gap in a room key's chain (§4.1), not as another layout.
- A server compositor's receipt fixes which objects it composed and under which set. A
compositor that left a participant out, read a layer other than the composed one, or
composed a stale set is refused by §8.3. The receipt does not fix the output's pixels
or samples, which a lossy encode does not reproduce (§6); a party that needs them
composes locally from the inputs.
- Composition on each sink needs no server to decode the room's media, so a room whose
tracks are end-to-end encrypted ([confidentiality](confidentiality.md)) can still be
laid out: the layout reads the room's events and the grants, never media.
- The participant set names every publisher, its grant and its join order. Composing a
room reveals nothing a member of it does not already hold, but a set digest published
outside the room links the participants it names; a room that needs participation
unlinkable across rooms gives principals a key per room (live-events §11).
- Attributes are read as the descriptor's keys name them and nothing else is read of a
grant's attributes, so an attribute a descriptor does not name changes nothing.

13
wai/extensions/live-rooms.md
View file

@ -32,7 +32,7 @@ slots, and the `detail` of every lifecycle kind of live-events §4.13 (§7); the
profiles (§8); room verification (§9); and the bindings onto token formats (§10).
Out of scope: a transport; how a principal obtains a grant; composition, which reads the
grant's attributes (a later extension); the review workflow, of which this extension
grant's attributes ([live-compose](live-compose.md)); the review workflow, of which this extension
fixes only the policy's fields and their meaning (§3.6); and what a recording holds,
which the recording policy constrains (§3.7) and later extensions define: a device's
master recording is [capture-binding](capture-binding.md)'s take.
@ -108,7 +108,8 @@ A **role** maps a name to a **capability set**:
- `events`: the event kinds it may send, a kinds bitmap within the ceiling's `kinds`.
- `powers`: bit 0 **moderate** (its `decision` events, live-events §4.11, are a
moderator's), bit 1 **direct** (it may send `control` commands other than recording),
bit 2 **compose** (it may author the room's composition, a later extension), bit 3
bit 2 **compose** (it may sign the room's composition descriptor and publish a composed
track, [live-compose](live-compose.md) §3.5, §7), bit 3
**record** (it may send `record_start` and `record_stop`).
A role whose `events` permit `decision` has the moderate power; one whose `events` permit
@ -260,9 +261,11 @@ value (a long field holding UTF-8, RFC 3629). Keys are in strictly ascending byt
and the entries take at most 1024 bytes, not counting the field's own length. An entry
that runs past the field's end breaks the rule.
Attributes are for composition: a later extension lays a room out from them. It reads
`featured` (`true` or `false`), `order` (a decimal integer) and `pip` (`true` or
`false`); this revision reads their syntax alone. Capture binding
Attributes are for composition: [live-compose](live-compose.md) lays a room out from
them. It reads `featured` (`true` or `false`), `order` (a decimal integer) and `pip`
(`true` or `false`), and `gain` and `pan` (decimal integers) for the mix, under the keys a
composition descriptor names, and gives each its syntax and its meaning (live-compose
§4.3); this revision checks only the form of the entries. Capture binding
([capture-binding](capture-binding.md) §3) reads two more: `subject`, the principal a
device captures for, as 64 lowercase hexadecimal digits of its key; and `minor` (`true`
or `false`), whether the authority marks that principal a minor. Attributes are signed with the grant,

9
wai/live-conformance/README.md
View file

@ -8,11 +8,13 @@ and the independent verifier of each, beside it.
| [`events/`](events/) | [live-events](../extensions/live-events.md) (`wai.live.events`) | `live_events_verify.py` |
| [`rooms/`](rooms/) | [live-rooms](../extensions/live-rooms.md) (`wai.live.room`) | `live_rooms_verify.py` |
| [`takes/`](takes/) | [capture-binding](../extensions/capture-binding.md) (`wai.capture.take`) | `capture_take_verify.py` |
| [`compose/`](compose/) | [live-compose](../extensions/live-compose.md) (`wai.live.compose`) | `live_compose_verify.py` |
**Provenance.** Generated here: `wai_live_events_vectors gen` writes `events/`,
`wai_live_room_vectors gen` writes `rooms/` and `wai_capture_take_vectors gen` writes
`takes/`, each from records authored in code, from fixed seeds (`events/README.md`,
`rooms/README.md`, `takes/README.md`). No recorded data.
`wai_live_room_vectors gen` writes `rooms/`, `wai_capture_take_vectors gen` writes
`takes/` and `wai_live_compose_vectors gen` writes `compose/`, each from records authored in
code, from fixed seeds (`events/README.md`, `rooms/README.md`, `takes/README.md`,
`compose/README.md`). No recorded data.
Each verifier is written from its extension's text, shares no code with `wai-rs`, and is
run by `.github/workflows/wai-conformance.yml` beside the reference's own check:
@ -21,4 +23,5 @@ run by `.github/workflows/wai-conformance.yml` beside the reference's own check:
python3 live_events_verify.py events
python3 live_rooms_verify.py rooms
python3 capture_take_verify.py takes
python3 live_compose_verify.py compose
```

111
wai/live-conformance/compose/README.md Normal file
View file

@ -0,0 +1,111 @@
# live-conformance/compose — live composition
The corpus of [`extensions/live-compose.md`](../../extensions/live-compose.md) §11: the
layout function and the mixer on their own, `WCD1` composition descriptors, and rooms (a
signed `WRD1` descriptor, its `WRG1` grants and its principals' `WEV1` chains) composed at
a list of instants, with `WCR1` composition receipts checked against them.
**Composition equivalence.** A conforming implementation reaches, for every case, the
outcome the case states, rectangle for rectangle and sample for sample:
- a **layout** case's `params` (the descriptor's fields, §3.1) and `entrants` (each a
`principal`, its `join` order, its `video` as `[width, height]` or `null`, whether it has
`audio`, and its `attrs` as `[key, value]` pairs) give the `order`, the `tiles` and the
`voices` of §5 and §6, or the refusal `reason`. A tile is its `principal`, its `place`
(`grid`, `featured` or `pip`), its `z`, its `tile` rectangle and its `dst` and `src`
rectangles (`null` when its video is stopped), each `[x, y, w, h]`; a voice its
`principal`, `gain` and `pan`;
- a **mix** case's `channels`, `frames` and `inputs` (each its `channels`, `gain`, `pan` and
`pcm`, signed 16-bit little-endian samples in hexadecimal) give the mixed `pcm` (§6), or
the refusal;
- a **descriptor** case's `bytes` read (§3.3) to its `digest` and `fields`, or are refused;
- a **composition** case's `room`, `descriptor` and `instants` give the descriptor's
`digest` and a frame per instant (§9): its `participants` (each its `principal`, its
`grant`, its `join` order, its `attrs`, and its `video` and `audio` sources with their
slot, media kind, profile, layers, composed layer, layering, dimensions and `tracks`),
its `set_digest` (§4.5), and its `order`, `tiles` and `voices`; or a refusal and where,
`at`: `["room"]`, `["room", …]` with the room verifier's own place (live-rooms §9.1),
`["descriptor"]` or `["set", t]`;
- a **receipt** case's `room`, `descriptor` and `receipt` are accepted (§8.3) with the
receipt's `digest` and the `set_digest` it names, or refused, `at` as above or
`["receipt"]`.
A verifier that reaches another outcome, reason, place, rectangle, sample or digest does
not conform.
**Provenance.** Generated here: `wai_live_compose_vectors gen` writes every case, room,
descriptor, record and receipt from scenes authored in code, from fixed seeds. Each key is
SHA-256 of a fixed label and the principal's index, each pure layout case's participant a
SHA-256 of a fixed label and an index (a name, not a key), each block of PCM SHA-256 output
under a fixed label (it stands for decoded audio and is no recording), each object digest
of a receipt SHA-256 of a fixed label (it stands for an object and hashes none), every time
is a constant (the room's `t0` is 2026-10-17T00:00:00Z), and Ed25519 signing is
deterministic (RFC 8032), so the file is the same bytes on every machine. The generator
asserts that each case reaches the outcome it was written to have before it writes it. No
recorded data.
## Layout
One file, `vectors.json`:
- `principals`: the room's keys, by name (informative).
- `layouts`, `mixes`, `descriptors`: the cases above, each with `name` and `about`.
- `rooms`: by name, each a `descriptor` and `grants` (hexadecimal), `key_log_head` (`null`:
the room names no origin) and `chains` (lists of records in hexadecimal).
- `compositions`, `receipts`: the cases above, each naming its room.
The **studio** room has a director (its role has the direct and compose powers), a
performer (camera and music, featured, order 1, gain 5120), a guest (speech only at first,
order 3, then exchanged at 5 s for a grant of order 0, gain 2048 and pan −2048, then video
at 7 s), a screen-sharing device, an encoder live in the slot `main` under a staged
profile of the room's own, a mixer (the compose power; it publishes the composed tracks),
and a viewer. The director's video stops at 6 s and the performer leaves at 8 s. Its
variants: `studio_bad_order` (the guest's order is `third`), `studio_lifecycle_gap` (the
relay's record at 6 s not delivered), `studio_viewer_gap` (the viewer's first chat not
delivered) and `studio_refused` (the viewer sends a `control` it may not).
Cases:
- layouts: `grid_0` to `grid_12`; portrait, square and auto canvases; `fill_fill`,
`fill_cover`, `fill_contain`, `fill_default` and extreme aspects; the featured
participant alone, of four, on a portrait canvas, kept past the tile limit, first of two,
absent, and unread; order by attribute and ties broken by key; stopped video shown and
omitted; the tile limit and the voice limit; picture in picture in each corner, static
and dynamic, alone, with derived sizes, with stopped video, first of two, and over a
featured tile; and attributes that do not read;
- mixes: passing through, both widths into both, saturation, pans, gains, ties, silence and
five refusals;
- descriptors: the room's three and canvases at their limits, and each refusal a descriptor
alone can have;
- compositions: `grid_session` (every second from 1 s to 9 s), `pip_session`,
`participant_gap`, `signer_with_power`, and a refusal for a signer without the power, one
not in the room, another room, a bad signature, an instant before `effective`, an
attribute that does not read, a lifecycle gap and a refused room;
- receipts: `video`, `audio` and `pip_video` accepted, and one refused for each reason a
receipt can have.
## Running
```
cd ../../wai-rs
cargo run --no-default-features --features live_compose --bin wai_live_compose_vectors -- verify ../live-conformance/compose
cargo run --no-default-features --features live_compose --bin wai_live_compose_vectors -- gen ../live-conformance/compose # regenerate
cargo test --no-default-features --features live_compose --test live_compose
cd ../live-conformance
python3 live_compose_verify.py compose
```
`wai_live_compose_vectors verify` and `tests/live_compose.rs` require the file to regenerate
byte for byte and `wai-rs` to give every case its outcome; the test also requires every
reason of §10 to refuse a case, the cases above to be there, the grid session to change its
order at the guest's exchange, and §10 to list exactly the reasons the reference reports.
`live_compose_verify.py` is the second implementation, written from the extension's text
and sharing no code with `wai-rs`: it verifies each room with `live_rooms_verify.py` and
steps through its lifecycle replay to take the room at each instant, gives every case the
same outcome, requires the same coverage of reasons, and then mutates copies of the corpus,
one defect each (a tile moved, tiles swapped, the PiP window's z, a source or destination
rectangle, the order, an entrant's attribute, a parameter, a stated reason, a voice, a mixed
or input sample, a pan, a descriptor's byte or digest, a frame's tile, set digest, join
order or tracks, an instant, a lifecycle record not delivered, a grant's byte, a receipt's
byte or digest, and a stated outcome), each of which must fail. Pass = an exit status of 0
from each.

19209
wai/live-conformance/compose/vectors.json Normal file
View file

File diff suppressed because it is too large Load diff

886
wai/live-conformance/live_compose_verify.py Normal file
View file

@ -0,0 +1,886 @@
#!/usr/bin/env python3
"""Independent verifier of the live-composition corpus (extensions/live-compose.md).
Written from the extension's text. It shares no code with wai-rs, which generates the
corpus, and never runs it. It checks Ed25519 signatures with
../claim-conformance/_ed25519.py, WAI's strict rule (SPEC §7 *Signatures*) written from
RFC 8032 with the standard library. Each room is verified by the live-rooms corpus's own
second implementation, live_rooms_verify.py (live-rooms §9), whose lifecycle replay this
verifier steps through itself, record by record, to take the room at each instant (§4.1).
Everything else is here, from the text:
- the descriptor (§3): read, its canvas, its field rules, its signature and who may sign;
- the participant set (§4): sources, attributes, order, and the set's digest;
- the layout function (§5): the grid and its candidates, the featured tile, picture in
picture, the fill modes; and the mixer (§6), over the corpus's PCM;
- the receipt (§8): read, its inputs, and its verification.
For every case it requires the outcome, reason and place, and the order, tiles, voices,
samples, digests, participants and sources the corpus states; of the corpus, that every
reason of §10 refuses a case. Then it mutates copies of the corpus, one defect each, and
requires every copy to fail.
python3 live_compose_verify.py <live-conformance/compose>
Exit status 0 only if every case verifies and every mutated copy fails. Python 3.8 or
later; standard library only.
"""
import copy
import hashlib
import json
import os
import struct
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
sys.path.insert(0, os.path.join(HERE, "..", "claim-conformance"))
import live_events_verify as LEV # noqa: E402
import live_rooms_verify as LRV # noqa: E402
import _ed25519 as ED # noqa: E402
Refuse = LEV.Refuse
DESCRIPTOR_DOMAIN = b"wai:live-compose\x01"
RECEIPT_DOMAIN = b"wai:live-compose-receipt\x01"
SET_DOMAIN = b"wai:live-compose-set\x01"
ZERO = bytes(32)
AUDIO, VIDEO, SCREEN = 1, 2, 3
MEDIA_NAME = {1: "audio", 2: "video", 3: "screen"}
COMPOSE_POWER = 4
JOIN = 0x10
GRID, PIP = 1, 2
AUTO, VIDEO_ASPECT, SQUARE, PORTRAIT = 0, 1, 2, 3
DEFAULT, FILL, COVER, CONTAIN = 0, 1, 2, 3
STATIC, DYNAMIC = 1, 2
SINGLE, SIMULCAST, SVC, STAGED = 0, 1, 2, 3
# §10, in its order.
REASONS = [
"canvas_out_of_range", "compose_invalid", "compose_not_permitted", "not_yet_effective",
"lifecycle_incomplete", "attr_malformed", "mix_input_malformed", "receipt_invalid",
"descriptor_mismatch", "compositor_not_permitted", "output_not_published", "state_mismatch",
"inputs_mismatch",
]
PARAM_KEYS = ["width", "height", "fps", "layout", "gap", "aspect", "fill", "featured_attr", "order_attr",
"omit_stopped", "pip_attr", "corner", "pip_width", "pip_height", "offset", "behaviour",
"channels", "sample_rate", "gain_attr", "pan_attr", "max_tiles", "max_voices"]
def sha(*parts):
h = hashlib.sha256()
for p in parts:
h.update(p)
return h.digest()
class Reader:
"""Fields in order; running out is `truncated`."""
def __init__(self, b, p=0):
self.b, self.p = b, p
def take(self, n):
if self.p + n > len(self.b):
raise Refuse("truncated")
s = self.b[self.p:self.p + n]
self.p += n
return s
def u8(self):
return self.take(1)[0]
def u16(self):
return struct.unpack("<H", self.take(2))[0]
def u32(self):
return struct.unpack("<I", self.take(4))[0]
def u64(self):
return struct.unpack("<Q", self.take(8))[0]
def short(self):
return self.take(self.u8())
def read_head(b, magic):
"""§3.3 step 1."""
if len(b) < 4:
raise Refuse("truncated")
if b[:4] != magic:
raise Refuse("malformed")
if len(b) < 5:
raise Refuse("truncated")
if b[4] != 1:
raise Refuse("unknown_version")
def is_key(k):
"""live-rooms §2: a slot name, 1 to 32 of a-z, 0-9 and _."""
return 1 <= len(k) <= 32 and all(c in b"abcdefghijklmnopqrstuvwxyz0123456789_" for c in k)
# ---------------------------------------------------------------- §3 the descriptor
def canvas_ok(p):
"""§3.4."""
def edge(e):
return 16 <= e <= 1920 and e % 2 == 0
return edge(p["width"]) and edge(p["height"]) and p["width"] * p["height"] <= 2073600 and 1 <= p["fps"] <= 60
def least_root(n):
k = 0
while k * k < n:
k += 1
return k
def fields_ok(p):
"""§3.3's rules after the canvas's."""
w, h, g = p["width"], p["height"], p["gap"]
if p["layout"] not in (GRID, PIP) or p["aspect"] > 3 or p["fill"] > 3:
return False
for k in ("featured_attr", "order_attr", "pip_attr", "gain_attr", "pan_attr"):
if p[k] != "" and not is_key(p[k].encode()):
return False
if p["layout"] == GRID:
if p["pip_attr"] != "" or p["corner"] or p["pip_width"] or p["pip_height"] or p["offset"] or p["behaviour"]:
return False
else:
o = p["offset"]
if not (1 <= p["corner"] <= 4 and p["behaviour"] in (STATIC, DYNAMIC) and o < w and o < h
and p["pip_width"] + o <= w and p["pip_height"] + o <= h):
return False
if p["channels"] not in (1, 2) or not 8000 <= p["sample_rate"] <= 192000:
return False
if not (1 <= p["max_tiles"] <= 64 and 1 <= p["max_voices"] <= 64):
return False
k = least_root(p["max_tiles"])
need = k + (k - 1) * g
return g < h and w >= need and h - g - (h - g) * 3 // 4 >= need
def check_params(p):
if not canvas_ok(p):
raise Refuse("canvas_out_of_range")
if not fields_ok(p):
raise Refuse("compose_invalid")
def read_descriptor(b):
"""§3.3: the descriptor, or Refuse."""
read_head(b, b"WCD1")
r = Reader(b, 5)
c = {"room": r.take(32), "signer": r.take(32), "effective": r.u64(), "compositor": r.take(32)}
p = {"width": r.u16(), "height": r.u16(), "fps": r.u8()}
c["background"], c["layer"] = r.take(3), r.u8()
p["layout"], p["gap"], p["aspect"], p["fill"] = r.u8(), r.u16(), r.u8(), r.u8()
raw = {"featured_attr": r.short(), "order_attr": r.short()}
omit = r.u8()
raw["pip_attr"] = r.short()
p["corner"], p["pip_width"], p["pip_height"], p["offset"], p["behaviour"] = r.u8(), r.u16(), r.u16(), r.u16(), r.u8()
p["channels"], p["sample_rate"] = r.u8(), r.u32()
raw["gain_attr"], raw["pan_attr"] = r.short(), r.short()
p["max_tiles"], p["max_voices"] = r.u8(), r.u8()
body = b[:r.p]
c["sig"] = r.take(64)
if r.p != len(b):
raise Refuse("malformed")
keys_ok = all(v == b"" or is_key(v) for v in raw.values())
for k, v in raw.items():
p[k] = v.decode("latin-1")
p["omit_stopped"] = omit == 1
if not canvas_ok(p):
raise Refuse("canvas_out_of_range")
if not keys_ok or omit > 1 or not fields_ok(p):
raise Refuse("compose_invalid")
bad = ED.verify(c["signer"], DESCRIPTOR_DOMAIN + body, c["sig"])
if bad:
raise Refuse(bad)
c["params"] = p
c["digest"] = sha(body)
return c
def fields_of(c):
out = {"room": c["room"].hex(), "signer": c["signer"].hex(), "effective": c["effective"],
"compositor": c["compositor"].hex(), "background": c["background"].hex(), "layer": c["layer"]}
for k in PARAM_KEYS:
out[k] = c["params"][k]
return out
# ---------------------------------------------------------------- §4.3 attributes
def decimal(v):
"""A decimal integer of §4.3, or None."""
neg = v.startswith("-")
digits = v[1:] if neg else v
if not 1 <= len(digits) <= 10 or any(ch not in "0123456789" for ch in digits):
return None
if digits[0] == "0" and (len(digits) > 1 or neg):
return None
n = int(digits)
return -n if neg else n
def read_attributes(p, attrs):
"""§4.3 for one participant: (featured, order, pip, gain, pan), or Refuse."""
def look(key):
if key == "":
return None
return next((v for k, v in attrs if k == key), None)
def boolean(key):
v = look(key)
if v is None or v == "false":
return False
if v == "true":
return True
raise Refuse("attr_malformed")
def ranged(key, lo, hi, absent):
v = look(key)
if v is None:
return absent
n = decimal(v)
if n is None or not lo <= n <= hi:
raise Refuse("attr_malformed")
return n
featured = boolean(p["featured_attr"])
order = ranged(p["order_attr"], -2 ** 31, 2 ** 31 - 1, None)
pip = boolean(p["pip_attr"])
gain = ranged(p["gain_attr"], 0, 16384, 4096)
pan = ranged(p["pan_attr"], -4096, 4096, 0)
return featured, order, pip, gain, pan
# ---------------------------------------------------------------- §5 the layout
def fit(bw, bh, a, b):
"""§5.1."""
if bw * b <= bh * a:
return bw, max(1, bw * b // a)
return max(1, bh * a // b), bh
def centred(outer, w, h):
x, y, ow, oh = outer
return (x + (ow - w) // 2, y + (oh - h) // 2, w, h)
def reference(aspect):
return {SQUARE: (1, 1), PORTRAIT: (3, 4)}.get(aspect, (16, 9))
def grid(region, n, gap, aspect):
"""§5.3: n tile rectangles."""
X, Y, W, H = region
a, b = reference(aspect)
chosen = None
for c in range(1, n + 1):
r = -(-n // c)
if (c - 1) * gap >= W or (r - 1) * gap >= H:
continue
cw, ch = (W - (c - 1) * gap) // c, (H - (r - 1) * gap) // r
if cw < 1 or ch < 1:
continue
tw, th = fit(cw, ch, a, b)
if chosen is None or tw * th >= chosen[0]:
chosen = (tw * th, c, r, cw, ch)
_, c, r, cw, ch = chosen
y0 = Y + (H - (r * ch + (r - 1) * gap)) // 2
out = []
for k in range(n):
i, j = k // c, k % c
m = n - (r - 1) * c if i == r - 1 else c
x0 = X + (W - (m * cw + (m - 1) * gap)) // 2
cell = (x0 + j * (cw + gap), y0 + i * (ch + gap), cw, ch)
out.append(cell if aspect == AUTO else centred(cell, *fit(cw, ch, a, b)))
return out
def show(tile, size, mode):
"""§5.6: (destination, source) of a source of `size` in `tile`."""
tx, ty, tw, th = tile
sw, sh = size
wider = sw * th >= sh * tw
whole = (0, 0, sw, sh)
if mode == FILL:
return tile, whole
if mode == CONTAIN:
if wider:
dh = max(1, tw * sh // sw)
return (tx, ty + (th - dh) // 2, tw, dh), whole
dw = max(1, th * sw // sh)
return (tx + (tw - dw) // 2, ty, dw, th), whole
if wider:
cw = max(1, sh * tw // th)
return tile, ((sw - cw) // 2, 0, cw, sh)
ch = max(1, sw * th // tw)
return tile, (0, (sh - ch) // 2, sw, ch)
def layout(p, entrants):
"""§5 and §6's voices over entrants (principal, join, video, audio, attrs): a dict."""
check_params(p)
members = []
for e in entrants:
featured, order, pip, gain, pan = read_attributes(p, e["attrs"])
members.append(dict(e, featured=featured, order=order, pip=pip, gain=gain, pan=pan))
# §4.4
members.sort(key=lambda m: (m["order"] is None, m["order"] or 0, m["join"], m["principal"]))
voices = [{"principal": m["principal"].hex(), "gain": m["gain"], "pan": m["pan"]}
for m in members if m["audio"]][:p["max_voices"]]
# §5.2
shown = [m for m in members if not (p["omit_stopped"] and m["video"] is None)]
pip_m = next((m for m in shown if m["pip"]), None) if p["layout"] == PIP else None
rest = [m for m in shown if m is not pip_m]
f = next((i for i, m in enumerate(rest) if m["featured"]), None)
if f is not None:
rest.insert(0, rest.pop(f))
rest = rest[:p["max_tiles"] - (1 if pip_m is not None else 0)]
featured = f is not None and bool(rest)
window = pip_m
if pip_m is not None and not rest and p["behaviour"] == DYNAMIC:
rest, featured, window = [pip_m], pip_m["featured"], None
tiles = grid_tiles(p, rest, featured)
if window is not None:
tiles.append(pip_tile(p, window))
return {"order": [m["principal"].hex() for m in members], "tiles": tiles, "voices": voices}
def tile_of(p, m, rect, place, z):
mode = p["fill"] if p["fill"] != DEFAULT else (CONTAIN if place == "featured" else COVER)
out = {"principal": m["principal"].hex(), "place": place, "z": z, "tile": list(rect), "dst": None, "src": None}
if m["video"] is not None:
dst, src = show(rect, m["video"], mode)
out["dst"], out["src"] = list(dst), list(src)
return out
def grid_tiles(p, ms, featured):
"""§5.3, §5.4."""
w, h, g = p["width"], p["height"], p["gap"]
if not ms:
return []
if not featured:
return [tile_of(p, m, r, "grid", 0) for m, r in zip(ms, grid((0, 0, w, h), len(ms), g, p["aspect"]))]
if len(ms) == 1:
return [tile_of(p, ms[0], (0, 0, w, h), "featured", 0)]
mh = (h - g) * 3 // 4
out = [tile_of(p, ms[0], (0, 0, w, mh), "featured", 0)]
band = (0, mh + g, w, h - mh - g)
out += [tile_of(p, m, r, "grid", 0) for m, r in zip(ms[1:], grid(band, len(ms) - 1, g, p["aspect"]))]
return out
def pip_tile(p, m):
"""§5.5."""
w, h, o = p["width"], p["height"], p["offset"]
if p["pip_width"] > 0 and p["pip_height"] > 0:
pw, ph = p["pip_width"], p["pip_height"]
else:
a, b = m["video"] if m["video"] is not None else (w, h)
if p["pip_width"] > 0:
bw = p["pip_width"]
elif p["pip_height"] > 0:
bw = w - o
else:
bw = min(w // 4, w - o)
bh = p["pip_height"] if p["pip_height"] > 0 else h - o
pw, ph = fit(bw, bh, a, b)
x = {1: o, 2: w - o - pw, 3: o, 4: w - o - pw}[p["corner"]]
y = {1: o, 2: o, 3: h - o - ph, 4: h - o - ph}[p["corner"]]
return tile_of(p, m, (x, y, pw, ph), "pip", 1)
# ---------------------------------------------------------------- §6 the mix
def mix(channels, frames, inputs):
"""§6: the mixed samples, or Refuse."""
if channels not in (1, 2):
raise Refuse("mix_input_malformed")
for v in inputs:
if v["channels"] not in (1, 2) or not 0 <= v["gain"] <= 16384 or not -4096 <= v["pan"] <= 4096 \
or len(v["pcm"]) != frames * v["channels"]:
raise Refuse("mix_input_malformed")
sums = [0] * (frames * channels)
for v in inputs:
x, g, ch = v["pcm"], v["gain"], v["channels"]
if channels == 1:
for f in range(frames):
sums[f] += (x[f] * 4096 if ch == 1 else (x[2 * f] + x[2 * f + 1]) * 2048) * g
else:
al, ar = 4096 - max(v["pan"], 0), 4096 + min(v["pan"], 0)
for f in range(frames):
left, right = (x[f], x[f]) if ch == 1 else (x[2 * f], x[2 * f + 1])
sums[2 * f] += left * g * al
sums[2 * f + 1] += right * g * ar
return [min(32767, max(-32768, (s + (1 << 23)) >> 24)) for s in sums]
def pcm_of(h):
b = bytes.fromhex(h)
if len(b) % 2:
raise ValueError("odd PCM")
return list(struct.unpack("<%dh" % (len(b) // 2), b))
def pcm_hex(samples):
return struct.pack("<%dh" % len(samples), *samples).hex()
# ---------------------------------------------------------------- §4 the room at an instant
def render_field(f):
"""MOQT §8.8."""
return "".join(chr(c) if (chr(c).isascii() and chr(c).isalnum()) or c == 0x5F else ".%02x" % c for c in f.encode())
def track_digest(room_id, owner, media, name):
"""live-events §3.1's anchor track: the rendered namespace, a zero byte, the name."""
ns = "-".join(render_field(f) for f in (room_id.hex(), owner, MEDIA_NAME[media]))
return sha(ns.encode(), b"\x00", name.encode())
class RoomView:
"""A room that verified (§4.1), its replay stepped record by record."""
def __init__(self, d, held, steps, final):
self.d, self.held, self.steps, self.final = d, held, steps, final
def grant(self, digest):
return next(g for g in self.held if g["digest"] == digest)
def at(self, t):
"""The room at t: the state before the first record of the merged order past t."""
return next((s for rt, s in self.steps if rt > t), self.final)
def snapshot(room, joins):
return {"present": dict(room.present), "own": dict(room.own), "slots": copy.deepcopy(room.slots), "joins": dict(joins)}
def view_room(room):
"""live-rooms §9 over the room, stepped: a RoomView, or (reason, place)."""
descriptor = bytes.fromhex(room["descriptor"])
grants = [bytes.fromhex(g) for g in room["grants"]]
head = room["key_log_head"]
chains = [[bytes.fromhex(r) for r in c] for c in room["chains"]]
try:
d = LRV.read_descriptor(descriptor)
except Refuse as e:
return e.reason, ["room", "descriptor"]
try:
revoked = LRV.check_key_log(d, head)
except Refuse as e:
return e.reason, ["room", "key_log"]
held, bad = LRV.check_grants(d, grants)
if bad:
return bad[1], ["room", "grant", bad[0]]
state = LRV.Room(d, held, revoked)
auth = LRV.authority_policy()
ver = LEV.Verifier(d["room"], auth, {})
seen, joins, steps, at_index = {}, {}, [], []
numbered = 0
for c, i in LRV.merged_order(d, chains):
b = chains[c][i]
at_index.append((c, i))
try:
ev = LEV.read_record(b)
duplicate = ev["digest"] in seen
except Refuse:
ev, duplicate = None, False
if ev is not None:
steps.append((ev["t"], snapshot(state, joins)))
try:
LRV.step(d, state, ver, seen, auth, b)
except Refuse as e:
return e.reason, ["room", "record", c, i]
if not duplicate and LRV.is_room_key(d, ev["principal"]) and ev["kind"] == JOIN:
joins[ver.accepted[-1][2]["subject"]] = numbered
numbered += 1
if ver.gaps:
if any(LRV.is_room_key(d, p) for p, _, _ in ver.gaps):
return "lifecycle_incomplete", ["room"]
else:
mismatch = LEV.stated_tally_mismatch(ver.accepted)
if mismatch is not None:
c, i = at_index[mismatch]
return "tally_mismatch", ["room", "record", c, i]
return RoomView(d, held, steps, snapshot(state, joins))
def source(view, c, profile, layers, slot, media):
"""§4.2: a source and its composed layer."""
prof = view.d["profiles"][profile]
layer = min(layers - 1, c["layer"])
w, h = prof["layers"][layer][0], prof["layers"][layer][1]
return {"slot": slot, "media": media, "profile": profile, "layers": layers, "layer": layer,
"layering": prof["layering"], "width": w, "height": h}
def tracks(view, s, principal):
"""§8.2: the tracks a source is read from."""
owner = principal.hex() if s["slot"] == "" else "_" + s["slot"]
if s["layering"] == SIMULCAST:
names = [s["layer"]]
elif s["layering"] == STAGED:
names = list(range(s["layer"] + 1))
else:
names = [0]
return [track_digest(view.d["room"], owner, s["media"], str(n)) for n in names]
def participants(view, c, t):
"""§4.2 at t, in ascending order of key."""
st = view.at(t)
out = []
slots = list(zip(view.d["slots"], st["slots"]))
def own(p, m):
h = st["own"].get((p, m))
return None if h is None else source(view, c, h[0], h[1], "", m)
def live_in(p, kinds):
for s, ss in slots:
if ss["live"] and ss["live"][0] == p and s["media"] in kinds:
return source(view, c, ss["live"][1], ss["live"][2], s["name"].decode(), s["media"])
return None
for p in sorted(st["present"]):
if p == c["compositor"]:
continue
if not (any(q == p for q, _ in st["own"]) or any(ss["live"] and ss["live"][0] == p for _, ss in slots)):
continue
video = own(p, VIDEO) or own(p, SCREEN) or live_in(p, (VIDEO, SCREEN))
audio = own(p, AUDIO) or live_in(p, (AUDIO,))
g = view.grant(st["present"][p])
out.append({"principal": p, "grant": g["digest"], "join": st["joins"][p],
"attrs": LRV.read_attrs(g["attrs"]), "video": video, "audio": audio})
return out
def set_digest(view, t, ps):
"""§4.5."""
def src(s):
if s is None:
return b"\x00"
slot = s["slot"].encode()
return bytes([s["media"], len(slot)]) + slot + bytes([s["profile"], s["layers"], s["layer"]]) \
+ struct.pack("<HH", s["width"], s["height"])
out = SET_DOMAIN + view.d["room"] + struct.pack("<QI", t, len(ps))
for p in ps:
out += p["principal"] + p["grant"] + struct.pack("<I", p["join"]) + src(p["video"]) + src(p["audio"])
return sha(out)
def frame(view, c, t):
"""§4 and §5 at t: the frame as the corpus writes it, with what §8.2 reads."""
if t < c["effective"]:
raise Refuse("not_yet_effective")
ps = participants(view, c, t)
entrants = [{"principal": p["principal"], "join": p["join"],
"video": None if p["video"] is None else (p["video"]["width"], p["video"]["height"]),
"audio": p["audio"] is not None, "attrs": p["attrs"]} for p in ps]
lay = layout(c["params"], entrants)
def src_json(s, principal):
if s is None:
return None
return dict(s, tracks=[x.hex() for x in tracks(view, s, principal)])
out = {"t": t, "participants": [{"principal": p["principal"].hex(), "grant": p["grant"].hex(), "join": p["join"],
"attrs": [[k, v] for k, v in p["attrs"]],
"video": src_json(p["video"], p["principal"]),
"audio": src_json(p["audio"], p["principal"])} for p in ps],
"set_digest": set_digest(view, t, ps).hex()}
out.update(lay)
by = {p["principal"].hex(): p for p in ps}
inputs = {VIDEO: [], AUDIO: []}
for tl in lay["tiles"]:
if tl["dst"] is not None:
p = by[tl["principal"]]
inputs[VIDEO] += [(p["principal"], x) for x in tracks(view, p["video"], p["principal"])]
for v in lay["voices"]:
p = by[v["principal"]]
inputs[AUDIO] += [(p["principal"], x) for x in tracks(view, p["audio"], p["principal"])]
return out, inputs
def check_descriptor(view, b):
"""§3.3, §3.5."""
c = read_descriptor(b)
if c["room"] != view.d["room"]:
raise Refuse("wrong_room")
if c["signer"] != view.d["authority"]:
st = view.at(c["effective"])
g = st["present"].get(c["signer"])
if g is None or view.grant(g)["powers"] & COMPOSE_POWER == 0:
raise Refuse("compose_not_permitted")
return c
# ---------------------------------------------------------------- §8 the receipt
def read_object(r):
return {"track": r.take(32), "group": r.u64(), "object": r.u64(), "digest": r.take(32)}
def read_receipt(b):
"""§8.1: the receipt, or Refuse."""
read_head(b, b"WCR1")
r = Reader(b, 5)
rc = {"room": r.take(32), "compositor": r.take(32), "descriptor": r.take(32), "t": r.u64(), "state": r.take(32),
"media": r.u8(), "output": read_object(r)}
rc["inputs"] = [(r.take(32), read_object(r)) for _ in range(r.u8())]
body = b[:r.p]
rc["sig"] = r.take(64)
if r.p != len(b):
raise Refuse("malformed")
pairs = [(p, o["track"]) for p, o in rc["inputs"]]
if rc["media"] not in (AUDIO, VIDEO) or len(set(pairs)) != len(pairs):
raise Refuse("receipt_invalid")
bad = ED.verify(rc["compositor"], RECEIPT_DOMAIN + body, rc["sig"])
if bad:
raise Refuse(bad)
rc["digest"] = sha(body)
return rc
def verify_receipt(view, dbytes, rbytes):
"""§8.3: what the corpus states for a receipt case, the room already viewed."""
try:
c = check_descriptor(view, dbytes)
except Refuse as e:
return {"outcome": "refuse", "reason": e.reason, "at": ["descriptor"]}
def no(reason):
return {"outcome": "refuse", "reason": reason, "at": ["receipt"]}
try:
rc = read_receipt(rbytes)
except Refuse as e:
return no(e.reason)
if rc["room"] != view.d["room"]:
return no("wrong_room")
if rc["descriptor"] != c["digest"]:
return no("descriptor_mismatch")
if rc["t"] < c["effective"]:
return no("not_yet_effective")
st = view.at(rc["t"])
g = st["present"].get(rc["compositor"])
if c["compositor"] == ZERO or rc["compositor"] != c["compositor"] or g is None \
or view.grant(g)["powers"] & COMPOSE_POWER == 0:
return no("compositor_not_permitted")
held = st["own"].get((rc["compositor"], rc["media"]))
if held is None or rc["output"]["track"] not in [
track_digest(view.d["room"], rc["compositor"].hex(), rc["media"], str(n)) for n in range(held[1])]:
return no("output_not_published")
try:
f, inputs = frame(view, c, rc["t"])
except Refuse as e:
return {"outcome": "refuse", "reason": e.reason, "at": ["set", rc["t"]]}
if rc["state"].hex() != f["set_digest"]:
return no("state_mismatch")
if [(p, o["track"]) for p, o in rc["inputs"]] != inputs[rc["media"]]:
return no("inputs_mismatch")
return {"outcome": "accept", "digest": rc["digest"].hex(), "set_digest": f["set_digest"]}
# ---------------------------------------------------------------- the corpus
def refuse(reason, at=None):
out = {"outcome": "refuse", "reason": reason}
if at is not None:
out["at"] = at
return out
def run_layout(v):
p = v["params"]
entrants = [{"principal": bytes.fromhex(e["principal"]), "join": e["join"],
"video": None if e["video"] is None else tuple(e["video"]), "audio": e["audio"],
"attrs": [tuple(kv) for kv in e["attrs"]]} for e in v["entrants"]]
try:
return dict(outcome="layout", **layout(p, entrants))
except Refuse as e:
return refuse(e.reason)
def run_mix(v):
inputs = [{"channels": x["channels"], "gain": x["gain"], "pan": x["pan"], "pcm": pcm_of(x["pcm"])} for x in v["inputs"]]
try:
return {"outcome": "mix", "pcm": pcm_hex(mix(v["channels"], v["frames"], inputs))}
except Refuse as e:
return refuse(e.reason)
def run_descriptor(v):
try:
c = read_descriptor(bytes.fromhex(v["bytes"]))
except Refuse as e:
return refuse(e.reason)
return {"outcome": "read", "digest": c["digest"].hex(), "fields": fields_of(c)}
class Rooms:
"""Each room of the corpus viewed once, by its content."""
def __init__(self):
self.cache = {}
def get(self, room):
k = json.dumps(room, sort_keys=True)
if k not in self.cache:
self.cache[k] = view_room(room)
return self.cache[k]
def run_composition(v, rooms, doc):
view = rooms.get(doc["rooms"][v["room"]])
if not isinstance(view, RoomView):
return refuse(view[0], view[1])
try:
c = check_descriptor(view, bytes.fromhex(v["descriptor"]))
except Refuse as e:
return refuse(e.reason, ["descriptor"])
frames = []
for t in v["instants"]:
try:
frames.append(frame(view, c, t)[0])
except Refuse as e:
return refuse(e.reason, ["set", t])
return {"outcome": "compose", "digest": c["digest"].hex(), "frames": frames}
def run_receipt(v, rooms, doc):
view = rooms.get(doc["rooms"][v["room"]])
if not isinstance(view, RoomView):
return refuse(view[0], view[1])
return verify_receipt(view, bytes.fromhex(v["descriptor"]), bytes.fromhex(v["receipt"]))
SINGULAR = {"layouts": "layout", "mixes": "mix", "descriptors": "descriptor", "compositions": "composition",
"receipts": "receipt"}
SECTIONS = [("layouts", lambda v, rooms, doc: run_layout(v)), ("mixes", lambda v, rooms, doc: run_mix(v)),
("descriptors", lambda v, rooms, doc: run_descriptor(v)), ("compositions", run_composition),
("receipts", run_receipt)]
def check(doc, rooms, quiet=False):
"""Every case against its statement: the failures."""
fails = []
for section, run in SECTIONS:
for v in doc[section]:
try:
got = run(v, rooms, doc)
except Exception as e: # a crash on a mutated copy is a failure to read it
got, problems = None, ["the case does not verify: %s: %s" % (type(e).__name__, e)]
else:
problems = [k for k in got if v.get(k) != got[k]]
problems += [k for k in ("outcome", "reason", "at") if k in v and k not in got]
if not quiet:
what = "?" if got is None else got["outcome"] + (" " + got["reason"] if got["outcome"] == "refuse" else "")
print(("PASS" if not problems else "FAIL") + " %s %s: %s" % (SINGULAR[section], v["name"], what))
for k in problems:
fails.append("%s %s: %s is %s, the corpus states %s" % (
SINGULAR[section], v["name"], k, json.dumps(got.get(k))[:200] if got else k, json.dumps(v.get(k))[:200]))
return fails
def flip(h, at, bit=0x01):
b = bytearray(bytes.fromhex(h))
b[at] ^= bit
return b.hex()
def mutations(doc):
"""Copies of the corpus, each with one defect a conforming verifier must catch."""
idx = {s: {v["name"]: i for i, v in enumerate(doc[s])} for s, _ in SECTIONS}
def case(d, section, name):
return d[section][idx[section][name]]
def mutate(label, f):
d = copy.deepcopy(doc)
f(d)
return label, d
def frame_at(d, name, t):
return next(f for f in case(d, "compositions", name)["frames"] if f["t"] == t)
def relay_drop(d):
relay = doc["principals"]["relay"]
chain = next(c for c in d["rooms"]["studio"]["chains"] if LEV.read_record(bytes.fromhex(c[0]))["principal"].hex() == relay)
chain.pop(5)
return [
mutate("grid_5: a tile one pixel right", lambda d: case(d, "layouts", "grid_5")["tiles"][4]["tile"].__setitem__(0, case(d, "layouts", "grid_5")["tiles"][4]["tile"][0] + 1)),
mutate("featured_4: two tiles swapped", lambda d: case(d, "layouts", "featured_4")["tiles"].reverse()),
mutate("pip_bottom_right_static: the window under the grid", lambda d: case(d, "layouts", "pip_bottom_right_static")["tiles"][-1].update(z=0)),
mutate("fill_cover: a source rectangle", lambda d: case(d, "layouts", "fill_cover")["tiles"][0]["src"].__setitem__(1, 1)),
mutate("fill_contain: a destination rectangle", lambda d: case(d, "layouts", "fill_contain")["tiles"][0]["dst"].__setitem__(3, 1)),
mutate("order_key_tie_break: two of the order swapped", lambda d: case(d, "layouts", "order_key_tie_break")["order"].reverse()),
mutate("order_attribute: an entrant's order", lambda d: case(d, "layouts", "order_attribute")["entrants"][2]["attrs"][0].__setitem__(1, "9")),
mutate("stopped_omitted: omit_stopped off", lambda d: case(d, "layouts", "stopped_omitted")["params"].update(omit_stopped=False)),
mutate("pip_alone_dynamic: stated static", lambda d: case(d, "layouts", "pip_alone_dynamic")["params"].update(behaviour=STATIC)),
mutate("attr_order_leading_zero: stated compose_invalid", lambda d: case(d, "layouts", "attr_order_leading_zero").update(reason="compose_invalid")),
mutate("voices: a gain", lambda d: case(d, "layouts", "voices")["voices"][0].update(gain=4097)),
mutate("sum_saturates_high: an output sample", lambda d: case(d, "mixes", "sum_saturates_high").update(pcm=flip(case(d, "mixes", "sum_saturates_high")["pcm"], 0))),
mutate("half_gain_ties: an input sample", lambda d: case(d, "mixes", "half_gain_ties")["inputs"][0].update(pcm=flip(case(d, "mixes", "half_gain_ties")["inputs"][0]["pcm"], 0))),
mutate("pan_hard_left: panned right", lambda d: case(d, "mixes", "pan_hard_left")["inputs"][0].update(pan=4096)),
mutate("descriptor grid: a byte of its body", lambda d: case(d, "descriptors", "grid").update(bytes=flip(case(d, "descriptors", "grid")["bytes"], 40))),
mutate("descriptor grid: its stated digest", lambda d: case(d, "descriptors", "grid").update(digest="00" * 32)),
mutate("descriptor canvas_too_wide: stated compose_invalid", lambda d: case(d, "descriptors", "canvas_too_wide").update(reason="compose_invalid")),
mutate("grid_session: a tile at 5 s", lambda d: frame_at(d, "grid_session", 5000000)["tiles"][1]["tile"].__setitem__(2, 1)),
mutate("grid_session: the set digest at 7 s", lambda d: frame_at(d, "grid_session", 7000000).update(set_digest="00" * 32)),
mutate("grid_session: a participant's join order", lambda d: frame_at(d, "grid_session", 4000000)["participants"][0].update(join=9)),
mutate("grid_session: a staged source's tracks reordered", lambda d: next(
p for p in frame_at(d, "grid_session", 4000000)["participants"] if p["video"] and len(p["video"]["tracks"]) > 1)["video"]["tracks"].reverse()),
mutate("grid_session: an instant moved", lambda d: case(d, "compositions", "grid_session")["instants"].__setitem__(4, 4500000)),
mutate("pip_session: a voice's pan", lambda d: frame_at(d, "pip_session", 6000000)["voices"][0].update(pan=1)),
mutate("studio: a lifecycle record not delivered", relay_drop),
mutate("studio: a byte of a grant", lambda d: d["rooms"]["studio"]["grants"].__setitem__(1, flip(d["rooms"]["studio"]["grants"][1], 200))),
mutate("receipt video: a byte of it", lambda d: case(d, "receipts", "video").update(receipt=flip(case(d, "receipts", "video")["receipt"], 150))),
mutate("receipt video: its stated digest", lambda d: case(d, "receipts", "video").update(digest="00" * 32)),
mutate("receipt inputs_missing: stated accepted", lambda d: case(d, "receipts", "inputs_missing").update(outcome="accept")),
mutate("receipt state_stale: stated inputs_mismatch", lambda d: case(d, "receipts", "state_stale").update(reason="inputs_mismatch")),
]
def main(corpus_dir):
fails = []
st = ED._self_test()
print(("PASS" if not st else "FAIL") + " Ed25519 strict rule self-test " + ",".join(st))
if st:
fails.append("ed25519 self-test")
with open(os.path.join(corpus_dir, "vectors.json"), encoding="utf-8") as f:
doc = json.load(f)
rooms = Rooms()
fails += check(doc, rooms)
seen = {v.get("reason") for s, _ in SECTIONS for v in doc[s]}
missing = [r for r in REASONS if r not in seen]
if missing:
fails.append("no case is refused %s" % ", ".join(missing))
print("%s every reason of §10 refuses a case" % ("PASS" if not missing else "FAIL"))
for label, mutated in mutations(doc):
caught = check(mutated, rooms, quiet=True)
print(("PASS" if caught else "FAIL") + " mutated copy fails: %s%s" % (label, " (%s)" % caught[0][:120] if caught else ""))
if not caught:
fails.append("a mutated copy verifies: %s" % label)
for f in fails:
print("FAIL " + f)
total = sum(len(doc[s]) for s, _ in SECTIONS)
print(("PASS" if not fails else "FAIL") + " live-composition corpus: %d cases" % total)
return 1 if fails else 0
if __name__ == "__main__":
if len(sys.argv) != 2:
print("usage: live_compose_verify.py <live-conformance/compose>", file=sys.stderr)
sys.exit(2)
sys.exit(main(sys.argv[1]))

28
wai/registry/capabilities.json
View file

@ -558,6 +558,20 @@
"edition": null,
"registered": "2026-06-12"
},
{
"capability": "wai.live.compose",
"kind": "record",
"status": "registered",
"media": null,
"role": null,
"tier": null,
"licence": null,
"replicate": null,
"sets": [],
"spec_section": "§5 Interactive, temporal & computed content",
"edition": null,
"registered": "2026-10-17"
},
{
"capability": "wai.live.events",
"kind": "record",
@ -1685,6 +1699,13 @@
"section": "§5",
"status": "defined"
},
{
"name": "WCD1",
"kind": "magic",
"document": "extensions/live-compose.md",
"section": "§3.1",
"status": "defined"
},
{
"name": "WCN1",
"kind": "magic",
@ -1692,6 +1713,13 @@
"section": "§8.1",
"status": "defined"
},
{
"name": "WCR1",
"kind": "magic",
"document": "extensions/live-compose.md",
"section": "§8.1",
"status": "defined"
},
{
"name": "WEP1",
"kind": "magic",

7
wai/vocab/v2.ttl
View file

@ -522,6 +522,13 @@ wai:wai.image.png a wai:Capability ;
wai:specSection "§5 Mandatory floor" ;
wai:registeredOn "2026-06-12"^^xsd:date .
wai:wai.live.compose a wai:Capability ;
rdfs:label "wai.live.compose" ;
wai:kind wai:kind-record ;
wai:status wai:status-registered ;
wai:specSection "§5 Interactive, temporal & computed content" ;
wai:registeredOn "2026-10-17"^^xsd:date .
wai:wai.live.events a wai:Capability ;
rdfs:label "wai.live.events" ;
wai:kind wai:kind-record ;

17
wai/wai-rs/Cargo.toml
View file

@ -192,6 +192,13 @@ live_room = ["live_events", "key_discovery"]
# with its single-take, aligned, complete and publishable verdicts. Ed25519, SHA-256 and
# the key log only; no new dependency.
capture_take = ["live_room"]
# Live composition (extensions/live-compose.md): `wai.live.compose` — the signed
# composition descriptor (WCD1), the participant set a room's lifecycle replay gives at
# each instant, the integer layout function (grid, featured tile, picture in picture, fill
# modes) and the saturating integer mixer, and the compositor's signed receipt (WCR1).
# The layout and the mixer use nothing beyond core and alloc. Ed25519 and SHA-256 only; no
# new dependency.
live_compose = ["live_room"]
# Object-based spatial audio (extensions/spatial-audio): the deterministic
# Fx mixdown engine. Reuses the world `Fx` floor + blake3; no codec/ML deps
# (sources arrive as decoded mono PCM — decoding the wai.audio.* assets is
@ -634,6 +641,11 @@ name = "wai_capture_take_vectors"
path = "src/bin/wai_capture_take_vectors.rs"
required-features = ["capture_take"]
[[bin]]
name = "wai_live_compose_vectors"
path = "src/bin/wai_live_compose_vectors.rs"
required-features = ["live_compose"]
[[bin]]
name = "wai_world_meter"
path = "src/bin/wai_world_meter.rs"
@ -1259,6 +1271,11 @@ name = "capture_take"
path = "tests/capture_take.rs"
required-features = ["capture_take"]
[[test]]
name = "live_compose"
path = "tests/live_compose.rs"
required-features = ["live_compose"]
[[test]]
name = "world_replay"
path = "tests/world_replay.rs"

53
wai/wai-rs/src/bin/wai_live_compose_vectors.rs Normal file
View file

@ -0,0 +1,53 @@
//! Conformance corpus for live composition (`live-conformance/compose/`, extensions/live-compose.md §11).
//!
//! wai_live_compose_vectors gen <corpus-dir> write vectors.json, from fixed seeds
//! wai_live_compose_vectors verify <corpus-dir> vectors.json regenerates byte for byte, and
//! every case gets the outcome it states
//!
//! The generator and this verifier are one implementation. The corpus's second
//! implementation is `live-conformance/live_compose_verify.py`, written from the extension's
//! text and sharing no code with this binary.
use std::path::Path;
use wai::live_compose::corpus;
fn main() {
let a: Vec<String> = std::env::args().collect();
match (a.get(1).map(String::as_str), a.get(2)) {
(Some("gen"), Some(d)) => {
let p = Path::new(d).join("vectors.json");
if let Err(e) = std::fs::create_dir_all(d).and_then(|_| std::fs::write(&p, corpus::generate())) {
eprintln!("write {}: {e}", p.display());
std::process::exit(1);
}
println!("wrote {}", p.display());
}
(Some("verify"), Some(d)) => std::process::exit(verify(Path::new(d))),
_ => {
eprintln!("usage: wai_live_compose_vectors gen|verify <corpus-dir>");
std::process::exit(2);
}
}
}
fn verify(dir: &Path) -> i32 {
let p = dir.join("vectors.json");
let doc = match std::fs::read_to_string(&p) {
Ok(s) => s,
Err(e) => {
eprintln!("read {}: {e}", p.display());
return 1;
}
};
let mut fail = 0;
let same = doc == corpus::generate();
println!("{} vectors.json regenerates byte for byte from its seeds", if same { "PASS" } else { "FAIL" });
fail += (!same) as i32;
let fails = corpus::verify(&doc);
for f in &fails {
println!("FAIL {f}");
}
println!("{} every case gets the outcome the corpus states", if fails.is_empty() { "PASS" } else { "FAIL" });
fail += (!fails.is_empty()) as i32;
(fail != 0) as i32
}

11
wai/wai-rs/src/codecs/registry.rs
View file

@ -212,6 +212,7 @@ pub const CAP_VIDEO_CHANNEL: &str = "wai.video.channel";
pub const CAP_LIVE_EVENTS: &str = "wai.live.events";
pub const CAP_LIVE_ROOM: &str = "wai.live.room";
pub const CAP_CAPTURE_TAKE: &str = "wai.capture.take";
pub const CAP_LIVE_COMPOSE: &str = "wai.live.compose";
/// One row per string SPEC §5 registers that is a payload, companion or record:
/// `(string, SPEC section, edition, registered)`.
@ -292,6 +293,7 @@ const ROWS: &[(&str, &str, Option<&str>, &str)] = &[
(CAP_LIVE_EVENTS, S_INTERACTIVE, None, "2026-10-16"),
(CAP_LIVE_ROOM, S_INTERACTIVE, None, "2026-10-16"),
(CAP_CAPTURE_TAKE, S_INTERACTIVE, None, "2026-10-17"),
(CAP_LIVE_COMPOSE, S_INTERACTIVE, None, "2026-10-17"),
(CAP_AUDIO_IAMF, S_IMMERSIVE, None, "2026-06-17"),
(CAP_AUDIO_AC4, S_IMMERSIVE, None, "2026-06-17"),
(CAP_AUDIO_IVAS, S_IMMERSIVE, None, "2026-06-17"),
@ -338,7 +340,7 @@ const DERIVATIONS_REGISTERED: &str = "2026-09-28";
const RECORDS: &[&str] = &[
CAP_ASSET_PROVENANCE, CAP_QUANTUM_CALIBRATION, CAP_QUANTUM_JOB,
CAP_VIDEO_MANIFEST, CAP_VIDEO_SSAI, CAP_VIDEO_KEYS, CAP_VIDEO_PACKAGE, CAP_VIDEO_CHANNEL,
CAP_LIVE_EVENTS, CAP_LIVE_ROOM, CAP_CAPTURE_TAKE,
CAP_LIVE_EVENTS, CAP_LIVE_ROOM, CAP_CAPTURE_TAKE, CAP_LIVE_COMPOSE,
];
/// A named identifier SPEC uses that has the shape of a capability string but is
@ -479,6 +481,7 @@ const PACKAGING: &str = "extensions/packaging.md";
const LIVE_EVENTS: &str = "extensions/live-events.md";
const LIVE_ROOMS: &str = "extensions/live-rooms.md";
const CAPTURE_BINDING: &str = "extensions/capture-binding.md";
const LIVE_COMPOSE: &str = "extensions/live-compose.md";
const fn wire(name: &'static str, kind: WireKind, document: &'static str, section: &'static str,
status: WireStatus) -> WireIdentifier {
@ -487,8 +490,8 @@ const fn wire(name: &'static str, kind: WireKind, document: &'static str, sectio
/// The wire identifiers of the container, of its two delivery bindings
/// (http-delivery, moq-streaming-format), of the key discovery those bindings serve
/// (jwp-receipts §7), of packaging, of live events, of live rooms and of capture binding, in document order. The
/// export lists them sorted by name and kind.
/// (jwp-receipts §7), of packaging, of live events, of live rooms, of capture binding and
/// of live composition, in document order. The export lists them sorted by name and kind.
pub const WIRE_IDENTIFIERS: &[WireIdentifier] = &[
wire("WAI1", WireKind::Magic, "SPEC.md", "§2", WireStatus::Defined),
wire("WAI2", WireKind::Magic, "SPEC.md", "§7.1", WireStatus::Defined),
@ -542,6 +545,8 @@ pub const WIRE_IDENTIFIERS: &[WireIdentifier] = &[
wire("WUR1", WireKind::Magic, CAPTURE_BINDING, "§6.2", WireStatus::Defined),
wire("WTM1", WireKind::Magic, CAPTURE_BINDING, "§7.1", WireStatus::Defined),
wire("WCN1", WireKind::Magic, CAPTURE_BINDING, "§8.1", WireStatus::Defined),
wire("WCD1", WireKind::Magic, LIVE_COMPOSE, "§3.1", WireStatus::Defined),
wire("WCR1", WireKind::Magic, LIVE_COMPOSE, "§8.1", WireStatus::Defined),
];
/// A string with the shape of a capability that the reference implementation uses

9
wai/wai-rs/src/lib.rs
View file

@ -406,6 +406,15 @@ pub mod live_room;
#[cfg(feature = "capture_take")]
pub mod capture_take;
// Live composition (extensions/live-compose.md): `wai.live.compose` — the signed
// composition descriptor, the participant set the room's lifecycle replay gives at each
// room-clock instant, the integer layout function and mixer that every sink runs alike,
// and the receipt a server compositor signs for a composed track. Rust-only, like the
// other live modules: its constants must not become macros in include/wai.h.
/// cbindgen:ignore
#[cfg(feature = "live_compose")]
pub mod live_compose;
// Typed receipt classes (extensions/jwp-receipts.md §2.4.3–§2.4.12, §2.6, §3.1–§3.2):
// energy, lawful-basis, log-inclusion, soft-binding, binding-recovery, measurement,
// operating-point, stage, delivery-gap and reconstruct-session statements about

920
wai/wai-rs/src/live_compose.rs Normal file
View file

@ -0,0 +1,920 @@
//! Live composition — `wai.live.compose` ([extensions/live-compose.md]).
//!
//! A room's composition is an integer function. A signed **composition descriptor**
//! (`WCD1`), by the room authority or a principal whose grant has the compose power
//! (live-rooms §3.3), fixes the canvas, the background, the layout (a grid, or a grid with
//! a picture-in-picture window), the attribute keys it reads and its limits. The
//! **participant set** at a room-clock instant is derived from the room's lifecycle
//! replay ([`crate::live_room`]): who publishes, each one's video and audio source and its
//! dimensions, the attributes of its grant in effect, and its join order. The layout and
//! the voices of the mix are then a pure function of the two ([`layout::layout`]), and the
//! mix a pure function of decoded PCM ([`layout::mix`]), so every sink composes the same
//! rectangles locally, and a server compositor that publishes a composed track signs a
//! **composition receipt** (`WCR1`) that anyone can check against the same function.
//!
//! What this module implements, by section of the extension:
//!
//! - §3: the descriptor, read, its canvas limits and field rules, its digest and
//! signature, and who may sign it;
//! - §4: the room state at an instant, the participant set, the attributes the descriptor
//! reads, the order, and the set's digest;
//! - §5, §6: the layout and the mixer, in [`layout`], which uses nothing beyond `core`,
//! `Vec` and `String`;
//! - §8: the receipt, read, signed, and verified against the room and the descriptor.
//!
//! The corpus `live-conformance/compose/` is written by [`corpus::generate`] and checked
//! by [`corpus::verify`] here, and by `live-conformance/live_compose_verify.py`, which
//! shares no code with this module.
//!
//! [extensions/live-compose.md]: ../../extensions/live-compose.md
use std::collections::{BTreeMap, BTreeSet};
use ed25519_dalek::{Signer, SigningKey};
use sha2::{Digest, Sha256};
use crate::live_events::{self as events, anchor_track, kind, Event, Payload};
use crate::live_room::{
self as room, layering, media, media_name, power, render_namespace, track_name, track_namespace, verify_room, At, Descriptor, Grant, Held, Owner, RoomOutcome, RoomRun,
RoomState,
};
pub mod corpus;
pub mod layout;
use layout::{Entrant, Layout, Params, Refusal};
/// The record this extension registers.
pub const CAPABILITY: &str = "wai.live.compose";
/// The magic of a composition descriptor (§3.1).
pub const DESCRIPTOR_MAGIC: [u8; 4] = *b"WCD1";
/// The magic of a composition receipt (§8.1).
pub const RECEIPT_MAGIC: [u8; 4] = *b"WCR1";
/// The version of both records in this revision.
pub const RECORD_VERSION: u8 = 1;
/// The domain a descriptor's signature is made over (§3.2).
pub const DESCRIPTOR_DOMAIN: &[u8] = b"wai:live-compose\x01";
/// The domain a receipt's signature is made over (§8.1).
pub const RECEIPT_DOMAIN: &[u8] = b"wai:live-compose-receipt\x01";
/// The domain of a participant set's digest (§4.5).
pub const SET_DOMAIN: &[u8] = b"wai:live-compose-set\x01";
/// A descriptor's `compositor` when no server composes (§3.1, §7).
pub const NO_COMPOSITOR: [u8; 32] = [0; 32];
// ---------------------------------------------------------------------------------------
// §10 Reasons
// ---------------------------------------------------------------------------------------
/// Why a composition, a mix or a receipt is refused (§10).
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum Reason {
/// A code of live-rooms §11, or of live-events §9 through it.
Room(room::Reason),
CanvasOutOfRange,
ComposeInvalid,
ComposeNotPermitted,
NotYetEffective,
LifecycleIncomplete,
AttrMalformed,
MixInputMalformed,
ReceiptInvalid,
DescriptorMismatch,
CompositorNotPermitted,
OutputNotPublished,
StateMismatch,
InputsMismatch,
}
impl From<room::Reason> for Reason {
fn from(r: room::Reason) -> Self {
Reason::Room(r)
}
}
impl From<events::Reason> for Reason {
fn from(r: events::Reason) -> Self {
Reason::Room(room::Reason::Events(r))
}
}
impl From<Refusal> for Reason {
fn from(r: Refusal) -> Self {
match r {
Refusal::CanvasOutOfRange => Reason::CanvasOutOfRange,
Refusal::ComposeInvalid => Reason::ComposeInvalid,
Refusal::AttrMalformed => Reason::AttrMalformed,
Refusal::MixInputMalformed => Reason::MixInputMalformed,
}
}
}
const TRUNCATED: Reason = Reason::Room(room::Reason::Events(events::Reason::Truncated));
const MALFORMED: Reason = Reason::Room(room::Reason::Events(events::Reason::Malformed));
const UNKNOWN_VERSION: Reason = Reason::Room(room::Reason::Events(events::Reason::UnknownVersion));
const WRONG_ROOM: Reason = Reason::Room(room::Reason::Events(events::Reason::WrongRoom));
impl Reason {
/// The codes this extension adds, in §10's order.
pub const ALL: [Reason; 13] = [
Reason::CanvasOutOfRange,
Reason::ComposeInvalid,
Reason::ComposeNotPermitted,
Reason::NotYetEffective,
Reason::LifecycleIncomplete,
Reason::AttrMalformed,
Reason::MixInputMalformed,
Reason::ReceiptInvalid,
Reason::DescriptorMismatch,
Reason::CompositorNotPermitted,
Reason::OutputNotPublished,
Reason::StateMismatch,
Reason::InputsMismatch,
];
/// The reason's code.
pub fn code(self) -> &'static str {
match self {
Reason::Room(r) => r.code(),
Reason::CanvasOutOfRange => "canvas_out_of_range",
Reason::ComposeInvalid => "compose_invalid",
Reason::ComposeNotPermitted => "compose_not_permitted",
Reason::NotYetEffective => "not_yet_effective",
Reason::LifecycleIncomplete => "lifecycle_incomplete",
Reason::AttrMalformed => "attr_malformed",
Reason::MixInputMalformed => "mix_input_malformed",
Reason::ReceiptInvalid => "receipt_invalid",
Reason::DescriptorMismatch => "descriptor_mismatch",
Reason::CompositorNotPermitted => "compositor_not_permitted",
Reason::OutputNotPublished => "output_not_published",
Reason::StateMismatch => "state_mismatch",
Reason::InputsMismatch => "inputs_mismatch",
}
}
}
impl std::fmt::Display for Reason {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(self.code())
}
}
impl std::error::Error for Reason {}
/// Where a verification stopped (§9).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Place {
/// The room: where live-rooms §9 refused it, or none for the room as a whole.
Room(Option<At>),
Descriptor,
Receipt,
/// The participant set or the layout at this instant.
Set(u64),
}
// ---------------------------------------------------------------------------------------
// Reading
// ---------------------------------------------------------------------------------------
fn sha256(parts: &[&[u8]]) -> [u8; 32] {
let mut h = Sha256::new();
for p in parts {
h.update(p);
}
h.finalize().into()
}
/// A cursor; running out is `truncated`.
struct Rd<'a> {
b: &'a [u8],
p: usize,
}
impl<'a> Rd<'a> {
fn take(&mut self, n: usize) -> Result<&'a [u8], Reason> {
let e = self.p.checked_add(n).filter(|&e| e <= self.b.len()).ok_or(TRUNCATED)?;
let s = &self.b[self.p..e];
self.p = e;
Ok(s)
}
fn arr<const N: usize>(&mut self) -> Result<[u8; N], Reason> {
Ok(self.take(N)?.try_into().expect("N bytes"))
}
fn u8(&mut self) -> Result<u8, Reason> {
Ok(self.take(1)?[0])
}
fn u16(&mut self) -> Result<u16, Reason> {
Ok(u16::from_le_bytes(self.arr()?))
}
fn u32(&mut self) -> Result<u32, Reason> {
Ok(u32::from_le_bytes(self.arr()?))
}
fn u64(&mut self) -> Result<u64, Reason> {
Ok(u64::from_le_bytes(self.arr()?))
}
/// A short field: a u8 length and that many bytes.
fn short(&mut self) -> Result<&'a [u8], Reason> {
let n = self.u8()? as usize;
self.take(n)
}
}
/// live-events §3.3's first two steps, with `magic` for `WEV1`.
fn read_head(b: &[u8], magic: &[u8; 4]) -> Result<(), Reason> {
if b.len() < 4 {
return Err(TRUNCATED);
}
if b[..4] != *magic {
return Err(MALFORMED);
}
if b.len() < 5 {
return Err(TRUNCATED);
}
if b[4] != RECORD_VERSION {
return Err(UNKNOWN_VERSION);
}
Ok(())
}
fn verify_sig(key: &[u8; 32], domain: &[u8], body: &[u8], sig: &[u8; 64]) -> Result<(), Reason> {
let mut m = domain.to_vec();
m.extend_from_slice(body);
crate::sig::verify_strict(key, &m, sig).map_err(|e| match e {
crate::sig::SigError::BadKey => events::Reason::BadKey.into(),
crate::sig::SigError::BadSignature => events::Reason::BadSignature.into(),
})
}
fn put_short(o: &mut Vec<u8>, s: &[u8]) {
assert!(s.len() <= u8::MAX as usize, "a short field is at most 255 bytes");
o.push(s.len() as u8);
o.extend_from_slice(s);
}
fn u16_of(v: u32) -> [u8; 2] {
u16::try_from(v).expect("a field of 16 bits").to_le_bytes()
}
// ---------------------------------------------------------------------------------------
// §3 The composition descriptor
// ---------------------------------------------------------------------------------------
/// A composition descriptor (§3.1).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Compose {
/// The room id (live-rooms §3.2).
pub room: [u8; 32],
/// The key that signs it: the room authority, or a principal with the compose power.
pub signer: [u8; 32],
/// The room-clock instant from which it applies.
pub effective: u64,
/// The principal that may publish the composed tracks; [`NO_COMPOSITOR`] for none.
pub compositor: [u8; 32],
/// sRGB, 8 bits each of red, green and blue.
pub background: [u8; 3],
/// The highest layer composed (§4.2).
pub layer: u8,
/// Everything else, as the layout reads it.
pub params: Params,
pub sig: [u8; 64],
}
impl Compose {
/// The descriptor without `sig` (§3.1).
pub fn body(&self) -> Vec<u8> {
let p = &self.params;
let mut o = Vec::with_capacity(256);
o.extend_from_slice(&DESCRIPTOR_MAGIC);
o.push(RECORD_VERSION);
o.extend_from_slice(&self.room);
o.extend_from_slice(&self.signer);
o.extend_from_slice(&self.effective.to_le_bytes());
o.extend_from_slice(&self.compositor);
o.extend_from_slice(&u16_of(p.width));
o.extend_from_slice(&u16_of(p.height));
o.push(p.fps);
o.extend_from_slice(&self.background);
o.push(self.layer);
o.push(p.layout);
o.extend_from_slice(&u16_of(p.gap));
o.push(p.aspect);
o.push(p.fill);
put_short(&mut o, p.featured_attr.as_bytes());
put_short(&mut o, p.order_attr.as_bytes());
o.push(p.omit_stopped as u8);
put_short(&mut o, p.pip_attr.as_bytes());
o.push(p.corner);
o.extend_from_slice(&u16_of(p.pip_width));
o.extend_from_slice(&u16_of(p.pip_height));
o.extend_from_slice(&u16_of(p.offset));
o.push(p.behaviour);
o.push(p.channels);
o.extend_from_slice(&p.sample_rate.to_le_bytes());
put_short(&mut o, p.gain_attr.as_bytes());
put_short(&mut o, p.pan_attr.as_bytes());
o.push(u8::try_from(p.max_tiles).expect("max_tiles fits a byte"));
o.push(u8::try_from(p.max_voices).expect("max_voices fits a byte"));
o
}
pub fn to_bytes(&self) -> Vec<u8> {
let mut o = self.body();
o.extend_from_slice(&self.sig);
o
}
/// The descriptor's digest (§3.2): SHA-256 over [`Compose::body`].
pub fn digest(&self) -> [u8; 32] {
sha256(&[&self.body()])
}
/// Signed under `key`; `signer` is left as it is.
pub fn sign(mut self, key: &SigningKey) -> Compose {
let mut m = DESCRIPTOR_DOMAIN.to_vec();
m.extend_from_slice(&self.body());
self.sig = key.sign(&m).to_bytes();
self
}
/// Read a descriptor (§3.3): the record, its canvas, its field rules, its signature.
pub fn parse(b: &[u8]) -> Result<Compose, Reason> {
read_head(b, &DESCRIPTOR_MAGIC)?;
let mut r = Rd { b, p: 5 };
let room = r.arr()?;
let signer = r.arr()?;
let effective = r.u64()?;
let compositor = r.arr()?;
let width = r.u16()? as u32;
let height = r.u16()? as u32;
let fps = r.u8()?;
let background = r.arr()?;
let layer = r.u8()?;
let layout = r.u8()?;
let gap = r.u16()? as u32;
let aspect = r.u8()?;
let fill = r.u8()?;
let featured = r.short()?;
let order = r.short()?;
let omit = r.u8()?;
let pip_attr = r.short()?;
let corner = r.u8()?;
let pip_width = r.u16()? as u32;
let pip_height = r.u16()? as u32;
let offset = r.u16()? as u32;
let behaviour = r.u8()?;
let channels = r.u8()?;
let sample_rate = r.u32()?;
let gain = r.short()?;
let pan = r.short()?;
let max_tiles = r.u8()? as u32;
let max_voices = r.u8()? as u32;
let sig = r.arr()?;
if r.p != b.len() {
return Err(MALFORMED);
}
let keys = [featured, order, pip_attr, gain, pan];
let keys_ok = keys.iter().all(|k| k.is_empty() || layout::key_ok(k));
let s = |k: &[u8]| String::from_utf8_lossy(k).into_owned();
let params = Params {
width,
height,
fps,
layout,
gap,
aspect,
fill,
featured_attr: s(featured),
order_attr: s(order),
omit_stopped: omit == 1,
pip_attr: s(pip_attr),
corner,
pip_width,
pip_height,
offset,
behaviour,
channels,
sample_rate,
gain_attr: s(gain),
pan_attr: s(pan),
max_tiles,
max_voices,
};
params.check_canvas()?;
if !keys_ok || omit > 1 {
return Err(Reason::ComposeInvalid);
}
params.check_fields()?;
let c = Compose { room, signer, effective, compositor, background, layer, params, sig };
verify_sig(&c.signer, DESCRIPTOR_DOMAIN, &b[..b.len() - 64], &c.sig)?;
debug_assert_eq!(c.body(), b[..b.len() - 64], "a descriptor that reads re-encodes to its bytes");
Ok(c)
}
}
// ---------------------------------------------------------------------------------------
// §4 The participant set
// ---------------------------------------------------------------------------------------
/// A room as live-rooms §9 is given it.
#[derive(Clone, Copy, Debug)]
pub struct RoomInput<'a> {
pub descriptor: &'a [u8],
pub grants: &'a [Vec<u8>],
pub head: Option<&'a str>,
pub chains: &'a [Vec<Vec<u8>>],
}
/// The room at one instant (§4.1): its state, and each present principal's join order.
#[derive(Clone, Debug)]
pub struct Instant {
pub state: RoomState,
pub joins: BTreeMap<[u8; 32], u32>,
}
/// A room that verified (§9 step 1), with what composition reads of it.
pub struct VerifiedRoom<'a> {
pub input: RoomInput<'a>,
pub run: RoomRun,
pub d: Descriptor,
pub id: [u8; 32],
}
impl<'a> VerifiedRoom<'a> {
/// live-rooms §9 over the room: its refusal; `lifecycle_incomplete` when a room key's
/// chain has a gap (§4.1). A gap in another principal's chain is not a refusal here.
pub fn verify(input: RoomInput<'a>) -> Result<VerifiedRoom<'a>, (Place, Reason)> {
let run = verify_room(input.descriptor, input.grants, input.head, input.chains);
if let RoomOutcome::Refuse { at, reason } = run.outcome {
return Err((Place::Room(Some(at)), reason.into()));
}
let d = run.descriptor.clone().expect("a room that is not refused read its descriptor");
if let RoomOutcome::Gap(gaps) = &run.outcome
&& gaps.iter().any(|g| d.is_room_key(&g.principal))
{
return Err((Place::Room(None), Reason::LifecycleIncomplete));
}
let id = d.room_id();
Ok(VerifiedRoom { input, run, d, id })
}
/// The grant a digest names.
pub fn grant(&self, digest: &[u8; 32]) -> Option<&Grant> {
self.run.grants.iter().find(|h| h.digest == *digest).map(|h| &h.grant)
}
/// The room at `t` (§4.1): live-rooms §9 over each chain's records up to the first
/// whose `t` is above `t`, and the join order of each `join` among them.
pub fn at(&self, t: u64) -> Instant {
let chains: Vec<Vec<Vec<u8>>> = self
.input
.chains
.iter()
.map(|c| c.iter().take_while(|r| Event::parse(r).is_ok_and(|e| e.t <= t)).cloned().collect())
.collect();
let run = verify_room(self.input.descriptor, self.input.grants, self.input.head, &chains);
let mut joins = BTreeMap::new();
let mut seen = BTreeSet::new();
let mut n = 0u32;
for &(c, i, _) in &run.order {
let Ok(ev) = Event::parse(&chains[c][i]) else { continue };
if !seen.insert(ev.digest()) || ev.kind != kind::JOIN || !self.d.is_room_key(&ev.principal) {
continue;
}
if let Ok(Payload::Lifecycle { subject, .. }) = events::parse_payload(ev.kind, &ev.payload, u16::MAX) {
joins.insert(subject, n);
n += 1;
}
}
Instant { state: run.state, joins }
}
/// The grant in effect of `p` at `at`, if `p` is present.
pub fn grant_in_effect(&self, at: &Instant, p: &[u8; 32]) -> Option<&Grant> {
at.state.present.get(p).and_then(|g| self.grant(g))
}
}
/// A participant's video or audio source (§4.2).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Source {
/// Empty for the participant's own track; otherwise the slot it is live in.
pub slot: String,
pub media: u8,
/// The index of its profile in the room's descriptor, and the layers it publishes.
pub profile: u8,
pub layers: u8,
/// The layer composed: the lower of `layers` − 1 and the descriptor's `layer`.
pub layer: u8,
pub layering: u8,
/// That layer's dimensions in the profile; 0 for audio.
pub width: u16,
pub height: u16,
}
impl Source {
/// The tracks a compositor reads it from (§8.2), each as an anchor's `track` digest
/// (live-rooms §6.2): track 0 for `single` and `svc`, track `layer` for `simulcast`,
/// and tracks 0 to `layer` for `staged`.
pub fn tracks(&self, room_id: &[u8; 32], principal: &[u8; 32]) -> Vec<[u8; 32]> {
let owner = if self.slot.is_empty() { Owner::Principal(*principal) } else { Owner::Slot(self.slot.clone()) };
let ns = render_namespace(&track_namespace(room_id, &owner, media_name(self.media).expect("a media kind")));
let layers: Vec<u8> = match self.layering {
layering::SIMULCAST => vec![self.layer],
layering::STAGED => (0..=self.layer).collect(),
_ => vec![0],
};
layers.into_iter().map(|l| anchor_track(&ns, &track_name(l))).collect()
}
fn encode(s: Option<&Source>, o: &mut Vec<u8>) {
match s {
None => o.push(0),
Some(s) => {
o.push(s.media);
put_short(o, s.slot.as_bytes());
o.extend_from_slice(&[s.profile, s.layers, s.layer]);
o.extend_from_slice(&s.width.to_le_bytes());
o.extend_from_slice(&s.height.to_le_bytes());
}
}
}
}
/// One participant of the set (§4.2).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Participant {
pub principal: [u8; 32],
/// The digest of its grant in effect.
pub grant: [u8; 32],
pub join: u32,
/// Its grant's attributes (live-rooms §4.4).
pub attrs: Vec<(String, String)>,
pub video: Option<Source>,
pub audio: Option<Source>,
}
fn source(d: &Descriptor, c: &Compose, held: &Held, slot: &str, m: u8) -> Source {
let p = &d.profiles[held.profile as usize];
let layer = (held.layers - 1).min(c.layer);
let l = &p.layers[layer as usize];
Source { slot: slot.into(), media: m, profile: held.profile, layers: held.layers, layer, layering: p.layering, width: l.width, height: l.height }
}
/// §4.2: the participant set of `c` at `at`, in ascending order of key.
pub fn participants(c: &Compose, vr: &VerifiedRoom, at: &Instant) -> Vec<Participant> {
let st = &at.state;
let d = &vr.d;
let live_in = |p: &[u8; 32], kinds: &[u8]| -> Option<Source> {
d.slots.iter().zip(&st.slots).find_map(|(s, ss)| match ss.live {
Some(h) if h.principal == *p && kinds.contains(&s.media) => Some(source(d, c, &h, &s.name, s.media)),
_ => None,
})
};
let own = |p: &[u8; 32], m: u8| st.own.get(&(*p, m)).map(|h| source(d, c, h, "", m));
let mut out = Vec::new();
for (p, g) in &st.present {
if *p == c.compositor {
continue;
}
let publishes = st.own.keys().any(|(q, _)| q == p) || st.slots.iter().any(|s| s.live.is_some_and(|h| h.principal == *p));
if !publishes {
continue;
}
let video = own(p, media::VIDEO).or_else(|| own(p, media::SCREEN)).or_else(|| live_in(p, &[media::VIDEO, media::SCREEN]));
let audio = own(p, media::AUDIO).or_else(|| live_in(p, &[media::AUDIO]));
let attrs = vr.grant(g).map(|g| g.attrs.clone()).unwrap_or_default();
let join = *at.joins.get(p).expect("a present principal joined");
out.push(Participant { principal: *p, grant: *g, join, attrs, video, audio });
}
out
}
/// §4.5: the digest of the participant set at `t`.
pub fn set_digest(room_id: &[u8; 32], t: u64, ps: &[Participant]) -> [u8; 32] {
let mut o = Vec::new();
o.extend_from_slice(SET_DOMAIN);
o.extend_from_slice(room_id);
o.extend_from_slice(&t.to_le_bytes());
o.extend_from_slice(&(ps.len() as u32).to_le_bytes());
for p in ps {
o.extend_from_slice(&p.principal);
o.extend_from_slice(&p.grant);
o.extend_from_slice(&p.join.to_le_bytes());
Source::encode(p.video.as_ref(), &mut o);
Source::encode(p.audio.as_ref(), &mut o);
}
sha256(&[&o])
}
/// The composition at one instant: the set, its digest, and the layout and voices.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Frame {
pub t: u64,
pub participants: Vec<Participant>,
pub set_digest: [u8; 32],
pub layout: Layout,
}
impl Frame {
fn participant(&self, p: &[u8; 32]) -> &Participant {
self.participants.iter().find(|q| q.principal == *p).expect("a tile's participant is in the set")
}
/// §8.2: the inputs a composed track of `m` reads, in order, as (principal, track):
/// for video, each tile that shows video, in drawing order; for audio, each voice.
pub fn inputs(&self, room_id: &[u8; 32], m: u8) -> Vec<([u8; 32], [u8; 32])> {
let mut out = Vec::new();
let sources: Vec<(&[u8; 32], Option<&Source>)> = if m == media::AUDIO {
self.layout.voices.iter().map(|v| (&v.principal, self.participant(&v.principal).audio.as_ref())).collect()
} else {
self.layout.tiles.iter().filter(|t| t.video.is_some()).map(|t| (&t.principal, self.participant(&t.principal).video.as_ref())).collect()
};
for (p, s) in sources {
for tr in s.expect("a composed source").tracks(room_id, p) {
out.push((*p, tr));
}
}
out
}
}
/// §4, §5 at `t` over the room as it is at `at`.
fn frame_at(c: &Compose, vr: &VerifiedRoom, t: u64, at: &Instant) -> Result<Frame, Reason> {
if t < c.effective {
return Err(Reason::NotYetEffective);
}
let participants = participants(c, vr, at);
let entrants: Vec<Entrant> = participants
.iter()
.map(|p| Entrant {
principal: p.principal,
join: p.join,
video: p.video.as_ref().map(|s| (s.width as u32, s.height as u32)),
audio: p.audio.is_some(),
attrs: &p.attrs,
})
.collect();
let layout = layout::layout(&c.params, &entrants)?;
Ok(Frame { t, set_digest: set_digest(&vr.id, t, &participants), participants, layout })
}
/// The composition of `c` at `t` (§4, §5).
pub fn compose_at(c: &Compose, vr: &VerifiedRoom, t: u64) -> Result<Frame, Reason> {
frame_at(c, vr, t, &vr.at(t))
}
/// §3.3, §3.5: a descriptor read, of this room, signed by its authority or by a principal
/// whose grant in effect at its `effective` has the compose power.
pub fn check_descriptor(b: &[u8], vr: &VerifiedRoom) -> Result<Compose, Reason> {
let c = Compose::parse(b)?;
if c.room != vr.id {
return Err(WRONG_ROOM);
}
if c.signer != vr.d.authority {
let at = vr.at(c.effective);
if !vr.grant_in_effect(&at, &c.signer).is_some_and(|g| g.caps.powers & power::COMPOSE != 0) {
return Err(Reason::ComposeNotPermitted);
}
}
Ok(c)
}
/// The outcome of composing a room at a list of instants.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ComposeOutcome {
Frames(Vec<Frame>),
Refuse { at: Place, reason: Reason },
}
/// §9: the room, the descriptor, then the composition at each instant.
pub fn verify_composition(room: RoomInput, descriptor: &[u8], instants: &[u64]) -> (ComposeOutcome, Option<[u8; 32]>) {
let vr = match VerifiedRoom::verify(room) {
Ok(v) => v,
Err((at, reason)) => return (ComposeOutcome::Refuse { at, reason }, None),
};
let c = match check_descriptor(descriptor, &vr) {
Ok(c) => c,
Err(reason) => return (ComposeOutcome::Refuse { at: Place::Descriptor, reason }, None),
};
let mut frames = Vec::with_capacity(instants.len());
for &t in instants {
match compose_at(&c, &vr, t) {
Ok(f) => frames.push(f),
Err(reason) => return (ComposeOutcome::Refuse { at: Place::Set(t), reason }, Some(c.digest())),
}
}
(ComposeOutcome::Frames(frames), Some(c.digest()))
}
// ---------------------------------------------------------------------------------------
// §8 The composition receipt
// ---------------------------------------------------------------------------------------
/// One MoQ object (§8.1): its track, as an anchor's `track` digest, its Location, and the
/// SHA-256 of its payload.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ObjectRef {
pub track: [u8; 32],
pub group: u64,
pub object: u64,
pub digest: [u8; 32],
}
/// One input object, and the participant it is read from.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct InputRef {
pub principal: [u8; 32],
pub object: ObjectRef,
}
/// A composition receipt (§8.1).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Receipt {
pub room: [u8; 32],
pub compositor: [u8; 32],
/// The descriptor's digest.
pub descriptor: [u8; 32],
/// The room-clock instant of the output object.
pub t: u64,
/// The participant set's digest at `t` (§4.5).
pub state: [u8; 32],
/// 1 audio (a mix) or 2 video (a layout).
pub media: u8,
pub output: ObjectRef,
pub inputs: Vec<InputRef>,
pub sig: [u8; 64],
}
fn put_object(o: &mut Vec<u8>, r: &ObjectRef) {
o.extend_from_slice(&r.track);
o.extend_from_slice(&r.group.to_le_bytes());
o.extend_from_slice(&r.object.to_le_bytes());
o.extend_from_slice(&r.digest);
}
fn read_object(r: &mut Rd) -> Result<ObjectRef, Reason> {
Ok(ObjectRef { track: r.arr()?, group: r.u64()?, object: r.u64()?, digest: r.arr()? })
}
impl Receipt {
/// The receipt without `sig` (§8.1).
pub fn body(&self) -> Vec<u8> {
let mut o = Vec::new();
o.extend_from_slice(&RECEIPT_MAGIC);
o.push(RECORD_VERSION);
o.extend_from_slice(&self.room);
o.extend_from_slice(&self.compositor);
o.extend_from_slice(&self.descriptor);
o.extend_from_slice(&self.t.to_le_bytes());
o.extend_from_slice(&self.state);
o.push(self.media);
put_object(&mut o, &self.output);
assert!(self.inputs.len() <= u8::MAX as usize, "at most 255 inputs");
o.push(self.inputs.len() as u8);
for i in &self.inputs {
o.extend_from_slice(&i.principal);
put_object(&mut o, &i.object);
}
o
}
pub fn to_bytes(&self) -> Vec<u8> {
let mut o = self.body();
o.extend_from_slice(&self.sig);
o
}
/// The receipt's digest (§8.1).
pub fn digest(&self) -> [u8; 32] {
sha256(&[&self.body()])
}
/// Signed under `key`; `compositor` is left as it is.
pub fn sign(mut self, key: &SigningKey) -> Receipt {
let mut m = RECEIPT_DOMAIN.to_vec();
m.extend_from_slice(&self.body());
self.sig = key.sign(&m).to_bytes();
self
}
/// Read a receipt (§8.3 step 1): the record, its rules, its signature.
pub fn parse(b: &[u8]) -> Result<Receipt, Reason> {
read_head(b, &RECEIPT_MAGIC)?;
let mut r = Rd { b, p: 5 };
let room = r.arr()?;
let compositor = r.arr()?;
let descriptor = r.arr()?;
let t = r.u64()?;
let state = r.arr()?;
let media = r.u8()?;
let output = read_object(&mut r)?;
let n = r.u8()? as usize;
let mut inputs = Vec::with_capacity(n);
for _ in 0..n {
inputs.push(InputRef { principal: r.arr()?, object: read_object(&mut r)? });
}
let sig = r.arr()?;
if r.p != b.len() {
return Err(MALFORMED);
}
let mut pairs = BTreeSet::new();
if !matches!(media, media::AUDIO | media::VIDEO) || !inputs.iter().all(|i| pairs.insert((i.principal, i.object.track))) {
return Err(Reason::ReceiptInvalid);
}
let rc = Receipt { room, compositor, descriptor, t, state, media, output, inputs, sig };
verify_sig(&rc.compositor, RECEIPT_DOMAIN, &b[..b.len() - 64], &rc.sig)?;
Ok(rc)
}
}
/// The outcome of verifying a receipt (§8.3).
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ReceiptOutcome {
Accept { digest: [u8; 32], frame: Frame },
Refuse { at: Place, reason: Reason },
}
/// §8.3: a receipt checked against the room and the descriptor it names.
pub fn verify_receipt(room: RoomInput, descriptor: &[u8], receipt: &[u8]) -> ReceiptOutcome {
let refuse = |at, reason| ReceiptOutcome::Refuse { at, reason };
let vr = match VerifiedRoom::verify(room) {
Ok(v) => v,
Err((at, reason)) => return refuse(at, reason),
};
let c = match check_descriptor(descriptor, &vr) {
Ok(c) => c,
Err(reason) => return refuse(Place::Descriptor, reason),
};
let r = match Receipt::parse(receipt) {
Ok(r) => r,
Err(reason) => return refuse(Place::Receipt, reason),
};
if r.room != vr.id {
return refuse(Place::Receipt, WRONG_ROOM);
}
if r.descriptor != c.digest() {
return refuse(Place::Receipt, Reason::DescriptorMismatch);
}
if r.t < c.effective {
return refuse(Place::Receipt, Reason::NotYetEffective);
}
let at = vr.at(r.t);
let composes = vr.grant_in_effect(&at, &r.compositor).is_some_and(|g| g.caps.powers & power::COMPOSE != 0);
if c.compositor == NO_COMPOSITOR || r.compositor != c.compositor || !composes {
return refuse(Place::Receipt, Reason::CompositorNotPermitted);
}
let ns = render_namespace(&track_namespace(&vr.id, &Owner::Principal(r.compositor), media_name(r.media).expect("a media kind")));
let published = at.state.own.get(&(r.compositor, r.media)).is_some_and(|h| (0..h.layers).any(|l| anchor_track(&ns, &track_name(l)) == r.output.track));
if !published {
return refuse(Place::Receipt, Reason::OutputNotPublished);
}
let frame = match frame_at(&c, &vr, r.t, &at) {
Ok(f) => f,
Err(reason) => return refuse(Place::Set(r.t), reason),
};
if r.state != frame.set_digest {
return refuse(Place::Receipt, Reason::StateMismatch);
}
let stated: Vec<([u8; 32], [u8; 32])> = r.inputs.iter().map(|i| (i.principal, i.object.track)).collect();
if stated != frame.inputs(&vr.id, r.media) {
return refuse(Place::Receipt, Reason::InputsMismatch);
}
ReceiptOutcome::Accept { digest: r.digest(), frame }
}
#[cfg(test)]
mod tests {
use super::*;
/// The corpus regenerates byte for byte, and this module gives every case of it the
/// outcome it states.
#[test]
fn corpus_round_trip() {
let doc = corpus::generate();
let fails = corpus::verify(&doc);
assert!(fails.is_empty(), "{}", fails.join("\n"));
}
/// A descriptor re-encodes to the bytes it was read from, and one bit of its body
/// changed fails its signature.
#[test]
fn descriptor_round_trip() {
let key = SigningKey::from_bytes(&[7; 32]);
let c = Compose {
room: [1; 32],
signer: key.verifying_key().to_bytes(),
effective: 0,
compositor: NO_COMPOSITOR,
background: [0, 0, 0],
layer: 2,
params: corpus::grid_params(),
sig: [0; 64],
}
.sign(&key);
let b = c.to_bytes();
assert_eq!(Compose::parse(&b), Ok(c));
let mut bad = b.clone();
bad[40] ^= 1;
assert_eq!(Compose::parse(&bad), Err(events::Reason::BadSignature.into()));
assert_eq!(Compose::parse(&b[..b.len() - 1]), Err(TRUNCATED));
}
}

1324
wai/wai-rs/src/live_compose/corpus.rs Normal file
View file

File diff suppressed because it is too large Load diff

748
wai/wai-rs/src/live_compose/layout.rs Normal file
View file

@ -0,0 +1,748 @@
//! The layout function and the mixer of live-compose §5 and §6.
//!
//! Both are pure integer functions of their arguments: no floating point, no clock, no
//! I/O, no hashing, and nothing beyond `core`, `Vec` and `String`, so that a later slice
//! can build this file for the browser unchanged. Every division here is of non-negative
//! integers and rounds down; the mixer's one shift of a signed sum is stated where it is
//! made. [`layout`] is §5 over the participants §4 derives from a room; [`mix`] is §6.
use core::cmp::{max, min, Ordering};
/// The longest canvas edge, in pixels (§3.4).
pub const MAX_EDGE: u32 = 1920;
/// The shortest canvas edge, in pixels (§3.4).
pub const MIN_EDGE: u32 = 16;
/// The most pixels a canvas holds (§3.4): 1920 × 1080.
pub const MAX_PIXELS: u32 = 2_073_600;
/// The highest frame rate (§3.4).
pub const MAX_FPS: u8 = 60;
/// The most tiles, and the most voices, a descriptor allows (§3.3).
pub const MAX_COMPOSED: u32 = 64;
/// A gain of one (§6): gains are in units of 1/4096.
pub const UNITY_GAIN: u32 = 4096;
/// The highest gain (§6): four.
pub const MAX_GAIN: u32 = 16_384;
/// The pan of a source hard right (§6); hard left is its negation.
pub const MAX_PAN: i32 = 4096;
/// The sample rates a descriptor may state (§3.3), in Hz.
pub const SAMPLE_RATES: core::ops::RangeInclusive<u32> = 8_000..=192_000;
/// The layout kinds (§3.1).
pub mod kind {
pub const GRID: u8 = 1;
pub const PIP: u8 = 2;
}
/// The tile aspects (§5.2): `AUTO` lays tiles out as `VIDEO` does and lets each fill its
/// cell.
pub mod aspect {
pub const AUTO: u8 = 0;
pub const VIDEO: u8 = 1;
pub const SQUARE: u8 = 2;
pub const PORTRAIT: u8 = 3;
}
/// The fill modes (§5.6); `DEFAULT` is cover for grid tiles and the PiP window, and
/// contain for the featured tile.
pub mod fill {
pub const DEFAULT: u8 = 0;
pub const FILL: u8 = 1;
pub const COVER: u8 = 2;
pub const CONTAIN: u8 = 3;
}
/// The PiP window's corners (§5.5).
pub mod corner {
pub const TOP_LEFT: u8 = 1;
pub const TOP_RIGHT: u8 = 2;
pub const BOTTOM_LEFT: u8 = 3;
pub const BOTTOM_RIGHT: u8 = 4;
}
/// What the PiP participant does when it is the only one composed (§5.5).
pub mod behaviour {
pub const STATIC: u8 = 1;
pub const DYNAMIC: u8 = 2;
}
/// Where a tile is placed (§5.7).
pub mod place {
pub const GRID: u8 = 0;
pub const FEATURED: u8 = 1;
pub const PIP: u8 = 2;
/// Each place's name, as the corpus writes it.
pub fn name(p: u8) -> &'static str {
match p {
FEATURED => "featured",
PIP => "pip",
_ => "grid",
}
}
}
/// Why the layout or the mix is refused (§10).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Refusal {
CanvasOutOfRange,
ComposeInvalid,
AttrMalformed,
MixInputMalformed,
}
/// A rectangle, in pixels: its left edge, its top edge, its width and its height.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)]
pub struct Rect {
pub x: u32,
pub y: u32,
pub w: u32,
pub h: u32,
}
impl Rect {
pub fn new(x: u32, y: u32, w: u32, h: u32) -> Rect {
Rect { x, y, w, h }
}
}
/// What the layout and the mix read of a descriptor (§3.1), its fields as integers.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Params {
pub width: u32,
pub height: u32,
pub fps: u8,
pub layout: u8,
pub gap: u32,
pub aspect: u8,
pub fill: u8,
pub featured_attr: String,
pub order_attr: String,
pub omit_stopped: bool,
pub pip_attr: String,
pub corner: u8,
pub pip_width: u32,
pub pip_height: u32,
pub offset: u32,
pub behaviour: u8,
pub channels: u8,
pub sample_rate: u32,
pub gain_attr: String,
pub pan_attr: String,
pub max_tiles: u32,
pub max_voices: u32,
}
/// An attribute key (live-rooms §4.4): 1 to 32 of `a`–`z`, `0`–`9` and `_`.
pub fn key_ok(k: &[u8]) -> bool {
(1..=32).contains(&k.len()) && k.iter().all(|&c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == b'_')
}
/// The least `k` with `k × k` ≥ `n`.
fn ceil_sqrt(n: u64) -> u64 {
let mut k = 0;
while k * k < n {
k += 1;
}
k
}
/// The height of the featured tile's region (§5.4): three quarters of what the gap leaves,
/// rounded down. `gap` < `height`.
fn main_height(height: u64, gap: u64) -> u64 {
(height - gap) * 3 / 4
}
impl Params {
/// §3.4: the canvas within its limits. Otherwise `canvas_out_of_range`.
pub fn check_canvas(&self) -> Result<(), Refusal> {
let edge = |e: u32| (MIN_EDGE..=MAX_EDGE).contains(&e) && e.is_multiple_of(2);
if !edge(self.width) || !edge(self.height) || self.width * self.height > MAX_PIXELS || !(1..=MAX_FPS).contains(&self.fps) {
return Err(Refusal::CanvasOutOfRange);
}
Ok(())
}
/// §3.3's field rules after the canvas's. Otherwise `compose_invalid`.
pub fn check_fields(&self) -> Result<(), Refusal> {
let bad = Err(Refusal::ComposeInvalid);
let (w, h, g) = (self.width as u64, self.height as u64, self.gap as u64);
if !matches!(self.layout, kind::GRID | kind::PIP) || self.aspect > aspect::PORTRAIT || self.fill > fill::CONTAIN {
return bad;
}
for k in [&self.featured_attr, &self.order_attr, &self.pip_attr, &self.gain_attr, &self.pan_attr] {
if !k.is_empty() && !key_ok(k.as_bytes()) {
return bad;
}
}
if self.layout == kind::GRID {
if !self.pip_attr.is_empty() || self.corner != 0 || self.pip_width != 0 || self.pip_height != 0 || self.offset != 0 || self.behaviour != 0 {
return bad;
}
} else {
let o = self.offset as u64;
if !(corner::TOP_LEFT..=corner::BOTTOM_RIGHT).contains(&self.corner)
|| !(behaviour::STATIC..=behaviour::DYNAMIC).contains(&self.behaviour)
|| o >= min(w, h)
|| self.pip_width as u64 + o > w
|| self.pip_height as u64 + o > h
{
return bad;
}
}
if !(1..=2).contains(&self.channels) || !SAMPLE_RATES.contains(&self.sample_rate) {
return bad;
}
if !(1..=MAX_COMPOSED).contains(&self.max_tiles) || !(1..=MAX_COMPOSED).contains(&self.max_voices) {
return bad;
}
// Every grid the descriptor can lay out has cells of at least one pixel (§3.3).
let k = ceil_sqrt(self.max_tiles as u64);
let need = k + (k - 1) * g;
if g >= h || w < need || h - g - main_height(h, g) < need {
return bad;
}
Ok(())
}
/// Both.
pub fn check(&self) -> Result<(), Refusal> {
self.check_canvas()?;
self.check_fields()
}
}
// ---------------------------------------------------------------------------------------
// §4.3 Attributes
// ---------------------------------------------------------------------------------------
/// The value of `key` among `attrs`; none when `key` is empty or not there.
pub fn attr<'a>(attrs: &'a [(String, String)], key: &str) -> Option<&'a str> {
if key.is_empty() {
return None;
}
attrs.iter().find(|(k, _)| k == key).map(|(_, v)| v.as_str())
}
/// `true` or `false`, exactly (§4.3); absent is `false`.
pub fn read_bool(v: Option<&str>) -> Result<bool, Refusal> {
match v {
None | Some("false") => Ok(false),
Some("true") => Ok(true),
Some(_) => Err(Refusal::AttrMalformed),
}
}
/// A decimal integer (§4.3): `0`, or an optional `-` and 1 to 10 digits not starting with
/// `0`.
pub fn read_int(v: &str) -> Option<i64> {
let b = v.as_bytes();
let (neg, digits) = match b.first() {
Some(b'-') => (true, &b[1..]),
_ => (false, b),
};
if digits.is_empty() || digits.len() > 10 || !digits.iter().all(u8::is_ascii_digit) {
return None;
}
if digits[0] == b'0' && (digits.len() > 1 || neg) {
return None;
}
let n = digits.iter().fold(0i64, |n, &d| n * 10 + (d - b'0') as i64);
Some(if neg { -n } else { n })
}
fn read_ranged(v: Option<&str>, lo: i64, hi: i64, absent: i64) -> Result<i64, Refusal> {
match v {
None => Ok(absent),
Some(s) => read_int(s).filter(|n| (lo..=hi).contains(n)).ok_or(Refusal::AttrMalformed),
}
}
/// One participant as the layout reads it (§4.2).
#[derive(Clone, Copy, Debug)]
pub struct Entrant<'a> {
pub principal: [u8; 32],
/// Its join order (§4.2).
pub join: u32,
/// Its video source's width and height, each at least 1; none when its video is
/// stopped.
pub video: Option<(u32, u32)>,
/// Whether it has an audio source.
pub audio: bool,
/// The attributes of its grant in effect (live-rooms §4.4).
pub attrs: &'a [(String, String)],
}
/// The attributes a descriptor reads, read for one participant (§4.3).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Read {
pub featured: bool,
pub order: Option<i32>,
pub pip: bool,
pub gain: u32,
pub pan: i32,
}
/// §4.3 over one participant's attributes: `attr_malformed` when one the descriptor names
/// does not read.
pub fn read_attrs(p: &Params, attrs: &[(String, String)]) -> Result<Read, Refusal> {
let featured = read_bool(attr(attrs, &p.featured_attr))?;
let order = match attr(attrs, &p.order_attr) {
None => None,
Some(v) => Some(read_int(v).filter(|n| (i32::MIN as i64..=i32::MAX as i64).contains(n)).ok_or(Refusal::AttrMalformed)? as i32),
};
let pip = read_bool(attr(attrs, &p.pip_attr))?;
let gain = read_ranged(attr(attrs, &p.gain_attr), 0, MAX_GAIN as i64, UNITY_GAIN as i64)? as u32;
let pan = read_ranged(attr(attrs, &p.pan_attr), -(MAX_PAN as i64), MAX_PAN as i64, 0)? as i32;
Ok(Read { featured, order, pip, gain, pan })
}
// ---------------------------------------------------------------------------------------
// §5 The layout
// ---------------------------------------------------------------------------------------
/// A source's video on the canvas (§5.6): the rectangle it is drawn into, and the
/// rectangle of the source drawn there.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Video {
pub dst: Rect,
pub src: Rect,
}
/// One tile (§5.7).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Tile {
pub principal: [u8; 32],
/// [`place`]: grid, featured or PiP.
pub place: u8,
/// 0 for a grid or featured tile, 1 for the PiP window, drawn over the others.
pub z: u8,
pub tile: Rect,
/// None when the participant's video is stopped: the tile shows the background.
pub video: Option<Video>,
}
/// One voice of the mix (§6): a participant with an audio source, its gain and its pan.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Voice {
pub principal: [u8; 32],
pub gain: u32,
pub pan: i32,
}
/// What the layout function gives (§5.7).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Layout {
/// Every participant, in the order of §4.4, before any is omitted.
pub order: Vec<[u8; 32]>,
/// The tiles, in drawing order: the grid's (the featured tile first), then the PiP
/// window.
pub tiles: Vec<Tile>,
/// The voices, in the order of §4.4.
pub voices: Vec<Voice>,
}
/// The aspect a grid is chosen by (§5.3), as `a`:`b`.
fn reference(a: u8) -> (u64, u64) {
match a {
aspect::SQUARE => (1, 1),
aspect::PORTRAIT => (3, 4),
_ => (16, 9),
}
}
/// The largest rectangle of aspect `a`:`b` within `bw` × `bh`, each side at least 1
/// (§5.1). `bw`, `bh`, `a`, `b` ≥ 1.
pub fn fit(bw: u64, bh: u64, a: u64, b: u64) -> (u64, u64) {
if bw * b <= bh * a {
(bw, max(1, bw * b / a))
} else {
(max(1, bh * a / b), bh)
}
}
fn centred(outer: Rect, w: u64, h: u64) -> Rect {
Rect::new(outer.x + (outer.w - w as u32) / 2, outer.y + (outer.h - h as u32) / 2, w as u32, h as u32)
}
/// §5.3: the tiles of a grid of `n` ≥ 1 over `region`.
fn grid(region: Rect, n: u64, gap: u64, asp: u8) -> Vec<Rect> {
let (a, b) = reference(asp);
let (rw, rh) = (region.w as u64, region.h as u64);
// (area, columns, rows, cell width, cell height) of the best candidate so far.
let mut best: Option<(u64, u64, u64, u64, u64)> = None;
for c in 1..=n {
let r = n.div_ceil(c);
if (c - 1) * gap >= rw || (r - 1) * gap >= rh {
continue;
}
let (cw, ch) = ((rw - (c - 1) * gap) / c, (rh - (r - 1) * gap) / r);
if cw == 0 || ch == 0 {
continue;
}
let (tw, th) = fit(cw, ch, a, b);
if best.is_none_or(|(area, ..)| tw * th >= area) {
best = Some((tw * th, c, r, cw, ch));
}
}
// §3.3's rule on the gap makes the square grid a candidate for every n it allows.
let Some((_, c, r, cw, ch)) = best else { return Vec::new() };
let y0 = region.y as u64 + (rh - (r * ch + (r - 1) * gap)) / 2;
(0..n)
.map(|k| {
let (i, j) = (k / c, k % c);
let in_row = if i == r - 1 { n - (r - 1) * c } else { c };
let x0 = region.x as u64 + (rw - (in_row * cw + (in_row - 1) * gap)) / 2;
let cell = Rect::new((x0 + j * (cw + gap)) as u32, (y0 + i * (ch + gap)) as u32, cw as u32, ch as u32);
if asp == aspect::AUTO {
cell
} else {
let (tw, th) = fit(cw, ch, a, b);
centred(cell, tw, th)
}
})
.collect()
}
/// §5.6: a source of `sw` × `sh` in `tile` by fill mode `mode` (not the default).
pub fn place_video(tile: Rect, sw: u32, sh: u32, mode: u8) -> Video {
let (tw, th) = (tile.w as u64, tile.h as u64);
let (sw, sh) = (max(1, sw) as u64, max(1, sh) as u64);
let whole = Rect::new(0, 0, sw as u32, sh as u32);
let wider = sw * th >= sh * tw;
match mode {
fill::FILL => Video { dst: tile, src: whole },
fill::CONTAIN => {
let dst = if wider {
let dh = max(1, tw * sh / sw);
Rect::new(tile.x, tile.y + ((th - dh) / 2) as u32, tile.w, dh as u32)
} else {
let dw = max(1, th * sw / sh);
Rect::new(tile.x + ((tw - dw) / 2) as u32, tile.y, dw as u32, tile.h)
};
Video { dst, src: whole }
}
_ => {
let src = if wider {
let cw = max(1, sh * tw / th);
Rect::new(((sw - cw) / 2) as u32, 0, cw as u32, sh as u32)
} else {
let ch = max(1, sw * th / tw);
Rect::new(0, ((sh - ch) / 2) as u32, sw as u32, ch as u32)
};
Video { dst: tile, src }
}
}
}
/// One participant, read.
struct Member<'a> {
e: &'a Entrant<'a>,
r: Read,
}
/// §4.4: an order attribute first, ascending; then join order; then the principal's key.
fn by_order(a: &Member, b: &Member) -> Ordering {
(a.r.order.is_none(), a.r.order.unwrap_or(0), a.e.join, a.e.principal).cmp(&(b.r.order.is_none(), b.r.order.unwrap_or(0), b.e.join, b.e.principal))
}
/// The layout function (§5) and the voices of the mix (§6) for `p`, over `entrants` in
/// any order. `p` is checked (§3.3) first.
pub fn layout(p: &Params, entrants: &[Entrant]) -> Result<Layout, Refusal> {
p.check()?;
let mut ms: Vec<Member> = Vec::with_capacity(entrants.len());
for e in entrants {
ms.push(Member { e, r: read_attrs(p, e.attrs)? });
}
ms.sort_by(by_order);
let order = ms.iter().map(|m| m.e.principal).collect();
let voices = ms
.iter()
.filter(|m| m.e.audio)
.take(p.max_voices as usize)
.map(|m| Voice { principal: m.e.principal, gain: m.r.gain, pan: m.r.pan })
.collect();
// §5.2: the tiles' participants.
let shown: Vec<&Member> = ms.iter().filter(|m| !p.omit_stopped || m.e.video.is_some()).collect();
let pip = if p.layout == kind::PIP { shown.iter().position(|m| m.r.pip) } else { None };
let pip_member = pip.map(|i| shown[i]);
let rest: Vec<&Member> = shown.iter().enumerate().filter(|(i, _)| Some(*i) != pip).map(|(_, m)| *m).collect();
let (rest, featured) = featured_first(rest, p.max_tiles as usize - pip.is_some() as usize);
let (on_grid, featured, window) = match pip_member {
// §5.5: alone and dynamic, the PiP participant is laid out as if it had no pip
// attribute.
Some(m) if rest.is_empty() && p.behaviour == behaviour::DYNAMIC => (vec![m], m.r.featured, None),
other => (rest, featured, other),
};
let mut tiles = grid_tiles(p, on_grid, featured);
if let Some(m) = window {
tiles.push(pip_tile(p, m));
}
Ok(Layout { order, tiles, voices })
}
/// §5.2: the featured participant (the first in order whose featured attribute is
/// `true`) moved first, then the list cut to `cap`, so that the cut never drops it; and
/// whether a featured participant is among those kept.
fn featured_first<'a, 'b>(mut members: Vec<&'b Member<'a>>, cap: usize) -> (Vec<&'b Member<'a>>, bool) {
let f = members.iter().position(|m| m.r.featured);
if let Some(i) = f {
let m = members.remove(i);
members.insert(0, m);
}
members.truncate(cap);
let kept = f.is_some() && !members.is_empty();
(members, kept)
}
fn video_of(p: &Params, m: &Member, tile: Rect, place: u8) -> Option<Video> {
let mode = match (p.fill, place) {
(fill::DEFAULT, place::FEATURED) => fill::CONTAIN,
(fill::DEFAULT, _) => fill::COVER,
(f, _) => f,
};
m.e.video.map(|(sw, sh)| place_video(tile, sw, sh, mode))
}
/// §5.3, §5.4: the grid's tiles over the canvas; when `featured`, `members[0]` is the
/// featured participant.
fn grid_tiles(p: &Params, members: Vec<&Member>, featured: bool) -> Vec<Tile> {
let n = members.len();
let (w, h, g) = (p.width as u64, p.height as u64, p.gap as u64);
let canvas = Rect::new(0, 0, p.width, p.height);
let mut out = Vec::with_capacity(n);
let mut put = |m: &Member, tile: Rect, place: u8| out.push(Tile { principal: m.e.principal, place, z: 0, tile, video: video_of(p, m, tile, place) });
if n == 0 {
return Vec::new();
}
if featured {
if n == 1 {
put(members[0], canvas, place::FEATURED);
} else {
let mh = main_height(h, g);
put(members[0], Rect::new(0, 0, p.width, mh as u32), place::FEATURED);
let band = Rect::new(0, (mh + g) as u32, w as u32, (h - mh - g) as u32);
for (m, t) in members[1..].iter().zip(grid(band, n as u64 - 1, g, p.aspect)) {
put(m, t, place::GRID);
}
}
} else {
for (m, t) in members.iter().zip(grid(canvas, n as u64, g, p.aspect)) {
put(m, t, place::GRID);
}
}
out
}
/// §5.5: the PiP window.
fn pip_tile(p: &Params, m: &Member) -> Tile {
let (w, h, o) = (p.width as u64, p.height as u64, p.offset as u64);
let (pw, ph) = if p.pip_width > 0 && p.pip_height > 0 {
(p.pip_width as u64, p.pip_height as u64)
} else {
let (a, b) = m.e.video.map_or((w, h), |(sw, sh)| (max(1, sw) as u64, max(1, sh) as u64));
let bw = if p.pip_width > 0 {
p.pip_width as u64
} else if p.pip_height > 0 {
w - o
} else {
min(w / 4, w - o)
};
let bh = if p.pip_height > 0 { p.pip_height as u64 } else { h - o };
fit(bw, bh, a, b)
};
let (x, y) = match p.corner {
corner::TOP_LEFT => (o, o),
corner::TOP_RIGHT => (w - o - pw, o),
corner::BOTTOM_LEFT => (o, h - o - ph),
_ => (w - o - pw, h - o - ph),
};
let tile = Rect::new(x as u32, y as u32, pw as u32, ph as u32);
Tile { principal: m.e.principal, place: place::PIP, z: 1, tile, video: video_of(p, m, tile, place::PIP) }
}
// ---------------------------------------------------------------------------------------
// §6 The audio mix
// ---------------------------------------------------------------------------------------
/// One voice's decoded audio for one block (§6): its channels (1 or 2), its gain and pan,
/// and its samples, interleaved when stereo.
#[derive(Clone, Copy, Debug)]
pub struct Input<'a> {
pub channels: u8,
pub gain: u32,
pub pan: i32,
pub pcm: &'a [i16],
}
/// §6: `frames` frames of `channels` (1 or 2) channels mixed from `inputs`, each exactly
/// `frames` frames of its own channels. `mix_input_malformed` otherwise.
pub fn mix(channels: u8, frames: usize, inputs: &[Input]) -> Result<Vec<i16>, Refusal> {
let bad = Err(Refusal::MixInputMalformed);
if !(1..=2).contains(&channels) {
return bad;
}
for v in inputs {
if !(1..=2).contains(&v.channels)
|| v.gain > MAX_GAIN
|| v.pan.unsigned_abs() > MAX_PAN as u32
|| Some(v.pcm.len()) != frames.checked_mul(v.channels as usize)
{
return bad;
}
}
let out_ch = channels as usize;
let mut acc = vec![0i64; frames * out_ch];
for v in inputs {
let g = v.gain as i64;
let s = |i: usize| v.pcm[i] as i64;
if out_ch == 1 {
for (f, a) in acc.iter_mut().enumerate() {
let x = if v.channels == 1 { s(f) * 4096 } else { (s(2 * f) + s(2 * f + 1)) * 2048 };
*a += x * g;
}
} else {
let left = (MAX_PAN - max(v.pan, 0)) as i64;
let right = (MAX_PAN + min(v.pan, 0)) as i64;
for f in 0..frames {
let (l, r) = if v.channels == 1 { (s(f), s(f)) } else { (s(2 * f), s(2 * f + 1)) };
acc[2 * f] += l * g * left;
acc[2 * f + 1] += r * g * right;
}
}
}
// Round half up by one arithmetic shift of the exact sum, then saturate once.
Ok(acc.iter().map(|&a| ((a + (1 << 23)) >> 24).clamp(i16::MIN as i64, i16::MAX as i64) as i16).collect())
}
#[cfg(test)]
mod tests {
use super::*;
fn params() -> Params {
Params {
width: 1920,
height: 1080,
fps: 30,
layout: kind::GRID,
gap: 0,
aspect: aspect::VIDEO,
fill: fill::DEFAULT,
featured_attr: "featured".into(),
order_attr: "order".into(),
omit_stopped: false,
pip_attr: String::new(),
corner: 0,
pip_width: 0,
pip_height: 0,
offset: 0,
behaviour: 0,
channels: 2,
sample_rate: 48_000,
gain_attr: "gain".into(),
pan_attr: "pan".into(),
max_tiles: 12,
max_voices: 12,
}
}
fn who(i: u8) -> [u8; 32] {
[i; 32]
}
fn rects(p: &Params, n: u8) -> Vec<Rect> {
featured_rects(p, n, None)
}
/// The tiles of `n` participants, the one at `featured` (if any) marked featured.
fn featured_rects(p: &Params, n: u8, featured: Option<u8>) -> Vec<Rect> {
let none: Vec<(String, String)> = Vec::new();
let star = vec![("featured".to_string(), "true".to_string())];
let es: Vec<Entrant> = (0..n)
.map(|i| Entrant { principal: who(i), join: i as u32, video: Some((1280, 720)), audio: true, attrs: if Some(i) == featured { &star } else { &none } })
.collect();
layout(p, &es).expect("a layout").tiles.iter().map(|t| t.tile).collect()
}
/// The grid choice of §5.3 on a 16:9 canvas with no gap: one tile fills it, two sit
/// side by side (the tie goes to more columns), three and four make two by two, and
/// ten to twelve four by three.
#[test]
fn grid_choice() {
let p = params();
assert_eq!(rects(&p, 1), vec![Rect::new(0, 0, 1920, 1080)]);
assert_eq!(rects(&p, 2), vec![Rect::new(0, 270, 960, 540), Rect::new(960, 270, 960, 540)]);
let three = rects(&p, 3);
assert_eq!(three[2], Rect::new(480, 540, 960, 540), "the last row is centred");
assert_eq!(rects(&p, 12)[11], Rect::new(1440, 765, 480, 270), "four by three cells of 480 by 360");
assert!(rects(&p, 0).is_empty());
}
/// Every tile of every grid of 1 to 64 tiles on each canvas, with and without a featured
/// participant, lies within the canvas, is at least a pixel each way, and no two tiles
/// overlap.
#[test]
fn tiles_stay_on_the_canvas() {
for (w, h, g) in [(1920, 1080, 8), (1080, 1920, 4), (1440, 1440, 0), (640, 360, 2)] {
let mut p = Params { width: w, height: h, gap: g, max_tiles: 64, max_voices: 64, ..params() };
for a in 0..=3 {
p.aspect = a;
for (n, f) in (0..=64u8).flat_map(|n| [(n, None), (n, n.checked_sub(1))]) {
let rs = featured_rects(&p, n, f);
assert_eq!(rs.len(), n as usize);
for (i, r) in rs.iter().enumerate() {
assert!(r.w >= 1 && r.h >= 1 && r.x + r.w <= w && r.y + r.h <= h, "{w}x{h} n={n} {r:?}");
for s in &rs[..i] {
assert!(r.x >= s.x + s.w || s.x >= r.x + r.w || r.y >= s.y + s.h || s.y >= r.y + r.h, "overlap {r:?} {s:?}");
}
}
}
}
}
}
/// Cover crops the source to the tile's aspect; contain letterboxes inside the tile;
/// fill stretches.
#[test]
fn fill_modes() {
let t = Rect::new(10, 20, 400, 400);
assert_eq!(place_video(t, 1920, 1080, fill::COVER), Video { dst: t, src: Rect::new(420, 0, 1080, 1080) });
assert_eq!(place_video(t, 1920, 1080, fill::CONTAIN), Video { dst: Rect::new(10, 107, 400, 225), src: Rect::new(0, 0, 1920, 1080) });
assert_eq!(place_video(t, 1920, 1080, fill::FILL), Video { dst: t, src: Rect::new(0, 0, 1920, 1080) });
assert_eq!(place_video(t, 1080, 1920, fill::CONTAIN).dst, Rect::new(97, 20, 225, 400));
}
#[test]
fn attributes() {
for (v, n) in [("0", 0), ("-7", -7), ("2147483647", 2_147_483_647), ("-2147483648", -2_147_483_648)] {
assert_eq!(read_int(v), Some(n), "{v}");
}
for v in ["", "-", "-0", "01", "+1", " 1", "1 ", "12345678901", "1e3", "٣"] {
assert_eq!(read_int(v), None, "{v:?}");
}
let p = params();
let a = |k: &str, v: &str| vec![(k.to_string(), v.to_string())];
assert_eq!(read_attrs(&p, &a("order", "2147483648")), Err(Refusal::AttrMalformed));
assert_eq!(read_attrs(&p, &a("featured", "TRUE")), Err(Refusal::AttrMalformed));
assert_eq!(read_attrs(&p, &a("gain", "16385")), Err(Refusal::AttrMalformed));
assert_eq!(read_attrs(&p, &a("pan", "-4096")).map(|r| r.pan), Ok(-4096));
assert_eq!(read_attrs(&p, &[]), Ok(Read { featured: false, order: None, pip: false, gain: UNITY_GAIN, pan: 0 }));
}
/// A single mono voice at unity gain and centre passes through both channels; a sum
/// that overflows saturates; the shift rounds half up.
#[test]
fn mixing() {
let x = [0i16, 1, -1, 32767, -32768, 1234];
let one = Input { channels: 1, gain: UNITY_GAIN, pan: 0, pcm: &x };
assert_eq!(mix(1, 6, &[one]).unwrap(), x.to_vec());
let st = mix(2, 6, &[one]).unwrap();
assert!(st.chunks(2).zip(x).all(|(lr, s)| lr == [s, s]));
let loud = [30000i16; 2];
let two = Input { channels: 1, gain: UNITY_GAIN, pan: 0, pcm: &loud };
assert_eq!(mix(1, 2, &[two, two]).unwrap(), vec![32767, 32767]);
let half = Input { channels: 1, gain: 2048, pan: 0, pcm: &[1, -1, 3, -3] };
assert_eq!(mix(1, 4, &[half]).unwrap(), vec![1, 0, 2, -1], "x/2 rounds half up");
let left = Input { channels: 1, gain: UNITY_GAIN, pan: -MAX_PAN, pcm: &[1000] };
assert_eq!(mix(2, 1, &[left]).unwrap(), vec![1000, 0]);
assert_eq!(mix(1, 3, &[one]), Err(Refusal::MixInputMalformed));
}
}

2
wai/wai-rs/tests/int_kernels_vectors.rs
View file

@ -223,7 +223,7 @@ fn every_decoded_output_states_a_sha256() {
("pin-conformance", "parameter-set pins: every digest there is a SHA-256 of a parameter set"),
("ops-conformance", "integer operators: every operator, stage and graph vector, the lower_* decodes of the sets under sets/ among them, states its output integers in full or its refusal code, never a digest of them; certificates/ states each lowered set's SHA-256 (a parameter-set identity) and holds the lowered graphs as bytes, compared byte for byte; the output digests a capability decoded through its lowering reproduces (integer-operators §14.4) are stated by that capability's own corpus"),
("signer-conformance", "key logs, grants and statuses: signed documents, no decoded output"),
("live-conformance", "live events, rooms and takes: signed records, commitments and their verdicts, no decoded output"),
("live-conformance", "live events, rooms, takes and compositions: signed records, commitments and their verdicts, and layouts and mixes stated in full, rectangle for rectangle and sample for sample; nothing decoded against a parameter set"),
("http-conformance", "objects and head proofs for transport: no case decodes a payload, and the component a fetch case picks is stated by its SHA-256"),
("cache-conformance", "cache rules: a driver, its controls and their reports, with no objects of its own"),
("byte-exact-conformance", "goldens over tests/vectors, whose sidecars state the SHA-256"),

81
wai/wai-rs/tests/live_compose.rs Normal file
View file

@ -0,0 +1,81 @@
//! `wai.live.compose` (extensions/live-compose.md): the committed corpus
//! `live-conformance/compose/vectors.json` regenerates byte for byte from its fixed seeds,
//! every case in it gets the outcome it states, every reason the extension adds refuses a
//! case of it, the cases §11 names are there, and the extension's §10 lists exactly the
//! reasons of the reference.
#![cfg(feature = "live_compose")]
use std::collections::BTreeSet;
use std::path::PathBuf;
use wai::live_compose::{corpus, Reason};
fn wai_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("..")
}
fn committed() -> String {
std::fs::read_to_string(wai_dir().join("live-conformance/compose/vectors.json")).expect("the committed corpus")
}
#[test]
fn the_committed_corpus_regenerates_from_its_seeds() {
assert!(committed() == corpus::generate(), "live-conformance/compose/vectors.json differs from `wai_live_compose_vectors gen`");
}
#[test]
fn every_case_gets_its_stated_outcome() {
let fails = corpus::verify(&committed());
assert!(fails.is_empty(), "{}", fails.join("\n"));
}
/// Every reason this extension adds refuses a case; grids of 0 to 12 tiles, every fill
/// mode, every corner under both behaviours, and both sessions are there; and the grid
/// session lays its room out anew at the guest's grant exchange.
#[test]
fn the_corpus_covers_every_reason_and_case() {
let doc: serde_json::Value = serde_json::from_str(&committed()).unwrap();
let mut seen: BTreeSet<String> = BTreeSet::new();
let mut names: BTreeSet<String> = BTreeSet::new();
for section in ["layouts", "mixes", "descriptors", "compositions", "receipts"] {
for v in doc[section].as_array().unwrap() {
names.insert(v["name"].as_str().unwrap().to_string());
if let Some(r) = v["reason"].as_str() {
seen.insert(r.into());
}
}
}
for r in Reason::ALL {
assert!(seen.contains(r.code()), "nothing in the corpus is refused {r}");
}
let mut want: Vec<String> = (0..=12).map(|n| format!("grid_{n}")).collect();
want.extend(["fill_fill", "fill_cover", "fill_contain", "featured_4", "featured_absent", "order_key_tie_break", "stopped_omitted"].map(String::from));
for c in ["top_left", "top_right", "bottom_left", "bottom_right"] {
for b in ["static", "dynamic"] {
want.push(format!("pip_{c}_{b}"));
}
}
want.extend(["pip_alone_static", "pip_alone_dynamic", "sum_saturates_high", "grid_session", "pip_session", "video", "audio"].map(String::from));
for n in &want {
assert!(names.contains(n), "no case {n}");
}
let session = doc["compositions"].as_array().unwrap().iter().find(|v| v["name"] == "grid_session").unwrap();
let frames = session["frames"].as_array().unwrap();
let at = |t: u64| frames.iter().find(|f| f["t"] == t).unwrap();
assert_ne!(at(4_000_000)["order"], at(5_000_000)["order"], "the guest's exchange at 5 s reorders the room");
assert_ne!(at(4_000_000)["set_digest"], at(5_000_000)["set_digest"]);
}
/// §10 of the extension lists exactly the codes `Reason::ALL` has, each once, in order.
#[test]
fn the_extension_lists_the_reasons() {
let text = std::fs::read_to_string(wai_dir().join("extensions/live-compose.md")).expect("the extension");
let at = text.find("\n## 10. Reasons").expect("a §10");
let end = text[at + 1..].find("\n## ").map_or(text.len(), |e| at + 1 + e);
let listed: Vec<String> = text[at..end]
.lines()
.filter(|l| l.starts_with("| `"))
.filter_map(|l| l.split('`').nth(1).map(String::from))
.collect();
let want: Vec<String> = Reason::ALL.iter().map(|r| r.code().to_string()).collect();
assert_eq!(listed, want, "§10's table and Reason::ALL differ");
}