wai: live-compose, a room's composition as an integer function (L3) #296

Merged

dcharlot merged 10 commits from wai/live-compose into main 2026-10-17 16:04:02 -04:00

Conversation 0 Commits 10 Files changed 21 +24099 -17

dcharlot commented 2026-10-17 09:49:40 -04:00

Owner

Copy link

wai.live.compose: a room's composition as an integer function (L3)

This PR adds slice L3 of the live and interactive work, the extension
wai/extensions/live-compose.md. A room's layout is a
pure integer function of two things: a signed descriptor, and the participant set that the
room's lifecycle replay gives at an instant. That lets every sink compose locally, with no
server encode. A server compositor's output can then be checked against the same function
through a signed receipt. The base is main, which already holds L1, L2 and L4 (#289).

What is in it

• Spec, wai/extensions/live-compose.md:
  • §3 the WCD1 descriptor. It is signed by the room authority, or by a principal whose
    grant in effect at effective has the compose power. It carries:
    • the canvas: each edge even, 16–1920 px; at most 2 073 600 px; 1–60 fps;
    • an sRGB background, and the highest layer composed;
    • a layout: a grid, or a grid under a PiP window;
    • the attribute keys it reads, and its limits on tiles and voices.
  • §4 the participant set at t:
    • The room state comes from the lifecycle events with t ≤ t. A gap in a room key's
      chain is lifecycle_incomplete; a gap in any other chain changes nothing.
    • The participants are every present publisher except the compositor.
    • Each has a video source (own video, own screen, or a live place in a slot), with the
      dimensions of its composed layer, and an audio source.
    • Each has the attributes of its grant in effect, and its join order.
    • Ordering is by the order attribute, then join order, then key bytes. The set has a
      digest.
  • §5 the layout. Every rounding is stated.
    • The grid is the column count whose tile of the reference aspect is largest; ties go
      to more columns. Rows are centred, and the last row is centred too.
    • Aspects: auto (laid out as 16:9, tiles fill their cells), 16:9, 1:1 and 3:4.
    • A featured tile sits above a band that holds the others.
    • Fill modes are FILL, COVER and CONTAIN. The default is cover for grid tiles and the
      PiP window, and contain for the featured tile.
    • Stopped video is shown or omitted.
    • The PiP window goes in any corner. Its size is set, or derived from the source's
      aspect. With an offset.
    • STATIC or DYNAMIC: a dynamic PiP participant that is alone is laid out as though it
      had no pip attribute.
    • N = 0 gives the background only. A featured participant moved first is never cut by
      the tile limit.
    • Re-layout happens at every lifecycle event and grant exchange.
  • §6 the mix:
    • Integer gain (0–16384 in units of 1/4096) and balance pan (±4096), from attributes.
    • Signed 16-bit PCM, summed exactly in i64, rounded half up by one shift, and saturated
      once at the end.
    • The mix is byte-exact over decoded PCM. The spec says plainly that a decode and a
      lossy re-encode are not, so the receipt binds the objects.
  • §7 where composition runs: client-side by default. Server-side requires being the
    descriptor's compositor with the compose power, and the compositor is not composed.
  • §8 the WCR1 receipt:
    • It names the descriptor digest, the set digest at t, the output object, and the
      input objects.
    • The inputs are the tracks the layout reads, in drawing or voice order: track 0 for
      single/svc, track L for simulcast, tracks 0..L for staged. A zero object digest means
      "held none".
    • §8.3 is the ten-step check.
  • §10 its reason codes; §11 conformance.
• Reference, wai-rs feature live_compose (it implies live_room):
  • src/live_compose.rs: the descriptor, the room at t, participants, the set digest,
    the receipt, and verification.
  • src/live_compose/layout.rs: the layout and the mixer, using nothing beyond core,
    Vec and String, so L5c can build it for wasm.
  • The module is cbindgen:ignore.
  • No new dependency.
• Corpus, wai/live-conformance/compose/, written from fixed seeds by
  wai_live_compose_vectors gen, with a provenance line in its README. It covers:
  • grids of 0–12 tiles on several canvases and aspects;
  • every fill mode, including extreme source aspects;
  • featured present, absent, past the tile limit, and two featured;
  • ordering ties broken by join order and by key;
  • omit-stopped on and off;
  • PiP in each corner, static and dynamic, alone static or dynamic, with derived sizes,
    with stopped video;
  • attributes that do not parse;
  • mixes with saturation (high, low, only at the end), pans, gains, ties, silence, and
    refusals;
  • descriptors at and past the canvas limits, and every descriptor refusal;
  • a studio room with four variants, composed every second across a grant exchange that
    re-lays it out;
  • good video, audio and PiP receipts, and a bad receipt for each reason.
• Independent verifier, wai/live-conformance/live_compose_verify.py, standard
  library only:
  • It reimplements the descriptor, the layout, the mixer, the set digest and the receipt
    from the spec text.
  • It verifies rooms with live_rooms_verify.py, and steps that verifier's lifecycle
    replay itself to take the room at each instant. The Rust reference instead re-verifies
    the prefix of each chain cut at t, so the two derivations differ.
  • Rectangles and PCM match the reference on every case, and every mutated copy of the
    corpus fails.
• CI and registry:
  • A live compose step in the independent-verifiers job, and live-compose.md in both
    path lists. The tests are declared with required-features.
  • SPEC §5 paragraph and record list, REGISTRY prose, and registry rows (the record, and
    the magics WCD1 and WCR1); wai registry export was re-run.
  • CHANGELOG entry.
  • The live-conformance exclusion in int_kernels_vectors now names compositions.
• live-rooms.md §1, §3.3 and §4.4 now point to live-compose for the compose power and
  for the meaning of featured, order, pip (and the added gain and pan). The
  capture-binding attributes subject and minor are unchanged.

Checks run (build host, rb, under memguard)

Every cargo command ran on the build host through ~/kani-logs/remote/rb <worktree> -- '<command>'.
Output went to logs on the host, which were then grepped. Nothing was built on the dev Mac.

run	command (in wai/wai-rs unless noted)	exit
1	cargo test --no-default-features --features live_compose --lib live_compose::	0
2 (--pull)	cargo run --no-default-features --features live_compose --bin wai_live_compose_vectors -- gen ../live-conformance/compose	0
3 (--pull)	cargo run -q --no-default-features --bin wai -- registry export	0
3	cargo run -q --no-default-features --bin wai -- registry export --check	0
3	touch cbindgen.toml && cargo build --no-default-features --features live_compose --lib, then git diff --exit-code -- wai/include/wai.h	0, 0: wai.h unchanged
3	cargo test … --features live_compose --lib live_compose:: and --test live_compose	0, 0
3	cargo run … --bin wai_live_compose_vectors -- verify ../live-conformance/compose	0
3	cargo test --no-default-features --features capture_take --test live_events --test live_room --test capture_take (live-rooms.md was edited)	0
3	cargo test --no-default-features --lib codecs::registry	0
3	cargo test --no-default-features --test cli_registry --test corpus_provenance	0
3	cargo clippy --no-default-features --features live_compose --lib --tests --bins	0, with two findings in live_compose (ok().expect, a manual is_multiple_of), fixed in bbbe4137
4	the same clippy	0, with no finding in live_compose
4	cargo test --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate,score,staged,stage_measure,legacy_int --test int_kernels_vectors (the cargo-class step's exact features)	0
5, on 12a7b7b7	the CI step exactly: the --lib live_compose:: and --test live_compose runs, then (in wai) python3 -I live-conformance/live_compose_verify.py live-conformance/compose; and --lib codecs::registry	0

The light checks were run locally:

• python3 -I on live_compose_verify.py, live_events_verify.py, live_rooms_verify.py and capture_take_verify.py: all exit 0, and every mutated copy of the compose corpus fails.
• The repo scripts check-tests-in-ci.py, check-spec-refs.py, check-wai-section-refs.py, check-no-vanity-metrics.sh, check-comparative-prose.sh, check-orphan-sources.py, check-workflow-concurrency.sh and pr-touches-paths-selftest.sh: all exit 0.

Two checks were not run:

• check-lockfiles.sh runs cargo. This PR adds no dependency, and no Cargo.lock changed.
• The company and product name check needs main's denylist secret. Instead, the added lines were scanned by hand and name no company, product or person.

Deferred to L5c (the browser compositor)

• Building layout.rs for wasm and exporting it to the page.
• Rendering:
  • scaling or filtering each source rectangle into its destination;
  • converting the sRGB background into the output's colour format;
  • a placeholder for stopped tiles.
• WebCodecs decode and per-block PCM gathering into the mixer, and resampling to
  sample_rate.
• The /room/ page, with an e2e test on a browser's fake media.
• A server compositor that encodes, publishes and signs receipts live. This PR specifies
  the receipt and its verification, and signs receipts only in the corpus.

Not done here: per-hop receipts of the relay (L9).

🤖 Generated with Claude Code

## `wai.live.compose`: a room's composition as an integer function (L3) This PR adds slice L3 of the live and interactive work, the extension [`wai/extensions/live-compose.md`](wai/extensions/live-compose.md). A room's layout is a pure integer function of two things: a signed descriptor, and the participant set that the room's lifecycle replay gives at an instant. That lets every sink compose locally, with no server encode. A server compositor's output can then be checked against the same function through a signed receipt. The base is `main`, which already holds L1, L2 and L4 (#289). ### What is in it - **Spec**, `wai/extensions/live-compose.md`: - §3 the `WCD1` descriptor. It is signed by the room authority, or by a principal whose grant in effect at `effective` has the compose power. It carries: - the canvas: each edge even, 16–1920 px; at most 2 073 600 px; 1–60 fps; - an sRGB background, and the highest layer composed; - a layout: a grid, or a grid under a PiP window; - the attribute keys it reads, and its limits on tiles and voices. - §4 the participant set at *t*: - The room state comes from the lifecycle events with `t` ≤ *t*. A gap in a room key's chain is `lifecycle_incomplete`; a gap in any other chain changes nothing. - The participants are every present publisher except the compositor. - Each has a video source (own video, own screen, or a live place in a slot), with the dimensions of its composed layer, and an audio source. - Each has the attributes of its grant in effect, and its join order. - Ordering is by the order attribute, then join order, then key bytes. The set has a digest. - §5 the layout. Every rounding is stated. - The grid is the column count whose tile of the reference aspect is largest; ties go to more columns. Rows are centred, and the last row is centred too. - Aspects: auto (laid out as 16:9, tiles fill their cells), 16:9, 1:1 and 3:4. - A featured tile sits above a band that holds the others. - Fill modes are FILL, COVER and CONTAIN. The default is cover for grid tiles and the PiP window, and contain for the featured tile. - Stopped video is shown or omitted. - The PiP window goes in any corner. Its size is set, or derived from the source's aspect. With an offset. - STATIC or DYNAMIC: a dynamic PiP participant that is alone is laid out as though it had no pip attribute. - N = 0 gives the background only. A featured participant moved first is never cut by the tile limit. - Re-layout happens at every lifecycle event and grant exchange. - §6 the mix: - Integer gain (0–16384 in units of 1/4096) and balance pan (±4096), from attributes. - Signed 16-bit PCM, summed exactly in i64, rounded half up by one shift, and saturated once at the end. - The mix is byte-exact over decoded PCM. The spec says plainly that a decode and a lossy re-encode are not, so the receipt binds the objects. - §7 where composition runs: client-side by default. Server-side requires being the descriptor's `compositor` with the compose power, and the compositor is not composed. - §8 the `WCR1` receipt: - It names the descriptor digest, the set digest at *t*, the output object, and the input objects. - The inputs are the tracks the layout reads, in drawing or voice order: track 0 for single/svc, track L for simulcast, tracks 0..L for staged. A zero object digest means "held none". - §8.3 is the ten-step check. - §10 its reason codes; §11 conformance. - **Reference**, `wai-rs` feature `live_compose` (it implies `live_room`): - `src/live_compose.rs`: the descriptor, the room at *t*, participants, the set digest, the receipt, and verification. - `src/live_compose/layout.rs`: the layout and the mixer, using nothing beyond `core`, `Vec` and `String`, so L5c can build it for wasm. - The module is `cbindgen:ignore`. - No new dependency. - **Corpus**, `wai/live-conformance/compose/`, written from fixed seeds by `wai_live_compose_vectors gen`, with a provenance line in its README. It covers: - grids of 0–12 tiles on several canvases and aspects; - every fill mode, including extreme source aspects; - featured present, absent, past the tile limit, and two featured; - ordering ties broken by join order and by key; - omit-stopped on and off; - PiP in each corner, static and dynamic, alone static or dynamic, with derived sizes, with stopped video; - attributes that do not parse; - mixes with saturation (high, low, only at the end), pans, gains, ties, silence, and refusals; - descriptors at and past the canvas limits, and every descriptor refusal; - a studio room with four variants, composed every second across a grant exchange that re-lays it out; - good video, audio and PiP receipts, and a bad receipt for each reason. - **Independent verifier**, `wai/live-conformance/live_compose_verify.py`, standard library only: - It reimplements the descriptor, the layout, the mixer, the set digest and the receipt from the spec text. - It verifies rooms with `live_rooms_verify.py`, and steps that verifier's lifecycle replay itself to take the room at each instant. The Rust reference instead re-verifies the prefix of each chain cut at *t*, so the two derivations differ. - Rectangles and PCM match the reference on every case, and every mutated copy of the corpus fails. - **CI and registry**: - A `live compose` step in the independent-verifiers job, and `live-compose.md` in both path lists. The tests are declared with `required-features`. - SPEC §5 paragraph and record list, REGISTRY prose, and registry rows (the record, and the magics `WCD1` and `WCR1`); `wai registry export` was re-run. - CHANGELOG entry. - The `live-conformance` exclusion in `int_kernels_vectors` now names compositions. - **live-rooms.md** §1, §3.3 and §4.4 now point to live-compose for the compose power and for the meaning of `featured`, `order`, `pip` (and the added `gain` and `pan`). The capture-binding attributes `subject` and `minor` are unchanged. ### Checks run (build host, `rb`, under memguard) Every cargo command ran on the build host through `~/kani-logs/remote/rb <worktree> -- '<command>'`. Output went to logs on the host, which were then grepped. Nothing was built on the dev Mac. | run | command (in `wai/wai-rs` unless noted) | exit | |---|---|---| | 1 | `cargo test --no-default-features --features live_compose --lib live_compose::` | 0 | | 2 (`--pull`) | `cargo run --no-default-features --features live_compose --bin wai_live_compose_vectors -- gen ../live-conformance/compose` | 0 | | 3 (`--pull`) | `cargo run -q --no-default-features --bin wai -- registry export` | 0 | | 3 | `cargo run -q --no-default-features --bin wai -- registry export --check` | 0 | | 3 | `touch cbindgen.toml && cargo build --no-default-features --features live_compose --lib`, then `git diff --exit-code -- wai/include/wai.h` | 0, 0: `wai.h` unchanged | | 3 | `cargo test … --features live_compose --lib live_compose::` and `--test live_compose` | 0, 0 | | 3 | `cargo run … --bin wai_live_compose_vectors -- verify ../live-conformance/compose` | 0 | | 3 | `cargo test --no-default-features --features capture_take --test live_events --test live_room --test capture_take` (live-rooms.md was edited) | 0 | | 3 | `cargo test --no-default-features --lib codecs::registry` | 0 | | 3 | `cargo test --no-default-features --test cli_registry --test corpus_provenance` | 0 | | 3 | `cargo clippy --no-default-features --features live_compose --lib --tests --bins` | 0, with two findings in `live_compose` (`ok().expect`, a manual `is_multiple_of`), fixed in `bbbe4137` | | 4 | the same clippy | 0, with no finding in `live_compose` | | 4 | `cargo test --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate,score,staged,stage_measure,legacy_int --test int_kernels_vectors` (the cargo-class step's exact features) | 0 | | 5, on `12a7b7b7` | the CI step exactly: the `--lib live_compose::` and `--test live_compose` runs, then (in `wai`) `python3 -I live-conformance/live_compose_verify.py live-conformance/compose`; and `--lib codecs::registry` | 0 | The light checks were run locally: - `python3 -I` on `live_compose_verify.py`, `live_events_verify.py`, `live_rooms_verify.py` and `capture_take_verify.py`: all exit 0, and every mutated copy of the compose corpus fails. - The repo scripts `check-tests-in-ci.py`, `check-spec-refs.py`, `check-wai-section-refs.py`, `check-no-vanity-metrics.sh`, `check-comparative-prose.sh`, `check-orphan-sources.py`, `check-workflow-concurrency.sh` and `pr-touches-paths-selftest.sh`: all exit 0. Two checks were not run: - `check-lockfiles.sh` runs cargo. This PR adds no dependency, and no `Cargo.lock` changed. - The company and product name check needs main's denylist secret. Instead, the added lines were scanned by hand and name no company, product or person. ### Deferred to L5c (the browser compositor) - Building `layout.rs` for wasm and exporting it to the page. - Rendering: - scaling or filtering each source rectangle into its destination; - converting the sRGB background into the output's colour format; - a placeholder for stopped tiles. - WebCodecs decode and per-block PCM gathering into the mixer, and resampling to `sample_rate`. - The `/room/` page, with an e2e test on a browser's fake media. - A server compositor that encodes, publishes and signs receipts live. This PR specifies the receipt and its verification, and signs receipts only in the corpus. Not done here: per-hop receipts of the relay (L9). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 10 commits 2026-10-17 09:49:40 -04:00

wai: live_compose, the layout function, the mixer, the descriptor, the set and the receipt (WIP) ... 61e58fe986

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge remote-tracking branch 'origin/main' into wai/live-compose 8d4b5761f5

wai: live_compose corpus generator, vectors binary, test, feature and module registration (WIP) ... 051c1bef43

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: live-compose.md and the generated live-conformance/compose corpus ... 15a7057b7f

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: live_compose_verify.py, the corpus's second implementation, with mutated copies ... 560965a9fd

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: registry, SPEC §5, REGISTRY, CHANGELOG, corpus READMEs; live-rooms points at live-compose ... 2e6d8d2f07

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: CI step for live_compose and its verifier; the extension in the path lists; the live-conformance exclusion names compositions ... 4a3d4de1bd

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: live-compose §6 cites RFC 6716 §6 without a count-shaped phrase ... aea575770a

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: registry export for wai.live.compose; two clippy findings in live_compose ... bbbe413738

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: live-compose §5.8 says every grant exchange changes the set; the header rewrapped ... 12a7b7b765

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-10-17 10:44:11 -04:00

wai: wai.live.recording — a room session as one package, and provable clips (L7a) #299

dcharlot referenced this pull request 2026-10-17 11:26:51 -04:00

wai: timed text and live captions, wai.text.timed (L8) #303

dcharlot merged commit 68d9842e42 into main 2026-10-17 16:04:02 -04:00

dcharlot referenced this pull request from a commit 2026-10-17 16:04:03 -04:00

Merge pull request 'wai: live-compose, a room's composition as an integer function (L3)' (#296) from wai/live-compose into main

dcharlot referenced this pull request from a commit 2026-10-17 16:10:26 -04:00

Merge origin/main into wai/d2c-m7a; CHANGELOG keeps both entries

dcharlot referenced this pull request from a commit 2026-10-17 16:16:46 -04:00

Merge origin/main into wai/live-record; live-compose and live-recording side by side

dcharlot referenced this pull request 2026-10-17 16:38:35 -04:00

wai: D2c M8 — WIV1 and WIR1 through wai-ops/1's stage operators, goldens unchanged #308

dcharlot referenced this pull request from a commit 2026-10-17 16:47:07 -04:00

Merge wai/live-record (#299, with main) into wai/timed-text; captions beside compose and recording

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!296

No description provided.