open-standards/wai/extensions/live-compose.md

# WAI Extension: Live Composition

> Status: Draft. Registers the record `wai.live.compose`: the deterministic composition
> of a live room's participants. A signed **composition descriptor** fixes a canvas, a
> background, a layout (a grid, with a featured tile, or a grid under a
> picture-in-picture window, or an ad frame: a program squeezed into a corner with an ad
> in the L around it, or a program and an ad in two boxes side by side), the grant
> attributes it reads and its limits. The
> **participant set** at a room-clock instant is derived from the room's lifecycle
> replay ([live-rooms](live-rooms.md) §9): who publishes, each one's video and audio
> source and its dimensions, the attributes of its grant in effect, and its join order.
> The layout is then a pure integer function of the descriptor and the set, and the mix
> a pure integer function of decoded PCM, so every sink composes the same rectangles
> locally, with no server encode, and a server compositor that publishes a composed track
> signs a **composition receipt** that anyone can check against the same function. No new
> transport: a composed track is a track of the room, and the receipt names the objects it
> was made from. The reference implementation is the `live_compose` module of `wai-rs`
> (feature `live_compose`); the corpus is
> [`live-conformance/compose/`](../live-conformance/compose/), and its second
> implementation is `live-conformance/live_compose_verify.py`.
> In an ad frame, a session's sink may show the session's own ad: a **per-session ad track**,
> decided for that session by a [stitched-presentation](stitched-presentation.md) pod-input
> record and bound to the frame by an **ad-track binding**, `WAT1`, which the record's signer
> signs (§5.10). A binding that does not match its session's composition is refused. Its
> reference is the `compose_ad_track` module of `wai-rs` (feature `compose_ad_track`); its corpus
> is [`live-conformance/compose-ad-track/`](../live-conformance/compose-ad-track/), and its second
> implementation `live-conformance/compose_ad_track_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision: `draft-ietf-moq-transport-22` (MOQT), an Internet-Draft
> not past Working Group Last Call, for the Full Track Name (§2.4.1) and the Location
> (§8.2); RFC 6716 §6; RFC 8032; FIPS 180-4; and IEC 61966-2-1 (sRGB). A section number of
> that draft in this text is that revision's.

## 1. Scope

In scope: the composition descriptor, its digest and signature, its limits and who may
sign it (§3); the participant set at an instant, the attributes the descriptor reads, the
order, and the set's digest (§4); the layout function (§5), and a session's own ad in an ad
frame with its binding (§5.10); the audio mix (§6); where
composition runs (§7); the composition receipt and its verification (§8, §9).

Out of scope: a transport; how a renderer scales and filters a source into its rectangle,
and how it converts the background's colour into its output's (the layout fixes
rectangles, not pixels); decoding, resampling and encoding; and the browser compositor
that renders a layout in a page, which a later slice builds from this function.

Relationship to the core spec: `wai.live.compose` is a **record** ([SPEC](../SPEC.md) §5):
a signed record a WAI tool emits and verifies. It names no payload format and is not
dispatchable.

## 2. Terms

- **Room**, **room clock**, **principal**, **event**, **chain**, **grant**, **grant in
  effect**, **room key**, **merged order**, **slot**, **publish profile**: as in
  live-events §2 and live-rooms §2, §8, §9.2.
- **Room authority**: the key that signs the room's descriptor (live-rooms §2).
- **Compose power**: bit 2 of a capability set's `powers` (live-rooms §3.3).
- **Canvas**: the picture a composition makes, `width` × `height` pixels.
- **Source**: the track a participant is composed from (§4.2).
- **Compositor**: the principal that publishes a composed track (§7).
- **Rectangle**: (`x`, `y`, `w`, `h`): its left edge and top edge, in pixels from the top
  left of the canvas (or of a source), and its width and height.
- A **short** field is a u8 length and that many bytes. Integers are little-endian. A
  digest is SHA-256 (FIPS 180-4), 32 bytes.

## 3. The composition descriptor

### 3.1 Layout

| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WCD1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id (live-rooms §3.2) |
| `signer` | 32 | the Ed25519 key that signs the descriptor (§3.5) |
| `effective` | 8 | u64: the room-clock instant from which the descriptor applies |
| `compositor` | 32 | the principal that may publish the composed tracks (§7); 32 zero bytes for none |
| `width`, `height` | 2 each | u16: the canvas, in pixels (§3.4) |
| `fps` | 1 | the canvas's frame rate, in frames per second (§3.4) |
| `background` | 3 | the canvas's colour: red, green and blue, 8 bits each, in sRGB (IEC 61966-2-1) |
| `layer` | 1 | the highest layer of a source composed (§4.2) |
| `layout` | 1 | 1 grid, 2 picture in picture, 3 squeeze, 4 double box (§5; 3 and 4 are the ad frames of §5.9) |
| `gap` | 2 | u16: the pixels between two grid tiles |
| `aspect` | 1 | the grid tiles' aspect: 0 auto, 1 video (16:9), 2 square (1:1), 3 portrait (3:4) (§5.3) |
| `fill` | 1 | 0 default, 1 fill, 2 cover, 3 contain (§5.6) |
| `featured_attr` | short | the attribute key that marks the featured participant; empty for none (§4.3) |
| `order_attr` | short | the attribute key that orders participants; empty for none |
| `omit_stopped` | 1 | 1: a participant whose video is stopped has no tile; 0: it has one (§5.2) |
| `pip_attr` | short | the attribute key that marks the PiP participant, or in an ad frame the ad participant (§5.9); empty for none |
| `corner` | 1 | the PiP window's corner, or a squeeze's program window's: 1 top left, 2 top right, 3 bottom left, 4 bottom right; a double box's program side: 1 left, 2 right |
| `pip_width`, `pip_height` | 2 each | u16: the PiP window's size, 0 derives it (§5.5); a squeeze's program window's size (§5.9) |
| `offset` | 2 | u16: the window's distance, in pixels, from both edges of its corner |
| `behaviour` | 1 | what the PiP participant does when it is the only one composed: 1 static, 2 dynamic (§5.5) |
| `channels` | 1 | the mix's channels: 1 mono, 2 stereo (§6) |
| `sample_rate` | 4 | u32: the rate, in Hz, every voice is decoded to before it is mixed |
| `gain_attr` | short | the attribute key of a participant's gain; empty for none (§6) |
| `pan_attr` | short | the attribute key of a participant's pan; empty for none |
| `max_tiles` | 1 | the most tiles (§5.2) |
| `max_voices` | 1 | the most voices of the mix (§6) |
| `sig` | 64 | §3.2 |

A grid's `pip_attr` is empty and its `corner`, `pip_width`, `pip_height`, `offset` and
`behaviour` are 0 (§3.3).

### 3.2 Digest and signature

The **digest** of a descriptor is SHA-256 over its record from `magic` through
`max_voices`: every byte but `sig`. The **signature** is Ed25519 (RFC 8032) by `signer`
over the 17 bytes `wai:live-compose` (ASCII) and 0x01, followed by the bytes the digest
covers. It is verified strictly, as [SPEC](../SPEC.md) §7 *Signatures* requires: a key
that fails step 1 or 2 is `bad_key`, and a signature that fails a later step is
`bad_signature`.

### 3.3 Reading and field rules

A reader reads exactly the bytes it is given as one descriptor, in the order of §3.1, and
refuses at the first of these that holds:

1. Fewer than 4 bytes: `truncated`. A magic other than `WCD1`: `malformed`. No byte after
   the magic: `truncated`. A version other than 1: `unknown_version`.
2. The bytes end before a field the layout requires: `truncated`.
3. Bytes after `sig`: `malformed`.

It then holds the canvas to §3.4, and refuses one beyond its limits as
`canvas_out_of_range`. It then refuses, as `compose_invalid`, a descriptor that breaks any
of these rules:

- `layout` is 1 or 2, `aspect` at most 3, `fill` at most 3, and `omit_stopped` 0 or 1.
- Each attribute key is empty or a slot name of live-rooms §2 (1 to 32 bytes of `a`–`z`,
  `0`–`9` and `_`), which is what a grant attribute's key is (live-rooms §4.4).
- A grid has an empty `pip_attr`, and `corner`, `pip_width`, `pip_height`, `offset` and
  `behaviour` 0. A picture in picture has a `corner` of 1 to 4 and a `behaviour` of 1 or 2,
  an `offset` below both `width` and `height`, `pip_width` + `offset` at most `width`,
  and `pip_height` + `offset` at most `height`.
- An ad frame (§5.9) has a `pip_attr` that is not empty, `max_tiles` 2, and `offset` and
  `behaviour` 0. A squeeze has a `corner` of 1 to 4, `pip_width` from 1 to `width` − 1 and
  `pip_height` from 1 to `height` − 1; a double box has a `corner` of 1 or 2, `pip_width`
  and `pip_height` 0, and `gap` + 2 at most `width`.
- `channels` is 1 or 2, and `sample_rate` 8 000 to 192 000.
- `max_tiles` and `max_voices` are each 1 to 64.
- With *k* the least integer whose square is at least `max_tiles`: `gap` is below
  `height`; `width` is at least *k* + (*k* − 1) × `gap`; and the band under a featured
  tile (§5.4), `height` − `gap` − ⌊(`height` − `gap`) × 3 / 4⌋, is at least *k* +
  (*k* − 1) × `gap`. Every grid the layout makes then has a candidate whose cells are at
  least a pixel each way (§5.3).

Then the signature (§3.2): `bad_key` or `bad_signature`.

### 3.4 The canvas

`width` and `height` are each even, and each 16 to 1920; `width` × `height` is at most
2 073 600, the pixels of 1920 × 1080; and `fps` is 1 to 60. The edges are even so that a
picture subsampled 4:2:0 has whole chroma samples. A portrait 1080 × 1920 and a square
1440 × 1440 canvas are within these limits.

### 3.5 Who may sign

Against a room, a descriptor that reads (§3.3) is refused as `wrong_room` when its `room`
is not the room id; and as `compose_not_permitted` unless its `signer` is the room
authority, or a principal present at `effective` whose grant in effect then (§4.1) has the
compose power.

The power is checked at `effective`, when the descriptor takes effect: a descriptor does
not lapse when its signer leaves or later loses the power. A composition has one
descriptor; a room that changes its layout signs another, and each composed track and
each receipt names the one it follows (§8).

## 4. The participant set

### 4.1 The room at an instant

A composition reads a room that verifies (live-rooms §9): one refused is refused here
with the room's reason, at `room`. A room whose outcome is a gap is composed only when no
gap is in a room key's chain: the lifecycle is then whole, and a gap in another
principal's chain (a chat not delivered) changes no room state. A gap in a room key's
chain is `lifecycle_incomplete`, at `room`: the state at no instant after it is known.

The **room at** *t* is the room state (live-rooms §9.3) replayed from the lifecycle events
whose `t` is at most *t*, in the merged order: equally, the room each chain gives when it
is cut before its first record whose `t` is above *t*. Events at *t* itself are applied,
so a principal that joins at *t* is in the set at *t*.

The **join order** of a principal present at *t* is the number of the `join` that made it
present (its last), where the room's accepted `join` events up to *t* are numbered from 0
in the merged order, a duplicate (live-events §6.2 step 1) counted once.

### 4.2 Participants and their sources

The **participants** at *t* are the principals present at *t*, other than the
descriptor's `compositor`, that publish at *t*: each has an own track of any media kind
published, or is the live publisher of a slot (live-rooms §7.1). A standby is not
composed for its place in a slot, since its media is not delivered.

Each participant has:

- its **video source**: the first of its own `video` track; its own `screen` track; and
  its live place in a slot of media kind `video` or `screen`, the first such slot in the
  room descriptor's order. With none, its video is **stopped**.
- its **audio source**: its own `audio` track; or else its live place in the first slot of
  media kind `audio`. With none, it has no voice.
- the attributes of its grant in effect at *t* (live-rooms §4.4).

A source's **profile** is the profile its track was published under (the `publish` or
`takeover` that made it, live-rooms §7.2), and its **layers** the number of layers
published. Its **composed layer** *L* is the lower of `layers` − 1 and the descriptor's
`layer`, and its width and height are those of layer *L* of the profile (live-rooms §8.1);
an audio source's are 0. A descriptor whose `layer` is below a publisher's top layer
composes from a layer the compositor's grant can subscribe to, and every sink lays the
room out in that layer's pixels.

### 4.3 Attributes

For each participant, each attribute key the descriptor names (not empty) is looked up
among the attributes of its grant in effect, and read:

| key | value | when absent |
|---|---|---|
| `featured_attr`, `pip_attr` | `true` or `false` | `false` |
| `order_attr` | a decimal integer from −2 147 483 648 to 2 147 483 647 | no order |
| `gain_attr` | a decimal integer from 0 to 16 384 | 4096 |
| `pan_attr` | a decimal integer from −4096 to 4096 | 0 |

A **decimal integer** is `0`, or an optional `-` followed by 1 to 10 of the digits `0`–`9`
of which the first is not `0`: no `+`, no space, no `-0`. Every attribute the descriptor
names is read for every participant, whether or not it has the source the attribute is
about; one whose value is not of its row is `attr_malformed`, and the composition at *t*
is refused, at `set`. The keys `featured`, `order` and `pip` that live-rooms §4.4 names
are these attributes under the keys a descriptor gives them; `gain` and `pan` are the
mix's.

### 4.4 Order

The participants are ordered: those with an order first, by order ascending, then those
with none; between two of one order (or none), by join order ascending; and between two
of one join order, by key, as unsigned bytes. This order is **S**.

### 4.5 The set's digest

The digest of the participant set at *t* is SHA-256 over the 21 bytes
`wai:live-compose-set` (ASCII) and 0x01, the room id, *t* (u64), the number of
participants (u32), and then, for each participant in ascending order of key:

| field | size | meaning |
|---|---|---|
| `principal` | 32 | its key |
| `grant` | 32 | the digest of its grant in effect |
| `join` | 4 | u32: its join order |
| `video`, `audio` | 1 or 9 + *s* each | its source: one byte 0 for none; otherwise `media` (u8, the media kind of live-rooms §6.1), `slot` (short: empty for an own track), `profile` (u8, the profile's index in the room descriptor), `layers` (u8), the composed layer *L* (u8), `width` and `height` (u16 each) |

The grant's digest fixes its attributes, so the set's digest fixes everything §5 and §6
read of the room at *t*.

## 5. The layout

### 5.1 Integers

Every quantity of this section is an integer, and none is negative. ⌊*a* / *b*⌋, for *b*
above 0, rounds down. **fit**(*bw*, *bh*, *a*, *b*), the largest rectangle of aspect
*a*:*b* within *bw* × *bh*, is (*bw*, max(1, ⌊*bw* × *b* / *a*⌋)) when *bw* × *b* ≤ *bh*
× *a*, and (max(1, ⌊*bh* × *a* / *b*⌋), *bh*) otherwise. A rectangle of *w* × *h*
**centred** in (*x*, *y*, *W*, *H*) is (*x* + ⌊(*W* − *w*) / 2⌋, *y* + ⌊(*H* − *h*) / 2⌋,
*w*, *h*). No product here exceeds 2^63.

### 5.2 Which participants have tiles

An ad frame's tiles are §5.9's. Otherwise, from S (§4.4):

1. **V** is S, less every participant whose video is stopped when `omit_stopped` is 1.
2. For a picture in picture, the **PiP participant** P is the first of V whose `pip_attr`
   attribute is `true`; with none there is no PiP window. For a grid there is none.
3. **G** is V without P, in order. The **featured participant** F is the first of G whose
   `featured_attr` attribute is `true`; it is moved to the front of G.
4. G is cut to its first `max_tiles` members, less one when there is a P. F, being first,
   is kept; when G is then empty there is no F.

### 5.3 The grid

A grid of *n* ≥ 1 tiles over a region (*X*, *Y*, *W*, *H*) is chosen by its **reference
aspect** *a*:*b*: 16:9 for auto and video, 1:1 for square, 3:4 for portrait.

- **Candidates.** For each number of columns *c* from 1 to *n*, with *r* = ⌈*n* / *c*⌉
  rows: *c* is a candidate when (*c* − 1) × `gap` is below *W*, (*r* − 1) × `gap` below
  *H*, and the cells, *cw* = ⌊(*W* − (*c* − 1) × `gap`) / *c*⌋ wide and *ch* = ⌊(*H* −
  (*r* − 1) × `gap`) / *r*⌋ high, are each at least 1. Its **area** is *tw* × *th*, where
  (*tw*, *th*) = fit(*cw*, *ch*, *a*, *b*).
- **The choice.** The candidate of the largest area; of several, the one with the most
  columns. §3.3's rule on `gap` makes *c* = ⌈√*n*⌉ a candidate for every *n* a
  descriptor allows.
- **Placement.** The rows are centred: the first row's top is *y*0 = *Y* + ⌊(*H* − (*r* ×
  *ch* + (*r* − 1) × `gap`)) / 2⌋. Tile *k*, from 0, is in row *i* = ⌊*k* / *c*⌋ and column
  *j* = *k* mod *c*. Every row holds *c* tiles but the last, which holds *n* − (*r* − 1) ×
  *c*; a row of *m* tiles is centred, beginning at *x*0 = *X* + ⌊(*W* − (*m* × *cw* + (*m* −
  1) × `gap`)) / 2⌋. Cell *k* is (*x*0 + *j* × (*cw* + `gap`), *y*0 + *i* × (*ch* + `gap`),
  *cw*, *ch*).
- **The tile.** Under auto, tile *k* is its cell; otherwise it is fit(*cw*, *ch*, *a*, *b*)
  centred in its cell.

With no tile the canvas is its background.

### 5.4 The featured tile

With F and *n* ≥ 2 members of G, the **featured tile** is (0, 0, `width`, *mh*), with *mh*
= ⌊(`height` − `gap`) × 3 / 4⌋, and the other *n* − 1 members are a grid (§5.3) over the
band below it, (0, *mh* + `gap`, `width`, `height` − *mh* − `gap`). The featured tile is its
whole region: `aspect` is the grid tiles'. With F alone the featured tile is the whole
canvas. With no F, the members of G are a grid over the whole canvas.

### 5.5 Picture in picture

With P and a G that is not empty, G is laid out over the canvas as §5.3 and §5.4 lay it out,
and P's **window** is drawn over it. With P and an empty G:

- **static**: P's window, and the background;
- **dynamic**: P is laid out as though it had no `pip_attr` attribute: a grid of P alone,
  or, when P is featured, the featured tile, the whole canvas.

The window is `pip_width` × `pip_height` when both are above 0. Otherwise, with *a*:*b* the
width and height of P's video source, or the canvas's when P's video is stopped, it is
fit(*bw*, *bh*, *a*, *b*), where *bw* is `pip_width` when that is above 0, `width` −
`offset` when only `pip_height` is, and the lower of ⌊`width` / 4⌋ and `width` − `offset`
when neither is; and *bh* is `pip_height` when that is above 0 and `height` − `offset`
otherwise. With *o* the `offset` and *pw* × *ph* the window, it is at (*o*, *o*) in the top
left corner, (`width` − *o* − *pw*, *o*) in the top right, (*o*, `height` − *o* − *ph*) in
the bottom left, and (`width` − *o* − *pw*, `height` − *o* − *ph*) in the bottom right.

### 5.6 Fill

A tile (*tx*, *ty*, *tw*, *th*) shows a source of *sw* × *sh* by its fill mode. The source
is **wider** when *sw* × *th* ≥ *sh* × *tw*.

- **Fill**: the whole source, (0, 0, *sw*, *sh*), stretched over the whole tile.
- **Cover**: the source cropped to the tile's aspect, over the whole tile. A wider source's
  crop is *cw* = max(1, ⌊*sh* × *tw* / *th*⌋) wide, at (⌊(*sw* − *cw*) / 2⌋, 0, *cw*, *sh*);
  any other's is *ch* = max(1, ⌊*sw* × *th* / *tw*⌋) high, at (0, ⌊(*sh* − *ch*) / 2⌋, *sw*,
  *ch*).
- **Contain**: the whole source, letterboxed in the tile. A wider source is drawn into
  (*tx*, *ty* + ⌊(*th* − *dh*) / 2⌋, *tw*, *dh*) with *dh* = max(1, ⌊*tw* × *sh* / *sw*⌋);
  any other into (*tx* + ⌊(*tw* − *dw*) / 2⌋, *ty*, *dw*, *th*) with *dw* = max(1, ⌊*th* ×
  *sw* / *sh*⌋).
- **Default**: cover for a grid tile, the PiP window and an ad tile, contain for the
  featured tile and a program tile (§5.9).

Each mode gives a **source rectangle** and a **destination rectangle**: the part of the
source drawn, and where on the canvas it is drawn. A tile of stopped video has neither, and
shows the background, as does the part of a tile outside its destination. How the source
rectangle is scaled into the destination is the renderer's; a sink that holds a source at a
layer below its composed one scales it to the composed layer's width and height first.

### 5.7 What the layout gives

The layout of a descriptor and a participant set gives:

- **the order**: S, every participant, by key;
- **the tiles**, in drawing order: the featured tile, then the grid's in G's order, then
  the PiP window; or an ad frame's, in §5.9's order. Each has its participant (or, for a
  session's ad track, the track, §5.10.4), its
  **place** (`grid`, `featured`, `pip`, `program` or `ad`), its **z** (0, or 1 for the PiP
  window and a squeeze's program window, drawn over the others), the tile, and, unless its
  video is stopped, its source and destination rectangles;
- **the voices** (§6).

A renderer paints the background over the canvas, then the tiles in order. No two tiles
of z 0 overlap.

### 5.8 Re-layout

The participant set changes only at the `t` of a lifecycle event: a `join`, `leave`,
`publish`, `unpublish`, `grant_exchange`, `takeover` or `failover` (an exchange always
changes it, since the set names each participant's grant). At every such instant at or
after `effective` the layout is computed again, and it changes when the sources, the
attributes or the order do; between two such instants it does not change. The layout at *t*
is the function of this section over the descriptor and the set at *t*, and of nothing a
sink sees otherwise, so a sink that joins late lays the room out as one that was there
throughout, and a grant exchange that changes an attribute re-lays the room out at the
exchange's tick, with no participant leaving. On the sink of a session that holds an ad-track
binding, an ad frame's layout also changes where the binding comes into effect and where it
ends (§5.10.4).

### 5.9 Ad frames

An ad frame composes a program and an ad into one picture: a **squeeze** (layout 3) shrinks
the program into a window in one corner and shows the ad in the L around it; a **double box**
(layout 4) shows the program and the ad in two boxes side by side. The frame is an integer
function of the canvas and the fields below, so the part of the picture the ad shows in is
fixed by what the descriptor's signer signed, and a record of another extension that states
the same fields fixes the same part (stitched-presentation §6). The ad is the room's, one
participant every sink shows, or each session's own, on a per-session ad track (§5.10).

**The frame** of a canvas of `width` × `height`, with *W* its width and *H* its height:

- **Squeeze**: with *pw* = `pip_width` and *ph* = `pip_height`, the **program window** is *pw*
  × *ph* in the corner `corner` names: at (0, 0) top left, (*W* − *pw*, 0) top right, (0, *H* −
  *ph*) bottom left and (*W* − *pw*, *H* − *ph*) bottom right. The **ad tile** is the whole
  canvas. The **ad region** is the canvas less the window, two rectangles: the **side band**,
  beside the window and as high as it, (*pw*, *y*, *W* − *pw*, *ph*) for a window on the
  left and (0, *y*, *W* − *pw*, *ph*) for one on the right, with *y* the window's top; and
  the **full band**, the rest of the rows, (0, *ph*, *W*, *H* − *ph*) for a window at the top
  and (0, 0, *W*, *H* − *ph*) for one at the bottom.
- **Double box**: with *bw* = ⌊(*W* − `gap`) / 2⌋, the left box is (0, 0, *bw*, *H*) and the
  right box (*bw* + `gap`, 0, *W* − *bw* − `gap`, *H*). The program's box is the left for a
  `corner` of 1 and the right for 2, and the ad's the other. Each tile is its box under
  `aspect` auto; otherwise it is fit(*box width*, *box height*, *a*, *b*) centred in its box,
  with *a*:*b* the reference aspect of §5.3. The **program tile** and the **ad tile** are
  those, and the ad region is the ad tile alone.

The ad region's rectangles are in order of their top edge, and none of them overlaps the
program's tile. Its **share** of the picture is ⌊10 000 × *A* / (*W* × *H*)⌋ basis points,
with *A* the sum of its rectangles' areas.

**The participants.** From V (§5.2 step 1), the **ad participant** is the first whose
`pip_attr` attribute is `true`, and the **program participant** is the first of the others
once the featured participant (§5.2 step 3) is moved to their front. Then:

- with both, the tiles are, for a squeeze, the ad tile (place `ad`, z 0) and then the
  program window (place `program`, z 1, drawn over it); for a double box, the program tile
  and then the ad tile (both z 0);
- with the program participant alone, one tile, the canvas, of place `program`; with the ad
  participant alone, one tile, the canvas, of place `ad`; with neither, none.

No other participant has a tile; the voices are §6's, as for every layout.

### 5.10 Per-session ad tracks

An ad frame's ad region may show each session an ad decided for it alone: an ad of a
pod-input record of the session ([stitched-presentation](stitched-presentation.md) §4.6). The
program is the room's, laid out as §5.9 lays it, and every sink draws the same rectangles; the
ad is the session's **ad track**, which the record's signer publishes for that session and binds
to the frame with an **ad-track binding**.

#### 5.10.1 The session's ad track

A **session** is the 16 bytes stitched-presentation §5.2 derives from a player's session key.
The ad track of session *s* is named as live-rooms §6.1 names a track, with the owner `ad_`
followed by *s* as 32 lowercase hexadecimal digits: its Track Namespace is the room id, that
owner and `video`, and its Track Name is `0`. The owner is neither a key (64 digits) nor a slot
(`_` first), so it names no participant, and each of its bytes is one of `a`–`z`, `0`–`9` and
`_`. Its **track** is the digest live-events §3.1 gives an anchor's `track`: SHA-256 over the
namespace's rendering (live-rooms §6.2), one zero byte, and the name.

#### 5.10.2 The binding

| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WAT1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id |
| `descriptor` | 32 | the digest of the descriptor whose ad frame shows the track (§3.2) |
| `record` | 32 | the digest of the session's pod-input record (stitched-presentation §4.4) |
| `session` | 16 | the session |
| `slot` | 1 | the ad's slot in the record (stitched-presentation §4.2) |
| `start` | 8 | u64: the room-clock instant from which the ad region shows the track |
| `width`, `height` | 2 each | u16: the track's picture, in pixels, each 1 to 1920 |
| `signer` | 32 | the record's signer, which signs the binding |
| `sig` | 64 | Ed25519 by `signer` |

The **digest** of a binding is SHA-256 over its bytes before `sig`; its **signature** is Ed25519
by `signer` over the 26 bytes `wai:live-compose-ad-track` (ASCII) and 0x01, followed by the bytes
the digest covers, verified strictly (SPEC §7 *Signatures*). A reader reads exactly the bytes it
is given, as §3.3 steps 1 to 3 read a descriptor with `WAT1` for `WCD1` (`truncated`,
`malformed`, `unknown_version`); refuses as `ad_track_invalid` a binding whose `width` or
`height` is 0 or above 1920; and then verifies the signature (`bad_key`, `bad_signature`).

A binding holds for its ad's `duration_ms` (stitched-presentation §4.2): it is **in effect** at a
room-clock instant *t* when `start` ≤ *t* < `start` + 1000 × `duration_ms`.

#### 5.10.3 Verifying a binding

A binding verifier is given a descriptor, a pod-input record and a binding. In this order, the
first that fails names the refusal and where:

1. **The descriptor** reads (§3.3): its refusal, at `descriptor`. Who may sign it (§3.5) is
   checked against the room, where the room is composed.
2. **The record** reads and its signature verifies (stitched-presentation §4.4, §7.2 steps 1 and
   2): `pod_input_malformed`, `bad_key` or `bad_signature`, at `record`.
3. **The binding** reads (§5.10.2), at `binding`; from here on every refusal is at `binding`.
4. Its `signer` is the record's `signer`: otherwise `ad_track_not_permitted`.
5. Its `room` is the descriptor's: otherwise `wrong_room`.
6. Its `descriptor` is the descriptor's digest: otherwise `descriptor_mismatch`.
7. It matches its session's composition: otherwise `ad_track_mismatch`, naming the first of
   these that differs:
   - `layout`: the descriptor's `layout` is an ad frame, 3 or 4;
   - `record`: its `record` is the record's digest;
   - `session`: the record is a record of a session, not a shared one, and its `session` is the
     binding's;
   - `frame`: the record's frame (stitched-presentation §4.3) is the descriptor's: its `kind` the
     `layout`, its `width` and `height` the canvas's, its `corner` the `corner`, its `window_w`
     and `window_h` the `pip_width` and `pip_height` for a squeeze and 0 for a double box, its
     `gap` the `gap` for a double box and 0 for a squeeze, and its `aspect` the `aspect` for a
     double box and 0 for a squeeze;
   - `slot`: the record has an ad of its `slot`.
8. Its `start` is at or after the descriptor's `effective`: otherwise `not_yet_effective`.

A binding that passes is **accepted**: the session's ad track shows the ad of `slot` of the
session's own record in the ad region the descriptor fixes, from `start` for that ad's
duration. A verifier states its digest, the session, the slot, the track, the span from `start`
to `start` + 1000 × `duration_ms`, and the ad region and its share (§5.9), which are the record's
frame's (stitched-presentation §6).

#### 5.10.4 The layout of a session

On the sink of a session that holds an accepted binding in effect at *t*, the layout of an ad
frame at *t* is §5.9's, with the session's ad track for the ad:

- the program participant is §5.9's, the first of V other than the room's ad participant, a
  featured one first; the room's ad participant, if there is one, has no tile and no voice;
- with the program participant, the tiles are §5.9's of both, the ad tile showing the session's
  ad track at the binding's `width` and `height`; without one, one tile, the canvas, of place
  `ad`, showing the track;
- the voices are §6's, less the room's ad participant's. The session's ad track is a picture:
  this revision mixes no sound of it.

A tile of the track names the track (§5.10.1) where every other tile names a participant. With no
binding in effect, the layout is §5.9's. A session's layout is composed on its sink (§7): a
composed track that a server compositor publishes (§8) is the room's, never a session's.

#### 5.10.5 Reasons

| reason | raised at |
|---|---|
| `ad_track_invalid` | §5.10.2 |
| `ad_track_not_permitted` | §5.10.3 step 4 |
| `ad_track_mismatch` | §5.10.3 step 7 |

A binding verifier also raises `truncated`, `malformed`, `unknown_version`, `bad_key` and
`bad_signature` for a binding that does not read, `wrong_room`, `descriptor_mismatch` and
`not_yet_effective` (§10) at `binding`, a descriptor's refusals at `descriptor` and a record's at
`record`.

## 6. The audio mix

The **voices** are the members of S (before any is omitted for its video) that have an
audio source, in S's order, the first `max_voices` of them. Each has its **gain** *g*,
from 0 to 16 384 in units of 1/4096 (4096 is unity), and its **pan** *p*, from −4096
(left) to 4096 (right) (§4.3).

The mixer mixes one block of *N* frames. Each voice's input is its decoded audio for the
block: *N* frames of signed 16-bit samples, of 1 or 2 channels (interleaved, left first),
at the descriptor's `sample_rate`. Each output sample is a sum over the voices in signed
64-bit integers, one term per voice:

- **mono output** (`channels` 1): *x* × 4096 × *g* for a mono voice's sample *x*; (*x*L +
  *x*R) × 2048 × *g* for a stereo voice's samples *x*L and *x*R.
- **stereo output** (`channels` 2), with *a*L = 4096 − max(*p*, 0) and *a*R = 4096 +
  min(*p*, 0): to the left channel *x* × *g* × *a*L and to the right *x* × *g* × *a*R for
  a mono voice; *x*L × *g* × *a*L and *x*R × *g* × *a*R for a stereo one.

The output sample is ⌊(*s* + 2^23) / 2^24⌋ of the sum *s*, the arithmetic shift right by
24 bits of *s* + 2^23, which rounds half up; then saturated once to −32 768 … 32 767. A
term is at most 2^41 in magnitude and a mix has at most 64 voices, so the sum is exact,
the order of its terms changes nothing, and saturation applies to the whole once. The pan
is a balance: at the centre both channels are at the voice's gain, and panning attenuates
one channel only. A single voice at unity gain and the centre is passed through
unchanged. A mixer that holds no decoded block of a voice for the output mixes it as
silence.

A mixer is refused as `mix_input_malformed` when an output has other than 1 or 2
channels, or an input has other than 1 or 2 channels, has other than *N* frames, or a
gain or pan out of range.

The mix is byte-exact over the decoded PCM: two mixers given the same blocks give the
same samples. It is not byte-exact over what precedes and follows it. A decoder of a lossy
codec need not reproduce another decoder's PCM bit for bit (RFC 6716 §6 holds a decoder to
a quality measure, not to equality), and a lossy encode of the mix is not
byte-reproducible: two encoders, or one encoder under two settings, give different bytes.
The receipt (§8) therefore binds the input objects and the output object, not samples.

## 7. Where composition runs

**On each sink, by default.** A sink composes the room locally from the tracks it
subscribes to, by §5 and §6, with no server decode or encode. It needs no power: the
descriptor states how the room is laid out, and laying it out is reading it. A session's own
ad in an ad frame is composed on its sink alone, from the session's ad track (§5.10.4).

**On a server, under the compose power.** A principal that publishes a composed track (its
own `video` track for a layout, its own `audio` track for a mix, live-rooms §6.1) **MUST**
be the descriptor's `compositor`, **MUST** hold the compose power in its grant in effect,
and **MUST** sign a receipt (§8) for each object of a composed track it publishes. It is
not a participant of its own composition (§4.2), so a composed track is never composed
again. A subscriber subscribes to a composed track as to any other; the descriptor and the
receipts let it check that the track is the room's composition.

## 8. The composition receipt

### 8.1 Layout

| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WCR1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id |
| `compositor` | 32 | the compositor's key, which signs the receipt |
| `descriptor` | 32 | the digest of the descriptor it follows (§3.2) |
| `t` | 8 | u64: the room-clock instant of the output object |
| `state` | 32 | the digest of the participant set at `t` (§4.5) |
| `media` | 1 | 1 audio (a mix), 2 video (a layout) |
| `output` | 80 | the output object |
| `inputs` | 1 + 112 × *n* | a u8 count *n*, then each input: `principal` (32), the participant it is read from, and an object (80) |
| `sig` | 64 | Ed25519 by `compositor` |

An **object** is 80 bytes: `track` (32), the digest live-events §3.1 gives an anchor's
`track`, over the track's namespace as live-rooms §6.2 renders it, one zero byte, and its
name; `group` and `object` (u64 each), its MOQT Location (MOQT §8.2); and `digest` (32),
the SHA-256 of the object's payload. An input whose `digest` is 32 zero bytes states that
the compositor held no object of that track for the output: it drew the tile with no
video, or mixed the voice as silence.

The **digest** of a receipt is SHA-256 over its bytes before `sig`; its **signature** is
Ed25519 by `compositor` over the 25 bytes `wai:live-compose-receipt` (ASCII) and 0x01,
followed by the bytes the digest covers, verified strictly (SPEC §7 *Signatures*). A
reader reads exactly the bytes it is given, as §3.3 steps 1 to 3 read a descriptor with
`WCR1` for `WCD1` (`truncated`, `malformed`, `unknown_version`); refuses as
`receipt_invalid` a receipt whose `media` is not 1 or 2, or that names two inputs of one
participant and one track; and then verifies the signature (`bad_key`, `bad_signature`).

### 8.2 The inputs

The inputs of an output at *t* are, in this order:

- **video** (`media` 2): for each tile of the layout at *t* that shows video, in drawing
  order (§5.7), the tracks of its participant's video source;
- **audio** (`media` 1): for each voice, in order (§6), the tracks of its audio source.

The **tracks** of a source are named as live-rooms §6.1 names them: the namespace is the
room, the owner (the participant's key, or `_` and the slot's name for a slot) and the
media kind; the names follow the profile's `layering` (live-rooms §6.3) at the composed
layer *L*: track `0` for `single` and `svc` (an `svc` track carries every layer), track
*L* for `simulcast`, and tracks `0` to *L* for `staged`, since a refinement is applied to
the layers below it ([staged-delivery](staged-delivery.md)). The receipt names, for each,
the object it read.

### 8.3 Verifying a receipt

A receipt verifier is given a room (as live-rooms §9.1), a descriptor and a receipt. In
this order, the first that fails names the refusal and where:

1. **The room** (§4.1): its refusal, at `room`.
2. **The descriptor** (§3.3, §3.5): at `descriptor`.
3. **The receipt** reads (§8.1), at `receipt`; and its `room` is the room id, otherwise
   `wrong_room`.
4. Its `descriptor` is the descriptor's digest: otherwise `descriptor_mismatch`.
5. Its `t` is at or after the descriptor's `effective`: otherwise `not_yet_effective`.
6. The descriptor names a compositor, the receipt's `compositor` is it, and that principal
   is present at `t` with a grant in effect that has the compose power: otherwise
   `compositor_not_permitted`.
7. Its `output`'s `track` is a track of the compositor's own track of the receipt's media
   kind, published at `t`, at a layer from 0 to its `layers` − 1: otherwise
   `output_not_published`.
8. **The set and the layout** at `t` (§4, §5): their refusal, at `set`.
9. Its `state` is the set's digest: otherwise `state_mismatch`.
10. Its inputs, each its `principal` and its object's `track`, are §8.2's, in order:
    otherwise `inputs_mismatch`.

A receipt that passes is **accepted**: its output object was composed under this
descriptor, from this set, out of objects of exactly the tracks the layout reads. Anyone
holding the room's events and the descriptor recomputes the layout and checks that the
inputs were the right ones; a party holding the objects checks their digests. The receipt
does not fix the output's pixels or samples (§6).

## 9. Verification

A **composition verifier** is given a room, a descriptor and a list of instants. It runs
these steps and stops at the first refusal, which it reports with the reason and where:

1. **The room** (§4.1): at `room`, with the room's own place (live-rooms §9.1) when the
   room is refused.
2. **The descriptor** (§3.3, §3.5): at `descriptor`.
3. **Each instant**, in the list's order: the instant is at or after `effective`
   (otherwise `not_yet_effective`), and the participants' attributes read (§4.3,
   otherwise `attr_malformed`): at `set`, with the instant.

Its outcome is **refuse**, with the reason and the place; or the **composition**: the
descriptor's digest and, for each instant, the participants with their sources and
tracks, the set's digest, the order, the tiles and the voices. A receipt verifier (§8.3)
reports its refusals at the same places, and at `receipt`.

## 10. Reasons

The codes this extension adds, each with where it is raised:

| reason | raised at |
|---|---|
| `canvas_out_of_range` | §3.3, §3.4 |
| `compose_invalid` | §3.3 |
| `compose_not_permitted` | §3.5 |
| `not_yet_effective` | §8.3 step 5, §9 step 3 |
| `lifecycle_incomplete` | §4.1 |
| `attr_malformed` | §4.3 |
| `mix_input_malformed` | §6 |
| `receipt_invalid` | §8.1 |
| `descriptor_mismatch` | §8.3 step 4 |
| `compositor_not_permitted` | §7, §8.3 step 6 |
| `output_not_published` | §8.3 step 7 |
| `state_mismatch` | §8.3 step 9 |
| `inputs_mismatch` | §8.3 step 10 |

The ad-track binding's codes are §5.10.5's.

A verifier of this extension also raises `truncated`, `malformed` and `unknown_version`
for a descriptor or a receipt that does not read; `bad_key` and `bad_signature` for their
signatures; `wrong_room` for one of another room (§3.5, §8.3); and every code a room
verifier raises (live-rooms §11 and the live-events codes it lists), at `room`.

## 11. Conformance

[`live-conformance/compose/`](../live-conformance/compose/) holds the corpus, in one file,
`vectors.json`, written from fixed seeds by `wai_live_compose_vectors gen`. It has:

- **layouts**: the layout function alone (§5, §6's voices), each case its parameters and
  its participants with their join orders, video sizes, audio and attributes, and the
  order, tiles and voices it gives, or its refusal. Grids of 0 to 12 tiles on a 16:9
  canvas, and on portrait, square and auto ones; every fill mode over sources of four
  shapes and of extreme aspects; a featured participant present, absent, kept past the
  tile limit, and first in order of two; ties in order broken by join order and by key;
  stopped video shown and omitted; picture in picture in every corner under both
  behaviours; squeezes in every corner and double boxes on both sides, with the program
  alone, the ad alone, and a featured program, alone static and alone dynamic, with derived sizes and stopped video; and
  attributes that do not read.
- **mixes**: blocks of PCM and the mix of them (§6): passing through, both widths into
  both, saturation high and low and only at the end, pans and gains at their limits,
  rounding at ties, silence, and inputs refused.
- **descriptors**: records read (§3), with their digest and fields, or refused with each
  reason a descriptor alone can have.
- **rooms**: a studio room under the live-rooms descriptor, grants and chains that its
  vectors name, and four variants of it: a grant whose order does not read, a lifecycle
  record not delivered, a viewer's chat not delivered, and a viewer's record refused.
- **compositions**: a room, a descriptor and instants (§9), with the composition or the
  refusal. The grid over the whole session re-lays the room out at each lifecycle
  instant, among them a grant exchange that reorders it; the picture in picture composes
  a lower layer.
- **receipts**: a room, a descriptor and a receipt (§8.3), accepted with its digest and
  the set's, or refused with each reason a receipt can have.

A conforming verifier gives every case the outcome, the reason, the place and the data the
corpus states. `wai_live_compose_vectors verify` and the test `tests/live_compose.rs`
check that the file regenerates byte for byte, that `wai-rs` gives every case its
outcome, that every reason of §10 refuses a case, and that §10 lists exactly the
reference's codes. `live-conformance/live_compose_verify.py` is the second implementation,
written from this text: it shares no code with `wai-rs`, verifies signatures with WAI's
strict rule written from RFC 8032 (`claim-conformance/_ed25519.py`), verifies each room
with the live-rooms corpus's own second implementation (`live_rooms_verify.py`) and
replays its lifecycle to each instant itself, and gives every case the same outcome,
rectangle for rectangle and sample for sample; it also mutates copies of the corpus and
requires every copy to fail.

[`live-conformance/compose-ad-track/`](../live-conformance/compose-ad-track/) holds the corpus of
§5.10, in one file, `vectors.json`, written from fixed seeds by `wai_compose_ad_track_vectors
gen`. It has:

- **bindings**: a descriptor, a pod-input record and a binding (§5.10.3), accepted with its
  digest, session, slot, track, span, ad region and share, under a squeeze and a double box; or
  refused with each reason a binding can have: bindings that do not read, out of their bounds, and
  signed over the wrong bytes (without the domain, under the pod-input record's domain, by a key
  that is not its signer, flipped); a signer that is not the record's; another room; another
  descriptor; and, for `ad_track_mismatch`, a grid's descriptor, another record, a shared record,
  another session's record, a record whose frame is another layout's, another window's, another
  corner's, another canvas's, another gap's and another aspect's, and a slot the record does not
  have; a start before the descriptor is effective; and descriptors and records that do not read;
- **layouts**: the layout of a session (§5.10.4) at instants before, at the start of, during, at
  the last microsecond of and at the end of its binding: under a squeeze and a double box, with a
  program and the room's own ad participant, with a program alone, with the room's ad
  participant alone, with no participant, under each fill, and with voices.

`wai_compose_ad_track_vectors verify` and the test `tests/compose_ad_track.rs` check that the
file regenerates byte for byte, that `wai-rs` gives every case its outcome, and that §5.10.5 lists
exactly the codes the reference adds. `live-conformance/compose_ad_track_verify.py` is the second
implementation, written from this text: it shares no code with `wai-rs`; it reads descriptors and
lays rooms out with the live-compose corpus's second implementation, reads records with the
stitched-presentation corpus's, and verifies Ed25519 with WAI's strict rule; and it mutates copies
of the corpus and drops each of its own checks in turn, and requires each to be caught.

## 12. Security and privacy considerations

- The layout is computed from what the room authority signed: the descriptor, or a
  principal's under the power the authority granted, and the attributes of grants the
  authority signed. A participant cannot place itself; a relay that drops or reorders
  lifecycle events is seen as a gap in a room key's chain (§4.1), not as another layout.
- A server compositor's receipt fixes which objects it composed and under which set. A
  compositor that left a participant out, read a layer other than the composed one, or
  composed a stale set is refused by §8.3. The receipt does not fix the output's pixels
  or samples, which a lossy encode does not reproduce (§6); a party that needs them
  composes locally from the inputs.
- Composition on each sink needs no server to decode the room's media, so a room whose
  tracks are end-to-end encrypted ([confidentiality](confidentiality.md)) can still be
  laid out: the layout reads the room's events and the grants, never media.
- The participant set names every publisher, its grant and its join order. Composing a
  room reveals nothing a member of it does not already hold, but a set digest published
  outside the room links the participants it names; a room that needs participation
  unlinkable across rooms gives principals a key per room (live-events §11).
- Attributes are read as the descriptor's keys name them and nothing else is read of a
  grant's attributes, so an attribute a descriptor does not name changes nothing.
- A per-session ad track is named by its session (§5.10.1), which stitched-presentation §5.2
  derives from a key made for one playback session, so it links nothing across sessions. A
  binding proves that the signer of the session's record stated that the session's ad track
  shows that record's ad in this descriptor's ad region; one whose record's frame is not the
  descriptor's, or whose record is shared or another session's, is refused (§5.10.3), so the ad
  region a session's claim measures is the one its sink draws.