open-standards/wai/extensions/live-compose.md

WAI Extension: Live Composition

Status: Draft. Registers the record wai.live.compose: the deterministic composition of a live room's participants. A signed composition descriptor fixes a canvas, a background, a layout (a grid, with a featured tile, or a grid under a picture-in-picture window, or an ad frame: a program squeezed into a corner with an ad in the L around it, or a program and an ad in two boxes side by side), the grant attributes it reads and its limits. The participant set at a room-clock instant is derived from the room's lifecycle replay (live-rooms §9): who publishes, each one's video and audio source and its dimensions, the attributes of its grant in effect, and its join order. The layout is then a pure integer function of the descriptor and the set, and the mix a pure integer function of decoded PCM, so every sink composes the same rectangles locally, with no server encode, and a server compositor that publishes a composed track signs a composition receipt that anyone can check against the same function. No new transport: a composed track is a track of the room, and the receipt names the objects it was made from. The reference implementation is the live_compose module of wai-rs (feature live_compose); the corpus is live-conformance/compose/, and its second implementation is live-conformance/live_compose_verify.py. In an ad frame, a session's sink may show the session's own ad: a per-session ad track, decided for that session by a stitched-presentation pod-input record and bound to the frame by an ad-track binding, WAT1, which the record's signer signs (§5.10). A binding that does not match its session's composition is refused. Its reference is the compose_ad_track module of wai-rs (feature compose_ad_track); its corpus is live-conformance/compose-ad-track/, and its second implementation live-conformance/compose_ad_track_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: draft-ietf-moq-transport-22 (MOQT), an Internet-Draft not past Working Group Last Call, for the Full Track Name (§2.4.1) and the Location (§8.2); RFC 6716 §6; RFC 8032; FIPS 180-4; and IEC 61966-2-1 (sRGB). A section number of that draft in this text is that revision's.

1. Scope

In scope: the composition descriptor, its digest and signature, its limits and who may sign it (§3); the participant set at an instant, the attributes the descriptor reads, the order, and the set's digest (§4); the layout function (§5), and a session's own ad in an ad frame with its binding (§5.10); the audio mix (§6); where composition runs (§7); the composition receipt and its verification (§8, §9).

Out of scope: a transport; how a renderer scales and filters a source into its rectangle, and how it converts the background's colour into its output's (the layout fixes rectangles, not pixels); decoding, resampling and encoding; and the browser compositor that renders a layout in a page, which a later slice builds from this function.

Relationship to the core spec: wai.live.compose is a record (SPEC §5): a signed record a WAI tool emits and verifies. It names no payload format and is not dispatchable.

2. Terms

• Room, room clock, principal, event, chain, grant, grant in effect, room key, merged order, slot, publish profile: as in live-events §2 and live-rooms §2, §8, §9.2.
• Room authority: the key that signs the room's descriptor (live-rooms §2).
• Compose power: bit 2 of a capability set's powers (live-rooms §3.3).
• Canvas: the picture a composition makes, width × height pixels.
• Source: the track a participant is composed from (§4.2).
• Compositor: the principal that publishes a composed track (§7).
• Rectangle: (x, y, w, h): its left edge and top edge, in pixels from the top left of the canvas (or of a source), and its width and height.
• A short field is a u8 length and that many bytes. Integers are little-endian. A digest is SHA-256 (FIPS 180-4), 32 bytes.

3. The composition descriptor

3.1 Layout

field	size	meaning
magic	4	WCD1 (ASCII)
version	1	1 in this revision
room	32	the room id (live-rooms §3.2)
signer	32	the Ed25519 key that signs the descriptor (§3.5)
effective	8	u64: the room-clock instant from which the descriptor applies
compositor	32	the principal that may publish the composed tracks (§7); 32 zero bytes for none
width, height	2 each	u16: the canvas, in pixels (§3.4)
fps	1	the canvas's frame rate, in frames per second (§3.4)
background	3	the canvas's colour: red, green and blue, 8 bits each, in sRGB (IEC 61966-2-1)
layer	1	the highest layer of a source composed (§4.2)
layout	1	1 grid, 2 picture in picture, 3 squeeze, 4 double box (§5; 3 and 4 are the ad frames of §5.9)
gap	2	u16: the pixels between two grid tiles
aspect	1	the grid tiles' aspect: 0 auto, 1 video (16:9), 2 square (1:1), 3 portrait (3:4) (§5.3)
fill	1	0 default, 1 fill, 2 cover, 3 contain (§5.6)
featured_attr	short	the attribute key that marks the featured participant; empty for none (§4.3)
order_attr	short	the attribute key that orders participants; empty for none
omit_stopped	1	1: a participant whose video is stopped has no tile; 0: it has one (§5.2)
pip_attr	short	the attribute key that marks the PiP participant, or in an ad frame the ad participant (§5.9); empty for none
corner	1	the PiP window's corner, or a squeeze's program window's: 1 top left, 2 top right, 3 bottom left, 4 bottom right; a double box's program side: 1 left, 2 right
pip_width, pip_height	2 each	u16: the PiP window's size, 0 derives it (§5.5); a squeeze's program window's size (§5.9)
offset	2	u16: the window's distance, in pixels, from both edges of its corner
behaviour	1	what the PiP participant does when it is the only one composed: 1 static, 2 dynamic (§5.5)
channels	1	the mix's channels: 1 mono, 2 stereo (§6)
sample_rate	4	u32: the rate, in Hz, every voice is decoded to before it is mixed
gain_attr	short	the attribute key of a participant's gain; empty for none (§6)
pan_attr	short	the attribute key of a participant's pan; empty for none
max_tiles	1	the most tiles (§5.2)
max_voices	1	the most voices of the mix (§6)
sig	64	§3.2

A grid's pip_attr is empty and its corner, pip_width, pip_height, offset and behaviour are 0 (§3.3).

3.2 Digest and signature

The digest of a descriptor is SHA-256 over its record from magic through max_voices: every byte but sig. The signature is Ed25519 (RFC 8032) by signer over the 17 bytes wai:live-compose (ASCII) and 0x01, followed by the bytes the digest covers. It is verified strictly, as SPEC §7 Signatures requires: a key that fails step 1 or 2 is bad_key, and a signature that fails a later step is bad_signature.

3.3 Reading and field rules

A reader reads exactly the bytes it is given as one descriptor, in the order of §3.1, and refuses at the first of these that holds:

1. Fewer than 4 bytes: truncated. A magic other than WCD1: malformed. No byte after the magic: truncated. A version other than 1: unknown_version.
2. The bytes end before a field the layout requires: truncated.
3. Bytes after sig: malformed.

It then holds the canvas to §3.4, and refuses one beyond its limits as canvas_out_of_range. It then refuses, as compose_invalid, a descriptor that breaks any of these rules:

• layout is 1 or 2, aspect at most 3, fill at most 3, and omit_stopped 0 or 1.
• Each attribute key is empty or a slot name of live-rooms §2 (1 to 32 bytes of a–z, 0–9 and _), which is what a grant attribute's key is (live-rooms §4.4).
• A grid has an empty pip_attr, and corner, pip_width, pip_height, offset and behaviour 0. A picture in picture has a corner of 1 to 4 and a behaviour of 1 or 2, an offset below both width and height, pip_width + offset at most width, and pip_height + offset at most height.
• An ad frame (§5.9) has a pip_attr that is not empty, max_tiles 2, and offset and behaviour 0. A squeeze has a corner of 1 to 4, pip_width from 1 to width − 1 and pip_height from 1 to height − 1; a double box has a corner of 1 or 2, pip_width and pip_height 0, and gap + 2 at most width.
• channels is 1 or 2, and sample_rate 8 000 to 192 000.
• max_tiles and max_voices are each 1 to 64.
• With k the least integer whose square is at least max_tiles: gap is below height; width is at least k + (k − 1) × gap; and the band under a featured tile (§5.4), height − gap − ⌊(height − gap) × 3 / 4⌋, is at least k + (k − 1) × gap. Every grid the layout makes then has a candidate whose cells are at least a pixel each way (§5.3).

Then the signature (§3.2): bad_key or bad_signature.

3.4 The canvas

width and height are each even, and each 16 to 1920; width × height is at most 2 073 600, the pixels of 1920 × 1080; and fps is 1 to 60. The edges are even so that a picture subsampled 4:2:0 has whole chroma samples. A portrait 1080 × 1920 and a square 1440 × 1440 canvas are within these limits.

3.5 Who may sign

Against a room, a descriptor that reads (§3.3) is refused as wrong_room when its room is not the room id; and as compose_not_permitted unless its signer is the room authority, or a principal present at effective whose grant in effect then (§4.1) has the compose power.

The power is checked at effective, when the descriptor takes effect: a descriptor does not lapse when its signer leaves or later loses the power. A composition has one descriptor; a room that changes its layout signs another, and each composed track and each receipt names the one it follows (§8).

4. The participant set

4.1 The room at an instant

A composition reads a room that verifies (live-rooms §9): one refused is refused here with the room's reason, at room. A room whose outcome is a gap is composed only when no gap is in a room key's chain: the lifecycle is then whole, and a gap in another principal's chain (a chat not delivered) changes no room state. A gap in a room key's chain is lifecycle_incomplete, at room: the state at no instant after it is known.

The room at t is the room state (live-rooms §9.3) replayed from the lifecycle events whose t is at most t, in the merged order: equally, the room each chain gives when it is cut before its first record whose t is above t. Events at t itself are applied, so a principal that joins at t is in the set at t.

The join order of a principal present at t is the number of the join that made it present (its last), where the room's accepted join events up to t are numbered from 0 in the merged order, a duplicate (live-events §6.2 step 1) counted once.

4.2 Participants and their sources

The participants at t are the principals present at t, other than the descriptor's compositor, that publish at t: each has an own track of any media kind published, or is the live publisher of a slot (live-rooms §7.1). A standby is not composed for its place in a slot, since its media is not delivered.

Each participant has:

• its video source: the first of its own video track; its own screen track; and its live place in a slot of media kind video or screen, the first such slot in the room descriptor's order. With none, its video is stopped.
• its audio source: its own audio track; or else its live place in the first slot of media kind audio. With none, it has no voice.
• the attributes of its grant in effect at t (live-rooms §4.4).

A source's profile is the profile its track was published under (the publish or takeover that made it, live-rooms §7.2), and its layers the number of layers published. Its composed layer L is the lower of layers − 1 and the descriptor's layer, and its width and height are those of layer L of the profile (live-rooms §8.1); an audio source's are 0. A descriptor whose layer is below a publisher's top layer composes from a layer the compositor's grant can subscribe to, and every sink lays the room out in that layer's pixels.

4.3 Attributes

For each participant, each attribute key the descriptor names (not empty) is looked up among the attributes of its grant in effect, and read:

key	value	when absent
featured_attr, pip_attr	true or false	false
order_attr	a decimal integer from −2 147 483 648 to 2 147 483 647	no order
gain_attr	a decimal integer from 0 to 16 384	4096
pan_attr	a decimal integer from −4096 to 4096	0

A decimal integer is 0, or an optional - followed by 1 to 10 of the digits 0–9 of which the first is not 0: no +, no space, no -0. Every attribute the descriptor names is read for every participant, whether or not it has the source the attribute is about; one whose value is not of its row is attr_malformed, and the composition at t is refused, at set. The keys featured, order and pip that live-rooms §4.4 names are these attributes under the keys a descriptor gives them; gain and pan are the mix's.

4.4 Order

The participants are ordered: those with an order first, by order ascending, then those with none; between two of one order (or none), by join order ascending; and between two of one join order, by key, as unsigned bytes. This order is S.

4.5 The set's digest

The digest of the participant set at t is SHA-256 over the 21 bytes wai:live-compose-set (ASCII) and 0x01, the room id, t (u64), the number of participants (u32), and then, for each participant in ascending order of key:

field	size	meaning
principal	32	its key
grant	32	the digest of its grant in effect
join	4	u32: its join order
video, audio	1 or 9 + s each	its source: one byte 0 for none; otherwise media (u8, the media kind of live-rooms §6.1), slot (short: empty for an own track), profile (u8, the profile's index in the room descriptor), layers (u8), the composed layer L (u8), width and height (u16 each)

The grant's digest fixes its attributes, so the set's digest fixes everything §5 and §6 read of the room at t.

5. The layout

5.1 Integers

Every quantity of this section is an integer, and none is negative. ⌊a / b⌋, for b above 0, rounds down. fit(bw, bh, a, b), the largest rectangle of aspect a:b within bw × bh, is (bw, max(1, ⌊bw × b / a⌋)) when bw × b ≤ bh × a, and (max(1, ⌊bh × a / b⌋), bh) otherwise. A rectangle of w × h centred in (x, y, W, H) is (x + ⌊(W − w) / 2⌋, y + ⌊(H − h) / 2⌋, w, h). No product here exceeds 2^63.

5.2 Which participants have tiles

An ad frame's tiles are §5.9's. Otherwise, from S (§4.4):

1. V is S, less every participant whose video is stopped when omit_stopped is 1.
2. For a picture in picture, the PiP participant P is the first of V whose pip_attr attribute is true; with none there is no PiP window. For a grid there is none.
3. G is V without P, in order. The featured participant F is the first of G whose featured_attr attribute is true; it is moved to the front of G.
4. G is cut to its first max_tiles members, less one when there is a P. F, being first, is kept; when G is then empty there is no F.

5.3 The grid

A grid of n ≥ 1 tiles over a region (X, Y, W, H) is chosen by its reference aspect a:b: 16:9 for auto and video, 1:1 for square, 3:4 for portrait.

• Candidates. For each number of columns c from 1 to n, with r = ⌈n / c⌉ rows: c is a candidate when (c − 1) × gap is below W, (r − 1) × gap below H, and the cells, cw = ⌊(W − (c − 1) × gap) / c⌋ wide and ch = ⌊(H − (r − 1) × gap) / r⌋ high, are each at least 1. Its area is tw × th, where (tw, th) = fit(cw, ch, a, b).
• The choice. The candidate of the largest area; of several, the one with the most columns. §3.3's rule on gap makes c = ⌈√n⌉ a candidate for every n a descriptor allows.
• Placement. The rows are centred: the first row's top is y0 = Y + ⌊(H − (r × ch + (r − 1) × gap)) / 2⌋. Tile k, from 0, is in row i = ⌊k / c⌋ and column j = k mod c. Every row holds c tiles but the last, which holds n − (r − 1) × c; a row of m tiles is centred, beginning at x0 = X + ⌊(W − (m × cw + (m −
  1. × gap)) / 2⌋. Cell k is (x0 + j × (cw + gap), y0 + i × (ch + gap), cw, ch).
• The tile. Under auto, tile k is its cell; otherwise it is fit(cw, ch, a, b) centred in its cell.

With no tile the canvas is its background.

5.4 The featured tile

With F and n ≥ 2 members of G, the featured tile is (0, 0, width, mh), with mh = ⌊(height − gap) × 3 / 4⌋, and the other n − 1 members are a grid (§5.3) over the band below it, (0, mh + gap, width, height − mh − gap). The featured tile is its whole region: aspect is the grid tiles'. With F alone the featured tile is the whole canvas. With no F, the members of G are a grid over the whole canvas.

5.5 Picture in picture

With P and a G that is not empty, G is laid out over the canvas as §5.3 and §5.4 lay it out, and P's window is drawn over it. With P and an empty G:

• static: P's window, and the background;
• dynamic: P is laid out as though it had no pip_attr attribute: a grid of P alone, or, when P is featured, the featured tile, the whole canvas.

The window is pip_width × pip_height when both are above 0. Otherwise, with a:b the width and height of P's video source, or the canvas's when P's video is stopped, it is fit(bw, bh, a, b), where bw is pip_width when that is above 0, width − offset when only pip_height is, and the lower of ⌊width / 4⌋ and width − offset when neither is; and bh is pip_height when that is above 0 and height − offset otherwise. With o the offset and pw × ph the window, it is at (o, o) in the top left corner, (width − o − pw, o) in the top right, (o, height − o − ph) in the bottom left, and (width − o − pw, height − o − ph) in the bottom right.

5.6 Fill

A tile (tx, ty, tw, th) shows a source of sw × sh by its fill mode. The source is wider when sw × th ≥ sh × tw.

• Fill: the whole source, (0, 0, sw, sh), stretched over the whole tile.
• Cover: the source cropped to the tile's aspect, over the whole tile. A wider source's crop is cw = max(1, ⌊sh × tw / th⌋) wide, at (⌊(sw − cw) / 2⌋, 0, cw, sh); any other's is ch = max(1, ⌊sw × th / tw⌋) high, at (0, ⌊(sh − ch) / 2⌋, sw, ch).
• Contain: the whole source, letterboxed in the tile. A wider source is drawn into (tx, ty + ⌊(th − dh) / 2⌋, tw, dh) with dh = max(1, ⌊tw × sh / sw⌋); any other into (tx + ⌊(tw − dw) / 2⌋, ty, dw, th) with dw = max(1, ⌊th × sw / sh⌋).
• Default: cover for a grid tile, the PiP window and an ad tile, contain for the featured tile and a program tile (§5.9).

Each mode gives a source rectangle and a destination rectangle: the part of the source drawn, and where on the canvas it is drawn. A tile of stopped video has neither, and shows the background, as does the part of a tile outside its destination. How the source rectangle is scaled into the destination is the renderer's; a sink that holds a source at a layer below its composed one scales it to the composed layer's width and height first.

5.7 What the layout gives

The layout of a descriptor and a participant set gives:

• the order: S, every participant, by key;
• the tiles, in drawing order: the featured tile, then the grid's in G's order, then the PiP window; or an ad frame's, in §5.9's order. Each has its participant (or, for a session's ad track, the track, §5.10.4), its place (grid, featured, pip, program or ad), its z (0, or 1 for the PiP window and a squeeze's program window, drawn over the others), the tile, and, unless its video is stopped, its source and destination rectangles;
• the voices (§6).

A renderer paints the background over the canvas, then the tiles in order. No two tiles of z 0 overlap.

5.8 Re-layout

The participant set changes only at the t of a lifecycle event: a join, leave, publish, unpublish, grant_exchange, takeover or failover (an exchange always changes it, since the set names each participant's grant). At every such instant at or after effective the layout is computed again, and it changes when the sources, the attributes or the order do; between two such instants it does not change. The layout at t is the function of this section over the descriptor and the set at t, and of nothing a sink sees otherwise, so a sink that joins late lays the room out as one that was there throughout, and a grant exchange that changes an attribute re-lays the room out at the exchange's tick, with no participant leaving. On the sink of a session that holds an ad-track binding, an ad frame's layout also changes where the binding comes into effect and where it ends (§5.10.4).

5.9 Ad frames

An ad frame composes a program and an ad into one picture: a squeeze (layout 3) shrinks the program into a window in one corner and shows the ad in the L around it; a double box (layout 4) shows the program and the ad in two boxes side by side. The frame is an integer function of the canvas and the fields below, so the part of the picture the ad shows in is fixed by what the descriptor's signer signed, and a record of another extension that states the same fields fixes the same part (stitched-presentation §6). The ad is the room's, one participant every sink shows, or each session's own, on a per-session ad track (§5.10).

The frame of a canvas of width × height, with W its width and H its height:

• Squeeze: with pw = pip_width and ph = pip_height, the program window is pw × ph in the corner corner names: at (0, 0) top left, (W − pw, 0) top right, (0, H − ph) bottom left and (W − pw, H − ph) bottom right. The ad tile is the whole canvas. The ad region is the canvas less the window, two rectangles: the side band, beside the window and as high as it, (pw, y, W − pw, ph) for a window on the left and (0, y, W − pw, ph) for one on the right, with y the window's top; and the full band, the rest of the rows, (0, ph, W, H − ph) for a window at the top and (0, 0, W, H − ph) for one at the bottom.
• Double box: with bw = ⌊(W − gap) / 2⌋, the left box is (0, 0, bw, H) and the right box (bw + gap, 0, W − bw − gap, H). The program's box is the left for a corner of 1 and the right for 2, and the ad's the other. Each tile is its box under aspect auto; otherwise it is fit(box width, box height, a, b) centred in its box, with a:b the reference aspect of §5.3. The program tile and the ad tile are those, and the ad region is the ad tile alone.

The ad region's rectangles are in order of their top edge, and none of them overlaps the program's tile. Its share of the picture is ⌊10 000 × A / (W × H)⌋ basis points, with A the sum of its rectangles' areas.

The participants. From V (§5.2 step 1), the ad participant is the first whose pip_attr attribute is true, and the program participant is the first of the others once the featured participant (§5.2 step 3) is moved to their front. Then:

• with both, the tiles are, for a squeeze, the ad tile (place ad, z 0) and then the program window (place program, z 1, drawn over it); for a double box, the program tile and then the ad tile (both z 0);
• with the program participant alone, one tile, the canvas, of place program; with the ad participant alone, one tile, the canvas, of place ad; with neither, none.

No other participant has a tile; the voices are §6's, as for every layout.

5.10 Per-session ad tracks

An ad frame's ad region may show each session an ad decided for it alone: an ad of a pod-input record of the session (stitched-presentation §4.6). The program is the room's, laid out as §5.9 lays it, and every sink draws the same rectangles; the ad is the session's ad track, which the record's signer publishes for that session and binds to the frame with an ad-track binding.

5.10.1 The session's ad track

A session is the 16 bytes stitched-presentation §5.2 derives from a player's session key. The ad track of session s is named as live-rooms §6.1 names a track, with the owner ad_ followed by s as 32 lowercase hexadecimal digits: its Track Namespace is the room id, that owner and video, and its Track Name is 0. The owner is neither a key (64 digits) nor a slot (_ first), so it names no participant, and each of its bytes is one of a–z, 0–9 and _. Its track is the digest live-events §3.1 gives an anchor's track: SHA-256 over the namespace's rendering (live-rooms §6.2), one zero byte, and the name.

5.10.2 The binding

field	size	meaning
magic	4	WAT1 (ASCII)
version	1	1 in this revision
room	32	the room id
descriptor	32	the digest of the descriptor whose ad frame shows the track (§3.2)
record	32	the digest of the session's pod-input record (stitched-presentation §4.4)
session	16	the session
slot	1	the ad's slot in the record (stitched-presentation §4.2)
start	8	u64: the room-clock instant from which the ad region shows the track
width, height	2 each	u16: the track's picture, in pixels, each 1 to 1920
signer	32	the record's signer, which signs the binding
sig	64	Ed25519 by signer

The digest of a binding is SHA-256 over its bytes before sig; its signature is Ed25519 by signer over the 26 bytes wai:live-compose-ad-track (ASCII) and 0x01, followed by the bytes the digest covers, verified strictly (SPEC §7 Signatures). A reader reads exactly the bytes it is given, as §3.3 steps 1 to 3 read a descriptor with WAT1 for WCD1 (truncated, malformed, unknown_version); refuses as ad_track_invalid a binding whose width or height is 0 or above 1920; and then verifies the signature (bad_key, bad_signature).

A binding holds for its ad's duration_ms (stitched-presentation §4.2): it is in effect at a room-clock instant t when start ≤ t < start + 1000 × duration_ms.

5.10.3 Verifying a binding

A binding verifier is given a descriptor, a pod-input record and a binding. In this order, the first that fails names the refusal and where:

1. The descriptor reads (§3.3): its refusal, at descriptor. Who may sign it (§3.5) is checked against the room, where the room is composed.
2. The record reads and its signature verifies (stitched-presentation §4.4, §7.2 steps 1 and 2): pod_input_malformed, bad_key or bad_signature, at record.
3. The binding reads (§5.10.2), at binding; from here on every refusal is at binding.
4. Its signer is the record's signer: otherwise ad_track_not_permitted.
5. Its room is the descriptor's: otherwise wrong_room.
6. Its descriptor is the descriptor's digest: otherwise descriptor_mismatch.
7. It matches its session's composition: otherwise ad_track_mismatch, naming the first of these that differs:
   • layout: the descriptor's layout is an ad frame, 3 or 4;
   • record: its record is the record's digest;
   • session: the record is a record of a session, not a shared one, and its session is the binding's;
   • frame: the record's frame (stitched-presentation §4.3) is the descriptor's: its kind the layout, its width and height the canvas's, its corner the corner, its window_w and window_h the pip_width and pip_height for a squeeze and 0 for a double box, its gap the gap for a double box and 0 for a squeeze, and its aspect the aspect for a double box and 0 for a squeeze;
   • slot: the record has an ad of its slot.
8. Its start is at or after the descriptor's effective: otherwise not_yet_effective.

A binding that passes is accepted: the session's ad track shows the ad of slot of the session's own record in the ad region the descriptor fixes, from start for that ad's duration. A verifier states its digest, the session, the slot, the track, the span from start to start + 1000 × duration_ms, and the ad region and its share (§5.9), which are the record's frame's (stitched-presentation §6).

5.10.4 The layout of a session

On the sink of a session that holds an accepted binding in effect at t, the layout of an ad frame at t is §5.9's, with the session's ad track for the ad:

• the program participant is §5.9's, the first of V other than the room's ad participant, a featured one first; the room's ad participant, if there is one, has no tile and no voice;
• with the program participant, the tiles are §5.9's of both, the ad tile showing the session's ad track at the binding's width and height; without one, one tile, the canvas, of place ad, showing the track;
• the voices are §6's, less the room's ad participant's. The session's ad track is a picture: this revision mixes no sound of it.

A tile of the track names the track (§5.10.1) where every other tile names a participant. With no binding in effect, the layout is §5.9's. A session's layout is composed on its sink (§7): a composed track that a server compositor publishes (§8) is the room's, never a session's.

5.10.5 Reasons

reason	raised at
ad_track_invalid	§5.10.2
ad_track_not_permitted	§5.10.3 step 4
ad_track_mismatch	§5.10.3 step 7

A binding verifier also raises truncated, malformed, unknown_version, bad_key and bad_signature for a binding that does not read, wrong_room, descriptor_mismatch and not_yet_effective (§10) at binding, a descriptor's refusals at descriptor and a record's at record.

6. The audio mix

The voices are the members of S (before any is omitted for its video) that have an audio source, in S's order, the first max_voices of them. Each has its gain g, from 0 to 16 384 in units of 1/4096 (4096 is unity), and its pan p, from −4096 (left) to 4096 (right) (§4.3).

The mixer mixes one block of N frames. Each voice's input is its decoded audio for the block: N frames of signed 16-bit samples, of 1 or 2 channels (interleaved, left first), at the descriptor's sample_rate. Each output sample is a sum over the voices in signed 64-bit integers, one term per voice:

• mono output (channels 1): x × 4096 × g for a mono voice's sample x; (xL + xR) × 2048 × g for a stereo voice's samples xL and xR.
• stereo output (channels 2), with aL = 4096 − max(p, 0) and aR = 4096 + min(p, 0): to the left channel x × g × aL and to the right x × g × aR for a mono voice; xL × g × aL and xR × g × aR for a stereo one.

The output sample is ⌊(s + 2^23) / 2^24⌋ of the sum s, the arithmetic shift right by 24 bits of s + 2^23, which rounds half up; then saturated once to −32 768 … 32 767. A term is at most 2^41 in magnitude and a mix has at most 64 voices, so the sum is exact, the order of its terms changes nothing, and saturation applies to the whole once. The pan is a balance: at the centre both channels are at the voice's gain, and panning attenuates one channel only. A single voice at unity gain and the centre is passed through unchanged. A mixer that holds no decoded block of a voice for the output mixes it as silence.

A mixer is refused as mix_input_malformed when an output has other than 1 or 2 channels, or an input has other than 1 or 2 channels, has other than N frames, or a gain or pan out of range.

The mix is byte-exact over the decoded PCM: two mixers given the same blocks give the same samples. It is not byte-exact over what precedes and follows it. A decoder of a lossy codec need not reproduce another decoder's PCM bit for bit (RFC 6716 §6 holds a decoder to a quality measure, not to equality), and a lossy encode of the mix is not byte-reproducible: two encoders, or one encoder under two settings, give different bytes. The receipt (§8) therefore binds the input objects and the output object, not samples.

7. Where composition runs

On each sink, by default. A sink composes the room locally from the tracks it subscribes to, by §5 and §6, with no server decode or encode. It needs no power: the descriptor states how the room is laid out, and laying it out is reading it. A session's own ad in an ad frame is composed on its sink alone, from the session's ad track (§5.10.4).

On a server, under the compose power. A principal that publishes a composed track (its own video track for a layout, its own audio track for a mix, live-rooms §6.1) MUST be the descriptor's compositor, MUST hold the compose power in its grant in effect, and MUST sign a receipt (§8) for each object of a composed track it publishes. It is not a participant of its own composition (§4.2), so a composed track is never composed again. A subscriber subscribes to a composed track as to any other; the descriptor and the receipts let it check that the track is the room's composition.

8. The composition receipt

8.1 Layout

field	size	meaning
magic	4	WCR1 (ASCII)
version	1	1 in this revision
room	32	the room id
compositor	32	the compositor's key, which signs the receipt
descriptor	32	the digest of the descriptor it follows (§3.2)
t	8	u64: the room-clock instant of the output object
state	32	the digest of the participant set at t (§4.5)
media	1	1 audio (a mix), 2 video (a layout)
output	80	the output object
inputs	1 + 112 × n	a u8 count n, then each input: principal (32), the participant it is read from, and an object (80)
sig	64	Ed25519 by compositor

An object is 80 bytes: track (32), the digest live-events §3.1 gives an anchor's track, over the track's namespace as live-rooms §6.2 renders it, one zero byte, and its name; group and object (u64 each), its MOQT Location (MOQT §8.2); and digest (32), the SHA-256 of the object's payload. An input whose digest is 32 zero bytes states that the compositor held no object of that track for the output: it drew the tile with no video, or mixed the voice as silence.

The digest of a receipt is SHA-256 over its bytes before sig; its signature is Ed25519 by compositor over the 25 bytes wai:live-compose-receipt (ASCII) and 0x01, followed by the bytes the digest covers, verified strictly (SPEC §7 Signatures). A reader reads exactly the bytes it is given, as §3.3 steps 1 to 3 read a descriptor with WCR1 for WCD1 (truncated, malformed, unknown_version); refuses as receipt_invalid a receipt whose media is not 1 or 2, or that names two inputs of one participant and one track; and then verifies the signature (bad_key, bad_signature).

8.2 The inputs

The inputs of an output at t are, in this order:

• video (media 2): for each tile of the layout at t that shows video, in drawing order (§5.7), the tracks of its participant's video source;
• audio (media 1): for each voice, in order (§6), the tracks of its audio source.

The tracks of a source are named as live-rooms §6.1 names them: the namespace is the room, the owner (the participant's key, or _ and the slot's name for a slot) and the media kind; the names follow the profile's layering (live-rooms §6.3) at the composed layer L: track 0 for single and svc (an svc track carries every layer), track L for simulcast, and tracks 0 to L for staged, since a refinement is applied to the layers below it (staged-delivery). The receipt names, for each, the object it read.

8.3 Verifying a receipt

A receipt verifier is given a room (as live-rooms §9.1), a descriptor and a receipt. In this order, the first that fails names the refusal and where:

1. The room (§4.1): its refusal, at room.
2. The descriptor (§3.3, §3.5): at descriptor.
3. The receipt reads (§8.1), at receipt; and its room is the room id, otherwise wrong_room.
4. Its descriptor is the descriptor's digest: otherwise descriptor_mismatch.
5. Its t is at or after the descriptor's effective: otherwise not_yet_effective.
6. The descriptor names a compositor, the receipt's compositor is it, and that principal is present at t with a grant in effect that has the compose power: otherwise compositor_not_permitted.
7. Its output's track is a track of the compositor's own track of the receipt's media kind, published at t, at a layer from 0 to its layers − 1: otherwise output_not_published.
8. The set and the layout at t (§4, §5): their refusal, at set.
9. Its state is the set's digest: otherwise state_mismatch.
10. Its inputs, each its principal and its object's track, are §8.2's, in order: otherwise inputs_mismatch.

A receipt that passes is accepted: its output object was composed under this descriptor, from this set, out of objects of exactly the tracks the layout reads. Anyone holding the room's events and the descriptor recomputes the layout and checks that the inputs were the right ones; a party holding the objects checks their digests. The receipt does not fix the output's pixels or samples (§6).

9. Verification

A composition verifier is given a room, a descriptor and a list of instants. It runs these steps and stops at the first refusal, which it reports with the reason and where:

1. The room (§4.1): at room, with the room's own place (live-rooms §9.1) when the room is refused.
2. The descriptor (§3.3, §3.5): at descriptor.
3. Each instant, in the list's order: the instant is at or after effective (otherwise not_yet_effective), and the participants' attributes read (§4.3, otherwise attr_malformed): at set, with the instant.

Its outcome is refuse, with the reason and the place; or the composition: the descriptor's digest and, for each instant, the participants with their sources and tracks, the set's digest, the order, the tiles and the voices. A receipt verifier (§8.3) reports its refusals at the same places, and at receipt.

10. Reasons

The codes this extension adds, each with where it is raised:

reason	raised at
canvas_out_of_range	§3.3, §3.4
compose_invalid	§3.3
compose_not_permitted	§3.5
not_yet_effective	§8.3 step 5, §9 step 3
lifecycle_incomplete	§4.1
attr_malformed	§4.3
mix_input_malformed	§6
receipt_invalid	§8.1
descriptor_mismatch	§8.3 step 4
compositor_not_permitted	§7, §8.3 step 6
output_not_published	§8.3 step 7
state_mismatch	§8.3 step 9
inputs_mismatch	§8.3 step 10

The ad-track binding's codes are §5.10.5's.

A verifier of this extension also raises truncated, malformed and unknown_version for a descriptor or a receipt that does not read; bad_key and bad_signature for their signatures; wrong_room for one of another room (§3.5, §8.3); and every code a room verifier raises (live-rooms §11 and the live-events codes it lists), at room.

11. Conformance

live-conformance/compose/ holds the corpus, in one file, vectors.json, written from fixed seeds by wai_live_compose_vectors gen. It has:

• layouts: the layout function alone (§5, §6's voices), each case its parameters and its participants with their join orders, video sizes, audio and attributes, and the order, tiles and voices it gives, or its refusal. Grids of 0 to 12 tiles on a 16:9 canvas, and on portrait, square and auto ones; every fill mode over sources of four shapes and of extreme aspects; a featured participant present, absent, kept past the tile limit, and first in order of two; ties in order broken by join order and by key; stopped video shown and omitted; picture in picture in every corner under both behaviours; squeezes in every corner and double boxes on both sides, with the program alone, the ad alone, and a featured program, alone static and alone dynamic, with derived sizes and stopped video; and attributes that do not read.
• mixes: blocks of PCM and the mix of them (§6): passing through, both widths into both, saturation high and low and only at the end, pans and gains at their limits, rounding at ties, silence, and inputs refused.
• descriptors: records read (§3), with their digest and fields, or refused with each reason a descriptor alone can have.
• rooms: a studio room under the live-rooms descriptor, grants and chains that its vectors name, and four variants of it: a grant whose order does not read, a lifecycle record not delivered, a viewer's chat not delivered, and a viewer's record refused.
• compositions: a room, a descriptor and instants (§9), with the composition or the refusal. The grid over the whole session re-lays the room out at each lifecycle instant, among them a grant exchange that reorders it; the picture in picture composes a lower layer.
• receipts: a room, a descriptor and a receipt (§8.3), accepted with its digest and the set's, or refused with each reason a receipt can have.

A conforming verifier gives every case the outcome, the reason, the place and the data the corpus states. wai_live_compose_vectors verify and the test tests/live_compose.rs check that the file regenerates byte for byte, that wai-rs gives every case its outcome, that every reason of §10 refuses a case, and that §10 lists exactly the reference's codes. live-conformance/live_compose_verify.py is the second implementation, written from this text: it shares no code with wai-rs, verifies signatures with WAI's strict rule written from RFC 8032 (claim-conformance/_ed25519.py), verifies each room with the live-rooms corpus's own second implementation (live_rooms_verify.py) and replays its lifecycle to each instant itself, and gives every case the same outcome, rectangle for rectangle and sample for sample; it also mutates copies of the corpus and requires every copy to fail.

live-conformance/compose-ad-track/ holds the corpus of §5.10, in one file, vectors.json, written from fixed seeds by wai_compose_ad_track_vectors gen. It has:

• bindings: a descriptor, a pod-input record and a binding (§5.10.3), accepted with its digest, session, slot, track, span, ad region and share, under a squeeze and a double box; or refused with each reason a binding can have: bindings that do not read, out of their bounds, and signed over the wrong bytes (without the domain, under the pod-input record's domain, by a key that is not its signer, flipped); a signer that is not the record's; another room; another descriptor; and, for ad_track_mismatch, a grid's descriptor, another record, a shared record, another session's record, a record whose frame is another layout's, another window's, another corner's, another canvas's, another gap's and another aspect's, and a slot the record does not have; a start before the descriptor is effective; and descriptors and records that do not read;
• layouts: the layout of a session (§5.10.4) at instants before, at the start of, during, at the last microsecond of and at the end of its binding: under a squeeze and a double box, with a program and the room's own ad participant, with a program alone, with the room's ad participant alone, with no participant, under each fill, and with voices.

wai_compose_ad_track_vectors verify and the test tests/compose_ad_track.rs check that the file regenerates byte for byte, that wai-rs gives every case its outcome, and that §5.10.5 lists exactly the codes the reference adds. live-conformance/compose_ad_track_verify.py is the second implementation, written from this text: it shares no code with wai-rs; it reads descriptors and lays rooms out with the live-compose corpus's second implementation, reads records with the stitched-presentation corpus's, and verifies Ed25519 with WAI's strict rule; and it mutates copies of the corpus and drops each of its own checks in turn, and requires each to be caught.

12. Security and privacy considerations

• The layout is computed from what the room authority signed: the descriptor, or a principal's under the power the authority granted, and the attributes of grants the authority signed. A participant cannot place itself; a relay that drops or reorders lifecycle events is seen as a gap in a room key's chain (§4.1), not as another layout.
• A server compositor's receipt fixes which objects it composed and under which set. A compositor that left a participant out, read a layer other than the composed one, or composed a stale set is refused by §8.3. The receipt does not fix the output's pixels or samples, which a lossy encode does not reproduce (§6); a party that needs them composes locally from the inputs.
• Composition on each sink needs no server to decode the room's media, so a room whose tracks are end-to-end encrypted (confidentiality) can still be laid out: the layout reads the room's events and the grants, never media.
• The participant set names every publisher, its grant and its join order. Composing a room reveals nothing a member of it does not already hold, but a set digest published outside the room links the participants it names; a room that needs participation unlinkable across rooms gives principals a key per room (live-events §11).
• Attributes are read as the descriptor's keys name them and nothing else is read of a grant's attributes, so an attribute a descriptor does not name changes nothing.
• A per-session ad track is named by its session (§5.10.1), which stitched-presentation §5.2 derives from a key made for one playback session, so it links nothing across sessions. A binding proves that the signer of the session's record stated that the session's ad track shows that record's ad in this descriptor's ad region; one whose record's frame is not the descriptor's, or whose record is shared or another session's, is refused (§5.10.3), so the ad region a session's claim measures is the one its sink draws.