open-standards/wai/extensions/co-performance.md

# WAI Extension: Co-performance

> Status: Draft. Registers the record `wai.live.coperformance`: people and AI participants
> performing together in a room, and an audience joining in. A participant publishes an
> **intent**: the material it means to play next, ahead of the instant it plays, by the digest
> of an object or as a symbolic form (notes, a text line, a camera move), so the others can see
> what it is about to do. When the material plays, a verifier compares what played with what
> was announced, under a pinned rule, and a room key signs a **fidelity statement**: per
> participant, how many intents it announced and how many it kept, which anyone holding the
> room recomputes. A **turn schedule** says who holds the floor and for how long, in
> microseconds or in bars at a stated tempo, as solo turns or call-and-response pairs, and a
> **handover** passes the floor within a turn; an object a performer publishes outside its turn
> is refused by the relay or marked by the room clock, as the schedule states, and an object
> that arrives later than its grant's **latency budget** is marked, never mixed unmarked. An
> audience member whose grant carries the **audience-input power** submits signed inputs (a
> prompt, a sketch or a movement sample by digest, a vote) to a **selection window**; the room
> clock commits to the inputs once the window closes, a declared rule (a plurality vote, the
> top inputs by reactions, or a draw from a seed committed before the window opened) selects
> from what was committed, and every selected input is **credited** to the member who sent
> it. All of it rides in the room's events, as `meta` payloads that the room's authors sign, under
> one room-level **co-performance policy**, `WPP1`, that the room authority signs. No new
> transport. The reference implementation is the `co_performance` module of `wai-rs` (feature
> `co_performance`); the corpus is
> [`live-conformance/coperformance/`](../live-conformance/coperformance/), and its second
> implementation is `live-conformance/co_performance_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision: the live family this extension builds on, as merged
> ([live-events](live-events.md), [live-rooms](live-rooms.md), [live-compose](live-compose.md),
> [live-metering](live-metering.md), [live-moderation](live-moderation.md),
> [synthesis](synthesis.md) §8 and [ai-usage](ai-usage.md)); RFC 8032 and FIPS 180-4. No
> outside text is pinned: the rules of this extension are WAI's own.

## 1. Scope

In scope: the co-performance policy, its signature and the policy in effect at each instant
(§3); how this extension's statements ride in a room's events, who may sign each, and the grant
attributes it reads (§4); intents, the forms they plan, what played, the pinned rules that
compare the two, and the fidelity statement (§5); turn schedules, handovers, the floor, refusal
and marking, latency budgets and marks (§6); audience inputs, their windows, the commitment,
the selection rules, the selection and its credits (§7); where each rule is enforced (§8); and
verification (§9).

Out of scope: a transport; how a participant decides what to play, and how a model produces it;
how a performer transcribes what it played into a symbolic form (§5.4 compares the forms it is
given); how a sink renders an intent, a schedule or a mark; the contribution record that
credits a work's makers sample by sample, which is a separate extension, `wai.media.contribution`,
whose slot §7.8 names; and the review of what an audience submits, which is
[live-moderation](live-moderation.md)'s (§7.4).

Relationship to the core spec: `wai.live.coperformance` is a **record** ([SPEC](../SPEC.md)
§5): signed records a WAI tool emits and verifies. It names no payload format and is not
dispatchable.

Relationship to [live-events](live-events.md): this extension changes nothing in it. Its
statements are payloads of the `meta` kind (§4.2 there), which live-events leaves to
applications, as [live-moderation](live-moderation.md)'s statement is, so every statement is
signed by its author, chained, stamped with the room clock and rate-bounded by its author's
policy (live-events §5, §6). An audience member's reaction to an input is a `reaction` event
(§4.5 there) whose target is the input's digest; a moderator's verdict on an input is a
`decision` event (§4.11 there).

Relationship to [live-rooms](live-rooms.md): the room descriptor's and the grant's layouts are
unchanged. A room states its co-performance policy in a record of its own, `WPP1` (§3), chained
like the review policy of live-moderation §3.4 and the room usage record of ai-usage §4.2. The
audience-input power and the latency budget are grant attributes (§4.4), which live-rooms §4.4
signs with the grant and leaves to the extensions that read them; a new power bit would be
refused by every room verifier of the current revision (live-rooms §3.4: no power above bit 4).
The direct power (live-rooms §3.3, bit 1) is the power to sign a schedule, a handover or a
window.

Relationship to [synthesis](synthesis.md) §8 and live-rooms §4.4: an AI participant is marked in
its grant, its chat, reactions and captions state their machine origin, and its tracks are
synthesized media. Its intents are `meta` events and carry no origin of their own: the grant in
effect at their `t` says whose they are.

## 2. Terms

- **Room**, **room clock**, **principal**, **event**, **chain**, **grant**, **grant in effect**,
  **room key**, **merged order**: as in live-events §2 and live-rooms §2 and §9.2. A room key is
  the room authority's key or the clock's.
- **Statement**: the payload of a `meta` event whose first four bytes are one of the magics of
  §4.1. It is named by the digest of its event (live-events §3.2).
- **Performer**: a principal that publishes media. **Audience member**: a principal whose grant
  carries the audience-input power (§4.4).
- **Object**: a MOQT object of a track of the room (live-rooms §6.1), named by its track (the
  32-byte digest an anchor names a track by, live-events §3.1), its group and its object ID, and
  by the SHA-256 of its payload. An object's **arrival** is the room-clock instant it reached the
  relay; its **due** instant is the room-clock instant its material is for (§9.1).
- **Before**, of two events: earlier in the merged order.
- A **text** field is a u16 length and that many bytes of UTF-8 (RFC 3629): no overlong form, no
  surrogate, nothing above U+10FFFF.

Integers are little-endian; a signed integer is two's complement. A digest is SHA-256 (FIPS
180-4), 32 bytes. Ed25519 is RFC 8032's, verified strictly ([SPEC](../SPEC.md) §7
*Signatures*): a key that fails step 1 or 2 is `bad_key`, and a signature that fails a later
step is `bad_signature`. *µs* is microseconds of room clock.

## 3. The co-performance policy

### 3.1 Layout

| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WPP1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id (live-rooms §3.2) |
| `prev` | 32 | the digest of the policy this one replaces (§3.4); 32 zero bytes for the first |
| `effective` | 8 | u64: the room-clock instant from which this policy applies; 0 for the first |
| `horizon_ms` | 4 | u32: how far ahead of its own `t` an intent may reach (§5.3), above 0 |
| `window_ms` | 4 | u32: the match window (§5.4): how far from its planned instant material may play and still be the intent's, above 0 |
| `time_tol_us` | 4 | u32: the timing tolerance of the notes rule (§5.5) |
| `angle_tol_mdeg` | 4 | u32: the angle tolerance of the camera rule (§5.5), in thousandths of a degree |
| `modes` | 1 | the turn modes a schedule may state (§6.5): bit 0 refuse, bit 1 mark; at least one, no other bit |
| `sig` | 64 | §3.2 |

### 3.2 Digest and signature

The **digest** of a policy is SHA-256 over its record from `magic` through `modes`. The
**signature** is Ed25519 by the room descriptor's `authority` over the 26 bytes
`wai:co-performance-policy` (ASCII) and 0x01, followed by the bytes the digest covers.

### 3.3 Reading and rules

A reader reads exactly the bytes it is given as one policy, in the order of §3.1, and refuses at
the first of these that holds:

1. Fewer than 4 bytes: `truncated`. A magic other than `WPP1`: `malformed`. No byte after the
   magic: `truncated`. A version other than 1: `unknown_version`.
2. The bytes end before a field the layout requires: `truncated`.
3. Bytes after `sig`: `malformed`.

It then holds the policy to these rules, and refuses one that breaks any of them as
`co_policy_malformed`: `horizon_ms` and `window_ms` are above 0, and `modes` is 1, 2 or 3.
Then its `room` is the room id (`wrong_room`), its signature verifies (`bad_key`,
`bad_signature`), and its link holds (§3.4, `co_policy_unlinked`).

### 3.4 The policy in effect

A verifier is given a room's policies in a list. The first has `prev` zero and `effective` 0.
Each later one has as `prev` the digest of the one before it in the list, and an `effective`
above that one's. A policy that breaks this is refused `co_policy_unlinked`. The **policy in
effect** at room-clock instant *t* is the last of the list whose `effective` is at or below *t*.

A policy is replaced, never amended. A statement is read under the policy in effect at its
event's `t`, and an intent is compared (§5.4) under the policy it was announced under. A
statement of this extension at an instant with no policy in effect is refused `co_no_policy`.

## 4. Statements

### 4.1 Carriage

| magic | statement | section | who signs it |
|---|---|---|---|
| `WPI1` | intent | §5.1 | any principal of the room |
| `WPF1` | fidelity statement | §5.6 | a room key |
| `WPT1` | turn schedule | §6.1 | a room key, or a principal whose grant in effect has the direct power |
| `WPH1` | handover | §6.3 | the holder of the turn, a principal with the direct power, or a room key |
| `WPM1` | mark | §6.7 | a room key |
| `WPW1` | selection window | §7.2 | a room key, or a principal with the direct power |
| `WPA1` | audience input | §7.3 | an audience member |
| `WPK1` | commitment | §7.5 | a room key |
| `WPS1` | selection | §7.7 | a room key |

Each statement is the payload of a `meta` event: its magic, a `version` byte of 1, and the
fields its section lists. A `meta` payload whose first four bytes are none of these magics is
not this extension's, and a sink that does not implement it shows none of them: they are timed
metadata for its application (live-events §4.2). Every statement is bounded by its author's
policy (live-events §5, `max_payload`), 1024 bytes for a room key, which bounds the counts each
section allows.

### 4.2 Reading

A statement that ends before a field, that holds bytes after its last field, whose `version` is
not 1, or whose field holds a value its section does not allow, is `co_statement_malformed`. A
text field that is not UTF-8 is `co_statement_malformed`.

### 4.3 Who signs

A statement whose author is not one the table of §4.1 allows is `co_not_permitted`, but for a
handover, whose author §6.3 checks once it has found the turn (`handover_not_holder`), and an
audience input (`input_not_permitted`, §7.3). The
direct power is that of the author's grant in effect at its event's `t` (live-rooms §9.2 step
4); a room key holds every power. Since a statement is a `meta` event, the room verifier has
already held it to its author's grant: a principal whose policy does not permit `meta` cannot
send one (live-rooms §9.2 step 4, `kind_not_permitted`).

### 4.4 Attributes

This extension reads two grant attributes (live-rooms §4.4):

| key | value | meaning |
|---|---|---|
| `audience_input` | a decimal integer from 1 to 255 | the **audience-input power**: the principal may submit inputs (§7.3), at most this many to one window |
| `latency_budget_ms` | a decimal integer from 1 to 600 000 | the principal's **latency budget** (§6.6) |

A **decimal integer** is 1 to 6 of the digits `0`–`9`, the first not `0`. A grant whose
attribute of either key is not of its row is refused `co_attr_malformed`. A grant without
`audience_input` gives no audience-input power; one without `latency_budget_ms` gives no budget.
The authority states a budget per role by giving every grant of the role the same value; an AI
participant's role is where it matters most, since its output is computed and may be late.

## 5. Intents

### 5.1 The intent

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPI1`, 1 |
| `track` | 32 | the track the material plays on (§2); 32 zero bytes for a text intent |
| `at` | 8 | u64: the room-clock instant the material plays from |
| `span` | 4 | u32: how long it lasts, in µs from `at`, above 0 |
| `form` | 1 | 1 object, 2 notes, 3 text, 4 camera |
| `body` | … | by `form` (§5.2) |

An intent's **author** is its event's principal, and its **horizon** is the stretch from its
event's `t` to `at` + `span`: what it says about the time ahead. The intent track of a
participant is its intents in its events track (live-rooms §6.1): every subscriber of the room
that subscribes to that track receives each intent when it is sent, before the material it
announces plays.

### 5.2 Forms

- **object** (1): `digest` (32), the SHA-256 of the payload of the object the author will
  publish.
- **notes** (2): `count` (u8, 1 to 64), then each note: `onset` (u32, µs from `at`, below
  `span`), `duration` (u32, above 0), `pitch` (u8, 0 to 127, a MIDI note number) and `velocity`
  (u8, 1 to 127), in ascending order of (`onset`, `pitch`), no two notes with both equal. A
  phrase is its notes.
- **text** (3): `text`, a text field of 1 to 500 bytes: a line the author will say, as a `chat`
  or a `caption`.
- **camera** (4): `pan` (i32, −180 000 to 180 000) and `tilt` (i32, −90 000 to 90 000), in
  thousandths of a degree, and `zoom` (u32, Q16.16, above 0): where the author's camera will be
  pointed and how far zoomed, from `at` for `span`.

`track` is zero exactly for a text intent. A form outside 1 to 4, a body that does not read as
its form says, and a `track` that breaks the rule, are `co_statement_malformed`.

The bodies of forms 2 and 4 are also the **symbolic payloads** a performer publishes: an object
whose payload is a notes body (the count and the notes, with no `at`) or a camera body (12
bytes) states in symbols what it played (§5.4).

### 5.3 Rules

An intent is announced ahead, and for a short horizon:

1. `at` is above its event's `t`: otherwise `intent_not_ahead`.
2. `at` + `span` − `t` is at most `horizon_ms` × 1000 of the policy in effect at `t`:
   otherwise `intent_beyond_horizon`.

### 5.4 What played

The intent's **window** is `window_ms` × 1000 µs of the policy in effect at its event's `t`.
Its **candidates** are:

- for forms 1, 2 and 4: the objects the verifier is given (§9.1) whose principal is the
  intent's author and whose track is the intent's `track`, with a due instant within the window
  of `at` (|`due` − `at`| at most the window);
- for form 3: the author's `chat` and `caption` events the room accepted, with a `t` within the
  window of `at`.

What **played** is the candidate nearest `at`, by |`due` − `at`| or |`t` − `at`|, and of two
equally near, the first of the objects given, or the first event in the merged order. With no
candidate the intent was **missed**. Otherwise it was **kept** when what played matches it
under its form's rule (§5.5), and **diverged** when it does not.

For forms 2 and 4 the rule reads the played object's payload, which the verifier is given by
its digest (§9.1). A played object whose payload is not given is `payload_missing`, at the
intent: what played cannot be compared.

### 5.5 The pinned rules

- **object**: the played object's digest is the intent's `digest`.
- **notes**: the played payload reads as a notes body (§5.2), with the same `count`; and for
  each *i*, in order, the two notes have the same `pitch`, their absolute onsets (`at` +
  `onset` of the intent's, `due` + `onset` of the played object's) differ by at most
  `time_tol_us`, and their durations by at most `time_tol_us`. `velocity` is not compared: it is
  how a note is played, not which.
- **text**: the event's text is the intent's text, byte for byte.
- **camera**: the played payload reads as a camera body, its `pan` and `tilt` each differ from
  the intent's by at most `angle_tol_mdeg`, and its `zoom` is the intent's.

`time_tol_us` and `angle_tol_mdeg` are those of the policy in effect at the intent's `t`. A
payload that does not read as the form is a diverged intent, not a malformed one: the object
is media, and it did not play what was announced.

### 5.6 The fidelity statement

A room key states, per participant, how often it did what it said it would:

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPF1`, 1 |
| `through` | 8 | u64: the room-clock instant it is stated through, at or below its event's `t` |
| `count` | 1 | u8: the entries |
| `entries` | 52 × *n* | each: `principal` (32), `announced`, `kept`, `diverged`, `missed` (u32 each) |

A fidelity statement whose `through` is above its event's `t`, or whose entries are not in
strictly ascending order of `principal`, is `co_statement_malformed`.

It covers the intents before it whose `at` plus their window is at or below `through`, and it
reads them as of its own instant: their candidates are the objects whose arrival is at or below
its event's `t`, and the events before it. Its entries are, in ascending order of `principal`,
every author of a covered intent, with the number it announced and the number of each outcome.
A verifier recomputes the entries from the room; a statement whose entries differ, in any
principal or any count, is `fidelity_mismatch`. A participant kept *k* of *a* intents: a
fraction a sink may show beside it, as two integers.

## 6. Turns

### 6.1 The schedule

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPT1`, 1 |
| `t0` | 8 | u64: the room's `t0` (live-rooms §3.1), for a relay that holds no descriptor (§8) |
| `from` | 8 | u64: the instant from which this schedule governs |
| `mode` | 1 | 1 refuse, 2 mark (§6.5) |
| `beat_us` | 4 | u32: µs per beat, the tempo; 0 when lengths are in µs |
| `beats_per_bar` | 1 | u8: 1 to 32 when `beat_us` is above 0; 0 otherwise |
| `segments` | 1 + … | a u8 count (1 to 4), then each segment: `start` (u64), `length` (u32), `turns` (u16, 1 to 1024), `pairs` (u8, 0 or 1), `holders` (a u8 count, 1 to 4, then each holder's key, 32 bytes) |

A segment's **turn length** *L* is `length` µs when `beat_us` is 0, and `length` ×
`beats_per_bar` × `beat_us` µs otherwise: `length` bars at the stated tempo. A schedule is
`co_statement_malformed` when: `t0` is not the descriptor's; `mode` is not 1 or 2; `beat_us` and
`beats_per_bar` break their rule; a segment's *L* is 0; a segment starts before `from`, or before
the end of the segment before it (its `start` + `turns` × *L*); an end does not fit a u64; two
holders of a segment are one key; or a segment with `pairs` 1 has an odd number of turns or one
holder. Then: `from` is at or after its event's `t`, since a schedule never changes the past
(`schedule_retroactive`); and `mode` is one the policy's `modes` allows (`mode_not_allowed`).

The **schedule in effect** at instant *x* is, of the schedules the room holds whose `from` is at
or below *x*, the last in the merged order. A later schedule replaces an earlier one from its
`from` on.

### 6.2 Turns and pairs

Turn *i* of a segment, from 0, runs from `start` + *i* × *L* up to, but not including,
`start` + (*i* + 1) × *L*, and its **scheduled holder** is the segment's holder *i* mod *n*, of
its *n* holders. The holders of a segment take turns in order, so two holders trading four-bar
phrases at 120 beats a minute are one segment: `beat_us` 500 000, `beats_per_bar` 4, `length`
4, two holders.

In a segment with `pairs` 1, turn 2*j* is a **call** and turn 2*j* + 1 its **response**: the
next holder answers what the call played. A verifier states each pair. The floor (§6.4) is held
the same way in either.

### 6.3 Handovers

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPH1`, 1 |
| `at` | 8 | u64: the instant the floor passes, at or after its event's `t` |
| `to` | 32 | the principal that takes it |

A handover passes the turn that holds `at`, under the schedule in effect at `at` of those before
it, to `to`, from `at` to the turn's end; the schedule's next turn is unchanged. It is `handover_invalid` when
`at` is before its event's `t`, when no turn of the schedule in effect holds `at`, or when `to`
is not a holder of that turn's segment. Its author is the turn's holder at `at` (below), a
principal with the direct power, or a room key: otherwise `handover_not_holder`.

The **holder** of a turn at instant *x* is the `to` of the handover into that turn whose `at` is
the latest at or below *x* (of two at one `at`, the later in the merged order), or its
scheduled holder when there is none.

### 6.4 The floor

A segment **governs** its holders while it runs. An object of principal *P* is **out of turn**
when, at its arrival, the schedule in effect has a segment that is running and whose holders
include *P*, and the turn running then is held, at that instant, by another principal. An
object of a principal no running segment names, or that arrives while no segment runs, is not
governed: the floor is open. An accompanist that plays under every turn is left out of the
holders.

The floor governs the objects of performers' media tracks. It does not govern events: a
principal's intents and chat stay its own to send.

### 6.5 Refuse or mark

A schedule's `mode` says what happens to an out-of-turn object:

- **refuse** (1): the relay **MUST NOT** forward it. The relay is the component on the path of
  every object before it fans out, so a refusal there is the floor held for every subscriber at
  once (§8).
- **mark** (2): the relay forwards it, and the room clock **MUST** sign a mark (§6.7) naming it
  as out of turn. Every sink and every compositor then receives it with the mark.

Which to state is the room's, through its policy's `modes` and each schedule's `mode`. Refuse
fits an exchange where the floor is the point: timed spoken turns, a game's moves. Mark fits
music: a pickup that starts before the bar line, or a phrase that rings past it, is part of the
music, and dropping it would cut the performance where the overlap is the performance; the mark
keeps the overlap visible and leaves a mixer to decide.

### 6.6 Latency budgets

An object of principal *P* is **late** when its arrival is after its due instant by more than
*P*'s budget: `latency_budget_ms` × 1000 µs of the grant in effect for *P* at the object's
arrival (§9.1). The relay, or the room clock acting for the room, **MUST** mark a late object
(§6.7) before it is mixed: a late object is forwarded with its mark, and a mixer that holds the
mark decides whether the object is still of use. A compositor ([live-compose](live-compose.md)
§7) **MUST NOT** mix an object it knows to be late without the mark, and a sink that mixes
locally takes the same mark. A late object is never mixed unmarked.

### 6.7 Marks

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPM1`, 1 |
| `track` | 32 | the object's track |
| `group`, `object` | 8 each | u64: its group and object ID |
| `flags` | 1 | bit 0: out of turn; bit 1: late; at least one, no other bit |

A mark is the room clock's, signed as it forwards the object. A verifier checks it against the
objects it is given (§9.2 step 7): the object it names is one of them, and each flag it sets
holds of that object. Otherwise `mark_unfounded`. A mark can be no accusation the room's own
records do not support.

## 7. Audience co-creation

### 7.1 The audience-input power

A principal is an **audience member** while its grant in effect carries `audience_input`
(§4.4). Its grant's event policy permits `meta`, and, to react to inputs, `reaction`; the
policy's rate windows (live-events §6.3) bound its inputs as they bound every event, and
`audience_input` bounds how many it submits to one window (§7.3).

### 7.2 Selection windows

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPW1`, 1 |
| `rule` | 1 | 1 plurality, 2 reactions, 3 draw (§7.6) |
| `types` | 1 | the input types it takes: bit 0 prompt, bit 1 sketch, bit 2 vote, bit 3 movement |
| `options` | 1 | for rule 1, the number of options, 2 to 32; otherwise 0 |
| `k` | 1 | how many it selects, 1 to 8 |
| `close` | 8 | u64: the instant it closes, above its event's `t` |
| `seed_commit` | 32 | for rule 3: SHA-256 over the 24 bytes `wai:co-performance-seed` (ASCII) and 0x01 and the 32-byte seed; otherwise 32 zero bytes |
| `question` | text | what it asks, 0 to 500 bytes |

A window of rule 1 takes votes alone (`types` 4); one of rules 2 and 3 takes any of prompts,
sketches and movement samples, and no vote. A window is `co_statement_malformed` when it breaks
these rules, when its `rule`, `options`, `k`, `seed_commit` or `question` is outside what the
table allows, or when its `close` is not above its event's `t`. A window is named by the digest
of its event.

### 7.3 Inputs

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPA1`, 1 |
| `window` | 32 | the window's digest |
| `type` | 1 | 1 prompt, 2 sketch, 3 vote, 4 movement |
| `body` | … | a prompt: a text field of 1 to 500 bytes; a sketch or a movement sample: the digest (32) of the object that holds it; a vote: `option` (u8) |

An input is checked, in this order:

1. its author is an audience member at its `t`: otherwise `input_not_permitted`;
2. `window` names a window before it: otherwise `window_unknown`;
3. its `type` is one the window takes, and a vote's `option` is below the window's `options`:
   otherwise `input_type_mismatch`;
4. counting it, its author has sent at most `audience_input` inputs naming that window:
   otherwise `input_over_limit`.

A relay **MUST NOT** forward an input these checks refuse, nor one naming a window whose
commitment (§7.5) it has forwarded: the commitment fixed the window's inputs, and an input
the room holds before the commitment in the merged order, with a `t` below `close`, is one it
counts (§7.5, `commit_mismatch`). A relay holds an input to these rules with the grant its
author's session proved (live-rooms §10.5) and the windows and commitments of the events tracks
it forwards, and withholds an input it refuses, as live-moderation §8.1 withholds a record:
never replaced, so its author's chain shows the gap. An input whose `t` is at or after the
window's `close` is not refused: it was in flight when the window closed, and it is not
**eligible** (§7.5). A sketch's or a movement sample's object is published as any object of the
room, under the member's grant; the input names it, and a performer that uses it fetches it by
that name.

A relay **MUST** count an author's inputs in the order of the author's chain (live-events
§6.2), as a verifier does, and never in the order they reach it: the records of one chain can
reach it on several streams, one for each group (live-events §8.1), which it may read side by
side. It decides an input once it has received every record of the author's chain below the
input's `seq` and decided every input among them. An input that reaches it ahead of a gap in
the author's chain it holds, for at most a bound of its own, and decides once the gap fills; one
whose gap has not filled when the bound runs out it refuses as `input_out_of_order`, and it
counts the author's chain from the record after it. It also refuses as `input_out_of_order` an
input that reaches it after it has forwarded a later input of the same author, since counting it
then would count it after an input that follows it in the chain. A relay that holds no earlier
record of an author's chain, because it started after the chain did, so refuses that author's
first input once its bound runs out. Neither refusal is a verifier's: a verifier holds each
chain in its order and counts each input where §9.2 step 4 meets it.

### 7.4 Moderation

[Live-moderation](live-moderation.md) applies to inputs as to any `meta` event: a review policy
may hold them for review (its §6.1), and a moderator decides on them (its §4). An input or a
reaction is **restricted** at an instant when, of the `decision` events the room accepted before
that instant whose target is its digest, the last denies or redacts it (live-events §4.11;
live-moderation §4.5 keeps that decision the standing one of its line). A restricted input is
not eligible, and a restricted reaction is not counted.

### 7.5 The commitment

Once a window has closed, a room key commits to its inputs, before any rule runs:

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPK1`, 1 |
| `window` | 32 | the window's digest |
| `count` | 4 | u32: the eligible inputs |
| `root` | 32 | SHA-256 over the 26 bytes `wai:co-performance-commit` (ASCII) and 0x01, `window`, `count` (u32), and each eligible input's digest in the merged order |

The **eligible** inputs of a window are the inputs naming it that the room accepted, with a `t`
below `close`, that are not restricted at the commitment. A commitment is checked: `window`
names a window before it (`window_unknown`); its `t` is at or after `close` (`commit_early`);
the window has no commitment before it (`statement_repeated`); and its `count` and `root` are
those of the eligible inputs (`commit_mismatch`). The commitment fixes the set a rule runs over
before the rule runs, so no input can be added or removed after the outcome is known; for a
draw, the seed was committed when the window opened, before any input (§7.2).

### 7.6 The rules

Each rule is a function of the window, the eligible inputs, and the room's events before the
commitment:

- **plurality** (1): each member's last eligible vote, in the merged order, counts. The
  options are ordered by their counts, the greatest first, and by option number between equal
  counts; the selection is the first `k` of them with a count above 0.
- **reactions** (2): an eligible input's score is the number of principals whose last
  `reaction` event targeting its digest, with a `t` below `close`, does not retract
  (live-events §4.5, `flags` bit 1 clear) and is not restricted at the commitment. The inputs
  are ordered by score, the greatest first, and in the merged order between equal scores; the
  selection is the first `k`.
- **draw** (3): the inputs are ordered by SHA-256 over the 24 bytes `wai:co-performance-draw`
  (ASCII) and 0x01, the seed, and the input's digest, the least first; the selection is the
  first `k`.

A selection holds fewer than `k` when fewer are eligible (or, for plurality, fewer options were
voted for).

### 7.7 The selection

| field | size | meaning |
|---|---|---|
| `magic`, `version` | 5 | `WPS1`, 1 |
| `rule` | 1 | the window's rule, 1 to 3, so that the selection reads on its own |
| `window` | 32 | the window's digest |
| `commit` | 32 | the digest of the window's commitment |
| `seed` | 32 | for a draw, the seed; otherwise 32 zero bytes |
| `picks` | 1 + … | a u8 count, then each pick: for plurality, an option (u8); otherwise an input's digest (32) |
| `credits` | 1 + 69 × *m* | a u8 count, then each credit: `step` (u8, 1 or 2), `party` (32), `ref` (32), `count` (u32) (§7.8) |

A selection is checked, in this order:

1. `window` names a window before it (`window_unknown`); it is the window's first selection
   (`statement_repeated`); and `commit` is the digest of the window's commitment, before it
   (`selection_uncommitted`).
2. For a draw, SHA-256 over `wai:co-performance-seed`, 0x01 and `seed` is the window's
   `seed_commit` (`seed_mismatch`); for the other rules `seed` is zero
   (`co_statement_malformed`).
3. No pick names an input whose `t` is at or after `close`: such an input was submitted after
   the window closed, and a selection that counts it is `input_late`.
4. Its `rule` is the window's, and its picks are that rule's selection over the committed
   inputs (§7.6), in its order: otherwise `selection_mismatch`.
5. The credits are one per pick, in the picks' order (§7.8). A pick with no credit is
   `input_uncredited`; a credit that is not the one its pick gives, or one past the last pick,
   is `credit_mismatch`.

### 7.8 Credit and the contribution slot

Every input that reaches the performance is credited, and a selection is where it reaches it:

- a pick of rule 2 or 3, input *X*: `step` 1 (**author**), `party` the key of *X*'s author,
  `ref` *X*'s digest, `count` 1. The member made material the performance uses.
- a pick of rule 1, option *o*: `step` 2 (**direct**), `party` SHA-256 over the 26 bytes
  `wai:co-performance-voters` (ASCII) and 0x01 followed by the keys of the members whose
  counted vote is *o*, in ascending order, `ref` SHA-256 over the 26 bytes
  `wai:co-performance-option` (ASCII) and 0x01, the window's digest and *o*, and `count` the
  number of those members. The members steered the performance together; each finds its own
  key among them by replaying the room.

**The contribution slot.** A contribution record (`wai.media.contribution`, a separate
extension) credits the makers of a piece of media by party and by step. Each credit here maps
onto it: an author credit onto a party of role `audience` with the member's key and a step
whose act is `author`, over the material its `ref` names; a direct credit onto a step whose
act is `direct`, on behalf of the members its `party` names. This revision writes no
contribution record and depends on none; the credit is complete without one, and a record that
cites a selection takes its parties from these fields.

## 8. Where each rule is enforced

- **The relay** holds the floor (§6.5): it reads the schedules and handovers from the events
  tracks it already forwards (as it reads decisions, live-moderation §6.3), puts each media
  object's arrival on the room clock with the schedule's `t0`, and under refuse does not forward
  an out-of-turn object. It does not forward an input §7.3 refuses, nor an event live-rooms
  §9.2 refuses. Refusal is the relay's because only the relay sees each object before every
  subscriber does. A relay **SHOULD** count what it held back in its per-session record
  ([live-metering](live-metering.md) §3): the objects out of turn, and the inputs by the reason
  §7.3 gives each, `window_committed` for one naming a window already committed, or
  `input_out_of_order` for one it could not count in its author's chain order (§7.3), which are
  the relay's alone and no verifier's.
- **Across relays.** Where a room is carried by several relays ([live-rooms](live-rooms.md)
  §10.7), the relay that holds a performer's object to the floor is the one the performer's
  session publishes to: the object's arrival there is the one put on the room clock. A relay
  that receives the object over a hop link forwards it as that relay decided and **MUST NOT**
  hold it to the floor again: its arrival trails by the link, and a second decision could refuse
  an object the first forwarded, so that subscribers on two relays would be given two
  performances. An audience input is likewise decided by §7.3, in its author's chain order, at
  the relay its author's session publishes to; a relay that receives it over a hop link neither
  holds it nor decides it again, and forwards what that relay forwarded, in the order it
  arrives, so that the room clock and every subscriber, on whichever relay, count the same inputs
  in the same order. Such a relay still reads the schedules, handovers, windows and commitments
  of the events tracks it forwards, as §6.3 and §7.2 give whose statements it honours, so that
  it holds the floor and the inputs for the performers and the authors whose sessions reach it.
- **The room clock** signs marks (§6.7), commitments, selections and fidelity statements: what
  the room states about what happened. A clock that is the relay's key signs marks as it
  forwards. A clock that is a session of its own on the relay takes each performer's objects
  as the relay forwards them to it: an object's arrival is the instant the clock received it,
  its due instant the object's timestamp, which a performer states on the room clock (§9.1), and
  the clock signs each mark as it takes the object. In a room that admits by age, such a
  session proves a room key in its SETUP and is the room's own service, which the serving rule
  of rated tracks does not hold ([age-assurance](age-assurance.md) §5.5): the relay serves it
  every performer's objects whatever their rating. Its log of the objects it took is the
  objects of §9.1 for the room, so a verifier given the room and that log finds every mark
  founded and no object out of turn or late without its mark. Such a clock's arrival trails the
  relay's by the hop between them: a performer that sends only while it holds the floor, from a
  little after its turn starts to a little before it ends, keeps its objects in its turn by both
  instants, which a schedule that refuses needs (§12).
- **The room clock, over time.** It commits to a window once it has closed and an input sent
  before the close has had time to arrive, and selects from it once the commitment is held; a
  draw's seed is the clock's own, drawn when the window opens and kept until its selection. It
  states fidelity through an instant some seconds behind its own, so that what played for the
  intents it covers, and every event about them, has reached it. It holds the room as a
  verifier does (every principal's chain and its own, the grants, the policy and its log), and
  each of these statements is what the verifier will recompute from them.
- **A room key on request.** A room key states a schedule, a handover or a window as its own.
  A room service whose clock does so for a participant that asks holds the request to that
  participant's grant and powers as §4.3 holds the participant's own statement (the direct
  power for a schedule or a window; the turn's holder, or the direct power, for a handover) and
  authenticates it with the grant's proof of possession over HTTP (live-rooms §10.5.3). In the
  room the statement is the room key's.
- **The room authority** signs the policy (§3), and grants the audience-input power and the
  latency budgets (§4.4).
- **Sinks and compositors** show an intent with its author, never mix a late object without its
  mark (§6.6), and show a selection with its credits.

After the fact, a verifier (§9) checks every one of these against the room: a relay that
forwarded an out-of-turn object under refuse, a clock that left one unmarked, or one that
marked an object that was neither, is found out.

## 9. Verification

### 9.1 Inputs

A co-performance verifier is given what a room verifier is given (live-rooms §9.1: a
descriptor, grants, a key-log head or none, and the chains), and:

- **policies**: the room's co-performance policies, in a list (§3.4);
- **objects**: the media objects the relay forwarded from the room's performers, each its
  principal, its track, group and object ID, its digest, its arrival and its due instant. The
  due instant is the room-clock instant its material is for: its capture instant on the room
  clock, or the planned instant of the beat it plays on. A performer in a room with this
  extension states it as the object's timestamp (a `WAI1` envelope's `target.timestamp_us`), on
  the room clock. The objects are the caller's statement, as live-moderation §7.1's are: a
  relay's log, a room clock's (§8), its recording or its DVR window states them;
- **payloads**: the symbolic payloads of objects, each named by its SHA-256.

The **grant in effect** for a principal at an instant, for an object, is the grant of its last
`join` or `grant_exchange` at or before that instant, unless a `leave` of it follows, when it
has none (live-rooms §7.2).

### 9.2 Steps

It runs these steps and stops at the first refusal, which it reports with the reason and where:

1. **The room** (live-rooms §9): a refusal, at `room` with the room verifier's own place. A gap
   is not a refusal; the steps go on, and those that recompute from the whole room are not made
   (below).
2. **The policies**, in the list's order (§3.3, §3.4): at the policy's index.
3. **The grants' attributes** (§4.4), in the grants' order: at the grant's index.
4. **The statements**, in the merged order: each `meta` event the room accepted whose payload
   begins with a magic of §4.1, by its section: §4.2, then §3.4 (`co_no_policy`), then §4.3,
   then the rules of its section, in the order each section gives. A commitment, a selection
   and a fidelity statement are recomputed at their place in the merged order. At the record.
5. **The intents** (§5.4), in the merged order: each one's outcome, and `payload_missing`, at
   the intent's record.
6. **The objects**, in the order given: out of turn under refuse (`turn_violated`); out of
   turn under mark without a mark that sets bit 0 (`turn_unmarked`); late without a mark that
   sets bit 1 (`late_unflagged`). At the object's index.
7. **The marks**, in the merged order (§6.7): `mark_unfounded`, at the record.
8. **The outcome** (§9.3).

With a gap, step 4 does not check a commitment's `count` and `root` (`commit_mismatch`), a
selection's picks and credits (step 3 onward of §7.7), or a fidelity statement's entries: each
depends on events the verifier may not hold. Every other check is made, over what it holds.

### 9.3 Outcomes and what a verification states

- **refuse**: the reason and where (§9.2).
- **gap**: no refusal, and the room has a gap: the gaps, under the code `delivery_gap`, and what
  **accept** states, computed from what the verifier holds.
- **accept**: no refusal and no gap.

With accept or gap, the verifier states:

- the policies' digests;
- each intent, in the merged order: its digest, author, form, `at`, its outcome (`kept`,
  `diverged` or `missed`) and what played (an object, by track, group and object ID, or an
  event, by digest), or none;
- each participant's fidelity: announced, kept, diverged and missed, over every intent;
- each schedule, in the merged order: its digest, `mode`, `from`, and its turns, each with its
  segment, index, start, end, scheduled holder and role (`solo`, `call` or `response`); and each
  handover: its digest, `at`, `to`, and the turn it passes;
- each object, in the order given: whether the floor was open, or it was in turn or out of
  turn, the schedule's mode, whether it was late, by how many µs it arrived after its due
  instant (0 when not after), and the flags of the marks naming it;
- each window, in the merged order: its digest, rule, `close`, its eligible inputs, its
  commitment, its picks, and its credits; with, for plurality, the count of each option, and,
  for reactions, each eligible input's score.

## 10. Reasons

The codes this extension adds, each with where it is raised:

| reason | raised at |
|---|---|
| `co_policy_malformed` | §3.3 |
| `co_policy_unlinked` | §3.4 |
| `co_no_policy` | §3.4, §9.2 step 4 |
| `co_attr_malformed` | §4.4, §9.2 step 3 |
| `co_statement_malformed` | §4.2, §5.2, §5.6, §6.1, §7.2, §7.7 step 2 |
| `co_not_permitted` | §4.3 |
| `intent_not_ahead` | §5.3 |
| `intent_beyond_horizon` | §5.3 |
| `payload_missing` | §5.4, §9.2 step 5 |
| `fidelity_mismatch` | §5.6 |
| `schedule_retroactive` | §6.1 |
| `mode_not_allowed` | §6.1 |
| `handover_invalid` | §6.3 |
| `handover_not_holder` | §6.3 |
| `turn_violated` | §6.5, §9.2 step 6 |
| `turn_unmarked` | §6.5, §9.2 step 6 |
| `late_unflagged` | §6.6, §9.2 step 6 |
| `mark_unfounded` | §6.7, §9.2 step 7 |
| `input_not_permitted` | §7.3 step 1 |
| `window_unknown` | §7.3 step 2, §7.5, §7.7 step 1 |
| `input_type_mismatch` | §7.3 step 3 |
| `input_over_limit` | §7.3 step 4 |
| `statement_repeated` | §7.5, §7.7 step 1 |
| `commit_early` | §7.5 |
| `commit_mismatch` | §7.5 |
| `selection_uncommitted` | §7.7 step 1 |
| `seed_mismatch` | §7.7 step 2 |
| `input_late` | §7.7 step 3 |
| `selection_mismatch` | §7.7 step 4 |
| `input_uncredited` | §7.7 step 5 |
| `credit_mismatch` | §7.7 step 5 |

A co-performance verifier also raises every code a room verifier raises (live-rooms §11 and the
live-events codes it lists), at the room; `truncated`, `malformed` and `unknown_version` for a
policy that does not read; `wrong_room`, `bad_key` and `bad_signature` for a policy of another
room or whose signature fails; and `delivery_gap` for a gap, which is not a refusal.

## 11. Conformance

[`live-conformance/coperformance/`](../live-conformance/coperformance/) holds the corpus, in
one file, `vectors.json`, written from fixed seeds by `wai_co_performance_vectors gen`. It has
rooms, each a room (descriptor, grants, key-log head or none, chains), its policies, objects and
payloads, and the outcome §9.3 gives them with everything it states:

- **Accepted**: a jam room, in which a person and an AI participant trade four-bar phrases as
  call-and-response pairs at a stated tempo under a schedule that marks, and a later schedule
  that refuses; the AI participant announces intents of every form, kept, diverged and missed;
  an object out of turn and marked, a late object marked, a handover by the director, and the
  clock's fidelity statement; and an audience room, with a plurality vote, a window selected by
  reactions and a seeded draw, an input denied by a moderator and left out, an input sent after
  its window closed and left out, each window committed, selected and credited.
- **Gap**: the audience room with one audience member's record not delivered.
- **Refused**: each reason of §10 at least once, each vector a control of one defect: among
  them a fidelity statement counting as kept an intent that played something else, an object
  out of turn under refuse, an input sent after its window closed and selected, a selection that
  does not recompute from the committed inputs, a selected input with no credit, and a late
  object with no mark; and policies that do not read or are of another room.

A conforming verifier gives every room the outcome, the reason, the place, and everything else
the corpus states. `wai_co_performance_vectors verify` and the test `tests/co_performance.rs`
check that the file regenerates byte for byte, that `wai-rs` gives every room its outcome, that
every reason of §10 appears, and that §10 lists exactly the reference's codes.
`live-conformance/co_performance_verify.py` is the second implementation, written from this
text: it shares no code with `wai-rs`; it verifies each room with the live-rooms corpus's second
implementation (`live_rooms_verify.py`) and every signature with WAI's strict rule
(`claim-conformance/_ed25519.py`); and it mutates copies of the corpus and requires every copy
to fail.

## 12. Security and privacy considerations

- Every statement is signed by its author and rides in its author's chain, and every policy is
  signed by the room authority. A relay can withhold a statement but alter none, and a withheld
  statement is a gap in its author's chain.
- An intent binds its author to what it announced, in advance and under its key. A participant
  that announces one thing and plays another is not refused for it: it diverged, and the
  fidelity it is credited with says so. A fidelity statement that credits more than the room
  holds is refused.
- The objects, their arrivals and their due instants are the caller's statement, as a relay's
  log states them. A relay that misstates an arrival misstates the floor and the budget; its
  signed per-hop record ([live-metering](live-metering.md) §3) and the subscribers' own
  observations of arrival are the evidence against it. A clock that is a session of its own
  takes each arrival a hop after the relay does (§8): under a schedule that refuses, an object
  the relay forwarded within that hop of a turn's end is in the clock's log out of turn, which
  the relay's own log of arrivals, where one is kept, settles. A due instant is a function of the
  object's timestamp and the room clock, and a subscriber holding the object recomputes it.
- A performer's symbolic payload states in symbols what its object played. A party holding both
  can check one against the other; this extension compares the forms and does not transcribe
  media.
- The commitment is made before the rule runs and covers exactly the eligible inputs, each
  signed by its member before the window closed; the seed of a draw is committed before any
  input exists. A selection that does not recompute from them is refused, so an outcome cannot
  be chosen after the inputs are seen. What remains to the clock is to omit an input from the
  room, which leaves a gap in the member's chain (live-events §6.2), or to commit before an
  input in flight arrives, which the input's `t` below `close` shows.
- A relay holds an input that reaches it ahead of a gap in its author's chain for a bound of its
  own, and refuses it once the bound runs out (§7.3). An author that leaves a gap in its own
  chain holds up only its own inputs, and for no longer than that bound.
- A sketch or a movement sample is named by digest, and its object is fetched only by a party
  the room's grants let subscribe; a movement sample can identify the person who made it, and it
  is personal data. Taking part in a room under a grant is taking part in its mechanics, and this
  extension adds no consent step to them. Where a regime asks the operator for a lawful basis for
  processing what an audience submits, the operator records it as a `lawful-basis` claim
  ([jwp-receipts](jwp-receipts.md) §2.4.6).
- A vote is credited to its voters together, by a digest of their keys: a member proves it voted
  by replaying the room, and the credit lists no key in clear.
- An audience member's inputs are linkable to one another through its key, as every event is
  (live-events §11). A member that needs inputs unlinkable across rooms uses a key per room.