Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 219
open-standards/wai/extensions/live-recording.md

835 lines
53 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# WAI Extension: Live Recording
> Status: Draft. Registers the record `wai.live.recording`: a finished room session as one
> WAI package, and clips of it that prove what they are. The package is a directory of files
> and its listing, as a package of [packaging](packaging.md) §11 is; its components are the
> room (its descriptor, grants and key-log head, [live-rooms](live-rooms.md)), its events
> (each principal's chain whole, [live-events](live-events.md) §8.6), one component for
> each participant's take ([capture-binding](capture-binding.md)), and, optionally, the
> composite; and a signed **recording manifest** names each component by its digest, with
> the signer, the room and the session's span on the room clock. A recording verifier gives
> `complete` and `publishable` for each purpose their verdicts. A **clip** is a span of one
> take, or one window of several takes aligned on the room clock; its signed **clip record**
> carries, for each take, the descriptors of the chunks it covers and a range proof that
> they are leaves of a commitment the device made while it captured, so a verifier holding
> the clip, the take's commitments, the device's key and the room descriptor knows the
> clip's bytes are exactly those bytes of that take, unedited, without the take. The HLS and
> LL-CMAF presentation of a recording (§7) and the addressing of its chunks for a MoQ FETCH
> (§8.1) are mappings. A relay's DVR window of a room's tracks (§8.2 to §8.5) keeps them, with
> their receipts, for a viewer to seek back into by FETCH. No new transport and no new
> container. The reference implementation is
> the `live_record` module of `wai-rs` (feature `live_record`); the corpus is
> [`live-conformance/recordings/`](../live-conformance/recordings/), and its second
> implementation is `live-conformance/live_record_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision: RFC 6962 §2.1 and RFC 9162 §2.1.1, §2.1.3 and §2.1.4 (the
> Merkle tree hash, inclusion proofs and their verification); RFC 8216 §3.3, §4.3.2.2,
> §4.3.2.5, §4.3.2.6, §4.3.3.1, §4.3.3.5 and §7; `draft-pantos-hls-rfc8216bis-22` §3.1.2,
> §4.4.3.7, §4.4.3.8, §4.4.4.5 and §4.4.4.9; and `draft-ietf-moq-transport-22` (MOQT)
> §2.4.1, §3.2, §3.2.1, §8.2, §8.7, §9.11 and §9.20.9. The two drafts are Internet-Drafts,
> and a section number of either in this text is that revision's.
## 1. Scope
In scope: the recording package, its files and its components (§3); the recording manifest
and recording verification, with the verdicts of `complete` and `publishable` (§4); the clip
record, the range proof it carries, clip verification, and cutting a clip by chunks, by
bytes or by a window of the room clock (§5); a clip across takes, its time correspondence
and its bound (§6); the mapping of a recording onto HLS and LL-CMAF (§7); and the addressing
of a recording's chunks for a MoQ FETCH, with the DVR window a relay keeps of a room's tracks
and serves to FETCH (§8).
Out of scope: a transport; how a relay lays its DVR window out on disk; a muxer for a
master that is not CMAF; the composition itself, which `wai.live.compose` defines
(live-compose §8, a later extension); and what a party does with a verdict.
Relationship to the core spec: `wai.live.recording` is a **record** ([SPEC](../SPEC.md) §5):
a signed record a WAI tool emits and verifies. It names no payload format and is not
dispatchable.
## 2. Terms
- **Room**, **room clock**, **principal**, **event**, **chain**, **grant**, **room key**:
as in live-events §2 and live-rooms §2.
- **Take**, **device**, **master**, **chunk**, **descriptor**, **take tree**,
**commitment**, **offset**, **purpose**: as in capture-binding §2 and §4. *I* and *J* are
the room's commit interval and jitter in microseconds (capture-binding §2).
- **Session span**: the room-clock instants a recording covers and its signer vouches for,
`from` to `to`, both included.
- **Package**: the directory of files a recording is (§3).
- **Component**: a part of a package named by a kind, an `id` and a `sha256` (§3.2).
- **Head**: the commitment of a take a clip's proof is made against: one of the take's
`take_commit` events (capture-binding §5.1), its size and its root.
- **Part** (of a clip): the span of one take a clip covers (§5.1).
- **Cutter**: whoever makes a clip and signs its record.
Integers are little-endian. A digest is SHA-256 (FIPS 180-4), 32 bytes. Ed25519 is RFC
8032's, verified strictly (SPEC §7 *Signatures*).
## 3. The recording package
### 3.1 Files and the listing
A recording is a directory of these files and no other:
```
recording.wrm the recording manifest (§4)
room/descriptor.wrd the room descriptor
room/grant-<k>.wrg each room grant
room/key-log.json the head of the room authority's key log
events/<principal>.wev each principal's chain
takes/<take>/manifest.wtm a take's manifest
takes/<take>/chunk-<index>.bin each chunk of the take the package holds
takes/<take>/receipt-<k>.wur the store's upload receipts
takes/<take>/consent-<k>.wcn the consent records
composite/receipt-<digest>.wcr the composite's receipts, optionally (§3.5)
composite/… and the objects they name
LISTING
```
`LISTING` is the listing (SPEC §3.1) of every other file, and its SHA-256 is the **package
digest**, as a package of packaging §11 has its own. Every path is one a listing can carry.
`<principal>` and `<take>` are a key and a take id as 64 lowercase hexadecimal digits,
`<index>` a chunk's index as ten decimal digits, and `<k>` a position as four decimal
digits from `0000`.
### 3.2 Components
A package has these components, each the files below one directory:
| kind | component | `id` | directory |
|---|---|---|---|
| 1 | the room | `room` | `room/` |
| 2 | the events | `events` | `events/` |
| 3 | a take | the take id, 64 lowercase hexadecimal digits | `takes/<take>/` |
| 4 | the composite | `composite` | `composite/` |
A component's **`sha256`** is the SHA-256 of the listing (SPEC §3.1) of its files, each by
its path below the component's directory. That is SPEC §7.1's component identity for a
component of several files: an `id`, of SPEC §7.1's form, and the `sha256` of its bytes,
here serialised as the listing that SPEC §3.1 gives a set of several files, as a pin in the
form `wai.prior.set` covers one. No other per-component digest is defined.
### 3.3 The room and the events
The room component holds `descriptor.wrd`, the room descriptor (live-rooms §3); one
`grant-<k>.wrg` for each room grant (live-rooms §4), numbered from `0000` without a gap in
the order a room verifier is given them (live-rooms §9.1); and `key-log.json`, the head of
the room authority's key log that the room verifier is given (live-rooms §5), when the
descriptor names an origin. It holds no other file.
The events component holds one file for each principal whose chain the recording holds,
named by the principal's key: its chain whole, its records concatenated in its stream order
(live-events §8.6). A record's length follows from its own fields (live-events §3.1), so the
file splits into records with no framing; a file that does not split into records that read
(live-events §3.3), each of that principal, is not a chain. A room verifier is given the
chains in the order of their files' paths, ascending order of the principals' keys, and that
order is the chain index by which live-rooms §9.2's merged order breaks ties.
### 3.4 A take
A take component's directory is `takes/<take>/`, its `id` the take id (capture-binding
§3.2). It holds `manifest.wtm`, the take manifest (capture-binding §7); one
`chunk-<index>.bin` for each chunk of the master the package holds, the chunk's bytes; the
store's upload receipts in their order as `receipt-<k>.wur` (capture-binding §6.2); and the
consent records as `consent-<k>.wcn` (capture-binding §8.1). It holds no other file.
The chunk files are the take's **parts** (capture-binding §6.1), given to the take verifier
in the order of their indexes. A chunk the store never received has no file: the package
holds what was uploaded, and the take's `complete` verdict says whether that is every
committed chunk. The master is its chunks concatenated in index order (capture-binding
§4.1).
### 3.5 The composite
A recording **MAY** hold the room's composite: the objects of a composed track, each with
the composition receipt `WCR1` its compositor signed for it, which `wai.live.compose`
defines (live-compose §8). The component `composite` holds each receipt as
`receipt-<digest>.wcr`, `<digest>` the receipt's digest (SHA-256 over its bytes before its
signature, live-compose §8.1) in 64 lowercase hexadecimal digits, and the objects the
receipts name beside them. The recording manifest names the component by its `sha256` like
any other, so the listing binds each receipt's bytes and its name its digest. A recording verifier of this revision
checks that digest and reads nothing in the component: verifying a composition receipt is
live-compose §8.3's, a later extension, and this extension depends on nothing of it but the
name and digest of its record.
## 4. The recording manifest
### 4.1 Layout
When the session is over, the room authority, or a principal that held the record power over
the whole span (§4.3), signs a recording manifest, `WRM1`:
| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WRM1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id |
| `signer` | 32 | the key that signs the manifest |
| `from` | 8 | u64: the session span's first instant, room clock in microseconds |
| `to` | 8 | u64: its last instant |
| `claims` | 1 | the claims the manifest makes: bit 0 `complete` |
| `purposes` | 1 | the purposes it claims every take `publishable` for (capture-binding §2) |
| `region` | 2 | ISO 3166-1 alpha-2: where the package is kept |
| `components` | 2 + … | a u16 count, then each component: `kind` (1), the length of its `id` (1), the `id`, and its `sha256` (32) |
| `sig` | 64 | Ed25519 by `signer` |
The **digest** of a manifest is SHA-256 over its bytes before `sig`; its **signature** is
over the 19 bytes `wai:live-recording` (ASCII) and 0x01, followed by the bytes the digest
covers.
### 4.2 Reading and rules
A reader reads exactly the bytes it is given, as live-events §3.3 reads a record with
`WRM1` for `WEV1` (`truncated`, `malformed`, `unknown_version`), and refuses a manifest
that breaks any of these rules as `recording_manifest_invalid`:
- `claims` sets no bit above bit 0, and `purposes` none above bit 3; `from` is at most `to`;
`region` is two capitals `A`–`Z`;
- the first component is the room (kind 1, `id` `room`) and the second the events (kind 2,
`id` `events`); then come the takes, each of kind 3 with an `id` of 64 lowercase
hexadecimal digits, in strictly ascending order of `id`; then, last, at most one
composite (kind 4, `id` `composite`); and no other component.
It then refuses one whose signature does not verify as `bad_key` or `bad_signature`. Each
`id` the rules admit is of SPEC §7.1's form, and the order makes the manifest of one
package one sequence of bytes.
### 4.3 Who signs
The room authority (the descriptor's `authority`, live-rooms §3.1) may sign any span. Any
other signer **MUST** hold the **record** power (live-rooms §3.3) at every instant of the
span, from `from` to `to` inclusive.
A principal holds the power at room-clock instant *t* when, in the room state replayed to
*t* (live-rooms §9.3: every `join`, `grant_exchange` and `leave` of it that the room verifier
accepted with its `t` at or below *t*, in the merged order), it is present, and its grant in
effect has the record power, is not revoked (live-rooms §5), and covers *t*: `nbf` ≤
⌊(`t0` + *t*) / 10^6⌋ < `exp`. So the grant took effect, by a `join` or a `grant_exchange`,
at or before `from`, and nothing takes the power away before `to`: no `leave`, no
`grant_exchange` to a grant without it, and no expiry.
A signer that does not hold it throughout is refused `recording_signer_not_permitted`, and
the verifier states the **first instant** of the span at which it does not hold it: `from`
when it held no grant with the power there; otherwise the `t` of the `leave` or the
`grant_exchange` that took the power away, or the first instant its grant does not cover.
One code with the instant, not one code per cause: the power over the span is one property,
each way it lapses is already a record of the room (a lifecycle event, or a grant's window)
that the instant points at, and the instant is what a signer needs to know to seal a span it
did hold the power over. A span that fails several ways fails first where the instant says.
The record power is the power to start and stop a recording, and sealing the session is the
end of that act. Held only at the end, it would let a principal granted it just before the
end seal a session it never held the power to record. A recorder that gained the power
partway through signs a manifest whose span starts no earlier than its grant took effect;
the package still holds the room's chains whole (§3.3), and §4.4 step 6 holds every take it
holds to that span. A device needs no record power to commit to its take, and a device whose
grant has none cannot seal the room's recording.
A revocation in the key log is not bounded by a time (live-rooms §5): the room verifier
refuses every use of a revoked grant, so a head that revokes the signer's grant refuses the
recording at the room (§4.4 step 4), before this rule is reached.
### 4.4 Verification
A recording verifier is given a package's files by path, and an evaluation instant `at` in
seconds since 1970-01-01T00:00:00Z, which it gives each take verifier (capture-binding
§9.1). It runs these steps and stops at the first refusal, which it reports with the reason
and where:
1. **The listing.** `LISTING` is present and is the listing of every other file:
otherwise `listing_mismatch`, at `listing`.
2. **The manifest.** `recording.wrm` is present, otherwise `recording_manifest_invalid`,
and it reads (§4.2); at `manifest`.
3. **The components.** For each component of the manifest, in its order: its directory
holds a file, otherwise `component_invalid`, and the SHA-256 of its listing is its
`sha256`, otherwise `component_digest_mismatch`; at that component. Then every file is
`LISTING`, `recording.wrm`, or a file below a component's directory: otherwise
`component_unlisted`, at `listing`.
4. **The room.** The room component holds the files of §3.3 and no other, and each file of
the events component is a chain of the principal it names (§3.3): otherwise
`component_invalid`, at that component. The room verifier (live-rooms §9) is given the
descriptor, the grants, the key-log head (or none) and the chains: a refusal is the
recording's, at `room` with the room verifier's own place. A gap is not a refusal: the
steps go on, and §4.5 says what a gap fails. The manifest's `room` is the room id:
otherwise `wrong_room`, at `manifest`.
5. **The recording policy**: the room's recording policy (live-rooms §3.7) has a `mode` other
than 0, otherwise `recording_not_permitted`, capture-binding §12's code, at `manifest`: a room
that is never recorded has no recording, takes or no takes. **The signer** (§4.3): otherwise
`recording_signer_not_permitted`, at `signer` with the
first instant of the span at which it does not hold the record power. **The region**:
where the room's recording policy pins regions (live-rooms §3.7), `region` is one of
them, otherwise `store_outside_region`, capture-binding §6.3's code, at `manifest`.
6. **The takes.** For each take component, in the manifest's order: it holds the files of
§3.4 and no other, otherwise `component_invalid`; the take verifier (capture-binding §9)
is given the room as in step 4, the take manifest, the parts, the receipts, the
consents and `at`, and its refusal is the recording's, at `take` with the component's
index and the take verifier's own place; the take id it reads is the component's `id`,
otherwise `component_invalid`; and the take lies within the span, its start (the
`record_start` it answers) at or after `from` and its last held commitment at or before
`to`, otherwise `outside_span`; at that component. The records before `from` or after
`to` that the package holds are the room's chains whole, which the room verifier replays
the room from; the manifest states nothing of them.
7. **The verdicts** (§4.5).
### 4.5 Verdicts and outcome
**`complete`** holds when the recording holds every take of the session that starts within
the span, each whole, and every event the room verifier was offered: the first reason below
that applies.
1. `delivery_gap`: the room verifier holds a gap (live-rooms §9.4).
2. `take_omitted`: a take of the session whose `record_start` lies within the span has no
component. A **take of the session** is the take id (capture-binding §3.2) that a
`record_start` the room verifier accepted (live-events §4.9) names for its target, where
the room holds a `take_commit` of that target whose payload reads (capture-binding §5.2)
and names that take: a start its device answered.
3. For each take component, in the manifest's order: its `single-take` verdict fails with
`delivery_gap`, `commit_chain_broken` or `take_not_closed`, so its commitments are not
one unbroken chain from the opening commitment to the closing one: that reason; or its
`complete` verdict fails: that reason (capture-binding §9.4).
**`publishable`** for purpose *p* holds when every take is `publishable` for *p*: for the
first take component, in the manifest's order, whose `publishable` for *p* fails, that
reason.
A recording verification gives one outcome:
- **refuse**: the reason and where (§4.4);
- **verdicts**: the package digest; the manifest's digest; the room id; the verdicts of
`complete` and of `publishable` for each of the four purposes; each take's id and verdicts
(capture-binding §9.5); and whether the manifest **stands**: every claim it makes, and
`publishable` for every purpose it names, holds.
## 5. Clips
### 5.1 The clip record
A clip is a span of one take, or one window of several takes aligned on the room clock
(§6). A cutter signs a clip record, `WCL1`:
| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WCL1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `room` | 32 | the room id |
| `window_in` | 8 | u64: the first room-clock instant the clip presents, in microseconds |
| `window_out` | 8 | u64: the instant it presents up to (excluded) |
| `parts` | 1 + … | a u8 count *k*, then each part as below |
| `signer` | 32 | the cutter's key |
| `sig` | 64 | Ed25519 by `signer` |
Each part:
| field | size | meaning |
|---|---|---|
| `device` | 32 | the device's key |
| `take` | 32 | the take id |
| `head` | 32 | the digest of the `take_commit` event whose commitment the proof is against |
| `size` | 8 | u64: that commitment's size |
| `root` | 32 | that commitment's root |
| `first` | 8 | u64: the index of the first chunk the part covers |
| `count` | 4 | u32: the number of chunks it covers |
| `trim_in` | 8 | u64: the bytes of the first chunk before the clip's bytes |
| `trim_out` | 8 | u64: the bytes of the last chunk after them |
| `chunks` | 64 × `count` | the descriptor of each chunk it covers (capture-binding §4.1), in index order |
| `proof_len` | 1 | the number of nodes in `proof` |
| `proof` | 32 × `proof_len` | the range proof (§5.3) |
The **digest** of a clip record is SHA-256 over its bytes before `sig`; its **signature**
is over the 14 bytes `wai:live-clip` (ASCII) and 0x01, followed by the bytes the digest
covers. A part's bytes are its chunks' bytes concatenated, less `trim_in` bytes at the
start and `trim_out` at the end; the clip's bytes are its parts'.
### 5.2 Reading and rules
A reader reads exactly the bytes it is given, as live-events §3.3 reads a record with
`WCL1` for `WEV1` (`truncated`, `malformed`, `unknown_version`), and refuses a record that
breaks any of these rules as `clip_record_invalid`:
- *k* is 1 to 16, and no take is named by two parts;
- in each part, `count` is at least 1; each chunk's `index` is `first` plus its position,
and each `len` is at least 1; `proof_len` is at most 128; `trim_in` is below the first
chunk's `len`, and `trim_out` below the last chunk's; and, with one chunk, `trim_in` +
`trim_out` is below its `len`;
- a part with a trim that is not 0 is the clip's only part;
- `window_in` is below `window_out`.
It then refuses one whose signature does not verify as `bad_key` or `bad_signature`.
**Partial chunks.** A clip whose range begins or ends inside a chunk carries that whole
chunk and states the bytes it leaves out, as `trim_in` and `trim_out`. A leaf commits to a
chunk's digest over all its bytes (capture-binding §4.2), so a part of a chunk can never be
proved on its own: with the whole chunk the proof stays exact, and the trims say which of
the proved bytes the clip presents. Refusing a range that does not fall on chunk boundaries
would make a byte range of the master, or a range request, unprovable unless it happened to
land on them. A trim is in bytes, which have no committed time within a chunk, so a trimmed
clip's window is its chunks' span (§5.4 step 3j), and trims are a single take's: an aligned
clip (§6) presents a window of time, not of bytes. A cutter that must not disclose the bytes
outside the range cuts on chunk boundaries, since the trimmed bytes travel with the clip.
### 5.3 The range proof
A part proves its chunks by one proof over the run of leaves they are, against the head.
With `D[n]` the take tree's leaves (capture-binding §4.2) under the head's size *n*,
`MTH` RFC 9162 §2.1.1's Merkle tree hash with the take tree's node, and *k* the largest
power of two below *n*, the proof of leaves *a* to *b* − 1 (0 ≤ *a* < *b* ≤ *n*) is:
```
RANGE(a, b, D[n]) = {} a = 0 and b = n
= RANGE(a, b, D[0:k]) : MTH(D[k:n]) b ≤ k
= RANGE(a − k, b − k, D[k:n]) : MTH(D[0:k]) a ≥ k
= RANGE(a, k, D[0:k]) : RANGE(0, b − k, D[k:n]) otherwise
```
where `:` appends. A run of one leaf is never split, so `RANGE(i, i + 1, D[n])` is RFC 6962
§2.1.1's `PATH(i, D[n])`, the inclusion proof RFC 9162 §2.1.3.1 generates, node for node.
A proof of a whole tree is empty. A proof has at most 2⌈log₂ *n*⌉ nodes, at most 128 for any
size a commitment can state, and every node of it is a node of the inclusion proof of a leaf
in the run.
A verifier given the leaves *L* of the run (*b* − *a* of them), the size *n* and the
nodes, used in order from the first, computes `ROOT(a, b, n, L)`:
```
ROOT(a, b, n, L) = MTH(L) a = 0 and b = n
= node(ROOT(a, b, k, L), next) b ≤ k
= node(next, ROOT(a − k, b − k, n − k, L)) a ≥ k
= node(ROOT(a, k, k, L[0 : k − a]),
ROOT(0, b − k, n − k, L[k − a :])) otherwise
```
where `next` is the next unused node, and `node` is the take tree's (staged-measurement
Appendix C). The proof verifies when no `next` is asked for past the last node, every node
is used, and `ROOT` is the head's root. For a run of one leaf this is RFC 9162 §2.1.3.2's
check of an inclusion proof, and gives its answer.
The size is the head's, which the device signed, never the cutter's: as for an inclusion
proof, the same nodes can reach one root under two sizes (staged-measurement Appendix C),
so a size the cutter chose would let it choose the leaves' positions. Under the head's size
the recursion fixes which subtree each node stands for; the leaf and node domains differ, so
no node passes for a leaf; and each leaf binds its chunk's index, capture range, length and
digest (capture-binding §4.2). A proof that verifies therefore shows, short of a collision of
SHA-256, that the run's descriptors are the head's leaves *a* to *b* − 1.
### 5.4 Verification
A clip verifier is given the room descriptor; the clip record; and, for each part in order,
the device's key, the take's commitments (its `take_commit` events, from its opening one to
the head, in the order of the take's chain), and the bytes of each chunk the part covers,
each present or absent. It is not given the take manifest, any other chunk, the upload, or
the room's other events. It runs these steps and stops at the first refusal, which it
reports with the reason and where:
1. **The descriptor** reads (live-rooms §3.4): its refusal, at `descriptor`. The room id is
its digest, and *I* and *J* are its recording policy's (live-rooms §3.7).
2. **The record** reads (§5.2), at `clip`. Its `room` is the room id, otherwise
`wrong_room`; and the verifier is given one input for each part, otherwise
`clip_record_invalid`; at `clip`.
3. **Each part**, in order, steps (a) to (k):
- **(a)** The key given is the part's `device`: otherwise `clip_device_mismatch`, at the
part.
- **(b)** Each commitment given reads (live-events §3.3), names the room
(`wrong_room`), is the device's (`clip_device_mismatch`), verifies (`bad_key`,
`bad_signature`), is a `take_commit` whose payload reads (capture-binding §5.2,
`commit_malformed`), and names the part's `take` (`clip_take_mismatch`); at that
commitment.
- **(c)** The commitments are a chain (capture-binding §5.3): the first is an opening one,
and each later one has the `prev`, `first` and `n` that follow its predecessor, which
is not a closing one, and is itself not an opening one. A later one whose `n` is more
than one above its predecessor's is `delivery_gap`; any other break is
`commit_chain_broken`. Then each extends its predecessor (capture-binding §4.3),
otherwise `root_not_extending`. At the first commitment that fails, the chain checked
over all of them before any extension, as capture-binding §9.4 orders them. None given
is `commit_chain_broken` at the part.
- **(d)** The last commitment given is the head: its digest is `head`, and its size and
root are the part's `size` and `root`. Otherwise `tree_head_not_committed`, at the
part: a tree the take never committed proves nothing of it.
- **(e)** `first` + `count` is at most `size`: otherwise `clip_out_of_range`, at the part.
- **(f)** The proof (§5.3) shows the part's descriptors are the head's leaves `first`
onward, under `size`: otherwise `clip_not_included`, at the part.
- **(g)** Every commitment's offset is within *J* of the opening one's, *o* (capture-binding
§9.4 `single-take` 6): otherwise `clock_unstable`, at that commitment.
- **(h)** The chunks tile their capture, each `start` below its `end` and each `start` the
previous chunk's `end` (`capture_discontinuous`), and none spans more than *I*
(`chunk_too_long`) (`single-take` 7, 8); at the part.
- **(i)** With each chunk mapped by *o* (capture-binding §9.3) and *t* the room-clock
instant of the commitment that adds it (the one whose `first` ≤ `index` < `size`): no
chunk has *t* + *J* below its mapped end (`commit_early`), and then none has *t* above
its mapped end plus *I* plus *J* (`commit_late`) (`single-take` 9, 10); at the part.
- **(j)** The window: with the part's **span** the first chunk's mapped `start` to the last
one's mapped `end`, the span covers the window, the first chunk ends after
`window_in` and the last begins before `window_out`; and a trimmed part's window is
its span. Otherwise `clip_window_invalid`, at the part.
- **(k)** The bytes: for each chunk in order, its bytes are given (`clip_chunk_missing`)
and are that chunk's, their digest and length its descriptor's; bytes that are
another covered chunk's are `chunk_reordered`, any others `chunk_altered`; at that
chunk. Bytes given past the last chunk are `chunk_extra`, at the chunk past it.
Steps (b) to (k) are `single-take` over the span: the rules of capture-binding §9.4
`single-take` that hold of the chunks the clip covers, checked from the commitments up to the
head. A clip does not need rule 4 (closed), since it may be cut while the take runs (§8);
nor rule 5 (the manifest), since inclusion binds its chunks to the head instead; nor rule
9's clauses on the start and the stop, whose events it is not given. A take whose
`single-take` fails only at chunks outside a span, as by a late commitment, still gives
clips of that span that verify.
A clip that passes is **accepted**, and the verifier states: the record's digest and
signer; the window; the bound (§6.2); and for each part its take, `first`, `count`, the
take's offset *o*, its span, the window on its device's clock (§6.1), the length and SHA-256
of the bytes it presents, and its FETCH address (§8). Where the take's `single-take` holds, an
accepted clip's descriptors are its manifest's at those indexes, since rule 5 binds the
manifest's chunks to every commitment's root; the clip verifier does not need the manifest.
A clip over chunks a moderation decision restricts is verified as
[live-moderation](live-moderation.md) §8.3 has it: step (k) passes over the restricted
chunks, whose descriptors the proof still covers.
### 5.5 Cutting a clip
A cutter holds a take's manifest, its commitments and the chunks it covers. It picks a head:
the take's closing commitment, or, while the take runs, any commitment whose size covers the
range. It cuts:
- **by chunks**: `first` and `count`, with no trim;
- **by bytes**: a byte range [*x*, *y*) of the master, with chunk *i* at the master's bytes
from the sum of the `len` of the chunks before it: the chunks from the one holding byte
*x* to the one holding byte *y* − 1, `trim_in` *x* less the first one's offset, and
`trim_out` the last one's end less *y*;
- **by a window** [*w₀*, *w₁*) of the room clock: in each take, the chunks from the one whose
mapped capture holds *w₀* to the one whose mapped capture holds *w₁* − 1, with no trim.
A clip cut by chunks or bytes presents its chunks' span as its window. The proof is §5.3's
over the manifest's descriptors up to the head's size; the cutter signs the record.
## 6. Across takes
### 6.1 Time correspondence
An aligned clip has a part for each take it spans, over one window of the room clock. Each
take is placed on the room clock by its own mapping, the offset *o* of its opening
commitment (capture-binding §9.3): a room-clock instant *t* in the window is the device-clock
instant *t* − *o_p* in part *p*, in the chunk whose capture range holds it, at its position
in that chunk and in the master (capture-binding §10). Two parts *p* and *q* correspond at
*t*: device instant *d* in *p* is *d* + *o_p* − *o_q* in *q*. The verifier states each part's
window on its device's clock, [`window_in` − *o_p*, `window_out` − *o_p*).
### 6.2 The bound
A part's mapping is held, over every commitment up to its head, to within *J* of the offset
the device measured at that commitment (step 3g), and each chunk it covers was committed
within [−*J*, *I* + *J*] of its mapped end on the room clock (step 3i). The clip states its
**bound**, `bound_us`: *J* for a clip of one take, its mapping's to the room clock; and
2*J* for an aligned clip, between any two of its parts, each strays by *J* from the room
clock as its device measured it. A sink that presents the parts of an aligned clip together
aligns them within that bound of what the devices measured. The measurement itself, and each
commitment's `t`, are their authors' statements (live-events §11), held by the room's other
records as capture-binding §14 says.
## 7. HLS and LL-CMAF
This section is a mapping: what a standard HLS client plays of a recording, with no new
format. Where a take's master is CMAF (ISO/IEC 23000-19), cut as capture-binding §4.1 cuts
it, chunk 0 is the CMAF header and the first CMAF segment, and each later chunk one CMAF
segment, so the take's chunk files are already an HLS presentation of fragmented MPEG-4
segments (RFC 8216 §3.3; `draft-pantos-hls-rfc8216bis-22` §3.1.2):
- **One Media Playlist per take.** `EXT-X-VERSION` is 6 or higher, as `EXT-X-MAP` without
`EXT-X-I-FRAMES-ONLY` requires (RFC 8216 §4.3.2.5, §7; the draft's §4.4.4.5).
`EXT-X-TARGETDURATION` is the longest chunk's `end` − `start`, in seconds rounded up
(RFC 8216 §4.3.3.1), never above *I* by `single-take` rule 8. `EXT-X-PLAYLIST-TYPE` is
`VOD` for a closed take and `EVENT` while it runs, segments added only at the end (RFC
8216 §4.3.3.5), with `EXT-X-ENDLIST` once the take is closed.
- **The initialization section** is `EXT-X-MAP` with `URI` chunk 0's file and `BYTERANGE`
`<h>@0` (the draft's §4.4.4.5), *h* the length of chunk 0's top-level boxes (ISO/IEC
14496-12) before its first `styp` or `moof`. Chunk 0's segment is then
`EXT-X-BYTERANGE:<len − h>@<h>` of the same file (RFC 8216 §4.3.2.2), and each later
chunk its own file whole.
- **Durations and dates.** Each segment's `EXTINF` is its chunk's `end` − `start` in
seconds. The first segment carries `EXT-X-PROGRAM-DATE-TIME` (RFC 8216 §4.3.2.6), the
instant `t0` + `start` + *o* of the room clock in UTC; a gap-free take needs no other.
- **Events.** The room's events ride the playlist as live-events §8.3 maps them, as
`EXT-X-DATERANGE`, so a marker lands at its instant in the master (capture-binding §10).
- **Several takes.** Each take is its own Media Playlist. An HLS client presents one
rendition of a medium at a time; one that shows two takes together aligns them by
`EXT-X-PROGRAM-DATE-TIME`, within §6.2's bound.
- **LL-CMAF.** A CMAF segment holds CMAF chunks, each a `moof` and its `mdat`. A
low-latency playlist of a running take lists them as `EXT-X-PART` with `BYTERANGE` into
the chunk's file (the draft's §4.4.4.9), with `EXT-X-PART-INF` and
`EXT-X-SERVER-CONTROL` (§4.4.3.7, §4.4.3.8). The take tree commits to whole chunks, so a
part is proved only as part of its chunk, once the commitment that adds the chunk is
held, within *I* + *J* of the chunk's end (`single-take` 10): a sink that verifies
presents parts ahead of that and holds its verdict until the chunk is committed.
A master that is not CMAF is played this way only through a muxer for its codec, which this
extension does not define. What this revision implements of this section: nothing. The
corpus's masters are not CMAF, the reference has no muxer for a device's master (packaging
§7's CMAF writer writes a floor's AV1 fragments), and it writes no playlist.
## 8. DVR over MoQ FETCH
This section fixes how a recording's chunks are addressed for a FETCH (§8.1), and what a
relay that keeps a window of a room's tracks, a **DVR window**, does with it: what it keeps
and evicts (§8.2), how it answers a FETCH from it (§8.3), what moderation withholds from it
(§8.4), and how it keeps it across a restart (§8.5); and the window's presentation over HTTP
(§8.6).
### 8.1 Addressing a take
- **The track.** A recorded take is one track. Its Track Namespace (MOQT §2.4.1) has three
fields: the room id and the device's key, each as 64 lowercase hexadecimal digits, and
`take`; its Track Name is the take id as 64 lowercase hexadecimal digits. Every byte is one
of `a`–`z` and `0`–`9`, as in live-rooms §6.1's names, and its fields' 196 bytes are within MOQT §8.7's limit.
- **Groups and objects.** Group *g* is chunk *g*, and its one object, Object ID 0, is the
chunk's bytes whole: the unit a leaf commits to. A group's Location (MOQT §8.2) is
(*g*, 0).
- **A clip's part** is fetched by a FETCH (MOQT §3.2, §9.11) of that track whose
`LOCATION_FILTER` (MOQT §9.20.9) has StartGroup `first`, StartObject 0, EndGroupDelta
`count` − 1 and no EndObject, so every object of the end group: the range is inclusive
(MOQT §3.2), and the groups arrive in the requested order (MOQT §3.2.1). The clip
verifier states this address for each part (§5.4).
- **The commitments** are events of the device's events track (live-rooms §6.1), whose
group is the minute of room clock an event falls in (live-events §8.1). A FETCH of its
groups from the opening commitment's to the head's holds them, among the device's other
events; the clip verifier is given the take's `take_commit` events alone.
- **A running take.** The groups below the latest commitment's size are committed; a group
at or past it is a chunk no commitment has added yet, which a sink holds unverified until
one does (§5.5: a clip of a running take is proved against the commitment that covers it).
A take's track kept in a DVR window is kept and answered as any other track (§8.2, §8.3), so
a clip's part is a FETCH the window answers.
### 8.2 The window
A relay **MAY** keep a window of the tracks of a room: its media tracks with their receipts
tracks, its events tracks, its catalogs and its takes.
- **What it keeps.** A group, once the relay holds it whole: its first object arrived (or
a Prior Object ID Gap says the objects before do not exist, MOQT §10.9), none of its
subgroup streams is open or was reset, and its end is known. The objects are kept with
every field a fetch stream carries (MOQT §11.4.1).
- **Its bounds.** An age, counted from when a group's first object reached the relay, of at
most 86 400 seconds: 24 hours, the start-over buffer a live-to-VOD harvest clips from;
and a byte budget over the window's tracks together. Past either, the oldest group goes
first, whole. Group *g* of a media track and group *g* of its receipts track
([moq-streaming-format](moq-streaming-format.md) §4.1) go together, so that every group
the window serves comes with its receipts and a viewer checks a recorded object exactly as
it checks a live one ([jwp-receipts](jwp-receipts.md) §4.1). A group that becomes whole
after the window has moved past its group number is not kept.
- **Where.** The window is media and events the relay keeps beyond delivery, so the room's
region pin (live-rooms §3.7) binds where it is stored.
- **Who reads it.** A FETCH or fill below a track's live edge is served only to a grant
with the seek power (live-rooms §10.6).
- **A room it never keeps.** A room with a confidentiality record is never recorded, and a
relay keeps nothing of its tracks beyond delivery ([confidentiality](confidentiality.md)
§9). A relay knows such a room by its first sealed object, one whose Immutable Properties
carry `SECURE_OBJECT_KEY_ID` (confidentiality §4.3), which it reads without a key. It
**MUST NOT** keep that object's group. From then on it keeps nothing of the room's tracks,
deletes what it kept of them before (the room's events and receipts tracks are in the clear,
and can be whole before its first sealed group), and remembers the room across a restart.
It still honours the moderation decisions it reads on the room's events tracks for what it
delivers, and does not keep them.
- **A minors' room.** A room in which a minor publishes is never kept either
([age-assurance](age-assurance.md) §8.4): from the moment a session proves a minor's
publishing grant of the room, the relay keeps nothing of its tracks, deletes what it kept, and
remembers the room across a restart, as for a room whose objects are sealed.
### 8.3 Answering a FETCH
- **Where from.** A FETCH whose range the relay's cache holds is answered from the cache.
Otherwise, when every group from its start to its end is held whole by the cache or the
window, is known not to exist (MOQT §10.8), or was evicted from the window, the relay
answers it from them; otherwise it asks upstream (MOQT §3.2.3). Groups come in the
requested order (MOQT §3.2.1), objects within a group in Object ID order. A FETCH of the
receipts track over the same range brings each group's receipts the same way.
- **What was evicted** is no longer known to the relay. The relay marks the evicted groups
of the range with an End of Unknown Range (MOQT §11.4.1.2, Serialization Flags `0x10C`)
through the last of them, then sends the groups it holds: a range partly evicted gets what
remains and the marker, and a range wholly evicted gets FETCH_OK and the marker alone. It
uses neither an unmarked gap nor an End of Non-Existent Range: an evicted object existed,
an unmarked gap would say it did not (MOQT §3.2.2), and MOQT §10.3 makes an object
that has left a cache an object of unknown state (§2.1 there).
- **Inside a group.** An End of Range moves to a later group than the entry before it: its
Group ID is coded as an object's Group ID Delta (MOQT §11.4.1.1, §11.4.1.2), the reading
the `joule-moq` codec and the MOQT -22 vectors of `joule-moq` take. So when objects of a
group cannot be served (withheld, §8.4; dropped, §8.5), one End of Unknown Range from the
group's start through the last of them goes first, and the group's objects after it
follow. The objects before the last such one are not sent.
- **Across relays.** A window one relay keeps is fetched through another relay as any FETCH
that relay cannot answer alone is (MOQT §3.2.3): forwarded once, on the hop link between them
([live-rooms](live-rooms.md) §10.7), and its fetch stream piped through, groups in the order
asked. The relay the viewer's session reaches holds the request to live-rooms §10.6 against
the live edge it learns from the relay it asks, and the relay that keeps the window serves
the hop link as the link's token allows; each answers the same range with the same objects
and the same End of Unknown Range, so a viewer on either relay is given the same window.
- **A window whose end lies ahead.** A live-to-VOD harvest clips a window of at most 24 hours
whose end may lie in the future. Its past is a FETCH. The whole is one SUBSCRIBE whose
`LOCATION_FILTER` is the window, as an AbsoluteRange (MOQT §9.20.9), with
`FILL_PARAMETERS` (MOQT §3.4): the fill is answered from the cache and the window as a
FETCH is, and the subscription delivers the rest up to the window's end.
### 8.4 Moderation
What [live-moderation](live-moderation.md) withholds is withheld from the window, as §6.3
there has a relay withhold it in answer to a FETCH:
- **How the relay learns of decisions.** It reads the `decision` events (live-events §4.11)
on the room's events tracks, which it already forwards. It honours a decision whose record
verifies under its principal (live-events §3.2), whose principal owns the events track it
rides (live-rooms §6.1), and whose principal is a moderator it knows: a room key, or the
principal of a grant with the moderate power (live-rooms §3.3) that its publishing session
proved (live-rooms §10.5). The last decision it honoured on a target stands
(live-moderation §4.5), and the honoured decisions are kept with the window.
- **Deny and redact** withhold the target from every answer to every session but a
moderator's; **redact** also erases it from the window. **Allow** restores what deny
withheld. A withheld media object is the one whose receipt names it by its digest
(live-moderation §4.1); its receipts stay where they are (live-moderation §8.1). A withheld
event is left out of the events-track object that carries it, the object's other records
sent byte for byte; an object left with none is withheld whole. Either is withheld, never
replaced: each is an End of Unknown Range in its place (§8.3), and the decision that
restricts it says why.
### 8.5 Across a restart
A window kept across a restart is checked again before it is served. Every object is
checked against its receipt: a media object of a track with a receipts track against group
*g*'s receipts, as jwp-receipts §4.1 checks one at the door (the group receipt's signature
under the trust the relay holds for the namespace, the object's receipt at its position,
its Merkle proof to the signed root, its content hash); an events-track object's records each
under its principal (live-events §3.2). An object that does not verify, or whose stored
bytes do not hold, is dropped and counted, and never served: a FETCH that reaches it is told
End of Unknown Range (§8.3). How a relay lays the window out on disk is its own; the
reference relay's layout and its durability rule are in its README.
### 8.6 Over HTTP
The window is a time-shift buffer. In DASH (ISO/IEC 23009-1) its age is the MPD's
`timeShiftBufferDepth`. In HLS a live Media Playlist keeps the window's segments and
removes them from its start as the window evicts their groups; `EXT-X-PLAYLIST-TYPE`
`EVENT` (RFC 8216 §4.3.3.5), whose segments are added only at the end, fits only a window
that holds the session from its start and evicts nothing.
What this revision implements of §8.2 to §8.5: the reference relay, `joule-moq-relay`
(`joule-code/joule-code-rs/crates/joule-moq-relay`), keeps a window per namespace prefix
and its tests show each rule above. §8.6 is a mapping, and nothing writes it.
## 9. Reasons
The codes this extension adds, each with where it is raised:
| reason | raised at |
|---|---|
| `listing_mismatch` | §4.4 step 1 |
| `recording_manifest_invalid` | §4.2, §4.4 step 2 |
| `component_invalid` | §4.4 steps 3, 4 and 6 |
| `component_digest_mismatch` | §4.4 step 3; SPEC §7.1's code, for a component of a recording |
| `component_unlisted` | §4.4 step 3 |
| `recording_signer_not_permitted` | §4.3, §4.4 step 5, with the first instant of the span without the power |
| `outside_span` | §4.4 step 6 |
| `take_omitted` | §4.5 `complete` 2 |
| `clip_record_invalid` | §5.2, §5.4 step 2 |
| `clip_device_mismatch` | §5.4 steps 3a, 3b |
| `clip_take_mismatch` | §5.4 step 3b |
| `tree_head_not_committed` | §5.4 step 3d |
| `clip_out_of_range` | §5.4 step 3e |
| `clip_not_included` | §5.4 step 3f |
| `clip_window_invalid` | §5.4 step 3j |
| `clip_chunk_missing` | §5.4 step 3k |
`take_omitted` is a verdict; the rest are refusals. A recording verifier also raises every
code a room verifier or a take verifier raises (live-rooms §11, capture-binding §12, and the
live-events codes they list), at the room or the take; `truncated`, `malformed`,
`unknown_version`, `bad_key` and `bad_signature` for a manifest that does not read or whose
signature fails; `wrong_room` for a manifest of another room; `recording_not_permitted` for a
package of a room never recorded; `store_outside_region` for a package kept outside the room's
region pin; and `delivery_gap` and every
verdict of capture-binding §12 as a verdict. A clip verifier also raises the codes of a
descriptor that does not read (live-rooms §11); `truncated`, `malformed`, `unknown_version`,
`bad_key`, `bad_signature` and `wrong_room` for its record and its commitments; and
`commit_malformed`, `delivery_gap`, `commit_chain_broken`, `root_not_extending`,
`clock_unstable`, `capture_discontinuous`, `chunk_too_long`, `commit_early`, `commit_late`,
`chunk_reordered`, `chunk_altered` and `chunk_extra` at §5.4's steps, each with the meaning
capture-binding §12 gives it.
## 10. Conformance
[`live-conformance/recordings/`](../live-conformance/recordings/) holds the corpus, in one
file, `vectors.json`, written from fixed seeds by `wai_live_record_vectors gen`. Its
recordings are packages, each with its files and an evaluation instant and the outcome §4.5
gives them; its clips are each a room descriptor, a clip record and, per part, the device's
key, the commitments and the chunks' bytes, with the outcome §5.4 gives them.
- **The session.** A lesson room with two takes: the performer's, which is capture
binding's `lesson` take byte for byte (its manifest, chunks, receipts, consent and
commitments), and the accompanist's, captured beside it in chunks of another length on a
device whose clock runs ahead of the room's. The recording sealed by the room authority,
and by the director, stands, and so does one with a composite component, whose file
stands for a composed object and whose digest alone the verifier checks.
- **Recordings that fail a claim:** a take left out (`take_omitted`); capture binding's
`missing_chunk` take, whose chunk 6 was never uploaded (`chunk_missing`); a commitment
the relay did not deliver (`delivery_gap`); a take without its consent
(`consent_missing`).
- **The record power over the span:** a director that joins at 1 s seals the span from 1 s
and stands; sealing from 0 is refused at 0; and a director removed at 18 s, exchanged at
17 s to a grant without the power, or whose grant expires at 18 s, is refused at that
instant; one whose grant the key log revokes is refused at the room.
- **Recordings refused:** a file changed after sealing (`listing_mismatch`) and with the
listing rewritten (`component_digest_mismatch`), a take's or the composite's; a file of no component; a component
with a file §3 does not define; a manifest that breaks §4.2, of another room, or with a
bad signature; a manifest signed by a principal without the record power; a package kept
outside the room's region pin; a package of a room whose recording policy has `mode` 0; a span that
ends before a take's closing commitment; a take, and a grant, whose signature fails.
- **Good clips:** a single chunk, a byte range with trims, a whole take, the first chunk,
the last chunk, a span of the second take, a span proved against a commitment made while
the take was running, a two-take clip aligned over one window, and a span of capture
binding's `late_commit` take that its late commitment does not touch.
- **Clips refused,** one for each reason of §9 a clip raises and for each `single-take`
rule of §5.4: a proof that does not reach the head, a range past the head's size, a
span over the chunk missing from capture binding's `missing_chunk` take, a clip spliced
from two takes, a reordered, altered or extra chunk, a tree head the take never
committed, a commitment left out or malformed, another take's or another device's
commitments, a record of another room, with a bad signature, cut short, or with a trim
in an aligned clip, a window its chunks do not cover, and spans of capture binding's
`late_commit`, `commit_early`, `root_not_extending`, `clock_unstable`,
`capture_discontinuous`, `chunk_too_long` and `commit_chain_broken` takes.
A conforming verifier gives every vector the outcome, the reason, the place and the data
the corpus states. `wai_live_record_vectors verify` and the test `tests/live_record.rs`
check that the file regenerates byte for byte, that `wai-rs` gives every vector its
outcome, that every reason of §9 appears in the corpus, that §9 lists exactly the
reference's codes, and that every take the corpus says it reuses is capture binding's,
byte for byte, against `live-conformance/takes/vectors.json`.
`live-conformance/live_record_verify.py` is the second implementation, written from this
text: it shares no code with `wai-rs`; it checks range proofs by §5.3's recursion and, for
a run of one chunk, by RFC 9162 §2.1.3.2's algorithm as that RFC states it; it verifies
signatures with WAI's strict rule (`claim-conformance/_ed25519.py`), each take with the
capture-binding corpus's second implementation (`capture_take_verify.py`) and the room with
the live-rooms corpus's (`live_rooms_verify.py`); it gives every vector the same outcome;
and it mutates copies of the corpus and requires every copy to fail.
## 11. Security and privacy considerations
- The recording manifest is the signer's statement of which components the session has
and of its span; it adds nothing to a take's verdicts, which the take verifier gives from
the room's events. Its signer held the record power over the whole span, so a
participant who can only publish or watch cannot seal a recording of the room, and one
granted the power late cannot seal what came before it. The package digest names the whole;
the manifest names each component, so a component can be fetched and checked alone.
- A recording is personal data of everyone it records: the masters, and the events, whose
chats, markers and captions are principals' words. The room's region pin and
`retention_days` (live-rooms §3.7) apply to the package as they apply to the takes: the
manifest states where the package is kept, and the verifier holds it to the pin; and
a take's consents bound what `publishable` admits.
- A clip proves that its bytes are those its take's device committed while it captured,
at those indexes, under a commitment the device signed. Its trust rests on the device's
key: a verifier takes it from the device's grant or the take manifest, never from the
clip. The cutter's signature binds the cutter's choice of span, window and head and
nothing more.
- A range proof's nodes are digests of digests of other chunks' descriptors; they disclose
no byte of another chunk. A trimmed clip carries its boundary chunks whole, so the bytes
it trims travel with it; a cutter that must not disclose them cuts on chunk boundaries.
- The time a clip states is the devices' measurement of the room clock, held to *J* at
every commitment; the bytes are proved exactly, the instants to within §6.2's bound.
- A clip cut while a take runs is proved against the commitment that covers it, which the
device can no longer rewrite: any later commitment must extend it (capture-binding §4.3),
so the clip stays a span of the take as it is finally committed.
Reference in a new issue View git blame Copy permalink