live-contribution: the link record's magic is WLK1, no longer the clip record's WCL1 #604

Closed
dcharlot wants to merge 2 commits from wai/contribution-link-magic into main
150 changed files with 378 additions and 147 deletions

View file

@ -4,6 +4,13 @@ All notable changes to the Joule Code standard are recorded here. The format fol
## [Unreleased]
### Changed — `joule-moq-ingest`: the contribution link record's magic is `WLK1`
- **Wire change.** `linkrec::MAGIC` is `WLK1` (WAI `extensions/live-contribution.md` §6.2), so the link record no
longer shares `WCL1` with WAI's clip record (live-recording §5.1), which had it first. A part written before this
change is refused as `malformed`; nothing else in the record changes. `joule-moq-ingest-link-vectors` writes
`.wlk1` files, and adds `clip_record_magic`, a part bearing the clip record's magic, refused `malformed`.
### Changed — `joule-moq-relay`: one object held once, and a socket that keeps what arrives
- **`limits.quic_socket_buffer_bytes`** (4 MiB by default, and for the links a relay dials too): the

View file

@ -222,7 +222,7 @@ stream and the first AAC stream go to the video and audio tracks as WHIP's frame
access unit (one ADTS frame) an object, the PTS on the room clock; a PES missing a packet is
dropped whole. The catalog declares the owner's `link` track.
**The link record** (`linkrec`, `WCL1`): every `[contrib] record_interval_ms` and when the
**The link record** (`linkrec`, `WLK1`): every `[contrib] record_interval_ms` and when the
session ends, the gateway signs a part: the sequence numbers passed, each delivered at the
first try, recovered, dropped or outstanding, the loss runs, requests, retransmissions, round
trips and recovery times, chained to the part before and naming each media track's receipt

View file

@ -58,7 +58,7 @@ retry_after_s = 2 # what a 409 asks a player to wait
[summary]
# out_path = "sessions.jsonl" # session records, one JSON line each; absent: kept in memory only
# link_out_path = "links.hex" # contribution link records (WCL1), one hexadecimal line each
# link_out_path = "links.hex" # contribution link records (WLK1), one hexadecimal line each
[rtmp]
# Enhanced RTMP ingest (live-ingest §12); off unless a listener is set.

View file

@ -1,5 +1,5 @@
//! `joule-moq-ingest-link-vectors`: the conformance corpus of the contribution link record,
//! `WCL1` (WAI `extensions/live-contribution.md` §6), written from fixed seeds, so the
//! `WLK1` (WAI `extensions/live-contribution.md` §6), written from fixed seeds, so the
//! same bytes every run.
//!
//! ```text
@ -7,7 +7,7 @@
//! joule-moq-ingest-link-vectors --check <wai/live-conformance/links> regenerate and compare
//! ```
//!
//! Each vector is a part (one `.wcl1` file) or a series (several, in order), with the
//! Each vector is a part (one `.wlk1` file) or a series (several, in order), with the
//! outcome §6.5 and §6.6 give it: accepted, or refused with its reason (and, in a series,
//! the index of the part refused). Each refused vector breaks one rule and keeps every
//! other: where a rule after the signature is broken, the record is signed again.
@ -111,7 +111,7 @@ struct Out {
impl Out {
fn part(&mut self, name: &str, bytes: Vec<u8>, outcome: Result<(), Reason>, note: &str) {
let file = format!("{name}.wcl1");
let file = format!("{name}.wlk1");
let mut v = json!({"name": name, "kind": "part", "files": [file], "note": note});
match outcome {
Ok(()) => {
@ -132,7 +132,7 @@ impl Out {
}
fn series(&mut self, name: &str, parts: Vec<Vec<u8>>, outcome: Result<(), (usize, Reason)>, note: &str) {
let files: Vec<String> = (0..parts.len()).map(|i| format!("{name}.{i}.wcl1")).collect();
let files: Vec<String> = (0..parts.len()).map(|i| format!("{name}.{i}.wlk1")).collect();
let mut v = json!({"name": name, "kind": "series", "files": files, "note": note});
let got = linkrec::verify_series(&parts);
match outcome {
@ -204,7 +204,10 @@ fn corpus() -> Out {
o.part("truncated", good[..good.len() - 1].to_vec(), Err(Reason::Truncated), "the signature's last byte missing");
let mut b = good.clone();
b[0] = b'X';
o.part("bad_magic", b, Err(Reason::Malformed), "not WCL1");
o.part("bad_magic", b, Err(Reason::Malformed), "not WLK1");
let mut b = good.clone();
b[..4].copy_from_slice(b"WCL1");
o.part("clip_record_magic", b, Err(Reason::Malformed), "the clip record's magic, WCL1 (live-recording §5.1)");
let mut b = good.clone();
b.push(0);
o.part("trailing_byte", b, Err(Reason::Malformed), "a byte after the signature");
@ -440,7 +443,7 @@ fn main() -> ExitCode {
if let Ok(rd) = std::fs::read_dir(&dir) {
for e in rd.flatten() {
let n = e.file_name().to_string_lossy().into_owned();
if (n.ends_with(".wcl1") || n == "vectors.json") && !files.contains_key(&n) {
if (n.ends_with(".wlk1") || n == "vectors.json") && !files.contains_key(&n) {
eprintln!("not generated: {n}");
bad += 1;
}

View file

@ -544,7 +544,7 @@ pub struct SummaryConfig {
/// Each session's record is appended here as one JSON line, when set.
#[serde(default)]
pub out_path: Option<PathBuf>,
/// Each contribution link record part (`WCL1`) is appended here, as one line of
/// Each contribution link record part (`WLK1`) is appended here, as one line of
/// hexadecimal, when set.
#[serde(default)]
pub link_out_path: Option<PathBuf>,

View file

@ -15,7 +15,7 @@
//! (live-ingest §7) and its arrival from the link as its arrival: the same objects,
//! groups, catalog and receipts as a WHIP session's (live-ingest §4 to §8).
//! - **The link record.** Every `record_interval_ms`, and when the session ends, the gateway
//! signs a part of the link's `WCL1` record ([`crate::linkrec`]) and publishes it on the
//! signs a part of the link's `WLK1` record ([`crate::linkrec`]) and publishes it on the
//! owner's `link` track with its receipts: every sequence number delivered, recovered,
//! dropped or outstanding, the requests and the round trips, and the head of every media
//! track's receipt chain at the part's end.

View file

@ -422,7 +422,7 @@ impl Gateway {
self.rtmp.lock().unwrap().iter().map(|(k, h)| (k.clone(), h.frames.load(Ordering::Relaxed))).collect()
}
/// Every contribution link record part (`WCL1`) signed so far.
/// Every contribution link record part (`WLK1`) signed so far.
pub fn link_records(&self) -> Vec<Vec<u8>> {
self.contrib.records.lock().unwrap().clone()
}

View file

@ -25,7 +25,7 @@
//! - **SRT and RIST** ([`contrib`]): contribution over UDP with retransmission, from
//! `joule-contrib`, into the same tracks: a one-time session from `/srt` or `/rist`
//! under the room grant, the transport stream demultiplexed, H.264, HEVC and AAC
//! carried; and the link's own record, `WCL1` ([`linkrec`]), signed part by part on the
//! carried; and the link's own record, `WLK1` ([`linkrec`]), signed part by part on the
//! owner's `link` track.
//!
//! The WebRTC stack is `str0m` (sans-IO, MIT OR Apache-2.0) with its pure-Rust crypto

View file

@ -1,4 +1,4 @@
//! The contribution link record, `WCL1` (WAI `extensions/live-contribution.md` §6): what a
//! The contribution link record, `WLK1` (WAI `extensions/live-contribution.md` §6): what a
//! contribution link over SRT or RIST carried, lost, recovered and dropped, and how long it
//! took, signed by the gateway that received it, part by part, each part chained to the one
//! before and naming the head of every media track's receipt chain at its end.
@ -11,7 +11,7 @@ use crate::room::{sha256, verify_strict, Reason as RoomReason};
use ed25519_dalek::{Signer, SigningKey};
use joule_contrib::stats::{Part, Run, Summary};
pub const MAGIC: &[u8; 4] = b"WCL1";
pub const MAGIC: &[u8; 4] = b"WLK1";
pub const VERSION: u8 = 1;
pub const DOMAIN: &[u8] = b"wai:contrib-link\x01";
pub const MAX_TRACKS: usize = 8;

View file

@ -6,6 +6,128 @@ wire format may change and such changes are recorded here.
## Unreleased
### The contribution link record has its own magic, `WLK1`
- **Wire change.** The contribution link record ([extensions/live-contribution.md](extensions/live-contribution.md)
§6.2) begins `WLK1`, no longer `WCL1`. The clip record of live-recording §5.1 had `WCL1` first; the link record,
added later, took the same four bytes, so a reader that tells records apart by their magic could not tell a link
record from a clip record, and the registry listed `WCL1` for the clip record alone. `WCL1` stays the clip
record's. A link record written before this change is `malformed` to a reader of this revision (§6.5 step 2), as a
new one is to a reader of the earlier revision; the signature's domain, `wai:contrib-link` 0x01, and every other
field are unchanged.
- **The registry** lists `WLK1` as a magic of live-contribution §6.2 (REGISTRY.md and `codecs/registry.rs`).
- **The corpus** [`live-conformance/links/`](live-conformance/links/) is regenerated with the new magic, each vector
file now `.wlk1`, and gains `clip_record_magic`: a part that bears the clip record's magic, refused `malformed`.
`live_contribution_verify.py` reads `WLK1`, and requires that a copy of each accepted part given the clip record's
magic is refused. `joule-moq-ingest` writes and reads `WLK1` (`linkrec::MAGIC`).
- **The reverse:** a record with the link record's magic is not a clip record. The live-recording corpus
[`live-conformance/recordings/`](live-conformance/recordings/) gains the clip `link_record_magic`: chunk 4's clip
record beginning `WLK1` and signed under it, refused `malformed` at the clip (live-recording §5.2, which now names
`WLK1` among the other magics). `live_record_verify.py` requires the same refusal of a copy of each accepted clip
given `WLK1`.
- **Breaking:** the corpus's index names the new files, and the files under the old magic are gone:
`record:corpus:wai/live-conformance/links/vectors.json`,
`record:corpus:wai/live-conformance/links/altered_count.wcl1`,
`record:corpus:wai/live-conformance/links/bad_key.wcl1`,
`record:corpus:wai/live-conformance/links/bad_magic.wcl1`,
`record:corpus:wai/live-conformance/links/bad_signature.wcl1`,
`record:corpus:wai/live-conformance/links/bytes_without_delivery.wcl1`,
`record:corpus:wai/live-conformance/links/count_mismatch.wcl1`,
`record:corpus:wai/live-conformance/links/empty_span.wcl1`,
`record:corpus:wai/live-conformance/links/first_part_with_parent.wcl1`,
`record:corpus:wai/live-conformance/links/first_try_over_delivered.wcl1`,
`record:corpus:wai/live-conformance/links/last_byte_two.wcl1`,
`record:corpus:wai/live-conformance/links/last_without_reason.wcl1`,
`record:corpus:wai/live-conformance/links/later_part_without_parent.wcl1`,
`record:corpus:wai/live-conformance/links/margin_past_window.wcl1`,
`record:corpus:wai/live-conformance/links/margin_without_recovery.wcl1`,
`record:corpus:wai/live-conformance/links/mode_unknown.wcl1`,
`record:corpus:wai/live-conformance/links/no_window.wcl1`,
`record:corpus:wai/live-conformance/links/outstanding_not_last.wcl1`,
`record:corpus:wai/live-conformance/links/protocol_unknown.wcl1`,
`record:corpus:wai/live-conformance/links/reason_unknown.wcl1`,
`record:corpus:wai/live-conformance/links/reason_without_last.wcl1`,
`record:corpus:wai/live-conformance/links/recovery_disordered.wcl1`,
`record:corpus:wai/live-conformance/links/recovery_empty_with_figures.wcl1`,
`record:corpus:wai/live-conformance/links/recovery_samples.wcl1`,
`record:corpus:wai/live-conformance/links/requested_below_requests.wcl1`,
`record:corpus:wai/live-conformance/links/rist_main_aes192.wcl1`,
`record:corpus:wai/live-conformance/links/rist_main_psk256.wcl1`,
`record:corpus:wai/live-conformance/links/rist_simple.wcl1`,
`record:corpus:wai/live-conformance/links/rist_simple_encrypted.wcl1`,
`record:corpus:wai/live-conformance/links/rtt_disordered.wcl1`,
`record:corpus:wai/live-conformance/links/rtt_empty_with_figures.wcl1`,
`record:corpus:wai/live-conformance/links/run_empty.wcl1`,
`record:corpus:wai/live-conformance/links/run_outside.wcl1`,
`record:corpus:wai/live-conformance/links/run_overfull.wcl1`,
`record:corpus:wai/live-conformance/links/run_overfull_alone.wcl1`,
`record:corpus:wai/live-conformance/links/runs_overlap.wcl1`,
`record:corpus:wai/live-conformance/links/runs_past_bound.wcl1`,
`record:corpus:wai/live-conformance/links/runs_short.wcl1`,
`record:corpus:wai/live-conformance/links/runs_truncated.wcl1`,
`record:corpus:wai/live-conformance/links/series_head_moved_without_groups.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_head_moved_without_groups.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_cipher.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_cipher.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_grant.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_grant.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_hop.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_hop.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_mode.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_mode.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_principal.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_principal.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_protocol.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_protocol.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_room.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_room.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_session.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_session.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_window.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_other_window.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_part_after_last.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_part_after_last.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_sequence_gap.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_sequence_gap.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_skips_a_part.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_skips_a_part.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_skips_a_part.2.wcl1`,
`record:corpus:wai/live-conformance/links/series_starts_late.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_starts_late.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_three_parts.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_three_parts.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_three_parts.2.wcl1`,
`record:corpus:wai/live-conformance/links/series_time_gap.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_time_gap.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_added.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_added.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_added.2.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_alias_changed.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_alias_changed.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_dropped.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_track_dropped.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_wrong_parent.0.wcl1`,
`record:corpus:wai/live-conformance/links/series_wrong_parent.1.wcl1`,
`record:corpus:wai/live-conformance/links/series_wrong_parent.2.wcl1`,
`record:corpus:wai/live-conformance/links/small_order_r.wcl1`,
`record:corpus:wai/live-conformance/links/srt_caller_aes192.wcl1`,
`record:corpus:wai/live-conformance/links/srt_cipher_100.wcl1`,
`record:corpus:wai/live-conformance/links/srt_last_outstanding.wcl1`,
`record:corpus:wai/live-conformance/links/srt_listener_clean.wcl1`,
`record:corpus:wai/live-conformance/links/srt_recovered.wcl1`,
`record:corpus:wai/live-conformance/links/too_large.wcl1`,
`record:corpus:wai/live-conformance/links/track_head_without_groups.wcl1`,
`record:corpus:wai/live-conformance/links/track_zero_digest.wcl1`,
`record:corpus:wai/live-conformance/links/tracks_not_begun.wcl1`,
`record:corpus:wai/live-conformance/links/tracks_past_bound.wcl1`,
`record:corpus:wai/live-conformance/links/tracks_unsorted.wcl1`,
`record:corpus:wai/live-conformance/links/trailing_byte.wcl1`,
`record:corpus:wai/live-conformance/links/truncated.wcl1`,
`record:corpus:wai/live-conformance/links/truncated_byte_two.wcl1`,
`record:corpus:wai/live-conformance/links/truncated_not_full.wcl1`,
`record:corpus:wai/live-conformance/links/two_dropped.wcl1`,
`record:corpus:wai/live-conformance/links/unknown_version.wcl1`.
### Tariff version 6: a job priced per endpoint, a third party's usage billed, and an upscale and a tone mapping billed by what re-derives them
- **Tariff and invoice version 6** ([extensions/receipt-ledger.md](extensions/receipt-ledger.md) §4.9). A job price names

View file

@ -407,6 +407,7 @@ and a role are fixed once registered.
| `WKL1` | magic | defined | key-release §4.1 |
| `WKT1` | magic | defined | key-release §9.3 |
| `WLI1` | magic | defined | receipt-ledger §6.1 |
| `WLK1` | magic | defined | live-contribution §6.2 |
| `WLT1` | magic | defined | receipt-ledger §4.1 |
| `WLU1` | magic | defined | channel-lineup §6.1 |
| `WMA1` | magic | defined | live-messaging §8 |

View file

@ -1098,7 +1098,7 @@ And for **contribution over UDP with retransmission** — `wai.live.contribution
Profile, VSF TR-06-2:2024 Main Profile), each admitted by a one-time session a publisher
obtains under its room grant, its MPEG transport stream demultiplexed into the room's tracks
as one envelope an access unit with first-hop receipts. The gateway signs the **link
record**, `WCL1`, part by part: every sequence number delivered at the first try,
record**, `WLK1`, part by part: every sequence number delivered at the first try,
recovered within the receive window, dropped, or outstanding when the link ended, the
retransmission requests, the round trips and the recovery times, each part chained to the
one before and naming the head of every media track's receipt chain; a verifier recomputes

View file

@ -7,7 +7,7 @@
> session under its room grant; its encoder sends an MPEG transport stream over the
> transport; the gateway demultiplexes it and carries each access unit as one envelope on
> the room's tracks with first-hop receipts, as it carries a WHIP publisher's frames.
> Registers the record `wai.live.contribution`: the **contribution link record**, `WCL1`,
> Registers the record `wai.live.contribution`: the **contribution link record**, `WLK1`,
> which the gateway signs part by part while the link runs: what became of every sequence
> number (delivered at the first try, recovered within the receive window, dropped when its
> time passed, or outstanding when the link ended), the retransmission requests and the
@ -339,7 +339,7 @@ measured before the first sequence number fall into the first part.
| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WCL1` (ASCII) |
| `magic` | 4 | `WLK1` (ASCII) |
| `version` | 1 | 1 |
| `room` | 32 | the room id |
| `hop` | 32 | the gateway's key, which signs the record and the tracks' receipts |
@ -416,8 +416,9 @@ fails, with its reason:
1. The record is at most 11 295 bytes (§6.7): otherwise `record_too_large`.
2. It reads, field by field in §6.2's order, to exactly its last byte, with `tracks` at most
8 and runs at most 512: `truncated` when it ends early; `unknown_version` for a version
other than 1; `malformed` for another magic, a byte after `sig`, a count past its bound,
or a `last` or `truncated` other than 0 and 1.
other than 1; `malformed` for another magic (live-recording §5.1's clip record, `WCL1`,
among them), a byte after `sig`, a count past its bound, or a `last` or `truncated` other
than 0 and 1.
3. The signature verifies under `hop`, strictly, as SPEC §7 *Signatures* requires: a key
that fails that rule's steps 1 or 2 is `bad_key`, a signature that fails a later step
`bad_signature`.

View file

@ -345,8 +345,9 @@ start and `trim_out` at the end; the clip's bytes are its parts'.
### 5.2 Reading and rules
A reader reads exactly the bytes it is given, as live-events §3.3 reads a record with
`WCL1` for `WEV1` (`truncated`, `malformed`, `unknown_version`), and refuses a record that
breaks any of these rules as `clip_record_invalid`:
`WCL1` for `WEV1` (`truncated`, `malformed`, `unknown_version`; another magic is
`malformed`, live-contribution §6.2's link record, `WLK1`, among them), and refuses a
record that breaks any of these rules as `clip_record_invalid`:
- *k* is 1 to 16, and no take is named by two parts;
- in each part, `count` is at least 1; each chunk's `index` is `first` plus its position,

View file

@ -1,4 +1,4 @@
# links — the contribution link record, `WCL1`
# links — the contribution link record, `WLK1`
The corpus of [live-contribution](../../extensions/live-contribution.md) §6: parts of the
link record a gateway signs while it receives an SRT or RIST link, and series of them.
@ -8,7 +8,7 @@ link record a gateway signs while it receives an SRT or RIST link, and series of
the index of the part refused), for an accepted part its `digest` and a `summary` of its
fields, and a `note` on what it shows. `hop` is the signing key, `hop_seed` the seed it is
made from, so anyone can sign a new vector the same way.
- Each `.wcl1` file is one record's bytes.
- Each `.wlk1` file is one record's bytes.
Accepted vectors cover each protocol and mode, each key length, a session's last part with
sequence numbers outstanding, a part whose loss runs are truncated, tracks not yet begun,

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show more