Transaction-Science/open-standards

Watch

Fork 0

joule-moq-ingest: Enhanced RTMP v2 ingest, and SCTE 104 automation into signed cues (wai.live.injection) #479

Closed

dcharlot wants to merge 0 commits from wai/ingest-ertmp-scte104 into main

dcharlot commented

2026-10-26 02:26:56 -04:00

Owner

Enhanced RTMP v2 ingest into WAI rooms, and SCTE 104 automation translated to SCTE 35 and signed as the room's cues (wai.live.injection). Both run in joule-moq-ingest, beside WHIP and WHEP.

What is in it

Enhanced RTMP v2 ingest

live-ingest.md §12 was a mapping; it is now implemented. The modules:

rtmp::chunk: the simple handshake, and a sans-IO chunk-stream decoder and encoder. Extended timestamps go on type-3 chunks, and a type-3 chunk that starts a message takes the delta of a fmt-0 header.
rtmp::amf0: AMF0.
rtmp::flv: legacy and enhanced FLV bodies, ModEx TimestampOffsetNano, multitrack (OneTrack, ManyTracks, ManyTracksManyCodecs), and the AVC, HEVC, AV1, VP, Opus, FLAC and AAC configuration records. It builds each frame's payload:
- Annex B, with the record's parameter sets before a key access unit that carries none;
- AV1 temporal units, with the sequence header restored;
- ADTS for AAC;
- a FLAC stream.
rtmp::tracks: the sans-IO binding of RTMP tracks to room tracks, and the packagers.
rtmp::session: the connection and its commands, capsEx 0x0F and the FourCC info maps, onMetaData per track, aggregate messages, the session record (protocol: ertmp), and the reconnect request at shutdown.

Admission works like this:

An RTMP encoder can't answer a proof-of-possession challenge. So the publisher POSTs to /rtmp with its room grant and a proof, admitted as WHIP is.
It gets a one-time stream key: 128 random bits, good for one publish within stream_key_ttl_s.
When the key is used, the grant is checked again.

Tracks and framing:

The room tracks are published at the relay when publish is admitted.
A video track's layer is the grant's publish-profile layer that its onMetaData entry matches (width, height, frame rate rounded to an integer).
What matches no layer is counted, not carried. So is anything that isn't audio track 0, or a codec the profile doesn't list.
live.rtp is the decode time in µs (rate 1 000 000). The new live.cto carries the composition offset, so B-frames keep a monotonic t.
RTMPS runs on tokio-rustls 0.26.6 (MIT OR Apache-2.0), on the workspace's rustls and ring. No C library is added.

SCTE 104 into signed cues (`wai/extensions/cue-injection.md`, new; registered record `wai.live.injection`)

The modules:

scte::scte104 and scte::scte35: reading and translation, a pure function of the message bytes, the arrival instant and the parameters. The program's 90 kHz clock is the room clock.
scte::Engine: holds each cue until its splice instant. It applies withdrawals: a cancel, a cancelled segmentation descriptor, or an early return that withdraws a derived auto-return in. It drops what live-ads §4.3 would refuse in its own chain: an overlap, an unmatched in, or a zero-duration in. It signs each cue as a cue event, after a meta event carrying its injection record WIJ1: the message verbatim, its arrival, leap, the frame rate and the cue's index.
scte::Injector: the TCP API connection (init, alive, one automation system at a time, inject_response and inject_complete_response with SCTE 104's result codes), and the principal's events track at the relay, published from the injector's start, with the join set.

What it supports and refuses:

Implemented: splice_request, time_signal, splice_null, with avail, DTMF, segmentation, time, tier and private descriptor requests.
inject_section_data: answered 124. Its command image carries pts_time on the automation system's clock, not the room's.
Encryption and control words, component mode, schedules, audio descriptors and provisioning, and alternate break duration: answered 124.
VITC and GPI timestamps: answered 123.
VANC carriage (SMPTE ST 2010): a mapping note, §10.

Vectors written from the standards' text

The request whose fields are SCTE 35-1 2023r2 §14.1's sample builds that sample's section byte for byte, CRC included.
§14.2's splice_insert is written byte for byte by the section writer from its fields. No SCTE 104 request can state its break duration.

Corpora, each generated through the reference's own code paths and regenerating byte for byte (tests/conformance_corpus.rs):

wai/live-conformance/injection/: translations, refusals with result codes, scenarios on a clock with the signed chain and the breaks (written by hand from live-ads §4.3), and negative chains.
wai/live-conformance/ertmp/: RTMP message sequences with arrivals, and every object each room track carries.

The independent Python verifiers are injection_verify.py and ertmp_verify.py. They are written from the texts, run as python3 -I, and share no code with the reference. Each has a --self-test that requires every mutated copy of its corpus to fail.

Tests with a local encoder as the test client

tests/ertmp_ffmpeg.rs is ignored unless an encoder is present; it was run on the build host. A local encoder publishes through a proxy that holds NetStream.Publish.Start until the relay subscriber is in place, so every group is seen. Every receipt is verified, each track's frames are written out as an elementary stream, and the encoder's own decoder decodes every frame. The cases:

libx264 with B-frames and AAC (legacy FLV);
HEVC and AVC on the build host's media engine (the platform's hardware encoder interface);
x265 with Opus;
SVT-AV1 with FLAC;
VP9 with Opus.

The encoder's output showed two forms the E-RTMP table does not, and the gateway now reads both:

a FLAC sequence start of STREAMINFO alone;
a coded-frames packet of no bytes at the end of a stream, which is not carried.

Design decisions

The stream key. It is issued by HTTP under the grant plus a proof of possession, and is single-use within 60 s. A bare grant as the stream name would carry no proof, and live-ingest §3.1 says a grant alone proves nothing. This follows §13's SRT session-ID pattern.
Layer binding needs onMetaData. A layer is never inferred from a track id; E-RTMP says ids imply no order.
The decode time is t; the composition offset rides as live.cto. §7's monotonic mapping would otherwise clamp B-frames.
The program clock for SCTE 35 is the room clock, with pts_adjustment 0. The cue's t and its section's pts_time agree by construction.
Cues are signed at their splice instant, not at receipt. A cancel or early return that comes in time leaves nothing to undo in a signed chain. The cost is notice: players learn of a break when it begins.
An auto-return yields a derived in cue (form 0) at the end of the break, so every break is out and in, and an early return can replace it.
The injection record WIJ1 makes each cue recomputable from what the automation system sent; §8 defines its verification.
cw_index 0xFF and event_id_compliance_flag 1, as SCTE 35-1 §14's samples write them: the injector is not told whether the automation system's ids comply.
vvc1 is mapped and not carried: no encoder available to the tests sends it.

Checks

check	command	result
joule-moq-ingest, every target	`rb … cargo test --locked -p joule-moq-ingest --all-targets` on the build host (macOS arm64), and again on the Linux x86_64 test box	all pass on both: lib 42; auth_and_ice 2; codecs 3; conformance_corpus 3; ertmp 6; room_corpus 3; scte104_injection 1; whep_play 1; whip_publish 1; ertmp_ffmpeg 6 ignored
local-encoder interop	`rb … WAI_TEST_FFMPEG=/opt/homebrew/bin/ffmpeg cargo test --locked -p joule-moq-ingest --test ertmp_ffmpeg -- --ignored`	6 of 6 pass, every frame decoded
timing-dependent tests, looped	`ertmp` + `scte104_injection`, 6 runs	6 of 6 pass each
clippy	`cargo clippy --locked -p joule-moq-ingest --all-targets`, on both hosts	0 notes in joule-moq-ingest
lockfile	`cargo metadata --locked --offline` in joule-code-rs	ok; the only addition is `tokio-rustls` 0.26.6
corpora regenerate	`tests/conformance_corpus.rs`	byte for byte
injection verifier	`python3 -I wai/live-conformance/injection_verify.py --self-test wai/live-conformance/injection`, on both hosts	231/231 checks agree; 113/113 mutated corpora refused
E-RTMP verifier	`python3 -I wai/live-conformance/ertmp_verify.py --self-test wai/live-conformance/ertmp`, on both hosts	11 sessions, 58 objects, every one agrees; 98/98 mutated corpora refused
registry	`cargo run -q --no-default-features --bin wai -- registry export` then `--check`; `cargo test --no-default-features --lib codecs::registry`	up to date; 14 pass
repo checks	check-tests-in-ci, check-workflow-concurrency, check-spec-refs, check-wai-section-refs, check-no-vanity-metrics, test-public-text-checks, check-comparative-prose, check-orphan-sources	all clean
merge of `origin/main`	`c90c4d547`: conflicts in CHANGELOG, SPEC §5 and the registry table, each two additions, both kept	registry export regenerated (`--check`: up to date), `codecs::registry` 14 pass, the suite and both verifiers re-run on the merged tree
workflows	`wfsteps.py HEAD origin/main feff9d71b`; `ruby -ryaml` over `.github/workflows/*.yml`	0 problems; all parse
public names	`scripts/check-public-names.sh` (wai/ scope); the changed files; the commit messages	clean. Two joule-code files outside the check's scope carry a pre-existing WebRTC library type name

Falsifications

The baseline was committed before falsifying. Each mutation was applied on the build host, its anchor matched exactly once, and the crate compiled. The named test or vector failed (exit 101 for cargo, 1 for Python), and git checkout restored the tree, which was verified clean.

#	mutation	caught by
M1	parameter sets not put before a key access unit	`ertmp::legacy_avc_and_aac`
M2	ADTS profile off by one	`ertmp::legacy_avc_and_aac`
M3	media time ignored (arrival used)	`ertmp::legacy_avc_and_aac`
M4	the ModEx nanosecond offset dropped	`ertmp::av1_with_flac_and_vp9_with_aac_and_a_nano_offset`
M5	a stream key past its time accepted	`ertmp::stream_key_refusals_and_admission`
M6	the profile's layers ignored	`ertmp::multitrack_layers_by_the_grant_s_profile`
M7	`cw_index` 0	`scte35::the_standard_s_splice_insert_sample`
M8	`pts_time` one tick late	`scte104::the_standard_s_time_signal_sample`
M9	cues signed on receipt	`the_injection_corpus_regenerates_byte_for_byte`
M10	an early return keeps the auto-return in	the same
M11	an overlapping out signed	the same
M12	the injection record's index always 0	`scte104_injection`
M13	no extended timestamp on type-3 chunks	`chunk::encode_and_decode_agree_at_every_split`
M14	a codec the profile does not list carried	`the_ertmp_corpus_regenerates_byte_for_byte`
M15	a STREAMINFO-only FLAC sequence start refused	the same
M16	an empty coded-frames packet handed on	the same
P1	injection_verify.py without its signature check	corpus vector `cue-signature-flipped`
P2	injection_verify.py without the cue-against-translation check	corpus vector `cue-not-the-translation`
P3	ertmp_verify.py without the parameter sets	corpus session `legacy-avc-aac`
P4	ertmp_verify.py without the skew bound	corpus session `wrap-and-reanchor`

Hardware

Build host (macOS, Apple silicon): every build and test, the local-encoder interop, and the energy measurement.
Linux x86_64 test box: the portable suite, clippy and both verifiers, run again.
Its media engine: the encode side of the HEVC and AVC interop cases (the platform's hardware encoder interface), and the pre-encoded clips of the energy measurement.
Ingest: CPU only; no ingest path uses a GPU, NPU or media engine.
Not used: jetson-hub and the AMD box. This slice decodes nothing, so their encode and decode engines belong to the ENC slice.

Energy

Measured on the build host (macOS, Apple silicon) under EN-A's macos-task profile (#460, not merged yet). The host is shared: its load average was 12 to 25 during the run.

Method:

Counter: proc_pid_rusage at RUSAGE_INFO_V6, reading ri_energy_nj + ri_billed_energy − ri_serviced_energy, every 200 ms, per process.
Classification: class OnChipCounter, filtering Unknown; attribution task; fabrics cpu.
Coverage is Partial: the profile reads no GPU or media engine. The ingest path uses neither.
Setup: the gateway and the relay each run as their own process (examples/ingest_energy.rs); there is no subscriber. A local encoder in copy mode replays a pre-encoded 20 s clip into the gateway in real time. The encoder is its own process and is not read, so the encode's energy is not in these figures.
Seven rounds per workload. Each round reads an idle bracket of 5 s, then the publish's busy window (about 19.7 s). Both idle brackets read 0 nJ, since an idle process accrues nothing.
Each figure is the busy energy less the idle mean power over the window.

Results (median, min–max over 7 rounds):

workload	gateway, mJ per s of content	gateway, µJ per video frame (audio included)	relay, mJ per s
HEVC 1280×720, 30 fps, 4 Mb/s (encoded on the media engine), and AAC 128 kb/s	6.96 (6.74–7.37)	229 (222–243)	5.14 (5.02–6.02)
AVC 1920×1080, 30 fps, 6 Mb/s, 1 s GOP, and AAC 128 kb/s	8.66 (7.39–9.37)	285 (243–308)	6.87 (5.64–7.12)

So the gateway hop (receive, package, sign every group's receipts, publish to the relay) draws about 7 to 9 mW of task-attributed CPU energy at these rates. The raw readings and the driver are kept with the session's scratch files, not committed.

Not done

Per-track sizes from the bitstream. One encoder's multitrack FLV writes track 0 legacy and track 1 enhanced, with no videoTrackIdInfoMap. So under a layered profile its track 1 matches no layer and is not carried. Reading a track's size from its configuration record (AVC/HEVC SPS, AV1 sequence header) is the follow-up.
Cross-gateway stream keys. After a reconnect request, the key is re-armed on this gateway only; a reconnect to another gateway needs a new key.
vvc1: mapped, not carried.
SCTE 104 in VANC: SMPTE ST 2010 was not available to read; §10 is a mapping only.
The requests answered 124 (listed above).
An "announce at receipt" mode for cues: not built.
The ffmpeg interop in CI. It does not run there: the linux-arm64 runner has no encoder, and the cases need the platform's media engine. CI lists those tests as ignored, not passed.

Needs a lead or owner decision

A relay door limitation, outside this slice. joule-moq-relay's door awaits each object on its stream's read path until the group's receipts arrive, and those come only after the group ends. A video group larger than the QUIC stream receive window stalls until hold_ms, then is refused, and the publisher ends on backpressure. Seen with 1080p at 8 Mb/s and 2 s GOPs; 4 Mb/s at 2 s and 6 Mb/s at 1 s pass. The same bound applies to WHIP groups.
The relay's door rules. A door rule on the bare room prefix also holds events tracks, which carry no receipts. Deployments should write door rules per media kind ([room, *, audio|video]); the injection test does.
Whether a stream key may be valid across gateways (shared key store or signed tickets).

🤖 Generated with Claude Code

Closed: Landed on main through merge train E (#499, merge bf39ec9a); verified by content: this PR's head is in main

Enhanced RTMP v2 ingest into WAI rooms, and SCTE 104 automation translated to SCTE 35 and signed as the room's cues (`wai.live.injection`). Both run in `joule-moq-ingest`, beside WHIP and WHEP. ## What is in it ### Enhanced RTMP v2 ingest `live-ingest.md` §12 was a mapping; it is now implemented. The modules: - `rtmp::chunk`: the simple handshake, and a sans-IO chunk-stream decoder and encoder. Extended timestamps go on type-3 chunks, and a type-3 chunk that starts a message takes the delta of a fmt-0 header. - `rtmp::amf0`: AMF0. - `rtmp::flv`: legacy and enhanced FLV bodies, ModEx `TimestampOffsetNano`, multitrack (OneTrack, ManyTracks, ManyTracksManyCodecs), and the AVC, HEVC, AV1, VP, Opus, FLAC and AAC configuration records. It builds each frame's payload: - Annex B, with the record's parameter sets before a key access unit that carries none; - AV1 temporal units, with the sequence header restored; - ADTS for AAC; - a FLAC stream. - `rtmp::tracks`: the sans-IO binding of RTMP tracks to room tracks, and the packagers. - `rtmp::session`: the connection and its commands, `capsEx` 0x0F and the FourCC info maps, `onMetaData` per track, aggregate messages, the session record (`protocol: ertmp`), and the reconnect request at shutdown. Admission works like this: - An RTMP encoder can't answer a proof-of-possession challenge. So the publisher POSTs to `/rtmp` with its room grant and a proof, admitted as WHIP is. - It gets a one-time stream key: 128 random bits, good for one publish within `stream_key_ttl_s`. - When the key is used, the grant is checked again. Tracks and framing: - The room tracks are published at the relay when `publish` is admitted. - A video track's layer is the grant's publish-profile layer that its `onMetaData` entry matches (width, height, frame rate rounded to an integer). - What matches no layer is counted, not carried. So is anything that isn't audio track 0, or a codec the profile doesn't list. - `live.rtp` is the decode time in µs (`rate` 1 000 000). The new `live.cto` carries the composition offset, so B-frames keep a monotonic `t`. - RTMPS runs on `tokio-rustls` 0.26.6 (MIT OR Apache-2.0), on the workspace's rustls and ring. No C library is added. ### SCTE 104 into signed cues (`wai/extensions/cue-injection.md`, new; registered record `wai.live.injection`) The modules: - `scte::scte104` and `scte::scte35`: reading and translation, a pure function of the message bytes, the arrival instant and the parameters. The program's 90 kHz clock is the room clock. - `scte::Engine`: holds each cue until its splice instant. It applies withdrawals: a cancel, a cancelled segmentation descriptor, or an early return that withdraws a derived auto-return in. It drops what live-ads §4.3 would refuse in its own chain: an overlap, an unmatched in, or a zero-duration in. It signs each cue as a `cue` event, after a `meta` event carrying its injection record `WIJ1`: the message verbatim, its arrival, `leap`, the frame rate and the cue's index. - `scte::Injector`: the TCP API connection (init, alive, one automation system at a time, `inject_response` and `inject_complete_response` with SCTE 104's result codes), and the principal's events track at the relay, published from the injector's start, with the join set. What it supports and refuses: - Implemented: splice_request, time_signal, splice_null, with avail, DTMF, segmentation, time, tier and private descriptor requests. - `inject_section_data`: answered 124. Its command image carries `pts_time` on the automation system's clock, not the room's. - Encryption and control words, component mode, schedules, audio descriptors and provisioning, and alternate break duration: answered 124. - VITC and GPI timestamps: answered 123. - VANC carriage (SMPTE ST 2010): a mapping note, §10. ### Vectors written from the standards' text - The request whose fields are SCTE 35-1 2023r2 §14.1's sample builds that sample's section byte for byte, CRC included. - §14.2's splice_insert is written byte for byte by the section writer from its fields. No SCTE 104 request can state its break duration. Corpora, each generated through the reference's own code paths and regenerating byte for byte (`tests/conformance_corpus.rs`): - `wai/live-conformance/injection/`: translations, refusals with result codes, scenarios on a clock with the signed chain and the breaks (written by hand from live-ads §4.3), and negative chains. - `wai/live-conformance/ertmp/`: RTMP message sequences with arrivals, and every object each room track carries. The independent Python verifiers are `injection_verify.py` and `ertmp_verify.py`. They are written from the texts, run as `python3 -I`, and share no code with the reference. Each has a `--self-test` that requires every mutated copy of its corpus to fail. ### Tests with a local encoder as the test client `tests/ertmp_ffmpeg.rs` is ignored unless an encoder is present; it was run on the build host. A local encoder publishes through a proxy that holds `NetStream.Publish.Start` until the relay subscriber is in place, so every group is seen. Every receipt is verified, each track's frames are written out as an elementary stream, and the encoder's own decoder decodes every frame. The cases: - libx264 with B-frames and AAC (legacy FLV); - HEVC and AVC on the build host's media engine (the platform's hardware encoder interface); - x265 with Opus; - SVT-AV1 with FLAC; - VP9 with Opus. The encoder's output showed two forms the E-RTMP table does not, and the gateway now reads both: - a FLAC sequence start of STREAMINFO alone; - a coded-frames packet of no bytes at the end of a stream, which is not carried. ## Design decisions - **The stream key.** It is issued by HTTP under the grant plus a proof of possession, and is single-use within 60 s. A bare grant as the stream name would carry no proof, and live-ingest §3.1 says a grant alone proves nothing. This follows §13's SRT session-ID pattern. - **Layer binding needs `onMetaData`.** A layer is never inferred from a track id; E-RTMP says ids imply no order. - **The decode time is `t`; the composition offset rides as `live.cto`.** §7's monotonic mapping would otherwise clamp B-frames. - **The program clock for SCTE 35 is the room clock**, with `pts_adjustment` 0. The cue's `t` and its section's `pts_time` agree by construction. - **Cues are signed at their splice instant, not at receipt.** A cancel or early return that comes in time leaves nothing to undo in a signed chain. The cost is notice: players learn of a break when it begins. - **An auto-return yields a derived in cue** (form 0) at the end of the break, so every break is out and in, and an early return can replace it. - **The injection record `WIJ1`** makes each cue recomputable from what the automation system sent; §8 defines its verification. - **`cw_index` 0xFF and `event_id_compliance_flag` 1, as SCTE 35-1 §14's samples write them**: the injector is not told whether the automation system's ids comply. - **`vvc1` is mapped and not carried**: no encoder available to the tests sends it. ## Checks | check | command | result | |---|---|---| | joule-moq-ingest, every target | `rb … cargo test --locked -p joule-moq-ingest --all-targets` on the build host (macOS arm64), and again on the Linux x86_64 test box | all pass on both: lib 42; auth_and_ice 2; codecs 3; conformance_corpus 3; ertmp 6; room_corpus 3; scte104_injection 1; whep_play 1; whip_publish 1; ertmp_ffmpeg 6 ignored | | local-encoder interop | `rb … WAI_TEST_FFMPEG=/opt/homebrew/bin/ffmpeg cargo test --locked -p joule-moq-ingest --test ertmp_ffmpeg -- --ignored` | 6 of 6 pass, every frame decoded | | timing-dependent tests, looped | `ertmp` + `scte104_injection`, 6 runs | 6 of 6 pass each | | clippy | `cargo clippy --locked -p joule-moq-ingest --all-targets`, on both hosts | 0 notes in joule-moq-ingest | | lockfile | `cargo metadata --locked --offline` in joule-code-rs | ok; the only addition is `tokio-rustls` 0.26.6 | | corpora regenerate | `tests/conformance_corpus.rs` | byte for byte | | injection verifier | `python3 -I wai/live-conformance/injection_verify.py --self-test wai/live-conformance/injection`, on both hosts | 231/231 checks agree; 113/113 mutated corpora refused | | E-RTMP verifier | `python3 -I wai/live-conformance/ertmp_verify.py --self-test wai/live-conformance/ertmp`, on both hosts | 11 sessions, 58 objects, every one agrees; 98/98 mutated corpora refused | | registry | `cargo run -q --no-default-features --bin wai -- registry export` then `--check`; `cargo test --no-default-features --lib codecs::registry` | up to date; 14 pass | | repo checks | check-tests-in-ci, check-workflow-concurrency, check-spec-refs, check-wai-section-refs, check-no-vanity-metrics, test-public-text-checks, check-comparative-prose, check-orphan-sources | all clean | | merge of `origin/main` | `c90c4d547`: conflicts in CHANGELOG, SPEC §5 and the registry table, each two additions, both kept | registry export regenerated (`--check`: up to date), `codecs::registry` 14 pass, the suite and both verifiers re-run on the merged tree | | workflows | `wfsteps.py HEAD origin/main feff9d71b`; `ruby -ryaml` over `.github/workflows/*.yml` | 0 problems; all parse | | public names | `scripts/check-public-names.sh` (wai/ scope); the changed files; the commit messages | clean. Two joule-code files outside the check's scope carry a pre-existing WebRTC library type name | ## Falsifications The baseline was committed before falsifying. Each mutation was applied on the build host, its anchor matched exactly once, and the crate compiled. The named test or vector failed (exit 101 for cargo, 1 for Python), and `git checkout` restored the tree, which was verified clean. | # | mutation | caught by | |---|---|---| | M1 | parameter sets not put before a key access unit | `ertmp::legacy_avc_and_aac` | | M2 | ADTS profile off by one | `ertmp::legacy_avc_and_aac` | | M3 | media time ignored (arrival used) | `ertmp::legacy_avc_and_aac` | | M4 | the ModEx nanosecond offset dropped | `ertmp::av1_with_flac_and_vp9_with_aac_and_a_nano_offset` | | M5 | a stream key past its time accepted | `ertmp::stream_key_refusals_and_admission` | | M6 | the profile's layers ignored | `ertmp::multitrack_layers_by_the_grant_s_profile` | | M7 | `cw_index` 0 | `scte35::the_standard_s_splice_insert_sample` | | M8 | `pts_time` one tick late | `scte104::the_standard_s_time_signal_sample` | | M9 | cues signed on receipt | `the_injection_corpus_regenerates_byte_for_byte` | | M10 | an early return keeps the auto-return in | the same | | M11 | an overlapping out signed | the same | | M12 | the injection record's index always 0 | `scte104_injection` | | M13 | no extended timestamp on type-3 chunks | `chunk::encode_and_decode_agree_at_every_split` | | M14 | a codec the profile does not list carried | `the_ertmp_corpus_regenerates_byte_for_byte` | | M15 | a STREAMINFO-only FLAC sequence start refused | the same | | M16 | an empty coded-frames packet handed on | the same | | P1 | injection_verify.py without its signature check | corpus vector `cue-signature-flipped` | | P2 | injection_verify.py without the cue-against-translation check | corpus vector `cue-not-the-translation` | | P3 | ertmp_verify.py without the parameter sets | corpus session `legacy-avc-aac` | | P4 | ertmp_verify.py without the skew bound | corpus session `wrap-and-reanchor` | ## Hardware - **Build host (macOS, Apple silicon):** every build and test, the local-encoder interop, and the energy measurement. - **Linux x86_64 test box:** the portable suite, clippy and both verifiers, run again. - **Its media engine:** the encode side of the HEVC and AVC interop cases (the platform's hardware encoder interface), and the pre-encoded clips of the energy measurement. - **Ingest:** CPU only; no ingest path uses a GPU, NPU or media engine. - **Not used:** jetson-hub and the AMD box. This slice decodes nothing, so their encode and decode engines belong to the ENC slice. ## Energy Measured on the build host (macOS, Apple silicon) under EN-A's `macos-task` profile (#460, not merged yet). The host is shared: its load average was 12 to 25 during the run. **Method:** - Counter: `proc_pid_rusage` at RUSAGE_INFO_V6, reading `ri_energy_nj + ri_billed_energy − ri_serviced_energy`, every 200 ms, per process. - Classification: class `OnChipCounter`, filtering `Unknown`; attribution `task`; fabrics `cpu`. - Coverage is `Partial`: the profile reads no GPU or media engine. The ingest path uses neither. - Setup: the gateway and the relay each run as their own process (`examples/ingest_energy.rs`); there is no subscriber. A local encoder in copy mode replays a pre-encoded 20 s clip into the gateway in real time. The encoder is its own process and is not read, so the encode's energy is not in these figures. - Seven rounds per workload. Each round reads an idle bracket of 5 s, then the publish's busy window (about 19.7 s). Both idle brackets read 0 nJ, since an idle process accrues nothing. - Each figure is the busy energy less the idle mean power over the window. Results (median, min–max over 7 rounds): | workload | gateway, mJ per s of content | gateway, µJ per video frame (audio included) | relay, mJ per s | |---|---|---|---| | HEVC 1280×720, 30 fps, 4 Mb/s (encoded on the media engine), and AAC 128 kb/s | 6.96 (6.74–7.37) | 229 (222–243) | 5.14 (5.02–6.02) | | AVC 1920×1080, 30 fps, 6 Mb/s, 1 s GOP, and AAC 128 kb/s | 8.66 (7.39–9.37) | 285 (243–308) | 6.87 (5.64–7.12) | So the gateway hop (receive, package, sign every group's receipts, publish to the relay) draws about 7 to 9 mW of task-attributed CPU energy at these rates. The raw readings and the driver are kept with the session's scratch files, not committed. ## Not done - **Per-track sizes from the bitstream.** One encoder's multitrack FLV writes track 0 legacy and track 1 enhanced, with no `videoTrackIdInfoMap`. So under a layered profile its track 1 matches no layer and is not carried. Reading a track's size from its configuration record (AVC/HEVC SPS, AV1 sequence header) is the follow-up. - **Cross-gateway stream keys.** After a reconnect request, the key is re-armed on this gateway only; a reconnect to another gateway needs a new key. - **`vvc1`**: mapped, not carried. - **SCTE 104 in VANC**: SMPTE ST 2010 was not available to read; §10 is a mapping only. - **The requests answered 124** (listed above). - **An "announce at receipt" mode** for cues: not built. - **The ffmpeg interop in CI.** It does not run there: the linux-arm64 runner has no encoder, and the cases need the platform's media engine. CI lists those tests as ignored, not passed. ## Needs a lead or owner decision - **A relay door limitation, outside this slice.** `joule-moq-relay`'s door awaits each object on its stream's read path until the group's receipts arrive, and those come only after the group ends. A video group larger than the QUIC stream receive window stalls until `hold_ms`, then is refused, and the publisher ends on backpressure. Seen with 1080p at 8 Mb/s and 2 s GOPs; 4 Mb/s at 2 s and 6 Mb/s at 1 s pass. The same bound applies to WHIP groups. - **The relay's door rules.** A door rule on the bare room prefix also holds events tracks, which carry no receipts. Deployments should write door rules per media kind (`[room, *, audio|video]`); the injection test does. - **Whether a stream key may be valid across gateways** (shared key store or signed tickets). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- **Closed:** Landed on main through merge train E (#499, merge bf39ec9a); verified by content: this PR's head is in main

dcharlot added 12 commits

2026-10-26 02:26:56 -04:00

joule-moq-ingest: Enhanced RTMP ingest and SCTE 104 injection, first cut 66b7127689

The RTMP handshake and chunk stream, AMF0, FLV legacy and enhanced tag bodies
(ModEx, multitrack), configuration records and the payload of each frame; a
stream key issued under a room grant; SCTE 104 messages translated to SCTE 35
sections and cues, signed with an injection record before each.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-moq-ingest: Enhanced RTMP end-to-end tests with a publisher of our own 28b0449522

Legacy AVC and AAC; enhanced HEVC with composition offsets and Opus; AV1 with FLAC,
VP9 with enhanced AAC, a ModEx nanosecond offset; multitrack layers bound by the
grant's publish profile; stream-key refusals; RTMPS and the reconnect request.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-moq-ingest: an RTMP publish's tracks, sans IO 8001296940

The binding rules, sequence starts and frames of one publish move to rtmp::Tracks,
which the connection feeds and the conformance corpus is generated through.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-moq-ingest: SCTE 104 injection end to end 41ac1b0ebd

An automation system of our own initialises the injector, keeps it alive, and sends
splice and time-signal requests; each cue is signed when its splice comes, after its
injection record, and reaches a relay subscriber on the principal's events track,
published from the injector's start.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai/live-conformance: the injection and E-RTMP corpora, generated through joule-moq-ingest c090454438

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: cue-injection, SCTE 104 to SCTE 35 to signed cues, and injection_verify.py d694f5356b

The extension text; the corpus's scenarios state the breaks live-ads §4.3 makes,
written by hand; the second implementation, from the text, agrees on every vector.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: live-ingest §12, Enhanced RTMP v2 implemented; ertmp_verify.py bc8e80c75e

The stream key, the connection, the binding of tracks to layers, the FourCCs and
their payloads, time, groups, bounds and the reconnect request; §6's cto member; the
reasons. The corpus's second implementation agrees on every object.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-moq-ingest: a local encoder as the test client; STREAMINFO-only FLAC; empty coded frames 3888cfff74

tests/ertmp_ffmpeg.rs (ignored unless an encoder is present) publishes from a local
encoder through a proxy that holds Publish.Start until the subscriber is in place,
and decodes each track's frames in a conformant decoder: AVC with B-frames and AAC,
HEVC and AVC on the platform's media engine, HEVC with Opus, AV1 with FLAC, VP9 with
Opus. The encoder sends a FLAC sequence start as STREAMINFO alone, and a coded-frames
packet of no bytes at the end: both now read, the latter not carried. The corpus
gains a session for each.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

wai: register wai.live.injection; SPEC §5, CHANGELOG, the crate's README; CI 1eabee8f24

The registry row and its export; the verifiers' mutation self-tests; a step for both
verifiers in wai-conformance, and joule-code-conformance's path filters for the
corpora and texts its tests read.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-moq-ingest: clippy notes cleared d3deeefa9a

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-moq-ingest: examples/ingest_energy, a relay and a gateway as two processes feff9d71b2

What a per-process energy reader (energy-measurement §9.2, macos-task) reads one hop
of: fixed seeds, loopback listeners, a stream key per line of input.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge origin/main into wai/ingest-ertmp-scte104

wai-pack / wai-pack · independent verifier (Python) (pull_request) Has been cancelled

Details

wai-pack / wai-pack · the floor through a media prober (pull_request) Has been cancelled

Details

wai-pack / wai-pack · dependency licences (pull_request) Has been cancelled

Details

WAI energy receipts / does this change touch it (pull_request) Has been cancelled

Details

WAI energy receipts / energy receipts · linux (pull_request) Has been cancelled

Details

WAI energy receipts / energy receipts · linux-arm64 (pull_request) Has been cancelled

Details

WAI energy receipts / energy receipts · macos (pull_request) Has been cancelled

Details

WAI conformance / does this change touch it (pull_request) Has been cancelled

Details

WAI conformance / cargo-class corpora · linux (pull_request) Has been cancelled

Details

WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Has been cancelled

Details

WAI conformance / cargo-class corpora · macos (pull_request) Has been cancelled

Details

WAI conformance / quantum core and corpus under wasm32-wasip2 (wasmtime) (pull_request) Has been cancelled

Details

WAI conformance / v2 vocabulary · manifests validate as stated (pull_request) Has been cancelled

Details

WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Has been cancelled

Details

WAI conformance / delivery telemetry · CMCD and CMSD against an independent library (pull_request) Has been cancelled

Details

WAI conformance / cross-arch byte-equality · linux (pull_request) Has been cancelled

Details

WAI conformance / cross-arch byte-equality · linux-arm64 (pull_request) Has been cancelled

Details

WAI conformance / cross-arch byte-equality · macos (pull_request) Has been cancelled

Details

WAI conformance / codec wrappers · fuzz smoke, text decode limits, C entry points, provenance, the semantic path (pull_request) Has been cancelled

Details

WAI conformance / C2PA manifest emission · signed, read back, validated (pull_request) Has been cancelled

Details

standards workspaces build + test / which workspaces this change touches (pull_request) Has been cancelled

Details

standards workspaces build + test / macOS energy backends (pull_request) Has been cancelled

Details

repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Has been cancelled

Details

joule-moq-web browser client / joule-moq-web · does this change touch it (pull_request) Has been cancelled

Details

joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (pull_request) Has been cancelled

Details

joule-moq-room room service / joule-moq-room · does this change touch it (pull_request) Has been cancelled

Details

joule-moq-room room service / joule-moq-room, every target (pull_request) Has been cancelled

Details

joule-code conformance gate / Build + round-trip C1-C4 vectors (pull_request) Has been cancelled

Details

joule-code conformance gate / joule-moq-relay, every target (pull_request) Has been cancelled

Details

joule-code conformance gate / joule-moq-ingest, every target (pull_request) Has been cancelled

Details

c90c4d5479

Conflicts in wai/CHANGELOG.md, wai/SPEC.md and wai-rs's registry table, each two
additions at one place: both kept (main's registrations and §5 paragraphs before
wai.live.injection; this branch's changelog entry first, as the newest).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request

2026-10-26 17:15:43 -04:00

wai merge train E: twenty-five PRs onto main, the relay's upstream window bounded, and three fixes the merged tree needed #499

dcharlot referenced this pull request from a commit

2026-10-26 20:23:54 -04:00

Merge #479 (wai/ingest-ertmp-scte104 @c90c4d54): Enhanced RTMP v2 ingest and SCTE 104 into signed cues

dcharlot referenced this pull request from a commit

2026-10-26 20:23:54 -04:00

Merge #483 (wai/ingest-srt-rist @b54dc94c): SRT and RIST contribution ingest, with a signed link record

dcharlot closed this pull request

2026-10-26 20:59:00 -04:00

wai-web wasm32 / does this change touch it (pull_request) Has been cancelled

Details

wai-web wasm32 / wai-web wasm32 build · corpus, menu and writers (pull_request) Has been cancelled

Details

wai-web reach builds / wai-web reach · does this change touch it (pull_request) Has been cancelled

Details

wai-web reach builds / wai-web reach builds · the package of record, every output as recorded (pull_request) Has been cancelled

Details

wai-web reach builds / wai-web reach builds · this tree's wai-web, every rung on its engine line (pull_request) Has been cancelled

Details

wai-quantum studio and desktop / does this change touch it (pull_request) Has been cancelled

Details

wai-quantum studio and desktop / studio · browser (pull_request) Has been cancelled

Details

wai-quantum studio and desktop / desktop app · macos (pull_request) Has been cancelled

Details

wai-pack / wai-pack · does this change touch it (pull_request) Has been cancelled

Details

wai-pack / wai-pack · linux (pull_request) Has been cancelled

Details

wai-pack / wai-pack · linux-arm64 (pull_request) Has been cancelled

Details

wai-pack / wai-pack · macos (pull_request) Has been cancelled

Details

wai-pack / wai-pack · independent verifier (Python) (pull_request) Has been cancelled

Details

wai-pack / wai-pack · the floor through a media prober (pull_request) Has been cancelled

Details

wai-pack / wai-pack · dependency licences (pull_request) Has been cancelled

Details

WAI energy receipts / does this change touch it (pull_request) Has been cancelled

Details

WAI energy receipts / energy receipts · linux (pull_request) Has been cancelled

Details

WAI energy receipts / energy receipts · linux-arm64 (pull_request) Has been cancelled

Details

WAI energy receipts / energy receipts · macos (pull_request) Has been cancelled

Details

WAI conformance / does this change touch it (pull_request) Has been cancelled

Details

WAI conformance / cargo-class corpora · linux (pull_request) Has been cancelled

Details

WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Has been cancelled

Details

WAI conformance / cargo-class corpora · macos (pull_request) Has been cancelled

Details

WAI conformance / quantum core and corpus under wasm32-wasip2 (wasmtime) (pull_request) Has been cancelled

Details

WAI conformance / v2 vocabulary · manifests validate as stated (pull_request) Has been cancelled

Details

WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Has been cancelled

Details

WAI conformance / delivery telemetry · CMCD and CMSD against an independent library (pull_request) Has been cancelled

Details

WAI conformance / cross-arch byte-equality · linux (pull_request) Has been cancelled

Details

WAI conformance / cross-arch byte-equality · linux-arm64 (pull_request) Has been cancelled

Details

WAI conformance / cross-arch byte-equality · macos (pull_request) Has been cancelled

Details

WAI conformance / codec wrappers · fuzz smoke, text decode limits, C entry points, provenance, the semantic path (pull_request) Has been cancelled

Details

WAI conformance / C2PA manifest emission · signed, read back, validated (pull_request) Has been cancelled

Details

standards workspaces build + test / which workspaces this change touches (pull_request) Has been cancelled

Details

standards workspaces build + test / macOS energy backends (pull_request) Has been cancelled

Details

repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Has been cancelled

Details

joule-moq-web browser client / joule-moq-web · does this change touch it (pull_request) Has been cancelled

Details

joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (pull_request) Has been cancelled

Details

joule-moq-room room service / joule-moq-room · does this change touch it (pull_request) Has been cancelled

Details

joule-moq-room room service / joule-moq-room, every target (pull_request) Has been cancelled

Details

joule-code conformance gate / Build + round-trip C1-C4 vectors (pull_request) Has been cancelled

Details

joule-code conformance gate / joule-moq-relay, every target (pull_request) Has been cancelled

Details

joule-code conformance gate / joule-moq-ingest, every target (pull_request) Has been cancelled

Details

Pull request closed

Please reopen this pull request to perform a merge.

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!479

No description provided.

Rows
Columns

joule-moq-ingest: Enhanced RTMP v2 ingest, and SCTE 104 automation into signed cues (wai.live.injection) #479

What is in it

Enhanced RTMP v2 ingest

SCTE 104 into signed cues (wai/extensions/cue-injection.md, new; registered record wai.live.injection)

Vectors written from the standards' text

Tests with a local encoder as the test client

Design decisions

Checks

Falsifications

Hardware

Energy

Not done

Needs a lead or owner decision

Pull request closed

SCTE 104 into signed cues (`wai/extensions/cue-injection.md`, new; registered record `wai.live.injection`)