diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 70e2f667b..7e9e76b20 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -6,6 +6,143 @@ wire format may change and such changes are recorded here. ## v0.4.0 — current +### Broadcast and streaming codecs — registered, licence-classified, never defaults + +- SPEC §5 registers the elementary video and audio bitstreams distribution chains + carry: `wai.video.{av2,vvc,hevc,avc,vp9,evc,evc.baseline,avs3,lcevc}` and + `wai.audio.{mpegh,avs3,aac}`. Each names one payload framing, which §8 then + fixes forever: the Annex B byte stream for AVC, HEVC, VVC and LCEVC (each + standard also defines a bare NAL unit stream, which is not the capability), + AV2's Annex B length-delimited bitstream, IVF for VP9, the Annex B raw + bitstream file storage format for EVC, MHAS for MPEG-H 3D Audio and ADTS for + AAC. The AVS3 video and audio rows name each standard's elementary + bitstream; those texts were not read, so whether either defines a second + form is unchecked. Each also carries a licence class with its recorded + basis; two are classifications rather than verified statuses (AVS3: no + published fee schedule; EVC Baseline: royalty-free by design, no formal + guarantee). All are sink-supplied; none is in the floor or the recommended + set. No decoder and no C dependency is added. +- AV2 is registered royalty-free but not recommended: the reference decoder + exists, deployable decode is not yet confirmed. Promotion is a MINOR revision. +- LCEVC is an enhancement layer, carried embedded in its base bitstream or as a + `WAI2` companion whose `companion_of` must name one of its base codecs (AVC, + HEVC, VVC, EVC, VP9, AV1, under every capability naming those bitstreams; + `codecs::enhancement_bases`) and resolve to exactly one selectable rendition + of it; a base takes at most one enhancement layer. The base list is WAI's + registration, not a restriction of ISO/IEC 23094-2, which leaves the base + decoder unspecified; a later revision may add a base (a widening, §8). +- `wai.video.lcevc` and the HDR metadata descriptors (`wai.meta.hdr10plus`, + `wai.meta.dovi`) are companion-only, now stated normatively: never a v1.0 + `capability` or `fallback`, never a v1.1 selectable rendition. `pick` / + `primary` never return them; `Wai::check_component_roles` (v1.0) and + `WaiMulti::check_component_roles` (v1.1) report envelopes that break the + rules (`RoleError`, `#[non_exhaustive]`). A `WAI1` naming one only as its + `fallback` does not conform and never takes dispatch step 5, but still decodes + via its primary capability (§4 step 3); new `Wai::dispatch_capability` applies + §4 with that rule, and `wai decode` and `wai inspect` both use it, so they + agree on such a file. `wai wrap` refuses to write a non-conforming envelope + and, with `--companion `, writes the `WAI2` companion form, + whose intent follows the same rule as the `WAI1` form's (`--intent`, below), + checked over the base payload's capability and each companion's. HDR + metadata may bind to a neural video payload, and the base check is what + refuses `wai wrap wai.neural.dcvc_rt … --companion wai.meta.hdr10plus …` + (`NeuralFloat`, SPEC §7) unless another intent is given. Every companion-only + capability is in a standard class, so the companion check refuses nothing + yet. `wai unwrap --component ` extracts a `WAI2` component. +- SPEC §7.1 states the three rules a `WAI2` companion meets, and + `WaiMulti::check_component_roles` enforces each: (1) its capability is + companion-only or unregistered — a registered standalone capability marked + companion is `RoleError::StandaloneAsCompanion`; (2) `companion_of` names a + capability it applies to (`codecs::companion_applies_to`: for HDR dynamic + metadata a registered capability in the `video` media class — new + `codecs::is_video_class`: every `wai.video.*` capability that is not + companion-only, the neural video codecs `wai.neural.{video_bmshj2018,dcvc_rt, + dcvc_fm}` and the films `wai.film.{linear,reel}` — or an unregistered one, + whose class is unknown; for LCEVC a base codec) — + `RoleError::IncompatibleBase`; (3) that base is present, and unique for an + enhancement layer. The check previously let an HDR descriptor bind to an audio + rendition, LCEVC bind to any `wai.video.*` rendition (JPEG 2000 included), and + any standalone codec ride as a companion; the reference test that carried an + IAMF stream as a companion now carries Dolby Vision metadata instead. +- SPEC §8 now says what these rules are, so they do not read as a redefinition + inside `1.x`: a registration fixes a capability's payload format **and its + role** (standalone media, or companion-only); changing either, or narrowing + what a companion-only capability applies to, needs a new MAJOR. A MINOR + revision may add a rule that enforces a registration, or a manifest field's + definition, as it was made, if it lists every kind of envelope it makes + non-conforming. The companion rules are such rules: the HDR descriptors were + registered as companion descriptors to a video payload, not standalone media; + `wai.video.lcevc` was registered companion-only with these rules; and + `role = "companion"` was defined as naming, in `companion_of`, the component + it binds to. SPEC §7.1 lists the envelopes they make non-conforming: an HDR + descriptor as a `WAI1` `capability` or `fallback`, an HDR descriptor as a + `WAI2` selectable rendition (which `pick` / `primary` no longer select), an + HDR descriptor bound to a registered capability outside the `video` media + class, a registered standalone capability marked companion, and a companion + of any capability with no `companion_of` or none that names a selectable + rendition in the envelope. Such envelopes still parse. The list also records + that the earlier §7.1 named immersive-audio metadata as a companion use and a + reference test carried `wai.audio.iamf` that way, which rule 1 now rejects, + since `wai.audio.iamf` was registered as standalone media. +- SPEC §8 also says what registering a capability does to an envelope that used + its string while no revision registered it. §7.1 leaves such a use unchecked + (an HDR descriptor's unregistered base is not constrained; an unregistered + companion needs only a present base), so the registering revision may make the envelope + non-conforming where it uses the string against the new registration: a + companion whose capability is registered standalone, an HDR descriptor bound + to a capability registered outside the `video` class, a companion-only + capability carried as a payload or bound to a base it does not admit. §8 + lists these kinds once for every registration; they still parse. This covers + `wai.video.lcevc`, which no earlier revision registered. +- `WaiMulti::companions_of` no longer says every companion it returns is meant + to be applied: the role check accepts an unregistered companion with a + present base, and SPEC §7.1 forbids a sink to apply a companion whose + capability it does not recognise. +- The broadcast-codec dispatch guidance asks for a recommended-set `WAI2` + rendition, not a bare v1.0 `fallback`: step 5 decodes the same payload with + the fallback's codec, which cannot read an HEVC, VVC or AV2 bitstream. +- `codecs::capability_license` is now exhaustive with no default class, so a + registered capability can no longer become royalty-free by omission. New + `codecs::mandatory_floor()` / `recommended_set()`; tests hold that every + default is royalty-free. `wai capabilities` gains a `role` column. +- **Replicate classes: exhaustive, and the exact class needs recorded evidence.** + `codecs::capability_replicate_class` no longer has a default arm (the + determinism-tiers change below still ended it in a `StandardDefined` + fall-through). Every registered capability is listed under one class; one + listed nowhere returns `None`, which the tests reject. `StandardDefined`, which + promises the same bytes on every conforming sink, is admitted only on a recorded + basis — a conformance clause that requires identical output, or the WAI + extension whose equivalence pins the reconstruction — pinned one row per + capability in the tests (`standard_defined_rests_on_recorded_evidence`). The + broadcast codecs enter with: + - `StandardDefined`: AVC, HEVC and VVC (ITU-T H.264 §C.4, H.265 and H.266 + §C.5.1: every sample a conforming decoder outputs equals the specified + decoding process's) and VP9 (VP9 specification v0.6 §8.1); + - `StandardTolerance`: AV2 (AV2 v1.0.0 §7.1: film grain, as AV1); + - `StandardUnclassified`: EVC (both entries), AVS3 video and audio, LCEVC, + MPEG-H 3D Audio and AAC, whose conformance documents were not read. + + `wai.audio.ivas` moves from `StandardUnclassified` to `StandardTolerance` on + 3GPP TS 26.252 V19.2.0 §7.2.1: implementations based on the floating-point + reference are tested by non-bit-exact procedures. New + `codecs::is_cross_decoder_exact(cap)` is true where the class's + `replicate_promise` is `ByteExact`, except for a capability whose output is + computed from other envelopes' decoded output (new + `codecs::composes_decoded_sources`: `wai.audio.scene`). A scene's mixdown is + exact given its sources' PCM, and a source decoded through `wai.audio.opus` may + decode differently on two conforming sinks, so the capability alone does not + make it portable; new `codecs::is_cross_decoder_exact_with_sources(cap, + sources)` is true once every source's capability is known and exact, and + false when it cannot tell (no source given, or a source outside + `wai.audio.*`). The class's `ByteExact` promise for a scene holds given its + sources' PCM, so SPEC §7 names `is_cross_decoder_exact`, not the promise, as + the audit for the same bytes on every sink. The + promise covers the output the standard or extension specifies, not a + conversion a sink applies afterwards. New + `tools/wai_replicate_probe.sh` decodes each format with every independent + decoder on the host and fails if a `StandardDefined` capability's decoders + disagree — a falsification check, not the basis of any class. + ### JPEG AI exact decoder: no platform math library on the path, and a cross-platform check - The exact decoder (`jpegai_decode::decode`) called the platform math library's diff --git a/wai/SPEC.md b/wai/SPEC.md index 4fbd6353e..f4d653c73 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -144,8 +144,11 @@ this algorithm on the plaintext; see §6.4. ## 5. Registered capabilities (the zeroth menu) -WAI v1.0 registers the following capabilities. Each names a SOTA -royalty-free standard codec. The "payload format" column is the +WAI v1.0 registers the following capabilities. Each names a standard +codec or a WAI-native class. The mandatory floor and the recommended +set are royalty-free; every registered capability carries a licence +class (see the end of this section), and one that is not royalty-free +is never a default. The "payload format" column is the on-the-wire payload bytes; in every case it is the canonical bytes that codec normally emits (a JXL file, an Opus stream, etc.) — WAI does not wrap or re-frame them. @@ -330,26 +333,178 @@ named standard's canonical bytes. | class | capabilities | |---|---| -| immersive audio | `wai.audio.iamf` (IAMF/Eclipsa, royalty-free) · `wai.audio.ac4` (Dolby AC-4, proprietary) · `wai.audio.ivas` (3GPP IVAS, SEP-pool) | +| immersive audio | `wai.audio.iamf` (IAMF/Eclipsa, royalty-free) · `wai.audio.ac4` (Dolby AC-4, proprietary) · `wai.audio.ivas` (3GPP IVAS, SEP-pool) · `wai.audio.mpegh` (MPEG-H 3D Audio, SEP-pool) · `wai.audio.avs3` (AVS3-P3 / Audio Vivid, SEP-pool) — the last two with the broadcast codecs below | | cinema mastering | `wai.image.jp2` · `wai.video.j2k` (JPEG 2000 / DCP, OpenJPEG) | | volumetric / 3D | `wai.mesh.draco` (royalty-free) · `wai.pc.gpcc` · `wai.pc.vpcc` (MPEG point clouds) · `wai.splat.sog` (Gaussian-splat compression, candidate) | | haptics | `wai.haptic.mihs` (MPEG-I Haptics, ISO/IEC 23090-31; its 23090-33 ships bit-exact conformance vectors) | | HDR dynamic metadata | `wai.meta.hdr10plus` (royalty-free) · `wai.meta.dovi` (Dolby Vision, proprietary) — **companion descriptors** to a video payload, not standalone media | -Every registered capability also carries an IP class — +### Broadcast and streaming codecs (registered, sink-supplied) + +The elementary video and audio bitstreams that terrestrial and satellite +broadcast, streaming and device playback carry. Registering them lets one +envelope name the bitstream a distribution chain already holds, and dispatch it +with a declared fallback or a sibling rendition. Every entry is +**sink-supplied** — the reference implementation bundles no decoder for any of +them — and **none is in the mandatory floor or the recommended set**: +registration never makes a capability a default. Payload is the named +standard's canonical elementary bitstream in the framing given; WAI does not +re-frame it. + +| capability | standard | payload (canonical bytes) | licence | +|---|---|---|---| +| `wai.video.av2` | AV2 (AOMedia) — AV2 Bitstream & Decoding Process Specification v1.0.0, 28 May 2026 | the length-delimited bitstream of that specification's Annex B (each OBU preceded by its `leb128` byte length) | royalty-free | +| `wai.video.vvc` | VVC — ITU-T H.266 \| ISO/IEC 23090-3 | H.266 Annex B byte stream | SEP-pool | +| `wai.video.hevc` | HEVC — ITU-T H.265 \| ISO/IEC 23008-2 | H.265 Annex B byte stream | SEP-pool | +| `wai.video.avc` | AVC — ITU-T H.264 \| ISO/IEC 14496-10 | H.264 Annex B byte stream | SEP-pool | +| `wai.video.vp9` | VP9 Bitstream & Decoding Process Specification | VP9 frames in the IVF frame container (FourCC `VP90`) | royalty-free | +| `wai.video.evc` | EVC — MPEG-5 Part 1, ISO/IEC 23094-1, any profile | raw bitstream file storage format, ISO/IEC 23094-1 Annex B | SEP-pool (Main-profile tools) | +| `wai.video.evc.baseline` | EVC, Baseline profile only | as `wai.video.evc`, Baseline-conformant | royalty-free by design | +| `wai.video.avs3` | AVS3 video — AVS3-P2, T/AI 109.2-2021, IEEE 1857.10 | AVS3-P2 elementary bitstream | SEP-pool | +| `wai.video.lcevc` | LCEVC — MPEG-5 Part 2, ISO/IEC 23094-2 | LCEVC enhancement as the ISO/IEC 23094-2 Annex B byte stream (start-code-delimited NAL units) — **companion-only**, see below | SEP-pool | +| `wai.audio.mpegh` | MPEG-H 3D Audio — ISO/IEC 23008-3 | MHAS packet stream (MPEG-H 3D Audio Stream) | SEP-pool | +| `wai.audio.avs3` | AVS3 audio / Audio Vivid — AVS3-P3, T/AI 109.3-2023; GY/T 363-2023; GB/T 46271-2025 | AVS3-P3 elementary bitstream | SEP-pool | +| `wai.audio.aac` | AAC-LC, HE-AAC, HE-AAC v2 — ISO/IEC 14496-3 | ADTS stream (ISO/IEC 13818-7 / 14496-3); Extended HE-AAC is not this capability | SEP-pool | + +*SEP-pool* means the standard's essential patents are licensed for royalties — +through patent pools, a single standard-essential-patent licensor, or FRAND +holders. Per-entry sources and caveats are in the research registry linked +below. Each is tiered `Lossy` (its distribution role; the lossless modes some +of them have are signalled in the bitstream, as for AVIF). No model WAI runs +is involved in any of them. For replication (§7) they fall in three classes: + +- **`StandardDefined`** — AVC, HEVC, VVC and VP9. Each standard's decoder + conformance requires every sample a conforming decoder outputs to equal the + value its decoding process produces (H.264 §C.4; H.265 and H.266 §C.5.1; VP9 + §8.1), so every conforming decoder outputs the same cropped decoded pictures. + The guarantee stops at those pictures: film-grain synthesis an SEI message + signals, colour conversion and other post-decoding processing are outside it. +- **`StandardTolerance`** — AV2. As for AV1, the output is identical unless the + payload signals film grain; then grain synthesis is held only to the absence + of "perceptually significant differences" from the reference (AV2 v1.0.0 + §7.1). +- **`StandardUnclassified`** — EVC (both entries), AVS3 video, LCEVC, MPEG-H 3D + Audio, AVS3 audio and AAC. Their conformance documents were not read for this + revision, so the registry promises no byte identity for them. For AAC, three + decoders on one host gave three different PCM streams from one ADTS stream. + LCEVC's decoding process includes a signalled dither step (ISO/IEC 23094-2, + clause 8.11). Any of them moves to `StandardDefined` only with the conformance + clause that fixes its output. + +The clauses behind each class, what was not read, and the cross-decoder +measurements are in the research registry. + +**Dispatch.** These dispatch exactly as §4 describes. Because none is a floor or +recommended capability, no sink can be assumed to hold one: an encoder +addressing sinks it does not control SHOULD ship such a payload in a `WAI2` +envelope (§7.1) together with a recommended-set rendition of the same content, +and that rendition is what keeps the file from being inert at a conforming +sink. A v1.0 `fallback` alone does not: dispatch step 5 decodes the *same* +payload with the fallback's codec (§4), so a fallback helps only where that +codec's decoder accepts the primary's bitstream (`wai.video.evc.baseline` → +`wai.video.evc` is such a pair; an HEVC, VVC or AV2 bitstream is not an AV1 +bitstream). A deployer that must not ship pool-licensed decode filters on the +licence class below, not on a hand-kept list. + +**AV2 placement.** AV2 is royalty-free, with a published specification and a +reference decoder (AVM v1.0.0). It is registered here and **not** in the +recommended set, because that set states what every conforming sink SHOULD +implement, and in September 2026 that is not yet a reasonable ask: AVM is a +reference model, a production software decoder is still in development, and +hardware decode in shipping devices is not confirmed. Promotion to the +recommended set is a MINOR revision (§8) once deployable decode exists; the +capability string and its payload framing do not change. Until then an AV2 +payload SHOULD travel in a `WAI2` envelope with a `wai.video.av1` rendition of +the same content. + +**Enhancement layers (LCEVC).** An LCEVC enhancement is not a picture: it holds +residuals that an LCEVC decoder adds to the upscaled output of **one specific +base encode**. This registration names its **base codecs**: AVC, HEVC, VVC, +EVC, VP9 and AV1, as the capabilities that name those bitstreams — +`wai.video.avc`, `wai.video.hevc`, `wai.video.vvc`, `wai.video.evc`, +`wai.video.evc.baseline`, `wai.video.vp9`, `wai.video.av1` and +`wai.video.av1.lossless` — and not AV2, AVS3, JPEG 2000 or the WAI-native +integer codecs. The list is WAI's, not the LCEVC standard's: ISO/IEC +23094-2:2021 takes its decoded base picture from "a base decoder" (definition +3.17) and names no base codec among its normative references. A later revision +MAY add a base, which widens what the enhancement layer applies to (§8). WAI +carries it one of two ways. + +- **Embedded.** The enhancement rides inside the base bitstream (SEI / + registered user data). The payload is the base codec's bytes, under the base + capability; a sink without LCEVC decodes the base and ignores the enhancement + under the base codec's own rules. No `wai.video.lcevc` component exists. +- **Separate.** The enhancement, in the byte-stream framing given below, is a + `WAI2` companion (§7.1): `role = "companion"`, `companion_of` naming the base + capability. This is the shape of LCEVC's dual-track storage in ISO/IEC + 14496-15, where an enhancement track refers to exactly one base track, and the + companion mechanism fits it with one added rule that binds in both directions: + `companion_of` MUST name one of its base codecs and resolve to **exactly + one** selectable rendition of it in the envelope, and that rendition MUST NOT + be the base of any other enhancement layer. `companion_of` names a capability, + not a component, so with two renditions of the base capability it cannot say + which encode the residuals belong to; and a sink applies every companion bound + to the rendition it picks (§7.1), so two enhancement layers on one base would + add two sets of residuals to one picture. Either way the picture is + corrupted. Such an envelope does not conform; a sink MUST NOT apply its + enhancement layers, and MAY still present a base rendition. + +The separate carriage's payload is the enhancement in the byte-stream form of +ISO/IEC 23094-2 Annex B, where a start code prefix (`0x000001`) precedes every +NAL unit. ISO/IEC 23094-2 defines an LCEVC bitstream as either a NAL unit +stream or a byte stream (definition 3.2); the byte stream is the form that +delimits its own NAL units, so it is the one `wai.video.lcevc` names. A NAL +unit stream without start code prefixes (for example with each NAL unit +preceded by its length) is not a `wai.video.lcevc` payload; carrying that form +would need its own capability string (§8). + +**Companion-only capabilities.** `wai.video.lcevc` and the HDR dynamic-metadata +descriptors `wai.meta.hdr10plus` and `wai.meta.dovi` are **companion-only**: +their payload means nothing without the base it accompanies. An HDR descriptor +applies to a rendition whose payload is video (§7.1, rule 2), `wai.video.lcevc` +to one rendition of one of its base codecs (above). Every other registered +capability is **standalone media** and is never carried as a companion (§7.1). +A companion-only +capability MUST NOT be a v1.0 `capability` (a `WAI1` carries one payload, so +there is nothing for it to accompany) or a v1.0 `fallback` (dispatch step 5 +would decode it on its own), nor a v1.1 selectable rendition; a sink never +dispatches it standalone. A `WAI1` naming one as its `capability` is inert: its +payload is companion data. A `WAI1` naming one only as its `fallback` does not +conform, and a sink MUST NOT take dispatch step 5 with it; step 3 is unaffected, +so a sink that advertises the envelope's `capability` still decodes it (§4). +The reference implementation checks these rules in +`Wai::check_component_roles` (v1.0) and `WaiMulti::check_component_roles` +(v1.1: the companion rules of §7.1, including both enhancement-layer rules +above), dispatches `WAI1` +envelopes through `Wai::dispatch_capability`, which applies both sentences +above, never returns a companion-only component from `pick` / `primary`, and +its `wai wrap` tool refuses to write an envelope that fails either check. Parsing stays +format-agnostic: a non-conforming envelope still parses, and the check is what +reports it. + +The companion binding carries no per-picture timing. The framing above fixes +the bytes of a separately carried enhancement stream, not how its access units +align picture by picture with its base outside a timed container; this +revision does not pin that alignment, so the embedded form is the unambiguous +one until it does. + +**Licence classes.** Every registered capability also carries an IP class — `codecs::capability_license(cap)` ∈ {royalty-free, patent-pool, proprietary, non-commercial}, with `is_royalty_free(cap)` as the predicate. It is a registry property (not a per-file manifest claim — a codec's licence can't be honestly asserted in an envelope's header), so a deployer can audit a title or a whole catalogue in code: encode only to royalty-free capabilities, or flag every pool/proprietary one. The per-title choice of which toll road to pay is a -property of the format, enforceable, not a policy on a slide. +property of the format, enforceable, not a policy on a slide. The +classification is exhaustive with no default class: a capability registered +without an explicit class is a registry defect the reference tests reject, so no +codec becomes royalty-free by omission. The same tests hold that every floor and +recommended-set capability is royalty-free. See [`research/media-codec-registry.md`](research/media-codec-registry.md) for the per-capability standard / spec number / verified royalty status / reference library, what each named standard's decoder conformance fixes (§7), and the -remaining spec follow-up (companion-component binding for the -HDR / audio metadata descriptors). +remaining spec follow-ups. --- @@ -447,21 +602,27 @@ selects this path. Live: https://wai.transaction.science/private-channel - A **conforming encoder** MUST emit containers (§2) that any conforming sink can read; the payload MUST be the canonical byte stream for the registered capability. -- Conformance is verified by **bit-exact decode-equivalence with the - reference library** named in §5 for each registered capability, except where - this section states a weaker criterion through the capability's registry class - (below): a `StandardTolerance` capability conforms by its named standard's own +- A decode is conformant by the criterion its capability's registry class + (below) sets. A `StandardDefined` or `NeuralIntegerExact` capability is + verified by **bit-exact decode-equivalence**: against the in-crate decoder or + wrapped reference library for the capabilities the reference implementation + decodes (`codecs::sink_capabilities()` and the in-crate neural and WAI-native + decoders), and against the named standard's own conformance bitstreams and + decoded outputs for a sink-supplied codec. A `GenerativeDeterministic` + capability is verified by bit-exact generation-equivalence at its pin. A + `StandardTolerance` capability conforms by its named standard's own decoder-conformance criterion around a reference, not by bit-exact equality with one library's output; a `StandardUnclassified` capability conforms by its - named standard's criterion, which this revision has not classified; a - `NeuralEntropyExact` capability conforms, as WAI registers it, - at the entropy-decoder output (for `wai.image.jpegai`, the criterion its named - standard sets on the reconstruction is not read in this revision); and - `NeuralFloat` and `Behavioral` capabilities make no claim that their - reconstruction is bit-exact across sinks. The WAI Rust - reference impl in `wai-rs/` is built on those - very libraries and ships a test suite that exercises every registered - capability through the full WAI envelope (`cargo test --lib`). + named standard's criterion, which this revision has not classified; WAI adds + no bit-exact requirement to either. A `NeuralEntropyExact` capability + conforms, as WAI registers it, at the entropy-decoder output (for + `wai.image.jpegai`, the criterion its named standard sets on the + reconstruction is not read in this revision). + `NeuralFloat` capabilities make no claim that their reconstruction is + bit-exact across sinks, and `Behavioral` ones conform by similarity. The WAI + Rust reference impl in `wai-rs/` ships a test suite that carries every + registered capability through the full WAI envelope (`cargo test --lib`, + `tests/carriage.rs`). ### Replicate-soundness — the integer-entropy contract (normative) @@ -568,28 +729,42 @@ registry as `codecs::capability_replicate_class(cap) -> ReplicateClass`: tier for these. - **`StandardDefined`** — the reconstruction is specified exactly: by the named standard, for the lossless formats whose decoded output it determines - (`wai.image.png`, `wai.audio.flac`, `wai.text.zstd`, `wai.text.xz`), or by the + (`wai.image.png`, `wai.audio.flac`, `wai.text.zstd`, `wai.text.xz`) and for the + video codecs whose decoder conformance requires every output sample to equal + the specified decoding process's (`wai.video.avc`, `wai.video.hevc`, + `wai.video.vvc`, `wai.video.vp9`; clauses in the table below), or by the WAI extension that defines the capability's `*-equivalence` over integer / fixed-point state (world, score, film, quantum, ternary, the haptic signal, the integer inter-frame codec and the other WAI-native capabilities). `replicate` promises the same bytes on every conforming sink. An extension's equivalence holds given its inputs: where it composes other capabilities' decoded output (a `wai.audio.scene` source decoded through `wai.audio.opus`, say), it is only as - exact as those capabilities' classes. No neural capability is in this class. + exact as those capabilities' classes. `wai.audio.scene` is the one registered + capability whose output depends on envelopes it references rather than + carries, so the reference registry does not report it byte-exact from the + capability alone (below). No neural decode is in this class, and a + capability enters it only with a recorded basis — the conformance clause or + extension that fixes its output (the reference registry tests pin one per + member); without one, a standard codec is `StandardUnclassified`. - **`StandardTolerance`** — a standard codec whose named standard, for at least some conforming payloads, holds decoders to a criterion around a reference rather than to identical output. For Opus and JPEG XL the clauses read state it as numeric thresholds. For JPEG and JPEG 2000 the text read points to a criterion without stating its bound (T.81 leaves the accuracy of dequantisation and IDCT to its Part 2, which was not read; only the table of contents and terms - of ISO/IEC 15444-4 were read), so this revision states none. For AV1 and AVIF - when film grain is signalled it is the absence of "perceptually significant - differences" from the reference grain synthesis, which the AV1 specification - leaves undefined (§7.2: its definition "is beyond the scope of this - specification"). `replicate` promises what that conformance promises: every + of ISO/IEC 15444-4 were read), so this revision states none. For IVAS, an + implementation based on its floating-point reference code whose output is not + bit-exact is tested by a non-bit-exact procedure instead: a per-frame + loudness-difference corridor for stereo and immersive operation, and for mono + operation the procedure of TS 26.444, which was not read (3GPP TS 26.252 + §7.2.1). For AV1, AVIF and AV2 when film grain is signalled it is the absence + of "perceptually significant differences" from the reference grain synthesis, + which the AV1 specification leaves undefined (§7.2: its definition "is beyond + the scope of this specification") and the AV2 specification leaves to others + to define (§7.1). `replicate` promises what that conformance promises: every conforming sink reconstructs within the standard's criterion — for film-grain - AV1, a criterion no document WAI cites quantifies. It does not promise byte - identity across sinks. + AV1 and AV2, a criterion no document WAI cites quantifies. It does not promise + byte identity across sinks. - **`StandardUnclassified`** — a standard codec whose named standard's decoder-conformance clauses this revision has not checked. `replicate` promises what that conformance promises; the registry does not say whether that is byte @@ -608,14 +783,19 @@ cited are the ones read; the evidence, and what was not read, is in | class | capabilities | what the named standard fixes | |---|---|---| | `StandardDefined` | `wai.image.png`, `wai.audio.flac`, `wai.text.zstd`, `wai.text.xz` | the decoded output exactly: lossless formats (PNG, RFC 2083 — its filters are exact integer operations; FLAC, RFC 9639) and lossless byte formats | +| | `wai.video.avc`, `wai.video.hevc`, `wai.video.vvc` | every sample of every cropped decoded picture a conforming decoder outputs equals the value the specified decoding process produces (ITU-T H.264 (V16, 06/2026) §C.4; H.265 (V11, 01/2026) §C.5.1; H.266 (V4, 01/2026) §C.5.1); processing an SEI message only signals, such as film grain synthesis, is outside that decoding process | +| | `wai.video.vp9` | output frames "identical in all respects", in the same order, to those of the specified decoding process (VP9 Bitstream & Decoding Process Specification v0.6 §8.1) | | `StandardTolerance` | `wai.audio.opus` | output within `opus_compare`'s thresholds of the reference decoder, with the same final range-decoder state (RFC 6716 §6) | | | `wai.image.jpeg` | decoding "with appropriate accuracy", the accuracy of dequantisation and IDCT specified in Part 2 (ITU-T T.81 §7, A.3.3); Part 2 (ITU-T T.83) was not read, so the bound is not stated here | | | `wai.image.jxl` | decoded samples within per-test peak-error and RMSE thresholds of reference images (ISO/IEC 18181-3 §4.1–§4.3) | | | `wai.image.jp2`, `wai.video.j2k` | decoded components compared with reference components by error metrics, and a reversible path its definitions describe as lossless (ISO/IEC 15444-4); only the table of contents and terms of its public preview were read, so the metrics and thresholds are not stated here | | | `wai.video.av1`, `wai.video.av1.lossless`, `wai.image.avif` | identical output frames, unless the payload signals film grain; then identical frames before grain synthesis, and output frames that "do not have perceptually significant differences" from the reference grain synthesis, a criterion whose definition the specification places outside its scope (AV1 §7.2) | +| | `wai.video.av2` | as AV1: identical output unless film grain is signalled, then one of the same two options (AV2 v1.0.0 §7.1) | +| | `wai.audio.ivas` | output bit-exact to the test sequences, or, for an implementation based on the floating-point reference code whose output is not, a non-bit-exact procedure: TS 26.444's for mono operation, a per-frame loudness-difference corridor for stereo and immersive operation (3GPP TS 26.252 V19.2.0 §7.1, §7.2.1) | | | `wai.audio.iamf` | its substreams include Opus (above), and renderers are out of its scope | -| `StandardUnclassified` | `wai.audio.ac4`, `wai.audio.ivas`, `wai.mesh.draco`, `wai.pc.gpcc`, `wai.pc.vpcc`, `wai.splat.sog`, `wai.haptic.mihs`, `wai.meta.hdr10plus`, `wai.meta.dovi` | not checked in this revision (all sink-supplied; `wai.splat.sog` names no frozen standard yet) | -| | `wai.mesh.glb`, `wai.splat.spz` | the payload bytes pass through intact; what a sink decodes from them is not classified here (glTF can embed JPEG images and compressed geometry; SPZ stores quantised attributes) | +| `StandardUnclassified` | `wai.video.evc`, `wai.video.evc.baseline`, `wai.video.avs3`, `wai.video.lcevc`, `wai.audio.mpegh`, `wai.audio.avs3`, `wai.audio.ac4`, `wai.mesh.draco`, `wai.pc.gpcc`, `wai.pc.vpcc`, `wai.splat.sog`, `wai.haptic.mihs`, `wai.meta.hdr10plus`, `wai.meta.dovi` | not checked in this revision (all sink-supplied; `wai.splat.sog` names no frozen standard yet) | +| | `wai.audio.aac` | not checked in this revision; three decoders on one host gave three different PCM streams from one ADTS stream, so it is not placed where byte identity would be promised | +| | `wai.mesh.glb`, `wai.splat.spz` | the payload bytes pass through intact; what a sink decodes from them is not classified here (a glTF binary may carry JPEG images in its buffer and compressed geometry through extensions, and glTF 2.0 specifies an asset format, not a decoder conformance over what is rendered; SPZ stores quantised attributes) | A conversion a sink applies after the standard's decode is not part of the promise. A standard fixes its decoded samples at their own depth, layout and @@ -677,11 +857,12 @@ it whenever the prior declared an integer contract. **Registry change in v0.4.0 — the standard codecs.** Every standard codec was classed `StandardDefined`, and the first form of this revision's §3 promised the -same bytes on every conforming sink over all of them. The capabilities in the `StandardTolerance` and -`StandardUnclassified` rows above are now in those classes. `replicate` over them -is still honoured; what it promises is narrower — the named standard's decoder -conformance, not byte identity across sinks. No other registered capability -changes class. +same bytes on every conforming sink over all of them. The previously registered +capabilities in the `StandardTolerance` and `StandardUnclassified` rows above are +now in those classes. `replicate` over them is still honoured; what it promises is +narrower — the named standard's decoder conformance, not byte identity across +sinks. No other previously registered capability changes class. The broadcast and +streaming codecs this revision registers (§5) enter with the classes in the table. A conforming sink that receives `intent = replicate` for a capability whose class is `NeuralEntropyExact`, `NeuralFloat`, `GenerativeDeterministic` or @@ -700,13 +881,36 @@ capability is refused. `decode_envelope_strict` decodes only through `model_requirement.capability`, so that is the one it checks. On the encoder side, `wai wrap`, which pins no prior, writes `intent = replicate` only where that check accepts it unpinned — over the three standard classes — for the capability -and for any fallback it declares, and otherwise refuses unless given another -intent (`codecs::unpinned_envelope_intent`). The registry class is also the *static* -audit of the same property: `wai capabilities` prints it per row and reports the -count of neural integer-exact capabilities, so a deployer can require "dispatch -`replicate` only to a cross-hardware-sound decode", or only to a capability whose -class promises the same bytes (`ReplicatePromise::ByteExact`), in code, exactly -as it audits royalty status. +and for any fallback it declares, or, in the `WAI2` companion form (§7.1), for the +base payload's capability and each companion's, and otherwise refuses unless +given another intent (`codecs::unpinned_envelope_intent`). The registry class is +also the *static* audit of the same property: `wai capabilities` prints it per +row and reports the count of neural integer-exact capabilities, so a deployer can +require "dispatch `replicate` only to a cross-hardware-sound decode", or only to +a capability whose output is the same bytes on every conforming sink +(`codecs::is_cross_decoder_exact`), in code, exactly as it audits royalty +status. The class's promise +(`ReplicateClass::replicate_promise`) and `is_cross_decoder_exact` say the same +about byte identity for every registered capability but one. A capability whose +output is computed from other envelopes' decoded output +(`codecs::composes_decoded_sources`: `wai.audio.scene`) has the class +`StandardDefined` and the promise `ReplicatePromise::ByteExact`, because its mix +is exact given the PCM its sources decode to; `is_cross_decoder_exact` answers +from the capability alone and is false for it, because a source decoded through +`wai.audio.opus`, say, may decode differently on two conforming sinks. A deployer +auditing for the same bytes uses `is_cross_decoder_exact`, not the promise. Once +the capabilities of the envelopes a scene references are known, +`codecs::is_cross_decoder_exact_with_sources` is true when the scene's class and +every source's promise the same bytes; it answers false when it cannot tell — no +source given, or a source outside `wai.audio.*`, which a scene may not reference. +The `replicate` check above covers one envelope, so `replicate` over a scene is +honoured and promises the same mix only given the same source PCM; each source is +an envelope of its own, checked by its own class. That promise covers the output +the standard or extension specifies, before any conversion a sink applies +afterwards (above), so a hash over a converted output is portable only between +sinks that also agree on the conversion. The classification is exhaustive with no +default class: a capability registered without one has no class, and the registry +tests reject it. This makes the cross-hardware property explicit per capability: a neural capability whose decode is *integer all the way down* can put its reconstruction @@ -848,28 +1052,115 @@ hide a major-version mismatch. A `WAI2` component carries an OPTIONAL `role`. Absent ⇒ `"rendition"` (a selectable alternative of the same content — the default above). -`"companion"` marks a **descriptor applied *with* the chosen rendition, not -instead of it**, naming the component it binds to in `companion_of`. This -resolves the one media class that is not a standalone payload: **HDR dynamic -metadata** (`wai.meta.hdr10plus` / `wai.meta.dovi`, SMPTE ST 2094) and -immersive-audio metadata accompany a video/audio payload rather than replacing -it. Carried as companions, the descriptor and its target ride in the **same -envelope**, so the binding is intrinsic — not a spoofable sidecar. +`"companion"` marks a component **applied *with* the chosen rendition, not +instead of it**, naming the capability it binds to in `companion_of`. It +carries the payloads that are not standalone media — the capabilities §5 +registers as **companion-only**: the HDR dynamic-metadata descriptors +`wai.meta.hdr10plus` / `wai.meta.dovi` (SMPTE ST 2094), which describe a video +payload, and the enhancement layer `wai.video.lcevc`, which adds residuals to +one. Carried as companions, the companion and its base ride in the **same +envelope**, so the binding is intrinsic — not a spoofable sidecar. This +revision registers no capability for immersive-audio mix metadata; +`wai.audio.iamf` carries its own audio substreams, so it is standalone media +and travels as a rendition. -The selection algorithm is unchanged: a sink picks a rendition among the +Selection: a sink picks a rendition among the `role != "companion"` entries, then applies every companion whose `companion_of` names the chosen component (e.g. the HDR10+ block for the AV1 -video the sink picked). `pick`/`primary` never return a companion. Static HDR +video the sink picked). `pick`/`primary` never return a companion, nor a +capability §5 registers as companion-only. Static HDR characteristics (ST 2086 mastering display, HLG transfer) stay in-band in the video bitstream / a manifest color field and need no companion. +A component with `role = "companion"` conforms only if: + +1. its capability is companion-only, or is not registered in this revision. A + registered capability that is not companion-only is standalone media — its + payload decodes on its own — and MUST be carried as a rendition, not as a + companion; +2. `companion_of` is present and names a capability the companion applies to. + An HDR dynamic-metadata descriptor, registered as a companion to a video + payload, applies to a registered capability whose payload is video — the + `video` media class (§3), as the capability's registration places it: every + `wai.video.*` capability that is not companion-only, the neural video codecs + `wai.neural.video_bmshj2018`, `wai.neural.dcvc_rt` and `wai.neural.dcvc_fm` + (§5, *Video:*), and the films `wai.film.linear` and `wai.film.reel` + ([linear-film](extensions/linear-film.md) gives a linear film's `media` the + value `"video"`, and a reel is a film with cuts). A capability registered as + video in a later revision joins that class. + `wai.video.lcevc` applies to one of the base codecs §5 lists for it. No + companion applies to a companion-only capability. This revision constrains + no base for a companion whose capability it does not register, nor an HDR + descriptor's base whose capability it does not register; +3. a selectable rendition of that capability is present in the envelope — for + an **enhancement layer**, exactly one, the base encode its residuals were + produced against, and that rendition is the base of no other enhancement + layer (§5). Several descriptor companions MAY bind to one base, and a + descriptor MAY bind to several renditions of one capability. + +An envelope with a companion that breaks any of these does not conform (the +reference `WaiMulti::check_component_roles` reports the first break). A sink +MUST NOT apply a companion that breaks rule 1 or 2, nor one whose capability it +does not recognise; a companion that breaks rule 3 binds to nothing, or, for an +enhancement layer, is handled as §5 states. The sink MAY still present the +rendition it picked. + ### v1.0 compatibility -v1.0 envelopes (`WAI1`) remain valid and are unchanged. Encoders +v1.0 envelopes (`WAI1`) remain valid and their format is unchanged, except that +a `WAI1` naming a companion-only capability as its `capability` or `fallback` +does not conform (§5; below). Encoders SHOULD emit `WAI1` when only one rendition is present (simpler form, wider sink support). `WAI2` is for sources that genuinely benefit from shipping multiple renditions per file. +### Envelopes the companion rules make non-conforming + +The companion rules of §5 and this section change no capability's payload +format and no capability's role (§8). Each enforces something fixed before the +rules were stated: + +- **a registration.** `wai.meta.hdr10plus` and `wai.meta.dovi` were registered + as companion descriptors to a video payload, not standalone media. + `wai.video.lcevc` was registered companion-only, with its base codecs, in the + same revision as these rules. Every other registered capability is standalone + media. +- **the definition of a companion.** When this section introduced + `role = "companion"`, it defined a companion as a component applied with the + chosen rendition and naming, in `companion_of`, the component it binds to. + +Before these rules were stated, the text rejected none of the envelopes below, +and the reference implementation had no check that reported them (its `pick` +and `primary` could select an HDR descriptor). These do not conform: + +- a `WAI1` that names an HDR descriptor as its `capability` or `fallback`, + carrying it as standalone media (§5); +- a `WAI2` that carries an HDR descriptor as a selectable rendition instead of + with `role = "companion"`. `pick` and `primary` no longer select it + (*Selection*, above); +- a `WAI2` that binds an HDR descriptor to a registered capability outside the + `video` media class (rule 2) — an audio, image or text payload, or any other + registered capability rule 2 does not list, a volumetric or world payload + among them; +- a `WAI2` that marks a registered standalone capability as a companion + (rule 1). This includes `wai.audio.iamf` carried as immersive-audio mix + metadata: the earlier text of this section named immersive-audio metadata + beside the HDR descriptors as a payload that accompanies another, and a + reference test carried `wai.audio.iamf` that way. That use did not follow + the registration — `wai.audio.iamf` was registered as an immersive-audio + carrier, standalone media, and no capability for mix metadata alone is + registered — and rule 1 follows the registration; +- a `WAI2` with a companion, of any capability — one this revision does not + register included — that has no `companion_of`, or whose `companion_of` + names no selectable rendition in the envelope (rules 2 and 3). + +The first four enforce the registrations; the last enforces the definition of +a companion. Each still parses, and §5 and the rules above state what a sink +does with it. No earlier revision registered `wai.video.lcevc`; this one +registers it with these rules. An envelope that used that string while no +revision registered it is covered by §8's rule for a new registration, as is an +envelope that uses any string a later revision registers. + --- ## 8. Versioning @@ -882,7 +1173,43 @@ shipping multiple renditions per file. - New capabilities (§5) MAY be added in any MINOR revision. Existing capabilities MUST NOT be redefined — once a capability string is registered for a codec, its payload format is fixed forever. -- Removing or repurposing an existing capability requires a new MAJOR. +- A registration also fixes the capability's **role** (§5, §7.1): standalone + media, or companion-only — applied with a base rendition, never on its own. + Changing a capability's role, or narrowing the renditions a companion-only + capability applies to, redefines it. Widening them does not: no conforming + envelope stops conforming, and a reader of an earlier revision that reports + the new binding as not conforming does not apply that companion and can + still present its base. +- Removing or repurposing an existing capability — changing its payload format + or its role — requires a new MAJOR. +- A MINOR revision MAY add a conformance rule that enforces a registration, + or the definition of a manifest field, as it was made, changing no payload + format and no role. Such a rule makes non-conforming only envelopes that used + a capability against its registration or a field against its definition, + and the revision lists every kind of envelope it makes non-conforming; they + still parse. The companion rules of §5 and §7.1 are such rules, and §7.1 + lists the envelopes they affect (*Envelopes the companion rules make + non-conforming*). +- Registering a capability gives a string a payload format and a role that no + earlier revision gave it, and the rules of §5 and §7.1 then apply to it. An + envelope that used the string before then used a capability no revision + defined, and no rule checked that use against a registration it did not yet + have: §7.1 constrains an HDR descriptor's base only when the base's + capability is registered, and holds a companion of an unregistered + capability only to naming a base that is present (rules 2 and 3). Leaving a + use unchecked does not make it conform to a registration made later, so the + registering revision MAY make such an envelope non-conforming where it uses + the string against the new registration. This narrows nothing a registration + fixed and is not a redefinition. The kinds of envelope are listed here, once, + for every registration, so a registering revision need not list them again: + registering a standalone capability makes non-conforming a `WAI2` that + carried it as a companion (rule 1) and, when it is outside the `video` media + class, one that bound an HDR descriptor to it (rule 2); registering a + companion-only capability makes non-conforming an envelope that carried it as + a `WAI1` `capability` or `fallback` or as a selectable rendition, that bound + it to a base or a number of bases its registration does not admit, or that + bound another companion to it (no companion applies to a companion-only + capability). Such envelopes still parse. --- diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index baa58b2a6..137fb710a 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -483,7 +483,7 @@ day. | `float_repro`, the decode's `exp` and rounding: the same bits over a fixed grid of inputs; `exp` within 2 ULPs of the host math library (1 ULP measured); rounding equal to the standard library's | library unit tests under the library-test step of the conformance workflow (`neural_int`), which has not yet run them on a runner. The pinned-bits and rounding tests never skip. The host comparison asserts the host library's accuracy as much as this module's. Its gate selects by target, not by host: it runs on macOS arm64, on Linux x86_64 and arm64 with glibc, and on WASI, and is reported as ignored, with the reason, elsewhere, Windows included. On the source of the change that added this row it was run against these hosts' `exp`, with a worst case of 1 ULP on each: Apple's math library on macOS arm64; the WASI C library under wasmtime; glibc 2.36 on Linux arm64, and glibc 2.35 in the Ubuntu 22.04 image the workflow's Linux arm64 runner used on 2026-09-28; glibc 2.36 on Linux x86_64 under user-mode emulation, against three of the four `exp` variants glibc chooses among by processor feature (the variant each run used follows from the features glibc's loader reported active): SSE2 (QEMU 7.0, whose emulated processor reports none of AVX, FMA, AVX2 and FMA4), AVX (QEMU 7.2, `-cpu max` with FMA and AVX2 masked by glibc's `hwcaps` tunable, which leaves AVX active) and FMA (QEMU 7.2, `-cpu max`). On a separate probe of 20 million inputs, the SSE2 and AVX variants returned the same bits, and the FMA variant different bits for some of them. Not compared: the FMA4 variant, which glibc chooses on a processor that has FMA4 but not both FMA and AVX2, and which neither emulator offers. Not run on x86_64 hardware or on other glibc versions, where the gate still runs it; a failure there would say that the host's `exp` and this one are more than 2 ULPs apart, not that `float_repro`'s bits changed | `wai-rs/src/float_repro.rs` tests | | The exact decode path calls no platform math library | two checks. Clippy's `disallowed_methods` (`wai-rs/clippy.toml`), denied in each module on the decode path, rejects any use of a listed float method, including `f64::exp(x)` and `.map(f64::exp)`. `tests/jpegai_decode_symbols.rs` lists the symbols of an executable that links only the decoder's entry points and weight loaders. It rejects any C math-library function the executable imports, and any it defines other than the round-to-integral functions (`floor`, `round`, `rint` and the like), whose results IEEE 754 fixes. It allows those definitions because a definition shows what the linker kept, not what the decode calls: linked with GNU ld 2.40 for x86_64 Linux, the debug executable defined the compiler builtins' `floor`, though nothing in it referred to `floor`. On the source of the change that added this row, both checks pass on macOS arm64, the symbol check in debug and release builds. The symbol check also passes, in debug builds, on Linux arm64 (the container above) and on Linux x86_64 under user-mode emulation, linked with GNU ld (reporting that `floor` as allowed) and with LLD. With a platform `exp` put back as `f64::exp(…)`, both fail on macOS arm64 while the decoder's unit tests still pass, and the symbol check fails on Linux x86_64 with GNU ld. With a float `%` put into the decode, the symbol check fails on macOS arm64 and clippy, which does not see operators, passes. The conformance workflow runs both on its unix legs; neither has run on a runner yet | `wai-rs/clippy.toml`, `wai-rs/tests/jpegai_decode_symbols.rs`, `.github/workflows/wai-conformance.yml` | | `wai.image.jpegai` scope: the reference codestream is accepted; copies with β, `model_id`, the colour transform or an LSBS flag changed are refused; a flush-to-zero or round-toward-zero environment is detected. The float ONNX path calls the same check | library unit tests over a committed copy of the reference codestream, which never skip and fall under the library-test step of the conformance workflow (`neural_int`); the environment test runs on arm64 only. Run on the four platforms above, on the source of the change that added the cross-platform row (the environment test on macOS arm64 and Linux arm64). The float path's refusal (β and `model_id` edits) is tested only with its ONNX bundle supplied (`jpegai_scope_ort.rs`, skips otherwise) | `wai-rs/src/jpegai_decode.rs` tests, `wai-rs/tests/vectors/jpegai/` | -| SPEC §7's replicate rule: the registry class decides first, so an integer-declared prior over `wai.image.jpegai` or a float codec is refused; `wai.image.jpegai` is the only `NeuralEntropyExact` capability; the standard codecs sit in the classes SPEC §7 lists, and the tolerance and unclassified ones are not refused; an encoder that pins no prior (`wai wrap`) writes `replicate` only over the standard classes, for the capability and for any declared fallback | library unit tests (`codecs`, `container`) that never skip and fall under the same library-test step. The strict-decode test in `tests/neural_native_decode.rs` needs the `neural` feature; `wai-neural-gates.yml` runs it on the Linux runner, in an Ubuntu 24.04 image because the `neural` test binary does not link in that runner's default 22.04 image. The workflow has not run through the forge; its job passed under `forgejo-runner exec` on the Linux runner's host (Linux x86_64, 2026-09-27). The test that runs the `wai wrap` binary needs the `codecs` feature, which no CI job builds. Otherwise run so far on macOS arm64 only | `wai-rs/src/codecs/mod.rs`, `container.rs` tests; `wai-rs/tests/neural_native_decode.rs`, `cli_wrap_intent.rs` | +| SPEC §7's replicate rule: the registry class decides first, so an integer-declared prior over `wai.image.jpegai` or a float codec is refused; `wai.image.jpegai` is the only `NeuralEntropyExact` capability; the standard codecs sit in the classes SPEC §7 lists, and the tolerance and unclassified ones are not refused; an encoder that pins no prior (`wai wrap`) writes `replicate` only over the standard classes, for the capability and for any declared fallback, or, in the `WAI2` companion form, for the base payload's capability and each companion's | library unit tests (`codecs`, `container`) that never skip and fall under the same library-test step. The strict-decode test in `tests/neural_native_decode.rs` needs the `neural` feature; `wai-neural-gates.yml` runs it on the Linux runner, in an Ubuntu 24.04 image because the `neural` test binary does not link in that runner's default 22.04 image. The workflow has not run through the forge; its job passed under `forgejo-runner exec` on the Linux runner's host (Linux x86_64, 2026-09-27). The test that runs the `wai wrap` binary needs the `codecs` feature, which no CI job builds. Otherwise run so far on macOS arm64 only | `wai-rs/src/codecs/mod.rs`, `container.rs` tests; `wai-rs/tests/neural_native_decode.rs`, `cli_wrap_intent.rs` | | `wai.neural.int_hyper` and `wai.video.int_hyper` at full model scale | tests that skip when the exported model is absent; not run in CI | `wai-rs/tests/int_hyper_synth.rs`, `int_video_sota_keyframe.rs` | | `wai.audio.int_codec`, `wai.splat.int_codec` | committed vectors that never skip, matched against an independent Python integer reference; their cargo features (`audio_int`, `splat_int`) are enabled by no CI job | `wai-rs/tests/int_audio.rs`, `int_splat.rs` | diff --git a/wai/extensions/spatial-audio.md b/wai/extensions/spatial-audio.md index 040894ae2..4358c1389 100644 --- a/wai/extensions/spatial-audio.md +++ b/wai/extensions/spatial-audio.md @@ -118,7 +118,13 @@ only as far as that PCM is. A source whose capability promises the same bytes on every sink (`wai.audio.flac`, an integer codec) keeps it portable; a source decoded through a codec whose standard conforms decoders to a tolerance (`wai.audio.opus`, SPEC §7, `StandardTolerance`) may decode differently on two conforming sinks, and -then so does the mixdown. +then so does the mixdown. The reference registry reports this: +`codecs::is_cross_decoder_exact("wai.audio.scene")` is false, because a scene's +payload names its sources by hash and not by capability, and +`codecs::is_cross_decoder_exact_with_sources` is true once the capability of +every source envelope is known and each promises the same bytes on every sink. It +answers false when it cannot tell: when no source is given (a scene with no +objects included), or when a source is not a `wai.audio.*` capability (§2.2). The `stereo` floor is mandatory and bit-exact. `binaural@` is perceptual (conformance is similarity, not bytes — a sink advertising diff --git a/wai/research/2030-transport-landscape.md b/wai/research/2030-transport-landscape.md index 057e9acd9..2a8e679e9 100644 --- a/wai/research/2030-transport-landscape.md +++ b/wai/research/2030-transport-landscape.md @@ -965,8 +965,11 @@ sits. It is the strongest evidence in this survey for §1.5. **Correct ingest.** A **typed receipt class** in the `jwp-receipts.md` envelope (which already has kinds and parent linkage), plus a **companion-rendition -descriptor** for the declaration fragment — the slot HDR10+ and IAMF mix metadata -already use. Not a new receipt format. +descriptor** for the declaration fragment — the slot HDR10+ metadata already +uses. A companion's capability is companion-only (SPEC §7.1, rule 1), and a +sink applies only one it recognises, so the fragment would need a companion-only +capability registered for it; IAMF is standalone media and rides as a +rendition. Not a new receipt format. ### 7.3 Regional items examined and rejected diff --git a/wai/research/media-codec-registry.md b/wai/research/media-codec-registry.md index b4be51310..b62b66ab0 100644 --- a/wai/research/media-codec-registry.md +++ b/wai/research/media-codec-registry.md @@ -1,9 +1,10 @@ -# Media-Codec Registry — immersive audio, cinema, volumetric, haptics, HDR +# Media-Codec Registry — immersive audio, cinema, volumetric, haptics, HDR, broadcast and streaming > Reference for the capabilities WAI dispatches to beyond the flat-media floor > (PNG/FLAC/zstd) and modern menu (AVIF/JXL/Opus/AV1) and the neural bench. > These cover object/scene audio, the theatrical mezzanine path, 3D-geometry -> compression, the standardized haptics codec, and HDR dynamic metadata. +> compression, the standardized haptics codec, HDR dynamic metadata, and the +> broadcast and streaming video / audio bitstreams distribution chains carry. > > **Registration ≠ a bundled decoder.** As everywhere in WAI, registering a > capability fixes the *capability string* and says "payload = the named @@ -14,8 +15,11 @@ > honestly, and never make a non-floor capability a default. Royalty key: **RF** = royalty-free · **PROP** = proprietary license · **POOL** -= patent-pool / SEP royalties apply. Status verified against primary sources -**June 2026** (see footnotes for residual caveats). += patent-pool / SEP royalties apply. Status checked against primary sources +**June 2026**, and **September 2026** for the broadcast and streaming rows. The +footnotes give each row's basis and residual caveats; two broadcast rows are +classifications rather than verified statuses — AVS3 (¹⁰: the fee schedule is not +published) and EVC Baseline (⁹: royalty-free by design, with no formal guarantee). ## License classification is enforceable, not just documented @@ -31,6 +35,89 @@ That is the *per-title toll-road* choice made into a property of the format rather than a slide. `wai capabilities` prints the class per row (the registry reports the royalty-free count outright). +The classification is **exhaustive, with no default class**. Every registered +capability sits under an explicit arm of `capability_license`; one registered +without a class returns `None`, and `licence_classification_is_exhaustive_with_no_default` +fails. That test also pins the full non-royalty-free set, so adding or +reclassifying a pool / proprietary / non-commercial capability is a reviewed +diff. (Until September 2026 the function ended in a royalty-free fall-through, +so a newly registered pool codec would have been classified royalty-free by +omission.) `defaults_are_royalty_free_and_never_pool_or_proprietary` holds the +other half of the guarantee: the mandatory floor and the recommended set +(`codecs::mandatory_floor()` / `recommended_set()`, equal to the native sink +set) are all royalty-free, and no pool, proprietary or non-commercial +capability is among them. + +## Replicate class: which reconstructions are byte-identical across decoders + +`codecs::capability_replicate_class(cap)` answers a second registry question: +what does every conforming sink reconstruct? For a standard codec the answer is +what its named standard's decoder conformance fixes, not anything WAI adds; the +per-standard reading is in *Decoder conformance* below. Two rules keep +the answer honest: + +- **Exhaustive, with no default.** Every registered capability sits under an + explicit arm; one registered without a class returns `None`, and the registry + tests reject it. Until September 2026 the function ended in a + `StandardDefined` fall-through, which gave AAC, Opus, JPEG and the other + tolerance-conformance codecs — and `wai.mesh.glb` and `wai.image.jpegai` — a + same-bytes guarantee nothing had established. +- **An exact class needs recorded evidence.** `StandardDefined` promises the same + bytes on every conforming sink, so a capability enters it only with a basis: a + conformance clause that requires identical output, or the WAI extension whose + `*-equivalence` pins the reconstruction. The registry tests pin one basis per + member (`STANDARD_DEFINED_EVIDENCE`), and a `StandardDefined` capability without + one fails `standard_defined_rests_on_recorded_evidence`. Without that evidence a + standard codec is `StandardUnclassified`, the conservative class, which + promises no byte identity. + +`is_cross_decoder_exact(cap)` is true for the classes whose +`replicate_promise` is `ByteExact` (`StandardDefined`, `NeuralIntegerExact`), +except a capability whose output is computed from other envelopes' decoded +output (`composes_decoded_sources`: `wai.audio.scene`, whose mixdown is exact +given its sources' PCM). It is the one-call audit for "a hash over the decoded +output verifies at another conforming sink", answered from the capability +alone; for a scene, `is_cross_decoder_exact_with_sources` answers once its +sources' capabilities are known, and answers false when it cannot tell (no source +given, or a source outside `wai.audio.*`). It covers the output the standard or extension specifies — for a video +codec the cropped decoded pictures, before colour conversion — not a conversion a +sink applies afterwards. + +**Measured** with `tools/wai_replicate_probe.sh` on an Apple M5 Max (arm64, +macOS 27.0) with ffmpeg 9.0.2, September 2026. Each format was encoded from a +synthetic source: 20 frames of 320×240 `testsrc2` video, a 1 s 440 Hz 48 kHz +mono tone, and a 320×240 grayscale still, so that no colour conversion enters +the comparison. Each was decoded by every independent decoder present, and +the raw outputs were compared byte for byte. VideoToolbox decodes were forced +onto hardware surfaces; a software-only format under the same flags fails +rather than falling back. + +| capability | decoders | result | +|---|---|---| +| `wai.video.avc` (libx264, Annex B) | ffmpeg `h264` · VideoToolbox hardware | identical | +| `wai.video.hevc` (libx265, Annex B) | ffmpeg `hevc` · VideoToolbox hardware | identical | +| `wai.video.vp9` (libvpx, IVF) | ffmpeg `vp9` · libvpx · VideoToolbox hardware | identical | +| `wai.video.av1` (SVT-AV1, IVF) | dav1d · VideoToolbox hardware | identical | +| `wai.audio.flac` | ffmpeg · libFLAC · Core Audio, against the source PCM | identical | +| `wai.audio.aac` (ffmpeg `aac`, ADTS) | ffmpeg `aac` · ffmpeg `aac_fixed` · AudioToolbox | differ: 238 of 49 152 samples (max 1 LSB) float vs fixed; 8 629 (max 23 LSB) against AudioToolbox | +| `wai.audio.opus` (libopus, 64 kb/s) | ffmpeg `opus` · libopus | differ: 13 of 48 000 samples, max 1 LSB | +| `wai.image.jpeg` (cjpeg q75, grayscale) | libjpeg-turbo accurate-integer IDCT · its float IDCT · ffmpeg `mjpeg` | differ: 169 and 673 of 76 800 samples, max 1 | +| `wai.image.jp2`, irreversible 9/7 (OpenJPEG, 20:1) | OpenJPEG · ffmpeg `jpeg2000` | differ: 67 of 76 800 samples, max 1 | +| `wai.image.jp2`, reversible 5/3 | OpenJPEG · ffmpeg `jpeg2000`, against the source | identical (the lossless path) | + +One stream per format on one host is a demonstration, not a survey, and it is not +the basis of any class: the classes rest on the conformance clauses in *Decoder +conformance* below. The measurement is a falsification check. An identical result on a +`StandardDefined` row is what its standard requires; a difference there would +contradict the class, and the probe exits non-zero (a perturbed hardware decode +run through the probe does exactly that). A `StandardTolerance` or +`StandardUnclassified` row may match on a given stream; the class permits it. +AV1 was measured identical here, but it is `StandardTolerance`: its +specification requires identical output only from streams that signal no film +grain. AAC is `StandardUnclassified`: its decoders differ here, and its +conformance document (ISO/IEC 14496-26) was not read, so the registry does not +say whether that difference is within its criterion. + ## Immersive / object-based audio | capability | standard | royalty | reference | tier | @@ -132,26 +219,38 @@ resolve in SPEC before an integration partner's engineers ask.** ## Decoder conformance — what each named standard fixes -SPEC §7 classes every standard codec — the floor and modern menu as well as the -capabilities below — by what its named standard's decoder conformance fixes, +SPEC §7 classes every standard codec — the floor and modern menu, the broadcast +codecs, and the capabilities above and below — by what its named standard's +decoder conformance fixes, because that is what `intent = replicate` over it promises: the same bytes on every conforming sink (`StandardDefined`), a criterion around a reference (`StandardTolerance`), or not yet classified (`StandardUnclassified`). Checked 2026-09-27. The last column says what was read; a row is only as strong as that. +A row reaches `StandardDefined` only on a clause that requires identical output; +a standard whose conformance text was not read stays `StandardUnclassified`. | capability | class | what the standard fixes | read | |---|---|---|---| | `wai.image.png` | `StandardDefined` | lossless: "All PNG filters are strictly lossless", and the Paeth predictor's calculations "must be performed exactly". Reducing to a lesser sample depth is the decoder's choice: §10.4 gives a rounding formula and a right shift | RFC 2083, full text | | `wai.audio.flac` | `StandardDefined` | lossless: decompressing "returns exactly the original data" (§3); STREAMINFO carries an MD5 of the unencoded audio | RFC 9639, definitions and STREAMINFO table | | `wai.text.zstd`, `wai.text.xz` | `StandardDefined` | lossless byte formats | not re-read | +| `wai.video.avc` | `StandardDefined` | §C.4: test bitstreams go to the HRD and the decoder under test; "for each picture output by the HRD, the values of all samples that are output by the DUT for the corresponding picture shall be equal to the values of the samples output by the HRD". Film grain characteristics are an SEI message, outside the decoding process | Rec. ITU-T H.264 (V16) (06/2026), §C.4 | +| `wai.video.hevc`, `wai.video.vvc` | `StandardDefined` | §C.5.1: "for each such cropped decoded picture output by the DUT, the values of all samples that are output shall be equal to the values of the samples produced by the specified decoding process". Film grain characteristics (and, in H.266, other post-processing hints) are SEI messages, outside the decoding process | Rec. ITU-T H.265 (V11) (01/2026) and H.266 (V4) (01/2026), §C.5.1 | +| `wai.video.vp9` | `StandardDefined` | §8.1: "Decoders shall produce output frames that are identical in all respects and have the same output order as those produced by the decoding process specified herein" | VP9 Bitstream & Decoding Process Specification v0.6 (31 March 2016), §8.1 | | `wai.audio.opus` | `StandardTolerance` | §6: conformance is "defined through the reference implementation"; a decoder's output "MUST also be within the thresholds specified by the opus_compare.c tool" against the reference on the test vectors, with "the same final range decoder state". §6.1: quality 0 is the passing threshold, calibrated to additive white noise at 48 dB SNR; 100 means identical to the reference. §1.1: "Various operations in the codec require bit-exact fixed-point behavior"; §4.2.7.9: the rest of SILK frame reconstruction "does not need to be bit-exact" | RFC 6716 §1.1, §4.2.7.9, §6–§6.1 | | `wai.image.jpeg` | `StandardTolerance` | §7: a decoder shall "with appropriate accuracy, convert to reconstructed image data"; A.3.3 NOTE: the IDCT equations "contain terms which cannot be represented with perfect accuracy by any real implementation", and the accuracy of dequantisation + IDCT is specified in Part 2; §4.1: colour space "is outside the scope of this Specification". The lossless process (Annex H) is exact, but the capability's role is the DCT processes | ITU-T T.81 (1992), full text. Part 2 (ITU-T T.83 \| ISO/IEC 10918-2) not read: the download failed | | `wai.image.jxl` | `StandardTolerance` | §4.1.1 NOTE: JPEG XL specifies infinite-precision arithmetic, and "this document therefore specifies tolerances for the decoded samples"; §4.2: a decoded image is similar to the reference iff every sample is within a peak-error threshold and each channel's RMSE within an RMSE threshold; §4.3: thresholds per test (`peak_error`, `rms_error`) | ISO/IEC 18181-3:2022, public preview (clauses 1–4.3) | | `wai.image.jp2`, `wai.video.j2k` | `StandardTolerance` | decoder compliance compares decoded components with reference components and their error metrics with specified values (Annex B.2.4, B.2.5; "Tolerances", G.4.3; MSE among the abbreviations; a "9-7I precision guarantee", A.1.7); the reversible 5-3 path is lossless (definitions 3.41, 3.58) | ISO/IEC 15444-4:2024 \| ITU-T T.803, public preview: table of contents, terms and abbreviations only. The clauses that state the metrics and thresholds were not in the preview | | `wai.video.av1`, `wai.video.av1.lossless`, `wai.image.avif` | `StandardTolerance` | §7.2: without film grain, decoders "shall produce output frames that are identical in all respects" to the decoding process's; with film grain, a decoder may instead produce identical frames before grain synthesis and output frames that "do not have perceptually significant differences" from the reference grain synthesis; "The definition of “perceptually significant differences” is beyond the scope of this specification and may be specified, for example, by a service provider as part of their accreditation program", so this criterion is not a stated bound. The film-grain syntax does not depend on lossless coding, so `av1.lossless` is in the same class. A text search of AVIF v1.2.0 found no film-grain restriction | AV1 Bitstream & Decoding Process Specification v1.0.0 with Errata 1, §7.2 and §5.9.30; AVIF v1.2.0, text search | +| `wai.video.av2` | `StandardTolerance` | §7.1: with `film_grain_params_present` equal to 0, output frames "identical in all respects" to the decoding process's; equal to 1, a conformant decoder satisfies one of two options — the exact reference film grain synthesis (§7.21.7), or identical intermediate frames (§7.21.2) and output frames that "do not have perceptually significant differences" from the reference grain synthesis, a term the specification leaves to others to define. Annex A.5 (decoder conformance) adds only level capability and the display schedule | AV2 Bitstream & Decoding Process Specification v1.0.0 (28 May 2026), §7.1 and Annex A.5 | +| `wai.audio.ivas` | `StandardTolerance` | §7.1: bit-exact conformance, output matching the reference test sequences, for every implementation; §7.2.1: an implementation "based on floating–point code (TS 26.258)" whose output is not bit-exact is tested instead by a non-bit-exact procedure — TS 26.444's for mono operation (§7.2.1.1), and for stereo and immersive operation a per-frame maximum-loudness-difference corridor derived from reference builds that "are not bit-exact with each other nor with the reference implementation" (§7.2.1.2, Annex A) | 3GPP TS 26.252 V19.2.0 (2026-06), §7 and Annex A. TS 26.444 not read | | `wai.audio.iamf` | `StandardTolerance` | substreams are Opus, AAC-LC, FLAC or LPCM (`codec_id`); "specific renderers are out of scope" (with reference to §2.2) | IAMF working draft of 21 April 2025, text search | -| `wai.audio.ac4`, `wai.audio.ivas`, `wai.mesh.draco`, `wai.pc.gpcc`, `wai.pc.vpcc`, `wai.splat.sog`, `wai.haptic.mihs`, `wai.meta.hdr10plus`, `wai.meta.dovi` | `StandardUnclassified` | not checked | nothing read for this purpose. (`wai.haptic.mihs`'s registry comment says ISO/IEC 23090-33 defines bit-exact decoder verification; that was not checked here) | -| `wai.mesh.glb`, `wai.splat.spz` | `StandardUnclassified` | the payload bytes pass through intact; a sink's decode of them is not classified (glTF can embed JPEG images and compressed geometry; SPZ stores quantised attributes) | not read | +| `wai.video.evc`, `wai.video.evc.baseline`, `wai.video.avs3`, `wai.audio.mpegh`, `wai.audio.avs3` | `StandardUnclassified` | not checked | nothing read for this purpose: ISO/IEC 23094-1 and 23008-3 and their conformance parts are not freely available, and T/AI 109.2 / 109.3 were not obtained | +| `wai.video.lcevc` | `StandardUnclassified` | not checked; its decoding process includes a signalled dither step (clause 8.11, "Decoding process for dither filter"), so identical output across decoders is not presumed | ISO/IEC 23094-2:2021 public preview sample (front matter to clause 5.8): table of contents and clause 3 only. The conformance part (ISO/IEC 23094-3) not read | +| `wai.audio.aac` | `StandardUnclassified` | not checked. Three decoders on one host gave three different PCM streams from one ADTS stream (measured above), so deployed decoders do not agree byte for byte; whether ISO/IEC 14496-26 admits that, or makes some of them non-conforming, was not established | nothing read for this purpose | +| `wai.audio.ac4`, `wai.mesh.draco`, `wai.pc.gpcc`, `wai.pc.vpcc`, `wai.splat.sog`, `wai.haptic.mihs`, `wai.meta.hdr10plus`, `wai.meta.dovi` | `StandardUnclassified` | not checked | nothing read for this purpose. (`wai.haptic.mihs`'s registry comment and SPEC §5 say ISO/IEC 23090-33 defines bit-exact decoder verification; that rests on a summary, not the standard text, and was not checked here) | +| `wai.mesh.glb` | `StandardUnclassified` | the payload bytes pass through intact; what a sink decodes from them is not classified. A glTF binary carries more than raw buffers: glTF 2.0 images may be PNG or JPEG ("JPEG images MUST be compatible with JPEG File Interchange Format"), embedded through a `bufferView`, and geometry may be compressed through extensions (`KHR_draco_mesh_compression`; the vendor extension `EXT_meshopt_compression`). JPEG is `StandardTolerance` and Draco `StandardUnclassified` above, so a scene built from those can differ between conforming sinks; and glTF specifies an asset format, not a decoder conformance over rendered output (edge appearance, for one, "MAY be subject to implementation-specific techniques") | glTF 2.0.1 specification (Khronos registry HTML), text search; Khronos glTF extension registry README | +| `wai.splat.spz` | `StandardUnclassified` | the payload bytes pass through intact; SPZ stores quantised attributes, and its decode is not classified | not read | Two limits apply to every row. The standard fixes its decoded output in its own domain (sample depth, planes, colour representation); a conversion a sink applies @@ -183,6 +282,11 @@ was read is an overview of the standard (arXiv 2510.13867, read 2026-09-27): Read together, these passages point to a criterion around a reference on the reconstruction — the kind `StandardTolerance` records — but the overview does not state one, nor its form or width. +A measurement of one decoder does not bear on this class. WAI's exact JPEG AI +decoder pins one reconstruction across machines under the conditions +[determinism-tiers](../extensions/determinism-tiers.md) states; that shows one +implementation is reproducible, and says nothing about what another conforming +decoder outputs, which is what the class records. Part 4 itself was not read. A web search on 2026-09-27 returned the ISO catalogue entry for ISO/IEC 6048-4 as under publication (stage 60.00); the catalogue page refused a direct fetch (HTTP 403), so that status is the search result's, not @@ -190,7 +294,201 @@ confirmed here. Keeping `wai.image.jpegai` at `NeuralEntropyExact`, where `replicate` is refused, is therefore a conservative choice pending Part 4, not a reading of the standard; if Part 4 states a criterion on the reconstruction, the capability is a candidate for `StandardTolerance` (SPEC §7, *`wai.image.jpegai` -and the standard classes*). +and the standard classes*). Only a Part 4 that required a bit-exact +reconstruction could support a same-bytes class, and none of the text read +suggests it does. + +What would upgrade each conservative row: for a `StandardUnclassified` standard +codec, its conformance clause, read, requiring identical output (then +`StandardDefined`) or stating a criterion (then `StandardTolerance`). For +`wai.mesh.glb`, either a conformance document for glTF decoding, or a narrower +capability whose payload is restricted to uncompressed buffers with no embedded +images, whose reconstruction could then be defined as the parsed buffers — a new +capability string, since §8 fixes this one's payload. + +## Broadcast and streaming video / audio + +The elementary bitstreams terrestrial and satellite broadcast, streaming and +device playback carry. Registered so an envelope can *name* the bitstream a +distribution chain already holds; all sink-supplied, none a default. + +| capability | standard | payload (canonical bytes) | royalty | reference / open decoder | tier | +|---|---|---|---|---|---| +| `wai.video.av2` | AV2 (AOMedia), spec v1.0.0, 28 May 2026 | AV2 Annex B length-delimited bitstream (`leb128` size before each OBU) | **RF**⁶ | AVM v1.0.0 | Lossy | +| `wai.video.vvc` | ITU-T H.266 \| ISO/IEC 23090-3 | H.266 Annex B byte stream | **POOL**⁷ | VTM | Lossy | +| `wai.video.hevc` | ITU-T H.265 \| ISO/IEC 23008-2 | H.265 Annex B byte stream | **POOL**⁷ | HM | Lossy | +| `wai.video.avc` | ITU-T H.264 \| ISO/IEC 14496-10 | H.264 Annex B byte stream | **POOL**⁷ | JM | Lossy | +| `wai.video.vp9` | VP9 Bitstream & Decoding Process Specification | IVF frame container, FourCC `VP90` | **RF**⁸ | libvpx | Lossy | +| `wai.video.evc` | MPEG-5 Part 1, ISO/IEC 23094-1 (any profile) | Annex B raw bitstream file storage format | **POOL**⁹ | XEVD | Lossy | +| `wai.video.evc.baseline` | ISO/IEC 23094-1, Baseline profile only | as above, Baseline-conformant | RF by design⁹ | XEVD | Lossy | +| `wai.video.avs3` | AVS3-P2: T/AI 109.2-2021; IEEE 1857.10 | AVS3-P2 elementary bitstream | **POOL**¹⁰ | uavs3d (BSD-3-Clause; Baseline profile) | Lossy | +| `wai.video.lcevc` | MPEG-5 Part 2, ISO/IEC 23094-2 | LCEVC enhancement as the ISO/IEC 23094-2 Annex B byte stream (start-code-delimited NAL units) — **companion-only** | **POOL**¹¹ | LCEVCdec (BSD-3-Clause-Clear, patent licences excluded) | Lossy | +| `wai.audio.mpegh` | MPEG-H 3D Audio, ISO/IEC 23008-3 | MHAS packet stream | **POOL**¹² | mpeghdec | Lossy | +| `wai.audio.avs3` | AVS3-P3: T/AI 109.3-2023; GY/T 363-2023 ("Audio Vivid"); GB/T 46271-2025 | AVS3-P3 elementary bitstream | **POOL**¹⁰ | — | Lossy | +| `wai.audio.aac` | ISO/IEC 14496-3: AAC-LC, HE-AAC, HE-AAC v2 | ADTS stream | **POOL**¹³ | — | Lossy | + +**Where these bitstreams are carried** (the reason for registering them, not a +ranking): Brazil's TV 3.0 / DTV+ — VVC video, LCEVC enhancement, MPEG-H 3D +Audio — launched by decree on 27 Aug 2025 and switched on in Rio de Janeiro, São +Paulo and Brasília in June 2026. DVB's video/audio coding specification (A001, +ETSI TS 101 154) added AVS3 video in 2022 and AVS3-P3 immersive audio in +November 2024 (BlueBook A001r22). China's GB/T 46271-2025 (3D audio coding, +distribution and presentation) was published 29 Aug 2025, in force 1 Mar 2026. +Japan's 4K/8K satellite broadcasting (on air since 1 Dec 2018) uses HEVC. AVC with AAC, and VP9, are +widely deployed in streaming and device playback. WAI registers LCEVC over +AVC, HEVC, VVC, EVC, VP9 and AV1, the base codecs a published description of +LCEVC gives as examples ("e.g., AVC, HEVC, VP9, AV1, EVC or VVC", +); with all six registered, every LCEVC +base an envelope names is itself a registered capability. The list is WAI's +registration, not a limit of the standard: ISO/IEC 23094-2:2021 defines the +decoded base picture as the output of "a base decoder" (definition 3.17), says +the base decoder's IDR picture "is not specified in this document" (3.30, Note +1), and its normative references (clause 2: ISO/IEC 11578, ITU-T H.273 | +ISO/IEC 23091-2, ITU-T T.35) include no video codec; these clauses are in the +public preview sample cited below. AV2, AVS3 video, JPEG 2000 and the +WAI-native integer codecs are not registered bases; a later revision may add +one, which SPEC §8 treats as a widening. + +**Payload framing is named per entry** because §8 fixes a capability's payload +format forever once registered, so each entry names one form. Where the +standard defines a self-delimiting elementary form the entry names it (EVC's +Annex B raw bitstream file storage format, MHAS for MPEG-H 3D Audio, ADTS for +AAC). H.264, H.265, H.266 and LCEVC each define a bitstream as either a NAL +unit stream or a byte stream (H.264 3.15, H.265 3.13, H.266 3.17, ISO/IEC +23094-2 3.2), and only the byte stream delimits its own NAL units, with the +three bytes `0x000001` "as a prefix to each NAL unit" (ISO/IEC 23094-2 3.51; +H.264 3.163, H.265 3.156 and H.266 3.138 say the same). So each of those pins +is that standard's Annex B byte stream, and a bare or length-prefixed NAL unit +stream is not the capability. For LCEVC this rests on ISO/IEC 23094-2:2021's +public preview sample, which runs from the front matter to clause 5.8; the +table of contents and clause 3 (definitions 3.1, 3.2, 3.6, 3.35, 3.36 and 3.51) +were read (, sample at +). +The table of contents lists "Annex B (normative) Byte stream format"; the +annex's body is not in the sample, so its syntax was not read. AV2 OBUs carry +no size of their own and the AV2 specification states that framing is outside +its scope *except* for the one method it defines, Annex B — so that is the pin. +VP9 frames are not self-delimiting and the VP9 specification defines no stream +framing, so the pin is IVF, the minimal frame container the VP9 reference +library (libvpx) reads and writes. The two AVS3 rows name the elementary +bitstream of T/AI 109.2 (video) and T/AI 109.3 (audio). Neither text was +obtained, so whether either standard defines a second form of its bitstream was +not checked; if one does, the row must name the form before §8 fixes the +string (open item 5(h)). + +**AV2 placement — registered, not recommended.** The recommended set is what +every conforming sink SHOULD implement, and WAI's value there is availability. +AV2 is royalty-free (⁶) and has a reference decoder, but as of September 2026 +AVM is a reference model, a production software decoder is still in +development, and hardware decode in shipping devices is not confirmed. Asking +every sink to carry it +now would ask for a decoder most cannot deploy. So it is registered (a WAI file +can carry AV2 today, royalty-free) and SPEC §5 tells encoders to ship it in a +`WAI2` envelope with a `wai.video.av1` rendition of the same content. A v1.0 +`fallback` of `wai.video.av1` would not do: dispatch step 5 hands the *same* +AV2 payload to the fallback's decoder, which cannot read it. Promotion is a +MINOR revision once deployable decode exists; the capability string and +framing stay fixed. +`defaults_are_royalty_free_and_never_pool_or_proprietary` asserts it is RF and +not (yet) a default. + +**LCEVC and the companion mechanism — the fit, and where it stops.** LCEVC is an +enhancement layer: residuals added to the upscaled output of one specific base +encode. SPEC §7.1's companion components fit most of it: + +- *Applied with, never instead of.* A companion is never returned by + `pick` / `primary`, so a sink with no LCEVC decoder presents the base — the + backward-compatible behaviour LCEVC is designed for (TV 3.0 uses it to carry a + UHD service on top of a usable HD one). +- *Intrinsic binding.* Base and enhancement ride in one envelope, not a sidecar. +- *One base.* ISO/IEC 14496-15's dual-track storage has the enhancement track + refer to exactly one base track; `companion_of` is the analogue. + +Two gaps, one closed here and one not: + +1. **Binding granularity (closed by a rule in both directions).** + `companion_of` names a *capability*, not a component. HDR metadata describes + content and can legitimately apply to several renditions of one codec, and + several descriptors can apply to one base; LCEVC residuals belong to one + encode and are added once. So an enhancement layer's `companion_of` MUST + name one of its base codecs — the capabilities that name the AVC, HEVC, VVC, + EVC, VP9 and AV1 bitstreams, EVC Baseline and lossless AV1 included — and + resolve to exactly one selectable rendition of it, and that base takes at + most one enhancement layer — a sink applies every companion of the rendition + it picks, so a second one would stack a second set of residuals. + `WaiMulti::check_component_roles` rejects a base outside that list (JPEG + 2000, AV2, AVS3, the WAI-native integer codecs, anything not video), zero or + several renditions of the named base, a second enhancement on one base, and + a companion-only capability mislabelled as a rendition (which `pick` / + `primary` also never select). A component-id binding would lift the + one-base-rendition limit; it is not part of this change. + The v1.0 side has its own check: `Wai::check_component_roles` rejects a + companion-only capability as a `WAI1` `capability` or `fallback`, and the + `wai wrap` tool refuses to write either (with `--companion` it writes the + conforming `WAI2` form instead). +2. **Per-picture timing (open).** A timed container aligns base and enhancement + samples by timestamp; a `WAI2` payload is an untimed elementary stream. The + payload's framing is pinned (the Annex B byte stream, above), and ISO/IEC + 23094-2 defines an access unit as the NAL units of one coded picture + "associated with a particular output time" (3.1), but this revision does + not pin how a separately carried enhancement's access units align picture + by picture with its base, so the **embedded** carriage — enhancement inside + the base bitstream as SEI / registered user data, carried under the base + capability, no separate component — is the unambiguous form until it does. + +**Not registered this round, and why.** + +- *E-AC-3 / AC-3.* Patent expiry is reported (AC-3 in 2017; the last E-AC-3 + patent reportedly in 2019) but not confirmed here, and a licensing programme + still exists. Registering needs a verified class first. +- *Extended HE-AAC (USAC).* ADTS cannot carry it; it needs LOAS/LATM or + ISOBMFF framing, i.e. a separate capability with its own pinned payload. +- *HDR Vivid.* The HDR dynamic-metadata companion published alongside Audio + Vivid in the same national-standard set; licence not verified. + +⁶ **Verified RF.** The AVM v1.0.0 reference software's `PATENTS` file is the +*Alliance for Open Media Patent License 1.0* ("no-charge, royalty-free, +irrevocable … patent license"); AOMedia's 9 June 2026 release states AV2 was +"developed under AOMedia's royalty-free patent policy". Residual caveat, the same +as for AV1 and VP9: a third-party licensing company announced (Dec 2025) an +intent to form a pool asserting AV2 patents, and on 25 Aug 2026 a second +licensing administrator launched an exploratory phase for a pool covering devices +and software implementing AV2 and AV1, saying it expects AV2 to be added to its +existing video-distribution pool licence (which covers VP9 and AV1) in 2027. WAI +classifies a format by its own licence and records such assertions here rather +than reclassifying on them. Sources: , +, +. +⁷ **Verified POOL.** VVC, HEVC and AVC essential patents are licensed for +royalties through patent pools (several for VVC and HEVC, the long-running +H.264 pool for AVC) plus bilateral licensors. None is royalty-free. +⁸ **RF.** VP9 is published with BSD-licensed source and a royalty-free patent +grant from its developer. Residual caveat as ⁶: a third-party pool asserts VP9 +patents; recorded, not adopted as the class. +⁹ **EVC split by profile.** ISO/IEC 23094-1's Baseline profile uses only tools +more than 20 years old or offered royalty-free, and its proponents describe it +as royalty-free; the Main profile adds tools whose holders license them on FRAND +terms. So `wai.video.evc` (any profile — a decoder must handle Main) is POOL and +`wai.video.evc.baseline` is RF *by design*. Residual caveat, like JPEG 2000 +Part 1 (³): no formal royalty-free guarantee exists for Baseline. +¹⁰ **Classified POOL.** The AVS patent pool issued calls for essential patents +for AVS3-P2 video and AVS3 audio on 11 Jan 2024, licensing on RAND terms with +"competitive licensing fees"; AVS licensing has historically charged devices +only. The fee schedule for AVS3 was not found published, but the pool is +royalty-bearing, not royalty-free. AVS3-P3 and Audio Vivid are treated as one +bitstream family (DVB and the open-source tooling name them together); the +bitstream identity of T/AI 109.3, GY/T 363 and GB/T 46271 was not checked +against the standard texts, which were not available. +¹¹ **Verified POOL (single SEP licensor).** LCEVC's essential patents are +licensed under programmes announced Aug 2025: royalties on consumer devices and +on video-distribution services, a free integration licence, and free-to-air +broadcast royalties waived. The open decoder library is BSD-3-Clause-Clear, +which expressly excludes patent licences. +¹² **Verified POOL.** MPEG-H 3D Audio essential patents are licensed through a +patent-pool programme. +¹³ **Verified POOL.** The AAC pool programme covers AAC-LC, HE-AAC, HE-AAC v2 and +Extended HE-AAC; a product shipping an AAC encoder or decoder pays a per-unit +royalty. ## Implementation & demonstration status @@ -201,9 +499,12 @@ floor codecs are *wrapped* reference libraries, not re-implemented). So - **Carriage proven for every registered class** — `wai wrap ` builds an envelope around the canonical bytes (over a class that does not back an unpinned `replicate`, only with `--intent`, SPEC §7); `wai unwrap` returns them - **byte-identically**; the registry reports the tier + license. The - `tests/carriage.rs` conformance asserts this for the *entire* registry (0 B … - 64 KiB payloads) and runs in the no-system-deps CI matrix. + **byte-identically**; the registry reports the tier + license. A + companion-only capability is carried as a `WAI2` companion instead + (`wai wrap --companion `, then + `wai unwrap --component 1`). The `tests/carriage.rs` conformance asserts this + for the *entire* registry (0 B … 64 KiB payloads), each capability in the + form SPEC §5 allows it, and runs in the no-system-deps CI matrix. - **Real end-to-end decode demonstrated** for the open formats with an installed/built reference tool — the full carry → dispatch → decode path on a real file: **JPEG 2000** (`wai.image.jp2`, OpenJPEG — cinema), **IAMF** @@ -231,6 +532,31 @@ floor codecs are *wrapped* reference libraries, not re-implemented). So `wai.haptic.mihs`, `wai.meta.hdr10plus`. - **Proprietary** (no open decoder, by design — carried, sink-supplied): `wai.audio.ac4`, `wai.meta.dovi`. +- **Companion-only** (`wai.meta.hdr10plus`, `wai.meta.dovi`, + `wai.video.lcevc`): carried in the demo the one conforming way, as a `WAI2` + companion bound to a base rendition (a real AVC Annex B stream when libx264 + is installed). Each row requires the companion back byte-exact, the role + check to pass, `wai wrap` to refuse the `WAI1` form, and the base to decode + to all 10 frames; the LCEVC row also requires a second enhancement on the + same base to be refused. The companion payloads are representative bytes. +- **Broadcast and streaming codecs.** Every row of that table is covered by + `tests/carriage.rs` (byte-exact wrap / unwrap, tier + licence present) and by + the registry tests in `codecs/mod.rs`; the LCEVC binding rules by the + `container.rs` unit tests. The crate gains no decoder and no C dependency: + every one is sink-supplied. **Real encode + decode demonstrated** for + `wai.video.avc`, `wai.video.hevc`, `wai.video.vp9` and `wai.audio.aac` in + `tools/wai_carriage_demo.sh` when ffmpeg has the encoders: the payload is + built in the framing SPEC §5 pins (Annex B byte stream, IVF, ADTS), carried + byte-exact, and decoded by ffmpeg with the matching raw demuxer *forced*. A + video row passes only if all 10 encoded frames come back; the AAC row only if + the decoded frame count equals the ADTS frame count of the encoded stream, + counted by walking its frame headers rather than by the decoder (a payload + cut to 600 B, or by its last byte, fails; random bytes under the forced + demuxers are rejected). **Carriage only** for `wai.video.vvc` (no VVC encoder + installed on the demo host), `wai.video.av2`, `wai.video.evc`, + `wai.video.evc.baseline`, `wai.video.avs3`, `wai.audio.mpegh` and + `wai.audio.avs3`. `wai.video.lcevc` is companion-only and is demonstrated as a + `WAI2` companion (above). The reference sink decoders for the volumetric + immersive-audio rows (MPEG `tmc13` for G-PCC, `tmc2` for V-PCC, 3GPP `IVAS_cod`/`IVAS_dec`) are fetched @@ -255,11 +581,20 @@ point, and the reason WAI does not need to re-implement any of them. 1. **Companion-component binding — DONE.** The `WAI2` multi-component envelope gains a component `role`: `"companion"` (with `companion_of`) marks a descriptor applied *with* the chosen rendition. HDR dynamic metadata - (`wai.meta.hdr10plus` / `wai.meta.dovi`) and immersive-audio metadata now - bind to a video/audio payload in the **same envelope** — `pick`/`primary` - skip companions, `companions_of(cap)` returns the descriptors for the chosen - component. Back-compat: `role` absent ⇒ rendition. SPEC §6.1 + a conformance - test (`companion_descriptors_bind_to_a_primary`). + (`wai.meta.hdr10plus` / `wai.meta.dovi`) binds to a video payload in the + **same envelope** — `pick`/`primary` skip companions, `companions_of(cap)` + returns the descriptors for the chosen component. Back-compat: `role` absent + ⇒ rendition. SPEC §7.1 + a conformance test + (`companion_descriptors_bind_to_a_primary`). Immersive-audio metadata was + named here too, and that test first carried an IAMF stream as a companion; + no capability for immersive-audio mix metadata is registered, and IAMF + carries its own audio substreams, so it is standalone media. Since + September 2026 SPEC §7.1 states three rules a companion must meet — its + capability is companion-only (or unregistered), `companion_of` names a base + it applies to (a `video`-class capability for HDR metadata — `wai.video.*`, + the neural video codecs, the films — and a base codec for LCEVC), and + that base is present — and `check_component_roles` enforces each, with a + negative test per rule. 2. **License classification — DONE.** `codecs::capability_license()` / `is_royalty_free()` classify every capability; `wai capabilities` reports them; a coherence test enforces that no capability is unclassified. Built as @@ -271,3 +606,47 @@ point, and the reason WAI does not need to re-implement any of them. confirmed pool; AC-4 / Dolby Vision → proprietary. Residual: JPEG 2000 extensions/submarine-patent caveat (³), MPEG-I Haptics pool status unconfirmed (⁵). +4. **Broadcast and streaming codecs — REGISTERED (September 2026).** AV2, VVC, + HEVC, AVC, VP9, EVC (any profile / Baseline), AVS3 video, LCEVC, MPEG-H 3D + Audio, AVS3 audio / Audio Vivid and AAC, each naming one payload framing + (LCEVC's is the ISO/IEC 23094-2 Annex B byte stream; the AVS3 rows' are + unconfirmed against their standards, (h)) and a licence class with its + basis (⁶–¹³). LCEVC is companion-only, binds to exactly one rendition of + one of its base codecs, and a base takes at most one enhancement; + companion-only capabilities are refused as a v1.0 `capability`, and as a + `fallback` wherever the fallback would be used (§4 step 5). Licence + classification made exhaustive (no default class), and so was the replicate + classification, with an exact class admitted only on recorded evidence. AVC, + HEVC, VVC and VP9 are `StandardDefined` on their conformance clauses; AV2 is + `StandardTolerance`, as AV1; the other broadcast rows are + `StandardUnclassified` (see *Decoder conformance* above). + Residual: EVC Baseline's RF-by-design status (⁹), unpublished AVS3 fee + schedule and unverified AVS3-P3 / Audio Vivid bitstream identity (¹⁰), + third-party pool assertions against AV2 / VP9 (⁶, ⁸). +5. **Open.** (a) Per-picture alignment of a separately carried LCEVC + enhancement with its base. Its framing is pinned (the Annex B byte stream), + but how its access units map to the base's pictures outside a timed + container is not; until it is, the embedded carriage is the unambiguous + form. (b) A component-id `companion_of` binding, which would let one + envelope carry enhancements for several renditions of one base codec. + (c) The neural capabilities' model-weight licences were carried over, not + re-verified, when the classification became explicit — the EnCodec 32 kHz + checkpoint is distributed with a model collection whose weights are licensed + CC-BY-NC 4.0, which would make `wai.neural.encodec32` non-commercial if it + applies. (d) E-AC-3 / AC-3, Extended HE-AAC and HDR Vivid (see above). + (e) The `StandardUnclassified` rows (EVC, AVS3 video and audio, LCEVC, + MPEG-H 3D Audio, AAC, AC-4, G-PCC, V-PCC, Draco, SOG, MIHS, the HDR + descriptors, glb, SPZ) are waiting on each standard's conformance text; any + whose clause requires identical decoded output moves to `StandardDefined`, + with that clause recorded, and any with a stated criterion to + `StandardTolerance`. (f) Closed: the spatial-audio extension now says a + mixdown is only as portable as its sources' decoded PCM, and + `is_cross_decoder_exact` no longer reports `wai.audio.scene` exact from the + capability alone (`is_cross_decoder_exact_with_sources` checks the sources). (g) Closed: + `wai.image.jpegai` is `NeuralEntropyExact` and does not back `replicate` + (SPEC §7); it is not cross-decoder-exact. A cross-platform measurement of + WAI's own decoder cannot change that, as *Decoder conformance* explains. + (h) AVS3 framing. `wai.video.avs3` and `wai.audio.avs3` name the elementary + bitstream of T/AI 109.2 and T/AI 109.3, which were not obtained. Before §8 + fixes the strings, confirm from those texts that each defines one form of + its bitstream, or name the form the capability carries. diff --git a/wai/tools/wai_carriage_demo.sh b/wai/tools/wai_carriage_demo.sh index 76e656794..940ed6105 100755 --- a/wai/tools/wai_carriage_demo.sh +++ b/wai/tools/wai_carriage_demo.sh @@ -11,7 +11,13 @@ # # Honest status per row: # real+decode = real canonical bytes, carried byte-exact, decoded by a real -# reference tool acting as the sink (JPEG 2000, IAMF). +# reference tool acting as the sink (JPEG 2000, IAMF, and the +# AVC / HEVC / VP9 / AAC broadcast rows via ffmpeg). +# companion = a companion-only capability (SPEC §5: HDR dynamic metadata, +# LCEVC) carried the one conforming way, as a WAI2 companion +# bound to a base rendition: the companion comes back +# byte-exact, the envelope passes the role check, `wai wrap` +# refuses the WAI1 form, and a real base decodes. # real+parse = real canonical bytes (a genuine encoder built the bitstream), # carried byte-exact, structurally decoded into the format's # units by a real reference parser as the sink (V-PCC → V3C). @@ -175,8 +181,112 @@ else rep "$TMP/vpcc" 4096; demo wai.pc.vpcc "$TMP/vpcc" "carriage"; fi rep "$TMP/sog" 4096; demo wai.splat.sog "$TMP/sog" "carriage" echo '{"version":1,"avatars":[]}' > "$TMP/h.hjif" demo wai.haptic.mihs "$TMP/h.hjif" "carriage" -rep "$TMP/hdr" 256; demo wai.meta.hdr10plus "$TMP/hdr" "carriage" -rep "$TMP/dovi" 256; demo wai.meta.dovi "$TMP/dovi" "proprietary" + +# ---- broadcast and streaming codecs ---- +# Where ffmpeg has an encoder, build the REAL payload in the framing SPEC §5 pins +# (Annex B byte stream / IVF / ADTS) and decode the carried bytes with the +# matching raw demuxer FORCED (-f h264 / hevc / ivf / aac), so the row shows a +# real decoder consumes exactly the pinned framing. The others carry a +# representative payload; their decoders are sink-supplied. +# The decode must yield every encoded frame, not merely exit 0: 10 video frames, +# and for AAC the frame count of the encoded ADTS stream, counted independently +# of the decoder by walking its frame headers. +# Encoder list captured once: `ffmpeg -encoders | grep -q` would SIGPIPE ffmpeg +# and trip `set -o pipefail` into a false "no encoder". +ENCODERS=""; have ffmpeg && ENCODERS="$(ffmpeg -hide_banner -encoders 2>/dev/null)" +has_enc() { grep -q " $1 " <<<"$ENCODERS"; } +# $1 capability $2 encoder $3 muxer (the pinned framing) $4 extension +bcast_video() { + local cap="$1" enc="$2" fmt="$3" ext="$4" + if has_enc "$enc"; then + ffmpeg -hide_banner -loglevel error -y -f lavfi -i "testsrc=size=128x96:rate=10:duration=1" \ + -c:v "$enc" -pix_fmt yuv420p -f "$fmt" "$TMP/b.$ext" >/dev/null 2>&1 + if [ -s "$TMP/b.$ext" ]; then + demo "$cap" "$TMP/b.$ext" "real+decode" \ + "ffmpeg -hide_banner -loglevel error -y -f $fmt -i $TMP/p.out -f framemd5 $TMP/d.md5 && [ \$(grep -vc '^#' $TMP/d.md5) -eq 10 ]" + return + fi + fi + rep "$TMP/b.$ext" 4096; demo "$cap" "$TMP/b.$ext" "carriage" +} +bcast_video wai.video.avc libx264 h264 h264 +bcast_video wai.video.hevc libx265 hevc hevc +bcast_video wai.video.vp9 libvpx-vp9 ivf ivf +# Count the ADTS frames of a stream by walking each header's frame_length. +# Prints nothing unless the walk lands exactly on the end of the file. +adts_frames() { + python3 - "$1" <<'ADTS' +import sys +b = open(sys.argv[1], 'rb').read(); i = n = 0 +while i + 7 <= len(b) and b[i] == 0xFF and (b[i+1] & 0xF6) == 0xF0: + fl = ((b[i+3] & 3) << 11) | (b[i+4] << 3) | (b[i+5] >> 5) + if fl < 7 or i + fl > len(b): break + i += fl; n += 1 +print(n if n > 0 and i == len(b) else "") +ADTS +} +if has_enc aac; then + ffmpeg -hide_banner -loglevel error -y -f lavfi -i "sine=frequency=440:duration=1" \ + -c:a aac -f adts "$TMP/b.aac" >/dev/null 2>&1 +fi +if [ -s "$TMP/b.aac" ]; then + AAC_FRAMES="$(adts_frames "$TMP/b.aac")" + demo wai.audio.aac "$TMP/b.aac" "real+decode" \ + "[ -n '$AAC_FRAMES' ] && ffmpeg -hide_banner -loglevel error -y -f aac -i $TMP/p.out -f framemd5 $TMP/d.md5 && [ \$(grep -vc '^#' $TMP/d.md5) -eq '$AAC_FRAMES' ]" +else rep "$TMP/b.aac" 2048; demo wai.audio.aac "$TMP/b.aac" "carriage"; fi +for cap in wai.video.av2 wai.video.vvc wai.video.evc wai.video.evc.baseline \ + wai.video.avs3 wai.audio.mpegh wai.audio.avs3; do + rep "$TMP/b.bin" 4096; demo "$cap" "$TMP/b.bin" "carriage" +done + +# ---- companion-only capabilities: HDR dynamic metadata, LCEVC (SPEC §5) ---- +# Their payload means nothing without its base, so the only conforming carriage +# is a WAI2 companion bound to a base rendition. Each row wraps base + companion, +# requires the role check to pass, unwraps the companion byte-exact, requires +# `wai wrap` to refuse the WAI1 form, and (for a real base) decodes the base — +# the picture a sink without the companion's decoder presents. The base is a +# real AVC Annex B stream when libx264 is installed. The companion payloads are +# representative: no open encoder for them is installed here. +BASE_CAP=wai.video.avc; BASE="$TMP/base.h264"; BASE_REAL="" +if has_enc libx264; then + ffmpeg -hide_banner -loglevel error -y -f lavfi -i "testsrc=size=128x96:rate=10:duration=1" \ + -c:v libx264 -pix_fmt yuv420p -f h264 "$BASE" >/dev/null 2>&1 + [ -s "$BASE" ] && BASE_REAL=1 +fi +[ -n "$BASE_REAL" ] || rep "$BASE" 4096 +# $1 companion capability $2 companion payload [$3 a second payload of the +# same capability, which must be refused when stacked on the same base] +demo_companion() { + local cap="$1" pay="$2" env="$TMP/c.wai" out="$TMP/c.out" ins + local tier; tier="$("$WAI" capabilities 2>/dev/null | awk -v c="$cap" '$1==c{print $2}')" + rm -f "$env" "$out" + "$WAI" wrap "$BASE_CAP" "$BASE" -o "$env" --companion "$cap" "$pay" >/dev/null 2>&1 + "$WAI" unwrap "$env" --component 1 -o "$out" >/dev/null 2>&1 + local rt="✗ MISMATCH" + if cmp -s "$pay" "$out"; then rt="✓ byte-exact"; fi + # capture first: `wai inspect | grep -q` would SIGPIPE wai under pipefail + ins="$("$WAI" inspect "$env" 2>/dev/null)" + if grep -q 'component roles: *✓' <<<"$ins"; then rt="$rt · roles ✓"; else rt="$rt · roles ✗"; fi + if "$WAI" wrap "$cap" "$pay" -o "$TMP/refused.wai" >/dev/null 2>&1; then + rt="$rt · WAI1 ✗ ACCEPTED"; else rt="$rt · WAI1 refused"; fi + if [ -n "${3:-}" ]; then + if "$WAI" wrap "$BASE_CAP" "$BASE" -o "$TMP/stacked.wai" --companion "$cap" "$pay" \ + --companion "$cap" "$3" >/dev/null 2>&1; then + rt="$rt · 2 on 1 base ✗ ACCEPTED"; else rt="$rt · 2 on 1 base refused"; fi + fi + if [ -n "$BASE_REAL" ]; then + rm -f "$TMP/base.out" "$TMP/base.md5" + "$WAI" unwrap "$env" -o "$TMP/base.out" >/dev/null 2>&1 + if ffmpeg -hide_banner -loglevel error -y -f h264 -i "$TMP/base.out" -f framemd5 "$TMP/base.md5" >/dev/null 2>&1 \ + && [ "$(grep -vc '^#' "$TMP/base.md5")" -eq 10 ]; then + rt="$rt → base decoded"; else rt="$rt → base decode FAILED"; fi + fi + printf '%-22s %-7s %-13s %s\n' "$cap" "${tier:-?}" "companion" "$rt" +} +rep "$TMP/hdr" 256; demo_companion wai.meta.hdr10plus "$TMP/hdr" +rep "$TMP/dovi" 256; demo_companion wai.meta.dovi "$TMP/dovi" +rep "$TMP/lcevc" 1024; rep "$TMP/lcevc2" 1024 + demo_companion wai.video.lcevc "$TMP/lcevc" "$TMP/lcevc2" echo echo "carriage is byte-exact for every class (§2). 'real+decode' rows ran a real" diff --git a/wai/tools/wai_replicate_probe.sh b/wai/tools/wai_replicate_probe.sh new file mode 100755 index 000000000..cb22b78d7 --- /dev/null +++ b/wai/tools/wai_replicate_probe.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# Probe the registry's replicate classes against real decoders. +# +# `codecs::capability_replicate_class` says, per capability, whether every +# conforming decoder outputs the identical reconstruction (`standard-defined`, +# resting on a conformance clause that requires it) or whether conforming +# decoders may differ (`standard-tolerance`: the standard states a criterion +# around a reference; `standard-unclassified`: its conformance was not read). +# This script is a falsification check on that claim where this host has more +# than one independent decoder for a format — it does not establish any class: +# it encodes a short synthetic source, decodes it with each decoder to raw +# samples in one common layout, and compares the outputs byte for byte. +# +# Verdicts, per row: +# identical every decoder gave the same bytes. +# differ decoders disagree; the count and the largest per-sample +# difference are printed. +# skipped fewer than two decoders (or no encoder) on this host. A +# skipped row is not a pass. +# +# The script FAILS (exit 1) only when a `standard-defined` capability's +# decoders disagree — that would contradict the registry. A `standard-tolerance` +# or `standard-unclassified` capability may come out identical on a given +# stream; the class permits that and claims no difference, so it is reported, +# not failed. +# +# Decoders used when present: ffmpeg's own decoders; libdav1d, libvpx and +# libopus through ffmpeg; Apple VideoToolbox hardware decode (forced onto +# hardware surfaces, so it fails rather than falling back to software); +# Apple AudioToolbox AAC; libjpeg-turbo `djpeg`; OpenJPEG `opj_decompress`; +# libFLAC `flac`; Core Audio `afconvert`. +# +# ./tools/wai_replicate_probe.sh +# +# The CLI is built with `cargo build --bin wai` (default features, so it needs +# libjxl); CARGO_TARGET_DIR is honoured. + +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT +have() { command -v "$1" >/dev/null 2>&1; } +have ffmpeg || { echo "ffmpeg not found — nothing to probe"; exit 1; } +have python3 || { echo "python3 not found — needed to compare samples"; exit 1; } + +WAI="${CARGO_TARGET_DIR:-$ROOT/wai-rs/target}/debug/wai" +echo "building wai CLI…" +( cd "$ROOT/wai-rs" && cargo build --bin wai >/dev/null 2>&1 ) || { echo "build failed"; exit 1; } +"$WAI" capabilities > "$TMP/caps.txt" || { echo "wai capabilities failed"; exit 1; } +class_of() { awk -v c="$1" '$1 == c { print $5 }' "$TMP/caps.txt"; } + +echo "host: $(uname -sm) · $(sysctl -n machdep.cpu.brand_string 2>/dev/null || uname -p) · $(ffmpeg -version | head -1 | cut -d' ' -f1-3)" + +ffdec_has() { ffmpeg -hide_banner -decoders 2>/dev/null | awk '{print $2}' | grep -qx "$1"; } +ffenc_has() { ffmpeg -hide_banner -encoders 2>/dev/null | awk '{print $2}' | grep -qx "$1"; } +FF=(ffmpeg -hide_banner -loglevel error -y) + +# Compare decoder outputs. $1 sample width in bytes (1 = u8, 2 = s16le); +# remaining args are name=file pairs. Prints "identical" or "differ …"; +# exits 3 when a file is missing or sizes disagree. +compare() { + python3 - "$@" <<'PY' +import sys, array +w = int(sys.argv[1]); pairs = [a.split("=", 1) for a in sys.argv[2:]] +def load(p): + b = open(p, "rb").read() + if w == 1: return b + a = array.array("h"); a.frombytes(b[: len(b) // 2 * 2]); return a +try: + data = [(n, load(p)) for n, p in pairs] +except OSError as e: + print(f"missing output ({e.filename})"); sys.exit(3) +ref_n, ref = data[0] +if any(len(d) != len(ref) for _, d in data): + print("sizes differ: " + ", ".join(f"{n}={len(d)}" for n, d in data)); sys.exit(3) +worst = [] +for n, d in data[1:]: + diff = [abs(x - y) for x, y in zip(ref, d)] + k = sum(1 for v in diff if v) + if k: worst.append(f"{ref_n}≠{n}: {k}/{len(ref)} samples, max {max(diff)}") +print("identical" if not worst else "differ — " + "; ".join(worst)) +PY +} + +FAIL=0 +row() { # $1 capability $2 sample width $3.. name=file + local cap="$1" w="$2"; shift 2 + local class; class="$(class_of "$cap")" + if [ -z "$class" ] || [ "$class" = "?" ]; then + printf '%-22s %-22s ✗ no replicate class in `wai capabilities`\n' "$cap" "${class:-missing}"; FAIL=1; return + fi + local have_n=0 args=() names="" + for p in "$@"; do + [ -s "${p#*=}" ] && { have_n=$((have_n + 1)); args+=("$p"); names="$names${p%%=*} "; } + done + if [ "$have_n" -lt 2 ]; then + printf '%-22s %-22s skipped (decoders available: %s)\n' "$cap" "$class" "${names:-none}"; return + fi + local out; out="$(compare "$w" "${args[@]}")"; local rc=$? + local note="" + if [ $rc -ne 0 ]; then note=" ✗ comparison error"; FAIL=1 + elif [ "${out%% *}" = "differ" ] && [ "$class" = "standard-defined" ]; then + note=" ✗ CONTRADICTS standard-defined"; FAIL=1 + elif [ "${out%% *}" = "differ" ]; then note=" (consistent with $class)" + elif [ "$class" != "standard-defined" ]; then note=" (the class permits this)" + fi + printf '%-22s %-22s %s [%s]%s\n' "$cap" "$class" "$out" "${names% }" "$note" +} + +# Video: software decode, single-threaded, to planar yuv420p. +vsw() { "${FF[@]}" -threads 1 -c:v "$2" -i "$3" -f rawvideo -pix_fmt yuv420p "$TMP/$1.yuv" 2>/dev/null; } +# VideoToolbox: frames must stay on hardware surfaces, else hwdownload fails. +vvt() { "${FF[@]}" -hwaccel videotoolbox -hwaccel_output_format videotoolbox_vld -i "$2" \ + -vf hwdownload,format=nv12,format=yuv420p -f rawvideo "$TMP/$1.yuv" 2>/dev/null; } +SRC=(-f lavfi -i "testsrc2=size=320x240:rate=25:duration=0.8" -pix_fmt yuv420p) + +printf '\n%-22s %-22s %s\n' "capability" "registry class" "decoded outputs" +printf '%-22s %-22s %s\n' "----------------------" "----------------------" "---------------" + +if ffenc_has libx264; then + "${FF[@]}" "${SRC[@]}" -c:v libx264 -bf 2 -f h264 "$TMP/v.264" 2>/dev/null + vsw avc_ff h264 "$TMP/v.264"; vvt avc_vt "$TMP/v.264" +fi +row wai.video.avc 1 "ffmpeg=$TMP/avc_ff.yuv" "videotoolbox=$TMP/avc_vt.yuv" + +if ffenc_has libx265; then + "${FF[@]}" "${SRC[@]}" -c:v libx265 -x265-params log-level=error -f hevc "$TMP/v.265" 2>/dev/null + vsw hevc_ff hevc "$TMP/v.265"; vvt hevc_vt "$TMP/v.265" +fi +row wai.video.hevc 1 "ffmpeg=$TMP/hevc_ff.yuv" "videotoolbox=$TMP/hevc_vt.yuv" + +if ffenc_has libvpx-vp9; then + "${FF[@]}" "${SRC[@]}" -c:v libvpx-vp9 -f ivf "$TMP/v.ivf" 2>/dev/null + vsw vp9_ff vp9 "$TMP/v.ivf" + ffdec_has libvpx-vp9 && vsw vp9_vpx libvpx-vp9 "$TMP/v.ivf" + vvt vp9_vt "$TMP/v.ivf" +fi +row wai.video.vp9 1 "ffmpeg=$TMP/vp9_ff.yuv" "libvpx=$TMP/vp9_vpx.yuv" "videotoolbox=$TMP/vp9_vt.yuv" + +AV1ENC=""; for e in libsvtav1 libaom-av1 librav1e; do ffenc_has "$e" && { AV1ENC="$e"; break; }; done +if [ -n "$AV1ENC" ]; then + "${FF[@]}" "${SRC[@]}" -c:v "$AV1ENC" -f ivf "$TMP/av1.ivf" >/dev/null 2>&1 + ffdec_has libdav1d && vsw av1_dav1d libdav1d "$TMP/av1.ivf" + ffdec_has libaom-av1 && vsw av1_aom libaom-av1 "$TMP/av1.ivf" + vvt av1_vt "$TMP/av1.ivf" +fi +row wai.video.av1 1 "dav1d=$TMP/av1_dav1d.yuv" "libaom=$TMP/av1_aom.yuv" "videotoolbox=$TMP/av1_vt.yuv" + +# Audio: decode to mono s16le. +TONE=(-f lavfi -i "sine=frequency=440:duration=1:sample_rate=48000" -ac 1) +adec() { "${FF[@]}" -c:a "$2" -i "$3" -f s16le -ac 1 "$TMP/$1.pcm" 2>/dev/null; } + +"${FF[@]}" "${TONE[@]}" -c:a aac -f adts "$TMP/a.aac" 2>/dev/null +for d in aac aac_fixed aac_at; do ffdec_has "$d" && adec "aac_$d" "$d" "$TMP/a.aac"; done +row wai.audio.aac 2 "ffmpeg-aac=$TMP/aac_aac.pcm" "ffmpeg-aac_fixed=$TMP/aac_aac_fixed.pcm" "audiotoolbox=$TMP/aac_aac_at.pcm" + +if ffenc_has libopus; then + "${FF[@]}" "${TONE[@]}" -c:a libopus -b:a 64k "$TMP/o.opus" 2>/dev/null + ffdec_has opus && adec opus_ff opus "$TMP/o.opus" + ffdec_has libopus && adec opus_lib libopus "$TMP/o.opus" +fi +row wai.audio.opus 2 "ffmpeg-opus=$TMP/opus_ff.pcm" "libopus=$TMP/opus_lib.pcm" + +"${FF[@]}" "${TONE[@]}" -c:a pcm_s16le -f s16le "$TMP/flac_src.pcm" 2>/dev/null +"${FF[@]}" -f s16le -ar 48000 -ac 1 -i "$TMP/flac_src.pcm" -c:a flac "$TMP/f.flac" 2>/dev/null +adec flac_ff flac "$TMP/f.flac" +have flac && flac -s -d --force-raw-format --endian=little --sign=signed -o "$TMP/flac_lib.pcm" "$TMP/f.flac" 2>/dev/null +have afconvert && afconvert -f WAVE -d LEI16 "$TMP/f.flac" "$TMP/flac_at.wav" 2>/dev/null \ + && "${FF[@]}" -i "$TMP/flac_at.wav" -f s16le -ac 1 "$TMP/flac_at.pcm" 2>/dev/null +row wai.audio.flac 2 "source=$TMP/flac_src.pcm" "ffmpeg=$TMP/flac_ff.pcm" "libFLAC=$TMP/flac_lib.pcm" "coreaudio=$TMP/flac_at.pcm" + +# Still images: grayscale, so no colour conversion enters the comparison. +"${FF[@]}" -f lavfi -i "testsrc2=size=320x240:rate=1:duration=1" -frames:v 1 -pix_fmt gray -f rawvideo "$TMP/src.gray" 2>/dev/null +"${FF[@]}" -f rawvideo -pix_fmt gray -s 320x240 -i "$TMP/src.gray" -f image2 -c:v pgm "$TMP/src.pgm" 2>/dev/null +pgm2raw() { "${FF[@]}" -i "$1" -pix_fmt gray -f rawvideo "$2" 2>/dev/null; } + +if have cjpeg; then + cjpeg -quality 75 -grayscale "$TMP/src.pgm" > "$TMP/g.jpg" 2>/dev/null + if have djpeg; then + djpeg -dct int -pnm "$TMP/g.jpg" > "$TMP/j_int.pgm" && pgm2raw "$TMP/j_int.pgm" "$TMP/j_int.gray" + djpeg -dct float -pnm "$TMP/g.jpg" > "$TMP/j_flt.pgm" && pgm2raw "$TMP/j_flt.pgm" "$TMP/j_flt.gray" + fi + "${FF[@]}" -i "$TMP/g.jpg" -pix_fmt gray -f rawvideo "$TMP/j_ff.gray" 2>/dev/null +fi +row wai.image.jpeg 1 "djpeg-int=$TMP/j_int.gray" "djpeg-float=$TMP/j_flt.gray" "ffmpeg=$TMP/j_ff.gray" + +if have opj_compress; then + opj_compress -i "$TMP/src.pgm" -o "$TMP/i.j2k" -I -r 20 >/dev/null 2>&1 # irreversible 9/7 + opj_compress -i "$TMP/src.pgm" -o "$TMP/r.j2k" >/dev/null 2>&1 # reversible 5/3 + for k in i r; do + have opj_decompress && opj_decompress -i "$TMP/$k.j2k" -o "$TMP/${k}_opj.pgm" >/dev/null 2>&1 \ + && pgm2raw "$TMP/${k}_opj.pgm" "$TMP/${k}_opj.gray" + "${FF[@]}" -c:v jpeg2000 -i "$TMP/$k.j2k" -pix_fmt gray -f rawvideo "$TMP/${k}_ff.gray" 2>/dev/null + done +fi +row wai.image.jp2 1 "openjpeg=$TMP/i_opj.gray" "ffmpeg=$TMP/i_ff.gray" +printf '%-22s %-22s ' " (reversible 5/3)" "" +[ -s "$TMP/r_opj.gray" ] && [ -s "$TMP/r_ff.gray" ] \ + && echo "$(compare 1 "source=$TMP/src.gray" "openjpeg=$TMP/r_opj.gray" "ffmpeg=$TMP/r_ff.gray") [source openjpeg ffmpeg] (lossless path; informational)" \ + || echo "skipped" + +echo +if [ $FAIL -ne 0 ]; then + echo "FAIL — a standard-defined capability's decoders disagreed, a comparison errored, or a probed capability has no registry class." + exit 1 +fi +echo "OK — no standard-defined capability's decoders disagreed on this host (skipped rows prove nothing)." diff --git a/wai/wai-rs/src/bin/wai.rs b/wai/wai-rs/src/bin/wai.rs index 7f3b409f2..38e2c5efb 100644 --- a/wai/wai-rs/src/bin/wai.rs +++ b/wai/wai-rs/src/bin/wai.rs @@ -21,8 +21,9 @@ fn usage() { eprintln!("wai decode -o "); eprintln!("wai inspect "); eprintln!("wai wrap [-o out.wai] [--fallback ] [--intent ]"); - eprintln!("wai unwrap [-o payload] carry/extract ANY registered codec's bytes"); - eprintln!("wai capabilities list registered capabilities + tier + license"); + eprintln!(" [--companion ]... WAI2: companions bound to the payload"); + eprintln!("wai unwrap [-o payload] [--component ] carry/extract ANY registered codec's bytes"); + eprintln!("wai capabilities list registered capabilities + tier + license + replicate class + role"); eprintln!(); eprintln!("codecs: png jpeg avif jxl jxl-lossless opus flac av1 av1-lossless zstd xz"); eprintln!("inputs:"); @@ -37,6 +38,24 @@ fn opt<'a>(args: &'a [String], flag: &str) -> Option<&'a str> { .and_then(|i| args.get(i + 1).map(|s| s.as_str())) } +/// Every `--companion ` pair among `args`. +fn companion_args(args: &[String]) -> Result, String> { + let mut pairs = Vec::new(); + let mut i = 0; + while i < args.len() { + if args[i] == "--companion" { + let (Some(c), Some(p)) = (args.get(i + 1), args.get(i + 2)) else { + return Err("--companion takes ".into()); + }; + pairs.push((c.clone(), p.clone())); + i += 3; + } else { + i += 1; + } + } + Ok(pairs) +} + fn read_meta(path: &str) -> Result, String> { let s = fs::read_to_string(format!("{path}.meta")) .map_err(|e| format!("missing {path}.meta: {e}"))?; @@ -169,7 +188,24 @@ fn encode(codec: &str, input: &str, out: &str, q: u8) -> Result<(), String> { fn decode(input: &str, out: &str) -> Result<(), String> { let wai = Wai::read(input).map_err(|e| e.to_string())?; - let cap = wai.capability(); + // SPEC §4 dispatch, shared with `inspect` so the two agree: the primary + // capability if this sink decodes it (step 3), else the fallback (step 5). + // A companion-only capability is never dispatched standalone (SPEC §5). + let native = wai::sink_capabilities(); + let roles = wai.check_component_roles(); + let Some(cap) = wai.dispatch_capability(|c| native.contains(&c)) else { + return Err(match roles { + Err(e) => format!("{e} (SPEC \u{a7}5)"), + Ok(()) => format!("inert at this sink: {} is not decoded here{}", + wai.capability(), + wai.fallback().map(|f| format!(", nor its fallback {f}")).unwrap_or_default()), + }); + }; + // A non-conforming envelope whose primary decodes here still decodes (§4 + // step 3); say why it does not conform. + if let Err(e) = roles { + eprintln!("warning: the envelope does not conform: {e} (SPEC \u{a7}5); decoding via {cap}"); + } match cap { c if c == CAP_IMAGE_PNG => { let (rgb, h, w) = image::png_decode(&wai.payload).map_err(|e| format!("{:?}", e))?; @@ -244,8 +280,14 @@ fn inspect(path: &str) -> Result<(), String> { let w = wai::container::Wai::from_bytes(&bytes).map_err(|e| e.to_string())?; let cap = w.capability(); let fb = w.fallback(); - let cap_ok = known.contains(cap); - let fb_ok = fb.map(|f| known.contains(f)); + // A companion-only capability is never dispatched standalone (SPEC §5), + // so it resolves neither as the capability nor as the fallback. The + // verdict comes from the same dispatch `decode` runs. + let usable = |c: &str| known.contains(c) && !wai::codecs::is_companion_only(c); + let cap_ok = usable(cap); + let fb_ok = fb.map(usable); + let chosen = w.dispatch_capability(|c| known.contains(c)); + let roles = w.check_component_roles(); println!(" payload size: {:>10} B", w.payload.len()); println!(" media: {}", w.media()); println!(" intent: {}", w.intent()); @@ -259,15 +301,27 @@ fn inspect(path: &str) -> Result<(), String> { println!(" fallback: {f}"); println!(" resolution: {}", if fb_ok.unwrap() { "\u{2713} this sink advertises it" } + else if wai::codecs::is_companion_only(f) { "\u{2717} companion-only, never dispatched standalone" } else { "\u{2717} NOT installed either" }); } - let verdict = match (cap_ok, fb_ok) { - (true, _) => "DECODABLE via primary capability", - (false, Some(true)) => "DECODABLE via declared fallback (informational only in v1.0)", - (false, Some(false)) => "INERT here \u{2014} neither capability installed", - (false, None) => "INERT here \u{2014} capability missing, no fallback declared", + let verdict = match chosen { + _ if wai::codecs::is_companion_only(cap) => + "NOT CONFORMING \u{2014} companion-only capability, never dispatched standalone", + Some(c) if c == cap => "DECODABLE via primary capability", + // §4 step 5 decodes the SAME payload with the fallback's codec. + Some(_) => "DECODABLE via declared fallback only if its decoder accepts this payload (\u{a7}4 step 5)", + None if fb.is_some_and(wai::codecs::is_companion_only) => + "INERT here \u{2014} capability missing; the fallback is companion-only and never dispatched", + None if fb.is_some() => "INERT here \u{2014} neither capability usable", + None => "INERT here \u{2014} capability missing, no fallback declared", }; - println!(" verdict: {verdict}"); + let conforms = if roles.is_ok() || wai::codecs::is_companion_only(cap) { "" } + else { " (the envelope does not conform; see component roles)" }; + println!(" verdict: {verdict}{conforms}"); + match roles { + Ok(()) => println!(" component roles: \u{2713} conform (SPEC \u{a7}5)"), + Err(e) => println!(" component roles: \u{2717} {e}"), + } } else { // WAI2 inspection let m = wai::container::WaiMulti::from_bytes(&bytes).map_err(|e| e.to_string())?; @@ -283,8 +337,15 @@ fn inspect(path: &str) -> Result<(), String> { { let avail = known.contains(meta.capability.as_str()); let mark = if avail { "\u{2713}" } else { "\u{2717}" }; - if avail && picked.is_none() { picked = Some(i); } - println!(" [{i}] {:>10} {:<28} {} {}", + // Companions (and companion-only capabilities) are applied with the + // chosen rendition, never picked instead of it (SPEC §7.1). + if avail && meta.is_selectable() && picked.is_none() { picked = Some(i); } + let role = match (meta.is_companion(), meta.companion_of.as_deref()) { + (true, Some(of)) => format!(" companion of {of}"), + (true, None) => " companion (unbound)".to_string(), + (false, _) => String::new(), + }; + println!(" [{i}] {:>10} {:<28} {} {}{role}", payload.len(), meta.capability, meta.kind, mark); } println!(); @@ -293,6 +354,10 @@ fn inspect(path: &str) -> Result<(), String> { m.manifest.renditions[i].capability), None => println!(" default pick: none \u{2014} no rendition's capability installed here"), } + match m.check_component_roles() { + Ok(()) => println!(" component roles: \u{2713} conform (SPEC \u{a7}5, \u{a7}7.1)"), + Err(e) => println!(" component roles: \u{2717} {e}"), + } } println!(); @@ -318,6 +383,19 @@ fn media_for(cap: &str) -> &'static str { } } +/// How dispatch treats a capability's payload: a standalone payload, a +/// companion applied with a base rendition, or an enhancement layer bound to +/// exactly one base encode (SPEC §5, §7.1). +fn role_str(cap: &str) -> &'static str { + if wai::codecs::is_enhancement_layer(cap) { + "enhancement" + } else if wai::codecs::is_companion_only(cap) { + "companion" + } else { + "standalone" + } +} + fn license_str(cap: &str) -> &'static str { match wai::codecs::capability_license(cap) { Some(wai::codecs::LicenseClass::RoyaltyFree) => "royalty-free", @@ -340,8 +418,18 @@ fn license_str(cap: &str) -> &'static str { /// any other capability (`wai.image.jpegai`, a float or integer-exact neural codec, /// a generator, a behavioral path) `wrap` refuses and writes nothing unless another /// intent is given. +/// +/// With one or more `--companion ` the output is a `WAI2` +/// envelope instead: the payload as its one rendition, and each companion bound +/// to it (`companion_of` = the payload's capability) — how HDR dynamic metadata +/// or an LCEVC enhancement travels with its base (SPEC §5, §7.1). It carries the +/// same intent, held to the same rule over the payload's capability and over +/// each companion's. It takes no `--fallback`. Either way the +/// envelope must pass the role check before it is written: a companion-only +/// capability is refused as a `WAI1` capability or fallback, and a `WAI2` table +/// that breaks the binding rules is refused. fn wrap(cap: &str, input: &str, out: &str, fallback: Option<&str>, - intent: Option<&str>) -> Result<(), String> { + intent: Option<&str>, companions: &[(String, String)]) -> Result<(), String> { if !wai::codecs::is_known_capability(cap) { return Err(format!("'{cap}' is not a registered capability (see `wai capabilities`)")); } @@ -349,6 +437,10 @@ fn wrap(cap: &str, input: &str, out: &str, fallback: Option<&str>, "not writing intent=replicate over {cap}: {why} (SPEC \u{a7}7). `wai wrap` pins no \ prior; pass --intent to carry the payload without a reconstruction claim \ (`create` for a generator, `improve` otherwise)"))?; + if !companions.is_empty() { + let payload = std::fs::read(input).map_err(|e| format!("read {input}: {e}"))?; + return wrap_with_companions(cap, payload, out, fallback, intent, companions); + } // A sink that lacks `cap` decodes through the fallback (SPEC §4 step 5), so the // intent has to hold over the fallback's class as well. if let Some(fb) = fallback { @@ -373,7 +465,14 @@ fn wrap(cap: &str, input: &str, out: &str, fallback: Option<&str>, conditioning: Conditioning { kind: "bytes".into(), prior: None }, target: serde_json::Value::Null, }; - let n = Wai::new(manifest, payload).write(out).map_err(|e| e.to_string())?; + let envelope = Wai::new(manifest, payload); + // The reference tool does not write an envelope that breaks the role rules. + envelope.check_component_roles().map_err(|e| match e { + wai::container::RoleError::CompanionOnlyAsCapability { .. } => format!( + "{e} (SPEC \u{a7}5): `wai wrap --companion {cap} `"), + _ => format!("{e} (SPEC \u{a7}5)"), + })?; + let n = envelope.write(out).map_err(|e| e.to_string())?; let tier = wai::codecs::capability_tier(cap).map(|t| format!("{t:?}")).unwrap_or_else(|| "?".into()); println!("wrapped → {out} ({n} B envelope, {n_in} B payload)"); println!(" capability {cap} · media {} · {tier} · {} · intent {intent}", @@ -381,13 +480,73 @@ fn wrap(cap: &str, input: &str, out: &str, fallback: Option<&str>, Ok(()) } -/// `wai unwrap [-o payload]` — parse the envelope, report the named -/// capability (+ tier + license + fallback), and extract the EXACT payload -/// bytes for the sink's decoder. The extraction is byte-identical to what -/// `wrap` carried. -fn unwrap(input: &str, out: Option<&str>) -> Result<(), String> { - let w = Wai::from_bytes(&std::fs::read(input).map_err(|e| format!("read {input}: {e}"))?) - .map_err(|e| e.to_string())?; +/// The `WAI2` half of [`wrap`]: the base payload as the one selectable +/// rendition, each companion bound to it. +fn wrap_with_companions(cap: &str, payload: Vec, out: &str, fallback: Option<&str>, + intent: &str, companions: &[(String, String)]) -> Result<(), String> { + if fallback.is_some() { + return Err("--fallback names a WAI1 alternative; a WAI2 envelope carries alternatives \ + as renditions (SPEC \u{a7}7.1)".into()); + } + let mut components = vec![wai::container::Rendition::media(cap, "bytes", payload)]; + for (c, p) in companions { + if !wai::codecs::is_known_capability(c) { + return Err(format!("'{c}' is not a registered capability (see `wai capabilities`)")); + } + // A sink that recognises the companion applies it to the base's output + // (SPEC §7.1), so the intent has to hold over its class as well. Every + // companion-only capability registered today is in a standard class, so + // this refuses nothing yet; it keeps the rule if one is registered in + // another class. + wai::codecs::unpinned_envelope_intent(c, Some(intent)).map_err(|why| format!( + "not writing intent=replicate over {cap} with companion {c}: a sink that \ + recognises {c} applies it to {cap}'s output (SPEC \u{a7}7.1), and over {c}: \ + {why} (SPEC \u{a7}7). Pass --intent to carry the payload without a \ + reconstruction claim"))?; + let bytes = std::fs::read(p).map_err(|e| format!("read {p}: {e}"))?; + let mut comp = wai::container::Rendition::companion(c.as_str(), bytes, cap); + comp.kind = "bytes".into(); + components.push(comp); + } + let envelope = wai::container::WaiMulti::new( + media_for(cap), intent, serde_json::Value::Null, components); + envelope.check_component_roles() + .map_err(|e| format!("{e} (SPEC \u{a7}5, \u{a7}7.1)"))?; + let n = envelope.write(out).map_err(|e| e.to_string())?; + println!("wrapped → {out} ({n} B WAI2 envelope, {} components, intent {intent})", + envelope.renditions.len()); + print_components(&envelope); + Ok(()) +} + +/// One line per `WAI2` component: index, capability, role, size, licence. +fn print_components(m: &wai::container::WaiMulti) { + for (i, (meta, p)) in m.manifest.renditions.iter().zip(m.renditions.iter()).enumerate() { + let role = match (meta.is_companion(), meta.companion_of.as_deref()) { + (true, Some(of)) => format!("companion of {of}"), + (true, None) => "companion (unbound)".into(), + (false, _) => "rendition".into(), + }; + println!(" [{i}] {:<26} {:<30} {:>10} B {}", meta.capability, role, p.len(), + license_str(&meta.capability)); + } +} + +/// `wai unwrap [-o payload] [--component ]` — parse the envelope, +/// report the named capability (+ tier + license + fallback + role check), and +/// extract the EXACT payload bytes for the sink's decoder. The extraction is +/// byte-identical to what `wrap` carried. For a `WAI2` envelope it lists the +/// components and extracts component `i` (default: the first selectable +/// rendition). +fn unwrap(input: &str, out: Option<&str>, component: Option<&str>) -> Result<(), String> { + let bytes = std::fs::read(input).map_err(|e| format!("read {input}: {e}"))?; + if wai::container::detect_version(&bytes).map_err(|e| e.to_string())? == 2 { + return unwrap_component(input, &bytes, out, component); + } + if component.is_some_and(|c| c != "0") { + return Err("a WAI1 envelope carries one payload (component 0)".into()); + } + let w = Wai::from_bytes(&bytes).map_err(|e| e.to_string())?; let cap = w.capability().to_string(); let tier = wai::codecs::capability_tier(&cap).map(|t| format!("{t:?}")).unwrap_or_else(|| "?".into()); let native = wai::codecs::sink_capabilities().contains(&cap.as_str()); @@ -397,7 +556,12 @@ fn unwrap(input: &str, out: Option<&str>) -> Result<(), String> { println!(" fidelity {tier}"); println!(" license {}", license_str(&cap)); println!(" decode {}", if native { "native (this crate)" } else { "sink-supplied" }); + println!(" role {}", role_str(&cap)); println!(" fallback {}", w.fallback().unwrap_or("(none)")); + match w.check_component_roles() { + Ok(()) => println!(" conformance \u{2713} component roles (SPEC \u{a7}5)"), + Err(e) => println!(" conformance \u{2717} {e}"), + } println!(" payload {} bytes", w.payload.len()); if let Some(out) = out { std::fs::write(out, &w.payload).map_err(|e| format!("write {out}: {e}"))?; @@ -406,6 +570,31 @@ fn unwrap(input: &str, out: Option<&str>) -> Result<(), String> { Ok(()) } +/// The `WAI2` half of [`unwrap`]. +fn unwrap_component(input: &str, bytes: &[u8], out: Option<&str>, + component: Option<&str>) -> Result<(), String> { + let m = wai::container::WaiMulti::from_bytes(bytes).map_err(|e| e.to_string())?; + let idx = match component { + Some(s) => s.parse::().map_err(|_| format!("--component takes an index, got '{s}'"))?, + None => m.manifest.renditions.iter().position(|r| r.is_selectable()) + .ok_or("no selectable rendition; name one with --component ")?, + }; + let (meta, payload) = m.manifest.renditions.get(idx).zip(m.renditions.get(idx)) + .ok_or_else(|| format!("component {idx} out of range ({} components)", m.renditions.len()))?; + println!("envelope: {input} (WAI2, {} components)", m.renditions.len()); + print_components(&m); + match m.check_component_roles() { + Ok(()) => println!(" conformance \u{2713} component roles (SPEC \u{a7}5, \u{a7}7.1)"), + Err(e) => println!(" conformance \u{2717} {e}"), + } + println!(" component [{idx}] {} · {} bytes", meta.capability, payload.len()); + if let Some(out) = out { + std::fs::write(out, payload).map_err(|e| format!("write {out}: {e}"))?; + println!(" → wrote component {idx} to {out} (byte-identical to the wrapped input)"); + } + Ok(()) +} + fn capabilities() -> Result<(), String> { let native: std::collections::HashSet<&str> = wai::codecs::sink_capabilities().iter().copied().collect(); @@ -417,8 +606,8 @@ fn capabilities() -> Result<(), String> { "wai capabilities — {} registered ({rf} royalty-free; {nie}/{neural} neural integer-exact)", caps.len() ); - println!(" {:<26} {:<11} {:<13} {:<14} {}", "capability", "fidelity", "decode", "license", "replicate"); - println!(" {:-<26} {:-<11} {:-<13} {:-<14} {:-<18}", "", "", "", "", ""); + println!(" {:<26} {:<11} {:<13} {:<14} {:<21} {}", "capability", "fidelity", "decode", "license", "replicate", "role"); + println!(" {:-<26} {:-<11} {:-<13} {:-<14} {:-<21} {:-<11}", "", "", "", "", "", ""); for &c in caps { let tier = wai::codecs::capability_tier(c) .map(|t| format!("{t:?}")).unwrap_or_else(|| "?".into()); @@ -433,7 +622,7 @@ fn capabilities() -> Result<(), String> { let replicate = wai::codecs::capability_replicate_class(c) .map(wai::codecs::ReplicateClass::label) .unwrap_or("?"); - println!(" {c:<26} {tier:<11} {decode:<13} {license:<14} {replicate}"); + println!(" {c:<26} {tier:<11} {decode:<13} {license:<14} {replicate:<21} {}", role_str(c)); } Ok(()) } @@ -453,10 +642,14 @@ fn main() -> ExitCode { "decode" if args.len() >= 4 => decode(&args[1], &args[3]), "wrap" if args.len() >= 3 => { let out = opt(&args[3..], "-o").map(String::from).unwrap_or("out.wai".into()); - wrap(&args[1], &args[2], &out, opt(&args[3..], "--fallback"), - opt(&args[3..], "--intent")) + match companion_args(&args[3..]) { + Ok(comps) => wrap(&args[1], &args[2], &out, opt(&args[3..], "--fallback"), + opt(&args[3..], "--intent"), &comps), + Err(e) => Err(e), + } } - "unwrap" if args.len() >= 2 => unwrap(&args[1], opt(&args[2..], "-o")), + "unwrap" if args.len() >= 2 => unwrap(&args[1], opt(&args[2..], "-o"), + opt(&args[2..], "--component")), "inspect" if args.len() >= 2 => inspect(&args[1]), "capabilities" => capabilities(), _ => { usage(); return ExitCode::from(2); } diff --git a/wai/wai-rs/src/codecs/mod.rs b/wai/wai-rs/src/codecs/mod.rs index 1cad185a6..6fb98d82a 100644 --- a/wai/wai-rs/src/codecs/mod.rs +++ b/wai/wai-rs/src/codecs/mod.rs @@ -306,6 +306,97 @@ pub const CAP_META_HDR10PLUS: &str = "wai.meta.hdr10plus"; // license-flagged premium slot (preserve the brand for marquee content). pub const CAP_META_DOVI: &str = "wai.meta.dovi"; +// --------------------------------------------------------------------------- +// Broadcast and streaming video / audio codecs: the elementary bitstreams that +// terrestrial and satellite broadcast, streaming and device playback carry, so +// a WAI envelope can name them. Every one is SINK-SUPPLIED — this crate bundles +// no decoder for any of them — and none is in the mandatory floor, the +// recommended set or sink_capabilities(). Payload = the named standard's +// canonical elementary bitstream in the framing stated per entry (SPEC §5, +// "Broadcast and streaming codecs"). Licence classes live in +// capability_license(); sources in research/media-codec-registry.md. +// --------------------------------------------------------------------------- + +// AV2 (AOMedia; AV2 Bitstream & Decoding Process Specification v1.0.0, 28 May +// 2026) — ROYALTY-FREE (AOMedia Patent License 1.0, shipped with the AVM +// v1.0.0 reference software). Payload: the length-delimited bitstream of the +// AV2 specification's Annex B (every OBU preceded by its leb128 byte length) — +// AV2 OBUs carry no size of their own, and Annex B is the one framing the +// specification itself defines. Registered, not recommended: a reference +// decoder exists, deployable decode (a production decoder, shipping hardware) +// is not yet confirmed. See SPEC §5 for the placement rule. +pub const CAP_VIDEO_AV2: &str = "wai.video.av2"; + +// VVC / H.266 (ITU-T H.266 | ISO/IEC 23090-3) — PATENT POOL. Payload: the +// Annex B byte stream (start-code-delimited NAL units, parameter sets in-band). +pub const CAP_VIDEO_VVC: &str = "wai.video.vvc"; + +// HEVC / H.265 (ITU-T H.265 | ISO/IEC 23008-2) — PATENT POOL. Payload: the +// Annex B byte stream. +pub const CAP_VIDEO_HEVC: &str = "wai.video.hevc"; + +// AVC / H.264 (ITU-T H.264 | ISO/IEC 14496-10) — PATENT POOL. Payload: the +// Annex B byte stream. +pub const CAP_VIDEO_AVC: &str = "wai.video.avc"; + +// VP9 (VP9 Bitstream & Decoding Process Specification) — ROYALTY-FREE (BSD +// source + the WebM project's patent grant). VP9 frames are not +// self-delimiting, so the payload is the IVF frame container (FourCC "VP90"): +// a 32-byte file header, then a 12-byte header before each frame. +pub const CAP_VIDEO_VP9: &str = "wai.video.vp9"; + +// EVC (MPEG-5 Part 1, ISO/IEC 23094-1), ANY profile — PATENT POOL: the Main +// profile's coding tools are licensed by their holders on FRAND terms, so a +// decoder for this capability needs that licence. Payload: the raw bitstream +// file storage format of ISO/IEC 23094-1 Annex B. +pub const CAP_VIDEO_EVC: &str = "wai.video.evc"; +// EVC constrained to the BASELINE profile — ROYALTY-FREE by design (its tools +// are more than 20 years old or were offered royalty-free). No formal +// guarantee; the residual caveat is recorded in the research doc, as for JPEG +// 2000 Part 1. Same payload framing as wai.video.evc, Baseline-conformant only +// (the relation wai.video.av1.lossless has to wai.video.av1). +pub const CAP_VIDEO_EVC_BASELINE: &str = "wai.video.evc.baseline"; + +// AVS3 video (AVS3-P2: T/AI 109.2-2021; IEEE 1857.10) — PATENT POOL (the AVS +// patent pool; RAND, fees on devices). Payload: the AVS3-P2 video elementary +// bitstream as T/AI 109.2 defines it. +pub const CAP_VIDEO_AVS3: &str = "wai.video.avs3"; + +// LCEVC (MPEG-5 Part 2, ISO/IEC 23094-2) — PATENT POOL (SEP licence, royalty +// on devices and on distribution services). NOT a standalone codec: an +// ENHANCEMENT LAYER whose residuals are added to the upscaled output of one +// specific base encode (AVC, HEVC, VVC, EVC, VP9, AV1). Two carriages: +// - embedded: the enhancement rides inside the base bitstream (SEI / +// registered user data); the payload is the base codec's bytes, under the +// base capability, and no separate component exists; +// - separate: payload = the LCEVC enhancement in the byte-stream form of +// ISO/IEC 23094-2 Annex B (a 0x000001 start code prefix before every NAL +// unit), carried ONLY as a WAI2 companion (SPEC §7.1) whose companion_of +// resolves to EXACTLY ONE selectable video rendition — the base it was +// encoded against. See is_companion_only / is_enhancement_layer. +// 23094-2 admits two bitstream forms, a NAL unit stream or a byte stream +// (definition 3.2); only the byte stream delimits itself, so it is the pin. +// A bare or length-prefixed NAL unit stream is NOT this capability. +pub const CAP_VIDEO_LCEVC: &str = "wai.video.lcevc"; + +// MPEG-H 3D Audio (ISO/IEC 23008-3) — PATENT POOL. Channel, object and +// scene-based (HOA) next-generation audio. Payload: the MHAS packet stream +// (MPEG-H 3D Audio Stream, ISO/IEC 23008-3), the self-synchronising form +// broadcast carriage uses. +pub const CAP_AUDIO_MPEGH: &str = "wai.audio.mpegh"; + +// AVS3 audio / Audio Vivid (AVS3-P3: T/AI 109.3-2023; GY/T 363-2023; the +// 3D-audio national standard GB/T 46271-2025) — PATENT POOL (the AVS patent +// pool). Immersive audio with lossy and lossless modes. Payload: the AVS3-P3 +// audio elementary bitstream as T/AI 109.3 defines it. +pub const CAP_AUDIO_AVS3: &str = "wai.audio.avs3"; + +// AAC family (ISO/IEC 14496-3: AAC-LC, HE-AAC, HE-AAC v2) — PATENT POOL. +// Payload: an ADTS stream (ISO/IEC 13818-7 / 14496-3), the self-synchronising +// elementary form. Extended HE-AAC (USAC) is NOT this capability — ADTS +// cannot carry it. +pub const CAP_AUDIO_AAC: &str = "wai.audio.aac"; + /// Returns the set of capabilities this sink (the WAI Rust crate) /// supports natively. A wrapping application is free to advertise /// additional neural capabilities backed by its own ML runtime. @@ -368,6 +459,30 @@ pub fn known_capabilities() -> &'static [&'static str] { CAP_HAPTIC_MIHS, // HDR dynamic-metadata descriptors (companion to a video payload) CAP_META_HDR10PLUS, CAP_META_DOVI, + // broadcast and streaming video (sink-supplied; none is a default) + CAP_VIDEO_AV2, CAP_VIDEO_VVC, CAP_VIDEO_HEVC, CAP_VIDEO_AVC, CAP_VIDEO_VP9, + CAP_VIDEO_EVC, CAP_VIDEO_EVC_BASELINE, CAP_VIDEO_AVS3, + // enhancement layer (companion-only, bound to one base video rendition) + CAP_VIDEO_LCEVC, + // broadcast and streaming audio (sink-supplied; none is a default) + CAP_AUDIO_MPEGH, CAP_AUDIO_AVS3, CAP_AUDIO_AAC, + ] +} + +/// SPEC §5 "Mandatory floor": every conforming sink MUST implement these. +pub fn mandatory_floor() -> &'static [&'static str] { + &[CAP_IMAGE_PNG, CAP_AUDIO_FLAC, CAP_TEXT_ZSTD] +} + +/// SPEC §5 "Recommended modern set": every conforming sink SHOULD implement +/// these. Together with [`mandatory_floor`] these are the capabilities WAI +/// makes defaults; every one is royalty-free (a test enforces it). +pub fn recommended_set() -> &'static [&'static str] { + &[ + CAP_IMAGE_JXL, CAP_IMAGE_AVIF, CAP_IMAGE_JPEG, + CAP_AUDIO_OPUS, + CAP_VIDEO_AV1, CAP_VIDEO_AV1_LOSSLESS, + CAP_TEXT_XZ, ] } @@ -376,6 +491,111 @@ pub fn is_known_capability(cap: &str) -> bool { known_capabilities().contains(&cap) } +/// Whether a registered capability is **companion-only**: its payload has no +/// meaning on its own and is only ever applied *with* a base payload, never +/// selected instead of one (SPEC §7.1). HDR dynamic metadata describes a video +/// payload; an LCEVC enhancement adds residuals to one. Such a capability is +/// never dispatched standalone — not as a v1.0 `capability` or `fallback`, not +/// as a v1.1 selectable rendition — only as a v1.1 companion. The converse +/// holds too: every other registered capability is standalone media, and is +/// never a companion (SPEC §7.1, rule 1). +pub fn is_companion_only(cap: &str) -> bool { + matches!(cap, CAP_META_HDR10PLUS | CAP_META_DOVI | CAP_VIDEO_LCEVC) +} + +/// Whether a registered capability is an **enhancement layer**: a companion +/// whose payload was encoded against one specific base encode, so it binds to +/// exactly one selectable rendition of one of its [`enhancement_bases`], never +/// to "every rendition of that codec" (SPEC §5, §7.1). Every enhancement layer +/// is also companion-only. +pub fn is_enhancement_layer(cap: &str) -> bool { + cap == CAP_VIDEO_LCEVC +} + +/// The base codecs WAI registers `wai.video.lcevc` over (SPEC §5): AVC, HEVC, +/// VVC, EVC, VP9 and AV1, each under every capability that names its bitstream +/// — the EVC Baseline entry is an EVC bitstream and the lossless AV1 entry an +/// AV1 bitstream. Not AV2, AVS3, JPEG 2000 or the WAI-native integer codecs. +/// +/// The list is this registration's, not a limit of the LCEVC standard: +/// ISO/IEC 23094-2:2021 takes its decoded base picture from "a base decoder" +/// (definition 3.17) and names no base codec among its normative references +/// (clause 2). Adding a base widens what the enhancement layer applies to, which +/// a MINOR revision may do; removing one narrows it, which needs a new MAJOR +/// (SPEC §8). +const LCEVC_BASES: &[&str] = &[ + CAP_VIDEO_AVC, CAP_VIDEO_HEVC, CAP_VIDEO_VVC, CAP_VIDEO_EVC, CAP_VIDEO_EVC_BASELINE, + CAP_VIDEO_VP9, CAP_VIDEO_AV1, CAP_VIDEO_AV1_LOSSLESS, +]; + +/// The capabilities an enhancement layer may be bound to: the base codecs its +/// registration names (SPEC §5), which for LCEVC are WAI's choice rather than a +/// restriction of its standard. Empty for a capability that is not an +/// enhancement layer. +pub fn enhancement_bases(cap: &str) -> &'static [&'static str] { + if cap == CAP_VIDEO_LCEVC { LCEVC_BASES } else { &[] } +} + +/// The registered capabilities outside the `wai.video.*` namespace whose payload +/// is video: the neural video codecs SPEC §5 lists under *Video:* (the `video` +/// media class, in its sink-architecture note), and the films of the +/// linear-film extension, which gives a linear film's `media` the value +/// `"video"` (a reel is a film with cuts). +const VIDEO_CLASS_OUTSIDE_NAMESPACE: &[&str] = &[ + CAP_NEURAL_VIDEO_BMSHJ2018, CAP_NEURAL_DCVC_RT, CAP_NEURAL_DCVC_FM, + CAP_FILM_LINEAR, CAP_FILM_REEL, +]; + +/// Whether a registered capability's payload is video — the `video` media +/// class of SPEC §3, as the capability's registration places it (the +/// manifest's `media` field is informational; the capability is +/// authoritative): +/// +/// - every registered `wai.video.*` capability that is not companion-only; +/// - the neural video codecs `wai.neural.video_bmshj2018`, `wai.neural.dcvc_rt` +/// and `wai.neural.dcvc_fm`; +/// - the films `wai.film.linear` and `wai.film.reel`. +/// +/// False for a capability this revision does not register: its class is not +/// known here. A video capability a later revision registers joins the class +/// in that revision. +pub fn is_video_class(cap: &str) -> bool { + is_known_capability(cap) + && ((cap.starts_with("wai.video.") && !is_companion_only(cap)) + || VIDEO_CLASS_OUTSIDE_NAMESPACE.contains(&cap)) +} + +/// Whether a companion of capability `companion` may be applied to a rendition +/// of capability `base` — SPEC §7.1, rule 2: +/// +/// - an HDR dynamic-metadata descriptor (`wai.meta.hdr10plus`, +/// `wai.meta.dovi`) was registered as a companion to a video payload, so a +/// registered `base` is one in the `video` media class ([`is_video_class`]): +/// a `wai.video.*` capability that is not itself companion-only, a neural +/// video codec, or a film. A `base` this revision does not register is not +/// constrained — its class is not known here, and a sink that does not +/// recognise it cannot present it — so a video capability a later revision +/// registers, in any namespace, is already a base older checkers accept. A +/// later revision that registers such a base outside the video class makes +/// that binding non-conforming, as SPEC §8 allows a new registration to; +/// - an enhancement layer applies to one of its [`enhancement_bases`] only. +/// +/// False for every other `companion`: a standalone capability is never a +/// companion (rule 1), and for a capability this revision does not register the +/// registry knows no bases — [`crate::container::WaiMulti::check_component_roles`] +/// checks such a companion only for a present base (rule 3). No companion +/// applies to a companion-only `base`. +pub fn companion_applies_to(companion: &str, base: &str) -> bool { + if is_companion_only(base) { + return false; + } + match companion { + CAP_META_HDR10PLUS | CAP_META_DOVI => !is_known_capability(base) || is_video_class(base), + c if is_enhancement_layer(c) => enhancement_bases(c).contains(&base), + _ => false, + } +} + /// The fidelity a capability delivers — the honest answer to "what do I get /// back?". This is *independent* of the determinism contract /// ([`crate::container::DeterminismClass`], about cross-hardware @@ -452,7 +672,15 @@ CAP_WORLD_SCENE | CAP_WORLD_REPLAY | CAP_WORLD_LIVE | CAP_WORLD_MODEL | CAP_MESH // ternary-MPS quantizes the transform to trits (lossy vs source) but its // decode is integer/byte-exact (reconstruction-equivalence) — the // StandardDefined replicate class, the deterministic floor, like quantum. - | CAP_TERNARY_TT => Lossy, + | CAP_TERNARY_TT + // Broadcast and streaming codecs, tiered by their primary role in the + // menu (lossy distribution). Several have lossless modes (HEVC/VVC + // lossless coding, AVS3-P3 lossless audio) — as for AVIF, the bitstream + // carries the actual fidelity; the tier is the role, not a per-file + // guarantee. LCEVC is lossy residual enhancement of a lossy base. + | CAP_VIDEO_AV2 | CAP_VIDEO_VVC | CAP_VIDEO_HEVC | CAP_VIDEO_AVC | CAP_VIDEO_VP9 + | CAP_VIDEO_EVC | CAP_VIDEO_EVC_BASELINE | CAP_VIDEO_AVS3 | CAP_VIDEO_LCEVC + | CAP_AUDIO_MPEGH | CAP_AUDIO_AVS3 | CAP_AUDIO_AAC => Lossy, // Generative / similarity-conformant. CAP_SEMANTIC_CLIP | CAP_SEMANTIC_PERCO2 => Perceptual, _ => return None, @@ -481,9 +709,18 @@ pub enum LicenseClass { } /// The [`LicenseClass`] of a registered capability, or `None` if unknown. -/// Verified against primary sources June 2026; the residual caveats (JPEG 2000 -/// extensions / submarine patents, MPEG-I Haptics pool status unconfirmed) are -/// documented in `research/media-codec-registry.md`. +/// Verified against primary sources (June 2026 for the immersive / cinema / +/// volumetric / haptics / HDR rows, September 2026 for the broadcast and +/// streaming codecs); the residual caveats (JPEG 2000 extensions / submarine +/// patents, MPEG-I Haptics pool status unconfirmed, EVC Baseline's +/// by-design royalty-free status, third-party pool assertions against the +/// AOMedia and VP9 formats) are documented in `research/media-codec-registry.md`. +/// +/// The classification is **exhaustive, with no default class**: every +/// registered capability is listed under exactly one arm, and a capability +/// that is registered but listed nowhere returns `None` — which the registry +/// tests reject. A newly registered codec therefore cannot become +/// "royalty-free" by omission. pub fn capability_license(cap: &str) -> Option { use LicenseClass::*; if !is_known_capability(cap) { @@ -497,13 +734,53 @@ pub fn capability_license(cap: &str) -> Option { // declarations apply — pool status unconfirmed, classified // conservatively). CAP_AUDIO_IVAS | CAP_PC_GPCC | CAP_PC_VPCC | CAP_HAPTIC_MIHS => PatentPool, + // Broadcast and streaming codecs whose essential patents are licensed + // for royalties — through pools, a single SEP licensor, or FRAND + // holders (September 2026): VVC, HEVC and AVC (several pools plus + // bilateral licensors); EVC Main-profile tools (FRAND holders); LCEVC + // (SEP licence with device and distribution royalties); AVS3 video and + // audio (the AVS patent pool, RAND, fees on devices); MPEG-H 3D Audio + // and AAC (pool programmes). + CAP_VIDEO_VVC | CAP_VIDEO_HEVC | CAP_VIDEO_AVC | CAP_VIDEO_EVC | CAP_VIDEO_LCEVC + | CAP_VIDEO_AVS3 | CAP_AUDIO_MPEGH | CAP_AUDIO_AVS3 | CAP_AUDIO_AAC => PatentPool, // Non-commercial model licence (PerCoV2 is CC-BY-NC-SA). CAP_SEMANTIC_PERCO2 => NonCommercial, - // Everything else WAI registers is royalty-free. JPEG 2000 Part 1 is - // fee-free under ITU-T policy 2.1 (extensions/submarine-patent caveats - // noted); IAMF under the AOMedia Patent License 1.0; Draco Apache-2.0; - // HDR10+ royalty-free for content + devices (only the logo is licensed). - _ => RoyaltyFree, + // Royalty-free — listed explicitly, never a fall-through. + // Floor + modern menu: open standards and open references. + CAP_IMAGE_PNG | CAP_IMAGE_JPEG | CAP_IMAGE_AVIF | CAP_IMAGE_JXL + | CAP_AUDIO_OPUS | CAP_AUDIO_FLAC + | CAP_VIDEO_AV1 | CAP_VIDEO_AV1_LOSSLESS + | CAP_TEXT_ZSTD | CAP_TEXT_XZ + // JPEG AI and the neural / semantic / generative capabilities. This is + // the classification the registry already gave them (by fall-through); + // it was made explicit here, not re-verified — model-weight licences + // are an open item in research/media-codec-registry.md. + | CAP_IMAGE_JPEGAI + | CAP_NEURAL_ENCODEC32 | CAP_NEURAL_DAC | CAP_NEURAL_MIMI | CAP_NEURAL_WAVTOKENIZER + | CAP_NEURAL_SNAC + | CAP_NEURAL_BMSHJ2018 | CAP_NEURAL_MBT2018_MEAN | CAP_NEURAL_INT_SYNTH | CAP_NEURAL_INT_HYPER + | CAP_NEURAL_ELIC | CAP_NEURAL_MLICPP | CAP_NEURAL_MLICV2 | CAP_NEURAL_CMIC + | CAP_NEURAL_VIDEO_BMSHJ2018 | CAP_NEURAL_GLC | CAP_NEURAL_DCVC_RT | CAP_NEURAL_DCVC_FM + | CAP_SEMANTIC_CLIP | CAP_GENERATE_SPECTRAL + // WAI-native classes: the reference implementation is Apache-2.0. + | CAP_VIDEO_INT_MOTION | CAP_VIDEO_INT_HYPER | CAP_AUDIO_INT_CODEC | CAP_SPLAT_INT_CODEC + | CAP_WORLD_SCENE | CAP_WORLD_REPLAY | CAP_WORLD_LIVE | CAP_WORLD_MODEL + | CAP_AUDIO_SCENE | CAP_AUDIO_SCORE | CAP_HAPTIC_SIGNAL | CAP_FEED_STATE + | CAP_AVATAR_POSE | CAP_SPLAT_4D | CAP_FILM_LINEAR | CAP_FILM_REEL + | CAP_QUANTUM_CIRCUIT | CAP_TERNARY_TT + // Open asset formats: glTF (Khronos), SPZ (open source). + | CAP_MESH_GLB | CAP_SPLAT_SPZ + // JPEG 2000 Part 1 is fee-free under ITU-T policy 2.1 (extensions / + // submarine-patent caveats noted); IAMF under the AOMedia Patent + // License 1.0; Draco Apache-2.0; SOG open work; HDR10+ royalty-free for + // content + devices (only the logo is licensed). + | CAP_IMAGE_JP2 | CAP_VIDEO_J2K | CAP_AUDIO_IAMF | CAP_MESH_DRACO | CAP_SPLAT_SOG + | CAP_META_HDR10PLUS + // AV2 under the AOMedia Patent License 1.0; VP9 under its BSD source + + // patent grant; EVC Baseline royalty-free by design (caveat recorded). + | CAP_VIDEO_AV2 | CAP_VIDEO_VP9 | CAP_VIDEO_EVC_BASELINE => RoyaltyFree, + // Registered but unclassified: a registry defect, surfaced as `None`. + _ => return None, }) } @@ -541,18 +818,32 @@ pub fn is_royalty_free(cap: &str) -> bool { #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum ReplicateClass { /// The reconstruction is specified exactly, so `replicate` promises the same - /// bytes on every conforming sink: by the named standard, for the lossless - /// formats whose decoded output it determines (PNG, FLAC, the zstd and XZ byte - /// formats), or by the WAI extension that defines the capability's - /// `*-equivalence` over integer / fixed-point state (`world`, `score`, `film`, - /// `quantum`, `ternary`, the haptic signal, the integer inter-frame codec, …). - /// The promise covers the output the standard or extension specifies, at its - /// own sample depth and layout; a conversion a sink applies afterwards (PNG - /// sample-depth rescaling, for which RFC 2083 §10.4 gives two methods) is not - /// part of it. An extension's equivalence holds given its inputs: where it - /// composes other capabilities' decoded output (a `wai.audio.scene` source - /// decoded through `wai.audio.opus`, say), it is only as exact as their - /// classes. No neural capability is in this class. + /// bytes on every conforming sink. Either the named standard requires it — the + /// lossless formats whose decoded output it determines (PNG, FLAC, the zstd and + /// XZ byte formats), and the video codecs whose decoder-conformance clause + /// requires every output sample to equal the specified decoding process's: + /// `wai.video.avc` (ITU-T H.264 (V16, 06/2026) §C.4), `wai.video.hevc` (H.265 + /// (V11, 01/2026) §C.5.1), `wai.video.vvc` (H.266 (V4, 01/2026) §C.5.1), + /// `wai.video.vp9` (VP9 Bitstream & Decoding Process Specification v0.6 §8.1, + /// "identical in all respects") — or the WAI extension that defines the + /// capability's `*-equivalence` over integer / fixed-point state (`world`, + /// `score`, `film`, `quantum`, `ternary`, the haptic signal, the integer + /// inter-frame codec, …). The promise covers the output the standard or + /// extension specifies, at its own sample depth and layout: for the video codecs + /// the cropped decoded pictures, before any processing a message only signals + /// (film grain characteristics, post-filter hints) and before colour + /// conversion; a conversion a sink applies afterwards (PNG sample-depth + /// rescaling, for which RFC 2083 §10.4 gives two methods) is not part of it. An + /// extension's equivalence holds given its inputs: where it composes other + /// capabilities' decoded output (a `wai.audio.scene` source decoded through + /// `wai.audio.opus`, say), it is only as exact as their classes + /// ([`composes_decoded_sources`]; [`is_cross_decoder_exact`] is false for such + /// a capability until its sources are known); for + /// `wai.world.model` it covers the signed action trace and session hash, not + /// the frames its declared renderer generates. No neural decode is in this + /// class. Every member's basis is recorded, one row per capability, in the + /// registry tests (`STANDARD_DEFINED_EVIDENCE`); a capability without a + /// recorded basis belongs in [`StandardUnclassified`](Self::StandardUnclassified). StandardDefined, /// A standard codec whose named standard holds decoders to a criterion around /// a reference instead of to identical output, for at least some conforming @@ -561,11 +852,13 @@ pub enum ReplicateClass { /// JPEG and JPEG 2000 the text read points to a criterion without stating its /// bound — T.81 leaves the accuracy of dequantisation and IDCT to its Part 2, /// which was not read, and only the table of contents and terms of ISO/IEC - /// 15444-4 were read — so no bound is given here. For AV1 and AVIF with film - /// grain it is the absence of "perceptually significant differences", whose - /// definition AV1 §7.2 places outside the specification. `replicate` over it - /// promises what that conformance promises — every conforming sink within the - /// standard's criterion — and not byte identity across sinks. + /// 15444-4 were read — so no bound is given here. For IVAS the clauses read + /// hold a non-bit-exact implementation to a stated procedure (below). For AV1, + /// AVIF and AV2 with film grain it is the absence of "perceptually significant + /// differences", whose definition AV1 §7.2 places outside the specification and + /// AV2 §7.1 leaves to others. `replicate` over it promises what that + /// conformance promises — every conforming sink within the standard's + /// criterion — and not byte identity across sinks. /// `wai.audio.opus` (RFC 6716 §6: within `opus_compare`'s thresholds of the /// reference decoder, with the same final range-decoder state); /// `wai.image.jpeg` (ITU-T T.81 §7 and A.3.3: decoding "with appropriate @@ -579,21 +872,36 @@ pub enum ReplicateClass { /// `wai.image.avif` (AV1 §7.2: identical output unless the payload signals film /// grain, and then identical frames before grain synthesis, and output frames /// that "do not have perceptually significant differences" from the reference - /// grain synthesis); `wai.audio.iamf` (its substreams - /// include Opus, and its renderers are out of its scope). + /// grain synthesis); `wai.video.av2` (AV2 v1.0.0 §7.1, the same two options); + /// `wai.audio.ivas` (3GPP TS 26.252 V19.2.0 §7.2.1: an implementation based on + /// the floating-point reference whose output is not bit-exact to the test + /// sequences is tested by a non-bit-exact procedure — TS 26.444's for mono, a + /// per-frame loudness-difference corridor for stereo and immersive operation); + /// `wai.audio.iamf` (its substreams include Opus, and its renderers are out of + /// its scope). StandardTolerance, /// A standard codec this registry has not yet classified between /// [`StandardDefined`](Self::StandardDefined) and /// [`StandardTolerance`](Self::StandardTolerance): its named standard's /// decoder-conformance clauses have not been checked. `replicate` over it /// promises what that conformance promises, and the registry does not say - /// whether that is byte identity. `wai.audio.ac4`, `wai.audio.ivas`, - /// `wai.mesh.draco`, `wai.pc.gpcc`, `wai.pc.vpcc`, `wai.splat.sog` (no frozen - /// standard yet), `wai.haptic.mihs`, and the HDR metadata descriptors - /// `wai.meta.hdr10plus` / `wai.meta.dovi`; and the asset capabilities - /// `wai.mesh.glb` and `wai.splat.spz`, whose payload bytes pass through intact - /// but whose decode is unclassified (glTF can embed JPEG images and compressed - /// geometry; SPZ stores quantised attributes). All are sink-supplied. + /// whether that is byte identity. The sink-supplied broadcast codecs whose + /// conformance documents were not read — `wai.video.evc` and + /// `wai.video.evc.baseline` (ISO/IEC 23094-1), `wai.video.avs3` (T/AI 109.2), + /// `wai.video.lcevc` (ISO/IEC 23094-2, whose decoding process includes a + /// signalled dither step), `wai.audio.mpegh` (ISO/IEC 23008-3), + /// `wai.audio.avs3` (T/AI 109.3), `wai.audio.aac` (three decoders measured on + /// one host give three different PCM streams from one ADTS stream; whether its + /// conformance admits that, or makes some of them non-conforming, was not + /// established), `wai.audio.ac4`; `wai.mesh.draco`, `wai.pc.gpcc`, + /// `wai.pc.vpcc`, `wai.splat.sog` (no frozen standard yet), `wai.haptic.mihs`, + /// and the HDR metadata descriptors `wai.meta.hdr10plus` / `wai.meta.dovi` + /// (what they reconstruct is a display-mapped picture); and the asset + /// capabilities `wai.mesh.glb` and `wai.splat.spz`, whose payload bytes pass + /// through intact but whose decode is unclassified (a glTF binary may carry JPEG + /// images in its buffers and compressed geometry through extensions, and glTF + /// specifies an asset format, not a decoder conformance over what is rendered; + /// SPZ stores quantised attributes). All are sink-supplied. StandardUnclassified, /// A neural regenerate path that is integer/fixed-point **by construction** /// — no float inference anywhere on the decode, entropy and synthesis — so it @@ -713,12 +1021,27 @@ impl ReplicateClass { } /// The [`ReplicateClass`] of a registered capability, or `None` if unknown. +/// +/// The classification is **exhaustive, with no default class**, as +/// [`capability_license`] is: every registered capability is listed under +/// exactly one arm, and one registered but listed nowhere returns `None`, which +/// the registry tests reject. A newly registered codec therefore cannot become +/// "reproducible on every machine" by omission. The basis for each standard +/// codec's class — the conformance clause read, and where a second decoder was +/// available a cross-decoder measurement — is recorded in +/// `research/media-codec-registry.md`; the basis of every +/// [`StandardDefined`](ReplicateClass::StandardDefined) member is also pinned in +/// the registry tests. pub fn capability_replicate_class(cap: &str) -> Option { use ReplicateClass::*; if !is_known_capability(cap) { return None; } Some(match cap { + // Every registered capability is listed under exactly one arm below; there + // is no default. A new registration takes the conservative class of its + // kind — StandardUnclassified for a standard codec, NeuralFloat for a neural + // decode — until evidence supports a stronger one. // The neural decodes that are integer/fixed-point by construction (all-i64, // no float/ONNX) — byte-identical everywhere. int_synth is synthesis only; // int_hyper is the FULL conditional codec (integer hyper-decoder σ source + @@ -756,26 +1079,48 @@ pub fn capability_replicate_class(cap: &str) -> Option { CAP_GENERATE_SPECTRAL => GenerativeDeterministic, // Sink-generator regeneration; conforms by similarity, not bytes. CAP_SEMANTIC_CLIP | CAP_SEMANTIC_PERCO2 => Behavioral, - // Standard codecs whose named standard holds decoders to a criterion around - // a reference, not to identical output (clauses in the variant's doc and in - // research/media-codec-registry.md): Opus (RFC 6716 §6), JPEG (T.81 §7, - // A.3.3), JPEG XL (18181-3 §4.2), JPEG 2000 (15444-4), AV1 and AVIF (AV1 - // §7.2, when film grain is signalled), IAMF (Opus substreams; renderers out - // of scope). Earlier in v0.4.0 these were StandardDefined. + + // ---- Standard classes: what the named standard's decoder conformance fixes. + // Specified exactly. Each member's basis is one row of + // STANDARD_DEFINED_EVIDENCE in the tests; an exact class without one fails + // `standard_defined_rests_on_recorded_evidence`. + // Lossless formats: the decode reproduces the encoded samples / bytes. + CAP_IMAGE_PNG | CAP_AUDIO_FLAC | CAP_TEXT_ZSTD | CAP_TEXT_XZ + // Video codecs whose decoder conformance requires every output sample to + // equal the specified decoding process's: H.264 §C.4, H.265 §C.5.1, H.266 + // §C.5.1, the VP9 specification §8.1. It covers the cropped decoded + // pictures, not processing an SEI message only signals (film grain). + | CAP_VIDEO_AVC | CAP_VIDEO_HEVC | CAP_VIDEO_VVC | CAP_VIDEO_VP9 + // WAI-native capabilities, by the extension that defines each one's + // `*-equivalence` over integer / fixed-point state. The wai.audio.scene + // mixdown is exact given its sources' decoded PCM; wai.world.model's + // equivalence covers its action trace, not its renderer's frames. + | CAP_VIDEO_INT_MOTION + | CAP_WORLD_SCENE | CAP_WORLD_REPLAY | CAP_WORLD_LIVE | CAP_WORLD_MODEL + | CAP_AUDIO_SCENE | CAP_AUDIO_SCORE | CAP_HAPTIC_SIGNAL | CAP_FEED_STATE + | CAP_AVATAR_POSE | CAP_SPLAT_4D | CAP_FILM_LINEAR | CAP_FILM_REEL + | CAP_QUANTUM_CIRCUIT | CAP_TERNARY_TT => StandardDefined, + // A criterion around a reference, not identical output (clauses in the + // variant's doc and in research/media-codec-registry.md): Opus (RFC 6716 + // §6), JPEG (T.81 §7, A.3.3), JPEG XL (18181-3 §4.2), JPEG 2000 (15444-4), + // AV1 and AVIF (AV1 §7.2, when film grain is signalled), AV2 (AV2 v1.0.0 + // §7.1, likewise), IVAS (TS 26.252 V19.2.0 §7.2.1, floating-point + // implementations), IAMF (Opus substreams; renderers out of scope). CAP_AUDIO_OPUS | CAP_IMAGE_JPEG | CAP_IMAGE_JXL | CAP_IMAGE_JP2 | CAP_VIDEO_J2K - | CAP_VIDEO_AV1 | CAP_VIDEO_AV1_LOSSLESS | CAP_IMAGE_AVIF | CAP_AUDIO_IAMF => { - StandardTolerance - } - // Sink-supplied standards whose decoder-conformance clauses the registry has - // not checked, and the two pass-through asset formats whose decode it has not - // classified. Earlier in v0.4.0 these were StandardDefined. - CAP_AUDIO_AC4 | CAP_AUDIO_IVAS | CAP_MESH_DRACO | CAP_PC_GPCC | CAP_PC_VPCC - | CAP_SPLAT_SOG | CAP_HAPTIC_MIHS | CAP_META_HDR10PLUS | CAP_META_DOVI + | CAP_VIDEO_AV1 | CAP_VIDEO_AV1_LOSSLESS | CAP_IMAGE_AVIF | CAP_VIDEO_AV2 + | CAP_AUDIO_IVAS | CAP_AUDIO_IAMF => StandardTolerance, + // Sink-supplied standards whose decoder-conformance clauses the registry + // has not checked, and the two pass-through asset formats whose decode it + // has not classified. The conservative class: it promises no byte identity. + // Moving one to StandardDefined needs a recorded basis (see above). + CAP_VIDEO_EVC | CAP_VIDEO_EVC_BASELINE | CAP_VIDEO_AVS3 | CAP_VIDEO_LCEVC + | CAP_AUDIO_MPEGH | CAP_AUDIO_AVS3 | CAP_AUDIO_AAC | CAP_AUDIO_AC4 + | CAP_MESH_DRACO | CAP_PC_GPCC | CAP_PC_VPCC | CAP_SPLAT_SOG | CAP_HAPTIC_MIHS + | CAP_META_HDR10PLUS | CAP_META_DOVI | CAP_MESH_GLB | CAP_SPLAT_SPZ => StandardUnclassified, - // Everything else: the reconstruction is specified exactly — PNG, FLAC, zstd - // and XZ by their standards, the WAI-native capabilities by the extension - // that defines their `*-equivalence`. - _ => StandardDefined, + + // Registered but unclassified: a registry defect, surfaced as `None`. + _ => return None, }) } @@ -784,7 +1129,9 @@ pub fn capability_replicate_class(cap: &str) -> Option { /// guarantees byte-identical on every machine, so a deployer can require /// `intent=replicate` to dispatch only to a cross-hardware-sound neural decode. /// Standard codecs are not neural; what `replicate` promises over them is their -/// class's [`ReplicateClass::replicate_promise`]. +/// class's [`ReplicateClass::replicate_promise`]. For the whole-registry question +/// ("does every conforming sink reconstruct the same bytes?", neural or not) use +/// [`is_cross_decoder_exact`]. pub fn is_neural_integer_exact(cap: &str) -> bool { capability_replicate_class(cap) == Some(ReplicateClass::NeuralIntegerExact) } @@ -843,7 +1190,13 @@ impl std::fmt::Display for ReplicateRefusal { /// [`ReplicateClass::StandardDefined`], the named standard's decoder conformance for /// [`ReplicateClass::StandardTolerance`] and [`ReplicateClass::StandardUnclassified`]. /// Nothing here checks a sink's decoder against its standard. Intents other than -/// `replicate` are unconstrained. +/// `replicate` are unconstrained. For a capability whose output is computed from +/// other envelopes' decoded output ([`composes_decoded_sources`]: a +/// `wai.audio.scene` mixes the PCM of the `wai.audio.*` envelopes it references), +/// this checks the one envelope it is given: its class's promise holds given its +/// sources, each of which is its own envelope with its own class, so the composite +/// is byte-exact only when every source is too +/// ([`is_cross_decoder_exact_with_sources`]). pub fn replicate_refusal( cap: &str, intent: &str, @@ -864,6 +1217,94 @@ pub fn replicate_refusal( } } +/// Whether a capability's output is computed from **other envelopes' decoded +/// output** — sources it references rather than carries — so it is only as exact +/// as they are. `wai.audio.scene` is the one such capability: its payload names +/// each object's source by the content hash of a `wai.audio.*` envelope, and its +/// mixdown is exact given the PCM those sources decode to +/// (`extensions/spatial-audio.md` §4), whatever their codec. The other WAI-native +/// capabilities hash only state they carry: a world's, a film's or an avatar's +/// assets are presentation and never enter its state hash. +pub fn composes_decoded_sources(cap: &str) -> bool { + cap == CAP_AUDIO_SCENE +} + +/// Whether the class promises byte identity for the capability's own output. +fn promises_byte_exact(cap: &str) -> bool { + capability_replicate_class(cap).and_then(ReplicateClass::replicate_promise) + == Some(ReplicatePromise::ByteExact) +} + +/// Whether every conforming sink decoding an envelope of this capability +/// reconstructs the same bytes, **from the capability alone**: its class's +/// [`ReplicateClass::replicate_promise`] is [`ReplicatePromise::ByteExact`] +/// ([`StandardDefined`](ReplicateClass::StandardDefined) or +/// [`NeuralIntegerExact`](ReplicateClass::NeuralIntegerExact)), and its output +/// does not depend on sources carried in other envelopes. A deployer that needs a +/// hash or receipt over a reconstruction to verify at another conforming sink +/// dispatches only to capabilities for which this is true. +/// +/// A capability that [`composes_decoded_sources`] is **false** here, although its +/// class is `StandardDefined`: `wai.audio.scene`'s mixdown is exact given its +/// sources' PCM, and a source decoded through `wai.audio.opus` may decode +/// differently on two conforming sinks. Once its sources' capabilities are known, +/// [`is_cross_decoder_exact_with_sources`] answers for the scene. +/// +/// It covers the output the named standard or WAI extension specifies, at that +/// output's own sample depth and layout — the cropped decoded pictures of a video +/// codec, the samples of an audio codec, the state an extension's equivalence +/// hashes. A conversion a sink applies afterwards (to RGB, to another bit depth, a +/// display mapping) is not covered; a hash over converted output is portable only +/// between sinks that also agree on the conversion. `StandardTolerance`, +/// `StandardUnclassified`, `NeuralEntropyExact`, `NeuralFloat` and `Behavioral` +/// capabilities are false; so is a generator (`GenerativeDeterministic` reproduces +/// generated content, it does not reconstruct a source). For a neural capability +/// this is the portability of the reconstruction only; whether an envelope may +/// claim `intent = replicate` over it is [`replicate_refusal`]'s rule (SPEC §7). +pub fn is_cross_decoder_exact(cap: &str) -> bool { + promises_byte_exact(cap) && !composes_decoded_sources(cap) +} + +/// Whether `source` is a capability a composing capability's payload may +/// reference: for `wai.audio.scene`, a `wai.audio.*` envelope — the asset table +/// names each object's source as one (`extensions/spatial-audio.md` §2.2). +fn is_source_of(cap: &str, source: &str) -> bool { + match cap { + CAP_AUDIO_SCENE => source.starts_with("wai.audio."), + _ => false, + } +} + +/// [`is_cross_decoder_exact`] for an envelope whose sources are known: `sources` +/// is the capability of every envelope its payload references — for a +/// `wai.audio.scene`, the envelope behind each object's `source_hash`, one entry +/// per object. True when the capability's class promises byte identity and every +/// source is a source it may reference and is itself cross-decoder exact. +/// +/// It answers **false when it cannot tell**, never true: +/// +/// - `sources` is empty. This cannot tell a scene that references no source +/// from a caller that resolved none, so a scene with no objects, whose mix is +/// silence, is also false here; +/// - a source is not one the capability may reference — for a scene, a +/// capability outside `wai.audio.*`; +/// - a source is unregistered, composes sources of its own, or is not +/// cross-decoder exact. +/// +/// The answer is still only as complete as `sources`: a source left out is not +/// checked. It covers the output spatial-audio §4 holds to mixdown-equivalence, +/// the declared output layout's canonical PCM; a binaural render is perceptual +/// there, and nothing here makes it exact. For a capability that composes +/// nothing, `sources` is ignored and this equals [`is_cross_decoder_exact`]. +pub fn is_cross_decoder_exact_with_sources(cap: &str, sources: &[&str]) -> bool { + if !composes_decoded_sources(cap) { + return is_cross_decoder_exact(cap); + } + promises_byte_exact(cap) + && !sources.is_empty() + && sources.iter().all(|s| is_source_of(cap, s) && is_cross_decoder_exact(s)) +} + /// The intent an encoder that pins **no prior** may write over `cap`: `requested`, /// or `replicate` when nothing is requested — unless [`replicate_refusal`] refuses /// that intent over `cap` with no prior, in which case the refusal comes back and @@ -907,7 +1348,8 @@ mod tests { // natively decodable — registration is not a bundled decoder. for &c in &[CAP_NEURAL_MLICPP, CAP_NEURAL_MLICV2, CAP_NEURAL_CMIC, CAP_NEURAL_SNAC, CAP_NEURAL_GLC, CAP_NEURAL_DCVC_RT, CAP_NEURAL_DCVC_FM, CAP_IMAGE_JPEGAI, - CAP_SEMANTIC_CLIP, CAP_SEMANTIC_PERCO2] { + CAP_SEMANTIC_CLIP, CAP_SEMANTIC_PERCO2, + CAP_VIDEO_AV2, CAP_VIDEO_VVC, CAP_VIDEO_LCEVC, CAP_AUDIO_MPEGH] { assert!(is_known_capability(c)); assert!(!sink_capabilities().contains(&c), "{c} is sink-supplied and must not claim native decode"); @@ -972,6 +1414,206 @@ mod tests { assert!(!is_royalty_free("wai.image.nope")); } + /// The broadcast and streaming codecs, each with the licence class recorded + /// September 2026 (sources and residual caveats: research/media-codec-registry.md) + /// and its replicate class. One row per capability so a reclassification is a + /// visible, reviewed diff. + const BROADCAST: &[(&str, LicenseClass, ReplicateClass)] = &[ + // Video whose decoder conformance requires identical decoded samples + // (H.266/H.265 §C.5.1, H.264 §C.4, VP9 §8.1). + (CAP_VIDEO_VVC, LicenseClass::PatentPool, ReplicateClass::StandardDefined), // H.266 pools + bilateral + (CAP_VIDEO_HEVC, LicenseClass::PatentPool, ReplicateClass::StandardDefined), // H.265 pools + bilateral + (CAP_VIDEO_AVC, LicenseClass::PatentPool, ReplicateClass::StandardDefined), // H.264 pool + (CAP_VIDEO_VP9, LicenseClass::RoyaltyFree, ReplicateClass::StandardDefined), // BSD + patent grant + // AV2 v1.0.0 §7.1: identical output unless film grain is signalled, then a + // perceptual criterion — as AV1. + (CAP_VIDEO_AV2, LicenseClass::RoyaltyFree, ReplicateClass::StandardTolerance), // AOMedia Patent License 1.0 + // Conformance documents not read: the conservative class. + (CAP_VIDEO_EVC, LicenseClass::PatentPool, ReplicateClass::StandardUnclassified), // Main-profile FRAND holders + (CAP_VIDEO_EVC_BASELINE, LicenseClass::RoyaltyFree, ReplicateClass::StandardUnclassified), // RF by design (caveat) + (CAP_VIDEO_AVS3, LicenseClass::PatentPool, ReplicateClass::StandardUnclassified), // AVS patent pool + (CAP_VIDEO_LCEVC, LicenseClass::PatentPool, ReplicateClass::StandardUnclassified), // SEP licence, royalties + (CAP_AUDIO_MPEGH, LicenseClass::PatentPool, ReplicateClass::StandardUnclassified), // MPEG-H 3D Audio pool + (CAP_AUDIO_AVS3, LicenseClass::PatentPool, ReplicateClass::StandardUnclassified), // AVS patent pool + // Measured: three decoders give three different PCM streams. + (CAP_AUDIO_AAC, LicenseClass::PatentPool, ReplicateClass::StandardUnclassified), // AAC pool + ]; + + #[test] + fn broadcast_codecs_are_registered_classified_and_sink_supplied() { + for &(c, class, replicate) in BROADCAST { + assert!(is_known_capability(c), "{c} not registered"); + assert_eq!(capability_license(c), Some(class), "{c} licence class"); + assert_eq!(is_royalty_free(c), class == LicenseClass::RoyaltyFree, "{c}"); + // Lossy distribution role. + assert_eq!(capability_tier(c), Some(FidelityTier::Lossy), "{c} tier"); + // Replication: pinned per row. No model is involved; the split is what + // the named standard's decoder conformance fixes, as far as it was read. + assert_eq!(capability_replicate_class(c), Some(replicate), "{c} replicate class"); + assert_eq!(is_cross_decoder_exact(c), replicate == ReplicateClass::StandardDefined, "{c}"); + // Registration is not a bundled decoder: every one is sink-supplied. + assert!(!sink_capabilities().contains(&c), "{c} must not claim native decode"); + } + // Every broadcast capability string is in the video or audio class, and + // no broadcast audio codec claims a cross-decoder-exact reconstruction. + for &(c, _, replicate) in BROADCAST { + assert!(c.starts_with("wai.video.") || c.starts_with("wai.audio."), "{c}"); + if c.starts_with("wai.audio.") { + assert_ne!(replicate, ReplicateClass::StandardDefined, "{c}"); + } + } + } + + #[test] + fn licence_classification_is_exhaustive_with_no_default() { + // Every registered capability is classified by an explicit arm... + for &c in known_capabilities() { + assert!(capability_license(c).is_some(), "{c} is registered but unclassified"); + } + // ...and the non-royalty-free set is exactly what the registry says it + // is — adding a pool / proprietary / non-commercial capability, or + // reclassifying one, has to change this list in review. + let mut non_rf: Vec<&str> = known_capabilities().iter().copied() + .filter(|c| !is_royalty_free(c)).collect(); + non_rf.sort_unstable(); + let mut want = vec![ + CAP_AUDIO_AC4, CAP_META_DOVI, + CAP_AUDIO_IVAS, CAP_PC_GPCC, CAP_PC_VPCC, CAP_HAPTIC_MIHS, + CAP_SEMANTIC_PERCO2, + CAP_VIDEO_VVC, CAP_VIDEO_HEVC, CAP_VIDEO_AVC, CAP_VIDEO_EVC, CAP_VIDEO_LCEVC, + CAP_VIDEO_AVS3, CAP_AUDIO_MPEGH, CAP_AUDIO_AVS3, CAP_AUDIO_AAC, + ]; + want.sort_unstable(); + assert_eq!(non_rf, want); + } + + #[test] + fn defaults_are_royalty_free_and_never_pool_or_proprietary() { + // The floor and the recommended set are the defaults; the native sink + // decodes exactly those. SPEC §5 and the code agree on the list. + let mut defaults: Vec<&str> = mandatory_floor().iter().chain(recommended_set()).copied().collect(); + defaults.sort_unstable(); + let mut native: Vec<&str> = sink_capabilities().to_vec(); + native.sort_unstable(); + assert_eq!(defaults, native, "floor ∪ recommended must equal the native sink set"); + // Every default is royalty-free. + for &c in &defaults { + assert_eq!(capability_license(c), Some(LicenseClass::RoyaltyFree), "default {c}"); + assert!(!is_companion_only(c), "default {c} must be a standalone payload"); + } + // Stated the other way round: no pool, proprietary or non-commercial + // capability — including every new broadcast codec — is a default. + for &c in known_capabilities() { + if !is_royalty_free(c) { + assert!(!defaults.contains(&c), "{c} is not royalty-free and must never be a default"); + } + } + // Royalty-free is necessary, not sufficient: AV2 is royalty-free but + // deliberately NOT a default until deployable decode exists (SPEC §5). + assert!(is_royalty_free(CAP_VIDEO_AV2)); + assert!(!defaults.contains(&CAP_VIDEO_AV2)); + } + + #[test] + fn companion_only_and_enhancement_layers() { + // Companion-only: HDR dynamic metadata and the LCEVC enhancement layer. + for &c in &[CAP_META_HDR10PLUS, CAP_META_DOVI, CAP_VIDEO_LCEVC] { + assert!(is_known_capability(c)); + assert!(is_companion_only(c), "{c} is companion-only"); + } + // Enhancement layers are exactly LCEVC, and every one is companion-only. + let enh: Vec<&str> = known_capabilities().iter().copied() + .filter(|c| is_enhancement_layer(c)).collect(); + assert_eq!(enh, vec![CAP_VIDEO_LCEVC]); + for &c in &enh { + assert!(is_companion_only(c)); + } + // LCEVC's registered base codecs are standalone payloads. + for &c in &[CAP_VIDEO_AVC, CAP_VIDEO_HEVC, CAP_VIDEO_VVC, CAP_VIDEO_EVC, CAP_VIDEO_VP9, CAP_VIDEO_AV1] { + assert!(!is_companion_only(c), "{c} is a base, not a companion"); + } + // IAMF carries its own audio substreams: standalone, never companion-only. + assert!(!is_companion_only(CAP_AUDIO_IAMF)); + assert!(!is_companion_only("wai.image.nope")); + } + + /// SPEC §5 and §7.1, rule 2: what each companion-only capability applies to. + /// Pinned in both directions, so widening or narrowing a base set is a + /// reviewed change to this test. + #[test] + fn companions_apply_only_to_the_bases_their_registration_names() { + // The video media class, pinned: every standalone wai.video.* capability, + // the neural video codecs SPEC §5 lists under *Video:*, and the films. + let mut video: Vec<&str> = known_capabilities().iter().copied() + .filter(|c| c.starts_with("wai.video.") && !is_companion_only(c)).collect(); + assert!(video.contains(&CAP_VIDEO_J2K) && video.contains(&CAP_VIDEO_AV2)); + video.extend([CAP_NEURAL_VIDEO_BMSHJ2018, CAP_NEURAL_DCVC_RT, CAP_NEURAL_DCVC_FM, + CAP_FILM_LINEAR, CAP_FILM_REEL]); + video.sort_unstable(); + let mut class: Vec<&str> = known_capabilities().iter().copied() + .filter(|c| is_video_class(c)).collect(); + class.sort_unstable(); + assert_eq!(class, video, "the video media class"); + + // HDR dynamic metadata was registered as a companion to a video payload: + // it applies to every video-class capability and to nothing else. + for &hdr in &[CAP_META_HDR10PLUS, CAP_META_DOVI] { + for &base in known_capabilities() { + let want = video.contains(&base); + assert_eq!(companion_applies_to(hdr, base), want, "{hdr} on {base}"); + } + // A base this revision does not register is not constrained: its class + // is unknown here, so a later revision's video codec, in any namespace, + // is already accepted. + for base in ["wai.video.future", "wai.neural.video_future"] { + assert!(!is_video_class(base)); + assert!(companion_applies_to(hdr, base), "{hdr} on unregistered {base}"); + } + // Never an audio, image, volumetric or world payload, nor a companion. + for base in [CAP_AUDIO_OPUS, CAP_AUDIO_AAC, CAP_AUDIO_IAMF, CAP_IMAGE_AVIF, + CAP_IMAGE_JP2, CAP_NEURAL_GLC, CAP_SPLAT_4D, CAP_WORLD_REPLAY, + CAP_PC_VPCC, CAP_VIDEO_LCEVC, CAP_META_DOVI] { + assert!(!companion_applies_to(hdr, base), "{hdr} must not apply to {base}"); + } + } + + // LCEVC: exactly the base codecs SPEC §5 names, among registered capabilities. + let mut lcevc: Vec<&str> = known_capabilities().iter().copied() + .filter(|b| companion_applies_to(CAP_VIDEO_LCEVC, b)).collect(); + lcevc.sort_unstable(); + let mut want = vec![ + CAP_VIDEO_AVC, CAP_VIDEO_HEVC, CAP_VIDEO_VVC, CAP_VIDEO_EVC, CAP_VIDEO_EVC_BASELINE, + CAP_VIDEO_VP9, CAP_VIDEO_AV1, CAP_VIDEO_AV1_LOSSLESS, + ]; + want.sort_unstable(); + assert_eq!(lcevc, want); + assert_eq!(enhancement_bases(CAP_VIDEO_LCEVC).len(), want.len()); + // Registered video codecs outside LCEVC's registered bases, and an unregistered one. + for base in [CAP_VIDEO_J2K, CAP_VIDEO_AV2, CAP_VIDEO_AVS3, CAP_VIDEO_INT_MOTION, + CAP_VIDEO_INT_HYPER, CAP_VIDEO_LCEVC, "wai.video.future"] { + assert!(!companion_applies_to(CAP_VIDEO_LCEVC, base), "LCEVC must not apply to {base}"); + } + // Every base is a registered, standalone video capability. + for &b in enhancement_bases(CAP_VIDEO_LCEVC) { + assert!(is_known_capability(b) && !is_companion_only(b) && b.starts_with("wai.video."), "{b}"); + } + assert!(enhancement_bases(CAP_META_HDR10PLUS).is_empty()); + assert!(enhancement_bases(CAP_VIDEO_HEVC).is_empty()); + + // Rule 1: a standalone capability applies to nothing — it is never a + // companion. Neither does a capability this revision does not register. + for &c in known_capabilities() { + if is_companion_only(c) { + assert!(known_capabilities().iter().any(|b| companion_applies_to(c, b)), + "companion-only {c} applies to no registered capability"); + } else { + assert!(!known_capabilities().iter().any(|b| companion_applies_to(c, b)), + "standalone {c} must apply to nothing"); + } + } + assert!(!companion_applies_to("wai.meta.future", CAP_VIDEO_AV1)); + } + #[test] fn replicate_class_isolates_the_integer_entropy_contract() { use ReplicateClass::*; @@ -1075,13 +1717,18 @@ mod tests { }; let sorted = |mut v: Vec<&'static str>| { v.sort_unstable(); v }; // Opus (RFC 6716 §6), JPEG (T.81 §7, A.3.3), JPEG XL (18181-3 §4.2), - // JPEG 2000 (15444-4), AV1/AVIF (AV1 §7.2 with film grain), IAMF. + // JPEG 2000 (15444-4), AV1/AVIF (AV1 §7.2 with film grain), AV2 (AV2 + // v1.0.0 §7.1, likewise), IVAS (TS 26.252 V19.2.0 §7.2.1), IAMF. assert_eq!(of(StandardTolerance), sorted(vec![ CAP_AUDIO_OPUS, CAP_IMAGE_JPEG, CAP_IMAGE_JXL, CAP_IMAGE_JP2, CAP_VIDEO_J2K, - CAP_VIDEO_AV1, CAP_VIDEO_AV1_LOSSLESS, CAP_IMAGE_AVIF, CAP_AUDIO_IAMF])); - // The sink-supplied standards and asset formats not yet classified. + CAP_VIDEO_AV1, CAP_VIDEO_AV1_LOSSLESS, CAP_IMAGE_AVIF, CAP_VIDEO_AV2, + CAP_AUDIO_IVAS, CAP_AUDIO_IAMF])); + // The sink-supplied standards and asset formats not yet classified — + // including every broadcast codec whose conformance document was not read. assert_eq!(of(StandardUnclassified), sorted(vec![ - CAP_AUDIO_AC4, CAP_AUDIO_IVAS, CAP_MESH_DRACO, CAP_PC_GPCC, CAP_PC_VPCC, + CAP_VIDEO_EVC, CAP_VIDEO_EVC_BASELINE, CAP_VIDEO_AVS3, CAP_VIDEO_LCEVC, + CAP_AUDIO_MPEGH, CAP_AUDIO_AVS3, CAP_AUDIO_AAC, CAP_AUDIO_AC4, + CAP_MESH_DRACO, CAP_PC_GPCC, CAP_PC_VPCC, CAP_SPLAT_SOG, CAP_HAPTIC_MIHS, CAP_META_HDR10PLUS, CAP_META_DOVI, CAP_MESH_GLB, CAP_SPLAT_SPZ])); // Every natively decoded standard codec is classified: none is unclassified. @@ -1161,6 +1808,147 @@ mod tests { assert!(why.contains("entropy-consistency"), "{why}"); } + /// Every capability classed StandardDefined, with the evidence its class rests + /// on. The class promises the same bytes on every conforming sink (SPEC §7), so + /// a capability enters it only with a recorded basis: a conformance clause + /// that requires identical output, or the WAI extension whose equivalence pins + /// the reconstruction. Adding a row is a reviewed change. A capability classed + /// StandardDefined without a row here fails + /// `standard_defined_rests_on_recorded_evidence`; one that has no such basis + /// belongs in StandardUnclassified. + const STANDARD_DEFINED_EVIDENCE: &[(&str, &str)] = &[ + (CAP_IMAGE_PNG, "lossless: decoding restores the encoded samples"), + (CAP_AUDIO_FLAC, "lossless: decoding restores the encoded samples"), + (CAP_TEXT_ZSTD, "lossless byte format: decoding restores the encoded bytes"), + (CAP_TEXT_XZ, "lossless byte format: decoding restores the encoded bytes"), + (CAP_VIDEO_AVC, "ITU-T H.264 (V16, 06/2026) §C.4: every output sample equals the HRD's"), + (CAP_VIDEO_HEVC, "ITU-T H.265 (V11, 01/2026) §C.5.1: every output sample equals the specified decoding process's"), + (CAP_VIDEO_VVC, "ITU-T H.266 (V4, 01/2026) §C.5.1: every output sample equals the specified decoding process's"), + (CAP_VIDEO_VP9, "VP9 Bitstream & Decoding Process Specification v0.6 §8.1: output frames identical in all respects"), + (CAP_VIDEO_INT_MOTION, "int_video.rs: pure-integer decode, frame-state-equivalence"), + (CAP_WORLD_SCENE, "extensions/interactive-worlds.md: replay-equivalence"), + (CAP_WORLD_REPLAY, "extensions/interactive-worlds.md: replay-equivalence"), + (CAP_WORLD_LIVE, "extensions/interactive-worlds.md: replay-equivalence"), + (CAP_WORLD_MODEL, "world_model.rs: replay-equivalence of the action trace (not the rendered frames)"), + (CAP_AUDIO_SCENE, "extensions/spatial-audio.md: mixdown-equivalence, given its sources' decoded PCM"), + (CAP_AUDIO_SCORE, "extensions/score.md: mixdown-equivalence"), + (CAP_HAPTIC_SIGNAL, "extensions/haptics.md: sample-equivalence"), + (CAP_FEED_STATE, "extensions/state-feeds.md: replay-equivalence"), + (CAP_AVATAR_POSE, "extensions/avatars.md: pose-equivalence"), + (CAP_SPLAT_4D, "extensions/volumetric.md: frame-state-equivalence"), + (CAP_FILM_LINEAR, "extensions/linear-film.md: film-equivalence"), + (CAP_FILM_REEL, "extensions/linear-film.md: reel-equivalence"), + (CAP_QUANTUM_CIRCUIT, "extensions/quantum-sim.md: statevector-equivalence"), + (CAP_TERNARY_TT, "extensions/ternary-tt.md: reconstruction-equivalence"), + ]; + + #[test] + fn standard_defined_rests_on_recorded_evidence() { + use ReplicateClass::*; + // Every registered capability is classified by an explicit arm (the + // function has no default class; an unlisted one returns None). + for &c in known_capabilities() { + assert!(capability_replicate_class(c).is_some(), "{c} is registered but has no replicate class"); + } + // The exact class is exactly the capabilities with a recorded basis. + for &c in known_capabilities() { + if capability_replicate_class(c) == Some(StandardDefined) { + assert!(STANDARD_DEFINED_EVIDENCE.iter().any(|&(e, _)| e == c), + "{c} is classed StandardDefined with no recorded evidence: record the \ + conformance clause or extension that fixes its output, or class it \ + StandardUnclassified"); + } + } + for &(c, basis) in STANDARD_DEFINED_EVIDENCE { + assert_eq!(capability_replicate_class(c), Some(StandardDefined), "{c}: {basis}"); + assert!(!basis.is_empty(), "{c}"); + } + let mut rows: Vec<&str> = STANDARD_DEFINED_EVIDENCE.iter().map(|&(c, _)| c).collect(); + rows.sort_unstable(); + rows.dedup(); + assert_eq!(rows.len(), STANDARD_DEFINED_EVIDENCE.len(), "duplicate evidence row"); + + // is_cross_decoder_exact is exactly StandardDefined ∪ NeuralIntegerExact, + // less the capabilities whose output depends on other envelopes' decode. + for &c in known_capabilities() { + let class = capability_replicate_class(c).unwrap(); + assert_eq!(is_cross_decoder_exact(c), + matches!(class, StandardDefined | NeuralIntegerExact) && !composes_decoded_sources(c), + "{c}"); + } + assert!(!is_cross_decoder_exact("wai.image.nope")); + // The mandatory floor reconstructs identically at every conforming sink. + for &c in mandatory_floor() { + assert!(is_cross_decoder_exact(c), "floor {c}"); + } + // The same family can land on either side: the decoded pictures of the + // ITU-T video codecs are fixed by their conformance clauses, the AAC PCM + // is not established, and film-grain AV1 is held to a perceptual criterion. + assert!(is_cross_decoder_exact(CAP_VIDEO_AVC) && !is_cross_decoder_exact(CAP_AUDIO_AAC)); + assert!(is_cross_decoder_exact(CAP_VIDEO_VP9) && !is_cross_decoder_exact(CAP_VIDEO_AV1)); + assert!(is_cross_decoder_exact(CAP_AUDIO_FLAC) && !is_cross_decoder_exact(CAP_AUDIO_IAMF)); + // The two classifications round one got wrong are not exact: a glTF binary + // may embed JPEG images and compressed geometry, and JPEG AI's reconstruction + // runs float stages that its standard does not fix. + assert!(!is_cross_decoder_exact(CAP_MESH_GLB)); + assert!(!is_cross_decoder_exact(CAP_IMAGE_JPEGAI)); + // Neither neural float nor a generator nor similarity is exact. + assert!(!is_cross_decoder_exact(CAP_NEURAL_DCVC_FM)); + assert!(!is_cross_decoder_exact(CAP_GENERATE_SPECTRAL)); + assert!(!is_cross_decoder_exact(CAP_SEMANTIC_CLIP)); + } + + /// A `wai.audio.scene` mixes the decoded PCM of sources carried in other + /// envelopes, so its class (StandardDefined: the mix is exact given that PCM) + /// does not make it portable by itself. + #[test] + fn a_scene_is_only_as_exact_as_its_sources() { + // The one capability that composes other envelopes' decoded output. + let composing: Vec<&str> = known_capabilities().iter().copied() + .filter(|c| composes_decoded_sources(c)).collect(); + assert_eq!(composing, vec![CAP_AUDIO_SCENE]); + assert_eq!(capability_replicate_class(CAP_AUDIO_SCENE), Some(ReplicateClass::StandardDefined)); + + // Not portable from the capability alone... + assert!(!is_cross_decoder_exact(CAP_AUDIO_SCENE)); + // ...portable when every source is: lossless FLAC, the integer audio codec. + assert!(is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_AUDIO_FLAC])); + assert!(is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_AUDIO_FLAC, CAP_AUDIO_INT_CODEC])); + // One source whose standard admits a tolerance, or is unclassified, or is + // unregistered, and the mix is not. + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_AUDIO_FLAC, CAP_AUDIO_OPUS])); + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_AUDIO_AAC])); + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_NEURAL_ENCODEC32])); + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &["wai.audio.nope"])); + // A scene used as a source is not portable by itself either. + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_AUDIO_SCENE])); + // It answers false when it cannot tell. No sources: a scene with no + // objects cannot be told from a caller that resolved none. + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[])); + // A source outside wai.audio.* is not a scene source (spatial-audio §2.2), + // even when its own class is exact: a PNG image, AVC video, a WAI world. + for bad in [CAP_IMAGE_PNG, CAP_VIDEO_AVC, CAP_WORLD_REPLAY] { + assert!(is_cross_decoder_exact(bad), "{bad} is exact on its own"); + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[bad]), "{bad}"); + assert!(!is_cross_decoder_exact_with_sources(CAP_AUDIO_SCENE, &[CAP_AUDIO_FLAC, bad]), "{bad}"); + } + + // A capability that composes nothing ignores `sources`. + for &c in known_capabilities() { + if !composes_decoded_sources(c) { + assert_eq!(is_cross_decoder_exact_with_sources(c, &[CAP_AUDIO_OPUS]), is_cross_decoder_exact(c), "{c}"); + assert_eq!(is_cross_decoder_exact_with_sources(c, &[]), is_cross_decoder_exact(c), "{c}"); + } + } + + // The replicate gate checks the one envelope it is given; the sources are + // envelopes of their own, each gated by its own class. + assert_eq!(replicate_refusal(CAP_AUDIO_SCENE, "replicate", None), None); + assert_eq!(replicate_refusal(CAP_AUDIO_FLAC, "replicate", None), None); + assert_eq!(replicate_refusal(CAP_AUDIO_OPUS, "replicate", None), None, + "opus backs replicate at its standard's tolerance, not byte identity"); + } + /// `wai wrap` pins no prior, and used to write `intent = replicate` over every /// capability, `wai.image.jpegai` included, against SPEC §7's MUST NOT. The /// intent it writes now comes from `unpinned_envelope_intent`, which must agree @@ -1228,5 +2016,17 @@ mod tests { CAP_TEXT_ZSTD, CAP_TEXT_XZ] { assert_eq!(unpinned_envelope_intent(c, None), Ok("replicate"), "{c}"); } + + // `wai wrap --companion` holds the intent over the base payload's + // capability and each companion's. Every companion-only capability + // registered today is in a standard class, so the companion check refuses + // nothing yet; this fails when one is registered in another class, which + // is when that check needs a test of its own. + let companions: Vec<&str> = known_capabilities().iter().copied() + .filter(|c| is_companion_only(c)).collect(); + assert!(!companions.is_empty()); + for c in companions { + assert_eq!(unpinned_envelope_intent(c, None), Ok("replicate"), "{c}"); + } } } diff --git a/wai/wai-rs/src/container.rs b/wai/wai-rs/src/container.rs index f9db68ba3..6342ed8d1 100644 --- a/wai/wai-rs/src/container.rs +++ b/wai/wai-rs/src/container.rs @@ -241,6 +241,29 @@ impl Wai { self.manifest.model_requirement.fallback.as_deref() } + /// SPEC §4 dispatch for this envelope at a sink that advertises the + /// capabilities `advertises` accepts: the capability whose codec decodes the + /// payload, or `None` if the envelope is inert there. Step 3 takes the + /// primary capability whenever the sink advertises it, whatever the fallback + /// names; step 5 takes the fallback only if the primary is not advertised. + /// A companion-only capability ([`crate::codecs::is_companion_only`]) is + /// never dispatched standalone (SPEC §5): named as the primary, the payload + /// is companion data and the envelope is inert; named as the fallback, step 5 + /// is skipped, and step 3 is unaffected. So an envelope that + /// [`Self::check_component_roles`] reports for its fallback alone still + /// decodes at a sink that advertises its primary capability. + pub fn dispatch_capability(&self, advertises: impl Fn(&str) -> bool) -> Option<&str> { + let usable = |c: &str| advertises(c) && !crate::codecs::is_companion_only(c); + let cap = self.capability(); + if crate::codecs::is_companion_only(cap) { + return None; + } + if usable(cap) { + return Some(cap); + } + self.fallback().filter(|&fb| usable(fb)) + } + pub fn media(&self) -> &str { &self.manifest.media } @@ -285,6 +308,28 @@ impl Wai { self.prior(), ) } + + /// Check this v1.0 envelope against the component-role rules of SPEC §5. + /// A companion-only capability ([`crate::codecs::is_companion_only`]: HDR + /// dynamic metadata, an LCEVC enhancement) means nothing without the base + /// payload it accompanies, and a `WAI1` carries exactly one payload, so it + /// MUST NOT be the envelope's `capability` — nor its `fallback`, because + /// dispatch step 5 would decode the payload as that capability on its own. + /// Parsing ([`Self::from_bytes`]) stays format-agnostic, so a sink or tool + /// runs this before dispatch. The v1.1 counterpart is + /// [`WaiMulti::check_component_roles`]. + pub fn check_component_roles(&self) -> Result<(), RoleError> { + let capability = self.capability(); + if crate::codecs::is_companion_only(capability) { + return Err(RoleError::CompanionOnlyAsCapability { capability: capability.to_string() }); + } + if let Some(fb) = self.fallback() { + if crate::codecs::is_companion_only(fb) { + return Err(RoleError::CompanionOnlyAsFallback { capability: fb.to_string() }); + } + } + Ok(()) + } } // ---- v1.1: multi-rendition envelope (WAI2) ---------------------- @@ -304,11 +349,13 @@ pub struct RenditionMeta { #[serde(default, skip_serializing_if = "Option::is_none")] pub prior: Option, /// Component role. Absent ⇒ `"rendition"` (a selectable alternative of the - /// same content — the v1.1 default). `"companion"` marks a descriptor that - /// is applied *with* the chosen rendition, not instead of it: HDR dynamic - /// metadata (`wai.meta.hdr10plus` / `wai.meta.dovi`) or an immersive-audio - /// metadata stream that accompanies a video/audio payload. A companion is - /// never offered to `pick`/`primary`; it is bound to the component named in + /// same content — the v1.1 default). `"companion"` marks a component that + /// is applied *with* the chosen rendition, not instead of it: a + /// companion-only capability ([`crate::codecs::is_companion_only`]) — HDR + /// dynamic metadata (`wai.meta.hdr10plus` / `wai.meta.dovi`) describing a + /// video payload, or an LCEVC enhancement layer added to one. A standalone + /// capability is never a companion (SPEC §7.1). A companion is never offered + /// to `pick`/`primary`; it is bound to the component named in /// [`Self::companion_of`]. #[serde(default, skip_serializing_if = "Option::is_none")] pub role: Option, @@ -325,8 +372,97 @@ impl RenditionMeta { pub fn is_companion(&self) -> bool { self.role.as_deref() == Some("companion") } + + /// Whether `pick`/`primary` may select this component: it is not marked a + /// companion **and** its capability is not registered companion-only + /// ([`crate::codecs::is_companion_only`]). An HDR metadata block or an + /// LCEVC enhancement mislabelled as a rendition is still never selected — + /// it has no picture or sound of its own. [`WaiMulti::check_component_roles`] + /// reports the mislabelling. + pub fn is_selectable(&self) -> bool { + !self.is_companion() && !crate::codecs::is_companion_only(&self.capability) + } } +/// An envelope that breaks the component-role rules of SPEC §5 / §7.1. +/// Returned by [`Wai::check_component_roles`] (v1.0) and +/// [`WaiMulti::check_component_roles`] (v1.1); `index` is the component's +/// position in a `WAI2` `manifest.renditions`. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub enum RoleError { + /// v1.0: a companion-only capability named as the `WAI1` `capability`. A + /// `WAI1` carries one payload, so there is no base for it to accompany. + CompanionOnlyAsCapability { capability: String }, + /// v1.0: a companion-only capability named as the `WAI1` `fallback`. + /// Dispatch step 5 would decode the payload as that capability on its own. + CompanionOnlyAsFallback { capability: String }, + /// A companion-only capability (HDR dynamic metadata, an enhancement + /// layer) listed as a selectable rendition instead of as a companion. + CompanionOnlyAsRendition { index: usize, capability: String }, + /// A registered standalone capability (one that is not companion-only, such + /// as `wai.audio.iamf` or `wai.video.hevc`) marked `role = "companion"`. Its + /// payload decodes on its own, so it is carried as a rendition; as a + /// companion no sink would ever select it, and applying it to another + /// rendition has no meaning. + StandaloneAsCompanion { index: usize, capability: String }, + /// A companion without `companion_of`, or whose `companion_of` names no + /// selectable rendition in this envelope — there is nothing to apply it to. + UnboundCompanion { index: usize, capability: String }, + /// A companion whose `companion_of` names a capability it does not apply to + /// ([`crate::codecs::companion_applies_to`]): an HDR dynamic-metadata + /// descriptor bound to a payload outside the `video` media class + /// ([`crate::codecs::is_video_class`]), or an enhancement layer bound to a + /// codec outside its [`crate::codecs::enhancement_bases`]. + IncompatibleBase { index: usize, capability: String, base: String }, + /// An enhancement layer whose `companion_of` names one of its base codecs + /// but does not resolve to exactly one selectable rendition of it. Its + /// residuals were encoded against one specific base encode; with zero + /// matches there is no base, and with more than one the binding is + /// ambiguous and applying it to the wrong base corrupts the picture. A + /// `companion_of` outside its base codecs is [`RoleError::IncompatibleBase`]. + EnhancementBaseNotUnique { index: usize, capability: String, matches: usize }, + /// A second enhancement layer bound to a base that already has one (the + /// component at `first`). A sink applies every companion bound to the + /// rendition it picked, so two sets of residuals would be stacked on one + /// picture; each was encoded to be added once, and together they corrupt it. + DuplicateEnhancementLayer { index: usize, capability: String, base: String, first: usize }, +} + +impl std::fmt::Display for RoleError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + RoleError::CompanionOnlyAsCapability { capability } => write!( + f, "{capability} is companion-only and cannot be a WAI1 capability; carry it as a WAI2 companion bound to its base rendition"), + RoleError::CompanionOnlyAsFallback { capability } => write!( + f, "{capability} is companion-only and cannot be a WAI1 fallback; it is never dispatched on its own"), + RoleError::DuplicateEnhancementLayer { index, capability, base, first } => write!( + f, "enhancement layer {index} ({capability}) binds to {base}, which already has enhancement layer {first}; a base takes at most one"), + RoleError::CompanionOnlyAsRendition { index, capability } => write!( + f, "component {index} ({capability}) is companion-only but is not marked role=\"companion\""), + RoleError::StandaloneAsCompanion { index, capability } => write!( + f, "component {index} ({capability}) is standalone media, not a companion; carry it as a rendition"), + RoleError::UnboundCompanion { index, capability } => write!( + f, "companion {index} ({capability}) names no selectable rendition in companion_of"), + RoleError::IncompatibleBase { index, capability, base } => { + let bases = crate::codecs::enhancement_bases(capability); + if bases.is_empty() { + write!(f, "companion {index} ({capability}) does not apply to {base}; \ + it describes a video payload (a wai.video.* capability, \ + a neural video codec or a film)") + } else { + write!(f, "companion {index} ({capability}) does not apply to {base}; \ + its base codecs are {}", bases.join(", ")) + } + } + RoleError::EnhancementBaseNotUnique { index, capability, matches } => write!( + f, "enhancement layer {index} ({capability}) must bind to exactly one selectable video rendition; companion_of matches {matches}"), + } + } +} + +impl std::error::Error for RoleError {} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct ManifestV2 { pub wai: String, // "1.1" @@ -511,11 +647,12 @@ impl WaiMulti { Ok(Self { manifest, renditions }) } - /// Pick the first *selectable rendition* (never a companion) whose + /// Pick the first *selectable rendition* (never a companion, never a + /// companion-only capability — see [`RenditionMeta::is_selectable`]) whose /// capability matches `pred`. pub fn pick bool>(&self, pred: F) -> Option<(&RenditionMeta, &[u8])> { for (meta, payload) in self.manifest.renditions.iter().zip(self.renditions.iter()) { - if !meta.is_companion() && pred(&meta.capability) { + if meta.is_selectable() && pred(&meta.capability) { return Some((meta, payload)); } } @@ -523,15 +660,103 @@ impl WaiMulti { } /// First selectable rendition in deployer-preferred order (the canonical - /// default pick when the sink has no further policy). Skips companions. + /// default pick when the sink has no further policy). Skips companions + /// and companion-only capabilities. pub fn primary(&self) -> Option<(&RenditionMeta, &[u8])> { self.manifest.renditions.iter().zip(self.renditions.iter()) - .find(|(m, _)| !m.is_companion()) + .find(|(m, _)| m.is_selectable()) .map(|(m, p)| (m, p.as_slice())) } - /// All companion descriptors (HDR / audio metadata) carried in this - /// envelope — applied *with* the chosen rendition, not instead of it. + /// Check the component table against the role rules of SPEC §5 / §7.1, + /// using the capability registry. Parsing ([`Self::from_bytes`]) stays + /// format-agnostic — an unknown capability is still carried — so a sink + /// runs this before applying companions. Checked in component order (the + /// first violation is returned): + /// + /// - a companion-only capability ([`crate::codecs::is_companion_only`]) + /// MUST be carried with `role = "companion"` + /// ([`RoleError::CompanionOnlyAsRendition`]); + /// + /// and, for each `role = "companion"` component, SPEC §7.1's three rules: + /// + /// 1. its capability is companion-only or unregistered: a registered + /// standalone capability is a rendition, never a companion + /// ([`RoleError::StandaloneAsCompanion`]); + /// 2. `companion_of` is present ([`RoleError::UnboundCompanion`]) and, for a + /// registered companion, names a capability it applies to + /// ([`crate::codecs::companion_applies_to`]: for HDR dynamic metadata a + /// video-class payload ([`crate::codecs::is_video_class`]) or one this + /// revision does not register, for an enhancement layer one of its + /// [`crate::codecs::enhancement_bases`]; [`RoleError::IncompatibleBase`]); + /// 3. a selectable rendition of that capability is present + /// ([`RoleError::UnboundCompanion`]); for an enhancement layer + /// ([`crate::codecs::is_enhancement_layer`], e.g. `wai.video.lcevc`) + /// **exactly one** — the base encode it was produced against + /// ([`RoleError::EnhancementBaseNotUnique`]) — and that base takes **at + /// most one** enhancement layer ([`RoleError::DuplicateEnhancementLayer`]). + /// A sink applies every companion bound to the rendition it picked + /// ([`Self::companions_of`]), so a second one would stack a second set of + /// residuals on the picture. + /// + /// A descriptor companion (HDR dynamic metadata) may bind to several + /// renditions of one capability, and several descriptors may bind to one + /// base; only enhancement layers are unique in both directions. A companion + /// whose capability this revision does not register is held to rule 3's + /// presence check only: its bases are not known here, and a sink that does + /// not recognise it does not apply it. The v1.0 counterpart is + /// [`Wai::check_component_roles`]. + pub fn check_component_roles(&self) -> Result<(), RoleError> { + let metas = &self.manifest.renditions; + for (index, m) in metas.iter().enumerate() { + let capability = m.capability.clone(); + let companion_only = crate::codecs::is_companion_only(&m.capability); + if !m.is_companion() { + if companion_only { + return Err(RoleError::CompanionOnlyAsRendition { index, capability }); + } + continue; + } + // Rule 1: only a companion-only (or unregistered) capability is a companion. + if !companion_only && crate::codecs::is_known_capability(&m.capability) { + return Err(RoleError::StandaloneAsCompanion { index, capability }); + } + let Some(target) = m.companion_of.as_deref() else { + return Err(RoleError::UnboundCompanion { index, capability }); + }; + // Rule 2: the named base is one this companion applies to. + if companion_only && !crate::codecs::companion_applies_to(&m.capability, target) { + let base = target.to_string(); + return Err(RoleError::IncompatibleBase { index, capability, base }); + } + // Rule 3: the base is present, and unique for an enhancement layer. + let bases = metas.iter() + .filter(|b| b.is_selectable() && b.capability == target) + .count(); + if crate::codecs::is_enhancement_layer(&m.capability) { + if bases != 1 { + return Err(RoleError::EnhancementBaseNotUnique { index, capability, matches: bases }); + } + // The base resolved uniquely, so an earlier enhancement layer + // naming the same capability binds to the same base encode. + let earlier = metas[..index].iter().position(|e| { + e.is_companion() + && crate::codecs::is_enhancement_layer(&e.capability) + && e.companion_of.as_deref() == Some(target) + }); + if let Some(first) = earlier { + let base = target.to_string(); + return Err(RoleError::DuplicateEnhancementLayer { index, capability, base, first }); + } + } else if bases == 0 { + return Err(RoleError::UnboundCompanion { index, capability }); + } + } + Ok(()) + } + + /// All companions (HDR dynamic metadata, enhancement layers) carried in + /// this envelope — applied *with* the chosen rendition, not instead of it. pub fn companions(&self) -> Vec<(&RenditionMeta, &[u8])> { self.manifest.renditions.iter().zip(self.renditions.iter()) .filter(|(m, _)| m.is_companion()) @@ -539,8 +764,16 @@ impl WaiMulti { .collect() } - /// The companion descriptors bound to a given component capability — e.g. - /// the HDR10+ metadata for the `wai.video.av1` rendition the sink picked. + /// The companions bound to a given component capability — e.g. the HDR10+ + /// metadata for the `wai.video.av1` rendition the sink picked. Each one + /// returned names `capability` in `companion_of`; that alone does not make + /// it applicable. A sink applies one only if the envelope passes + /// [`Self::check_component_roles`] (which also guarantees at most one + /// enhancement layer per base) **and** it recognises the companion's + /// capability (SPEC §7.1). The role check accepts a companion whose + /// capability this revision does not register, provided its base is + /// present, and this returns it; a sink that does not recognise that + /// capability MUST NOT apply it and may still present the rendition. pub fn companions_of(&self, capability: &str) -> Vec<(&RenditionMeta, &[u8])> { self.companions().into_iter() .filter(|(m, _)| m.companion_of.as_deref() == Some(capability)) @@ -743,9 +976,9 @@ mod tests { #[test] fn companion_descriptors_bind_to_a_primary() { - // one HDR video payload (two codec renditions of it) + an HDR10+ - // dynamic-metadata companion bound to the AV1 rendition + an immersive - // audio metadata companion. One envelope, intrinsic binding. + // one HDR video payload (two codec renditions of it) + two HDR + // dynamic-metadata companions (HDR10+ and Dolby Vision) bound to the AV1 + // rendition. One envelope, intrinsic binding. let m = WaiMulti::new( "video", "replicate", @@ -754,7 +987,7 @@ mod tests { Rendition::media("wai.video.av1", "av1", b"AV1-HDR-VIDEO".to_vec()), Rendition::media("wai.image.jp2", "j2k", b"J2K-MEZZANINE".to_vec()), Rendition::companion("wai.meta.hdr10plus", b"ST2094-40-DYNAMIC-METADATA".to_vec(), "wai.video.av1"), - Rendition::companion("wai.audio.iamf", b"IAMF-MIX-METADATA".to_vec(), "wai.video.av1"), + Rendition::companion("wai.meta.dovi", b"ST2094-10-RPU".to_vec(), "wai.video.av1"), ], ); let dec = WaiMulti::from_bytes(&m.to_bytes().unwrap()).unwrap(); @@ -773,6 +1006,358 @@ mod tests { assert!(dec.companions_of("wai.image.jp2").is_empty()); // a v1.1 rendition (role absent) is not a companion — back-compat. assert!(!dec.manifest.renditions[0].is_companion()); + // and the envelope satisfies the registry-aware role rules. + assert_eq!(dec.check_component_roles(), Ok(())); + } + + #[test] + fn lcevc_enhancement_binds_to_exactly_one_base() { + // The separate (dual-track-shaped) LCEVC carriage: an HD VVC base that + // any VVC sink presents on its own, plus the enhancement companion that + // an LCEVC-capable sink adds on top. An AV1 rendition of the same + // content rides alongside as a royalty-free alternative. + let m = WaiMulti::new( + "video", + "replicate", + serde_json::Value::Null, + vec![ + Rendition::media("wai.video.vvc", "vvc", b"VVC-HD-BASE".to_vec()), + Rendition::media("wai.video.av1", "av1", b"AV1-UHD".to_vec()), + Rendition::companion("wai.video.lcevc", b"LCEVC-ENHANCEMENT".to_vec(), "wai.video.vvc"), + ], + ); + let dec = WaiMulti::from_bytes(&m.to_bytes().unwrap()).unwrap(); + assert_eq!(dec.check_component_roles(), Ok(())); + // The enhancement is never selectable; the base is, and the enhancement + // is bound to it byte-exact. + assert!(dec.pick(|c| c == "wai.video.lcevc").is_none()); + assert_eq!(dec.primary().unwrap().0.capability, "wai.video.vvc"); + let enh = dec.companions_of("wai.video.vvc"); + assert_eq!(enh.len(), 1); + assert_eq!(enh[0].0.capability, "wai.video.lcevc"); + assert_eq!(enh[0].1, b"LCEVC-ENHANCEMENT"); + assert!(dec.companions_of("wai.video.av1").is_empty()); + } + + #[test] + fn lcevc_binding_rejects_ambiguous_missing_and_mislabelled_bases() { + let lcevc = || Rendition::companion("wai.video.lcevc", b"ENH".to_vec(), "wai.video.hevc"); + // Two HEVC renditions: the enhancement was encoded against ONE of them, + // and companion_of (a capability) cannot say which. + let two = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", b"HEVC-A".to_vec()), + Rendition::media("wai.video.hevc", "hevc", b"HEVC-B".to_vec()), + lcevc(), + ]); + assert_eq!(two.check_component_roles(), + Err(RoleError::EnhancementBaseNotUnique { index: 2, capability: "wai.video.lcevc".into(), matches: 2 })); + // No HEVC base at all. + let none = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + lcevc(), + ]); + assert_eq!(none.check_component_roles(), + Err(RoleError::EnhancementBaseNotUnique { index: 1, capability: "wai.video.lcevc".into(), matches: 0 })); + // A non-video base (an enhancement of an audio payload is meaningless): + // present and unique, but not one of LCEVC's registered base codecs. + let audio = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.audio.aac", "aac", b"AAC".to_vec()), + Rendition::companion("wai.video.lcevc", b"ENH".to_vec(), "wai.audio.aac"), + ]); + assert_eq!(audio.check_component_roles(), + Err(RoleError::IncompatibleBase { index: 1, capability: "wai.video.lcevc".into(), + base: "wai.audio.aac".into() })); + // The enhancement mislabelled as a rendition: reported, and never picked + // even by a sink that advertises an LCEVC decoder. + let mis = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.lcevc", "lcevc", b"ENH".to_vec()), + Rendition::media("wai.video.hevc", "hevc", b"HEVC".to_vec()), + ]); + assert_eq!(mis.check_component_roles(), + Err(RoleError::CompanionOnlyAsRendition { index: 0, capability: "wai.video.lcevc".into() })); + assert_eq!(mis.primary().unwrap().0.capability, "wai.video.hevc"); + assert!(mis.pick(|c| c == "wai.video.lcevc").is_none()); + } + + #[test] + fn a_base_takes_at_most_one_enhancement_layer() { + // Two LCEVC enhancements bound to one HEVC base: the base resolves + // uniquely, but a sink applies every companion of the rendition it + // picked, so it would stack two sets of residuals on one picture. + let enh = |p: &[u8], of: &str| Rendition::companion("wai.video.lcevc", p.to_vec(), of); + let stacked = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", b"HEVC".to_vec()), + enh(b"ENH-1", "wai.video.hevc"), + enh(b"ENH-2", "wai.video.hevc"), + ]); + assert_eq!(stacked.companions_of("wai.video.hevc").len(), 2, "both would be applied"); + assert_eq!(stacked.check_component_roles(), + Err(RoleError::DuplicateEnhancementLayer { + index: 2, capability: "wai.video.lcevc".into(), + base: "wai.video.hevc".into(), first: 1 })); + // One enhancement per base across two different bases conforms, and a + // descriptor companion on an enhanced base does not count against it. + let one_each = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", b"HEVC".to_vec()), + Rendition::media("wai.video.avc", "avc", b"AVC".to_vec()), + enh(b"ENH-HEVC", "wai.video.hevc"), + enh(b"ENH-AVC", "wai.video.avc"), + Rendition::companion("wai.meta.hdr10plus", b"ST2094-40".to_vec(), "wai.video.hevc"), + ]); + assert_eq!(one_each.check_component_roles(), Ok(())); + } + + #[test] + fn v1_envelope_rejects_companion_only_capability_and_fallback() { + let v1 = |cap: &str, fallback: Option<&str>| Wai::new( + Manifest { + wai: "1.0".into(), + media: "video".into(), + intent: "replicate".into(), + model_requirement: ModelRequirement { + capability: cap.into(), + fallback: fallback.map(String::from), + min_sim: None, + sim_metric: None, + }, + conditioning: Conditioning { kind: "bytes".into(), prior: None }, + target: serde_json::Value::Null, + }, + b"BYTES".to_vec(), + ); + // A WAI1 has one payload, so a companion-only capability has no base. + for cap in ["wai.video.lcevc", "wai.meta.hdr10plus", "wai.meta.dovi"] { + assert_eq!(v1(cap, None).check_component_roles(), + Err(RoleError::CompanionOnlyAsCapability { capability: cap.into() })); + // ...and it survives the wire, because parsing stays format-agnostic; + // the check is what reports it. + let back = Wai::from_bytes(&v1(cap, None).to_bytes().unwrap()).unwrap(); + assert!(back.check_component_roles().is_err()); + } + // Named as a fallback, dispatch step 5 would decode it standalone. + assert_eq!(v1("wai.video.hevc", Some("wai.video.lcevc")).check_component_roles(), + Err(RoleError::CompanionOnlyAsFallback { capability: "wai.video.lcevc".into() })); + // Ordinary WAI1 envelopes conform, with or without a fallback. + assert_eq!(v1("wai.video.hevc", None).check_component_roles(), Ok(())); + assert_eq!(v1("wai.video.evc.baseline", Some("wai.video.evc")).check_component_roles(), Ok(())); + } + + /// SPEC §4 with the §5 companion-only rule. A companion-only fallback makes + /// the envelope non-conforming, but step 3 still decodes via a primary the + /// sink advertises; only step 5 is closed to it. A companion-only primary is + /// never dispatched, even with a usable fallback. + #[test] + fn dispatch_skips_a_companion_only_fallback_but_keeps_step_3() { + let v1 = |cap: &str, fallback: Option<&str>| Wai::new( + Manifest { + wai: "1.0".into(), + media: "image".into(), + intent: "replicate".into(), + model_requirement: ModelRequirement { + capability: cap.into(), + fallback: fallback.map(String::from), + min_sim: None, + sim_metric: None, + }, + conditioning: Conditioning { kind: "bytes".into(), prior: None }, + target: serde_json::Value::Null, + }, + b"BYTES".to_vec(), + ); + // A sink that advertises everything, including the companion-only + // capabilities: advertising one never makes it standalone-dispatchable. + let all = |_: &str| true; + let only = |set: &'static [&'static str]| move |c: &str| set.iter().any(|s| *s == c); + + // Primary advertised: step 3, whatever the fallback names. + let png_lcevc = v1("wai.image.png", Some("wai.video.lcevc")); + assert!(png_lcevc.check_component_roles().is_err(), "non-conforming envelope"); + assert_eq!(png_lcevc.dispatch_capability(all), Some("wai.image.png")); + assert_eq!(png_lcevc.dispatch_capability(only(&["wai.image.png"])), Some("wai.image.png")); + // Primary not advertised: step 5 would dispatch the fallback, but a + // companion-only one is never dispatched, so the envelope is inert. + assert_eq!(png_lcevc.dispatch_capability(only(&["wai.video.lcevc"])), None); + // A companion-only primary is inert even with a usable fallback. + assert_eq!(v1("wai.video.lcevc", Some("wai.video.hevc")).dispatch_capability(all), None); + // Ordinary envelopes: step 3, then step 5, then inert. + let evc = v1("wai.video.evc.baseline", Some("wai.video.evc")); + assert_eq!(evc.dispatch_capability(only(&["wai.video.evc.baseline", "wai.video.evc"])), + Some("wai.video.evc.baseline")); + assert_eq!(evc.dispatch_capability(only(&["wai.video.evc"])), Some("wai.video.evc")); + assert_eq!(evc.dispatch_capability(only(&["wai.image.png"])), None); + assert_eq!(v1("wai.image.png", None).dispatch_capability(only(&["wai.audio.flac"])), None); + } + + #[test] + fn descriptor_companions_bind_but_need_not_be_unique() { + // HDR10+ metadata describes the content, so it may apply to either of + // two renditions of one capability — unlike an enhancement layer. + let m = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", b"HEVC-HI".to_vec()), + Rendition::media("wai.video.hevc", "hevc", b"HEVC-LO".to_vec()), + Rendition::companion("wai.meta.hdr10plus", b"ST2094-40".to_vec(), "wai.video.hevc"), + ]); + assert_eq!(m.check_component_roles(), Ok(())); + // ...but it must bind to something present. + let dangling = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::companion("wai.meta.hdr10plus", b"ST2094-40".to_vec(), "wai.video.hevc"), + ]); + assert_eq!(dangling.check_component_roles(), + Err(RoleError::UnboundCompanion { index: 1, capability: "wai.meta.hdr10plus".into() })); + // A companion with no companion_of at all binds to nothing. + let mut no_target = Rendition::companion("wai.meta.dovi", b"RPU".to_vec(), "x"); + no_target.companion_of = None; + let unbound = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", b"HEVC".to_vec()), + no_target, + ]); + assert!(matches!(unbound.check_component_roles(), Err(RoleError::UnboundCompanion { index: 1, .. }))); + } + + /// SPEC §7.1, rule 1: a registered capability that is not companion-only is + /// standalone media — its payload decodes on its own — and is carried as a + /// rendition, never marked companion. + #[test] + fn a_standalone_codec_is_never_a_companion() { + for (cap, of) in [ + // IAMF carries its own audio substreams; it is not mix metadata. + ("wai.audio.iamf", "wai.video.av1"), + // A second video codec, bound to a video base of the same content. + ("wai.video.hevc", "wai.video.av1"), + // An audio track bound to its video. + ("wai.audio.opus", "wai.video.av1"), + // A standalone codec bound to a base of its own kind. + ("wai.image.png", "wai.image.jp2"), + ] { + let m = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::media("wai.image.jp2", "j2k", b"J2K".to_vec()), + Rendition::companion(cap, b"PAYLOAD".to_vec(), of), + ]); + assert_eq!(m.check_component_roles(), + Err(RoleError::StandaloneAsCompanion { index: 2, capability: cap.into() }), "{cap}"); + // Marked companion, it is never selectable either, so the check is + // the only thing that says the envelope is wrong. + assert!(m.pick(|c| c == cap).is_none(), "{cap}"); + } + // Reported even with no companion_of: rule 1 comes first. + let mut unbound = Rendition::companion("wai.audio.iamf", b"IAMF".to_vec(), "x"); + unbound.companion_of = None; + let m = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + unbound, + ]); + assert_eq!(m.check_component_roles(), + Err(RoleError::StandaloneAsCompanion { index: 1, capability: "wai.audio.iamf".into() })); + // The same IAMF stream as a rendition beside the video conforms. + let ok = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::media("wai.audio.iamf", "iamf", b"IAMF".to_vec()), + ]); + assert_eq!(ok.check_component_roles(), Ok(())); + } + + /// SPEC §7.1, rule 1 and 3 for a capability this revision does not register: + /// its role and bases are unknown here, so it is held only to a present + /// base — a later revision may register it as a companion. + #[test] + fn an_unregistered_companion_needs_only_a_present_base() { + let m = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::companion("wai.meta.future", b"DESCRIPTOR".to_vec(), "wai.video.av1"), + ]); + assert_eq!(m.check_component_roles(), Ok(())); + let dangling = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::companion("wai.meta.future", b"DESCRIPTOR".to_vec(), "wai.video.hevc"), + ]); + assert_eq!(dangling.check_component_roles(), + Err(RoleError::UnboundCompanion { index: 1, capability: "wai.meta.future".into() })); + } + + /// SPEC §5 / §7.1, rule 2 for HDR dynamic metadata: it was registered as a + /// companion to a video payload, so among registered capabilities it binds + /// to one in the `video` media class — `wai.video.*`, the neural video + /// codecs, the films — and to nothing else. An unregistered base is held to + /// rule 3 only. + #[test] + fn hdr_descriptors_bind_only_to_video() { + let env = |hdr: &str, base: &str, kind: &str| WaiMulti::new( + "video", "replicate", serde_json::Value::Null, vec![ + Rendition::media(base, kind, b"BASE".to_vec()), + Rendition::companion(hdr, b"ST2094".to_vec(), base), + ]); + for hdr in ["wai.meta.hdr10plus", "wai.meta.dovi"] { + // Not video: audio, images (a neural image codec among them), a + // volumetric and a world payload. + for base in ["wai.audio.opus", "wai.audio.aac", "wai.audio.iamf", "wai.image.avif", + "wai.image.jp2", "wai.neural.glc", "wai.splat.4d", "wai.world.replay"] { + assert_eq!(env(hdr, base, "x").check_component_roles(), + Err(RoleError::IncompatibleBase { index: 1, capability: hdr.into(), base: base.into() }), + "{hdr} on {base}"); + } + // Video: wai.video.* (including the codecs LCEVC is not registered + // over), the neural video codecs outside that namespace, the films. + for base in ["wai.video.av1", "wai.video.hevc", "wai.video.j2k", "wai.video.av2", + "wai.video.int_motion", "wai.neural.video_bmshj2018", "wai.neural.dcvc_rt", + "wai.neural.dcvc_fm", "wai.film.linear", "wai.film.reel"] { + assert_eq!(env(hdr, base, "x").check_component_roles(), Ok(()), "{hdr} on {base}"); + } + // A base this revision does not register is held only to rule 3: present + // it binds, absent it is unbound. + assert_eq!(env(hdr, "wai.neural.video_future", "x").check_component_roles(), Ok(())); + let absent_unknown = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::companion(hdr, b"ST2094".to_vec(), "wai.neural.video_future"), + ]); + assert_eq!(absent_unknown.check_component_roles(), + Err(RoleError::UnboundCompanion { index: 1, capability: hdr.into() })); + } + // The refusal says what the descriptor does apply to. + let msg = env("wai.meta.hdr10plus", "wai.audio.opus", "opus").check_component_roles() + .unwrap_err().to_string(); + assert!(msg.contains("video payload") && msg.contains("neural video"), "{msg}"); + // Never to a companion: HDR metadata bound to an enhancement layer. + let on_enh = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", b"HEVC".to_vec()), + Rendition::companion("wai.video.lcevc", b"ENH".to_vec(), "wai.video.hevc"), + Rendition::companion("wai.meta.hdr10plus", b"ST2094".to_vec(), "wai.video.lcevc"), + ]); + assert_eq!(on_enh.check_component_roles(), + Err(RoleError::IncompatibleBase { index: 2, capability: "wai.meta.hdr10plus".into(), + base: "wai.video.lcevc".into() })); + // The mismatch is caught even when the named base is absent: the name alone + // says it cannot apply. + let absent = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.av1", "av1", b"AV1".to_vec()), + Rendition::companion("wai.meta.dovi", b"RPU".to_vec(), "wai.audio.opus"), + ]); + assert!(matches!(absent.check_component_roles(), Err(RoleError::IncompatibleBase { index: 1, .. }))); + } + + /// SPEC §5 / §7.1, rule 2 for LCEVC: it binds only to a base codec its + /// registration names (AVC, HEVC, VVC, EVC, VP9, AV1), not to any + /// `wai.video.*`. + #[test] + fn lcevc_binds_only_to_its_base_codecs() { + let env = |base: &str| WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media(base, "x", b"BASE".to_vec()), + Rendition::companion("wai.video.lcevc", b"ENH".to_vec(), base), + ]); + for &base in crate::codecs::enhancement_bases("wai.video.lcevc") { + assert_eq!(env(base).check_component_roles(), Ok(()), "LCEVC on {base}"); + } + for base in ["wai.video.j2k", "wai.video.av2", "wai.video.avs3", "wai.video.int_motion", + "wai.video.int_hyper", "wai.video.future", "wai.image.jp2"] { + assert_eq!(env(base).check_component_roles(), + Err(RoleError::IncompatibleBase { index: 1, capability: "wai.video.lcevc".into(), + base: base.into() }), + "LCEVC on {base}"); + } + // The error names the bases it would accept. + let msg = env("wai.video.j2k").check_component_roles().unwrap_err().to_string(); + assert!(msg.contains("wai.video.hevc") && msg.contains("wai.video.av1"), "{msg}"); } #[test] diff --git a/wai/wai-rs/tests/carriage.rs b/wai/wai-rs/tests/carriage.rs index a8ba063f0..a54cee698 100644 --- a/wai/wai-rs/tests/carriage.rs +++ b/wai/wai-rs/tests/carriage.rs @@ -12,10 +12,20 @@ //! decode of the open formats (JPEG 2000 via OpenJPEG, IAMF via ffmpeg) is //! demonstrated in tools/wai_carriage_demo.sh. //! +//! A companion-only capability (HDR dynamic metadata, an LCEVC enhancement) is +//! carried the only way SPEC §5 allows: as a `WAI2` companion bound to a base +//! rendition. As a `WAI1` capability it still survives the wire (parsing is +//! format-agnostic) but fails the role check. The converse holds for every +//! other registered capability: marked companion, it survives the wire and +//! fails the role check (SPEC §7.1, rule 1). +//! //! Pure container + registry — no codec feature required, so it runs in the //! same no-system-deps matrix as the cargo classes. -use wai::codecs::{capability_license, capability_tier, is_known_capability, known_capabilities}; +use wai::codecs::{ + capability_license, capability_tier, is_companion_only, is_known_capability, known_capabilities, +}; +use wai::container::{Rendition, RoleError, WaiMulti}; use wai::{Conditioning, Manifest, ModelRequirement, Wai}; fn media_for(cap: &str) -> &'static str { @@ -66,6 +76,17 @@ fn wrap(cap: &str, payload: Vec) -> Wai { ) } +/// A companion-only capability in the form SPEC §5 allows: a `WAI2` companion +/// bound to a base video rendition. Returns the parsed envelope. +fn wrap_companion(cap: &str, payload: Vec) -> WaiMulti { + let base = payload_for("wai.video.hevc", 64); + let env = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", base), + Rendition::companion(cap, payload, "wai.video.hevc"), + ]); + WaiMulti::from_bytes(&env.to_bytes().expect("serialize envelope")).expect("parse envelope") +} + #[test] fn every_registered_capability_carries_bytes_byte_exact() { let mut n_checked = 0; @@ -73,9 +94,22 @@ fn every_registered_capability_carries_bytes_byte_exact() { // a few payload sizes incl. 0 (empty) and a chunky one for &len in &[0usize, 1, 31, 4096, 65537] { let payload = payload_for(cap, len); + if is_companion_only(cap) { + let back = wrap_companion(cap, payload.clone()); + assert_eq!(back.check_component_roles(), Ok(()), "{cap}: companion carriage must conform"); + let bound = back.companions_of("wai.video.hevc"); + assert_eq!(bound.len(), 1, "{cap}: companion not bound to its base"); + assert_eq!(bound[0].0.capability, cap, "{cap}: capability not preserved"); + assert_eq!(bound[0].1, payload.as_slice(), "{cap}: payload not carried byte-identically at {len} B"); + assert_eq!(back.primary().unwrap().0.capability, "wai.video.hevc", "{cap}: companion was selectable"); + assert!(capability_tier(cap).is_some(), "{cap}: no fidelity tier"); + assert!(capability_license(cap).is_some(), "{cap}: no license class"); + continue; + } let env = wrap(cap, payload.clone()); let bytes = env.to_bytes().expect("serialize envelope"); let back = Wai::from_bytes(&bytes).expect("parse envelope"); + assert_eq!(back.check_component_roles(), Ok(()), "{cap}: standalone WAI1 must conform"); assert_eq!(back.payload, payload, "{cap}: payload not carried byte-identically at {len} B"); assert_eq!(back.capability(), cap, "{cap}: capability not preserved"); assert_eq!(back.fallback(), Some("wai.image.png"), "{cap}: fallback not preserved"); @@ -91,15 +125,22 @@ fn every_registered_capability_carries_bytes_byte_exact() { #[test] fn dispatch_distinguishes_known_from_unknown() { - // the floor + the new immersive/cinema/volumetric/haptics classes are known + // the floor + the immersive/cinema/volumetric/haptics classes + the + // broadcast and streaming codecs are known for cap in [ "wai.image.png", "wai.audio.iamf", "wai.image.jp2", "wai.mesh.draco", "wai.pc.gpcc", "wai.haptic.mihs", "wai.meta.hdr10plus", "wai.audio.ac4", + "wai.video.av2", "wai.video.vvc", "wai.video.hevc", "wai.video.avc", "wai.video.vp9", + "wai.video.evc", "wai.video.evc.baseline", "wai.video.avs3", "wai.video.lcevc", + "wai.audio.mpegh", "wai.audio.avs3", "wai.audio.aac", ] { assert!(is_known_capability(cap), "{cap} should be registered"); let env = wrap(cap, payload_for(cap, 128)); let back = Wai::from_bytes(&env.to_bytes().unwrap()).unwrap(); assert_eq!(back.capability(), cap); + // The v1.0 role check separates a standalone payload from a + // companion-only one, which a WAI1 cannot carry conformantly. + assert_eq!(back.check_component_roles().is_ok(), !is_companion_only(cap), "{cap}"); } // an unregistered capability is still CARRIED (the envelope is // format-agnostic) but is NOT known — a sink returns the loud @@ -109,3 +150,27 @@ fn dispatch_distinguishes_known_from_unknown() { assert_eq!(back.capability(), "wai.image.nope"); assert!(!is_known_capability(back.capability())); } + +#[test] +fn no_standalone_capability_is_carried_as_a_companion() { + // SPEC §7.1, rule 1: every registered capability that is not companion-only + // is standalone media. Bound as a companion to a video base it still + // survives the wire byte-exact, and the role check reports it. + let mut n_checked = 0; + for &cap in known_capabilities() { + if is_companion_only(cap) { + continue; + } + let payload = payload_for(cap, 257); + let env = WaiMulti::new("video", "replicate", serde_json::Value::Null, vec![ + Rendition::media("wai.video.hevc", "hevc", payload_for("wai.video.hevc", 64)), + Rendition::companion(cap, payload.clone(), "wai.video.hevc"), + ]); + let back = WaiMulti::from_bytes(&env.to_bytes().expect("serialize envelope")).expect("parse envelope"); + assert_eq!(back.companions()[0].1, payload.as_slice(), "{cap}: payload not carried byte-identically"); + assert_eq!(back.check_component_roles(), + Err(RoleError::StandaloneAsCompanion { index: 1, capability: cap.into() }), "{cap}"); + n_checked += 1; + } + assert!(n_checked >= 50, "expected the full registry, got {n_checked}"); +} diff --git a/wai/wai-rs/tests/cli_wrap_intent.rs b/wai/wai-rs/tests/cli_wrap_intent.rs index 51137d50c..88e966307 100644 --- a/wai/wai-rs/tests/cli_wrap_intent.rs +++ b/wai/wai-rs/tests/cli_wrap_intent.rs @@ -12,7 +12,7 @@ use std::path::{Path, PathBuf}; use std::process::{Command, Output}; -use wai::container::Wai; +use wai::container::{Wai, WaiMulti}; /// The committed JPEG AI reference codestream; `wrap` treats any payload as opaque. fn payload() -> PathBuf { @@ -115,3 +115,45 @@ fn wrap_still_writes_replicate_over_a_standard_codec() { assert_eq!(env.replicate_refusal(), None, "a sink accepts what wrap wrote"); assert_eq!(env.payload, std::fs::read(payload()).unwrap()); } + +#[test] +fn wrap_holds_a_companion_envelope_to_the_same_intent_rule() { + // With `--companion` the output is a WAI2 envelope; its intent is checked over + // the base payload's capability as the WAI1 form's is (and over each + // companion's). HDR dynamic metadata may bind to a neural video payload + // (SPEC §7.1), whose class is NeuralFloat. + let meta = payload(); + let meta = meta.to_str().unwrap(); + let hdr = ["--companion", "wai.meta.hdr10plus", meta]; + for extra in [hdr.to_vec(), [&["--intent", "replicate"][..], &hdr[..]].concat()] { + let out = scratch("dcvc-hdr.wai"); + let run = wrap("wai.neural.dcvc_rt", &out, &extra); + let stderr = String::from_utf8_lossy(&run.stderr); + assert!(!run.status.success(), "{extra:?}: wrap must refuse; stderr: {stderr}"); + assert!(stderr.contains("not writing intent=replicate over wai.neural.dcvc_rt"), "{stderr}"); + assert!(!out.exists(), "{extra:?}: a refused wrap must not write an envelope"); + } + // Another intent is carried, companion and all. + let out = scratch("dcvc-hdr-improve.wai"); + let run = wrap("wai.neural.dcvc_rt", &out, &[&["--intent", "improve"][..], &hdr[..]].concat()); + assert!(run.status.success(), "{}", String::from_utf8_lossy(&run.stderr)); + let env = WaiMulti::read(&out).unwrap(); + assert_eq!(env.intent(), "improve"); + assert_eq!(env.check_component_roles(), Ok(())); + // A standard base still gets `replicate`. + let out = scratch("hevc-hdr.wai"); + let run = wrap("wai.video.hevc", &out, &hdr); + assert!(run.status.success(), "{}", String::from_utf8_lossy(&run.stderr)); + let env = WaiMulti::read(&out).unwrap(); + assert_eq!(env.intent(), "replicate"); + assert_eq!(env.check_component_roles(), Ok(())); + // The WAI2 form takes no fallback; that is the refusal, whatever the + // fallback's class, rather than the WAI1 fallback-intent check. + let out = scratch("hevc-hdr-fb.wai"); + let run = wrap("wai.video.hevc", &out, + &[&["--fallback", "wai.image.jpegai"][..], &hdr[..]].concat()); + let stderr = String::from_utf8_lossy(&run.stderr); + assert!(!run.status.success(), "wrap must refuse; stderr: {stderr}"); + assert!(stderr.contains("--fallback names a WAI1 alternative"), "{stderr}"); + assert!(!out.exists(), "a refused wrap must not write an envelope"); +}