cf4c7bb8bd
Add tools/build_reference_decoders.sh: fetches and builds the open MPEG/3GPP
reference codecs that act as the sink decoders for the volumetric and
immersive-audio rows of the carriage demo, each pinned to a verified ref:
- G-PCC mpeg-pcc-tmc13 release-v23.0-rc2 -> tmc3
- V-PCC mpeg-pcc-tmc2 release-v25.0 -> PccApp{Encoder,Parser,...}
- IVAS 3GPP ivas-codec main @ 3882460 -> IVAS_cod / IVAS_dec
It handles the cmake 4.x policy override the MPEG trees need (their build.sh
rejects cmake major != 3), is idempotent (skips already-built trees), and
prints the exact PATH/TMC2_CFG export line to run wai_carriage_demo.sh. So the
real+decode / real+parse rows reproduce from a clean checkout instead of the
ad-hoc /tmp builds.
- demo header + registry: cross-reference the build script.
- .gitignore: ignore the script's default ./.reference-tools prefix.
Verified: refs resolve on all three remotes; the printed exports drive the
demo to 5 real+decode + 1 real+parse rows, all byte-exact.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
210 lines
12 KiB
Markdown
210 lines
12 KiB
Markdown
# Media-Codec Registry — immersive audio, cinema, volumetric, haptics, HDR
|
||
|
||
> Reference for the capabilities WAI dispatches to beyond the flat-media floor
|
||
> (PNG/FLAC/zstd) and modern menu (AVIF/JXL/Opus/AV1) and the neural bench.
|
||
> These cover object/scene audio, the theatrical mezzanine path, 3D-geometry
|
||
> compression, the standardized haptics codec, and HDR dynamic metadata.
|
||
>
|
||
> **Registration ≠ a bundled decoder.** As everywhere in WAI, registering a
|
||
> capability fixes the *capability string* and says "payload = the named
|
||
> standard's canonical bytes; the decoder is sink/library-supplied." The point
|
||
> is one envelope that can *name* any of these and fall back cleanly — not
|
||
> re-implementing them. The "ingest, not compete" posture: dispatch to the
|
||
> field's open reference where one exists, flag the proprietary/SEP-pool ones
|
||
> honestly, and never make a non-floor capability a default.
|
||
|
||
Royalty key: **RF** = royalty-free · **PROP** = proprietary license · **POOL**
|
||
= patent-pool / SEP royalties apply. Status verified against primary sources
|
||
**June 2026** (see footnotes for residual caveats).
|
||
|
||
## License classification is enforceable, not just documented
|
||
|
||
Every registered capability carries an IP class in the registry —
|
||
`codecs::capability_license(cap) -> LicenseClass` ∈ {`RoyaltyFree`,
|
||
`PatentPool`, `Proprietary`, `NonCommercial`}, with `is_royalty_free(cap)` as
|
||
the predicate. It is a property of the **capability**, looked up in the
|
||
registry — deliberately **not** a per-file manifest claim, because a codec's
|
||
licence can't be honestly asserted in an envelope's own header. A deployer
|
||
audits a title or a whole catalogue in code: "encode only to royalty-free
|
||
capabilities", or "flag every pool/proprietary capability in this library."
|
||
That is the *per-title toll-road* choice made into a property of the format
|
||
rather than a slide. `wai capabilities` prints the class per row (the registry
|
||
reports the royalty-free count outright).
|
||
|
||
## Immersive / object-based audio
|
||
|
||
| capability | standard | royalty | reference | tier |
|
||
|---|---|---|---|---|
|
||
| `wai.audio.iamf` | IAMF / "Eclipsa Audio" (AOMedia Immersive Audio Model & Formats) | **RF** | libiamf (Apache-2.0) | Lossless¹ |
|
||
| `wai.audio.ac4` | Dolby AC-4 (codec under the Atmos umbrella) | **PROP** | Dolby | Lossy |
|
||
| `wai.audio.ivas` | 3GPP IVAS (Immersive Voice & Audio Services, EVS-based) | **POOL**² | 3GPP | Lossy |
|
||
|
||
The floor (`wai.audio.{opus,flac}`) + the deterministic object mix
|
||
(`wai.audio.scene`) cover stereo and the bit-exact mixdown; these add the
|
||
field's standard *immersive carriers*. **IAMF is the pick** — open,
|
||
royalty-free, AV1-lineage, and codec-agnostic (it carries Opus / FLAC / LPCM
|
||
for channel, scene/ambisonic, and binaural audio), which fits "name the
|
||
capability" exactly. AC-4 is registered as a license-flagged slot so marquee
|
||
object-audio content can ride *inside* a WAI envelope without forcing the brand
|
||
out. IVAS is the conversational/telepresence immersive codec.
|
||
|
||
¹ IAMF can carry lossless codecs (FLAC/LPCM) → lossless ceiling. **Verified
|
||
RF**: published under the AOMedia Patent License 1.0 ("no-charge, royalty-free,
|
||
irrevocable"); reference under BSD-3-Clause-Clear. (Adopted by YouTube +
|
||
Samsung 2025.)
|
||
² **Verified POOL**: IVAS is in Via Licensing Alliance's Voice Codec pool
|
||
(launched Dec 2024, alongside EVS; Fraunhofer / NTT / Ericsson / Panasonic et
|
||
al.) — **not** royalty-free.
|
||
|
||
## Cinema mastering / intra-frame mezzanine
|
||
|
||
| capability | standard | royalty | reference | tier |
|
||
|---|---|---|---|---|
|
||
| `wai.image.jp2` | JPEG 2000 still (ISO/IEC 15444-1; DCP via SMPTE ST 429 / MXF) | RF core³ | OpenJPEG (BSD-2) | Lossless |
|
||
| `wai.video.j2k` | Intra-frame JPEG 2000 video (each frame an independent codestream) | RF core³ | OpenJPEG | Lossless |
|
||
|
||
The theatrical interchange path: a DCP is JPEG 2000 in MXF. ProRes / DNxHR are
|
||
the production mezzanines but proprietary, so out of scope for a royalty-free
|
||
floor (a license-flagged slot could be added if a workflow needs it).
|
||
|
||
³ **Verified RF (Part 1)**: JPEG 2000 Part 1 (the core coding system) is
|
||
licensable fee-free under ITU-T patent policy 2.1 — 20+ organisations agreed to
|
||
license essential IP for the standard without royalties. Residual caveats: the
|
||
committee never gave a *formal* guarantee, some **extensions** may carry fees,
|
||
and a 2004 note acknowledged possible undeclared "submarine" patents. Safe for
|
||
the DCP/mezzanine Part-1 path; don't extend the RF claim to Part 2+.
|
||
|
||
## Volumetric / 3D geometry compression
|
||
|
||
| capability | standard | royalty | reference | tier |
|
||
|---|---|---|---|---|
|
||
| `wai.mesh.draco` | Draco (Google) mesh + point-cloud geometry | **RF** (Apache-2.0) | draco | Lossless⁴ |
|
||
| `wai.pc.gpcc` | MPEG G-PCC (ISO/IEC 23090-9), sparse / LiDAR clouds | **POOL** | mpeg-pcc-tmc13 | Lossy |
|
||
| `wai.pc.vpcc` | MPEG V-PCC (ISO/IEC 23090-5), dense / dynamic clouds | **POOL** | mpeg-pcc-tmc2 | Lossy |
|
||
| `wai.splat.sog` | Gaussian-splat compression (Self-Organizing Gaussians; MPEG WG4 WIP) | RF (open work) | SOG / SOGS | Lossy |
|
||
|
||
These compress the heavy asset clouds the existing capabilities already ship
|
||
(`wai.mesh.glb`, `wai.splat.spz`, `wai.splat.4d`). `wai.splat.sog` is a
|
||
**candidate** — registered so a capability exists ahead of an MPEG freeze; its
|
||
payload format pins when an open reference firms up (the same "track until the
|
||
license/anchor firms" discipline used for the codec newcomers).
|
||
|
||
⁴ Draco mesh geometry has a lossless mode (exact-geometry ceiling, like
|
||
`wai.mesh.glb`); its quantization knobs are the encoder's choice.
|
||
|
||
## Standardized haptics
|
||
|
||
| capability | standard | royalty | reference | tier |
|
||
|---|---|---|---|---|
|
||
| `wai.haptic.mihs` | MPEG-I Haptics (ISO/IEC 23090-31, Jan 2025) | flag⁵ | MPEG RM | Lossy |
|
||
|
||
Complements the native `wai.haptic.signal` (the keyframe-envelope, Fx-floor,
|
||
sample-equivalence path). MIHS is the ISO standard codec: human-readable HJIF
|
||
(JSON) + a binary MIHS stream (~8–16 kbps). **ISO/IEC 23090-33 defines
|
||
bit-exact decoder verification + valid/invalid conformance datasets** — the
|
||
conformance-vector philosophy already in the standard, so a `*-equivalence`
|
||
conformance corpus is straightforward to mirror.
|
||
|
||
⁵ **Unconfirmed → classified POOL conservatively.** No public royalty-free
|
||
declaration was found; it is an MPEG/ISO standard, so the ISO patent-declaration
|
||
process applies and a pool is the likely outcome (as with the MPEG point
|
||
clouds). Treat as pool until the ITTF patent database / a pool announcement
|
||
confirms otherwise — do not assert royalty-free.
|
||
|
||
## HDR dynamic metadata (descriptors, not media payloads)
|
||
|
||
| capability | standard | royalty | tier |
|
||
|---|---|---|---|
|
||
| `wai.meta.hdr10plus` | HDR10+ dynamic metadata (SMPTE ST 2094-40) | **RF** (content) | Lossless |
|
||
| `wai.meta.dovi` | Dolby Vision dynamic metadata (SMPTE ST 2094-10, RPU) | **PROP** | Lossless |
|
||
|
||
**Architectural note.** Unlike the codecs above, HDR metadata is *not* a
|
||
standalone media payload — it is color / dynamic-range metadata that
|
||
accompanies a video payload (carried in SEI / ST 2094). It stresses the §2
|
||
invariant ("payload = the named codec's canonical bytes"). The clean design:
|
||
these are **companion descriptors** a manifest declares alongside a video
|
||
capability; the video bytes stay the video capability's, and the descriptor is
|
||
bound as a companion component (the multi-component / multi-rendition envelope's
|
||
job — v1.1). Static HDR10 (SMPTE ST 2086) and HLG (ARIB STD-B67) are baseline
|
||
characteristics already in-band in the video bitstream and need only a manifest
|
||
color-descriptor field, not a separate capability. **This is the one place to
|
||
resolve in SPEC before an integration partner's engineers ask.**
|
||
|
||
## Implementation & demonstration status
|
||
|
||
WAI's job is **carriage + dispatch**; decoders are sink-supplied (even the
|
||
floor codecs are *wrapped* reference libraries, not re-implemented). So
|
||
"implemented and demonstrated" means, per capability:
|
||
|
||
- **Carriage proven for every registered class** — `wai wrap <cap> <payload>`
|
||
builds an envelope around the canonical bytes; `wai unwrap` returns them
|
||
**byte-identically**; the registry reports the tier + license. The
|
||
`tests/carriage.rs` conformance asserts this for the *entire* registry (0 B …
|
||
64 KiB payloads) and runs in the no-system-deps CI matrix.
|
||
- **Real end-to-end decode demonstrated** for the open formats with an
|
||
installed/built reference tool — the full carry → dispatch → decode path on a
|
||
real file: **JPEG 2000** (`wai.image.jp2`, OpenJPEG — cinema), **IAMF**
|
||
(`wai.audio.iamf`, ffmpeg — immersive audio), **Draco** (`wai.mesh.draco`,
|
||
draco_decoder — 3D geometry), **G-PCC** (`wai.pc.gpcc`, MPEG `tmc3` built
|
||
from source — point clouds; patent-pool, so demonstrated-but-not-royalty-free),
|
||
and **IVAS** (`wai.audio.ivas`, 3GPP TS 26.258 `IVAS_cod`/`IVAS_dec` built
|
||
from source — immersive voice/audio: real stereo PCM → IVAS bitstream → full
|
||
PCM reconstruction, and the *carried* bitstream decodes bit-identically to a
|
||
direct decode; patent-pool, so demonstrated-but-not-royalty-free).
|
||
See `tools/wai_carriage_demo.sh`.
|
||
- **Real encode + structural decode demonstrated** — **V-PCC** (`wai.pc.vpcc`,
|
||
MPEG `tmc2` built from source with its bundled HM video codec): a real point
|
||
cloud is encoded end-to-end through the full V-PCC pipeline (patch projection
|
||
→ occupancy/geometry video → HM intra encode) into a conformant **V3C
|
||
bitstream**, which `PccAppParser` then structurally decodes into its V3C units
|
||
(`V3C_VPS` / `V3C_AD` / `V3C_OVD` / `V3C_GVD`), and which WAI carries
|
||
byte-exact. Full *pixel* reconstruction needs HM's standalone decoder, which
|
||
the V-PCC-patched HM source builds only as a namespaced library (the
|
||
`PccAppVideoDecoder` wrapper's job), not a standalone `TAppDecoder` — so the
|
||
reconstruct step stays sink-supplied while the encode + V3C-conformance +
|
||
carriage are real. Patent-pool, like G-PCC.
|
||
- **Decoder sink-supplied** (open reference exists, not bundled / not installed
|
||
here): `wai.video.j2k`, `wai.splat.sog`,
|
||
`wai.haptic.mihs`, `wai.meta.hdr10plus`.
|
||
- **Proprietary** (no open decoder, by design — carried, sink-supplied):
|
||
`wai.audio.ac4`, `wai.meta.dovi`.
|
||
|
||
The reference sink decoders for the volumetric + immersive-audio rows (MPEG
|
||
`tmc13` for G-PCC, `tmc2` for V-PCC, 3GPP `IVAS_cod`/`IVAS_dec`) are fetched
|
||
and built at pinned versions by `tools/build_reference_decoders.sh`, which
|
||
prints the `PATH` / `TMC2_CFG` to export before `tools/wai_carriage_demo.sh`
|
||
— so every `real+decode` / `real+parse` row reproduces from a clean checkout.
|
||
|
||
`wai wrap` / `unwrap` is the universal carriage operation: the payload is opaque
|
||
to WAI, so the same envelope carries a JPEG 2000 codestream, an IAMF stream, a
|
||
Draco buffer, or a proprietary AC-4 frame identically — that opacity is the
|
||
point, and the reason WAI does not need to re-implement any of them.
|
||
|
||
## What stays the manifest's job (not new capabilities)
|
||
|
||
- **ADM / S-ADM** (ITU-R BS.2076) — the object-audio *authoring metadata model*
|
||
that feeds IAMF; agnostic, not a codec. A manifest/authoring concern.
|
||
- **Static HDR10 / HLG color descriptors** — a manifest color field (primaries,
|
||
transfer, mastering-display), not a payload.
|
||
|
||
## Spec follow-ups flagged by this round
|
||
|
||
1. **Companion-component binding — DONE.** The `WAI2` multi-component envelope
|
||
gains a component `role`: `"companion"` (with `companion_of`) marks a
|
||
descriptor applied *with* the chosen rendition. HDR dynamic metadata
|
||
(`wai.meta.hdr10plus` / `wai.meta.dovi`) and immersive-audio metadata now
|
||
bind to a video/audio payload in the **same envelope** — `pick`/`primary`
|
||
skip companions, `companions_of(cap)` returns the descriptors for the chosen
|
||
component. Back-compat: `role` absent ⇒ rendition. SPEC §6.1 + a conformance
|
||
test (`companion_descriptors_bind_to_a_primary`).
|
||
2. **License classification — DONE.** `codecs::capability_license()` /
|
||
`is_royalty_free()` classify every capability; `wai capabilities` reports
|
||
them; a coherence test enforces that no capability is unclassified. Built as
|
||
a **registry property**, not a manifest field (a per-file licence claim would
|
||
be spoofable; the codec's licence is intrinsic to the capability).
|
||
3. **Royalty footnotes — VERIFIED (June 2026).** IAMF (AOMedia PL 1.0), JPEG
|
||
2000 Part 1 (ITU-T 2.1), Draco (Apache-2.0), HDR10+ (content+device RF) →
|
||
confirmed royalty-free; IVAS (Via-LA Voice Codec pool), G-PCC/V-PCC (MPEG) →
|
||
confirmed pool; AC-4 / Dolby Vision → proprietary. Residual: JPEG 2000
|
||
extensions/submarine-patent caveat (³), MPEG-I Haptics pool status
|
||
unconfirmed (⁵).
|