open-standards/wai/research/media-codec-registry.md

Media-Codec Registry — immersive audio, cinema, volumetric, haptics, HDR

Reference for the capabilities WAI dispatches to beyond the flat-media floor (PNG/FLAC/zstd) and modern menu (AVIF/JXL/Opus/AV1) and the neural bench. These cover object/scene audio, the theatrical mezzanine path, 3D-geometry compression, the standardized haptics codec, and HDR dynamic metadata.

Registration ≠ a bundled decoder. As everywhere in WAI, registering a capability fixes the capability string and says "payload = the named standard's canonical bytes; the decoder is sink/library-supplied." The point is one envelope that can name any of these and fall back cleanly — not re-implementing them. The "ingest, not compete" posture: dispatch to the field's open reference where one exists, flag the proprietary/SEP-pool ones honestly, and never make a non-floor capability a default.

Royalty key: RF = royalty-free · PROP = proprietary license · POOL = patent-pool / SEP royalties apply. Status verified against primary sources June 2026 (see footnotes for residual caveats).

License classification is enforceable, not just documented

Every registered capability carries an IP class in the registry — codecs::capability_license(cap) -> LicenseClass ∈ {RoyaltyFree, PatentPool, Proprietary, NonCommercial}, with is_royalty_free(cap) as the predicate. It is a property of the capability, looked up in the registry — deliberately not a per-file manifest claim, because a codec's licence can't be honestly asserted in an envelope's own header. A deployer audits a title or a whole catalogue in code: "encode only to royalty-free capabilities", or "flag every pool/proprietary capability in this library." That is the per-title toll-road choice made into a property of the format rather than a slide. wai capabilities prints the class per row (the registry reports the royalty-free count outright).

Immersive / object-based audio

capability	standard	royalty	reference	tier
wai.audio.iamf	IAMF / "Eclipsa Audio" (AOMedia Immersive Audio Model & Formats)	RF	libiamf (Apache-2.0)	Lossless¹
wai.audio.ac4	Dolby AC-4 (codec under the Atmos umbrella)	PROP	Dolby	Lossy
wai.audio.ivas	3GPP IVAS (Immersive Voice & Audio Services, EVS-based)	POOL²	3GPP	Lossy

The floor (wai.audio.{opus,flac}) + the deterministic object mix (wai.audio.scene) cover stereo and the bit-exact mixdown; these add the field's standard immersive carriers. IAMF is the pick — open, royalty-free, AV1-lineage, and codec-agnostic (it carries Opus / FLAC / LPCM for channel, scene/ambisonic, and binaural audio), which fits "name the capability" exactly. AC-4 is registered as a license-flagged slot so marquee object-audio content can ride inside a WAI envelope without forcing the brand out. IVAS is the conversational/telepresence immersive codec.

¹ IAMF can carry lossless codecs (FLAC/LPCM) → lossless ceiling. Verified RF: published under the AOMedia Patent License 1.0 ("no-charge, royalty-free, irrevocable"); reference under BSD-3-Clause-Clear. (Adopted by YouTube + Samsung 2025.) ² Verified POOL: IVAS is in Via Licensing Alliance's Voice Codec pool (launched Dec 2024, alongside EVS; Fraunhofer / NTT / Ericsson / Panasonic et al.) — not royalty-free.

Cinema mastering / intra-frame mezzanine

capability	standard	royalty	reference	tier
wai.image.jp2	JPEG 2000 still (ISO/IEC 15444-1; DCP via SMPTE ST 429 / MXF)	RF core³	OpenJPEG (BSD-2)	Lossless
wai.video.j2k	Intra-frame JPEG 2000 video (each frame an independent codestream)	RF core³	OpenJPEG	Lossless

The theatrical interchange path: a DCP is JPEG 2000 in MXF. ProRes / DNxHR are the production mezzanines but proprietary, so out of scope for a royalty-free floor (a license-flagged slot could be added if a workflow needs it).

³ Verified RF (Part 1): JPEG 2000 Part 1 (the core coding system) is licensable fee-free under ITU-T patent policy 2.1 — 20+ organisations agreed to license essential IP for the standard without royalties. Residual caveats: the committee never gave a formal guarantee, some extensions may carry fees, and a 2004 note acknowledged possible undeclared "submarine" patents. Safe for the DCP/mezzanine Part-1 path; don't extend the RF claim to Part 2+.

Volumetric / 3D geometry compression

capability	standard	royalty	reference	tier
wai.mesh.draco	Draco (Google) mesh + point-cloud geometry	RF (Apache-2.0)	draco	Lossless⁴
wai.pc.gpcc	MPEG G-PCC (ISO/IEC 23090-9), sparse / LiDAR clouds	POOL	mpeg-pcc-tmc13	Lossy
wai.pc.vpcc	MPEG V-PCC (ISO/IEC 23090-5), dense / dynamic clouds	POOL	mpeg-pcc-tmc2	Lossy
wai.splat.sog	Gaussian-splat compression (Self-Organizing Gaussians; MPEG WG4 WIP)	RF (open work)	SOG / SOGS	Lossy

These compress the heavy asset clouds the existing capabilities already ship (wai.mesh.glb, wai.splat.spz, wai.splat.4d). wai.splat.sog is a candidate — registered so a capability exists ahead of an MPEG freeze; its payload format pins when an open reference firms up (the same "track until the license/anchor firms" discipline used for the codec newcomers).

⁴ Draco mesh geometry has a lossless mode (exact-geometry ceiling, like wai.mesh.glb); its quantization knobs are the encoder's choice.

Standardized haptics

capability	standard	royalty	reference	tier
wai.haptic.mihs	MPEG-I Haptics (ISO/IEC 23090-31, Jan 2025)	flag⁵	MPEG RM	Lossy

Complements the native wai.haptic.signal (the keyframe-envelope, Fx-floor, sample-equivalence path). MIHS is the ISO standard codec: human-readable HJIF (JSON) + a binary MIHS stream (~8–16 kbps). ISO/IEC 23090-33 defines bit-exact decoder verification + valid/invalid conformance datasets — the conformance-vector philosophy already in the standard, so a *-equivalence conformance corpus is straightforward to mirror.

⁵ Unconfirmed → classified POOL conservatively. No public royalty-free declaration was found; it is an MPEG/ISO standard, so the ISO patent-declaration process applies and a pool is the likely outcome (as with the MPEG point clouds). Treat as pool until the ITTF patent database / a pool announcement confirms otherwise — do not assert royalty-free.

HDR dynamic metadata (descriptors, not media payloads)

capability	standard	royalty	tier
wai.meta.hdr10plus	HDR10+ dynamic metadata (SMPTE ST 2094-40)	RF (content)	Lossless
wai.meta.dovi	Dolby Vision dynamic metadata (SMPTE ST 2094-10, RPU)	PROP	Lossless

Architectural note. Unlike the codecs above, HDR metadata is not a standalone media payload — it is color / dynamic-range metadata that accompanies a video payload (carried in SEI / ST 2094). It stresses the §2 invariant ("payload = the named codec's canonical bytes"). The clean design: these are companion descriptors a manifest declares alongside a video capability; the video bytes stay the video capability's, and the descriptor is bound as a companion component (the multi-component / multi-rendition envelope's job — v1.1). Static HDR10 (SMPTE ST 2086) and HLG (ARIB STD-B67) are baseline characteristics already in-band in the video bitstream and need only a manifest color-descriptor field, not a separate capability. This is the one place to resolve in SPEC before an integration partner's engineers ask.

Implementation & demonstration status

WAI's job is carriage + dispatch; decoders are sink-supplied (even the floor codecs are wrapped reference libraries, not re-implemented). So "implemented and demonstrated" means, per capability:

• Carriage proven for every registered class — wai wrap <cap> <payload> builds an envelope around the canonical bytes; wai unwrap returns them byte-identically; the registry reports the tier + license. The tests/carriage.rs conformance asserts this for the entire registry (0 B … 64 KiB payloads) and runs in the no-system-deps CI matrix.
• Real end-to-end decode demonstrated for the open formats with an installed/built reference tool — the full carry → dispatch → decode path on a real file: JPEG 2000 (wai.image.jp2, OpenJPEG — cinema), IAMF (wai.audio.iamf, ffmpeg — immersive audio), Draco (wai.mesh.draco, draco_decoder — 3D geometry), G-PCC (wai.pc.gpcc, MPEG tmc3 built from source — point clouds; patent-pool, so demonstrated-but-not-royalty-free), and IVAS (wai.audio.ivas, 3GPP TS 26.258 IVAS_cod/IVAS_dec built from source — immersive voice/audio: real stereo PCM → IVAS bitstream → full PCM reconstruction, and the carried bitstream decodes bit-identically to a direct decode; patent-pool, so demonstrated-but-not-royalty-free). See tools/wai_carriage_demo.sh.
• Real encode + structural decode demonstrated — V-PCC (wai.pc.vpcc, MPEG tmc2 built from source with its bundled HM video codec): a real point cloud is encoded end-to-end through the full V-PCC pipeline (patch projection → occupancy/geometry video → HM intra encode) into a conformant V3C bitstream, which PccAppParser then structurally decodes into its V3C units (V3C_VPS / V3C_AD / V3C_OVD / V3C_GVD), and which WAI carries byte-exact. Full pixel reconstruction needs HM's standalone decoder, which the V-PCC-patched HM source builds only as a namespaced library (the PccAppVideoDecoder wrapper's job), not a standalone TAppDecoder — so the reconstruct step stays sink-supplied while the encode + V3C-conformance + carriage are real. Patent-pool, like G-PCC.
• Decoder sink-supplied (open reference exists, not bundled / not installed here): wai.video.j2k, wai.splat.sog, wai.haptic.mihs, wai.meta.hdr10plus.
• Proprietary (no open decoder, by design — carried, sink-supplied): wai.audio.ac4, wai.meta.dovi.

The reference sink decoders for the volumetric + immersive-audio rows (MPEG tmc13 for G-PCC, tmc2 for V-PCC, 3GPP IVAS_cod/IVAS_dec) are fetched and built at pinned versions by tools/build_reference_decoders.sh, which prints the PATH / TMC2_CFG to export before tools/wai_carriage_demo.sh — so every real+decode / real+parse row reproduces from a clean checkout.

wai wrap / unwrap is the universal carriage operation: the payload is opaque to WAI, so the same envelope carries a JPEG 2000 codestream, an IAMF stream, a Draco buffer, or a proprietary AC-4 frame identically — that opacity is the point, and the reason WAI does not need to re-implement any of them.

What stays the manifest's job (not new capabilities)

• ADM / S-ADM (ITU-R BS.2076) — the object-audio authoring metadata model that feeds IAMF; agnostic, not a codec. A manifest/authoring concern.
• Static HDR10 / HLG color descriptors — a manifest color field (primaries, transfer, mastering-display), not a payload.

Spec follow-ups flagged by this round

1. Companion-component binding — DONE. The WAI2 multi-component envelope gains a component role: "companion" (with companion_of) marks a descriptor applied with the chosen rendition. HDR dynamic metadata (wai.meta.hdr10plus / wai.meta.dovi) and immersive-audio metadata now bind to a video/audio payload in the same envelope — pick/primary skip companions, companions_of(cap) returns the descriptors for the chosen component. Back-compat: role absent ⇒ rendition. SPEC §6.1 + a conformance test (companion_descriptors_bind_to_a_primary).
2. License classification — DONE. codecs::capability_license() / is_royalty_free() classify every capability; wai capabilities reports them; a coherence test enforces that no capability is unclassified. Built as a registry property, not a manifest field (a per-file licence claim would be spoofable; the codec's licence is intrinsic to the capability).
3. Royalty footnotes — VERIFIED (June 2026). IAMF (AOMedia PL 1.0), JPEG 2000 Part 1 (ITU-T 2.1), Draco (Apache-2.0), HDR10+ (content+device RF) → confirmed royalty-free; IVAS (Via-LA Voice Codec pool), G-PCC/V-PCC (MPEG) → confirmed pool; AC-4 / Dolby Vision → proprietary. Residual: JPEG 2000 extensions/submarine-patent caveat (³), MPEG-I Haptics pool status unconfirmed (⁵).