# Media-Codec Registry — immersive audio, cinema, volumetric, haptics, HDR

> Reference for the capabilities WAI dispatches to beyond the flat-media floor
> (PNG/FLAC/zstd) and modern menu (AVIF/JXL/Opus/AV1) and the neural bench.
> These cover object/scene audio, the theatrical mezzanine path, 3D-geometry
> compression, the standardized haptics codec, and HDR dynamic metadata.
>
> **Registration ≠ a bundled decoder.** As everywhere in WAI, registering a
> capability fixes the *capability string* and says "payload = the named
> standard's canonical bytes; the decoder is sink/library-supplied." The point
> is one envelope that can *name* any of these and fall back cleanly — not
> re-implementing them. The "ingest, not compete" posture: dispatch to the
> field's open reference where one exists, flag the proprietary/SEP-pool ones
> honestly, and never make a non-floor capability a default.

Royalty key: **RF** = royalty-free · **PROP** = proprietary license · **POOL**
= patent-pool / SEP royalties apply. Status verified against primary sources
**June 2026** (see footnotes for residual caveats).

## License classification is enforceable, not just documented

Every registered capability carries an IP class in the registry —
`codecs::capability_license(cap) -> LicenseClass` ∈ {`RoyaltyFree`,
`PatentPool`, `Proprietary`, `NonCommercial`}, with `is_royalty_free(cap)` as
the predicate. It is a property of the **capability**, looked up in the
registry — deliberately **not** a per-file manifest claim, because a codec's
licence can't be honestly asserted in an envelope's own header. A deployer
audits a title or a whole catalogue in code: "encode only to royalty-free
capabilities", or "flag every pool/proprietary capability in this library."
That is the *per-title toll-road* choice made into a property of the format
rather than a slide. `wai capabilities` prints the class per row (the registry
reports the royalty-free count outright).

## Immersive / object-based audio

| capability | standard | royalty | reference | tier |
|---|---|---|---|---|
| `wai.audio.iamf` | IAMF / "Eclipsa Audio" (AOMedia Immersive Audio Model & Formats) | **RF** | libiamf (Apache-2.0) | Lossless¹ |
| `wai.audio.ac4` | Dolby AC-4 (codec under the Atmos umbrella) | **PROP** | Dolby | Lossy |
| `wai.audio.ivas` | 3GPP IVAS (Immersive Voice & Audio Services, EVS-based) | **POOL**² | 3GPP | Lossy |

The floor (`wai.audio.{opus,flac}`) + the deterministic object mix
(`wai.audio.scene`) cover stereo and the bit-exact mixdown; these add the
field's standard *immersive carriers*. **IAMF is the pick** — open,
royalty-free, AV1-lineage, and codec-agnostic (it carries Opus / FLAC / LPCM
for channel, scene/ambisonic, and binaural audio), which fits "name the
capability" exactly. AC-4 is registered as a license-flagged slot so marquee
object-audio content can ride *inside* a WAI envelope without forcing the brand
out. IVAS is the conversational/telepresence immersive codec.

¹ IAMF can carry lossless codecs (FLAC/LPCM) → lossless ceiling. **Verified
RF**: published under the AOMedia Patent License 1.0 ("no-charge, royalty-free,
irrevocable"); reference under BSD-3-Clause-Clear. (Adopted by YouTube +
Samsung 2025.)
² **Verified POOL**: IVAS is in Via Licensing Alliance's Voice Codec pool
(launched Dec 2024, alongside EVS; Fraunhofer / NTT / Ericsson / Panasonic et
al.) — **not** royalty-free.

## Cinema mastering / intra-frame mezzanine

| capability | standard | royalty | reference | tier |
|---|---|---|---|---|
| `wai.image.jp2` | JPEG 2000 still (ISO/IEC 15444-1; DCP via SMPTE ST 429 / MXF) | RF core³ | OpenJPEG (BSD-2) | Lossless |
| `wai.video.j2k` | Intra-frame JPEG 2000 video (each frame an independent codestream) | RF core³ | OpenJPEG | Lossless |

The theatrical interchange path: a DCP is JPEG 2000 in MXF. ProRes / DNxHR are
the production mezzanines but proprietary, so out of scope for a royalty-free
floor (a license-flagged slot could be added if a workflow needs it).

³ **Verified RF (Part 1)**: JPEG 2000 Part 1 (the core coding system) is
licensable fee-free under ITU-T patent policy 2.1 — 20+ organisations agreed to
license essential IP for the standard without royalties. Residual caveats: the
committee never gave a *formal* guarantee, some **extensions** may carry fees,
and a 2004 note acknowledged possible undeclared "submarine" patents. Safe for
the DCP/mezzanine Part-1 path; don't extend the RF claim to Part 2+.

## Volumetric / 3D geometry compression

| capability | standard | royalty | reference | tier |
|---|---|---|---|---|
| `wai.mesh.draco` | Draco (Google) mesh + point-cloud geometry | **RF** (Apache-2.0) | draco | Lossless⁴ |
| `wai.pc.gpcc` | MPEG G-PCC (ISO/IEC 23090-9), sparse / LiDAR clouds | **POOL** | mpeg-pcc-tmc13 | Lossy |
| `wai.pc.vpcc` | MPEG V-PCC (ISO/IEC 23090-5), dense / dynamic clouds | **POOL** | mpeg-pcc-tmc2 | Lossy |
| `wai.splat.sog` | Gaussian-splat compression (Self-Organizing Gaussians; MPEG WG4 WIP) | RF (open work) | SOG / SOGS | Lossy |

These compress the heavy asset clouds the existing capabilities already ship
(`wai.mesh.glb`, `wai.splat.spz`, `wai.splat.4d`). `wai.splat.sog` is a
**candidate** — registered so a capability exists ahead of an MPEG freeze; its
payload format pins when an open reference firms up (the same "track until the
license/anchor firms" discipline used for the codec newcomers).

⁴ Draco mesh geometry has a lossless mode (exact-geometry ceiling, like
`wai.mesh.glb`); its quantization knobs are the encoder's choice.

## Standardized haptics

| capability | standard | royalty | reference | tier |
|---|---|---|---|---|
| `wai.haptic.mihs` | MPEG-I Haptics (ISO/IEC 23090-31, Jan 2025) | flag⁵ | MPEG RM | Lossy |

Complements the native `wai.haptic.signal` (the keyframe-envelope, Fx-floor,
sample-equivalence path). MIHS is the ISO standard codec: human-readable HJIF
(JSON) + a binary MIHS stream (~8–16 kbps). **ISO/IEC 23090-33 defines
bit-exact decoder verification + valid/invalid conformance datasets** — the
conformance-vector philosophy already in the standard, so a `*-equivalence`
conformance corpus is straightforward to mirror.

⁵ **Unconfirmed → classified POOL conservatively.** No public royalty-free
declaration was found; it is an MPEG/ISO standard, so the ISO patent-declaration
process applies and a pool is the likely outcome (as with the MPEG point
clouds). Treat as pool until the ITTF patent database / a pool announcement
confirms otherwise — do not assert royalty-free.

## HDR dynamic metadata (descriptors, not media payloads)

| capability | standard | royalty | tier |
|---|---|---|---|
| `wai.meta.hdr10plus` | HDR10+ dynamic metadata (SMPTE ST 2094-40) | **RF** (content) | Lossless |
| `wai.meta.dovi` | Dolby Vision dynamic metadata (SMPTE ST 2094-10, RPU) | **PROP** | Lossless |

**Architectural note.** Unlike the codecs above, HDR metadata is *not* a
standalone media payload — it is color / dynamic-range metadata that
accompanies a video payload (carried in SEI / ST 2094). It stresses the §2
invariant ("payload = the named codec's canonical bytes"). The clean design:
these are **companion descriptors** a manifest declares alongside a video
capability; the video bytes stay the video capability's, and the descriptor is
bound as a companion component (the multi-component / multi-rendition envelope's
job — v1.1). Static HDR10 (SMPTE ST 2086) and HLG (ARIB STD-B67) are baseline
characteristics already in-band in the video bitstream and need only a manifest
color-descriptor field, not a separate capability. **This is the one place to
resolve in SPEC before an integration partner's engineers ask.**

## Implementation & demonstration status

WAI's job is **carriage + dispatch**; decoders are sink-supplied (even the
floor codecs are *wrapped* reference libraries, not re-implemented). So
"implemented and demonstrated" means, per capability:

- **Carriage proven for every registered class** — `wai wrap <cap> <payload>`
  builds an envelope around the canonical bytes; `wai unwrap` returns them
  **byte-identically**; the registry reports the tier + license. The
  `tests/carriage.rs` conformance asserts this for the *entire* registry (0 B …
  64 KiB payloads) and runs in the no-system-deps CI matrix.
- **Real end-to-end decode demonstrated** for the open formats with an
  installed/built reference tool — the full carry → dispatch → decode path on a
  real file: **JPEG 2000** (`wai.image.jp2`, OpenJPEG — cinema), **IAMF**
  (`wai.audio.iamf`, ffmpeg — immersive audio), **Draco** (`wai.mesh.draco`,
  draco_decoder — 3D geometry), **G-PCC** (`wai.pc.gpcc`, MPEG `tmc3` built
  from source — point clouds; patent-pool, so demonstrated-but-not-royalty-free),
  and **IVAS** (`wai.audio.ivas`, 3GPP TS 26.258 `IVAS_cod`/`IVAS_dec` built
  from source — immersive voice/audio: real stereo PCM → IVAS bitstream → full
  PCM reconstruction, and the *carried* bitstream decodes bit-identically to a
  direct decode; patent-pool, so demonstrated-but-not-royalty-free).
  See `tools/wai_carriage_demo.sh`.
- **Real encode + structural decode demonstrated** — **V-PCC** (`wai.pc.vpcc`,
  MPEG `tmc2` built from source with its bundled HM video codec): a real point
  cloud is encoded end-to-end through the full V-PCC pipeline (patch projection
  → occupancy/geometry video → HM intra encode) into a conformant **V3C
  bitstream**, which `PccAppParser` then structurally decodes into its V3C units
  (`V3C_VPS` / `V3C_AD` / `V3C_OVD` / `V3C_GVD`), and which WAI carries
  byte-exact. Full *pixel* reconstruction needs HM's standalone decoder, which
  the V-PCC-patched HM source builds only as a namespaced library (the
  `PccAppVideoDecoder` wrapper's job), not a standalone `TAppDecoder` — so the
  reconstruct step stays sink-supplied while the encode + V3C-conformance +
  carriage are real. Patent-pool, like G-PCC.
- **Decoder sink-supplied** (open reference exists, not bundled / not installed
  here): `wai.video.j2k`, `wai.splat.sog`,
  `wai.haptic.mihs`, `wai.meta.hdr10plus`.
- **Proprietary** (no open decoder, by design — carried, sink-supplied):
  `wai.audio.ac4`, `wai.meta.dovi`.

The reference sink decoders for the volumetric + immersive-audio rows (MPEG
`tmc13` for G-PCC, `tmc2` for V-PCC, 3GPP `IVAS_cod`/`IVAS_dec`) are fetched
and built at pinned versions by `tools/build_reference_decoders.sh`, which
prints the `PATH` / `TMC2_CFG` to export before `tools/wai_carriage_demo.sh`
— so every `real+decode` / `real+parse` row reproduces from a clean checkout.

`wai wrap` / `unwrap` is the universal carriage operation: the payload is opaque
to WAI, so the same envelope carries a JPEG 2000 codestream, an IAMF stream, a
Draco buffer, or a proprietary AC-4 frame identically — that opacity is the
point, and the reason WAI does not need to re-implement any of them.

## What stays the manifest's job (not new capabilities)

- **ADM / S-ADM** (ITU-R BS.2076) — the object-audio *authoring metadata model*
  that feeds IAMF; agnostic, not a codec. A manifest/authoring concern.
- **Static HDR10 / HLG color descriptors** — a manifest color field (primaries,
  transfer, mastering-display), not a payload.

## Spec follow-ups flagged by this round

1. **Companion-component binding — DONE.** The `WAI2` multi-component envelope
   gains a component `role`: `"companion"` (with `companion_of`) marks a
   descriptor applied *with* the chosen rendition. HDR dynamic metadata
   (`wai.meta.hdr10plus` / `wai.meta.dovi`) and immersive-audio metadata now
   bind to a video/audio payload in the **same envelope** — `pick`/`primary`
   skip companions, `companions_of(cap)` returns the descriptors for the chosen
   component. Back-compat: `role` absent ⇒ rendition. SPEC §6.1 + a conformance
   test (`companion_descriptors_bind_to_a_primary`).
2. **License classification — DONE.** `codecs::capability_license()` /
   `is_royalty_free()` classify every capability; `wai capabilities` reports
   them; a coherence test enforces that no capability is unclassified. Built as
   a **registry property**, not a manifest field (a per-file licence claim would
   be spoofable; the codec's licence is intrinsic to the capability).
3. **Royalty footnotes — VERIFIED (June 2026).** IAMF (AOMedia PL 1.0), JPEG
   2000 Part 1 (ITU-T 2.1), Draco (Apache-2.0), HDR10+ (content+device RF) →
   confirmed royalty-free; IVAS (Via-LA Voice Codec pool), G-PCC/V-PCC (MPEG) →
   confirmed pool; AC-4 / Dolby Vision → proprietary. Residual: JPEG 2000
   extensions/submarine-patent caveat (³), MPEG-I Haptics pool status
   unconfirmed (⁵).