open-standards/wai/SPEC.md

# WAI v1.0 — Web AI Media Transport & Execution

Status: **Draft Standard**. Reference implementation: `wai-rs/` (Rust
lib + cdylib + staticlib, C FFI). Two paths open every conforming
file:

- the **neural condition** — the sink regenerates the media from a
  compact conditioning payload against a *shared ambient prior* (a
  model). The model is a **requirement of existence**, never
  transported and never hash-pinned, exactly as `.mp3` does not ship
  its decoder.
- the **zeroth condition** — a *menu of registered SOTA standard
  codecs*. Always satisfiable on any device that ships ffmpeg or
  equivalent. WAI does **not** define new codecs; it dispatches to the
  field's best-in-class lossy+lossless libraries (AVIF, JPEG-XL, PNG,
  Opus, FLAC, AV1, zstd, XZ).

The capability a file requires is *named*, not supplied. Sinks that
have the named capability use it; sinks that don't fall back to a
declared `fallback` capability.

Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119.

---

## 1. Scope and model

WAI is a **container + capability dispatch standard**. It is media-
general (image, audio, video, text), royalty-free, and built around the
principle that the standard's value is the envelope and the dispatch
rules — not in re-implementing existing SOTA codecs.

A WAI file declares (in its manifest):
- what media it carries,
- what *capability* a sink must have to decode it,
- a fallback capability the sink can use if it lacks the primary,
- and the conditioning bytes for that capability.

Conforming sinks MUST implement the container envelope (§2), the
manifest schema (§3), the capability-dispatch algorithm (§4), and at
least the **mandatory floor** capabilities (§5). Other capabilities
(neural and additional zeroth) are OPTIONAL but RECOMMENDED.

---

## 2. Container

```
+--------+----------------+-------------------+----------------+-----------+
| "WAI1" | u32  man_len   | manifest (JSON)   | u32 payload_len| payload   |
| 4 B    | little-endian  | man_len bytes     | little-endian  | bytes     |
+--------+----------------+-------------------+----------------+-----------+
```

- Bytes 0..4 MUST be ASCII `WAI1`. A reader MUST reject any file whose
  first 4 bytes differ.
- `man_len` and `payload_len` are unsigned 32-bit little-endian.
- The manifest is UTF-8 JSON, serialized with no insignificant
  whitespace, preserving key insertion order. Readers MUST NOT depend
  on key order for *semantics*; encoders SHOULD preserve order so files
  round-trip byte-identically.
- The payload is opaque codec bytes whose interpretation is fully
  determined by the manifest's `model_requirement.capability`.

All WAI multi-byte integers are little-endian unless stated otherwise.

---

## 3. Manifest schema

```json
{
  "wai": "1.0",
  "media": "image" | "audio" | "video" | "text",
  "intent": "replicate" | "create" | "improve" | "...",
  "model_requirement": { "capability": "<capability-string>",
                         "fallback":   "<capability-string>" | null },
  "conditioning": { "kind": "<codec-id>" },
  "security": { "encryption": "<scheme>" | "none", "kid": <u64> },
  "target":   { ... }
}
```

- `wai` — spec version. This document defines `"1.0"`. Unknown major →
  reject. Unknown minor → accept by ignoring unknown fields.
- `media` — media class; informational. The capability is authoritative.
- `intent` — generative intent. `replicate` (reconstruct the source) is
  the only intent with normative reconstruction semantics in v1.0. For a
  **neural** capability, `replicate` is sound only under the integer-entropy
  contract (§7, *Replicate-soundness*) — a float neural decode MUST NOT claim it.
- `model_requirement.capability` — the capability string a sink MUST
  resolve. See §5 for the registered set.
- `model_requirement.fallback` — a second capability the sink MAY use
  if it lacks the primary. `null` means the file is inert without the
  primary.
- `conditioning.kind` — short codec id matching the capability. Helps
  multiplexers route bytes without parsing the full capability string.
- `security` — OPTIONAL. Absent, `null`, or `encryption: "none"` ⇒ the
  payload is plaintext (the default). Otherwise the payload is encrypted
  and MUST be recovered before dispatch; see §6. The capability, media,
  and conditioning fields always describe the **plaintext** codec.
- `target` — codec-specific reconstruction parameters (dimensions,
  sample rate, duration, frame count). Informational; the payload is
  self-describing per the codec's own format.

---

## 4. Capability dispatch

A conforming sink MUST implement this algorithm:

1. Read the manifest. If `wai` major version is unknown → reject.
2. Let `cap = model_requirement.capability`.
3. If the sink advertises `cap`, decode `payload` via the codec
   registered for `cap` (§5) and return.
4. Else, let `fb = model_requirement.fallback`.
5. If `fb != null` and the sink advertises `fb`, decode via `fb`'s
   codec and return.
6. Else the file is **inert at this sink** — return a clear "missing
   capability" error to the calling application.

A WAI **payload always corresponds to exactly one capability**. v1.0
does not support multi-codec multiplexed payloads.

If the manifest declares `security.encryption` other than `"none"`
(§6), the sink MUST recover the plaintext payload before step 3 and run
this algorithm on the plaintext; see §6.4.

---

## 5. Registered capabilities (the zeroth menu)

WAI v1.0 registers the following capabilities. Each names a SOTA
royalty-free standard codec. The "payload format" column is the
on-the-wire payload bytes; in every case it is the canonical bytes
that codec normally emits (a JXL file, an Opus stream, etc.) — WAI
does not wrap or re-frame them.

### Mandatory floor

Every conforming sink MUST implement:

| capability | payload | role |
|---|---|---|
| `wai.image.png` | PNG (RFC 2083) | universal lossless image |
| `wai.audio.flac` | FLAC (xiph.org/flac) | universal lossless audio |
| `wai.text.zstd` | zstd (RFC 8478) | universal compressed bytes |

These three guarantee that any sink with the standard system codec
libraries can open *some* file in every WAI-supported media class.

### Recommended modern set

Every conforming sink SHOULD implement:

| capability | payload | role |
|---|---|---|
| `wai.image.jxl` | JPEG-XL (ISO/IEC 18181) | lossless + lossy, modern SOTA |
| `wai.image.avif` | AVIF (ISO/IEC 23000-22) | lossy SOTA (AV1-based) |
| `wai.image.jpeg` | JPEG (ITU-T T.81) | legacy compatibility |
| `wai.audio.opus` | Opus (RFC 6716) | lossy SOTA |
| `wai.video.av1` | AV1 (AOMedia) | lossy SOTA |
| `wai.video.av1.lossless` | AV1 (AOMedia), `quantizer=0` | mathematically lossless YUV (lossy in RGB due to color conversion) |
| `wai.text.xz` | XZ/LZMA2 | maximum classical text ratio |

### Neural capabilities (OPTIONAL, sink advertises if its ML runtime supports them)

Each names a model the sink MAY have installed. Payload is a compact
conditioning representation specific to the capability (latent tokens,
prompt embeddings, etc.). The standard does NOT specify the byte
layout of these payloads — the capability owner does.

Neural capabilities are picked on the **best-quality-per-bit at each
bitrate band** principle — not on whatever model is famous or small.
2026 SOTA per medium, with the wire-byte win each one delivers. The
decode route for each is implemented in the reference sink (see the
`source` column / `neural/*.rs`); the model weights are sink-supplied.

*Audio:*

| capability | model | win vs zeroth | source |
|---|---|---|---|
| `wai.neural.encodec32` | Meta EnCodec at 32 kHz, 4 codebooks | ~3 kbps acceptable music | `facebook/encodec_32khz` |
| `wai.neural.dac` | Descript Audio Codec at 44.1 kHz | 6 kbps near-transparent music | `descript/dac_44khz` |
| `wai.neural.mimi` | Kyutai Mimi, 12.5 Hz frame rate | ~1.1 kbps real-time speech | `kyutai/mimi` |
| `wai.neural.wavtokenizer` | WavTokenizer single-codebook VQ | 0.9 kbps beats DAC @ 9 kbps on UTMOS | `novateur/WavTokenizer-large-*` |
| `wai.neural.snac` | SNAC multi-resolution RVQ (hierarchical code levels, own sample rate) | low-bitrate TTS standard | `neural/snac.rs` |

*Image:*

| capability | model | win vs zeroth | source |
|---|---|---|---|
| `wai.neural.bmshj2018` | bmshj2018-factorized + rANS (byte-exact CompressAI port) | ~80× vs raw RGB at q=3 (~0.26 bpp, 29 dB PSNR) | CompressAI |
| `wai.neural.mbt2018_mean` | mean-scale hyperprior (Gaussian-conditional entropy) | conditional-entropy family, stronger than factorized | `neural/hyperprior.rs` |
| `wai.neural.int_synth` | integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) | bit-identical on every machine — soundly backs `intent = replicate` | `neural/int_synth.rs` (QAT weights sink-supplied) |
| `wai.neural.int_hyper` | integer fixed-point **conditional** codec — whole bitstream→image path integer (integer hyper-decoder σ → integer-σ entropy → integer synthesis) | the integer-entropy contract fully assembled; SOTA quality (PTQ of mbt2018_mean) bit-identical on every machine | `int_hyper_synth.rs` (tables sink-supplied) |
| `wai.generate.spectral` | deterministic integer GENERATIVE synthesis (`intent=create`) — prompt+seed → integer-LUT spectral composition, no float/ONNX | byte-identical on every machine, pinned to (model, prompt, seed, steps); the create-side of "conformance is a hash" (generation-equivalence) | `generate.rs` (small pinned model bundle) |
| `wai.neural.elic` | checkerboard context model (hyperprior + 2-pass spatial context) | the JPEG-AI / MLIC++ entropy class | `neural/context.rs` |
| `wai.neural.mlicpp` | MLIC++ multi-reference entropy (channel-AR slices + checkerboard + global attention) | −13.39% BD-rate vs VVC/VTM-17 | research code, `neural/mlicpp.rs` |
| `wai.neural.mlicv2` | MLICv2 / MLICv2+ (2025), MetaFormer transform | −20.46% BD-rate vs VTM-17 — current SOTA learned image; shares the mlicpp route | research code, `neural/mlicpp.rs` |
| `wai.neural.cmic` | Content-Adaptive Mamba (ICLR 2026), SSM transform + SCCTX entropy | learned-image frontier; shares the mlicpp route | arXiv 2508.02192, `neural/mlicpp.rs` |
| `wai.neural.glc` | Generative Latent Coding — categorical / VQ entropy | <0.05 bpp where JPEG-XL collapses | research code, `neural/glc.rs` |

*Video:*

| capability | model | win vs zeroth | source |
|---|---|---|---|
| `wai.neural.video_bmshj2018` | per-frame bmshj2018 | 1000×+ vs raw RGB, browser-decodable on WebGPU | CompressAI |
| `wai.neural.dcvc_rt` | DCVC-RT (2025), inter-frame with reference-feature propagation | −21% vs VTM; real-time at 1080p/4K | research code, `neural/video_dcvc.rs` |
| `wai.neural.dcvc_fm` | DCVC-FM (feature modulation), wide single-model quality range (RGB + YUV) | SOTA-RD neural video; shares the inter-frame route | research code, `neural/video_dcvc.rs` |
| `wai.video.int_motion` | DETERMINISTIC INTEGER inter-frame — I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS, no float/ONNX; motion is whole-pel or **half-pel** (`mv_precision`), the latter predicting at the half-grid by integer bilinear interpolation (`(a+b+1)>>1`, `(a+b+c+d+2)>>2`) so real footage — which pans a non-integer number of pixels per frame — costs fewer residual bits while staying byte-exact | byte-identical frame-by-frame on every machine (*frame-state-equivalence*, a per-frame portable hash); the temporal extension of the integer contract, vs the float DCVC family that desyncs down the GOP | `int_video.rs` (whole-pel `tools/wai_int_video_encode.py`; half-pel from real footage `tools/wai_int_video_real_encode.py`) |
| `wai.video.int_hyper` | NEURAL-KEYFRAME inter-frame — the I-frame is the LEARNED integer codec `int_hyper` decoded byte-exact, the P-frames are integer block-MC residuals on top | a learned keyframe you can put in a receipt + integer inter-frames; whole decode integer, byte-identical everywhere | `int_video::decode_pframes` ∘ `int_hyper_synth::decode_conditional` |

**Sink-architecture note:** the inter-frame neural video codecs (DCVC-RT,
DCVC-FM) decode through `ort` and are **CPU-capable** for native sinks —
their real-time figures come from GPU execution, but the decode is not
CUDA-only. They are heavier than per-frame image codecs (a temporal net
plus reference-feature state), so a browser sink that cannot host the
temporal net should declare only `wai.neural.video_bmshj2018` (or another
browser-decodable neural video codec) for the `video` media class. The
choice is the deployer's, advertised per sink — the envelope is identical.

**TAESD is intentionally absent.** TAESD is a distilled SD VAE decoder
— it inverts a generative pipeline's encoder, not a general image
codec. Outside the SD pipeline it underperforms JPEG-XL at most
bitrates while costing ~5 MB of weights and ONNX compute. Don't ship
codecs for the brand; ship them for the bit budget.

**Image/video scope is honest:** zeroth codecs (JPEG-XL, AV1) are
already SOTA-adjacent in the bitrate bands where most media lives.
Neural image/video capabilities are registered for the bands where
zeroth codecs collapse (ultra-low bpp images, bandwidth-constrained
streaming) — not as defaults.

Each capability owns its payload format. Decoder implementations
(Python source-side, ONNX-export, browser onnxruntime-web) live in
`tools/wai_<capability>_*.py` and `wai-web/demo/`. New neural
capabilities are added by registration; the WAI envelope is unchanged.

See [`extensions/jwp-receipts.md`](extensions/jwp-receipts.md) for the
receipt-chain sub-spec covering per-object content hashes, per-group
Merkle roots, Ed25519 group signatures, and cross-hop preservation.
See [`extensions/confidentiality.md`](extensions/confidentiality.md) for the
private-channel sub-spec: MLS (RFC 9420) group keys + SFrame (RFC 9605)
object encryption, the byte-level framing behind §6.

### Interactive & temporal content (extensions)

Media that consumes input or moves over time. Each registers its own
payload format and a deterministic `*-equivalence` conformance criterion
over the `wai.det.fixed64` fixed-point floor — the reconstruct half of
the standard. See the linked extension for each.

| capability | payload | conformance · extension |
|---|---|---|
| `wai.world.scene` / `.replay` / `.live` | WWLD container (op log) | replay-equivalence · [interactive-worlds](extensions/interactive-worlds.md) |
| `wai.world.model` | generative-world transport: a signed **action trace** (the world CausalOp log) + a declared `model_id` (`world_model.rs`) | the wire carries the *action trace*, not the pixels — it replays under canonical total order to one **session hash** on every machine (replay-equivalence), the generated frames vouched for by a signed, joules-accounted provenance receipt. The neural sibling of `.replay`; with an `int_*` renderer the frames hash-converge too (a generative interactive world you can put in a receipt) |
| `wai.mesh.glb` | glTF 2.0 Binary (Khronos) | asset, content-addressed (decoder sink-supplied) |
| `wai.splat.spz` | SPZ (Niantic) | asset, content-addressed; presentation only |
| `wai.audio.scene` | WSCN scene (objects + trajectories) | mixdown-equivalence · [spatial-audio](extensions/spatial-audio.md) |
| `wai.audio.score` | WSCR container (note events) | music as instructions; sink synthesizes, mixdown-equivalence · [score](extensions/score.md) |
| `wai.audio.int_codec` | LEARNED 1-D transform codec — integer transposed-conv synthesis + integer IGDN over a factorized-rANS latent (model + bitstream from `tools/wai_int_audio_export.py`) | **sample-equivalence**: int16 PCM byte-identical on every machine (a portable PCM hash); the audio analog of `wai.neural.int_hyper`, vs the float SoundStream/EnCodec family (`wai.neural.{encodec32,dac,mimi,…}`) that desyncs on ~1e-5 round-off. Pure i64, no float/ONNX (`int_audio.rs`) |
| `wai.haptic.signal` | WHAP envelope (intensity/sharpness keyframes) | sample-equivalence · [haptics](extensions/haptics.md) |
| `wai.feed.state` | WWLD, sim `wai.world.sim.feed0` | replay-equivalence · [state-feeds](extensions/state-feeds.md) |
| `wai.avatar.pose` | WWLD, sim `wai.world.sim.skeletal0` | pose-equivalence · [avatars](extensions/avatars.md) |
| `wai.splat.4d` | W4DS container (per-Gaussian keyframed tracks) | frame-state-equivalence · [volumetric](extensions/volumetric.md) |
| `wai.splat.int_codec` | LEARNED Gaussian-splat ATTRIBUTE codec — a learned transform over the `[14,S,S]` attribute image (pos/scale/rot/color/opacity), integer synthesis (int_hyper machinery, 14 channels) over a factorized-rANS latent (`int_splat.rs`; model + bitstream from `tools/wai_int_splat_export.py`) | **splat-equivalence**: the decoded attribute tensor is byte-identical on every machine (a portable cloud hash) — the int_hyper of 3D, vs the float SOG / self-organizing-Gaussian codecs whose decode desyncs. The float rasterizer stays sink-supplied presentation; the *attributes* are the contract. Pure i64, no float/ONNX |
| `wai.film.linear` | WFLM container (sealed world + camera track + frame schedule) | film-equivalence · [linear-film](extensions/linear-film.md) |
| `wai.film.reel` | WREL container (sealed world + a sequence of shots = camera track + schedule each) | reel-equivalence (film-equivalence with cuts) · [linear-film](extensions/linear-film.md) |

And the verify half — a signed, energy-accounted lineage for opaque
captured content WAI does **not** reconstruct (content-addressing +
receipts + measured joules): [`extensions/provenance.md`](extensions/provenance.md)
(`wai.asset.provenance`).

### Immersive audio, cinema, volumetric, haptics, HDR (registered, sink-supplied)

The same "name the capability, the sink supplies the decoder" pattern extends
to the standard carriers of the immersive / theatrical / volumetric media a
full distributor needs. Each names an open reference where one exists; the
proprietary and SEP-pool entries are flagged, never defaults. Payload is the
named standard's canonical bytes.

| class | capabilities |
|---|---|
| immersive audio | `wai.audio.iamf` (IAMF/Eclipsa, royalty-free) · `wai.audio.ac4` (Dolby AC-4, proprietary) · `wai.audio.ivas` (3GPP IVAS, SEP-pool) |
| cinema mastering | `wai.image.jp2` · `wai.video.j2k` (JPEG 2000 / DCP, OpenJPEG) |
| volumetric / 3D | `wai.mesh.draco` (royalty-free) · `wai.pc.gpcc` · `wai.pc.vpcc` (MPEG point clouds) · `wai.splat.sog` (Gaussian-splat compression, candidate) |
| haptics | `wai.haptic.mihs` (MPEG-I Haptics, ISO/IEC 23090-31; its 23090-33 ships bit-exact conformance vectors) |
| HDR dynamic metadata | `wai.meta.hdr10plus` (royalty-free) · `wai.meta.dovi` (Dolby Vision, proprietary) — **companion descriptors** to a video payload, not standalone media |

Every registered capability also carries an IP class —
`codecs::capability_license(cap)` ∈ {royalty-free, patent-pool, proprietary,
non-commercial}, with `is_royalty_free(cap)` as the predicate. It is a registry
property (not a per-file manifest claim — a codec's licence can't be honestly
asserted in an envelope's header), so a deployer can audit a title or a whole
catalogue in code: encode only to royalty-free capabilities, or flag every
pool/proprietary one. The per-title choice of which toll road to pay is a
property of the format, enforceable, not a policy on a slide.

See [`research/media-codec-registry.md`](research/media-codec-registry.md) for
the per-capability standard / spec number / verified royalty status / reference
library and the remaining spec follow-up (companion-component binding for the
HDR / audio metadata descriptors).

---

## 6. Confidentiality (OPTIONAL — encrypted payloads / private channels)

By default a WAI payload is **plaintext**: any sink that advertises the
named capability (§4–§5) can decode it. This is correct for public
media — a shared world, a broadcast film. Some channels are instead
**private**: only an intended group may recover the content, and any
relay or multiplexer that carries it MUST be able to route, meter, and
attest the bytes **without** reading them.

Confidentiality is an OPTIONAL layer that is **orthogonal to the
capability**. It wraps the payload bytes; the capability still names how
to decode the *plaintext* once recovered. A sink decrypts first, then
runs capability dispatch (§4) on the plaintext.

### 6.1 Manifest field

A manifest MAY carry a `security` object:

```json
"security": { "encryption": "mls-sframe", "kid": 0 }
```

- `security.encryption` — encryption scheme id. Absent, `null`, or
  `"none"` ⇒ the payload is plaintext (the default; §1–§5 unchanged).
  `"mls-sframe"` ⇒ the payload is an SFrame (RFC 9605) ciphertext object
  whose group key is agreed by MLS (RFC 9420) — the
  `draft-jennings-moq-e2ee-mls` binding.
- `security.kid` — the SFrame key id selecting which sender/epoch key
  opens the object. It is also carried in the SFrame header; the
  manifest copy lets a multiplexer route without parsing the payload.

`media`, `intent`, `conditioning.kind`, and
`model_requirement.capability` continue to describe the **plaintext**
codec. They are NOT encrypted and remain visible for routing —
confidentiality protects the *content*, not the *shape*.

### 6.2 Keys are a requirement of existence

Exactly as the model behind a neural capability is never transported
(§1), the **group key is never transported in the WAI file**. Group
membership is established out of band via MLS; each member derives the
32-byte SFrame base key from the group's exporter secret under the label
`"moq-jwp sframe base key"`. A sink that is not a group member cannot
derive the key and the file is **inert at that sink** — the
confidentiality analog of a missing capability.

### 6.3 Relay transparency and attestation

A WAI relay or multiplexer that carries an encrypted payload MUST treat
it as opaque: it MUST NOT require the plaintext to route, meter, or
attest, and MUST NOT attempt capability dispatch on ciphertext. The JWP
receipt chain ([`extensions/jwp-receipts.md`](extensions/jwp-receipts.md))
commits over the **ciphertext** content hash, so per-object delivery,
energy, and integrity are attested end-to-end without any relay holding
the key. Content secrecy (SFrame/MLS) and delivery attestation (JWP) are
independent layers over the same bytes and compose: a relay proves it
forwarded the exact sealed bytes it was given while never learning what
they say.

### 6.4 Decrypt-then-dispatch

A confidentiality-aware sink MUST, before running §4:

1. If `security.encryption` is absent / `null` / `"none"`, proceed to §4
   on the payload unchanged.
2. Else, if the sink does not implement the named scheme, or holds no
   key for `security.kid`, the file is **inert at this sink** — return a
   clear "no key / unsupported encryption" error. It MUST NOT feed
   ciphertext to a codec.
3. Else recover the plaintext payload via the named scheme (for
   `mls-sframe`: SFrame `open` under the member's base key), then run §4
   on the recovered plaintext.

A sink that does not implement confidentiality at all MUST refuse an
envelope whose manifest carries a non-`none` `security.encryption`
cleanly, rather than mis-decode ciphertext.

Byte-level SFrame framing, the SFrame key schedule, and the MLS binding
(including per-sender keying for multi-sender channels) are specified in
[`extensions/confidentiality.md`](extensions/confidentiality.md). The
reference implementation is the `moq-sframe` (SFrame) and
`joule-comm-mls` (MLS exporter → base key) crates, with browser bindings
in `joule-comm-mls-wasm`; `Track::e2e_encrypted` is the switch that
selects this path. Live: https://wai.transaction.science/private-channel

---

## 7. Conformance

- A **conforming sink** MUST: implement the container (§2), capability
  dispatch (§4), and the mandatory floor (§5).
- A **conforming encoder** MUST emit containers (§2) that any
  conforming sink can read; the payload MUST be the canonical byte
  stream for the registered capability.
- Conformance is verified by **bit-exact decode-equivalence with the
  reference library** named in §5 for each registered capability. The
  WAI Rust reference impl in `wai-rs/` is built on those very
  libraries and ships a test suite that exercises every registered
  capability through the full WAI envelope (`cargo test --lib`).

### Replicate-soundness — the integer-entropy contract (normative)

`intent = replicate` claims the sink reproduces the *exact* source bytes. For a
**neural** capability that claim is only sound if the decode is byte-identical
on every machine — and that is **not** automatic. A learned codec's entropy
decode desyncs the instant the decoder's scale-bucket / σ computation drifts
from the encoder's by one float ULP: the bucket assignment flips and the rest
of the stream corrupts. So a neural decode that runs **float** inference is
reproducible only on the *same build*, never guaranteed across hardware.

Therefore a neural capability MAY back `intent = replicate` **only** if its
entire decode path — entropy parameterization *and* synthesis — is
integer / fixed-point, with no float operation anywhere (the **integer-entropy
contract**). The reference impl factors the halves into ORT-free, pure-i64
modules a codec composes: `int_entropy` (the deterministic scale→bucket step
that removes the float-σ desync), `int_synth` (integer synthesis), and the
integer rANS — assembled into a full conditional decode by `int_hyper_synth`
(`wai.neural.int_hyper`: factorized rANS-decode `z` → integer hyper-decoder
`h_s` → σ,μ → integer-σ bucket → rANS-decode residual → `y_hat = r + μ` →
integer synthesis — both bitstreams entropy-coded, no float anywhere). This is
a property of the capability, classified in the
registry as `codecs::capability_replicate_class(cap) -> ReplicateClass`:

- **`NeuralIntegerExact`** — integer by construction; byte-identical on every
  machine; the only neural class that soundly backs cross-hardware `replicate`.
  (`wai.neural.int_synth` is synthesis only; `wai.neural.int_hyper` is the full
  conditional codec — σ source, entropy, and synthesis all integer.)
- **`GenerativeDeterministic`** — the *create*-side of byte-exact conformance.
  A generator (`intent = create`) whose entire pipeline is integer/fixed-point
  and pinned to `(model, prompt, seed, steps)`, so the **generated** content is
  byte-identical on every machine. There is no original to reconstruct, yet the
  output hash is portable and receiptable — *generation-equivalence*. The
  fixed-point sampler is mandatory: a float RNG or float sampling step desyncs
  the stream exactly as a float σ does. (`wai.generate.spectral`: a pure-integer
  spectral synthesis — prompt+seed draw integer-LUT sinusoid components by a
  `splitmix64` PRNG; the float `sin()` that bakes the LUT runs once at
  model-build, never at generation.) Distinct from `Behavioral`, which conforms
  by similarity and never reproduces bytes.
- **`NeuralFloat`** — runs float inference; exact only on the same build. An
  encoder MUST NOT emit `intent = replicate` for one of these; `perceptual` /
  "exact on this build" is the honest claim.
- **`StandardDefined`** — reconstruction is fixed by the named codec/standard
  (the floor + menu codecs) or by WAI's deterministic floor (§ extensions);
  the neural float hazard does not apply.
- **`Behavioral`** — a sink-generator path that conforms by similarity, not
  bytes (`wai.semantic.*`).

A conforming sink that receives `intent = replicate` for a `NeuralFloat`
capability MUST refuse the replicate claim (decode as perceptual or reject).
The reference impl enforces this per-envelope in `decode_envelope_strict`, which
rejects a `replicate` envelope whose pinned prior is not integer-exact
(`DeterminismClass`) — and a `NeuralFloat` capability cannot honestly carry an
integer-exact prior. The registry class is the *static* audit of the same
property: `wai capabilities` prints it per row and reports the count of neural
integer-exact capabilities, so a deployer can require "dispatch `replicate` only
to a cross-hardware-sound decode" in code, exactly as it audits royalty status.

This is the standard fixing the one axis the learned-codec field leaves
implicit: a neural codec that is *integer all the way down* is the only one that
can put a reconstruction in a receipt and have it verify on someone else's
machine.

---

## 7.1. Multi-rendition envelope (v1.1 extension)

The v1.0 single-payload envelope is sufficient when one source produces
one rendition for one capability. v1.1 adds an OPTIONAL multi-payload
form for cases where the source emits the SAME content in multiple
renditions and lets the sink pick — typically because the sink's
compute budget, bandwidth budget, or runtime preference (browser-
native vs WebGPU vs WebNN vs native ML) differs at decode time.

The `model_requirement.fallback` field in v1.0 is **declarative** — it
names a capability the sink CAN look for. v1.1 makes fallback (and
multi-rendition selection generally) **executable** by carrying every
declared capability's payload in one envelope.

### Self-contained-ecosystem note

In a controlled-deployment ecosystem (every sink ships from one
authority and therefore has a known capability set) the multi-rendition
form is NOT a "graceful fallback for missing capability" — every sink
has every required capability. Renditions exist for **policy selection**:
high-quality on a desktop sink with WebGPU; low-quality on a phone sink;
quality-vs-latency by bandwidth headroom; native vs WebNN by which
runtime is initialized. The sink picks by deployer-defined policy, not
by which capability happens to be installed.

### v1.1 wire format

```
+--------+----------------+-------------------+----------------+----...---+
| "WAI2" | u32  man_len   | manifest (JSON)   | rendition table| payloads |
+--------+----------------+-------------------+----------------+----...---+
```

- Magic is `WAI2` (distinct from v1.0's `WAI1`; sinks dispatch by magic).
- Manifest is JSON with a single `renditions` array of self-contained
  rendition entries, in deployer-preferred order:
  ```json
  {
    "wai":    "1.1",
    "media":  "audio",
    "intent": "replicate",
    "renditions": [
      { "capability": "wai.neural.encodec32", "kind": "encodec_tokens" },
      { "capability": "wai.audio.opus",       "kind": "opus"           }
    ],
    "target": { "sr": 48000, "dur": 5.0 }
  }
  ```
  Each entry has its own `capability` (the dispatch key) and `kind`
  (the codec id matching the bytes in the corresponding payload).
- Rendition table: `u16 n_renditions | n × (u32 offset, u32 length)`,
  offsets relative to the start of the payloads block. The table's
  i-th entry corresponds to `renditions[i]` in the manifest.
- Payloads block: the rendition byte ranges, packed contiguously.
- `n_renditions` MUST equal the manifest's `renditions.length`.

### Selection algorithm

A v1.1-aware sink MUST:

1. Read the manifest, get the ordered list of `model_requirement`
   entries.
2. Walk the list in order; pick the first entry whose capability the
   sink either advertises OR whose declared backend the sink prefers
   under its deployer policy.
3. Read the corresponding rendition's bytes from the payload block.
4. Dispatch as in v1.0 §4.

A v1.0 sink that encounters a `WAI2` envelope MUST refuse it cleanly
(unknown major). No silent fallback to the first payload — that would
hide a major-version mismatch.

### Companion components (descriptors bound to a payload)

A `WAI2` component carries an OPTIONAL `role`. Absent ⇒ `"rendition"` (a
selectable alternative of the same content — the default above).
`"companion"` marks a **descriptor applied *with* the chosen rendition, not
instead of it**, naming the component it binds to in `companion_of`. This
resolves the one media class that is not a standalone payload: **HDR dynamic
metadata** (`wai.meta.hdr10plus` / `wai.meta.dovi`, SMPTE ST 2094) and
immersive-audio metadata accompany a video/audio payload rather than replacing
it. Carried as companions, the descriptor and its target ride in the **same
envelope**, so the binding is intrinsic — not a spoofable sidecar.

The selection algorithm is unchanged: a sink picks a rendition among the
`role != "companion"` entries, then applies every companion whose
`companion_of` names the chosen component (e.g. the HDR10+ block for the AV1
video the sink picked). `pick`/`primary` never return a companion. Static HDR
characteristics (ST 2086 mastering display, HLG transfer) stay in-band in the
video bitstream / a manifest color field and need no companion.

### v1.0 compatibility

v1.0 envelopes (`WAI1`) remain valid and are unchanged. Encoders
SHOULD emit `WAI1` when only one rendition is present (simpler form,
wider sink support). `WAI2` is for sources that genuinely benefit from
shipping multiple renditions per file.

---

## 8. Versioning

- The `wai` manifest field is `MAJOR.MINOR`. A reader MUST reject an
  unknown MAJOR. A reader MUST accept an unknown MINOR by ignoring
  fields it does not understand, provided MAJOR matches.
- The container format (§2) and the manifest schema (§3) are frozen
  for all of `1.x`.
- New capabilities (§5) MAY be added in any MINOR revision. Existing
  capabilities MUST NOT be redefined — once a capability string is
  registered for a codec, its payload format is fixed forever.
- Removing or repurposing an existing capability requires a new MAJOR.

---

## Appendix A — Reference implementation

`wai-rs/` (Rust). Builds as `lib`, `cdylib`, `staticlib` so the C ABI
in `wai-rs/src/ffi.rs` drops in as `libwai.dylib` / `.so` / `.dll` and
can be called from any language. The crate wraps:

- **image**: `image` crate (PNG, JPEG, AVIF via `ravif`), `jpegxl-rs`
  (libjxl 0.11.x).
- **audio**: `opus` (libopus 1.x), `claxon` + `flac-bound` (libFLAC).
- **video**: `rav1e` (AV1 encode) + `dav1d` (AV1 decode).
- **text**: `zstd` (libzstd), `xz2` (liblzma).

Run `cargo test --lib` (with `RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"`
on macOS, or via the committed `.cargo/config.toml`).

## Appendix B — Why a *capability menu*, not a custom codec

Earlier drafts of WAI defined custom transform + entropy stages for
each medium. That direction was dropped: re-implementing JPEG / MPEG /
Opus poorly is strictly worse than calling the field's mature
libraries. WAI's value is the envelope + capability dispatch + the
neural-shared-prior model, all of which are absent from existing
standards. The zeroth condition's purpose is **availability**, not
codec novelty.

## Appendix C — License

Apache-2.0. The standard and reference implementation are open;
royalty-free implementation is a precondition for adoption (avoiding
the H.264/H.265 royalty trap; following the VP9/AV1/AVIF/JPEG-XL open-
codec precedent).