open-standards/wai/SPEC.md

# WAI v1.0 — Web AI Media Transport & Execution

Status: **Draft Standard**. Reference implementation: `wai-rs/` (Rust
lib + cdylib + staticlib, C FFI). Two paths open every conforming
file:

- the **neural condition** — the sink regenerates the media from a
  compact conditioning payload against a *shared ambient prior* (a
  model). The model is a **requirement of existence**, never
  transported and never hash-pinned, exactly as `.mp3` does not ship
  its decoder.
- the **zeroth condition** — a *menu of registered SOTA standard
  codecs*. Always satisfiable on any device that ships ffmpeg or
  equivalent. WAI does **not** define new codecs; it dispatches to the
  field's best-in-class lossy+lossless libraries (AVIF, JPEG-XL, PNG,
  Opus, FLAC, AV1, zstd, XZ).

The capability a file requires is *named*, not supplied. Sinks that
have the named capability use it; sinks that don't fall back to a
declared `fallback` capability.

Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119.

---

## 1. Scope and model

WAI is a **container + capability dispatch standard**. It is media-
general (image, audio, video, text), royalty-free, and built around the
principle that the standard's value is the envelope and the dispatch
rules — not in re-implementing existing SOTA codecs.

A WAI file declares (in its manifest):
- what media it carries,
- what *capability* a sink must have to decode it,
- a fallback capability the sink can use if it lacks the primary,
- and the conditioning bytes for that capability.

Conforming sinks MUST implement the container envelope (§2), the
manifest schema (§3), the capability-dispatch algorithm (§4), and at
least the **mandatory floor** capabilities (§5). Other capabilities
(neural and additional zeroth) are OPTIONAL but RECOMMENDED.

---

## 2. Container

```
+--------+----------------+-------------------+----------------+-----------+
| "WAI1" | u32  man_len   | manifest (JSON)   | u32 payload_len| payload   |
| 4 B    | little-endian  | man_len bytes     | little-endian  | bytes     |
+--------+----------------+-------------------+----------------+-----------+
```

- Bytes 0..4 MUST be ASCII `WAI1`. A reader MUST reject any file whose
  first 4 bytes differ.
- `man_len` and `payload_len` are unsigned 32-bit little-endian.
- The manifest is UTF-8 JSON, serialized with no insignificant
  whitespace, preserving key insertion order. Readers MUST NOT depend
  on key order for *semantics*; encoders SHOULD preserve order so files
  round-trip byte-identically.
- The payload is opaque codec bytes whose interpretation is fully
  determined by the manifest's `model_requirement.capability`.

All WAI multi-byte integers are little-endian unless stated otherwise.

---

## 3. Manifest schema

```json
{
  "wai": "1.0",
  "media": "image" | "audio" | "video" | "text",
  "intent": "replicate" | "create" | "improve" | "...",
  "model_requirement": { "capability": "<capability-string>",
                         "fallback":   "<capability-string>" | null },
  "conditioning": { "kind": "<codec-id>" },
  "security": { "encryption": "<scheme>" | "none", "kid": <u64> },
  "target":   { ... }
}
```

- `wai` — spec version. This document defines `"1.0"`. Unknown major →
  reject. Unknown minor → accept by ignoring unknown fields.
- `media` — media class; informational. The capability is authoritative.
  The interactive-worlds extension adds `"world"`; spatial audio `"audio-scene"`; haptics `"haptic"`.
- `intent` — generative intent. `replicate` (reconstruct the source) is
  the only intent with normative reconstruction semantics in v1.0.
- `model_requirement.capability` — the capability string a sink MUST
  resolve. See §5 for the registered set.
- `model_requirement.fallback` — a second capability the sink MAY use
  if it lacks the primary. `null` means the file is inert without the
  primary.
- `conditioning.kind` — short codec id matching the capability. Helps
  multiplexers route bytes without parsing the full capability string.
- `security` — OPTIONAL. Absent, `null`, or `encryption: "none"` ⇒ the
  payload is plaintext (the default). Otherwise the payload is encrypted
  and MUST be recovered before dispatch; see §6. The capability, media,
  and conditioning fields always describe the **plaintext** codec.
- `target` — codec-specific reconstruction parameters (dimensions,
  sample rate, duration, frame count). Informational; the payload is
  self-describing per the codec's own format.

---

## 4. Capability dispatch

A conforming sink MUST implement this algorithm:

1. Read the manifest. If `wai` major version is unknown → reject.
2. Let `cap = model_requirement.capability`.
3. If the sink advertises `cap`, decode `payload` via the codec
   registered for `cap` (§5) and return.
4. Else, let `fb = model_requirement.fallback`.
5. If `fb != null` and the sink advertises `fb`, decode via `fb`'s
   codec and return.
6. Else the file is **inert at this sink** — return a clear "missing
   capability" error to the calling application.

A WAI **payload always corresponds to exactly one capability**. v1.0
does not support multi-codec multiplexed payloads.

If the manifest declares `security.encryption` other than `"none"`
(§6), the sink MUST recover the plaintext payload before step 3 and run
this algorithm on the plaintext; see §6.4.

---

## 5. Registered capabilities (the zeroth menu)

WAI v1.0 registers the following capabilities. Each names a SOTA
royalty-free standard codec. The "payload format" column is the
on-the-wire payload bytes; in every case it is the canonical bytes
that codec normally emits (a JXL file, an Opus stream, etc.) — WAI
does not wrap or re-frame them.

### Mandatory floor

Every conforming sink MUST implement:

| capability | payload | role |
|---|---|---|
| `wai.image.png` | PNG (RFC 2083) | universal lossless image |
| `wai.audio.flac` | FLAC (xiph.org/flac) | universal lossless audio |
| `wai.text.zstd` | zstd (RFC 8478) | universal compressed bytes |

These three guarantee that any sink with the standard system codec
libraries can open *some* file in every WAI-supported media class.

### Recommended modern set

Every conforming sink SHOULD implement:

| capability | payload | role |
|---|---|---|
| `wai.image.jxl` | JPEG-XL (ISO/IEC 18181) | lossless + lossy, modern SOTA |
| `wai.image.avif` | AVIF (ISO/IEC 23000-22) | lossy SOTA (AV1-based) |
| `wai.image.jpeg` | JPEG (ITU-T T.81) | legacy compatibility |
| `wai.audio.opus` | Opus (RFC 6716) | lossy SOTA |
| `wai.video.av1` | AV1 (AOMedia) | lossy SOTA |
| `wai.video.av1.lossless` | AV1 (AOMedia), `quantizer=0` | mathematically lossless YUV (lossy in RGB due to color conversion) |
| `wai.text.xz` | XZ/LZMA2 | maximum classical text ratio |

### Neural capabilities (OPTIONAL, sink advertises if its ML runtime supports them)

Each names a model the sink MAY have installed. Payload is a compact
conditioning representation specific to the capability (latent tokens,
prompt embeddings, etc.). The standard does NOT specify the byte
layout of these payloads — the capability owner does.

Neural capabilities are picked on the **best-quality-per-bit at each
bitrate band** principle — not on whatever model is famous or small.
2026 SOTA per medium, with what each one delivers (a bitrate band, or —
for `int_synth` — a determinism guarantee):

**Audio** (RVQ / hierarchical token codecs):

| capability | model | characterization | source |
|---|---|---|---|
| `wai.neural.encodec32` | Meta EnCodec at 32 kHz, 4 codebooks | ~3 kbps acceptable music | `facebook/encodec_32khz` |
| `wai.neural.dac` | Descript Audio Codec at 44.1 kHz | 6 kbps near-transparent music | `descript/dac_44khz` |
| `wai.neural.mimi` | Kyutai Mimi, 12.5 Hz frame rate | ~1.1 kbps real-time speech | `kyutai/mimi` |
| `wai.neural.wavtokenizer` | WavTokenizer single-codebook VQ | 0.9 kbps, ultra-low-bitrate speech/music | `novateur/WavTokenizer-large-*` |
| `wai.neural.snac` | SNAC multi-resolution RVQ | ~0.98 kbps hierarchical speech; N variable-rate code streams | `hubertsiuzdak/snac_*` |

**Learned image** — one conditional-entropy decode core, increasingly expressive entropy + transform. The reference Rust crate decodes this whole family natively (the entropy decode is in-crate; the analysis/synthesis nets are sink-supplied ONNX); decoders are cross-checked byte-for-byte against the CompressAI reference where a pretrained model exists.

| capability | entropy model | characterization | source |
|---|---|---|---|
| `wai.neural.bmshj2018` | factorized prior | ~80× vs raw RGB at q=3 (~0.26 bpp, 29 dB PSNR) | CompressAI |
| `wai.neural.mbt2018_mean` | mean-scale **hyperprior** (Gaussian-conditional) | per-element scale/mean over the factorized baseline | CompressAI |
| `wai.neural.elic` | checkerboard **spatial context** | two-pass anchor→context→non-anchor | ELIC class |
| `wai.neural.mlicpp` | **channel-autoregressive** multi-reference | channel slices + checkerboard + global attention; −13.4% BD-rate vs VTM-17 | MLIC++ |
| `wai.neural.mlicv2` | enhanced multi-reference | MetaFormer transform; −20.5% BD-rate vs VTM-17 (decodes via the mlicpp core) | MLICv2 |
| `wai.neural.cmic` | SCCTX (space-channel context) | Content-Aware Mamba transform; −15.9% BD-rate vs VTM-21 (decodes via the mlicpp core) | CMIC |
| `wai.neural.glc` | **categorical / VQ** (not Gaussian): hyper + checkerboard categorical context | codebook indices via softmax → quantized CDF → categorical rANS → VQ synthesis; ultra-low bpp | GLC |
| `wai.neural.int_synth` | (integer fixed-point synthesis) | bit-identical decode on every machine — the cross-hardware `replicate` member | WAI-native |

**Video:**

| capability | model | characterization | source |
|---|---|---|---|
| `wai.neural.video_bmshj2018` | per-frame bmshj2018 (intra-only) | browser-decodable on WebGPU | CompressAI |
| `wai.neural.dcvc_rt` | DCVC-RT (inter-frame) | AV1-class quality at ~21% less bitrate; 100+ fps 1080p with an accelerator | DCVC-RT |
| `wai.neural.dcvc_fm` | DCVC-FM (feature modulation) | wide single-model quality range, RGB + YUV | DCVC-FM |

The inter-frame decode is a frame loop that propagates a reference feature; the
reference Rust crate runs it on **CPU** via ONNX Runtime (`neural/video_dcvc.rs`)
— an accelerator is a SPEED choice, not a decode requirement. The per-frame
entropy decode reuses the CompressAI-validated hyperprior path; the temporal net
is sink-supplied. Browser sinks that lack the temporal net fall back to
`wai.neural.video_bmshj2018` for the `video` media class.

**TAESD is intentionally absent.** TAESD is a distilled SD VAE decoder
— it inverts a generative pipeline's encoder, not a general image
codec. Outside the SD pipeline it underperforms JPEG-XL at most
bitrates while costing ~5 MB of weights and ONNX compute. Don't ship
codecs for the brand; ship them for the bit budget.

**Image/video scope is honest:** zeroth codecs (JPEG-XL, AV1) are
already SOTA-adjacent in the bitrate bands where most media lives.
Neural image/video capabilities are registered for the bands where
zeroth codecs collapse (ultra-low bpp images, bandwidth-constrained
streaming) — not as defaults.

### Interactive worlds (OPTIONAL — media that consumes input)

Registered by the [interactive-worlds extension](extensions/interactive-worlds.md);
payload formats and conformance (replay-equivalence: identical state
hash at every checkpoint, no tolerance parameter) are defined there.
Reference engine: the `world` feature of `wai-rs` (pure Rust, no ML
dep chain); conformance corpus: `world-conformance/`.

| capability | payload | role |
|---|---|---|
| `wai.world.scene` | WWLD container, op log empty | a world at tick 0 — shareable, explorable |
| `wai.world.replay` | WWLD container, op log sealed | a finished session — re-derives every state from kilobytes |
| `wai.world.live` | WWLD container, op log open | a joinable world; ops arrive by out-of-scope transport |
| `wai.mesh.glb` | glTF 2.0 Binary (Khronos) | asset media: meshes, materials, textures, armatures (decoder sink/library-supplied) |
| `wai.audio.scene` | WSCN scene (objects + trajectories) | object-based spatial audio; the sink mixes to its layout (stereo floor bit-exact) — see extensions/spatial-audio |
| `wai.haptic.signal` | WHAP envelope (intensity/sharpness keyframes) | haptics; the sink reconstructs an actuator signal (sample-equivalence) — see extensions/haptics |
| `wai.feed.state` | WWLD container, sim `wai.world.sim.feed0` | live state-feeds (scoreboard / telemetry / market tape): a signed typed-field op log, non-physics; replay-equivalence — see extensions/state-feeds |
| `wai.avatar.pose` | WWLD container, sim `wai.world.sim.skeletal0` | digital humans / avatars: a signed skeletal joint-transform op stream skinning a `wai.mesh.glb`; pose-equivalence — see extensions/avatars |
| `wai.splat.spz` | SPZ (Niantic, open-source) | asset media: Gaussian-splat scenes; presentation only, content-addressed; renderer sink-supplied |
| `wai.splat.4d` | W4DS container (per-Gaussian keyframed tracks) | volumetric / 4D temporal Gaussian splats; the sink interpolates the cloud at frame N; frame-state-equivalence — see extensions/volumetric |

Each capability owns its payload format. Decoder implementations
(Python source-side, ONNX-export, browser onnxruntime-web) live in
`tools/wai_<capability>_*.py` and `wai-web/demo/`. New neural
capabilities are added by registration; the WAI envelope is unchanged.

See [`extensions/jwp-receipts.md`](extensions/jwp-receipts.md) for the
receipt-chain sub-spec covering per-object content hashes, per-group
Merkle roots, Ed25519 group signatures, and cross-hop preservation.
See [`extensions/interactive-worlds.md`](extensions/interactive-worlds.md)
for the interactive-worlds sub-spec: deterministic content-addressed
worlds, signed causal-op logs, simulation contracts, session receipts,
and the passive-media fallback floor. See
[`extensions/spatial-audio.md`](extensions/spatial-audio.md) for object-based
spatial audio (mixdown-equivalence; the audio analog of interactive worlds).
See [`extensions/confidentiality.md`](extensions/confidentiality.md) for the
private-channel sub-spec: MLS (RFC 9420) group keys + SFrame (RFC 9605)
object encryption, the byte-level framing behind §6.

---

## 6. Confidentiality (OPTIONAL — encrypted payloads / private channels)

By default a WAI payload is **plaintext**: any sink that advertises the
named capability (§4–§5) can decode it. This is correct for public
media — a shared world, a broadcast film. Some channels are instead
**private**: only an intended group may recover the content, and any
relay or multiplexer that carries it MUST be able to route, meter, and
attest the bytes **without** reading them.

Confidentiality is an OPTIONAL layer that is **orthogonal to the
capability**. It wraps the payload bytes; the capability still names how
to decode the *plaintext* once recovered. A sink decrypts first, then
runs capability dispatch (§4) on the plaintext.

### 6.1 Manifest field

A manifest MAY carry a `security` object:

```json
"security": { "encryption": "mls-sframe", "kid": 0 }
```

- `security.encryption` — encryption scheme id. Absent, `null`, or
  `"none"` ⇒ the payload is plaintext (the default; §1–§5 unchanged).
  `"mls-sframe"` ⇒ the payload is an SFrame (RFC 9605) ciphertext object
  whose group key is agreed by MLS (RFC 9420) — the
  `draft-jennings-moq-e2ee-mls` binding.
- `security.kid` — the SFrame key id selecting which sender/epoch key
  opens the object. It is also carried in the SFrame header; the
  manifest copy lets a multiplexer route without parsing the payload.

`media`, `intent`, `conditioning.kind`, and
`model_requirement.capability` continue to describe the **plaintext**
codec. They are NOT encrypted and remain visible for routing —
confidentiality protects the *content*, not the *shape*.

### 6.2 Keys are a requirement of existence

Exactly as the model behind a neural capability is never transported
(§1), the **group key is never transported in the WAI file**. Group
membership is established out of band via MLS; each member derives the
32-byte SFrame base key from the group's exporter secret under the label
`"moq-jwp sframe base key"`. A sink that is not a group member cannot
derive the key and the file is **inert at that sink** — the
confidentiality analog of a missing capability.

### 6.3 Relay transparency and attestation

A WAI relay or multiplexer that carries an encrypted payload MUST treat
it as opaque: it MUST NOT require the plaintext to route, meter, or
attest, and MUST NOT attempt capability dispatch on ciphertext. The JWP
receipt chain ([`extensions/jwp-receipts.md`](extensions/jwp-receipts.md))
commits over the **ciphertext** content hash, so per-object delivery,
energy, and integrity are attested end-to-end without any relay holding
the key. Content secrecy (SFrame/MLS) and delivery attestation (JWP) are
independent layers over the same bytes and compose: a relay proves it
forwarded the exact sealed bytes it was given while never learning what
they say.

### 6.4 Decrypt-then-dispatch

A confidentiality-aware sink MUST, before running §4:

1. If `security.encryption` is absent / `null` / `"none"`, proceed to §4
   on the payload unchanged.
2. Else, if the sink does not implement the named scheme, or holds no
   key for `security.kid`, the file is **inert at this sink** — return a
   clear "no key / unsupported encryption" error. It MUST NOT feed
   ciphertext to a codec.
3. Else recover the plaintext payload via the named scheme (for
   `mls-sframe`: SFrame `open` under the member's base key), then run §4
   on the recovered plaintext.

A sink that does not implement confidentiality at all MUST refuse an
envelope whose manifest carries a non-`none` `security.encryption`
cleanly, rather than mis-decode ciphertext.

Byte-level SFrame framing, the SFrame key schedule, and the MLS binding
(including per-sender keying for multi-sender channels) are specified in
[`extensions/confidentiality.md`](extensions/confidentiality.md). The
reference implementation is the `moq-sframe` (SFrame) and
`joule-comm-mls` (MLS exporter → base key) crates, with browser bindings
in `joule-comm-mls-wasm`; `Track::e2e_encrypted` is the switch that
selects this path. Live: https://wai.transaction.science/private-channel

---

## 7. Conformance

- A **conforming sink** MUST: implement the container (§2), capability
  dispatch (§4), and the mandatory floor (§5).
- A **conforming encoder** MUST emit containers (§2) that any
  conforming sink can read; the payload MUST be the canonical byte
  stream for the registered capability.
- Conformance is verified by **bit-exact decode-equivalence with the
  reference library** named in §5 for each registered capability. The
  WAI Rust reference impl in `wai-rs/` is built on those very
  libraries and ships a test suite that exercises every registered
  capability through the full WAI envelope (`cargo test --lib`).

---

## 7.1. Multi-rendition envelope (v1.1 extension)

The v1.0 single-payload envelope is sufficient when one source produces
one rendition for one capability. v1.1 adds an OPTIONAL multi-payload
form for cases where the source emits the SAME content in multiple
renditions and lets the sink pick — typically because the sink's
compute budget, bandwidth budget, or runtime preference (browser-
native vs WebGPU vs WebNN vs native ML) differs at decode time.

The `model_requirement.fallback` field in v1.0 is **declarative** — it
names a capability the sink CAN look for. v1.1 makes fallback (and
multi-rendition selection generally) **executable** by carrying every
declared capability's payload in one envelope.

### Self-contained-ecosystem note

In a controlled-deployment ecosystem (every sink ships from one
authority and therefore has a known capability set) the multi-rendition
form is NOT a "graceful fallback for missing capability" — every sink
has every required capability. Renditions exist for **policy selection**:
high-quality on a desktop sink with WebGPU; low-quality on a phone sink;
quality-vs-latency by bandwidth headroom; native vs WebNN by which
runtime is initialized. The sink picks by deployer-defined policy, not
by which capability happens to be installed.

### v1.1 wire format

```
+--------+----------------+-------------------+----------------+----...---+
| "WAI2" | u32  man_len   | manifest (JSON)   | rendition table| payloads |
+--------+----------------+-------------------+----------------+----...---+
```

- Magic is `WAI2` (distinct from v1.0's `WAI1`; sinks dispatch by magic).
- Manifest is JSON with a single `renditions` array of self-contained
  rendition entries, in deployer-preferred order:
  ```json
  {
    "wai":    "1.1",
    "media":  "audio",
    "intent": "replicate",
    "renditions": [
      { "capability": "wai.neural.encodec32", "kind": "encodec_tokens" },
      { "capability": "wai.audio.opus",       "kind": "opus"           }
    ],
    "target": { "sr": 48000, "dur": 5.0 }
  }
  ```
  Each entry has its own `capability` (the dispatch key) and `kind`
  (the codec id matching the bytes in the corresponding payload).
- Rendition table: `u16 n_renditions | n × (u32 offset, u32 length)`,
  offsets relative to the start of the payloads block. The table's
  i-th entry corresponds to `renditions[i]` in the manifest.
- Payloads block: the rendition byte ranges, packed contiguously.
- `n_renditions` MUST equal the manifest's `renditions.length`.

### Selection algorithm

A v1.1-aware sink MUST:

1. Read the manifest, get the ordered list of `model_requirement`
   entries.
2. Walk the list in order; pick the first entry whose capability the
   sink either advertises OR whose declared backend the sink prefers
   under its deployer policy.
3. Read the corresponding rendition's bytes from the payload block.
4. Dispatch as in v1.0 §4.

A v1.0 sink that encounters a `WAI2` envelope MUST refuse it cleanly
(unknown major). No silent fallback to the first payload — that would
hide a major-version mismatch.

### v1.0 compatibility

v1.0 envelopes (`WAI1`) remain valid and are unchanged. Encoders
SHOULD emit `WAI1` when only one rendition is present (simpler form,
wider sink support). `WAI2` is for sources that genuinely benefit from
shipping multiple renditions per file.

---

## 8. Versioning

- The `wai` manifest field is `MAJOR.MINOR`. A reader MUST reject an
  unknown MAJOR. A reader MUST accept an unknown MINOR by ignoring
  fields it does not understand, provided MAJOR matches.
- The container format (§2) and the manifest schema (§3) are frozen
  for all of `1.x`.
- New capabilities (§5) MAY be added in any MINOR revision. Existing
  capabilities MUST NOT be redefined — once a capability string is
  registered for a codec, its payload format is fixed forever.
- Removing or repurposing an existing capability requires a new MAJOR.

---

## Appendix A — Reference implementation

`wai-rs/` (Rust). Builds as `lib`, `cdylib`, `staticlib` so the C ABI
in `wai-rs/src/ffi.rs` drops in as `libwai.dylib` / `.so` / `.dll` and
can be called from any language. The crate wraps:

- **image**: `image` crate (PNG, JPEG, AVIF via `ravif`), `jpegxl-rs`
  (libjxl 0.11.x).
- **audio**: `opus` (libopus 1.x), `claxon` + `flac-bound` (libFLAC).
- **video**: `rav1e` (AV1 encode) + `dav1d` (AV1 decode).
- **text**: `zstd` (libzstd), `xz2` (liblzma).

Run `cargo test --lib` (with `RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"`
on macOS, or via the committed `.cargo/config.toml`).

## Appendix B — Why a *capability menu*, not a custom codec

Earlier drafts of WAI defined custom transform + entropy stages for
each medium. That direction was dropped: re-implementing JPEG / MPEG /
Opus poorly is strictly worse than calling the field's mature
libraries. WAI's value is the envelope + capability dispatch + the
neural-shared-prior model, all of which are absent from existing
standards. The zeroth condition's purpose is **availability**, not
codec novelty.

## Appendix C — License

Apache-2.0. The standard and reference implementation are open;
royalty-free implementation is a precondition for adoption (avoiding
the H.264/H.265 royalty trap; following the VP9/AV1/AVIF/JPEG-XL open-
codec precedent).