open-standards/wai/README.md

# WAI — Web AI Media Transport & Execution

A container + capability-dispatch standard for media. **WAI does not
re-implement codecs**; it dispatches to SOTA standard libraries (AVIF,
JPEG-XL, PNG, JPEG, Opus, FLAC, AV1, zstd, XZ) and adds an envelope
that lets the *neural-shared-prior* path coexist with the model-free
floor.

Two paths open every conforming file:

- **Neural condition** — the sink regenerates the media against a
  shared ambient prior (a model). The model is a *requirement of
  existence*, never shipped (an `.mp3` doesn't ship its decoder
  either).
- **Zeroth condition** — a *menu of registered SOTA codecs*. Always
  satisfiable on any device that ships ffmpeg or equivalent.

The capability a file requires is *named*, not supplied.

## What's here

| Path | Contents |
|------|----------|
| [`SPEC.md`](SPEC.md) | WAI v1.0 spec — envelope, manifest, capability menu |
| [`wai-rs/`](wai-rs/) | Rust reference implementation (lib + cdylib + staticlib + C FFI) |
| [`glossary/`](glossary/) | Reference catalog of media codecs surveyed for capability registration. [`codecs.json`](glossary/codecs.json) is the canonical 342-entry dataset; [`SCHEMA.md`](glossary/SCHEMA.md) defines the shape and contribution rules. Rendered at [wai.transaction.science/glossary](https://wai.transaction.science/glossary). |
| [`corpus/`](corpus/) | Standard test material (Kodak/CLIC/Tecnick/SIPI images, Xiph Derf + UVG video, SQAM-class audio, enwik8 + Silesia text). Not committed; `CORPUS.md` documents reacquisition. |

## Quickstart

```bash
cd wai-rs
cargo test --lib    # every registered capability + envelope round-trip
cargo build --release --bin wai
cargo build --release --lib    # produces libwai.dylib/.so/.dll + libwai.a
```

On macOS, libflac lives under the Homebrew "flac" formula prefix that
Cargo doesn't probe automatically. The committed `.cargo/config.toml`
adds the path; if you build outside the repo set
`RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"`.

## SDK consumption

`wai-rs` exposes a stable C ABI in [`src/ffi.rs`](wai-rs/src/ffi.rs) —
the cdylib drops in as `libwai.dylib`/`.so`/`.dll`. Bindings for any
language (Python ctypes, Node N-API, Swift, JNI, Go cgo, etc.) call
into:

- `wai_image_{png,jpeg,avif,jxl}_{encode,decode}`
- `wai_audio_{opus,flac}_{encode,decode}`
- `wai_text_{zstd,xz}_{encode,decode}`
- `wai_envelope_{pack,unpack}` — wrap/unwrap the WAI container
- `wai_buffer_free`, `wai_last_error` — memory + error handling

## Registered capabilities (the zeroth menu)

| Capability | Codec | Lossless? | Mandatory? |
|---|---|---|---|
| `wai.image.png` | PNG | yes | **mandatory floor** |
| `wai.image.jxl` | JPEG-XL | both | recommended |
| `wai.image.avif` | AVIF (AV1-based) | lossy | recommended |
| `wai.image.jpeg` | JPEG | lossy | recommended (legacy compat) |
| `wai.audio.flac` | FLAC | yes | **mandatory floor** |
| `wai.audio.opus` | Opus | lossy | recommended |
| `wai.video.av1` | AV1 | lossy | recommended |
| `wai.video.av1.lossless` | AV1 (`quantizer=0`) | YUV-lossless | recommended |
| `wai.text.zstd` | zstd | yes | **mandatory floor** |
| `wai.text.xz` | XZ/LZMA2 | yes | recommended |

Neural capabilities (`wai.neural.*`) are OPTIONAL and sink-advertised. Most
run float inference, so their decode is exact only on the build that produced
it. A distinguished subset is **integer-exact by construction** — an all-`i64`
decode with no float and no ONNX — so the reconstructed artifact is
byte-identical on every machine and its hash is portable. Conformance is a
hash: a deployer can require `intent=replicate` to dispatch only to these.

| Capability | Integer-exact decode of | Conformance |
|---|---|---|
| `wai.neural.int_hyper` | image — integer hyperprior σ source + integer entropy + integer synthesis | byte-exact (hash-portable) |
| `wai.neural.int_synth` | image — integer synthesis only | byte-exact (hash-portable) |
| `wai.video.int_hyper` | video — `int_hyper` I-frame + integer half-pel motion-compensated P-frames | byte-exact (hash-portable) |
| `wai.audio.int_codec` | audio — learned 1-D transform, integer synthesis → int16 PCM | byte-exact (hash-portable) |
| `wai.splat.int_codec` | 3D — learned Gaussian-splat-attribute transform, integer synthesis | byte-exact (hash-portable) |

## Interactive & temporal content (extensions)

Beyond the codec menu, WAI carries content that consumes input or moves
over time — reconstructed from instructions at the sink, over one
`wai.det.fixed64` fixed-point floor, each with its own deterministic
`*-equivalence` conformance criterion:

| Capability | What it carries | Conformance |
|---|---|---|
| `wai.world.{scene,replay,live}` | interactive worlds (signed op log) | replay-equivalence |
| `wai.world.model` | action-conditioned world model (signed action ops → next frame, integer transition + synthesis) | replay-equivalence |
| `wai.mesh.glb` / `wai.splat.spz` | 3D mesh / Gaussian-splat assets | content-addressed |
| `wai.audio.scene` | object-based spatial audio | mixdown-equivalence |
| `wai.audio.score` | music as note events (synthesized at the sink) | mixdown-equivalence |
| `wai.haptic.signal` | keyframed touch envelopes | sample-equivalence |
| `wai.feed.state` | scoreboards / telemetry / market feeds | replay-equivalence |
| `wai.avatar.pose` | digital humans (skeletal pose stream) | pose-equivalence |
| `wai.splat.4d` | volumetric / 4D temporal Gaussian splats | frame-state-equivalence |
| `wai.film.linear` | animation / CGI film (sealed world + camera track) | film-equivalence |
| `wai.film.reel` | a film with cuts (one world, a sequence of shots) | reel-equivalence |

And the verify half — a signed, energy-accounted lineage for opaque
captured content WAI does **not** reconstruct (`wai.asset.provenance`).
See [`SPEC.md`](SPEC.md) §5 and [`extensions/`](extensions/) for each.

## License

Apache-2.0 — see [`LICENSE`](LICENSE).