fc5c19b696
All checks were successful
no-vanity-metrics / Scan committed Markdown for vanity metrics (push) Successful in 3s
WAI conformance / cargo-class corpora · linux (push) Successful in 30s
WAI conformance / cargo-class corpora · linux-arm64 (push) Successful in 41s
WAI conformance / cross-arch byte-equality · linux (push) Successful in 13s
WAI conformance / cross-arch byte-equality · linux-arm64 (push) Successful in 18s
WAI conformance / cargo-class corpora · windows (push) Successful in 1m46s
WAI conformance / cross-arch byte-equality · windows (push) Successful in 31s
WAI conformance / cargo-class corpora · macos (push) Successful in 1m18s
WAI conformance / cross-arch byte-equality · macos (push) Successful in 19s
`wai-rs` builds the codec chain by default, and those crates link system libraries rather than vendoring them -- which follows from "WAI does not re-implement codecs", but means a fresh clone fails inside a codec crate's build script with no hint that the missing piece is an OS package. Nothing in the tree said so. Lists the packages for macOS and Debian/Ubuntu, and records the `--no-default-features` path that skips the C chain entirely (the envelope, manifest and `world` engine don't need it -- it's how wasm32 is built). Debian names checked against Ubuntu 24.04; macOS set verified by building wai-rs with default features after installing them. Also corrects the jpegxl-rs comment, which claimed system libjxl 0.11.x specifically -- 0.12.0 builds fine. Co-Authored-By: Claude Opus 5 <[email protected]>
135 lines
6.4 KiB
Markdown
135 lines
6.4 KiB
Markdown
# WAI — Web AI Media Transport & Execution
|
||
|
||
A container + capability-dispatch standard for media. **WAI does not
|
||
re-implement codecs**; it dispatches to SOTA standard libraries (AVIF,
|
||
JPEG-XL, PNG, JPEG, Opus, FLAC, AV1, zstd, XZ) and adds an envelope
|
||
that lets the *neural-shared-prior* path coexist with the model-free
|
||
floor.
|
||
|
||
Two paths open every conforming file:
|
||
|
||
- **Neural condition** — the sink regenerates the media against a
|
||
shared ambient prior (a model). The model is a *requirement of
|
||
existence*, never shipped (an `.mp3` doesn't ship its decoder
|
||
either).
|
||
- **Zeroth condition** — a *menu of registered SOTA codecs*. Always
|
||
satisfiable on any device that ships ffmpeg or equivalent.
|
||
|
||
The capability a file requires is *named*, not supplied.
|
||
|
||
## Building
|
||
|
||
`wai-rs` builds the codec chain by default (the `codecs` feature), and those
|
||
crates link **system** libraries rather than vendoring them — consistent with
|
||
"WAI does not re-implement codecs". Without them the build stops in a codec
|
||
crate's build script, not in WAI's own code:
|
||
|
||
```sh
|
||
# macOS
|
||
brew install jpeg-xl libavif opus dav1d xz
|
||
|
||
# Debian / Ubuntu
|
||
apt-get install libjxl-dev libavif-dev libopus-dev libdav1d-dev liblzma-dev
|
||
```
|
||
|
||
To build without the C codec chain — the envelope, manifest and `world`
|
||
engine do not need it, which is also how the wasm32 target is built:
|
||
|
||
```sh
|
||
cargo check -p wai --no-default-features
|
||
```
|
||
|
||
## What's here
|
||
|
||
| Path | Contents |
|
||
|------|----------|
|
||
| [`SPEC.md`](SPEC.md) | WAI v1.0 spec — envelope, manifest, capability menu |
|
||
| [`wai-rs/`](wai-rs/) | Rust reference implementation (lib + cdylib + staticlib + C FFI) |
|
||
| [`glossary/`](glossary/) | Reference catalog of media codecs surveyed for capability registration. [`codecs.json`](glossary/codecs.json) is the canonical survey dataset; [`SCHEMA.md`](glossary/SCHEMA.md) defines the shape and contribution rules. Rendered at [wai.transaction.science/glossary](https://wai.transaction.science/glossary). |
|
||
| [`corpus/`](corpus/) | Standard test material (Kodak/CLIC/Tecnick/SIPI images, Xiph Derf + UVG video, SQAM-class audio, enwik8 + Silesia text). Not committed; `CORPUS.md` documents reacquisition. |
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
cd wai-rs
|
||
cargo test --lib # every registered capability + envelope round-trip
|
||
cargo build --release --bin wai
|
||
cargo build --release --lib # produces libwai.dylib/.so/.dll + libwai.a
|
||
```
|
||
|
||
On macOS, libflac lives under the Homebrew "flac" formula prefix that
|
||
Cargo doesn't probe automatically. The committed `.cargo/config.toml`
|
||
adds the path; if you build outside the repo set
|
||
`RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"`.
|
||
|
||
## SDK consumption
|
||
|
||
`wai-rs` exposes a stable C ABI in [`src/ffi.rs`](wai-rs/src/ffi.rs) —
|
||
the cdylib drops in as `libwai.dylib`/`.so`/`.dll`. Bindings for any
|
||
language (Python ctypes, Node N-API, Swift, JNI, Go cgo, etc.) call
|
||
into:
|
||
|
||
- `wai_image_{png,jpeg,avif,jxl}_{encode,decode}`
|
||
- `wai_audio_{opus,flac}_{encode,decode}`
|
||
- `wai_text_{zstd,xz}_{encode,decode}`
|
||
- `wai_envelope_{pack,unpack}` — wrap/unwrap the WAI container
|
||
- `wai_buffer_free`, `wai_last_error` — memory + error handling
|
||
|
||
## Registered capabilities (the zeroth menu)
|
||
|
||
| Capability | Codec | Lossless? | Mandatory? |
|
||
|---|---|---|---|
|
||
| `wai.image.png` | PNG | yes | **mandatory floor** |
|
||
| `wai.image.jxl` | JPEG-XL | both | recommended |
|
||
| `wai.image.avif` | AVIF (AV1-based) | lossy | recommended |
|
||
| `wai.image.jpeg` | JPEG | lossy | recommended (legacy compat) |
|
||
| `wai.audio.flac` | FLAC | yes | **mandatory floor** |
|
||
| `wai.audio.opus` | Opus | lossy | recommended |
|
||
| `wai.video.av1` | AV1 | lossy | recommended |
|
||
| `wai.video.av1.lossless` | AV1 (`quantizer=0`) | YUV-lossless | recommended |
|
||
| `wai.text.zstd` | zstd | yes | **mandatory floor** |
|
||
| `wai.text.xz` | XZ/LZMA2 | yes | recommended |
|
||
|
||
Neural capabilities (`wai.neural.*`) are OPTIONAL and sink-advertised. Most
|
||
run float inference, so their decode is exact only on the build that produced
|
||
it. A distinguished subset is **integer-exact by construction** — an all-`i64`
|
||
decode with no float and no ONNX — so the reconstructed artifact is
|
||
byte-identical on every machine and its hash is portable. Conformance is a
|
||
hash: a deployer can require `intent=replicate` to dispatch only to these.
|
||
|
||
| Capability | Integer-exact decode of | Conformance |
|
||
|---|---|---|
|
||
| `wai.neural.int_hyper` | image — integer hyperprior σ source + integer entropy + integer synthesis | byte-exact (hash-portable) |
|
||
| `wai.neural.int_synth` | image — integer synthesis only | byte-exact (hash-portable) |
|
||
| `wai.video.int_hyper` | video — `int_hyper` I-frame + integer half-pel motion-compensated P-frames | byte-exact (hash-portable) |
|
||
| `wai.audio.int_codec` | audio — learned 1-D transform, integer synthesis → int16 PCM | byte-exact (hash-portable) |
|
||
| `wai.splat.int_codec` | 3D — learned Gaussian-splat-attribute transform, integer synthesis | byte-exact (hash-portable) |
|
||
|
||
## Interactive & temporal content (extensions)
|
||
|
||
Beyond the codec menu, WAI carries content that consumes input or moves
|
||
over time — reconstructed from instructions at the sink, over one
|
||
`wai.det.fixed64` fixed-point floor, each with its own deterministic
|
||
`*-equivalence` conformance criterion:
|
||
|
||
| Capability | What it carries | Conformance |
|
||
|---|---|---|
|
||
| `wai.world.{scene,replay,live}` | interactive worlds (signed op log) | replay-equivalence |
|
||
| `wai.world.model` | action-conditioned world model (signed action ops → next frame, integer transition + synthesis) | replay-equivalence |
|
||
| `wai.mesh.glb` / `wai.splat.spz` | 3D mesh / Gaussian-splat assets | content-addressed |
|
||
| `wai.audio.scene` | object-based spatial audio | mixdown-equivalence |
|
||
| `wai.audio.score` | music as note events (synthesized at the sink) | mixdown-equivalence |
|
||
| `wai.haptic.signal` | keyframed touch envelopes | sample-equivalence |
|
||
| `wai.feed.state` | scoreboards / telemetry / market feeds | replay-equivalence |
|
||
| `wai.avatar.pose` | digital humans (skeletal pose stream) | pose-equivalence |
|
||
| `wai.splat.4d` | volumetric / 4D temporal Gaussian splats | frame-state-equivalence |
|
||
| `wai.film.linear` | animation / CGI film (sealed world + camera track) | film-equivalence |
|
||
| `wai.film.reel` | a film with cuts (one world, a sequence of shots) | reel-equivalence |
|
||
|
||
And the verify half — a signed, energy-accounted lineage for opaque
|
||
captured content WAI does **not** reconstruct (`wai.asset.provenance`).
|
||
See [`SPEC.md`](SPEC.md) §5 and [`extensions/`](extensions/) for each.
|
||
|
||
## License
|
||
|
||
Apache-2.0 — see [`LICENSE`](LICENSE).
|