ea31d22854
Bring the git-visible surfaces in line with the framing rules (self-referential;
no competitive comparisons, no vanity metrics, describe by capability):
- SPEC.md neural table: drop company prefixes (Meta/Descript-company/Kyutai → the
capability's own model identifier), drop BD-rate "vs VTM/VVC" comparisons and the
"current SOTA" superlative (describe each capability's regime instead), and reword
the "vs the float X family that desyncs" lines as self-referential statements of
why the integer contract exists ("a float decode desyncs on…").
- README status line: "passes 11 … tests" → "round-trips every registered capability
through the full envelope" (a count that describes coverage, not a number that drifts).
- glossary: "canonical 342-entry dataset" → "canonical survey dataset".
Unchanged on purpose: "dispatches to SOTA standard codecs" and similar — that is the
ingest thesis (we route to the best existing codecs), self-referential, not a
comparison of our own work; and the source-column model IDs, which are factual
dependency pointers.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
113 lines
5.8 KiB
Markdown
113 lines
5.8 KiB
Markdown
# WAI — Web AI Media Transport & Execution
|
||
|
||
A container + capability-dispatch standard for media. **WAI does not
|
||
re-implement codecs**; it dispatches to SOTA standard libraries (AVIF,
|
||
JPEG-XL, PNG, JPEG, Opus, FLAC, AV1, zstd, XZ) and adds an envelope
|
||
that lets the *neural-shared-prior* path coexist with the model-free
|
||
floor.
|
||
|
||
Two paths open every conforming file:
|
||
|
||
- **Neural condition** — the sink regenerates the media against a
|
||
shared ambient prior (a model). The model is a *requirement of
|
||
existence*, never shipped (an `.mp3` doesn't ship its decoder
|
||
either).
|
||
- **Zeroth condition** — a *menu of registered SOTA codecs*. Always
|
||
satisfiable on any device that ships ffmpeg or equivalent.
|
||
|
||
The capability a file requires is *named*, not supplied.
|
||
|
||
## What's here
|
||
|
||
| Path | Contents |
|
||
|------|----------|
|
||
| [`SPEC.md`](SPEC.md) | WAI v1.0 spec — envelope, manifest, capability menu |
|
||
| [`wai-rs/`](wai-rs/) | Rust reference implementation (lib + cdylib + staticlib + C FFI) |
|
||
| [`glossary/`](glossary/) | Reference catalog of media codecs surveyed for capability registration. [`codecs.json`](glossary/codecs.json) is the canonical survey dataset; [`SCHEMA.md`](glossary/SCHEMA.md) defines the shape and contribution rules. Rendered at [wai.transaction.science/glossary](https://wai.transaction.science/glossary). |
|
||
| [`corpus/`](corpus/) | Standard test material (Kodak/CLIC/Tecnick/SIPI images, Xiph Derf + UVG video, SQAM-class audio, enwik8 + Silesia text). Not committed; `CORPUS.md` documents reacquisition. |
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
cd wai-rs
|
||
cargo test --lib # every registered capability + envelope round-trip
|
||
cargo build --release --bin wai
|
||
cargo build --release --lib # produces libwai.dylib/.so/.dll + libwai.a
|
||
```
|
||
|
||
On macOS, libflac lives under the Homebrew "flac" formula prefix that
|
||
Cargo doesn't probe automatically. The committed `.cargo/config.toml`
|
||
adds the path; if you build outside the repo set
|
||
`RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"`.
|
||
|
||
## SDK consumption
|
||
|
||
`wai-rs` exposes a stable C ABI in [`src/ffi.rs`](wai-rs/src/ffi.rs) —
|
||
the cdylib drops in as `libwai.dylib`/`.so`/`.dll`. Bindings for any
|
||
language (Python ctypes, Node N-API, Swift, JNI, Go cgo, etc.) call
|
||
into:
|
||
|
||
- `wai_image_{png,jpeg,avif,jxl}_{encode,decode}`
|
||
- `wai_audio_{opus,flac}_{encode,decode}`
|
||
- `wai_text_{zstd,xz}_{encode,decode}`
|
||
- `wai_envelope_{pack,unpack}` — wrap/unwrap the WAI container
|
||
- `wai_buffer_free`, `wai_last_error` — memory + error handling
|
||
|
||
## Registered capabilities (the zeroth menu)
|
||
|
||
| Capability | Codec | Lossless? | Mandatory? |
|
||
|---|---|---|---|
|
||
| `wai.image.png` | PNG | yes | **mandatory floor** |
|
||
| `wai.image.jxl` | JPEG-XL | both | recommended |
|
||
| `wai.image.avif` | AVIF (AV1-based) | lossy | recommended |
|
||
| `wai.image.jpeg` | JPEG | lossy | recommended (legacy compat) |
|
||
| `wai.audio.flac` | FLAC | yes | **mandatory floor** |
|
||
| `wai.audio.opus` | Opus | lossy | recommended |
|
||
| `wai.video.av1` | AV1 | lossy | recommended |
|
||
| `wai.video.av1.lossless` | AV1 (`quantizer=0`) | YUV-lossless | recommended |
|
||
| `wai.text.zstd` | zstd | yes | **mandatory floor** |
|
||
| `wai.text.xz` | XZ/LZMA2 | yes | recommended |
|
||
|
||
Neural capabilities (`wai.neural.*`) are OPTIONAL and sink-advertised. Most
|
||
run float inference, so their decode is exact only on the build that produced
|
||
it. A distinguished subset is **integer-exact by construction** — an all-`i64`
|
||
decode with no float and no ONNX — so the reconstructed artifact is
|
||
byte-identical on every machine and its hash is portable. Conformance is a
|
||
hash: a deployer can require `intent=replicate` to dispatch only to these.
|
||
|
||
| Capability | Integer-exact decode of | Conformance |
|
||
|---|---|---|
|
||
| `wai.neural.int_hyper` | image — integer hyperprior σ source + integer entropy + integer synthesis | byte-exact (hash-portable) |
|
||
| `wai.neural.int_synth` | image — integer synthesis only | byte-exact (hash-portable) |
|
||
| `wai.video.int_hyper` | video — `int_hyper` I-frame + integer half-pel motion-compensated P-frames | byte-exact (hash-portable) |
|
||
| `wai.audio.int_codec` | audio — learned 1-D transform, integer synthesis → int16 PCM | byte-exact (hash-portable) |
|
||
| `wai.splat.int_codec` | 3D — learned Gaussian-splat-attribute transform, integer synthesis | byte-exact (hash-portable) |
|
||
|
||
## Interactive & temporal content (extensions)
|
||
|
||
Beyond the codec menu, WAI carries content that consumes input or moves
|
||
over time — reconstructed from instructions at the sink, over one
|
||
`wai.det.fixed64` fixed-point floor, each with its own deterministic
|
||
`*-equivalence` conformance criterion:
|
||
|
||
| Capability | What it carries | Conformance |
|
||
|---|---|---|
|
||
| `wai.world.{scene,replay,live}` | interactive worlds (signed op log) | replay-equivalence |
|
||
| `wai.world.model` | action-conditioned world model (signed action ops → next frame, integer transition + synthesis) | replay-equivalence |
|
||
| `wai.mesh.glb` / `wai.splat.spz` | 3D mesh / Gaussian-splat assets | content-addressed |
|
||
| `wai.audio.scene` | object-based spatial audio | mixdown-equivalence |
|
||
| `wai.audio.score` | music as note events (synthesized at the sink) | mixdown-equivalence |
|
||
| `wai.haptic.signal` | keyframed touch envelopes | sample-equivalence |
|
||
| `wai.feed.state` | scoreboards / telemetry / market feeds | replay-equivalence |
|
||
| `wai.avatar.pose` | digital humans (skeletal pose stream) | pose-equivalence |
|
||
| `wai.splat.4d` | volumetric / 4D temporal Gaussian splats | frame-state-equivalence |
|
||
| `wai.film.linear` | animation / CGI film (sealed world + camera track) | film-equivalence |
|
||
| `wai.film.reel` | a film with cuts (one world, a sequence of shots) | reel-equivalence |
|
||
|
||
And the verify half — a signed, energy-accounted lineage for opaque
|
||
captured content WAI does **not** reconstruct (`wai.asset.provenance`).
|
||
See [`SPEC.md`](SPEC.md) §5 and [`extensions/`](extensions/) for each.
|
||
|
||
## License
|
||
|
||
Apache-2.0 — see [`LICENSE`](LICENSE).
|