810 lines
49 KiB
Markdown
810 lines
49 KiB
Markdown
# WAI — changelog
|
||
|
||
Notable changes to the WAI standard and the reference implementation.
|
||
Newest at the top. WAI is a draft standard; until v1.0 freezes, the
|
||
wire format may change and such changes are recorded here.
|
||
|
||
## v0.4.0 — current
|
||
|
||
### JPEG AI — a live, self-sufficient `wai.image.jpegai` decode (2026-06-22)
|
||
|
||
- `neural::jpegai::decode(codestream, &JpegAiModel)` decodes a real ISO/IEC 6048
|
||
codestream to RGB with **no per-image schedule fixtures**: every per-image
|
||
quantity is derived from the bitstream, and only checkpoint constants come
|
||
from a sink-supplied model bundle. Wired into `decode_envelope` — JPEG AI is
|
||
now a dispatchable capability, not just reference/conformance support.
|
||
- The two derivations that closed it, traced bit-exact from the reference: the
|
||
residual coding mask is `sigma_skip > 382` (the σ surface *before* the rvs
|
||
addition — on the bucketed index it looks ambiguous, on `sigma_skip` it is
|
||
exact, 0 violations / 65536 positions); the dequant is
|
||
`resi × rvs_dq[grfs[ch]][lk] / 2^16 / scaler_vec[ch]`, with `grfs` determined
|
||
the same way the σ rvs does so the per-channel `scaler_vec` is consistent.
|
||
Matches the reference decoder's PNG to within rounding (max diff 1).
|
||
|
||
### Film reel — `wai.film.reel`, cuts over one world (2026-06-17)
|
||
|
||
- The editor's layer over `wai.film.linear`: a `WREL` embeds one sealed world
|
||
+ a sequence of **shots** (each its own camera track + frame schedule) played
|
||
back to back. Cutting wide→close, or one beat from two angles, is film
|
||
grammar; a reel carries it as a few kilobytes of camera keys, the world
|
||
shared across every shot. Conformance is **reel-equivalence** —
|
||
film-equivalence extended over cuts.
|
||
- Shots MAY revisit earlier world ticks (a cut back in time from a new angle),
|
||
so the world is sampled at the sorted/de-duplicated union of all frame ticks
|
||
— convergence still holds because replay-equivalence is order-independent.
|
||
- Reuses the film machinery (shared camera interpolation + pose bytes) + the
|
||
`FilmReceipt` profile (identity = `reel_hash`); `wai_meter` grows a `.wrel`
|
||
branch. Registered + tiered Lossless. Corpus vector `coverage` (the character
|
||
beat cut three ways, incl. a cut back in time) joins `film-conformance/`,
|
||
verified by `wai_film_vectors` on the four-platform matrix. SPEC §5 row +
|
||
`extensions/linear-film.md §7`.
|
||
|
||
### Linear film — `wai.film.linear` (2026-06-16)
|
||
|
||
- "A film is a sealed, camera-on-rails world." A generated film ships not
|
||
as pixels but as the **world** that produces it (an embedded sealed
|
||
`wai.world.replay`) + a **camera track** (keyframed pose over the world's
|
||
ticks) + a **frame schedule** (which tick each frame samples). The sink
|
||
renders the frames — lever 3 (instructions-at-the-sink) for the
|
||
synthesizable slice of "video" (animation / CGI / motion graphics). The
|
||
conformance corpus carries whole films in ~1 KB.
|
||
- `wai-rs` `film` feature (reuses the world engine + `Fx` floor + blake3):
|
||
the `WFLM` container (contract + embedded `WWLD` + camera track) and the
|
||
deterministic reconstruct contract. Conformance is **film-equivalence** —
|
||
per-frame `BLAKE3(world_state_hash(frame_tick) || interpolated camera
|
||
pose)` is bit-identical on every machine. It **composes** the embedded
|
||
world's replay-equivalence (inherited unchanged) with the camera track in
|
||
the `wai.det.fixed64` floor; the render (pixels) stays the sink's.
|
||
- The world engine gained one general primitive: `collect_checkpoints_
|
||
sampled` / `run_sampled_dispatch` — read the state hash at an arbitrary
|
||
frame cadence, independent of the contract's checkpoint cadence (a
|
||
renderer samples the timeline at its own frame rate).
|
||
- Corpus `film-conformance/` + `wai_film_vectors gen|verify` (dolly,
|
||
two-shot, crane-up) — all pass, on the four-platform CI matrix. Registered
|
||
+ tiered Lossless (film-equivalence is exact). `FilmReceipt` (per-frame
|
||
Merkle leaves) + a `wai_meter` `.wflm` branch emitting **measured** joules.
|
||
- Spec PROMOTED to `extensions/linear-film.md`, SPEC §5 temporal-content
|
||
row, content-taxonomy updated (animation-film → ✅). The biggest
|
||
commercial wedge in the taxonomy, now shipped.
|
||
|
||
### Music as instructions — `wai.audio.score`, LIVE at /score (2026-06-14)
|
||
|
||
- The deepest audio lever: a score carries **no recording**, only note
|
||
events — ship the composition, not the recording. `wai-rs` `score`
|
||
feature (reuses the world `Fx` floor + blake3): the `WSCR` container
|
||
(contract + note events) and a **trig-free** deterministic reference
|
||
synth — a phase accumulator drives a piecewise-linear oscillator
|
||
(saw/square/triangle/pulse) shaped by a linear ADSR, all `+ − × ÷`.
|
||
Pitch is a per-sample phase increment authored off-line (no exponential
|
||
in the floor). `mixdown_hash` is the **mixdown-equivalence** quantity,
|
||
the music counterpart of spatial audio; a soundfont/neural instrument
|
||
is the sink's presentation.
|
||
- Corpus `score-conformance/` + `wai_score_vectors gen|verify` (scale,
|
||
chord/polyphony, timbre-changing arpeggio) — all pass. Registered +
|
||
tiered Lossless (exact reference mix).
|
||
- Browser sink `ScoreSink` (`wai-web`) + `/score` page, LIVE: synthesizes
|
||
the score in-tab, verifies the mixdown hash vs the corpus, draws a
|
||
piano roll, and plays the reconstructed PCM via Web Audio. wasm pkg
|
||
uniquely named `wai_score`.
|
||
- Spec PROMOTED to `extensions/score.md` (site mirror), SPEC §5 row,
|
||
content-taxonomy updated (music → ✅). Completes the audio surface:
|
||
recorded objects (audio.scene) and synthesized notes (audio.score).
|
||
|
||
### Neural subsystem reconciled onto main (2026-06-14)
|
||
|
||
- Wires the full learned-image/audio decoder family into `neural/mod.rs`
|
||
and brings the categorical `rANS` (`decode_categorical`) it needs:
|
||
hyperprior (`mbt2018_mean`), checkerboard context (`elic`),
|
||
channel-autoregressive (`mlicpp`/`mlicv2`/`cmic`), integer-exact
|
||
(`int_synth`), `glc` (VQ), inter-frame video (`dcvc_rt`/`dcvc_fm`),
|
||
`snac` audio — plus the ISO/IEC 6048-1 JPEG-AI entropy + codestream
|
||
+ σ/value/filter port (`jpegai_*`).
|
||
- Restores the **semantic / generative path** (`semantic.rs`,
|
||
CLIP-similarity conformance, `ModelRequirement.min_sim`/`sim_metric`)
|
||
and the shared deterministic ONNX loader (`ort_session.rs`), behind
|
||
the `semantic` feature.
|
||
- Re-adds `byte-exact-conformance` (cross-arch rANS + int_synth + JPEG-AI
|
||
mANS goldens) and the model export/encode tooling under `tools/`.
|
||
- All features build + test green together (`neural` + `semantic` +
|
||
the interactive/temporal cargo classes); clippy clean.
|
||
|
||
### Asset provenance — `wai.asset.provenance`, LIVE at /provenance (2026-06-13)
|
||
|
||
- **Lever 4 — the "verifies whatever it carries" half.** For content that
|
||
cannot be reduced to instructions (a photo, a recording), WAI does not
|
||
reconstruct — it carries the opaque bytes content-addressed and attaches
|
||
a signed, energy-accounted **lineage**. `AssetReceipt` (`provenance.rs`,
|
||
`provenance` feature): a JWP group whose objects are processing
|
||
**steps** (op + input/output content hash + work + measured joules),
|
||
Merkle-bound, Ed25519-signed, with chain-linkage verification (each
|
||
step's input = the previous output, final output = the asset hash) and
|
||
`parent_receipt_hash` chaining. `from_json` so a verifying sink can
|
||
re-parse it.
|
||
- **`wai_prov` tool**: synthesizes a real image, builds a 3-step lineage
|
||
(origin → PNG master → published JPEG) with the **real measured CPU
|
||
energy** of each encode (IOReport via macmon, no sudo), wraps the JPEG
|
||
as a content-addressed `wai.image.jpeg` envelope, and emits the signed
|
||
receipt (origin 0 + PNG 0.43 J + JPEG 0.14 J = 0.57 J measured).
|
||
- **Browser sink — `AssetProvenance` (`wai-web`)** + `/provenance` demo,
|
||
LIVE: recomputes the asset's content hash in-tab and compares to the
|
||
receipt (tamper-evidence — a "corrupt one byte" button flips it red),
|
||
verifies the Ed25519 signature + Merkle chain, shows the energy-
|
||
accounted lineage, and hands the carried JPEG to a native `<img>` (the
|
||
browser decodes it; WAI does not). wasm pkg uniquely named `wai_prov`.
|
||
- **Spec PROMOTED to `extensions/provenance.md`** (site mirror). It is a
|
||
receipt profile *about* a carried asset (like `jwp-receipts`), not a §5
|
||
payload codec — the asset keeps its own capability (`wai.image.jpeg`).
|
||
Completes the thesis: carries whatever it can reconstruct, verifies
|
||
whatever it carries.
|
||
|
||
### Cargo-class polish wave — real cloud, EWA render, receipts, telepresence (2026-06-13)
|
||
|
||
Four follow-on builds across the new cargo classes:
|
||
|
||
- **Real-SPZ-derived 4D clip** (`wai-spz-gen 4d`): the `bloom` clip animates
|
||
1200 genuine Niantic-SPZ Gaussians (the same cloud the static
|
||
`wai.splat.spz` asset ships, round-tripped through the SPZ library) —
|
||
real cloud, authored motion. LIVE on `/volumetric`. Honest framing: the
|
||
win is a verifiable per-frame state, not capture fidelity.
|
||
- **WebGPU EWA render on `/volumetric`**: the page now renders the
|
||
reconstructed cloud with a real anisotropic EWA splat pipeline (the
|
||
same one the worlds demo uses) — per-Gaussian covariance projection,
|
||
conic, 3σ bounding quad, premultiplied blend, back-to-front sort — with
|
||
a 2D alpha-disc fallback where WebGPU is unavailable. The hash-verify
|
||
badge is render-independent. Confirmed on WebGPU (Chrome) + 2D fallback.
|
||
- **Per-class receipts with measured joules**: `HapticReceipt`
|
||
(`haptic/receipt.rs`) + `Splat4dReceipt` (`splat4d/receipt.rs`) — the
|
||
worlds/audio JWP receipt profile transposed to touch (per-channel
|
||
blocks) and volumetric (per-frame blocks), reusing `world::receipt::
|
||
merkle_root`. New `wai_meter` bin measures **real CPU energy** (IOReport
|
||
via macmon, no sudo) for haptic/splat4d reconstruction and emits a
|
||
signed, self-verified receipt carrying the measured `joules_micro`;
|
||
`wai_world_meter` now also dispatches `feed0`/`skeletal0`, so feed and
|
||
avatar worlds get the same measured `SessionReceipt`. Every cargo class
|
||
now has a measured-energy receipt path (haptic 57 µJ, splat4d 0.21 J for
|
||
1200 Gaussians × 17 frames, feed/avatar mJ-scale).
|
||
- **Avatar telepresence — LIVE at `/telepresence`**: a live skeletal world
|
||
of N avatars over the existing worlds relay. New `SkeletalActor`
|
||
(wasm) signs `CREATE` + `SET_ROTATION` `CausalOp`s; the worlds transport
|
||
(relay-first + BroadcastChannel fallback, hello/sync/op/hash,
|
||
boundary-hash convergence) is reused verbatim. Each tab creates and
|
||
animates its own arm rig; sinks converge on the pose hash ("poses agree
|
||
@ tick N"). Verified on prod through the real relay with two isolated
|
||
contexts — identical pose hashes, zero errors.
|
||
|
||
### Volumetric / 4D splats — `wai.splat.4d`, LIVE at /volumetric (2026-06-13)
|
||
|
||
- **Phase 5 complete — the roadmap frontier.** A `wai.splat.4d` is a
|
||
**temporal Gaussian splat**: the static splat extended to time. Each
|
||
Gaussian carries a keyframed track (position / scale / rotation /
|
||
color / opacity); the sink interpolates the cloud at frame N in the
|
||
`wai.det.fixed64` floor. Conformance is **frame-state-equivalence**:
|
||
the reconstructed Gaussian set at frame N hashes identically on every
|
||
sink, no tolerance — the temporal counterpart of the static splat.
|
||
- **`wai-rs` `splat4d` feature** (reuses the world `Fx` floor + blake3,
|
||
no codec/ML deps): the `W4DS` container (contract + per-Gaussian
|
||
keyframe tracks, 14 `Fx` per key) + a deterministic reconstruct (Fx
|
||
linear interpolation, hold outside the track). Trig-free (quaternions
|
||
interpolate component-wise; the sink normalizes for rendering).
|
||
`frame_hash` = the frame-state-equivalence quantity; `clip_hash` =
|
||
frame-independent identity. Registered + tiered **Lossy** (consistent
|
||
with `wai.splat.spz` — the splat representation approximates the
|
||
captured scene, even though the reconstruction is frame-state-exact).
|
||
- **`volumetric-conformance/` corpus** + `wai_splat4d_vectors gen|verify`:
|
||
orbit (two Gaussians swap places), pulse (scale + fade), morph (color
|
||
shift) — every frame's hash checked, all pass.
|
||
- **Browser sink — `Splat4dSink` (`wai-web`)** + `/volumetric` demo page,
|
||
LIVE: reconstructs the cloud in-tab and **verifies every frame's hash**
|
||
against the corpus; a frame scrubber re-derives the cloud, drawn as
|
||
painter-sorted alpha Gaussians (a sink with WebGPU renders the same set
|
||
with the full EWA pipeline — presentation). wasm pkg uniquely named
|
||
`wai_splat4d`.
|
||
- **Spec PROMOTED to `extensions/volumetric.md`** (site mirror),
|
||
registered in SPEC.md §5 (`wai.splat.4d`). **Honest about size**: the
|
||
win is content-addressing + dedup + a verifiable per-frame state, NOT
|
||
payload reduction (a real cloud is MB-scale) — stated plainly.
|
||
- **The cargo set is complete across the medium**: sight (worlds/replay),
|
||
hearing (audio/mixdown), touch (haptics/sample), live state
|
||
(feed/replay), a moving body (avatar/pose), and now a moving volume
|
||
(splat.4d/frame-state) — one `wai.det.fixed64` floor, one verification.
|
||
|
||
### Avatars / digital humans — `wai.avatar.pose`, LIVE at /avatar (2026-06-13)
|
||
|
||
- **Phase 4 complete — digital humans**: motion as instructions. An
|
||
avatar is a **skeletal world** — the worlds container/op-log/ordering/
|
||
sealing/state-hash/receipt-chain, with the `step` reduced to the
|
||
identity. Conformance is **pose-equivalence** (the per-bone local
|
||
transforms hash identically), a special case of replay-equivalence.
|
||
- **`wai.world.sim.skeletal0`** (in `wai-rs` `world` feature): a bone is
|
||
a `parent[16] + translation[3] + rotation-quaternion[4] + scale[3]`
|
||
(all `wai.det.fixed64`). Mutations: `CREATE` / `DELETE` /
|
||
`SET_ROTATION` (the animation driver) / `SET_TRANSLATION` / `SET_SCALE`,
|
||
all LWW. **Trig-free floor**: rotations are quaternions and the FK
|
||
composition is `+ − ×` only; axis-angle authoring happens off-line.
|
||
The forward-kinematics → world space and the skinning of a
|
||
`wai.mesh.glb` are presentation (a pure function of the hashed pose),
|
||
never in the hash. Registered + tiered `Lossless`; dispatched in
|
||
`run_sealed_dispatch`.
|
||
- **`avatar-conformance/` corpus** + `wai_avatar_vectors gen|verify`: a
|
||
sealed 3-bone arm wave, a sealed 5-bone humanoid pose, and three seeded
|
||
adversarial-arrival shuffles of the arm wave — all converge, all pass.
|
||
- **Browser sink — `SkeletalSink` (`wai-web`)** + `/avatar` demo page,
|
||
LIVE: replays a sealed animation in-tab, **re-derives the sealed pose
|
||
hash** (checked against the native corpus and the file's own seal), and
|
||
exposes FK world-space joints + bone segments to draw the skeleton on a
|
||
rotating orthographic canvas with a tick scrubber. wasm pkg uniquely
|
||
named `wai_avatar`.
|
||
- **Spec PROMOTED to `extensions/avatars.md`** (site mirror), registered
|
||
in SPEC.md §5 (`wai.avatar.pose`). A moving body joins sight (worlds),
|
||
sound (audio), touch (haptics), and live state (feeds) under one floor.
|
||
|
||
### State-feeds — `wai.feed.state`, LIVE at /feed (2026-06-13)
|
||
|
||
- **Phase 3 complete — live state-feeds**: state as instructions. A
|
||
feed is a **non-physics world** — the same `WWLD` container, op log,
|
||
canonical ordering, sealing, state hash, and receipt chain as
|
||
interactive-worlds, with the simulation `step` reduced to the
|
||
identity. Conformance is therefore **replay-equivalence**, reused
|
||
wholesale: no new criterion.
|
||
- **`wai.world.sim.feed0`** (in `wai-rs` `world` feature): nodes are
|
||
records of self-describing typed cells (`Int` i64, `Fixed` Q32.32,
|
||
`Text` UTF-8 — hashed without a schema). Mutations: `CREATE` /
|
||
`DELETE` / `SET_CELL` (LWW) and `ADD_INT` / `ADD_FIXED` (SUM —
|
||
commutative accumulation). Wrong-typed / out-of-range / absent-node
|
||
updates are deterministic no-ops. Registered + tiered `Lossless` in
|
||
the central `codecs` registry; dispatched in `run_sealed_dispatch`.
|
||
- **`feed-conformance/` corpus** + `wai_feed_vectors gen|verify`:
|
||
sealed scoreboard (SUM scoring + an LWW correction), telemetry
|
||
(LWW gauge + SUM packet count, incl. a negative `ADD_FIXED`), market
|
||
tape (LWW last + SUM volume), and three seeded adversarial-arrival
|
||
shuffles of the scoreboard — all converge, all pass (hash equality).
|
||
- **Browser sink — `FeedSink` (`wai-web`)** + `/feed` demo page, LIVE:
|
||
loads a sealed feed, replays it in-tab with the same engine that made
|
||
the corpus, and **re-derives the publisher's sealed BLAKE3 state hash**
|
||
(the file's seal also equals the native corpus hash — both checked).
|
||
A tick scrubber re-derives the feed at any tick; the record table
|
||
renders the typed cells generically. wasm pkg uniquely named
|
||
`wai_feed` (avoids the deploy fingerprinter's basename collision).
|
||
- **Spec PROMOTED to `extensions/state-feeds.md`** (site mirror),
|
||
registered in SPEC.md §5 (`wai.feed.state`). The **broadcast profile**
|
||
(one authoritative publisher, many read-only sinks) is documented as a
|
||
deployment shape of `wai.world.live`, not a new format. State joins
|
||
sight (worlds/replay), hearing (audio/mixdown), and touch
|
||
(haptics/sample) under one floor — verified the same way.
|
||
|
||
### Interactive worlds — `world` feature + conformance corpus (2026-06-11)
|
||
|
||
- **New cargo class implemented**: the interactive-worlds extension
|
||
(spec draft staged in `wai-transaction-science-web/research/`,
|
||
destined for `extensions/interactive-worlds.md`). Media that consumes
|
||
input: a world is a content-addressed artifact, every event is a
|
||
signed `CausalOp`, and conformance is **replay-equivalence** —
|
||
identical BLAKE3 state hash at every checkpoint, no tolerance
|
||
parameter. One container (`WWLD`) serves `wai.world.scene` /
|
||
`wai.world.replay` / `wai.world.live`, differing only in op-log seal
|
||
state; an input log and a multiplayer op log are the same object.
|
||
- **`wai-rs` `world` feature** (pure Rust, no ML dep chain; deps:
|
||
blake3 + ed25519-dalek): `Fx` Q32.32 fixed-point (`wai.det.fixed64`,
|
||
the floor profile — bit-exact unconditionally), xoshiro256++ contract
|
||
rng, signed/dep-gated ops with canonical `(effect_tick, lamport,
|
||
actor)` ordering, coordination-free BLAKE3 node minting, WWLD
|
||
section container with content-binding seal, `Sim` trait (the
|
||
extension standardizes the contract, not a physics engine) +
|
||
`kinematic0` reference registrant, sealed `Replayer` + `LiveWorld`
|
||
engines (buffering, past-tick re-derivation, equivocation rejection,
|
||
receipted-checkpoint history sealing).
|
||
- **`world-conformance/` corpus** + `wai_world_vectors gen|verify`:
|
||
sealed replays, LWW/SUM conflict batteries, dep-gap / past-tick /
|
||
shuffled-arrival live streams, expected hashes. All vectors pass;
|
||
clippy clean on the new code.
|
||
- **Spec PROMOTED to `extensions/interactive-worlds.md`** (site mirror
|
||
`src/pages/extensions/`), registered in SPEC.md §5: `wai.world.scene`
|
||
/ `wai.world.replay` / `wai.world.live` + asset media `wai.mesh.glb`
|
||
(canonical glTF 2.0 Binary, Khronos — decoder sink/library-supplied);
|
||
`media: "world"` noted in §3. Capability strings live in the central
|
||
`codecs` registry (`known_capabilities()`), tiered `Lossless` —
|
||
replay-equivalence is exact state reproduction, the simulation analog
|
||
of lossless.
|
||
- **Second sim registrant `wai.world.sim.ballistic0`** (gravity +
|
||
ground-plane bounce, per-node radius/restitution, `G = −10`, all
|
||
Q32.32 — bit-exact through every bounce) proving the registration
|
||
story: entered registry, dispatch, and corpus (`ballistic_bounce`
|
||
vector) without one line of the extension changing. New
|
||
`run_sealed_dispatch` routes on `contract.sim`; a missing sim
|
||
reports the world inert (`UnknownSimCapability`), mirroring SPEC.md
|
||
§4 step-6 semantics so callers can fall back per the envelope.
|
||
- **Native energy meter — measured joules, no sudo (2026-06-13)**:
|
||
corrected a wrong assumption that real energy needed RAPL/sudo
|
||
(unavailable on the arm64 prod box). On Apple Silicon the SoC energy
|
||
counters are readable through IOReport with **no privileges** (the
|
||
source `macmon`/`asitop` use; verified euid≠0 reads CPU/GPU/ANE power
|
||
on the M3 Ultra), and `proc_pid_rusage` gives exact per-process
|
||
retired cycles/instructions. New `wai_world_meter` bin (`world`
|
||
feature): brackets a tight replay burst of a sealed `.wwld` against an
|
||
idle baseline, samples CPU power via macmon, and reports the marginal
|
||
**measured** CPU energy — e.g. the truss replay metered at ~512 µJ /
|
||
replay (~4.3 µJ/tick) at 3.2 W marginal, a real number that fills a
|
||
receipt's `joules_micro` where a native sink runs. The burst runs on
|
||
N threads so the marginal draw clears a shared box's idle fluctuation
|
||
(single-thread 512 µJ vs 10-thread 524 µJ/replay cross-validates the
|
||
method — per-replay energy is invariant to parallelism). With a second
|
||
arg the meter emits a full §8 `SessionReceipt` carrying the MEASURED
|
||
joules + exact `work_steps`, signed and self-verified — real measured
|
||
energy end to end. Spec §8 + the `/worlds` page corrected: the
|
||
sink-metering path needs no sudo (IOReport on Apple Silicon, RAPL on
|
||
x86); the browser estimate is the fallback, not the ceiling. clippy
|
||
clean, 94 tests green.
|
||
- **Phase 2 begun — haptics (`wai.haptic.signal`) (2026-06-13)**: touch
|
||
as instructions. Spec drafted (`research/haptics-draft.md`), capability
|
||
registered + tiered. `wai-rs` `haptic` feature (reuses the world `Fx`
|
||
floor, no codec/ML deps): the `WHAP` container + a deterministic
|
||
reconstruct — per channel, a keyframed **intensity/sharpness** envelope
|
||
(the Core-Haptics pair every actuator API maps to) linear-interpolated
|
||
in `Fx` and quantized to `u8`. Trig-free by construction (amplitude +
|
||
sharpness, not a modulated carrier), so the floor reconstructs it
|
||
exactly with `+,−,×,÷` alone. `signal_hash` is the **sample-equivalence**
|
||
quantity. `wai_haptic_vectors gen|verify` + `haptic-conformance/` (ramp,
|
||
thud, dual-rumble pulse) — all pass, hash equality. 104 tests, clippy
|
||
clean. Next: Gamepad/Vibration browser sink + `/haptics` demo, then
|
||
promote. This completes the sight → hearing → touch trio's engines.
|
||
- **Spatial-audio: receipt profile + spec PROMOTED (2026-06-13)** —
|
||
Phase 1 complete. `audio_scene::SceneReceipt` (the worlds receipt
|
||
transposed to audio): identity `scene_hash`, the JWP group's objects
|
||
are the mix's **PCM blocks** (`BLAKE3("wai:audio-block")`), Merkle-bound
|
||
with JWP's exact domains, one Ed25519 sig over scene_hash + root +
|
||
blocks + `work_mac` (samples × objects, exact signed) + carried
|
||
`joules_micro`; `parent_receipt_hash` chains renders. Reuses
|
||
`world::receipt::merkle_root` — minimal new code. 2 receipt tests (101
|
||
total). Draft promoted to `extensions/spatial-audio.md` (+ site mirror,
|
||
SPEC §5 row + extension pointer) — and the canonical spec corrected to
|
||
the **as-built** `sqrt_constpower` pan (sqrt-only, no transcendental
|
||
tables) and the real receipt shape, since registration froze the
|
||
format. clippy clean; spec page live. Phase 1 (spatial audio) DONE:
|
||
spec · register · mixer · corpus · live /audio sink · receipt ·
|
||
promoted. Next: Phase 2 — haptics.
|
||
- **Spatial-audio browser sink — LIVE at /audio (2026-06-13)**: `wai-web`
|
||
`AudioSink` (wasm-bindgen): parse `WSCN`, load the corpus sources, run
|
||
the exact deterministic mixdown, expose `mixdown_hash_hex` +
|
||
`scene_hash_hex` + `mix` (i16 PCM) + `object_xz(t)`. The `/audio` page
|
||
(site-styled, in nav) loads a scene, **verifies the mixdown hash
|
||
in-browser against the corpus** (mixdown-equivalence, the same proof
|
||
worlds does for state), plays the bit-exact stereo mix via an
|
||
AudioBuffer, and shows a top-down radar of the objects panning. Proven
|
||
on prod: all three scenes verify, zero errors — wasm-in-Chrome mixes
|
||
byte-identically to native. Honest split kept: plays the exact stereo
|
||
floor; binaural/HRTF noted as the richer optional presentation.
|
||
- **Spatial-audio conformance corpus (2026-06-13)**: `wai_audio_vectors
|
||
gen|verify` + `audio-conformance/` — the mixdown-equivalence analog of
|
||
the worlds corpus. Three vectors (static stereo pair, distance
|
||
near/far, keyframed left→right sweep) ship a `.wscn` scene + a
|
||
`.sources.bin` (decoded mono tones) + expected `scene_hash` and
|
||
`mixdown_hash`; `verify` mixes each and checks hash equality, no
|
||
tolerance. All pass; clippy clean.
|
||
- **Spatial-audio reference mixer — mixdown-equivalence (2026-06-13)**:
|
||
`wai-rs` `audio_scene` feature (reuses the world `Fx` floor + blake3,
|
||
no codec/ML deps): the `WSCN` container (scene contract + object table
|
||
+ asset table + optional seal, mirrors WWLD) and a fully deterministic
|
||
mixdown — `Fx` position interpolation, `inverse_clamped` distance
|
||
attenuation, `sqrt_constpower` pan (sqrt only, no transcendental
|
||
tables), `Fx` accumulate, round-half-away i16 quantization in integer
|
||
space. `mixdown_hash` (BLAKE3 over canonical i16LE PCM for a declared
|
||
layout) is the **mixdown-equivalence** quantity; `scene_hash` is the
|
||
identity. 5 tests prove it: centered = equal L/R, hard-left = silent
|
||
right, deterministic + layout-bound hash, distance attenuates, a
|
||
moving source sweeps L→R + container round-trips. Sources arrive as
|
||
decoded mono PCM keyed by content hash — decoding the `wai.audio.*`
|
||
assets is the sink's job. clippy clean. Next: conformance corpus
|
||
bin + Web Audio browser sink + demo.
|
||
- **Content taxonomy + cargo roadmap; spatial-audio Phase 1 begun (2026-06-13)**:
|
||
mapped the full space of web-transported content WAI's approach covers
|
||
(`research/content-taxonomy.md`) — a capture→synthesizable→executable→
|
||
symbolic spectrum × four levers (codec dispatch, neural prior,
|
||
instructions-at-sink, determinism+receipts), with a per-row
|
||
mechanism/extension/status table and the white space called out. A
|
||
phased build plan (`research/cargo-roadmap.md`) takes each white-space
|
||
class through the proven interactive-worlds recipe. **Phase 1 kicked
|
||
off — object-based spatial audio** (`wai.audio.scene`): the audio
|
||
analog of interactive-worlds (ship sound objects + a scene, render to
|
||
the sink's layout; **mixdown-equivalence** = replay-equivalence with
|
||
the tick replaced by the audio sample). Spec drafted
|
||
(`research/spatial-audio-draft.md`, `WSCN` container, `Fx` mixdown,
|
||
content-addressed `wai.audio.*` sources, stereo bit-exact floor +
|
||
binaural perceptual tier, WAI2 recorded-mix fallback, JWP receipt
|
||
profile); `wai.audio.scene` registered + tiered Lossless. 94 tests
|
||
green. Remaining phases: haptics, live state-feeds, digital humans,
|
||
volumetric/4D splats.
|
||
- **Real SPZ decode — `wai.splat.spz` end to end (2026-06-13)**: the
|
||
splat gallery now renders a genuine Niantic-SPZ file, not a
|
||
mesh-derived stand-in. New `wai-spz-gen` crate depends on the field's
|
||
`spz` reference library (the sink-supplied codec WAI dispatches to),
|
||
encodes a 2,400-Gaussian cube cloud to a real `.spz` (validated by
|
||
re-loading through the library), and wraps it as a content-addressed
|
||
`wai.splat.spz` WAI object (`nebula.spz.wai`). `gen-gallery` adds it
|
||
to the scene's asset table (role 1) beside the glb. Browser:
|
||
`WorldSink.asset_payload(name, env, cap)` generalizes the §4.1 verify
|
||
to any capability; the page gunzips (DecompressionStream) and decodes
|
||
the SPZ bitstream in-tab — 16-byte header, non-interleaved arrays,
|
||
24-bit fixed-point positions, log scales, smallest-three quaternions,
|
||
byte/255 opacity+color — into per-Gaussian world covariance for the
|
||
EWA pipeline. Decoded count matches the encoded 2,400 (validates the
|
||
layout); the cube silhouette + surface-aligned discs validate the
|
||
quaternion math. Page copy corrected: the splat view decodes a real
|
||
SPZ asset (no longer "derived from mesh"). Verified locally (2,400
|
||
Gaussians, zero errors); prod serves the identical artifacts.
|
||
- **Anisotropic EWA splatting (2026-06-13)**: upgraded the splat
|
||
renderer from isotropic billboards to real 3D Gaussian splatting. Each
|
||
splat carries an anisotropic 3×3 covariance (surface-aligned flat
|
||
discs: wide in the triangle tangent plane, thin along the normal); the
|
||
vertex shader projects it to a screen-space conic
|
||
(Σ' = J·W·Σ·Wᵀ·Jᵀ with the perspective Jacobian), emits a tight
|
||
3σ bounding quad, and the fragment evaluates the conic — the textbook
|
||
3DGS pipeline, still CPU-sorted back-to-front and alpha-blended.
|
||
Cam uniform carries view + proj separately (+ viewport/focal) for the
|
||
Jacobian. Visibly sharper, surface-coherent splats; proven on prod,
|
||
zero errors.
|
||
- **3D render layer — PBR mesh + Gaussian splats (2026-06-13)**: the
|
||
`/worlds` gallery now renders the content-addressed, hash-verified
|
||
`wai.mesh.glb` two ways in a real 3D path (perspective camera + depth
|
||
buffer, all in `wai-web/demo/world.js`, no wasm change): a **solid
|
||
PBR-shaded cube** (flat normals from screen-space derivatives, Lambert
|
||
+ spec + sky-ambient, gamma) and a **Gaussian-splat cloud** —
|
||
instanced camera-facing quads with a Gaussian falloff, back-to-front
|
||
depth-sorted each frame (painter's order) and alpha-blended. The splat
|
||
cloud is *derived from the same verified mesh*, making the thesis
|
||
visible: one content-addressed asset, two sink-side renderings,
|
||
presentation never touches the state hash. SPZ binary decode stays
|
||
sink-supplied (stated honestly on the page); the renderer is the real
|
||
deliverable. Two gallery vectors (`gallery` / `gallery_splat`); the 2D
|
||
sim path is untouched. Proven on prod: both render, 768 splats from
|
||
the cube, zero errors.
|
||
- **Receipt chaining + work-meter + `wai.splat.spz` (2026-06-12)**:
|
||
three follow-ons. (1) `SessionReceipt` gains `parent_receipt_hash`
|
||
(BLAKE3 of the prior receipt's canonical bytes — the JWP linkage) +
|
||
`receipt_hash()` + `verify_chain`: successive sealed sessions of one
|
||
world (shared `world_hash`, `group_id`+1) form an append-only signed
|
||
timeline. (2) An **honest energy meter**: receipts carry exact,
|
||
signed, deterministic `work_steps` (Σ node-steps, from
|
||
`collect_checkpoints_and_work`) AND a separately *carried*
|
||
`joules_micro` — never derived in the receipt; the browser supplies a
|
||
wall-clock estimate at a **declared** coefficient (labeled "est@4W"),
|
||
native sinks supply real RAPL, an unmetered sink signs `0`. (3)
|
||
Registered **`wai.splat.spz`** — Gaussian-splat scenes as an asset
|
||
capability parallel to `wai.mesh.glb` (presentation only, never in the
|
||
state hash; the win is content-addressing + dedup, not payload size;
|
||
renderer sink-supplied). Browser `seal_session(seed, id, joules,
|
||
group, parent)` chains via localStorage; the `/worlds` HUD shows
|
||
`chain #N ↳linked · N steps · ≈X mJ est@4W · ✓ re-verified`. Proven on
|
||
prod: two seals chain (g0 parent null → g1 parent = g0.receipt_hash),
|
||
exact work 248 steps, est joules carried, zero errors. Spec §8 updated
|
||
(chaining + the two-quantity energy model). 94 native tests, 10/10
|
||
vectors, clippy clean.
|
||
- **Asset path SHIPPED — `wai.mesh.glb`, content-addressed (2026-06-12,
|
||
extension §4.1)**: worlds are made of WAI objects. New
|
||
`world::asset_content_hash` (BLAKE3 `wai:asset\x01` over a complete
|
||
WAI envelope) + `DOMAIN_ASSET`; native tests PROVE the load-bearing
|
||
claim — a scene with an asset table has the byte-identical state hash
|
||
AND world hash as the same scene without it (assets are
|
||
presentation-only, never in the state hash), and the asset hash binds
|
||
the exact envelope. Generator `gen-gallery` writes a `gallery.wwld`
|
||
scene + `assets/cube.glb.wai`: a real glTF 2.0 Binary cube
|
||
(hand-written GLB writer) wrapped as a `wai.mesh.glb` WAI envelope and
|
||
content-addressed in the table. Browser: `WorldSink.assets()` +
|
||
`asset_glb(name, envelope)` (verifies content hash in WASM, rejects a
|
||
mismatch, unwraps the WAI envelope to the `.glb` payload); the page
|
||
decodes the GLB (minimal JS decoder) and renders the verified mesh as
|
||
a spinning wireframe. Proven on prod: `1/1 verified ✓ · cube.glb`,
|
||
and the state hash stays fixed while the mesh spins — §4.1 made
|
||
visible. 92 native tests, 10/10 vectors, clippy clean.
|
||
- **Session receipts + seal-to-replay SHIPPED** (2026-06-12, extension
|
||
§8 implemented): `world/receipt.rs` builds a `SessionReceipt` — a JWP
|
||
*profile*, not a fork: op batches (one per checkpoint interval,
|
||
content-hashed `wai:world-ops\x01`) are Merkle-bound with JWP's exact
|
||
leaf/node domain separators, and one Ed25519 signature covers the
|
||
world-profile payload (world hash + checkpoint `state_hash` list +
|
||
carried microjoules). Joules are **carried, never fabricated** — the
|
||
builder takes the figure; an unmetered session signs `0` and says so.
|
||
New `world_hash` (BLAKE3 over scene = contract + tick-0 graph, the
|
||
§7.4 shareable identity), `collect_checkpoints` (unsealed replay →
|
||
checkpoint list). Browser: `WorldSink.seal_session(seed, signer_id,
|
||
joules)` → `SealedSession{replay_bytes, receipt_json, root_hash, …}`;
|
||
the `/worlds` page's "⏷ seal & download replay" turns a live, shared,
|
||
edited world into a portable `.wwld` + `.receipt.json`, re-opens the
|
||
fresh replay in a throwaway sink to prove it re-verifies, and shows
|
||
the Merkle root + batch count in the HUD. Proven on prod: fracture a
|
||
tower across the relay → seal → 3-batch receipt, valid sig, replay
|
||
re-verified. 90 native tests, 10/10 vectors, clippy clean.
|
||
- **Cross-device multiplayer SHIPPED** (2026-06-12, `wai-relay/` +
|
||
`wai-web` Actor binding): every tab/device is a signed actor. New
|
||
`Actor` wasm-bindgen class (browser-held Ed25519 key; clicks author
|
||
real `CausalOp`s; Lamport discipline via `observe_lamport`);
|
||
`wai-world-relay` reference transport (axum WS, per-room fan-out,
|
||
retains the signed op log for late joiners, **verifies every op's
|
||
Ed25519 signature at the door** with the same `wai` crate — it
|
||
adjudicates nothing; sinks converge because the math does). Page
|
||
transport is relay-first (room in the URL hash) with BroadcastChannel
|
||
fallback. Deployed at `wai.transaction.science/worlds-ws/*`
|
||
(launchd `com.wai.world-relay`, localhost:9670). Proven on prod with
|
||
two ISOLATED browser contexts: point-blank axial shoves snap two
|
||
members (23→21 nodes) and both contexts converge — "✓ hashes agree".
|
||
Also fixed `build.rs` emitting libFLAC link flags regardless of
|
||
features (pkg-config probe side effect): now gated on `codecs`.
|
||
- **Browser sink SHIPPED** (2026-06-12, `wai-web/src/world.rs` +
|
||
`wai-web/demo/world.{html,js}`): a `wai.world.replay` sink in a tab.
|
||
NOT a port — the `wai` crate itself now compiles to wasm32 via the
|
||
new default-on **`codecs` feature** (C codec chain gated; envelope +
|
||
`world` engine build with `--no-default-features --features world`),
|
||
and wai-web binds it (`WorldSink`: advance/particles/members/
|
||
state-hash/verify-seal; sim state stays exact Q32.32, f32 only for
|
||
display). WebGPU renderer (instanced SDF circles + member lines).
|
||
Proven in real Chrome via Playwright: the 1,282-byte truss_fracture
|
||
payload replays 120 ticks, the member snaps, and the browser's
|
||
BLAKE3 state hash equals the native corpus hash byte-for-byte —
|
||
**replay-equivalence across arm64-native and wasm32-in-Chrome,
|
||
demonstrated**, with the Ed25519 seal verified in-page.
|
||
- **Solver-backed registrant `wai.world.sim.truss0`** (2026-06-12,
|
||
`world/truss.rs`): structural simulation — particles + distance
|
||
members solved per tick by position-based dynamics (fixed 8-pass
|
||
Gauss–Seidel projection in canonical member order), entirely in
|
||
`wai.det.fixed64` including a new floor-rounded `Fx::sqrt` (128-bit
|
||
integer Newton). Members carry a yield strain measured on predicted
|
||
positions before the solve; an overloaded member **fractures** as an
|
||
emergent solver consequence — two actors pulling the same member in
|
||
opposite directions (the SUM rule feeding the solver) snap it
|
||
bit-identically on every machine (`truss_fracture` corpus vector).
|
||
Registration constants: `G = −10`, `ITERATIONS = 8`, ground at
|
||
`y = 0`, velocity damping `63/64`/tick (static equilibrium).
|
||
`LiveWorld` gained `state()`/`current_tick()` accessors (what a
|
||
renderer reads). Where the float-based CAD solvers can't transplant
|
||
(floats aren't cross-platform bit-exact), solver *form* does:
|
||
iterative constraint projection in the deterministic profile.
|
||
|
||
### Inter-frame video, categorical/VQ images, and real-model validation
|
||
|
||
- **`wai.neural.dcvc_rt` / `wai.neural.dcvc_fm` — inter-frame video decoder
|
||
implemented** (`neural/video_dcvc.rs`). "Requires CUDA" was a *speed* claim:
|
||
inter-frame decode is a frame loop that propagates a reference feature, and
|
||
ONNX Runtime runs it on CPU. The per-frame entropy decode reuses the
|
||
CompressAI-validated hyperprior path; the temporal net is sink-supplied.
|
||
Validated end-to-end (3 inter-frames, feature propagation confirmed).
|
||
- **`wai.neural.glc` — categorical / VQ decoder implemented** (`neural/glc.rs`).
|
||
Not the Gaussian-conditional family: codebook indices decoded via a
|
||
per-position softmax → `pmf_to_quantized_cdf` (bit-exact with CompressAI's
|
||
C++, unit-tested) → a bypass-free categorical rANS over a checkerboard → VQ
|
||
synthesis. Surfaced that categorical entropy is the most determinism-sensitive
|
||
path — a per-position CDF flips on ~1e-5 logit drift where Gaussian scale
|
||
buckets don't.
|
||
- **Real-model validation** beyond synthetic stand-ins. SNAC's genuine
|
||
`snac_24khz` now exports (`tools/wai_snac_export.py`: zero-noise so the decode
|
||
is deterministic at all + a reshape-free snake) and `wai-rs` decodes it to
|
||
~9e-7 of the torch reference. A real PTQ-trained, activation-calibrated
|
||
`int_synth` model decodes **bit-identically** (max|Δ|=0) to its integer-forward
|
||
reference (`tools/wai_int_synth_train.py`; also fixed a zero-gradient
|
||
`torch.floor` bug in the QAT trainer).
|
||
|
||
### Learned-image codec family — decoders implemented and validated
|
||
|
||
- The reference crate now decodes the full conditional-entropy family
|
||
natively, not just the factorized baseline: **`wai.neural.mbt2018_mean`**
|
||
(mean-scale hyperprior, Gaussian-conditional), **`wai.neural.elic`**
|
||
(checkerboard spatial context), **`wai.neural.mlicpp`**
|
||
(channel-autoregressive multi-reference), and the integer-exact
|
||
**`wai.neural.int_synth`** (all-integer transposed-conv synthesis — a
|
||
decode that is bit-identical on every machine *by construction*, the
|
||
first capability that can soundly back `intent=replicate` cross-hardware).
|
||
The rANS + entropy decode live in `wai-rs`; the analysis/synthesis nets
|
||
are sink-supplied ONNX.
|
||
- **Current-SOTA entries added as live routes:** `wai.neural.mlicv2`
|
||
(MLICv2/v2+, the current SOTA learned image) and `wai.neural.cmic`
|
||
(Content-Aware Mamba) both decode through the multi-reference core —
|
||
their entropy model is the same channel-slice + checkerboard structure;
|
||
the gains live in their sink-supplied transforms.
|
||
- **Validation against the reference codec.** `bmshj2018` and
|
||
`mbt2018_mean` are checked byte-for-byte against pretrained CompressAI
|
||
models (reconstruction matches to within final-layer float rounding);
|
||
`elic`/`mlicpp` are checked end-to-end against minimal CompressAI-
|
||
compatible models; `int_synth` against a constructed integer model.
|
||
Surfacing this also caught (and the spec now documents) a real
|
||
determinism hazard: a learned codec's entropy decode desyncs unless the
|
||
encoder and decoder compute the scale buckets with the *same* model —
|
||
float drift between two implementations of `h_s` flips bucket
|
||
assignments and corrupts the stream.
|
||
|
||
### Audio: SNAC hierarchical codec
|
||
|
||
- **`wai.neural.snac`** — SNAC's multi-resolution RVQ, distinct from the
|
||
flat single-grid RVQ runtime: N variable-length code streams at
|
||
hierarchical rates, decoded via dynamic-arity ONNX inputs. Ubiquitous in
|
||
low-bitrate TTS.
|
||
|
||
### Removed: off-charter Python runtime (`wai-neural/`)
|
||
|
||
- Moved `wai-neural/` OUT of this repo to `../../wai-neural-prototype/`.
|
||
It was a parallel **Python** runtime (`WAI4` generative-semantic
|
||
container + CLIP-embedding payloads + SD-XL-Turbo regeneration +
|
||
CLIP-similarity conformance) — a disconnected second implementation in
|
||
the wrong language. The reference implementation is **Rust** (`wai-rs`);
|
||
nothing referenced the Python package. The *idea* it proved — a
|
||
behavioral/perceptual neural path that conforms by semantic similarity
|
||
rather than a weights hash — is sound and belongs in `wai-rs` as a future
|
||
Rust path alongside the bit-exact, prior-pinned path. The Python is kept
|
||
only as a prototype reference (and it hardcodes a machine-local model
|
||
path, so it never ran portably). This does NOT touch the Python
|
||
model-export tools in `tools/`, which are legitimate dev tooling.
|
||
|
||
### Neural condition: working end-to-end
|
||
|
||
- `tools/wai_encodec_encode.py` / `wai_encodec_decode.py` — first
|
||
working **neural capability** (`wai.neural.encodec32`): source
|
||
encodes audio via locally-installed EnCodec, ships only the
|
||
quantized tokens in a standard WAI envelope (236 MB of model
|
||
weights stay on the source's disk, never enter the envelope); sink
|
||
with EnCodec installed decodes. Measured: **5 s of audio → 2.3 KB
|
||
envelope (141× compressed)**, model on the wire = 0 B.
|
||
- Rust SDK gains `wai inspect` CLI: parses an envelope, resolves the
|
||
capability + fallback against `sink_capabilities()`, and prints
|
||
verdict (`DECODABLE via primary` / `via fallback` / `INERT here`).
|
||
|
||
### Browser runtime (`wai-web/`)
|
||
|
||
- New crate: wasm-bindgen wrapper exposing `parse(bytes) → WaiEnvelope`,
|
||
`pack(manifest_json, payload)`, and `known_capabilities()`. 107 KB
|
||
wasm + 16 KB JS glue.
|
||
- `wai-web/demo/wai.js` — thin JS shim with `WaiRuntime.register(cap,
|
||
handler)` dispatch. Backends are **pluggable** (browser-native APIs,
|
||
transformers.js, WebGPU compute, WebNN, ONNX runtime web) — WAI
|
||
doesn't bundle any. The sink-side compute fabric is heterogeneous;
|
||
the deployer picks per capability.
|
||
- `wai-web/demo/index.html` — drop-and-decode demo with handlers wired
|
||
to: `createImageBitmap` for `wai.image.{png,jpeg,avif,jxl}`,
|
||
`AudioContext` for `wai.audio.{opus,flac}`, placeholders for
|
||
`wai.neural.*` showing where transformers.js / WebNN / WebGPU plug
|
||
in. Run with `python3 wai-web/demo/serve.py`.
|
||
- 4 sample `.wai` files in `wai-web/demo/samples/` covering PNG, JXL,
|
||
Opus, and EnCodec for testing.
|
||
|
||
### Spec: v1.1 multi-rendition envelope (SPEC.md §6.1)
|
||
|
||
- New magic `WAI2` for envelopes that carry multiple renditions of the
|
||
same content (e.g. EnCodec + Opus, or AVIF + JPEG-XL). Reframed from
|
||
"graceful fallback for missing capability" to **policy selection** —
|
||
in a self-contained ecosystem every sink has every required
|
||
capability; renditions exist so the sink picks by compute budget,
|
||
bandwidth, or runtime preference (browser-native vs WebGPU vs WebNN
|
||
vs native ML). v1.0 (`WAI1`) remains valid and unchanged.
|
||
|
||
## v0.3.0
|
||
|
||
**Pivot**: WAI is the **envelope + capability-dispatch + neural-shared-
|
||
prior** standard. The zeroth condition is a *menu of registered SOTA
|
||
codec capabilities* (AVIF, JPEG-XL, PNG, JPEG, Opus, FLAC, AV1, zstd,
|
||
XZ), not a custom codec. The Python prototype and all hand-rolled
|
||
codec implementations from v0.2.x were removed; Rust is the single
|
||
reference implementation and the SDK.
|
||
|
||
### Standard
|
||
|
||
- `SPEC.md` rewritten as a capability-menu spec. Mandatory floor:
|
||
`wai.image.png`, `wai.audio.flac`, `wai.text.zstd` (every conforming
|
||
sink MUST decode these). Recommended modern set: `wai.image.jxl`,
|
||
`wai.image.avif`, `wai.image.jpeg`, `wai.audio.opus`, `wai.video.av1`,
|
||
`wai.video.av1.lossless`, `wai.text.xz`. Neural capabilities
|
||
(`wai.neural.*`) remain OPTIONAL and sink-advertised.
|
||
- Container envelope (`WAI1` magic + length-prefixed manifest + length-
|
||
prefixed payload) and manifest schema are unchanged from v0.2.x.
|
||
- Conformance: bit-exact decode-equivalence with each capability's
|
||
reference library (libjxl / libavif / libopus / libFLAC / rav1e / etc.).
|
||
|
||
### Rust reference (`wai-rs/`, crate `wai` 0.3.0)
|
||
|
||
- `cargo build --release` produces `libwai.dylib`/`.so`/`.dll` (cdylib),
|
||
`libwai.a` (staticlib), and `libwai.rlib` (Rust-native). The crate
|
||
type list is `["lib", "cdylib", "staticlib"]` — drop-in for any
|
||
language's C-FFI binding.
|
||
- Codec wrappers: `image` (PNG, JPEG via `jpeg-encoder`, AVIF via
|
||
`ravif` + `libavif-image` for decode, JPEG-XL via `jpegxl-rs`), `opus`
|
||
(RFC 7845 Ogg-Opus container so output drops into ffmpeg/VLC/browsers),
|
||
`claxon`+`flac-bound` for FLAC, `rav1e`+`dav1d` for AV1 (lossy +
|
||
lossless), `zstd`+`xz2` for text.
|
||
- Generated C header: `include/wai.h` is regenerated by `build.rs` via
|
||
`cbindgen` on every build. SDK consumers get a typed header rather
|
||
than hand-writing `extern "C"` decls.
|
||
- Cross-platform linkage: `build.rs` probes `pkg-config` for libflac;
|
||
the macOS-hardcoded `.cargo/config.toml` from v0.2.x is gone. Builds
|
||
clean on macOS (Homebrew) and Linux (apt) without `RUSTFLAGS`.
|
||
|
||
### C FFI / SDK surface
|
||
|
||
All `wai_*` functions return a `wai_status_t`:
|
||
|
||
| code | meaning |
|
||
|---|---|
|
||
| `WAI_OK` (0) | success |
|
||
| `WAI_ERR_INVALID_ARG` (1) | null pointer / mismatched length / out-of-range dimension |
|
||
| `WAI_ERR_CODEC` (2) | codec-level failure (details in `wai_last_error()`) |
|
||
| `WAI_ERR_CONTAINER` (3) | bad magic / truncated envelope / unparseable manifest JSON |
|
||
| `WAI_ERR_PANIC` (4) | internal Rust panic caught at the boundary (a wrapper bug — never UB into the C caller) |
|
||
|
||
Every FFI function is wrapped in `std::panic::catch_unwind` so Rust
|
||
panics never unwind into C (UB). `av1_decode` validates header bounds
|
||
to prevent libdav1d's C-level `abort()`. `wai_last_error()` is per-
|
||
thread, populated on any non-OK return.
|
||
|
||
### Testing
|
||
|
||
- **13 lib tests**: per-codec round-trip + envelope round-trip
|
||
- **21 FFI robustness tests**: null pointers, length mismatches,
|
||
dimension overflow, malformed magic, truncated payloads, bad JSON,
|
||
invalid Opus sample rates, zero fps, garbage decoder input
|
||
- **10 fuzz smoke targets**: 1024 adversarial random byte sequences per
|
||
decoder asserting no panic. Stable Rust, no `cargo-fuzz` toolchain
|
||
needed. Found and fixed a real envelope-truncation panic.
|
||
- **`wai_bench` corpus walker**: quick mode (4 images + 4 audio + 3
|
||
text + 2 video) for spot-checks; `--full` walks the entire corpus
|
||
(169 images + 5 audio + 13 text + 12 video sequences = 588 codec
|
||
runs in ~9.4 min). Per-codec aggregate stats: runs, ok / fail counts,
|
||
WAI vs reference byte sizes, PSNR delta vs ffmpeg, encode time.
|
||
- **`--report file.json`** writes a structured bench report;
|
||
`wai_bench_diff baseline.json head.json` flags regressions on bytes
|
||
(±5% default tolerance), encode time (±25%), or new failures. Non-zero
|
||
exit on regression — ready for CI gating.
|
||
- **GitHub Actions CI** (`.github/workflows/ci.yml`): ubuntu + macOS
|
||
matrix, apt/brew codec deps, `cargo test --release`, cdylib/staticlib
|
||
artifact verification, quick bench, JSON report uploaded as artifact.
|
||
|
||
### Honest codec measurements (vs ffmpeg reference, full Kodak/Tecnick/
|
||
CLIC/synthetic corpus = 169 images, plus SQAM audio + Silesia text + Derf CIF video)
|
||
|
||
| codec | runs | size vs ref | PSNR Δ vs ref | notes |
|
||
|---|---:|---:|---:|---|
|
||
| PNG | 169 | — | lossless | ffmpeg-interop ok |
|
||
| JPEG q=50 | 169 | +12.7% | +0.27 dB | q-knob calibrated to libjpeg |
|
||
| JPEG q=80 | 169 | +19.7% | −0.31 dB | residual gap is pure-Rust vs libjpeg-turbo asm |
|
||
| AVIF q=70 | 169 | — | — | ffmpeg-interop ok |
|
||
| JPEG-XL lossless | 169 | — | lossless | bit-exact via libjxl |
|
||
| FLAC | 5 | **−10.1%** | lossless | WAI compression_level 8 vs ffmpeg default 5 |
|
||
| Opus 64k | 5 | — | — | standard Ogg-Opus container, ffmpeg-interop ok |
|
||
| zstd-19 | 13 | ±0.0% | — | **byte-identical to system zstd** |
|
||
| xz-6 | 13 | ±0.0% | — | **byte-identical to system xz** |
|
||
| AV1 lossy | 12 | +28.3% | — | rav1e vs ffmpeg libsvtav1 (different encoders) |
|
||
|
||
All 588 corpus runs and all 44 tests pass.
|
||
|
||
### Removed
|
||
|
||
- The Python prototype (`wai/`) and Python BD-rate harness (`bench/`).
|
||
v0.2.x's hand-rolled wire format is dead; Rust is the single source
|
||
of truth.
|
||
- The `webm_lottie_new/wai-rs/` pre-pivot Rust crate (Wasm-decoder-in-
|
||
bundle model). Replaced by the clean `wai-rs/` SOTA-wrapper crate.
|
||
|
||
---
|
||
|
||
## v0.2.0 — superseded (historical record)
|
||
|
||
The v0.2.x direction was hand-rolling custom codec wire formats (DCT +
|
||
quantization + rANS for image, MDCT for audio, I/P inter-frame for
|
||
video). Two byte-interoperable reference implementations (Python +
|
||
Rust) were built and cross-validated.
|
||
|
||
**Why superseded**: re-implementing JPEG/MPEG/Opus poorly is strictly
|
||
worse than wrapping the field's SOTA libraries. WAI's actual value is
|
||
the envelope + capability dispatch + the neural-shared-prior model,
|
||
which v0.3 keeps. Custom codecs are gone; standard libraries are in.
|
||
|
||
### What v0.2 had
|
||
|
||
- `SPEC.md` defined byte-precise wire formats for image (8×8 DCT-II +
|
||
YCbCr + JPEG-Q + zigzag + DC-Δ + rANS), audio (orthonormal MDCT +
|
||
sine window + TDAC), video (I/P inter-frame with integer-pel motion).
|
||
- Python (`wai/`) + Rust (`wai-rs/`) implementations, byte-equivalent on
|
||
the lossless paths (rANS, container) and reconstruction-equivalent on
|
||
the lossy paths.
|
||
- Honest measured standing: image at JPEG-1992 parity (~5% behind
|
||
libjpeg), audio uniform-MDCT (not perceptually competitive with MP3),
|
||
video I/P at ~41% smaller than MJPEG floor but far behind H.264.
|
||
|
||
### Licensing (unchanged)
|
||
|
||
- Apache-2.0 for the standard and the reference implementation.
|
||
Royalty-free implementation is a precondition for adoption (VP9/AV1
|
||
open-codec precedent; the H.264/H.265 royalty trap is avoided).
|