# WAI — changelog Notable changes to the WAI standard and the reference implementation. Newest at the top. WAI is a draft standard; until v1.0 freezes, the wire format may change and such changes are recorded here. ## v0.4.0 — current ### JPEG AI — a live, self-sufficient `wai.image.jpegai` decode (2026-06-22) - `neural::jpegai::decode(codestream, &JpegAiModel)` decodes a real ISO/IEC 6048 codestream to RGB with **no per-image schedule fixtures**: every per-image quantity is derived from the bitstream, and only checkpoint constants come from a sink-supplied model bundle. Wired into `decode_envelope` — JPEG AI is now a dispatchable capability, not just reference/conformance support. - The two derivations that closed it, traced bit-exact from the reference: the residual coding mask is `sigma_skip > 382` (the σ surface *before* the rvs addition — on the bucketed index it looks ambiguous, on `sigma_skip` it is exact, 0 violations / 65536 positions); the dequant is `resi × rvs_dq[grfs[ch]][lk] / 2^16 / scaler_vec[ch]`, with `grfs` determined the same way the σ rvs does so the per-channel `scaler_vec` is consistent. Matches the reference decoder's PNG to within rounding (max diff 1). ### Film reel — `wai.film.reel`, cuts over one world (2026-06-17) - The editor's layer over `wai.film.linear`: a `WREL` embeds one sealed world + a sequence of **shots** (each its own camera track + frame schedule) played back to back. Cutting wide→close, or one beat from two angles, is film grammar; a reel carries it as a few kilobytes of camera keys, the world shared across every shot. Conformance is **reel-equivalence** — film-equivalence extended over cuts. - Shots MAY revisit earlier world ticks (a cut back in time from a new angle), so the world is sampled at the sorted/de-duplicated union of all frame ticks — convergence still holds because replay-equivalence is order-independent. - Reuses the film machinery (shared camera interpolation + pose bytes) + the `FilmReceipt` profile (identity = `reel_hash`); `wai_meter` grows a `.wrel` branch. Registered + tiered Lossless. Corpus vector `coverage` (the character beat cut three ways, incl. a cut back in time) joins `film-conformance/`, verified by `wai_film_vectors` on the four-platform matrix. SPEC §5 row + `extensions/linear-film.md §7`. ### Linear film — `wai.film.linear` (2026-06-16) - "A film is a sealed, camera-on-rails world." A generated film ships not as pixels but as the **world** that produces it (an embedded sealed `wai.world.replay`) + a **camera track** (keyframed pose over the world's ticks) + a **frame schedule** (which tick each frame samples). The sink renders the frames — lever 3 (instructions-at-the-sink) for the synthesizable slice of "video" (animation / CGI / motion graphics). The conformance corpus carries whole films in ~1 KB. - `wai-rs` `film` feature (reuses the world engine + `Fx` floor + blake3): the `WFLM` container (contract + embedded `WWLD` + camera track) and the deterministic reconstruct contract. Conformance is **film-equivalence** — per-frame `BLAKE3(world_state_hash(frame_tick) || interpolated camera pose)` is bit-identical on every machine. It **composes** the embedded world's replay-equivalence (inherited unchanged) with the camera track in the `wai.det.fixed64` floor; the render (pixels) stays the sink's. - The world engine gained one general primitive: `collect_checkpoints_ sampled` / `run_sampled_dispatch` — read the state hash at an arbitrary frame cadence, independent of the contract's checkpoint cadence (a renderer samples the timeline at its own frame rate). - Corpus `film-conformance/` + `wai_film_vectors gen|verify` (dolly, two-shot, crane-up) — all pass, on the four-platform CI matrix. Registered + tiered Lossless (film-equivalence is exact). `FilmReceipt` (per-frame Merkle leaves) + a `wai_meter` `.wflm` branch emitting **measured** joules. - Spec PROMOTED to `extensions/linear-film.md`, SPEC §5 temporal-content row, content-taxonomy updated (animation-film → ✅). The biggest commercial wedge in the taxonomy, now shipped. ### Music as instructions — `wai.audio.score`, LIVE at /score (2026-06-14) - The deepest audio lever: a score carries **no recording**, only note events — ship the composition, not the recording. `wai-rs` `score` feature (reuses the world `Fx` floor + blake3): the `WSCR` container (contract + note events) and a **trig-free** deterministic reference synth — a phase accumulator drives a piecewise-linear oscillator (saw/square/triangle/pulse) shaped by a linear ADSR, all `+ − × ÷`. Pitch is a per-sample phase increment authored off-line (no exponential in the floor). `mixdown_hash` is the **mixdown-equivalence** quantity, the music counterpart of spatial audio; a soundfont/neural instrument is the sink's presentation. - Corpus `score-conformance/` + `wai_score_vectors gen|verify` (scale, chord/polyphony, timbre-changing arpeggio) — all pass. Registered + tiered Lossless (exact reference mix). - Browser sink `ScoreSink` (`wai-web`) + `/score` page, LIVE: synthesizes the score in-tab, verifies the mixdown hash vs the corpus, draws a piano roll, and plays the reconstructed PCM via Web Audio. wasm pkg uniquely named `wai_score`. - Spec PROMOTED to `extensions/score.md` (site mirror), SPEC §5 row, content-taxonomy updated (music → ✅). Completes the audio surface: recorded objects (audio.scene) and synthesized notes (audio.score). ### Neural subsystem reconciled onto main (2026-06-14) - Wires the full learned-image/audio decoder family into `neural/mod.rs` and brings the categorical `rANS` (`decode_categorical`) it needs: hyperprior (`mbt2018_mean`), checkerboard context (`elic`), channel-autoregressive (`mlicpp`/`mlicv2`/`cmic`), integer-exact (`int_synth`), `glc` (VQ), inter-frame video (`dcvc_rt`/`dcvc_fm`), `snac` audio — plus the ISO/IEC 6048-1 JPEG-AI entropy + codestream + σ/value/filter port (`jpegai_*`). - Restores the **semantic / generative path** (`semantic.rs`, CLIP-similarity conformance, `ModelRequirement.min_sim`/`sim_metric`) and the shared deterministic ONNX loader (`ort_session.rs`), behind the `semantic` feature. - Re-adds `byte-exact-conformance` (cross-arch rANS + int_synth + JPEG-AI mANS goldens) and the model export/encode tooling under `tools/`. - All features build + test green together (`neural` + `semantic` + the interactive/temporal cargo classes); clippy clean. ### Asset provenance — `wai.asset.provenance`, LIVE at /provenance (2026-06-13) - **Lever 4 — the "verifies whatever it carries" half.** For content that cannot be reduced to instructions (a photo, a recording), WAI does not reconstruct — it carries the opaque bytes content-addressed and attaches a signed, energy-accounted **lineage**. `AssetReceipt` (`provenance.rs`, `provenance` feature): a JWP group whose objects are processing **steps** (op + input/output content hash + work + measured joules), Merkle-bound, Ed25519-signed, with chain-linkage verification (each step's input = the previous output, final output = the asset hash) and `parent_receipt_hash` chaining. `from_json` so a verifying sink can re-parse it. - **`wai_prov` tool**: synthesizes a real image, builds a 3-step lineage (origin → PNG master → published JPEG) with the **real measured CPU energy** of each encode (IOReport via macmon, no sudo), wraps the JPEG as a content-addressed `wai.image.jpeg` envelope, and emits the signed receipt (origin 0 + PNG 0.43 J + JPEG 0.14 J = 0.57 J measured). - **Browser sink — `AssetProvenance` (`wai-web`)** + `/provenance` demo, LIVE: recomputes the asset's content hash in-tab and compares to the receipt (tamper-evidence — a "corrupt one byte" button flips it red), verifies the Ed25519 signature + Merkle chain, shows the energy- accounted lineage, and hands the carried JPEG to a native `` (the browser decodes it; WAI does not). wasm pkg uniquely named `wai_prov`. - **Spec PROMOTED to `extensions/provenance.md`** (site mirror). It is a receipt profile *about* a carried asset (like `jwp-receipts`), not a §5 payload codec — the asset keeps its own capability (`wai.image.jpeg`). Completes the thesis: carries whatever it can reconstruct, verifies whatever it carries. ### Cargo-class polish wave — real cloud, EWA render, receipts, telepresence (2026-06-13) Four follow-on builds across the new cargo classes: - **Real-SPZ-derived 4D clip** (`wai-spz-gen 4d`): the `bloom` clip animates 1200 genuine Niantic-SPZ Gaussians (the same cloud the static `wai.splat.spz` asset ships, round-tripped through the SPZ library) — real cloud, authored motion. LIVE on `/volumetric`. Honest framing: the win is a verifiable per-frame state, not capture fidelity. - **WebGPU EWA render on `/volumetric`**: the page now renders the reconstructed cloud with a real anisotropic EWA splat pipeline (the same one the worlds demo uses) — per-Gaussian covariance projection, conic, 3σ bounding quad, premultiplied blend, back-to-front sort — with a 2D alpha-disc fallback where WebGPU is unavailable. The hash-verify badge is render-independent. Confirmed on WebGPU (Chrome) + 2D fallback. - **Per-class receipts with measured joules**: `HapticReceipt` (`haptic/receipt.rs`) + `Splat4dReceipt` (`splat4d/receipt.rs`) — the worlds/audio JWP receipt profile transposed to touch (per-channel blocks) and volumetric (per-frame blocks), reusing `world::receipt:: merkle_root`. New `wai_meter` bin measures **real CPU energy** (IOReport via macmon, no sudo) for haptic/splat4d reconstruction and emits a signed, self-verified receipt carrying the measured `joules_micro`; `wai_world_meter` now also dispatches `feed0`/`skeletal0`, so feed and avatar worlds get the same measured `SessionReceipt`. Every cargo class now has a measured-energy receipt path (haptic 57 µJ, splat4d 0.21 J for 1200 Gaussians × 17 frames, feed/avatar mJ-scale). - **Avatar telepresence — LIVE at `/telepresence`**: a live skeletal world of N avatars over the existing worlds relay. New `SkeletalActor` (wasm) signs `CREATE` + `SET_ROTATION` `CausalOp`s; the worlds transport (relay-first + BroadcastChannel fallback, hello/sync/op/hash, boundary-hash convergence) is reused verbatim. Each tab creates and animates its own arm rig; sinks converge on the pose hash ("poses agree @ tick N"). Verified on prod through the real relay with two isolated contexts — identical pose hashes, zero errors. ### Volumetric / 4D splats — `wai.splat.4d`, LIVE at /volumetric (2026-06-13) - **Phase 5 complete — the roadmap frontier.** A `wai.splat.4d` is a **temporal Gaussian splat**: the static splat extended to time. Each Gaussian carries a keyframed track (position / scale / rotation / color / opacity); the sink interpolates the cloud at frame N in the `wai.det.fixed64` floor. Conformance is **frame-state-equivalence**: the reconstructed Gaussian set at frame N hashes identically on every sink, no tolerance — the temporal counterpart of the static splat. - **`wai-rs` `splat4d` feature** (reuses the world `Fx` floor + blake3, no codec/ML deps): the `W4DS` container (contract + per-Gaussian keyframe tracks, 14 `Fx` per key) + a deterministic reconstruct (Fx linear interpolation, hold outside the track). Trig-free (quaternions interpolate component-wise; the sink normalizes for rendering). `frame_hash` = the frame-state-equivalence quantity; `clip_hash` = frame-independent identity. Registered + tiered **Lossy** (consistent with `wai.splat.spz` — the splat representation approximates the captured scene, even though the reconstruction is frame-state-exact). - **`volumetric-conformance/` corpus** + `wai_splat4d_vectors gen|verify`: orbit (two Gaussians swap places), pulse (scale + fade), morph (color shift) — every frame's hash checked, all pass. - **Browser sink — `Splat4dSink` (`wai-web`)** + `/volumetric` demo page, LIVE: reconstructs the cloud in-tab and **verifies every frame's hash** against the corpus; a frame scrubber re-derives the cloud, drawn as painter-sorted alpha Gaussians (a sink with WebGPU renders the same set with the full EWA pipeline — presentation). wasm pkg uniquely named `wai_splat4d`. - **Spec PROMOTED to `extensions/volumetric.md`** (site mirror), registered in SPEC.md §5 (`wai.splat.4d`). **Honest about size**: the win is content-addressing + dedup + a verifiable per-frame state, NOT payload reduction (a real cloud is MB-scale) — stated plainly. - **The cargo set is complete across the medium**: sight (worlds/replay), hearing (audio/mixdown), touch (haptics/sample), live state (feed/replay), a moving body (avatar/pose), and now a moving volume (splat.4d/frame-state) — one `wai.det.fixed64` floor, one verification. ### Avatars / digital humans — `wai.avatar.pose`, LIVE at /avatar (2026-06-13) - **Phase 4 complete — digital humans**: motion as instructions. An avatar is a **skeletal world** — the worlds container/op-log/ordering/ sealing/state-hash/receipt-chain, with the `step` reduced to the identity. Conformance is **pose-equivalence** (the per-bone local transforms hash identically), a special case of replay-equivalence. - **`wai.world.sim.skeletal0`** (in `wai-rs` `world` feature): a bone is a `parent[16] + translation[3] + rotation-quaternion[4] + scale[3]` (all `wai.det.fixed64`). Mutations: `CREATE` / `DELETE` / `SET_ROTATION` (the animation driver) / `SET_TRANSLATION` / `SET_SCALE`, all LWW. **Trig-free floor**: rotations are quaternions and the FK composition is `+ − ×` only; axis-angle authoring happens off-line. The forward-kinematics → world space and the skinning of a `wai.mesh.glb` are presentation (a pure function of the hashed pose), never in the hash. Registered + tiered `Lossless`; dispatched in `run_sealed_dispatch`. - **`avatar-conformance/` corpus** + `wai_avatar_vectors gen|verify`: a sealed 3-bone arm wave, a sealed 5-bone humanoid pose, and three seeded adversarial-arrival shuffles of the arm wave — all converge, all pass. - **Browser sink — `SkeletalSink` (`wai-web`)** + `/avatar` demo page, LIVE: replays a sealed animation in-tab, **re-derives the sealed pose hash** (checked against the native corpus and the file's own seal), and exposes FK world-space joints + bone segments to draw the skeleton on a rotating orthographic canvas with a tick scrubber. wasm pkg uniquely named `wai_avatar`. - **Spec PROMOTED to `extensions/avatars.md`** (site mirror), registered in SPEC.md §5 (`wai.avatar.pose`). A moving body joins sight (worlds), sound (audio), touch (haptics), and live state (feeds) under one floor. ### State-feeds — `wai.feed.state`, LIVE at /feed (2026-06-13) - **Phase 3 complete — live state-feeds**: state as instructions. A feed is a **non-physics world** — the same `WWLD` container, op log, canonical ordering, sealing, state hash, and receipt chain as interactive-worlds, with the simulation `step` reduced to the identity. Conformance is therefore **replay-equivalence**, reused wholesale: no new criterion. - **`wai.world.sim.feed0`** (in `wai-rs` `world` feature): nodes are records of self-describing typed cells (`Int` i64, `Fixed` Q32.32, `Text` UTF-8 — hashed without a schema). Mutations: `CREATE` / `DELETE` / `SET_CELL` (LWW) and `ADD_INT` / `ADD_FIXED` (SUM — commutative accumulation). Wrong-typed / out-of-range / absent-node updates are deterministic no-ops. Registered + tiered `Lossless` in the central `codecs` registry; dispatched in `run_sealed_dispatch`. - **`feed-conformance/` corpus** + `wai_feed_vectors gen|verify`: sealed scoreboard (SUM scoring + an LWW correction), telemetry (LWW gauge + SUM packet count, incl. a negative `ADD_FIXED`), market tape (LWW last + SUM volume), and three seeded adversarial-arrival shuffles of the scoreboard — all converge, all pass (hash equality). - **Browser sink — `FeedSink` (`wai-web`)** + `/feed` demo page, LIVE: loads a sealed feed, replays it in-tab with the same engine that made the corpus, and **re-derives the publisher's sealed BLAKE3 state hash** (the file's seal also equals the native corpus hash — both checked). A tick scrubber re-derives the feed at any tick; the record table renders the typed cells generically. wasm pkg uniquely named `wai_feed` (avoids the deploy fingerprinter's basename collision). - **Spec PROMOTED to `extensions/state-feeds.md`** (site mirror), registered in SPEC.md §5 (`wai.feed.state`). The **broadcast profile** (one authoritative publisher, many read-only sinks) is documented as a deployment shape of `wai.world.live`, not a new format. State joins sight (worlds/replay), hearing (audio/mixdown), and touch (haptics/sample) under one floor — verified the same way. ### Interactive worlds — `world` feature + conformance corpus (2026-06-11) - **New cargo class implemented**: the interactive-worlds extension (spec draft staged in `wai-transaction-science-web/research/`, destined for `extensions/interactive-worlds.md`). Media that consumes input: a world is a content-addressed artifact, every event is a signed `CausalOp`, and conformance is **replay-equivalence** — identical BLAKE3 state hash at every checkpoint, no tolerance parameter. One container (`WWLD`) serves `wai.world.scene` / `wai.world.replay` / `wai.world.live`, differing only in op-log seal state; an input log and a multiplayer op log are the same object. - **`wai-rs` `world` feature** (pure Rust, no ML dep chain; deps: blake3 + ed25519-dalek): `Fx` Q32.32 fixed-point (`wai.det.fixed64`, the floor profile — bit-exact unconditionally), xoshiro256++ contract rng, signed/dep-gated ops with canonical `(effect_tick, lamport, actor)` ordering, coordination-free BLAKE3 node minting, WWLD section container with content-binding seal, `Sim` trait (the extension standardizes the contract, not a physics engine) + `kinematic0` reference registrant, sealed `Replayer` + `LiveWorld` engines (buffering, past-tick re-derivation, equivocation rejection, receipted-checkpoint history sealing). - **`world-conformance/` corpus** + `wai_world_vectors gen|verify`: sealed replays, LWW/SUM conflict batteries, dep-gap / past-tick / shuffled-arrival live streams, expected hashes. All vectors pass; clippy clean on the new code. - **Spec PROMOTED to `extensions/interactive-worlds.md`** (site mirror `src/pages/extensions/`), registered in SPEC.md §5: `wai.world.scene` / `wai.world.replay` / `wai.world.live` + asset media `wai.mesh.glb` (canonical glTF 2.0 Binary, Khronos — decoder sink/library-supplied); `media: "world"` noted in §3. Capability strings live in the central `codecs` registry (`known_capabilities()`), tiered `Lossless` — replay-equivalence is exact state reproduction, the simulation analog of lossless. - **Second sim registrant `wai.world.sim.ballistic0`** (gravity + ground-plane bounce, per-node radius/restitution, `G = −10`, all Q32.32 — bit-exact through every bounce) proving the registration story: entered registry, dispatch, and corpus (`ballistic_bounce` vector) without one line of the extension changing. New `run_sealed_dispatch` routes on `contract.sim`; a missing sim reports the world inert (`UnknownSimCapability`), mirroring SPEC.md §4 step-6 semantics so callers can fall back per the envelope. - **Native energy meter — measured joules, no sudo (2026-06-13)**: corrected a wrong assumption that real energy needed RAPL/sudo (unavailable on the arm64 prod box). On Apple Silicon the SoC energy counters are readable through IOReport with **no privileges** (the source `macmon`/`asitop` use; verified euid≠0 reads CPU/GPU/ANE power on the M3 Ultra), and `proc_pid_rusage` gives exact per-process retired cycles/instructions. New `wai_world_meter` bin (`world` feature): brackets a tight replay burst of a sealed `.wwld` against an idle baseline, samples CPU power via macmon, and reports the marginal **measured** CPU energy — e.g. the truss replay metered at ~512 µJ / replay (~4.3 µJ/tick) at 3.2 W marginal, a real number that fills a receipt's `joules_micro` where a native sink runs. The burst runs on N threads so the marginal draw clears a shared box's idle fluctuation (single-thread 512 µJ vs 10-thread 524 µJ/replay cross-validates the method — per-replay energy is invariant to parallelism). With a second arg the meter emits a full §8 `SessionReceipt` carrying the MEASURED joules + exact `work_steps`, signed and self-verified — real measured energy end to end. Spec §8 + the `/worlds` page corrected: the sink-metering path needs no sudo (IOReport on Apple Silicon, RAPL on x86); the browser estimate is the fallback, not the ceiling. clippy clean, 94 tests green. - **Phase 2 begun — haptics (`wai.haptic.signal`) (2026-06-13)**: touch as instructions. Spec drafted (`research/haptics-draft.md`), capability registered + tiered. `wai-rs` `haptic` feature (reuses the world `Fx` floor, no codec/ML deps): the `WHAP` container + a deterministic reconstruct — per channel, a keyframed **intensity/sharpness** envelope (the Core-Haptics pair every actuator API maps to) linear-interpolated in `Fx` and quantized to `u8`. Trig-free by construction (amplitude + sharpness, not a modulated carrier), so the floor reconstructs it exactly with `+,−,×,÷` alone. `signal_hash` is the **sample-equivalence** quantity. `wai_haptic_vectors gen|verify` + `haptic-conformance/` (ramp, thud, dual-rumble pulse) — all pass, hash equality. 104 tests, clippy clean. Next: Gamepad/Vibration browser sink + `/haptics` demo, then promote. This completes the sight → hearing → touch trio's engines. - **Spatial-audio: receipt profile + spec PROMOTED (2026-06-13)** — Phase 1 complete. `audio_scene::SceneReceipt` (the worlds receipt transposed to audio): identity `scene_hash`, the JWP group's objects are the mix's **PCM blocks** (`BLAKE3("wai:audio-block")`), Merkle-bound with JWP's exact domains, one Ed25519 sig over scene_hash + root + blocks + `work_mac` (samples × objects, exact signed) + carried `joules_micro`; `parent_receipt_hash` chains renders. Reuses `world::receipt::merkle_root` — minimal new code. 2 receipt tests (101 total). Draft promoted to `extensions/spatial-audio.md` (+ site mirror, SPEC §5 row + extension pointer) — and the canonical spec corrected to the **as-built** `sqrt_constpower` pan (sqrt-only, no transcendental tables) and the real receipt shape, since registration froze the format. clippy clean; spec page live. Phase 1 (spatial audio) DONE: spec · register · mixer · corpus · live /audio sink · receipt · promoted. Next: Phase 2 — haptics. - **Spatial-audio browser sink — LIVE at /audio (2026-06-13)**: `wai-web` `AudioSink` (wasm-bindgen): parse `WSCN`, load the corpus sources, run the exact deterministic mixdown, expose `mixdown_hash_hex` + `scene_hash_hex` + `mix` (i16 PCM) + `object_xz(t)`. The `/audio` page (site-styled, in nav) loads a scene, **verifies the mixdown hash in-browser against the corpus** (mixdown-equivalence, the same proof worlds does for state), plays the bit-exact stereo mix via an AudioBuffer, and shows a top-down radar of the objects panning. Proven on prod: all three scenes verify, zero errors — wasm-in-Chrome mixes byte-identically to native. Honest split kept: plays the exact stereo floor; binaural/HRTF noted as the richer optional presentation. - **Spatial-audio conformance corpus (2026-06-13)**: `wai_audio_vectors gen|verify` + `audio-conformance/` — the mixdown-equivalence analog of the worlds corpus. Three vectors (static stereo pair, distance near/far, keyframed left→right sweep) ship a `.wscn` scene + a `.sources.bin` (decoded mono tones) + expected `scene_hash` and `mixdown_hash`; `verify` mixes each and checks hash equality, no tolerance. All pass; clippy clean. - **Spatial-audio reference mixer — mixdown-equivalence (2026-06-13)**: `wai-rs` `audio_scene` feature (reuses the world `Fx` floor + blake3, no codec/ML deps): the `WSCN` container (scene contract + object table + asset table + optional seal, mirrors WWLD) and a fully deterministic mixdown — `Fx` position interpolation, `inverse_clamped` distance attenuation, `sqrt_constpower` pan (sqrt only, no transcendental tables), `Fx` accumulate, round-half-away i16 quantization in integer space. `mixdown_hash` (BLAKE3 over canonical i16LE PCM for a declared layout) is the **mixdown-equivalence** quantity; `scene_hash` is the identity. 5 tests prove it: centered = equal L/R, hard-left = silent right, deterministic + layout-bound hash, distance attenuates, a moving source sweeps L→R + container round-trips. Sources arrive as decoded mono PCM keyed by content hash — decoding the `wai.audio.*` assets is the sink's job. clippy clean. Next: conformance corpus bin + Web Audio browser sink + demo. - **Content taxonomy + cargo roadmap; spatial-audio Phase 1 begun (2026-06-13)**: mapped the full space of web-transported content WAI's approach covers (`research/content-taxonomy.md`) — a capture→synthesizable→executable→ symbolic spectrum × four levers (codec dispatch, neural prior, instructions-at-sink, determinism+receipts), with a per-row mechanism/extension/status table and the white space called out. A phased build plan (`research/cargo-roadmap.md`) takes each white-space class through the proven interactive-worlds recipe. **Phase 1 kicked off — object-based spatial audio** (`wai.audio.scene`): the audio analog of interactive-worlds (ship sound objects + a scene, render to the sink's layout; **mixdown-equivalence** = replay-equivalence with the tick replaced by the audio sample). Spec drafted (`research/spatial-audio-draft.md`, `WSCN` container, `Fx` mixdown, content-addressed `wai.audio.*` sources, stereo bit-exact floor + binaural perceptual tier, WAI2 recorded-mix fallback, JWP receipt profile); `wai.audio.scene` registered + tiered Lossless. 94 tests green. Remaining phases: haptics, live state-feeds, digital humans, volumetric/4D splats. - **Real SPZ decode — `wai.splat.spz` end to end (2026-06-13)**: the splat gallery now renders a genuine Niantic-SPZ file, not a mesh-derived stand-in. New `wai-spz-gen` crate depends on the field's `spz` reference library (the sink-supplied codec WAI dispatches to), encodes a 2,400-Gaussian cube cloud to a real `.spz` (validated by re-loading through the library), and wraps it as a content-addressed `wai.splat.spz` WAI object (`nebula.spz.wai`). `gen-gallery` adds it to the scene's asset table (role 1) beside the glb. Browser: `WorldSink.asset_payload(name, env, cap)` generalizes the §4.1 verify to any capability; the page gunzips (DecompressionStream) and decodes the SPZ bitstream in-tab — 16-byte header, non-interleaved arrays, 24-bit fixed-point positions, log scales, smallest-three quaternions, byte/255 opacity+color — into per-Gaussian world covariance for the EWA pipeline. Decoded count matches the encoded 2,400 (validates the layout); the cube silhouette + surface-aligned discs validate the quaternion math. Page copy corrected: the splat view decodes a real SPZ asset (no longer "derived from mesh"). Verified locally (2,400 Gaussians, zero errors); prod serves the identical artifacts. - **Anisotropic EWA splatting (2026-06-13)**: upgraded the splat renderer from isotropic billboards to real 3D Gaussian splatting. Each splat carries an anisotropic 3×3 covariance (surface-aligned flat discs: wide in the triangle tangent plane, thin along the normal); the vertex shader projects it to a screen-space conic (Σ' = J·W·Σ·Wᵀ·Jᵀ with the perspective Jacobian), emits a tight 3σ bounding quad, and the fragment evaluates the conic — the textbook 3DGS pipeline, still CPU-sorted back-to-front and alpha-blended. Cam uniform carries view + proj separately (+ viewport/focal) for the Jacobian. Visibly sharper, surface-coherent splats; proven on prod, zero errors. - **3D render layer — PBR mesh + Gaussian splats (2026-06-13)**: the `/worlds` gallery now renders the content-addressed, hash-verified `wai.mesh.glb` two ways in a real 3D path (perspective camera + depth buffer, all in `wai-web/demo/world.js`, no wasm change): a **solid PBR-shaded cube** (flat normals from screen-space derivatives, Lambert + spec + sky-ambient, gamma) and a **Gaussian-splat cloud** — instanced camera-facing quads with a Gaussian falloff, back-to-front depth-sorted each frame (painter's order) and alpha-blended. The splat cloud is *derived from the same verified mesh*, making the thesis visible: one content-addressed asset, two sink-side renderings, presentation never touches the state hash. SPZ binary decode stays sink-supplied (stated honestly on the page); the renderer is the real deliverable. Two gallery vectors (`gallery` / `gallery_splat`); the 2D sim path is untouched. Proven on prod: both render, 768 splats from the cube, zero errors. - **Receipt chaining + work-meter + `wai.splat.spz` (2026-06-12)**: three follow-ons. (1) `SessionReceipt` gains `parent_receipt_hash` (BLAKE3 of the prior receipt's canonical bytes — the JWP linkage) + `receipt_hash()` + `verify_chain`: successive sealed sessions of one world (shared `world_hash`, `group_id`+1) form an append-only signed timeline. (2) An **honest energy meter**: receipts carry exact, signed, deterministic `work_steps` (Σ node-steps, from `collect_checkpoints_and_work`) AND a separately *carried* `joules_micro` — never derived in the receipt; the browser supplies a wall-clock estimate at a **declared** coefficient (labeled "est@4W"), native sinks supply real RAPL, an unmetered sink signs `0`. (3) Registered **`wai.splat.spz`** — Gaussian-splat scenes as an asset capability parallel to `wai.mesh.glb` (presentation only, never in the state hash; the win is content-addressing + dedup, not payload size; renderer sink-supplied). Browser `seal_session(seed, id, joules, group, parent)` chains via localStorage; the `/worlds` HUD shows `chain #N ↳linked · N steps · ≈X mJ est@4W · ✓ re-verified`. Proven on prod: two seals chain (g0 parent null → g1 parent = g0.receipt_hash), exact work 248 steps, est joules carried, zero errors. Spec §8 updated (chaining + the two-quantity energy model). 94 native tests, 10/10 vectors, clippy clean. - **Asset path SHIPPED — `wai.mesh.glb`, content-addressed (2026-06-12, extension §4.1)**: worlds are made of WAI objects. New `world::asset_content_hash` (BLAKE3 `wai:asset\x01` over a complete WAI envelope) + `DOMAIN_ASSET`; native tests PROVE the load-bearing claim — a scene with an asset table has the byte-identical state hash AND world hash as the same scene without it (assets are presentation-only, never in the state hash), and the asset hash binds the exact envelope. Generator `gen-gallery` writes a `gallery.wwld` scene + `assets/cube.glb.wai`: a real glTF 2.0 Binary cube (hand-written GLB writer) wrapped as a `wai.mesh.glb` WAI envelope and content-addressed in the table. Browser: `WorldSink.assets()` + `asset_glb(name, envelope)` (verifies content hash in WASM, rejects a mismatch, unwraps the WAI envelope to the `.glb` payload); the page decodes the GLB (minimal JS decoder) and renders the verified mesh as a spinning wireframe. Proven on prod: `1/1 verified ✓ · cube.glb`, and the state hash stays fixed while the mesh spins — §4.1 made visible. 92 native tests, 10/10 vectors, clippy clean. - **Session receipts + seal-to-replay SHIPPED** (2026-06-12, extension §8 implemented): `world/receipt.rs` builds a `SessionReceipt` — a JWP *profile*, not a fork: op batches (one per checkpoint interval, content-hashed `wai:world-ops\x01`) are Merkle-bound with JWP's exact leaf/node domain separators, and one Ed25519 signature covers the world-profile payload (world hash + checkpoint `state_hash` list + carried microjoules). Joules are **carried, never fabricated** — the builder takes the figure; an unmetered session signs `0` and says so. New `world_hash` (BLAKE3 over scene = contract + tick-0 graph, the §7.4 shareable identity), `collect_checkpoints` (unsealed replay → checkpoint list). Browser: `WorldSink.seal_session(seed, signer_id, joules)` → `SealedSession{replay_bytes, receipt_json, root_hash, …}`; the `/worlds` page's "⏷ seal & download replay" turns a live, shared, edited world into a portable `.wwld` + `.receipt.json`, re-opens the fresh replay in a throwaway sink to prove it re-verifies, and shows the Merkle root + batch count in the HUD. Proven on prod: fracture a tower across the relay → seal → 3-batch receipt, valid sig, replay re-verified. 90 native tests, 10/10 vectors, clippy clean. - **Cross-device multiplayer SHIPPED** (2026-06-12, `wai-relay/` + `wai-web` Actor binding): every tab/device is a signed actor. New `Actor` wasm-bindgen class (browser-held Ed25519 key; clicks author real `CausalOp`s; Lamport discipline via `observe_lamport`); `wai-world-relay` reference transport (axum WS, per-room fan-out, retains the signed op log for late joiners, **verifies every op's Ed25519 signature at the door** with the same `wai` crate — it adjudicates nothing; sinks converge because the math does). Page transport is relay-first (room in the URL hash) with BroadcastChannel fallback. Deployed at `wai.transaction.science/worlds-ws/*` (launchd `com.wai.world-relay`, localhost:9670). Proven on prod with two ISOLATED browser contexts: point-blank axial shoves snap two members (23→21 nodes) and both contexts converge — "✓ hashes agree". Also fixed `build.rs` emitting libFLAC link flags regardless of features (pkg-config probe side effect): now gated on `codecs`. - **Browser sink SHIPPED** (2026-06-12, `wai-web/src/world.rs` + `wai-web/demo/world.{html,js}`): a `wai.world.replay` sink in a tab. NOT a port — the `wai` crate itself now compiles to wasm32 via the new default-on **`codecs` feature** (C codec chain gated; envelope + `world` engine build with `--no-default-features --features world`), and wai-web binds it (`WorldSink`: advance/particles/members/ state-hash/verify-seal; sim state stays exact Q32.32, f32 only for display). WebGPU renderer (instanced SDF circles + member lines). Proven in real Chrome via Playwright: the 1,282-byte truss_fracture payload replays 120 ticks, the member snaps, and the browser's BLAKE3 state hash equals the native corpus hash byte-for-byte — **replay-equivalence across arm64-native and wasm32-in-Chrome, demonstrated**, with the Ed25519 seal verified in-page. - **Solver-backed registrant `wai.world.sim.truss0`** (2026-06-12, `world/truss.rs`): structural simulation — particles + distance members solved per tick by position-based dynamics (fixed 8-pass Gauss–Seidel projection in canonical member order), entirely in `wai.det.fixed64` including a new floor-rounded `Fx::sqrt` (128-bit integer Newton). Members carry a yield strain measured on predicted positions before the solve; an overloaded member **fractures** as an emergent solver consequence — two actors pulling the same member in opposite directions (the SUM rule feeding the solver) snap it bit-identically on every machine (`truss_fracture` corpus vector). Registration constants: `G = −10`, `ITERATIONS = 8`, ground at `y = 0`, velocity damping `63/64`/tick (static equilibrium). `LiveWorld` gained `state()`/`current_tick()` accessors (what a renderer reads). Where the float-based CAD solvers can't transplant (floats aren't cross-platform bit-exact), solver *form* does: iterative constraint projection in the deterministic profile. ### Inter-frame video, categorical/VQ images, and real-model validation - **`wai.neural.dcvc_rt` / `wai.neural.dcvc_fm` — inter-frame video decoder implemented** (`neural/video_dcvc.rs`). "Requires CUDA" was a *speed* claim: inter-frame decode is a frame loop that propagates a reference feature, and ONNX Runtime runs it on CPU. The per-frame entropy decode reuses the CompressAI-validated hyperprior path; the temporal net is sink-supplied. Validated end-to-end (3 inter-frames, feature propagation confirmed). - **`wai.neural.glc` — categorical / VQ decoder implemented** (`neural/glc.rs`). Not the Gaussian-conditional family: codebook indices decoded via a per-position softmax → `pmf_to_quantized_cdf` (bit-exact with CompressAI's C++, unit-tested) → a bypass-free categorical rANS over a checkerboard → VQ synthesis. Surfaced that categorical entropy is the most determinism-sensitive path — a per-position CDF flips on ~1e-5 logit drift where Gaussian scale buckets don't. - **Real-model validation** beyond synthetic stand-ins. SNAC's genuine `snac_24khz` now exports (`tools/wai_snac_export.py`: zero-noise so the decode is deterministic at all + a reshape-free snake) and `wai-rs` decodes it to ~9e-7 of the torch reference. A real PTQ-trained, activation-calibrated `int_synth` model decodes **bit-identically** (max|Δ|=0) to its integer-forward reference (`tools/wai_int_synth_train.py`; also fixed a zero-gradient `torch.floor` bug in the QAT trainer). ### Learned-image codec family — decoders implemented and validated - The reference crate now decodes the full conditional-entropy family natively, not just the factorized baseline: **`wai.neural.mbt2018_mean`** (mean-scale hyperprior, Gaussian-conditional), **`wai.neural.elic`** (checkerboard spatial context), **`wai.neural.mlicpp`** (channel-autoregressive multi-reference), and the integer-exact **`wai.neural.int_synth`** (all-integer transposed-conv synthesis — a decode that is bit-identical on every machine *by construction*, the first capability that can soundly back `intent=replicate` cross-hardware). The rANS + entropy decode live in `wai-rs`; the analysis/synthesis nets are sink-supplied ONNX. - **Current-SOTA entries added as live routes:** `wai.neural.mlicv2` (MLICv2/v2+, the current SOTA learned image) and `wai.neural.cmic` (Content-Aware Mamba) both decode through the multi-reference core — their entropy model is the same channel-slice + checkerboard structure; the gains live in their sink-supplied transforms. - **Validation against the reference codec.** `bmshj2018` and `mbt2018_mean` are checked byte-for-byte against pretrained CompressAI models (reconstruction matches to within final-layer float rounding); `elic`/`mlicpp` are checked end-to-end against minimal CompressAI- compatible models; `int_synth` against a constructed integer model. Surfacing this also caught (and the spec now documents) a real determinism hazard: a learned codec's entropy decode desyncs unless the encoder and decoder compute the scale buckets with the *same* model — float drift between two implementations of `h_s` flips bucket assignments and corrupts the stream. ### Audio: SNAC hierarchical codec - **`wai.neural.snac`** — SNAC's multi-resolution RVQ, distinct from the flat single-grid RVQ runtime: N variable-length code streams at hierarchical rates, decoded via dynamic-arity ONNX inputs. Ubiquitous in low-bitrate TTS. ### Removed: off-charter Python runtime (`wai-neural/`) - Moved `wai-neural/` OUT of this repo to `../../wai-neural-prototype/`. It was a parallel **Python** runtime (`WAI4` generative-semantic container + CLIP-embedding payloads + SD-XL-Turbo regeneration + CLIP-similarity conformance) — a disconnected second implementation in the wrong language. The reference implementation is **Rust** (`wai-rs`); nothing referenced the Python package. The *idea* it proved — a behavioral/perceptual neural path that conforms by semantic similarity rather than a weights hash — is sound and belongs in `wai-rs` as a future Rust path alongside the bit-exact, prior-pinned path. The Python is kept only as a prototype reference (and it hardcodes a machine-local model path, so it never ran portably). This does NOT touch the Python model-export tools in `tools/`, which are legitimate dev tooling. ### Neural condition: working end-to-end - `tools/wai_encodec_encode.py` / `wai_encodec_decode.py` — first working **neural capability** (`wai.neural.encodec32`): source encodes audio via locally-installed EnCodec, ships only the quantized tokens in a standard WAI envelope (236 MB of model weights stay on the source's disk, never enter the envelope); sink with EnCodec installed decodes. Measured: **5 s of audio → 2.3 KB envelope (141× compressed)**, model on the wire = 0 B. - Rust SDK gains `wai inspect` CLI: parses an envelope, resolves the capability + fallback against `sink_capabilities()`, and prints verdict (`DECODABLE via primary` / `via fallback` / `INERT here`). ### Browser runtime (`wai-web/`) - New crate: wasm-bindgen wrapper exposing `parse(bytes) → WaiEnvelope`, `pack(manifest_json, payload)`, and `known_capabilities()`. 107 KB wasm + 16 KB JS glue. - `wai-web/demo/wai.js` — thin JS shim with `WaiRuntime.register(cap, handler)` dispatch. Backends are **pluggable** (browser-native APIs, transformers.js, WebGPU compute, WebNN, ONNX runtime web) — WAI doesn't bundle any. The sink-side compute fabric is heterogeneous; the deployer picks per capability. - `wai-web/demo/index.html` — drop-and-decode demo with handlers wired to: `createImageBitmap` for `wai.image.{png,jpeg,avif,jxl}`, `AudioContext` for `wai.audio.{opus,flac}`, placeholders for `wai.neural.*` showing where transformers.js / WebNN / WebGPU plug in. Run with `python3 wai-web/demo/serve.py`. - 4 sample `.wai` files in `wai-web/demo/samples/` covering PNG, JXL, Opus, and EnCodec for testing. ### Spec: v1.1 multi-rendition envelope (SPEC.md §6.1) - New magic `WAI2` for envelopes that carry multiple renditions of the same content (e.g. EnCodec + Opus, or AVIF + JPEG-XL). Reframed from "graceful fallback for missing capability" to **policy selection** — in a self-contained ecosystem every sink has every required capability; renditions exist so the sink picks by compute budget, bandwidth, or runtime preference (browser-native vs WebGPU vs WebNN vs native ML). v1.0 (`WAI1`) remains valid and unchanged. ## v0.3.0 **Pivot**: WAI is the **envelope + capability-dispatch + neural-shared- prior** standard. The zeroth condition is a *menu of registered SOTA codec capabilities* (AVIF, JPEG-XL, PNG, JPEG, Opus, FLAC, AV1, zstd, XZ), not a custom codec. The Python prototype and all hand-rolled codec implementations from v0.2.x were removed; Rust is the single reference implementation and the SDK. ### Standard - `SPEC.md` rewritten as a capability-menu spec. Mandatory floor: `wai.image.png`, `wai.audio.flac`, `wai.text.zstd` (every conforming sink MUST decode these). Recommended modern set: `wai.image.jxl`, `wai.image.avif`, `wai.image.jpeg`, `wai.audio.opus`, `wai.video.av1`, `wai.video.av1.lossless`, `wai.text.xz`. Neural capabilities (`wai.neural.*`) remain OPTIONAL and sink-advertised. - Container envelope (`WAI1` magic + length-prefixed manifest + length- prefixed payload) and manifest schema are unchanged from v0.2.x. - Conformance: bit-exact decode-equivalence with each capability's reference library (libjxl / libavif / libopus / libFLAC / rav1e / etc.). ### Rust reference (`wai-rs/`, crate `wai` 0.3.0) - `cargo build --release` produces `libwai.dylib`/`.so`/`.dll` (cdylib), `libwai.a` (staticlib), and `libwai.rlib` (Rust-native). The crate type list is `["lib", "cdylib", "staticlib"]` — drop-in for any language's C-FFI binding. - Codec wrappers: `image` (PNG, JPEG via `jpeg-encoder`, AVIF via `ravif` + `libavif-image` for decode, JPEG-XL via `jpegxl-rs`), `opus` (RFC 7845 Ogg-Opus container so output drops into ffmpeg/VLC/browsers), `claxon`+`flac-bound` for FLAC, `rav1e`+`dav1d` for AV1 (lossy + lossless), `zstd`+`xz2` for text. - Generated C header: `include/wai.h` is regenerated by `build.rs` via `cbindgen` on every build. SDK consumers get a typed header rather than hand-writing `extern "C"` decls. - Cross-platform linkage: `build.rs` probes `pkg-config` for libflac; the macOS-hardcoded `.cargo/config.toml` from v0.2.x is gone. Builds clean on macOS (Homebrew) and Linux (apt) without `RUSTFLAGS`. ### C FFI / SDK surface All `wai_*` functions return a `wai_status_t`: | code | meaning | |---|---| | `WAI_OK` (0) | success | | `WAI_ERR_INVALID_ARG` (1) | null pointer / mismatched length / out-of-range dimension | | `WAI_ERR_CODEC` (2) | codec-level failure (details in `wai_last_error()`) | | `WAI_ERR_CONTAINER` (3) | bad magic / truncated envelope / unparseable manifest JSON | | `WAI_ERR_PANIC` (4) | internal Rust panic caught at the boundary (a wrapper bug — never UB into the C caller) | Every FFI function is wrapped in `std::panic::catch_unwind` so Rust panics never unwind into C (UB). `av1_decode` validates header bounds to prevent libdav1d's C-level `abort()`. `wai_last_error()` is per- thread, populated on any non-OK return. ### Testing - **13 lib tests**: per-codec round-trip + envelope round-trip - **21 FFI robustness tests**: null pointers, length mismatches, dimension overflow, malformed magic, truncated payloads, bad JSON, invalid Opus sample rates, zero fps, garbage decoder input - **10 fuzz smoke targets**: 1024 adversarial random byte sequences per decoder asserting no panic. Stable Rust, no `cargo-fuzz` toolchain needed. Found and fixed a real envelope-truncation panic. - **`wai_bench` corpus walker**: quick mode (4 images + 4 audio + 3 text + 2 video) for spot-checks; `--full` walks the entire corpus (169 images + 5 audio + 13 text + 12 video sequences = 588 codec runs in ~9.4 min). Per-codec aggregate stats: runs, ok / fail counts, WAI vs reference byte sizes, PSNR delta vs ffmpeg, encode time. - **`--report file.json`** writes a structured bench report; `wai_bench_diff baseline.json head.json` flags regressions on bytes (±5% default tolerance), encode time (±25%), or new failures. Non-zero exit on regression — ready for CI gating. - **GitHub Actions CI** (`.github/workflows/ci.yml`): ubuntu + macOS matrix, apt/brew codec deps, `cargo test --release`, cdylib/staticlib artifact verification, quick bench, JSON report uploaded as artifact. ### Honest codec measurements (vs ffmpeg reference, full Kodak/Tecnick/ CLIC/synthetic corpus = 169 images, plus SQAM audio + Silesia text + Derf CIF video) | codec | runs | size vs ref | PSNR Δ vs ref | notes | |---|---:|---:|---:|---| | PNG | 169 | — | lossless | ffmpeg-interop ok | | JPEG q=50 | 169 | +12.7% | +0.27 dB | q-knob calibrated to libjpeg | | JPEG q=80 | 169 | +19.7% | −0.31 dB | residual gap is pure-Rust vs libjpeg-turbo asm | | AVIF q=70 | 169 | — | — | ffmpeg-interop ok | | JPEG-XL lossless | 169 | — | lossless | bit-exact via libjxl | | FLAC | 5 | **−10.1%** | lossless | WAI compression_level 8 vs ffmpeg default 5 | | Opus 64k | 5 | — | — | standard Ogg-Opus container, ffmpeg-interop ok | | zstd-19 | 13 | ±0.0% | — | **byte-identical to system zstd** | | xz-6 | 13 | ±0.0% | — | **byte-identical to system xz** | | AV1 lossy | 12 | +28.3% | — | rav1e vs ffmpeg libsvtav1 (different encoders) | All 588 corpus runs and all 44 tests pass. ### Removed - The Python prototype (`wai/`) and Python BD-rate harness (`bench/`). v0.2.x's hand-rolled wire format is dead; Rust is the single source of truth. - The `webm_lottie_new/wai-rs/` pre-pivot Rust crate (Wasm-decoder-in- bundle model). Replaced by the clean `wai-rs/` SOTA-wrapper crate. --- ## v0.2.0 — superseded (historical record) The v0.2.x direction was hand-rolling custom codec wire formats (DCT + quantization + rANS for image, MDCT for audio, I/P inter-frame for video). Two byte-interoperable reference implementations (Python + Rust) were built and cross-validated. **Why superseded**: re-implementing JPEG/MPEG/Opus poorly is strictly worse than wrapping the field's SOTA libraries. WAI's actual value is the envelope + capability dispatch + the neural-shared-prior model, which v0.3 keeps. Custom codecs are gone; standard libraries are in. ### What v0.2 had - `SPEC.md` defined byte-precise wire formats for image (8×8 DCT-II + YCbCr + JPEG-Q + zigzag + DC-Δ + rANS), audio (orthonormal MDCT + sine window + TDAC), video (I/P inter-frame with integer-pel motion). - Python (`wai/`) + Rust (`wai-rs/`) implementations, byte-equivalent on the lossless paths (rANS, container) and reconstruction-equivalent on the lossy paths. - Honest measured standing: image at JPEG-1992 parity (~5% behind libjpeg), audio uniform-MDCT (not perceptually competitive with MP3), video I/P at ~41% smaller than MJPEG floor but far behind H.264. ### Licensing (unchanged) - Apache-2.0 for the standard and the reference implementation. Royalty-free implementation is a precondition for adoption (VP9/AV1 open-codec precedent; the H.264/H.265 royalty trap is avoided).