fd2a41cd0f
- content-taxonomy: feed/avatar/volumetric marked ✅ (shipped), not ⬜. - cargo-roadmap: header notes Phases 1–5 complete (archived plan). - README: add the interactive/temporal capability table + provenance. - codecs: fix CMIC label ("Content-Adaptive Mamba", arXiv 2508.02192); drop the external BD-rate-vs-VTM comparison (self-referential copy). Co-Authored-By: Claude Opus 4.8 <[email protected]>
193 lines
8.4 KiB
Markdown
193 lines
8.4 KiB
Markdown
# WAI Cargo-Class Roadmap
|
||
|
||
> **Status: Phases 1–5 complete (v0.4.0).** Spatial audio, haptics,
|
||
> state-feeds, avatars, and volumetric/4D splats all shipped, promoted to
|
||
> `extensions/`, and live. This document is now the archived plan of
|
||
> record — see `CHANGELOG.md` for current state and `content-taxonomy.md`
|
||
> for what remains white space (e.g. music-as-score).
|
||
>
|
||
> The build plan for the white-space content classes in
|
||
> `content-taxonomy.md`. Every phase follows the **same proven recipe**
|
||
> the interactive-worlds extension established — so this is execution,
|
||
> not invention.
|
||
|
||
## The recipe (every cargo class follows it)
|
||
|
||
1. **Draft the extension spec** in `research/<class>-draft.md` — scope,
|
||
payload format, conformance criterion, fallback floor, receipt profile.
|
||
2. **Register the capability** in `wai-rs/src/codecs/mod.rs`
|
||
(`known_capabilities` + fidelity tier). Registration freezes the
|
||
payload format forever (SPEC §7) — so it lands *with* the drafted format.
|
||
3. **Reference implementation** in `wai-rs` behind a feature: the
|
||
container, the deterministic core, and the conformance criterion.
|
||
4. **Conformance corpus** + a `gen|verify` bin: vectors + expected
|
||
hashes; pass = hash equality, no tolerance parameter.
|
||
5. **Browser sink** in `wai-web` (wasm-bindgen) + a `/worlds`-style demo
|
||
page; prove it on real hardware via Playwright.
|
||
6. **JWP receipt profile** — what the §8 receipt carries for this class.
|
||
7. **Promote** the draft to `extensions/`, mirror to the site, update
|
||
SPEC §5/§8, CHANGELOG, deploy.
|
||
|
||
The invariants every class inherits for free: content-addressing,
|
||
fallback floor, Ed25519 seals, the JWP receipt chain, exact signed
|
||
`work_steps` + measured `joules_micro` (the native `wai_world_meter`
|
||
pattern), and a conformance criterion that is the class's own form of
|
||
*-equivalence*.
|
||
|
||
## Conformance criterion per class (the load-bearing design choice)
|
||
|
||
Each class needs a deterministic, hash-checkable equivalence — the
|
||
generalization of decode-equivalence:
|
||
|
||
| class | equivalence | the determinism floor |
|
||
|---|---|---|
|
||
| interactive worlds | **replay**-equivalence | `wai.det.fixed64` (shipped) |
|
||
| spatial audio | **mixdown**-equivalence | fixed-point PCM mix to a declared layout |
|
||
| haptics | **sample**-equivalence | fixed-point actuator samples at a declared rate |
|
||
| state-feeds | **replay**-equivalence | reuse the worlds op-log + state hash |
|
||
| digital humans | **pose**-equivalence | skeletal transforms in `wai.det.fixed64` |
|
||
| volumetric / 4D splats | **frame-state**-equivalence | per-frame Gaussian-cloud hash |
|
||
|
||
## Phase 0 — shipped (baseline)
|
||
|
||
Core media codecs (image/audio/video/text + neural family) · JWP
|
||
receipts · interactive-worlds (`wai.world.{scene,replay,live}`, three sim
|
||
registrants, corpus, browser sink, tab + cross-device relay multiplayer,
|
||
seal-to-replay, session receipts) · `wai.mesh.glb` + `wai.splat.spz`
|
||
assets · PBR + EWA-splat renderers · native energy meter. Siblings:
|
||
`map`, `joule-ui`, `Access`/`Lumina`.
|
||
|
||
---
|
||
|
||
## Phase 1 — Object-based spatial audio · `wai.audio.scene`
|
||
|
||
**Why first:** the cleanest technical analog to interactive-worlds.
|
||
Ship sound *objects* + a scene; the sink renders to its own speaker
|
||
layout. Mixdown-equivalence is the audio counterpart of
|
||
replay-equivalence, and it composes the asset path (each source is a
|
||
content-addressed `wai.audio.*` object) + receipts + determinism.
|
||
|
||
**Payload (`WSCN` container, sketch):**
|
||
- scene contract: sample rate, channel layout target(s), pan law id,
|
||
numeric profile (`wai.det.fixed64` for the exact mix).
|
||
- object table: N objects, each = `{ source_asset_hash (a wai.audio.*
|
||
WAI object), gain, trajectory (keyframed position over time) }`.
|
||
- a deterministic **render contract**: given the objects + a declared
|
||
output layout (`stereo`, `5.1`, `binaural@<hrtf-id>`), the mixed PCM
|
||
is bit-reproducible.
|
||
|
||
**Conformance:** mixdown-equivalence — same scene + same declared layout
|
||
→ identical BLAKE3 over the canonical PCM. Reference mixer: distance
|
||
attenuation + constant-power panning, fixed-point accumulate.
|
||
|
||
**Reference impl:** `wai-rs` `audio_scene` feature — scene container,
|
||
deterministic mixer to stereo (+ a binaural profile naming an HRTF as a
|
||
requirement-of-existence, like a neural prior). Sources decoded via the
|
||
existing audio path.
|
||
|
||
**Browser sink:** Web Audio API for the *nice* render (PannerNode/HRTF),
|
||
plus the deterministic reference mixer for the hash check — same
|
||
honest split as worlds (exact state vs. presentation).
|
||
|
||
**Effort:** medium (DSP: resample, pan, fixed-point mix). **Risk:** the
|
||
fixed-point mix must be exact across machines — same discipline as
|
||
`Fx`.
|
||
|
||
---
|
||
|
||
## Phase 2 — Haptics · `wai.haptic.signal`
|
||
|
||
**Why second:** lowest effort, pure instructions-at-sink, trivially
|
||
deterministic. A tiny event/waveform stream rendered on actuators.
|
||
|
||
**Payload:** a contract (sample rate, channel→actuator map) + either a
|
||
keyframed envelope or a fixed-point waveform per channel. Often a few
|
||
hundred bytes for an effect.
|
||
|
||
**Conformance:** sample-equivalence — the reconstructed actuator sample
|
||
buffer hashes identically.
|
||
|
||
**Reference impl:** `wai-rs` `haptic` feature — container + reconstruct
|
||
to a PCM-like buffer. **Browser sink:** Gamepad `vibrationActuator` /
|
||
the Vibration API; visualize the waveform on canvas as the fallback.
|
||
|
||
**Effort:** low. **Risk:** low. Good momentum builder after Phase 1.
|
||
|
||
---
|
||
|
||
## Phase 3 — Live state-feeds · broadcast profile + `wai.feed.state`
|
||
|
||
**Why:** the multiplayer-games precedent applied to broadcast — "ship
|
||
state, not pixels" for sports/events/telemetry. Most of it already
|
||
exists.
|
||
|
||
**Two pieces:**
|
||
1. **Broadcast profile of `wai.world.live`** — one authoritative
|
||
publisher, many read-only sinks, over the existing relay. A spectator
|
||
floor (an AV1 stream of a capable participant's render) per §9.
|
||
2. **`wai.feed.state`** — a *non-physics* state feed (scoreboard,
|
||
telemetry, market ticks): a signed op log over a declared schema with
|
||
the same canonical ordering + state hash, no sim. Effectively
|
||
interactive-worlds with a trivial "sim = apply the typed field
|
||
updates."
|
||
|
||
**Conformance:** replay-equivalence (reuse the worlds machinery).
|
||
|
||
**Effort:** medium (mostly profiling + relay broadcast mode). Reuses
|
||
relay, receipts, op log, state hash.
|
||
|
||
---
|
||
|
||
## Phase 4 — Digital humans / avatars · skinned mesh + skeletal animation
|
||
|
||
**Why:** highest-bandwidth capture, most reducible to rig + stream.
|
||
Composes shipped pieces.
|
||
|
||
**Pieces:**
|
||
- **Skinned-mesh asset**: `wai.mesh.glb` already carries skeletal
|
||
armatures + skin weights (glTF) — decode is sink-supplied.
|
||
- **Skeletal sim registrant** (`wai.world.sim.skeletal`): nodes are
|
||
bones; ops set joint transforms in `wai.det.fixed64`; the sink skins
|
||
the mesh by the deterministic pose.
|
||
- **Animation / expression op stream**: the existing CausalOp log drives
|
||
the rig — a telepresence stream is a live world of one avatar.
|
||
|
||
**Conformance:** pose-equivalence (bone transforms hash identically) —
|
||
a special case of replay-equivalence.
|
||
|
||
**Effort:** medium-high (skinning on the GPU; pose determinism).
|
||
|
||
---
|
||
|
||
## Phase 5 — Volumetric / light-field · `wai.splat.4d` (temporal Gaussians)
|
||
|
||
**Why last:** the frontier of "video"; no clean envelope exists. Extends
|
||
the splat path to time.
|
||
|
||
**Payload:** a base Gaussian cloud + per-frame deltas, OR per-Gaussian
|
||
trajectories (position/rotation/scale/opacity keyframes). Sink
|
||
interpolates and renders with the existing EWA pipeline.
|
||
|
||
**Conformance:** frame-state-equivalence — the Gaussian set at frame N
|
||
hashes identically (presentation render stays the renderer's business).
|
||
|
||
**Effort:** high. **Risk:** size (volumetric is heavy — the win is
|
||
content-addressing + dedup + receipts, *not* payload reduction; state
|
||
that honestly, as with static splats).
|
||
|
||
---
|
||
|
||
## Sequencing rationale
|
||
|
||
1. **Spatial audio** — cleanest analog, highest "proves the thesis
|
||
generalizes" value, composes the asset path.
|
||
2. **Haptics** — fast win, completes the "every sense" story (sight via
|
||
worlds/splats, hearing via audio-scene, touch via haptics).
|
||
3. **State-feeds** — biggest commercial wedge, mostly already built.
|
||
4. **Digital humans** — high value, composes worlds + mesh.
|
||
5. **Volumetric** — frontier, heaviest, do last.
|
||
|
||
Cross-cutting, anytime: surface **content-addressing + JWP receipts +
|
||
measured joules** on the captured-reality classes (photos, recordings,
|
||
telemetry) — the lever-4 win that needs no reconstruction and applies to
|
||
the content that can't be reduced to instructions.
|