open-standards/wai/research/cargo-roadmap.md

WAI Cargo-Class Roadmap

Status: Phases 1–5 complete (v0.4.0). Spatial audio, haptics, state-feeds, avatars, and volumetric/4D splats all shipped, promoted to extensions/, and live. This document is now the archived plan of record — see CHANGELOG.md for current state and content-taxonomy.md for what remains white space (e.g. music-as-score).

The build plan for the white-space content classes in content-taxonomy.md. Every phase follows the same proven recipe the interactive-worlds extension established — so this is execution, not invention.

The recipe (every cargo class follows it)

1. Draft the extension spec in research/<class>-draft.md — scope, payload format, conformance criterion, fallback floor, receipt profile.
2. Register the capability in wai-rs/src/codecs/mod.rs (known_capabilities + fidelity tier). Registration freezes the payload format forever (SPEC §7) — so it lands with the drafted format.
3. Reference implementation in wai-rs behind a feature: the container, the deterministic core, and the conformance criterion.
4. Conformance corpus + a gen|verify bin: vectors + expected hashes; pass = hash equality, no tolerance parameter.
5. Browser sink in wai-web (wasm-bindgen) + a /worlds-style demo page; prove it on real hardware via Playwright.
6. JWP receipt profile — what the §8 receipt carries for this class.
7. Promote the draft to extensions/, mirror to the site, update SPEC §5/§8, CHANGELOG, deploy.

The invariants every class inherits for free: content-addressing, fallback floor, Ed25519 seals, the JWP receipt chain, exact signed work_steps + measured joules_micro (the native wai_world_meter pattern), and a conformance criterion that is the class's own form of -equivalence.

Conformance criterion per class (the load-bearing design choice)

Each class needs a deterministic, hash-checkable equivalence — the generalization of decode-equivalence:

class	equivalence	the determinism floor
interactive worlds	replay-equivalence	wai.det.fixed64 (shipped)
spatial audio	mixdown-equivalence	fixed-point PCM mix to a declared layout
haptics	sample-equivalence	fixed-point actuator samples at a declared rate
state-feeds	replay-equivalence	reuse the worlds op-log + state hash
digital humans	pose-equivalence	skeletal transforms in wai.det.fixed64
volumetric / 4D splats	frame-state-equivalence	per-frame Gaussian-cloud hash

Phase 0 — shipped (baseline)

Core media codecs (image/audio/video/text + neural family) · JWP receipts · interactive-worlds (wai.world.{scene,replay,live}, three sim registrants, corpus, browser sink, tab + cross-device relay multiplayer, seal-to-replay, session receipts) · wai.mesh.glb + wai.splat.spz assets · PBR + EWA-splat renderers · native energy meter. Siblings: map, joule-ui, Access/Lumina.

---

Phase 1 — Object-based spatial audio · wai.audio.scene

Why first: the cleanest technical analog to interactive-worlds. Ship sound objects + a scene; the sink renders to its own speaker layout. Mixdown-equivalence is the audio counterpart of replay-equivalence, and it composes the asset path (each source is a content-addressed wai.audio.* object) + receipts + determinism.

Payload (WSCN container, sketch):

• scene contract: sample rate, channel layout target(s), pan law id, numeric profile (wai.det.fixed64 for the exact mix).
• object table: N objects, each = { source_asset_hash (a wai.audio.* WAI object), gain, trajectory (keyframed position over time) }.
• a deterministic render contract: given the objects + a declared output layout (stereo, 5.1, binaural@<hrtf-id>), the mixed PCM is bit-reproducible.

Conformance: mixdown-equivalence — same scene + same declared layout → identical BLAKE3 over the canonical PCM. Reference mixer: distance attenuation + constant-power panning, fixed-point accumulate.

Reference impl: wai-rs audio_scene feature — scene container, deterministic mixer to stereo (+ a binaural profile naming an HRTF as a requirement-of-existence, like a neural prior). Sources decoded via the existing audio path.

Browser sink: Web Audio API for the nice render (PannerNode/HRTF), plus the deterministic reference mixer for the hash check — same honest split as worlds (exact state vs. presentation).

Effort: medium (DSP: resample, pan, fixed-point mix). Risk: the fixed-point mix must be exact across machines — same discipline as Fx.

---

Phase 2 — Haptics · wai.haptic.signal

Why second: lowest effort, pure instructions-at-sink, trivially deterministic. A tiny event/waveform stream rendered on actuators.

Payload: a contract (sample rate, channel→actuator map) + either a keyframed envelope or a fixed-point waveform per channel. Often a few hundred bytes for an effect.

Conformance: sample-equivalence — the reconstructed actuator sample buffer hashes identically.

Reference impl: wai-rs haptic feature — container + reconstruct to a PCM-like buffer. Browser sink: Gamepad vibrationActuator / the Vibration API; visualize the waveform on canvas as the fallback.

Effort: low. Risk: low. Good momentum builder after Phase 1.

---

Phase 3 — Live state-feeds · broadcast profile + wai.feed.state

Why: the multiplayer-games precedent applied to broadcast — "ship state, not pixels" for sports/events/telemetry. Most of it already exists.

Two pieces:

1. Broadcast profile of wai.world.live — one authoritative publisher, many read-only sinks, over the existing relay. A spectator floor (an AV1 stream of a capable participant's render) per §9.
2. wai.feed.state — a non-physics state feed (scoreboard, telemetry, market ticks): a signed op log over a declared schema with the same canonical ordering + state hash, no sim. Effectively interactive-worlds with a trivial "sim = apply the typed field updates."

Conformance: replay-equivalence (reuse the worlds machinery).

Effort: medium (mostly profiling + relay broadcast mode). Reuses relay, receipts, op log, state hash.

---

Phase 4 — Digital humans / avatars · skinned mesh + skeletal animation

Why: highest-bandwidth capture, most reducible to rig + stream. Composes shipped pieces.

Pieces:

• Skinned-mesh asset: wai.mesh.glb already carries skeletal armatures + skin weights (glTF) — decode is sink-supplied.
• Skeletal sim registrant (wai.world.sim.skeletal): nodes are bones; ops set joint transforms in wai.det.fixed64; the sink skins the mesh by the deterministic pose.
• Animation / expression op stream: the existing CausalOp log drives the rig — a telepresence stream is a live world of one avatar.

Conformance: pose-equivalence (bone transforms hash identically) — a special case of replay-equivalence.

Effort: medium-high (skinning on the GPU; pose determinism).

---

Phase 5 — Volumetric / light-field · wai.splat.4d (temporal Gaussians)

Why last: the frontier of "video"; no clean envelope exists. Extends the splat path to time.

Payload: a base Gaussian cloud + per-frame deltas, OR per-Gaussian trajectories (position/rotation/scale/opacity keyframes). Sink interpolates and renders with the existing EWA pipeline.

Conformance: frame-state-equivalence — the Gaussian set at frame N hashes identically (presentation render stays the renderer's business).

Effort: high. Risk: size (volumetric is heavy — the win is content-addressing + dedup + receipts, not payload reduction; state that honestly, as with static splats).

---

Sequencing rationale

1. Spatial audio — cleanest analog, highest "proves the thesis generalizes" value, composes the asset path.
2. Haptics — fast win, completes the "every sense" story (sight via worlds/splats, hearing via audio-scene, touch via haptics).
3. State-feeds — biggest commercial wedge, mostly already built.
4. Digital humans — high value, composes worlds + mesh.
5. Volumetric — frontier, heaviest, do last.

Cross-cutting, anytime: surface content-addressing + JWP receipts + measured joules on the captured-reality classes (photos, recordings, telemetry) — the lever-4 win that needs no reconstruction and applies to the content that can't be reduced to instructions.