46c5688b86
Commit the WAI standard source: a content + capability-dispatch standard
whose thesis is "carries whatever it can reconstruct, verifies whatever
it carries." Source only — large vendored/model assets and build output
are gitignored.
Reconstruct classes (each: spec + conformance corpus + in-tab browser
sink that re-derives the corpus hash, one wai.det.fixed64 fixed-point
floor, one *-equivalence apiece):
- interactive worlds (replay-equivalence) + 3 sim registrants,
multiplayer over a relay, session receipts, glTF + real SPZ assets
- spatial audio (mixdown-equivalence)
- haptics (sample-equivalence)
- state-feeds (replay-equivalence, non-physics sim)
- avatars / digital humans (pose-equivalence) + telepresence over the relay
- volumetric / 4D splats (frame-state-equivalence) + WebGPU EWA render,
real-SPZ-derived clip
Verify half (lever 4):
- asset provenance: a signed, energy-accounted lineage for opaque
captured content WAI does not reconstruct; tamper-evident, in-tab.
Cross-cutting: JWP receipt profiles per class; a native energy meter
(IOReport via macmon, no sudo) emitting receipts with REAL measured
joules; the learned-image/neural codec family and JPEG AI entropy port
from earlier work.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
7.6 KiB
7.6 KiB
WAI Content Taxonomy — the bearings doc
Living reference for what web-transported content benefits from WAI, how, and which extension covers it. Not normative; it's the map we plan cargo classes against. Edit as classes ship.
1. The reframe: a spectrum, not a list of media types
Everything transported over the web sits on one axis:
captured reality ──► synthesizable ──► executable ──► symbolic
(samples) (rendered from (it IS a (already
a description) program) instructions)
WAI's leverage is proportional to how much of the content the sink can reconstruct instead of receiving pre-rendered. That's the whole map. The discipline it enforces: do not claim "everything becomes instructions" — a real photograph is irreducibly samples. Different position on the axis → different lever → different size of win.
2. The four levers
| # | Lever | What it does | Where it applies |
|---|---|---|---|
| 1 | Codec dispatch | name the SOTA royalty-free codec; sink has it or falls back | all media |
| 2 | Neural shared-prior | ship conditioning, regenerate against a requirement-of-existence model | perceptual media a model can reconstruct |
| 3 | Instructions-at-the-sink | ship the recipe / state / intent; the sink renders | anything whose payload is secretly a rendered result |
| 4 | Determinism + receipts | replay/decode/mixdown-equivalence, content-addressing, JWP signatures, joules | cross-cutting — applies even to content that can't be reconstructed |
Levers 1–2 are the compression story. Lever 3 is the medium-shift story (the Disney "price per byte → joules per experience" thesis). Lever 4 is the trust-and-energy story, and it rides on top of all of them.
3. The taxonomy (with mechanism + coverage)
Status: ✅ shipped · 🟡 partial/composable from shipped pieces · ⬜ white space.
A. Captured reality — irreducible samples (levers 1, 2, 4)
| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| Photographs / images | codec dispatch + neural | wai.image.* (PNG/JXL/AVIF/JPEG, neural family) |
✅ |
| Recorded audio | codec dispatch + neural | wai.audio.{flac,opus}, wai.neural.{dac,encodec32,…} |
✅ |
| Recorded / cinematographic video | codec dispatch + neural | wai.video.av1*, wai.neural.{dcvc_rt,dcvc_fm} |
✅ |
| Live capture: conferencing, telephony, radio, broadcast | codec dispatch + receipts | core audio/video caps + JWP per object | 🟡 (transport out of scope; caps + receipts exist) |
| Sensor / telemetry / IoT | content-addressing + receipts | wai.text.zstd + JWP (or a wai.feed.* class) |
🟡 |
| Scientific captures: microscopy, medical imaging, LiDAR/point clouds, genomic | codec dispatch + content-addressing | core caps; point clouds → splat/volumetric path | 🟡 |
| Biosignals (EEG/ECG) | content-addressing + receipts | wai.text.zstd + JWP |
🟡 |
B. Synthesizable media — shipped today as the render (lever 3)
| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| Animation / CGI film | instructions-at-sink | interactive-worlds (a film is a sealed, camera-on-rails world) | 🟡 (substrate exists; "linear film" profile unbuilt) |
| Motion graphics / titles / ads | instructions-at-sink | interactive-worlds / joule-ui | 🟡 |
| Music as score / synthesis | instructions-at-sink | ⬜ (a wai.audio.score / synth cargo) |
⬜ |
| Procedural / game audio, soundscapes | instructions-at-sink | spatial-audio (wai.audio.scene) |
🟡 Phase 1 |
| Text-to-speech | neural prior | ship text + a voice prior (wai.neural.* speech) |
🟡 |
| Data visualizations / charts | instructions-at-sink | ship data + spec (joule-ui) | 🟡 |
| Diagrams, vector art, fonts | instructions-at-sink | the precedents that already won the web; envelope only | ✅ (precedent) |
| Maps / geospatial | instructions-at-sink | map standard |
✅ (sibling) |
C. Interactive / executable — it IS a program (levers 3, 4)
| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| Videogames / interactive worlds | instructions + determinism + receipts | interactive-worlds (wai.world.*) |
✅ |
| 3D scenes / models | content-addressed assets | wai.mesh.glb |
✅ |
| Gaussian splats | content-addressed assets | wai.splat.spz |
✅ |
| AR / VR / XR immersive scenes | instructions-at-sink | interactive-worlds + a sink pose/camera mode | 🟡 (substrate exists; XR sink mode unbuilt) |
| Volumetric / light-field / holographic / 6DoF video | instructions-at-sink | ⬜ temporal splats (wai.splat.4d) / wai.video.volumetric |
⬜ Phase 5 |
| Avatars / digital humans | instructions-at-sink | ⬜ skinned mesh asset + skeletal sim + animation ops | ⬜ Phase 4 |
| UIs / applications | instructions-at-sink + receipts | joule-ui standard |
✅ (sibling) |
| Documents-as-programs (interactive/reactive) | instructions-at-sink | joule-ui / interactive-worlds | 🟡 |
| Simulations / digital twins | instructions + determinism | interactive-worlds sim registrants; CAD/mathground lineage | 🟡 |
D. Structured / symbolic — already instructions (levers 1, 4)
| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| Text / markup / documents | codec + content-addressing | wai.text.{zstd,xz}; reading surface = Access/Lumina |
✅ |
| Code / software / packages | content-addressing + receipts | wai.text.zstd + JWP |
🟡 |
| Datasets / tabular / scientific data / knowledge graphs | content-addressing + receipts | wai.text.* + JWP; JouleDB lineage |
🟡 |
| Notifications / messages / event-&-state feeds | instructions + receipts | ⬜ wai.feed.state (broadcast profile) |
⬜ Phase 3 |
The under-the-radar classes (flagged white space)
| Content | Why it matters | Plan |
|---|---|---|
| Object-based / spatial audio | the exact audio analog of the video thesis: ship sound objects + a scene, render to the listener's speaker config — the cleanest technical fit | Phase 1 (wai.audio.scene) |
| Haptics | instruction streams to actuators; tiny payloads, pure recipe, deterministic | Phase 2 (wai.haptic.signal) |
| Live state-feeds | sports/events as state, not pixels — the multiplayer-games precedent applied to broadcast; a top commercial wedge | Phase 3 (broadcast profile of wai.world.live + wai.feed.state) |
| Digital humans / avatars | the highest-bandwidth capture today, the most reducible to rig + stream — "personal at broadcast scale" | Phase 4 |
| Volumetric / light fields | where "video" goes next; no clean envelope exists for it yet | Phase 5 |
| Full-sensory / olfactory | the holodeck endgame | on the axis; not scheduled |
4. The one sentence to navigate by
WAI carries whatever it can reconstruct, and verifies whatever it carries. Reconstruction (instructions-at-sink, neural priors) is the compression story; verification (determinism, content-addressing, receipts, joules) is the trust-and-energy story — and the second applies even to the content that can't be reconstructed.
5. Reading the map
- The biggest commercial wedges: animation-film (a sealed camera-on-rails world) and live state-feeds.
- The cleanest technical analog to what's already proven: object-based spatial audio (mixdown-equivalence is the audio counterpart of replay-equivalence).
- The cross-cutting win that needs no reconstruction at all: content-addressing + JWP receipts + measured joules on every class, including raw captured reality.
See cargo-roadmap.md for the phased build plan.