open-standards/wai/research/content-taxonomy.md

# WAI Content Taxonomy — the bearings doc

> Living reference for *what web-transported content benefits from WAI,
> how, and which extension covers it.* Not normative; it's the map we
> plan cargo classes against. Edit as classes ship.

## 1. The reframe: a spectrum, not a list of media types

Everything transported over the web sits on one axis:

```
captured reality  ──►  synthesizable  ──►  executable  ──►  symbolic
   (samples)          (rendered from        (it IS a        (already
                       a description)        program)        instructions)
```

**WAI's leverage is proportional to how much of the content the sink can
reconstruct instead of receiving pre-rendered.** That's the whole map.
The discipline it enforces: do *not* claim "everything becomes
instructions" — a real photograph is irreducibly samples. Different
position on the axis → different lever → different size of win.

## 2. The four levers

| # | Lever | What it does | Where it applies |
|---|---|---|---|
| 1 | **Codec dispatch** | name the SOTA royalty-free codec; sink has it or falls back | all media |
| 2 | **Neural shared-prior** | ship conditioning, regenerate against a requirement-of-existence model | perceptual media a model can reconstruct |
| 3 | **Instructions-at-the-sink** | ship the recipe / state / intent; the sink renders | anything whose payload is secretly a rendered result |
| 4 | **Determinism + receipts** | replay/decode/mixdown-equivalence, content-addressing, JWP signatures, joules | **cross-cutting** — applies even to content that can't be reconstructed |

Levers 1–2 are the **compression** story. Lever 3 is the **medium-shift**
story (the Disney "price per byte → joules per experience" thesis).
Lever 4 is the **trust-and-energy** story, and it rides on top of all of
them.

## 3. The taxonomy (with mechanism + coverage)

Status: ✅ shipped · 🟡 partial/composable from shipped pieces · ⬜ white space.

### A. Captured reality — irreducible samples (levers 1, 2, 4)

| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| Photographs / images | codec dispatch + neural | `wai.image.*` (PNG/JXL/AVIF/JPEG, neural family) | ✅ |
| Recorded audio | codec dispatch + neural | `wai.audio.{flac,opus}`, `wai.neural.{dac,encodec32,…}` | ✅ |
| Recorded / cinematographic video | codec dispatch + neural | `wai.video.av1*`, `wai.neural.{dcvc_rt,dcvc_fm}` | ✅ |
| Live capture: conferencing, telephony, **radio**, broadcast | codec dispatch + receipts | core audio/video caps + JWP per object | 🟡 (transport out of scope; caps + receipts exist) |
| Sensor / telemetry / IoT | content-addressing + receipts | `wai.text.zstd` + JWP (or a `wai.feed.*` class) | 🟡 |
| Scientific captures: microscopy, medical imaging, LiDAR/point clouds, genomic | codec dispatch + content-addressing | core caps; point clouds → splat/volumetric path | 🟡 |
| Biosignals (EEG/ECG) | content-addressing + receipts | `wai.text.zstd` + JWP | 🟡 |

### B. Synthesizable media — shipped today as the render (lever 3)

| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| **Animation / CGI film** | instructions-at-sink | interactive-worlds (a film is a sealed, camera-on-rails world) | 🟡 (substrate exists; "linear film" profile unbuilt) |
| Motion graphics / titles / ads | instructions-at-sink | interactive-worlds / joule-ui | 🟡 |
| Music as score / synthesis | instructions-at-sink | ⬜ (a `wai.audio.score` / synth cargo) | ⬜ |
| Procedural / game audio, soundscapes | instructions-at-sink | spatial-audio (`wai.audio.scene`) | 🟡 Phase 1 |
| Text-to-speech | neural prior | ship text + a voice prior (`wai.neural.*` speech) | 🟡 |
| Data visualizations / charts | instructions-at-sink | ship data + spec (joule-ui) | 🟡 |
| Diagrams, vector art, **fonts** | instructions-at-sink | the precedents that already won the web; envelope only | ✅ (precedent) |
| **Maps / geospatial** | instructions-at-sink | `map` standard | ✅ (sibling) |

### C. Interactive / executable — it IS a program (levers 3, 4)

| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| **Videogames / interactive worlds** | instructions + determinism + receipts | interactive-worlds (`wai.world.*`) | ✅ |
| 3D scenes / models | content-addressed assets | `wai.mesh.glb` | ✅ |
| Gaussian splats | content-addressed assets | `wai.splat.spz` | ✅ |
| **AR / VR / XR** immersive scenes | instructions-at-sink | interactive-worlds + a sink pose/camera mode | 🟡 (substrate exists; XR sink mode unbuilt) |
| **Volumetric / light-field / holographic / 6DoF video** | instructions-at-sink | ⬜ temporal splats (`wai.splat.4d`) / `wai.video.volumetric` | ⬜ Phase 5 |
| **Avatars / digital humans** | instructions-at-sink | ⬜ skinned mesh asset + skeletal sim + animation ops | ⬜ Phase 4 |
| UIs / applications | instructions-at-sink + receipts | `joule-ui` standard | ✅ (sibling) |
| Documents-as-programs (interactive/reactive) | instructions-at-sink | joule-ui / interactive-worlds | 🟡 |
| Simulations / digital twins | instructions + determinism | interactive-worlds sim registrants; CAD/mathground lineage | 🟡 |

### D. Structured / symbolic — already instructions (levers 1, 4)

| Content | Primary lever | Capability / extension | Status |
|---|---|---|---|
| Text / markup / documents | codec + content-addressing | `wai.text.{zstd,xz}`; reading surface = `Access`/`Lumina` | ✅ |
| Code / software / packages | content-addressing + receipts | `wai.text.zstd` + JWP | 🟡 |
| Datasets / tabular / scientific data / knowledge graphs | content-addressing + receipts | `wai.text.*` + JWP; `JouleDB` lineage | 🟡 |
| Notifications / messages / event-&-state feeds | instructions + receipts | ⬜ `wai.feed.state` (broadcast profile) | ⬜ Phase 3 |

### The under-the-radar classes (flagged white space)

| Content | Why it matters | Plan |
|---|---|---|
| **Object-based / spatial audio** | the *exact audio analog* of the video thesis: ship sound objects + a scene, render to the listener's speaker config — the cleanest technical fit | **Phase 1** (`wai.audio.scene`) |
| **Haptics** | instruction streams to actuators; tiny payloads, pure recipe, deterministic | **Phase 2** (`wai.haptic.signal`) |
| **Live state-feeds** | sports/events as *state*, not pixels — the multiplayer-games precedent applied to broadcast; a top commercial wedge | **Phase 3** (broadcast profile of `wai.world.live` + `wai.feed.state`) |
| **Digital humans / avatars** | the highest-bandwidth capture today, the most reducible to rig + stream — "personal at broadcast scale" | **Phase 4** |
| **Volumetric / light fields** | where "video" goes next; no clean envelope exists for it yet | **Phase 5** |
| **Full-sensory / olfactory** | the holodeck endgame | on the axis; not scheduled |

## 4. The one sentence to navigate by

> **WAI carries whatever it can reconstruct, and verifies whatever it
> carries.** Reconstruction (instructions-at-sink, neural priors) is the
> compression story; verification (determinism, content-addressing,
> receipts, joules) is the trust-and-energy story — and the second
> applies even to the content that can't be reconstructed.

## 5. Reading the map

- The **biggest commercial wedges**: animation-film (a sealed
  camera-on-rails world) and live state-feeds.
- The **cleanest technical analog** to what's already proven:
  object-based spatial audio (mixdown-equivalence is the audio
  counterpart of replay-equivalence).
- The **cross-cutting win** that needs no reconstruction at all:
  content-addressing + JWP receipts + measured joules on *every* class,
  including raw captured reality.

See `cargo-roadmap.md` for the phased build plan.