open-standards/wai/extensions/webcodecs-negotiation.md

# WAI Extension: Browser Capability Negotiation (WebCodecs)

> Status: Draft. How a **browser** sink implements the §4 dispatch algorithm's
> "if the sink advertises `cap`" test — using the platform's own capability-
> negotiation primitive (WebCodecs `isConfigSupported()`) for the native tier, and
> a WebAssembly decoder as the always-satisfiable fallback. Informative: it maps §4
> onto the browser; it does not change the wire format or the algorithm.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. The dispatch algorithm already is the browser's pattern

§4 resolves a capability by asking whether the sink *advertises* it, else falling
back. That is the exact shape of the browser's media-capability primitive:
WebCodecs `VideoDecoder.isConfigSupported()` / `AudioDecoder.isConfigSupported()`
(and `ImageDecoder`) answer, at runtime, whether a codec configuration will decode
on this device — and the established pattern when the answer is *no* is to fall
back to a WebAssembly decoder. A WAI browser sink is that pattern with a named
capability instead of a bare codec string, and a guaranteed WASM floor.

So a browser sink resolves §4 in **two tiers**:

- **Native tier** — the platform decodes the capability's payload with a built-in
  codec, gated by `isConfigSupported()`.
- **WASM tier** — WAI's own pure-integer decoders, compiled to wasm32, decode the
  payload with no platform codec and no ML runtime. This tier is *always
  satisfiable*: it depends on nothing the device must ship.

## 2. Native tier — `isConfigSupported()` as "advertises `cap`"

For the floor and recommended-menu capabilities (§5) that correspond to a codec
the browser may implement, a sink SHOULD treat "advertises `cap`" as "the platform
reports the codec supported." It builds the WebCodecs config for the capability's
payload and calls `isConfigSupported(config)`; a `supported: true` result means the
sink advertises that capability and decodes via the platform decoder.

| WAI capability | browser decode path (when `isConfigSupported` is true) |
|---|---|
| `wai.image.png` / `wai.image.jpeg` / `wai.image.jxl` / `wai.image.avif` | `ImageDecoder` (or `<img>`/`createImageBitmap`) with the format's MIME type |
| `wai.video.av1` / `wai.video.av1.lossless` | `VideoDecoder` with the `av01.*` codec string |
| `wai.audio.opus` / `wai.audio.flac` | `AudioDecoder` with the `opus` / `flac` codec string |

A `false` result (or a browser without WebCodecs) means the sink does **not**
advertise the capability at the native tier; resolution proceeds to the WASM tier,
then to §4's `fallback`, then to inert.

## 3. WASM tier — the always-satisfiable floor and the neural path

The neural and computed capabilities (`wai.image.jpegai`, `wai.neural.int_*`,
`wai.audio.int_codec`, `wai.splat.int_codec`, the `wai.world/film/quantum/...`
family) have **no** native browser codec by design — the decoder is *named, not
shipped*. For these the WASM tier is the decode path: WAI's pure-integer decoders
run in wasm32 exactly as they run natively. Because the integer decoders satisfy
`decode-equivalence` (see [determinism-tiers](determinism-tiers.md)), the WASM
decode is byte-identical to the native one — the reconstruction has a portable hash
in the browser too. The live `/jpegai` and `/int_hyper` sinks are this tier.

The WASM tier also backs the **classical floor** (§5): a sink MAY carry wasm
decoders for PNG / FLAC / zstd so that even a browser whose `isConfigSupported()`
declines every native codec can still open *some* file in each media class. This is
the browser realization of the floor's guarantee — always-decodable, depending on
nothing the platform must provide.

## 4. Resolution order in a browser sink

For `cap = model_requirement.capability`, a browser sink SHOULD resolve:

1. **Native**, if a WebCodecs mapping exists for `cap` and
   `isConfigSupported()` reports it supported → decode with the platform decoder.
2. **WASM**, if the sink bundles a wasm decoder for `cap` → decode in wasm.
3. **Fallback** `model_requirement.fallback`, resolved by the same two tiers.
4. Otherwise **inert at this sink** (§4 step 6).

This never contradicts §4 — it is §4's "advertises `cap`" made precise for the
browser: native when the platform reports it, WASM otherwise, and the classical
floor always reachable through wasm.

## 5. The native tier is already a constraint check

[`capability-constraints.md`](capability-constraints.md) makes dispatch a
precondition check: a sink satisfies a capability when it advertises `cap` **and**
the object's declared parameters satisfy one of the capability's registered
constraint sets. That extension landed after this mapping was written, and it
sharpens it rather than competing with it.

`isConfigSupported()` is not a codec-string test. It takes a full configuration —
for video, the codec string plus `codedWidth`, `codedHeight`, the description, and
the other members of `VideoDecoderConfig` — and answers for *that configuration*.
It is therefore a constraint evaluator already, and the browser satisfies the
extension's requirements structurally rather than by additional machinery:

- **Evaluated before dispatch.** `isConfigSupported()` is a pre-flight query; it
  does not construct or configure a decoder. This is the "move the failure earlier"
  requirement, met by the API's own shape.
- **Answered per object, not per sink.** A sink that decodes one object of a
  capability and refuses another is expressing exactly what a constraint set
  expresses. The parameters come from the object.
- **Unsupported, not a decode error.** A `supported: false` result is a negative
  answer to a question, returned before any payload is committed to — which is the
  required treatment where an object satisfies no set.

The result also carries a `config` back: the normalised configuration the platform
would accept. A browser sink **SHOULD** treat that returned configuration as the
accepted parameter set for the object, rather than re-deriving it.

**The honest limit.** WebCodecs' parameter vocabulary is the conventional-media
one — codec, resolution, frame rate, bit depth, profile. That is the same shape as
the ratified registers `capability-constraints.md` §3 draws on, and it has the same
gap: there is no WebCodecs configuration for the neural, quantum, world or haptic
families. Those capabilities have no native tier to constrain at all. They resolve
through the WASM tier (§3) and remain unconstrained until WAI authors their
register entries, which is the state §3 of that extension describes rather than a
shortfall of this mapping.

## 6. Framing

WAI's capability dispatch is not a bespoke negotiation scheme; it is the pattern
the browser's own media stack already uses — ask whether a config is supported,
fall back to WebAssembly when it is not. WAI adds a *named* capability and a
guaranteed WASM floor, so the fallback is always defined rather than
device-dependent. Stated as a property of the standard, not a comparison.