diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 864eb7ef7..55ec7222f 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -19,6 +19,21 @@ wire format may change and such changes are recorded here. the tier WAI adds on top and the precondition for a hard reconstruction binding. Referenced from SPEC §7 (Replicate-soundness); framed self-referentially. +### Browser capability negotiation — WebCodecs mapping + +- New [`extensions/webcodecs-negotiation.md`]: how a browser sink implements §4's + "advertises `cap`" test — the platform's `isConfigSupported()` for the native tier, + a WebAssembly decoder as the always-satisfiable fallback. Grounds WAI's dispatch + in the browser's own capability-negotiation pattern (ask if a config is supported; + fall back to wasm) and documents what the live wasm sinks already do: native codec + when the platform reports it, WAI's pure-integer wasm decoders otherwise, the + classical floor always reachable. Referenced from SPEC §4. Its §5 reconciles the + mapping with [`extensions/capability-constraints.md`], which landed later: + `isConfigSupported()` takes a whole `VideoDecoderConfig`, so the native tier is + already a per-object constraint check evaluated before dispatch — and it carries + the same register gap, since no WebCodecs configuration describes the neural, + quantum, world or haptic families. + ### Reproducible-reconstruction binding — a C2PA/JPEG-Trust hard binding over pixels - New `wai.*` provenance capability: `reconstruction_binding` (feature diff --git a/wai/SPEC.md b/wai/SPEC.md index e44dc8b9b..0988924c2 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -126,6 +126,13 @@ If the manifest declares `security.encryption` other than `"none"` (§6), the sink MUST recover the plaintext payload before step 3 and run this algorithm on the plaintext; see §6.4. +In a **browser** sink, "the sink advertises `cap`" (step 3) is answered by the +platform's own capability primitive — WebCodecs `isConfigSupported()` — with a +WebAssembly decoder as the always-satisfiable fallback tier (native when the +platform reports the codec supported, WASM otherwise; the classical floor is always +reachable through wasm). See +[`extensions/webcodecs-negotiation.md`](extensions/webcodecs-negotiation.md). + --- ## 5. Registered capabilities (the zeroth menu) diff --git a/wai/extensions/webcodecs-negotiation.md b/wai/extensions/webcodecs-negotiation.md new file mode 100644 index 000000000..2f4ed9833 --- /dev/null +++ b/wai/extensions/webcodecs-negotiation.md @@ -0,0 +1,120 @@ +# WAI Extension: Browser Capability Negotiation (WebCodecs) + +> Status: Draft. How a **browser** sink implements the §4 dispatch algorithm's +> "if the sink advertises `cap`" test — using the platform's own capability- +> negotiation primitive (WebCodecs `isConfigSupported()`) for the native tier, and +> a WebAssembly decoder as the always-satisfiable fallback. Informative: it maps §4 +> onto the browser; it does not change the wire format or the algorithm. +> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174. + +## 1. The dispatch algorithm already is the browser's pattern + +§4 resolves a capability by asking whether the sink *advertises* it, else falling +back. That is the exact shape of the browser's media-capability primitive: +WebCodecs `VideoDecoder.isConfigSupported()` / `AudioDecoder.isConfigSupported()` +(and `ImageDecoder`) answer, at runtime, whether a codec configuration will decode +on this device — and the established pattern when the answer is *no* is to fall +back to a WebAssembly decoder. A WAI browser sink is that pattern with a named +capability instead of a bare codec string, and a guaranteed WASM floor. + +So a browser sink resolves §4 in **two tiers**: + +- **Native tier** — the platform decodes the capability's payload with a built-in + codec, gated by `isConfigSupported()`. +- **WASM tier** — WAI's own pure-integer decoders, compiled to wasm32, decode the + payload with no platform codec and no ML runtime. This tier is *always + satisfiable*: it depends on nothing the device must ship. + +## 2. Native tier — `isConfigSupported()` as "advertises `cap`" + +For the floor and recommended-menu capabilities (§5) that correspond to a codec +the browser may implement, a sink SHOULD treat "advertises `cap`" as "the platform +reports the codec supported." It builds the WebCodecs config for the capability's +payload and calls `isConfigSupported(config)`; a `supported: true` result means the +sink advertises that capability and decodes via the platform decoder. + +| WAI capability | browser decode path (when `isConfigSupported` is true) | +|---|---| +| `wai.image.png` / `wai.image.jpeg` / `wai.image.jxl` / `wai.image.avif` | `ImageDecoder` (or ``/`createImageBitmap`) with the format's MIME type | +| `wai.video.av1` / `wai.video.av1.lossless` | `VideoDecoder` with the `av01.*` codec string | +| `wai.audio.opus` / `wai.audio.flac` | `AudioDecoder` with the `opus` / `flac` codec string | + +A `false` result (or a browser without WebCodecs) means the sink does **not** +advertise the capability at the native tier; resolution proceeds to the WASM tier, +then to §4's `fallback`, then to inert. + +## 3. WASM tier — the always-satisfiable floor and the neural path + +The neural and computed capabilities (`wai.image.jpegai`, `wai.neural.int_*`, +`wai.audio.int_codec`, `wai.splat.int_codec`, the `wai.world/film/quantum/...` +family) have **no** native browser codec by design — the decoder is *named, not +shipped*. For these the WASM tier is the decode path: WAI's pure-integer decoders +run in wasm32 exactly as they run natively. Because the integer decoders satisfy +`decode-equivalence` (see [determinism-tiers](determinism-tiers.md)), the WASM +decode is byte-identical to the native one — the reconstruction has a portable hash +in the browser too. The live `/jpegai` and `/int_hyper` sinks are this tier. + +The WASM tier also backs the **classical floor** (§5): a sink MAY carry wasm +decoders for PNG / FLAC / zstd so that even a browser whose `isConfigSupported()` +declines every native codec can still open *some* file in each media class. This is +the browser realization of the floor's guarantee — always-decodable, depending on +nothing the platform must provide. + +## 4. Resolution order in a browser sink + +For `cap = model_requirement.capability`, a browser sink SHOULD resolve: + +1. **Native**, if a WebCodecs mapping exists for `cap` and + `isConfigSupported()` reports it supported → decode with the platform decoder. +2. **WASM**, if the sink bundles a wasm decoder for `cap` → decode in wasm. +3. **Fallback** `model_requirement.fallback`, resolved by the same two tiers. +4. Otherwise **inert at this sink** (§4 step 6). + +This never contradicts §4 — it is §4's "advertises `cap`" made precise for the +browser: native when the platform reports it, WASM otherwise, and the classical +floor always reachable through wasm. + +## 5. The native tier is already a constraint check + +[`capability-constraints.md`](capability-constraints.md) makes dispatch a +precondition check: a sink satisfies a capability when it advertises `cap` **and** +the object's declared parameters satisfy one of the capability's registered +constraint sets. That extension landed after this mapping was written, and it +sharpens it rather than competing with it. + +`isConfigSupported()` is not a codec-string test. It takes a full configuration — +for video, the codec string plus `codedWidth`, `codedHeight`, the description, and +the other members of `VideoDecoderConfig` — and answers for *that configuration*. +It is therefore a constraint evaluator already, and the browser satisfies the +extension's requirements structurally rather than by additional machinery: + +- **Evaluated before dispatch.** `isConfigSupported()` is a pre-flight query; it + does not construct or configure a decoder. This is the "move the failure earlier" + requirement, met by the API's own shape. +- **Answered per object, not per sink.** A sink that decodes one object of a + capability and refuses another is expressing exactly what a constraint set + expresses. The parameters come from the object. +- **Unsupported, not a decode error.** A `supported: false` result is a negative + answer to a question, returned before any payload is committed to — which is the + required treatment where an object satisfies no set. + +The result also carries a `config` back: the normalised configuration the platform +would accept. A browser sink **SHOULD** treat that returned configuration as the +accepted parameter set for the object, rather than re-deriving it. + +**The honest limit.** WebCodecs' parameter vocabulary is the conventional-media +one — codec, resolution, frame rate, bit depth, profile. That is the same shape as +the ratified registers `capability-constraints.md` §3 draws on, and it has the same +gap: there is no WebCodecs configuration for the neural, quantum, world or haptic +families. Those capabilities have no native tier to constrain at all. They resolve +through the WASM tier (§3) and remain unconstrained until WAI authors their +register entries, which is the state §3 of that extension describes rather than a +shortfall of this mapping. + +## 6. Framing + +WAI's capability dispatch is not a bespoke negotiation scheme; it is the pattern +the browser's own media stack already uses — ask whether a config is supported, +fall back to WebAssembly when it is not. WAI adds a *named* capability and a +guaranteed WASM floor, so the fallback is always defined rather than +device-dependent. Stated as a property of the standard, not a comparison.