open-standards/wai/extensions/platform-decode.md

WAI Extension: Platform Decode Capabilities

Status: Draft. Maps two web platform interfaces, Media Capabilities decodingInfo() and WebCodecs isConfigSupported() with its hardware-acceleration preference, onto the capability report of browser-sink §3: what a sink asks the platform about an entry, read from the entry's payload (§2), and how the answers enter the report (§3). The answers are hints. They order the alternatives a page can decode, inside browser-sink's routing, by SPEC §4's energy order (browser-sink §4.7); they never change a route, are never an energy figure and are never evidence of hardware. No manifest member and no wire format. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

A sink running in a web platform can decode a standard video codec through the platform's own decoders, which may be fixed-function hardware, or through a decoder of its own: WebAssembly, or a learned reconstruction such as WAI's integer codecs. The platform says, for a configuration, whether it can decode it, whether it can do so without dropping frames, and whether it judges the decode power-efficient (Media Capabilities §2.2). A learned reconstruction runs on the sink's CPU, GPU or neural engine, and what it draws there is known only where it has been measured. This extension says how a WAI sink asks the platform about an entry, and how it enters what the platform says in browser-sink's capability report, where the routing (browser-sink §4) decides what the page can decode and the energy order (SPEC §4, Energy order; browser-sink §4.7) orders what it can.

It reads these documents:

• Media Capabilities, W3C Working Draft, 18 October 2026: MediaDecodingConfiguration, VideoConfiguration, MediaCapabilitiesInfo (§2.1, §2.2), the decoding-info algorithm (§2.3.2), and the privacy considerations (§3);
• WebCodecs, W3C Working Draft, 19 October 2026: VideoDecoderConfig (§7.6), isConfigSupported() and Check Configuration Support (§4, §7.1), and HardwareAcceleration (§7.9);
• the WebCodecs AV1 codec registration (Group Note Draft, 13 October 2026) and HEVC codec registration (Group Note Draft, 8 June 2026), which define the codec strings by the AV1 ISOBMFF binding (AV1 Codec ISO Media File Format Binding v1.3.0, §5) and by ISO/IEC 14496-15 Annex E.3.

ISO/IEC 14496-15 was not read. The HEVC codec string §2 writes follows the form the HEVC registration describes (four dot-separated fields of Annex E.3 and the constraint bytes after them), checked against an open-source parser of that form (Chromium's ParseHEVCCodecId) and against a platform that decodes with it (§5).

2. The query

For an entry whose capability the table maps, the sink forms the query from the entry's payload. It MUST NOT form it from anything the envelope declares apart from the payload: the codec string a payload's encoder declared can differ from the stream (§5 records one that did).

capability	codec string	size	frame rate	bit rate
wai.video.av1, wai.video.av1.lossless	from the first temporal unit's sequence header OBU: av01., the profile, the level (two digits) and the tier (M or H), the bit depth (two digits); then the monochrome flag, the chroma subsampling (three digits), the colour primaries, transfer characteristics and matrix coefficients (two digits each, 01 each where the sequence header carries no colour description), and the full-range flag; the part after the bit depth omitted where it is exactly .0.110.01.01.01.0 (AV1-ISOBMFF §5)	w × h of SPEC §5's AV1 framing	fps_num / fps_den of the framing	the payload's bits × the frame rate ÷ n, rounded up
wai.video.hevc	from the first SPS NAL unit (H.265 §7.3.2.2, §7.3.3): hev1.; the profile space as nothing, A, B or C, then general_profile_idc in decimal; the 32 general_profile_compatibility_flags as one hexadecimal value with flag j at bit j; L or H for general_tier_flag, then general_level_idc in decimal; then the six bytes from general_progressive_source_flag on, each in hexadecimal, trailing zero bytes omitted. hev1 because an Annex B stream carries its parameter sets in band	pic_width_in_luma_samples × pic_height_in_luma_samples, less the conformance window; the coded size before it	none in the payload: the rate the sink will present at, where it has one	the payload's bits × the frame rate ÷ the pictures the stream holds, rounded up

A capability the table does not map is not queried: the sink decodes it, or does not, as it otherwise would, and the order of §4 places it by its class alone.

The sink asks:

• navigator.mediaCapabilities.decodingInfo() with a MediaDecodingConfiguration of type "media-source" and a video member: contentType video/mp4; codecs="<codec string>", the size, the bit rate and the frame rate. Where the query has no frame rate it does not ask: framerate and bitrate are required members, and a sink MUST NOT invent them.
• VideoDecoder.isConfigSupported() with a VideoDecoderConfig of the codec string, the coded size as codedWidth and codedHeight, and the hardwareAcceleration the sink will configure, when it decodes the entry through WebCodecs.

WebCodecs gives no smoothness or power answer, and Media Capabilities has no WebCodecs decoding type. A sink that decodes through WebCodecs asks isConfigSupported() first, and asks decodingInfo() of the media-source query of the same configuration only once isConfigSupported() answers supported, and only for a codec string browser-sink §4.2's registry gate passes: Media Capabilities is a further question about a decoder already found (http-delivery §12.2). It reads smooth and powerEfficient from that answer as hints and nothing more (§3).

3. The answers

A page enters what the platform said in browser-sink §3's capability report, one answer per codec string, {supported, smooth, power_efficient, probe}, and in no other shape (platform_decode::report_answer):

• supported is isConfigSupported()'s answer for the preference the sink will configure, alone. A Media Capabilities supported is not read: a media-source answer is about the media-source pipeline, its container support included, not about the decoder WebCodecs found, so a decodingInfo() answer of supported: false overrides nothing. supported: false makes the user agent's leg unavailable for that object (browser-sink §4.3 step 4.5, ua_unsupported); the object then routes to a host or a wasm decoder where the page holds one, or nowhere. It is not a decode error and not a reason to try the decode anyway.
• smooth and power_efficient are decodingInfo()'s, entered only where supported is true, and null otherwise or where nothing was asked. They order the alternatives a page can decode, and only for an object routed ua (browser-sink §4.7): an object the page routes to a host or a wasm decoder, because the user agent's tier is below what the object needs (AV1's user-agent tier is none, browser-sink §4.1) or because no probe ran for a codec whose need is decode-equivalence (browser-sink §4.4), is not ranked on an answer about a decoder that will not run it. powerEfficient is the user agent's judgement that the decode's power draw is optimal, its definition left to the user agent, which SHOULD NOT mark hardware decoding power-efficient by default (Media Capabilities §2.2). It is a hint, not a measurement and not evidence of hardware. A sink MUST NOT write it, or anything derived from it, into a receipt, a measurement claim or an energy figure, and MUST NOT present it as an acquisition class (energy-measurement §2).
• probe is not_run: this extension runs no probe (browser-sink §4.4).

The hardware-acceleration preference. prefer-hardware and prefer-software are hints a user agent may ignore for any reason (WebCodecs §7.9). An isConfigSupported() answer of supported: true with prefer-hardware says the platform accepts that configuration with that preference; it is not evidence that the decode runs on hardware, and a sink MUST NOT report it as such.

Privacy. The answers can be used to fingerprint the sink (Media Capabilities §3.2). A sink SHOULD NOT send them, or the order they give, to any party; the order is computed where the answers are. A platform that fakes its capabilities to resist fingerprinting (§3.2 recommends a fixed set rather than always yes or always no) gives answers the order uses as given: the result is a choice that platform made, and the decode still conforms.

4. The order the answers feed

This extension defines no order of its own, and no second meaning of "the sink advertises cap". What a page advertises for an object is its route (browser-sink §4.3, SPEC §4). Among alternatives the page can route, SPEC §4's energy order applies, as browser-sink §4.7 writes it: first for classical decode routed ua that the platform answers supported, smooth and power-efficient, then smooth, then classical (any other routed classical decode, and a learned one with a qualifying sink-metered figure in browser-sink §6's energy entry), then learned-without-budget. A WAI1 envelope's capability and fallback are two such alternatives. The order reads no figure's value (browser-sink §6).

5. Conformance

The reference implementation is wai::platform_decode (wai/wai-rs/src/platform_decode.rs): query forms §2's query, and report_answer §3's answer. It calls no platform: the answers are inputs. The vectors are in wai/platform-decode-conformance/: queries.json, two floor objects of the packaging corpus, whose codec string is the one the packager wrote into its HLS master, and an HEVC SPS the platform's encoder wrote through WebCodecs, configured with prefer-hardware. The order's vectors are the choices of live-conformance/browser/.

An implementation conforms when it gives every query's members. The reference's tests (platform_decode::tests) also hold the AV1 codec string to the AV1-ISOBMFF binding's own examples, derive every floor of the packaging corpus to the codec string and size the packager wrote into the pack's HLS master, read constructed SPSs with sub-layers and with emulation-prevention bytes, and hold report_answer to §3.

wai-web's platform_decode bundle binds the same code for a page (platformQuery, platformAnswer, and platformChoose, browser-sink §4.7's choice), and wai-web/demo/platform_decode.html (verify_platform_decode.mjs) runs it against a browser's platform: every AV1 floor object of the packaging corpus, queried and decoded through WebCodecs with the configuration its payload's query gives; an HEVC stream the platform's encoder writes, queried from that stream and decoded back; and the choice between a learned alternative and an AV1 floor object, from the report the page built from those answers. Every query the page asks is read from a payload.

Measured 2026-10-22 with that page on an arm64 desktop (macOS 27.0.1), in headless Chromium 148 and a Chromium-based browser at 155, with the same results in both, before the page's order moved into browser-sink's routing:

• every AV1 floor object of the packaging corpus (64×36, av01.0.00M.08) decoded every temporal unit through WebCodecs. The platform answered supported and smooth, not powerEfficient, and did not accept prefer-hardware at that size;
• the platform's HEVC encoder, configured as hvc1.1.6.L93.B0 for 640×360 with prefer-hardware, wrote an SPS with general_level_idc 63, and the codec string it declared for its output repeated the configured one. The query read hev1.1.6.L63.B0 from the stream; the platform accepted that configuration with both preferences, answered supported, smooth and powerEfficient, and decoded every picture back with it.

The choice step of the page was rewritten with the order and has not been run in a browser in this revision; a 1920×1080 answer the earlier page reported was for a configuration written by hand, not read from a payload, and is not restated. Not measured: a platform without hardware AV1 or HEVC decode, a platform that fakes its answers, and any engine but Chromium's.

6. Security and privacy considerations

The answers are fingerprinting surface (§3, Privacy). The order reads them where they are produced and sends nothing. The query reads only the payload's headers; a malformed payload gives no query, and the entry is decoded, or not, as it would be without this extension. An encoder cannot raise an entry's rank by what it declares: the codec string is read from the payload, the class from the registry, the route from browser-sink's routing, and a learned entry ranks with classical decode only on a sink-metered figure the sink holds, never on one an envelope carries.