open-standards/wai/extensions/browser-sink.md
Transaction Science f5a242a9a2
All checks were successful
joule-moq-remote host agent / joule-moq-remote · does this change touch it (push) Successful in 23s
joule-moq-room room service / joule-moq-room · does this change touch it (push) Successful in 21s
joule-moq-web browser client / joule-moq-web · does this change touch it (push) Successful in 22s
wai-app and Create / wai-app · does this change touch it (push) Successful in 31s
WAI operation catalogue / does this change touch it (push) Successful in 20s
wai-audit collector and player integration / does this change touch it (push) Successful in 30s
wai-sink-swift native sink package / does this change touch it (push) Successful in 18s
wai-quantum studio and desktop / does this change touch it (push) Successful in 14s
wai-ledger service / does this change touch it (push) Successful in 45s
wai-fleet service / does this change touch it (push) Successful in 1m1s
wai-sink-android native sink library / does this change touch it (push) Successful in 19s
wai-quantum for JavaScript / does this change touch it (push) Successful in 51s
WAI Studio / does this change touch it (push) Successful in 52s
joule-moq-remote host agent / joule-moq-remote, every target (linux-arm64) (push) Successful in 2m14s
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (push) Successful in 5m12s
wai-sink-android native sink library / the JNI layer and the platform-free half on the host JVM · linux-arm64 (push) Successful in 5m30s
joule-moq-room room service / joule-moq-room, every target (push) Successful in 7m13s
WAI breaking changes and the conformance runner / WAI breaking · does this change touch WAI (push) Successful in 27s
wai-app and Create / wai-app · tests, clippy, the line language's verifier (linux) (push) Successful in 41s
wai-audit collector and player integration / collector · linux (push) Successful in 2m30s
wai-ledger service / service · linux (push) Successful in 1m53s
WAI Studio / the catalogue's actions, the unit tests and the core · linux (push) Successful in 1m5s
wai-quantum studio and desktop / studio · browser (push) Successful in 4m38s
wai-quantum for JavaScript / javascript · linux (push) Successful in 3m51s
wai-fleet service / service · linux (push) Successful in 4m2s
WAI operation catalogue / the catalogue and every door · linux (push) Successful in 8m54s
wai-quantum studio and desktop / desktop app · linux (push) Successful in 11m31s
WAI Studio / Studio in WebKit · linux (push) Successful in 13m53s
wai-quantum studio and desktop / desktop app · windows (push) Successful in 9m58s
joule-moq-remote host agent / joule-moq-remote, every target (macos) (push) Successful in 1m1s
wai-audit collector and player integration / collector, player integration and the browser end to end · macos (push) Successful in 2m17s
wai-app and Create / wai-app · Create, browser end-to-end (macos) (push) Successful in 8m46s
wai-quantum studio and desktop / desktop app · macos (push) Successful in 5m29s
wai-ledger service / service · macos (push) Successful in 1m5s
joule-moq-web browser client / joule-moq-web · browser end-to-end (macos) (push) Successful in 17m5s
wai-fleet service / service · macos (push) Successful in 44s
wai-sink-android native sink library / every ABI, the AAR, and the instrumented tests in a headless emulator · macos (push) Successful in 8m34s
wai-sink-swift native sink package / libwai slices, desktop tests, simulator tests · macos (push) Successful in 18m3s
WAI Studio / Studio in Chromium and Firefox · macos (push) Successful in 12m15s
WAI breaking changes and the conformance runner / WAI breaking · wire, record and API against main · linux (push) Successful in 1m46s
WAI breaking changes and the conformance runner / WAI breaking and conform · their tests, and the independent verifiers through wai conform · linux (push) Successful in 9m10s
wai: header probes for AV2, AC-4, IAMF and IVAS, their codec strings, and why EVC has none
format-probes §14 to §17 add probes of basis `standard`, each read from its published text:

- AV2 (`wai.video.av2`, AV2 v1.0.0): the length-delimited bitstream of Annex B, the OBU
  headers, the temporal delimiter a payload begins with, and the first sequence header OBU
  through its cropping window: profile, level, tier, chroma format, bit depth, maximum and
  cropped size, layer counts, with Annex A's reserved profiles and levels and each profile's
  chroma formats refused. No codec string: AOMedia's ISO BMFF binding is a working draft.
- AC-4 (`wai.audio.ac4`, ETSI TS 103 190-1 V1.4.1 and -2 V1.3.1): sync frames with their CRC
  checked and the first table of contents to the first presentation's md_compat; the codec
  string of TS 103 190-2 E.13.
- IAMF (`wai.audio.iamf`, IAMF v2.0.0): a standalone IA Sequence, its profiles, the order of
  its Descriptors, the Opus, FLAC and LPCM configurations as §3.13 constrains them, the start of
  each audio element; the codec string of §6.4.
- IVAS (`wai.audio.ivas`, 3GPP TS 26.253 V19.3.0 Annex A): one IVAS payload's E and ToC bytes,
  frame sizes, processing information, padding and the first IVAS frame's format; `sivs`
  (TS 26.244 A.2.2).

§18 says why EVC has no probe: its framing, NAL unit types and SPS are in ISO/IEC 23094-1,
which is not openly readable, and RFC 9584 gives only the NAL unit header. §2.6 and
platform-decode §2.5: the probes' `codec`, which browser-sink §4.2's gate refuses, so no user
agent is asked; a native sink reads the string from the payload. SPEC §5 pins the payload forms
of AC-4, IAMF and IVAS to what their probes read. Nothing is added to the floor or the
recommended set.

format-probe-conformance: hand-built vectors with known field values under gen/av2, gen/ac4,
gen/iamf and gen/ivas, the texts' codec-string examples among them, and negative controls;
format_probe_verify.py reads them with its own readers and requires the new planted faults
(stated profiles, levels, versions, bit rates and codec strings, header bytes under restated
digests) to fail, as tests/format_probe.rs requires of corpus::verify.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-10 15:29:02 -05:00

69 KiB
Raw Permalink Blame History

WAI Extension: Browser Sink

Status: Draft. How a sink that runs in a web page answers SPEC §4's "the sink advertises cap", and what the frames it decodes can be checked against. For each object it is about to decode, the page routes the object's capability to the user agent's decoder, a native host's or WAI's own WebAssembly decoder (§4): the user agent is asked only about codec strings the WebCodecs Codec Registry lists, and its decoder is preferred where it is exact for the tier the object needs. Among objects that are alternatives of one content, the page decodes the one SPEC §4's energy order puts first, inside the routing: a decode routed to the user agent that it answers supported, smooth and power-efficient before a software one, and a learned one without a sink-metered figure of its decode last (§4.7). The page evaluates the constraint sets of capability-constraints itself (§5), and reports the joules a decode was measured to cost beside the user agent's powerEfficient boolean, which stays the user agent's word (§6). Every frame the page decodes carries a content binding (§7): the content hash of the receipted object it was decoded from, the link hash of the group receipt that object verified under, and the tier its samples hold. The binding's life through clone(), transfer, copyTo() and re-encode is in §8, its carriage in a page in §9, and the state of the frame on screen, which joins the binding to the receipts and to live-provenance's verdict, in §10. The member a frame carries, contentBinding, is proposed for the WebCodecs VideoFrame Metadata Registry in proposals/webcodecs-content-binding.md, a draft that has not been filed; until a registry lists the member, a page carries it in a side table (§9.2). No transport, no record, no change to the wire format or to SPEC §4's algorithm. The reference implementation is the browser_sink module of wai-rs (feature browser_sink), which wai/wai-live-web builds for the page; the page's own code is joule-code/joule-code-rs/crates/joule-moq-web/web/room/binding.js; the corpus is live-conformance/browser/, and its second implementation is live-conformance/browser_sink_verify.py. Each takes a wasm frame's decoder to be a module the page holds at the integer level of the WAISM™ profile (§4.3 step 6, §12). Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision, each pinned to the version named: WebCodecs, W3C Working Draft 19 October 2026: the VideoDecoder's Output VideoFrames steps, Create a VideoFrame, the VideoFrameMetadata dictionary and metadata(), Copy VideoFrame metadata, Clone VideoFrame, the constructors' metadata steps, the serialization and transfer steps, copyTo() and EncodedVideoChunk; the WebCodecs Codec Registry, W3C Registry Draft 24 September 2026, its video and audio tables; the WebCodecs VideoFrame Metadata Registry, W3C Registry Draft 14 April 2026, its entries and its registration requirements; and Media Capabilities, W3C Working Draft 18 October 2026, MediaCapabilitiesInfo (supported, smooth, powerEfficient) and its privacy considerations. A section number in this text is this extension's unless it names another document.

1. Scope

In scope: the capability report a page builds (§3); routing an object to a decoder (§4), and choosing among alternatives (§4.7); constraint sets evaluated in the page (§5); the energy entry beside powerEfficient (§6); the content binding, its form and when a page writes one (§7); what happens to it through a frame's life (§8); how a page carries it (§9); the state of the frame on screen (§10); the reasons (§11); and a page that composes a package of several components on one timeline (§14).

Out of scope: a transport; the user agent's decoders and whether they conform to their standards, which a page cannot see beyond the answers they give and a probe (§4.4); the decoders of a native host; how a meter measures (energy-measurement §3); and what a party does with a state.

Relationship to the core spec and the other extensions:

  • SPEC §4. A page answers step 3's "the sink advertises cap" for an object with its route (§4): it advertises cap for that object when the route is not none. Among alternatives it applies SPEC §4's energy order (§4.7).
  • http-delivery §4.3 and §12. A WAI element is never put to a user-agent interface (§4.3 there). The route gives the Sink Capability Descriptor's via (§4.6).
  • capability-constraints. §5 is its evaluation in a page, over the register of its §3.1.
  • energy-measurement §2. The classes a figure carries (§6).
  • platform-decode §2, §3. The query a page asks the platform, read from the payload, and how the answers enter the report.
  • jwp-receipts §2.1 and §3 step 3. The content hash a binding names, and the link hash of the group receipt.
  • determinism-tiers. The tier vocabulary a need and a binding use.
  • live-provenance §11. The verdict on the segment a frame's object belongs to, for a CMAF track (§10).

2. Terms

  • Page: a sink running in a web page.

  • UA decoder: a decoder of the user agent: WebCodecs VideoDecoder, AudioDecoder or ImageDecoder. Wasm decoder: a WAI decoder compiled to WebAssembly that the page loads. Host decoder: a decoder of a native host the page hands a payload to (http-delivery §12.1, host).

  • Route: ua, host, wasm or none.

  • Receipted object: the bytes a JWP content hash covers (jwp-receipts §2.1): a WAI envelope; in an end-to-end encrypted room, the sealed object, over which the receipt is signed (confidentiality); in a CMAF track, a chunk (live-provenance §8.1).

  • Tier: decode-equivalence, entropy-consistency or none (determinism-tiers §2), ordered none < entropy-consistency < decode-equivalence. A token outside this vocabulary is read as none, as a pin's is (determinism-tiers §3).

  • Registered tier of a capability: the tier its decoded output holds on every conforming sink, from its replicate class (SPEC §7):

    replicate class registered tier
    StandardDefined, NeuralIntegerExact decode-equivalence
    NeuralEntropyExact entropy-consistency
    every other class, and a string this revision does not register none

    StandardDefined is read as decode-equivalence because SPEC §7 has its decoded output the same bytes on every conforming sink, which is the property that tier names; determinism-tiers defines the tiers for neural capabilities, and this extension uses the same tokens for that property and claims nothing more. GenerativeExact output also reproduces, but it is generated rather than decoded, so it is held at none here.

  • Need: the tier the page holds an object's decoded output to: the object's pin tier (SPEC §3.1) where it has one, else its track's wai_determinism (moq-streaming-format §4), else none.

  • Frame: a WebCodecs VideoFrame. Binding: a contentBinding value (§7).

Hexadecimal is lowercase. JSON in this text is RFC 8259.

3. The capability report

A page states what it knows about its decoders, and what it is about to decode, in a capability report: one JSON object.

{ "browser_sink": 1,
  "ua": { "<codec string>": { "supported": true, "smooth": true, "power_efficient": false,
                              "probe": "match" | "mismatch" | "not_run" } },
  "host": { "<capability>": "<tier>" },
  "wasm": [ "<capability>" ],
  "constraints": { "<capability>": [ { "<parameter>": { "enum": [ ], "minimum": 0, "maximum": 0 } } ] },
  "energy": { "<capability> <route>": { "joules_micro": 0, "decodes": 0, "window_us": 0,
                                        "class": "<class>", "relative_ppm": 0,
                                        "filtering": "On" | "Off" | "Unknown",
                                        "calibration_ref": "<text>",
                                        "coverage": "Whole" | "Partial", "fabrics": [ "cpu" ] } },
  "objects": [ { "name": "<text>", "capability": "<capability>", "codec": "<codec string>" | null,
                 "need": "<tier>", "params": { "<parameter>": 0 }, "choice": "<text>" } ] }
  • browser_sink is 1.
  • ua holds the user agent's answers, by codec string, video and audio alike: supported, from isConfigSupported (VideoDecoder's or AudioDecoder's) for the configuration the page built from the object (§4.6), read from its payload (platform-decode §2); smooth and power_efficient, from Media Capabilities' smooth and powerEfficient for the same configuration, asked with a video or an audio member, or null where the page did not ask; probe, from §4.4. Where the page asked about several configurations under one codec string, the answer is their merge, as platform-decode §3 has it: supported only where each was, and a hint true only where each one's was.
  • host names the capabilities a native host decodes for the page, each with the tier its decode holds, as the host declares it.
  • wasm lists the capabilities the page's wasm decoders decode. A wasm decoder of an integer neural capability may run its wai-ops/1-i32 graphs (integer-operators §9.1) on the page's WebGPU, through wai/wai-ops-gpu's WGSL kernels compiled by the page's own WGSL compiler (the gpu unit on the wgsl backend, an integer path in neural-dispatch §1), once the adapter has given the host's result for every word of the arithmetic probe and WGSL's result for every guarded case of the integer-semantics suite in the page. Its output is then the decode's on the page's CPU, byte for byte, so the capability's tier is unchanged; every graph the device does not run, and every device error, runs on the page's CPU, which is also the decoder where no adapter passes. A capability that plays is listed only as §4.6 has it.
  • constraints holds the constraint sets the page's decoders publish, by capability (§5). A capability with no entry is unconstrained.
  • energy holds measurements, by capability and route (§6).
  • objects lists what the page is about to decode, each object (or each track, whose objects share these values) with a name, its capability, the codec string a UA decoder would be asked under (null where none applies), its need, and its declared parameters (§5); and, optionally, a choice, a non-empty string that objects which are alternatives of one content share (§4.7).

A report that does not read in this shape, or whose browser_sink is not 1, is refused whole (report_malformed): every member above is required but an object's choice, which, when present, is a non-empty string; objects may be empty, and a member this text does not define is ignored. A codec string with an answer in ua that §4.2's gate refuses is never read (§4.2).

4. Routing

4.1 The user agent's decoders

A capability is put to a UA decoder only through a row of this table:

capability UA decoder codec string UA tier
wai.video.vp9 VideoDecoder vp09. and more decode-equivalence
wai.video.avc VideoDecoder avc1. or avc3. and more decode-equivalence
wai.video.hevc VideoDecoder hev1. or hvc1. and more decode-equivalence
wai.video.av1, wai.video.av1.lossless VideoDecoder av01. and more none
wai.video.vp8 VideoDecoder vp8 none
wai.audio.opus, wai.audio.opus.packet AudioDecoder opus none
wai.audio.flac AudioDecoder flac none
wai.audio.aac AudioDecoder mp4a. and more none

The UA tier is what a UA decoder's output holds when the decoder conforms to its standard:

  • decode-equivalence for the video codecs whose decoder conformance requires every output sample to equal the decoding process's (SPEC §7, StandardDefined), over the samples of the frame the decoder outputs, in the format it outputs them in (§8);
  • none for AV1, whose film-grain synthesis is held to a perceptual criterion, for VP8, whose conformance this revision has not read, and for Opus and AAC (SPEC §7); and for FLAC, whose decoded samples are exact but which an AudioDecoder hands back in a sample format the user agent chooses, a conversion no standard fixes (SPEC §7, What replicate promises over a standard codec).

AV1's row stays at none because a codec string does not say whether a stream signals film grain. A stream that signals none decodes to the decoding process's frames on a conforming decoder (AV1 §7.2), and a page that has shown its decoder conforming by a probe (§4.4) may use the frames of a grain-free stream as the base of a residual refinement (platform-decode §6): staged delivery holds that base step at decode-equivalence (staged-delivery §5). That changes no route and no row of this table.

A capability with no row has no UA decoder: wai.video.av2, wai.video.vvc, wai.video.evc, wai.audio.iamf, wai.audio.ac4 and wai.audio.ivas among the registered ones, since the registry of §4.2 lists no codec string for them (the strings their header probes read are platform-decode §2.5's), and every WAI-native capability, whose decoder is WAI's own. ImageDecoder has no row in this revision: an image decode in a user agent may apply colour conversion, and the user agent does not say whether it did.

4.2 The registry gate

The WebCodecs Codec Registry (W3C Registry Draft, 24 September 2026) lists these codec strings:

media rows
video av01.*, avc1.*, avc3.*, hev1.*, hvc1.*, vp8, vp09.*
audio flac, mp3, mp4a.*, opus, vorbis, ulaw, alaw, pcm-*

A codec string matches a row equal to it; a row ending .* or -* is matched by a string that begins with the row up to and including its . or - and has at least one character after it. A codec string that matches no row:

  • MUST NOT be passed to isConfigSupported, to Media Capabilities or to any other user-agent interface by a page that routes by this extension;
  • is never answerable: an answer for it in a report is not read, whatever it says, and the object's UA leg is refused codec_unregistered.

The registry is how WebCodecs names a codec. Its abstract says no implementer is required to support any entry; a string outside it names nothing the registry defines, so two user agents' answers about it need not be about the same codec, and a page has no basis to read one. Such a capability resolves through a host decoder or a wasm decoder, or not at all. A row the registry adds later enters this table by a revision of this extension.

4.3 The steps

For each object o of a report, in order, the page computes its route:

  1. Kind. If o.capability is not dispatchable (SPEC §4: a record, a derivation, a candidate, a companion-only or a refinement-only capability), the route is none (not_dispatchable).
  2. Need. If o.need is above the capability's registered tier, the route is none (tier_exceeds_class, determinism-tiers §3).
  3. Constraints. §5 over the report's sets for o.capability and o.params: if they cannot be evaluated the route is none (constraints_unevaluable), and if no set is satisfied it is none (constraints_unmet) — the capability is unsupported for this object, which is not a decode error (capability-constraints §2).
  4. The UA leg. It is taken when every check holds; otherwise its reason is the first that fails:
    1. o.codec is a string (no_ua_mapping: the object names nothing to ask a UA decoder);
    2. o.codec passes §4.2's gate (codec_unregistered), whatever the capability;
    3. o.capability has a row in §4.1 (no_ua_mapping);
    4. o.codec is a string of that row's form (codec_mismatch);
    5. the report holds an answer for o.codec (ua_not_asked), and its supported is true (ua_unsupported);
    6. the need is at most the row's UA tier (ua_inexact);
    7. where the need is decode-equivalence, the answer's probe is match (ua_unprobed for not_run, ua_probe_mismatch for mismatch).
  5. The host leg. It is taken when o.capability is in host (no_host) with a tier at least the need (host_inexact).
  6. The wasm leg. It is taken when o.capability is in wasm (no_wasm). The route's tier is the registered tier, which step 2 has already held to be at least the need. A wasm decoder's output holds that tier only where the decoder is a module the page holds at the integer level of the WAISM profile (waism §6), authorised under jwp-receipts §7.9 and checked before the page instantiated it; the frames of any other wasm decoder are bound at none (§7.2).
  7. Route. The route is the first leg taken, in the order UA, host, wasm. Where none is taken the route is none (no_decoder), and each leg's reason is reported.

The UA leg comes first because a decoder the device ships is the one a page is expected to use, and checks 4.6 and 4.7 take it only where its output is exact for what the object needs; the wasm decoder is the fallback that depends on nothing the device must provide. A page MUST decode an object only through its route, and MUST NOT decode one whose route is none.

Each object's entry in the routing table states: its name, its route, its reason (null unless the route is none), each leg's outcome ("taken", "not_reached" for a leg after the one taken and for every leg when steps 1 to 3 decide, or the leg's reason), asked (whether the page may put o.codec to the user agent: true exactly when steps 1 to 3 pass and checks 4.1 to 4.4 hold), the route's tier, and the energy entry of §6. The entry's reason is the code of the step that decided, or no_decoder from step 7. The route's tier is the row's UA tier for ua (lowered to none unless the probe matched), the host's declared tier for host, the registered tier for wasm, and none for none; in each case no higher than the registered tier.

4.4 The probe

An answer of supported says a UA decoder accepts the configuration; it does not say the decoder's output is the decoding process's. Where an object needs decode-equivalence, the page decodes a probe: an object of the same capability and configuration whose decoded frames' SHA-256 digests are published (http-delivery §12.4's probe object), each over the frame's samples as copyTo() writes them with no format, rect or layout option (planes in order, rows tightly packed). A frame output in NV12 is compared after its interleaved chroma plane is split into its two planes, which reorders samples and changes none. The probe is match when every frame's digest is the published one, mismatch when any is not, and not_run when the page decoded no probe. This revision publishes no probe for routing, so a page reports not_run and a need of decode-equivalence is never routed to a UA decoder. It publishes one probe of another use: a grain-free AV1 stream whose frames' digests are in platform-decode-conformance/grain-free-av1/probe.json, by whose outcome a page takes its decoder's frames of a grain-free stream of that configuration as a refinement base or does not (platform-decode §6.2).

4.5 The user agent's answer is a constraint check too

isConfigSupported answers for a whole configuration, before any decoder is configured, and returns the configuration it would accept. It already does, for the parameters a configuration carries, what capability-constraints asks of a sink: an answer before dispatch, for one object, that is "unsupported" rather than a decode error. A page SHOULD build the configuration from the object's own declared parameters, and SHOULD configure the decoder with the configuration the answer returns. The two checks are separate and both apply: an object routes ua only when its constraint sets are satisfied (step 3) and the user agent answered supported (check 4.5). Neither covers the neural, quantum, world or haptic families, which have no UA decoder; their constraints are evaluated by §5 alone.

4.6 What a page asks, and what it advertises

A page builds a report in two passes: from the objects alone it computes asked for each, and puts to the user agent exactly the codec strings an entry marks asked; then it adds the answers and computes the routes. Media Capabilities is asked only about a codec string the user agent answered supported (http-delivery §12.2). A capability routed ua, wasm or host is, for http-delivery §12.1's descriptor, via "ua" (with the codec strings asked and their answers, provisional true), "wasm" or "host"; one routed none is absent from it.

A capability whose objects play to a deadline, such as wai.video.int_hyper, is in wasm only where the page's decoder keeps up with the object on this device, from the decode times the page measured on the path it decodes with (on the page's CPU, or with its graphs on WebGPU as §3 has it): a group of n frames at frame_rate_milli frames per 1000 s, its keyframe and the frames after it, decodes in no more microseconds, each time rounded up to a whole microsecond, than it plays, n · 10⁹ / frame_rate_milli. A time that is not finite or is negative, and a group of no frames or no rate, never keeps up. A capability whose objects are stills, such as wai.neural.int_hyper and wai.neural.int_synth, is listed wherever the page decodes it. A capability left out is routed none, its wasm leg no_wasm. One listed is routed wasm, and §4.7 ranks it as it ranks any learned capability: after a classical alternative unless the report holds a sink-metered figure of its decode. So a piece whose floor this device can route plays the floor; a piece whose only alternative is the clip plays it where the decoder keeps up, and nothing where it does not (no_alternative). The reference is wai/wai-web's neural_webgpu bundle: IntNeural decodes on either path, and its capabilities and route give the leg and the report routed and chosen with it (src/neural_leg.rs).

4.7 Choosing among alternatives

Objects of a report that carry the same choice are alternatives: renditions of one content of which the page decodes one, such as the entries of a WAI2 envelope, a WAI1 envelope's capability and fallback, or the renditions of one track. For each choice, in order of its first appearance in objects, the page puts the alternatives in SPEC §4's energy order. The order sits inside the routing: each alternative's route is the one §4.3 gives it, and the order changes none.

  • An alternative routed ua ranks by the user agent's answer for its codec string (platform-decode §3): first when supported, smooth and power_efficient are all true; smooth when smooth is true and power_efficient is not; classical otherwise. A false or a null is not a true: no answer is not a power-efficient one.
  • An alternative routed host or wasm ranks classical. The user agent's answers are about its own decoder, which does not run this object, so they are not read for it, whatever they say: AV1, whose user-agent tier is none, and a codec whose need is decode-equivalence while no probe ran (§4.4) route elsewhere and rank on nothing the user agent said.
  • An alternative whose capability is learned (SPEC §4: its replicate class is not one of the three standard classes, or the string is not registered) ranks classical when the report holds a measurement under its route's key (§6, "<capability> <route>") that reads under §6 with the class OnChipCounter or CalibratedInstrument and the coverage Whole: a sink-metered figure of its decode, with its class, uncertainty and scope. It ranks learned-without-budget otherwise. The figure's value is not read.
  • An alternative routed none comes after every other, with no rank.

Report order within a rank. powerEfficient is the user agent's hint, used as the user agent gave it, as an order and never as an energy figure or as evidence of hardware. The chosen alternative is the first in that order whose route is not none, and where every route is none nothing is chosen (no_alternative). A page MUST decode the chosen alternative of a choice and no other, so a learned alternative without a qualifying measurement is decoded only where no classical alternative can be routed. For a WAI1 envelope this is SPEC §4's order of steps 3 and 5: the chosen alternative is tried first, and the other after it if the first does not decode.

The choice table has one entry a choice: choice; order, the alternatives' names in that order, those routed none included; ranks, each one's rank in the same order, null for one routed none; chosen, a name or null; and reason, null or no_alternative. An object with no choice is in no entry, and a report whose objects carry none has an empty table.

5. Constraint sets in the page

A page evaluates capability-constraints §2 in this form:

  • A capability's sets are a JSON array of one or more constraint sets. A set is a JSON object whose members are parameter names, each mapped to a constraint: a JSON object with one or more of enum (a non-empty array of values), minimum and maximum (integers), and no other member. A value is a JSON string, or an integer: a JSON number with no fraction and no exponent, from −(2^53 − 1) to 2^53 − 1. A page whose own numbers do not keep 1920.0 apart from 1920 writes both as 1920, so the integers it writes are exactly its integral values; a value with a fractional part stays a fraction.
  • A parameter name MUST be one of the register of capability-constraints §3.1.
  • An object's declared parameters are a JSON object of the same names and values.

evaluate(sets, params) is:

  1. No sets for the capability: unconstrained.
  2. The sets break the form above — not an array, an empty array, a set that is not an object, a constraint that is not an object, a constraint with no member or with a member other than the three, an empty enum or one holding something other than a value, a minimum or maximum that is not an integer, or a minimum above a maximum: unevaluable (constraint_malformed).
  3. A set names a parameter outside the register: unevaluable (parameter_unregistered). A page MUST NOT advertise a capability whose sets it cannot evaluate (capability-constraints §2).
  4. A set is satisfied when, for each parameter it names, the object declares a value for it and the value meets each member of the constraint: for enum, it equals one of the values, a string only a string and an integer only an integer; for minimum and maximum, it is an integer at least, or at most, the bound. An undeclared parameter, or a declared value that is neither a string nor an integer, meets no constraint. An empty set is satisfied by every object.
  5. satisfied when one set is; otherwise unsatisfied (constraints_unmet).

The grammar is the one capability-constraints adopts, enumerations, minima and maxima ORed over sets, with integers in place of rationals, so every evaluation is exact.

6. Energy beside powerEfficient

Each entry of the routing table carries an energy entry:

{ "power_efficient": true | false | null, "figure": null | { … }, "reason": null | "<reason>" }
  • power_efficient is Media Capabilities' powerEfficient for the object's codec string, copied as the user agent gave it, for an object routed ua; it is null for any other route and where the page did not ask. It is the user agent's word: its own specification leaves the meaning of optimal power draw to the user agent. A page MUST NOT set it, invert it or compute it from a figure, MUST NOT state it for a route the user agent did not answer for, and MUST NOT derive a figure from it.
  • figure is the measurement the report holds under the key "<capability> <route>", where <route> is the object's route, and null when it holds none or the route is none. A measurement under another route's key is not this route's figure. A page MAY take a measurement from its sink's decode table (sink-energy §4): a row whose status is figure gives one, as that extension's §1 maps it, and a row of any other status gives none.

A measurement is read as energy-measurement §2 has it, and refused, with figure null and the entry's reason, when:

  • its class is not one of CalibratedInstrument, OnChipCounter, ModelBased and Estimator (energy_class_invalid; HwShunt is never emitted, energy-measurement §2.6);
  • joules_micro is 0 (unmetered_as_zero): an unmetered decode has no measurement, and a class beside a zero claims a measurement that did not occur;
  • joules_micro, decodes or window_us is not a positive integer below 2^53 (energy_count_invalid);
  • a hardware class lacks its evidence: relative_ppm, a positive integer, for CalibratedInstrument and OnChipCounter; filtering, one of On, Off and Unknown, for OnChipCounter; calibration_ref, a non-empty string, for CalibratedInstrument (energy_evidence_missing);
  • coverage is not Whole or Partial, or fabrics is not a non-empty array of distinct strings (energy_evidence_missing).

A figure that reads is reported with every member it was given and per_decode_micro = ⌊joules_micro / decodes⌋; joules_micro and decodes stay beside it, so the floor loses nothing. An object with no measurement is unmetered: figure and reason are both null, and a page presents it as unmetered, never as 0 J. In this revision a figure never changes a route, and none is steered on: §4.7 reads only whether a learned alternative's figure qualifies, never its value. How a figure is measured, bracketed and attributed is energy-measurement §3 and §4's, and a Partial coverage says the meter saw only the fabrics it names.

7. The content binding

7.1 The value

A content binding is a JSON object, and in a page a dictionary of the same members, the contentBinding member of a frame's metadata: proposed for the WebCodecs VideoFrame Metadata Registry, and not registered (§9):

member value
alg the algorithm digest is computed with. This revision defines jwp-object-1: the content hash of jwp-receipts §2.1, BLAKE3("moq-jwp:object\x01" ‖ object).
digest 64 hexadecimal digits: the content hash of the receipted object the frame was decoded from.
receipt optional; 64 hexadecimal digits: the link hash (jwp-receipts §3 step 3) of the group receipt that object verified under.
tier the tier the frame's samples hold (§7.2).

Its canonical form is the JSON text of those members in that order, receipt left out when absent, with no whitespace and no other member. A reader:

  • refuses a value that is not an object, that lacks alg, digest or tier, whose members are not strings, whose digest or receipt is not 64 hexadecimal digits (binding_malformed);
  • refuses an alg it does not know (binding_alg_unknown): it cannot check the digest;
  • reads a tier outside the vocabulary as none;
  • ignores a member this text does not define, as a dictionary drops one.

7.2 When a page binds a frame

A page binds a frame it decoded, and only then, when:

  1. the frame was output for a chunk whose coded data came from one receipted object, and the page computed digest over that object's bytes as received, or holds it as the content hash of the object receipt that object verified against;
  2. the content hash of every object the decoder was given since its last key frame — its last key chunk, or its configuration — was established: the object verified against its JWP object receipt (jwp-receipts §3 steps 1 to 5), or it was fetched whole under that hash as its content identifier, or through a verified head proof (http-delivery §6, §8.3). A frame predicted from data whose hash nobody established is not bound (binding_unverified_reference). A page that verifies every object before a decoder sees it meets this by construction.

receipt is written exactly when the object verified against a group receipt (jwp-receipts §3 steps 1 to 5), and is that receipt's link hash; a page MUST NOT write a receipt it did not verify the object against. For an object carried over MoQ, digest and receipt are then the content_hash and group_receipt of its WAI_RECEIPT property (moq-streaming-format §5). tier is the route's tier (§4.3), lowered to none where:

  • the route is ua and the frame's format is null: its samples cannot be read without a conversion, so nothing can check them;
  • the route is wasm and its decoder is not a module the page holds at the integer level of the WAISM profile (§4.3 step 6);
  • the page altered the samples after the decode (determinism-tiers §2, What a tier covers).

Where the frame's tier is below the object's need, the page MUST NOT present the frame as meeting it.

On the wasm route the wasm decoder writes the binding itself: it holds the object's bytes, hashes them, decodes them, and returns every frame with the binding.

8. Through the frame's life

operation the binding
a decoder's output none from the decoder: WebCodecs' Create a VideoFrame takes no metadata. The page binds it (§7.2, §9).
clone() carried: the clone references the same samples, and Clone VideoFrame assigns it every slot but the resource reference, [[metadata]] among them.
serialization (postMessage without transfer) carried: the serialization steps copy [[metadata]].
transfer carried to the receiving realm; the source is closed and keeps none.
close() gone with the frame.
copyTo() the bytes written carry none. With no format option they are the frame's samples, and a page MAY check them against a published digest (§4.4); with one they are a conversion, which no tier covers.
new VideoFrame(frame, init) carried only when the page puts it in init.metadata: the constructor copies init's metadata, not the source's. The new frame references the source's samples; a page MAY copy the binding into it, and a narrower visibleRect presents part of the same samples, from the same object.
new VideoFrame from a canvas, an image or a buffer a frame of other samples: none, except on the wasm route, where the samples are the ones the wasm decoder output from the bound object (§7.2).
drawing the frame (drawImage, a texture) the drawn pixels are a rendering and carry none; §10 says which frame a region of the screen shows.
VideoEncoder.encode(), and a decode of its output does not survive. A re-encoded frame is a new object: its chunks carry no binding, and a frame decoded from them is bound, if at all, to that new object and its own receipt.

9. Carriage in a page

9.1 As metadata

Where the user agent's VideoFrameMetadata has a contentBinding member, a page binds a decoded frame f by constructing new VideoFrame(f, { metadata: { ...f.metadata(), contentBinding } }) and closing f. clone(), serialization and transfer then carry it by WebCodecs' own steps (§8).

9.2 In a side table

VideoFrameMetadata is a dictionary whose members are the registry's, and a dictionary keeps only the members it declares, so a member no registry lists is dropped from init.metadata. Until contentBinding is listed, a page carries the binding in a side table:

  1. Pending. Before it gives a decoder a chunk, the page enters the chunk's binding under (that decoder, the chunk's timestamp). A decoder outputs a frame with the timestamp of the chunk it came from (WebCodecs Output VideoFrames), and nothing else in a frame says which chunk that was. Where a second chunk is given the same decoder under a timestamp still pending, both entries are struck and neither frame is bound (binding_ambiguous). A page bounds the entries pending per decoder, and strikes the oldest past the bound.
  2. Output. On a decoder's output, the page takes the entry under (that decoder, the frame's timestamp), removes it, and attaches it to the frame object, held weakly so that it lives as long as the frame does. An output with no entry is unbound.
  3. Reset. reset(), close() and a decoder error strike that decoder's pending entries, and so does flush() once it resolves, for the entries whose frames it did not output; a frame output afterwards for a struck entry is unbound.
  4. Clone. clone() attaches the source's binding to the clone.
  5. Transfer. A frame posted to another realm is posted with its binding beside it, and the receiver attaches it on arrival. A frame that arrives without one is unbound.
  6. Close. close() detaches it.

A page MAY test whether the user agent keeps the member (construct a frame with it in init.metadata and read metadata()), and use §9.1 where it does.

10. The frame on screen

For each region of the screen a page draws frames into (a tile), the page keeps the binding of the frame it last drew there. The state of that frame is computed from:

  • the binding, or none;
  • object: verified (the object verified against its receipt), refused, or unverified (no receipt has been checked);
  • chain: the group receipt's admission to its chain (jwp-receipts §3 step 3): linked, anchor, history, held (not yet admitted: parent_not_held) or refused;
  • provenance: for a track with a provenance track (live-provenance §10.2), the verdict of live-provenance §11.4 on the segment of the frame's group, accepted or refused, or missing while its provenance object has not arrived; not_applicable for a track packaged as WAI envelopes, which has no C2PA provenance (live-provenance §1).

The state is, in this order:

  1. unbound, with no binding or with one the reader refuses (§7.1);
  2. refused, when object, chain or provenance is refused;
  3. pending, when object is unverified, chain is held or provenance is missing;
  4. verified otherwise.

anchor and history are admissions: a chain a joiner anchored, or filled below its anchor. A page that shows verified for them MUST also say from which group it holds the chain, as jwp-receipts §3 step 3 requires of a verifier that re-anchors, and MUST NOT state the groups before it verified.

A page that shows the state, behind a control the reader turns on, shows for each tile the state, the tier, the first digits of the digest and of the receipt, the provenance, the route and the energy entry. The state is about the frame the tile last drew, not about the pixels on the display: what the display emits is a rendering of the frame, scaled and converted to the display's colour space, which no tier covers.

11. Reasons

reason raised at
report_malformed §3
not_dispatchable §4.3 step 1
tier_exceeds_class §4.3 step 2
constraints_unevaluable §4.3 step 3
constraints_unmet §4.3 step 3, §5 step 5
no_ua_mapping §4.3 steps 4.1, 4.3
codec_mismatch §4.3 step 4.4
codec_unregistered §4.2, §4.3 step 4.2
ua_not_asked §4.3 step 4.5
ua_unsupported §4.3 step 4.5
ua_inexact §4.3 step 4.6
ua_unprobed §4.3 step 4.7
ua_probe_mismatch §4.3 step 4.7
no_host §4.3 step 5
host_inexact §4.3 step 5
no_wasm §4.3 step 6
no_decoder §4.3 step 7
no_alternative §4.7
constraint_malformed §5 step 2
parameter_unregistered §5 step 3
energy_class_invalid §6
unmetered_as_zero §6
energy_count_invalid §6
energy_evidence_missing §6
binding_malformed §7.1
binding_alg_unknown §7.1
binding_unverified_reference §7.2
binding_ambiguous §9.2 step 1

An object routed none is inert at the sink for that object, as SPEC §4 step 6 has it; a refused energy measurement leaves the route as it is; a refused binding leaves the frame unbound.

12. Conformance

live-conformance/browser/ holds the corpus, vectors.json, written by wai_browser_sink_vectors gen. It has:

  • Routes: capability reports, each with the routing table §4 and §6 give it: every route and every reason of §4.3, the registry gate with codec strings for AV2, VVC and IAMF answered supported and never read, a need of decode-equivalence refused a UA decoder whose probe did not run and taken by one whose probe matched, the host and wasm fallbacks, and a malformed report.
  • Choices: reports whose objects are alternatives, each with its routing table and the choice table §4.7 gives it: first, smooth and classical before learned-without-budget; a power-efficient answer that is not smooth below a smooth one; a power-efficient answer for a codec routed to a host decoder (its need above the user agent's tier) read for nothing; a learned alternative with a qualifying measurement (OnChipCounter, CalibratedInstrument) ranked classical, and with a Partial or a ModelBased one ranked last; a learned alternative chosen where no other routes; no answer read as not power-efficient; two choices beside an object in none; a choice with nothing to choose (no_alternative); and an empty and a non-string choice, each refused whole. Every measurement in them is synthetic, marked "note": "synthetic: no measurement".
  • Constraints: constraint sets and declared parameters, with §5's outcome: satisfied by one set of several, by an empty set, by enum, minimum and maximum; unsatisfied by an undeclared parameter and by a value of the wrong type; each malformed form and an unregistered parameter.
  • Energy, among the routes: measurements beside answers, each with its energy entry: every class, the unmetered case, a figure under another route's key, and each refusal of §6.
  • Bindings: values with their reading: canonical forms, each refusal of §7.1, an unknown tier read as none, and a member the text does not define; the tier §7.2 gives a frame for each route, format and alteration, a wasm frame's decoder being a module the page holds at the integer level of the WAISM profile (§4.3 step 6); and objects bound under a signed group receipt, each binding's digest the object's content hash and its receipt the receipt's link hash, the receipt's signature and the object's inclusion verifiable from the corpus.
  • Frames: inputs to §10 with their state, covering every row.
  • Side tables: sequences of chunks, outputs, clones, transfers and resets with the bindings §9.2 leaves each frame, binding_ambiguous among them.

wai_browser_sink_vectors verify and the test tests/browser_sink.rs check that vectors.json regenerates byte for byte, that wai-rs gives every case its stated outcome, that every reason of §11 appears, and that §11 lists exactly the reference's codes. frame_tier in wai-rs, which the page calls as frameTier, is given no module status: it gives a wasm frame the tier §7.2 gives the frame of a decoder held at the integer level, and every wasm frame of the corpus, the chain's among them, is of such a decoder. The page whose code the header names loads no wasm decoder for a room's tracks, so it binds no frame on the wasm route. live-conformance/browser_sink_verify.py is the second implementation, written from this text: it shares no code with wai-rs, reads each capability's kind, status and replicate class from the registry export (registry/capabilities.json), checks signatures with WAI's strict rule and hashes with BLAKE3 from claim-conformance/, and mutates copies of the corpus and requires every copy to fail. The page runs the route, choice, constraint, energy, binding and frame cases through its wasm build, and its side table through the sequences, in a browser (joule-moq-web's end-to-end scenario binding). wai conform sink --suite browser runs every call of the corpus but the chain against another implementation, its answer withheld, and signs a record of the run (conformance-results); wai_browser_sink_vectors answers is the reference's answer to one call.

13. Security and privacy considerations

  • A binding is a statement, not a signature. Any script in the page can bind any frame. What a binding gives is where to look: the receipt it names commits to the digest under a key, and a party that holds the object, the receipt and its key log checks them. The state of §10 is the page's own verdict on what it checked.
  • What verified covers. The object and its group receipt, and, for a CMAF track, the segment's C2PA provenance. It does not cover what the display shows, how a frame was scaled, or a user agent's decoder: a UA decoder that misdecodes a verified object outputs a frame bound to that object. §4.4's probe is how a page holds a UA decoder to the decoding process where the tier matters.
  • What a report reveals. A report holds the user agent's answers and, with a meter attached, an energy figure; together they say a good deal about a device (Media Capabilities, privacy considerations). Nothing in this extension sends a report anywhere: routing, binding and the state of a frame are computed in the page. A service that collects reports learns that much about each device it collects from; where it needs to record a legal basis for doing so, jwp-receipts §2.4.6's lawful-basis claim is the place.
  • An energy figure is a side channel. A per-process energy reading, taken often enough, leaks what a process does. A page receives a figure only from a meter the host chose to attach, at the granularity the host chose, and this extension never has a page read a counter itself.
  • Ambiguity is refused, not guessed. A side table keyed by timestamp cannot tell two chunks of one decoder with one timestamp apart, so neither frame is bound (§9.2).
  • Unregistered codec strings are not asked about. Besides having no defined answer, a question about a codec string outside the registry adds to what a page learns about a device for nothing it can use.

14. Composing a package

A package (packaging) may carry a programme as several components, each in the capability that suits it: camera footage in a video codec or an integer rung, titles and credits as vector graphics, captions as timed text, music as a score, data as signed state. A composing page opens such a package, dispatches each component by §4, checks everything it is about to present against the package's signed receipt, decodes each component on the route it chose, and presents them all on one timeline. A piece's base may have layers (staged-delivery), and the page computes two kinds of them itself, in WebAssembly, before it presents any stage they give: the levels of an integer neural field (neural-fields §7) and a picture refined by a layer conditioned on side planes (staged-delivery §6.3) (§14.5). The reference is wai/wai-web's compose bundle (src/compose.rs and src/compose_stage.rs, feature compose) and its page, wai-web/compose/ (compose.js, index.html); the package it plays is wai-web/compose/package/, which wai-pack writes under compose/policy.toml (packaging §5.8), and the staged packages beside it are wai-web/compose/staged/ (§14.7).

14.1 What the page reads

The page reads a version-2 package receipt (packaging §9.1), the index of each component it names (packaging §6), and the objects the indexes list (objects/<content hash>.wai). Every component is a run of pieces on its own timebase, the lead's cut at its scene cuts and every other component's at the lead's piece starts (packaging §4.5, §5.8), each piece one object or more. Within a component, a piece's alternatives are its base entry and each of its floor entries; a refinement rides on the base it refines and is not an alternative. An asset entry of a piece is an object a layer of that piece may name as a side object (staged-delivery §3.1); the page reads it only when a layer names it (§14.5). The page reads nothing else of a package: no playlist, catalog or listing, none of which the receipt binds. A text piece names its language in its object's target (packaging §5.8), which is how the page reads its segment.

14.2 Layers

The layer a decoded object is presented in follows from its capability, never from its component's media class:

capability layer drawn as
a video capability, wai.image.png in a video or an image component video frames on the picture, at the lead's frame rate; a still held for its piece
wai.field.int_grid video the field's render at the view its payload pins (neural-fields §6), held for its piece
wai.graphics.vector overlay an RGBA8P frame (graphics-vector §2.9) composited source-over above the picture
wai.text.timed text text, in the page's own text layer, at each cue's anchor
an audio capability, wai.audio.score audio samples on the page's audio output
wai.feed.state state the records of the state at the clock's tick

The vector overlay is rasterised at a size its payload lists (graphics-vector §3.10): the smallest listed size at least as large as the overlay's area on the display in device pixels, else the largest listed. The platform scales it from there (graphics-vector §1.1).

14.3 Dispatch

For each piece of each component, the page builds §3's report over the piece's alternatives, each object carrying the choice "<component>/<piece>", and decodes the alternative §4.7 chooses. The report's wasm, host and ua say what the device has: a decoder the page holds in WebAssembly is in wasm, a capability that plays only where the decoder keeps up (§4.6); the user agent is asked, as §4.6 has it, about every alternative whose capability platform-decode maps (AV1, HEVC, AVC, VP9 and VP8 video; Opus, FLAC and AAC audio) and that the page does not decode in WebAssembly itself, from a query read from that object's payload (platform-decode §2), after the object has been checked (§14.4); each configuration is asked, and answers under one codec string are merged (platform-decode §3). A capability the page decodes in WebAssembly (FLAC in the reference page) is not put to the user agent, so its UA leg is ua_not_asked and it routes wasm: the page's decode holds the capability's registered tier, the user agent's none (§4.1). A profile without the page's decoder of it asks the user agent. A field (wai.field.int_grid) and the objects a conditioned layer reads as side planes (wai.image.png, and wai.splat.int_codec for a render) the page computes in WebAssembly too: platform-decode maps none of them, so their UA leg is no_ua_mapping, the user agent is never asked about them, and they route wasm. The capabilities whose layers the page applies in WebAssembly (wai.field.int_refine, wai.image.int_cond_refine) are listed in the report's wasm leg beside its decoders. A layer is not an alternative and is not routed: it is applied as §14.5 has it where its capability is in that leg and its piece's base was chosen, and is otherwise not applied, its piece presenting the stage below (a floor chosen takes no layer). An alternative routed ua is decoded through WebCodecs from the chunks platform-decode §2.3 reads from its payload, with the first configuration the user agent accepted under which it decodes; a video one is presented in the video layer, an audio one on the page's audio output. An Ogg Opus stream (wai.audio.opus) states a pre-skip and an end (RFC 7845 §4) that a decoder handed its packets does not apply: the page drops the pre-skip, ends the samples at the last page's granule position, and holds an Opus floor's piece (packaging §7.9) to exactly its piece's samples (compose_frames otherwise). A forced profile is a report from which named capabilities are left out of every leg and never asked about; it routes and chooses as any other.

  • The chosen alternative is decoded first. Where the user agent's decoder does not decode it, having answered the configuration supported, the page decodes the next alternative in §4.7's order whose route is not none, after checking that object as §14.4 has it: §4.7's "the chosen alternative is tried first, and the other after it". The dispatch record keeps what was tried and why it did not decode (tried).
  • A piece whose alternative is a record (wai.text.timed) is not routed: SPEC §4 dispatches no decoder for a record, and the page draws it in its text layer.
  • A component with a piece for which nothing is chosen (no_alternative) is not presented on this device, and the page says so. The lead component (the receipt's first) carries the clock (§14.6): a package whose lead has such a piece is refused whole.

The dispatch record is what this device dispatched, and why, as data a page can show:

{ "device": { "user_agent": "<text>", "forced_without": [ "<capability>" ],
              "wasm": [ "<capability>" ], "ua": { "<codec string>": { "supported": true, … } } },
  "receipt": { "receipt_hash": "<hex>", "signer_id": "<text>", … },
  "components": [ { "id": "<id>", "layer": "<layer>" | null, "presented": true,
                    "pieces": [ { "piece": 0, "chosen": "<name>" | null, "capability": "<capability>",
                                  "role": "base" | "floor" | null, "route": "ua" | "host" | "wasm" | "page" | "none",
                                  "why": "<text>", "order": [ "<name>" ],
                                  "tried": [ { "name": "<name>", "error": "<text>" } ],
                                  "alternatives": [ { "name": "<name>", "capability": "<capability>",
                                                      "role": "base" | "floor", "route": "<route>",
                                                      "reason": "<reason>" | null, "legs": { … },
                                                      "tier": "<tier>", "rank": "<rank>" | null } ],
                                  "layers": [ { "name": "<component>/<piece>/L<layer>/<capability>",
                                                "capability": "<capability>", "layer": 1,
                                                "route": "wasm" | "none",
                                                "outcome": "pending" | "applied" | "refused" | "not-applied",
                                                "reason": "<reason>" | null, "detail": "<text>" | null,
                                                "stage": 1 | null, "sha256": "<hex>" | null,
                                                "sides": [ { "entry": "<name>" | null, "object": "<hex>",
                                                             "render": { … } | null, "held": true,
                                                             "exact": true | null, "capability": "<capability>",
                                                             "via_fallback": false, "tier": "<tier>",
                                                             "why": "<text>" | null } ] } ] } ] } ] }

An alternative's route, reason, legs and tier are its §4.3 routing-table entry, and its rank its §4.7 rank; why says in words which rule decided: the base first in §4.7's order, the floor first in that order, or the floor because the base has no route here, with the base's reason and legs. A piece with layers lists each, in layer order: its outcome (pending until it arrives), the reason it was refused or not applied (a reason of staged-delivery §4.2, an object check's code (packaging §6.3), or prefix-digest), the stage it presented and that stage's SHA-256, and each side plane it named: the object, the render's pose for a render entry, whether the page held its plane, whether exactly, how it was decoded, and why a plane was not held.

14.4 Checks before presenting

The page presents nothing of a package until each of these holds, in this order; the first that fails refuses the package whole, and the page states which check failed and with which code:

  1. the receipt's JSON, its indexes, their digests and stage roots, its counts and its Ed25519 signature (packaging §9.1, PackageReceiptV2::verify_detailed; in the page, verifyPackageReceipt, and verifyPackageReceiptSignature for the signature alone);
  2. each index, read now that the receipt holds it (packaging §6);
  3. every object the page is about to decode, and every floor object it reads a query from, against its index entry: its length and content hash, its envelope, lengths and capability, its refines and derived_from (packaging §6.3);
  4. each decoded output whose entry states decode-equivalence, against the entry's prefix digest (packaging §6.5), piece by piece: a video base's clip digest (a still's frame held for its piece); a vector piece's clip digest over its frames rasterised in the page as RGBA8P at the first size its payload lists and its component's rate; a text piece's SHA-256 over its segment, once the segment reads to its cues (timed-text §5.3); a score piece's SHA-256 over its samples as signed 16-bit little-endian integers, a two-channel score's interleaved (score §3.4), and a FLAC piece's over its samples, interleaved, as the page decodes them in WebAssembly; a field's render at its own view, computed in the page, whose SHA-256 is the field's stage hash (staged-delivery §5); a picture of an image component, its RGB8 frame decoded in the page (packaging §6.5); and a feed piece's state, replayed from its signed ops to its seal, against the sealed state hash (state-feeds), each piece after a component's first starting from the state the one before it seals.

A floor decoded by the user agent is held to no digest: its tier is none (§4.1), and the page says so; an audio floor among them too, whatever its codec, since the user agent chooses the sample format it hands back. A check's outcome is listed with what it compared, so a page can show every check it made. A piece whose base has layers is presented at its base once these hold; each layer is checked as it arrives (§14.5), and one that fails a check is not applied, without refusing the package.

14.5 Refinement layers

A piece whose chosen alternative is its base presents the base first. Its layers then arrive one at a time, in layer order, and for each the page, in this order:

  1. fetches it and checks it against its entry: its length and content hash, its envelope and lengths, its capability and its refines (packaging §6.3);
  2. computes the next stage in WebAssembly, with the reference implementation of staged-delivery §4.1 compiled for the page, which refuses with that section's reasons (§4.2):
    • over a field, the payload's level as the field's next level (neural-fields §7.3, staged-delivery §6.2), the field holding no layer for later (a hold bound of zero), so a level whose level below is not applied is refused layer-gap; then the field rendered again at its own view;
    • over a picture, wai.image.int_cond_refine (staged-delivery §6.3). First each side plane its refines.side names (§3.1), held by the page or not. An object the piece's asset entries list is fetched once and held only where the page decodes its capability in WebAssembly under the device's profile, its bytes pass their entry's checks (packaging §6.3), and its output, decoded through its manifest's capability or its fallback as SPEC §4 dispatches, holds the digest its entry states. A render entry's plane is the frame wai-splat-render:i1 gives at the entry's pose (splat-exact §5), which the page renders itself from a splat set held so, decoded through wai.splat.int_codec with the parameter set the page holds, its pin verified. Then §4.1 step 4 refuses the layer where the page does not hold a plane (side-absent), or holds one that is not exact (side-not-exact: decoded through a fallback, or below decode-equivalence);
  3. holds the stage's output to the prefix digest its entry states: for a field the SHA-256 of its render at its own view, for a picture of its RGB8 frame, each the stage hash staged-delivery §5 defines (prefix-digest);
  4. presents the stage.

A layer refused at step 1 or 2 is not applied, and one refused at step 3 is computed and never presented; either way the page presents the stage below it (the base, for a first layer) and says why: the dispatch record states each layer's outcome and reason, and each side plane, held or not, exact or not, and why not (§14.3). The page then takes the next layer, and the reference implementation decides it (a level whose level below was refused at step 1 or 2 is refused layer-gap). The page never presents a stage whose digest it has not compared, nor one that failed. An asset's prefix digest is its decoded output's stage hash (staged-delivery §5), the attribute tensor's bytes for a splat set.

14.6 One timeline

The lead component's video is the master clock. Its frame n is presented from n / fps seconds, fps its index's timebase; every other layer is placed by that clock:

  • the overlay presents its frame g = ⌊t · num / den⌋ at its rate num/den, from the piece whose window holds g (a piece's frames are the whole document's, graphics-vector §3.11), and nothing outside its pieces' frames;
  • the text layer shows the cues active at t of the piece the clock is in (a cue across a piece start is in both pieces' segments, with its whole interval, timed-text §5.2);
  • the state layer shows the state at tick ⌊t · tb_num / tb_den⌋ of its component's timebase, from the piece that holds the tick, never past that piece's seal;
  • the audio layer plays its pieces' samples end to end, in their channels and at their rate (every piece of a component the same, or the page refuses it, compose_audio_format), from the position the clock gives, and re-places them at the clock's position whenever they drift from it by more than one video frame.

The reference page plays a piece's decoded frames from a clock it starts on the page's own timer, and the score through Web Audio, so the measure of drift is the audio context's own time. A layered piece's picture is the latest stage it presented; the reference page moves the clock to a piece as it presents that piece's next stage, so the stage is on screen.

14.7 The package's shape

packaging §5 (Components beside the lead, 5.8) is the normative home of a package of several components: the media classes graphics (7) and text (8) beside packaging's others, the rungs each medium takes, how each component is cut at the lead's piece starts, each piece's object and target, and the decoded-output digest each entry states. The reference package is written by wai-pack under compose/policy.toml: the lead v, a test pattern of 320 × 180 (the fixture compose-01) with an integer base and a supplied AV1 floor for every piece; g, vector-conformance's title card and credit roll; t, three captions; s, a voiced score of two channels at 48 kHz with a FLAC floor of each piece's mixdown, which the page decodes in WebAssembly where the score is not routed (a floor states no digest, so the page holds it to none; the page asks the user agent about no audio codec); and f, a scoreboard feed. The corpus check of wai-pack packs it again from those sources and requires every byte the same. A composing page does not read a component's media class for a layer (§14.2); it reads it for the digest a picture states (packaging §6.5: an image's RGB8 frame, a video's clip).

Beside it, the page plays the staged packages under wai-web/compose/staged/. src/compose_stage.rs's tests seal them with the package's key from objects of the corpora, byte for byte, each entry stating the prefix digest computed natively and held there to the corpus's own statement, and require the committed files to be those bytes. Each is one image component on the timebase 1/1, one unit a piece:

package its pieces
fields three fields of volumetric-conformance/fields/: the orb, its base carrying two of its four levels and its two further levels as layers 1 and 2; mlp; edge
fields-digest fields, the orb's layer 2 stating the digest of its stage 1
cond three pictures of staged-conformance/cond/, cond_side_planes, cond_dual_gbuffer and cond_render, each a PNG base, its wai.image.int_cond_refine layer and its side objects as asset entries (two planes, three G-buffers, a splat set)
cond-not-exact cond_side_not_exact's layer over a plane carried as wai.image.jxl with a wai.image.png fallback (an asset of tier none)

They are sealed under no policy (their policy digest is zero), and an image lead and asset entries are not among the shapes of a package's components (packaging §5.8); a page reads them as it reads any package (§14.1).

14.8 Conformance

wai-web/compose/run.mjs plays the reference package in browser engines and passes when, in each: the package opens with every check of §14.4 holding and every component presented, and plays to its end with frames, overlay frames, cues, state and audio presented; the score's samples, synthesised in the page in the channels and at the rate each piece's target states, have piece by piece the SHA-256 its index entries state, which wai-pack computed natively; every overlay frame rasterised in the page has the SHA-256 vector-conformance/ states, and each piece's frames the clip digest its entry states; each text piece holds to its segment's digest and each feed piece to its seal, chained to the piece before; under a profile without the AV1 decoder each integer video base decodes in WebAssembly to the clip digest its entry states; under a profile without the integer rung and the still, every video piece plays its AV1 floor through the user agent, the base unrouted for want of a WebAssembly decoder (no_wasm), and the page says the floor plays; under a profile without the score, every score piece plays its FLAC floor, decoded in WebAssembly to the samples whose SHA-256 the score's signed base states for its mixdown, and the page says the floor plays; in pack-conformance/multi/multi-opus-01, under a profile without FLAC, where the engine's AudioDecoder takes Opus every piece of the sound track plays its Opus floor through it and comes to exactly the piece's samples, and where it does not the sound track is not presented and the picture is; a receipt signed by another key (compose/mutations/forged.receipt.json, which wai-pack's corpus check writes and holds) is refused receipt-signature; and an object with one byte changed in delivery is refused object-hash; each of the last two with nothing presented. Over the staged packages (§14.7), held to what volumetric-conformance/fields/ and staged-conformance/cond/ state as well as to their signed entries: every field of fields renders in the page at its pinned view to the frame digest its scene's vectors state, read back so from the drawn picture; with each of the orb's levels held back in delivery and released one at a time, the coarse field is presented before its first level is asked for, and each level released presents the next stage, whose digest is the stage hash the corpus states, before the next level is asked for; in fields-digest the orb's stage 2 is computed, refused prefix-digest, and stage 1 stays presented; a byte of the orb's first level changed in delivery is refused object-hash, its second level layer-gap, and the coarse field stays presented; every picture of cond presents the stage hash its case states, from two exact planes, three G-buffers and a splat set's frame rendered in the page; a side object withheld in delivery, and the splat decoder left out of the profile, each refuse their layer side-absent with the base presented and the page saying why; cond-not-exact's layer is refused side-not-exact, its plane held through its fallback, with the base presented; and a side object with a byte changed in delivery is refused object-hash, never held, and its layer side-absent. It exits 0 only when every check passes in every engine named, and fails on a page error, on an engine that cannot be launched or runs fewer checks than it makes, and on an error of its own.