Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 102
open-standards/wai/extensions/browser-sink.md
Transaction Science bfa44f4c1d Merge #463 (wai/energy-honesty @13964c94, replayed): EN-C, the energy order of dispatch, a self-metered relay, HwShunt closed to new profiles 
#463's head carries #452's original history, and with it #438's first
research note (docs/research/wai-cross-fabric.md), which names fleet hosts and
addresses; #465 replaces that history with one squash so the note never enters
main's. So #463 is not merged by its head: its own commits are replayed with
cherry-pick -x onto the train's #465 merge (767d2aa7a), the second parent here:

- #452's platform-decode commits, which the review's split gave to EN-C and
  #465 left out: 6ad92e2e8, 8695bc75f, fed8694f4;
- EN-C's fourteen: df5ea962f 11b255682 ff069acc2 244a715b3 2a8d45aa0 a203eae23
  9b6e4bada 1ffceb508 6c03ba72c ce38948da 1ca593124 507f4cd9e c95337739
  13964c944.

#463's merges of main and of #452 are not replayed; their conflict
resolutions (SPEC §7.1 step 2, the CHANGELOG) are reached by the later commits.
Checked: the replay's tree equals a three-way merge of #463's head onto the
replay's platform-decode commits with #452's head merged into main 1a465f23 as
the base, file for file, except the two files that merge conflicts on
(int_kernels_vectors.rs: #465's wgsl line and EN-C's platform-decode line, both
kept), and its CHANGELOG is #465's with #463's energy-honesty entry exactly.
No address of the fleet appears in the replayed commits.

Merging the replay here: CHANGELOG only, #463's entry on top. The registry
export is up to date after the merge.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-26 08:32:50 -04:00

658 lines
42 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# WAI Extension: Browser Sink
> Status: Draft. How a sink that runs in a web page answers SPEC §4's "the sink advertises
> `cap`", and what the frames it decodes can be checked against. For each object it is about
> to decode, the page **routes** the object's capability to the user agent's decoder, a native
> host's or WAI's own WebAssembly decoder (§4): the user agent is asked only about codec strings
> the WebCodecs Codec Registry lists, and its decoder is preferred where it is exact for the
> tier the object needs. Among objects that are alternatives of one content, the page decodes
> the one SPEC §4's energy order puts first, inside the routing: a decode routed to the user agent
> that it answers supported, smooth and power-efficient before a software one, and a learned one
> without a sink-metered figure of its decode last (§4.7). The page evaluates the constraint sets of
> [capability-constraints](capability-constraints.md) itself (§5), and reports the joules a
> decode was measured to cost beside the user agent's `powerEfficient` boolean, which stays the
> user agent's word (§6). Every frame the page decodes carries a **content binding** (§7): the
> content hash of the receipted object it was decoded from, the link hash of the group receipt
> that object verified under, and the tier its samples hold. The binding's life through
> `clone()`, transfer, `copyTo()` and re-encode is in §8, its carriage in a page in §9, and the
> state of the frame on screen, which joins the binding to the receipts and to
> [live-provenance](live-provenance.md)'s verdict, in §10. The member a frame carries,
> `contentBinding`, is proposed for the WebCodecs VideoFrame Metadata Registry in
> [`proposals/webcodecs-content-binding.md`](../proposals/webcodecs-content-binding.md), a
> draft that has not been filed; until a registry lists the member, a page carries it in a side
> table (§9.2). No transport, no record, no change to the wire format or to SPEC §4's algorithm.
> The reference implementation is the `browser_sink` module of `wai-rs` (feature
> `browser_sink`), which `wai/wai-live-web` builds for the page; the page's own code is
> `joule-code/joule-code-rs/crates/joule-moq-web/web/room/binding.js`; the corpus is
> [`live-conformance/browser/`](../live-conformance/browser/), and its second implementation is
> `live-conformance/browser_sink_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
>
> Texts read for this revision, each pinned to the version named: WebCodecs, W3C Working Draft
> 19 October 2026: the VideoDecoder's *Output VideoFrames* steps, *Create a VideoFrame*, the
> `VideoFrameMetadata` dictionary and `metadata()`, *Copy VideoFrame metadata*, *Clone
> VideoFrame*, the constructors' metadata steps, the serialization and transfer steps, `copyTo()`
> and `EncodedVideoChunk`; the WebCodecs Codec Registry, W3C Registry Draft 24 September 2026,
> its video and audio tables; the WebCodecs VideoFrame Metadata Registry, W3C Registry Draft
> 14 April 2026, its entries and its registration requirements; and Media Capabilities, W3C
> Working Draft 18 October 2026, `MediaCapabilitiesInfo` (`supported`, `smooth`,
> `powerEfficient`) and its privacy considerations. A section number in this text is this
> extension's unless it names another document.
## 1. Scope
In scope: the capability report a page builds (§3); routing an object to a decoder (§4), and
choosing among alternatives (§4.7);
constraint sets evaluated in the page (§5); the energy entry beside `powerEfficient` (§6); the
content binding, its form and when a page writes one (§7); what happens to it through a frame's
life (§8); how a page carries it (§9); the state of the frame on screen (§10); and the reasons
(§11).
Out of scope: a transport; the user agent's decoders and whether they conform to their
standards, which a page cannot see beyond the answers they give and a probe (§4.4); the
decoders of a native host; how a meter measures (energy-measurement §3); and what a party does
with a state.
Relationship to the core spec and the other extensions:
- **SPEC §4.** A page answers step 3's "the sink advertises `cap`" for an object with its route
(§4): it advertises `cap` for that object when the route is not `none`. Among alternatives it
applies SPEC §4's energy order (§4.7).
- **[http-delivery](http-delivery.md) §4.3 and §12.** A WAI element is never put to a user-agent
interface (§4.3 there). The route gives the Sink Capability Descriptor's `via` (§4.6).
- **[capability-constraints](capability-constraints.md).** §5 is its evaluation in a page, over
the register of its §3.1.
- **[energy-measurement](energy-measurement.md) §2.** The classes a figure carries (§6).
- **[platform-decode](platform-decode.md) §2, §3.** The query a page asks the platform, read
from the payload, and how the answers enter the report.
- **[jwp-receipts](jwp-receipts.md) §2.1 and §3 step 3.** The content hash a binding names, and
the link hash of the group receipt.
- **[determinism-tiers](determinism-tiers.md).** The tier vocabulary a need and a binding use.
- **[live-provenance](live-provenance.md) §11.** The verdict on the segment a frame's object
belongs to, for a CMAF track (§10).
## 2. Terms
- **Page**: a sink running in a web page.
- **UA decoder**: a decoder of the user agent: WebCodecs `VideoDecoder`, `AudioDecoder` or
`ImageDecoder`. **Wasm decoder**: a WAI decoder compiled to WebAssembly that the page loads.
**Host decoder**: a decoder of a native host the page hands a payload to (http-delivery §12.1,
`host`).
- **Route**: `ua`, `host`, `wasm` or `none`.
- **Receipted object**: the bytes a JWP content hash covers (jwp-receipts §2.1): a WAI envelope;
in an end-to-end encrypted room, the sealed object, over which the receipt is signed
([confidentiality](confidentiality.md)); in a CMAF track, a chunk (live-provenance §8.1).
- **Tier**: `decode-equivalence`, `entropy-consistency` or `none`
([determinism-tiers](determinism-tiers.md) §2), ordered `none` < `entropy-consistency` <
`decode-equivalence`. A token outside this vocabulary is read as `none`, as a pin's is
(determinism-tiers §3).
- **Registered tier** of a capability: the tier its decoded output holds on every conforming
sink, from its replicate class ([SPEC](../SPEC.md) §7):
| replicate class | registered tier |
|---|---|
| `StandardDefined`, `NeuralIntegerExact` | `decode-equivalence` |
| `NeuralEntropyExact` | `entropy-consistency` |
| every other class, and a string this revision does not register | `none` |
`StandardDefined` is read as `decode-equivalence` because SPEC §7 has its decoded output the
same bytes on every conforming sink, which is the property that tier names; determinism-tiers
defines the tiers for neural capabilities, and this extension uses the same tokens for that
property and claims nothing more. `GenerativeExact` output also reproduces, but it is
generated rather than decoded, so it is held at `none` here.
- **Need**: the tier the page holds an object's decoded output to: the object's pin `tier`
([SPEC](../SPEC.md) §3.1) where it has one, else its track's `wai_determinism`
([moq-streaming-format](moq-streaming-format.md) §4), else `none`.
- **Frame**: a WebCodecs `VideoFrame`. **Binding**: a `contentBinding` value (§7).
Hexadecimal is lowercase. JSON in this text is RFC 8259.
## 3. The capability report
A page states what it knows about its decoders, and what it is about to decode, in a **capability
report**: one JSON object.
```json
{ "browser_sink": 1,
"ua": { "<codec string>": { "supported": true, "smooth": true, "power_efficient": false,
"probe": "match" | "mismatch" | "not_run" } },
"host": { "<capability>": "<tier>" },
"wasm": [ "<capability>" ],
"constraints": { "<capability>": [ { "<parameter>": { "enum": [ ], "minimum": 0, "maximum": 0 } } ] },
"energy": { "<capability> <route>": { "joules_micro": 0, "decodes": 0, "window_us": 0,
"class": "<class>", "relative_ppm": 0,
"filtering": "On" | "Off" | "Unknown",
"calibration_ref": "<text>",
"coverage": "Whole" | "Partial", "fabrics": [ "cpu" ] } },
"objects": [ { "name": "<text>", "capability": "<capability>", "codec": "<codec string>" | null,
"need": "<tier>", "params": { "<parameter>": 0 }, "choice": "<text>" } ] }
```
- `browser_sink` is `1`.
- `ua` holds the user agent's answers, by codec string: `supported`, from `isConfigSupported` for
the configuration the page built from the object (§4.6); `smooth` and `power_efficient`, from
Media Capabilities' `smooth` and `powerEfficient` for the same codec string, or `null` where the
page did not ask; `probe`, from §4.4.
- `host` names the capabilities a native host decodes for the page, each with the tier its
decode holds, as the host declares it.
- `wasm` lists the capabilities the page's wasm decoders decode.
- `constraints` holds the constraint sets the page's decoders publish, by capability (§5). A
capability with no entry is unconstrained.
- `energy` holds measurements, by capability and route (§6).
- `objects` lists what the page is about to decode, each object (or each track, whose objects
share these values) with a `name`, its `capability`, the codec string a UA decoder would be
asked under (`null` where none applies), its need, and its declared parameters (§5); and,
optionally, a `choice`, a non-empty string that objects which are alternatives of one content
share (§4.7).
A report that does not read in this shape, or whose `browser_sink` is not `1`, is refused whole
(`report_malformed`): every member above is required but an object's `choice`, which, when
present, is a non-empty string; `objects` may be empty, and a member this text does not define is
ignored. A codec string with an answer in `ua` that §4.2's gate refuses is
never read (§4.2).
## 4. Routing
### 4.1 The user agent's decoders
A capability is put to a UA decoder only through a row of this table:
| capability | UA decoder | codec string | UA tier |
|---|---|---|---|
| `wai.video.vp9` | `VideoDecoder` | `vp09.` and more | `decode-equivalence` |
| `wai.video.avc` | `VideoDecoder` | `avc1.` or `avc3.` and more | `decode-equivalence` |
| `wai.video.hevc` | `VideoDecoder` | `hev1.` or `hvc1.` and more | `decode-equivalence` |
| `wai.video.av1`, `wai.video.av1.lossless` | `VideoDecoder` | `av01.` and more | `none` |
| `wai.video.vp8` | `VideoDecoder` | `vp8` | `none` |
| `wai.audio.opus`, `wai.audio.opus.packet` | `AudioDecoder` | `opus` | `none` |
| `wai.audio.flac` | `AudioDecoder` | `flac` | `none` |
| `wai.audio.aac` | `AudioDecoder` | `mp4a.` and more | `none` |
The **UA tier** is what a UA decoder's output holds when the decoder conforms to its standard:
- `decode-equivalence` for the video codecs whose decoder conformance requires every output
sample to equal the decoding process's (SPEC §7, `StandardDefined`), over the samples of the
frame the decoder outputs, in the format it outputs them in (§8);
- `none` for AV1, whose film-grain synthesis is held to a perceptual criterion, for VP8, whose
conformance this revision has not read, and for Opus and AAC (SPEC §7); and for FLAC, whose
decoded samples are exact but which an `AudioDecoder` hands back in a sample format the user
agent chooses, a conversion no standard fixes (SPEC §7, *What `replicate` promises over a
standard codec*).
A capability with no row has no UA decoder: `wai.video.av2`, `wai.video.vvc` and
`wai.audio.iamf` among the registered ones, since the registry of §4.2 lists no codec string for
them, and every WAI-native capability, whose decoder is WAI's own. `ImageDecoder` has no row in
this revision: an image decode in a user agent may apply colour conversion, and the user agent
does not say whether it did.
### 4.2 The registry gate
The WebCodecs Codec Registry (W3C Registry Draft, 24 September 2026) lists these codec strings:
| media | rows |
|---|---|
| video | `av01.*`, `avc1.*`, `avc3.*`, `hev1.*`, `hvc1.*`, `vp8`, `vp09.*` |
| audio | `flac`, `mp3`, `mp4a.*`, `opus`, `vorbis`, `ulaw`, `alaw`, `pcm-*` |
A codec string **matches** a row equal to it; a row ending `.*` or `-*` is matched by a string
that begins with the row up to and including its `.` or `-` and has at least one character after
it. A codec string that matches no row:
- **MUST NOT** be passed to `isConfigSupported`, to Media Capabilities or to any other
user-agent interface by a page that routes by this extension;
- is never answerable: an answer for it in a report is not read, whatever it says, and the
object's UA leg is refused `codec_unregistered`.
The registry is how WebCodecs names a codec. Its abstract says no implementer is required to
support any entry; a string outside it names nothing the registry defines, so two user agents'
answers about it need not be about the same codec, and a page has no basis to read one. Such a
capability resolves through a host decoder or a wasm decoder, or not at all. A row the registry
adds later enters this table by a revision of this extension.
### 4.3 The steps
For each object `o` of a report, in order, the page computes its route:
1. **Kind.** If `o.capability` is not dispatchable (SPEC §4: a record, a derivation, a candidate,
a companion-only or a refinement-only capability), the route is `none`
(`not_dispatchable`).
2. **Need.** If `o.need` is above the capability's registered tier, the route is `none`
(`tier_exceeds_class`, determinism-tiers §3).
3. **Constraints.** §5 over the report's sets for `o.capability` and `o.params`: if they cannot
be evaluated the route is `none` (`constraints_unevaluable`), and if no set is satisfied it is
`none` (`constraints_unmet`) — the capability is unsupported for this object, which is not a
decode error (capability-constraints §2).
4. **The UA leg.** It is taken when every check holds; otherwise its reason is the first that
fails:
1. `o.codec` is a string (`no_ua_mapping`: the object names nothing to ask a UA decoder);
2. `o.codec` passes §4.2's gate (`codec_unregistered`), whatever the capability;
3. `o.capability` has a row in §4.1 (`no_ua_mapping`);
4. `o.codec` is a string of that row's form (`codec_mismatch`);
5. the report holds an answer for `o.codec` (`ua_not_asked`), and its `supported` is true
(`ua_unsupported`);
6. the need is at most the row's UA tier (`ua_inexact`);
7. where the need is `decode-equivalence`, the answer's `probe` is `match` (`ua_unprobed`
for `not_run`, `ua_probe_mismatch` for `mismatch`).
5. **The host leg.** It is taken when `o.capability` is in `host` (`no_host`) with a tier at
least the need (`host_inexact`).
6. **The wasm leg.** It is taken when `o.capability` is in `wasm` (`no_wasm`). A wasm decoder
is WAI's own, so its output holds the registered tier, which step 2 has already held to be at
least the need.
7. **Route.** The route is the first leg taken, in the order UA, host, wasm. Where none is taken
the route is `none` (`no_decoder`), and each leg's reason is reported.
The UA leg comes first because a decoder the device ships is the one a page is expected to use,
and checks 4.6 and 4.7 take it only where its output is exact for what the object needs; the wasm
decoder is the fallback that depends on nothing the device must provide. A page **MUST** decode
an object only through its route, and **MUST NOT** decode one whose route is `none`.
Each object's entry in the **routing table** states: its `name`, its `route`, its `reason`
(`null` unless the route is `none`), each leg's outcome (`"taken"`, `"not_reached"` for a leg
after the one taken and for every leg when steps 1 to 3 decide, or the leg's reason), `asked`
(whether the page may put `o.codec` to the user agent: true exactly when steps 1 to 3 pass and
checks 4.1 to 4.4 hold), the route's `tier`, and the energy entry of §6. The entry's `reason` is
the code of the step that decided, or `no_decoder` from step 7. The route's tier is the row's UA
tier for `ua` (lowered to `none` unless the probe matched), the host's declared tier for `host`,
the registered tier for `wasm`, and `none` for `none`; in each case no higher than the
registered tier.
### 4.4 The probe
An answer of `supported` says a UA decoder accepts the configuration; it does not say the
decoder's output is the decoding process's. Where an object needs `decode-equivalence`, the page
decodes a **probe**: an object of the same capability and configuration whose decoded frames'
SHA-256 digests are published (http-delivery §12.4's probe object), each over the frame's samples
as `copyTo()` writes them with no `format`, `rect` or `layout` option (planes in order, rows
tightly packed). A frame output in `NV12` is compared after its interleaved chroma plane is split
into its two planes, which reorders samples and changes none. The probe is `match` when every
frame's digest is the published one, `mismatch` when any is not, and `not_run` when the page
decoded no probe. This revision publishes no probe for a UA codec, so a page reports `not_run`
and a need of `decode-equivalence` is never routed to a UA decoder.
### 4.5 The user agent's answer is a constraint check too
`isConfigSupported` answers for a whole configuration, before any decoder is configured, and
returns the configuration it would accept. It already does, for the parameters a configuration
carries, what capability-constraints asks of a sink: an answer before dispatch, for one object,
that is "unsupported" rather than a decode error. A page **SHOULD** build the configuration from
the object's own declared parameters, and **SHOULD** configure the decoder with the
configuration the answer returns. The two checks are separate and both apply: an object routes
`ua` only when its constraint sets are satisfied (step 3) and the user agent answered
`supported` (check 4.5). Neither covers the neural, quantum, world or haptic families, which have
no UA decoder; their constraints are evaluated by §5 alone.
### 4.6 What a page asks, and what it advertises
A page builds a report in two passes: from the objects alone it computes `asked` for each, and
puts to the user agent exactly the codec strings an entry marks asked; then it adds the answers
and computes the routes. Media Capabilities is asked only about a codec string the user agent
answered `supported` (http-delivery §12.2). A capability routed `ua`, `wasm` or `host` is, for
http-delivery §12.1's descriptor, `via` `"ua"` (with the codec strings asked and their answers,
`provisional` true), `"wasm"` or `"host"`; one routed `none` is absent from it.
### 4.7 Choosing among alternatives
Objects of a report that carry the same `choice` are **alternatives**: renditions of one content
of which the page decodes one, such as the entries of a `WAI2` envelope, a `WAI1` envelope's
capability and fallback, or the renditions of one track. For each `choice`, in order of its first
appearance in `objects`, the page puts the alternatives in SPEC §4's energy order. The order sits
inside the routing: each alternative's route is the one §4.3 gives it, and the order changes none.
- An alternative routed `ua` ranks by the user agent's answer for its codec string
([platform-decode](platform-decode.md) §3): `first` when `supported`, `smooth` and
`power_efficient` are all `true`; `smooth` when `smooth` is `true` and `power_efficient` is
not; `classical` otherwise. A `false` or a `null` is not a `true`: no answer is not a
power-efficient one.
- An alternative routed `host` or `wasm` ranks `classical`. The user agent's answers are about
its own decoder, which does not run this object, so they are not read for it, whatever they
say: AV1, whose user-agent tier is `none`, and a codec whose need is `decode-equivalence` while
no probe ran (§4.4) route elsewhere and rank on nothing the user agent said.
- An alternative whose capability is learned (SPEC §4: its replicate class is not one of the
three standard classes, or the string is not registered) ranks `classical` when the report
holds a measurement under its route's key (§6, `"<capability> <route>"`) that reads under §6
with the class `OnChipCounter` or `CalibratedInstrument` and the coverage `Whole`: a
sink-metered figure of its decode, with its class, uncertainty and scope. It ranks
`learned-without-budget` otherwise. The figure's value is not read.
- An alternative routed `none` comes after every other, with no rank.
Report order within a rank. `powerEfficient` is the user agent's hint, used as the user agent
gave it, as an order and never as an energy figure or as evidence of hardware. The **chosen**
alternative is the first in that order whose route is not `none`, and where every route is `none`
nothing is chosen (`no_alternative`). A page **MUST** decode the chosen alternative of a choice
and no other, so a learned alternative without a qualifying measurement is decoded only where no
classical alternative can be routed. For a `WAI1` envelope this is SPEC §4's order of steps 3 and
5: the chosen alternative is tried first, and the other after it if the first does not decode.
The **choice table** has one entry a choice: `choice`; `order`, the alternatives' names in that
order, those routed `none` included; `ranks`, each one's rank in the same order, `null` for one
routed `none`; `chosen`, a name or `null`; and `reason`, `null` or `no_alternative`. An object with
no `choice` is in no entry, and a report whose objects carry none has an empty table.
## 5. Constraint sets in the page
A page evaluates [capability-constraints](capability-constraints.md) §2 in this form:
- A capability's sets are a JSON array of one or more **constraint sets**. A set is a JSON
object whose members are parameter names, each mapped to a **constraint**: a JSON object with
one or more of `enum` (a non-empty array of values), `minimum` and `maximum` (integers), and no
other member. A value is a JSON string, or an integer: a JSON number with no fraction and no
exponent, from −(2^53 − 1) to 2^53 − 1. A page whose own numbers do not keep `1920.0` apart
from `1920` writes both as `1920`, so the integers it writes are exactly its integral
values; a value with a fractional part stays a fraction.
- A parameter name **MUST** be one of the register of capability-constraints §3.1.
- An object's declared parameters are a JSON object of the same names and values.
`evaluate(sets, params)` is:
1. No sets for the capability: `unconstrained`.
2. The sets break the form above — not an array, an empty array, a set that is not an object, a
constraint that is not an object, a constraint with no member or with a member other than the
three, an empty `enum` or one holding something other than a value, a `minimum` or `maximum`
that is not an integer, or a `minimum` above a `maximum`: `unevaluable`
(`constraint_malformed`).
3. A set names a parameter outside the register: `unevaluable` (`parameter_unregistered`). A
page **MUST NOT** advertise a capability whose sets it cannot evaluate (capability-constraints
§2).
4. A set is **satisfied** when, for each parameter it names, the object declares a value for it
and the value meets each member of the constraint: for `enum`, it equals one of the values,
a string only a string and an integer only an integer; for `minimum` and `maximum`, it is an
integer at least, or at most, the bound. An undeclared parameter, or a declared value that is
neither a string nor an integer, meets no constraint. An empty set is satisfied by every
object.
5. `satisfied` when one set is; otherwise `unsatisfied` (`constraints_unmet`).
The grammar is the one capability-constraints adopts, enumerations, minima and maxima ORed over
sets, with integers in place of rationals, so every evaluation is exact.
## 6. Energy beside `powerEfficient`
Each entry of the routing table carries an **energy entry**:
```json
{ "power_efficient": true | false | null, "figure": null | { … }, "reason": null | "<reason>" }
```
- `power_efficient` is Media Capabilities' `powerEfficient` for the object's codec string,
copied as the user agent gave it, for an object routed `ua`; it is `null` for any other route
and where the page did not ask. It is the user agent's word: its own specification leaves the
meaning of optimal power draw to the user agent. A page **MUST NOT** set it, invert it or
compute it from a figure, **MUST NOT** state it for a route the user agent did not answer for,
and **MUST NOT** derive a figure from it.
- `figure` is the measurement the report holds under the key `"<capability> <route>"`, where
`<route>` is the object's route, and `null` when it holds none or the route is `none`. A
measurement under another route's key is not this route's figure. A page **MAY** take a measurement
from its sink's decode table ([sink-energy](sink-energy.md) §4): a row whose status is
`figure` gives one, as that extension's §1 maps it, and a row of any other status gives
none.
A measurement is read as energy-measurement §2 has it, and refused, with `figure` `null` and the
entry's `reason`, when:
- its `class` is not one of `CalibratedInstrument`, `OnChipCounter`, `ModelBased` and `Estimator`
(`energy_class_invalid`; `HwShunt` is never emitted, energy-measurement §2.6);
- `joules_micro` is `0` (`unmetered_as_zero`): an unmetered decode has no measurement, and a
class beside a zero claims a measurement that did not occur;
- `joules_micro`, `decodes` or `window_us` is not a positive integer below 2^53
(`energy_count_invalid`);
- a hardware class lacks its evidence: `relative_ppm`, a positive integer, for
`CalibratedInstrument` and `OnChipCounter`; `filtering`, one of `On`, `Off` and `Unknown`, for
`OnChipCounter`; `calibration_ref`, a non-empty string, for `CalibratedInstrument`
(`energy_evidence_missing`);
- `coverage` is not `Whole` or `Partial`, or `fabrics` is not a non-empty array of distinct
strings (`energy_evidence_missing`).
A figure that reads is reported with every member it was given and
`per_decode_micro = ⌊joules_micro / decodes⌋`; `joules_micro` and `decodes` stay beside it, so the
floor loses nothing. An object with no measurement is **unmetered**: `figure` and `reason` are
both `null`, and a page presents it as unmetered, never as 0 J. In this revision a figure never
changes a route, and none is steered on: §4.7 reads only whether a learned alternative's figure
qualifies, never its value. How a figure is measured, bracketed and
attributed is energy-measurement §3 and §4's, and a `Partial` coverage says the meter saw only
the fabrics it names.
## 7. The content binding
### 7.1 The value
A **content binding** is a JSON object, and in a page a dictionary of the same members, the
`contentBinding` member of a frame's metadata: proposed for the WebCodecs VideoFrame Metadata
Registry, and not registered (§9):
| member | value |
|---|---|
| `alg` | the algorithm `digest` is computed with. This revision defines `jwp-object-1`: the content hash of jwp-receipts §2.1, `BLAKE3("moq-jwp:object\x01" ‖ object)`. |
| `digest` | 64 hexadecimal digits: the content hash of the receipted object the frame was decoded from. |
| `receipt` | optional; 64 hexadecimal digits: the link hash (jwp-receipts §3 step 3) of the group receipt that object verified under. |
| `tier` | the tier the frame's samples hold (§7.2). |
Its **canonical form** is the JSON text of those members in that order, `receipt` left out when
absent, with no whitespace and no other member. A reader:
- refuses a value that is not an object, that lacks `alg`, `digest` or `tier`, whose members
are not strings, whose `digest` or `receipt` is not 64 hexadecimal digits
(`binding_malformed`);
- refuses an `alg` it does not know (`binding_alg_unknown`): it cannot check the digest;
- reads a `tier` outside the vocabulary as `none`;
- ignores a member this text does not define, as a dictionary drops one.
### 7.2 When a page binds a frame
A page binds a frame it decoded, and only then, when:
1. the frame was output for a chunk whose coded data came from one receipted object, and the page
computed `digest` over that object's bytes as received, or holds it as the content hash of the
object receipt that object verified against;
2. the content hash of every object the decoder was given since its last key frame — its last
`key` chunk, or its configuration — was established: the object verified against its JWP
object receipt (jwp-receipts §3 steps 1 to 5), or it was fetched whole under that hash as its
content identifier, or through a verified head proof (http-delivery §6, §8.3). A frame
predicted from data whose hash nobody established is not bound
(`binding_unverified_reference`). A page that verifies every object before a decoder sees it
meets this by construction.
`receipt` is written exactly when the object verified against a group receipt (jwp-receipts §3
steps 1 to 5), and is that receipt's link hash; a page **MUST NOT** write a receipt it did not
verify the object against. For an object carried over MoQ, `digest` and `receipt` are then the
`content_hash` and `group_receipt` of its `WAI_RECEIPT` property
([moq-streaming-format](moq-streaming-format.md) §5). `tier` is the route's tier (§4.3),
lowered to `none` where:
- the route is `ua` and the frame's `format` is `null`: its samples cannot be read without a
conversion, so nothing can check them;
- the page altered the samples after the decode (determinism-tiers §2, *What a tier covers*).
Where the frame's tier is below the object's need, the page **MUST NOT** present the frame as
meeting it.
On the wasm route the wasm decoder writes the binding itself: it holds the object's bytes, hashes
them, decodes them, and returns every frame with the binding.
## 8. Through the frame's life
| operation | the binding |
|---|---|
| a decoder's output | none from the decoder: WebCodecs' *Create a VideoFrame* takes no metadata. The page binds it (§7.2, §9). |
| `clone()` | carried: the clone references the same samples, and *Clone VideoFrame* assigns it every slot but the resource reference, `[[metadata]]` among them. |
| serialization (`postMessage` without transfer) | carried: the serialization steps copy `[[metadata]]`. |
| transfer | carried to the receiving realm; the source is closed and keeps none. |
| `close()` | gone with the frame. |
| `copyTo()` | the bytes written carry none. With no `format` option they are the frame's samples, and a page **MAY** check them against a published digest (§4.4); with one they are a conversion, which no tier covers. |
| `new VideoFrame(frame, init)` | carried only when the page puts it in `init.metadata`: the constructor copies `init`'s metadata, not the source's. The new frame references the source's samples; a page **MAY** copy the binding into it, and a narrower `visibleRect` presents part of the same samples, from the same object. |
| `new VideoFrame` from a canvas, an image or a buffer | a frame of other samples: none, except on the wasm route, where the samples are the ones the wasm decoder output from the bound object (§7.2). |
| drawing the frame (`drawImage`, a texture) | the drawn pixels are a rendering and carry none; §10 says which frame a region of the screen shows. |
| `VideoEncoder.encode()`, and a decode of its output | **does not survive.** A re-encoded frame is a new object: its chunks carry no binding, and a frame decoded from them is bound, if at all, to that new object and its own receipt. |
## 9. Carriage in a page
### 9.1 As metadata
Where the user agent's `VideoFrameMetadata` has a `contentBinding` member, a page binds a decoded
frame `f` by constructing `new VideoFrame(f, { metadata: { ...f.metadata(), contentBinding } })`
and closing `f`. `clone()`, serialization and transfer then carry it by WebCodecs' own steps (§8).
### 9.2 In a side table
`VideoFrameMetadata` is a dictionary whose members are the registry's, and a dictionary keeps only
the members it declares, so a member no registry lists is dropped from `init.metadata`. Until
`contentBinding` is listed, a page carries the binding in a side table:
1. **Pending.** Before it gives a decoder a chunk, the page enters the chunk's binding under (that
decoder, the chunk's `timestamp`). A decoder outputs a frame with the timestamp of the chunk it
came from (WebCodecs *Output VideoFrames*), and nothing else in a frame says which chunk that
was. Where a second chunk is given the same decoder under a timestamp still pending, both
entries are struck and neither frame is bound (`binding_ambiguous`). A page bounds the entries
pending per decoder, and strikes the oldest past the bound.
2. **Output.** On a decoder's output, the page takes the entry under (that decoder, the frame's
timestamp), removes it, and attaches it to the frame object, held weakly so that it lives as
long as the frame does. An output with no entry is unbound.
3. **Reset.** `reset()`, `close()` and a decoder error strike that decoder's pending entries, and
so does `flush()` once it resolves, for the entries whose frames it did not output; a frame
output afterwards for a struck entry is unbound.
4. **Clone.** `clone()` attaches the source's binding to the clone.
5. **Transfer.** A frame posted to another realm is posted with its binding beside it, and the
receiver attaches it on arrival. A frame that arrives without one is unbound.
6. **Close.** `close()` detaches it.
A page **MAY** test whether the user agent keeps the member (construct a frame with it in
`init.metadata` and read `metadata()`), and use §9.1 where it does.
## 10. The frame on screen
For each region of the screen a page draws frames into (a tile), the page keeps the binding of the
frame it last drew there. The **state** of that frame is computed from:
- the binding, or none;
- `object`: `verified` (the object verified against its receipt), `refused`, or `unverified` (no
receipt has been checked);
- `chain`: the group receipt's admission to its chain (jwp-receipts §3 step 3): `linked`,
`anchor`, `history`, `held` (not yet admitted: `parent_not_held`) or `refused`;
- `provenance`: for a track with a provenance track (live-provenance §10.2), the verdict of
live-provenance §11.4 on the segment of the frame's group, `accepted` or `refused`, or `missing`
while its provenance object has not arrived; `not_applicable` for a track packaged as WAI
envelopes, which has no C2PA provenance (live-provenance §1).
The state is, in this order:
1. `unbound`, with no binding or with one the reader refuses (§7.1);
2. `refused`, when `object`, `chain` or `provenance` is `refused`;
3. `pending`, when `object` is `unverified`, `chain` is `held` or `provenance` is `missing`;
4. `verified` otherwise.
`anchor` and `history` are admissions: a chain a joiner anchored, or filled below its anchor. A
page that shows `verified` for them **MUST** also say from which group it holds the chain, as
jwp-receipts §3 step 3 requires of a verifier that re-anchors, and **MUST NOT** state the groups
before it verified.
A page that shows the state, behind a control the reader turns on, shows for each tile the state,
the tier, the first digits of the digest and of the receipt, the provenance, the route and the
energy entry. The state is about the frame the tile last drew, not about the pixels on the
display: what the display emits is a rendering of the frame, scaled and converted to the
display's colour space, which no tier covers.
## 11. Reasons
| reason | raised at |
|---|---|
| `report_malformed` | §3 |
| `not_dispatchable` | §4.3 step 1 |
| `tier_exceeds_class` | §4.3 step 2 |
| `constraints_unevaluable` | §4.3 step 3 |
| `constraints_unmet` | §4.3 step 3, §5 step 5 |
| `no_ua_mapping` | §4.3 steps 4.1, 4.3 |
| `codec_mismatch` | §4.3 step 4.4 |
| `codec_unregistered` | §4.2, §4.3 step 4.2 |
| `ua_not_asked` | §4.3 step 4.5 |
| `ua_unsupported` | §4.3 step 4.5 |
| `ua_inexact` | §4.3 step 4.6 |
| `ua_unprobed` | §4.3 step 4.7 |
| `ua_probe_mismatch` | §4.3 step 4.7 |
| `no_host` | §4.3 step 5 |
| `host_inexact` | §4.3 step 5 |
| `no_wasm` | §4.3 step 6 |
| `no_decoder` | §4.3 step 7 |
| `no_alternative` | §4.7 |
| `constraint_malformed` | §5 step 2 |
| `parameter_unregistered` | §5 step 3 |
| `energy_class_invalid` | §6 |
| `unmetered_as_zero` | §6 |
| `energy_count_invalid` | §6 |
| `energy_evidence_missing` | §6 |
| `binding_malformed` | §7.1 |
| `binding_alg_unknown` | §7.1 |
| `binding_unverified_reference` | §7.2 |
| `binding_ambiguous` | §9.2 step 1 |
An object routed `none` is inert at the sink for that object, as SPEC §4 step 6 has it; a refused
energy measurement leaves the route as it is; a refused binding leaves the frame unbound.
## 12. Conformance
[`live-conformance/browser/`](../live-conformance/browser/) holds the corpus, `vectors.json`,
written by `wai_browser_sink_vectors gen`. It has:
- **Routes**: capability reports, each with the routing table §4 and §6 give it: every route and
every reason of §4.3, the registry gate with codec strings for AV2, VVC and IAMF answered
`supported` and never read, a need of `decode-equivalence` refused a UA decoder whose probe did
not run and taken by one whose probe matched, the host and wasm fallbacks, and a malformed
report.
- **Choices**: reports whose objects are alternatives, each with its routing table and the choice
table §4.7 gives it: `first`, `smooth` and `classical` before `learned-without-budget`; a
power-efficient answer that is not smooth below a smooth one; a power-efficient answer for a
codec routed to a host decoder (its need above the user agent's tier) read for nothing; a
learned alternative with a qualifying measurement (`OnChipCounter`, `CalibratedInstrument`)
ranked `classical`, and with a `Partial` or a `ModelBased` one ranked last; a learned
alternative chosen where no other routes; no answer read as not power-efficient; two choices
beside an object in none; a choice with nothing to choose (`no_alternative`); and an empty and a
non-string `choice`, each refused whole. Every measurement in them is synthetic, marked
`"note": "synthetic: no measurement"`.
- **Constraints**: constraint sets and declared parameters, with §5's outcome: satisfied by one
set of several, by an empty set, by `enum`, `minimum` and `maximum`; unsatisfied by an
undeclared parameter and by a value of the wrong type; each malformed form and an unregistered
parameter.
- **Energy**, among the routes: measurements beside answers, each with its energy entry: every
class, the unmetered case, a figure under another route's key, and each refusal of §6.
- **Bindings**: values with their reading: canonical forms, each refusal of §7.1, an unknown tier
read as `none`, and a member the text does not define; the tier §7.2 gives a frame for each
route, format and alteration; and objects bound under a signed group
receipt, each binding's `digest` the object's content hash and its `receipt` the receipt's link
hash, the receipt's signature and the object's inclusion verifiable from the corpus.
- **Frames**: inputs to §10 with their state, covering every row.
- **Side tables**: sequences of chunks, outputs, clones, transfers and resets with the bindings
§9.2 leaves each frame, `binding_ambiguous` among them.
`wai_browser_sink_vectors verify` and the test `tests/browser_sink.rs` check that `vectors.json`
regenerates byte for byte, that `wai-rs` gives every case its stated outcome, that every reason of
§11 appears, and that §11 lists exactly the reference's codes.
`live-conformance/browser_sink_verify.py` is the second implementation, written from this text:
it shares no code with `wai-rs`, reads each capability's kind, status and replicate class from the
registry export ([`registry/capabilities.json`](../registry/capabilities.json)), checks
signatures with WAI's strict rule and hashes with BLAKE3 from `claim-conformance/`, and mutates
copies of the corpus and requires every copy to fail. The page runs the route, choice, constraint,
energy, binding and frame cases through its wasm build, and its side table through the sequences, in a
browser (`joule-moq-web`'s end-to-end scenario `binding`).
## 13. Security and privacy considerations
- **A binding is a statement, not a signature.** Any script in the page can bind any frame. What
a binding gives is where to look: the receipt it names commits to the digest under a key, and a
party that holds the object, the receipt and its key log checks them. The state of §10 is the
page's own verdict on what it checked.
- **What `verified` covers.** The object and its group receipt, and, for a CMAF track, the
segment's C2PA provenance. It does not cover what the display shows, how a frame was scaled, or a
user agent's decoder: a UA decoder that misdecodes a verified object outputs a frame bound to that
object. §4.4's probe is how a page holds a UA decoder to the decoding process where the tier
matters.
- **What a report reveals.** A report holds the user agent's answers and, with a meter attached,
an energy figure; together they say a good deal about a device (Media Capabilities, privacy
considerations). Nothing in this extension sends a report anywhere: routing, binding and the
state of a frame are computed in the page. A service that collects reports learns that much
about each device it collects from; where it needs to record a legal basis for doing so,
jwp-receipts §2.4.6's `lawful-basis` claim is the place.
- **An energy figure is a side channel.** A per-process energy reading, taken often enough, leaks
what a process does. A page receives a figure only from a meter the host chose to attach, at the
granularity the host chose, and this extension never has a page read a counter itself.
- **Ambiguity is refused, not guessed.** A side table keyed by timestamp cannot tell two chunks of
one decoder with one timestamp apart, so neither frame is bound (§9.2).
- **Unregistered codec strings are not asked about.** Besides having no defined answer, a question
about a codec string outside the registry adds to what a page learns about a device for nothing
it can use.
Reference in a new issue View git blame Copy permalink