158 lines
10 KiB
Markdown
158 lines
10 KiB
Markdown
# Proposal: `contentBinding`, a VideoFrame metadata member that ties a decoded frame to the content it was decoded from
|
||
|
||
> **Status: draft. Not filed.** Whether, when and where to file it is the owner's decision.
|
||
> This file holds what a filer would need: the gap, a draft registry entry, a draft of the
|
||
> defining text, the issue text ready to paste, what was tested and what is still open. The
|
||
> normative WAI text, which a page that uses WAI's receipts follows today, is
|
||
> [`extensions/browser-sink.md`](../extensions/browser-sink.md) §7–§10.
|
||
>
|
||
> Texts read, each pinned to the version named: WebCodecs, W3C Working Draft 19 October 2026;
|
||
> the WebCodecs VideoFrame Metadata Registry, W3C Registry Draft 14 April 2026; the WebCodecs
|
||
> Codec Registry, W3C Registry Draft 24 September 2026; Media Capabilities, W3C Working Draft
|
||
> 18 October 2026.
|
||
|
||
## 1. The gap
|
||
|
||
A page can verify the bytes it fetched: a segment against a signed manifest, an object against
|
||
a receipt. Nothing ties that verification to the frame the page then shows. Between the two
|
||
sit a decoder, which outputs frames that name no input but a timestamp, and the page's own
|
||
handling, in which a frame is cloned, transferred to a worker, wrapped in a new `VideoFrame`,
|
||
drawn and re-encoded. A page that wants to say "the frame on screen is the one decoded from
|
||
the bytes I verified" has to keep that association itself, keyed by timestamp, and loses it at
|
||
every realm boundary unless it carries it beside the frame by hand.
|
||
|
||
`VideoFrameMetadata` is where WebCodecs carries per-frame facts through exactly those
|
||
operations: *Clone VideoFrame* assigns `[[metadata]]` to the clone, and the serialization and
|
||
transfer steps carry it across realms. Its registry lists capture-side facts (`captureTime`,
|
||
`receiveTime`, `rtpTimestamp`, segmentation and blur state). It has no member that says what
|
||
encoded content a frame was decoded from.
|
||
|
||
## 2. The registry's requirements, read against this proposal
|
||
|
||
The registry (Registry Draft, 14 April 2026, *Registration*) asks for an issue in the
|
||
WebCodecs issue tracker, interest from at least one implementer, Media Working Group consensus,
|
||
and an entry that:
|
||
|
||
| requirement | this proposal |
|
||
|---|---|
|
||
| is a separate `VideoFrameMetadata` member | `contentBinding`, one member whose value is a dictionary |
|
||
| is serializable | its members are strings; a dictionary of strings is serializable |
|
||
| is defined by a W3C specification that has reached consensus in its originating Working Group | **not met by this document.** WAI's extension is a profile of a producer, not a W3C specification. The issue asks the group where a definition belongs (a section of WebCodecs, a Media WG note, or another group's document) |
|
||
| has clear semantics, including how it interacts with encoders, decoders and renderers | §4 below |
|
||
|
||
## 3. Draft registry entry
|
||
|
||
| Member name | Public specification |
|
||
|---|---|
|
||
| `contentBinding` | *(to be decided with the Media WG: the document that defines §4 below)* |
|
||
|
||
## 4. Draft defining text
|
||
|
||
```webidl
|
||
dictionary VideoFrameContentBinding {
|
||
required DOMString alg;
|
||
required DOMString digest;
|
||
DOMString receipt;
|
||
DOMString tier;
|
||
};
|
||
|
||
partial dictionary VideoFrameMetadata {
|
||
VideoFrameContentBinding contentBinding;
|
||
};
|
||
```
|
||
|
||
- `alg` names the algorithm `digest` was computed with **and which bytes it covers**. Values
|
||
are defined by the specification of whoever writes the binding; a reader that does not know
|
||
`alg` cannot check `digest` and treats the binding as absent. (WAI's value is `jwp-object-1`:
|
||
BLAKE3 over a domain string and the encoded object as received.)
|
||
- `digest` is the lowercase hexadecimal digest of the encoded object the frame was decoded
|
||
from: the bytes the page received and, where it checked them, verified.
|
||
- `receipt`, optional, is the lowercase hexadecimal identity of a signed record that commits to
|
||
`digest` and that the page verified the object against. Absent when the page verified nothing
|
||
beyond the digest.
|
||
- `tier`, optional, names how reproducible the frame's samples are as the writer's
|
||
specification defines it (WAI's: `decode-equivalence`, `entropy-consistency`, `none`); a reader
|
||
that does not know the token reads it as no claim.
|
||
|
||
**Semantics through the pipeline.**
|
||
|
||
| operation | the member |
|
||
|---|---|
|
||
| a decoder's output | not set by the decoder (*Create a VideoFrame* takes no metadata). A page sets it by constructing `new VideoFrame(output, { metadata: { ...output.metadata(), contentBinding } })` |
|
||
| `clone()` | carried (*Clone VideoFrame* assigns `[[metadata]]`) |
|
||
| serialization and transfer | carried (the serialization and transfer steps carry `[[metadata]]`) |
|
||
| `new VideoFrame(frame, init)` | what `init.metadata` holds, as for every member: a page copies it when the new frame presents the same samples |
|
||
| `new VideoFrame` from a canvas, an image or a buffer | not set, unless the page wrote those samples itself from the bound content (a decoder implemented in WebAssembly, say) |
|
||
| `copyTo()` | the bytes written carry nothing; with no `format` option they are the frame's samples, which a page may hash |
|
||
| `VideoEncoder.encode()` | **does not survive.** An encoded chunk is new content; a frame decoded from it is bound, if at all, to that content |
|
||
| rendering (`drawImage`, a texture) | the rendered pixels carry nothing; a page that wants to say which frame a region shows keeps that itself |
|
||
|
||
**What it is not.** It is not a signature and proves nothing by itself: any script that can
|
||
construct a frame can write any value. It says where to look — which content, under which
|
||
receipt — so that a party holding that content and that receipt can check them, and so that the
|
||
association survives the operations above without each page re-implementing a side table.
|
||
|
||
## 5. Draft issue text
|
||
|
||
> **Title:** VideoFrame metadata registry: `contentBinding`, the content a decoded frame came from
|
||
>
|
||
> **Problem.** A page that verifies the encoded media it receives (against a signed manifest,
|
||
> a receipt, a content address) cannot carry that verification to the frames it decodes.
|
||
> Decoder outputs identify their input only by timestamp, and a page that keeps its own
|
||
> association loses it at `clone()`, at `new VideoFrame(frame)`, and at every
|
||
> `postMessage` to a worker unless it posts the association beside each frame by hand.
|
||
> `VideoFrameMetadata` is the dictionary WebCodecs already carries through clone,
|
||
> serialization and transfer; it has no member for what content a frame was decoded from.
|
||
>
|
||
> **Proposal.** A registry entry `contentBinding`: a dictionary of strings `{ alg, digest,
|
||
> receipt?, tier? }` naming the digest algorithm and the digest of the encoded object the frame
|
||
> was decoded from, optionally the identity of a signed record committing to that digest which
|
||
> the page verified, and optionally a reproducibility claim for the samples. Semantics: set by
|
||
> the page (not by the decoder) on a decoder's output; carried by `clone()`, serialization and
|
||
> transfer as every member is; not carried into a re-encoded chunk or anything decoded from it;
|
||
> carries nothing into `copyTo()` output or rendered pixels. A draft IDL and a table of the
|
||
> pipeline semantics are attached.
|
||
>
|
||
> **Prior experience.** We implemented the semantics as a page-side polyfill and tested it in a
|
||
> current user agent under automation: a side table keyed by (decoder, timestamp), with
|
||
> `clone()` and `close()` patched and the binding posted beside each transferred frame. It
|
||
> works, with three costs a registered member would remove: the page must refuse to bind when
|
||
> one decoder is given two chunks with one timestamp (nothing else says which chunk a frame
|
||
> came from); every realm needs its own table and every `postMessage` its own convention; and
|
||
> code that does not know the table (a library, a worker the page did not write) drops the
|
||
> binding silently. In the user agent tested, `metadata()` returned an empty dictionary for a
|
||
> frame constructed with `init.metadata` holding `contentBinding`, through both the buffer and
|
||
> the frame constructor, as a dictionary that does not declare a member drops it; it was also
|
||
> empty for `rtpTimestamp`, a registered member, given the same way.
|
||
>
|
||
> **Questions for the group.** (1) Which W3C document should define the member, given the
|
||
> registry's requirement that its definition reach consensus in a Working Group? (2) Strings or
|
||
> `ArrayBuffer`s for `digest` and `receipt`? (Strings are immutable and trivially
|
||
> serializable; buffers avoid hex.) (3) Should `alg` values be drawn from an existing registry of
|
||
> hash algorithm names, given that a value must also say which bytes are covered? (4) Is there
|
||
> interest in letting a decoder carry the member from an `EncodedVideoChunk` to its output
|
||
> frame, which would remove the timestamp matching altogether? That is beyond a registry entry
|
||
> and is asked only to learn whether the group sees it as in scope.
|
||
|
||
## 6. What was tested
|
||
|
||
`joule-moq-web`'s end-to-end scenario `binding` drives a current user agent headless (its
|
||
version is in the run's log) against the page `web/binding/` and a room with `?debug=binding`:
|
||
|
||
- the side table over the user agent's own VP9 decoder: every output bound to the chunk it came
|
||
from by decoder and timestamp; two chunks under one timestamp refused (`binding_ambiguous`);
|
||
- a WebAssembly decoder (`wai.video.int_motion`) that verifies each object against its signed
|
||
receipts, decodes it and binds every frame; each frame's SHA-256 equals the published one;
|
||
- the binding through `clone()`, `close()`, `new VideoFrame(frame)`, transfer to a worker and
|
||
back, `copyTo()` with and without a format, and re-encode, as §4's table has it;
|
||
- `init.metadata` with `contentBinding` and with `rtpTimestamp`: the result quoted in §5;
|
||
- in a two-party room, the frame on each tile bound to an object the page verified, under that
|
||
object's group receipt, and shown as verified.
|
||
|
||
## 7. Open
|
||
|
||
- Whether the group wants the member at all, and where its definition lives (§2).
|
||
- No audio counterpart is proposed: `AudioData` has no metadata dictionary.
|
||
- A frame predicted from earlier frames depends on their content too. WAI's text has a page
|
||
bind only when every object the decoder was given since its last key frame was verified; a
|
||
registered member's definition would need to say whether it names one object or a chain.
|