# Proposal: `contentBinding`, a VideoFrame metadata member that ties a decoded frame to the content it was decoded from

> **Status: draft. Not filed.** Whether, when and where to file it is the owner's decision.
> This file holds what a filer would need: the gap, a draft registry entry, a draft of the
> defining text, the issue text ready to paste, what was tested and what is still open. The
> normative WAI text, which a page that uses WAI's receipts follows today, is
> [`extensions/browser-sink.md`](../extensions/browser-sink.md) §7–§10.
>
> Texts read, each pinned to the version named: WebCodecs, W3C Working Draft 19 October 2026;
> the WebCodecs VideoFrame Metadata Registry, W3C Registry Draft 14 April 2026; the WebCodecs
> Codec Registry, W3C Registry Draft 24 September 2026; Media Capabilities, W3C Working Draft
> 18 October 2026.

## 1. The gap

A page can verify the bytes it fetched: a segment against a signed manifest, an object against
a receipt. Nothing ties that verification to the frame the page then shows. Between the two
sit a decoder, which outputs frames that name no input but a timestamp, and the page's own
handling, in which a frame is cloned, transferred to a worker, wrapped in a new `VideoFrame`,
drawn and re-encoded. A page that wants to say "the frame on screen is the one decoded from
the bytes I verified" has to keep that association itself, keyed by timestamp, and loses it at
every realm boundary unless it carries it beside the frame by hand.

`VideoFrameMetadata` is where WebCodecs carries per-frame facts through exactly those
operations: *Clone VideoFrame* assigns `[[metadata]]` to the clone, and the serialization and
transfer steps carry it across realms. Its registry lists capture-side facts (`captureTime`,
`receiveTime`, `rtpTimestamp`, segmentation and blur state). It has no member that says what
encoded content a frame was decoded from.

## 2. The registry's requirements, read against this proposal

The registry (Registry Draft, 14 April 2026, *Registration*) asks for an issue in the
WebCodecs issue tracker, interest from at least one implementer, Media Working Group consensus,
and an entry that:

| requirement | this proposal |
|---|---|
| is a separate `VideoFrameMetadata` member | `contentBinding`, one member whose value is a dictionary |
| is serializable | its members are strings; a dictionary of strings is serializable |
| is defined by a W3C specification that has reached consensus in its originating Working Group | **not met by this document.** WAI's extension is a profile of a producer, not a W3C specification. The issue asks the group where a definition belongs (a section of WebCodecs, a Media WG note, or another group's document) |
| has clear semantics, including how it interacts with encoders, decoders and renderers | §4 below |

## 3. Draft registry entry

| Member name | Public specification |
|---|---|
| `contentBinding` | *(to be decided with the Media WG: the document that defines §4 below)* |

## 4. Draft defining text

```webidl
dictionary VideoFrameContentBinding {
  required DOMString alg;
  required DOMString digest;
  DOMString receipt;
  DOMString tier;
};

partial dictionary VideoFrameMetadata {
  VideoFrameContentBinding contentBinding;
};
```

- `alg` names the algorithm `digest` was computed with **and which bytes it covers**. Values
  are defined by the specification of whoever writes the binding; a reader that does not know
  `alg` cannot check `digest` and treats the binding as absent. (WAI's value is `jwp-object-1`:
  BLAKE3 over a domain string and the encoded object as received.)
- `digest` is the lowercase hexadecimal digest of the encoded object the frame was decoded
  from: the bytes the page received and, where it checked them, verified.
- `receipt`, optional, is the lowercase hexadecimal identity of a signed record that commits to
  `digest` and that the page verified the object against. Absent when the page verified nothing
  beyond the digest.
- `tier`, optional, names how reproducible the frame's samples are as the writer's
  specification defines it (WAI's: `decode-equivalence`, `entropy-consistency`, `none`); a reader
  that does not know the token reads it as no claim.

**Semantics through the pipeline.**

| operation | the member |
|---|---|
| a decoder's output | not set by the decoder (*Create a VideoFrame* takes no metadata). A page sets it by constructing `new VideoFrame(output, { metadata: { ...output.metadata(), contentBinding } })` |
| `clone()` | carried (*Clone VideoFrame* assigns `[[metadata]]`) |
| serialization and transfer | carried (the serialization and transfer steps carry `[[metadata]]`) |
| `new VideoFrame(frame, init)` | what `init.metadata` holds, as for every member: a page copies it when the new frame presents the same samples |
| `new VideoFrame` from a canvas, an image or a buffer | not set, unless the page wrote those samples itself from the bound content (a decoder implemented in WebAssembly, say) |
| `copyTo()` | the bytes written carry nothing; with no `format` option they are the frame's samples, which a page may hash |
| `VideoEncoder.encode()` | **does not survive.** An encoded chunk is new content; a frame decoded from it is bound, if at all, to that content |
| rendering (`drawImage`, a texture) | the rendered pixels carry nothing; a page that wants to say which frame a region shows keeps that itself |

**What it is not.** It is not a signature and proves nothing by itself: any script that can
construct a frame can write any value. It says where to look — which content, under which
receipt — so that a party holding that content and that receipt can check them, and so that the
association survives the operations above without each page re-implementing a side table.

## 5. Draft issue text

> **Title:** VideoFrame metadata registry: `contentBinding`, the content a decoded frame came from
>
> **Problem.** A page that verifies the encoded media it receives (against a signed manifest,
> a receipt, a content address) cannot carry that verification to the frames it decodes.
> Decoder outputs identify their input only by timestamp, and a page that keeps its own
> association loses it at `clone()`, at `new VideoFrame(frame)`, and at every
> `postMessage` to a worker unless it posts the association beside each frame by hand.
> `VideoFrameMetadata` is the dictionary WebCodecs already carries through clone,
> serialization and transfer; it has no member for what content a frame was decoded from.
>
> **Proposal.** A registry entry `contentBinding`: a dictionary of strings `{ alg, digest,
> receipt?, tier? }` naming the digest algorithm and the digest of the encoded object the frame
> was decoded from, optionally the identity of a signed record committing to that digest which
> the page verified, and optionally a reproducibility claim for the samples. Semantics: set by
> the page (not by the decoder) on a decoder's output; carried by `clone()`, serialization and
> transfer as every member is; not carried into a re-encoded chunk or anything decoded from it;
> carries nothing into `copyTo()` output or rendered pixels. A draft IDL and a table of the
> pipeline semantics are attached.
>
> **Prior experience.** We implemented the semantics as a page-side polyfill and tested it in a
> current user agent under automation: a side table keyed by (decoder, timestamp), with
> `clone()` and `close()` patched and the binding posted beside each transferred frame. It
> works, with three costs a registered member would remove: the page must refuse to bind when
> one decoder is given two chunks with one timestamp (nothing else says which chunk a frame
> came from); every realm needs its own table and every `postMessage` its own convention; and
> code that does not know the table (a library, a worker the page did not write) drops the
> binding silently. In the user agent tested, `metadata()` returned an empty dictionary for a
> frame constructed with `init.metadata` holding `contentBinding`, through both the buffer and
> the frame constructor, as a dictionary that does not declare a member drops it; it was also
> empty for `rtpTimestamp`, a registered member, given the same way.
>
> **Questions for the group.** (1) Which W3C document should define the member, given the
> registry's requirement that its definition reach consensus in a Working Group? (2) Strings or
> `ArrayBuffer`s for `digest` and `receipt`? (Strings are immutable and trivially
> serializable; buffers avoid hex.) (3) Should `alg` values be drawn from an existing registry of
> hash algorithm names, given that a value must also say which bytes are covered? (4) Is there
> interest in letting a decoder carry the member from an `EncodedVideoChunk` to its output
> frame, which would remove the timestamp matching altogether? That is beyond a registry entry
> and is asked only to learn whether the group sees it as in scope.

## 6. What was tested

`joule-moq-web`'s end-to-end scenario `binding` drives a current user agent headless (its
version is in the run's log) against the page `web/binding/` and a room with `?debug=binding`:

- the side table over the user agent's own VP9 decoder: every output bound to the chunk it came
  from by decoder and timestamp; two chunks under one timestamp refused (`binding_ambiguous`);
- a WebAssembly decoder (`wai.video.int_motion`) that verifies each object against its signed
  receipts, decodes it and binds every frame; each frame's SHA-256 equals the published one;
- the binding through `clone()`, `close()`, `new VideoFrame(frame)`, transfer to a worker and
  back, `copyTo()` with and without a format, and re-encode, as §4's table has it;
- `init.metadata` with `contentBinding` and with `rtpTimestamp`: the result quoted in §5;
- in a two-party room, the frame on each tile bound to an object the page verified, under that
  object's group receipt, and shown as verified.

## 7. Open

- Whether the group wants the member at all, and where its definition lives (§2).
- No audio counterpart is proposed: `AudioData` has no metadata dictionary.
- A frame predicted from earlier frames depends on their content too. WAI's text has a page
  bind only when every object the decoder was given since its last key frame was verified; a
  registered member's definition would need to say whether it names one object or a chain.