open-standards/wai/proposals/webcodecs-content-binding.md

Proposal: contentBinding, a VideoFrame metadata member that ties a decoded frame to the content it was decoded from

Status: draft. Not filed. Whether, when and where to file it is the owner's decision. This file holds what a filer would need: the gap, a draft registry entry, a draft of the defining text, the issue text ready to paste, what was tested and what is still open. The normative WAI text, which a page that uses WAI's receipts follows today, is extensions/browser-sink.md §7–§10.

Texts read, each pinned to the version named: WebCodecs, W3C Working Draft 19 October 2026; the WebCodecs VideoFrame Metadata Registry, W3C Registry Draft 14 April 2026; the WebCodecs Codec Registry, W3C Registry Draft 24 September 2026; Media Capabilities, W3C Working Draft 18 October 2026.

1. The gap

A page can verify the bytes it fetched: a segment against a signed manifest, an object against a receipt. Nothing ties that verification to the frame the page then shows. Between the two sit a decoder, which outputs frames that name no input but a timestamp, and the page's own handling, in which a frame is cloned, transferred to a worker, wrapped in a new VideoFrame, drawn and re-encoded. A page that wants to say "the frame on screen is the one decoded from the bytes I verified" has to keep that association itself, keyed by timestamp, and loses it at every realm boundary unless it carries it beside the frame by hand.

VideoFrameMetadata is where WebCodecs carries per-frame facts through exactly those operations: Clone VideoFrame assigns [[metadata]] to the clone, and the serialization and transfer steps carry it across realms. Its registry lists capture-side facts (captureTime, receiveTime, rtpTimestamp, segmentation and blur state). It has no member that says what encoded content a frame was decoded from.

2. The registry's requirements, read against this proposal

The registry (Registry Draft, 14 April 2026, Registration) asks for an issue in the WebCodecs issue tracker, interest from at least one implementer, Media Working Group consensus, and an entry that:

requirement	this proposal
is a separate VideoFrameMetadata member	contentBinding, one member whose value is a dictionary
is serializable	its members are strings; a dictionary of strings is serializable
is defined by a W3C specification that has reached consensus in its originating Working Group	not met by this document. WAI's extension is a profile of a producer, not a W3C specification. The issue asks the group where a definition belongs (a section of WebCodecs, a Media WG note, or another group's document)
has clear semantics, including how it interacts with encoders, decoders and renderers	§4 below

3. Draft registry entry

Member name	Public specification
contentBinding	(to be decided with the Media WG: the document that defines §4 below)

4. Draft defining text

dictionary VideoFrameContentBinding {
  required DOMString alg;
  required DOMString digest;
  DOMString receipt;
  DOMString tier;
};

partial dictionary VideoFrameMetadata {
  VideoFrameContentBinding contentBinding;
};

• alg names the algorithm digest was computed with and which bytes it covers. Values are defined by the specification of whoever writes the binding; a reader that does not know alg cannot check digest and treats the binding as absent. (WAI's value is jwp-object-1: BLAKE3 over a domain string and the encoded object as received.)
• digest is the lowercase hexadecimal digest of the encoded object the frame was decoded from: the bytes the page received and, where it checked them, verified.
• receipt, optional, is the lowercase hexadecimal identity of a signed record that commits to digest and that the page verified the object against. Absent when the page verified nothing beyond the digest.
• tier, optional, names how reproducible the frame's samples are as the writer's specification defines it (WAI's: decode-equivalence, entropy-consistency, none); a reader that does not know the token reads it as no claim.

Semantics through the pipeline.

operation	the member
a decoder's output	not set by the decoder (Create a VideoFrame takes no metadata). A page sets it by constructing new VideoFrame(output, { metadata: { ...output.metadata(), contentBinding } })
clone()	carried (Clone VideoFrame assigns [[metadata]])
serialization and transfer	carried (the serialization and transfer steps carry [[metadata]])
new VideoFrame(frame, init)	what init.metadata holds, as for every member: a page copies it when the new frame presents the same samples
new VideoFrame from a canvas, an image or a buffer	not set, unless the page wrote those samples itself from the bound content (a decoder implemented in WebAssembly, say)
copyTo()	the bytes written carry nothing; with no format option they are the frame's samples, which a page may hash
VideoEncoder.encode()	does not survive. An encoded chunk is new content; a frame decoded from it is bound, if at all, to that content
rendering (drawImage, a texture)	the rendered pixels carry nothing; a page that wants to say which frame a region shows keeps that itself

What it is not. It is not a signature and proves nothing by itself: any script that can construct a frame can write any value. It says where to look — which content, under which receipt — so that a party holding that content and that receipt can check them, and so that the association survives the operations above without each page re-implementing a side table.

5. Draft issue text

Title: VideoFrame metadata registry: contentBinding, the content a decoded frame came from

Problem. A page that verifies the encoded media it receives (against a signed manifest, a receipt, a content address) cannot carry that verification to the frames it decodes. Decoder outputs identify their input only by timestamp, and a page that keeps its own association loses it at clone(), at new VideoFrame(frame), and at every postMessage to a worker unless it posts the association beside each frame by hand. VideoFrameMetadata is the dictionary WebCodecs already carries through clone, serialization and transfer; it has no member for what content a frame was decoded from.

Proposal. A registry entry contentBinding: a dictionary of strings { alg, digest, receipt?, tier? } naming the digest algorithm and the digest of the encoded object the frame was decoded from, optionally the identity of a signed record committing to that digest which the page verified, and optionally a reproducibility claim for the samples. Semantics: set by the page (not by the decoder) on a decoder's output; carried by clone(), serialization and transfer as every member is; not carried into a re-encoded chunk or anything decoded from it; carries nothing into copyTo() output or rendered pixels. A draft IDL and a table of the pipeline semantics are attached.

Prior experience. We implemented the semantics as a page-side polyfill and tested it in a current user agent under automation: a side table keyed by (decoder, timestamp), with clone() and close() patched and the binding posted beside each transferred frame. It works, with three costs a registered member would remove: the page must refuse to bind when one decoder is given two chunks with one timestamp (nothing else says which chunk a frame came from); every realm needs its own table and every postMessage its own convention; and code that does not know the table (a library, a worker the page did not write) drops the binding silently. In the user agent tested, metadata() returned an empty dictionary for a frame constructed with init.metadata holding contentBinding, through both the buffer and the frame constructor, as a dictionary that does not declare a member drops it; it was also empty for rtpTimestamp, a registered member, given the same way.

Questions for the group. (1) Which W3C document should define the member, given the registry's requirement that its definition reach consensus in a Working Group? (2) Strings or ArrayBuffers for digest and receipt? (Strings are immutable and trivially serializable; buffers avoid hex.) (3) Should alg values be drawn from an existing registry of hash algorithm names, given that a value must also say which bytes are covered? (4) Is there interest in letting a decoder carry the member from an EncodedVideoChunk to its output frame, which would remove the timestamp matching altogether? That is beyond a registry entry and is asked only to learn whether the group sees it as in scope.

6. What was tested

joule-moq-web's end-to-end scenario binding drives a current user agent headless (its version is in the run's log) against the page web/binding/ and a room with ?debug=binding:

• the side table over the user agent's own VP9 decoder: every output bound to the chunk it came from by decoder and timestamp; two chunks under one timestamp refused (binding_ambiguous);
• a WebAssembly decoder (wai.video.int_motion) that verifies each object against its signed receipts, decodes it and binds every frame; each frame's SHA-256 equals the published one;
• the binding through clone(), close(), new VideoFrame(frame), transfer to a worker and back, copyTo() with and without a format, and re-encode, as §4's table has it;
• init.metadata with contentBinding and with rtpTimestamp: the result quoted in §5;
• in a two-party room, the frame on each tile bound to an object the page verified, under that object's group receipt, and shown as verified.

7. Open

• Whether the group wants the member at all, and where its definition lives (§2).
• No audio counterpart is proposed: AudioData has no metadata dictionary.
• A frame predicted from earlier frames depends on their content too. WAI's text has a page bind only when every object the decoder was given since its last key frame was verified; a registered member's definition would need to say whether it names one object or a chain.