af46947469
SPEC §5 registers `wai.video.vcm` (video coding for machines, ISO/IEC 23888-2) and `wai.tensor.fcm` (feature coding for machines, ISO/IEC 23888-4) as candidates: neither part is published and neither's text is openly readable (the committee's project pages give each the status `ongoing` and list only common test conditions), so no payload format is pinned, neither is emitted or dispatched, a packager refuses a policy that asks for one (`capability-candidate`), and format-probes §19 says why neither has a probe: `no_probe`, whatever the bytes. Licence class `patent-pool`, provisional; replicate class `StandardUnclassified`. Nothing joins the floor or the recommended set. No benchmark was run and no task accuracy is claimed. task-fidelity §5, which named the split-computation case without solving it, now registers the claim class `split-computation` (jwp-receipts §2.4.4 lets the text that defines a class register it; no core text changes). When a network is cut in two, each side signs a claim about the object that carried the tensor across the cut: its side, the fabric it ran on, what it consumed and produced, the digests of both halves' parameter sets, the cut, a network digest over the cut and both sets, and its energy with its acquisition class, or `null` when unmetered and never 0. Two halves compose only as a head and a tail about one tensor, stating one cut, one network and the same two sets; a set the verifier holds must have the digest they name. The pair's account gives the figures by acquisition class (one class's figures are added, two classes' never are, and there is no total), names the unmetered sides, and states its coverage. machine-descriptors, a new extension of the `live` profile: labels, picture regions and embeddings by digest a model produced about one media object, as the payload (`WMD1`) of a signed `meta` event of live-events anchored to that object, signed by the machine that ran the model and stating the model by digest. A zero model digest is refused (`descriptor_no_model`), as are an origin that is not machine or machine-reviewed, an event with no anchor and a model a reader does not admit. No event kind and no transport is added: the event rides an MSF event timeline indexed by the Location of the object it describes. The references are `claim_receipt::split` and the `machine_coding` module of wai-rs (feature `machine_coding`). The corpus is live-conformance/machine-coding/ (2 candidates, 4 network digests, 18 claims, 12 pairs, 26 descriptor events) and format-probe-conformance gains the two `no_probe` vectors. machine_coding_verify.py, the second implementation, agrees on every case, fails every mutated copy (24 of 24) and catches every weakened verifier (32 of 32); claim_verify.py holds the class's body and rules and replays the corpus's claims through steps 1 to 4. Planted faults, each refused with its own reason: an envelope naming a candidate reported and inert at dispatch; halves of two networks, two cuts, two tensors, two heads, and a held parameter set of another digest; a total added to a body; an unmetered half written as 0 J; a descriptor with no model identity, a person's origin, no anchor, an unlisted model, a changed payload under its old signature. decisions D9 records what was read of ISO/IEC 23888 and what follows. Co-Authored-By: Claude <[email protected]>
9.3 KiB
WAI Extension: Machine Descriptors
Status: Draft. A machine descriptor is what a model produced about one media object: labels, regions of the picture, and embeddings named by digest. It is carried as the payload of a signed
metaevent of live-events, anchored to the object it describes, and it states the model that produced it by digest. No new event kind, no new record and no new transport: the event is aWEV1as live-events §3 lays it out, signed by the machine that ran the model, and it rides an MSF event timeline, anEXT-X-DATERANGEtag, anemsgbox, a data channel and a recording as every event does (live-events §8). The reference implementation is themachine_codingmodule ofwai-rs(featuremachine_coding); the corpus islive-conformance/machine-coding/, and its second implementation islive-conformance/machine_coding_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.
1. Scope
In scope: the descriptor's payload (§3), how a reader reads it from an event (§4), what its carriage is (§5), and its refusals (§6).
Out of scope: how a model is run, what its labels mean, and how good they are. A descriptor
is a statement by its signer that a named model gave these outputs for a named object. It is
not a claim about a task's accuracy (task-fidelity §2 is the shape of
one), and a reader that takes a score for a measurement has misread it. A descriptor carries
no embedding vector: an embedding travels as an object of its own and is named here by its
digest. The descriptors that video coding for machines codes inside its own bitstream
(ISO/IEC 23888-2) are that format's; wai.video.vcm is a candidate with no pinned payload
(format-probes §19), and nothing here reads them.
2. Terms
- Machine: the principal that signs the event (live-events §2): the key of the device or service that ran the model.
- Model: named by a model digest, 32 bytes: the SHA-256 a pin of the model's
parameter set carries (SPEC §3.1), as a caption's
modelis (live-events §4.10). A digest of 32 zero bytes names nothing. - Object: the media object the descriptor is about, named by the event's anchor (live-events §3.1): a track, a group and an object.
- Picture extent: a region's coordinates are in ten-thousandths of the picture's width and of its height, from its top left corner, so they hold at any size the object is presented at.
3. The payload
The payload of the meta event, integers little-endian:
| field | size | meaning |
|---|---|---|
magic |
4 | WMD1 (ASCII) |
version |
1 | 1 in this revision |
origin |
1 | 2 machine, 3 machine-reviewed: live-events §4.10's values. 1, human, is not a machine descriptor |
model |
32 | the model digest (§2), not zero |
duration_ms |
4 | u32: how long the descriptor holds from the anchored object; 0, that object alone |
count |
1 | the number of items, 1 to 16 |
| items | count items, each a form byte and that form's fields |
An item's form:
form |
item | fields |
|---|---|---|
| 1 | label: a label for the whole object | score u16; label, a text of 1 to 255 bytes |
| 2 | region: a rectangle of the picture and what it holds | x, y, w, h, each u16, in the picture extent (§2); score u16; label, a text of 0 to 255 bytes |
| 3 | embedding: an embedding of the object, by digest | space, 32 bytes: the digest that names the space the embedding lies in, not zero; dims u32, its number of dimensions, at least 1; digest, 32 bytes: the SHA-256 of the embedding's bytes, not zero |
A text is one length byte and that many bytes of UTF-8 (RFC 3629), with no code point
below U+0020 and no U+007F. A score is in thousandths, 0 to 1000, or 0xFFFF when the
model stated none; it is the model's own output and has no unit this extension defines. A
region has w and h of at least 1, x + w at most 10 000 and y + h at most 10 000.
A descriptor of three items is under 200 bytes; live-events' default max_payload is 1024
(live-events §5). A model that says more about one object signs more events.
4. Reading
A reader is given one event record, the room it reads for, and, where it has one, the list of models it admits. In this order; the first rule that fails names the refusal:
- The record reads as live-events §3.3 reads one, its
roomis the reader's, and its signature verifies (live-events §3.2): otherwisedescriptor_event. - Its kind is
meta(live-events §4.2): otherwisedescriptor_not_meta. - It has an anchor: otherwise
descriptor_unanchored. A descriptor is about one object. - Its payload begins with
WMD1: otherwisedescriptor_no_magic. Ametaevent's payload is opaque to live-events, and one that is not a descriptor is some other application's. versionis 1: otherwisedescriptor_version. A reader reads no further field of a later version.- The fields are read in §3's order. A payload that ends inside a field is
descriptor_truncated. As each is read: anoriginother than 2 or 3 isdescriptor_origin; amodelof 32 zero bytes isdescriptor_no_model; acountof 0 or above 16 isdescriptor_count; aformthis section does not define, a score above 1000 that is not0xFFFF, a region outside the picture extent or of no width or height, or an embedding withdimsof 0 or a zerospaceordigest, isdescriptor_item; a text that is not UTF-8, holds a code point this section excludes, or is empty where a label must not be, isdescriptor_text. - No byte follows the last item: otherwise
descriptor_trailing. - Where the reader was given the models it admits,
modelis one of them: otherwisedescriptor_model_unknown.
A reader that accepts a descriptor has the machine that signed it, the object it is about,
the model it names and its items. It has not checked the model's outputs: nothing in a
descriptor lets it. A sink that shows a descriptor's labels SHOULD say that a machine
produced them, as live-events §4.10 has it label a machine's caption, and takes the label
from origin.
A room's verifier (live-events §6) carries the event as any meta event: it holds the
machine to its policy, its chain and its rates, and reads the payload for its length alone.
The rules of this section are a descriptor reader's, applied to events that verifier
accepted or to one record alone.
5. Carriage
A descriptor event is an event, and live-events §8 carries it unchanged. On MoQ it is a
record of the machine's MSF event timeline (live-events §8.1): the event is anchored, so its
record is {"l":[<group>,<object>],"data":{"wev1":"<base64url>"}}, indexed by the MOQT
Location of the object it describes, and a subscriber that joins mid-stream is not owed
earlier descriptors (live-events §8.2 lists what a join set must hold; a publisher MAY
repeat a descriptor whose duration_ms spans the group's start). The machine publishes on a
track of its own, as every principal does, so a sink that wants no descriptors subscribes to
none.
An embedding's bytes are an object the publisher names elsewhere; a reader that fetches them
checks their SHA-256 against the item's digest and their length against dims under the
element type the space fixes. This extension defines no space.
6. Reasons
| reason | raised at |
|---|---|
descriptor_event |
§4 step 1 |
descriptor_not_meta |
§4 step 2 |
descriptor_unanchored |
§4 step 3 |
descriptor_no_magic |
§4 step 4 |
descriptor_version |
§4 step 5 |
descriptor_truncated |
§4 step 6 |
descriptor_origin |
§4 step 6 |
descriptor_no_model |
§4 step 6 |
descriptor_count |
§4 step 6 |
descriptor_item |
§4 step 6 |
descriptor_text |
§4 step 6 |
descriptor_trailing |
§4 step 7 |
descriptor_model_unknown |
§4 step 8 |
7. Conformance
live-conformance/machine-coding/ holds, in
vectors.json, descriptor events with the outcome each was written to have: the descriptor
read, with the MSF record that carries its event, or its refusal. Every reason of §6 is
given. wai_machine_coding_vectors verify and wai-rs/tests/machine_coding.rs replay it;
machine_coding_verify.py, written from this text and live-events and sharing no code with
wai-rs, reads each record with its own readers, writes each MSF record again, requires
every mutated copy of the corpus to fail, and drops each of its own checks in turn.
8. Security and privacy considerations
A descriptor is as trustworthy as its signer. Its signature shows which machine made the statement and that the statement was not changed; it does not show that the named model was the one run, nor that the model is right. A reader that acts on labels admits models by digest (§4 step 8) and signers by the room's policy (live-events §5).
Labels and regions can describe people. A descriptor says where in a picture something is and what a model called it; a publisher that sends descriptors of people sends what a viewer could not have computed without the model, and the basis on which that may be processed is not this extension's to grant (jwp-receipts §2.4.6 states one). An embedding is carried by digest so that a sink which does not fetch it does not hold it.