open-standards/wai/extensions/machine-descriptors.md

# WAI Extension: Machine Descriptors

> Status: Draft. A **machine descriptor** is what a model produced about one media object:
> labels, regions of the picture, and embeddings named by digest. It is carried as the payload
> of a signed `meta` event of [live-events](live-events.md), anchored to the object it
> describes, and it states the model that produced it by digest. No new event kind, no new
> record and no new transport: the event is a `WEV1` as live-events §3 lays it out, signed by
> the machine that ran the model, and it rides an MSF event timeline, an `EXT-X-DATERANGE`
> tag, an `emsg` box, a data channel and a recording as every event does (live-events §8).
> The reference implementation is the `machine_coding` module of `wai-rs` (feature
> `machine_coding`); the corpus is
> [`live-conformance/machine-coding/`](../live-conformance/machine-coding/), and its second
> implementation is `live-conformance/machine_coding_verify.py`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Scope

In scope: the descriptor's payload (§3), how a reader reads it from an event (§4), what its
carriage is (§5), and its refusals (§6).

Out of scope: how a model is run, what its labels mean, and how good they are. A descriptor
is a statement by its signer that a named model gave these outputs for a named object. It is
not a claim about a task's accuracy ([task-fidelity](task-fidelity.md) §2 is the shape of
one), and a reader that takes a score for a measurement has misread it. A descriptor carries
no embedding vector: an embedding travels as an object of its own and is named here by its
digest. The descriptors that video coding for machines codes inside its own bitstream
(ISO/IEC 23888-2) are that format's; `wai.video.vcm` is a candidate with no pinned payload
([format-probes](format-probes.md) §19), and nothing here reads them.

## 2. Terms

- **Machine**: the principal that signs the event (live-events §2): the key of the device or
  service that ran the model.
- **Model**: named by a **model digest**, 32 bytes: the SHA-256 a pin of the model's
  parameter set carries (SPEC §3.1), as a caption's `model` is (live-events §4.10). A digest
  of 32 zero bytes names nothing.
- **Object**: the media object the descriptor is about, named by the event's anchor
  (live-events §3.1): a track, a group and an object.
- **Picture extent**: a region's coordinates are in ten-thousandths of the picture's width
  and of its height, from its top left corner, so they hold at any size the object is
  presented at.

## 3. The payload

The payload of the `meta` event, integers little-endian:

| field | size | meaning |
|---|---|---|
| `magic` | 4 | `WMD1` (ASCII) |
| `version` | 1 | 1 in this revision |
| `origin` | 1 | 2 machine, 3 machine-reviewed: live-events §4.10's values. 1, human, is not a machine descriptor |
| `model` | 32 | the model digest (§2), not zero |
| `duration_ms` | 4 | u32: how long the descriptor holds from the anchored object; 0, that object alone |
| `count` | 1 | the number of items, 1 to 16 |
| items | | `count` items, each a `form` byte and that form's fields |

An item's `form`:

| `form` | item | fields |
|---|---|---|
| 1 | **label**: a label for the whole object | `score` u16; `label`, a text of 1 to 255 bytes |
| 2 | **region**: a rectangle of the picture and what it holds | `x`, `y`, `w`, `h`, each u16, in the picture extent (§2); `score` u16; `label`, a text of 0 to 255 bytes |
| 3 | **embedding**: an embedding of the object, by digest | `space`, 32 bytes: the digest that names the space the embedding lies in, not zero; `dims` u32, its number of dimensions, at least 1; `digest`, 32 bytes: the SHA-256 of the embedding's bytes, not zero |

A **text** is one length byte and that many bytes of UTF-8 (RFC 3629), with no code point
below U+0020 and no U+007F. A **score** is in thousandths, 0 to 1000, or `0xFFFF` when the
model stated none; it is the model's own output and has no unit this extension defines. A
region has `w` and `h` of at least 1, `x + w` at most 10 000 and `y + h` at most 10 000.

A descriptor of three items is under 200 bytes; live-events' default `max_payload` is 1024
(live-events §5). A model that says more about one object signs more events.

## 4. Reading

A reader is given one event record, the room it reads for, and, where it has one, the list of
models it admits. In this order; the first rule that fails names the refusal:

1. The record reads as live-events §3.3 reads one, its `room` is the reader's, and its
   signature verifies (live-events §3.2): otherwise `descriptor_event`.
2. Its kind is `meta` (live-events §4.2): otherwise `descriptor_not_meta`.
3. It has an anchor: otherwise `descriptor_unanchored`. A descriptor is about one object.
4. Its payload begins with `WMD1`: otherwise `descriptor_no_magic`. A `meta` event's payload
   is opaque to live-events, and one that is not a descriptor is some other application's.
5. `version` is 1: otherwise `descriptor_version`. A reader reads no further field of a later
   version.
6. The fields are read in §3's order. A payload that ends inside a field is
   `descriptor_truncated`. As each is read: an `origin` other than 2 or 3 is
   `descriptor_origin`; a `model` of 32 zero bytes is `descriptor_no_model`; a `count` of 0 or
   above 16 is `descriptor_count`; a `form` this section does not define, a score above 1000
   that is not `0xFFFF`, a region outside the picture extent or of no width or height, or an
   embedding with `dims` of 0 or a zero `space` or `digest`, is `descriptor_item`; a text that
   is not UTF-8, holds a code point this section excludes, or is empty where a label must not
   be, is `descriptor_text`.
7. No byte follows the last item: otherwise `descriptor_trailing`.
8. Where the reader was given the models it admits, `model` is one of them: otherwise
   `descriptor_model_unknown`.

A reader that accepts a descriptor has the machine that signed it, the object it is about,
the model it names and its items. It has not checked the model's outputs: nothing in a
descriptor lets it. A sink that shows a descriptor's labels **SHOULD** say that a machine
produced them, as live-events §4.10 has it label a machine's caption, and takes the label
from `origin`.

A room's verifier (live-events §6) carries the event as any `meta` event: it holds the
machine to its policy, its chain and its rates, and reads the payload for its length alone.
The rules of this section are a descriptor reader's, applied to events that verifier
accepted or to one record alone.

## 5. Carriage

A descriptor event is an event, and live-events §8 carries it unchanged. On MoQ it is a
record of the machine's MSF event timeline (live-events §8.1): the event is anchored, so its
record is `{"l":[<group>,<object>],"data":{"wev1":"<base64url>"}}`, indexed by the MOQT
Location of the object it describes, and a subscriber that joins mid-stream is not owed
earlier descriptors (live-events §8.2 lists what a join set must hold; a publisher **MAY**
repeat a descriptor whose `duration_ms` spans the group's start). The machine publishes on a
track of its own, as every principal does, so a sink that wants no descriptors subscribes to
none.

An embedding's bytes are an object the publisher names elsewhere; a reader that fetches them
checks their SHA-256 against the item's `digest` and their length against `dims` under the
element type the `space` fixes. This extension defines no space.

## 6. Reasons

| reason | raised at |
|---|---|
| `descriptor_event` | §4 step 1 |
| `descriptor_not_meta` | §4 step 2 |
| `descriptor_unanchored` | §4 step 3 |
| `descriptor_no_magic` | §4 step 4 |
| `descriptor_version` | §4 step 5 |
| `descriptor_truncated` | §4 step 6 |
| `descriptor_origin` | §4 step 6 |
| `descriptor_no_model` | §4 step 6 |
| `descriptor_count` | §4 step 6 |
| `descriptor_item` | §4 step 6 |
| `descriptor_text` | §4 step 6 |
| `descriptor_trailing` | §4 step 7 |
| `descriptor_model_unknown` | §4 step 8 |

## 7. Conformance

[`live-conformance/machine-coding/`](../live-conformance/machine-coding/) holds, in
`vectors.json`, descriptor events with the outcome each was written to have: the descriptor
read, with the MSF record that carries its event, or its refusal. Every reason of §6 is
given. `wai_machine_coding_vectors verify` and `wai-rs/tests/machine_coding.rs` replay it;
`machine_coding_verify.py`, written from this text and live-events and sharing no code with
`wai-rs`, reads each record with its own readers, writes each MSF record again, requires
every mutated copy of the corpus to fail, and drops each of its own checks in turn.

## 8. Security and privacy considerations

A descriptor is as trustworthy as its signer. Its signature shows which machine made the
statement and that the statement was not changed; it does not show that the named model was
the one run, nor that the model is right. A reader that acts on labels admits models by
digest (§4 step 8) and signers by the room's policy (live-events §5).

Labels and regions can describe people. A descriptor says where in a picture something is
and what a model called it; a publisher that sends descriptors of people sends what a viewer
could not have computed without the model, and the basis on which that may be processed is
not this extension's to grant ([jwp-receipts](jwp-receipts.md) §2.4.6 states one). An
embedding is carried by digest so that a sink which does not fetch it does not hold it.