diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index d84516e7f..ee65cf1a6 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -88,6 +88,12 @@ construction; the reverse does not hold. ## 4. Framing +Both tiers here are **reproducibility** guarantees: a sink can check them from the +bytes it received. A separate axis exists for media whose correctness is a +downstream model's behaviour rather than a byte comparison — see +[task-fidelity](task-fidelity.md), which is a sibling rather than a third tier +precisely because such a claim is *not* sink-checkable from the bytes. + WAI spans the spectrum rather than competing on one point of it. The `entropy-consistency` tier is the base learned-codec floor — the guarantee the field and the underlying standards already make — and WAI supports it directly. diff --git a/wai/extensions/task-fidelity.md b/wai/extensions/task-fidelity.md new file mode 100644 index 000000000..84789e10c --- /dev/null +++ b/wai/extensions/task-fidelity.md @@ -0,0 +1,128 @@ +# WAI Extension: Task Fidelity + +> Status: Draft. Names the contract for media whose correctness is a **downstream +> model's behaviour**, not a byte comparison — the class +> [`determinism-tiers`](determinism-tiers.md) cannot express. Sibling to that +> file, not a tier inside it: the tiers there are *reproducibility* guarantees a +> sink can check from the bytes alone, and a task claim is not one. +> No new wire format and no new receipt mechanism; a claim shape + an evidence +> grading over §7, modelled on +> [`energy-measurement`](energy-measurement.md)'s acquisition class. +> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174. + +## 1. Why a second axis + +`determinism-tiers.md` answers one question: *does the sink reproduce the same +bytes?* A growing class of coded media is not answering that question at all. +Three independent arrivals, none of them WAI's: + +- **Video coding for machines.** ISO/IEC 23888-2 (MPEG-AI Part 2; **DIS**, FDIS + planned late 2026) codes video *and* descriptors extracted from it, jointly + optimised for bitrate and for machine-task performance after decoding. Its + fidelity target is a task's accuracy, not a pixel comparison. +- **Feature coding for machines.** ISO/IEC 23888-4 (MPEG-AI Part 4; **CD**) codes + intermediate feature tensors from inside a machine-vision network — the origin + runs the first N layers, ships the tensor, the sink runs the remainder. **No + pixels are reconstructed at any point**, so there is no medium to hash. +- **Vendor framing.** At least one vendor markets compression as safe for machine + consumption when the change it induces in a downstream model's behaviour is + statistically indistinguishable from the variance that model already tolerates. + No standards body, no designation — and the vendor's own material states the + industry lacks a shared framework for the claim, which is what makes the seam + worth naming rather than ceding. + +WAI today can register these as capabilities but cannot state what conformance +means for them. `ReplicateClass::Behavioral` is a disclaimer, not a criterion. +*Task-equivalent but not bit-equal* is unstateable, so an implementation either +overclaims a reproducibility tier it does not meet or says nothing at all. + +**The distinction that makes this a separate file.** A reproducibility tier is +checkable by the sink from the bytes it received. A task-fidelity claim is not: +checking it requires a model, a corpus, and a test run. Listing the two together +would blur the one property `determinism-tiers.md` currently keeps clean — that +its tiers are verifiable without leaving the object. + +## 2. The claim + +A task-fidelity claim is a tuple, and it is meaningless with any element missing: + +| element | meaning | +|---|---| +| `model` | content address of the network the claim is about — the identity under which δ was observed | +| `test` | content address of the task, corpus and metric the claim was measured under | +| `tolerance` | τ — the behavioural change the claimant asserts is acceptable, in the metric `test` names | +| `observed` | δ — the behavioural change actually measured under `test` | +| `evidence` | the class in §3 | + +A claim asserts **δ ≤ τ under `test`, for `model`**. It asserts nothing about any +other model, any other corpus, or any metric `test` does not name. + +- A claim **MUST** carry all five elements. A tolerance without an observation is + a target, not a claim, and **MUST NOT** be sealed as one. +- `model` and `test` **MUST** be content addresses, not names. "Tested on a + standard detector" is unfalsifiable; a hash is not. +- A claim **MUST NOT** be restated for a different model or corpus without a new + observation. Behavioural tolerance does not transfer by assertion. + +## 3. Evidence class (REQUIRED) + +What separates the rungs is **provenance and scope, not accuracy** — the same +principle as `energy-measurement.md` §2, and for the same reason: a confidently +formatted δ says nothing about whether anyone ran the test. + +| class | meaning | requires | +|---|---|---| +| `Measured` | δ was observed by running `test` against `model` on the stated corpus. | retained artifacts sufficient to re-run | +| `Transferred` | δ was observed under a *different* model or corpus and is argued to carry over. | the source claim, and the argument, both named | +| `Asserted` | δ is claimed without an observation. | — | + +- An implementation **MUST NOT** promote a claim's evidence class in transit, as + §2 of `energy-measurement.md` forbids for acquisition class. +- A verifier that cannot resolve `model` and `test` **MUST** treat the claim as + `Asserted` regardless of the class stated, and **SHOULD** say so rather than + discard the claim silently. +- `Transferred` **MUST NOT** be used to cross a modality or a task family. A + detector's tolerance is not a segmenter's. + +## 4. What the claim is, and is not + +- It is **not** a reproducibility tier, and **MUST NOT** be presented as one. A + capability carrying a task-fidelity claim **MUST NOT** thereby claim + `entropy-consistency` or `decode-equivalence`; those are separate assertions + about separate properties, and a capability may hold both, one, or neither. +- It is **not** sink-checkable from the bytes. This is the defining limitation, + and stating it is the point: a receipt bearing a task-fidelity claim carries a + *graded assertion*, not a verification, and a relying party that treats it as + the latter has misread it. +- It is **not** a quality metric. δ is behavioural change in a named downstream + task, not perceptual fidelity, and the two **MUST NOT** be reported in the same + field. +- It does **not** license degradation elsewhere. A task-fidelity claim says + nothing about energy, and an object honouring a decoding-operation-reduction + hint is still bound by `determinism-tiers.md` §2. + +## 5. The split-computation case — named, not solved + +FCM makes the origin/sink **work partition** a negotiated wire parameter: the cut +point N decides how much of the network each side runs. That breaks an assumption +in `jwp-receipts.md`, where `joules_micro` is one `u64` for producing the object +by one producer. + +A cut-point-aware receipt needs three things WAI does not have: joules before the +cut, joules attributable after it, and a network-identity hash making the halves +provably compose. The third is a carriage problem — the artifact a network +identity names has no normative wire form in WAI today. + +This extension **does not** solve that. It records the gap so a task-fidelity +claim over a split computation is not mistaken for a complete accounting: such a +claim **MUST** state the cut point, and **MUST NOT** report a single joule figure +as if one producer had done all the work. + +## 6. Framing + +Byte-comparability is becoming one fidelity contract among several rather than +the only one. WAI's position is unchanged by that — it names contracts and grades +the evidence behind them, and does not adjudicate which contract a domain should +want. What would be wrong is silence: a ratified class of media arriving with a +conformance notion the container cannot state, described instead in the vocabulary +of a guarantee it does not make.