From 00dcf1f1aa561e78b169185e53281bc6225ffb41 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Mon, 7 Sep 2026 04:00:07 -0400 Subject: [PATCH] task-fidelity: name the contract for media a model consumes, not a hash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `determinism-tiers.md` answers one question — does the sink reproduce the same bytes? A growing class of coded media does not answer that question at all, and WAI can register these capabilities today without being able to say what conformance means for them. `ReplicateClass::Behavioral` is a disclaimer, not a criterion; "task-equivalent but not bit-equal" is unstateable, so an implementation either overclaims a reproducibility tier it does not meet or says nothing. Three independent arrivals, none of them ours: * ISO/IEC 23888-2 (VCM, DIS, FDIS planned late 2026) codes video AND descriptors, optimised for machine-task performance after decoding. * ISO/IEC 23888-4 (FCM, CD) codes intermediate feature tensors from inside a network — no pixels are reconstructed at any point, so there is no medium to hash. * A vendor markets compression as safe for machine consumption when the behavioural change is statistically indistinguishable from the variance the model already tolerates — with, by its own account, no shared framework for the claim. Deliberately a SIBLING file, not a third tier. The two tiers in `determinism-tiers.md` are reproducibility guarantees a sink can check from the bytes it received. A task claim cannot be checked that way — it needs a model, a corpus and a test run. Listing them together would blur the one property that file keeps clean. The claim is a five-tuple and is meaningless with any element missing: model, test, tolerance τ, observed δ, evidence class. `model` and `test` MUST be content addresses — "tested on a standard detector" is unfalsifiable, a hash is not — and a tolerance without an observation is a target, not a claim. Evidence is graded the way `energy-measurement.md` grades acquisition, on the same principle that what separates the rungs is provenance and scope, not accuracy: Measured / Transferred / Asserted, no promotion in transit, and a verifier that cannot resolve model and test MUST treat the claim as Asserted whatever it says. §4 states the limitation rather than hiding it: this is a graded assertion, not a verification, and a relying party that reads it as the latter has misread it. §5 names FCM's split budget without pretending to solve it. The cut point N makes the origin/sink work partition a wire parameter, which breaks `joules_micro` as one u64 for one producer. A cut-point-aware receipt needs joules either side plus a network-identity hash making the halves provably compose — and that third piece is a carriage problem WAI has not solved. Until then such a claim MUST state the cut point and MUST NOT report a single joule figure as if one producer did all the work. No new wire format, no new receipt mechanism. Second item in the landscape survey (#80) ingest queue, and the second of the two with no external dependency. Co-Authored-By: Claude Opus 5 --- wai/extensions/determinism-tiers.md | 6 ++ wai/extensions/task-fidelity.md | 128 ++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+) create mode 100644 wai/extensions/task-fidelity.md diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index d84516e7f..ee65cf1a6 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -88,6 +88,12 @@ construction; the reverse does not hold. ## 4. Framing +Both tiers here are **reproducibility** guarantees: a sink can check them from the +bytes it received. A separate axis exists for media whose correctness is a +downstream model's behaviour rather than a byte comparison — see +[task-fidelity](task-fidelity.md), which is a sibling rather than a third tier +precisely because such a claim is *not* sink-checkable from the bytes. + WAI spans the spectrum rather than competing on one point of it. The `entropy-consistency` tier is the base learned-codec floor — the guarantee the field and the underlying standards already make — and WAI supports it directly. diff --git a/wai/extensions/task-fidelity.md b/wai/extensions/task-fidelity.md new file mode 100644 index 000000000..84789e10c --- /dev/null +++ b/wai/extensions/task-fidelity.md @@ -0,0 +1,128 @@ +# WAI Extension: Task Fidelity + +> Status: Draft. Names the contract for media whose correctness is a **downstream +> model's behaviour**, not a byte comparison — the class +> [`determinism-tiers`](determinism-tiers.md) cannot express. Sibling to that +> file, not a tier inside it: the tiers there are *reproducibility* guarantees a +> sink can check from the bytes alone, and a task claim is not one. +> No new wire format and no new receipt mechanism; a claim shape + an evidence +> grading over §7, modelled on +> [`energy-measurement`](energy-measurement.md)'s acquisition class. +> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174. + +## 1. Why a second axis + +`determinism-tiers.md` answers one question: *does the sink reproduce the same +bytes?* A growing class of coded media is not answering that question at all. +Three independent arrivals, none of them WAI's: + +- **Video coding for machines.** ISO/IEC 23888-2 (MPEG-AI Part 2; **DIS**, FDIS + planned late 2026) codes video *and* descriptors extracted from it, jointly + optimised for bitrate and for machine-task performance after decoding. Its + fidelity target is a task's accuracy, not a pixel comparison. +- **Feature coding for machines.** ISO/IEC 23888-4 (MPEG-AI Part 4; **CD**) codes + intermediate feature tensors from inside a machine-vision network — the origin + runs the first N layers, ships the tensor, the sink runs the remainder. **No + pixels are reconstructed at any point**, so there is no medium to hash. +- **Vendor framing.** At least one vendor markets compression as safe for machine + consumption when the change it induces in a downstream model's behaviour is + statistically indistinguishable from the variance that model already tolerates. + No standards body, no designation — and the vendor's own material states the + industry lacks a shared framework for the claim, which is what makes the seam + worth naming rather than ceding. + +WAI today can register these as capabilities but cannot state what conformance +means for them. `ReplicateClass::Behavioral` is a disclaimer, not a criterion. +*Task-equivalent but not bit-equal* is unstateable, so an implementation either +overclaims a reproducibility tier it does not meet or says nothing at all. + +**The distinction that makes this a separate file.** A reproducibility tier is +checkable by the sink from the bytes it received. A task-fidelity claim is not: +checking it requires a model, a corpus, and a test run. Listing the two together +would blur the one property `determinism-tiers.md` currently keeps clean — that +its tiers are verifiable without leaving the object. + +## 2. The claim + +A task-fidelity claim is a tuple, and it is meaningless with any element missing: + +| element | meaning | +|---|---| +| `model` | content address of the network the claim is about — the identity under which δ was observed | +| `test` | content address of the task, corpus and metric the claim was measured under | +| `tolerance` | τ — the behavioural change the claimant asserts is acceptable, in the metric `test` names | +| `observed` | δ — the behavioural change actually measured under `test` | +| `evidence` | the class in §3 | + +A claim asserts **δ ≤ τ under `test`, for `model`**. It asserts nothing about any +other model, any other corpus, or any metric `test` does not name. + +- A claim **MUST** carry all five elements. A tolerance without an observation is + a target, not a claim, and **MUST NOT** be sealed as one. +- `model` and `test` **MUST** be content addresses, not names. "Tested on a + standard detector" is unfalsifiable; a hash is not. +- A claim **MUST NOT** be restated for a different model or corpus without a new + observation. Behavioural tolerance does not transfer by assertion. + +## 3. Evidence class (REQUIRED) + +What separates the rungs is **provenance and scope, not accuracy** — the same +principle as `energy-measurement.md` §2, and for the same reason: a confidently +formatted δ says nothing about whether anyone ran the test. + +| class | meaning | requires | +|---|---|---| +| `Measured` | δ was observed by running `test` against `model` on the stated corpus. | retained artifacts sufficient to re-run | +| `Transferred` | δ was observed under a *different* model or corpus and is argued to carry over. | the source claim, and the argument, both named | +| `Asserted` | δ is claimed without an observation. | — | + +- An implementation **MUST NOT** promote a claim's evidence class in transit, as + §2 of `energy-measurement.md` forbids for acquisition class. +- A verifier that cannot resolve `model` and `test` **MUST** treat the claim as + `Asserted` regardless of the class stated, and **SHOULD** say so rather than + discard the claim silently. +- `Transferred` **MUST NOT** be used to cross a modality or a task family. A + detector's tolerance is not a segmenter's. + +## 4. What the claim is, and is not + +- It is **not** a reproducibility tier, and **MUST NOT** be presented as one. A + capability carrying a task-fidelity claim **MUST NOT** thereby claim + `entropy-consistency` or `decode-equivalence`; those are separate assertions + about separate properties, and a capability may hold both, one, or neither. +- It is **not** sink-checkable from the bytes. This is the defining limitation, + and stating it is the point: a receipt bearing a task-fidelity claim carries a + *graded assertion*, not a verification, and a relying party that treats it as + the latter has misread it. +- It is **not** a quality metric. δ is behavioural change in a named downstream + task, not perceptual fidelity, and the two **MUST NOT** be reported in the same + field. +- It does **not** license degradation elsewhere. A task-fidelity claim says + nothing about energy, and an object honouring a decoding-operation-reduction + hint is still bound by `determinism-tiers.md` §2. + +## 5. The split-computation case — named, not solved + +FCM makes the origin/sink **work partition** a negotiated wire parameter: the cut +point N decides how much of the network each side runs. That breaks an assumption +in `jwp-receipts.md`, where `joules_micro` is one `u64` for producing the object +by one producer. + +A cut-point-aware receipt needs three things WAI does not have: joules before the +cut, joules attributable after it, and a network-identity hash making the halves +provably compose. The third is a carriage problem — the artifact a network +identity names has no normative wire form in WAI today. + +This extension **does not** solve that. It records the gap so a task-fidelity +claim over a split computation is not mistaken for a complete accounting: such a +claim **MUST** state the cut point, and **MUST NOT** report a single joule figure +as if one producer had done all the work. + +## 6. Framing + +Byte-comparability is becoming one fidelity contract among several rather than +the only one. WAI's position is unchanged by that — it names contracts and grades +the evidence behind them, and does not adjudicate which contract a domain should +want. What would be wrong is silence: a ratified class of media arriving with a +conformance notion the container cannot state, described instead in the vocabulary +of a guarantee it does not make. -- 2.54.0 (Apple Git-157)