open-standards/wai/extensions/derived-tracks.md

# WAI Extension: Derived Tracks

> Status: Draft. Registers the derivation operations of ISO/IEC 23001-16:2021
> as `wai.derive.*` capability strings, each with a **per-operation determinism
> tier**, and states how a tier composes over a derived sample. No wire format,
> and no compositing. The geometric conditions listed per operation are read
> from the operations' titles: clause 8 of ISO/IEC 23001-16, which defines each
> operation, was not read. Reference impl: `wai-rs` `codecs::derivation`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.

## 1. Prior art, stated plainly

A track that carries a process the receiver evaluates, rather than the processed
result, is not new. **ISO/IEC 23001-16:2021**, *Derived visual tracks in the ISO
base media file format* (1st edition, 2021-11), ratified it. A derived visual
track (sample entry `dtrk`) carries derived samples, each active for a duration
of the track's composition timeline. Each derived sample is an ordered list of
derivation operations, each applying a derivation transformation, with its
parameters, to an ordered list of inputs. An input is either a **visual input**
(an image item, an interval of an input track or track group possibly spanning
several samples, the visual output of a preceding operation, or a default fill
picture) or a **parameter input** (a metadata item, or an interval of a metadata
track possibly spanning several samples). A parameter takes the value the derived
sample gives it, else the value in the sample entry, else the default its
transformation's clause defines. A visual output is one frame or a sequence of
frames, and any transformation may have internal time structure (the standard's
example is a cross-fade), so the picture may change during the sample's duration.
Operations chain, and a derived track can be an input to another. The first
edition defines a base set of ten transformations, each identified by a
four-character code, with `'uuid'` reserved for vendor transformations identified
by UUID.

The parts of that specification read for this extension (clauses 1–4, 5.1 and
5.2, and the table of contents, from the published sample, which ends at 5.2) do
not say whether an operation's output reproduces byte for byte across conforming
readers. The rest was not read: clause 5.3 (the semantics of the derivation
boxes), clauses 6 and 7 (the sample entry, the configuration record and the
sample format), clause 8, which gives each operation's definition, syntax and
semantics, and so any blend or resampling formula, and Annex A. This extension
states, per operation, when a
claim over its output may hold, in the vocabulary of
[determinism-tiers](determinism-tiers.md), so a receipt or binding over a derived
sample can say which tier it actually holds. Every tier rests on conditions that
the sink which executed an instance checks on that instance, in space and in
time. Clause 8 decides what an instance does, and so which conditions it can
meet. What the registry takes from outside clause 8 is each operation's purpose,
read from its title, which decides the geometric conditions listed for it (§2,
*Basis of the tiers*). The tiers are not uniform. Dissolve and Scaling compute
new sample values by their nature. The others are exact only for instances that
copy samples without converting them, and whose output does not change over the
span a claim covers.

**Scope.** WAI does not composite. Presentation stays with the sink, as in
[linear-film](linear-film.md) and [interactive-worlds](interactive-worlds.md).
Registering grid and overlay composition states what a reproducibility claim over
their output would need; it does not make WAI a renderer. Registering the
operations aligns WAI's vocabulary with the standard; it says nothing about which
players evaluate derived visual tracks.

## 2. Registered operations

An executed instance of an operation is **exact** when every output sample is a
copy of an input sample, or of a constant the claim names, placed by integer
geometry, with no arithmetic on sample values, no change of sample
representation, and no change over the span the claim covers. Inputs at
`decode-equivalence` give an exact instance's output at `decode-equivalence`. A
per-operation tier is one of:

- **exact when (conditions)**: an executed instance is exact when it meets the
  three universal conditions below and every condition the table lists for its
  operation, and untiered otherwise. The conditions are properties of the
  executed instance, checked by the sink that executed it. **No operation is
  exact unconditionally**, Identity included.
- **untiered**: the operation computes new sample values, by blending or
  resampling, and WAI pins no arithmetic for it. The output carries no tier,
  whatever its inputs held, until an arithmetic is pinned and an instance is
  shown to follow it. Dependence on time is not what makes an operation
  untiered: every operation needs time-invariant.

The three **universal conditions**, required of every tiered operation:

- **same-representation**: every visual input, and every constant the instance
  writes (a fill colour, the default fill picture), is already in the output's
  sample representation: the same bit depth, chroma format and chroma siting,
  colour primaries, transfer characteristics, matrix coefficients, range and
  component set. No sample value is converted. Clause 4 requires the visual
  inputs of a derived sample to share pixel aspect ratio and bit depth with each
  other; the clauses read say nothing about the other properties, or about the
  output's representation relative to its inputs. So an 8-bit input composed into
  a 10-bit output, a BT.709 input composed into a BT.2020 output, or an Identity
  whose output representation differs from its input's holds no tier.
- **copies-only**: every output sample value is, bit for bit, the value of an
  input sample or of a constant whose value the claim names. The instance
  computed no sample value.
- **time-invariant**: over the span the claim covers, every visual input the
  instance reads is one frame and every parameter it uses has one value. The
  instance also has no time structure of its own, such as a transition or a
  parameter interpolated over time. So its output over the span is one frame,
  fixed by those frames and values. The span is either the derived sample's
  whole duration or one output frame at a composition instant the claim names
  (§3 rule 3). A parameter counts whatever its source: the derived sample, the
  sample entry, the default of the operation's clause, or a parameter input.
  Clause 4 lets one derivation duration span several samples of an input track,
  and lets any transformation have internal time structure. A visual or
  parameter input may be an interval of a track spanning several samples, and a
  visual output may be a sequence of frames. So two instances hold no tier for a
  claim over the whole duration: an ROI selection whose region is driven by an
  interval of a metadata track whose value changes within the sample, and an
  Identity over an input interval holding several samples. Either can hold one
  for a claim over one output frame, at an instant at which nothing it reads
  changes, if it has no time structure of its own.

Because every tiered operation requires all three, an instance is untiered, not
over-claimed, if it uses any parameter that clause 8 allows to convert or compute
sample values, or to vary its output over time. This holds as long as the
executing sink lists only the conditions it actually checked.

| capability | 23001-16 clause | operation | tier | conditions beyond the universal three (read from the operation's title; clause 8 was not read), or why untiered |
|---|---|---|---|---|
| `wai.derive.identity` | 8.2 | Identity | exact when | none. It copies its input; uncovered output samples come from the default fill picture, a constant the claim must name (§3 rule 5). Whether clause 8 lets Identity emit a representation other than its input's was not read, so same-representation is checked, not assumed |
| `wai.derive.srgb_fill` | 8.3 | sRGB Fill | exact when | none. The colour is stated in sRGB; writing it into any other representation is a conversion, which same-representation excludes. As its title is read, it reads no visual input, and it is the only operation registered as reading none |
| `wai.derive.dissolve` | 8.4 | Dissolve | untiered | a transition that mixes two inputs, so its output values are computed, not copied. A transition is also time structure of its own, which time-invariant excludes for every operation (clause 4's example of internal time structure is a cross-fade). Untiered until a blend arithmetic, and how its weights depend on the instant, are pinned; clause 8.4, which may fix them, was not read |
| `wai.derive.crop` | 8.5 | Crop | exact when | the kept region lies on the sample grid of every component (no component is resampled) |
| `wai.derive.rotation` | 8.6 | Rotation | exact when | the angle is a multiple of 90°, and every component is sampled at the same resolution, so the rotation permutes samples |
| `wai.derive.mirror` | 8.7 | Mirror | exact when | the reflection maps every component's sample grid, including its siting, onto itself (always so without chroma subsampling) |
| `wai.derive.scaling` | 8.8 | Scaling | untiered | resamples, so its output values are computed, not copied. Untiered until an interpolation filter and its rounding are pinned; clause 8.8, which may fix them, was not read |
| `wai.derive.roi_selection` | 8.9 | Region of interest (ROI) selection | exact when | the region lies on the sample grid of every component, and is emitted at its own size |
| `wai.derive.grid_composition` | 8.10 | Grid composition | exact when | every input is placed at its own size, at offsets on every component's sample grid |
| `wai.derive.overlay_composition` | 8.11 | Overlay composition | exact when | every input is placed at its own size, at offsets on every component's sample grid, and opaquely (no alpha, opacity or blending) |

- **Identification.** Each capability string stands for the transformation its
  clause defines, including that clause's four-character code. The registry does
  not yet record the codes: clause 8 of the published text was not available when
  this was written. A sink maps each capability to the code its clause defines,
  and MUST NOT map one by a code taken from any other source. Codes from earlier
  drafts circulate in secondary sources.
- **Basis of the tiers.** Tiers are assigned from what each operation does to
  sample values, in space and in time, not from clause 8, which was not read.
  Each condition is a property of an executed instance, not of parameter syntax.
  The three universal conditions exclude any conversion or computation of sample
  values, and any change over the span a claim covers. What is taken from outside
  clause 8 is each operation's purpose, read from its title (a crop selects, a
  rotation rotates), which decides the geometric conditions listed for it, and
  whether it reads a visual input at all (only sRGB Fill is registered as reading
  none). Reading clause 8 may show one of those lists to be incomplete. It may
  also make some conditions unnecessary (for example, if Rotation admits only
  quarter turns, or if the output representation is always the inputs'). Clause 8
  may also fix a blend or resampling arithmetic bit-exactly; that would be the
  start of a pin for Dissolve or Scaling, not a tier in itself. Clause 4, which
  was read, already shows where a list may fall short: wherever an input's size
  differs from the output's, the input is cropped or filled implicitly, for every
  operation unless the operation's clause overrides that. The fill is covered by
  §3 rule 5. The clauses read do not say where such an input is placed, and no
  list, Identity's included, carries a condition on that placement.
- **Correction.** An earlier revision said that no tier here rested on anything
  clause 8 might fix. The time dimension did. Only Dissolve was untiered for
  depending on time, as if clause 4's cross-fade were particular to it, but
  clause 4 states generally that a derivation transformation may have internal
  time structure. Time-invariant is now a universal condition.
- **Not payload capabilities.** An operation names no payload format; it acts on
  samples another capability decoded. SPEC §4 assigns every payload exactly one
  capability, so these strings are not in `codecs::known_capabilities` and a
  manifest cannot name one as its payload's capability. A sink advertises the
  operations it evaluates the way it advertises codecs.
- **Licence.** The Introduction of ISO/IEC 23001-16:2021 records a patent claimed
  to be required for compliance, whose holder has given a
  reasonable-and-non-discriminatory licensing assurance. The registry classifies
  the operations as patent-pool, conservatively
  (`codecs::derivation::derivation_license`).

## 3. Composing a tier over a derived sample

1. **Weakest step.** A derived sample holds `decode-equivalence` only if every
   visual input holds `decode-equivalence` and every executed operation instance
   is exact: it meets the three universal conditions and every condition listed
   for its operation. Otherwise it holds no tier, and a receipt over it MUST NOT
   carry a reproducible derived-sample hash.
2. **`entropy-consistency` does not pass through.** Derivation operations act on
   reconstructed samples, which that tier does not fix. A derived sample over an
   input at `entropy-consistency` holds no tier.
3. **The claim is about what was executed, with what, and when.** ISO/IEC
   23001-16 lets a reader treat an unsupported non-essential operation as a null
   operation, so two conforming readers can emit different samples from one
   derived sample. A parameter can come from a metadata track, and the output
   can change within the sample's duration. A tier claim or receipt over a
   derived sample MUST name:
   - the operations executed, in order;
   - for each instance, the parameter values it applied, whatever their source
     (the derived sample, the sample entry, the default of the operation's
     clause, or a parameter input), with any region, offset or size expressed in
     samples;
   - the span it covers. For a claim over the whole duration, that is the derived
     sample's composition time and duration on the derived track's timeline, in
     its timescale. Otherwise it is the composition instant of each output frame
     the claim covers;
   - its inputs, visual and parameter, by content hash. For an interval of a
     track, the hash covers the samples the instance read.

   Two sinks' derived-sample hashes are comparable only when all four are equal.
   A sink MUST NOT present a hash computed with an operation skipped as the hash
   of the sample with that operation, and MUST NOT present the hash of one output
   frame as the hash of a derived sample whose output changes over its duration.
4. **Transforms applied before derivation count.** ISO/IEC 23001-16 applies the
   transformative properties of inputs (clean aperture, track matrix, and the
   like) before the derivation operation. For a tier claim each counts as a step,
   tiered as the registered operation it performs. A unity matrix is identity; a
   quarter-turn or reflection matrix is rotation or mirror; a clean aperture is a
   crop. Any other matrix is untiered, whether or not it resamples (a scale, a
   shear or a sub-sample translation does): the registry pins no arithmetic for
   it.
5. **The default fill picture is an input.** Where output samples come from the
   configuration record's default fill (black, grey or white), whatever the
   operation, the claim MUST name the fill's sample values in the output
   representation. A fill whose values the claim does not name is an input with
   no tier, and an instance that writes it does not meet copies-only. An input
   that neither the derived sample nor the sample entry lists is the default fill
   picture (clause 5.1). So an operation reads the fill for every visual input it
   reads that neither lists, and a claim over it names the fill as an input.
   Where an instance sits in the executed list does not say what it read: clause
   4 lets an operation that is not first read the output of any previous
   operation, only new inputs, or both. A claim therefore names, for each
   instance, the visual inputs it read: an input, or the output of an operation
   executed before it. An instance of an operation that reads a visual input and
   names none holds no tier, wherever it sits in the list.
6. **Unregistered operations cannot be tiered.** A `'uuid'` vendor operation, or
   any transformation this registry does not name, makes the derived sample
   untiered. The reference implementation returns an error rather than guess.
7. **Relation to [determinism-tiers](determinism-tiers.md) §2.** That section's
   rule is that a determinism claim covers what the sink emits. A derived sample
   is an emitted medium, and this section says when a claim over it holds. It
   does not relax the rule for the inputs: a hash over a decoded input is not a
   hash over the derived sample.

## 4. Receipt

A receipt over a derived sample SHOULD carry:

- the executed operation list, as capability strings in order, with the
  conditions each instance met, the universal ones included, the visual inputs
  each instance read (§3 rule 5), and the parameter values each instance applied
  (§3 rule 3);
- the span the claim covers: the derived sample's composition time and
  duration, or the composition instant of each output frame covered;
- the content hashes of the inputs, visual and parameter;
- the resulting tier;
- only when that tier is `decode-equivalence`, the hash of each output frame the
  claim covers. A claim over the whole duration covers one frame.

The energy of evaluating the graph is measured and attributed as for any other
work ([energy-measurement](energy-measurement.md)). The tier and the energy are
separate claims, and neither implies the other. This extension defines the
content of such a receipt, not a field layout.

## 5. Reference implementation

`wai-rs`, module `codecs::derivation`, with no feature gate:

- `derivation_operations()`, the ten registered operations in clause order
  (`DerivationOp { capability, name, clause, fourcc, tier, basis,
  reads_visual_input }`, with `fourcc` `None` throughout and
  `reads_visual_input` false only for sRGB Fill);
- `derivation_op(cap)` and `is_derivation_operation(cap)`;
- `Condition` (`SameRepresentation`, `CopiesOnly`, `TimeInvariant` — together
  `UNIVERSAL_CONDITIONS` — and the per-operation conditions), with
  `Condition::statement()`;
- `OperationTier { ExactWhen(&[Condition]), Untiered }`,
  `DerivationOp::exact_conditions()` (the universal conditions followed by the
  operation's own) and `DeterminismTier { EntropyConsistency, DecodeEquivalence }`
  (the registry's `codecs::DeterminismTier`, re-exported);
- `derived_sample_tier(inputs, executed) -> Result<Option<DeterminismTier>,
  DerivedSampleError>`, which implements rules 1, 2, 5 and 6 of §3 over an
  executed list of `ExecutedOp { capability, conditions_met, reads }`. The
  conditions met are listed one by one, so a sink that checked only geometry
  cannot meet same-representation or time-invariant by omission. `inputs` lists
  the tier of every visual input read other than a preceding operation's output,
  the default fill picture included. `reads` names what each instance read, as
  `VisualInput::Listed(k)` (entry `k` of `inputs`) or `VisualInput::Preceding(j)`
  (the output of executed operation `j`, which comes before it). An instance of
  an operation that reads a visual input and names nothing gives no tier,
  wherever it sits in the list. When at least one operation executed, a call that
  contradicts itself is an error: a reference to an input that is not listed, to
  an operation that does not precede the reader, or a listed input that no
  instance names. An empty executed list gives no tier before any of these
  checks, whatever `inputs` holds. What cannot be
  detected is a `reads` that leaves out an input its instance read, unless that
  input is listed and no instance names it. The result is as sound as the sink's
  report of what each instance read, as it is as sound as its report of the
  conditions it checked. An earlier revision
  refused a tier on an empty `inputs` only when the first executed operation
  read a visual input, so an operation after an sRGB Fill could still read an
  input nobody listed and be reported exact;
- `derivation_license()`.

The crate evaluates no derivation operation, and has no receipt emitter. Rules 3
and 4 of §3, and the receipt content of §4, are therefore not implemented.
`ExecutedOp` carries only what the tier depends on (the conditions met and the
inputs read), not the parameter values, span or input hashes a claim binds. The
`wai` command-line tool does not yet list these operations.

## 6. Sources

- ISO/IEC 23001-16:2021, *Information technology — MPEG systems technologies —
  Part 16: Derived visual tracks in the ISO base media file format*. The published
  sample (clauses 1–4, 5.1 and 5.2, and the table of contents; it ends at 5.2) is
  the source for the operation
  names and clause numbers, 4CC identification and the `'uuid'` escape, the
  essential and non-essential rule, the default fill picture, pre-derivation
  transforms, the rule that visual inputs share pixel aspect ratio and bit depth,
  the kinds of input and output (definitions 3.5–3.8: parameter inputs, and
  visual and parameter inputs that are intervals of a track spanning several
  samples, and visual outputs that are sequences of frames), the statement that
  any derivation transformation may have internal time structure (clause 4, with
  a cross-fade as its example), the order in which a parameter or an input takes
  its value, an unlisted input being the default fill picture (clause 5.1), the
  implicit crop and fill of an input whose size differs from the output's
  (clause 4), and the patent statement.
  The parts read say nothing about bit-exactness either way. What the parts not
  read say about it is unknown: clause 5.3 (the semantics of the derivation
  boxes), clauses 6 and 7 (the sample entry, the configuration record and the
  sample format), clause 8 (each operation's definition, syntax and semantics,
  §8.2–§8.11) and Annex A.
- [research/2030-transport-landscape.md](../research/2030-transport-landscape.md)
  §3.1 and §8 item 12.