open-standards/wai/extensions/derived-tracks.md

WAI Extension: Derived Tracks

Status: Draft. Registers the derivation operations of ISO/IEC 23001-16:2021 as wai.derive.* capability strings, each with a per-operation determinism tier, and states how a tier composes over a derived sample. No wire format, and no compositing. The geometric conditions listed per operation are read from the operations' titles: clause 8 of ISO/IEC 23001-16, which defines each operation, was not read. Reference impl: wai-rs codecs::derivation. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Prior art, stated plainly

A track that carries a process the receiver evaluates, rather than the processed result, is not new. ISO/IEC 23001-16:2021, Derived visual tracks in the ISO base media file format (1st edition, 2021-11), ratified it. A derived visual track (sample entry dtrk) carries derived samples, each active for a duration of the track's composition timeline. Each derived sample is an ordered list of derivation operations, each applying a derivation transformation, with its parameters, to an ordered list of inputs. An input is either a visual input (an image item, an interval of an input track or track group possibly spanning several samples, the visual output of a preceding operation, or a default fill picture) or a parameter input (a metadata item, or an interval of a metadata track possibly spanning several samples). A parameter takes the value the derived sample gives it, else the value in the sample entry, else the default its transformation's clause defines. A visual output is one frame or a sequence of frames, and any transformation may have internal time structure (the standard's example is a cross-fade), so the picture may change during the sample's duration. Operations chain, and a derived track can be an input to another. The first edition defines a base set of ten transformations, each identified by a four-character code, with 'uuid' reserved for vendor transformations identified by UUID.

The parts of that specification read for this extension (clauses 1–4, 5.1 and 5.2, and the table of contents, from the published sample, which ends at 5.2) do not say whether an operation's output reproduces byte for byte across conforming readers. The rest was not read: clause 5.3 (the semantics of the derivation boxes), clauses 6 and 7 (the sample entry, the configuration record and the sample format), clause 8, which gives each operation's definition, syntax and semantics, and so any blend or resampling formula, and Annex A. This extension states, per operation, when a claim over its output may hold, in the vocabulary of determinism-tiers, so a receipt or binding over a derived sample can say which tier it actually holds. Every tier rests on conditions that the sink which executed an instance checks on that instance, in space and in time. Clause 8 decides what an instance does, and so which conditions it can meet. What the registry takes from outside clause 8 is each operation's purpose, read from its title, which decides the geometric conditions listed for it (§2, Basis of the tiers). The tiers are not uniform. Dissolve and Scaling compute new sample values by their nature. The others are exact only for instances that copy samples without converting them, and whose output does not change over the span a claim covers.

Scope. WAI does not composite. Presentation stays with the sink, as in linear-film and interactive-worlds. Registering grid and overlay composition states what a reproducibility claim over their output would need; it does not make WAI a renderer. Registering the operations aligns WAI's vocabulary with the standard; it says nothing about which players evaluate derived visual tracks.

2. Registered operations

An executed instance of an operation is exact when every output sample is a copy of an input sample, or of a constant the claim names, placed by integer geometry, with no arithmetic on sample values, no change of sample representation, and no change over the span the claim covers. Inputs at decode-equivalence give an exact instance's output at decode-equivalence. A per-operation tier is one of:

• exact when (conditions): an executed instance is exact when it meets the three universal conditions below and every condition the table lists for its operation, and untiered otherwise. The conditions are properties of the executed instance, checked by the sink that executed it. No operation is exact unconditionally, Identity included.
• untiered: the operation computes new sample values, by blending or resampling, and WAI pins no arithmetic for it. The output carries no tier, whatever its inputs held, until an arithmetic is pinned and an instance is shown to follow it. Dependence on time is not what makes an operation untiered: every operation needs time-invariant.

The three universal conditions, required of every tiered operation:

• same-representation: every visual input, and every constant the instance writes (a fill colour, the default fill picture), is already in the output's sample representation: the same bit depth, chroma format and chroma siting, colour primaries, transfer characteristics, matrix coefficients, range and component set. No sample value is converted. Clause 4 requires the visual inputs of a derived sample to share pixel aspect ratio and bit depth with each other; the clauses read say nothing about the other properties, or about the output's representation relative to its inputs. So an 8-bit input composed into a 10-bit output, a BT.709 input composed into a BT.2020 output, or an Identity whose output representation differs from its input's holds no tier.
• copies-only: every output sample value is, bit for bit, the value of an input sample or of a constant whose value the claim names. The instance computed no sample value.
• time-invariant: over the span the claim covers, every visual input the instance reads is one frame and every parameter it uses has one value. The instance also has no time structure of its own, such as a transition or a parameter interpolated over time. So its output over the span is one frame, fixed by those frames and values. The span is either the derived sample's whole duration or one output frame at a composition instant the claim names (§3 rule 3). A parameter counts whatever its source: the derived sample, the sample entry, the default of the operation's clause, or a parameter input. Clause 4 lets one derivation duration span several samples of an input track, and lets any transformation have internal time structure. A visual or parameter input may be an interval of a track spanning several samples, and a visual output may be a sequence of frames. So two instances hold no tier for a claim over the whole duration: an ROI selection whose region is driven by an interval of a metadata track whose value changes within the sample, and an Identity over an input interval holding several samples. Either can hold one for a claim over one output frame, at an instant at which nothing it reads changes, if it has no time structure of its own.

Because every tiered operation requires all three, an instance is untiered, not over-claimed, if it uses any parameter that clause 8 allows to convert or compute sample values, or to vary its output over time. This holds as long as the executing sink lists only the conditions it actually checked.

capability	23001-16 clause	operation	tier	conditions beyond the universal three (read from the operation's title; clause 8 was not read), or why untiered
wai.derive.identity	8.2	Identity	exact when	none. It copies its input; uncovered output samples come from the default fill picture, a constant the claim must name (§3 rule 5). Whether clause 8 lets Identity emit a representation other than its input's was not read, so same-representation is checked, not assumed
wai.derive.srgb_fill	8.3	sRGB Fill	exact when	none. The colour is stated in sRGB; writing it into any other representation is a conversion, which same-representation excludes. As its title is read, it reads no visual input, and it is the only operation registered as reading none
wai.derive.dissolve	8.4	Dissolve	untiered	a transition that mixes two inputs, so its output values are computed, not copied. A transition is also time structure of its own, which time-invariant excludes for every operation (clause 4's example of internal time structure is a cross-fade). Untiered until a blend arithmetic, and how its weights depend on the instant, are pinned; clause 8.4, which may fix them, was not read
wai.derive.crop	8.5	Crop	exact when	the kept region lies on the sample grid of every component (no component is resampled)
wai.derive.rotation	8.6	Rotation	exact when	the angle is a multiple of 90°, and every component is sampled at the same resolution, so the rotation permutes samples
wai.derive.mirror	8.7	Mirror	exact when	the reflection maps every component's sample grid, including its siting, onto itself (always so without chroma subsampling)
wai.derive.scaling	8.8	Scaling	untiered	resamples, so its output values are computed, not copied. Untiered until an interpolation filter and its rounding are pinned; clause 8.8, which may fix them, was not read
wai.derive.roi_selection	8.9	Region of interest (ROI) selection	exact when	the region lies on the sample grid of every component, and is emitted at its own size
wai.derive.grid_composition	8.10	Grid composition	exact when	every input is placed at its own size, at offsets on every component's sample grid
wai.derive.overlay_composition	8.11	Overlay composition	exact when	every input is placed at its own size, at offsets on every component's sample grid, and opaquely (no alpha, opacity or blending)

• Identification. Each capability string stands for the transformation its clause defines, including that clause's four-character code. The registry does not yet record the codes: clause 8 of the published text was not available when this was written. A sink maps each capability to the code its clause defines, and MUST NOT map one by a code taken from any other source. Codes from earlier drafts circulate in secondary sources.
• Basis of the tiers. Tiers are assigned from what each operation does to sample values, in space and in time, not from clause 8, which was not read. Each condition is a property of an executed instance, not of parameter syntax. The three universal conditions exclude any conversion or computation of sample values, and any change over the span a claim covers. What is taken from outside clause 8 is each operation's purpose, read from its title (a crop selects, a rotation rotates), which decides the geometric conditions listed for it, and whether it reads a visual input at all (only sRGB Fill is registered as reading none). Reading clause 8 may show one of those lists to be incomplete. It may also make some conditions unnecessary (for example, if Rotation admits only quarter turns, or if the output representation is always the inputs'). Clause 8 may also fix a blend or resampling arithmetic bit-exactly; that would be the start of a pin for Dissolve or Scaling, not a tier in itself. Clause 4, which was read, already shows where a list may fall short: wherever an input's size differs from the output's, the input is cropped or filled implicitly, for every operation unless the operation's clause overrides that. The fill is covered by §3 rule 5. The clauses read do not say where such an input is placed, and no list, Identity's included, carries a condition on that placement.
• Correction. An earlier revision said that no tier here rested on anything clause 8 might fix. The time dimension did. Only Dissolve was untiered for depending on time, as if clause 4's cross-fade were particular to it, but clause 4 states generally that a derivation transformation may have internal time structure. Time-invariant is now a universal condition.
• Not payload capabilities. An operation names no payload format; it acts on samples another capability decoded. SPEC §4 assigns every payload exactly one capability, so these strings are not in codecs::known_capabilities and a manifest cannot name one as its payload's capability. A sink advertises the operations it evaluates the way it advertises codecs.
• Licence. The Introduction of ISO/IEC 23001-16:2021 records a patent claimed to be required for compliance, whose holder has given a reasonable-and-non-discriminatory licensing assurance. The registry classifies the operations as patent-pool, conservatively (codecs::derivation::derivation_license).

3. Composing a tier over a derived sample

1. Weakest step. A derived sample holds decode-equivalence only if every visual input holds decode-equivalence and every executed operation instance is exact: it meets the three universal conditions and every condition listed for its operation. Otherwise it holds no tier, and a receipt over it MUST NOT carry a reproducible derived-sample hash.

2. entropy-consistency does not pass through. Derivation operations act on reconstructed samples, which that tier does not fix. A derived sample over an input at entropy-consistency holds no tier.

3. The claim is about what was executed, with what, and when. ISO/IEC 23001-16 lets a reader treat an unsupported non-essential operation as a null operation, so two conforming readers can emit different samples from one derived sample. A parameter can come from a metadata track, and the output can change within the sample's duration. A tier claim or receipt over a derived sample MUST name:

   • the operations executed, in order;
   • for each instance, the parameter values it applied, whatever their source (the derived sample, the sample entry, the default of the operation's clause, or a parameter input), with any region, offset or size expressed in samples;
   • the span it covers. For a claim over the whole duration, that is the derived sample's composition time and duration on the derived track's timeline, in its timescale. Otherwise it is the composition instant of each output frame the claim covers;
   • its inputs, visual and parameter, by content hash. For an interval of a track, the hash covers the samples the instance read.

   Two sinks' derived-sample hashes are comparable only when all four are equal. A sink MUST NOT present a hash computed with an operation skipped as the hash of the sample with that operation, and MUST NOT present the hash of one output frame as the hash of a derived sample whose output changes over its duration.

4. Transforms applied before derivation count. ISO/IEC 23001-16 applies the transformative properties of inputs (clean aperture, track matrix, and the like) before the derivation operation. For a tier claim each counts as a step, tiered as the registered operation it performs. A unity matrix is identity; a quarter-turn or reflection matrix is rotation or mirror; a clean aperture is a crop. Any other matrix is untiered, whether or not it resamples (a scale, a shear or a sub-sample translation does): the registry pins no arithmetic for it.

5. The default fill picture is an input. Where output samples come from the configuration record's default fill (black, grey or white), whatever the operation, the claim MUST name the fill's sample values in the output representation. A fill whose values the claim does not name is an input with no tier, and an instance that writes it does not meet copies-only. An input that neither the derived sample nor the sample entry lists is the default fill picture (clause 5.1). So an operation reads the fill for every visual input it reads that neither lists, and a claim over it names the fill as an input. Where an instance sits in the executed list does not say what it read: clause 4 lets an operation that is not first read the output of any previous operation, only new inputs, or both. A claim therefore names, for each instance, the visual inputs it read: an input, or the output of an operation executed before it. An instance of an operation that reads a visual input and names none holds no tier, wherever it sits in the list.

6. Unregistered operations cannot be tiered. A 'uuid' vendor operation, or any transformation this registry does not name, makes the derived sample untiered. The reference implementation returns an error rather than guess.

7. Relation to determinism-tiers §2. That section's rule is that a determinism claim covers what the sink emits. A derived sample is an emitted medium, and this section says when a claim over it holds. It does not relax the rule for the inputs: a hash over a decoded input is not a hash over the derived sample.

4. Receipt

A receipt over a derived sample SHOULD carry:

• the executed operation list, as capability strings in order, with the conditions each instance met, the universal ones included, the visual inputs each instance read (§3 rule 5), and the parameter values each instance applied (§3 rule 3);
• the span the claim covers: the derived sample's composition time and duration, or the composition instant of each output frame covered;
• the content hashes of the inputs, visual and parameter;
• the resulting tier;
• only when that tier is decode-equivalence, the hash of each output frame the claim covers. A claim over the whole duration covers one frame.

The energy of evaluating the graph is measured and attributed as for any other work (energy-measurement). The tier and the energy are separate claims, and neither implies the other. This extension defines the content of such a receipt, not a field layout; deriving-edge §5 gives one, for the derivations an edge makes and publishes.

5. Reference implementation

wai-rs, module codecs::derivation, with no feature gate:

• derivation_operations(), the ten registered operations in clause order (DerivationOp { capability, name, clause, fourcc, tier, basis, reads_visual_input }, with fourcc None throughout and reads_visual_input false only for sRGB Fill);
• derivation_op(cap) and is_derivation_operation(cap);
• Condition (SameRepresentation, CopiesOnly, TimeInvariant — together UNIVERSAL_CONDITIONS — and the per-operation conditions), with Condition::statement();
• OperationTier { ExactWhen(&[Condition]), Untiered }, DerivationOp::exact_conditions() (the universal conditions followed by the operation's own) and DeterminismTier { EntropyConsistency, DecodeEquivalence } (the registry's codecs::DeterminismTier, re-exported);
• derived_sample_tier(inputs, executed) -> Result<Option<DeterminismTier>, DerivedSampleError>, which implements rules 1, 2, 5 and 6 of §3 over an executed list of ExecutedOp { capability, conditions_met, reads }. The conditions met are listed one by one, so a sink that checked only geometry cannot meet same-representation or time-invariant by omission. inputs lists the tier of every visual input read other than a preceding operation's output, the default fill picture included. reads names what each instance read, as VisualInput::Listed(k) (entry k of inputs) or VisualInput::Preceding(j) (the output of executed operation j, which comes before it). An instance of an operation that reads a visual input and names nothing gives no tier, wherever it sits in the list. When at least one operation executed, a call that contradicts itself is an error: a reference to an input that is not listed, to an operation that does not precede the reader, or a listed input that no instance names. An empty executed list gives no tier before any of these checks, whatever inputs holds. What cannot be detected is a reads that leaves out an input its instance read, unless that input is listed and no instance names it. The result is as sound as the sink's report of what each instance read, as it is as sound as its report of the conditions it checked. An earlier revision refused a tier on an empty inputs only when the first executed operation read a visual input, so an operation after an sRGB Fill could still read an input nobody listed and be reported exact;
• derivation_license().

codecs::derivation evaluates no derivation operation and emits no receipt, so rules 3 and 4 of §3, and the receipt content of §4, are not implemented there. The deriving edge's module (derive_edge, feature derive_edge; deriving-edge) evaluates the operations that extension registers, over 8-bit RGB stills read at one frame each, reads which of them are exact from derived_sample_tier, and signs a record carrying §4's content. ExecutedOp carries only what the tier depends on (the conditions met and the inputs read), not the parameter values, span or input hashes a claim binds. The wai command-line tool does not yet list these operations.

6. Sources

• ISO/IEC 23001-16:2021, Information technology — MPEG systems technologies — Part 16: Derived visual tracks in the ISO base media file format. The published sample (clauses 1–4, 5.1 and 5.2, and the table of contents; it ends at 5.2) is the source for the operation names and clause numbers, 4CC identification and the 'uuid' escape, the essential and non-essential rule, the default fill picture, pre-derivation transforms, the rule that visual inputs share pixel aspect ratio and bit depth, the kinds of input and output (definitions 3.5–3.8: parameter inputs, and visual and parameter inputs that are intervals of a track spanning several samples, and visual outputs that are sequences of frames), the statement that any derivation transformation may have internal time structure (clause 4, with a cross-fade as its example), the order in which a parameter or an input takes its value, an unlisted input being the default fill picture (clause 5.1), the implicit crop and fill of an input whose size differs from the output's (clause 4), and the patent statement. The parts read say nothing about bit-exactness either way. What the parts not read say about it is unknown: clause 5.3 (the semantics of the derivation boxes), clauses 6 and 7 (the sample entry, the configuration record and the sample format), clause 8 (each operation's definition, syntax and semantics, §8.2–§8.11) and Annex A.
• research/2030-transport-landscape.md §3.1 and §8 item 12.