open-standards/wai/staged-measure-conformance

staged-measure-conformance — staged-delivery measurement

The corpus of extensions/staged-measurement.md §12.3: measured titles, and every figure continuous integration may assert about them (§12.1).

What a conforming implementation reproduces. For each title: the counted bytes of every object, split into envelope header, manifest and payload (§3), and their sums per stage and per prefix, per base and over the title; the bytes of each base's parameter set; the first-presentable bytes (§5.1); every presentation's output digest (SPEC §7 Digests), with its quality integers against the source and against the final prefix (§4), and every prefix's pooled quality; the decoder entries — capability, pin and the envelope decoded — and the output tree's root and leaf count (Appendix C); the layering overhead's rate–quality points, the full title's and each single-layer reference's (§6); the deduplication totals (§9); and the title's content identity (Appendix D). Across the corpus: the record-wide deduplication totals (§9.2).

• fixtures/<name>/ — one title (Appendix D): title.json; its layers (L*.wai) and its single-layer references' layers (R*.wai); source.bin, each unit's canonical buffer at its final shape; expect.json, the fixture tool's own reconstruction digests. Each title's base is a base of a staged-conformance case, read where it is, with the case's stage 0 and parameter set.
• goldens.json — every title's figures, in fixture order, and the record-wide totals.
• quality.json — §4.2's integers for picture pairs drawn from a stated formula, at sizes that place the similarity's windows every way the rule does: narrower or shorter than a window, offsets that reach the edge at a multiple of 4, and an edge-aligned last window on one axis or both.
• counting/ and counting.json — objects whose counting §3.2 decides, each with its counted bytes or the code it is not countable with: a WIH1 base, a WIR1 layer and a WAI2 (counted); the JSON form of WIH1, its base64 text, a JSON string, and WIH1 behind a byte-order mark (bad_magic); a PNG base and an unregistered capability (no-format-check); WIH1 under a wai.video.int_motion fallback (format_capability_mismatch) and under a fallback WAI checks no format for (counted); a WAI2 component with another payload's sha256 (sha256-mismatch); an encrypted payload; bytes after an envelope (trailing); and a WAI2 whose second rendition, PNG, is its classical rendition (no-format-check), with its encrypted twin (encrypted), which a session receiving either states the classical length of (jwp-receipts §2.4.12).
• refusals/<name>/title.json — titles a tool must refuse, each with the code it states (refused): a base whose units pass the chain budget its title states (chain-over-limit, over the staged-delivery corpus's base_over_budget case); the two bound vectors under ../wai-rs/tests/vectors/staged_bounds/, whose headers declare 64 MiB and 1 GiB of output, past the tools' stated limit (cost_over_limit, refused before anything is allocated for them), the second also under a title whose limits state 2^40 for every figure, which a title cannot raise; and the two envelopes there whose capability is refinement-only or companion-only (missing_capability: inert, the fallback never reached); and the still title's up2 layer under a cost limit its base passes and the layer's declared output does not (cost-over-limit).
• sim/ — the delivery simulator's corpus (§10, Appendix B): models/ (a synthetic model of three groups of five units, each a base and two layers per unit, captured as each group ends), plans/ (wai-sim/1), cases.json (a model — a file, or fixture:<name> for a title's objects and counted bytes — a plan, and the case's claim: within-bound, exceeds:<plan> or unbounded) and results.json (every case simulated: each object's delivery, the presentations, first presentable, the byte totals, the missed deadlines, each unit's stall, the plan's bound and the largest stall). The plans cover both carriages and both bounded link schedules, a slow link on which refinements miss their timeouts (with a window, so that a reset takes a subgroup's tail before its own deadlines), a layer cap, live playout with late refinements discarded or presented, two models on which bases were once sent or decoded out of the order they became available (reorder-link, reorder-decode), and a counter-plan for each part of §10.4's bound a schedule can lose: a fifo link with no refinement timeout, whose stall passes the bound of the same fifo link with a short timeout (fifo-timeout-short), and fifo decode on one worker, whose stall passes the bound of the same plan with base-first decode (strict-window). The model per-track gives each refinement a delivery timeout of its own, one per layer, as a MoQ track's DELIVERY_TIMEOUT does (Appendix B): with them a fifo link with no plan timeout is bounded, by the longest of them.

Provenance. Generated here. Each base is a committed vector under ../wai-rs/tests/vectors/, generated too (that directory's README states how), as the staged-delivery corpus wraps it: a wai.neural.int_hyper picture, a wai.video.int_hyper clip, a wai.audio.int_codec clip and a wai.splat.int_codec attribute tensor, each pinned to its parameter set; and the unpinned wai.neural.int_hyper picture of image_int_hyper_unpinned, decoded against the set the sink holds, which its decoder entry names by the digest a pin over it names (SPEC §3.1, Pin by default): the digest the pinned picture's pin names, over the same files. Each source is the base's output with fine detail added by integer functions of position, channel and a seed (for audio, two triangle waves), raised to the final shape by the layer's own predictor where the title changes the shape. The layers are WIR1 payloads coded toward the source by the reference encoder (staged::wir1::encode): up2 then identity for the picture; one identity layer per frame, twice, for the clip; a rate layer to twice the sample rate, then identity, for the audio; identity for the attribute tensor and for the unpinned picture. Each title's single-layer references code one layer straight from stage 0 at the final step and, for some titles, at further steps (the curve of §6.3).

Verifiers.

• wai_stage_measure verify . (from ../wai-rs, feature stage_measure): regenerates every fixture and refusal title in memory and requires it byte for byte; decodes each base through the sink's own base decode (staged::decode_base_pinned) under the title's limits and requires the case's stage 0, capability, fallback, tier and pin; applies the layers through the reference sink (staged::StageSink) under the title's chain budget; requires every refusal title refused with its code; requires every presentation's digest and every reference's final digests to be the fixture tool's (expect.json, written from the encoder's own reconstruction); and requires goldens.json byte for byte. wai_stage_measure generate-goldens . and wai_stage_measure gen-fixtures . write them; wai_stage_measure eval fixtures/<name> prints one title's figures.
• wai_stage_measure verify . also regenerates the simulator's models, plans and cases byte for byte, simulates every case (stage_measure::sim), checks each claim, and requires sim/results.json byte for byte; wai_stage_measure sim <model> <plan> runs one.
• wai_stage_measure bench fixtures/<name> measures a title on the machine it runs on and is never run in CI (§12.2): the decode time of each case — one base decoded or one layer applied, for the (unit, stage) outputs it gives, with its preparation, a base's comparison with its stage 0 and the hashing of its outputs outside the clock — over interleaved rounds whose order rotates, every run's outputs compared with the evaluation's and the bench aborted on a difference (§7.1–§7.3), with each run, the order and the operator's statement of whether the machine was idle in the record (§7.4); and, with --energy, its decode figure after a probe of the meter's rail at rest (§8.6). With --sign it seals the session claim a sink decoding the title would seal (jwp-receipts §2.4.12), whose output tree must be the title's — the root and leaf count goldens.json states — and, beside a clean figure, the measurement claim on it (§8.7). A lib test seals every fixture's session and checks its tree against the evaluation's, and tests/stage_measure.rs that every case, run on its own, gives the evaluation's outputs.
• sim_verify.py .: pure Python with the standard library, sharing no code with wai-rs: Appendix B implemented from the text, with each fixture's model taken from measure_verify.py's own evaluation. It reproduces every result, checks every claim, and requires a counter-case for each part of the bound a schedule can lose.
• sim_diff.py <wai_stage_measure> [seed] [count] runs the simulator differentially: seeded random models (groups of a base and up to three layers, each layer one object or one per unit, a refinement with its own delivery timeout or the plan's) under random plans, each simulated by wai_stage_measure sim and by sim_verify.py. The two must agree on the refusal, or on the whole result, the stall bound (§10.4) and the longest stall; and where the bound applies, every stall must be within it.
• measure_verify.py .: pure Python with the standard library, sharing no code with wai-rs. It reimplements, from the extension and SPEC: the byte rule and the formats it counts; each base's units from its case's stage 0; every layer applied in layer order to every unit it covers, with its predecessor checked; the integer luma, SSE, the Q32 structural similarity with its window placement (Python's // floors toward −∞, as §4.2 states), checked over quality.json as well, PCM and tensor sums and their pooling; the output tree; the prefixes, first presentable, overhead points, deduplication and content identity. It requires its decode to be the fixture tool's and every figure to be goldens.json's. Before it uses a base it runs its own base checks (SPEC §4's dispatch, the output and symbols the payload's header declares against the title's cost limit, the units against its chain budget), and it requires every refusal title refused, by those checks, with the code the title states. It takes BLAKE3 from ../claim-conformance/_blake3.py, the integer payload parsers from ../int-payload-conformance/int_payload_verify.py, and the WAI1 reader, refines, WIR1 and its predictors from ../staged-conformance/staged_verify.py, each written from its specification. Each base's stage 0 is the staged-delivery corpus's, which staged_verify.py checks against the vectors' sidecars.

Every figure here is integer, and the same on every machine. The simulator's times are virtual and labelled simulated; no measured time and no energy is stated: those are measured, never asserted (§12.2).