diff --git a/.github/workflows/wai-conformance.yml b/.github/workflows/wai-conformance.yml index 80346dcd8..89681b505 100644 --- a/.github/workflows/wai-conformance.yml +++ b/.github/workflows/wai-conformance.yml @@ -141,11 +141,14 @@ jobs: # is in the set because the reference meters (wai_meter, wai_world_meter) write a # measurement report by default and require it: without it they would drop out # of this build on every leg of this matrix, without a word. staged and the integer - # media features build the staged-delivery core, its corpus tool and `wai stage`. - run: cargo build --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,receipt_claims,staged,video_int,audio_int,splat_int + # media features build the staged-delivery core, its corpus tool and `wai stage`; + # stage_measure builds the staged-measurement core and wai_stage_measure; telemetry + # the delivery-telemetry mapping and wai_telemetry. + run: cargo build --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,receipt_claims,staged,video_int,audio_int,splat_int,stage_measure,telemetry + # video_package runs the package receipt's dedup tests, which content_dedup feeds. - name: Library unit tests (the fixed-point determinism floors) working-directory: wai/wai-rs - run: cargo test --lib --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,oer,receipt_claims,key_discovery,staged,video_int,audio_int,splat_int + run: cargo test --lib --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,oer,receipt_claims,key_discovery,staged,video_int,audio_int,splat_int,stage_measure,telemetry,video_package # The JPEG AI exact decoder reproduces across platforms only if no platform math # library is on its path. Two guards, each failing the step: clippy's # disallowed-methods (wai-rs/clippy.toml), which each module on the decode path @@ -201,7 +204,11 @@ jobs: # own binary, a counting allocator) checks that the staged sink's work and memory per # layer grow linearly in the layer's units, that a layer past the chain budget is # refused before any stream is decoded, that what no unit holds is dropped, and that a - # WIR1 parse holds no more than its declared working bytes. Cargo.toml declares them, and the other + # WIR1 parse holds no more than its declared working bytes. tests/stage_measure.rs counts every + # committed staged envelope as exactly its length, refuses JSON, base64 and another capability's + # format, and evaluates a measured title read from a map of files to its golden, refusing a + # changed byte of a layer, the base, the source or a parameter-set file (extensions/ + # staged-measurement.md). Cargo.toml declares them, and the other # feature-gated targets named here, with required-features, so if this step ever loses a # feature, cargo stops with an error instead of compiling an empty binary that reports ok. # Each committed vector's decoded output is also checked against the SHA-256 in @@ -211,24 +218,32 @@ jobs: # wai.generate.spectral vector, which `generate` enables. - name: Fast integer kernels ≡ reference kernels (overflow checks on) working-directory: wai/wai-rs - run: cargo test --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate,staged --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate --test int_payload --test int_payload_preflight --test int_payload_cost --test int_video_enc_memory --test cli_stage --test staged_resources + run: cargo test --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate,staged,stage_measure --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate --test int_payload --test int_payload_preflight --test int_payload_cost --test int_video_enc_memory --test cli_stage --test staged_resources --test stage_measure --test stage_measure_bounds - name: Fast integer kernels ≡ reference kernels (release, overflow checks off) working-directory: wai/wai-rs - run: cargo test --release --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate,staged --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate --test int_payload --test int_payload_preflight --test int_payload_cost --test int_video_enc_memory --test cli_stage --test staged_resources + run: cargo test --release --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate,staged,stage_measure --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate --test int_payload --test int_payload_preflight --test int_payload_cost --test int_video_enc_memory --test cli_stage --test staged_resources --test stage_measure --test stage_measure_bounds # The integer decode path builds under each of its features alone, so a sink can take # one medium without the others, and for wasm32 (a browser sink); `int_payload` arms a - # medium only under its feature, and none needs `neural`. + # medium only under its feature, and none needs `neural`. So does the staged-measurement + # core (`stage_measure`), which pulls no ONNX and reads no file on wasm32. - name: Each integer feature alone, and wasm32 (unix) if: matrix.os != 'windows' shell: bash working-directory: wai/wai-rs run: | set -euo pipefail - for f in neural_int video_int audio_int splat_int staged; do + for f in neural_int video_int audio_int splat_int staged stage_measure telemetry; do cargo check --lib --no-default-features --features "$f" done + # The decode bench prints its machine line from bench_env; no other step builds it. + cargo build --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int --bin wai_decode_bench rustup target add --toolchain 1.98.0 wasm32-unknown-unknown cargo check --lib --target wasm32-unknown-unknown --no-default-features --features video_int,audio_int,splat_int,staged + # The staged-measurement core is wasm32-clean: a browser computes the same figures. + # Built, not only checked, so codegen and linking of the library run for wasm32. + cargo build --lib --target wasm32-unknown-unknown --no-default-features --features stage_measure + # So is delivery telemetry: a browser sink writes the same reports. + cargo build --lib --target wasm32-unknown-unknown --no-default-features --features telemetry # carriage: every registered capability round-trips byte-exact. cli_wrap_intent and # cli_registry run the `wai` binary, which builds without the codecs feature: the # intent rule of `wai wrap`, its refusal of candidate capabilities (SPEC §5), its media @@ -299,7 +314,7 @@ jobs: working-directory: wai/wai-rs run: | set -euo pipefail - F=world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged + F=world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged,stage_measure,telemetry v() { echo "::group::$1"; cargo run -q --no-default-features --features "$F" --bin "$2" -- verify "$3"; echo "::endgroup::"; } v worlds wai_world_vectors ../world-conformance v spatial-audio wai_audio_vectors ../audio-conformance @@ -319,6 +334,8 @@ jobs: v signers wai_signer_vectors ../signer-conformance v int-payload wai_int_payload_vectors ../int-payload-conformance v staged wai_staged_vectors ../staged-conformance + v staged-measure wai_stage_measure ../staged-measure-conformance + v telemetry wai_telemetry ../telemetry-conformance # The pin corpus's second implementation: Python standard library only, sharing no # code with wai-rs. One leg is enough — it checks digests and listing rules, which # do not depend on the architecture — and the macOS host has python3. @@ -347,7 +364,7 @@ jobs: shell: pwsh working-directory: wai/wai-rs run: | - $F = "world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged" + $F = "world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged,stage_measure,telemetry" function v($grp, $bin, $path) { Write-Output "::group::$grp" cargo run -q --no-default-features --features $F --bin $bin -- verify $path @@ -372,6 +389,8 @@ jobs: v signers wai_signer_vectors ../signer-conformance v int-payload wai_int_payload_vectors ../int-payload-conformance v staged wai_staged_vectors ../staged-conformance + v staged-measure wai_stage_measure ../staged-measure-conformance + v telemetry wai_telemetry ../telemetry-conformance # The generated v2 vocabulary (wai/vocab/v2.*) is checked as a consumer uses it: # sample manifests, typed and untyped, WAI1 and WAI2, are expanded through the @@ -457,6 +476,22 @@ jobs: working-directory: wai shell: bash run: python3 staged-conformance/staged_verify.py staged-conformance + # The staged-measurement corpus's second implementation (staged-measure-conformance/README): + # it counts every object, applies every layer from its base's stage 0, computes the quality + # integers, the output tree, the prefixes, the overhead points and the dedup totals itself, + # and requires its decode to be the fixture tool's and its figures to be goldens.json's. + # Standard library only. + - name: measure_verify.py + working-directory: wai + shell: bash + run: python3 staged-measure-conformance/measure_verify.py staged-measure-conformance + # The delivery simulator's second implementation: Appendix B from the text, every case + # of staged-measure-conformance/sim/ reproduced and its claim checked, with a + # counter-case required for each part of the stall bound a schedule can lose. + - name: sim_verify.py + working-directory: wai + shell: bash + run: python3 staged-measure-conformance/sim_verify.py staged-measure-conformance - name: claim_verify.py, differential.py, jwp_verify.py, signer_verify.py working-directory: wai shell: bash @@ -470,6 +505,28 @@ jobs: "$GITHUB_WORKSPACE/wai/wai-rs/target/debug/wai_signer_vectors" verdicts signer-conformance/vectors.json > "$RUNNER_TEMP/signer-rs.tsv" diff "$RUNNER_TEMP/signer-py.tsv" "$RUNNER_TEMP/signer-rs.tsv" + # Delivery telemetry's second implementation is an independent open-source CMCD and CMSD + # library at locked versions (telemetry-conformance/cml/package-lock.json). The + # differential builds each observation's report from the extension's tables, encodes it + # with the library, requires the bytes of expected/cmcd.json and expected/cmsd.json, reads + # the expected strings back as dictionary, query and header shards, and validates every + # report with no error. One platform: string encoding, not arithmetic. + telemetry-differential: + name: delivery telemetry · CMCD and CMSD against an independent library + runs-on: linux + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-node@v4 + with: + node-version: '22' + - name: npm ci, then the differential + working-directory: wai/telemetry-conformance/cml + shell: bash + run: | + set -euo pipefail + npm ci --no-audit --no-fund + node differential.mjs .. + byte-exact: name: cross-arch byte-equality · ${{ matrix.os }} runs-on: ${{ matrix.os }} diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index d28ee9e05..888adadd0 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -227,6 +227,110 @@ wire format may change and such changes are recorded here. `rdiv` and the rate indices. `rdiv` and the rate indices are computed in checked 64-bit arithmetic. +### Staged-delivery measurement: counted bytes, integer quality, the output tree + +- **The extension** [staged-measurement](extensions/staged-measurement.md): how a + figure about staged delivery is measured and published. The counted bytes of an + object are its envelope to the byte (`12 + man_len + P` for `WAI1`, + `8 + man_len + 2 + 8n + Σ P` for `WAI2`), and an object is countable only where WAI + checks the format of every payload it carries: the pinned integer formats and `WIR1`, + by the parse a sink runs, for the capability and for a `WAI1` fallback with a pinned + format. Any other capability, a classical codestream or an unregistered string among + them, is not countable (`no-format-check`), and neither is a `WAI2` component whose + `sha256` is not its payload's, nor an encrypted payload; no estimate stands in for a + figure that is not countable. A figure carries the framing label `registered`. Quality is integers per + presentation and per prefix, against the source and against the final prefix: SSE + over RGB and over an integer BT.601 luma, a Q32 structural similarity over clipped + 8×8 windows at stride 4 with an edge-aligned last window on each axis, so every pixel + is in a window (bounded in 128-bit integers), PCM SSE and reference energy, + and tensor SSE and largest difference, each pooled over units; decibels are display + only. First presentable, layering overhead as two rate–quality points (never a + ratio), delivery simulation, decode time and energy (only a clean attribution verdict is a figure; a rail + that is not live is unmetered), deduplication, dated records, what CI asserts, and a + media-class matrix are stated. The output tree (Appendix C) commits to every + presentation, one SHA-256 leaf per (unit, layers), under RFC 6962's tree hash, by its + root and leaf count together, with audit paths checked as RFC 9162 checks them under + the committed count (`output_tree::OutputCommitment`). +- **Measured titles** (Appendix D): a staged set over a staged-conformance base, read + where it is, with only what the stage model does not carry beside it (unit duration, + capture times, the source, single-layer references, the fixture tool's own + reconstruction digests), and a content identity over the measured files. +- **Reference implementation.** `wai-rs` feature `stage_measure` (pure integer, no + ONNX, wasm32-clean; CI builds it for wasm32): `measure_framing`, `output_tree`, + `stage_measure::{title, decode, run}`, which decodes each base through the sink's own + base decode, `staged::decode_base_pinned` (which also gives the pin it verified), + under the cost limit, and applies the layers through `staged::StageSink` + under the chain budget, handing each stage's output over as it is reached. The tools' + stated limits are `staged::TOOL_DEFAULT_LIMIT` and `staged::TOOL_DEFAULT_CHAIN_BUDGET` + (staged-measurement Appendix D states each figure and why); `wai stage` (`apply`, + `refine`, `base`), `wai_stage_measure eval` and `wai_stage_bench` decode every base + under them, hold every layer and chain to them, and take + `--limit OUT,SYMBOLS,WORKING,MACS`. A title's `limits` lower a limit and never raise + one: each figure is the lower of the title's and the tool's. + The corpus's `refusals/` hold titles a tool must refuse, with the code: a base past its + chain budget, the two declared-clip bound vectors (refused before anything is + allocated for them; a counting-allocator test shows it), the 1 GiB one again under a + title that states limits of 2^40, two inert envelopes, and a layer past its title's + cost limit. `tests/vectors/staged_bounds/` gains a base whose up2 layer declares + 67,092,480 output bytes, which `wai stage apply` and `wai_stage_bench` refuse with no + flag. Always on: + `quality` and `content_dedup` (the package receipt's dedup figures now go through + it; a map that gives one content hash two lengths no longer verifies); native only: + `bench_env`, the machine record, which `wai_decode_bench` now prints from. The + `wai_stage_measure` tool evaluates a title (`eval`), verifies the corpus (`verify`) + and writes it (`gen-fixtures`, `generate-goldens`). +- **The corpus** [`staged-measure-conformance/`](staged-measure-conformance/): a + picture, a clip, an audio clip and a splat attribute tensor, each with its figures in + `goldens.json`; `wai_stage_measure verify` and an independent Python verifier, + `measure_verify.py`, both reproduce every figure, and CI runs both. +- **The delivery simulator** (§10, Appendix B; `stage_measure::sim`, + `wai_stage_measure sim`): a discrete-event model in integer microseconds of a sender + putting a title's objects on one link (`strict-priority` or `fifo`; `datagram` or + `subgroup` carriage with delivery timeouts that drop or reset; a layer cap; a VOD + window or live capture) and a sink decoding and presenting them (workers, `base-first` + or `fifo` decode, playout deadlines). Its figures are labelled simulated. Bases are + sent, and decoded under `base-first`, in the order they became available. A + refinement's stall of its base is then bounded by one chunk (strict priority) or a + timeout and one chunk (`fifo`), plus one refinement decode (`base-first`): §10.4 + proves it, for every plan with no base timeout, a bounded link schedule and + `base-first` decode. It is also checked on generated plans drawn over that whole + range and on the corpus's plans, and for each unbounded part a counter-plan's stall + passes the bound of the same plan with that part restored. The corpus `sim/` and an independent + `sim_verify.py` reproduce every result. +- **Wire.** No envelope, payload, receipt, claim, pin or corpus file of an earlier + revision changes bytes. + +### Delivery telemetry: CMCD version 2 and CMSD for WAI objects + +- **The extension** [delivery-telemetry](extensions/delivery-telemetry.md): an unsigned, + advisory observation about WAI delivery (no joule figure, no acquisition class, no + quality figure; content and receipt hashes only as pointers), read strictly with + exact members; its CMCD version 2 report, in Request mode for requests and Event mode + otherwise (`e=rr` for responses, play states, custom events for switches, dispatch + outcomes, layer-cap changes and dropped frames), with header shards; custom keys + under `science.transaction.wai-`, each carrying a String; switch reasons; CMSD-Static + and CMSD-Dynamic for a WAI object; session metrics from one session's log (video + start time, exit before start, start failure, stalls, engagement, units, counted + bytes); and privacy rules: a content id is a content hash or a short token, and an + emitter puts no address, query, token or personal data in it; an object URL holding a + `?`, `#`, `@` or `\` is refused rather than stripped, so no query, signed-URL token or + user information enters an observation however a parser splits the URL; and a + Request-mode report goes only to the host serving the object. Cohort assignment and capable share are not in this revision. +- **Reference implementation.** `wai-rs` feature `telemetry` (no dependency; + wasm32-clean): `telemetry::{sf, observation, cmcd, cmsd, qoe}`, with the RFC 9651 + subset the reports use; the `wai_telemetry` tool. +- **The corpus** [`telemetry-conformance/`](telemetry-conformance/): observations of + every event kind with their reports, refused observations, CMSD descriptions and + session logs. `wai_telemetry verify` requires every value; a differential encodes + every report with an independent open-source CMCD and CMSD library at locked versions + and requires the same bytes, reads them back, and validates them; CI runs both. +- **Other text.** [moq-streaming-format](extensions/moq-streaming-format.md) §5 requires + that telemetry **MUST NOT** restate the per-object property's figure, class or evidence; + [energy-exposure-mapping](research/energy-exposure-mapping.md) gains a dated + addendum. +- **Wire.** No envelope, payload, receipt, claim or corpus file of an earlier revision + changes bytes. + ### The integer codecs' payloads pinned in place: `WIV1`, `WIH1`, `WIA1`, `WIS1` - **Pinned in place** (SPEC §5, *Payloads of the WAI-native integer codecs*, and the diff --git a/wai/extensions/delivery-telemetry.md b/wai/extensions/delivery-telemetry.md new file mode 100644 index 000000000..1e1d1b29b --- /dev/null +++ b/wai/extensions/delivery-telemetry.md @@ -0,0 +1,248 @@ +# WAI Extension: Delivery Telemetry + +> Status: Draft. An **unsigned, advisory** observation about WAI delivery that a sink, +> edge or origin emits; its mapping onto the keys of CMCD version 2 (CTA-5004-B) and +> CMSD (CTA-5006), written as RFC 9651 Structured Field Values; and the session +> metrics a log of observations gives. Reference impl: `wai-rs` — +> `telemetry::{sf, observation, cmcd, cmsd, qoe}`; the `wai_telemetry` tool; the corpus +> [`telemetry-conformance/`](../telemetry-conformance/), with a differential against an +> independent open-source CMCD and CMSD library. Keywords **MUST**, **MUST NOT**, +> **SHOULD**, **MAY** are RFC 2119/8174. + +## 1. Status of an observation + +§1.1 No signature covers an observation. A relying party **MUST NOT** treat one as +evidence of delivery, presentation, quality or energy. What a session did is attested, +where it is, by receipts and claims ([jwp-receipts](jwp-receipts.md)), never by +telemetry. + +§1.2 An observation **MUST NOT** carry a joule figure, an acquisition class or a quality +figure. It **MAY** carry content hashes and link hashes as pointers. + +§1.3 Telemetry binds no receipt to CMCD or CMSD, and no receipt to telemetry. + +## 2. The observation + +§2.1 An observation is a JSON object with exactly these members; a member that does not +apply is `null`. Integers are unsigned and at most 2⁵³ − 1, and a member a report +carries as an Integer (§3.3: `sn`, `ts_ms`, the object's `d_ms`, `dl_ms`, `ttfb_ms`, +`ttlb_ms` and `rc`, and the metrics) at most 10¹⁵ − 1, the largest RFC 9651 Integer, so +every observation can be reported. A string a report carries is printable ASCII. + +| member | type | meaning | +|---|---|---| +| `v` | integer | `1` | +| `sid` | string | 32 lowercase hexadecimal digits, fresh per session, derived from no identifier (§8) | +| `sn` | integer | the observation's sequence number in the session | +| `ts_ms` | integer or `null` | Unix time of the event in milliseconds; required for every event reported in Event mode (§3.1) | +| `event` | string | `request`, `response`, `start`, `present`, `switch`, `layer-change`, `stall-start`, `stall-end`, `frame-drop`, `capable`, `abandon` or `end` | +| `cid` | string or `null` | the entry object's content hash (64 lowercase hexadecimal digits), or a deployer's content id of 1–64 letters, digits, `.`, `_` and `-` | +| `st` | string | stream type: `v` (on demand), `l` (live) or `ll` (low-latency live) | +| `sf` | string | `o` for WAI objects over the HTTP binding; `h` or `d` for a WAI rendition referenced from HLS or DASH | +| `object` | object or `null` | for `request` and `response`: the object (§2.2) | +| `present` | object or `null` | for `present`: `{ unit, layers, first, late_ms, decode_us }`; `first` is `true` only for the session's first presentation; `layers` is at least 1 | +| `switch` | object or `null` | for `switch`: `{ unit, from, to, reason }`; `from` and `to` are `primary` or `fallback` and differ; `reason` is a §5 term | +| `layer_change` | object or `null` | for `layer-change`: `{ from, to }`, a change of `layer_cap`; `null` is no cap | +| `capable` | object or `null` | for `capable`: `{ outcome, reason, classical_floor }` (§2.3) | +| `cohort` | string or `null` | `control` or `wai`: the arm the content's service assigned the session to | +| `metrics` | object or `null` | `{ msd_ms, bsa, bsda_ms, dfa }`: media start delay, buffer starvations and their total duration since the session began, and dropped frames since the session began | +| `claim` | 64 hexadecimal digits or `null` | on `end` only: the link hash of the session's `reconstruct-session` claim, where the sink sealed one | + +§2.2 **The object** has exactly the members `ot`, `url`, `content_hash`, `capability`, +`rendition`, `layer`, `layers_total`, `layer_cap`, `counted`, `transferred`, `d_ms`, +`dl_ms`, `ttfb_ms`, `ttlb_ms`, `rc`, `path` and `receipt`, each `null` where it does +not apply: + +- `ot` is a CTA-5004-B object type token (`m`, `a`, `v`, `av`, `i`, `c`, `tt`, `k`, + `o`); `content_hash` the object's content hash and `receipt` a receipt's link hash, + each 64 lowercase hexadecimal digits; `capability` at most 64 characters; + `rendition` the `WAI2` rendition index; `layer` and `layers_total` the object's + stage and its stage set's stage count; `layer_cap` the cap the request asks for; + `counted` its counted bytes and `transferred` its transferred bytes + ([staged-measurement](staged-measurement.md) §3); `path` `primary` or `fallback`; + `url` the object's address, at most 2048 characters, holding no `?`, `#`, `@` or `\` + (§8.2). +- The request keys `ot`, `d_ms`, `dl_ms` and `layer_cap` are `null` on a `response`; + the response keys `rc`, `ttfb_ms`, `ttlb_ms` and `url` are `null` on a `request`. + +§2.3 **A `capable` observation** reports one WAI-arm dispatch outcome: `not-reached` +(no dispatch decision for a WAI object was reached), `missing-capability`, +`pin-refused`, `module-refused`, `below-rate` or `capable`. `reason` is SPEC §3.1's pin +refusal for `pin-refused` (one of the terms §5 lists before `missing-capability`), and +`null` otherwise. `classical_floor` is the platform's answer for the classical +rendition's codec string, `supported` or `unsupported`, where the sink asked. The +outcome comes from WAI's own dispatch: advertised capabilities, pin verification and a +timed decode. A sink **MUST NOT** derive it from a platform media-capability query, +which answers only for the platform's own decoders; it reports that answer separately, +as `classical_floor`. How sessions are assigned to arms, and the shares computed over +outcomes, are not specified in this revision. + +§2.4 An observation that breaks §2.1–§2.3 is refused as a whole: a member missing or +not listed, a value of another type or outside its range, an object on an event that +has none, or none on one that has one. + +## 3. CMCD version 2 mapping + +§3.1 The modes: + +- a `request` observation is reported in Request mode; +- every other observation is reported in Event mode, with `ts`: a `response` as an + event report with `e=rr`, whose response keys (`rc`, `ttfb`, `ttlb`, `url`) are never + carried in Request mode or in a header shard; +- a `present` observation with `first: false` is not reported in CMCD. + +Every report carries `v=2`, `sid`, `sn`, `st` and `sf`, and `cid` where it is not `null`. + +| observation | CMCD | mode | +|---|---|---| +| `sid`, `sn`, `cid`, `st`, `sf` | same keys | every report | +| `object.ot`, `object.d_ms`, `object.dl_ms` | `ot`, `d`, `dl` | request | +| `object.rc`, `object.ttfb_ms`, `object.ttlb_ms`, `object.url` | `rc`, `ttfb`, `ttlb`, `url`, with `e=rr` | event | +| `ts_ms` | `ts` | event | +| `start` / `end` / `abandon` | `e=ps`, `sta=s` / `e` / `q` | event | +| `present` with `first: true` | `e=ps`, `sta=p` | event | +| `stall-start` / `stall-end` | `e=ps`, `sta=r` / `p` | event | +| `metrics.msd_ms`, `metrics.dfa` | `msd`, `dfa` | any report | +| `metrics.bsa`, `metrics.bsda_ms` | `bsa`, `bsda`, each an inner list of one Integer | any report | +| `switch`, `capable`, `layer-change`, `frame-drop` | `e=ce`, `cen` = `"wai-switch"`, `"wai-capable"`, `"wai-layer-change"`, `"wai-frame-drop"` | event | + +§3.2 An emitter **MUST NOT** put counted bytes, a layer or a capability into `br`, `tb`, +`pb` or any other bitrate key. + +§3.3 A report is an RFC 9651 Dictionary: Integers for counts and times, Tokens for +`st`, `sf`, `ot`, `e` and `sta`, Strings for `sid`, `cid`, `url` and `cen`. Its keys are +written in bytewise order and without optional whitespace, and a key is written only +when its value is not `null`. The corpus fixes the output byte for byte. + +## 4. Custom keys + +§4.1 Custom keys have the form `science.transaction.wai-`: a lowercase first letter, +then lowercase letters, digits, `.` and `-`, with an inner hyphen (CTA-5004-B's +custom-key rule). Every custom key's value is a String, the type CTA-5004-B admits for +custom keys besides a Token; an integer member is written as its decimal digits. + +| key | from | report | header shard (Request mode) | +|---|---|---|---| +| `science.transaction.wai-b` | `object.counted` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-tx` | `object.transferred` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-ch` | `object.content_hash` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-cap` | `object.capability` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-ri` | `object.rendition` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-l` | `object.layer` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-lt` | `object.layers_total` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-p` | `object.path`, or `switch.to`: `"p"` or `"f"` | request, `e=rr`, `wai-switch` | `CMCD-Object` | +| `science.transaction.wai-rcpt` | `object.receipt` | request, `e=rr` | `CMCD-Object` | +| `science.transaction.wai-lc` | `object.layer_cap`, or `layer_change.to` | request, `wai-layer-change` | `CMCD-Request` | +| `science.transaction.wai-lcf` | `layer_change.from` | `wai-layer-change` | — | +| `science.transaction.wai-coh` | `cohort` | every report | `CMCD-Session` | +| `science.transaction.wai-u` | `present.unit`, or `switch.unit` | `sta=p` (first presentation), `wai-switch` | — | +| `science.transaction.wai-pl`, `-late`, `-dec` | `present.layers`, `present.late_ms`, `present.decode_us` | `sta=p` (first presentation) | — | +| `science.transaction.wai-sw` | `switch.reason` | `wai-switch` | — | +| `science.transaction.wai-co`, `-cr`, `-cf` | `capable.outcome`, `capable.reason`, `capable.classical_floor` | `wai-capable` | — | +| `science.transaction.wai-clm` | `claim` | `sta=e` | — | + +§4.2 In Request mode the report also goes in header shards: each key to the shard +CTA-5004-B assigns it (`v`, `sid`, `cid`, `st`, `sf` and `msd` to `CMCD-Session`; `sn`, +`dl` and `dfa` to `CMCD-Request`; `ot` and `d` to `CMCD-Object`; `bsa` and `bsda` to +`CMCD-Status`), and each custom key to the shard of §4.1. Each shard is a Dictionary +written as §3.3 writes a report; a shard with no key is not sent. + +## 5. Switch reasons + +§5.1 A switch reason is one of: + +- SPEC §3.1's pin refusals, in the order of its checks: `pin-malformed`, + `form-not-registered`, `form-unreadable`, `form-required`, `prior-revoked`, + `prior-not-authorised` (check 5, at a sink that holds a key log), `prior-not-held`, + `digest-superseded`, `set-malformed`, `part-mismatch`, `set-incomplete`; +- `missing-capability`; +- `module-unverified`, `module-revoked`, `trap`, `fuel-exhausted`, `memory-exhausted`, + `deadline-threshold`, `unit-hash-mismatch`, `budget-exhausted`; +- `recovered`. + +A later extension adds its reasons here. + +## 6. CMSD mapping + +§6.1 An origin or edge **MAY** describe a WAI object in `CMSD-Static` with `at`, `d`, +`ht`, `nor`, `nrr`, `ot`, `sf`, `st` and `su`, and the custom keys +`science.transaction.wai-ch` (its content hash), `-l` and `-lt` (its layer and its +stage set's stage count) and `-rcpt` (a delivery receipt's link hash), each a String. +`v` is not written for version 1. + +§6.2 A deriving node adds `science.transaction.wai-op`, its operation token, as a Token. + +§6.3 `CMSD-Dynamic` **MAY** state, for each node, `etp`, `rtt`, `mb`, `rd` and `du`, and +`science.transaction.wai-drop` (a String): the number of objects the node discarded on +delivery timeout since its previous response to the same session. Each node adds its +own entry, in order; within an entry the parameters are in bytewise order. + +§6.4 Keys and parameters are written as §3.3 writes a report: in bytewise order, a key +only when its value is not `null`, and a Boolean (`su`, `du`) only when true. + +§6.5 CMSD carries no joule figure. An edge's energy travels in a claim. + +## 7. Session metrics + +§7.1 A **log** is one session's observations in `sn` order: one `sid`, `sn` strictly +increasing, timestamps never decreasing, and nothing after `end` or `abandon`. + +§7.2 A log gives, in integer milliseconds and bytes: + +- `started`: the log has a `start`; +- `vst_ms`, the video start time: the first presentation's `ts_ms` less the `start`'s, + `null` without both; +- `exit_before_start`: an `abandon` before any first presentation; +- `start_failed`: a session that started, never presented, and ended with `end`; +- `stalls` and `stall_ms`: the stalls begun after the first presentation, each from its + `stall-start` to its `stall-end`, or to `end` or `abandon` where it is still open, + and their total duration; +- `engaged_ms`: from the first presentation to `end` or `abandon`; +- `units`: the distinct units presented; +- `counted`: the counted bytes of every `response`; +- `claim`: the `end` observation's `claim`. + +§7.3 These are statements of the emitter, as the observations are (§1.1). + +## 8. Privacy + +§8.1 `sid` is fresh per session. An emitter **MUST NOT** add a persistent identifier, or +send telemetry to a party other than the content's own service, without the person's +agreement. + +§8.2 Neither a URL's query nor a signed URL's token enters an observation. A reader +refuses an `object.url` that holds a `?`, a `#`, a `@` or a `\`: a query, a fragment, +user information, or a separator some URL parsers read as `/` and so a way to user +information past a check for it. It refuses rather than strips them, and a path that +holds a `@` is refused with the rest. An emitter whose URLs carry a token in their +path **MUST** report `url` as `null`. `cid`'s syntax (§2.1) admits a content hash or +a short identifier, and no syntax keeps an address, a token or personal data out of an +identifier: an emitter **MUST NOT** put an address, a query, a token or personal data +in `cid`, and reports the content hash where it has no identifier free of them. + +§8.3 A Request-mode report travels with its request, to the host that serves the object +and to no other party. Event-mode reports go to the content's own service (§8.1). + +§8.4 A published aggregate covers at least 50 sessions per cell. A cell with fewer is +suppressed, not rounded. + +## 9. Conformance + +§9.1 [`telemetry-conformance/`](../telemetry-conformance/) holds: + +- observations of every event kind, with their expected CMCD reports and header shards + (`expected/cmcd.json`); +- observations every reader refuses (`invalid/`); +- CMSD-Static and CMSD-Dynamic descriptions with their expected fields + (`expected/cmsd.json`); +- session logs with their expected metrics (`expected/qoe.json`). + +§9.2 `wai_telemetry verify` requires every expected value byte for byte, and every +`invalid/` observation refused. + +§9.3 `cml/differential.mjs` builds each report from this extension's tables, encodes it +with an independent open-source CMCD and CMSD library at the versions +`cml/package-lock.json` locks, and requires the expected bytes; reads the +expected strings back as a dictionary, as a query string and as header shards, and +requires the report's members; and validates every report, and the Event-mode reports +together, with no error. diff --git a/wai/extensions/moq-streaming-format.md b/wai/extensions/moq-streaming-format.md index 8f1370591..6a96faf9f 100644 --- a/wai/extensions/moq-streaming-format.md +++ b/wai/extensions/moq-streaming-format.md @@ -145,6 +145,9 @@ commitment: claim, a relay that rewrites this property included. - A relay **MUST** preserve this property unchanged, per `jwp-receipts` §4.1. A relay that recomputes it is asserting something it did not measure. +- Telemetry about objects on a WAI track is + [delivery-telemetry](delivery-telemetry.md)'s. It is unsigned and advisory, and it + **MUST NOT** restate this property's figure, class or evidence. - Absence of the property is not evidence of absence of a receipt; receipts MAY ride elsewhere (§1 of `jwp-receipts` lists sidecar and trailer carriage). A sink **MUST NOT** infer that an object without the property is unattested. diff --git a/wai/extensions/staged-measurement.md b/wai/extensions/staged-measurement.md new file mode 100644 index 000000000..5f62e4ca9 --- /dev/null +++ b/wai/extensions/staged-measurement.md @@ -0,0 +1,671 @@ +# WAI Extension: Staged-Delivery Measurement + +> Status: Draft. How a figure about staged delivery +> ([staged-delivery](staged-delivery.md)) is measured and published: which bytes are +> counted, how quality is computed per prefix, what "first presentable" means, how +> layering overhead is stated, how decode time and energy are reported, how +> deduplication is computed, how delivery is simulated, and what a record carries. +> Reference impl: `wai-rs` — `measure_framing`, `quality`, `output_tree`, +> `content_dedup`, `bench_env`, `stage_measure::{title, decode, run, sim}`; the +> `wai_stage_measure` tool; the corpus +> [`staged-measure-conformance/`](../staged-measure-conformance/). Keywords **MUST**, +> **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174. + +## 1. Scope + +§1.1 This extension binds any party that publishes a figure about WAI staged or split +delivery: bytes, time, quality, overhead, decode time, energy or deduplication. It +binds the reference tools. It does not bind a sink's decode behaviour, which +[staged-delivery](staged-delivery.md) states. + +§1.2 A figure that does not meet this extension **MUST NOT** be presented as a WAI +measurement. + +§1.3 This extension states no price, and a record **MUST NOT** carry one. A deployer +that derives a cost from transferred bytes does so outside WAI. + +## 2. Terms + +| term | meaning | +|---|---| +| **title** | What a session plays: one or more bases and the layers that refine them ([staged-delivery](staged-delivery.md) §1). | +| **unit** | A presentable output, as staged-delivery §2 fixes it per capability: a frame, a picture, an audio clip, an attribute tensor. A title numbers its units globally: a base's unit `i` is the title's unit `first_unit + i`. | +| **stage** | A unit's stage: 0 after its base decodes, `k` after a layer whose `refines.layer` is `k` applies to it. | +| **prefix** | The stages `0…k` of a title: every object whose layer is at most `k`. Its **layers** count is `k + 1`. | +| **presentation** | A (unit, layers) pair a sink handed to output, with the output it presented. | +| **counted bytes** | Bytes under §3. Apart from transferred bytes, the only byte figure this extension reports. | +| **transferred bytes** | Bytes of WAI objects as a sink received them, whole or as byte ranges, without transport or receipts. | + +A stage's position is its layer (0 for the base). A prototype MoQ binding in this +repository carries it as `mrl_layer`, a stage set's stage count as `mrl_total_layers`, +and a subscriber's upper bound on the layers it accepts as `max_mrl_layer`; this +extension names that bound `layer_cap`. Staged-delivery §13 says which of them a sink +reads. + +## 3. Byte accounting + +§3.1 **Counted bytes of an object.** For a `WAI1` envelope, `12 + man_len + P`; for a +`WAI2` envelope, `8 + man_len + 2 + 8·n + Σ Pᵢ`. Here `man_len` is the manifest's +length as written, `n` is the component count, and `P` or `Pᵢ` is each payload's +length (SPEC §2, §7.1). These are the envelope's bytes, exactly: an object with bytes +after its last payload is not the envelope its lengths describe, and is not counted. +Transport bytes (HTTP, QUIC, MoQ object headers, TLS) are not counted bytes; a figure +that includes them **MUST** state them separately. + +§3.2 **Only a checked format is counted.** An object is countable only where WAI checks +the format of every payload it carries: a capability SPEC §5 pins to a binary format +(`WIV1`, `WIH1`, `WIA1`, `WIS1`, and `WIR1` for a refinement-only capability; Appendix +A) whose payload parses as that format, by the parse a sink runs. Where a `WAI1` +manifest names a fallback with a pinned format, the payload must parse as the +fallback's format too, since the fallback decodes the same payload (SPEC §4 step 5). +Otherwise the object is **not countable**, with a code: + +- `no-format-check`: a capability WAI checks no format for, a classical codestream or + a string WAI does not register among them. Nothing in WAI tells such a payload's bytes + from a text encoding of them (a JSON text, base64, JSON arrays of byte values), which + **MUST NOT** be counted, as stored or as an estimate of what it would be in binary; +- the format parse's code (`bad_magic`, `format_capability_mismatch`, …: + [integer-payloads](integer-payloads.md) §8, staged-delivery A.6), for a payload that + is not its capability's or its fallback's format. A JSON form of a pinned format, its + base64 text, a JSON string, or the format's bytes behind a byte-order mark all fail + here; +- `sha256-mismatch`: a `WAI2` component whose `sha256` is not its payload's (SPEC §7.1); +- `encrypted`: an encrypted payload, whose plaintext's format cannot be checked. + +No byte figure is published for an object that is not countable. The JSON that the +integer codecs' encode and export tools write is a converter input +([integer-payloads](integer-payloads.md) §10); its counted bytes are those of the +envelope that carries the converted payload. + +§3.3 **Framing label.** A byte figure carries a framing label. A figure is +`registered` when every payload it counts is in its capability's registered format, +which §3.2 requires of every figure this revision defines. A later revision that +counts a payload under another framing names its label. Two figures with different +framing labels **MUST NOT** be compared, summed or placed on one curve. + +§3.4 A record **MUST** state: + +- the counted bytes of each object, split into envelope header, manifest and payload; +- the sums per (base, stage) and per (base, prefix), and over the title per stage and + per prefix; +- the framing label; +- the bytes of each base's parameter set, separately (staged-delivery §11). + +§3.5 Receipt and claim bytes are counted separately, as receipt bytes. + +## 4. Quality per prefix + +§4.1 Quality is stated per presentation and per prefix against a **reference**: the +source, where the title pins it by SHA-256, and the final prefix's output, labelled +"against the final prefix". A figure against the final prefix **MUST NOT** be +presented as distortion against the source. A unit whose shape at a stage differs from +the reference's (below an up2 or a rate layer) has no figure at that stage; a figure +pooled over a prefix covers the units that have one, and says how many. + +§4.2 **Pictures and frames** (`rgb8`). For two RGB8 buffers of width `W ≥ 1` and +height `H ≥ 1`: + +- `sse_rgb` = Σ (aᵢ − bᵢ)² over the `3·W·H` bytes; +- `sse_y` is the same sum over the luma planes, with + `Y = (77·R + 150·G + 29·B + 128) >> 8`; +- `ssim_y_q32` is the mean structural similarity of the luma planes in Q32 fixed + point (`2³²` is 1). + +The similarity's windows measure `ww = min(8, W)` by `wh = min(8, H)` pixels. Their +left edges are at every multiple of 4 up to `W − ww`, and at `W − ww` itself when it is +not a multiple of 4; their top edges are at every multiple of 4 up to `H − wh`, and at +`H − wh` itself when it is not one. A window is at every pair of a left and a top edge, +so every pixel is in at least one window. For a window of `n = ww·wh` pixels, with `Sx = Σx`, +`Sy = Σy`, `Sxx = Σx²`, `Syy = Σy²` and `Sxy = Σxy` over the reference's luma `x` and +the test's luma `y`: + +``` +A1 = 20000·Sx·Sy + 65025·n² +A2 = 20000·(n·Sxy − Sx·Sy) + 585225·n² +B1 = 10000·(Sx² + Sy²) + 65025·n² +B2 = 10000·((n·Sxx − Sx²) + (n·Syy − Sy²)) + 585225·n² +s_w = ⌊ A1·A2·2³² / (B1·B2) ⌋ (floor toward −∞) +ssim_y_q32 = ⌊ (Σ_w s_w) / (window count) ⌋ (floor toward −∞) +``` + +This is the structural similarity of Wang et al. (2004) with K1 = 0.01, K2 = 0.03, a +peak of 255, uniform windows at stride 4 with an edge-aligned last window on each +axis, and population statistics, multiplied out so +that no floating-point operation occurs. Because `n ≤ 64`, `|A1·A2·2³²| < 2¹¹⁵` and +`B1·B2 > 0` (the reference implementation's `quality` module proves both), so the +computation fits signed 128-bit integers. + +§4.3 **Audio** (`pcm16`). For two sequences of `i16` samples, channel-interleaved and +in output order: `sse` = Σ (aᵢ − bᵢ)² and `ref_energy` = Σ aᵢ² over the reference. +Both are unsigned integers of at least 96 bits, written in records as decimal +strings. Sequences of different lengths are refused. + +§4.4 **Tensors** (`tensor-i64`), for splat attributes, volumetric attributes and +haptic samples: `sse` over the capability's canonical integer output, written as a +decimal string, and `max_abs` = max |aᵢ − bᵢ|. + +§4.5 **Pooling.** A figure over several units (a clip, or a prefix) sums each sum over +the units; for pictures, `ssim_y_q32` is the mean of every window of every unit, one +division over the pooled `Σ s_w` and window count, and `max_abs` is the largest. + +§4.6 **Exact outputs** (`state-exact`), for worlds, feeds, scores, avatars, quantum +and every capability whose output is a state: a prefix is `equal` or `differs`, by +SPEC §7 *Digests*. It is never stated as a distance. + +§4.7 PSNR = `10·log10(255²·N / sse)` and SNR = `10·log10(ref_energy / sse)` are derived +for display only. A record **MUST** carry the integers. An `sse` of 0 has no finite +PSNR, and is shown as exact. + +§4.8 A record **MAY** add another full-reference metric. It names the metric's +implementation and version, and the metric is informative only. + +## 5. First presentable + +§5.1 `bfpf_min` is the sum of the counted bytes of the smallest set of objects needed +to present unit 0 at one layer: the base that holds it, and every index or manifest +object a sink reads first. It is computed from the title alone. A base's parameter +set is stated beside it (§3.4), not in it. + +§5.2 `bfpf_sched` is the counted bytes that had arrived when unit 0 was first +presented, under a stated delivery schedule (§10). It is at least `bfpf_min`. + +§5.3 `tfpf` is the time from the session's first request to the first presentation of +unit 0, in microseconds. Under §10 it is virtual and deterministic. Measured, it is a distribution over repeated runs, with the +clock and the presentation event named; in a browser that event is the frame callback +after the canvas write. + +§5.4 `tffs_k` is the time to the first presentation of unit 0 at `k + 1` layers, stated +as §5.3 states `tfpf`. + +## 6. Layering overhead + +§6.1 A figure that states a saving from staging **MUST** be accompanied by its layering +overhead. + +§6.2 The **single-layer reference** is the same content coded by the same staged form +as one layer at the final layer's operating point: the title's bases and, over each, +one layer straight from stage 0 to the source, with the final layer's step and the +predictor that reaches the final shape. A reference at further operating points is +the same with another step. + +§6.3 The overhead is two rate–quality points: (counted bytes of the full title, quality +of its final prefix against the source) and (counted bytes of the reference, its +quality against the source). A record **MUST NOT** reduce them to a ratio, and +**MUST NOT** say "at equal quality" unless the two quality integers are equal. It +**SHOULD** add the reference at further operating points as a curve, and state which +two reference points bracket the quality of the final prefix. + +## 7. Decode time + +§7.1 Decode time is measured per (unit, stage) on a stated machine and build. A record +states the runs; the minimum, median, 10th and 90th percentiles; the order in which +cases ran; and the discarded warm-up runs. + +§7.2 Cases **MUST** be interleaved, with the order rotated each round. + +§7.3 A run whose output digest differs from the reference is not a timing; the bench +aborts. + +§7.4 A record states the machine (its processor, cores, operating system and +architecture, and the build profile), its load before and after each block, its power +source, and whether it was otherwise idle. Continuous integration **MUST NOT** assert +any timing. + +## 8. Energy + +§8.1 An energy figure **MUST** follow [energy-measurement](energy-measurement.md): +an acquisition class (§2); for a figure obtained under its §3, the bracket a +`measurement` claim publishes (§6); the build (§5); and a recomputable work unit. For +staged decode, the work unit is the (unit, stage) decodes of one completion. + +§8.2 A figure the meter could not obtain is **unmetered**: it is written as `null` +with a reason, never as 0. + +§8.3 Only a figure whose attribution verdict (energy-measurement §4, §6.2) is clean is +published as a figure. A bracket usable for relative comparison **MAY** be shown, +labelled "relative comparison only, not a figure", and never as an absolute joule +figure. Noisy and unusable brackets are kept in the record and not shown. + +§8.4 A figure covers decode on the measured machine's processor rails. A record **MUST +NOT** state, imply or chart energy for displays, radios, networks or other devices. A +reading from a whole-system rail that includes a display is not published. + +§8.5 A rail that is not live in a bracket (energy-measurement §3.1) gives no figure; the +decode is unmetered. + +## 9. Deduplication + +§9.1 Over a set of objects: + +- `objects` is their count, a repeated object counted each time; +- `unique` is the count of distinct content hashes ([jwp-receipts](jwp-receipts.md) + §2.1); +- `referenced_bytes` is the sum of their counted bytes; +- `stored_bytes` is the sum over one object per content hash. + +All four are recomputed, never declared. One content hash has one length; a set that +gives one hash two lengths has no totals. + +§9.2 Deduplication is stated per title, over every object the title and its +single-layer references reference (a reference's bases counted again), and, where +content repeats across titles, across the record. + +## 10. Delivery simulation + +§10.1 A deterministic delivery figure comes from the **staged delivery simulator**, a +discrete-event model in integer microseconds defined by its model and plan +(Appendix B). Its figures are labelled "simulated" with the plan. A simulated figure +is never a measurement of a network, and is never presented as one. + +§10.2 **Delivery timeout.** A plan gives each object a carriage, after the +`DELIVERY TIMEOUT` of MoQ Transport (draft-ietf-moq-transport): + +- `datagram`: each object is one datagram, and one not started by its deadline (its + availability plus its timeout) is **dropped**. +- `subgroup`: each (base, layer) is one subgroup, whose objects are sent in order. When + an object of a subgroup is not fully sent by its deadline, the subgroup is reset: the + bytes already sent of it and of every later object of the subgroup are **abandoned**, + and the bytes not yet sent are **dropped**, whatever the later objects' own + deadlines. + +Dropped and abandoned bytes are never delivered, and every offered byte is delivered, +dropped or abandoned. + +§10.3 **Layer cap.** An object whose layer exceeds `layer_cap` is never sent and never +counted as offered. + +§10.4 **A refinement never stalls its base beyond a bound.** A plan's *base-only* +model removes every refinement. The **stall** of a unit is its base presentation time +under the plan minus the same time under the base-only model. The stall bound is +`S = S_link + S_decode`: + +- `S_link`: under `strict-priority`, the serialisation time of one chunk; under `fifo` + with a refinement timeout `T`, `T` plus one chunk; otherwise there is no bound. +- `S_decode`: under `base-first` decode, one refinement's decode time; otherwise there + is no bound. + +**Preconditions.** The bound holds for every model and every plan with no base +timeout (`delivery_timeout_us.base` is `null`), a bounded link schedule (`strict-priority`, +or `fifo` with a refinement timeout) and `base-first` decode, whatever else the plan +states (carriage, window, live captures, layer cap, workers, decode times, playout +deadlines). A plan outside them has no bound: a base its refinements delay past its own +timeout is not presented at all, and the two unbounded parts above have counter-plans. + +**Proof.** Write `c` for one chunk's serialisation time, `⌈chunk_bytes·1000 / +bytes_per_ms⌉`, the longest any chunk takes; `L = c` under `strict-priority` and +`L = T + c` under `fifo`; and `d_r` for one refinement's decode time. A *base* below is +one layer-0 object (a base split into several layer-0 objects contributes each). +Number the bases `b_1, b_2, …` in the order the sender sends them. That order is +greedy under subgroup precedence: whenever the sender starts a base, it takes, of the +bases available then and not held behind an unfinished earlier object of their +subgroup (§10.2; under `subgroup`, a base's layer-0 objects form one subgroup, sent in +title order), the least by the order key (Appendix B, *Order*), whose first part is the +availability under either schedule. It is the same order under the plan and under its +base-only model. By induction on `k`: when either starts its `k`-th base, the bases +finished are `b_1, …, b_{k−1}` in both, so the same bases are held. The base-only +model starts it at the first time after `b_{k−1}` ends at which a base is available; +the plan starts it no earlier, since `b_{k−1}` ends no earlier under the plan and it +too needs a base available. A base that becomes available after the base-only model's +start has a later availability, and so a larger key, than every base eligible then, so +the least is `b_k` in both. The order can differ from the order of the keys alone: +a base listed after another of its subgroup waits for it, even when it is available +earlier. A base's service time `s_k` (the sum of its chunks' times) is the same in +both. + +*The link.* Let `a_k` be `b_k`'s availability, `start_k`, `end_k` its first chunk's start +and last chunk's end under the plan, and `start⁰_k`, `end⁰_k` under the base-only model. + +1. A base, once started, is sent to its end without a chunk of another object between. + At each boundary while `b_k` is sent, the eligible objects are those eligible when + it started, less any dropped or reset, and those that became available since, whose + availability, and so whose key under either schedule, is larger than `b_k`'s. None + becomes eligible by its subgroup in between: an earlier object of a subgroup is + finished only by being sent, which does not happen while `b_k` is sent, and a reset + ends the whole tail. `b_k` was the least eligible object when it started (under + `strict-priority` every base precedes every refinement; under `fifo` it was chosen), + so it stays the least. So `end_k = start_k + s_k`. +2. In the base-only model, `start⁰_k = max(a_k, end⁰_{k−1})`: the bases are sent in + order, and the link is idle only when no base is available. +3. Under the plan, at `t_k = max(a_k, end_{k−1})` base `b_k` is eligible (the bases + its subgroup holds it behind are among `b_1, …, b_{k−1}`) and, by the order above, + the eligible base of least key. Under `strict-priority`, a base precedes every refinement, so `b_k` starts + at the first boundary at or after `t_k`; a chunk in flight at `t_k` is a refinement's + (step 1), and ends within `c`. So `start_k ≤ t_k + c`. Under `fifo`, objects that + precede `b_k` are bases (all before it in the order) and refinements available no + later than `a_k`, whose deadlines are therefore at most `a_k + T`. At the first + boundary after `a_k + T`, which comes within `c` of it, every such refinement not + yet finished has been dropped or reset (§10.2); one already finished took no + longer. So `start_k ≤ max(end_{k−1}, a_k + T + c)`, and in both cases + `start_k ≤ max(a_k + L, end_{k−1})`, with `end_{k−1}` exact when it is the larger: + a base that ends at a boundary where `b_k` is eligible is followed by `b_k`. +4. By induction on `k`, `start_k ≤ start⁰_k + L`: with step 3, step 1 and the + hypothesis, `start_k ≤ max(a_k + L, end⁰_{k−1} + L) = start⁰_k + L`. Each base + therefore arrives at most `L` later than in the base-only model, and the bases + arrive in the same order in both. + +*The decode.* Number the base decode jobs in arrival order, which is the order of step +4. Under `base-first` the sink takes bases in the order they became ready (Appendix B, +*Workers*), which is that order, and before any refinement. Let `r_k ≤ r⁰_k + L` be +`b_k`'s arrival under the plan and the base-only model, `W` the workers and `d_b` a +base's decode time. + +5. In the base-only model, with equal decode times and jobs started in order, + `u⁰_k = max(r⁰_k, u⁰_{k−1}, u⁰_{k−W} + d_b)` is `b_k`'s start (terms with an index + below 1 omitted): at that time `b_{k−1}` has started, every base up to `b_{k−W}` has + finished, so at most `W − 1` workers are busy. +6. Under the plan, no refinement starts while a base is ready and waiting. A + refinement that holds a worker after `r_k + d_r` therefore started after `r_k` and + so after `b_k` started. At `τ = max(r_k + d_r, u_{k−1}, u_{k−W} + d_b)`, if `b_k` has + not started, it is the waiting base of least order, no refinement holds a worker, + and at most `W − 1` bases do; so `u_k ≤ τ`. +7. By induction, `u_k ≤ u⁰_k + L + d_r`: each term of `τ` is at most the matching term + of step 5 plus `L + d_r`. + +So every base is presented at most `S_link + S_decode = L + d_r` after its time in the +base-only model, and a late base is presented all the same (Appendix B, +*Deadlines*). ∎ + +The reference implementation checks the bound on generated models and plans drawn +over the whole range of the preconditions (groups and captures in any order, objects +listed in any order, slow links, zero and short timeouts, caps, one to four workers), +and on draws aimed at the orders the bound depends on (captures on and beside chunk +boundaries and out of order, bases split into two layer-0 objects, chunks of one byte), +which fail on their own when the send or the decode order of bases is not the one +above; the corpus checks it on every bounded plan it holds. For each part of the bound a +plan can lose, the corpus holds a counter-plan whose stall passes the bound of the +same plan with that part restored: a `fifo` link without a refinement timeout against +the same `fifo` link with one, and `fifo` decode against the same plan with +`base-first` decode. + +## 11. Records + +§11.1 A published figure **MUST** cite a dated record in +`wai/research/staged-measurements/` by path and commit. The record carries: + +- the date and the commit; +- the title's content identity (Appendix D) and the framing label; +- the machine and build; +- every deterministic figure of §3–§6 and §9, and every measured figure of §5.3–§5.4, + §7 and §8 with its method; +- the link hashes of its claims. + +§11.2 Records are immutable. A correction is a new record that names the one it +corrects. + +§11.3 A classical baseline is a reference curve of points (counted bytes, quality per +§4) from a named encoder at stated versions and settings, on the same source. A +record **MUST NOT** state a multiplier, percentage or ratio between a WAI figure and a +baseline. + +## 12. What continuous integration asserts + +§12.1 Only values that are the same on every machine: + +- counted bytes, `bfpf_min` and simulator outputs; +- the integers of §4, output digests and the output tree's root and leaf count; +- deduplication totals and the overhead points; +- every figure a conformance corpus states. + +§12.2 Measured times and energy are stored as artifacts and never asserted. + +§12.3 The corpus [`staged-measure-conformance/`](../staged-measure-conformance/) holds +titles (Appendix D) and, for each, every figure of §12.1 (`goldens.json`) and the +fixture tool's own reconstruction digests (`expect.json`). A conforming implementation +reproduces every figure and every digest. The corpus's README names its second, +independent implementation. + +## 13. Media-class matrix + +The last column follows §3.2: an object is countable only under a capability whose +format WAI checks (`count_object` gives `no-format-check` for any other), so a class +whose staged form or base has no checked format is not countable in this revision, +whatever else it supports. + +| class | unit | staged form | edge operation | quality kind | in this revision | +|---|---|---|---|---|---| +| image | a picture | pinned integer base (`wai.neural.int_hyper`) + `WIR1` layers (`wai.image.int_refine`: identity, up2) | serve a prefix of layers | `rgb8` | measured (corpus); a classical base (PNG and the like) is not countable in this revision | +| video, WAI-native | a frame | `wai.video.int_motion` or `wai.video.int_hyper` base + `WIR1` layers, per frame or per span (`wai.video.int_refine`) | drop layers above a cap | `rgb8` per frame, pooled per clip | measured (corpus) | +| audio | the clip | `wai.audio.int_codec` base + `WIR1` layers (`wai.audio.int_refine`: rate, identity) | drop layers above a cap | `pcm16` | measured (corpus); a `wai.audio.flac` base is not countable in this revision | +| splat | the attribute tensor | `wai.splat.int_codec` base + `WIR1` layers (`wai.splat.int_refine`) | drop layers above a cap | `tensor-i64` | measured (corpus) | +| video, enhancement layer | a frame | registered base + `wai.video.lcevc` companion (SPEC §7.1) | drop the companion | `rgb8` after the sink's decode | not countable in this revision (the companion's format is not checked); quality needs a sink-supplied decoder | +| image, progressive passes | a picture | byte prefixes of one progressive codestream | serve a byte prefix | `rgb8` after the sink's decode | a staged form not yet registered (staged-delivery §6.2); not countable in this revision | +| audio, codebook prefix | the clip | the further codebooks of a residual-vector-quantised code | drop codebooks above a cap | `pcm16` | a staged form not yet registered; not countable in this revision | +| volumetric (4D splat) | a keyframe span | keyframe refinement | serve a prefix | `tensor-i64` | a staged form not yet registered; not countable in this revision | +| world / feed / film / avatar | a segment | snapshot and operation deltas | snapshot and tail | `state-exact` | a staged form not yet registered; not countable in this revision | +| score | a section | none | none | `state-exact` (mixdown digest) | not countable in this revision | +| haptics | an envelope | keyframe refinement | serve a prefix | `tensor-i64` | a staged form not yet registered; not countable in this revision | +| quantum | a circuit | none | none | `state-exact` | not countable in this revision | +| text and records | an object | none | none | none | not countable in this revision | + +## Appendix A. Counted formats + +The formats §3.2 counts, and how the reference implementation +(`measure_framing::count_object`) checks each before it counts it. A payload that fails +its check is not countable, with the check's code. + +| capabilities | format | check | +|---|---|---| +| `wai.video.int_motion` (keyframe 0), `wai.video.int_hyper` (keyframe 1) | `WIV1` | [integer-payloads](integer-payloads.md) §2–§3, in its order | +| `wai.neural.int_hyper` | `WIH1` | integer-payloads §2, §4 | +| `wai.audio.int_codec` | `WIA1` | integer-payloads §2, §5 | +| `wai.splat.int_codec` | `WIS1` | integer-payloads §2, §6 | +| `wai.image.int_refine`, `wai.video.int_refine`, `wai.audio.int_refine`, `wai.splat.int_refine` | `WIR1` | staged-delivery Appendix A.1–A.3, for the capability's unit kind | +| every other capability | — | none: not countable (`no-format-check`) | + +A `WAI1` fallback with a pinned format is checked as its capability is. Parameter sets +(SPEC §3.1) are not objects and are never counted bytes; §3.4 states them separately. +The corpus's `counting.json` holds an object for each outcome. + +## Appendix B. Delivery simulator `wai-sim/1` + +A **model** is a title's objects in title order, the object index, as delivery sees +them (`wai-sim-model/1`): + +```json +{ "format": "wai-sim-model/1", "unit_us": 40000, + "objects": [ { "id": "base.wai", "base": 0, "layer": 0, "units": [0, 5], "bytes": 2519, "capture_us": null }, … ] } +``` + +`units` is `[from, count]` over the title's units and `bytes` the object's counted +bytes (§3). A model is well formed when every object has at least one byte and one +unit, no two objects cover one unit at one layer, and every unit an object of layer +`k ≥ 1` covers is covered at layer `k − 1` by an object of the same base. A measured +title's model is its objects with their counted bytes, and its bases' capture times. + +A **plan** (every member required, none other allowed): + +```json +{ "plan": "wai-sim/1", "mode": "vod" | "live", + "link": { "rtt_us": 40000, "bytes_per_ms": 2500, "chunk_bytes": 1200 }, + "sender": { "schedule": "strict-priority" | "fifo", "carriage": "datagram" | "subgroup", + "layer_cap": null, "delivery_timeout_us": { "base": null, "refinement": 30000 }, + "vod_window_us": null }, + "sink": { "workers": 1, "decode_order": "base-first" | "fifo", + "decode_us": { "base": 4000, "refinement": 2000 }, + "playout_start_us": null, "late_refinement": "discard" | "present" } } +``` + +`bytes_per_ms`, `chunk_bytes`, `workers` and both decode times are at least 1. Every +integer of a model or plan is at most 2⁵³ − 1. All arithmetic is in integer +microseconds, and `h = ⌊rtt_us / 2⌋`; a time or a byte sum past 64 bits is refused +(`overflow`), never wrapped. + +**The sender.** + +- *Availability.* The sink requests at 0 and the sender holds the request at `h`. An + object is available at `h + max(0, from·unit_us − vod_window_us)` on demand (`h` + with no window), and at `max(h, capture_us)` live; a live plan over an object with no + capture time is refused. +- *Deadline.* An object's availability plus its layer's timeout (`base` for layer 0, + `refinement` otherwise); none when that timeout is `null`. +- *Cap.* An object above `layer_cap` is capped: never sent, never offered. +- *Datagrams.* Under `datagram`, an object of more than `chunk_bytes` that is not capped + is refused (`datagram-too-large`). +- *Link.* The link carries one chunk at a time. At each **boundary** — time 0, and the + end of each chunk — the sender first applies the timeouts, then sends the next chunk: + - *Timeouts.* In object order, each object still pending whose deadline is before the + boundary's time: under `datagram` it is dropped; under `subgroup` its subgroup is + reset from it (§10.2). + - *Eligibility.* An object is eligible when it is pending, available, and, under + `subgroup`, no earlier object of its subgroup is pending. + - *Order.* Of the eligible objects, the least by (layer, then for a base its + availability, `from`, object index) under `strict-priority`, and by + (availability, object index) under `fifo`: under either, bases go in the order + they became available. + - *Chunk.* `c = min(chunk_bytes, bytes not yet sent)`, occupying the link + `⌈c·1000 / bytes_per_ms⌉` µs from the boundary, and arriving `h` after it ends. An + object's `sent_first_us` is its first chunk's start. An object whose last chunk + ends at `e` is delivered, arriving at `e + h`, under `datagram`, with no deadline, + or when `e` is at most its deadline; otherwise it stays pending and the next + boundary resets its subgroup. + - With nothing eligible, the time moves to the next availability of a pending + object, which is a boundary too; the run ends when nothing is pending. + +**The sink.** + +- *Ready.* A delivered object is ready when it has arrived and, for layer `k ≥ 1`, + every unit it covers is at stage `k − 1`; it became ready at the later of its arrival + and the time the last of those units reached that stage. +- *Workers.* The sink has `workers` non-preemptive workers. At each event time it first + completes the jobs ending then, then gives each free worker the least ready job: by + (layer, then for a base the time it became ready, `from`, object index) under + `base-first`, so bases are decoded in the order they arrived, and by (the time it + became ready, object index) under `fifo`. A job lasts `decode_us.base` or `decode_us.refinement`, + once per object whatever its units. The next event is the next job end or the next + time a job becomes ready. +- *Presentation.* When a job ends at `t`, every unit it covers is presented at + `layers = k + 1` at `t`, and reaches stage `k`. +- *Deadlines.* With `playout_start_us` set, unit `u`'s deadline is + `playout_start_us + u·unit_us`. Each unit a job presents after its deadline counts + one **missed**: a base is presented all the same; a refinement is presented under + `present`, and under `discard` is not presented and the unit stays at its stage. + +**Outputs.** + +- per object: `sent_first_us`, `arrived_us`, and its state: `delivered`, `dropped`, + `abandoned` or `capped`; +- the presentations `(unit, layers, t_us)`, ordered by unit, then layers; +- `bfpf_sched`, the bytes of the chunks arrived by unit 0's first presentation; + `tfpf_us`, that presentation's time; and `tffs_us`, unit 0's first presentation at + each layer count, `null` where none; +- `offered`, `delivered`, `dropped`, `abandoned` and `never_sent_capped`, in bytes, with + `offered = delivered + dropped + abandoned`; +- `missed`; +- each unit's stall (§10.4), `null` where the plan or the base-only model never presents + its base. + +A **case** of the corpus names a model, a plan, and its claim: `within-bound` (every +stall at most the plan's bound), `exceeds:` (the plan has no bound, and its +largest stall passes the named plan's), or `unbounded` (the plan has no bound, and only +its figures are checked). + +## Appendix C. Output tree + +A commitment to every presentation of a session or a title: one leaf per (unit, +layers), so that a record or a claim commits to its outputs by the pair (root, +`leaf_count`) and any one output is provable by an inclusion path. + +``` +out_digest = SHA-256(canonical output) SPEC §7 Digests: RGB8, s16le PCM, the attribute tensor, a 32-byte state hash +leaf = SHA-256("wai:out-leaf\x01" ‖ u64_be(unit) ‖ u16_be(layers) ‖ u16_be(decoder) ‖ out_digest) +node = SHA-256("wai:out-node\x01" ‖ left ‖ right) +``` + +- `decoder` is the index of the decoder entry that produced the output: entries are + (capability, pin) pairs in first-use order, presentations taken by (layers, unit). + For a title, the base's capability with the pin it verified, then each refinement + capability. +- Leaves are ordered by (unit, layers), and a (unit, layers) given twice is refused. +- The root over `n ≥ 1` leaves is RFC 6962's Merkle tree hash: one leaf is itself; + otherwise `node(MTH(D[0:k]), MTH(D[k:n]))`, with `k` the largest power of two below + `n`. No leaf is ever duplicated, so two leaf lists never give one root. +- A commitment is the root with its leaf count; a root alone is not one. A record or a + claim that commits to an output tree states both. +- An inclusion proof is RFC 6962's audit path, checked as RFC 9162 §2.1.3.2 checks one + under the leaf count the commitment states, never a count the proof's holder + supplies. An audit path does not fix the tree's size: leaf 0's path in a tree of 7 + leaves also walks to that root when 8 is taken as its size. +- `layers` is 16 bits, so it holds any stage count. + +The JWP group root ([jwp-receipts](jwp-receipts.md)) is a separate, fixed wire format +and is not this tree. + +## Appendix D. Measured titles + +A measured title is a fixture directory with a `title.json`: + +```json +{ "format": "wai-measure-title/1", "title": "still", "about": "…", + "unit_us": 40000, + "bases": [ { "case": "staged-conformance/cases/image_int_hyper_2l", "base": 0, + "first_unit": 0, "capture_us": null } ], + "layers": [ "L01.wai", "L02.wai", "L03.wai" ], + "source": { "file": "source.bin", "sha256": "<64 hex>" }, + "references": [ { "label": "one layer, q 1", "layers": [ "R0_00.wai" ] } ], + "expect": "expect.json" } +``` + +- The stage structure is the staged set's own. Each base is a base of a + [staged-conformance](../staged-conformance/) case, named by the case directory and + its index there: the case states its envelope, the parameter set it decodes against + (by member and path) and its stage-0 output. Each layer's `refines` names its base, + producers, layer and span (staged-delivery §3). `title.json` adds only what that + model does not carry. +- `unit_us` is one unit's duration, and `capture_us` a base's capture time for a live + delivery plan (`null` for one delivered on demand). +- `source` holds each unit's canonical buffer at its final shape, in unit order, with + its SHA-256. +- `references` are the single-layer references of §6.2: each is the title's bases with + the layers it lists. +- `expect.json` holds, from the fixture tool's own encoder, the SHA-256 of every + unit's output at every stage, and of every reference's final units. An evaluation + **MUST** reproduce them. +- A base **MAY** instead be named by its envelope alone, `{ "envelope": , "set": + { : }, "first_unit", "capture_us" }`: it has no stage-0 output or case + to check its decode against. +- `limits` (optional) lowers the limits the title is measured under: `cost`, four + figures (output bytes, symbols, working bytes, multiply-accumulates), and + `chain_budget`, each `null` for the tool's limit below. A title lowers a limit and + never raises one (below). +- `refused` (optional) names the code the title's evaluation **MUST** be refused with: + a title that exists to be refused has no figures. +- Every path is below `wai/`; a layer, source and expectation file is a file name in + the fixture directory. + +A title's **content identity** is BLAKE3 over `"wai:measure-title\x01"` and then, in +path order, each measured file's path and bytes, each prefixed by its length as a +64-bit big-endian integer. The measured files are its objects (bases, layers and the +references' layers), its bases' parameter-set files and its source; labels, +`title.json`, stage-0 outputs and `expect.json` are not part of it. + +**Evaluation.** Each base is decoded as a sink decodes one (SPEC §4: its capability, +then its fallback, a refinement-only or companion-only capability making the envelope +inert), against the parameter set its pin names (SPEC §3.1), and under the cost +limit, which its payload's declared cost (integer-payloads §7) is compared with +before any symbol is decoded or any output allocated. It must give the case's stage 0, +and the capability, fallback, tier and pin the case states. Its layers are applied as a +sink applies them (staged-delivery §4) under the chain budget, with the same +limit on each layer's declared cost (Appendix A.5), one layer index at a time and, +within one, in the order the title lists them; every layer **MUST** apply to every unit +it covers, as in a set delivered whole and in order. The figures of §3–§6 and §9 and +the output tree follow from the outputs at every stage. A refusal on the way names its +code: the base decode's (`cost_over_limit`, `missing_capability`, a pin refusal) or the +sink's (`chain-over-limit`, a layer's §4.2 reason). This composition of library +functions is for measurement, not a sink's dispatch path. + +**The tools' limits.** The reference tools (`wai stage`, `wai_stage_measure`, +`wai_stage_bench`) decode every base under these cost figures, hold every layer to +their output, symbols and working bytes (staged-delivery Appendix A.5), and hold every +chain to the chain budget. A tool's `--limit OUT,SYMBOLS,WORKING,MACS` replaces the four +cost figures. A title is measured under the lower, figure by figure, of its own +`limits` and the tool's: a title lowers a limit and **MUST NOT** raise one, because +whoever writes a title is not whoever runs the tool, and raising a limit takes the +tool's own `--limit`. + +| figure | default | why | +|---|---|---| +| output bytes | 2²⁵ (32 MiB) | five 1920 × 1080 RGB8 frames, or minutes of 48 kHz stereo PCM: more than one stage set's base carries, and below the declared clips a tool must refuse | +| symbols | 2²⁵ | one entropy-decode step per output byte at the output limit | +| working bytes | 2²⁸ (256 MiB) | eight bytes per network activation, for networks a few times the output's size | +| multiply-accumulates | 2³⁸ | a learned decode of several megapixels | +| chain budget | 2²⁸ (256 MiB) | eight times the output limit, for a base's units and the layers retained over them | diff --git a/wai/research/energy-exposure-mapping.md b/wai/research/energy-exposure-mapping.md index cf204ce89..8930c82ba 100644 --- a/wai/research/energy-exposure-mapping.md +++ b/wai/research/energy-exposure-mapping.md @@ -648,3 +648,12 @@ Read on 2026-09-27 unless noted. 11. [2026-09-global-sweep](2026-09-global-sweep.md), for the CM-EE announcement (§2.4), the ISO/IEC 23001-19 stage (§2.1) and the annexes of C(2026) 3472 (§5.2). + +## 2026-10 addendum: delivery telemetry + +CMSD headers and 5GMS remain outside WAI's receipt layer. Since this note was +written, [delivery-telemetry](../extensions/delivery-telemetry.md) maps an unsigned, +advisory observation about WAI delivery onto CMCD version 2 and CMSD keys. It carries +content hashes and receipt link hashes as pointers, never a joule figure, an +acquisition class or a quality figure, and it binds no receipt to CMCD or CMSD. The +text above is unchanged. diff --git a/wai/staged-measure-conformance/.gitattributes b/wai/staged-measure-conformance/.gitattributes new file mode 100644 index 000000000..eff1df73a --- /dev/null +++ b/wai/staged-measure-conformance/.gitattributes @@ -0,0 +1,2 @@ +# Hashed files: a line-ending conversion on checkout would change every digest. +* -text diff --git a/wai/staged-measure-conformance/README.md b/wai/staged-measure-conformance/README.md new file mode 100644 index 000000000..ef8af084f --- /dev/null +++ b/wai/staged-measure-conformance/README.md @@ -0,0 +1,121 @@ +# staged-measure-conformance — staged-delivery measurement + +The corpus of [`extensions/staged-measurement.md`](../extensions/staged-measurement.md) +§12.3: measured titles, and every figure continuous integration may assert about them +(§12.1). + +**What a conforming implementation reproduces.** For each title: the counted bytes of +every object, split into envelope header, manifest and payload (§3), and their sums per +stage and per prefix, per base and over the title; the bytes of each base's parameter +set; the first-presentable bytes (§5.1); every presentation's output digest (SPEC §7 +*Digests*), with its quality integers against the source and against the final prefix +(§4), and every prefix's pooled quality; the decoder entries and the output tree's root +and leaf count (Appendix C); the layering overhead's rate–quality points, the full title's and each +single-layer reference's (§6); the deduplication totals (§9); and the title's content +identity (Appendix D). Across the corpus: the record-wide deduplication totals (§9.2). + +- `fixtures//` — one title (Appendix D): `title.json`; its layers (`L*.wai`) and + its single-layer references' layers (`R*.wai`); `source.bin`, each unit's canonical + buffer at its final shape; `expect.json`, the fixture tool's own reconstruction + digests. Each title's base is a base of a [staged-conformance](../staged-conformance/) + case, read where it is, with the case's stage 0 and parameter set. +- `goldens.json` — every title's figures, in fixture order, and the record-wide + totals. +- `quality.json` — §4.2's integers for picture pairs drawn from a stated formula, at + sizes that place the similarity's windows every way the rule does: narrower or + shorter than a window, offsets that reach the edge at a multiple of 4, and an + edge-aligned last window on one axis or both. +- `counting/` and `counting.json` — objects whose counting §3.2 decides, each with its + counted bytes or the code it is not countable with: a `WIH1` base, a `WIR1` layer and + a `WAI2` (counted); the JSON form of `WIH1`, its base64 text, a JSON string, and + `WIH1` behind a byte-order mark (`bad_magic`); a PNG base and an unregistered + capability (`no-format-check`); `WIH1` under a `wai.video.int_motion` fallback + (`format_capability_mismatch`) and under a fallback WAI checks no format for + (counted); a `WAI2` component with another payload's `sha256` (`sha256-mismatch`); an + encrypted payload; and bytes after an envelope (`trailing`). +- `refusals//title.json` — titles a tool must refuse, each with the code it states + (`refused`): a base whose units pass the chain budget its title states + (`chain-over-limit`, over the staged-delivery corpus's `base_over_budget` case); the + two bound vectors under `../wai-rs/tests/vectors/staged_bounds/`, whose headers declare + 64 MiB and 1 GiB of output, past the tools' stated limit (`cost_over_limit`, refused + before anything is allocated for them), the second also under a title whose `limits` + state 2^40 for every figure, which a title cannot raise; and the two envelopes there whose capability + is refinement-only or companion-only (`missing_capability`: inert, the fallback never + reached); and the still title's up2 layer under a cost limit its base passes and the + layer's declared output does not (`cost-over-limit`). +- `sim/` — the delivery simulator's corpus (§10, Appendix B): `models/` (a synthetic + model of three groups of five units, each a base and two layers per unit, captured as + each group ends), `plans/` (`wai-sim/1`), `cases.json` (a model — a file, or + `fixture:` for a title's objects and counted bytes — a plan, and the case's + claim: `within-bound`, `exceeds:` or `unbounded`) and `results.json` (every + case simulated: each object's delivery, the presentations, first presentable, the + byte totals, the missed deadlines, each unit's stall, the plan's bound and the + largest stall). The plans cover both carriages and both bounded link schedules, a slow + link on which refinements miss their timeouts (with a window, so that a reset takes a + subgroup's tail before its own deadlines), a layer cap, live playout with late + refinements discarded or presented, two models on which bases were once sent or + decoded out of the order they became available (`reorder-link`, `reorder-decode`), + and a counter-plan for each part of §10.4's bound a schedule can lose: a `fifo` link + with no refinement timeout, whose stall passes the bound of the same `fifo` link + with a short timeout (`fifo-timeout-short`), and `fifo` decode on one worker, whose + stall passes the bound of the same plan with `base-first` decode (`strict-window`). + +**The content is synthetic.** Each base is a committed vector under +[`../wai-rs/tests/vectors/`](../wai-rs/tests/vectors/), as the staged-delivery corpus +wraps it: a `wai.neural.int_hyper` picture, a `wai.video.int_hyper` clip, a +`wai.audio.int_codec` clip and a `wai.splat.int_codec` attribute tensor, each pinned to +its parameter set. Each source is the base's output with fine detail added by integer +functions of position, channel and a seed (for audio, two triangle waves), raised to the +final shape by the layer's own predictor where the title changes the shape. The layers +are `WIR1` payloads coded toward the source by the reference encoder +(`staged::wir1::encode`): up2 then identity for the picture; one identity layer per +frame, twice, for the clip; a rate layer to twice the sample rate, then identity, for +the audio; identity for the attribute tensor. Each title's single-layer references code +one layer straight from stage 0 at the final step and, for some titles, at further +steps (the curve of §6.3). + +**Verifiers.** + +- `wai_stage_measure verify .` (from `../wai-rs`, feature `stage_measure`): regenerates + every fixture and refusal title in memory and requires it byte for byte; decodes each + base through the sink's own base decode (`staged::decode_base_pinned`) under the + title's limits and requires the case's stage 0, capability, fallback, tier and pin; + applies the layers through the reference sink (`staged::StageSink`) under the title's + chain budget; requires every refusal title refused with its code; requires every + presentation's digest and + every reference's final digests to be the fixture tool's (`expect.json`, written from + the encoder's own reconstruction); and requires `goldens.json` byte for byte. + `wai_stage_measure generate-goldens .` and `wai_stage_measure gen-fixtures .` write + them; `wai_stage_measure eval fixtures/` prints one title's figures. +- `wai_stage_measure verify .` also regenerates the simulator's models, plans and cases + byte for byte, simulates every case (`stage_measure::sim`), checks each claim, and + requires `sim/results.json` byte for byte; `wai_stage_measure sim ` + runs one. +- `sim_verify.py .`: pure Python with the standard library, sharing no code with + `wai-rs`: Appendix B implemented from the text, with each fixture's model taken from + `measure_verify.py`'s own evaluation. It reproduces every result, checks every claim, + and requires a counter-case for each part of the bound a schedule can lose. +- `measure_verify.py .`: pure Python with the standard library, sharing no code with + `wai-rs`. It reimplements, from the extension and SPEC: the byte rule and the formats + it counts; each base's units from its case's stage 0; every layer applied in layer + order to every unit it covers, with its predecessor checked; the integer luma, SSE, + the Q32 structural similarity with its window placement (Python's `//` floors toward + −∞, as §4.2 states), checked over `quality.json` as well, PCM and tensor sums and + their pooling; the output tree; the prefixes, first presentable, + overhead points, deduplication and content identity. It requires its decode to be the + fixture tool's and every figure to be `goldens.json`'s. Before it uses a base it runs + its own base checks (SPEC §4's dispatch, the output and symbols the payload's header + declares against the title's cost limit, the units against its chain budget), and it + requires every refusal title refused, by those checks, with the code the title + states. It takes BLAKE3 from + [`../claim-conformance/_blake3.py`](../claim-conformance/_blake3.py), the integer + payload parsers from + [`../int-payload-conformance/int_payload_verify.py`](../int-payload-conformance/int_payload_verify.py), + and the `WAI1` reader, `refines`, `WIR1` and its predictors from + [`../staged-conformance/staged_verify.py`](../staged-conformance/staged_verify.py), + each written from its specification. Each base's stage 0 is the staged-delivery + corpus's, which `staged_verify.py` checks against the vectors' sidecars. + +Every figure here is integer, and the same on every machine. The simulator's times are +virtual and labelled simulated; no measured time and no energy is stated: those are +measured, never asserted (§12.2). diff --git a/wai/staged-measure-conformance/counting.json b/wai/staged-measure-conformance/counting.json new file mode 100644 index 000000000..546aee5aa --- /dev/null +++ b/wai/staged-measure-conformance/counting.json @@ -0,0 +1,110 @@ +{ + "format": "wai-measure-counting/1", + "about": "Objects whose counting extensions/staged-measurement.md §3.2 decides: each object's counted bytes, or the code it is not countable with (and the WAI2 component). Written by wai_stage_measure gen-fixtures; required by wai_stage_measure verify and measure_verify.py.", + "cases": [ + { + "file": "staged-measure-conformance/counting/wih1.wai", + "counted": { + "header": 12, + "manifest": 158, + "payload": 88, + "total": 258 + }, + "refused": null, + "component": null + }, + { + "file": "staged-measure-conformance/fixtures/still/L01.wai", + "counted": { + "header": 12, + "manifest": 323, + "payload": 616, + "total": 951 + }, + "refused": null, + "component": null + }, + { + "file": "staged-measure-conformance/counting/json-form.wai", + "counted": null, + "refused": "bad_magic", + "component": null + }, + { + "file": "staged-measure-conformance/counting/base64.wai", + "counted": null, + "refused": "bad_magic", + "component": null + }, + { + "file": "staged-measure-conformance/counting/json-string.wai", + "counted": null, + "refused": "bad_magic", + "component": null + }, + { + "file": "staged-measure-conformance/counting/bom.wai", + "counted": null, + "refused": "bad_magic", + "component": null + }, + { + "file": "staged-conformance/cases/image_png_up2/base.wai", + "counted": null, + "refused": "no-format-check", + "component": null + }, + { + "file": "staged-measure-conformance/counting/unregistered-capability.wai", + "counted": null, + "refused": "no-format-check", + "component": null + }, + { + "file": "staged-measure-conformance/counting/fallback-mismatch.wai", + "counted": null, + "refused": "format_capability_mismatch", + "component": null + }, + { + "file": "staged-measure-conformance/counting/fallback-unchecked.wai", + "counted": { + "header": 12, + "manifest": 185, + "payload": 88, + "total": 285 + }, + "refused": null, + "component": null + }, + { + "file": "staged-measure-conformance/counting/wai2.wai", + "counted": { + "header": 18, + "manifest": 212, + "payload": 88, + "total": 318 + }, + "refused": null, + "component": null + }, + { + "file": "staged-measure-conformance/counting/wai2-sha256-mismatch.wai", + "counted": null, + "refused": "sha256-mismatch", + "component": 0 + }, + { + "file": "staged-measure-conformance/counting/encrypted.wai", + "counted": null, + "refused": "encrypted", + "component": null + }, + { + "file": "staged-measure-conformance/counting/trailing.wai", + "counted": null, + "refused": "trailing", + "component": null + } + ] +} diff --git a/wai/staged-measure-conformance/counting/base64.wai b/wai/staged-measure-conformance/counting/base64.wai new file mode 100644 index 000000000..c7e337547 Binary files /dev/null and b/wai/staged-measure-conformance/counting/base64.wai differ diff --git a/wai/staged-measure-conformance/counting/bom.wai b/wai/staged-measure-conformance/counting/bom.wai new file mode 100644 index 000000000..ca0f57eaf Binary files /dev/null and b/wai/staged-measure-conformance/counting/bom.wai differ diff --git a/wai/staged-measure-conformance/counting/encrypted.wai b/wai/staged-measure-conformance/counting/encrypted.wai new file mode 100644 index 000000000..b3d24a03b Binary files /dev/null and b/wai/staged-measure-conformance/counting/encrypted.wai differ diff --git a/wai/staged-measure-conformance/counting/fallback-mismatch.wai b/wai/staged-measure-conformance/counting/fallback-mismatch.wai new file mode 100644 index 000000000..58b498934 Binary files /dev/null and b/wai/staged-measure-conformance/counting/fallback-mismatch.wai differ diff --git a/wai/staged-measure-conformance/counting/fallback-unchecked.wai b/wai/staged-measure-conformance/counting/fallback-unchecked.wai new file mode 100644 index 000000000..ba2ef1ff3 Binary files /dev/null and b/wai/staged-measure-conformance/counting/fallback-unchecked.wai differ diff --git a/wai/staged-measure-conformance/counting/json-form.wai b/wai/staged-measure-conformance/counting/json-form.wai new file mode 100644 index 000000000..61289d284 Binary files /dev/null and b/wai/staged-measure-conformance/counting/json-form.wai differ diff --git a/wai/staged-measure-conformance/counting/json-string.wai b/wai/staged-measure-conformance/counting/json-string.wai new file mode 100644 index 000000000..bc46fb573 Binary files /dev/null and b/wai/staged-measure-conformance/counting/json-string.wai differ diff --git a/wai/staged-measure-conformance/counting/trailing.wai b/wai/staged-measure-conformance/counting/trailing.wai new file mode 100644 index 000000000..14b61e09c Binary files /dev/null and b/wai/staged-measure-conformance/counting/trailing.wai differ diff --git a/wai/staged-measure-conformance/counting/unregistered-capability.wai b/wai/staged-measure-conformance/counting/unregistered-capability.wai new file mode 100644 index 000000000..add8867d3 Binary files /dev/null and b/wai/staged-measure-conformance/counting/unregistered-capability.wai differ diff --git a/wai/staged-measure-conformance/counting/wai2-sha256-mismatch.wai b/wai/staged-measure-conformance/counting/wai2-sha256-mismatch.wai new file mode 100644 index 000000000..07399c787 Binary files /dev/null and b/wai/staged-measure-conformance/counting/wai2-sha256-mismatch.wai differ diff --git a/wai/staged-measure-conformance/counting/wai2.wai b/wai/staged-measure-conformance/counting/wai2.wai new file mode 100644 index 000000000..0ee9f2c6d Binary files /dev/null and b/wai/staged-measure-conformance/counting/wai2.wai differ diff --git a/wai/staged-measure-conformance/counting/wih1.wai b/wai/staged-measure-conformance/counting/wih1.wai new file mode 100644 index 000000000..444eb9dea Binary files /dev/null and b/wai/staged-measure-conformance/counting/wih1.wai differ diff --git a/wai/staged-measure-conformance/fixtures/audio/L01.wai b/wai/staged-measure-conformance/fixtures/audio/L01.wai new file mode 100644 index 000000000..73ae8da1c Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/audio/L01.wai differ diff --git a/wai/staged-measure-conformance/fixtures/audio/L02.wai b/wai/staged-measure-conformance/fixtures/audio/L02.wai new file mode 100644 index 000000000..4d2689fbe Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/audio/L02.wai differ diff --git a/wai/staged-measure-conformance/fixtures/audio/L03.wai b/wai/staged-measure-conformance/fixtures/audio/L03.wai new file mode 100644 index 000000000..a12b9c1f2 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/audio/L03.wai differ diff --git a/wai/staged-measure-conformance/fixtures/audio/R0_00.wai b/wai/staged-measure-conformance/fixtures/audio/R0_00.wai new file mode 100644 index 000000000..e615fa00c Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/audio/R0_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/audio/R1_00.wai b/wai/staged-measure-conformance/fixtures/audio/R1_00.wai new file mode 100644 index 000000000..b52afac83 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/audio/R1_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/audio/expect.json b/wai/staged-measure-conformance/fixtures/audio/expect.json new file mode 100644 index 000000000..c27725d72 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/audio/expect.json @@ -0,0 +1,40 @@ +{ + "format": "wai-measure-expect/1", + "about": "The fixture tool's own reconstruction: the SHA-256 of each unit's canonical buffer at each stage it coded, and of each reference's final units, from the encoder's reconstruction loop.", + "presentations": [ + { + "unit": 0, + "layers": 1, + "out_sha256": "b8ff83a442c112661ec7a41d6e51cb32559510a699a0e9bb37fa301f8952978a" + }, + { + "unit": 0, + "layers": 2, + "out_sha256": "7cf4ee1daabc1f8779252bb3e02f16ba2ab2d8a65b374c8cbb1c3c3354121cdd" + }, + { + "unit": 0, + "layers": 3, + "out_sha256": "02c3e3540210f1e5fce2b9f75d38e74d9b877cf17f2233b7825a8a10b823c64c" + }, + { + "unit": 0, + "layers": 4, + "out_sha256": "8405e88c07e18cd93aecd578493cb3ff060c98be1790ebca0edeb0d1af1c512a" + } + ], + "references": [ + { + "label": "one layer, q 1", + "out_sha256": [ + "8405e88c07e18cd93aecd578493cb3ff060c98be1790ebca0edeb0d1af1c512a" + ] + }, + { + "label": "one layer, q 8", + "out_sha256": [ + "9b34a3dcbc301e180a7a675b3304ed57ddee43858863e113a55455a926d614e8" + ] + } + ] +} diff --git a/wai/staged-measure-conformance/fixtures/audio/source.bin b/wai/staged-measure-conformance/fixtures/audio/source.bin new file mode 100644 index 000000000..e536cf120 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/audio/source.bin differ diff --git a/wai/staged-measure-conformance/fixtures/audio/title.json b/wai/staged-measure-conformance/fixtures/audio/title.json new file mode 100644 index 000000000..17f511284 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/audio/title.json @@ -0,0 +1,38 @@ +{ + "format": "wai-measure-title/1", + "title": "audio", + "about": "a pinned wai.audio.int_codec clip (8192 samples, mono); a rate layer to twice the sample rate at q 64, then identity layers at q 8 and q 1; references: one rate layer at q 1 and at q 8", + "unit_us": 512000, + "bases": [ + { + "case": "staged-conformance/cases/audio_int_codec_1l", + "base": 0, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [ + "L01.wai", + "L02.wai", + "L03.wai" + ], + "source": { + "file": "source.bin", + "sha256": "8405e88c07e18cd93aecd578493cb3ff060c98be1790ebca0edeb0d1af1c512a" + }, + "references": [ + { + "label": "one layer, q 1", + "layers": [ + "R0_00.wai" + ] + }, + { + "label": "one layer, q 8", + "layers": [ + "R1_00.wai" + ] + } + ], + "expect": "expect.json" +} diff --git a/wai/staged-measure-conformance/fixtures/clip/L01.wai b/wai/staged-measure-conformance/fixtures/clip/L01.wai new file mode 100644 index 000000000..0e890a125 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L01.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L02.wai b/wai/staged-measure-conformance/fixtures/clip/L02.wai new file mode 100644 index 000000000..f4e6a453c Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L02.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L03.wai b/wai/staged-measure-conformance/fixtures/clip/L03.wai new file mode 100644 index 000000000..e31445f39 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L03.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L04.wai b/wai/staged-measure-conformance/fixtures/clip/L04.wai new file mode 100644 index 000000000..ecc28919d Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L04.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L05.wai b/wai/staged-measure-conformance/fixtures/clip/L05.wai new file mode 100644 index 000000000..b96b75c87 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L05.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L06.wai b/wai/staged-measure-conformance/fixtures/clip/L06.wai new file mode 100644 index 000000000..514234aa1 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L06.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L07.wai b/wai/staged-measure-conformance/fixtures/clip/L07.wai new file mode 100644 index 000000000..724611eda Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L07.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L08.wai b/wai/staged-measure-conformance/fixtures/clip/L08.wai new file mode 100644 index 000000000..6aafc99cf Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L08.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L09.wai b/wai/staged-measure-conformance/fixtures/clip/L09.wai new file mode 100644 index 000000000..b5cd2c10b Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L09.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/L10.wai b/wai/staged-measure-conformance/fixtures/clip/L10.wai new file mode 100644 index 000000000..4326b621c Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/L10.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R0_00.wai b/wai/staged-measure-conformance/fixtures/clip/R0_00.wai new file mode 100644 index 000000000..fd31d576f Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R0_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R0_01.wai b/wai/staged-measure-conformance/fixtures/clip/R0_01.wai new file mode 100644 index 000000000..cdc836ad8 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R0_01.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R0_02.wai b/wai/staged-measure-conformance/fixtures/clip/R0_02.wai new file mode 100644 index 000000000..167b447d4 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R0_02.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R0_03.wai b/wai/staged-measure-conformance/fixtures/clip/R0_03.wai new file mode 100644 index 000000000..3a8bb82ee Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R0_03.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R0_04.wai b/wai/staged-measure-conformance/fixtures/clip/R0_04.wai new file mode 100644 index 000000000..8e206ef87 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R0_04.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R1_00.wai b/wai/staged-measure-conformance/fixtures/clip/R1_00.wai new file mode 100644 index 000000000..0e890a125 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R1_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R1_01.wai b/wai/staged-measure-conformance/fixtures/clip/R1_01.wai new file mode 100644 index 000000000..f4e6a453c Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R1_01.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R1_02.wai b/wai/staged-measure-conformance/fixtures/clip/R1_02.wai new file mode 100644 index 000000000..e31445f39 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R1_02.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R1_03.wai b/wai/staged-measure-conformance/fixtures/clip/R1_03.wai new file mode 100644 index 000000000..ecc28919d Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R1_03.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/R1_04.wai b/wai/staged-measure-conformance/fixtures/clip/R1_04.wai new file mode 100644 index 000000000..b96b75c87 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/R1_04.wai differ diff --git a/wai/staged-measure-conformance/fixtures/clip/expect.json b/wai/staged-measure-conformance/fixtures/clip/expect.json new file mode 100644 index 000000000..87f318676 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/clip/expect.json @@ -0,0 +1,103 @@ +{ + "format": "wai-measure-expect/1", + "about": "The fixture tool's own reconstruction: the SHA-256 of each unit's canonical buffer at each stage it coded, and of each reference's final units, from the encoder's reconstruction loop.", + "presentations": [ + { + "unit": 0, + "layers": 1, + "out_sha256": "dce7282d46f88f03d5dc0fd42f325756d310fe1fc982b6e1792a1fbfbeb482cf" + }, + { + "unit": 0, + "layers": 2, + "out_sha256": "71639c87086d0fe69eb0e9ba8a976e882cb7ac38056a3c388d901dd3f0ac4fd2" + }, + { + "unit": 0, + "layers": 3, + "out_sha256": "7655e2d7055f1ea34084785b13e0becbec58301023ee9beff3e75a11abb7b621" + }, + { + "unit": 1, + "layers": 1, + "out_sha256": "eb22eaca420f922c3e65f7583e4ba15195a013a4694d3d704c1c393477279197" + }, + { + "unit": 1, + "layers": 2, + "out_sha256": "5ca748b74ae00febec767d871f336c61cb1f67db1d0451a02112191f7f9c664f" + }, + { + "unit": 1, + "layers": 3, + "out_sha256": "06e01123126a93ebd5f845883f66c959a57270662fc7982e32d0768637b48722" + }, + { + "unit": 2, + "layers": 1, + "out_sha256": "904e9064df9f95582e2ac4b9ef594a27810922c325c5144acd4d17e08294ecdf" + }, + { + "unit": 2, + "layers": 2, + "out_sha256": "804b060fcb5fe3ea6a1588c5768a4520eb25bb67df51cbdc4de392253bcfd387" + }, + { + "unit": 2, + "layers": 3, + "out_sha256": "1e946bd550123521b8d6e67485c2058159e6b3d82b58f709ad23ceb223c6bb0c" + }, + { + "unit": 3, + "layers": 1, + "out_sha256": "ba33b5f7ffe2dd451106b43596f7be854ba0a5e011039a66bc8f221b4321c2df" + }, + { + "unit": 3, + "layers": 2, + "out_sha256": "14eb7bbe7d69932210f029c7319f19f2e911f43c2222b08fc345f9a04069264f" + }, + { + "unit": 3, + "layers": 3, + "out_sha256": "7b577471db6e6c1b9f301c34470678407d6c6d74e44d89ca6152d1c12f177fe6" + }, + { + "unit": 4, + "layers": 1, + "out_sha256": "38ccbd2a62233c72087b136588551fdb742ae98a1e78a5e4f6b68c25b7778e3b" + }, + { + "unit": 4, + "layers": 2, + "out_sha256": "0261a1b9ae00788df786e4cc542d36629cff822c52195ce8be5c12d33ff66905" + }, + { + "unit": 4, + "layers": 3, + "out_sha256": "6d6ff21e4b9654c757a98fafc949e760e01ea22953f201affd4a738a15be6726" + } + ], + "references": [ + { + "label": "one layer, q 2", + "out_sha256": [ + "7655e2d7055f1ea34084785b13e0becbec58301023ee9beff3e75a11abb7b621", + "06e01123126a93ebd5f845883f66c959a57270662fc7982e32d0768637b48722", + "1e946bd550123521b8d6e67485c2058159e6b3d82b58f709ad23ceb223c6bb0c", + "7b577471db6e6c1b9f301c34470678407d6c6d74e44d89ca6152d1c12f177fe6", + "6d6ff21e4b9654c757a98fafc949e760e01ea22953f201affd4a738a15be6726" + ] + }, + { + "label": "one layer, q 6", + "out_sha256": [ + "71639c87086d0fe69eb0e9ba8a976e882cb7ac38056a3c388d901dd3f0ac4fd2", + "5ca748b74ae00febec767d871f336c61cb1f67db1d0451a02112191f7f9c664f", + "804b060fcb5fe3ea6a1588c5768a4520eb25bb67df51cbdc4de392253bcfd387", + "14eb7bbe7d69932210f029c7319f19f2e911f43c2222b08fc345f9a04069264f", + "0261a1b9ae00788df786e4cc542d36629cff822c52195ce8be5c12d33ff66905" + ] + } + ] +} diff --git a/wai/staged-measure-conformance/fixtures/clip/source.bin b/wai/staged-measure-conformance/fixtures/clip/source.bin new file mode 100644 index 000000000..bd32f65b2 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/clip/source.bin differ diff --git a/wai/staged-measure-conformance/fixtures/clip/title.json b/wai/staged-measure-conformance/fixtures/clip/title.json new file mode 100644 index 000000000..5ff065dd2 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/clip/title.json @@ -0,0 +1,53 @@ +{ + "format": "wai-measure-title/1", + "title": "clip", + "about": "a pinned wai.video.int_hyper clip (5 frames, 16×16); one identity layer per frame at q 6, then one per frame at q 2; references: one layer per frame at q 2 and at q 6", + "unit_us": 40000, + "bases": [ + { + "case": "staged-conformance/cases/video_int_hyper_keyframe", + "base": 0, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [ + "L01.wai", + "L02.wai", + "L03.wai", + "L04.wai", + "L05.wai", + "L06.wai", + "L07.wai", + "L08.wai", + "L09.wai", + "L10.wai" + ], + "source": { + "file": "source.bin", + "sha256": "9148bb9adeb87d45530d07ded7a4f648861f119af5347ab0d84962f75762175d" + }, + "references": [ + { + "label": "one layer, q 2", + "layers": [ + "R0_00.wai", + "R0_01.wai", + "R0_02.wai", + "R0_03.wai", + "R0_04.wai" + ] + }, + { + "label": "one layer, q 6", + "layers": [ + "R1_00.wai", + "R1_01.wai", + "R1_02.wai", + "R1_03.wai", + "R1_04.wai" + ] + } + ], + "expect": "expect.json" +} diff --git a/wai/staged-measure-conformance/fixtures/splat/L01.wai b/wai/staged-measure-conformance/fixtures/splat/L01.wai new file mode 100644 index 000000000..a7ca70f6e Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/splat/L01.wai differ diff --git a/wai/staged-measure-conformance/fixtures/splat/L02.wai b/wai/staged-measure-conformance/fixtures/splat/L02.wai new file mode 100644 index 000000000..84032f313 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/splat/L02.wai differ diff --git a/wai/staged-measure-conformance/fixtures/splat/R0_00.wai b/wai/staged-measure-conformance/fixtures/splat/R0_00.wai new file mode 100644 index 000000000..36f645fd7 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/splat/R0_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/splat/expect.json b/wai/staged-measure-conformance/fixtures/splat/expect.json new file mode 100644 index 000000000..f90c10e06 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/splat/expect.json @@ -0,0 +1,29 @@ +{ + "format": "wai-measure-expect/1", + "about": "The fixture tool's own reconstruction: the SHA-256 of each unit's canonical buffer at each stage it coded, and of each reference's final units, from the encoder's reconstruction loop.", + "presentations": [ + { + "unit": 0, + "layers": 1, + "out_sha256": "f07679de87ff7c3d4fc4890b80919ef857fb85bd4d9e4cfa16105211d580427f" + }, + { + "unit": 0, + "layers": 2, + "out_sha256": "4603b50563302461ce2baeaa1810268c9ebe834599f62790d7acf01da33c66fd" + }, + { + "unit": 0, + "layers": 3, + "out_sha256": "9a177a9b1879bc66804f7c96d98755d39ee16aca5d8e042b31706aff4c539bee" + } + ], + "references": [ + { + "label": "one layer, q 1", + "out_sha256": [ + "9a177a9b1879bc66804f7c96d98755d39ee16aca5d8e042b31706aff4c539bee" + ] + } + ] +} diff --git a/wai/staged-measure-conformance/fixtures/splat/source.bin b/wai/staged-measure-conformance/fixtures/splat/source.bin new file mode 100644 index 000000000..3aa0830f8 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/splat/source.bin differ diff --git a/wai/staged-measure-conformance/fixtures/splat/title.json b/wai/staged-measure-conformance/fixtures/splat/title.json new file mode 100644 index 000000000..adb0753d0 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/splat/title.json @@ -0,0 +1,31 @@ +{ + "format": "wai-measure-title/1", + "title": "splat", + "about": "a pinned wai.splat.int_codec attribute tensor (14 planes of 32×32); an identity layer at q 6, then one at q 1; reference: one identity layer at q 1", + "unit_us": 40000, + "bases": [ + { + "case": "staged-conformance/cases/splat_attr_1l", + "base": 0, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [ + "L01.wai", + "L02.wai" + ], + "source": { + "file": "source.bin", + "sha256": "9a177a9b1879bc66804f7c96d98755d39ee16aca5d8e042b31706aff4c539bee" + }, + "references": [ + { + "label": "one layer, q 1", + "layers": [ + "R0_00.wai" + ] + } + ], + "expect": "expect.json" +} diff --git a/wai/staged-measure-conformance/fixtures/still/L01.wai b/wai/staged-measure-conformance/fixtures/still/L01.wai new file mode 100644 index 000000000..ff421c828 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/L01.wai differ diff --git a/wai/staged-measure-conformance/fixtures/still/L02.wai b/wai/staged-measure-conformance/fixtures/still/L02.wai new file mode 100644 index 000000000..38eafbf37 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/L02.wai differ diff --git a/wai/staged-measure-conformance/fixtures/still/L03.wai b/wai/staged-measure-conformance/fixtures/still/L03.wai new file mode 100644 index 000000000..53292627c Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/L03.wai differ diff --git a/wai/staged-measure-conformance/fixtures/still/R0_00.wai b/wai/staged-measure-conformance/fixtures/still/R0_00.wai new file mode 100644 index 000000000..d9fbb2f9b Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/R0_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/still/R1_00.wai b/wai/staged-measure-conformance/fixtures/still/R1_00.wai new file mode 100644 index 000000000..252531106 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/R1_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/still/R2_00.wai b/wai/staged-measure-conformance/fixtures/still/R2_00.wai new file mode 100644 index 000000000..ff421c828 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/R2_00.wai differ diff --git a/wai/staged-measure-conformance/fixtures/still/expect.json b/wai/staged-measure-conformance/fixtures/still/expect.json new file mode 100644 index 000000000..e5b0a7b38 --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/still/expect.json @@ -0,0 +1,46 @@ +{ + "format": "wai-measure-expect/1", + "about": "The fixture tool's own reconstruction: the SHA-256 of each unit's canonical buffer at each stage it coded, and of each reference's final units, from the encoder's reconstruction loop.", + "presentations": [ + { + "unit": 0, + "layers": 1, + "out_sha256": "dce7282d46f88f03d5dc0fd42f325756d310fe1fc982b6e1792a1fbfbeb482cf" + }, + { + "unit": 0, + "layers": 2, + "out_sha256": "827c831aef7df8d700616cd896dcb5d2cde2a926afaea503d85487026836a34a" + }, + { + "unit": 0, + "layers": 3, + "out_sha256": "952ebbf5a937f327c0d056684ed88fef9d9489be4735a456ae8ecdd2d299ace6" + }, + { + "unit": 0, + "layers": 4, + "out_sha256": "fef81da98166b22c30e66ce6dda9a835aa95a7d1aa278c6046dd01a8c56f0efd" + } + ], + "references": [ + { + "label": "one layer, q 1", + "out_sha256": [ + "fef81da98166b22c30e66ce6dda9a835aa95a7d1aa278c6046dd01a8c56f0efd" + ] + }, + { + "label": "one layer, q 3", + "out_sha256": [ + "8181c12af11507de87344cbf5a3f9217025c4f2f946ac1a12011e531806e449d" + ] + }, + { + "label": "one layer, q 8", + "out_sha256": [ + "827c831aef7df8d700616cd896dcb5d2cde2a926afaea503d85487026836a34a" + ] + } + ] +} diff --git a/wai/staged-measure-conformance/fixtures/still/source.bin b/wai/staged-measure-conformance/fixtures/still/source.bin new file mode 100644 index 000000000..26e744120 Binary files /dev/null and b/wai/staged-measure-conformance/fixtures/still/source.bin differ diff --git a/wai/staged-measure-conformance/fixtures/still/title.json b/wai/staged-measure-conformance/fixtures/still/title.json new file mode 100644 index 000000000..ab370025b --- /dev/null +++ b/wai/staged-measure-conformance/fixtures/still/title.json @@ -0,0 +1,44 @@ +{ + "format": "wai-measure-title/1", + "title": "still", + "about": "a pinned wai.neural.int_hyper picture (16×16); an up2 layer to 32×32 at q 8, then identity layers at q 3 and q 1; references: one up2 layer at q 1, 3 and 8", + "unit_us": 40000, + "bases": [ + { + "case": "staged-conformance/cases/image_int_hyper_2l", + "base": 0, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [ + "L01.wai", + "L02.wai", + "L03.wai" + ], + "source": { + "file": "source.bin", + "sha256": "fef81da98166b22c30e66ce6dda9a835aa95a7d1aa278c6046dd01a8c56f0efd" + }, + "references": [ + { + "label": "one layer, q 1", + "layers": [ + "R0_00.wai" + ] + }, + { + "label": "one layer, q 3", + "layers": [ + "R1_00.wai" + ] + }, + { + "label": "one layer, q 8", + "layers": [ + "R2_00.wai" + ] + } + ], + "expect": "expect.json" +} diff --git a/wai/staged-measure-conformance/goldens.json b/wai/staged-measure-conformance/goldens.json new file mode 100644 index 000000000..52df14cc4 --- /dev/null +++ b/wai/staged-measure-conformance/goldens.json @@ -0,0 +1,1744 @@ +{ + "format": "wai-measure-goldens/1", + "about": "The deterministic figures of every fixture (extensions/staged-measurement.md §12.1), written by wai_stage_measure generate-goldens and required by wai_stage_measure verify and measure_verify.py.", + "fixtures": [ + { + "title": "audio", + "content_id": "dd64e04d02388d970bf07a6759ea97f073e2329abcb03acb910d5d0a25ac8462", + "framing": "registered", + "unit_class": "pcm16", + "unit_us": 512000, + "bases": [ + { + "object": "d45db2bc98dd38873553beb15fdb026a4e6b7fd7c8c7dbe4f4a0b6fcc0709a54", + "capability": "wai.audio.int_codec", + "pin": "98b382c92112660a7d3909f07da831125fcdb75a548aa0cf64ad17a0901bd9ef", + "tier": "decode-equivalence", + "first_unit": 0, + "capture_us": null, + "units": 1, + "param_set_bytes": 12215 + } + ], + "objects": [ + { + "file": "staged-conformance/cases/audio_int_codec_1l/base.wai", + "object": "d45db2bc98dd38873553beb15fdb026a4e6b7fd7c8c7dbe4f4a0b6fcc0709a54", + "capability": "wai.audio.int_codec", + "base": 0, + "layer": 0, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 338, + "payload": 1444, + "total": 1794 + } + }, + { + "file": "staged-measure-conformance/fixtures/audio/L01.wai", + "object": "14c630b04c617542d3df5a40b398c2fa5c960d6020f84a961046ff5f7831dc69", + "capability": "wai.audio.int_refine", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 323, + "payload": 10016, + "total": 10351 + } + }, + { + "file": "staged-measure-conformance/fixtures/audio/L02.wai", + "object": "4f935f7fade9bd09ba65d65fca37083e18b57c6251c4f1ded91d7bab2c1640b0", + "capability": "wai.audio.int_refine", + "base": 0, + "layer": 2, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 323, + "payload": 6524, + "total": 6859 + } + }, + { + "file": "staged-measure-conformance/fixtures/audio/L03.wai", + "object": "ed29fe08a0e7aafba5c71016408cbab959ddcda07914ee0ca08cc58f2d7cf549", + "capability": "wai.audio.int_refine", + "base": 0, + "layer": 3, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 323, + "payload": 6496, + "total": 6831 + } + } + ], + "bfpf_min": 1794, + "prefixes": [ + { + "layers": 1, + "stage_counted": 1794, + "counted": 1794, + "per_base": [ + { + "stage_counted": 1794, + "counted": 1794 + } + ], + "units": 1, + "vs_source": null, + "vs_final": null + }, + { + "layers": 2, + "stage_counted": 10351, + "counted": 12145, + "per_base": [ + { + "stage_counted": 10351, + "counted": 12145 + } + ], + "units": 1, + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "5600413", + "ref_energy": "77076760253" + }, + "vs_final": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "5600413", + "ref_energy": "77076760253" + } + }, + { + "layers": 3, + "stage_counted": 6859, + "counted": 19004, + "per_base": [ + { + "stage_counted": 6859, + "counted": 19004 + } + ], + "units": 1, + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "89997", + "ref_energy": "77076760253" + }, + "vs_final": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "89997", + "ref_energy": "77076760253" + } + }, + { + "layers": 4, + "stage_counted": 6831, + "counted": 25835, + "per_base": [ + { + "stage_counted": 6831, + "counted": 25835 + } + ], + "units": 1, + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "0", + "ref_energy": "77076760253" + }, + "vs_final": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "0", + "ref_energy": "77076760253" + } + } + ], + "decoders": [ + { + "capability": "wai.audio.int_codec", + "pin": "98b382c92112660a7d3909f07da831125fcdb75a548aa0cf64ad17a0901bd9ef" + }, + { + "capability": "wai.audio.int_refine", + "pin": null + } + ], + "presentations": [ + { + "unit": 0, + "layers": 1, + "decoder": 0, + "shape": [ + 8192, + 1, + 1 + ], + "out_sha256": "b8ff83a442c112661ec7a41d6e51cb32559510a699a0e9bb37fa301f8952978a", + "vs_source": null, + "vs_final": null + }, + { + "unit": 0, + "layers": 2, + "decoder": 1, + "shape": [ + 16384, + 1, + 1 + ], + "out_sha256": "7cf4ee1daabc1f8779252bb3e02f16ba2ab2d8a65b374c8cbb1c3c3354121cdd", + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "5600413", + "ref_energy": "77076760253" + }, + "vs_final": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "5600413", + "ref_energy": "77076760253" + } + }, + { + "unit": 0, + "layers": 3, + "decoder": 1, + "shape": [ + 16384, + 1, + 1 + ], + "out_sha256": "02c3e3540210f1e5fce2b9f75d38e74d9b877cf17f2233b7825a8a10b823c64c", + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "89997", + "ref_energy": "77076760253" + }, + "vs_final": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "89997", + "ref_energy": "77076760253" + } + }, + { + "unit": 0, + "layers": 4, + "decoder": 1, + "shape": [ + 16384, + 1, + 1 + ], + "out_sha256": "8405e88c07e18cd93aecd578493cb3ff060c98be1790ebca0edeb0d1af1c512a", + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "0", + "ref_energy": "77076760253" + }, + "vs_final": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "0", + "ref_energy": "77076760253" + } + } + ], + "output_root": "cbefe0dba0775d039b833791d17f22c15649959a3f1e9cd84473fea68a58985c", + "leaf_count": 4, + "overhead": [ + { + "label": "staged", + "counted": 25835, + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "0", + "ref_energy": "77076760253" + }, + "out_sha256": [ + "8405e88c07e18cd93aecd578493cb3ff060c98be1790ebca0edeb0d1af1c512a" + ] + }, + { + "label": "one layer, q 1", + "counted": 32145, + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "0", + "ref_energy": "77076760253" + }, + "out_sha256": [ + "8405e88c07e18cd93aecd578493cb3ff060c98be1790ebca0edeb0d1af1c512a" + ] + }, + { + "label": "one layer, q 8", + "counted": 19137, + "vs_source": { + "kind": "pcm16", + "n_samples": 16384, + "sse": "89997", + "ref_energy": "77076760253" + }, + "out_sha256": [ + "9b34a3dcbc301e180a7a675b3304ed57ddee43858863e113a55455a926d614e8" + ] + } + ], + "dedup": { + "objects": 8, + "unique": 6, + "referenced_bytes": 77117, + "stored_bytes": 73529 + } + }, + { + "title": "clip", + "content_id": "0072fc77dc1dc516eb2baed8072c577351f0316086675043482e54e729f60b92", + "framing": "registered", + "unit_class": "rgb8", + "unit_us": 40000, + "bases": [ + { + "object": "85f50eb426d09ee3a5006544e10cbba572fc5e05cb62f3992bd095d9cb3c7e8f", + "capability": "wai.video.int_hyper", + "pin": "13e8b0d18e001eb9f0160309a02755b35a14f71ba0bcaa13f373e32b931ab818", + "tier": "decode-equivalence", + "first_unit": 0, + "capture_us": null, + "units": 5, + "param_set_bytes": 2676 + } + ], + "objects": [ + { + "file": "staged-conformance/cases/video_int_hyper_keyframe/base.wai", + "object": "85f50eb426d09ee3a5006544e10cbba572fc5e05cb62f3992bd095d9cb3c7e8f", + "capability": "wai.video.int_hyper", + "base": 0, + "layer": 0, + "units": [ + 0, + 5 + ], + "counted": { + "header": 12, + "manifest": 359, + "payload": 2148, + "total": 2519 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L01.wai", + "object": "431bfb81f4e5bda35152786d2a580d3378c5cfd6196d271a6cddd122e93f09f0", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 284, + "total": 647 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L02.wai", + "object": "836a7a4314b389304bf1e12113468b063e2853420e66ab41b4ac51ea42a97fd3", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 1, + "units": [ + 1, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 288, + "total": 651 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L03.wai", + "object": "3ba12c0683b76c4e1aec05de9bab726ad3c5e4ef5ab729c4610ef3e6ed1704fc", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 1, + "units": [ + 2, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 284, + "total": 647 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L04.wai", + "object": "9ebbe2a5888366479c4fe8f17f61cec3a0b9513953dee4ae5a35f0c25175840d", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 1, + "units": [ + 3, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 296, + "total": 659 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L05.wai", + "object": "8367e3e2a08e02cbd0c44edfc9b1265d653cd14cdd22a202fec6c8dd81fbd210", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 1, + "units": [ + 4, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 304, + "total": 667 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L06.wai", + "object": "dc49106d143d7ef3cc73ef3b286f411e863ba885c1db361012bef09966c9cf26", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 2, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 204, + "total": 567 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L07.wai", + "object": "8e8ce0262bb55022c8f9670985535f99804b3f364de631e04e117273f794f285", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 2, + "units": [ + 1, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 200, + "total": 563 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L08.wai", + "object": "a29103cc845996ca04557fca953b58401163940793d8a729ffc804688a70166a", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 2, + "units": [ + 2, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 196, + "total": 559 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L09.wai", + "object": "43e35ee383faa6a0b86c5c8270447635ebffe2a890818371f34fb6819ec0eda5", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 2, + "units": [ + 3, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 192, + "total": 555 + } + }, + { + "file": "staged-measure-conformance/fixtures/clip/L10.wai", + "object": "130c3ca0cba51a191e896e9e9da1a6eec1300bae5637a3390493f53b69a714b4", + "capability": "wai.video.int_refine", + "base": 0, + "layer": 2, + "units": [ + 4, + 1 + ], + "counted": { + "header": 12, + "manifest": 351, + "payload": 204, + "total": 567 + } + } + ], + "bfpf_min": 2519, + "prefixes": [ + { + "layers": 1, + "stage_counted": 2519, + "counted": 2519, + "per_base": [ + { + "stage_counted": 2519, + "counted": 2519 + } + ], + "units": 5, + "vs_source": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "189317", + "sse_y": "27660", + "ssim_y_q32": 4287926497, + "windows": 45 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "189317", + "sse_y": "27660", + "ssim_y_q32": 4287926497, + "windows": 45 + } + }, + { + "layers": 2, + "stage_counted": 3271, + "counted": 5790, + "per_base": [ + { + "stage_counted": 3271, + "counted": 5790 + } + ], + "units": 5, + "vs_source": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "5492", + "sse_y": "976", + "ssim_y_q32": 4294716641, + "windows": 45 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "5492", + "sse_y": "976", + "ssim_y_q32": 4294716641, + "windows": 45 + } + }, + { + "layers": 3, + "stage_counted": 2811, + "counted": 8601, + "per_base": [ + { + "stage_counted": 2811, + "counted": 8601 + } + ], + "units": 5, + "vs_source": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 45 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 45 + } + } + ], + "decoders": [ + { + "capability": "wai.video.int_hyper", + "pin": "13e8b0d18e001eb9f0160309a02755b35a14f71ba0bcaa13f373e32b931ab818" + }, + { + "capability": "wai.video.int_refine", + "pin": null + } + ], + "presentations": [ + { + "unit": 0, + "layers": 1, + "decoder": 0, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "dce7282d46f88f03d5dc0fd42f325756d310fe1fc982b6e1792a1fbfbeb482cf", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "39980", + "sse_y": "5688", + "ssim_y_q32": 4287319116, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "39980", + "sse_y": "5688", + "ssim_y_q32": 4287319116, + "windows": 9 + } + }, + { + "unit": 0, + "layers": 2, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "71639c87086d0fe69eb0e9ba8a976e882cb7ac38056a3c388d901dd3f0ac4fd2", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1160", + "sse_y": "210", + "ssim_y_q32": 4294692926, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1160", + "sse_y": "210", + "ssim_y_q32": 4294692926, + "windows": 9 + } + }, + { + "unit": 0, + "layers": 3, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "7655e2d7055f1ea34084785b13e0becbec58301023ee9beff3e75a11abb7b621", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + } + }, + { + "unit": 1, + "layers": 1, + "decoder": 0, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "eb22eaca420f922c3e65f7583e4ba15195a013a4694d3d704c1c393477279197", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "38833", + "sse_y": "5108", + "ssim_y_q32": 4288419105, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "38833", + "sse_y": "5108", + "ssim_y_q32": 4288419105, + "windows": 9 + } + }, + { + "unit": 1, + "layers": 2, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "5ca748b74ae00febec767d871f336c61cb1f67db1d0451a02112191f7f9c664f", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1096", + "sse_y": "196", + "ssim_y_q32": 4294722977, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1096", + "sse_y": "196", + "ssim_y_q32": 4294722977, + "windows": 9 + } + }, + { + "unit": 1, + "layers": 3, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "06e01123126a93ebd5f845883f66c959a57270662fc7982e32d0768637b48722", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + } + }, + { + "unit": 2, + "layers": 1, + "decoder": 0, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "904e9064df9f95582e2ac4b9ef594a27810922c325c5144acd4d17e08294ecdf", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "35388", + "sse_y": "5682", + "ssim_y_q32": 4287779398, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "35388", + "sse_y": "5682", + "ssim_y_q32": 4287779398, + "windows": 9 + } + }, + { + "unit": 2, + "layers": 2, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "804b060fcb5fe3ea6a1588c5768a4520eb25bb67df51cbdc4de392253bcfd387", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1056", + "sse_y": "160", + "ssim_y_q32": 4294758223, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1056", + "sse_y": "160", + "ssim_y_q32": 4294758223, + "windows": 9 + } + }, + { + "unit": 2, + "layers": 3, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "1e946bd550123521b8d6e67485c2058159e6b3d82b58f709ad23ceb223c6bb0c", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + } + }, + { + "unit": 3, + "layers": 1, + "decoder": 0, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "ba33b5f7ffe2dd451106b43596f7be854ba0a5e011039a66bc8f221b4321c2df", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "36106", + "sse_y": "5201", + "ssim_y_q32": 4288634741, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "36106", + "sse_y": "5201", + "ssim_y_q32": 4288634741, + "windows": 9 + } + }, + { + "unit": 3, + "layers": 2, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "14eb7bbe7d69932210f029c7319f19f2e911f43c2222b08fc345f9a04069264f", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1012", + "sse_y": "201", + "ssim_y_q32": 4294697080, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1012", + "sse_y": "201", + "ssim_y_q32": 4294697080, + "windows": 9 + } + }, + { + "unit": 3, + "layers": 3, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "7b577471db6e6c1b9f301c34470678407d6c6d74e44d89ca6152d1c12f177fe6", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + } + }, + { + "unit": 4, + "layers": 1, + "decoder": 0, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "38ccbd2a62233c72087b136588551fdb742ae98a1e78a5e4f6b68c25b7778e3b", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "39010", + "sse_y": "5981", + "ssim_y_q32": 4287480124, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "39010", + "sse_y": "5981", + "ssim_y_q32": 4287480124, + "windows": 9 + } + }, + { + "unit": 4, + "layers": 2, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "0261a1b9ae00788df786e4cc542d36629cff822c52195ce8be5c12d33ff66905", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1168", + "sse_y": "209", + "ssim_y_q32": 4294711997, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "1168", + "sse_y": "209", + "ssim_y_q32": 4294711997, + "windows": 9 + } + }, + { + "unit": 4, + "layers": 3, + "decoder": 1, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "6d6ff21e4b9654c757a98fafc949e760e01ea22953f201affd4a738a15be6726", + "vs_source": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 768, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 9 + } + } + ], + "output_root": "c372d238be5ce75f44c296045cbf7b39009f72556e246bb20b9b68f576867261", + "leaf_count": 15, + "overhead": [ + { + "label": "staged", + "counted": 8601, + "vs_source": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 45 + }, + "out_sha256": [ + "7655e2d7055f1ea34084785b13e0becbec58301023ee9beff3e75a11abb7b621", + "06e01123126a93ebd5f845883f66c959a57270662fc7982e32d0768637b48722", + "1e946bd550123521b8d6e67485c2058159e6b3d82b58f709ad23ceb223c6bb0c", + "7b577471db6e6c1b9f301c34470678407d6c6d74e44d89ca6152d1c12f177fe6", + "6d6ff21e4b9654c757a98fafc949e760e01ea22953f201affd4a738a15be6726" + ] + }, + { + "label": "one layer, q 2", + "counted": 6454, + "vs_source": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 45 + }, + "out_sha256": [ + "7655e2d7055f1ea34084785b13e0becbec58301023ee9beff3e75a11abb7b621", + "06e01123126a93ebd5f845883f66c959a57270662fc7982e32d0768637b48722", + "1e946bd550123521b8d6e67485c2058159e6b3d82b58f709ad23ceb223c6bb0c", + "7b577471db6e6c1b9f301c34470678407d6c6d74e44d89ca6152d1c12f177fe6", + "6d6ff21e4b9654c757a98fafc949e760e01ea22953f201affd4a738a15be6726" + ] + }, + { + "label": "one layer, q 6", + "counted": 5790, + "vs_source": { + "kind": "rgb8", + "n_samples": 3840, + "sse_rgb": "5492", + "sse_y": "976", + "ssim_y_q32": 4294716641, + "windows": 45 + }, + "out_sha256": [ + "71639c87086d0fe69eb0e9ba8a976e882cb7ac38056a3c388d901dd3f0ac4fd2", + "5ca748b74ae00febec767d871f336c61cb1f67db1d0451a02112191f7f9c664f", + "804b060fcb5fe3ea6a1588c5768a4520eb25bb67df51cbdc4de392253bcfd387", + "14eb7bbe7d69932210f029c7319f19f2e911f43c2222b08fc345f9a04069264f", + "0261a1b9ae00788df786e4cc542d36629cff822c52195ce8be5c12d33ff66905" + ] + } + ], + "dedup": { + "objects": 23, + "unique": 16, + "referenced_bytes": 20845, + "stored_bytes": 12536 + } + }, + { + "title": "splat", + "content_id": "41837ee65bb05f40907d3e20a7d9602d7e8b9e73a92bc1de80116fd99cc47a87", + "framing": "registered", + "unit_class": "tensor-i64", + "unit_us": 40000, + "bases": [ + { + "object": "bc283dfb0eb717c129db5a1cadad67daefc788e4f6880ecf4c7ac8a2d35204d8", + "capability": "wai.splat.int_codec", + "pin": "033a37c9d29d7ea9675a0b4a3e269d74f2ab874b5a3b1425eec69bf3bad8b26f", + "tier": "decode-equivalence", + "first_unit": 0, + "capture_us": null, + "units": 1, + "param_set_bytes": 78763 + } + ], + "objects": [ + { + "file": "staged-conformance/cases/splat_attr_1l/base.wai", + "object": "bc283dfb0eb717c129db5a1cadad67daefc788e4f6880ecf4c7ac8a2d35204d8", + "capability": "wai.splat.int_codec", + "base": 0, + "layer": 0, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 343, + "payload": 2964, + "total": 3319 + } + }, + { + "file": "staged-measure-conformance/fixtures/splat/L01.wai", + "object": "de3eb7d9fb70a8f7f745b572a164c3edc49f474cc0403616e6a90c2b122fa5ff", + "capability": "wai.splat.int_refine", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 328, + "payload": 4800, + "total": 5140 + } + }, + { + "file": "staged-measure-conformance/fixtures/splat/L02.wai", + "object": "d041bed2c68c8f943051a11a51ff8b7595f751f83e452d179bfb23b8b31bfe5a", + "capability": "wai.splat.int_refine", + "base": 0, + "layer": 2, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 328, + "payload": 3040, + "total": 3380 + } + } + ], + "bfpf_min": 3319, + "prefixes": [ + { + "layers": 1, + "stage_counted": 3319, + "counted": 3319, + "per_base": [ + { + "stage_counted": 3319, + "counted": 3319 + } + ], + "units": 1, + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "1197651", + "max_abs": 16 + }, + "vs_final": { + "kind": "tensor-i64", + "n": 14336, + "sse": "1197651", + "max_abs": 16 + } + }, + { + "layers": 2, + "stage_counted": 5140, + "counted": 8459, + "per_base": [ + { + "stage_counted": 5140, + "counted": 8459 + } + ], + "units": 1, + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "35867", + "max_abs": 2 + }, + "vs_final": { + "kind": "tensor-i64", + "n": 14336, + "sse": "35867", + "max_abs": 2 + } + }, + { + "layers": 3, + "stage_counted": 3380, + "counted": 11839, + "per_base": [ + { + "stage_counted": 3380, + "counted": 11839 + } + ], + "units": 1, + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "0", + "max_abs": 0 + }, + "vs_final": { + "kind": "tensor-i64", + "n": 14336, + "sse": "0", + "max_abs": 0 + } + } + ], + "decoders": [ + { + "capability": "wai.splat.int_codec", + "pin": "033a37c9d29d7ea9675a0b4a3e269d74f2ab874b5a3b1425eec69bf3bad8b26f" + }, + { + "capability": "wai.splat.int_refine", + "pin": null + } + ], + "presentations": [ + { + "unit": 0, + "layers": 1, + "decoder": 0, + "shape": [ + 14, + 32, + 32 + ], + "out_sha256": "f07679de87ff7c3d4fc4890b80919ef857fb85bd4d9e4cfa16105211d580427f", + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "1197651", + "max_abs": 16 + }, + "vs_final": { + "kind": "tensor-i64", + "n": 14336, + "sse": "1197651", + "max_abs": 16 + } + }, + { + "unit": 0, + "layers": 2, + "decoder": 1, + "shape": [ + 14, + 32, + 32 + ], + "out_sha256": "4603b50563302461ce2baeaa1810268c9ebe834599f62790d7acf01da33c66fd", + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "35867", + "max_abs": 2 + }, + "vs_final": { + "kind": "tensor-i64", + "n": 14336, + "sse": "35867", + "max_abs": 2 + } + }, + { + "unit": 0, + "layers": 3, + "decoder": 1, + "shape": [ + 14, + 32, + 32 + ], + "out_sha256": "9a177a9b1879bc66804f7c96d98755d39ee16aca5d8e042b31706aff4c539bee", + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "0", + "max_abs": 0 + }, + "vs_final": { + "kind": "tensor-i64", + "n": 14336, + "sse": "0", + "max_abs": 0 + } + } + ], + "output_root": "78b1c64959dddb38ddd8efd963bffd5fe0517084e12259d929db388e9717b2be", + "leaf_count": 3, + "overhead": [ + { + "label": "staged", + "counted": 11839, + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "0", + "max_abs": 0 + }, + "out_sha256": [ + "9a177a9b1879bc66804f7c96d98755d39ee16aca5d8e042b31706aff4c539bee" + ] + }, + { + "label": "one layer, q 1", + "counted": 11283, + "vs_source": { + "kind": "tensor-i64", + "n": 14336, + "sse": "0", + "max_abs": 0 + }, + "out_sha256": [ + "9a177a9b1879bc66804f7c96d98755d39ee16aca5d8e042b31706aff4c539bee" + ] + } + ], + "dedup": { + "objects": 5, + "unique": 4, + "referenced_bytes": 23122, + "stored_bytes": 19803 + } + }, + { + "title": "still", + "content_id": "e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277", + "framing": "registered", + "unit_class": "rgb8", + "unit_us": 40000, + "bases": [ + { + "object": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "capability": "wai.neural.int_hyper", + "pin": "13e8b0d18e001eb9f0160309a02755b35a14f71ba0bcaa13f373e32b931ab818", + "tier": "decode-equivalence", + "first_unit": 0, + "capture_us": null, + "units": 1, + "param_set_bytes": 2676 + } + ], + "objects": [ + { + "file": "staged-conformance/cases/image_int_hyper_2l/base.wai", + "object": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "capability": "wai.neural.int_hyper", + "base": 0, + "layer": 0, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 360, + "payload": 88, + "total": 460 + } + }, + { + "file": "staged-measure-conformance/fixtures/still/L01.wai", + "object": "e35e4888e6dbca631928480d0f3e4c5404f337ed7fc4cfeab8b54bdb46968d05", + "capability": "wai.image.int_refine", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 323, + "payload": 616, + "total": 951 + } + }, + { + "file": "staged-measure-conformance/fixtures/still/L02.wai", + "object": "b8f85108e6f3eb6b2671ddb326aa2fd22ca634187d904dff6ea9dc0d67fb6b92", + "capability": "wai.image.int_refine", + "base": 0, + "layer": 2, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 323, + "payload": 612, + "total": 947 + } + }, + { + "file": "staged-measure-conformance/fixtures/still/L03.wai", + "object": "469e20139ecc596d8aa1fb210e05d60fea605d63ee294ac099cf6eb6329ceb37", + "capability": "wai.image.int_refine", + "base": 0, + "layer": 3, + "units": [ + 0, + 1 + ], + "counted": { + "header": 12, + "manifest": 323, + "payload": 608, + "total": 943 + } + } + ], + "bfpf_min": 460, + "prefixes": [ + { + "layers": 1, + "stage_counted": 460, + "counted": 460, + "per_base": [ + { + "stage_counted": 460, + "counted": 460 + } + ], + "units": 1, + "vs_source": null, + "vs_final": null + }, + { + "layers": 2, + "stage_counted": 951, + "counted": 1411, + "per_base": [ + { + "stage_counted": 951, + "counted": 1411 + } + ], + "units": 1, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "11560", + "sse_y": "1659", + "ssim_y_q32": 4293926148, + "windows": 49 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "11560", + "sse_y": "1659", + "ssim_y_q32": 4293926148, + "windows": 49 + } + }, + { + "layers": 3, + "stage_counted": 947, + "counted": 2358, + "per_base": [ + { + "stage_counted": 947, + "counted": 2358 + } + ], + "units": 1, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "1267", + "sse_y": "326", + "ssim_y_q32": 4294755931, + "windows": 49 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "1267", + "sse_y": "326", + "ssim_y_q32": 4294755931, + "windows": 49 + } + }, + { + "layers": 4, + "stage_counted": 943, + "counted": 3301, + "per_base": [ + { + "stage_counted": 943, + "counted": 3301 + } + ], + "units": 1, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 49 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 49 + } + } + ], + "decoders": [ + { + "capability": "wai.neural.int_hyper", + "pin": "13e8b0d18e001eb9f0160309a02755b35a14f71ba0bcaa13f373e32b931ab818" + }, + { + "capability": "wai.image.int_refine", + "pin": null + } + ], + "presentations": [ + { + "unit": 0, + "layers": 1, + "decoder": 0, + "shape": [ + 16, + 16, + 3 + ], + "out_sha256": "dce7282d46f88f03d5dc0fd42f325756d310fe1fc982b6e1792a1fbfbeb482cf", + "vs_source": null, + "vs_final": null + }, + { + "unit": 0, + "layers": 2, + "decoder": 1, + "shape": [ + 32, + 32, + 3 + ], + "out_sha256": "827c831aef7df8d700616cd896dcb5d2cde2a926afaea503d85487026836a34a", + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "11560", + "sse_y": "1659", + "ssim_y_q32": 4293926148, + "windows": 49 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "11560", + "sse_y": "1659", + "ssim_y_q32": 4293926148, + "windows": 49 + } + }, + { + "unit": 0, + "layers": 3, + "decoder": 1, + "shape": [ + 32, + 32, + 3 + ], + "out_sha256": "952ebbf5a937f327c0d056684ed88fef9d9489be4735a456ae8ecdd2d299ace6", + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "1267", + "sse_y": "326", + "ssim_y_q32": 4294755931, + "windows": 49 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "1267", + "sse_y": "326", + "ssim_y_q32": 4294755931, + "windows": 49 + } + }, + { + "unit": 0, + "layers": 4, + "decoder": 1, + "shape": [ + 32, + 32, + 3 + ], + "out_sha256": "fef81da98166b22c30e66ce6dda9a835aa95a7d1aa278c6046dd01a8c56f0efd", + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 49 + }, + "vs_final": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 49 + } + } + ], + "output_root": "4afc5e26ef90233e520d7c5e44273cbc5a36d329a2bf41c48a09d7457f4c0fee", + "leaf_count": 4, + "overhead": [ + { + "label": "staged", + "counted": 3301, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 49 + }, + "out_sha256": [ + "fef81da98166b22c30e66ce6dda9a835aa95a7d1aa278c6046dd01a8c56f0efd" + ] + }, + { + "label": "one layer, q 1", + "counted": 2311, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "0", + "sse_y": "0", + "ssim_y_q32": 4294967296, + "windows": 49 + }, + "out_sha256": [ + "fef81da98166b22c30e66ce6dda9a835aa95a7d1aa278c6046dd01a8c56f0efd" + ] + }, + { + "label": "one layer, q 3", + "counted": 1823, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "1538", + "sse_y": "382", + "ssim_y_q32": 4294724670, + "windows": 49 + }, + "out_sha256": [ + "8181c12af11507de87344cbf5a3f9217025c4f2f946ac1a12011e531806e449d" + ] + }, + { + "label": "one layer, q 8", + "counted": 1411, + "vs_source": { + "kind": "rgb8", + "n_samples": 3072, + "sse_rgb": "11560", + "sse_y": "1659", + "ssim_y_q32": 4293926148, + "windows": 49 + }, + "out_sha256": [ + "827c831aef7df8d700616cd896dcb5d2cde2a926afaea503d85487026836a34a" + ] + } + ], + "dedup": { + "objects": 10, + "unique": 6, + "referenced_bytes": 8846, + "stored_bytes": 6515 + } + } + ], + "record": { + "dedup": { + "objects": 46, + "unique": 32, + "referenced_bytes": 129930, + "stored_bytes": 112383 + } + } +} diff --git a/wai/staged-measure-conformance/measure_verify.py b/wai/staged-measure-conformance/measure_verify.py new file mode 100755 index 000000000..f6d7a2416 --- /dev/null +++ b/wai/staged-measure-conformance/measure_verify.py @@ -0,0 +1,660 @@ +#!/usr/bin/env python3 +"""Independent verifier of the staged-measurement corpus (extensions/staged-measurement.md). + +Written from the extension text and SPEC §2, §7. It shares no code with wai-rs, which +writes the corpus. It takes BLAKE3 from ../claim-conformance/_blake3.py, and the +WAI1 reader, `refines`, the WIR1 parse, its stream checks and its decode (the three +predictors) from ../staged-conformance/staged_verify.py, and the integer payload +parsers from ../int-payload-conformance/int_payload_verify.py, all written from their +specifications. It implements the rest itself: + +- the byte rule (§3.1): a WAI1 envelope is 12 + man_len + P bytes and a WAI2 envelope + 8 + man_len + 2 + 8n + Σ P, and a payload is counted only where WAI checks its + capability's format (§3.2): WIR1 for a refinement, the pinned integer formats, by + their parsers, the capability's and a checked fallback's; anything else is + `no-format-check`, and a WAI2 component whose sha256 is not its payload's is + `sha256-mismatch`; every case of counting.json is required; +- each base's stage 0 from its staged-conformance case (whose stage0.bin + staged_verify.py checks against the vectors' sidecars), and every layer applied in + layer order, each to every unit it covers, with its predecessor checked; +- the quality integers (§4): integer BT.601 luma, SSE, the Q32 structural similarity + over clipped 8×8 windows at stride 4 with an edge-aligned last window on each axis + (Python's // floors toward −∞, as the method states), PCM and tensor sums, each pooled over several units; +- the output tree (Appendix C): RFC 6962's tree hash over SHA-256 leaves; +- the prefixes' counted bytes, first presentable, the layering overhead's points, + deduplication (§9), and the title's content identity. + +Before it uses a base it runs its own base checks, in the order the measurement +refuses: SPEC §4's dispatch (a refinement-only or companion-only capability makes the +envelope inert, `missing_capability`, and its fallback is never reached); the figures a +payload's header fixes, its output bytes and its symbols, against the title's cost +limit (`cost_over_limit`; the working bytes and multiply-accumulates need the +parameter set's networks, which it does not read); and the units' bytes against the +title's chain budget (`chain-over-limit`). A title's limits are its `limits` member's, +within the tools' stated defaults: each figure the lower of the two (staged-measurement +Appendix D). + +For every fixture it requires its own decode's digests to equal the fixture tool's +(expect.json), and its whole evaluation to equal the fixture's block of goldens.json; +it requires the record-wide dedup totals; it requires §4.2's integers for every +picture pair of quality.json; and it requires every title under `refusals/` to be +refused, by its own checks, with the code the title states. + + python3 measure_verify.py + +Exit status 0 only if every fixture verifies. +""" +import hashlib +import json +import os +import struct +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +WAI = os.path.normpath(os.path.join(HERE, "..")) +sys.path.insert(0, os.path.join(WAI, "claim-conformance")) +sys.path.insert(0, os.path.join(WAI, "int-payload-conformance")) +sys.path.insert(0, os.path.join(WAI, "staged-conformance")) +from _blake3 import blake3 # noqa: E402 +import int_payload_verify as IPV # noqa: E402 +import staged_verify as SV # noqa: E402 + +# SPEC §5: the media class of each base capability the corpus uses, which fixes its +# quality kind (§4.2–§4.4). +UNIT_CLASS = { + "wai.neural.int_hyper": "rgb8", + "wai.video.int_hyper": "rgb8", + "wai.video.int_motion": "rgb8", + "wai.audio.int_codec": "pcm16", + "wai.splat.int_codec": "tensor-i64", +} +KIND = {"rgb8": "rgb8", "pcm16": "pcm", "tensor-i64": "attr"} + +# staged-measurement Appendix D: the tools' stated limits, and the order Rust's +# DeclaredCost::exceeds compares the figures in. +TOOL_COST = [1 << 25, 1 << 25, 1 << 28, 1 << 38] +TOOL_CHAIN_BUDGET = 1 << 28 +INT_PATH = set(IPV.CAP_MAGIC) | {"wai.neural.int_synth", "wai.neural.int_mlicpp"} + + +class Refusal(Exception): + def __init__(self, code, detail): + super().__init__(f"{code}: {detail}") + self.code = code + + +def companion_only(cap): + # SPEC §5: the enhancement layer and the HDR metadata descriptors. + return cap == "wai.video.lcevc" or cap.startswith("wai.meta.") + + +def limits_of(t): + """The limits a title is measured under: each figure the lower of the title's and + the tools' stated one. A title lowers a limit and never raises one.""" + l = t.get("limits") or {} + cost = [min(a, b) for a, b in zip(l.get("cost") or TOOL_COST, TOOL_COST)] + budget = l.get("chain_budget") + return cost, TOOL_CHAIN_BUDGET if budget is None else min(budget, TOOL_CHAIN_BUDGET) + + +def header_cost(cap, payload): + """The output bytes and symbols a payload's header fixes (integer-payloads §7); + None for a figure the header alone does not fix.""" + v = IPV.parse(cap, payload) + magic = IPV.CAP_MAGIC[cap] + if magic == b"WIV1": + frame = v["h"] * v["w"] * 3 + if v["keyframe"] == 0: + return frame * v["n_frames"], frame * v["n_frames"] + k = IPV.parse_wih1(v["wih1"]) + return frame * v["n_frames"], frame * (v["n_frames"] - 1) + k["n_z"] * k["zh"] * k["zw"] + k["m"] * k["sh"] * k["sw"] + if magic == b"WIH1": + return None, v["n_z"] * v["zh"] * v["zw"] + v["m"] * v["sh"] * v["sw"] + if magic == b"WIA1": + return 2 * v["t_out"], v["n_ch"] * v["lat_len"] + return v["attr"] * v["s"] * v["s"], v["c"] * v["sh"] * v["sw"] + + +def base_checks(env, cost): + """SPEC §4's dispatch and the header's cost: the capability a base decodes through, + or a Refusal.""" + m, _, payload = SV.read_wai1(env) + cap = m["model_requirement"]["capability"] + fb = m["model_requirement"].get("fallback") + if cap in SV.UNIT_KIND or companion_only(cap): + raise Refusal("missing_capability", f"{cap} is never dispatched; its fallback is not reached") + for via in [cap, fb]: + if via is None or via not in INT_PATH: + continue + if via in IPV.CAP_MAGIC: + out, sym = header_cost(via, payload) + if (out is not None and out > cost[0]) or sym > cost[1]: + # A cost past the limit makes the capability unsupported; the fallback is + # tried next (SPEC §4). + continue + return via + if cap in IPV.CAP_MAGIC: + raise Refusal("cost_over_limit", "the header declares a decode past the limit") + raise Refusal("missing_capability", "no capability this decode takes") + + +def sha(b): + return hashlib.sha256(b).hexdigest() + + +def read(rel): + with open(os.path.join(WAI, rel), "rb") as f: + return f.read() + + +# ---------------------------------------------------------------- §3, the byte rule + +class NotCountable(Exception): + def __init__(self, code, component=None): + super().__init__(code) + self.code = code + self.component = component + + +def has_format_check(cap): + return cap in IPV.CAP_MAGIC or cap in SV.UNIT_KIND + + +def format_check(cap, payload): + """§3.2: a payload counts only in a format WAI checks for its capability.""" + if not has_format_check(cap): + raise NotCountable("no-format-check") + try: + if cap in SV.UNIT_KIND: + SV.parse_wir1(payload, SV.UNIT_KIND[cap]) + else: + IPV.parse(cap, payload) + except (SV.Refused, IPV.Refused) as e: + raise NotCountable(e.code) + + +def count_any(b): + """The counted bytes of a WAI1 or WAI2 envelope (§3.1), or NotCountable with the + code: `not-an-envelope`, `encrypted`, a format's code or `no-format-check` (the + capability's, then a WAI1 fallback's that WAI checks), `sha256-mismatch` (a WAI2 + component), `trailing`.""" + if b[:4] == b"WAI1": + try: + m, text, payload = SV.read_wai1(b) + except SV.NotWai1: + raise NotCountable("not-an-envelope") + if not SV.plaintext(m): + raise NotCountable("encrypted") + mr = m["model_requirement"] + format_check(mr["capability"], payload) + fb = mr.get("fallback") + if fb is not None and has_format_check(fb): + format_check(fb, payload) + header, man, pl = 12, len(text), len(payload) + elif b[:4] == b"WAI2": + try: + text, entries, payloads = SV.read_wai2(b) + except SV.NotWai2: + raise NotCountable("not-an-envelope") + sec = json.loads(text).get("security") + if sec is not None and sec.get("encryption") not in (None, "none"): + raise NotCountable("encrypted") + for i, (e, p) in enumerate(zip(entries, payloads)): + if e.get("sha256") is not None and e["sha256"] != sha(p): + raise NotCountable("sha256-mismatch", i) + try: + format_check(e["capability"], p) + except NotCountable as x: + raise NotCountable(x.code, i) + header, man, pl = 8 + 2 + 8 * len(entries), len(text.encode()), sum(len(p) for p in payloads) + else: + raise NotCountable("not-an-envelope") + total = header + man + pl + if total != len(b): + raise NotCountable("trailing") + return {"header": header, "manifest": man, "payload": pl, "total": total} + + +def count(env): + """A measured WAI1 object: its manifest, payload and counted bytes.""" + counted = count_any(env) + m, _, payload = SV.read_wai1(env) + return m, payload, counted + + +# ---------------------------------------------------------------- §4, quality + +def luma(px): + return [(77 * px[i] + 150 * px[i + 1] + 29 * px[i + 2] + 128) >> 8 for i in range(0, len(px), 3)] + + +def starts(length, win): + """A window's offsets along an axis: every multiple of 4 up to length - win, then + length - win itself when it is not one, so every pixel is in a window.""" + last = length - win + return list(range(0, last + 1, 4)) + ([last] if last % 4 else []) + + +def image_parts(h, w, a, b): + sse_rgb = sum((x - y) ** 2 for x, y in zip(a, b)) + ya, yb = luma(a), luma(b) + sse_y = sum((x - y) ** 2 for x, y in zip(ya, yb)) + ww, wh = min(8, w), min(8, h) + n = ww * wh + tot = cnt = 0 + for y0 in starts(h, wh): + for x0 in starts(w, ww): + sx = sy = sxx = syy = sxy = 0 + for yy in range(y0, y0 + wh): + row = yy * w + for xx in range(x0, x0 + ww): + p, q = ya[row + xx], yb[row + xx] + sx += p + sy += q + sxx += p * p + syy += q * q + sxy += p * q + a1 = 20000 * sx * sy + 65025 * n * n + a2 = 20000 * (n * sxy - sx * sy) + 585225 * n * n + b1 = 10000 * (sx * sx + sy * sy) + 65025 * n * n + b2 = 10000 * ((n * sxx - sx * sx) + (n * syy - sy * sy)) + 585225 * n * n + tot += (a1 * a2 * (1 << 32)) // (b1 * b2) + cnt += 1 + return len(a), sse_rgb, sse_y, tot, cnt + + +def picture(w, h, seed, mul): + """quality.json's picture: byte c of pixel (x, y).""" + return [(x * mul + y * (mul + 3) + 29 * c + 17 * seed + 5 * ((x * y + seed) % 13)) % 256 + for y in range(h) for x in range(w) for c in range(3)] + + +def s16(b): + return list(struct.unpack(f"<{len(b) // 2}h", b)) + + +def pooled(cls, pairs): + """pairs: (test samples, test shape, reference canonical bytes, reference shape). + Pooled over the pairs of equal shape; None when there is none.""" + pairs = [(t, s, r) for t, s, r, rs in pairs if s == rs] + if not pairs: + return None + if cls == "rgb8": + n = sr = sy = tot = cnt = 0 + for t, s, r in pairs: + a, b, c, d, e = image_parts(s[0], s[1], list(r), t) + n, sr, sy, tot, cnt = n + a, sr + b, sy + c, tot + d, cnt + e + return {"kind": "rgb8", "n_samples": n, "sse_rgb": str(sr), "sse_y": str(sy), "ssim_y_q32": tot // cnt, "windows": cnt} + if cls == "pcm16": + n = e = en = 0 + for t, s, r in pairs: + ref = s16(r) + n += len(ref) + e += sum((x - y) ** 2 for x, y in zip(ref, t)) + en += sum(x * x for x in ref) + return {"kind": "pcm16", "n_samples": n, "sse": str(e), "ref_energy": str(en)} + n = e = mx = 0 + for t, s, r in pairs: + n += len(r) + for x, y in zip(r, t): + d = abs(x - y) + e += d * d + mx = max(mx, d) + return {"kind": "tensor-i64", "n": n, "sse": str(e), "max_abs": mx} + + +def canonical(cls, t): + return struct.pack(f"<{len(t)}h", *t) if cls == "pcm16" else bytes(t) + + +# ---------------------------------------------------------------- Appendix C + +def leaf(unit, layers, dec, digest): + return hashlib.sha256(b"wai:out-leaf\x01" + struct.pack(">QHH", unit, layers, dec) + bytes.fromhex(digest)).digest() + + +def mth(d): + if len(d) == 1: + return d[0] + k = 1 + while k * 2 < len(d): + k *= 2 + return hashlib.sha256(b"wai:out-node\x01" + mth(d[:k]) + mth(d[k:])).digest() + + +# ---------------------------------------------------------------- a title + +class Fixture: + def __init__(self, corpus_rel, name, sub="fixtures"): + self.dir = f"{corpus_rel}/{sub}/{name}" + self.t = json.loads(read(f"{self.dir}/title.json")) + if self.t["format"] != "wai-measure-title/1": + raise ValueError("title format") + self.read = {} # path -> bytes, every file the content identity covers + + def obj(self, rel, measured=True): + b = read(rel) + if measured: + self.read[rel] = b + return b + + +def replay(bases, layers, cost=None): + """bases: list of dicts (hash, cap, pin, first, units: [(shape, samples)]); + layers: list of (path, env). Returns per unit (global, base index) the list of + stages (layers count, capability, pin, shape, samples).""" + traces = {} + producer = {} + for bi, b in enumerate(bases): + for u, (shape, s) in enumerate(b["units"]): + g = b["first"] + u + traces[g] = [(1, b["cap"], b["pin"], shape, s)] + producer[g] = b["hash"] + parsed = [] + for path, env in layers: + m, _, payload = SV.read_wai1(env) + cap = m["model_requirement"]["capability"] + ref = SV.parse_refines(m["refines"]) + bi = next(i for i, b in enumerate(bases) if b["hash"] == ref["base"]) + frm, cnt = SV.units_of(ref, len(bases[bi]["units"])) + w = SV.parse_wir1(payload, SV.UNIT_KIND[cap]) + out, sym, work = SV.cost(w) + if cost is not None and (out > cost[0] or sym > cost[1] or work > cost[2]): + raise Refusal("cost-over-limit", f"{path}: the layer's declared cost passes the limit") + SV.check_streams(w) + if w["n_units"] != cnt: + raise ValueError(f"{path}: n_units is not the span") + parsed.append((path, cap, ref, bi, frm, cnt, w, SV.content_hash(env))) + top = max([p[2]["layer"] for p in parsed], default=0) + for k in range(1, top + 1): + for path, cap, ref, bi, frm, cnt, w, h in parsed: + if ref["layer"] != k: + continue + for i in range(cnt): + g = bases[bi]["first"] + frm + i + stage = len(traces[g]) - 1 + want = SV.producer_for(ref, i) + if stage != k - 1 or producer[g] != want: + raise ValueError(f"{path}: unit {g} is not ready for layer {k}") + _, _, _, shape, s = traces[g][-1] + if not SV.accepts_below(w, shape): + raise ValueError(f"{path}: unit {g}'s shape") + out = SV.apply_unit(w, i, s, shape) + traces[g].append((k + 1, cap, None, w["d"], out)) + producer[g] = h + return [traces[g] for g in sorted(traces)] + + +def evaluate(f): + t = f.t + bases = [] + objects = [] + base_extra = [] + cost, budget = limits_of(t) + for bi, b in enumerate(t["bases"]): + if b.get("envelope") is not None: + # A base by its envelope alone: the corpus names one only to be refused. + env = f.obj(b["envelope"]) + base_checks(env, cost) + raise ValueError(f"{b['envelope']}: a base with no stage 0 to check passes the base checks") + case = json.loads(read(f"{b['case']}/case.json")) + cb = case["bases"][b["base"]] + path = f"{b['case']}/{cb['file']}" + env = f.obj(path) + via = base_checks(env, cost) + m, payload, counted = count(env) + cap = m["model_requirement"]["capability"] + if SV.content_hash(env) != cb["object"] or via != cb["decoded_via"] or (via != cap) != cb["via_fallback"]: + raise ValueError(f"{path}: not the case's base") + stage0 = read(f"{b['case']}/{cb['stage0']}") + if sha(stage0) != cb["stage0_sha256"]: + raise ValueError(f"{path}: stage0") + set_bytes = 0 + for _, sp in sorted(cb["decode"]["set"].items()): + set_bytes += len(f.obj(sp)) + units, at = [], 0 + for u in cb["units"]: + shape = tuple(u["shape"]) + n = shape[0] * shape[1] * shape[2] + if u["elem"] == "i16": + units.append((shape, s16(stage0[at:at + 2 * n]))) + at += 2 * n + else: + units.append((shape, list(stage0[at:at + n]))) + at += n + if sum(len(x) * (2 if u["elem"] == "i16" else 1) for x, u in zip((v for _, v in units), cb["units"])) > budget: + raise Refusal("chain-over-limit", "the base's units pass the chain budget") + prior = (m.get("conditioning") or {}).get("prior") + pin = prior["sha256"] if cb["pin_resolved"] and prior else None + bases.append({"hash": cb["object"], "cap": cap, "pin": pin, "first": b["first_unit"], "units": units}) + base_extra.append({"object": cb["object"], "capability": cap, "pin": pin, "tier": cb["tier"], + "first_unit": b["first_unit"], "capture_us": b["capture_us"], "units": len(units), + "param_set_bytes": set_bytes}) + objects.append({"file": path, "object": cb["object"], "capability": cap, "base": bi, "layer": 0, + "units": [b["first_unit"], len(units)], "counted": counted}) + cls = UNIT_CLASS[bases[0]["cap"]] + layers = [] + for name in t["layers"]: + path = f"{f.dir}/{name}" + env = f.obj(path) + m, _, counted = count(env) + ref = SV.parse_refines(m["refines"]) + bi = next(i for i, b in enumerate(bases) if b["hash"] == ref["base"]) + frm, cnt = SV.units_of(ref, len(bases[bi]["units"])) + objects.append({"file": path, "object": SV.content_hash(env), "capability": m["model_requirement"]["capability"], + "base": bi, "layer": ref["layer"], "units": [bases[bi]["first"] + frm, cnt], "counted": counted}) + layers.append((path, env)) + traces = replay(bases, layers, cost) + finals = [tr[-1] for tr in traces] + src = None + if t.get("source") is not None: + sb = f.obj(f"{f.dir}/{t['source']['file']}") + if sha(sb) != t["source"]["sha256"]: + raise ValueError("source sha256") + src, at = [], 0 + for fin in finals: + n = len(canonical(cls, fin[4])) + src.append(sb[at:at + n]) + at += n + if at != len(sb): + raise ValueError("source length") + # decoders, by first use as presentations run by (layers, unit) + order = sorted(((st[0], tr_i, s_i) for tr_i, tr in enumerate(traces) for s_i, st in enumerate(tr))) + decoders = [] + for _, tr_i, s_i in order: + st = traces[tr_i][s_i] + key = (st[1], st[2]) + if key not in decoders: + decoders.append(key) + units_g = sorted(bases[bi]["first"] + u for bi in range(len(bases)) for u in range(len(bases[bi]["units"]))) + presentations = [] + leaves = [] + for tr_i, tr in enumerate(traces): + fin = finals[tr_i] + fin_bytes = canonical(cls, fin[4]) + for st in tr: + digest = sha(canonical(cls, st[4])) + dec = decoders.index((st[1], st[2])) + presentations.append({ + "unit": units_g[tr_i], "layers": st[0], "decoder": dec, "shape": list(st[3]), "out_sha256": digest, + "vs_source": pooled(cls, [(st[4], st[3], src[tr_i], fin[3])]) if src is not None else None, + "vs_final": pooled(cls, [(st[4], st[3], fin_bytes, fin[3])]), + }) + leaves.append(((units_g[tr_i], st[0]), leaf(units_g[tr_i], st[0], dec, digest))) + leaves.sort() + root = mth([l for _, l in leaves]).hex() if leaves else None + top = max(o["layer"] for o in objects) + prefixes = [] + cum = [0] * len(bases) + for k in range(top + 1): + per_base = [] + for bi in range(len(bases)): + s = sum(o["counted"]["total"] for o in objects if o["base"] == bi and o["layer"] == k) + cum[bi] += s + per_base.append({"stage_counted": s, "counted": cum[bi]}) + at = [tr[min(k, len(tr) - 1)] for tr in traces] + prefixes.append({ + "layers": k + 1, + "stage_counted": sum(p["stage_counted"] for p in per_base), + "counted": sum(p["counted"] for p in per_base), + "per_base": per_base, + "units": sum(1 for tr in traces if len(tr) > k), + "vs_source": pooled(cls, [(st[4], st[3], src[i], finals[i][3]) for i, st in enumerate(at)]) if src is not None else None, + "vs_final": pooled(cls, [(st[4], st[3], canonical(cls, finals[i][4]), finals[i][3]) for i, st in enumerate(at)]), + }) + bfpf = next((o["counted"]["total"] for o in objects if o["layer"] == 0 and o["units"][0] == 0 and o["units"][1] >= 1), None) + overhead = [] + items = [(o["object"], o["counted"]["total"]) for o in objects] + base_total = sum(o["counted"]["total"] for o in objects if o["layer"] == 0) + ref_items = [] + if src is not None: + overhead.append({"label": "staged", "counted": sum(o["counted"]["total"] for o in objects), + "vs_source": pooled(cls, [(fin[4], fin[3], src[i], fin[3]) for i, fin in enumerate(finals)]), + "out_sha256": [sha(canonical(cls, fin[4])) for fin in finals]}) + for r in t["references"]: + rl = [] + counted = base_total + for name in r["layers"]: + path = f"{f.dir}/{name}" + env = f.obj(path) + counted += count(env)[2]["total"] + rl.append((path, env)) + ref_items.append((SV.content_hash(env), count(env)[2]["total"])) + ref_items.extend((o["object"], o["counted"]["total"]) for o in objects if o["layer"] == 0) + rt = replay(bases, rl, cost) + rf = [tr[-1] for tr in rt] + overhead.append({"label": r["label"], "counted": counted, + "vs_source": pooled(cls, [(x[4], x[3], src[i], finals[i][3]) for i, x in enumerate(rf)]), + "out_sha256": [sha(canonical(cls, x[4])) for x in rf]}) + items += ref_items + dedup = dedup_totals(items) + # §11.1's content identity: the measured content, by path. + h = b"wai:measure-title\x01" + for p in sorted(f.read): + b = f.read[p] + h += struct.pack(">Q", len(p)) + p.encode() + struct.pack(">Q", len(b)) + b + return { + "title": t["title"], + "content_id": blake3(h).hex(), + "framing": "registered", + "unit_class": cls, + "unit_us": t["unit_us"], + "bases": base_extra, + "objects": objects, + "bfpf_min": bfpf, + "prefixes": prefixes, + "decoders": [{"capability": c, "pin": p} for c, p in decoders], + "presentations": presentations, + "output_root": root, + "leaf_count": len(presentations), + "overhead": overhead, + "dedup": dedup, + }, items + + +def dedup_totals(items): + seen = {} + d = {"objects": 0, "unique": 0, "referenced_bytes": 0, "stored_bytes": 0} + for h, n in items: + d["objects"] += 1 + d["referenced_bytes"] += n + if h in seen: + if seen[h] != n: + raise ValueError(f"object {h} has two lengths") + else: + seen[h] = n + d["unique"] += 1 + d["stored_bytes"] += n + return d + + +def diff(a, b, at, out): + """Where two JSON values differ, for the report.""" + if type(a) is not type(b) or (not isinstance(a, (dict, list)) and a != b): + out.append(f"{at}: {json.dumps(a)[:80]} != {json.dumps(b)[:80]}") + elif isinstance(a, dict): + for k in sorted(set(a) | set(b)): + diff(a.get(k), b.get(k), f"{at}.{k}", out) + elif isinstance(a, list): + if len(a) != len(b): + out.append(f"{at}: {len(a)} items != {len(b)}") + for i, (x, y) in enumerate(zip(a, b)): + diff(x, y, f"{at}[{i}]", out) + + +def main(d): + d = os.path.abspath(d) + rel = os.path.relpath(d, WAI) + goldens = json.load(open(os.path.join(d, "goldens.json"))) + names = sorted(n for n in os.listdir(os.path.join(d, "fixtures")) if os.path.isfile(os.path.join(d, "fixtures", n, "title.json"))) + fails = [] + got_titles = [g["title"] for g in goldens["fixtures"]] + if got_titles != names: + fails.append(f"goldens.json covers {got_titles}; the fixtures are {names}") + all_items = [] + for name in names: + f = Fixture(rel, name) + try: + ev, items = evaluate(f) + except (ValueError, Refusal, NotCountable, SV.Refused, SV.Malformed, IPV.Refused, KeyError, StopIteration) as e: + fails.append(f"{name}: {e!r}") + continue + all_items += items + exp = json.load(open(os.path.join(d, "fixtures", name, f.t["expect"]))) + mine = [{"unit": p["unit"], "layers": p["layers"], "out_sha256": p["out_sha256"]} for p in ev["presentations"]] + if exp["presentations"] != mine: + fails.append(f"{name}: this decode is not the fixture tool's reconstruction (expect.json)") + refs = [{"label": o["label"], "out_sha256": o["out_sha256"]} for o in ev["overhead"] if o["label"] != "staged"] + if exp["references"] != refs: + fails.append(f"{name}: the references' decodes are not the fixture tool's reconstruction") + g = next((x for x in goldens["fixtures"] if x["title"] == name), None) + out = [] + diff(ev, g, name, out) + fails += out[:20] + print(f" {name}: {len(ev['objects'])} objects, {ev['leaf_count']} presentations, root {ev['output_root']}" + f"{'' if not out else ' — DIFFERS'}") + # The counting cases: each object's counted bytes, or its code, as counting.json + # states them. + for c in json.load(open(os.path.join(d, "counting.json")))["cases"]: + b = read(c["file"]) + try: + got = {"counted": count_any(b), "refused": None, "component": None} + except NotCountable as e: + got = {"counted": None, "refused": e.code, "component": e.component} + want = {k: c[k] for k in ("counted", "refused", "component")} + if got != want: + fails.append(f"counting {c['file']}: {got} != {want}") + # The quality cases: §4.2's integers for each picture pair quality.json states. + for c in json.load(open(os.path.join(d, "quality.json")))["cases"]: + w, h = c["w"], c["h"] + n, sse, ssey, tot, cnt = image_parts(h, w, picture(w, h, *c["reference"]), picture(w, h, *c["test"])) + got = {"n_samples": n, "sse_rgb": str(sse), "sse_y": str(ssey), "ssim_y_q32": tot // cnt, "windows": cnt} + want = {k: c[k] for k in got} + if got != want: + fails.append(f"quality {w}x{h}: {got} != {want}") + refusals = os.path.join(d, "refusals") + for name in sorted(os.listdir(refusals)) if os.path.isdir(refusals) else []: + f = Fixture(rel, name, "refusals") + want = f.t.get("refused") + try: + evaluate(f) + fails.append(f"refusals/{name}: evaluated, though it must be refused {want}") + except Refusal as e: + if e.code != want: + fails.append(f"refusals/{name}: refused {e.code}, the title states {want}") + else: + print(f" refusals/{name}: refused {e.code}") + except (ValueError, SV.Refused, SV.Malformed, SV.NotWai1, IPV.Refused, KeyError) as e: + fails.append(f"refusals/{name}: {e!r}") + rec = dedup_totals(all_items) + if goldens["record"]["dedup"] != rec: + fails.append(f"record dedup {rec} != goldens {goldens['record']['dedup']}") + if fails: + for x in fails: + print("FAIL", x) + sys.exit(1) + print("staged-measure: every fixture verifies independently") + + +if __name__ == "__main__": + main(sys.argv[1] if len(sys.argv) > 1 else HERE) diff --git a/wai/staged-measure-conformance/quality.json b/wai/staged-measure-conformance/quality.json new file mode 100644 index 000000000..0ca025593 --- /dev/null +++ b/wai/staged-measure-conformance/quality.json @@ -0,0 +1,244 @@ +{ + "format": "wai-measure-quality/1", + "about": "Picture pairs and their §4.2 integers (extensions/staged-measurement.md), written by wai_stage_measure generate-goldens and required by wai_stage_measure verify and measure_verify.py. Byte c of pixel (x, y) of the picture [seed, mul] is (x*mul + y*(mul + 3) + 29*c + 17*seed + 5*((x*y + seed) mod 13)) mod 256.", + "cases": [ + { + "w": 1, + "h": 1, + "reference": [ + 1, + 5 + ], + "test": [ + 2, + 7 + ], + "n_samples": 3, + "sse_rgb": "1452", + "sse_y": "484", + "ssim_y_q32": 3986842579, + "windows": 1 + }, + { + "w": 5, + "h": 5, + "reference": [ + 3, + 11 + ], + "test": [ + 3, + 13 + ], + "n_samples": 75, + "sse_rgb": "65392", + "sse_y": "2145", + "ssim_y_q32": 4227018163, + "windows": 1 + }, + { + "w": 3, + "h": 100, + "reference": [ + 1, + 7 + ], + "test": [ + 2, + 7 + ], + "n_samples": 900, + "sse_rgb": "4705154", + "sse_y": "532902", + "ssim_y_q32": 2841543333, + "windows": 24 + }, + { + "w": 8, + "h": 8, + "reference": [ + 2, + 9 + ], + "test": [ + 4, + 9 + ], + "n_samples": 192, + "sse_rgb": "1321821", + "sse_y": "175220", + "ssim_y_q32": 1280208992, + "windows": 1 + }, + { + "w": 9, + "h": 9, + "reference": [ + 5, + 3 + ], + "test": [ + 5, + 4 + ], + "n_samples": 243, + "sse_rgb": "256360", + "sse_y": "6844", + "ssim_y_q32": 4122222193, + "windows": 4 + }, + { + "w": 11, + "h": 11, + "reference": [ + 1, + 5 + ], + "test": [ + 2, + 7 + ], + "n_samples": 363, + "sse_rgb": "1366484", + "sse_y": "201849", + "ssim_y_q32": 3172402069, + "windows": 4 + }, + { + "w": 13, + "h": 5, + "reference": [ + 6, + 2 + ], + "test": [ + 7, + 2 + ], + "n_samples": 195, + "sse_rgb": "425640", + "sse_y": "33092", + "ssim_y_q32": 2427457119, + "windows": 3 + }, + { + "w": 12, + "h": 20, + "reference": [ + 4, + 9 + ], + "test": [ + 5, + 9 + ], + "n_samples": 720, + "sse_rgb": "4072752", + "sse_y": "548256", + "ssim_y_q32": 2404129353, + "windows": 8 + }, + { + "w": 16, + "h": 16, + "reference": [ + 1, + 5 + ], + "test": [ + 1, + 6 + ], + "n_samples": 768, + "sse_rgb": "3329152", + "sse_y": "245154", + "ssim_y_q32": 3542145100, + "windows": 9 + }, + { + "w": 17, + "h": 9, + "reference": [ + 2, + 3 + ], + "test": [ + 5, + 3 + ], + "n_samples": 459, + "sse_rgb": "2191911", + "sse_y": "500794", + "ssim_y_q32": 1115121540, + "windows": 8 + }, + { + "w": 9, + "h": 33, + "reference": [ + 8, + 13 + ], + "test": [ + 9, + 14 + ], + "n_samples": 891, + "sse_rgb": "8244787", + "sse_y": "1469842", + "ssim_y_q32": 1291000068, + "windows": 16 + }, + { + "w": 19, + "h": 23, + "reference": [ + 3, + 1 + ], + "test": [ + 3, + 2 + ], + "n_samples": 1311, + "sse_rgb": "3944806", + "sse_y": "322676", + "ssim_y_q32": 3344287242, + "windows": 20 + }, + { + "w": 40, + "h": 24, + "reference": [ + 3, + 7 + ], + "test": [ + 3, + 8 + ], + "n_samples": 2880, + "sse_rgb": "19201376", + "sse_y": "3078475", + "ssim_y_q32": 2198626257, + "windows": 45 + }, + { + "w": 41, + "h": 27, + "reference": [ + 9, + 4 + ], + "test": [ + 1, + 4 + ], + "n_samples": 3321, + "sse_rgb": "50241520", + "sse_y": "11485039", + "ssim_y_q32": -168294012, + "windows": 60 + } + ] +} diff --git a/wai/staged-measure-conformance/refusals/base-over-budget/title.json b/wai/staged-measure-conformance/refusals/base-over-budget/title.json new file mode 100644 index 000000000..dcd5468c5 --- /dev/null +++ b/wai/staged-measure-conformance/refusals/base-over-budget/title.json @@ -0,0 +1,23 @@ +{ + "format": "wai-measure-title/1", + "title": "base-over-budget", + "about": "a base whose units' output passes the chain budget the title states", + "unit_us": 40000, + "bases": [ + { + "case": "staged-conformance/cases/base_over_budget", + "base": 0, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [], + "source": null, + "references": [], + "expect": null, + "limits": { + "cost": null, + "chain_budget": 383 + }, + "refused": "chain-over-limit" +} diff --git a/wai/staged-measure-conformance/refusals/clip-1gib-limits-raised/title.json b/wai/staged-measure-conformance/refusals/clip-1gib-limits-raised/title.json new file mode 100644 index 000000000..dada1bd43 --- /dev/null +++ b/wai/staged-measure-conformance/refusals/clip-1gib-limits-raised/title.json @@ -0,0 +1,28 @@ +{ + "format": "wai-measure-title/1", + "title": "clip-1gib-limits-raised", + "about": "the 1 GiB clip under a title that states limits of 2^40: a title lowers a limit and never raises one, so the tools' stated limit refuses it before anything is allocated", + "unit_us": 40000, + "bases": [ + { + "envelope": "wai-rs/tests/vectors/staged_bounds/clip_1gib.wai", + "set": {}, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [], + "source": null, + "references": [], + "expect": null, + "limits": { + "cost": [ + 1099511627776, + 1099511627776, + 1099511627776, + 1099511627776 + ], + "chain_budget": 1099511627776 + }, + "refused": "cost_over_limit" +} diff --git a/wai/staged-measure-conformance/refusals/clip-1gib/title.json b/wai/staged-measure-conformance/refusals/clip-1gib/title.json new file mode 100644 index 000000000..2d0fcecf1 --- /dev/null +++ b/wai/staged-measure-conformance/refusals/clip-1gib/title.json @@ -0,0 +1,20 @@ +{ + "format": "wai-measure-title/1", + "title": "clip-1gib", + "about": "a wai.video.int_motion base whose header declares 1 GiB of output, refused before anything is allocated for it", + "unit_us": 40000, + "bases": [ + { + "envelope": "wai-rs/tests/vectors/staged_bounds/clip_1gib.wai", + "set": {}, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [], + "source": null, + "references": [], + "expect": null, + "limits": null, + "refused": "cost_over_limit" +} diff --git a/wai/staged-measure-conformance/refusals/clip-64mib/title.json b/wai/staged-measure-conformance/refusals/clip-64mib/title.json new file mode 100644 index 000000000..33c2cdd42 --- /dev/null +++ b/wai/staged-measure-conformance/refusals/clip-64mib/title.json @@ -0,0 +1,20 @@ +{ + "format": "wai-measure-title/1", + "title": "clip-64mib", + "about": "a wai.video.int_motion base whose header declares 64 MiB of output, past the tools' stated limit", + "unit_us": 40000, + "bases": [ + { + "envelope": "wai-rs/tests/vectors/staged_bounds/clip_64mib.wai", + "set": {}, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [], + "source": null, + "references": [], + "expect": null, + "limits": null, + "refused": "cost_over_limit" +} diff --git a/wai/staged-measure-conformance/refusals/inert-companion-primary/title.json b/wai/staged-measure-conformance/refusals/inert-companion-primary/title.json new file mode 100644 index 000000000..28d26965a --- /dev/null +++ b/wai/staged-measure-conformance/refusals/inert-companion-primary/title.json @@ -0,0 +1,20 @@ +{ + "format": "wai-measure-title/1", + "title": "inert-companion-primary", + "about": "a companion-only capability with an integer fallback: inert, the fallback never reached", + "unit_us": 40000, + "bases": [ + { + "envelope": "wai-rs/tests/vectors/staged_bounds/inert_companion_primary.wai", + "set": {}, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [], + "source": null, + "references": [], + "expect": null, + "limits": null, + "refused": "missing_capability" +} diff --git a/wai/staged-measure-conformance/refusals/inert-refinement-primary/title.json b/wai/staged-measure-conformance/refusals/inert-refinement-primary/title.json new file mode 100644 index 000000000..e3ebdd474 --- /dev/null +++ b/wai/staged-measure-conformance/refusals/inert-refinement-primary/title.json @@ -0,0 +1,20 @@ +{ + "format": "wai-measure-title/1", + "title": "inert-refinement-primary", + "about": "a refinement-only capability with an integer fallback: inert, the fallback never reached", + "unit_us": 40000, + "bases": [ + { + "envelope": "wai-rs/tests/vectors/staged_bounds/inert_refinement_primary.wai", + "set": {}, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [], + "source": null, + "references": [], + "expect": null, + "limits": null, + "refused": "missing_capability" +} diff --git a/wai/staged-measure-conformance/refusals/layer-over-limit/L01.wai b/wai/staged-measure-conformance/refusals/layer-over-limit/L01.wai new file mode 100644 index 000000000..ff421c828 Binary files /dev/null and b/wai/staged-measure-conformance/refusals/layer-over-limit/L01.wai differ diff --git a/wai/staged-measure-conformance/refusals/layer-over-limit/title.json b/wai/staged-measure-conformance/refusals/layer-over-limit/title.json new file mode 100644 index 000000000..6c48a35f9 --- /dev/null +++ b/wai/staged-measure-conformance/refusals/layer-over-limit/title.json @@ -0,0 +1,30 @@ +{ + "format": "wai-measure-title/1", + "title": "layer-over-limit", + "about": "the still title's up2 layer under a cost limit its base passes and its layer does not: the layer's declared output, 3072 bytes, passes 1024", + "unit_us": 40000, + "bases": [ + { + "case": "staged-conformance/cases/image_int_hyper_2l", + "base": 0, + "first_unit": 0, + "capture_us": null + } + ], + "layers": [ + "L01.wai" + ], + "source": null, + "references": [], + "expect": null, + "limits": { + "cost": [ + 1024, + 1024, + 268435456, + 274877906944 + ], + "chain_budget": null + }, + "refused": "cost-over-limit" +} diff --git a/wai/staged-measure-conformance/sim/cases.json b/wai/staged-measure-conformance/sim/cases.json new file mode 100644 index 000000000..08e336a8d --- /dev/null +++ b/wai/staged-measure-conformance/sim/cases.json @@ -0,0 +1,186 @@ +{ + "format": "wai-sim-cases/1", + "about": "Delivery simulator cases (extensions/staged-measurement.md §10, Appendix B): a model (a file, or fixture: for a measured title's objects and counted bytes), a plan, and what the plan must show: within-bound, exceeds: (its largest stall passes that plan's bound), or unbounded.", + "cases": [ + { + "name": "gop3/strict-subgroup", + "model": "models/gop3.json", + "plan": "plans/strict-subgroup.json", + "claim": "within-bound" + }, + { + "name": "gop3/strict-slow-timeout", + "model": "models/gop3.json", + "plan": "plans/strict-slow-timeout.json", + "claim": "within-bound" + }, + { + "name": "gop3/fifo-timeout-subgroup", + "model": "models/gop3.json", + "plan": "plans/fifo-timeout-subgroup.json", + "claim": "within-bound" + }, + { + "name": "gop3/fifo-timeout-window", + "model": "models/gop3.json", + "plan": "plans/fifo-timeout-window.json", + "claim": "within-bound" + }, + { + "name": "gop3/fifo-timeout-datagram", + "model": "models/gop3.json", + "plan": "plans/fifo-timeout-datagram.json", + "claim": "within-bound" + }, + { + "name": "gop3/fifo-timeout-short", + "model": "models/gop3.json", + "plan": "plans/fifo-timeout-short.json", + "claim": "within-bound" + }, + { + "name": "gop3/fifo-no-timeout", + "model": "models/gop3.json", + "plan": "plans/fifo-no-timeout.json", + "claim": "exceeds:fifo-timeout-short" + }, + { + "name": "gop3/strict-window", + "model": "models/gop3.json", + "plan": "plans/strict-window.json", + "claim": "within-bound" + }, + { + "name": "gop3/fifo-decode", + "model": "models/gop3.json", + "plan": "plans/fifo-decode.json", + "claim": "exceeds:strict-window" + }, + { + "name": "gop3/capped", + "model": "models/gop3.json", + "plan": "plans/capped.json", + "claim": "within-bound" + }, + { + "name": "gop3/live-playout", + "model": "models/gop3.json", + "plan": "plans/live-playout.json", + "claim": "within-bound" + }, + { + "name": "gop3/live-playout-present", + "model": "models/gop3.json", + "plan": "plans/live-playout-present.json", + "claim": "within-bound" + }, + { + "name": "reorder-link", + "model": "models/reorder-link.json", + "plan": "plans/reorder-link.json", + "claim": "within-bound" + }, + { + "name": "reorder-decode", + "model": "models/reorder-decode.json", + "plan": "plans/reorder-decode.json", + "claim": "within-bound" + }, + { + "name": "audio/strict-subgroup", + "model": "fixture:audio", + "plan": "plans/strict-subgroup.json", + "claim": "within-bound" + }, + { + "name": "audio/fifo-timeout-subgroup", + "model": "fixture:audio", + "plan": "plans/fifo-timeout-subgroup.json", + "claim": "within-bound" + }, + { + "name": "audio/fifo-timeout-datagram", + "model": "fixture:audio", + "plan": "plans/fifo-timeout-datagram.json", + "claim": "within-bound" + }, + { + "name": "audio/fifo-no-timeout", + "model": "fixture:audio", + "plan": "plans/fifo-no-timeout.json", + "claim": "unbounded" + }, + { + "name": "clip/strict-subgroup", + "model": "fixture:clip", + "plan": "plans/strict-subgroup.json", + "claim": "within-bound" + }, + { + "name": "clip/fifo-timeout-subgroup", + "model": "fixture:clip", + "plan": "plans/fifo-timeout-subgroup.json", + "claim": "within-bound" + }, + { + "name": "clip/fifo-timeout-datagram", + "model": "fixture:clip", + "plan": "plans/fifo-timeout-datagram.json", + "claim": "within-bound" + }, + { + "name": "clip/fifo-no-timeout", + "model": "fixture:clip", + "plan": "plans/fifo-no-timeout.json", + "claim": "unbounded" + }, + { + "name": "splat/strict-subgroup", + "model": "fixture:splat", + "plan": "plans/strict-subgroup.json", + "claim": "within-bound" + }, + { + "name": "splat/fifo-timeout-subgroup", + "model": "fixture:splat", + "plan": "plans/fifo-timeout-subgroup.json", + "claim": "within-bound" + }, + { + "name": "splat/fifo-timeout-datagram", + "model": "fixture:splat", + "plan": "plans/fifo-timeout-datagram.json", + "claim": "within-bound" + }, + { + "name": "splat/fifo-no-timeout", + "model": "fixture:splat", + "plan": "plans/fifo-no-timeout.json", + "claim": "unbounded" + }, + { + "name": "still/strict-subgroup", + "model": "fixture:still", + "plan": "plans/strict-subgroup.json", + "claim": "within-bound" + }, + { + "name": "still/fifo-timeout-subgroup", + "model": "fixture:still", + "plan": "plans/fifo-timeout-subgroup.json", + "claim": "within-bound" + }, + { + "name": "still/fifo-timeout-datagram", + "model": "fixture:still", + "plan": "plans/fifo-timeout-datagram.json", + "claim": "within-bound" + }, + { + "name": "still/fifo-no-timeout", + "model": "fixture:still", + "plan": "plans/fifo-no-timeout.json", + "claim": "unbounded" + } + ] +} diff --git a/wai/staged-measure-conformance/sim/models/gop3.json b/wai/staged-measure-conformance/sim/models/gop3.json new file mode 100644 index 000000000..f30ccd3e4 --- /dev/null +++ b/wai/staged-measure-conformance/sim/models/gop3.json @@ -0,0 +1,369 @@ +{ + "format": "wai-sim-model/1", + "unit_us": 40000, + "objects": [ + { + "id": "g0-base", + "base": 0, + "layer": 0, + "units": [ + 0, + 5 + ], + "bytes": 5000, + "capture_us": 200000 + }, + { + "id": "g0-l1-u0", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "bytes": 1500, + "capture_us": 200000 + }, + { + "id": "g0-l1-u1", + "base": 0, + "layer": 1, + "units": [ + 1, + 1 + ], + "bytes": 1537, + "capture_us": 200000 + }, + { + "id": "g0-l1-u2", + "base": 0, + "layer": 1, + "units": [ + 2, + 1 + ], + "bytes": 1574, + "capture_us": 200000 + }, + { + "id": "g0-l1-u3", + "base": 0, + "layer": 1, + "units": [ + 3, + 1 + ], + "bytes": 1611, + "capture_us": 200000 + }, + { + "id": "g0-l1-u4", + "base": 0, + "layer": 1, + "units": [ + 4, + 1 + ], + "bytes": 1648, + "capture_us": 200000 + }, + { + "id": "g0-l2-u0", + "base": 0, + "layer": 2, + "units": [ + 0, + 1 + ], + "bytes": 900, + "capture_us": 200000 + }, + { + "id": "g0-l2-u1", + "base": 0, + "layer": 2, + "units": [ + 1, + 1 + ], + "bytes": 937, + "capture_us": 200000 + }, + { + "id": "g0-l2-u2", + "base": 0, + "layer": 2, + "units": [ + 2, + 1 + ], + "bytes": 974, + "capture_us": 200000 + }, + { + "id": "g0-l2-u3", + "base": 0, + "layer": 2, + "units": [ + 3, + 1 + ], + "bytes": 1011, + "capture_us": 200000 + }, + { + "id": "g0-l2-u4", + "base": 0, + "layer": 2, + "units": [ + 4, + 1 + ], + "bytes": 1048, + "capture_us": 200000 + }, + { + "id": "g1-base", + "base": 1, + "layer": 0, + "units": [ + 5, + 5 + ], + "bytes": 5700, + "capture_us": 400000 + }, + { + "id": "g1-l1-u0", + "base": 1, + "layer": 1, + "units": [ + 5, + 1 + ], + "bytes": 1511, + "capture_us": 400000 + }, + { + "id": "g1-l1-u1", + "base": 1, + "layer": 1, + "units": [ + 6, + 1 + ], + "bytes": 1548, + "capture_us": 400000 + }, + { + "id": "g1-l1-u2", + "base": 1, + "layer": 1, + "units": [ + 7, + 1 + ], + "bytes": 1585, + "capture_us": 400000 + }, + { + "id": "g1-l1-u3", + "base": 1, + "layer": 1, + "units": [ + 8, + 1 + ], + "bytes": 1622, + "capture_us": 400000 + }, + { + "id": "g1-l1-u4", + "base": 1, + "layer": 1, + "units": [ + 9, + 1 + ], + "bytes": 1659, + "capture_us": 400000 + }, + { + "id": "g1-l2-u0", + "base": 1, + "layer": 2, + "units": [ + 5, + 1 + ], + "bytes": 911, + "capture_us": 400000 + }, + { + "id": "g1-l2-u1", + "base": 1, + "layer": 2, + "units": [ + 6, + 1 + ], + "bytes": 948, + "capture_us": 400000 + }, + { + "id": "g1-l2-u2", + "base": 1, + "layer": 2, + "units": [ + 7, + 1 + ], + "bytes": 985, + "capture_us": 400000 + }, + { + "id": "g1-l2-u3", + "base": 1, + "layer": 2, + "units": [ + 8, + 1 + ], + "bytes": 1022, + "capture_us": 400000 + }, + { + "id": "g1-l2-u4", + "base": 1, + "layer": 2, + "units": [ + 9, + 1 + ], + "bytes": 1059, + "capture_us": 400000 + }, + { + "id": "g2-base", + "base": 2, + "layer": 0, + "units": [ + 10, + 5 + ], + "bytes": 6400, + "capture_us": 600000 + }, + { + "id": "g2-l1-u0", + "base": 2, + "layer": 1, + "units": [ + 10, + 1 + ], + "bytes": 1522, + "capture_us": 600000 + }, + { + "id": "g2-l1-u1", + "base": 2, + "layer": 1, + "units": [ + 11, + 1 + ], + "bytes": 1559, + "capture_us": 600000 + }, + { + "id": "g2-l1-u2", + "base": 2, + "layer": 1, + "units": [ + 12, + 1 + ], + "bytes": 1596, + "capture_us": 600000 + }, + { + "id": "g2-l1-u3", + "base": 2, + "layer": 1, + "units": [ + 13, + 1 + ], + "bytes": 1633, + "capture_us": 600000 + }, + { + "id": "g2-l1-u4", + "base": 2, + "layer": 1, + "units": [ + 14, + 1 + ], + "bytes": 1670, + "capture_us": 600000 + }, + { + "id": "g2-l2-u0", + "base": 2, + "layer": 2, + "units": [ + 10, + 1 + ], + "bytes": 922, + "capture_us": 600000 + }, + { + "id": "g2-l2-u1", + "base": 2, + "layer": 2, + "units": [ + 11, + 1 + ], + "bytes": 959, + "capture_us": 600000 + }, + { + "id": "g2-l2-u2", + "base": 2, + "layer": 2, + "units": [ + 12, + 1 + ], + "bytes": 996, + "capture_us": 600000 + }, + { + "id": "g2-l2-u3", + "base": 2, + "layer": 2, + "units": [ + 13, + 1 + ], + "bytes": 1033, + "capture_us": 600000 + }, + { + "id": "g2-l2-u4", + "base": 2, + "layer": 2, + "units": [ + 14, + 1 + ], + "bytes": 1070, + "capture_us": 600000 + } + ] +} diff --git a/wai/staged-measure-conformance/sim/models/reorder-decode.json b/wai/staged-measure-conformance/sim/models/reorder-decode.json new file mode 100644 index 000000000..b59411545 --- /dev/null +++ b/wai/staged-measure-conformance/sim/models/reorder-decode.json @@ -0,0 +1,50 @@ +{ + "format": "wai-sim-model/1", + "unit_us": 40000, + "objects": [ + { + "id": "B0", + "base": 0, + "layer": 0, + "units": [ + 0, + 1 + ], + "bytes": 10, + "capture_us": null + }, + { + "id": "R0", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "bytes": 10, + "capture_us": null + }, + { + "id": "Bb", + "base": 2, + "layer": 0, + "units": [ + 2, + 1 + ], + "bytes": 100000, + "capture_us": null + }, + { + "id": "Ba", + "base": 1, + "layer": 0, + "units": [ + 1, + 1 + ], + "bytes": 10, + "capture_us": null + } + ] +} diff --git a/wai/staged-measure-conformance/sim/models/reorder-link.json b/wai/staged-measure-conformance/sim/models/reorder-link.json new file mode 100644 index 000000000..8b1446ff6 --- /dev/null +++ b/wai/staged-measure-conformance/sim/models/reorder-link.json @@ -0,0 +1,50 @@ +{ + "format": "wai-sim-model/1", + "unit_us": 40000, + "objects": [ + { + "id": "B0", + "base": 0, + "layer": 0, + "units": [ + 0, + 1 + ], + "bytes": 10, + "capture_us": 0 + }, + { + "id": "R0", + "base": 0, + "layer": 1, + "units": [ + 0, + 1 + ], + "bytes": 1000, + "capture_us": 0 + }, + { + "id": "Bb", + "base": 2, + "layer": 0, + "units": [ + 2, + 1 + ], + "bytes": 10, + "capture_us": 15 + }, + { + "id": "Ba", + "base": 1, + "layer": 0, + "units": [ + 1, + 1 + ], + "bytes": 50000, + "capture_us": 20 + } + ] +} diff --git a/wai/staged-measure-conformance/sim/plans/capped.json b/wai/staged-measure-conformance/sim/plans/capped.json new file mode 100644 index 000000000..297076293 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/capped.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": 1, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/fifo-decode.json b/wai/staged-measure-conformance/sim/plans/fifo-decode.json new file mode 100644 index 000000000..a1a1e754b --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/fifo-decode.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": 180000 + }, + "sink": { + "workers": 1, + "decode_order": "fifo", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/fifo-no-timeout.json b/wai/staged-measure-conformance/sim/plans/fifo-no-timeout.json new file mode 100644 index 000000000..77f3070b3 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/fifo-no-timeout.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "fifo", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/fifo-timeout-datagram.json b/wai/staged-measure-conformance/sim/plans/fifo-timeout-datagram.json new file mode 100644 index 000000000..e7256a325 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/fifo-timeout-datagram.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 400, + "chunk_bytes": 12000 + }, + "sender": { + "schedule": "fifo", + "carriage": "datagram", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 30000 + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/fifo-timeout-short.json b/wai/staged-measure-conformance/sim/plans/fifo-timeout-short.json new file mode 100644 index 000000000..5e6146468 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/fifo-timeout-short.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "fifo", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 1000 + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/fifo-timeout-subgroup.json b/wai/staged-measure-conformance/sim/plans/fifo-timeout-subgroup.json new file mode 100644 index 000000000..342b4e39e --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/fifo-timeout-subgroup.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 400, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "fifo", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 30000 + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/fifo-timeout-window.json b/wai/staged-measure-conformance/sim/plans/fifo-timeout-window.json new file mode 100644 index 000000000..2a78ad488 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/fifo-timeout-window.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 400, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "fifo", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 30000 + }, + "vod_window_us": 100000 + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/live-playout-present.json b/wai/staged-measure-conformance/sim/plans/live-playout-present.json new file mode 100644 index 000000000..8efb553c4 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/live-playout-present.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "live", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": null + }, + "sink": { + "workers": 2, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": 215000, + "late_refinement": "present" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/live-playout.json b/wai/staged-measure-conformance/sim/plans/live-playout.json new file mode 100644 index 000000000..0788c5c36 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/live-playout.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "live", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 100000 + }, + "vod_window_us": null + }, + "sink": { + "workers": 2, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": 215000, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/reorder-decode.json b/wai/staged-measure-conformance/sim/plans/reorder-decode.json new file mode 100644 index 000000000..6b0de029c --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/reorder-decode.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 0, + "bytes_per_ms": 1000, + "chunk_bytes": 1000 + }, + "sender": { + "schedule": "fifo", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 20 + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 100000, + "refinement": 100 + }, + "playout_start_us": null, + "late_refinement": "present" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/reorder-link.json b/wai/staged-measure-conformance/sim/plans/reorder-link.json new file mode 100644 index 000000000..b3244d35f --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/reorder-link.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "live", + "link": { + "rtt_us": 0, + "bytes_per_ms": 1000, + "chunk_bytes": 1000 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 100, + "refinement": 100 + }, + "playout_start_us": null, + "late_refinement": "present" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/strict-slow-timeout.json b/wai/staged-measure-conformance/sim/plans/strict-slow-timeout.json new file mode 100644 index 000000000..b5fb253b4 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/strict-slow-timeout.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 400, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": 30000 + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/strict-subgroup.json b/wai/staged-measure-conformance/sim/plans/strict-subgroup.json new file mode 100644 index 000000000..393837b04 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/strict-subgroup.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": null + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/plans/strict-window.json b/wai/staged-measure-conformance/sim/plans/strict-window.json new file mode 100644 index 000000000..d5d9b0003 --- /dev/null +++ b/wai/staged-measure-conformance/sim/plans/strict-window.json @@ -0,0 +1,29 @@ +{ + "plan": "wai-sim/1", + "mode": "vod", + "link": { + "rtt_us": 40000, + "bytes_per_ms": 2500, + "chunk_bytes": 1200 + }, + "sender": { + "schedule": "strict-priority", + "carriage": "subgroup", + "layer_cap": null, + "delivery_timeout_us": { + "base": null, + "refinement": null + }, + "vod_window_us": 180000 + }, + "sink": { + "workers": 1, + "decode_order": "base-first", + "decode_us": { + "base": 4000, + "refinement": 2000 + }, + "playout_start_us": null, + "late_refinement": "discard" + } +} diff --git a/wai/staged-measure-conformance/sim/results.json b/wai/staged-measure-conformance/sim/results.json new file mode 100644 index 000000000..0ea784788 --- /dev/null +++ b/wai/staged-measure-conformance/sim/results.json @@ -0,0 +1,7176 @@ +{ + "format": "wai-sim-results/1", + "about": "Every case of sim/cases.json simulated (extensions/staged-measurement.md Appendix B), written by wai_stage_measure generate-goldens and required by wai_stage_measure verify and sim_verify.py. Simulated, not measured.", + "results": [ + { + "name": "gop3/strict-subgroup", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 42000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 26840, + "arrived_us": 47440, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 27440, + "arrived_us": 48055, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 28055, + "arrived_us": 48685, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 28685, + "arrived_us": 49330, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 29330, + "arrived_us": 49990, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 36356, + "arrived_us": 56716, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 36716, + "arrived_us": 57091, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 37091, + "arrived_us": 57481, + "state": "delivered" + }, + { + "id": "g0-l2-u3", + "sent_first_us": 37481, + "arrived_us": 57886, + "state": "delivered" + }, + { + "id": "g0-l2-u4", + "sent_first_us": 37886, + "arrived_us": 58306, + "state": "delivered" + }, + { + "id": "g1-base", + "sent_first_us": 22000, + "arrived_us": 44280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 29990, + "arrived_us": 50595, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 30595, + "arrived_us": 51215, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 31215, + "arrived_us": 51849, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 31849, + "arrived_us": 52498, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 32498, + "arrived_us": 53162, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 38306, + "arrived_us": 58671, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 38671, + "arrived_us": 59051, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 39051, + "arrived_us": 59445, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 39445, + "arrived_us": 59854, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 39854, + "arrived_us": 60278, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 24280, + "arrived_us": 46840, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 33162, + "arrived_us": 53771, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 33771, + "arrived_us": 54395, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 34395, + "arrived_us": 55034, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 35034, + "arrived_us": 55688, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 35688, + "arrived_us": 56356, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 40278, + "arrived_us": 60647, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 40647, + "arrived_us": 61031, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 41031, + "arrived_us": 61430, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 41430, + "arrived_us": 61844, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 41844, + "arrived_us": 62272, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 46000 + ], + [ + 0, + 2, + 56000 + ], + [ + 0, + 3, + 86000 + ], + [ + 1, + 1, + 46000 + ], + [ + 1, + 2, + 58000 + ], + [ + 1, + 3, + 88000 + ], + [ + 2, + 1, + 46000 + ], + [ + 2, + 2, + 60000 + ], + [ + 2, + 3, + 90000 + ], + [ + 3, + 1, + 46000 + ], + [ + 3, + 2, + 62000 + ], + [ + 3, + 3, + 92000 + ], + [ + 4, + 1, + 46000 + ], + [ + 4, + 2, + 64000 + ], + [ + 4, + 3, + 94000 + ], + [ + 5, + 1, + 50000 + ], + [ + 5, + 2, + 66000 + ], + [ + 5, + 3, + 96000 + ], + [ + 6, + 1, + 50000 + ], + [ + 6, + 2, + 68000 + ], + [ + 6, + 3, + 98000 + ], + [ + 7, + 1, + 50000 + ], + [ + 7, + 2, + 70000 + ], + [ + 7, + 3, + 100000 + ], + [ + 8, + 1, + 50000 + ], + [ + 8, + 2, + 72000 + ], + [ + 8, + 3, + 102000 + ], + [ + 9, + 1, + 50000 + ], + [ + 9, + 2, + 74000 + ], + [ + 9, + 3, + 104000 + ], + [ + 10, + 1, + 54000 + ], + [ + 10, + 2, + 76000 + ], + [ + 10, + 3, + 106000 + ], + [ + 11, + 1, + 54000 + ], + [ + 11, + 2, + 78000 + ], + [ + 11, + 3, + 108000 + ], + [ + 12, + 1, + 54000 + ], + [ + 12, + 2, + 80000 + ], + [ + 12, + 3, + 110000 + ], + [ + 13, + 1, + 54000 + ], + [ + 13, + 2, + 82000 + ], + [ + 13, + 3, + 112000 + ], + [ + 14, + 1, + 54000 + ], + [ + 14, + 2, + 84000 + ], + [ + 14, + 3, + 114000 + ] + ], + "bfpf_sched": 14300, + "tfpf_us": 46000, + "tffs_us": [ + 46000, + 56000, + 86000 + ], + "offered": 55650, + "delivered": 55650, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/strict-slow-timeout", + "stall_bound_us": 5000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 52500, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-base", + "sent_first_us": 32500, + "arrived_us": 66750, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-base", + "sent_first_us": 46750, + "arrived_us": 82750, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + } + ], + "presentations": [ + [ + 0, + 1, + 56500 + ], + [ + 1, + 1, + 56500 + ], + [ + 2, + 1, + 56500 + ], + [ + 3, + 1, + 56500 + ], + [ + 4, + 1, + 56500 + ], + [ + 5, + 1, + 70750 + ], + [ + 6, + 1, + 70750 + ], + [ + 7, + 1, + 70750 + ], + [ + 8, + 1, + 70750 + ], + [ + 9, + 1, + 70750 + ], + [ + 10, + 1, + 86750 + ], + [ + 11, + 1, + 86750 + ], + [ + 12, + 1, + 86750 + ], + [ + 13, + 1, + 86750 + ], + [ + 14, + 1, + 86750 + ] + ], + "bfpf_sched": 6200, + "tfpf_us": 56500, + "tffs_us": [ + 56500, + null, + null + ], + "offered": 55650, + "delivered": 17100, + "dropped": 38550, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/fifo-timeout-subgroup", + "stall_bound_us": 35000, + "max_stall_us": 18556, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 52500, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 32500, + "arrived_us": 56250, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 36250, + "arrived_us": 60093, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 40093, + "arrived_us": 64028, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 44028, + "arrived_us": 68056, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 48056, + "arrived_us": null, + "state": "abandoned" + }, + { + "id": "g0-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-base", + "sent_first_us": 51056, + "arrived_us": 85306, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-base", + "sent_first_us": 65306, + "arrived_us": 101306, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + } + ], + "presentations": [ + [ + 0, + 1, + 56500 + ], + [ + 0, + 2, + 58500 + ], + [ + 1, + 1, + 56500 + ], + [ + 1, + 2, + 62093 + ], + [ + 2, + 1, + 56500 + ], + [ + 2, + 2, + 66028 + ], + [ + 3, + 1, + 56500 + ], + [ + 3, + 2, + 70056 + ], + [ + 4, + 1, + 56500 + ], + [ + 5, + 1, + 89306 + ], + [ + 6, + 1, + 89306 + ], + [ + 7, + 1, + 89306 + ], + [ + 8, + 1, + 89306 + ], + [ + 9, + 1, + 89306 + ], + [ + 10, + 1, + 105306 + ], + [ + 11, + 1, + 105306 + ], + [ + 12, + 1, + 105306 + ], + [ + 13, + 1, + 105306 + ], + [ + 14, + 1, + 105306 + ] + ], + "bfpf_sched": 6500, + "tfpf_us": 56500, + "tffs_us": [ + 56500, + 58500, + null + ], + "offered": 55650, + "delivered": 23322, + "dropped": 31128, + "abandoned": 1200, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 18556 + ], + [ + 6, + 18556 + ], + [ + 7, + 18556 + ], + [ + 8, + 18556 + ], + [ + 9, + 18556 + ], + [ + 10, + 18556 + ], + [ + 11, + 18556 + ], + [ + 12, + 18556 + ], + [ + 13, + 18556 + ], + [ + 14, + 18556 + ] + ] + } + }, + { + "name": "gop3/fifo-timeout-window", + "stall_bound_us": 35000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 52500, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 32500, + "arrived_us": 56250, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 36250, + "arrived_us": 60093, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 40093, + "arrived_us": 64028, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 51056, + "arrived_us": 75084, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 80000, + "arrived_us": 104120, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 44028, + "arrived_us": 66278, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 46278, + "arrived_us": 68621, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 48621, + "arrived_us": null, + "state": "abandoned" + }, + { + "id": "g0-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-base", + "sent_first_us": 120000, + "arrived_us": 154250, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 134250, + "arrived_us": 158028, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 160000, + "arrived_us": 183870, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 200000, + "arrived_us": 223963, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 240000, + "arrived_us": 264055, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 280000, + "arrived_us": 304148, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 138028, + "arrived_us": 160306, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 163870, + "arrived_us": 186240, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 203963, + "arrived_us": 226426, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 244055, + "arrived_us": 266610, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 284148, + "arrived_us": 306796, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 320000, + "arrived_us": 356000, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 336000, + "arrived_us": 359805, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 360000, + "arrived_us": 383898, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 400000, + "arrived_us": 423990, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 440000, + "arrived_us": 464083, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 480000, + "arrived_us": 504175, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 339805, + "arrived_us": 362110, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 363898, + "arrived_us": 386296, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 403990, + "arrived_us": 426480, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 444083, + "arrived_us": 466666, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 484175, + "arrived_us": 506850, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 56500 + ], + [ + 0, + 2, + 58500 + ], + [ + 0, + 3, + 68278 + ], + [ + 1, + 1, + 56500 + ], + [ + 1, + 2, + 62093 + ], + [ + 1, + 3, + 70621 + ], + [ + 2, + 1, + 56500 + ], + [ + 2, + 2, + 66028 + ], + [ + 3, + 1, + 56500 + ], + [ + 3, + 2, + 77084 + ], + [ + 4, + 1, + 56500 + ], + [ + 4, + 2, + 106120 + ], + [ + 5, + 1, + 158250 + ], + [ + 5, + 2, + 160250 + ], + [ + 5, + 3, + 162306 + ], + [ + 6, + 1, + 158250 + ], + [ + 6, + 2, + 185870 + ], + [ + 6, + 3, + 188240 + ], + [ + 7, + 1, + 158250 + ], + [ + 7, + 2, + 225963 + ], + [ + 7, + 3, + 228426 + ], + [ + 8, + 1, + 158250 + ], + [ + 8, + 2, + 266055 + ], + [ + 8, + 3, + 268610 + ], + [ + 9, + 1, + 158250 + ], + [ + 9, + 2, + 306148 + ], + [ + 9, + 3, + 308796 + ], + [ + 10, + 1, + 360000 + ], + [ + 10, + 2, + 362000 + ], + [ + 10, + 3, + 364110 + ], + [ + 11, + 1, + 360000 + ], + [ + 11, + 2, + 385898 + ], + [ + 11, + 3, + 388296 + ], + [ + 12, + 1, + 360000 + ], + [ + 12, + 2, + 425990 + ], + [ + 12, + 3, + 428480 + ], + [ + 13, + 1, + 360000 + ], + [ + 13, + 2, + 466083 + ], + [ + 13, + 3, + 468666 + ], + [ + 14, + 1, + 360000 + ], + [ + 14, + 2, + 506175 + ], + [ + 14, + 3, + 508850 + ] + ], + "bfpf_sched": 6500, + "tfpf_us": 56500, + "tffs_us": [ + 56500, + 58500, + 68278 + ], + "offered": 55650, + "delivered": 52617, + "dropped": 2059, + "abandoned": 974, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/fifo-timeout-datagram", + "stall_bound_us": 62000, + "max_stall_us": 19676, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 52500, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 32500, + "arrived_us": 56250, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 36250, + "arrived_us": 60093, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 40093, + "arrived_us": 64028, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 44028, + "arrived_us": 68056, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 48056, + "arrived_us": 72176, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-base", + "sent_first_us": 52176, + "arrived_us": 86426, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-base", + "sent_first_us": 66426, + "arrived_us": 102426, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + } + ], + "presentations": [ + [ + 0, + 1, + 56500 + ], + [ + 0, + 2, + 58500 + ], + [ + 1, + 1, + 56500 + ], + [ + 1, + 2, + 62093 + ], + [ + 2, + 1, + 56500 + ], + [ + 2, + 2, + 66028 + ], + [ + 3, + 1, + 56500 + ], + [ + 3, + 2, + 70056 + ], + [ + 4, + 1, + 56500 + ], + [ + 4, + 2, + 74176 + ], + [ + 5, + 1, + 90426 + ], + [ + 6, + 1, + 90426 + ], + [ + 7, + 1, + 90426 + ], + [ + 8, + 1, + 90426 + ], + [ + 9, + 1, + 90426 + ], + [ + 10, + 1, + 106426 + ], + [ + 11, + 1, + 106426 + ], + [ + 12, + 1, + 106426 + ], + [ + 13, + 1, + 106426 + ], + [ + 14, + 1, + 106426 + ] + ], + "bfpf_sched": 6500, + "tfpf_us": 56500, + "tffs_us": [ + 56500, + 58500, + null + ], + "offered": 55650, + "delivered": 24970, + "dropped": 30680, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 19676 + ], + [ + 6, + 19676 + ], + [ + 7, + 19676 + ], + [ + 8, + 19676 + ], + [ + 9, + 19676 + ], + [ + 10, + 19676 + ], + [ + 11, + 19676 + ], + [ + 12, + 19676 + ], + [ + 13, + 19676 + ], + [ + 14, + 19676 + ] + ] + } + }, + { + "name": "gop3/fifo-timeout-short", + "stall_bound_us": 3480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 42000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g0-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-base", + "sent_first_us": 22000, + "arrived_us": 44280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g1-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-base", + "sent_first_us": 24280, + "arrived_us": 46840, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l1-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "g2-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + } + ], + "presentations": [ + [ + 0, + 1, + 46000 + ], + [ + 1, + 1, + 46000 + ], + [ + 2, + 1, + 46000 + ], + [ + 3, + 1, + 46000 + ], + [ + 4, + 1, + 46000 + ], + [ + 5, + 1, + 50000 + ], + [ + 6, + 1, + 50000 + ], + [ + 7, + 1, + 50000 + ], + [ + 8, + 1, + 50000 + ], + [ + 9, + 1, + 50000 + ], + [ + 10, + 1, + 54000 + ], + [ + 11, + 1, + 54000 + ], + [ + 12, + 1, + 54000 + ], + [ + 13, + 1, + 54000 + ], + [ + 14, + 1, + 54000 + ] + ], + "bfpf_sched": 14300, + "tfpf_us": 46000, + "tffs_us": [ + 46000, + null, + null + ], + "offered": 55650, + "delivered": 17100, + "dropped": 38550, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/fifo-no-timeout", + "stall_bound_us": null, + "max_stall_us": 8000, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 42000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 22000, + "arrived_us": 42600, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 22600, + "arrived_us": 43215, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 23215, + "arrived_us": 43845, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 23845, + "arrived_us": 44490, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 24490, + "arrived_us": 45150, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 25150, + "arrived_us": 45510, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 25510, + "arrived_us": 45885, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 25885, + "arrived_us": 46275, + "state": "delivered" + }, + { + "id": "g0-l2-u3", + "sent_first_us": 26275, + "arrived_us": 46680, + "state": "delivered" + }, + { + "id": "g0-l2-u4", + "sent_first_us": 26680, + "arrived_us": 47100, + "state": "delivered" + }, + { + "id": "g1-base", + "sent_first_us": 27100, + "arrived_us": 49380, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 29380, + "arrived_us": 49985, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 29985, + "arrived_us": 50605, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 30605, + "arrived_us": 51239, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 31239, + "arrived_us": 51888, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 31888, + "arrived_us": 52552, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 32552, + "arrived_us": 52917, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 32917, + "arrived_us": 53297, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 33297, + "arrived_us": 53691, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 33691, + "arrived_us": 54100, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 34100, + "arrived_us": 54524, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 34524, + "arrived_us": 57084, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 37084, + "arrived_us": 57693, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 37693, + "arrived_us": 58317, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 38317, + "arrived_us": 58956, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 38956, + "arrived_us": 59610, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 39610, + "arrived_us": 60278, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 40278, + "arrived_us": 60647, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 40647, + "arrived_us": 61031, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 41031, + "arrived_us": 61430, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 41430, + "arrived_us": 61844, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 41844, + "arrived_us": 62272, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 46000 + ], + [ + 0, + 2, + 48000 + ], + [ + 0, + 3, + 86000 + ], + [ + 1, + 1, + 46000 + ], + [ + 1, + 2, + 50000 + ], + [ + 1, + 3, + 88000 + ], + [ + 2, + 1, + 46000 + ], + [ + 2, + 2, + 56000 + ], + [ + 2, + 3, + 90000 + ], + [ + 3, + 1, + 46000 + ], + [ + 3, + 2, + 58000 + ], + [ + 3, + 3, + 92000 + ], + [ + 4, + 1, + 46000 + ], + [ + 4, + 2, + 64000 + ], + [ + 4, + 3, + 94000 + ], + [ + 5, + 1, + 54000 + ], + [ + 5, + 2, + 66000 + ], + [ + 5, + 3, + 96000 + ], + [ + 6, + 1, + 54000 + ], + [ + 6, + 2, + 68000 + ], + [ + 6, + 3, + 98000 + ], + [ + 7, + 1, + 54000 + ], + [ + 7, + 2, + 70000 + ], + [ + 7, + 3, + 100000 + ], + [ + 8, + 1, + 54000 + ], + [ + 8, + 2, + 72000 + ], + [ + 8, + 3, + 102000 + ], + [ + 9, + 1, + 54000 + ], + [ + 9, + 2, + 74000 + ], + [ + 9, + 3, + 104000 + ], + [ + 10, + 1, + 62000 + ], + [ + 10, + 2, + 76000 + ], + [ + 10, + 3, + 106000 + ], + [ + 11, + 1, + 62000 + ], + [ + 11, + 2, + 78000 + ], + [ + 11, + 3, + 108000 + ], + [ + 12, + 1, + 62000 + ], + [ + 12, + 2, + 80000 + ], + [ + 12, + 3, + 110000 + ], + [ + 13, + 1, + 62000 + ], + [ + 13, + 2, + 82000 + ], + [ + 13, + 3, + 112000 + ], + [ + 14, + 1, + 62000 + ], + [ + 14, + 2, + 84000 + ], + [ + 14, + 3, + 114000 + ] + ], + "bfpf_sched": 14707, + "tfpf_us": 46000, + "tffs_us": [ + 46000, + 48000, + 86000 + ], + "offered": 55650, + "delivered": 55650, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 4000 + ], + [ + 6, + 4000 + ], + [ + 7, + 4000 + ], + [ + 8, + 4000 + ], + [ + 9, + 4000 + ], + [ + 10, + 8000 + ], + [ + 11, + 8000 + ], + [ + 12, + 8000 + ], + [ + 13, + 8000 + ], + [ + 14, + 8000 + ] + ] + } + }, + { + "name": "gop3/strict-window", + "stall_bound_us": 2480, + "max_stall_us": 1720, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 42000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 22000, + "arrived_us": 42600, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 22600, + "arrived_us": 43215, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 23215, + "arrived_us": 43845, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 23845, + "arrived_us": 44490, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 24490, + "arrived_us": 45150, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 25150, + "arrived_us": 45510, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 25510, + "arrived_us": 45885, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 25885, + "arrived_us": 46275, + "state": "delivered" + }, + { + "id": "g0-l2-u3", + "sent_first_us": 26275, + "arrived_us": 46680, + "state": "delivered" + }, + { + "id": "g0-l2-u4", + "sent_first_us": 26680, + "arrived_us": 47100, + "state": "delivered" + }, + { + "id": "g1-base", + "sent_first_us": 40000, + "arrived_us": 62280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 42280, + "arrived_us": 62885, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 80000, + "arrived_us": 100620, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 120000, + "arrived_us": 140634, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 160000, + "arrived_us": 180649, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 200000, + "arrived_us": 220664, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 42885, + "arrived_us": 63250, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 80620, + "arrived_us": 101000, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 120634, + "arrived_us": 141028, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 160649, + "arrived_us": 181058, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 200664, + "arrived_us": 221088, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 240000, + "arrived_us": 262560, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 242560, + "arrived_us": 263169, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 280000, + "arrived_us": 300624, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 320000, + "arrived_us": 340639, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 360000, + "arrived_us": 380654, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 400000, + "arrived_us": 420668, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 243169, + "arrived_us": 263538, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 280624, + "arrived_us": 301008, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 320639, + "arrived_us": 341038, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 360654, + "arrived_us": 381068, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 400668, + "arrived_us": 421096, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 46000 + ], + [ + 0, + 2, + 48000 + ], + [ + 0, + 3, + 58000 + ], + [ + 1, + 1, + 46000 + ], + [ + 1, + 2, + 50000 + ], + [ + 1, + 3, + 60000 + ], + [ + 2, + 1, + 46000 + ], + [ + 2, + 2, + 52000 + ], + [ + 2, + 3, + 62000 + ], + [ + 3, + 1, + 46000 + ], + [ + 3, + 2, + 54000 + ], + [ + 3, + 3, + 64000 + ], + [ + 4, + 1, + 46000 + ], + [ + 4, + 2, + 56000 + ], + [ + 4, + 3, + 72000 + ], + [ + 5, + 1, + 68000 + ], + [ + 5, + 2, + 70000 + ], + [ + 5, + 3, + 74000 + ], + [ + 6, + 1, + 68000 + ], + [ + 6, + 2, + 102620 + ], + [ + 6, + 3, + 104620 + ], + [ + 7, + 1, + 68000 + ], + [ + 7, + 2, + 142634 + ], + [ + 7, + 3, + 144634 + ], + [ + 8, + 1, + 68000 + ], + [ + 8, + 2, + 182649 + ], + [ + 8, + 3, + 184649 + ], + [ + 9, + 1, + 68000 + ], + [ + 9, + 2, + 222664 + ], + [ + 9, + 3, + 224664 + ], + [ + 10, + 1, + 266560 + ], + [ + 10, + 2, + 268560 + ], + [ + 10, + 3, + 270560 + ], + [ + 11, + 1, + 266560 + ], + [ + 11, + 2, + 302624 + ], + [ + 11, + 3, + 304624 + ], + [ + 12, + 1, + 266560 + ], + [ + 12, + 2, + 342639 + ], + [ + 12, + 3, + 344639 + ], + [ + 13, + 1, + 266560 + ], + [ + 13, + 2, + 382654 + ], + [ + 13, + 3, + 384654 + ], + [ + 14, + 1, + 266560 + ], + [ + 14, + 2, + 422668 + ], + [ + 14, + 3, + 424668 + ] + ], + "bfpf_sched": 14707, + "tfpf_us": 46000, + "tffs_us": [ + 46000, + 48000, + 58000 + ], + "offered": 55650, + "delivered": 55650, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 1720 + ], + [ + 6, + 1720 + ], + [ + 7, + 1720 + ], + [ + 8, + 1720 + ], + [ + 9, + 1720 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/fifo-decode", + "stall_bound_us": null, + "max_stall_us": 3720, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 42000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 22000, + "arrived_us": 42600, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 22600, + "arrived_us": 43215, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 23215, + "arrived_us": 43845, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 23845, + "arrived_us": 44490, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 24490, + "arrived_us": 45150, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 25150, + "arrived_us": 45510, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 25510, + "arrived_us": 45885, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 25885, + "arrived_us": 46275, + "state": "delivered" + }, + { + "id": "g0-l2-u3", + "sent_first_us": 26275, + "arrived_us": 46680, + "state": "delivered" + }, + { + "id": "g0-l2-u4", + "sent_first_us": 26680, + "arrived_us": 47100, + "state": "delivered" + }, + { + "id": "g1-base", + "sent_first_us": 40000, + "arrived_us": 62280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 42280, + "arrived_us": 62885, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 80000, + "arrived_us": 100620, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 120000, + "arrived_us": 140634, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 160000, + "arrived_us": 180649, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 200000, + "arrived_us": 220664, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 42885, + "arrived_us": 63250, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 80620, + "arrived_us": 101000, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 120634, + "arrived_us": 141028, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 160649, + "arrived_us": 181058, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 200664, + "arrived_us": 221088, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 240000, + "arrived_us": 262560, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 242560, + "arrived_us": 263169, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 280000, + "arrived_us": 300624, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 320000, + "arrived_us": 340639, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 360000, + "arrived_us": 380654, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 400000, + "arrived_us": 420668, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 243169, + "arrived_us": 263538, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 280624, + "arrived_us": 301008, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 320639, + "arrived_us": 341038, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 360654, + "arrived_us": 381068, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 400668, + "arrived_us": 421096, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 46000 + ], + [ + 0, + 2, + 48000 + ], + [ + 0, + 3, + 58000 + ], + [ + 1, + 1, + 46000 + ], + [ + 1, + 2, + 50000 + ], + [ + 1, + 3, + 60000 + ], + [ + 2, + 1, + 46000 + ], + [ + 2, + 2, + 52000 + ], + [ + 2, + 3, + 62000 + ], + [ + 3, + 1, + 46000 + ], + [ + 3, + 2, + 54000 + ], + [ + 3, + 3, + 64000 + ], + [ + 4, + 1, + 46000 + ], + [ + 4, + 2, + 56000 + ], + [ + 4, + 3, + 66000 + ], + [ + 5, + 1, + 70000 + ], + [ + 5, + 2, + 72000 + ], + [ + 5, + 3, + 74000 + ], + [ + 6, + 1, + 70000 + ], + [ + 6, + 2, + 102620 + ], + [ + 6, + 3, + 104620 + ], + [ + 7, + 1, + 70000 + ], + [ + 7, + 2, + 142634 + ], + [ + 7, + 3, + 144634 + ], + [ + 8, + 1, + 70000 + ], + [ + 8, + 2, + 182649 + ], + [ + 8, + 3, + 184649 + ], + [ + 9, + 1, + 70000 + ], + [ + 9, + 2, + 222664 + ], + [ + 9, + 3, + 224664 + ], + [ + 10, + 1, + 266560 + ], + [ + 10, + 2, + 268560 + ], + [ + 10, + 3, + 270560 + ], + [ + 11, + 1, + 266560 + ], + [ + 11, + 2, + 302624 + ], + [ + 11, + 3, + 304624 + ], + [ + 12, + 1, + 266560 + ], + [ + 12, + 2, + 342639 + ], + [ + 12, + 3, + 344639 + ], + [ + 13, + 1, + 266560 + ], + [ + 13, + 2, + 382654 + ], + [ + 13, + 3, + 384654 + ], + [ + 14, + 1, + 266560 + ], + [ + 14, + 2, + 422668 + ], + [ + 14, + 3, + 424668 + ] + ], + "bfpf_sched": 14707, + "tfpf_us": 46000, + "tffs_us": [ + 46000, + 48000, + 58000 + ], + "offered": 55650, + "delivered": 55650, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 3720 + ], + [ + 6, + 3720 + ], + [ + 7, + 3720 + ], + [ + 8, + 3720 + ], + [ + 9, + 3720 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/capped", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 20000, + "arrived_us": 42000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 26840, + "arrived_us": 47440, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 27440, + "arrived_us": 48055, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 28055, + "arrived_us": 48685, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 28685, + "arrived_us": 49330, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 29330, + "arrived_us": 49990, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g0-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g0-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g0-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g0-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g1-base", + "sent_first_us": 22000, + "arrived_us": 44280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 29990, + "arrived_us": 50595, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 30595, + "arrived_us": 51215, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 31215, + "arrived_us": 51849, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 31849, + "arrived_us": 52498, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 32498, + "arrived_us": 53162, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g1-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g1-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g1-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g1-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g2-base", + "sent_first_us": 24280, + "arrived_us": 46840, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 33162, + "arrived_us": 53771, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 33771, + "arrived_us": 54395, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 34395, + "arrived_us": 55034, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 35034, + "arrived_us": 55688, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 35688, + "arrived_us": 56356, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g2-l2-u1", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g2-l2-u2", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g2-l2-u3", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + }, + { + "id": "g2-l2-u4", + "sent_first_us": null, + "arrived_us": null, + "state": "capped" + } + ], + "presentations": [ + [ + 0, + 1, + 46000 + ], + [ + 0, + 2, + 56000 + ], + [ + 1, + 1, + 46000 + ], + [ + 1, + 2, + 58000 + ], + [ + 2, + 1, + 46000 + ], + [ + 2, + 2, + 60000 + ], + [ + 3, + 1, + 46000 + ], + [ + 3, + 2, + 62000 + ], + [ + 4, + 1, + 46000 + ], + [ + 4, + 2, + 64000 + ], + [ + 5, + 1, + 50000 + ], + [ + 5, + 2, + 66000 + ], + [ + 6, + 1, + 50000 + ], + [ + 6, + 2, + 68000 + ], + [ + 7, + 1, + 50000 + ], + [ + 7, + 2, + 70000 + ], + [ + 8, + 1, + 50000 + ], + [ + 8, + 2, + 72000 + ], + [ + 9, + 1, + 50000 + ], + [ + 9, + 2, + 74000 + ], + [ + 10, + 1, + 54000 + ], + [ + 10, + 2, + 76000 + ], + [ + 11, + 1, + 54000 + ], + [ + 11, + 2, + 78000 + ], + [ + 12, + 1, + 54000 + ], + [ + 12, + 2, + 80000 + ], + [ + 13, + 1, + 54000 + ], + [ + 13, + 2, + 82000 + ], + [ + 14, + 1, + 54000 + ], + [ + 14, + 2, + 84000 + ] + ], + "bfpf_sched": 14300, + "tfpf_us": 46000, + "tffs_us": [ + 46000, + 56000, + null + ], + "offered": 40875, + "delivered": 40875, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 14775, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/live-playout", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 200000, + "arrived_us": 222000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 202000, + "arrived_us": 222600, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 202600, + "arrived_us": 223215, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 203215, + "arrived_us": 223845, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 203845, + "arrived_us": 224490, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 204490, + "arrived_us": 225150, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 205150, + "arrived_us": 225510, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 205510, + "arrived_us": 225885, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 205885, + "arrived_us": 226275, + "state": "delivered" + }, + { + "id": "g0-l2-u3", + "sent_first_us": 206275, + "arrived_us": 226680, + "state": "delivered" + }, + { + "id": "g0-l2-u4", + "sent_first_us": 206680, + "arrived_us": 227100, + "state": "delivered" + }, + { + "id": "g1-base", + "sent_first_us": 400000, + "arrived_us": 422280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 402280, + "arrived_us": 422885, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 402885, + "arrived_us": 423505, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 403505, + "arrived_us": 424139, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 404139, + "arrived_us": 424788, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 404788, + "arrived_us": 425452, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 405452, + "arrived_us": 425817, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 405817, + "arrived_us": 426197, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 406197, + "arrived_us": 426591, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 406591, + "arrived_us": 427000, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 407000, + "arrived_us": 427424, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 600000, + "arrived_us": 622560, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 602560, + "arrived_us": 623169, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 603169, + "arrived_us": 623793, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 603793, + "arrived_us": 624432, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 604432, + "arrived_us": 625086, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 605086, + "arrived_us": 625754, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 605754, + "arrived_us": 626123, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 606123, + "arrived_us": 626507, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 606507, + "arrived_us": 626906, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 606906, + "arrived_us": 627320, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 607320, + "arrived_us": 627748, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 226000 + ], + [ + 1, + 1, + 226000 + ], + [ + 1, + 2, + 228000 + ], + [ + 1, + 3, + 232000 + ], + [ + 2, + 1, + 226000 + ], + [ + 2, + 2, + 230000 + ], + [ + 2, + 3, + 234000 + ], + [ + 3, + 1, + 226000 + ], + [ + 3, + 2, + 230000 + ], + [ + 3, + 3, + 234000 + ], + [ + 4, + 1, + 226000 + ], + [ + 4, + 2, + 232000 + ], + [ + 4, + 3, + 236000 + ], + [ + 5, + 1, + 426280 + ], + [ + 6, + 1, + 426280 + ], + [ + 6, + 2, + 428280 + ], + [ + 6, + 3, + 432280 + ], + [ + 7, + 1, + 426280 + ], + [ + 7, + 2, + 430280 + ], + [ + 7, + 3, + 434280 + ], + [ + 8, + 1, + 426280 + ], + [ + 8, + 2, + 430280 + ], + [ + 8, + 3, + 434280 + ], + [ + 9, + 1, + 426280 + ], + [ + 9, + 2, + 432280 + ], + [ + 9, + 3, + 436280 + ], + [ + 10, + 1, + 626560 + ], + [ + 11, + 1, + 626560 + ], + [ + 11, + 2, + 628560 + ], + [ + 11, + 3, + 632560 + ], + [ + 12, + 1, + 626560 + ], + [ + 12, + 2, + 630560 + ], + [ + 12, + 3, + 634560 + ], + [ + 13, + 1, + 626560 + ], + [ + 13, + 2, + 630560 + ], + [ + 13, + 3, + 634560 + ], + [ + 14, + 1, + 626560 + ], + [ + 14, + 2, + 632560 + ], + [ + 14, + 3, + 636560 + ] + ], + "bfpf_sched": 14707, + "tfpf_us": 226000, + "tffs_us": [ + 226000, + null, + null + ], + "offered": 55650, + "delivered": 55650, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 6, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "gop3/live-playout-present", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "g0-base", + "sent_first_us": 200000, + "arrived_us": 222000, + "state": "delivered" + }, + { + "id": "g0-l1-u0", + "sent_first_us": 202000, + "arrived_us": 222600, + "state": "delivered" + }, + { + "id": "g0-l1-u1", + "sent_first_us": 202600, + "arrived_us": 223215, + "state": "delivered" + }, + { + "id": "g0-l1-u2", + "sent_first_us": 203215, + "arrived_us": 223845, + "state": "delivered" + }, + { + "id": "g0-l1-u3", + "sent_first_us": 203845, + "arrived_us": 224490, + "state": "delivered" + }, + { + "id": "g0-l1-u4", + "sent_first_us": 204490, + "arrived_us": 225150, + "state": "delivered" + }, + { + "id": "g0-l2-u0", + "sent_first_us": 205150, + "arrived_us": 225510, + "state": "delivered" + }, + { + "id": "g0-l2-u1", + "sent_first_us": 205510, + "arrived_us": 225885, + "state": "delivered" + }, + { + "id": "g0-l2-u2", + "sent_first_us": 205885, + "arrived_us": 226275, + "state": "delivered" + }, + { + "id": "g0-l2-u3", + "sent_first_us": 206275, + "arrived_us": 226680, + "state": "delivered" + }, + { + "id": "g0-l2-u4", + "sent_first_us": 206680, + "arrived_us": 227100, + "state": "delivered" + }, + { + "id": "g1-base", + "sent_first_us": 400000, + "arrived_us": 422280, + "state": "delivered" + }, + { + "id": "g1-l1-u0", + "sent_first_us": 402280, + "arrived_us": 422885, + "state": "delivered" + }, + { + "id": "g1-l1-u1", + "sent_first_us": 402885, + "arrived_us": 423505, + "state": "delivered" + }, + { + "id": "g1-l1-u2", + "sent_first_us": 403505, + "arrived_us": 424139, + "state": "delivered" + }, + { + "id": "g1-l1-u3", + "sent_first_us": 404139, + "arrived_us": 424788, + "state": "delivered" + }, + { + "id": "g1-l1-u4", + "sent_first_us": 404788, + "arrived_us": 425452, + "state": "delivered" + }, + { + "id": "g1-l2-u0", + "sent_first_us": 405452, + "arrived_us": 425817, + "state": "delivered" + }, + { + "id": "g1-l2-u1", + "sent_first_us": 405817, + "arrived_us": 426197, + "state": "delivered" + }, + { + "id": "g1-l2-u2", + "sent_first_us": 406197, + "arrived_us": 426591, + "state": "delivered" + }, + { + "id": "g1-l2-u3", + "sent_first_us": 406591, + "arrived_us": 427000, + "state": "delivered" + }, + { + "id": "g1-l2-u4", + "sent_first_us": 407000, + "arrived_us": 427424, + "state": "delivered" + }, + { + "id": "g2-base", + "sent_first_us": 600000, + "arrived_us": 622560, + "state": "delivered" + }, + { + "id": "g2-l1-u0", + "sent_first_us": 602560, + "arrived_us": 623169, + "state": "delivered" + }, + { + "id": "g2-l1-u1", + "sent_first_us": 603169, + "arrived_us": 623793, + "state": "delivered" + }, + { + "id": "g2-l1-u2", + "sent_first_us": 603793, + "arrived_us": 624432, + "state": "delivered" + }, + { + "id": "g2-l1-u3", + "sent_first_us": 604432, + "arrived_us": 625086, + "state": "delivered" + }, + { + "id": "g2-l1-u4", + "sent_first_us": 605086, + "arrived_us": 625754, + "state": "delivered" + }, + { + "id": "g2-l2-u0", + "sent_first_us": 605754, + "arrived_us": 626123, + "state": "delivered" + }, + { + "id": "g2-l2-u1", + "sent_first_us": 606123, + "arrived_us": 626507, + "state": "delivered" + }, + { + "id": "g2-l2-u2", + "sent_first_us": 606507, + "arrived_us": 626906, + "state": "delivered" + }, + { + "id": "g2-l2-u3", + "sent_first_us": 606906, + "arrived_us": 627320, + "state": "delivered" + }, + { + "id": "g2-l2-u4", + "sent_first_us": 607320, + "arrived_us": 627748, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 226000 + ], + [ + 0, + 2, + 228000 + ], + [ + 0, + 3, + 232000 + ], + [ + 1, + 1, + 226000 + ], + [ + 1, + 2, + 228000 + ], + [ + 1, + 3, + 234000 + ], + [ + 2, + 1, + 226000 + ], + [ + 2, + 2, + 230000 + ], + [ + 2, + 3, + 234000 + ], + [ + 3, + 1, + 226000 + ], + [ + 3, + 2, + 230000 + ], + [ + 3, + 3, + 236000 + ], + [ + 4, + 1, + 226000 + ], + [ + 4, + 2, + 232000 + ], + [ + 4, + 3, + 236000 + ], + [ + 5, + 1, + 426280 + ], + [ + 5, + 2, + 428280 + ], + [ + 5, + 3, + 432280 + ], + [ + 6, + 1, + 426280 + ], + [ + 6, + 2, + 428280 + ], + [ + 6, + 3, + 434280 + ], + [ + 7, + 1, + 426280 + ], + [ + 7, + 2, + 430280 + ], + [ + 7, + 3, + 434280 + ], + [ + 8, + 1, + 426280 + ], + [ + 8, + 2, + 430280 + ], + [ + 8, + 3, + 436280 + ], + [ + 9, + 1, + 426280 + ], + [ + 9, + 2, + 432280 + ], + [ + 9, + 3, + 436280 + ], + [ + 10, + 1, + 626560 + ], + [ + 10, + 2, + 628560 + ], + [ + 10, + 3, + 632560 + ], + [ + 11, + 1, + 626560 + ], + [ + 11, + 2, + 628560 + ], + [ + 11, + 3, + 634560 + ], + [ + 12, + 1, + 626560 + ], + [ + 12, + 2, + 630560 + ], + [ + 12, + 3, + 634560 + ], + [ + 13, + 1, + 626560 + ], + [ + 13, + 2, + 630560 + ], + [ + 13, + 3, + 636560 + ], + [ + 14, + 1, + 626560 + ], + [ + 14, + 2, + 632560 + ], + [ + 14, + 3, + 636560 + ] + ], + "bfpf_sched": 14707, + "tfpf_us": 226000, + "tffs_us": [ + 226000, + 228000, + 232000 + ], + "offered": 55650, + "delivered": 55650, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 9, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ], + [ + 5, + 0 + ], + [ + 6, + 0 + ], + [ + 7, + 0 + ], + [ + 8, + 0 + ], + [ + 9, + 0 + ], + [ + 10, + 0 + ], + [ + 11, + 0 + ], + [ + 12, + 0 + ], + [ + 13, + 0 + ], + [ + 14, + 0 + ] + ] + } + }, + { + "name": "reorder-link", + "stall_bound_us": 1100, + "max_stall_us": 1000, + "result": { + "objects": [ + { + "id": "B0", + "sent_first_us": 0, + "arrived_us": 10, + "state": "delivered" + }, + { + "id": "R0", + "sent_first_us": 10, + "arrived_us": 1010, + "state": "delivered" + }, + { + "id": "Bb", + "sent_first_us": 1010, + "arrived_us": 1020, + "state": "delivered" + }, + { + "id": "Ba", + "sent_first_us": 1020, + "arrived_us": 51020, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 110 + ], + [ + 0, + 2, + 1110 + ], + [ + 1, + 1, + 51120 + ], + [ + 2, + 1, + 1210 + ] + ], + "bfpf_sched": 10, + "tfpf_us": 110, + "tffs_us": [ + 110, + 1110 + ], + "offered": 51020, + "delivered": 51020, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 995 + ], + [ + 2, + 1000 + ] + ] + } + }, + { + "name": "reorder-decode", + "stall_bound_us": 1120, + "max_stall_us": 100, + "result": { + "objects": [ + { + "id": "B0", + "sent_first_us": 0, + "arrived_us": 10, + "state": "delivered" + }, + { + "id": "R0", + "sent_first_us": 10, + "arrived_us": 20, + "state": "delivered" + }, + { + "id": "Bb", + "sent_first_us": 20, + "arrived_us": 100020, + "state": "delivered" + }, + { + "id": "Ba", + "sent_first_us": 100020, + "arrived_us": 100030, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 100010 + ], + [ + 0, + 2, + 100110 + ], + [ + 1, + 1, + 300110 + ], + [ + 2, + 1, + 200110 + ] + ], + "bfpf_sched": 99020, + "tfpf_us": 100010, + "tffs_us": [ + 100010, + 100110 + ], + "offered": 100030, + "delivered": 100030, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 100 + ], + [ + 2, + 100 + ] + ] + } + }, + { + "name": "audio/strict-subgroup", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 40718, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 20718, + "arrived_us": 44859, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 24859, + "arrived_us": 47603, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 27603, + "arrived_us": 50336, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 44718 + ], + [ + 0, + 2, + 46859 + ], + [ + 0, + 3, + 49603 + ], + [ + 0, + 4, + 52336 + ] + ], + "bfpf_sched": 11394, + "tfpf_us": 44718, + "tffs_us": [ + 44718, + 46859, + 49603, + 52336 + ], + "offered": 25835, + "delivered": 25835, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "audio/fifo-timeout-subgroup", + "stall_bound_us": 35000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 44485, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 24485, + "arrived_us": null, + "state": "abandoned" + }, + { + "id": "L02.wai", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "L03.wai", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + } + ], + "presentations": [ + [ + 0, + 1, + 48485 + ] + ], + "bfpf_sched": 2994, + "tfpf_us": 48485, + "tffs_us": [ + 48485, + null, + null, + null + ], + "offered": 25835, + "delivered": 1794, + "dropped": 13690, + "abandoned": 10351, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "audio/fifo-timeout-datagram", + "stall_bound_us": 62000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 44485, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 24485, + "arrived_us": 70363, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + }, + { + "id": "L03.wai", + "sent_first_us": null, + "arrived_us": null, + "state": "dropped" + } + ], + "presentations": [ + [ + 0, + 1, + 48485 + ], + [ + 0, + 2, + 72363 + ] + ], + "bfpf_sched": 1794, + "tfpf_us": 48485, + "tffs_us": [ + 48485, + 72363, + null, + null + ], + "offered": 25835, + "delivered": 12145, + "dropped": 13690, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "audio/fifo-no-timeout", + "stall_bound_us": null, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 40718, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 20718, + "arrived_us": 44859, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 24859, + "arrived_us": 47603, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 27603, + "arrived_us": 50336, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 44718 + ], + [ + 0, + 2, + 46859 + ], + [ + 0, + 3, + 49603 + ], + [ + 0, + 4, + 52336 + ] + ], + "bfpf_sched": 11394, + "tfpf_us": 44718, + "tffs_us": [ + 44718, + 46859, + 49603, + 52336 + ], + "offered": 25835, + "delivered": 25835, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "clip/strict-subgroup", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 41008, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 21008, + "arrived_us": 41267, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 21267, + "arrived_us": 41528, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 21528, + "arrived_us": 41787, + "state": "delivered" + }, + { + "id": "L04.wai", + "sent_first_us": 21787, + "arrived_us": 42051, + "state": "delivered" + }, + { + "id": "L05.wai", + "sent_first_us": 22051, + "arrived_us": 42318, + "state": "delivered" + }, + { + "id": "L06.wai", + "sent_first_us": 22318, + "arrived_us": 42545, + "state": "delivered" + }, + { + "id": "L07.wai", + "sent_first_us": 22545, + "arrived_us": 42771, + "state": "delivered" + }, + { + "id": "L08.wai", + "sent_first_us": 22771, + "arrived_us": 42995, + "state": "delivered" + }, + { + "id": "L09.wai", + "sent_first_us": 22995, + "arrived_us": 43217, + "state": "delivered" + }, + { + "id": "L10.wai", + "sent_first_us": 23217, + "arrived_us": 43444, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 45008 + ], + [ + 0, + 2, + 47008 + ], + [ + 0, + 3, + 57008 + ], + [ + 1, + 1, + 45008 + ], + [ + 1, + 2, + 49008 + ], + [ + 1, + 3, + 59008 + ], + [ + 2, + 1, + 45008 + ], + [ + 2, + 2, + 51008 + ], + [ + 2, + 3, + 61008 + ], + [ + 3, + 1, + 45008 + ], + [ + 3, + 2, + 53008 + ], + [ + 3, + 3, + 63008 + ], + [ + 4, + 1, + 45008 + ], + [ + 4, + 2, + 55008 + ], + [ + 4, + 3, + 65008 + ] + ], + "bfpf_sched": 8601, + "tfpf_us": 45008, + "tffs_us": [ + 45008, + 47008, + 57008 + ], + "offered": 8601, + "delivered": 8601, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ] + ] + } + }, + { + "name": "clip/fifo-timeout-subgroup", + "stall_bound_us": 35000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 46298, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 26298, + "arrived_us": 47916, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 27916, + "arrived_us": 49544, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 29544, + "arrived_us": 51162, + "state": "delivered" + }, + { + "id": "L04.wai", + "sent_first_us": 31162, + "arrived_us": 52810, + "state": "delivered" + }, + { + "id": "L05.wai", + "sent_first_us": 32810, + "arrived_us": 54478, + "state": "delivered" + }, + { + "id": "L06.wai", + "sent_first_us": 34478, + "arrived_us": 55896, + "state": "delivered" + }, + { + "id": "L07.wai", + "sent_first_us": 35896, + "arrived_us": 57304, + "state": "delivered" + }, + { + "id": "L08.wai", + "sent_first_us": 37304, + "arrived_us": 58702, + "state": "delivered" + }, + { + "id": "L09.wai", + "sent_first_us": 38702, + "arrived_us": 60090, + "state": "delivered" + }, + { + "id": "L10.wai", + "sent_first_us": 40090, + "arrived_us": 61508, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 50298 + ], + [ + 0, + 2, + 52298 + ], + [ + 0, + 3, + 62298 + ], + [ + 1, + 1, + 50298 + ], + [ + 1, + 2, + 54298 + ], + [ + 1, + 3, + 64298 + ], + [ + 2, + 1, + 50298 + ], + [ + 2, + 2, + 56298 + ], + [ + 2, + 3, + 66298 + ], + [ + 3, + 1, + 50298 + ], + [ + 3, + 2, + 58298 + ], + [ + 3, + 3, + 68298 + ], + [ + 4, + 1, + 50298 + ], + [ + 4, + 2, + 60298 + ], + [ + 4, + 3, + 70298 + ] + ], + "bfpf_sched": 3817, + "tfpf_us": 50298, + "tffs_us": [ + 50298, + 52298, + 62298 + ], + "offered": 8601, + "delivered": 8601, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ] + ] + } + }, + { + "name": "clip/fifo-timeout-datagram", + "stall_bound_us": 62000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 46298, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 26298, + "arrived_us": 47916, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 27916, + "arrived_us": 49544, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 29544, + "arrived_us": 51162, + "state": "delivered" + }, + { + "id": "L04.wai", + "sent_first_us": 31162, + "arrived_us": 52810, + "state": "delivered" + }, + { + "id": "L05.wai", + "sent_first_us": 32810, + "arrived_us": 54478, + "state": "delivered" + }, + { + "id": "L06.wai", + "sent_first_us": 34478, + "arrived_us": 55896, + "state": "delivered" + }, + { + "id": "L07.wai", + "sent_first_us": 35896, + "arrived_us": 57304, + "state": "delivered" + }, + { + "id": "L08.wai", + "sent_first_us": 37304, + "arrived_us": 58702, + "state": "delivered" + }, + { + "id": "L09.wai", + "sent_first_us": 38702, + "arrived_us": 60090, + "state": "delivered" + }, + { + "id": "L10.wai", + "sent_first_us": 40090, + "arrived_us": 61508, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 50298 + ], + [ + 0, + 2, + 52298 + ], + [ + 0, + 3, + 62298 + ], + [ + 1, + 1, + 50298 + ], + [ + 1, + 2, + 54298 + ], + [ + 1, + 3, + 64298 + ], + [ + 2, + 1, + 50298 + ], + [ + 2, + 2, + 56298 + ], + [ + 2, + 3, + 66298 + ], + [ + 3, + 1, + 50298 + ], + [ + 3, + 2, + 58298 + ], + [ + 3, + 3, + 68298 + ], + [ + 4, + 1, + 50298 + ], + [ + 4, + 2, + 60298 + ], + [ + 4, + 3, + 70298 + ] + ], + "bfpf_sched": 3817, + "tfpf_us": 50298, + "tffs_us": [ + 50298, + 52298, + 62298 + ], + "offered": 8601, + "delivered": 8601, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ] + ] + } + }, + { + "name": "clip/fifo-no-timeout", + "stall_bound_us": null, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 41008, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 21008, + "arrived_us": 41267, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 21267, + "arrived_us": 41528, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 21528, + "arrived_us": 41787, + "state": "delivered" + }, + { + "id": "L04.wai", + "sent_first_us": 21787, + "arrived_us": 42051, + "state": "delivered" + }, + { + "id": "L05.wai", + "sent_first_us": 22051, + "arrived_us": 42318, + "state": "delivered" + }, + { + "id": "L06.wai", + "sent_first_us": 22318, + "arrived_us": 42545, + "state": "delivered" + }, + { + "id": "L07.wai", + "sent_first_us": 22545, + "arrived_us": 42771, + "state": "delivered" + }, + { + "id": "L08.wai", + "sent_first_us": 22771, + "arrived_us": 42995, + "state": "delivered" + }, + { + "id": "L09.wai", + "sent_first_us": 22995, + "arrived_us": 43217, + "state": "delivered" + }, + { + "id": "L10.wai", + "sent_first_us": 23217, + "arrived_us": 43444, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 45008 + ], + [ + 0, + 2, + 47008 + ], + [ + 0, + 3, + 57008 + ], + [ + 1, + 1, + 45008 + ], + [ + 1, + 2, + 49008 + ], + [ + 1, + 3, + 59008 + ], + [ + 2, + 1, + 45008 + ], + [ + 2, + 2, + 51008 + ], + [ + 2, + 3, + 61008 + ], + [ + 3, + 1, + 45008 + ], + [ + 3, + 2, + 53008 + ], + [ + 3, + 3, + 63008 + ], + [ + 4, + 1, + 45008 + ], + [ + 4, + 2, + 55008 + ], + [ + 4, + 3, + 65008 + ] + ], + "bfpf_sched": 8601, + "tfpf_us": 45008, + "tffs_us": [ + 45008, + 47008, + 57008 + ], + "offered": 8601, + "delivered": 8601, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ], + [ + 1, + 0 + ], + [ + 2, + 0 + ], + [ + 3, + 0 + ], + [ + 4, + 0 + ] + ] + } + }, + { + "name": "splat/strict-subgroup", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 41328, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 21328, + "arrived_us": 43384, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 23384, + "arrived_us": 44736, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 45328 + ], + [ + 0, + 2, + 47328 + ], + [ + 0, + 3, + 49328 + ] + ], + "bfpf_sched": 11839, + "tfpf_us": 45328, + "tffs_us": [ + 45328, + 47328, + 49328 + ], + "offered": 11839, + "delivered": 11839, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "splat/fifo-timeout-subgroup", + "stall_bound_us": 35000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 48298, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 28298, + "arrived_us": 61148, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 41148, + "arrived_us": 69598, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 52298 + ], + [ + 0, + 2, + 63148 + ], + [ + 0, + 3, + 71598 + ] + ], + "bfpf_sched": 4519, + "tfpf_us": 52298, + "tffs_us": [ + 52298, + 63148, + 71598 + ], + "offered": 11839, + "delivered": 11839, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "splat/fifo-timeout-datagram", + "stall_bound_us": 62000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 48298, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 28298, + "arrived_us": 61148, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 41148, + "arrived_us": 69598, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 52298 + ], + [ + 0, + 2, + 63148 + ], + [ + 0, + 3, + 71598 + ] + ], + "bfpf_sched": 3319, + "tfpf_us": 52298, + "tffs_us": [ + 52298, + 63148, + 71598 + ], + "offered": 11839, + "delivered": 11839, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "splat/fifo-no-timeout", + "stall_bound_us": null, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 41328, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 21328, + "arrived_us": 43384, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 23384, + "arrived_us": 44736, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 45328 + ], + [ + 0, + 2, + 47328 + ], + [ + 0, + 3, + 49328 + ] + ], + "bfpf_sched": 11839, + "tfpf_us": 45328, + "tffs_us": [ + 45328, + 47328, + 49328 + ], + "offered": 11839, + "delivered": 11839, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "still/strict-subgroup", + "stall_bound_us": 2480, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 40184, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 20184, + "arrived_us": 40565, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 20565, + "arrived_us": 40944, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 20944, + "arrived_us": 41322, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 44184 + ], + [ + 0, + 2, + 46184 + ], + [ + 0, + 3, + 48184 + ], + [ + 0, + 4, + 50184 + ] + ], + "bfpf_sched": 3301, + "tfpf_us": 44184, + "tffs_us": [ + 44184, + 46184, + 48184, + 50184 + ], + "offered": 3301, + "delivered": 3301, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "still/fifo-timeout-subgroup", + "stall_bound_us": 35000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 41150, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 21150, + "arrived_us": 43528, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 23528, + "arrived_us": 45896, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 25896, + "arrived_us": 48254, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 45150 + ], + [ + 0, + 2, + 47150 + ], + [ + 0, + 3, + 49150 + ], + [ + 0, + 4, + 51150 + ] + ], + "bfpf_sched": 1411, + "tfpf_us": 45150, + "tffs_us": [ + 45150, + 47150, + 49150, + 51150 + ], + "offered": 3301, + "delivered": 3301, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "still/fifo-timeout-datagram", + "stall_bound_us": 62000, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 41150, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 21150, + "arrived_us": 43528, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 23528, + "arrived_us": 45896, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 25896, + "arrived_us": 48254, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 45150 + ], + [ + 0, + 2, + 47150 + ], + [ + 0, + 3, + 49150 + ], + [ + 0, + 4, + 51150 + ] + ], + "bfpf_sched": 1411, + "tfpf_us": 45150, + "tffs_us": [ + 45150, + 47150, + 49150, + 51150 + ], + "offered": 3301, + "delivered": 3301, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + }, + { + "name": "still/fifo-no-timeout", + "stall_bound_us": null, + "max_stall_us": 0, + "result": { + "objects": [ + { + "id": "base.wai", + "sent_first_us": 20000, + "arrived_us": 40184, + "state": "delivered" + }, + { + "id": "L01.wai", + "sent_first_us": 20184, + "arrived_us": 40565, + "state": "delivered" + }, + { + "id": "L02.wai", + "sent_first_us": 20565, + "arrived_us": 40944, + "state": "delivered" + }, + { + "id": "L03.wai", + "sent_first_us": 20944, + "arrived_us": 41322, + "state": "delivered" + } + ], + "presentations": [ + [ + 0, + 1, + 44184 + ], + [ + 0, + 2, + 46184 + ], + [ + 0, + 3, + 48184 + ], + [ + 0, + 4, + 50184 + ] + ], + "bfpf_sched": 3301, + "tfpf_us": 44184, + "tffs_us": [ + 44184, + 46184, + 48184, + 50184 + ], + "offered": 3301, + "delivered": 3301, + "dropped": 0, + "abandoned": 0, + "never_sent_capped": 0, + "missed": 0, + "stall_us": [ + [ + 0, + 0 + ] + ] + } + } + ] +} diff --git a/wai/staged-measure-conformance/sim_verify.py b/wai/staged-measure-conformance/sim_verify.py new file mode 100755 index 000000000..8c72c11bf --- /dev/null +++ b/wai/staged-measure-conformance/sim_verify.py @@ -0,0 +1,329 @@ +#!/usr/bin/env python3 +"""Independent verifier of the delivery simulator's corpus (extensions/staged-measurement.md +§10, Appendix B). + +Written from the extension text; it shares no code with wai-rs, which writes the +corpus. It implements Appendix B itself: availability, eligibility and the sender's +order, the link chunk by chunk, delivery timeouts (a datagram not started by its +deadline is dropped; a subgroup is reset at the deadline of an object not fully sent, +its sent bytes abandoned and the rest dropped), the layer cap, the sink's workers and +decode order, playout deadlines, the outputs, conservation, the base-only plan and each +unit's stall, and §10.4's bound. A fixture's model comes from measure_verify.py's own +evaluation of the title (each object's counted bytes, base, layer and units). + +For every case of sim/cases.json it requires its own result, bound and largest stall +to equal sim/results.json's, and the case's claim to hold: `within-bound` (every stall +within the plan's bound), `exceeds:` (the largest stall passes that plan's bound), +or `unbounded` (the plan has none). It also requires the corpus to hold, for each part +of the bound a plan can lose (a fifo link with no refinement timeout; fifo decode), a +case whose stall passes a bounded plan's bound. + + python3 sim_verify.py + +Exit status 0 only if every case verifies. +""" +import json +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import measure_verify as MV # noqa: E402 + + +def ceil_div(a, b): + return -(-a // b) + + +def check_model(m): + covered = {} + for o in m["objects"]: + frm, cnt = o["units"] + if o["bytes"] < 1 or cnt < 1: + raise ValueError(f"{o['id']}: no bytes or no units") + for u in range(frm, frm + cnt): + if (u, o["layer"]) in covered: + raise ValueError(f"{o['id']}: unit {u} covered twice") + covered[(u, o["layer"])] = o["base"] + for o in m["objects"]: + if o["layer"] >= 1: + frm, cnt = o["units"] + for u in range(frm, frm + cnt): + if covered.get((u, o["layer"] - 1)) != o["base"]: + raise ValueError(f"{o['id']}: nothing below unit {u}") + + +def run_link(m, p): + objs = m["objects"] + n = len(objs) + snd, lnk = p["sender"], p["link"] + h = lnk["rtt_us"] // 2 + ser = lambda c: ceil_div(c * 1000, lnk["bytes_per_ms"]) # noqa: E731 + cap = snd["layer_cap"] + capped = [cap is not None and o["layer"] > cap for o in objs] + avail = [] + for o in objs: + if p["mode"] == "vod": + w = snd["vod_window_us"] + avail.append(h + (0 if w is None else max(0, o["units"][0] * m["unit_us"] - w))) + else: + if o["capture_us"] is None: + raise ValueError(f"{o['id']}: no capture time") + avail.append(max(h, o["capture_us"])) + to = snd["delivery_timeout_us"] + deadline = [None if (to["base"] if o["layer"] == 0 else to["refinement"]) is None + else avail[i] + (to["base"] if o["layer"] == 0 else to["refinement"]) for i, o in enumerate(objs)] + datagram = snd["carriage"] == "datagram" + if datagram and any(not capped[i] and o["bytes"] > lnk["chunk_bytes"] for i, o in enumerate(objs)): + raise ValueError("datagram-too-large") + state = ["capped" if c else None for c in capped] + pending = [not c for c in capped] + sent = [0] * n + first = [None] * n + arrived = [None] * n + chunks = [] + dropped = abandoned = 0 + sub = [(o["base"], o["layer"]) for o in objs] + t = 0 + while True: + for i in range(n): + if not pending[i] or deadline[i] is None or deadline[i] >= t: + continue + if datagram: + pending[i] = False + state[i] = "dropped" + dropped += objs[i]["bytes"] + else: + for j in range(i, n): + if pending[j] and sub[j] == sub[i]: + pending[j] = False + abandoned += sent[j] + dropped += objs[j]["bytes"] - sent[j] + state[j] = "abandoned" if sent[j] > 0 else "dropped" + + def eligible(i): + if not pending[i] or avail[i] > t: + return False + return datagram or not any(pending[j] and sub[j] == sub[i] for j in range(i)) + el = [i for i in range(n) if eligible(i)] + if not el: + later = [avail[i] for i in range(n) if pending[i] and avail[i] > t] + if later: + t = min(later) + continue + if any(pending): + raise ValueError("stuck") + break + if snd["schedule"] == "strict-priority": + # Bases in the order they became available (Appendix B, Order). + i = min(el, key=lambda i: (objs[i]["layer"], avail[i] if objs[i]["layer"] == 0 else 0, objs[i]["units"][0], i)) + else: + i = min(el, key=lambda i: (avail[i], i)) + c = min(lnk["chunk_bytes"], objs[i]["bytes"] - sent[i]) + if sent[i] == 0: + first[i] = t + end = t + ser(c) + sent[i] += c + chunks.append((end + h, c)) + if sent[i] == objs[i]["bytes"] and (datagram or deadline[i] is None or end <= deadline[i]): + pending[i] = False + state[i] = "delivered" + arrived[i] = end + h + t = end + return state, first, arrived, chunks, dropped, abandoned + + +def run_once(m, p): + objs = m["objects"] + n = len(objs) + sink = p["sink"] + state, first, arrived, chunks, dropped, abandoned = run_link(m, p) + stage = {} # unit -> (stage, when) + done = [False] * n + running = [] # (end, i) + pres = [] + missed = 0 + jobs = [i for i in range(n) if state[i] == "delivered"] + + def ready_at(i): + o = objs[i] + r = arrived[i] + if o["layer"] >= 1: + frm, cnt = o["units"] + for u in range(frm, frm + cnt): + s = stage.get(u) + if s is None or s[0] != o["layer"] - 1: + return None + r = max(r, s[1]) + return r + t = 0 + while True: + running.sort() + while running and running[0][0] <= t: + end, i = running.pop(0) + o = objs[i] + frm, cnt = o["units"] + for u in range(frm, frm + cnt): + ps = sink["playout_start_us"] + late = ps is not None and end > ps + u * m["unit_us"] + if late: + missed += 1 + if late and o["layer"] >= 1 and sink["late_refinement"] == "discard": + continue + pres.append((u, o["layer"] + 1, end)) + stage[u] = (o["layer"], end) + while len(running) < sink["workers"]: + cands = [] + for i in jobs: + if done[i]: + continue + r = ready_at(i) + if r is not None and r <= t: + cands.append((i, r)) + if not cands: + break + if sink["decode_order"] == "base-first": + # Bases in the order they became ready (Appendix B, Workers). + i, _ = min(cands, key=lambda x: (objs[x[0]]["layer"], x[1] if objs[x[0]]["layer"] == 0 else 0, objs[x[0]]["units"][0], x[0])) + else: + i, _ = min(cands, key=lambda x: (x[1], x[0])) + done[i] = True + d = sink["decode_us"]["base"] if objs[i]["layer"] == 0 else sink["decode_us"]["refinement"] + running.append((t + d, i)) + nexts = [e for e, _ in running] + nexts += [r for i in jobs if not done[i] for r in [ready_at(i)] if r is not None and r > t] + if not nexts: + break + t = min(nexts) + pres.sort() + t0 = min((x[2] for x in pres if x[0] == 0 and x[1] == 1), default=None) + top = max(o["layer"] for o in objs) + tffs = [min((x[2] for x in pres if x[0] == 0 and x[1] == k + 1), default=None) for k in range(top + 1)] + bfpf = None if t0 is None else sum(c for a, c in chunks if a <= t0) + return { + "objects": [{"id": o["id"], "sent_first_us": first[i], "arrived_us": arrived[i], "state": state[i]} for i, o in enumerate(objs)], + "presentations": [list(x) for x in pres], + "bfpf_sched": bfpf, + "tfpf_us": t0, + "tffs_us": tffs, + "offered": sum(o["bytes"] for i, o in enumerate(objs) if state[i] != "capped"), + "delivered": sum(o["bytes"] for i, o in enumerate(objs) if state[i] == "delivered"), + "dropped": dropped, + "abandoned": abandoned, + "never_sent_capped": sum(o["bytes"] for i, o in enumerate(objs) if state[i] == "capped"), + "missed": missed, + } + + +def simulate(m, p): + check_model(m) + full = run_once(m, p) + base = run_once({"unit_us": m["unit_us"], "objects": [o for o in m["objects"] if o["layer"] == 0]}, p) + + def first_base(r): + out = {} + for u, l, t in r["presentations"]: + if l == 1 and u not in out: + out[u] = t + return out + fb, bb = first_base(full), first_base(base) + units = sorted(u for o in m["objects"] if o["layer"] == 0 for u in range(o["units"][0], o["units"][0] + o["units"][1])) + full["stall_us"] = [[u, fb[u] - bb[u] if u in fb and u in bb else None] for u in units] + return full + + +def bound(p): + snd, sink = p["sender"], p["sink"] + if snd["delivery_timeout_us"]["base"] is not None: + return None + chunk = ceil_div(p["link"]["chunk_bytes"] * 1000, p["link"]["bytes_per_ms"]) + if snd["schedule"] == "strict-priority": + link = chunk + elif snd["delivery_timeout_us"]["refinement"] is not None: + link = snd["delivery_timeout_us"]["refinement"] + chunk + else: + return None + if sink["decode_order"] != "base-first": + return None + return link + sink["decode_us"]["refinement"] + + +def max_stall(r): + m = 0 + for _, s in r["stall_us"]: + if s is None: + return None + m = max(m, s) + return m + + +def fixture_model(d, name): + rel = os.path.relpath(d, MV.WAI) + ev, _ = MV.evaluate(MV.Fixture(rel, name)) + caps = [b["capture_us"] for b in ev["bases"]] + return {"unit_us": ev["unit_us"], "objects": [ + {"id": o["file"].rsplit("/", 1)[-1], "base": o["base"], "layer": o["layer"], "units": o["units"], + "bytes": o["counted"]["total"], "capture_us": caps[o["base"]]} for o in ev["objects"]]} + + +def main(d): + d = os.path.abspath(d) + cases = json.load(open(os.path.join(d, "sim", "cases.json")))["cases"] + results = {r["name"]: r for r in json.load(open(os.path.join(d, "sim", "results.json")))["results"]} + fails = [] + shown = {"link": False, "decode": False} + if set(results) != {c["name"] for c in cases}: + fails.append("results.json does not cover exactly the cases") + for c in cases: + name = c["name"] + if c["model"].startswith("fixture:"): + m = fixture_model(d, c["model"].split(":", 1)[1]) + else: + m = json.load(open(os.path.join(d, "sim", c["model"]))) + p = json.load(open(os.path.join(d, "sim", c["plan"]))) + try: + r = simulate(m, p) + except ValueError as e: + fails.append(f"{name}: {e}") + continue + if r["offered"] != r["delivered"] + r["dropped"] + r["abandoned"]: + fails.append(f"{name}: bytes are not conserved") + b, w = bound(p), max_stall(r) + claim = c["claim"] + if claim == "within-bound": + if b is None or w is None or w > b: + fails.append(f"{name}: stall {w} not within the bound {b}") + elif claim == "unbounded": + if b is not None: + fails.append(f"{name}: the plan has a bound") + elif claim.startswith("exceeds:"): + other = json.load(open(os.path.join(d, "sim", "plans", claim.split(":", 1)[1] + ".json"))) + ob = bound(other) + if b is not None or ob is None or (w is not None and w <= ob): + fails.append(f"{name}: stall {w} does not pass {claim}'s bound {ob}") + else: + if p["sender"]["schedule"] == "fifo" and p["sender"]["delivery_timeout_us"]["refinement"] is None: + shown["link"] = True + if p["sink"]["decode_order"] == "fifo": + shown["decode"] = True + else: + fails.append(f"{name}: claim {claim}") + g = results.get(name) + if g is None: + continue + out = [] + MV.diff({"stall_bound_us": b, "max_stall_us": w, "result": r}, + {"stall_bound_us": g["stall_bound_us"], "max_stall_us": g["max_stall_us"], "result": g["result"]}, name, out) + fails += out[:10] + for part, ok in shown.items(): + if not ok: + fails.append(f"no case shows a stall past a bounded plan's bound for an unbounded {part} schedule") + if fails: + for x in fails: + print("FAIL", x) + sys.exit(1) + print(f"staged-measure sim: {len(cases)} cases verify independently") + + +if __name__ == "__main__": + main(sys.argv[1] if len(sys.argv) > 1 else HERE) diff --git a/wai/telemetry-conformance/.gitattributes b/wai/telemetry-conformance/.gitattributes new file mode 100644 index 000000000..eff1df73a --- /dev/null +++ b/wai/telemetry-conformance/.gitattributes @@ -0,0 +1,2 @@ +# Hashed files: a line-ending conversion on checkout would change every digest. +* -text diff --git a/wai/telemetry-conformance/README.md b/wai/telemetry-conformance/README.md new file mode 100644 index 000000000..fcd8c8e23 --- /dev/null +++ b/wai/telemetry-conformance/README.md @@ -0,0 +1,49 @@ +# telemetry-conformance — delivery telemetry + +The corpus of [`extensions/delivery-telemetry.md`](../extensions/delivery-telemetry.md) +§9: observations, their CMCD version 2 reports and CMSD fields, and session logs with +their metrics. Every value here is a string or an integer, the same on every machine. + +- `observations/` — one observation (§2) each, every event kind among them: requests + with the object's members, a response, the play states, the first presentation and a + later one (which CMCD does not report), stalls with their metrics, a switch, a layer + change, a dropped frame, dispatch outcomes, an abandon, an end with its claim pointer, + a low-latency live session, and strings RFC 9651 escapes (a quote and a backslash in + a capability in a custom key, a quote in a URL in an Event-mode report). +- `expected/cmcd.json` — for each observation: its mode (`request`, `event` or `none`), + its report (§3, §4), and for Request mode its header shards (§4.2). +- `invalid/` — observations every reader refuses (§2.4): a member unknown or missing, + the wrong version, a session id that is not hexadecimal, an Event-mode observation + with no time, an integer past 2⁵³ − 1, a time past an RFC 9651 Integer, a claim + before the end, an unknown event, a response key on a request and a request key on a + response, a switch to its own path or for an unknown reason, a pin refusal with no + reason, an unknown object type, a content id past 64 characters or holding an address + or a URL, an object URL with a query, a fragment, user information (behind `//`, behind + backslashes, behind no slash, or behind a mixed separator), a backslash, or a `@` in + its path (§8.2), a presentation at no layer, + an unknown cohort. + `expected/refused.json` lists them. +- `cmsd/` — CMSD-Static descriptions of a base and of a derived layer, and a + CMSD-Dynamic field from two nodes (§6); `expected/cmsd.json`, their fields. +- `logs/` — session logs (§7.1), one observation per line: a session with stalls, one + open at its end; an exit before start; a start failure. `expected/qoe.json`, their + metrics (§7.2). +- `cml/` — the differential (§9.3). + +**Verifiers.** + +- `wai_telemetry verify .` (from `../wai-rs`, feature `telemetry`): reads every input + from disk, requires each observation to write back as given, recomputes every + expected file and requires it byte for byte, reads every report back, and requires + every `invalid/` observation refused. `wai_telemetry gen .` writes the corpus. +- `cml/differential.mjs` (`npm ci && node differential.mjs ..` in `cml/`): builds each + report from the extension's tables in JavaScript, not from `wai-rs`'s output; encodes + it with an independent open-source CMCD and CMSD library at the versions + `package-lock.json` locks, in the mode the event fixes, with the header shards for + Request mode; requires the bytes of `expected/cmcd.json` and `expected/cmsd.json`; + reads the expected strings back with the library's dictionary, query and header + readers and requires the members; and validates every report, and the Event-mode + reports together, requiring no error. + +The content is synthetic: session ids, content hashes and timestamps are fixed values +chosen for the corpus. diff --git a/wai/telemetry-conformance/cml/.gitignore b/wai/telemetry-conformance/cml/.gitignore new file mode 100644 index 000000000..c2658d7d1 --- /dev/null +++ b/wai/telemetry-conformance/cml/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/wai/telemetry-conformance/cml/differential.mjs b/wai/telemetry-conformance/cml/differential.mjs new file mode 100644 index 000000000..b78de3803 --- /dev/null +++ b/wai/telemetry-conformance/cml/differential.mjs @@ -0,0 +1,158 @@ +// The delivery-telemetry corpus against an independent open-source CMCD and CMSD +// library (extensions/delivery-telemetry.md §9.3). +// +// For each observation it builds the report's members from the extension's tables +// (§3, §4) here, not from wai-rs's output; encodes them with the library in the mode the +// event fixes, and the header shards for Request mode; requires the bytes of +// expected/cmcd.json; reads the expected strings back with the library's dictionary, +// query and header readers and requires the members; and validates every report, and +// the Event-mode reports together, requiring no error. CMSD-Static and CMSD-Dynamic +// are encoded and read back the same way. Exit status 0 only if everything agrees. +// +// npm ci && node differential.mjs + +import fs from 'node:fs'; +import path from 'node:path'; +import { encodeCmcd, toCmcdHeaders, decodeCmcd, fromCmcdQuery, fromCmcdHeaders, validateCmcd, validateCmcdEvents } from '@svta/cml-cmcd'; +import { encodeCmsdStatic, encodeCmsdDynamic, decodeCmsdStatic, decodeCmsdDynamic } from '@svta/cml-cmsd'; +import { SfItem, SfToken } from '@svta/cml-structured-field-values'; + +const dir = path.resolve(process.argv[2] ?? '..'); +const read = (p) => JSON.parse(fs.readFileSync(path.join(dir, p), 'utf8')); +const P = 'science.transaction.wai-'; +const fails = []; +const fail = (s) => fails.push(s); + +// §4.1: the custom keys' header shards (Request mode). +const CUSTOM_HEADERS = { + 'CMCD-Object': ['b', 'tx', 'ch', 'cap', 'ri', 'l', 'lt', 'p', 'rcpt'].map((k) => P + k), + 'CMCD-Request': [P + 'lc'], + 'CMCD-Session': [P + 'coh'], +}; + +// §3.1's event table. +const EVENT = { + response: ['rr'], start: ['ps', 's'], present: ['ps', 'p'], 'stall-start': ['ps', 'r'], 'stall-end': ['ps', 'p'], + abandon: ['ps', 'q'], end: ['ps', 'e'], switch: ['ce', null, 'wai-switch'], capable: ['ce', null, 'wai-capable'], + 'layer-change': ['ce', null, 'wai-layer-change'], 'frame-drop': ['ce', null, 'wai-frame-drop'], +}; + +const pathToken = (p) => (p === 'primary' ? 'p' : 'f'); + +// An observation's report members, from the extension's tables. +function build(o) { + if (o.event === 'present' && !o.present.first) return null; + const mode = o.event === 'request' ? 'request' : 'event'; + const c = { v: 2, sid: o.sid, sn: o.sn, st: o.st, sf: o.sf }; + const put = (k, v) => { if (v !== null && v !== undefined) c[k] = v; }; + const custom = (k, v) => { if (v !== null && v !== undefined) c[P + k] = String(v); }; + put('cid', o.cid); + custom('coh', o.cohort); + if (o.metrics) { + put('msd', o.metrics.msd_ms); + if (o.metrics.bsa !== null) c.bsa = [o.metrics.bsa]; + if (o.metrics.bsda_ms !== null) c.bsda = [o.metrics.bsda_ms]; + put('dfa', o.metrics.dfa); + } + const ob = o.object; + if (ob) { + custom('b', ob.counted); custom('tx', ob.transferred); custom('ch', ob.content_hash); custom('cap', ob.capability); + custom('ri', ob.rendition); custom('l', ob.layer); custom('lt', ob.layers_total); + custom('p', ob.path === null ? null : pathToken(ob.path)); custom('rcpt', ob.receipt); + put('ot', ob.ot); put('d', ob.d_ms); put('dl', ob.dl_ms); custom('lc', ob.layer_cap); + put('rc', ob.rc); put('ttfb', ob.ttfb_ms); put('ttlb', ob.ttlb_ms); put('url', ob.url); + } + if (mode === 'event') { + const [e, sta, cen] = EVENT[o.event]; + c.e = e; c.ts = o.ts_ms; + put('sta', sta ?? null); put('cen', cen ?? null); + if (o.present) { custom('u', o.present.unit); custom('pl', o.present.layers); custom('late', o.present.late_ms); custom('dec', o.present.decode_us); } + if (o.switch) { custom('u', o.switch.unit); custom('p', pathToken(o.switch.to)); custom('sw', o.switch.reason); } + if (o.capable) { custom('co', o.capable.outcome); custom('cr', o.capable.reason); custom('cf', o.capable.classical_floor); } + if (o.layer_change) { custom('lc', o.layer_change.to); custom('lcf', o.layer_change.from); } + custom('clm', o.claim); + } + return { mode, c }; +} + +// Two members agree: the library reads Tokens as strings and inner lists as arrays. +function same(a, b) { + const norm = (v) => (v instanceof SfToken ? v.description ?? String(v) : typeof v === 'symbol' ? v.description : v); + const ka = Object.keys(a).sort(), kb = Object.keys(b).sort(); + if (JSON.stringify(ka) !== JSON.stringify(kb)) return false; + return ka.every((k) => JSON.stringify(norm(a[k])) === JSON.stringify(norm(b[k]))); +} + +const errors = (r) => (r.issues ?? []).filter((i) => i.severity === 'error'); + +const expected = read('expected/cmcd.json'); +const events = []; +for (const f of fs.readdirSync(path.join(dir, 'observations')).sort()) { + const name = f.replace(/\.json$/, ''); + const o = read(`observations/${f}`); + const x = expected[name]; + if (!x) { fail(`${name}: not in expected/cmcd.json`); continue; } + const b = build(o); + if (b === null) { + if (x.mode !== 'none' || x.report !== null) fail(`${name}: not reported, expected says ${x.mode}`); + continue; + } + if (x.mode !== b.mode) { fail(`${name}: mode ${b.mode}, expected ${x.mode}`); continue; } + const enc = encodeCmcd(b.c, { version: 2, reportingMode: b.mode }); + if (enc !== x.report) fail(`${name}: the library writes\n ${enc}\n expected\n ${x.report}`); + const dec = decodeCmcd(x.report); + if (!same(dec, b.c)) fail(`${name}: the expected report reads back as ${JSON.stringify(dec)}`); + const v = validateCmcd(dec, { reportingMode: b.mode }); + if (errors(v).length) fail(`${name}: ${JSON.stringify(errors(v))}`); + if (b.mode === 'request') { + const h = toCmcdHeaders(b.c, { version: 2, customHeaderMap: CUSTOM_HEADERS }); + if (JSON.stringify(Object.fromEntries(Object.entries(h).sort())) !== JSON.stringify(x.headers)) { + fail(`${name}: the library's header shards are ${JSON.stringify(h)}`); + } + if (!same(fromCmcdQuery('CMCD=' + encodeURIComponent(x.report)), dec)) fail(`${name}: the query form reads back otherwise`); + if (!same(fromCmcdHeaders(x.headers), dec)) fail(`${name}: the header shards read back otherwise`); + } else { + events.push(x.report); + } +} +const ev = validateCmcdEvents(events.join('\n')); +if (errors(ev).length || ev.valid === false) fail(`event reports: ${JSON.stringify(ev.issues)}`); + +// CMSD: the extension's §6 members, keys and parameters in bytewise order. +const cmsdExpected = read('expected/cmsd.json'); +for (const f of fs.readdirSync(path.join(dir, 'cmsd')).sort()) { + const name = f.replace(/\.json$/, ''); + const v = read(`cmsd/${f}`); + if (name.startsWith('static')) { + const m = {}; + const put = (k, x) => { if (x !== null) m[k] = x; }; + put('at', v.at); put('d', v.d); put('ht', v.ht); put('nor', v.nor); put('nrr', v.nrr); put('ot', v.ot); + put(P + 'ch', v.content_hash); put(P + 'l', v.layer === null ? null : String(v.layer)); + put(P + 'lt', v.layers_total === null ? null : String(v.layers_total)); + put(P + 'op', v.op === null ? null : new SfToken(v.op)); put(P + 'rcpt', v.receipt); + put('sf', v.sf); put('st', v.st); if (v.su) m.su = true; + const enc = encodeCmsdStatic(m); + if (enc !== cmsdExpected[name]) fail(`cmsd ${name}: the library writes ${enc}`); + if (!same(decodeCmsdStatic(cmsdExpected[name]), m)) fail(`cmsd ${name}: reads back otherwise`); + } else { + const items = v.map((e) => { + const p = {}; + if (e.du) p.du = true; + for (const k of ['etp', 'mb', 'rd', 'rtt']) if (e[k] !== null) p[k] = e[k]; + if (e.drop !== null) p[P + 'drop'] = String(e.drop); + return new SfItem(e.server, p); + }); + const enc = encodeCmsdDynamic(items); + if (enc !== cmsdExpected[name]) fail(`cmsd ${name}: the library writes ${enc}`); + const back = decodeCmsdDynamic(cmsdExpected[name]); + if (back.length !== items.length || back.some((b, i) => b.value !== items[i].value || !same(b.params, items[i].params))) { + fail(`cmsd ${name}: reads back otherwise: ${JSON.stringify(back)}`); + } + } +} + +if (fails.length) { + for (const f of fails) console.log('FAIL', f); + process.exit(1); +} +console.log(`telemetry: every report agrees with the independent library (${Object.keys(expected).length} observations, ${events.length} Event-mode reports, ${Object.keys(cmsdExpected).length} CMSD fields)`); diff --git a/wai/telemetry-conformance/cml/package-lock.json b/wai/telemetry-conformance/cml/package-lock.json new file mode 100644 index 000000000..bf33f221c --- /dev/null +++ b/wai/telemetry-conformance/cml/package-lock.json @@ -0,0 +1,78 @@ +{ + "name": "wai-telemetry-differential", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "wai-telemetry-differential", + "dependencies": { + "@svta/cml-cmcd": "2.8.1", + "@svta/cml-cmsd": "1.1.1", + "@svta/cml-structured-field-values": "1.1.6" + } + }, + "node_modules/@svta/cml-cmcd": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/@svta/cml-cmcd/-/cml-cmcd-2.8.1.tgz", + "integrity": "sha512-+QzjQNbZCAtyRHI/qAfKlY6w0VC/ofFastXqzD9+uONMTqACkPu2qfkkYT6OvcQmhXe0ULNNAvI26sVq4fAntw==", + "license": "Apache-2.0", + "engines": { + "node": ">=20" + }, + "peerDependencies": { + "@svta/cml-structured-field-values": "1.1.6", + "@svta/cml-utils": "1.6.1" + } + }, + "node_modules/@svta/cml-cmsd": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@svta/cml-cmsd/-/cml-cmsd-1.1.1.tgz", + "integrity": "sha512-Om3w7qU33E7i/K3Gd0GOG8njIDP9GD3R3XjqxIGc/ZXFexTVvMK3eQIN9cyVLdJrLR7XLMUIih7Kc1D+tNLl8Q==", + "license": "Apache-2.0", + "engines": { + "node": ">=20" + }, + "peerDependencies": { + "@svta/cml-cta": "1.0.9", + "@svta/cml-structured-field-values": "1.1.6", + "@svta/cml-utils": "1.6.1" + } + }, + "node_modules/@svta/cml-cta": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/@svta/cml-cta/-/cml-cta-1.0.9.tgz", + "integrity": "sha512-qEz9dYeHSDQtKiIfDQNWlBQgBsKJZcUojI8JLNhfc4d4IcAmqwwCd7UVXxEiAt7S6hNLQgWBE/YH43ot8x0Fbw==", + "license": "Apache-2.0", + "peer": true, + "engines": { + "node": ">=20" + }, + "peerDependencies": { + "@svta/cml-structured-field-values": "1.1.6", + "@svta/cml-utils": "1.6.1" + } + }, + "node_modules/@svta/cml-structured-field-values": { + "version": "1.1.6", + "resolved": "https://registry.npmjs.org/@svta/cml-structured-field-values/-/cml-structured-field-values-1.1.6.tgz", + "integrity": "sha512-+aPvf3Ak9DdJZC+z5/R1A8Swq0AI4uoVD6QV5e8gM9W8im+LuzP69rPh7lrnA7RfANTPhSwd6zxTf/IXyPlY8A==", + "license": "Apache-2.0", + "engines": { + "node": ">=20" + }, + "peerDependencies": { + "@svta/cml-utils": "1.6.1" + } + }, + "node_modules/@svta/cml-utils": { + "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@svta/cml-utils/-/cml-utils-1.6.1.tgz", + "integrity": "sha512-jPKb+it53Pi8rbO8fNX8qpbdE+JfDAeOXF7lcG+uVU5BCcR0wbMTT2VtFdMeVsUN57TFPAurRavE3aEUSzC+zw==", + "license": "Apache-2.0", + "peer": true, + "engines": { + "node": ">=20" + } + } + } +} diff --git a/wai/telemetry-conformance/cml/package.json b/wai/telemetry-conformance/cml/package.json new file mode 100644 index 000000000..0064fde0e --- /dev/null +++ b/wai/telemetry-conformance/cml/package.json @@ -0,0 +1,14 @@ +{ + "name": "wai-telemetry-differential", + "private": true, + "description": "Encodes the corpus's observations with an independent open-source CMCD and CMSD library and requires the same bytes (extensions/delivery-telemetry.md §9).", + "type": "module", + "scripts": { + "differential": "node differential.mjs .." + }, + "dependencies": { + "@svta/cml-cmcd": "2.8.1", + "@svta/cml-cmsd": "1.1.1", + "@svta/cml-structured-field-values": "1.1.6" + } +} diff --git a/wai/telemetry-conformance/cmsd/dynamic-two-nodes.json b/wai/telemetry-conformance/cmsd/dynamic-two-nodes.json new file mode 100644 index 000000000..171559579 --- /dev/null +++ b/wai/telemetry-conformance/cmsd/dynamic-two-nodes.json @@ -0,0 +1,20 @@ +[ + { + "server": "edge-7", + "du": false, + "etp": 25000, + "mb": 8000, + "rd": 3, + "rtt": 40, + "drop": 2 + }, + { + "server": "origin-1", + "du": true, + "etp": null, + "mb": null, + "rd": 12, + "rtt": null, + "drop": null + } +] diff --git a/wai/telemetry-conformance/cmsd/static-base.json b/wai/telemetry-conformance/cmsd/static-base.json new file mode 100644 index 000000000..0fde1ab93 --- /dev/null +++ b/wai/telemetry-conformance/cmsd/static-base.json @@ -0,0 +1,16 @@ +{ + "at": 1790000000000, + "d": 40, + "ht": null, + "nor": "L01.wai", + "nrr": null, + "ot": "v", + "sf": "o", + "st": "v", + "su": true, + "content_hash": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "layer": 0, + "layers_total": 4, + "receipt": "7bf0f413db62aa185969b9ccf925d40f0281601268899bb7f2e97344487f4189", + "op": null +} diff --git a/wai/telemetry-conformance/cmsd/static-derived.json b/wai/telemetry-conformance/cmsd/static-derived.json new file mode 100644 index 000000000..c83d86979 --- /dev/null +++ b/wai/telemetry-conformance/cmsd/static-derived.json @@ -0,0 +1,16 @@ +{ + "at": 1790000000005, + "d": null, + "ht": 20, + "nor": null, + "nrr": "0-1023", + "ot": "o", + "sf": "o", + "st": "l", + "su": false, + "content_hash": "4fc1bd94bd48b2c1ded99c0053eed742f5399cec09dbd2f8694cca6a30e44875", + "layer": 1, + "layers_total": 4, + "receipt": null, + "op": "crop" +} diff --git a/wai/telemetry-conformance/expected/cmcd.json b/wai/telemetry-conformance/expected/cmcd.json new file mode 100644 index 000000000..613ce57d2 --- /dev/null +++ b/wai/telemetry-conformance/expected/cmcd.json @@ -0,0 +1,123 @@ +{ + "01-request-base": { + "mode": "request", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",d=40,dl=200,ot=v,science.transaction.wai-b=\"460\",science.transaction.wai-cap=\"wai.neural.int_hyper\",science.transaction.wai-ch=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",science.transaction.wai-coh=\"wai\",science.transaction.wai-l=\"0\",science.transaction.wai-lt=\"4\",science.transaction.wai-p=\"p\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=0,st=v,v=2", + "headers": { + "CMCD-Object": "d=40,ot=v,science.transaction.wai-b=\"460\",science.transaction.wai-cap=\"wai.neural.int_hyper\",science.transaction.wai-ch=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",science.transaction.wai-l=\"0\",science.transaction.wai-lt=\"4\",science.transaction.wai-p=\"p\"", + "CMCD-Request": "dl=200,sn=0", + "CMCD-Session": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",science.transaction.wai-coh=\"wai\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",st=v,v=2" + } + }, + "02-request-layer": { + "mode": "request", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",dl=400,ot=v,science.transaction.wai-b=\"951\",science.transaction.wai-cap=\"wai.image.int_refine\",science.transaction.wai-ch=\"4fc1bd94bd48b2c1ded99c0053eed742f5399cec09dbd2f8694cca6a30e44875\",science.transaction.wai-l=\"1\",science.transaction.wai-lc=\"2\",science.transaction.wai-lt=\"4\",science.transaction.wai-p=\"p\",science.transaction.wai-rcpt=\"7bf0f413db62aa185969b9ccf925d40f0281601268899bb7f2e97344487f4189\",science.transaction.wai-tx=\"951\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=1,st=v,v=2", + "headers": { + "CMCD-Object": "ot=v,science.transaction.wai-b=\"951\",science.transaction.wai-cap=\"wai.image.int_refine\",science.transaction.wai-ch=\"4fc1bd94bd48b2c1ded99c0053eed742f5399cec09dbd2f8694cca6a30e44875\",science.transaction.wai-l=\"1\",science.transaction.wai-lt=\"4\",science.transaction.wai-p=\"p\",science.transaction.wai-rcpt=\"7bf0f413db62aa185969b9ccf925d40f0281601268899bb7f2e97344487f4189\",science.transaction.wai-tx=\"951\"", + "CMCD-Request": "dl=400,science.transaction.wai-lc=\"2\",sn=1", + "CMCD-Session": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",st=v,v=2" + } + }, + "03-request-rendition": { + "mode": "request", + "report": "ot=o,science.transaction.wai-b=\"2519\",science.transaction.wai-p=\"f\",science.transaction.wai-ri=\"1\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=2,st=v,v=2", + "headers": { + "CMCD-Object": "ot=o,science.transaction.wai-b=\"2519\",science.transaction.wai-p=\"f\",science.transaction.wai-ri=\"1\"", + "CMCD-Request": "sn=2", + "CMCD-Session": "sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",st=v,v=2" + } + }, + "04-response": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=rr,rc=200,science.transaction.wai-b=\"460\",science.transaction.wai-ch=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",science.transaction.wai-p=\"p\",science.transaction.wai-tx=\"460\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=3,st=v,ts=1790000000120,ttfb=38,ttlb=41,url=\"https://media.example/t/base.wai\",v=2", + "headers": null + }, + "05-start": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=4,st=v,sta=s,ts=1790000000000,v=2", + "headers": null + }, + "06-present-first": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,msd=160,science.transaction.wai-dec=\"2900\",science.transaction.wai-pl=\"1\",science.transaction.wai-u=\"0\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=5,st=v,sta=p,ts=1790000000160,v=2", + "headers": null + }, + "07-present-later": { + "mode": "none", + "report": null, + "headers": null + }, + "08-stall-start": { + "mode": "event", + "report": "bsa=(1),bsda=(0),cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=7,st=v,sta=r,ts=1790000004000,v=2", + "headers": null + }, + "09-stall-end": { + "mode": "event", + "report": "bsa=(1),bsda=(350),cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=8,st=v,sta=p,ts=1790000004350,v=2", + "headers": null + }, + "10-switch": { + "mode": "event", + "report": "cen=\"wai-switch\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ce,science.transaction.wai-p=\"f\",science.transaction.wai-sw=\"prior-not-held\",science.transaction.wai-u=\"125\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=9,st=v,ts=1790000005000,v=2", + "headers": null + }, + "11-layer-change": { + "mode": "event", + "report": "cen=\"wai-layer-change\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ce,science.transaction.wai-lc=\"1\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=10,st=v,ts=1790000006000,v=2", + "headers": null + }, + "12-frame-drop": { + "mode": "event", + "report": "cen=\"wai-frame-drop\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",dfa=3,e=ce,sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=11,st=v,ts=1790000006500,v=2", + "headers": null + }, + "13-capable-pin-refused": { + "mode": "event", + "report": "cen=\"wai-capable\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ce,science.transaction.wai-cf=\"supported\",science.transaction.wai-co=\"pin-refused\",science.transaction.wai-cr=\"part-mismatch\",sf=o,sid=\"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4\",sn=0,st=v,ts=1790000000010,v=2", + "headers": null + }, + "14-capable": { + "mode": "event", + "report": "cen=\"wai-capable\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ce,science.transaction.wai-co=\"capable\",science.transaction.wai-coh=\"control\",sf=o,sid=\"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4\",sn=1,st=v,ts=1790000000011,v=2", + "headers": null + }, + "15-abandon": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,sf=o,sid=\"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4\",sn=2,st=v,sta=q,ts=1790000000900,v=2", + "headers": null + }, + "16-end": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,science.transaction.wai-clm=\"e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=12,st=v,sta=e,ts=1790000009000,v=2", + "headers": null + }, + "17-live-low-latency": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ps,sf=h,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=13,st=ll,sta=s,ts=1790000009100,v=2", + "headers": null + }, + "18-switch-prior-revoked": { + "mode": "event", + "report": "cen=\"wai-switch\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ce,science.transaction.wai-p=\"f\",science.transaction.wai-sw=\"prior-revoked\",science.transaction.wai-u=\"126\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=14,st=v,ts=1790000009200,v=2", + "headers": null + }, + "19-capable-not-authorised": { + "mode": "event", + "report": "cen=\"wai-capable\",cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=ce,science.transaction.wai-co=\"pin-refused\",science.transaction.wai-cr=\"prior-not-authorised\",sf=o,sid=\"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4\",sn=3,st=v,ts=1790000000012,v=2", + "headers": null + }, + "20-request-escaped": { + "mode": "request", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",ot=v,science.transaction.wai-b=\"12\",science.transaction.wai-cap=\"x.test.\\\"quoted\\\"\\\\cap\",science.transaction.wai-p=\"p\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=15,st=v,v=2", + "headers": { + "CMCD-Object": "ot=v,science.transaction.wai-b=\"12\",science.transaction.wai-cap=\"x.test.\\\"quoted\\\"\\\\cap\",science.transaction.wai-p=\"p\"", + "CMCD-Request": "sn=15", + "CMCD-Session": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",st=v,v=2" + } + }, + "21-response-escaped": { + "mode": "event", + "report": "cid=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",e=rr,rc=200,science.transaction.wai-b=\"12\",science.transaction.wai-p=\"p\",sf=o,sid=\"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968\",sn=16,st=v,ts=1790000009300,url=\"https://media.example/t/a\\\"b.wai\",v=2", + "headers": null + } +} diff --git a/wai/telemetry-conformance/expected/cmsd.json b/wai/telemetry-conformance/expected/cmsd.json new file mode 100644 index 000000000..d06ed03d0 --- /dev/null +++ b/wai/telemetry-conformance/expected/cmsd.json @@ -0,0 +1,5 @@ +{ + "static-base": "at=1790000000000,d=40,nor=\"L01.wai\",ot=v,science.transaction.wai-ch=\"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f\",science.transaction.wai-l=\"0\",science.transaction.wai-lt=\"4\",science.transaction.wai-rcpt=\"7bf0f413db62aa185969b9ccf925d40f0281601268899bb7f2e97344487f4189\",sf=o,st=v,su", + "static-derived": "at=1790000000005,ht=20,nrr=\"0-1023\",ot=o,science.transaction.wai-ch=\"4fc1bd94bd48b2c1ded99c0053eed742f5399cec09dbd2f8694cca6a30e44875\",science.transaction.wai-l=\"1\",science.transaction.wai-lt=\"4\",science.transaction.wai-op=crop,sf=o,st=l", + "dynamic-two-nodes": "\"edge-7\";etp=25000;mb=8000;rd=3;rtt=40;science.transaction.wai-drop=\"2\",\"origin-1\";du;rd=12" +} diff --git a/wai/telemetry-conformance/expected/qoe.json b/wai/telemetry-conformance/expected/qoe.json new file mode 100644 index 000000000..37f946b93 --- /dev/null +++ b/wai/telemetry-conformance/expected/qoe.json @@ -0,0 +1,38 @@ +{ + "session-stalls": { + "started": true, + "vst_ms": 160, + "exit_before_start": false, + "start_failed": false, + "stalls": 2, + "stall_ms": 1350, + "engaged_ms": 8840, + "units": 2, + "counted": 1411, + "claim": "e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277" + }, + "session-exit-before-start": { + "started": true, + "vst_ms": null, + "exit_before_start": true, + "start_failed": false, + "stalls": 0, + "stall_ms": 0, + "engaged_ms": 0, + "units": 0, + "counted": 2519, + "claim": null + }, + "session-start-failure": { + "started": true, + "vst_ms": null, + "exit_before_start": false, + "start_failed": true, + "stalls": 0, + "stall_ms": 0, + "engaged_ms": 0, + "units": 0, + "counted": 0, + "claim": null + } +} diff --git a/wai/telemetry-conformance/expected/refused.json b/wai/telemetry-conformance/expected/refused.json new file mode 100644 index 000000000..83872b9cd --- /dev/null +++ b/wai/telemetry-conformance/expected/refused.json @@ -0,0 +1,30 @@ +[ + "unknown-member", + "missing-member", + "version-2", + "sid-not-hex", + "event-mode-without-time", + "sn-past-2-53", + "ts-past-sf-integer", + "claim-before-end", + "unknown-event", + "response-key-on-request", + "request-key-on-response", + "switch-to-same-path", + "switch-unknown-reason", + "pin-refused-without-reason", + "object-type-unknown", + "cid-past-64", + "cid-an-address", + "cid-a-url", + "url-with-query", + "url-with-fragment", + "url-with-userinfo", + "url-userinfo-behind-backslashes", + "url-userinfo-without-slashes", + "url-userinfo-behind-a-mixed-separator", + "url-with-a-backslash", + "url-with-an-at-in-its-path", + "present-zero-layers", + "cohort-unknown" +] diff --git a/wai/telemetry-conformance/invalid/cid-a-url.json b/wai/telemetry-conformance/invalid/cid-a-url.json new file mode 100644 index 000000000..b2c361c1d --- /dev/null +++ b/wai/telemetry-conformance/invalid/cid-a-url.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "https://media.example/t", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/cid-an-address.json b/wai/telemetry-conformance/invalid/cid-an-address.json new file mode 100644 index 000000000..c90ee158d --- /dev/null +++ b/wai/telemetry-conformance/invalid/cid-an-address.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "viewer@example.org", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/cid-past-64.json b/wai/telemetry-conformance/invalid/cid-past-64.json new file mode 100644 index 000000000..e254ee7f7 --- /dev/null +++ b/wai/telemetry-conformance/invalid/cid-past-64.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/claim-before-end.json b/wai/telemetry-conformance/invalid/claim-before-end.json new file mode 100644 index 000000000..946b43975 --- /dev/null +++ b/wai/telemetry-conformance/invalid/claim-before-end.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": "e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277" +} diff --git a/wai/telemetry-conformance/invalid/cohort-unknown.json b/wai/telemetry-conformance/invalid/cohort-unknown.json new file mode 100644 index 000000000..4b898740c --- /dev/null +++ b/wai/telemetry-conformance/invalid/cohort-unknown.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": "treatment", + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/event-mode-without-time.json b/wai/telemetry-conformance/invalid/event-mode-without-time.json new file mode 100644 index 000000000..c2c557568 --- /dev/null +++ b/wai/telemetry-conformance/invalid/event-mode-without-time.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": null, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/missing-member.json b/wai/telemetry-conformance/invalid/missing-member.json new file mode 100644 index 000000000..458e0a6c9 --- /dev/null +++ b/wai/telemetry-conformance/invalid/missing-member.json @@ -0,0 +1,17 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "claim": null, + "metrics": null +} diff --git a/wai/telemetry-conformance/invalid/object-type-unknown.json b/wai/telemetry-conformance/invalid/object-type-unknown.json new file mode 100644 index 000000000..4d23333ca --- /dev/null +++ b/wai/telemetry-conformance/invalid/object-type-unknown.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": null, + "event": "request", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": "x", + "url": null, + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": null, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/pin-refused-without-reason.json b/wai/telemetry-conformance/invalid/pin-refused-without-reason.json new file mode 100644 index 000000000..eb864d9e0 --- /dev/null +++ b/wai/telemetry-conformance/invalid/pin-refused-without-reason.json @@ -0,0 +1,22 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "capable", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": { + "outcome": "pin-refused", + "reason": null, + "classical_floor": null + }, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/present-zero-layers.json b/wai/telemetry-conformance/invalid/present-zero-layers.json new file mode 100644 index 000000000..7cdbe1a79 --- /dev/null +++ b/wai/telemetry-conformance/invalid/present-zero-layers.json @@ -0,0 +1,24 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "present", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": { + "unit": 0, + "layers": 0, + "first": true, + "late_ms": null, + "decode_us": null + }, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/request-key-on-response.json b/wai/telemetry-conformance/invalid/request-key-on-response.json new file mode 100644 index 000000000..d29bfba8f --- /dev/null +++ b/wai/telemetry-conformance/invalid/request-key-on-response.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": null, + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": 200, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": null, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/response-key-on-request.json b/wai/telemetry-conformance/invalid/response-key-on-request.json new file mode 100644 index 000000000..cce98e8b4 --- /dev/null +++ b/wai/telemetry-conformance/invalid/response-key-on-request.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": null, + "event": "request", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": null, + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/sid-not-hex.json b/wai/telemetry-conformance/invalid/sid-not-hex.json new file mode 100644 index 000000000..042a0f4ff --- /dev/null +++ b/wai/telemetry-conformance/invalid/sid-not-hex.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "not-a-session-id-not-a-session-i", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/sn-past-2-53.json b/wai/telemetry-conformance/invalid/sn-past-2-53.json new file mode 100644 index 000000000..b58a15bed --- /dev/null +++ b/wai/telemetry-conformance/invalid/sn-past-2-53.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 9007199254740992, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/switch-to-same-path.json b/wai/telemetry-conformance/invalid/switch-to-same-path.json new file mode 100644 index 000000000..f4c132db7 --- /dev/null +++ b/wai/telemetry-conformance/invalid/switch-to-same-path.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "switch", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": { + "unit": 1, + "from": "primary", + "to": "primary", + "reason": "trap" + }, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/switch-unknown-reason.json b/wai/telemetry-conformance/invalid/switch-unknown-reason.json new file mode 100644 index 000000000..515a24fd5 --- /dev/null +++ b/wai/telemetry-conformance/invalid/switch-unknown-reason.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "switch", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": { + "unit": 1, + "from": "primary", + "to": "fallback", + "reason": "slow" + }, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/ts-past-sf-integer.json b/wai/telemetry-conformance/invalid/ts-past-sf-integer.json new file mode 100644 index 000000000..2d79421e7 --- /dev/null +++ b/wai/telemetry-conformance/invalid/ts-past-sf-integer.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1000000000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/unknown-event.json b/wai/telemetry-conformance/invalid/unknown-event.json new file mode 100644 index 000000000..1b095dc1b --- /dev/null +++ b/wai/telemetry-conformance/invalid/unknown-event.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "pause", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/unknown-member.json b/wai/telemetry-conformance/invalid/unknown-member.json new file mode 100644 index 000000000..b44c8b3ee --- /dev/null +++ b/wai/telemetry-conformance/invalid/unknown-member.json @@ -0,0 +1,19 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null, + "extra": 1 +} diff --git a/wai/telemetry-conformance/invalid/url-userinfo-behind-a-mixed-separator.json b/wai/telemetry-conformance/invalid/url-userinfo-behind-a-mixed-separator.json new file mode 100644 index 000000000..ee3da73a8 --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-userinfo-behind-a-mixed-separator.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https:/\\viewer:secret@media.example/t/base.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-userinfo-behind-backslashes.json b/wai/telemetry-conformance/invalid/url-userinfo-behind-backslashes.json new file mode 100644 index 000000000..43de79e4d --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-userinfo-behind-backslashes.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https:\\\\viewer:secret@media.example/t/base.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-userinfo-without-slashes.json b/wai/telemetry-conformance/invalid/url-userinfo-without-slashes.json new file mode 100644 index 000000000..0862634f7 --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-userinfo-without-slashes.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https:viewer:secret@media.example/t/base.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-with-a-backslash.json b/wai/telemetry-conformance/invalid/url-with-a-backslash.json new file mode 100644 index 000000000..4e2634cce --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-with-a-backslash.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://media.example\\t/base.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-with-an-at-in-its-path.json b/wai/telemetry-conformance/invalid/url-with-an-at-in-its-path.json new file mode 100644 index 000000000..c5b25bab0 --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-with-an-at-in-its-path.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://media.example/u/viewer@example.org/base.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-with-fragment.json b/wai/telemetry-conformance/invalid/url-with-fragment.json new file mode 100644 index 000000000..d24ee1cbc --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-with-fragment.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://media.example/t/base.wai#t=1", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-with-query.json b/wai/telemetry-conformance/invalid/url-with-query.json new file mode 100644 index 000000000..fff0ef5a2 --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-with-query.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://media.example/t/base.wai?token=abc", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/url-with-userinfo.json b/wai/telemetry-conformance/invalid/url-with-userinfo.json new file mode 100644 index 000000000..1c62557a9 --- /dev/null +++ b/wai/telemetry-conformance/invalid/url-with-userinfo.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://viewer:secret@media.example/t/base.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": null, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": null, + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/invalid/version-2.json b/wai/telemetry-conformance/invalid/version-2.json new file mode 100644 index 000000000..aea23de39 --- /dev/null +++ b/wai/telemetry-conformance/invalid/version-2.json @@ -0,0 +1,18 @@ +{ + "v": 2, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/logs/session-exit-before-start.jsonl b/wai/telemetry-conformance/logs/session-exit-before-start.jsonl new file mode 100644 index 000000000..68657f133 --- /dev/null +++ b/wai/telemetry-conformance/logs/session-exit-before-start.jsonl @@ -0,0 +1,3 @@ +{"v":1,"sid":"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4","sn":0,"ts_ms":1790000000000,"event":"start","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4","sn":1,"ts_ms":1790000000300,"event":"response","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":{"ot":null,"url":null,"content_hash":null,"capability":null,"rendition":null,"layer":null,"layers_total":null,"layer_cap":null,"counted":2519,"transferred":null,"d_ms":null,"dl_ms":null,"ttfb_ms":null,"ttlb_ms":null,"rc":200,"path":null,"receipt":null},"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4","sn":2,"ts_ms":1790000000900,"event":"abandon","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} diff --git a/wai/telemetry-conformance/logs/session-stalls.jsonl b/wai/telemetry-conformance/logs/session-stalls.jsonl new file mode 100644 index 000000000..cb2062565 --- /dev/null +++ b/wai/telemetry-conformance/logs/session-stalls.jsonl @@ -0,0 +1,10 @@ +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":0,"ts_ms":1790000000000,"event":"start","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":1,"ts_ms":1790000000100,"event":"response","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":{"ot":null,"url":null,"content_hash":null,"capability":null,"rendition":null,"layer":null,"layers_total":null,"layer_cap":null,"counted":460,"transferred":null,"d_ms":null,"dl_ms":null,"ttfb_ms":null,"ttlb_ms":null,"rc":200,"path":null,"receipt":null},"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":2,"ts_ms":1790000000160,"event":"present","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":{"unit":0,"layers":1,"first":true,"late_ms":null,"decode_us":null},"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":3,"ts_ms":1790000000180,"event":"response","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":{"ot":null,"url":null,"content_hash":null,"capability":null,"rendition":null,"layer":null,"layers_total":null,"layer_cap":null,"counted":951,"transferred":null,"d_ms":null,"dl_ms":null,"ttfb_ms":null,"ttlb_ms":null,"rc":200,"path":null,"receipt":null},"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":4,"ts_ms":1790000000210,"event":"present","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":{"unit":0,"layers":2,"first":false,"late_ms":null,"decode_us":null},"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":5,"ts_ms":1790000004000,"event":"stall-start","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":6,"ts_ms":1790000004350,"event":"stall-end","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":7,"ts_ms":1790000004400,"event":"present","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":{"unit":1,"layers":1,"first":false,"late_ms":30,"decode_us":null},"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":8,"ts_ms":1790000008000,"event":"stall-start","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968","sn":9,"ts_ms":1790000009000,"event":"end","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":"e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277"} diff --git a/wai/telemetry-conformance/logs/session-start-failure.jsonl b/wai/telemetry-conformance/logs/session-start-failure.jsonl new file mode 100644 index 000000000..45b61f5cc --- /dev/null +++ b/wai/telemetry-conformance/logs/session-start-failure.jsonl @@ -0,0 +1,3 @@ +{"v":1,"sid":"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4","sn":0,"ts_ms":1790000000000,"event":"start","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4","sn":1,"ts_ms":1790000000050,"event":"capable","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":{"outcome":"missing-capability","reason":null,"classical_floor":"unsupported"},"cohort":null,"metrics":null,"claim":null} +{"v":1,"sid":"0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4","sn":2,"ts_ms":1790000003000,"event":"end","cid":"aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f","st":"v","sf":"o","object":null,"present":null,"switch":null,"layer_change":null,"capable":null,"cohort":null,"metrics":null,"claim":null} diff --git a/wai/telemetry-conformance/observations/01-request-base.json b/wai/telemetry-conformance/observations/01-request-base.json new file mode 100644 index 000000000..ac9e3e059 --- /dev/null +++ b/wai/telemetry-conformance/observations/01-request-base.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 0, + "ts_ms": null, + "event": "request", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": "v", + "url": null, + "content_hash": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "capability": "wai.neural.int_hyper", + "rendition": null, + "layer": 0, + "layers_total": 4, + "layer_cap": null, + "counted": 460, + "transferred": null, + "d_ms": 40, + "dl_ms": 200, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": null, + "path": "primary", + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": "wai", + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/02-request-layer.json b/wai/telemetry-conformance/observations/02-request-layer.json new file mode 100644 index 000000000..64295a700 --- /dev/null +++ b/wai/telemetry-conformance/observations/02-request-layer.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 1, + "ts_ms": null, + "event": "request", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": "v", + "url": null, + "content_hash": "4fc1bd94bd48b2c1ded99c0053eed742f5399cec09dbd2f8694cca6a30e44875", + "capability": "wai.image.int_refine", + "rendition": null, + "layer": 1, + "layers_total": 4, + "layer_cap": 2, + "counted": 951, + "transferred": 951, + "d_ms": null, + "dl_ms": 400, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": null, + "path": "primary", + "receipt": "7bf0f413db62aa185969b9ccf925d40f0281601268899bb7f2e97344487f4189" + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/03-request-rendition.json b/wai/telemetry-conformance/observations/03-request-rendition.json new file mode 100644 index 000000000..6a8e8d99c --- /dev/null +++ b/wai/telemetry-conformance/observations/03-request-rendition.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 2, + "ts_ms": null, + "event": "request", + "cid": null, + "st": "v", + "sf": "o", + "object": { + "ot": "o", + "url": null, + "content_hash": null, + "capability": null, + "rendition": 1, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": 2519, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": null, + "path": "fallback", + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/04-response.json b/wai/telemetry-conformance/observations/04-response.json new file mode 100644 index 000000000..f6e3e4ec2 --- /dev/null +++ b/wai/telemetry-conformance/observations/04-response.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 3, + "ts_ms": 1790000000120, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://media.example/t/base.wai", + "content_hash": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": 460, + "transferred": 460, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": 38, + "ttlb_ms": 41, + "rc": 200, + "path": "primary", + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/05-start.json b/wai/telemetry-conformance/observations/05-start.json new file mode 100644 index 000000000..93abd27c6 --- /dev/null +++ b/wai/telemetry-conformance/observations/05-start.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 4, + "ts_ms": 1790000000000, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/06-present-first.json b/wai/telemetry-conformance/observations/06-present-first.json new file mode 100644 index 000000000..1cf21062f --- /dev/null +++ b/wai/telemetry-conformance/observations/06-present-first.json @@ -0,0 +1,29 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 5, + "ts_ms": 1790000000160, + "event": "present", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": { + "unit": 0, + "layers": 1, + "first": true, + "late_ms": null, + "decode_us": 2900 + }, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": { + "msd_ms": 160, + "bsa": null, + "bsda_ms": null, + "dfa": null + }, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/07-present-later.json b/wai/telemetry-conformance/observations/07-present-later.json new file mode 100644 index 000000000..0db6e9cab --- /dev/null +++ b/wai/telemetry-conformance/observations/07-present-later.json @@ -0,0 +1,24 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 6, + "ts_ms": 1790000000210, + "event": "present", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": { + "unit": 0, + "layers": 2, + "first": false, + "late_ms": 10, + "decode_us": 1400 + }, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/08-stall-start.json b/wai/telemetry-conformance/observations/08-stall-start.json new file mode 100644 index 000000000..28dc27971 --- /dev/null +++ b/wai/telemetry-conformance/observations/08-stall-start.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 7, + "ts_ms": 1790000004000, + "event": "stall-start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": { + "msd_ms": null, + "bsa": 1, + "bsda_ms": 0, + "dfa": null + }, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/09-stall-end.json b/wai/telemetry-conformance/observations/09-stall-end.json new file mode 100644 index 000000000..68da09c6f --- /dev/null +++ b/wai/telemetry-conformance/observations/09-stall-end.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 8, + "ts_ms": 1790000004350, + "event": "stall-end", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": { + "msd_ms": null, + "bsa": 1, + "bsda_ms": 350, + "dfa": null + }, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/10-switch.json b/wai/telemetry-conformance/observations/10-switch.json new file mode 100644 index 000000000..bd2fdbac0 --- /dev/null +++ b/wai/telemetry-conformance/observations/10-switch.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 9, + "ts_ms": 1790000005000, + "event": "switch", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": { + "unit": 125, + "from": "primary", + "to": "fallback", + "reason": "prior-not-held" + }, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/11-layer-change.json b/wai/telemetry-conformance/observations/11-layer-change.json new file mode 100644 index 000000000..c5463f031 --- /dev/null +++ b/wai/telemetry-conformance/observations/11-layer-change.json @@ -0,0 +1,21 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 10, + "ts_ms": 1790000006000, + "event": "layer-change", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": { + "from": null, + "to": 1 + }, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/12-frame-drop.json b/wai/telemetry-conformance/observations/12-frame-drop.json new file mode 100644 index 000000000..0c6948989 --- /dev/null +++ b/wai/telemetry-conformance/observations/12-frame-drop.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 11, + "ts_ms": 1790000006500, + "event": "frame-drop", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": { + "msd_ms": null, + "bsa": null, + "bsda_ms": null, + "dfa": 3 + }, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/13-capable-pin-refused.json b/wai/telemetry-conformance/observations/13-capable-pin-refused.json new file mode 100644 index 000000000..8e52eebea --- /dev/null +++ b/wai/telemetry-conformance/observations/13-capable-pin-refused.json @@ -0,0 +1,22 @@ +{ + "v": 1, + "sid": "0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4", + "sn": 0, + "ts_ms": 1790000000010, + "event": "capable", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": { + "outcome": "pin-refused", + "reason": "part-mismatch", + "classical_floor": "supported" + }, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/14-capable.json b/wai/telemetry-conformance/observations/14-capable.json new file mode 100644 index 000000000..edbba0877 --- /dev/null +++ b/wai/telemetry-conformance/observations/14-capable.json @@ -0,0 +1,22 @@ +{ + "v": 1, + "sid": "0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4", + "sn": 1, + "ts_ms": 1790000000011, + "event": "capable", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": { + "outcome": "capable", + "reason": null, + "classical_floor": null + }, + "cohort": "control", + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/15-abandon.json b/wai/telemetry-conformance/observations/15-abandon.json new file mode 100644 index 000000000..23ee857c5 --- /dev/null +++ b/wai/telemetry-conformance/observations/15-abandon.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4", + "sn": 2, + "ts_ms": 1790000000900, + "event": "abandon", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/16-end.json b/wai/telemetry-conformance/observations/16-end.json new file mode 100644 index 000000000..9c15e1534 --- /dev/null +++ b/wai/telemetry-conformance/observations/16-end.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 12, + "ts_ms": 1790000009000, + "event": "end", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": "e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277" +} diff --git a/wai/telemetry-conformance/observations/17-live-low-latency.json b/wai/telemetry-conformance/observations/17-live-low-latency.json new file mode 100644 index 000000000..cfc5ff5bb --- /dev/null +++ b/wai/telemetry-conformance/observations/17-live-low-latency.json @@ -0,0 +1,18 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 13, + "ts_ms": 1790000009100, + "event": "start", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "ll", + "sf": "h", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/18-switch-prior-revoked.json b/wai/telemetry-conformance/observations/18-switch-prior-revoked.json new file mode 100644 index 000000000..6e8c8d916 --- /dev/null +++ b/wai/telemetry-conformance/observations/18-switch-prior-revoked.json @@ -0,0 +1,23 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 14, + "ts_ms": 1790000009200, + "event": "switch", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": { + "unit": 126, + "from": "primary", + "to": "fallback", + "reason": "prior-revoked" + }, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/19-capable-not-authorised.json b/wai/telemetry-conformance/observations/19-capable-not-authorised.json new file mode 100644 index 000000000..dde24e21f --- /dev/null +++ b/wai/telemetry-conformance/observations/19-capable-not-authorised.json @@ -0,0 +1,22 @@ +{ + "v": 1, + "sid": "0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4", + "sn": 3, + "ts_ms": 1790000000012, + "event": "capable", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": null, + "present": null, + "switch": null, + "layer_change": null, + "capable": { + "outcome": "pin-refused", + "reason": "prior-not-authorised", + "classical_floor": null + }, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/20-request-escaped.json b/wai/telemetry-conformance/observations/20-request-escaped.json new file mode 100644 index 000000000..2579dab19 --- /dev/null +++ b/wai/telemetry-conformance/observations/20-request-escaped.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 15, + "ts_ms": null, + "event": "request", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": "v", + "url": null, + "content_hash": null, + "capability": "x.test.\"quoted\"\\cap", + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": 12, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": null, + "path": "primary", + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/telemetry-conformance/observations/21-response-escaped.json b/wai/telemetry-conformance/observations/21-response-escaped.json new file mode 100644 index 000000000..b287dc3c8 --- /dev/null +++ b/wai/telemetry-conformance/observations/21-response-escaped.json @@ -0,0 +1,36 @@ +{ + "v": 1, + "sid": "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968", + "sn": 16, + "ts_ms": 1790000009300, + "event": "response", + "cid": "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f", + "st": "v", + "sf": "o", + "object": { + "ot": null, + "url": "https://media.example/t/a\"b.wai", + "content_hash": null, + "capability": null, + "rendition": null, + "layer": null, + "layers_total": null, + "layer_cap": null, + "counted": 12, + "transferred": null, + "d_ms": null, + "dl_ms": null, + "ttfb_ms": null, + "ttlb_ms": null, + "rc": 200, + "path": "primary", + "receipt": null + }, + "present": null, + "switch": null, + "layer_change": null, + "capable": null, + "cohort": null, + "metrics": null, + "claim": null +} diff --git a/wai/wai-rs/Cargo.toml b/wai/wai-rs/Cargo.toml index e06b16118..a8e231313 100644 --- a/wai/wai-rs/Cargo.toml +++ b/wai/wai-rs/Cargo.toml @@ -184,6 +184,16 @@ jwp_hash = ["dep:blake3"] # (WIR1) is integer rANS (neural_int); content hashes are jwp_hash's BLAKE3; stage # hashes are SHA-256, always on. staged = ["neural_int", "jwp_hash"] +# Staged-delivery measurement (extensions/staged-measurement.md): the counted bytes of +# an envelope, the output tree, and the deterministic evaluation of a staged title — +# its prefixes' counted bytes, outputs and quality integers, first presentable, +# layering overhead and deduplication — over the staged sink and the integer decoders. +# Pure integer, no ONNX, no file system on wasm32. +stage_measure = ["staged", "video_int", "audio_int", "splat_int"] +# Delivery telemetry (extensions/delivery-telemetry.md): unsigned, advisory observations, +# their CMCD version 2 and CMSD reports as RFC 9651 Structured Field Values, and session +# metrics. No dependency beyond serde_json; wasm32-clean. +telemetry = [] # Typed receipt classes on the JWP chain (extensions/jwp-receipts.md §2.4.3–§2.4.7, # §2.6, §3.1–§3.2): `claim` receipts of class energy / lawful-basis / log-inclusion, # the soft-binding class, and recovery under the ancestor rule. Needs `provenance` @@ -770,3 +780,31 @@ codegen-units = 1 name = "wai_score_vectors" path = "src/bin/wai_score_vectors.rs" required-features = ["score"] + +# Staged-delivery measurement (extensions/staged-measurement.md): `eval` one title, +# `verify` the corpus staged-measure-conformance/ (fixtures regenerated in memory, +# evaluations, the fixture tool's own reconstruction, goldens), `generate-goldens` +# and `gen-fixtures` (never in CI). +[[bin]] +name = "wai_stage_measure" +path = "src/bin/wai_stage_measure.rs" +required-features = ["stage_measure"] + +[[test]] +name = "stage_measure" +path = "tests/stage_measure.rs" +required-features = ["stage_measure"] + +# Its own binary: a counting allocator shows a title over an outsized base is refused +# before anything is allocated for it. +[[test]] +name = "stage_measure_bounds" +path = "tests/stage_measure_bounds.rs" +required-features = ["stage_measure"] + +# Delivery telemetry (extensions/delivery-telemetry.md): `encode` an observation's CMCD +# report, `decode` one, `verify` the corpus telemetry-conformance/, `gen` (never in CI). +[[bin]] +name = "wai_telemetry" +path = "src/bin/wai_telemetry.rs" +required-features = ["telemetry"] diff --git a/wai/wai-rs/src/bench_env.rs b/wai/wai-rs/src/bench_env.rs new file mode 100644 index 000000000..000d18403 --- /dev/null +++ b/wai/wai-rs/src/bench_env.rs @@ -0,0 +1,114 @@ +//! The machine a measurement ran on (`extensions/staged-measurement.md` §7.4, §11.1): +//! its processor, cores, operating system and architecture, the build's profile, its +//! load, and its power source, as the operating system reports them. Native only: a +//! browser reports none of these. +//! +//! Every reading is what the OS says, unparsed beyond trimming; a reading the OS does +//! not give is `None` (or "unknown"), never estimated. + +/// The machine and build a measurement ran on. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct MachineRecord { + /// The processor's name as the OS reports it, or `unknown CPU`. + pub cpu: String, + /// `N logical cores`, followed, where the OS reports performance levels, by each + /// level's core count and name in parentheses. + pub cores: String, + pub os: &'static str, + pub arch: &'static str, + /// `release`, or `DEBUG (numbers not meaningful)` for a build with debug assertions. + pub build: &'static str, +} + +impl std::fmt::Display for MachineRecord { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}, {}; {}-{}; {} build", self.cpu, self.cores, self.os, self.arch, self.build) + } +} + +fn sh(cmd: &str, args: &[&str]) -> Option { + let o = std::process::Command::new(cmd).args(args).output().ok()?; + o.status.success().then(|| String::from_utf8_lossy(&o.stdout).trim().to_string()) +} + +/// The core types as the OS reports them, e.g. `6 Super + 12 Performance`: each macOS performance +/// level's logical-core count and its own name (`hw.perflevelN.name`), in the OS's +/// order. Names are reported as given, not inferred from the level's index, since what +/// level 0 and 1 are differs between chips. `None` where the OS does not report levels. +fn core_levels() -> Option { + let n: usize = sh("sysctl", &["-n", "hw.nperflevels"])?.parse().ok()?; + let parts: Vec = (0..n) + .map(|i| { + let cpus = sh("sysctl", &["-n", &format!("hw.perflevel{i}.logicalcpu")])?; + let name = sh("sysctl", &["-n", &format!("hw.perflevel{i}.name")]).unwrap_or_else(|| format!("level-{i}")); + Some(format!("{cpus} {name}")) + }) + .collect::>()?; + (!parts.is_empty()).then(|| parts.join(" + ")) +} + +/// This machine and build. +pub fn machine() -> MachineRecord { + let cpu = sh("sysctl", &["-n", "machdep.cpu.brand_string"]) + .or_else(|| { + std::fs::read_to_string("/proc/cpuinfo").ok()?.lines().find(|l| l.starts_with("model name")).map(|l| l.split(':').nth(1).unwrap_or("").trim().to_string()) + }) + .unwrap_or_else(|| "unknown CPU".into()); + let levels = core_levels().map(|l| format!(" ({l})")).unwrap_or_default(); + MachineRecord { + cpu, + cores: format!("{} logical cores{levels}", std::thread::available_parallelism().map(|n| n.get()).unwrap_or(0)), + os: std::env::consts::OS, + arch: std::env::consts::ARCH, + build: if cfg!(debug_assertions) { "DEBUG (numbers not meaningful)" } else { "release" }, + } +} + +/// The 1-, 5- and 15-minute load averages, as the OS writes them. +pub fn loadavg() -> Option<(String, String, String)> { + let line = sh("sysctl", &["-n", "vm.loadavg"]) + .map(|s| s.trim_matches(|c| c == '{' || c == '}' || c == ' ').to_string()) + .or_else(|| std::fs::read_to_string("/proc/loadavg").ok())?; + let mut it = line.split_whitespace().map(str::to_string); + Some((it.next()?, it.next()?, it.next()?)) +} + +/// Where the machine draws power from, as the OS names it: macOS's `pmset` line +/// (`AC Power`, `Battery Power`), or on Linux `AC Power` when a mains supply is +/// online and `Battery Power` when only a battery reports. `None` when the OS says +/// nothing. +pub fn power_source() -> Option { + if let Some(out) = sh("pmset", &["-g", "batt"]) { + let first = out.lines().next()?; + let (_, rest) = first.split_once('\'')?; + let (name, _) = rest.split_once('\'')?; + return Some(name.to_string()); + } + let dir = std::fs::read_dir("/sys/class/power_supply").ok()?; + let mut battery = false; + for e in dir.flatten() { + let p = e.path(); + let kind = std::fs::read_to_string(p.join("type")).unwrap_or_default(); + match kind.trim() { + "Mains" if std::fs::read_to_string(p.join("online")).is_ok_and(|s| s.trim() == "1") => return Some("AC Power".into()), + "Battery" => battery = true, + _ => {} + } + } + battery.then(|| "Battery Power".into()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_record_prints_as_the_decode_bench_printed_it() { + let m = MachineRecord { cpu: "X".into(), cores: "8 logical cores (4 A + 4 B)".into(), os: "macos", arch: "aarch64", build: "release" }; + assert_eq!(m.to_string(), "X, 8 logical cores (4 A + 4 B); macos-aarch64; release build"); + // On this machine: the record reads, and it names this OS and architecture. + let here = machine(); + assert_eq!((here.os, here.arch), (std::env::consts::OS, std::env::consts::ARCH)); + assert!(here.cores.ends_with(')') || here.cores.ends_with("cores"), "{}", here.cores); + } +} diff --git a/wai/wai-rs/src/bin/wai_cli/stage.rs b/wai/wai-rs/src/bin/wai_cli/stage.rs index 78f41cf47..4d76f9340 100644 --- a/wai/wai-rs/src/bin/wai_cli/stage.rs +++ b/wai/wai-rs/src/bin/wai_cli/stage.rs @@ -6,7 +6,11 @@ //! A base of an integer capability is decoded through `staged::decode_base`: with no //! parameter set for `wai.video.int_motion`, otherwise against the set in the //! directory `--set` names (its files by the names SPEC §5 gives them), with the base's -//! pin verified when it carries one. Any other base (PNG, FLAC: this tool has no +//! pin verified when it carries one, under the cost limit `--limit OUT,SYMBOLS,WORKING,MACS` +//! gives (by default `staged::TOOL_DEFAULT_LIMIT`), applied before any symbol is decoded. +//! Every layer is held to the same limit's output, symbols and working bytes +//! (`staged::layer_limit`, staged-delivery Appendix A.5), and every chain to the +//! stated chain budget (`staged::TOOL_DEFAULT_CHAIN_BUDGET`). Any other base (PNG, FLAC: this tool has no //! `codecs` decoder for them) is read from `--stage0`, the units' canonical buffers //! concatenated, with `--units N --dims D0,D1,D2 --elem u8|i16`. @@ -91,8 +95,15 @@ fn int_base(path: &str, env: &[u8], args: &[String]) -> Result<(BaseInfo, Vec Result), wai::staged::BaseError>, String> { let set = param_set(args)?; let files: BTreeMap = set.iter().map(|(k, v)| (k.clone(), v.as_slice())).collect(); - // No cost limit and no key log: a tool over files its user chose (check 5 skipped). - Ok(wai::staged::decode_base(env, &files, None, None)) + // The tool's stated cost limit, or `--limit`; no key log: a tool over files its user + // chose (check 5 skipped). + Ok(wai::staged::decode_base(env, &files, Some(&limit(args)?), None)) +} + +/// The cost limit a base is decoded under: `--limit OUT,SYMBOLS,WORKING,MACS`, or the +/// tools' stated default (`staged::TOOL_DEFAULT_LIMIT`). +fn limit(args: &[String]) -> Result { + opt(args, "--limit").map_or(Ok(wai::staged::TOOL_DEFAULT_LIMIT), wai::staged::parse_limit) } /// The base's info and units, decoded as the module doc states. @@ -141,8 +152,17 @@ fn below(s: &Samples) -> Below<'_> { } } -fn cfg() -> ChainConfig { - ChainConfig { bound: HoldBound { max_envelopes: 1024, max_age_ms: u64::MAX }, cost_limit: None, budget: u64::MAX, record_budget: u64::MAX, live: None } +/// The chain configuration of `apply` and `refine`: each layer held to the output, +/// symbols and working bytes of the cost limit (`--limit`, by default the tools' +/// stated limit), each chain to the stated chain budget. +fn cfg(args: &[String]) -> Result { + Ok(ChainConfig { + bound: HoldBound { max_envelopes: 1024, max_age_ms: u64::MAX }, + cost_limit: Some(wai::staged::layer_limit(&limit(args)?)), + budget: wai::staged::TOOL_DEFAULT_CHAIN_BUDGET, + record_budget: u64::MAX, + live: None, + }) } fn print_units(c: &StageChain) { @@ -271,7 +291,7 @@ fn apply(args: &[String]) -> Result<(), String> { if pos.is_empty() { return Err("wai stage apply … [--json]".into()); } - let mut sink = StageSink::new(cfg()); + let mut sink = StageSink::new(cfg(args)?); sink.set_trace(true); let mut bases: Vec<[u8; 32]> = Vec::new(); let mut frozen: BTreeMap<([u8; 32], u32), bool> = BTreeMap::new(); @@ -438,7 +458,7 @@ fn base_cmd(args: &[String]) -> Result<(), String> { let cap = wai::codecs::CAP_VIDEO_INT_MOTION; let m = wai::Manifest::new("video", "replicate", wai::ModelRequirement::new(cap), wai::Conditioning::new("int_motion")); let bytes = Wai::new(m, payload).to_emit_bytes().map_err(|e| e.to_string())?; - let (info, units) = wai::staged::decode_base(&bytes, &BTreeMap::new(), None, None).map_err(|e| e.to_string())?; + let (info, units) = wai::staged::decode_base(&bytes, &BTreeMap::new(), Some(&limit(args)?), None).map_err(|e| e.to_string())?; if units.len() != recon.len() || units.iter().zip(&recon).any(|(u, r)| u.samples != Samples::U8(r.clone())) { return Err("the base does not decode to the encoder's reconstruction".into()); } @@ -468,7 +488,7 @@ fn refine(args: &[String]) -> Result<(), String> { }; let base_hash = info.content_hash; let n_units = units.len() as u32; - let mut c = StageChain::new(info, units, cfg()); + let mut c = StageChain::new(info, units, cfg(args)?); for l in opts(args, "--on") { let bytes = read(l)?; c.offer(&bytes, 0, ReceiptEvidence::None).map_err(|e| format!("{l}: {e}"))?; @@ -656,5 +676,8 @@ wai stage base --frames --dims N,H,W --q Q -o base.wai a wai.video.int_ wai stage inspect a layer's refines, WIR1 header and declared cost wai stage pack … -o x.wai pack WAI1 envelopes into a WAI2 (manifests verbatim) wai stage explode [-o out.wai] a WAI2 component as its WAI1 envelope - a base of an integer codec decodes with --set (its parameter files by name); any - other base is read from --stage0 --units N --dims D0,D1,D2 [--elem u8|i16]"; + a base of an integer codec decodes with --set (its parameter files by name), under + --limit OUT,SYMBOLS,WORKING,MACS (by default the tools' stated limit), which also bounds each + layer's output, symbols and working bytes; a chain's units are held to the stated chain + budget; any other base is + read from --stage0 --units N --dims D0,D1,D2 [--elem u8|i16]"; diff --git a/wai/wai-rs/src/bin/wai_decode_bench.rs b/wai/wai-rs/src/bin/wai_decode_bench.rs index 37004ed89..d065f0249 100644 --- a/wai/wai-rs/src/bin/wai_decode_bench.rs +++ b/wai/wai-rs/src/bin/wai_decode_bench.rs @@ -479,49 +479,13 @@ fn clone_layer(l: &IntLayer) -> IntLayer { // ─────────────────────────────── harness ─────────────────────────────── -fn sh(cmd: &str, args: &[&str]) -> Option { - let o = std::process::Command::new(cmd).args(args).output().ok()?; - o.status.success().then(|| String::from_utf8_lossy(&o.stdout).trim().to_string()) -} - -/// The core types as the OS reports them, e.g. `6 Super + 12 Performance`: each -/// macOS performance level's logical-core count and its own name -/// (`hw.perflevelN.name`), in the OS's order. Names are reported as given, not -/// inferred from the level's index, since what level 0 and 1 are differs between -/// chips. `None` where the OS does not report levels. -fn core_levels() -> Option { - let n: usize = sh("sysctl", &["-n", "hw.nperflevels"])?.parse().ok()?; - let parts: Vec = (0..n) - .map(|i| { - let cpus = sh("sysctl", &["-n", &format!("hw.perflevel{i}.logicalcpu")])?; - let name = sh("sysctl", &["-n", &format!("hw.perflevel{i}.name")]).unwrap_or_else(|| format!("level-{i}")); - Some(format!("{cpus} {name}")) - }) - .collect::>()?; - (!parts.is_empty()).then(|| parts.join(" + ")) -} - +/// The machine line (`wai::bench_env`, shared with the staged-measurement bench). fn machine() -> String { - let cpu = sh("sysctl", &["-n", "machdep.cpu.brand_string"]) - .or_else(|| { - std::fs::read_to_string("/proc/cpuinfo").ok()?.lines().find(|l| l.starts_with("model name")).map(|l| l.split(':').nth(1).unwrap_or("").trim().to_string()) - }) - .unwrap_or_else(|| "unknown CPU".into()); - let levels = core_levels().map(|l| format!(" ({l})")).unwrap_or_default(); - format!( - "{cpu}, {} logical cores{levels}; {}-{}; {} build", - std::thread::available_parallelism().map(|n| n.get()).unwrap_or(0), - std::env::consts::OS, - std::env::consts::ARCH, - if cfg!(debug_assertions) { "DEBUG (numbers not meaningful)" } else { "release" }, - ) + wai::bench_env::machine().to_string() } fn loadavg() -> String { - sh("sysctl", &["-n", "vm.loadavg"]) - .map(|s| s.trim_matches(|c| c == '{' || c == '}' || c == ' ').to_string()) - .or_else(|| std::fs::read_to_string("/proc/loadavg").ok().map(|s| s.split_whitespace().take(3).collect::>().join(" "))) - .unwrap_or_else(|| "n/a".into()) + wai::bench_env::loadavg().map(|(a, b, c)| format!("{a} {b} {c}")).unwrap_or_else(|| "n/a".into()) } fn ms(d: Duration) -> f64 { diff --git a/wai/wai-rs/src/bin/wai_stage_bench.rs b/wai/wai-rs/src/bin/wai_stage_bench.rs index 1a6ee8573..b811cf509 100644 --- a/wai/wai-rs/src/bin/wai_stage_bench.rs +++ b/wai/wai-rs/src/bin/wai_stage_bench.rs @@ -1,11 +1,14 @@ //! `wai_stage_bench` — measure one staged set natively (`extensions/staged-delivery.md`; //! the measurement method of the staged-delivery design, §9). //! -//! wai_stage_bench --source [--set ] [--runs N] [--out report.json] … +//! wai_stage_bench --source [--set ] [--limit OUT,SYMBOLS,WORKING,MACS] [--runs N] [--out report.json] … //! //! The envelopes are a stage set: bases (no `refines`), decoded through //! `staged::decode_base` against the parameter set in `--set` (none for -//! `wai.video.int_motion`), and their refinement layers. Stage 0 is every base decoded; +//! `wai.video.int_motion`), and their refinement layers. Every base is decoded under the +//! cost limit `--limit` gives (by default `staged::TOOL_DEFAULT_LIMIT`), applied before +//! any symbol is decoded; every layer is held to that limit's output, symbols and +//! working bytes, and every chain to `staged::TOOL_DEFAULT_CHAIN_BUDGET`. Stage 0 is every base decoded; //! stage `k` is every layer whose `refines.layer` is `k` applied, in the order given. //! `--source` holds what the units approximate, at their final shape: each unit's //! canonical buffer as the last stage shapes it, base by base in the order given and @@ -69,8 +72,33 @@ fn read(path: &str) -> Result, String> { std::fs::read(path).map_err(|e| format!("read {path}: {e}")) } -fn cfg() -> ChainConfig { - ChainConfig { bound: HoldBound { max_envelopes: 4096, max_age_ms: u64::MAX }, cost_limit: None, budget: u64::MAX, record_budget: u64::MAX, live: None } +/// The cost limit: `--limit OUT,SYMBOLS,WORKING,MACS`, or the tools' stated default. +fn limit(args: &[String]) -> Result { + opt(args, "--limit").map_or(Ok(wai::staged::TOOL_DEFAULT_LIMIT), wai::staged::parse_limit) +} + +/// The sink's configuration: each layer held to the cost limit's output, symbols and +/// working bytes, each chain to the stated chain budget. +fn cfg(limit: &wai::int_payload::DeclaredCost) -> ChainConfig { + ChainConfig { + bound: HoldBound { max_envelopes: 4096, max_age_ms: u64::MAX }, + cost_limit: Some(wai::staged::layer_limit(limit)), + budget: wai::staged::TOOL_DEFAULT_CHAIN_BUDGET, + record_budget: u64::MAX, + live: None, + } +} + +/// Add a base to the sink, refusing one the sink does not take whole (its units past +/// the chain budget). +fn add_base(sink: &mut StageSink, file: &str, info: BaseInfo, units: Vec) -> Result<(), String> { + let h = info.content_hash; + let reports = sink.add_base(info, units); + if sink.chain(&h).is_none() { + let why = reports.iter().find_map(|r| r.envelope).map_or("not taken", |r| r.code()); + return Err(format!("{file}: the base is refused ({why})")); + } + Ok(()) } /// One envelope of the set, read once. @@ -173,6 +201,7 @@ fn run(args: &[String]) -> Result<(), String> { None => BTreeMap::new(), }; let files: BTreeMap = set.iter().map(|(k, v)| (k.clone(), v.as_slice())).collect(); + let limit = limit(args)?; let envs: Vec = positional(args).iter().map(|f| read_env(f)).collect::>()?; let n_stages = envs.iter().map(|e| e.stage).max().ok_or("no envelopes")?; if !envs.iter().any(|e| e.stage == 0) { @@ -183,15 +212,15 @@ fn run(args: &[String]) -> Result<(), String> { // apply whole: the bench measures a set delivered whole and in order. let mut bases: Vec<(BaseInfo, Vec)> = Vec::new(); for e in envs.iter().filter(|e| e.stage == 0) { - bases.push(wai::staged::decode_base(&e.bytes, &files, None, None).map_err(|x| format!("{}: {x}", e.file))?); + bases.push(wai::staged::decode_base(&e.bytes, &files, Some(&limit), None).map_err(|x| format!("{}: {x}", e.file))?); } let hashes: Vec<[u8; 32]> = bases.iter().map(|(i, _)| i.content_hash).collect(); // stages[k][b][u]: the unit's samples, stage and stage hash at stage k. let mut stages_seen: Vec)>>> = Vec::new(); { - let mut sink = StageSink::new(cfg()); - for (info, units) in &bases { - sink.add_base(info.clone(), units.clone()); + let mut sink = StageSink::new(cfg(&limit)); + for ((info, units), e) in bases.iter().zip(envs.iter().filter(|e| e.stage == 0)) { + add_base(&mut sink, &e.file, info.clone(), units.clone())?; } let snap = |sink: &StageSink| -> Vec)>> { hashes.iter().map(|h| { @@ -204,7 +233,8 @@ fn run(args: &[String]) -> Result<(), String> { for (i, e) in envs.iter().enumerate().filter(|(_, e)| e.stage == k) { let reports = sink.offer(&e.bytes, i as u64, ReceiptEvidence::None).map_err(|x| format!("{}: {x}", e.file))?; if let Some(bad) = reports.iter().find(|x| x.envelope.is_some() || !x.refused.is_empty() || !x.held.is_empty() || x.held_for_base) { - return Err(format!("{}: not applied whole ({:?}); the bench measures a set delivered whole and in order", e.file, bad)); + let why = bad.envelope.map(|r| r.code()).or_else(|| bad.refused.first().map(|&(_, r)| r.code())).unwrap_or("held"); + return Err(format!("{}: not applied whole ({why}); the bench measures a set delivered whole and in order", e.file)); } } stages_seen.push(snap(&sink)); @@ -258,11 +288,11 @@ fn run(args: &[String]) -> Result<(), String> { // Timed replays. let mut times: Vec> = vec![Vec::new(); n_stages as usize + 1]; for _ in 0..runs { - let mut sink = StageSink::new(cfg()); + let mut sink = StageSink::new(cfg(&limit)); let t = Instant::now(); for e in envs.iter().filter(|e| e.stage == 0) { - let (info, units) = wai::staged::decode_base(&e.bytes, &files, None, None).map_err(|x| format!("{}: {x}", e.file))?; - sink.add_base(info, units); + let (info, units) = wai::staged::decode_base(&e.bytes, &files, Some(&limit), None).map_err(|x| format!("{}: {x}", e.file))?; + add_base(&mut sink, &e.file, info, units)?; } times[0].push(t.elapsed().as_secs_f64() * 1e6); for k in 1..=n_stages { @@ -391,7 +421,7 @@ fn run(args: &[String]) -> Result<(), String> { fn main() -> ExitCode { let args: Vec = std::env::args().skip(1).collect(); if args.is_empty() { - eprintln!("usage: wai_stage_bench --source [--set ] [--runs N≥20] [--out report.json] …"); + eprintln!("usage: wai_stage_bench --source [--set ] [--limit OUT,SYMBOLS,WORKING,MACS] [--runs N≥20] [--out report.json] …"); return ExitCode::from(2); } match run(&args) { diff --git a/wai/wai-rs/src/bin/wai_stage_measure.rs b/wai/wai-rs/src/bin/wai_stage_measure.rs new file mode 100644 index 000000000..1632e1b92 --- /dev/null +++ b/wai/wai-rs/src/bin/wai_stage_measure.rs @@ -0,0 +1,1080 @@ +//! `wai_stage_measure` — the deterministic figures of staged delivery +//! (`extensions/staged-measurement.md`), and the corpus that pins them +//! (`staged-measure-conformance/`). +//! +//! wai_stage_measure eval [--json] [--limit OUT,SYMBOLS,WORKING,MACS] +//! wai_stage_measure sim +//! wai_stage_measure verify +//! wai_stage_measure generate-goldens +//! wai_stage_measure gen-fixtures +//! +//! `eval` evaluates one title (`stage_measure::run::evaluate`) and prints its figures, +//! or with `--json` the block `goldens.json` holds for it. `sim` runs one delivery plan +//! (`stage_measure::sim`) over a model or a title, and prints the simulated result. `verify` regenerates every +//! fixture file in memory and requires it byte for byte, evaluates every title, checks +//! it against the fixture tool's own reconstruction (`expect.json`), and requires +//! `goldens.json` byte for byte; it does the same for the simulator's models, plans and +//! cases, simulates every case, checks its claim against §10.4's bound and requires +//! `sim/results.json` byte for byte. `generate-goldens` writes `goldens.json` and +//! `sim/results.json`; +//! `gen-fixtures` writes the fixtures. Neither runs in CI: CI runs `verify`. +//! +//! The fixtures are staged sets over bases the staged-delivery corpus already holds +//! (`staged-conformance/cases/*/base.wai`, decoded against their pinned parameter +//! sets), with refinement layers coded here by `staged::wir1::encode` toward a source +//! computed from the base by integer functions. No third-party content is used. + +use std::collections::BTreeMap; +use std::path::PathBuf; +use std::process::ExitCode; + +use serde_json::{json, Value}; +use wai::container::{Conditioning, Manifest, ModelRequirement, Prev, Refines, Span, Wai}; +use wai::stage_measure::decode::{decode_base, DecodedBase}; +use wai::stage_measure::run::{check_expect, check_refusal, evaluate, record_dedup, record_dedup_json, Evaluation, QualityKind}; +use wai::stage_measure::sim::{self, Carriage, DecodeOrder, LateRefinement, Mode, Model, Plan, Schedule, SimObject}; +use wai::stage_measure::title::{CaseFacts, Limits, Obj, SetFile, Title, TitleBase}; +use wai::stage_measure::{MeasureError, EXPECT_FORMAT, GOLDENS_FORMAT}; +use wai::staged::wir1::{self, Below, Next, Predictor, RefineParams}; +use wai::staged::{Samples, UnitBuf}; + +fn hex(b: &[u8]) -> String { + b.iter().map(|x| format!("{x:02x}")).collect() +} + +fn sha(b: &[u8]) -> String { + wai::pin::sha256_hex(b) +} + +fn pretty(v: &Value) -> Vec { + let mut s = serde_json::to_string_pretty(v).expect("JSON"); + s.push('\n'); + s.into_bytes() +} + +/// The corpus directory, its name below `wai/`, and `wai/`. +struct Corpus { + dir: PathBuf, + rel: String, + wai: PathBuf, +} + +impl Corpus { + fn new(p: &str) -> Result { + let dir = std::fs::canonicalize(p).map_err(|e| format!("{p}: {e}"))?; + let rel = dir.file_name().and_then(|n| n.to_str()).ok_or("the corpus has no name")?.to_string(); + let wai = dir.parent().ok_or("the corpus has no parent")?.to_path_buf(); + Ok(Corpus { dir, rel, wai }) + } + fn fixture_rel(&self, name: &str) -> String { + format!("{}/fixtures/{name}", self.rel) + } + fn refusal_rel(&self, name: &str) -> String { + format!("{}/refusals/{name}", self.rel) + } + /// The fixture directories, by name. + fn fixtures(&self) -> Result, String> { + let mut v: Vec = std::fs::read_dir(self.dir.join("fixtures")) + .map_err(|e| format!("fixtures: {e}"))? + .filter_map(|e| e.ok()) + .filter(|e| e.path().join("title.json").is_file()) + .filter_map(|e| e.file_name().to_str().map(str::to_string)) + .collect(); + v.sort(); + Ok(v) + } +} + +// ───────────────────────── generating the fixtures ───────────────────────── + +/// A base of the staged-delivery corpus, read as a title reads it. +fn case_base(c: &Corpus, case: &str, index: usize) -> Result { + let read = |p: &str| std::fs::read(c.wai.join(p)).map_err(|e| format!("{p}: {e}")); + let cj: Value = serde_json::from_slice(&read(&format!("{case}/case.json"))?).map_err(|e| e.to_string())?; + let b = &cj["bases"][index]; + let file = format!("{case}/{}", b["file"].as_str().ok_or("base file")?); + let mut set = Vec::new(); + for (m, p) in b["decode"]["set"].as_object().ok_or("decode set")? { + let p = p.as_str().ok_or("set path")?; + set.push(SetFile { member: m.clone(), path: p.to_string(), bytes: read(p)? }); + } + let stage0 = read(&format!("{case}/{}", b["stage0"].as_str().ok_or("stage0")?))?; + let case = CaseFacts { + decoded_via: b["decoded_via"].as_str().ok_or("decoded_via")?.to_string(), + via_fallback: b["via_fallback"].as_bool().ok_or("via_fallback")?, + tier: b["tier"].as_str().ok_or("tier")?.to_string(), + pin_resolved: b["pin_resolved"].as_bool().ok_or("pin_resolved")?, + }; + Ok(TitleBase { obj: Obj { bytes: read(&file)?, path: file }, first_unit: 0, capture_us: None, set, stage0: Some(stage0), case: Some(case) }) +} + +/// Fine detail the base lacks, from integer functions of position, channel and seed: +/// a value in `−8..=8`. +fn tex(x: u64, y: u64, c: u64, seed: u64) -> i64 { + ((((x * 7 + y * 13 + c * 5 + seed * 11) ^ (x * y + seed)) % 17) as i64) - 8 +} + +/// A triangle wave of `period` samples and amplitude `amp`. +fn tri(t: u64, period: u64, amp: i64) -> i64 { + let p = (t % period) as i64; + let half = period as i64 / 2; + amp * (if p < half { 2 * p - half } else { 3 * half - 2 * p }) / half.max(1) +} + +fn u8_of(s: &Samples) -> &[u8] { + match s { + Samples::U8(v) => v, + Samples::I16(_) => panic!("u8 samples expected"), + } +} + +fn i16_of(s: &Samples) -> &[i16] { + match s { + Samples::I16(v) => v, + Samples::U8(_) => panic!("i16 samples expected"), + } +} + +fn below(s: &Samples) -> Below<'_> { + match s { + Samples::U8(v) => Below::U8(v), + Samples::I16(v) => Below::I16(v), + } +} + +fn samples(n: Next) -> Samples { + match n { + Next::U8(v) => Samples::U8(v), + Next::I16(v) => Samples::I16(v), + } +} + +/// One layer to code: its predictor, step and (for rate) ratio, over every unit of the +/// base, or one envelope per unit. +struct LayerSpec { + predictor: Predictor, + q: u32, + rate: (u32, u32), + per_unit: bool, +} + +/// The fixture tool's own encoder state: each unit's current reconstruction and its +/// producer, and the envelopes written. +struct Coder<'a> { + base: &'a DecodedBase, + cap: &'static str, + media: &'static str, + units: Vec, + producers: Vec, + layer: u8, + /// Each layer's envelopes, in order. + out: Vec>, + /// (unit, layers, digest) of every stage coded. + digests: Vec<(u64, u16, String)>, +} + +impl<'a> Coder<'a> { + fn new(base: &'a DecodedBase, cap: &'static str, media: &'static str) -> Self { + let units = base.units.clone(); + let producers = vec![hex(&base.info.content_hash); units.len()]; + let digests = units.iter().enumerate().map(|(u, b)| (u as u64, 1u16, sha(&b.samples.canonical_bytes()))).collect(); + Coder { base, cap, media, units, producers, layer: 0, out: vec![], digests } + } + + /// Code one layer toward `targets` (each unit's target at the layer's output shape). + fn layer(&mut self, spec: &LayerSpec, targets: &[UnitBuf]) -> Result<(), String> { + self.layer += 1; + let kind = wai::staged::unit_kind(self.cap).ok_or("not a refinement capability")?; + let groups: Vec> = if spec.per_unit { (0..self.units.len()).map(|u| vec![u]).collect() } else { vec![(0..self.units.len()).collect()] }; + for g in groups { + let below_shape = self.units[g[0]].shape; + let dims = targets[g[0]].shape; + let belows: Vec = g.iter().map(|&u| below(&self.units[u].samples)).collect(); + let tg: Vec = g.iter().map(|&u| below(&targets[u].samples)).collect(); + let params = RefineParams { q: spec.q, predictor: spec.predictor, rate: spec.rate, table_per_unit: false, max_symbols: 4095 }; + let (payload, recon) = wir1::encode(kind, &belows, below_shape, &tg, dims, ¶ms).map_err(|e| e.to_string())?; + let prev = self.producers[g[0]].clone(); + if g.iter().any(|&u| self.producers[u] != prev) { + return Err("a layer's units have different producers".into()); + } + let span = if g.len() == self.units.len() { None } else { Some(Span { from: g[0] as u32, count: g.len() as u32 }) }; + let refines = Refines { base: hex(&self.base.info.content_hash), prev: Prev::One(prev), layer: self.layer, span }; + let m = Manifest::new(self.media, "replicate", ModelRequirement::new(self.cap), Conditioning::new("wir1")).with_refines(&refines); + let env = Wai::new(m, payload).to_emit_bytes().map_err(|e| e.to_string())?; + let h = hex(&wai::jwp_hash::content_hash(&env)); + for (&u, r) in g.iter().zip(recon) { + self.units[u] = UnitBuf { shape: dims, samples: samples(r) }; + self.producers[u] = h.clone(); + self.digests.push((u as u64, self.layer as u16 + 1, sha(&self.units[u].samples.canonical_bytes()))); + } + self.out.push(env); + } + Ok(()) + } +} + +/// What one fixture is. +struct FixtureSpec { + name: &'static str, + about: &'static str, + case: &'static str, + cap: &'static str, + media: &'static str, + layers: Vec, + /// Each reference: its label and its one layer. + references: Vec<(&'static str, LayerSpec)>, +} + +fn specs() -> Vec { + use Predictor::*; + let l = |predictor, q, rate, per_unit| LayerSpec { predictor, q, rate, per_unit }; + vec![ + FixtureSpec { + name: "still", + about: "a pinned wai.neural.int_hyper picture (16×16); an up2 layer to 32×32 at q 8, then identity layers at q 3 and q 1; references: one up2 layer at q 1, 3 and 8", + case: "staged-conformance/cases/image_int_hyper_2l", + cap: wai::codecs::CAP_IMAGE_INT_REFINE, + media: "image", + layers: vec![l(Up2, 8, (0, 0), false), l(Identity, 3, (0, 0), false), l(Identity, 1, (0, 0), false)], + references: vec![("one layer, q 1", l(Up2, 1, (0, 0), false)), ("one layer, q 3", l(Up2, 3, (0, 0), false)), ("one layer, q 8", l(Up2, 8, (0, 0), false))], + }, + FixtureSpec { + name: "clip", + about: "a pinned wai.video.int_hyper clip (5 frames, 16×16); one identity layer per frame at q 6, then one per frame at q 2; references: one layer per frame at q 2 and at q 6", + case: "staged-conformance/cases/video_int_hyper_keyframe", + cap: wai::codecs::CAP_VIDEO_INT_REFINE, + media: "video", + layers: vec![l(Identity, 6, (0, 0), true), l(Identity, 2, (0, 0), true)], + references: vec![("one layer, q 2", l(Identity, 2, (0, 0), true)), ("one layer, q 6", l(Identity, 6, (0, 0), true))], + }, + FixtureSpec { + name: "audio", + about: "a pinned wai.audio.int_codec clip (8192 samples, mono); a rate layer to twice the sample rate at q 64, then identity layers at q 8 and q 1; references: one rate layer at q 1 and at q 8", + case: "staged-conformance/cases/audio_int_codec_1l", + cap: wai::codecs::CAP_AUDIO_INT_REFINE, + media: "audio", + layers: vec![l(Rate, 64, (2, 1), false), l(Identity, 8, (0, 0), false), l(Identity, 1, (0, 0), false)], + references: vec![("one layer, q 1", l(Rate, 1, (2, 1), false)), ("one layer, q 8", l(Rate, 8, (2, 1), false))], + }, + FixtureSpec { + name: "splat", + about: "a pinned wai.splat.int_codec attribute tensor (14 planes of 32×32); an identity layer at q 6, then one at q 1; reference: one identity layer at q 1", + case: "staged-conformance/cases/splat_attr_1l", + cap: wai::codecs::CAP_SPLAT_INT_REFINE, + media: "volumetric", + layers: vec![l(Identity, 6, (0, 0), false), l(Identity, 1, (0, 0), false)], + references: vec![("one layer, q 1", l(Identity, 1, (0, 0), false))], + }, + ] +} + +/// The source of a fixture: each unit at its final shape, from the base's decode and +/// integer detail. +fn source_units(spec: &FixtureSpec, base: &DecodedBase) -> Vec { + base.units + .iter() + .enumerate() + .map(|(u, b)| { + let (d0, d1, d2) = b.shape; + match spec.name { + "still" => { + let (h, w) = (2 * d0 as usize, 2 * d1 as usize); + let prev = u8_of(&b.samples); + let mut px = Vec::with_capacity(h * w * 3); + for y in 0..h { + for x in 0..w { + for c in 0..3 { + let p = wai::staged::predict::up2_at(prev, d0 as usize, d1 as usize, 3, y, x, c); + px.push((p + tex(x as u64, y as u64, c as u64, 3)).clamp(0, 255) as u8); + } + } + } + UnitBuf { shape: (h as u32, w as u32, 3), samples: Samples::U8(px) } + } + "audio" => { + let prev = i16_of(&b.samples); + let c = d2 as usize; + let n = 2 * d0 as usize; + let mut v = Vec::with_capacity(n * c); + for i in 0..n { + for ch in 0..c { + let p = wai::staged::predict::rate_at(prev, c, 2, 1, i, ch); + let detail = tri(i as u64, 37, 700) + tri(i as u64 * 3, 1009, 300); + v.push((p + detail).clamp(-32768, 32767) as i16); + } + } + UnitBuf { shape: (n as u32, 1, d2), samples: Samples::I16(v) } + } + _ => { + // A frame of the clip, or the splat's attribute planes: the same shape. + let prev = u8_of(&b.samples); + let px = prev + .iter() + .enumerate() + .map(|(i, &p)| { + let (a, r) = (i as u64 / (d1 * d2) as u64, i as u64 % (d1 * d2) as u64); + (p as i64 + 2 * tex(r % d2 as u64 + a, r / d2 as u64, a, 5 + u as u64)).clamp(0, 255) as u8 + }) + .collect(); + UnitBuf { shape: b.shape, samples: Samples::U8(px) } + } + } + }) + .collect() +} + +/// Every file of one fixture, by name, and nothing written. +fn build_fixture(c: &Corpus, spec: &FixtureSpec) -> Result>, String> { + let tb = case_base(c, spec.case, 0)?; + let base = decode_base(&tb, &Limits::default()).map_err(|e| e.to_string())?; + let src = source_units(spec, &base); + let mut files = BTreeMap::new(); + let mut coder = Coder::new(&base, spec.cap, spec.media); + // A layer that changes the shape (up2, rate) is coded toward the source as it is; + // the units are already at the source's shape for every later layer. + for ls in &spec.layers { + coder.layer(ls, &src)?; + } + let mut layer_names = Vec::new(); + for (k, env) in coder.out.iter().enumerate() { + let name = format!("L{:02}.wai", k + 1); + files.insert(name.clone(), env.clone()); + layer_names.push(name); + } + let mut digests = coder.digests.clone(); + digests.sort(); + let mut refs_json = Vec::new(); + let mut refs_expect = Vec::new(); + for (ri, (label, ls)) in spec.references.iter().enumerate() { + let mut rc = Coder::new(&base, spec.cap, spec.media); + rc.layer(ls, &src)?; + let mut names = Vec::new(); + for (j, env) in rc.out.iter().enumerate() { + let name = format!("R{ri}_{j:02}.wai"); + files.insert(name.clone(), env.clone()); + names.push(name); + } + refs_json.push(json!({"label": label, "layers": names})); + refs_expect.push(json!({"label": label, "out_sha256": rc.units.iter().map(|u| sha(&u.samples.canonical_bytes())).collect::>()})); + } + let source: Vec = src.iter().flat_map(|u| u.samples.canonical_bytes()).collect(); + let unit_us = match spec.name { + "audio" => { + let w = Wai::from_bytes(&tb.obj.bytes).map_err(|e| e.to_string())?; + let sr = wai::int_wire::parse_wia1(&w.payload).map_err(|e| format!("{e:?}"))?.sr as u64; + base.units[0].shape.0 as u64 * 1_000_000 / sr + } + _ => 40_000, + }; + let title = json!({ + "format": wai::stage_measure::title::FORMAT, + "title": spec.name, + "about": spec.about, + "unit_us": unit_us, + "bases": [{"case": spec.case, "base": 0, "first_unit": 0, "capture_us": null}], + "layers": layer_names, + "source": {"file": "source.bin", "sha256": sha(&source)}, + "references": refs_json, + "expect": "expect.json", + }); + let expect = json!({ + "format": EXPECT_FORMAT, + "about": "The fixture tool's own reconstruction: the SHA-256 of each unit's canonical buffer at each stage it coded, and of each reference's final units, from the encoder's reconstruction loop.", + "presentations": digests.iter().map(|(u, l, d)| json!({"unit": u, "layers": l, "out_sha256": d})).collect::>(), + "references": refs_expect, + }); + files.insert("title.json".into(), pretty(&title)); + files.insert("source.bin".into(), source); + files.insert("expect.json".into(), pretty(&expect)); + Ok(files) +} + +/// Objects whose counting §3.2 decides one way or the other: each written here, or an +/// object the corpora already hold, by path below `wai/`. +fn counting_cases(c: &Corpus) -> Result>)>, String> { + use wai::container::{Conditioning, Manifest, ModelRequirement, Rendition, Security, WaiMulti}; + let wih1 = std::fs::read(c.wai.join("wai-rs/tests/vectors/int_hyper/tiny_bitstream.wih1")).map_err(|e| e.to_string())?; + let json_form = std::fs::read(c.wai.join("wai-rs/tests/vectors/int_hyper/tiny_bitstream.json")).map_err(|e| e.to_string())?; + let hyper = wai::codecs::CAP_NEURAL_INT_HYPER; + let one = |cap: &str, fb: Option<&str>, payload: Vec| -> Vec { + let mut mr = ModelRequirement::new(cap); + if let Some(f) = fb { + mr = mr.with_fallback(f); + } + Wai::new(Manifest::new("image", "replicate", mr, Conditioning::new("int_hyper")), payload).to_bytes().expect("serialise") + }; + const B64: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let base64: Vec = wih1 + .chunks(3) + .flat_map(|ch| { + let n = (ch[0] as u32) << 16 | (*ch.get(1).unwrap_or(&0) as u32) << 8 | *ch.get(2).unwrap_or(&0) as u32; + (0..4).map(move |i| if i <= ch.len() { B64[(n >> (18 - 6 * i) & 63) as usize] } else { b'=' }) + }) + .collect(); + let quoted = [b"\"".as_slice(), &base64, b"\""].concat(); + let wai2 = |digest_ok: bool| -> Result, String> { + let mut w = WaiMulti::new("image", "replicate", Value::Null, vec![Rendition::media(hyper, "int_hyper", wih1.clone()).with_payload_digest()]); + if !digest_ok { + w.manifest.renditions[0].sha256 = Some(sha(b"another payload")); + } + w.to_bytes().map_err(|e| e.to_string()) + }; + let mut enc = Manifest::new("image", "replicate", ModelRequirement::new(hyper), Conditioning::new("int_hyper")); + enc.security = Some(Security::new("aes-256-gcm", 1)); + let mut trailing = one(hyper, None, wih1.clone()); + trailing.push(0); + Ok(vec![ + ("wih1".into(), Some(one(hyper, None, wih1.clone()))), + ("staged-measure-conformance/fixtures/still/L01.wai".into(), None), + ("json-form".into(), Some(one(hyper, None, json_form))), + ("base64".into(), Some(one(hyper, None, base64))), + ("json-string".into(), Some(one(hyper, None, quoted))), + ("bom".into(), Some(one(hyper, None, [b"\xef\xbb\xbf".as_slice(), &wih1].concat()))), + ("staged-conformance/cases/image_png_up2/base.wai".into(), None), + ("unregistered-capability".into(), Some(one("wai.image.unregistered", None, wih1.clone()))), + ("fallback-mismatch".into(), Some(one(hyper, Some(wai::codecs::CAP_VIDEO_INT_MOTION), wih1.clone()))), + ("fallback-unchecked".into(), Some(one(hyper, Some(wai::codecs::CAP_IMAGE_PNG), wih1.clone()))), + ("wai2".into(), Some(wai2(true)?)), + ("wai2-sha256-mismatch".into(), Some(wai2(false)?)), + ("encrypted".into(), Some(Wai::new(enc, wih1.clone()).to_bytes().map_err(|e| e.to_string())?)), + ("trailing".into(), Some(trailing)), + ]) +} + +/// `counting/`: the objects written here, and `counting.json`, each case's counted bytes +/// or its refusal's code. +fn counting_files(c: &Corpus) -> Result>, String> { + let mut files = BTreeMap::new(); + let mut cases = Vec::new(); + for (name, bytes) in counting_cases(c)? { + let (path, b) = match bytes { + Some(b) => { + let rel = format!("counting/{name}.wai"); + files.insert(rel.clone(), b.clone()); + (format!("{}/{rel}", c.rel), b) + } + None => (name.clone(), std::fs::read(c.wai.join(&name)).map_err(|e| format!("{name}: {e}"))?), + }; + cases.push(match wai::measure_framing::count_object(&b) { + Ok(k) => json!({"file": path, "counted": {"header": k.header, "manifest": k.manifest, "payload": k.payload, "total": k.total}, "refused": null, "component": null}), + Err(e) => json!({"file": path, "counted": null, "refused": e.code(), "component": match &e { + wai::measure_framing::FramingError::NotCountable { component, .. } => json!(component), + _ => Value::Null, + }}), + }); + } + files.insert("counting.json".into(), pretty(&json!({ + "format": "wai-measure-counting/1", + "about": "Objects whose counting extensions/staged-measurement.md §3.2 decides: each object's counted bytes, or the code it is not countable with (and the WAI2 component). Written by wai_stage_measure gen-fixtures; required by wai_stage_measure verify and measure_verify.py.", + "cases": cases, + }))); + Ok(files) +} + +/// Titles a tool must refuse, each with the code: a base past the chain budget its +/// title states, the two bound vectors whose headers declare outsized clips (one under +/// a title that states limits past the tools'), two envelopes SPEC §4 never dispatches, +/// and a layer past the cost limit its title states. +fn refusals() -> Vec<(&'static str, &'static str, Value)> { + let env = |path: &str| json!({"envelope": path, "set": {}, "first_unit": 0, "capture_us": null}); + vec![ + ("base-over-budget", "chain-over-limit", json!({"base": {"case": "staged-conformance/cases/base_over_budget", "base": 0, "first_unit": 0, "capture_us": null}, "limits": {"cost": null, "chain_budget": 383}, + "about": "a base whose units' output passes the chain budget the title states"})), + ("clip-64mib", "cost_over_limit", json!({"base": env("wai-rs/tests/vectors/staged_bounds/clip_64mib.wai"), "limits": null, + "about": "a wai.video.int_motion base whose header declares 64 MiB of output, past the tools' stated limit"})), + ("clip-1gib", "cost_over_limit", json!({"base": env("wai-rs/tests/vectors/staged_bounds/clip_1gib.wai"), "limits": null, + "about": "a wai.video.int_motion base whose header declares 1 GiB of output, refused before anything is allocated for it"})), + ("clip-1gib-limits-raised", "cost_over_limit", json!({"base": env("wai-rs/tests/vectors/staged_bounds/clip_1gib.wai"), + "limits": {"cost": [1u64 << 40, 1u64 << 40, 1u64 << 40, 1u64 << 40], "chain_budget": 1u64 << 40}, + "about": "the 1 GiB clip under a title that states limits of 2^40: a title lowers a limit and never raises one, so the tools' stated limit refuses it before anything is allocated"})), + ("inert-companion-primary", "missing_capability", json!({"base": env("wai-rs/tests/vectors/staged_bounds/inert_companion_primary.wai"), "limits": null, + "about": "a companion-only capability with an integer fallback: inert, the fallback never reached"})), + ("inert-refinement-primary", "missing_capability", json!({"base": env("wai-rs/tests/vectors/staged_bounds/inert_refinement_primary.wai"), "limits": null, + "about": "a refinement-only capability with an integer fallback: inert, the fallback never reached"})), + ("layer-over-limit", "cost-over-limit", json!({"base": {"case": "staged-conformance/cases/image_int_hyper_2l", "base": 0, "first_unit": 0, "capture_us": null}, + "layers": ["L01.wai"], "limits": {"cost": [1024, 1024, 1u64 << 28, 1u64 << 38], "chain_budget": null}, + "about": "the still title's up2 layer under a cost limit its base passes and its layer does not: the layer's declared output, 3072 bytes, passes 1024"})), + ] +} + +/// The layers a refusal title carries, copied from the fixture that writes them. +fn refusal_layers(c: &Corpus, name: &str) -> Result)>, String> { + match name { + "layer-over-limit" => { + let spec = specs().into_iter().find(|s| s.name == "still").expect("the still fixture"); + let files = build_fixture(c, &spec)?; + Ok(vec![("L01.wai".to_string(), files["L01.wai"].clone())]) + } + _ => Ok(vec![]), + } +} + +fn refusal_title(name: &str, code: &str, v: &Value) -> Value { + json!({ + "format": wai::stage_measure::title::FORMAT, + "title": name, + "about": v["about"], + "unit_us": 40000, + "bases": [v["base"]], + "layers": v.get("layers").cloned().unwrap_or(json!([])), + "source": null, + "references": [], + "expect": null, + "limits": v["limits"], + "refused": code, + }) +} + +fn gen_fixtures(c: &Corpus) -> Result<(), String> { + for spec in specs() { + let dir = c.dir.join("fixtures").join(spec.name); + if dir.exists() { + std::fs::remove_dir_all(&dir).map_err(|e| e.to_string())?; + } + std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?; + let files = build_fixture(c, &spec)?; + for (n, b) in &files { + std::fs::write(dir.join(n), b).map_err(|e| format!("{n}: {e}"))?; + } + println!("fixtures/{}: {} files", spec.name, files.len()); + } + for (name, code, v) in refusals() { + let dir = c.dir.join("refusals").join(name); + std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?; + std::fs::write(dir.join("title.json"), pretty(&refusal_title(name, code, &v))).map_err(|e| e.to_string())?; + for (f, b) in refusal_layers(c, name)? { + std::fs::write(dir.join(f), b).map_err(|e| e.to_string())?; + } + } + println!("refusals: {} titles", refusals().len()); + for (rel, b) in counting_files(c)? { + let p = c.dir.join(&rel); + std::fs::create_dir_all(p.parent().expect("a directory")).map_err(|e| e.to_string())?; + std::fs::write(&p, b).map_err(|e| format!("{rel}: {e}"))?; + } + for (rel, b) in sim_inputs() { + let p = c.dir.join(&rel); + std::fs::create_dir_all(p.parent().expect("a directory")).map_err(|e| e.to_string())?; + std::fs::write(&p, b).map_err(|e| format!("{rel}: {e}"))?; + } + println!("sim: models, plans and cases"); + Ok(()) +} + +// ───────────────────────── the simulator corpus ───────────────────────── + +/// The synthetic delivery model: three groups of five units (a 25 Hz clip in groups of +/// five frames), each a base and two layers per unit, each group captured when its last +/// frame is. +fn gop3() -> Model { + let unit_us = 40_000; + let mut objects = Vec::new(); + for g in 0..3u32 { + let from = g as u64 * 5; + let cap = Some((from + 5) * unit_us); + objects.push(SimObject { id: format!("g{g}-base"), base: g, layer: 0, from, count: 5, bytes: 5000 + 700 * g as u64, capture_us: cap }); + for (k, size) in [(1u8, 1500u64), (2, 900)] { + for u in 0..5u64 { + objects.push(SimObject { id: format!("g{g}-l{k}-u{u}"), base: g, layer: k, from: from + u, count: 1, bytes: size + 37 * u + 11 * g as u64, capture_us: cap }); + } + } + } + Model { unit_us, objects } +} + +/// The plans: two bounded link schedules over both carriages, a counter-plan for each +/// unbounded part of §10.4's bound, a layer cap, and a live plan with playout +/// deadlines. +fn sim_plans() -> Vec<(&'static str, Plan)> { + let base = Plan { + mode: Mode::Vod, + rtt_us: 40_000, + bytes_per_ms: 2500, + chunk_bytes: 1200, + schedule: Schedule::StrictPriority, + carriage: Carriage::Subgroup, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: None, + vod_window_us: None, + workers: 1, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: 4000, + decode_refinement_us: 2000, + playout_start_us: None, + late_refinement: LateRefinement::Discard, + }; + // A slow link, on which refinements miss their delivery timeouts. + let slow = Plan { bytes_per_ms: 400, timeout_refinement_us: Some(30_000), ..base.clone() }; + vec![ + ("strict-subgroup", base.clone()), + ("strict-slow-timeout", slow.clone()), + ("fifo-timeout-subgroup", Plan { schedule: Schedule::Fifo, ..slow.clone() }), + // A window staggers each group's per-unit layers: a reset takes a subgroup's + // tail, whose own deadlines have not passed. + ("fifo-timeout-window", Plan { schedule: Schedule::Fifo, vod_window_us: Some(100_000), ..slow.clone() }), + ("fifo-timeout-datagram", Plan { schedule: Schedule::Fifo, carriage: Carriage::Datagram, chunk_bytes: 12_000, ..slow }), + ("fifo-no-timeout", Plan { schedule: Schedule::Fifo, ..base.clone() }), + // The same fifo schedule with a short refinement timeout: what isolates "no + // timeout" in the counter-plan above. + ("fifo-timeout-short", Plan { schedule: Schedule::Fifo, timeout_refinement_us: Some(1000), ..base.clone() }), + ("fifo-decode", Plan { vod_window_us: Some(180_000), decode_order: DecodeOrder::Fifo, ..base.clone() }), + // The same plan with base-first decode: what isolates the decode order in the + // counter-plan above. + ("strict-window", Plan { vod_window_us: Some(180_000), ..base.clone() }), + ("capped", Plan { layer_cap: Some(1), ..base.clone() }), + ("live-playout", Plan { mode: Mode::Live, timeout_refinement_us: Some(100_000), workers: 2, playout_start_us: Some(215_000), ..base.clone() }), + ("live-playout-present", Plan { mode: Mode::Live, workers: 2, playout_start_us: Some(215_000), late_refinement: LateRefinement::Present, ..base }), + ] +} + +/// Each case: a model (`gop3`, or a fixture's), a plan, and what it must show: +/// `within-bound`, `exceeds:` (its largest stall passes that plan's bound), or +/// `unbounded` (a plan with no bound, checked for its figures only). +fn sim_cases() -> Vec<(String, String, &'static str, String)> { + let mut out = Vec::new(); + for (plan, claim) in [ + ("strict-subgroup", "within-bound"), + ("strict-slow-timeout", "within-bound"), + ("fifo-timeout-subgroup", "within-bound"), + ("fifo-timeout-window", "within-bound"), + ("fifo-timeout-datagram", "within-bound"), + ("fifo-timeout-short", "within-bound"), + ("fifo-no-timeout", "exceeds:fifo-timeout-short"), + ("strict-window", "within-bound"), + ("fifo-decode", "exceeds:strict-window"), + ("capped", "within-bound"), + ("live-playout", "within-bound"), + ("live-playout-present", "within-bound"), + ] { + out.push((format!("gop3/{plan}"), "models/gop3.json".to_string(), plan, claim.to_string())); + } + for (name, ..) in reorder_cases() { + out.push((name.to_string(), format!("models/{name}.json"), name, "within-bound".to_string())); + } + for f in ["audio", "clip", "splat", "still"] { + for (plan, claim) in [("strict-subgroup", "within-bound"), ("fifo-timeout-subgroup", "within-bound"), ("fifo-timeout-datagram", "within-bound"), ("fifo-no-timeout", "unbounded")] { + out.push((format!("{f}/{plan}"), format!("fixture:{f}"), plan, claim.to_string())); + } + } + out +} + +/// Two models on which bases were once sent or decoded out of the order they became +/// available, and a refinement then stalled a base past §10.4's bound: a base captured +/// later but listed first under strict priority (`reorder-link`), and a small base whose +/// decode overtook a large one's under base-first decode (`reorder-decode`). Each comes +/// with its plan. +fn reorder_cases() -> Vec<(&'static str, Model, Plan)> { + let o = |id: &str, base, layer, from, bytes, cap| SimObject { id: id.into(), base, layer, from, count: 1, bytes, capture_us: cap }; + let link = Plan { + mode: Mode::Live, + rtt_us: 0, + bytes_per_ms: 1000, + chunk_bytes: 1000, + schedule: Schedule::StrictPriority, + carriage: Carriage::Subgroup, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: None, + vod_window_us: None, + workers: 1, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: 100, + decode_refinement_us: 100, + playout_start_us: None, + late_refinement: LateRefinement::Present, + }; + vec![ + ("reorder-link", Model { unit_us: 40_000, objects: vec![o("B0", 0, 0, 0, 10, Some(0)), o("R0", 0, 1, 0, 1000, Some(0)), o("Bb", 2, 0, 2, 10, Some(15)), o("Ba", 1, 0, 1, 50_000, Some(20))] }, link.clone()), + ("reorder-decode", Model { unit_us: 40_000, objects: vec![o("B0", 0, 0, 0, 10, None), o("R0", 0, 1, 0, 10, None), o("Bb", 2, 0, 2, 100_000, None), o("Ba", 1, 0, 1, 10, None)] }, + Plan { mode: Mode::Vod, schedule: Schedule::Fifo, timeout_refinement_us: Some(20), decode_base_us: 100_000, ..link }), + ] +} + +/// The simulator corpus's input files, by path below the corpus. +fn sim_inputs() -> BTreeMap> { + let mut files = BTreeMap::new(); + files.insert("sim/models/gop3.json".to_string(), pretty(&gop3().to_json())); + for (name, m, p) in reorder_cases() { + files.insert(format!("sim/models/{name}.json"), pretty(&m.to_json())); + files.insert(format!("sim/plans/{name}.json"), pretty(&p.to_json())); + } + for (name, p) in sim_plans() { + files.insert(format!("sim/plans/{name}.json"), pretty(&p.to_json())); + } + let cases: Vec = sim_cases().into_iter().map(|(name, model, plan, claim)| json!({"name": name, "model": model, "plan": format!("plans/{plan}.json"), "claim": claim})).collect(); + files.insert("sim/cases.json".to_string(), pretty(&json!({ + "format": "wai-sim-cases/1", + "about": "Delivery simulator cases (extensions/staged-measurement.md §10, Appendix B): a model (a file, or fixture: for a measured title's objects and counted bytes), a plan, and what the plan must show: within-bound, exceeds: (its largest stall passes that plan's bound), or unbounded.", + "cases": cases, + }))); + files +} + +/// The model a case names. +fn case_model(c: &Corpus, model: &str, evals: &[(Title, Evaluation)]) -> Result { + match model.strip_prefix("fixture:") { + Some(f) => evals.iter().find(|(t, _)| t.name == f).map(|(_, e)| Model::from_evaluation(e)).ok_or(format!("no fixture {f}")), + None => { + let v: Value = serde_json::from_slice(&std::fs::read(c.dir.join("sim").join(model)).map_err(|e| format!("{model}: {e}"))?).map_err(|e| e.to_string())?; + Model::from_json(&v).map_err(|e| format!("{model}: {e}")) + } + } +} + +fn read_plan(c: &Corpus, rel: &str) -> Result { + let v: Value = serde_json::from_slice(&std::fs::read(c.dir.join("sim").join(rel)).map_err(|e| format!("{rel}: {e}"))?).map_err(|e| e.to_string())?; + Plan::from_json(&v).map_err(|e| format!("{rel}: {e}")) +} + +/// Every case simulated, and each claim checked: `sim/results.json`. +fn sim_results(c: &Corpus, evals: &[(Title, Evaluation)]) -> Result { + let cases: Value = serde_json::from_slice(&std::fs::read(c.dir.join("sim/cases.json")).map_err(|e| format!("sim/cases.json: {e}"))?).map_err(|e| e.to_string())?; + let mut out = Vec::new(); + for k in cases["cases"].as_array().ok_or("sim/cases.json: no cases")? { + let name = k["name"].as_str().ok_or("a case name")?; + let m = case_model(c, k["model"].as_str().ok_or("a model")?, evals)?; + let plan_rel = k["plan"].as_str().ok_or("a plan")?; + let p = read_plan(c, plan_rel)?; + let r = sim::simulate(&m, &p).map_err(|e| format!("{name}: {e}"))?; + if r.offered != r.delivered + r.dropped + r.abandoned { + return Err(format!("{name}: bytes are not conserved")); + } + let bound = sim::stall_bound(&p); + let worst = r.max_stall(); + let claim = k["claim"].as_str().ok_or("a claim")?; + match claim.split_once(':') { + None if claim == "within-bound" => { + let b = bound.ok_or(format!("{name}: within-bound, but the plan has no bound"))?; + if worst.is_none_or(|w| w > b as i64) { + return Err(format!("{name}: the stall {worst:?} passes the bound {b}")); + } + } + None if claim == "unbounded" => { + if bound.is_some() { + return Err(format!("{name}: unbounded, but the plan has a bound")); + } + } + Some(("exceeds", other)) => { + let b = sim::stall_bound(&read_plan(c, &format!("plans/{other}.json"))?).ok_or(format!("{name}: {other} has no bound"))?; + if bound.is_some() || worst.is_some_and(|w| w <= b as i64) { + return Err(format!("{name}: the stall {worst:?} does not pass {other}'s bound {b}")); + } + } + _ => return Err(format!("{name}: claim `{claim}`")), + } + out.push(json!({"name": name, "stall_bound_us": bound, "max_stall_us": worst, "result": r.to_json()})); + } + Ok(json!({ + "format": "wai-sim-results/1", + "about": "Every case of sim/cases.json simulated (extensions/staged-measurement.md Appendix B), written by wai_stage_measure generate-goldens and required by wai_stage_measure verify and sim_verify.py. Simulated, not measured.", + "results": out, + })) +} + +// ───────────────────────── goldens and verification ───────────────────────── + +/// The picture `quality.json` draws its cases from: byte `c` of pixel `(x, y)` is +/// `(x·mul + y·(mul + 3) + 29·c + 17·seed + 5·((x·y + seed) mod 13)) mod 256`. +fn picture(w: usize, h: usize, seed: usize, mul: usize) -> Vec { + let mut out = Vec::with_capacity(w * h * 3); + for y in 0..h { + for x in 0..w { + for c in 0..3 { + out.push(((x * mul + y * (mul + 3) + c * 29 + seed * 17 + ((x * y + seed) % 13) * 5) % 256) as u8); + } + } + } + out +} + +/// `quality.json`: §4.2's integers for picture pairs whose sizes place the windows +/// every way the rule does: narrower or shorter than a window, offsets that reach the +/// edge at a multiple of 4, and an edge-aligned last window on one axis or both. +fn quality_cases() -> Value { + // (w, h, seed a, mul a, seed b, mul b) + let sizes: [(usize, usize, usize, usize, usize, usize); 14] = [ + (1, 1, 1, 5, 2, 7), + (5, 5, 3, 11, 3, 13), + (3, 100, 1, 7, 2, 7), + (8, 8, 2, 9, 4, 9), + (9, 9, 5, 3, 5, 4), + (11, 11, 1, 5, 2, 7), + (13, 5, 6, 2, 7, 2), + (12, 20, 4, 9, 5, 9), + (16, 16, 1, 5, 1, 6), + (17, 9, 2, 3, 5, 3), + (9, 33, 8, 13, 9, 14), + (19, 23, 3, 1, 3, 2), + (40, 24, 3, 7, 3, 8), + (41, 27, 9, 4, 1, 4), + ]; + let cases: Vec = sizes + .iter() + .map(|&(w, h, s1, m1, s2, m2)| { + let (a, b) = (picture(w, h, s1, m1), picture(w, h, s2, m2)); + let q = wai::quality::image(wai::quality::Rgb8 { w, h, px: &a }, wai::quality::Rgb8 { w, h, px: &b }).expect("two pictures of one size"); + json!({"w": w, "h": h, "reference": [s1, m1], "test": [s2, m2], "n_samples": q.n_samples, "sse_rgb": q.sse_rgb.to_string(), + "sse_y": q.sse_y.to_string(), "ssim_y_q32": q.ssim_y_q32, "windows": q.windows}) + }) + .collect(); + json!({ + "format": "wai-measure-quality/1", + "about": "Picture pairs and their §4.2 integers (extensions/staged-measurement.md), written by wai_stage_measure generate-goldens and required by wai_stage_measure verify and measure_verify.py. Byte c of pixel (x, y) of the picture [seed, mul] is (x*mul + y*(mul + 3) + 29*c + 17*seed + 5*((x*y + seed) mod 13)) mod 256.", + "cases": cases, + }) +} + +fn evaluate_all(c: &Corpus) -> Result<(Vec<(Title, Evaluation)>, Value), String> { + let mut out = Vec::new(); + for name in c.fixtures()? { + let t = Title::load(&c.wai, &c.fixture_rel(&name)).map_err(|e| format!("{name}: {e}"))?; + let e = evaluate(&t).map_err(|e| format!("{name}: {e}"))?; + check_expect(&t, &e).map_err(|e| format!("{name}: {e}"))?; + out.push((t, e)); + } + let pairs: Vec<(&Title, &Evaluation)> = out.iter().map(|(t, e)| (t, e)).collect(); + let rec = record_dedup(&pairs).map_err(|e| e.to_string())?; + let goldens = json!({ + "format": GOLDENS_FORMAT, + "about": "The deterministic figures of every fixture (extensions/staged-measurement.md §12.1), written by wai_stage_measure generate-goldens and required by wai_stage_measure verify and measure_verify.py.", + "fixtures": out.iter().map(|(_, e)| e.to_json()).collect::>(), + "record": {"dedup": record_dedup_json(&rec)}, + }); + Ok((out, goldens)) +} + +fn verify(c: &Corpus) -> Result<(), String> { + let mut fails = Vec::new(); + // Every fixture file is what the fixture tool writes, byte for byte. + let names = c.fixtures()?; + let want: Vec<&str> = specs().iter().map(|s| s.name).collect::>(); + let mut sorted = want.clone(); + sorted.sort(); + if names != sorted { + fails.push(format!("the fixtures are {names:?}; the fixture tool writes {sorted:?}")); + } + for spec in specs() { + let dir = c.dir.join("fixtures").join(spec.name); + match build_fixture(c, &spec) { + Err(e) => fails.push(format!("{}: {e}", spec.name)), + Ok(files) => { + let mut on_disk: Vec = std::fs::read_dir(&dir).map(|d| d.filter_map(|e| e.ok()).filter_map(|e| e.file_name().to_str().map(str::to_string)).collect()).unwrap_or_default(); + on_disk.sort(); + let mut written: Vec = files.keys().cloned().collect(); + written.sort(); + if on_disk != written { + fails.push(format!("{}: holds {on_disk:?}, the fixture tool writes {written:?}", spec.name)); + } + for (n, b) in &files { + if std::fs::read(dir.join(n)).ok().as_ref() != Some(b) { + fails.push(format!("{}/{n}: not what the fixture tool writes", spec.name)); + } + } + } + } + } + for (rel, b) in sim_inputs() { + if std::fs::read(c.dir.join(&rel)).ok().as_ref() != Some(&b) { + fails.push(format!("{rel}: not what the fixture tool writes")); + } + } + // The counting cases: every object and counting.json are the tool's. + match counting_files(c) { + Err(e) => fails.push(format!("counting: {e}")), + Ok(files) => { + for (rel, b) in files { + if std::fs::read(c.dir.join(&rel)).ok().as_ref() != Some(&b) { + fails.push(format!("{rel}: not what the fixture tool writes")); + } + } + } + } + // Every refusal title is the tool's, and is refused with its code. + let mut on_disk: Vec = std::fs::read_dir(c.dir.join("refusals")).map(|d| d.filter_map(|e| e.ok()).filter_map(|e| e.file_name().to_str().map(str::to_string)).collect()).unwrap_or_default(); + on_disk.sort(); + let mut want: Vec = refusals().iter().map(|r| r.0.to_string()).collect(); + want.sort(); + if on_disk != want { + fails.push(format!("refusals: holds {on_disk:?}, the fixture tool writes {want:?}")); + } + for (name, code, v) in refusals() { + if std::fs::read(c.dir.join("refusals").join(name).join("title.json")).ok() != Some(pretty(&refusal_title(name, code, &v))) { + fails.push(format!("refusals/{name}/title.json: not what the fixture tool writes")); + } + match refusal_layers(c, name) { + Err(e) => fails.push(format!("refusals/{name}: {e}")), + Ok(ls) => { + for (f, b) in ls { + if std::fs::read(c.dir.join("refusals").join(name).join(&f)).ok() != Some(b) { + fails.push(format!("refusals/{name}/{f}: not what the fixture tool writes")); + } + } + } + } + match Title::load(&c.wai, &c.refusal_rel(name)).map_err(|e| e.to_string()).and_then(|t| check_refusal(&t).map_err(|e| e.to_string())) { + Ok(()) => println!(" refusals/{name}: refused {code}"), + Err(e) => fails.push(format!("refusals/{name}: {e}")), + } + } + if std::fs::read(c.dir.join("quality.json")).ok() != Some(pretty(&quality_cases())) { + fails.push("quality.json is not what generate-goldens writes".into()); + } + match evaluate_all(c) { + Err(e) => fails.push(e), + Ok((evals, goldens)) => { + match sim_results(c, &evals) { + Err(e) => fails.push(format!("sim: {e}")), + Ok(r) => { + if std::fs::read(c.dir.join("sim/results.json")).unwrap_or_default() != pretty(&r) { + fails.push("sim/results.json is not what generate-goldens writes".into()); + } + println!(" sim: {} cases, every claim holds", r["results"].as_array().map_or(0, Vec::len)); + } + } + let committed = std::fs::read(c.dir.join("goldens.json")).unwrap_or_default(); + if committed != pretty(&goldens) { + let theirs: Value = serde_json::from_slice(&committed).unwrap_or(Value::Null); + for (i, (t, e)) in evals.iter().enumerate() { + if theirs["fixtures"][i] != e.to_json() { + fails.push(format!("{}: the evaluation differs from goldens.json", t.name)); + } + } + fails.push("goldens.json is not what generate-goldens writes".into()); + } + for (t, e) in &evals { + println!(" {}: {} objects, {} presentations, root {}", t.name, e.objects.len(), e.presentations.len(), e.output.map(|c| hex(&c.root)).unwrap_or_default()); + } + } + } + if fails.is_empty() { + println!("staged-measure: every fixture, evaluation and golden verifies"); + Ok(()) + } else { + for f in &fails { + eprintln!("FAIL {f}"); + } + Err(format!("{} failures", fails.len())) + } +} + +// ───────────────────────── eval ───────────────────────── + +fn quality_line(q: &Option) -> String { + match q { + None => "-".into(), + Some(QualityKind::Rgb8(q)) => match wai::quality::psnr_db_display(q.sse_rgb, q.n_samples) { + Some(db) => format!("sse {} (PSNR {db:.2} dB), SSIM {:.4}", q.sse_rgb, wai::quality::ssim_display(q.ssim_y_q32)), + None => "exact".into(), + }, + Some(QualityKind::Pcm16(q)) => match wai::quality::snr_db_display(q.ref_energy, q.sse) { + Some(db) => format!("sse {} (SNR {db:.2} dB)", q.sse), + None if q.sse == 0 => "exact".into(), + None => format!("sse {}", q.sse), + }, + Some(QualityKind::Tensor(q)) => if q.sse == 0 { "exact".into() } else { format!("sse {}, max |d| {}", q.sse, q.max_abs) }, + } +} + +/// A fixture directory's title, evaluated under the tool's limits (`limit` where given, +/// otherwise the stated defaults), which the title may lower and never raise, and +/// checked against its expectation. +fn load_eval_with(dir: &str, limit: Option) -> Result<(Title, Evaluation), String> { + let path = std::fs::canonicalize(dir).map_err(|e| format!("{dir}: {e}"))?; + // The fixture's path below wai/: wai/ is the directory that holds the corpus. + let fixtures = path.parent().ok_or("no parent")?; + let corpus = fixtures.parent().ok_or("no corpus")?; + let wai_root = corpus.parent().ok_or("no wai/")?; + let rel = path.strip_prefix(wai_root).map_err(|e| e.to_string())?.to_str().ok_or("path")?.replace('\\', "/"); + let mut t = Title::load(wai_root, &rel).map_err(|e: MeasureError| e.to_string())?; + if let Some(l) = limit { + t.under(&wai::stage_measure::title::Limits { cost: l, chain_budget: wai::staged::TOOL_DEFAULT_CHAIN_BUDGET }); + } + let e = evaluate(&t).map_err(|e| e.to_string())?; + check_expect(&t, &e).map_err(|e| e.to_string())?; + Ok((t, e)) +} + +fn load_eval(dir: &str) -> Result<(Title, Evaluation), String> { + load_eval_with(dir, None) +} + +/// Simulate one plan over a model file or a fixture's title. +fn sim_one(model: &str, plan: &str) -> Result<(), String> { + let m = if std::path::Path::new(model).is_dir() { + Model::from_evaluation(&load_eval(model)?.1) + } else { + let v: Value = serde_json::from_slice(&std::fs::read(model).map_err(|e| format!("{model}: {e}"))?).map_err(|e| e.to_string())?; + Model::from_json(&v).map_err(|e| e.to_string())? + }; + let v: Value = serde_json::from_slice(&std::fs::read(plan).map_err(|e| format!("{plan}: {e}"))?).map_err(|e| e.to_string())?; + let p = Plan::from_json(&v).map_err(|e| e.to_string())?; + let r = sim::simulate(&m, &p).map_err(|e| e.to_string())?; + let out = json!({"label": "simulated", "plan": p.to_json(), "stall_bound_us": sim::stall_bound(&p), "max_stall_us": r.max_stall(), "result": r.to_json()}); + print!("{}", String::from_utf8(pretty(&out)).expect("UTF-8")); + Ok(()) +} + +fn eval(dir: &str, as_json: bool, limit: Option) -> Result<(), String> { + let (_, e) = load_eval_with(dir, limit)?; + if as_json { + print!("{}", String::from_utf8(pretty(&e.to_json())).expect("UTF-8")); + return Ok(()); + } + println!("title {} ({}), framing {}, content {}", e.title, e.class.token(), e.framing, hex(&e.content_id)); + println!("bfpf_min {} bytes", e.bfpf_min.map(|b| b.to_string()).unwrap_or("-".into())); + println!("layers stage bytes prefix bytes units against the source / against the final prefix"); + for p in &e.prefixes { + println!("{:>6} {:>11} {:>12} {:>5} {} / {}", p.layers, p.stage_counted, p.counted, p.units, quality_line(&p.vs_source), quality_line(&p.vs_final)); + } + println!("layering overhead (counted bytes, quality against the source):"); + for o in &e.overhead { + println!(" {:<16} {:>9} {}", o.label, o.counted, quality_line(&o.vs_source)); + } + let d = &e.dedup; + println!("dedup: {} objects, {} unique, {} referenced bytes, {} stored bytes", d.objects, d.unique, d.referenced_bytes, d.stored_bytes); + println!("output root {} over {} leaves", e.output.map(|c| hex(&c.root)).unwrap_or("-".into()), e.output.map_or(0, |c| c.leaf_count)); + Ok(()) +} + +fn main() -> ExitCode { + let args: Vec = std::env::args().skip(1).collect(); + let r = match args.iter().map(String::as_str).collect::>().as_slice() { + ["eval", dir, rest @ ..] => (|| { + let mut json = false; + let mut limit = None; + let mut it = rest.iter(); + while let Some(a) = it.next() { + match *a { + "--json" => json = true, + "--limit" => limit = Some(wai::staged::parse_limit(it.next().ok_or("--limit OUT,SYMBOLS,WORKING,MACS")?)?), + other => return Err(format!("eval: unknown argument {other}")), + } + } + eval(dir, json, limit) + })(), + ["verify", corpus] => Corpus::new(corpus).and_then(|c| verify(&c)), + ["generate-goldens", corpus] => Corpus::new(corpus).and_then(|c| { + let (evals, g) = evaluate_all(&c)?; + std::fs::write(c.dir.join("goldens.json"), pretty(&g)).map_err(|e| e.to_string())?; + std::fs::write(c.dir.join("sim/results.json"), pretty(&sim_results(&c, &evals)?)).map_err(|e| e.to_string())?; + std::fs::write(c.dir.join("quality.json"), pretty(&quality_cases())).map_err(|e| e.to_string())?; + println!("goldens.json, quality.json and sim/results.json written"); + Ok(()) + }), + ["sim", model, plan] => sim_one(model, plan), + ["gen-fixtures", corpus] => Corpus::new(corpus).and_then(|c| gen_fixtures(&c)), + _ => { + eprintln!("usage: wai_stage_measure eval [--json] [--limit OUT,SYMBOLS,WORKING,MACS] | sim | verify | generate-goldens | gen-fixtures "); + return ExitCode::from(2); + } + }; + match r { + Ok(()) => ExitCode::SUCCESS, + Err(e) => { + eprintln!("error: {e}"); + ExitCode::from(1) + } + } +} diff --git a/wai/wai-rs/src/bin/wai_staged_vectors.rs b/wai/wai-rs/src/bin/wai_staged_vectors.rs index 0d18cf8a4..86ff98e2c 100644 --- a/wai/wai-rs/src/bin/wai_staged_vectors.rs +++ b/wai/wai-rs/src/bin/wai_staged_vectors.rs @@ -1850,15 +1850,24 @@ fn verify(dir: &Path) -> Result<(), Vec> { .iter() .map(|(m, p)| (m.clone(), p.as_str().unwrap().to_string())) .collect(); - let w = Wai::from_bytes(&env).expect("a base envelope"); - let pin = w.prior().cloned(); - let decoded = decode_with(&ctx, &via, &w.payload, &set, pin.as_ref()); - if decoded != units { - fail(format!("{rel}: the base's decode is not stage0")); - } - let t = tier::base_tier(&via, if b["pin_resolved"] == true { pin.as_ref() } else { None }); - if tier::token(t) != b["tier"] { - fail(format!("{rel}: the base tier is {}, the case states {}", tier::token(t), b["tier"])); + // The sink's own base decode, through the capability the case names: + // its units, its tier, whether it went through the fallback and + // whether its pin resolved, each read from the decode. + let bytes: Vec<(String, Vec)> = set.iter().map(|(m, p)| (m.clone(), ctx.read(p))).collect(); + let files: BTreeMap = bytes.iter().map(|(m, b)| (m.clone(), b.as_slice())).collect(); + match wai::staged::decode_base_pinned(&env, Some(&via), &files, None, None) { + Err(e) => fail(format!("{rel}: the base does not decode: {e}")), + Ok(d) => { + if d.units != units { + fail(format!("{rel}: the base's decode is not stage0")); + } + if tier::token(d.info.tier) != b["tier"] { + fail(format!("{rel}: the base tier is {}, the case states {}", tier::token(d.info.tier), b["tier"])); + } + if d.info.via_fallback != (b["via_fallback"] == true) || d.pin.is_some() != (b["pin_resolved"] == true) { + fail(format!("{rel}: the decode's fallback and pin are not what the case states")); + } + } } } Some(k) => println!(" {rel}: the {k} base is a stage-0 fixture in this build (staged_verify.py decodes it)"), diff --git a/wai/wai-rs/src/bin/wai_telemetry.rs b/wai/wai-rs/src/bin/wai_telemetry.rs new file mode 100644 index 000000000..19f417637 --- /dev/null +++ b/wai/wai-rs/src/bin/wai_telemetry.rs @@ -0,0 +1,334 @@ +//! `wai_telemetry` — delivery telemetry (`extensions/delivery-telemetry.md`): an +//! observation's CMCD report, reading a report back, and the corpus +//! `telemetry-conformance/`. +//! +//! wai_telemetry encode +//! wai_telemetry decode +//! wai_telemetry verify +//! wai_telemetry gen +//! +//! `verify` reads every observation, CMSD description and session log of the corpus, +//! requires each observation to read back to its own bytes, recomputes every expected +//! file and requires it byte for byte, and requires every observation under `invalid/` +//! to be refused. `gen` writes the corpus; it never runs in CI. + +use std::collections::BTreeMap; +use std::path::Path; +use std::process::ExitCode; + +use serde_json::{json, Value}; +use wai::telemetry::cmcd::{cmcd, cmcd_headers, from_cmcd, Report}; +use wai::telemetry::cmsd::{encode_dynamic, CmsdDynamic, CmsdStatic}; +use wai::telemetry::observation::Observation; +use wai::telemetry::qoe::session_metrics; +use wai::telemetry::sf::{SfBare, SfMember}; + +const SID: &str = "6f1c2a9e0b7d4c3a8e5f1d2c3b4a5968"; +const SID2: &str = "0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4"; +const CH: &str = "4fc1bd94bd48b2c1ded99c0053eed742f5399cec09dbd2f8694cca6a30e44875"; +const BASE: &str = "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f"; +const RCPT: &str = "7bf0f413db62aa185969b9ccf925d40f0281601268899bb7f2e97344487f4189"; +const CLAIM: &str = "e6e42e268f003a39dd887e6bdaf1501a270bcc63c29d5c40bbeedcb0b29d4277"; +const T0: u64 = 1_790_000_000_000; + +fn pretty(v: &Value) -> Vec { + let mut s = serde_json::to_string_pretty(v).expect("JSON"); + s.push('\n'); + s.into_bytes() +} + +fn obs(sid: &str, sn: u64, ts: Option, event: &str) -> Value { + json!({ + "v": 1, "sid": sid, "sn": sn, "ts_ms": ts, "event": event, "cid": BASE, "st": "v", "sf": "o", + "object": null, "present": null, "switch": null, "layer_change": null, "capable": null, "cohort": null, + "metrics": null, "claim": null, + }) +} + +fn object(members: Value) -> Value { + let mut o = json!({ + "ot": null, "url": null, "content_hash": null, "capability": null, "rendition": null, "layer": null, + "layers_total": null, "layer_cap": null, "counted": null, "transferred": null, "d_ms": null, "dl_ms": null, + "ttfb_ms": null, "ttlb_ms": null, "rc": null, "path": null, "receipt": null, + }); + for (k, x) in members.as_object().expect("members") { + o[k] = x.clone(); + } + o +} + +fn with(mut v: Value, k: &str, x: Value) -> Value { + v[k] = x; + v +} + +fn present(unit: u64, layers: u64, first: bool, late: Option, dec: Option) -> Value { + json!({"unit": unit, "layers": layers, "first": first, "late_ms": late, "decode_us": dec}) +} + +/// The observations: every event kind, and the members each maps. +fn observations() -> Vec<(&'static str, Value)> { + let req = |sn, m: Value| with(obs(SID, sn, None, "request"), "object", object(m)); + vec![ + ("01-request-base", with(req(0, json!({"ot": "v", "content_hash": BASE, "capability": "wai.neural.int_hyper", "layer": 0, "layers_total": 4, "counted": 460, "d_ms": 40, "dl_ms": 200, "path": "primary"})), "cohort", json!("wai"))), + ("02-request-layer", req(1, json!({"ot": "v", "content_hash": CH, "capability": "wai.image.int_refine", "layer": 1, "layers_total": 4, "layer_cap": 2, "counted": 951, "transferred": 951, "dl_ms": 400, "path": "primary", "receipt": RCPT}))), + ("03-request-rendition", with(req(2, json!({"ot": "o", "rendition": 1, "counted": 2519, "path": "fallback"})), "cid", Value::Null)), + ("04-response", with(obs(SID, 3, Some(T0 + 120), "response"), "object", object(json!({"url": "https://media.example/t/base.wai", "content_hash": BASE, "counted": 460, "transferred": 460, "ttfb_ms": 38, "ttlb_ms": 41, "rc": 200, "path": "primary"})))), + ("05-start", obs(SID, 4, Some(T0), "start")), + ("06-present-first", with(with(obs(SID, 5, Some(T0 + 160), "present"), "present", present(0, 1, true, None, Some(2900))), "metrics", json!({"msd_ms": 160, "bsa": null, "bsda_ms": null, "dfa": null}))), + ("07-present-later", with(obs(SID, 6, Some(T0 + 210), "present"), "present", present(0, 2, false, Some(10), Some(1400)))), + ("08-stall-start", with(obs(SID, 7, Some(T0 + 4000), "stall-start"), "metrics", json!({"msd_ms": null, "bsa": 1, "bsda_ms": 0, "dfa": null}))), + ("09-stall-end", with(obs(SID, 8, Some(T0 + 4350), "stall-end"), "metrics", json!({"msd_ms": null, "bsa": 1, "bsda_ms": 350, "dfa": null}))), + ("10-switch", with(obs(SID, 9, Some(T0 + 5000), "switch"), "switch", json!({"unit": 125, "from": "primary", "to": "fallback", "reason": "prior-not-held"}))), + ("11-layer-change", with(obs(SID, 10, Some(T0 + 6000), "layer-change"), "layer_change", json!({"from": null, "to": 1}))), + ("12-frame-drop", with(obs(SID, 11, Some(T0 + 6500), "frame-drop"), "metrics", json!({"msd_ms": null, "bsa": null, "bsda_ms": null, "dfa": 3}))), + ("13-capable-pin-refused", with(obs(SID2, 0, Some(T0 + 10), "capable"), "capable", json!({"outcome": "pin-refused", "reason": "part-mismatch", "classical_floor": "supported"}))), + ("14-capable", with(with(obs(SID2, 1, Some(T0 + 11), "capable"), "capable", json!({"outcome": "capable", "reason": null, "classical_floor": null})), "cohort", json!("control"))), + ("15-abandon", obs(SID2, 2, Some(T0 + 900), "abandon")), + ("16-end", with(obs(SID, 12, Some(T0 + 9000), "end"), "claim", json!(CLAIM))), + ("17-live-low-latency", with(with(obs(SID, 13, Some(T0 + 9100), "start"), "st", json!("ll")), "sf", json!("h"))), + ("18-switch-prior-revoked", with(obs(SID, 14, Some(T0 + 9200), "switch"), "switch", json!({"unit": 126, "from": "primary", "to": "fallback", "reason": "prior-revoked"}))), + ("19-capable-not-authorised", with(obs(SID2, 3, Some(T0 + 12), "capable"), "capable", json!({"outcome": "pin-refused", "reason": "prior-not-authorised", "classical_floor": null}))), + // Strings an RFC 9651 String escapes: a quote and a backslash in a Request-mode + // custom key, and a quote in an Event-mode report's url (which holds no `\`). + ("20-request-escaped", req(15, json!({"ot": "v", "capability": "x.test.\"quoted\"\\cap", "counted": 12, "path": "primary"}))), + ("21-response-escaped", with(obs(SID, 16, Some(T0 + 9300), "response"), "object", object(json!({"url": "https://media.example/t/a\"b.wai", "counted": 12, "rc": 200, "path": "primary"})))), + ] +} + +/// Observations every reader refuses, with what is wrong. +fn invalid() -> Vec<(&'static str, Value)> { + let base = obs(SID, 0, Some(T0), "start"); + let mut missing = base.clone(); + missing.as_object_mut().expect("object").remove("cohort"); + vec![ + ("unknown-member", with(base.clone(), "extra", json!(1))), + ("missing-member", missing), + ("version-2", with(base.clone(), "v", json!(2))), + ("sid-not-hex", with(base.clone(), "sid", json!("not-a-session-id-not-a-session-i"))), + ("event-mode-without-time", with(base.clone(), "ts_ms", Value::Null)), + ("sn-past-2-53", with(base.clone(), "sn", json!(9_007_199_254_740_992u64))), + ("ts-past-sf-integer", with(base.clone(), "ts_ms", json!(1_000_000_000_000_000u64))), + ("claim-before-end", with(base.clone(), "claim", json!(CLAIM))), + ("unknown-event", with(base.clone(), "event", json!("pause"))), + ("response-key-on-request", with(obs(SID, 0, None, "request"), "object", object(json!({"rc": 200})))), + ("request-key-on-response", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"dl_ms": 200})))), + ("switch-to-same-path", with(obs(SID, 0, Some(T0), "switch"), "switch", json!({"unit": 1, "from": "primary", "to": "primary", "reason": "trap"}))), + ("switch-unknown-reason", with(obs(SID, 0, Some(T0), "switch"), "switch", json!({"unit": 1, "from": "primary", "to": "fallback", "reason": "slow"}))), + ("pin-refused-without-reason", with(obs(SID, 0, Some(T0), "capable"), "capable", json!({"outcome": "pin-refused", "reason": null, "classical_floor": null}))), + ("object-type-unknown", with(obs(SID, 0, None, "request"), "object", object(json!({"ot": "x"})))), + ("cid-past-64", with(base.clone(), "cid", json!("c".repeat(65)))), + ("cid-an-address", with(base.clone(), "cid", json!("viewer@example.org"))), + ("cid-a-url", with(base.clone(), "cid", json!("https://media.example/t"))), + ("url-with-query", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https://media.example/t/base.wai?token=abc", "rc": 200})))), + ("url-with-fragment", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https://media.example/t/base.wai#t=1", "rc": 200})))), + ("url-with-userinfo", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https://viewer:secret@media.example/t/base.wai", "rc": 200})))), + ("url-userinfo-behind-backslashes", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https:\\\\viewer:secret@media.example/t/base.wai", "rc": 200})))), + ("url-userinfo-without-slashes", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https:viewer:secret@media.example/t/base.wai", "rc": 200})))), + ("url-userinfo-behind-a-mixed-separator", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https:/\\viewer:secret@media.example/t/base.wai", "rc": 200})))), + ("url-with-a-backslash", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https://media.example\\t/base.wai", "rc": 200})))), + ("url-with-an-at-in-its-path", with(obs(SID, 0, Some(T0), "response"), "object", object(json!({"url": "https://media.example/u/viewer@example.org/base.wai", "rc": 200})))), + ("present-zero-layers", with(obs(SID, 0, Some(T0), "present"), "present", present(0, 0, true, None, None))), + ("cohort-unknown", with(base, "cohort", json!("treatment"))), + ] +} + +fn cmsd_cases() -> Vec<(&'static str, Value)> { + vec![ + ("static-base", json!({"at": T0, "d": 40, "ht": null, "nor": "L01.wai", "nrr": null, "ot": "v", "sf": "o", "st": "v", "su": true, "content_hash": BASE, "layer": 0, "layers_total": 4, "receipt": RCPT, "op": null})), + ("static-derived", json!({"at": T0 + 5, "d": null, "ht": 20, "nor": null, "nrr": "0-1023", "ot": "o", "sf": "o", "st": "l", "su": false, "content_hash": CH, "layer": 1, "layers_total": 4, "receipt": null, "op": "crop"})), + ("dynamic-two-nodes", json!([ + {"server": "edge-7", "du": false, "etp": 25000, "mb": 8000, "rd": 3, "rtt": 40, "drop": 2}, + {"server": "origin-1", "du": true, "etp": null, "mb": null, "rd": 12, "rtt": null, "drop": null}, + ])), + ] +} + +/// Session logs: one session each. +fn logs() -> Vec<(&'static str, Vec)> { + let o = |sid, sn, t, e| obs(sid, sn, Some(T0 + t), e); + let resp = |sid, sn, t, counted: u64| with(o(sid, sn, t, "response"), "object", object(json!({"counted": counted, "rc": 200}))); + vec![ + ("session-stalls", vec![ + o(SID, 0, 0, "start"), + resp(SID, 1, 100, 460), + with(o(SID, 2, 160, "present"), "present", present(0, 1, true, None, None)), + resp(SID, 3, 180, 951), + with(o(SID, 4, 210, "present"), "present", present(0, 2, false, None, None)), + o(SID, 5, 4000, "stall-start"), + o(SID, 6, 4350, "stall-end"), + with(o(SID, 7, 4400, "present"), "present", present(1, 1, false, Some(30), None)), + o(SID, 8, 8000, "stall-start"), + with(o(SID, 9, 9000, "end"), "claim", json!(CLAIM)), + ]), + ("session-exit-before-start", vec![o(SID2, 0, 0, "start"), resp(SID2, 1, 300, 2519), o(SID2, 2, 900, "abandon")]), + ("session-start-failure", vec![ + o(SID2, 0, 0, "start"), + with(o(SID2, 1, 50, "capable"), "capable", json!({"outcome": "missing-capability", "reason": null, "classical_floor": "unsupported"})), + o(SID2, 2, 3000, "end"), + ]), + ] +} + +fn expected_cmcd(o: &Observation) -> Result { + Ok(match cmcd(o).map_err(|e| e.to_string())? { + Report::None => json!({"mode": "none", "report": null, "headers": null}), + Report::Event(r) => json!({"mode": "event", "report": r, "headers": null}), + Report::Request(r) => json!({"mode": "request", "report": r, "headers": cmcd_headers(o).map_err(|e| e.to_string())?}), + }) +} + +/// Every file of the corpus but its README and the differential's own files. +fn corpus() -> Result>, String> { + let mut files = BTreeMap::new(); + let mut cm = serde_json::Map::new(); + for (n, v) in observations() { + let o = Observation::from_json(&v).map_err(|e| format!("{n}: {e}"))?; + if o.to_json() != v { + return Err(format!("{n}: does not write back as given")); + } + files.insert(format!("observations/{n}.json"), pretty(&v)); + cm.insert(n.to_string(), expected_cmcd(&o)?); + } + files.insert("expected/cmcd.json".into(), pretty(&Value::Object(cm))); + let mut refused = Vec::new(); + for (n, v) in invalid() { + if Observation::from_json(&v).is_ok() { + return Err(format!("invalid/{n}: accepted")); + } + files.insert(format!("invalid/{n}.json"), pretty(&v)); + refused.push(n); + } + files.insert("expected/refused.json".into(), pretty(&json!(refused))); + let mut cs = serde_json::Map::new(); + for (n, v) in cmsd_cases() { + let s = if n.starts_with("static") { + CmsdStatic::from_json(&v).and_then(|s| s.encode()) + } else { + v.as_array() + .ok_or_else(|| wai::telemetry::TelemetryError::Invalid("a list".into())) + .and_then(|a| a.iter().map(CmsdDynamic::from_json).collect::, _>>()) + .and_then(|d| encode_dynamic(&d)) + } + .map_err(|e| format!("cmsd/{n}: {e}"))?; + files.insert(format!("cmsd/{n}.json"), pretty(&v)); + cs.insert(n.to_string(), json!(s)); + } + files.insert("expected/cmsd.json".into(), pretty(&Value::Object(cs))); + let mut q = serde_json::Map::new(); + for (n, log) in logs() { + let mut text = String::new(); + let mut os = Vec::new(); + for v in &log { + os.push(Observation::from_json(v).map_err(|e| format!("logs/{n}: {e}"))?); + text.push_str(&serde_json::to_string(v).expect("JSON")); + text.push('\n'); + } + files.insert(format!("logs/{n}.jsonl"), text.into_bytes()); + q.insert(n.to_string(), session_metrics(&os).map_err(|e| format!("logs/{n}: {e}"))?.to_json()); + } + files.insert("expected/qoe.json".into(), pretty(&Value::Object(q))); + Ok(files) +} + +fn write_corpus(dir: &Path) -> Result<(), String> { + for (rel, b) in corpus()? { + let p = dir.join(&rel); + std::fs::create_dir_all(p.parent().expect("a directory")).map_err(|e| e.to_string())?; + std::fs::write(&p, b).map_err(|e| format!("{rel}: {e}"))?; + } + println!("telemetry-conformance written"); + Ok(()) +} + +/// Read every input from disk (so a hand edit is caught), recompute the expected +/// files, and compare; and require the written corpus to be the generator's. +fn verify(dir: &Path) -> Result<(), String> { + let mut fails = Vec::new(); + let want = corpus()?; + for (rel, b) in &want { + if std::fs::read(dir.join(rel)).ok().as_ref() != Some(b) { + fails.push(format!("{rel}: not what the generator writes")); + } + } + for sub in ["observations", "invalid", "cmsd", "logs"] { + for e in std::fs::read_dir(dir.join(sub)).map_err(|e| format!("{sub}: {e}"))?.flatten() { + let rel = format!("{sub}/{}", e.file_name().to_string_lossy()); + if !want.contains_key(&rel) { + fails.push(format!("{rel}: not in the corpus the generator writes")); + } + } + } + // The expected reports, recomputed from the files on disk. + let expected: Value = serde_json::from_slice(&std::fs::read(dir.join("expected/cmcd.json")).map_err(|e| e.to_string())?).map_err(|e| e.to_string())?; + for (n, _) in observations() { + let v: Value = serde_json::from_slice(&std::fs::read(dir.join(format!("observations/{n}.json"))).map_err(|e| e.to_string())?).map_err(|e| e.to_string())?; + match Observation::from_json(&v) { + Err(e) => fails.push(format!("{n}: refused: {e}")), + Ok(o) => { + let got = expected_cmcd(&o)?; + if got != expected[n] { + fails.push(format!("{n}: the report is not expected/cmcd.json's")); + } + if let Some(r) = got["report"].as_str() { + if let Err(e) = from_cmcd(r) { + fails.push(format!("{n}: the report does not read back: {e}")); + } + } + } + } + } + for (n, _) in invalid() { + let v: Value = serde_json::from_slice(&std::fs::read(dir.join(format!("invalid/{n}.json"))).map_err(|e| e.to_string())?).map_err(|e| e.to_string())?; + if Observation::from_json(&v).is_ok() { + fails.push(format!("invalid/{n}: accepted")); + } + } + if fails.is_empty() { + println!("telemetry: {} observations, {} refusals, {} CMSD descriptions and {} session logs verify", observations().len(), invalid().len(), cmsd_cases().len(), logs().len()); + Ok(()) + } else { + for f in &fails { + eprintln!("FAIL {f}"); + } + Err(format!("{} failures", fails.len())) + } +} + +fn member_json(m: &SfMember) -> Value { + match m { + SfMember::Item(SfBare::Int(i)) => json!(i), + SfMember::Item(SfBare::Str(s)) => json!({"string": s}), + SfMember::Item(SfBare::Token(t)) => json!({"token": t}), + SfMember::Item(SfBare::Bool(b)) => json!(b), + SfMember::IntList(l) => json!(l), + } +} + +fn main() -> ExitCode { + let args: Vec = std::env::args().skip(1).collect(); + let r = match args.iter().map(String::as_str).collect::>().as_slice() { + ["encode", path] => std::fs::read(path) + .map_err(|e| format!("{path}: {e}")) + .and_then(|b| serde_json::from_slice::(&b).map_err(|e| e.to_string())) + .and_then(|v| Observation::from_json(&v).map_err(|e| e.to_string())) + .and_then(|o| expected_cmcd(&o)) + .map(|v| print!("{}", String::from_utf8(pretty(&v)).expect("UTF-8"))), + ["decode", dict] => from_cmcd(dict) + .map_err(|e| e.to_string()) + .map(|m| print!("{}", String::from_utf8(pretty(&Value::Object(m.iter().map(|(k, v)| (k.clone(), member_json(v))).collect()))).expect("UTF-8"))), + ["verify", dir] => verify(Path::new(dir)), + ["gen", dir] => write_corpus(Path::new(dir)), + _ => { + eprintln!("usage: wai_telemetry encode | decode | verify | gen "); + return ExitCode::from(2); + } + }; + match r { + Ok(()) => ExitCode::SUCCESS, + Err(e) => { + eprintln!("error: {e}"); + ExitCode::from(1) + } + } +} diff --git a/wai/wai-rs/src/content_dedup.rs b/wai/wai-rs/src/content_dedup.rs new file mode 100644 index 000000000..e5bf7e5cc --- /dev/null +++ b/wai/wai-rs/src/content_dedup.rs @@ -0,0 +1,73 @@ +//! Deduplication totals over a set of objects (`extensions/staged-measurement.md` §9). +//! +//! Every figure is recomputed from the objects themselves, never declared: how many +//! objects a set references, how many distinct contents they have, the bytes the set +//! references, and the bytes a store keeps when it holds one copy of each content. +//! The same function serves a staged title's objects and a package's segments. + +use std::collections::BTreeMap; + +/// The four totals of staged-measurement §9.1. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)] +pub struct DedupTotals { + /// Objects referenced, a repeated content counted each time. + pub objects: u64, + /// Distinct content hashes. + pub unique: u64, + /// The bytes of every object referenced, a repeated content counted each time. + pub referenced_bytes: u64, + /// The bytes of one object per distinct content hash. + pub stored_bytes: u64, +} + +/// The totals over `items`, each a content hash and the object's bytes. +/// +/// One content hash names one content, so it has one length. A hash that comes with +/// two lengths is a set that cannot be accounted: the first such hash is returned +/// with the length seen first and the conflicting one, and no totals are given. +/// A byte sum past `u64` is not a set of real objects either; it saturates, which +/// no set that fits in memory reaches. +pub fn totals_checked>(items: I) -> Result { + let mut seen: BTreeMap<[u8; 32], u64> = BTreeMap::new(); + let mut t = DedupTotals::default(); + for (hash, len) in items { + t.objects += 1; + t.referenced_bytes = t.referenced_bytes.saturating_add(len); + match seen.get(&hash) { + Some(&first) if first != len => return Err((hash, first, len)), + Some(_) => {} + None => { + seen.insert(hash, len); + t.unique += 1; + t.stored_bytes = t.stored_bytes.saturating_add(len); + } + } + } + Ok(t) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn h(b: u8) -> [u8; 32] { + [b; 32] + } + + #[test] + fn totals_of_a_set_with_a_repeat() { + let t = totals_checked([(h(1), 100), (h(2), 40), (h(1), 100), (h(3), 7)]).unwrap(); + assert_eq!(t, DedupTotals { objects: 4, unique: 3, referenced_bytes: 247, stored_bytes: 147 }); + } + + #[test] + fn empty_set_is_all_zero() { + assert_eq!(totals_checked(std::iter::empty()).unwrap(), DedupTotals::default()); + } + + #[test] + fn one_hash_with_two_lengths_is_refused() { + // The first hash seen twice with a different length is named, with both lengths. + assert_eq!(totals_checked([(h(1), 10), (h(2), 5), (h(1), 11), (h(2), 6)]), Err((h(1), 10, 11))); + } +} diff --git a/wai/wai-rs/src/lib.rs b/wai/wai-rs/src/lib.rs index db13734df..b52d96750 100644 --- a/wai/wai-rs/src/lib.rs +++ b/wai/wai-rs/src/lib.rs @@ -22,6 +22,17 @@ pub mod codecs; pub mod container; +// Deduplication totals (extensions/staged-measurement.md §9): objects, distinct +// contents, referenced and stored bytes, recomputed from the objects. Pure integer, +// no dependency, so every build (wasm32 included) has it. +pub mod content_dedup; +// Quality per prefix (extensions/staged-measurement.md §4): SSE, luma, a Q32 +// structural similarity, PCM and tensor sums, all in integers; decibels for display. +pub mod quality; +// The machine a measurement ran on (extensions/staged-measurement.md §7.4): CPU, +// cores, OS, arch, build, load and power source, as the OS reports them. Native only. +#[cfg(not(target_arch = "wasm32"))] +pub mod bench_env; // Parameter-set pins (SPEC §3.1): the listing form, and verifying a pin against // the sets a sink holds. Pure Rust over bytes, so it is always on. pub mod pin; @@ -321,6 +332,21 @@ mod container_explode; #[cfg(feature = "staged")] pub mod staged; +// Staged-delivery measurement (extensions/staged-measurement.md): the output tree +// (Appendix C), the counted bytes of an envelope (§3), and the evaluation of a staged +// title (§3–§6, §9) through the staged sink. +#[cfg(feature = "stage_measure")] +pub mod output_tree; +#[cfg(feature = "stage_measure")] +pub mod measure_framing; +#[cfg(feature = "stage_measure")] +pub mod stage_measure; + +// Delivery telemetry (extensions/delivery-telemetry.md): advisory, unsigned observations +// about delivery, as CMCD version 2 and CMSD reports, and session metrics. +#[cfg(feature = "telemetry")] +pub mod telemetry; + // Typed receipt classes (extensions/jwp-receipts.md §2.4.3–§2.4.7, §2.6, §3.1–§3.2): // energy, lawful-basis, log-inclusion, soft-binding and binding-recovery statements // about delivered objects, on one `claim` envelope and one signature rule beside the diff --git a/wai/wai-rs/src/measure_framing.rs b/wai/wai-rs/src/measure_framing.rs new file mode 100644 index 000000000..2e9ce2bdd --- /dev/null +++ b/wai/wai-rs/src/measure_framing.rs @@ -0,0 +1,229 @@ +//! Counted bytes of an object (`extensions/staged-measurement.md` §3). +//! +//! The counted bytes of a `WAI1` envelope are `12 + man_len + P`, and of a `WAI2` +//! envelope `8 + man_len + 2 + 8·n + Σ Pᵢ` (SPEC §2, §7.1): the envelope exactly as +//! written, which a well-formed envelope is to the byte. An object is countable only +//! where WAI checks the format of every payload it carries (§3.2): a capability SPEC §5 +//! pins to a binary format (`WIV1`, `WIH1`, `WIA1`, `WIS1`, and `WIR1` for a +//! refinement-only capability), whose payload parses as that format by the parse a sink +//! runs (`container::check_int_payload_framing`), and, where the manifest names a +//! fallback with a pinned format, parses as the fallback's too (as +//! `Wai::check_int_payload_framing` checks a payload a fallback decodes). Every other +//! capability, a classical codestream or a string WAI does not register among them, is +//! not countable (`no-format-check`): nothing in WAI tells its bytes from a text +//! encoding of them. A `WAI2` component whose `sha256` does not match its payload is not +//! countable (`sha256-mismatch`), and neither is an encrypted payload (`encrypted`), +//! whose plaintext's format cannot be checked. +//! +//! A countable object's figure carries the framing label `registered` (§3.3). Nothing +//! here estimates a figure for an object that is not countable. + +use crate::container::{check_int_payload_framing, manifest_v2_security, Wai, WaiMulti, MAGIC, MAGIC_V2}; + +/// The framing label of a figure whose every payload is in its capability's registered +/// format (§3.3). +pub const REGISTERED: &str = "registered"; + +/// An object's counted bytes, split as §3.4 states them. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Counted { + /// The envelope's own structure: magic and length prefixes, and for `WAI2` the + /// rendition count and table. + pub header: u64, + /// The manifest's bytes as written. + pub manifest: u64, + /// The payloads' bytes. + pub payload: u64, + /// `header + manifest + payload`: the envelope's length. + pub total: u64, + /// The framing label (§3.3). + pub framing: &'static str, +} + +/// Why an object has no counted bytes. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum FramingError { + /// The bytes are not a `WAI1` or `WAI2` envelope. + Envelope(String), + /// Bytes follow the envelope's last payload: the object is not the envelope its + /// lengths describe. + Trailing { envelope: u64, object: u64 }, + /// A payload is not countable (§3.2): `component` is the `WAI2` entry's index, + /// `None` for a `WAI1`; `capability` the capability whose check refused it; `code` + /// `no-format-check`, `sha256-mismatch`, `encrypted`, or the format parse's code + /// (`bad_magic`, `format_capability_mismatch`, … integer-payloads §8, staged-delivery + /// A.6). + NotCountable { component: Option, capability: String, code: &'static str, detail: String }, +} + +impl FramingError { + /// A short token for the error: `not-an-envelope`, `trailing`, or the code of a + /// payload that is not countable. + pub fn code(&self) -> &'static str { + match self { + Self::Envelope(_) => "not-an-envelope", + Self::Trailing { .. } => "trailing", + Self::NotCountable { code, .. } => code, + } + } +} + +impl std::fmt::Display for FramingError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Envelope(e) => write!(f, "not an envelope: {e}"), + Self::Trailing { envelope, object } => write!(f, "{} bytes follow the envelope's {envelope}", object - envelope), + Self::NotCountable { component: Some(i), capability, code, detail } => write!(f, "component {i} ({capability}) is not countable: {code}: {detail}"), + Self::NotCountable { component: None, capability, code, detail } => write!(f, "the payload ({capability}) is not countable: {code}: {detail}"), + } + } +} + +impl std::error::Error for FramingError {} + +/// Whether WAI checks the format of `capability`'s payload: a pinned integer format, or +/// `WIR1` for a refinement-only capability. +pub fn has_format_check(capability: &str) -> bool { + crate::codecs::pinned_int_payload(capability).is_some() || crate::codecs::is_refinement_only(capability) +} + +/// Why `payload` is not countable under `capability` (its code and what failed), or +/// `None` when it is (§3.2). +pub fn not_countable(capability: &str, payload: &[u8]) -> Option<(&'static str, String)> { + if !has_format_check(capability) { + return Some(("no-format-check", "WAI checks no format for this capability's payload".into())); + } + check_int_payload_framing(capability, payload).err().map(|e| (e.code, e.detail.to_string())) +} + +fn refuse(component: Option, capability: &str, (code, detail): (&'static str, String)) -> FramingError { + FramingError::NotCountable { component, capability: capability.to_string(), code, detail } +} + +/// The counted bytes of one object (§3.1). +pub fn count_object(envelope: &[u8]) -> Result { + let len = envelope.len() as u64; + let magic = envelope.get(..4).ok_or_else(|| FramingError::Envelope("shorter than a magic".into()))?; + let encrypted = || ("encrypted", "an encrypted payload's format cannot be checked".to_string()); + let (header, manifest, payload) = if magic == MAGIC { + let w = Wai::from_bytes(envelope).map_err(|e| FramingError::Envelope(e.to_string()))?; + let man = u32::from_le_bytes(envelope[4..8].try_into().expect("checked by the parse")) as u64; + let cap = w.capability(); + if w.manifest.security.as_ref().is_some_and(|s| !s.is_plaintext()) { + return Err(refuse(None, cap, encrypted())); + } + if let Some(why) = not_countable(cap, &w.payload) { + return Err(refuse(None, cap, why)); + } + // The fallback decodes the same payload (SPEC §4 step 5), so a fallback with a + // pinned format must find it in that format too. + if let Some(fb) = w.fallback().filter(|fb| has_format_check(fb)) { + if let Some(why) = not_countable(fb, &w.payload) { + return Err(refuse(None, fb, why)); + } + } + (12, man, w.payload.len() as u64) + } else if magic == MAGIC_V2 { + let w = WaiMulti::from_bytes(envelope).map_err(|e| FramingError::Envelope(e.to_string()))?; + let man = u32::from_le_bytes(envelope[4..8].try_into().expect("checked by the parse")) as u64; + if manifest_v2_security(&w.manifest).map_err(FramingError::Envelope)?.is_some_and(|s| !s.is_plaintext()) { + return Err(refuse(None, "", encrypted())); + } + let mut payload = 0u64; + for (i, (meta, p)) in w.manifest.renditions.iter().zip(&w.renditions).enumerate() { + if meta.sha256.is_some() && !meta.payload_digest_matches(p) { + return Err(refuse(Some(i), &meta.capability, ("sha256-mismatch", "the entry's sha256 is not its payload's".into()))); + } + if let Some(why) = not_countable(&meta.capability, p) { + return Err(refuse(Some(i), &meta.capability, why)); + } + payload += p.len() as u64; + } + (8 + 2 + 8 * w.renditions.len() as u64, man, payload) + } else { + return Err(FramingError::Envelope("neither WAI1 nor WAI2".into())); + }; + let total = header + manifest + payload; + if total != len { + return Err(FramingError::Trailing { envelope: total, object: len }); + } + Ok(Counted { header, manifest, payload, total, framing: REGISTERED }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::container::{Conditioning, Manifest, ModelRequirement, Rendition, Security}; + + const WIH1: &[u8] = include_bytes!("../tests/vectors/int_hyper/tiny_bitstream.wih1"); + const HYPER: &str = "wai.neural.int_hyper"; + + fn wai1(cap: &str, fallback: Option<&str>, payload: Vec) -> Vec { + let mut mr = ModelRequirement::new(cap); + if let Some(f) = fallback { + mr = mr.with_fallback(f); + } + Wai::new(Manifest::new("image", "replicate", mr, Conditioning::new("k")), payload).to_bytes().unwrap() + } + + fn code(r: Result) -> &'static str { + r.err().map(|e| e.code()).unwrap_or("counted") + } + + #[test] + fn a_wai1_counts_its_bytes_split_three_ways() { + let e = wai1(HYPER, None, WIH1.to_vec()); + let c = count_object(&e).unwrap(); + let man = u32::from_le_bytes(e[4..8].try_into().unwrap()) as u64; + assert_eq!(c, Counted { header: 12, manifest: man, payload: WIH1.len() as u64, total: e.len() as u64, framing: REGISTERED }); + } + + #[test] + fn a_wai2_counts_its_table_and_checks_its_digests() { + let two = || { + WaiMulti::new("image", "replicate", serde_json::Value::Null, vec![ + Rendition::media(HYPER, "int_hyper", WIH1.to_vec()).with_payload_digest(), + Rendition::media(HYPER, "int_hyper", WIH1.to_vec()), + ]) + }; + let e = two().to_bytes().unwrap(); + let c = count_object(&e).unwrap(); + assert_eq!((c.header, c.payload, c.total), (8 + 2 + 16, 2 * WIH1.len() as u64, e.len() as u64)); + // A stated digest that is not the payload's. + let mut bad = two(); + bad.manifest.renditions[0].sha256 = Some("0".repeat(64)); + assert!(matches!(count_object(&bad.to_bytes().unwrap()), Err(FramingError::NotCountable { component: Some(0), code: "sha256-mismatch", .. }))); + } + + #[test] + fn only_a_checked_format_is_counted() { + // A capability WAI checks no format for, a classical codestream or a string WAI + // does not register, is not countable. + assert_eq!(code(count_object(&wai1("wai.image.png", None, b"\x89PNG\r\n\x1a\n".to_vec()))), "no-format-check"); + assert_eq!(code(count_object(&wai1("wai.image.unregistered", None, vec![1, 2, 3]))), "no-format-check"); + // Text forms of a pinned format, and bytes before it, fail its parse. + let b64: Vec = WIH1.iter().map(|b| b"ABCDEFGHIJKLMNOP"[(b & 15) as usize]).collect(); + for p in [b"{\"rans_b64\": \"AAAA\"}".to_vec(), b"\"V0lIMQE=\"".to_vec(), b64, [b"\xef\xbb\xbf".as_slice(), WIH1].concat()] { + assert_eq!(code(count_object(&wai1(HYPER, None, p))), "bad_magic"); + } + // The fallback decodes the same payload: a WIH1 payload under an int_motion + // fallback is not that fallback's format. + assert_eq!(code(count_object(&wai1(HYPER, Some("wai.video.int_motion"), WIH1.to_vec()))), "format_capability_mismatch"); + // A fallback WAI checks no format for is not checked. + assert_eq!(code(count_object(&wai1(HYPER, Some("wai.image.png"), WIH1.to_vec()))), "counted"); + } + + #[test] + fn trailing_truncated_and_encrypted_are_refused() { + let mut e = wai1(HYPER, None, WIH1.to_vec()); + e.push(0); + assert!(matches!(count_object(&e), Err(FramingError::Trailing { .. }))); + e.truncate(e.len() - 3); + assert!(matches!(count_object(&e), Err(FramingError::Envelope(_)))); + assert!(matches!(count_object(b"WAI"), Err(FramingError::Envelope(_)))); + let mut m = Manifest::new("image", "replicate", ModelRequirement::new(HYPER), Conditioning::new("k")); + m.security = Some(Security::new("aes-256-gcm", 1)); + let enc = Wai::new(m, WIH1.to_vec()).to_bytes().unwrap(); + assert_eq!(code(count_object(&enc)), "encrypted"); + } +} diff --git a/wai/wai-rs/src/output_tree.rs b/wai/wai-rs/src/output_tree.rs new file mode 100644 index 000000000..29116519f --- /dev/null +++ b/wai/wai-rs/src/output_tree.rs @@ -0,0 +1,283 @@ +//! The output tree (`extensions/staged-measurement.md` Appendix C): a commitment to +//! every presentation a sink made, one leaf per (unit, layers), so that a record or a +//! session claim commits to its outputs by the tree's root and its leaf count +//! ([`OutputCommitment`]) and any one output is provable by an inclusion path. +//! +//! ```text +//! out_digest = SHA-256(canonical output) SPEC §7 Digests: RGB8, s16le PCM, … +//! leaf = SHA-256("wai:out-leaf\x01" ‖ u64_be(unit) ‖ u16_be(layers) ‖ u16_be(decoder) ‖ out_digest) +//! node = SHA-256("wai:out-node\x01" ‖ left ‖ right) +//! ``` +//! +//! Leaves are ordered by (unit, layers), and a (unit, layers) given twice is refused. +//! The root over `n ≥ 1` leaves is RFC 6962's Merkle tree hash: one leaf is itself; +//! otherwise `node(MTH(D[0:k]), MTH(D[k:n]))` with `k` the largest power of two below +//! `n`. No leaf is ever duplicated, so two different leaf lists never give one root and +//! the leaf count is fixed by the tree. An inclusion proof is RFC 6962's audit path, +//! checked as RFC 9162 §2.1.3.2 checks one, against the leaf count the commitment +//! states: one audit path can verify against one root under two tree sizes, so a root +//! without its leaf count fixes neither a leaf's position nor the tree's size, and +//! [`verify_proof`] takes no size from the proof's holder. + +use crate::pin::sha256; + +/// The leaf domain. +pub const LEAF_DOMAIN: &[u8] = b"wai:out-leaf\x01"; +/// The node domain. +pub const NODE_DOMAIN: &[u8] = b"wai:out-node\x01"; + +/// One presentation: a unit, the layers it was presented at (the base alone is 1), +/// the index of the decoder entry that produced it, and its output's digest. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Presentation { + pub unit: u64, + pub layers: u16, + pub decoder: u16, + pub out_digest: [u8; 32], +} + +/// Why no root or proof was given. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TreeError { + /// The same (unit, layers) was presented twice. + Duplicate { unit: u64, layers: u16 }, + /// A proof was asked for a leaf the tree does not have. + IndexOutOfRange, +} + +impl std::fmt::Display for TreeError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Duplicate { unit, layers } => write!(f, "unit {unit} at {layers} layers is presented twice"), + Self::IndexOutOfRange => f.write_str("no leaf at that index"), + } + } +} + +impl std::error::Error for TreeError {} + +/// The SHA-256 of an output's canonical buffer. +pub fn out_digest(canonical: &[u8]) -> [u8; 32] { + sha256(canonical) +} + +/// A presentation's leaf hash. +pub fn leaf(p: &Presentation) -> [u8; 32] { + let mut b = Vec::with_capacity(LEAF_DOMAIN.len() + 8 + 2 + 2 + 32); + b.extend_from_slice(LEAF_DOMAIN); + b.extend_from_slice(&p.unit.to_be_bytes()); + b.extend_from_slice(&p.layers.to_be_bytes()); + b.extend_from_slice(&p.decoder.to_be_bytes()); + b.extend_from_slice(&p.out_digest); + sha256(&b) +} + +fn node(l: &[u8; 32], r: &[u8; 32]) -> [u8; 32] { + let mut b = Vec::with_capacity(NODE_DOMAIN.len() + 64); + b.extend_from_slice(NODE_DOMAIN); + b.extend_from_slice(l); + b.extend_from_slice(r); + sha256(&b) +} + +/// The largest power of two below `n` (`n ≥ 2`). +fn split(n: usize) -> usize { + let mut k = 1; + while k * 2 < n { + k *= 2; + } + k +} + +fn mth(d: &[[u8; 32]]) -> [u8; 32] { + if d.len() == 1 { + return d[0]; + } + let k = split(d.len()); + node(&mth(&d[..k]), &mth(&d[k..])) +} + +/// The presentations in leaf order, refusing a (unit, layers) given twice. +pub fn ordered(ps: &[Presentation]) -> Result, TreeError> { + let mut v = ps.to_vec(); + v.sort_by_key(|p| (p.unit, p.layers)); + if let Some(w) = v.windows(2).find(|w| (w[0].unit, w[0].layers) == (w[1].unit, w[1].layers)) { + return Err(TreeError::Duplicate { unit: w[0].unit, layers: w[0].layers }); + } + Ok(v) +} + +/// The root over `ps`, in any order; `None` for no presentation. A root alone is not a +/// commitment: [`commit`] gives it with its leaf count. +pub fn root(ps: &[Presentation]) -> Result, TreeError> { + let v = ordered(ps)?; + if v.is_empty() { + return Ok(None); + } + let leaves: Vec<[u8; 32]> = v.iter().map(leaf).collect(); + Ok(Some(mth(&leaves))) +} + +/// What a record or a claim commits to: the tree's root and its leaf count. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct OutputCommitment { + pub root: [u8; 32], + pub leaf_count: u64, +} + +/// The commitment to `ps`, in any order; `None` for no presentation. +pub fn commit(ps: &[Presentation]) -> Result, TreeError> { + Ok(root(ps)?.map(|root| OutputCommitment { root, leaf_count: ps.len() as u64 })) +} + +fn path(m: usize, d: &[[u8; 32]], out: &mut Vec<[u8; 32]>) { + if d.len() <= 1 { + return; + } + let k = split(d.len()); + if m < k { + path(m, &d[..k], out); + out.push(mth(&d[k..])); + } else { + path(m - k, &d[k..], out); + out.push(mth(&d[..k])); + } +} + +/// The audit path of the leaf at `index` in leaf order (RFC 6962 §2.1.1), from the leaf +/// up. +pub fn proof(ps: &[Presentation], index: usize) -> Result, TreeError> { + let v = ordered(ps)?; + if index >= v.len() { + return Err(TreeError::IndexOutOfRange); + } + let leaves: Vec<[u8; 32]> = v.iter().map(leaf).collect(); + let mut out = Vec::new(); + path(index, &leaves, &mut out); + Ok(out) +} + +/// Whether `path` proves the leaf hash `leaf` at `index` in the tree `c` commits to: its +/// root, under its leaf count (RFC 9162 §2.1.3.2). +pub fn verify_proof(leaf: [u8; 32], index: u64, path: &[[u8; 32]], c: &OutputCommitment) -> bool { + let (n, root) = (c.leaf_count, c.root); + if index >= n { + return false; + } + let (mut fnn, mut sn) = (index, n - 1); + let mut r = leaf; + for p in path { + if sn == 0 { + return false; + } + if fnn & 1 == 1 || fnn == sn { + r = node(p, &r); + while fnn & 1 == 0 && fnn != 0 { + fnn >>= 1; + sn >>= 1; + } + } else { + r = node(&r, p); + } + fnn >>= 1; + sn >>= 1; + } + sn == 0 && r == root +} + +#[cfg(test)] +mod tests { + use super::*; + + fn hex(b: &[u8]) -> String { + b.iter().map(|x| format!("{x:02x}")).collect() + } + + /// The leaves the independent reference builds: unit `i`, `1 + i mod 3` layers, + /// decoder `i mod 2`, the output digest SHA-256 of the one byte `i`. + fn ps(n: usize) -> Vec { + (0..n).map(|i| Presentation { unit: i as u64, layers: 1 + (i % 3) as u16, decoder: (i % 2) as u16, out_digest: sha256(&[i as u8]) }).collect() + } + + #[test] + fn rfc6962_shape() { + let want = [ + "327b97c1dd3a9f79f5e895d4e1ee9e3effbad78c8e10621446e596e07b29c509", + "fde25c8d3bb8ed348cb33af1afec1540030f1c893f98a7dfa7527e435081becd", + "e4e9974c8bb77c1e6fbe06db3104cfbc1e75eddd981d0cd9f020c6db7aa0d0d3", + "cdbccfb7b7c445d27226aa3d76982b7c852c59945f3ec337cdcfdbb9d3c29214", + "769801393e51285b485e0dc26425ceb7bb1e43a4f769c270720ee95244f0131b", + "0c593a450f2cca51bd29198e6c896139bf9e88d90b67aebc587047268cd1e300", + "8bf0aebf1ce0fdc1c4325c81a2c327ec24be8b63d567dc7d6592b6a83075aae0", + "a8a37cd9ebf83637962e84a13b929c9279e0fb79736fd60b692091ce43f86661", + "d73a02a24875919cec4c06e4bc7c7a20f5da9be2dccd1754820be5994ef93410", + ]; + for (n, w) in (1..=9).zip(want) { + assert_eq!(hex(&root(&ps(n)).unwrap().unwrap()), w, "n = {n}"); + } + // Leaf order is (unit, layers), whatever order the list comes in. + let mut rev = ps(7); + rev.reverse(); + assert_eq!(root(&rev).unwrap(), root(&ps(7)).unwrap()); + assert_eq!(root(&[]).unwrap(), None); + } + + #[test] + fn proof_roundtrip() { + for n in 1..=33usize { + let p = ps(n); + let c = commit(&p).unwrap().unwrap(); + assert_eq!(c, OutputCommitment { root: root(&p).unwrap().unwrap(), leaf_count: n as u64 }); + let v = ordered(&p).unwrap(); + for i in 0..n { + let path = proof(&p, i).unwrap(); + let l = leaf(&v[i]); + assert!(verify_proof(l, i as u64, &path, &c), "n = {n}, i = {i}"); + // A flipped bit anywhere or a wrong index fails. + let mut bad = l; + bad[0] ^= 1; + assert!(!verify_proof(bad, i as u64, &path, &c)); + if n > 1 { + assert!(!verify_proof(l, ((i + 1) % n) as u64, &path, &c), "n = {n}, i = {i}"); + let mut p2 = path.clone(); + p2[0][31] ^= 0x80; + assert!(!verify_proof(l, i as u64, &p2, &c)); + } + } + assert_eq!(proof(&p, n), Err(TreeError::IndexOutOfRange)); + } + } + + #[test] + fn a_proof_verifies_only_under_the_committed_leaf_count() { + // An audit path does not fix the tree's size: RFC 9162's check walks it against + // whatever size it is given. So a claim commits to (root, leaf_count), and the + // size a proof is checked under is the commitment's. + let p = ps(7); + let c = commit(&p).unwrap().unwrap(); + assert_eq!(c.leaf_count, 7); + let v = ordered(&p).unwrap(); + let paths: Vec<_> = (0..7).map(|i| proof(&p, i).unwrap()).collect(); + // A 7-leaf proof does not verify against the commitment to 8 leaves, whose root + // differs. + let c8 = commit(&ps(8)).unwrap().unwrap(); + assert_ne!(c8.root, c.root); + for i in 0..7 { + assert!(verify_proof(leaf(&v[i]), i as u64, &paths[i], &c)); + assert!(!verify_proof(leaf(&v[i]), i as u64, &paths[i], &c8), "i = {i}"); + } + // Why the count is committed: under this root with 8 stated as its size, leaf 0's + // path still walks to the root (MTH(D[4:7]) stands where MTH(D[4:8]) would), and + // only leaf 6's is refused. A root alone would let the proof's holder pick the size. + let as8 = OutputCommitment { leaf_count: 8, ..c }; + assert!(verify_proof(leaf(&v[0]), 0, &paths[0], &as8)); + assert!(!verify_proof(leaf(&v[6]), 6, &paths[6], &as8)); + } + + #[test] + fn duplicate_presentation_is_refused() { + let mut p = ps(4); + p.push(Presentation { decoder: 1, ..p[2] }); + assert_eq!(root(&p), Err(TreeError::Duplicate { unit: 2, layers: 3 })); + } +} diff --git a/wai/wai-rs/src/quality.rs b/wai/wai-rs/src/quality.rs new file mode 100644 index 000000000..5fbae779a --- /dev/null +++ b/wai/wai-rs/src/quality.rs @@ -0,0 +1,491 @@ +//! Quality per prefix (`extensions/staged-measurement.md` §4), in integers. +//! +//! A record states quality as integers that every machine computes alike: sums of +//! squared differences, the reference's energy, a largest difference, and a structural +//! similarity in Q32 fixed point. Decibel figures are derived from those integers for +//! display only ([`psnr_db_display`], [`snr_db_display`], [`ssim_display`]), and the +//! only floating-point arithmetic in this module is in those three functions. +//! +//! **Structural similarity** (§4.2). Luma is `Y = (77·R + 150·G + 29·B + 128) >> 8`. +//! Windows are `ww = min(8, W)` by `wh = min(8, H)` pixels. Their left edges are at +//! every multiple of 4 up to `W − ww`, and at `W − ww` itself when it is not one; their +//! top edges likewise up to `H − wh`. A window is at every pair of the two, so every +//! pixel is in one. For a window of `n = ww·wh` pixels, with `Sx = Σx`, `Sy = Σy`, +//! `Sxx = Σx²`, `Syy = Σy²` and `Sxy = Σxy` over the reference's luma `x` and the +//! test's luma `y`: +//! +//! ```text +//! A1 = 20000·Sx·Sy + 65025·n² +//! A2 = 20000·(n·Sxy − Sx·Sy) + 585225·n² +//! B1 = 10000·(Sx² + Sy²) + 65025·n² +//! B2 = 10000·((n·Sxx − Sx²) + (n·Syy − Sy²)) + 585225·n² +//! s_w = ⌊A1·A2·2³² / (B1·B2)⌋ +//! ``` +//! +//! and the figure is `⌊Σ s_w / windows⌋`. Both divisions floor toward −∞. This is the +//! structural similarity of Wang et al. (2004) with `K1 = 0.01`, `K2 = 0.03`, a peak of +//! 255, uniform windows and population statistics, with every factor multiplied out. +//! +//! **Why it fits `i128`.** A window holds `n ≤ 64` pixels of values `0..=255`, so +//! `0 ≤ Sx, Sy ≤ 16320` and `A1 ≤ 20000·16320² + 65025·64² < 5.33·10¹²`. +//! `n·Sxy − Sx·Sy` is `n²` times the window's covariance, whose magnitude is at most +//! the product of the two standard deviations, each at most 127.5, so +//! `|n·Sxy − Sx·Sy| ≤ 4096·16256.25 < 6.66·10⁷` and `|A2| < 1.34·10¹²`. Then +//! `|A1·A2·2³²| < 7.2·10²⁴·2³² < 2¹¹⁵`, well inside `i128`. `B1` and `B2` are each at +//! least their `n²` term, so `B1·B2 > 0`. `B1 ≤ 10000·2·16320² + 65025·64² < 5.33·10¹²`, +//! and `n·Sxx − Sx²` is `n²` times a variance of at most `127.5²`, so +//! `B2 ≤ 10000·2·4096·16256.25 + 585225·64² < 1.34·10¹²`; then +//! `B1·B2 < 7.2·10²⁴ < 2⁸³`. A similarity lies in `[−1, 1]`, so `|s_w| ≤ 2³²`, and a +//! sum over fewer than 2⁹⁰ windows fits as well: a picture has at most +//! `(⌊(W − 1)/4⌋ + 1)·(⌊(H − 1)/4⌋ + 1)` windows, never more than it has pixels. +//! +//! Several units are pooled (a clip, or every unit of a prefix) by summing each sum and +//! each window's `s_w` over the units and dividing once ([`image_pooled`]). +#![deny(clippy::float_arithmetic)] + +/// An RGB8 picture: `h` rows of `w` pixels, `R, G, B` bytes, rows top to bottom. +#[derive(Clone, Copy, Debug)] +pub struct Rgb8<'a> { + pub w: usize, + pub h: usize, + pub px: &'a [u8], +} + +/// §4.2's integers for one picture, or for several pooled. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct ImageQuality { + /// Bytes compared: `3·W·H` per picture. + pub n_samples: u64, + /// `Σ (aᵢ − bᵢ)²` over the RGB bytes. + pub sse_rgb: u128, + /// The same sum over the luma planes. + pub sse_y: u128, + /// The mean structural similarity of the luma planes, in Q32 (`2³²` is 1). + pub ssim_y_q32: i64, + /// The windows the mean is over. + pub windows: u64, +} + +/// §4.3's integers for two sequences of `i16` samples. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PcmQuality { + pub n_samples: u64, + /// `Σ (aᵢ − bᵢ)²`. + pub sse: u128, + /// `Σ aᵢ²` over the reference. + pub ref_energy: u128, +} + +/// §4.4's integers for two integer tensors. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct TensorQuality { + pub n: u64, + /// `Σ (aᵢ − bᵢ)²`. + pub sse: u128, + /// `max |aᵢ − bᵢ|`, 0 when equal. + pub max_abs: u64, +} + +/// Why two outputs have no quality figure. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum QualityError { + /// Their sizes differ, or a picture's buffer is not `3·W·H` bytes. + SizeMismatch, + /// There is nothing to compare: no sample, or a picture with no pixel. + Empty, + /// A sum passed `u128` (only tensors of 64-bit values far apart reach it). + Overflow, +} + +impl std::fmt::Display for QualityError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(match self { + Self::SizeMismatch => "the two outputs differ in size", + Self::Empty => "nothing to compare", + Self::Overflow => "a sum passes 128 bits", + }) + } +} + +impl std::error::Error for QualityError {} + +/// The integer BT.601 luma of RGB8 bytes: `(77·R + 150·G + 29·B + 128) >> 8` per +/// pixel. A trailing partial pixel is ignored. +pub fn luma_bt601_int(px: &[u8]) -> Vec { + px.chunks_exact(3) + .map(|p| ((77 * p[0] as u32 + 150 * p[1] as u32 + 29 * p[2] as u32 + 128) >> 8) as u8) + .collect() +} + +/// One picture's sums, with `Σ s_w` and its window count, before the mean is taken. +struct ImageSums { + n: u64, + sse_rgb: u128, + sse_y: u128, + ssim_sum: i128, + windows: u64, +} + +fn sq(a: u8, b: u8) -> u128 { + let d = a.abs_diff(b) as u128; + d * d +} + +/// A window's offsets along an axis of `len` pixels: every multiple of 4 up to +/// `len − win`, then `len − win` itself when it is not one, so the last window ends at +/// the edge and every pixel is in a window. +fn starts(len: usize, win: usize) -> impl Iterator { + let last = len - win; + (0..=last).step_by(4).chain((last % 4 != 0).then_some(last)) +} + +fn image_sums(r: Rgb8<'_>, t: Rgb8<'_>) -> Result { + if r.w != t.w || r.h != t.h { + return Err(QualityError::SizeMismatch); + } + if r.w == 0 || r.h == 0 { + return Err(QualityError::Empty); + } + let len = r.w.checked_mul(r.h).and_then(|n| n.checked_mul(3)).ok_or(QualityError::SizeMismatch)?; + if r.px.len() != len || t.px.len() != len { + return Err(QualityError::SizeMismatch); + } + let sse_rgb = r.px.iter().zip(t.px).map(|(&a, &b)| sq(a, b)).sum(); + let (ya, yb) = (luma_bt601_int(r.px), luma_bt601_int(t.px)); + let sse_y = ya.iter().zip(&yb).map(|(&a, &b)| sq(a, b)).sum(); + let (w, h) = (r.w, r.h); + let (ww, wh) = (w.min(8), h.min(8)); + let n = (ww * wh) as i128; + let (c1, c2) = (65025 * n * n, 585225 * n * n); + let mut ssim_sum: i128 = 0; + let mut windows = 0u64; + for y0 in starts(h, wh) { + for x0 in starts(w, ww) { + let (mut sx, mut sy, mut sxx, mut syy, mut sxy) = (0i128, 0i128, 0i128, 0i128, 0i128); + for y in y0..y0 + wh { + for x in x0..x0 + ww { + let (p, q) = (ya[y * w + x] as i128, yb[y * w + x] as i128); + sx += p; + sy += q; + sxx += p * p; + syy += q * q; + sxy += p * q; + } + } + let a1 = 20000 * sx * sy + c1; + let a2 = 20000 * (n * sxy - sx * sy) + c2; + let b1 = 10000 * (sx * sx + sy * sy) + c1; + let b2 = 10000 * ((n * sxx - sx * sx) + (n * syy - sy * sy)) + c2; + ssim_sum += (a1 * a2 * (1i128 << 32)).div_euclid(b1 * b2); + windows += 1; + } + } + Ok(ImageSums { n: len as u64, sse_rgb, sse_y, ssim_sum, windows }) +} + +/// §4.2 for one picture against its reference. +pub fn image(reference: Rgb8<'_>, test: Rgb8<'_>) -> Result { + image_pooled(&[(reference, test)]) +} + +/// §4.2's sums pooled over pictures added one at a time (§4.5): every sum is summed +/// over the pictures, and the similarity is the mean of every window of every +/// picture, one division at the end. A pool holds no picture, so pooling a prefix's +/// units needs only one unit's output at a time. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ImagePool { + n: u64, + sse_rgb: u128, + sse_y: u128, + ssim_sum: i128, + windows: u64, +} + +impl ImagePool { + /// Add one picture against its reference; the pair must match in size. + pub fn add(&mut self, reference: Rgb8<'_>, test: Rgb8<'_>) -> Result<(), QualityError> { + let s = image_sums(reference, test)?; + self.n += s.n; + self.sse_rgb += s.sse_rgb; + self.sse_y += s.sse_y; + self.ssim_sum += s.ssim_sum; + self.windows += s.windows; + Ok(()) + } + + /// The pooled figure, or `None` when no picture was added. + pub fn finish(&self) -> Option { + // Every picture has at least one window (`ww ≤ W`, `wh ≤ H`), so a pool with a + // picture has `windows ≥ 1`. + (self.windows > 0).then(|| ImageQuality { + n_samples: self.n, + sse_rgb: self.sse_rgb, + sse_y: self.sse_y, + ssim_y_q32: self.ssim_sum.div_euclid(self.windows as i128) as i64, + windows: self.windows, + }) + } +} + +/// §4.2 pooled over several pictures (a clip's frames, or a prefix's units): every +/// sum is summed over the pictures, and the similarity is the mean of every window +/// of every picture. Each pair must match in size; the pictures may differ. +pub fn image_pooled(pairs: &[(Rgb8<'_>, Rgb8<'_>)]) -> Result { + let mut pool = ImagePool::default(); + for &(r, t) in pairs { + pool.add(r, t)?; + } + pool.finish().ok_or(QualityError::Empty) +} + +/// §4.3 for two sequences of `i16` samples, channel-interleaved in output order. +/// Sequences of different lengths are refused. +pub fn pcm16(reference: &[i16], test: &[i16]) -> Result { + if reference.len() != test.len() { + return Err(QualityError::SizeMismatch); + } + if reference.is_empty() { + return Err(QualityError::Empty); + } + let mut q = PcmQuality { n_samples: reference.len() as u64, sse: 0, ref_energy: 0 }; + for (&a, &b) in reference.iter().zip(test) { + let d = (a as i32 - b as i32).unsigned_abs() as u128; + q.sse += d * d; + let e = (a as i32).unsigned_abs() as u128; + q.ref_energy += e * e; + } + Ok(q) +} + +/// [`pcm16`] pooled over several pairs. +pub fn pcm16_pooled(pairs: &[(&[i16], &[i16])]) -> Result { + if pairs.is_empty() { + return Err(QualityError::Empty); + } + let mut out = PcmQuality { n_samples: 0, sse: 0, ref_energy: 0 }; + for &(r, t) in pairs { + let q = pcm16(r, t)?; + out.n_samples += q.n_samples; + out.sse += q.sse; + out.ref_energy += q.ref_energy; + } + Ok(out) +} + +/// §4.4 for two integer tensors in the capability's canonical order. +pub fn tensor_i64(reference: &[i64], test: &[i64]) -> Result { + tensor_pooled(&[(reference, test)]) +} + +/// [`tensor_i64`] pooled over several pairs: the sums are summed and `max_abs` is the +/// largest. +pub fn tensor_pooled(pairs: &[(&[i64], &[i64])]) -> Result { + if pairs.is_empty() { + return Err(QualityError::Empty); + } + let mut out = TensorQuality { n: 0, sse: 0, max_abs: 0 }; + for &(r, t) in pairs { + if r.len() != t.len() { + return Err(QualityError::SizeMismatch); + } + if r.is_empty() { + return Err(QualityError::Empty); + } + for (&a, &b) in r.iter().zip(t) { + let d = (a as i128 - b as i128).unsigned_abs(); + // |a − b| < 2⁶⁴ for two i64, so it fits u64; its square fits u128. + out.max_abs = out.max_abs.max(d as u64); + out.sse = out.sse.checked_add(d * d).ok_or(QualityError::Overflow)?; + } + out.n += r.len() as u64; + } + Ok(out) +} + +/// PSNR in dB for `n` 8-bit samples with squared error `sse`: +/// `10·log10(255²·n / sse)`. For display only (§4.6); `None` when `sse` is 0 (no +/// finite figure) or `n` is 0. +#[allow(clippy::float_arithmetic)] +pub fn psnr_db_display(sse: u128, n: u64) -> Option { + if sse == 0 || n == 0 { + return None; + } + Some(10.0 * (65025.0 * n as f64 / sse as f64).log10()) +} + +/// SNR in dB: `10·log10(ref_energy / sse)`. For display only (§4.6); `None` when +/// `sse` or `ref_energy` is 0. +#[allow(clippy::float_arithmetic)] +pub fn snr_db_display(ref_energy: u128, sse: u128) -> Option { + if sse == 0 || ref_energy == 0 { + return None; + } + Some(10.0 * (ref_energy as f64 / sse as f64).log10()) +} + +/// A Q32 similarity as a number in `[−1, 1]`, for display only. +#[allow(clippy::float_arithmetic)] +pub fn ssim_display(q32: i64) -> f64 { + q32 as f64 / 4_294_967_296.0 +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A deterministic test picture. The literals below were computed from it in Python + /// twice, by `staged-measure-conformance/measure_verify.py`'s `image_parts` and by a + /// rational-arithmetic structural similarity with its own window placement. + fn picture(w: usize, h: usize, seed: usize, mul: usize) -> Vec { + let mut out = Vec::with_capacity(w * h * 3); + for y in 0..h { + for x in 0..w { + for c in 0..3 { + out.push(((x * mul + y * (mul + 3) + c * 29 + seed * 17 + ((x * y + seed) % 13) * 5) % 256) as u8); + } + } + } + out + } + + fn q(w: usize, h: usize, a: &[u8], b: &[u8]) -> ImageQuality { + image(Rgb8 { w, h, px: a }, Rgb8 { w, h, px: b }).unwrap() + } + + fn iq(n_samples: u64, sse_rgb: u128, sse_y: u128, ssim_y_q32: i64, windows: u64) -> ImageQuality { + ImageQuality { n_samples, sse_rgb, sse_y, ssim_y_q32, windows } + } + + #[test] + fn sse_literals() { + // Hand values: (255² · 3) + (3² + 4²) + (1 + 1 + 9) = 195075 + 25 + 11. + let a = [0, 0, 0, 255, 255, 255, 10, 20, 30, 200, 100, 50]; + let b = [0, 0, 0, 0, 0, 0, 13, 16, 30, 201, 99, 53]; + assert_eq!(q(2, 2, &a, &b), iq(12, 195111, 65026, 187211645, 1)); + assert_eq!(q(2, 2, &[7; 12], &[7; 12]), iq(12, 0, 0, 1 << 32, 1)); + assert_eq!(q(2, 2, &[255; 12], &[0; 12]), iq(12, 780300, 260100, 429453, 1)); + } + + #[test] + fn ssim_identical_is_one() { + for (w, h) in [(1, 1), (8, 8), (13, 5), (40, 24)] { + let p = picture(w, h, 2, 9); + assert_eq!(q(w, h, &p, &p).ssim_y_q32, 1 << 32, "{w}x{h}"); + } + } + + #[test] + fn ssim_extreme_windows() { + // A full 8×8 window at the extremes: all white against all black, and a + // checkerboard against its inverse (the most negative covariance a window has). + assert_eq!(q(8, 8, &[255; 192], &[0; 192]), iq(192, 12484800, 4161600, 429453, 1)); + let cb: Vec = (0..64).flat_map(|i| [if (i % 8 + i / 8) % 2 == 1 { 255u8 } else { 0 }; 3]).collect(); + let inv: Vec = cb.iter().map(|v| 255 - v).collect(); + assert_eq!(q(8, 8, &cb, &inv), iq(192, 12484800, 4161600, -4279533196, 1)); + } + + #[test] + fn ssim_small_planes() { + // Planes narrower or shorter than a window use clipped windows: 3 × 8 down a + // 3 × 100 plane at stride 4 (24 windows), and one 5 × 5 window. + assert_eq!(q(3, 100, &picture(3, 100, 1, 7), &picture(3, 100, 2, 7)), iq(900, 4705154, 532902, 2841543333, 24)); + assert_eq!(q(5, 5, &picture(5, 5, 3, 11), &picture(5, 5, 3, 13)), iq(75, 65392, 2145, 4227018163, 1)); + } + + #[test] + fn ssim_literals() { + // (w, h, seed a, mul a, seed b, mul b) → n, sse_rgb, sse_y, ssim_y_q32, windows, + // as the independent reference computes them. + // 17 × 9, 9 × 33 and 11 × 11 have an edge-aligned last window on an axis. + let cases: [((usize, usize, usize, usize, usize, usize), ImageQuality); 7] = [ + ((16, 16, 1, 5, 1, 6), iq(768, 3329152, 245154, 3542145100, 9)), + ((17, 9, 2, 3, 5, 3), iq(459, 2191911, 500794, 1115121540, 8)), + ((12, 20, 4, 9, 5, 9), iq(720, 4072752, 548256, 2404129353, 8)), + ((32, 8, 6, 1, 7, 2), iq(768, 8646871, 893709, -424684373, 7)), + ((9, 33, 8, 13, 9, 14), iq(891, 8244787, 1469842, 1291000068, 16)), + ((40, 24, 3, 7, 3, 8), iq(2880, 19201376, 3078475, 2198626257, 45)), + ((11, 11, 1, 5, 2, 7), iq(363, 1366484, 201849, 3172402069, 4)), + ]; + for ((w, h, s1, m1, s2, m2), want) in cases { + assert_eq!(q(w, h, &picture(w, h, s1, m1), &picture(w, h, s2, m2)), want, "{w}x{h}"); + } + } + + #[test] + fn pooled_is_one_mean_over_every_window() { + let (a0, b0) = (picture(16, 16, 1, 5), picture(16, 16, 1, 6)); + let (a1, b1) = (picture(17, 9, 2, 3), picture(17, 9, 5, 3)); + let p = image_pooled(&[ + (Rgb8 { w: 16, h: 16, px: &a0 }, Rgb8 { w: 16, h: 16, px: &b0 }), + (Rgb8 { w: 17, h: 9, px: &a1 }, Rgb8 { w: 17, h: 9, px: &b1 }), + ]) + .unwrap(); + // ⌊(31879305901 + 8920972324) / 17⌋, not the mean of the two pictures' means. + assert_eq!(p, iq(1227, 5521063, 745948, 2400016366, 17)); + } + + #[test] + fn every_pixel_is_in_a_window() { + // Offsets per axis: multiples of 4 up to `len − win`, then the edge-aligned one. + assert_eq!(starts(11, 8).collect::>(), [0, 3]); + assert_eq!(starts(16, 8).collect::>(), [0, 4, 8]); + assert_eq!(starts(17, 8).collect::>(), [0, 4, 8, 9]); + assert_eq!(starts(5, 5).collect::>(), [0]); + for len in 1..=70 { + let win = len.min(8); + let mut seen = vec![false; len]; + let v: Vec = starts(len, win).collect(); + for &x in &v { + seen[x..x + win].iter_mut().for_each(|s| *s = true); + } + assert!(seen.iter().all(|&s| s), "len {len}"); + // Never more offsets than the stated bound, and no offset twice. + assert!(v.len() <= (len - 1) / 4 + 1 && v.windows(2).all(|w| w[0] < w[1]), "len {len}"); + } + // An 11 × 11 picture whose only difference is its last row and column: with + // windows at 0 alone it would score 1; the edge-aligned windows see it. + let a = picture(11, 11, 4, 3); + let mut b = a.clone(); + for i in 0..11 { + for c in 0..3 { + b[(10 * 11 + i) * 3 + c] ^= 0x40; + b[(i * 11 + 10) * 3 + c] ^= 0x40; + } + } + let got = q(11, 11, &a, &b); + assert_eq!(got.windows, 4); + assert!(got.ssim_y_q32 < 1 << 32); + } + + #[test] + fn sizes_are_checked() { + let p = [0u8; 12]; + assert_eq!(image(Rgb8 { w: 2, h: 2, px: &p }, Rgb8 { w: 1, h: 4, px: &p }), Err(QualityError::SizeMismatch)); + assert_eq!(image(Rgb8 { w: 2, h: 2, px: &p }, Rgb8 { w: 2, h: 2, px: &p[..9] }), Err(QualityError::SizeMismatch)); + assert_eq!(image(Rgb8 { w: 0, h: 4, px: &[] }, Rgb8 { w: 0, h: 4, px: &[] }), Err(QualityError::Empty)); + assert_eq!(pcm16(&[1, 2], &[1]), Err(QualityError::SizeMismatch)); + assert_eq!(pcm16(&[], &[]), Err(QualityError::Empty)); + assert_eq!(tensor_i64(&[1], &[1, 2]), Err(QualityError::SizeMismatch)); + } + + #[test] + fn pcm_and_tensor_literals() { + // sse = 2² + 32767² + 65535²; ref_energy = 10² + 32767² + 32768². + let q = pcm16(&[10, -32767, -32768], &[12, 0, 32767]).unwrap(); + assert_eq!(q, PcmQuality { n_samples: 3, sse: 4 + 1073676289 + 4294836225, ref_energy: 100 + 1073676289 + 1073741824 }); + // The widest i64 gap: |i64::MIN − i64::MAX| = 2⁶⁴ − 1, squared within u128. + let t = tensor_i64(&[i64::MIN, 5, 7], &[i64::MAX, 5, 4]).unwrap(); + assert_eq!(t, TensorQuality { n: 3, sse: (u64::MAX as u128) * (u64::MAX as u128) + 9, max_abs: u64::MAX }); + // Two such gaps pass u128. + assert_eq!(tensor_i64(&[i64::MIN, i64::MIN], &[i64::MAX, i64::MAX]), Err(QualityError::Overflow)); + } + + #[test] + fn display_figures() { + assert_eq!(psnr_db_display(0, 10), None); + // 255²·1 / 65025 = 1 → 0 dB. + assert_eq!(psnr_db_display(65025, 1), Some(0.0)); + assert_eq!(snr_db_display(7, 7), Some(0.0)); + assert_eq!(ssim_display(1 << 31), 0.5); + } +} diff --git a/wai/wai-rs/src/stage_measure/decode.rs b/wai/wai-rs/src/stage_measure/decode.rs new file mode 100644 index 000000000..cd1bcacc3 --- /dev/null +++ b/wai/wai-rs/src/stage_measure/decode.rs @@ -0,0 +1,205 @@ +//! Every stage of a staged set, decoded through the reference sink. +//! +//! A base is decoded by `staged::decode_base_pinned`, the one base decode the sink, the +//! `wai stage` tool and the browser binding share: SPEC §4's dispatch through the +//! integer decoders, the pin verified against the parameter set (SPEC §3.1), and the +//! title's cost limit applied to the payload's declared cost before any symbol is +//! decoded or any output allocated. Its layers are applied by `staged::StageSink` +//! (`extensions/staged-delivery.md` §4) under the title's chain budget, with the same +//! limit on each layer's declared cost (Appendix A.5), one layer index at a time; each +//! unit's output at each stage is handed to the caller as it is reached and not kept, +//! so a replay holds one output per unit. This is a composition of library functions +//! for measurement, not a sink's dispatch path: it requires every layer to apply to +//! every unit it covers, as a set delivered whole and in order does, and refuses +//! anything else. + +use std::collections::BTreeMap; + +use crate::container::Wai; +use crate::staged::{tier, BaseInfo, ChainConfig, HoldBound, ReceiptEvidence, StageRefusal, StageSink, UnitBuf}; + +use super::title::{Limits, TitleBase}; +use super::MeasureError; + +fn invalid(s: impl Into) -> MeasureError { + MeasureError::Invalid(s.into()) +} + +/// A base decoded: its identity as the sink holds it, its units, and the SHA-256 of +/// the pin its decode verified (`None` when unpinned). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DecodedBase { + pub info: BaseInfo, + pub units: Vec, + pub pin: Option, +} + +/// Decode a base under `limits`, and require what its case states: its stage-0 output, +/// the capability it was decoded through, whether that was the fallback, its tier and +/// whether its pin resolved. A refusal carries the decode's code. +pub fn decode_base(b: &TitleBase, limits: &Limits) -> Result { + let at = &b.obj.path; + let set: BTreeMap = b.set.iter().map(|s| (s.member.clone(), s.bytes.as_slice())).collect(); + let d = crate::staged::decode_base_pinned(&b.obj.bytes, None, &set, Some(&limits.cost), None) + .map_err(|e| MeasureError::Refused { at: at.clone(), code: e.code().to_string(), detail: e.to_string() })?; + let mut info = d.info; + info.first_unit = b.first_unit; + if let Some(s0) = &b.stage0 { + let mine: Vec = d.units.iter().flat_map(|u| u.samples.canonical_bytes()).collect(); + if &mine != s0 { + return Err(invalid(format!("{at}: the decode is not the case's stage 0"))); + } + } + if let Some(c) = &b.case { + let got = (info.capability.as_str(), info.via_fallback, tier::token(info.tier), d.pin.is_some()); + let want = (c.decoded_via.as_str(), c.via_fallback, c.tier.as_str(), c.pin_resolved); + if got != want { + return Err(invalid(format!("{at}: the decode gives (via, fallback, tier, pin) {got:?}; the case states {want:?}"))); + } + } + Ok(DecodedBase { info, units: d.units, pin: d.pin.map(|p| p.sha256) }) +} + +/// The chain configuration a replay runs under: the title's chain budget, its cost +/// limit on each layer, a hold bound no in-order replay reaches, and no live schedule. +pub fn chain_config(limits: &Limits) -> ChainConfig { + ChainConfig { + // A replay offers each layer after the layers it needs, so nothing is held; a + // layer held at all is refused by the replay, whatever the bound. + bound: HoldBound { max_envelopes: 4096, max_age_ms: u64::MAX }, + cost_limit: Some(crate::staged::layer_limit(&limits.cost)), + budget: limits.chain_budget, + // A replay never drops a chain, so it keeps no record of one. + record_budget: 0, + live: None, + } +} + +/// One stage a unit reached, as a replay reports it. +#[derive(Debug, Clone, Copy)] +pub struct StageEvent<'a> { + /// The global unit index: the base's `first_unit` plus the unit's index in it. + pub unit: u64, + /// The base's index in the title. + pub base: usize, + /// The stage's layer count: 1 for the base alone. + pub layers: u16, + /// The capability that produced it: the base's, or the layer's. + pub capability: &'a str, + /// The pin the base's decode verified, for stage 0. + pub pin: Option<&'a str>, + pub buf: &'a UnitBuf, +} + +/// A unit's output at the last stage it reached. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct FinalUnit { + pub unit: u64, + pub base: usize, + /// Its stage's layer count. + pub layers: u16, + pub buf: UnitBuf, +} + +/// What one layer envelope is: its base, its layer index, its units, its capability. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LayerInfo { + pub base: usize, + pub layer: u8, + /// The global units it covers: `[from, from + count)`. + pub from: u64, + pub count: u64, + pub capability: String, +} + +fn hex(b: &[u8]) -> String { + b.iter().map(|x| format!("{x:02x}")).collect() +} + +/// Read a layer envelope's `refines` against the bases. +pub fn layer_info(bytes: &[u8], bases: &[DecodedBase], at: &str) -> Result { + let w = Wai::from_bytes(bytes).map_err(|e| invalid(format!("{at}: {e}")))?; + let r = w.refines().ok_or_else(|| invalid(format!("{at}: not a layer (no refines)")))?.map_err(|e| invalid(format!("{at}: {e}")))?; + let b = bases + .iter() + .position(|d| hex(&d.info.content_hash) == r.base) + .ok_or_else(|| invalid(format!("{at}: its base is not one of the title's")))?; + let n = bases[b].units.len() as u32; + let (from, count) = r.units(n); + Ok(LayerInfo { base: b, layer: r.layer, from: bases[b].info.first_unit + from as u64, count: count as u64, capability: w.capability().to_string() }) +} + +/// Apply `layers` (envelope bytes, with a name for messages) to the decoded bases under +/// `limits`, layer index by layer index and, within one, in the order given. `on_stage` +/// sees every stage every unit reaches, stage 0 first, as it is reached; the result is +/// each unit's output at its last stage, in unit order. A base past the chain budget is +/// refused (`chain-over-limit`), and so is a layer that does not apply to every unit it +/// covers (its reason's code). +pub fn replay( + bases: &[DecodedBase], + layers: &[(&str, &[u8])], + limits: &Limits, + on_stage: &mut dyn FnMut(StageEvent<'_>) -> Result<(), MeasureError>, +) -> Result, MeasureError> { + let mut sink = StageSink::new(chain_config(limits)); + let mut seen = std::collections::BTreeSet::new(); + for (bi, b) in bases.iter().enumerate() { + for (u, buf) in b.units.iter().enumerate() { + let unit = b.info.first_unit + u as u64; + if !seen.insert(unit) { + return Err(invalid(format!("two bases give the global unit {unit}"))); + } + on_stage(StageEvent { unit, base: bi, layers: 1, capability: &b.info.capability, pin: b.pin.as_deref(), buf })?; + } + let reports = sink.add_base(b.info.clone(), b.units.clone()); + if let Some(r) = reports.iter().find(|r| r.envelope == Some(StageRefusal::ChainOverLimit)) { + return Err(MeasureError::Refused { + at: format!("base {bi} ({})", hex(&r.content_hash)), + code: StageRefusal::ChainOverLimit.code().into(), + detail: "the base's units pass the chain budget".into(), + }); + } + if !reports.is_empty() { + return Err(invalid("a base released something on decode: two bases with one content hash")); + } + } + let mut infos = Vec::new(); + for (name, bytes) in layers { + infos.push(layer_info(bytes, bases, name)?); + } + let top = infos.iter().map(|l| l.layer).max().unwrap_or(0); + for k in 1..=top { + for (i, ((name, bytes), info)) in layers.iter().zip(&infos).enumerate() { + if info.layer != k { + continue; + } + let reports = sink.offer(bytes, i as u64, ReceiptEvidence::None).map_err(|e| invalid(format!("{name}: {e}")))?; + let r = reports.first().filter(|_| reports.len() == 1).ok_or_else(|| invalid(format!("{name}: {} reports for one offer", reports.len())))?; + if let Some(why) = r.envelope { + return Err(MeasureError::Refused { at: name.to_string(), code: why.code().into(), detail: r.detail.unwrap_or("the layer is refused").into() }); + } + if let Some(&(u, why)) = r.refused.first() { + return Err(MeasureError::Refused { at: name.to_string(), code: why.code().into(), detail: format!("unit {u}") }); + } + if !r.held.is_empty() || r.held_for_base || r.applied.len() as u64 != info.count { + return Err(invalid(format!("{name}: does not apply to every unit it covers"))); + } + let base = &bases[info.base]; + let chain = sink.chain(&base.info.content_hash).expect("a base the sink holds"); + for &u in &r.applied { + let st = chain.unit(u).expect("an applied unit"); + let unit = base.info.first_unit + u as u64; + on_stage(StageEvent { unit, base: info.base, layers: st.stage as u16 + 1, capability: &info.capability, pin: None, buf: &st.buf })?; + } + } + } + let mut finals = Vec::new(); + for (bi, b) in bases.iter().enumerate() { + let chain = sink.chain(&b.info.content_hash).expect("a base the sink holds"); + for (u, st) in chain.units().iter().enumerate() { + finals.push(FinalUnit { unit: b.info.first_unit + u as u64, base: bi, layers: st.stage as u16 + 1, buf: st.buf.clone() }); + } + } + finals.sort_by_key(|f| f.unit); + Ok(finals) +} diff --git a/wai/wai-rs/src/stage_measure/mod.rs b/wai/wai-rs/src/stage_measure/mod.rs new file mode 100644 index 000000000..6b449db75 --- /dev/null +++ b/wai/wai-rs/src/stage_measure/mod.rs @@ -0,0 +1,65 @@ +//! Staged-delivery measurement (`extensions/staged-measurement.md`): the deterministic +//! figures of a staged title, computed through the reference sink. +//! +//! - [`title`]: a measured title, read from a fixture directory or from a map of files. +//! - [`decode`]: its bases decoded by `staged::decode_base_pinned`, the sink's own base +//! decode, under the title's cost limit, and its layers applied by +//! `staged::StageSink` under its chain budget, each unit's output at each stage handed +//! over as it is reached. +//! - [`run`]: the evaluation — counted bytes per object, stage and prefix (§3), quality +//! per presentation and per prefix (§4), first presentable (§5.1), the layering +//! overhead's rate–quality points (§6), deduplication (§9) and the output tree +//! (Appendix C) — and its check against the fixture tool's own reconstruction. +//! - [`sim`]: the staged delivery simulator (§10, Appendix B), its stall bound and the +//! check of a plan against it. +//! +//! Everything here is integer and reads no clock and no file outside [`title::Title::load`], +//! so it builds for wasm32 and every machine computes the same figures. + +pub mod decode; +pub mod run; +pub mod sim; +pub mod title; + +/// The `format` of a fixture's `expect.json`: the reconstruction digests the fixture +/// tool computed from its own encoder. +pub const EXPECT_FORMAT: &str = "wai-measure-expect/1"; + +/// The `format` of a corpus's `goldens.json`. +pub const GOLDENS_FORMAT: &str = "wai-measure-goldens/1"; + +/// Why a title could not be read or evaluated. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum MeasureError { + /// A file the title needs is not among those given: its path below `wai/`. + Missing(String), + /// The title, an object or a figure is not what the method requires. + Invalid(String), + /// A base or a layer is refused under the title's limits, with the refusal's code: + /// the base decode's (`staged::BaseError::code`: `cost_over_limit`, + /// `missing_capability`, a pin refusal, …), or the sink's (`chain-over-limit`, a + /// layer's §4.2 reason). + Refused { at: String, code: String, detail: String }, +} + +impl MeasureError { + /// The refusal's code, for a refusal. + pub fn code(&self) -> Option<&str> { + match self { + Self::Refused { code, .. } => Some(code), + _ => None, + } + } +} + +impl std::fmt::Display for MeasureError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Missing(p) => write!(f, "{p}: not given"), + Self::Invalid(s) => f.write_str(s), + Self::Refused { at, code, detail } => write!(f, "{at}: refused, {code}: {detail}"), + } + } +} + +impl std::error::Error for MeasureError {} diff --git a/wai/wai-rs/src/stage_measure/run.rs b/wai/wai-rs/src/stage_measure/run.rs new file mode 100644 index 000000000..77cbfb855 --- /dev/null +++ b/wai/wai-rs/src/stage_measure/run.rs @@ -0,0 +1,538 @@ +//! The deterministic evaluation of a title (`extensions/staged-measurement.md` §3–§6, +//! §9, Appendix C): every figure CI may assert, computed the same on every machine. + +use serde_json::{json, Value}; + +use crate::codecs::{capability_media, Media}; +use crate::content_dedup::{totals_checked, DedupTotals}; +use crate::measure_framing::{count_object, Counted, REGISTERED}; +use crate::output_tree::{out_digest, Presentation}; +use crate::quality::{self, ImageQuality, PcmQuality, Rgb8, TensorQuality}; +use crate::staged::{Samples, UnitBuf}; + +use std::collections::BTreeMap; + +use super::decode::{decode_base, layer_info, replay, DecodedBase, FinalUnit}; +use super::title::{Obj, Title}; +use super::MeasureError; + +fn invalid(s: impl Into) -> MeasureError { + MeasureError::Invalid(s.into()) +} + +fn hex(b: &[u8]) -> String { + b.iter().map(|x| format!("{x:02x}")).collect() +} + +/// What a title's units hold, which fixes the quality arithmetic (§4). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UnitClass { + /// Pictures and frames, RGB8 (§4.2). + Rgb8, + /// PCM, `i16` (§4.3). + Pcm16, + /// An integer tensor: a splat's attribute planes (§4.4). + Tensor, +} + +impl UnitClass { + /// The class of a base capability's units. + pub fn of(capability: &str) -> Option { + match capability_media(capability)? { + Media::Image | Media::Video => Some(UnitClass::Rgb8), + Media::Audio => Some(UnitClass::Pcm16), + Media::Volumetric => Some(UnitClass::Tensor), + _ => None, + } + } + pub fn token(self) -> &'static str { + match self { + UnitClass::Rgb8 => "rgb8", + UnitClass::Pcm16 => "pcm16", + UnitClass::Tensor => "tensor-i64", + } + } +} + +/// A quality figure of one of §4's kinds. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum QualityKind { + Rgb8(ImageQuality), + Pcm16(PcmQuality), + Tensor(TensorQuality), +} + +impl QualityKind { + /// The figure as a record writes it: counts and similarities as numbers, every sum + /// of squares as a decimal string (§4.3). + pub fn to_json(&self) -> Value { + match self { + QualityKind::Rgb8(q) => json!({"kind": "rgb8", "n_samples": q.n_samples, "sse_rgb": q.sse_rgb.to_string(), + "sse_y": q.sse_y.to_string(), "ssim_y_q32": q.ssim_y_q32, "windows": q.windows}), + QualityKind::Pcm16(q) => json!({"kind": "pcm16", "n_samples": q.n_samples, "sse": q.sse.to_string(), "ref_energy": q.ref_energy.to_string()}), + QualityKind::Tensor(q) => json!({"kind": "tensor-i64", "n": q.n, "sse": q.sse.to_string(), "max_abs": q.max_abs}), + } + } +} + +fn opt_json(q: Option) -> Value { + q.map(|q| q.to_json()).unwrap_or(Value::Null) +} + +/// A quality figure pooled over units added one at a time (§4.5), of the title's kind. +/// A unit whose shape differs from its reference's is not added: a stage below an up2 +/// or rate layer has no figure against the final shape. +#[derive(Debug, Clone, Copy)] +struct Pool { + class: UnitClass, + image: quality::ImagePool, + pcm: Option, + tensor: Option, +} + +impl Pool { + fn new(class: UnitClass) -> Pool { + Pool { class, image: quality::ImagePool::default(), pcm: None, tensor: None } + } + + /// Add `test` against the reference `r`, whose shape is `shape`. + fn add(&mut self, test: &UnitBuf, r: &[u8], shape: (u32, u32, u32)) -> Result<(), MeasureError> { + if test.shape != shape || test.samples.byte_len() != r.len() as u64 { + return Ok(()); + } + let q = |e: quality::QualityError| invalid(format!("quality: {e}")); + match (self.class, &test.samples) { + (UnitClass::Rgb8, Samples::U8(t)) => { + let (h, w) = (shape.0 as usize, shape.1 as usize); + self.image.add(Rgb8 { w, h, px: r }, Rgb8 { w, h, px: t }).map_err(q)?; + } + (UnitClass::Pcm16, Samples::I16(t)) => { + let rs: Vec = r.chunks_exact(2).map(|c| i16::from_le_bytes([c[0], c[1]])).collect(); + let x = quality::pcm16(&rs, t).map_err(q)?; + let acc = self.pcm.get_or_insert(PcmQuality { n_samples: 0, sse: 0, ref_energy: 0 }); + acc.n_samples += x.n_samples; + acc.sse += x.sse; + acc.ref_energy += x.ref_energy; + } + (UnitClass::Tensor, Samples::U8(t)) => { + let rs: Vec = r.iter().map(|&x| x as i64).collect(); + let ts: Vec = t.iter().map(|&x| x as i64).collect(); + let x = quality::tensor_i64(&rs, &ts).map_err(q)?; + let acc = self.tensor.get_or_insert(TensorQuality { n: 0, sse: 0, max_abs: 0 }); + acc.n += x.n; + acc.sse = acc.sse.checked_add(x.sse).ok_or_else(|| invalid("quality: a sum passes 128 bits"))?; + acc.max_abs = acc.max_abs.max(x.max_abs); + } + _ => return Err(invalid("a unit holds samples of another kind than the title's")), + } + Ok(()) + } + + fn finish(&self) -> Option { + match self.class { + UnitClass::Rgb8 => self.image.finish().map(QualityKind::Rgb8), + UnitClass::Pcm16 => self.pcm.map(QualityKind::Pcm16), + UnitClass::Tensor => self.tensor.map(QualityKind::Tensor), + } + } +} + +/// One unit's figure against one reference. +fn single(class: UnitClass, test: &UnitBuf, r: &[u8], shape: (u32, u32, u32)) -> Result, MeasureError> { + let mut p = Pool::new(class); + p.add(test, r, shape)?; + Ok(p.finish()) +} + +/// One object of the title, counted (§3). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CountedObject { + pub path: String, + pub content_hash: [u8; 32], + pub capability: String, + pub base: usize, + /// 0 for a base. + pub layer: u8, + pub from: u64, + pub count: u64, + pub counted: Counted, +} + +impl CountedObject { + fn to_json(&self) -> Value { + json!({ + "file": self.path, "object": hex(&self.content_hash), "capability": self.capability, + "base": self.base, "layer": self.layer, "units": [self.from, self.count], + "counted": {"header": self.counted.header, "manifest": self.counted.manifest, "payload": self.counted.payload, "total": self.counted.total}, + }) + } +} + +fn count(o: &Obj) -> Result { + count_object(&o.bytes).map_err(|e| invalid(format!("{}: {e}", o.path))) +} + +/// One presentation of the full delivery, with its quality. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PresentationResult { + pub p: Presentation, + pub shape: (u32, u32, u32), + pub vs_source: Option, + pub vs_final: Option, +} + +/// A rate–quality point of the layering overhead (§6.3). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OverheadPoint { + pub label: String, + pub counted: u64, + pub vs_source: Option, + /// Each unit's final output digest, in unit order. + pub out_digests: Vec<[u8; 32]>, +} + +/// The prefix of `layers` layer counts (§3.5, §4.1). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PrefixResult { + pub layers: u16, + /// The counted bytes of the objects of this stage. + pub stage_counted: u64, + /// The counted bytes of the prefix: every object of this stage and below. + pub counted: u64, + /// Per base: (stage counted, prefix counted). + pub per_base: Vec<(u64, u64)>, + /// Units that reach this stage. + pub units: u64, + pub vs_source: Option, + pub vs_final: Option, +} + +/// Everything §12.1 lets CI assert about a title. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Evaluation { + pub title: String, + pub content_id: [u8; 32], + pub framing: &'static str, + pub unit_us: u64, + pub class: UnitClass, + pub bases: Vec<(DecodedBase, Option, u64)>, + pub objects: Vec, + pub bfpf_min: Option, + pub prefixes: Vec, + /// Decoder entries in first-use order: capability and the pin it verified. + pub decoders: Vec<(String, Option)>, + pub presentations: Vec, + /// The output tree's root and leaf count (Appendix C). + pub output: Option, + pub overhead: Vec, + pub dedup: DedupTotals, +} + +/// Each unit's slice of the source, at its final shape, in global unit order. +fn split_source<'a>(source: &'a [u8], finals: &[FinalUnit]) -> Result, MeasureError> { + let mut at = 0usize; + let mut out = Vec::new(); + for f in finals { + let n = f.buf.samples.byte_len() as usize; + out.push(source.get(at..at + n).ok_or_else(|| invalid("the source is shorter than the units at their final stage"))?); + at += n; + } + if at != source.len() { + return Err(invalid(format!("the source holds {} bytes; the units at their final stage hold {at}", source.len()))); + } + Ok(out) +} + +/// One stage a unit reached, as the first replay records it: no output, its digest. +struct StageRecord { + unit: u64, + layers: u16, + capability: String, + pin: Option, + digest: [u8; 32], + shape: (u32, u32, u32), +} + +/// Evaluate a title under its limits. A title that states the code it must be refused +/// with is evaluated all the same; [`check_refusal`] compares. +pub fn evaluate(t: &Title) -> Result { + let decoded: Vec = t.bases.iter().map(|b| decode_base(b, &t.limits)).collect::>()?; + let class = UnitClass::of(&decoded[0].info.capability).ok_or_else(|| invalid("the base's media has no quality kind"))?; + if decoded.iter().any(|d| UnitClass::of(&d.info.capability) != Some(class)) { + return Err(invalid("the bases' units are of different kinds")); + } + // §3: every object counted. + let mut objects = Vec::new(); + for (i, (b, d)) in t.bases.iter().zip(&decoded).enumerate() { + objects.push(CountedObject { + path: b.obj.path.clone(), + content_hash: d.info.content_hash, + capability: d.info.capability.clone(), + base: i, + layer: 0, + from: d.info.first_unit, + count: d.units.len() as u64, + counted: count(&b.obj)?, + }); + } + for o in &t.layers { + let l = layer_info(&o.bytes, &decoded, &o.path)?; + objects.push(CountedObject { + path: o.path.clone(), + content_hash: crate::jwp_hash::content_hash(&o.bytes), + capability: l.capability, + base: l.base, + layer: l.layer, + from: l.from, + count: l.count, + counted: count(o)?, + }); + } + let named: Vec<(&str, &[u8])> = t.layers.iter().map(|o| (o.path.as_str(), o.bytes.as_slice())).collect(); + + // The first replay: every stage's digest, and each unit's final output. + let mut records: Vec = Vec::new(); + let finals = replay(&decoded, &named, &t.limits, &mut |e| { + records.push(StageRecord { + unit: e.unit, + layers: e.layers, + capability: e.capability.to_string(), + pin: e.pin.map(str::to_string), + digest: out_digest(&e.buf.samples.canonical_bytes()), + shape: e.buf.shape, + }); + Ok(()) + })?; + let source: Option> = t.source.as_ref().map(|s| split_source(&s.bytes, &finals)).transpose()?; + let index: BTreeMap = finals.iter().enumerate().map(|(i, f)| (f.unit, i)).collect(); + let final_bytes: Vec> = finals.iter().map(|f| f.buf.samples.canonical_bytes()).collect(); + + // Decoder entries, in first use as presentations run by (layers, unit). + records.sort_by_key(|r| (r.layers, r.unit)); + let mut decoders: Vec<(String, Option)> = Vec::new(); + for r in &records { + let key = (r.capability.clone(), r.pin.clone()); + if !decoders.contains(&key) { + decoders.push(key); + } + } + records.sort_by_key(|r| (r.unit, r.layers)); + let top = objects.iter().map(|o| o.layer).max().unwrap_or(0) as usize; + + // The second replay: each stage's quality against the source and the final output, + // and the prefixes' pools; a unit's final output also stands for every prefix above + // its last stage. + let mut by_stage: BTreeMap<(u64, u16), (Option, Option)> = BTreeMap::new(); + let mut src_pools: Vec = vec![Pool::new(class); top + 1]; + let mut fin_pools: Vec = vec![Pool::new(class); top + 1]; + replay(&decoded, &named, &t.limits, &mut |e| { + let i = index[&e.unit]; + let shape = finals[i].buf.shape; + let k = e.layers as usize - 1; + let vs_source = match &source { + Some(src) => { + src_pools[k].add(e.buf, src[i], shape)?; + single(class, e.buf, src[i], shape)? + } + None => None, + }; + fin_pools[k].add(e.buf, &final_bytes[i], shape)?; + by_stage.insert((e.unit, e.layers), (vs_source, single(class, e.buf, &final_bytes[i], shape)?)); + Ok(()) + })?; + for (i, f) in finals.iter().enumerate() { + for k in f.layers as usize..=top { + if let Some(src) = &source { + src_pools[k].add(&f.buf, src[i], f.buf.shape)?; + } + fin_pools[k].add(&f.buf, &final_bytes[i], f.buf.shape)?; + } + } + let mut presentations = Vec::new(); + for r in &records { + let decoder = decoders.iter().position(|d| d.0 == r.capability && d.1 == r.pin).expect("listed") as u16; + let (vs_source, vs_final) = by_stage.remove(&(r.unit, r.layers)).ok_or_else(|| invalid("the two replays disagree"))?; + presentations.push(PresentationResult { + p: Presentation { unit: r.unit, layers: r.layers, decoder, out_digest: r.digest }, + shape: r.shape, + vs_source, + vs_final, + }); + } + let ps: Vec = presentations.iter().map(|r| r.p).collect(); + let output = crate::output_tree::commit(&ps).map_err(|e| invalid(e.to_string()))?; + + // §3.5 and §4.1 per prefix. + let mut prefixes = Vec::new(); + let mut cumulative = vec![0u64; decoded.len()]; + for k in 0..=top { + let mut per_base = Vec::new(); + for (b, cum) in cumulative.iter_mut().enumerate() { + let s: u64 = objects.iter().filter(|o| o.base == b && o.layer as usize == k).map(|o| o.counted.total).sum(); + *cum += s; + per_base.push((s, *cum)); + } + prefixes.push(PrefixResult { + layers: k as u16 + 1, + stage_counted: per_base.iter().map(|p| p.0).sum(), + counted: per_base.iter().map(|p| p.1).sum(), + per_base, + units: finals.iter().filter(|f| f.layers as usize > k).count() as u64, + vs_source: src_pools[k].finish(), + vs_final: fin_pools[k].finish(), + }); + } + + // §5.1: the base that holds unit 0, the one object it needs. + let bfpf_min = objects.iter().find(|o| o.layer == 0 && o.from == 0 && o.count >= 1).map(|o| o.counted.total); + + // §6: the staged point, then each single-layer reference. + let mut overhead = Vec::new(); + if let Some(src) = &source { + let mut staged = Pool::new(class); + for (i, f) in finals.iter().enumerate() { + staged.add(&f.buf, src[i], f.buf.shape)?; + } + overhead.push(OverheadPoint { + label: "staged".into(), + counted: objects.iter().map(|o| o.counted.total).sum(), + vs_source: staged.finish(), + out_digests: final_bytes.iter().map(|b| out_digest(b)).collect(), + }); + for r in &t.references { + let named: Vec<(&str, &[u8])> = r.layers.iter().map(|o| (o.path.as_str(), o.bytes.as_slice())).collect(); + let rf = replay(&decoded, &named, &t.limits, &mut |_| Ok(()))?; + let mut counted: u64 = objects.iter().filter(|o| o.layer == 0).map(|o| o.counted.total).sum(); + for o in &r.layers { + counted += count(o)?.total; + } + let mut pool = Pool::new(class); + for (i, f) in rf.iter().enumerate() { + pool.add(&f.buf, src[i], finals[i].buf.shape)?; + } + overhead.push(OverheadPoint { + label: r.label.clone(), + counted, + vs_source: pool.finish(), + out_digests: rf.iter().map(|f| out_digest(&f.buf.samples.canonical_bytes())).collect(), + }); + } + } + + // §9: every object the title references, a reference's bases counted again. + let mut items: Vec<([u8; 32], u64)> = objects.iter().map(|o| (o.content_hash, o.counted.total)).collect(); + for r in &t.references { + items.extend(objects.iter().filter(|o| o.layer == 0).map(|o| (o.content_hash, o.counted.total))); + for o in &r.layers { + items.push((crate::jwp_hash::content_hash(&o.bytes), count(o)?.total)); + } + } + let dedup = totals_checked(items).map_err(|(h, a, b)| invalid(format!("object {} has two lengths, {a} and {b}", hex(&h))))?; + + let bases = decoded.into_iter().zip(&t.bases).map(|(d, b)| (d, b.capture_us, b.set.iter().map(|s| s.bytes.len() as u64).sum())).collect(); + Ok(Evaluation { + title: t.name.clone(), + content_id: t.content_id(), + framing: REGISTERED, + unit_us: t.unit_us, + class, + bases, + objects, + bfpf_min, + prefixes, + decoders, + presentations, + output, + overhead, + dedup, + }) +} + +/// A title that exists to be refused: its evaluation must fail with the code it states. +pub fn check_refusal(t: &Title) -> Result<(), MeasureError> { + let Some(want) = &t.refused else { return Err(invalid(format!("{}: states no refusal", t.dir))) }; + match evaluate(t) { + Err(MeasureError::Refused { code, .. }) if &code == want => Ok(()), + Err(e) => Err(invalid(format!("{}: refused otherwise than `{want}`: {e}", t.dir))), + Ok(_) => Err(invalid(format!("{}: evaluated, though it must be refused `{want}`", t.dir))), + } +} + +fn dedup_json(d: &DedupTotals) -> Value { + json!({"objects": d.objects, "unique": d.unique, "referenced_bytes": d.referenced_bytes, "stored_bytes": d.stored_bytes}) +} + +impl Evaluation { + /// The deterministic block: the JSON a corpus's `goldens.json` holds for the title. + pub fn to_json(&self) -> Value { + json!({ + "title": self.title, + "content_id": hex(&self.content_id), + "framing": self.framing, + "unit_class": self.class.token(), + "unit_us": self.unit_us, + "bases": self.bases.iter().map(|(d, capture, set)| json!({ + "object": hex(&d.info.content_hash), "capability": d.info.capability, "pin": d.pin, + "tier": crate::staged::tier::token(d.info.tier), "first_unit": d.info.first_unit, + "capture_us": capture, "units": d.units.len(), "param_set_bytes": set, + })).collect::>(), + "objects": self.objects.iter().map(CountedObject::to_json).collect::>(), + "bfpf_min": self.bfpf_min, + "prefixes": self.prefixes.iter().map(|p| json!({ + "layers": p.layers, "stage_counted": p.stage_counted, "counted": p.counted, + "per_base": p.per_base.iter().map(|(s, c)| json!({"stage_counted": s, "counted": c})).collect::>(), + "units": p.units, "vs_source": opt_json(p.vs_source), "vs_final": opt_json(p.vs_final), + })).collect::>(), + "decoders": self.decoders.iter().map(|(c, p)| json!({"capability": c, "pin": p})).collect::>(), + "presentations": self.presentations.iter().map(|r| json!({ + "unit": r.p.unit, "layers": r.p.layers, "decoder": r.p.decoder, "shape": [r.shape.0, r.shape.1, r.shape.2], + "out_sha256": hex(&r.p.out_digest), "vs_source": opt_json(r.vs_source), "vs_final": opt_json(r.vs_final), + })).collect::>(), + "output_root": self.output.map(|c| hex(&c.root)), + "leaf_count": self.output.map_or(0, |c| c.leaf_count), + "overhead": self.overhead.iter().map(|o| json!({ + "label": o.label, "counted": o.counted, "vs_source": opt_json(o.vs_source), + "out_sha256": o.out_digests.iter().map(|d| hex(d)).collect::>(), + })).collect::>(), + "dedup": dedup_json(&self.dedup), + }) + } +} + +/// Check the evaluation against the fixture tool's own reconstruction digests +/// (`expect.json`): every presentation's digest, and every reference's final digests. +pub fn check_expect(t: &Title, e: &Evaluation) -> Result<(), MeasureError> { + let Some(x) = &t.expect else { return Ok(()) }; + if x["format"] != super::EXPECT_FORMAT { + return Err(invalid(format!("{}: expect is not {}", t.dir, super::EXPECT_FORMAT))); + } + let got: Vec = e.presentations.iter().map(|r| json!({"unit": r.p.unit, "layers": r.p.layers, "out_sha256": hex(&r.p.out_digest)})).collect(); + if x["presentations"] != Value::Array(got) { + return Err(invalid(format!("{}: the decode's presentations are not the fixture tool's reconstruction", t.dir))); + } + let refs: Vec = e.overhead.iter().filter(|o| o.label != "staged").map(|o| json!({"label": o.label, "out_sha256": o.out_digests.iter().map(|d| hex(d)).collect::>()})).collect(); + if x["references"] != Value::Array(refs) { + return Err(invalid(format!("{}: the references' decodes are not the fixture tool's reconstruction", t.dir))); + } + Ok(()) +} + +/// Dedup over several titles' evaluations (§9.2, across the record). +pub fn record_dedup(evals: &[(&Title, &Evaluation)]) -> Result { + let mut items = Vec::new(); + for (t, e) in evals { + items.extend(e.objects.iter().map(|o| (o.content_hash, o.counted.total))); + for r in &t.references { + items.extend(e.objects.iter().filter(|o| o.layer == 0).map(|o| (o.content_hash, o.counted.total))); + for o in &r.layers { + items.push((crate::jwp_hash::content_hash(&o.bytes), count(o)?.total)); + } + } + } + totals_checked(items).map_err(|(h, a, b)| invalid(format!("object {} has two lengths, {a} and {b}", hex(&h)))) +} + +/// The record-wide dedup block of `goldens.json`. +pub fn record_dedup_json(d: &DedupTotals) -> Value { + dedup_json(d) +} diff --git a/wai/wai-rs/src/stage_measure/sim.rs b/wai/wai-rs/src/stage_measure/sim.rs new file mode 100644 index 000000000..f613b91a9 --- /dev/null +++ b/wai/wai-rs/src/stage_measure/sim.rs @@ -0,0 +1,774 @@ +//! The staged delivery simulator (`extensions/staged-measurement.md` §10, Appendix B): +//! a discrete-event model, in integer microseconds, of a sender putting a title's +//! objects on one link and a sink decoding and presenting them. Every figure it gives +//! is the same on every machine, so continuous integration may assert it; it is +//! labelled "simulated" with its plan, and is never a measurement of a network. +//! +//! A [`Model`] is a title's objects as delivery sees them: each object's base, layer, +//! units and counted bytes ([`Model::from_evaluation`] takes them from an evaluated +//! title). A [`Plan`] (`wai-sim/1`) is the link, the sender's schedule, carriage, +//! layer cap and delivery timeouts, and the sink's decode workers, order, times and +//! playout deadlines. [`simulate`] runs one against the other; [`stall_bound`] states +//! the bound §10.4 gives a plan on how far its refinements may delay a base, and +//! [`check_stall`] checks a plan against it. + +use serde_json::{json, Map, Value}; + +use super::run::Evaluation; + +/// The `plan` member of a `wai-sim/1` plan. +pub const PLAN_FORMAT: &str = "wai-sim/1"; +/// The `format` of a delivery model. +pub const MODEL_FORMAT: &str = "wai-sim-model/1"; + +/// Why a model or plan cannot be simulated. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum SimError { + /// The model or plan is not well formed. + Invalid(String), + /// Under `datagram` carriage, an object larger than one chunk (`datagram-too-large`). + DatagramTooLarge(String), + /// A live plan over an object with no capture time. + NoCaptureTime(String), + /// A time or a byte sum the simulation reaches passes 64 bits: refused, the same in + /// every build. + Overflow, +} + +impl std::fmt::Display for SimError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Invalid(s) => f.write_str(s), + Self::DatagramTooLarge(id) => write!(f, "datagram-too-large: {id} does not fit one chunk"), + Self::NoCaptureTime(id) => write!(f, "{id}: a live plan needs a capture time"), + Self::Overflow => f.write_str("a time or a byte sum passes 64 bits"), + } + } +} + +impl std::error::Error for SimError {} + +fn invalid(s: impl Into) -> SimError { + SimError::Invalid(s.into()) +} + +/// One object as delivery sees it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SimObject { + pub id: String, + /// The base (group) it belongs to. + pub base: u32, + /// 0 for a base. + pub layer: u8, + /// The global units it covers: `[from, from + count)`. + pub from: u64, + pub count: u64, + /// Its counted bytes (§3). + pub bytes: u64, + /// When its content is captured, for a live plan. + pub capture_us: Option, +} + +/// A title's objects, in the order the title lists them (the object index). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Model { + pub unit_us: u64, + pub objects: Vec, +} + +/// The largest integer a model or plan carries: `2⁵³ − 1`, what a JSON number holds +/// exactly. +/// +/// cbindgen:ignore +pub const MAX_FIGURE: u64 = (1 << 53) - 1; + +fn u64_of(v: &Value, k: &str, at: &str) -> Result { + v[k].as_u64().filter(|&x| x <= MAX_FIGURE).ok_or_else(|| invalid(format!("{at}: `{k}` is not an unsigned integer up to 2^53 - 1"))) +} + +fn opt_u64(v: &Value, k: &str, at: &str) -> Result, SimError> { + match &v[k] { + Value::Null => Ok(None), + x => x.as_u64().filter(|&x| x <= MAX_FIGURE).map(Some).ok_or_else(|| invalid(format!("{at}: `{k}` is not an unsigned integer up to 2^53 - 1, or null"))), + } +} + +fn add(a: u64, b: u64) -> Result { + a.checked_add(b).ok_or(SimError::Overflow) +} + +fn mul(a: u64, b: u64) -> Result { + a.checked_mul(b).ok_or(SimError::Overflow) +} + +/// Refuse members a section does not define, so a misspelt member is not ignored. +fn only(v: &Value, members: &[&str], at: &str) -> Result<(), SimError> { + let o = v.as_object().ok_or_else(|| invalid(format!("{at}: not an object")))?; + for k in o.keys() { + if !members.contains(&k.as_str()) { + return Err(invalid(format!("{at}: unknown member `{k}`"))); + } + } + for k in members { + if !o.contains_key(*k) { + return Err(invalid(format!("{at}: missing member `{k}`"))); + } + } + Ok(()) +} + +impl Model { + /// A model from its JSON (`wai-sim-model/1`). + pub fn from_json(v: &Value) -> Result { + only(v, &["format", "unit_us", "objects"], "model")?; + if v["format"] != MODEL_FORMAT { + return Err(invalid(format!("model: format is not {MODEL_FORMAT}"))); + } + let mut objects = Vec::new(); + for (i, o) in v["objects"].as_array().ok_or_else(|| invalid("model: no objects"))?.iter().enumerate() { + let at = format!("model object {i}"); + only(o, &["id", "base", "layer", "units", "bytes", "capture_us"], &at)?; + let units = o["units"].as_array().filter(|a| a.len() == 2).ok_or_else(|| invalid(format!("{at}: units is not [from, count]")))?; + objects.push(SimObject { + id: o["id"].as_str().ok_or_else(|| invalid(format!("{at}: id")))?.to_string(), + base: u32::try_from(u64_of(o, "base", &at)?).map_err(|_| invalid(format!("{at}: base")))?, + layer: u8::try_from(u64_of(o, "layer", &at)?).map_err(|_| invalid(format!("{at}: layer")))?, + from: units[0].as_u64().ok_or_else(|| invalid(format!("{at}: units")))?, + count: units[1].as_u64().ok_or_else(|| invalid(format!("{at}: units")))?, + bytes: u64_of(o, "bytes", &at)?, + capture_us: opt_u64(o, "capture_us", &at)?, + }); + } + let m = Model { unit_us: u64_of(v, "unit_us", "model")?, objects }; + m.check()?; + Ok(m) + } + + /// The model as JSON. + pub fn to_json(&self) -> Value { + json!({ + "format": MODEL_FORMAT, + "unit_us": self.unit_us, + "objects": self.objects.iter().map(|o| json!({ + "id": o.id, "base": o.base, "layer": o.layer, "units": [o.from, o.count], "bytes": o.bytes, "capture_us": o.capture_us, + })).collect::>(), + }) + } + + /// The model of an evaluated title: its objects in title order, with their counted + /// bytes, and each base's capture time on every object of the base. + pub fn from_evaluation(e: &Evaluation) -> Model { + let objects = e + .objects + .iter() + .map(|o| SimObject { + id: o.path.rsplit('/').next().unwrap_or(&o.path).to_string(), + base: o.base as u32, + layer: o.layer, + from: o.from, + count: o.count, + bytes: o.counted.total, + capture_us: e.bases.get(o.base).and_then(|b| b.1), + }) + .collect(); + Model { unit_us: e.unit_us, objects } + } + + /// The same model with every refinement removed: what §10.4's stall is measured + /// against. + pub fn base_only(&self) -> Model { + Model { unit_us: self.unit_us, objects: self.objects.iter().filter(|o| o.layer == 0).cloned().collect() } + } + + /// A model is well formed when every object has bytes and units; no two objects + /// cover one unit at one layer; and every unit a layer `k ≥ 1` covers is covered at + /// layer `k − 1` by an object of the same base. + pub fn check(&self) -> Result<(), SimError> { + if self.unit_us == 0 { + return Err(invalid("model: unit_us is 0")); + } + if self.objects.is_empty() { + return Err(invalid("model: no objects")); + } + let mut covered: std::collections::BTreeMap<(u64, u8), u32> = std::collections::BTreeMap::new(); + for o in &self.objects { + if o.bytes == 0 || o.count == 0 { + return Err(invalid(format!("{}: no bytes or no units", o.id))); + } + let end = o.from.checked_add(o.count).ok_or_else(|| invalid(format!("{}: units past u64", o.id)))?; + if o.count > 1 << 24 { + return Err(invalid(format!("{}: too many units", o.id))); + } + for u in o.from..end { + if covered.insert((u, o.layer), o.base).is_some() { + return Err(invalid(format!("{}: unit {u} is covered twice at layer {}", o.id, o.layer))); + } + } + } + for o in &self.objects { + if o.layer >= 1 { + for u in o.from..o.from + o.count { + if covered.get(&(u, o.layer - 1)) != Some(&o.base) { + return Err(invalid(format!("{}: unit {u} has no layer {} of its base below it", o.id, o.layer - 1))); + } + } + } + } + Ok(()) + } +} + +/// Delivery on demand or live (Appendix B, *Availability*). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Mode { + Vod, + Live, +} + +/// The order the sender puts objects on the link. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Schedule { + /// By layer, then first unit, then object index. + StrictPriority, + /// By availability, then object index. + Fifo, +} + +/// How objects are carried, which fixes what a delivery timeout does (§10.2). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Carriage { + /// Each object one datagram, dropped when not started by its deadline. + Datagram, + /// Each (base, layer) a subgroup, reset at the deadline of an object not fully sent. + Subgroup, +} + +/// The order the sink's workers take ready decode jobs. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DecodeOrder { + /// By layer, then first unit, then object index. + BaseFirst, + /// By the time the job became ready, then object index. + Fifo, +} + +/// What the sink does with a refinement decoded after its unit's deadline. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LateRefinement { + /// Not presented; the unit stays at its stage. Counted missed. + Discard, + /// Presented, and counted missed. + Present, +} + +/// A `wai-sim/1` plan. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Plan { + pub mode: Mode, + pub rtt_us: u64, + pub bytes_per_ms: u64, + pub chunk_bytes: u64, + pub schedule: Schedule, + pub carriage: Carriage, + pub layer_cap: Option, + pub timeout_base_us: Option, + pub timeout_refinement_us: Option, + pub vod_window_us: Option, + pub workers: u32, + pub decode_order: DecodeOrder, + pub decode_base_us: u64, + pub decode_refinement_us: u64, + pub playout_start_us: Option, + pub late_refinement: LateRefinement, +} + +fn token<'a>(v: &'a Value, k: &str, at: &str) -> Result<&'a str, SimError> { + v[k].as_str().ok_or_else(|| invalid(format!("{at}: `{k}` is not a string"))) +} + +impl Plan { + /// A plan from its JSON (Appendix B). Every member is required, and a member the + /// plan does not define is refused. + pub fn from_json(v: &Value) -> Result { + only(v, &["plan", "mode", "link", "sender", "sink"], "plan")?; + if v["plan"] != PLAN_FORMAT { + return Err(invalid(format!("plan: `plan` is not {PLAN_FORMAT}"))); + } + let (l, s, k) = (&v["link"], &v["sender"], &v["sink"]); + only(l, &["rtt_us", "bytes_per_ms", "chunk_bytes"], "link")?; + only(s, &["schedule", "carriage", "layer_cap", "delivery_timeout_us", "vod_window_us"], "sender")?; + only(&s["delivery_timeout_us"], &["base", "refinement"], "sender.delivery_timeout_us")?; + only(k, &["workers", "decode_order", "decode_us", "playout_start_us", "late_refinement"], "sink")?; + only(&k["decode_us"], &["base", "refinement"], "sink.decode_us")?; + let p = Plan { + mode: match token(v, "mode", "plan")? { + "vod" => Mode::Vod, + "live" => Mode::Live, + x => return Err(invalid(format!("plan: mode `{x}`"))), + }, + rtt_us: u64_of(l, "rtt_us", "link")?, + bytes_per_ms: u64_of(l, "bytes_per_ms", "link")?, + chunk_bytes: u64_of(l, "chunk_bytes", "link")?, + schedule: match token(s, "schedule", "sender")? { + "strict-priority" => Schedule::StrictPriority, + "fifo" => Schedule::Fifo, + x => return Err(invalid(format!("sender: schedule `{x}`"))), + }, + carriage: match token(s, "carriage", "sender")? { + "datagram" => Carriage::Datagram, + "subgroup" => Carriage::Subgroup, + x => return Err(invalid(format!("sender: carriage `{x}`"))), + }, + layer_cap: opt_u64(s, "layer_cap", "sender")?.map(|c| u8::try_from(c).map_err(|_| invalid("sender: layer_cap past 255"))).transpose()?, + timeout_base_us: opt_u64(&s["delivery_timeout_us"], "base", "sender.delivery_timeout_us")?, + timeout_refinement_us: opt_u64(&s["delivery_timeout_us"], "refinement", "sender.delivery_timeout_us")?, + vod_window_us: opt_u64(s, "vod_window_us", "sender")?, + workers: u32::try_from(u64_of(k, "workers", "sink")?).map_err(|_| invalid("sink: workers"))?, + decode_order: match token(k, "decode_order", "sink")? { + "base-first" => DecodeOrder::BaseFirst, + "fifo" => DecodeOrder::Fifo, + x => return Err(invalid(format!("sink: decode_order `{x}`"))), + }, + decode_base_us: u64_of(&k["decode_us"], "base", "sink.decode_us")?, + decode_refinement_us: u64_of(&k["decode_us"], "refinement", "sink.decode_us")?, + playout_start_us: opt_u64(k, "playout_start_us", "sink")?, + late_refinement: match token(k, "late_refinement", "sink")? { + "discard" => LateRefinement::Discard, + "present" => LateRefinement::Present, + x => return Err(invalid(format!("sink: late_refinement `{x}`"))), + }, + }; + if p.bytes_per_ms == 0 || p.chunk_bytes == 0 || p.workers == 0 || p.decode_base_us == 0 || p.decode_refinement_us == 0 { + return Err(invalid("plan: bytes_per_ms, chunk_bytes, workers and decode times are at least 1")); + } + Ok(p) + } + + /// The plan as JSON. + pub fn to_json(&self) -> Value { + json!({ + "plan": PLAN_FORMAT, + "mode": match self.mode { Mode::Vod => "vod", Mode::Live => "live" }, + "link": {"rtt_us": self.rtt_us, "bytes_per_ms": self.bytes_per_ms, "chunk_bytes": self.chunk_bytes}, + "sender": { + "schedule": match self.schedule { Schedule::StrictPriority => "strict-priority", Schedule::Fifo => "fifo" }, + "carriage": match self.carriage { Carriage::Datagram => "datagram", Carriage::Subgroup => "subgroup" }, + "layer_cap": self.layer_cap, + "delivery_timeout_us": {"base": self.timeout_base_us, "refinement": self.timeout_refinement_us}, + "vod_window_us": self.vod_window_us, + }, + "sink": { + "workers": self.workers, + "decode_order": match self.decode_order { DecodeOrder::BaseFirst => "base-first", DecodeOrder::Fifo => "fifo" }, + "decode_us": {"base": self.decode_base_us, "refinement": self.decode_refinement_us}, + "playout_start_us": self.playout_start_us, + "late_refinement": match self.late_refinement { LateRefinement::Discard => "discard", LateRefinement::Present => "present" }, + }, + }) + } + + /// The time one chunk of `c` bytes occupies the link: `⌈c·1000 / bytes_per_ms⌉`, or + /// `None` past 64 bits. + pub fn serialise_us(&self, c: u64) -> Option { + Some(c.checked_mul(1000)?.div_ceil(self.bytes_per_ms)) + } +} + +/// What became of an object. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ObjState { + /// Every byte arrived. + Delivered, + /// Never started: its bytes were never sent. + Dropped, + /// Started and not finished by its deadline: its sent bytes are abandoned, the rest + /// dropped. + Abandoned, + /// Above the layer cap: never sent and never offered. + Capped, +} + +impl ObjState { + pub fn token(self) -> &'static str { + match self { + Self::Delivered => "delivered", + Self::Dropped => "dropped", + Self::Abandoned => "abandoned", + Self::Capped => "capped", + } + } +} + +/// One object's delivery. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ObjEvent { + pub id: String, + pub sent_first_us: Option, + pub arrived_us: Option, + pub state: ObjState, +} + +/// What a simulation gives (Appendix B, *Outputs*). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SimResult { + pub objects: Vec, + /// `(unit, layers, t_us)`, ordered by unit, then layers. + pub presentations: Vec<(u64, u16, u64)>, + pub bfpf_sched: Option, + pub tfpf_us: Option, + /// For each layer count `k + 1` the model has, unit 0's first presentation at it. + pub tffs_us: Vec>, + pub offered: u64, + pub delivered: u64, + pub dropped: u64, + pub abandoned: u64, + pub never_sent_capped: u64, + pub missed: u64, + /// For each unit, its base presentation under the plan less under the base-only + /// plan; `None` where either never presents it. + pub stall_us: Vec<(u64, Option)>, +} + +impl SimResult { + pub fn to_json(&self) -> Value { + let mut m = Map::new(); + m.insert("objects".into(), Value::Array(self.objects.iter().map(|o| json!({ + "id": o.id, "sent_first_us": o.sent_first_us, "arrived_us": o.arrived_us, "state": o.state.token(), + })).collect())); + m.insert("presentations".into(), json!(self.presentations.iter().map(|(u, l, t)| json!([u, l, t])).collect::>())); + m.insert("bfpf_sched".into(), json!(self.bfpf_sched)); + m.insert("tfpf_us".into(), json!(self.tfpf_us)); + m.insert("tffs_us".into(), json!(self.tffs_us)); + m.insert("offered".into(), json!(self.offered)); + m.insert("delivered".into(), json!(self.delivered)); + m.insert("dropped".into(), json!(self.dropped)); + m.insert("abandoned".into(), json!(self.abandoned)); + m.insert("never_sent_capped".into(), json!(self.never_sent_capped)); + m.insert("missed".into(), json!(self.missed)); + m.insert("stall_us".into(), json!(self.stall_us.iter().map(|(u, s)| json!([u, s])).collect::>())); + Value::Object(m) + } + + /// The largest stall, or `None` when a unit the base-only plan presents is never + /// presented under the plan (an unbounded stall); `Some(0)` with no unit. + pub fn max_stall(&self) -> Option { + let mut m = 0i64; + for (_, s) in &self.stall_us { + m = m.max((*s)?); + } + Some(m) + } +} + +/// The link and the sender: when each object's chunks leave and arrive. +struct Link { + state: Vec, + pending: Vec, + sent: Vec, + first: Vec>, + arrived: Vec>, + /// `(arrival time, bytes)` of every chunk. + chunks: Vec<(u64, u64)>, + dropped: u64, + abandoned: u64, +} + +fn run_link(m: &Model, p: &Plan) -> Result { + let n = m.objects.len(); + let h = p.rtt_us / 2; + let capped: Vec = m.objects.iter().map(|o| p.layer_cap.is_some_and(|c| o.layer > c)).collect(); + let mut avail = Vec::with_capacity(n); + for o in &m.objects { + avail.push(match p.mode { + Mode::Vod => add(h, p.vod_window_us.map_or(Ok(0), |w| mul(o.from, m.unit_us).map(|x| x.saturating_sub(w)))?)?, + Mode::Live => h.max(o.capture_us.ok_or_else(|| SimError::NoCaptureTime(o.id.clone()))?), + }); + } + let deadline: Vec> = m + .objects + .iter() + .zip(&avail) + .map(|(o, &a)| if o.layer == 0 { p.timeout_base_us } else { p.timeout_refinement_us }.map(|t| add(a, t)).transpose()) + .collect::>()?; + if p.carriage == Carriage::Datagram { + if let Some(o) = m.objects.iter().zip(&capped).find(|(o, c)| !**c && o.bytes > p.chunk_bytes) { + return Err(SimError::DatagramTooLarge(o.0.id.clone())); + } + } + let mut l = Link { + state: capped.iter().map(|&c| if c { ObjState::Capped } else { ObjState::Delivered }).collect(), + pending: capped.iter().map(|&c| !c).collect(), + sent: vec![0; n], + first: vec![None; n], + arrived: vec![None; n], + chunks: Vec::new(), + dropped: 0, + abandoned: 0, + }; + let sub = |i: usize| (m.objects[i].base, m.objects[i].layer); + let mut t = 0u64; + loop { + // Timeouts at this boundary: every object still pending whose deadline is past. + for i in 0..n { + let Some(d) = deadline[i] else { continue }; + if !l.pending[i] || d >= t { + continue; + } + match p.carriage { + Carriage::Datagram => { + // A datagram is one chunk, so a pending one has not started. + l.pending[i] = false; + l.state[i] = ObjState::Dropped; + l.dropped = add(l.dropped, m.objects[i].bytes)?; + } + Carriage::Subgroup => { + // Reset the subgroup from here: this and every later pending object. + for j in i..n { + if l.pending[j] && sub(j) == sub(i) { + l.pending[j] = false; + l.abandoned = add(l.abandoned, l.sent[j])?; + l.dropped = add(l.dropped, m.objects[j].bytes - l.sent[j])?; + l.state[j] = if l.sent[j] > 0 { ObjState::Abandoned } else { ObjState::Dropped }; + } + } + } + } + } + let eligible = |i: usize| { + l.pending[i] && avail[i] <= t && (p.carriage == Carriage::Datagram || (0..i).all(|j| !(l.pending[j] && sub(j) == sub(i)))) + }; + // Bases go in the order they became available under either schedule, so the + // base-only model sends them in the same order (§10.4's proof). + let pick = (0..n).filter(|&i| eligible(i)).min_by_key(|&i| match p.schedule { + Schedule::StrictPriority => { + let o = &m.objects[i]; + (o.layer as u64, if o.layer == 0 { avail[i] } else { 0 }, o.from, i as u64) + } + Schedule::Fifo => (avail[i], i as u64, 0, 0), + }); + let Some(i) = pick else { + // Idle: wait for the next object to become available, or stop. + match (0..n).filter(|&i| l.pending[i] && avail[i] > t).map(|i| avail[i]).min() { + Some(next) => { + t = next; + continue; + } + None if (0..n).any(|i| l.pending[i]) => return Err(invalid("the sender can send nothing and nothing is coming")), + None => break, + } + }; + let c = p.chunk_bytes.min(m.objects[i].bytes - l.sent[i]); + if l.sent[i] == 0 { + l.first[i] = Some(t); + } + let end = add(t, p.serialise_us(c).ok_or(SimError::Overflow)?)?; + let arrives = add(end, h)?; + l.sent[i] += c; + l.chunks.push((arrives, c)); + if l.sent[i] == m.objects[i].bytes { + let on_time = p.carriage == Carriage::Datagram || deadline[i].is_none_or(|d| end <= d); + if on_time { + l.pending[i] = false; + l.state[i] = ObjState::Delivered; + l.arrived[i] = Some(arrives); + } + // Otherwise it stays pending, fully sent, and the next boundary resets it. + } + t = end; + } + Ok(l) +} + +/// Run `plan` over `model` (Appendix B). +pub fn simulate(model: &Model, plan: &Plan) -> Result { + model.check()?; + let full = simulate_once(model, plan)?; + let base = simulate_once(&model.base_only(), plan)?; + let first_base = |ps: &[(u64, u16, u64)]| -> std::collections::BTreeMap { + let mut out = std::collections::BTreeMap::new(); + for &(u, l, t) in ps { + if l == 1 { + out.entry(u).or_insert(t); + } + } + out + }; + let (fb, bb) = (first_base(&full.presentations), first_base(&base.presentations)); + let mut units: Vec = model.objects.iter().filter(|o| o.layer == 0).flat_map(|o| o.from..o.from + o.count).collect(); + units.sort_unstable(); + let stall_us = units + .into_iter() + .map(|u| { + Ok((u, match (fb.get(&u), bb.get(&u)) { + (Some(&a), Some(&b)) => Some(i64::try_from(a).map_err(|_| SimError::Overflow)? - i64::try_from(b).map_err(|_| SimError::Overflow)?), + _ => None, + })) + }) + .collect::>()?; + Ok(SimResult { stall_us, ..full }) +} + +fn simulate_once(m: &Model, p: &Plan) -> Result { + let l = run_link(m, p)?; + let n = m.objects.len(); + // The sink: workers take ready decode jobs, and a job's completion presents every + // unit it covers. + let mut stage: std::collections::BTreeMap = std::collections::BTreeMap::new(); // unit → (stage, when) + let mut done = vec![false; n]; + let mut running: Vec<(u64, usize)> = Vec::new(); + let mut presentations = Vec::new(); + let mut missed = 0u64; + let jobs: Vec = (0..n).filter(|&i| l.state[i] == ObjState::Delivered).collect(); + let mut t = 0u64; + let ready_at = |i: usize, stage: &std::collections::BTreeMap| -> Option { + let o = &m.objects[i]; + let mut r = l.arrived[i]?; + if o.layer >= 1 { + for u in o.from..o.from + o.count { + match stage.get(&u) { + Some(&(s, when)) if s == o.layer - 1 => r = r.max(when), + _ => return None, + } + } + } + Some(r) + }; + loop { + // Completions at t, in object order. + running.sort_unstable(); + while let Some(&(end, i)) = running.first() { + if end > t { + break; + } + running.remove(0); + let o = &m.objects[i]; + for u in o.from..o.from + o.count { + let late = match p.playout_start_us { + Some(s) => end > add(s, mul(u, m.unit_us)?)?, + None => false, + }; + if late { + missed += 1; + } + if late && o.layer >= 1 && p.late_refinement == LateRefinement::Discard { + continue; + } + presentations.push((u, o.layer as u16 + 1, end)); + stage.insert(u, (o.layer, end)); + } + } + // Start ready jobs on free workers. + loop { + if running.len() as u32 >= p.workers { + break; + } + let pick = jobs + .iter() + .copied() + .filter(|&i| !done[i]) + .filter_map(|i| ready_at(i, &stage).filter(|&r| r <= t).map(|r| (i, r))) + // Bases in the order they became ready, which is the order they arrived, + // so the base-only model decodes them in the same order (§10.4's proof). + .min_by_key(|&(i, r)| match p.decode_order { + DecodeOrder::BaseFirst => { + let o = &m.objects[i]; + (o.layer as u64, if o.layer == 0 { r } else { 0 }, o.from, i as u64) + } + DecodeOrder::Fifo => (r, i as u64, 0, 0), + }); + let Some((i, _)) = pick else { break }; + done[i] = true; + let d = if m.objects[i].layer == 0 { p.decode_base_us } else { p.decode_refinement_us }; + running.push((add(t, d)?, i)); + } + // The next event: a completion, or a job becoming ready by its arrival. + let next_end = running.iter().map(|r| r.0).min(); + let next_ready = jobs.iter().copied().filter(|&i| !done[i]).filter_map(|i| ready_at(i, &stage)).filter(|&r| r > t).min(); + match next_end.into_iter().chain(next_ready).min() { + Some(nt) => t = nt, + None => break, + } + } + presentations.sort_unstable(); + let first_unit0 = presentations.iter().filter(|p| p.0 == 0 && p.1 == 1).map(|p| p.2).min(); + let top = m.objects.iter().map(|o| o.layer).max().unwrap_or(0); + let tffs_us = (0..=top as u16).map(|k| presentations.iter().filter(|p| p.0 == 0 && p.1 == k + 1).map(|p| p.2).min()).collect(); + fn sum(it: impl Iterator) -> Result { + it.into_iter().try_fold(0u64, add) + } + let bfpf_sched = first_unit0.map(|t0| sum(l.chunks.iter().filter(|c| c.0 <= t0).map(|c| c.1))).transpose()?; + let offered = sum(m.objects.iter().zip(&l.state).filter(|(_, s)| **s != ObjState::Capped).map(|(o, _)| o.bytes))?; + let delivered = sum(m.objects.iter().zip(&l.state).filter(|(_, s)| **s == ObjState::Delivered).map(|(o, _)| o.bytes))?; + let never_sent_capped = sum(m.objects.iter().zip(&l.state).filter(|(_, s)| **s == ObjState::Capped).map(|(o, _)| o.bytes))?; + let objects = (0..n) + .map(|i| ObjEvent { id: m.objects[i].id.clone(), sent_first_us: l.first[i], arrived_us: l.arrived[i], state: l.state[i] }) + .collect(); + Ok(SimResult { + objects, + presentations, + bfpf_sched, + tfpf_us: first_unit0, + tffs_us, + offered, + delivered, + dropped: l.dropped, + abandoned: l.abandoned, + never_sent_capped, + missed, + stall_us: Vec::new(), + }) +} + +/// §10.4's bound on how far a plan's refinements may delay a base's presentation: +/// `S_link + S_decode`, or `None` where either part is unbounded. +/// +/// - `S_link`: one chunk's serialisation under `strict-priority`; under `fifo` with a +/// refinement timeout `T`, `T` plus one chunk; otherwise unbounded. +/// - `S_decode`: under `base-first` decode, one refinement's decode time; otherwise +/// unbounded. +/// +/// A plan with a base timeout has no bound: a base the refinements delay past its own +/// timeout is not presented at all. §10.4 proves the bound for every other plan with a +/// bounded link schedule and `base-first` decode, given that bases are sent and decoded +/// in the order they became available, as [`simulate`] sends and decodes them. +pub fn stall_bound(p: &Plan) -> Option { + if p.timeout_base_us.is_some() { + return None; + } + let chunk = p.serialise_us(p.chunk_bytes)?; + let link = match (p.schedule, p.timeout_refinement_us) { + (Schedule::StrictPriority, _) => chunk, + (Schedule::Fifo, Some(t)) => t.checked_add(chunk)?, + (Schedule::Fifo, None) => return None, + }; + let decode = match p.decode_order { + DecodeOrder::BaseFirst => p.decode_refinement_us, + DecodeOrder::Fifo => return None, + }; + // A bound past 64 bits is stated as none. + link.checked_add(decode) +} + +/// A plan whose stall passes its bound. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct StallViolation { + pub unit: u64, + /// `None`: the base-only plan presents the unit's base, and the plan never does. + pub stall_us: Option, + pub bound_us: u64, +} + +/// Check `plan` over `model` against its bound; `Ok` for a plan with none. +pub fn check_stall(model: &Model, plan: &Plan) -> Result<(), StallViolation> { + let Some(bound) = stall_bound(plan) else { return Ok(()) }; + let r = simulate(model, plan).map_err(|_| StallViolation { unit: 0, stall_us: None, bound_us: bound })?; + for &(unit, s) in &r.stall_us { + if s.is_none_or(|s| s > bound as i64) { + return Err(StallViolation { unit, stall_us: s, bound_us: bound }); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests; diff --git a/wai/wai-rs/src/stage_measure/sim/tests.rs b/wai/wai-rs/src/stage_measure/sim/tests.rs new file mode 100644 index 000000000..66537ecb5 --- /dev/null +++ b/wai/wai-rs/src/stage_measure/sim/tests.rs @@ -0,0 +1,576 @@ +//! The simulator against hand-computed timelines, its conservation and cap rules, and +//! §10.4's stall bound on generated plans, with a counter-plan for each part of the +//! bound a schedule can lose. + +use super::*; + +/// SplitMix64, for generated models and plans. +struct Rng(u64); + +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn range(&mut self, lo: u64, hi: u64) -> u64 { + lo + self.next() % (hi - lo + 1) + } + fn coin(&mut self) -> bool { + self.next() & 1 == 1 + } +} + +fn obj(id: &str, base: u32, layer: u8, from: u64, count: u64, bytes: u64, capture_us: Option) -> SimObject { + SimObject { id: id.into(), base, layer, from, count, bytes, capture_us } +} + +/// `bases` groups of `per` units, each a base and `layers` refinement layers, each +/// layer one object per unit or one for the group; capture at the end of each group. +fn model(r: &mut Rng, bases: u32, per: u64, layers: u8, per_unit: bool) -> Model { + let unit_us = 40_000; + let mut objects = Vec::new(); + for b in 0..bases { + let from = b as u64 * per; + let cap = Some((from + per) * unit_us); + objects.push(obj(&format!("b{b}"), b, 0, from, per, r.range(1500, 9000), cap)); + for k in 1..=layers { + if per_unit { + for u in 0..per { + objects.push(obj(&format!("b{b}l{k}u{u}"), b, k, from + u, 1, r.range(200, 2500), cap)); + } + } else { + objects.push(obj(&format!("b{b}l{k}"), b, k, from, per, r.range(800, 7000), cap)); + } + } + } + Model { unit_us, objects } +} + +/// A model drawn with nothing in order: groups take their unit ranges in a random +/// order, capture times are random (not rising with the units), object sizes span +/// tiny to large, and the objects are listed in a random order. +fn wild_model(r: &mut Rng) -> Model { + let unit_us = r.range(1, 80_000); + let bases = r.range(1, 5) as u32; + let per = r.range(1, 6); + let layers = r.range(0, 3) as u8; + let mut slots: Vec = (0..bases as u64).collect(); + for i in (1..slots.len()).rev() { + slots.swap(i, r.range(0, i as u64) as usize); + } + let mut objects = Vec::new(); + for b in 0..bases { + let from = slots[b as usize] * per; + let cap = Some(r.range(0, 600_000)); + let size = |r: &mut Rng| if r.coin() { r.range(1, 400) } else { r.range(400, 60_000) }; + objects.push(obj(&format!("b{b}"), b, 0, from, per, size(r), cap)); + for k in 1..=layers { + if r.coin() { + for u in 0..per { + objects.push(obj(&format!("b{b}l{k}u{u}"), b, k, from + u, 1, size(r), cap)); + } + } else { + objects.push(obj(&format!("b{b}l{k}"), b, k, from, per, size(r), cap)); + } + } + } + for i in (1..objects.len()).rev() { + objects.swap(i, r.range(0, i as u64) as usize); + } + Model { unit_us, objects } +} + +/// A bounded plan drawn over the whole range: either bounded link schedule (a `fifo` +/// link always with a refinement timeout), either carriage, any window, live or on +/// demand, one to four workers, decode times from 1 µs, slow and fast links. +fn wild_plan(r: &mut Rng, m: &Model) -> Plan { + let schedule = if r.coin() { Schedule::Fifo } else { Schedule::StrictPriority }; + let carriage = if r.coin() { Carriage::Datagram } else { Carriage::Subgroup }; + let largest = m.objects.iter().map(|o| o.bytes).max().unwrap_or(1); + let timeout = |r: &mut Rng| match r.range(0, 3) { + 0 => 0, + 1 => r.range(1, 2000), + _ => r.range(2000, 400_000), + }; + Plan { + mode: if r.coin() { Mode::Vod } else { Mode::Live }, + rtt_us: if r.coin() { 0 } else { r.range(1, 200_000) }, + bytes_per_ms: r.range(1, 20_000), + chunk_bytes: if carriage == Carriage::Datagram { largest + r.range(0, 2000) } else { r.range(1, 3000) }, + schedule, + carriage, + layer_cap: if r.range(0, 4) == 0 { Some(r.range(0, 2) as u8) } else { None }, + timeout_base_us: None, + timeout_refinement_us: if schedule == Schedule::Fifo || r.coin() { Some(timeout(r)) } else { None }, + vod_window_us: if r.coin() { Some(r.range(0, 300_000)) } else { None }, + workers: r.range(1, 4) as u32, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: r.range(1, 20_000), + decode_refinement_us: r.range(1, 10_000), + playout_start_us: if r.coin() { Some(r.range(0, 400_000)) } else { None }, + late_refinement: if r.coin() { LateRefinement::Discard } else { LateRefinement::Present }, + } +} + +fn plan(r: &mut Rng, schedule: Schedule, carriage: Carriage, timeout_ref: Option, decode_order: DecodeOrder) -> Plan { + Plan { + mode: if r.coin() { Mode::Vod } else { Mode::Live }, + rtt_us: r.range(0, 120_000), + bytes_per_ms: r.range(150, 6000), + chunk_bytes: if carriage == Carriage::Datagram { 10_000 } else { r.range(300, 1500) }, + schedule, + carriage, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: timeout_ref, + vod_window_us: if r.coin() { Some(r.range(0, 200_000)) } else { None }, + workers: r.range(1, 3) as u32, + decode_order, + decode_base_us: r.range(1000, 9000), + decode_refinement_us: r.range(300, 5000), + playout_start_us: if r.coin() { Some(r.range(0, 400_000)) } else { None }, + late_refinement: if r.coin() { LateRefinement::Discard } else { LateRefinement::Present }, + } +} + +fn conserved(r: &SimResult) { + assert_eq!(r.offered, r.delivered + r.dropped + r.abandoned, "offered = delivered + dropped + abandoned"); +} + +#[test] +fn a_hand_computed_timeline() { + // One unit: a 2500-byte base and a 1250-byte layer; h = 20 ms, 2500 bytes/ms, + // 1250-byte chunks (500 µs each). Base chunks leave 20.0–20.5 and 20.5–21.0 ms and + // arrive 20 ms later; the layer's chunk leaves 21.0–21.5 ms. The base decodes 41.0– + // 44.0 ms, the layer 44.0–45.5 ms. + let m = Model { unit_us: 40_000, objects: vec![obj("b", 0, 0, 0, 1, 2500, None), obj("l", 0, 1, 0, 1, 1250, None)] }; + let p = Plan { + mode: Mode::Vod, + rtt_us: 40_000, + bytes_per_ms: 2500, + chunk_bytes: 1250, + schedule: Schedule::StrictPriority, + carriage: Carriage::Subgroup, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: None, + vod_window_us: None, + workers: 1, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: 3000, + decode_refinement_us: 1500, + playout_start_us: None, + late_refinement: LateRefinement::Discard, + }; + let r = simulate(&m, &p).unwrap(); + assert_eq!(r.objects[0], ObjEvent { id: "b".into(), sent_first_us: Some(20_000), arrived_us: Some(41_000), state: ObjState::Delivered }); + assert_eq!(r.objects[1], ObjEvent { id: "l".into(), sent_first_us: Some(21_000), arrived_us: Some(41_500), state: ObjState::Delivered }); + assert_eq!(r.presentations, vec![(0, 1, 44_000), (0, 2, 45_500)]); + assert_eq!((r.tfpf_us, r.tffs_us.clone(), r.bfpf_sched), (Some(44_000), vec![Some(44_000), Some(45_500)], Some(3750))); + assert_eq!((r.offered, r.delivered, r.dropped, r.abandoned, r.missed), (3750, 3750, 0, 0, 0)); + assert_eq!(r.stall_us, vec![(0, Some(0))]); + // With playout at 44.5 ms the layer is late: discarded and missed, or presented and missed. + let late = Plan { playout_start_us: Some(44_500), ..p.clone() }; + let r = simulate(&m, &late).unwrap(); + assert_eq!((r.presentations.clone(), r.missed), (vec![(0, 1, 44_000)], 1)); + let r = simulate(&m, &Plan { late_refinement: LateRefinement::Present, ..late }).unwrap(); + assert_eq!((r.presentations.clone(), r.missed), (vec![(0, 1, 44_000), (0, 2, 45_500)], 1)); + // A 400 µs refinement timeout: the layer's deadline is 20.4 ms, and at the next chunk + // boundary (20.5 ms) it has not started, so its subgroup is reset and its bytes + // dropped; as a datagram, not started by its deadline, it is dropped too. + let t = Plan { timeout_refinement_us: Some(400), ..p.clone() }; + let r = simulate(&m, &t).unwrap(); + assert_eq!((r.objects[1].state, r.objects[1].sent_first_us, r.abandoned, r.dropped, r.presentations.len()), (ObjState::Dropped, None, 0, 1250, 1)); + let dg = Plan { carriage: Carriage::Datagram, chunk_bytes: 2500, ..t }; + let r = simulate(&m, &dg).unwrap(); + assert_eq!((r.objects[1].state, r.objects[1].sent_first_us, r.dropped), (ObjState::Dropped, None, 1250)); + conserved(&r); + // A 2500-byte layer and a 1200 µs timeout (deadline 21.2 ms): its first chunk leaves + // 21.0–21.5 ms, and at 21.5 ms the subgroup is reset: 1250 bytes abandoned, 1250 + // dropped, nothing presented above the base. + let m2 = Model { unit_us: 40_000, objects: vec![obj("b", 0, 0, 0, 1, 2500, None), obj("l", 0, 1, 0, 1, 2500, None)] }; + let r = simulate(&m2, &Plan { timeout_refinement_us: Some(1200), ..p.clone() }).unwrap(); + assert_eq!((r.objects[1].state, r.objects[1].sent_first_us, r.abandoned, r.dropped), (ObjState::Abandoned, Some(21_000), 1250, 1250)); + assert_eq!(r.presentations, vec![(0, 1, 44_000)]); + conserved(&r); + // A reset takes the subgroup's tail with it: two units' layers in one subgroup, + // the second available 40 ms later (a window of 0) with a deadline far off, is + // dropped when the first misses its deadline at 21.5 ms. + let m3 = Model { + unit_us: 40_000, + objects: vec![obj("b", 0, 0, 0, 2, 2500, None), obj("l0", 0, 1, 0, 1, 2500, None), obj("l1", 0, 1, 1, 1, 1250, None)], + }; + let r = simulate(&m3, &Plan { timeout_refinement_us: Some(1200), vod_window_us: Some(0), ..p.clone() }).unwrap(); + assert_eq!((r.objects[1].state, r.objects[2].state, r.objects[2].sent_first_us), (ObjState::Abandoned, ObjState::Dropped, None)); + assert_eq!((r.abandoned, r.dropped), (1250, 2500)); +} + +#[test] +fn strict_priority_bound() { + let mut r = Rng(1); + for i in 0..12 { + let m = model(&mut r, 3, 4, 2, i % 2 == 0); + let carriage = if r.coin() { Carriage::Datagram } else { Carriage::Subgroup }; + let t = if r.coin() { Some(r.range(0, 300_000)) } else { None }; + let p = plan(&mut r, Schedule::StrictPriority, carriage, t, DecodeOrder::BaseFirst); + assert!(stall_bound(&p).is_some()); + conserved(&simulate(&m, &p).unwrap()); + check_stall(&m, &p).unwrap_or_else(|v| panic!("plan {i}: {v:?} {p:?}")); + } +} + +#[test] +fn fifo_timeout_bound() { + let mut r = Rng(2); + for carriage in [Carriage::Datagram, Carriage::Subgroup] { + for i in 0..12 { + let m = model(&mut r, 3, 4, 2, i % 2 == 1); + let t = r.range(0, 300_000); + let p = plan(&mut r, Schedule::Fifo, carriage, Some(t), DecodeOrder::BaseFirst); + conserved(&simulate(&m, &p).unwrap()); + check_stall(&m, &p).unwrap_or_else(|v| panic!("{carriage:?} plan {i}: {v:?} {p:?}")); + } + } +} + +#[test] +fn bound_holds_on_mixed_plans() { + // Groups, units per group and layers drawn too, under either bounded schedule and + // either carriage: the stall stays within §10.4's bound and the bytes are conserved. + for seed in 0..300u64 { + let mut r = Rng(1000 + seed); + let (nb, per, nl, per_unit) = (r.range(1, 4) as u32, r.range(1, 6), r.range(1, 3) as u8, r.coin()); + let m = model(&mut r, nb, per, nl, per_unit); + let schedule = if r.coin() { Schedule::Fifo } else { Schedule::StrictPriority }; + let carriage = if r.coin() { Carriage::Datagram } else { Carriage::Subgroup }; + let t = if schedule == Schedule::Fifo || r.coin() { Some(r.range(0, 300_000)) } else { None }; + let p = plan(&mut r, schedule, carriage, t, DecodeOrder::BaseFirst); + conserved(&simulate(&m, &p).unwrap()); + check_stall(&m, &p).unwrap_or_else(|v| panic!("seed {seed}: {v:?} {p:?}")); + } +} + +/// §10.4's bound, checked on models and plans drawn over the whole range +/// ([`wild_model`], [`wild_plan`]). An exploratory run of the same draw over far more +/// seeds is reported with the change that added it; this is its bounded part. +#[test] +fn bound_holds_on_wild_plans() { + for seed in 0..2000u64 { + let mut r = Rng(0x5eed_0000 + seed); + let m = wild_model(&mut r); + let p = wild_plan(&mut r, &m); + assert!(stall_bound(&p).is_some()); + let res = simulate(&m, &p).unwrap_or_else(|e| panic!("seed {seed}: {e}")); + conserved(&res); + check_stall(&m, &p).unwrap_or_else(|v| panic!("seed {seed}: {v:?}\n{p:?}\n{}", m.to_json())); + } +} + +/// A model and plan drawn toward the cases where send and decode order matter: capture +/// times on, just before and just after chunk boundaries, repeated and out of order; +/// a base split into two layer-0 objects with their own captures; objects listed in any +/// order; chunks of one byte to a datagram's size; slow links; zero, one-microsecond and +/// chunk-length timeouts; one to four workers. Every plan is inside §10.4's +/// preconditions. +fn targeted(r: &mut Rng) -> (Model, Plan) { + fn pick(r: &mut Rng, v: &[T]) -> T { + v[r.range(0, v.len() as u64 - 1) as usize] + } + let chunk = pick(r, &[1u64, 2, 3, 7, 1200]); + let bpm = pick(r, &[1u64, 3, 1000, 2500, 7000]); + let c = (chunk * 1000).div_ceil(bpm); + let unit_us = pick(r, &[1u64, 7, c, c + 1, 40_000]); + let datagram = r.range(0, 2) == 0; + let size = |r: &mut Rng| -> u64 { + if datagram { + pick(r, &[1, chunk.max(2) - 1, chunk]).max(1) + } else { + pick(r, &[1, chunk.max(2) - 1, chunk, chunk + 1, 2 * chunk + 1, 5 * chunk]).max(1) + } + }; + let live = r.coin(); + let caps = [0, 1, c.saturating_sub(1), c, c + 1, 2 * c, 2 * c + 1, 3 * c + 2, 7 * c]; + let nb = r.range(1, 4) as u32; + let per = r.range(1, 3); + let mut objects = Vec::new(); + for b in 0..nb { + let from = b as u64 * per; + let cap0 = live.then(|| pick(r, &caps)); + if per >= 2 && r.range(0, 2) == 0 { + let split = r.range(1, per - 1); + objects.push(obj(&format!("b{b}a"), b, 0, from, split, size(r), cap0)); + let cap1 = live.then(|| pick(r, &caps)); + objects.push(obj(&format!("b{b}b"), b, 0, from + split, per - split, size(r), cap1)); + } else { + objects.push(obj(&format!("b{b}"), b, 0, from, per, size(r), cap0)); + } + for k in 1..=r.range(0, 3) as u8 { + if r.coin() { + for u in 0..per { + let cap = live.then(|| pick(r, &caps)); + objects.push(obj(&format!("b{b}l{k}u{u}"), b, k, from + u, 1, size(r), cap)); + } + } else { + let cap = live.then(|| pick(r, &caps)); + objects.push(obj(&format!("b{b}l{k}"), b, k, from, per, size(r), cap)); + } + } + } + for i in (1..objects.len()).rev() { + objects.swap(i, r.range(0, i as u64) as usize); + } + let strict = r.coin(); + let plan = Plan { + mode: if live { Mode::Live } else { Mode::Vod }, + rtt_us: pick(r, &[0, 1, 2, 7, 40_000]), + bytes_per_ms: bpm, + chunk_bytes: chunk, + schedule: if strict { Schedule::StrictPriority } else { Schedule::Fifo }, + carriage: if datagram { Carriage::Datagram } else { Carriage::Subgroup }, + layer_cap: pick(r, &[None, Some(0), Some(1), Some(2)]), + timeout_base_us: None, + timeout_refinement_us: if strict { + pick(r, &[None, Some(0), Some(1), Some(c), Some(3 * c)]) + } else { + Some(pick(r, &[0, 1, c.saturating_sub(1), c, c + 1, 3 * c, 10 * c])) + }, + vod_window_us: if live { None } else { pick(r, &[None, Some(0), Some(1), Some(unit_us / 2), Some(unit_us * 2), Some(unit_us * 100)]) }, + workers: r.range(1, 4) as u32, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: pick(r, &[1, c.max(1), 10 * c.max(1), 1000, 3]), + decode_refinement_us: pick(r, &[1, c.saturating_sub(1).max(1), c.max(1), 5 * c.max(1), 2]), + playout_start_us: pick(r, &[None, Some(0), Some(c), Some(unit_us)]), + late_refinement: if r.coin() { LateRefinement::Discard } else { LateRefinement::Present }, + }; + (Model { unit_us, objects }, plan) +} + +/// §10.4's bound on [`targeted`] draws, which reach the orders the bound depends on: with +/// the send or decode order of bases reverted to listing order, this test fails on its +/// own, without [`bases_keep_their_order`]'s two models. +#[test] +fn bound_holds_on_targeted_plans() { + let mut r = Rng(0x7a76_e7ed); + for i in 0..TARGETED_DRAWS { + let (m, p) = targeted(&mut r); + assert!(stall_bound(&p).is_some()); + let res = simulate(&m, &p).unwrap_or_else(|e| panic!("draw {i}: {e}\n{p:?}\n{}", m.to_json())); + conserved(&res); + check_stall(&m, &p).unwrap_or_else(|v| panic!("draw {i}: {v:?}\n{p:?}\n{}", m.to_json())); + } +} + +const TARGETED_DRAWS: u64 = 20_000; + +/// The two models a review found past the bound before bases were sent and decoded in +/// the order they became available: a later-captured base ahead of an earlier one under +/// strict priority, and a base whose decode a smaller one overtook under base-first. +#[test] +fn bases_keep_their_order() { + let a = Model { + unit_us: 40_000, + objects: vec![ + obj("B0", 0, 0, 0, 1, 10, Some(0)), + obj("R0", 0, 1, 0, 1, 1000, Some(0)), + obj("Bb", 2, 0, 2, 1, 10, Some(15)), + obj("Ba", 1, 0, 1, 1, 50_000, Some(20)), + ], + }; + let pa = Plan { + mode: Mode::Live, + rtt_us: 0, + bytes_per_ms: 1000, + chunk_bytes: 1000, + schedule: Schedule::StrictPriority, + carriage: Carriage::Subgroup, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: None, + vod_window_us: None, + workers: 1, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: 100, + decode_refinement_us: 100, + playout_start_us: None, + late_refinement: LateRefinement::Present, + }; + check_stall(&a, &pa).unwrap(); + let b = Model { + unit_us: 40_000, + objects: vec![ + obj("B0", 0, 0, 0, 1, 10, None), + obj("R0", 0, 1, 0, 1, 10, None), + obj("Bb", 2, 0, 2, 1, 100_000, None), + obj("Ba", 1, 0, 1, 1, 10, None), + ], + }; + let pb = Plan { mode: Mode::Vod, schedule: Schedule::Fifo, timeout_refinement_us: Some(20), decode_base_us: 100_000, ..pa }; + check_stall(&b, &pb).unwrap(); +} + +/// The counter-plans' model: three groups of five units, two layers per unit, all +/// available at once, on a slow link. +fn counter_model() -> Model { + let mut objects = Vec::new(); + for b in 0..3u32 { + let from = b as u64 * 5; + objects.push(obj(&format!("b{b}"), b, 0, from, 5, 4000, None)); + for k in 1..=2u8 { + for u in 0..5 { + objects.push(obj(&format!("b{b}l{k}u{u}"), b, k, from + u, 1, 1800, None)); + } + } + } + Model { unit_us: 40_000, objects } +} + +fn counter_plan(schedule: Schedule, timeout_ref: Option, decode_order: DecodeOrder) -> Plan { + Plan { + mode: Mode::Vod, + rtt_us: 40_000, + bytes_per_ms: 1000, + chunk_bytes: 1200, + schedule, + carriage: Carriage::Subgroup, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: timeout_ref, + vod_window_us: None, + workers: 1, + decode_order, + decode_base_us: 4000, + decode_refinement_us: 3000, + playout_start_us: None, + late_refinement: LateRefinement::Discard, + } +} + +#[test] +fn fifo_no_timeout_violates() { + // FIFO without a timeout sends every layer of a group before the next group's base: + // its stall passes the bound of the same plan under strict priority. + let m = counter_model(); + let fifo = counter_plan(Schedule::Fifo, None, DecodeOrder::BaseFirst); + let strict = counter_plan(Schedule::StrictPriority, None, DecodeOrder::BaseFirst); + assert_eq!(stall_bound(&fifo), None); + let bound = stall_bound(&strict).unwrap(); + check_stall(&m, &strict).unwrap(); + let worst = simulate(&m, &fifo).unwrap().max_stall().unwrap(); + assert!(worst > bound as i64, "fifo with no timeout stalls {worst} µs, within the strict bound {bound}"); + // With a timeout it is bounded again. + let timed = counter_plan(Schedule::Fifo, Some(30_000), DecodeOrder::BaseFirst); + check_stall(&m, &timed).unwrap(); +} + +#[test] +fn fifo_decode_violates() { + // One worker taking jobs in ready order decodes a group's layers before the next + // base: its stall passes the bound base-first decode would have. + // The next group becomes available 20 ms after the first (a 180 ms window over + // 200 ms groups), while the first group's layers still wait to be decoded. + let m = counter_model(); + let fifo = Plan { vod_window_us: Some(180_000), ..counter_plan(Schedule::StrictPriority, None, DecodeOrder::Fifo) }; + let base_first = Plan { decode_order: DecodeOrder::BaseFirst, ..fifo.clone() }; + check_stall(&m, &base_first).unwrap(); + assert_eq!(stall_bound(&fifo), None); + let worst = simulate(&m, &fifo).unwrap().max_stall().unwrap(); + assert!(worst > stall_bound(&base_first).unwrap() as i64, "fifo decode stalls {worst} µs"); +} + +#[test] +fn conservation_and_layer_cap() { + let mut r = Rng(3); + for i in 0..24 { + let m = model(&mut r, 2, 3, 3, i % 3 == 0); + let schedule = if r.coin() { Schedule::Fifo } else { Schedule::StrictPriority }; + let carriage = if r.coin() { Carriage::Datagram } else { Carriage::Subgroup }; + let t = if r.coin() { Some(r.range(0, 100_000)) } else { None }; + let mut p = plan(&mut r, schedule, carriage, t, DecodeOrder::BaseFirst); + if r.coin() { + p.timeout_base_us = Some(r.range(0, 200_000)); + } + let cap = (i % 4) as u8; + p.layer_cap = Some(cap); + let res = simulate(&m, &p).unwrap(); + conserved(&res); + let capped: u64 = m.objects.iter().filter(|o| o.layer > cap).map(|o| o.bytes).sum(); + assert_eq!(res.never_sent_capped, capped); + for (o, e) in m.objects.iter().zip(&res.objects) { + assert_eq!(o.layer > cap, e.state == ObjState::Capped, "{}", o.id); + if o.layer > cap { + assert_eq!(e.sent_first_us, None, "{}: a capped object is never sent", o.id); + } + } + assert!(res.presentations.iter().all(|p| p.1 <= cap as u16 + 1), "no presentation above the cap"); + } +} + +#[test] +fn malformed_models_and_plans_are_refused() { + let m = counter_model(); + let p = counter_plan(Schedule::StrictPriority, None, DecodeOrder::BaseFirst); + // A datagram larger than a chunk; a live plan with no capture time. + assert!(matches!(simulate(&m, &Plan { carriage: Carriage::Datagram, ..p.clone() }), Err(SimError::DatagramTooLarge(_)))); + assert!(matches!(simulate(&m, &Plan { mode: Mode::Live, ..p.clone() }), Err(SimError::NoCaptureTime(_)))); + // A unit covered twice at one layer; a layer with nothing below it. + let mut twice = m.clone(); + twice.objects.push(obj("x", 0, 1, 0, 1, 10, None)); + assert!(matches!(simulate(&twice, &p), Err(SimError::Invalid(_)))); + let mut gap = m.clone(); + gap.objects.push(obj("y", 0, 4, 0, 1, 10, None)); + assert!(matches!(simulate(&gap, &p), Err(SimError::Invalid(_)))); + // The JSON forms round-trip, and an unknown member is refused. + assert_eq!(Plan::from_json(&p.to_json()).unwrap(), p); + assert_eq!(Model::from_json(&m.to_json()).unwrap(), m); + let mut v = p.to_json(); + v["sink"]["extra"] = json!(1); + assert!(Plan::from_json(&v).is_err()); +} + +/// A model and plan whose times and byte sums pass 64 bits are refused, the same in +/// every build; a figure past `2⁵³ − 1` is refused when read. +#[test] +fn sums_past_64_bits_are_refused() { + let big = Model { unit_us: 40_000, objects: vec![obj("B0", 0, 0, 0, 1, u64::MAX / 2, None), obj("B1", 1, 0, 1, 1, u64::MAX / 2 + 2, None)] }; + let p = Plan { + mode: Mode::Vod, + rtt_us: 0, + bytes_per_ms: 1, + chunk_bytes: u64::MAX, + schedule: Schedule::StrictPriority, + carriage: Carriage::Subgroup, + layer_cap: None, + timeout_base_us: None, + timeout_refinement_us: None, + vod_window_us: None, + workers: 1, + decode_order: DecodeOrder::BaseFirst, + decode_base_us: 1, + decode_refinement_us: 1, + playout_start_us: None, + late_refinement: LateRefinement::Present, + }; + assert_eq!(simulate(&big, &p), Err(SimError::Overflow)); + // Each chunk's time fits; the link's clock after two of them does not. + let three = Model { + unit_us: 40_000, + objects: (0..3).map(|b| obj(&format!("B{b}"), b, 0, b as u64, 1, 1 << 53, None)).collect(), + }; + assert_eq!(simulate(&three, &Plan { chunk_bytes: 1 << 53, ..p.clone() }), Err(SimError::Overflow)); + // A decode that would end past 64 bits. + let small = Model { unit_us: 40_000, objects: vec![obj("B0", 0, 0, 0, 1, 10, None)] }; + assert_eq!(simulate(&small, &Plan { chunk_bytes: 1000, bytes_per_ms: 1000, decode_base_us: u64::MAX, ..p.clone() }), Err(SimError::Overflow)); + assert_eq!(stall_bound(&p), None, "a bound past 64 bits is none"); + // The review's model, read from its JSON: its byte counts pass 2^53 - 1. + let o = serde_json::json!({"format": "wai-sim-model/1", "unit_us": 40000, "objects": [ + {"id": "B0", "base": 0, "layer": 0, "units": [0, 1], "bytes": 18446744073709551u64, "capture_us": null}, + {"id": "B1", "base": 1, "layer": 0, "units": [1, 1], "bytes": 18446744073709552u64, "capture_us": null}]}); + assert!(matches!(Model::from_json(&o), Err(SimError::Invalid(_)))); + // At the largest figures a JSON number holds, the sums still fit or are refused. + let edge = Model { unit_us: MAX_FIGURE, objects: vec![obj("B0", 0, 0, MAX_FIGURE - 1, 1, MAX_FIGURE, None)] }; + let q = Plan { chunk_bytes: MAX_FIGURE, rtt_us: MAX_FIGURE, vod_window_us: Some(0), ..p }; + assert!(matches!(simulate(&edge, &q), Err(SimError::Overflow))); +} diff --git a/wai/wai-rs/src/stage_measure/title.rs b/wai/wai-rs/src/stage_measure/title.rs new file mode 100644 index 000000000..80b1d679d --- /dev/null +++ b/wai/wai-rs/src/stage_measure/title.rs @@ -0,0 +1,380 @@ +//! A measured title (`extensions/staged-measurement.md` Appendix D): a staged set and +//! what a measurement needs beside it. +//! +//! The stage structure is the staged set's own: each base is a staged-conformance base +//! (its envelope, the parameter set it decodes against, and its stage-0 output), and +//! each layer's `refines` names its base, its producers, its layer and its span +//! (`extensions/staged-delivery.md` §3). `title.json` adds only what that model does +//! not carry: the units' duration and capture times (for the delivery simulator), the +//! source the units approximate, the single-layer references of the layering overhead, +//! and the reconstruction digests the fixture tool wrote from its own encoder. +//! +//! A base may also be named by its envelope alone (with its parameter set): such a base +//! has no stage-0 output or case facts to check its decode against, and the corpus uses +//! the form only for titles a tool must refuse. A title may lower the limits it is +//! measured under (`limits`), never raise them: each figure is the lower of the title's +//! and the tool's (`staged::TOOL_DEFAULT_LIMIT` and `staged::TOOL_DEFAULT_CHAIN_BUDGET`, +//! or the limit the tool is given). A title also may state the code its evaluation must +//! be refused with (`refused`). +//! +//! Every path is relative to the `wai/` directory, so a title reads the staged corpus's +//! bases where they are. [`Title::from_files`] reads from a map of path → bytes and +//! needs no file system (a browser fetches the files); it reports the first file it +//! needs and does not hold ([`MeasureError::Missing`]), so a caller can fetch it and +//! call again. [`Title::load`] does that from a directory. + +use std::collections::BTreeMap; + +use serde_json::Value; + +use super::MeasureError; + +/// The `format` of a `title.json`. +pub const FORMAT: &str = "wai-measure-title/1"; + +/// An object of the title: its path below `wai/` and its bytes. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Obj { + pub path: String, + pub bytes: Vec, +} + +/// A file of a base's parameter set: the member name its decode reads it as, its path, +/// its bytes. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SetFile { + pub member: String, + pub path: String, + pub bytes: Vec, +} + +/// What a staged-conformance case states of how its base decodes: the capability it +/// was decoded through, whether that was the fallback, its tier, and whether its pin +/// resolved. A measurement's decode must state the same. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CaseFacts { + pub decoded_via: String, + pub via_fallback: bool, + pub tier: String, + pub pin_resolved: bool, +} + +/// A base of the title, as its staged-conformance case states it, or by its envelope +/// alone. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TitleBase { + pub obj: Obj, + /// The global index of the base's unit 0. + pub first_unit: u64, + /// When the base's content is captured, for a live delivery plan; `None` for a + /// title delivered on demand. + pub capture_us: Option, + pub set: Vec, + /// The case's stage-0 output: the units' canonical buffers in unit order; `None` + /// for a base named by its envelope alone. + pub stage0: Option>, + /// The case's statement of the decode; `None` for a base named by its envelope. + pub case: Option, +} + +/// The limits a title is measured under: the cost limit its bases are decoded under +/// (and whose output, symbols and working bytes its layers are held to, +/// staged-delivery Appendix A.5), and its chains' budget (§4.5). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Limits { + pub cost: crate::int_payload::DeclaredCost, + pub chain_budget: u64, +} + +impl Default for Limits { + /// The tools' stated limits. + fn default() -> Self { + Limits { cost: crate::staged::TOOL_DEFAULT_LIMIT, chain_budget: crate::staged::TOOL_DEFAULT_CHAIN_BUDGET } + } +} + +/// What a title states of its limits: each one it lowers, `None` where it states none. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct StatedLimits { + pub cost: Option, + pub chain_budget: Option, +} + +impl Limits { + /// The limits a title is measured under by a tool whose own limits are `tool`: each + /// figure the lower of the title's and the tool's. A title lowers a limit and never + /// raises one; a limit past the tool's takes the tool's own (`--limit`), because + /// whoever writes a title is not whoever runs the tool. + pub fn within(stated: &StatedLimits, tool: &Limits) -> Limits { + let t = &tool.cost; + let cost = stated.cost.map_or(*t, |s| { + crate::int_payload::DeclaredCost::new( + s.output_bytes.min(t.output_bytes), + s.symbols.min(t.symbols), + s.working_bytes.min(t.working_bytes), + s.macs.min(t.macs), + ) + }); + Limits { cost, chain_budget: stated.chain_budget.map_or(tool.chain_budget, |b| b.min(tool.chain_budget)) } + } +} + +/// A single-layer reference (§6.2): the title's bases and the layers given here. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Reference { + pub label: String, + pub layers: Vec, +} + +/// A measured title. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Title { + pub name: String, + /// The fixture directory, below `wai/`. + pub dir: String, + pub about: String, + /// The duration of one unit. + pub unit_us: u64, + pub bases: Vec, + /// The layers, in the order the title lists them. + pub layers: Vec, + /// The source the units approximate: each unit's canonical buffer at its final + /// shape, in global unit order. + pub source: Option, + pub references: Vec, + /// The fixture tool's own reconstruction digests (`expect.json`). + pub expect: Option, + /// What the title states of its limits. + pub stated: StatedLimits, + /// The limits it is measured under: [`Limits::within`] the tools' stated limits, + /// until [`Title::under`] gives a tool's own. + pub limits: Limits, + /// The code the title's evaluation must be refused with, for a title that exists + /// to be refused. + pub refused: Option, +} + +fn invalid(s: impl Into) -> MeasureError { + MeasureError::Invalid(s.into()) +} + +fn get<'a>(files: &'a BTreeMap>, path: &str) -> Result<&'a [u8], MeasureError> { + files.get(path).map(Vec::as_slice).ok_or_else(|| MeasureError::Missing(path.to_string())) +} + +fn json(files: &BTreeMap>, path: &str) -> Result { + serde_json::from_slice(get(files, path)?).map_err(|e| invalid(format!("{path}: {e}"))) +} + +fn str_of<'a>(v: &'a Value, k: &str, at: &str) -> Result<&'a str, MeasureError> { + v[k].as_str().ok_or_else(|| invalid(format!("{at}: `{k}` is not a string"))) +} + +/// A file name inside a directory: one path segment, no `..`. +fn name<'a>(v: &'a Value, at: &str) -> Result<&'a str, MeasureError> { + let s = v.as_str().ok_or_else(|| invalid(format!("{at}: a file name is not a string")))?; + if s.is_empty() || s.contains('/') || s.contains('\\') || s == "." || s == ".." { + return Err(invalid(format!("{at}: `{s}` is not a file name"))); + } + Ok(s) +} + +/// A path below `wai/`: relative, `/`-separated, no `.` or `..` segment. +fn wai_path<'a>(v: &'a Value, at: &str) -> Result<&'a str, MeasureError> { + let s = v.as_str().ok_or_else(|| invalid(format!("{at}: a path is not a string")))?; + if s.is_empty() || s.starts_with('/') || s.contains('\\') || s.split('/').any(|p| p.is_empty() || p == "." || p == "..") { + return Err(invalid(format!("{at}: `{s}` is not a path below wai/"))); + } + Ok(s) +} + +fn hex(b: &[u8]) -> String { + b.iter().map(|x| format!("{x:02x}")).collect() +} + +impl Title { + /// Measure the title under a tool's own limits `tool`: each figure the lower of the + /// title's and the tool's ([`Limits::within`]). + pub fn under(&mut self, tool: &Limits) { + self.limits = Limits::within(&self.stated, tool); + } + + /// The title in the fixture directory `dir` (below `wai/`), from `files` (path + /// below `wai/` → bytes). + pub fn from_files(dir: &str, files: &BTreeMap>) -> Result { + let at = format!("{dir}/title.json"); + let t = json(files, &at)?; + if t["format"] != FORMAT { + return Err(invalid(format!("{at}: format is not {FORMAT}"))); + } + let unit_us = t["unit_us"].as_u64().filter(|&u| u >= 1).ok_or_else(|| invalid(format!("{at}: unit_us is not an integer ≥ 1")))?; + let local = |v: &Value, what: &str| -> Result { + let path = format!("{dir}/{}", name(v, &format!("{at}: {what}"))?); + Ok(Obj { bytes: get(files, &path)?.to_vec(), path }) + }; + let mut bases = Vec::new(); + for (i, b) in t["bases"].as_array().ok_or_else(|| invalid(format!("{at}: no bases")))?.iter().enumerate() { + let first_unit = b["first_unit"].as_u64().ok_or_else(|| invalid(format!("{at}: base {i}: first_unit")))?; + let capture_us = match &b["capture_us"] { + Value::Null => None, + v => Some(v.as_u64().ok_or_else(|| invalid(format!("{at}: base {i}: capture_us")))?), + }; + let read_set = |set: &Value, at: &str| -> Result, MeasureError> { + let mut out = Vec::new(); + for (member, p) in set.as_object().ok_or_else(|| invalid(format!("{at}: no decode set")))? { + let sp = wai_path(p, at)?; + out.push(SetFile { member: member.clone(), path: sp.to_string(), bytes: get(files, sp)?.to_vec() }); + } + Ok(out) + }; + if !b["envelope"].is_null() { + // A base by its envelope alone. + let path = wai_path(&b["envelope"], &format!("{at}: base {i}"))?.to_string(); + let set = read_set(&b["set"], &format!("{at}: base {i}"))?; + bases.push(TitleBase { obj: Obj { bytes: get(files, &path)?.to_vec(), path }, first_unit, capture_us, set, stage0: None, case: None }); + continue; + } + let case = wai_path(&b["case"], &format!("{at}: base {i}"))?; + let cpath = format!("{case}/case.json"); + let c = json(files, &cpath)?; + let k = b["base"].as_u64().ok_or_else(|| invalid(format!("{at}: base {i}: `base` is not an index")))? as usize; + let cb = c["bases"].get(k).ok_or_else(|| invalid(format!("{cpath}: no base {k}")))?; + let path = format!("{case}/{}", name(&cb["file"], &cpath)?); + let bytes = get(files, &path)?.to_vec(); + if hex(&crate::jwp_hash::content_hash(&bytes)) != str_of(cb, "object", &cpath)? { + return Err(invalid(format!("{path}: not the content hash {cpath} states"))); + } + if cb["decode"]["kind"] != "int" { + return Err(invalid(format!("{cpath}: base {k} is not decoded by an integer decoder"))); + } + let set = read_set(&cb["decode"]["set"], &cpath)?; + let s0 = format!("{case}/{}", name(&cb["stage0"], &cpath)?); + let stage0 = get(files, &s0)?.to_vec(); + if crate::pin::sha256_hex(&stage0) != str_of(cb, "stage0_sha256", &cpath)? { + return Err(invalid(format!("{s0}: not the SHA-256 {cpath} states"))); + } + let facts = CaseFacts { + decoded_via: str_of(cb, "decoded_via", &cpath)?.to_string(), + via_fallback: cb["via_fallback"].as_bool().ok_or_else(|| invalid(format!("{cpath}: via_fallback")))?, + tier: str_of(cb, "tier", &cpath)?.to_string(), + pin_resolved: cb["pin_resolved"].as_bool().ok_or_else(|| invalid(format!("{cpath}: pin_resolved")))?, + }; + bases.push(TitleBase { obj: Obj { path, bytes }, first_unit, capture_us, set, stage0: Some(stage0), case: Some(facts) }); + } + if bases.is_empty() { + return Err(invalid(format!("{at}: no bases"))); + } + let layers = t["layers"] + .as_array() + .ok_or_else(|| invalid(format!("{at}: `layers` is not an array")))? + .iter() + .map(|v| local(v, "layers")) + .collect::, _>>()?; + let source = match &t["source"] { + Value::Null => None, + s => { + let o = local(&s["file"], "source")?; + if crate::pin::sha256_hex(&o.bytes) != str_of(s, "sha256", &at)? { + return Err(invalid(format!("{}: not the SHA-256 {at} states", o.path))); + } + Some(o) + } + }; + let mut references = Vec::new(); + for r in t["references"].as_array().map(Vec::as_slice).unwrap_or_default() { + let label = str_of(r, "label", &at)?.to_string(); + let layers = r["layers"] + .as_array() + .ok_or_else(|| invalid(format!("{at}: reference `{label}` has no layers")))? + .iter() + .map(|v| local(v, "references")) + .collect::, _>>()?; + references.push(Reference { label, layers }); + } + let expect = match &t["expect"] { + Value::Null => None, + v => Some(json(files, &format!("{dir}/{}", name(v, &at)?))?), + }; + let stated = match &t["limits"] { + Value::Null => StatedLimits::default(), + l => { + let cost = match &l["cost"] { + Value::Null => None, + c => { + let f: Vec = c.as_array().filter(|a| a.len() == 4).and_then(|a| a.iter().map(Value::as_u64).collect()).ok_or_else(|| invalid(format!("{at}: limits.cost is not four integers")))?; + Some(crate::int_payload::DeclaredCost::new(f[0], f[1], f[2], f[3] as u128)) + } + }; + let chain_budget = match &l["chain_budget"] { + Value::Null => None, + v => Some(v.as_u64().ok_or_else(|| invalid(format!("{at}: limits.chain_budget")))?), + }; + StatedLimits { cost, chain_budget } + } + }; + let limits = Limits::within(&stated, &Limits::default()); + let refused = match &t["refused"] { + Value::Null => None, + v => Some(v.as_str().ok_or_else(|| invalid(format!("{at}: refused is not a string")))?.to_string()), + }; + Ok(Title { + stated, + limits, + refused, + name: str_of(&t, "title", &at)?.to_string(), + dir: dir.to_string(), + about: t["about"].as_str().unwrap_or_default().to_string(), + unit_us, + bases, + layers, + source, + references, + expect, + }) + } + + /// The title in `dir` (below `wai_root`), reading each file it needs from disk. + #[cfg(not(target_arch = "wasm32"))] + pub fn load(wai_root: &std::path::Path, dir: &str) -> Result { + let mut files: BTreeMap> = BTreeMap::new(); + loop { + match Title::from_files(dir, &files) { + Err(MeasureError::Missing(p)) => { + let b = std::fs::read(wai_root.join(&p)).map_err(|e| invalid(format!("{p}: {e}")))?; + files.insert(p, b); + } + other => return other, + } + } + } + + /// The title's content identity: BLAKE3 over the measured content, the paths and + /// bytes of its objects (bases, layers and the references' layers), its bases' + /// parameter-set files and its source, in path order, each path and each file + /// length-prefixed. Labels, verification aids (`stage0`, `expect.json`) and + /// `title.json` itself are not part of it. + pub fn content_id(&self) -> [u8; 32] { + let mut all: BTreeMap<&str, &[u8]> = BTreeMap::new(); + for b in &self.bases { + all.insert(&b.obj.path, &b.obj.bytes); + for s in &b.set { + all.insert(&s.path, &s.bytes); + } + } + for o in self.layers.iter().chain(self.references.iter().flat_map(|r| &r.layers)) { + all.insert(&o.path, &o.bytes); + } + if let Some(s) = &self.source { + all.insert(&s.path, &s.bytes); + } + let mut h = blake3::Hasher::new(); + h.update(b"wai:measure-title\x01"); + for (p, b) in all { + h.update(&(p.len() as u64).to_be_bytes()); + h.update(p.as_bytes()); + h.update(&(b.len() as u64).to_be_bytes()); + h.update(b); + } + *h.finalize().as_bytes() + } +} diff --git a/wai/wai-rs/src/staged/mod.rs b/wai/wai-rs/src/staged/mod.rs index 298af3a08..14efd0379 100644 --- a/wai/wai-rs/src/staged/mod.rs +++ b/wai/wai-rs/src/staged/mod.rs @@ -112,6 +112,52 @@ impl std::fmt::Display for BaseError { impl std::error::Error for BaseError {} +/// The cost limit the reference tools (`wai stage`, `wai_stage_measure`) apply to a +/// base by default ([`decode_base`]'s `limit`; `extensions/staged-measurement.md` +/// Appendix D). Each figure is chosen to admit any base a tool is asked to measure and to +/// refuse a payload whose header declares an outsized decode before anything is +/// allocated for it; a tool's `--limit OUT,SYMBOLS,WORKING,MACS` replaces it. +/// +/// - **Output, 32 MiB (2²⁵ bytes):** five 1920 × 1080 RGB8 frames, or more than three +/// minutes of 48 kHz stereo PCM: more than one stage set's base carries in practice, +/// and well under the bound vectors of `tests/vectors/staged_bounds` (64 MiB and +/// 1 GiB), which a tool must refuse. +/// - **Symbols, 2²⁵:** one entropy-decode step per output byte at the output limit, the +/// most an integer payload's symbols per output byte reach (`WIV1` with keyframe 0). +/// - **Working bytes, 256 MiB (2²⁸):** eight bytes per network activation for networks +/// whose largest layers hold a few times the output limit's elements. +/// - **Multiply-accumulates, 2³⁸:** a learned decode of several megapixels on the +/// reference kernels. +/// +/// cbindgen:ignore +pub const TOOL_DEFAULT_LIMIT: crate::int_payload::DeclaredCost = crate::int_payload::DeclaredCost::new(1 << 25, 1 << 25, 1 << 28, 1 << 38); + +/// The chain budget the reference tools give a base's chain by default (§4.5): +/// 256 MiB, eight times [`TOOL_DEFAULT_LIMIT`]'s output, for a base's units and the +/// layers retained over them. +/// +/// cbindgen:ignore +pub const TOOL_DEFAULT_CHAIN_BUDGET: u64 = 1 << 28; + +/// A cost limit given as four figures, `OUT,SYMBOLS,WORKING,MACS`: output bytes, +/// symbols, working bytes and multiply-accumulates (`int_payload::DeclaredCost`), as a +/// tool's `--limit` takes it. +pub fn parse_limit(s: &str) -> Result { + let v: Vec<&str> = s.split(',').map(str::trim).collect(); + let [o, y, w, m] = v[..] else { + return Err(format!("--limit {s}: four figures, OUT,SYMBOLS,WORKING,MACS")); + }; + let n = |x: &str| x.parse::().map_err(|e| format!("--limit {s}: {x}: {e}")); + let macs = m.parse::().map_err(|e| format!("--limit {s}: {m}: {e}"))?; + Ok(crate::int_payload::DeclaredCost::new(n(o)?, n(y)?, n(w)?, macs)) +} + +/// The layer cost limit (Appendix A.5) that matches a base cost limit: its output, +/// symbols and working bytes. +pub fn layer_limit(limit: &crate::int_payload::DeclaredCost) -> Wir1Cost { + Wir1Cost { output_bytes: limit.output_bytes, symbols: limit.symbols, working_bytes: limit.working_bytes } +} + /// Whether `cap` decodes through `int_payload::decode_int`: the capabilities SPEC §5 /// pins an integer payload format to, and `wai.neural.int_synth` and /// `wai.neural.int_mlicpp`. What a base decode advertises to SPEC §4. @@ -228,6 +274,31 @@ pub fn decode_base_via( limit: Option<&crate::int_payload::DeclaredCost>, authority: Option<&dyn crate::pin::PinAuthority>, ) -> Result<(BaseInfo, Vec), BaseError> { + decode_base_pinned(envelope, via, set, limit, authority).map(|d| (d.info, d.units)) +} + +/// A base decoded for a sink ([`decode_base_pinned`]): its info, its units, and the pin +/// its decode verified. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct BaseDecode { + pub info: BaseInfo, + pub units: Vec, + /// The envelope's pin where it resolved against the parameter set (SPEC §3.1), so + /// the decode read only bytes it verified; `None` for an unpinned decode, or a + /// capability that reads no parameter set. + pub pin: Option, +} + +/// [`decode_base_via`], also giving the pin the decode verified: what a record of the +/// decode names as its parameter set (`extensions/staged-measurement.md` Appendix C, +/// the decoder entries). +pub fn decode_base_pinned( + envelope: &[u8], + via: Option<&str>, + set: &BTreeMap, + limit: Option<&crate::int_payload::DeclaredCost>, + authority: Option<&dyn crate::pin::PinAuthority>, +) -> Result { use crate::container::Inert; use crate::int_payload::{check_cost, decode_int, resolve}; let w = Wai::from_bytes(envelope).map_err(|e| BaseError::NotWai1(e.to_string()))?; @@ -298,16 +369,16 @@ pub fn decode_base_via( Err(e) => return Err(BaseError::Decode(e)), }; let units = units_from_decoded(decoded).filter(|u| !u.is_empty()).ok_or(BaseError::NoUnits)?; - let pinned = params.is_pinned(); + let pin = if params.is_pinned() { w.prior().cloned() } else { None }; let info = BaseInfo { content_hash: crate::jwp_hash::content_hash(envelope), - tier: tier::base_tier(&cap, if pinned { w.prior() } else { None }), + tier: tier::base_tier(&cap, pin.as_ref()), via_fallback: cap != primary, capability: cap, attested: false, first_unit: 0, }; - return Ok((info, units)); + return Ok(BaseDecode { info, units, pin }); } } diff --git a/wai/wai-rs/src/telemetry/cmcd.rs b/wai/wai-rs/src/telemetry/cmcd.rs new file mode 100644 index 000000000..6302d3a42 --- /dev/null +++ b/wai/wai-rs/src/telemetry/cmcd.rs @@ -0,0 +1,219 @@ +//! An observation as a CMCD version 2 report (CTA-5004-B; `extensions/delivery-telemetry.md` +//! §3, §4): which mode reports it, which standard and custom keys carry which members, +//! and, for Request mode, the header shards. +//! +//! A report is an RFC 9651 dictionary with its keys in bytewise order and no optional +//! whitespace. A key is written only when its value is not `null`. Custom keys carry +//! Strings, which is the only custom value type CTA-5004-B admits besides Tokens, so an +//! integer member is written as its decimal digits. + +use std::collections::BTreeMap; + +use super::observation::Observation; +use super::sf::{parse_dict, serialize_dict, SfBare, SfMember}; +use super::TelemetryError; + +/// The prefix of every custom key this extension defines. +pub const PREFIX: &str = "science.transaction.wai-"; + +/// The keys of CMCD version 2: request, response and event keys (CTA-5004-B), and the +/// version 1 key `nrr`. +pub const CMCD_KEYS: &[&str] = &[ + "ab", "bg", "bl", "br", "bs", "bsa", "bsd", "bsda", "cdn", "cid", "cs", "d", "dfa", "dl", "ec", "lab", "lb", "ltc", "msd", "mtp", "nor", + "nr", "nrr", "ot", "pb", "pr", "pt", "rtp", "sf", "sid", "sn", "st", "sta", "su", "tab", "tb", "tbl", "tpb", "v", "cmsdd", "cmsds", "rc", + "smrt", "ttfb", "ttfbb", "ttlb", "url", "cen", "e", "h", "ts", +]; + +/// The CMCD header shard a Request-mode key goes to (CTA-5004-B), custom keys included. +pub fn header_shard(key: &str) -> Option<&'static str> { + Some(match key { + "ab" | "br" | "d" | "lab" | "lb" | "ot" | "tab" | "tb" | "tpb" => "CMCD-Object", + "bl" | "cs" | "dfa" | "dl" | "ltc" | "mtp" | "nor" | "nrr" | "pb" | "sn" | "sta" | "su" | "tbl" => "CMCD-Request", + "cid" | "msd" | "sf" | "sid" | "st" | "v" => "CMCD-Session", + "bg" | "bs" | "bsa" | "bsd" | "bsda" | "cdn" | "ec" | "nr" | "pr" | "pt" | "rtp" => "CMCD-Status", + _ => { + let x = key.strip_prefix(PREFIX)?; + match x { + "b" | "tx" | "ch" | "cap" | "ri" | "l" | "lt" | "p" | "rcpt" => "CMCD-Object", + "lc" => "CMCD-Request", + "coh" => "CMCD-Session", + _ => return None, + } + } + }) +} + +/// Whether `k` has the form of a CMCD custom key: a lowercase first letter, then +/// lowercase letters, digits, `.` and `-`, with a hyphen that is neither first nor last. +pub fn is_custom_key(k: &str) -> bool { + let b = k.as_bytes(); + // An inner hyphen needs a character on each side, so a key has at least three. + b.len() >= 3 + && b[0].is_ascii_lowercase() + && b.iter().all(|&c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == b'.' || c == b'-') + && b[1..b.len() - 1].contains(&b'-') +} + +/// How an observation is reported. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Report { + /// Request mode: the dictionary a request carries. + Request(String), + /// Event mode: one report line. + Event(String), + /// Not reported in CMCD (a `present` that is not the session's first). + None, +} + +fn s(v: impl ToString) -> SfMember { + SfMember::Item(SfBare::Str(v.to_string())) +} + +fn int(v: u64) -> SfMember { + SfMember::Item(SfBare::Int(v as i64)) +} + +fn tok(v: &str) -> SfMember { + SfMember::Item(SfBare::Token(v.to_string())) +} + +fn path_token(p: &str) -> &'static str { + if p == "primary" { "p" } else { "f" } +} + +/// The report's members, in bytewise key order, and whether it is Event mode; `None` +/// when the observation is not reported. +pub fn members(o: &Observation) -> Result)>, TelemetryError> { + let event = o.is_event_mode(); + if o.event == "present" && !event { + return Ok(None); + } + let mut m: BTreeMap = BTreeMap::new(); + let mut put = |k: &str, v: Option| { + if let Some(v) = v { + m.insert(k.to_string(), v); + } + }; + let c = |x: &str| format!("{PREFIX}{x}"); + put("v", Some(int(2))); + put("sid", Some(s(&o.sid))); + put("sn", Some(int(o.sn))); + put("cid", o.cid.as_ref().map(s)); + put("st", Some(tok(&o.st))); + put("sf", Some(tok(&o.sf))); + put(&c("coh"), o.cohort.as_ref().map(s)); + if let Some(x) = &o.metrics { + put("msd", x.msd_ms.map(int)); + put("bsa", x.bsa.map(|n| SfMember::IntList(vec![n as i64]))); + put("bsda", x.bsda_ms.map(|n| SfMember::IntList(vec![n as i64]))); + put("dfa", x.dfa.map(int)); + } + if let Some(ob) = &o.object { + put(&c("b"), ob.counted.map(s)); + put(&c("tx"), ob.transferred.map(s)); + put(&c("ch"), ob.content_hash.as_ref().map(s)); + put(&c("cap"), ob.capability.as_ref().map(s)); + put(&c("ri"), ob.rendition.map(s)); + put(&c("l"), ob.layer.map(s)); + put(&c("lt"), ob.layers_total.map(s)); + put(&c("p"), ob.path.as_deref().map(|p| s(path_token(p)))); + put(&c("rcpt"), ob.receipt.as_ref().map(s)); + // Request keys, and response keys (Event mode, `e=rr`); the observation's + // checks keep each on its own side. + put("ot", ob.ot.as_deref().map(tok)); + put("d", ob.d_ms.map(int)); + put("dl", ob.dl_ms.map(int)); + put(&c("lc"), ob.layer_cap.map(s)); + put("rc", ob.rc.map(int)); + put("ttfb", ob.ttfb_ms.map(int)); + put("ttlb", ob.ttlb_ms.map(int)); + put("url", ob.url.as_ref().map(s)); + } + if event { + put("ts", o.ts_ms.map(int)); + let (e, sta, cen) = match o.event.as_str() { + "response" => ("rr", None, None), + "start" => ("ps", Some("s"), None), + "present" | "stall-end" => ("ps", Some("p"), None), + "stall-start" => ("ps", Some("r"), None), + "abandon" => ("ps", Some("q"), None), + "end" => ("ps", Some("e"), None), + "switch" => ("ce", None, Some("wai-switch")), + "capable" => ("ce", None, Some("wai-capable")), + "layer-change" => ("ce", None, Some("wai-layer-change")), + "frame-drop" => ("ce", None, Some("wai-frame-drop")), + other => return Err(TelemetryError::Invalid(format!("event `{other}` has no Event-mode report"))), + }; + put("e", Some(tok(e))); + put("sta", sta.map(tok)); + put("cen", cen.map(s)); + if let Some(p) = &o.present { + put(&c("u"), Some(s(p.unit))); + put(&c("pl"), Some(s(p.layers))); + put(&c("late"), p.late_ms.map(s)); + put(&c("dec"), p.decode_us.map(s)); + } + if let Some(w) = &o.switch { + put(&c("u"), Some(s(w.unit))); + put(&c("p"), Some(s(path_token(&w.to)))); + put(&c("sw"), Some(s(&w.reason))); + } + if let Some(k) = &o.capable { + put(&c("co"), Some(s(&k.outcome))); + put(&c("cr"), k.reason.as_ref().map(s)); + put(&c("cf"), k.classical_floor.as_ref().map(s)); + } + if let Some(l) = &o.layer_change { + put(&c("lc"), l.to.map(s)); + put(&c("lcf"), l.from.map(s)); + } + put(&c("clm"), o.claim.as_ref().map(s)); + } + Ok(Some((event, m.into_iter().collect()))) +} + +/// The CMCD report of an observation (§3). +pub fn cmcd(o: &Observation) -> Result { + Ok(match members(o)? { + None => Report::None, + Some((event, m)) => { + let d = serialize_dict(&m).map_err(|e| TelemetryError::Invalid(e.to_string()))?; + if event { Report::Event(d) } else { Report::Request(d) } + } + }) +} + +/// The header shards of a Request-mode report: each shard's dictionary, keys in bytewise +/// order; a shard with no key is not written. +pub fn cmcd_headers(o: &Observation) -> Result, TelemetryError> { + let Some((false, m)) = members(o)? else { + return Err(TelemetryError::Invalid("header shards carry Request-mode reports only".into())); + }; + let mut shards: BTreeMap<&'static str, Vec<(String, SfMember)>> = BTreeMap::new(); + for (k, v) in m { + let h = header_shard(&k).ok_or_else(|| TelemetryError::Invalid(format!("`{k}` has no header shard")))?; + shards.entry(h).or_default().push((k, v)); + } + shards + .into_iter() + .map(|(h, m)| serialize_dict(&m).map(|d| (h, d)).map_err(|e| TelemetryError::Invalid(e.to_string()))) + .collect() +} + +/// Read a CMCD dictionary: every key a CMCD version 2 key or a custom key, and every +/// custom key's value a String or a Token. +pub fn from_cmcd(dict: &str) -> Result, TelemetryError> { + let m = parse_dict(dict).map_err(|e| TelemetryError::Invalid(e.to_string()))?; + for (k, v) in &m { + if CMCD_KEYS.contains(&k.as_str()) { + continue; + } + if !is_custom_key(k) { + return Err(TelemetryError::Invalid(format!("`{k}` is neither a CMCD key nor a custom key"))); + } + if !matches!(v, SfMember::Item(SfBare::Str(_) | SfBare::Token(_))) { + return Err(TelemetryError::Invalid(format!("custom key `{k}` carries neither a String nor a Token"))); + } + } + Ok(m) +} diff --git a/wai/wai-rs/src/telemetry/cmsd.rs b/wai/wai-rs/src/telemetry/cmsd.rs new file mode 100644 index 000000000..af50aecd7 --- /dev/null +++ b/wai/wai-rs/src/telemetry/cmsd.rs @@ -0,0 +1,197 @@ +//! CMSD (CTA-5006) for a WAI object (`extensions/delivery-telemetry.md` §6): the +//! static description an origin or edge gives a response, and the dynamic state each +//! node adds. Keys and parameters are written in bytewise order; a key is written only +//! when its value is not `null`, and a Boolean only when true. CMSD version 1 is not +//! written (CTA-5006: a sender omits `v` for version 1). + +use serde_json::{Map, Value}; + +use super::cmcd::PREFIX; +use super::observation::OBJECT_TYPES; +use super::sf::{serialize_dict, serialize_list, SfBare, SfItem, SfMember}; +use super::TelemetryError; + +/// CMSD-Static for a WAI object (§6.1, §6.2). +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct CmsdStatic { + pub at: Option, + pub d: Option, + pub ht: Option, + pub nor: Option, + pub nrr: Option, + pub ot: Option, + pub sf: Option, + pub st: Option, + pub su: bool, + /// The object's content hash (`wai-ch`). + pub content_hash: Option, + /// Its layer and its stage set's stage count (`wai-l`, `wai-lt`). + pub layer: Option, + pub layers_total: Option, + /// A delivery receipt's link hash (`wai-rcpt`). + pub receipt: Option, + /// A deriving node's operation token (`wai-op`). + pub op: Option, +} + +/// One node's CMSD-Dynamic entry (§6.3). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CmsdDynamic { + pub server: String, + pub du: bool, + pub etp: Option, + pub mb: Option, + pub rd: Option, + pub rtt: Option, + /// Objects discarded on delivery timeout since the node's previous response to the + /// same session (`wai-drop`). + pub drop: Option, +} + +fn bad(s: impl Into) -> TelemetryError { + TelemetryError::Invalid(s.into()) +} + +fn only<'a>(v: &'a Value, members: &[&str], at: &str) -> Result<&'a Map, TelemetryError> { + let o = v.as_object().ok_or_else(|| bad(format!("{at}: not an object")))?; + for k in o.keys() { + if !members.contains(&k.as_str()) { + return Err(bad(format!("{at}: unknown member `{k}`"))); + } + } + for k in members { + if !o.contains_key(*k) { + return Err(bad(format!("{at}: missing member `{k}`"))); + } + } + Ok(o) +} + +/// An integer member: at most `10¹⁵ − 1`, which an RFC 9651 Integer holds (the custom +/// keys, which carry Strings, take the same bound). +fn oint(v: &Value, at: &str) -> Result, TelemetryError> { + match v { + Value::Null => Ok(None), + x => x.as_u64().filter(|&n| n <= super::observation::MAX_SF_INT).map(Some).ok_or_else(|| bad(format!("{at}: not an integer in 0…10¹⁵ − 1"))), + } +} + +fn ostr(v: &Value, at: &str) -> Result, TelemetryError> { + match v { + Value::Null => Ok(None), + Value::String(s) if !s.is_empty() && s.bytes().all(|c| (0x20..=0x7e).contains(&c)) => Ok(Some(s.clone())), + _ => Err(bad(format!("{at}: not a string of printable ASCII"))), + } +} + +fn otok(v: &Value, set: &[&str], at: &str) -> Result, TelemetryError> { + match v { + Value::Null => Ok(None), + Value::String(s) if set.contains(&s.as_str()) => Ok(Some(s.clone())), + _ => Err(bad(format!("{at}: not one of {set:?}"))), + } +} + +fn ohex(v: &Value, at: &str) -> Result, TelemetryError> { + match v { + Value::Null => Ok(None), + Value::String(s) if s.len() == 64 && s.bytes().all(|c| c.is_ascii_digit() || (b'a'..=b'f').contains(&c)) => Ok(Some(s.clone())), + _ => Err(bad(format!("{at}: not 64 lowercase hexadecimal digits"))), + } +} + +impl CmsdStatic { + pub fn from_json(v: &Value) -> Result { + let o = only(v, &["at", "d", "ht", "nor", "nrr", "ot", "sf", "st", "su", "content_hash", "layer", "layers_total", "receipt", "op"], "cmsd-static")?; + let op = ostr(&o["op"], "op")?; + if op.as_deref().is_some_and(|t| !super::sf::is_token(t)) { + return Err(bad("op: not a token")); + } + Ok(CmsdStatic { + at: oint(&o["at"], "at")?, + d: oint(&o["d"], "d")?, + ht: oint(&o["ht"], "ht")?, + nor: ostr(&o["nor"], "nor")?, + nrr: ostr(&o["nrr"], "nrr")?, + ot: otok(&o["ot"], OBJECT_TYPES, "ot")?, + sf: otok(&o["sf"], &["o", "h", "d"], "sf")?, + st: otok(&o["st"], &["v", "l", "ll"], "st")?, + su: o["su"].as_bool().ok_or_else(|| bad("su: not a boolean"))?, + content_hash: ohex(&o["content_hash"], "content_hash")?, + layer: oint(&o["layer"], "layer")?, + layers_total: oint(&o["layers_total"], "layers_total")?, + receipt: ohex(&o["receipt"], "receipt")?, + op, + }) + } + + /// The `CMSD-Static` field value. + pub fn encode(&self) -> Result { + let mut m: Vec<(String, SfMember)> = Vec::new(); + let mut put = |k: String, v: Option| { + if let Some(v) = v { + m.push((k, SfMember::Item(v))); + } + }; + let i = |x: Option| x.map(|n| SfBare::Int(n as i64)); + let st = |x: &Option| x.as_ref().map(|s| SfBare::Str(s.clone())); + let tk = |x: &Option| x.as_ref().map(|s| SfBare::Token(s.clone())); + let c = |x: &str| format!("{PREFIX}{x}"); + put("at".into(), i(self.at)); + put("d".into(), i(self.d)); + put("ht".into(), i(self.ht)); + put("nor".into(), st(&self.nor)); + put("nrr".into(), st(&self.nrr)); + put("ot".into(), tk(&self.ot)); + put(c("ch"), st(&self.content_hash)); + put(c("l"), self.layer.map(|n| SfBare::Str(n.to_string()))); + put(c("lt"), self.layers_total.map(|n| SfBare::Str(n.to_string()))); + put(c("op"), tk(&self.op)); + put(c("rcpt"), st(&self.receipt)); + put("sf".into(), tk(&self.sf)); + put("st".into(), tk(&self.st)); + put("su".into(), self.su.then_some(SfBare::Bool(true))); + m.sort_by(|a, b| a.0.cmp(&b.0)); + serialize_dict(&m).map_err(|e| bad(e.to_string())) + } +} + +impl CmsdDynamic { + pub fn from_json(v: &Value) -> Result { + let o = only(v, &["server", "du", "etp", "mb", "rd", "rtt", "drop"], "cmsd-dynamic")?; + Ok(CmsdDynamic { + server: ostr(&o["server"], "server")?.ok_or_else(|| bad("server: required"))?, + du: o["du"].as_bool().ok_or_else(|| bad("du: not a boolean"))?, + etp: oint(&o["etp"], "etp")?, + mb: oint(&o["mb"], "mb")?, + rd: oint(&o["rd"], "rd")?, + rtt: oint(&o["rtt"], "rtt")?, + drop: oint(&o["drop"], "drop")?, + }) + } +} + +/// The `CMSD-Dynamic` field value: one entry per node, in the order given (each node +/// adds its own entry), each node's parameters in bytewise order. +pub fn encode_dynamic(entries: &[CmsdDynamic]) -> Result { + let items: Vec = entries + .iter() + .map(|e| { + let mut params: Vec<(String, SfBare)> = Vec::new(); + if e.du { + params.push(("du".into(), SfBare::Bool(true))); + } + for (k, v) in [("etp", e.etp), ("mb", e.mb), ("rd", e.rd), ("rtt", e.rtt)] { + if let Some(v) = v { + params.push((k.into(), SfBare::Int(v as i64))); + } + } + if let Some(d) = e.drop { + params.push((format!("{PREFIX}drop"), SfBare::Str(d.to_string()))); + } + params.sort_by(|a, b| a.0.cmp(&b.0)); + SfItem { value: SfBare::Str(e.server.clone()), params } + }) + .collect(); + serialize_list(&items).map_err(|e| bad(e.to_string())) +} diff --git a/wai/wai-rs/src/telemetry/mod.rs b/wai/wai-rs/src/telemetry/mod.rs new file mode 100644 index 000000000..3cafa6721 --- /dev/null +++ b/wai/wai-rs/src/telemetry/mod.rs @@ -0,0 +1,296 @@ +//! Delivery telemetry (`extensions/delivery-telemetry.md`): unsigned, advisory +//! observations about WAI delivery, their mapping onto CMCD version 2 and CMSD as RFC +//! 9651 Structured Field Values, and the session metrics a log of them gives. +//! +//! - [`sf`]: the RFC 9651 subset the reports use; +//! - [`observation`]: an observation, read strictly and written in one order; +//! - [`cmcd`]: an observation as a CMCD report, its header shards, and reading one back; +//! - [`cmsd`]: CMSD-Static and CMSD-Dynamic for a WAI object; +//! - [`qoe`]: one session's metrics. +//! +//! An observation is never evidence of delivery, presentation, quality or energy: no +//! signature covers it, and it carries no joule figure. + +pub mod cmcd; +pub mod cmsd; +pub mod observation; +pub mod qoe; +pub mod sf; + +/// Why an observation, a report or a log is refused. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum TelemetryError { + Invalid(String), +} + +impl std::fmt::Display for TelemetryError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Invalid(s) => f.write_str(s), + } + } +} + +impl std::error::Error for TelemetryError {} + +#[cfg(test)] +mod tests { + use serde_json::{json, Value}; + + use super::cmcd::{cmcd, cmcd_headers, from_cmcd, is_custom_key, Report}; + use super::cmsd::{encode_dynamic, CmsdDynamic, CmsdStatic}; + use super::observation::Observation; + use super::qoe::session_metrics; + + const SID: &str = "0123456789abcdef0123456789abcdef"; + const H: &str = "aaf90792d6ff5318e5f306fbb000dfcbe73637e244aad1a2eff6dae537cd6b5f"; + + fn obs(event: &str, sn: u64, ts: Option, f: impl FnOnce(&mut Value)) -> Observation { + let mut v = json!({ + "v": 1, "sid": SID, "sn": sn, "ts_ms": ts, "event": event, "cid": null, "st": "v", "sf": "o", + "object": null, "present": null, "switch": null, "layer_change": null, "capable": null, "cohort": null, + "metrics": null, "claim": null, + }); + f(&mut v); + Observation::from_json(&v).unwrap_or_else(|e| panic!("{event}: {e}")) + } + + fn object(v: &mut Value, members: Value) { + let mut o = json!({ + "ot": null, "url": null, "content_hash": null, "capability": null, "rendition": null, "layer": null, + "layers_total": null, "layer_cap": null, "counted": null, "transferred": null, "d_ms": null, "dl_ms": null, + "ttfb_ms": null, "ttlb_ms": null, "rc": null, "path": null, "receipt": null, + }); + for (k, x) in members.as_object().unwrap() { + o[k] = x.clone(); + } + v["object"] = o; + } + + #[test] + fn a_request_and_its_shards() { + let o = obs("request", 3, None, |v| { + v["cid"] = json!("abc"); + v["cohort"] = json!("wai"); + object(v, json!({"ot": "v", "d_ms": 4000, "dl_ms": 1200, "counted": 1234, "content_hash": H, "path": "primary", "layer": 1, "layers_total": 3, "layer_cap": 2})); + }); + let want = format!( + "cid=\"abc\",d=4000,dl=1200,ot=v,science.transaction.wai-b=\"1234\",science.transaction.wai-ch=\"{H}\",\ + science.transaction.wai-coh=\"wai\",science.transaction.wai-l=\"1\",science.transaction.wai-lc=\"2\",\ + science.transaction.wai-lt=\"3\",science.transaction.wai-p=\"p\",sf=o,sid=\"{SID}\",sn=3,st=v,v=2" + ); + assert_eq!(cmcd(&o).unwrap(), Report::Request(want.clone())); + let h = cmcd_headers(&o).unwrap(); + assert_eq!(h["CMCD-Session"], format!("cid=\"abc\",science.transaction.wai-coh=\"wai\",sf=o,sid=\"{SID}\",st=v,v=2")); + assert_eq!(h["CMCD-Request"], "dl=1200,science.transaction.wai-lc=\"2\",sn=3"); + assert!(h["CMCD-Object"].starts_with("d=4000,ot=v,science.transaction.wai-b=\"1234\"")); + assert!(!h.contains_key("CMCD-Status")); + assert_eq!(from_cmcd(&want).unwrap().len(), 16); + } + + #[test] + fn event_reports() { + let start = obs("start", 4, Some(1_700_000_000_000), |_| {}); + assert_eq!(cmcd(&start).unwrap(), Report::Event(format!("e=ps,sf=o,sid=\"{SID}\",sn=4,st=v,sta=s,ts=1700000000000,v=2"))); + let rr = obs("response", 5, Some(9), |v| object(v, json!({"rc": 200, "ttfb_ms": 30, "ttlb_ms": 80, "url": "https://x.example/a.wai", "counted": 99}))); + assert_eq!( + cmcd(&rr).unwrap(), + Report::Event(format!("e=rr,rc=200,science.transaction.wai-b=\"99\",sf=o,sid=\"{SID}\",sn=5,st=v,ts=9,ttfb=30,ttlb=80,url=\"https://x.example/a.wai\",v=2")) + ); + let sw = obs("switch", 6, Some(10), |v| v["switch"] = json!({"unit": 3, "from": "primary", "to": "fallback", "reason": "missing-capability"})); + assert_eq!( + cmcd(&sw).unwrap(), + Report::Event(format!("cen=\"wai-switch\",e=ce,science.transaction.wai-p=\"f\",science.transaction.wai-sw=\"missing-capability\",science.transaction.wai-u=\"3\",sf=o,sid=\"{SID}\",sn=6,st=v,ts=10,v=2")) + ); + let stall = obs("stall-start", 7, Some(11), |v| v["metrics"] = json!({"msd_ms": null, "bsa": 2, "bsda_ms": 300, "dfa": 1})); + assert_eq!(cmcd(&stall).unwrap(), Report::Event(format!("bsa=(2),bsda=(300),dfa=1,e=ps,sf=o,sid=\"{SID}\",sn=7,st=v,sta=r,ts=11,v=2"))); + // Only the first presentation is reported; header shards are Request mode only. + let later = obs("present", 8, None, |v| v["present"] = json!({"unit": 4, "layers": 2, "first": false, "late_ms": null, "decode_us": 900})); + assert_eq!(cmcd(&later).unwrap(), Report::None); + assert!(cmcd_headers(&start).is_err()); + } + + #[test] + fn observations_are_read_strictly() { + let base = json!({ + "v": 1, "sid": SID, "sn": 0, "ts_ms": 1, "event": "start", "cid": null, "st": "v", "sf": "o", + "object": null, "present": null, "switch": null, "layer_change": null, "capable": null, "cohort": null, + "metrics": null, "claim": null, + }); + assert!(Observation::from_json(&base).is_ok()); + assert_eq!(Observation::from_json(&base).unwrap().to_json(), base); + let mutate = |f: &dyn Fn(&mut Value)| { + let mut v = base.clone(); + f(&mut v); + Observation::from_json(&v) + }; + for (what, f) in [ + ("an unknown member", &(|v: &mut Value| v["x"] = json!(1)) as &dyn Fn(&mut Value)), + ("a missing member", &|v: &mut Value| { v.as_object_mut().unwrap().remove("claim"); }), + ("v 2", &|v: &mut Value| v["v"] = json!(2)), + ("an uppercase sid", &|v: &mut Value| v["sid"] = json!(SID.to_uppercase())), + ("an event mode report with no time", &|v: &mut Value| v["ts_ms"] = json!(null)), + ("an integer past 2^53 − 1", &|v: &mut Value| v["sn"] = json!(1u64 << 53)), + ("a claim before the end", &|v: &mut Value| v["claim"] = json!(H)), + ("an object on a start", &|v: &mut Value| v["object"] = json!({})), + ("an unknown event", &|v: &mut Value| v["event"] = json!("pause")), + ] { + assert!(mutate(f).is_err(), "{what}"); + } + let mut req = base.clone(); + req["event"] = json!("request"); + object(&mut req, json!({"rc": 200})); + assert!(Observation::from_json(&req).is_err(), "a response key on a request"); + } + + #[test] + fn integers_a_report_cannot_carry_are_refused() { + // 10^15 is past an RFC 9651 Integer: an observation that holds it is refused, + // so every observation read is one cmcd() can write. + let at = |k: &str, x: u64| { + let mut v = obs("start", 0, Some(1), |_| {}).to_json(); + v[k] = json!(x); + Observation::from_json(&v) + }; + assert!(at("ts_ms", 999_999_999_999_999).is_ok()); + assert!(at("ts_ms", 1_000_000_000_000_000).is_err()); + assert!(at("sn", 1_000_000_000_000_000).is_err()); + // A log whose counted bytes pass 64 bits is refused, not wrapped. + let mut log = vec![obs("start", 0, Some(1), |_| {})]; + for i in 1..=2049u64 { + log.push(obs("response", i, Some(1), |v| object(v, json!({"counted": (1u64 << 53) - 1})))); + } + assert!(session_metrics(&log).is_err()); + log.truncate(2048); + assert!(session_metrics(&log).is_ok()); + } + + #[test] + fn short_keys_do_not_panic() { + // One- and two-character keys have no inner hyphen: neither CMCD keys nor + // custom keys, refused, never a panic. + for k in ["q", "x", "-", "a-", "zz"] { + assert!(!is_custom_key(k), "{k}"); + assert!(from_cmcd(k).is_err(), "{k}"); + } + assert!(from_cmcd("v=2,x=\"1\"").is_err()); + assert!(is_custom_key("a-b")); + } + + /// Every parser of the module, on inputs drawn from a fixed seed: structured-field + /// text built from the characters the grammar uses, and the corpus's JSON documents + /// with bytes changed. Each returns, `Ok` or `Err`; none panics. + #[test] + fn parsers_never_panic_on_drawn_inputs() { + struct Rng(u64); + impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn below(&mut self, n: u64) -> u64 { + self.next() % n + } + } + let mut r = Rng(0x7e1e_0001); + const PIECES: &[&str] = &[ + "v", "=", ",", ";", " ", "(", ")", "\"", "\\", "?", "?1", "?0", "-", ".", "*", "2", "0", "999999999999999", "9999999999999999", + "-999999999999999", "sid", "science.transaction.wai-b", "a-b", "x", "q", "ot", "v=2", "\t", "é", "\u{0}", "1.5", "e=ps", ":", "/", + "bsa=(1 2)", "\"s\"", "A", "~", + ]; + for _ in 0..20_000 { + let n = r.below(12) as usize; + let s: String = (0..n).map(|_| PIECES[r.below(PIECES.len() as u64) as usize]).collect(); + let _ = super::sf::parse_dict(&s); + let _ = super::sf::parse_list(&s); + let _ = from_cmcd(&s); + let _ = super::cmcd::is_custom_key(&s); + } + // JSON documents with bytes changed, read as observations and as CMSD. + let base = obs("request", 3, None, |v| object(v, json!({"ot": "v", "counted": 12, "path": "primary"}))).to_json().to_string(); + let cmsd = r#"{"at": 1, "d": 2, "ht": null, "nor": "n", "nrr": null, "ot": "v", "sf": "o", "st": "v", "su": true, "content_hash": null, "layer": 1, "layers_total": 2, "receipt": null, "op": "crop"}"#; + let dynamic = r#"{"server": "s", "du": true, "etp": 1, "mb": null, "rd": 2, "rtt": null, "drop": 3}"#; + let swaps: &[&[u8]] = &[b"0", b"1", b"-", b"\"", b"null", b"true", b"[", b"{", b"}", b"99999999999999999999", b"1e3", b"-1", b"\"x\""]; + for doc in [base.as_str(), cmsd, dynamic] { + for _ in 0..4000 { + let mut b = doc.as_bytes().to_vec(); + for _ in 0..1 + r.below(3) { + let at = r.below(b.len() as u64) as usize; + let len = (r.below(4) as usize).min(b.len() - at); + let with = swaps[r.below(swaps.len() as u64) as usize]; + b.splice(at..at + len, with.iter().copied()); + } + if let Ok(v) = serde_json::from_slice::(&b) { + let _ = Observation::from_json(&v); + let _ = CmsdStatic::from_json(&v); + let _ = CmsdDynamic::from_json(&v); + if let Ok(o) = Observation::from_json(&v) { + let _ = cmcd(&o); + let _ = cmcd_headers(&o); + } + } + } + } + } + + #[test] + fn custom_keys_and_reading_reports() { + assert!(is_custom_key("science.transaction.wai-b") && is_custom_key("com.example-x")); + assert!(!is_custom_key("wai") && !is_custom_key("-x") && !is_custom_key("x-") && !is_custom_key("Abc-d")); + assert!(from_cmcd("v=2,science.transaction.wai-b=12").is_err(), "a custom key carries a String or a Token"); + assert!(from_cmcd("v=2,zz=1").is_err(), "neither a CMCD key nor a custom key"); + assert!(from_cmcd("v=2,sid=\"s\",com.example-x=tok").is_ok()); + } + + #[test] + fn cmsd_static_and_dynamic() { + let s = CmsdStatic::from_json(&json!({ + "at": 1_700_000_000_000u64, "d": 4000, "ht": null, "nor": "next.wai", "nrr": null, "ot": "v", "sf": "o", "st": "v", + "su": true, "content_hash": H, "layer": 1, "layers_total": 3, "receipt": null, "op": "crop", + })) + .unwrap(); + assert_eq!( + s.encode().unwrap(), + format!("at=1700000000000,d=4000,nor=\"next.wai\",ot=v,science.transaction.wai-ch=\"{H}\",science.transaction.wai-l=\"1\",science.transaction.wai-lt=\"3\",science.transaction.wai-op=crop,sf=o,st=v,su") + ); + let d = [ + CmsdDynamic { server: "edge-1".into(), du: true, etp: Some(25000), mb: None, rd: Some(5), rtt: Some(40), drop: Some(2) }, + CmsdDynamic { server: "origin".into(), du: false, etp: None, mb: None, rd: Some(12), rtt: None, drop: None }, + ]; + assert_eq!(encode_dynamic(&d).unwrap(), "\"edge-1\";du;etp=25000;rd=5;rtt=40;science.transaction.wai-drop=\"2\",\"origin\";rd=12"); + } + + #[test] + fn a_session_log() { + let log = vec![ + obs("start", 0, Some(1000), |_| {}), + obs("response", 1, Some(1100), |v| object(v, json!({"counted": 500}))), + obs("present", 2, Some(1250), |v| v["present"] = json!({"unit": 0, "layers": 1, "first": true, "late_ms": null, "decode_us": null})), + obs("present", 3, Some(1300), |v| v["present"] = json!({"unit": 0, "layers": 2, "first": false, "late_ms": null, "decode_us": null})), + obs("stall-start", 4, Some(2000), |_| {}), + obs("stall-end", 5, Some(2400), |_| {}), + obs("present", 6, Some(2450), |v| v["present"] = json!({"unit": 1, "layers": 1, "first": false, "late_ms": 40, "decode_us": null})), + obs("stall-start", 7, Some(3000), |_| {}), + obs("end", 8, Some(3100), |v| v["claim"] = json!(H)), + ]; + let m = session_metrics(&log).unwrap(); + assert_eq!( + (m.started, m.vst_ms, m.exit_before_start, m.start_failed, m.stalls, m.stall_ms, m.engaged_ms, m.units, m.counted), + (true, Some(250), false, false, 2, 500, 1850, 2, 500) + ); + assert_eq!(m.claim.as_deref(), Some(H)); + // An abandon before any presentation is an exit before start; an end is a start + // failure. + let quit = vec![obs("start", 0, Some(1), |_| {}), obs("abandon", 1, Some(9), |_| {})]; + let m = session_metrics(&quit).unwrap(); + assert!(m.exit_before_start && !m.start_failed); + let fail = vec![obs("start", 0, Some(1), |_| {}), obs("end", 1, Some(9), |_| {})]; + assert!(session_metrics(&fail).unwrap().start_failed); + // Out of order, or after the end, is refused. + assert!(session_metrics(&[obs("start", 2, Some(1), |_| {}), obs("end", 1, Some(2), |_| {})]).is_err()); + assert!(session_metrics(&[obs("end", 0, Some(1), |_| {}), obs("start", 1, Some(2), |_| {})]).is_err()); + } +} diff --git a/wai/wai-rs/src/telemetry/observation.rs b/wai/wai-rs/src/telemetry/observation.rs new file mode 100644 index 000000000..073867eff --- /dev/null +++ b/wai/wai-rs/src/telemetry/observation.rs @@ -0,0 +1,458 @@ +//! An observation (`extensions/delivery-telemetry.md` §2): one unsigned, advisory +//! event about WAI delivery that a sink, edge or origin emits. +//! +//! An observation is a JSON object with exactly the members of §2.1, each `null` +//! where it does not apply. [`Observation::from_json`] refuses anything else, so two +//! implementations read one observation alike; [`Observation::to_json`] writes the +//! members in §2.1's order. + +use serde_json::{json, Map, Value}; + +use super::TelemetryError; + +/// The largest integer an observation carries: `2⁵³ − 1`. +/// +/// cbindgen:ignore +pub const MAX_INT: u64 = (1 << 53) - 1; + +/// The event kinds of §2.1. +pub const EVENTS: &[&str] = &[ + "request", "response", "start", "present", "switch", "layer-change", "stall-start", "stall-end", "frame-drop", "capable", "abandon", "end", +]; + +/// The object types a request may name (CTA-5004-B `ot`). +pub const OBJECT_TYPES: &[&str] = &["m", "a", "v", "av", "i", "c", "tt", "k", "o"]; + +/// The switch reasons of §5. +pub const SWITCH_REASONS: &[&str] = &[ + "pin-malformed", "form-not-registered", "form-unreadable", "form-required", "prior-revoked", "prior-not-authorised", + "prior-not-held", "digest-superseded", "set-malformed", "part-mismatch", "set-incomplete", "missing-capability", "module-unverified", "module-revoked", "trap", + "fuel-exhausted", "memory-exhausted", "deadline-threshold", "unit-hash-mismatch", "budget-exhausted", "recovered", +]; + +/// The dispatch outcomes a `capable` observation reports (§2.3). +pub const CAPABLE_OUTCOMES: &[&str] = &["not-reached", "missing-capability", "pin-refused", "module-refused", "below-rate", "capable"]; + +/// The SPEC §3.1 pin refusals a `pin-refused` outcome names, in the order of its checks, +/// check 5 (`prior-revoked`, `prior-not-authorised`) included. +pub const PIN_REFUSALS: &[&str] = &[ + "pin-malformed", "form-not-registered", "form-unreadable", "form-required", "prior-revoked", "prior-not-authorised", + "prior-not-held", "digest-superseded", "set-malformed", "part-mismatch", "set-incomplete", +]; + +/// The object a `request` or `response` observation is about. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct ObjectObs { + pub ot: Option, + pub url: Option, + pub content_hash: Option, + pub capability: Option, + pub rendition: Option, + pub layer: Option, + pub layers_total: Option, + pub layer_cap: Option, + pub counted: Option, + pub transferred: Option, + pub d_ms: Option, + pub dl_ms: Option, + pub ttfb_ms: Option, + pub ttlb_ms: Option, + pub rc: Option, + /// `primary` or `fallback`. + pub path: Option, + /// A receipt's link hash, 64 lowercase hexadecimal digits. + pub receipt: Option, +} + +/// A presentation. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PresentObs { + pub unit: u64, + pub layers: u64, + /// The session's first presentation. + pub first: bool, + pub late_ms: Option, + pub decode_us: Option, +} + +/// A switch between paths. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SwitchObs { + pub unit: u64, + pub from: String, + pub to: String, + pub reason: String, +} + +/// A change of `layer_cap`; `None` is no cap. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LayerChangeObs { + pub from: Option, + pub to: Option, +} + +/// A dispatch outcome. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CapableObs { + pub outcome: String, + /// For `pin-refused`, the refusal; otherwise `None`. + pub reason: Option, + /// The answer for the classical rendition's codec string: `supported` or + /// `unsupported`. + pub classical_floor: Option, +} + +/// Session metrics a report may carry. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct MetricsObs { + pub msd_ms: Option, + pub bsa: Option, + pub bsda_ms: Option, + pub dfa: Option, +} + +/// One observation. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Observation { + pub sid: String, + pub sn: u64, + pub ts_ms: Option, + pub event: String, + pub cid: Option, + pub st: String, + pub sf: String, + pub object: Option, + pub present: Option, + pub switch: Option, + pub layer_change: Option, + pub capable: Option, + pub cohort: Option, + pub metrics: Option, + pub claim: Option, +} + +const MEMBERS: &[&str] = &[ + "v", "sid", "sn", "ts_ms", "event", "cid", "st", "sf", "object", "present", "switch", "layer_change", "capable", "cohort", "metrics", "claim", +]; +const OBJECT_MEMBERS: &[&str] = &[ + "ot", "url", "content_hash", "capability", "rendition", "layer", "layers_total", "layer_cap", "counted", "transferred", "d_ms", "dl_ms", + "ttfb_ms", "ttlb_ms", "rc", "path", "receipt", +]; + +fn bad(code: &str) -> TelemetryError { + TelemetryError::Invalid(code.to_string()) +} + +fn exact<'a>(v: &'a Value, members: &[&str], at: &str) -> Result<&'a Map, TelemetryError> { + let o = v.as_object().ok_or_else(|| bad(&format!("{at}: not an object")))?; + for k in o.keys() { + if !members.contains(&k.as_str()) { + return Err(bad(&format!("{at}: unknown member `{k}`"))); + } + } + for k in members { + if !o.contains_key(*k) { + return Err(bad(&format!("{at}: missing member `{k}`"))); + } + } + Ok(o) +} + +fn int(v: &Value, at: &str) -> Result { + v.as_u64().filter(|&x| x <= MAX_INT).ok_or_else(|| bad(&format!("{at}: not an integer in 0…2⁵³ − 1"))) +} + +fn opt_int(v: &Value, at: &str) -> Result, TelemetryError> { + if v.is_null() { Ok(None) } else { int(v, at).map(Some) } +} + +/// The largest integer a report carries as an RFC 9651 Integer: `10¹⁵ − 1`. +/// +/// cbindgen:ignore +pub const MAX_SF_INT: u64 = 999_999_999_999_999; + +/// A member a report carries as an Integer (§3.3): at most [`MAX_SF_INT`], so every +/// observation read is one a report can carry. +fn sf_int(v: &Value, at: &str) -> Result { + v.as_u64().filter(|&x| x <= MAX_SF_INT).ok_or_else(|| bad(&format!("{at}: not an integer in 0…10¹⁵ − 1, which an RFC 9651 Integer holds"))) +} + +fn opt_sf_int(v: &Value, at: &str) -> Result, TelemetryError> { + if v.is_null() { Ok(None) } else { sf_int(v, at).map(Some) } +} + +/// A string of printable ASCII (what a Structured Field String carries) of at most `max` +/// characters. +fn text(v: &Value, max: usize, at: &str) -> Result { + let s = v.as_str().ok_or_else(|| bad(&format!("{at}: not a string")))?; + if s.is_empty() || s.len() > max || !s.bytes().all(|c| (0x20..=0x7e).contains(&c)) { + return Err(bad(&format!("{at}: not 1–{max} printable ASCII characters"))); + } + Ok(s.to_string()) +} + +fn opt_text(v: &Value, max: usize, at: &str) -> Result, TelemetryError> { + if v.is_null() { Ok(None) } else { text(v, max, at).map(Some) } +} + +/// `cid` (§2.1, §8): the entry object's content hash, 64 lowercase hexadecimal digits, +/// or a deployer's content id of 1–64 letters, digits, `.`, `_` and `-`, which can hold +/// no address, query or token. +fn opt_cid(v: &Value) -> Result, TelemetryError> { + match v { + Value::Null => Ok(None), + Value::String(s) if !s.is_empty() && s.len() <= 64 && s.bytes().all(|c| c.is_ascii_alphanumeric() || matches!(c, b'.' | b'_' | b'-')) => Ok(Some(s.clone())), + _ => Err(bad("cid: not a content hash or 1–64 letters, digits, `.`, `_` and `-`")), + } +} + +/// `object.url` (§2.2, §8.2): printable ASCII, at most 2048 characters, holding no `?`, +/// `#`, `@` or `\`, so no query, fragment or user information enters an observation, +/// however a URL parser would split it (some read `\` as `/`, which moves user +/// information past a check of the authority alone). A URL that holds one is refused, +/// not stripped, and a path that holds a `@` with it. +fn opt_url(v: &Value) -> Result, TelemetryError> { + let u = opt_text(v, 2048, "object.url")?; + if let Some(c) = u.as_deref().and_then(|u| u.chars().find(|c| matches!(c, '?' | '#' | '@' | '\\'))) { + return Err(bad(&format!("object.url: holds `{c}` (a query, a fragment, user information or a backslash)"))); + } + Ok(u) +} + +fn one_of(v: &Value, set: &[&str], at: &str) -> Result { + let s = v.as_str().filter(|s| set.contains(s)).ok_or_else(|| bad(&format!("{at}: not one of {set:?}")))?; + Ok(s.to_string()) +} + +fn opt_one_of(v: &Value, set: &[&str], at: &str) -> Result, TelemetryError> { + if v.is_null() { Ok(None) } else { one_of(v, set, at).map(Some) } +} + +fn is_hex(s: &str, n: usize) -> bool { + s.len() == n && s.bytes().all(|c| c.is_ascii_digit() || (b'a'..=b'f').contains(&c)) +} + +fn opt_hex64(v: &Value, at: &str) -> Result, TelemetryError> { + match v { + Value::Null => Ok(None), + Value::String(s) if is_hex(s, 64) => Ok(Some(s.clone())), + _ => Err(bad(&format!("{at}: not 64 lowercase hexadecimal digits"))), + } +} + +impl Observation { + /// Whether this observation is reported in CMCD Event mode (§3.1): every event but + /// `request`, and a `present` only when it is the session's first. + pub fn is_event_mode(&self) -> bool { + match self.event.as_str() { + "request" => false, + "present" => self.present.as_ref().is_some_and(|p| p.first), + _ => true, + } + } + + /// An observation from its JSON (§2.1). + pub fn from_json(v: &Value) -> Result { + let o = exact(v, MEMBERS, "observation")?; + if o["v"] != json!(1) { + return Err(bad("v: not 1")); + } + let sid = o["sid"].as_str().filter(|s| is_hex(s, 32)).ok_or_else(|| bad("sid: not 32 lowercase hexadecimal digits"))?.to_string(); + let event = one_of(&o["event"], EVENTS, "event")?; + let cid = opt_cid(&o["cid"])?; + let object = match &o["object"] { + Value::Null => None, + x => { + let m = exact(x, OBJECT_MEMBERS, "object")?; + let ob = ObjectObs { + ot: opt_one_of(&m["ot"], OBJECT_TYPES, "object.ot")?, + url: opt_url(&m["url"])?, + content_hash: opt_hex64(&m["content_hash"], "object.content_hash")?, + capability: opt_text(&m["capability"], 64, "object.capability")?, + rendition: opt_int(&m["rendition"], "object.rendition")?, + layer: opt_int(&m["layer"], "object.layer")?, + layers_total: opt_int(&m["layers_total"], "object.layers_total")?, + layer_cap: opt_int(&m["layer_cap"], "object.layer_cap")?, + counted: opt_int(&m["counted"], "object.counted")?, + transferred: opt_int(&m["transferred"], "object.transferred")?, + d_ms: opt_sf_int(&m["d_ms"], "object.d_ms")?, + dl_ms: opt_sf_int(&m["dl_ms"], "object.dl_ms")?, + ttfb_ms: opt_sf_int(&m["ttfb_ms"], "object.ttfb_ms")?, + ttlb_ms: opt_sf_int(&m["ttlb_ms"], "object.ttlb_ms")?, + rc: opt_sf_int(&m["rc"], "object.rc")?, + path: opt_one_of(&m["path"], &["primary", "fallback"], "object.path")?, + receipt: opt_hex64(&m["receipt"], "object.receipt")?, + }; + Some(ob) + } + }; + let present = match &o["present"] { + Value::Null => None, + x => { + let m = exact(x, &["unit", "layers", "first", "late_ms", "decode_us"], "present")?; + Some(PresentObs { + unit: int(&m["unit"], "present.unit")?, + layers: int(&m["layers"], "present.layers")?, + first: m["first"].as_bool().ok_or_else(|| bad("present.first: not a boolean"))?, + late_ms: opt_int(&m["late_ms"], "present.late_ms")?, + decode_us: opt_int(&m["decode_us"], "present.decode_us")?, + }) + } + }; + let switch = match &o["switch"] { + Value::Null => None, + x => { + let m = exact(x, &["unit", "from", "to", "reason"], "switch")?; + Some(SwitchObs { + unit: int(&m["unit"], "switch.unit")?, + from: one_of(&m["from"], &["primary", "fallback"], "switch.from")?, + to: one_of(&m["to"], &["primary", "fallback"], "switch.to")?, + reason: one_of(&m["reason"], SWITCH_REASONS, "switch.reason")?, + }) + } + }; + let layer_change = match &o["layer_change"] { + Value::Null => None, + x => { + let m = exact(x, &["from", "to"], "layer_change")?; + Some(LayerChangeObs { from: opt_int(&m["from"], "layer_change.from")?, to: opt_int(&m["to"], "layer_change.to")? }) + } + }; + let capable = match &o["capable"] { + Value::Null => None, + x => { + let m = exact(x, &["outcome", "reason", "classical_floor"], "capable")?; + Some(CapableObs { + outcome: one_of(&m["outcome"], CAPABLE_OUTCOMES, "capable.outcome")?, + reason: opt_one_of(&m["reason"], PIN_REFUSALS, "capable.reason")?, + classical_floor: opt_one_of(&m["classical_floor"], &["supported", "unsupported"], "capable.classical_floor")?, + }) + } + }; + let metrics = match &o["metrics"] { + Value::Null => None, + x => { + let m = exact(x, &["msd_ms", "bsa", "bsda_ms", "dfa"], "metrics")?; + Some(MetricsObs { + msd_ms: opt_sf_int(&m["msd_ms"], "metrics.msd_ms")?, + bsa: opt_sf_int(&m["bsa"], "metrics.bsa")?, + bsda_ms: opt_sf_int(&m["bsda_ms"], "metrics.bsda_ms")?, + dfa: opt_sf_int(&m["dfa"], "metrics.dfa")?, + }) + } + }; + let ob = Observation { + sid, + sn: sf_int(&o["sn"], "sn")?, + ts_ms: opt_sf_int(&o["ts_ms"], "ts_ms")?, + cid, + st: one_of(&o["st"], &["v", "l", "ll"], "st")?, + sf: one_of(&o["sf"], &["o", "h", "d"], "sf")?, + object, + present, + switch, + layer_change, + capable, + cohort: opt_one_of(&o["cohort"], &["control", "wai"], "cohort")?, + metrics, + claim: opt_hex64(&o["claim"], "claim")?, + event, + }; + ob.check()?; + Ok(ob) + } + + /// The rules that tie the members to the event (§2.1). + fn check(&self) -> Result<(), TelemetryError> { + let e = self.event.as_str(); + let want = |member: &str, present: bool, applies: bool| -> Result<(), TelemetryError> { + if present != applies { + Err(bad(&format!("{member}: {} for event `{e}`", if applies { "required" } else { "not allowed" }))) + } else { + Ok(()) + } + }; + want("object", self.object.is_some(), matches!(e, "request" | "response"))?; + want("present", self.present.is_some(), e == "present")?; + want("switch", self.switch.is_some(), e == "switch")?; + want("layer_change", self.layer_change.is_some(), e == "layer-change")?; + want("capable", self.capable.is_some(), e == "capable")?; + if self.claim.is_some() && e != "end" { + return Err(bad("claim: only on `end`")); + } + if self.is_event_mode() && self.ts_ms.is_none() { + return Err(bad("ts_ms: required for an event reported in Event mode")); + } + if let Some(ob) = &self.object { + let response_only = [ob.ttfb_ms, ob.ttlb_ms, ob.rc].iter().any(Option::is_some) || ob.url.is_some(); + let request_only = [ob.d_ms, ob.dl_ms, ob.layer_cap].iter().any(Option::is_some) || ob.ot.is_some(); + if e == "request" && response_only { + return Err(bad("object: a response key on a request")); + } + if e == "response" && request_only { + return Err(bad("object: a request key on a response")); + } + } + if let Some(s) = &self.switch { + if s.from == s.to { + return Err(bad("switch: from and to are one path")); + } + } + if let Some(c) = &self.capable { + if (c.outcome == "pin-refused") != c.reason.is_some() { + return Err(bad("capable.reason: exactly for pin-refused")); + } + } + if let Some(p) = &self.present { + if p.layers == 0 { + return Err(bad("present.layers: at least 1")); + } + } + Ok(()) + } + + /// The observation as JSON, members in §2.1's order. + pub fn to_json(&self) -> Value { + let mut m = Map::new(); + m.insert("v".into(), json!(1)); + m.insert("sid".into(), json!(self.sid)); + m.insert("sn".into(), json!(self.sn)); + m.insert("ts_ms".into(), json!(self.ts_ms)); + m.insert("event".into(), json!(self.event)); + m.insert("cid".into(), json!(self.cid)); + m.insert("st".into(), json!(self.st)); + m.insert("sf".into(), json!(self.sf)); + m.insert("object".into(), match &self.object { + None => Value::Null, + Some(o) => json!({ + "ot": o.ot, "url": o.url, "content_hash": o.content_hash, "capability": o.capability, "rendition": o.rendition, + "layer": o.layer, "layers_total": o.layers_total, "layer_cap": o.layer_cap, "counted": o.counted, + "transferred": o.transferred, "d_ms": o.d_ms, "dl_ms": o.dl_ms, "ttfb_ms": o.ttfb_ms, "ttlb_ms": o.ttlb_ms, + "rc": o.rc, "path": o.path, "receipt": o.receipt, + }), + }); + m.insert("present".into(), match &self.present { + None => Value::Null, + Some(p) => json!({"unit": p.unit, "layers": p.layers, "first": p.first, "late_ms": p.late_ms, "decode_us": p.decode_us}), + }); + m.insert("switch".into(), match &self.switch { + None => Value::Null, + Some(s) => json!({"unit": s.unit, "from": s.from, "to": s.to, "reason": s.reason}), + }); + m.insert("layer_change".into(), match &self.layer_change { + None => Value::Null, + Some(l) => json!({"from": l.from, "to": l.to}), + }); + m.insert("capable".into(), match &self.capable { + None => Value::Null, + Some(c) => json!({"outcome": c.outcome, "reason": c.reason, "classical_floor": c.classical_floor}), + }); + m.insert("cohort".into(), json!(self.cohort)); + m.insert("metrics".into(), match &self.metrics { + None => Value::Null, + Some(x) => json!({"msd_ms": x.msd_ms, "bsa": x.bsa, "bsda_ms": x.bsda_ms, "dfa": x.dfa}), + }); + m.insert("claim".into(), json!(self.claim)); + Value::Object(m) + } +} diff --git a/wai/wai-rs/src/telemetry/qoe.rs b/wai/wai-rs/src/telemetry/qoe.rs new file mode 100644 index 000000000..4f0b24c64 --- /dev/null +++ b/wai/wai-rs/src/telemetry/qoe.rs @@ -0,0 +1,142 @@ +//! Session metrics from one session's observations (`extensions/delivery-telemetry.md` +//! §7): start, first presentation, exit before start, start failure, stalls, +//! engagement, units presented and counted bytes, in integer milliseconds and bytes, +//! with no clock but the observations' own timestamps. + +use std::collections::BTreeSet; + +use serde_json::{json, Value}; + +use super::observation::Observation; +use super::TelemetryError; + +/// What one session's log gives (§7.2). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SessionMetrics { + /// The log has a `start`. + pub started: bool, + /// The video start time: the first presentation's time less the start's. + pub vst_ms: Option, + /// Exit before video start: an `abandon` before any first presentation. + pub exit_before_start: bool, + /// Start failure: a session that started, never presented, and ended without an + /// `abandon`. + pub start_failed: bool, + /// Stalls begun after the first presentation, and their total duration; a stall + /// still open at `end` or `abandon` closes there. + pub stalls: u64, + pub stall_ms: u64, + /// From the first presentation to `end` or `abandon`. + pub engaged_ms: u64, + /// Distinct units presented. + pub units: u64, + /// The counted bytes of every response. + pub counted: u64, + /// The `end` observation's claim pointer. + pub claim: Option, +} + +impl SessionMetrics { + pub fn to_json(&self) -> Value { + json!({ + "started": self.started, "vst_ms": self.vst_ms, "exit_before_start": self.exit_before_start, + "start_failed": self.start_failed, "stalls": self.stalls, "stall_ms": self.stall_ms, + "engaged_ms": self.engaged_ms, "units": self.units, "counted": self.counted, "claim": self.claim, + }) + } +} + +fn bad(s: impl Into) -> TelemetryError { + TelemetryError::Invalid(s.into()) +} + +/// The metrics of one session's log, in `sn` order. Every observation has the log's +/// `sid`, `sn` rises strictly, timestamps never decrease, and nothing follows `end` or +/// `abandon`. +pub fn session_metrics(log: &[Observation]) -> Result { + let first = log.first().ok_or_else(|| bad("an empty log"))?; + let mut m = SessionMetrics { + started: false, + vst_ms: None, + exit_before_start: false, + start_failed: false, + stalls: 0, + stall_ms: 0, + engaged_ms: 0, + units: 0, + counted: 0, + claim: None, + }; + let mut start_ms: Option = None; + let mut present_ms: Option = None; + let mut stall_open: Option = None; + let mut units: BTreeSet = BTreeSet::new(); + let mut last_ts = 0u64; + let mut ended = false; + for (i, o) in log.iter().enumerate() { + if o.sid != first.sid { + return Err(bad(format!("observation {i}: another session"))); + } + if i > 0 && o.sn <= log[i - 1].sn { + return Err(bad(format!("observation {i}: sn does not rise"))); + } + if ended { + return Err(bad(format!("observation {i}: after the session's end"))); + } + if let Some(t) = o.ts_ms { + if t < last_ts { + return Err(bad(format!("observation {i}: time runs backwards"))); + } + last_ts = t; + } + let t = o.ts_ms; + match o.event.as_str() { + "start" if start_ms.is_none() => { + m.started = true; + start_ms = t; + } + "present" => { + let p = o.present.as_ref().expect("checked"); + units.insert(p.unit); + if p.first { + if present_ms.is_some() { + return Err(bad(format!("observation {i}: a second first presentation"))); + } + present_ms = t; + m.vst_ms = match (start_ms, t) { + (Some(s), Some(t)) => Some(t - s.min(t)), + _ => None, + }; + } + } + "stall-start" if present_ms.is_some() && stall_open.is_none() => stall_open = t, + "stall-end" => { + if let (Some(s), Some(e)) = (stall_open.take(), t) { + m.stalls += 1; + m.stall_ms = m.stall_ms.checked_add(e - s).ok_or_else(|| bad("stall time passes 64 bits"))?; + } + } + "response" => { + let c = o.object.as_ref().and_then(|x| x.counted).unwrap_or(0); + m.counted = m.counted.checked_add(c).ok_or_else(|| bad(format!("observation {i}: counted bytes pass 64 bits")))?; + } + "abandon" | "end" => { + ended = true; + let e = t.expect("an Event-mode report carries a timestamp"); + if let Some(s) = stall_open.take() { + m.stalls += 1; + m.stall_ms = m.stall_ms.checked_add(e - s).ok_or_else(|| bad("stall time passes 64 bits"))?; + } + match present_ms { + Some(p) => m.engaged_ms = e - p, + None if o.event == "abandon" => m.exit_before_start = true, + None => m.start_failed = m.started, + } + m.claim = o.claim.clone(); + } + _ => {} + } + } + m.units = units.len() as u64; + Ok(m) +} diff --git a/wai/wai-rs/src/telemetry/sf.rs b/wai/wai-rs/src/telemetry/sf.rs new file mode 100644 index 000000000..b9030a263 --- /dev/null +++ b/wai/wai-rs/src/telemetry/sf.rs @@ -0,0 +1,410 @@ +//! The subset of RFC 9651 Structured Field Values that CMCD and CMSD reports use: +//! dictionaries whose members are bare items (Integer, String, Token, Boolean) or inner +//! lists of Integers, and lists of items with parameters. Serialisation follows RFC +//! 9651 §4.1 and writes no optional whitespace; parsing follows §4.2 and refuses what +//! this subset does not hold. + +/// A bare item (RFC 9651 §3.3). +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum SfBare { + /// −999 999 999 999 999 … 999 999 999 999 999. + Int(i64), + /// Printable ASCII. + Str(String), + Token(String), + Bool(bool), +} + +/// A dictionary member: a bare item, or an inner list of Integers with no parameters. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum SfMember { + Item(SfBare), + IntList(Vec), +} + +/// An item of a list, with its parameters in order. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SfItem { + pub value: SfBare, + pub params: Vec<(String, SfBare)>, +} + +/// Why a value cannot be serialised or a field cannot be parsed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SfError(pub String); + +impl std::fmt::Display for SfError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "structured field: {}", self.0) + } +} + +impl std::error::Error for SfError {} + +fn err(s: impl Into) -> SfError { + SfError(s.into()) +} + +const INT_MAX: i64 = 999_999_999_999_999; + +/// Whether `k` is a key (RFC 9651 §3.1.2): `lcalpha` or `*` first, then `lcalpha`, +/// digits, `_`, `-`, `.` or `*`. +pub fn is_key(k: &str) -> bool { + let b = k.as_bytes(); + !b.is_empty() + && (b[0].is_ascii_lowercase() || b[0] == b'*') + && b.iter().all(|&c| c.is_ascii_lowercase() || c.is_ascii_digit() || matches!(c, b'_' | b'-' | b'.' | b'*')) +} + +fn is_tchar(c: u8) -> bool { + c.is_ascii_alphanumeric() || b"!#$%&'*+-.^_`|~".contains(&c) +} + +/// Whether `t` is a Token (RFC 9651 §3.3.4). +pub fn is_token(t: &str) -> bool { + let b = t.as_bytes(); + !b.is_empty() && (b[0].is_ascii_alphabetic() || b[0] == b'*') && b.iter().all(|&c| is_tchar(c) || c == b':' || c == b'/') +} + +fn bare(v: &SfBare, out: &mut String) -> Result<(), SfError> { + match v { + SfBare::Int(i) => { + if !(-INT_MAX..=INT_MAX).contains(i) { + return Err(err(format!("integer {i} out of range"))); + } + out.push_str(&i.to_string()); + } + SfBare::Str(s) => { + if !s.bytes().all(|c| (0x20..=0x7e).contains(&c)) { + return Err(err("a string holds a byte outside printable ASCII")); + } + out.push('"'); + for c in s.chars() { + if c == '"' || c == '\\' { + out.push('\\'); + } + out.push(c); + } + out.push('"'); + } + SfBare::Token(t) => { + if !is_token(t) { + return Err(err(format!("`{t}` is not a token"))); + } + out.push_str(t); + } + SfBare::Bool(b) => out.push_str(if *b { "?1" } else { "?0" }), + } + Ok(()) +} + +fn params(ps: &[(String, SfBare)], out: &mut String) -> Result<(), SfError> { + for (k, v) in ps { + if !is_key(k) { + return Err(err(format!("`{k}` is not a key"))); + } + out.push(';'); + out.push_str(k); + if *v != SfBare::Bool(true) { + out.push('='); + bare(v, out)?; + } + } + Ok(()) +} + +/// Serialise a dictionary, members in the order given (RFC 9651 §4.1.2). A Boolean +/// true is written as its key alone. +pub fn serialize_dict(m: &[(String, SfMember)]) -> Result { + let mut out = String::new(); + for (i, (k, v)) in m.iter().enumerate() { + if !is_key(k) { + return Err(err(format!("`{k}` is not a key"))); + } + if i > 0 { + out.push(','); + } + out.push_str(k); + match v { + SfMember::Item(SfBare::Bool(true)) => {} + SfMember::Item(b) => { + out.push('='); + bare(b, &mut out)?; + } + SfMember::IntList(l) => { + out.push_str("=("); + for (j, x) in l.iter().enumerate() { + if j > 0 { + out.push(' '); + } + bare(&SfBare::Int(*x), &mut out)?; + } + out.push(')'); + } + } + } + Ok(out) +} + +/// Serialise a list of items with parameters (RFC 9651 §4.1.1). +pub fn serialize_list(items: &[SfItem]) -> Result { + let mut out = String::new(); + for (i, it) in items.iter().enumerate() { + if i > 0 { + out.push(','); + } + bare(&it.value, &mut out)?; + params(&it.params, &mut out)?; + } + Ok(out) +} + +struct P<'a> { + b: &'a [u8], + i: usize, +} + +impl P<'_> { + fn peek(&self) -> Option { + self.b.get(self.i).copied() + } + fn ows(&mut self) { + while matches!(self.peek(), Some(b' ' | b'\t')) { + self.i += 1; + } + } + fn sp(&mut self) { + while self.peek() == Some(b' ') { + self.i += 1; + } + } + fn key(&mut self) -> Result { + let s = self.i; + match self.peek() { + Some(c) if c.is_ascii_lowercase() || c == b'*' => {} + _ => return Err(err(format!("a key expected at {}", self.i))), + } + while matches!(self.peek(), Some(c) if c.is_ascii_lowercase() || c.is_ascii_digit() || matches!(c, b'_' | b'-' | b'.' | b'*')) { + self.i += 1; + } + Ok(String::from_utf8_lossy(&self.b[s..self.i]).into_owned()) + } + fn bare(&mut self) -> Result { + match self.peek() { + Some(b'-' | b'0'..=b'9') => { + let s = self.i; + if self.peek() == Some(b'-') { + self.i += 1; + } + let d = self.i; + while matches!(self.peek(), Some(b'0'..=b'9')) { + self.i += 1; + } + if self.peek() == Some(b'.') { + return Err(err("a decimal is not in this subset")); + } + if self.i == d || self.i - d > 15 { + return Err(err("an integer has no digits or more than 15")); + } + let t = std::str::from_utf8(&self.b[s..self.i]).expect("ASCII"); + Ok(SfBare::Int(t.parse().map_err(|_| err("an integer"))?)) + } + Some(b'"') => { + self.i += 1; + let mut out = String::new(); + loop { + match self.peek() { + None => return Err(err("an unterminated string")), + Some(b'"') => { + self.i += 1; + return Ok(SfBare::Str(out)); + } + Some(b'\\') => { + self.i += 1; + match self.peek() { + Some(c @ (b'"' | b'\\')) => { + out.push(c as char); + self.i += 1; + } + _ => return Err(err("a backslash escapes only a quote or a backslash")), + } + } + Some(c) if (0x20..=0x7e).contains(&c) => { + out.push(c as char); + self.i += 1; + } + Some(_) => return Err(err("a string holds a byte outside printable ASCII")), + } + } + } + Some(b'?') => { + self.i += 1; + let v = match self.peek() { + Some(b'1') => true, + Some(b'0') => false, + _ => return Err(err("a boolean is ?0 or ?1")), + }; + self.i += 1; + Ok(SfBare::Bool(v)) + } + Some(c) if c.is_ascii_alphabetic() || c == b'*' => { + let s = self.i; + while matches!(self.peek(), Some(c) if is_tchar(c) || c == b':' || c == b'/') { + self.i += 1; + } + Ok(SfBare::Token(String::from_utf8_lossy(&self.b[s..self.i]).into_owned())) + } + _ => Err(err(format!("an item expected at {}", self.i))), + } + } + fn params(&mut self) -> Result, SfError> { + let mut out: Vec<(String, SfBare)> = Vec::new(); + while self.peek() == Some(b';') { + self.i += 1; + self.sp(); + let k = self.key()?; + let v = if self.peek() == Some(b'=') { + self.i += 1; + self.bare()? + } else { + SfBare::Bool(true) + }; + match out.iter_mut().find(|(x, _)| *x == k) { + Some(slot) => slot.1 = v, + None => out.push((k, v)), + } + } + Ok(out) + } + fn sep(&mut self) -> Result { + self.ows(); + if self.i == self.b.len() { + return Ok(false); + } + if self.peek() != Some(b',') { + return Err(err(format!("a comma expected at {}", self.i))); + } + self.i += 1; + self.ows(); + if self.i == self.b.len() { + return Err(err("a trailing comma")); + } + Ok(true) + } +} + +/// Parse a dictionary (RFC 9651 §4.2.2). A key given twice keeps its first place and +/// takes its last value. Members with parameters are not in this subset. +pub fn parse_dict(s: &str) -> Result, SfError> { + let mut p = P { b: s.as_bytes(), i: 0 }; + p.sp(); + let mut out: Vec<(String, SfMember)> = Vec::new(); + if p.i == p.b.len() { + return Ok(out); + } + loop { + let k = p.key()?; + let v = if p.peek() == Some(b'=') { + p.i += 1; + if p.peek() == Some(b'(') { + p.i += 1; + let mut l = Vec::new(); + loop { + p.sp(); + if p.peek() == Some(b')') { + p.i += 1; + break; + } + match p.bare()? { + SfBare::Int(x) => l.push(x), + _ => return Err(err("an inner list here holds integers only")), + } + if !matches!(p.peek(), Some(b' ' | b')')) { + return Err(err("an inner list item is followed by a space or `)`")); + } + } + SfMember::IntList(l) + } else { + SfMember::Item(p.bare()?) + } + } else { + SfMember::Item(SfBare::Bool(true)) + }; + if p.peek() == Some(b';') { + return Err(err("parameters on a dictionary member are not in this subset")); + } + match out.iter_mut().find(|(x, _)| *x == k) { + Some(slot) => slot.1 = v, + None => out.push((k, v)), + } + if !p.sep()? { + return Ok(out); + } + } +} + +/// Parse a list of items with parameters (RFC 9651 §4.2.1). +pub fn parse_list(s: &str) -> Result, SfError> { + let mut p = P { b: s.as_bytes(), i: 0 }; + p.sp(); + let mut out = Vec::new(); + if p.i == p.b.len() { + return Ok(out); + } + loop { + let value = p.bare()?; + let params = p.params()?; + out.push(SfItem { value, params }); + if !p.sep()? { + return Ok(out); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn item(b: SfBare) -> SfMember { + SfMember::Item(b) + } + + #[test] + fn dictionary_round_trip() { + let d = vec![ + ("bs".to_string(), item(SfBare::Bool(true))), + ("bsa".to_string(), SfMember::IntList(vec![2, 3])), + ("cid".to_string(), item(SfBare::Str("a\"b\\c".into()))), + ("ot".to_string(), item(SfBare::Token("v".into()))), + ("su".to_string(), item(SfBare::Bool(false))), + ("ts".to_string(), item(SfBare::Int(-999_999_999_999_999))), + ]; + let s = serialize_dict(&d).unwrap(); + assert_eq!(s, r#"bs,bsa=(2 3),cid="a\"b\\c",ot=v,su=?0,ts=-999999999999999"#); + assert_eq!(parse_dict(&s).unwrap(), d); + // Whitespace around commas is read; a later value of a key replaces the first. + assert_eq!(parse_dict("a=1 , b=2,a=3").unwrap(), vec![("a".into(), item(SfBare::Int(3))), ("b".into(), item(SfBare::Int(2)))]); + } + + #[test] + fn what_this_subset_refuses() { + for bad in ["A=1", "a=1,", "a=1;p", "a=1.5", "a=1234567890123456", "a=\"\\x\"", "a=(1 \"s\")", "a=?2", "a,,b", "a=\"open"] { + assert!(parse_dict(bad).is_err(), "{bad}"); + } + assert!(serialize_dict(&[("a".into(), item(SfBare::Int(1_000_000_000_000_000)))]).is_err()); + assert!(serialize_dict(&[("a".into(), item(SfBare::Str("é".into())))]).is_err()); + assert!(serialize_dict(&[("a".into(), item(SfBare::Token("1x".into())))]).is_err()); + assert!(serialize_dict(&[("A".into(), item(SfBare::Int(1)))]).is_err()); + } + + #[test] + fn list_with_parameters() { + let l = vec![ + SfItem { value: SfBare::Str("edge-1".into()), params: vec![("du".into(), SfBare::Bool(true)), ("etp".into(), SfBare::Int(25000))] }, + SfItem { value: SfBare::Str("origin".into()), params: vec![("rd".into(), SfBare::Int(12))] }, + ]; + let s = serialize_list(&l).unwrap(); + assert_eq!(s, r#""edge-1";du;etp=25000,"origin";rd=12"#); + assert_eq!(parse_list(&s).unwrap(), l); + } +} diff --git a/wai/wai-rs/src/wai_video_package.rs b/wai/wai-rs/src/wai_video_package.rs index be70aaca2..ae2a07e91 100644 --- a/wai/wai-rs/src/wai_video_package.rs +++ b/wai/wai-rs/src/wai_video_package.rs @@ -21,7 +21,6 @@ //! evidence — is measured-attested: signed, and not re-measurable. Reuses the //! manifest module's `EnergyProvenance` and the world Merkle floor's Ed25519 + `GrantRef`. -use std::collections::BTreeSet; use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey}; @@ -109,16 +108,12 @@ pub fn segment(gops: &[Gop], target_ms: u64) -> Vec { /// The count of distinct segment contents, and the bytes a dedup store keeps /// (one copy per distinct content). Both recomputed from the segments so a -/// receipt cannot overstate the saving. -fn dedup_stats(segments: &[SegmentEntry]) -> (u64, u64) { - let mut seen: BTreeSet<[u8; 32]> = BTreeSet::new(); - let mut stored = 0u64; - for s in segments { - if seen.insert(s.content_hash) { - stored += s.byte_len; - } - } - (seen.len() as u64, stored) +/// receipt cannot overstate the saving (`content_dedup::totals_checked`, the one +/// dedup computation of the crate). `None` for a map that gives one content hash two +/// byte lengths: it has no dedup accounting, and a receipt over it does not verify. +fn dedup_stats(segments: &[SegmentEntry]) -> Option<(u64, u64)> { + let t = crate::content_dedup::totals_checked(segments.iter().map(|s| (s.content_hash, s.byte_len))).ok()?; + Some((t.unique, t.stored_bytes)) } /// A signed SegmentMap with an attested dedup accounting. @@ -201,7 +196,9 @@ impl PackageReceipt { parent_receipt_hash: Option<[u8; 32]>, ) -> PackageReceipt { let leaves: Vec<[u8; 32]> = segments.iter().map(SegmentEntry::leaf).collect(); - let (unique_segments, stored_bytes) = dedup_stats(&segments); + // A map that gives one content hash two lengths has no dedup accounting: it is + // sealed with both figures 0, and `verify` refuses it. + let (unique_segments, stored_bytes) = dedup_stats(&segments).unwrap_or((0, 0)); let total_bytes = segments.iter().map(|s| s.byte_len).sum(); let mut r = PackageReceipt { asset_id: asset_id.into(), @@ -238,7 +235,7 @@ impl PackageReceipt { if self.work_segment_ops != self.segments.len() as u64 { return false; } - let (uniq, stored) = dedup_stats(&self.segments); + let Some((uniq, stored)) = dedup_stats(&self.segments) else { return false }; if uniq != self.unique_segments || stored != self.stored_bytes { return false; } @@ -317,7 +314,7 @@ mod tests { assert_eq!(a, b, "segmentation must be a pure function"); assert_eq!(a.len(), 4); assert_eq!(a[1].content_hash, a[3].content_hash, "the repeated ad dedups"); - let (uniq, _) = dedup_stats(&a); + let (uniq, _) = dedup_stats(&a).unwrap(); assert_eq!(uniq, 3, "4 segments, 3 distinct"); } @@ -347,6 +344,18 @@ mod tests { assert!(!r.verify(), "a dedup figure the segments do not support must fail"); } + #[test] + fn one_content_with_two_lengths_has_no_dedup_accounting() { + // Segment 3 repeats segment 1's content hash with another length: no store can + // keep "one copy" of it, so the map has no dedup figure and its receipt fails. + let mut segs = segment(&source(), 2000); + segs[3].byte_len += 1; + let g = GrantRef::unbounded("video.package"); + let r = PackageReceipt::seal(&key(5), "o", "asset", segs, 0, None, g, None); + assert_eq!((r.unique_segments, r.stored_bytes), (0, 0)); + assert!(!r.verify(), "a map with one hash at two lengths must not verify"); + } + #[test] fn tampered_segment_breaks_the_map() { let segs = segment(&source(), 2000); diff --git a/wai/wai-rs/tests/cli_stage.rs b/wai/wai-rs/tests/cli_stage.rs index 13e24f393..94b260561 100644 --- a/wai/wai-rs/tests/cli_stage.rs +++ b/wai/wai-rs/tests/cli_stage.rs @@ -537,3 +537,49 @@ fn apply_records_a_refused_base_and_goes_on() { } } } + +/// The bound vectors of `tests/vectors/staged_bounds/`. +fn bounds(file: &str) -> PathBuf { + wai_dir().join("wai-rs/tests/vectors/staged_bounds").join(file) +} + +/// `wai stage apply` holds every layer to the stated cost limit with no flag: a 275-byte +/// base whose 587-byte up2 layer declares 67,092,480 output bytes, past the stated limit +/// of 2^25, is refused (`cost-over-limit`) before any symbol of it is decoded, and the +/// base stays at stage 0. +#[test] +fn apply_holds_a_layer_to_the_stated_limit() { + let (b, l) = (bounds("layer_bomb_base.wai"), bounds("layer_bomb_up2.wai")); + let o = wai(&[os("stage"), os("inspect"), l.as_os_str()]); + assert!(text(&o).contains("declared cost: 67092480 output bytes"), "the vector declares what its README states: {}", text(&o)); + let o = wai(&[os("stage"), os("apply"), os("--json"), b.as_os_str(), l.as_os_str()]); + assert!(o.status.success(), "{}", text(&o)); + let doc: serde_json::Value = serde_json::from_slice(&o.stdout).unwrap(); + let r = &doc["steps"][1]["reports"][0]; + assert_eq!((r["envelope"].as_str(), r["detail"].as_str()), (Some("cost-over-limit"), Some("output bytes")), "{r}"); + assert!(r["applied"].as_array().unwrap().is_empty()); + assert_eq!(doc["bases"][0]["units"][0]["stage"], 0); +} + +/// `wai_stage_bench` decodes under the stated limit: the 64 MiB bound vector is refused +/// (`cost_over_limit`) before it is decoded, and the layer past the limit is refused +/// rather than measured. +#[test] +fn stage_bench_applies_the_stated_limit() { + let src = scratch("bench_bounds_source.bin"); + std::fs::write(&src, [0u8; 3]).unwrap(); + let bench = |envs: &[&Path]| { + let mut c = Command::new(env!("CARGO_BIN_EXE_wai_stage_bench")); + c.args([os("--source"), src.as_os_str(), os("--runs"), os("20")]); + c.args(envs); + let t = std::time::Instant::now(); + let o = c.output().unwrap(); + (o, t.elapsed()) + }; + let (o, took) = bench(&[&bounds("clip_64mib.wai")]); + assert!(!o.status.success() && text(&o).contains("cost_over_limit"), "{}", text(&o)); + // Refused from its header: a decode of 64 MiB of output is not what happened. + assert!(took < std::time::Duration::from_secs(20), "{took:?}"); + let (o, _) = bench(&[&bounds("layer_bomb_base.wai"), &bounds("layer_bomb_up2.wai")]); + assert!(!o.status.success() && text(&o).contains("layer_bomb_up2.wai: not applied whole (cost-over-limit)"), "{}", text(&o)); +} diff --git a/wai/wai-rs/tests/int_kernels_vectors.rs b/wai/wai-rs/tests/int_kernels_vectors.rs index e226b7c6d..58e8176cc 100644 --- a/wai/wai-rs/tests/int_kernels_vectors.rs +++ b/wai/wai-rs/tests/int_kernels_vectors.rs @@ -225,9 +225,11 @@ fn every_decoded_output_states_a_sha256() { ("byte-exact-conformance", "goldens over tests/vectors, whose sidecars state the SHA-256"), ("int-payload-conformance", "refusal fixtures and the index of the payloads under tests/vectors, whose sidecars state the SHA-256"), ("staged-conformance", "each case.json states the SHA-256 of every stage-0 buffer and of every unit's stage (SPEC §7 Digests)"), + ("staged-measure-conformance", "each fixture's expect.json and goldens.json state the SHA-256 of every unit's output at every stage (SPEC §7 Digests)"), + ("telemetry-conformance", "advisory observations and their CMCD and CMSD strings: no decoded output"), ("vectors/int_mlicpp", "expected.json states rgb_sha256 (RGB8)"), ("vectors/jpegai", "inputs only; the decoded pixels' SHA-256 is pinned by the JPEG AI e2e job"), - ("vectors/staged_bounds", "bases that declare clips past a sink's cost limit, refused before any symbol is decoded, and inert envelopes: nothing here is decoded"), + ("vectors/staged_bounds", "bases and a layer that declare outputs past a sink's cost limit, refused before any symbol is decoded, and inert envelopes; the one base decoded is wai.video.int_motion at keyframe 0, which decodes against no parameter set"), ]; let wai = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(".."); let mut dirs: Vec<(String, PathBuf)> = Vec::new(); diff --git a/wai/wai-rs/tests/stage_measure.rs b/wai/wai-rs/tests/stage_measure.rs new file mode 100644 index 000000000..f7d2d5707 --- /dev/null +++ b/wai/wai-rs/tests/stage_measure.rs @@ -0,0 +1,244 @@ +//! Staged-delivery measurement against the committed corpora +//! (`extensions/staged-measurement.md`): the byte rule over every committed envelope, +//! the refusal of text forms, and a title read and evaluated from a map of files, as a +//! browser reads one. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; + +use wai::container::{Conditioning, Manifest, ModelRequirement, Wai}; +use wai::measure_framing::{count_object, FramingError, REGISTERED}; +use wai::stage_measure::run::evaluate; +use wai::stage_measure::title::Title; +use wai::stage_measure::MeasureError; + +fn wai_root() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("..") +} + +fn envelopes(dir: &Path, out: &mut Vec) { + for e in std::fs::read_dir(dir).unwrap().flatten() { + let p = e.path(); + if p.is_dir() { + envelopes(&p, out); + } else if p.extension().is_some_and(|x| x == "wai") { + out.push(p); + } + } +} + +/// Every committed envelope of the staged corpora is either counted as exactly its +/// length (§3.1: no byte of an envelope is left out, none is added) or refused; and the +/// measurement corpus's envelopes, all well formed, are all counted. +#[test] +fn binary_counts_whole_envelope() { + let mut files = Vec::new(); + envelopes(&wai_root().join("staged-conformance"), &mut files); + let mut measured = Vec::new(); + envelopes(&wai_root().join("staged-measure-conformance/fixtures"), &mut measured); + assert!(!files.is_empty() && !measured.is_empty()); + let (mut counted, mut wai2) = (0, 0); + for p in files.iter().chain(&measured) { + let b = std::fs::read(p).unwrap(); + match count_object(&b) { + Ok(c) => { + assert_eq!(c.total, b.len() as u64, "{}", p.display()); + assert_eq!(c.header + c.manifest + c.payload, c.total); + assert_eq!(c.framing, REGISTERED); + counted += 1; + wai2 += (&b[..4] == b"WAI2") as usize; + } + Err(e) => assert!(!measured.contains(p), "{}: {e}", p.display()), + } + } + assert!(counted > measured.len() && wai2 > 0, "the corpora's well-formed envelopes, WAI2 among them, are counted"); +} + +/// §3.2: a text form is never counted. Each pinned integer format's JSON source, as +/// the encode and export tools write it, is refused, and the binary payload it converts +/// to is counted, and is the smaller. +#[test] +fn stored_json_is_refused_and_overstates() { + let v = wai_root().join("wai-rs/tests/vectors"); + for (cap, json, bin) in [ + ("wai.neural.int_hyper", "int_hyper/tiny_bitstream.json", "int_hyper/tiny_bitstream.wih1"), + ("wai.video.int_motion", "int_video/int_video_bitstream.json", "int_video/int_video_bitstream.wiv1"), + ("wai.audio.int_codec", "int_audio/audio_bitstream.json", "int_audio/audio_bitstream.wia1"), + ("wai.splat.int_codec", "int_splat/splat.json", "int_splat/splat.wis1"), + ] { + let wrap = |p: &str| { + let m = Manifest::new("image", "replicate", ModelRequirement::new(cap), Conditioning::new("k")); + Wai::new(m, std::fs::read(v.join(p)).unwrap()).to_bytes().unwrap() + }; + let (j, b) = (wrap(json), wrap(bin)); + assert!(matches!(count_object(&j), Err(FramingError::NotCountable { .. })), "{cap}: its JSON form is not countable"); + let c = count_object(&b).unwrap_or_else(|e| panic!("{cap}: {e}")); + assert!(c.total < j.len() as u64, "{cap}: the stored JSON is larger than the payload it converts to"); + } + // A pinned capability counts only its own format: the binary payload in base64 + // text, and another capability's binary format, are not countable either. + let wih1 = std::fs::read(v.join("int_hyper/tiny_bitstream.wih1")).unwrap(); + let b64: String = { + const A: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + wih1.chunks(3) + .flat_map(|c| { + let n = (c[0] as u32) << 16 | (*c.get(1).unwrap_or(&0) as u32) << 8 | *c.get(2).unwrap_or(&0) as u32; + (0..4).map(move |i| if i <= c.len() { A[(n >> (18 - 6 * i) & 63) as usize] as char } else { '=' }) + }) + .collect() + }; + for (cap, payload) in [("wai.neural.int_hyper", b64.into_bytes()), ("wai.video.int_motion", wih1)] { + let m = Manifest::new("image", "replicate", ModelRequirement::new(cap), Conditioning::new("k")); + let e = Wai::new(m, payload).to_bytes().unwrap(); + assert!(matches!(count_object(&e), Err(FramingError::NotCountable { .. })), "{cap}"); + } +} + +fn fixture_files(dir: &str) -> BTreeMap> { + // Read exactly what the title asks for, as a browser would fetch it. + let mut files = BTreeMap::new(); + loop { + match Title::from_files(dir, &files) { + Err(MeasureError::Missing(p)) => { + files.insert(p.clone(), std::fs::read(wai_root().join(&p)).unwrap()); + } + Err(e) => panic!("{dir}: {e}"), + Ok(_) => return files, + } + } +} + +/// A title read from a map of files evaluates to its golden, and every file it read +/// is one it needs: nothing is read from disk by the library. +#[test] +fn a_title_from_files_evaluates_to_its_golden() { + let goldens: serde_json::Value = serde_json::from_slice(&std::fs::read(wai_root().join("staged-measure-conformance/goldens.json")).unwrap()).unwrap(); + let dir = "staged-measure-conformance/fixtures/still"; + let files = fixture_files(dir); + assert_eq!(Title::from_files(dir, &BTreeMap::new()), Err(MeasureError::Missing(format!("{dir}/title.json")))); + let t = Title::from_files(dir, &files).unwrap(); + let e = evaluate(&t).unwrap(); + let g = goldens["fixtures"].as_array().unwrap().iter().find(|f| f["title"] == "still").unwrap(); + assert_eq!(&e.to_json(), g); +} + +/// A changed byte anywhere the measurement reads is refused, never measured: a layer +/// (it no longer applies whole, or no longer parses), the base (its content hash is +/// the case's), the source (its SHA-256 is the title's), a parameter-set file (the pin +/// refuses it). +#[test] +fn a_changed_byte_is_refused() { + let dir = "staged-measure-conformance/fixtures/still"; + let files = fixture_files(dir); + let flip = |path: &str, at: usize| { + let mut f = files.clone(); + let b = f.get_mut(path).unwrap_or_else(|| panic!("{path} is read")); + let i = at.min(b.len() - 1); + b[i] ^= 0x01; + f + }; + let set = files.keys().find(|k| k.ends_with("tiny_g_s.bin")).unwrap().clone(); + let base = files.keys().find(|k| k.ends_with("image_int_hyper_2l/base.wai")).unwrap().clone(); + // A refusal names its code: the layer's (§4.2) and the pin's (SPEC §3.1); a base or + // a source that is not the case's or the title's is invalid. + for (path, at, code) in [ + (format!("{dir}/L02.wai"), usize::MAX, Some("payload-malformed")), + (format!("{dir}/source.bin"), 7, None), + (base, usize::MAX, None), + (set, 100, Some("prior_not_held")), + ] { + let f = flip(&path, at); + let r = Title::from_files(dir, &f).and_then(|t| evaluate(&t)); + match (&r, code) { + (Err(MeasureError::Refused { code: got, .. }), Some(want)) => assert_eq!(got, want, "{path}"), + (Err(MeasureError::Invalid(_)), None) => {} + _ => panic!("{path}: {r:?}"), + } + } +} + +/// The case's statement of its base's decode is checked against the decode: a case +/// that says the pin did not resolve, of a base whose pin does, is refused. +#[test] +fn the_case_facts_are_checked() { + let dir = "staged-measure-conformance/fixtures/still"; + let mut files = fixture_files(dir); + let case = files.keys().find(|k| k.ends_with("image_int_hyper_2l/case.json")).unwrap().clone(); + let text = String::from_utf8(files[&case].clone()).unwrap(); + assert!(text.contains("\"pin_resolved\": true")); + files.insert(case, text.replacen("\"pin_resolved\": true", "\"pin_resolved\": false", 1).into_bytes()); + let r = Title::from_files(dir, &files).and_then(|t| evaluate(&t)); + assert!(matches!(r, Err(MeasureError::Invalid(_))), "{r:?}"); +} + +/// Every case of `counting.json` is what `count_object` gives: its counted bytes, or the +/// code (and the `WAI2` component) it is not countable with. +#[test] +fn the_counting_cases_hold() { + let doc: serde_json::Value = serde_json::from_slice(&std::fs::read(wai_root().join("staged-measure-conformance/counting.json")).unwrap()).unwrap(); + let cases = doc["cases"].as_array().unwrap(); + assert!(cases.iter().any(|c| c["refused"].is_null()) && cases.iter().any(|c| !c["refused"].is_null())); + for c in cases { + let b = std::fs::read(wai_root().join(c["file"].as_str().unwrap())).unwrap(); + match count_object(&b) { + Ok(k) => assert_eq!(c["counted"]["total"], k.total, "{}", c["file"]), + Err(e) => { + assert_eq!(c["refused"], e.code(), "{}", c["file"]); + if let FramingError::NotCountable { component, .. } = e { + assert_eq!(c["component"], serde_json::json!(component), "{}", c["file"]); + } + } + } + } +} + +/// A title lowers its limits and never raises them (Appendix D): each figure is the +/// lower of the title's and the tool's, and only the tool's own limit raises one. +#[test] +fn a_title_lowers_its_limits_and_never_raises_them() { + use wai::int_payload::DeclaredCost; + use wai::stage_measure::title::{Limits, StatedLimits}; + let tool = Limits::default(); + let raised = StatedLimits { cost: Some(DeclaredCost::new(1 << 40, 1 << 40, 1 << 40, 1 << 40)), chain_budget: Some(1 << 40) }; + assert_eq!(Limits::within(&raised, &tool), tool); + let mixed = StatedLimits { cost: Some(DeclaredCost::new(1024, 1 << 40, 1 << 20, 1 << 40)), chain_budget: Some(383) }; + let l = Limits::within(&mixed, &tool); + assert_eq!(l.cost, DeclaredCost::new(1024, tool.cost.symbols, 1 << 20, tool.cost.macs)); + assert_eq!(l.chain_budget, 383); + // The title stating 2^40 is measured under the tools' stated limit, and under a tool's + // own larger one only as far as that one goes. + let mut t = Title::load(&wai_root(), "staged-measure-conformance/refusals/clip-1gib-limits-raised").unwrap(); + assert_eq!(t.limits, tool); + let big = Limits { cost: DeclaredCost::new(1 << 30, 1 << 31, 1 << 32, 1 << 39), chain_budget: 1 << 30 }; + t.under(&big); + assert_eq!(t.limits, big); +} + +/// §13's matrix: every class it marks not countable in this revision is, under the +/// capability its row names (`no-format-check`), whatever the payload holds. +#[test] +fn the_matrix_rows_not_countable_are_not() { + use wai::codecs::*; + let not_countable = [ + CAP_IMAGE_PNG, + CAP_AUDIO_FLAC, + CAP_VIDEO_LCEVC, + CAP_SPLAT_4D, + CAP_WORLD_REPLAY, + CAP_FEED_STATE, + CAP_FILM_LINEAR, + CAP_AVATAR_POSE, + CAP_AUDIO_SCORE, + CAP_HAPTIC_SIGNAL, + CAP_QUANTUM_CIRCUIT, + CAP_TEXT_ZSTD, + ]; + for cap in not_countable { + let m = Manifest::new("image", "replicate", ModelRequirement::new(cap), Conditioning::new("k")); + let env = Wai::new(m, b"\x89PNG\r\n\x1a\nWIV1WIH1".to_vec()).to_bytes().unwrap(); + match count_object(&env) { + Err(FramingError::NotCountable { code, .. }) => assert_eq!(code, "no-format-check", "{cap}"), + other => panic!("{cap}: {other:?}"), + } + } +} diff --git a/wai/wai-rs/tests/stage_measure_bounds.rs b/wai/wai-rs/tests/stage_measure_bounds.rs new file mode 100644 index 000000000..98f4ba9ff --- /dev/null +++ b/wai/wai-rs/tests/stage_measure_bounds.rs @@ -0,0 +1,101 @@ +//! A title whose base declares an outsized decode is refused before anything is +//! allocated for it (`extensions/staged-measurement.md` Appendix D): the measurement +//! decodes a base only through `staged::decode_base_pinned`, under the title's cost +//! limit. Its own binary: it installs a counting global allocator and reads the peak of +//! the bytes live at once while the 1 GiB bound vector's title is evaluated. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::collections::BTreeMap; +use std::path::Path; +use std::sync::atomic::{AtomicUsize, Ordering::Relaxed}; + +use wai::stage_measure::run::{check_refusal, evaluate}; +use wai::stage_measure::title::Title; +use wai::stage_measure::MeasureError; + +struct Count; +static CUR: AtomicUsize = AtomicUsize::new(0); +static PEAK: AtomicUsize = AtomicUsize::new(0); + +fn grow(n: usize) { + PEAK.fetch_max(CUR.fetch_add(n, Relaxed) + n, Relaxed); +} + +unsafe impl GlobalAlloc for Count { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + grow(l.size()); + unsafe { System.alloc(l) } + } + unsafe fn alloc_zeroed(&self, l: Layout) -> *mut u8 { + grow(l.size()); + unsafe { System.alloc_zeroed(l) } + } + unsafe fn realloc(&self, p: *mut u8, l: Layout, new: usize) -> *mut u8 { + if new >= l.size() { + grow(new - l.size()); + } else { + CUR.fetch_sub(l.size() - new, Relaxed); + } + unsafe { System.realloc(p, l, new) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + CUR.fetch_sub(l.size(), Relaxed); + unsafe { System.dealloc(p, l) } + } +} + +#[global_allocator] +static ALLOC: Count = Count; + +/// The tests share the counters, so they run one at a time. +static ONE_AT_A_TIME: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +fn files(dir: &str) -> BTreeMap> { + let wai = Path::new(env!("CARGO_MANIFEST_DIR")).join(".."); + let mut files = BTreeMap::new(); + loop { + match Title::from_files(dir, &files) { + Err(MeasureError::Missing(p)) => { + files.insert(p.clone(), std::fs::read(wai.join(&p)).unwrap()); + } + Err(e) => panic!("{dir}: {e}"), + Ok(_) => return files, + } + } +} + +#[test] +fn a_1_gib_declared_clip_is_refused_before_it_is_allocated() { + let _one = ONE_AT_A_TIME.lock().unwrap_or_else(|e| e.into_inner()); + // The second title states limits of 2^40 for every figure: a title lowers a limit + // and never raises one, so the tools' stated limit still refuses the clip. + for dir in ["staged-measure-conformance/refusals/clip-1gib", "staged-measure-conformance/refusals/clip-1gib-limits-raised"] { + let f = files(dir); + let t = Title::from_files(dir, &f).unwrap(); + let start = CUR.load(Relaxed); + PEAK.store(start, Relaxed); + let r = evaluate(&t); + let peak = PEAK.load(Relaxed) - start; + assert!(matches!(&r, Err(MeasureError::Refused { code, .. }) if code == "cost_over_limit"), "{dir}: {r:?}"); + // The refusal reads the header and the cost; it holds nothing sized from the clip. + assert!(peak < 1 << 20, "{dir}: the refusal held {peak} bytes at once"); + check_refusal(&t).unwrap(); + } +} + +#[test] +fn every_refusal_title_is_refused_with_its_code() { + let _one = ONE_AT_A_TIME.lock().unwrap_or_else(|e| e.into_inner()); + let wai = Path::new(env!("CARGO_MANIFEST_DIR")).join(".."); + let mut n = 0; + for e in std::fs::read_dir(wai.join("staged-measure-conformance/refusals")).unwrap().flatten() { + let dir = format!("staged-measure-conformance/refusals/{}", e.file_name().to_string_lossy()); + let t = Title::from_files(&dir, &files(&dir)).unwrap(); + let start = CUR.load(Relaxed); + PEAK.store(start, Relaxed); + check_refusal(&t).unwrap_or_else(|e| panic!("{dir}: {e}")); + assert!(PEAK.load(Relaxed) - start < 1 << 20, "{dir}"); + n += 1; + } + assert!(n > 0); +} diff --git a/wai/wai-rs/tests/vectors/staged_bounds/README.md b/wai/wai-rs/tests/vectors/staged_bounds/README.md index 6a0235530..16f0a6620 100644 --- a/wai/wai-rs/tests/vectors/staged_bounds/README.md +++ b/wai/wai-rs/tests/vectors/staged_bounds/README.md @@ -24,3 +24,13 @@ dispatches, with `wai.video.int_motion` as the fallback: `inert_refinement_prima capability nor, through it, the fallback (`missing_capability`). They are what `staged::tests::decode_base_runs_spec_4` builds, which checks them byte for byte (`WAI_STAGED_BOUNDS_WRITE=1` rewrites them). + +A last pair holds a layer, not a base, past the limit: `layer_bomb_base.wai` (275 +bytes) is a `wai.video.int_motion` base of one 2048 × 2730 frame (16,773,120 bytes of +output, within the tools' stated limit), and `layer_bomb_up2.wai` (587 bytes) is a +`wai.video.int_refine` up2 layer over it whose `WIR1` header declares one 4096 × 5460 +unit: 67,092,480 output bytes and as many symbols, past the stated limit of 2^25 +(`staged::TOOL_DEFAULT_LIMIT`). Every reference tool holds the layer to that limit with +no flag and refuses it (`cost-over-limit`) before any symbol of it is decoded: +`tests/cli_stage.rs` checks `wai stage apply` and `wai_stage_bench`, and the stated +declared output through `wai stage inspect`. diff --git a/wai/wai-rs/tests/vectors/staged_bounds/layer_bomb_base.wai b/wai/wai-rs/tests/vectors/staged_bounds/layer_bomb_base.wai new file mode 100644 index 000000000..279b890c9 Binary files /dev/null and b/wai/wai-rs/tests/vectors/staged_bounds/layer_bomb_base.wai differ diff --git a/wai/wai-rs/tests/vectors/staged_bounds/layer_bomb_up2.wai b/wai/wai-rs/tests/vectors/staged_bounds/layer_bomb_up2.wai new file mode 100644 index 000000000..7999cb738 Binary files /dev/null and b/wai/wai-rs/tests/vectors/staged_bounds/layer_bomb_up2.wai differ