diff --git a/.github/workflows/wai-conformance.yml b/.github/workflows/wai-conformance.yml index 89681b505..46dc4e65e 100644 --- a/.github/workflows/wai-conformance.yml +++ b/.github/workflows/wai-conformance.yml @@ -56,6 +56,11 @@ on: # A library test (staged::tests::staged_delivery_text_lists_the_codes) checks # that this extension's §4.2 and A.6 list the codes the sink reports. - 'wai/extensions/staged-delivery.md' + # A library test (codecs::registry::tests::wire_identifiers_are_in_their_documents) + # checks every wire identifier against the section of these texts it names. + - 'wai/extensions/http-delivery.md' + - 'wai/extensions/moq-streaming-format.md' + - 'wai/extensions/jwp-receipts.md' - 'wai/wai-web/**' - '.github/workflows/wai-conformance.yml' pull_request: @@ -81,6 +86,11 @@ on: # A library test (staged::tests::staged_delivery_text_lists_the_codes) checks # that this extension's §4.2 and A.6 list the codes the sink reports. - 'wai/extensions/staged-delivery.md' + # A library test (codecs::registry::tests::wire_identifiers_are_in_their_documents) + # checks every wire identifier against the section of these texts it names. + - 'wai/extensions/http-delivery.md' + - 'wai/extensions/moq-streaming-format.md' + - 'wai/extensions/jwp-receipts.md' - 'wai/wai-web/**' - '.github/workflows/wai-conformance.yml' @@ -143,12 +153,16 @@ jobs: # of this build on every leg of this matrix, without a word. staged and the integer # media features build the staged-delivery core, its corpus tool and `wai stage`; # stage_measure builds the staged-measurement core and wai_stage_measure; telemetry - # the delivery-telemetry mapping and wai_telemetry. - run: cargo build --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,receipt_claims,staged,video_int,audio_int,splat_int,stage_measure,telemetry + # the delivery-telemetry mapping and wai_telemetry; http the HTTP delivery binding, + # its corpus tool and `wai http`. + run: cargo build --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,receipt_claims,staged,video_int,audio_int,splat_int,stage_measure,telemetry,http # video_package runs the package receipt's dedup tests, which content_dedup feeds. + # The dev profile keeps overflow checks on, so an arithmetic overflow in the HTTP + # binding (http_binding::tests, among them a seeded test of every parser of untrusted + # input) fails here rather than wrapping. - name: Library unit tests (the fixed-point determinism floors) working-directory: wai/wai-rs - run: cargo test --lib --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,oer,receipt_claims,key_discovery,staged,video_int,audio_int,splat_int,stage_measure,telemetry,video_package + run: cargo test --lib --no-default-features --features world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,oer,receipt_claims,key_discovery,staged,video_int,audio_int,splat_int,stage_measure,telemetry,video_package,http # The JPEG AI exact decoder reproduces across platforms only if no platform math # library is on its path. Two guards, each failing the step: clippy's # disallowed-methods (wai-rs/clippy.toml), which each module on the decode path @@ -225,20 +239,21 @@ jobs: # The integer decode path builds under each of its features alone, so a sink can take # one medium without the others, and for wasm32 (a browser sink); `int_payload` arms a # medium only under its feature, and none needs `neural`. So does the staged-measurement - # core (`stage_measure`), which pulls no ONNX and reads no file on wasm32. + # core (`stage_measure`), which pulls no ONNX and reads no file on wasm32. The HTTP + # binding (`http`) performs no I/O, so it builds alone and for wasm32 too. - name: Each integer feature alone, and wasm32 (unix) if: matrix.os != 'windows' shell: bash working-directory: wai/wai-rs run: | set -euo pipefail - for f in neural_int video_int audio_int splat_int staged stage_measure telemetry; do + for f in neural_int video_int audio_int splat_int staged stage_measure telemetry http; do cargo check --lib --no-default-features --features "$f" done # The decode bench prints its machine line from bench_env; no other step builds it. cargo build --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int --bin wai_decode_bench rustup target add --toolchain 1.98.0 wasm32-unknown-unknown - cargo check --lib --target wasm32-unknown-unknown --no-default-features --features video_int,audio_int,splat_int,staged + cargo check --lib --target wasm32-unknown-unknown --no-default-features --features video_int,audio_int,splat_int,staged,http # The staged-measurement core is wasm32-clean: a browser computes the same figures. # Built, not only checked, so codegen and linking of the library run for wasm32. cargo build --lib --target wasm32-unknown-unknown --no-default-features --features stage_measure @@ -254,6 +269,14 @@ jobs: - name: Carriage conformance and the wai CLI (wrap, registry export) working-directory: wai/wai-rs run: cargo test --no-default-features --test carriage --test manifest_builders --test cli_wrap_intent --test cli_registry + # tests/cli_http.rs runs `wai http publish`, `codecs`, `origin` and `bench` as a user + # does: objects named by their content hash with their head proofs, a sequence's union + # and floor, the origin's headers, ranges and If-Range on loopback, and a fetch by + # the byte-range procedure whose record states energy as not measured. Cargo.toml + # declares it with required-features = ["http"]. + - name: The wai http commands + working-directory: wai/wai-rs + run: cargo test --no-default-features --features http --test cli_http # The machine-readable registry (wai/registry/, wai/vocab/v2.*, the table in # wai/REGISTRY.md) is generated from wai-rs/src/codecs/registry.rs. Regenerate it in # memory and fail on any difference; the message says how to regenerate. One leg: the @@ -314,7 +337,7 @@ jobs: working-directory: wai/wai-rs run: | set -euo pipefail - F=world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged,stage_measure,telemetry + F=world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged,stage_measure,telemetry,http v() { echo "::group::$1"; cargo run -q --no-default-features --features "$F" --bin "$2" -- verify "$3"; echo "::endgroup::"; } v worlds wai_world_vectors ../world-conformance v spatial-audio wai_audio_vectors ../audio-conformance @@ -334,6 +357,7 @@ jobs: v signers wai_signer_vectors ../signer-conformance v int-payload wai_int_payload_vectors ../int-payload-conformance v staged wai_staged_vectors ../staged-conformance + v http wai_http_vectors ../http-conformance v staged-measure wai_stage_measure ../staged-measure-conformance v telemetry wai_telemetry ../telemetry-conformance # The pin corpus's second implementation: Python standard library only, sharing no @@ -364,7 +388,7 @@ jobs: shell: pwsh working-directory: wai/wai-rs run: | - $F = "world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged,stage_measure,telemetry" + $F = "world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int,staged,stage_measure,telemetry,http" function v($grp, $bin, $path) { Write-Output "::group::$grp" cargo run -q --no-default-features --features $F --bin $bin -- verify $path @@ -389,6 +413,7 @@ jobs: v signers wai_signer_vectors ../signer-conformance v int-payload wai_int_payload_vectors ../int-payload-conformance v staged wai_staged_vectors ../staged-conformance + v http wai_http_vectors ../http-conformance v staged-measure wai_stage_measure ../staged-measure-conformance v telemetry wai_telemetry ../telemetry-conformance @@ -422,7 +447,8 @@ jobs: # signer_verify.py (every key-discovery vector and the canonical fixture, with its # verdicts required to equal wai_signer_vectors' line for line), and # int_payload_verify.py (the integer payloads and their refusal fixtures), and - # staged_verify.py (the staged-delivery corpus). + # staged_verify.py (the staged-delivery corpus), and http_verify.py (the HTTP delivery + # corpus). # Each exits non-zero on any failure or any difference, and the step fails with it. # One platform: what they check is the reading of the text, not the architecture. independent-verifiers: @@ -476,6 +502,31 @@ jobs: working-directory: wai shell: bash run: python3 staged-conformance/staged_verify.py staged-conformance + # The HTTP delivery corpus's second implementation (http-conformance/README): the head, + # the head proof (every committed proof rebuilt from the text), response checks, plans, + # selection, the byte-range procedure and its fallback, and the codecs union and floor, + # checked first against published BLAKE3 vectors. Standard library only. Its + # mutations and the differential against wai-rs run in wai-http.yml's falsify job. + - name: http_verify.py + working-directory: wai + shell: bash + run: python3 http-conformance/http_verify.py http-conformance + # Outside the corpus: wai_http_vectors fuzz writes seeded cases (Content-Range values, + # codecs values, #WAI-LAYER lines, manifests and their member types, encryption, + # refines, layers declared against their bases, layers over WAI2 components, objects + # past the first request through a tampering origin) with wai-rs's outcomes, and + # http_verify.py must give every one, a refusal by the same code. It prints each + # family's count of cases and of refusals. Two fixed seeds, so a run is reproducible. + - name: The HTTP differential outside the corpus (wai-rs against http_verify.py) + working-directory: wai + shell: bash + run: | + set -euo pipefail + (cd wai-rs && cargo build --no-default-features --features http --bin wai_http_vectors) + for seed in 11 12; do + wai-rs/target/debug/wai_http_vectors fuzz http-conformance --seed "$seed" --count 400 --out "$RUNNER_TEMP/http-fuzz-$seed.json" + python3 http-conformance/http_verify.py cases "$RUNNER_TEMP/http-fuzz-$seed.json" http-conformance + done # The staged-measurement corpus's second implementation (staged-measure-conformance/README): # it counts every object, applies every layer from its base's stage 0, computes the quality # integers, the output tree, the prefixes, the overhead points and the dedup totals itself, diff --git a/.github/workflows/wai-http.yml b/.github/workflows/wai-http.yml new file mode 100644 index 000000000..8b815fd61 --- /dev/null +++ b/.github/workflows/wai-http.yml @@ -0,0 +1,157 @@ +name: WAI HTTP delivery + +# extensions/http-delivery.md: two jobs beside the corpus checks in wai-conformance.yml. +# +# falsify — the HTTP corpus tool and its independent verifier must catch defects. Each +# patch in wai/http-conformance/falsify/ breaks one function of wai-rs (the content +# hash's domain, the root flag, the left-subtree rule, the chunk counter, the component +# digest check, the Content-Range and content-coding checks, ASCII-only numbers, the +# union, layers' place in it and their dispatch, the wai prefix's case, duplicate +# capabilities, a record beside a fallback, the trailing-byte rule, the missing-digest +# plan, the class merge, the base-only threshold, refused refinement responses, stale +# bytes after a mismatch, an unanswered base, the selection digest test); it +# is applied to a fresh worktree, the build must succeed, and `wai_http_vectors verify` +# must fail naming the cases on the patch's first line. Then each `http_verify.py +# --mutate` must fail, and the differential: wai-rs writes random envelopes' covered +# prefixes and head proofs, the Python verifier accepts every one, and refuses exactly +# the ten it changes one chaining-value byte in. +# +# cache-conformance — the cache rules (§9). wai/cache-conformance/run.sh stages the +# corpus's objects behind the reference origin (`wai http origin`) and runs driver.py +# against the origin, which must pass every MUST test, and against four negative-control +# proxies, which must each fail the tests their defect reaches. Labelled caches run only +# where the environment provides them (/opt/wai-cache); this repository holds none, and +# the job summary says when none ran. The reports are uploaded as an artifact. +# +# No workflow_dispatch: it does not start runs on this forge. A weekly schedule keeps the +# jobs exercised between changes. + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + push: + branches: [main] + paths: + - 'wai/wai-rs/src/http_binding/**' + - 'wai/wai-rs/src/container.rs' + - 'wai/wai-rs/src/jwp_hash.rs' + - 'wai/wai-rs/src/bin/wai.rs' + - 'wai/wai-rs/src/bin/wai_http_vectors.rs' + - 'wai/wai-rs/src/bin/wai_cli/http.rs' + - 'wai/wai-rs/src/bin/wai_cli/http_vectors.rs' + - 'wai/wai-rs/Cargo.toml' + - 'wai/http-conformance/**' + - 'wai/cache-conformance/**' + - 'wai/claim-conformance/_blake3.py' + # http_verify.py reads staged_verify.py (and through it int_payload_verify.py), and + # the corpus carries the staged ladder's base and layers. + - 'wai/staged-conformance/staged_verify.py' + - 'wai/staged-conformance/cases/audio_flac_rate_ladder/**' + - 'wai/int-payload-conformance/int_payload_verify.py' + - 'wai/registry/**' + - '.github/workflows/wai-http.yml' + pull_request: + paths: + - 'wai/wai-rs/src/http_binding/**' + - 'wai/wai-rs/src/container.rs' + - 'wai/wai-rs/src/jwp_hash.rs' + - 'wai/wai-rs/src/bin/wai.rs' + - 'wai/wai-rs/src/bin/wai_http_vectors.rs' + - 'wai/wai-rs/src/bin/wai_cli/http.rs' + - 'wai/wai-rs/src/bin/wai_cli/http_vectors.rs' + - 'wai/wai-rs/Cargo.toml' + - 'wai/http-conformance/**' + - 'wai/cache-conformance/**' + - 'wai/claim-conformance/_blake3.py' + # http_verify.py reads staged_verify.py (and through it int_payload_verify.py), and + # the corpus carries the staged ladder's base and layers. + - 'wai/staged-conformance/staged_verify.py' + - 'wai/staged-conformance/cases/audio_flac_rate_ladder/**' + - 'wai/int-payload-conformance/int_payload_verify.py' + - 'wai/registry/**' + - '.github/workflows/wai-http.yml' + schedule: + - cron: '17 4 * * 1' + +jobs: + falsify: + name: HTTP corpus · falsification and differential + runs-on: linux + steps: + - uses: actions/checkout@v5 + + - name: Install Rust 1.98.0 + shell: bash + run: | + export PATH="$HOME/.cargo/bin:$PATH" + command -v rustup >/dev/null 2>&1 || \ + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0 + rustup toolchain install 1.98.0 --profile minimal + rustup default 1.98.0 + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + + # One target directory for every patched build, so only the wai crate rebuilds. + - name: Every patch is caught; every mutation is caught; the differential holds + shell: bash + env: + CARGO_TARGET_DIR: ${{ runner.temp }}/falsify-target + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + bash wai/http-conformance/falsify/run.sh "$RUNNER_TEMP/falsify" + + cache-conformance: + name: cache rules · controls and labelled caches + runs-on: linux + steps: + - uses: actions/checkout@v5 + + - name: Install Rust 1.98.0 + shell: bash + run: | + export PATH="$HOME/.cargo/bin:$PATH" + command -v rustup >/dev/null 2>&1 || \ + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0 + rustup toolchain install 1.98.0 --profile minimal + rustup default 1.98.0 + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Build the reference origin + working-directory: wai/wai-rs + shell: bash + run: cargo build --release --no-default-features --features http --bin wai + + # The verdict: run.sh exits non-zero when the origin fails a MUST test, a control is + # not caught, or a labelled cache fails one. + - name: The origin passes, the controls are caught, labelled caches report + shell: bash + run: WAI="$GITHUB_WORKSPACE/wai/wai-rs/target/release/wai" bash wai/cache-conformance/run.sh "$RUNNER_TEMP/cache" + + # The evidence in the log, whatever becomes of the upload: each report's path and + # SHA-256, and the summary. + - name: The reports, in the log + if: always() + shell: bash + run: | + cd "$RUNNER_TEMP/cache/reports" 2>/dev/null || { echo "no reports were written"; exit 0; } + for f in *; do printf '%s %s\n' "$(sha256sum "$f" | cut -d' ' -f1)" "$RUNNER_TEMP/cache/reports/$f"; done + cat summary.md 2>/dev/null || true + + # An upload this forge cannot take must not decide the job: the verdict is the step + # above it. + - name: Upload the reports + if: always() + continue-on-error: true + uses: actions/upload-artifact@v3 + with: + name: cache-conformance-reports + path: ${{ runner.temp }}/cache/reports diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 888adadd0..1176f4e33 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -6,6 +6,106 @@ wire format may change and such changes are recorded here. ## v0.4.0 — current +### HTTP delivery (Draft) + +- **The extension** [http-delivery](extensions/http-delivery.md) binds envelopes to + HTTP, with no envelope byte and no manifest member changed: + - the proposed media type `application/wai`, not registered; + - `codecs` elements `wai1.`/`wai2.` that list the exhaustive union of the + capabilities a variant may dispatch to or apply, with the floor every object of it + offers carried separately (`X-WAI-FLOOR`, a DASH `SupplementalProperty`); + - objects named by their receipt content hash (`.wai`), immutable, with no + content coding and no `Vary`, and exactly one envelope each; + - the `WHP1` head proof: the BLAKE3 chaining values that prove an envelope's head + against its content hash without the rest of the object; + - verified range fetch of one `WAI2` component that picks as whole-envelope + selection does, and falls back to the whole object when a pick has no digest, a + digest fails, or the plan needs most of the object; + - receipt, key, grant, parameter-set and playlist sidecars; cache rules; priorities, + deadlines and the carriage of enhancement companions, refinement components and + layer objects ([staged-delivery](extensions/staged-delivery.md)); + - HLS and DASH adjacency for on-demand, live and low-latency live; a capability + probe that never asks a user agent about a WAI element. +- **jwp-receipts**: §3 step 4 accepts the content hash a verified head proof + establishes for an object fetched in part, and checks a receipt about a component's + exploded envelope over the envelope the verifier builds. New §4.4 (objects not + delivered) and §4.5 (an HTTP cache is not a relay). §1 and §7.9 point to the HTTP + locations. moq-streaming-format §6 names the HTTP binding. +- **SPEC**: §2 and §7.1 name the HTTP object and the `WAI2` head; the + `wai.video.manifest` row admits `render` and `extras` on a playlist receipt. +- **The registry's wire identifiers.** `codecs::registry::WIRE_IDENTIFIERS` lists the + names the container, the two delivery bindings and key discovery put on the wire + outside a manifest, each with its kind, defining section and status; `wai registry + export` writes them to `registry/capabilities.json` (`wire_identifiers`) and the table + in `REGISTRY.md`, and a library test checks each against the section it names. +- **The reference binding** (`wai-rs`, feature `http`): `http_binding::{head, head_proof, + plan, session, codecs, origin}` — the head and §5.1, the `WHP1` proof built and checked + with BLAKE3's tree, the root checked before any covered byte is read as a head; plans by + priority class; a sans-IO fetch session with §7.1's response checks, component digests, + the whole-object fallback and refinement deadlines; the `codecs` grammar, union, floor + and support; an origin's headers and Range answers. `container::select_entries` is SPEC + §7.1's walk with the digest test passed in, which `WaiMulti::pick` and a client holding + only the head both run, and `companion_indices` / `refinement_indices` bind by entry. + The `wai` tool gains `wai http publish|codecs|origin|vectors|bench`. +- **The corpus** [`http-conformance/`](http-conformance/): objects with their head proofs + and ETag sidecars (a pinned learned image beside AVIF, JPEG XL and PNG of the one + picture; AV1 at two rates with HDR10+ bound by id; a missing and a wrong digest; an AVC + base with an LCEVC enhancement; a segment sequence; Appendix D's envelope; a head of + several chunks) and cases for §4–§7. `wai_http_vectors verify` and the independent + `http_verify.py` (standard library, written from the text, checked first against + published BLAKE3 vectors) agree on every case. Falsification patches and Python + mutations each must be caught, and a differential checks wai-rs's proofs of random + envelopes in Python (`.github/workflows/wai-http.yml`). +- **Cache conformance** [`cache-conformance/`](cache-conformance/): a driver for §9's + rules, run against the reference origin (which must pass), against four + negative-control proxies (which must each be caught) and against labelled caches where + an environment provides them; reports name a cache by label, version and configuration + digest only, and CI uploads them. +- **What review found, fixed** (http-delivery §3.1, §4, §7, §7.1, §10, §13, Appendix D): + - a layer object (a `WAI1` whose capability is refinement-only) is never dispatched: + it is fetched whole at refinement urgency, verified against its cid and handed to + staged delivery, and joins a union after `+` through `refines.base` + (`not_a_layer`, `codecs_layer_unbound`, `layer_tag_malformed`). A `WAI1` whose + capability is a payload is an ordinary object even when it carries `refines`: it is + dispatched, described by its capability, and `not_a_layer` when listed as a layer. + A layer is declared only in the `alt` of its own base's capability; + - a refinement never moves its base: only base-class bytes count toward the 90 % + rule, a refinement is always requested on its own, a refinement with no digest or a + failed one is dropped alone on the range path and on the whole path alike, and a + refused refinement response abandons only it; + - a client presents a base as soon as it verifies and applies its refinements as they + arrive; a refinement's deadline runs from that moment on the caller's clock + (`FetchSession::base_ready`, `tick`), and `wai http bench` reports the time to the + verified base apart from the time to the complete fetch; + - after a component mismatch every byte past `covered_len` is discarded before the + rest is fetched again; a base request never answered is `base_unanswered`; + - `Content-Range` numbers are ASCII digits only; an element beginning `wai` in any + case must be a well-formed WAI element; an empty element or a capability named + twice is malformed; a `WAI1` naming a record beside a payload fallback is described + by its fallback; a `codecs` value with non-ASCII text, and a head proof whose lengths + reach the end of 64 bits, are refused rather than panicking; + - the reference origin caps what it reads of a request head (16 KiB); gives the whole + head a deadline from the connection's accept (default 10 s, `--read-timeout-ms`), + answered by 408 however slowly its bytes trickle in; bounds each connection's whole + life from its accept, writing included (default 60 s, `--max-connection-ms`), so a + peer reading a large response slowly is cut off then; serves at most 64 connections + at once (`--max-connections`), answering 503 to one more, with a cap per peer + address only when `--max-per-peer` is given (it listens on 127.0.0.1, where every + client has one address, and a default cap would let a few reconnecting clients shut + the rest out); decodes `/__hits` as bytes; and refuses `\`, `:` and NUL in a path. + The wire-identifier test reads each name in its section's body. + - a verified base is decided: a component that verified is never read again, and of a + refinement's response only the range requested is kept, so a refinement answered + `200` with the whole object, a byte changed, never reaches the base. Every response + carries the caller's clock (`FetchSession::on_response(…, now_ms)`), the base is + stamped with the time of the response that verified it, and the deadline's + arithmetic saturates. The corpus's `fetch_timed` cases run the session on a clock; + `http_verify.py` agrees on them. + The corpus gains a case for each, and `http_verify.py` agrees with wai-rs on them and + on a seeded differential outside the corpus, in which a refusal must match by its + code and each family reports how many cases it generated and how many were refused. +- No envelope byte or receipt byte changes for existing material. + ### Packaging: a declared policy, its digest, and pieces cut at scene changes - **The extension** [packaging](extensions/packaging.md), first sections: a package is diff --git a/wai/REGISTRY.md b/wai/REGISTRY.md index 5ff7d8222..b6561bcd9 100644 --- a/wai/REGISTRY.md +++ b/wai/REGISTRY.md @@ -8,7 +8,7 @@ together with the class functions of [`wai-rs/src/codecs/mod.rs`](wai-rs/src/cod | file | what | |---|---| -| [`registry/capabilities.json`](registry/capabilities.json) | every registered string with its kind, status, media class, role, fidelity tier, licence class, replicate class, set membership, registering section, edition and registration date; the named identifiers that are not capabilities; the unregistered labels | +| [`registry/capabilities.json`](registry/capabilities.json) | every registered string with its kind, status, media class, role, fidelity tier, licence class, replicate class, set membership, registering section, edition and registration date; the named identifiers that are not capabilities; the unregistered labels; the wire identifiers | | [`registry/capabilities.schema.json`](registry/capabilities.schema.json) | JSON Schema (2020-12) for that file; its enums are the label sets below | | [`vocab/v2.ttl`](vocab/v2.ttl), [`vocab/v2.jsonld`](vocab/v2.jsonld), [`vocab/v2.shapes.ttl`](vocab/v2.shapes.ttl) | the same registry as RDF, a JSON-LD context for manifests, and a SHACL shape that checks a manifest's capability and fallback against it, only the form of its media, and its parameter-set pins (`vocab/README.md`); a violation for a string the registry does not hold means the string is not registered in this revision, not that the envelope fails to conform or cannot be dispatched (SPEC §4, §8) | | the table at the end of this file | the same registry, for reading | @@ -54,6 +54,16 @@ Two further tables hold strings that look like capabilities and are not: - **unregistered labels** are strings the reference implementation uses as labels that no revision registers, each with the reason. +A third holds the **wire identifiers**: the names the container, its delivery +bindings ([http-delivery](extensions/http-delivery.md), +[moq-streaming-format](extensions/moq-streaming-format.md)) and the key discovery +they serve ([jwp-receipts](extensions/jwp-receipts.md) §7) put on the wire outside a +manifest — magic numbers, a media type, `codecs` prefixes, paths, playlist tags and +attributes, document kinds and catalog fields — each with its kind, the section that +defines it, and whether it is defined, provisional (test material only until +registered) or proposed (not registered). A library test checks that each is written +in the section it names. + ## Status - **registered** — the payload format (for a record or derivation, the @@ -250,4 +260,38 @@ and a role are fixed once registered. | `wai.quantum.compile` | a receipt-kind label the wai-quantum crate seals; no revision of SPEC or of an extension registers it, so it is not a registered record | | `wai.quantum.rearrange` | a receipt-kind label the wai-quantum crate seals; no revision of SPEC or of an extension registers it, so it is not a registered record | | `wai.quantum.qbom` | a receipt-kind label the wai-quantum crate seals; no revision of SPEC or of an extension registers it, so it is not a registered record | + +| wire identifier (not a capability) | kind | status | defined in | +|---|---|---|---| +| `#WAI-LAYER` | playlist-tag | defined | http-delivery §10.4 | +| `.wai` | path | defined | http-delivery §5.2 | +| `.wai.hp` | path | defined | http-delivery §6.2 | +| `.wai.receipts` | path | defined | http-delivery §8.2 | +| `/.well-known/wai-grants/` | well-known | defined | http-delivery §8.4 | +| `/.well-known/wai-key-status` | well-known | defined | http-delivery §8.4 | +| `/.well-known/wai-keys` | well-known | defined | jwp-receipts §7.1 | +| `WAI1` | magic | defined | SPEC §2 | +| `WAI2` | magic | defined | SPEC §7.1 | +| `WAI_RECEIPT` | catalog | defined | moq-streaming-format §5 | +| `WHP1` | magic | defined | http-delivery §6.2 | +| `X-WAI-FLOOR` | playlist-attribute | defined | http-delivery §11.1 | +| `application/wai` | media-type | proposed | http-delivery §3 | +| `mr/` | path | defined | http-delivery §8.6 | +| `rcpt/` | path | defined | http-delivery §8.1 | +| `science.transaction.wai.multivariant` | data-id | defined | http-delivery §11.1 | +| `tag:transaction.science,2026:wai/floor` | scheme | defined | http-delivery §11.1 | +| `tag:transaction.science,2026:wai/http-delivery` | scheme | defined | http-delivery §11.1 | +| `wai` | catalog | defined | moq-streaming-format §3 | +| `wai-key-status` | document-kind | defined | jwp-receipts §7.9 | +| `wai-module-grant` | document-kind | defined | jwp-receipts §7.9 | +| `wai-module-grants` | document-kind | defined | http-delivery §8.4 | +| `wai-receipt-index` | document-kind | defined | http-delivery §8.2 | +| `wai1` | codecs-prefix | defined | http-delivery §4.1 | +| `wai1` | fourcc | provisional | http-delivery Appendix B | +| `wai2` | codecs-prefix | defined | http-delivery §4.1 | +| `wai2` | fourcc | provisional | http-delivery Appendix B | +| `waiC` | fourcc | provisional | http-delivery Appendix B | +| `wai_capabilities` | catalog | defined | moq-streaming-format §4 | +| `wai_determinism` | catalog | defined | moq-streaming-format §4 | +| `wai_receipts` | catalog | defined | moq-streaming-format §4 | diff --git a/wai/SPEC.md b/wai/SPEC.md index 7f0831d27..6b6d5d5c2 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -86,6 +86,11 @@ OPTIONAL; §5 says which are RECOMMENDED. All WAI multi-byte integers are little-endian unless stated otherwise. +An envelope delivered over HTTP is one object of the proposed media type +`application/wai`, named by its receipt content hash, and fetched whole or, for a +`WAI2`, one verified component at a time +([http-delivery](extensions/http-delivery.md)). + --- ## 3. Manifest schema @@ -614,7 +619,7 @@ registration and makes no conforming envelope non-conforming (§8). | `wai.neural.dcvc_fm` | DCVC-FM (feature modulation), wide single-model quality range (RGB + YUV) | a wide single-model quality range; shares the inter-frame route | research code, `neural/video_dcvc.rs` | | `wai.video.int_motion` | DETERMINISTIC INTEGER inter-frame — I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS, no float/ONNX; motion is whole-pel or **half-pel** (`mv_precision`), the latter predicting at the half-grid by integer bilinear interpolation (`(a+b+1)>>1`, `(a+b+c+d+2)>>2`) so real footage — which pans a non-integer number of pixels per frame — costs fewer residual bits while staying byte-exact | byte-identical frame-by-frame on every machine (*frame-state-equivalence*, a per-frame portable hash); the temporal extension of the integer contract, where a float inter-frame decode desyncs down the GOP | payload `WIV1`, keyframe 0; `int_video.rs` (the encode tools write a JSON form a converter reads) | | `wai.video.int_hyper` | NEURAL-KEYFRAME inter-frame — the I-frame is the LEARNED integer codec `int_hyper` decoded byte-exact, the P-frames are integer block-MC residuals on top | a learned keyframe you can put in a receipt + integer inter-frames; whole decode integer, byte-identical everywhere | payload `WIV1`, keyframe 1 (a nested `WIH1`); `int_video::decode_pframes_typed` ∘ `int_hyper_synth::decode_conditional_checked` | -| `wai.video.manifest` | ATTESTED PER-SESSION MANIFEST — the OTT delivery layer, not a codec: a personalised HLS manifest (ads stitched at `EXT-X-DISCONTINUITY`, blackouts + geo enforced) rendered as a **pure function of the session *class*** `(content, entitlement, geo, blackout set, ad pod, window)` rather than the viewer, and sealed in an Ed25519 receipt that binds the JCP grant (capability, joule ceiling, `funds_ceiling` = ad budget), the composition, the stitched ads and the policy applied | **manifest-equivalence**: `manifest_hash == hash(render(segments))` and `session_class_hash` are recomputable by anyone holding the bytes, so the ads, blackouts and compose-work stop being dashboard claims; class-determinism means one generation serves every viewer of the class, and `uj_per_viewer` is the amortised energy that a 1:1-unique generator cannot report below its per-viewer cost. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_manifest.rs` | +| `wai.video.manifest` | ATTESTED PER-SESSION MANIFEST — the OTT delivery layer, not a codec: a personalised HLS manifest (ads stitched at `EXT-X-DISCONTINUITY`, blackouts + geo enforced) rendered as a **pure function of the session *class*** `(content, entitlement, geo, blackout set, ad pod, window)` rather than the viewer, and sealed in an Ed25519 receipt that binds the JCP grant (capability, joule ceiling, `funds_ceiling` = ad budget), the composition, the stitched ads and the policy applied. A receipt over a WAI playlist MAY carry `render` and `extras`, the rendering parameters and per-segment lines that playlist adds, both bound through `manifest_hash` ([http-delivery](extensions/http-delivery.md) §8.6) | **manifest-equivalence**: `manifest_hash == hash(render(segments))` and `session_class_hash` are recomputable by anyone holding the bytes, so the ads, blackouts and compose-work stop being dashboard claims; class-determinism means one generation serves every viewer of the class, and `uj_per_viewer` is the amortised energy that a 1:1-unique generator cannot report below its per-viewer cost. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_manifest.rs` | | `wai.video.ssai` | ATTESTED SERVER-SIDE AD INSERTION — a **deterministic** ad decision (greedy by price within the avail duration and the funds remaining) plus a per-impression Ed25519 receipt binding the candidate pod, the manifest it belongs to, the served `creative_hash`es and the VAST quartile each reached | **spend-equivalence**: `budget_spent_micros` is recomputed from the *billable* impressions and rejected if it exceeds the grant's `funds_ceiling` — an overspent receipt fails verification; every creative and quartile is checkable, so a buyer and seller reconcile impressions from the same bytes instead of two logs that only a dashboard joins. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_ssai.rs` | | `wai.video.keys` | ATTESTED CONTENT-KEY RELEASE — the DRM licensing decision as a record: every key release, and every refusal, is an Ed25519 receipt binding the KID, the DRM system, the entitlement demanded and the window enforced, with a BLAKE3 `key_commit` proving *which* key was released without carrying it | **decision-equivalence**: `verify()` re-runs the release policy over the receipt's own `(grant capability, entitlement, issued, expiry)` and rejects any misreported outcome, and enforces granted-iff-committed — so "keys went only to entitled, unexpired sessions" is checkable against the signed log, not asserted. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_keys.rs` | | `wai.video.package` | DETERMINISTIC PACKAGING + ATTESTED SEGMENTMAP — a source sliced into content-hashed segments on GOP boundaries; the ordered `(index → content_hash)` map is a signed Merkle commitment, and identical content (ads, bumpers, replays, simulcast overlap) dedups to one stored/encoded copy | **map-equivalence + verifiable dedup**: `segment_map_root` lets a `wai.video.manifest` reference be checked against the package that produced it, and `verify()` recomputes `unique_segments`/`stored_bytes` from the segments so the storage/encode saving cannot be overstated; `uj_per_unique_segment` is the encode-once energy. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_package.rs` | @@ -1727,6 +1732,10 @@ by which capability happens to be installed. i-th entry corresponds to `renditions[i]` in the manifest. - Payloads block: the rendition byte ranges, packed contiguously. - `n_renditions` MUST equal the manifest's `renditions.length`. +- The bytes from offset 0 to the end of the rendition table are the envelope's + **head**. They give every component's byte range and, with each entry's `sha256`, + its digest, so a sink can fetch and verify one component without the others + ([http-delivery](extensions/http-delivery.md) §7). ### Selection algorithm diff --git a/wai/cache-conformance/README.md b/wai/cache-conformance/README.md new file mode 100644 index 000000000..80fcb83d9 --- /dev/null +++ b/wai/cache-conformance/README.md @@ -0,0 +1,47 @@ +# cache-conformance — HTTP caches under http-delivery §9 + +Checks the cache rules of [http-delivery](../extensions/http-delivery.md) §9 against +an HTTP cache that fronts an origin serving the objects of +[`http-conformance/`](../http-conformance/), and reports, per cache label and version, +whether the configuration meets each rule. + +| file | what | +|---|---| +| `tests.json` | the tests: id, rule, level (MUST or informative) and what each checks | +| `driver.py` | stages the origin's files and runs the tests against a cache; standard library only, with `../claim-conformance/_blake3.py` to check each object against its content hash | +| `mutators.py` | four proxies that each break §9 in one way: the negative controls | +| `run.sh` | stages, starts `wai http origin`, and runs the driver against the origin, each control and each labelled cache | + +## What runs + +1. **The origin, directly** (`origin-direct`): the reference origin + (`wai http origin`) must pass every MUST test. It answers byte ranges and `If-Range`, + serves a live-edge alias with a shorter `max-age`, sends §8.4's `Cache-Control` for + the key log, and counts the requests it receives (`--count-hits`) so the informative + tests can tell whether a cache answered from its store. +2. **The negative controls** (`control-flip`, `control-gzip`, `control-truncate`, + `control-vary`): the driver must fail, and must name the tests each defect reaches — + `flip` C1, C3 and C11; `gzip` C7; `truncate` C1 and C2; `vary` C13. A control the + driver passes is a defect in the driver. +3. **Labelled caches.** An environment that provides them lists their labels in + `/opt/wai-cache/labels`; each `/opt/wai-cache/