Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 17 Packages Wiki Activity Actions 60

wai: the integer codecs' binary payloads, pinned in place #139

Merged
dcharlot merged 35 commits from wai/land-16-binary-payloads into main 2026-10-04 14:05:11 -04:00
Conversation 0 Commits 35 Files changed 143 +10379 -549
143 changed files with 10379 additions and 549 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.github/workflows/wai-byte-equality.yml vendored
View file

@ -14,7 +14,8 @@ name: wai cross-architecture byte-equality
# (`wai.video.int_hyper`) whose I-frame comes out of a learned conditional codec,
# and the MLIC++-topology codec (`wai.neural.int_mlicpp`), whose entropy network runs
# before every checkerboard pass and must give every platform the encoder's scale
# bucket for every element. Video is the strictest leg — inter-frame decode is a
# bucket for every element. The binary payloads SPEC §5 pins (WIV1, WIH1) are parsed
# with the wire parser a sink runs and decoded to the same digests. Video is the strictest leg — inter-frame decode is a
# feedback loop, so one divergent half-pel rounding does not stay one byte, it
# predicts the next frame and compounds for the rest of the clip.
#
@ -38,6 +39,7 @@ on:
- 'wai/wai-rs/src/int_transform.rs'
- 'wai/wai-rs/src/int_hyper_synth.rs'
- 'wai/wai-rs/src/int_mlicpp.rs'
- 'wai/wai-rs/src/int_payload/wire.rs'
# The fast kernels int_synth, int_transform and int_hyper_synth dispatch to
# by default: the goldens here decode through them.
- 'wai/wai-rs/src/int_kernels.rs'
@ -62,6 +64,7 @@ on:
- 'wai/wai-rs/src/int_transform.rs'
- 'wai/wai-rs/src/int_hyper_synth.rs'
- 'wai/wai-rs/src/int_mlicpp.rs'
- 'wai/wai-rs/src/int_payload/wire.rs'
# The fast kernels int_synth, int_transform and int_hyper_synth dispatch to
# by default: the goldens here decode through them.
- 'wai/wai-rs/src/int_kernels.rs'

55
.github/workflows/wai-conformance.yml vendored
View file

@ -30,6 +30,9 @@ on:
- 'wai/glossary/**'
- 'wai/SPEC.md'
- 'wai/REGISTRY.md'
# A library test checks SPEC §5's pinned integer payloads against this
# extension's sections, so an edit of it is checked here too.
- 'wai/extensions/integer-payloads.md'
- 'wai/wai-web/**'
- '.github/workflows/wai-conformance.yml'
pull_request:
@ -49,6 +52,9 @@ on:
- 'wai/glossary/**'
- 'wai/SPEC.md'
- 'wai/REGISTRY.md'
# A library test checks SPEC §5's pinned integer payloads against this
# extension's sections, so an edit of it is checked here too.
- 'wai/extensions/integer-payloads.md'
- 'wai/wai-web/**'
- '.github/workflows/wai-conformance.yml'
@ -153,7 +159,16 @@ jobs:
# decode allocates anything sized from it.
# tests/int_audio.rs and tests/int_splat.rs decode the committed wai.audio.int_codec and
# wai.splat.int_codec vectors against the hashes the Python integer reference computed at
# export, and check each vector's metadata. Cargo.toml declares them, and the other
# export, and check each vector's metadata. `--lib` runs int_payload's unit tests: each
# binary payload (WIV1, WIH1, WIA1, WIS1) against the JSON it is converted from, under
# every kernel policy, and decode_int for every integer capability in this build, which
# has no `neural`. tests/int_payload.rs refuses every fixture of int-payload-conformance
# with its code, every strict prefix of every payload, and runs a byte-change sweep in
# which nothing may panic; tests/int_payload_preflight.rs (its own binary, a counting
# allocator) checks that a header past the caps is refused before anything sized from it
# is allocated; tests/int_payload_cost.rs (its own binary, a live-bytes allocator) checks
# that int_payload::declared_cost bounds what each decode holds, and, in the release
# step only, decodes a 2^30-byte clip a 3 KB payload codes. Cargo.toml declares them, and the other
# feature-gated targets named here, with required-features, so if this step ever loses a
# feature, cargo stops with an error instead of compiling an empty binary that reports ok.
# Each committed vector's decoded output is also checked against the SHA-256 in
@ -163,10 +178,24 @@ jobs:
# wai.generate.spectral vector, which `generate` enables.
- name: Fast integer kernels ≡ reference kernels (overflow checks on)
working-directory: wai/wai-rs
run: cargo test --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate
run: cargo test --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate --test int_payload --test int_payload_preflight --test int_payload_cost
- name: Fast integer kernels ≡ reference kernels (release, overflow checks off)
working-directory: wai/wai-rs
run: cargo test --release --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate
run: cargo test --release --no-default-features --features neural_int,video_int,audio_int,world_int,splat_int,generate --lib --test int_kernels_vectors --test int_kernels_geometry --test int_mlicpp --test int_mlicpp_preflight --test int_audio --test int_splat --test generate --test int_payload --test int_payload_preflight --test int_payload_cost
# The integer decode path builds under each of its features alone, so a sink can take
# one medium without the others, and for wasm32 (a browser sink); `int_payload` arms a
# medium only under its feature, and none needs `neural`.
- name: Each integer feature alone, and wasm32 (unix)
if: matrix.os != 'windows'
shell: bash
working-directory: wai/wai-rs
run: |
set -euo pipefail
for f in neural_int video_int audio_int splat_int; do
cargo check --lib --no-default-features --features "$f"
done
rustup target add --toolchain 1.98.0 wasm32-unknown-unknown
cargo check --lib --target wasm32-unknown-unknown --no-default-features --features video_int,audio_int,splat_int
# carriage: every registered capability round-trips byte-exact. cli_wrap_intent and
# cli_registry run the `wai` binary, which builds without the codecs feature: the
# intent rule of `wai wrap`, its refusal of candidate capabilities (SPEC §5), its media
@ -237,7 +266,7 @@ jobs:
working-directory: wai/wai-rs
run: |
set -euo pipefail
F=world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery
F=world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int
v() { echo "::group::$1"; cargo run -q --no-default-features --features "$F" --bin "$2" -- verify "$3"; echo "::endgroup::"; }
v worlds wai_world_vectors ../world-conformance
v spatial-audio wai_audio_vectors ../audio-conformance
@ -255,6 +284,7 @@ jobs:
v claims wai_claim_vectors ../claim-conformance
v pin wai_pin_vectors ../pin-conformance
v signers wai_signer_vectors ../signer-conformance
v int-payload wai_int_payload_vectors ../int-payload-conformance
# The pin corpus's second implementation: Python standard library only, sharing no
# code with wai-rs. One leg is enough — it checks digests and listing rules, which
# do not depend on the architecture — and the macOS host has python3.
@ -283,7 +313,7 @@ jobs:
shell: pwsh
working-directory: wai/wai-rs
run: |
$F = "world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery"
$F = "world,audio_scene,haptic,splat4d,score,provenance,neural_int,film,quantum,quantum_ops,oer,receipt_claims,key_discovery,video_int,audio_int,splat_int"
function v($grp, $bin, $path) {
Write-Output "::group::$grp"
cargo run -q --no-default-features --features $F --bin $bin -- verify $path
@ -306,6 +336,7 @@ jobs:
v claims wai_claim_vectors ../claim-conformance
v pin wai_pin_vectors ../pin-conformance
v signers wai_signer_vectors ../signer-conformance
v int-payload wai_int_payload_vectors ../int-payload-conformance
# The generated v2 vocabulary (wai/vocab/v2.*) is checked as a consumer uses it:
# sample manifests, typed and untyped, WAI1 and WAI2, are expanded through the
@ -335,11 +366,12 @@ jobs:
# differential.py (both verifiers' verdicts on generated inputs the corpus does not hold,
# under a fixed seed, so a run is reproducible), jwp_verify.py (every JWP vector), and
# signer_verify.py (every key-discovery vector and the canonical fixture, with its
# verdicts required to equal wai_signer_vectors' line for line).
# verdicts required to equal wai_signer_vectors' line for line), and
# int_payload_verify.py (the integer payloads and their refusal fixtures).
# Each exits non-zero on any failure or any difference, and the step fails with it.
# One platform: what they check is the reading of the text, not the architecture.
independent-verifiers:
name: independent verifiers · claims, JWP and key discovery (Python)
name: independent verifiers · claims, JWP, key discovery and integer payloads (Python)
runs-on: linux
steps:
- uses: actions/checkout@v5
@ -371,6 +403,15 @@ jobs:
shell: bash
run: cargo build --no-default-features --features key_discovery --bin wai_claim_vectors --bin wai_jwp_vectors --bin wai_signer_vectors
# The integer payload corpus's second implementation (int-payload-conformance/README):
# it parses every payload with the extension's rules and check order, converts every
# JSON source independently and requires the committed bytes, decodes the WIV1
# keyframe-0 payloads in full to their sidecars' SHA-256, and checks every refusal
# fixture. Standard library only, so it needs nothing installed.
- name: int_payload_verify.py
working-directory: wai
shell: bash
run: python3 int-payload-conformance/int_payload_verify.py int-payload-conformance
- name: claim_verify.py, differential.py, jwp_verify.py, signer_verify.py
working-directory: wai
shell: bash

4
.github/workflows/wai-energy-receipts.yml vendored
View file

@ -70,6 +70,8 @@ on:
- 'wai/extensions/**'
- 'wai/SPEC.md'
- 'wai/CHANGELOG.md'
# tests/video_receipt.rs reads the WIV1 refusal fixtures.
- 'wai/int-payload-conformance/**'
- '.github/workflows/wai-energy-receipts.yml'
pull_request:
paths:
@ -79,6 +81,8 @@ on:
- 'wai/extensions/**'
- 'wai/SPEC.md'
- 'wai/CHANGELOG.md'
# tests/video_receipt.rs reads the WIV1 refusal fixtures.
- 'wai/int-payload-conformance/**'
- '.github/workflows/wai-energy-receipts.yml'
jobs:

63
.github/workflows/wai-neural-gates.yml vendored
View file

@ -16,7 +16,10 @@ name: WAI neural sink gates
# committed int_mlicpp vector and the int_synth corpus: the form-less and set-form
# pins through the strict gate, the int_synth pin over the spec it reads, the
# several-file capabilities refusing a form-less pin, a pinned decode reading only
# the bytes it verified, a WAI2 envelope whose pins pick the held model, the strict
# the bytes it verified, an unpinned decode at a sink with a key log checked over
# every identity of what it read once and reading only that (or, holding the
# capability's sets to §7.9, the part §7.9 authorised), an encrypted envelope inert at the
# sink (SPEC §6.4 step 2), a WAI2 envelope whose pins pick the held model, the strict
# rule read for the capability decoded through (a sound fallback behind an unsound
# primary), SPEC §4 step 6's order of reasons after a set_incomplete decode, and the
# replicate_* reason the strict WAI2 policy reports for an entry it leaves out,
@ -43,6 +46,7 @@ on:
- 'wai/wai-rs/src/ffi.rs'
- 'wai/wai-rs/src/ffi_neural.rs'
- 'wai/wai-rs/tests/ffi_neural_pin.rs'
- 'wai/wai-rs/tests/neural_cost_limit.rs'
- 'wai/wai-rs/src/int_synth.rs'
- 'wai/wai-rs/src/ort_session.rs'
- 'wai/neural-conformance/vectors/**'
@ -50,6 +54,17 @@ on:
- 'wai/wai-rs/tests/neural_native_decode.rs'
- 'wai/wai-rs/src/int_mlicpp.rs'
- 'wai/wai-rs/tests/vectors/int_mlicpp/**'
# The integer capabilities the neural sink delegates to int_payload::decode_int,
# and the vectors its tests decode through it.
- 'wai/wai-rs/src/int_payload/**'
- 'wai/wai-rs/src/int_video.rs'
- 'wai/wai-rs/src/int_hyper_synth.rs'
- 'wai/wai-rs/src/int_audio.rs'
- 'wai/wai-rs/src/int_splat.rs'
- 'wai/wai-rs/tests/vectors/int_hyper/**'
- 'wai/wai-rs/tests/vectors/int_video/**'
- 'wai/wai-rs/tests/vectors/int_audio/**'
- 'wai/wai-rs/tests/vectors/int_splat/**'
- 'wai/wai-rs/Cargo.toml'
- 'wai/wai-rs/Cargo.lock'
- '.github/workflows/wai-neural-gates.yml'
@ -68,6 +83,7 @@ on:
- 'wai/wai-rs/src/ffi.rs'
- 'wai/wai-rs/src/ffi_neural.rs'
- 'wai/wai-rs/tests/ffi_neural_pin.rs'
- 'wai/wai-rs/tests/neural_cost_limit.rs'
- 'wai/wai-rs/src/int_synth.rs'
- 'wai/wai-rs/src/ort_session.rs'
- 'wai/neural-conformance/vectors/**'
@ -75,6 +91,17 @@ on:
- 'wai/wai-rs/tests/neural_native_decode.rs'
- 'wai/wai-rs/src/int_mlicpp.rs'
- 'wai/wai-rs/tests/vectors/int_mlicpp/**'
# The integer capabilities the neural sink delegates to int_payload::decode_int,
# and the vectors its tests decode through it.
- 'wai/wai-rs/src/int_payload/**'
- 'wai/wai-rs/src/int_video.rs'
- 'wai/wai-rs/src/int_hyper_synth.rs'
- 'wai/wai-rs/src/int_audio.rs'
- 'wai/wai-rs/src/int_splat.rs'
- 'wai/wai-rs/tests/vectors/int_hyper/**'
- 'wai/wai-rs/tests/vectors/int_video/**'
- 'wai/wai-rs/tests/vectors/int_audio/**'
- 'wai/wai-rs/tests/vectors/int_splat/**'
- 'wai/wai-rs/Cargo.toml'
- 'wai/wai-rs/Cargo.lock'
- '.github/workflows/wai-neural-gates.yml'
@ -115,8 +142,40 @@ jobs:
pinned_decode_reads_only_verified_bytes wai2_decode_multi_picks_the_verified_rendition \
strict_gate_reads_the_capability_it_decodes_through set_incomplete_at_step_3_outranks_the_fallback_refusal \
decode_multi_strict_reports_replicate_reasons strict_gate_ignores_a_held_capability_it_cannot_dispatch \
int_hyper_decodes_through_the_neural_dispatch int_synth_check_5_covers_the_spec_a_model_directory_holds \
an_integer_capability_without_a_decoder_moves_on_to_the_fallback \
a_listing_that_does_not_match_the_files_read_authorises_nothing \
a_pinned_set_revoked_under_its_file_digest_is_refused_through_a_listing_pin \
a_granted_listing_authorises_only_the_files_it_names \
an_encrypted_envelope_is_inert_at_the_reference_sink \
a_revoke_only_key_log_checks_and_reads_the_set_once \
2>&1 | tee "$RUNNER_TEMP/neural-gates.log"
grep -q "test result: ok. 12 passed; 0 failed" "$RUNNER_TEMP/neural-gates.log"
grep -q "test result: ok. 20 passed; 0 failed" "$RUNNER_TEMP/neural-gates.log"
# The integer video, audio and splat capabilities through the neural dispatch, which
# delegates them to int_payload::decode_int: each decodes to Decoded::Int with its
# committed SHA-256, and wai.video.int_motion is advertised with ModelRegistry::builtin.
- name: The neural sink decodes the integer video, audio and splat payloads
working-directory: wai/wai-rs
shell: bash
run: |
set -euo pipefail
cargo test --no-default-features --features neural,video_int,audio_int,splat_int --test neural_native_decode -- \
--exact integer_media_decode_through_the_neural_dispatch \
2>&1 | tee "$RUNNER_TEMP/neural-int.log"
grep -q "test result: ok. 1 passed; 0 failed" "$RUNNER_TEMP/neural-int.log"
# The registry's cost limit refuses an integer payload over it before any symbol is
# decoded (cost_over_limit) and moves on to the next entry; the C ABI refuses an
# integer video it has no output kind for before decoding it; and the declared
# cost over a registry's set, and through the C ABI given the set's path, is the
# library's. Its own binary: a live-bytes allocator measures each refusal.
- name: The neural sink applies the declared cost before decoding
working-directory: wai/wai-rs
shell: bash
run: |
set -euo pipefail
cargo test --no-default-features --features neural,ffi,video_int,audio_int,splat_int --test neural_cost_limit \
2>&1 | tee "$RUNNER_TEMP/neural-cost.log"
grep -q "test result: ok. 3 passed; 0 failed" "$RUNNER_TEMP/neural-cost.log"
- name: A key log refuses a revoked pinned set at the neural sink (SPEC §3.1 check 5)
working-directory: wai/wai-rs
shell: bash

22
.github/workflows/wai-web-wasm.yml vendored
View file

@ -13,7 +13,8 @@ name: wai-web wasm32
# - demo/verify_pack.mjs: the browser writers `pack` and `packMulti` refuse
# every candidate the export lists, in every place an envelope can name one;
# - a byte comparison of the demo's staged int_synth copies with the corpus;
# - the crate's wasm_tests under wasm-bindgen-test-runner.
# - the crate's wasm_tests under wasm-bindgen-test-runner, and its native unit
# tests on the host.
#
# The full-scale int_hyper wasm decode needs vectors regenerated from a
# pretrained model; wai-sota-vectors.yml runs it after regenerating them.
@ -85,10 +86,29 @@ jobs:
CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=wasm-bindgen-test-runner \
cargo test --locked --target wasm32-unknown-unknown 2>&1 | tee "$RUNNER_TEMP/wai-web-wasm.log"
grep -q "test result: ok. 4 passed; 0 failed" "$RUNNER_TEMP/wai-web-wasm.log"
# The crate's native unit tests, run on the host: the pure-Rust halves of the
# bindings, which wasm_tests does not run. They decode the committed integer
# payloads to their sidecars' SHA-256 (int_payload::tests) and check that the
# writers `pack` and `packMulti` refuse a payload that is not the format SPEC §5
# pins its integer capability to (tests). The count check turns a test that
# stopped being compiled into a failure instead of a silent "0 passed".
- name: wai-web native unit tests
shell: bash
working-directory: wai/wai-web
run: |
set -euo pipefail
cargo test --locked --lib 2>&1 | tee "$RUNNER_TEMP/wai-web-native.log"
grep -q "test result: ok. 8 passed; 0 failed" "$RUNNER_TEMP/wai-web-native.log"
- name: wasm32 int_synth decode equals the frozen native corpus
shell: bash
working-directory: wai/wai-web
run: node demo/verify_int_synth_wasm.mjs ./pkg-node/wai_web.js ../neural-conformance
# decodeWiv1, decodeWih1, decodeWia1 and decodeWis1 on the committed integer
# payloads give the SHA-256 their sidecars state, and a JSON form is refused.
- name: wasm32 integer payload decode equals the native sidecars
shell: bash
working-directory: wai/wai-web
run: node demo/verify_int_payload_wasm.mjs ./pkg-node/wai_web.js ../wai-rs/tests/vectors
- name: The browser menu is registered and names no candidate
shell: bash
working-directory: wai/wai-web

6
wai/.gitattributes vendored
View file

@ -1,3 +1,9 @@
# REGISTRY.md carries a table `wai registry export` generates, which a library test
# compares byte for byte (as registry/ and vocab/v2.* are): no line-ending conversion.
REGISTRY.md -text
# The binary payloads of the integer codecs (extensions/integer-payloads.md) beside
# their JSON sources: pinned by SHA-256, so never converted.
*.wiv1 -text
*.wih1 -text
*.wia1 -text
*.wis1 -text

178
wai/CHANGELOG.md
View file

@ -6,6 +6,184 @@ wire format may change and such changes are recorded here.
## v0.4.0 — current
### The integer codecs' payloads pinned in place: `WIV1`, `WIH1`, `WIA1`, `WIS1`
- **Pinned in place** (SPEC §5, *Payloads of the WAI-native integer codecs*, and the
new extension [integer-payloads](extensions/integer-payloads.md)): the payload of
`wai.video.int_motion` is `WIV1` with keyframe 0; of `wai.video.int_hyper`, `WIV1`
with keyframe 1, a nested `WIH1`; of `wai.neural.int_hyper`, `WIH1`; of
`wai.audio.int_codec`, `WIA1`, which carries the sample rate; of
`wai.splat.int_codec`, `WIS1`, which carries each attribute's presentation range.
These strings' rows named a decoder and no byte layout; three of them
(`wai.video.int_motion`, `wai.audio.int_codec`, `wai.splat.int_codec`) also named
the tool that wrote a JSON bitstream, which SPEC §5 called the bytes the codec
normally emits. So the pin changes what these strings carry. SPEC §8 gains
*Completing a registration*, which permits it for a WAI-native string whose row
fixed no byte layout and lists the envelopes it makes non-conforming; from this
revision their formats are fixed as SPEC §8 states. No new string is registered.
`codecs::PINNED_INT_PAYLOADS` is the one table of strings and magics.
- **The JSON is a converter input, never a payload.** The encode and export tools
still write JSON; `int_payload::{wiv1,wih1,wia1,wis1}_from_json` convert it and
state what it left implicit (an absent `mv_precision` is 1; the first frame carries
no motion vectors; a short motion-vector list is padded with `(0, 0)`; the audio
sample rate comes from the export's `audio.json`; a splat range must be exactly a
binary32 value; the JSON `afrac` must be the model's; the channel count must match
the medians, tables and offsets). *Envelopes this makes
non-conforming:* a `WAI1` or `WAI2` that carries anything but the pinned format
under these strings, a tool's JSON output among them. It still parses; a sink
refuses it at decode (a JSON form with `bad_magic`) and does not try the fallback.
Every emit path refuses to write it: `Wai::to_emit_bytes` and
`WaiMulti::to_emit_bytes` (so the C packers `wai_envelope_pack` and
`wai_envelope_pack_multi`, `WAI_ERR_INVALID_ARG`), `wai-web`'s `pack` and
`packMulti`, and `wai wrap`. They run `container::check_int_payload_framing`, which
parses the payload with the wire parser a sink runs and the capability's keyframe
kind, so it checks every parse rule of the extension's §2–§6 and leaves to the
decode only what needs a parameter set or the streams (§7); `wai inspect` reports
what it finds. A refused emit is the new `EmitError::IntPayload`. An encrypted
envelope's payload is ciphertext (SPEC §6), so the emit paths skip it and a sink
checks the plaintext after decrypting (SPEC §6.4); `wai inspect` says so. No
committed envelope or conformance corpus carried such a payload. The reference
dispatchers (`Wai::dispatch_verified`, `WaiMulti::pick`, and so the neural sink,
its C ABI and `wai-web`'s `dispatchCapability`) implement SPEC §6.4 step 2: they
open no scheme, so an envelope whose manifest declares an encryption other than
`"none"` is inert at them (`unsupported_encryption`; `container::Inert::Encrypted`,
`container::Skip::Encrypted`, `neural::DecodeError::Encrypted`), and a payload sent
in the clear under a claim of encryption is not decoded as plaintext. A `WAI2`
manifest's `security` member (now stated in SPEC §7.1) must be a security object to
be written (`EmitError::Security`, `container::manifest_v2_security`); one that is
not is read as encrypted.
- **One integer decode path.** `int_payload::decode_int` decodes the strings above
and `wai.neural.int_synth` and `wai.neural.int_mlicpp` from parameter files held in
memory, with no ONNX runtime and no filesystem, natively and for `wasm32`, through a
dispatchable capability only. A refusal is one code from a closed list
([integer-payloads](extensions/integer-payloads.md) §8). `int_payload::resolve`
verifies a pin over sets held in memory and runs SPEC §3.1's check 5 for a pinned
decode and over the set an unpinned decode reads. The neural sink delegates every
integer capability to it: integer video, PCM and splat attributes come back as
`neural::Decoded::Int`, never converted; `ModelRegistry::builtin` advertises
`wai.video.int_motion`, which reads no set; the C ABI refuses an `Int` output. A
capability this build has no integer decoder for (`unsupported_capability`) moves
dispatch on to the fallback, as SPEC §4 treats a capability a sink cannot decode.
Check 5 covers every identity of the bytes a decode reads (SPEC §3.1): after a pin
verifies, it runs again over the listing of the members read and a one-file set's
file digest (`int_payload::read_identities`), so a set revoked under one identity
is not reached through a pin naming it by another; and a listing beside an
unpinned set counts only when it matches the files read. Outside the integer path,
at a sink that holds a key log, an unpinned decode reads what it can read once —
the registered file of a one-file set, or every file below the directory its
decoder reads from (`pin::read_dir_set`) — and check 5 runs over every digest a pin
over it could name: the file's, or the directory's derived listing
(`pin::listing_of_dir`) and each file's, and each listing beside them that matches.
The decode reads only those bytes, never the path again; where the key log holds
the capability's sets to jwp-receipts §7.9, only the part §7.9 authorised (the
whole, the files a granted listing names, or one granted file). So a revoked set
is refused with no listing beside it, a granted listing authorises the files it
names and nothing else, a file changed after the check is not read, and a file the
decode needs outside the set is `set_incomplete` (SPEC §4, *Module authorisation*).
`pin::PinAuthority` gains `requires_grant`, with a default.
*Breaking:* `neural::Decoded` gains the variant `Int` and is now `#[non_exhaustive]`,
so a `match` on it outside this crate needs a wildcard arm; `neural::ParamSource`
gains `UnpinnedRead` (an unpinned integer set read into memory once) and
`Checked` (what an unpinned decode at a sink with a key log reads, read once) and is
`#[non_exhaustive]` too; `int_payload::IntParams` is opaque, built by
`int_payload::resolve` (or `IntParams::none()`).
- **Checked decoders.** `int_video`, `int_audio` and `int_splat` decode a typed
stream that the binary and JSON forms both read into. Every size is capped at
`2^26` elements (a `WIV1` clip at `2^30` bytes and `2^20` frames, its motion-vector
field at `2^26` elements) the same on every target, every network's shapes are
planned before anything is allocated, every latent and layer is checked for
exactness, and a stream that runs out is refused at the first symbol it cannot
give, so a build with overflow checks and one without accept the same bytes. A
frame, a clip or a latent grows only as its symbols are decoded, to its declared
size and no further, so a short stream stops early; `int_video` holds frames as
RGB8. A stream that carries its symbols is decoded in full, and a few kilobytes can
code a clip at the cap, so the header bounds the work: a sink reads the declared
cost before decoding (`int_payload::declared_cost`, `container::declared_size`, the
C ABI's `wai_int_payload_declared`, `wai-web`'s `intPayloadCost`) and applies its own
size policy (integer-payloads §7, *Memory and work*). The cost counts the networks'
multiply-accumulates layer by layer, exactly (`DeclaredCost::macs`, a `u128`;
`macs` on `intPayloadCost`, a `BigInt`). The reference sink applies one:
`neural::ModelRegistry::with_cost_limit` refuses a payload whose declared cost
exceeds the limit before any symbol is decoded, with the new code `cost_over_limit`
(`int_payload::check_cost`, `neural::DecodeError::OverCostLimit`), and dispatch moves
on as for a capability the sink cannot decode (§8). `neural::declared_cost` reads the
cost over the set a registry resolved, pinned or not. The C ABI's
`wai_int_payload_declared` takes the set's path (or null for the header alone) and
gives the working bytes and the multiply-accumulates too (`wai_int_declared_t`
gains `working_bytes`, `macs_lo`, `macs_hi` and `cost_known`); its neural decode
entry points no longer decode integer video, PCM or splat attributes, which they
have no output kind for, and refuse them before decoding. A `WIA1`'s latent may be no
longer than its `t_out` needs (`geometry_mismatch`). The JSON reader
(`int_video::VideoJson`) caps every size before it pads a motion-vector list, and
reads a field that does not fit the target's `usize` as absent rather than
truncating it. In the browser, `decodeWiv1` keeps the decoder's frames
(`DecodedVideo.frame(i)`) instead of copying the clip into one buffer.
- **One refusal order.** The extension states the order of a decode's checks
exactly (§7): the set's members, the set against the payload, every network
planned with its caps, the shapes, then each stream symbol by symbol, a run-out
before the value it would give. Networks are planned layer by layer in 64-bit
arithmetic (`int_hyper_synth::plan`, the audio synthesis's plan), a size past 64
bits being past the cap, so a 32-bit and a 64-bit sink refuse a malformed network
with the same code; a planned network's parameter lengths are checked before any
symbol is decoded, and a convolution's padded side is capped. A set's entropy
tables obey §2's length rule (3 to 4097 entries). The parser reads a table of one
per channel table by table. A `WIV1` keyframe's planned size is checked against the
clip's `h × w` before any of the keyframe's symbols is decoded, so
`int_payload::declared_cost` and the decode refuse the same bytes with the same
code; the Rust verifier checks that for every fixture. Fixtures pin each case.
- **One header for a one-frame clip.** A `WIV1` of one frame codes no motion
vector, so its `mv_precision` and `block` must be 1 (`field_range` otherwise); the
converters write them so. The extension's §2 says a payload's hash identifies the
payload, not its output (unused table entries, say, code the same clip in other
bytes), and that one output is identified by its decoded digest (a clip's
`clip_hash`).
- **Canonical streams.** After its last symbol, each rANS stream of these payloads
must have been read to its last word and end in the state `2^31`, where an encoder
starts (`rans::RansDecoder::is_canonical_end`); otherwise the payload is refused
with the new code `stream_noncanonical`. An unread word or another starting state
coded the same symbols in other bytes. Every committed stream, the JSON forms'
included, ends canonically. `int_hyper_synth` gains
`decode_conditional_checked`; `int_synth` refuses a side of `2^64` or more instead of
shifting by it.
- **Parameter sets.** SPEC §5 names the members the integer decoders read:
`g_s.whm`, `h_s.whm` and `entropy.json` for `wai.neural.int_hyper` and
`wai.video.int_hyper`. `wai.audio.int_codec` (`model.wam`) and
`wai.splat.int_codec` (`g_s.whm`) are registered as one file, which widens their
registrations (SPEC §8). `wai registry export` writes nothing new: no exported field
records a set's shape. A `WHM1` or `WAM1` network holds nothing after its last layer
(`IntModel::from_bytes` and `AudioModel::from_bytes` refuse trailing bytes; every
committed network has none), so one network has one file and one digest.
- **Receipts.** A `wai.video.int_motion` receipt identifies its `WIV1` payload
(`VideoReceipt::seal_decode_wiv1`, `verify_reconstruction_wiv1`), and
`wai_video_meter` seals over it. The entry points that read the JSON form are
deprecated; on a conforming JSON bitstream their output is unchanged, so every
receipt sealed over a committed one, and every pinned digest of one, still
verifies. They now read the JSON through `int_video::VideoJson` and the checked
decoder, so they refuse JSON the old decoder took: a stream that runs out (it
decoded on from the exhausted state) or does not end canonically, a table that
breaks the table rules, `q` outside 1..=65535, an `mv_precision` other than 1 or 2,
other than one motion-vector list per frame, vectors on the first frame or more
vectors than blocks, a vector outside `i16`, a zero size, and a size past the caps. jwp-receipts §2.1's object content hash moves to its own
module and feature (`jwp_hash`), which `receipt_claims` enables;
`claim_receipt::object_content_hash` re-exports it, and no corpus changed.
- **JSON readers that refuse.** `IntModel::try_from_json` and
`CondBitstream::try_from_value` read the conditional codec's debug JSON without
panicking; `wai-web`'s `decodeIntHyper` and `decodeIntHyperBin` read it through
them, `CondEntropy::try_from_value` and `decode_conditional_checked`, so an empty
model or bitstream, or sizes past the caps, are refused. `int_video::VideoJson`
refuses a frame whose motion vectors are not a list.
- **Browser.** `wai-web` decodes the four formats (`decodeWiv1`, `decodeWih1`,
`decodeWia1`, `decodeWis1`); the entry points that read the JSON forms are for
debugging and conversion.
- **Conformance.** The payloads sit beside their JSON sources in
`wai-rs/tests/vectors/`, written by `wai_int_payload_vectors`, with refusal fixtures
and an index in `int-payload-conformance/`; `int_payload_verify.py` checks them
independently in Python's standard library and decodes the `WIV1` keyframe-0
payloads in full; `byte-exact-conformance` parses and decodes the `WIV1` and `WIH1`
payloads on every leg of its matrix. CI runs the new tests in both build profiles,
the generator in the corpus loop, the verifier, a `cargo check` of each integer
feature alone, and a `wasm32` check.
### The registry as one table, candidate capabilities, the codec glossary under test, and the public text retrofit
- **Candidate capabilities** (SPEC §5): a registered string whose payload format no

5
wai/README.md
View file

@ -111,6 +111,11 @@ hash: a deployer can require `intent=replicate` to dispatch only to these.
| `wai.audio.int_codec` | audio — learned 1-D transform, integer synthesis → int16 PCM | byte-exact (hash-portable) |
| `wai.splat.int_codec` | 3D — learned Gaussian-splat-attribute transform, integer synthesis | byte-exact (hash-portable) |
Their payloads are binary formats SPEC §5 pins — `WIV1` for the integer video
codecs, `WIH1`, `WIA1` and `WIS1`
([`extensions/integer-payloads.md`](extensions/integer-payloads.md)). The JSON the
encode and export tools write is converted to them, never carried.
## Interactive & temporal content (extensions)
Beyond the codec menu, WAI carries content that consumes input or moves

7
wai/REGISTRY.md
View file

@ -49,7 +49,12 @@ Two further tables hold strings that look like capabilities and are not:
## Status
- **registered** — the payload format (for a record or derivation, the
definition) is pinned.
definition) is pinned. The WAI-native integer codecs `wai.video.int_motion`,
`wai.video.int_hyper`, `wai.neural.int_hyper`, `wai.audio.int_codec` and
`wai.splat.int_codec` were registered with a decoder and no byte layout (the first,
the fourth and the fifth also with the tool that wrote their bitstreams); v0.4.0
completes their registrations by pinning their formats (SPEC §8, *Completing a
registration*; SPEC §5, *Payloads of the WAI-native integer codecs*).
- **candidate** — the string is registered but no revision has pinned its payload
format. An encoder MUST NOT emit a candidate, and a sink MUST treat it as
unsupported (SPEC §5, *Candidate capabilities*). The reference implementation

120
wai/SPEC.md
View file

@ -249,6 +249,14 @@ order, and the first that fails is the reason reported:
is offline or has no clock it trusts, a grant outside its window or for another
capability — or the sink holds no grant for the set. The sink reports §7.9's
check and reason with it. A sink that holds no key log skips this check.
Check 5 covers every identity of the bytes the decode reads, not only the
pin's: once checks 6 to 8 have found the set, the sink runs this check again
over each digest a pin over those bytes could name — the listing of the files
the decode reads, and, for a capability registered with a one-file set, that
file's SHA-256 — so a set refused under one identity is not reached through a
pin that names it by another (a set-form pin over a listing of one revoked file,
or a listing that adds a file the decode does not read). Authorisation needs
one of them to be held; the pin's own digest, which passed above, is one.
6. `prior_not_held`: the sink holds no parameter set for the capability whose
digest, in the pin's form, equals `sha256`; a file the sink cannot read is not
held. A sink that holds the set a well-formed `supersedes` names — a file when
@ -414,7 +422,12 @@ pin does not resolve. An envelope that pins no set is decoded against the set th
sink holds for `cap`, and a sink that holds a key log runs check 5 over that set
too, identified by each digest a pin over it could name: it does not decode `cap`
for that envelope when any of them is revoked, or, for a capability it holds to
§7.9, when §7.9 holds the set under none of them. Either way
§7.9, when §7.9 holds the set under none of them. It then decodes from the bytes
check 5 ran over alone, as a pinned decode reads only what its pin verified (§3.1):
read once, so a file changed after the check is not what the decode reads, and, for
a capability it holds to §7.9, only the part §7.9 holds (the set, the files a held
listing names, or one held file); a file the decode needs outside them is
`set_incomplete`. Either way
the algorithm above runs unchanged, so a sink MUST NOT report a file inert because
of module authorisation while a fallback it can decode without that module remains,
and an encoder emitting a capability whose decode needs a module MUST declare such a
@ -422,11 +435,16 @@ fallback (§3). A sink configured with no key log holds no module under §7.9, s
runs no module obtained apart from an envelope, and it does not run check 5. The
reference dispatcher in `wai-rs` runs no module obtained apart from an envelope; it
runs check 5 when the sink is configured with a key log (`key_log::SinkKeyLog`,
feature `key_discovery`, given to `pin::verify_with` or to
`neural::ModelRegistry::with_key_log`, which also runs it before an unpinned
decode): the revocation part for every capability a pin applies to, and §7.9's
authorisation for the capabilities the sink holds to it. `wai-web` and the C ABI
take no key log.
feature `key_discovery`, given to `pin::verify_with`, to `int_payload::resolve`
(which runs it for a pinned decode and over the files an unpinned integer decode
reads) or to `neural::ModelRegistry::with_key_log`, which also runs it before an
unpinned decode, over the bytes that decode then reads, read once: an integer
decode's members, and otherwise the registered file of a one-file set, or every
file below the directory the decoder reads from, identified by the listing
`pin::listing_of_dir` derives, each file's digest and each listing beside them that
matches): the revocation part for
every capability a pin applies to, and §7.9's authorisation for the capabilities the
sink holds to it. `wai-web` and the C ABI take no key log.
---
@ -437,9 +455,12 @@ codec or a WAI-native class. The mandatory floor and the recommended
set are royalty-free; every payload and companion capability carries a
licence class (see the end of this section), and one that is not
royalty-free is never a default. The "payload format" column is the
on-the-wire payload bytes; in every case it is the canonical bytes
that codec normally emits (a JXL file, an Opus stream, etc.) — WAI
does not wrap or re-frame them.
on-the-wire payload bytes. For a standard codec it is the canonical
bytes that codec normally emits (a JXL file, an Opus stream, etc.) —
WAI does not wrap or re-frame them. For a WAI-native class it is the
format the class's registration or extension defines; for the
WAI-native integer codecs, the format *Payloads of the WAI-native
integer codecs* (below) pins.
The registry is also published in machine-readable form, generated from
the reference implementation: [`registry/capabilities.json`](registry/capabilities.json)
@ -502,8 +523,10 @@ Every conforming sink SHOULD implement:
Each names a model the sink MAY have installed. Payload is a compact
conditioning representation specific to the capability (latent tokens,
prompt embeddings, etc.). The standard does NOT specify the byte
layout of these payloads — the capability owner does.
prompt embeddings, etc.). The standard does not specify the byte
layout of these payloads — the capability owner does — except where this
section pins one: the integer codecs' payloads are pinned below
(*Payloads of the WAI-native integer codecs*).
Each row names the model and the operating point its own publication
reports (bitrate, sample rate) or the property WAI's integer construction
@ -515,11 +538,15 @@ sink-supplied.
A capability that decodes against a parameter set (§3.1) is registered with that
set's shape. **One file:** `wai.neural.int_synth` (its model specification,
`int_synth.json`); `wai.neural.int_mlicpp` (its `WMB1` bundle, read from a set as
`model.wmb`); and `wai.neural.encodec32`, `wai.neural.dac`, `wai.neural.mimi`,
`model.wmb`); `wai.audio.int_codec` (its `WAM1` synthesis, read from a set as
`model.wam`); `wai.splat.int_codec` (its `WHM1` synthesis, read from a set as
`g_s.whm`); and `wai.neural.encodec32`, `wai.neural.dac`, `wai.neural.mimi`,
`wai.neural.wavtokenizer`, `wai.neural.snac` and `wai.neural.video_bmshj2018` (a
self-contained ONNX model; one that keeps tensors in external files is several
files). **Several files:** every other capability that decodes against a parameter
set. A later revision MAY register a capability of several files as one file when
set; `wai.neural.int_hyper` and `wai.video.int_hyper` read `g_s.whm`, `h_s.whm` and
`entropy.json` from theirs. `wai.audio.int_codec` and `wai.splat.int_codec` were
registered as several files until this revision named their one-file formats. A later revision MAY register a capability of several files as one file when
it defines a one-file format that holds the whole set; that widens the
registration and makes no conforming envelope non-conforming (§8).
@ -540,7 +567,7 @@ registration and makes no conforming envelope non-conforming (§8).
| `wai.neural.bmshj2018` | bmshj2018-factorized + rANS (byte-exact CompressAI port) | about 0.26 bpp at 29 dB PSNR (quality 3) | CompressAI |
| `wai.neural.mbt2018_mean` | mean-scale hyperprior (Gaussian-conditional entropy) | the conditional-entropy family (arXiv 1809.02736) | `neural/hyperprior.rs` |
| `wai.neural.int_synth` | integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) | bit-identical on every machine — soundly backs `intent = replicate` | `neural/int_synth.rs` (QAT weights sink-supplied) |
| `wai.neural.int_hyper` | integer fixed-point **conditional** codec — whole bitstream→image path integer (integer hyper-decoder σ → integer-σ entropy → integer synthesis) | the integer-entropy contract fully assembled: a PTQ of mbt2018_mean, bit-identical on every machine, whose integer decode CI holds within 1 dB of the same model's float decode on a pinned photograph | `int_hyper_synth.rs` (tables sink-supplied) |
| `wai.neural.int_hyper` | integer fixed-point **conditional** codec — whole bitstream→image path integer (integer hyper-decoder σ → integer-σ entropy → integer synthesis) | the integer-entropy contract fully assembled: a PTQ of mbt2018_mean, bit-identical on every machine, whose integer decode CI holds within 1 dB of the same model's float decode on a pinned photograph | payload `WIH1`; `int_hyper_synth.rs` (tables sink-supplied) |
| `wai.neural.int_mlicpp` | integer fixed-point **MLIC++ entropy topology** — `K` channel slices of the latent, each decoded as a checkerboard anchor pass then a non-anchor pass, with an entropy network over concat[latent decoded so far, hyper] re-run before every pass to give σ‖μ; hyper-decoder, entropy network and synthesis all integer | the channel-autoregressive × checkerboard entropy model at **decode-equivalence**: the encoder and every decoder compute the same σ, so the same distribution for every element, at any thread count; bit-identical on every machine. The committed conformance vector has synthetic weights, so it demonstrates decode-equivalence and states no quality figure | `int_mlicpp.rs` (one-file model bundle sink-supplied; `tools/wai_int_mlicpp_export.py`) |
| `wai.image.jpegai` | JPEG AI (ISO/IEC 6048-1:2025) codestream → RGB — integer entropy front end (mANS, with an integer scale-index derivation) + 16-bit-PTQ integer hyper-decoder, context-model and synthesis networks + dequantisation, latent refinement, post-filters (the eICCI CNN among them) and colour conversion in IEEE-754 float with a fixed evaluation order. Scope: one configuration — 256×256 4:4:4, `model_id` 0 (tools_0), `beta_displacement_log` 0, the high-operation-point synthesis, the eICCI model pair the weight bundle carries, and the other header conditions in [determinism-tiers](extensions/determinism-tiers.md) §5; both decode paths refuse any other codestream. Within it, a sink without the integer weight bundle, or in a non-default floating-point environment, takes a float ONNX decode instead. Only checkpoint constants are sink-supplied | **entropy-consistency** (registry class `NeuralEntropyExact`): the entropy decode, including the scale-index derivation, is integer on both decode paths, so the decoded symbols reproduce on every machine. The exact decoder also pins one reconstruction to RGB, reproducible across machines under the build and floating-point conditions in [determinism-tiers](extensions/determinism-tiers.md) §5–§6 (it checks the floating-point environment before decoding) — a property of that decoder, not of the capability, so `wai.image.jpegai` does not back `replicate` (§7). On one reference codestream the exact decoder stays within max\|Δ\| ≤ 2 of the reference decoder (recorded max\|Δ\| = 1 on ~4.4% of samples); evidence in [determinism-tiers](extensions/determinism-tiers.md) §6. **Downgraded in v0.4.0:** this row previously claimed decode-equivalence, and the registry classed the capability `StandardDefined` | `jpegai_decode.rs` (+ `jpegai_{bitstream,mans,pipeline,synthesis,icci}.rs`) |
| `wai.generate.spectral` | deterministic integer GENERATIVE synthesis (`intent=create`) — prompt+seed → integer-LUT spectral composition, no float/ONNX | byte-identical on every machine, pinned to (model, prompt, seed, steps); the create-side of "conformance is a hash" (generation-equivalence) | `generate.rs` (small pinned model bundle) |
@ -559,8 +586,8 @@ registration and makes no conforming envelope non-conforming (§8).
| `wai.neural.video_bmshj2018` | per-frame bmshj2018 | per-frame image coding; decodable in a browser on WebGPU | CompressAI |
| `wai.neural.dcvc_rt` | DCVC-RT (2025), inter-frame with reference-feature propagation | inter-frame neural video; its paper reports real-time 1080p and 4K decode on a GPU | research code, `neural/video_dcvc.rs` |
| `wai.neural.dcvc_fm` | DCVC-FM (feature modulation), wide single-model quality range (RGB + YUV) | a wide single-model quality range; shares the inter-frame route | research code, `neural/video_dcvc.rs` |
| `wai.video.int_motion` | DETERMINISTIC INTEGER inter-frame — I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS, no float/ONNX; motion is whole-pel or **half-pel** (`mv_precision`), the latter predicting at the half-grid by integer bilinear interpolation (`(a+b+1)>>1`, `(a+b+c+d+2)>>2`) so real footage — which pans a non-integer number of pixels per frame — costs fewer residual bits while staying byte-exact | byte-identical frame-by-frame on every machine (*frame-state-equivalence*, a per-frame portable hash); the temporal extension of the integer contract, where a float inter-frame decode desyncs down the GOP | `int_video.rs` (whole-pel `tools/wai_int_video_encode.py`; half-pel from real footage `tools/wai_int_video_real_encode.py`) |
| `wai.video.int_hyper` | NEURAL-KEYFRAME inter-frame — the I-frame is the LEARNED integer codec `int_hyper` decoded byte-exact, the P-frames are integer block-MC residuals on top | a learned keyframe you can put in a receipt + integer inter-frames; whole decode integer, byte-identical everywhere | `int_video::decode_pframes` ∘ `int_hyper_synth::decode_conditional` |
| `wai.video.int_motion` | DETERMINISTIC INTEGER inter-frame — I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS, no float/ONNX; motion is whole-pel or **half-pel** (`mv_precision`), the latter predicting at the half-grid by integer bilinear interpolation (`(a+b+1)>>1`, `(a+b+c+d+2)>>2`) so real footage — which pans a non-integer number of pixels per frame — costs fewer residual bits while staying byte-exact | byte-identical frame-by-frame on every machine (*frame-state-equivalence*, a per-frame portable hash); the temporal extension of the integer contract, where a float inter-frame decode desyncs down the GOP | payload `WIV1`, keyframe 0; `int_video.rs` (the encode tools write a JSON form a converter reads) |
| `wai.video.int_hyper` | NEURAL-KEYFRAME inter-frame — the I-frame is the LEARNED integer codec `int_hyper` decoded byte-exact, the P-frames are integer block-MC residuals on top | a learned keyframe you can put in a receipt + integer inter-frames; whole decode integer, byte-identical everywhere | payload `WIV1`, keyframe 1 (a nested `WIH1`); `int_video::decode_pframes_typed` ∘ `int_hyper_synth::decode_conditional_checked` |
| `wai.video.manifest` | ATTESTED PER-SESSION MANIFEST — the OTT delivery layer, not a codec: a personalised HLS manifest (ads stitched at `EXT-X-DISCONTINUITY`, blackouts + geo enforced) rendered as a **pure function of the session *class*** `(content, entitlement, geo, blackout set, ad pod, window)` rather than the viewer, and sealed in an Ed25519 receipt that binds the JCP grant (capability, joule ceiling, `funds_ceiling` = ad budget), the composition, the stitched ads and the policy applied | **manifest-equivalence**: `manifest_hash == hash(render(segments))` and `session_class_hash` are recomputable by anyone holding the bytes, so the ads, blackouts and compose-work stop being dashboard claims; class-determinism means one generation serves every viewer of the class, and `uj_per_viewer` is the amortised energy that a 1:1-unique generator cannot report below its per-viewer cost. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_manifest.rs` |
| `wai.video.ssai` | ATTESTED SERVER-SIDE AD INSERTION — a **deterministic** ad decision (greedy by price within the avail duration and the funds remaining) plus a per-impression Ed25519 receipt binding the candidate pod, the manifest it belongs to, the served `creative_hash`es and the VAST quartile each reached | **spend-equivalence**: `budget_spent_micros` is recomputed from the *billable* impressions and rejected if it exceeds the grant's `funds_ceiling` — an overspent receipt fails verification; every creative and quartile is checkable, so a buyer and seller reconcile impressions from the same bytes instead of two logs that only a dashboard joins. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_ssai.rs` |
| `wai.video.keys` | ATTESTED CONTENT-KEY RELEASE — the DRM licensing decision as a record: every key release, and every refusal, is an Ed25519 receipt binding the KID, the DRM system, the entitlement demanded and the window enforced, with a BLAKE3 `key_commit` proving *which* key was released without carrying it | **decision-equivalence**: `verify()` re-runs the release policy over the receipt's own `(grant capability, entitlement, issued, expiry)` and rejects any misreported outcome, and enforces granted-iff-committed — so "keys went only to entitled, unexpired sessions" is checkable against the signed log, not asserted. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_keys.rs` |
@ -657,12 +684,12 @@ standard. See the linked extension for each.
| `wai.splat.spz` | SPZ (an open-source Gaussian-splat format) | asset, content-addressed; presentation only |
| `wai.audio.scene` | WSCN scene (objects + trajectories) | mixdown-equivalence · [spatial-audio](extensions/spatial-audio.md) |
| `wai.audio.score` | WSCR container (note events) | music as instructions; sink synthesizes, mixdown-equivalence · [score](extensions/score.md) |
| `wai.audio.int_codec` | LEARNED 1-D transform codec — integer transposed-conv synthesis + integer IGDN over a factorized-rANS latent (model + bitstream from `tools/wai_int_audio_export.py`) | **sample-equivalence**: int16 PCM byte-identical on every machine (a portable PCM hash); the audio analog of `wai.neural.int_hyper`, where a float learned-audio decode desyncs on ~1e-5 round-off. Pure i64, no float/ONNX (`int_audio.rs`) |
| `wai.audio.int_codec` | `WIA1` — a LEARNED 1-D transform codec: integer transposed-conv synthesis + integer IGDN over a factorized-rANS latent (`tools/wai_int_audio_export.py` writes the model and a JSON form a converter reads) | **sample-equivalence**: int16 PCM byte-identical on every machine (a portable PCM hash); the audio analog of `wai.neural.int_hyper`, where a float learned-audio decode desyncs on ~1e-5 round-off. Pure i64, no float/ONNX (`int_audio.rs`) |
| `wai.haptic.signal` | WHAP envelope (intensity/sharpness keyframes) | sample-equivalence · [haptics](extensions/haptics.md) |
| `wai.feed.state` | WWLD, sim `wai.world.sim.feed0` | replay-equivalence · [state-feeds](extensions/state-feeds.md) |
| `wai.avatar.pose` | WWLD, sim `wai.world.sim.skeletal0` | pose-equivalence · [avatars](extensions/avatars.md) |
| `wai.splat.4d` | W4DS container (per-Gaussian keyframed tracks) | frame-state-equivalence · [volumetric](extensions/volumetric.md) |
| `wai.splat.int_codec` | LEARNED Gaussian-splat ATTRIBUTE codec — a learned transform over the `[14,S,S]` attribute image (pos/scale/rot/color/opacity), integer synthesis (int_hyper machinery, 14 channels) over a factorized-rANS latent (`int_splat.rs`; model + bitstream from `tools/wai_int_splat_export.py`) | **splat-equivalence**: the decoded attribute tensor is byte-identical on every machine (a portable cloud hash) — the int_hyper of 3D, where a float splat-attribute decode desyncs. The float rasterizer stays sink-supplied presentation; the *attributes* are the contract. Pure i64, no float/ONNX |
| `wai.splat.int_codec` | `WIS1` — a LEARNED Gaussian-splat ATTRIBUTE codec: a learned transform over the `[14,S,S]` attribute image (pos/scale/rot/color/opacity), integer synthesis (int_hyper machinery, 14 channels) over a factorized-rANS latent (`int_splat.rs`; `tools/wai_int_splat_export.py` writes the model and a JSON form a converter reads) | **splat-equivalence**: the decoded attribute tensor is byte-identical on every machine (a portable cloud hash) — the int_hyper of 3D, where a float splat-attribute decode desyncs. The float rasterizer stays sink-supplied presentation; the *attributes* are the contract. Pure i64, no float/ONNX |
| `wai.film.linear` | WFLM container (sealed world + camera track + frame schedule) | film-equivalence · [linear-film](extensions/linear-film.md) |
| `wai.film.reel` | WREL container (sealed world + a sequence of shots = camera track + schedule each) | reel-equivalence (film-equivalence with cuts) · [linear-film](extensions/linear-film.md) |
| `wai.ternary.tt` | WQTT container (balanced-ternary MPS cores — packed trits + integer per-column scales) | a **quantum-inspired** (tensor-network) transform ported from the ternary/MPS fabric with ONE change: the per-column scale is an **integer**, so the whole decode is `i64`/`i128` and the reconstruction is byte-identical on every machine — **reconstruction-equivalence** (a portable recon hash), receiptable. Scope: a **provenance** transform, measured at 13.3 dB SNR for 3.8× with six residual layers (`Lossy` tier; the extension's §5). Encoder-side float SVD only · [ternary-tt](extensions/ternary-tt.md) |
@ -681,6 +708,38 @@ grant's joule ceiling (gate ⊗ meter ⊗ receipt in one object). WAI owns the
receipt; the sink's control system runs the hardware:
[`extensions/quantum-ops.md`](extensions/quantum-ops.md).
### Payloads of the WAI-native integer codecs
This revision pins a payload format to each WAI-native integer codec whose row named
its decoder and no byte layout. The rows of `wai.video.int_motion`,
`wai.audio.int_codec` and `wai.splat.int_codec` also named the tool that wrote a
bitstream, a JSON document, which this section's opening called the bytes the codec
normally emits; those of `wai.neural.int_hyper` and `wai.video.int_hyper` named their
decoders only. The pin completes these registrations as §8 permits (*Completing a
registration*). [integer-payloads](extensions/integer-payloads.md) §3–§6 defines
each layout and its decode, §7 the checks a sink makes, and §8 the closed list of
codes it refuses with. The rows above keep their headings.
| capability | payload format | parameter set |
|---|---|---|
| `wai.video.int_motion` | `WIV1`, keyframe 0 | none |
| `wai.video.int_hyper` | `WIV1`, keyframe 1 (a nested `WIH1`) | several files: `g_s.whm`, `h_s.whm`, `entropy.json` |
| `wai.neural.int_hyper` | `WIH1` | several files: `g_s.whm`, `h_s.whm`, `entropy.json` |
| `wai.audio.int_codec` | `WIA1` | one file: `model.wam` |
| `wai.splat.int_codec` | `WIS1` | one file: `g_s.whm` |
From this revision on, the format of each is fixed as §8 states. The JSON the encode
and export tools write is a converter input
([integer-payloads](extensions/integer-payloads.md) §10), never a payload: an encoder
MUST NOT carry it under these strings, and a sink refuses it (`bad_magic`).
*Envelopes the pin makes non-conforming* (they still parse): a `WAI1` whose
`capability` or `fallback` is one of these strings and whose payload is not that
string's format, a tool's JSON form among them; and a `WAI2` with a component of one
of these strings whose bytes are not its format. A sink refuses such a payload when it
decodes it, and does not try the fallback
([integer-payloads](extensions/integer-payloads.md) §11).
### Immersive audio, cinema, volumetric, haptics, HDR (registered, sink-supplied)
The same "name the capability, the sink supplies the decoder" pattern extends
@ -999,7 +1058,9 @@ A confidentiality-aware sink MUST, before running §4:
2. Else, if the sink does not implement the named scheme, or holds no
key for `security.kid`, the file is **inert at this sink** — return a
clear "no key / unsupported encryption" error. It MUST NOT feed
ciphertext to a codec.
ciphertext to a codec. (The reference dispatchers in `wai-rs` implement
no scheme and report `unsupported_encryption`, for a `WAI2` envelope
against every entry.)
3. Else recover the plaintext payload via the named scheme (for
`mls-sframe`: SFrame `open` under the member's base key), then run §4
on the recovered plaintext.
@ -1529,7 +1590,10 @@ by which capability happens to be installed.
}
```
Each entry has its own `capability` (the dispatch key) and `kind`
(the codec id matching the bytes in the corresponding payload). An entry MAY
(the codec id matching the bytes in the corresponding payload). The manifest
MAY carry `security` (§6.1), which covers every payload: a sink applies §6.4
before step 2 of the selection below, and an encoder writes only a security
object there. An entry MAY
carry `prior`, the pin (§3.1) of the parameter set its payload was coded
against. Two entries MAY name one capability with different pins: the same
content coded against two parameter sets, such as a set and its successor:
@ -1592,7 +1656,8 @@ A v1.1-aware sink MUST:
which step 2 never picks), `missing_capability`, `component_digest_mismatch`,
a pin reason (§3.1), or, where a deployer policy restricts step 2 to the entries §7's
`replicate` rule admits, the `replicate_*` reason the rule gives for an entry
it leaves out.
it leaves out; an envelope inert under §6.4 step 2 names that reason for every
entry.
A sink MUST NOT decode one entry's payload against the parameter set another
entry pins. The envelope's `intent` applies to the rendition the sink picks, and
@ -1765,7 +1830,18 @@ envelope that uses any string a later revision registers.
whole envelope where an earlier reader ignored the member.
- New capabilities (§5) MAY be added in any MINOR revision. Existing
capabilities MUST NOT be redefined — once a capability string is
registered for a codec, its payload format is fixed forever.
registered for a codec, its payload format is fixed forever; for a registration
completed as the next item says, from the revision that completes it.
- **Completing a registration.** A WAI-native string whose §5 row named the decoder
that reads its payload and fixed no byte layout (naming, at most, a tool that wrote
it) has a registration with no payload format to hold. A MINOR revision MAY
complete such a registration by pinning a byte layout, which is not a
redefinition. Envelopes that carried another form under the string, a tool's
output among them, become non-conforming; they still parse, and the completing
revision lists them in §5. From that revision on, the string's payload format is
fixed as the item above states. §5, *Payloads of the WAI-native integer codecs*,
completes the registrations of `wai.video.int_motion`, `wai.video.int_hyper`,
`wai.neural.int_hyper`, `wai.audio.int_codec` and `wai.splat.int_codec` this way.
- **Editions.** A registration that names an edition (or version) of its
standard carries that edition. A registration that names none carries the
edition of its named standard current on the date it was registered, and no

118
wai/byte-exact-conformance/src/int_payload_golden.rs Normal file
View file

@ -0,0 +1,118 @@
//! Cross-architecture byte-equality for the binary payloads SPEC §5 pins
//! (`extensions/integer-payloads.md`): `WIV1` (`wai.video.int_motion` with keyframe 0,
//! `wai.video.int_hyper` with keyframe 1) and `WIH1` (`wai.neural.int_hyper`).
//!
//! The goldens next door decode the JSON forms the encode tools write. These parse the
//! committed binary payloads with the wire parser a sink runs (`int_payload/wire.rs`,
//! `#[path]`-included and self-contained) and decode them through the checked typed
//! decoders, so every leg of the matrix checks the normative parse as well as the
//! decode. Each must reproduce the same SHA-256 digests the JSON forms do, from the
//! vectors' `sha256.json` sidecars, and the payloads themselves are pinned by their
//! SHA-256.
#![cfg(test)]
use std::path::PathBuf;
use serde_json::Value;
use sha2::{Digest, Sha256};
use crate::int_hyper_synth::{decode_conditional_checked, CondEntropy, CondStream, IntModel};
use crate::int_payload_wire::{parse_wih1, parse_wiv1, Wih1, Wiv1};
use crate::int_video::{decode_pframes_typed, decode_typed, VideoStream};
const INT_VIDEO_WIV1_SHA256: &str = "3bbf0a864d9528e73aff036d1a636893f839d3f6c5f31dde7824a59e6c116565";
const VIDEO_REAL_WIV1_SHA256: &str = "efce5ada256cb3443a673ee0c10aa4e0a8ce4a372102cd918627e5eec746880f";
const VIDEO_NK_WIV1_SHA256: &str = "d2c62da643328bfbdf9eab129283322e9077a491c5ddd269464577f51a958bc8";
const TINY_WIH1_SHA256: &str = "6d8a195a5a138452292b55d70a575d1fbb4372fd8614b3b821b676e5090c3c17";
fn vectors() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../wai-rs/tests/vectors")
}
fn read(rel: &str) -> Vec<u8> {
let p = vectors().join(rel);
std::fs::read(&p).unwrap_or_else(|e| panic!("read {}: {e}", p.display()))
}
fn jload(rel: &str) -> Value {
serde_json::from_slice(&read(rel)).unwrap_or_else(|e| panic!("parse {rel}: {e}"))
}
fn hex(b: &[u8]) -> String {
b.iter().map(|x| format!("{x:02x}")).collect()
}
fn stream<'a>(v: &'a Wiv1<'a>) -> VideoStream<'a> {
VideoStream {
n_frames: v.n_frames as usize, h: v.h as usize, w: v.w as usize, block: v.block as usize,
q: v.q as i64, offset: v.offset, mv_precision: v.mv_precision, cdf: &v.cdf, mvs: &v.mvs, rans: v.rans,
}
}
/// Every frame's SHA-256 and the clip's against the sidecar entry `key`.
fn assert_clip(frames: &[Vec<u8>], key: &str) -> String {
let side = jload("int_video/sha256.json");
let want = side[key]["frames_sha256"].as_array().expect("frames_sha256");
assert_eq!(want.len(), frames.len(), "{key}: frame count");
let mut all = Vec::new();
for (i, f) in frames.iter().enumerate() {
let d = Sha256::digest(f);
assert_eq!(hex(&d), want[i].as_str().unwrap(), "{key}: frame {i} diverged");
all.extend_from_slice(&d);
}
let clip = hex(&Sha256::digest(&all));
assert_eq!(clip, side[key]["clip_sha256"].as_str().unwrap(), "{key}: clip");
clip
}
/// The tiny conditional codec's keyframe, from its committed set (the compact `WHM1`
/// networks and the entropy tables).
fn keyframe(k: &Wih1<'_>) -> (Vec<u8>, usize, usize) {
let g_s = IntModel::from_bytes(&read("int_hyper/tiny_g_s.bin")).expect("g_s");
let h_s = IntModel::from_bytes(&read("int_hyper/tiny_h_s.bin")).expect("h_s");
let ent = CondEntropy::try_from_value(&jload("int_hyper/tiny_entropy.json")).expect("entropy tables");
let s = CondStream {
m: k.m as usize, sh: k.sh as usize, sw: k.sw as usize, n_z: k.n_z as usize, zh: k.zh as usize, zw: k.zw as usize,
z_rans: k.z_rans, rans: k.rans,
};
decode_conditional_checked(&g_s, &h_s, &ent, &s).expect("the keyframe decodes")
}
/// `WIV1`, keyframe 0: the synthetic whole-pel clip and the half-pel real footage.
#[test]
fn wiv1_keyframe_0_payloads_are_byte_identical() {
for (bin, key, payload_sha, clip_sha) in [
("int_video/int_video_bitstream.wiv1", "int_video", INT_VIDEO_WIV1_SHA256,
"5afd6477d09525b7db4485027588ba3bc43de3d29c996eb57f2d3274bf8309f9"),
("int_video/video_real_bitstream.wiv1", "video_real", VIDEO_REAL_WIV1_SHA256,
"e6724ca214694e012c3e2b49df4e116a0a7e150435a12d7b144560ce2d5c7bdf"),
] {
let p = read(bin);
assert_eq!(hex(&Sha256::digest(&p)), payload_sha, "{key}: the committed payload");
let v = parse_wiv1(&p, Some(0)).expect("the payload parses");
let clip = decode_typed(&stream(&v)).expect("decode must succeed on every architecture");
assert_eq!(assert_clip(&clip.frames, key), clip_sha);
}
}
/// `WIV1`, keyframe 1: the nested `WIH1` keyframe, then integer P-frames on top.
#[test]
fn wiv1_keyframe_1_payload_is_byte_identical() {
let p = read("int_video/video_nk_bitstream.wiv1");
assert_eq!(hex(&Sha256::digest(&p)), VIDEO_NK_WIV1_SHA256);
let v = parse_wiv1(&p, Some(1)).expect("the payload parses");
let k = parse_wih1(v.wih1.expect("keyframe 1 nests a WIH1")).expect("the keyframe parses");
let (frame0, h, w) = keyframe(&k);
assert_eq!((h, w), (v.h as usize, v.w as usize), "the keyframe is the clip's size");
let clip = decode_pframes_typed(&frame0, &stream(&v)).expect("integer P-frames");
assert_eq!(assert_clip(&clip.frames, "video_nk"), "4a81f0c94ccd938a7ca80df203303d5735d9dfcdfa150da155d9f2e4ca283e8f");
}
/// `WIH1` alone: the learned keyframe's image.
#[test]
fn wih1_payload_is_byte_identical() {
let p = read("int_hyper/tiny_bitstream.wih1");
assert_eq!(hex(&Sha256::digest(&p)), TINY_WIH1_SHA256);
let (rgb, h, w) = keyframe(&parse_wih1(&p).expect("the payload parses"));
assert_eq!((h, w), (16, 16));
assert_eq!(hex(&Sha256::digest(&rgb)), "dce7282d46f88f03d5dc0fd42f325756d310fe1fc982b6e1792a1fbfbeb482cf");
assert_eq!(hex(&Sha256::digest(&rgb)), jload("int_hyper/sha256.json")["tiny"]["sha256"].as_str().unwrap());
}

7
wai/byte-exact-conformance/src/lib.rs
View file

@ -69,7 +69,14 @@ pub mod int_hyper_synth;
#[path = "../../wai-rs/src/int_mlicpp.rs"]
pub mod int_mlicpp;
// The wire parser of the integer payloads SPEC §5 pins (WIV1, WIH1, WIA1, WIS1).
// Self-contained — std's Vec only, nothing from the crate — so the matrix parses the
// committed binary payloads with the parser a sink runs.
#[path = "../../wai-rs/src/int_payload/wire.rs"]
pub mod int_payload_wire;
mod golden;
mod int_payload_golden;
mod int_video_golden;
mod neural_keyframe_golden;
mod jpegai_golden;

559
wai/extensions/integer-payloads.md Normal file
View file

@ -0,0 +1,559 @@
# WAI Extension: Integer Payloads
> Status: Draft. Pins the payload formats of the WAI-native integer codecs that
> [SPEC.md §5](../SPEC.md) registers without one (*Payloads of the WAI-native integer
> codecs*): `WIV1` for `wai.video.int_motion` and `wai.video.int_hyper`, `WIH1` for
> `wai.neural.int_hyper`, `WIA1` for `wai.audio.int_codec` and `WIS1` for
> `wai.splat.int_codec`. Defines each layout, its decode, the checks a sink makes and
> the closed list of refusal codes; names each capability's parameter-set members;
> and makes the JSON the encode and export tools write a converter input, never a
> payload. Reference impl: `wai-rs` — `int_payload::{wire, decode_int, resolve}`,
> `int_payload::{wiv1_from_json, wih1_from_json, wia1_from_json, wis1_from_json}`,
> `codecs::PINNED_INT_PAYLOADS`, `container::check_int_payload_framing`.
> Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174.
## 1. Scope
| capability | payload | parameter set (§9) |
|---|---|---|
| `wai.video.int_motion` | `WIV1` with keyframe 0 (§3) | none |
| `wai.video.int_hyper` | `WIV1` with keyframe 1, a nested `WIH1` (§3) | several files: `entropy.json`, `g_s.whm`, `h_s.whm` |
| `wai.neural.int_hyper` | `WIH1` (§4) | several files: `entropy.json`, `g_s.whm`, `h_s.whm` |
| `wai.audio.int_codec` | `WIA1` (§5) | one file: `model.wam` |
| `wai.splat.int_codec` | `WIS1` (§6) | one file: `g_s.whm` |
**Why this pin is permitted.** Until this revision, these capabilities' SPEC §5 rows
named the decoder that read their payload, and no byte layout. The rows of
`wai.video.int_motion`, `wai.audio.int_codec` and `wai.splat.int_codec` also named the
encode or export tool that wrote a bitstream, a JSON document; those of
`wai.neural.int_hyper` and `wai.video.int_hyper` named their decoders only. SPEC §5
called every row's payload the bytes its codec normally emits, which for these rows
was the tools' JSON. So this revision changes what these strings carry. SPEC §8
permits it under *Completing a registration*: a revision may complete the
registration of a WAI-native string whose row named a decoder and fixed no byte layout
by pinning one, and it lists the envelopes that become non-conforming (§11; SPEC §5).
SPEC §8's rule that a registered string's payload format is fixed forever applies to
these strings from this revision on. The encode and export tools still write JSON
that carries the same fields; it is a converter input (§10).
`wai.neural.int_synth` and `wai.neural.int_mlicpp` keep the payload formats SPEC §5
gives them. A sink decodes them and the five strings above (`wai.video.int_motion`,
`wai.video.int_hyper`, `wai.neural.int_hyper`, `wai.audio.int_codec`,
`wai.splat.int_codec`) through one path (`int_payload::decode_int`), and §8 names the
codes it reports for those two.
## 2. Common rules
**Encoding.** Every multi-byte field is little-endian. Offsets are in bytes.
**Framing.** A payload starts with its four-byte magic and a version byte. This
revision defines version 1 for every format. Every reserved field is zero. Nothing
follows the last section.
**rANS streams.** A stream is a `u32` length `L` followed by `L` bytes. `L` is a
multiple of 4 and at least 8. It is decoded as follows (precision 16, with a bypass
escape of 4-bit groups):
- The state `x` starts as `(w1 << 32) | w0`, from the stream's first two `u32`
words. *Renormalising*: if `x < 2^31`, then `x = (x << 32) | w`, where `w` is the
next unread word; if no word is left, the stream has **run out** and `x` is left
as it is.
- `bits(n)`: `v = x mod 2^n`, `x = x >> n`, renormalise, return `v`.
- Decoding a symbol against a table `cdf` of length `n` with offset `o`: let
`cum = x mod 2^16` and `s` be the largest index with `cdf[s] ≤ cum`
(`0 ≤ s ≤ n − 2`); then `x = (cdf[s+1] − cdf[s]) · (x >> 16) + cum − cdf[s]`, and
renormalise. If `s < n − 2`, the value is `s`. If `s = n − 2` (the escape): let
`v = bits(4)` and `count = v`; then, at most 8 times, while `v = 15` and the stream
has not run out, `v = bits(4)` and `count = count + v`; then `count = min(count, 8)`
and `raw = Σ_{j < count} bits(4) << 4j`. The value is `−(raw >> 1) − 1` if `raw`
is odd and `(raw >> 1) + (n − 2)` if it is even, saturated to `i32`. The symbol
is `value + o`, saturated to `i32`.
- A decode stops at the first symbol during whose decode its stream runs out, and
refuses the payload (`stream_exhausted`): the value that symbol would give is not
used, and nothing after it is decoded. A stream that never runs out decodes every
symbol, so a conforming payload is unaffected.
- After the last symbol a decode reads from a stream, the stream ends **canonically**:
every word of it has been read, and the state is `2^31`. Otherwise the payload is
refused (`stream_noncanonical`). An encoder starts its state at `2^31` and writes
the words it flushes and no others, so its streams end this way. A word after the
last symbol's, or a state the encoder did not start from, codes the same symbols in
other bytes; the rule closes both, so neither can give one decode several content
hashes. A stream that codes no symbol (the residual stream of a one-frame `WIV1`
with keyframe 1) is the state `2^31` alone: the words `2^31` and `0`.
**One clip, several payloads.** These rules fix how a payload is read, not the
only payload of a decoded output: tables that differ in entries the stream never
selects (a symbol the clip does not use, say), or a different escape layout, code
the same output in other bytes. A payload's hash therefore identifies the payload,
not what it decodes to. Where one output must have one identity, it is identified by
the digest of its decoded output: for a clip, its frame-state-equivalence digests
(SPEC §7, a video receipt's `clip_hash`); for the other formats, their capabilities'
equivalence digests.
**CDF tables.** A table is `n` entries of `u32` with `3 ≤ n ≤ 4097`, `cdf[0] = 0`,
`cdf[n − 1] = 65536` and every entry greater than the one before. These rules are
what keep the symbol search and the state update above in range.
**Order of the checks.** A parser checks, in this order, and refuses with the first
that fails:
1. the magic: the capability's (§1), or else `format_capability_mismatch` if it is
another of the four magics, or `bad_magic`;
2. the version byte: present (`stream_length`), and 1 (`version`);
3. the fixed header: present (`stream_length`);
4. the reserved fields (`reserved_nonzero`);
5. the header's fields, in the order the format's section lists their rules
(`field_range`), then, for `WIV1`, the keyframe kind (`keyframe_capability_mismatch`);
6. the caps of §7 on the sizes the header declares (`too_large`), in the order the
format's section lists them;
7. each later section in order: present (`stream_length`), then its rules
(`field_range`, `cdf_malformed`). A section of one table per channel (`WIA1`,
`WIS1`) is read table by table: each table present, then checked, before the next;
8. nothing after the last section (`trailing`, or `nested_length` for `WIV1`'s
nested keyframe).
A decode then makes the checks of §7, and refuses a stream that runs out or does not
end canonically (§2). Two conforming sinks refuse the same bytes with the same code.
## 3. `WIV1` — integer inter-frame video
| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIV1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `mv_precision` | `1` (whole-pel) or `2` (half-pel); `1` when `n_frames` is 1 |
| 6 | 1 | `keyframe` | `0` or `1` |
| 7 | 1 | reserved | `0` |
| 8 | 4 | `n_frames` | ≥ 1 |
| 12 | 4 | `h` | ≥ 1 |
| 16 | 4 | `w` | ≥ 1 |
| 20 | 4 | `block` | ≥ 1; `1` when `n_frames` is 1 |
| 24 | 4 | `q` | 1..=65535 |
| 28 | 4 | `offset` | `i32` |
| 32 | 2 | `cdf_len` | 3..=4097 |
| 34 | 2 | reserved | `0` |
| 36 | `cdf_len`·4 | the residual table | §2 |
| … | (`n_frames` − 1)·`nb`·4 | motion vectors | `i16 dy, i16 dx` |
| … | 4 + `L` | the residual stream | §2 |
| … | 4 + `K` | keyframe 1 only: `u32 K`, then a `WIH1` payload | `K` is exactly the bytes that remain |
The header's field rules are checked in the order `mv_precision`, `keyframe`,
`n_frames`, `h` and `w`, `block`, `q`, `cdf_len`, then, for a one-frame clip,
`mv_precision` 1 and `block` 1: such a clip codes no motion vector, so neither field
would change its decode, and they are fixed so that it has one header. `keyframe` 0 is
`wai.video.int_motion`'s and 1 is `wai.video.int_hyper`'s; the other is
`keyframe_capability_mismatch`. The frame `h·w·3`, the clip `n_frames·h·w·3`, the frame
count `n_frames` and the motion-vector field `2·nb·(n_frames − 1)` are capped (§7), in
that order.
**Motion vectors.** `nb = ⌈h/block⌉·⌈w/block⌉` and `nbx = ⌈w/block⌉`. The vectors
are frames 1 to `n_frames − 1` in order, each its block grid row-major: the vector of
pixel `(y, x)` in frame `f` is entry `(f − 1)·nb + (y div block)·nbx + (x div block)`,
in units of `1/mv_precision` pel.
**Decode.** One decoder reads the residual stream for the whole clip, against the
residual table at `offset`. For each frame in order, for each pixel in raster order
`(y, x)`, for each channel `c` of R, G, B: decode the symbol `r`, and the sample is
`clamp(pred + r·q, 0, 255)`. Since `|r| < 2^31` and `q < 2^16`, every step fits `i64`.
- Frame 0 with keyframe 0: `pred = 128`.
- Frame 0 with keyframe 1: the frame is the nested `WIH1`'s decode (§4), which reads
no symbol of the residual stream. It must be `h × w` (`geometry_mismatch`).
- Frame `f ≥ 1`: `pred` reads the previous **decoded** frame `R` with the pixel's
vector `(dy, dx)`, where `p(Y, X)` is `R` at row `clamp(Y, 0, h − 1)`, column
`clamp(X, 0, w − 1)`, channel `c`:
- whole-pel: `pred = p(y + dy, x + dx)`;
- half-pel: `iy = y + (dy >> 1)`, `fy = dy & 1`, `ix = x + (dx >> 1)`,
`fx = dx & 1` (arithmetic shift); then `p(iy, ix)` when both are 0,
`(p(iy, ix) + p(iy, ix + 1) + 1) >> 1` when only `fx` is 1,
`(p(iy, ix) + p(iy + 1, ix) + 1) >> 1` when only `fy` is 1, and
`(p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2`
when both are.
The output is `n_frames` RGB8 frames of `h × w` (rows top to bottom, pixels left to
right, R, G, B bytes): the buffers of the capability's frame-state-equivalence digests
(SPEC §7 *Digests*).
## 4. `WIH1` — the integer conditional image codec
| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIH1` | |
| 4 | 1 | version | `1` |
| 5 | 3 | reserved | `0` |
| 8 | 4 | `m` | ≥ 1: latent channels |
| 12 | 4 | `sh` | ≥ 1: latent height |
| 16 | 4 | `sw` | ≥ 1: latent width |
| 20 | 4 | `n_z` | ≥ 1: hyper-latent channels |
| 24 | 4 | `zh` | ≥ 1: hyper-latent height |
| 28 | 4 | `zw` | ≥ 1: hyper-latent width |
| 32 | 4 + `Lz` | the hyper-latent stream | §2 |
| … | 4 + `Ly` | the latent stream | §2 |
The caps (§7) apply to `n_z·zh·zw` and to `2·m·sh·sw`.
**Decode,** against the set of §9 (`g_s.whm` and `h_s.whm`, two integer networks at
one fixed-point scale `2^afrac`; `entropy.json`, the tables):
1. For each hyper-latent channel `k` and each of its `zh·zw` elements, decode a
symbol `s` from the hyper-latent stream against channel `k`'s table and offset;
the element is `s·2^afrac + median_k`.
2. `h_s` on the hyper-latent `[n_z, zh, zw]` gives `[2·m, sh, sw]`: its first `m`
channels are the scales `σ`, the rest the means `μ`.
3. For each latent element `p` in order (channel, then row, then column), its table
is `b`, where, with the scale edges `e_0 … e_{N−1}` and `σ' = max(σ_p, e_0)`,
`b = N − 1 − #{ i < N − 1 : σ' ≤ e_i }`. Decode a symbol `r` from the latent
stream against table `b` and its offset; the element is `r·2^afrac + μ_p`.
4. `g_s` on the latent `[m, sh, sw]` gives three channels; each value `v` is clamped
to `[0, 2^afrac]` and becomes `(v·255 + 2^(afrac−1)) >> afrac`, as RGB8.
Every value is formed in `i64` and must fit it (§7).
## 5. `WIA1` — the integer learned audio codec
| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIA1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `afrac` | 1..=30; equal to the model's (§7) |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `sr` | ≥ 1: the sample rate the PCM is presented at, in hertz |
| 12 | 4 | `n_ch` | ≥ 1: latent channels |
| 16 | 4 | `lat_len` | ≥ 1: latent length |
| 20 | 4 | `t_out` | ≥ 1: samples presented |
| 24 | 4 | `offset` | `i32`: the symbol offset of every channel's table |
| 28 | 2 | `cdf_len` | 3..=4097 |
| 30 | 2 | reserved | `0` |
| 32 | `n_ch`·8 | medians | `i64` each, `|median| ≤ 2^47` |
| … | `n_ch`·`cdf_len`·4 | one table per channel | §2 |
| … | 4 + `L` | the latent stream | §2 |
The header's field rules are checked in the order `afrac`, then `sr`, `n_ch`,
`lat_len` and `t_out`, then `cdf_len`. The caps (§7) apply to `n_ch·lat_len` and to
`t_out`.
**Decode,** against the set of §9 (`model.wam`, the integer synthesis):
1. For each channel `k` and each of its `lat_len` elements, decode a symbol `s`
against channel `k`'s table at `offset`; the element is `s·2^afrac + median_k`.
2. The synthesis (one-dimensional integer transposed convolutions and integer
inverse GDN) on `[n_ch, lat_len]` gives one channel of `t ≥ t_out` samples.
3. The first `t_out` samples `v` become `(v·32767 + 2^(afrac−1)) >> afrac`, clamped
to `[−32768, 32767]`.
The output is `t_out` samples of `i16` PCM, one channel, at `sr` hertz: the buffer of
the capability's sample-equivalence digest (signed 16-bit little-endian samples).
## 6. `WIS1` — the integer learned Gaussian-splat attribute codec
| off | size | field | rule |
|---|---|---|---|
| 0 | 4 | magic `WIS1` | |
| 4 | 1 | version | `1` |
| 5 | 1 | `afrac` | 1..=30; equal to the model's (§7) |
| 6 | 2 | reserved | `0` |
| 8 | 4 | `c` | ≥ 1: latent channels |
| 12 | 4 | `sh` | ≥ 1: latent height |
| 16 | 4 | `sw` | ≥ 1: latent width |
| 20 | 4 | `s` | ≥ 1: the attribute planes' side |
| 24 | 4 | `attr` | ≥ 1: attribute planes |
| 28 | 4 | `n_gaussians` | 1..=`s·s` |
| 32 | `attr`·8 | ranges | per plane, `f32 lo` then `f32 hi` (IEEE 754 binary32): finite, `lo ≤ hi` |
| … | `c`·2 | table lengths | `u16` each, 3..=4097 |
| … | `c`·4 | symbol offsets | `i32` each |
| … | `c`·8 | medians | `i64` each, `|median| ≤ 2^47` |
| … | the lengths' sum ·4 | one table per channel | §2 |
| … | 4 + `L` | the latent stream | §2 |
The header's field rules are checked in the order `afrac`, the dimensions,
`n_gaussians`. The caps (§7) apply to `c·sh·sw` and to `attr·s·s`.
**Decode,** against the set of §9 (`g_s.whm`, the integer synthesis):
1. For each channel `k` and each of its `sh·sw` elements, decode a symbol `s`
against channel `k`'s table and offset; the element is `s·2^afrac + median_k`.
2. `g_s` on `[c, sh, sw]` gives `[attr, s, s]` (`geometry_mismatch` otherwise).
3. Each value `v` is clamped to `[0, 2^afrac]` and becomes
`(v·255 + 2^(afrac−1)) >> afrac`, a byte.
The output, plane after plane, is the buffer of the capability's splat-equivalence
digest. The first `n_gaussians` cells of the planes, in raster order, hold one
Gaussian each. The ranges are presentation data: a sink presents a plane's byte `u`
at about `lo + (hi − lo)·u/255`. They are outside the digest, and this extension does
not fix presentation.
## 7. Caps and the checks a decode makes
**Caps, the same on every target.** No tensor a payload sizes may exceed `2^26`
elements: a frame `h·w·3`, a `WIV1` motion-vector field `2·nb·(n_frames − 1)`, a
latent, a hyper-latent, a network's output, an attribute tensor, a clip's `t_out`. A
`WIV1` clip may not exceed `2^30` bytes, nor `2^20` frames.
**Planning the networks.** Before a decode allocates a latent, it plans every network
on the payload's shapes from the layers' geometry and parameter lengths alone. The
network's input must be within the cap (`too_large`); then, layer by layer in order:
1. the layer fits the running channel count (`model_mismatch`): a convolution's output
channels, kernel and stride are non-zero, it has `c·out·k·k` weights (`c·out·k` for
the audio synthesis) and `out` multipliers and biases, and its requantisation shift
is under 128; an inverse GDN has `c·c` and `c` parameters and shifts under 128; a
leaky ReLU's shift is under 64;
2. every size the layer forms is within `2^26` (`too_large`): a convolution's padded
input side `n + 2·pad`, a transposed convolution's uncropped side
`(n − 1)·stride + k + opad` and uncropped plane, and the output;
3. the layer leaves an output (`model_mismatch`): a kernel no larger than the padded
input, a crop `2·pad` smaller than the uncropped side.
Every size is formed in 64-bit arithmetic with overflow checked, and a size past 64
bits is past the cap, so a 32-bit sink refuses exactly what a 64-bit one does, with
the same code; once a network is planned, every size it forms fits a 32-bit index.
**Memory and work.** A payload's header declares its decode's output and the work
that output takes, and a sink can read both before it decodes anything. A stream does
not bound them: a table can make one symbol cost a small fraction of a bit, so a few
kilobytes of stream can code a clip at the `2^30`-byte cap, as any codec can describe
a large output in a small file. The bound is the header's, within the caps:
- the output: a `WIV1` clip's `n_frames·h·w·3` bytes (one per sample), a `WIH1`
image's `3·h·w` bytes at its synthesis network's size, a `WIA1` clip's `2·t_out`
bytes (`i16` samples), a `WIS1` attribute tensor's `attr·s·s` bytes;
- the entropy decode: one step per symbol, `n_frames·h·w·3` for a keyframe-0 `WIV1`
(`h·w·3` fewer, plus its keyframe's, for keyframe 1), `n_z·zh·zw + m·sh·sw` for a
`WIH1`, `n_ch·lat_len` for a `WIA1`, `c·sh·sw` for a `WIS1`;
- the buffers: latents and network activations of 8 bytes per element (`i64`); while
a network layer runs, its input, its output and, for a transposed convolution, the
uncropped plane it accumulates into, each at most `2^26` elements (*Planning the
networks*); a `WIV1` decode's frames (one byte per sample) and its motion vectors
(4 bytes each);
- each network's work, its multiply-accumulates layer by layer: a convolution's
every output times `c·k·k` (every tap of its padded input), a transposed
convolution's every input times `out·k·k` (`out·k` for the audio synthesis), an
inverse GDN's every element times `c`; a `WIV1` with keyframe 0 runs no network. A
kernel may skip a zero term, so it performs at most this many.
A `WIA1`'s latent is the one its `t_out` needs and no longer: the synthesis of
`lat_len` samples gives at least `t_out` samples and that of `lat_len − 1` fewer
(`geometry_mismatch`), so a payload cannot declare a large latent to present a few
samples.
A decode refuses a stream that runs out at its first missing symbol (§2), so a short
stream stops early, but a stream that carries its symbols is decoded in full: a sink
that will not spend what a header declares refuses before decoding, by its own size
policy, applied to the declared cost (`int_payload::declared_cost` and
`int_payload::check_cost`, with the set; `neural::declared_cost` over the set a
registry resolved; the C ABI's `wai_int_payload_declared`, given the set's path;
`wai-web`'s `intPayloadCost`; the header alone through `container::declared_size`),
`cost_over_limit` (§8). The reference neural sink applies one when it is given one
(`neural::ModelRegistry::with_cost_limit`); its C ABI does not decode integer video,
PCM or splat attributes, which it has no output kind for. This extension fixes the
caps; a sink's policy may be stricter.
**The parameter set must fit the payload** (`model_mismatch`):
- every network parses (the `WHM1` and `WAM1` forms, §9), and the networks of one set
share one fixed-point scale `afrac` in 1..=30;
- a `WIA1` or `WIS1` payload's `afrac` is the model's;
- the entropy tables of `entropy.json` are consistent (a table, length and offset per
scale edge, at least one edge; a table, length, offset and median per hyper-latent
channel) and every table is a CDF of §2 over its length; a payload has no more
hyper-latent channels than the tables;
- each layer's parameters have the lengths the running channel count needs, and its
geometry leaves an output (*Planning the networks*);
- the audio synthesis gives one channel; the conditional codec's `g_s` gives three.
**The shapes the payload declares must be the ones the decode gives**
(`geometry_mismatch`): `h_s`'s output and `[2·m, sh, sw]`; a `WIV1` keyframe and
`h × w`; the audio synthesis's length and `t_out` (it may be longer, by less than one
latent sample's worth: the synthesis of `lat_len − 1` samples gives fewer than
`t_out`); `g_s`'s output and `[attr, s, s]`.
**The order of a decode's checks.** After the parse (§2), a decode checks in this
order and refuses with the first that fails:
1. the set's members, in the order of §9's list, each present (`set_incomplete`) and
parsed (`model_mismatch`);
2. the parameter set against the payload (`model_mismatch`): one fixed-point scale,
the payload's `afrac`, the entropy tables, the hyper-latent channels;
3. every network planned, in decode order (`h_s` then `g_s`; the audio synthesis;
the splat `g_s`), each layer by the three rules above (`model_mismatch`,
`too_large`): the caps come before any shape is compared;
4. what the networks give against the payload: for `WIH1`, `h_s`'s shape
(`geometry_mismatch`), then `g_s`'s three channels (`model_mismatch`); for `WIV1`
with keyframe 1, its keyframe's as for `WIH1`, then the keyframe's size against
`h × w` (`geometry_mismatch`); for `WIA1`,
one channel (`model_mismatch`), then at least `t_out` samples, then a latent no
longer than that needs (both `geometry_mismatch`); for `WIS1`, `g_s`'s shape
(`geometry_mismatch`);
5. each stream in decode order (`WIH1`: the hyper-latent's, then, after `h_s` runs,
the latent's; `WIV1` with keyframe 1: the keyframe's streams, then the residual
stream), symbol by symbol: a
symbol that runs its stream out (`stream_exhausted`) before its value's range
(`field_range`); after its last symbol, its canonical end (`stream_noncanonical`);
6. each network as it runs, layer by layer (`field_range`, below).
So a run-out is reported before the value it would have given, and a size past the
caps before a shape that disagrees. A sink that applies a cost limit (*Memory and
work*) compares the declared cost with it after step 4, once everything the cost reads
has been checked, and before any symbol is decoded (`cost_over_limit`, §8).
**Exactness.** A latent rebuilt as `s·2^afrac + base` must fit `i64`; before each
network layer runs, the decode checks on the values present that no product, partial
sum, requantised value or inverse-GDN term can leave the type its arithmetic is
formed in. A stream whose values fail either check is refused (`field_range`). So a
decode with overflow checks and one without compute the same integers, and neither
panics.
## 8. Refusal codes
The list is closed: a sink reports exactly one of these, the first that applies in the
order of §2 and §7.
A refusal of the payload or of the parameter set against it, `bad_magic` through
`model_mismatch` in the table below (and `payload_malformed`), is final: the sink does
not decode the payload by another path, and SPEC §4's fallback is not tried, since it
would decode the same bytes. The other codes say the sink cannot decode through the
capability at all, and route as SPEC §4 routes such a capability:
- `set_incomplete` is a pin that did not resolve (SPEC §3.1): a `WAI1` continues at
SPEC §4 step 4 (or reports it at step 6), and a `WAI2` moves on to its next entry
(SPEC §7.1);
- `not_dispatchable` and `unsupported_capability` name a capability the sink does not
decode: SPEC §4 does not take step 3 (or 5) with it, so a `WAI1` continues at step 4
and a `WAI2` at its next entry, as for a capability the sink does not advertise;
- `cost_over_limit` is the sink's own policy, not a fault of the payload: a payload
whose declared cost exceeds the limit the sink applies (§7, *Memory and work*) is
refused before any symbol is decoded, and routes as `unsupported_capability` does,
so a `WAI1` continues at step 4 and a `WAI2` at its next entry.
| code | when |
|---|---|
| `bad_magic` | the payload does not start with any of the four magics: a tool's JSON form lands here |
| `format_capability_mismatch` | the payload is another of the four formats than the capability's |
| `keyframe_capability_mismatch` | a `WIV1` keyframe kind that is not the capability's |
| `version` | a version this revision does not define |
| `reserved_nonzero` | a reserved field that is not zero |
| `field_range` | a field of any section outside its range (a header field, a `WIS1` range or table length, a median); or a value the stream decodes that leaves the exact integer range (§7) |
| `too_large` | a size past the caps of §7 |
| `cdf_malformed` | a table that breaks the rules of §2 |
| `stream_length` | the payload ends before a field or section it declares; or a stream length that is not a multiple of 4, or is under 8 |
| `nested_length` | a `WIV1` keyframe length `K` that is not the rest of the payload |
| `trailing` | bytes after the last section |
| `stream_exhausted` | a stream that ran out before its last symbol |
| `stream_noncanonical` | a stream that does not end canonically after its last symbol: a word left unread, or a state other than `2^31` (§2) |
| `geometry_mismatch` | a shape the payload declares that the decode does not give (§7) |
| `model_mismatch` | a parameter set that does not fit the payload (§7) |
| `set_incomplete` | a parameter set without a member the decode reads (§9); a pinned decode reports SPEC §3.1's `set_incomplete` |
| `not_dispatchable` | a capability SPEC §4 does not dispatch |
| `unsupported_capability` | a capability the sink has no integer decoder for |
| `payload_malformed` | a `wai.neural.int_synth` or `wai.neural.int_mlicpp` payload its decoder refuses |
| `cost_over_limit` | a payload whose declared cost (§7) exceeds the limit the sink applies before decoding |
## 9. Parameter sets
A capability that decodes against a parameter set reads these members, by the names a
set's listing gives them ([prior-carriage](prior-carriage.md) §7):
| capability | shape | members |
|---|---|---|
| `wai.neural.int_hyper`, `wai.video.int_hyper` | several files | `g_s.whm` and `h_s.whm` (networks in `WHM1`), `entropy.json` (the tables) |
| `wai.audio.int_codec` | one file | `model.wam` (the synthesis in `WAM1`) |
| `wai.splat.int_codec` | one file | `g_s.whm` (the synthesis in `WHM1`) |
| `wai.video.int_motion` | none | — |
A set of several files is pinned through its listing (`wai.prior.set`); a one-file set
MAY be pinned form-less, by its file's SHA-256 (SPEC §3.1). `wai.audio.int_codec` and
`wai.splat.int_codec` were registered with sets of several files, because no one-file
format had been named for them; this revision registers each as one file, the file its
decoder reads, which holds every parameter its output depends on. That widens their
registrations, as SPEC §8 allows a MINOR revision to; a set pinned through its listing
still verifies.
**`WHM1`** (`int_hyper_synth::IntModel::from_bytes`): magic, then `afrac`, `gf`,
`shift` and the layer count (`u32` each), then per layer a tag byte: 0 a transposed
convolution and 1 a convolution (input and output channels, kernel, stride, padding and,
for 0, output padding as `u32`; weights as `i16`; per-output-channel multipliers and
biases as `i64`), 2 an inverse GDN (channels as `u32`; `gamma` as `i32`, `beta` as
`i64`), 3 a ReLU, 4 a leaky ReLU (slope `i64`, its shift `u32`). **`WAM1`**
(`int_audio::AudioModel::from_bytes`): the same header, then layers of tag 0, a
one-dimensional transposed convolution, and tag 2, an inverse GDN. In both forms
nothing follows the last layer: a file with bytes after it is refused
(`model_mismatch`), so one network has one file and one digest. The layers' integer
arithmetic is the reference decoders' (`int_hyper_synth.rs`, `int_audio.rs`,
`int_transform.rs`), which the conformance vectors pin.
## 10. The tools' JSON forms
The encode and export tools under `tools/` write a JSON bitstream. It is a converter input and a
debugging aid, never a payload. An encoder MUST NOT carry it as the payload of any
capability of §1, and a sink refuses it (`bad_magic`). The converters
(`int_payload::{wiv1,wih1,wia1,wis1}_from_json`, and the independent
`int-payload-conformance/int_payload_verify.py`) make explicit what the JSON leaves
implicit:
- `WIV1`: an absent `mv_precision` is 1, and only 1 and 2 are converted; a one-frame
clip is written with `mv_precision` 1 and `block` 1, whatever the JSON states; `mvs` has one
list per frame, and the first is empty; a frame's list shorter than its block grid is
padded with `(0, 0)`, the vector the JSON decoder read for a missing block; a list
longer than the grid, or a vector outside `i16`, is refused; `rmax` is not carried.
The keyframe of `wai.video.int_hyper` is the `WIH1` converted from its own JSON.
- `WIH1`: the fields of `cond_bitstream.json`, the streams as they are.
- `WIA1`: `sr` is the export's `sample_rate` (in `audio.json`, which the bitstream
does not carry); `lo` is the symbol offset; every channel's table must have the first
table's length; `afrac` is carried and must be the model's.
- `WIS1`: the ranges `lo` and `hi` must each be a binary32 value, carried exactly (a
JSON integer included: `2^60 + 1` is refused, not rounded to `2^60`);
`afrac` is carried and must be the model's.
A payload and the JSON it is converted from decode to the same bytes; the conformance
vectors check that for every committed JSON bitstream.
## 11. Envelopes this revision makes non-conforming
This revision completes these strings' registrations with a payload format they did
not have (SPEC §8, *Completing a registration*). It makes non-conforming, and they
still parse:
- a `WAI1` envelope whose `capability` or `fallback` is a string of §1 and whose
payload is not that string's format: a tool's JSON form among them;
- a `WAI2` envelope with a component of a string of §1 whose bytes are not that
string's format.
A sink refuses such a payload when it decodes it (§8), with no fallback attempt. An
encoder MUST NOT write one; the reference implementation's emit paths (`wai wrap`, the
C packers and the browser writers) parse the payload as a sink does and refuse to. An
encrypted envelope's payload (SPEC §6) is ciphertext until a sink decrypts it, so the
emit paths do not check it; the sink checks the plaintext when it decodes it (SPEC
§6.4).
No committed envelope or conformance corpus carried such a payload when this revision
was made.
## 12. Reference implementation and conformance
- `int_payload::wire` parses and writes the four formats, with no dependency on the
rest of the crate; `int_payload::decode_int` decodes every integer capability from
parameter files held in memory, with no ONNX runtime and no filesystem, for native
and `wasm32` sinks; `int_payload::resolve` verifies a pin over sets held in memory
(`pin::verify_bytes_with`) and runs SPEC §3.1's check 5, for a pinned decode and over
the files an unpinned decode reads (a listing the set carries must match them, and
the digests include the listing of the files read); it is the only way to build the
parameters a decode reads (`IntParams`). The neural sink (`neural::decode_payload`)
delegates every integer capability to `decode_int`, and its registry reads an
unpinned integer set once and runs check 5 over those bytes.
- `codecs::PINNED_INT_PAYLOADS` is the one table of strings and magics;
`container::check_int_payload_framing` parses a payload with `int_payload::wire` and
the capability's keyframe kind, which needs no integer decoder and no feature, so
every emit path runs it: `Wai::to_emit_bytes` and `WaiMulti::to_emit_bytes` (and so
the C packers), `wai-web`'s `pack` and `packMulti`, and `wai wrap`; `wai inspect`
reports it.
- `video_receipt::VideoReceipt::seal_decode_wiv1` seals a `wai.video.int_motion`
receipt whose identity is BLAKE3 over the `WIV1` payload.
- `wai-web` decodes the four formats in the browser (`decodeWiv1`, `decodeWih1`,
`decodeWia1`, `decodeWis1`).
- Conformance: the payloads beside their JSON sources in `wai-rs/tests/vectors/`,
and the refusal fixtures and index in
[`int-payload-conformance/`](../int-payload-conformance/), written by
`wai_int_payload_vectors gen` and checked by `wai_int_payload_vectors verify`; the
same corpus checked by `int_payload_verify.py`, which shares no code with wai-rs and
decodes the `WIV1` keyframe-0 payloads in full; and the cross-architecture goldens of
`byte-exact-conformance`, which parse and decode the `WIV1` and `WIH1` payloads on
every platform of its matrix.

12
wai/extensions/jwp-receipts.md
View file

@ -2602,10 +2602,14 @@ left the file inert because no fallback was declared.
`key_log::module_check` in `wai-rs` implements these checks. The
reference dispatcher runs SPEC §3.1's check 5 when the sink is
configured with a key log (`key_log::SinkKeyLog`, given to
`pin::verify_with` or to `neural::ModelRegistry::with_key_log`, which
also runs it before an unpinned decode): the revocation lists for every
capability a pin applies to, and these checks for the capabilities a sink
holds to this section. A sink configured with no key log runs neither. The reference dispatcher runs no module obtained
`pin::verify_with`, to `int_payload::resolve`, which runs it for a pinned
decode and over the files an unpinned integer decode reads, or to
`neural::ModelRegistry::with_key_log`, which also runs it before an
unpinned decode, over the bytes that decode then reads, read once, by
every digest a pin over them could name): the revocation
lists for every capability a pin applies to, and these checks for the
capabilities a sink holds to this section. A sink configured with no key
log runs neither. The reference dispatcher runs no module obtained
apart from an envelope, and `wai-web` and the C ABI take no key log
([SPEC](../SPEC.md) §4).

10
wai/extensions/prior-carriage.md
View file

@ -287,7 +287,10 @@ WAI's binding sits beside the standard's, never in place of it.
decode through.
- The decoders read parameters only through `neural::ParamSource`, which hands a
pinned decode the verified bytes; ONNX sessions are built from those bytes, and
a file the verified set lacks is refused with `set_incomplete`.
a file the verified set lacks is refused with `set_incomplete`. The integer
capabilities decode through `int_payload::decode_int`, which reads its set members
from memory; `int_payload::resolve` verifies a pin over sets held in memory and
runs check 5, pinned or unpinned ([integer-payloads](integer-payloads.md) §12).
`wai.neural.bmshj2018` caches its CDF tables by content digest.
- The C ABI (`ffi`) returns each manifest as its original bytes, carries a `WAI2`
rendition's identity and pin in the rendition table, and reports a refused pin
@ -311,6 +314,9 @@ file the set lacks is refused with `set_incomplete`.
|---|---|---|
| `wai.neural.int_mlicpp` | one file | `model.wmb` |
| `wai.neural.int_synth` | one file | `int_synth.json` |
| `wai.audio.int_codec` | one file | `model.wam` |
| `wai.splat.int_codec` | one file | `g_s.whm` |
| `wai.neural.int_hyper`, `wai.video.int_hyper` | several | `g_s.whm`, `h_s.whm`, `entropy.json` |
| `wai.neural.encodec32`, `.dac`, `.mimi`, `.wavtokenizer`, `.snac`, `.video_bmshj2018` | one file | `decoder.onnx` |
| `wai.neural.bmshj2018` | several | `decoder.onnx`, `cdfs_q<q>.json` |
| `wai.neural.mbt2018_mean` | several | `hyper_cdfs_q<q>.json`, `hyper_synthesis.onnx`, `gauss_cdfs_q<q>.json`, `synthesis.onnx` |
@ -319,7 +325,7 @@ file the set lacks is refused with `set_incomplete`.
| `wai.neural.glc` | several | `hyper_cdfs_q<q>.json`, `hyper_entropy.onnx`, `context_entropy.onnx`, `synthesis.onnx` |
| `wai.neural.dcvc_rt`, `.dcvc_fm` | several | `hyper_cdfs_q<q>.json`, `hyper_synthesis.onnx`, `gauss_cdfs_q<q>.json`, `temporal_decoder.onnx` |
| `wai.image.jpegai` | several | exact path: `norm.json`, `weights/{hs,hyp,synth}_{y,uv}.json`, `weights/mcm{0..3}_y.json`, `weights/icci.json`. Float path: `hyper_scale_decoder_int_weights.json`, `model_constants.json`, `jpegai_normative_tables.json`, `me-tANS/bounds.csv`, `jpegai_factorized_cdfs.json`, `jpegai_scaler_vec.json`, `jpegai_dequant_consts.json`, `model/CCS_SGMM/tools_0/model_{y,uv}/common_modules/hyper_decoder.onnx`, `model/CCS_SGMM/tools_0/model_y/common_modules/MCM/stage{0..3}.onnx`, `model16/CCS_SGMM/tools_0/model_{y,uv}/synthesis16.onnx`, `e2e/pfchain/icci_{y_m6,uv_m3}.onnx`. A set holding only the exact path's files decodes through that path; where that path refuses (a non-default floating-point environment), the float path needs files the set lacks, and the decode is refused with `set_incomplete` instead of running on unverified files. |
| `wai.neural.int_hyper`, `wai.video.int_hyper`, `wai.audio.int_codec`, `wai.splat.int_codec`, `wai.generate.spectral` | several, until an envelope decode path registers its files | — (no envelope decode path in the reference sink) |
| `wai.generate.spectral` | several, until an envelope decode path registers its files | — (no envelope decode path in the reference sink) |
## 8. Sources

2
wai/int-payload-conformance/.gitattributes vendored Normal file
View file

@ -0,0 +1,2 @@
# Hashed files: a line-ending conversion on checkout would change every digest.
* -text

80
wai/int-payload-conformance/README.md Normal file
View file

@ -0,0 +1,80 @@
# int-payload-conformance — the integer payloads
Vectors for the payload formats SPEC §5 pins to the WAI-native integer codecs
([`extensions/integer-payloads.md`](../extensions/integer-payloads.md)): `WIV1` for
`wai.video.int_motion` (keyframe 0) and `wai.video.int_hyper` (keyframe 1), `WIH1`
for `wai.neural.int_hyper`, `WIA1` for `wai.audio.int_codec` and `WIS1` for
`wai.splat.int_codec`.
**Code-equivalence.** A conforming sink decodes every payload to the output digests
its vector states, and refuses every fixture with the code it states, from the
extension's closed list (§8).
- `vectors.json` — the index. For each payload: its capability, its path, its
SHA-256, the JSON it is converted from, the parameter set it decodes against (each
member's name in the set, and the committed file that holds it), and the
`sha256.json` sidecar that states its decoded output's SHA-256. For each refusal
fixture: its capability, its path, its code, and whether parsing alone refuses it
(`parse`) or the decode does (`decode`). Paths are relative to `wai/`.
- The payloads sit beside their JSON sources under
[`../wai-rs/tests/vectors/`](../wai-rs/tests/vectors/): `int_video/*.wiv1`,
`int_hyper/tiny_bitstream.wih1`, `int_audio/audio_bitstream.wia1` and
`int_splat/splat.wis1`. Their sidecars are the ones the JSON forms' decodes are
checked against: a payload and its JSON source decode to the same bytes.
- `refusals/*.bin` — each a committed payload changed in one place (a byte, a
field, a truncation, a word appended to a stream), so that it is refused with one
code, or, for `wiv1_end_state` and `wiv1_one_frame_end_state`, a small clip written
to end its stream in a state other than `2^31`. The fixtures named `*_order_*`
break two rules at once and pin which one a sink reports (the extension's §2 and
§7), and so do the `wiv1_geometry_flip_*` fixtures: `wiv1_geometry` with one byte
of its keyframe's last stream words changed, refused for the keyframe's size before
that stream is read. The fixtures for `bad_magic` are the JSON sources themselves: a tool's JSON
output is a converter input, never a payload.
- `sets/` — a parameter-set member a fixture decodes against in place of a committed
one: `entropy_median_max.json` is the tiny conditional codec's tables with one
median at `i64::MAX`.
Files are compared as bytes; `.gitattributes` turns off line-ending conversion.
**Generation.** Every file here, and every payload under `tests/vectors/`, is
written by one generator from the committed JSON alone, so it can be rebuilt and
compared:
cd wai/wai-rs
cargo run --no-default-features --features video_int,audio_int,splat_int \
--bin wai_int_payload_vectors -- gen ../int-payload-conformance
**Verifiers.**
- `wai_int_payload_vectors verify ../int-payload-conformance` (wai-rs) regenerates
every file and requires it to equal the committed one byte for byte, decodes every
payload through `int_payload::decode_int` to its sidecar's SHA-256, and requires
every fixture to be refused with its code at its layer, and by
`int_payload::declared_cost` with the same code unless the decode's refusal comes
from a stream or a network run. `tests/int_payload.rs` runs
the fixtures again in both build profiles, refuses every strict prefix of every
payload, and runs a sweep of single-byte changes in which nothing may panic;
`tests/int_payload_preflight.rs` checks that a header past the caps or past what the
parameter set can fit is refused before anything sized from it is allocated, and
that a payload whose streams cannot carry what its header declares is refused at
its first exhausted symbol, having allocated only what it decoded.
- `int_payload_verify.py`, Python standard library only and sharing no code with
wai-rs, written from the extension: it parses every payload with the extension's
rules and check order, converts every JSON source independently and requires the
committed bytes, decodes the `WIV1` payloads with keyframe 0 in full (the rANS
decoder with its bypass escape, its stop at the first exhausted symbol and its
canonical end, whole-pel and half-pel motion compensation) to their sidecars'
SHA-256, and checks every fixture: a `parse` fixture's code, and that a `decode`
fixture parses (a `WIV1` keyframe-0 one is decoded and must be refused with its
code).
python3 int-payload-conformance/int_payload_verify.py int-payload-conformance
**Not decoded independently.** The Python verifier does not decode `WIH1`, `WIV1`
with keyframe 1 (its keyframe is a `WIH1`), `WIA1` or `WIS1`: their decodes run
integer networks it does not implement. Their outputs are checked by wai-rs against
the sidecars above, and the `WIV1` and `WIH1` payloads also by the
cross-architecture goldens in
[`../byte-exact-conformance/`](../byte-exact-conformance/). For their `decode`
fixtures the Python verifier checks that they parse; their codes are checked by
wai-rs.

544
wai/int-payload-conformance/int_payload_verify.py Normal file
View file

@ -0,0 +1,544 @@
#!/usr/bin/env python3
"""Independent verifier for the integer payload corpus (extensions/integer-payloads.md).
Python standard library only; it shares no code with wai-rs. Written from the
extension's text, it:
- parses every committed payload (WIV1, WIH1, WIA1, WIS1) with the rules and in the
check order of the extension's §2-§6, and requires each to parse;
- converts each payload's JSON source independently (§10) and requires the result to
equal the committed payload byte for byte, and the payload's SHA-256 to be the one
the index states;
- decodes the WIV1 payloads with keyframe 0 in full - the rANS decoder with its
bypass escape, its stop at the first exhausted symbol and its canonical end,
whole-pel and half-pel motion compensation - and requires every frame's SHA-256
and the clip's to be the ones the vector's sha256.json states;
- parses every refusal fixture and requires a parse-layer fixture to be refused with
its code, and a decode-layer fixture to parse; a decode-layer fixture of WIV1
keyframe 0 is decoded and must be refused with its code (stream_exhausted,
stream_noncanonical).
Not decoded independently: WIH1, WIV1 with keyframe 1 (its keyframe is a WIH1), WIA1
and WIS1, whose decodes run integer networks this script does not implement. Their
decoded outputs are checked by wai-rs against the same sidecars, on every leg of the
cross-architecture matrix for WIV1 and WIH1. The decode-layer fixtures of those
formats are checked here to parse, and by wai-rs for their codes.
python3 int-payload-conformance/int_payload_verify.py int-payload-conformance
Exit status 1 on any failure.
"""
import base64
import hashlib
import json
import math
import os
import struct
import sys
MAGICS = (b"WIV1", b"WIH1", b"WIA1", b"WIS1")
MAX_TENSOR = 1 << 26
MAX_CLIP = 1 << 30
MAX_FRAMES = 1 << 20
MAX_MEDIAN_Q = 1 << 47
CAP_MAGIC = {
"wai.video.int_motion": b"WIV1",
"wai.video.int_hyper": b"WIV1",
"wai.neural.int_hyper": b"WIH1",
"wai.audio.int_codec": b"WIA1",
"wai.splat.int_codec": b"WIS1",
}
class Refused(Exception):
def __init__(self, code, why):
super().__init__(f"{code}: {why}")
self.code = code
class Reader:
def __init__(self, b, p):
self.b, self.p = b, p
def left(self):
return len(self.b) - self.p
def take(self, n):
if n > self.left():
raise Refused("stream_length", "the payload ends inside a section")
s = self.b[self.p:self.p + n]
self.p += n
return s
def u8(self):
return self.take(1)[0]
def u16(self):
return struct.unpack("<H", self.take(2))[0]
def u32(self):
return struct.unpack("<I", self.take(4))[0]
def i32(self):
return struct.unpack("<i", self.take(4))[0]
def cdf(self, n):
t = list(struct.unpack(f"<{n}I", self.take(4 * n)))
check_cdf(t)
return t
def stream(self):
n = self.u32()
if n % 4 or n < 8:
raise Refused("stream_length", "a rANS stream's length is not a multiple of 4, or is under 8")
return self.take(n)
def end(self):
if self.left():
raise Refused("trailing", "bytes follow the last section")
def check_cdf(t):
if not 3 <= len(t) <= 4097:
raise Refused("cdf_malformed", "length outside 3..=4097")
if t[0] != 0 or t[-1] != 65536:
raise Refused("cdf_malformed", "not from 0 to 65536")
if any(a >= b for a, b in zip(t, t[1:])):
raise Refused("cdf_malformed", "not strictly increasing")
def check_magic(p, want):
m = p[:4]
if len(m) == 4 and m == want:
return
if len(m) == 4 and m in MAGICS:
raise Refused("format_capability_mismatch", "another integer format")
raise Refused("bad_magic", "not the capability's magic")
def opening(p, magic, header):
check_magic(p, magic)
if len(p) < 5:
raise Refused("stream_length", "no version byte")
if p[4] != 1:
raise Refused("version", "a version this revision does not define")
if len(p) < header:
raise Refused("stream_length", "the payload ends inside its fixed header")
return Reader(p, 5)
def too_large(dims, cap):
n = 1
for d in dims:
n *= d
if n > cap:
raise Refused("too_large", "over the cap")
def parse_wiv1(p, want_keyframe=None):
r = opening(p, b"WIV1", 36)
mvp, kf, res0 = r.u8(), r.u8(), r.u8()
n_frames, h, w, block, q = r.u32(), r.u32(), r.u32(), r.u32(), r.u32()
offset, cdf_len, res1 = r.i32(), r.u16(), r.u16()
if res0 or res1:
raise Refused("reserved_nonzero", "reserved")
if mvp not in (1, 2):
raise Refused("field_range", "mv_precision")
if kf > 1:
raise Refused("field_range", "keyframe")
if 0 in (n_frames, h, w):
raise Refused("field_range", "n_frames, h or w")
if block == 0:
raise Refused("field_range", "block")
if not 1 <= q <= 65535:
raise Refused("field_range", "q")
if not 3 <= cdf_len <= 4097:
raise Refused("field_range", "cdf_len")
if n_frames == 1 and (mvp != 1 or block != 1):
raise Refused("field_range", "a one-frame clip has mv_precision 1 and block 1")
if want_keyframe is not None and want_keyframe != kf:
raise Refused("keyframe_capability_mismatch", "keyframe")
too_large([h, w, 3], MAX_TENSOR)
too_large([n_frames, h, w, 3], MAX_CLIP)
too_large([n_frames], MAX_FRAMES)
nb = -(-h // block) * -(-w // block)
too_large([2, n_frames - 1, nb], MAX_TENSOR)
cdf = r.cdf(cdf_len)
raw = r.take((n_frames - 1) * nb * 4)
mvs = [struct.unpack_from("<hh", raw, 4 * i) for i in range((n_frames - 1) * nb)]
rans = r.stream()
wih1 = None
if kf == 1:
k = r.u32()
if k != r.left():
raise Refused("nested_length", "K is not the rest")
wih1 = r.take(k)
parse_wih1(wih1)
else:
r.end()
return dict(mv_precision=mvp, keyframe=kf, n_frames=n_frames, h=h, w=w, block=block, q=q,
offset=offset, cdf=cdf, mvs=mvs, rans=rans, wih1=wih1)
def parse_wih1(p):
r = opening(p, b"WIH1", 32)
res = r.take(3)
m, sh, sw, n_z, zh, zw = (r.u32() for _ in range(6))
if any(res):
raise Refused("reserved_nonzero", "reserved")
if 0 in (m, sh, sw, n_z, zh, zw):
raise Refused("field_range", "a dimension")
too_large([n_z, zh, zw], MAX_TENSOR)
too_large([2, m, sh, sw], MAX_TENSOR)
z, y = r.stream(), r.stream()
r.end()
return dict(m=m, sh=sh, sw=sw, n_z=n_z, zh=zh, zw=zw, z_rans=z, rans=y)
def parse_wia1(p):
r = opening(p, b"WIA1", 32)
afrac, res0 = r.u8(), r.u16()
sr, n_ch, lat_len, t_out = r.u32(), r.u32(), r.u32(), r.u32()
offset, cdf_len, res1 = r.i32(), r.u16(), r.u16()
if res0 or res1:
raise Refused("reserved_nonzero", "reserved")
if not 1 <= afrac <= 30:
raise Refused("field_range", "afrac")
if 0 in (sr, n_ch, lat_len, t_out):
raise Refused("field_range", "a dimension")
if not 3 <= cdf_len <= 4097:
raise Refused("field_range", "cdf_len")
too_large([n_ch, lat_len], MAX_TENSOR)
too_large([t_out], MAX_TENSOR)
med = list(struct.unpack(f"<{n_ch}q", r.take(8 * n_ch)))
if any(abs(x) > MAX_MEDIAN_Q for x in med):
raise Refused("field_range", "median")
cdfs = [r.cdf(cdf_len) for _ in range(n_ch)]
rans = r.stream()
r.end()
return dict(afrac=afrac, sr=sr, n_ch=n_ch, lat_len=lat_len, t_out=t_out, offset=offset,
median_q=med, cdfs=cdfs, rans=rans)
def parse_wis1(p):
r = opening(p, b"WIS1", 32)
afrac, res = r.u8(), r.u16()
c, sh, sw, s, attr, n_g = (r.u32() for _ in range(6))
if res:
raise Refused("reserved_nonzero", "reserved")
if not 1 <= afrac <= 30:
raise Refused("field_range", "afrac")
if 0 in (c, sh, sw, s, attr, n_g):
raise Refused("field_range", "a dimension")
if n_g > s * s:
raise Refused("field_range", "n_gaussians")
too_large([c, sh, sw], MAX_TENSOR)
too_large([attr, s, s], MAX_TENSOR)
raw = r.take(8 * attr)
ranges = [struct.unpack_from("<ff", raw, 8 * i) for i in range(attr)]
for lo, hi in ranges:
if not (math.isfinite(lo) and math.isfinite(hi)) or lo > hi:
raise Refused("field_range", "a range")
lens = list(struct.unpack(f"<{c}H", r.take(2 * c)))
if any(not 3 <= n <= 4097 for n in lens):
raise Refused("field_range", "a CDF length")
offs = list(struct.unpack(f"<{c}i", r.take(4 * c)))
med = list(struct.unpack(f"<{c}q", r.take(8 * c)))
if any(abs(x) > MAX_MEDIAN_Q for x in med):
raise Refused("field_range", "median")
cdfs = [r.cdf(n) for n in lens]
rans = r.stream()
r.end()
return dict(afrac=afrac, c=c, sh=sh, sw=sw, s=s, attr=attr, n_gaussians=n_g, ranges=ranges,
cdfs=cdfs, offsets=offs, median_q=med, rans=rans)
def parse(cap, p):
m = CAP_MAGIC[cap]
if m == b"WIV1":
return parse_wiv1(p, 1 if cap == "wai.video.int_hyper" else 0)
return {b"WIH1": parse_wih1, b"WIA1": parse_wia1, b"WIS1": parse_wis1}[m](p)
# ---- writers: the JSON sources converted independently (extensions/integer-payloads.md §10) ----
def put_stream(b):
return struct.pack("<I", len(b)) + b
class NotConvertible(Exception):
"""A JSON source the converters of §10 refuse. Raised explicitly, never through
`assert`, so `python3 -O` checks the same rules."""
def require(ok, why):
if not ok:
raise NotConvertible(why)
def wiv1_from_json(j, keyframe=None):
nf, h, w, block = j["n_frames"], j["h"], j["w"], j["block"]
mvp = j.get("mv_precision", 1)
require(mvp in (1, 2), "mv_precision is neither 1 nor 2")
if nf == 1:
# A one-frame clip's header fixes mv_precision 1 and block 1 (§3, §10).
mvp, block = 1, 1
nb = -(-h // block) * -(-w // block)
mvs = j["mvs"]
require(len(mvs) == nf and not mvs[0], "mvs is not one list per frame, the first empty")
out = b"".join([b"WIV1", bytes([1, mvp, 1 if keyframe else 0, 0]),
struct.pack("<IIIIIiHH", nf, h, w, block, j["q"], j["offset"], len(j["cdf"]), 0),
struct.pack(f"<{len(j['cdf'])}I", *j["cdf"])])
for fr in mvs[1:]:
require(isinstance(fr, list), "a frame's motion vectors are not a list")
require(len(fr) <= nb, "a frame carries more motion vectors than blocks")
for dy, dx in list(fr) + [[0, 0]] * (nb - len(fr)):
out += struct.pack("<hh", dy, dx)
out += put_stream(base64.b64decode(j["rans_b64"]))
if keyframe:
out += put_stream(keyframe)
return out
def wih1_from_json(j):
return (b"WIH1" + bytes([1, 0, 0, 0]) + struct.pack("<6I", j["m"], j["sh"], j["sw"], j["n_z"], j["zh"], j["zw"])
+ put_stream(bytes(j["z_rans"])) + put_stream(bytes(j["rans"])))
def wia1_from_json(j, sr):
n = len(j["cdfs"][0])
require(all(len(t) == n for t in j["cdfs"]), "the channels' tables are not of one length")
return (b"WIA1" + bytes([1, j["afrac"], 0, 0])
+ struct.pack("<IIIIiHH", sr, j["n_ch"], j["lat_len"], j["t_out"], j["lo"], n, 0)
+ struct.pack(f"<{j['n_ch']}q", *j["median_q"])
+ b"".join(struct.pack(f"<{n}I", *t) for t in j["cdfs"])
+ put_stream(base64.b64decode(j["rans_b64"])))
def f32(x):
"""`x` as IEEE 754 binary32, refused unless it is exactly a binary32 value (§10)."""
require(isinstance(x, (int, float)) and math.isfinite(x), f"{x!r} is not a finite number")
try:
b = struct.pack("<f", x)
except OverflowError:
raise NotConvertible(f"{x!r} is not a binary32 value")
require(struct.unpack("<f", b)[0] == x, f"{x!r} is not a binary32 value")
return b
def wis1_from_json(j):
c = j["c"]
return (b"WIS1" + bytes([1, j["afrac"], 0, 0])
+ struct.pack("<6I", c, j["sh"], j["sw"], j["s"], j["attr"], j["n_gaussians"])
+ b"".join(f32(lo) + f32(hi) for lo, hi in zip(j["lo"], j["hi"]))
+ struct.pack(f"<{c}H", *(len(t) for t in j["cdfs"]))
+ struct.pack(f"<{c}i", *j["offsets"])
+ struct.pack(f"<{c}q", *j["median_q"])
+ b"".join(struct.pack(f"<{len(t)}I", *t) for t in j["cdfs"])
+ put_stream(base64.b64decode(j["rans_b64"])))
# ---- the rANS decoder (precision 16, bypass 4) and the WIV1 keyframe-0 decode ----
I32_MIN, I32_MAX = -(1 << 31), (1 << 31) - 1
def sat(v):
return max(I32_MIN, min(I32_MAX, v))
class Rans:
def __init__(self, b):
self.w = [struct.unpack_from("<I", b, i)[0] for i in range(0, len(b), 4)]
self.i, self.x, self.exhausted = 2, (self.w[1] << 32) | self.w[0], False
def renorm(self):
if self.x < (1 << 31):
if self.i >= len(self.w):
self.exhausted = True
return
self.x = (self.x << 32) | self.w[self.i]
self.i += 1
def bits(self, n):
v = self.x & ((1 << n) - 1)
self.x >>= n
self.renorm()
return v
def decode(self, cdf, offset):
n = len(cdf)
max_value = n - 2
cum = self.x & 0xFFFF
lo, hi = 1, n - 1
while lo < hi:
mid = (lo + hi) // 2
if cdf[mid] > cum:
hi = mid
else:
lo = mid + 1
s = lo - 1
self.x = (cdf[s + 1] - cdf[s]) * (self.x >> 16) + (self.x & 0xFFFF) - cdf[s]
self.renorm()
value = s
if value == max_value:
val = self.bits(4)
count, groups = val, 0
while val == 15 and groups < 8 and not self.exhausted:
val = self.bits(4)
count += val
groups += 1
raw = 0
for j in range(min(count, 8)):
raw |= self.bits(4) << (4 * j)
value = raw >> 1
value = sat(-value - 1) if raw & 1 else sat(value + max_value)
return sat(value + offset)
def decode_wiv1_k0(v):
h, w, block, q, prec = v["h"], v["w"], v["block"], v["q"], v["mv_precision"]
nbx = -(-w // block)
nb = nbx * -(-h // block)
d = Rans(v["rans"])
frames, prev = [], None
for f in range(v["n_frames"]):
mvs = v["mvs"][(f - 1) * nb:f * nb] if f else None
# The frame grows a sample at a time as its symbols decode: nothing is
# allocated ahead of the stream that fills it.
cur = bytearray()
for y in range(h):
for x in range(w):
dy, dx = mvs[(y // block) * nbx + x // block] if f else (0, 0)
for c in range(3):
if f == 0:
pred = 128
else:
def p(yy, xx):
return prev[(min(max(yy, 0), h - 1) * w + min(max(xx, 0), w - 1)) * 3 + c]
if prec == 1:
pred = p(y + dy, x + dx)
else:
iy, fy, ix, fx = y + (dy >> 1), dy & 1, x + (dx >> 1), dx & 1
if not fy and not fx:
pred = p(iy, ix)
elif not fy:
pred = (p(iy, ix) + p(iy, ix + 1) + 1) >> 1
elif not fx:
pred = (p(iy, ix) + p(iy + 1, ix) + 1) >> 1
else:
pred = (p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2
r = d.decode(v["cdf"], v["offset"])
if d.exhausted:
# §2: the first symbol during whose decode the stream runs out.
raise Refused("stream_exhausted", "the residual stream ran out")
cur.append(min(max(pred + r * q, 0), 255))
frames.append(bytes(cur))
prev = cur
if d.i != len(d.w) or d.x != 1 << 31:
# §2: after the last symbol every word has been read and the state is 2^31.
raise Refused("stream_noncanonical", "the residual stream does not end canonically")
return frames
def main(cdir):
wai = os.path.normpath(os.path.join(cdir, ".."))
def rd(rel):
with open(os.path.join(wai, rel), "rb") as f:
return f.read()
def jl(rel):
return json.loads(rd(rel))
idx = jl("int-payload-conformance/vectors.json")
fails, decoded, parsed_only = [], 0, []
for v in idx["vectors"]:
name, cap = v["name"], v["capability"]
p = rd(v["payload"])
try:
fields = parse(cap, p)
except Refused as e:
fails.append(f"{name}: does not parse: {e}")
continue
if hashlib.sha256(p).hexdigest() != v["payload_sha256"]:
fails.append(f"{name}: payload SHA-256 differs from the index")
src = v["converted_from"]
j = jl(src["json"])
m = CAP_MAGIC[cap]
try:
if m == b"WIV1":
conv = wiv1_from_json(j, rd(src["keyframe"]) if "keyframe" in src else None)
elif m == b"WIH1":
conv = wih1_from_json(j)
elif m == b"WIA1":
conv = wia1_from_json(j, jl(src["sample_rate"])["sample_rate"])
else:
conv = wis1_from_json(j)
except NotConvertible as e:
fails.append(f"{name}: {src['json']} is not convertible: {e}")
continue
if conv != p:
fails.append(f"{name}: the independent conversion of {src['json']} differs from the payload")
if m == b"WIV1" and fields["keyframe"] == 0:
frames = decode_wiv1_k0(fields)
side = jl(v["sidecar"])[name]
digests = [hashlib.sha256(f).digest() for f in frames]
if [d.hex() for d in digests] != side["frames_sha256"] or hashlib.sha256(b"".join(digests)).hexdigest() != side["clip_sha256"]:
fails.append(f"{name}: the independent decode differs from {v['sidecar']}")
else:
decoded += 1
print(f"ok {name}: parsed, converted, decoded to its sidecar's SHA-256 ({len(frames)} frames)")
else:
parsed_only.append(name)
print(f"ok {name}: parsed and converted (not decoded independently)")
checked, decode_layer = 0, []
for f in idx["refusals"]:
name, cap, code, layer = f["name"], f["capability"], f["code"], f["layer"]
p = rd(f["payload"])
try:
fields = parse(cap, p)
got = None
except Refused as e:
got = e.code
if layer == "parse":
if got != code:
fails.append(f"{name}: parsed with {got}, want {code}")
continue
else:
if got is not None:
fails.append(f"{name}: refused at parse with {got}, want a decode-layer {code}")
continue
if CAP_MAGIC[cap] == b"WIV1" and fields["keyframe"] == 0:
try:
decode_wiv1_k0(fields)
fails.append(f"{name}: decoded, want {code}")
continue
except Refused as e:
if e.code != code:
fails.append(f"{name}: decode refused with {e.code}, want {code}")
continue
else:
decode_layer.append(name)
checked += 1
print(f"ok {name}: {code}" + ("" if layer == "parse" or name not in decode_layer else " (parses; the code is the decode's)"))
# wiv1_end_state codes three symbols from the state 2^31 + 1, as the words [1, 4]; its
# canonical twin, the words [0, 4], decodes to one grey pixel.
twin = bytearray(rd("int-payload-conformance/refusals/wiv1_end_state.bin"))
twin[-8:-4] = struct.pack("<I", 0)
try:
if decode_wiv1_k0(parse("wai.video.int_motion", bytes(twin))) != [bytes([128, 128, 128])]:
fails.append("wiv1_end_state's canonical twin does not decode to one grey pixel")
else:
print("ok wiv1_end_state's canonical twin decodes to one grey pixel")
except Refused as e:
fails.append(f"wiv1_end_state's canonical twin is refused: {e}")
print(f"vectors: {decoded} decoded independently; parsed and converted only: {', '.join(parsed_only)}")
print(f"refusal fixtures checked: {checked}; decode-layer codes left to wai-rs: {', '.join(decode_layer)}")
for x in fails:
print("FAIL", x)
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else os.path.dirname(os.path.abspath(__file__))))

BIN
wai/int-payload-conformance/refusals/wia1_afrac_model.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_afrac_zero.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_cdf_malformed.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_exhausted.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_latent_too_long.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_median_range.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_order_table_before_length.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_sr_zero.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_t_out.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_too_large.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_trailing.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_under_splat.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wia1_unread_word.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_exhausted.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_geometry.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_model_mismatch.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_order_caps_before_geometry.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_order_exhausted_before_value.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_reserved.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_stream_length.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_too_large.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_trailing.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_under_video.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_unread_word.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wih1_zero_dim.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_afrac_model.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_cdf_len.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_exhausted.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_geometry.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_n_gaussians.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_range_nan.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_range_order.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_trailing.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wis1_unread_word.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_cdf_malformed.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_end_state.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_exhausted.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2133_fe.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2136_96.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2136_9b.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2137_8c.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2138_be.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2139_03.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_geometry_flip_2146_4a.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_keyframe_mismatch.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_mv_field.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_mv_precision.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_nested_length.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_nested_version.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_one_frame_block.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_one_frame_end_state.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_one_frame_mv_precision.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_q_zero.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_reserved.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_stream_length.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_too_large.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_too_many_frames.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_trailing.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_truncated.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_under_audio.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_unread_word.bin Normal file
View file

Binary file not shown.

BIN
wai/int-payload-conformance/refusals/wiv1_version.bin Normal file
View file

Binary file not shown.

1
wai/int-payload-conformance/sets/entropy_median_max.json Normal file
View file

@ -0,0 +1 @@
{"afrac":12,"frac":12,"scale_edges":[4096,8192,16384,32768],"cdfs":[[0,1,2,3,4,6,296,3834,19692,45837,61695,65233,65523,65532,65533,65534,65535,65536],[0,4,33,178,752,2521,6765,14695,26232,39306,50843,58773,63017,64786,65360,65505,65534,65536],[0,927,2410,4636,7775,11933,17108,23158,29803,36659,43304,49354,54529,58687,61826,64052,65535,65536],[0,2905,6173,9790,13731,17960,22426,27070,31824,36616,41370,46014,50480,54709,58650,62267,65535,65536]],"cdf_lengths":[18,18,18,18],"offsets":[-8,-8,-8,-8],"z_cdfs":[[0,4,33,178,752,2521,6765,14695,26232,39306,50843,58773,63017,64786,65360,65505,65534,65536],[0,4,33,178,752,2521,6765,14695,26232,39306,50843,58773,63017,64786,65360,65505,65534,65536]],"z_cdf_lengths":[18,18],"z_offsets":[1,-8],"z_median_q":[9223372036854775807,0]}

806
wai/int-payload-conformance/vectors.json Normal file
View file

@ -0,0 +1,806 @@
{
"format": "wai-int-payload-vectors/1",
"about": "Binary payload vectors of the WAI-native integer codecs and their refusal fixtures (extensions/integer-payloads.md §12). Paths are relative to wai/. Written by wai_int_payload_vectors gen.",
"vectors": [
{
"name": "int_video",
"capability": "wai.video.int_motion",
"payload": "wai-rs/tests/vectors/int_video/int_video_bitstream.wiv1",
"payload_sha256": "3bbf0a864d9528e73aff036d1a636893f839d3f6c5f31dde7824a59e6c116565",
"converted_from": {
"json": "wai-rs/tests/vectors/int_video/int_video_bitstream.json"
},
"set": {},
"sidecar": "wai-rs/tests/vectors/int_video/sha256.json"
},
{
"name": "video_real",
"capability": "wai.video.int_motion",
"payload": "wai-rs/tests/vectors/int_video/video_real_bitstream.wiv1",
"payload_sha256": "efce5ada256cb3443a673ee0c10aa4e0a8ce4a372102cd918627e5eec746880f",
"converted_from": {
"json": "wai-rs/tests/vectors/int_video/video_real_bitstream.json"
},
"set": {},
"sidecar": "wai-rs/tests/vectors/int_video/sha256.json"
},
{
"name": "tiny",
"capability": "wai.neural.int_hyper",
"payload": "wai-rs/tests/vectors/int_hyper/tiny_bitstream.wih1",
"payload_sha256": "6d8a195a5a138452292b55d70a575d1fbb4372fd8614b3b821b676e5090c3c17",
"converted_from": {
"json": "wai-rs/tests/vectors/int_hyper/tiny_bitstream.json"
},
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
},
"sidecar": "wai-rs/tests/vectors/int_hyper/sha256.json"
},
{
"name": "video_nk",
"capability": "wai.video.int_hyper",
"payload": "wai-rs/tests/vectors/int_video/video_nk_bitstream.wiv1",
"payload_sha256": "d2c62da643328bfbdf9eab129283322e9077a491c5ddd269464577f51a958bc8",
"converted_from": {
"json": "wai-rs/tests/vectors/int_video/video_nk_bitstream.json",
"keyframe": "wai-rs/tests/vectors/int_hyper/tiny_bitstream.wih1"
},
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
},
"sidecar": "wai-rs/tests/vectors/int_video/sha256.json"
},
{
"name": "audio",
"capability": "wai.audio.int_codec",
"payload": "wai-rs/tests/vectors/int_audio/audio_bitstream.wia1",
"payload_sha256": "b45e727525f06ac1ac5a5a7121886c85a7c33613451acbb482c20e37f3c79055",
"converted_from": {
"json": "wai-rs/tests/vectors/int_audio/audio_bitstream.json",
"sample_rate": "wai-rs/tests/vectors/int_audio/audio.json"
},
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
},
"sidecar": "wai-rs/tests/vectors/int_audio/sha256.json"
},
{
"name": "splat",
"capability": "wai.splat.int_codec",
"payload": "wai-rs/tests/vectors/int_splat/splat.wis1",
"payload_sha256": "473ea0223840758b26ff7d540d861e3b2c1565dbc2bec68b0235b3f50f19d422",
"converted_from": {
"json": "wai-rs/tests/vectors/int_splat/splat.json"
},
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
},
"sidecar": "wai-rs/tests/vectors/int_splat/sha256.json"
}
],
"refusals": [
{
"name": "wiv1_json_form",
"capability": "wai.video.int_motion",
"payload": "wai-rs/tests/vectors/int_video/int_video_bitstream.json",
"code": "bad_magic",
"layer": "parse",
"set": {}
},
{
"name": "wiv1_under_audio",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wiv1_under_audio.bin",
"code": "format_capability_mismatch",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wiv1_keyframe_mismatch",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_keyframe_mismatch.bin",
"code": "keyframe_capability_mismatch",
"layer": "parse",
"set": {}
},
{
"name": "wiv1_version",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_version.bin",
"code": "version",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_reserved",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_reserved.bin",
"code": "reserved_nonzero",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_mv_precision",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_mv_precision.bin",
"code": "field_range",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_q_zero",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_q_zero.bin",
"code": "field_range",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_too_large",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_too_large.bin",
"code": "too_large",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_too_many_frames",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_too_many_frames.bin",
"code": "too_large",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_mv_field",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_mv_field.bin",
"code": "too_large",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_cdf_malformed",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_cdf_malformed.bin",
"code": "cdf_malformed",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_stream_length",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_stream_length.bin",
"code": "stream_length",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_truncated",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_truncated.bin",
"code": "stream_length",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_nested_length",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_nested_length.bin",
"code": "nested_length",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_nested_version",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_nested_version.bin",
"code": "version",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_trailing",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_trailing.bin",
"code": "trailing",
"layer": "parse",
"set": {}
},
{
"name": "wiv1_exhausted",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_exhausted.bin",
"code": "stream_exhausted",
"layer": "decode",
"set": {}
},
{
"name": "wiv1_unread_word",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_unread_word.bin",
"code": "stream_noncanonical",
"layer": "decode",
"set": {}
},
{
"name": "wiv1_end_state",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_end_state.bin",
"code": "stream_noncanonical",
"layer": "decode",
"set": {}
},
{
"name": "wiv1_one_frame_block",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_one_frame_block.bin",
"code": "field_range",
"layer": "parse",
"set": {}
},
{
"name": "wiv1_one_frame_mv_precision",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wiv1_one_frame_mv_precision.bin",
"code": "field_range",
"layer": "parse",
"set": {}
},
{
"name": "wiv1_one_frame_end_state",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_one_frame_end_state.bin",
"code": "stream_noncanonical",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2146_4a",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2146_4a.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2138_be",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2138_be.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2136_96",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2136_96.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2139_03",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2139_03.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2133_fe",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2133_fe.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2136_9b",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2136_9b.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wiv1_geometry_flip_2137_8c",
"capability": "wai.video.int_hyper",
"payload": "int-payload-conformance/refusals/wiv1_geometry_flip_2137_8c.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_json_form",
"capability": "wai.neural.int_hyper",
"payload": "wai-rs/tests/vectors/int_hyper/tiny_bitstream.json",
"code": "bad_magic",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_under_video",
"capability": "wai.video.int_motion",
"payload": "int-payload-conformance/refusals/wih1_under_video.bin",
"code": "format_capability_mismatch",
"layer": "parse",
"set": {}
},
{
"name": "wih1_reserved",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_reserved.bin",
"code": "reserved_nonzero",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_zero_dim",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_zero_dim.bin",
"code": "field_range",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_too_large",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_too_large.bin",
"code": "too_large",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_stream_length",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_stream_length.bin",
"code": "stream_length",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_trailing",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_trailing.bin",
"code": "trailing",
"layer": "parse",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_model_mismatch",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_model_mismatch.bin",
"code": "model_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_geometry",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_geometry.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_exhausted",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_exhausted.bin",
"code": "stream_exhausted",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_order_caps_before_geometry",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_order_caps_before_geometry.bin",
"code": "too_large",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_order_exhausted_before_value",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_order_exhausted_before_value.bin",
"code": "stream_exhausted",
"layer": "decode",
"set": {
"entropy.json": "int-payload-conformance/sets/entropy_median_max.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wih1_unread_word",
"capability": "wai.neural.int_hyper",
"payload": "int-payload-conformance/refusals/wih1_unread_word.bin",
"code": "stream_noncanonical",
"layer": "decode",
"set": {
"entropy.json": "wai-rs/tests/vectors/int_hyper/tiny_entropy.json",
"g_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_g_s.bin",
"h_s.whm": "wai-rs/tests/vectors/int_hyper/tiny_h_s.bin"
}
},
{
"name": "wia1_json_form",
"capability": "wai.audio.int_codec",
"payload": "wai-rs/tests/vectors/int_audio/audio_bitstream.json",
"code": "bad_magic",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_under_splat",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wia1_under_splat.bin",
"code": "format_capability_mismatch",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wia1_afrac_zero",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_afrac_zero.bin",
"code": "field_range",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_sr_zero",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_sr_zero.bin",
"code": "field_range",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_median_range",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_median_range.bin",
"code": "field_range",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_cdf_malformed",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_cdf_malformed.bin",
"code": "cdf_malformed",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_order_table_before_length",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_order_table_before_length.bin",
"code": "cdf_malformed",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_too_large",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_too_large.bin",
"code": "too_large",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_trailing",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_trailing.bin",
"code": "trailing",
"layer": "parse",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_afrac_model",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_afrac_model.bin",
"code": "model_mismatch",
"layer": "decode",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_t_out",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_t_out.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_latent_too_long",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_latent_too_long.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_exhausted",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_exhausted.bin",
"code": "stream_exhausted",
"layer": "decode",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wia1_unread_word",
"capability": "wai.audio.int_codec",
"payload": "int-payload-conformance/refusals/wia1_unread_word.bin",
"code": "stream_noncanonical",
"layer": "decode",
"set": {
"model.wam": "wai-rs/tests/vectors/int_audio/audio_model.bin"
}
},
{
"name": "wis1_json_form",
"capability": "wai.splat.int_codec",
"payload": "wai-rs/tests/vectors/int_splat/splat.json",
"code": "bad_magic",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_range_order",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_range_order.bin",
"code": "field_range",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_range_nan",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_range_nan.bin",
"code": "field_range",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_n_gaussians",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_n_gaussians.bin",
"code": "field_range",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_cdf_len",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_cdf_len.bin",
"code": "field_range",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_trailing",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_trailing.bin",
"code": "trailing",
"layer": "parse",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_geometry",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_geometry.bin",
"code": "geometry_mismatch",
"layer": "decode",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_afrac_model",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_afrac_model.bin",
"code": "model_mismatch",
"layer": "decode",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_exhausted",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_exhausted.bin",
"code": "stream_exhausted",
"layer": "decode",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
},
{
"name": "wis1_unread_word",
"capability": "wai.splat.int_codec",
"payload": "int-payload-conformance/refusals/wis1_unread_word.bin",
"code": "stream_noncanonical",
"layer": "decode",
"set": {
"g_s.whm": "wai-rs/tests/vectors/int_splat/splat_gs.bin"
}
}
]
}

30
wai/wai-rs/Cargo.toml
View file

@ -142,7 +142,7 @@ world_int = ["neural_int"]
# Integer-exact LEARNED Gaussian-splat ATTRIBUTE codec ("the int_hyper of 3D"):
# learned transform over the [14,S,S] attribute image → byte-exact attribute tensor
# (splat-equivalence, a portable cloud hash). Reuses int_hyper_synth (neural_int).
splat_int = ["neural_int"]
splat_int = ["neural_int", "serde_json/float_roundtrip"]
neural = ["neural_int", "dep:ort", "dep:ndarray", "dep:zstd-safe"]
# The perceptual / generative-semantic path: CLIP-similarity conformance,
# regenerate-at-sink. Reuses the ONNX runtime for the reference CLIP embedder;
@ -176,13 +176,16 @@ provenance = ["world"]
# content-address and chain it, and sign it. Conformance is reason-set equality with
# the reference verifier and the portable C kernel over oer-conformance/.
oer = ["dep:blake3", "dep:ed25519-dalek", "dep:num-bigint", "serde_json/float_roundtrip"]
# jwp-receipts §2.1's object content hash (src/jwp_hash.rs), on its own so any
# receipt family that names an object can compute it with BLAKE3 alone.
jwp_hash = ["dep:blake3"]
# Typed receipt classes on the JWP chain (extensions/jwp-receipts.md §2.4.3–§2.4.7,
# §2.6, §3.1–§3.2): `claim` receipts of class energy / lawful-basis / log-inclusion,
# the soft-binding class, and recovery under the ancestor rule. Needs `provenance`
# for the coverage vocabulary and `oer` for the evidential grades and the OER receipt
# an energy claim may name, which it reads and verifies through `crate::oer`. Its one
# direct dependency, curve25519-dalek, is already in the graph under ed25519-dalek.
receipt_claims = ["provenance", "oer", "dep:curve25519-dalek"]
receipt_claims = ["provenance", "oer", "jwp_hash", "dep:curve25519-dalek"]
# Key discovery and signing authority (extensions/jwp-receipts.md §7): the origin key
# log a `did:web` signer_id names, anchoring, the confirmation reported beside a receipt's
# or claim's verdict, and the authorisation and revocation of decoder and reconstruct
@ -467,6 +470,14 @@ name = "wai_int_synth_vectors"
path = "src/bin/wai_int_synth_vectors.rs"
required-features = ["neural_int"]
# The binary payload vectors of the integer codecs and their refusal fixtures
# (extensions/integer-payloads.md §12): `gen` converts the committed JSON, `verify`
# requires byte-identical regeneration, the sidecars' SHA-256 and every fixture's code.
[[bin]]
name = "wai_int_payload_vectors"
path = "src/bin/wai_int_payload_vectors.rs"
required-features = ["video_int", "audio_int", "splat_int"]
[[bin]]
name = "wai_film_vectors"
path = "src/bin/wai_film_vectors.rs"
@ -627,6 +638,21 @@ name = "int_mlicpp_preflight"
path = "tests/int_mlicpp_preflight.rs"
required-features = ["neural_int"]
[[test]]
name = "int_payload"
path = "tests/int_payload.rs"
required-features = ["video_int", "audio_int", "splat_int"]
[[test]]
name = "int_payload_preflight"
path = "tests/int_payload_preflight.rs"
required-features = ["video_int", "audio_int", "splat_int"]
[[test]]
name = "int_payload_cost"
path = "tests/int_payload_cost.rs"
required-features = ["video_int", "audio_int", "splat_int"]
[[test]]
name = "generate"
path = "tests/generate.rs"

1
wai/wai-rs/cbindgen.toml
View file

@ -41,6 +41,7 @@ prefix = ""
[export.rename]
"WaiBuffer" = "wai_buffer_t"
"WaiDecoded" = "wai_decoded_t"
"WaiIntDeclared" = "wai_int_declared_t"
[fn]
prefix = ""

60
wai/wai-rs/include/wai.h
View file

@ -345,6 +345,24 @@ typedef struct wai_buffer_t {
uintptr_t cap;
} wai_buffer_t;
/**
* What decoding an integer payload will cost (`wai_int_payload_declared`,
* `extensions/integer-payloads.md` §7, *Memory and work*): the decoded output's
* bytes (`output_known` 1 where they are filled), the symbols its streams code, and,
* where `cost_known` is 1, the most bytes the reference decode holds at once beyond
* the payload and the parsed set (`working_bytes`) and the networks'
* multiply-accumulates, `macs_hi·2^64 + macs_lo`.
*/
typedef struct wai_int_declared_t {
uint64_t output_bytes;
uint8_t output_known;
uint64_t symbols;
uint64_t working_bytes;
uint64_t macs_lo;
uint64_t macs_hi;
uint8_t cost_known;
} wai_int_declared_t;
/**
* What `wai_neural_decode_ex` decoded.
*
@ -397,13 +415,45 @@ extern "C" {
*/
const char *wai_last_error(void);
/**
* Read what decoding the payload of an integer capability will cost before decoding
* it, so a sink can refuse a payload past its own size policy first
* (`extensions/integer-payloads.md` §7, *Memory and work*). `capability` is the
* envelope's capability (or the `WAI2` component's). The payload is parsed as a sink
* parses it: one that is not the format SPEC §5 pins `capability` to is
* `WAI_ERR_INVALID_ARG`, with its code in `wai_last_error`, and so is a capability
* with no pinned integer format.
*
* With `set_path` null, only the header is read: a `wai.neural.int_hyper` image's
* size is its synthesis network's, so `output_known` is 0 for it, and `cost_known` is
* 1 only for a `WIV1` with keyframe 0, which runs no network. With `set_path` naming
* the capability's parameter set — a directory holding its members by the names §9
* gives, or the file of a one-file set — every figure is filled
* (`int_payload::declared_cost`), and the set is checked against the payload as a
* decode checks it first (`set_incomplete`, `model_mismatch`, `geometry_mismatch`,
* …). The set is read as given, unverified: a pin is verified when the envelope is
* decoded. A build without `neural_int` reads no set.
*
* # Safety
* `capability` NUL-terminated; `payload` valid for `payload_len` bytes; `set_path`
* null or NUL-terminated UTF-8; `out` writable.
*/
int wai_int_payload_declared(const char *capability,
const uint8_t *payload,
uintptr_t payload_len,
const char *set_path,
struct wai_int_declared_t *out);
/**
* Builds a complete WAI file from a codec-produced payload + manifest
* fields. `manifest_json` is a UTF-8 JSON object overriding/supplying
* any fields; `payload` is the bytes from one of the codec encoders.
* Refuses (`WAI_ERR_INVALID_ARG`) a manifest that names a candidate capability
* as its capability or fallback, which an encoder MUST NOT emit (SPEC §5,
* *Candidate capabilities*): it emits through `Wai::to_emit_bytes`.
* *Candidate capabilities*), and a payload that is not the format SPEC §5 pins its
* integer capability or fallback to (*Payloads of the WAI-native integer codecs*):
* it emits through `Wai::to_emit_bytes`.
*/
int wai_envelope_pack(const char *manifest_json,
@ -426,7 +476,8 @@ int wai_envelope_pack(const char *manifest_json,
* - The i-th payload pairs with `manifest.renditions[i]`.
*
* Refuses (`WAI_ERR_INVALID_ARG`) a table with a component, in either role,
* whose capability is a candidate (SPEC §5, *Candidate capabilities*): it emits
* whose capability is a candidate (SPEC §5, *Candidate capabilities*), or whose
* payload is not the format SPEC §5 pins its integer capability to: it emits
* through `WaiMulti::to_emit_bytes`.
*
* # Safety
@ -680,7 +731,10 @@ int wai_neural_decode_video(const uint8_t *envelope,
* array of paths. The first path is the registered artifact; the others are
* further sets a pin may resolve against (a set and its successor). Every pin is
* verified first (SPEC §3.1); a `WAI2` rendition is picked by SPEC §7.1. With
* `flags & WAI_DECODE_STRICT`, SPEC §7's replicate rule applies too.
* `flags & WAI_DECODE_STRICT`, SPEC §7's replicate rule applies too. A capability
* whose output has no `wai_decoded_t` kind (integer video, PCM, splat attributes) is
* not advertised, whatever `registry_json` says: dispatch moves on past it before
* decoding, and when nothing else decodes the error names it.
*
* `*out` is written on every call with a non-null `out`, before anything else is
* read: on failure it holds `kind` 0 and an empty `data`, so the caller may free

23
wai/wai-rs/src/bin/wai.rs
View file

@ -364,6 +364,15 @@ fn inspect(path: &str) -> Result<(), String> {
if let Err(e) = w.check_candidates() {
println!(" candidates: \u{2717} {e}");
}
let encrypted = !w.manifest.security.as_ref().is_none_or(wai::container::Security::is_plaintext);
if [Some(cap), fb].into_iter().flatten().any(|c| wai::codecs::pinned_int_payload(c).is_some()) && encrypted {
println!(" integer payload: encrypted; a sink checks it after decrypting (SPEC \u{a7}6.4)");
} else if [Some(cap), fb].into_iter().flatten().any(|c| wai::codecs::pinned_int_payload(c).is_some()) {
match w.check_int_payload_framing() {
Ok(()) => println!(" integer payload: \u{2713} parses as the format SPEC \u{a7}5 pins (every parse rule; the decode checks the rest)"),
Err(e) => println!(" integer payload: \u{2717} {e}; a sink refuses it at decode"),
}
}
} else {
// WAI2 inspection
let m = wai::container::WaiMulti::from_bytes(&bytes).map_err(|e| e.to_string())?;
@ -422,6 +431,9 @@ fn inspect(path: &str) -> Result<(), String> {
if let Err(e) = m.check_candidates() {
println!(" candidates: \u{2717} {e}");
}
if let Err((i, e)) = m.check_int_payload_framing() {
println!(" integer payload: \u{2717} [{i}] {e}; a sink refuses it at decode");
}
}
println!();
@ -486,7 +498,9 @@ fn license_str(cap: &str) -> &'static str {
/// capability is refused as a `WAI1` capability or fallback, and a `WAI2` table
/// that breaks the binding rules is refused. A candidate capability (SPEC §5,
/// *Candidate capabilities*) is refused in every place — capability, fallback or
/// companion — and nothing is written.
/// companion — and nothing is written. So is a payload that is not the format SPEC §5
/// pins its integer capability to (*Payloads of the WAI-native integer codecs*): the
/// JSON a tool writes is converted first, never wrapped.
fn wrap(cap: &str, input: &str, out: &str, fallback: Option<&str>,
intent: Option<&str>, companions: &[(String, String)]) -> Result<(), String> {
if !wai::codecs::is_known_capability(cap) {
@ -526,9 +540,11 @@ fn wrap(cap: &str, input: &str, out: &str, fallback: Option<&str>,
};
let manifest = Manifest::new(media_for(cap), intent, requirement, Conditioning::new("bytes"));
let envelope = Wai::new(manifest, payload);
// The reference tool does not write an envelope that names a candidate, or
// one that breaks the role rules.
// The reference tool does not write an envelope that names a candidate, one that
// breaks the role rules, or one whose payload is not the format SPEC §5 pins its
// integer capability to (a tool's JSON output, say).
envelope.check_candidates().map_err(|e| e.to_string())?;
envelope.check_int_payload_framing().map_err(|e| format!("not writing {out}: {e}"))?;
envelope.check_component_roles().map_err(|e| match e {
wai::container::RoleError::CompanionOnlyAsCapability { .. } => format!(
"{e} (SPEC \u{a7}5): `wai wrap <base-cap> <base-payload> --companion {cap} <payload>`"),
@ -573,6 +589,7 @@ fn wrap_with_companions(cap: &str, payload: Vec<u8>, out: &str, fallback: Option
let envelope = wai::container::WaiMulti::new(
media_for(cap), intent, serde_json::Value::Null, components);
envelope.check_candidates().map_err(|e| e.to_string())?;
envelope.check_int_payload_framing().map_err(|(i, e)| format!("not writing {out}: component {i}: {e}"))?;
envelope.check_component_roles()
.map_err(|e| format!("{e} (SPEC \u{a7}5, \u{a7}7.1)"))?;
let n = envelope.write(out).map_err(|e| e.to_string())?;

532
wai/wai-rs/src/bin/wai_int_payload_vectors.rs Normal file
View file

@ -0,0 +1,532 @@
//! `wai_int_payload_vectors` — writes and verifies the binary payload vectors of the
//! WAI-native integer codecs (`extensions/integer-payloads.md` §12).
//!
//! wai_int_payload_vectors gen <int-payload-conformance dir>
//! wai_int_payload_vectors verify <int-payload-conformance dir>
//!
//! `gen` converts each committed JSON bitstream under `wai-rs/tests/vectors/` to its
//! pinned payload, beside the JSON (`*.wiv1`, `*.wih1`, `*.wia1`, `*.wis1`), derives the
//! refusal fixtures from those payloads (`refusals/*.bin`), and writes the index
//! `vectors.json`. Every file it writes is a function of the committed JSON alone, so
//! a second run writes the same bytes.
//!
//! `verify` regenerates every file in memory and requires it to equal the committed
//! one byte for byte; decodes each payload through `int_payload::decode_int` against
//! the parameter set the index names, and requires the output's SHA-256 to be the one
//! the vector's `sha256.json` sidecar states; and requires each refusal fixture to be
//! refused with the code the index states. Exit status 1 on any failure.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::process::ExitCode;
use serde_json::{json, Value};
use sha2::{Digest, Sha256};
use wai::int_payload::{declared_cost, decode_int, wire, IntDecoded, IntParams};
use wai::pin::HeldBytes;
fn hex(b: &[u8]) -> String {
b.iter().map(|x| format!("{x:02x}")).collect()
}
fn sha256(b: &[u8]) -> String {
hex(&Sha256::digest(b))
}
/// The tree below `wai/`, read from and written to by path relative to it.
struct Tree {
wai: PathBuf,
out: BTreeMap<String, Vec<u8>>,
}
impl Tree {
fn read(&self, rel: &str) -> Vec<u8> {
if let Some(b) = self.out.get(rel) {
return b.clone();
}
let p = self.wai.join(rel);
std::fs::read(&p).unwrap_or_else(|e| panic!("read {}: {e}", p.display()))
}
fn json(&self, rel: &str) -> Value {
serde_json::from_slice(&self.read(rel)).unwrap_or_else(|e| panic!("parse {rel}: {e}"))
}
fn put(&mut self, rel: &str, bytes: Vec<u8>) {
self.out.insert(rel.to_string(), bytes);
}
}
const V: &str = "wai-rs/tests/vectors";
const R: &str = "int-payload-conformance/refusals";
/// One decodable vector: its name, capability, payload path, the JSON it is converted
/// from, the set members it decodes against (member name → path) and its sidecar entry.
struct Vector {
name: &'static str,
cap: &'static str,
payload: String,
from: Value,
set: Vec<(&'static str, String)>,
sidecar: String,
}
fn int_hyper_set() -> Vec<(&'static str, String)> {
vec![
("entropy.json", format!("{V}/int_hyper/tiny_entropy.json")),
("g_s.whm", format!("{V}/int_hyper/tiny_g_s.bin")),
("h_s.whm", format!("{V}/int_hyper/tiny_h_s.bin")),
]
}
/// Convert every committed JSON bitstream to its payload.
fn vectors(t: &mut Tree) -> Vec<Vector> {
use wai::int_payload::{wia1_from_json, wih1_from_json, wis1_from_json, wiv1_from_json};
let mut out = Vec::new();
for (name, json_rel, wiv1) in [
("int_video", "int_video/int_video_bitstream.json", "int_video/int_video_bitstream.wiv1"),
("video_real", "int_video/video_real_bitstream.json", "int_video/video_real_bitstream.wiv1"),
] {
let p = wiv1_from_json(&t.json(&format!("{V}/{json_rel}")), None).unwrap_or_else(|e| panic!("{name}: {e}"));
t.put(&format!("{V}/{wiv1}"), p);
out.push(Vector {
name,
cap: wai::codecs::CAP_VIDEO_INT_MOTION,
payload: format!("{V}/{wiv1}"),
from: json!({ "json": format!("{V}/{json_rel}") }),
set: vec![],
sidecar: format!("{V}/int_video/sha256.json"),
});
}
let wih1 = wih1_from_json(&t.json(&format!("{V}/int_hyper/tiny_bitstream.json"))).expect("tiny WIH1");
t.put(&format!("{V}/int_hyper/tiny_bitstream.wih1"), wih1.clone());
out.push(Vector {
name: "tiny",
cap: wai::codecs::CAP_NEURAL_INT_HYPER,
payload: format!("{V}/int_hyper/tiny_bitstream.wih1"),
from: json!({ "json": format!("{V}/int_hyper/tiny_bitstream.json") }),
set: int_hyper_set(),
sidecar: format!("{V}/int_hyper/sha256.json"),
});
let nk = wiv1_from_json(&t.json(&format!("{V}/int_video/video_nk_bitstream.json")), Some(&wih1)).expect("video_nk WIV1");
t.put(&format!("{V}/int_video/video_nk_bitstream.wiv1"), nk);
out.push(Vector {
name: "video_nk",
cap: wai::codecs::CAP_VIDEO_INT_HYPER,
payload: format!("{V}/int_video/video_nk_bitstream.wiv1"),
from: json!({ "json": format!("{V}/int_video/video_nk_bitstream.json"), "keyframe": format!("{V}/int_hyper/tiny_bitstream.wih1") }),
set: int_hyper_set(),
sidecar: format!("{V}/int_video/sha256.json"),
});
let sr = t.json(&format!("{V}/int_audio/audio.json"))["sample_rate"].as_u64().expect("sample_rate") as u32;
let wia1 = wia1_from_json(&t.json(&format!("{V}/int_audio/audio_bitstream.json")), sr).expect("audio WIA1");
t.put(&format!("{V}/int_audio/audio_bitstream.wia1"), wia1);
out.push(Vector {
name: "audio",
cap: wai::codecs::CAP_AUDIO_INT_CODEC,
payload: format!("{V}/int_audio/audio_bitstream.wia1"),
from: json!({ "json": format!("{V}/int_audio/audio_bitstream.json"), "sample_rate": format!("{V}/int_audio/audio.json") }),
set: vec![("model.wam", format!("{V}/int_audio/audio_model.bin"))],
sidecar: format!("{V}/int_audio/sha256.json"),
});
let wis1 = wis1_from_json(&t.json(&format!("{V}/int_splat/splat.json"))).expect("splat WIS1");
t.put(&format!("{V}/int_splat/splat.wis1"), wis1);
out.push(Vector {
name: "splat",
cap: wai::codecs::CAP_SPLAT_INT_CODEC,
payload: format!("{V}/int_splat/splat.wis1"),
from: json!({ "json": format!("{V}/int_splat/splat.json") }),
set: vec![("g_s.whm", format!("{V}/int_splat/splat_gs.bin"))],
sidecar: format!("{V}/int_splat/sha256.json"),
});
out
}
/// One refusal fixture: its name, capability, payload path, the code it is refused
/// with, whether parsing alone refuses it (`parse`) or the decode does (`decode`), and
/// the set its decode reads.
struct Fixture {
name: &'static str,
cap: &'static str,
payload: String,
code: &'static str,
layer: &'static str,
set: Vec<(&'static str, String)>,
}
fn set_u32(b: &mut [u8], off: usize, v: u32) {
b[off..off + 4].copy_from_slice(&v.to_le_bytes());
}
fn get_u32(b: &[u8], off: usize) -> u32 {
u32::from_le_bytes(b[off..off + 4].try_into().unwrap())
}
/// The refusal fixtures, each the committed payload changed in one place.
fn fixtures(t: &mut Tree) -> Vec<Fixture> {
use wai::codecs::{CAP_AUDIO_INT_CODEC as AUD, CAP_NEURAL_INT_HYPER as HYP, CAP_SPLAT_INT_CODEC as SPL,
CAP_VIDEO_INT_HYPER as VNK, CAP_VIDEO_INT_MOTION as MOT};
let iv = t.read(&format!("{V}/int_video/int_video_bitstream.wiv1"));
let nk = t.read(&format!("{V}/int_video/video_nk_bitstream.wiv1"));
let ih = t.read(&format!("{V}/int_hyper/tiny_bitstream.wih1"));
let au = t.read(&format!("{V}/int_audio/audio_bitstream.wia1"));
let sp = t.read(&format!("{V}/int_splat/splat.wis1"));
let mut out = Vec::new();
let mut fx = |t: &mut Tree, name: &'static str, cap: &'static str, bytes: Option<Vec<u8>>, payload: Option<String>, code: &'static str, layer: &'static str| {
let payload = match bytes {
Some(b) => {
let rel = format!("{R}/{name}.bin");
t.put(&rel, b);
rel
}
None => payload.expect("a fixture names its payload"),
};
let set = match cap {
c if c == HYP || c == VNK => int_hyper_set(),
c if c == AUD => vec![("model.wam", format!("{V}/int_audio/audio_model.bin"))],
c if c == SPL => vec![("g_s.whm", format!("{V}/int_splat/splat_gs.bin"))],
_ => vec![],
};
out.push(Fixture { name, cap, payload, code, layer, set });
};
let edit = |base: &[u8], f: &dyn Fn(&mut Vec<u8>)| {
let mut b = base.to_vec();
f(&mut b);
Some(b)
};
// ── WIV1 ── (keyframe 1 is video_nk: 16×16, the nested tiny WIH1)
let parsed = wire::parse_wiv1(&nk, Some(1)).expect("video_nk parses");
let nk_l = at(&nk, parsed.rans) - 4; // offset of L
let nk_k = at(&nk, parsed.wih1.expect("keyframe 1")) - 4; // offset of K
fx(t, "wiv1_json_form", MOT, None, Some(format!("{V}/int_video/int_video_bitstream.json")), "bad_magic", "parse");
fx(t, "wiv1_under_audio", AUD, Some(nk.clone()), None, "format_capability_mismatch", "parse");
fx(t, "wiv1_keyframe_mismatch", MOT, Some(nk.clone()), None, "keyframe_capability_mismatch", "parse");
fx(t, "wiv1_version", VNK, edit(&nk, &|b| b[4] = 2), None, "version", "parse");
fx(t, "wiv1_reserved", VNK, edit(&nk, &|b| b[7] = 1), None, "reserved_nonzero", "parse");
fx(t, "wiv1_mv_precision", VNK, edit(&nk, &|b| b[5] = 3), None, "field_range", "parse");
fx(t, "wiv1_q_zero", VNK, edit(&nk, &|b| set_u32(b, 24, 0)), None, "field_range", "parse");
fx(t, "wiv1_too_large", VNK, edit(&nk, &|b| { set_u32(b, 12, 8192); set_u32(b, 16, 8192) }), None, "too_large", "parse");
// 2^20 + 1 frames of 16 × 16: within the clip's 2^30 bytes, past the frame count.
fx(t, "wiv1_too_many_frames", VNK, edit(&nk, &|b| set_u32(b, 8, (1 << 20) + 1)), None, "too_large", "parse");
// Block 1 over 131074 frames: 2·256·131073 motion-vector elements, past 2^26.
fx(t, "wiv1_mv_field", VNK, edit(&nk, &|b| { set_u32(b, 8, 131_074); set_u32(b, 20, 1) }), None, "too_large", "parse");
fx(t, "wiv1_cdf_malformed", VNK, edit(&nk, &|b| { let v = get_u32(b, 40); set_u32(b, 44, v) }), None, "cdf_malformed", "parse");
fx(t, "wiv1_stream_length", VNK, edit(&nk, &|b| { let l = get_u32(b, nk_l); set_u32(b, nk_l, l - 2) }), None, "stream_length", "parse");
fx(t, "wiv1_truncated", VNK, edit(&nk, &|b| b.truncate(nk_k + 2)), None, "stream_length", "parse");
fx(t, "wiv1_nested_length", VNK, edit(&nk, &|b| { let k = get_u32(b, nk_k); set_u32(b, nk_k, k + 4) }), None, "nested_length", "parse");
fx(t, "wiv1_nested_version", VNK, edit(&nk, &|b| b[nk_k + 4 + 4] = 2), None, "version", "parse");
fx(t, "wiv1_trailing", MOT, edit(&iv, &|b| b.push(0)), None, "trailing", "parse");
let iv_l = at(&iv, wire::parse_wiv1(&iv, Some(0)).expect("int_video parses").rans) - 4;
fx(t, "wiv1_exhausted", MOT, edit(&iv, &|b| { set_u32(b, iv_l, 8); b.truncate(iv_l + 4 + 8) }), None, "stream_exhausted", "decode");
// A word appended to the residual stream (the last section), which the decode
// never reads.
fx(t, "wiv1_unread_word", MOT, edit(&iv, &|b| { let l = get_u32(b, iv_l); set_u32(b, iv_l, l + 4); b.extend_from_slice(&[0; 4]) }), None, "stream_noncanonical", "decode");
// A one-pixel clip whose three symbols are 0 (frequency 2^15 of 2^16), encoded
// from the state 2^31 + 1: the stream is the words [1, 4], and the decode ends at
// 2^31 + 1. Encoded from 2^31, the canonical stream is [0, 4] (`grey_pixel`).
fx(t, "wiv1_end_state", MOT, Some(grey_pixel(1)), None, "stream_noncanonical", "decode");
// A one-frame clip's header fixes mv_precision 1 and block 1 (§3): the one-pixel
// clip with block 4, or half-pel, is refused at parse.
fx(t, "wiv1_one_frame_block", MOT, edit(&grey_pixel(0), &|b| set_u32(b, 20, 4)), None, "field_range", "parse");
fx(t, "wiv1_one_frame_mv_precision", MOT, edit(&grey_pixel(0), &|b| b[5] = 2), None, "field_range", "parse");
// One frame, the nested keyframe: the residual stream codes no symbol, so it must be
// the state 2^31 alone, and 2^31 + 1 is refused.
let one = |state: u32| {
let mut v = wire::parse_wiv1(&nk, Some(1)).expect("video_nk parses");
let rans: Vec<u8> = [state, 0].iter().flat_map(|w| w.to_le_bytes()).collect();
(v.n_frames, v.mv_precision, v.block) = (1, 1, 1);
v.mvs.clear();
wire::write_wiv1(&wire::Wiv1 { rans: &rans, ..v })
};
fx(t, "wiv1_one_frame_end_state", VNK, Some(one((1 << 31) + 1)), None, "stream_noncanonical", "decode");
let geo = {
let mut v = wire::parse_wiv1(&nk, Some(1)).expect("video_nk parses");
(v.h, v.w) = (12, 12); // the same 2 × 2 block grid; the keyframe decodes 16 × 16
wire::write_wiv1(&v)
};
fx(t, "wiv1_geometry", VNK, Some(geo.clone()), None, "geometry_mismatch", "decode");
// The keyframe's size is checked against the clip's h × w before any symbol of the
// keyframe is decoded (§7): with one byte of the keyframe's last stream words
// changed, so that its stream no longer ends canonically, each is still
// geometry_mismatch. Named by the byte changed and the value XORed into it.
for (name, back, x) in [
("wiv1_geometry_flip_2146_4a", 2, 0x4a), ("wiv1_geometry_flip_2138_be", 10, 0xbe),
("wiv1_geometry_flip_2136_96", 12, 0x96), ("wiv1_geometry_flip_2139_03", 9, 0x03),
("wiv1_geometry_flip_2133_fe", 15, 0xfe), ("wiv1_geometry_flip_2136_9b", 12, 0x9b),
("wiv1_geometry_flip_2137_8c", 11, 0x8c),
] {
fx(t, name, VNK, edit(&geo, &|b| { let at = b.len() - back; b[at] ^= x }), None, "geometry_mismatch", "decode");
}
// ── WIH1 ── (the tiny conditional codec: m 4, 4×4; n_z 2, 2×2)
fx(t, "wih1_json_form", HYP, None, Some(format!("{V}/int_hyper/tiny_bitstream.json")), "bad_magic", "parse");
fx(t, "wih1_under_video", MOT, Some(ih.clone()), None, "format_capability_mismatch", "parse");
fx(t, "wih1_reserved", HYP, edit(&ih, &|b| b[6] = 1), None, "reserved_nonzero", "parse");
fx(t, "wih1_zero_dim", HYP, edit(&ih, &|b| set_u32(b, 8, 0)), None, "field_range", "parse");
fx(t, "wih1_too_large", HYP, edit(&ih, &|b| { set_u32(b, 24, 65_536); set_u32(b, 28, 65_536) }), None, "too_large", "parse");
fx(t, "wih1_stream_length", HYP, edit(&ih, &|b| set_u32(b, 32, 6)), None, "stream_length", "parse");
fx(t, "wih1_trailing", HYP, edit(&ih, &|b| b.push(0)), None, "trailing", "parse");
fx(t, "wih1_model_mismatch", HYP, edit(&ih, &|b| set_u32(b, 20, 3)), None, "model_mismatch", "decode");
fx(t, "wih1_geometry", HYP, edit(&ih, &|b| { set_u32(b, 12, 8); set_u32(b, 16, 8) }), None, "geometry_mismatch", "decode");
let ih_ly = at(&ih, wire::parse_wih1(&ih).expect("tiny parses").rans) - 4;
fx(t, "wih1_exhausted", HYP, edit(&ih, &|b| { set_u32(b, ih_ly, 8); b.truncate(ih_ly + 4 + 8) }), None, "stream_exhausted", "decode");
// The order of the decode's checks (§7): every network is planned, caps included,
// before a shape is compared, so a 2100 × 2100 latent, whose g_s would form a
// buffer past the cap, is too_large rather than h_s's geometry_mismatch; and a
// symbol that runs its stream out is refused before its value is rebuilt, so a z
// stream of 8 zero bytes, whose first symbol rebuilt against a median of i64::MAX
// would leave i64, is stream_exhausted (the set's entropy.json is
// sets/entropy_median_max.json).
fx(t, "wih1_order_caps_before_geometry", HYP, edit(&ih, &|b| { set_u32(b, 12, 2100); set_u32(b, 16, 2100) }), None, "too_large", "decode");
let short_z = {
let mut k = wire::parse_wih1(&ih).expect("tiny parses");
k.z_rans = &[0; 8];
wire::write_wih1(&k)
};
fx(t, "wih1_order_exhausted_before_value", HYP, Some(short_z), None, "stream_exhausted", "decode");
fx(t, "wih1_unread_word", HYP, edit(&ih, &|b| { let l = get_u32(b, ih_ly); set_u32(b, ih_ly, l + 4); b.extend_from_slice(&[0; 4]) }), None, "stream_noncanonical", "decode");
// ── WIA1 ── (16 channels × 512, tables of 14)
let au_n = at(&au, wire::parse_wia1(&au).expect("audio parses").rans) - 4; // offset of L
fx(t, "wia1_json_form", AUD, None, Some(format!("{V}/int_audio/audio_bitstream.json")), "bad_magic", "parse");
fx(t, "wia1_under_splat", SPL, Some(au.clone()), None, "format_capability_mismatch", "parse");
fx(t, "wia1_afrac_zero", AUD, edit(&au, &|b| b[5] = 0), None, "field_range", "parse");
fx(t, "wia1_sr_zero", AUD, edit(&au, &|b| set_u32(b, 8, 0)), None, "field_range", "parse");
fx(t, "wia1_median_range", AUD, edit(&au, &|b| b[32..40].copy_from_slice(&((1i64 << 47) + 1).to_le_bytes())), None, "field_range", "parse");
fx(t, "wia1_cdf_malformed", AUD, edit(&au, &|b| set_u32(b, 160 + 13 * 4, 65_535)), None, "cdf_malformed", "parse");
// §2's order: the tables are read one at a time, each present and then checked, so
// a malformed first table is cdf_malformed though the payload ends inside the second.
fx(t, "wia1_order_table_before_length", AUD, edit(&au, &|b| { set_u32(b, 160 + 13 * 4, 65_535); b.truncate(160 + 14 * 4 + 10) }), None, "cdf_malformed", "parse");
fx(t, "wia1_too_large", AUD, edit(&au, &|b| { set_u32(b, 12, 65_536); set_u32(b, 16, 2_048) }), None, "too_large", "parse");
fx(t, "wia1_trailing", AUD, edit(&au, &|b| b.push(0)), None, "trailing", "parse");
fx(t, "wia1_afrac_model", AUD, edit(&au, &|b| b[5] = 11), None, "model_mismatch", "decode");
fx(t, "wia1_t_out", AUD, edit(&au, &|b| { let n = get_u32(b, 20); set_u32(b, 20, n + 1) }), None, "geometry_mismatch", "decode");
// A latent twice as long as t_out needs: one sample fewer still gives t_out (§7).
fx(t, "wia1_latent_too_long", AUD, edit(&au, &|b| { let n = get_u32(b, 16); set_u32(b, 16, 2 * n) }), None, "geometry_mismatch", "decode");
fx(t, "wia1_exhausted", AUD, edit(&au, &|b| { set_u32(b, au_n, 8); b.truncate(au_n + 4 + 8) }), None, "stream_exhausted", "decode");
fx(t, "wia1_unread_word", AUD, edit(&au, &|b| { let l = get_u32(b, au_n); set_u32(b, au_n, l + 4); b.extend_from_slice(&[0; 4]) }), None, "stream_noncanonical", "decode");
// ── WIS1 ── (32 channels of 8×8 → 14 planes of 32×32)
let wis = |f: &dyn Fn(&mut wire::Wis1<'_>)| {
let mut v = wire::parse_wis1(&sp).expect("splat parses");
f(&mut v);
Some(wire::write_wis1(&v))
};
fx(t, "wis1_json_form", SPL, None, Some(format!("{V}/int_splat/splat.json")), "bad_magic", "parse");
fx(t, "wis1_range_order", SPL, wis(&|v| v.ranges[0] = (1.0, -1.0)), None, "field_range", "parse");
fx(t, "wis1_range_nan", SPL, wis(&|v| v.ranges[0].0 = f32::NAN), None, "field_range", "parse");
fx(t, "wis1_n_gaussians", SPL, wis(&|v| v.n_gaussians = v.s * v.s + 1), None, "field_range", "parse");
fx(t, "wis1_cdf_len", SPL, edit(&sp, &|b| b[32 + 14 * 8..32 + 14 * 8 + 2].copy_from_slice(&2u16.to_le_bytes())), None, "field_range", "parse");
fx(t, "wis1_trailing", SPL, edit(&sp, &|b| b.push(0)), None, "trailing", "parse");
fx(t, "wis1_geometry", SPL, wis(&|v| { v.attr = 13; v.ranges.truncate(13) }), None, "geometry_mismatch", "decode");
fx(t, "wis1_afrac_model", SPL, edit(&sp, &|b| b[5] = 11), None, "model_mismatch", "decode");
let sp_l = at(&sp, wire::parse_wis1(&sp).expect("splat parses").rans) - 4;
fx(t, "wis1_exhausted", SPL, edit(&sp, &|b| { set_u32(b, sp_l, 8); b.truncate(sp_l + 4 + 8) }), None, "stream_exhausted", "decode");
fx(t, "wis1_unread_word", SPL, edit(&sp, &|b| { let l = get_u32(b, sp_l); set_u32(b, sp_l, l + 4); b.extend_from_slice(&[0; 4]) }), None, "stream_noncanonical", "decode");
let sets = "int-payload-conformance/sets";
t.put(&format!("{sets}/entropy_median_max.json"), entropy_median_max(t));
for f in &mut out {
if f.name == "wih1_order_exhausted_before_value" {
f.set = vec![
("entropy.json", format!("{sets}/entropy_median_max.json")),
("g_s.whm", format!("{V}/int_hyper/tiny_g_s.bin")),
("h_s.whm", format!("{V}/int_hyper/tiny_h_s.bin")),
];
}
}
out
}
/// A `WIV1` keyframe-0 clip of one 1 × 1 frame whose three residual symbols are 0
/// against the table `[0, 32768, 49152, 65536]`, with the stream `[w0, 4]`: `w0 = 0` is
/// the canonical coding of those symbols, which decodes to a grey pixel; `w0 = 1`
/// codes the same symbols from the state `2^31 + 1`.
fn grey_pixel(w0: u32) -> Vec<u8> {
let rans: Vec<u8> = [w0, 4].iter().flat_map(|w| w.to_le_bytes()).collect();
wire::write_wiv1(&wire::Wiv1 {
mv_precision: 1, keyframe: 0, n_frames: 1, h: 1, w: 1, block: 1, q: 1, offset: 0,
cdf: vec![0, 32_768, 49_152, 65_536], mvs: vec![], rans: &rans, wih1: None,
})
}
/// The tiny conditional codec's entropy tables with the first hyper-latent channel's
/// median at `i64::MAX` and its symbol offset at 1, so that channel's first symbol, if
/// its value were used, would rebuild past `i64`.
fn entropy_median_max(t: &Tree) -> Vec<u8> {
let mut e = t.json(&format!("{V}/int_hyper/tiny_entropy.json"));
e["z_median_q"][0] = json!(i64::MAX);
e["z_offsets"][0] = json!(1);
let mut text = serde_json::to_string(&e).expect("entropy");
text.push('\n');
text.into_bytes()
}
/// The offset in `base` at which `part`, a slice of it, starts.
fn at(base: &[u8], part: &[u8]) -> usize {
part.as_ptr() as usize - base.as_ptr() as usize
}
/// Build every file: the payloads, the fixtures, the index.
fn build(wai: &Path) -> Tree {
let mut t = Tree { wai: wai.to_path_buf(), out: BTreeMap::new() };
let vs = vectors(&mut t);
let fs = fixtures(&mut t);
let set = |s: &[(&str, String)]| -> Value { s.iter().map(|(k, v)| (k.to_string(), json!(v))).collect::<serde_json::Map<_, _>>().into() };
let index = json!({
"format": "wai-int-payload-vectors/1",
"about": "Binary payload vectors of the WAI-native integer codecs and their refusal fixtures (extensions/integer-payloads.md §12). Paths are relative to wai/. Written by wai_int_payload_vectors gen.",
"vectors": vs.iter().map(|v| json!({
"name": v.name,
"capability": v.cap,
"payload": v.payload,
"payload_sha256": sha256(&t.read(&v.payload)),
"converted_from": v.from,
"set": set(&v.set),
"sidecar": v.sidecar,
})).collect::<Vec<_>>(),
"refusals": fs.iter().map(|f| json!({
"name": f.name,
"capability": f.cap,
"payload": f.payload,
"code": f.code,
"layer": f.layer,
"set": set(&f.set),
})).collect::<Vec<_>>(),
});
let mut text = serde_json::to_string_pretty(&index).expect("index");
text.push('\n');
t.put("int-payload-conformance/vectors.json", text.into_bytes());
t
}
/// The SHA-256 digests of a decode's output, in the form its sidecar entry states.
fn digests(d: &IntDecoded) -> Value {
match d {
IntDecoded::Video { frames, .. } => {
let per: Vec<[u8; 32]> = frames.iter().map(|f| Sha256::digest(f).into()).collect();
let clip = Sha256::digest(per.concat());
json!({ "frames_sha256": per.iter().map(|d| hex(d)).collect::<Vec<_>>(), "clip_sha256": hex(&clip) })
}
IntDecoded::Image { rgb, .. } => json!({ "sha256": sha256(rgb) }),
IntDecoded::Pcm { samples, .. } => {
let le: Vec<u8> = samples.iter().flat_map(|s| s.to_le_bytes()).collect();
json!({ "sha256": sha256(&le) })
}
IntDecoded::Attr { bytes, .. } => json!({ "sha256": sha256(bytes) }),
_ => json!(null),
}
}
/// `f` with the parameters of the set `set` names (member → path) for `cap`.
fn with_set<R>(t: &Tree, cap: &str, set: &Value, f: impl FnOnce(&IntParams<'_>) -> R) -> R {
let owned: BTreeMap<String, Vec<u8>> = set.as_object().into_iter().flatten()
.map(|(k, v)| (k.clone(), t.read(v.as_str().expect("a set path")))).collect();
let view: BTreeMap<String, &[u8]> = owned.iter().map(|(k, v)| (k.clone(), v.as_slice())).collect();
let params = wai::int_payload::resolve(cap, None, &[HeldBytes::Set { listing: b"", files: &view }], None)
.unwrap_or_else(|_| IntParams::none());
f(&params)
}
/// Decode `payload` through `cap` against the set `set` names (member → path).
fn decode(t: &Tree, cap: &str, payload: &[u8], set: &Value) -> Result<IntDecoded, wai::int_payload::IntPayloadError> {
with_set(t, cap, set, |p| decode_int(cap, payload, p))
}
fn verify(wai: &Path) -> bool {
let t = build(wai);
let mut ok = true;
for (rel, bytes) in &t.out {
match std::fs::read(wai.join(rel)) {
Ok(have) if &have == bytes => {}
Ok(_) => { println!("FAIL {rel}: differs from its regeneration"); ok = false }
Err(e) => { println!("FAIL {rel}: {e}"); ok = false }
}
}
let index: Value = serde_json::from_slice(&t.out["int-payload-conformance/vectors.json"]).expect("index");
for v in index["vectors"].as_array().expect("vectors") {
let (name, cap) = (v["name"].as_str().unwrap(), v["capability"].as_str().unwrap());
let payload = t.read(v["payload"].as_str().unwrap());
match decode(&t, cap, &payload, &v["set"]) {
Ok(d) => {
let want = &t.json(v["sidecar"].as_str().unwrap())[name];
let got = digests(&d);
let same = got.as_object().unwrap().iter().all(|(k, g)| &want[k] == g);
if same {
println!("ok {name} ({cap}): decodes to its sidecar's SHA-256");
} else {
println!("FAIL {name} ({cap}): decoded {got} ≠ sidecar");
ok = false;
}
}
Err(e) => { println!("FAIL {name} ({cap}): {e}"); ok = false }
}
}
let mut seen = std::collections::BTreeSet::new();
for f in index["refusals"].as_array().expect("refusals") {
let (name, cap, code) = (f["name"].as_str().unwrap(), f["capability"].as_str().unwrap(), f["code"].as_str().unwrap());
let payload = t.read(f["payload"].as_str().unwrap());
let parsed = match wai::codecs::pinned_int_payload(cap) {
Some(wire::MAGIC_WIV1) => wire::parse_wiv1(&payload, Some((cap == wai::codecs::CAP_VIDEO_INT_HYPER) as u8)).err(),
Some(wire::MAGIC_WIH1) => wire::parse_wih1(&payload).err(),
Some(wire::MAGIC_WIA1) => wire::parse_wia1(&payload).err(),
Some(wire::MAGIC_WIS1) => wire::parse_wis1(&payload).err(),
_ => None,
};
let layer_ok = (f["layer"] == "parse") == parsed.is_some();
// The declared cost reads what a decode reads before its first symbol, so it
// refuses with the decode's code, or admits the payload when the decode's
// refusal comes from a stream or a network run (§7, steps 5 and 6).
let declared = with_set(&t, cap, &f["set"], |p| declared_cost(cap, &payload, p).map(|_| ()).map_err(|e| e.code()));
let declared_ok = match declared {
Err(c) => c == code,
Ok(()) => matches!(code, "stream_exhausted" | "stream_noncanonical" | "field_range"),
};
match decode(&t, cap, &payload, &f["set"]) {
Err(e) if e.code() == code && layer_ok && declared_ok => { seen.insert(code); println!("ok {name}: {code}") }
Err(e) if e.code() == code && layer_ok => {
println!("FAIL {name}: the decode gives {code}, the declared cost {declared:?}");
ok = false
}
Err(e) => { println!("FAIL {name}: refused with {e}, want {code} at {}", f["layer"]); ok = false }
Ok(_) => { println!("FAIL {name}: decoded, want {code}"); ok = false }
}
}
println!("{} refusal codes covered by fixtures", seen.len());
// The end-state fixture's canonical twin, the same symbols coded from 2^31, decodes.
match decode(&t, wai::codecs::CAP_VIDEO_INT_MOTION, &grey_pixel(0), &Value::Null) {
Ok(IntDecoded::Video { frames, .. }) if frames == [vec![128u8; 3]] => {
println!("ok grey_pixel: wiv1_end_state's canonical twin decodes to one grey pixel")
}
other => { println!("FAIL grey_pixel: {other:?}"); ok = false }
}
ok
}
fn main() -> ExitCode {
let args: Vec<String> = std::env::args().collect();
let (Some(cmd), Some(dir)) = (args.get(1), args.get(2)) else {
eprintln!("usage: wai_int_payload_vectors gen|verify <int-payload-conformance dir>");
return ExitCode::from(2);
};
let wai = Path::new(dir).join("..");
match cmd.as_str() {
"gen" => {
let t = build(&wai);
for (rel, bytes) in &t.out {
let p = wai.join(rel);
if let Some(d) = p.parent() {
std::fs::create_dir_all(d).expect("create directory");
}
std::fs::write(&p, bytes).unwrap_or_else(|e| panic!("write {}: {e}", p.display()));
}
println!("wrote {} files", t.out.len());
ExitCode::SUCCESS
}
"verify" => {
if verify(&wai) { ExitCode::SUCCESS } else { ExitCode::FAILURE }
}
_ => {
eprintln!("usage: wai_int_payload_vectors gen|verify <int-payload-conformance dir>");
ExitCode::from(2)
}
}
}

44
wai/wai-rs/src/bin/wai_video_meter.rs
View file

@ -8,9 +8,14 @@
//! decode burst against an idle baseline, and reports the **marginal** CPU
//! energy attributable to the decode.
//!
//! wai_video_meter <bitstream.json> [receipt_out.json [report_out.json]]
//! wai_video_meter <payload.wiv1 | bitstream.json> [receipt_out.json [report_out.json]]
//! [--no-report] [--previous-report <report.json>]
//!
//! The codestream is the `WIV1` payload `wai.video.int_motion` carries
//! (`extensions/integer-payloads.md` §3), and the receipt identifies it. Given a
//! tool's JSON bitstream instead, the meter converts it to `WIV1` first
//! (`int_payload::wiv1_from_json`) and meters and seals the converted payload.
//!
//! Video was the last cargo class with no meter: haptics, 4D splats, scores and
//! film are metered by `wai_meter`, worlds/feeds/avatars by `wai_world_meter`,
//! quantum circuits by `wai_quantum_meter`. It also has the strongest claim of
@ -42,21 +47,28 @@ use ed25519_dalek::SigningKey;
use wai::energy_burst::{measure_with_macmon, power_reader};
use wai::energy_figure::meter_report;
use wai::energy_class::CounterFiltering;
use wai::int_video::decode;
use wai::video_receipt::{work_pixel_ops, Provenance, VideoReceipt};
use wai::video_receipt::{decode_wiv1, work_pixel_ops, Provenance, VideoReceipt};
fn main() {
let report_args = meter_report::parse_args_or_exit(&std::env::args().collect::<Vec<_>>());
let args = &report_args.positional;
let Some(path) = args.get(1) else {
eprintln!("usage: wai_video_meter <bitstream.json> [receipt_out.json [report_out.json]] [--no-report] [--previous-report <report.json>]");
eprintln!("usage: wai_video_meter <payload.wiv1 | bitstream.json> [receipt_out.json [report_out.json]] [--no-report] [--previous-report <report.json>]");
std::process::exit(2);
};
let bytes = std::fs::read(path).expect("read bitstream");
let bs: serde_json::Value = serde_json::from_slice(&bytes).expect("parse bitstream JSON");
let bytes = std::fs::read(path).expect("read the codestream");
// The payload is WIV1; a JSON bitstream is the tools' form, converted first.
let payload = if bytes.starts_with(b"WIV1") {
bytes
} else {
let bs: serde_json::Value = serde_json::from_slice(&bytes).expect("neither a WIV1 payload nor a JSON bitstream");
let p = wai::int_payload::wiv1_from_json(&bs, None).expect("the JSON bitstream converts to WIV1");
eprintln!("{path} is a JSON bitstream: metering and sealing the WIV1 payload converted from it");
p
};
// A decode that does not even run is not meterable.
let clip = decode(&bs).expect("codestream decodes");
let clip = decode_wiv1(&payload).expect("codestream decodes");
let (n, h, w) = (clip.frames.len() as u64, clip.h as u64, clip.w as u64);
let work = work_pixel_ops(n, h, w);
@ -78,8 +90,8 @@ fn main() {
// as one.
let mut values = None;
let measured: Option<(u64, Provenance)> =
match measure_with_macmon("decode", &bs, |b| {
let _ = decode(b);
match measure_with_macmon("decode", &payload, |p| {
let _ = decode_wiv1(p);
}) {
None => {
eprintln!(
@ -144,19 +156,11 @@ fn main() {
// label it also did not earn.
None => (0, None),
};
let receipt = VideoReceipt::seal_decode(
&signer,
signer_id,
&bytes,
&bs,
uj,
class,
None,
)
.expect("seal receipt");
let receipt = VideoReceipt::seal_decode_wiv1(&signer, signer_id, &payload, uj, class, None)
.expect("seal receipt");
assert!(receipt.verify(), "emitted receipt must verify");
assert!(
receipt.verify_reconstruction(&bytes, &bs),
receipt.verify_reconstruction_wiv1(&payload),
"emitted receipt must re-decode to the same frames"
);
let tag = if measured.is_some() { "MEASURED" } else { "UNMETERED" };

13
wai/wai-rs/src/claim_receipt.rs
View file

@ -113,22 +113,15 @@ use std::fmt::Write as _;
pub const DOMAIN_CLAIM: &[u8] = b"moq-jwp:claim\x01";
/// Domain of a claim receipt's link hash (its identity for `parent_receipt_hash`).
pub const DOMAIN_CLAIM_LINK: &[u8] = b"moq-jwp:claim-link\x01";
/// jwp-receipts §2.1: the per-object content hash domain.
pub const DOMAIN_OBJECT: &[u8] = b"moq-jwp:object\x01";
// jwp-receipts §2.1: the per-object content hash and its domain, kept under their
// names here so every existing caller of this module's API stays valid.
pub use crate::jwp_hash::{content_hash as object_content_hash, DOMAIN_OBJECT};
/// The largest integer a claim may carry: I-JSON's (RFC 7493) exact-integer bound,
/// which is what keeps RFC 8785 number serialisation trivial. Crate-private so the
/// C header (cbindgen exports public integer constants) does not gain a generic name.
pub(crate) const MAX_INTEGER: u64 = (1 << 53) - 1;
/// jwp-receipts §2.1 content hash of a WAI envelope — the hard binding.
pub fn object_content_hash(envelope_bytes: &[u8]) -> [u8; 32] {
let mut h = blake3::Hasher::new();
h.update(DOMAIN_OBJECT);
h.update(envelope_bytes);
*h.finalize().as_bytes()
}
// ───────────────────────────── reasons ─────────────────────────────
/// Why a claim, or a recovery, was rejected. The label is the conformance name.

50
wai/wai-rs/src/codecs/mod.rs
View file

@ -47,8 +47,8 @@ pub const CAP_AUDIO_FLAC: &str = "wai.audio.flac"; // lossless
// Video
pub const CAP_VIDEO_AV1: &str = "wai.video.av1"; // lossy AV1
pub const CAP_VIDEO_AV1_LOSSLESS: &str = "wai.video.av1.lossless"; // lossless AV1
pub const CAP_VIDEO_INT_MOTION: &str = "wai.video.int_motion"; // DETERMINISTIC INTEGER inter-frame codec (int_video.rs): I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS. Decoder IMPLEMENTED, pure i64, no float/ONNX — every frame is byte-identical on every machine (frame-state-equivalence; per-frame portable hash). The temporal extension of the integer contract. Bitstream from tools/wai_int_video_encode.py
pub const CAP_VIDEO_INT_HYPER: &str = "wai.video.int_hyper"; // NEURAL-KEYFRAME inter-frame codec: the I-frame is the LEARNED integer codec int_hyper (wai.neural.int_hyper) decoded byte-exact, the P-frames are integer block-MC residuals on top (int_video::decode_pframes). A learned keyframe you can put in a receipt + integer inter-frames — the whole decode integer, byte-identical everywhere. Composes int_hyper_synth + int_video; pure i64
pub const CAP_VIDEO_INT_MOTION: &str = "wai.video.int_motion"; // DETERMINISTIC INTEGER inter-frame codec (int_video.rs): I-frame + integer block motion-compensated P-frames + integer-quantized residual over rANS. Decoder IMPLEMENTED, pure i64, no float/ONNX — every frame is byte-identical on every machine (frame-state-equivalence; per-frame portable hash). The temporal extension of the integer contract. Payload WIV1, keyframe 0 (extensions/integer-payloads.md); the encode tools write a JSON form a converter reads
pub const CAP_VIDEO_INT_HYPER: &str = "wai.video.int_hyper"; // NEURAL-KEYFRAME inter-frame codec: the I-frame is the LEARNED integer codec int_hyper (wai.neural.int_hyper) decoded byte-exact, the P-frames are integer block-MC residuals on top (int_video::decode_pframes_typed). A learned keyframe you can put in a receipt + integer inter-frames — the whole decode integer, byte-identical everywhere. Composes int_hyper_synth + int_video; pure i64. Payload WIV1, keyframe 1, a nested WIH1 (extensions/integer-payloads.md)
// Text
pub const CAP_TEXT_ZSTD: &str = "wai.text.zstd"; // general-purpose
pub const CAP_TEXT_XZ: &str = "wai.text.xz"; // maximum classical ratio
@ -61,7 +61,7 @@ pub const CAP_NEURAL_WAVTOKENIZER: &str = "wai.neural.wavtokenizer"; // ultra-lo
pub const CAP_NEURAL_BMSHJ2018: &str = "wai.neural.bmshj2018"; // bmshj2018-factorized image codec (2018 baseline)
pub const CAP_NEURAL_MBT2018_MEAN: &str = "wai.neural.mbt2018_mean"; // mean-scale HYPERPRIOR image codec (Gaussian-conditional entropy) — decoder IMPLEMENTED in this crate (neural/hyperprior.rs); the conditional-entropy family: a mean-scale hyperprior conditions each latent's Gaussian; sink-supplied ONNX (hyper_synthesis + synthesis)
pub const CAP_NEURAL_INT_SYNTH: &str = "wai.neural.int_synth"; // INTEGER fixed-point synthesis (neural/int_synth.rs): all-i64 transposed-conv decode, no float/ONNX — bit-identical on every machine BY CONSTRUCTION. The first capability that can soundly back intent=replicate cross-hardware (determinism IS integer-exact). Weights are sink-supplied (int_synth.json from a QAT model)
pub const CAP_NEURAL_INT_HYPER: &str = "wai.neural.int_hyper"; // INTEGER fixed-point CONDITIONAL codec (int_hyper_synth.rs): the full Gaussian-conditional decode integer end-to-end — factorized rANS-decode z (z_q = s·2^afrac + median_q) → integer hyper-decoder h_s → σ,μ → integer-σ bucket (int_entropy) → integer rANS-decode residual → y_hat = r·2^afrac + μ → integer synthesis g_s. Both bitstreams entropy-coded, no raw latents. No float/ONNX anywhere, so the WHOLE bitstream→image decode is bit-identical on every machine BY CONSTRUCTION — int_synth's sibling that adds the conditional entropy. Quality: PTQ of the pretrained mbt2018_mean (quality 6); CI (wai-sota-vectors.yml) requires the integer decode within 1 dB of the same model's float decode on a pinned test photograph — measured cost there 0.19 dB (35.985 vs 36.175 dB, arm64 macOS); image-dependent, see int_hyper_synth.rs. Tables sink-supplied
pub const CAP_NEURAL_INT_HYPER: &str = "wai.neural.int_hyper"; // INTEGER fixed-point CONDITIONAL codec (int_hyper_synth.rs): the full Gaussian-conditional decode integer end-to-end — factorized rANS-decode z (z_q = s·2^afrac + median_q) → integer hyper-decoder h_s → σ,μ → integer-σ bucket (int_entropy) → integer rANS-decode residual → y_hat = r·2^afrac + μ → integer synthesis g_s. Both bitstreams entropy-coded, no raw latents; payload WIH1 (extensions/integer-payloads.md). No float/ONNX anywhere, so the WHOLE bitstream→image decode is bit-identical on every machine BY CONSTRUCTION — int_synth's sibling that adds the conditional entropy. Quality: PTQ of the pretrained mbt2018_mean (quality 6); CI (wai-sota-vectors.yml) requires the integer decode within 1 dB of the same model's float decode on a pinned test photograph — measured cost there 0.19 dB (35.985 vs 36.175 dB, arm64 macOS); image-dependent, see int_hyper_synth.rs. Tables sink-supplied
pub const CAP_NEURAL_INT_MLICPP: &str = "wai.neural.int_mlicpp"; // INTEGER MLIC++-topology codec (int_mlicpp.rs): the channel-autoregressive × checkerboard entropy model — K channel slices, each decoded as an anchor pass then a non-anchor pass, with an entropy network over concat[y decoded so far, hyper] giving σ‖μ before every pass — with every decode-side network integer (int_hyper_synth networks, int_entropy bucketing, integer rANS). The encoder and every decoder compute the same σ, so the same bucket, at any thread count: no float/ONNX anywhere, byte-identical on every machine BY CONSTRUCTION. Refuses any stream value outside the exact integer range. Payload (WIM1) and one-file model bundle (WMB1, so a prior pins every network and table) specified in int_mlicpp.rs. Conformance vector tests/vectors/int_mlicpp/ has SYNTHETIC weights (seeded, quantised like a trained model): it shows decode-equivalence, not compression quality. Model sink-supplied
pub const CAP_NEURAL_ELIC: &str = "wai.neural.elic"; // checkerboard CONTEXT-MODEL image codec (neural/context.rs) — the JPEG-AI/MLIC++ entropy class: hyperprior + 2-pass spatial context (anchor → context net → non-anchor). Decoder IMPLEMENTED; sink-supplied ONNX (hyper_synthesis + context + synthesis). WAI-native, NOT the ISO JPEG AI bitstream
pub const CAP_NEURAL_MLICPP: &str = "wai.neural.mlicpp"; // MLIC++: the multi-reference entropy model — channel-autoregressive slices + checkerboard + global attention. Decoder IMPLEMENTED (neural/mlicpp.rs): the slice/checkerboard decode orchestration; the attention nets are sink-supplied ONNX (hyper_synthesis + entropy + synthesis)
@ -136,8 +136,10 @@ pub const CAP_SPLAT_SPZ: &str = "wai.splat.spz";
// synthesis (int_hyper machinery, 14 channels) over a factorized-rANS latent →
// byte-exact attribute tensor on every machine (splat-equivalence; a portable cloud
// hash). The int_hyper of 3D — the decoded attributes you can put in a receipt.
// Decoder IMPLEMENTED, pure i64, no float/ONNX. Model + bitstream from
// tools/wai_int_splat_export.py. The float rasterizer stays sink-supplied.
// Decoder IMPLEMENTED, pure i64, no float/ONNX. Payload WIS1, with each attribute's
// presentation range (extensions/integer-payloads.md); its parameter set is one file,
// the WHM1 synthesis. tools/wai_int_splat_export.py writes the model and a JSON form a
// converter reads. The float rasterizer stays sink-supplied.
pub const CAP_SPLAT_INT_CODEC: &str = "wai.splat.int_codec";
// Object-based spatial audio (extensions/spatial-audio, Phase 1). Ship
@ -161,10 +163,29 @@ pub const CAP_AUDIO_SCORE: &str = "wai.audio.score";
// decoding the per-channel factorized-rANS latent to int16 PCM byte-identically on
// every machine (sample-equivalence; a portable PCM hash). The audio analog of
// int_hyper — a neural audio codec whose reconstruction you can put in a receipt.
// Decoder IMPLEMENTED, pure i64, no float/ONNX. Model + bitstream from
// tools/wai_int_audio_export.py
// Decoder IMPLEMENTED, pure i64, no float/ONNX. Payload WIA1, with its sample rate
// (extensions/integer-payloads.md); its parameter set is one file, the WAM1 synthesis.
// tools/wai_int_audio_export.py writes the model and a JSON form a converter reads.
pub const CAP_AUDIO_INT_CODEC: &str = "wai.audio.int_codec";
/// SPEC §5, *Payloads of the WAI-native integer codecs*: each capability whose payload
/// this revision pins to a format of `extensions/integer-payloads.md`, with the
/// format's magic. The one table `int_payload`, `container::check_int_payload_framing`
/// (and so `wai wrap` and `wai inspect`) and the SPEC test read. `WIV1` is pinned twice:
/// keyframe 0 for `wai.video.int_motion`, keyframe 1 for `wai.video.int_hyper`.
pub const PINNED_INT_PAYLOADS: &[(&str, [u8; 4])] = &[
(CAP_VIDEO_INT_MOTION, *b"WIV1"),
(CAP_VIDEO_INT_HYPER, *b"WIV1"),
(CAP_NEURAL_INT_HYPER, *b"WIH1"),
(CAP_AUDIO_INT_CODEC, *b"WIA1"),
(CAP_SPLAT_INT_CODEC, *b"WIS1"),
];
/// The magic of the format SPEC §5 pins `cap`'s payload to, or `None`.
pub fn pinned_int_payload(cap: &str) -> Option<[u8; 4]> {
PINNED_INT_PAYLOADS.iter().find(|(c, _)| *c == cap).map(|&(_, m)| m)
}
// Haptics (extensions/haptics): per-channel intensity/sharpness envelope,
// reconstructed to an actuator signal (sample-equivalence). Engine
// behind the `haptic` feature.
@ -1265,9 +1286,11 @@ pub fn parameter_set_shape(cap: &str) -> Option<ParameterSetShape> {
}
Some(match cap {
// int_synth: its model specification, int_synth.json. int_mlicpp: its WMB1
// bundle. The audio codecs and the per-frame video codec: one self-contained
// ONNX decoder (a model that keeps tensors in external files is several files).
CAP_NEURAL_INT_SYNTH | CAP_NEURAL_INT_MLICPP
// bundle. audio.int_codec: its WAM1 synthesis (model.wam). splat.int_codec: its
// WHM1 synthesis (g_s.whm). The audio codecs and the per-frame video codec: one
// self-contained ONNX decoder (a model that keeps tensors in external files is
// several files).
CAP_NEURAL_INT_SYNTH | CAP_NEURAL_INT_MLICPP | CAP_AUDIO_INT_CODEC | CAP_SPLAT_INT_CODEC
| CAP_NEURAL_ENCODEC32 | CAP_NEURAL_DAC | CAP_NEURAL_MIMI | CAP_NEURAL_WAVTOKENIZER
| CAP_NEURAL_SNAC | CAP_NEURAL_VIDEO_BMSHJ2018 => ParameterSetShape::OneFile,
_ => ParameterSetShape::SeveralFiles,
@ -1993,14 +2016,15 @@ mod tests {
}
}
one_file.sort_unstable();
let mut want = vec![CAP_NEURAL_INT_SYNTH, CAP_NEURAL_INT_MLICPP, CAP_NEURAL_ENCODEC32, CAP_NEURAL_DAC,
CAP_NEURAL_MIMI, CAP_NEURAL_WAVTOKENIZER, CAP_NEURAL_SNAC, CAP_NEURAL_VIDEO_BMSHJ2018];
let mut want = vec![CAP_NEURAL_INT_SYNTH, CAP_NEURAL_INT_MLICPP, CAP_AUDIO_INT_CODEC, CAP_SPLAT_INT_CODEC,
CAP_NEURAL_ENCODEC32, CAP_NEURAL_DAC, CAP_NEURAL_MIMI, CAP_NEURAL_WAVTOKENIZER,
CAP_NEURAL_SNAC, CAP_NEURAL_VIDEO_BMSHJ2018];
want.sort_unstable();
assert_eq!(one_file, want);
for c in [CAP_IMAGE_PNG, CAP_SEMANTIC_CLIP, CAP_PRIOR_SET, CAP_WORLD_MODEL] {
assert!(!decodes_against_parameter_set(c), "{c}");
}
for c in [CAP_IMAGE_JPEGAI, CAP_NEURAL_GLC, CAP_GENERATE_SPECTRAL, CAP_AUDIO_INT_CODEC] {
for c in [CAP_IMAGE_JPEGAI, CAP_NEURAL_GLC, CAP_GENERATE_SPECTRAL, CAP_NEURAL_INT_HYPER, CAP_VIDEO_INT_HYPER] {
assert_eq!(parameter_set_shape(c), Some(ParameterSetShape::SeveralFiles), "{c}");
}
// The listing form: lossless, royalty-free, StandardDefined.

48
wai/wai-rs/src/codecs/registry.rs
View file

@ -1212,6 +1212,54 @@ mod tests {
assert!(problems.is_empty(), "SPEC and the registry disagree:\n{}", problems.join("\n"));
}
/// SPEC §5, *Payloads of the WAI-native integer codecs*, pins exactly the strings
/// and formats of `PINNED_INT_PAYLOADS`, and names each string's parameter-set
/// members as `int_payload` reads them; the extension defines each format in its
/// own section.
#[test]
fn spec_pins_the_integer_payloads_the_registry_lists() {
let spec = read("SPEC.md");
let at = spec.find("### Payloads of the WAI-native integer codecs").expect("the SPEC §5 subsection");
let section = &spec[at..];
let section = &section[..section[4..].find("\n### ").map_or(section.len(), |e| e + 4)];
let mut rows: Vec<(String, String, String)> = Vec::new();
for line in section.lines().filter(|l| l.starts_with("| `wai.")) {
let cells: Vec<&str> = line.split('|').map(str::trim).collect();
let tick = |c: &str| c.split('`').nth(1).unwrap_or_default().to_string();
rows.push((tick(cells[1]), tick(cells[2]), cells[3].to_string()));
}
let mut want: Vec<(String, String)> = super::super::PINNED_INT_PAYLOADS.iter()
.map(|(c, m)| (c.to_string(), String::from_utf8_lossy(m).into_owned())).collect();
let mut got: Vec<(String, String)> = rows.iter().map(|(c, m, _)| (c.clone(), m.clone())).collect();
want.sort();
got.sort();
assert_eq!(got, want, "SPEC's table and PINNED_INT_PAYLOADS");
// Each row's set: its shape is the registry's, and its members are named.
let members: &[(&str, &[&str])] = &[
(CAP_VIDEO_INT_MOTION, &[]),
(CAP_VIDEO_INT_HYPER, &["g_s.whm", "h_s.whm", "entropy.json"]),
(CAP_NEURAL_INT_HYPER, &["g_s.whm", "h_s.whm", "entropy.json"]),
(CAP_AUDIO_INT_CODEC, &["model.wam"]),
(CAP_SPLAT_INT_CODEC, &["g_s.whm"]),
];
for (cap, _, set) in &rows {
let (_, names) = members.iter().find(|(c, _)| c == cap).expect("a pinned string");
let shape = match crate::codecs::parameter_set_shape(cap) {
None => "none",
Some(crate::codecs::ParameterSetShape::OneFile) => "one file",
Some(crate::codecs::ParameterSetShape::SeveralFiles) => "several files",
};
assert!(set.starts_with(shape), "{cap}: SPEC says `{set}`, the registry {shape}");
for n in *names {
assert!(set.contains(&format!("`{n}`")), "{cap}: SPEC names no `{n}`");
}
}
let ext = read("extensions/integer-payloads.md");
for (heading, magic) in [("## 3.", "WIV1"), ("## 4.", "WIH1"), ("## 5.", "WIA1"), ("## 6.", "WIS1")] {
assert!(ext.lines().any(|l| l.starts_with(heading) && l.contains(&format!("`{magic}`"))), "the extension's {heading} is {magic}");
}
}
#[test]
fn vocab_v2_names_every_registered_string_and_no_candidate_as_emittable() {
let shapes = vocab_v2_shapes();

341
wai/wai-rs/src/container.rs
View file

@ -733,6 +733,15 @@ impl Security {
}
}
/// SPEC §6.4 step 2 at a sink that implements no encryption scheme, as the reference
/// dispatchers are: a manifest whose `security` declares an encryption other than
/// `"none"` makes the envelope inert here, and no ciphertext reaches a codec. A sink
/// that opens a scheme decrypts first and dispatches the plaintext envelope (§6.4
/// step 3), whose `security` is plaintext. `declared` is the encryption as written.
fn encryption_inert(declared: impl Into<String>) -> Inert {
Inert::Encrypted { encryption: declared.into() }
}
/// A `WAI1` manifest (SPEC §3). Build one with [`Manifest::new`].
#[derive(Debug, Clone, Serialize, Deserialize)]
#[non_exhaustive]
@ -809,14 +818,19 @@ pub enum Inert {
/// The pin was refused for `capability` (the first refusal kept, step 3 before
/// step 5).
PinRefused { capability: String, refusal: PinRefusal },
/// The manifest declares an encryption this sink cannot open (SPEC §6.4 step 2,
/// `unsupported_encryption`): the reference dispatchers implement no scheme, so an
/// encrypted envelope is inert at them before any capability is considered.
Encrypted { encryption: String },
}
impl Inert {
/// `missing_capability`, or the pin's reason.
/// `missing_capability`, `unsupported_encryption`, or the pin's reason.
pub fn code(&self) -> &'static str {
match self {
Self::MissingCapability => "missing_capability",
Self::PinRefused { refusal, .. } => refusal.code(),
Self::Encrypted { .. } => "unsupported_encryption",
}
}
}
@ -826,6 +840,11 @@ impl std::fmt::Display for Inert {
match self {
Self::MissingCapability => write!(f, "missing_capability: the sink advertises neither the capability nor the fallback"),
Self::PinRefused { capability, refusal } => write!(f, "{}: {capability}: {refusal}", refusal.code()),
Self::Encrypted { encryption } => write!(
f,
"unsupported_encryption: the manifest declares the encryption {encryption}, which this sink cannot \
open (SPEC §6.4): the file is inert here, and its payload is not decoded"
),
}
}
}
@ -904,7 +923,9 @@ impl Wai {
}
/// SPEC §4 dispatch at a sink that advertises the capabilities `advertises`
/// accepts and verifies pins with `verify` (SPEC §3.1). Step 3 takes the
/// accepts and verifies pins with `verify` (SPEC §3.1). First, SPEC §6.4 step 2:
/// a manifest that declares an encryption other than `"none"` is inert here
/// ([`Inert::Encrypted`]), since this dispatcher opens no scheme. Step 3 takes the
/// capability when the sink advertises it and, where the pin applies to it,
/// `verify` accepts the pin; step 5 does the same for the fallback. A refused
/// pin makes that capability unsupported for this envelope: dispatch moves on,
@ -935,6 +956,11 @@ impl Wai {
advertises: impl Fn(&str) -> bool,
mut verify: impl FnMut(&str, &Prior) -> Result<V, PinRefusal>,
) -> Result<Dispatch<'_, V>, Inert> {
// SPEC §6.4 step 2, before §4: an encrypted payload is inert at a sink that
// opens no scheme.
if let Some(sec) = self.manifest.security.as_ref().filter(|s| !s.is_plaintext()) {
return Err(encryption_inert(format!("{:?}", sec.encryption.as_deref().unwrap_or_default())));
}
let primary = self.capability();
if crate::codecs::is_companion_only(primary) {
return Err(Inert::MissingCapability);
@ -1036,11 +1062,31 @@ impl Wai {
check_candidates_v1(self.capability(), self.fallback())
}
/// The envelope's bytes, for an encoder to emit: [`Self::check_candidates`],
/// then [`Self::to_bytes`]. The C packer `wai_envelope_pack` emits through
/// this, so an encoder that uses it cannot emit a candidate (SPEC §5).
/// Check that the payload is the format SPEC §5 pins for the `capability`, and for
/// the `fallback` (step 5 decodes the same payload through it), where either is an
/// integer capability with a pinned format ([`check_int_payload_framing`]). An
/// encrypted payload (`security.encryption` other than `"none"`) is ciphertext, so
/// it is not checked here: a sink decrypts it first (SPEC §6.4) and checks the
/// plaintext when it decodes it.
pub fn check_int_payload_framing(&self) -> Result<(), FramingError> {
if !self.manifest.security.as_ref().is_none_or(Security::is_plaintext) {
return Ok(());
}
check_int_payload_framing(self.capability(), &self.payload)?;
match self.fallback() {
Some(fb) => check_int_payload_framing(fb, &self.payload),
None => Ok(()),
}
}
/// The envelope's bytes, for an encoder to emit: [`Self::check_candidates`], then
/// [`Self::check_int_payload_framing`], then [`Self::to_bytes`]. The C packer
/// `wai_envelope_pack` emits through this, so an encoder that uses it cannot emit a
/// candidate, or an integer payload that is not its capability's pinned format
/// (SPEC §5).
pub fn to_emit_bytes(&self) -> Result<Vec<u8>, EmitError> {
self.check_candidates().map_err(EmitError::Candidate)?;
self.check_int_payload_framing().map_err(|error| EmitError::IntPayload { component: None, error })?;
self.to_bytes().map_err(EmitError::Serialize)
}
@ -1330,6 +1376,131 @@ pub fn check_component_candidates<'a>(capabilities: impl IntoIterator<Item = &'a
}
}
/// Whether a `WAI2` manifest's payloads are plaintext: no `security` member, or one
/// whose `encryption` is absent, `null` or `"none"` (SPEC §6). A `security` member
/// that is not a security object is not plaintext: a sink treats the payloads as
/// ciphertext it cannot open, and an encoder does not write it
/// ([`manifest_v2_security`]).
pub fn manifest_v2_plaintext(m: &ManifestV2) -> bool {
matches!(manifest_v2_security(m), Ok(s) if s.as_ref().is_none_or(Security::is_plaintext))
}
/// A `WAI2` manifest's `security` member (SPEC §6.1, §7.1): `None` when absent or
/// `null`; the object; or why the member is not a security object (`true`, a string,
/// an array, an `encryption` that is not a string, …), which an encoder does not write
/// ([`WaiMulti::to_emit_bytes`]).
pub fn manifest_v2_security(m: &ManifestV2) -> Result<Option<Security>, String> {
match m.extra.get("security") {
None | Some(Value::Null) => Ok(None),
Some(v) => serde_json::from_value::<Security>(v.clone()).map(Some)
.map_err(|e| format!("the manifest's `security` member {v} is not a security object (SPEC §6.1): {e}")),
}
}
/// Why a payload is not the format SPEC §5 pins its capability to (*Payloads of the
/// WAI-native integer codecs*, `extensions/integer-payloads.md`): the capability, the
/// code the extension's §8 names and what failed. The payload is parsed as a sink parses
/// it (§2–§6), so every code a parse can give is possible here: `bad_magic` (none of the
/// four formats; a JSON form lands here), `format_capability_mismatch`,
/// `keyframe_capability_mismatch`, `version`, `reserved_nonzero`, `field_range`,
/// `too_large`, `cdf_malformed`, `stream_length`, `nested_length` and `trailing`. What
/// only a decode checks, against a parameter set and through the streams (§7), is not.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FramingError {
pub capability: String,
pub code: &'static str,
pub detail: &'static str,
}
impl std::fmt::Display for FramingError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let want = crate::codecs::pinned_int_payload(&self.capability).map(|m| String::from_utf8_lossy(&m).into_owned());
write!(f, "the payload is not the {} payload SPEC \u{a7}5 pins {} to ({}: {}; extensions/integer-payloads.md)",
want.as_deref().unwrap_or("?"), self.capability, self.code, self.detail)
}
}
impl std::error::Error for FramingError {}
/// Check that `payload` is the format SPEC §5 pins `capability`'s payload to
/// ([`crate::codecs::PINNED_INT_PAYLOADS`]): it is parsed with the wire parser a sink
/// runs (`int_payload::wire`, with the capability's keyframe kind for `WIV1`), which
/// checks every rule of the extension's §2–§6 in its order and reports the first that
/// fails ([`FramingError`]). `Ok` for a capability with no pinned integer format. It
/// needs no parameter set and no integer decoder, so every emit path runs it
/// ([`Wai::to_emit_bytes`], [`WaiMulti::to_emit_bytes`], `wai wrap`, the browser's
/// writers). `payload` is plaintext: an encrypted envelope's payload is checked after
/// it is decrypted (SPEC §6.4), and its emit paths skip the check.
pub fn check_int_payload_framing(capability: &str, payload: &[u8]) -> Result<(), FramingError> {
use crate::int_wire as w;
let parsed = match crate::codecs::pinned_int_payload(capability) {
None => return Ok(()),
Some(w::MAGIC_WIV1) => w::parse_wiv1(payload, Some((capability == crate::codecs::CAP_VIDEO_INT_HYPER) as u8)).map(|_| ()),
Some(w::MAGIC_WIH1) => w::parse_wih1(payload).map(|_| ()),
Some(w::MAGIC_WIA1) => w::parse_wia1(payload).map(|_| ()),
Some(_) => w::parse_wis1(payload).map(|_| ()),
};
parsed.map_err(|e| FramingError { capability: capability.to_string(), code: e.code(), detail: e.detail })
}
/// What an integer payload's header declares about its decode, read before any
/// symbol is decoded and with no parameter set (`extensions/integer-payloads.md` §7).
/// A sink applies its own size policy to it and refuses a payload past that policy
/// before decoding; `int_payload::declared_cost` adds what the parameter set fixes
/// (the networks' buffers, a `WIH1` image's size).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[non_exhaustive]
pub struct DeclaredSize {
/// The decoded output's bytes, where the header alone fixes it: the frames of a
/// `WIV1` (`n_frames·h·w·3`), the PCM of a `WIA1` (`2·t_out`), the attribute tensor
/// of a `WIS1` (`attr·s·s`). `None` for a `WIH1`, whose image size is its
/// synthesis network's.
pub output_bytes: Option<u64>,
/// The symbols the payload's streams code: one entropy-decode step each.
pub symbols: u64,
/// The most bytes the reference decode holds at once beyond the payload, where the
/// header alone fixes it: a `WIV1` with keyframe 0
/// ([`crate::int_wire::Wiv1::keyframe0_working_bytes`]). `None` where a network
/// runs, whose buffers the parameter set sizes.
pub working_bytes: Option<u64>,
/// The networks' multiply-accumulates, where the header alone fixes them: 0 for a
/// `WIV1` with keyframe 0, which runs no network; `None` otherwise.
pub macs: Option<u128>,
}
/// [`DeclaredSize`] for `payload` under `capability`, after the parse
/// [`check_int_payload_framing`] makes. `Ok(None)` for a capability with no pinned
/// integer format.
pub fn declared_size(capability: &str, payload: &[u8]) -> Result<Option<DeclaredSize>, FramingError> {
use crate::int_wire as w;
let fail = |e: w::WireError| FramingError { capability: capability.to_string(), code: e.code(), detail: e.detail };
let wih1 = |k: &w::Wih1<'_>| k.n_z as u64 * k.zh as u64 * k.zw as u64 + k.m as u64 * k.sh as u64 * k.sw as u64;
Ok(Some(match crate::codecs::pinned_int_payload(capability) {
None => return Ok(None),
Some(w::MAGIC_WIV1) => {
let v = w::parse_wiv1(payload, Some((capability == crate::codecs::CAP_VIDEO_INT_HYPER) as u8)).map_err(fail)?;
let frame = v.h as u64 * v.w as u64 * 3;
let output_bytes = Some(frame * v.n_frames as u64);
match v.wih1 {
None => DeclaredSize { output_bytes, symbols: frame * v.n_frames as u64, working_bytes: Some(v.keyframe0_working_bytes()), macs: Some(0) },
Some(k) => {
let symbols = frame * (v.n_frames as u64 - 1) + wih1(&w::parse_wih1(k).map_err(fail)?);
DeclaredSize { output_bytes, symbols, working_bytes: None, macs: None }
}
}
}
Some(w::MAGIC_WIH1) => DeclaredSize { output_bytes: None, symbols: wih1(&w::parse_wih1(payload).map_err(fail)?), working_bytes: None, macs: None },
Some(w::MAGIC_WIA1) => {
let a = w::parse_wia1(payload).map_err(fail)?;
DeclaredSize { output_bytes: Some(2 * a.t_out as u64), symbols: a.n_ch as u64 * a.lat_len as u64, working_bytes: None, macs: None }
}
Some(_) => {
let v = w::parse_wis1(payload).map_err(fail)?;
DeclaredSize { output_bytes: Some(v.attr as u64 * v.s as u64 * v.s as u64), symbols: v.c as u64 * v.sh as u64 * v.sw as u64, working_bytes: None, macs: None }
}
}))
}
/// Why [`Wai::to_emit_bytes`] or [`WaiMulti::to_emit_bytes`] wrote nothing.
#[derive(Debug)]
#[non_exhaustive]
@ -1338,6 +1509,14 @@ pub enum EmitError {
Candidate(CandidateError),
/// The envelope did not serialise.
Serialize(std::io::Error),
/// A payload is not the format SPEC §5 pins its integer capability to
/// ([`check_int_payload_framing`]): the `WAI2` component's index, or `None` for a
/// `WAI1`'s payload.
IntPayload { component: Option<usize>, error: FramingError },
/// A `WAI2` manifest's `security` member is not a security object
/// ([`manifest_v2_security`]): an encoder that writes one could claim an
/// encryption no sink can read, or none.
Security(String),
}
impl std::fmt::Display for EmitError {
@ -1345,6 +1524,9 @@ impl std::fmt::Display for EmitError {
match self {
EmitError::Candidate(e) => e.fmt(f),
EmitError::Serialize(e) => write!(f, "envelope serialize: {e}"),
EmitError::IntPayload { component: Some(i), error } => write!(f, "component {i}: {error}"),
EmitError::IntPayload { component: None, error } => error.fmt(f),
EmitError::Security(why) => f.write_str(why),
}
}
}
@ -1456,11 +1638,15 @@ pub enum Skip {
/// A policy that serves only sound `replicate` claims left the entry out:
/// SPEC §7's rule refuses `replicate` through it (`replicate_*`).
Replicate(crate::codecs::ReplicateRefusal),
/// The manifest declares an encryption the sink cannot open (SPEC §6.4 step 2,
/// `unsupported_encryption`): every entry's payload is ciphertext here.
Encrypted,
}
impl Skip {
/// The reason: `not_selectable`, `missing_capability`,
/// `component_digest_mismatch`, the pin's reason, or the `replicate_*` reason.
/// `component_digest_mismatch`, the pin's reason, the `replicate_*` reason, or
/// `unsupported_encryption`.
pub fn code(&self) -> &'static str {
match self {
Self::NotSelectable => "not_selectable",
@ -1468,6 +1654,7 @@ impl Skip {
Self::ComponentDigest => "component_digest_mismatch",
Self::Pin(r) => r.code(),
Self::Replicate(r) => r.code(),
Self::Encrypted => "unsupported_encryption",
}
}
}
@ -1645,7 +1832,10 @@ impl WaiMulti {
/// pin applies ([`pin_applies_to`]), `verify` accepts. An entry whose pin is
/// refused is skipped as if its capability were not advertised, and its reason
/// kept. When nothing is picked the envelope is inert at this sink, and the
/// error gives every entry's index and reason.
/// error gives every entry's index and reason. An envelope whose manifest declares
/// an encryption other than `"none"` is inert before any entry is considered (SPEC
/// §6.4 step 2; this dispatcher opens no scheme): every entry is
/// [`Skip::Encrypted`].
pub fn pick<V>(
&self,
advertises: impl Fn(&str) -> bool,
@ -1672,6 +1862,11 @@ impl WaiMulti {
eligible: impl Fn(usize, &RenditionMeta) -> Result<(), Skip>,
mut verify: impl FnMut(&str, &Prior) -> Result<V, PinRefusal>,
) -> Result<Picked<'_, V>, Vec<(usize, Skip)>> {
// SPEC §6.4 step 2, before selection: an encrypted envelope is inert at a sink
// that opens no scheme, and every entry is reported so.
if !manifest_v2_plaintext(&self.manifest) {
return Err((0..self.manifest.renditions.len()).map(|i| (i, Skip::Encrypted)).collect());
}
let mut skipped = Vec::new();
for (index, (meta, payload)) in self.manifest.renditions.iter().zip(self.renditions.iter()).enumerate() {
if !meta.is_selectable() {
@ -1708,11 +1903,31 @@ impl WaiMulti {
check_component_candidates(self.manifest.renditions.iter().map(|m| m.capability.as_str()))
}
/// The envelope's bytes, for an encoder to emit: [`Self::check_candidates`],
/// then [`Self::to_bytes`]. The C packer `wai_envelope_pack_multi` emits
/// through this, so an encoder that uses it cannot emit a candidate (SPEC §5).
/// [`check_int_payload_framing`] over every component, in table order: the
/// first component whose payload is not its capability's pinned format, with its
/// index. Skipped for an encrypted envelope (a `security` member whose
/// `encryption` is not `"none"`), whose payloads are ciphertext until a sink
/// decrypts them (SPEC §6.4).
pub fn check_int_payload_framing(&self) -> Result<(), (usize, FramingError)> {
if !manifest_v2_plaintext(&self.manifest) {
return Ok(());
}
for (i, (m, p)) in self.manifest.renditions.iter().zip(&self.renditions).enumerate() {
check_int_payload_framing(&m.capability, p).map_err(|e| (i, e))?;
}
Ok(())
}
/// The envelope's bytes, for an encoder to emit: [`Self::check_candidates`], then
/// [`manifest_v2_security`], then [`Self::check_int_payload_framing`], then
/// [`Self::to_bytes`]. The C packer `wai_envelope_pack_multi` emits through this, so
/// an encoder that uses it cannot emit a candidate, a `security` member that is not
/// a security object (SPEC §6.1), or a component whose integer payload is not its
/// capability's pinned format (SPEC §5).
pub fn to_emit_bytes(&self) -> Result<Vec<u8>, EmitError> {
self.check_candidates().map_err(EmitError::Candidate)?;
manifest_v2_security(&self.manifest).map_err(EmitError::Security)?;
self.check_int_payload_framing().map_err(|(i, error)| EmitError::IntPayload { component: Some(i), error })?;
self.to_bytes().map_err(EmitError::Serialize)
}
@ -2676,6 +2891,112 @@ mod tests {
assert_eq!(m.pick(|c| c == "wai.image.nope", no_parameter_sets()).unwrap().meta.capability, "wai.image.nope");
}
/// SPEC §5, *Payloads of the WAI-native integer codecs*: `to_emit_bytes` (the C
/// packers' emit path) parses an integer payload as a sink does and writes nothing
/// when it is not its capability's pinned format, as the capability, the fallback
/// or a `WAI2` component: a keyframe-0 `WIV1` under `wai.video.int_hyper`, a `WIV1`
/// cut inside its tables, a tool's JSON form. The committed payloads are written.
#[test]
fn the_emit_path_refuses_an_integer_payload_that_is_not_its_format() {
let read = |rel: &str| std::fs::read(std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/vectors").join(rel)).unwrap();
let (k0, k1) = (read("int_video/int_video_bitstream.wiv1"), read("int_video/video_nk_bitstream.wiv1"));
let (mot, hyp) = (crate::codecs::CAP_VIDEO_INT_MOTION, crate::codecs::CAP_VIDEO_INT_HYPER);
let v1 = |cap: &str, fb: Option<&str>, p: &[u8]| {
let mut req = ModelRequirement::new(cap);
if let Some(f) = fb {
req = req.with_fallback(f);
}
Wai::new(Manifest::new("video", "improve", req, Conditioning::new("bytes")), p.to_vec()).to_emit_bytes()
};
let code = |r: Result<Vec<u8>, EmitError>| match r {
Err(EmitError::IntPayload { component, error }) => Some((component, error.code)),
Err(e) => panic!("{e}"),
Ok(_) => None,
};
assert_eq!(code(v1(hyp, None, &k0)), Some((None, "keyframe_capability_mismatch")));
assert_eq!(code(v1(mot, None, &k0[..40])), Some((None, "stream_length")));
assert_eq!(code(v1(mot, None, &read("int_video/int_video_bitstream.json"))), Some((None, "bad_magic")));
assert_eq!(code(v1(crate::codecs::CAP_VIDEO_AV1, Some(mot), &k1)), Some((None, "keyframe_capability_mismatch")));
assert_eq!(code(v1(mot, None, &k0)), None);
assert_eq!(code(v1(hyp, None, &k1)), None);
let v2 = |a: &[u8], b: &[u8]| WaiMulti::new("video", "improve", serde_json::Value::Null, vec![
Rendition::media(mot, "bytes", a.to_vec()), Rendition::media(hyp, "bytes", b.to_vec()),
]).to_emit_bytes();
assert_eq!(code(v2(&k0, &k0)), Some((Some(1), "keyframe_capability_mismatch")));
assert_eq!(code(v2(&k0, &k1)), None);
}
/// An encrypted envelope's payload is ciphertext (SPEC §6): the emit paths do not
/// check it against the integer format, since a sink decrypts it first (§6.4) and
/// checks the plaintext when it decodes it. A `WAI1` and a `WAI2` that declare an
/// encryption are written; the same bytes in plaintext are refused.
#[test]
fn an_encrypted_integer_payload_is_emitted() {
let read = |rel: &str| std::fs::read(std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/vectors").join(rel)).unwrap();
let plain = read("int_video/int_video_bitstream.wiv1");
let cipher: Vec<u8> = plain.iter().enumerate().map(|(i, b)| b ^ (i as u8).wrapping_mul(151).wrapping_add(7)).collect();
let mot = crate::codecs::CAP_VIDEO_INT_MOTION;
let v1 = |security: Option<Security>| {
let mut m = Manifest::new("video", "improve", ModelRequirement::new(mot), Conditioning::new("bytes"));
m.security = security;
Wai::new(m, cipher.clone()).to_emit_bytes()
};
assert!(v1(Some(Security::new("aes-256-gcm", 7))).is_ok(), "an encrypted WAI1 is written");
assert!(matches!(v1(None), Err(EmitError::IntPayload { .. })), "the ciphertext in plaintext is refused");
assert!(matches!(v1(Some(Security::new("none", 0))), Err(EmitError::IntPayload { .. })), "`none` is plaintext");
let mut multi = WaiMulti::new("video", "improve", serde_json::Value::Null, vec![Rendition::media(mot, "bytes", cipher.clone())]);
assert!(matches!(multi.to_emit_bytes(), Err(EmitError::IntPayload { .. })));
multi.manifest.extra.insert("security".into(), serde_json::json!({"encryption": "mls-sframe", "kid": 0}));
assert!(multi.to_emit_bytes().is_ok(), "an encrypted WAI2 is written");
}
/// SPEC §6.4 step 2 at the reference dispatchers, which open no scheme: a manifest
/// that declares an encryption other than `"none"` is inert before any capability
/// is considered (`unsupported_encryption`), so a payload sent in the clear under a
/// false claim of encryption is not decoded as plaintext; `"none"`, `null` and an
/// absent member are plaintext. A `WAI2` `security` member that is not a security
/// object is inert too, and is never written.
#[test]
fn an_encrypted_envelope_is_inert_at_the_reference_dispatchers() {
let png = "wai.image.png";
let all = |_: &str| true;
let v1 = |security: Option<Security>| {
let mut m = Manifest::new("image", "improve", ModelRequirement::new(png), Conditioning::new("bytes"));
m.security = security;
Wai::new(m, b"bytes".to_vec())
};
assert_eq!(v1(None).dispatch_verified(all, no_parameter_sets()).unwrap().capability, png);
assert_eq!(v1(Some(Security::new("none", 0))).dispatch_verified(all, no_parameter_sets()).unwrap().capability, png);
let e = v1(Some(Security::new("aes-256-gcm", 7))).dispatch_verified(all, no_parameter_sets()).unwrap_err();
assert_eq!(e.code(), "unsupported_encryption");
assert!(e.to_string().contains("\"aes-256-gcm\""), "{e}");
let multi = |security: Option<Value>| {
let mut m = WaiMulti::new("image", "improve", Value::Null,
vec![Rendition::media(png, "bytes", b"a".to_vec()), Rendition::media(png, "bytes", b"b".to_vec())]);
if let Some(v) = security {
m.manifest.extra.insert("security".into(), v);
}
m
};
for plain in [None, Some(Value::Null), Some(serde_json::json!({"encryption": "none"})), Some(serde_json::json!({}))] {
let m = multi(plain.clone());
assert_eq!(m.pick(all, no_parameter_sets()).unwrap().index, 0, "{plain:?}");
assert!(m.to_emit_bytes().is_ok(), "{plain:?}");
}
let encrypted = multi(Some(serde_json::json!({"encryption": "mls-sframe", "kid": 0})));
assert!(encrypted.to_emit_bytes().is_ok(), "an encrypted WAI2 is written");
let skips = encrypted.pick(all, no_parameter_sets()).unwrap_err();
assert_eq!(skips, vec![(0, Skip::Encrypted), (1, Skip::Encrypted)]);
assert_eq!(skips[0].1.code(), "unsupported_encryption");
for bad in [serde_json::json!(true), serde_json::json!("aes"), serde_json::json!([]), serde_json::json!({"encryption": 5})] {
let m = multi(Some(bad.clone()));
assert!(matches!(m.to_emit_bytes(), Err(EmitError::Security(ref why)) if why.contains("is not a security object")), "{bad}");
let back = WaiMulti::from_bytes(&m.to_bytes().unwrap()).unwrap();
assert_eq!(back.pick(all, no_parameter_sets()).unwrap_err()[0].1, Skip::Encrypted, "{bad}: read, it is inert");
}
}
/// The emit path of the C packers (`wai_envelope_pack`,
/// `wai_envelope_pack_multi`), driven exactly as they drive it: the manifest
/// JSON parsed into the manifest type, the envelope built, and the bytes taken

5
wai/wai-rs/src/energy_figure/tests.rs
View file

@ -96,8 +96,8 @@ fn a_delivered_object_is_viewed_at_its_index_within_its_group() {
use crate::jwp_delivery::{GroupBuilder, ObjectLabel};
let env = |i: u8| vec![i; 9];
let mut b = GroupBuilder::default();
b.add(crate::claim_receipt::object_content_hash(&env(0)), 300, "o", Some(ObjectLabel::Class(chip(20_000))));
b.add(crate::claim_receipt::object_content_hash(&env(1)), 200, "o", None);
b.add(crate::jwp_hash::content_hash(&env(0)), 300, "o", Some(ObjectLabel::Class(chip(20_000))));
b.add(crate::jwp_hash::content_hash(&env(1)), 200, "o", None);
let (g, objs) = b.finish(&key(0x11), "p", 1, 2, "2026-09-30T12:00:00Z", "2026-09-30T12:00:01Z", None, false).unwrap();
let v = DeliveredObject { object: &objs[1], group: &g, envelope: &env(1) }.figure_view().unwrap();
assert_eq!(v.profile, "moq-jwp:object");
@ -143,6 +143,7 @@ fn a_provenance_receipt_is_viewed_step_by_step() {
#[cfg(feature = "video_receipt")]
#[test]
#[allow(deprecated)] // the receipt sealed over the JSON form, as its legacy pins are
fn the_video_receipt_is_viewed_by_its_receipt_hash() {
let p = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/vectors/int_video/int_video_bitstream.json");
let bytes = std::fs::read(p).unwrap();

139
wai/wai-rs/src/ffi.rs
View file

@ -162,17 +162,149 @@ where F: FnOnce() -> std::result::Result<Vec<u8>, (c_int, String)> + panic::Unwi
/// container's (`WAI_ERR_CONTAINER`).
fn emit_err(e: crate::container::EmitError) -> (c_int, String) {
match e {
crate::container::EmitError::Candidate(_) => (WAI_ERR_INVALID_ARG, e.to_string()),
crate::container::EmitError::Candidate(_) | crate::container::EmitError::IntPayload { .. } => (WAI_ERR_INVALID_ARG, e.to_string()),
_ => (WAI_ERR_CONTAINER, e.to_string()),
}
}
/// What decoding an integer payload will cost (`wai_int_payload_declared`,
/// `extensions/integer-payloads.md` §7, *Memory and work*): the decoded output's
/// bytes (`output_known` 1 where they are filled), the symbols its streams code, and,
/// where `cost_known` is 1, the most bytes the reference decode holds at once beyond
/// the payload and the parsed set (`working_bytes`) and the networks'
/// multiply-accumulates, `macs_hi·2^64 + macs_lo`.
#[repr(C)]
#[derive(Debug, Clone, Copy, Default)]
pub struct WaiIntDeclared {
pub output_bytes: u64,
pub output_known: u8,
pub symbols: u64,
pub working_bytes: u64,
pub macs_lo: u64,
pub macs_hi: u64,
pub cost_known: u8,
}
impl WaiIntDeclared {
/// The figures `container::declared_size` reads from the header alone.
fn from_size(d: crate::container::DeclaredSize) -> Self {
let macs = d.macs.unwrap_or(0);
WaiIntDeclared {
output_bytes: d.output_bytes.unwrap_or(0),
output_known: d.output_bytes.is_some() as u8,
symbols: d.symbols,
working_bytes: d.working_bytes.unwrap_or(0),
macs_lo: macs as u64,
macs_hi: (macs >> 64) as u64,
cost_known: (d.working_bytes.is_some() && d.macs.is_some()) as u8,
}
}
}
/// The members of `capability`'s parameter set read from `set_path`: inside it where
/// it is a directory, by the names the set lists them under; the file itself for a set
/// of one file; else the file named for the member beside it. A member that cannot be
/// read is absent, and the cost then refuses with `set_incomplete`.
#[cfg(feature = "neural_int")]
fn read_set(capability: &str, set_path: &std::path::Path) -> std::collections::BTreeMap<String, Vec<u8>> {
let members = crate::int_payload::set_members(capability);
members.iter()
.filter_map(|m| {
let f = if set_path.is_dir() {
set_path.join(m)
} else if members.len() == 1 {
set_path.to_path_buf()
} else {
set_path.parent().map(|d| d.join(m)).unwrap_or_else(|| std::path::PathBuf::from(m))
};
std::fs::read(f).ok().map(|b| (m.to_string(), b))
})
.collect()
}
/// Every figure of the declared cost, over the set read from `set_path`.
#[cfg(feature = "neural_int")]
fn cost_with_set(capability: &str, payload: &[u8], set_path: &str) -> std::result::Result<WaiIntDeclared, (c_int, String)> {
use crate::int_payload::{declared_cost, resolve};
let files = read_set(capability, std::path::Path::new(set_path));
let view: std::collections::BTreeMap<String, &[u8]> = files.iter().map(|(k, v)| (k.clone(), v.as_slice())).collect();
let params = resolve(capability, None, &[crate::pin::HeldBytes::Set { listing: b"", files: &view }], None)
.map_err(|r| (WAI_ERR_INVALID_ARG, format!("{}: {r}", r.code())))?;
let c = declared_cost(capability, payload, &params).map_err(|e| (WAI_ERR_INVALID_ARG, e.to_string()))?;
Ok(WaiIntDeclared {
output_bytes: c.output_bytes, output_known: 1, symbols: c.symbols, working_bytes: c.working_bytes,
macs_lo: c.macs as u64, macs_hi: (c.macs >> 64) as u64, cost_known: 1,
})
}
#[cfg(not(feature = "neural_int"))]
fn cost_with_set(capability: &str, _: &[u8], _: &str) -> std::result::Result<WaiIntDeclared, (c_int, String)> {
Err((WAI_ERR_INVALID_ARG, format!("{capability}: this build reads no parameter set (feature neural_int)")))
}
/// Read what decoding the payload of an integer capability will cost before decoding
/// it, so a sink can refuse a payload past its own size policy first
/// (`extensions/integer-payloads.md` §7, *Memory and work*). `capability` is the
/// envelope's capability (or the `WAI2` component's). The payload is parsed as a sink
/// parses it: one that is not the format SPEC §5 pins `capability` to is
/// `WAI_ERR_INVALID_ARG`, with its code in `wai_last_error`, and so is a capability
/// with no pinned integer format.
///
/// With `set_path` null, only the header is read: a `wai.neural.int_hyper` image's
/// size is its synthesis network's, so `output_known` is 0 for it, and `cost_known` is
/// 1 only for a `WIV1` with keyframe 0, which runs no network. With `set_path` naming
/// the capability's parameter set — a directory holding its members by the names §9
/// gives, or the file of a one-file set — every figure is filled
/// (`int_payload::declared_cost`), and the set is checked against the payload as a
/// decode checks it first (`set_incomplete`, `model_mismatch`, `geometry_mismatch`,
/// …). The set is read as given, unverified: a pin is verified when the envelope is
/// decoded. A build without `neural_int` reads no set.
///
/// # Safety
/// `capability` NUL-terminated; `payload` valid for `payload_len` bytes; `set_path`
/// null or NUL-terminated UTF-8; `out` writable.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn wai_int_payload_declared(
capability: *const c_char,
payload: *const u8, payload_len: usize,
set_path: *const c_char,
out: *mut WaiIntDeclared,
) -> c_int {
if capability.is_null() || out.is_null() {
set_err("int_payload_declared: capability/out null");
return WAI_ERR_INVALID_ARG;
}
let r = panic::catch_unwind(|| -> std::result::Result<WaiIntDeclared, (c_int, String)> {
let cap = unsafe { std::ffi::CStr::from_ptr(capability) }.to_string_lossy().to_string();
let p = unsafe { input_slice(payload, payload_len) }
.ok_or((WAI_ERR_INVALID_ARG, "payload null + len>0".into()))?;
let header = match crate::container::declared_size(&cap, p) {
Ok(Some(d)) => WaiIntDeclared::from_size(d),
Ok(None) => return Err((WAI_ERR_INVALID_ARG, format!("{cap} has no pinned integer payload format"))),
Err(e) => return Err((WAI_ERR_INVALID_ARG, e.to_string())),
};
if set_path.is_null() {
return Ok(header);
}
let set = unsafe { std::ffi::CStr::from_ptr(set_path) }.to_str()
.map_err(|_| (WAI_ERR_INVALID_ARG, "set_path is not UTF-8".to_string()))?;
cost_with_set(&cap, p, set)
});
match r {
Ok(Ok(d)) => { unsafe { *out = d; } WAI_OK }
Ok(Err((code, msg))) => { set_err(msg); code }
Err(_) => { set_err("internal panic in int_payload_declared"); WAI_ERR_PANIC }
}
}
/// Builds a complete WAI file from a codec-produced payload + manifest
/// fields. `manifest_json` is a UTF-8 JSON object overriding/supplying
/// any fields; `payload` is the bytes from one of the codec encoders.
/// Refuses (`WAI_ERR_INVALID_ARG`) a manifest that names a candidate capability
/// as its capability or fallback, which an encoder MUST NOT emit (SPEC §5,
/// *Candidate capabilities*): it emits through `Wai::to_emit_bytes`.
/// *Candidate capabilities*), and a payload that is not the format SPEC §5 pins its
/// integer capability or fallback to (*Payloads of the WAI-native integer codecs*):
/// it emits through `Wai::to_emit_bytes`.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn wai_envelope_pack(
manifest_json: *const c_char,
@ -226,7 +358,8 @@ pub unsafe extern "C" fn wai_envelope_detect_version(
/// - The i-th payload pairs with `manifest.renditions[i]`.
///
/// Refuses (`WAI_ERR_INVALID_ARG`) a table with a component, in either role,
/// whose capability is a candidate (SPEC §5, *Candidate capabilities*): it emits
/// whose capability is a candidate (SPEC §5, *Candidate capabilities*), or whose
/// payload is not the format SPEC §5 pins its integer capability to: it emits
/// through `WaiMulti::to_emit_bytes`.
///
/// # Safety

51
wai/wai-rs/src/ffi_neural.rs
View file

@ -19,6 +19,12 @@
//! `WAI_ERR_UNSOUND_REPLICATE`, and a `WAI2` envelope with nothing to pick is
//! `WAI_ERR_NO_RENDITION`. Output buffers are heap-allocated here; free them with
//! `wai_buffer_free`.
//!
//! Integer video, PCM and splat attributes (`neural::decodes_to_int`) have no
//! `wai_decoded_t` kind; the Rust API returns them (`int_payload::decode_int`). These
//! entry points do not advertise those capabilities, so such a payload is refused
//! before anything is decoded, and dispatch moves on to a fallback this ABI can
//! return.
use std::ffi::{c_char, c_int, CStr};
use std::panic;
@ -28,8 +34,8 @@ use std::slice;
use crate::container::{Wai, WaiMulti};
use crate::ffi::{set_err, WaiBuffer, WAI_ERR_CODEC, WAI_ERR_CONTAINER, WAI_ERR_INVALID_ARG,
WAI_ERR_NO_RENDITION, WAI_ERR_PANIC, WAI_ERR_PRIOR, WAI_ERR_UNSOUND_REPLICATE, WAI_OK};
use crate::neural::{decode_envelope, decode_envelope_strict, decode_multi, decode_multi_strict, DecodeError,
Decoded, ModelRegistry};
use crate::neural::{decode_envelope, decode_envelope_strict, decode_multi, decode_multi_strict, decodes_to_int,
DecodeError, Decoded, ModelRegistry};
unsafe fn input_envelope<'a>(bytes: *const u8, len: usize) -> Option<&'a [u8]> {
if bytes.is_null() && len > 0 { return None; }
@ -48,6 +54,7 @@ fn decode_status(e: DecodeError) -> (c_int, String) {
DecodeError::PriorMismatch(r) => (WAI_ERR_PRIOR, format!("{}: {r}", r.code())),
DecodeError::UnsoundReplicate(why) => (WAI_ERR_UNSOUND_REPLICATE, why),
DecodeError::NoRendition(_) => (WAI_ERR_NO_RENDITION, e.to_string()),
DecodeError::Encrypted(why) => (WAI_ERR_CONTAINER, why),
other => (WAI_ERR_CODEC, other.to_string()),
}
}
@ -66,14 +73,25 @@ fn run<F: FnOnce() -> Result<(), (c_int, String)> + panic::UnwindSafe>(f: F) ->
}
}
/// The refusal of a capability whose output this ABI has no kind for.
fn no_kind(caps: &[&str]) -> (c_int, String) {
(WAI_ERR_CODEC, format!(
"{} decodes to integer video, PCM or splat attributes, which have no wai_decoded_t kind: \
not decoded here; decode it with the Rust API (int_payload::decode_int)", caps.join(", ")))
}
/// Parse a `WAI1` envelope and decode it with `model_path` registered for its
/// capability.
/// capability. A capability whose output has no `wai_decoded_t` kind is refused before
/// anything is decoded.
fn decode_one(envelope: *const u8, envelope_len: usize, model_path: *const c_char) -> Result<Decoded, (c_int, String)> {
let bytes = unsafe { input_envelope(envelope, envelope_len) }
.ok_or((WAI_ERR_INVALID_ARG, "envelope null with len>0".to_string()))?;
let path = unsafe { input_cstr(model_path) }
.ok_or((WAI_ERR_INVALID_ARG, "model_path null or not UTF-8".to_string()))?;
let env = Wai::from_bytes(bytes).map_err(|e| (WAI_ERR_CONTAINER, format!("envelope parse: {e}")))?;
if decodes_to_int(env.capability()) {
return Err(no_kind(&[env.capability()]));
}
let reg = ModelRegistry::new().register(env.capability().to_string(), Path::new(path));
decode_envelope(&env, &reg).map_err(decode_status)
}
@ -211,7 +229,10 @@ impl WaiDecoded {
/// array of paths. The first path is the registered artifact; the others are
/// further sets a pin may resolve against (a set and its successor). Every pin is
/// verified first (SPEC §3.1); a `WAI2` rendition is picked by SPEC §7.1. With
/// `flags & WAI_DECODE_STRICT`, SPEC §7's replicate rule applies too.
/// `flags & WAI_DECODE_STRICT`, SPEC §7's replicate rule applies too. A capability
/// whose output has no `wai_decoded_t` kind (integer video, PCM, splat attributes) is
/// not advertised, whatever `registry_json` says: dispatch moves on past it before
/// decoding, and when nothing else decodes the error names it.
///
/// `*out` is written on every call with a non-null `out`, before anything else is
/// read: on failure it holds `kind` 0 and an empty `data`, so the caller may free
@ -241,7 +262,12 @@ pub unsafe extern "C" fn wai_neural_decode_ex(
let map: std::collections::BTreeMap<String, Vec<String>> = serde_json::from_str(rj)
.map_err(|e| (WAI_ERR_INVALID_ARG, format!("registry_json: {e}")))?;
let mut reg = ModelRegistry::new();
let mut kindless: Vec<String> = Vec::new();
for (cap, paths) in map {
if decodes_to_int(&cap) {
kindless.push(cap);
continue;
}
let mut it = paths.into_iter();
if let Some(first) = it.next() {
reg = reg.register(cap.clone(), first);
@ -253,14 +279,22 @@ pub unsafe extern "C" fn wai_neural_decode_ex(
let strict = flags & WAI_DECODE_STRICT != 0;
let version = crate::container::detect_version(bytes)
.map_err(|e| (WAI_ERR_CONTAINER, e.to_string()))?;
// A refusal at an envelope that names a capability left unadvertised above says so.
let explain = |named: Vec<&str>, e: DecodeError| {
let left: Vec<&str> = named.into_iter().filter(|c| kindless.iter().any(|k| k == c)).collect();
let (code, msg) = decode_status(e);
if left.is_empty() { (code, msg) } else { (code, format!("{msg}; {}", no_kind(&left).1)) }
};
let (index, decoded) = if version == 1 {
let env = Wai::from_bytes(bytes).map_err(|e| (WAI_ERR_CONTAINER, format!("envelope parse: {e}")))?;
let d = if strict { decode_envelope_strict(&env, &reg) } else { decode_envelope(&env, &reg) };
(0, d.map_err(decode_status)?)
let named = [Some(env.capability()), env.fallback()].into_iter().flatten().collect();
(0, d.map_err(|e| explain(named, e))?)
} else {
let m = WaiMulti::from_bytes(bytes).map_err(|e| (WAI_ERR_CONTAINER, format!("envelope parse: {e}")))?;
let r = if strict { decode_multi_strict(&m, &reg) } else { decode_multi(&m, &reg) };
r.map_err(decode_status)?
let named = m.manifest.renditions.iter().map(|r| r.capability.as_str()).collect();
r.map_err(|e| explain(named, e))?
};
let mut o = WaiDecoded { index: index as u32, ..WaiDecoded::empty() };
match decoded {
@ -283,6 +317,11 @@ pub unsafe extern "C" fn wai_neural_decode_ex(
o.fps_x_1000 = (v.fps * 1000.0).round() as u32;
o.data = WaiBuffer::from_vec(v.frames_rgb.concat());
}
// Not reached: such a capability is not advertised above. Refused rather
// than converted, so no caller could read one as another kind.
Decoded::Int(_) => {
return Err((WAI_ERR_CODEC, "this integer output (video, PCM or splat attributes) has no wai_decoded_t kind; decode it with the Rust API".to_string()));
}
}
unsafe { out.write(o) };
Ok(())

421
wai/wai-rs/src/int_audio.rs
View file

@ -35,47 +35,48 @@ pub struct AudioModel {
pub layers: Vec<ALayer>,
}
/// Little-endian, bounds-checked cursor over the binary model.
/// Little-endian cursor over the binary model, bounds-checked: every read is of bytes
/// that are present, with its end formed in checked arithmetic, so a damaged model is
/// refused and never makes the parser allocate more than the bytes it holds.
struct Cur<'a> {
b: &'a [u8],
p: usize,
}
impl Cur<'_> {
fn u32(&mut self) -> Result<usize, &'static str> {
let e = self.p + 4;
impl<'a> Cur<'a> {
/// The next `n` elements of `size` bytes each.
fn take(&mut self, n: usize, size: usize) -> Result<&'a [u8], &'static str> {
let e = n.checked_mul(size).and_then(|len| self.p.checked_add(len)).ok_or("model truncated")?;
let s = self.b.get(self.p..e).ok_or("model truncated")?;
self.p = e;
Ok(u32::from_le_bytes(s.try_into().unwrap()) as usize)
Ok(s)
}
fn u32(&mut self) -> Result<usize, &'static str> {
Ok(u32::from_le_bytes(self.take(1, 4)?.try_into().unwrap()) as usize)
}
fn u8(&mut self) -> Result<u8, &'static str> {
let v = *self.b.get(self.p).ok_or("model truncated")?;
self.p += 1;
Ok(v)
Ok(self.take(1, 1)?[0])
}
fn i16v(&mut self, n: usize) -> Result<Vec<i64>, &'static str> {
let e = self.p + 2 * n;
let s = self.b.get(self.p..e).ok_or("model truncated")?;
self.p = e;
Ok(s.chunks_exact(2).map(|c| i16::from_le_bytes([c[0], c[1]]) as i64).collect())
Ok(self.take(n, 2)?.chunks_exact(2).map(|c| i16::from_le_bytes([c[0], c[1]]) as i64).collect())
}
fn i32v(&mut self, n: usize) -> Result<Vec<i64>, &'static str> {
let e = self.p + 4 * n;
let s = self.b.get(self.p..e).ok_or("model truncated")?;
self.p = e;
Ok(s.chunks_exact(4).map(|c| i32::from_le_bytes([c[0], c[1], c[2], c[3]]) as i64).collect())
Ok(self.take(n, 4)?.chunks_exact(4).map(|c| i32::from_le_bytes([c[0], c[1], c[2], c[3]]) as i64).collect())
}
fn i64v(&mut self, n: usize) -> Result<Vec<i64>, &'static str> {
let e = self.p + 8 * n;
let s = self.b.get(self.p..e).ok_or("model truncated")?;
self.p = e;
Ok(s.chunks_exact(8).map(|c| i64::from_le_bytes(c.try_into().unwrap())).collect())
Ok(self.take(n, 8)?.chunks_exact(8).map(|c| i64::from_le_bytes(c.try_into().unwrap())).collect())
}
}
/// The product of a layer's dimensions, refused where it overflows `usize`.
fn count(dims: &[usize]) -> Result<usize, &'static str> {
dims.iter().try_fold(1usize, |a, &d| a.checked_mul(d)).ok_or("layer size overflows")
}
impl AudioModel {
/// Parse the export tool's compact binary model: magic `WAM1`, then
/// `afrac/gf/shift/n_layers` (u32) and per layer a `u8` tag — 0 convT1d, 2 igdn
/// — with int12 weights as `i16`, IGDN `gamma` as `i32`, the rest `i64`.
/// — with int12 weights as `i16`, IGDN `gamma` as `i32`, the rest `i64`, and
/// nothing after the last layer.
pub fn from_bytes(b: &[u8]) -> Result<Self, &'static str> {
if b.get(0..4) != Some(b"WAM1") {
return Err("bad model magic (want WAM1)");
@ -83,7 +84,10 @@ impl AudioModel {
let mut c = Cur { b, p: 4 };
let (afrac, gf, shift) = (c.u32()? as u32, c.u32()? as u32, c.u32()? as u32);
let n = c.u32()?;
let mut layers = Vec::with_capacity(n);
// Each layer is at least its tag byte, so a count beyond the bytes left is a
// truncated model, and reserving for it would let four damaged bytes ask for a
// large allocation.
let mut layers = Vec::with_capacity(n.min(b.len() - c.p));
for _ in 0..n {
layers.push(match c.u8()? {
0 => {
@ -91,15 +95,19 @@ impl AudioModel {
let out = c.u32()?;
let (k, stride, pad, opad) = (c.u32()?, c.u32()?, c.u32()?, c.u32()?);
ALayer::ConvT1d { out_ch: out, k, stride, pad, opad,
weight: c.i16v(inc * out * k)?, mult: c.i64v(out)?, bias: c.i64v(out)? }
weight: c.i16v(count(&[inc, out, k])?)?, mult: c.i64v(out)?, bias: c.i64v(out)? }
}
2 => {
let cc = c.u32()?;
ALayer::Igdn { c: cc, gamma: c.i32v(cc * cc)?, beta: c.i64v(cc)? }
ALayer::Igdn { c: cc, gamma: c.i32v(count(&[cc, cc])?)?, beta: c.i64v(cc)? }
}
_ => return Err("bad layer tag"),
});
}
// Nothing follows the last layer (`extensions/integer-payloads.md` §9).
if c.p != b.len() {
return Err("bytes follow the last layer");
}
Ok(Self { afrac, gf, shift, layers })
}
}
@ -167,81 +175,288 @@ pub mod reference {
}
}
/// Run the integer synthesis `g_s` on the fixed-point latent `x` (`[c, t]` at
/// `2^afrac`) → `([1, to], to)` waveform at `2^afrac`. Pure i64.
fn run_g_s(model: &AudioModel, x: &[i64], mut c: usize, mut t: usize) -> (Vec<i64>, usize) {
let (afrac, gf, shift) = (model.afrac, model.gf, model.shift);
let mut x = x.to_vec();
for layer in &model.layers {
match layer {
ALayer::ConvT1d { out_ch, k, stride, pad, opad, weight, mult, bias } => {
let (nx, to) = conv_t_1d(&x, c, t, weight, *out_ch, *k, *stride, *pad, *opad, mult, bias, shift);
x = nx;
c = *out_ch;
t = to;
}
ALayer::Igdn { c: _, gamma, beta } => {
x = igdn(&x, c, t, afrac, gamma, beta, gf);
}
}
}
(x, t)
}
fn i64s(v: &Value) -> Vec<i64> {
v.as_array().map(|a| a.iter().filter_map(|n| n.as_i64()).collect()).unwrap_or_default()
}
/// Decode a full audio **bitstream → int16 PCM**, end to end and fully integer:
///
/// 1. per channel, factorized rANS-decode the symbol `s`; rebuild the fixed-point
/// latent `y_q = s·2^afrac + median_q` (no float);
/// 2. integer synthesis `g_s` (transposed convs + IGDN) → waveform at `2^afrac`;
/// 3. quantize → int16 PCM (round-half-up, clamp).
///
/// `bs` is the export tool's `audio_bitstream.json`. Byte-identical on every machine.
pub fn decode(model: &AudioModel, bs: &Value) -> Result<Vec<i16>, &'static str> {
let c = bs["n_ch"].as_u64().ok_or("n_ch")? as usize;
let l = bs["lat_len"].as_u64().ok_or("lat_len")? as usize;
let t_out = bs["t_out"].as_u64().ok_or("t_out")? as usize;
let afrac = model.afrac;
let one = 1i64 << afrac;
let lo = bs["lo"].as_i64().ok_or("lo")? as i32;
let median_q = i64s(&bs["median_q"]);
if median_q.len() != c {
return Err("median_q length disagrees with n_ch");
}
let cdfs: Vec<Vec<i32>> = bs["cdfs"]
.as_array()
.ok_or("cdfs")?
.iter()
.map(|row| row.as_array().unwrap().iter().map(|n| n.as_i64().unwrap() as i32).collect())
.collect();
let cdf_len = cdfs.first().ok_or("empty cdfs")?.len();
let rans = b64_decode(bs["rans_b64"].as_str().ok_or("rans_b64")?)?;
use crate::int_hyper_synth::{refused, Refused, RefusalKind, MAX_AFRAC, MAX_TENSOR};
// 1. factorized rANS-decode the per-channel latent (channel-major), rebuild y_q.
let mut y = vec![0i64; c * l];
let mut dec = RansDecoder::new(&rans)?;
for ch in 0..c {
for p in 0..l {
let s = dec.decode(&cdfs[ch], cdf_len, lo);
y[ch * l + p] = s as i64 * one + median_q[ch];
/// A clip's coded form, typed: what a `WIA1` payload carries
/// (`extensions/integer-payloads.md` §5), and what the tool's JSON form carries, read
/// by [`AudioJson`]. `cdfs` holds one table per latent channel, all of one length.
#[derive(Debug, Clone, Copy)]
pub struct AudioStream<'a> {
/// The fixed-point scale the stream was coded at; it must equal the model's.
pub afrac: u32,
pub n_ch: usize,
pub lat_len: usize,
/// Samples presented: the first `t_out` of the synthesis.
pub t_out: usize,
/// The symbol offset every channel's table is coded at (the JSON form's `lo`).
pub offset: i32,
pub median_q: &'a [i64],
pub cdfs: &'a [Vec<i32>],
pub rans: &'a [u8],
}
/// The tool's JSON bitstream (`tools/wai_int_audio_export.py`'s
/// `audio_bitstream.json`), read into the typed form. A converter input and a
/// debugging aid; the payload is `WIA1`. What the old JSON decoder left implicit is
/// explicit here: its `afrac` is read (the old decoder ignored it) and must be the
/// model's; every channel's table must have the first table's length (the old decoder
/// read every table at that length).
#[derive(Debug, Clone)]
pub struct AudioJson {
pub afrac: u32,
pub n_ch: usize,
pub lat_len: usize,
pub t_out: usize,
pub offset: i32,
pub median_q: Vec<i64>,
pub cdfs: Vec<Vec<i32>>,
pub rans: Vec<u8>,
}
impl AudioJson {
/// Read the tool's JSON bitstream.
pub fn from_value(bs: &Value) -> Result<Self, &'static str> {
let uz = |k: &'static str| bs[k].as_u64().and_then(|v| usize::try_from(v).ok()).ok_or(k);
let (n_ch, lat_len, t_out) = (uz("n_ch")?, uz("lat_len")?, uz("t_out")?);
let afrac = bs["afrac"].as_u64().and_then(|v| u32::try_from(v).ok()).ok_or("afrac")?;
let offset = bs["lo"].as_i64().and_then(|v| i32::try_from(v).ok()).ok_or("lo")?;
let median_q = i64s(&bs["median_q"]);
let cdfs = bs["cdfs"].as_array().ok_or("cdfs")?.iter()
.map(|row| row.as_array().and_then(|r| r.iter().map(|n| n.as_i64().and_then(|v| i32::try_from(v).ok())).collect::<Option<Vec<i32>>>()))
.collect::<Option<Vec<_>>>().ok_or("cdfs holds a value outside i32")?;
let first = cdfs.first().ok_or("empty cdfs")?.len();
if cdfs.iter().any(|c| c.len() != first) {
return Err("the channels' tables are not of one length");
}
let rans = b64_decode(bs["rans_b64"].as_str().ok_or("rans_b64")?)?;
Ok(AudioJson { afrac, n_ch, lat_len, t_out, offset, median_q, cdfs, rans })
}
/// The typed form, borrowing this one's tables.
pub fn stream(&self) -> AudioStream<'_> {
AudioStream {
afrac: self.afrac, n_ch: self.n_ch, lat_len: self.lat_len, t_out: self.t_out, offset: self.offset,
median_q: &self.median_q, cdfs: &self.cdfs, rans: &self.rans,
}
}
// 2. integer synthesis → waveform.
let (x, t) = run_g_s(model, &y, c, l);
// 3. waveform → int16 PCM (round-half-up, clamp). Take t_out samples.
}
/// The model's output `[c, t]` on a `[c, t]` latent, planned from its layers' geometry
/// and parameter lengths alone, before anything is allocated
/// (`extensions/integer-payloads.md` §7). Layer by layer in order: the layer fits the
/// running channel count (`ModelMismatch`: output channels, kernel and stride
/// non-zero, `c·out·k` weights, `out` multipliers and biases, a requantisation shift
/// under 128; an inverse GDN of the running channel count, with `c·c` and `c`
/// parameters and shifts under 128); the uncropped length `(t − 1)·stride + k + opad`
/// and the buffer of `out` channels of it are within [`MAX_TENSOR`] (`TooLarge`); and
/// the crop `2·pad` leaves an output (`ModelMismatch`). Every size is formed in `u64`
/// with checked arithmetic, a size past `u64` being over the cap, so every target
/// refuses alike. Returns the output `[c, t]`, the most elements one layer holds, and
/// the multiply-accumulates the layers comprise (`extensions/integer-payloads.md` §7):
/// a transposed convolution's every input times `out·k`, an inverse GDN's every
/// element times `c`, summed in `u128`.
fn plan(model: &AudioModel, c: usize, t: usize) -> Result<(usize, usize, u64, u128), Refused> {
use RefusalKind::*;
const CAP: u64 = MAX_TENSOR as u64;
let (mut c, mut t) = (c as u64, t as u64);
// The most elements the reference kernels hold while one layer runs, its input
// included: a transposed convolution's input, uncropped buffer and output; an
// inverse GDN's input, output and one position's squares (`c` values of 16 bytes).
let mut live = c * t;
let mut macs: u128 = 0;
for layer in &model.layers {
match layer {
ALayer::ConvT1d { out_ch, k, stride, pad, opad, weight, mult, bias } => {
let (cout, k, s, p, op) = (*out_ch as u64, *k as u64, *stride as u64, *pad as u64, *opad as u64);
if cout == 0 || k == 0 || s == 0 {
return Err(refused(ModelMismatch, "a layer has zero output channels, kernel or stride"));
}
if c.checked_mul(cout).and_then(|v| v.checked_mul(k)) != Some(weight.len() as u64)
|| mult.len() as u64 != cout || bias.len() as u64 != cout || model.shift > 127
{
return Err(refused(ModelMismatch, "a layer's parameters do not match the running channel count"));
}
let full = (t - 1).checked_mul(s).and_then(|v| v.checked_add(k)).and_then(|v| v.checked_add(op));
let full = full.filter(|&n| n <= CAP).ok_or(refused(TooLarge, "a layer's uncropped length exceeds 2^26"))?;
if cout.checked_mul(full).is_none_or(|n| n > CAP) {
return Err(refused(TooLarge, "a layer would form a buffer over 2^26 elements"));
}
let crop = p.checked_mul(2).unwrap_or(u64::MAX);
if full <= crop {
return Err(refused(ModelMismatch, "a layer's padding leaves no output"));
}
live = live.max(c * t + cout * full + cout * (full - crop));
macs = macs.saturating_add((c * t) as u128 * (cout * k) as u128);
(c, t) = (cout, full - crop);
}
ALayer::Igdn { c: cc, gamma, beta } => {
if *cc as u64 != c || c.checked_mul(c) != Some(gamma.len() as u64) || beta.len() as u64 != c
|| model.gf > 127 || model.afrac > 127
{
return Err(refused(ModelMismatch, "an IGDN layer does not match the running channel count"));
}
live = live.max(2 * c * t + 2 * c);
macs = macs.saturating_add((c * t) as u128 * c as u128);
}
}
}
// Every running size is at most 2^26 here, so it fits usize on every target.
Ok((c as usize, t as usize, live, macs))
}
/// What [`decode_typed`] checks before it decodes a symbol, and the buffers it then
/// forms. Returns the synthesis's length, the most bytes the decode holds at once
/// on the reference kernels beyond the payload and the parsed model (the latent and
/// each layer's buffers, `i64` each, then the synthesis's output, `i64`, with the
/// PCM, `i16`), and the synthesis's multiply-accumulates on the latent.
///
/// The latent's length is the one `t_out` needs and no longer: the synthesis of
/// `lat_len` latent samples gives at least `t_out` samples (`GeometryMismatch`
/// otherwise), and that of `lat_len − 1` gives fewer (`GeometryMismatch`), so a
/// payload cannot declare a large latent to present a short clip. A length for which
/// the model leaves no output gives none.
pub(crate) fn audio_plan(model: &AudioModel, s: &AudioStream<'_>) -> Result<(usize, u64, u128), Refused> {
use RefusalKind::*;
let afrac = model.afrac;
if !(1..=MAX_AFRAC).contains(&afrac) {
return Err(refused(ModelMismatch, "the model's fixed-point scale is outside 1..=30"));
}
if s.afrac != afrac {
return Err(refused(ModelMismatch, "the stream's afrac is not the model's"));
}
let (c, l) = (s.n_ch, s.lat_len);
if c == 0 || l == 0 || s.t_out == 0 || s.median_q.len() != c || s.cdfs.len() != c {
return Err(refused(FieldRange, "a dimension is zero, or the medians or tables are not one per channel"));
}
if s.median_q.iter().any(|m| m.unsigned_abs() > 1 << 47) {
return Err(refused(FieldRange, "a median's magnitude exceeds 2^47"));
}
let len = s.cdfs[0].len();
let ok = |t: &Vec<i32>| t.len() == len && (3..=4097).contains(&len) && t[0] == 0 && t[len - 1] == 1 << 16
&& t.windows(2).all(|p| p[0] < p[1]);
if !s.cdfs.iter().all(ok) {
return Err(refused(CdfMalformed, "a table is not a CDF from 0 to 65536 of the channels' one length"));
}
if c.checked_mul(l).is_none_or(|n| n > MAX_TENSOR) || s.t_out > MAX_TENSOR {
return Err(refused(TooLarge, "the latent or the clip exceeds 2^26 elements"));
}
let (oc, t, live, macs) = plan(model, c, l)?;
if oc != 1 {
return Err(refused(ModelMismatch, "the model does not give one channel"));
}
if t < s.t_out {
return Err(refused(GeometryMismatch, "the synthesis gives fewer samples than t_out"));
}
let shorter = if l > 1 { plan(model, c, l - 1).map_or(0, |(_, t, ..)| t) } else { 0 };
if shorter >= s.t_out {
return Err(refused(GeometryMismatch, "the latent is longer than t_out needs: one sample fewer gives t_out samples"));
}
let working = (8 * live).max(8 * t as u64 + 2 * s.t_out as u64);
Ok((t, working, macs))
}
/// One layer's arithmetic on the values `x` holds, checked before it runs, as
/// [`crate::int_hyper_synth::run_net_checked`] checks a 2-D layer: every product and
/// partial sum and the requantised value of a convolution fit `i64`; IGDN's terms,
/// norm and root fit their types.
fn layer_in_range(model: &AudioModel, layer: &ALayer, x: &[i64], c: usize) -> Result<(), Refused> {
use crate::int_kernels::max_abs;
const I64: u128 = i64::MAX as u128;
match layer {
ALayer::ConvT1d { k, weight, mult, bias, .. } => {
if model.shift > 127 {
return Err(refused(RefusalKind::ModelMismatch, "requantisation shift of 128 or more"));
}
let terms = (c as u128).saturating_mul(*k as u128);
let acc = terms.saturating_mul(max_abs(weight) as u128).saturating_mul(max_abs(x) as u128);
if acc > I64 {
return Err(refused(RefusalKind::FieldRange, "a latent could take an accumulation outside i64"));
}
let q = ((acc * max_abs(mult) as u128) >> model.shift) + 1;
if q.saturating_add(max_abs(bias) as u128) > I64 {
return Err(refused(RefusalKind::FieldRange, "a latent could take a requantised value outside i64"));
}
Ok(())
}
ALayer::Igdn { gamma, beta, .. } => crate::int_hyper_synth::igdn_check(x, c, gamma, beta, model.gf, model.afrac)
.map_err(|(why, values)| refused(if values { RefusalKind::FieldRange } else { RefusalKind::ModelMismatch }, why)),
}
}
/// Decode a typed clip → int16 PCM, mono, byte-exact on every machine, refusing
/// rather than panicking or overflowing on any input:
///
/// 1. the stream's `afrac` is the model's, in `1..=30`; every table is well formed;
/// the model's layers fit the latent's shape, give one channel, and give at least
/// `t_out` samples from a latent no longer than that needs, with no buffer over
/// [`MAX_TENSOR`] elements — all before anything is allocated ([`audio_plan`]);
/// 2. per channel, rANS-decode the symbol `s`; the latent is `s·2^afrac + median_q`
/// ([`crate::int_entropy::dequantize`]), and a stream that runs out before its last
/// symbol is refused;
/// 3. integer synthesis `g_s`, each layer's arithmetic checked before it runs;
/// 4. the first `t_out` samples, `(x·32767 + 2^(afrac−1)) >> afrac` clamped to `i16`.
pub fn decode_typed(model: &AudioModel, s: &AudioStream<'_>) -> Result<Vec<i16>, Refused> {
use crate::int_entropy::dequantize;
use RefusalKind::*;
audio_plan(model, s)?;
let (afrac, c, l, len) = (model.afrac, s.n_ch, s.lat_len, s.cdfs[0].len());
// 1. factorized rANS-decode the per-channel latent (channel-major), rebuild y_q.
// The latent grows a symbol at a time, and the first symbol that runs the stream
// out refuses the payload before its value is used.
let mut y = Vec::new();
let mut dec = RansDecoder::new(s.rans).map_err(|_| refused(FieldRange, "the latent stream is not a rANS stream"))?;
for ch in 0..c {
for _ in 0..l {
let sym = dec.decode(&s.cdfs[ch], len, s.offset);
if dec.is_exhausted() {
return Err(refused(StreamExhausted, "the latent stream ended before its last symbol"));
}
crate::int_hyper_synth::push_capped(&mut y, dequantize(sym, afrac, s.median_q[ch]).ok_or(refused(FieldRange, "a latent value leaves i64"))?, c * l);
}
}
if !dec.is_canonical_end() {
return Err(refused(StreamNoncanonical, "the latent stream does not end canonically after its last symbol"));
}
// 2. integer synthesis → waveform, each layer checked on the values it is given.
let (mut x, mut cc, mut tt) = (y, c, l);
for layer in &model.layers {
layer_in_range(model, layer, &x, cc)?;
(x, cc, tt) = run_layer(model, layer, x, cc, tt);
}
// 3. waveform → int16 PCM (round-half-up, clamp): the first t_out samples.
let half = 1i64 << (afrac - 1);
let n = t_out.min(t);
let mut pcm = vec![0i16; n];
for (i, s) in pcm.iter_mut().enumerate() {
let num = x[i] * 32767 + half;
*s = (num >> afrac).clamp(-32768, 32767) as i16;
let mut pcm = Vec::with_capacity(s.t_out);
for &v in &x[..s.t_out] {
let num = v.checked_mul(32767).and_then(|n| n.checked_add(half))
.ok_or(refused(FieldRange, "a synthesised sample leaves i64 when scaled"))?;
pcm.push((num >> afrac).clamp(-32768, 32767) as i16);
}
Ok(pcm)
}
/// One layer of the synthesis on `x` (`[c, t]`): the new activations and shape.
fn run_layer(model: &AudioModel, layer: &ALayer, x: Vec<i64>, c: usize, t: usize) -> (Vec<i64>, usize, usize) {
match layer {
ALayer::ConvT1d { out_ch, k, stride, pad, opad, weight, mult, bias } => {
let (nx, to) = conv_t_1d(&x, c, t, weight, *out_ch, *k, *stride, *pad, *opad, mult, bias, model.shift);
(nx, *out_ch, to)
}
ALayer::Igdn { gamma, beta, .. } => (igdn(&x, c, t, model.afrac, gamma, beta, model.gf), c, t),
}
}
/// Decode the tool's JSON bitstream (`audio_bitstream.json`) → int16 PCM:
/// [`AudioJson`] then [`decode_typed`]. A debugging and converter path; the payload
/// is `WIA1`. Byte-identical on every machine.
pub fn decode(model: &AudioModel, bs: &Value) -> Result<Vec<i16>, &'static str> {
decode_typed(model, &AudioJson::from_value(bs)?.stream()).map_err(|r| r.detail)
}
/// FNV-1a 64-bit over the little-endian int16 PCM bytes — the dependency-free
/// checksum the export tool and this decoder share to prove byte-for-byte agreement
/// across languages (sample-equivalence).
@ -286,6 +501,38 @@ fn b64_decode(s: &str) -> Result<Vec<u8>, &'static str> {
}
/// The fast 1-D transposed conv against the scalar reference, element by element.
#[cfg(test)]
mod plan_tests {
use super::*;
/// The committed synthesis's planned multiply-accumulates on the committed latent are
/// its layers' terms at the lengths the reference kernel gives (run here on a zero
/// latent): a transposed convolution's every input times `out·k`, an inverse GDN's
/// every element times `c`.
#[test]
fn the_planned_macs_are_every_layers_terms() {
let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/vectors/int_audio");
let model = AudioModel::from_bytes(&std::fs::read(dir.join("audio_model.bin")).unwrap()).unwrap();
let wia1 = std::fs::read(dir.join("audio_bitstream.wia1")).unwrap();
let a = crate::int_wire::parse_wia1(&wia1).unwrap();
let (c0, l) = (a.n_ch as usize, a.lat_len as usize);
let (.., macs) = plan(&model, c0, l).unwrap();
let (mut c, mut t, mut terms) = (c0, l, 0u128);
for layer in &model.layers {
match layer {
ALayer::ConvT1d { out_ch, k, stride, pad, opad, weight, mult, bias } => {
let (_, to) = reference::conv_t_1d(&vec![0; c * t], c, t, weight, *out_ch, *k, *stride, *pad, *opad, mult, bias, model.shift);
terms += (c * t) as u128 * (out_ch * k) as u128;
(c, t) = (*out_ch, to);
}
ALayer::Igdn { .. } => terms += (c * t) as u128 * c as u128,
}
}
assert!(terms > 0);
assert_eq!(macs, terms);
}
}
#[cfg(test)]
mod kernel_equivalence {
use super::*;

590
wai/wai-rs/src/int_hyper_synth.rs
View file

@ -264,7 +264,52 @@ fn count(dims: &[usize]) -> Result<usize, &'static str> {
}
impl IntModel {
/// [`Self::from_json`], refusing rather than panicking on any JSON. It reads what
/// `from_json` reads, an absent number as 0 and an absent list as empty, as that
/// does, but refuses a member of the wrong type, a size that does not fit (`u32`
/// for `afrac`, `gf`, `shift` and a shift, `usize` for the rest), and a layer type
/// it does not know. A layer's parameters are checked against the running shapes
/// when the network is planned ([`decode_conditional_checked`]).
pub fn try_from_json(v: &Value) -> Result<Self, &'static str> {
let word = |o: &Value, k: &'static str| match o.get(k) {
None => Ok(0),
Some(n) => n.as_u64().and_then(|n| u32::try_from(n).ok()).ok_or(k),
};
let size = |o: &Value, k: &'static str| match o.get(k) {
None => Ok(0),
Some(n) => n.as_u64().and_then(|n| usize::try_from(n).ok()).ok_or(k),
};
let ints = |o: &Value, k: &'static str| -> Result<Vec<i64>, &'static str> {
match o.get(k) {
None => Ok(Vec::new()),
Some(a) => a.as_array().ok_or(k)?.iter().map(Value::as_i64).collect::<Option<_>>().ok_or(k),
}
};
let layers = v.get("layers").and_then(Value::as_array).ok_or("layers")?.iter().map(|l| {
Ok(match l.get("type").and_then(Value::as_str).ok_or("a layer's type")? {
"convT" => IntLayer::ConvT {
out_ch: size(l, "out_ch")?, k: size(l, "k")?, stride: size(l, "stride")?, pad: size(l, "pad")?,
opad: size(l, "opad")?, weight: ints(l, "weight")?, mult: ints(l, "mult")?, bias: ints(l, "bias")?,
},
"conv" => IntLayer::Conv {
out_ch: size(l, "out_ch")?, k: size(l, "k")?, stride: size(l, "stride")?, pad: size(l, "pad")?,
weight: ints(l, "weight")?, mult: ints(l, "mult")?, bias: ints(l, "bias")?,
},
"igdn" => IntLayer::Igdn { gamma: ints(l, "gamma")?, beta: ints(l, "beta")? },
"relu" => IntLayer::Relu,
"leaky" => IntLayer::Leaky {
slope_q: match l.get("slope_q") { None => 655, Some(q) => q.as_i64().ok_or("slope_q")? },
slope_frac: word(l, "slope_frac")?,
},
_ => return Err("an unknown layer type"),
})
}).collect::<Result<Vec<_>, &'static str>>()?;
Ok(Self { afrac: word(v, "afrac")?, gf: word(v, "gf")?, shift: word(v, "shift")?, layers })
}
/// Parse the export tool's JSON model (`int_synth_sota_model.json` / `int_hs_model.json`).
/// Panics on a JSON the tool would not write; [`Self::try_from_json`] refuses
/// instead.
pub fn from_json(v: &Value) -> Self {
let layers = v["layers"]
.as_array()
@ -291,7 +336,8 @@ impl IntModel {
/// Parse the export tool's **compact binary** model (`*_model.bin`): magic
/// `WHM1`, then `afrac/gf/shift/n_layers` (u32) and per layer a `u8` tag —
/// 0 convT, 1 conv, 2 igdn, 3 relu, 4 leaky — with int12 weights as `i16`,
/// IGDN `gamma` as `i32`, the rest `i64`. Decodes identically to [`from_json`].
/// IGDN `gamma` as `i32`, the rest `i64`, and nothing after the last layer.
/// Decodes identically to [`from_json`].
pub fn from_bytes(b: &[u8]) -> Result<Self, &'static str> {
if b.get(0..4) != Some(b"WHM1") {
return Err("bad model magic (want WHM1)");
@ -328,6 +374,11 @@ impl IntModel {
_ => return Err("bad layer tag"),
});
}
// Nothing follows the last layer (`extensions/integer-payloads.md` §9), so one
// network has one file.
if c.p != b.len() {
return Err("bytes follow the last layer");
}
Ok(Self { afrac, gf, shift, layers })
}
}
@ -449,7 +500,7 @@ pub fn run_net_checked(
/// A check [`check_layer`] failed: its reason, and whether it concerned the values
/// presented (`true`) or the layer itself (`false`).
type Failed = (&'static str, bool);
pub(crate) type Failed = (&'static str, bool);
const fn shape(reason: &'static str) -> Failed {
(reason, false)
}
@ -548,7 +599,6 @@ fn check_layer(model: &IntModel, layer: &IntLayer, x: &[i64], c: usize, h: usize
use crate::int_kernels::max_abs;
let (out, _) = layer_geometry(layer, c, h, w)?;
const I64: u128 = i64::MAX as u128;
const I128: u128 = i128::MAX as u128;
let xm = max_abs(x) as u128;
// One (transposed) convolution's arithmetic: every product and partial sum, then
// the requantised value, must fit i64 (the reference's accumulator and result).
@ -579,28 +629,7 @@ fn check_layer(model: &IntModel, layer: &IntLayer, x: &[i64], c: usize, h: usize
Ok(out)
}
IntLayer::Igdn { gamma, beta } => {
if c.checked_mul(c) != Some(gamma.len()) || beta.len() != c {
return Err(shape("IGDN parameter lengths do not match the channel count"));
}
if model.gf > 127 || model.afrac > 127 {
return Err(shape("IGDN shift of 128 or more"));
}
// x² ≤ 2^126 always fits; each term gamma·x², and the norm, must fit i128.
let x2 = xm * xm;
let term = (max_abs(gamma) as u128).checked_mul(x2).filter(|&t| t <= I128).ok_or(range("IGDN term could leave i128"))?;
let norm = (max_abs(beta) as u128).saturating_add((c as u128).saturating_mul((term >> model.gf) + 1));
if norm > I128 {
return Err(range("IGDN norm could leave i128"));
}
// The reference narrows the root to i64 (`isqrt(norm) as i64`), so it must
// fit; then |x · root| < 2^63 · 2^63 fits u128, and the result must fit i64.
let root = norm.isqrt();
if root > I64 {
return Err(range("IGDN root could leave i64"));
}
if ((xm * root) >> model.afrac) + 1 > I64 {
return Err(range("IGDN output could leave i64"));
}
igdn_check(x, c, gamma, beta, model.gf, model.afrac)?;
Ok((c, h, w))
}
IntLayer::Relu => Ok((c, h, w)),
@ -616,6 +645,39 @@ fn check_layer(model: &IntModel, layer: &IntLayer, x: &[i64], c: usize, h: usize
}
}
/// The IGDN checks [`run_net_checked`] documents, for activations `x` of `c` channels
/// at `2^afrac` against `gamma` (`c × c`, at `2^gf`) and `beta` (`c`). Shared with the
/// 1-D synthesis of `int_audio`, whose IGDN is the same [`igdn`].
pub(crate) fn igdn_check(x: &[i64], c: usize, gamma: &[i64], beta: &[i64], gf: u32, afrac: u32) -> Result<(), Failed> {
use crate::int_kernels::max_abs;
const I64: u128 = i64::MAX as u128;
const I128: u128 = i128::MAX as u128;
if c.checked_mul(c) != Some(gamma.len()) || beta.len() != c {
return Err(shape("IGDN parameter lengths do not match the channel count"));
}
if gf > 127 || afrac > 127 {
return Err(shape("IGDN shift of 128 or more"));
}
let xm = max_abs(x) as u128;
// x² ≤ 2^126 always fits; each term gamma·x², and the norm, must fit i128.
let x2 = xm * xm;
let term = (max_abs(gamma) as u128).checked_mul(x2).filter(|&t| t <= I128).ok_or(range("IGDN term could leave i128"))?;
let norm = (max_abs(beta) as u128).saturating_add((c as u128).saturating_mul((term >> gf) + 1));
if norm > I128 {
return Err(range("IGDN norm could leave i128"));
}
// The reference narrows the root to i64 (`isqrt(norm) as i64`), so it must
// fit; then |x · root| < 2^63 · 2^63 fits u128, and the result must fit i64.
let root = norm.isqrt();
if root > I64 {
return Err(range("IGDN root could leave i64"));
}
if ((xm * root) >> afrac) + 1 > I64 {
return Err(range("IGDN output could leave i64"));
}
Ok(())
}
/// Decode the integer synthesis `model` on the latent `input` (`[c, h, w]` at
/// `2^afrac`) → RGB, row-major **HWC** `u8`. Byte-identical on every machine.
pub fn decode(model: &IntModel, input: &[i64], c: usize, h: usize, w: usize) -> (Vec<u8>, usize, usize) {
@ -719,7 +781,24 @@ fn bytes(v: &Value) -> Vec<u8> {
}
impl CondBitstream {
/// Parse the bitstream the export tool emits (`cond_bitstream.json`).
/// [`Self::from_value`], refusing rather than panicking on any JSON: every member
/// present, a size an integer that fits `usize`, a stream an array of bytes. A
/// reader of JSON it did not write (the browser's debug decode) uses this, and
/// decodes with [`decode_conditional_checked`].
pub fn try_from_value(v: &Value) -> Result<Self, &'static str> {
let dim = |k: &'static str| v.get(k).and_then(Value::as_u64).and_then(|n| usize::try_from(n).ok()).ok_or(k);
let bytes = |k: &'static str| -> Result<Vec<u8>, &'static str> {
v.get(k).and_then(Value::as_array).ok_or(k)?.iter()
.map(|b| b.as_u64().and_then(|b| u8::try_from(b).ok())).collect::<Option<_>>().ok_or(k)
};
Ok(Self {
z_rans: bytes("z_rans")?, n_z: dim("n_z")?, zh: dim("zh")?, zw: dim("zw")?,
m: dim("m")?, sh: dim("sh")?, sw: dim("sw")?, rans: bytes("rans")?,
})
}
/// Parse the bitstream the export tool emits (`cond_bitstream.json`). Panics on a
/// JSON the tool would not write; [`Self::try_from_value`] refuses instead.
pub fn from_value(v: &Value) -> Self {
Self {
z_rans: bytes(&v["z_rans"]),
@ -780,6 +859,411 @@ pub fn decode_conditional(
Ok(decode(g_s, &y_hat, bs.m, sh, sw))
}
/// The kind of rule a checked decode refused on. The integer-payload decoders
/// (`int_payload`, and `int_audio` and `int_splat`'s typed decoders) report these as
/// the refusal codes of `extensions/integer-payloads.md` §8.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RefusalKind {
/// A value the payload carries, or a latent its stream decodes, is outside the
/// range the decode is exact over.
FieldRange,
/// A tensor or an activation would exceed [`MAX_TENSOR`] elements.
TooLarge,
/// A CDF table breaks the table rules.
CdfMalformed,
/// A rANS stream ran out before its last symbol.
StreamExhausted,
/// A rANS stream does not end canonically after its last symbol.
StreamNoncanonical,
/// A shape the payload declares disagrees with the one the decode produces.
GeometryMismatch,
/// The parameter set does not fit the payload.
ModelMismatch,
}
/// A checked decode's refusal: the kind of rule and which.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Refused {
pub kind: RefusalKind,
pub detail: &'static str,
}
impl std::fmt::Display for Refused {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{:?}: {}", self.kind, self.detail)
}
}
pub(crate) const fn refused(kind: RefusalKind, detail: &'static str) -> Refused {
Refused { kind, detail }
}
/// The most elements any one buffer a checked decode forms may hold, the same on
/// every target (`extensions/integer-payloads.md` §7).
///
/// cbindgen:ignore
pub const MAX_TENSOR: usize = 1 << 26;
/// The largest fixed-point scale a checked decode accepts.
///
/// cbindgen:ignore
pub const MAX_AFRAC: u32 = 30;
/// A table of `extensions/integer-payloads.md` §2 over its first `len` entries:
/// `3 ≤ len ≤ 4097` and `len` within the row, `cdf[0] = 0`, `cdf[len − 1] = 2^16`,
/// strictly increasing over `len`.
fn table_ok(cdf: &[i32], len: usize) -> bool {
(3..=4097).contains(&len) && len <= cdf.len() && cdf[0] == 0 && cdf[len - 1] == 1 << 16
&& cdf[..len].windows(2).all(|p| p[0] < p[1])
}
/// The entropy model's consistency: a non-empty scale table with one Gaussian table,
/// length and offset per edge; one `z` table, length, offset and median per channel;
/// every table one the rANS decoder is total over.
fn check_entropy(t: &CondEntropy) -> Result<(), &'static str> {
let nb = t.edges.len();
if nb == 0 || t.cdfs.len() != nb || t.cdf_lengths.len() != nb || t.offsets.len() != nb {
return Err("one Gaussian table, length and offset per scale edge, and at least one edge");
}
let nz = t.z_cdfs.len();
if nz == 0 || t.z_cdf_lengths.len() != nz || t.z_offsets.len() != nz || t.z_median_q.len() != nz {
return Err("one z table, length, offset and median per channel");
}
let all = t.cdfs.iter().zip(&t.cdf_lengths).chain(t.z_cdfs.iter().zip(&t.z_cdf_lengths));
if all.into_iter().any(|(c, &l)| !table_ok(c, l)) {
return Err("a table is not a CDF from 0 to 65536, strictly increasing over its length");
}
Ok(())
}
impl CondEntropy {
/// Parse and check the entropy tables (`cond_entropy.json`, read from a set as
/// `entropy.json`): every member present with integers in range, the lengths
/// consistent and every table well formed. Refuses rather than panics.
pub fn try_from_value(v: &Value) -> Result<Self, &'static str> {
fn arr<'a>(v: &'a Value, k: &str) -> Result<&'a Vec<Value>, &'static str> {
v.get(k).and_then(Value::as_array).ok_or("an entropy member is missing or not an array")
}
fn ints<T: TryFrom<i64>>(v: &Value, k: &str) -> Result<Vec<T>, &'static str> {
arr(v, k)?.iter().map(|x| x.as_i64().and_then(|i| T::try_from(i).ok())).collect::<Option<_>>()
.ok_or("an entropy member holds a value out of range")
}
fn rows(v: &Value, k: &str) -> Result<Vec<Vec<i32>>, &'static str> {
arr(v, k)?.iter().map(|r| r.as_array().ok_or("an entropy table is not an array")
.and_then(|r| r.iter().map(|x| x.as_i64().and_then(|i| i32::try_from(i).ok())).collect::<Option<_>>()
.ok_or("an entropy table holds a value out of range"))).collect()
}
let t = CondEntropy {
edges: ints(v, "scale_edges")?,
cdfs: rows(v, "cdfs")?,
cdf_lengths: ints(v, "cdf_lengths")?,
offsets: ints(v, "offsets")?,
z_cdfs: rows(v, "z_cdfs")?,
z_cdf_lengths: ints(v, "z_cdf_lengths")?,
z_offsets: ints(v, "z_offsets")?,
z_median_q: ints(v, "z_median_q")?,
};
check_entropy(&t)?;
Ok(t)
}
}
/// A conditional bitstream's fields, borrowed: what a `WIH1` payload carries.
#[derive(Debug, Clone, Copy)]
pub struct CondStream<'a> {
pub m: usize,
pub sh: usize,
pub sw: usize,
pub n_z: usize,
pub zh: usize,
pub zw: usize,
pub z_rans: &'a [u8],
pub rans: &'a [u8],
}
impl CondBitstream {
/// The borrowed form [`decode_conditional_checked`] reads.
pub fn stream(&self) -> CondStream<'_> {
CondStream { m: self.m, sh: self.sh, sw: self.sw, n_z: self.n_z, zh: self.zh, zw: self.zw, z_rans: &self.z_rans, rans: &self.rans }
}
}
/// A network's output on a `[c, h, w]` input, planned from its layers' geometry and
/// parameter lengths alone, before anything is allocated or any value read
/// (`extensions/integer-payloads.md` §7). The input must be within [`MAX_TENSOR`]
/// (`TooLarge`). Then, layer by layer in order:
///
/// 1. the layer fits the running channel count (`ModelMismatch`): a convolution's
/// output channels, kernel and stride are non-zero, its weights number
/// `c·out·k·k` and its multipliers and biases `out`, and its requantisation shift is
/// under 128; an inverse GDN's `gamma` numbers `c·c` and its `beta` `c`, and its
/// shifts are under 128; a leaky ReLU's shift is under 64;
/// 2. every size the layer forms is within [`MAX_TENSOR`] (`TooLarge`): a
/// convolution's padded input side `n + 2·pad`, a transposed convolution's uncropped
/// side `(n − 1)·stride + k + opad` and uncropped plane, and the output;
/// 3. the layer leaves an output (`ModelMismatch`): a kernel no larger than the padded
/// input, a crop `2·pad` smaller than the uncropped side.
///
/// Every size is formed in `u64` with checked arithmetic, a size past `u64` being over
/// the cap, so a 32-bit sink refuses exactly what a 64-bit one does, with the same
/// kind; and once a network is planned, every size [`run_net_checked`] forms fits
/// `usize` on every target. The first rule that fails is the refusal.
pub(crate) fn plan_net(net: &IntModel, c: usize, h: usize, w: usize) -> Result<NetPlan, Refused> {
plan_layers(net, c, h, w)
}
/// What [`plan_net`] finds about a network on a `[c, h, w]` input, in elements.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct NetPlan {
/// The output `[c, h, w]`.
pub out: (usize, usize, usize),
/// The largest single buffer the network forms: its input, a layer's output, a
/// transposed convolution's uncropped plane.
pub largest: u64,
/// The most elements the reference kernels hold at once while one layer runs, its
/// input included: a transposed convolution's input, uncropped plane and output; a
/// convolution's input and output; an inverse GDN's input, output and one
/// position's squares (`c` values of 16 bytes, counted as `2·c`); a ReLU's input,
/// rewritten in place. Each element is an `i64`.
pub live: u64,
/// The multiply-accumulates the layers comprise (`extensions/integer-payloads.md`
/// §7): a convolution's every output times `c·k·k` (every tap of its padded
/// input), a transposed convolution's every input times `out·k·k`, an inverse
/// GDN's every element times `c`. A kernel may skip a zero term, so it performs
/// at most this many. Summed in `u128`: a layer's terms are its weights times at
/// most `2^26` positions, so no set a sink can hold reaches the type's limit.
pub macs: u128,
}
fn plan_layers(net: &IntModel, c: usize, h: usize, w: usize) -> Result<NetPlan, Refused> {
use RefusalKind::{ModelMismatch, TooLarge};
const CAP: u64 = MAX_TENSOR as u64;
let big = |what: &'static str| refused(TooLarge, what);
let within = |v: Option<u64>, what: &'static str| v.filter(|&n| n <= CAP).ok_or(big(what));
let (mut c, mut h, mut w) = (c as u64, h as u64, w as u64);
if c == 0 || h == 0 || w == 0 {
return Err(refused(ModelMismatch, "a network's input is empty"));
}
let input = within(c.checked_mul(h).and_then(|v| v.checked_mul(w)), "a network's input exceeds 2^26 elements")?;
let (mut largest, mut live) = (input, input);
let mut macs: u128 = 0;
let params = |cout: u64, k: u64, weight: usize, mult: usize, bias: usize, c: u64| {
let n = c.checked_mul(cout).and_then(|v| v.checked_mul(k)).and_then(|v| v.checked_mul(k));
n == Some(weight as u64) && mult as u64 == cout && bias as u64 == cout
};
for layer in &net.layers {
match layer {
IntLayer::ConvT { out_ch, k, stride, pad, opad, weight, mult, bias } => {
let (cout, k, s, p, op) = (*out_ch as u64, *k as u64, *stride as u64, *pad as u64, *opad as u64);
if cout == 0 || k == 0 || s == 0 || !params(cout, k, weight.len(), mult.len(), bias.len(), c) || net.shift > 127 {
return Err(refused(ModelMismatch, "a transposed convolution does not fit the running channel count"));
}
let full = |n: u64| (n - 1).checked_mul(s).and_then(|v| v.checked_add(k)).and_then(|v| v.checked_add(op));
let (fh, fw) = (within(full(h), "a layer's uncropped side exceeds 2^26")?, within(full(w), "a layer's uncropped side exceeds 2^26")?);
let plane = within(cout.checked_mul(fh).and_then(|v| v.checked_mul(fw)), "a layer's uncropped plane exceeds 2^26 elements")?;
let crop = p.checked_mul(2).unwrap_or(u64::MAX);
if fh <= crop || fw <= crop {
return Err(refused(ModelMismatch, "a transposed convolution's padding leaves no output"));
}
let a = c * h * w;
macs = macs.saturating_add(a as u128 * (cout * k * k) as u128);
(c, h, w) = (cout, fh - crop, fw - crop);
let b = c * h * w;
largest = largest.max(plane).max(b);
live = live.max(a + plane + b);
}
IntLayer::Conv { out_ch, k, stride, pad, weight, mult, bias } => {
let (cout, k, s, p) = (*out_ch as u64, *k as u64, *stride as u64, *pad as u64);
if cout == 0 || k == 0 || s == 0 || !params(cout, k, weight.len(), mult.len(), bias.len(), c) || net.shift > 127 {
return Err(refused(ModelMismatch, "a convolution does not fit the running channel count"));
}
let padded = |n: u64| p.checked_mul(2).and_then(|v| v.checked_add(n));
let (hp, wp) = (within(padded(h), "a layer's padded side exceeds 2^26")?, within(padded(w), "a layer's padded side exceeds 2^26")?);
if hp < k || wp < k {
return Err(refused(ModelMismatch, "a convolution's kernel is larger than its padded input"));
}
let (ho, wo) = ((hp - k) / s + 1, (wp - k) / s + 1);
let b = within(cout.checked_mul(ho).and_then(|v| v.checked_mul(wo)), "a layer's output exceeds 2^26 elements")?;
live = live.max(c * h * w + b);
largest = largest.max(b);
macs = macs.saturating_add(b as u128 * (c * k * k) as u128);
(c, h, w) = (cout, ho, wo);
}
IntLayer::Igdn { gamma, beta } => {
if c.checked_mul(c) != Some(gamma.len() as u64) || beta.len() as u64 != c || net.gf > 127 || net.afrac > 127 {
return Err(refused(ModelMismatch, "an inverse GDN does not fit the running channel count"));
}
live = live.max(2 * c * h * w + 2 * c);
macs = macs.saturating_add((c * h * w) as u128 * c as u128);
}
IntLayer::Relu => {}
IntLayer::Leaky { slope_frac, .. } => {
if *slope_frac > 63 {
return Err(refused(ModelMismatch, "a leaky ReLU's shift is 64 or more"));
}
}
}
}
// Every running size is at most 2^26 here, so it fits usize on every target.
Ok(NetPlan { out: (c as usize, h as usize, w as usize), largest, live, macs })
}
/// [`run_net_checked`] with its refusal as a [`Refused`]: values that could overflow a
/// layer are the stream's (`FieldRange`); a layer that does not fit is the model's.
fn run_checked(net: &IntModel, x: &[i64], c: usize, h: usize, w: usize) -> Result<Vec<i64>, Refused> {
run_net_checked(net, x, c, h, w).map(|(o, ..)| o).map_err(|r| {
if r.out_of_range {
refused(RefusalKind::FieldRange, "a decoded latent could take a layer outside the exact integer range")
} else {
refused(RefusalKind::ModelMismatch, "a network's parameters do not fit the running shapes")
}
})
}
/// [`decode_conditional`], refusing instead of panicking or overflowing on any input:
/// the decode the `WIH1` payload (`wai.neural.int_hyper`, and the keyframe of
/// `wai.video.int_hyper`) runs. On a stream [`decode_conditional`] decodes, both give
/// the same image.
///
/// Before anything is allocated, in this order: the networks share one fixed-point
/// scale in `1..=`[`MAX_AFRAC`]; the entropy model is consistent and has a `z` table
/// for each of the payload's channels; the latents are within [`MAX_TENSOR`]; `h_s` on
/// the hyper-latent's shape, then `g_s` on `[m, sh, sw]`, are planned ([`plan_net`]: each
/// layer fits, forms nothing over the cap, and leaves an output); then `h_s` must give
/// `[2·m, sh, sw]` (`GeometryMismatch`) and `g_s` three channels. Then the hyper-latent
/// is decoded symbol by symbol (a symbol that runs the stream out is refused before
/// its value is rebuilt with [`dequantize`]), its stream must end canonically, `h_s`
/// runs through [`run_net_checked`], the latent is decoded the same way, and `g_s`
/// runs. Returns RGB HWC with its height and width.
pub fn decode_conditional_checked(
g_s: &IntModel, h_s: &IntModel, ent: &CondEntropy, bs: &CondStream<'_>,
) -> Result<(Vec<u8>, usize, usize), Refused> {
use crate::int_entropy::dequantize;
use RefusalKind::*;
let p = cond_plan(g_s, h_s, ent, bs)?;
let (afrac, n_z, zh, zw, m, sh, sw) = (g_s.afrac, bs.n_z, bs.zh, bs.zw, bs.m, bs.sh, bs.sw);
let (zplane, n, (oc, oh, ow)) = (zh * zw, m * sh * sw, p.gs.out);
// 1. factorized-decode z, rebuild z_q = s·2^afrac + median_q. The latent grows a
// symbol at a time, to its declared length and no further, and the first symbol
// that runs its stream out refuses the payload before its value is used.
let mut z = Vec::new();
let mut dz = RansDecoder::new(bs.z_rans).map_err(|_| refused(FieldRange, "the z stream is not a rANS stream"))?;
for ch in 0..n_z {
for _ in 0..zplane {
let s = dz.decode(&ent.z_cdfs[ch], ent.z_cdf_lengths[ch], ent.z_offsets[ch]);
if dz.is_exhausted() {
return Err(refused(StreamExhausted, "the z stream ended before its last symbol"));
}
push_capped(&mut z, dequantize(s, afrac, ent.z_median_q[ch]).ok_or(refused(FieldRange, "a hyper-latent value leaves i64"))?, n_z * zplane);
}
}
if !dz.is_canonical_end() {
return Err(refused(StreamNoncanonical, "the z stream does not end canonically after its last symbol"));
}
let sm = run_checked(h_s, &z, n_z, zh, zw)?;
drop(z);
let (sigma, mu) = sm.split_at(n);
// 2. y, conditioned on the integer σ: y_hat = r·2^afrac + μ.
let mut dec = RansDecoder::new(bs.rans).map_err(|_| refused(FieldRange, "the y stream is not a rANS stream"))?;
let mut y_hat = Vec::new();
for p in 0..n {
let b = det_build_index(sigma[p], &ent.edges);
let r = dec.decode(&ent.cdfs[b], ent.cdf_lengths[b], ent.offsets[b]);
if dec.is_exhausted() {
return Err(refused(StreamExhausted, "the y stream ended before its last symbol"));
}
push_capped(&mut y_hat, dequantize(r, afrac, mu[p]).ok_or(refused(FieldRange, "a latent value leaves i64"))?, n);
}
if !dec.is_canonical_end() {
return Err(refused(StreamNoncanonical, "the y stream does not end canonically after its last symbol"));
}
drop(sm);
// 3. integer synthesis g_s → pixels.
let x = run_checked(g_s, &y_hat, m, sh, sw)?;
drop(y_hat);
Ok((to_rgb8(&x, oc, oh, ow, afrac), oh, ow))
}
/// Append `x` to `v`, a buffer that will hold `total` elements, growing it by doubling
/// but never past `total`: a decode holds no more than it declared, and no buffer is
/// reserved ahead of the symbols that fill it.
pub(crate) fn push_capped<T>(v: &mut Vec<T>, x: T, total: usize) {
if v.len() == v.capacity() {
let want = v.capacity().saturating_mul(2).max(64).min(total).max(v.len() + 1);
v.reserve_exact(want - v.len());
}
v.push(x);
}
/// What [`decode_conditional_checked`] checks before it decodes a symbol, in its
/// order, and the buffers it then forms: the networks share one fixed-point scale in
/// `1..=`[`MAX_AFRAC`]; the entropy model is consistent and has a `z` table for each
/// of the payload's channels; the latents are within [`MAX_TENSOR`]; `h_s` on the
/// hyper-latent's shape, then `g_s` on `[m, sh, sw]`, are planned ([`plan_net`]); then `h_s`
/// must give `[2·m, sh, sw]` (`GeometryMismatch`) and `g_s` three channels.
pub(crate) fn cond_plan(g_s: &IntModel, h_s: &IntModel, ent: &CondEntropy, bs: &CondStream<'_>) -> Result<CondPlan, Refused> {
use RefusalKind::*;
let afrac = g_s.afrac;
if h_s.afrac != afrac || !(1..=MAX_AFRAC).contains(&afrac) {
return Err(refused(ModelMismatch, "g_s and h_s do not share one fixed-point scale in 1..=30"));
}
check_entropy(ent).map_err(|e| refused(ModelMismatch, e))?;
let (n_z, zh, zw, m, sh, sw) = (bs.n_z, bs.zh, bs.zw, bs.m, bs.sh, bs.sw);
if [n_z, zh, zw, m, sh, sw].contains(&0) {
return Err(refused(FieldRange, "a latent dimension is zero"));
}
if n_z > ent.z_cdfs.len() {
return Err(refused(ModelMismatch, "the hyper-latent has more channels than the entropy model has z tables"));
}
let size = |d: &[usize]| d.iter().try_fold(1usize, |a, &x| a.checked_mul(x)).filter(|&n| n <= MAX_TENSOR);
if size(&[n_z, zh, zw]).is_none() || size(&[2, m, sh, sw]).is_none() {
return Err(refused(TooLarge, "a latent exceeds 2^26 elements"));
}
// Both networks are planned, caps included, before either output shape is
// compared with the payload's.
let hs = plan_net(h_s, n_z, zh, zw)?;
let gs = plan_net(g_s, m, sh, sw)?;
if hs.out != (2 * m, sh, sw) {
return Err(refused(GeometryMismatch, "h_s does not give the latent shape the payload declares"));
}
if gs.out.0 != 3 {
return Err(refused(ModelMismatch, "g_s does not give three channels"));
}
Ok(CondPlan { hs, gs, z: (n_z * zh * zw) as u64, y: (m * sh * sw) as u64 })
}
/// The buffers [`decode_conditional_checked`] forms on a payload [`cond_plan`] admits.
#[derive(Debug, Clone, Copy)]
pub(crate) struct CondPlan {
pub hs: NetPlan,
pub gs: NetPlan,
/// The hyper-latent's and the latent's elements: the symbols each stream codes.
pub z: u64,
pub y: u64,
}
impl CondPlan {
/// The image's bytes (RGB8).
pub fn output_bytes(&self) -> u64 {
let (c, h, w) = self.gs.out;
(c * h * w) as u64
}
/// The most bytes the decode holds at once on the reference kernels, beyond the
/// payload and the parsed set: the hyper-latent while `h_s` runs; `h_s`'s output
/// (`[2·m, sh, sw]`) while the latent is decoded; the latent while `g_s` runs; and
/// `g_s`'s output while it becomes RGB8. Every value is an `i64`.
pub fn working_bytes(&self) -> u64 {
let gs_out = self.output_bytes();
[8 * (self.z + self.hs.live), 8 * (2 * self.y + self.y), 8 * (self.y + self.gs.live), 8 * gs_out + gs_out]
.into_iter().max().unwrap_or(0)
}
/// The networks' multiply-accumulates: `h_s`'s, then `g_s`'s ([`NetPlan::macs`]).
pub fn macs(&self) -> u128 {
self.hs.macs.saturating_add(self.gs.macs)
}
}
/// The fast kernels against the scalar reference, element by element. Operand
/// ranges cover the `i32` fast path, the `i64` path, and sparse (zero-heavy)
/// inputs; geometries cover strides, paddings, output paddings and kernels smaller
@ -1118,6 +1602,60 @@ mod checked {
assert!(ran > 150, "too few nets passed the checks ({ran})");
}
/// `NetPlan::macs` is every layer's terms at the shapes the network runs at:
/// counted here one term at a time — a convolution's every output and every tap of
/// every input channel, a transposed convolution's every input and every tap of
/// every output channel, an inverse GDN's every element and every channel — over
/// the shapes `apply_layer` forms, not the plan's.
#[test]
fn the_planned_macs_are_every_layers_terms() {
let mut g = Rng(0x5EED_0003);
let mut ran = 0;
for _ in 0..200 {
let cin = 1 + g.below(4);
let (model, _) = random_net(&mut g, cin);
let (h, w) = (1 + g.below(6), 1 + g.below(6));
let x = g.vec(cin * h * w, 1 << 16, 20);
if run_net_checked(&model, &x, cin, h, w).is_err() {
continue;
}
let plan = plan_net(&model, cin, h, w).expect("the checked run ran, so the plan admits it");
let (mut a, mut c, mut hh, mut ww) = (x, cin, h, w);
let mut terms: u128 = 0;
for layer in &model.layers {
let (na, nc, nh, nw) = apply_layer(&model, layer, a, c, hh, ww);
match layer {
IntLayer::Conv { k, .. } => {
for _output in 0..nc * nh * nw {
for _tap in 0..c * k * k {
terms += 1;
}
}
}
IntLayer::ConvT { out_ch, k, .. } => {
for _input in 0..c * hh * ww {
for _tap in 0..out_ch * k * k {
terms += 1;
}
}
}
IntLayer::Igdn { .. } => {
for _element in 0..c * hh * ww {
for _channel in 0..c {
terms += 1;
}
}
}
IntLayer::Relu | IntLayer::Leaky { .. } => {}
}
(a, c, hh, ww) = (na, nc, nh, nw);
}
assert_eq!(plan.macs, terms);
ran += 1;
}
assert!(ran > 150, "too few nets passed the checks ({ran})");
}
#[test]
fn a_convolution_runs_up_to_the_accumulation_bound_and_no_further() {
// One term: the bound is |w|·|x|. Exactly i64::MAX runs; one past refuses.

8
wai/wai-rs/src/int_mlicpp.rs
View file

@ -361,6 +361,14 @@ pub struct Trace {
pub symbols: Vec<i32>,
}
/// Parse the `WMB1` bundle `bundle` and decode the `WIM1` payload with it: the
/// byte-input entry of the integer decoders (`int_payload`, the browser crate). A pin
/// over the bundle is verified before this is called (`pin::verify_bytes`,
/// `int_payload::resolve`), so it takes no pin.
pub fn decode_bytes(payload: &[u8], bundle: &[u8]) -> Result<IntMlicImage, IntMlicError> {
decode(&IntMlicModel::from_bundle(bundle)?, payload)
}
/// Decode a `WIM1` payload with `model` → RGB. Byte-identical on every machine.
///
/// Before it allocates anything, the decode derives from the model's layer geometry

829
wai/wai-rs/src/int_payload/mod.rs Normal file
View file

@ -0,0 +1,829 @@
//! Payloads of the WAI-native integer codecs
//! ([`extensions/integer-payloads.md`](../../../extensions/integer-payloads.md)) and
//! the one integer decode path every sink shares.
//!
//! SPEC §5 pins each of these capabilities' payload to a binary format
//! ([`crate::codecs::PINNED_INT_PAYLOADS`]): `WIV1` for `wai.video.int_motion`
//! (keyframe 0) and `wai.video.int_hyper` (keyframe 1, a nested `WIH1`), `WIH1` for
//! `wai.neural.int_hyper`, `WIA1` for `wai.audio.int_codec` and `WIS1` for
//! `wai.splat.int_codec`. [`wire`] parses and writes them. The JSON the export and
//! encode tools write is a converter input ([`wiv1_from_json`], [`wih1_from_json`],
//! [`wia1_from_json`], [`wis1_from_json`]) and a debugging aid, never a payload: a
//! payload that is not the pinned format is refused (`bad_magic`).
//!
//! [`decode_int`] decodes every integer capability, these five and
//! `wai.neural.int_synth` and `wai.neural.int_mlicpp`, from the parameter files a
//! [`IntParams`] holds. It needs no ONNX runtime and no filesystem, so it builds under
//! `neural_int` and its media features alone, natively and for `wasm32`; the neural
//! sink (`neural::decode_payload`, feature `neural`) delegates these capabilities to
//! it. [`resolve`] verifies a pin over sets held in memory, running SPEC §3.1's check
//! 5 when the sink holds a key log, and runs the same check over the set an unpinned
//! decode reads (SPEC §4, *Module authorisation*).
//!
//! Every refusal is a code from the closed list of the extension's §8
//! ([`Refusal`]). A refusal of the payload or of the parameter set against it is final:
//! nothing is decoded in its place, and SPEC §4's fallback, which would decode the same
//! bytes, is not tried. `set_incomplete`, `not_dispatchable` and
//! `unsupported_capability` say the sink cannot decode through the capability, and
//! dispatch moves on as SPEC §4 says (the extension's §8).
pub use crate::int_wire as wire;
#[cfg(test)]
mod tests;
pub use wire::{Refusal, WireError};
use crate::codecs::{
CAP_AUDIO_INT_CODEC, CAP_NEURAL_INT_HYPER, CAP_NEURAL_INT_MLICPP, CAP_NEURAL_INT_SYNTH, CAP_SPLAT_INT_CODEC,
CAP_VIDEO_INT_HYPER,
};
use crate::container::{PinRefusal, Prior};
use crate::pin::{HeldBytes, PinAuthority, Verified, VerifiedRef};
/// Why [`decode_int`] or a converter refused: a code of the extension's closed list,
/// and what failed.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct IntPayloadError {
refusal: Refusal,
detail: String,
}
impl IntPayloadError {
pub fn new(refusal: Refusal, detail: impl Into<String>) -> Self {
IntPayloadError { refusal, detail: detail.into() }
}
/// The refusal.
pub fn refusal(&self) -> Refusal {
self.refusal
}
/// The code the extension's §8 names.
pub fn code(&self) -> &'static str {
self.refusal.code()
}
/// What failed. For [`Refusal::SetIncomplete`], the path of the file the set lacks.
pub fn detail(&self) -> &str {
&self.detail
}
}
impl std::fmt::Display for IntPayloadError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}: {}", self.refusal.code(), self.detail)
}
}
impl std::error::Error for IntPayloadError {}
impl From<WireError> for IntPayloadError {
fn from(e: WireError) -> Self {
IntPayloadError::new(e.refusal, e.detail)
}
}
impl From<crate::int_hyper_synth::Refused> for IntPayloadError {
fn from(r: crate::int_hyper_synth::Refused) -> Self {
use crate::int_hyper_synth::RefusalKind as K;
let refusal = match r.kind {
K::FieldRange => Refusal::FieldRange,
K::TooLarge => Refusal::TooLarge,
K::CdfMalformed => Refusal::CdfMalformed,
K::StreamExhausted => Refusal::StreamExhausted,
K::StreamNoncanonical => Refusal::StreamNoncanonical,
K::GeometryMismatch => Refusal::GeometryMismatch,
K::ModelMismatch => Refusal::ModelMismatch,
};
IntPayloadError::new(refusal, r.detail)
}
}
fn model(detail: impl Into<String>) -> IntPayloadError {
IntPayloadError::new(Refusal::ModelMismatch, detail)
}
/// The files a capability's decode reads from its parameter set, by the names a set
/// lists them under (SPEC §5; `extensions/prior-carriage.md` §7). Empty for a
/// capability that decodes against no set, or that this module does not decode.
pub fn set_members(cap: &str) -> &'static [&'static str] {
match cap {
CAP_NEURAL_INT_HYPER | CAP_VIDEO_INT_HYPER => &["entropy.json", "g_s.whm", "h_s.whm"],
CAP_AUDIO_INT_CODEC => &["model.wam"],
CAP_SPLAT_INT_CODEC => &["g_s.whm"],
CAP_NEURAL_INT_SYNTH => &["int_synth.json"],
CAP_NEURAL_INT_MLICPP => &["model.wmb"],
_ => &[],
}
}
/// The parameter set one integer decode reads, in memory.
///
/// A pinned decode reads only bytes its pin verified: what `pin::verify` returned (a
/// native sink), or the held set `pin::verify_bytes` matched and, for the form
/// `wai.prior.set`, the paths its listing names (a sink with no filesystem). An
/// unpinned decode reads the set the sink registers for the capability: only the
/// files check 5 was run over ([`resolve`]).
///
/// Only [`resolve`] builds one that holds parameters, so a decode never reads a set
/// that no pin verified while reporting it pinned, nor an unpinned set other than the
/// one check 5 identified. [`IntParams::none`] holds none.
#[derive(Debug, Clone)]
pub struct IntParams<'a>(Params<'a>);
#[derive(Debug, Clone)]
enum Params<'a> {
/// No parameter set: a capability that decodes against none (`wai.video.int_motion`).
None,
/// A set `pin::verify` verified, as the bytes that were hashed.
#[cfg_attr(not(feature = "neural"), allow(dead_code))]
Verified(&'a Verified),
/// A held set: one file, or a set whose readable files are `paths` (`None`: a
/// form-less pin's one file). `pinned` when a pin verified it.
Held { set: HeldBytes<'a>, paths: Option<Vec<String>>, pinned: bool },
}
impl<'a> IntParams<'a> {
/// No parameter set. A decode through a capability that reads one refuses with
/// `set_incomplete`.
pub fn none() -> IntParams<'static> {
IntParams(Params::None)
}
/// The set `pin::verify` returned for a pin over `cap`: the neural sink's pinned
/// decode, whose registry ran SPEC §3.1's checks.
#[cfg(feature = "neural")]
pub(crate) fn verified(v: &'a Verified) -> Self {
IntParams(Params::Verified(v))
}
/// The parameters a pin verified with `pin::verify_bytes` over `held`.
fn from_verified_ref(cap: &str, held: &[HeldBytes<'a>], r: &VerifiedRef) -> Result<Self, PinRefusal> {
let (index, paths) = match r {
VerifiedRef::File { index } => (*index, None),
VerifiedRef::Set { index, paths } => (*index, Some(paths.clone())),
};
let set = *held.get(index).ok_or_else(|| PinRefusal::NotHeld { capability: cap.to_string() })?;
Ok(IntParams(Params::Held { set, paths, pinned: true }))
}
/// Whether a pin verified these parameters.
pub fn is_pinned(&self) -> bool {
matches!(self.0, Params::Verified(_) | Params::Held { pinned: true, .. })
}
/// The members of `cap`'s set these parameters hold, by name: the bytes a decode
/// reads.
pub fn read(&self, cap: &str) -> std::collections::BTreeMap<String, &'a [u8]> {
set_members(cap).iter().filter_map(|m| self.member(cap, m).ok().map(|b| (m.to_string(), b))).collect()
}
/// Whether these parameters hold no set ([`IntParams::none`], or [`resolve`] for a
/// capability that decodes against none).
pub fn is_none(&self) -> bool {
matches!(self.0, Params::None)
}
/// The set member `name` that `cap`'s decode reads. A one-file set (a form-less
/// pin's, or a one-file capability's unpinned file) answers for the capability's
/// one member; a listed set answers for a path it lists. A file the set does not
/// hold is `set_incomplete`, naming it.
pub fn member(&self, cap: &str, name: &str) -> Result<&'a [u8], IntPayloadError> {
let one = set_members(cap) == [name];
let found = match &self.0 {
Params::None => None,
Params::Verified(Verified::File(b)) => one.then_some(b.as_slice()),
Params::Verified(v @ Verified::Set { .. }) => v.file(name).ok(),
Params::Held { set: HeldBytes::File(b), .. } => one.then_some(*b),
Params::Held { set: HeldBytes::Set { files, .. }, paths, .. } => {
if paths.as_ref().is_none_or(|p| p.iter().any(|x| x == name)) { files.get(name).copied() } else { None }
}
};
found.ok_or_else(|| IntPayloadError::new(Refusal::SetIncomplete, name))
}
}
/// What an unpinned decode of `cap` reads from `set`, and every digest a pin over
/// those bytes could name, for check 5 (SPEC §4, *Module authorisation*).
///
/// The decode reads the capability's members (`set_members`) and nothing else. Where
/// the set carries a listing, it must be well formed (`set_malformed`) and every file
/// it names must be held with its digest (`part_mismatch`), as for a pinned set (SPEC
/// §3.1 checks 7 and 8), and the decode reads only the members it lists. The digests:
/// the listing's, where there is one; the listing of the members the decode reads
/// (`pin::listing_from`), which is what a pin in the form `wai.prior.set` over exactly
/// those files names; and, for a one-file capability, its one file's. So an empty
/// listing does not hide the set: its digest is the one derived from the files read.
/// A set whose listing is malformed or does not match its files has no one identity,
/// so it is refused before check 5 runs.
pub(crate) fn unpinned_read(cap: &str, set: &HeldBytes<'_>) -> Result<(Vec<String>, Option<Vec<String>>), PinRefusal> {
use crate::pin::{parse_listing, sha256, sha256_hex};
let one_file = crate::container::pin_shape(cap) == crate::codecs::ParameterSetShape::OneFile;
let members = set_members(cap);
let mut digests = Vec::new();
let mut read: std::collections::BTreeMap<String, &[u8]> = std::collections::BTreeMap::new();
let paths = match set {
HeldBytes::File(b) => {
if one_file {
if let Some(m) = members.first() {
read.insert(m.to_string(), *b);
}
}
None
}
HeldBytes::Set { listing, files } => {
let listed: Option<Vec<String>> = if listing.is_empty() {
None
} else {
digests.push(sha256_hex(listing));
let lines = parse_listing(listing)?;
if let Some((path, _)) = lines.iter().find(|(path, d)| files.get(path).is_none_or(|b| sha256(b) != *d)) {
return Err(PinRefusal::PartMismatch { path: path.clone() });
}
Some(lines.into_iter().map(|(p, _)| p).collect())
};
for m in members {
if listed.as_ref().is_none_or(|l| l.iter().any(|p| p == m)) {
if let Some(b) = files.get(*m) {
read.insert(m.to_string(), *b);
}
}
}
Some(read.keys().cloned().collect())
}
};
digests.extend(read_identities(cap, &read));
digests.sort_unstable();
digests.dedup();
Ok((digests, paths))
}
/// Every digest a pin over the bytes a decode of `cap` reads, `read` (member name →
/// bytes), could name besides its own: the listing of those members
/// (`pin::listing_from`, what a pin in the form `wai.prior.set` over exactly them
/// names), and, for a one-file capability, its one file's SHA-256 (what a form-less
/// pin names). Check 5 runs over these, pinned or unpinned, so a set is refused under
/// any identity of the bytes the decode reads (SPEC §3.1 check 5).
pub fn read_identities(cap: &str, read: &std::collections::BTreeMap<String, &[u8]>) -> Vec<String> {
use crate::pin::{listing_from, sha256_hex};
let mut out = Vec::new();
if crate::container::pin_shape(cap) == crate::codecs::ParameterSetShape::OneFile {
if let Some(b) = set_members(cap).first().and_then(|m| read.get(*m)) {
out.push(sha256_hex(b));
}
}
if let Ok(l) = listing_from(read) {
out.push(sha256_hex(&l));
}
out
}
/// The parameters `cap`'s decode reads, at a sink that holds the sets `held` in memory
/// and, where it has one, the key log `authority` (SPEC §3.1, §4). The one way to build
/// [`IntParams`] that hold a set.
///
/// - A capability that decodes against no parameter set reads none.
/// - Where a pin applies and the envelope carries one, `pin::verify_bytes_with` runs
/// SPEC §3.1's checks in order, check 5 included where `authority` is given, and the
/// decode reads the bytes it verified. Where `authority` is given, check 5 then runs
/// over every identity of those bytes as well ([`read_identities`]) and the pin's
/// digest.
/// - Otherwise the decode is unpinned and reads `held[0]`, the set the sink registers
/// for the capability, as [`unpinned_read`] states: a listing the set carries must
/// match its files, and the decode reads only the capability's members (only those
/// the listing names, where there is one). Where `authority` is given, check 5 runs
/// over every digest a pin over the bytes read could name, and refuses a revoked or
/// unauthorised set.
pub fn resolve<'a>(
cap: &str,
prior: Option<&Prior>,
held: &[HeldBytes<'a>],
authority: Option<&dyn PinAuthority>,
) -> Result<IntParams<'a>, PinRefusal> {
if crate::codecs::is_known_capability(cap) && !crate::codecs::decodes_against_parameter_set(cap) {
return Ok(IntParams(Params::None));
}
match prior {
Some(p) if crate::container::pin_applies_to(cap) => {
let r = crate::pin::verify_bytes_with(cap, p, held, crate::pin::READABLE_FORMS, authority)?;
let params = IntParams::from_verified_ref(cap, held, &r)?;
// Check 5 ran over the pin's digest; it runs again over every identity of the
// bytes the decode will read, so a set revoked (or not granted) under its
// file's digest or under the listing of its members is not reached through
// a pin that names it another way.
if let Some(a) = authority {
let mut ids = read_identities(cap, &params.read(cap));
ids.push(p.sha256.clone());
a.check_unpinned(cap, &ids)?;
}
Ok(params)
}
_ => {
let set = *held.first().ok_or_else(|| PinRefusal::NotHeld { capability: cap.to_string() })?;
let (digests, paths) = unpinned_read(cap, &set)?;
if let Some(a) = authority {
a.check_unpinned(cap, &digests)?;
}
Ok(IntParams(Params::Held { set, paths, pinned: false }))
}
}
}
/// What an integer decode gives. Pixels are RGB8, rows top to bottom, pixels left to
/// right; PCM is `i16`, channels interleaved.
#[derive(Debug, Clone, PartialEq)]
#[non_exhaustive]
pub enum IntDecoded {
/// `wai.neural.int_hyper`, `wai.neural.int_synth`, `wai.neural.int_mlicpp`.
Image { rgb: Vec<u8>, width: u32, height: u32 },
/// `wai.video.int_motion`, `wai.video.int_hyper`: every frame, in order.
Video { frames: Vec<Vec<u8>>, width: u32, height: u32 },
/// `wai.audio.int_codec`: the PCM, its sample rate and channel count.
Pcm { samples: Vec<i16>, sample_rate: u32, channels: u32 },
/// `wai.splat.int_codec`: the `[n_attr, s, s]` attribute tensor (the
/// splat-equivalence digest's buffer), the cloud's Gaussian count, and each
/// attribute's presentation range `(lo, hi)`, which the digest does not cover.
Attr { bytes: Vec<u8>, n_attr: u32, s: u32, n_gaussians: u32, ranges: Vec<(f32, f32)> },
}
/// Decode `payload` through the integer capability `cap` against `params`.
///
/// The capability must be dispatchable (SPEC §4, `codecs::is_dispatchable`). For the
/// five capabilities SPEC §5 pins a format to, the payload must be that format
/// (`bad_magic` otherwise; a JSON form is refused here, before any parameter is
/// read) and is checked as the extension's §2–§7 give, in that order; the decode then
/// reads its set members, checks that they fit the payload, and runs the checked
/// integer decoder. `wai.neural.int_synth` and `wai.neural.int_mlicpp` decode through
/// their own decoders. A capability this build has no decoder for is
/// `unsupported_capability`.
pub fn decode_int(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
if !crate::codecs::is_dispatchable(cap) {
return Err(IntPayloadError::new(Refusal::NotDispatchable, format!("{cap} is not dispatchable (SPEC \u{a7}4)")));
}
match crate::codecs::pinned_int_payload(cap) {
Some(wire::MAGIC_WIV1) => decode_wiv1(cap, payload, params),
Some(wire::MAGIC_WIH1) => decode_wih1(cap, payload, params),
Some(wire::MAGIC_WIA1) => decode_wia1(cap, payload, params),
Some(wire::MAGIC_WIS1) => decode_wis1(cap, payload, params),
_ => match cap {
CAP_NEURAL_INT_SYNTH => {
let spec = params.member(cap, "int_synth.json")?;
let img = crate::int_synth::decode_from_spec(payload, spec).map_err(|e| {
let crate::int_synth::IntSynthError::InvalidPayload(why) = e;
IntPayloadError::new(Refusal::PayloadMalformed, why)
})?;
Ok(IntDecoded::Image { rgb: img.rgb, width: img.width, height: img.height })
}
CAP_NEURAL_INT_MLICPP => {
use crate::int_mlicpp::IntMlicError;
let mlic = |e: IntMlicError| match e {
IntMlicError::Payload(why) => IntPayloadError::new(Refusal::PayloadMalformed, why),
IntMlicError::Model(why) => model(why),
};
let m = crate::int_mlicpp::IntMlicModel::from_bundle(params.member(cap, "model.wmb")?).map_err(mlic)?;
let img = crate::int_mlicpp::decode(&m, payload).map_err(mlic)?;
Ok(IntDecoded::Image { rgb: img.rgb, width: img.width as u32, height: img.height as u32 })
}
_ => Err(unsupported(cap, "no integer decoder for this capability")),
},
}
}
/// What decoding a payload will cost, read before any symbol is decoded, from the
/// header and the parameter set's layer geometry alone (`extensions/integer-payloads.md`
/// §7, *Memory and work*). The header declares the output, and the caps bound it; a
/// stream can make a large output from few bytes (a table can make one symbol nearly
/// free), so it is the header, not the stream's length, that bounds the work. A sink
/// applies its own size policy to this and refuses a payload past it before
/// decoding.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[non_exhaustive]
pub struct DeclaredCost {
/// The decoded output's bytes: frames (`n_frames·h·w·3`), an image (`h·w·3`), PCM
/// (`2·t_out`), an attribute tensor (`attr·s·s`).
pub output_bytes: u64,
/// The symbols the payload's streams code: one entropy-decode step each.
pub symbols: u64,
/// The most bytes the reference decode (`decode_int` on the reference kernels,
/// `int_kernels::Policy::reference()`) holds at once, beyond the payload and the
/// parameter set as parsed: the payload's tables as parsed, the output, the
/// latents and network activations (8 bytes per element), each network layer's
/// input, output and, for a transposed convolution, its uncropped plane, and a
/// clip's frame list. The fast kernels
/// trade the uncropped plane for a padded copy of the layer's input, at most 16
/// times the elements the layer reads and writes, and per-worker rows.
pub working_bytes: u64,
/// The multiply-accumulates the decode's networks comprise, layer by layer: a
/// convolution's every output times `c·k·k`, a transposed convolution's every
/// input times `out·k·k` (`out·k` in the audio synthesis), an inverse GDN's every
/// element times `c` (§7). Exact, and 0 for a `WIV1` with keyframe 0, which runs
/// no network. A kernel may skip a zero term, so it performs at most this many.
pub macs: u128,
}
impl DeclaredCost {
/// A cost from its four figures: what a sink gives [`check_cost`] (and the neural
/// registry's `with_cost_limit`) as its limit. A figure the sink does not bound is
/// its type's maximum.
pub const fn new(output_bytes: u64, symbols: u64, working_bytes: u64, macs: u128) -> Self {
Self { output_bytes, symbols, working_bytes, macs }
}
/// The first figure of this cost over `limit`'s, by name, or `None` where every
/// figure is within it.
pub fn exceeds(&self, limit: &DeclaredCost) -> Option<&'static str> {
if self.output_bytes > limit.output_bytes {
Some("output bytes")
} else if self.symbols > limit.symbols {
Some("symbols")
} else if self.working_bytes > limit.working_bytes {
Some("working bytes")
} else if self.macs > limit.macs {
Some("multiply-accumulates")
} else {
None
}
}
}
/// A sink's cost limit (§7, *Memory and work*), applied before any symbol is decoded:
/// [`declared_cost`], with its refusal where the payload or the set fails a check it
/// reads, and `cost_over_limit` where the cost exceeds `limit` in any figure. A
/// capability with no declared cost (`wai.neural.int_synth`, `wai.neural.int_mlicpp`)
/// has nothing to check, `Ok(None)`; it is decoded as before.
pub fn check_cost(cap: &str, payload: &[u8], params: &IntParams<'_>, limit: &DeclaredCost) -> Result<Option<DeclaredCost>, IntPayloadError> {
if crate::codecs::pinned_int_payload(cap).is_none() {
return Ok(None);
}
let cost = declared_cost(cap, payload, params)?;
match cost.exceeds(limit) {
None => Ok(Some(cost)),
Some(what) => Err(IntPayloadError::new(Refusal::CostOverLimit, format!("{cap}: the declared {what} exceed the sink's limit"))),
}
}
/// [`DeclaredCost`] for `payload` under `cap` against `params`: the parse and every
/// check a decode makes before its first symbol (the set's members and networks
/// planned), with the refusal a decode would give, then the sizes those checks fix.
/// A `WIV1` with keyframe 1 is also refused here when its keyframe's size is not the
/// clip's (`geometry_mismatch`), as a decode refuses it before it decodes the keyframe.
/// So a payload the decode would refuse before its first symbol is refused here with
/// the same code. Only the five
/// capabilities SPEC §5 pins a format to have a declared cost
/// (`unsupported_capability` otherwise).
pub fn declared_cost(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
if !crate::codecs::is_dispatchable(cap) {
return Err(IntPayloadError::new(Refusal::NotDispatchable, format!("{cap} is not dispatchable (SPEC \u{a7}4)")));
}
match crate::codecs::pinned_int_payload(cap) {
Some(wire::MAGIC_WIV1) => cost_wiv1(cap, payload, params),
Some(wire::MAGIC_WIH1) => {
let k = wire::parse_wih1(payload)?;
let p = cond_cost(cap, &k, params)?;
Ok(DeclaredCost { output_bytes: p.output_bytes(), symbols: p.z + p.y, working_bytes: p.working_bytes(), macs: p.macs() })
}
Some(wire::MAGIC_WIA1) => cost_wia1(cap, payload, params),
Some(wire::MAGIC_WIS1) => cost_wis1(cap, payload, params),
_ => Err(unsupported(cap, "no declared cost: its payload format is not one SPEC \u{a7}5 pins here")),
}
}
/// The bytes of one `Vec`'s header on this target, for a table of tables.
#[cfg(any(feature = "audio_int", feature = "splat_int"))]
const VEC: u64 = std::mem::size_of::<Vec<i32>>() as u64;
/// The conditional codec's plan for a parsed `WIH1` against the capability's set.
fn cond_cost(cap: &str, k: &wire::Wih1<'_>, params: &IntParams<'_>) -> Result<crate::int_hyper_synth::CondPlan, IntPayloadError> {
let (g_s, h_s, ent) = int_hyper_models(cap, params)?;
let s = crate::int_hyper_synth::CondStream {
m: k.m as usize, sh: k.sh as usize, sw: k.sw as usize,
n_z: k.n_z as usize, zh: k.zh as usize, zw: k.zw as usize,
z_rans: k.z_rans, rans: k.rans,
};
Ok(crate::int_hyper_synth::cond_plan(&g_s, &h_s, &ent, &s)?)
}
#[cfg(feature = "video_int")]
fn cost_wiv1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
use crate::int_video::{check, working_bytes, VideoStream};
let keyframe = if cap == crate::codecs::CAP_VIDEO_INT_MOTION { 0 } else { 1 };
let v = wire::parse_wiv1(payload, Some(keyframe))?;
let s = VideoStream {
n_frames: v.n_frames as usize, h: v.h as usize, w: v.w as usize, block: v.block as usize,
q: v.q as i64, offset: v.offset, mv_precision: v.mv_precision, cdf: &v.cdf, mvs: &v.mvs, rans: v.rans,
};
check(&s).map_err(|e| IntPayloadError::new(Refusal::FieldRange, e))?;
// The parsed table and motion vectors, held for the whole decode.
let tables = 4 * v.cdf.len() as u64 + 4 * v.mvs.len() as u64;
let frame = v.h as u64 * v.w as u64 * 3;
let output_bytes = frame * v.n_frames as u64;
match v.wih1 {
None => Ok(DeclaredCost { output_bytes, symbols: output_bytes, working_bytes: v.keyframe0_working_bytes(), macs: 0 }),
Some(nested) => {
let p = cond_cost(cap, &wire::parse_wih1(nested)?, params)?;
if p.gs.out != (3, v.h as usize, v.w as usize) {
return Err(IntPayloadError::new(Refusal::GeometryMismatch, "the keyframe's size is not the clip's h × w"));
}
// The keyframe's decode, then the clip with the keyframe's image beside it.
let working = p.working_bytes().max(working_bytes(&s) + frame);
Ok(DeclaredCost { output_bytes, symbols: frame * (v.n_frames as u64 - 1) + p.z + p.y, working_bytes: tables + working, macs: p.macs() })
}
}
}
#[cfg(not(feature = "video_int"))]
fn cost_wiv1(cap: &str, _: &[u8], _: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
Err(unsupported(cap, "this build has no integer video decoder (feature video_int)"))
}
#[cfg(feature = "audio_int")]
fn cost_wia1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
use crate::int_audio::{audio_plan, AudioModel, AudioStream};
let a = wire::parse_wia1(payload)?;
let m = AudioModel::from_bytes(params.member(cap, "model.wam")?).map_err(|e| model(format!("model.wam: {e}")))?;
let s = AudioStream {
afrac: a.afrac as u32, n_ch: a.n_ch as usize, lat_len: a.lat_len as usize, t_out: a.t_out as usize,
offset: a.offset, median_q: &a.median_q, cdfs: &a.cdfs, rans: a.rans,
};
let (_, working, macs) = audio_plan(&m, &s)?;
// The parsed medians and tables, held for the whole decode.
let tables = a.n_ch as u64 * (8 + VEC + 4 * a.cdfs[0].len() as u64);
Ok(DeclaredCost { output_bytes: 2 * a.t_out as u64, symbols: a.n_ch as u64 * a.lat_len as u64, working_bytes: tables + working, macs })
}
#[cfg(not(feature = "audio_int"))]
fn cost_wia1(cap: &str, _: &[u8], _: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
Err(unsupported(cap, "this build has no integer audio decoder (feature audio_int)"))
}
#[cfg(feature = "splat_int")]
fn cost_wis1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
use crate::int_hyper_synth::IntModel;
use crate::int_splat::{splat_plan, SplatStream};
let v = wire::parse_wis1(payload)?;
let gs = IntModel::from_bytes(params.member(cap, "g_s.whm")?).map_err(|e| model(format!("g_s.whm: {e}")))?;
let s = SplatStream {
afrac: v.afrac as u32, c: v.c as usize, sh: v.sh as usize, sw: v.sw as usize, s: v.s as usize,
n_attr: v.attr as usize, cdfs: &v.cdfs, offsets: &v.offsets, median_q: &v.median_q, rans: v.rans,
};
let (working, macs) = splat_plan(&gs, &s)?;
// The parsed ranges, offsets, medians and tables, held for the whole decode.
let cdfs: u64 = v.cdfs.iter().map(|t| 4 * t.len() as u64).sum();
let tables = 8 * v.attr as u64 + v.c as u64 * (4 + 8 + VEC) + cdfs;
Ok(DeclaredCost { output_bytes: v.attr as u64 * v.s as u64 * v.s as u64, symbols: v.c as u64 * v.sh as u64 * v.sw as u64, working_bytes: tables + working, macs })
}
#[cfg(not(feature = "splat_int"))]
fn cost_wis1(cap: &str, _: &[u8], _: &IntParams<'_>) -> Result<DeclaredCost, IntPayloadError> {
Err(unsupported(cap, "this build has no integer splat decoder (feature splat_int)"))
}
fn unsupported(cap: &str, why: &str) -> IntPayloadError {
IntPayloadError::new(Refusal::UnsupportedCapability, format!("{cap}: {why}"))
}
/// The conditional codec's three set members, parsed and checked.
fn int_hyper_models(
cap: &str,
params: &IntParams<'_>,
) -> Result<(crate::int_hyper_synth::IntModel, crate::int_hyper_synth::IntModel, crate::int_hyper_synth::CondEntropy), IntPayloadError> {
use crate::int_hyper_synth::{CondEntropy, IntModel};
let net = |name: &str| -> Result<IntModel, IntPayloadError> {
IntModel::from_bytes(params.member(cap, name)?).map_err(|e| model(format!("{name}: {e}")))
};
let g_s = net("g_s.whm")?;
let h_s = net("h_s.whm")?;
let ent: serde_json::Value = serde_json::from_slice(params.member(cap, "entropy.json")?)
.map_err(|e| model(format!("entropy.json: {e}")))?;
let ent = CondEntropy::try_from_value(&ent).map_err(|e| model(format!("entropy.json: {e}")))?;
Ok((g_s, h_s, ent))
}
/// A parsed `WIH1` decoded to RGB with the capability's set. A `WIV1` keyframe passes
/// the clip's `h × w` as `size`: the keyframe's planned image is checked against it
/// (`geometry_mismatch`) before any of its symbols is decoded (§7).
fn wih1_image(cap: &str, k: &wire::Wih1<'_>, params: &IntParams<'_>, size: Option<(u32, u32)>) -> Result<(Vec<u8>, u32, u32), IntPayloadError> {
let (g_s, h_s, ent) = int_hyper_models(cap, params)?;
let s = crate::int_hyper_synth::CondStream {
m: k.m as usize, sh: k.sh as usize, sw: k.sw as usize,
n_z: k.n_z as usize, zh: k.zh as usize, zw: k.zw as usize,
z_rans: k.z_rans, rans: k.rans,
};
if let Some((h, w)) = size {
let p = crate::int_hyper_synth::cond_plan(&g_s, &h_s, &ent, &s)?;
if p.gs.out != (3, h as usize, w as usize) {
return Err(IntPayloadError::new(Refusal::GeometryMismatch, "the keyframe's size is not the clip's h × w"));
}
}
let (rgb, h, w) = crate::int_hyper_synth::decode_conditional_checked(&g_s, &h_s, &ent, &s)?;
Ok((rgb, h as u32, w as u32))
}
fn decode_wih1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
let k = wire::parse_wih1(payload)?;
let (rgb, height, width) = wih1_image(cap, &k, params, None)?;
Ok(IntDecoded::Image { rgb, width, height })
}
#[cfg(feature = "video_int")]
fn decode_wiv1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
use crate::int_video::{decode_pframes_typed, decode_typed, VideoStream, STREAM_EXHAUSTED, STREAM_NONCANONICAL};
let keyframe = if cap == crate::codecs::CAP_VIDEO_INT_MOTION { 0 } else { 1 };
let v = wire::parse_wiv1(payload, Some(keyframe))?;
let s = VideoStream {
n_frames: v.n_frames as usize, h: v.h as usize, w: v.w as usize, block: v.block as usize,
q: v.q as i64, offset: v.offset, mv_precision: v.mv_precision, cdf: &v.cdf, mvs: &v.mvs, rans: v.rans,
};
let video = |e: &'static str| {
let refusal = match e {
STREAM_EXHAUSTED => Refusal::StreamExhausted,
STREAM_NONCANONICAL => Refusal::StreamNoncanonical,
_ => Refusal::FieldRange,
};
IntPayloadError::new(refusal, e)
};
let clip = match v.wih1 {
None => decode_typed(&s).map_err(video)?,
Some(nested) => {
let k = wire::parse_wih1(nested)?;
let (frame0, ..) = wih1_image(cap, &k, params, Some((v.h, v.w)))?;
decode_pframes_typed(&frame0, &s).map_err(video)?
}
};
Ok(IntDecoded::Video { frames: clip.frames, width: v.w, height: v.h })
}
#[cfg(not(feature = "video_int"))]
fn decode_wiv1(cap: &str, _: &[u8], _: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
Err(unsupported(cap, "this build has no integer video decoder (feature video_int)"))
}
#[cfg(feature = "audio_int")]
fn decode_wia1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
use crate::int_audio::{decode_typed, AudioModel, AudioStream};
let a = wire::parse_wia1(payload)?;
let m = AudioModel::from_bytes(params.member(cap, "model.wam")?).map_err(|e| model(format!("model.wam: {e}")))?;
let s = AudioStream {
afrac: a.afrac as u32, n_ch: a.n_ch as usize, lat_len: a.lat_len as usize, t_out: a.t_out as usize,
offset: a.offset, median_q: &a.median_q, cdfs: &a.cdfs, rans: a.rans,
};
let samples = decode_typed(&m, &s)?;
Ok(IntDecoded::Pcm { samples, sample_rate: a.sr, channels: 1 })
}
#[cfg(not(feature = "audio_int"))]
fn decode_wia1(cap: &str, _: &[u8], _: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
Err(unsupported(cap, "this build has no integer audio decoder (feature audio_int)"))
}
#[cfg(feature = "splat_int")]
fn decode_wis1(cap: &str, payload: &[u8], params: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
use crate::int_hyper_synth::IntModel;
use crate::int_splat::{decode_typed, SplatStream};
let v = wire::parse_wis1(payload)?;
let gs = IntModel::from_bytes(params.member(cap, "g_s.whm")?).map_err(|e| model(format!("g_s.whm: {e}")))?;
let s = SplatStream {
afrac: v.afrac as u32, c: v.c as usize, sh: v.sh as usize, sw: v.sw as usize, s: v.s as usize,
n_attr: v.attr as usize, cdfs: &v.cdfs, offsets: &v.offsets, median_q: &v.median_q, rans: v.rans,
};
let d = decode_typed(&gs, &s)?;
Ok(IntDecoded::Attr { bytes: d.attr, n_attr: v.attr, s: v.s, n_gaussians: v.n_gaussians, ranges: v.ranges })
}
#[cfg(not(feature = "splat_int"))]
fn decode_wis1(cap: &str, _: &[u8], _: &IntParams<'_>) -> Result<IntDecoded, IntPayloadError> {
Err(unsupported(cap, "this build has no integer splat decoder (feature splat_int)"))
}
// ───────────────────────── converters (JSON tool output → payload) ─────────────────────────
/// A converter's refusal of its JSON input.
fn json(why: &str) -> IntPayloadError {
IntPayloadError::new(Refusal::FieldRange, format!("the JSON form: {why}"))
}
#[cfg(any(feature = "video_int", feature = "audio_int", feature = "splat_int"))]
fn to_u32(v: usize, what: &str) -> Result<u32, IntPayloadError> {
u32::try_from(v).map_err(|_| json(&format!("{what} exceeds u32")))
}
#[cfg(any(feature = "video_int", feature = "audio_int", feature = "splat_int"))]
fn table_len(len: usize) -> Result<(), IntPayloadError> {
if !(wire::CDF_LEN_MIN..=wire::CDF_LEN_MAX).contains(&len) {
return Err(json("a CDF table's length is outside 3..=4097"));
}
Ok(())
}
/// Convert the video encoders' JSON bitstream to `WIV1`: keyframe 0, or keyframe 1
/// with `keyframe_wih1` (a `WIH1` payload) as its nested I-frame. The JSON's implicit
/// rules are made explicit as [`crate::int_video::VideoJson`] states; `q` must be in
/// `1..=65535`. The result is parsed before it is returned, so a converter never
/// emits a payload a sink would refuse at parse.
#[cfg(feature = "video_int")]
pub fn wiv1_from_json(bs: &serde_json::Value, keyframe_wih1: Option<&[u8]>) -> Result<Vec<u8>, IntPayloadError> {
let j = crate::int_video::VideoJson::from_value(bs).map_err(json)?;
let q = u32::try_from(j.q).ok().filter(|q| (1..=wire::MAX_Q).contains(q)).ok_or_else(|| json("q is outside 1..=65535"))?;
table_len(j.cdf.len())?;
// A one-frame clip carries no motion vector: its header is written with
// mv_precision 1 and block 1, the values the format fixes for it.
let one = j.n_frames == 1;
let v = wire::Wiv1 {
mv_precision: if one { 1 } else { j.mv_precision },
keyframe: keyframe_wih1.is_some() as u8,
n_frames: to_u32(j.n_frames, "n_frames")?,
h: to_u32(j.h, "h")?,
w: to_u32(j.w, "w")?,
block: if one { 1 } else { to_u32(j.block, "block")? },
q,
offset: j.offset,
cdf: j.cdf.clone(),
mvs: j.mvs.clone(),
rans: &j.rans,
wih1: keyframe_wih1,
};
let out = wire::write_wiv1(&v);
wire::parse_wiv1(&out, None)?;
Ok(out)
}
/// Convert the conditional codec's JSON bitstream (`cond_bitstream.json`) to `WIH1`.
pub fn wih1_from_json(bs: &serde_json::Value) -> Result<Vec<u8>, IntPayloadError> {
let dim = |k: &str| bs[k].as_u64().and_then(|v| u32::try_from(v).ok()).ok_or_else(|| json(&format!("`{k}` is missing or out of range")));
let bytes = |k: &str| -> Result<Vec<u8>, IntPayloadError> {
bs[k].as_array().and_then(|a| a.iter().map(|b| b.as_u64().and_then(|b| u8::try_from(b).ok())).collect::<Option<Vec<u8>>>())
.ok_or_else(|| json(&format!("`{k}` is not an array of bytes")))
};
let (z_rans, rans) = (bytes("z_rans")?, bytes("rans")?);
let v = wire::Wih1 {
m: dim("m")?, sh: dim("sh")?, sw: dim("sw")?, n_z: dim("n_z")?, zh: dim("zh")?, zw: dim("zw")?,
z_rans: &z_rans, rans: &rans,
};
let out = wire::write_wih1(&v);
wire::parse_wih1(&out)?;
Ok(out)
}
/// Convert the audio export's JSON bitstream (`audio_bitstream.json`) to `WIA1` at
/// the sample rate `sr` (the export's `audio.json` carries it as `sample_rate`; the
/// bitstream does not). The JSON's implicit rules are made explicit as
/// [`crate::int_audio::AudioJson`] states.
#[cfg(feature = "audio_int")]
pub fn wia1_from_json(bs: &serde_json::Value, sr: u32) -> Result<Vec<u8>, IntPayloadError> {
let j = crate::int_audio::AudioJson::from_value(bs).map_err(json)?;
if j.median_q.len() != j.n_ch || j.cdfs.len() != j.n_ch {
return Err(json("the medians and tables are not one per channel (n_ch)"));
}
table_len(j.cdfs[0].len())?;
let v = wire::Wia1 {
afrac: u8::try_from(j.afrac).map_err(|_| json("afrac exceeds a byte"))?,
sr,
n_ch: to_u32(j.n_ch, "n_ch")?,
lat_len: to_u32(j.lat_len, "lat_len")?,
t_out: to_u32(j.t_out, "t_out")?,
offset: j.offset,
median_q: j.median_q.clone(),
cdfs: j.cdfs.clone(),
rans: &j.rans,
};
let out = wire::write_wia1(&v);
wire::parse_wia1(&out)?;
Ok(out)
}
/// Convert the splat export's JSON bitstream (`splat.json`) to `WIS1`, with its
/// per-attribute presentation ranges. The export writes each range as a binary32
/// value; a range that is not exactly one is refused, so the payload carries what the
/// JSON states. `splat_int` reads JSON numbers correctly rounded
/// (`serde_json/float_roundtrip`), so a decimal that denotes a binary32 value reads as
/// exactly that value, and no tolerance is needed.
#[cfg(feature = "splat_int")]
pub fn wis1_from_json(bs: &serde_json::Value) -> Result<Vec<u8>, IntPayloadError> {
let j = crate::int_splat::SplatJson::from_value(bs).map_err(json)?;
if j.cdfs.len() != j.c || j.offsets.len() != j.c || j.median_q.len() != j.c {
return Err(json("the tables, offsets and medians are not one per channel (c)"));
}
for t in &j.cdfs {
table_len(t.len())?;
}
let exact = |x: f64| -> Result<f32, IntPayloadError> {
let f = x as f32;
if f as f64 == x { Ok(f) } else { Err(json("a range is not exactly a binary32 value")) }
};
let ranges = j.ranges.iter().map(|&(lo, hi)| Ok((exact(lo)?, exact(hi)?))).collect::<Result<Vec<_>, IntPayloadError>>()?;
let v = wire::Wis1 {
afrac: u8::try_from(j.afrac).map_err(|_| json("afrac exceeds a byte"))?,
c: to_u32(j.c, "c")?,
sh: to_u32(j.sh, "sh")?,
sw: to_u32(j.sw, "sw")?,
s: to_u32(j.s, "s")?,
attr: to_u32(j.n_attr, "attr")?,
n_gaussians: to_u32(j.n_gaussians, "n_gaussians")?,
ranges,
cdfs: j.cdfs.clone(),
offsets: j.offsets.clone(),
median_q: j.median_q.clone(),
rans: &j.rans,
};
let out = wire::write_wis1(&v);
wire::parse_wis1(&out)?;
Ok(out)
}

Some files were not shown because too many files have changed in this diff Show more