From 08b156829198931c13201cc2ca171b602f5ae975 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 11:55:53 -0400 Subject: [PATCH 01/15] wai: int_synth's weights are quantised after float training, and two engine paths moved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SPEC §5 and determinism-tiers §5 said `wai.neural.int_synth` decodes "QAT weights". No committed model was trained that way. The trained corpus vector was made by `tools/wai_int_synth_real.py`: float training, then post-training quantisation with each layer's shift derived from its measured activation range, the recipe `tools/wai_int_synth_train.py` documents. The synthetic vectors' weights are generated from a seed. Both texts, the capability constant's comment, the module doc and a test comment now say that. `tools/wai_int_synth_qat.py` said it "closes the one open gap" and tunes "the weights AND the shifts"; it trains the weights against one fixed shift and made no committed model, and it now says so. `tools/wai_int_synth_export.py` told users to train with QAT upstream; it now names the working path. Stale paths: the int_synth decoder is `int_synth.rs` at the crate root, not `neural/int_synth.rs` (SPEC §5, the export tool, the capability comment); the quantum-ops engine is `wai-quantum/src/quantum_ops.rs`, and the quantum-sim receipt's feature is `quantum_receipt` of `wai-quantum`, which `wai-rs` forwards, not `["quantum", "world"]`. Co-Authored-By: Claude Opus 5.5 --- wai/SPEC.md | 2 +- wai/extensions/determinism-tiers.md | 2 +- wai/extensions/quantum-ops.md | 2 +- wai/extensions/quantum-sim.md | 8 +++++--- wai/extensions/ternary-tt.md | 3 ++- wai/tools/wai_int_synth_export.py | 17 ++++++++++------- wai/tools/wai_int_synth_qat.py | 22 +++++++++++++--------- wai/wai-rs/src/codecs/mod.rs | 2 +- wai/wai-rs/src/int_synth.rs | 11 +++++++---- wai/wai-rs/src/ternary.rs | 4 ++-- wai/wai-rs/tests/neural_native_decode.rs | 2 +- 11 files changed, 44 insertions(+), 31 deletions(-) diff --git a/wai/SPEC.md b/wai/SPEC.md index a38e0a598..8756b766e 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -601,7 +601,7 @@ registration and makes no conforming envelope non-conforming (§8). |---|---|---|---| | `wai.neural.bmshj2018` | bmshj2018-factorized + rANS (byte-exact CompressAI port) | about 0.26 bpp at 29 dB PSNR (quality 3) | CompressAI | | `wai.neural.mbt2018_mean` | mean-scale hyperprior (Gaussian-conditional entropy) | the conditional-entropy family (arXiv 1809.02736) | `neural/hyperprior.rs` | -| `wai.neural.int_synth` | integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) | bit-identical on every machine — soundly backs `intent = replicate` | `neural/int_synth.rs` (QAT weights sink-supplied) | +| `wai.neural.int_synth` | integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) | bit-identical on every machine — soundly backs `intent = replicate` | `int_synth.rs` (integer weights sink-supplied; the reference tools make them by training in float, then quantising after training with per-layer shifts calibrated on measured activations, `tools/wai_int_synth_train.py`) | | `wai.neural.int_hyper` | integer fixed-point **conditional** codec — whole bitstream→image path integer (integer hyper-decoder σ → integer-σ entropy → integer synthesis) | the integer-entropy contract fully assembled: a PTQ of mbt2018_mean, bit-identical on every machine, whose integer decode CI holds within 1 dB of the same model's float decode on a pinned photograph | payload `WIH1`; `int_hyper_synth.rs` (tables sink-supplied) | | `wai.neural.int_mlicpp` | integer fixed-point **MLIC++ entropy topology** — `K` channel slices of the latent, each decoded as a checkerboard anchor pass then a non-anchor pass, with an entropy network over concat[latent decoded so far, hyper] re-run before every pass to give σ‖μ; hyper-decoder, entropy network and synthesis all integer | the channel-autoregressive × checkerboard entropy model at **decode-equivalence**: the encoder and every decoder compute the same σ, so the same distribution for every element, at any thread count; bit-identical on every machine. The committed conformance vector has synthetic weights, so it demonstrates decode-equivalence and states no quality figure | `int_mlicpp.rs` (one-file model bundle sink-supplied; `tools/wai_int_mlicpp_export.py`) | | `wai.image.jpegai` | JPEG AI (ISO/IEC 6048-1:2025) codestream → RGB — integer entropy front end (mANS, with an integer scale-index derivation) + 16-bit-PTQ integer hyper-decoder, context-model and synthesis networks + dequantisation, latent refinement, post-filters (the eICCI CNN among them) and colour conversion in IEEE-754 float with a fixed evaluation order. Scope: one configuration — 256×256 4:4:4, `model_id` 0 (tools_0), `beta_displacement_log` 0, the high-operation-point synthesis, the eICCI model pair the weight bundle carries, and the other header conditions in [determinism-tiers](extensions/determinism-tiers.md) §5; both decode paths refuse any other codestream. Within it, a sink without the integer weight bundle, or in a non-default floating-point environment, takes a float ONNX decode instead. Only checkpoint constants are sink-supplied | **entropy-consistency** (registry class `NeuralEntropyExact`): the entropy decode, including the scale-index derivation, is integer on both decode paths, so the decoded symbols reproduce on every machine. The exact decoder also pins one reconstruction to RGB, reproducible across machines under the build and floating-point conditions in [determinism-tiers](extensions/determinism-tiers.md) §5–§6 (it checks the floating-point environment before decoding) — a property of that decoder, not of the capability, so `wai.image.jpegai` does not back `replicate` (§7). On one reference codestream the exact decoder stays within max\|Δ\| ≤ 2 of the reference decoder (recorded max\|Δ\| = 1 on ~4.4% of samples); evidence in [determinism-tiers](extensions/determinism-tiers.md) §6. **Downgraded in v0.4.0:** this row previously claimed decode-equivalence, and the registry classed the capability `StandardDefined` | `jpegai_decode.rs` (+ `jpegai_{bitstream,mans,pipeline,synthesis,icci}.rs`) | diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index 9297817b4..33313c0a0 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -355,7 +355,7 @@ tier costs them. | capability | hashable medium | how the reconstruction is pinned | |---|---|---| -| `wai.neural.int_synth` | RGB | integer synthesis (QAT weights) | +| `wai.neural.int_synth` | RGB | integer synthesis (post-training quantisation of a float model, with per-layer shifts calibrated on measured activations) | | `wai.neural.int_hyper` | RGB | integer hyper-decoder, integer scale → distribution step, integer synthesis (post-training quantisation of a float model) | | `wai.video.int_hyper` | per-frame | an `int_hyper` keyframe, then integer motion-compensated P-frames | | `wai.audio.int_codec` | int16 PCM | integer synthesis over a factorized rANS latent | diff --git a/wai/extensions/quantum-ops.md b/wai/extensions/quantum-ops.md index 49804e95e..0df00f997 100644 --- a/wai/extensions/quantum-ops.md +++ b/wai/extensions/quantum-ops.md @@ -1,6 +1,6 @@ # WAI Extension: Quantum-Operations Attestation (`wai.quantum.calibration` · `wai.quantum.job`) -Status: **Draft.** Engine: `wai-rs/src/quantum_ops.rs` (feature `quantum_ops`). +Status: **Draft.** Engine: `wai-quantum/src/quantum_ops.rs` (feature `quantum_ops`, which `wai-rs` forwards under the same name and re-exports as `wai::quantum_ops`). The **verify-half** of WAI (`wai.asset.provenance`'s lineage, specialized) applied to the quantum processing industry. diff --git a/wai/extensions/quantum-sim.md b/wai/extensions/quantum-sim.md index 0cab4fbe2..19ce7dffe 100644 --- a/wai/extensions/quantum-sim.md +++ b/wai/extensions/quantum-sim.md @@ -6,7 +6,8 @@ > and the reconstructed statevector hashes identically on every machine. > Conformance is **statevector-equivalence** (and, for measured circuits, > **shot-histogram-equivalence**). Reference impl: the `quantum` feature of -> `wai-rs` (`crate::quantum`); corpus: `quantum-conformance/`; capability +> `wai-quantum` (`wai_quantum::quantum`), which `wai-rs` forwards and re-exports +> as `wai::quantum`; corpus: `quantum-conformance/`; capability > `wai.quantum.circuit`. Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY** are RFC 2119/8174. @@ -185,8 +186,9 @@ It is not a media codec and it is not a route to quantum advantage. ## 6. Receipt (JWP profile) -The receipt (`quantum_receipt.rs`, feature `quantum_receipt = ["quantum", -"world"]`) is the artifact this whole extension exists to make possible. Its +The receipt (`wai-quantum/src/quantum_receipt.rs`, feature `quantum_receipt`, +which `wai-rs` forwards under the same name) is the artifact this whole extension +exists to make possible. Its design principle: **separate the two halves of the reconstruction's cost so each is trusted the way it honestly can be.** diff --git a/wai/extensions/ternary-tt.md b/wai/extensions/ternary-tt.md index 23a88ec01..bc82af3e7 100644 --- a/wai/extensions/ternary-tt.md +++ b/wai/extensions/ternary-tt.md @@ -39,7 +39,8 @@ desyncs across machines exactly as a float σ desyncs a learned entropy coder. integer shift). So the entire *decode* path is `i64`/`i128` with no float, and a reconstruction is byte-identical on every machine. Encoder-side float (the SVD, the optimal per-column scale) runs once at encode and never at decode — the same -posture as QAT and as `wai.neural.int_hyper`. +posture as `wai.neural.int_hyper`, whose float model is quantised once, after +training, and never runs in float at decode. ## 2. Container (`WQTT`) diff --git a/wai/tools/wai_int_synth_export.py b/wai/tools/wai_int_synth_export.py index 267774d75..f32d2aefe 100644 --- a/wai/tools/wai_int_synth_export.py +++ b/wai/tools/wai_int_synth_export.py @@ -1,5 +1,5 @@ """Quantize a synthesis transform to the integer `int_synth.json` spec decoded -bit-exactly by `wai-rs/src/neural/int_synth.rs` — the integer-exact `replicate` +bit-exactly by `wai-rs/src/int_synth.rs` — the integer-exact `replicate` path. `int_synth` runs an all-integer transposed-conv stack (no float, no ONNX), so @@ -8,10 +8,13 @@ synthesis (ConvTranspose2d + ReLU layers) into the integer weights + per-layer requantize shifts the Rust decoder needs. IMPORTANT on quality: integer fixed-point inference is bit-exact regardless, but -to match the float model's *quality* the model should be **quantization-aware -trained** (QAT) for these shifts. A post-hoc round of an arbitrary float model -will decode deterministically but may look worse — the determinism is free, the -quality is the training step. This tool does the export; QAT is upstream. +this tool rounds every layer with ONE fixed shift, so a float model exported here +will decode deterministically and may look much worse than it does in float — the +determinism is free, the quality is the quantisation step. The working path is +`tools/wai_int_synth_train.py` (and `wai_int_synth_real.py`, which wrote the trained +corpus vector): float training, then post-training quantisation with each layer's +shift derived from its measured activation range. `wai_int_synth_qat.py` is an +experimental attempt to train through a fixed shift; no committed model came from it. Spec written (matches int_synth.rs): { "layers": [ { in_ch, out_ch, k, stride, pad, out_pad, @@ -73,7 +76,7 @@ def main() -> int: ap.add_argument("--out", required=True) ap.add_argument("--w-bits", type=int, default=8) ap.add_argument("--shift", type=int, default=8, - help="per-layer requantize right-shift; tune with QAT for quality.") + help="per-layer requantize right-shift (one value for every layer).") args = ap.parse_args() model = torch.load(args.weights, map_location="cpu", weights_only=False) @@ -93,7 +96,7 @@ def main() -> int: print(f"wrote {len(layers)}-layer integer synthesis → {out} " f"(w_bits={args.w_bits}, shift={args.shift})") print("determinism: bit-exact on any machine by construction; " - "quality depends on QAT for these shifts.") + "quality depends on the shift; see wai_int_synth_train.py for calibrated shifts.") return 0 diff --git a/wai/tools/wai_int_synth_qat.py b/wai/tools/wai_int_synth_qat.py index faa5c4b6a..a8776a560 100644 --- a/wai/tools/wai_int_synth_qat.py +++ b/wai/tools/wai_int_synth_qat.py @@ -1,13 +1,17 @@ -"""Quantization-aware training (QAT) of an integer synthesis for the -`wai.neural.int_synth` capability — closes the one open gap: integer-exact -decode is already proven (`neural/int_synth.rs`), this makes it look *good*. +"""An EXPERIMENTAL quantization-aware training (QAT) attempt for the +`wai.neural.int_synth` capability (decoder: `wai-rs/src/int_synth.rs`). No +committed model was made with it. `int_synth` runs an all-integer transposed-conv stack with a per-layer requantize `>> shift`. The decode is bit-exact on every machine regardless of -weights; QAT tunes the weights AND the shifts so the integer reconstruction -matches the float teacher. We train with straight-through fake-quant (forward = -quantized integers, backward = float gradients) so the network learns weights -that survive int8 + the chosen shifts. +weights. This tool trains the weights with straight-through fake-quant (forward = +quantized integers, backward = float gradients) against ONE fixed shift for every +layer; it does not learn the shifts. With a fixed shift, small activations can +collapse to 0 (see the comment in `IntSynth.forward`). The working path is +`tools/wai_int_synth_train.py`: train in +float, then quantise after training with each layer's shift derived from its +measured activation range. The trained corpus vector was made that way +(`tools/wai_int_synth_real.py`). Inputs: paired (latent, target_image) tensors — e.g. dump `(y_hat, x)` from a trained float codec on a dataset, or co-train with an analysis transform. This @@ -114,8 +118,8 @@ def main() -> int: out = Path(args.out); out.parent.mkdir(parents=True, exist_ok=True) out.write_text(json.dumps(net.eval().export_spec(), separators=(",", ":"))) - print(f"wrote QAT integer synthesis → {out} (shift={args.shift}); " - f"decode is bit-exact on any machine by construction.") + print(f"wrote an experimental QAT integer synthesis → {out} (shift={args.shift}); " + f"decode is bit-exact on any machine by construction, quality is not checked.") return 0 diff --git a/wai/wai-rs/src/codecs/mod.rs b/wai/wai-rs/src/codecs/mod.rs index e1c0f0c8f..7e1112607 100644 --- a/wai/wai-rs/src/codecs/mod.rs +++ b/wai/wai-rs/src/codecs/mod.rs @@ -60,7 +60,7 @@ pub const CAP_NEURAL_MIMI: &str = "wai.neural.mimi"; // Mimi, re pub const CAP_NEURAL_WAVTOKENIZER: &str = "wai.neural.wavtokenizer"; // ultra-low bitrate audio pub const CAP_NEURAL_BMSHJ2018: &str = "wai.neural.bmshj2018"; // bmshj2018-factorized image codec (2018 baseline) pub const CAP_NEURAL_MBT2018_MEAN: &str = "wai.neural.mbt2018_mean"; // mean-scale HYPERPRIOR image codec (Gaussian-conditional entropy) — decoder IMPLEMENTED in this crate (neural/hyperprior.rs); the conditional-entropy family: a mean-scale hyperprior conditions each latent's Gaussian; sink-supplied ONNX (hyper_synthesis + synthesis) -pub const CAP_NEURAL_INT_SYNTH: &str = "wai.neural.int_synth"; // INTEGER fixed-point synthesis (neural/int_synth.rs): all-i64 transposed-conv decode, no float/ONNX — bit-identical on every machine BY CONSTRUCTION. The first capability that can soundly back intent=replicate cross-hardware (determinism IS integer-exact). Weights are sink-supplied (int_synth.json from a QAT model) +pub const CAP_NEURAL_INT_SYNTH: &str = "wai.neural.int_synth"; // INTEGER fixed-point synthesis (int_synth.rs): all-i64 transposed-conv decode, no float/ONNX — bit-identical on every machine BY CONSTRUCTION. The first capability that can soundly back intent=replicate cross-hardware (determinism IS integer-exact). Weights are sink-supplied (int_synth.json; the reference tools train in float, then quantise after training with calibrated per-layer shifts, tools/wai_int_synth_train.py) pub const CAP_NEURAL_INT_HYPER: &str = "wai.neural.int_hyper"; // INTEGER fixed-point CONDITIONAL codec (int_hyper_synth.rs): the full Gaussian-conditional decode integer end-to-end — factorized rANS-decode z (z_q = s·2^afrac + median_q) → integer hyper-decoder h_s → σ,μ → integer-σ bucket (int_entropy) → integer rANS-decode residual → y_hat = r·2^afrac + μ → integer synthesis g_s. Both bitstreams entropy-coded, no raw latents; payload WIH1 (extensions/integer-payloads.md). No float/ONNX anywhere, so the WHOLE bitstream→image decode is bit-identical on every machine BY CONSTRUCTION — int_synth's sibling that adds the conditional entropy. Quality: PTQ of the pretrained mbt2018_mean (quality 6); CI (wai-sota-vectors.yml) requires the integer decode within 1 dB of the same model's float decode on a pinned test photograph — measured cost there 0.19 dB (35.985 vs 36.175 dB, arm64 macOS); image-dependent, see int_hyper_synth.rs. Tables sink-supplied pub const CAP_NEURAL_INT_MLICPP: &str = "wai.neural.int_mlicpp"; // INTEGER MLIC++-topology codec (int_mlicpp.rs): the channel-autoregressive × checkerboard entropy model — K channel slices, each decoded as an anchor pass then a non-anchor pass, with an entropy network over concat[y decoded so far, hyper] giving σ‖μ before every pass — with every decode-side network integer (int_hyper_synth networks, int_entropy bucketing, integer rANS). The encoder and every decoder compute the same σ, so the same bucket, at any thread count: no float/ONNX anywhere, byte-identical on every machine BY CONSTRUCTION. Refuses any stream value outside the exact integer range. Payload (WIM1) and one-file model bundle (WMB1, so a prior pins every network and table) specified in int_mlicpp.rs. Conformance vector tests/vectors/int_mlicpp/ has SYNTHETIC weights (seeded, quantised like a trained model): it shows decode-equivalence, not compression quality. Model sink-supplied pub const CAP_NEURAL_ELIC: &str = "wai.neural.elic"; // checkerboard CONTEXT-MODEL image codec (neural/context.rs) — the JPEG-AI/MLIC++ entropy class: hyperprior + 2-pass spatial context (anchor → context net → non-anchor). Decoder IMPLEMENTED; sink-supplied ONNX (hyper_synthesis + context + synthesis). WAI-native, NOT the ISO JPEG AI bitstream diff --git a/wai/wai-rs/src/int_synth.rs b/wai/wai-rs/src/int_synth.rs index 916f2fb3b..409c4f94c 100644 --- a/wai/wai-rs/src/int_synth.rs +++ b/wai/wai-rs/src/int_synth.rs @@ -17,10 +17,13 @@ //! Runtime present. The `neural` module re-exposes it through the //! `wai.neural.int_synth` capability. //! -//! Quality needs a quantization-aware-trained model (the weights are integers); -//! producing that model is a tooling step (`tools/wai_int_synth_export.py`). The -//! *decode contract* — the thing that makes cross-hardware bit-exactness real — -//! lives here and is exact regardless. +//! Quality depends on how the integer weights were made, which is a tooling step +//! outside the decode contract. The reference tools train in float, then quantise +//! after training, deriving each layer's shift from its measured activation range +//! (`tools/wai_int_synth_train.py`, `tools/wai_int_synth_real.py`); the trained +//! model in the conformance corpus was made that way. The *decode contract* — the +//! thing that makes cross-hardware bit-exactness real — lives here and is exact +//! whatever the weights. //! //! Wire format (little-endian): //! u32 H, u32 W, u16 C, u8 S_log2, u8 _reserved (12 B) diff --git a/wai/wai-rs/src/ternary.rs b/wai/wai-rs/src/ternary.rs index 0a37b2f4d..b527241de 100644 --- a/wai/wai-rs/src/ternary.rs +++ b/wai/wai-rs/src/ternary.rs @@ -17,8 +17,8 @@ //! byte-identical on every machine — a portable reconstruction hash you can put //! in a receipt — the same `NeuralIntegerExact` discipline as `int_hyper`, now //! for the quantum-inspired tensor-network transform. Encoder-side float (the -//! SVD, the optimal per-column scale) is fine and expected: like QAT it runs -//! once at encode and never at decode. +//! SVD, the optimal per-column scale) is fine and expected: like the float +//! training behind a quantised model, it runs once at encode and never at decode. //! //! This is a **lossy** codec whose decode is **deterministic**. It does not claim //! to recover an arbitrary source losslessly; it claims that whatever it *does* diff --git a/wai/wai-rs/tests/neural_native_decode.rs b/wai/wai-rs/tests/neural_native_decode.rs index 5ec8433c7..f17614894 100644 --- a/wai/wai-rs/tests/neural_native_decode.rs +++ b/wai/wai-rs/tests/neural_native_decode.rs @@ -1118,7 +1118,7 @@ fn snac_real_model_matches_reference() { assert!(maxd < 1e-3, "real snac decode diverges (max {maxd})"); } -/// int_synth against a REAL QAT/PTQ-trained model (not the constructed unit +/// int_synth against a REAL trained model, quantised after training (not the constructed unit /// test): a float autoencoder trained on synthetic images, then quantized to /// the integer spec with per-layer activation-calibrated shifts. Confirms the /// Rust integer decode is BIT-IDENTICAL to the numpy integer-forward reference -- 2.54.0 (Apple Git-157) From ce4646278f6c46b3421209d03c39c64bb323a7c7 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 12:05:30 -0400 Subject: [PATCH 02/15] =?UTF-8?q?wai:=20SPEC=20=C2=A76.5,=20protected=20co?= =?UTF-8?q?ntent=20in=20two=20lanes,=20and=20where=20the=20exact=20tier=20?= =?UTF-8?q?holds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A new normative section names the two lanes protected content travels in, and what a WAI record covers in each. - Lane A, hardware-protected content: a standard codec decrypted and decoded inside the platform's protected media path. This revision registers no `security.encryption` value for it, so no WAI envelope carries Lane A media. WAI's records about such a title (the served manifest, the package's segment map, the key-release decisions, the content hashes of WAI objects) are all it attests; none covers a decoded frame. Parametric synthesis applies only inside a decoder the protection system certifies, with no tier and no output digest. - Lane B, end-to-end encryption at the application layer (`mls-sframe`): the application decrypts and any decoder runs, so every record and tier applies as to plaintext. It is stated not to be a protected-path robustness level: a publisher must not offer it for a licence term that requires one. - The exact tier covers only output a sink can hash. determinism-tiers §5 gains a subsection listing the fabrics `decode-equivalence` is shown on in this revision (native CPU; wasm32, whose CI job passed on main at 412a08c5), none on GPU or NPU, forbids relaxed-SIMD instructions and NaN-bit dependence in a Wasm decoder at the tier, and rules out any tier, digest or binding for protected-path output. The `wai.video.keys` row and the confidentiality extension now say which lane they belong to. Two more stale references: README and SPEC Appendix A sent macOS builders to a committed `.cargo/config.toml` that no longer exists; `wai-rs/build.rs` finds libFLAC through pkg-config, then the Homebrew prefix. Co-Authored-By: Claude Opus 5.5 --- wai/README.md | 8 ++-- wai/SPEC.md | 70 +++++++++++++++++++++++++++-- wai/extensions/confidentiality.md | 4 ++ wai/extensions/determinism-tiers.md | 31 +++++++++++++ 4 files changed, 106 insertions(+), 7 deletions(-) diff --git a/wai/README.md b/wai/README.md index aa0d59a36..b8ad259b9 100644 --- a/wai/README.md +++ b/wai/README.md @@ -62,10 +62,10 @@ cargo run --no-default-features --bin wai -- capabilities # the registry, one r cargo build --release --lib # with the codecs feature: libwai.dylib/.so/.dll + libwai.a ``` -On macOS, libflac lives under the Homebrew "flac" formula prefix that -Cargo doesn't probe automatically. The committed `.cargo/config.toml` -adds the path; if you build outside the repo set -`RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"`. +With the `codecs` feature, `wai-rs/build.rs` finds libFLAC through pkg-config +and, on macOS without pkg-config, in the Homebrew "flac" formula prefix +(`/opt/homebrew/opt/flac/lib` or `/usr/local/opt/flac/lib`). Where neither finds +it, set `RUSTFLAGS="-L "`. ## SDK consumption diff --git a/wai/SPEC.md b/wai/SPEC.md index 8756b766e..a687d8530 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -625,7 +625,7 @@ registration and makes no conforming envelope non-conforming (§8). | `wai.video.int_hyper` | NEURAL-KEYFRAME inter-frame — the I-frame is the LEARNED integer codec `int_hyper` decoded byte-exact, the P-frames are integer block-MC residuals on top | a learned keyframe you can put in a receipt + integer inter-frames; whole decode integer, byte-identical everywhere | payload `WIV1`, keyframe 1 (a nested `WIH1`); `int_video::decode_pframes_typed` ∘ `int_hyper_synth::decode_conditional_checked` | | `wai.video.manifest` | ATTESTED PER-SESSION MANIFEST — the OTT delivery layer, not a codec: a personalised HLS manifest (ads stitched at `EXT-X-DISCONTINUITY`, blackouts + geo enforced) rendered as a **pure function of the session *class*** `(content, entitlement, geo, blackout set, ad pod, window)` rather than the viewer, and sealed in an Ed25519 receipt that binds the JCP grant (capability, joule ceiling, `funds_ceiling` = ad budget), the composition, the stitched ads and the policy applied. A receipt over a WAI playlist MAY carry `render` and `extras`, the rendering parameters and per-segment lines that playlist adds, both bound through `manifest_hash` and through a render digest the signature covers ([http-delivery](extensions/http-delivery.md) §8.6) | **manifest-equivalence**: `manifest_hash == hash(render(segments))` and `session_class_hash` are recomputable by anyone holding the bytes, so the ads, blackouts and compose-work stop being dashboard claims; class-determinism means one generation serves every viewer of the class, and `uj_per_viewer` is the amortised energy that a 1:1-unique generator cannot report below its per-viewer cost. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_manifest.rs` | | `wai.video.ssai` | ATTESTED SERVER-SIDE AD INSERTION — a **deterministic** ad decision (greedy by price within the avail duration and the funds remaining) plus a per-impression Ed25519 receipt binding the candidate pod, the manifest it belongs to, the served `creative_hash`es and the VAST quartile each reached | **spend-equivalence**: `budget_spent_micros` is recomputed from the *billable* impressions and rejected if it exceeds the grant's `funds_ceiling` — an overspent receipt fails verification; every creative and quartile is checkable, so a buyer and seller reconcile impressions from the same bytes instead of two logs that only a dashboard joins. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_ssai.rs` | -| `wai.video.keys` | ATTESTED CONTENT-KEY RELEASE — the DRM licensing decision as a record: every key release, and every refusal, is an Ed25519 receipt binding the KID, the DRM system, the entitlement demanded and the window enforced, with a BLAKE3 `key_commit` proving *which* key was released without carrying it | **decision-equivalence**: `verify()` re-runs the release policy over the receipt's own `(grant capability, entitlement, issued, expiry)` and rejects any misreported outcome, and enforces granted-iff-committed — so "keys went only to entitled, unexpired sessions" is checkable against the signed log, not asserted. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_keys.rs` | +| `wai.video.keys` | ATTESTED CONTENT-KEY RELEASE — the DRM licensing decision as a record: every key release, and every refusal, is an Ed25519 receipt binding the KID, the DRM system, the entitlement demanded and the window enforced, with a BLAKE3 `key_commit` proving *which* key was released without carrying it. It is one of Lane A's records (§6.5): it covers the release decision, never a frame the key opens | **decision-equivalence**: `verify()` re-runs the release policy over the receipt's own `(grant capability, entitlement, issued, expiry)` and rejects any misreported outcome, and enforces granted-iff-committed — so "keys went only to entitled, unexpired sessions" is checkable against the signed log, not asserted. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_keys.rs` | | `wai.video.package` | DETERMINISTIC PACKAGING + ATTESTED SEGMENTMAP — a source sliced into content-hashed segments on GOP boundaries; the ordered `(index → content_hash)` map is a signed Merkle commitment, and identical content (ads, bumpers, replays, simulcast overlap) dedups to one stored/encoded copy | **map-equivalence + verifiable dedup**: `segment_map_root` lets a `wai.video.manifest` reference be checked against the package that produced it, and `verify()` recomputes `unique_segments`/`stored_bytes` from the segments so the storage/encode saving cannot be overstated; `uj_per_unique_segment` is the encode-once energy. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable. Version 2 ([packaging](extensions/packaging.md) §9) binds the policy digest and one stage index per component, names each object by its jwp-receipts §2.1 content hash, and covers every media class a packager packs; the string keeps `video` for history | `wai_video_package.rs`, `package.rs` | | `wai.video.channel` | VIRTUAL LINEAR / FAST ASSEMBLY — a continuous schedule of assets over wall-clock time, sliced into attested playback windows; windows chain via `parent_receipt_hash`, so a channel's history is a hash chain of program-as-run records | **schedule-continuity + recomputable program-as-run**: `verify()` rejects a schedule with a gap or overlap (no dead air, no double-booking) and recomputes `aired` from the schedule and the window, so what actually aired is checkable by a licensor, advertiser or regulator against the signed chain rather than an operator-held play-out log. Only the measurement — `joules_micro`, its acquisition class and the class's evidence — is measured-attested: signed, and not re-measurable | `wai_video_channel.rs` | @@ -1062,6 +1062,9 @@ capability**. It wraps the payload bytes; the capability still names how to decode the *plaintext* once recovered. A sink decrypts first, then runs capability dispatch (§4) on the plaintext. +§6.1–§6.4 define that layer. §6.5 sets it beside hardware-protected content: the +two lanes protected content travels in, and what a WAI record covers in each. + ### 6.1 Manifest field A manifest MAY carry a `security` object: @@ -1153,6 +1156,66 @@ exporter → base key) crates, with browser bindings in `joule-comm-mls-wasm`; `Track::e2e_encrypted` is the switch that selects this path. Live: https://wai.transaction.science/private-channel +### 6.5 Protected content: two lanes + +Content whose licence requires protection travels in one of two lanes. They keep the +content from different parties, and a WAI record covers different things in each. A +sink, a receipt, a tier claim or a status text MUST NOT present content in one lane as +being in the other. + +**Lane A: hardware-protected content.** The media is a bitstream of a capability whose +registration names its standard (§5), encrypted for a platform's protected media path: +a content decryption module (in a browser, the one the W3C Encrypted Media Extensions +interface reaches) and a decoder whose output the application cannot read. Decryption +and decode happen inside that path. + +- This revision registers no `security.encryption` value for Lane A, so no WAI + envelope carries Lane A media: it travels in the form its protection system defines, + and WAI carries records about it. An envelope that names an encryption the sink does + not implement is inert at that sink (§6.4 step 2). +- The records are the whole of what WAI attests about a Lane A title: the manifest a + session was served (`wai.video.manifest`), the package's segment map over the bytes + it sliced (`wai.video.package`), each key-release decision, which binds the released + key by commitment and never carries it (`wai.video.keys`), and the content hash of + each WAI object delivered (jwp-receipts §2.1). None of them covers a decoded frame + or sample. +- A capability that synthesises output from parameters (grain, an enhancement layer, a + generative stage) MAY apply to Lane A content only inside the protected path, in a + decoder that the protection system certifies. WAI does not certify that decoder and + cannot read its output, so such synthesis has no tier and no output digest, and a + record about it states at most which capability the sink declared it ran. + +**Lane B: end-to-end encryption at the application layer.** The payload is sealed per +object under a group key, `mls-sframe` (§6.1–§6.4, +[confidentiality](extensions/confidentiality.md) §3–§5). A member's application opens +it and runs §4 on the plaintext, so any decoder a sink has can decode it: native code, +WebAssembly, or a neural decoder. + +- Every WAI record applies as it does to plaintext: the receipt chain over the + ciphertext (§6.3), the capability's tier over the decoded output + ([determinism-tiers](extensions/determinism-tiers.md) §2), the output digests of a + `reconstruct-session` claim ([jwp-receipts](extensions/jwp-receipts.md) §2.4.12), and + reconstruction and energy bindings + ([reconstruction-binding](extensions/reconstruction-binding.md), + [energy-binding](extensions/energy-binding.md) §3). +- **Lane B is not a protected-path robustness level.** It keeps the content from relays + and from anyone outside the group (§6.2, §6.3). It does not keep it from a member's + device: the base key, the plaintext and the decoded output are in the application's + memory, readable by the application and by anything that can read that memory. A + publisher MUST NOT offer Lane B to meet a licence term that requires a + hardware-protected path, and a sink or a receipt MUST NOT describe Lane B content as + hardware-protected. + +**What the exact tier covers.** `decode-equivalence` covers the medium a sink emits and +can hash ([determinism-tiers](extensions/determinism-tiers.md) §2), on the fabrics +determinism-tiers §5 lists: native CPU and WebAssembly in this revision, with no GPU or +NPU decoder. It never covers output the application cannot read. A sink MUST NOT state +a tier, an output digest, a reconstruction binding or an energy binding for output +decoded inside a protected media path, and a `reconstruct-session` claim +(jwp-receipts §2.4.12), whose every presentation carries the SHA-256 of its output, +lists no presentation whose output its sink could not hash. In Lane B the tier holds as +it does for plaintext; in Lane A there is no output for it to cover. + --- ## 7. Conformance @@ -2150,8 +2213,9 @@ can be called from any language. The crate wraps: - **video**: `rav1e` (AV1 encode) + `dav1d` (AV1 decode). - **text**: `zstd` (libzstd), `xz2` (liblzma). -Run `cargo test --lib` (with `RUSTFLAGS="-L /opt/homebrew/opt/flac/lib"` -on macOS, or via the committed `.cargo/config.toml`). +Run `cargo test --lib`. With the `codecs` feature, `wai-rs/build.rs` finds libFLAC +through pkg-config (on macOS without pkg-config, in the Homebrew prefix); where it +does not, set `RUSTFLAGS="-L "`. Decode speed is an implementation property, not a conformance property: the integer decoders run faster kernels (SIMD, threads) built to give the same diff --git a/wai/extensions/confidentiality.md b/wai/extensions/confidentiality.md index a7f5334a8..47254347c 100644 --- a/wai/extensions/confidentiality.md +++ b/wai/extensions/confidentiality.md @@ -5,6 +5,10 @@ > `moq-sframe` (SFrame object encryption) and `joule-comm-mls` (MLS group > key agreement) crates, with browser bindings in `joule-comm-mls-wasm`. > Live demonstration: https://wai.transaction.science/private-channel +> +> This is Lane B of SPEC §6.5: end-to-end encryption at the application layer, +> under which every WAI record and tier applies to the decoded output. It is not a +> protected-path robustness level (SPEC §6.5). Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY**, **REQUIRED**, **OPTIONAL** are RFC 2119 / RFC 8174. diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index 33313c0a0..7ae205fce 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -480,6 +480,37 @@ Parameter transmission pays in bitrate and leaves the decoder's arithmetic free; the integer decode pays in quantisation, compute and model preparation and adds no bits. A capability's tier states which trade it made. +### Where `decode-equivalence` holds: fabrics, and output a sink cannot read + +The tier is a property of the integer arithmetic a capability's registration fixes, +so a fabric meets it when it executes that arithmetic exactly. What this revision +shows, by fabric: + +- **CPU, native.** The reference decoders, whose fast kernels give the same integer + for every output as the scalar reference kernels (*Compute*, above). The + golden-byte suite passes on Linux x86_64, Linux arm64 and macOS arm64 (§6). +- **WebAssembly.** The same Rust compiled to wasm32. `wai-web-wasm.yml` builds the + browser crate for `wasm32-unknown-unknown` and, under Node, requires the `int_synth` + corpus and the committed `WIV1`, `WIH1`, `WIA1` and `WIS1` vectors to decode to the + hashes the native decode gives; it passed on the forge for main at `412a08c5` + (2026-10-09). The `int_mlicpp` vector decodes to its pinned bytes on + `wasm32-wasip2` under wasmtime (§6). The build enables no SIMD target feature. A + WebAssembly decoder at this tier MUST NOT use relaxed-SIMD instructions, whose + results the WebAssembly specification lets engines choose among, and MUST NOT + depend on the bits of a NaN. +- **GPU and NPU.** No decoder of these capabilities runs on either in this revision, + and the tier is not claimed there. A network evaluated in floating point on a GPU or + NPU does not meet it (§1). An integer GPU kernel could; until a run shows its output + equal to the reference's, none is claimed. + +The tier covers the medium a sink emits and can hash (§2, *What a tier covers*). A +sink that hands a payload to a protected media path whose output the application +cannot read, as Lane A of [SPEC](../SPEC.md) §6.5 does, holds no output to hash, and +the tier never covers that output: the sink MUST NOT claim the tier for it, or bind a +digest, a reconstruction or an energy figure to it. Content that the application +decrypts itself, as in Lane B of SPEC §6.5, is decoded like any plaintext, and the +tier holds for it as it does for plaintext. + ## 6. Evidence status What the repository demonstrates, claim by claim, as of 2026-09-27 (main at -- 2.54.0 (Apple Git-157) From 30a15bad94793004c043a007eee19597edec1b94 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 12:21:50 -0400 Subject: [PATCH 03/15] wai: a reader need hold no manifest past 1 MiB, and the writers never write one `man_len` is a u32, and nothing bounded it: a reader trusted it up to the bytes in hand, and the HTTP head parser answered a `man_len` of 4 GiB by asking for 4 GiB + 12 bytes. The reference now sets a reader limit: - `container::MANIFEST_MAX_LEN` is 1 MiB (1 048 576 bytes), far above any manifest in the committed corpora. `check_manifest_len` refuses a longer one with `manifest_too_large`, from the length field alone, before anything is sliced. - Both readers (`Wai::from_bytes`, `WaiMulti::from_bytes`), `manifest_bytes` and `http_binding::head::parse_head` (as `head_malformed`) apply it. - Both writers, and the browser crate's `pack` and `packMulti`, refuse to write a manifest past it, so the reference never emits what its reader refuses. - The writers also refuse a payload whose length does not fit its u32 field; `Wai::to_bytes` and `WaiMulti::to_bytes` wrote such a length modulo 2^32. - `include/wai.h` gains the constant, as cbindgen writes it. Tests: at the limit both containers read and write back byte for byte; one byte past it, or a declared u32::MAX, is refused by every reader from the first eight bytes, and neither writer writes it; the head parser refuses it instead of asking for it. The manifest's nesting limit, which the JSON reader already applied, is now pinned: 127 levels read, 128 refused, in WAI1 and WAI2 manifests. Co-Authored-By: Claude Opus 5.5 --- wai/wai-rs/include/wai.h | 9 ++ wai/wai-rs/src/container.rs | 143 ++++++++++++++++++++++++++- wai/wai-rs/src/http_binding/head.rs | 5 +- wai/wai-rs/src/http_binding/tests.rs | 18 ++++ wai/wai-web/src/lib.rs | 8 +- 5 files changed, 176 insertions(+), 7 deletions(-) diff --git a/wai/wai-rs/include/wai.h b/wai/wai-rs/include/wai.h index 024e56bbc..5f6bc40ef 100644 --- a/wai/wai-rs/include/wai.h +++ b/wai/wai-rs/include/wai.h @@ -30,6 +30,15 @@ #include #include +/** + * The longest manifest, in bytes, a reader must accept (SPEC §2, *Reader limits*): + * 1 MiB. The reference readers refuse a longer one before they read it, and the + * reference writers refuse to write one, so `man_len` never asks a reader to hold + * more. The manifest's nesting has its own limit: the JSON reader refuses 128 nested + * arrays and objects and takes 127 (`check_unique_members`). + */ +#define MANIFEST_MAX_LEN (1 << 20) + #define WAI_OK 0 #define WAI_ERR_INVALID_ARG 1 diff --git a/wai/wai-rs/src/container.rs b/wai/wai-rs/src/container.rs index e493a1f4e..0b1d0c908 100644 --- a/wai/wai-rs/src/container.rs +++ b/wai/wai-rs/src/container.rs @@ -35,6 +35,31 @@ use crate::codecs::{DeclaredTier, DeterminismTier, ParameterSetShape}; pub const MAGIC: &[u8; 4] = b"WAI1"; // v1.0 single-payload pub const MAGIC_V2: &[u8; 4] = b"WAI2"; // v1.1 multi-rendition +/// The longest manifest, in bytes, a reader must accept (SPEC §2, *Reader limits*): +/// 1 MiB. The reference readers refuse a longer one before they read it, and the +/// reference writers refuse to write one, so `man_len` never asks a reader to hold +/// more. The manifest's nesting has its own limit: the JSON reader refuses 128 nested +/// arrays and objects and takes 127 (`check_unique_members`). +pub const MANIFEST_MAX_LEN: usize = 1 << 20; + +/// SPEC §2, *Reader limits*: `Err` with the code `manifest_too_large` when a manifest +/// of `len` bytes is longer than [`MANIFEST_MAX_LEN`]. +pub fn check_manifest_len(len: u64) -> Result<(), String> { + if len > MANIFEST_MAX_LEN as u64 { + return Err(format!( + "manifest_too_large: the manifest is {len} bytes; a reader need accept no more than {MANIFEST_MAX_LEN} (SPEC §2)" + )); + } + Ok(()) +} + +/// A length a container writes into a `u32` field (SPEC §2, §7.1), or an error +/// naming the field when it does not fit. +fn u32_field(len: usize, what: &str) -> std::io::Result { + u32::try_from(len).map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, + format!("{what} is {len} bytes, past the u32 field that holds it"))) +} + fn is_none(v: &Option) -> bool { v.is_none() } @@ -1188,11 +1213,14 @@ impl Wai { // `preserve_order`), so the bytes round-trip identically. let m = serde_json::to_vec(&self.manifest) .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + check_manifest_len(m.len() as u64) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + let payload_len = u32_field(self.payload.len(), "the payload")?; let mut out = Vec::with_capacity(12 + m.len() + self.payload.len()); out.extend_from_slice(MAGIC); out.extend_from_slice(&(m.len() as u32).to_le_bytes()); out.extend_from_slice(&m); - out.extend_from_slice(&(self.payload.len() as u32).to_le_bytes()); + out.extend_from_slice(&payload_len.to_le_bytes()); out.extend_from_slice(&self.payload); Ok(out) } @@ -1220,6 +1248,8 @@ impl Wai { } need(4, 4, "manifest length")?; let ml = u32::from_le_bytes(bytes[4..8].try_into().unwrap()) as usize; + check_manifest_len(ml as u64) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; need(8, ml, "manifest body")?; check_unique_members(&bytes[8..8 + ml]) .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; @@ -1520,6 +1550,7 @@ pub fn manifest_bytes(envelope: &[u8]) -> std::io::Result<&[u8]> { return Err(bad("unknown magic (not WAI1/WAI2)".into())); } let ml = u32::from_le_bytes(envelope[4..8].try_into().unwrap()) as usize; + check_manifest_len(ml as u64).map_err(bad)?; if 8usize.saturating_add(ml) > envelope.len() { return Err(bad(format!("truncated envelope (manifest body needs {ml} bytes at offset 8, have {})", envelope.len()))); } @@ -2482,6 +2513,8 @@ impl WaiMulti { } let m = serde_json::to_vec(&self.manifest) .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + check_manifest_len(m.len() as u64) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; let table_bytes = 2 + self.renditions.len() * 8; let payload_bytes: usize = self.renditions.iter().map(|p| p.len()).sum(); @@ -2494,10 +2527,11 @@ impl WaiMulti { // Rendition table: (u32 offset, u32 length); offsets are relative to // the start of the payloads block (the byte right after the table). let mut off: u32 = 0; - for p in &self.renditions { + for (i, p) in self.renditions.iter().enumerate() { + let len = u32_field(p.len(), &format!("rendition {i}'s payload"))?; out.extend_from_slice(&off.to_le_bytes()); - out.extend_from_slice(&(p.len() as u32).to_le_bytes()); - off = off.checked_add(p.len() as u32).ok_or_else(|| std::io::Error::new( + out.extend_from_slice(&len.to_le_bytes()); + off = off.checked_add(len).ok_or_else(|| std::io::Error::new( std::io::ErrorKind::InvalidData, "payload sum overflows u32"))?; } for p in &self.renditions { @@ -2521,6 +2555,8 @@ impl WaiMulti { } need(4, 4, "manifest length")?; let ml = u32::from_le_bytes(bytes[4..8].try_into().unwrap()) as usize; + check_manifest_len(ml as u64) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; need(8, ml, "manifest body")?; check_unique_members(&bytes[8..8 + ml]) .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; @@ -4308,4 +4344,103 @@ mod tests { let back: RenditionMeta = serde_json::from_str(&j).unwrap(); assert_eq!(DerivedFrom::from_value(back.derived_from.as_ref().unwrap()).unwrap(), d); } + + /// A `WAI1` (or, with `v2`, a `WAI2`) envelope whose manifest is exactly `len` + /// bytes: a small manifest with one member no revision defines, `pad`, filled to + /// length (a reader ignores such a member, SPEC §8), over a 3-byte payload. + fn envelope_with_manifest_len(v2: bool, len: usize) -> Vec { + let head = if v2 { + r#"{"wai":"1.1","media":"text","intent":"replicate","renditions":[{"capability":"wai.text.zstd","kind":"zstd"}],"target":null,"pad":""# + } else { + r#"{"wai":"1.0","media":"text","intent":"replicate","model_requirement":{"capability":"wai.text.zstd"},"conditioning":{"kind":"zstd"},"target":null,"pad":""# + }; + let m = format!("{head}{}\"}}", "x".repeat(len - head.len() - 2)); + assert_eq!(m.len(), len); + let mut b = if v2 { MAGIC_V2.to_vec() } else { MAGIC.to_vec() }; + b.extend_from_slice(&(m.len() as u32).to_le_bytes()); + b.extend_from_slice(m.as_bytes()); + if v2 { + b.extend_from_slice(&1u16.to_le_bytes()); + b.extend_from_slice(&0u32.to_le_bytes()); + } + b.extend_from_slice(&3u32.to_le_bytes()); + b.extend_from_slice(b"abc"); + b + } + + /// SPEC §2, *Reader limits*: a manifest of exactly `MANIFEST_MAX_LEN` bytes is + /// read and written, by `WAI1` and `WAI2` alike; one byte more is refused by both + /// readers and by `manifest_bytes`, from the length field alone, and neither writer + /// writes one. + #[test] + fn a_manifest_past_the_reader_limit_is_refused_and_never_written() { + assert_eq!(MANIFEST_MAX_LEN, 1_048_576); + let too_large = |e: String| assert!(e.starts_with("manifest_too_large:"), "{e}"); + + let at = envelope_with_manifest_len(false, MANIFEST_MAX_LEN); + let w = Wai::from_bytes(&at).expect("a manifest at the limit is read"); + assert_eq!(manifest_bytes(&at).unwrap().len(), MANIFEST_MAX_LEN); + assert_eq!(w.to_bytes().unwrap(), at, "and written back byte for byte"); + let over = envelope_with_manifest_len(false, MANIFEST_MAX_LEN + 1); + too_large(Wai::from_bytes(&over).err().expect("refused").to_string()); + too_large(manifest_bytes(&over).unwrap_err().to_string()); + let mut big = w; + big.manifest.extra.insert("pad".into(), Value::from("x".repeat(MANIFEST_MAX_LEN))); + too_large(big.to_bytes().unwrap_err().to_string()); + + let at = envelope_with_manifest_len(true, MANIFEST_MAX_LEN); + let w = WaiMulti::from_bytes(&at).expect("a WAI2 manifest at the limit is read"); + assert_eq!(manifest_bytes(&at).unwrap().len(), MANIFEST_MAX_LEN); + assert_eq!(w.to_bytes().unwrap(), at, "and written back byte for byte"); + let over = envelope_with_manifest_len(true, MANIFEST_MAX_LEN + 1); + too_large(WaiMulti::from_bytes(&over).err().expect("refused").to_string()); + too_large(manifest_bytes(&over).unwrap_err().to_string()); + let mut big = w; + big.manifest.extra.insert("pad".into(), Value::from("x".repeat(MANIFEST_MAX_LEN))); + too_large(big.to_bytes().unwrap_err().to_string()); + + // From the length field alone: eight bytes that declare one byte past the limit, + // or the most a u32 can, are refused as too large, not as truncated. + for declared in [MANIFEST_MAX_LEN as u32 + 1, u32::MAX] { + for magic in [MAGIC, MAGIC_V2] { + let b = [&magic[..], &declared.to_le_bytes()].concat(); + too_large(manifest_bytes(&b).unwrap_err().to_string()); + let e = if magic == MAGIC { Wai::from_bytes(&b).err() } else { WaiMulti::from_bytes(&b).err() }.expect("refused"); + too_large(e.to_string()); + } + } + } + + /// The manifest's nesting limit the reference readers apply (SPEC §2, *Reader + /// limits*): 127 nested arrays and objects are read, 128 are refused, in `WAI1` and + /// `WAI2` manifests alike. + #[test] + fn a_manifest_nested_past_127_is_refused() { + for v2 in [false, true] { + let base = envelope_with_manifest_len(v2, 300); + let ml = u32::from_le_bytes(base[4..8].try_into().unwrap()) as usize; + let text = std::str::from_utf8(&base[8..8 + ml]).unwrap(); + let rest = &base[8 + ml..]; + // The manifest object is level 1, so `pad` holding n nested arrays is n + 1. + let pad = text.find("\"pad\":").unwrap(); + let nested = |levels: usize| { + let n = levels - 1; + let m = format!("{}\"pad\":{}{}}}", &text[..pad], "[".repeat(n), "]".repeat(n)); + [&base[..4], &(m.len() as u32).to_le_bytes(), m.as_bytes(), rest].concat() + }; + let read = |b: &[u8]| if v2 { WaiMulti::from_bytes(b).map(|_| ()) } else { Wai::from_bytes(b).map(|_| ()) }; + assert!(read(&nested(127)).is_ok(), "v2={v2}: 127 levels are read"); + assert!(read(&nested(128)).is_err(), "v2={v2}: 128 levels are refused"); + } + } + + /// A length a container writes into a u32 field must fit it: the writers refuse a + /// payload past 2^32 − 1 bytes instead of writing its length modulo 2^32. + #[test] + #[cfg(target_pointer_width = "64")] + fn a_length_past_a_u32_field_is_refused() { + assert_eq!(u32_field(u32::MAX as usize, "x").unwrap(), u32::MAX); + let e = u32_field(u32::MAX as usize + 1, "the payload").unwrap_err().to_string(); + assert!(e.contains("the payload is 4294967296 bytes, past the u32 field"), "{e}"); + } } diff --git a/wai/wai-rs/src/http_binding/head.rs b/wai/wai-rs/src/http_binding/head.rs index 97ff1fac8..83fe94929 100644 --- a/wai/wai-rs/src/http_binding/head.rs +++ b/wai/wai-rs/src/http_binding/head.rs @@ -52,7 +52,9 @@ fn u32_at(b: &[u8], at: usize) -> u64 { /// Parse the head at the start of `prefix`, an object of `envelope_len` bytes, and /// apply §5.1: the object is exactly one envelope. Returns `Need(n)` while `prefix` -/// holds fewer than the `n` bytes the head needs. +/// holds fewer than the `n` bytes the head needs. A `man_len` past SPEC §2's reader +/// limit ([`crate::container::MANIFEST_MAX_LEN`]) is refused from the first eight +/// bytes, so `n` never asks for a manifest longer than that. pub fn parse_head(prefix: &[u8], envelope_len: u64) -> Result { let have = prefix.len() as u64; if have < 8 { @@ -64,6 +66,7 @@ pub fn parse_head(prefix: &[u8], envelope_len: u64) -> Result return Err(RangeRefusal::HeadMalformed("the magic is neither WAI1 nor WAI2".into())), }; let man_len = u32_at(prefix, 4); + crate::container::check_manifest_len(man_len).map_err(RangeRefusal::HeadMalformed)?; let manifest = 8..8 + man_len; let (head_len, components) = match version { Version::Wai1 => { diff --git a/wai/wai-rs/src/http_binding/tests.rs b/wai/wai-rs/src/http_binding/tests.rs index a6560cfe9..00f4f8fd6 100644 --- a/wai/wai-rs/src/http_binding/tests.rs +++ b/wai/wai-rs/src/http_binding/tests.rs @@ -101,6 +101,24 @@ fn head_parse_needs_more() { assert!(matches!(parse_head(b"WAI3\0\0\0\0", 8), Err(RangeRefusal::HeadMalformed(_)))); } +/// SPEC §2, *Reader limits*: the first eight bytes of a head whose `man_len` passes +/// the limit are refused (`head_malformed`), so the head never asks for more than a +/// manifest of `MANIFEST_MAX_LEN` bytes; at the limit it asks for the head. +#[test] +fn head_parse_refuses_a_manifest_past_the_reader_limit() { + let max = crate::container::MANIFEST_MAX_LEN as u64; + for (magic, head_at_max) in [(b"WAI1", 12 + max), (b"WAI2", 10 + max)] { + let at = [&magic[..], &(max as u32).to_le_bytes()].concat(); + assert_eq!(parse_head(&at, u64::MAX).unwrap(), HeadParse::Need(head_at_max)); + for declared in [max as u32 + 1, u32::MAX] { + let over = [&magic[..], &declared.to_le_bytes()].concat(); + let e = parse_head(&over, u64::MAX).unwrap_err(); + assert_eq!(e.code(), "head_malformed"); + assert!(matches!(&e, RangeRefusal::HeadMalformed(m) if m.starts_with("manifest_too_large:")), "{e:?}"); + } + } +} + #[test] fn object_length_rule() { let d = appendix_d(); diff --git a/wai/wai-web/src/lib.rs b/wai/wai-web/src/lib.rs index 48711a466..76711eae0 100644 --- a/wai/wai-web/src/lib.rs +++ b/wai/wai-web/src/lib.rs @@ -521,9 +521,11 @@ fn pack_multi_refusal(manifest: &wai::container::ManifestV2, payloads: &[Vec /// encoder MUST NOT emit (SPEC §5, *Candidate capabilities*), and a payload that is /// not the format SPEC §5 pins its integer capability or fallback to. The manifest is /// written as given, so a manifest that names a member twice in one object is refused -/// (SPEC §2), as every reader refuses it. +/// (SPEC §2), as every reader refuses it, and so is one longer than SPEC §2's reader +/// limit (`manifest_too_large`). #[wasm_bindgen(js_name = pack)] pub fn pack(manifest_json: &str, payload: &[u8]) -> Result, JsError> { + wai::container::check_manifest_len(manifest_json.len() as u64).map_err(|e| JsError::new(&e))?; let m: wai::container::Manifest = read_manifest(manifest_json).map_err(|e| JsError::new(&e))?; pack_refusal(&m, payload).map_err(|e| JsError::new(&e))?; let mb = manifest_json.as_bytes(); @@ -671,9 +673,11 @@ pub fn parse_multi(bytes: &[u8]) -> Result { /// payloads. The manifest is written as given. Like `wai_envelope_pack_multi`, /// refuses a table with a component, in either role, whose capability is a /// candidate, or whose payload is not the format SPEC §5 pins its integer capability -/// to (SPEC §5). +/// to (SPEC §5), and a manifest longer than SPEC §2's reader limit +/// (`manifest_too_large`). #[wasm_bindgen(js_name = packMulti)] pub fn pack_multi(manifest_json: &str, payloads: js_sys::Array) -> Result, JsError> { + wai::container::check_manifest_len(manifest_json.len() as u64).map_err(|e| JsError::new(&e))?; let manifest: wai::container::ManifestV2 = read_manifest(manifest_json).map_err(|e| JsError::new(&e))?; wai::container::check_component_candidates(manifest.renditions.iter().map(|r| r.capability.as_str())) .map_err(|e| JsError::new(&e.to_string()))?; -- 2.54.0 (Apple Git-157) From 027dca6dc548436a73588d79c93fbbc3384b398e Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 12:41:10 -0400 Subject: [PATCH 04/15] wai: the Python verifiers apply the same manifest reader limit as wai-rs The stdlib verifiers already copy the reference reader's nesting limit, so that two readers agree on an envelope. They now copy its length limit too: `staged_verify.MANIFEST_MAX` (1 MiB) is refused by `read_wai1` and `read_wai2`, by `claim_verify.wai2_read`, by `pack_verify.read_envelope`, and by `http_verify.parse_head` as `head_malformed` before it asks for the head. Checked: every changed verifier passes its corpus as CI runs it (pack, staged, http, staged-measure, claim). A manifest of exactly 1 MiB is read by each reader and one byte more refused, as wai-rs does. On main, `read_wai1` read the longer one and `parse_head` answered eight bytes declaring u32::MAX by asking for 4 GiB. Co-Authored-By: Claude Opus 5.5 --- wai/claim-conformance/claim_verify.py | 2 +- wai/http-conformance/http_verify.py | 2 ++ wai/pack-conformance/pack_verify.py | 4 +++- wai/staged-conformance/staged_verify.py | 8 ++++++++ 4 files changed, 14 insertions(+), 2 deletions(-) diff --git a/wai/claim-conformance/claim_verify.py b/wai/claim-conformance/claim_verify.py index 138ffdf47..79d3ecb26 100644 --- a/wai/claim-conformance/claim_verify.py +++ b/wai/claim-conformance/claim_verify.py @@ -767,7 +767,7 @@ def wai2_read(data): if len(data) < 8 or data[:4] != b"WAI2": return None ml = struct.unpack_from(" len(data): + if ml > SV.MANIFEST_MAX or 8 + ml > len(data): return None try: m = SV.strict_json(data[8:8 + ml]) diff --git a/wai/http-conformance/http_verify.py b/wai/http-conformance/http_verify.py index 947bd8ed0..134cfc107 100644 --- a/wai/http-conformance/http_verify.py +++ b/wai/http-conformance/http_verify.py @@ -121,6 +121,8 @@ def parse_head(prefix, n): if magic not in (b"WAI1", b"WAI2"): raise Refusal("head_malformed") ml = u32(prefix, 4) + if ml > SV.MANIFEST_MAX: # SPEC §2, *Reader limits*: refused before it is asked for + raise Refusal("head_malformed") if magic == b"WAI1": head_len = 12 + ml if len(prefix) < head_len: diff --git a/wai/pack-conformance/pack_verify.py b/wai/pack-conformance/pack_verify.py index 9db9b0dbf..2458d8ace 100644 --- a/wai/pack-conformance/pack_verify.py +++ b/wai/pack-conformance/pack_verify.py @@ -78,6 +78,8 @@ import _ed25519 as ED # noqa: E402 import int_payload_verify as IPV # noqa: E402 MAX_INTEGER = (1 << 53) - 1 +# The longest manifest a reader must accept (SPEC §2, *Reader limits*). +MANIFEST_MAX = 1 << 20 U32 = (1 << 32) - 1 # The frames of each piece an encode vector codes: the first ENCODE_FRAMES. ENCODE_FRAMES = 12 @@ -1445,7 +1447,7 @@ def read_envelope(b): if len(b) < 8 or b[:4] not in (b"WAI1", b"WAI2"): return None ml = le(b, 4, 4) - if 8 + ml > len(b): + if ml > MANIFEST_MAX or 8 + ml > len(b): return None man = read_json(b[8:8 + ml]) # §6.3: "does not parse" is the typed parse with SPEC §3's members. diff --git a/wai/staged-conformance/staged_verify.py b/wai/staged-conformance/staged_verify.py index 921d71366..5159c4c6e 100644 --- a/wai/staged-conformance/staged_verify.py +++ b/wai/staged-conformance/staged_verify.py @@ -66,6 +66,10 @@ U64 = (1 << 64) - 1 # that two readers must agree on stays within. DEPTH_LIMIT = 127 +# The longest manifest a reader must accept (SPEC §2, *Reader limits*); the reference +# reader refuses a longer one from its length field alone (`manifest_too_large`). +MANIFEST_MAX = 1 << 20 + class NotJSON(ValueError): pass @@ -251,6 +255,8 @@ def read_wai1(b): if len(b) < 8 or b[:4] != b"WAI1": raise NotWai1("not a WAI1 envelope") ml = struct.unpack_from(" MANIFEST_MAX: + raise NotWai1("manifest_too_large") if 8 + ml + 4 > len(b): raise NotWai1("truncated manifest") text = b[8:8 + ml] @@ -1169,6 +1175,8 @@ def read_wai2(b): if len(b) < 8 or b[:4] != b"WAI2": raise NotWai2("not WAI2") ml = struct.unpack_from(" MANIFEST_MAX: + raise NotWai2("manifest_too_large") if 8 + ml > len(b): raise NotWai2("truncated manifest") try: -- 2.54.0 (Apple Git-157) From 6c3abb1530fe1274d8a67015c620233f485ad77d Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 13:01:30 -0400 Subject: [PATCH 05/15] =?UTF-8?q?wai:=20SECURITY.md,=20and=20SPEC=20=C2=A7?= =?UTF-8?q?2's=20reader=20limits=20stated=20as=20rules?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `wai/SECURITY.md` is engineering notes, not normative text: - how to report a vulnerability: security@transaction.science, the address the repository's sandbox standard already gives, and what to send; - manifest parsing: every bound the readers apply, with where it is specified and which tests check it, including the Python verifiers' share of them; - the native codec libraries behind the `codecs` and `neural` features, that they run in-process with no sandbox, what does not touch them, and two gaps stated as gaps: `wai.text.zstd` and `wai.text.xz` decompress without an output bound, and the random-input smoke test is in no CI workflow; - module authorisation (SPEC §4, jwp-receipts §7.9) and what it does not yet cover: the environment a module runs in; - the fingerprinting risk of `held_sets`, pointing at the privacy rules still to be written for http-delivery. SPEC §2 gains *Reader limits*: no reader refuses a manifest for its size or nesting within 1 MiB and 127 levels, any reader may refuse one past them, and a refusal for length comes from `man_len` alone (`manifest_too_large`). The container layout does not change. http-delivery §7 step 3 and the WAI2 packer rule point at it. `verify_pack.mjs`, which the wasm job runs, now checks that `pack` and `packMulti` write a manifest of exactly 1 MiB and refuse one byte more; it passes against a release wasm32 build of this source. That build also validates with WebAssembly's SIMD and relaxed-SIMD features disabled (a control with bulk memory disabled fails), which determinism-tiers §5 now records in place of a claim about build flags. Co-Authored-By: Claude Opus 5.5 --- wai/README.md | 1 + wai/SECURITY.md | 163 ++++++++++++++++++++++++++++ wai/SPEC.md | 13 ++- wai/extensions/determinism-tiers.md | 10 +- wai/extensions/http-delivery.md | 3 +- wai/wai-web/demo/verify_pack.mjs | 28 ++++- 6 files changed, 210 insertions(+), 8 deletions(-) create mode 100644 wai/SECURITY.md diff --git a/wai/README.md b/wai/README.md index b8ad259b9..f9bac8c3a 100644 --- a/wai/README.md +++ b/wai/README.md @@ -52,6 +52,7 @@ cargo check -p wai --no-default-features | [`research/`](research/) | Research notes and dated records: graded, not normative | | [`glossary/`](glossary/) | Reference catalog of media codecs surveyed for capability registration. [`codecs.json`](glossary/codecs.json) is the dataset, linked to the registry; [`SCHEMA.md`](glossary/SCHEMA.md) defines the shape and contribution rules. Rendered at [wai.transaction.science/glossary](https://wai.transaction.science/glossary). | | [`CONTRIBUTING.md`](CONTRIBUTING.md) | How to change the specification, register a capability and add tests CI runs | +| [`SECURITY.md`](SECURITY.md) | How to report a vulnerability, and where the reference implementation reads untrusted input: the manifest limits, the native codec libraries, module authorisation, the fingerprinting risk of advertised parameter sets | ## Quickstart diff --git a/wai/SECURITY.md b/wai/SECURITY.md new file mode 100644 index 000000000..683e37e74 --- /dev/null +++ b/wai/SECURITY.md @@ -0,0 +1,163 @@ +# Security + +This file says where WAI's reference implementation reads untrusted input, what it +bounds, and what it does not yet do. The rules a sink must follow are in +[`SPEC.md`](SPEC.md) and [`extensions/`](extensions/); this file points at them and +adds nothing normative. + +## Reporting a vulnerability + +Email **security@transaction.science**. Do not open a public issue or pull request +for a vulnerability that is not yet fixed. + +Say which component is affected (a crate and file, a verifier, or a section of the +specification), the commit you tested, the input or steps that trigger it, what +happened and what you expected. An input file is the most useful thing to send. + +In scope: + +- the specification and its extensions, where a rule lets a conforming sink or + verifier be made to misbehave; +- the reference implementation: `wai-rs`, `wai-quantum`, `wai-web`, `wai-pack`, + `wai-relay`, `wai-verify` and the other crates under this directory; +- the conformance verifiers (`*-conformance/*.py`) and their corpora. + +Out of scope: services anyone deploys with this code, which their operators answer for. + +WAI is pre-1.0. Fixes go to `main` of the canonical repository, +`https://git.transaction.science/Transaction-Science/open-standards`; no earlier +release line is maintained. + +## Manifest parsing + +Every envelope a sink receives is untrusted. Its container and manifest are read by +`wai-rs/src/container.rs` (`Wai::from_bytes`, `WaiMulti::from_bytes`, +`manifest_bytes`) and, over HTTP, by `wai-rs/src/http_binding/head.rs` +(`parse_head`). + +What the readers bound: + +| what | bound | where it is specified | +|---|---|---| +| every length field (`man_len`, `payload_len`, each rendition's offset and length) | checked against the bytes in hand before anything is sliced; sums formed so they cannot wrap (`saturating_add`; the `WAI2` payload total in `u64`) | SPEC §2, §7.1 | +| manifest length | at most 1 048 576 bytes. A longer one is refused from `man_len` alone, before it is read or, over HTTP, requested (`manifest_too_large`; `head_malformed` from `parse_head`) | SPEC §2, *Reader limits* | +| manifest nesting | at most 127 nested arrays and objects, the manifest object counting as one; 128 are refused | SPEC §2, *Reader limits* | +| member names | a member named twice in one object, at any depth, is refused | SPEC §2 | +| `WAI2` rendition table | at most 65 535 rows (`u16`), as many as the manifest's `renditions`, packed contiguously from offset 0 | SPEC §7.1 | +| an HTTP object | exactly one envelope; any other length is refused (`object_length_mismatch`) | http-delivery §5.1 | +| a head proof | at most 64 chaining values; `covered_len` is the object's length, or shorter and ending on a chunk boundary of the hashed input | http-delivery §6 | + +Allocation follows the input: a reader copies a payload only after its bounds are +checked, so it never allocates more than it was given. A `WAI1` reader ignores bytes +after the payload; an HTTP object that carries any is refused (§5.1). + +The library tests in `container.rs` and `http_binding/tests.rs` check the manifest +limits at their edges: a manifest of exactly 1 MiB is read and written back byte for +byte, one byte more is refused by every reader from its first eight bytes and written +by neither writer, and 127 levels are read where 128 are refused. The writers +(`Wai::to_bytes`, `WaiMulti::to_bytes`, and `pack` and `packMulti` in `wai-web`) +refuse a manifest past the length limit, and a payload whose length does not fit +its `u32` field. + +The stdlib Python verifiers read envelopes too. `staged_verify.py`, +`claim_verify.py`, `pack_verify.py` and `http_verify.py` refuse a manifest past the +length limit as the reference does. Those that read manifests through +`staged_verify.strict_json` refuse nesting past 127; `pack_verify.py` reads up to the +interpreter's recursion limit and refuses past it as unparseable. + +Payload formats that WAI defines carry their own caps, checked before the first symbol +is decoded: a tensor of at most 2²⁶ elements, a `WIV1` clip of at most 2³⁰ decoded +bytes and 2²⁰ frames, and a sink cost limit (`int_payload::check_cost`, +`cost_over_limit`) ([integer-payloads](extensions/integer-payloads.md) §7, §8). +`wai.neural.int_mlicpp` refuses a header whose shapes, or whose buffers, pass its cap +before it allocates them (`tests/int_mlicpp_preflight.rs`). + +## Native codec libraries and sandboxed decode + +The default `codecs` feature binds C and C++ libraries for the floor and the +recommended set: libzstd and liblzma for `wai.text.zstd` and `wai.text.xz`, libjxl +for JPEG XL, libavif for AVIF decode, libopus, dav1d for AV1 decode, and libFLAC for +FLAC encode. PNG, JPEG and FLAC decode run in Rust (the `image` and `claxon` crates). +The `neural` feature binds ONNX Runtime for the float neural decoders. These libraries +parse payload bytes directly, and they are the usual place a memory-safety defect +in a media stack lives. + +They run in the caller's process, with its privileges: the reference implementation +has no process or system-call sandbox. A sink that decodes payloads from origins it +does not trust through these libraries should run the decode in a separate process +with no file-system or network access, or use a decoder compiled to WebAssembly and +run it in a WebAssembly runtime. + +What does not touch them: + +- The builds CI runs (`--no-default-features` with named features) link none of + them. +- WAI's own decoders are Rust: the integer codecs, worlds, scores, films and the + generator. Outside the C ABI (`ffi.rs`, `ffi_codecs.rs`, `ffi_neural.rs`), the only + `unsafe` in `wai-rs`'s library code is in the AArch64 SIMD kernels of + `int_kernels.rs`, each block with the bounds argument beside it. +- `wai-web`, the browser crate, links none of the C libraries and runs inside the + browser's WebAssembly sandbox. + +Model checking covers part of the integer core: Kani harnesses beside the code in +`rans.rs`, `int_entropy.rs`, `int_transform.rs`, `int_kernels.rs`, +`wai_video_policy.rs` and `staged/`, hosted by `wai-verify` and run by +`.github/workflows/wai-formal-verification.yml`. + +Known gaps: + +- **Decompression output is not bounded.** `wai.text.zstd`, in the floor, and + `wai.text.xz` decode the whole stream into memory, so a small payload can expand to + any size the format allows. The image, audio and video decoders allocate what the + bitstream declares, within the libraries' own limits. The reference sets no output + limit of its own. +- **Fuzzing is shallow.** `tests/fuzz_smoke.rs` feeds seeded random bytes to each + codec wrapper and to `Wai::from_bytes` and requires that nothing panics. It needs + the `codecs` feature, which no CI workflow builds, so it runs only by hand. There is + no coverage-guided fuzzing of the manifest parser, the integer payload parsers or the + codec wrappers. + +## Module authorisation + +A module is code a sink obtains apart from an envelope and runs to decode a +capability or to rebuild a reconstruction ([SPEC](SPEC.md) §4, *Module +authorisation*). A sink runs one only if it holds it under +[jwp-receipts](extensions/jwp-receipts.md) §7.9: a grant, signed under a module +origin's key log, naming the SHA-256 of the module's bytes exactly as the sink +instantiates them, its form, the capabilities it may decode and a validity window, +with revocation through the same key log. A module the sink does not hold leaves its +capability unadvertised for that envelope, and the envelope's fallback still runs. +Parameter sets pinned by digest are checked before use and revoked the same way +(SPEC §3.1, checks 5–8). + +The reference dispatcher runs no module obtained apart from an envelope. With a key +log (`key_log::SinkKeyLog`, feature `key_discovery`) it refuses a revoked parameter +set (`prior_revoked`). + +Authorisation decides which code runs, not what that code can reach. The +environment a module runs in (its imports, a memory limit, a time limit, and the +determinism rules an exact replay needs) is not specified yet. Until it is, a sink +that runs modules confines them itself, for example in a WebAssembly runtime that +gives the module no file-system, network, clock or random-number imports. + +## Fingerprinting: advertised parameter sets + +The Sink Capability Descriptor ([http-delivery](extensions/http-delivery.md) §12.1) +can list `held_sets`: for each capability, the digests of the parameter sets the +sink holds. Which sets a sink holds depends on what it has fetched before, so the +list can identify a sink across the sites it is sent to, much as a cookie does. A +shared compression dictionary named by its hash raises the same concern (RFC 9842). + +Privacy rules for the descriptor and for parameter-set fetches are not written yet. +They are to be added to http-delivery: the list kept per top-level site, clearable by +the user, disclosed only as far as a choice of rendition needs it, never sent across +sites, and off by default. Until they land, send `held_sets` only to the origin that +needs it to choose a rendition, and never across sites. No code in this tree builds +or sends the descriptor. + +## Protected content + +End-to-end encryption at the application layer (`mls-sframe`, SPEC §6) keeps content +from relays and from anyone outside the group, not from a member's device: the +decrypted payload and the decoded output are in the application's memory. It is not a +protected-path robustness level ([SPEC](SPEC.md) §6.5). diff --git a/wai/SPEC.md b/wai/SPEC.md index a687d8530..3cbffa7c5 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -86,6 +86,17 @@ OPTIONAL; §5 says which are RECOMMENDED. All WAI multi-byte integers are little-endian unless stated otherwise. +**Reader limits.** A reader MUST NOT refuse a manifest for its length or its nesting +while it is at most 1 048 576 bytes (`man_len` ≤ 2²⁰) and its arrays and objects nest +at most 127 deep, the manifest object itself counting as one. It MAY refuse a longer or +deeper manifest. A reader that refuses one past the length limit does so from `man_len` +alone, before it reads or requests the manifest (`manifest_too_large`). The reference +readers refuse both. The same limits apply to the `WAI2` manifest (§7.1). They bound a +reader's work and move no byte of the layout above, which stays frozen (§8). Since a +conforming encoder emits only containers every conforming sink can read (§7), a +conforming encoder writes no manifest past either limit; the reference writers refuse +one past the length limit. + An envelope delivered over HTTP is one object of the proposed media type `application/wai`, named by its receipt content hash, and fetched whole or, for a `WAI2`, one verified component at a time @@ -1822,7 +1833,7 @@ by which capability happens to be installed. JSON value (whitespace around a member's value is not part of it, so an explode cannot write it back) and nothing follows its payload; a packer refuses any other envelope, and refuses an output that does not explode back to its sources (one - whose manifest, nested in the `WAI2` manifest, passes a reader's depth limit). Entry + whose manifest, nested in the `WAI2` manifest, passes §2's nesting limit). Entry *i*'s exploded envelope is defined by entry *i* and the entries its `refines` names, so an entry that cannot be exploded leaves every entry that does not name it intact. A tool that re-serialises a `WAI2` manifest may change these bytes, and with them diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index 7ae205fce..48c417f94 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -494,10 +494,12 @@ shows, by fabric: corpus and the committed `WIV1`, `WIH1`, `WIA1` and `WIS1` vectors to decode to the hashes the native decode gives; it passed on the forge for main at `412a08c5` (2026-10-09). The `int_mlicpp` vector decodes to its pinned bytes on - `wasm32-wasip2` under wasmtime (§6). The build enables no SIMD target feature. A - WebAssembly decoder at this tier MUST NOT use relaxed-SIMD instructions, whose - results the WebAssembly specification lets engines choose among, and MUST NOT - depend on the bits of a NaN. + `wasm32-wasip2` under wasmtime (§6). The browser crate's release build contains no + SIMD instruction: on the source of the change that added this subsection it + validated with WebAssembly's SIMD and relaxed-SIMD features disabled + (`wasm-tools validate`), which CI does not check. A WebAssembly decoder at this + tier MUST NOT use relaxed-SIMD instructions, whose results the WebAssembly + specification lets engines choose among, and MUST NOT depend on the bits of a NaN. - **GPU and NPU.** No decoder of these capabilities runs on either in this revision, and the tier is not claimed there. A network evaluated in floating point on a GPU or NPU does not meet it (§1). An integer GPU kernel could; until a run shows its output diff --git a/wai/extensions/http-delivery.md b/wai/extensions/http-delivery.md index 016177264..2d6ac04d6 100644 --- a/wai/extensions/http-delivery.md +++ b/wai/extensions/http-delivery.md @@ -317,7 +317,8 @@ The procedure: 16369 otherwise, and request `.wai.hp`. 2. Check each response (§7.1). If `covered_len` exceeds the bytes received, request the rest of the first `covered_len` bytes. -3. Verify the head (§6.3). Parse it (`head_malformed`); apply §5.1; refuse a +3. Verify the head (§6.3). Parse it (`head_malformed`, which covers a manifest past + [SPEC](../SPEC.md) §2's reader limits); apply §5.1; refuse a capability the variant's union does not list (`codecs_undeclared`). 4. Select from the head exactly as [SPEC](../SPEC.md) §7.1 selects, deferring only the component-digest test to the bytes. A deployer policy MAY reorder and restrict diff --git a/wai/wai-web/demo/verify_pack.mjs b/wai/wai-web/demo/verify_pack.mjs index 1b0524844..c2178f226 100644 --- a/wai/wai-web/demo/verify_pack.mjs +++ b/wai/wai-web/demo/verify_pack.mjs @@ -4,7 +4,8 @@ // either role), and still write an envelope that names none. The candidates are // read from the committed export wai/registry/capabilities.json, so a newly // registered candidate is checked without editing this script. They also refuse a -// manifest that names a member twice in one object (SPEC §2), which no reader reads. +// manifest that names a member twice in one object (SPEC §2), which no reader reads, +// and one longer than SPEC §2's reader limit, while writing one at the limit. // // node verify_pack.mjs [path/to/wai_web.js] [path/to/capabilities.json] // The package path is relative to this directory (as in the other verify scripts); @@ -73,6 +74,29 @@ refusesTwice('pack, a member twice in model_requirement', () => wai.pack(v1Text.replace('"capability":"wai.image.png"', '"capability":"wai.image.png","q":1,"q":2'), bytes)); refusesTwice('packMulti, a member twice in an entry', () => wai.packMulti(v2([['wai.video.av1']]).replace('"kind":"bytes"', '"kind":"bytes","zz":1,"zz":2'), [bytes])); +// SPEC §2, *Reader limits*: a manifest of exactly 1 MiB is written as given; one byte +// more is refused (`manifest_too_large`), as every reference reader refuses it. +const MAX = 1 << 20; +// `text` with a member `pad` added so that it is exactly `len` bytes (all ASCII). +const padTo = (text, len) => text.slice(0, -1) + ',"pad":"' + 'x'.repeat(len - text.length - 9) + '"}'; +function refusesTooLarge(what, write) { + try { + write(); + fail(`${what}: written, but it is past the reader limit`); + } catch (e) { + if (!String(e.message ?? e).includes('manifest_too_large')) fail(`${what}: refused for another reason: ${e.message ?? e}`); + } +} +const v2Text = v2([['wai.video.av1']]); +try { + for (const [what, env] of [['pack', wai.pack(padTo(v1Text, MAX), bytes)], ['packMulti', wai.packMulti(padTo(v2Text, MAX), [bytes])]]) { + if (new DataView(env.buffer, env.byteOffset, env.byteLength).getUint32(4, true) !== MAX) fail(`${what}: a manifest at the limit was not written as given`); + } +} catch (e) { + fail(`a manifest at the reader limit was refused: ${e.message ?? e}`); +} +refusesTooLarge('pack, a manifest one byte past the limit', () => wai.pack(padTo(v1Text, MAX + 1), bytes)); +refusesTooLarge('packMulti, a manifest one byte past the limit', () => wai.packMulti(padTo(v2Text, MAX + 1), [bytes])); // An envelope that names no candidate is still written, and parses back. try { const one = wai.pack(v1('wai.image.png', 'wai.image.jpeg'), bytes); @@ -82,6 +106,6 @@ try { } catch (e) { fail(`an envelope with no candidate was refused: ${e.message ?? e}`); } -console.log(pass ? `PASS pack and packMulti refuse every candidate (${candidates.join(', ')}) and a member named twice` +console.log(pass ? `PASS pack and packMulti refuse every candidate (${candidates.join(', ')}), a member named twice and a manifest past the reader limit` : '\n✗ a browser writer emits what no reader may take'); process.exit(pass ? 0 : 1); -- 2.54.0 (Apple Git-157) From b0bdea2949621fb08743662feede009e5be7000b Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 13:05:39 -0400 Subject: [PATCH 06/15] wai: CHANGELOG for the record made true Co-Authored-By: Claude Opus 5.5 --- wai/CHANGELOG.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 112c348c5..898c04d70 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -154,6 +154,47 @@ wire format may change and such changes are recorded here. ([jwp-receipts](extensions/jwp-receipts.md) §2.4.8's `build`): a WebAssembly build optimised for size is `release`. +### The record made true: training, paths, protected content, reader limits, security + +- **`int_synth`'s weights are quantised after float training.** SPEC §5 and + [determinism-tiers](extensions/determinism-tiers.md) §5 said the capability decodes + "QAT weights". No committed model was trained that way: the trained corpus vector was + made by `tools/wai_int_synth_real.py`, float training then post-training quantisation + with each layer's shift derived from its measured activation range, and the synthetic + vectors' weights come from a seed. Both texts now say so, and so do the tools: + `tools/wai_int_synth_qat.py` is marked experimental (it trains against one fixed shift + and made no committed model), and `tools/wai_int_synth_export.py` names the working + path. +- **Stale paths.** The int_synth decoder is `int_synth.rs` at the crate root, not + `neural/int_synth.rs`. The quantum-ops engine and the quantum-sim receipt live in + `wai-quantum`, whose features `wai-rs` forwards. README and SPEC Appendix A no longer + send macOS builders to a `.cargo/config.toml` that no longer exists: `wai-rs/build.rs` + finds libFLAC. +- **Protected content: two lanes** (SPEC §6.5). Lane A, hardware-protected content, + decodes inside the platform's protected media path; WAI's records about it (the served + manifest, the segment map, the key-release decisions, the content hashes of WAI + objects) are all WAI attests, and none covers a decoded frame. This revision registers + no `security.encryption` value for it. Lane B, end-to-end encryption at the + application layer (`mls-sframe`), decrypts in the application, so every WAI record and + tier applies; it is not a protected-path robustness level. `wai.video.keys` and the + confidentiality extension say which lane they belong to. +- **Where the exact tier holds** (determinism-tiers §5). `decode-equivalence` is shown + on native CPU and on wasm32, on no GPU or NPU; a WebAssembly decoder at the tier uses + no relaxed-SIMD instruction and depends on no NaN's bits; and the tier never covers + output a sink cannot read. +- **Reader limits** (SPEC §2). A reader need accept no manifest longer than 1 048 576 + bytes or nested deeper than 127, and refuses a longer one from `man_len` alone + (`manifest_too_large`). `wai-rs`'s readers, its HTTP head parser (`head_malformed`) and + the Python verifiers apply the length limit; the writers, `wai-web`'s included, refuse + to write past it, and refuse a payload whose length does not fit its `u32` field. + *Envelopes this makes non-conforming:* none of the layout's; an envelope whose manifest + is longer than 1 MiB, or nested deeper than 127, is one a conforming sink may refuse, + so a conforming encoder no longer writes it (SPEC §7). +- **`SECURITY.md`.** How to report a vulnerability, and where the reference + implementation reads untrusted input: the manifest limits, the native codec libraries + and their gaps (unbounded decompression output for `wai.text.zstd` and `wai.text.xz`; + no fuzzing in CI), module authorisation, and the fingerprinting risk of `held_sets`. + ### One Ed25519 rule for every signature - **SPEC §7 *Signatures*.** Every Ed25519 signature WAI defines is verified strictly: the -- 2.54.0 (Apple Git-157) From 07e923c7249b6a537ede7349a63f5aa484a92a93 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 13:52:19 -0400 Subject: [PATCH 07/15] wai: the manifest reader limit is 2^27, the largest manifest a staged layer has MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate falsified the 1 MiB limit: `tests/staged_resources.rs` offers layers over up to 65 536 units whose `refines.prev` names a different producer for each unit, and the writer refused its 1.47 MB manifest. staged-delivery §3 permits that: a `WIR1` layer covers up to 2^20 units, and a `prev` with one run per unit is about 90 bytes a run, some 94 MB in all. A reader floor below that would let a sink refuse layers the extension allows. `MANIFEST_MAX_LEN` is now 2^27 (134 217 728 bytes), which holds that layer, and SPEC §2 states why. The Python verifiers use the same bound. The writers now serialise through a capped writer that stops as soon as the manifest passes the limit, so a refused manifest is never built in full. Serialising 128 MiB takes a debug build about 17 s, so the tests check the writers' cap at a small limit, and the readers' boundary from the length field: at the limit a reader refuses eight bytes only as truncated, one past it as too large. A 2 MiB manifest, past the old bound, round-trips byte for byte. `verify_pack.mjs` checks a 3 MiB manifest written and one byte past 2^27 refused by the browser writers. Co-Authored-By: Claude Opus 5.5 --- wai/CHANGELOG.md | 10 +- wai/SECURITY.md | 21 +++-- wai/SPEC.md | 6 +- wai/pack-conformance/pack_verify.py | 2 +- wai/staged-conformance/staged_verify.py | 2 +- wai/wai-rs/include/wai.h | 13 ++- wai/wai-rs/src/container.rs | 117 ++++++++++++++---------- wai/wai-web/demo/verify_pack.mjs | 16 ++-- 8 files changed, 113 insertions(+), 74 deletions(-) diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 898c04d70..9647f073f 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -182,13 +182,15 @@ wire format may change and such changes are recorded here. on native CPU and on wasm32, on no GPU or NPU; a WebAssembly decoder at the tier uses no relaxed-SIMD instruction and depends on no NaN's bits; and the tier never covers output a sink cannot read. -- **Reader limits** (SPEC §2). A reader need accept no manifest longer than 1 048 576 - bytes or nested deeper than 127, and refuses a longer one from `man_len` alone - (`manifest_too_large`). `wai-rs`'s readers, its HTTP head parser (`head_malformed`) and +- **Reader limits** (SPEC §2). A reader need accept no manifest longer than 2²⁷ bytes + (128 MiB) or nested deeper than 127, and refuses a longer one from `man_len` alone + (`manifest_too_large`). The length limit is sized to the largest manifest a + conforming staged layer has: a `prev` naming a different producer for each of 2²⁰ + units is about 94 MB. `wai-rs`'s readers, its HTTP head parser (`head_malformed`) and the Python verifiers apply the length limit; the writers, `wai-web`'s included, refuse to write past it, and refuse a payload whose length does not fit its `u32` field. *Envelopes this makes non-conforming:* none of the layout's; an envelope whose manifest - is longer than 1 MiB, or nested deeper than 127, is one a conforming sink may refuse, + is longer than 128 MiB, or nested deeper than 127, is one a conforming sink may refuse, so a conforming encoder no longer writes it (SPEC §7). - **`SECURITY.md`.** How to report a vulnerability, and where the reference implementation reads untrusted input: the manifest limits, the native codec libraries diff --git a/wai/SECURITY.md b/wai/SECURITY.md index 683e37e74..32c3d3e69 100644 --- a/wai/SECURITY.md +++ b/wai/SECURITY.md @@ -40,7 +40,7 @@ What the readers bound: | what | bound | where it is specified | |---|---|---| | every length field (`man_len`, `payload_len`, each rendition's offset and length) | checked against the bytes in hand before anything is sliced; sums formed so they cannot wrap (`saturating_add`; the `WAI2` payload total in `u64`) | SPEC §2, §7.1 | -| manifest length | at most 1 048 576 bytes. A longer one is refused from `man_len` alone, before it is read or, over HTTP, requested (`manifest_too_large`; `head_malformed` from `parse_head`) | SPEC §2, *Reader limits* | +| manifest length | at most 134 217 728 bytes (2²⁷), sized to the largest manifest a conforming staged layer has. A longer one is refused from `man_len` alone, before it is read or, over HTTP, requested (`manifest_too_large`; `head_malformed` from `parse_head`) | SPEC §2, *Reader limits* | | manifest nesting | at most 127 nested arrays and objects, the manifest object counting as one; 128 are refused | SPEC §2, *Reader limits* | | member names | a member named twice in one object, at any depth, is refused | SPEC §2 | | `WAI2` rendition table | at most 65 535 rows (`u16`), as many as the manifest's `renditions`, packed contiguously from offset 0 | SPEC §7.1 | @@ -52,12 +52,19 @@ checked, so it never allocates more than it was given. A `WAI1` reader ignores b after the payload; an HTTP object that carries any is refused (§5.1). The library tests in `container.rs` and `http_binding/tests.rs` check the manifest -limits at their edges: a manifest of exactly 1 MiB is read and written back byte for -byte, one byte more is refused by every reader from its first eight bytes and written -by neither writer, and 127 levels are read where 128 are refused. The writers -(`Wai::to_bytes`, `WaiMulti::to_bytes`, and `pack` and `packMulti` in `wai-web`) -refuse a manifest past the length limit, and a payload whose length does not fit -its `u32` field. +limits at their edges: from its first eight bytes, every reader refuses a manifest one +byte past the length limit as too large and one at the limit only as truncated; a +2 MiB manifest is read and written back byte for byte; and 127 levels are read where +128 are refused. The writers (`Wai::to_bytes`, `WaiMulti::to_bytes`, and `pack` and +`packMulti` in `wai-web`) refuse a manifest past the length limit, and a payload whose +length does not fit its `u32` field. The Rust writers stop serialising as soon as the +manifest passes the limit; the test checks that at a small limit, since serialising +128 MiB takes a debug build many seconds. `verify_pack.mjs` checks the browser writers +at the real limit. + +The length limit bounds what a reader is asked to hold, not what a sink can afford: a +sink with less memory than a 128 MiB manifest needs cannot decode the layers that +reach that size either, and refuses them by its own cost limits. The stdlib Python verifiers read envelopes too. `staged_verify.py`, `claim_verify.py`, `pack_verify.py` and `http_verify.py` refuse a manifest past the diff --git a/wai/SPEC.md b/wai/SPEC.md index 3cbffa7c5..a0f432e56 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -87,9 +87,11 @@ OPTIONAL; §5 says which are RECOMMENDED. All WAI multi-byte integers are little-endian unless stated otherwise. **Reader limits.** A reader MUST NOT refuse a manifest for its length or its nesting -while it is at most 1 048 576 bytes (`man_len` ≤ 2²⁰) and its arrays and objects nest +while it is at most 134 217 728 bytes (`man_len` ≤ 2²⁷) and its arrays and objects nest at most 127 deep, the manifest object itself counting as one. It MAY refuse a longer or -deeper manifest. A reader that refuses one past the length limit does so from `man_len` +deeper manifest. The length limit holds the largest manifest a conforming layer has +([staged-delivery](extensions/staged-delivery.md) §3): a `prev` that names a different +producer for each of the 2²⁰ units a layer may cover is about 94 MB. A reader that refuses one past the length limit does so from `man_len` alone, before it reads or requests the manifest (`manifest_too_large`). The reference readers refuse both. The same limits apply to the `WAI2` manifest (§7.1). They bound a reader's work and move no byte of the layout above, which stays frozen (§8). Since a diff --git a/wai/pack-conformance/pack_verify.py b/wai/pack-conformance/pack_verify.py index 2458d8ace..bfbe279dc 100644 --- a/wai/pack-conformance/pack_verify.py +++ b/wai/pack-conformance/pack_verify.py @@ -79,7 +79,7 @@ import int_payload_verify as IPV # noqa: E402 MAX_INTEGER = (1 << 53) - 1 # The longest manifest a reader must accept (SPEC §2, *Reader limits*). -MANIFEST_MAX = 1 << 20 +MANIFEST_MAX = 1 << 27 U32 = (1 << 32) - 1 # The frames of each piece an encode vector codes: the first ENCODE_FRAMES. ENCODE_FRAMES = 12 diff --git a/wai/staged-conformance/staged_verify.py b/wai/staged-conformance/staged_verify.py index 5159c4c6e..7376717a0 100644 --- a/wai/staged-conformance/staged_verify.py +++ b/wai/staged-conformance/staged_verify.py @@ -68,7 +68,7 @@ DEPTH_LIMIT = 127 # The longest manifest a reader must accept (SPEC §2, *Reader limits*); the reference # reader refuses a longer one from its length field alone (`manifest_too_large`). -MANIFEST_MAX = 1 << 20 +MANIFEST_MAX = 1 << 27 class NotJSON(ValueError): diff --git a/wai/wai-rs/include/wai.h b/wai/wai-rs/include/wai.h index 5f6bc40ef..682155a7e 100644 --- a/wai/wai-rs/include/wai.h +++ b/wai/wai-rs/include/wai.h @@ -32,12 +32,15 @@ /** * The longest manifest, in bytes, a reader must accept (SPEC §2, *Reader limits*): - * 1 MiB. The reference readers refuse a longer one before they read it, and the - * reference writers refuse to write one, so `man_len` never asks a reader to hold - * more. The manifest's nesting has its own limit: the JSON reader refuses 128 nested - * arrays and objects and takes 127 (`check_unique_members`). + * 2^27 (128 MiB). It is sized to the largest manifest a conforming layer has + * (staged-delivery §3): a `prev` that names a different producer for each of the + * 2^20 units a `WIR1` layer may cover, about 90 bytes a run. The reference readers + * refuse a longer one before they read it, and the reference writers refuse to write + * one, so `man_len` never asks a reader to hold more. The manifest's nesting has its + * own limit: the JSON reader refuses 128 nested arrays and objects and takes 127 + * (`check_unique_members`). */ -#define MANIFEST_MAX_LEN (1 << 20) +#define MANIFEST_MAX_LEN (1 << 27) #define WAI_OK 0 diff --git a/wai/wai-rs/src/container.rs b/wai/wai-rs/src/container.rs index 0b1d0c908..3f734a19f 100644 --- a/wai/wai-rs/src/container.rs +++ b/wai/wai-rs/src/container.rs @@ -36,11 +36,14 @@ pub const MAGIC: &[u8; 4] = b"WAI1"; // v1.0 single-payload pub const MAGIC_V2: &[u8; 4] = b"WAI2"; // v1.1 multi-rendition /// The longest manifest, in bytes, a reader must accept (SPEC §2, *Reader limits*): -/// 1 MiB. The reference readers refuse a longer one before they read it, and the -/// reference writers refuse to write one, so `man_len` never asks a reader to hold -/// more. The manifest's nesting has its own limit: the JSON reader refuses 128 nested -/// arrays and objects and takes 127 (`check_unique_members`). -pub const MANIFEST_MAX_LEN: usize = 1 << 20; +/// 2^27 (128 MiB). It is sized to the largest manifest a conforming layer has +/// (staged-delivery §3): a `prev` that names a different producer for each of the +/// 2^20 units a `WIR1` layer may cover, about 90 bytes a run. The reference readers +/// refuse a longer one before they read it, and the reference writers refuse to write +/// one, so `man_len` never asks a reader to hold more. The manifest's nesting has its +/// own limit: the JSON reader refuses 128 nested arrays and objects and takes 127 +/// (`check_unique_members`). +pub const MANIFEST_MAX_LEN: usize = 1 << 27; /// SPEC §2, *Reader limits*: `Err` with the code `manifest_too_large` when a manifest /// of `len` bytes is longer than [`MANIFEST_MAX_LEN`]. @@ -53,6 +56,35 @@ pub fn check_manifest_len(len: u64) -> Result<(), String> { Ok(()) } +/// A manifest's JSON as the writers emit it (no insignificant whitespace, members in +/// order), refused with `manifest_too_large` as soon as it would pass +/// [`MANIFEST_MAX_LEN`], so a writer never builds more of it than that. +fn manifest_json(m: &T) -> std::io::Result> { + manifest_json_within(m, MANIFEST_MAX_LEN) +} + +/// [`manifest_json`] with the limit `cap` in place of [`MANIFEST_MAX_LEN`]: what the +/// writers run, with a limit a test can reach without serialising 128 MiB. +fn manifest_json_within(m: &T, cap: usize) -> std::io::Result> { + struct Capped(Vec, usize); + impl std::io::Write for Capped { + fn write(&mut self, b: &[u8]) -> std::io::Result { + if self.0.len().saturating_add(b.len()) > self.1 { + return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, format!( + "manifest_too_large: the manifest passes {} bytes, the most a reader need accept (SPEC §2)", self.1))); + } + self.0.extend_from_slice(b); + Ok(b.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + let mut w = Capped(Vec::new(), cap); + serde_json::to_writer(&mut w, m).map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + Ok(w.0) +} + /// A length a container writes into a `u32` field (SPEC §2, §7.1), or an error /// naming the field when it does not fit. fn u32_field(len: usize, what: &str) -> std::io::Result { @@ -1211,10 +1243,7 @@ impl Wai { pub fn to_bytes(&self) -> std::io::Result> { // No insignificant whitespace, key order preserved (serde_json's // `preserve_order`), so the bytes round-trip identically. - let m = serde_json::to_vec(&self.manifest) - .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; - check_manifest_len(m.len() as u64) - .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + let m = manifest_json(&self.manifest)?; let payload_len = u32_field(self.payload.len(), "the payload")?; let mut out = Vec::with_capacity(12 + m.len() + self.payload.len()); out.extend_from_slice(MAGIC); @@ -2511,10 +2540,7 @@ impl WaiMulti { return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "too many renditions for u16 count field")); } - let m = serde_json::to_vec(&self.manifest) - .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; - check_manifest_len(m.len() as u64) - .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + let m = manifest_json(&self.manifest)?; let table_bytes = 2 + self.renditions.len() * 8; let payload_bytes: usize = self.renditions.iter().map(|p| p.len()).sum(); @@ -4368,47 +4394,44 @@ mod tests { b } - /// SPEC §2, *Reader limits*: a manifest of exactly `MANIFEST_MAX_LEN` bytes is - /// read and written, by `WAI1` and `WAI2` alike; one byte more is refused by both - /// readers and by `manifest_bytes`, from the length field alone, and neither writer - /// writes one. + /// SPEC §2, *Reader limits*, from the length field: eight bytes declaring a manifest + /// of exactly `MANIFEST_MAX_LEN` bytes are refused only as truncated, and one byte + /// more, or the most a u32 can declare, as too large, by both readers and by + /// `manifest_bytes`. A 2 MiB manifest, the size a staged layer's `prev` reaches over + /// some 23 000 units with alternating producers, is read and written back byte for + /// byte. The writers' JSON stops at its limit: a manifest is written at a limit equal + /// to its length and refused one byte below it (at `MANIFEST_MAX_LEN` itself the + /// check would serialise 128 MiB, which a debug build takes many seconds over). #[test] fn a_manifest_past_the_reader_limit_is_refused_and_never_written() { - assert_eq!(MANIFEST_MAX_LEN, 1_048_576); + assert_eq!(MANIFEST_MAX_LEN, 134_217_728); let too_large = |e: String| assert!(e.starts_with("manifest_too_large:"), "{e}"); - - let at = envelope_with_manifest_len(false, MANIFEST_MAX_LEN); - let w = Wai::from_bytes(&at).expect("a manifest at the limit is read"); - assert_eq!(manifest_bytes(&at).unwrap().len(), MANIFEST_MAX_LEN); - assert_eq!(w.to_bytes().unwrap(), at, "and written back byte for byte"); - let over = envelope_with_manifest_len(false, MANIFEST_MAX_LEN + 1); - too_large(Wai::from_bytes(&over).err().expect("refused").to_string()); - too_large(manifest_bytes(&over).unwrap_err().to_string()); - let mut big = w; - big.manifest.extra.insert("pad".into(), Value::from("x".repeat(MANIFEST_MAX_LEN))); - too_large(big.to_bytes().unwrap_err().to_string()); - - let at = envelope_with_manifest_len(true, MANIFEST_MAX_LEN); - let w = WaiMulti::from_bytes(&at).expect("a WAI2 manifest at the limit is read"); - assert_eq!(manifest_bytes(&at).unwrap().len(), MANIFEST_MAX_LEN); - assert_eq!(w.to_bytes().unwrap(), at, "and written back byte for byte"); - let over = envelope_with_manifest_len(true, MANIFEST_MAX_LEN + 1); - too_large(WaiMulti::from_bytes(&over).err().expect("refused").to_string()); - too_large(manifest_bytes(&over).unwrap_err().to_string()); - let mut big = w; - big.manifest.extra.insert("pad".into(), Value::from("x".repeat(MANIFEST_MAX_LEN))); - too_large(big.to_bytes().unwrap_err().to_string()); - - // From the length field alone: eight bytes that declare one byte past the limit, - // or the most a u32 can, are refused as too large, not as truncated. - for declared in [MANIFEST_MAX_LEN as u32 + 1, u32::MAX] { - for magic in [MAGIC, MAGIC_V2] { + let truncated = |e: String| assert!(e.contains("truncated"), "{e}"); + for magic in [MAGIC, MAGIC_V2] { + let read = |b: &[u8]| if magic == MAGIC { Wai::from_bytes(b).err() } else { WaiMulti::from_bytes(b).err() } + .expect("refused").to_string(); + let at = [&magic[..], &(MANIFEST_MAX_LEN as u32).to_le_bytes()].concat(); + truncated(read(&at)); + truncated(manifest_bytes(&at).unwrap_err().to_string()); + for declared in [MANIFEST_MAX_LEN as u32 + 1, u32::MAX] { let b = [&magic[..], &declared.to_le_bytes()].concat(); + too_large(read(&b)); too_large(manifest_bytes(&b).unwrap_err().to_string()); - let e = if magic == MAGIC { Wai::from_bytes(&b).err() } else { WaiMulti::from_bytes(&b).err() }.expect("refused"); - too_large(e.to_string()); } } + + let len = 2 << 20; + let one = envelope_with_manifest_len(false, len); + let w = Wai::from_bytes(&one).expect("a 2 MiB manifest is read"); + assert_eq!(w.to_bytes().unwrap(), one, "and written back byte for byte"); + let multi = envelope_with_manifest_len(true, len); + let w2 = WaiMulti::from_bytes(&multi).expect("a 2 MiB WAI2 manifest is read"); + assert_eq!(w2.to_bytes().unwrap(), multi, "and written back byte for byte"); + + assert_eq!(manifest_json_within(&w.manifest, len).unwrap(), &one[8..8 + len]); + too_large(manifest_json_within(&w.manifest, len - 1).unwrap_err().to_string()); + assert_eq!(manifest_json_within(&w2.manifest, len).unwrap(), &multi[8..8 + len]); + too_large(manifest_json_within(&w2.manifest, len - 1).unwrap_err().to_string()); } /// The manifest's nesting limit the reference readers apply (SPEC §2, *Reader diff --git a/wai/wai-web/demo/verify_pack.mjs b/wai/wai-web/demo/verify_pack.mjs index c2178f226..6724d1c00 100644 --- a/wai/wai-web/demo/verify_pack.mjs +++ b/wai/wai-web/demo/verify_pack.mjs @@ -5,7 +5,7 @@ // read from the committed export wai/registry/capabilities.json, so a newly // registered candidate is checked without editing this script. They also refuse a // manifest that names a member twice in one object (SPEC §2), which no reader reads, -// and one longer than SPEC §2's reader limit, while writing one at the limit. +// and one longer than SPEC §2's reader limit, while writing a long one within it. // // node verify_pack.mjs [path/to/wai_web.js] [path/to/capabilities.json] // The package path is relative to this directory (as in the other verify scripts); @@ -74,9 +74,11 @@ refusesTwice('pack, a member twice in model_requirement', () => wai.pack(v1Text.replace('"capability":"wai.image.png"', '"capability":"wai.image.png","q":1,"q":2'), bytes)); refusesTwice('packMulti, a member twice in an entry', () => wai.packMulti(v2([['wai.video.av1']]).replace('"kind":"bytes"', '"kind":"bytes","zz":1,"zz":2'), [bytes])); -// SPEC §2, *Reader limits*: a manifest of exactly 1 MiB is written as given; one byte -// more is refused (`manifest_too_large`), as every reference reader refuses it. -const MAX = 1 << 20; +// SPEC §2, *Reader limits*: a manifest of 3 MiB is written as given; one byte past the +// limit, 2^27 bytes, is refused (`manifest_too_large`), as every reference reader +// refuses it. +const MAX = 1 << 27; +const LONG = 3 << 20; // `text` with a member `pad` added so that it is exactly `len` bytes (all ASCII). const padTo = (text, len) => text.slice(0, -1) + ',"pad":"' + 'x'.repeat(len - text.length - 9) + '"}'; function refusesTooLarge(what, write) { @@ -89,11 +91,11 @@ function refusesTooLarge(what, write) { } const v2Text = v2([['wai.video.av1']]); try { - for (const [what, env] of [['pack', wai.pack(padTo(v1Text, MAX), bytes)], ['packMulti', wai.packMulti(padTo(v2Text, MAX), [bytes])]]) { - if (new DataView(env.buffer, env.byteOffset, env.byteLength).getUint32(4, true) !== MAX) fail(`${what}: a manifest at the limit was not written as given`); + for (const [what, env] of [['pack', wai.pack(padTo(v1Text, LONG), bytes)], ['packMulti', wai.packMulti(padTo(v2Text, LONG), [bytes])]]) { + if (new DataView(env.buffer, env.byteOffset, env.byteLength).getUint32(4, true) !== LONG) fail(`${what}: a 3 MiB manifest was not written as given`); } } catch (e) { - fail(`a manifest at the reader limit was refused: ${e.message ?? e}`); + fail(`a manifest within the reader limit was refused: ${e.message ?? e}`); } refusesTooLarge('pack, a manifest one byte past the limit', () => wai.pack(padTo(v1Text, MAX + 1), bytes)); refusesTooLarge('packMulti, a manifest one byte past the limit', () => wai.packMulti(padTo(v2Text, MAX + 1), [bytes])); -- 2.54.0 (Apple Git-157) From 4961fccf1e492a4903fc6ae30ba2debd87e6b42c Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 16:46:13 -0400 Subject: [PATCH 08/15] wai-rs: test the writers' limits through the writers The test named for the writers called manifest_json_within and u32_field directly and never called Wai::to_bytes or WaiMulti::to_bytes, so putting serde_json::to_vec and `as u32` back in both writers left every test green. Both writers now run to_bytes_within(WriteLimits), and to_bytes passes SPEC's limits. The test writes each envelope at limits equal to its manifest's and payloads' lengths, and requires manifest_too_large one byte below the manifest, the field error one byte below a payload, and the sum error when two payloads each fit and their total does not. MANIFEST_MAX_LEN's doc cites staged-delivery Appendix A.3 for the 2^20-unit cap (section 3 bounds a count only by 32 bits), and says the limit bounds the bytes read, not the memory a parse takes. wai.h regenerated. Co-Authored-By: Claude Opus 5.5 --- wai/wai-rs/include/wai.h | 16 +++--- wai/wai-rs/src/container.rs | 109 +++++++++++++++++++++++++----------- 2 files changed, 85 insertions(+), 40 deletions(-) diff --git a/wai/wai-rs/include/wai.h b/wai/wai-rs/include/wai.h index 682155a7e..07d1f35c3 100644 --- a/wai/wai-rs/include/wai.h +++ b/wai/wai-rs/include/wai.h @@ -32,13 +32,15 @@ /** * The longest manifest, in bytes, a reader must accept (SPEC §2, *Reader limits*): - * 2^27 (128 MiB). It is sized to the largest manifest a conforming layer has - * (staged-delivery §3): a `prev` that names a different producer for each of the - * 2^20 units a `WIR1` layer may cover, about 90 bytes a run. The reference readers - * refuse a longer one before they read it, and the reference writers refuse to write - * one, so `man_len` never asks a reader to hold more. The manifest's nesting has its - * own limit: the JSON reader refuses 128 nested arrays and objects and takes 127 - * (`check_unique_members`). + * 2^27 (128 MiB). It is sized to the largest manifest a conforming layer has: a `prev` + * (staged-delivery §3) that names a different producer for each of the 2^20 units a + * `WIR1` layer may cover (staged-delivery Appendix A.3, `n_units ≤ 2^20`), about 90 + * bytes a run. The reference readers refuse a longer one before they read it, and the + * reference writers refuse to write one, so `man_len` never asks a reader to read + * more. It bounds the bytes read, not the memory a parse takes: the reference readers + * build a JSON tree, which can take some 40 times the manifest's length. The + * manifest's nesting has its own limit: the JSON reader refuses 128 nested arrays and + * objects and takes 127 (`check_unique_members`). */ #define MANIFEST_MAX_LEN (1 << 27) diff --git a/wai/wai-rs/src/container.rs b/wai/wai-rs/src/container.rs index 3f734a19f..afc5363ae 100644 --- a/wai/wai-rs/src/container.rs +++ b/wai/wai-rs/src/container.rs @@ -36,13 +36,15 @@ pub const MAGIC: &[u8; 4] = b"WAI1"; // v1.0 single-payload pub const MAGIC_V2: &[u8; 4] = b"WAI2"; // v1.1 multi-rendition /// The longest manifest, in bytes, a reader must accept (SPEC §2, *Reader limits*): -/// 2^27 (128 MiB). It is sized to the largest manifest a conforming layer has -/// (staged-delivery §3): a `prev` that names a different producer for each of the -/// 2^20 units a `WIR1` layer may cover, about 90 bytes a run. The reference readers -/// refuse a longer one before they read it, and the reference writers refuse to write -/// one, so `man_len` never asks a reader to hold more. The manifest's nesting has its -/// own limit: the JSON reader refuses 128 nested arrays and objects and takes 127 -/// (`check_unique_members`). +/// 2^27 (128 MiB). It is sized to the largest manifest a conforming layer has: a `prev` +/// (staged-delivery §3) that names a different producer for each of the 2^20 units a +/// `WIR1` layer may cover (staged-delivery Appendix A.3, `n_units ≤ 2^20`), about 90 +/// bytes a run. The reference readers refuse a longer one before they read it, and the +/// reference writers refuse to write one, so `man_len` never asks a reader to read +/// more. It bounds the bytes read, not the memory a parse takes: the reference readers +/// build a JSON tree, which can take some 40 times the manifest's length. The +/// manifest's nesting has its own limit: the JSON reader refuses 128 nested arrays and +/// objects and takes 127 (`check_unique_members`). pub const MANIFEST_MAX_LEN: usize = 1 << 27; /// SPEC §2, *Reader limits*: `Err` with the code `manifest_too_large` when a manifest @@ -56,16 +58,25 @@ pub fn check_manifest_len(len: u64) -> Result<(), String> { Ok(()) } -/// A manifest's JSON as the writers emit it (no insignificant whitespace, members in -/// order), refused with `manifest_too_large` as soon as it would pass -/// [`MANIFEST_MAX_LEN`], so a writer never builds more of it than that. -fn manifest_json(m: &T) -> std::io::Result> { - manifest_json_within(m, MANIFEST_MAX_LEN) +/// What a container writer may write: the longest manifest, and the largest length a +/// `u32` field holds. [`Wai::to_bytes`] and [`WaiMulti::to_bytes`] write within +/// [`WriteLimits::SPEC`]; a test passes limits it can reach without serialising 128 MiB +/// or holding 4 GiB. +#[derive(Clone, Copy, Debug)] +struct WriteLimits { + manifest: usize, + field: u32, } -/// [`manifest_json`] with the limit `cap` in place of [`MANIFEST_MAX_LEN`]: what the -/// writers run, with a limit a test can reach without serialising 128 MiB. -fn manifest_json_within(m: &T, cap: usize) -> std::io::Result> { +impl WriteLimits { + /// SPEC §2's reader limit for the manifest, and the whole of a `u32`. + const SPEC: Self = Self { manifest: MANIFEST_MAX_LEN, field: u32::MAX }; +} + +/// A manifest's JSON as the writers emit it (no insignificant whitespace, members in +/// order), refused with `manifest_too_large` as soon as it would pass `cap` bytes +/// ([`MANIFEST_MAX_LEN`] in the writers), so a writer never builds more of it than that. +fn manifest_json(m: &T, cap: usize) -> std::io::Result> { struct Capped(Vec, usize); impl std::io::Write for Capped { fn write(&mut self, b: &[u8]) -> std::io::Result { @@ -86,10 +97,11 @@ fn manifest_json_within(m: &T, cap: usize) -> std::io::Result std::io::Result { - u32::try_from(len).map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, - format!("{what} is {len} bytes, past the u32 field that holds it"))) +/// naming the field when it is past `max` (in the writers, `u32::MAX`: when it does not +/// fit). +fn u32_field(len: usize, max: u32, what: &str) -> std::io::Result { + u32::try_from(len).ok().filter(|&l| l <= max).ok_or_else(|| std::io::Error::new( + std::io::ErrorKind::InvalidData, format!("{what} is {len} bytes, past the u32 field that holds it"))) } fn is_none(v: &Option) -> bool { @@ -1240,11 +1252,18 @@ impl Wai { Ok(bytes.len()) } + /// The envelope's bytes. Refuses a manifest longer than SPEC §2's reader limit + /// (`manifest_too_large`), and a payload whose length does not fit its `u32` field. pub fn to_bytes(&self) -> std::io::Result> { + self.to_bytes_within(WriteLimits::SPEC) + } + + /// [`Self::to_bytes`] within `lim`. + fn to_bytes_within(&self, lim: WriteLimits) -> std::io::Result> { // No insignificant whitespace, key order preserved (serde_json's // `preserve_order`), so the bytes round-trip identically. - let m = manifest_json(&self.manifest)?; - let payload_len = u32_field(self.payload.len(), "the payload")?; + let m = manifest_json(&self.manifest, lim.manifest)?; + let payload_len = u32_field(self.payload.len(), lim.field, "the payload")?; let mut out = Vec::with_capacity(12 + m.len() + self.payload.len()); out.extend_from_slice(MAGIC); out.extend_from_slice(&(m.len() as u32).to_le_bytes()); @@ -2530,7 +2549,15 @@ impl WaiMulti { Self::from_bytes(&bytes) } + /// The envelope's bytes. Refuses a manifest longer than SPEC §2's reader limit + /// (`manifest_too_large`), and a payload, or the payloads' total, that does not fit + /// a `u32` field. pub fn to_bytes(&self) -> std::io::Result> { + self.to_bytes_within(WriteLimits::SPEC) + } + + /// [`Self::to_bytes`] within `lim`. + fn to_bytes_within(&self, lim: WriteLimits) -> std::io::Result> { if self.manifest.renditions.len() != self.renditions.len() { return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, format!("manifest has {} renditions but payload list has {}", @@ -2540,7 +2567,7 @@ impl WaiMulti { return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "too many renditions for u16 count field")); } - let m = manifest_json(&self.manifest)?; + let m = manifest_json(&self.manifest, lim.manifest)?; let table_bytes = 2 + self.renditions.len() * 8; let payload_bytes: usize = self.renditions.iter().map(|p| p.len()).sum(); @@ -2554,10 +2581,10 @@ impl WaiMulti { // the start of the payloads block (the byte right after the table). let mut off: u32 = 0; for (i, p) in self.renditions.iter().enumerate() { - let len = u32_field(p.len(), &format!("rendition {i}'s payload"))?; + let len = u32_field(p.len(), lim.field, &format!("rendition {i}'s payload"))?; out.extend_from_slice(&off.to_le_bytes()); out.extend_from_slice(&len.to_le_bytes()); - off = off.checked_add(len).ok_or_else(|| std::io::Error::new( + off = off.checked_add(len).filter(|&o| o <= lim.field).ok_or_else(|| std::io::Error::new( std::io::ErrorKind::InvalidData, "payload sum overflows u32"))?; } for p in &self.renditions { @@ -4399,9 +4426,10 @@ mod tests { /// more, or the most a u32 can declare, as too large, by both readers and by /// `manifest_bytes`. A 2 MiB manifest, the size a staged layer's `prev` reaches over /// some 23 000 units with alternating producers, is read and written back byte for - /// byte. The writers' JSON stops at its limit: a manifest is written at a limit equal - /// to its length and refused one byte below it (at `MANIFEST_MAX_LEN` itself the - /// check would serialise 128 MiB, which a debug build takes many seconds over). + /// byte. Both writers stop at their limits: an envelope is written at limits equal to + /// its manifest's and its payloads' lengths, and refused at one byte less of either + /// (at the real limits the check would serialise 128 MiB, which a debug build takes + /// many seconds over, or hold a 4 GiB payload). #[test] fn a_manifest_past_the_reader_limit_is_refused_and_never_written() { assert_eq!(MANIFEST_MAX_LEN, 134_217_728); @@ -4428,10 +4456,25 @@ mod tests { let w2 = WaiMulti::from_bytes(&multi).expect("a 2 MiB WAI2 manifest is read"); assert_eq!(w2.to_bytes().unwrap(), multi, "and written back byte for byte"); - assert_eq!(manifest_json_within(&w.manifest, len).unwrap(), &one[8..8 + len]); - too_large(manifest_json_within(&w.manifest, len - 1).unwrap_err().to_string()); - assert_eq!(manifest_json_within(&w2.manifest, len).unwrap(), &multi[8..8 + len]); - too_large(manifest_json_within(&w2.manifest, len - 1).unwrap_err().to_string()); + // Each payload is 3 bytes. + let lim = |manifest, field| WriteLimits { manifest, field }; + let past_field = |e: std::io::Error, what: &str| { + let e = e.to_string(); + assert!(e.contains(&format!("{what} is 3 bytes, past the u32 field")), "{e}"); + }; + assert_eq!(w.to_bytes_within(lim(len, 3)).unwrap(), one, "written at its own lengths"); + too_large(w.to_bytes_within(lim(len - 1, 3)).unwrap_err().to_string()); + past_field(w.to_bytes_within(lim(len, 2)).unwrap_err(), "the payload"); + assert_eq!(w2.to_bytes_within(lim(len, 3)).unwrap(), multi, "written at its own lengths"); + too_large(w2.to_bytes_within(lim(len - 1, 3)).unwrap_err().to_string()); + past_field(w2.to_bytes_within(lim(len, 2)).unwrap_err(), "rendition 0's payload"); + // A second rendition of 3 bytes: each fits a field of 5, their total does not. + let mut w3 = WaiMulti::from_bytes(&multi).unwrap(); + w3.manifest.renditions.push(w3.manifest.renditions[0].clone()); + w3.renditions.push(b"def".to_vec()); + assert!(w3.to_bytes_within(lim(MANIFEST_MAX_LEN, 6)).is_ok()); + let e = w3.to_bytes_within(lim(MANIFEST_MAX_LEN, 5)).unwrap_err().to_string(); + assert!(e.contains("payload sum overflows u32"), "{e}"); } /// The manifest's nesting limit the reference readers apply (SPEC §2, *Reader @@ -4462,8 +4505,8 @@ mod tests { #[test] #[cfg(target_pointer_width = "64")] fn a_length_past_a_u32_field_is_refused() { - assert_eq!(u32_field(u32::MAX as usize, "x").unwrap(), u32::MAX); - let e = u32_field(u32::MAX as usize + 1, "the payload").unwrap_err().to_string(); + assert_eq!(u32_field(u32::MAX as usize, u32::MAX, "x").unwrap(), u32::MAX); + let e = u32_field(u32::MAX as usize + 1, u32::MAX, "the payload").unwrap_err().to_string(); assert!(e.contains("the payload is 4294967296 bytes, past the u32 field"), "{e}"); } } -- 2.54.0 (Apple Git-157) From 010ee79b2836e33e71334926cabd785a78c1a57e Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 16:56:04 -0400 Subject: [PATCH 09/15] wai: the HTTP corpus holds the manifest reader limit at its edge No case reached the new limit: with the check deleted from http_verify.parse_head, every case still gave its outcome, and no committed envelope declares a man_len past 2^27. Four parse_head cases now do, on a WAI1 (O5) and a WAI2 (OD): eight bytes declaring 2^27 + 1 are refused (head_malformed), and eight declaring 2^27 ask for the head they declare (need 134217740 and 134217738). Both implementations give all four. falsify/manifest-limit-dropped.patch drops the check from head::parse_head, and http_verify.py --mutate manifest-limit-dropped drops it on the Python side; run.sh runs both. Co-Authored-By: Claude Opus 5.5 --- wai/http-conformance/README.md | 3 ++- wai/http-conformance/cases.json | 4 ++++ .../falsify/manifest-limit-dropped.patch | 14 ++++++++++++++ wai/http-conformance/falsify/run.sh | 2 +- wai/http-conformance/http_verify.py | 4 +++- wai/wai-rs/src/bin/wai_cli/http_vectors.rs | 10 ++++++++++ 6 files changed, 34 insertions(+), 3 deletions(-) create mode 100644 wai/http-conformance/falsify/manifest-limit-dropped.patch diff --git a/wai/http-conformance/README.md b/wai/http-conformance/README.md index d5f2cbae0..90f3e44e9 100644 --- a/wai/http-conformance/README.md +++ b/wai/http-conformance/README.md @@ -162,7 +162,8 @@ each to a fresh worktree, requires the build to succeed, and requires `wai_http_vectors verify` to exit non-zero naming those cases. `http_verify.py --mutate NAME` does the same on the Python side (`left-half`, `no-root`, `counter-zero` fail the proof cases; `floor-as-codecs` fails `union_O9`; `blake3-iv` -fails the self-test; `base-at-done`, `tick-noop` and `refinement-unclipped` fail the +fails the self-test; `manifest-limit-dropped` fails the reader-limit `parse_head` +cases; `base-at-done`, `tick-noop` and `refinement-unclipped` fail the `timed_*` cases; `pdt-ignored`, `dseq-ignored`, `parts-after-extinf`, `average-floor`, `render-digest-off`, `name-unchecked`, `chain-unchecked`, `ilog2-no-interp`, `bola-waits`, `unlabelled-as-zero` and `round-half-down` fail the playlist, receipt and diff --git a/wai/http-conformance/cases.json b/wai/http-conformance/cases.json index 148adc9ce..cecb7761a 100644 --- a/wai/http-conformance/cases.json +++ b/wai/http-conformance/cases.json @@ -12,6 +12,10 @@ {"name":"object_length_short","op":"parse_head","input":{"object":"O3","envelope_len":98614},"expect":{"error":"object_length_mismatch"}}, {"name":"head_malformed_magic","op":"parse_head","input":{"object":"O5","patch_object":[[3,"33"]]},"expect":{"error":"head_malformed"}}, {"name":"head_malformed_offset","op":"parse_head","input":{"object":"OD","patch_object":[[574,"01"]]},"expect":{"error":"head_malformed"}}, +{"name":"head_malformed_manifest_too_large_O5","op":"parse_head","input":{"object":"O5","prefix":8,"patch_object":[[4,"01"],[5,"00"],[6,"00"],[7,"08"]]},"expect":{"error":"head_malformed"}}, +{"name":"parse_head_need_manifest_at_limit_O5","op":"parse_head","input":{"object":"O5","prefix":8,"patch_object":[[4,"00"],[5,"00"],[6,"00"],[7,"08"]]},"expect":{"need":134217740}}, +{"name":"head_malformed_manifest_too_large_OD","op":"parse_head","input":{"object":"OD","prefix":8,"patch_object":[[4,"01"],[5,"00"],[6,"00"],[7,"08"]]},"expect":{"error":"head_malformed"}}, +{"name":"parse_head_need_manifest_at_limit_OD","op":"parse_head","input":{"object":"OD","prefix":8,"patch_object":[[4,"00"],[5,"00"],[6,"00"],[7,"08"]]},"expect":{"need":134217738}}, {"name":"head_proof_ok_O1","op":"head_proof","input":{"object":"O1"},"expect":{"covered_len":859,"n_cv":0,"envelope_len":859}}, {"name":"head_proof_ok_O10","op":"head_proof","input":{"object":"O10"},"expect":{"covered_len":1009,"n_cv":1,"envelope_len":1825}}, {"name":"head_proof_ok_O10L1","op":"head_proof","input":{"object":"O10L1"},"expect":{"covered_len":1009,"n_cv":4,"envelope_len":12711}}, diff --git a/wai/http-conformance/falsify/manifest-limit-dropped.patch b/wai/http-conformance/falsify/manifest-limit-dropped.patch new file mode 100644 index 000000000..7489a7de5 --- /dev/null +++ b/wai/http-conformance/falsify/manifest-limit-dropped.patch @@ -0,0 +1,14 @@ +# must fail: head_malformed_manifest_too_large_O5 head_malformed_manifest_too_large_OD +# a man_len past the reader limit is asked for, not refused (head::parse_head) +diff --git a/wai/wai-rs/src/http_binding/head.rs b/wai/wai-rs/src/http_binding/head.rs +index 83fe9492..53219b7c 100644 +--- a/wai/wai-rs/src/http_binding/head.rs ++++ b/wai/wai-rs/src/http_binding/head.rs +@@ -66,7 +66,6 @@ pub fn parse_head(prefix: &[u8], envelope_len: u64) -> Result return Err(RangeRefusal::HeadMalformed("the magic is neither WAI1 nor WAI2".into())), + }; + let man_len = u32_at(prefix, 4); +- crate::container::check_manifest_len(man_len).map_err(RangeRefusal::HeadMalformed)?; + let manifest = 8..8 + man_len; + let (head_len, components) = match version { + Version::Wai1 => { diff --git a/wai/http-conformance/falsify/run.sh b/wai/http-conformance/falsify/run.sh index df63f4073..b82f10968 100755 --- a/wai/http-conformance/falsify/run.sh +++ b/wai/http-conformance/falsify/run.sh @@ -48,7 +48,7 @@ cd "$ROOT/wai/http-conformance" for m in left-half no-root counter-zero floor-as-codecs blake3-iv base-at-done tick-noop refinement-unclipped \ pdt-ignored dseq-ignored parts-after-extinf average-floor render-digest-off name-unchecked chain-unchecked \ ilog2-no-interp bola-waits unlabelled-as-zero round-half-down small-order-key small-order-r \ - rebind-uri rebind-sequence key-unchecked walk-unbounded reload-walked \ + rebind-uri rebind-sequence key-unchecked walk-unbounded reload-walked manifest-limit-dropped \ xml-char-unchecked xml-attribute-twice dash-start-ignored dash-timescale-ignored \ unlabelled-escapes abr-unbounded gap-carries receipt-duplicates receipt-surrogates session-data-anywhere \ dash-r-expanded xml-attributes-scanned dash-timelines-unbounded budget-raises \ diff --git a/wai/http-conformance/http_verify.py b/wai/http-conformance/http_verify.py index 134cfc107..0868eb7a4 100644 --- a/wai/http-conformance/http_verify.py +++ b/wai/http-conformance/http_verify.py @@ -121,7 +121,9 @@ def parse_head(prefix, n): if magic not in (b"WAI1", b"WAI2"): raise Refusal("head_malformed") ml = u32(prefix, 4) - if ml > SV.MANIFEST_MAX: # SPEC §2, *Reader limits*: refused before it is asked for + # SPEC §2, *Reader limits*: refused before it is asked for (--mutate + # manifest-limit-dropped asks for it). + if ml > SV.MANIFEST_MAX and MUTATE != "manifest-limit-dropped": raise Refusal("head_malformed") if magic == b"WAI1": head_len = 12 + ml diff --git a/wai/wai-rs/src/bin/wai_cli/http_vectors.rs b/wai/wai-rs/src/bin/wai_cli/http_vectors.rs index 3e6da62aa..a62421462 100644 --- a/wai/wai-rs/src/bin/wai_cli/http_vectors.rs +++ b/wai/wai-rs/src/bin/wai_cli/http_vectors.rs @@ -855,6 +855,16 @@ fn cases(objs: &BTreeMap>, extra: &mut BTreeMap> v.push(refused("object_length_short", "parse_head", json!({ "object": "O3", "envelope_len": n("O3") + 1 }), "object_length_mismatch")); v.push(refused("head_malformed_magic", "parse_head", json!({ "object": "O5", "patch_object": [[3, "33"]] }), "head_malformed")); v.push(refused("head_malformed_offset", "parse_head", json!({ "object": "OD", "patch_object": [[574, "01"]] }), "head_malformed")); + // SPEC §2, *Reader limits*: a `man_len` of 2^27 + 1 is refused from the first eight + // bytes, and one of 2^27 asks for the head it declares. + let man_len = |v: u32| json!([[4, format!("{:02x}", v as u8)], [5, format!("{:02x}", (v >> 8) as u8)], + [6, format!("{:02x}", (v >> 16) as u8)], [7, format!("{:02x}", (v >> 24) as u8)]]); + for id in ["O5", "OD"] { + v.push(refused(format!("head_malformed_manifest_too_large_{id}"), "parse_head", + json!({ "object": id, "prefix": 8, "patch_object": man_len((1 << 27) + 1) }), "head_malformed")); + v.push(case(format!("parse_head_need_manifest_at_limit_{id}"), "parse_head", + json!({ "object": id, "prefix": 8, "patch_object": man_len(1 << 27) }))); + } // head_proof: every object, and every way a proof can be wrong. for id in objs.keys() { -- 2.54.0 (Apple Git-157) From 55df2cdc2c0facbe018d485baeb5f0b1727a9ab4 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 17:03:13 -0400 Subject: [PATCH 10/15] wai: the Python verifiers check their manifest reader limit in memory In staged_verify, claim_verify and pack_verify the man_len check is outcome-neutral for any input under 128 MiB: the bounds check after it refuses the same bytes. No corpus file is that large, so nothing exercised it. Each verifier now builds, in memory, a WAI1 and a WAI2 (claim_verify: the WAI2 its wai2_read reads) whose manifest is padded to 2^27 + 1 bytes by a member no revision defines, and requires it refused, and requires the same envelope with a 300-byte manifest read, so only the limit tells the two apart. pack_verify builds its own envelopes, as it shares no code. Co-Authored-By: Claude Opus 5.5 --- wai/claim-conformance/claim_verify.py | 3 ++ wai/pack-conformance/pack_verify.py | 20 +++++++++++ wai/staged-conformance/staged_verify.py | 46 +++++++++++++++++++++++++ 3 files changed, 69 insertions(+) diff --git a/wai/claim-conformance/claim_verify.py b/wai/claim-conformance/claim_verify.py index 79d3ecb26..7a42055ac 100644 --- a/wai/claim-conformance/claim_verify.py +++ b/wai/claim-conformance/claim_verify.py @@ -1963,6 +1963,9 @@ def main(argv): bm = brackets_match_wai_rs() print(("PASS" if not bm else "FAIL") + " bracket arithmetic equals the literals pinned in wai-rs " + ",".join(bm)) fail += bool(bm) + rl = SV.reader_limit_failures(lambda b: wai2_read(b) is not None, True) + print(("PASS" if not rl else "FAIL") + " wai2_read refuses a manifest past the reader limit (SPEC §2) " + "; ".join(rl)) + fail += bool(rl) rd = OER.check_canonical_form() # the OER reference's own fixtures, its reading of -0 among them print(("PASS" if not rd else "FAIL") + " the OER reference's canonical form and number reading hold " + ",".join(rd)) fail += bool(rd) diff --git a/wai/pack-conformance/pack_verify.py b/wai/pack-conformance/pack_verify.py index bfbe279dc..bbe78dcff 100644 --- a/wai/pack-conformance/pack_verify.py +++ b/wai/pack-conformance/pack_verify.py @@ -1476,6 +1476,25 @@ def read_envelope(b): return (ml, total, rs[0], rs[0].get("capability"), parts) if len(b) == head + total else None +def check_reader_limit(rep): + """SPEC §2, *Reader limits*, checked in memory, since no file in the corpus is past + 128 MiB: read_envelope reads a WAI1 and a WAI2 whose manifests are 300 bytes, and + refuses each with its manifest padded to MANIFEST_MAX + 1 bytes by a member no + revision defines, which a reader ignores (SPEC §8), so only the limit refuses it.""" + for magic, head, rest in ( + (b"WAI1", b'{"wai":"1.0","media":"text","intent":"replicate","model_requirement":' + b'{"capability":"wai.text.zstd"},"conditioning":{"kind":"zstd"},"pad":"', + (3).to_bytes(4, "little")), + (b"WAI2", b'{"wai":"1.1","media":"text","intent":"replicate","renditions":' + b'[{"capability":"wai.text.zstd","kind":"zstd"}],"pad":"', + (1).to_bytes(2, "little") + (0).to_bytes(4, "little") + (3).to_bytes(4, "little"))): + def env(n): + return b"".join([magic, n.to_bytes(4, "little"), head, b"x" * (n - len(head) - 2), b'"}', rest, b"abc"]) + rep.check(read_envelope(env(300)) is not None, f"reader limit: a {magic.decode()} with a 300-byte manifest is read") + rep.check(read_envelope(env(MANIFEST_MAX + 1)) is None, + f"reader limit: a {magic.decode()} manifest of {MANIFEST_MAX + 1} bytes, past the limit, is refused") + + WIR1_KIND = {"wai.image.int_refine": "rgb8", "wai.video.int_refine": "rgb8", "wai.audio.int_refine": "pcm", "wai.splat.int_refine": "tensor"} @@ -3098,6 +3117,7 @@ def main(argv): print(__doc__) return 2 rep = verify(argv[1]) + check_reader_limit(rep) print(f"pack_verify: {rep.checked - rep.failures}/{rep.checked} checks hold") return 0 if rep.failures == 0 else 1 diff --git a/wai/staged-conformance/staged_verify.py b/wai/staged-conformance/staged_verify.py index 7376717a0..fa14dc34a 100644 --- a/wai/staged-conformance/staged_verify.py +++ b/wai/staged-conformance/staged_verify.py @@ -1209,6 +1209,48 @@ def parse_wai2(b): return text, payloads +def envelope_with_manifest_len(v2, n): + """A WAI1 (or, with `v2`, a WAI2) envelope whose manifest is exactly `n` bytes: a small + manifest with one member no revision defines, `pad`, filled to length (a reader + ignores such a member, SPEC §8), over a 3-byte payload. Past MANIFEST_MAX, only the + reader limit tells it from one that reads.""" + if v2: + head = (b'{"wai":"1.1","media":"text","intent":"replicate","renditions":' + b'[{"capability":"wai.text.zstd","kind":"zstd"}],"target":null,"pad":"') + rest = struct.pack(" Date: Fri, 9 Oct 2026 17:14:52 -0400 Subject: [PATCH 11/15] wai-web: pack and packMulti refuse a payload past its u32 field SECURITY.md and the CHANGELOG named wai-web's writers among those that refuse a payload whose length does not fit its u32 field, but they still wrote `payload.len() as u32` and `p.len() as u32`; packMulti's checked_add caught only a total that overflows, and the claim held only because usize is 32 bits on wasm32. Both now convert with u32::try_from, as wai-rs's writers do, and name the payload that does not fit. verify_pack.mjs: the refused manifest is 2^27 + 1 bytes, one past the limit; the comment read as if it were 2^27. Co-Authored-By: Claude Opus 5.5 --- wai/wai-web/demo/verify_pack.mjs | 6 +++--- wai/wai-web/src/lib.rs | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/wai/wai-web/demo/verify_pack.mjs b/wai/wai-web/demo/verify_pack.mjs index 6724d1c00..fc3de67bc 100644 --- a/wai/wai-web/demo/verify_pack.mjs +++ b/wai/wai-web/demo/verify_pack.mjs @@ -74,9 +74,9 @@ refusesTwice('pack, a member twice in model_requirement', () => wai.pack(v1Text.replace('"capability":"wai.image.png"', '"capability":"wai.image.png","q":1,"q":2'), bytes)); refusesTwice('packMulti, a member twice in an entry', () => wai.packMulti(v2([['wai.video.av1']]).replace('"kind":"bytes"', '"kind":"bytes","zz":1,"zz":2'), [bytes])); -// SPEC §2, *Reader limits*: a manifest of 3 MiB is written as given; one byte past the -// limit, 2^27 bytes, is refused (`manifest_too_large`), as every reference reader -// refuses it. +// SPEC §2, *Reader limits*: a manifest of 3 MiB is written as given; one of 2^27 + 1 +// bytes, one byte past the limit, is refused (`manifest_too_large`), as every +// reference reader refuses it. const MAX = 1 << 27; const LONG = 3 << 20; // `text` with a member `pad` added so that it is exactly `len` bytes (all ASCII). diff --git a/wai/wai-web/src/lib.rs b/wai/wai-web/src/lib.rs index 76711eae0..853749644 100644 --- a/wai/wai-web/src/lib.rs +++ b/wai/wai-web/src/lib.rs @@ -514,6 +514,13 @@ fn pack_multi_refusal(manifest: &wai::container::ManifestV2, payloads: &[Vec Ok(()) } +/// A length the writers put in a `u32` field (SPEC §2, §7.1), or an error naming the +/// field when it does not fit, as `wai-rs`'s writers refuse it. (On wasm32 a `usize` +/// is 32 bits, so every length fits; the check holds on any target.) +fn u32_field(len: usize, what: &str) -> Result { + u32::try_from(len).map_err(|_| JsError::new(&format!("{what} is {len} bytes, past the u32 field that holds it"))) +} + /// Build a WAI envelope from a manifest JSON string + a payload byte /// array. Mirrors `wai_envelope_pack` from the native SDK so JS code /// can produce .wai files in the browser too: like it, refuses a manifest @@ -529,11 +536,12 @@ pub fn pack(manifest_json: &str, payload: &[u8]) -> Result, JsError> { let m: wai::container::Manifest = read_manifest(manifest_json).map_err(|e| JsError::new(&e))?; pack_refusal(&m, payload).map_err(|e| JsError::new(&e))?; let mb = manifest_json.as_bytes(); + let payload_len = u32_field(payload.len(), "the payload")?; let mut out = Vec::with_capacity(12 + mb.len() + payload.len()); out.extend_from_slice(wai::container::MAGIC); out.extend_from_slice(&(mb.len() as u32).to_le_bytes()); out.extend_from_slice(mb); - out.extend_from_slice(&(payload.len() as u32).to_le_bytes()); + out.extend_from_slice(&payload_len.to_le_bytes()); out.extend_from_slice(payload); Ok(out.into_boxed_slice()) } @@ -708,10 +716,11 @@ pub fn pack_multi(manifest_json: &str, payloads: js_sys::Array) -> Result Date: Fri, 9 Oct 2026 17:14:52 -0400 Subject: [PATCH 12/15] check-wai-section-refs: read wai/SECURITY.md too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SECURITY.md cites sections of SPEC, http-delivery, integer-payloads and jwp-receipts, and nothing checked them: the checker read only SPEC.md and extensions/*.md, and the workflow's paths left SECURITY.md out. It reads SECURITY.md now, and the workflow runs when it changes. Every reference in it resolves; a reference to http-delivery §66 put in by hand fails the check. CHANGELOG.md stays out: it is a dated record whose older references are kept as written. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/wai-section-refs.yml | 4 +++- scripts/check-wai-section-refs.py | 8 +++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.github/workflows/wai-section-refs.yml b/.github/workflows/wai-section-refs.yml index c1ecff11c..5352e46bc 100644 --- a/.github/workflows/wai-section-refs.yml +++ b/.github/workflows/wai-section-refs.yml @@ -1,6 +1,6 @@ name: WAI section references -# Every section that SPEC.md or an extension cites in another WAI text exists: +# Every section that SPEC.md, an extension or SECURITY.md cites in a WAI text exists: # a link followed by §N, a link whose text holds §N, or an extension's name (or # SPEC) in prose followed by §N, including references that break across lines. # A citation of a section that was never written — as transparency §6's pointer @@ -34,12 +34,14 @@ on: paths: - 'wai/extensions/**' - 'wai/SPEC.md' + - 'wai/SECURITY.md' - 'scripts/check-wai-section-refs.py' - '.github/workflows/wai-section-refs.yml' pull_request: paths: - 'wai/extensions/**' - 'wai/SPEC.md' + - 'wai/SECURITY.md' - 'scripts/check-wai-section-refs.py' - '.github/workflows/wai-section-refs.yml' diff --git a/scripts/check-wai-section-refs.py b/scripts/check-wai-section-refs.py index e4a7a0cb1..5acf2ab72 100644 --- a/scripts/check-wai-section-refs.py +++ b/scripts/check-wai-section-refs.py @@ -1,9 +1,10 @@ #!/usr/bin/env python3 """Every section a WAI specification text cites in another WAI text exists. -Reads wai/SPEC.md and wai/extensions/*.md. In each, a reference to a section of -SPEC.md or of an extension is checked against the headings of the file it names. Three -spellings are recognised, including where the reference breaks across lines: +Reads wai/SPEC.md, wai/extensions/*.md and wai/SECURITY.md. In each, a reference to a +section of SPEC.md or of an extension is checked against the headings of the file it +names. Three spellings are recognised, including where the reference breaks across +lines: [text](X.md) §N a link, then the section [text §N](X.md) the section inside the link text @@ -37,6 +38,7 @@ def sections(path): def texts(): out = [os.path.join(WAI, "SPEC.md")] out += sorted(os.path.join(EXT, n) for n in os.listdir(EXT) if n.endswith(".md")) + out.append(os.path.join(WAI, "SECURITY.md")) # it cites the texts it points readers to return out -- 2.54.0 (Apple Git-157) From 59ab0d9489883b21e3d0bf1a55481ce501f50350 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Fri, 9 Oct 2026 17:16:04 -0400 Subject: [PATCH 13/15] =?UTF-8?q?wai:=20SECURITY.md,=20SPEC=20=C2=A72=20an?= =?UTF-8?q?d=20=C2=A76.5=20say=20what=20holds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SECURITY.md: - "Allocation follows the input" was false for the manifest. The reference readers parse it into serde_json::Value; measured on the release build, a 94 MB prev of 2^20 runs peaks at about 7 times its length and a 134,217,000-byte array of zeros at about 38 times (5.1 GB RSS, 7.6 s in from_bytes, which returned Ok). The text now says so, and the argument that a sink short of that memory could not decode such layers either is gone. - CI does link ONNX Runtime and libzstd: wai-neural-gates.yml and wai-neural-e2e.yml build `neural`. No workflow builds `codecs`. - No Kani harness is in int_kernels.rs; the harnesses are in rans.rs, int_entropy.rs, int_transform.rs, wai_video_policy.rs, staged/predict.rs and staged/wir1_arith.rs. Its SIMD kernels, the only unsafe outside the C ABI, are held to the scalar kernels by equality tests, not by a model checker. - The bare "(§5.1)" names http-delivery; the writer test is described as it now runs; the Python verifiers' checks are described as they now run. SPEC §2: the 2^20-unit cap is staged-delivery Appendix A.3's (section 3 bounds a count only by 32 bits); the limit bounds what a reader reads, not the memory its parse takes; the paragraph is reflowed. SPEC §6.5: Lane A's records are "among them", and include wai.video.ssai and wai.video.channel. Lane B no longer says every record applies: staged-measurement counts no encrypted object, and a verifier without the key checks no payload-dependent rule of a reconstruct-session claim (jwp-receipts §2.4.12 rule 3); nothing in the reference states a Lane B tier. The confidentiality header and the CHANGELOG say the same. SPEC §5 and determinism-tiers §5: only `photo` comes from a trained model; gradient, bloom, facade and the cross-architecture golden use seeded integer weights, as the CHANGELOG said both texts did. Co-Authored-By: Claude Opus 5.5 --- wai/CHANGELOG.md | 46 ++++++++++++-------- wai/SECURITY.md | 55 ++++++++++++++++-------- wai/SPEC.md | 47 ++++++++++++-------- wai/extensions/confidentiality.md | 8 ++-- wai/extensions/determinism-tiers.md | 2 +- wai/wai-rs/tests/neural_native_decode.rs | 11 +++-- 6 files changed, 104 insertions(+), 65 deletions(-) diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 9647f073f..b17e2903c 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -158,10 +158,11 @@ wire format may change and such changes are recorded here. - **`int_synth`'s weights are quantised after float training.** SPEC §5 and [determinism-tiers](extensions/determinism-tiers.md) §5 said the capability decodes - "QAT weights". No committed model was trained that way: the trained corpus vector was - made by `tools/wai_int_synth_real.py`, float training then post-training quantisation - with each layer's shift derived from its measured activation range, and the synthetic - vectors' weights come from a seed. Both texts now say so, and so do the tools: + "QAT weights". No committed model was trained that way: the trained corpus vector, + `photo`, was made by `tools/wai_int_synth_real.py`, float training then post-training + quantisation with each layer's shift derived from its measured activation range, and + the weights of the synthetic vectors and of the cross-architecture golden come from a + seed. Both texts now say so, and so do the tools: `tools/wai_int_synth_qat.py` is marked experimental (it trains against one fixed shift and made no committed model), and `tools/wai_int_synth_export.py` names the working path. @@ -171,13 +172,17 @@ wire format may change and such changes are recorded here. send macOS builders to a `.cargo/config.toml` that no longer exists: `wai-rs/build.rs` finds libFLAC. - **Protected content: two lanes** (SPEC §6.5). Lane A, hardware-protected content, - decodes inside the platform's protected media path; WAI's records about it (the served - manifest, the segment map, the key-release decisions, the content hashes of WAI - objects) are all WAI attests, and none covers a decoded frame. This revision registers - no `security.encryption` value for it. Lane B, end-to-end encryption at the - application layer (`mls-sframe`), decrypts in the application, so every WAI record and - tier applies; it is not a protected-path robustness level. `wai.video.keys` and the - confidentiality extension say which lane they belong to. + decodes inside the platform's protected media path; WAI's records about it (among + them the served manifest, the segment map, the key-release decisions, the ad + decisions and impressions of `wai.video.ssai`, the program-as-run windows of + `wai.video.channel`, and the content hashes of WAI objects) cover no decoded frame. + This revision registers no `security.encryption` value for it. Lane B, end-to-end + encryption at the application layer (`mls-sframe`), decrypts in the application, so + the tier and the output digests of the sink that holds the key hold as they do for + plaintext; staged-measurement counts no encrypted object, and a verifier without the + key checks none of a `reconstruct-session` claim's rules that need a payload. Lane B + is not a protected-path robustness level. `wai.video.keys` and the confidentiality + extension say which lane they belong to. - **Where the exact tier holds** (determinism-tiers §5). `decode-equivalence` is shown on native CPU and on wasm32, on no GPU or NPU; a WebAssembly decoder at the tier uses no relaxed-SIMD instruction and depends on no NaN's bits; and the tier never covers @@ -185,17 +190,22 @@ wire format may change and such changes are recorded here. - **Reader limits** (SPEC §2). A reader need accept no manifest longer than 2²⁷ bytes (128 MiB) or nested deeper than 127, and refuses a longer one from `man_len` alone (`manifest_too_large`). The length limit is sized to the largest manifest a - conforming staged layer has: a `prev` naming a different producer for each of 2²⁰ - units is about 94 MB. `wai-rs`'s readers, its HTTP head parser (`head_malformed`) and - the Python verifiers apply the length limit; the writers, `wai-web`'s included, refuse - to write past it, and refuse a payload whose length does not fit its `u32` field. + conforming staged layer has: a `prev` (staged-delivery §3) naming a different + producer for each of the 2²⁰ units a `WIR1` layer may cover (staged-delivery + Appendix A.3) is about 94 MB. It bounds what a reader reads, not the memory a parse + takes, which in the reference readers is up to nearly 40 times the manifest's length + (`SECURITY.md`). `wai-rs`'s readers, its HTTP head parser (`head_malformed`) and the + Python verifiers apply the length limit, and the HTTP corpus holds it at its edge; + the writers, `wai-web`'s included, refuse to write past it, and refuse a payload whose + length does not fit its `u32` field. *Envelopes this makes non-conforming:* none of the layout's; an envelope whose manifest is longer than 128 MiB, or nested deeper than 127, is one a conforming sink may refuse, so a conforming encoder no longer writes it (SPEC §7). - **`SECURITY.md`.** How to report a vulnerability, and where the reference - implementation reads untrusted input: the manifest limits, the native codec libraries - and their gaps (unbounded decompression output for `wai.text.zstd` and `wai.text.xz`; - no fuzzing in CI), module authorisation, and the fingerprinting risk of `held_sets`. + implementation reads untrusted input: the manifest limits and the memory a manifest's + parse takes, the native codec libraries and their gaps (unbounded decompression output + for `wai.text.zstd` and `wai.text.xz`; no fuzzing in CI), module authorisation, and the + fingerprinting risk of `held_sets`. CI checks its section references. ### One Ed25519 rule for every signature diff --git a/wai/SECURITY.md b/wai/SECURITY.md index 32c3d3e69..bbaedeab9 100644 --- a/wai/SECURITY.md +++ b/wai/SECURITY.md @@ -47,9 +47,21 @@ What the readers bound: | an HTTP object | exactly one envelope; any other length is refused (`object_length_mismatch`) | http-delivery §5.1 | | a head proof | at most 64 chaining values; `covered_len` is the object's length, or shorter and ending on a chunk boundary of the hashed input | http-delivery §6 | -Allocation follows the input: a reader copies a payload only after its bounds are -checked, so it never allocates more than it was given. A `WAI1` reader ignores bytes -after the payload; an HTTP object that carries any is refused (§5.1). +A reader copies a payload only after its bounds are checked, so a payload never makes +it allocate more than it was given. A `WAI1` reader ignores bytes after the payload; an +HTTP object that carries any is refused ([http-delivery](extensions/http-delivery.md) +§5.1). + +**Parsing a manifest takes more memory than its length.** The reference readers parse +the manifest into a JSON tree (`serde_json::Value`, for `target`, `refines`, +`derived_from` and every member this revision does not define), and a parsed value +takes more memory than its text: about seven times the length of a 94 MB `prev` that +names a producer for each of 2²⁰ units, the largest manifest a conforming layer has, +and nearly 40 times the length of a manifest of many small values, such as one long +array of zeros. A manifest at the length limit can take some 5 GB and several seconds +to parse, and a reader may not refuse it for its length (SPEC §2). The length limit +bounds what a reader must read, not what parsing it costs; the reference sets no limit +of its own on the parse. The library tests in `container.rs` and `http_binding/tests.rs` check the manifest limits at their edges: from its first eight bytes, every reader refuses a manifest one @@ -58,19 +70,18 @@ byte past the length limit as too large and one at the limit only as truncated; 128 are refused. The writers (`Wai::to_bytes`, `WaiMulti::to_bytes`, and `pack` and `packMulti` in `wai-web`) refuse a manifest past the length limit, and a payload whose length does not fit its `u32` field. The Rust writers stop serialising as soon as the -manifest passes the limit; the test checks that at a small limit, since serialising -128 MiB takes a debug build many seconds. `verify_pack.mjs` checks the browser writers -at the real limit. - -The length limit bounds what a reader is asked to hold, not what a sink can afford: a -sink with less memory than a 128 MiB manifest needs cannot decode the layers that -reach that size either, and refuses them by its own cost limits. +manifest passes the limit; the test runs both writers at limits equal to an envelope's +manifest and payload lengths and one byte below them, since serialising 128 MiB takes a +debug build many seconds. `verify_pack.mjs` checks the browser writers at the real +limit. The stdlib Python verifiers read envelopes too. `staged_verify.py`, `claim_verify.py`, `pack_verify.py` and `http_verify.py` refuse a manifest past the -length limit as the reference does. Those that read manifests through -`staged_verify.strict_json` refuse nesting past 127; `pack_verify.py` reads up to the -interpreter's recursion limit and refuses past it as unparseable. +length limit as the reference does. `http_verify.py` and `wai_http_vectors` agree on +`parse_head` at the limit in the HTTP corpus; the other three each build an envelope in +memory with a manifest one byte past it and require it refused. Those that read +manifests through `staged_verify.strict_json` refuse nesting past 127; `pack_verify.py` +reads up to the interpreter's recursion limit and refuses past it as unparseable. Payload formats that WAI defines carry their own caps, checked before the first symbol is decoded: a tensor of at most 2²⁶ elements, a `WIV1` clip of at most 2³⁰ decoded @@ -97,8 +108,11 @@ run it in a WebAssembly runtime. What does not touch them: -- The builds CI runs (`--no-default-features` with named features) link none of - them. +- CI builds, except the neural workflows: no workflow enables `codecs` or `semantic`, + so no CI build links the codec libraries. `wai-neural-gates.yml` and + `wai-neural-e2e.yml` build the `neural` feature, which links ONNX Runtime and + libzstd, and run the float neural decoders through them; no other workflow's build + links any of these libraries. - WAI's own decoders are Rust: the integer codecs, worlds, scores, films and the generator. Outside the C ABI (`ffi.rs`, `ffi_codecs.rs`, `ffi_neural.rs`), the only `unsafe` in `wai-rs`'s library code is in the AArch64 SIMD kernels of @@ -107,9 +121,14 @@ What does not touch them: browser's WebAssembly sandbox. Model checking covers part of the integer core: Kani harnesses beside the code in -`rans.rs`, `int_entropy.rs`, `int_transform.rs`, `int_kernels.rs`, -`wai_video_policy.rs` and `staged/`, hosted by `wai-verify` and run by -`.github/workflows/wai-formal-verification.yml`. +`rans.rs`, `int_entropy.rs`, `int_transform.rs`, `wai_video_policy.rs`, +`staged/predict.rs` and `staged/wir1_arith.rs`, hosted by `wai-verify` and run by +`.github/workflows/wai-formal-verification.yml`. `wai-verify` also compiles +`int_kernels.rs`, on its scalar path, because `int_transform.rs` calls it, but no +harness reaches it: its AArch64 SIMD kernels, the only `unsafe` named above, are not +model-checked. Equality tests hold them to the scalar reference kernels, element by +element beside each kernel and over every committed integer-decoder vector +(`tests/int_kernels_vectors.rs`), on every platform leg of `wai-conformance.yml`. Known gaps: diff --git a/wai/SPEC.md b/wai/SPEC.md index a0f432e56..be9345ec8 100644 --- a/wai/SPEC.md +++ b/wai/SPEC.md @@ -89,12 +89,14 @@ All WAI multi-byte integers are little-endian unless stated otherwise. **Reader limits.** A reader MUST NOT refuse a manifest for its length or its nesting while it is at most 134 217 728 bytes (`man_len` ≤ 2²⁷) and its arrays and objects nest at most 127 deep, the manifest object itself counting as one. It MAY refuse a longer or -deeper manifest. The length limit holds the largest manifest a conforming layer has -([staged-delivery](extensions/staged-delivery.md) §3): a `prev` that names a different -producer for each of the 2²⁰ units a layer may cover is about 94 MB. A reader that refuses one past the length limit does so from `man_len` -alone, before it reads or requests the manifest (`manifest_too_large`). The reference -readers refuse both. The same limits apply to the `WAI2` manifest (§7.1). They bound a -reader's work and move no byte of the layout above, which stays frozen (§8). Since a +deeper manifest. The length limit holds the largest manifest a conforming layer has: a +`prev` ([staged-delivery](extensions/staged-delivery.md) §3) that names a different +producer for each of the 2²⁰ units a `WIR1` layer may cover (staged-delivery +Appendix A.3, `n_units` ≤ 2²⁰) is about 94 MB. A reader that refuses one past the +length limit does so from `man_len` alone, before it reads or requests the manifest +(`manifest_too_large`). The reference readers refuse both. The same limits apply to the +`WAI2` manifest (§7.1). They bound what a reader must read, not the memory its parse +takes, and move no byte of the layout above, which stays frozen (§8). Since a conforming encoder emits only containers every conforming sink can read (§7), a conforming encoder writes no manifest past either limit; the reference writers refuse one past the length limit. @@ -614,7 +616,7 @@ registration and makes no conforming envelope non-conforming (§8). |---|---|---|---| | `wai.neural.bmshj2018` | bmshj2018-factorized + rANS (byte-exact CompressAI port) | about 0.26 bpp at 29 dB PSNR (quality 3) | CompressAI | | `wai.neural.mbt2018_mean` | mean-scale hyperprior (Gaussian-conditional entropy) | the conditional-entropy family (arXiv 1809.02736) | `neural/hyperprior.rs` | -| `wai.neural.int_synth` | integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) | bit-identical on every machine — soundly backs `intent = replicate` | `int_synth.rs` (integer weights sink-supplied; the reference tools make them by training in float, then quantising after training with per-layer shifts calibrated on measured activations, `tools/wai_int_synth_train.py`) | +| `wai.neural.int_synth` | integer fixed-point synthesis (all-i64 transposed-conv, no float/ONNX) | bit-identical on every machine — soundly backs `intent = replicate` | `int_synth.rs` (integer weights sink-supplied; the reference tools make them by training in float, then quantising after training with per-layer shifts calibrated on measured activations, `tools/wai_int_synth_train.py`. Of the committed vectors only `photo` comes from a trained model, `tools/wai_int_synth_real.py`; `gradient`, `bloom`, `facade` and the cross-architecture golden use seeded integer weights) | | `wai.neural.int_hyper` | integer fixed-point **conditional** codec — whole bitstream→image path integer (integer hyper-decoder σ → integer-σ entropy → integer synthesis) | the integer-entropy contract fully assembled: a PTQ of mbt2018_mean, bit-identical on every machine, whose integer decode CI holds within 1 dB of the same model's float decode on a pinned photograph | payload `WIH1`; `int_hyper_synth.rs` (tables sink-supplied) | | `wai.neural.int_mlicpp` | integer fixed-point **MLIC++ entropy topology** — `K` channel slices of the latent, each decoded as a checkerboard anchor pass then a non-anchor pass, with an entropy network over concat[latent decoded so far, hyper] re-run before every pass to give σ‖μ; hyper-decoder, entropy network and synthesis all integer | the channel-autoregressive × checkerboard entropy model at **decode-equivalence**: the encoder and every decoder compute the same σ, so the same distribution for every element, at any thread count; bit-identical on every machine. The committed conformance vector has synthetic weights, so it demonstrates decode-equivalence and states no quality figure | `int_mlicpp.rs` (one-file model bundle sink-supplied; `tools/wai_int_mlicpp_export.py`) | | `wai.image.jpegai` | JPEG AI (ISO/IEC 6048-1:2025) codestream → RGB — integer entropy front end (mANS, with an integer scale-index derivation) + 16-bit-PTQ integer hyper-decoder, context-model and synthesis networks + dequantisation, latent refinement, post-filters (the eICCI CNN among them) and colour conversion in IEEE-754 float with a fixed evaluation order. Scope: one configuration — 256×256 4:4:4, `model_id` 0 (tools_0), `beta_displacement_log` 0, the high-operation-point synthesis, the eICCI model pair the weight bundle carries, and the other header conditions in [determinism-tiers](extensions/determinism-tiers.md) §5; both decode paths refuse any other codestream. Within it, a sink without the integer weight bundle, or in a non-default floating-point environment, takes a float ONNX decode instead. Only checkpoint constants are sink-supplied | **entropy-consistency** (registry class `NeuralEntropyExact`): the entropy decode, including the scale-index derivation, is integer on both decode paths, so the decoded symbols reproduce on every machine. The exact decoder also pins one reconstruction to RGB, reproducible across machines under the build and floating-point conditions in [determinism-tiers](extensions/determinism-tiers.md) §5–§6 (it checks the floating-point environment before decoding) — a property of that decoder, not of the capability, so `wai.image.jpegai` does not back `replicate` (§7). On one reference codestream the exact decoder stays within max\|Δ\| ≤ 2 of the reference decoder (recorded max\|Δ\| = 1 on ~4.4% of samples); evidence in [determinism-tiers](extensions/determinism-tiers.md) §6. **Downgraded in v0.4.0:** this row previously claimed decode-equivalence, and the registry classed the capability `StandardDefined` | `jpegai_decode.rs` (+ `jpegai_{bitstream,mans,pipeline,synthesis,icci}.rs`) | @@ -1186,12 +1188,14 @@ and decode happen inside that path. envelope carries Lane A media: it travels in the form its protection system defines, and WAI carries records about it. An envelope that names an encryption the sink does not implement is inert at that sink (§6.4 step 2). -- The records are the whole of what WAI attests about a Lane A title: the manifest a - session was served (`wai.video.manifest`), the package's segment map over the bytes +- WAI attests about a Lane A title only through its records, among them the manifest + a session was served (`wai.video.manifest`), the package's segment map over the bytes it sliced (`wai.video.package`), each key-release decision, which binds the released - key by commitment and never carries it (`wai.video.keys`), and the content hash of - each WAI object delivered (jwp-receipts §2.1). None of them covers a decoded frame - or sample. + key by commitment and never carries it (`wai.video.keys`), the ad decision and the + per-impression receipts that bind each served `creative_hash` and the quartile it + reached (`wai.video.ssai`), a channel's program-as-run windows (`wai.video.channel`), + and the content hash of each WAI object delivered (jwp-receipts §2.1). None of them + covers a decoded frame or sample. - A capability that synthesises output from parameters (grain, an enhancement layer, a generative stage) MAY apply to Lane A content only inside the protected path, in a decoder that the protection system certifies. WAI does not certify that decoder and @@ -1204,13 +1208,18 @@ object under a group key, `mls-sframe` (§6.1–§6.4, it and runs §4 on the plaintext, so any decoder a sink has can decode it: native code, WebAssembly, or a neural decoder. -- Every WAI record applies as it does to plaintext: the receipt chain over the - ciphertext (§6.3), the capability's tier over the decoded output - ([determinism-tiers](extensions/determinism-tiers.md) §2), the output digests of a - `reconstruct-session` claim ([jwp-receipts](extensions/jwp-receipts.md) §2.4.12), and - reconstruction and energy bindings - ([reconstruction-binding](extensions/reconstruction-binding.md), - [energy-binding](extensions/energy-binding.md) §3). +- The sink that holds the key decodes the plaintext, so the capability's tier over its + decoded output ([determinism-tiers](extensions/determinism-tiers.md) §2) and the + output digests it states in a `reconstruct-session` claim + ([jwp-receipts](extensions/jwp-receipts.md) §2.4.12) hold as they do for plaintext, + and the receipt chain covers the ciphertext (§6.3). Not every record carries over: + staged-measurement counts no encrypted object and publishes no byte figure for one + ([staged-measurement](extensions/staged-measurement.md) §3, *Only a checked format + is counted*, code `encrypted`), and a verifier of a `reconstruct-session` claim that + holds no key checks none of the rules that need a payload, which stay the signer's + statement (jwp-receipts §2.4.12, rule 3). The reference dispatchers implement no + scheme (§6.4 step 2), so nothing in the reference states a tier, an output digest or + a binding for Lane B content. - **Lane B is not a protected-path robustness level.** It keeps the content from relays and from anyone outside the group (§6.2, §6.3). It does not keep it from a member's device: the base key, the plaintext and the decoded output are in the application's diff --git a/wai/extensions/confidentiality.md b/wai/extensions/confidentiality.md index 47254347c..8ecdba045 100644 --- a/wai/extensions/confidentiality.md +++ b/wai/extensions/confidentiality.md @@ -6,9 +6,11 @@ > key agreement) crates, with browser bindings in `joule-comm-mls-wasm`. > Live demonstration: https://wai.transaction.science/private-channel > -> This is Lane B of SPEC §6.5: end-to-end encryption at the application layer, -> under which every WAI record and tier applies to the decoded output. It is not a -> protected-path robustness level (SPEC §6.5). +> This is Lane B of SPEC §6.5: end-to-end encryption at the application layer. The +> sink that holds the key decodes the plaintext, so the tier and the output digests it +> states hold as they do for plaintext; staged-measurement counts no encrypted object, +> and a verifier without the key checks none of a claim's rules that need a payload +> (SPEC §6.5). It is not a protected-path robustness level. Keywords **MUST**, **MUST NOT**, **SHOULD**, **MAY**, **REQUIRED**, **OPTIONAL** are RFC 2119 / RFC 8174. diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index 48c417f94..a36113b4e 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -355,7 +355,7 @@ tier costs them. | capability | hashable medium | how the reconstruction is pinned | |---|---|---| -| `wai.neural.int_synth` | RGB | integer synthesis (post-training quantisation of a float model, with per-layer shifts calibrated on measured activations) | +| `wai.neural.int_synth` | RGB | integer synthesis (post-training quantisation of a float model, with per-layer shifts calibrated on measured activations; of the committed vectors only `photo` is such a model, and `gradient`, `bloom`, `facade` and the cross-architecture golden of §6 use seeded integer weights) | | `wai.neural.int_hyper` | RGB | integer hyper-decoder, integer scale → distribution step, integer synthesis (post-training quantisation of a float model) | | `wai.video.int_hyper` | per-frame | an `int_hyper` keyframe, then integer motion-compensated P-frames | | `wai.audio.int_codec` | int16 PCM | integer synthesis over a factorized rANS latent | diff --git a/wai/wai-rs/tests/neural_native_decode.rs b/wai/wai-rs/tests/neural_native_decode.rs index f17614894..2d2480c90 100644 --- a/wai/wai-rs/tests/neural_native_decode.rs +++ b/wai/wai-rs/tests/neural_native_decode.rs @@ -1118,12 +1118,11 @@ fn snac_real_model_matches_reference() { assert!(maxd < 1e-3, "real snac decode diverges (max {maxd})"); } -/// int_synth against a REAL trained model, quantised after training (not the constructed unit -/// test): a float autoencoder trained on synthetic images, then quantized to -/// the integer spec with per-layer activation-calibrated shifts. Confirms the -/// Rust integer decode is BIT-IDENTICAL to the numpy integer-forward reference -/// on a trained model (the determinism guarantee, on real weights). Skipped -/// without /tmp artifacts. +/// int_synth against a REAL trained model (not the constructed unit test): a float +/// autoencoder trained on synthetic images, then quantised to the integer spec with +/// per-layer activation-calibrated shifts. Confirms the Rust integer decode is +/// BIT-IDENTICAL to the numpy integer-forward reference on a trained model (the +/// determinism guarantee, on real weights). Skipped without /tmp artifacts. #[test] fn int_synth_trained_model_matches_integer_forward() { let wai = Path::new("/tmp/test.intsynth.wai"); -- 2.54.0 (Apple Git-157) From 603c6bbb8c2349f37ef6b76d7fd94d5596a8b0d4 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Sun, 11 Oct 2026 03:44:06 -0400 Subject: [PATCH 14/15] wai-web reach: the record leg runs the record's own demo scripts The record leg holds the package of record (reach/record/, built from the commit reach/manifest.json names) to the demo verify scripts. It ran this tree's scripts, so a check added after the record's commit was run against code that predates it: #201's verify_pack test of the writers' manifest limit failed on every rung of the record leg (jobs 21114, 21188), because the recorded wai_web_bg.wasm was built before the writers refused a manifest past 2^27. run-modern.mjs takes --demo-commit : step 3 then runs that commit's demo scripts over that commit's corpora (its wai/, exported with git archive). A script this tree runs that the commit lacks is reported as SKIP, not as a pass; a commit with none of them fails the run, as does an empty or unknown commit. The record leg fetches input.commit (the checkout is shallow) and passes it. The tree leg is unchanged: this tree's scripts against this tree's build, where the new verify_pack check runs and passes. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/wai-web-reach.yml | 18 ++++++- wai/wai-web/reach/README.md | 13 +++-- wai/wai-web/reach/test/run-modern.mjs | 71 ++++++++++++++++++++++----- 3 files changed, 85 insertions(+), 17 deletions(-) diff --git a/.github/workflows/wai-web-reach.yml b/.github/workflows/wai-web-reach.yml index 6e7c6fcac..74b0b6b86 100644 --- a/.github/workflows/wai-web-reach.yml +++ b/.github/workflows/wai-web-reach.yml @@ -27,7 +27,10 @@ name: wai-web reach builds # 2. on current Node (reach/test/run-modern.mjs): the loader must choose the shipped # build; every rung and profile, forced, must give the shipped build's outputs over # the corpora; an engine stripped to ES5 must be given J and the same outputs; and -# the demo verify scripts must pass against shipped, L1, L2 and J; +# the demo verify scripts must pass against shipped, L1, L2 and J. On the record leg +# those scripts, and the corpora they read, are the record's commit's +# (--demo-commit): the package is held to the checks of the code it was built from, +# and a check added since meets that code on the tree leg; # 3. on older engine lines, each in its own Node image (reach/test/conformance.js, # which is ECMAScript 5): the loader must choose the rung that line needs, by # feature probes alone, and every workload must equal the shipped build's output @@ -175,6 +178,11 @@ jobs: "$REACH/manifest.json" "$REACH/record" mkdir -p "$RUNNER_TEMP/pkg" cp "$REACH/record/wai_web_bg.wasm" "$REACH/record/wai_web.js" "$RUNNER_TEMP/pkg/" + # The commit the package was built from (input.commit): its demo verify scripts + # are the ones run-modern.mjs runs on this leg. The checkout is shallow. + c=$(node -p 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).input.commit' "$REACH/manifest.json") + git fetch --no-tags --depth=1 origin "$c" + echo "RECORD_COMMIT=$c" >> "$GITHUB_ENV" # The build and the reference stay in the workspace: the engine images below # mount it, and nothing else. - name: Reach builds (reach/build.sh, pinned binaryen and JS tools) @@ -184,9 +192,15 @@ jobs: if [ "${{ matrix.leg }}" = record ]; then export WAI_REACH_REQUIRE_RECORD=1; fi WAI_REACH_TOOLS="$RUNNER_TEMP/reach-tools" bash "$REACH/build.sh" "$RUNNER_TEMP/pkg" "$DIST" cat "$DIST/manifest.json" + # The demo verify scripts: on the record leg, the record's commit's, over that + # commit's corpora; on the tree leg, this tree's. - name: Current Node — every rung forced, the demo verify scripts on every full build shell: bash - run: node "$REACH/test/run-modern.mjs" --dist "$DIST" --shipped "$RUNNER_TEMP/pkg" --out "$REACH/ci-modern-${{ matrix.leg }}" + run: | + set -euo pipefail + set -- + if [ "${{ matrix.leg }}" = record ]; then set -- --demo-commit "$RECORD_COMMIT"; fi + node "$REACH/test/run-modern.mjs" --dist "$DIST" --shipped "$RUNNER_TEMP/pkg" --out "$REACH/ci-modern-${{ matrix.leg }}" "$@" - name: node:0.12 (no WebAssembly) — J chosen and equal to the shipped build uses: docker://node:0.12 diff --git a/wai/wai-web/reach/README.md b/wai/wai-web/reach/README.md index 1258d4883..c67d92b37 100644 --- a/wai/wai-web/reach/README.md +++ b/wai/wai-web/reach/README.md @@ -164,10 +164,13 @@ Every rung must give the shipped build's outputs. The runners below check this: - the demo verify scripts (`demo/verify_*.mjs`: integer payloads, int_synth, the staged corpus with its guards and fuzz, the menu, the writers, the staged measurement) run against shipped, L1, L2 and J through a generated adapter module. The measurement - script's written files must equal the shipped build's. + script's written files must equal the shipped build's. They are this tree's scripts, + or, with `--demo-commit `, that commit's scripts over that commit's corpora + (its `wai/`, exported with `git archive`). A script this tree runs that the commit + lacks is reported as `SKIP`. ``` -node wai/wai-web/reach/test/run-modern.mjs --dist --shipped +node wai/wai-web/reach/test/run-modern.mjs --dist --shipped [--demo-commit ] node wai/wai-web/reach/test/conformance.js --dist --wai wai --rung J --profile receipt --reference ``` @@ -181,9 +184,13 @@ CI (`.github/workflows/wai-web-reach.yml`, on the `linux` runner) runs a leg per kept rather than rebuilt in CI: the same commit and recipe give a different wasm on a Linux x86_64 host than on the macOS arm64 host that built it (the glue is the same), so the commit's build is checked on that host (the site's `build-wasm-pkg.sh recorded wai`). + The demo verify scripts on this leg are that commit's, over that commit's corpora + (`run-modern.mjs --demo-commit`): a script a later commit changes checks code the + package does not hold. The reach workloads and the engine images are this tree's. - **The tree.** wai-web is built from the pull request's own tree, so a change to wai-web meets the reach builds before the package of record moves to it. These outputs are not - the recorded ones. + the recorded ones. The demo verify scripts are this tree's, so a check a pull request + adds runs here against the code it checks. On each leg: diff --git a/wai/wai-web/reach/test/run-modern.mjs b/wai/wai-web/reach/test/run-modern.mjs index 200d1e39d..fb0f01372 100644 --- a/wai/wai-web/reach/test/run-modern.mjs +++ b/wai/wai-web/reach/test/run-modern.mjs @@ -1,7 +1,7 @@ // Every rung forced, on a modern Node: the reach conformance at every profile, and the // existing wai-web verify scripts (demo/verify_*.mjs) on every full build. // -// node run-modern.mjs --dist --shipped [--out ] +// node run-modern.mjs --dist --shipped [--out ] [--demo-commit ] // // 1. The shipped package, chosen by the loader itself (it must choose `shipped` here), // runs the workloads; its outputs are the reference. @@ -16,6 +16,15 @@ // module; the measurement script's written outputs must equal the shipped build's // file for file. // +// The scripts are this tree's by default. With --demo-commit, they and the corpora +// they read are that commit's: its wai/, exported with `git archive` into +// /demo-tree. A package is held to the scripts of the commit it was built +// from, not to checks written after it: a check a later commit adds describes code +// the package does not hold. CI's leg on the package of record passes the commit +// reach/manifest.json names (input.commit). A script this tree runs that the commit +// does not have is reported as SKIP, not as a pass; if the commit has none of them, +// the run fails. +// // Exit 0 only if every run passes. --out keeps the reference, the logs and a JSON // summary there (default: a temporary folder, removed). import { spawnSync } from 'node:child_process'; @@ -28,7 +37,7 @@ const here = dirname(fileURLToPath(import.meta.url)); const web = resolve(here, '../..'); const wai = resolve(web, '..'); const args = Object.fromEntries(process.argv.slice(2).reduce((acc, a, i, all) => (a.startsWith('--') ? [...acc, [a.slice(2), all[i + 1]]] : acc), [])); -if (!args.dist || !args.shipped) { console.log('usage: node run-modern.mjs --dist --shipped [--out ]'); process.exit(2); } +if (!args.dist || !args.shipped) { console.log('usage: node run-modern.mjs --dist --shipped [--out ] [--demo-commit ]'); process.exit(2); } const dist = resolve(args.dist), shipped = resolve(args.shipped); const keep = !!args.out; const out = keep ? resolve(args.out) : mkdtempSync(join(tmpdir(), 'wai-reach-')); @@ -36,6 +45,33 @@ mkdirSync(out, { recursive: true }); const ref = join(out, 'reference.json'); const runs = []; +// Where step 3's scripts and corpora come from: this tree, or --demo-commit's wai/. +let demoWai = wai, demoFrom = 'this tree'; +if ('demo-commit' in args) { + if (!args['demo-commit'] || args['demo-commit'].startsWith('--')) { console.log('FAIL --demo-commit names no commit'); process.exit(2); } + // From the top of the repository: run in a subfolder, `git archive` keeps only that + // subfolder's part of the tree it is given. + const top = spawnSync('git', ['-C', wai, 'rev-parse', '--show-toplevel'], { encoding: 'utf8' }).stdout.trim(); + const git = (...a) => spawnSync('git', ['-C', top || wai, ...a], { encoding: 'utf8', maxBuffer: 1 << 26 }); + const rev = git('rev-parse', '--verify', '--quiet', `${args['demo-commit']}^{commit}`); + if (rev.status !== 0) { + console.log(`FAIL --demo-commit ${args['demo-commit']} is not a commit in this clone (a shallow clone must fetch it first)`); + process.exit(1); + } + const commit = rev.stdout.trim(); + const root = join(out, 'demo-tree'), tar = join(out, 'demo-tree.tar'); + rmSync(root, { recursive: true, force: true }); + mkdirSync(root, { recursive: true }); + const a = git('archive', '--format=tar', '--prefix=wai/', '-o', tar, `${commit}:wai`); + const x = a.status === 0 ? spawnSync('tar', ['-xf', tar, '-C', root], { encoding: 'utf8' }) : a; + rmSync(tar, { force: true }); + if (x.status !== 0) { console.log(`FAIL could not export wai/ at ${commit}: ${(x.stderr || '').trim()}`); process.exit(1); } + demoWai = join(root, 'wai'); + demoFrom = `commit ${commit}`; +} +const demoWeb = join(demoWai, 'wai-web'); +console.log(`the demo verify scripts and their corpora: ${demoFrom === 'this tree' ? demoFrom : `wai/ at ${demoFrom}`}`); + function conformance(label, extra) { const r = spawnSync(process.execPath, [join(here, 'conformance.js'), '--dist', dist, '--wai', wai, '--shipped', shipped, ...extra], { encoding: 'utf8', maxBuffer: 1 << 26 }); const text = (r.stdout || '') + (r.stderr || ''); @@ -86,24 +122,34 @@ export const { ${names.join(', ')} } = st.api; return file; } const SCRIPTS = [ - ['verify_int_payload_wasm', (p) => [p, join(wai, 'wai-rs/tests/vectors')]], - ['verify_int_synth_wasm', (p) => [p, join(wai, 'neural-conformance')]], - ['verify_staged_wasm', (p) => [p, join(wai, 'staged-conformance')]], - ['verify_menu', (p) => [p, join(wai, 'registry/capabilities.json')]], - ['verify_pack', (p) => [p, join(wai, 'registry/capabilities.json')]], - ['verify_measure_wasm', (p, rung) => [p, join(wai, 'staged-measure-conformance'), '--out', join(out, `measure.${rung}`)]], + ['verify_int_payload_wasm', (p) => [p, join(demoWai, 'wai-rs/tests/vectors')]], + ['verify_int_synth_wasm', (p) => [p, join(demoWai, 'neural-conformance')]], + ['verify_staged_wasm', (p) => [p, join(demoWai, 'staged-conformance')]], + ['verify_menu', (p) => [p, join(demoWai, 'registry/capabilities.json')]], + ['verify_pack', (p) => [p, join(demoWai, 'registry/capabilities.json')]], + ['verify_measure_wasm', (p, rung) => [p, join(demoWai, 'staged-measure-conformance'), '--out', join(out, `measure.${rung}`)]], ]; +const absent = SCRIPTS.map(([s]) => s).filter((s) => !existsSync(join(demoWeb, 'demo', `${s}.mjs`))); +if (absent.length === SCRIPTS.length) { + runs.push({ kind: 'demo', label: 'the demo verify scripts', ok: false, failed: [`${demoFrom} has none of them`] }); + console.log(`FAIL ${demoFrom} has none of the demo verify scripts`); +} for (const rung of ['shipped', 'L1', 'L2', 'J']) { const a = adapter(rung); for (const [script, argv] of SCRIPTS) { - const r = spawnSync(process.execPath, [join(web, 'demo', `${script}.mjs`), ...argv(a, rung)], { encoding: 'utf8', cwd: web, maxBuffer: 1 << 26 }); + if (absent.includes(script)) { + runs.push({ kind: 'demo', label: `${rung} ${script}`, ok: true, skipped: `${demoFrom} has no demo/${script}.mjs` }); + console.log(`SKIP ${rung} demo/${script}.mjs: ${demoFrom} has no such script`); + continue; + } + const r = spawnSync(process.execPath, [join(demoWeb, 'demo', `${script}.mjs`), ...argv(a, rung)], { encoding: 'utf8', cwd: demoWeb, maxBuffer: 1 << 26 }); const text = (r.stdout || '') + (r.stderr || ''); writeFileSync(join(out, `${script}.${rung}.log`), text); const ok = r.status === 0; runs.push({ kind: 'demo', label: `${rung} ${script}`, ok, failed: ok ? [] : text.split('\n').filter((l) => /FAIL|Error/.test(l)).slice(0, 8) }); console.log(`${ok ? 'PASS' : 'FAIL'} ${rung} demo/${script}.mjs${ok ? '' : '\n ' + text.trim().split('\n').slice(-6).join('\n ')}`); } - if (rung !== 'shipped') { + if (rung !== 'shipped' && !absent.includes('verify_measure_wasm')) { const A = join(out, 'measure.shipped'), B = join(out, `measure.${rung}`); const files = existsSync(A) ? readdirSync(A).sort() : []; const differ = files.filter((f) => !existsSync(join(B, f)) || !readFileSync(join(A, f)).equals(readFileSync(join(B, f)))); @@ -114,7 +160,8 @@ for (const rung of ['shipped', 'L1', 'L2', 'J']) { } const failed = runs.filter((r) => !r.ok); -writeFileSync(join(out, 'summary.json'), JSON.stringify({ engine: `node ${process.version}`, runs }, null, 1) + '\n'); -console.log(`\n${runs.length - failed.length} of ${runs.length} runs passed on node ${process.version}${keep ? `; logs in ${out}` : ''}`); +const skipped = runs.filter((r) => r.skipped); +writeFileSync(join(out, 'summary.json'), JSON.stringify({ engine: `node ${process.version}`, demo_scripts: demoFrom, runs }, null, 1) + '\n'); +console.log(`\n${runs.length - failed.length - skipped.length} of ${runs.length - skipped.length} runs passed${skipped.length ? ` (${skipped.length} skipped)` : ''} on node ${process.version}; demo scripts from ${demoFrom}${keep ? `; logs in ${out}` : ''}`); if (!keep) rmSync(out, { recursive: true, force: true }); process.exit(failed.length ? 1 : 0); -- 2.54.0 (Apple Git-157) From 1f4876653195363e85294eefdcb8b3efc023538c Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Sun, 11 Oct 2026 08:34:57 -0400 Subject: [PATCH 15/15] wai-conformance: build wai as one codegen unit for the wasip2 quantum leg The wasip2 leg's `cargo build --bin wai_quantum_vectors` also links wai's cdylib, and rust-lld 22 (Rust 1.98) segfaults linking it on #201's tree. The crash is LLD 22's ElemSection::writeBody dereferencing a null indirect function table: the linker keeps the flags of the first object that names __indirect_function_table, and on this tree that object (a codegen unit holding io::Error::is_interrupted, which the new capped manifest writer instantiates) names it without NO_STRIP. The table is then no GC root, nothing the cdylib keeps (only cabi_realloc is exported) calls through it, yet live function addresses still fill the element segment. LLD 23 ORs NO_STRIP across references and links the same inputs. Which object comes first follows the codegen units' hashed names, which depend on the code and the checkout path, so the crash comes and goes with unrelated changes. With one unit, wai's object comes first and takes function addresses, so it names the table with NO_STRIP. Measured on 603c6bbb at the CI path in an x86_64 Ubuntu 22.04 container: the step crashes as is and links with this setting. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/wai-conformance.yml | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/.github/workflows/wai-conformance.yml b/.github/workflows/wai-conformance.yml index 693f2c9a5..ff0975324 100644 --- a/.github/workflows/wai-conformance.yml +++ b/.github/workflows/wai-conformance.yml @@ -559,12 +559,25 @@ jobs: || { echo "::error::no test of $m ran"; exit 1; } done + # The wai package is built as one codegen unit here. Its lib target is also a cdylib, + # and the wasm linker in Rust 1.98 (LLD 22) can crash with SIGSEGV linking that cdylib: + # the linker keeps the flags of the first object that names the function table, so + # when that object names it without NO_STRIP the table is no garbage-collection root. + # Nothing the cdylib keeps calls through the table, but it does keep function + # addresses, and writing their element segment then dereferences a table that was + # never created. Which object comes first follows the hashed names of the codegen + # units, so an unrelated change to the crate, or a different checkout path, can bring + # the crash on or off. As one unit, wai's object comes first and takes function + # addresses, so it names the table with NO_STRIP. LLD 23 keeps NO_STRIP from any + # object. The units change how the crate is split for code generation, not what the + # program computes. - name: wai_quantum_vectors verify under wasmtime shell: bash working-directory: wai/wai-rs run: | set -euo pipefail - cargo build --locked --target wasm32-wasip2 --no-default-features --features quantum --bin wai_quantum_vectors + cargo build --locked --target wasm32-wasip2 --no-default-features --features quantum --bin wai_quantum_vectors \ + --config 'profile.dev.package.wai.codegen-units=1' wasmtime run --dir ../quantum-conformance::/corpus \ target/wasm32-wasip2/debug/wai_quantum_vectors.wasm verify /corpus -- 2.54.0 (Apple Git-157)