From 648f407f09e649a35dbdfe6d85a3a77eac1887a5 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Mon, 28 Sep 2026 18:40:59 -0400 Subject: [PATCH 1/2] wai: the headline decode tests fail on a missing input, and CI supplies their inputs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tests behind WAI's whole-decode claims read inputs from outside the repository and returned early — reporting a pass — when those inputs were absent. No workflow supplied them, so the claims were not checked continuously. - WAI_REQUIRE_SOTA_VECTORS=1 turns a missing input into a failure. A path variable naming a missing or incomplete input fails with or without the switch. Local runs without either still skip, and every skip prints a banner saying the claim was not verified. - wai-sota-vectors.yml regenerates the int_hyper vectors from the pretrained mbt2018_mean checkpoint (pinned by SHA-256, with a pinned Python stack) and runs int_hyper_synth and int_video_sota_keyframe with the switch set. The regenerated vectors identify one run, not one machine: the encoder side is float. - wai-jpegai-e2e.yml fetches the pinned JPEG AI inputs from the public models root, checks each against a SHA-256 manifest, and runs the whole-decode and reconstruction-binding tests with the switch set, requiring each test by name and the pinned pixel hash. - Both jobs fail on any skip banner, and trigger on the kernels and float helpers the decode imports. - The int_hyper claims are stated as the checks CI runs, with its quality measured on the pinned image; JPEG AI cross-machine identity is stated as a measurement, not a registry-class claim. determinism-tiers §5–§6 record the jobs. Neither workflow has yet run through the forge. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/wai-jpegai-e2e.yml | 166 ++++++++++ .github/workflows/wai-sota-vectors.yml | 254 +++++++++++++++ wai/CHANGELOG.md | 99 +++++- wai/extensions/determinism-tiers.md | 31 +- wai/tools/jpegai-e2e-v1.sha256 | 13 + wai/tools/requirements-sota-int.txt | 25 ++ wai/tools/wai_int_hyper_tiny_vector.py | 15 +- .../wai_int_video_sota_keyframe_encode.py | 22 +- wai/tools/wai_jpegai_fetch_e2e_inputs.sh | 54 ++++ wai/tools/wai_sota_int_synth_export.py | 77 ++++- wai/tools/wai_sota_source_image.py | 49 +++ wai/tools/wai_sota_vectors.sh | 71 +++++ wai/wai-rs/examples/int_hyper_decode_frame.rs | 4 +- wai/wai-rs/src/codecs/mod.rs | 2 +- wai/wai-rs/src/int_hyper_synth.rs | 22 +- wai/wai-rs/src/lib.rs | 7 +- wai/wai-rs/tests/int_hyper_synth.rs | 159 +++++++--- wai/wai-rs/tests/int_video_sota_keyframe.rs | 46 +-- wai/wai-rs/tests/jpegai_e2e_full.rs | 23 +- wai/wai-rs/tests/jpegai_live.rs | 49 ++- .../tests/jpegai_reconstruction_binding.rs | 52 ++- wai/wai-rs/tests/require_vectors_rule.rs | 300 ++++++++++++++++++ wai/wai-rs/tests/support/require_vectors.rs | 204 ++++++++++++ wai/wai-web/demo/int_hyper.html | 15 +- wai/wai-web/demo/int_hyper.js | 3 +- wai/wai-web/demo/verify_int_hyper_wasm.mjs | 4 +- wai/wai-web/src/neural_int.rs | 2 +- 27 files changed, 1624 insertions(+), 144 deletions(-) create mode 100644 .github/workflows/wai-jpegai-e2e.yml create mode 100644 .github/workflows/wai-sota-vectors.yml create mode 100644 wai/tools/jpegai-e2e-v1.sha256 create mode 100644 wai/tools/requirements-sota-int.txt create mode 100755 wai/tools/wai_jpegai_fetch_e2e_inputs.sh create mode 100644 wai/tools/wai_sota_source_image.py create mode 100755 wai/tools/wai_sota_vectors.sh create mode 100644 wai/wai-rs/tests/require_vectors_rule.rs create mode 100644 wai/wai-rs/tests/support/require_vectors.rs diff --git a/.github/workflows/wai-jpegai-e2e.yml b/.github/workflows/wai-jpegai-e2e.yml new file mode 100644 index 000000000..9f6c7d72d --- /dev/null +++ b/.github/workflows/wai-jpegai-e2e.yml @@ -0,0 +1,166 @@ +name: WAI JPEG-AI whole decode (pinned inputs, required) + +# WAI states that the exact decoder, `wai::jpegai_decode::decode`, decodes an +# ISO/IEC 6048 JPEG-AI codestream to RGB deterministically, so a hard binding +# taken over one decode verifies a second one, and that on one reference +# codestream its output stays within max|Δ| ≤ 2 of the reference decoder's +# (recorded max|Δ| = 1; wai/SPEC.md §5, the wai.image.jpegai row). +# Across machines the registry states less: it classes wai.image.jpegai as +# NeuralEntropyExact, a conservative class that guarantees only the entropy decode +# (the decoded symbols and their entropy parameters) to be byte-identical on every +# machine. That the exact decoder's pinned output is the same on other machines is +# a measurement, recorded in wai/extensions/determinism-tiers.md §6. The tests that +# carry these statements (tests/jpegai_e2e_full.rs, +# tests/jpegai_reconstruction_binding.rs) read their inputs from outside the tree +# and used to report PASS without them. +# +# This job fetches those inputs from WAI's versioned models root (jpegai/v1/), +# checks every file against the SHA-256 manifest committed in +# wai/tools/jpegai-e2e-v1.sha256, and runs both tests with +# WAI_REQUIRE_SOTA_VECTORS=1, so a missing input is a failure. What passes here: +# - the whole chain, codestream → RGB, through the one public `decode()` call, +# lands within max|Δ| ≤ 2 of the reference decoder's output, and at most 5% +# of samples differ from it at all (the test's bar); +# - two decodes in one process are byte-identical and the reconstruction binding +# verifies the second and rejects a one-byte change; +# - the decoded pixels hash to WAI_JPEGAI_EXPECT_SHA256 below: the value measured +# from these inputs on an Apple Silicon Mac (aarch64 macOS, native) and +# reproduced by an x86_64 Linux build (glibc, run under QEMU user-mode +# emulation), and the value the four-platform comparison in determinism-tiers +# §6 (macOS arm64, wasm32-wasip2, Linux arm64, Linux x86_64 under user-mode +# emulation) found on every platform. This runner is x86_64 hardware, so the +# job adds a decode on one more machine to that measurement. A match is a +# measurement on this runner, not something the registry class promises; a +# mismatch is a departure from the measured agreement. +# The pinned decode is max|Δ| = 1 from the reference on 4.446% of samples +# (on the Mac and the emulated x86_64 build), so a hash match also confirms +# that recorded max|Δ| = 1 for this codestream; the test's own tolerance +# above is looser than that. +# +# What this job does not cover: +# - Other codestreams: it decodes one 256×256 test codestream, so the +# max|Δ| ≤ 2 bound and the determinism statement are checked on that input +# only. +# - The extraction: the inputs are an extraction of the JPEG-AI +# reference-software assets (tools/wai_jpegai_build_e2e_inputs.py), which are +# not in the repository, so the extraction is not re-run here; the manifest +# pins its published bytes. +# - The ONNX-runtime decode (`neural::jpegai`, tests/jpegai_live.rs), which needs +# those local assets and is not run in CI. It also needs the `neural` feature, +# whose prebuilt ONNX Runtime does not link on the `linux` runner's image +# (glibc 2.35; the library references glibc 2.38 symbols), which is where +# wai-neural-e2e.yml's runs fail once they reach the Rust build. The +# features built here (neural_int, provenance) link no prebuilt native +# library. +# The job depends on the models root being reachable; an outage fails the job. +# +# The path filters list the source files the two tests' decode path imports, +# among them src/float_repro.rs (the exact decoder's `exp` and its roundings to +# an integer) and src/int_kernels.rs (the fast convolution and parallel-map +# kernels the integer networks run through). An entry for a file the tree does +# not have matches nothing until the file exists. A module the decode path +# starts importing belongs here too; otherwise a change to that module alone is +# first checked by the weekly run. + +on: + push: + branches: [main] + paths: + - 'wai/wai-rs/src/jpegai_*.rs' + - 'wai/wai-rs/src/reconstruction_binding.rs' + - 'wai/wai-rs/src/int_transformer.rs' + - 'wai/wai-rs/src/int_hyper_synth.rs' + - 'wai/wai-rs/src/int_kernels.rs' + - 'wai/wai-rs/src/float_repro.rs' + - 'wai/wai-rs/src/int_entropy.rs' + - 'wai/wai-rs/src/int_transform.rs' + - 'wai/wai-rs/src/rans.rs' + - 'wai/wai-rs/src/lib.rs' + - 'wai/wai-rs/Cargo.toml' + - 'wai/wai-rs/Cargo.lock' + - 'wai/wai-rs/tests/jpegai_e2e_full.rs' + - 'wai/wai-rs/tests/jpegai_reconstruction_binding.rs' + - 'wai/wai-rs/tests/support/**' + - 'wai/tools/wai_jpegai_fetch_e2e_inputs.sh' + - 'wai/tools/jpegai-e2e-v1.sha256' + - '.github/workflows/wai-jpegai-e2e.yml' + pull_request: + paths: + - 'wai/wai-rs/src/jpegai_*.rs' + - 'wai/wai-rs/src/reconstruction_binding.rs' + - 'wai/wai-rs/src/int_transformer.rs' + - 'wai/wai-rs/src/int_hyper_synth.rs' + - 'wai/wai-rs/src/int_kernels.rs' + - 'wai/wai-rs/src/float_repro.rs' + - 'wai/wai-rs/src/int_entropy.rs' + - 'wai/wai-rs/src/int_transform.rs' + - 'wai/wai-rs/src/rans.rs' + - 'wai/wai-rs/src/lib.rs' + - 'wai/wai-rs/Cargo.toml' + - 'wai/wai-rs/Cargo.lock' + - 'wai/wai-rs/tests/jpegai_e2e_full.rs' + - 'wai/wai-rs/tests/jpegai_reconstruction_binding.rs' + - 'wai/wai-rs/tests/support/**' + - 'wai/tools/wai_jpegai_fetch_e2e_inputs.sh' + - 'wai/tools/jpegai-e2e-v1.sha256' + - '.github/workflows/wai-jpegai-e2e.yml' + # Weekly, independent of changes: the inputs live outside the repository, and + # their disappearing should surface here, not at the next unrelated PR. + schedule: + - cron: '41 6 * * 1' + +jobs: + jpegai-e2e: + name: fetch pinned JPEG-AI inputs → whole decode + pixel hash, required + runs-on: linux + timeout-minutes: 30 + env: + WAI_REQUIRE_SOTA_VECTORS: '1' + WAI_JPEGAI_DIR: ${{ github.workspace }}/jpegai_e2e + # sha256 of the decoded RGB8 pixels for the jpegai/v1 inputs (see header). + WAI_JPEGAI_EXPECT_SHA256: dbc494a5a670914edf55c07c48afc0dd348cd05d3a82fff293bf384b2bb69ea3 + steps: + - uses: actions/checkout@v5 + # Bootstrap rustup: the docker runner image ships no Rust. + - name: Install Rust 1.98.0 + shell: bash + run: | + export PATH="$HOME/.cargo/bin:$PATH" + command -v rustup >/dev/null 2>&1 || \ + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0 + rustup toolchain install 1.98.0 --profile minimal + rustup default 1.98.0 + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + - name: Fetch the JPEG-AI inputs and check each against the pinned manifest + shell: bash + working-directory: wai + run: tools/wai_jpegai_fetch_e2e_inputs.sh "$WAI_JPEGAI_DIR" + - name: Whole decode vs the reference, determinism, pinned pixel hash (missing input = failure) + shell: bash + working-directory: wai/wai-rs + run: | + set -o pipefail + log="${RUNNER_TEMP:-/tmp}/jpegai-tests.log" + # Without the switch both tests would skip and report ok. + [ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \ + || { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; } + export WAI_JPEGAI_CODESTREAM="$WAI_JPEGAI_DIR/test256.bin" + export WAI_JPEGAI_WEIGHTS_DIR="$WAI_JPEGAI_DIR" + export WAI_JPEGAI_NORM="$WAI_JPEGAI_DIR/norm.json" + # --show-output rather than --nocapture: each test's output is printed + # after the results, so it cannot split a `test X ... ok` line below. + cargo test --release --no-default-features --features neural_int,provenance \ + --test jpegai_e2e_full --test jpegai_reconstruction_binding -- --show-output 2>&1 | tee "$log" + # A test that was renamed or compiled out would not fail — it would just + # not run. Require each one by name, and the pinned-hash check by its line. + for t in codestream_to_rgb_matches_reference_png real_decode_to_reconstruction_binding; do + grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; } + done + grep -q "matches the pinned WAI_JPEGAI_EXPECT_SHA256" "$log" \ + || { echo "::error::the pinned pixel hash was not checked"; exit 1; } + # A skip also prints `test X ... ok`; every skip writes a banner with + # "SKIPPED" and "NOT VERIFIED" (tests/support/require_vectors.rs). + if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then + echo "::error::a JPEG-AI test skipped instead of checking its input" + exit 1 + fi diff --git a/.github/workflows/wai-sota-vectors.yml b/.github/workflows/wai-sota-vectors.yml new file mode 100644 index 000000000..cf93aa572 --- /dev/null +++ b/.github/workflows/wai-sota-vectors.yml @@ -0,0 +1,254 @@ +name: WAI mbt2018_mean integer decode (regenerated vectors) + +# `wai.neural.int_hyper` is an integer export of one pretrained model: the +# mbt2018_mean mean-scale hyperprior from the CompressAI model zoo (quality 6, +# MSE), post-training-quantized to integers. WAI states that its whole +# bitstream→image decode is integer, so the Rust decoder reproduces the export +# tool's integer reference byte-for-byte, and that `wai.video.int_hyper` builds a +# clip on that keyframe. The model is too large to commit, so the tests that carry +# these statements read vectors from outside the tree; without them they used to +# return early and report PASS. +# +# This job checks them on every run: it regenerates the vectors from pinned +# inputs (a public-domain photograph shipped in the pinned scikit-image wheel and +# pinned by the SHA-256 of its pixels; the published mbt2018_mean checkpoint, +# pinned by its full SHA-256 and loaded as tensors only; a pinned Python stack), +# then runs the int_hyper vector tests with WAI_REQUIRE_SOTA_VECTORS=1, under which +# a missing vector is a failure. It also runs tests/require_vectors_rule.rs, which +# checks in child processes what the switch does not cover here: that without it a +# variable naming absent or incomplete inputs still fails, and a test skips only +# when no variable names what is missing (an unset variable, or an absent or +# incomplete default location). What passes here: +# - the Rust integer decode reproduces the export tool's integer reference +# byte-for-byte (synthesis, hyper-decoder, the whole conditional bitstream, the +# compact binary model, the eight-frame clip); +# - on the pinned photograph, the integer decode costs at most 1 dB against the +# same model's float decode (the bar CI enforces), measured by the Rust decoder +# on its own output. Measured on an Apple Silicon Mac: 35.985 dB for the integer +# decode (8-bit PSNR against the source pixels) against 36.175 dB for the float +# decode (the export tool's PSNR of the float reconstruction, before 8-bit +# rounding), a cost of 0.19 dB, from 6328 bytes of rANS payload for the +# 256×256 crop (about 0.77 bits per pixel). +# +# What this job does not cover: +# - How mbt2018_mean compares with any other codec. Nothing here measures rate or +# distortion against another model or standard; the job checks only that the +# integer decode is exact and within 1 dB of the same model's float decode. +# - Quality on other images. PSNR is image-dependent and the job measures one +# crop. The 42.6 dB development figure was measured on a photograph that is not +# in the repository. +# - Cross-architecture byte-equality of these vectors. They are regenerated on +# this runner, so the job checks Rust against the Python reference on one +# machine (x86_64 Linux). The cross-architecture check is the committed tiny +# int_hyper vector in wai-byte-equality.yml. +# - Portable or repeatable vector hashes. No expected hash of the regenerated +# vectors is pinned in the repository (the committed tiny vector's is, in +# tests/vectors/int_hyper/tiny.json). Each is written by the same run's export +# and checked in that run, and a hash taken from one run's vectors must not be +# pinned for another run, on this machine or another: it can differ with both +# decoders correct. What was measured, comparing all sixteen files by SHA-256: +# - Between an Apple Silicon Mac and this runner's x86_64 image, four files +# differed. int_synth_sota_input.json (the synthesis-only test's input +# latent, which the float encoder and hyper-decoder compute) differed, and +# with it the rgb_hash in int_synth_sota.json (fd9ca73a… on the Mac, +# f0fb6813… here); each side wrote the same latent in all of its runs. +# cond.json and int_synth_sota.json also differed. Beyond that rgb_hash, +# their difference is not established as a machine difference: both files +# also differ from run to run on this runner, most likely in psnr_float +# (next point). source.png differed in its compressed bytes only: step 1 +# writes it with the PNG encoder from the pinned pixels before any model +# runs, and its pixels (source.rgb) were identical, so that difference is +# the encoder's output, not float computation. +# - Two runs of the same export on this runner (same image, pins and tree, +# minutes apart) wrote different cond.json and int_synth_sota.json; the +# other fourteen files and both rgb_hash values matched. The field those two +# files share that comes from the float model is psnr_float, the float +# decode's PSNR, so it most likely varies from run to run here, although the +# export sets torch to deterministic algorithms on one thread. The runner's +# files were not kept, so the field is inferred, not diffed. Repeated runs on +# the Mac wrote identical files. +# Nothing the tests compare exactly reads psnr_float; only the quality test +# does, against its thresholds (the 1 dB bar and a 30 dB floor). The files +# that matched (both models, the conditional bitstream, frame0, the clip) are +# derived from float computation too, so matching here does not make them +# portable. The pins that hold across runs and machines are on the inputs: the +# photograph's pixels and the checkpoint's bytes. The Python stack is pinned +# by version, but each platform installs its own build of it. +# The JPEG-AI whole-decoder tests run in wai-jpegai-e2e.yml (see +# wai/wai-rs/tests/support/require_vectors.rs). +# +# The path filters list the source files the int_hyper decode imports, among +# them src/int_kernels.rs (the fast convolution kernels int_hyper_synth.rs runs +# through). An entry for a file the tree does not have matches nothing until the +# file exists. A module the decode path starts importing belongs here too; +# otherwise a change to that module alone is first checked by the weekly run. +# +# Licences of what is fetched: CompressAI and its model zoo, BSD-3-Clause-Clear; +# the astronaut photograph, public domain (NASA), per the scikit-image docstring. + +on: + push: + branches: [main] + paths: + - 'wai/wai-rs/src/int_hyper_synth.rs' + - 'wai/wai-rs/src/int_kernels.rs' + - 'wai/wai-rs/src/int_entropy.rs' + - 'wai/wai-rs/src/int_transform.rs' + - 'wai/wai-rs/src/int_video.rs' + - 'wai/wai-rs/src/rans.rs' + - 'wai/wai-rs/src/lib.rs' + - 'wai/wai-rs/Cargo.toml' + - 'wai/wai-rs/Cargo.lock' + - 'wai/wai-rs/examples/int_hyper_decode_frame.rs' + - 'wai/wai-rs/tests/int_hyper_synth.rs' + - 'wai/wai-rs/tests/int_video_sota_keyframe.rs' + - 'wai/wai-rs/tests/support/**' + - 'wai/wai-rs/tests/require_vectors_rule.rs' + - 'wai/tools/wai_sota_*' + - 'wai/tools/wai_int_video_sota_keyframe_encode.py' + - 'wai/tools/requirements-sota-int.txt' + - '.github/workflows/wai-sota-vectors.yml' + pull_request: + paths: + - 'wai/wai-rs/src/int_hyper_synth.rs' + - 'wai/wai-rs/src/int_kernels.rs' + - 'wai/wai-rs/src/int_entropy.rs' + - 'wai/wai-rs/src/int_transform.rs' + - 'wai/wai-rs/src/int_video.rs' + - 'wai/wai-rs/src/rans.rs' + - 'wai/wai-rs/src/lib.rs' + - 'wai/wai-rs/Cargo.toml' + - 'wai/wai-rs/Cargo.lock' + - 'wai/wai-rs/examples/int_hyper_decode_frame.rs' + - 'wai/wai-rs/tests/int_hyper_synth.rs' + - 'wai/wai-rs/tests/int_video_sota_keyframe.rs' + - 'wai/wai-rs/tests/support/**' + - 'wai/wai-rs/tests/require_vectors_rule.rs' + - 'wai/tools/wai_sota_*' + - 'wai/tools/wai_int_video_sota_keyframe_encode.py' + - 'wai/tools/requirements-sota-int.txt' + - '.github/workflows/wai-sota-vectors.yml' + # Weekly, independent of changes: the job depends on artifacts fetched from + # outside the repository (the checkpoint, the pinned wheels), and their + # disappearing should surface as a failure here, not at the next unrelated PR. + schedule: + - cron: '17 6 * * 1' + +jobs: + sota-vectors: + name: regenerate mbt2018_mean integer vectors → Rust decode, required + # The `linux` runner is a docker executor on catthehacker/ubuntu:act-22.04 + # (glibc 2.35, no Rust). The Python half of this setup — setup-python 3.11, + # CPU torch, compressai — completes there: every logged wai-neural-e2e.yml + # run got through it and generated its fixtures. Each of those runs that + # reached the Rust build then failed linking `ort`'s prebuilt ONNX Runtime, + # which references glibc 2.38 and newer-libstdc++ symbols (`__isoc23_strtol`, + # `_M_replace_cold`). The features built here (neural_int, video_int) are + # pure Rust and link no prebuilt native library; do not add `neural` here. + runs-on: linux + timeout-minutes: 45 + env: + WAI_SOTA_INT_DIR: ${{ github.workspace }}/sota_int + WAI_REQUIRE_SOTA_VECTORS: '1' + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + - name: Pinned ML stack (CPU torch first, then the rest) + shell: bash + working-directory: wai + run: | + python -m pip install --upgrade pip + # CPU wheels: the default index resolves the multi-GB CUDA builds. + python -m pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/cpu + python -m pip install -r tools/requirements-sota-int.txt + # Fail here, not mid-export, if the resolver installed anything but the + # pins, or if torchvision's compiled ops do not match torch (the + # `torchvision::nms does not exist` mismatch wai-neural-e2e.yml once hit). + python - tools/requirements-sota-int.txt <<'PY' + import sys, importlib.metadata as md + bad = [] + for line in open(sys.argv[1]): + line = line.split("#")[0].strip() + if not line: + continue + name, want = line.split("==") + got = md.version(name) + print(f"{name}=={got}") + if got.split("+")[0] != want: + bad.append(f"{name} {got}, pinned {want}") + if bad: + sys.exit("not the pinned stack: " + "; ".join(bad)) + import torch, torchvision, compressai, skimage + torchvision.ops.nms(torch.zeros(1, 4), torch.zeros(1), 0.5) + PY + # Bootstrap rustup: the docker runner image ships no Rust. + - name: Install Rust 1.98.0 + shell: bash + run: | + export PATH="$HOME/.cargo/bin:$PATH" + command -v rustup >/dev/null 2>&1 || \ + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0 + rustup toolchain install 1.98.0 --profile minimal + rustup default 1.98.0 + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + - name: Regenerate the int_hyper vectors (source photo → integer export → keyframe → clip) + shell: bash + working-directory: wai + run: | + tools/wai_sota_vectors.sh "$WAI_SOTA_INT_DIR" + # Recorded so a result can be tied to the exact vectors it checked. It + # identifies this run's vectors, not this machine's: two runs on this + # runner wrote different cond.json and int_synth_sota.json, and other + # files differ between machines (see the header). It is not a list any + # other run must reproduce, here or elsewhere. + (cd "$WAI_SOTA_INT_DIR" && sha256sum *) + - name: Rust decodes every int_hyper vector byte-for-byte (missing input = failure), and the missing-input rule holds + shell: bash + working-directory: wai/wai-rs + run: | + set -o pipefail + log="${RUNNER_TEMP:-/tmp}/sota-tests.log" + # Without the switch every vector test would skip and report ok. + [ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \ + || { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; } + # --show-output rather than --nocapture: each test's output is printed + # after the results, so it cannot split a `test X ... ok` line below. + # require_vectors_rule runs its cases in child processes with the switch + # and the input variables set per case, so the job's own env does not + # reach them; it prints nothing on success, so it cannot trip the skip + # backstop below. + cargo test --release --no-default-features --features neural_int,video_int \ + --test int_hyper_synth --test int_video_sota_keyframe --test require_vectors_rule \ + -- --show-output 2>&1 | tee "$log" + # A test that was renamed or compiled out would not fail — it would just + # not run. Require each one by name. + for t in rust_integer_synthesis_reproduces_the_export_hash \ + rust_integer_hyper_decoder_reproduces_the_export_hash \ + rust_decodes_the_full_conditional_bitstream_to_the_export_image \ + integer_decode_keeps_the_float_codec_quality \ + decodes_the_committed_tiny_conformance_vector \ + sota_int_hyper_keyframe_clip_decodes_byte_for_byte \ + without_the_switch_a_named_directory_that_does_not_exist_fails \ + without_the_switch_a_named_path_that_is_a_file_fails_as_a_directory \ + without_the_switch_a_named_empty_directory_fails \ + without_the_switch_an_unset_or_empty_directory_variable_skips_on_an_absent_default \ + without_the_switch_an_incomplete_default_directory_skips \ + a_named_directory_holding_the_input_loads_it \ + without_the_switch_a_path_variable_naming_nothing_fails \ + without_the_switch_a_path_variable_naming_nothing_fails_when_another_is_unset \ + without_the_switch_an_unset_path_variable_skips \ + path_variables_naming_files_that_exist_load_them \ + switch_values_that_leave_it_off_behave_as_unset \ + with_the_switch_an_absent_default_fails; do + grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; } + done + # A skipped test ALSO prints `test X ... ok`, so the name check alone + # cannot tell a skip from a pass. Every skip writes a banner containing + # "SKIPPED" and "NOT VERIFIED" (tests/support/require_vectors.rs); either + # word anywhere in the log fails the job, whatever the switch says. + if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then + echo "::error::a vector test skipped instead of checking its input" + exit 1 + fi diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 02c583e05..286cc2672 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -6,6 +6,92 @@ wire format may change and such changes are recorded here. ## v0.4.0 — current +### Headline decode tests fail on a missing input instead of passing + +- The `int_hyper` tests over the pretrained mbt2018_mean (`int_hyper_synth`, + `int_video_sota_keyframe`) and the JPEG-AI whole-decoder tests + (`jpegai_e2e_full`, `jpegai_reconstruction_binding`, `jpegai_live`) read inputs + from outside the tree, and used to `return` — reported as a pass — when those + were absent. With + `WAI_REQUIRE_SOTA_VECTORS=1` a missing input is now a failure that names the + tool producing it; without it the skip is written to the process's real stderr, + so it shows in an ordinary `cargo test` run. Either way, a corrupt input file + fails, and so does a variable that names inputs which are not all there: a + path that does not exist (`WAI_JPEGAI_CODESTREAM`/`_WEIGHTS_DIR`/`_NORM`, even + when another of the three is unset), or a directory that does not exist or + lacks a file the test reads (`WAI_SOTA_INT_DIR`, `WAI_JPEGAI_ASSETS`, the + JPEG-AI weights directory). So without the switch a test skips only when no + variable names what is missing: one of the three path variables is unset, or + `WAI_SOTA_INT_DIR` or `WAI_JPEGAI_ASSETS` is unset or empty and its default + directory (`/tmp/sota_int`, `../jpegai-vm-assets`) is absent or incomplete. + The CI jobs always set the switch, so the new test + `require_vectors_rule` checks this rule without it: it runs the helpers in + child processes with the switch and the variables set per case, and + `wai-sota-vectors.yml` runs it. +- New CI job `wai-sota-vectors.yml` regenerates the mbt2018_mean integer vectors + from pinned inputs (`tools/wai_sota_vectors.sh`, the Python stack in + `tools/requirements-sota-int.txt`, a public-domain photograph pinned by SHA-256, + the published checkpoint pinned by its full SHA-256) and runs those tests + with the switch set. It is triggered by changes to the decode path and by a + weekly schedule. Besides requiring each test by name, it fails if the test log + contains a skip banner, since a skipped test also reports `ok`. The export tool + checks the checkpoint's bytes before loading them (as tensors only), because the + model zoo's own download path checks no hash. The job checks the Rust decode + against the Python integer reference on one runner; cross-architecture + byte-equality remains the committed tiny vector's job in + `wai-byte-equality.yml`. The regenerated vectors are not portable, and part + of them is not repeatable run to run on every machine. Between an Apple + Silicon Mac and the x86_64 runner, the synthesis-only test's input latent, + which the float model computes, differed, and with it its expected + `rgb_hash`; each side wrote the same latent in all of its runs. Two runs on + the one runner wrote different `cond.json` and `int_synth_sota.json`, most + likely in the float decode's PSNR, which only the quality test reads, against + thresholds. The PNG bytes of the source also differed between the Mac and the + runner; the PNG encoder writes them from the pinned pixels before any model + runs, and the pixels matched. So no expected hash of the regenerated vectors + is pinned in the repository (the committed tiny vector's is, in + `tests/vectors/int_hyper/tiny.json`); each run checks the hashes its own export + wrote, and a hash from one run's vectors must not be pinned for another run, + on the same machine or another. The pins that hold across machines are on the + inputs: the photograph's pixels and the checkpoint's bytes. +- New test `integer_decode_keeps_the_float_codec_quality`: the Rust decoder + measures its own reconstruction against the source pixels and must stay within + 1 dB of the same model's float decode on the same image. On the pinned + photograph, measured on an Apple Silicon Mac: integer decode 35.985 dB (8-bit + PSNR of the Rust decode), float decode 36.175 dB (the export tool's PSNR of the + float reconstruction), a cost of 0.19 dB, from 6328 bytes of rANS payload for + 256×256 pixels. This is a relative bar on one image: nothing here compares + mbt2018_mean with another codec, and quality is image-dependent — the 42.6 dB + figure quoted for this codec was measured on a different photograph, which is + not in the repository. +- New CI job `wai-jpegai-e2e.yml` runs `jpegai_e2e_full` and + `jpegai_reconstruction_binding` with the switch set, with the same by-name and + skip-banner checks. Their inputs — the decoder + weights, normative tables and test codestream, extracted from the JPEG-AI + reference-software assets (which are not in this repository) — are fetched from + WAI's versioned models root (`jpegai/v1/`) by + `tools/wai_jpegai_fetch_e2e_inputs.sh` and checked against the SHA-256 manifest + `tools/jpegai-e2e-v1.sha256`. The job also pins the SHA-256 of the decoded + pixels (`WAI_JPEGAI_EXPECT_SHA256`): measured natively on an Apple Silicon Mac + and reproduced by an x86_64 Linux build under QEMU user-mode emulation, and + required again on the x86_64 runner. That extends a measurement, not a + registry guarantee: the registry classes `wai.image.jpegai` as + `NeuralEntropyExact`, a conservative class that guarantees only the entropy + decode to be byte-identical on every machine, while the exact decoder's pinned + output for these inputs was measured identical on four platforms + (determinism-tiers §6). +- `jpegai_live` (the ONNX-runtime `wai.image.jpegai` decode) follows the same + missing-input rule, but no CI job runs it: its model bundle exists only in the + local JPEG-AI asset directory, and it needs the `neural` and `codecs` features. +- determinism-tiers §5–§6 record the two jobs. The full-scale `int_hyper` row and + the JPEG AI whole-decode row (split from the index-derivation row, whose test CI + still does not run) now name their workflow, state that a missing input fails + there instead of skipping, and state that neither workflow has run through the + forge: each passed under `forgejo-runner exec` on the Linux runner's host in an + earlier form, and its current steps have run on macOS arm64 only. §5 quotes the + `int_hyper` cost measured on the pinned photograph in place of the 42.6 dB + figure, and names the bar each job re-checks. + ### Prior carriage and derived-track operations — registered with their limits - New [`extensions/prior-carriage.md`]: a `Prior` MAY declare the wire form of @@ -557,9 +643,12 @@ wire format may change and such changes are recorded here. external standard's codestream deterministically. - Modules: `jpegai_decode` (the composed entry point) over `jpegai_bitstream`, `jpegai_mans`, `jpegai_pipeline`, `jpegai_synthesis`, `jpegai_icci` - (feature `neural_int`). Inputs are regenerable from the shipped assets via - `tools/wai_jpegai_build_e2e_inputs.py`. The whole chain — codestream → RGB — - is exercised end to end against the reference PNG. + (feature `neural_int`). Inputs are regenerable via + `tools/wai_jpegai_build_e2e_inputs.py` from the JPEG-AI reference-software + assets, which are kept locally and are not in this repository; the extraction is + published under WAI's versioned models root (`jpegai/v1/`). The whole chain — + codestream → RGB — is exercised end to end against the reference PNG by + `wai-jpegai-e2e.yml`, which fetches those inputs pinned by SHA-256. ### JPEG AI — a live, self-sufficient `wai.image.jpegai` decode (2026-06-22) @@ -574,7 +663,9 @@ wire format may change and such changes are recorded here. exact, 0 violations / 65536 positions); the dequant is `resi × rvs_dq[grfs[ch]][lk] / 2^16 / scaler_vec[ch]`, with `grfs` determined the same way the σ rvs does so the per-channel `scaler_vec` is consistent. - Matches the reference decoder's PNG to within rounding (max diff 1). + Matches the reference decoder's PNG to within rounding (max diff 1), checked by + `tests/jpegai_live.rs` on a machine holding the local JPEG-AI assets; no CI job + runs it (its model bundle is not in the repository). ### Film reel — `wai.film.reel`, cuts over one world (2026-06-17) diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index 137fb710a..140a05108 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -433,14 +433,19 @@ same ones for its integer networks: - **Fidelity to the float model.** The quantised networks do not reproduce the float model's output, so the pinned reconstruction differs from it. For - `wai.neural.int_hyper`, `wai-rs/src/int_hyper_synth.rs` records 42.6 dB PSNR for - one pretrained model (`mbt2018_mean`), within ~0.6 dB of the float model; the - test image, quality level and machine are not recorded there. For the JPEG AI - exact decoder, `wai-rs/tests/jpegai_e2e_full.rs` compares against the reference - decoder's output on one 256×256 reference codestream and asserts max|Δ| ≤ 2 with - at most 5% of samples differing; the recorded result is max|Δ| = 1 on 4.446% of - samples (macOS arm64, release build, 2026-09-27). Neither figure is re-derived in - CI (§6). + `wai.neural.int_hyper`, over one pretrained model (`mbt2018_mean`, quality 6) + and one pinned test photograph, the integer decode of the full conditional + bitstream measured 35.985 dB PSNR against 36.175 dB for the same model's float + decode, a cost of 0.19 dB (macOS arm64; `wai-rs/src/int_hyper_synth.rs`). + Quality is image-dependent, and this compares the model only with its own float + decode. For the JPEG AI exact decoder, `wai-rs/tests/jpegai_e2e_full.rs` compares + against the reference decoder's output on one 256×256 reference codestream and + asserts max|Δ| ≤ 2 with at most 5% of samples differing; the recorded result is + max|Δ| = 1 on 4.446% of samples (macOS arm64, release build, 2026-09-27). CI + jobs re-check both bars: `wai-sota-vectors.yml` requires the `int_hyper` cost to + stay within 1 dB on that photograph, and `wai-jpegai-e2e.yml` runs the JPEG AI + comparison and requires the decoded pixels to hash to a pinned value. Neither + workflow has run through the forge yet (§6). - **Compute.** The reference decoders are single-threaded CPU code with no explicit SIMD, in integer arithmetic (i64, with i128 intermediates in places; plus fixed-order float for the JPEG AI exact decoder's float stages). They @@ -478,13 +483,14 @@ day. |---|---|---| | Cross-architecture identity of the rANS coder, `int_synth`, and the JPEG AI symbol decode (a reference-encoder codestream, given its distribution indices) | golden-byte assertions passed on Linux x86_64, Linux arm64 and macOS arm64 on main at `afc7ece` (2026-09-27, `wai-conformance.yml`; `wai-byte-equality.yml` last ran on main at `3f50097`, 2026-08-29, all three legs passing), and on Windows x86_64 last on main at `3e7811f` (2026-08-27). Between `3e7811f` and `afc7ece` the suite only gained the video and keyframe goldens; these goldens, the sources they decode and the suite's lockfile are unchanged | `byte-exact-conformance/`; `.github/workflows/wai-conformance.yml`, `wai-byte-equality.yml` | | Cross-architecture identity of integer inter-frame video, and of `int_hyper` on a small learned keyframe | the same suite, passing on Linux x86_64, Linux arm64 and macOS arm64 on main at `afc7ece` (2026-09-27). Windows x86_64 never decoded the keyframe clip: it was added in `d4c922b` (2026-08-28), whose Windows legs were cancelled. Windows decoded the integer video goldens only on the pre-merge commits `c9be2ad`, `256e5dd`, `7dfca26` and `3d34860` (2026-08-25/26), whose video sources, vectors and golden file match main but whose `byte-exact-conformance/Cargo.lock` differs; the change reached main at `3f50097`, after the last Windows run on main | as above | -| `wai.image.jpegai` exact decoder: the integer derivation of the distribution indices matches the reference decoder exactly; the whole decode stays within a stated bound of it (max\|Δ\| ≤ 2, ≤ 5% of samples) | tests on one machine that skip unless their fixtures (sink-supplied weights, normative tables, codestream) are provided; CI does not provide them. The whole-decode test last ran on macOS arm64 on 2026-09-27 with a local bundle | `wai-rs/tests/jpegai_hyperscale_calibration.rs`, `jpegai_e2e_full.rs` | +| `wai.image.jpegai` exact decoder: the integer derivation of the distribution indices matches the reference decoder exactly | a test on one machine that skips unless its fixtures (sink-supplied weights, normative tables, codestream) are provided; CI does not provide them | `wai-rs/tests/jpegai_hyperscale_calibration.rs` | +| `wai.image.jpegai` exact decoder, whole decode through the one public `decode()` call: it stays within a stated bound of the reference decoder's output (max\|Δ\| ≤ 2, ≤ 5% of samples); two decodes in one process are byte-identical, and a reconstruction binding taken over one verifies the other | `wai-jpegai-e2e.yml` fetches the pinned `jpegai/v1` inputs (an extraction of the reference-software assets, published under WAI's models root), checks each file against the SHA-256 manifest `tools/jpegai-e2e-v1.sha256`, and runs both tests with `WAI_REQUIRE_SOTA_VECTORS=1`, under which a missing input fails instead of skipping. The job also fails on any skip banner in the test log, and requires the decoded pixels to hash to a pinned SHA-256. One 256×256 codestream. The workflow has not run through the forge. Its job passed under `forgejo-runner exec` on the Linux runner's host (x86_64 hardware, 2026-09-27), with the exact decoder as it stood on main at `afc7ece`, which calls the platform `exp`; there the decoded pixels matched the pinned hash, the value measured on macOS arm64. The job's steps have otherwise run on macOS arm64 only, outside a runner | `wai-rs/tests/jpegai_e2e_full.rs`, `jpegai_reconstruction_binding.rs`; `.github/workflows/wai-jpegai-e2e.yml` | | `wai.image.jpegai` exact decoder across platforms: the `jpegai/v1` codestream decodes to the same RGB8 bytes, and to the same bits in the float values the decoder reports after each stage (values inside a stage are not hashed), on each platform measured | measured 2026-09-28 with rustc 1.98.0, release builds, inputs checked against their SHA-256, all four on the source of the change that added this row: macOS arm64 (native, on the development machine); wasm32-wasip2 (wasmtime 49, on the same machine); Linux arm64 (glibc 2.36, a container in an arm64 Linux VM on that machine); Linux x86_64 (glibc 2.36, the x86_64 build run in that container under user-mode emulation, QEMU 7.0, not on x86_64 hardware). All nine pinned hashes matched on each platform. On all four they also matched on an earlier state of that change, before the decode's `exp` and rounding moved to `float_repro`. Before the move the decode called the platform `exp` on every target, and its roundings compiled to calls to the symbols `round`, `roundf` and `rint` on x86_64 and `round` and `roundf` on wasm32, which the link resolves: an x86_64 Linux test executable inspected after the move, whose other code still rounds this way, took them from Rust's compiler builtins, not the C library (for wasm32 this was not checked). Not measured: macOS x86_64 (no x86_64 translation layer is installed on the development machine), Windows, x86_64 hardware, 32-bit targets. The test skips unless the inputs are supplied, and no CI job supplies them to it | `wai-rs/tests/jpegai_cross_platform.rs` | | `float_repro`, the decode's `exp` and rounding: the same bits over a fixed grid of inputs; `exp` within 2 ULPs of the host math library (1 ULP measured); rounding equal to the standard library's | library unit tests under the library-test step of the conformance workflow (`neural_int`), which has not yet run them on a runner. The pinned-bits and rounding tests never skip. The host comparison asserts the host library's accuracy as much as this module's. Its gate selects by target, not by host: it runs on macOS arm64, on Linux x86_64 and arm64 with glibc, and on WASI, and is reported as ignored, with the reason, elsewhere, Windows included. On the source of the change that added this row it was run against these hosts' `exp`, with a worst case of 1 ULP on each: Apple's math library on macOS arm64; the WASI C library under wasmtime; glibc 2.36 on Linux arm64, and glibc 2.35 in the Ubuntu 22.04 image the workflow's Linux arm64 runner used on 2026-09-28; glibc 2.36 on Linux x86_64 under user-mode emulation, against three of the four `exp` variants glibc chooses among by processor feature (the variant each run used follows from the features glibc's loader reported active): SSE2 (QEMU 7.0, whose emulated processor reports none of AVX, FMA, AVX2 and FMA4), AVX (QEMU 7.2, `-cpu max` with FMA and AVX2 masked by glibc's `hwcaps` tunable, which leaves AVX active) and FMA (QEMU 7.2, `-cpu max`). On a separate probe of 20 million inputs, the SSE2 and AVX variants returned the same bits, and the FMA variant different bits for some of them. Not compared: the FMA4 variant, which glibc chooses on a processor that has FMA4 but not both FMA and AVX2, and which neither emulator offers. Not run on x86_64 hardware or on other glibc versions, where the gate still runs it; a failure there would say that the host's `exp` and this one are more than 2 ULPs apart, not that `float_repro`'s bits changed | `wai-rs/src/float_repro.rs` tests | | The exact decode path calls no platform math library | two checks. Clippy's `disallowed_methods` (`wai-rs/clippy.toml`), denied in each module on the decode path, rejects any use of a listed float method, including `f64::exp(x)` and `.map(f64::exp)`. `tests/jpegai_decode_symbols.rs` lists the symbols of an executable that links only the decoder's entry points and weight loaders. It rejects any C math-library function the executable imports, and any it defines other than the round-to-integral functions (`floor`, `round`, `rint` and the like), whose results IEEE 754 fixes. It allows those definitions because a definition shows what the linker kept, not what the decode calls: linked with GNU ld 2.40 for x86_64 Linux, the debug executable defined the compiler builtins' `floor`, though nothing in it referred to `floor`. On the source of the change that added this row, both checks pass on macOS arm64, the symbol check in debug and release builds. The symbol check also passes, in debug builds, on Linux arm64 (the container above) and on Linux x86_64 under user-mode emulation, linked with GNU ld (reporting that `floor` as allowed) and with LLD. With a platform `exp` put back as `f64::exp(…)`, both fail on macOS arm64 while the decoder's unit tests still pass, and the symbol check fails on Linux x86_64 with GNU ld. With a float `%` put into the decode, the symbol check fails on macOS arm64 and clippy, which does not see operators, passes. The conformance workflow runs both on its unix legs; neither has run on a runner yet | `wai-rs/clippy.toml`, `wai-rs/tests/jpegai_decode_symbols.rs`, `.github/workflows/wai-conformance.yml` | | `wai.image.jpegai` scope: the reference codestream is accepted; copies with β, `model_id`, the colour transform or an LSBS flag changed are refused; a flush-to-zero or round-toward-zero environment is detected. The float ONNX path calls the same check | library unit tests over a committed copy of the reference codestream, which never skip and fall under the library-test step of the conformance workflow (`neural_int`); the environment test runs on arm64 only. Run on the four platforms above, on the source of the change that added the cross-platform row (the environment test on macOS arm64 and Linux arm64). The float path's refusal (β and `model_id` edits) is tested only with its ONNX bundle supplied (`jpegai_scope_ort.rs`, skips otherwise) | `wai-rs/src/jpegai_decode.rs` tests, `wai-rs/tests/vectors/jpegai/` | | SPEC §7's replicate rule: the registry class decides first, so an integer-declared prior over `wai.image.jpegai` or a float codec is refused; `wai.image.jpegai` is the only `NeuralEntropyExact` capability; the standard codecs sit in the classes SPEC §7 lists, and the tolerance and unclassified ones are not refused; an encoder that pins no prior (`wai wrap`) writes `replicate` only over the standard classes, for the capability and for any declared fallback, or, in the `WAI2` companion form, for the base payload's capability and each companion's | library unit tests (`codecs`, `container`) that never skip and fall under the same library-test step. The strict-decode test in `tests/neural_native_decode.rs` needs the `neural` feature; `wai-neural-gates.yml` runs it on the Linux runner, in an Ubuntu 24.04 image because the `neural` test binary does not link in that runner's default 22.04 image. The workflow has not run through the forge; its job passed under `forgejo-runner exec` on the Linux runner's host (Linux x86_64, 2026-09-27). The test that runs the `wai wrap` binary needs the `codecs` feature, which no CI job builds. Otherwise run so far on macOS arm64 only | `wai-rs/src/codecs/mod.rs`, `container.rs` tests; `wai-rs/tests/neural_native_decode.rs`, `cli_wrap_intent.rs` | -| `wai.neural.int_hyper` and `wai.video.int_hyper` at full model scale | tests that skip when the exported model is absent; not run in CI | `wai-rs/tests/int_hyper_synth.rs`, `int_video_sota_keyframe.rs` | +| `wai.neural.int_hyper` and `wai.video.int_hyper` at full model scale (the pretrained `mbt2018_mean`, exported to integers): the Rust decode reproduces the export tool's integer reference byte for byte (synthesis, hyper-decoder, the whole conditional bitstream, an eight-frame clip), and costs at most 1 dB against the same model's float decode on one pinned photograph | `wai-sota-vectors.yml` regenerates the vectors on the runner from pinned inputs (the photograph's pixels and the checkpoint's bytes by SHA-256, the Python stack by version) and runs both tests with `WAI_REQUIRE_SOTA_VECTORS=1`, under which a missing vector fails instead of skipping. The job also fails on any skip banner in the test log, and runs `require_vectors_rule.rs`, which checks that without the switch a variable naming absent or incomplete inputs fails as well. Rust is checked against the export of the same run on the same machine: the vectors come from float computation, and some of them differed between machines and between runs, so none of their hashes is pinned and none is compared across machines. The workflow has not run through the forge. Its job passed under `forgejo-runner exec` on the Linux runner's host (x86_64 hardware, 2026-09-27) in an earlier form that did not yet run `require_vectors_rule.rs`; its current steps have run on macOS arm64 only, outside a runner | `wai-rs/tests/int_hyper_synth.rs`, `int_video_sota_keyframe.rs`, `require_vectors_rule.rs`; `.github/workflows/wai-sota-vectors.yml` | | `wai.audio.int_codec`, `wai.splat.int_codec` | committed vectors that never skip, matched against an independent Python integer reference; their cargo features (`audio_int`, `splat_int`) are enabled by no CI job | `wai-rs/tests/int_audio.rs`, `int_splat.rs` | So at `entropy-consistency`, WAI's JPEG AI symbol decode is demonstrated across @@ -495,8 +501,9 @@ identity (x86_64 against arm64) is demonstrated for `int_synth`, integer video a `int_hyper` on committed small-scale vectors; of these, only `int_synth` has also been decoded on Windows on main. `wai.audio.int_codec` and `wai.splat.int_codec` reproduce a separate implementation's output, but no CI job runs them and no -cross-architecture comparison exists. For full-scale models, identity rests on -construction, not on a cross-machine comparison of decoded output. +cross-architecture comparison exists. For full-scale `int_hyper`, identity rests on +construction, not on a cross-machine comparison of decoded output: CI checks its +decode against the export's integer reference on one machine per run. For the JPEG AI exact decoder's pinned reconstruction (§5), identity across machines rests on construction, and on one comparison: a single codestream on the diff --git a/wai/tools/jpegai-e2e-v1.sha256 b/wai/tools/jpegai-e2e-v1.sha256 new file mode 100644 index 000000000..38e9adacf --- /dev/null +++ b/wai/tools/jpegai-e2e-v1.sha256 @@ -0,0 +1,13 @@ +f3b325c583c6b7613ccddc2d89bfaa50e0e8314ceecd8d82fc4df7fb159b6083 hs_uv.json +1d193f15b5d8d1531ef9ffef8676904d76cfedbd1d2df931d581dfb26daa2688 hs_y.json +2c5f6f12fcf9a36489553e0e02e95684c339d82bab64cb7bae76a4635f9ec56d hyp_uv.json +32ab3a153e33ded902b1d8a6c413047634bc39e8b74525dfbb40eded0763433d hyp_y.json +bd3aa0c23294a462895416eda38399468781163858229c386deaba58def5a1c3 icci.json +562740aba18ba4dccdc007b3d54a669ecd18feac9d0bd1fd7a29e66a0ba498ee mcm0_y.json +fbd4b7c05e81cd32df4db4e2f6dfa4d6e3fc2edb6c21150f80ef5ac04765228d mcm1_y.json +b42d6fa96bf865dd2db47955aba14d8d65b9152c941be3ee3e92eae5643834a7 mcm2_y.json +e06766d0a7bff16188a0e96297d6e498e28fbaf7850f68b0a29447a9391c9d9d mcm3_y.json +5353b76f25e59f52d85adc896c9dc6b4aa8552589e3e4e1d97b8f29692e37b13 norm.json +8ddf8fe9dd15569910ad5755a5495526df959835bcf86a259d06ab6445f02b4e synth_uv.json +4f2c9ae807ea22197adaae0263c9d6d4d53ef7d4fccb81848b7bb4da9ef0ea69 synth_y.json +99840e5f472423f5a1261c9ec219d12dc910e9623d752672555dfea69ea4a097 test256.bin diff --git a/wai/tools/requirements-sota-int.txt b/wai/tools/requirements-sota-int.txt new file mode 100644 index 000000000..0ff58b114 --- /dev/null +++ b/wai/tools/requirements-sota-int.txt @@ -0,0 +1,25 @@ +# Pinned Python stack for regenerating the int_hyper integer-codec vectors (the +# pretrained mbt2018_mean, exported to integers; tools/wai_sota_vectors.sh). The vectors are regenerated rather than committed, +# so the export must run against a fixed stack: an upgrade that changes the +# pretrained weights or the float analysis transform must arrive as a reviewed +# change here, not silently on the next CI run. +# +# torch/torchvision: install from the CPU wheel index first on Linux, or pip +# resolves the CUDA builds (several GB). `==2.14.0` matches `2.14.0+cpu`. +# pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/cpu +# pip install -r tools/requirements-sota-int.txt +torch==2.14.0 +torchvision==0.29.0 +# BSD-3-Clause-Clear. Supplies the mbt2018_mean architecture and the URL of its +# published checkpoint, mbt2018-mean-6-a19628ab.pth.tar. The checkpoint is NOT +# verified by compressai or torch.hub (the zoo calls torch.hub without +# check_hash), so wai_sota_int_synth_export.py pins its full SHA-256 and refuses +# to load any other bytes. +compressai==1.2.8 +# compressai 1.2.8 requires numpy<2. +numpy==1.26.4 +pillow==12.3.0 +# Source image only: `skimage.data.astronaut()` ships inside the wheel and is a +# NASA photograph documented there as "No known copyright restrictions, released +# into the public domain" (the function's docstring). +scikit-image==0.26.0 diff --git a/wai/tools/wai_int_hyper_tiny_vector.py b/wai/tools/wai_int_hyper_tiny_vector.py index 29cc5fc81..6eff59173 100644 --- a/wai/tools/wai_int_hyper_tiny_vector.py +++ b/wai/tools/wai_int_hyper_tiny_vector.py @@ -1,12 +1,13 @@ #!/usr/bin/env python3 """Generate a TINY, self-contained `wai.neural.int_hyper` conformance vector. -The SOTA int_hyper vector needs a ~23 MB PTQ model, too large to commit — so the -Rust conformance test SKIPS without it, and CI never exercises the decode. This -emits a SMALL synthetic conditional codec (deterministic random integer weights) -that drives the SAME decode path end to end — factorized z -> integer hyper- -decoder h_s -> integer-sigma entropy bucketing -> integer rANS -> integer -synthesis g_s -> RGB — with a committed expected FNV-1a. A few KB, so it commits +The pretrained int_hyper vector needs a ~23 MB PTQ model, too large to commit — +so it is regenerated per run (tools/wai_sota_vectors.sh, wai-sota-vectors.yml) +and checked on one machine at a time. This emits a SMALL synthetic conditional +codec (deterministic random integer weights) that drives the SAME decode path +end to end — factorized z -> integer hyper-decoder h_s -> integer-sigma entropy +bucketing -> integer rANS -> integer synthesis g_s -> RGB — with a committed +expected FNV-1a. A few KB, so it commits and the int_hyper decode is CI-verifiable with no external artifacts. python tools/wai_int_hyper_tiny_vector.py @@ -20,7 +21,7 @@ import numpy as np from compressai._CXX import pmf_to_quantized_cdf from compressai.ans import RansEncoder -# Canonical integer-net runner + helpers (the same the SOTA export uses). +# Canonical integer-net runner + helpers (the same the pretrained-model export uses). sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from wai_sota_int_synth_export import run_int_net, det_build_index_vec, fnv1a, model_to_bin diff --git a/wai/tools/wai_int_video_sota_keyframe_encode.py b/wai/tools/wai_int_video_sota_keyframe_encode.py index 71e1db83d..f6f8cd1e8 100644 --- a/wai/tools/wai_int_video_sota_keyframe_encode.py +++ b/wai/tools/wai_int_video_sota_keyframe_encode.py @@ -1,17 +1,20 @@ #!/usr/bin/env python3 -"""Encode a REAL 256x256 int_hyper-keyframe clip: the I-frame is the SOTA learned -integer codec int_hyper (its byte-exact 256x256 reconstruction, dumped by the -`int_hyper_decode_frame` Rust example), and the P-frames pan it with integer block -motion compensation + integer-quantized residual. Skip-if-absent: needs the SOTA -int_hyper vector dir (default /tmp/sota_int) holding frame0.rgb. +"""Encode a REAL 256x256 int_hyper-keyframe clip: the I-frame is the integer +learned codec int_hyper (the pretrained mbt2018_mean exported to integers; its +byte-exact 256x256 reconstruction, dumped by the `int_hyper_decode_frame` Rust +example), and the P-frames pan it with integer block motion compensation + +integer-quantized residual. Needs the int_hyper vector dir (default +/tmp/sota_int) holding frame0.rgb, and exits non-zero without it. +tools/wai_sota_vectors.sh runs the whole chain. - cargo run --release --example int_hyper_decode_frame --features neural_int -- /frame0.rgb + cargo run --release --no-default-features --features neural_int --example int_hyper_decode_frame -- /frame0.rgb python tools/wai_int_video_sota_keyframe_encode.py Emits /video_sota_bitstream.json + /video_sota.json (the P-frame stream + expected per-frame / clip hashes; frame 0's hash is int_hyper's). The Rust test `int_video_sota_keyframe` composes int_hyper::decode_conditional with -int_video::decode_pframes and checks these, skipping when the dir is absent. +int_video::decode_pframes and checks these (skipping loudly when they are absent, +failing instead under WAI_REQUIRE_SOTA_VECTORS=1). """ import sys, os, json, base64 import numpy as np @@ -75,8 +78,9 @@ def main(d, prefix='video_sota'): global H, W fp = f'{d}/frame0.rgb' if not os.path.exists(fp): - print(f'skip: {fp} absent (run the int_hyper_decode_frame example first)') - return + # A producer that exits 0 without producing lets every consumer downstream + # skip in turn, and the whole chain reports green having checked nothing. + sys.exit(f'error: {fp} absent (run the int_hyper_decode_frame example first)') raw = open(fp, 'rb').read() H = W = int(round((len(raw) / 3) ** 0.5)) # square frame inferred from the dump frame0 = np.frombuffer(raw, np.uint8).reshape(H, W, 3).astype(np.int64) diff --git a/wai/tools/wai_jpegai_fetch_e2e_inputs.sh b/wai/tools/wai_jpegai_fetch_e2e_inputs.sh new file mode 100755 index 000000000..1ec293a4c --- /dev/null +++ b/wai/tools/wai_jpegai_fetch_e2e_inputs.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# Fetch the JPEG-AI whole-decoder test inputs from WAI's content-versioned models +# root and check every byte against the SHA-256 manifest committed next to this +# script (tools/jpegai-e2e-v1.sha256). +# +# tools/wai_jpegai_fetch_e2e_inputs.sh [out-dir] # default /tmp/jpegai_e2e +# +# Then run the tests that read them (tests/jpegai_e2e_full.rs, +# tests/jpegai_reconstruction_binding.rs) with +# WAI_JPEGAI_CODESTREAM=/test256.bin +# WAI_JPEGAI_WEIGHTS_DIR= +# WAI_JPEGAI_NORM=/norm.json +# +# What is fetched: the eleven integer/PTQ weight files the decoder runs +# ({hs,hyp,synth}_{y,uv}.json, mcm{0..3}_y.json, icci.json), norm.json (normative +# tables plus the reference decoder's RGB output for the test picture) and +# test256.bin (the test codestream). They were extracted by +# tools/wai_jpegai_build_e2e_inputs.py from the JPEG-AI reference-software assets, +# which are not in this repository; the models root publishes the extraction under +# an immutable versioned path (jpegai/v1/). The manifest pins those exact bytes, so +# a changed or truncated file at the URL fails here instead of changing what the +# tests check. +# +# WAI_JPEGAI_INPUTS_URL overrides the base URL (a mirror holding the same files). +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +manifest="$here/jpegai-e2e-v1.sha256" +base="${WAI_JPEGAI_INPUTS_URL:-https://wai.transaction.science/models/jpegai/v1}" +out="${1:-/tmp/jpegai_e2e}" + +mkdir -p "$out" +out="$(cd "$out" && pwd)" + +if command -v sha256sum >/dev/null 2>&1; then + sha() { sha256sum "$@"; } +else + sha() { shasum -a 256 "$@"; } +fi + +while read -r want name; do + [ -n "$name" ] || continue + curl --proto '=https' --tlsv1.2 -fsSL --retry 3 -o "$out/$name.part" "$base/$name" + got="$(sha "$out/$name.part" | cut -d' ' -f1)" + if [ "$got" != "$want" ]; then + echo "error: $base/$name has sha256 $got, pinned $want" >&2 + rm -f "$out/$name.part" + exit 1 + fi + mv "$out/$name.part" "$out/$name" + echo "ok $want $name" +done < "$manifest" + +echo "JPEG-AI e2e inputs verified in $out" diff --git a/wai/tools/wai_sota_int_synth_export.py b/wai/tools/wai_sota_int_synth_export.py index b4ea512d5..d42df6103 100644 --- a/wai/tools/wai_sota_int_synth_export.py +++ b/wai/tools/wai_sota_int_synth_export.py @@ -1,17 +1,58 @@ #!/usr/bin/env python3 -"""Export a pretrained SOTA-class learned-codec SYNTHESIS as an INTEGER-exact -decoder and prove it preserves quality: per-channel int12 weights + fixed-point -activations + integer IGDN, all i64 — byte-identical on every machine -(NeuralIntegerExact), at the float codec's PSNR. +"""Export the pretrained mbt2018_mean mean-scale hyperprior (CompressAI zoo, +quality 6, MSE) as an INTEGER-exact decoder and measure what that costs: +per-channel int12 weights + fixed-point activations + integer IGDN, all i64 — +byte-identical on every machine (NeuralIntegerExact). The tool reports the +integer and float PSNR on the given image; tests/int_hyper_synth.rs requires the +integer decode within 1 dB of the float decode. python tools/wai_sota_int_synth_export.py + +Normally run through tools/wai_sota_vectors.sh, which pins the source image and +the Python stack and produces the whole vector directory the Rust int_hyper tests +read. """ -import sys, os, json, hashlib +import sys, os, io, json, hashlib import numpy as np, torch, warnings; warnings.filterwarnings('ignore') -from compressai.zoo import mbt2018_mean +from compressai.zoo import image as zoo_image +from compressai.zoo.pretrained import load_pretrained AFRAC = 12; WBITS = 12; SHIFT = 24; GF = 16 +# The pretrained checkpoint the model-derived vectors come from (all but the source +# image, source.png and source.rgb), pinned by its full SHA-256. +# CompressAI's own `mbt2018_mean(pretrained=True)` fetches it through torch.hub +# without checking any hash (and unpickles it with weights_only=False), so a changed +# object at the URL, or a stale file in the torch.hub cache, would silently yield +# different vectors that the Rust tests — comparing against Python output built +# from the same checkpoint — would still pass. So the bytes are checked here, and +# only then loaded, as tensors only (weights_only=True). +CKPT_URL = "https://compressai.s3.amazonaws.com/models/v1/mbt2018-mean-6-a19628ab.pth.tar" +CKPT_SHA256 = "a19628aba19732125e02caf5c38e233edbe2a641145a5b1779c51f4527f51dd6" + + +def load_pinned_mbt2018_mean(): + """mbt2018_mean quality 6 (MSE), built from checkpoint bytes whose SHA-256 is + CKPT_SHA256 — or exit non-zero. Downloads into the torch.hub cache if absent.""" + zoo_url = zoo_image.model_urls["mbt2018-mean"]["mse"][6] + if zoo_url != CKPT_URL: + sys.exit(f"error: the installed compressai points mbt2018_mean q6 at {zoo_url}, " + f"not the pinned {CKPT_URL}; review the new checkpoint before re-pinning") + path = os.path.join(torch.hub.get_dir(), "checkpoints", os.path.basename(CKPT_URL)) + if not os.path.exists(path): + os.makedirs(os.path.dirname(path), exist_ok=True) + torch.hub.download_url_to_file(CKPT_URL, path, progress=False) + with open(path, "rb") as f: + blob = f.read() + got = hashlib.sha256(blob).hexdigest() + if got != CKPT_SHA256: + sys.exit(f"error: {path} has sha256 {got}, pinned {CKPT_SHA256}. " + "Delete the file to re-download; if the published checkpoint itself " + "changed, re-pin only after reviewing it.") + print(f"checkpoint {os.path.basename(path)}: sha256 {got} (pinned, verified)") + state = torch.load(io.BytesIO(blob), map_location="cpu", weights_only=True) + return zoo_image.model_architectures["mbt2018-mean"].from_state_dict(load_pretrained(state)) + def q_weight_pc(w): # ConvTranspose [in,out,k,k] → per-OUT-channel int12 + scale s = np.abs(w).max(axis=(0,2,3), keepdims=True)/(2**(WBITS-1)-1) + 1e-12 wi = np.round(w/s).clip(-(2**(WBITS-1)), 2**(WBITS-1)-1).astype(np.int64) @@ -149,11 +190,27 @@ def model_to_bin(model): def main(imgpath, outdir): from PIL import Image + # The ENCODER side (g_a, h_a, the float h_s used for the reference) is float, so + # its latent can depend on kernel choice and reduction order. Both are pinned + # to make two runs of this tool on one machine emit identical vectors. That + # held for repeated runs on an Apple Silicon Mac, but two runs on an x86_64 + # Linux runner wrote different cond.json and int_synth_sota.json, most likely + # in psnr_float (`base` below), which the Rust tests read only as a quality + # threshold; the latent and both rgb_hash values matched. The decode being + # tested is integer and needs no such pinning. + torch.use_deterministic_algorithms(True) + torch.set_num_threads(1) + # Verify the checkpoint before writing anything, so a refused checkpoint leaves + # no partial vector directory behind. + m = load_pinned_mbt2018_mean().eval() os.makedirs(outdir, exist_ok=True) img = Image.open(imgpath).convert('RGB') - x = torch.from_numpy(np.array(img).astype(np.float32)/255.).permute(2,0,1).unsqueeze(0) + src8 = np.array(img) + x = torch.from_numpy(src8.astype(np.float32)/255.).permute(2,0,1).unsqueeze(0) h,w = x.shape[2]//64*64, x.shape[3]//64*64; x = x[:,:,:h,:w] - m = mbt2018_mean(quality=6, pretrained=True).eval() + # The exact pixels the codec saw (HWC u8), so a decoder can measure its own + # reconstruction quality instead of trusting the PSNR this tool reports. + np.ascontiguousarray(src8[:h, :w]).tofile(f'{outdir}/source.rgb') def psnr(a,b): return float(-10*np.log10(((a-b)**2).mean()+1e-12)) with torch.no_grad(): y = m.g_a(x); z = m.h_a(y); z_hat = torch.round(z) @@ -199,7 +256,7 @@ def main(imgpath, outdir): for byte in rgb8.tobytes(): h = ((h ^ byte) * 1099511628211) & 0xFFFFFFFFFFFFFFFF digest = format(h, '016x') - json.dump({'afrac':AFRAC,'wbits':WBITS,'shift':SHIFT,'psnr_int':p_int,'rgb_hash':digest, + json.dump({'afrac':AFRAC,'wbits':WBITS,'shift':SHIFT,'psnr_int':p_int,'psnr_float':base,'rgb_hash':digest, 'out_c':int(rec.shape[0]),'out_h':int(rec.shape[1]),'out_w':int(rec.shape[2])}, open(f'{outdir}/int_synth_sota.json','w')) json.dump(model, open(f'{outdir}/int_synth_sota_model.json','w')) @@ -308,7 +365,7 @@ def main(imgpath, outdir): 'n_z':int(Cz),'zh':int(z_sym.shape[1]),'zw':int(z_sym.shape[2]), 'z_rans':list(z_bits),'rans':list(bits)}, open(f'{outdir}/cond_bitstream.json','w')) - json.dump({'rgb_hash':cond_hash,'psnr_cond':p_cond, + json.dump({'rgb_hash':cond_hash,'psnr_cond':p_cond,'psnr_float':base, 'out_c':int(rec_int.shape[0]),'out_h':int(rec_int.shape[1]),'out_w':int(rec_int.shape[2])}, open(f'{outdir}/cond.json','w')) print(f'wrote conditional bitstream: {len(z_bits)} z + {len(bits)} y rANS bytes ' diff --git a/wai/tools/wai_sota_source_image.py b/wai/tools/wai_sota_source_image.py new file mode 100644 index 000000000..9089f8bb0 --- /dev/null +++ b/wai/tools/wai_sota_source_image.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Write the pinned source photograph the int_hyper integer-codec vectors (the +pretrained mbt2018_mean, exported to integers) are regenerated from. + + python tools/wai_sota_source_image.py + +The image is a 256x256 crop of `skimage.data.astronaut()` — a photograph that +ships inside the scikit-image wheel and is documented there as public domain +(NASA; "No known copyright restrictions, released into the public domain"). It +is pinned by the SHA-256 of the cropped RGB bytes, so a different image can never +be substituted silently: a mismatch is an error, not a warning. + +A real photograph, rather than a synthetic pattern, is what makes the quality +check meaningful: the integer decode is checked against the same model's float +decode on natural-image content, and must cost at most 1 dB there +(tests/int_hyper_synth.rs). That bar is relative, on this one image; it says +nothing about how the model compares with other codecs, or about other images. +The 42.6 dB development figure for this codec was taken on a different +photograph that is not in this repository. +""" +import hashlib +import sys + +import numpy as np +from PIL import Image +from skimage import data + +CROP = (slice(64, 320), slice(128, 384)) # rows, cols of the 512x512 original +SHA256 = "29cb3aee1a55f9585474d056b65edb0aaebbb80371ba200e85030d63665c19d6" + + +def main(out: str) -> int: + rgb = np.ascontiguousarray(data.astronaut()[CROP]) + got = hashlib.sha256(rgb.tobytes()).hexdigest() + if rgb.shape != (256, 256, 3) or got != SHA256: + print(f"error: source crop is {rgb.shape} sha256 {got}; pinned (256, 256, 3) {SHA256}. " + "The scikit-image sample changed — pin the new hash only after reviewing the image.", + file=sys.stderr) + return 1 + Image.fromarray(rgb).save(out) + print(f"wrote {out}: 256x256 RGB, sha256(rgb) {got}") + return 0 + + +if __name__ == "__main__": + if len(sys.argv) != 2: + print(__doc__, file=sys.stderr) + raise SystemExit(2) + raise SystemExit(main(sys.argv[1])) diff --git a/wai/tools/wai_sota_vectors.sh b/wai/tools/wai_sota_vectors.sh new file mode 100755 index 000000000..2f9bf9fdf --- /dev/null +++ b/wai/tools/wai_sota_vectors.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Regenerate the int_hyper integer-codec vectors (the pretrained mbt2018_mean, +# exported to integers) that the Rust tests read from WAI_SOTA_INT_DIR +# (tests/int_hyper_synth.rs, tests/int_video_sota_keyframe.rs). +# +# tools/wai_sota_vectors.sh [out-dir] # default /tmp/sota_int +# +# Needs the pinned Python stack (tools/requirements-sota-int.txt) on PATH as +# `python` (or set PYTHON), and cargo. The pretrained mbt2018_mean checkpoint is +# downloaded into the torch.hub cache on first use and checked against its pinned +# SHA-256 (CKPT_SHA256 in wai_sota_int_synth_export.py) before it is loaded. +# +# The chain, each step consuming the one before: +# 1. the pinned public-domain source photograph → source.png +# 2. the integer export of mbt2018_mean → model, bitstream, hashes +# 3. the Rust integer decode of that bitstream → frame0.rgb (the keyframe) +# 4. the P-frame encoder panning that keyframe → video bitstream, hashes +# Every step must succeed and every file the tests read must exist at the end, or +# this exits non-zero: a partial directory would let a test skip instead of fail. +# +# The output is not portable, and part of it is not repeatable run to run on +# every machine. Step 2 runs the float model (torch, CPU) to encode the +# photograph, on the pinned stack, which each platform installs as its own build. +# - Between an Apple Silicon Mac and an x86_64 Linux runner, +# int_synth_sota_input.json (the synthesis-only latent, which the float +# encoder and hyper-decoder compute) differed, and with it the rgb_hash in +# int_synth_sota.json. Each side wrote the same latent in all of its runs. +# - Two runs on that one runner wrote different cond.json and +# int_synth_sota.json, most likely in psnr_float, the float decode's PSNR +# (the runner's files were not kept, so the field is inferred). Repeated runs +# on the Mac wrote identical files. Only the quality test reads psnr_float, +# against thresholds (the 1 dB bar and a 30 dB floor). +# - source.png differed between the Mac and the runner in its compressed bytes. +# Step 1 writes it with the PNG encoder from the pinned pixels, before any +# model runs, and its pixels (source.rgb) were identical, so this is the +# encoder's output, not float computation. +# The other files matched in these comparisons, which does not make them +# portable. So check a decode against the vectors of the same run, and do not +# pin a hash from these files for another run, on this machine or another: it +# can differ with both decoders correct. The decode's cross-architecture check +# uses the committed tiny vector (wai-rs/tests/vectors/int_hyper/), whose hashes +# are pinned. +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +wai="$(dirname "$here")" +out="${1:-/tmp/sota_int}" +py="${PYTHON:-python}" + +mkdir -p "$out" +out="$(cd "$out" && pwd)" + +"$py" "$here/wai_sota_source_image.py" "$out/source.png" +"$py" "$here/wai_sota_int_synth_export.py" "$out/source.png" "$out" +(cd "$wai/wai-rs" && cargo run --release --no-default-features --features neural_int \ + --example int_hyper_decode_frame -- "$out" "$out/frame0.rgb") +"$py" "$here/wai_int_video_sota_keyframe_encode.py" "$out" + +missing=0 +for f in source.rgb \ + int_synth_sota.json int_synth_sota_model.json int_synth_sota_model.bin int_synth_sota_input.json \ + int_hs.json int_hs_model.json int_hs_model.bin int_hs_input.json \ + cond.json cond_entropy.json cond_bitstream.json \ + frame0.rgb video_sota.json video_sota_bitstream.json; do + if [ ! -s "$out/$f" ]; then + echo "error: $out/$f was not produced" >&2 + missing=1 + fi +done +[ "$missing" -eq 0 ] || exit 1 +echo "int_hyper vectors complete in $out" diff --git a/wai/wai-rs/examples/int_hyper_decode_frame.rs b/wai/wai-rs/examples/int_hyper_decode_frame.rs index 756ff9e4b..55ad259f1 100644 --- a/wai/wai-rs/examples/int_hyper_decode_frame.rs +++ b/wai/wai-rs/examples/int_hyper_decode_frame.rs @@ -1,8 +1,8 @@ -//! Decode a SOTA `int_hyper` conditional bitstream from a directory and dump the +//! Decode an `int_hyper` conditional bitstream from a directory and dump the //! reconstructed frame as raw RGB (HWC u8) — the byte-exact keyframe the //! `int_hyper`-keyframe video codec builds its P-frames on. //! -//! cargo run --release --example int_hyper_decode_frame --features neural_int -- +//! cargo run --release --no-default-features --features neural_int --example int_hyper_decode_frame -- //! //! `` is an int_hyper vector dir (e.g. `/tmp/sota_int`) holding //! `int_synth_sota_model.json`, `int_hs_model.json`, `cond_entropy.json`, diff --git a/wai/wai-rs/src/codecs/mod.rs b/wai/wai-rs/src/codecs/mod.rs index d09940ff4..67c753ae7 100644 --- a/wai/wai-rs/src/codecs/mod.rs +++ b/wai/wai-rs/src/codecs/mod.rs @@ -49,7 +49,7 @@ pub const CAP_NEURAL_WAVTOKENIZER: &str = "wai.neural.wavtokenizer"; // ultra-lo pub const CAP_NEURAL_BMSHJ2018: &str = "wai.neural.bmshj2018"; // bmshj2018-factorized image codec (2018 baseline) pub const CAP_NEURAL_MBT2018_MEAN: &str = "wai.neural.mbt2018_mean"; // mean-scale HYPERPRIOR image codec (Gaussian-conditional entropy) — decoder IMPLEMENTED in this crate (neural/hyperprior.rs); the conditional-entropy family, stronger than factorized bmshj2018; sink-supplied ONNX (hyper_synthesis + synthesis) pub const CAP_NEURAL_INT_SYNTH: &str = "wai.neural.int_synth"; // INTEGER fixed-point synthesis (neural/int_synth.rs): all-i64 transposed-conv decode, no float/ONNX — bit-identical on every machine BY CONSTRUCTION. The first capability that can soundly back intent=replicate cross-hardware (determinism IS integer-exact). Weights are sink-supplied (int_synth.json from a QAT model) -pub const CAP_NEURAL_INT_HYPER: &str = "wai.neural.int_hyper"; // INTEGER fixed-point CONDITIONAL codec (int_hyper_synth.rs): the full Gaussian-conditional decode integer end-to-end — factorized rANS-decode z (z_q = s·2^afrac + median_q) → integer hyper-decoder h_s → σ,μ → integer-σ bucket (int_entropy) → integer rANS-decode residual → y_hat = r·2^afrac + μ → integer synthesis g_s. Both bitstreams entropy-coded, no raw latents. No float/ONNX anywhere, so the WHOLE bitstream→image decode is bit-identical on every machine BY CONSTRUCTION — int_synth's sibling that adds the conditional entropy. SOTA quality (PTQ of mbt2018_mean, ~42.6 dB). Tables sink-supplied +pub const CAP_NEURAL_INT_HYPER: &str = "wai.neural.int_hyper"; // INTEGER fixed-point CONDITIONAL codec (int_hyper_synth.rs): the full Gaussian-conditional decode integer end-to-end — factorized rANS-decode z (z_q = s·2^afrac + median_q) → integer hyper-decoder h_s → σ,μ → integer-σ bucket (int_entropy) → integer rANS-decode residual → y_hat = r·2^afrac + μ → integer synthesis g_s. Both bitstreams entropy-coded, no raw latents. No float/ONNX anywhere, so the WHOLE bitstream→image decode is bit-identical on every machine BY CONSTRUCTION — int_synth's sibling that adds the conditional entropy. Quality: PTQ of the pretrained mbt2018_mean (quality 6); CI (wai-sota-vectors.yml) requires the integer decode within 1 dB of the same model's float decode on a pinned test photograph — measured cost there 0.19 dB (35.985 vs 36.175 dB, Apple Silicon Mac); image-dependent, see int_hyper_synth.rs. Tables sink-supplied pub const CAP_NEURAL_ELIC: &str = "wai.neural.elic"; // checkerboard CONTEXT-MODEL image codec (neural/context.rs) — the JPEG-AI/MLIC++ entropy class: hyperprior + 2-pass spatial context (anchor → context net → non-anchor). Decoder IMPLEMENTED; sink-supplied ONNX (hyper_synthesis + context + synthesis). WAI-native, NOT the ISO JPEG AI bitstream pub const CAP_NEURAL_MLICPP: &str = "wai.neural.mlicpp"; // MLIC++ (−13.39% BD-rate vs VVC/VTM-17): the multi-reference entropy model — channel-autoregressive slices + checkerboard + global attention. Decoder IMPLEMENTED (neural/mlicpp.rs): the slice/checkerboard decode orchestration; the attention nets are sink-supplied ONNX (hyper_synthesis + entropy + synthesis) pub const CAP_NEURAL_MLICV2: &str = "wai.neural.mlicv2"; // MLICv2 / MLICv2+ (2025): CURRENT SOTA learned image, −20.46% BD-rate vs VTM-17 (vs MLIC++'s −13.39%). SAME multi-reference channel-autoregressive decode orchestration as mlicpp — the gains are in the MetaFormer transform + enhanced entropy/global-context nets, all sink-supplied ONNX. Decoder route IMPLEMENTED (shares neural/mlicpp.rs) diff --git a/wai/wai-rs/src/int_hyper_synth.rs b/wai/wai-rs/src/int_hyper_synth.rs index 7b5348431..37657f39f 100644 --- a/wai/wai-rs/src/int_hyper_synth.rs +++ b/wai/wai-rs/src/int_hyper_synth.rs @@ -1,14 +1,26 @@ -//! Integer-exact synthesis of a SOTA-class learned codec (per-channel int12 +//! Integer-exact synthesis of a pretrained learned image codec (per-channel int12 //! weights + integer IGDN) — the decode side of a `NeuralIntegerExact` -//! conditional codec at SOTA quality. Pure i64, no float, no `ort`: the same +//! conditional codec, run on the pretrained mbt2018_mean mean-scale hyperprior +//! exported to integers. Pure i64, no float, no `ort`: the same //! discipline as `int_synth`/`int_entropy`, so the reconstruction is //! byte-identical on every machine. //! //! It runs the model the export tool emits (`tools/wai_sota_int_synth_export.py`): //! a stack of integer transposed-convolutions (per-output-channel `(mult,shift)` -//! requant) and inverse-GDN activations ([`crate::int_transform::igdn`]). On a -//! real pretrained `mbt2018_mean` it reconstructs at 42.6 dB — within ~0.6 dB of -//! the float baseline, fully integer. +//! requant) and inverse-GDN activations ([`crate::int_transform::igdn`]). +//! +//! Quality is image-dependent, so it is measured on a pinned image rather than +//! quoted. On the public-domain test photograph `tools/wai_sota_vectors.sh` +//! regenerates the vectors from (a 256×256 crop of scikit-image's `astronaut`), +//! measured on an Apple Silicon Mac: the integer decode of the pretrained +//! `mbt2018_mean`'s full conditional bitstream scores 35.985 dB (8-bit PSNR of this +//! decoder's output against the source pixels, `tests/int_hyper_synth.rs`), against +//! 36.175 dB for the same model's float decode (the export tool's PSNR of its float +//! reconstruction) — a cost of 0.19 dB. CI (`wai-sota-vectors.yml`) requires that +//! cost to stay within 1 dB on every run. It is a relative bar on one image and does +//! not compare the model with any other codec. An earlier development figure of +//! 42.6 dB (43.25 dB float), for the integer synthesis alone, was measured on a +//! photograph that is not in the repository and cannot be reproduced from it. use serde_json::Value; diff --git a/wai/wai-rs/src/lib.rs b/wai/wai-rs/src/lib.rs index e86a22b56..a4ca830b5 100644 --- a/wai/wai-rs/src/lib.rs +++ b/wai/wai-rs/src/lib.rs @@ -54,9 +54,10 @@ pub mod rans; #[cfg(feature = "neural_int")] pub mod int_transform; -// Integer-exact synthesis of a SOTA-class learned codec (per-channel int12 convs -// + integer IGDN) — the decode side of a NeuralIntegerExact conditional codec at -// SOTA quality. Pure i64, no `ort`. Runs the model the export tool emits. +// Integer-exact synthesis of a pretrained learned image codec (per-channel int12 +// convs + integer IGDN) — the decode side of a NeuralIntegerExact conditional +// codec (the pretrained mbt2018_mean, exported to integers; quality measured in +// int_hyper_synth.rs). Pure i64, no `ort`. Runs the model the export tool emits. #[cfg(feature = "neural_int")] pub mod int_hyper_synth; diff --git a/wai/wai-rs/tests/int_hyper_synth.rs b/wai/wai-rs/tests/int_hyper_synth.rs index 0c20c444b..3ff975267 100644 --- a/wai/wai-rs/tests/int_hyper_synth.rs +++ b/wai/wai-rs/tests/int_hyper_synth.rs @@ -1,32 +1,64 @@ -//! The Rust integer synthesis of a SOTA-class learned codec reproduces the -//! Python export tool's reconstruction BYTE-FOR-BYTE — proving the decode is -//! cross-language / cross-hardware exact at SOTA quality (mbt2018_mean, 42.6 dB, -//! fully integer). The model + input + expected FNV-1a checksum are emitted by -//! `tools/wai_sota_int_synth_export.py` into a directory pointed to by -//! `WAI_SOTA_INT_DIR` (default `/tmp/sota_int`); skipped if absent. +//! The Rust integer decode of `wai.neural.int_hyper` — the pretrained mbt2018_mean +//! mean-scale hyperprior (CompressAI zoo, quality 6), exported to integers — +//! reproduces the Python export tool's reconstruction BYTE-FOR-BYTE, and costs at +//! most 1 dB against the same model's float decode on the photograph the vectors +//! were built from. The decode is all-integer by construction; its +//! cross-architecture byte-equality is checked on the committed tiny vector +//! (`tests/vectors/int_hyper/`, `wai-byte-equality.yml`), not on these. The +//! pretrained-model tests below take their expected hashes from the export run +//! that wrote the vector directory, not from the repository (only the tiny +//! vector's test reads a committed hash). That export encodes with the float model, +//! and its files are not portable: the synthesis-only test's latent and its +//! `rgb_hash` were measured to differ between an Apple Silicon Mac and an x86_64 +//! Linux runner, and two runs on that runner wrote different `cond.json` and +//! `int_synth_sota.json` (most likely in `psnr_float`, which only the quality +//! test reads, against thresholds). None of these hashes is pinned for use by +//! another run (`tools/wai_sota_vectors.sh`). +//! +//! The model + bitstream + expected FNV-1a checksums are regenerated by +//! `tools/wai_sota_vectors.sh` (pretrained mbt2018_mean exported to integers by +//! `tools/wai_sota_int_synth_export.py`) into `WAI_SOTA_INT_DIR` (default +//! `/tmp/sota_int`). CI (`wai-sota-vectors.yml`) regenerates them from a pinned +//! public-domain photograph and runs this file with `WAI_REQUIRE_SOTA_VECTORS=1`, +//! so an absent vector fails there; a local run without them skips loudly. A +//! `WAI_SOTA_INT_DIR` naming a directory that does not exist, or one missing a +//! file a test reads, fails either way: only the default may be absent and skip. +//! +//! Quality is image-dependent: the 42.6 dB development figure for this codec was +//! measured on a photograph that is not in the repository. What is asserted here, +//! on whatever image produced the vectors, is the integer decode's cost against the +//! float codec on that same image. Nothing here compares the model with any other +//! codec. #![cfg(feature = "neural_int")] -use std::path::PathBuf; +#[path = "support/require_vectors.rs"] +mod require_vectors; +use require_vectors::{load_all, load_json, InputDir}; use wai::int_hyper_synth::{decode, fnv1a, IntModel}; -fn dir() -> PathBuf { - PathBuf::from(std::env::var("WAI_SOTA_INT_DIR").unwrap_or_else(|_| "/tmp/sota_int".into())) +const PRODUCE: &str = "tools/wai_sota_vectors.sh (then WAI_SOTA_INT_DIR=, default /tmp/sota_int)"; + +/// `WAI_SOTA_INT_DIR`, else `/tmp/sota_int`. Set to a directory that does not +/// exist, or to one missing a file the test reads, it fails the test instead of +/// letting it skip. +fn dir() -> InputDir { + require_vectors::env_dir("int_hyper_synth", "WAI_SOTA_INT_DIR", "/tmp/sota_int", PRODUCE) } -fn jload(p: &std::path::Path) -> Option { - serde_json::from_slice(&std::fs::read(p).ok()?).ok() +/// Every named file from the vector dir, or `None` after reporting what is absent. +fn inputs(test: &str, d: &InputDir, names: [&str; N]) -> Option<[serde_json::Value; N]> { + load_all(test, d, names, PRODUCE) } #[test] fn rust_integer_synthesis_reproduces_the_export_hash() { let d = dir(); - let (Some(conf), Some(model), Some(input)) = ( - jload(&d.join("int_synth_sota.json")), - jload(&d.join("int_synth_sota_model.json")), - jload(&d.join("int_synth_sota_input.json")), + let Some([conf, model, input]) = inputs( + "int_hyper_synth::rust_integer_synthesis_reproduces_the_export_hash", + &d, + ["int_synth_sota.json", "int_synth_sota_model.json", "int_synth_sota_input.json"], ) else { - eprintln!("skipping SOTA int-synth: artifacts not in {} (run tools/wai_sota_int_synth_export.py)", d.display()); return; }; @@ -44,19 +76,18 @@ fn rust_integer_synthesis_reproduces_the_export_hash() { let got = format!("{:016x}", fnv1a(&rgb)); let want = conf["rgb_hash"].as_str().unwrap(); - println!("SOTA integer synthesis: {oh}x{ow}, fnv1a {got} (psnr {:.2} dB)", conf["psnr_int"].as_f64().unwrap_or(0.0)); + println!("int_hyper integer synthesis: {oh}x{ow}, fnv1a {got} (psnr {:.2} dB)", conf["psnr_int"].as_f64().unwrap_or(0.0)); assert_eq!(got, want, "Rust integer synthesis must reproduce the export's reconstruction byte-for-byte"); } #[test] fn rust_integer_hyper_decoder_reproduces_the_export_hash() { let d = dir(); - let (Some(conf), Some(model), Some(input)) = ( - jload(&d.join("int_hs.json")), - jload(&d.join("int_hs_model.json")), - jload(&d.join("int_hs_input.json")), + let Some([conf, model, input]) = inputs( + "int_hyper_synth::rust_integer_hyper_decoder_reproduces_the_export_hash", + &d, + ["int_hs.json", "int_hs_model.json", "int_hs_input.json"], ) else { - eprintln!("skipping SOTA int h_s: artifacts not in {}", d.display()); return; }; let (c, h, w) = ( @@ -81,18 +112,18 @@ fn decodes_the_committed_tiny_conformance_vector() { // A SMALL synthetic int_hyper codec, committed under tests/vectors/int_hyper/, // drives the full decode path (factorized z → integer h_s → integer-σ entropy // → integer g_s) with NO external artifacts — so CI verifies the int_hyper - // capability without the 23 MB SOTA model. Both the JSON and the compact + // capability without the 23 MB pretrained model. Both the JSON and the compact // binary model must reproduce the committed FNV-1a, byte-for-byte. Never skips. use wai::int_hyper_synth::{decode_conditional, CondBitstream, CondEntropy}; let d = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/vectors/int_hyper"); - let conf = jload(&d.join("tiny.json")).expect("committed tiny.json"); - let ent = CondEntropy::from_value(&jload(&d.join("tiny_entropy.json")).unwrap()); - let bs = CondBitstream::from_value(&jload(&d.join("tiny_bitstream.json")).unwrap()); + let conf = load_json(&d.join("tiny.json")).expect("committed tiny.json"); + let ent = CondEntropy::from_value(&load_json(&d.join("tiny_entropy.json")).unwrap()); + let bs = CondBitstream::from_value(&load_json(&d.join("tiny_bitstream.json")).unwrap()); let want = conf["rgb_hash"].as_str().unwrap(); let (oh, ow) = (conf["out_h"].as_u64().unwrap() as usize, conf["out_w"].as_u64().unwrap() as usize); - let g_j = IntModel::from_json(&jload(&d.join("tiny_g_s.json")).unwrap()); - let h_j = IntModel::from_json(&jload(&d.join("tiny_h_s.json")).unwrap()); + let g_j = IntModel::from_json(&load_json(&d.join("tiny_g_s.json")).unwrap()); + let h_j = IntModel::from_json(&load_json(&d.join("tiny_h_s.json")).unwrap()); let (rgb, h, w) = decode_conditional(&g_j, &h_j, &ent, &bs).expect("tiny json decode"); assert_eq!((h, w), (oh, ow)); assert_eq!(format!("{:016x}", fnv1a(&rgb)), want, "tiny int_hyper (json model) must reproduce the committed hash"); @@ -113,14 +144,12 @@ fn rust_decodes_the_full_conditional_bitstream_to_the_export_image() { // identical on every machine). This is the integer-entropy contract assembled. use wai::int_hyper_synth::{decode_conditional, CondBitstream, CondEntropy}; let d = dir(); - let (Some(conf), Some(g_s), Some(h_s), Some(ent), Some(bs)) = ( - jload(&d.join("cond.json")), - jload(&d.join("int_synth_sota_model.json")), - jload(&d.join("int_hs_model.json")), - jload(&d.join("cond_entropy.json")), - jload(&d.join("cond_bitstream.json")), + let test = "int_hyper_synth::rust_decodes_the_full_conditional_bitstream_to_the_export_image"; + let Some([conf, g_s, h_s, ent, bs]) = inputs( + test, + &d, + ["cond.json", "int_synth_sota_model.json", "int_hs_model.json", "cond_entropy.json", "cond_bitstream.json"], ) else { - eprintln!("skipping conditional bitstream: artifacts not in {}", d.display()); return; }; let (g_s, h_s) = (IntModel::from_json(&g_s), IntModel::from_json(&h_s)); @@ -137,9 +166,12 @@ fn rust_decodes_the_full_conditional_bitstream_to_the_export_image() { // The COMPACT BINARY model (~½ the size) must decode bit-identically to the // JSON — same codec, smaller download (what makes the in-browser demo practical). - if let (Ok(g_bin), Ok(h_bin)) = - (std::fs::read(d.join("int_synth_sota_model.bin")), std::fs::read(d.join("int_hs_model.bin"))) - { + // The export writes both, so an absent binary is an incomplete vector dir. + let (gb, hb) = (d.join("int_synth_sota_model.bin"), d.join("int_hs_model.bin")); + if !gb.exists() || !hb.exists() { + d.absent(&format!("{test} (binary model)"), &format!("int_synth_sota_model.bin / int_hs_model.bin not in {}", d.display()), PRODUCE); + } else { + let (g_bin, h_bin) = (std::fs::read(&gb).unwrap(), std::fs::read(&hb).unwrap()); let g_s2 = IntModel::from_bytes(&g_bin).expect("g_s binary"); let h_s2 = IntModel::from_bytes(&h_bin).expect("h_s binary"); let (rgb2, _, _) = decode_conditional(&g_s2, &h_s2, &ent, &bs).expect("binary conditional decode"); @@ -148,3 +180,54 @@ fn rust_decodes_the_full_conditional_bitstream_to_the_export_image() { println!("compact binary model ({} + {} bytes) decodes byte-identically", g_bin.len(), h_bin.len()); } } + +/// The largest quality cost the integer decode may have against the float codec on +/// the same image. Measured costs: 0.19 dB for the full conditional decode on the +/// pinned photograph (Apple Silicon Mac); 0.62 dB for the integer synthesis alone +/// in development, on a photograph that is not in the repository. A larger gap +/// means the quantization has regressed, even when every byte-exactness check +/// above still passes (they compare the integer decode only with itself). +const MAX_INTEGER_COST_DB: f64 = 1.0; +/// A sanity floor for the float codec on the source image: below it the pretrained +/// checkpoint is not coding natural content at all, and a matching integer decode +/// of it proves nothing. A sanity floor only, not a quality claim. +const MIN_FLOAT_PSNR_DB: f64 = 30.0; + +#[test] +fn integer_decode_keeps_the_float_codec_quality() { + // Measured by this decoder on its own output, against the exact source pixels + // the export tool coded — not the PSNR the tool reports about itself. + use wai::int_hyper_synth::{decode_conditional, CondBitstream, CondEntropy}; + let d = dir(); + let test = "int_hyper_synth::integer_decode_keeps_the_float_codec_quality"; + let Some([conf, g_s, h_s, ent, bs]) = inputs( + test, + &d, + ["cond.json", "int_synth_sota_model.json", "int_hs_model.json", "cond_entropy.json", "cond_bitstream.json"], + ) else { + return; + }; + let src_path = d.join("source.rgb"); + let Some(psnr_float) = conf.get("psnr_float").and_then(|v| v.as_f64()) else { + d.absent(test, "cond.json predates psnr_float (regenerate the vectors)", PRODUCE); + return; + }; + if !src_path.exists() { + d.absent(test, &format!("source.rgb not in {}", d.display()), PRODUCE); + return; + } + let source = std::fs::read(&src_path).unwrap(); + + let (g_s, h_s) = (IntModel::from_json(&g_s), IntModel::from_json(&h_s)); + let (rgb, h, w) = decode_conditional(&g_s, &h_s, &CondEntropy::from_value(&ent), &CondBitstream::from_value(&bs)) + .expect("conditional decode"); + assert_eq!(source.len(), rgb.len(), "source.rgb is not the {h}x{w} RGB image the bitstream decodes to"); + + let sse: f64 = rgb.iter().zip(&source).map(|(a, b)| (f64::from(*a) - f64::from(*b)).powi(2)).sum(); + let mse = sse / rgb.len() as f64; + let psnr = 10.0 * (255.0f64 * 255.0 / mse).log10(); + let cost = psnr_float - psnr; + println!("integer decode {psnr:.3} dB vs float codec {psnr_float:.3} dB on the same {h}x{w} image (cost {cost:.3} dB)"); + assert!(psnr_float >= MIN_FLOAT_PSNR_DB, "float codec {psnr_float:.2} dB < {MIN_FLOAT_PSNR_DB} dB: the checkpoint or source is wrong"); + assert!(cost <= MAX_INTEGER_COST_DB, "integer decode costs {cost:.2} dB against the float codec (limit {MAX_INTEGER_COST_DB} dB)"); +} diff --git a/wai/wai-rs/tests/int_video_sota_keyframe.rs b/wai/wai-rs/tests/int_video_sota_keyframe.rs index 4cf89bec3..c0a681e89 100644 --- a/wai/wai-rs/tests/int_video_sota_keyframe.rs +++ b/wai/wai-rs/tests/int_video_sota_keyframe.rs @@ -1,12 +1,19 @@ -//! `wai.video.int_hyper` at SOTA scale — a real 256×256 clip whose I-frame is the -//! full SOTA `int_hyper` learned codec (42.6 dB photo) decoded byte-exact, panned -//! into P-frames by integer motion compensation. Reads the SOTA int_hyper vector + -//! the panned-clip bitstream from `WAI_SOTA_INT_DIR` (default `/tmp/sota_int`); -//! skips if absent (the SOTA model is too large to commit, like the int_hyper SOTA -//! test). The committed 16×16 `int_video_keyframe` test is the never-skip companion. +//! `wai.video.int_hyper` on the full pretrained model — a real 256×256 clip whose +//! I-frame is the byte-exact `int_hyper` decode of the pretrained mbt2018_mean +//! (exported to integers), panned into P-frames by integer motion compensation. +//! Reads the int_hyper vector + the panned-clip bitstream from `WAI_SOTA_INT_DIR` +//! (default `/tmp/sota_int`), regenerated by `tools/wai_sota_vectors.sh` (the +//! model is too large to commit). CI (`wai-sota-vectors.yml`) regenerates them and +//! runs this with `WAI_REQUIRE_SOTA_VECTORS=1`, so an absent vector fails there; +//! locally it skips loudly. A `WAI_SOTA_INT_DIR` naming a directory that does not +//! exist, or one missing a file this test reads, fails either way. The committed +//! 16×16 `int_video_keyframe` test is the never-skip companion. #![cfg(all(feature = "neural_int", feature = "video_int"))] -use std::path::PathBuf; +#[path = "support/require_vectors.rs"] +mod require_vectors; + +use require_vectors::{env_dir, load_all}; use wai::int_hyper_synth::{decode_conditional, CondBitstream, CondEntropy, IntModel}; use wai::int_video::{decode_pframes, fnv1a}; @@ -21,19 +28,22 @@ fn hex16(s: &str) -> [u8; 8] { #[test] fn sota_int_hyper_keyframe_clip_decodes_byte_for_byte() { - let d = PathBuf::from(std::env::var("WAI_SOTA_INT_DIR").unwrap_or_else(|_| "/tmp/sota_int".into())); - let jl = |n: &str| -> Option { - serde_json::from_slice(&std::fs::read(d.join(n)).ok()?).ok() - }; - let (Some(g), Some(hh), Some(e), Some(b), Some(vb), Some(conf)) = ( - jl("int_synth_sota_model.json"), jl("int_hs_model.json"), jl("cond_entropy.json"), - jl("cond_bitstream.json"), jl("video_sota_bitstream.json"), jl("video_sota.json"), + const TEST: &str = "int_video_sota_keyframe::sota_int_hyper_keyframe_clip_decodes_byte_for_byte"; + const PRODUCE: &str = "tools/wai_sota_vectors.sh (then WAI_SOTA_INT_DIR=, default /tmp/sota_int)"; + let d = env_dir(TEST, "WAI_SOTA_INT_DIR", "/tmp/sota_int", PRODUCE); + let Some([g, hh, e, b, vb, conf]) = load_all( + TEST, + &d, + [ + "int_synth_sota_model.json", "int_hs_model.json", "cond_entropy.json", + "cond_bitstream.json", "video_sota_bitstream.json", "video_sota.json", + ], + PRODUCE, ) else { - eprintln!("skipping SOTA int_hyper-keyframe video: artifacts not in {} (run the export, the int_hyper_decode_frame example, then tools/wai_int_video_sota_keyframe_encode.py)", d.display()); return; }; - // frame 0: the SOTA int_hyper learned I-frame (byte-exact 256×256 photo). + // frame 0: the int_hyper learned I-frame (byte-exact 256×256 photo). let g_s = IntModel::from_json(&g); let h_s = IntModel::from_json(&hh); let ent = CondEntropy::from_value(&e); @@ -44,7 +54,7 @@ fn sota_int_hyper_keyframe_clip_decodes_byte_for_byte() { let clip = decode_pframes(&frame0, h, w, &vb).expect("p-frames"); let want = conf["frame_hashes"].as_array().unwrap(); assert_eq!(clip.frames.len(), want.len()); - assert_eq!(format!("{:016x}", fnv1a(&frame0)), want[0].as_str().unwrap(), "I-frame = SOTA int_hyper decode"); + assert_eq!(format!("{:016x}", fnv1a(&frame0)), want[0].as_str().unwrap(), "I-frame = int_hyper decode"); let mut all = Vec::new(); for (i, frame) in clip.frames.iter().enumerate() { @@ -53,6 +63,6 @@ fn sota_int_hyper_keyframe_clip_decodes_byte_for_byte() { all.extend_from_slice(&hex16(&got)); } assert_eq!(format!("{:016x}", fnv1a(&all)), conf["clip_hash"].as_str().unwrap(), "clip hash"); - println!("SOTA int_hyper-keyframe video: {} frames {h}x{w}, I-frame={} (learned), clip {}", + println!("int_hyper-keyframe video (pretrained model): {} frames {h}x{w}, I-frame={} (learned), clip {}", clip.frames.len(), &want[0].as_str().unwrap()[..8], conf["clip_hash"].as_str().unwrap()); } diff --git a/wai/wai-rs/tests/jpegai_e2e_full.rs b/wai/wai-rs/tests/jpegai_e2e_full.rs index 50584a30c..e01ae755e 100644 --- a/wai/wai-rs/tests/jpegai_e2e_full.rs +++ b/wai/wai-rs/tests/jpegai_e2e_full.rs @@ -25,8 +25,21 @@ //! WAI_JPEGAI_WEIGHTS_DIR = dir of {hs,hyp}_{y,uv}.json, mcm{0..3}_y.json, //! synth_{y,uv}.json, icci.json //! WAI_JPEGAI_NORM = norm.json (see wai_jpegai_build_e2e_inputs.py) +//! +//! These inputs are not in the repository. They were extracted by +//! `tools/wai_jpegai_build_e2e_inputs.py` from JPEG-AI reference-software assets +//! (ISO/IEC 6048-1; kept locally in `wai/jpegai-vm-assets`, not committed), and the +//! extraction is published under WAI's versioned models root (`jpegai/v1/`). +//! `tools/wai_jpegai_fetch_e2e_inputs.sh` fetches it and checks every file against +//! the SHA-256 manifest `tools/jpegai-e2e-v1.sha256`; CI (`wai-jpegai-e2e.yml`) does +//! that and runs this file with `WAI_REQUIRE_SOTA_VECTORS=1`. Without the three +//! variables it skips loudly; with the switch it fails instead; a variable naming a +//! missing path, or a weights directory missing a file, fails either way. #![cfg(feature = "neural_int")] +#[path = "support/require_vectors.rs"] +mod require_vectors; + use wai::jpegai_decode::{decode, DecoderWeights, NormTables, PlaneWeights}; use wai::jpegai_icci::{Conv, IcciNet, IcciPath, ResBlock}; use wai::jpegai_synthesis::{IntNet, Weights}; @@ -80,12 +93,14 @@ fn path_of(v: &serde_json::Value) -> IcciPath { #[test] fn codestream_to_rgb_matches_reference_png() { - let (Ok(cs), Ok(dir), Ok(nf)) = (std::env::var("WAI_JPEGAI_CODESTREAM"), - std::env::var("WAI_JPEGAI_WEIGHTS_DIR"), std::env::var("WAI_JPEGAI_NORM")) else { - eprintln!("skip: set WAI_JPEGAI_CODESTREAM + WAI_JPEGAI_WEIGHTS_DIR + WAI_JPEGAI_NORM"); + let Some([cs, dir, nf]) = require_vectors::env_paths( + "jpegai_e2e_full::codestream_to_rgb_matches_reference_png", + ["WAI_JPEGAI_CODESTREAM", "WAI_JPEGAI_WEIGHTS_DIR", "WAI_JPEGAI_NORM"], + "tools/wai_jpegai_fetch_e2e_inputs.sh (pinned by tools/jpegai-e2e-v1.sha256)", + ) else { return; }; - let Ok(data) = std::fs::read(&cs) else { eprintln!("skip: no codestream"); return }; + let data = std::fs::read(&cs).unwrap_or_else(|e| panic!("read {cs}: {e}")); let nv = jload(std::path::Path::new(&nf)); let iw = jload(&std::path::Path::new(&dir).join("icci.json")); diff --git a/wai/wai-rs/tests/jpegai_live.rs b/wai/wai-rs/tests/jpegai_live.rs index ce2c4ad82..34055b727 100644 --- a/wai/wai-rs/tests/jpegai_live.rs +++ b/wai/wai-rs/tests/jpegai_live.rs @@ -4,33 +4,62 @@ //! rvs); the model bundle supplies only checkpoint constants. Compared against //! the reference decoder's PNG — the same bar as the staged `jpegai_e2e` test, //! but through the dispatchable library entry point. +//! +//! This is the whole-decode test behind the dispatchable `wai.image.jpegai` ONNX +//! path, so a missing input follows the headline-claim rule (support/ +//! require_vectors.rs): without its inputs it skips loudly, and with +//! `WAI_REQUIRE_SOTA_VECTORS=1` it fails. Its inputs — the float model bundle, the +//! traced nets under `model16/`, the extracted model constants and the reference +//! PNG — exist only in the local JPEG-AI asset directory +//! (`wai/jpegai-vm-assets`, exported from the ISO/IEC 6048-1 reference software; +//! not in the repository), and the test also needs the `neural` (ONNX Runtime) and +//! `codecs` features. No CI job runs it; it is verified only on a machine holding +//! those assets. `WAI_JPEGAI_ASSETS` points it at the directory from elsewhere; a +//! directory it names must exist and hold every input, or the test fails with or +//! without the switch. Unset or empty, the default below is used, and only that +//! default may be absent and skip. #![cfg(all(feature = "neural", feature = "codecs"))] -use std::path::PathBuf; +#[path = "support/require_vectors.rs"] +mod require_vectors; +use require_vectors::InputDir; use wai::neural::jpegai::{decode, JpegAiModel}; -fn assets() -> PathBuf { - PathBuf::from("../jpegai-vm-assets") +const TEST: &str = "jpegai_live::jpegai_live_decode_matches_reference"; +const PRODUCE: &str = "the local JPEG-AI asset directory wai/jpegai-vm-assets (exported from the \ +ISO/IEC 6048-1 reference software; not in the repository), or WAI_JPEGAI_ASSETS="; + +/// `WAI_JPEGAI_ASSETS`, else `../jpegai-vm-assets` beside the crate. A variable that +/// names a directory that does not exist, or one missing an input, is a +/// misconfigured run and fails. +fn assets() -> InputDir { + require_vectors::env_dir(TEST, "WAI_JPEGAI_ASSETS", "../jpegai-vm-assets", PRODUCE) } #[test] fn jpegai_live_decode_matches_reference() { let a = assets(); // Needs the model bundle (incl. the extracted model-constant files) + the - // 256×256 traced nets + the reference PNG. Skipped if archived out-of-band. - for need in [ + // 256×256 traced nets + the reference PNG. + let missing: Vec<&str> = [ "e2e/test256.bin", "e2e/test256_dec.png", "jpegai_factorized_cdfs.json", "jpegai_scaler_vec.json", "jpegai_dequant_consts.json", "model16/CCS_SGMM", - ] { - if !a.join(need).exists() { - eprintln!("skipping jpegai live decode: {need} not present"); - return; - } + ] + .into_iter() + .filter(|need| !a.join(need).exists()) + .collect(); + if !missing.is_empty() { + a.absent( + TEST, + &format!("{} not in {}", missing.join(", "), a.display()), + PRODUCE, + ); + return; } let model = JpegAiModel::load(&a).expect("load jpeg-ai model bundle"); diff --git a/wai/wai-rs/tests/jpegai_reconstruction_binding.rs b/wai/wai-rs/tests/jpegai_reconstruction_binding.rs index 2c16f9805..2f59bcbf7 100644 --- a/wai/wai-rs/tests/jpegai_reconstruction_binding.rs +++ b/wai/wai-rs/tests/jpegai_reconstruction_binding.rs @@ -6,14 +6,36 @@ //! It exercises the property the binding rests on: the exact decoder is deterministic, //! so the binding taken over one decode verifies a second decode, and a single flipped //! pixel breaks it. Both decodes run in this process; that the same pixels come out on -//! another machine rests on the conditions in `extensions/determinism-tiers.md` §5–§6, -//! not on this test. (`wai.image.jpegai` is registered at entropy-consistency; the -//! pinned reconstruction is a property of the exact decoder.) +//! another machine rests on the conditions and measurements in +//! `extensions/determinism-tiers.md` §5–§6. This test compares with other machines +//! only through `WAI_JPEGAI_EXPECT_SHA256` (below). (`wai.image.jpegai` is registered +//! at entropy-consistency; the pinned reconstruction is a property of the exact +//! decoder.) //! //! env: WAI_JPEGAI_CODESTREAM, WAI_JPEGAI_WEIGHTS_DIR, WAI_JPEGAI_NORM -//! (the bundle from `tools/wai_jpegai_build_e2e_inputs.py`; see jpegai_e2e_full). +//! (the bundle from `tools/wai_jpegai_fetch_e2e_inputs.sh`; see jpegai_e2e_full). +//! WAI_JPEGAI_EXPECT_SHA256 (optional) = the SHA-256 the decoded pixels must +//! have. Set, it also compares this machine's decode with a value decoded on +//! other machines, which the two in-process decodes cannot do. That value rests +//! on a measurement, not on the registry: the codec registry classes +//! `wai.image.jpegai` as `NeuralEntropyExact`, a conservative class that +//! guarantees only the entropy decode (the decoded symbols and their entropy +//! parameters) to be byte-identical across machines, not the reconstruction. +//! The exact decoder's pinned output for the `jpegai/v1` inputs was measured +//! identical on four platforms: macOS arm64, wasm32-wasip2, Linux arm64, and +//! Linux x86_64 under user-mode emulation (`extensions/determinism-tiers.md` +//! §6). `wai-jpegai-e2e.yml` pins that value, so a mismatch there is a departure +//! from that measurement. +//! +//! The inputs are not in the repository; see jpegai_e2e_full for where they come +//! from and how CI fetches them. Without the three variables this test skips +//! loudly; with `WAI_REQUIRE_SOTA_VECTORS=1` it fails instead; a variable naming a +//! missing path, or a weights directory missing a file, fails either way. #![cfg(all(feature = "neural_int", feature = "provenance"))] +#[path = "support/require_vectors.rs"] +mod require_vectors; + use wai::jpegai_decode::{decode, DecoderWeights, NormTables, PlaneWeights}; use wai::jpegai_icci::IcciNet; use wai::jpegai_synthesis::{IntNet, Weights}; @@ -25,15 +47,14 @@ fn jload(p: &std::path::Path) -> serde_json::Value { #[test] fn real_decode_to_reconstruction_binding() { - let (Ok(cs), Ok(dir), Ok(nf)) = ( - std::env::var("WAI_JPEGAI_CODESTREAM"), - std::env::var("WAI_JPEGAI_WEIGHTS_DIR"), - std::env::var("WAI_JPEGAI_NORM"), + let Some([cs, dir, nf]) = require_vectors::env_paths( + "jpegai_reconstruction_binding::real_decode_to_reconstruction_binding", + ["WAI_JPEGAI_CODESTREAM", "WAI_JPEGAI_WEIGHTS_DIR", "WAI_JPEGAI_NORM"], + "tools/wai_jpegai_fetch_e2e_inputs.sh (pinned by tools/jpegai-e2e-v1.sha256)", ) else { - eprintln!("skip: set WAI_JPEGAI_CODESTREAM + WAI_JPEGAI_WEIGHTS_DIR + WAI_JPEGAI_NORM"); return; }; - let Ok(data) = std::fs::read(&cs) else { eprintln!("skip: no codestream"); return }; + let data = std::fs::read(&cs).unwrap_or_else(|e| panic!("read {cs}: {e}")); // --- load the model bundle (same shape as jpegai_e2e_full) --- let dpath = std::path::Path::new(&dir); @@ -102,4 +123,15 @@ fn real_decode_to_reconstruction_binding() { eprintln!(" assertion = {}", ReconstructionBinding::C2PA_LABEL); eprintln!(" sha256 = {}", binding.hash_hex()); eprintln!(" json = {}", binding.to_json()); + + // Against other machines: the pixels this machine decoded must hash to the value + // measured from other machines' decodes of the same inputs (see the header). + if let Ok(want) = std::env::var("WAI_JPEGAI_EXPECT_SHA256") { + assert_eq!( + binding.hash_hex(), + want.trim().to_ascii_lowercase(), + "the decoded pixels differ from the pinned decode of these inputs" + ); + eprintln!(" matches the pinned WAI_JPEGAI_EXPECT_SHA256"); + } } diff --git a/wai/wai-rs/tests/require_vectors_rule.rs b/wai/wai-rs/tests/require_vectors_rule.rs new file mode 100644 index 000000000..1631ccc5f --- /dev/null +++ b/wai/wai-rs/tests/require_vectors_rule.rs @@ -0,0 +1,300 @@ +//! The missing-input rule of `support/require_vectors.rs`, checked in the +//! environment it is about. +//! +//! The vector tests' CI jobs always set `WAI_REQUIRE_SOTA_VECTORS=1`, so those runs +//! never exercise what the helpers do WITHOUT the switch: that a variable naming a +//! test's inputs still makes them required, and that a test skips only when no +//! variable names what is missing (an unset variable, or an absent or incomplete +//! default location). These tests check that rule directly. Each case starts this +//! test binary again as a child process running the ignored `probe` test, with the +//! switch and the input variables set exactly as the case needs (and every other +//! one removed, so the parent's environment cannot leak in), then checks whether +//! the child passed or failed and what it printed. +//! +//! Their inputs are scratch directories and small files that each case creates +//! under the system temporary directory, so these tests never skip. + +#[path = "support/require_vectors.rs"] +mod require_vectors; + +use std::path::PathBuf; +use std::process::Command; + +/// Which helper the probe runs: `dir` (`env_dir` + `load_all`, as the int_hyper +/// tests use them) or `paths` (`env_paths`, as the JPEG-AI tests use it). +const MODE: &str = "WAI_RV_PROBE"; +/// The default directory the probe passes to `env_dir`. +const DEFAULT: &str = "WAI_RV_PROBE_DEFAULT"; +/// The directory variable the probe reads, as `int_hyper_synth` does. +const DIR_VAR: &str = "WAI_SOTA_INT_DIR"; +/// The two path variables the probe reads, as the JPEG-AI tests read three. +const PATH_VAR: &str = "WAI_JPEGAI_CODESTREAM"; +const PATH_VAR2: &str = "WAI_JPEGAI_NORM"; +/// The one input file the `dir` probe needs. +const INPUT: &str = "cond.json"; + +/// Runs one helper the way a vector test does and prints what it returned. It is +/// only meaningful in a child started by [`run`], with the environment [`run`] +/// gives it; run any other way, it returns without doing anything. +#[test] +#[ignore = "a probe that the other tests in this file run in a child process"] +fn probe() { + let Ok(mode) = std::env::var(MODE) else { + return; + }; + let loaded = match mode.as_str() { + "dir" => { + let default = std::env::var(DEFAULT).expect("the default directory"); + let d = require_vectors::env_dir("probe", DIR_VAR, &default, "the probe"); + require_vectors::load_all("probe", &d, [INPUT], "the probe").is_some() + } + "paths" => require_vectors::env_paths("probe", [PATH_VAR, PATH_VAR2], "the probe").is_some(), + other => panic!("unknown probe mode {other}"), + }; + println!( + "probe: {}", + if loaded { + "inputs loaded" + } else { + "returned without inputs" + } + ); +} + +/// What a probe run did. +struct Outcome { + passed: bool, + /// The child's stdout followed by its stderr. + out: String, +} + +impl Outcome { + /// The probe passed because it found nothing to check: the loud skip. + fn assert_skipped_loudly(&self) { + assert!(self.passed, "expected a skip, got a failure:\n{}", self.out); + assert!( + self.out.contains("probe: returned without inputs"), + "{}", + self.out + ); + // The banner CI's backstop looks for, so a skip is never read as a pass. + assert!( + self.out.contains("SKIPPED") && self.out.contains("NOT VERIFIED"), + "{}", + self.out + ); + } + + fn assert_failed_with(&self, why: &str) { + assert!( + !self.passed, + "expected a failure naming {why:?}, got a pass:\n{}", + self.out + ); + assert!( + self.out.contains(why), + "the failure does not say {why:?}:\n{}", + self.out + ); + } +} + +/// Runs the probe in a child process in `mode`, with `env` set and the switch and +/// every input variable otherwise removed. +fn run(mode: &str, env: &[(&str, &str)]) -> Outcome { + let exe = std::env::current_exe().expect("this test binary"); + let mut cmd = Command::new(exe); + cmd.args([ + "probe", + "--exact", + "--ignored", + "--nocapture", + "--test-threads=1", + ]); + for v in [require_vectors::SWITCH, MODE, DEFAULT, DIR_VAR, PATH_VAR, PATH_VAR2] { + cmd.env_remove(v); + } + cmd.env(MODE, mode); + for (k, v) in env { + cmd.env(k, v); + } + let o = cmd.output().expect("start the probe"); + let out = format!( + "{}{}", + String::from_utf8_lossy(&o.stdout), + String::from_utf8_lossy(&o.stderr) + ); + // A filter that matched nothing would also exit 0. + assert!( + out.contains("running 1 test"), + "the probe did not run:\n{out}" + ); + Outcome { + passed: o.status.success(), + out, + } +} + +/// A scratch directory unique to one test case, removed when dropped. +struct Scratch(PathBuf); + +impl Scratch { + fn new(case: &str) -> Scratch { + let p = std::env::temp_dir().join(format!( + "wai-require-vectors-rule-{}-{case}", + std::process::id() + )); + let _ = std::fs::remove_dir_all(&p); + std::fs::create_dir_all(&p).expect("create the scratch directory"); + Scratch(p) + } + + /// A path inside the scratch directory, not created. + fn at(&self, name: &str) -> String { + self.0 + .join(name) + .to_str() + .expect("a UTF-8 temp path") + .to_owned() + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +// ---- without the switch ---- + +#[test] +fn without_the_switch_a_named_directory_that_does_not_exist_fails() { + let s = Scratch::new("dir-missing"); + let (named, default) = (s.at("not-there"), s.at("default")); + run("dir", &[(DIR_VAR, &named), (DEFAULT, &default)]).assert_failed_with("is not a directory"); +} + +#[test] +fn without_the_switch_a_named_path_that_is_a_file_fails_as_a_directory() { + let s = Scratch::new("dir-is-file"); + let (named, default) = (s.at("a-file"), s.at("default")); + std::fs::write(&named, b"{}").unwrap(); + run("dir", &[(DIR_VAR, &named), (DEFAULT, &default)]).assert_failed_with("is not a directory"); +} + +#[test] +fn without_the_switch_a_named_empty_directory_fails() { + // The case this file was written for: the directory exists, so only the + // missing file can fail the run. + let s = Scratch::new("dir-empty"); + let (named, default) = (s.at("vectors"), s.at("default")); + std::fs::create_dir(&named).unwrap(); + run("dir", &[(DIR_VAR, &named), (DEFAULT, &default)]) + .assert_failed_with(&format!("{DIR_VAR} names this directory")); +} + +#[test] +fn without_the_switch_an_unset_or_empty_directory_variable_skips_on_an_absent_default() { + let s = Scratch::new("dir-default"); + let default = s.at("default"); + run("dir", &[(DEFAULT, &default)]).assert_skipped_loudly(); + run("dir", &[(DIR_VAR, ""), (DEFAULT, &default)]).assert_skipped_loudly(); +} + +#[test] +fn without_the_switch_an_incomplete_default_directory_skips() { + let s = Scratch::new("dir-default-empty"); + let default = s.at("default"); + std::fs::create_dir(&default).unwrap(); + run("dir", &[(DEFAULT, &default)]).assert_skipped_loudly(); +} + +#[test] +fn a_named_directory_holding_the_input_loads_it() { + // The positive control: without it, a probe that always failed would make the + // failure cases above pass for the wrong reason. + let s = Scratch::new("dir-full"); + let (named, default) = (s.at("vectors"), s.at("default")); + std::fs::create_dir(&named).unwrap(); + std::fs::write(PathBuf::from(&named).join(INPUT), b"{}").unwrap(); + let o = run("dir", &[(DIR_VAR, &named), (DEFAULT, &default)]); + assert!( + o.passed && o.out.contains("probe: inputs loaded"), + "{}", + o.out + ); + assert!(!o.out.contains("NOT VERIFIED"), "{}", o.out); +} + +#[test] +fn without_the_switch_a_path_variable_naming_nothing_fails() { + let s = Scratch::new("path-missing"); + let other = s.at("norm.json"); + std::fs::write(&other, b"{}").unwrap(); + run("paths", &[(PATH_VAR, &s.at("not-there")), (PATH_VAR2, &other)]) + .assert_failed_with("does not exist"); + // env_paths has no default: an empty value names no existing path. + run("paths", &[(PATH_VAR, ""), (PATH_VAR2, &other)]).assert_failed_with("does not exist"); +} + +#[test] +fn without_the_switch_a_path_variable_naming_nothing_fails_when_another_is_unset() { + // The other variable being unset would end the test as a skip; the one that + // names a missing path must fail it first. + let s = Scratch::new("path-missing-partial"); + run("paths", &[(PATH_VAR, &s.at("not-there"))]).assert_failed_with("does not exist"); + run("paths", &[(PATH_VAR2, &s.at("not-there"))]).assert_failed_with("does not exist"); +} + +#[test] +fn without_the_switch_an_unset_path_variable_skips() { + run("paths", &[]).assert_skipped_loudly(); + // One set to a file that exists, the other unset: nothing named is missing. + let s = Scratch::new("path-partial"); + let file = s.at("codestream.bin"); + std::fs::write(&file, b"x").unwrap(); + run("paths", &[(PATH_VAR, &file)]).assert_skipped_loudly(); +} + +#[test] +fn path_variables_naming_files_that_exist_load_them() { + // The positive control for `env_paths`, as the directory one is for `env_dir`. + let s = Scratch::new("path-full"); + let (a, b) = (s.at("codestream.bin"), s.at("norm.json")); + std::fs::write(&a, b"x").unwrap(); + std::fs::write(&b, b"{}").unwrap(); + let o = run("paths", &[(PATH_VAR, &a), (PATH_VAR2, &b)]); + assert!( + o.passed && o.out.contains("probe: inputs loaded"), + "{}", + o.out + ); + assert!(!o.out.contains("NOT VERIFIED"), "{}", o.out); +} + +#[test] +fn switch_values_that_leave_it_off_behave_as_unset() { + let s = Scratch::new("switch-off"); + let default = s.at("default"); + for off in ["", "0", "false", "no", "FALSE", " no "] { + run( + "dir", + &[(require_vectors::SWITCH, off), (DEFAULT, &default)], + ) + .assert_skipped_loudly(); + } +} + +// ---- with the switch ---- + +#[test] +fn with_the_switch_an_absent_default_fails() { + let s = Scratch::new("switch-on"); + let default = s.at("default"); + for on in ["1", "true", "yes"] { + run("dir", &[(require_vectors::SWITCH, on), (DEFAULT, &default)]) + .assert_failed_with("a missing input is a failure, not a skip"); + run("paths", &[(require_vectors::SWITCH, on)]) + .assert_failed_with("a missing input is a failure, not a skip"); + } +} diff --git a/wai/wai-rs/tests/support/require_vectors.rs b/wai/wai-rs/tests/support/require_vectors.rs new file mode 100644 index 000000000..b5b1f4bd2 --- /dev/null +++ b/wai/wai-rs/tests/support/require_vectors.rs @@ -0,0 +1,204 @@ +//! What a test does when the out-of-tree inputs behind a headline decode claim are +//! absent. +//! +//! Some decode claims are tested against inputs too large, or not redistributable, +//! to commit: the `int_hyper` vectors (the pretrained mbt2018_mean, exported to +//! integers) and the JPEG-AI whole-decoder bundle. A test that `return`s when its +//! inputs are missing is reported as *passed*, so a run with nothing to check looks +//! exactly like a run that checked everything. These helpers remove that ambiguity: +//! +//! - `WAI_REQUIRE_SOTA_VECTORS` unset (or `0`/`false`/`no`): the test still skips, +//! but writes a banner to the process's real stderr — past libtest's output +//! capture, so it shows in an ordinary `cargo test` run, not only under +//! `--nocapture`. +//! - `WAI_REQUIRE_SOTA_VECTORS=1`: a missing input is a test failure naming the +//! tool that produces it. A run that claims to verify these decodes sets it. +//! +//! Either way, with or without the switch, a variable that names a test's inputs +//! makes them required: pointing a test at inputs and having it pass because they +//! are not there is the failure this module exists to prevent. So the test fails +//! when a variable read through [`env_paths`] names a path that does not exist, +//! and when a directory variable read through [`env_dir`] (`WAI_SOTA_INT_DIR`, +//! `WAI_JPEGAI_ASSETS`) names something that is not a directory, or a directory +//! missing a file the test reads ([`InputDir::absent`]). The JPEG-AI tests that +//! take a directory through [`env_paths`] fail the same way on a missing file, +//! because their loaders do not skip. So without the switch a test skips only when +//! no variable names what is missing: when a variable read through [`env_paths`] +//! is unset (the ones that are set must still name existing paths), or when the +//! variable of [`env_dir`] is unset or empty and the default location it falls +//! back to is absent or incomplete. [`env_paths`] has no default, so an empty value +//! there names no existing path and fails. `tests/require_vectors_rule.rs` checks +//! this rule by running these helpers in child processes, with the switch unset and +//! set. +//! +//! Where each covered input comes from, and who verifies it: +//! +//! | test | inputs | produced by | verified in CI | +//! |---|---|---|---| +//! | `int_hyper_synth` (pretrained-model cases) | `WAI_SOTA_INT_DIR` | `tools/wai_sota_vectors.sh` | yes — `wai-sota-vectors.yml` regenerates and requires them | +//! | `int_video_sota_keyframe` | `WAI_SOTA_INT_DIR` | `tools/wai_sota_vectors.sh` | yes — same job | +//! | `jpegai_e2e_full` | `WAI_JPEGAI_{CODESTREAM,WEIGHTS_DIR,NORM}` | `tools/wai_jpegai_fetch_e2e_inputs.sh` (the models root's `jpegai/v1/`, pinned by `tools/jpegai-e2e-v1.sha256`) | yes — `wai-jpegai-e2e.yml` fetches, checks and requires them | +//! | `jpegai_reconstruction_binding` | same, plus optional `WAI_JPEGAI_EXPECT_SHA256` | same | yes — same job, with the decoded-pixel hash pinned | +//! | `jpegai_live` | `WAI_JPEGAI_ASSETS` (default `../jpegai-vm-assets`) | the local JPEG-AI asset directory (not in the repository) | no — local only; also needs the `neural` + `codecs` features | +//! +//! Tests that still skip silently on absent inputs and are NOT covered by this +//! switch — an open gap, listed here so it stays visible: +//! `jpegai_*_calibration.rs`, `jpegai_e2e.rs`, `jpegai_e2e_{chain,chroma,dequant, +//! frontend,hyperlatent,synthesis}.rs`, `jpegai_mans_golden.rs`, +//! `jpegai_postfilter_chain.rs` (per-stage and staged JPEG-AI fixtures traced from +//! the reference software), `neural_native_decode.rs` (float ONNX decoders, models +//! and `/tmp` fixtures built per machine), and the `WAI_JPEGAI_*` unit tests in +//! `src/jpegai_synthesis.rs` and `src/neural/mod.rs`. + +use std::io::Write; +use std::path::{Path, PathBuf}; + +pub const SWITCH: &str = "WAI_REQUIRE_SOTA_VECTORS"; + +/// Whether this run has declared that the out-of-tree inputs must be present. +pub fn required() -> bool { + match std::env::var(SWITCH) { + Ok(v) => !matches!(v.trim().to_ascii_lowercase().as_str(), "" | "0" | "false" | "no"), + Err(_) => false, + } +} + +/// Call at the point a test would otherwise return early for lack of inputs. +/// Panics when [`required`]; otherwise announces the skip where it cannot be +/// missed. The caller returns (or skips the sub-check) after this. +pub fn absent(test: &str, missing: &str, produce: &str) { + let msg = format!("{test}: NOT VERIFIED — {missing}\n produce with: {produce}"); + if required() { + panic!("{msg}\n {SWITCH} is set, so a missing input is a failure, not a skip"); + } + // Deliberately not eprintln!: libtest captures the print macros of a passing + // test, which is how skips became invisible in the first place. + // wai-sota-vectors.yml and wai-jpegai-e2e.yml fail on "SKIPPED" or + // "NOT VERIFIED" in their test log, so keep both words in this banner. + let _ = writeln!( + std::io::stderr(), + "\n!!! SKIPPED — {msg}\n set {SWITCH}=1 to make a missing input fail\n" + ); +} + +/// Read a JSON input. `None` only when the file does not exist; a file that exists +/// but cannot be read or parsed is a broken input and fails the test either way. +#[allow(dead_code)] +pub fn load_json(p: &Path) -> Option { + if !p.exists() { + return None; + } + let bytes = std::fs::read(p).unwrap_or_else(|e| panic!("read {}: {e}", p.display())); + Some(serde_json::from_slice(&bytes).unwrap_or_else(|e| panic!("parse {}: {e}", p.display()))) +} + +/// Every named JSON file from `dir`, or `None` after [`InputDir::absent`] names +/// the ones missing (which fails instead when a variable named `dir`). +#[allow(dead_code)] +pub fn load_all( + test: &str, + dir: &InputDir, + names: [&str; N], + produce: &str, +) -> Option<[serde_json::Value; N]> { + let loaded = names.map(|n| load_json(&dir.join(n))); + let missing: Vec<&str> = + names.iter().zip(&loaded).filter(|(_, v)| v.is_none()).map(|(n, _)| *n).collect(); + if !missing.is_empty() { + dir.absent(test, &format!("{} not in {}", missing.join(", "), dir.display()), produce); + return None; + } + Some(loaded.map(Option::unwrap)) +} + +/// The environment variables a test reads its input paths from. `None` (after +/// [`absent`]) when any is unset. A variable that IS set but names a path that does +/// not exist is a misconfigured run — pointing a test at a file and having it pass +/// because the file is missing is the failure this module exists to prevent — so +/// that fails regardless of the switch, and whether or not the other variables are +/// set. +#[allow(dead_code)] +pub fn env_paths(test: &str, vars: [&str; N], produce: &str) -> Option<[String; N]> { + let got = vars.map(|v| std::env::var(v).ok()); + // The set ones first: a variable naming a missing path fails even when another + // variable the test reads is unset, which would otherwise end the test as a skip. + for (v, p) in vars.iter().zip(&got) { + if let Some(p) = p { + assert!(Path::new(p).exists(), "{test}: {v}={p} does not exist (produce with: {produce})"); + } + } + let unset: Vec<&str> = + vars.iter().zip(&got).filter(|(_, g)| g.is_none()).map(|(v, _)| *v).collect(); + if !unset.is_empty() { + absent(test, &format!("{} not set", unset.join(" + ")), produce); + return None; + } + Some(got.map(Option::unwrap)) +} + +/// The directory a test reads its inputs from, and whether a variable named it. +/// Made by [`env_dir`]. +#[allow(dead_code)] +pub struct InputDir { + path: PathBuf, + /// The variable that named `path`; `None` when `path` is the default. + named_by: Option, +} + +#[allow(dead_code)] +impl InputDir { + pub fn path(&self) -> &Path { + &self.path + } + + pub fn join(&self, name: impl AsRef) -> PathBuf { + self.path.join(name) + } + + pub fn display(&self) -> std::path::Display<'_> { + self.path.display() + } + + /// [`absent`] for an input that belongs in this directory. When a variable + /// named the directory, the run pointed the test at its inputs, so a missing + /// one fails whether or not the switch is set. Only the default directory may + /// be absent or incomplete and skip. + pub fn absent(&self, test: &str, missing: &str, produce: &str) { + if let Some(var) = &self.named_by { + panic!( + "{test}: NOT VERIFIED — {missing}\n produce with: {produce}\n \ + {var} names this directory, so a missing input is a failure with or \ + without {SWITCH}" + ); + } + absent(test, missing, produce); + } +} + +impl AsRef for InputDir { + fn as_ref(&self) -> &Path { + &self.path + } +} + +/// The directory a test reads its inputs from: the variable `var` when it is set +/// and non-empty, else `default`. Empty counts as unset, as it does for +/// `tools/wai_sota_vectors.sh`'s own `${1:-/tmp/sota_int}`, so the script and the +/// tests fall back to the same place. A variable that IS set names the test's +/// inputs: it fails the test, regardless of the switch, when it does not name a +/// directory (as in [`env_paths`]), and the returned [`InputDir`] fails it when a +/// file is missing from that directory. The default may be absent or incomplete: +/// the test then goes through [`absent`]. +#[allow(dead_code)] +pub fn env_dir(test: &str, var: &str, default: &str, produce: &str) -> InputDir { + match std::env::var(var) { + Ok(p) if !p.is_empty() => { + assert!( + Path::new(&p).is_dir(), + "{test}: {var}={p} is not a directory (produce with: {produce})" + ); + InputDir { path: PathBuf::from(p), named_by: Some(var.to_owned()) } + } + _ => InputDir { path: PathBuf::from(default), named_by: None }, + } +} diff --git a/wai/wai-web/demo/int_hyper.html b/wai/wai-web/demo/int_hyper.html index 81d8b1fc8..8350ca207 100644 --- a/wai/wai-web/demo/int_hyper.html +++ b/wai/wai-web/demo/int_hyper.html @@ -30,7 +30,7 @@ no float anywhere. The whole path is integer: factorized z → integer hyper-decoder → integer-σ entropy bucketing → integer synthesis. Because of that (the integer-entropy contract) the reconstruction is byte-identical on every machine, so the page re-checks its FNV-1a against - the hash the native reference produced — a SOTA-class learned codec verified client-side, no server. + the hash the native reference produced — a pretrained learned codec verified client-side, no server.

loading model…
@@ -42,12 +42,13 @@ native hash —

- Why this is the moat: the SOTA learned-image family is float, so its entropy decode desyncs on - ~1e-5 of cross-platform drift in the scale σ — reproducible only on the same build, never receiptable. - int_hyper is a real pretrained codec (post-training-quantized to integer, ~0.6 dB off - the float baseline) whose entire decode is integer, so the hash above is portable: this browser, a native - sink, and the conformance corpus all produce the same bytes. The wire carried a ~1.4 KB bitstream; - the model is the installed codec. Note: the model files are large (PTQ weights) and not committed — + Why the whole decode is integer: a float learned codec's entropy decode can desync on ~1e-5 of + cross-platform drift in the scale σ, so its output is reproducible only on the same build. + int_hyper is the pretrained mbt2018_mean mean-scale hyperprior, post-training-quantized to + integer (CI requires its decode within 1 dB of the same model's float decode on a pinned test + photograph), and its entire decode is integer, so the hash above is portable: this browser, a native + sink, and the conformance corpus all produce the same bytes. The wire carries only the bitstream (its + size is shown above); the model is the installed codec. Note: the model files are large (PTQ weights) and not committed — generate them with tools/wai_sota_int_synth_export.py into demo/int_hyper/.

diff --git a/wai/wai-web/demo/int_hyper.js b/wai/wai-web/demo/int_hyper.js index 7666389a8..27e29a860 100644 --- a/wai/wai-web/demo/int_hyper.js +++ b/wai/wai-web/demo/int_hyper.js @@ -3,7 +3,8 @@ // Rust that decodes natively, compiled to wasm32, with NO ML runtime (no // onnxruntime-web, no WebGPU, no float). Then verify the decoded image's FNV-1a // against the native reference hash, in-page — the integer-entropy contract: a -// SOTA-class learned codec whose reconstruction hash is portable. +// pretrained learned codec (mbt2018_mean, exported to integers) whose +// reconstruction hash is portable. // // The model files (g_s/h_s, PTQ weights) are large and NOT committed; place the // export tool's output under demo/int_hyper/ (the .bin models are ~½ the JSON). diff --git a/wai/wai-web/demo/verify_int_hyper_wasm.mjs b/wai/wai-web/demo/verify_int_hyper_wasm.mjs index 3c44f6164..df1841575 100644 --- a/wai/wai-web/demo/verify_int_hyper_wasm.mjs +++ b/wai/wai-web/demo/verify_int_hyper_wasm.mjs @@ -1,5 +1,5 @@ // Reproducible proof: the wasm32 FULL CONDITIONAL codec decode is byte-identical -// to the native reference — a SOTA-class learned image codec (PTQ of mbt2018_mean) +// to the native reference — a pretrained learned image codec (PTQ of mbt2018_mean) // whose whole bitstream→image decode (factorized z, integer h_s, integer-σ // entropy, integer g_s) runs in the browser engine with NO ML runtime, and whose // RGB verifies against the native FNV-1a hash. The integer-entropy contract, @@ -59,6 +59,6 @@ const bin = check('binary model', wai.decodeIntHyperBin( const pass = json && bin; console.log(pass - ? '\n✓ wasm32 conditional-codec decode (JSON + compact-binary model) is byte-identical to the native reference — a SOTA learned codec decoded in-browser, zero JS ML runtime.' + ? '\n✓ wasm32 conditional-codec decode (JSON + compact-binary model) is byte-identical to the native reference — a pretrained learned codec decoded in-browser, zero JS ML runtime.' : '\n✗ mismatch'); process.exit(pass ? 0 : 1); diff --git a/wai/wai-web/src/neural_int.rs b/wai/wai-web/src/neural_int.rs index f783cf48d..7194af16c 100644 --- a/wai/wai-web/src/neural_int.rs +++ b/wai/wai-web/src/neural_int.rs @@ -60,7 +60,7 @@ pub fn decode_int_synth(payload: &[u8], spec_json: &[u8]) -> Result Date: Mon, 28 Sep 2026 18:42:16 -0400 Subject: [PATCH 2/2] ci(wai): run the JPEG AI cross-platform pinned-output test in the whole-decode job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pinned-output test (tests/jpegai_cross_platform.rs) needs the same inputs the whole-decode job already fetches and checks, but no job ran it. The job now runs it with the other two tests, requires it by name, and triggers on changes to it. determinism-tiers §6 and the CHANGELOG say so. Verified locally with the job's own script and the pinned inputs: all three required tests ran and passed, the pinned pixel hash was checked, and no test printed a skip banner. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/wai-jpegai-e2e.yml | 8 ++++++-- wai/CHANGELOG.md | 2 +- wai/extensions/determinism-tiers.md | 2 +- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.github/workflows/wai-jpegai-e2e.yml b/.github/workflows/wai-jpegai-e2e.yml index 9f6c7d72d..43b41b85e 100644 --- a/.github/workflows/wai-jpegai-e2e.yml +++ b/.github/workflows/wai-jpegai-e2e.yml @@ -80,6 +80,7 @@ on: - 'wai/wai-rs/Cargo.lock' - 'wai/wai-rs/tests/jpegai_e2e_full.rs' - 'wai/wai-rs/tests/jpegai_reconstruction_binding.rs' + - 'wai/wai-rs/tests/jpegai_cross_platform.rs' - 'wai/wai-rs/tests/support/**' - 'wai/tools/wai_jpegai_fetch_e2e_inputs.sh' - 'wai/tools/jpegai-e2e-v1.sha256' @@ -100,6 +101,7 @@ on: - 'wai/wai-rs/Cargo.lock' - 'wai/wai-rs/tests/jpegai_e2e_full.rs' - 'wai/wai-rs/tests/jpegai_reconstruction_binding.rs' + - 'wai/wai-rs/tests/jpegai_cross_platform.rs' - 'wai/wai-rs/tests/support/**' - 'wai/tools/wai_jpegai_fetch_e2e_inputs.sh' - 'wai/tools/jpegai-e2e-v1.sha256' @@ -150,10 +152,12 @@ jobs: # --show-output rather than --nocapture: each test's output is printed # after the results, so it cannot split a `test X ... ok` line below. cargo test --release --no-default-features --features neural_int,provenance \ - --test jpegai_e2e_full --test jpegai_reconstruction_binding -- --show-output 2>&1 | tee "$log" + --test jpegai_e2e_full --test jpegai_reconstruction_binding --test jpegai_cross_platform \ + -- --show-output 2>&1 | tee "$log" # A test that was renamed or compiled out would not fail — it would just # not run. Require each one by name, and the pinned-hash check by its line. - for t in codestream_to_rgb_matches_reference_png real_decode_to_reconstruction_binding; do + for t in codestream_to_rgb_matches_reference_png real_decode_to_reconstruction_binding \ + decode_matches_pinned_hashes; do grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; } done grep -q "matches the pinned WAI_JPEGAI_EXPECT_SHA256" "$log" \ diff --git a/wai/CHANGELOG.md b/wai/CHANGELOG.md index 286cc2672..e5170f5c0 100644 --- a/wai/CHANGELOG.md +++ b/wai/CHANGELOG.md @@ -347,7 +347,7 @@ wire format may change and such changes are recorded here. this change's final source, and on all four also before the math-library calls were replaced. determinism-tiers §6 records the toolchain and the method for each. Not measured: macOS x86_64, Windows, x86_64 hardware. The test skips without its - inputs, and no CI job supplies them yet; `float_repro`'s own pinned-output test + inputs; `wai-jpegai-e2e.yml` supplies them and requires it to pass; `float_repro`'s own pinned-output test needs no inputs and runs in the library tests. - Two checks keep math-library calls off the decode path. Clippy's `disallowed_methods` (`wai-rs/clippy.toml`), denied in each module on the path, diff --git a/wai/extensions/determinism-tiers.md b/wai/extensions/determinism-tiers.md index 140a05108..f7b99c544 100644 --- a/wai/extensions/determinism-tiers.md +++ b/wai/extensions/determinism-tiers.md @@ -485,7 +485,7 @@ day. | Cross-architecture identity of integer inter-frame video, and of `int_hyper` on a small learned keyframe | the same suite, passing on Linux x86_64, Linux arm64 and macOS arm64 on main at `afc7ece` (2026-09-27). Windows x86_64 never decoded the keyframe clip: it was added in `d4c922b` (2026-08-28), whose Windows legs were cancelled. Windows decoded the integer video goldens only on the pre-merge commits `c9be2ad`, `256e5dd`, `7dfca26` and `3d34860` (2026-08-25/26), whose video sources, vectors and golden file match main but whose `byte-exact-conformance/Cargo.lock` differs; the change reached main at `3f50097`, after the last Windows run on main | as above | | `wai.image.jpegai` exact decoder: the integer derivation of the distribution indices matches the reference decoder exactly | a test on one machine that skips unless its fixtures (sink-supplied weights, normative tables, codestream) are provided; CI does not provide them | `wai-rs/tests/jpegai_hyperscale_calibration.rs` | | `wai.image.jpegai` exact decoder, whole decode through the one public `decode()` call: it stays within a stated bound of the reference decoder's output (max\|Δ\| ≤ 2, ≤ 5% of samples); two decodes in one process are byte-identical, and a reconstruction binding taken over one verifies the other | `wai-jpegai-e2e.yml` fetches the pinned `jpegai/v1` inputs (an extraction of the reference-software assets, published under WAI's models root), checks each file against the SHA-256 manifest `tools/jpegai-e2e-v1.sha256`, and runs both tests with `WAI_REQUIRE_SOTA_VECTORS=1`, under which a missing input fails instead of skipping. The job also fails on any skip banner in the test log, and requires the decoded pixels to hash to a pinned SHA-256. One 256×256 codestream. The workflow has not run through the forge. Its job passed under `forgejo-runner exec` on the Linux runner's host (x86_64 hardware, 2026-09-27), with the exact decoder as it stood on main at `afc7ece`, which calls the platform `exp`; there the decoded pixels matched the pinned hash, the value measured on macOS arm64. The job's steps have otherwise run on macOS arm64 only, outside a runner | `wai-rs/tests/jpegai_e2e_full.rs`, `jpegai_reconstruction_binding.rs`; `.github/workflows/wai-jpegai-e2e.yml` | -| `wai.image.jpegai` exact decoder across platforms: the `jpegai/v1` codestream decodes to the same RGB8 bytes, and to the same bits in the float values the decoder reports after each stage (values inside a stage are not hashed), on each platform measured | measured 2026-09-28 with rustc 1.98.0, release builds, inputs checked against their SHA-256, all four on the source of the change that added this row: macOS arm64 (native, on the development machine); wasm32-wasip2 (wasmtime 49, on the same machine); Linux arm64 (glibc 2.36, a container in an arm64 Linux VM on that machine); Linux x86_64 (glibc 2.36, the x86_64 build run in that container under user-mode emulation, QEMU 7.0, not on x86_64 hardware). All nine pinned hashes matched on each platform. On all four they also matched on an earlier state of that change, before the decode's `exp` and rounding moved to `float_repro`. Before the move the decode called the platform `exp` on every target, and its roundings compiled to calls to the symbols `round`, `roundf` and `rint` on x86_64 and `round` and `roundf` on wasm32, which the link resolves: an x86_64 Linux test executable inspected after the move, whose other code still rounds this way, took them from Rust's compiler builtins, not the C library (for wasm32 this was not checked). Not measured: macOS x86_64 (no x86_64 translation layer is installed on the development machine), Windows, x86_64 hardware, 32-bit targets. The test skips unless the inputs are supplied, and no CI job supplies them to it | `wai-rs/tests/jpegai_cross_platform.rs` | +| `wai.image.jpegai` exact decoder across platforms: the `jpegai/v1` codestream decodes to the same RGB8 bytes, and to the same bits in the float values the decoder reports after each stage (values inside a stage are not hashed), on each platform measured | measured 2026-09-28 with rustc 1.98.0, release builds, inputs checked against their SHA-256, all four on the source of the change that added this row: macOS arm64 (native, on the development machine); wasm32-wasip2 (wasmtime 49, on the same machine); Linux arm64 (glibc 2.36, a container in an arm64 Linux VM on that machine); Linux x86_64 (glibc 2.36, the x86_64 build run in that container under user-mode emulation, QEMU 7.0, not on x86_64 hardware). All nine pinned hashes matched on each platform. On all four they also matched on an earlier state of that change, before the decode's `exp` and rounding moved to `float_repro`. Before the move the decode called the platform `exp` on every target, and its roundings compiled to calls to the symbols `round`, `roundf` and `rint` on x86_64 and `round` and `roundf` on wasm32, which the link resolves: an x86_64 Linux test executable inspected after the move, whose other code still rounds this way, took them from Rust's compiler builtins, not the C library (for wasm32 this was not checked). Not measured: macOS x86_64 (no x86_64 translation layer is installed on the development machine), Windows, x86_64 hardware, 32-bit targets. The test skips unless the inputs are supplied. `wai-jpegai-e2e.yml` supplies them and requires this test to run and pass on its Linux runner; that workflow has not yet run through the forge | `wai-rs/tests/jpegai_cross_platform.rs` | | `float_repro`, the decode's `exp` and rounding: the same bits over a fixed grid of inputs; `exp` within 2 ULPs of the host math library (1 ULP measured); rounding equal to the standard library's | library unit tests under the library-test step of the conformance workflow (`neural_int`), which has not yet run them on a runner. The pinned-bits and rounding tests never skip. The host comparison asserts the host library's accuracy as much as this module's. Its gate selects by target, not by host: it runs on macOS arm64, on Linux x86_64 and arm64 with glibc, and on WASI, and is reported as ignored, with the reason, elsewhere, Windows included. On the source of the change that added this row it was run against these hosts' `exp`, with a worst case of 1 ULP on each: Apple's math library on macOS arm64; the WASI C library under wasmtime; glibc 2.36 on Linux arm64, and glibc 2.35 in the Ubuntu 22.04 image the workflow's Linux arm64 runner used on 2026-09-28; glibc 2.36 on Linux x86_64 under user-mode emulation, against three of the four `exp` variants glibc chooses among by processor feature (the variant each run used follows from the features glibc's loader reported active): SSE2 (QEMU 7.0, whose emulated processor reports none of AVX, FMA, AVX2 and FMA4), AVX (QEMU 7.2, `-cpu max` with FMA and AVX2 masked by glibc's `hwcaps` tunable, which leaves AVX active) and FMA (QEMU 7.2, `-cpu max`). On a separate probe of 20 million inputs, the SSE2 and AVX variants returned the same bits, and the FMA variant different bits for some of them. Not compared: the FMA4 variant, which glibc chooses on a processor that has FMA4 but not both FMA and AVX2, and which neither emulator offers. Not run on x86_64 hardware or on other glibc versions, where the gate still runs it; a failure there would say that the host's `exp` and this one are more than 2 ULPs apart, not that `float_repro`'s bits changed | `wai-rs/src/float_repro.rs` tests | | The exact decode path calls no platform math library | two checks. Clippy's `disallowed_methods` (`wai-rs/clippy.toml`), denied in each module on the decode path, rejects any use of a listed float method, including `f64::exp(x)` and `.map(f64::exp)`. `tests/jpegai_decode_symbols.rs` lists the symbols of an executable that links only the decoder's entry points and weight loaders. It rejects any C math-library function the executable imports, and any it defines other than the round-to-integral functions (`floor`, `round`, `rint` and the like), whose results IEEE 754 fixes. It allows those definitions because a definition shows what the linker kept, not what the decode calls: linked with GNU ld 2.40 for x86_64 Linux, the debug executable defined the compiler builtins' `floor`, though nothing in it referred to `floor`. On the source of the change that added this row, both checks pass on macOS arm64, the symbol check in debug and release builds. The symbol check also passes, in debug builds, on Linux arm64 (the container above) and on Linux x86_64 under user-mode emulation, linked with GNU ld (reporting that `floor` as allowed) and with LLD. With a platform `exp` put back as `f64::exp(…)`, both fail on macOS arm64 while the decoder's unit tests still pass, and the symbol check fails on Linux x86_64 with GNU ld. With a float `%` put into the decode, the symbol check fails on macOS arm64 and clippy, which does not see operators, passes. The conformance workflow runs both on its unix legs; neither has run on a runner yet | `wai-rs/clippy.toml`, `wai-rs/tests/jpegai_decode_symbols.rs`, `.github/workflows/wai-conformance.yml` | | `wai.image.jpegai` scope: the reference codestream is accepted; copies with β, `model_id`, the colour transform or an LSBS flag changed are refused; a flush-to-zero or round-toward-zero environment is detected. The float ONNX path calls the same check | library unit tests over a committed copy of the reference codestream, which never skip and fall under the library-test step of the conformance workflow (`neural_int`); the environment test runs on arm64 only. Run on the four platforms above, on the source of the change that added the cross-platform row (the environment test on macOS arm64 and Linux arm64). The float path's refusal (β and `model_id` edits) is tested only with its ONNX bundle supplied (`jpegai_scope_ort.rs`, skips otherwise) | `wai-rs/src/jpegai_decode.rs` tests, `wai-rs/tests/vectors/jpegai/` | -- 2.54.0 (Apple Git-157)