name: WAI mbt2018_mean integer decode (regenerated vectors) # `wai.neural.int_hyper` is an integer export of one pretrained model: the # mbt2018_mean mean-scale hyperprior from the CompressAI model zoo (quality 6, # MSE), post-training-quantized to integers. WAI states that its whole # bitstream→image decode is integer, so the Rust decoder reproduces the export # tool's integer reference byte-for-byte, and that `wai.video.int_hyper` builds a # clip on that keyframe. The model is too large to commit, so the tests that carry # these statements read vectors from outside the tree; without them they used to # return early and report PASS. # # This job checks them on every run: it regenerates the vectors from pinned # inputs (a public-domain photograph shipped in the pinned scikit-image wheel and # pinned by the SHA-256 of its pixels; the published mbt2018_mean checkpoint, # pinned by its full SHA-256 and loaded as tensors only; a pinned Python stack), # then runs the int_hyper vector tests with WAI_REQUIRE_SOTA_VECTORS=1, under which # a missing vector is a failure. It also runs tests/require_vectors_rule.rs, which # checks in child processes what the switch does not cover here: that without it a # variable naming absent or incomplete inputs still fails, and a test skips only # when no variable names what is missing (an unset variable, or an absent or # incomplete default location). What passes here: # - the Rust integer decode reproduces the export tool's integer reference # byte-for-byte (synthesis, hyper-decoder, the whole conditional bitstream, the # compact binary model, the eight-frame clip); # - on the pinned photograph, the integer decode costs at most 1 dB against the # same model's float decode (the bar CI enforces), measured by the Rust decoder # on its own output. Measured on an Apple Silicon Mac: 35.985 dB for the integer # decode (8-bit PSNR against the source pixels) against 36.175 dB for the float # decode (the export tool's PSNR of the float reconstruction, before 8-bit # rounding), a cost of 0.19 dB, from 6328 bytes of rANS payload for the # 256×256 crop (about 0.77 bits per pixel). # # What this job does not cover: # - How mbt2018_mean compares with any other codec. Nothing here measures rate or # distortion against another model or standard; the job checks only that the # integer decode is exact and within 1 dB of the same model's float decode. # - Quality on other images. PSNR is image-dependent and the job measures one # crop. The 42.6 dB development figure was measured on a photograph that is not # in the repository. # - Cross-architecture byte-equality of these vectors. They are regenerated on # this runner, so the job checks Rust against the Python reference on one # machine (x86_64 Linux). The cross-architecture check is the committed tiny # int_hyper vector in wai-byte-equality.yml. # - Portable or repeatable vector hashes. No expected hash of the regenerated # vectors is pinned in the repository (the committed tiny vector's is, in # tests/vectors/int_hyper/tiny.json). Each is written by the same run's export # and checked in that run, and a hash taken from one run's vectors must not be # pinned for another run, on this machine or another: it can differ with both # decoders correct. What was measured, comparing all sixteen files by SHA-256: # - Between an Apple Silicon Mac and this runner's x86_64 image, four files # differed. int_synth_sota_input.json (the synthesis-only test's input # latent, which the float encoder and hyper-decoder compute) differed, and # with it the rgb_hash in int_synth_sota.json (fd9ca73a… on the Mac, # f0fb6813… here); each side wrote the same latent in all of its runs. # cond.json and int_synth_sota.json also differed. Beyond that rgb_hash, # their difference is not established as a machine difference: both files # also differ from run to run on this runner, most likely in psnr_float # (next point). source.png differed in its compressed bytes only: step 1 # writes it with the PNG encoder from the pinned pixels before any model # runs, and its pixels (source.rgb) were identical, so that difference is # the encoder's output, not float computation. # - Two runs of the same export on this runner (same image, pins and tree, # minutes apart) wrote different cond.json and int_synth_sota.json; the # other fourteen files and both rgb_hash values matched. The field those two # files share that comes from the float model is psnr_float, the float # decode's PSNR, so it most likely varies from run to run here, although the # export sets torch to deterministic algorithms on one thread. The runner's # files were not kept, so the field is inferred, not diffed. Repeated runs on # the Mac wrote identical files. # Nothing the tests compare exactly reads psnr_float; only the quality test # does, against its thresholds (the 1 dB bar and a 30 dB floor). The files # that matched (both models, the conditional bitstream, frame0, the clip) are # derived from float computation too, so matching here does not make them # portable. The pins that hold across runs and machines are on the inputs: the # photograph's pixels and the checkpoint's bytes. The Python stack is pinned # by version, but each platform installs its own build of it. # The JPEG-AI whole-decoder tests run in wai-jpegai-e2e.yml (see # wai/wai-rs/tests/support/require_vectors.rs). # # The path filters list the source files the int_hyper decode imports, among # them src/int_kernels.rs (the fast convolution kernels int_hyper_synth.rs runs # through). An entry for a file the tree does not have matches nothing until the # file exists. A module the decode path starts importing belongs here too; # otherwise a change to that module alone is first checked by the weekly run. # # Licences of what is fetched: CompressAI and its model zoo, BSD-3-Clause-Clear; # the astronaut photograph, public domain (NASA), per the scikit-image docstring. # One run per branch at a time. A push supersedes the run before it. # # There is one `macos` runner and it is the dev Mac, so every macOS job in this # repo queues on it. With no concurrency group, a session pushing four versions # of a branch queued four complete sets of runs and every one of them had to # drain: on 2026-10-05 a PR's 21 contexts settled none in 70 minutes behind a # backlog of 25 runs, and the queue grew faster than it cleared. Only the newest # commit's verdict is wanted, so the older runs are cancelled. # # Verified on this forge (Forgejo 16.0.5) in `ci-test` before being added here: # the key parses, and a newer push moves an older run to `cancelled` even while # it is still queued, which is the case that matters. # # `cancel-in-progress: true` also applies to `main`. That is deliberate and # safe here: main advances by merge commits whose tree contains the commits # before them, so the newer run covers the older one's changes. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true on: push: branches: [main] paths: - 'wai/wai-rs/src/int_hyper_synth.rs' - 'wai/wai-rs/src/int_kernels.rs' - 'wai/wai-rs/src/int_entropy.rs' - 'wai/wai-rs/src/int_transform.rs' - 'wai/wai-rs/src/int_video.rs' - 'wai/wai-rs/src/rans.rs' - 'wai/wai-rs/src/lib.rs' - 'wai/wai-rs/Cargo.toml' - 'wai/wai-rs/Cargo.lock' - 'wai/wai-rs/examples/int_hyper_decode_frame.rs' - 'wai/wai-rs/tests/int_hyper_synth.rs' - 'wai/wai-rs/tests/int_video_sota_keyframe.rs' - 'wai/wai-rs/tests/support/**' - 'wai/wai-rs/tests/require_vectors_rule.rs' - 'wai/tools/wai_sota_*' - 'wai/tools/wai_int_video_sota_keyframe_encode.py' - 'wai/tools/requirements-sota-int.txt' # The wasm32 decode of the same vectors (the last steps below). - 'wai/wai-web/**' - '.github/workflows/wai-sota-vectors.yml' pull_request: paths: - 'wai/wai-rs/src/int_hyper_synth.rs' - 'wai/wai-rs/src/int_kernels.rs' - 'wai/wai-rs/src/int_entropy.rs' - 'wai/wai-rs/src/int_transform.rs' - 'wai/wai-rs/src/int_video.rs' - 'wai/wai-rs/src/rans.rs' - 'wai/wai-rs/src/lib.rs' - 'wai/wai-rs/Cargo.toml' - 'wai/wai-rs/Cargo.lock' - 'wai/wai-rs/examples/int_hyper_decode_frame.rs' - 'wai/wai-rs/tests/int_hyper_synth.rs' - 'wai/wai-rs/tests/int_video_sota_keyframe.rs' - 'wai/wai-rs/tests/support/**' - 'wai/wai-rs/tests/require_vectors_rule.rs' - 'wai/tools/wai_sota_*' - 'wai/tools/wai_int_video_sota_keyframe_encode.py' - 'wai/tools/requirements-sota-int.txt' # The wasm32 decode of the same vectors (the last steps below). - 'wai/wai-web/**' - '.github/workflows/wai-sota-vectors.yml' # Weekly, independent of changes: the job depends on artifacts fetched from # outside the repository (the checkpoint, the pinned wheels), and their # disappearing should surface as a failure here, not at the next unrelated PR. schedule: - cron: '17 6 * * 1' jobs: sota-vectors: name: regenerate mbt2018_mean integer vectors → Rust decode, required # The `linux` runner is a docker executor on catthehacker/ubuntu:act-22.04 # (glibc 2.35, no Rust). The Python half of this setup — setup-python 3.11, # CPU torch, compressai — completes there: every logged wai-neural-e2e.yml # run got through it and generated its fixtures. Each of those runs that # reached the Rust build then failed linking `ort`'s prebuilt ONNX Runtime, # which references glibc 2.38 and newer-libstdc++ symbols (`__isoc23_strtol`, # `_M_replace_cold`). The features built here (neural_int, video_int) are # pure Rust and link no prebuilt native library; do not add `neural` here. runs-on: linux timeout-minutes: 45 env: WAI_SOTA_INT_DIR: ${{ github.workspace }}/sota_int WAI_REQUIRE_SOTA_VECTORS: '1' steps: - uses: actions/checkout@v5 - uses: actions/setup-python@v5 with: python-version: '3.11' - name: Pinned ML stack (CPU torch first, then the rest) shell: bash working-directory: wai run: | python -m pip install --upgrade pip # CPU wheels: the default index resolves the multi-GB CUDA builds. python -m pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/cpu python -m pip install -r tools/requirements-sota-int.txt # Fail here, not mid-export, if the resolver installed anything but the # pins, or if torchvision's compiled ops do not match torch (the # `torchvision::nms does not exist` mismatch wai-neural-e2e.yml once hit). python - tools/requirements-sota-int.txt <<'PY' import sys, importlib.metadata as md bad = [] for line in open(sys.argv[1]): line = line.split("#")[0].strip() if not line: continue name, want = line.split("==") got = md.version(name) print(f"{name}=={got}") if got.split("+")[0] != want: bad.append(f"{name} {got}, pinned {want}") if bad: sys.exit("not the pinned stack: " + "; ".join(bad)) import torch, torchvision, compressai, skimage torchvision.ops.nms(torch.zeros(1, 4), torch.zeros(1), 0.5) PY # Bootstrap rustup: the docker runner image ships no Rust. - name: Install Rust 1.98.0 shell: bash run: | export PATH="$HOME/.cargo/bin:$PATH" command -v rustup >/dev/null 2>&1 || \ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0 rustup toolchain install 1.98.0 --profile minimal rustup default 1.98.0 echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - name: Regenerate the int_hyper vectors (source photo → integer export → keyframe → clip) shell: bash working-directory: wai run: | tools/wai_sota_vectors.sh "$WAI_SOTA_INT_DIR" # Recorded so a result can be tied to the exact vectors it checked. It # identifies this run's vectors, not this machine's: two runs on this # runner wrote different cond.json and int_synth_sota.json, and other # files differ between machines (see the header). It is not a list any # other run must reproduce, here or elsewhere. (cd "$WAI_SOTA_INT_DIR" && sha256sum *) - name: Rust decodes every int_hyper vector byte-for-byte (missing input = failure), and the missing-input rule holds shell: bash working-directory: wai/wai-rs run: | set -o pipefail log="${RUNNER_TEMP:-/tmp}/sota-tests.log" # Without the switch every vector test would skip and report ok. [ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \ || { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; } # --show-output rather than --nocapture: each test's output is printed # after the results, so it cannot split a `test X ... ok` line below. # require_vectors_rule runs its cases in child processes with the switch # and the input variables set per case, so the job's own env does not # reach them; it prints nothing on success, so it cannot trip the skip # backstop below. cargo test --release --no-default-features --features neural_int,video_int \ --test int_hyper_synth --test int_video_sota_keyframe --test require_vectors_rule \ -- --show-output 2>&1 | tee "$log" # A test that was renamed or compiled out would not fail — it would just # not run. Require each one by name. for t in rust_integer_synthesis_reproduces_the_export_hash \ rust_integer_hyper_decoder_reproduces_the_export_hash \ rust_decodes_the_full_conditional_bitstream_to_the_export_image \ integer_decode_keeps_the_float_codec_quality \ decodes_the_committed_tiny_conformance_vector \ sota_int_hyper_keyframe_clip_decodes_byte_for_byte \ without_the_switch_a_named_directory_that_does_not_exist_fails \ without_the_switch_a_named_path_that_is_a_file_fails_as_a_directory \ without_the_switch_a_named_empty_directory_fails \ without_the_switch_an_unset_or_empty_directory_variable_skips_on_an_absent_default \ without_the_switch_an_incomplete_default_directory_skips \ a_named_directory_holding_the_input_loads_it \ without_the_switch_a_path_variable_naming_nothing_fails \ without_the_switch_a_path_variable_naming_nothing_fails_when_another_is_unset \ without_the_switch_an_unset_path_variable_skips \ path_variables_naming_files_that_exist_load_them \ switch_values_that_leave_it_off_behave_as_unset \ with_the_switch_an_absent_default_fails; do grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; } done # A skipped test ALSO prints `test X ... ok`, so the name check alone # cannot tell a skip from a pass. Every skip writes a banner containing # "SKIPPED" and "NOT VERIFIED" (tests/support/require_vectors.rs); either # word anywhere in the log fails the job, whatever the switch says. if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then echo "::error::a vector test skipped instead of checking its input" exit 1 fi # The same vectors decoded by the wasm32 build of wai-web (the browser # sink), in the same job so they share $WAI_SOTA_INT_DIR. The verify script # fails on a missing artifact here (WAI_REQUIRE_SOTA_VECTORS=1, and the # directory is named), and compares the wasm decode's RGB hash, for the # JSON and the compact binary model, with the native export's. - name: wasm32 toolchain (target, wasm-bindgen CLI pinned to wai-web's lock) shell: bash run: | rustup target add wasm32-unknown-unknown cargo install --locked wasm-bindgen-cli --version 0.2.127 - uses: actions/setup-node@v4 with: node-version: '22' - name: wasm32 decode of the int_hyper vectors equals the native reference shell: bash working-directory: wai/wai-web run: | set -euo pipefail cargo build --locked --release --target wasm32-unknown-unknown wasm-bindgen --target nodejs --out-dir demo/pkg-node --out-name wai_web \ target/wasm32-unknown-unknown/release/wai_web.wasm node demo/verify_int_hyper_wasm.mjs ./pkg-node/wai_web.js "$WAI_SOTA_INT_DIR"