open-standards/.github/workflows/wai-sota-vectors.yml

name: WAI mbt2018_mean integer decode (regenerated vectors)

# `wai.neural.int_hyper` is an integer export of one pretrained model: the
# mbt2018_mean mean-scale hyperprior from the CompressAI model zoo (quality 6,
# MSE), post-training-quantized to integers. WAI states that its whole
# bitstream→image decode is integer, so the Rust decoder reproduces the export
# tool's integer reference byte-for-byte, and that `wai.video.int_hyper` builds a
# clip on that keyframe. The model is too large to commit, so the tests that carry
# these statements read vectors from outside the tree; without them they used to
# return early and report PASS.
#
# This job checks them on every run: it regenerates the vectors from pinned
# inputs (a public-domain photograph shipped in the pinned scikit-image wheel and
# pinned by the SHA-256 of its pixels; the published mbt2018_mean checkpoint,
# pinned by its full SHA-256 and loaded as tensors only; a pinned Python stack),
# then runs the int_hyper vector tests with WAI_REQUIRE_SOTA_VECTORS=1, under which
# a missing vector is a failure. It also runs tests/require_vectors_rule.rs, which
# checks in child processes what the switch does not cover here: that without it a
# variable naming absent or incomplete inputs still fails, and a test skips only
# when no variable names what is missing (an unset variable, or an absent or
# incomplete default location). What passes here:
# - the Rust integer decode reproduces the export tool's integer reference
#   byte-for-byte (synthesis, hyper-decoder, the whole conditional bitstream, the
#   compact binary model, the eight-frame clip);
# - on the pinned photograph, the integer decode costs at most 1 dB against the
#   same model's float decode (the bar CI enforces), measured by the Rust decoder
#   on its own output. Measured on an Apple Silicon Mac: 35.985 dB for the integer
#   decode (8-bit PSNR against the source pixels) against 36.175 dB for the float
#   decode (the export tool's PSNR of the float reconstruction, before 8-bit
#   rounding), a cost of 0.19 dB, from 6328 bytes of rANS payload for the
#   256×256 crop (about 0.77 bits per pixel).
#
# What this job does not cover:
# - How mbt2018_mean compares with any other codec. Nothing here measures rate or
#   distortion against another model or standard; the job checks only that the
#   integer decode is exact and within 1 dB of the same model's float decode.
# - Quality on other images. PSNR is image-dependent and the job measures one
#   crop. The 42.6 dB development figure was measured on a photograph that is not
#   in the repository.
# - Cross-architecture byte-equality of these vectors. They are regenerated on
#   this runner, so the job checks Rust against the Python reference on one
#   machine (x86_64 Linux). The cross-architecture check is the committed tiny
#   int_hyper vector in wai-byte-equality.yml.
# - Portable or repeatable vector hashes. No expected hash of the regenerated
#   vectors is pinned in the repository (the committed tiny vector's is, in
#   tests/vectors/int_hyper/tiny.json). Each is written by the same run's export
#   and checked in that run, and a hash taken from one run's vectors must not be
#   pinned for another run, on this machine or another: it can differ with both
#   decoders correct. What was measured, comparing all sixteen files by SHA-256:
#   - Between an Apple Silicon Mac and this runner's x86_64 image, four files
#     differed. int_synth_sota_input.json (the synthesis-only test's input
#     latent, which the float encoder and hyper-decoder compute) differed, and
#     with it the rgb_hash in int_synth_sota.json (fd9ca73a… on the Mac,
#     f0fb6813… here); each side wrote the same latent in all of its runs.
#     cond.json and int_synth_sota.json also differed. Beyond that rgb_hash,
#     their difference is not established as a machine difference: both files
#     also differ from run to run on this runner, most likely in psnr_float
#     (next point). source.png differed in its compressed bytes only: step 1
#     writes it with the PNG encoder from the pinned pixels before any model
#     runs, and its pixels (source.rgb) were identical, so that difference is
#     the encoder's output, not float computation.
#   - Two runs of the same export on this runner (same image, pins and tree,
#     minutes apart) wrote different cond.json and int_synth_sota.json; the
#     other fourteen files and both rgb_hash values matched. The field those two
#     files share that comes from the float model is psnr_float, the float
#     decode's PSNR, so it most likely varies from run to run here, although the
#     export sets torch to deterministic algorithms on one thread. The runner's
#     files were not kept, so the field is inferred, not diffed. Repeated runs on
#     the Mac wrote identical files.
#   Nothing the tests compare exactly reads psnr_float; only the quality test
#   does, against its thresholds (the 1 dB bar and a 30 dB floor). The files
#   that matched (both models, the conditional bitstream, frame0, the clip) are
#   derived from float computation too, so matching here does not make them
#   portable. The pins that hold across runs and machines are on the inputs: the
#   photograph's pixels and the checkpoint's bytes. The Python stack is pinned
#   by version, but each platform installs its own build of it.
# The JPEG-AI whole-decoder tests run in wai-jpegai-e2e.yml (see
# wai/wai-rs/tests/support/require_vectors.rs).
#
# The path filters list the source files the int_hyper decode imports, among
# them src/int_kernels.rs (the fast convolution kernels int_hyper_synth.rs runs
# through). An entry for a file the tree does not have matches nothing until the
# file exists. A module the decode path starts importing belongs here too;
# otherwise a change to that module alone is first checked by the weekly run.
#
# Licences of what is fetched: CompressAI and its model zoo, BSD-3-Clause-Clear;
# the astronaut photograph, public domain (NASA), per the scikit-image docstring.

on:
  push:
    branches: [main]
    paths:
      - 'wai/wai-rs/src/int_hyper_synth.rs'
      - 'wai/wai-rs/src/int_kernels.rs'
      - 'wai/wai-rs/src/int_entropy.rs'
      - 'wai/wai-rs/src/int_transform.rs'
      - 'wai/wai-rs/src/int_video.rs'
      - 'wai/wai-rs/src/rans.rs'
      - 'wai/wai-rs/src/lib.rs'
      - 'wai/wai-rs/Cargo.toml'
      - 'wai/wai-rs/Cargo.lock'
      - 'wai/wai-rs/examples/int_hyper_decode_frame.rs'
      - 'wai/wai-rs/tests/int_hyper_synth.rs'
      - 'wai/wai-rs/tests/int_video_sota_keyframe.rs'
      - 'wai/wai-rs/tests/support/**'
      - 'wai/wai-rs/tests/require_vectors_rule.rs'
      - 'wai/tools/wai_sota_*'
      - 'wai/tools/wai_int_video_sota_keyframe_encode.py'
      - 'wai/tools/requirements-sota-int.txt'
      # The wasm32 decode of the same vectors (the last steps below).
      - 'wai/wai-web/**'
      - '.github/workflows/wai-sota-vectors.yml'
  pull_request:
    paths:
      - 'wai/wai-rs/src/int_hyper_synth.rs'
      - 'wai/wai-rs/src/int_kernels.rs'
      - 'wai/wai-rs/src/int_entropy.rs'
      - 'wai/wai-rs/src/int_transform.rs'
      - 'wai/wai-rs/src/int_video.rs'
      - 'wai/wai-rs/src/rans.rs'
      - 'wai/wai-rs/src/lib.rs'
      - 'wai/wai-rs/Cargo.toml'
      - 'wai/wai-rs/Cargo.lock'
      - 'wai/wai-rs/examples/int_hyper_decode_frame.rs'
      - 'wai/wai-rs/tests/int_hyper_synth.rs'
      - 'wai/wai-rs/tests/int_video_sota_keyframe.rs'
      - 'wai/wai-rs/tests/support/**'
      - 'wai/wai-rs/tests/require_vectors_rule.rs'
      - 'wai/tools/wai_sota_*'
      - 'wai/tools/wai_int_video_sota_keyframe_encode.py'
      - 'wai/tools/requirements-sota-int.txt'
      # The wasm32 decode of the same vectors (the last steps below).
      - 'wai/wai-web/**'
      - '.github/workflows/wai-sota-vectors.yml'
  # Weekly, independent of changes: the job depends on artifacts fetched from
  # outside the repository (the checkpoint, the pinned wheels), and their
  # disappearing should surface as a failure here, not at the next unrelated PR.
  schedule:
    - cron: '17 6 * * 1'

jobs:
  sota-vectors:
    name: regenerate mbt2018_mean integer vectors → Rust decode, required
    # The `linux` runner is a docker executor on catthehacker/ubuntu:act-22.04
    # (glibc 2.35, no Rust). The Python half of this setup — setup-python 3.11,
    # CPU torch, compressai — completes there: every logged wai-neural-e2e.yml
    # run got through it and generated its fixtures. Each of those runs that
    # reached the Rust build then failed linking `ort`'s prebuilt ONNX Runtime,
    # which references glibc 2.38 and newer-libstdc++ symbols (`__isoc23_strtol`,
    # `_M_replace_cold`). The features built here (neural_int, video_int) are
    # pure Rust and link no prebuilt native library; do not add `neural` here.
    runs-on: linux
    timeout-minutes: 45
    env:
      WAI_SOTA_INT_DIR: ${{ github.workspace }}/sota_int
      WAI_REQUIRE_SOTA_VECTORS: '1'
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Pinned ML stack (CPU torch first, then the rest)
        shell: bash
        working-directory: wai
        run: |
          python -m pip install --upgrade pip
          # CPU wheels: the default index resolves the multi-GB CUDA builds.
          python -m pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/cpu
          python -m pip install -r tools/requirements-sota-int.txt
          # Fail here, not mid-export, if the resolver installed anything but the
          # pins, or if torchvision's compiled ops do not match torch (the
          # `torchvision::nms does not exist` mismatch wai-neural-e2e.yml once hit).
          python - tools/requirements-sota-int.txt <<'PY'
          import sys, importlib.metadata as md
          bad = []
          for line in open(sys.argv[1]):
              line = line.split("#")[0].strip()
              if not line:
                  continue
              name, want = line.split("==")
              got = md.version(name)
              print(f"{name}=={got}")
              if got.split("+")[0] != want:
                  bad.append(f"{name} {got}, pinned {want}")
          if bad:
              sys.exit("not the pinned stack: " + "; ".join(bad))
          import torch, torchvision, compressai, skimage
          torchvision.ops.nms(torch.zeros(1, 4), torch.zeros(1), 0.5)
          PY
      # Bootstrap rustup: the docker runner image ships no Rust.
      - name: Install Rust 1.98.0
        shell: bash
        run: |
          export PATH="$HOME/.cargo/bin:$PATH"
          command -v rustup >/dev/null 2>&1 || \
            curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
          rustup toolchain install 1.98.0 --profile minimal
          rustup default 1.98.0
          echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
      - name: Regenerate the int_hyper vectors (source photo → integer export → keyframe → clip)
        shell: bash
        working-directory: wai
        run: |
          tools/wai_sota_vectors.sh "$WAI_SOTA_INT_DIR"
          # Recorded so a result can be tied to the exact vectors it checked. It
          # identifies this run's vectors, not this machine's: two runs on this
          # runner wrote different cond.json and int_synth_sota.json, and other
          # files differ between machines (see the header). It is not a list any
          # other run must reproduce, here or elsewhere.
          (cd "$WAI_SOTA_INT_DIR" && sha256sum *)
      - name: Rust decodes every int_hyper vector byte-for-byte (missing input = failure), and the missing-input rule holds
        shell: bash
        working-directory: wai/wai-rs
        run: |
          set -o pipefail
          log="${RUNNER_TEMP:-/tmp}/sota-tests.log"
          # Without the switch every vector test would skip and report ok.
          [ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \
            || { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; }
          # --show-output rather than --nocapture: each test's output is printed
          # after the results, so it cannot split a `test X ... ok` line below.
          # require_vectors_rule runs its cases in child processes with the switch
          # and the input variables set per case, so the job's own env does not
          # reach them; it prints nothing on success, so it cannot trip the skip
          # backstop below.
          cargo test --release --no-default-features --features neural_int,video_int \
            --test int_hyper_synth --test int_video_sota_keyframe --test require_vectors_rule \
            -- --show-output 2>&1 | tee "$log"
          # A test that was renamed or compiled out would not fail — it would just
          # not run. Require each one by name.
          for t in rust_integer_synthesis_reproduces_the_export_hash \
                   rust_integer_hyper_decoder_reproduces_the_export_hash \
                   rust_decodes_the_full_conditional_bitstream_to_the_export_image \
                   integer_decode_keeps_the_float_codec_quality \
                   decodes_the_committed_tiny_conformance_vector \
                   sota_int_hyper_keyframe_clip_decodes_byte_for_byte \
                   without_the_switch_a_named_directory_that_does_not_exist_fails \
                   without_the_switch_a_named_path_that_is_a_file_fails_as_a_directory \
                   without_the_switch_a_named_empty_directory_fails \
                   without_the_switch_an_unset_or_empty_directory_variable_skips_on_an_absent_default \
                   without_the_switch_an_incomplete_default_directory_skips \
                   a_named_directory_holding_the_input_loads_it \
                   without_the_switch_a_path_variable_naming_nothing_fails \
                   without_the_switch_a_path_variable_naming_nothing_fails_when_another_is_unset \
                   without_the_switch_an_unset_path_variable_skips \
                   path_variables_naming_files_that_exist_load_them \
                   switch_values_that_leave_it_off_behave_as_unset \
                   with_the_switch_an_absent_default_fails; do
            grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; }
          done
          # A skipped test ALSO prints `test X ... ok`, so the name check alone
          # cannot tell a skip from a pass. Every skip writes a banner containing
          # "SKIPPED" and "NOT VERIFIED" (tests/support/require_vectors.rs); either
          # word anywhere in the log fails the job, whatever the switch says.
          if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then
            echo "::error::a vector test skipped instead of checking its input"
            exit 1
          fi
      # The same vectors decoded by the wasm32 build of wai-web (the browser
      # sink), in the same job so they share $WAI_SOTA_INT_DIR. The verify script
      # fails on a missing artifact here (WAI_REQUIRE_SOTA_VECTORS=1, and the
      # directory is named), and compares the wasm decode's RGB hash, for the
      # JSON and the compact binary model, with the native export's.
      - name: wasm32 toolchain (target, wasm-bindgen CLI pinned to wai-web's lock)
        shell: bash
        run: |
          rustup target add wasm32-unknown-unknown
          cargo install --locked wasm-bindgen-cli --version 0.2.127
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: wasm32 decode of the int_hyper vectors equals the native reference
        shell: bash
        working-directory: wai/wai-web
        run: |
          set -euo pipefail
          cargo build --locked --release --target wasm32-unknown-unknown
          wasm-bindgen --target nodejs --out-dir demo/pkg-node --out-name wai_web \
            target/wasm32-unknown-unknown/release/wai_web.wasm
          node demo/verify_int_hyper_wasm.mjs ./pkg-node/wai_web.js "$WAI_SOTA_INT_DIR"