d7ab19292f
The disk pre-check read df under the leg's four directories (the workspace, the runner's temporary directory, TMPDIR and HOME). Three gaps remained. 1. On linux-aarch64 df reads the runner VM's disk, not the host's. Run in a job container started as the runner starts one (its image and its two cache mounts), the step's loop put all four directories on the VM's data disk (84 GiB free), while the host volume that holds the disk image had 50 GiB. The image is a 200 GiB file that was already allocated on the host but for 0.7 GiB, so a full host volume could take back at most that much. The step now also writes and syncs 32 MiB once on each filesystem it measures (dd conv=fsync, which the macOS host's dd and the job image's both take), and names a write refused for space, an I/O error, a read-only filesystem or a quota infrastructure. That also catches a disk out of inodes, which df's block count does not show. Another failed write is a warning, not a verdict. 2. The second determinism matrix, wai-conformance.yml's byte-exact job, runs on the same three labels with no disk check. It now has both steps, with the same floor. 3. The floor had no reading behind it. The step's loop, run on 2026-11-04 in each label's job environment: linux-x86_64 201 GiB (one host) and 304 GiB (the other); linux-aarch64 84 GiB; macos 78 GiB under the workspace and the runner's temporary directory, 85 GiB under TMPDIR and HOME. The floor stays at 4 GiB: it fails a leg only on a host within a few GiB of full, and the macOS runner's internal disk, where HOME and TMPDIR sit, has gone from 7 GiB free to 118 MiB within hours, with its jobs failing under HOME. The env comment says so. The class of the failure is still not in the status context: the forge reports `wai cross-architecture byte-equality / <label>` as failed for a full disk as for a divergence. The comment now says so, and why the check is not a job of its own per label: the `linux` label has two hosts, so such a job could measure another host's disk than the leg's. Co-Authored-By: Claude Opus 5.5 <[email protected]>
294 lines
15 KiB
YAML
294 lines
15 KiB
YAML
name: wai cross-architecture byte-equality
|
|
|
|
# The `replicate` determinism contract, checked on real silicon. WAI claims its
|
|
# integer-exact decode paths (rANS entropy coder, integer fixed-point synthesis)
|
|
# produce BYTE-IDENTICAL output on every machine. This matrix proves it: every
|
|
# OS/arch leg of the matrix below decodes the same fixtures and asserts the same golden
|
|
# bytes — rANS streams produced by CompressAI's upstream encoder, and an
|
|
# int_synth output hash produced on the reference machine (macOS arm64). One
|
|
# diverging byte anywhere fails the suite.
|
|
#
|
|
# Coverage is still-image AND temporal, classical AND learned: the rANS coder,
|
|
# integer fixed-point synthesis, the JPEG-AI entropy stage, the inter-frame video
|
|
# decoder (`wai.video.int_motion`), and the neural-keyframe variant
|
|
# (`wai.video.int_hyper`) whose I-frame comes out of a learned conditional codec,
|
|
# and the MLIC++-topology codec (`wai.neural.int_mlicpp`), whose entropy network runs
|
|
# before every checkerboard pass and must give every platform the encoder's scale
|
|
# bucket for every element. The learned integer audio and Gaussian-splat attribute codecs
|
|
# (`wai.audio.int_codec`, `wai.splat.int_codec`) decode their committed WIA1 and WIS1
|
|
# payloads to their PCM and attribute digests. The binary payloads SPEC §5 pins (WIV1,
|
|
# WIH1, WIA1, WIS1) are parsed
|
|
# with the wire parser a sink runs and decoded to the same digests, and the staged-delivery
|
|
# residual layers (WIR1) of the staged corpus are applied stage by stage to pinned digests.
|
|
# Video is the strictest leg — inter-frame decode is a
|
|
# feedback loop, so one divergent half-pel rounding does not stay one byte, it
|
|
# predicts the next frame and compounds for the rest of the clip.
|
|
#
|
|
# The conformance crate (`wai/byte-exact-conformance`) #[path]-includes the
|
|
# pure-integer sources from wai-rs — no system codec libraries, no ONNX runtime
|
|
# — so every leg runs in about a minute.
|
|
|
|
# One run per branch at a time. A push supersedes the run before it.
|
|
#
|
|
# There is one `macos` runner and it is the dev Mac, so every macOS job in this
|
|
# repo queues on it. With no concurrency group, a session pushing four versions
|
|
# of a branch queued four complete sets of runs and every one of them had to
|
|
# drain: on 2026-10-05 a PR's 21 contexts settled none in 70 minutes behind a
|
|
# backlog of 25 runs, and the queue grew faster than it cleared. Only the newest
|
|
# commit's verdict is wanted, so the older runs are cancelled.
|
|
#
|
|
# Verified on this forge (Forgejo 16.0.5) in `ci-test` before being added here:
|
|
# the key parses, and a newer push moves an older run to `cancelled` even while
|
|
# it is still queued, which is the case that matters.
|
|
#
|
|
# `cancel-in-progress: true` also applies to `main`. That is deliberate and
|
|
# safe here: main advances by merge commits whose tree contains the commits
|
|
# before them, so the newer run covers the older one's changes.
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'wai/byte-exact-conformance/**'
|
|
- 'wai/wai-rs/src/neural/int_synth.rs'
|
|
- 'wai/wai-rs/src/neural/jpegai_mans.rs'
|
|
# Every source the crate #[path]-includes must be able to fire this
|
|
# workflow, or a change can break cross-arch equality and merge green.
|
|
- 'wai/wai-rs/src/int_synth.rs'
|
|
- 'wai/wai-rs/src/int_entropy.rs'
|
|
- 'wai/wai-rs/src/jpegai_mans.rs'
|
|
- 'wai/wai-rs/src/rans.rs'
|
|
- 'wai/wai-rs/src/int_video.rs'
|
|
- 'wai/wai-rs/src/int_transform.rs'
|
|
- 'wai/wai-rs/src/int_hyper_synth.rs'
|
|
- 'wai/wai-rs/src/int_mlicpp.rs'
|
|
- 'wai/wai-rs/src/int_audio.rs'
|
|
- 'wai/wai-rs/src/int_splat.rs'
|
|
- 'wai/wai-rs/src/int_payload/wire.rs'
|
|
- 'wai/wai-rs/src/staged/predict.rs'
|
|
- 'wai/wai-rs/src/staged/wir1.rs'
|
|
- 'wai/wai-rs/src/staged/wir1_arith.rs'
|
|
- 'wai/staged-conformance/**'
|
|
# The fast kernels int_synth, int_transform and int_hyper_synth dispatch to
|
|
# by default: the goldens here decode through them.
|
|
- 'wai/wai-rs/src/int_kernels.rs'
|
|
# The integer operator set and the bridge to it: the goldens decode int_synth,
|
|
# the audio and splat codecs, the conditional codec (WIH1, the WIV1 keyframe)
|
|
# and the MLIC++-topology codec through their lowerings too.
|
|
- 'wai/wai-rs/src/int_ops.rs'
|
|
- 'wai/wai-ops/**'
|
|
- 'wai/wai-rs/tests/vectors/int_video/**'
|
|
- 'wai/wai-rs/tests/vectors/int_hyper/**'
|
|
- 'wai/wai-rs/tests/vectors/int_mlicpp/**'
|
|
- 'wai/wai-rs/tests/vectors/int_audio/**'
|
|
- 'wai/wai-rs/tests/vectors/int_splat/**'
|
|
- '.github/workflows/wai-byte-equality.yml'
|
|
push:
|
|
branches:
|
|
- main
|
|
paths:
|
|
- 'wai/byte-exact-conformance/**'
|
|
- 'wai/wai-rs/src/neural/int_synth.rs'
|
|
- 'wai/wai-rs/src/neural/jpegai_mans.rs'
|
|
# Every source the crate #[path]-includes must be able to fire this
|
|
# workflow, or a change can break cross-arch equality and merge green.
|
|
- 'wai/wai-rs/src/int_synth.rs'
|
|
- 'wai/wai-rs/src/int_entropy.rs'
|
|
- 'wai/wai-rs/src/jpegai_mans.rs'
|
|
- 'wai/wai-rs/src/rans.rs'
|
|
- 'wai/wai-rs/src/int_video.rs'
|
|
- 'wai/wai-rs/src/int_transform.rs'
|
|
- 'wai/wai-rs/src/int_hyper_synth.rs'
|
|
- 'wai/wai-rs/src/int_mlicpp.rs'
|
|
- 'wai/wai-rs/src/int_audio.rs'
|
|
- 'wai/wai-rs/src/int_splat.rs'
|
|
- 'wai/wai-rs/src/int_payload/wire.rs'
|
|
- 'wai/wai-rs/src/staged/predict.rs'
|
|
- 'wai/wai-rs/src/staged/wir1.rs'
|
|
- 'wai/wai-rs/src/staged/wir1_arith.rs'
|
|
- 'wai/staged-conformance/**'
|
|
# The fast kernels int_synth, int_transform and int_hyper_synth dispatch to
|
|
# by default: the goldens here decode through them.
|
|
- 'wai/wai-rs/src/int_kernels.rs'
|
|
# The integer operator set and the bridge to it: the goldens decode int_synth,
|
|
# the audio and splat codecs, the conditional codec (WIH1, the WIV1 keyframe)
|
|
# and the MLIC++-topology codec through their lowerings too.
|
|
- 'wai/wai-rs/src/int_ops.rs'
|
|
- 'wai/wai-ops/**'
|
|
- 'wai/wai-rs/tests/vectors/int_video/**'
|
|
- 'wai/wai-rs/tests/vectors/int_hyper/**'
|
|
- 'wai/wai-rs/tests/vectors/int_mlicpp/**'
|
|
- 'wai/wai-rs/tests/vectors/int_audio/**'
|
|
- 'wai/wai-rs/tests/vectors/int_splat/**'
|
|
- '.github/workflows/wai-byte-equality.yml'
|
|
|
|
jobs:
|
|
# Does this pull request's own diff touch the paths below? This forge matches a
|
|
# workflow's `paths:` against the push, so a merge of main into a branch fires the
|
|
# workflow whatever the pull request changes; every job here waits on this answer.
|
|
# A push to main, a schedule, or anything the script cannot determine answers
|
|
# `true` (scripts/pr-touches-paths.py).
|
|
relevant:
|
|
name: does this change touch it
|
|
runs-on: linux
|
|
outputs:
|
|
run: ${{ steps.gate.outputs.run }}
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
with:
|
|
fetch-depth: 0
|
|
- id: gate
|
|
run: |
|
|
set -euo pipefail
|
|
r=$(python3 scripts/pr-touches-paths.py .github/workflows/wai-byte-equality.yml \
|
|
"${{ github.event.pull_request.base.sha }}" \
|
|
"${{ github.event.pull_request.head.sha }}")
|
|
echo "this change touches wai-byte-equality: $r"
|
|
echo "run=$r" >> "$GITHUB_OUTPUT"
|
|
|
|
byte-equality:
|
|
needs: relevant
|
|
if: needs.relevant.outputs.run == 'true'
|
|
name: ${{ matrix.label }}
|
|
runs-on: ${{ matrix.os }}
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
# Self-hosted fleet labels, not hosted images. macos-x86_64 is
|
|
# dropped: no Intel-Mac runner exists in the fleet, and by the same
|
|
# argument the cargo-classes matrix already makes, it adds no arch
|
|
# coverage — x86_64 is held by linux-x86_64, arm64 by linux-aarch64
|
|
# and the macos-arm64 reference.
|
|
# Windows x86_64 leg removed 2026-09-30: runner offline. Restore by
|
|
# re-adding `- { os: windows, label: windows-x86_64 }` here once the
|
|
# runner is back online.
|
|
# The two disk steps below are bash, and skip on Windows.
|
|
include:
|
|
- { os: linux, label: linux-x86_64 }
|
|
- { os: linux-arm64, label: linux-aarch64 }
|
|
- { os: macos, label: macos-arm64 (reference) }
|
|
env:
|
|
# The floor of the disk check below, in GiB. The leg writes under 1 GiB (a
|
|
# shallow checkout of about 0.3 GiB plus its pack, and a release target under
|
|
# 0.1 GiB, measured 2026-11-03), and a cold toolchain install about 1 GiB more
|
|
# under HOME; the rest is room for the jobs that share the disk. Measured with the
|
|
# step's own loop on 2026-11-04, every label had 50 GiB or more free under all
|
|
# four directories, so the floor fails a leg only on a host within a few GiB of
|
|
# full. The macOS runner's HOME and TMPDIR sit on its internal disk, which has
|
|
# gone from 7 GiB free to 118 MiB within hours, and at that level its jobs failed
|
|
# writing under HOME.
|
|
DISK_FLOOR_GIB: '4'
|
|
steps:
|
|
# Infrastructure first, before the checkout writes anything. Main's run 17676
|
|
# failed on both arm64 legs with "No space left on device", linux-aarch64 in
|
|
# `git fetch` and macOS in the toolchain's temporary files: the hosts' disks were
|
|
# full and no fixture was decoded, yet the matrix read as a byte-equality failure.
|
|
# This step measures the free space under every directory the leg writes to (the
|
|
# workspace, the runner's temporary directory, TMPDIR, and HOME, where the
|
|
# toolchain lives), and writes and syncs 32 MiB once on each filesystem, which a
|
|
# disk out of inodes or remounted read-only fails though df shows room. Below the
|
|
# floor, or on a write refused for space, an I/O error or a read-only filesystem,
|
|
# it fails, and its name and its error say infrastructure.
|
|
#
|
|
# On linux-aarch64 the four directories are on the runner VM's data disk, so df
|
|
# reads the guest's free space, not the host volume's. The disk is an image file
|
|
# on that volume, and on 2026-11-04 all but 0.7 of its 200 GiB were already
|
|
# allocated there, so a full host volume could take back at most that much; the
|
|
# write probe catches the rest.
|
|
#
|
|
# The forge still reports the leg's own status context as failed (`wai
|
|
# cross-architecture byte-equality / <label>`), as for a divergence; the class is
|
|
# in the failed step's name and its error. A job of its own per label would put it
|
|
# in the context, but the `linux` label has two hosts, so that job could measure
|
|
# another host's disk than the leg's.
|
|
- name: "infrastructure: this runner's disk has room for the leg"
|
|
id: disk
|
|
if: matrix.os != 'windows'
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
short=0; seen=""
|
|
for d in "${GITHUB_WORKSPACE:-$PWD}" "${RUNNER_TEMP:-/tmp}" "${TMPDIR:-/tmp}" "$HOME"; do
|
|
p=$d; while [ ! -e "$p" ]; do p=$(dirname "$p"); done
|
|
set -- $(df -Pk "$p" | awk 'NR == 2 {print $4, $6}')
|
|
echo "$d: $(($1 / 1048576)) GiB free on $2"
|
|
if [ "$1" -lt $((DISK_FLOOR_GIB * 1048576)) ]; then
|
|
echo "::error::infrastructure: ${{ matrix.label }}: $d is on $2, which has $(($1 / 1024)) MiB free, below the floor of $DISK_FLOOR_GIB GiB. The runner host's disk is full: no fixture was decoded, so this is not a byte-equality failure."
|
|
short=1; continue
|
|
fi
|
|
case " $seen " in *" $2 "*) continue ;; esac
|
|
seen="$seen $2"; f="$p/.wai-disk-probe.$$"
|
|
if ! err=$(dd if=/dev/zero of="$f" bs=1048576 count=32 conv=fsync 2>&1); then
|
|
case "$err" in
|
|
*"No space left"*|*"Input/output error"*|*"Read-only file system"*|*"quota"*)
|
|
echo "::error::infrastructure: ${{ matrix.label }}: a 32 MiB write under $p, on $2, failed: $(printf '%s' "$err" | head -n 1). The runner host's disk cannot take the leg's writes: no fixture was decoded, so this is not a byte-equality failure."
|
|
short=1 ;;
|
|
*) echo "::warning::a 32 MiB write under $p failed for another reason: $(printf '%s' "$err" | head -n 1)" ;;
|
|
esac
|
|
fi
|
|
rm -f "$f"
|
|
done
|
|
exit $short
|
|
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
# unix: bootstrap rustup (docker image ships no Rust; macOS host has it but
|
|
# launchd's PATH omits ~/.cargo/bin). Windows is a host executor with rustup
|
|
# already present, and must use pwsh — its `bash` is the WSL stub.
|
|
- name: Install Rust 1.98.0 (unix)
|
|
if: matrix.os != 'windows'
|
|
shell: bash
|
|
run: |
|
|
export PATH="$HOME/.cargo/bin:$PATH"
|
|
command -v rustup >/dev/null 2>&1 || \
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
|
|
rustup toolchain install 1.98.0 --profile minimal
|
|
rustup default 1.98.0
|
|
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
|
- name: Install Rust 1.98.0 (Windows)
|
|
if: matrix.os == 'windows'
|
|
shell: pwsh
|
|
run: |
|
|
rustup toolchain install 1.98.0 --profile minimal
|
|
rustup default 1.98.0
|
|
|
|
- name: Decode fixtures, assert golden bytes
|
|
working-directory: wai/byte-exact-conformance
|
|
run: cargo test --release -- --nocapture
|
|
|
|
# A disk can also fill while the leg runs, from the jobs that share it (three
|
|
# conformance jobs at once filled the arm64 VM's data disk mid-link). When a step
|
|
# after the check failed, the same measurement and write are made again: below the
|
|
# floor, or on a write refused for space, the failure is named infrastructure; with
|
|
# room, it stands as it is.
|
|
- name: "infrastructure: was the disk full when the leg failed?"
|
|
if: failure() && steps.disk.outcome == 'success'
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
short=0; seen=""
|
|
for d in "${GITHUB_WORKSPACE:-$PWD}" "${RUNNER_TEMP:-/tmp}" "${TMPDIR:-/tmp}" "$HOME"; do
|
|
p=$d; while [ ! -e "$p" ]; do p=$(dirname "$p"); done
|
|
set -- $(df -Pk "$p" | awk 'NR == 2 {print $4, $6}')
|
|
echo "$d: $(($1 / 1048576)) GiB free on $2"
|
|
if [ "$1" -lt $((DISK_FLOOR_GIB * 1048576)) ]; then
|
|
echo "::error::infrastructure: ${{ matrix.label }}: $d is on $2, which has $(($1 / 1024)) MiB free, below the floor of $DISK_FLOOR_GIB GiB. The disk filled while the leg ran: read the failure above as the runner host's disk, not as a byte-equality failure."
|
|
short=1; continue
|
|
fi
|
|
case " $seen " in *" $2 "*) continue ;; esac
|
|
seen="$seen $2"; f="$p/.wai-disk-probe.$$"
|
|
if ! err=$(dd if=/dev/zero of="$f" bs=1048576 count=32 conv=fsync 2>&1); then
|
|
case "$err" in
|
|
*"No space left"*|*"Input/output error"*|*"Read-only file system"*|*"quota"*)
|
|
echo "::error::infrastructure: ${{ matrix.label }}: a 32 MiB write under $p, on $2, failed: $(printf '%s' "$err" | head -n 1). The disk filled while the leg ran: read the failure above as the runner host's disk, not as a byte-equality failure."
|
|
short=1 ;;
|
|
*) echo "::warning::a 32 MiB write under $p failed for another reason: $(printf '%s' "$err" | head -n 1)" ;;
|
|
esac
|
|
fi
|
|
rm -f "$f"
|
|
done
|
|
[ "$short" = 1 ] || echo "Every disk this leg writes to has room: the failure above is not the disk."
|
|
exit $short
|