open-standards/.github/workflows/wai-ops-gpu.yml
Transaction Science 6517401f01 CI: the gate that decides whether a workflow runs was queued behind the work it gates
Every path-gated workflow begins with one small job -- `does this change touch
it`, or `which workspaces this change touches` for the standards build -- that
all of its real jobs declare in `needs`. All 55 of those gate jobs ran on
`linux-arm64`.

That is the one label the fleet cannot spare. Sampled while 267 runs were
queued, every waiting run had the same shape:

    [waiting]  linux-arm64    does this change touch it
    [blocked]  linux          member · dataflow device, simulated
    [blocked]  macos          member · dataflow device absent

237 runs, each holding a gate in the longest queue, and every real job blocked
behind it. Not only `linux` jobs: `macos`, `linux-hw`, `linux-arm64-hw` and
`macos-m3-ultra` jobs were blocked too. The measured effect was `linux-arm64`
263 deep against `linux` **zero**, with jetson-hub's 20 cores taking no job for
a full hour -- starved not for want of work but because the work could not be
unblocked. A gate waiting on the resource it is rationing is head-of-line
blocking, and it makes every other runner in the fleet idle on its behalf.

The gates move to `linux`. Nothing else changes: the diff is 55 `runs-on` lines
and no others, and every touched file still parses.

Why `linux` and not elsewhere. The gate is not free -- it compares
`BASE...HEAD`, which needs the merge base, so `fetch-depth: 0` is a full clone
rather than a ten-second script. `macos` is served by hosts on a Mac Studio
sitting at load 185; `linux` is a separate 20-core box that was measurably
empty. This moves a serialisation point to the least contended label; it does
not remove it, and the gate is still a single point of failure for its
workflow. The structural fix is an early-exit first step inside each job, which
costs one checkout per leg and is a larger change.

Follow-up, deliberately not bundled: jetson-hub runs at capacity 2 with 15 GB of
RAM, so 55 clones still serialise two at a time. Raising capacity helps gates
and hurts the heavy emulated-architecture jobs that share the label, so it wants
measuring on its own.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-11-03 15:19:13 -05:00

337 lines
20 KiB
YAML

name: wai-ops-gpu integer operators on GPUs
# wai-ops/1-i32 on GPUs (wai/wai-ops-gpu): the integer operator set's graph operators
# as WGSL kernels through wgpu, exact at acc32 (one 32-bit word per value) and at acc64
# (64- and 128-bit arithmetic built from 32-bit words), and the integer decoders with
# their graphs on the device (wai-rs `ops_gpu`).
#
# - metal (the `macos` runner, whose host has a GPU): WGSL's integer semantics
# (wai/wgsl-conformance, determinism-tiers §5, Claiming the tier on a GPU), the
# committed vectors and SplitMix64 cases of every operation, with an edited kernel
# refused at discovery; the arithmetic probe on a
# million cases; every graph vector of wai/ops-conformance that the subset covers,
# at both widths, against the committed results and the interpreter; the shader
# mutants, every one of which the corpus must catch; every committed vector and
# refusal fixture of every capability that runs a wai-ops/1 graph, the jpegai/v1
# inputs included, against the decode on the CPU; the exact-upscale derivations and
# the face generator's frames, against the CPU's; and the corpus again in a
# headless browser's WebGPU, the kernels compiled by the browser's own WGSL
# compiler, with the semantics suite there and its vectors held to the reference
# compiled to wasm32, in the page and under Node. WAI_OPS_GPU_EXPECT declares the one adapter the host must offer, and
# WAI_OPS_GPU_REQUIRE makes an absent adapter a failure, never a skip; each step
# then requires the report lines with their counts, so a filtered or empty run
# fails too. The splat renderer's compositing (extensions/splat-exact.md) runs there
# too: every view of the splat corpus equal to the CPU's frame, and its mutants caught.
# - checks (`linux`, a container with no GPU): the crate and its tests build; the
# WGSL rules on naga's IR (tests/wgsl_rules.rs) and the committed semantics vectors
# against the reference, which need no adapter, pass; the device tests print SKIPPED
# and pass, and the same tests under
# WAI_OPS_GPU_REQUIRE=1 fail: an absent adapter is not a pass. The browser module
# builds for wasm32-unknown-unknown; the bench tool builds.
#
# The equality on other adapters (a discrete and an integrated GPU on Vulkan, an
# integrated GPU on Vulkan elsewhere, software Vulkan) was run by hand and is recorded in
# wai/extensions/determinism-tiers.md §6; this workflow runs no job on those hosts.
# One run per branch at a time; a push supersedes the run before it.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
on:
pull_request:
paths:
- 'wai/wai-ops-gpu/**'
- 'wai/wai-ops/**'
- 'wai/ops-conformance/**'
- 'wai/wgsl-conformance/**'
# The decoders whose graphs run on the device, and what they decode.
- 'wai/wai-rs/Cargo.toml'
- 'wai/wai-rs/Cargo.lock'
- 'wai/wai-rs/src/int_*.rs'
- 'wai/wai-rs/src/int_payload/**'
- 'wai/wai-rs/src/jpegai_*.rs'
- 'wai/wai-rs/tests/ops_gpu.rs'
# The exact-upscale derivations' graph on the device, under the corpus's priors.
- 'wai/wai-rs/src/exact_upscale.rs'
- 'wai/wai-rs/src/exact_upscale/**'
- 'wai/wai-rs/tests/exact_upscale_gpu.rs'
- 'wai/live-conformance/upscale/priors/**'
- 'wai/wai-rs/src/dred_exact/**'
- 'wai/wai-rs/tests/dred_exact_gpu.rs'
- 'wai/live-conformance/dred/**'
# The splat renderer's compositing on the device (extensions/splat-exact.md), and the
# scenes it is held to.
- 'wai/wai-rs/src/splat_exact.rs'
- 'wai/wai-rs/src/splat_exact/**'
- 'wai/wai-rs/tests/splat_exact_gpu.rs'
- 'wai/live-conformance/splat/**'
# The face generator's graph on the device, over the corpus's frames.
- 'wai/wai-rs/src/avatar_face.rs'
- 'wai/wai-rs/src/avatar_face/**'
- 'wai/wai-rs/tests/avatar_face_gpu.rs'
- 'wai/live-conformance/avatar-face/**'
- 'wai/wai-rs/tests/vectors/**'
- 'wai/int-payload-conformance/**'
- 'wai/neural-conformance/**'
- 'wai/tools/ops-gpu-bench/**'
- 'wai/tools/wai_jpegai_fetch_e2e_inputs.sh'
- 'wai/tools/jpegai-e2e-v1.sha256'
- 'rust-toolchain.toml'
- '.github/workflows/wai-ops-gpu.yml'
schedule:
- cron: '17 6 * * *'
jobs:
# Does this pull request's own diff touch the paths below? This forge matches a
# workflow's `paths:` against the push, so a merge of main into a branch fires the
# workflow whatever the pull request changes; every job here waits on this answer.
# A push to main, a schedule, or anything the script cannot determine answers
# `true` (scripts/pr-touches-paths.py).
relevant:
name: does this change touch it
runs-on: linux
outputs:
run: ${{ steps.gate.outputs.run }}
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- id: gate
run: |
set -euo pipefail
r=$(python3 scripts/pr-touches-paths.py .github/workflows/wai-ops-gpu.yml \
"${{ github.event.pull_request.base.sha }}" \
"${{ github.event.pull_request.head.sha }}")
echo "this change touches wai-ops-gpu: $r"
echo "run=$r" >> "$GITHUB_OUTPUT"
checks:
needs: relevant
if: needs.relevant.outputs.run == 'true'
name: builds · no adapter is not a pass (linux)
runs-on: linux
timeout-minutes: 60
steps:
- uses: actions/checkout@v5
- name: Install Rust 1.98.0 with the wasm32-unknown-unknown target
shell: bash
run: |
export PATH="$HOME/.cargo/bin:$PATH"
command -v rustup >/dev/null 2>&1 || \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
rustup toolchain install 1.98.0 --profile minimal
rustup default 1.98.0
rustup target add --toolchain 1.98.0 wasm32-unknown-unknown
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
# This container offers no adapter. The device tests must say so and pass; under
# WAI_OPS_GPU_REQUIRE=1 the same tests must fail. Were an adapter ever offered
# here, its results must then be equal instead.
- name: The crate's tests, and an absent adapter under WAI_OPS_GPU_REQUIRE
shell: bash
working-directory: wai/wai-ops-gpu
run: |
set -euo pipefail
log="$RUNNER_TEMP/ops-gpu.log"
rep="$RUNNER_TEMP/ops-gpu.report"
: > "$rep"
WAI_OPS_GPU_REPORT="$rep" cargo test --locked --release -- --nocapture --test-threads=1 2>&1 | tee "$log"
if grep -q '^ops-gpu corpus: key=' "$rep"; then
echo "this runner offered an adapter"
if grep -E '^ops-gpu corpus: key=' "$rep" | grep -vq ' mismatches=0 errors=0 '; then
echo "::error::an adapter here gave a result that is not the interpreter's"; exit 1
fi
else
grep -q '^ops-gpu corpus: SKIPPED' "$rep" || { echo "::error::the corpus test printed neither a result nor SKIPPED"; exit 1; }
if WAI_OPS_GPU_REQUIRE=1 cargo test --locked --release --test corpus --test arith --test mutants --test semantics > "$RUNNER_TEMP/require.log" 2>&1; then
echo "::error::with no adapter, the device tests passed under WAI_OPS_GPU_REQUIRE=1"; exit 1
fi
grep -q 'WAI_OPS_GPU_REQUIRE is set' "$RUNNER_TEMP/require.log" \
|| { echo "::error::the tests failed under WAI_OPS_GPU_REQUIRE=1, but not for the absent adapter"; cat "$RUNNER_TEMP/require.log"; exit 1; }
echo "no adapter: SKIPPED without WAI_OPS_GPU_REQUIRE, a failure with it"
fi
- name: The browser module builds for wasm32-unknown-unknown
working-directory: wai/wai-ops-gpu/browser
run: cargo build --locked --release --target wasm32-unknown-unknown
- name: wai-rs builds with ops_gpu, and its GPU test targets
working-directory: wai/wai-rs
run: |
set -euo pipefail
cargo test --locked --no-default-features --features ops_gpu,video_int,audio_int,splat_int,world_int --test ops_gpu --no-run
cargo test --locked --no-default-features --features ops_gpu,exact_upscale --test exact_upscale_gpu --no-run
cargo test --locked --no-default-features --features splat_exact,ops_gpu --test splat_exact_gpu --no-run
cargo test --locked --no-default-features --features ops_gpu,avatar_face --test avatar_face_gpu --no-run
- name: wai-rs builds the DRED profile's GPU test target
working-directory: wai/wai-rs
run: cargo test --locked --no-default-features --features dred_exact,ops_gpu --test dred_exact_gpu --no-run
- name: The bench tool builds
working-directory: wai/tools/ops-gpu-bench
run: cargo build --locked --release
metal:
needs: relevant
if: needs.relevant.outputs.run == 'true'
name: Metal GPU and browser WebGPU (macos)
runs-on: macos
timeout-minutes: 90
env:
WAI_OPS_GPU_EXPECT: gpu-metal-0
WAI_OPS_GPU_REQUIRE: '1'
steps:
- uses: actions/checkout@v5
# launchd's PATH omits cargo on the macOS host runner.
- name: Put cargo on PATH
shell: bash
run: echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
- name: The probe, the corpus at both widths, and the mutants, on the GPU
shell: bash
working-directory: wai/wai-ops-gpu
run: |
set -euo pipefail
log="$RUNNER_TEMP/ops-gpu.log"
export WAI_OPS_GPU_REPORT="$RUNNER_TEMP/ops-gpu.report"
: > "$WAI_OPS_GPU_REPORT"
nice -n 10 cargo test --locked --release -- --nocapture --test-threads=1 2>&1 | tee "$log"
k=gpu-metal-0
# Seventeen tests in six binaries: the library's four, arith's two, corpus's two,
# mutants' one, semantics' three and wgsl_rules' five, none filtered, ignored or
# failed.
ran=$(awk '/^test result: ok\. [0-9]+ passed; 0 failed; 0 ignored; 0 measured; 0 filtered out/ { n += $4 } END { print n + 0 }' "$log")
[ "$ran" = 17 ] || { echo "::error::$ran tests ran and passed, not 17"; exit 1; }
! grep -q '^test result: FAILED' "$log" || { echo "::error::a test failed"; exit 1; }
grep -qxF "ops-gpu adapters: as declared ($k)" "$WAI_OPS_GPU_REPORT" || { echo "::error::the adapters found are not $k"; exit 1; }
grep -qE "^ops-gpu arith: key=$k .* cases=[0-9]+ words=[0-9]+ equal$" "$WAI_OPS_GPU_REPORT" || { echo "::error::no probe result for $k"; exit 1; }
grep -qE "^ops-gpu corpus: key=$k .* acc32 vectors=[1-9][0-9]* decodes=[1-9][0-9]*; acc64 vectors=[1-9][0-9]* decodes=[1-9][0-9]*; .* mismatches=0 errors=0 " "$WAI_OPS_GPU_REPORT" \
|| { echo "::error::no equal corpus run for $k"; exit 1; }
grep -qxF "ops-gpu mutants: key=$k killed 18 of 18" "$WAI_OPS_GPU_REPORT" || { echo "::error::a mutant survived on $k, or the mutant test did not run"; exit 1; }
grep -qE "^ops-gpu semantics: key=$k .* committed=[1-9][0-9]* cases=[1-9][0-9]* guarded=[1-9][0-9]* differ=0 " "$WAI_OPS_GPU_REPORT" \
|| { echo "::error::no run of WGSL's integer semantics on $k with every guarded case equal"; exit 1; }
grep -qE "^ops-gpu semantics mutant: key=$k refused: the WGSL semantics suite: rem_i32 " "$WAI_OPS_GPU_REPORT" \
|| { echo "::error::the edited semantics kernel was not refused on $k"; exit 1; }
- name: The jpegai/v1 inputs
shell: bash
run: bash wai/tools/wai_jpegai_fetch_e2e_inputs.sh "$RUNNER_TEMP/jpegai"
- name: Every capability that runs a wai-ops/1 graph, with its graphs on the GPU
shell: bash
working-directory: wai/wai-rs
run: |
set -euo pipefail
log="$RUNNER_TEMP/ops-gpu-rs.log"
rep="$RUNNER_TEMP/ops-gpu-rs.report"
: > "$rep"
J="$RUNNER_TEMP/jpegai"
WAI_OPS_GPU_REPORT="$rep" WAI_JPEGAI_CODESTREAM="$J/test256.bin" WAI_JPEGAI_WEIGHTS_DIR="$J" WAI_JPEGAI_NORM="$J/norm.json" WAI_REQUIRE_SOTA_VECTORS=1 \
nice -n 10 cargo test --locked --release --no-default-features --features ops_gpu,video_int,audio_int,splat_int,world_int \
--test ops_gpu -- --nocapture 2>&1 | tee "$log"
grep -q '^test result: ok\. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out' "$log" || { echo "::error::the capability test did not run and pass"; exit 1; }
for cap in wai.neural.int_hyper wai.video.int_hyper wai.audio.int_codec wai.splat.int_codec wai.neural.int_synth \
wai.neural.int_mlicpp wai.world.model wai.image.jpegai; do
grep -qE "^ops-gpu capability: key=gpu-metal-0 .* cap=$cap cases=[1-9][0-9]* .* differ=0 errors=0$" "$rep" \
|| { echo "::error::$cap: no equal run on the GPU"; exit 1; }
done
# The exact-upscale derivations (extensions/exact-upscale.md §3.4): under every prior of the
# corpus, sr2 and sr4 give the same bytes with their graph on the device as on the CPU,
# every graph run on the device and compared with the interpreter.
- name: The exact-upscale derivations with their graph on the GPU
shell: bash
working-directory: wai/wai-rs
run: |
set -euo pipefail
log="$RUNNER_TEMP/exact-upscale-gpu.log"
rep="$RUNNER_TEMP/exact-upscale-gpu.report"
: > "$rep"
WAI_OPS_GPU_REPORT="$rep" nice -n 10 cargo test --locked --release --no-default-features --features ops_gpu,exact_upscale \
--test exact_upscale_gpu -- --nocapture 2>&1 | tee "$log"
grep -q '^test result: ok\. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out' "$log" || { echo "::error::the test did not run and pass"; exit 1; }
grep -qE "^exact-upscale gpu: key=gpu-metal-0 .* cases=[1-9][0-9]* .* cpu=0 .* differ=0 errors=0 equal$" "$rep" \
|| { echo "::error::no equal run on the GPU"; exit 1; }
# wai.audio.opus.dred.int (extensions/opus-dred.md §4.8): the corpus's payloads and its
# streams' reconstructions decoded with both graphs of dred-int/1 on the GPU at acc64, every graph also run on the
# interpreter and compared, and the decode's integers held to the CPU's. The weights file
# is fetched and checked against the draft's digest (live-conformance/dred/fetch.sh).
- name: The DRED profile, with its graphs on the GPU
shell: bash
run: |
set -euo pipefail
d="$RUNNER_TEMP/dred"
bash wai/live-conformance/dred/fetch.sh "$d"
log="$RUNNER_TEMP/dred-gpu.log"
rep="$RUNNER_TEMP/dred-gpu.report"
: > "$rep"
cd wai/wai-rs
WAI_OPS_GPU_REPORT="$rep" WAI_DRED_WEIGHTS="$d/dred_unquantized_weights.bin" \
nice -n 10 cargo test --locked --release --no-default-features --features dred_exact,ops_gpu \
--test dred_exact_gpu -- --nocapture 2>&1 | tee "$log"
grep -q '^test result: ok\. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out' "$log" || { echo "::error::the DRED GPU test did not run and pass"; exit 1; }
grep -qE "^dred-gpu: key=gpu-metal-0 payloads=[1-9][0-9]* streams=[1-9][0-9]* same=true graphs_on_gpu=[1-9][0-9]* .* on_cpu=0 compared=[1-9][0-9]* differed=0 errors=\[\]$" "$rep" \
|| { echo "::error::no equal DRED run on the GPU"; exit 1; }
# wai-splat-render:i1 (extensions/splat-exact.md §5.6-§5.7) with its compositing on the
# GPU: every canonical and own-pose view of every scene of live-conformance/splat/ equal to
# the CPU's frame byte for byte, and each edited kernel caught.
- name: The splat renderer's compositing on the GPU
shell: bash
working-directory: wai/wai-rs
run: |
set -euo pipefail
log="$RUNNER_TEMP/splat-gpu.log"
rep="$RUNNER_TEMP/splat-gpu.report"
: > "$rep"
WAI_OPS_GPU_REPORT="$rep" nice -n 10 cargo test --locked --release --no-default-features --features splat_exact,ops_gpu \
--test splat_exact_gpu -- --nocapture 2>&1 | tee "$log"
grep -q '^test result: ok\. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out' "$log" || { echo "::error::the splat GPU test did not run and pass"; exit 1; }
grep -qE '^splat-exact gpu: key=gpu-metal-0 views=[1-9][0-9]* differ=0 mutants_survived=0 ' "$rep" \
|| { echo "::error::no equal splat run on gpu-metal-0, or a mutant survived"; exit 1; }
# The face generator (extensions/avatars.md §7.8): every frame the avatar-face corpus
# names, from the seeded and the registered generator, is the CPU's frame and the corpus's
# digest with the generator's graph on the device, every graph run on the device and
# compared with the interpreter.
- name: The face generator's frames with its graph on the GPU
shell: bash
working-directory: wai/wai-rs
run: |
set -euo pipefail
log="$RUNNER_TEMP/avatar-face-gpu.log"
rep="$RUNNER_TEMP/avatar-face-gpu.report"
: > "$rep"
WAI_OPS_GPU_REPORT="$rep" nice -n 10 cargo test --locked --release --no-default-features --features ops_gpu,avatar_face \
--test avatar_face_gpu -- --nocapture 2>&1 | tee "$log"
grep -q '^test result: ok\. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out' "$log" || { echo "::error::the test did not run and pass"; exit 1; }
grep -qE "^face-gpu: key=gpu-metal-0 frames=[1-9][0-9]* same=true graphs_on_gpu=[1-9][0-9]* .* on_cpu=0 .* differed=0 errors=\[\] " "$rep" \
|| { echo "::error::no equal run on the GPU"; exit 1; }
- name: wasm-bindgen CLI 0.2.127, and the wasm32-unknown-unknown target
shell: bash
run: |
set -euo pipefail
rustup target add wasm32-unknown-unknown
cargo install --locked wasm-bindgen-cli --version 0.2.127
# Playwright and its headless browser go into the job's temporary directory, not
# the host's caches. `chromium` is Playwright's name for its engine.
- name: Playwright 1.60.0 and its headless browser
shell: bash
run: |
set -euo pipefail
pw="$RUNNER_TEMP/playwright"
mkdir -p "$pw"
npm install --prefix "$pw" --no-save --no-audit --no-fund [email protected]
PLAYWRIGHT_BROWSERS_PATH="$pw/browsers" "$pw/node_modules/.bin/playwright" install --only-shell chromium
echo "PLAYWRIGHT_HOST=$pw/" >> "$GITHUB_ENV"
echo "PLAYWRIGHT_BROWSERS_PATH=$pw/browsers" >> "$GITHUB_ENV"
- name: The corpus in a browser's WebGPU
shell: bash
working-directory: wai/wai-ops-gpu/browser
run: |
set -euo pipefail
nice -n 10 cargo build --locked --release --target wasm32-unknown-unknown
t="${CARGO_TARGET_DIR:-target}"
wasm-bindgen --target web --out-dir pkg --out-name wai_ops_gpu_browser "$t/wasm32-unknown-unknown/release/wai_ops_gpu_browser.wasm"
# The committed semantics vectors against the reference compiled to wasm32,
# under Node, with no GPU.
node semantics_reference.mjs
log="$RUNNER_TEMP/browser.log"
nice -n 10 node run.mjs 2>&1 | tee "$log"
grep -qE '^browser webgpu: key=gpu-webgpu-[0-9a-f]+ .* acc32 vectors=[1-9][0-9]* decodes=[1-9][0-9]*; acc64 vectors=[1-9][0-9]* decodes=[1-9][0-9]*; .* mismatches=0 errors=0$' "$log" \
|| { echo "::error::no equal corpus run on a GPU adapter in the browser"; exit 1; }
grep -qE '^wasm reference: target=wasm32 .* cases=[1-9][0-9]* equal$' "$log" \
|| { echo "::error::the semantics vectors are not the wasm32 reference's in the page"; exit 1; }
grep -qE '^browser webgpu semantics: key=gpu-webgpu-[0-9a-f]+ .* guarded=[1-9][0-9]* differ=0 ' "$log" \
|| { echo "::error::no run of WGSL's integer semantics on a GPU adapter in the browser with every guarded case equal"; exit 1; }