6517401f01
Every path-gated workflow begins with one small job -- `does this change touch
it`, or `which workspaces this change touches` for the standards build -- that
all of its real jobs declare in `needs`. All 55 of those gate jobs ran on
`linux-arm64`.
That is the one label the fleet cannot spare. Sampled while 267 runs were
queued, every waiting run had the same shape:
[waiting] linux-arm64 does this change touch it
[blocked] linux member · dataflow device, simulated
[blocked] macos member · dataflow device absent
237 runs, each holding a gate in the longest queue, and every real job blocked
behind it. Not only `linux` jobs: `macos`, `linux-hw`, `linux-arm64-hw` and
`macos-m3-ultra` jobs were blocked too. The measured effect was `linux-arm64`
263 deep against `linux` **zero**, with jetson-hub's 20 cores taking no job for
a full hour -- starved not for want of work but because the work could not be
unblocked. A gate waiting on the resource it is rationing is head-of-line
blocking, and it makes every other runner in the fleet idle on its behalf.
The gates move to `linux`. Nothing else changes: the diff is 55 `runs-on` lines
and no others, and every touched file still parses.
Why `linux` and not elsewhere. The gate is not free -- it compares
`BASE...HEAD`, which needs the merge base, so `fetch-depth: 0` is a full clone
rather than a ten-second script. `macos` is served by hosts on a Mac Studio
sitting at load 185; `linux` is a separate 20-core box that was measurably
empty. This moves a serialisation point to the least contended label; it does
not remove it, and the gate is still a single point of failure for its
workflow. The structural fix is an early-exit first step inside each job, which
costs one checkout per leg and is a larger change.
Follow-up, deliberately not bundled: jetson-hub runs at capacity 2 with 15 GB of
RAM, so 55 clones still serialise two at a time. Raising capacity helps gates
and hurts the heavy emulated-architecture jobs that share the label, so it wants
measuring on its own.
Co-Authored-By: Claude Opus 5 <[email protected]>
338 lines
19 KiB
YAML
338 lines
19 KiB
YAML
name: WAI JPEG-AI whole decode (pinned inputs, required)
|
||
|
||
# WAI states that the exact decoder, `wai::jpegai_decode::decode`, decodes an
|
||
# ISO/IEC 6048 JPEG-AI codestream to RGB deterministically, so a hard binding
|
||
# taken over one decode verifies a second one, and that on one reference
|
||
# codestream its output stays within max|Δ| ≤ 2 of the reference decoder's
|
||
# (recorded max|Δ| = 1; wai/SPEC.md §5, the wai.image.jpegai row).
|
||
# Across machines the registry states less: it classes wai.image.jpegai as
|
||
# NeuralEntropyExact, a conservative class that guarantees only the entropy decode
|
||
# (the decoded symbols and their entropy parameters) to be byte-identical on every
|
||
# machine. That the exact decoder's pinned output is the same on other machines is
|
||
# a measurement, recorded in wai/extensions/determinism-tiers.md §6. The tests that
|
||
# carry these statements (tests/jpegai_e2e_full.rs,
|
||
# tests/jpegai_reconstruction_binding.rs) read their inputs from outside the tree
|
||
# and used to report PASS without them.
|
||
#
|
||
# This job fetches those inputs from WAI's versioned models root (jpegai/v1/),
|
||
# checks every file against the SHA-256 manifest committed in
|
||
# wai/tools/jpegai-e2e-v1.sha256, and runs both tests with
|
||
# WAI_REQUIRE_SOTA_VECTORS=1, so a missing input is a failure. What passes here:
|
||
# - the whole chain, codestream → RGB, through the one public `decode()` call,
|
||
# lands within max|Δ| ≤ 2 of the reference decoder's output, and at most 5%
|
||
# of samples differ from it at all (the test's bar);
|
||
# - two decodes in one process are byte-identical and the reconstruction binding
|
||
# verifies the second and rejects a one-byte change;
|
||
# - the bundle's bounds certificate (wai/extensions/integer-operators.md §8.5,
|
||
# §13.6: the two scale graphs and the four graphs of its other integer
|
||
# networks) is byte for byte the one committed under the set's digest in
|
||
# wai/ops-conformance/certificates/, which is the listing digest the set-form
|
||
# pin below covers, from both certifiers (wai-verify and the independent
|
||
# tools/wai_ops_check.py); its verdict meets acc64, and the scale path's graphs
|
||
# meet acc32 (§11, role_profiles);
|
||
# - the decoded pixels hash to WAI_JPEGAI_EXPECT_SHA256 below: the value measured
|
||
# from these inputs on an Apple Silicon Mac (aarch64 macOS, native) and
|
||
# reproduced by an x86_64 Linux build (glibc, run under QEMU user-mode
|
||
# emulation), and the value the four-platform comparison in determinism-tiers
|
||
# §6 (macOS arm64, wasm32-wasip2, Linux arm64, Linux x86_64 under user-mode
|
||
# emulation) found on every platform. This runner is x86_64 hardware, so the
|
||
# job adds a decode on one more machine to that measurement. A match is a
|
||
# measurement on this runner, not something the registry class promises; a
|
||
# mismatch is a departure from the measured agreement.
|
||
# The pinned decode is max|Δ| = 1 from the reference on 4.446% of samples
|
||
# (on the Mac and the emulated x86_64 build), so a hash match also confirms
|
||
# that recorded max|Δ| = 1 for this codestream; the test's own tolerance
|
||
# above is looser than that.
|
||
#
|
||
# What this job does not cover:
|
||
# - Other codestreams: it decodes one 256×256 test codestream, so the
|
||
# max|Δ| ≤ 2 bound and the determinism statement are checked on that input
|
||
# only.
|
||
# - The extraction: the inputs are an extraction of the JPEG-AI
|
||
# reference-software assets (tools/wai_jpegai_build_e2e_inputs.py), which are
|
||
# not in the repository, so the extraction is not re-run here; the manifest
|
||
# pins its published bytes.
|
||
# - The ONNX-runtime decode (`neural::jpegai`, tests/jpegai_live.rs), which needs
|
||
# those local assets and is not run in CI. It also needs the `neural` feature,
|
||
# whose prebuilt ONNX Runtime does not link on the `linux` runner's image
|
||
# (glibc 2.35; the library references glibc 2.38 symbols), which is where
|
||
# wai-neural-e2e.yml's runs fail once they reach the Rust build. The
|
||
# features built here (neural_int, provenance) link no prebuilt native
|
||
# library.
|
||
# The job depends on the models root being reachable; an outage fails the job.
|
||
#
|
||
# The path filters list the source files the two tests' decode path imports,
|
||
# among them src/float_repro.rs (the exact decoder's `exp` and its roundings to
|
||
# an integer) and src/int_kernels.rs (the fast convolution and parallel-map
|
||
# kernels the integer networks run through). An entry for a file the tree does
|
||
# not have matches nothing until the file exists. A module the decode path
|
||
# starts importing belongs here too; otherwise a change to that module alone is
|
||
# first checked by the weekly run.
|
||
|
||
# One run per branch at a time. A push supersedes the run before it.
|
||
#
|
||
# There is one `macos` runner and it is the dev Mac, so every macOS job in this
|
||
# repo queues on it. With no concurrency group, a session pushing four versions
|
||
# of a branch queued four complete sets of runs and every one of them had to
|
||
# drain: on 2026-10-05 a PR's 21 contexts settled none in 70 minutes behind a
|
||
# backlog of 25 runs, and the queue grew faster than it cleared. Only the newest
|
||
# commit's verdict is wanted, so the older runs are cancelled.
|
||
#
|
||
# Verified on this forge (Forgejo 16.0.5) in `ci-test` before being added here:
|
||
# the key parses, and a newer push moves an older run to `cancelled` even while
|
||
# it is still queued, which is the case that matters.
|
||
#
|
||
# `cancel-in-progress: true` also applies to `main`. That is deliberate and
|
||
# safe here: main advances by merge commits whose tree contains the commits
|
||
# before them, so the newer run covers the older one's changes.
|
||
concurrency:
|
||
group: ${{ github.workflow }}-${{ github.ref }}
|
||
cancel-in-progress: true
|
||
|
||
on:
|
||
pull_request:
|
||
paths:
|
||
- 'wai/wai-rs/src/jpegai_*.rs'
|
||
- 'wai/wai-rs/src/reconstruction_binding.rs'
|
||
# The set-form pin over the exact bundle (tests/jpegai_e2e_full.rs) verifies
|
||
# through the pin module.
|
||
- 'wai/wai-rs/src/pin.rs'
|
||
- 'wai/wai-rs/src/container.rs'
|
||
- 'wai/wai-rs/src/int_transformer.rs'
|
||
- 'wai/wai-rs/src/int_hyper_synth.rs'
|
||
- 'wai/wai-rs/src/int_kernels.rs'
|
||
- 'wai/wai-rs/src/float_repro.rs'
|
||
- 'wai/wai-rs/src/int_entropy.rs'
|
||
- 'wai/wai-rs/src/int_transform.rs'
|
||
- 'wai/wai-rs/src/rans.rs'
|
||
- 'wai/wai-rs/Cargo.lock'
|
||
- 'wai/wai-rs/tests/jpegai_e2e_full.rs'
|
||
- 'wai/wai-rs/tests/jpegai_reconstruction_binding.rs'
|
||
- 'wai/wai-rs/tests/jpegai_cross_platform.rs'
|
||
- 'wai/wai-rs/tests/jpegai_ops.rs'
|
||
- 'wai/wai-rs/tests/jpegai_nets_ops.rs'
|
||
- 'wai/wai-rs/tests/jpegai_memory.rs'
|
||
- 'wai/wai-rs/src/int_set_cache.rs'
|
||
- 'wai/wai-rs/src/int_ops.rs'
|
||
- 'wai/ops-conformance/sets/jpegai/**'
|
||
- 'wai/wai-rs/tests/support/**'
|
||
- 'wai/tools/wai_jpegai_fetch_e2e_inputs.sh'
|
||
- 'wai/tools/jpegai-e2e-v1.sha256'
|
||
# The bounds certificate of the bundle's scale path (the step after the fetch):
|
||
# the certifier, and the certificate committed by digest.
|
||
- 'wai/wai-ops/src/**'
|
||
- 'wai/wai-ops/tests/goldens.rs'
|
||
- 'wai/wai-verify/Cargo.toml'
|
||
- 'wai/wai-verify/Cargo.lock'
|
||
- 'wai/wai-verify/src/bin/**'
|
||
- 'wai/ops-conformance/certificates/external.json'
|
||
- 'wai/ops-conformance/certificates/wai.image.jpegai/3e243dc11566922f5cb7f4c628afd440b18d56a08b99b07cc33d5c8522aeec37.json'
|
||
- 'wai/tools/wai_ops_check.py'
|
||
- '.github/workflows/wai-jpegai-e2e.yml'
|
||
# Weekly, independent of changes: the inputs live outside the repository, and
|
||
# their disappearing should surface here, not at the next unrelated PR.
|
||
schedule:
|
||
- cron: '41 6 * * *'
|
||
|
||
jobs:
|
||
# Does this pull request's own diff touch the paths below? This forge matches a
|
||
# workflow's `paths:` against the push, so a merge of main into a branch fires the
|
||
# workflow whatever the pull request changes; every job here waits on this answer.
|
||
# A push to main, a schedule, or anything the script cannot determine answers
|
||
# `true` (scripts/pr-touches-paths.py).
|
||
relevant:
|
||
name: does this change touch it
|
||
runs-on: linux
|
||
outputs:
|
||
run: ${{ steps.gate.outputs.run }}
|
||
steps:
|
||
- uses: actions/checkout@v5
|
||
with:
|
||
fetch-depth: 0
|
||
- id: gate
|
||
run: |
|
||
set -euo pipefail
|
||
r=$(python3 scripts/pr-touches-paths.py .github/workflows/wai-jpegai-e2e.yml \
|
||
"${{ github.event.pull_request.base.sha }}" \
|
||
"${{ github.event.pull_request.head.sha }}")
|
||
echo "this change touches wai-jpegai-e2e: $r"
|
||
echo "run=$r" >> "$GITHUB_OUTPUT"
|
||
|
||
jpegai-e2e:
|
||
needs: relevant
|
||
if: needs.relevant.outputs.run == 'true'
|
||
name: fetch pinned JPEG-AI inputs → whole decode + pixel hash, required
|
||
runs-on: linux
|
||
timeout-minutes: 60
|
||
env:
|
||
WAI_REQUIRE_SOTA_VECTORS: '1'
|
||
WAI_JPEGAI_DIR: ${{ github.workspace }}/jpegai_e2e
|
||
# sha256 of the decoded RGB8 pixels for the jpegai/v1 inputs (see header).
|
||
WAI_JPEGAI_EXPECT_SHA256: dbc494a5a670914edf55c07c48afc0dd348cd05d3a82fff293bf384b2bb69ea3
|
||
steps:
|
||
- uses: actions/checkout@v5
|
||
# Bootstrap rustup: the docker runner image ships no Rust.
|
||
- name: Install Rust 1.98.0
|
||
shell: bash
|
||
run: |
|
||
export PATH="$HOME/.cargo/bin:$PATH"
|
||
command -v rustup >/dev/null 2>&1 || \
|
||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
|
||
rustup toolchain install 1.98.0 --profile minimal
|
||
rustup default 1.98.0
|
||
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
||
- name: Fetch the JPEG-AI inputs and check each against the pinned manifest
|
||
shell: bash
|
||
working-directory: wai
|
||
run: tools/wai_jpegai_fetch_e2e_inputs.sh "$WAI_JPEGAI_DIR"
|
||
# The scale path's bounds certificate (integer-operators §8.5, §13.6.1). The
|
||
# bundle is not committed, so its certificate is, under the set's digest (the
|
||
# listing of norm.json and weights/*, as the set-form pin covers it), and
|
||
# listed in wai/ops-conformance/certificates/external.json.
|
||
- name: The bundle's bounds certificate is the committed one; it meets acc64, and its scale path acc32
|
||
shell: bash
|
||
working-directory: wai
|
||
run: |
|
||
set -euo pipefail
|
||
(cd wai-verify && cargo build --locked --release --features cli --bin wai-verify)
|
||
D="$WAI_JPEGAI_DIR"
|
||
M=(--member "norm.json=$D/norm.json")
|
||
for n in hs_y hs_uv hyp_y hyp_uv synth_y synth_uv mcm0_y mcm1_y mcm2_y mcm3_y icci; do
|
||
M+=(--member "weights/$n.json=$D/$n.json")
|
||
done
|
||
wai-verify/target/release/wai-verify bounds --capability wai.image.jpegai "${M[@]}" \
|
||
--profile acc64 --role-profile hs_y=acc32 --role-profile hs_uv=acc32 \
|
||
--out "${RUNNER_TEMP:-/tmp}/jpegai-bounds.json" \
|
||
--check ops-conformance/certificates/wai.image.jpegai/3e243dc11566922f5cb7f4c628afd440b18d56a08b99b07cc33d5c8522aeec37.json
|
||
# The same certificate from the independent certifier (Python standard library
|
||
# only, sharing no code with wai-ops).
|
||
- uses: actions/setup-python@v5
|
||
with:
|
||
python-version: '3.11'
|
||
- name: The independent certifier writes the same certificate; it meets acc64, and its scale path acc32
|
||
shell: bash
|
||
working-directory: wai
|
||
run: |
|
||
set -euo pipefail
|
||
D="$WAI_JPEGAI_DIR"
|
||
M=(--member "norm.json=$D/norm.json")
|
||
for n in hs_y hs_uv hyp_y hyp_uv synth_y synth_uv mcm0_y mcm1_y mcm2_y mcm3_y icci; do
|
||
M+=(--member "weights/$n.json=$D/$n.json")
|
||
done
|
||
python3 -I tools/wai_ops_check.py certify --capability wai.image.jpegai --lowering jpegai-int/1 "${M[@]}" \
|
||
--profile acc64 --role-profile hs_y=acc32 --role-profile hs_uv=acc32 --out "${RUNNER_TEMP:-/tmp}/jpegai-bounds-py.json" \
|
||
--check ops-conformance/certificates/wai.image.jpegai/3e243dc11566922f5cb7f4c628afd440b18d56a08b99b07cc33d5c8522aeec37.json
|
||
# The operator set's lowering of the bundle's scale path (integer-operators
|
||
# §13.6.1), run through the interpreter with every value held to its certified
|
||
# interval, against the legacy decoder's arithmetic on the same bundle.
|
||
- name: The scale path's lowering decodes as the legacy arithmetic does
|
||
shell: bash
|
||
working-directory: wai/wai-ops
|
||
run: |
|
||
set -euo pipefail
|
||
WAI_OPS_JPEGAI_BUNDLE="$WAI_JPEGAI_DIR" \
|
||
cargo test --locked --release --test goldens -- --ignored --exact jpegai_scale_path_on_a_supplied_bundle
|
||
|
||
- name: Whole decode vs the reference, determinism, pinned pixel hash (missing input = failure)
|
||
shell: bash
|
||
working-directory: wai/wai-rs
|
||
run: |
|
||
set -o pipefail
|
||
log="${RUNNER_TEMP:-/tmp}/jpegai-tests.log"
|
||
# Without the switch both tests would skip and report ok.
|
||
[ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \
|
||
|| { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; }
|
||
export WAI_JPEGAI_CODESTREAM="$WAI_JPEGAI_DIR/test256.bin"
|
||
export WAI_JPEGAI_WEIGHTS_DIR="$WAI_JPEGAI_DIR"
|
||
export WAI_JPEGAI_NORM="$WAI_JPEGAI_DIR/norm.json"
|
||
# --show-output rather than --nocapture: each test's output is printed
|
||
# after the results, so it cannot split a `test X ... ok` line below.
|
||
cargo test --release --no-default-features --features neural_int,provenance \
|
||
--test jpegai_e2e_full --test jpegai_reconstruction_binding --test jpegai_cross_platform \
|
||
-- --show-output 2>&1 | tee "$log"
|
||
# A test that was renamed or compiled out would not fail — it would just
|
||
# not run. Require each one by name, and the pinned-hash check by its line.
|
||
for t in codestream_to_rgb_matches_reference_png real_decode_to_reconstruction_binding \
|
||
decode_matches_pinned_hashes jpegai_e2e_set_pin_verifies; do
|
||
grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; }
|
||
done
|
||
grep -q "matches the pinned WAI_JPEGAI_EXPECT_SHA256" "$log" \
|
||
|| { echo "::error::the pinned pixel hash was not checked"; exit 1; }
|
||
# A skip also prints `test X ... ok`; every skip writes a banner with
|
||
# "SKIPPED" and "NOT VERIFIED" (tests/support/require_vectors.rs).
|
||
if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then
|
||
echo "::error::a JPEG-AI test skipped instead of checking its input"
|
||
exit 1
|
||
fi
|
||
|
||
# The scale path and the entropy decodes through the integer operator set
|
||
# (integer-operators §13.6.1, lowered by jpegai-int/1), on this bundle: the lowering
|
||
# of the bundle as the decoder parses it is the lowering of its bytes and the
|
||
# committed certificate's graphs, under acc32; both planes' integers and the whole
|
||
# decode through the operator set are the direct decode's; a stream cut at every
|
||
# length is refused (stream_exhausted) exactly where the direct decode panics; and a
|
||
# codestream out of scope is refused with its reason.
|
||
- name: The scale path through the integer operator set, on the pinned bundle (missing input = failure)
|
||
shell: bash
|
||
working-directory: wai/wai-rs
|
||
run: |
|
||
set -o pipefail
|
||
log="${RUNNER_TEMP:-/tmp}/jpegai-ops.log"
|
||
[ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \
|
||
|| { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; }
|
||
export WAI_JPEGAI_CODESTREAM="$WAI_JPEGAI_DIR/test256.bin"
|
||
export WAI_JPEGAI_WEIGHTS_DIR="$WAI_JPEGAI_DIR"
|
||
export WAI_JPEGAI_NORM="$WAI_JPEGAI_DIR/norm.json"
|
||
cargo test --release --no-default-features --features neural_int,legacy_int --test jpegai_ops --test jpegai_memory \
|
||
-- --show-output 2>&1 | tee "$log"
|
||
for t in the_committed_bundle_lowers_to_its_committed_graphs_and_verdict \
|
||
the_pinned_bundle_lowers_to_its_certificate_and_meets_acc32 \
|
||
the_planes_decode_through_the_lowering_as_directly \
|
||
the_whole_decode_through_the_lowering_is_the_direct_decode \
|
||
every_load_check_of_the_bundle_refuses_model_mismatch \
|
||
a_stream_cut_short_is_refused_where_the_direct_decode_panics \
|
||
a_stream_cut_short_is_refused_where_the_direct_decode_panics_pinned \
|
||
a_codestream_out_of_scope_is_refused_with_its_reason \
|
||
every_kernel_policy_decodes_the_planes_alike \
|
||
the_decode_holds_no_more_than_the_direct_decode; do
|
||
grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; }
|
||
done
|
||
if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then
|
||
echo "::error::a JPEG-AI test skipped instead of checking its input"
|
||
exit 1
|
||
fi
|
||
|
||
# The other integer networks through the integer operator set (integer-operators
|
||
# §13.6.2, lowered by jpegai-int/1 to rec_y, rec_uv, syn_y and syn_uv), on this
|
||
# bundle: the lowering of the networks as the decoder parses them is the lowering of
|
||
# their bytes and the committed certificate's graphs, under acc64; each network
|
||
# through its graph gives the direct network's values on the test codestream; every
|
||
# value each operator forms there lies within its certified interval (the measured
|
||
# accumulators of §13.6.2); and every kernel policy runs the graphs alike.
|
||
- name: The hyper decoders, context stages and synthesis through the integer operator set, on the pinned bundle (missing input = failure)
|
||
shell: bash
|
||
working-directory: wai/wai-rs
|
||
run: |
|
||
set -o pipefail
|
||
log="${RUNNER_TEMP:-/tmp}/jpegai-nets-ops.log"
|
||
[ "${WAI_REQUIRE_SOTA_VECTORS:-}" = 1 ] \
|
||
|| { echo "::error::WAI_REQUIRE_SOTA_VECTORS must be 1 in this job"; exit 1; }
|
||
export WAI_JPEGAI_CODESTREAM="$WAI_JPEGAI_DIR/test256.bin"
|
||
export WAI_JPEGAI_WEIGHTS_DIR="$WAI_JPEGAI_DIR"
|
||
export WAI_JPEGAI_NORM="$WAI_JPEGAI_DIR/norm.json"
|
||
cargo test --release --no-default-features --features neural_int,legacy_int --test jpegai_nets_ops \
|
||
-- --show-output 2>&1 | tee "$log"
|
||
for t in the_committed_bundle_lowers_to_its_committed_networks \
|
||
the_pinned_bundle_lowers_its_networks_to_its_certificate \
|
||
the_networks_decode_through_the_lowering_as_directly \
|
||
the_whole_decode_through_the_networks_is_the_pinned_decode \
|
||
the_networks_keep_to_their_certificate_on_the_test_codestream \
|
||
every_load_check_of_the_networks_refuses_model_mismatch \
|
||
a_value_past_its_range_is_refused_field_range \
|
||
every_kernel_policy_runs_the_networks_alike; do
|
||
grep -q "^test $t \.\.\. ok$" "$log" || { echo "::error::$t did not run and pass"; exit 1; }
|
||
done
|
||
if grep -nE 'SKIPPED|NOT VERIFIED' "$log"; then
|
||
echo "::error::a JPEG-AI test skipped instead of checking its input"
|
||
exit 1
|
||
fi
|