6517401f01
Every path-gated workflow begins with one small job -- `does this change touch
it`, or `which workspaces this change touches` for the standards build -- that
all of its real jobs declare in `needs`. All 55 of those gate jobs ran on
`linux-arm64`.
That is the one label the fleet cannot spare. Sampled while 267 runs were
queued, every waiting run had the same shape:
[waiting] linux-arm64 does this change touch it
[blocked] linux member · dataflow device, simulated
[blocked] macos member · dataflow device absent
237 runs, each holding a gate in the longest queue, and every real job blocked
behind it. Not only `linux` jobs: `macos`, `linux-hw`, `linux-arm64-hw` and
`macos-m3-ultra` jobs were blocked too. The measured effect was `linux-arm64`
263 deep against `linux` **zero**, with jetson-hub's 20 cores taking no job for
a full hour -- starved not for want of work but because the work could not be
unblocked. A gate waiting on the resource it is rationing is head-of-line
blocking, and it makes every other runner in the fleet idle on its behalf.
The gates move to `linux`. Nothing else changes: the diff is 55 `runs-on` lines
and no others, and every touched file still parses.
Why `linux` and not elsewhere. The gate is not free -- it compares
`BASE...HEAD`, which needs the merge base, so `fetch-depth: 0` is a full clone
rather than a ten-second script. `macos` is served by hosts on a Mac Studio
sitting at load 185; `linux` is a separate 20-core box that was measurably
empty. This moves a serialisation point to the least contended label; it does
not remove it, and the gate is still a single point of failure for its
workflow. The structural fix is an early-exit first step inside each job, which
costs one checkout per leg and is a larger change.
Follow-up, deliberately not bundled: jetson-hub runs at capacity 2 with 15 GB of
RAM, so 55 clones still serialise two at a time. Raising capacity helps gates
and hurts the heavy emulated-architecture jobs that share the label, so it wants
measuring on its own.
Co-Authored-By: Claude Opus 5 <[email protected]>
227 lines
12 KiB
YAML
227 lines
12 KiB
YAML
name: WAI neural sink gates
|
|
|
|
# The neural sink refuses some envelopes before it touches a model: a pin that
|
|
# the parameter sets it holds do not satisfy, with SPEC §3.1's reason
|
|
# (ModelRegistry::resolve_verified, decode_envelope, decode_multi), and a `replicate` that
|
|
# SPEC §7 does not back (decode_envelope_strict): over a capability whose registry
|
|
# class cannot back it, or with a prior that is float or absent, or that declares a
|
|
# form not established to reconstruct the same tensors at every reader. Those
|
|
# refusals live behind the
|
|
# `neural` feature, which wai-conformance does not build, and the tests that
|
|
# exercise them need no model, so wai-neural-e2e's reference filter never selects
|
|
# them. This job runs exactly those tests and fails if any of them is not run:
|
|
# a filter that matches nothing reports "0 passed" and exits 0.
|
|
#
|
|
# It also runs the `neural` dispatch tests that need no model download, over the
|
|
# committed int_mlicpp vector and the int_synth corpus: the form-less and set-form
|
|
# pins through the strict gate, the int_synth pin over the spec it reads, the
|
|
# several-file capabilities refusing a form-less pin, a pinned decode reading only
|
|
# the bytes it verified, an unpinned decode at a sink with a key log checked over
|
|
# every identity of what it read once and reading only that (or, holding the
|
|
# capability's sets to §7.9, the part §7.9 authorised), an encrypted envelope inert at the
|
|
# sink (SPEC §6.4 step 2), a WAI2 envelope whose pins pick the held model, the strict
|
|
# rule read for the capability decoded through (a sound fallback behind an unsound
|
|
# primary), SPEC §4 step 6's order of reasons after a set_incomplete decode, and the
|
|
# replicate_* reason the strict WAI2 policy reports for an entry it leaves out,
|
|
# and a candidate held at the sink leaving the strict path's reason unchanged.
|
|
# With `key_discovery` as well, it runs SPEC §3.1's check 5 at the neural sink: a
|
|
# registry given a key log refuses the committed int_mlicpp set once the log revokes
|
|
# its digest, and when the sink holds no fresh status, and decodes it while a grant
|
|
# holds it.
|
|
|
|
# One run per branch at a time. A push supersedes the run before it.
|
|
#
|
|
# There is one `macos` runner and it is the dev Mac, so every macOS job in this
|
|
# repo queues on it. With no concurrency group, a session pushing four versions
|
|
# of a branch queued four complete sets of runs and every one of them had to
|
|
# drain: on 2026-10-05 a PR's 21 contexts settled none in 70 minutes behind a
|
|
# backlog of 25 runs, and the queue grew faster than it cleared. Only the newest
|
|
# commit's verdict is wanted, so the older runs are cancelled.
|
|
#
|
|
# Verified on this forge (Forgejo 16.0.5) in `ci-test` before being added here:
|
|
# the key parses, and a newer push moves an older run to `cancelled` even while
|
|
# it is still queued, which is the case that matters.
|
|
#
|
|
# `cancel-in-progress: true` also applies to `main`. That is deliberate and
|
|
# safe here: main advances by merge commits whose tree contains the commits
|
|
# before them, so the newer run covers the older one's changes.
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'wai/wai-rs/src/neural/**'
|
|
- 'wai/wai-rs/src/container.rs'
|
|
- 'wai/wai-rs/src/codecs/mod.rs'
|
|
- 'wai/wai-rs/src/codecs/prior_form.rs'
|
|
- 'wai/wai-rs/src/pin.rs'
|
|
- 'wai/wai-rs/src/key_log.rs'
|
|
- 'wai/wai-rs/src/key_log/**'
|
|
- 'wai/wai-rs/src/signer_id.rs'
|
|
- 'wai/wai-rs/src/claim_receipt.rs'
|
|
- 'wai/wai-rs/src/claim_text.rs'
|
|
- 'wai/wai-rs/src/ffi.rs'
|
|
- 'wai/wai-rs/src/ffi_neural.rs'
|
|
- 'wai/wai-rs/tests/ffi_neural_pin.rs'
|
|
- 'wai/wai-rs/tests/neural_cost_limit.rs'
|
|
- 'wai/wai-rs/src/int_synth.rs'
|
|
# The integer operator set the integer decoders load their sets through.
|
|
- 'wai/wai-rs/src/int_ops.rs'
|
|
- 'wai/wai-ops/**'
|
|
- 'wai/wai-rs/src/ort_session.rs'
|
|
- 'wai/neural-conformance/vectors/**'
|
|
- 'wai/pin-conformance/**'
|
|
- 'wai/wai-rs/tests/neural_native_decode.rs'
|
|
- 'wai/wai-rs/src/int_mlicpp.rs'
|
|
- 'wai/wai-rs/tests/vectors/int_mlicpp/**'
|
|
# The integer capabilities the neural sink delegates to int_payload::decode_int,
|
|
# and the vectors its tests decode through it.
|
|
- 'wai/wai-rs/src/int_payload/**'
|
|
- 'wai/wai-rs/src/int_video.rs'
|
|
- 'wai/wai-rs/src/int_hyper_synth.rs'
|
|
- 'wai/wai-rs/src/int_audio.rs'
|
|
- 'wai/wai-rs/src/int_splat.rs'
|
|
- 'wai/wai-rs/tests/vectors/int_hyper/**'
|
|
- 'wai/wai-rs/tests/vectors/int_video/**'
|
|
- 'wai/wai-rs/tests/vectors/int_audio/**'
|
|
- 'wai/wai-rs/tests/vectors/int_splat/**'
|
|
- 'wai/wai-rs/Cargo.lock'
|
|
- '.github/workflows/wai-neural-gates.yml'
|
|
schedule:
|
|
- cron: '25 5 * * *'
|
|
|
|
jobs:
|
|
# Does this pull request's own diff touch the paths below? This forge matches a
|
|
# workflow's `paths:` against the push, so a merge of main into a branch fires the
|
|
# workflow whatever the pull request changes; every job here waits on this answer.
|
|
# A push to main, a schedule, or anything the script cannot determine answers
|
|
# `true` (scripts/pr-touches-paths.py).
|
|
relevant:
|
|
name: does this change touch it
|
|
runs-on: linux
|
|
outputs:
|
|
run: ${{ steps.gate.outputs.run }}
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
with:
|
|
fetch-depth: 0
|
|
- id: gate
|
|
run: |
|
|
set -euo pipefail
|
|
r=$(python3 scripts/pr-touches-paths.py .github/workflows/wai-neural-gates.yml \
|
|
"${{ github.event.pull_request.base.sha }}" \
|
|
"${{ github.event.pull_request.head.sha }}")
|
|
echo "this change touches wai-neural-gates: $r"
|
|
echo "run=$r" >> "$GITHUB_OUTPUT"
|
|
|
|
neural-gates:
|
|
needs: relevant
|
|
if: needs.relevant.outputs.run == 'true'
|
|
name: model-free refusals of the neural sink
|
|
runs-on: linux
|
|
# The ONNX Runtime binary the `ort` crate downloads needs glibc 2.38 and a
|
|
# GCC 13 libstdc++ at link time (`__isoc23_strtol`, `_M_replace_cold`). The
|
|
# `linux` label's default image is Ubuntu 22.04 (glibc 2.35, GCC 11), where
|
|
# the `neural` test binary fails to link, so this job runs in the 24.04 image.
|
|
container:
|
|
image: catthehacker/ubuntu:act-24.04
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
# Bootstrap rustup: the docker runner image ships no Rust.
|
|
- name: Install Rust 1.98.0
|
|
run: |
|
|
export PATH="$HOME/.cargo/bin:$PATH"
|
|
command -v rustup >/dev/null 2>&1 || \
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.98.0
|
|
rustup toolchain install 1.98.0 --profile minimal
|
|
rustup default 1.98.0
|
|
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
|
- name: Prior-hash refusal, the strict replicate gate, and the int_mlicpp dispatch
|
|
working-directory: wai/wai-rs
|
|
shell: bash
|
|
# --no-default-features: `neural` needs only `ort`, not the C codec chain.
|
|
# --exact: each name selects one test. The count check turns a renamed or
|
|
# removed test into a failure instead of a silent "0 passed".
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --no-default-features --features neural --test neural_native_decode -- \
|
|
--exact resolve_checked_verifies_prior_hash strict_decode_gates_unsound_replicate \
|
|
int_mlicpp_decodes_through_the_strict_dispatch int_mlicpp_set_pin_through_the_strict_dispatch \
|
|
int_synth_pin_covers_the_spec_it_reads several_file_capabilities_need_the_set_form \
|
|
pinned_decode_reads_only_verified_bytes wai2_decode_multi_picks_the_verified_rendition \
|
|
strict_gate_reads_the_capability_it_decodes_through set_incomplete_at_step_3_outranks_the_fallback_refusal \
|
|
decode_multi_strict_reports_replicate_reasons strict_gate_ignores_a_held_capability_it_cannot_dispatch \
|
|
int_hyper_decodes_through_the_neural_dispatch int_synth_check_5_covers_the_spec_a_model_directory_holds \
|
|
an_integer_capability_without_a_decoder_moves_on_to_the_fallback \
|
|
a_listing_that_does_not_match_the_files_read_authorises_nothing \
|
|
a_pinned_set_revoked_under_its_file_digest_is_refused_through_a_listing_pin \
|
|
a_granted_listing_authorises_only_the_files_it_names \
|
|
an_encrypted_envelope_is_inert_at_the_reference_sink \
|
|
a_revoke_only_key_log_checks_and_reads_the_set_once \
|
|
2>&1 | tee "$RUNNER_TEMP/neural-gates.log"
|
|
grep -q "test result: ok. 20 passed; 0 failed" "$RUNNER_TEMP/neural-gates.log"
|
|
# The integer video, audio and splat capabilities through the neural dispatch, which
|
|
# delegates them to int_payload::decode_int: each decodes to Decoded::Int with its
|
|
# committed SHA-256, and wai.video.int_motion is advertised with ModelRegistry::builtin.
|
|
- name: The neural sink decodes the integer video, audio and splat payloads
|
|
working-directory: wai/wai-rs
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --no-default-features --features neural,video_int,audio_int,splat_int --test neural_native_decode -- \
|
|
--exact integer_media_decode_through_the_neural_dispatch \
|
|
2>&1 | tee "$RUNNER_TEMP/neural-int.log"
|
|
grep -q "test result: ok. 1 passed; 0 failed" "$RUNNER_TEMP/neural-int.log"
|
|
# The registry's cost limit refuses an integer payload over it before any symbol is
|
|
# decoded (cost_over_limit) and moves on to the next entry; the C ABI refuses an
|
|
# integer video it has no output kind for before decoding it; and the declared
|
|
# cost over a registry's set, and through the C ABI given the set's path, is the
|
|
# library's. Its own binary: a live-bytes allocator measures each refusal.
|
|
- name: The neural sink applies the declared cost before decoding
|
|
working-directory: wai/wai-rs
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --no-default-features --features neural,ffi,video_int,audio_int,splat_int --test neural_cost_limit \
|
|
2>&1 | tee "$RUNNER_TEMP/neural-cost.log"
|
|
grep -q "test result: ok. 3 passed; 0 failed" "$RUNNER_TEMP/neural-cost.log"
|
|
- name: A key log refuses a revoked pinned set at the neural sink (SPEC §3.1 check 5)
|
|
working-directory: wai/wai-rs
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --no-default-features --features neural,key_discovery --test neural_native_decode -- \
|
|
--exact int_mlicpp_revoked_set_is_refused_through_the_key_log \
|
|
2>&1 | tee "$RUNNER_TEMP/neural-key-log.log"
|
|
grep -q "test result: ok. 1 passed; 0 failed" "$RUNNER_TEMP/neural-key-log.log"
|
|
# Library unit tests that exist only under `neural` and need no model: the
|
|
# bmshj2018 CDF table cache stays bounded and evicts least recently used;
|
|
# the ONNX reader finds a tensor kept in another file, wherever it sits; and a
|
|
# pinned decode refuses such a model with set_incomplete before the runtime
|
|
# sees it, while the same graph with its tensor inside loads.
|
|
- name: Model-free neural unit tests
|
|
working-directory: wai/wai-rs
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --no-default-features --features neural --lib -- --exact neural::image::tests::cdf_cache_is_bounded \
|
|
neural::onnx_external::tests::a_tensor_kept_inside_the_model_is_not_external \
|
|
neural::onnx_external::tests::an_external_initializer_is_found_with_its_location \
|
|
neural::onnx_external::tests::an_external_tensor_in_a_subgraph_is_found \
|
|
neural::onnx_external::tests::external_data_without_a_location_still_marks_the_tensor \
|
|
neural::onnx_external::tests::bytes_that_are_not_a_protobuf_message_are_an_error \
|
|
neural::params::tests::a_pinned_model_with_external_tensors_is_set_incomplete \
|
|
2>&1 | tee "$RUNNER_TEMP/neural-lib.log"
|
|
grep -q "test result: ok. 7 passed; 0 failed" "$RUNNER_TEMP/neural-lib.log"
|
|
- name: The neural C ABI reports pin and replicate refusals and decodes WAI2
|
|
working-directory: wai/wai-rs
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
cargo test --no-default-features --features neural,ffi --test ffi_neural_pin -- \
|
|
--exact tampered_model_is_wai_err_prior strict_replicate_is_wai_err_unsound_replicate \
|
|
wai2_decode_ex_picks_the_held_pin failed_decode_ex_clears_out \
|
|
2>&1 | tee "$RUNNER_TEMP/neural-ffi.log"
|
|
grep -q "test result: ok. 4 passed; 0 failed" "$RUNNER_TEMP/neural-ffi.log"
|