CI: the gate that decides whether a workflow runs was queued behind the work it gates #598

Closed
dcharlot wants to merge 1 commit from ci/gates-off-arm64 into main

2026-11-03

Transaction Science
CI: the gate that decides whether a workflow runs was queued behind the work it gates
Some checks failed
wai-sink-android native sink library / every ABI, the AAR, and the instrumented tests in a headless emulator · macos (pull_request) Failing after 28m13s
wai-quantum WebGPU member / member · Metal (macos) (pull_request) Successful in 23m39s
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (pull_request) Successful in 3m39s
WAI conformance / v2 vocabulary · manifests validate as stated (pull_request) Successful in 35s
WAI conformance / each feature of the independent verifiers' steps compiles alone (pull_request) Successful in 21m12s
WAI conformance / delivery telemetry · CMCD and CMSD against an independent library (pull_request) Successful in 15s
WAI conformance / cross-arch byte-equality · linux-arm64 (pull_request) Successful in 52s
WAI conformance / C2PA manifest emission · signed, read back, validated (pull_request) Successful in 3m7s
wai formal verification (integer core) / kani (isqrt512) (pull_request) Failing after 1h59m46s
wai-ops integer operators / linux-aarch64 (pull_request) Successful in 6m39s
WAI energy receipts / energy receipts · linux-arm64 (pull_request) Successful in 30m37s
wai-ops-npu integer operators on neural processors / builds · no device is not a pass (linux-arm64) (pull_request) Successful in 2m45s
standards workspaces build + test / jouleclaw-rs (-nodebug, jouleclaw/jouleclaw-rs, -p jouleclaw-energy --features rapl,nvml, jouleclaw-rs, linux-arm64, -C target-feature=+fp16) (pull_request) Successful in 5m22s
WAI conformance / cargo-class corpora · linux-arm64 (pull_request) Successful in 1h14m41s
standards workspaces build + test / joule-code-rs (joule-code/joule-code-rs, joule-code-rs, linux-arm64, true) (pull_request) Successful in 8m35s
standards workspaces build + test / map-rs (map-conformance, map/map-rs, map-rs, linux-arm64) (pull_request) Successful in 43s
standards workspaces build + test / mesh-rs (mesh/mesh-rs, -p mesh-engine-gguf -p mesh-openai --features mesh-engine-gguf/gguf,mesh-openai/server, mesh-rs, linux-arm64) (pull_request) Successful in 45s
standards workspaces build + test / openpay (openpay, openpay, linux-arm64) (pull_request) Successful in 2m12s
wai cross-architecture byte-equality / linux-aarch64 (pull_request) Successful in 43s
standards workspaces build + test / joulecontract-rs (joulecontract/joulecontract-rs, joulecontract-rs, linux-arm64, true) (pull_request) Successful in 8m52s
wai-quantum HIP member / member · HIP absent (linux-arm64) (pull_request) Successful in 1m44s
wai-quantum npu-cores member / member · npu-cores absent (linux-arm64) (pull_request) Successful in 2m48s
WAI conformance / independent verifiers · claims, JWP, key discovery and integer payloads (Python) (pull_request) Successful in 1h24m11s
wai-quantum pulse and dynamics / the same bits · linux-arm64 (pull_request) Successful in 2m19s
wai-quantum variational and algorithm library / parameters, gradients, optimisers, VQE, QAOA and builders · linux-arm64, release (pull_request) Successful in 2m55s
wai-sink-android native sink library / the JNI layer and the platform-free half on the host JVM · linux-arm64 (pull_request) Successful in 4m6s
standards workspaces build + test / aac-rs (aac-conformance, aac/aac-rs, aac-rs, linux-arm64) (pull_request) Successful in 26m40s
wai-quantum tensor-engine member / member · tensor engine absent (linux-arm64) (pull_request) Successful in 28s
wai-vector vector graphics / linux-aarch64 (pull_request) Successful in 4m22s
wai-quantum extension points / the registry and the example · linux-arm64 (pull_request) Successful in 8m40s
Every path-gated workflow begins with one small job -- `does this change touch
it`, or `which workspaces this change touches` for the standards build -- that
all of its real jobs declare in `needs`. All 55 of those gate jobs ran on
`linux-arm64`.

That is the one label the fleet cannot spare. Sampled while 267 runs were
queued, every waiting run had the same shape:

    [waiting]  linux-arm64    does this change touch it
    [blocked]  linux          member · dataflow device, simulated
    [blocked]  macos          member · dataflow device absent

237 runs, each holding a gate in the longest queue, and every real job blocked
behind it. Not only `linux` jobs: `macos`, `linux-hw`, `linux-arm64-hw` and
`macos-m3-ultra` jobs were blocked too. The measured effect was `linux-arm64`
263 deep against `linux` **zero**, with jetson-hub's 20 cores taking no job for
a full hour -- starved not for want of work but because the work could not be
unblocked. A gate waiting on the resource it is rationing is head-of-line
blocking, and it makes every other runner in the fleet idle on its behalf.

The gates move to `linux`. Nothing else changes: the diff is 55 `runs-on` lines
and no others, and every touched file still parses.

Why `linux` and not elsewhere. The gate is not free -- it compares
`BASE...HEAD`, which needs the merge base, so `fetch-depth: 0` is a full clone
rather than a ten-second script. `macos` is served by hosts on a Mac Studio
sitting at load 185; `linux` is a separate 20-core box that was measurably
empty. This moves a serialisation point to the least contended label; it does
not remove it, and the gate is still a single point of failure for its
workflow. The structural fix is an early-exit first step inside each job, which
costs one checkout per leg and is a larger change.

Follow-up, deliberately not bundled: jetson-hub runs at capacity 2 with 15 GB of
RAM, so 55 clones still serialise two at a time. Raising capacity helps gates
and hurts the heavy emulated-architecture jobs that share the label, so it wants
measuring on its own.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-11-03 03:09:23 -05:00