CI: the gate that decides whether a workflow runs was queued behind the work it gates #603
Loading…
Reference in a new issue
No description provided.
Delete branch "ci/gates-off-arm64-r2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Every path-gated workflow in this repository opens with a cheap gate job —
relevant,shardsorplan— that reads the pull request's diff and answers whether the real work should run. Every real job in the workflow declares that gate inneeds, so nothing in the workflow starts until the gate finishes.58 of the repository's 66 gate jobs ran on
linux-arm64, which is the fleet's most contended label.That is head-of-line blocking, and it is the expensive kind: a script that takes ten seconds waits in the same queue as jobs that take an hour, and while it waits, every other runner the workflow would use sits idle, because no job may begin until the gate has spoken. A gate rations a queue it should never be standing in.
This is not hypothetical. It deadlocked a stack of pull requests earlier today: two of them had nothing left to settle but gate scripts, those gates were queued behind hour-long falsification shards on
linux-arm64, and a macOS job was reportedblockedwaiting on one of them while thelinuxrunners had no work at all.This moves all 58 gate jobs to
linux. The work jobs do not move. Where builds, tests and proofs run is a deliberate choice per workflow and is untouched here; only the question "should this workflow run at all" changes queue.The second commit
The first pass found the gates by the sentence they share in their
name:—does this change touch it. Four workflows prefix theirs (WAI HTTP · does this change touch it,WAI breaking · …,WAI examples · …,wai-pack · …), and one names itself for its job entirely, so five gates were missed by the sweep that claimed to be exhaustive.Two of them are in
wai-http.yml, which is the worst file to have missed: its gate decides whether sixfalsifyshards run, and the file's own comment puts those at about an hour and a half of runner time. Both the gate and the work it gates were in the same queue.The second commit finds them by the job key instead of the display name, because the key is the structural fact and the name is prose.
Verification
Checked with a different query from the one that found the targets, so the check cannot inherit the original mistake:
{"linux" => 66}— none remain onlinux-arm64or anywhere else.check-workflow-concurrency.sh: 77 workflows, each grouped per ref.check-tests-in-ci.pyandcheck-lockfiles.shpass.Why this replaces #598
Same two commits, rebased onto
mainafter #599 and #602 landed. #598's three red checks were all explained and are all now fixed onmain:joulespec-proofswas the proof-gate bug that #599 repairs,kani (isqrt512)was the shard that #602 moves to a host where it fits, and the android leg was a timing flake on a build host that was at load 476 from orphaned simulator processes since reaped. Rebasing in place needs a force-push, which is not available here, hence a new branch.#599 contained the identical
planchange instandards-build.yml, so git applied it once; that hunk is already onmainand is correctly absent from this diff, which is why this is 58 jobs across 57 files rather than 59 across 58.🤖 Generated with Claude Code
Every path-gated workflow begins with one small job -- `does this change touch it`, or `which workspaces this change touches` for the standards build -- that all of its real jobs declare in `needs`. All 55 of those gate jobs ran on `linux-arm64`. That is the one label the fleet cannot spare. Sampled while 267 runs were queued, every waiting run had the same shape: [waiting] linux-arm64 does this change touch it [blocked] linux member · dataflow device, simulated [blocked] macos member · dataflow device absent 237 runs, each holding a gate in the longest queue, and every real job blocked behind it. Not only `linux` jobs: `macos`, `linux-hw`, `linux-arm64-hw` and `macos-m3-ultra` jobs were blocked too. The measured effect was `linux-arm64` 263 deep against `linux` **zero**, with jetson-hub's 20 cores taking no job for a full hour -- starved not for want of work but because the work could not be unblocked. A gate waiting on the resource it is rationing is head-of-line blocking, and it makes every other runner in the fleet idle on its behalf. The gates move to `linux`. Nothing else changes: the diff is 55 `runs-on` lines and no others, and every touched file still parses. Why `linux` and not elsewhere. The gate is not free -- it compares `BASE...HEAD`, which needs the merge base, so `fetch-depth: 0` is a full clone rather than a ten-second script. `macos` is served by hosts on a Mac Studio sitting at load 185; `linux` is a separate 20-core box that was measurably empty. This moves a serialisation point to the least contended label; it does not remove it, and the gate is still a single point of failure for its workflow. The structural fix is an early-exit first step inside each job, which costs one checkout per leg and is a larger change. Follow-up, deliberately not bundled: jetson-hub runs at capacity 2 with 15 GB of RAM, so 55 clones still serialise two at a time. Raising capacity helps gates and hurts the heavy emulated-architecture jobs that share the label, so it wants measuring on its own. Co-Authored-By: Claude Opus 5 <[email protected]>