CI: the gate that decides whether a workflow runs was queued behind the work it gates #603

Merged

dcharlot merged 2 commits from ci/gates-off-arm64-r2 into main 2026-11-03 17:41:39 -05:00

Conversation 0 Commits 2 Files changed 57 +58 -58

2026-11-03

Transaction Science

CI: five more gates were in the queue they ration, missed by their own names ...

The commit before this one moved 55 gate jobs off `linux-arm64`. It found them
by the name every gate carried, `does this change touch it`. Four workflows
prefix theirs -- `WAI HTTP · does this change touch it`, `WAI breaking · does
this change touch WAI`, `WAI examples · does this change touch them`,
`wai-pack · does this change touch it` -- so a name match skipped them, and the
fix that was meant to be exhaustive left five jobs behind.

Searching on the job key instead of the display name finds them. They are the
same shape as the 55: `fetch-depth: 0`, one call to pr-touches-paths.py, and
every real job in the file declaring them in `needs`. The fifth is wai-http's
`shards`, the accounting job that checks each patch and mutation lands in
exactly one shard; it runs one bash script and gates nothing cheaper than
itself, so it belongs on the same label as wai-formal-verification's.

wai-http is the costly one to leave behind. Its six falsify shards are about an
hour and a half of runner time, its gate decides whether they run at all, and
the gate's own regex lists `.github/workflows/wai-http.yml`, so any branch that
edits that file -- every Rust-toolchain branch edits all of them -- waits on an
arm64 gate to find out it must then spend that hour and a half on arm64 too.

Measured while this was written: `linux-arm64` 188 runs deep, `linux` idle, and
the only jobs two pull requests still needed to merge were gate scripts of a
few seconds each, queued behind hour-long work. The work stays where it is.
Only the gates move.

Verified: searching by job key now finds no gate-class job on `linux-arm64`,
and all four files parse.

Co-Authored-By: Claude Opus 5 <[email protected]>

944a813684

• View at this point in history

2026-11-03 15:19:13 -05:00

Transaction Science

CI: the gate that decides whether a workflow runs was queued behind the work it gates ...

Every path-gated workflow begins with one small job -- `does this change touch
it`, or `which workspaces this change touches` for the standards build -- that
all of its real jobs declare in `needs`. All 55 of those gate jobs ran on
`linux-arm64`.

That is the one label the fleet cannot spare. Sampled while 267 runs were
queued, every waiting run had the same shape:

    [waiting]  linux-arm64    does this change touch it
    [blocked]  linux          member · dataflow device, simulated
    [blocked]  macos          member · dataflow device absent

237 runs, each holding a gate in the longest queue, and every real job blocked
behind it. Not only `linux` jobs: `macos`, `linux-hw`, `linux-arm64-hw` and
`macos-m3-ultra` jobs were blocked too. The measured effect was `linux-arm64`
263 deep against `linux` **zero**, with jetson-hub's 20 cores taking no job for
a full hour -- starved not for want of work but because the work could not be
unblocked. A gate waiting on the resource it is rationing is head-of-line
blocking, and it makes every other runner in the fleet idle on its behalf.

The gates move to `linux`. Nothing else changes: the diff is 55 `runs-on` lines
and no others, and every touched file still parses.

Why `linux` and not elsewhere. The gate is not free -- it compares
`BASE...HEAD`, which needs the merge base, so `fetch-depth: 0` is a full clone
rather than a ten-second script. `macos` is served by hosts on a Mac Studio
sitting at load 185; `linux` is a separate 20-core box that was measurably
empty. This moves a serialisation point to the least contended label; it does
not remove it, and the gate is still a single point of failure for its
workflow. The structural fix is an early-exit first step inside each job, which
costs one checkout per leg and is a larger change.

Follow-up, deliberately not bundled: jetson-hub runs at capacity 2 with 15 GB of
RAM, so 55 clones still serialise two at a time. Raising capacity helps gates
and hurts the heavy emulated-architecture jobs that share the label, so it wants
measuring on its own.

Co-Authored-By: Claude Opus 5 <[email protected]>

6517401f01

• View at this point in history

2026-11-03 15:19:13 -05:00