ci: one run per branch at a time, and where the queue actually bottlenecks #161
Loading…
Reference in a new issue
No description provided.
Delete branch "ci/concurrency-groups"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Not one of this repo's 15 workflows declared
concurrency:, so every push queued a complete set of runs and nothing superseded anything.That is what I hit twice this week. A session pushing
wai-quantum0.3.31 → 0.3.34 queued four complete sets of runs, all of which had to drain, when only the newest commit's verdict was wanted. PR #157's 21 contexts settled none in 70 minutes behind a backlog of 25 runs that was growing faster than it cleared, and I merged both #154 and #157 on locally-executed equivalents because the gate was effectively unavailable. A gate that never runs is not a gate.Every workflow now groups on
${{ github.workflow }}-${{ github.ref }}withcancel-in-progress: true.Verified on this forge, not assumed
An unparseable workflow would mean no CI at all — a far worse failure than a slow queue — and the fleet's existing
concurrency:users are all GitHub-hosted, so they prove GitHub parses the key, not Forgejo.So I pushed a probe to
ci-teston this forge (Forgejo 16.0.5) first. Runs were created, so the key parses. Then a second push moved both earlier runs tocancelledwhile they were still queued — which is precisely the case that matters, since the problem here is queue depth rather than long-running jobs.cancel-in-progress: trueapplies tomainas well. Deliberate: main advances by merge commits whose tree contains the commits before them, so a newer run covers an older one's changes.The queue's real shape — worse than "one busy runner"
I had been calling this a macOS-runner problem. Measuring it properly, it is not. The workspace matrix asks for three labels:
linuxlinux-arm64txarmVM, on the dev MacmacosmacOS energy backendsjob13 of 21 contexts land on the dev Mac through two runners — while that same machine was carrying another session's VVC/VTM encodes at load average 132 on 18 CPUs. That explains the pattern exactly:
jcp-rsandeoc-rscame back quickly because they are the two on jetson-hub;mesh-rs(linux-arm64) took 25 minutes; the macOS job never ran.I should also correct something I said earlier: I read an ssh timeout to jetson-hub as the linux runner being down. It is not — the runner connects outbound to the forge and the forge reports it idle. The ssh route being unavailable says nothing about the runner.
Concurrency groups cut queued runs for every label, which helps all three. They do not change one host serving 13 of 21 contexts. The structural options are to move some
linux-arm64workspaces onto jetson-hub'slinux, or to stand up a second arm64/macOS host — the Mac Studio M3 Ultra is online and idle. Both are fleet changes rather than repo ones, so they are flagged rather than attempted here.What is deliberately not included
No regression guard asserting that every workflow declares the key. Such a check would need its own CI job on the very runner this change exists to unblock, and an unwired check is worse than none — this repo already has a documented history of conformance packs that nothing read.
Proven live, on this repo's own workflows
The
ci-testprobe showed the key parses on this forge. This PR then demonstrated it on the real thing: a workflows-only change triggers every workflow, so opening it queued 15 runs of its own. Pushing one more commit to the branch moved all 15 tocancelledand queued 15 replacements, rather than stacking 30.That also settles the parsing risk empirically rather than by inspection. The forge read these exact modified files, created runs from them, and applied their concurrency block — a malformed workflow would have produced no runs to cancel.
For scale: the queue was 43 waiting at that moment, 15 of them mine, with the dev Mac at load average 188. Without the key my own PR would have contributed 30.
🤖 Generated with Claude Code
Not one of this repo's 15 workflows declared `concurrency:`, so every push queued a complete set of runs and nothing superseded anything. A session pushing four versions of a branch queued four complete sets, and all of them had to drain. On 2026-10-05 a PR's 21 contexts settled **none in 70 minutes** behind a backlog of 25 runs, and the backlog grew faster than it cleared — 0.3.31 through 0.3.34 of one branch, four full sets, when only the newest commit's verdict was wanted. Every workflow now groups on `${{ github.workflow }}-${{ github.ref }}` with `cancel-in-progress: true`. **Verified before being added here, not assumed.** An unparseable workflow would mean no CI at all, which is a worse failure than a slow queue, and none of the fleet's existing `concurrency:` users are on this forge — they are GitHub-hosted, so they prove GitHub parses the key, not Forgejo. Pushed a probe to `ci-test` on this forge (Forgejo 16.0.5) instead: runs were created, so the key parses; and a newer push moved two older runs to `cancelled` **while they were still queued**, which is exactly the case that matters here. `cancel-in-progress: true` applies to `main` too. That is deliberate: main advances by merge commits whose tree contains the commits before them, so a newer run covers an older one's changes. ## The queue's real shape, which is worse than one busy runner Worth recording because it decides where the next fix goes. The workspace matrix asks for three labels: - **`linux` — 5 workspaces** (jcp-rs, sandbox-rs, smart-byte-rs, proof-rs, eoc-rs) → jetson-hub, which the forge reports idle and connected. - **`linux-arm64` — 7 workspaces** (openpay, joule-code-rs, map-rs, aac-rs, joulecontract-rs, mesh-rs, jouleclaw-rs) → the colima `txarm` VM, **on the dev Mac**. - **`macos` — 5 workspaces** plus the dedicated `macOS energy backends` job → **the dev Mac**. So **13 of 21 contexts land on the dev Mac** through two runners, while that same machine was carrying another session's VVC/VTM encodes at load average 132 on 18 CPUs. That is why `jcp-rs` and `eoc-rs` returned quickly — they are the two on jetson-hub — while `mesh-rs` (linux-arm64) took 25 minutes and the macOS job never ran at all. Concurrency groups cut the number of queued runs, which helps every label. They do not change the fact that one host serves 13 of 21 contexts. The structural options are to move some `linux-arm64` workspaces onto jetson-hub's `linux`, or to stand up a second arm64/macOS host — the Mac Studio M3 Ultra is online and idle — and that is a fleet change rather than a repo one, so it is flagged here rather than attempted. No regression guard is added. A check that every workflow declares the key would itself need a CI job on the runner this change exists to unblock, and an unwired check is worse than none. Co-Authored-By: Claude Opus 5 <[email protected]>