ci: one run per branch at a time, and where the queue actually bottlenecks #161

Merged

dcharlot merged 2 commits from ci/concurrency-groups into main 2026-10-05 12:04:16 -04:00

Conversation 0 Commits 2 Files changed 15 +302

2026-10-05

Transaction Science

ci: touch one workflow to test that the new group supersedes its own queue ...

Co-Authored-By: Claude Opus 5 <[email protected]>

395f4a28df

• View at this point in history

2026-10-05 12:03:07 -04:00

Transaction Science

ci: one run per branch at a time — no workflow declared a concurrency group ...

Not one of this repo's 15 workflows declared `concurrency:`, so every push
queued a complete set of runs and nothing superseded anything. A session
pushing four versions of a branch queued four complete sets, and all of them
had to drain. On 2026-10-05 a PR's 21 contexts settled **none in 70 minutes**
behind a backlog of 25 runs, and the backlog grew faster than it cleared —
0.3.31 through 0.3.34 of one branch, four full sets, when only the newest
commit's verdict was wanted.

Every workflow now groups on `${{ github.workflow }}-${{ github.ref }}` with
`cancel-in-progress: true`.

**Verified before being added here, not assumed.** An unparseable workflow
would mean no CI at all, which is a worse failure than a slow queue, and none
of the fleet's existing `concurrency:` users are on this forge — they are
GitHub-hosted, so they prove GitHub parses the key, not Forgejo. Pushed a probe
to `ci-test` on this forge (Forgejo 16.0.5) instead: runs were created, so the
key parses; and a newer push moved two older runs to `cancelled` **while they
were still queued**, which is exactly the case that matters here.

`cancel-in-progress: true` applies to `main` too. That is deliberate: main
advances by merge commits whose tree contains the commits before them, so a
newer run covers an older one's changes.

## The queue's real shape, which is worse than one busy runner

Worth recording because it decides where the next fix goes. The workspace
matrix asks for three labels:

  - **`linux` — 5 workspaces** (jcp-rs, sandbox-rs, smart-byte-rs, proof-rs,
    eoc-rs) → jetson-hub, which the forge reports idle and connected.
  - **`linux-arm64` — 7 workspaces** (openpay, joule-code-rs, map-rs, aac-rs,
    joulecontract-rs, mesh-rs, jouleclaw-rs) → the colima `txarm` VM, **on the
    dev Mac**.
  - **`macos` — 5 workspaces** plus the dedicated `macOS energy backends` job
    → **the dev Mac**.

So **13 of 21 contexts land on the dev Mac** through two runners, while that
same machine was carrying another session's VVC/VTM encodes at load average
132 on 18 CPUs. That is why `jcp-rs` and `eoc-rs` returned quickly — they are
the two on jetson-hub — while `mesh-rs` (linux-arm64) took 25 minutes and the
macOS job never ran at all.

Concurrency groups cut the number of queued runs, which helps every label. They
do not change the fact that one host serves 13 of 21 contexts. The structural
options are to move some `linux-arm64` workspaces onto jetson-hub's `linux`, or
to stand up a second arm64/macOS host — the Mac Studio M3 Ultra is online and
idle — and that is a fleet change rather than a repo one, so it is flagged here
rather than attempted.

No regression guard is added. A check that every workflow declares the key
would itself need a CI job on the runner this change exists to unblock, and an
unwired check is worse than none.

Co-Authored-By: Claude Opus 5 <[email protected]>

23d8d31f65

• View at this point in history

2026-10-05 12:01:57 -04:00