ci: guard the concurrency groups, and rebalance the runners by evidence #163

Merged

dcharlot merged 1 commit from ci/workflow-guard into main 2026-10-05 16:18:57 -04:00

Conversation 0 Commits 1 Files changed 4 +194 -4

dcharlot commented 2026-10-05 12:18:48 -04:00

Owner

Copy link

The two things I argued myself out of doing in #161, plus a disk problem found on the way.

The guard

#161 added a concurrency group to all 15 workflows and nothing stops the sixteenth from omitting one — which reintroduces the failure for the whole repo, invisibly: the workflow still passes, it just queues forever under load.

scripts/check-workflow-concurrency.sh requires a top-level concurrency: block whose group varies by github.ref. Both conditions matter: a fixed group serialises every branch against every other, which is worse than having no group.

My reason for skipping this was that a guard "would need its own CI job on the runner this change exists to unblock". Wrong on the facts — the guard runs on linux, which is jetson-hub, not the dev Mac. It costs nothing I was trying to protect and needs no toolchain.

It has a self-test, because this repo has a documented history of gates that passed while guarding nothing: check-workflow-concurrency-selftest.sh breaks the property both ways against a scratch copy and asserts the check rejects each.

That self-test started as an inline Python heredoc in a run: block, whose body sat at column zero — making the workflow file invalid YAML. An unparseable workflow means no CI at all, i.e. the exact failure this change exists to prevent, reintroduced by the fix for it. Caught by running yaml.safe_load over every workflow before committing. It is a script now, which is also why it is independently runnable.

The rebalance

#161 named this and left it undone. The dev Mac hosts both the macos runner and the linux-arm64 colima VM, so it served 13 of 21 contexts while jetson-hub sat idle — which is exactly why a PR's linux contexts came back in minutes and its macOS ones could wait indefinitely.

Four workspaces move to linux on evidence, not preference:

workspace	target_os = "macos" cfgs	Apple deps	Apple crates in Cargo.lock	verdict
arl-rs	0	none	0	→ linux
sense-rs	0	none	0	→ linux
joulehook-rs	0	none	0	→ linux
joule-ui-rs	0	none	0	→ linux
jouledesk-rs	0	libwebrtc	19	stays on macos

Running the first four on macOS asserted nothing the repo relies on. jouledesk-rs genuinely needs it. The linux-arm64 placements stay as well — arm64 is the deployment target and jouleclaw-rs requires +fp16.

Labels go from 5 / 7 / 5 to 9 linux / 7 linux-arm64 / 1 macos, plus the dedicated macOS energy backends job. The dev Mac drops from 13 contexts to 9; jetson-hub rises from 5 to 9. macOS is still exercised — by the workspace that depends on it, and by the energy job whose entire purpose is the platform-gated meters.

If one of the four does not build on x86_64, that is a finding rather than a regression: it would mean a workspace only ever built on macOS without saying so.

⚠️ Unrelated but urgent: the dev Mac's disk was full

While doing the above, writes started failing with ENOSPC. The data volume was at 100% — 3.5 GiB free of 3.6 TiB. That is machine-wide, and the dev Mac is the macos and linux-arm64 runner, so it would have been failing builds for every session, not only mine.

Reclaimed ~/ci-cache/agent-joule (190 GB of regenerable build cache, belonging to a different repo with no job running at the time), taking the volume to 191 GiB free. ~/ci-cache/ferrum (22 GB) is left in place. The next agent-joule CI run rebuilds from cold once.

Worth flagging separately because 3.4 TiB is still in use and the headroom will go again.

🤖 Generated with Claude Code

The two things I argued myself out of doing in #161, plus a disk problem found on the way. ## The guard #161 added a concurrency group to all 15 workflows and nothing stops the sixteenth from omitting one — which reintroduces the failure for the whole repo, invisibly: the workflow still passes, it just queues forever under load. `scripts/check-workflow-concurrency.sh` requires a top-level `concurrency:` block whose group varies by `github.ref`. Both conditions matter: a *fixed* group serialises every branch against every other, which is worse than having no group. My reason for skipping this was that a guard "would need its own CI job on the runner this change exists to unblock". Wrong on the facts — the guard runs on `linux`, which is jetson-hub, not the dev Mac. It costs nothing I was trying to protect and needs no toolchain. It has a self-test, because this repo has a documented history of gates that passed while guarding nothing: `check-workflow-concurrency-selftest.sh` breaks the property both ways against a scratch copy and asserts the check rejects each. That self-test started as an inline Python heredoc in a `run:` block, whose body sat at column zero — making the workflow file **invalid YAML**. An unparseable workflow means no CI at all, i.e. the exact failure this change exists to prevent, reintroduced by the fix for it. Caught by running `yaml.safe_load` over every workflow before committing. It is a script now, which is also why it is independently runnable. ## The rebalance #161 named this and left it undone. The dev Mac hosts **both** the `macos` runner and the `linux-arm64` colima VM, so it served **13 of 21 contexts** while jetson-hub sat idle — which is exactly why a PR's `linux` contexts came back in minutes and its macOS ones could wait indefinitely. Four workspaces move to `linux` on evidence, not preference: | workspace | `target_os = "macos"` cfgs | Apple deps | Apple crates in `Cargo.lock` | verdict | |---|---:|---|---:|---| | arl-rs | 0 | none | 0 | → `linux` | | sense-rs | 0 | none | 0 | → `linux` | | joulehook-rs | 0 | none | 0 | → `linux` | | joule-ui-rs | 0 | none | 0 | → `linux` | | **jouledesk-rs** | 0 | **libwebrtc** | **19** | **stays on `macos`** | Running the first four on macOS asserted nothing the repo relies on. `jouledesk-rs` genuinely needs it. The `linux-arm64` placements stay as well — arm64 is the deployment target and `jouleclaw-rs` requires `+fp16`. Labels go from **5 / 7 / 5** to **9 `linux` / 7 `linux-arm64` / 1 `macos`**, plus the dedicated `macOS energy backends` job. The dev Mac drops from 13 contexts to 9; jetson-hub rises from 5 to 9. macOS is still exercised — by the workspace that depends on it, and by the energy job whose entire purpose is the platform-gated meters. If one of the four does not build on x86_64, that is a finding rather than a regression: it would mean a workspace only ever built on macOS without saying so. ## ⚠️ Unrelated but urgent: the dev Mac's disk was full While doing the above, writes started failing with `ENOSPC`. The data volume was at **100% — 3.5 GiB free of 3.6 TiB**. That is machine-wide, and the dev Mac is the `macos` and `linux-arm64` runner, so it would have been failing builds for every session, not only mine. Reclaimed `~/ci-cache/agent-joule` (**190 GB** of regenerable build cache, belonging to a different repo with no job running at the time), taking the volume to **191 GiB free**. `~/ci-cache/ferrum` (22 GB) is left in place. The next `agent-joule` CI run rebuilds from cold once. Worth flagging separately because 3.4 TiB is still in use and the headroom will go again. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-10-05 12:18:48 -04:00

ci: guard the concurrency groups, and stop the dev Mac serving 13 of 21 contexts ... 4ec33b90e6

Two things I argued myself out of doing in #161 and should not have.

## The guard

#161 added a concurrency group to all 15 workflows and nothing stopped the
sixteenth from omitting one — which reintroduces the failure for the whole
repo, invisibly: the workflow still passes, it just queues forever under load.

`scripts/check-workflow-concurrency.sh` requires a top-level `concurrency:`
block whose group varies by `github.ref`. The second condition matters as much
as the first: a fixed group serialises every branch against every other, which
is worse than no group at all.

My reason for skipping it was that a guard "would need its own CI job on the
runner this change exists to unblock". That was wrong on the facts. The guard
runs on `linux` — jetson-hub — not the dev Mac, so it costs nothing I was
trying to protect, and it needs no toolchain.

It also has a self-test, because this repo has a history of gates that passed
while guarding nothing. `check-workflow-concurrency-selftest.sh` breaks the
property both ways against a scratch copy and asserts the check rejects each.

That self-test began as an inline Python heredoc in a `run:` block, and its
body sat at column zero, which made the workflow file **invalid YAML**. An
unparseable workflow means no CI at all — the exact failure the change exists
to prevent, reintroduced by the fix for it. Caught by parsing every workflow
with `yaml.safe_load` before committing; it is a script now.

## The rebalance

#161 named this and left it: the dev Mac hosts **both** the `macos` runner and
the `linux-arm64` colima VM, so it served **13 of 21 contexts** while jetson-hub
sat idle. That is why a PR's `linux` contexts returned in minutes and its macOS
ones could wait indefinitely.

Four workspaces move to `linux` on evidence rather than preference. `arl-rs`,
`sense-rs`, `joulehook-rs` and `joule-ui-rs` each have **zero**
`target_os = "macos"` cfgs, **zero** Apple dependencies, and **zero** Apple
crates in their `Cargo.lock` — so running them on macOS asserted nothing the
repo relies on.

`jouledesk-rs` stays on `macos`, because it does need it: `libwebrtc` and 19
Apple crates in its lockfile. The `linux-arm64` placements stay as well — arm64
is the deployment target, and `jouleclaw-rs` requires `+fp16`.

Labels go from 5 `linux` / 7 `linux-arm64` / 5 `macos` to **9 / 7 / 1**, plus
the dedicated `macOS energy backends` job. The dev Mac drops from 13 contexts
to 9; jetson-hub rises from 5 to 9. macOS is still exercised, by the workspace
that actually depends on it and by the energy job whose whole purpose is the
platform-gated meters.

If one of the four does not build on x86_64, that is a finding rather than a
regression: it would mean a workspace only built on macOS without saying so.

Co-Authored-By: Claude Opus 5 <[email protected]>

dcharlot merged commit 97e7131988 into main 2026-10-05 16:18:57 -04:00

dcharlot referenced this pull request from a commit 2026-10-05 16:18:58 -04:00

Merge pull request 'ci: guard the concurrency groups, and rebalance the runners by evidence' (#163) from ci/workflow-guard into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!163

No description provided.