From 4ec33b90e609578d4aa2a3daacb760635d61a563 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Mon, 5 Oct 2026 12:18:21 -0400 Subject: [PATCH] ci: guard the concurrency groups, and stop the dev Mac serving 13 of 21 contexts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things I argued myself out of doing in #161 and should not have. ## The guard #161 added a concurrency group to all 15 workflows and nothing stopped the sixteenth from omitting one — which reintroduces the failure for the whole repo, invisibly: the workflow still passes, it just queues forever under load. `scripts/check-workflow-concurrency.sh` requires a top-level `concurrency:` block whose group varies by `github.ref`. The second condition matters as much as the first: a fixed group serialises every branch against every other, which is worse than no group at all. My reason for skipping it was that a guard "would need its own CI job on the runner this change exists to unblock". That was wrong on the facts. The guard runs on `linux` — jetson-hub — not the dev Mac, so it costs nothing I was trying to protect, and it needs no toolchain. It also has a self-test, because this repo has a history of gates that passed while guarding nothing. `check-workflow-concurrency-selftest.sh` breaks the property both ways against a scratch copy and asserts the check rejects each. That self-test began as an inline Python heredoc in a `run:` block, and its body sat at column zero, which made the workflow file **invalid YAML**. An unparseable workflow means no CI at all — the exact failure the change exists to prevent, reintroduced by the fix for it. Caught by parsing every workflow with `yaml.safe_load` before committing; it is a script now. ## The rebalance #161 named this and left it: the dev Mac hosts **both** the `macos` runner and the `linux-arm64` colima VM, so it served **13 of 21 contexts** while jetson-hub sat idle. That is why a PR's `linux` contexts returned in minutes and its macOS ones could wait indefinitely. Four workspaces move to `linux` on evidence rather than preference. `arl-rs`, `sense-rs`, `joulehook-rs` and `joule-ui-rs` each have **zero** `target_os = "macos"` cfgs, **zero** Apple dependencies, and **zero** Apple crates in their `Cargo.lock` — so running them on macOS asserted nothing the repo relies on. `jouledesk-rs` stays on `macos`, because it does need it: `libwebrtc` and 19 Apple crates in its lockfile. The `linux-arm64` placements stay as well — arm64 is the deployment target, and `jouleclaw-rs` requires `+fp16`. Labels go from 5 `linux` / 7 `linux-arm64` / 5 `macos` to **9 / 7 / 1**, plus the dedicated `macOS energy backends` job. The dev Mac drops from 13 contexts to 9; jetson-hub rises from 5 to 9. macOS is still exercised, by the workspace that actually depends on it and by the energy job whose whole purpose is the platform-gated meters. If one of the four does not build on x86_64, that is a finding rather than a regression: it would mean a workspace only built on macOS without saying so. Co-Authored-By: Claude Opus 5 --- .github/workflows/standards-build.yml | 23 +++++-- .github/workflows/workflow-hygiene.yml | 47 +++++++++++++ .../check-workflow-concurrency-selftest.sh | 61 +++++++++++++++++ scripts/check-workflow-concurrency.sh | 67 +++++++++++++++++++ 4 files changed, 194 insertions(+), 4 deletions(-) create mode 100644 .github/workflows/workflow-hygiene.yml create mode 100755 scripts/check-workflow-concurrency-selftest.sh create mode 100755 scripts/check-workflow-concurrency.sh diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index 31e2abac5..8f1c93109 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -78,14 +78,29 @@ jobs: - { name: map-rs, dir: map/map-rs, runner: linux-arm64, conformance: map-conformance } - { name: aac-rs, dir: aac/aac-rs, runner: linux-arm64, conformance: aac-conformance } - { name: joulecontract-rs, dir: joulecontract/joulecontract-rs, runner: linux-arm64, z3: true } + # Runner placement is by what a workspace actually needs, not by + # habit. The dev Mac hosts BOTH the `macos` runner and the + # `linux-arm64` colima VM, so it served 13 of this repo's 21 + # contexts while jetson-hub (`linux`) sat idle — which is why a PR's + # macOS contexts could wait indefinitely while its `linux` ones + # returned in minutes. + # + # `arl-rs`, `sense-rs`, `joulehook-rs` and `joule-ui-rs` moved to + # `linux` on that basis: each has zero `target_os = "macos"` cfgs, + # zero Apple dependencies, and zero Apple crates in its Cargo.lock, + # so running them on macOS asserted nothing the repo relies on. + # `jouledesk-rs` stays, because it does need it — `libwebrtc` and 19 + # Apple crates in its lockfile. `linux-arm64` placements stay too: + # arm64 is the deployment target, and `jouleclaw-rs` needs + # `+fp16`. - { name: jouledesk-rs, dir: jouledesk/jouledesk-rs, runner: macos } - - { name: arl-rs, dir: arl/arl-rs, runner: macos } - - { name: sense-rs, dir: sense/sense-rs, runner: macos } + - { name: arl-rs, dir: arl/arl-rs, runner: linux } + - { name: sense-rs, dir: sense/sense-rs, runner: linux } - { name: proof-rs, dir: proof/proof-rs, runner: linux, z3: true, conformance: proof-conformance } - - { name: joulehook-rs, dir: joulehook/joulehook-rs, runner: macos } + - { name: joulehook-rs, dir: joulehook/joulehook-rs, runner: linux } - { name: eoc-rs, dir: eoc/eoc-rs, runner: linux, feature_tests: "-p eoc-meter --features cuda" } - { name: mesh-rs, dir: mesh/mesh-rs, runner: linux-arm64 } - - { name: joule-ui-rs, dir: joule-ui/joule-ui-rs, runner: macos } + - { name: joule-ui-rs, dir: joule-ui/joule-ui-rs, runner: linux } # jouleclaw-rs is the one entry whose runner IS a correctness choice. # It reaches candle → ug → gemm, which uses inline aarch64 assembly # (`fmla {0:v}.8h, ...`) requiring the ARMv8.2 FP16 extension. diff --git a/.github/workflows/workflow-hygiene.yml b/.github/workflows/workflow-hygiene.yml new file mode 100644 index 000000000..8047972f8 --- /dev/null +++ b/.github/workflows/workflow-hygiene.yml @@ -0,0 +1,47 @@ +name: workflow hygiene + +# Properties of the workflows themselves, checked rather than remembered. +# +# Today that is one property: every workflow declares a concurrency group that +# varies by ref. None of them did until 2026-10-05, and the consequence was not +# a slow build — it was that CI stopped being usable for anyone. A session +# pushing four versions of a branch queued four complete sets of runs, all of +# which had to drain; one PR's 21 contexts settled none in 70 minutes behind a +# backlog of 25, and two PRs were merged on locally-executed equivalents +# because the gate could not run. One omitted block reintroduces that for the +# whole repo, invisibly: the workflow still passes, it just queues forever +# under load. +# +# This job runs on `linux`, which is jetson-hub — deliberately not the dev Mac, +# which already serves 13 of this repo's 21 contexts through two runners. It +# needs no toolchain and finishes in seconds. + +on: + push: + branches: [main] + pull_request: + paths: + - '.github/workflows/**' + - 'scripts/check-workflow-concurrency.sh' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + concurrency-groups: + name: every workflow supersedes its own queue + runs-on: linux + steps: + - uses: actions/checkout@v4 + + # The check must be able to fail, or it is decoration. The self-test + # breaks the property both ways against a scratch copy and asserts the + # check rejects each. It lives in a script because embedding its Python + # in this `run:` block made the file invalid YAML, which would have meant + # no CI at all. + - name: The check fails closed (self-test) + run: ./scripts/check-workflow-concurrency-selftest.sh + + - name: Every workflow is grouped per ref + run: ./scripts/check-workflow-concurrency.sh diff --git a/scripts/check-workflow-concurrency-selftest.sh b/scripts/check-workflow-concurrency-selftest.sh new file mode 100755 index 000000000..bb591153f --- /dev/null +++ b/scripts/check-workflow-concurrency-selftest.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# +# check-workflow-concurrency-selftest.sh +# +# Proves `check-workflow-concurrency.sh` can fail. A check that cannot fail is +# decoration, and this repo has a documented history of gates that passed +# because nothing they guarded was ever read. +# +# Both ways the property can break are exercised against a scratch copy, never +# the working tree: +# +# 1. a workflow with no top-level `concurrency:` block; +# 2. a group that does not vary by `github.ref` — which would serialise every +# branch against every other, worse than having no group at all. +# +# In a script rather than inline in the workflow on purpose. The first version +# embedded a Python heredoc in a `run:` block, and the heredoc body sat at +# column zero, which made the workflow file invalid YAML. An unparseable +# workflow means no CI at all — the exact failure this change exists to +# prevent, reintroduced by the fix for it. + +set -euo pipefail + +cd "$(dirname "$0")/.." +root=$(pwd) +target=.github/workflows/workflow-hygiene.yml + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +cp -R .github scripts "$tmp/" + +# ── 1. remove the block entirely ──────────────────────────────────────────── +python3 - "$tmp/$target" <<'PY' +import re, sys +p = sys.argv[1] +s = open(p).read() +out = re.sub(r"^concurrency:\n( .*\n)+", "", s, count=1, flags=re.M) +assert out != s, "fixture unchanged: there was no concurrency block to remove" +open(p, "w").write(out) +PY +if ( cd "$tmp" && ./scripts/check-workflow-concurrency.sh ) >/dev/null 2>&1; then + echo "::error::the check passed a workflow with no concurrency block" + exit 1 +fi + +# ── 2. a group that does not vary by ref ──────────────────────────────────── +cp -f "$root/$target" "$tmp/$target" +python3 - "$tmp/$target" <<'PY' +import re, sys +p = sys.argv[1] +s = open(p).read() +out = re.sub(r"^( group:).*$", r"\1 fixed-for-the-whole-repo", s, count=1, flags=re.M) +assert out != s, "fixture unchanged: there was no group line to rewrite" +open(p, "w").write(out) +PY +if ( cd "$tmp" && ./scripts/check-workflow-concurrency.sh ) >/dev/null 2>&1; then + echo "::error::the check passed a group that does not vary by ref" + exit 1 +fi + +echo "self-test: the check rejects both a missing block and a fixed group" diff --git a/scripts/check-workflow-concurrency.sh b/scripts/check-workflow-concurrency.sh new file mode 100755 index 000000000..5b4fa854a --- /dev/null +++ b/scripts/check-workflow-concurrency.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# +# check-workflow-concurrency.sh +# +# Every workflow must declare a concurrency group, so a push supersedes the run +# before it instead of queueing behind it. +# +# Why this is a gate and not a convention. None of the 15 workflows declared +# `concurrency:` until 2026-10-05. A session pushing four versions of a branch +# therefore queued four complete sets of runs, all of which had to drain: one +# PR's 21 contexts settled none in 70 minutes behind a backlog of 25 runs that +# grew faster than it cleared. The repo's CI became unusable for every other +# session, and two PRs were merged on locally-executed equivalents instead of +# on the gate. A gate that cannot run is not a gate. +# +# One omitted `concurrency:` block reintroduces that for the whole repo, and it +# is invisible: the workflow still passes, it just queues forever under load. +# So it is checked rather than remembered. +# +# Runs in seconds with no toolchain, as a step inside an existing job — adding +# a job of its own would spend runner capacity to protect runner capacity. + +set -euo pipefail + +cd "$(dirname "$0")/.." + +fail=0 +count=0 + +for f in .github/workflows/*.yml; do + count=$((count + 1)) + name=$(basename "$f") + + # A top-level key, so match at column zero: a `concurrency:` nested under a + # job is a different (and legal) thing that does not bound the workflow. + if ! grep -qE '^concurrency:' "$f"; then + echo "::error file=$f::$name declares no top-level \`concurrency:\` block." + echo " Add:" + echo " concurrency:" + echo " group: \${{ github.workflow }}-\${{ github.ref }}" + echo " cancel-in-progress: true" + fail=1 + continue + fi + + # The group must vary by ref. A fixed group serialises every branch against + # every other, which is worse than no group at all: one busy branch then + # blocks the repo instead of only itself. + if ! grep -A3 -E '^concurrency:' "$f" | grep -qE 'group:.*github\.ref'; then + echo "::error file=$f::$name has a concurrency group that does not vary by \`github.ref\`." + echo " A fixed group serialises unrelated branches against each other." + fail=1 + fi +done + +if [ "$count" -eq 0 ]; then + echo "::error::no workflows found — this check looked in the wrong place." + exit 1 +fi + +if [ "$fail" -ne 0 ]; then + echo + echo "checked $count workflow(s): at least one would queue instead of superseding." + exit 1 +fi + +echo "workflow concurrency: $count workflow(s), each grouped per ref." -- 2.54.0 (Apple Git-157)