CI: the process-calibration step retries a run a busy GPU left unmetered #476

Merged
dcharlot merged 1 commit from ci/energy-step-retry into main 2026-10-26 04:30:06 -04:00

View file

@ -558,6 +558,14 @@ jobs:
# and macOS. The fabric calibration rows' macOS stamp, and a CPU member
# priced on this process's own counters against its own energy count
# (tests/process_calibration.rs), each print their line only there.
#
# The calibration test meters the GPU counter only once a GPU liveness load has run, and
# that load gives up after 500 ms on a GPU too busy to grant it. The test then takes its
# refusal branch and passes, but prints no `process calibration:` line, so the grep fails
# on host load alone (seen 2026-10-25, the runner's host at load 55). Its own retries
# keep that from costing a whole pull request's rerun: up to three runs, 20 s apart, and
# the line must still come from a run on live counters. A failing assertion fails the
# step at once, as before.
- name: jouleclaw-loader-gguf (fabric-meter, process calibration)
shell: bash
working-directory: jouleclaw/jouleclaw-rs
@ -565,7 +573,12 @@ jobs:
set -o pipefail
log=$(mktemp)
cargo test -p jouleclaw-loader-gguf --features "rayon fabric-meter" marginal_energy -- --nocapture 2>&1 | tee "$log"
cargo test -p jouleclaw-loader-gguf --features "rayon fabric-meter" --test process_calibration -- --nocapture 2>&1 | tee -a "$log"
for attempt in 1 2 3; do
cargo test -p jouleclaw-loader-gguf --features "rayon fabric-meter" --test process_calibration -- --nocapture 2>&1 | tee -a "$log"
grep -q 'process calibration: CPU member' "$log" && break
echo "process calibration: attempt $attempt ran without live GPU counters (a busy GPU); trying again in 20 s"
sleep 20
done
grep -q 'calibration row stamped on macOS:' "$log"
grep -q 'process calibration: CPU member' "$log"