From a303d999e3f11d17cdd7074c643a3a592e59783c Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 01:19:33 -0400 Subject: [PATCH 1/5] ci(standards-build): run all five IOReport meters on macOS, and fail a step whose test never ran The macOS energy job ran only jcp-energy. Four more crates carry an Apple Silicon IOReport meter: proof-cascade, eoc-meter, jouleclaw-energy and sandbox-meter. Each is behind cfg(target_os = "macos"), jouleclaw-energy also behind its `ioreport` feature, and every one of their workspaces is pinned to a linux runner. None of them had ever run on the platform whose private API they call. This replaces this branch's first version (07fb8360), which rested on a wrong premise. That version added three of the crates and left sandbox-meter out with a comment saying it failed where jcp-energy passed "because they read IOReport differently". They do not. Both read the same CPU rail, and on macOS 27 an unprivileged reader does not see that rail move: it is republished only in rare lumps. jcp-energy's old test asserted that the summed rails advanced, which the live GPU rail satisfied whenever something drew on screen. sandbox-meter's asserted the CPU class itself, so it failed. The three steps the first version added passed for the same vacuous reason. The job's header was wrong in the same way. It described a 2026-09-19 zero that had "healed" by 09-21, and cited cumulative_advances_after_cpu_burn as proof that it catches the real symptom. That test was satisfied by the GPU rail, and the jcp-energy fix deletes it. What the job checks now. Each meter uses IOReport only when the CPU rail itself advances in at least two separate intervals while the probe burns CPU, and otherwise falls back to its estimator. Its tests assert against an independent oracle. That makes the job deterministic: green while every meter declines a rail that does not keep moving, red if any selects one. Each step runs with --nocapture and fails unless the output contains the `ioreport liveness:` line. Only the platform-gated frozen-rail test prints it, so a filter, a missing feature or a cfg that leaves zero tests to run can no longer pass as green. Without --nocapture a passing test's eprintln! is captured, which is why the line never appeared in the old job's log. Merge last: the steps run the new tests from the five meter branches, and sandbox-meter's current test fails on main. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/standards-build.yml | 111 +++++++++++++++++++------- 1 file changed, 82 insertions(+), 29 deletions(-) diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index 6e14d7353..1aad59b67 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -198,33 +198,40 @@ jobs: RUSTFLAGS: ${{ matrix.rustflags }} run: cargo run -q -p ${{ matrix.conformance }} --bin ${{ matrix.conformance }} - # ── The macOS energy backend is exercised NOWHERE otherwise ─────────────── + # ── The macOS energy backends are exercised NOWHERE otherwise ───────────── # - # `jcp-energy`'s IOReport meter — the Apple Silicon rung of the energy - # ladder — is behind `#[cfg(target_os = "macos")]`, and `jcp-rs` is pinned to - # the linux runner above. A platform-gated test plus a single-platform runner - # assignment is zero coverage, and it is indistinguishable from passing: the - # code compiles out, the suite reports green, and nothing has been measured. - # Same family as a feature-gated test with no `--features`, with `target_os` - # as the gate. + # Five crates carry an Apple Silicon IOReport energy meter: jcp-energy, + # proof-cascade, eoc-meter, jouleclaw-energy and sandbox-meter. Each is + # behind `#[cfg(target_os = "macos")]` (jouleclaw-energy also behind its + # `ioreport` feature), and every workspace holding one is pinned to a linux + # runner above. A platform-gated test on a single-platform runner is zero + # coverage, and it is indistinguishable from passing. # - # This is not hypothetical. On 2026-09-19 the counter read 0 µJ across a full - # run on the dev Mac — `IoReport::open()` succeeding while `read_uj()` - # returned nothing — and by 2026-09-21 it had healed with no code change in - # between. Both observations came from a human running the suite by hand; - # CI saw neither. A meter that silently reads zero and later recovers is - # exactly the failure a human spot-check is worst at catching. + # What this job checks, and why it used to flake. On macOS 27 an unprivileged + # reader does not see the IOReport "Energy Model" CPU rails (`CPU Energy`, + # `DIE__CPU Energy`) move under load. They are republished only in rare + # lumps: on the M3 Ultra both dies held for over 22 minutes, then jumped + # about 1.3 kJ each in a single 100 ms sample. On the M5 Max the GPU rail + # stays live. The old jcp-energy test asserted that the *summed* rails + # advanced during a CPU burn, and the GPU rail satisfied that whenever + # something drew on screen. It passed by day and failed at night. The + # 2026-09-19 zero that "healed" by 09-21 was this effect, not a recovery. # - # Deliberately narrow: `-p jcp-energy` only. `jcp-rs` stays on linux, so this - # ADDS the platform that was missing rather than trading one for the other, - # and it costs one small crate rather than a second full workspace run. + # Every meter now uses IOReport only when the CPU rail itself advances in at + # least two separate sample intervals while the probe burns CPU. One jump is + # a lump, not a live rail. Otherwise the meter falls back to its estimator + # rung. Its tests assert against an independent oracle that reads the CPU + # rail directly, so this job is deterministic: + # - green while each meter declines a rail that does not keep moving; + # - red if any meter selects one; + # - on a host whose rail is live, red unless CPU work meters non-zero. # - # Verified to catch the real symptom: forcing `read_uj()` to return `Some(0)` - # makes `cumulative_advances_after_cpu_burn` fail with - # `energy must advance: before=0 after=0` — byte-identical to what was - # observed on 2026-09-19. + # Each step fails unless its test printed the `ioreport liveness:` line. That + # line is the only proof the platform-gated test ran and asserted a branch, + # rather than a filter, a feature or a cfg leaving zero tests to pass. It goes + # to stderr and is captured on success, hence `--nocapture` and `2>&1`. macos-energy-backend: - name: macOS energy backend (jcp-energy) + name: macOS energy backends runs-on: macos steps: - uses: actions/checkout@v4 @@ -237,20 +244,66 @@ jobs: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" - - name: Cache cargo registry + target + - name: Cache cargo registry + targets uses: actions/cache@v4 with: path: | ~/.cargo/registry ~/.cargo/git jcp/jcp-rs/target - key: standards-jcp-energy-macos-${{ hashFiles('jcp/jcp-rs/Cargo.lock', 'rust-toolchain.toml') }} + proof/proof-rs/target + eoc/eoc-rs/target + jouleclaw/jouleclaw-rs/target + sandbox/sandbox-rs/target + key: standards-macos-energy-${{ hashFiles('jcp/jcp-rs/Cargo.lock', 'proof/proof-rs/Cargo.lock', 'eoc/eoc-rs/Cargo.lock', 'jouleclaw/jouleclaw-rs/Cargo.lock', 'sandbox/sandbox-rs/Cargo.lock', 'rust-toolchain.toml') }} restore-keys: | - standards-jcp-energy-macos- + standards-macos-energy- - # `--lib` keeps this to the crate's own tests, which is where every - # `cfg(target_os = "macos")` block in this crate lives. - - name: Energy backend tests (the cfg(macos) blocks) + # `--lib` keeps each run to the crate's own tests, which is where every + # `cfg(target_os = "macos")` block lives. + - name: jcp-energy + shell: bash working-directory: jcp/jcp-rs - run: cargo test --lib -p jcp-energy + run: | + set -o pipefail + log=$(mktemp) + cargo test --lib -p jcp-energy -- --nocapture 2>&1 | tee "$log" + grep -q 'ioreport liveness:' "$log" + - name: proof-cascade + shell: bash + working-directory: proof/proof-rs + run: | + set -o pipefail + log=$(mktemp) + cargo test --lib -p proof-cascade -- --nocapture 2>&1 | tee "$log" + grep -q 'ioreport liveness:' "$log" + + - name: eoc-meter + shell: bash + working-directory: eoc/eoc-rs + run: | + set -o pipefail + log=$(mktemp) + cargo test --lib -p eoc-meter -- --nocapture 2>&1 | tee "$log" + grep -q 'ioreport liveness:' "$log" + + # Double-gated: `cfg(all(feature = "ioreport", target_os = "macos"))`. + # A macOS runner alone reaches none of it. + - name: jouleclaw-energy (feature AND platform gated) + shell: bash + working-directory: jouleclaw/jouleclaw-rs + run: | + set -o pipefail + log=$(mktemp) + cargo test --lib -p jouleclaw-energy --features ioreport -- --nocapture 2>&1 | tee "$log" + grep -q 'ioreport liveness:' "$log" + + - name: sandbox-meter + shell: bash + working-directory: sandbox/sandbox-rs + run: | + set -o pipefail + log=$(mktemp) + cargo test --lib -p sandbox-meter -- --nocapture 2>&1 | tee "$log" + grep -q 'ioreport liveness:' "$log" -- 2.54.0 (Apple Git-157) From 6690eafb3dfde9be19a62fd92083b2d404ec54b6 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 18:00:06 -0400 Subject: [PATCH 2/5] ci(standards-build): run each IOReport meter's integration tests on macOS too Each of the five meter crates now has a tests/ file that checks its CPU-rail liveness probe runs exactly once per process: jcp-energy (ioreport_probed_once), sandbox-meter (ioreport_probe_once_per_process), proof-cascade (ioreport_once, ioreport_unopenable), eoc-meter (liveness_probe_once_per_process) and jouleclaw-energy. Those checks only hold in a fresh process, so they are integration tests. `cargo test --lib` skipped them, so a per-thread or re-probing cache would have passed this job. Dropping `--lib` runs them too, and the `ioreport liveness:` guard is unchanged. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/standards-build.yml | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index 1aad59b67..a84af9dff 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -259,15 +259,16 @@ jobs: restore-keys: | standards-macos-energy- - # `--lib` keeps each run to the crate's own tests, which is where every - # `cfg(target_os = "macos")` block lives. + # Each crate's lib tests plus its integration tests. Each crate has a + # tests/ file that checks the liveness probe runs exactly once per process, + # and that check only holds in a fresh process. `--lib` would skip it. - name: jcp-energy shell: bash working-directory: jcp/jcp-rs run: | set -o pipefail log=$(mktemp) - cargo test --lib -p jcp-energy -- --nocapture 2>&1 | tee "$log" + cargo test -p jcp-energy -- --nocapture 2>&1 | tee "$log" grep -q 'ioreport liveness:' "$log" - name: proof-cascade @@ -276,7 +277,7 @@ jobs: run: | set -o pipefail log=$(mktemp) - cargo test --lib -p proof-cascade -- --nocapture 2>&1 | tee "$log" + cargo test -p proof-cascade -- --nocapture 2>&1 | tee "$log" grep -q 'ioreport liveness:' "$log" - name: eoc-meter @@ -285,7 +286,7 @@ jobs: run: | set -o pipefail log=$(mktemp) - cargo test --lib -p eoc-meter -- --nocapture 2>&1 | tee "$log" + cargo test -p eoc-meter -- --nocapture 2>&1 | tee "$log" grep -q 'ioreport liveness:' "$log" # Double-gated: `cfg(all(feature = "ioreport", target_os = "macos"))`. @@ -296,7 +297,7 @@ jobs: run: | set -o pipefail log=$(mktemp) - cargo test --lib -p jouleclaw-energy --features ioreport -- --nocapture 2>&1 | tee "$log" + cargo test -p jouleclaw-energy --features ioreport -- --nocapture 2>&1 | tee "$log" grep -q 'ioreport liveness:' "$log" - name: sandbox-meter @@ -305,5 +306,5 @@ jobs: run: | set -o pipefail log=$(mktemp) - cargo test --lib -p sandbox-meter -- --nocapture 2>&1 | tee "$log" + cargo test -p sandbox-meter -- --nocapture 2>&1 | tee "$log" grep -q 'ioreport liveness:' "$log" -- 2.54.0 (Apple Git-157) From 06ef387c00b91236b1acbe9320dda50adb59a1a5 Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 18:58:51 -0400 Subject: [PATCH 3/5] ci(standards-build): describe the liveness rule the meters now use The job's comment said a meter uses IOReport once the CPU rail advances in two separate intervals. The five meters have since tightened that rule. A refresh can be published in steps: one was caught as +61.8 kJ, then +3.4 J 64 ms later. So two advancing intervals can still be a single refresh. Now the rail has to advance in at least 5 intervals, in at least 4 of every 5, over at least 300 ms. Comment only. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/standards-build.yml | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index a84af9dff..6b46ffc15 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -217,11 +217,13 @@ jobs: # something drew on screen. It passed by day and failed at night. The # 2026-09-19 zero that "healed" by 09-21 was this effect, not a recovery. # - # Every meter now uses IOReport only when the CPU rail itself advances in at - # least two separate sample intervals while the probe burns CPU. One jump is - # a lump, not a live rail. Otherwise the meter falls back to its estimator - # rung. Its tests assert against an independent oracle that reads the CPU - # rail directly, so this job is deterministic: + # Every meter now uses IOReport only when the CPU rail itself keeps + # advancing while the probe burns CPU: in at least 5 sample intervals, at + # least 4 in every 5, over at least 300 ms. A lump is published in steps (one + # was caught as +61.8 kJ, then +3.4 J 64 ms later), so two advancing + # intervals are still just a lump. Otherwise the meter falls back to its + # estimator rung. Its tests assert against an independent oracle that reads + # the CPU rail directly, so this job is deterministic: # - green while each meter declines a rail that does not keep moving; # - red if any meter selects one; # - on a host whose rail is live, red unless CPU work meters non-zero. -- 2.54.0 (Apple Git-157) From 3354465adf371eedd87525e0808e0d170a408a9e Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Wed, 30 Sep 2026 20:12:08 -0400 Subject: [PATCH 4/5] ci: run joule-code-remediate's formal gate against a real z3 joule-code-remediate's formal gate shells out to z3, and neither job that tests it installed a solver: joule-code-rs in standards-build, and the joule-code conformance gate. Its z3 tests used to return early without a solver, so CI passed without ever running the real-solver path. sandbox-meter/cpu-rail-liveness (#114) makes those tests assert in both branches, and adds JOULE_FIX_EXPECT_Z3=1 to turn a missing solver into a failure. joule-code-rs now joins the `z3: true` matrix entries, as joulecontract-rs already does on the same runner kind. The conformance gate installs z3 the same way. Both set JOULE_FIX_EXPECT_Z3=1 where z3 was installed. Elsewhere the variable is empty, and only joule-code- remediate reads it. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/joule-code-conformance.yml | 14 ++++++++++++++ .github/workflows/standards-build.yml | 13 ++++++++++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/.github/workflows/joule-code-conformance.yml b/.github/workflows/joule-code-conformance.yml index 2c433079c..8e59d8707 100644 --- a/.github/workflows/joule-code-conformance.yml +++ b/.github/workflows/joule-code-conformance.yml @@ -53,8 +53,22 @@ jobs: joule-code/joule-code-rs/target key: joule-code-${{ runner.os }}-${{ hashFiles('joule-code/joule-code-rs/Cargo.lock') }} + # joule-code-remediate's formal gate shells out to `z3`, and the runner + # image has no solver. Its tests assert in both branches. With a solver + # installed here, JOULE_FIX_EXPECT_Z3=1 makes its absence a failure, so + # this gate runs the real-solver branch rather than the fail-closed one. + - name: Install z3 + shell: bash + run: | + SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo" + $SUDO apt-get update -qq + $SUDO apt-get install -y -qq z3 + z3 --version + - name: Reference library tests working-directory: joule-code/joule-code-rs + env: + JOULE_FIX_EXPECT_Z3: '1' run: cargo test --workspace # joule-moq's QUIC transport and its example compile only with `quic`. Named diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index 6b46ffc15..7ee13e7f2 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -54,7 +54,7 @@ jobs: - { name: sandbox-rs, dir: sandbox/sandbox-rs, runner: linux, conformance: sandbox-conformance } - { name: smart-byte-rs, dir: smart-byte/smart-byte-rs, runner: linux } - { name: openpay, dir: openpay, runner: linux-arm64 } - - { name: joule-code-rs, dir: joule-code/joule-code-rs, runner: linux-arm64 } + - { name: joule-code-rs, dir: joule-code/joule-code-rs, runner: linux-arm64, z3: true } - { name: map-rs, dir: map/map-rs, runner: linux-arm64, conformance: map-conformance } - { name: aac-rs, dir: aac/aac-rs, runner: linux-arm64, conformance: aac-conformance } - { name: joulecontract-rs, dir: joulecontract/joulecontract-rs, runner: linux-arm64, z3: true } @@ -86,7 +86,7 @@ jobs: steps: - uses: actions/checkout@v5 - # Two workspaces shell out to `z3`, and the runner image has no solver. + # Three workspaces shell out to `z3`, and the runner image has no solver. # # This step used to be load-bearing for correctness: without a solver the # ladder escalated past its real rungs into a model STAND-IN that reported @@ -101,7 +101,12 @@ jobs: # stay covered either way — that regression test uses a goal z3 cannot # parse, so it asserts the same thing on a machine that has one. # - # Both z3 jobs are pinned to Linux runners deliberately: the macOS runner is + # joule-code-remediate's formal gate is the third. Its tests assert in both + # branches, so they pass without a solver, and JOULE_FIX_EXPECT_Z3=1 below + # makes a missing z3 fail them where this step installed one. Otherwise CI + # would only ever run the no-solver branch. + # + # The z3 jobs are pinned to Linux runners deliberately: the macOS runner is # a host executor with no apt. `sudo` may or may not exist depending on # whether the container runs as root, so fall back to a bare invocation. - name: Install z3 @@ -167,6 +172,8 @@ jobs: working-directory: ${{ matrix.dir }} env: RUSTFLAGS: ${{ matrix.rustflags }} + # Where z3 was installed above, its absence is a failure, not a skip. + JOULE_FIX_EXPECT_Z3: ${{ matrix.z3 && '1' || '' }} run: cargo test --workspace # Run each standard's conformance suite. -- 2.54.0 (Apple Git-157) From 4e23637dbac345698eb6661e94b63aa53736dc0e Mon Sep 17 00:00:00 2001 From: Transaction Science Date: Thu, 1 Oct 2026 02:13:15 -0400 Subject: [PATCH 5/5] ci: build and run the energy meters' NVML-gated tests jcp-energy `nvml`, eoc-meter `cuda` and jouleclaw-energy `nvml` are features no workspace member enables, so `cargo test --workspace` never compiled the NVML backends, nor ran the tests behind them. That includes #129's plausibility rule, which marks a counter that runs past its board Implausible. A matrix field `feature_tests` now runs those crates once more with the feature on. In the container there is no GPU and powercap is masked, so the hardware tests take their "unavailable, and the oracle agrees" branch. The unit tests behind the features run in full. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/standards-build.yml | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/.github/workflows/standards-build.yml b/.github/workflows/standards-build.yml index 7ee13e7f2..cd5c084ef 100644 --- a/.github/workflows/standards-build.yml +++ b/.github/workflows/standards-build.yml @@ -50,7 +50,7 @@ jobs: fail-fast: false matrix: include: - - { name: jcp-rs, dir: jcp/jcp-rs, runner: linux, conformance: jcp-conformance } + - { name: jcp-rs, dir: jcp/jcp-rs, runner: linux, conformance: jcp-conformance, feature_tests: "-p jcp-energy --features nvml" } - { name: sandbox-rs, dir: sandbox/sandbox-rs, runner: linux, conformance: sandbox-conformance } - { name: smart-byte-rs, dir: smart-byte/smart-byte-rs, runner: linux } - { name: openpay, dir: openpay, runner: linux-arm64 } @@ -63,7 +63,7 @@ jobs: - { name: sense-rs, dir: sense/sense-rs, runner: macos } - { name: proof-rs, dir: proof/proof-rs, runner: linux, z3: true, conformance: proof-conformance } - { name: joulehook-rs, dir: joulehook/joulehook-rs, runner: macos } - - { name: eoc-rs, dir: eoc/eoc-rs, runner: linux } + - { name: eoc-rs, dir: eoc/eoc-rs, runner: linux, feature_tests: "-p eoc-meter --features cuda" } - { name: mesh-rs, dir: mesh/mesh-rs, runner: linux-arm64 } - { name: joule-ui-rs, dir: joule-ui/joule-ui-rs, runner: macos } # jouleclaw-rs is the one entry whose runner IS a correctness choice. @@ -82,7 +82,7 @@ jobs: # failing for no apparent reason, check `df -h /mnt/lima-colima-txarm` # on the txarm VM before suspecting the code: a full cache volume # fails whatever happens to be building, not whatever changed. - - { name: jouleclaw-rs, dir: jouleclaw/jouleclaw-rs, runner: linux-arm64, rustflags: "-C target-feature=+fp16", cache_suffix: "-nodebug" } + - { name: jouleclaw-rs, dir: jouleclaw/jouleclaw-rs, runner: linux-arm64, rustflags: "-C target-feature=+fp16", cache_suffix: "-nodebug", feature_tests: "-p jouleclaw-energy --features rapl,nvml" } steps: - uses: actions/checkout@v5 @@ -176,6 +176,19 @@ jobs: JOULE_FIX_EXPECT_Z3: ${{ matrix.z3 && '1' || '' }} run: cargo test --workspace + # The NVML backends sit behind features no workspace member enables + # (jcp-energy `nvml`, eoc-meter `cuda`, jouleclaw-energy `nvml`), so the + # step above never compiles them, let alone runs their tests. Here they + # build and their tests run. The container has no GPU and Docker masks + # powercap, so the hardware tests take their "unavailable, and the + # oracle agrees" branch; the rule's unit tests run in full. + - name: Feature-gated energy tests + if: matrix.feature_tests + working-directory: ${{ matrix.dir }} + env: + RUSTFLAGS: ${{ matrix.rustflags }} + run: cargo test ${{ matrix.feature_tests }} + # Run each standard's conformance suite. # # These suites live in a binary's `main()`, not in `#[test]`, so -- 2.54.0 (Apple Git-157)