jcp-energy: IOReport only when its CPU rail keeps advancing, never on a batch refresh #110

Merged

dcharlot merged 10 commits from jcp-energy/cpu-rail-liveness into main 2026-10-01 05:45:02 -04:00

Conversation 0 Commits 10 Files changed 7 +5352 -156

dcharlot commented 2026-09-29 10:08:35 -04:00

Owner

Copy link

jcp-energy's detect() selected its IOReport rung the moment a subscription opened. measure() then took any rise in the summed rails as the task's energy. That contradicts this crate's own contract: "measured when the platform exposes it, estimated when it doesn't, and the provenance always says which … A reading never silently upgrades its own provenance."

What was wrong: on main

• M5 Max: the live GPU rail makes the sum rise, so GPU energy was reported as the energy of CPU work, as a whole-task ModelBased figure.
• M3 Ultra: nothing moves, so it fell back to time×power, but under the same tag.
• The macOS energy backend (jcp-energy) CI job flaked for the same reason. Its test asserted the summed total advanced during a CPU burn, which the GPU satisfies whenever something draws on screen. It passed by day and failed at night; this was the "healed" zero of 2026-09-19.

What this does

• detect() selects IOReport only when ioreport_status() is Live, under the rule below. Otherwise it falls back to TimePower (ModelBased), and ioreport_status() says why.
• The flaky aggregate test is replaced by tests against an independent oracle, so the CI job is deterministic.
• The IOReport counter counts CPU energy once. It no longer sums per-cluster and power-manager sub-rails beside the aggregate they add up to. This only matters on a host whose rail is live.

Verification of this PR (head 3fd35db5)

• Mutations: the round-6 run caught 17 of 17. Restoring the old two-advance rule fails 15 tests, including the two-step refresh run through the real host_status(). The old oracle is shown to vouch for a refresh: CpuRailMoved(true) on a two-step lump.
• Robustness: passed 5/5, then 3/3, beside an all-channel IOReport sampler taking 144–313 ms per sample.
• Both chips: M5 Max and M3 Ultra each give status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false.
• Dependants: jcp-cli, jcp-conformance (the binary prints "all checks passed", C9 included) and jcp-intercept are green.
• Count CPU energy once: per-cluster and power-manager sub-rails are no longer summed beside CPU Energy. The hierarchy was confirmed from absolute values on both chips (CPU Energy = MCPU0 + MCPU1 + PCPU, to within 1–7 mJ).
• Declined path: when IOReport is declined, detect() falls back to TimePower (ModelBased), and 100 ms of real CPU work is checked to meter non-zero through it.

Later rounds on this branch (after the verification above)

• 340313d6: a live IOReport reading is graded HwShunt, the same as RAPL and NVML in this crate. A span is a reading only if the CPU rail itself advanced across it. intercept/1 carries coverage: Partial for a partial figure; the key is omitted when Whole, so old receipts are byte-identical (pinned against goldens). SPEC §5 has the liveness MUST and a coverage section.
• 475680ab: RAPL and NVML are gated on the same liveness rule, through one shared module. An NVML figure is measured_partial. A span where no counter rose gets the time×power figure, never a hardware 0.
• 4f9d90ca: a counter is live only if every rail it sums keeps advancing: each die for IOReport, each package for RAPL. A host with no rail is never Live.
• 25bfdcd9: RAPL sums only the package-N zones, never psys or dram. Each zone wraps at its own range; an unreadable range used to default to u64::MAX and seal a ~1.8×10¹⁹ µJ figure.

Why: what IOReport actually does on macOS 27.0.1

Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1:

• CPU rails are republished in rare lumps. For an unprivileged reader, CPU Energy and DIE_<n>_CPU Energy do not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each.
• A lump can arrive in steps. A watcher sampling every ~16 ms caught one as +61,764,485 mJ, then +3,435 mJ 64 ms later.
• GPU Energy is live on the M5 Max (it changes every ~28 ms) and sporadic on the M3 Ultra.

A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it.

The rule: identical in all five backends (#110, #114, #115, #116, #117)

• The probe opens its own subscription to the "Energy Model" group. A sample costs about 6–12 ms of thread CPU, against about 115 ms for all channels.
• It decides per CPU rail (CPU Energy, or each DIE_<n>_CPU Energy), keyed by name and ordinal:
  • it compares raw integers from consecutive samples;
  • a decrease, or a rail's first appearance, is not an advance;
  • a rail advances at most once per interval.
• Live means every rail the meter sums kept advancing: in at least 5 intervals, in at least 4 of every 5 intervals so far, over a window of at least 300 ms.
  • A live rail meets that in about 300 ms. A lump does not, even one published in several steps.
  • Every rail is required because the meter sums them. One live die beside a frozen die would otherwise give a figure that silently misses the frozen die while claiming the whole CPU.
• The same rule gates RAPL and NVML.
  • RAPL sums the top-level zones whose name reads package-N, on every socket, and each must be live. It never sums psys, which already includes the packages, nor dram or the sub-zones.
  • Each zone wraps at its own max_energy_range_uj. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading.
  • An NVML (GPU-only) figure is Partial where the format can say so. Where it can't, it is never a hardware reading of the whole task.
• It samples until Live, or until the 1 s budget has elapsed and at least 5 intervals have been seen, with a cap of 5 budgets.
  • Each tick burns 25 ms of this thread's CPU time, not wall time, with a 500 ms wall cap. So the probe keeps working on a loaded host.
• The result is cached once per process. Anything but Live declines the backend, so the crate's existing honest fallback runs. The reason is public: IoReportStatus, ioreport_status(), rapl_status(), nvml_status().
• The production status is one branch-free composition, counted each time it runs. Every selection path reads it only through the cached status and a pure admits(status) (Live only).
• A live vendor counter's reading is a hardware reading: HwShunt, or Measured in eoc. That follows the owner's position that vendor counters are real readings and the trust gap is provenance. It holds only while the counter is live, per the MUST in #119.

How it was verified

Six rounds of implementation, each checked by independent adversarial verification.

Pure tests on scripted samples and a scripted clock. They cover:

• the real refreshes (one step, two steps, 3–5 steps, the two-die shape);
• decreases, first appearances and failed samples;
• live rails (every interval, 80% of intervals, on a loaded host);
• the sampling-loop bounds.

Fault hooks inside the real IOReport opener and sampler. The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test.

Hardware tests against an independent oracle. The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check:

• the probe's sampled values equal the oracle's reads on either side;
• the live GPU rail shows up in the probe's record, which proves the reads are fresh;
• every probe sample is group-only, by channel count;
• thread CPU per sample and per tick is checked against the tests' own clock;
• the probe runs exactly once per process, checked in a fresh process.

Mutation-tested. Each of the following was applied as a temporary edit and failed a test:

• restore the old one- or two-advance rule;
• compare every sample with the first;
• treat a decrease as an advance;
• map "no subscription" to Live inside the production status;
• use a no-op tick, a tick that sleeps, or a tick on the wrong clock;
• use a per-thread or non-atomic cache;
• probe uncached from the constructor;
• a sampler that returns constant, stale or summed values;
• an all-channel subscription;
• decide on the summed total or on the GPU rail;
• an admits() that accepts CpuRailAbsent.

The counts per crate are below.

Robustness. Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra.

CI: these tests are behind cfg(target_os = "macos"), and this workspace runs on a linux runner. They reach CI through #102, which merges last. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not print ioreport liveness:.

Merge order: #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them.

Open questions, now decided and implemented (per "fix it all")

• One grade for a live vendor counter. It is HwShunt, or Measured in eoc, everywhere: a vendor counter is a real reading, and the gap is provenance. It holds only while the counter is live (the MUST in #119).
• The GPU rail as a partial reading of CPU work: no. The GPU rail is machine-wide (display compositing moves it). Booking it to a task would misattribute energy and under-debit budgets.
• Wire keys named measured_uj that carry estimates. Their grade now sits beside them in plaintext: jcp-gateway's _meta gets energy_provenance, plus energy_coverage when partial. The keys stay, for compatibility.
• eoc cache hits are an estimated zero, never Measured.
• joule-ui. Meter defaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. Authorizer::charge carries the figure's provenance.
• A live CPU energy source on macOS 27 exists. It is the kernel's per-process counter ri_energy_nj: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privileged powermetrics is live too, but machine-wide and root-only, so it is not used.)

Still open

• The joule-ui demo on prod. It has run on macOS 27 since 2026-09-15. Its sealed chain was not inspected, because the read was blocked as a production read. After #114 merges, redeploying joule-ui-serve gives it honest grades, and the per-process probe in #121 gives it real readings.
• Linux RAPL and NVML have never run against real hardware. No Linux host with readable RAPL and no NVIDIA GPU were available. They are covered by fake-tree and scripted tests, type-checked for Linux, and declined fail-safe.

🤖 Generated with Claude Code

jcp-energy's `detect()` selected its IOReport rung the moment a subscription opened. `measure()` then took any rise in the **summed** rails as the task's energy. That contradicts this crate's own contract: *"measured when the platform exposes it, estimated when it doesn't, and the provenance always says which … A reading never silently upgrades its own provenance."* ### What was wrong: on main - **M5 Max:** the live GPU rail makes the sum rise, so GPU energy was reported as the energy of CPU work, as a whole-task ModelBased figure. - **M3 Ultra:** nothing moves, so it fell back to time×power, but under the same tag. - **The `macOS energy backend (jcp-energy)` CI job flaked for the same reason.** Its test asserted the summed total advanced during a CPU burn, which the GPU satisfies whenever something draws on screen. It passed by day and failed at night; this was the "healed" zero of 2026-09-19. ### What this does - **`detect()` selects IOReport only when `ioreport_status()` is `Live`,** under the rule below. Otherwise it falls back to TimePower (ModelBased), and `ioreport_status()` says why. - **The flaky aggregate test is replaced** by tests against an independent oracle, so the CI job is deterministic. - **The IOReport counter counts CPU energy once.** It no longer sums per-cluster and power-manager sub-rails beside the aggregate they add up to. This only matters on a host whose rail is live. ### Verification of this PR (head `3fd35db5`) - **Mutations:** the round-6 run caught 17 of 17. Restoring the old two-advance rule fails 15 tests, including the two-step refresh run through the real `host_status()`. The old oracle is shown to vouch for a refresh: `CpuRailMoved(true)` on a two-step lump. - **Robustness:** passed 5/5, then 3/3, beside an all-channel IOReport sampler taking 144–313 ms per sample. - **Both chips:** M5 Max and M3 Ultra each give `status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false`. - **Dependants:** jcp-cli, jcp-conformance (the binary prints "all checks passed", C9 included) and jcp-intercept are green. - **Count CPU energy once:** per-cluster and power-manager sub-rails are no longer summed beside `CPU Energy`. The hierarchy was confirmed from absolute values on both chips (`CPU Energy` = `MCPU0` + `MCPU1` + `PCPU`, to within 1–7 mJ). - **Declined path:** when IOReport is declined, detect() falls back to TimePower (ModelBased), and 100 ms of real CPU work is checked to meter non-zero through it. ### Later rounds on this branch (after the verification above) - `340313d6`: a live IOReport reading is graded HwShunt, the same as RAPL and NVML in this crate. A span is a reading only if the CPU rail itself advanced across it. intercept/1 carries `coverage: Partial` for a partial figure; the key is omitted when Whole, so old receipts are byte-identical (pinned against goldens). SPEC §5 has the liveness MUST and a coverage section. - `475680ab`: RAPL and NVML are gated on the same liveness rule, through one shared module. An NVML figure is `measured_partial`. A span where no counter rose gets the time×power figure, never a hardware 0. - `4f9d90ca`: a counter is live only if *every* rail it sums keeps advancing: each die for IOReport, each package for RAPL. A host with no rail is never Live. - `25bfdcd9`: RAPL sums only the `package-N` zones, never psys or dram. Each zone wraps at its own range; an unreadable range used to default to `u64::MAX` and seal a ~1.8×10¹⁹ µJ figure. ### Why: what IOReport actually does on macOS 27.0.1 Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1: - **CPU rails are republished in rare lumps.** For an unprivileged reader, `CPU Energy` and `DIE_<n>_CPU Energy` do not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each. - **A lump can arrive in steps.** A watcher sampling every ~16 ms caught one as `+61,764,485 mJ`, then `+3,435 mJ` 64 ms later. - **GPU Energy is live on the M5 Max** (it changes every ~28 ms) and sporadic on the M3 Ultra. A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it. ### The rule: identical in all five backends (#110, #114, #115, #116, #117) - **The probe opens its own subscription to the "Energy Model" group.** A sample costs about 6–12 ms of thread CPU, against about 115 ms for all channels. - **It decides per CPU rail** (`CPU Energy`, or each `DIE_<n>_CPU Energy`), keyed by name and ordinal: - it compares raw integers from consecutive samples; - a decrease, or a rail's first appearance, is not an advance; - a rail advances at most once per interval. - **Live means every rail the meter sums kept advancing:** in at least 5 intervals, in at least 4 of every 5 intervals so far, over a window of at least 300 ms. - A live rail meets that in about 300 ms. A lump does not, even one published in several steps. - Every rail is required because the meter sums them. One live die beside a frozen die would otherwise give a figure that silently misses the frozen die while claiming the whole CPU. - **The same rule gates RAPL and NVML.** - RAPL sums the top-level zones whose `name` reads `package-N`, on every socket, and each must be live. It never sums `psys`, which already includes the packages, nor `dram` or the sub-zones. - Each zone wraps at its own `max_energy_range_uj`. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading. - An NVML (GPU-only) figure is Partial where the format can say so. Where it can't, it is never a hardware reading of the whole task. - **It samples until Live,** or until the 1 s budget has elapsed and at least 5 intervals have been seen, with a cap of 5 budgets. - Each tick burns 25 ms of this thread's CPU time, not wall time, with a 500 ms wall cap. So the probe keeps working on a loaded host. - **The result is cached once per process.** Anything but `Live` declines the backend, so the crate's existing honest fallback runs. The reason is public: `IoReportStatus`, `ioreport_status()`, `rapl_status()`, `nvml_status()`. - **The production status is one branch-free composition,** counted each time it runs. Every selection path reads it only through the cached status and a pure `admits(status)` (Live only). - **A live vendor counter's reading is a hardware reading:** HwShunt, or Measured in eoc. That follows the owner's position that vendor counters are real readings and the trust gap is provenance. It holds only while the counter is live, per the MUST in #119. ### How it was verified Six rounds of implementation, each checked by independent adversarial verification. **Pure tests on scripted samples and a scripted clock.** They cover: - the real refreshes (one step, two steps, 3–5 steps, the two-die shape); - decreases, first appearances and failed samples; - live rails (every interval, 80% of intervals, on a loaded host); - the sampling-loop bounds. **Fault hooks inside the real IOReport opener and sampler.** The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test. **Hardware tests against an independent oracle.** The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check: - the probe's sampled values equal the oracle's reads on either side; - the live GPU rail shows up in the probe's record, which proves the reads are fresh; - every probe sample is group-only, by channel count; - thread CPU per sample and per tick is checked against the tests' own clock; - the probe runs exactly once per process, checked in a fresh process. **Mutation-tested.** Each of the following was applied as a temporary edit and failed a test: - restore the old one- or two-advance rule; - compare every sample with the first; - treat a decrease as an advance; - map "no subscription" to Live inside the production status; - use a no-op tick, a tick that sleeps, or a tick on the wrong clock; - use a per-thread or non-atomic cache; - probe uncached from the constructor; - a sampler that returns constant, stale or summed values; - an all-channel subscription; - decide on the summed total or on the GPU rail; - an `admits()` that accepts CpuRailAbsent. The counts per crate are below. **Robustness.** Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra. **CI:** these tests are behind `cfg(target_os = "macos")`, and this workspace runs on a linux runner. They reach CI through **#102**, which merges **last**. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not print `ioreport liveness:`. **Merge order:** #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them. ### Open questions, now decided and implemented (per "fix it all") - **One grade for a live vendor counter.** It is HwShunt, or Measured in eoc, everywhere: a vendor counter is a real reading, and the gap is provenance. It holds only while the counter is live (the MUST in #119). - **The GPU rail as a partial reading of CPU work: no.** The GPU rail is machine-wide (display compositing moves it). Booking it to a task would misattribute energy and under-debit budgets. - **Wire keys named `measured_uj` that carry estimates.** Their grade now sits beside them in plaintext: jcp-gateway's `_meta` gets `energy_provenance`, plus `energy_coverage` when partial. The keys stay, for compatibility. - **eoc cache hits** are an estimated zero, never Measured. - **joule-ui.** `Meter` defaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. `Authorizer::charge` carries the figure's provenance. - **A live CPU energy source on macOS 27 exists.** It is the kernel's per-process counter `ri_energy_nj`: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privileged `powermetrics` is live too, but machine-wide and root-only, so it is not used.) ### Still open - **The joule-ui demo on prod.** It has run on macOS 27 since 2026-09-15. Its sealed chain was not inspected, because the read was blocked as a production read. After #114 merges, redeploying `joule-ui-serve` gives it honest grades, and the per-process probe in #121 gives it real readings. - **Linux RAPL and NVML have never run against real hardware.** No Linux host with readable RAPL and no NVIDIA GPU were available. They are covered by fake-tree and scripted tests, type-checked for Linux, and declined fail-safe. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 1 commit 2026-09-29 10:08:35 -04:00

jcp-energy: never select IOReport when its CPU rail is frozen ... dfec5d44b0

This crate's contract, in its own module doc: "measured when the platform
exposes it, estimated when it doesn't, and the provenance always says which ...
A reading never silently upgrades its own provenance." The IOReport rung broke
that on current macOS, and the CI job meant to catch it could not.

What IOReport actually returns on macOS 27.0.1, measured by dumping the
channels rather than re-running the test:

- On an M5 Max, every CPU rail in the Energy Model group (`CPU Energy`, `PCPU`,
  `MCPU*`, `PACC_*`) holds the same cumulative value across 200 ms, 1 s, 3 s and
  ~4 minutes of full CPU load. `CPU Energy` read 73,176,481 mJ before and after.
  `GPU Energy` stays live.
- On an M3 Ultra, the same OS: all 645 Energy Model rails are frozen, CPU,
  DRAM and display included.
- Every channel is IOReport format 1 (Simple), so the accessor is right, and
  subscribing to the Energy Model group alone instead of all channels does not
  unfreeze anything. No channel in any group exposes live CPU energy to an
  unprivileged reader; "CPU Stats" residency moves, but that is activity.

So `detect()` selected IOReport the moment a subscription opened, and
`measure()` accepted any `a > b` on the summed rails as the task's energy. On
the M5 Max the GPU rail makes `a > b` true: GPU energy was reported as the energy
of CPU work, as a whole-task ModelBased figure. On the M3 Ultra nothing moves,
so it fell to time×power under the same tag, indistinguishable from a reading.

The CI job `macOS energy backend (jcp-energy)` flaked for the same reason. Its
test asserted the summed total advanced across a 200 ms CPU burn — satisfied by
the GPU, not the CPU. It passed when something happened to draw on screen and
failed when nothing did, which is why it failed overnight and "healed" by day.
That flake has been blocking unrelated PRs.

The fix enforces the contract where the rung is chosen. `detect()` selects
IOReport only when `ioreport_status()` is `Live`: the CPU rail itself — `CPU
Energy`, or `DIE_<n>_CPU Energy` on multi-die parts — must advance while this
thread burns CPU, polled every ~25 ms for up to one second, probed once per
process and cached. Liveness is decided on the raw integer before unit
conversion, and never on the summed total, because the live GPU rail is exactly
what corrupts the total. Otherwise the rung is declined and the time×power model
estimates, as the contract requires. `IoReportStatus` (`Unavailable`,
`CpuRailAbsent`, `CpuRailFrozen`, `Live`) is exposed so a caller or receipt can
say why an Apple Silicon host is being estimated. The change is additive:
`mod ioreport` is private and nothing outside the crate matches on `Meter`.

Tests. The flaky test is replaced by three that assert against an independent
oracle: it reads the CPU rail's ABSOLUTE cumulative value with no delta
samples, matches names with `ends_with("CPU Energy")` rather than the
implementation's die-prefix stripping, and burns for three times the probe's
budget. There is no early return that passes without asserting — a host with no
IOReport must report `Unavailable`, one with no CPU rail `CpuRailAbsent`.

- `the_ioreport_rung_is_never_selected_on_a_frozen_cpu_rail` — the property that
  matters. It logs what it observed so a CI log shows which branch was asserted.
- `the_liveness_probe_never_reports_a_frozen_rail_as_live` — the dangerous
  direction, asserted on its own.
- `a_selected_ioreport_meter_meters_cpu_work` — when the rung IS selected, a CPU
  burn meters as non-zero energy.

Observed on both machines: `status=CpuRailFrozen oracle=CpuRailMoved(false)
selected=false`, 10/10 passing — on the M3 Ultra via the `DIE_0_`/`DIE_1_` rails,
which confirms the multi-die path. The CI job becomes deterministic: green while
the meter correctly declines a frozen rail, red if it ever selects one.

Falsified: a probe that always says Live fails the test (the oracle saw the rail
hold); reverting `detect()` to select IOReport whenever it opens — main's
behaviour — fails it with "IOReport selected although its CPU rail did not
move", so main fails this test today.

jcp-cli, jcp-conformance and jcp-intercept (every dependant of this crate) pass,
as does the conformance suite's C9 energy-honesty check; clippy is clean on
jcp-energy. jcp-gateway fails on main as well and does not depend on this crate
— it meters through sandbox-meter's probe, which reads the same frozen rail.

Not done here, deliberately:
- A per-span check for a rail that freezes after `detect()` needs "CPU Stats"
  residency as a witness; the platform-level freeze observed here is caught at
  selection.
- Reporting the live GPU rail as an explicitly partial reading, instead of
  declining, would need a coverage field on the intercept/1 wire — a format
  change, so a separate decision.
- The same defect in jouleclaw-energy, eoc-meter, proof-cascade and
  sandbox-meter, and in the callers that seal a zero delta as a measurement.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request from a commit 2026-09-29 10:52:35 -04:00

proof-cascade: never meter CPU work through a frozen IOReport CPU rail

dcharlot referenced this pull request from a commit 2026-09-29 11:18:54 -04:00

eoc-meter: never construct the IOReport counter on a frozen CPU rail

dcharlot added 1 commit 2026-09-30 01:34:19 -04:00

jcp-energy: IOReport is live only when one CPU rail keeps advancing ... 2518f0d70a

The liveness gate added in dfec5d44 enforces the contract at selection, but
it had three defects. Each was measured on the M5 Max and the M3 Ultra (both
macOS 27.0.1).

1. One advance counted as Live. For an unprivileged reader the CPU rails are
   not permanently frozen: they are refreshed in rare batches. On the M3 Ultra
   both dies held for at least 22 minutes, then stepped +1,344,869 and
   +1,391,637 mJ in a single 100 ms sample. On the M5 Max the rail went from
   480,083,160 to 541,893,638 mJ in about 17 minutes, then did not change
   across 13,614 samples in a 25-minute watch. A probe whose window straddles
   a refresh sees one jump and calls the rail Live; this happened on the M5
   Max at ~10:46 on 2026-09-29. The rung then stays selected for the life of
   the process, and each later ~1.3 kJ refresh lands whole in whichever
   metered span straddles it, tagged ModelBased. Both machines have been
   refreshed again since: today the M5 Max rail reads 1,308,503,606 mJ and the
   M3 Ultra's read 21,170,718 / 23,300,102 mJ. Under load they still do not
   move.

2. The probe sampled the counter's all-channel subscription. Its doc said
   "re-sampling every ~25 ms", but one IOReportCreateSamples over every
   channel took 130-160 ms on the M5 Max today (0.4-0.8 s on 2026-09-29) and
   ~135 ms on the M3 Ultra. Instrumented: the probe took 7 samples in ~1.09 s
   on the M5 Max and burned CPU for only 175 ms of that; on the M3 Ultra, 6
   samples and 150 ms. It also compared every sample with the first one
   instead of with the previous one.

3. The oracle burned CPU on its sampling thread, between samples, so most of
   its "three times the budget" window was not under load. It burned for
   0.90-0.95 s of ~3.1 s on the M5 Max and 0.80 s on the M3 Ultra with
   today's ~130 ms samples; with 0.4-0.8 s samples that falls to ~250 ms. It
   also returned CpuRailMoved(true) on any change in the absolute value, and
   a single refresh is a change.

The fix ports the family liveness rule: RailWatch from jouleclaw-energy
(b9d7e945) and the group-only probe subscription from eoc-meter (c6378629).
The code is copied in, so the crate gains no dependency.

- The probe opens its own subscription to the "Energy Model" group only
  (IOReportCopyChannelsInGroup, then IOReportCreateSubscription). One sample
  costs 6-12 ms on the M5 Max and ~6.6 ms on the M3 Ultra, and it returns the
  same absolute CPU values as the all-channel subscription. The probe now
  burns 25 ms, samples, and repeats, which gives 29-31 intervals in its
  one-second budget on both machines. The counter's own read path, over all
  channels, is unchanged.
- Liveness is decided per CPU rail: a channel whose name, after any DIE_<n>_
  prefix is stripped, equals "CPU Energy" (case-insensitive). The test is the
  raw integer delta between consecutive samples, never a summed total. Live
  requires the same rail to advance in at least two separate intervals
  (LIVE_RAIL_ADVANCES). One jump is Frozen, and so is one jump on each die in
  different intervals.
- The probe keeps sampling until the rail is Live, or until the budget has
  elapsed and at least two intervals were observed. It never runs past five
  budgets.
- The outcomes are unchanged: Live, CpuRailFrozen, CpuRailAbsent and
  Unavailable. Unavailable now also covers a subscription that yields no
  sample. The result is still cached once per process in a OnceLock, and any
  outcome except Live declines the rung. If the Energy Model group cannot be
  subscribed, opening the all-channel subscription tells CpuRailAbsent from
  Unavailable.
- The rule itself is RailWatch, a pure struct with no FFI.

The public API is unchanged: IoReportStatus and its variants, ioreport_status(),
CPU_RAIL_LIVENESS_BUDGET (1 s), Meter, and the grades. One item is gone:
IoReport::probe_cpu_rail, a public method on an unnameable type that dfec5d44
added and nothing outside the crate used. The probe is now a free function
that takes its own subscription.

Docs made false by the change are corrected:
- the crate doc's IOReport rung, which now says what Live means and mentions
  the refresh;
- the probe doc ("every ~25 ms" was only achievable on the group
  subscription);
- the four IoReportStatus variant docs;
- CPU_RAIL_LIVENESS_BUDGET, which now describes the intervals and the
  five-budget cap instead of "updates within a few hundred ms";
- the cost described in the ioreport_status() doc;
- the detect() comment.

Tests. There are now 17 lib tests on macOS, up from 10, all in
mod ioreport::tests except the new off-macOS one.

- Rule tests, which need no hardware:
  - a_single_jump_is_a_batch_refresh_not_a_live_rail, including the M3 Ultra
    refresh with both dies in one interval
  - one_jump_on_each_die_in_different_intervals_is_not_a_live_rail
  - two_advances_of_the_same_rail_are_live
  - no_interval_is_no_verdict_and_no_cpu_rail_is_absent
  - the_probe_samples_until_live_or_long_enough_to_tell, including the
    five-budget cap
  - only_the_cpu_energy_aggregate_is_a_cpu_rail
- The oracle opens its own all-channel subscription and reads absolute
  values, matching names with ends_with("CPU Energy") in any group. A helper
  thread burns CPU for the whole 3 s window while the test thread reads. It
  reports CpuRailMoved(true) only if one rail's value changed at two or more
  separate reads, using its own literal threshold rather than the crate's
  constant. It also counts changes in GPU Energy, which shows its reads are
  fresh: 16-21 of 16-21 reads on the M5 Max, 17-18 of 19-20 on the M3 Ultra.
- the_ioreport_rung_is_never_selected_on_a_frozen_cpu_rail keeps its name. On
  every oracle branch it asserts the selection first, then the status. It
  prints one "ioreport liveness: status=..." line that includes the oracle's
  reads and its changes per rail.
- the_liveness_probe_never_reports_a_frozen_rail_as_live now runs the probe
  itself, uncached, on a fresh group subscription, beside the cached status.
  It asserts on every oracle outcome, not only on Live, and prints how long
  the probe took and how many intervals it saw.
- the_liveness_status_is_probed_once_and_cached is new.
- a_selected_ioreport_meter_meters_cpu_work now reads the counter directly,
  before and after a 200 ms burn. It used to go through Meter::measure, whose
  time x power fallback made e.uj() > 0 true whatever the counter did.
- ioreport_is_unavailable_off_macos is new and runs only off macOS.

Observed with the same test binary on both machines, three runs each:
  M5 Max:   ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
            selected=false (oracle: 21 reads over 3s with a helper thread
            burning CPU; changes per CPU rail [0], per GPU rail [21])
            probe: took=1.10-1.11s intervals=29-31
            test result: ok. 17 passed; 0 failed
  M3 Ultra: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
            selected=false (oracle: 20 reads over 3s with a helper thread
            burning CPU; changes per CPU rail [0, 0], per GPU rail [17]),
            which exercises the DIE_0_/DIE_1_ path
            probe: took=1.08-1.09s intervals=31
            test result: ok. 17 passed; 0 failed

Falsified on the M5 Max. Each mutation was reverted with git checkout.
- F-rule, LIVE_RAIL_ADVANCES = 1: four rule tests fail. For example, "one
  jump counted as a live rail: RailWatch { intervals: 5, rails: [("CPU
  Energy", 1)] }", "one jump per die counted as a live rail", "one interval
  cannot show two advances" and "one advance is not yet live".
- F1, probe_status always returns Live: the frozen-rail test fails with
  "IOReport selected (status=Live) although its CPU rail did not keep moving
  under 3s of CPU burn — it would report GPU-only energy, zero, or a batch
  refresh as the energy of CPU work". The probe test and
  a_selected_ioreport_meter_meters_cpu_work fail too, the latter with
  "IOReport selected but a fresh probe says Some(Frozen)".
- F2, detect() ungated as on main: the frozen-rail test fails with "IOReport
  selected (status=CpuRailFrozen) although ...".
- F-sum, liveness decided on the summed Energy Model total: through the GPU
  rail the probe says Live after 2-3 intervals (0.16-0.18 s). The frozen-rail
  test and the probe test fail in both of two runs. The same happens on the
  M3 Ultra, where GPU Energy changed at 16-17 of 18 reads today. A variant
  that treats every Energy Model channel as a rail fails the same way.

Dependants on the M5 Max all pass: jcp-cli (0 tests), jcp-conformance (1 lib
test and 5 pack tests) and jcp-intercept (8). So does the conformance suite:
"C9 energy provenance honesty" passes and it ends "all checks passed". On the
M3 Ultra the jcp-conformance lib, jcp-cli and jcp-intercept test binaries and
the jcp-conformance suite also pass (exit 0). The pack test was not run there:
it reads the pack from its compile-time path and does not touch this crate.
clippy -p jcp-energy --no-deps, lib and --tests, for aarch64-apple-darwin and
x86_64-unknown-linux-gnu: no warnings.

Not done here, deliberately:
- The counter's own reads still sample every channel, 130 ms to 0.8 s a read
  on the M5 Max. A group-only subscription returns the same values in
  ~6-12 ms, but switching changes what every read costs, so it belongs in a
  separate change.
- counts_once admits CPU sub-rails (PACC_*, MCPM*, ...) beside CPU Energy.
  That is a double count once the rail is live.
- A rail that stops tracking after selection is not caught: the probe runs
  once per process.
- Reporting the live GPU rail as an explicitly partial reading, and deciding
  which grade IOReport readings carry, are separate decisions. Both need a
  wire-format change.
- No live CPU rail exists on these hosts, so every Live-branch assertion has
  been exercised only by mutation.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 08:04:49 -04:00

jcp-energy: the liveness rule pairs consecutive samples, and tests pin each step ... 94e7cd3f24

This commit corrects dfec5d44 and 2518f0d7 where they were wrong, and closes
the gaps the round-2 review of 2518f0d7 found. The part of the probe where
the original defect lived could not be tested: putting that defect back
passed every test.

Where the two earlier commits were wrong:

- dfec5d44 says that on the M3 Ultra "all 645 Energy Model rails are frozen,
  CPU, DRAM and display included", and that there "nothing moves". Both are
  wrong. GPU Energy moves on the M3 Ultra, sporadically: 2518f0d7's oracle saw
  it change at 17-18 of 19-20 reads, the review at 18 of 19, and this
  commit's runs at 16-19 of 20. The CPU rails are not frozen for good either:
  they are refreshed in rare batches, as 2518f0d7 already said.
- dfec5d44 deleted cumulative_advances_after_cpu_burn. The comment on the
  macOS job in .github/workflows/standards-build.yml still says that test
  catches read_uj() returning Some(0), the symptom seen on 2026-09-19.
  Neither commit said so, and from dfec5d44 until this commit nothing caught
  that mutation. The job also runs without --nocapture, so a passing log
  never shows the "ioreport liveness:" line. The workflow belongs to PR #102
  and is not edited here.
- 2518f0d7 fixed the probe comparing every sample with the first one, but
  nothing tested the fix: with the defect put back, all 17 tests passed. The
  pairing lived in the FFI loop, and the rule's tests took precomputed
  booleans.
- 2518f0d7's cache test timed a second ioreport_status() against 200 ms. A
  re-probe on a live rail takes 145-155 ms, so on such a host it could not
  catch a missing cache.
- 2518f0d7's oracle doc says it "shares nothing with the liveness probe but
  the raw FFI declarations". It used the crate's cfstring_to_string. 2518f0d7
  also says the oracle's GPU Energy count "shows its reads are fresh"; that
  holds only while the GPU draws (the review saw 7 changes in 16-20 reads).
- 2518f0d7 gives 130-160 ms for one all-channel sample on the M5 Max "today".
  That depends on load. At today's load averages of 17-27 on 18 cores the
  oracle got 2-11 all-channel reads in 3 s, and the review measured ~425 ms,
  so the 0.4-0.8 s figure from 2026-09-29 comes back under load.

What changes:

- The FFI only reads. A probe sample is one plain sample of the Energy Model
  group: (channel name, absolute raw integer) for every channel. RailWatch,
  which is pure, picks out the CPU rails, keys each by (name, ordinal among
  same-named channels in that sample) because the M3 Ultra's group repeats
  264 of its 645 names, compares each sample with the one before it, and
  counts at most one advance per rail per interval. A failed sample is
  skipped; the next is compared with the last usable one.
- The probe loop is watch(sample, tick, elapsed, budget), generic over the
  sampler, the tick and the clock. The real probe passes the FFI sampler, a
  25 ms CPU spin and Instant; the tests pass scripted samples and a scripted
  clock.
- Behaviour change: a first sample that fails no longer ends the probe. Like
  any failed sample it keeps the probe sampling, up to the five-budget cap;
  with no two usable samples the status is Unavailable.
- status_of(verdict) is a pure mapping to IoReportStatus. detect() calls a
  private detect_with(status), so the gate can be driven with every status.
  A private counter, PROBE_RUNS, counts runs of the real probe. The public
  API is unchanged.
- IoReport::open no longer leaks the subscribed-channels dictionary when the
  subscription comes back null. The path is implausible, but since 2518f0d7
  probe_status reaches it.
- The oracle converts CFStrings itself, so its doc is now true.

Tests: 26 lib tests on macOS, up from 17.

- Scripted samples through watch() with a scripted clock:
  - the M5 Max refresh [480083160, 480083160, 541893638, ...] beside a rising
    GPU rail: Frozen, the step counted once, after the whole budget;
  - the M3 Ultra two-die refresh in one sample, among duplicated ECPUDTL00
    names: Frozen;
  - two same-named "CPU Energy" channels stepping together once: Frozen;
  - one rail rising by one raw unit at every sample: Live after exactly two
    intervals, three samples and two ticks;
  - every sample failing: sampling continues to the five-budget cap, no
    verdict, Unavailable;
  - past the budget: stops with Frozen once there are two intervals, samples
    again with only one;
  - DIE_0 and DIE_1 each stepping once, in different intervals: Frozen;
  - advances in non-consecutive intervals, and on one die only: Live;
  - failed samples between usable ones: the pairing goes on.
- each_verdict_reports_its_status covers all four verdicts.
  only_a_live_status_selects_the_ioreport_rung drives detect_with with every
  status: Live selects IOReport wherever the counter opens; the others give
  TimePower at the default 15 W.
- the_liveness_status_is_probed_once_and_cached counts instead of timing:
  after several status and detect() calls, PROBE_RUNS must equal 1.
- the_liveness_probe_never_reports_a_frozen_rail_as_live also pins cadence
  and burn. When a fresh probe's verdict is not Live it must have seen at
  least 10 intervals (group-only: ~30; all-channel: 2-8). The probing thread
  must use at least half a 25 ms tick of CPU per interval
  (CLOCK_THREAD_CPUTIME_ID): burning, 23-34 ms; sleeping, 6-10 ms. Per
  interval, not against the wall clock: in one run under load averages above
  20, the probe (unmutated there) spent only 36% of its wall time on CPU,
  because each IOReport sample waited longer while the burn still got its
  CPU.
- a_selected_ioreport_meter_meters_cpu_work, in its declined branch, meters
  100 ms of real CPU work with the detected meter and asserts uj() > 0, at
  least the time x power figure at the default 15 W, measured_uj None,
  provenance ModelBased and check_provenance().
- the_counter_advances_when_a_rail_it_sums_advanced is new. The counter takes
  a baseline; the oracle's own subscription reads GPU Energy (Energy Model
  group, absolute) until one channel rises by at least 10^6 raw units or 3 s
  pass; the counter reads again. If the GPU rose inside the counter's window
  the counter must have advanced. With the GPU idle only after >= base is
  asserted, and the printed line says which branch ran.
- the_ioreport_rung_is_never_selected_on_a_frozen_cpu_rail keeps its name
  and its "ioreport liveness: status=" line.

Falsified on the M5 Max against this tree. Each mutation was a temporary
edit, restored with git checkout; git status was clean after each. Every
mutation the review listed as surviving now fails:
- M6, compare each sample with the first (dfec5d44's defect): 6 tests fail,
  e.g. "a batch refresh read as live: [("CPU Energy#0", 2)]".
- M7, the probe on the all-channel subscription: "the probe saw 2 intervals
  in 1.689598583s: it is not re-sampling the Energy Model group after every
  25ms tick".
- M8, the detect() fallback TimePower { milliwatts: 0 }: "the declined meter
  metered 100ms of CPU work as 0 µJ" and "status Unavailable gave
  TimePower { milliwatts: 0 }".
- M9, the probe sleeping instead of burning: "the probing thread used
  207.32975ms of CPU over 27 intervals, 7.678879ms each, under half of each
  25ms tick".
- M10, no verdict mapped to Live: 2 tests fail, "left: Live, right:
  Unavailable".
- M11, the gate selecting on CpuRailAbsent: "status CpuRailAbsent gave
  IoReport(IoReport)".
- M13, the advance threshold raised to 1e9 raw units: 8 tests fail.
- M14, read_uj() returning Some(0): "GPU Energy rose 18851614 raw inside the
  counter's window, but the counter read 0 -> 0 µJ".
Also caught, on an intermediate tree with the same rule and loop: keying
rails by name alone, no OnceLock (the probe ran 13 times), summing advances
across rails, counting GPU Energy as a CPU rail, stopping at the budget with
one interval, LIVE_RAIL_ADVANCES = 1, and Absent for rails that held.

Observed on this tree:
  M5 Max, 3 runs, load averages 23-24: ioreport liveness:
    status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle:
    7-9 reads over 3s; changes per CPU rail [0], per GPU rail [7-9]); probe
    1.00-1.04 s, 16-27 intervals, 31.4 ms thread CPU per interval; declined
    meter 100 ms -> ~1,500,010 µJ ModelBased, measured_uj None;
    test result: ok. 26 passed; 0 failed.
  M3 Ultra, the same test binary (sha256 55d69f84...): status=CpuRailFrozen
    oracle=CpuRailMoved(false) selected=false (20 reads; CPU rails [0, 0],
    GPU rail [17]); probe 1.03 s, 28 intervals, 33.7 ms per interval;
    test result: ok. 26 passed; 0 failed.

Dependants on the M5 Max: jcp-cli (0 tests), jcp-conformance (1 lib, 5
pack) and jcp-intercept (8) pass, and the suite prints "✓ C9 energy
provenance honesty" and "all checks passed". On the M3 Ultra the jcp-cli,
jcp-conformance lib and jcp-intercept test binaries and the suite pass too.
Under M8, C9 and the consumers still pass (jcp-cli prints "ModelBased
backend → 0 µJ probe"); only this crate's tests catch it. clippy -p
jcp-energy --no-deps, lib and --tests, is clean on aarch64-apple-darwin,
x86_64-apple-darwin and x86_64-unknown-linux-gnu.

Caveats:
- M14 is caught only when a GPU Energy channel rises within 3 s. It did in
  every run today on both machines; on an idle headless host nothing can
  tell a constant reader from a true zero.
- The two new hardware floors depend on load, the interval floor most:
  12 and 16 intervals were seen at load averages of 20-27 on the 18-core
  M5 Max, which also hosts the macOS CI runner.
- Keying by ordinal assumes IOReport lists same-named channels in the same
  order in every sample of one subscription.
- The tests were not run on Linux (no docker); only clippy and check for
  x86_64-unknown-linux-gnu.
- The workflow comment and --nocapture are left to PR #102; C9 is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 09:06:52 -04:00

jcp-energy: the probe burns thread CPU, and its sampler answers to the oracle ... 7105154289

Round 3 left three mutations alive and one assertion whose margin shrank
with load. This Mac is also the macOS CI runner, often at load averages of
24-104, so every pin on how the probe paces itself had to stop depending on
wall time. This commit closes all four.

What changes in the probe:

- A tick now burns thread CPU time. It spins until the thread's own clock,
  clock_gettime(CLOCK_THREAD_CPUTIME_ID), has advanced 25 ms, with a 500 ms
  wall-clock cap. Before, a tick spun 25 ms of wall time, and on a loaded host
  that is a few milliseconds of CPU. Under 36 extra spinners, the old tick got
  a median of 3.2 ms of CPU in 48 ms.
- The probe records the wall time of each of its own FFI sample calls, and
  the CPU and wall time of each tick. Tests read how it sampled and burned
  from that record, not by timing it from outside.
- With no Energy Model subscription, the status is now Unavailable, through
  the same pure status_of as a probe that saw no interval. Before, a failed
  group open beside an openable counter said CpuRailAbsent, which claims a
  group was read and had no CPU rail. probe_with() is generic over the opener,
  so that path now runs in a test.

Tests:

- Group-only subscription: the MEDIAN wall time of the probe's own sample
  calls is under 50 ms. Group samples measured 5.7-12.8 ms; all-channel ones
  142-714 ms. This replaces the >= 10-interval floor. No hardware test asserts
  a wall-clock ratio.
- Burning: thread CPU per interval is at least 12.5 ms by the tests' own
  clock (clock_gettime_nsec_np); only the 500 ms wall cap could cut it short,
  and the message says so. From the probe's record, every tick burned 25 ms of
  thread CPU or hit the cap, and a tick came before every re-sample.
- New: the_probe_sampler_reads_each_cpu_rail_as_the_oracle_does. The oracle
  reads each Energy Model CPU and GPU rail absolutely, on its own all-channel
  subscription, before and after each of two probe samples. A rail it saw hold
  must read EQUAL in the probe's sample; a rail that moved must read between
  the two. This catches a sampler that returns constant 0 (round 3's X3), sums
  rails, or reads the wrong channel.
- Pure-rule pins, one test each: a decrease is not an advance (X5); a rail
  missing from the previous sample only sets its baseline; no subscription,
  or no sample, is Unavailable (X4); no verdict is Unavailable.
- Oracle evidence: fewer than two reads after the first is now a failure
  (Oracle::TooFewReads), never CpuRailMoved(false). The oracle extends its 3 s
  window up to 5x to get the two reads.
- The cache pin is process-wide. Eight threads, released together by a
  barrier, ask for the status and build the meter. The static PROBE_RUNS
  counter must read exactly 1.
- The IOReport hardware tests now share one lock. Side by side, one test's
  all-channel reads slowed another's group samples: max sample calls reached
  132-279 ms, and 12 probes at once pushed the median to 112 ms. Serialised,
  the median is 6-13 ms. The suite takes ~12-16 s instead of ~5 s.
- `the_ioreport_rung_is_never_selected_on_a_frozen_cpu_rail` still prints
  "ioreport liveness: status=". It now shows how long the oracle actually read.

Evidence, on this exact tree:

- M5 Max, 10 runs of the lib tests with 36 extra spinners (2 x hw.ncpu), at
  1-minute load averages of 61-180: 31/31 in all 10. Median sample cost was
  7.0-12.8 ms, thread CPU per interval 32.3-33.6 ms, CPU per tick 25.00 ms, and
  the oracle took 5-17 reads. Every run printed status=CpuRailFrozen
  oracle=CpuRailMoved(false) selected=false.
- M3 Ultra, same binary: 31/31. DIE_0 and DIE_1 read equal to the oracle
  (28237793 and 30845042), and the median sample cost was 5.9 ms.
- Each mutation below was restored with git checkout, and each one fails a
  test: X3 constant-0 sampler, X4 no subscription -> Live, X5 decrease counts,
  a new rail counts, per-thread cache, sum of CPU-named rails, GPU value given
  as CPU, tick sleeps, tick is a no-op (both the burn call and the closure),
  tick burns wall time, probe on the all-channel subscription, no OnceLock,
  compare with the first sample, TimePower{0} when declined, None -> Live, and
  a coarse threshold. Summing the CPU Energy rails is the identity on the M5
  Max's single rail, so it cannot fail there. On the M3 Ultra it fails: "held
  at 28237793 ... but the probe's sampler read 59082835".
- jcp-cli, jcp-conformance (lib and pack) and jcp-intercept pass. So does the
  conformance suite, including "C9 energy provenance honesty", on both
  machines. clippy --no-deps is clean for aarch64-apple-darwin,
  x86_64-apple-darwin and x86_64-unknown-linux-gnu, with and without --tests.

Not done here, deliberately:

- C9 still meters `|| ()`, so a declined meter giving 0 passes it. That is a
  conformance-suite change.
- The standards-build.yml comment is stale, and its job lacks --nocapture.
  PR #102 owns that workflow.
- read_uj still samples every channel. counts_once admits CPU sub-rails that
  do not contain "cpu" (e.g. PACC_*) beside CPU Energy, a double count once a
  rail is live. The receipt carries no IoReportStatus (a wire decision).
  CPU_RAIL_LIVENESS_BUDGET is public on macOS only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 14:31:46 -04:00

sandbox-meter: a CPU counter that does not keep moving is never a measurement, and no consumer relabels an estimate #114

dcharlot referenced this pull request 2026-09-30 14:31:48 -04:00

proof-cascade: never meter CPU work through an IOReport CPU rail that does not keep moving #115

dcharlot referenced this pull request 2026-09-30 14:31:50 -04:00

eoc-meter: never build the IOReport counter on a CPU rail that does not keep moving #116

dcharlot referenced this pull request 2026-09-30 14:31:52 -04:00

jouleclaw-energy: IOReport only when its CPU rail keeps moving; consumers never zero-fill or relabel #117

dcharlot referenced this pull request 2026-09-30 14:32:47 -04:00

ci(standards-build): run all five IOReport meters on macOS, and fail a step whose test never ran (merge LAST) #102

dcharlot added 1 commit 2026-09-30 17:59:01 -04:00

jcp-energy: the production probe runs under test, and no pin reads wall time ... a0daa3da62

Round 5's verifier found five problems. (1) The pin "median WALL time of the
probe's sample calls < 50 ms" failed on correct code while another process
sampled IOReport over all channels: a flake on this host, which is the macOS
CI runner. (2) The glue between the tested parts was itself untested.
probe_status() and the closure it handed probe_with() ran only against the
hardware, where the subscription always opens and the CPU rail never moves,
so a no-subscription -> Live branch, a single advance -> Live in that
closure, and a no-op tick all survived, and PROBE_RUNS counted calls to the
wrapper, not probe runs. (3) Nothing pinned which clock the probe reads as
thread CPU. (4) A sampler that repeated a stale sample passed on frozen
rails, where a stale value equals a fresh one. (5) Nothing pinned the values
the rule is handed.

What changes in the code:

- One production chain, and every link is thin. ioreport_status() caches
  host_status() in a process-wide OnceLock. host_status() has no branch: it
  counts itself in HOST_PROBE_RUNS, then returns status_of(probe_with(the
  real opener, burn_tick, HostClocks, the budget)). It is the only
  production caller of probe_with. Meter::detect() is
  detect_with(ioreport_status), which selects the rung only when the pure
  admits(status) does, and admits() takes only Live.
- probe_with() is generic over the opener, the sampler, the tick and the
  clocks, and returns a ProbeRun: the watch, and a record of every sample
  call (when it began, wall and thread CPU, and the keyed CPU and GPU Energy
  values handed to the rule) and of every tick (thread CPU and wall). The
  record is kept in every build; only tests read it.
- The FFI read only reads: (group, name, raw) per channel. group_sample()
  does the rest, and tests run it.
- Fault injection inside the real parts, in test builds only (#[cfg(test)],
  thread-local, undone by a guard): EnergyModelSubscription::open can be made
  to fail, and its FFI read can be passed through or replaced by a script. A
  tap keeps each ProbeRun. A dependant, or an integration test, links a
  library with none of this.
- counts_once, the counter's filter, drops PACC*, EACC*, MACC* and *CPM*
  (with their _SRAM rails) beside the CPU aggregates: eoc-meter's rule from
  7aa42418. On a live rail the counter would have counted that CPU energy
  twice.
- Two #[doc(hidden)] counters, ioreport_probe_runs() and
  ioreport_energy_model_opens(), so an integration test can count.

Tests (45: 44 in the library, 1 integration test):

- No pin reads wall time. Group-only subscription, by structure: every
  sample the real probe took carried exactly the Energy Model group's
  channels as the oracle counts them on its own all-channel subscription
  (364 of 11,886-12,038 on the M5 Max, 645 of 14,345 on the M3 Ultra), none
  from another group. By cost: the median THREAD CPU of the probe's FFI
  sample calls, by the tests' own clock, is under 50 ms. Measured: 5.8-6.2
  ms, with or without an all-channel sampler beside it.
- The real host_status(), uncached, under fault injection. Its opener
  failing: Unavailable, with no read, no tick and no CPU burned. A scripted
  batch refresh: Frozen, one advance. A scripted constant rail: Frozen, no
  advance. A rail rising at every read: Live, after three reads and two
  ticks. Two dies stepping once each, in different intervals: Frozen. In
  each, between every two reads the thread burned 25 ms of its CPU, or the
  tick ran into the 500 ms wall cap, by the tests' own clocks, and no tick
  shows more CPU than wall time.
- Clock identity, on the tests' own clock ids (their literals 16 and 12,
  read with clock_gettime_nsec_np). The probe's thread clock advances under
  5 ms across a 100 ms sleep, and under 5 ms while a helper thread burns 100
  ms and the process clock advances at least 100 ms. It tracks 30 ms of this
  thread's CPU to within 1 ms.
- Values handed to the rule. The real host_status() runs, uncached, while an
  oracle on another thread reads back to back. Rails only count up. So for
  each sample handed to the rule, a rail the oracle saw hold must have been
  handed the oracle's value exactly, and a rail that moved a value between
  the oracle reads either side. The CPU and GPU key sets must equal the
  oracle's. Freshness is forced: the probe's second read is held until the
  oracle sees a rail differ from the first read (or has looked twice, and
  3 s have passed). Every later read must then show the rise. In every run
  below, on both chips, the GPU rail gave that witness within 0.06-0.9 s, at
  the oracle's first or second read. Pure versions of the two-die case run
  through probe_with.
- Once per process, counted: eight threads, released by a barrier, ask for
  the status and build the meter. HOST_PROBE_RUNS and the Energy Model opens
  must both be 1. tests/ioreport_probed_once.rs does this in a process of its
  own. The macOS CI job runs only `--lib`, which skips that file, so a lib
  test does it too: the_liveness_probe_runs_once_in_a_fresh_process re-runs
  its own binary, filtered to itself, with a marker in the environment, and
  fails if that process fails or ran no test.
- admits() on all four statuses. detect_with() gives the time×power model on
  every status but Live, and the IOReport meter on Live wherever the counter
  opens.
- counts_once_drops_subrails pins the rule on the real channel names of both
  chips. Its summed lists are exactly what counts_once admits in a dump of
  each chip's Energy Model group taken for this commit (17 of 364 channels
  on the M5 Max, 29 of 645 on the M3 Ultra). In that dump, CPU Energy =
  MCPU0 + MCPU1 + PCPU to 1 mJ, and DIE_n_CPU Energy = EACC + PACC0 + PACC1
  to 1-2 mJ.
- No early return passes by default. Where a test finds no Energy Model
  subscription, or no counter, the oracle must find none either.
- Kept: every earlier pin, and the "ioreport liveness: status=" line.

Found while verifying: the freshness test could fail on correct code. Its
witness only waited 3 s. Under load one all-channel oracle read can take
longer, and a witness seen after the probe's second read had been released
was then asserted as a floor under it. It failed that way once, under an
unrelated mutation. The witness now counts only if it was seen before the
release, and the wait is counted in the oracle's reads.

Evidence, on this tree less one type alias (see the last item):

- Mutations. Each was a temporary edit, then the whole suite (library and
  integration tests), then git checkout; the tree was clean after each. All
  17 fail at least one test:
  - every sample compared with the first: 10 tests ("a batch refresh read
    as live: [("CPU Energy#0", 2)]");
  - a decrease counts as an advance: the pure decrease test;
  - no subscription -> Live inside host_status: host_status with its opener
    failing ("no Energy Model subscription");
  - a single advance -> Live in host_status's composition: 2 host_status
    tests; the same inside probe_with's production sample closure (it
    fabricates later advances): 4 tests;
  - a no-op tick passed by host_status: 5 tests ("tick 0 used 9.209µs of
    this thread's CPU in 9.291µs ... a tick must burn 25ms of thread CPU");
  - a per-thread cache: the lib fresh-process pin ("probe runs = 8") and
    tests/ioreport_probed_once.rs;
  - the sampler returns constant 0: both oracle tests ("CPU Energy#0 held at
    2588973125 ... the probe's sampler read 0");
  - a stale sampler, three ways: repeating the process's first sample (2
    tests), each subscription address's first sample (2), and each live
    subscription's first sample, so only later reads are stale (1, the
    forced witness: "GPU Energy#0 moved 126278986003314 -> 126278989490261
    ... probe read 1, but the probe handed the rule 126278982679214, outside
    them: a stale or made-up sample");
  - the thread clock read as CLOCK_MONOTONIC: 6 tests; as
    CLOCK_PROCESS_CPUTIME_ID: the clock-identity test;
  - admits() accepts CpuRailAbsent: 2 tests; detect() gates only on
    CpuRailFrozen: "status Unavailable gave IoReport(IoReport)";
  - the counter sums every Energy Model rail: counts_once_drops_subrails;
    the probe hands the rule the sum of every rail as its CPU rail: 10 tests,
    the hardware ones included.
- Contention. The lib test binary ran 5 times while another process sampled
  IOReport over all channels in a tight loop (384 samples, median 144-313 ms
  each), at 1-minute load averages of 10.6-12.1: 44/44 passed in all 5. The
  probe's median thread CPU per sample call was 5.99-6.22 ms, its median
  tick 25.06 ms of thread CPU, 30-31 intervals. Every run printed
  status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false, and every
  run asserted staleness. The helper was killed by a trap; none remains.
- M3 Ultra, macOS 27.0.1, the same binary once: 44/44. DIE_0 and DIE_1 read
  equal to the oracle (38945708 and 42131327 mJ), the median sample cost 5.8
  ms of thread CPU, staleness was asserted through the GPU rail, and the
  fresh process counted probe runs = 1.
- jcp-cli, jcp-conformance (lib and pack) and jcp-intercept pass. The
  conformance binary prints "all checks passed", C9 included. clippy
  -p jcp-energy --no-deps --tests is clean for aarch64-apple-darwin,
  x86_64-apple-darwin and x86_64-unknown-linux-gnu. The freshness test's new
  Witness struct drew one type_complexity warning. A type alias, added after
  the runs above, clears it; the suite passed again after it (44 + 1).

Not done here, deliberately:

- The macOS CI job's `cargo test --lib -p jcp-energy` still skips
  tests/ioreport_probed_once.rs; the lib pin covers the same ground there.
  The workflow is PR #102's.
- C9 still meters `|| ()`, so a declined meter giving 0 passes it. That is a
  conformance-suite change.
- read_uj still samples every channel. ANE energy is not summed on either
  chip: there is no `ANE Energy` aggregate, and counting its rails needs a
  whole-sample rule, which eoc-meter leaves for later too. The receipt
  carries no IoReportStatus (a wire decision). CPU_RAIL_LIVENESS_BUDGET is
  public on macOS only.
- The same defects in jouleclaw-energy, proof-cascade and sandbox-meter.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 18:55:23 -04:00

jcp-energy: a rail is live only if it keeps advancing, never on a two-step refresh ... 3fd35db5f9

The CPU-rail liveness rule said Live once the same CPU rail had advanced in
two separate sample intervals. On 2026-09-30 the jouleclaw-energy verifier
found a batch refresh that passes it. On this M5 Max (macOS 27.0.1), a
group-only watcher sampling the Energy Model every ~16 ms saw a refresh
published in two steps:

    CPU Energy 2643843378 -> 2705607863 (+61764485)
    CPU Energy 2705607863 -> 2705611298 (+3435)      64 ms later

and MCPU0, MCPU1 and PCPU each stepped twice with it. The probe's intervals
are ~31 ms (a 25 ms thread-CPU tick and a ~6 ms sample), so a probe window
holding such a refresh sees two advancing intervals and says Live, and the
Live is cached for the process. Two real probes did say Live while the
oracle saw no sustained movement, coinciding with a refresh. Refreshes are
~30 minutes apart, so this is rare, but it is exactly the failure the rule
exists to prevent: the rung selected on a frozen rail, and every later
refresh billed whole to whichever metered span straddles it. The rail on
this host still holds at 2,705,611,298, the value that refresh left.

The rule is now, for one CPU rail (name + ordinal):
- it advanced in at least MIN_LIVE_ADVANCES = 5 distinct intervals;
- those are at least 80% of all the intervals observed so far
  (advancing * 5 >= intervals * 4);
- the window observed so far spans at least MIN_LIVE_WINDOW = 300 ms of
  wall time from the first usable sample.
It is checked after each interval, and the rail is Live from the first
interval at which all three hold. The probe keeps sampling until Live, or
until the 1 s budget has elapsed and at least five intervals were
observed, capped at five budgets. A live rail is now called live about
300 ms in (12 intervals on an idle host) rather than ~60 ms.

The rule needs each interval's wall time, so watch() reads the clock after
every sample and hands that reading to RailWatch::observe; the tests script
it. Everything else stands: per-rail keys, consecutive samples, a fall or a
first appearance is no advance, one advance per rail per interval, the
group-only subscription, the thread-CPU tick, the pure status mapping, the
OnceLock, the test seams, and the "ioreport liveness: status=" line.

The oracle gets the same spirit. CpuRailMoved(true) now needs one rail to
have changed at 5 or more reads making up at least 80% of its reads. Its
all-channel reads are 0.13 s or more apart, so even a refresh spread over
150 ms shows as three changes at most, and a refresh coinciding with the
probe can no longer vouch for the Live it caused. It extends its window, up
to 5x, until it has five reads after the first, and still fails as
TooFewReads when it cannot. Its loop is now oracle_watch_with, over a read
function and a clock, so its verdict is tested on scripts.

Tests, on a scripted clock at 31 ms an interval unless stated; (*) fail
under the two-advance rule:
- (*) the observed two-step refresh, with the steps in consecutive
  intervals, 62 ms apart, and on a host at 160 ms an interval: Frozen;
- (*) a refresh in 3, 4 or 5 steps within 150 ms at the start of the
  window, and in 5 steps mid-window: Frozen;
- (*) the M3 Ultra shape, both dies stepping twice within 64 ms: Frozen;
- a rail advancing in every interval: Live at the 10th (310 ms), not at the
  5th (155 ms), on either die;
- missing 1 interval in 5 (exactly 80%): Live; missing 2 in 5: never; on a
  loaded host, 2 early misses: not at 5 of 7, 6 of 8 or 7 of 9, Live at 8
  of 10;
- a loaded host at 100 ms an interval: Live at the 5th (500 ms), not at the
  3rd, where the window is already met;
- keep-sampling: at 600 ms an interval the probe goes past the budget to
  five intervals (where a live rail is proven Live, at 3 s); at 1.5 s it
  stops at the 5 s cap with four;
- through probe_with: a sustained advance is Live at 10 intervals, and
  (*) the two-step refresh is Frozen;
- through host_status itself (the real opener, burn tick and clocks, with
  scripted FFI reads): (*) the two-step refresh is CpuRailFrozen; a
  sustained advance is Live with 5+ advances over a 300+ ms window by both
  the probe's clock and the tests' own, stopping as soon as it is live;
- the two-die summing check now has the dies advance in turn (each in half
  the intervals, summed in all): two one-step dies, summed, are only two
  advances, which no longer trips the rule;
- the oracle on scripts: the two-step refresh is CpuRailMoved(false) (it
  was true), a three-step refresh false, a change at every read true at the
  5th, 4 in 5 true, 3 in 5 false; slow reads are extended to five, reads
  too slow for that are TooFewReads.

Mutations, each a temporary edit restored with git checkout, the tree clean
after each:
- MIN_LIVE_ADVANCES = 2: 4 tests fail (the two-step refresh on a loaded
  host, loaded-host Live before the 5th interval, keep-sampling, the
  stopping rule);
- the 80% check removed: 4 fail (the 80% test, the 5-step refresh, two dies
  in turn through probe_with and through host_status);
- the 300 ms window removed: 6 fail (Live at the 10th, the 80% test, the
  5-step refresh at the start, the sustained advance through probe_with and
  through host_status, the stopping rule);
- the oracle's bar back to >= 2 changes: both oracle tests fail. The whole
  old oracle (2 changes, 2 reads) fails with "a two-step refresh vouched for
  a moving rail: OracleRun { verdict: CpuRailMoved(true), reads: 4, ...,
  cpu_changes: [2] ...";
- the whole old rule (2 advances, no share, no window): 15 fail, every (*)
  test among them.

Runs. The crate's tests pass (51 lib, 1 integration). Three runs under a
helper process sampling IOReport over all channels back to back (~250-280
ms a sample, load average 14-16), 51/51 each:

    ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 6 reads over 3.857275292s of a 3s window with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [4])
    ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 7 reads over 3.051596791s of a 3s window with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [6])
    ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 9 reads over 3.133878417s of a 3s window with a helper thread burning CPU; changes per CPU rail [0], per GPU rail [5])

On the M3 Ultra (macOS 27.0.1), the same binary, 51/51:

    ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false (oracle: 21 reads over 3.033503042s of a 3s window with a helper thread burning CPU; changes per CPU rail [0, 0], per GPU rail [19])

jcp-cli (it builds; it has no tests), jcp-conformance (with the conformance
binary: all checks passed, C9 energy provenance honesty among them) and
jcp-intercept pass. Clippy --no-deps and rustdoc over private items with
-D warnings are clean on jcp-energy.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot changed title from jcp-energy: never select IOReport when its CPU rail is frozen to jcp-energy: IOReport only when its CPU rail keeps advancing, never on a batch refresh 2026-09-30 18:59:59 -04:00

dcharlot added 1 commit 2026-09-30 20:06:22 -04:00

jcp: a live vendor counter is a reading; intercept/1 keeps a partial figure partial ... 340313d6b3

The questions this branch left open are now decided. A vendor energy counter
(RAPL, NVML, Apple IOReport) is a real reading; what it lacks is an
independent root, which is a gap in provenance, not in accuracy. It earns the
hardware grade only while it is live. The live GPU rail is not reported as a
partial reading of CPU work. And a partial figure must stay partial on the
intercept/1 wire.

The grade. jcp-energy already graded RAPL and NVML `HwShunt`. The IOReport rung
carried a live reading at `ModelBased`, in the modeled slot. It now takes the
same grade: `Meter::provenance()` says `HwShunt` and a live span is
`Energy::measured`. The declined paths keep their own grades: the time×power
model is `ModelBased`, the constant estimator `Estimator`.

Raising the grade raises the stakes of what `measure()` accepts, so it is
gated per span as well. It took any rise of the summed SoC total as the span's
energy, and that total rises through the GPU rail, which is machine-wide. On
this M5 Max, in this commit's own test runs, the counter counted 32 mJ and then
9.1 J between two reads while an independent oracle saw the CPU rail hold and
one other rail move. A rail that was live at selection can still hold across
one span: a span shorter than its update interval, or a rail that stops
publishing after the probe. The counter now reports the CPU rails' share beside
its total, from the same delta sample. `measure()` grades a span `HwShunt` only
when the CPU rail itself advanced. The figure is still the SoC total's rise,
which, like a RAPL package counter, over-debits the work rather than leaving
part of it out. A span whose CPU rail held, or whose read failed, is the
time×power figure at `ModelBased`. A model figure is never `HwShunt`. The grading
is one pure function, `ioreport_span`, so every branch runs on scripted reads on
any platform. The oracle test that checks the counter's total value for value,
against its own all-channel subscription, now checks the CPU share the same way.

The GPU rail is not reported as a partial reading. It is machine-wide (display
compositing moves it), so booking it to a task misattributes energy the task
did not spend, and a partial figure would under-debit the budget because the
CPU's share would go unmetered. The rung is declined to the whole-task estimate.
That rationale now sits in the rung's docs and in SPEC §5.1.

SPEC. §5 says `HwShunt` covers vendor counters as well as shunts, and carries
the liveness MUST in the shared wording. Its dashes are typeset and `Partial`
is code-formatted; the text is otherwise as decided. §5.1 adds the IOReport
rung. A new §5.2 defines coverage and its wire encodings, carried only when
`Partial`. One is jcp/2's `energy_coverage`, which jcp-receipt already emitted
but the SPEC never listed, so §8's core field list gains it. The other is
intercept/1's `coverage`.

intercept/1. jcp-intercept dropped coverage, so a `Partial` figure handed to it
was sealed as the whole step's energy. It now adds `"coverage": "Partial"` only
when the figure is partial. `verify` recovers it in a new
`InterceptReceipt::coverage`, and refuses any other value, an explicit "Whole"
included, because a sealer never produces one. A whole figure seals the same
bytes as before. The two envelopes a Redact-then-CallLimit chain sealed at
3fd35db5 were captured by building that commit in a scratch crate before the
change, and are pinned hex for hex. A whole `HwShunt` figure's object is
compared, key for key and in canonical bytes, against the pre-coverage object.
No conformance vector changes, and none contains an intercept/1 receipt.

Prose. Provenance::HwShunt's doc names vendor counters and the liveness rule.
The README's jcp-energy bullet lists the IOReport rung. Its jcp-gateway bullet
no longer says "metered on real hardware ... energy actually spent". jcp-efficiency's
MAPPING §4 now says "real readings of the vendor's own energy counter" instead of
"power-sensor readings", and that a counter counts only while live. It also no
longer calls RAPL "a model, not a meter".

Tests. jcp-energy has 54 lib tests (51 before) plus its integration test.
jcp-intercept has 12 (8 before). jcp-schema has 30, jcp-conformance its unit
test and 5 pack tests, all passing. `jcp-conformance` prints "all checks
passed", and jcp-cli builds and runs. clippy --no-deps is clean on jcp-energy
and jcp-intercept. jcp-schema's one warning, in flow.rs, is pre-existing.

Falsified, each by a temporary edit restored with git checkout:
- A live span built with `Energy::modeled` fails
  `a_span_the_ioreport_cpu_rail_advanced_over_is_a_hardware_reading` and
  `the_ioreport_rung_stamps_the_hardware_grade`.
- `Meter::provenance()` for IOReport set back to `ModelBased` fails
  `the_ioreport_rung_stamps_the_hardware_grade`.
- Gating the span on the SoC total instead of the CPU rail fails
  `a_span_the_ioreport_cpu_rail_held_over_is_the_models_figure` with
  "the GPU rail rose, the CPU rail held ... provenance: HwShunt".
- A CPU share that counts every rail fails the hardware oracle test: "CPU share
  rose 18127 µJ ... CPU energy rails rose 0 µJ". That one needs a non-CPU rail
  to move during the window, and the GPU rail did.
- Sealing coverage even when Whole fails the byte-identity tests (and every
  test that verifies a whole receipt).
- Dropping coverage when Partial fails `a_partial_figure_seals_coverage_partial`.
- Letting verify read an explicit "Whole" fails
  `verify_refuses_a_coverage_other_than_partial`.

Not done here:
- jcp-gateway still seals `Energy::measured` for whatever sandbox-meter returns,
  estimates included. The sandbox-meter liveness branch fixes that, so the
  README's gateway line states the rule rather than this branch's gateway.
- The NVML rung reports GPU energy as a whole-task `HwShunt` figure. Under the
  new MUST, a figure for work its counter does not cover (the CPU's share) is
  `Partial`. That needs its own decision, and nothing here is built with
  `--features nvml`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 20:46:07 -04:00

jcp-energy: RAPL and NVML count only while live, and an NVML figure is partial ... 475680abcd

340313d6 left two vendor counters outside the rule it adopted. SPEC §5 says a
vendor counter (RAPL, NVML, IOReport) is a hardware reading only while it is
live for the domain the figure claims, and that a figure must not claim a
domain its counter does not read. IOReport was held to that. RAPL and NVML were
selected as soon as they opened, and NVML's GPU-only figure claimed the whole
task.

One rule. The sustained-advance rule (one rail advancing in at least five
intervals, at least 80% of them, over at least 300 ms), its loop, its stopping
rule and the CPU-burning tick lived inside the macOS-only IOReport module. They
now live in a crate-level `liveness` module that compiles everywhere, and all
three rungs apply it. IOReport's `RailWatch` is now the shared rule fed only
its CPU rails. Its 43 tests are unchanged apart from one import line, and all
pass. The thread clock gains Linux's CLOCK_THREAD_CPUTIME_ID (3) beside
macOS's (16), so on Linux the RAPL probe also burns by its own thread's CPU.

RAPL. `rapl_status()` probes once per process and caches the result. Each
package's `energy_uj` is a rail, sampled after every 25 ms of the probing
thread's CPU, for up to one second. `detect()` opens RAPL only on Live; on any
other status `select` does not call the opener. With no readable powercap tree
the status is Unavailable and nothing is burned, so RAPL declines as before. On
macOS that costs one failed read_dir. A span over which no package counter rose
now gets the time×power figure instead of a hardware reading of zero. This
mirrors IOReport's per-span rule: the counter updates about once a
millisecond, so a shorter span can see no rise even though the work ran.

NVML. `nvml_status()` applies the same rule to device 0's total-energy
counter. The probe cannot load the GPU, and burning CPU does not load it
either, so it waits 100 ms between samples instead of burning. A counter
refreshed at least that often advances on the device's own draw in every
interval. One refreshed less often fails the 80% share and is declined, so it
fails safe. An NVML span is now `Energy::measured_partial`, `HwShunt` with
`Coverage::Partial`, because the GPU counter does not read the CPU's share of
the work. A span over which the counter did not advance, or fell (a driver
reload resets it), gets the model's whole-task figure. Without the feature
`nvml_status()` is Unavailable. With it, everything compiles and clippy is
clean on macOS and Linux x86_64. But no host here has an NVIDIA GPU, so the
NVML probe has never run against a real counter.

API, additive: `CounterStatus { Unavailable, Frozen, Live }`, `rapl_status()`
and `nvml_status()`. `Meter::detect()` keeps its signature. Its private
`detect_with` now takes a `Ladder` of statuses and the RAPL opener, so the
tests can drive each gate with every status.

SPEC §5.1 and the README's jcp-energy bullet now say that RAPL and NVML are
selected only while live, and that NVML's figure is `Partial`.

Tests. jcp-energy has 63 lib tests (54 before) and its integration test, all
passing with and without `--features nvml`:
- Scripted package counters, through the probe the RAPL rung runs. Live at
  the tenth interval (310 ms) when a package keeps advancing, and the probe
  stops there. Frozen when the counters hold, step in one lump, step in two or
  in five quick steps, advance in turn, or wrap at every other sample.
  Unavailable when every read fails, or when no tree opens (then nothing is
  sampled and nothing ticks).
- The real RAPL probe on fake powercap trees in the temp directory: real file
  reads, the CPU-burning tick, the real clock. "a fake tree that holds ->
  Frozen in 1.004322458s"; "a fake tree advancing every 2 ms -> Live in
  417.847167ms"; an absent or unparsable tree is Unavailable.
- The gate: only Live admits a counter rung, and on any other status the
  counter is not opened. `detect_with` on a fake tree selects RAPL on Live
  only.
- Spans: `measure` on a fake tree reads a rise, wraparound-corrected, as
  `HwShunt` and whole, and a span with no rise as `ModelBased`. NVML spans are
  `HwShunt` and `Partial`, or the model's whole figure when the counter held,
  fell or failed to read.
- NVML-shaped scripts at 100 ms intervals: Live at the fifth interval; a
  counter advancing in 1 of 4 intervals is Frozen.
- The fresh-process pin also counts the RAPL probe (one run) and, under the
  feature, the NVML probe (one run).
jcp-intercept (12), jcp-schema (30) and jcp-conformance (1 unit test and 5
pack tests) pass. The jcp-conformance binary prints "all checks passed", and
jcp-cli runs. clippy --no-deps is clean on jcp-energy, with and without `nvml`,
for aarch64-apple-darwin, x86_64-apple-darwin and x86_64-unknown-linux-gnu.
`cargo doc` with `-D warnings` is clean.

Falsified, each by a temporary edit restored with git checkout (git diff was
empty after each):
- `detect` opening RAPL without asking its status fails
  `only_a_live_status_selects_the_rapl_rung` ("status Unavailable gave
  Rapl(...)"). It also fails the fresh-process pin, because the probe never
  ran.
- A gate that admits Frozen fails the gate tests ("Frozen admits the rung").
- A probe that reports Live for any counter that opened fails the scripted,
  fake-tree and status tests ("a powercap tree whose counters hold was not
  declined").
- Requiring only two advances fails four IOReport rule tests and the NVML
  script.
- A RAPL sampler that returns a constant fails the fake-tree test ("a powercap
  counter advancing every 2 ms was not live").
- Grading a span that did not rise as a reading fails three span tests
  ("measured_uj: Some(0) ... provenance: HwShunt").
- Grading an NVML span `Energy::measured` fails the partial test ("a GPU-only
  figure claimed the whole task").
- An uncached `rapl_status` fails the pin ("the RAPL liveness probe did not
  run exactly once in this process").
- A `RailWatch` that hands the rule every rail, GPU included, fails four
  IOReport rule tests.
- A RAPL span that ignores the rise fails
  `a_rapl_span_is_a_reading_only_if_a_counter_rose`.

Not done here:
- The NVML gate in `detect_with` (`select(nvml, Nvml::open)`) cannot be
  exercised on a host with no NVIDIA device. It is the same `select` the RAPL
  gate tests drive, and the rule it gates on is tested on NVML-shaped scripts.
- The 100 ms NVML wait is a design choice, not a measured refresh period,
  because no NVML counter was observed here.
- RAPL was not exercised on Linux hardware. On most distributions `energy_uj`
  is readable only by root, and there RAPL stays Unavailable as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 21:10:59 -04:00

jcp-energy: meter a closure's CPU work with the per-process energy counter, which macOS 27 keeps live #122

dcharlot added 1 commit 2026-09-30 21:26:57 -04:00

jcp-energy: a counter is live only if every rail it sums keeps advancing ... 4f9d90ca94

The IOReport counter sums every CPU energy rail: `CPU Energy` on an M5 Max,
`DIE_0_CPU Energy` and `DIE_1_CPU Energy` on an M3 Ultra. The shared liveness
rule called a counter Live as soon as any one of its rails sustained advances
(five intervals, 80% of them, 300 ms). On a two-die part, one live die beside
a frozen one would select the rung. Every span's figure would then miss the
frozen die's energy and still be graded a whole-CPU HwShunt reading. SPEC §5
forbids that: a figure must not claim coverage of a domain its counter does
not read.

RAPL had the same gap. A RAPL span sums every top-level package's
`energy_uj`, and the rung was held to the same any-rail rule. The fake-tree
test even asserted Live with package 1 holding. proof-cascade's RAPL already
requires every package (9334ec0c); this crate's did not.

The fix is in the one shared rule, `liveness::Advances::observe`. A counter
is now Live once every rail seen meets all three conditions at the same
interval. A counter with no rail at all is never Live. That guard matters:
`all` over an empty list is true, so without it a group with no CPU rail would
read as Live once the window passed 300 ms. A rail missing from some samples
is still counted against every interval, so a rail that comes and goes holds
the counter Frozen. NVML and a single-die host have one rail, so for them
nothing changes: a single rail is Live at the tenth interval, as before.

On a two-die part whose rails are both live, the slower die now decides. If
an idle die's rail advanced in fewer than 80% of the intervals, the rung is
declined and the model estimates. That is the safe side: estimated, never
over-claimed. On macOS 27.0.1 every CPU rail is frozen on both machines here,
so nothing changes there.

The crate docs, `CounterStatus`, `IoReportStatus`, `rapl_status`, SPEC §5 and
§5.1, and the README's jcp-energy bullet now say "every rail". The SPEC adds
one sentence: where the figure sums several counters of the domain, every one
of them must be observed live.

Tests. jcp-energy has 66 lib tests (63 before) and its integration test, all
passing with and without `--features nvml`:
- `one_live_die_beside_a_frozen_die_is_frozen`, on scripted M3 Ultra-shaped
  samples. Both dies rising: Live at the tenth interval, the same interval as
  a single rail. DIE_0 rising, DIE_1 holding: Frozen after the whole budget
  (DIE_0 33 advances, DIE_1 0), and never Live when fed 200 intervals by hand.
  The mirror is Frozen too, and so is DIE_0 rising beside a DIE_1 republished
  once.
- The same through `probe_with`, and through the production composition
  `host_status` with its real tick and clocks:
  `host_status_on_one_live_die_beside_a_frozen_die_is_frozen`.
- The existing 300 ms test's second case was "the second die of two, the
  first holding -> Live". That was the any-rail behaviour, and it is now the
  mixed test. In its place, both dies rising -> Live.
- RAPL, scripted: package 0 rising beside a held package 1 is Frozen. RAPL,
  the real probe on fake powercap trees: "packages [0, 1] advancing every 2 ms
  -> Live in 301.846625ms"; "packages [0] advancing every 2 ms -> Frozen in
  1.06510775s".
- No rail at all, watched past 300 ms: an Energy Model group with no CPU rail
  stays Absent over the whole budget, and a counter whose samples carry no
  rail is never Live.

Falsified, each by a temporary edit restored with git checkout (git diff was
empty after each):
- The rule back to any rail fails five tests: the three two-die tests ("DIE_0
  beside a frozen DIE_1 became live", left Some(10), right None; Live vs
  Frozen; Live vs CpuRailFrozen) and both RAPL tests ("one live package beside
  a held one read as live"; "package 0 advancing beside a held package 1 was
  not declined").
- Dropping the no-rail guard first passed all 66 tests. The only no-rail test
  spanned 62 ms, short of the window. With the full-budget cases added, it
  fails two: "a group with no CPU rail, watched 310ms", left Some(Live), right
  Some(Absent), and the counter-status test.

jcp-intercept (12) and jcp-conformance (1 unit test and 5 pack tests) pass.
The jcp-conformance binary prints "all checks passed", and jcp-cli runs and
exits 0. clippy --no-deps -D warnings is clean on jcp-energy for macOS (with
and without `nvml`) and x86_64-unknown-linux-gnu, and `cargo doc` with
`-D warnings` is clean.

On an M3 Ultra (macOS 27.0.1), the lib test binary passed 66 of 66: "status=
CpuRailFrozen oracle=CpuRailMoved(false) selected=false", advances per CPU
rail [("DIE_0_CPU Energy#0", 0), ("DIE_1_CPU Energy#0", 0)].

Not done here:
- The hardware oracle in the IOReport tests still vouches for a Live when any
  one CPU rail kept moving. It matches `*CPU Energy` in every group, not only
  the Energy Model's. Making it require every rail could convict a correct
  probe over a channel the counter never reads, so it is unchanged. On a mixed
  host it would not convict an any-rail probe; the scripted tests do.
- `ioreport_span` still grades a span a reading when the summed CPU rails
  rose, that is, when any die advanced. Requiring every die per span would
  decline short spans over an idle die.
- This crate's RAPL reader sums every top-level `intel-rapl:N`, `psys`
  included where a platform exposes it. proof-cascade found that double count
  (9334ec0c). It is untouched here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 21:57:57 -04:00

jcp-energy: the RAPL figure sums the package zones by name, never psys or dram ... 25bfdcd98d

The RAPL reader opened every top-level powercap directory named
`intel-rapl:<n>` and summed them as the CPU package figure, and its liveness
probe watched every one of them. A top-level zone is not necessarily a
package. Many Intel hosts expose `psys`, the platform zone, at the top level
(`intel-rapl:1` on a typical laptop), and its count already includes the
packages', so the figure counted package energy twice and graded it
`HwShunt`. Some platforms put a `dram` zone at the top level, which is not
CPU energy at all. The probe had the mirror-image fault: a live `psys` and a
held `dram` zone decided whether the package counter was live.

The fix is the rule proof-cascade already follows (9334ec0c): the figure is
the sum of every top-level zone whose `name` file reads `package-<digits>`,
every socket's, and nothing else. `psys`, `dram`, a package's subzones
(`intel-rapl:0:0`, its `core`, `uncore` or `dram`) and the second interface
to the same packages (`intel-rapl-mmio:<n>`) are neither summed nor probed.
The liveness sample is built from the opened packages, so the rule that
every rail must keep advancing now applies to every package and only to
packages. A host whose top level holds no package zone (only `psys`) opens no
reader and is `Unavailable` without a probe.

Two holes that would leave a package out of a figure graded whole close with
it. A package zone whose `energy_uj` does not read used to be skipped, so on
a two-socket host the figure could be one socket's energy claiming the CPU;
now the reader does not open. A top-level zone whose `name` does not read may
be a package, so it declines the reader too.

Wrap handling, checked per zone. Each package was already corrected at its
own `max_energy_range_uj`, but a range that did not read defaulted to
`u64::MAX`, so a counter that fell across a span was booked as about 1.8e19
µJ, `HwShunt`. A range that does not read, or reads zero, is now unknown,
and a fall with an unknown range, or from above the range, gives no figure:
the span is the time×power model's, as a failed read is. A package snapshot
that does not match the opened packages gives none either.

The SPEC §5.1 RAPL entry now names the package zones and says why `psys`
and `dram` are not summed.

Tests, on fake powercap trees in the temp directory (they run on macOS):
- `rapl_sums_and_probes_the_package_zones_only`: a two-socket tree with
  `psys`, a top-level `dram` zone, `core`/`dram` subzones and an MMIO
  interface. The reader holds exactly `intel-rapl:0` and `intel-rapl:2`, in
  zone order, each with its own range; the liveness sample names those two;
  a span over which every zone rose is metered as the two packages' 8,000 µJ
  alone; package 1 wrapping at its own 65,532,610,987 µJ range while `psys`
  and `dram` also fall is 12,487 µJ. The real probe is Live with both
  packages advancing and `psys`/`dram` holding, and Frozen with `psys`,
  `dram`, the MMIO zone and package 0 advancing while package 1 holds.
- `a_rapl_figure_never_leaves_a_package_out_or_takes_another_zone`: a tree of
  `psys` and `dram` (with a `package-0` subzone) opens nothing and is
  Unavailable without burning; a package whose `energy_uj` does not parse
  beside a readable one, and an unnamed top-level zone, do not open; a
  package with no range meters a rise and gives the model's figure for a
  fall.
- `a_rapl_zone_wraps_at_its_own_range_or_gives_no_figure` and
  `a_rapl_package_is_a_zone_named_package_n`: the per-zone delta and the
  name and directory matchers, edge cases included.
- The existing fake trees now carry `name` files; every earlier RAPL test is
  unchanged in what it asserts.

Falsified, each restored through git before the next:
- selecting by directory name again (the `name` check disabled) fails
  `rapl_sums_and_probes_the_package_zones_only` — the reader held
  `intel-rapl:0..3`, `psys` and `dram` among them — and
  `a_rapl_figure_never_leaves_a_package_out_or_takes_another_zone` — "a tree
  with no package zone opened";
- an unreadable range defaulting to `u64::MAX` again fails the latter: the
  fall was metered as `HwShunt`, 18,446,744,073,709,550,715 µJ.

jcp-energy (70 unit tests + the fresh-process pin), jcp-intercept,
jcp-conformance (its tests and the binary: all checks passed) and jcp-cli
pass on macOS (M5 Max, macOS 27.0.1). `cargo clippy --all-targets` on those
four is clean for aarch64-apple-darwin and x86_64-unknown-linux-gnu apart
from a pre-existing jcp-schema lint. The reader was not run against a real
Linux sysfs here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-10-01 02:12:29 -04:00

energy meters: Linux RAPL and NVML on real hardware; an NVML counter past its board is Implausible #129

dcharlot merged commit b24bc89438 into main 2026-10-01 05:45:02 -04:00

dcharlot referenced this pull request from a commit 2026-10-01 05:45:03 -04:00

Merge pull request 'jcp-energy: IOReport only when its CPU rail keeps advancing, never on a batch refresh' (#110) from jcp-energy/cpu-rail-liveness into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!110

No description provided.