eoc-meter: never build the IOReport counter on a CPU rail that does not keep moving #116

Merged

dcharlot merged 22 commits from eoc-meter/cpu-rail-liveness into main 2026-10-01 05:50:58 -04:00

Conversation 0 Commits 22 Files changed 27 +6705 -647

dcharlot commented 2026-09-30 14:31:50 -04:00

Owner

Copy link

eoc-meter's IOReport counter was built whenever a subscription opened. eoc-cascade then tagged any positive overhead as JouleSource::Measured.

What was wrong: on main

• eoc query "totally novel question" on the M5 Max printed cost : 50136686 µJ (measured) in 4 of 4 runs. That is the 50 J neural-stage estimate plus a GPU-only IOReport delta, sealed into the receipt as measured.
• eoc meter printed counter: ioreport with idle : 307124 µJ on the M5 Max, and 10279 µJ on the M3 Ultra. Both are GPU-rail energy.
• The old test checked only the aggregate. It passed on the M5 Max through the GPU rail and failed on the M3 Ultra (energy must advance: before=0 after=0).

What this does

• MacosIoReportCounter::try_detect() returns Err(NoCounter) unless the family rule says Live. ioreport_status() gives the reason.
  • The fallback counters can only return an error or 0, so attach_cascade_cost keeps each stage's own provenance.
  • After, on both chips: cost : 50000000 µJ (estimated).
• eoc meter prints the IOReport status and never prints a failed read as 0 µJ over 1.0 s.
• eoc-streaming: a stream's energy is Measured only when the counter actually advanced over the span. Before, any non-stub counter's zero delta was sealed as Measured.
• The counter counts CPU energy once. counts_once admitted per-cluster and power-manager sub-rails (PACC_*, MCPM*, PCPM, *_CPM, and their _SRAM rails) beside the CPU Energy aggregate they sum into. The hierarchy was confirmed from absolute values on both chips (CPU Energy = MCPU0 + MCPU1 + PCPU, to within 1 mJ). On a live host CPU energy would have been summed twice. No reading so far was affected, because no host is live.

Verification of this PR (head 65ff45ba)

• Mutations: the round-6 run caught 24 of 24, including a gate that declines only on CpuRailFrozen, and a constructor that opens before it declines. Restoring the old two-step rule fails 13 tests.
• Robustness: 5/5 under all-channel contention, 5/5 under busy-loop load, then 3/3 with the new rule.
• M3 Ultra: passes.
• eoc workspace: 621 passed, 0 failed.

Later rounds on this branch

• f84f0b2f: a cache hit is an estimated zero, never a measured one. A total is Measured only if every microjoule was read.
• 0db9b8be and 6c37e299: eoc-carbon carries provenance through EvalReport, CarbonAccount and SCI. An SCI score at PUE > 1 is Estimated, because its overhead is modeled. A run with no correct answer has no score; it used to produce an 18.4 TJ energy term.
• 6a8d7c68: an unmetered stream has no figure, never 0 µJ.
• 15cf07c2 and 1535b8eb: eoc-local never turns a failed first read into a since-boot total, and inference that ran, including a llamacpp error, is never 0 µJ.
• 2c353d68, b2e8464c and 26165a77:
  • RAPL and NVML are gated on the liveness rule.
  • IOReport needs every CPU rail.
  • RAPL sums every package-N zone, never psys or dram, each wrapped at its own range.
• 52d9d1e2, 371fbe5e, 6cf8b60e and 1352aeea (eoc-agent):
  • A failed first read can no longer charge a since-boot total to an agent's budget.
  • Tool steps and Plan-and-Execute steps are never charged 0.
  • Every trace span says whether its figure was measured or estimated.

Why: what IOReport actually does on macOS 27.0.1

Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1:

• CPU rails are republished in rare lumps. For an unprivileged reader, CPU Energy and DIE_<n>_CPU Energy do not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each.
• A lump can arrive in steps. A watcher sampling every ~16 ms caught one as +61,764,485 mJ, then +3,435 mJ 64 ms later.
• GPU Energy is live on the M5 Max (it changes every ~28 ms) and sporadic on the M3 Ultra.

A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it.

The rule: identical in all five backends (#110, #114, #115, #116, #117)

• The probe opens its own subscription to the "Energy Model" group. A sample costs about 6–12 ms of thread CPU, against about 115 ms for all channels.
• It decides per CPU rail (CPU Energy, or each DIE_<n>_CPU Energy), keyed by name and ordinal:
  • it compares raw integers from consecutive samples;
  • a decrease, or a rail's first appearance, is not an advance;
  • a rail advances at most once per interval.
• Live means every rail the meter sums kept advancing: in at least 5 intervals, in at least 4 of every 5 intervals so far, over a window of at least 300 ms.
  • A live rail meets that in about 300 ms. A lump does not, even one published in several steps.
  • Every rail is required because the meter sums them. One live die beside a frozen die would otherwise give a figure that silently misses the frozen die while claiming the whole CPU.
• The same rule gates RAPL and NVML.
  • RAPL sums the top-level zones whose name reads package-N, on every socket, and each must be live. It never sums psys, which already includes the packages, nor dram or the sub-zones.
  • Each zone wraps at its own max_energy_range_uj. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading.
  • An NVML (GPU-only) figure is Partial where the format can say so. Where it can't, it is never a hardware reading of the whole task.
• It samples until Live, or until the 1 s budget has elapsed and at least 5 intervals have been seen, with a cap of 5 budgets.
  • Each tick burns 25 ms of this thread's CPU time, not wall time, with a 500 ms wall cap. So the probe keeps working on a loaded host.
• The result is cached once per process. Anything but Live declines the backend, so the crate's existing honest fallback runs. The reason is public: IoReportStatus, ioreport_status(), rapl_status(), nvml_status().
• The production status is one branch-free composition, counted each time it runs. Every selection path reads it only through the cached status and a pure admits(status) (Live only).
• A live vendor counter's reading is a hardware reading: HwShunt, or Measured in eoc. That follows the owner's position that vendor counters are real readings and the trust gap is provenance. It holds only while the counter is live, per the MUST in #119.

How it was verified

Six rounds of implementation, each checked by independent adversarial verification.

Pure tests on scripted samples and a scripted clock. They cover:

• the real refreshes (one step, two steps, 3–5 steps, the two-die shape);
• decreases, first appearances and failed samples;
• live rails (every interval, 80% of intervals, on a loaded host);
• the sampling-loop bounds.

Fault hooks inside the real IOReport opener and sampler. The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test.

Hardware tests against an independent oracle. The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check:

• the probe's sampled values equal the oracle's reads on either side;
• the live GPU rail shows up in the probe's record, which proves the reads are fresh;
• every probe sample is group-only, by channel count;
• thread CPU per sample and per tick is checked against the tests' own clock;
• the probe runs exactly once per process, checked in a fresh process.

Mutation-tested. Each of the following was applied as a temporary edit and failed a test:

• restore the old one- or two-advance rule;
• compare every sample with the first;
• treat a decrease as an advance;
• map "no subscription" to Live inside the production status;
• use a no-op tick, a tick that sleeps, or a tick on the wrong clock;
• use a per-thread or non-atomic cache;
• probe uncached from the constructor;
• a sampler that returns constant, stale or summed values;
• an all-channel subscription;
• decide on the summed total or on the GPU rail;
• an admits() that accepts CpuRailAbsent.

The counts per crate are below.

Robustness. Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra.

CI: these tests are behind cfg(target_os = "macos"), and this workspace runs on a linux runner. They reach CI through #102, which merges last. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not print ioreport liveness:.

Merge order: #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them.

Open questions, now decided and implemented (per "fix it all")

• One grade for a live vendor counter. It is HwShunt, or Measured in eoc, everywhere: a vendor counter is a real reading, and the gap is provenance. It holds only while the counter is live (the MUST in #119).
• The GPU rail as a partial reading of CPU work: no. The GPU rail is machine-wide (display compositing moves it). Booking it to a task would misattribute energy and under-debit budgets.
• Wire keys named measured_uj that carry estimates. Their grade now sits beside them in plaintext: jcp-gateway's _meta gets energy_provenance, plus energy_coverage when partial. The keys stay, for compatibility.
• eoc cache hits are an estimated zero, never Measured.
• joule-ui. Meter defaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. Authorizer::charge carries the figure's provenance.
• A live CPU energy source on macOS 27 exists. It is the kernel's per-process counter ri_energy_nj: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privileged powermetrics is live too, but machine-wide and root-only, so it is not used.)

Still open

• The joule-ui demo on prod. It has run on macOS 27 since 2026-09-15. Its sealed chain was not inspected, because the read was blocked as a production read. After #114 merges, redeploying joule-ui-serve gives it honest grades, and the per-process probe in #121 gives it real readings.
• Linux RAPL and NVML have never run against real hardware. No Linux host with readable RAPL and no NVIDIA GPU were available. They are covered by fake-tree and scripted tests, type-checked for Linux, and declined fail-safe.

🤖 Generated with Claude Code

eoc-meter's IOReport counter was built whenever a subscription opened. eoc-cascade then tagged any positive overhead as `JouleSource::Measured`. ### What was wrong: on main - **`eoc query "totally novel question"` on the M5 Max** printed `cost : 50136686 µJ (measured)` in 4 of 4 runs. That is the 50 J neural-stage *estimate* plus a GPU-only IOReport delta, sealed into the receipt as measured. - **`eoc meter`** printed `counter: ioreport` with `idle : 307124 µJ` on the M5 Max, and `10279 µJ` on the M3 Ultra. Both are GPU-rail energy. - **The old test checked only the aggregate.** It passed on the M5 Max through the GPU rail and failed on the M3 Ultra (`energy must advance: before=0 after=0`). ### What this does - **`MacosIoReportCounter::try_detect()`** returns `Err(NoCounter)` unless the family rule says Live. `ioreport_status()` gives the reason. - The fallback counters can only return an error or 0, so `attach_cascade_cost` keeps each stage's own provenance. - After, on both chips: `cost : 50000000 µJ (estimated)`. - **`eoc meter`** prints the IOReport status and never prints a failed read as `0 µJ over 1.0 s`. - **eoc-streaming:** a stream's energy is Measured only when the counter actually advanced over the span. Before, any non-stub counter's zero delta was sealed as Measured. - **The counter counts CPU energy once.** `counts_once` admitted per-cluster and power-manager sub-rails (`PACC_*`, `MCPM*`, `PCPM`, `*_CPM`, and their `_SRAM` rails) beside the `CPU Energy` aggregate they sum into. The hierarchy was confirmed from absolute values on both chips (`CPU Energy` = `MCPU0` + `MCPU1` + `PCPU`, to within 1 mJ). On a live host CPU energy would have been summed twice. No reading so far was affected, because no host is live. ### Verification of this PR (head `65ff45ba`) - **Mutations:** the round-6 run caught 24 of 24, including a gate that declines only on CpuRailFrozen, and a constructor that opens before it declines. Restoring the old two-step rule fails 13 tests. - **Robustness:** 5/5 under all-channel contention, 5/5 under busy-loop load, then 3/3 with the new rule. - **M3 Ultra:** passes. - **eoc workspace:** 621 passed, 0 failed. ### Later rounds on this branch - `f84f0b2f`: a cache hit is an estimated zero, never a measured one. A total is Measured only if every microjoule was read. - `0db9b8be` and `6c37e299`: eoc-carbon carries provenance through EvalReport, CarbonAccount and SCI. An SCI score at PUE > 1 is Estimated, because its overhead is modeled. A run with no correct answer has no score; it used to produce an 18.4 TJ energy term. - `6a8d7c68`: an unmetered stream has no figure, never 0 µJ. - `15cf07c2` and `1535b8eb`: eoc-local never turns a failed first read into a since-boot total, and inference that ran, including a llamacpp error, is never 0 µJ. - `2c353d68`, `b2e8464c` and `26165a77`: - RAPL and NVML are gated on the liveness rule. - IOReport needs every CPU rail. - RAPL sums every `package-N` zone, never psys or dram, each wrapped at its own range. - `52d9d1e2`, `371fbe5e`, `6cf8b60e` and `1352aeea` (eoc-agent): - A failed first read can no longer charge a since-boot total to an agent's budget. - Tool steps and Plan-and-Execute steps are never charged 0. - Every trace span says whether its figure was measured or estimated. ### Why: what IOReport actually does on macOS 27.0.1 Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1: - **CPU rails are republished in rare lumps.** For an unprivileged reader, `CPU Energy` and `DIE_<n>_CPU Energy` do not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each. - **A lump can arrive in steps.** A watcher sampling every ~16 ms caught one as `+61,764,485 mJ`, then `+3,435 mJ` 64 ms later. - **GPU Energy is live on the M5 Max** (it changes every ~28 ms) and sporadic on the M3 Ultra. A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it. ### The rule: identical in all five backends (#110, #114, #115, #116, #117) - **The probe opens its own subscription to the "Energy Model" group.** A sample costs about 6–12 ms of thread CPU, against about 115 ms for all channels. - **It decides per CPU rail** (`CPU Energy`, or each `DIE_<n>_CPU Energy`), keyed by name and ordinal: - it compares raw integers from consecutive samples; - a decrease, or a rail's first appearance, is not an advance; - a rail advances at most once per interval. - **Live means every rail the meter sums kept advancing:** in at least 5 intervals, in at least 4 of every 5 intervals so far, over a window of at least 300 ms. - A live rail meets that in about 300 ms. A lump does not, even one published in several steps. - Every rail is required because the meter sums them. One live die beside a frozen die would otherwise give a figure that silently misses the frozen die while claiming the whole CPU. - **The same rule gates RAPL and NVML.** - RAPL sums the top-level zones whose `name` reads `package-N`, on every socket, and each must be live. It never sums `psys`, which already includes the packages, nor `dram` or the sub-zones. - Each zone wraps at its own `max_energy_range_uj`. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading. - An NVML (GPU-only) figure is Partial where the format can say so. Where it can't, it is never a hardware reading of the whole task. - **It samples until Live,** or until the 1 s budget has elapsed and at least 5 intervals have been seen, with a cap of 5 budgets. - Each tick burns 25 ms of this thread's CPU time, not wall time, with a 500 ms wall cap. So the probe keeps working on a loaded host. - **The result is cached once per process.** Anything but `Live` declines the backend, so the crate's existing honest fallback runs. The reason is public: `IoReportStatus`, `ioreport_status()`, `rapl_status()`, `nvml_status()`. - **The production status is one branch-free composition,** counted each time it runs. Every selection path reads it only through the cached status and a pure `admits(status)` (Live only). - **A live vendor counter's reading is a hardware reading:** HwShunt, or Measured in eoc. That follows the owner's position that vendor counters are real readings and the trust gap is provenance. It holds only while the counter is live, per the MUST in #119. ### How it was verified Six rounds of implementation, each checked by independent adversarial verification. **Pure tests on scripted samples and a scripted clock.** They cover: - the real refreshes (one step, two steps, 3–5 steps, the two-die shape); - decreases, first appearances and failed samples; - live rails (every interval, 80% of intervals, on a loaded host); - the sampling-loop bounds. **Fault hooks inside the real IOReport opener and sampler.** The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test. **Hardware tests against an independent oracle.** The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check: - the probe's sampled values equal the oracle's reads on either side; - the live GPU rail shows up in the probe's record, which proves the reads are fresh; - every probe sample is group-only, by channel count; - thread CPU per sample and per tick is checked against the tests' own clock; - the probe runs exactly once per process, checked in a fresh process. **Mutation-tested.** Each of the following was applied as a temporary edit and failed a test: - restore the old one- or two-advance rule; - compare every sample with the first; - treat a decrease as an advance; - map "no subscription" to Live inside the production status; - use a no-op tick, a tick that sleeps, or a tick on the wrong clock; - use a per-thread or non-atomic cache; - probe uncached from the constructor; - a sampler that returns constant, stale or summed values; - an all-channel subscription; - decide on the summed total or on the GPU rail; - an `admits()` that accepts CpuRailAbsent. The counts per crate are below. **Robustness.** Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra. **CI:** these tests are behind `cfg(target_os = "macos")`, and this workspace runs on a linux runner. They reach CI through **#102**, which merges **last**. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not print `ioreport liveness:`. **Merge order:** #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them. ### Open questions, now decided and implemented (per "fix it all") - **One grade for a live vendor counter.** It is HwShunt, or Measured in eoc, everywhere: a vendor counter is a real reading, and the gap is provenance. It holds only while the counter is live (the MUST in #119). - **The GPU rail as a partial reading of CPU work: no.** The GPU rail is machine-wide (display compositing moves it). Booking it to a task would misattribute energy and under-debit budgets. - **Wire keys named `measured_uj` that carry estimates.** Their grade now sits beside them in plaintext: jcp-gateway's `_meta` gets `energy_provenance`, plus `energy_coverage` when partial. The keys stay, for compatibility. - **eoc cache hits** are an estimated zero, never Measured. - **joule-ui.** `Meter` defaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. `Authorizer::charge` carries the figure's provenance. - **A live CPU energy source on macOS 27 exists.** It is the kernel's per-process counter `ri_energy_nj`: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privileged `powermetrics` is live too, but machine-wide and root-only, so it is not used.) ### Still open - **The joule-ui demo on prod.** It has run on macOS 27 since 2026-09-15. Its sealed chain was not inspected, because the read was blocked as a production read. After #114 merges, redeploying `joule-ui-serve` gives it honest grades, and the per-process probe in #121 gives it real readings. - **Linux RAPL and NVML have never run against real hardware.** No Linux host with readable RAPL and no NVIDIA GPU were available. They are covered by fake-tree and scripted tests, type-checked for Linux, and declined fail-safe. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 6 commits 2026-09-30 14:31:50 -04:00

eoc-meter: never construct the IOReport counter on a frozen CPU rail ... c637862902

Every energy backend in this family owes one contract: measured when the
platform exposes it, estimated when it doesn't, and the provenance always says
which; a reading never silently upgrades its own provenance. eoc-meter's
IOReport counter broke it on current macOS, and eoc sealed the result into its
receipts as measured.

Ground truth, read from the channels on both machines (macOS 27.0.1), not
inferred from test outcomes:

- M5 Max: of 364 Energy Model channels, only `GPU Energy` moved across a 3 s
  single-thread CPU burn (+851,622,003 nJ). `CPU Energy` held at 541,893,638 mJ
  (absolute) and no CPU sub-rail moved.
- M3 Ultra: of 645 channels, none moved in 3 s; over 12 s only `GPU Energy`
  did (+14,809,813 nJ). `DIE_0_CPU Energy` / `DIE_1_CPU Energy` held at
  9,494,336 / 10,212,656 mJ.
- The CPU rails are not frozen for good; they are refreshed now and then. A
  25-minute watch, one sample a second, saw the M5 Max's `CPU Energy` hold at
  541,893,638 mJ throughout (PR #110 records 73,176,481 mJ for an M5 Max
  earlier today; this one has been up since 00:23). The M3 Ultra's rails held
  from at least 10:47 to 11:09, then both dies stepped within one sample,
  +1,344,869 and +1,391,637 mJ, and held again.

The defect, as eoc showed it on main (860ead8c):

- `MacosIoReportCounter::try_detect()` returned the counter whenever a
  subscription opened, so `detect()` chose it on every Apple Silicon host. Its
  reading sums the Energy Model rails, so on the M5 Max it advanced only
  through the GPU rail.
- `eoc query "totally novel question"` on the M5 Max printed
  `cost : 50136686 µJ (measured)`: the neural stage's 50 J estimate plus a
  136,686 µJ GPU-only delta, which `attach_cascade_cost` tags Measured because
  it is positive. The receipt seals that tag. Four runs of four did this
  (+136,686, +533,511, +272,220 and +231,665 µJ).
- `eoc meter` printed `counter: ioreport` and `idle : 307124 µJ over 1.0 s`
  on the M5 Max, `idle : 10279 µJ over 1.0 s` on the M3 Ultra: GPU energy,
  presented as the machine's idle draw.
- The test meant to guard this, `ioreport_advances_after_cpu_burn`, asserted
  only that the summed total advanced across a 200 ms burn. The GPU rail
  satisfied it on the M5 Max; on the M3 Ultra it failed (`energy must advance:
  before=0 after=0`); where IOReport can't open it returned early and passed.

The fix. `try_detect()` now returns the counter only when `ioreport_status()`
is `Live`, and `Err(NoCounter)` otherwise, so `detect()` falls through. The
probe is jcp-energy's (PR #110, dfec5d44), copied in to keep the crate
self-contained: burn CPU on the probing thread, re-sample after every ~25 ms of
it for up to a second, and decide on the raw integers of `CPU Energy` or
`DIE_<n>_CPU Energy`, never on a summed total, because the live GPU rail is
what corrupts the total. Probed once per process and cached. `IoReportStatus`
(`Unavailable`, `CpuRailAbsent`, `CpuRailFrozen`, `Live`) and
`ioreport_status()` are public so a caller can say why a Mac is not metered by
IOReport. `Err(NoCounter)` is reused rather than adding an `eoc_core::Error`
variant: the reason lives in the status, and the error enum is another crate's
public API.

Two deliberate differences from the reference, both measured:

- The probe samples its own subscription to the Energy Model group, not every
  channel. Sampling every channel took ~781 ms per call on the M5 Max and
  ~144 ms on the M3 Ultra, the group ~6 ms. On every channel a one-second probe
  took 2 samples and spent 3% of its window spinning on the M5 Max (12% on the
  M3 Ultra); on the group it takes 31 samples and spins 77% (75%). "Poll every
  ~25 ms while this thread spins" holds only on the group. Both subscriptions
  read the same absolute CPU values. The counter's own subscription is
  unchanged, and the test oracle reads that one.
- A rail counts as live only when one CPU channel steps at two separate
  samples, not one. A live rail sampled every ~30 ms under load steps many
  times a second; a refresh steps once. On a single step, a refresh landing
  in the probe's second would select the counter for the life of the process,
  and every later refresh (~1.3 kJ per die, above) would land whole in
  whichever caller's interval straddled it, tagged Measured. A rail too slow
  to step twice in a second is declined: estimated, never over-claimed.

What runs instead, end to end. `detect()` now falls through to
`MacosPowermetricsCounter`, every read of which is `Err(NoCounter)`, or to
`StubCounter`, which reads 0. Neither can produce a positive delta, so
`attach_cascade_cost` keeps each stage's own source and no consumer here
(eoc-cascade, eoc-local, eoc-streaming) can relabel the fallback as a reading;
eoc-cascade is unchanged. `eoc query` now prints `cost : 50000000 µJ
(estimated)` on both machines. `eoc meter` printed a failed read as `0 µJ over
1.0 s (0.000 W)`; it now prints the IOReport status and `not metered (counter
read failed: no joule counter available)`.

Tests. The aggregate test is replaced by three that assert against an
independent oracle. The oracle samples the counter's all-channel subscription,
reads the CPU rails' ABSOLUTE values (no deltas), matches names with
`ends_with("CPU Energy")` rather than the implementation's group filter and
die-prefix stripping, and burns CPU on its thread for three times the probe's
budget between one read before and one after. Every branch asserts. The
oracle is not blind: the same absolute reads show `GPU Energy` moving on the
M5 Max.

- `the_ioreport_counter_is_never_selected_on_a_frozen_cpu_rail`: the property.
  Logs `ioreport liveness: status=.. oracle=.. selected=..`.
- `the_liveness_probe_never_reports_a_frozen_rail_as_live`: the dangerous
  direction, on a fresh uncached probe, which must also agree with the oracle
  on what exists.
- `a_selected_ioreport_counter_meters_cpu_work`: when the counter is available
  its CPU rail is live and a 200 ms burn meters non-zero; when it is not, the
  status is not `Live`.
- `a_cpu_rail_that_steps_once_is_not_live` and
  `a_cpu_rail_that_keeps_stepping_is_live`: the step rule, fed the M3 Ultra's
  observed refresh. A refresh cannot be scheduled, so no hardware test can
  catch a single-step rule on demand.
- eoc: `a_counter_that_did_not_read_is_never_printed_as_a_figure`.

Observed on both machines: `status=CpuRailFrozen oracle=CpuRailMoved(false)
selected=false` and `probe=Some(Frozen)`, 9/9 passing; on the M3 Ultra through
the `DIE_0_`/`DIE_1_` rails.

Falsified, each reverted with git checkout:

- F1, a probe that always says Live: the frozen-rail test fails on both
  machines ("IOReport selected (status=Live) although its CPU rail did not
  move under 3s of CPU burn ..."), and so does the probe test. The third test
  fails too on the M3 Ultra (before=0 after=0) but passes on the M5 Max, where
  the GPU rail moves the summed counter: the defect itself.
- F2, main's ungated `try_detect()`: the frozen-rail test fails on both
  ("IOReport selected (status=CpuRailFrozen) ..."), and so does the third.
- F3, liveness decided on every Energy Model rail, as a total would: fails on
  the M5 Max, where the GPU rail steps inside the probe window; passes on the
  M3 Ultra, where no rail stepped twice in that second.
- F4, main's `eoc meter` rendering: the CLI test fails on "0 µJ over 1.0 s
  (0.000 W)".
- F5, the reference's single-step rule: `a_cpu_rail_that_steps_once_is_not_live`
  fails ("one refresh counted as a live rail").

Every crate that depends on eoc-meter or eoc-cascade passes: eoc, eoc-agent,
eoc-carbon, eoc-cascade, eoc-distributed, eoc-local, eoc-observability,
eoc-route-learned, eoc-spec-decode, eoc-streaming, and eoc-wasm (no tests).
Clippy adds nothing on eoc-meter or eoc; eoc-meter's two existing warnings
remain. The probe costs about a second of one thread's CPU, once per process,
on Apple Silicon; `eoc query` on the M5 Max still got faster (1.11-1.24 s,
against 1.39-1.43 s on main, which paid for two all-channel reads per query).

Not done here, deliberately:

- `counts_once` admits CPU sub-rails beside `CPU Energy` (`PACC_0`..`PACC_5`,
  `MCPM*`, `PCPM*` on the M5 Max; `DIE_n_PACC*_CPM*`, `DIE_n_EACC_CPM*` on the
  M3 Ultra): a double count if those rails go live.
- The counter's own reads sample every channel (~781 ms a read on the M5 Max)
  although its sum uses only the Energy Model group.
- The powermetrics rung's docs say it degrades to `StubCounter`. It never does;
  every read errors.
- With a live counter, `attach_cascade_cost` tags a stage's estimate plus the
  measured overhead as one Measured total.
- `JouleCost::zero()` is Measured, so a cache hit claims a measured zero; the
  streaming meter tags a zero delta from any non-stub counter Measured;
  `sci_from_eval_report` tags every report Measured.
- A rail that stops tracking after selection: the probe runs once a process.
- Reporting the live GPU rail as an explicitly partial reading, and the grade
  IOReport readings carry: separate decisions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-meter: CPU-rail liveness pairs consecutive samples, and every step of it is tested ... 3b827e2a52

c6378629 gated the IOReport counter on a CPU-rail liveness probe, but no one
had checked it against the family rule, and the part of the probe where this
family's defect first appeared was never tested. Auditing it against the rule
found these gaps:

- The probe loop stopped at the one-second budget however few intervals it
  had seen. The rule is to keep sampling until the rail is Live, or until the
  budget has elapsed AND at least two intervals were seen, capped at five
  budgets. The loop ran inline in the FFI function, so its pairing,
  termination and failed-sample handling could not be tested at all.
- A failed first sample, or a probe that got no interval, was reported as
  CpuRailAbsent ("no CPU rail"). The rule reports Unavailable: nothing was
  learned.
- The oracle read the rail twice, before and after a burn on its own thread,
  and called one change "moved". So a single batch refresh inside its window
  counted as a moving rail. It opened its subscription through the
  implementation's MacosIoReportCounter::open() and converted names with the
  implementation's cfstring_to_string. So c6378629's doc claim that it "shares
  nothing with the liveness probe but the raw FFI" was false, and so was the
  message's "the same absolute reads show GPU Energy moving": the oracle never
  read GPU Energy.
- Nothing pinned the once-per-process cache, the probe's cadence on the
  Energy Model group, the verdict-to-status mapping, the Live-only gate, or
  what a declined host meters.
- open_energy_model() leaked the subscribed-channel dictionary when
  IOReportCreateSubscription returned null. The counter's own open() did
  too, before this branch.

What holds now:

- The rule is pure (mod rail_liveness). It is compiled on every platform
  under test, so the Linux CI leg runs it too. The FFI only takes a sample
  and lists each Energy Model channel as (name, absolute raw integer).
  RailWatch::observe keys each CPU rail (the name after stripping DIE_<n>_ is
  "CPU Energy", case-insensitive) by (channel name, ordinal among
  same-named channels in that sample). It compares each usable sample with
  the previous usable one, never with the first. So a rail advances at most
  once per interval, and dies are never summed. The M3 Ultra lists 264 of its
  645 Energy Model names twice. Two same-named rails stepping together are
  one advance each, not two advances of one rail.
- watch(sample, tick, now, budget) is the probe loop. It is generic over the
  sampler, the tick and the clock. The real probe passes the IOReport
  sampler, 25 ms of CPU spin on the calling thread, and Instant. Live means
  the same rail advanced in >= 2 intervals. The loop keeps sampling until
  Live, or until the budget has elapsed with >= 2 intervals, capped at five
  budgets. No interval at all is no verdict, which maps to Unavailable.
- status_of() and admits_counter() are pure. try_detect() admits the counter
  only on Live.
- PROBE_RUNS counts the real probe behind ioreport_status().
- The oracle opens its own all-channel subscription with raw calls and
  converts names with its own helper. It reads absolute values for
  ends_with("CPU Energy") and ends_with("GPU Energy") over and over for 3 s,
  while a helper thread burns CPU for the whole window. It says the rail
  moved only if one rail changed at >= 2 reads, and it must get >= 3 reads
  before it may say Frozen.
- The powermetrics docs said the rung degrades to StubCounter. It never does:
  every read errors. The docs now say so, and a test pins it.

Tests added:

- Scripted samples and a scripted clock:
  - (a) the M5 Max refresh 480,083,160 -> 541,893,638 is Frozen: exactly 40
    intervals and 41 samples, while GPU Energy moves at every sample;
  - (b) the M3 Ultra's two dies stepping +1,344,869 / +1,391,637 in one
    sample is Frozen;
  - (c) same-named rails are counted apart;
  - (d) a rail stepping every sample is Live after exactly 2 intervals and
    3 samples;
  - (e) failed samples sample to the cap (201 samples) and give no verdict,
    which is Unavailable;
  - (f) past the budget the probe stops at once with two intervals, and keeps
    sampling with fewer;
  - (g) one step on each die, in different intervals, is Frozen.
  The verdict-to-status mapping, the Live-only gate and the name rule each
  have a test.
- Hardware tests:
  - the_liveness_probe_runs_once_per_process: exactly one run after repeated
    ioreport_status(), detect() and try_detect() calls;
  - the fresh-probe test requires >= 10 intervals, and the probe thread on
    CPU for >= 50% of the probe, whenever the verdict is not Live;
  - on a declined host, detect()'s fallback measures 100 ms of real CPU work
    as no figure (both reads error, or it is the stub).
- In the eoc binary, work_on_an_unmetered_host_keeps_its_estimate runs the
  CLI's own cascade around a backend that burns 100 ms. On this host it must
  come back as the backend's 50,000,000 µJ estimate, tagged estimated.
  (async-trait becomes a dev-dependency of eoc for that backend.) `eoc meter`
  now says Unavailable can also mean "yielded no samples".

Observed on the M5 Max, 3 runs of 3, macOS 27.0.1:

  ioreport probe: probed=Some(Some(Frozen)) cached=CpuRailFrozen
  took=1.11s on-cpu=95-98% intervals=30-31
  ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
  selected=false (oracle: 9-12 reads, CPU rail changes [0], GPU rail 8-11)
  eoc-meter 19 passed, eoc 2 passed

Observed on the M3 Ultra (10.10.10.2), same binaries by sha256:
intervals=32, on-cpu=100%, CPU rail changes [0, 0] on DIE_0/DIE_1, GPU rail
changed at 17 and 20 of 21 oracle reads. eoc-meter 19 passed, eoc 2 passed. On both machines,
`eoc query` prints `cost : 50000000 µJ (estimated)`, and `eoc meter` prints
the declined status and "not metered".

Falsified, each restored with git checkout (clean status every time). Each
of these fails at least one test:
- comparing every sample with the first, in RailWatch and in the loop;
- one advance counting as Live;
- advances summed across dies;
- rails keyed by name alone;
- GPU Energy counted as a CPU rail;
- no DIE_ stripping;
- c6378629's stop-at-budget loop;
- no five-budget cap;
- None mapped to Live or to CpuRailAbsent;
- the gate admitting CpuRailAbsent;
- Absent reported for rails that did not advance;
- a coarse 1e9 threshold;
- no OnceLock (the probe ran 14 times);
- an all-channel probe (2 intervals in 1.86 s);
- a probe that sleeps (26% on CPU);
- a powermetrics fallback reading Ok(0) or a time-based count;
- main's ungated try_detect();
- a probe that always says Live.
One mutation survives: the counter's own read returning Ok(0). On a frozen
host the counter is never constructed, so only a Live host runs that path.

Leak check: 150 probes plus 150 all-channel opens under leaks --atExit gave
"0 leaks for 0 total leaked bytes". A control with CFRelease(sample)
removed gave 100,080 leaks.

The whole eoc workspace passes: 594 passed, 0 failed, 3 ignored. That covers
every dependant c6378629 ran. Clippy adds nothing: the two eoc-meter
warnings are on lines that predate this branch.

Not done here: counts_once still admits CPU sub-rails; the counter's reads
still sample every channel; and a rail that freezes after selection is not
re-probed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-streaming: a stream's energy is measured only when the counter advanced ... 467acb385f

StreamMeter::start marked a stream measured as soon as one counter read
succeeded, as long as the counter was not named "stub". So any other counter
that read but did not move had its zero sealed as JouleSource::Measured. That
includes a frozen rail, a placeholder, or a counter with no resolution over
the span. The flag was also set before any checkpoint, so snapshot() reported
a measured 0 µJ for a stream that had not been metered at all. c6378629
listed this as deferred.

A second defect sat in the same function. When the start read failed, the
baseline was 0, so the first successful checkpoint reported the counter's
whole since-boot total as the stream's energy.

What holds now. The baseline is an Option. A checkpoint computes a delta only
when there is a baseline and a reading now. The account is `measured` only
when the counter advanced over the span (now > start). Without a baseline or
a reading, the account stays as it was. StreamAccount's fields and their types
are unchanged. The `measured` doc now says what the flag means.

Tests: a non-stub counter that reads a constant is not a measurement (before
and after a checkpoint). A counter that advances by 450 µJ is
JouleCost::measured(450). A failed start read followed by a reading of
9,000,000 µJ measures nothing. Falsified, each restored with git checkout:
- the old `name() != "stub"` rule fails two tests;
- `>=` in place of `>` fails three;
- a 0 baseline after a failed start fails with `left: (9000000, true)`;
- marking the stream measured at start fails with "measured before any
  checkpoint".

eoc-streaming passes: 20 unit tests plus its integration tests. The whole eoc
workspace passes: 594 passed, 0 failed, 3 ignored.

Not done: an unmetered stream still reports `0 µJ (estimated)`, because
StreamAccount has no way to say "not metered". Saying so honestly would
change a public type.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-meter: the liveness tests hold on a loaded host, and pin what the probe does ... bc6d497ef1

The round-3 review of 3b827e2a found the hardware liveness tests flaky on
the M5 Max that also runs the macOS CI. With three test processes at once
and a load average of 97-172, 8 of 12 runs failed on "the probe's thread
was on CPU for only 33% ... 50%", and the oracle once completed 2 reads in
its fixed 3 s window. Both assertions measured wall time, which host load
stretches. A sleeping probe (26-27% on CPU) sat close to a loaded burning
one (33-57%).

The probe now burns thread CPU, not wall time. Each tick spins until this
thread's CPU clock (clock_gettime(CLOCK_THREAD_CPUTIME_ID)) has advanced
25 ms. A descheduled thread's CPU clock stands still, so load stretches a
tick in wall time but never shortens its CPU. A wall cap of 500 ms bounds a
tick on a host that gives the thread under 5% of a core. The spin is a pure
function, spin_for_cpu, tested with scripted clocks, and the real tick is
still injected into the generic loop.

The probe records each sample call: its wall duration, and the thread's
CPU clock before and after. The tests read that record instead of timing
the probe from outside, and no hardware test asserts a wall-clock ratio.

- the_probe_burns_its_tick_of_thread_cpu_between_samples: the CPU consumed
  between one sample call and the next must lie between 12.5 and 50 ms,
  against a literal 25 ms in the test. Only the 500 ms wall cap could make a
  tick fall short, and the message says so. The upper bound catches a slower
  cadence: round 3's surviving PROBE_TICK = 100 ms now fails. The recorded
  tick CPU is also checked against the test's own reading of the thread
  clock over the whole probe.
- the_probe_samples_only_the_energy_model_group: the median wall duration
  of the probe's own sample calls must be under 50 ms. The Energy Model
  group takes ~6-12 ms and every channel 130-800 ms. This replaces the
  ">= 10 intervals" and ">= 50% on CPU" assertions.
- the_probe_sampler_reads_each_cpu_rail_as_the_oracle_does: for three
  samples, the oracle reads each CPU rail just before and just after the
  sample the probe's sampler takes. A rail that held must read EQUAL, and
  a rail that moved must read between the two oracle reads. The sampler
  must also list exactly the oracle's CPU rails. This catches a sampler
  that returns a constant 0 or another rail's value.

The oracle now reads until its window has passed AND it has 3 reads, the
fewest in which a rail can change twice, capped at five windows. Fewer
than 3 reads is a test failure, never a Frozen verdict. A failed first
read no longer reports "no IOReport".

Round-3 survivors:
- F21: the counter's own read path now has a test on the hosts we have.
  the_counter_advances_when_a_rail_it_sums_advanced builds the counter past
  the gate on purpose. It takes a baseline, then reads GPU Energy (which
  the counter sums) independently in absolute values, with the oracle's own
  unit table, until it has risen by 1 µJ or 3 s pass, then reads the
  counter again. If the GPU rose, the counter must have advanced. If not,
  the counter is printed, not asserted, and the line says so.
- probe_status mapped "no Energy Model subscription" inline, and a
  mutation to Live survived. It now goes through the pure
  status_from_probe(). Following the family rule, a group that cannot be
  subscribed is Unavailable, never CpuRailAbsent: CpuRailAbsent now means
  the group was sampled and held no CPU rail. The IoReportStatus docs and
  the `eoc meter` line for Unavailable say so. No variant changed.

The pure rules each have their own test now: a decrease is not an advance;
a rail absent from the previous sample only sets its baseline; no
subscription or no sample is Unavailable; no verdict is Unavailable. The
once-per-process pin releases six threads together, and each asks for the
status, try_detect() and detect(). The process-wide PROBE_RUNS must read
exactly 1. The test that prints "ioreport liveness: status=" keeps its name
and line.

On this M5 Max, unloaded, 28 passed: per-tick thread CPU 25.01-25.03 ms,
median sample 11.8 ms, CPU Energy read identically by the sampler and the
oracle, GPU Energy +10.6 mJ in 257 ms with the counter at 0 -> 53,238 µJ.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-meter: the IOReport counter counts CPU energy once ... 7aa424188c

counts_once, the filter on what the counter sums, admitted CPU rails whose
names do not say "CPU" beside the `CPU Energy` aggregate they add up to.
Round 3 carried this as latent. It is now confirmed from one absolute
sample of the Energy Model group on each chip:

- M5 Max: `CPU Energy` = `MCPU0` + `MCPU1` + `PCPU` (1,863,990,848 vs
  1,863,990,847 mJ). `PCPU` = `PACC_0..5` + `PCPU0..5_SRAM` + `PCPM` +
  `PCPM_SRAM`, and `MCPUn` = its cores + their SRAM + `MCPMn` + `MCPMn_SRAM`.
  counts_once admitted `PACC_*`, `MCPM*`, `PCPM` and their `_SRAM` rails.
- M3 Ultra: `DIE_n_CPU Energy` = its `EACC_CPU` + `PACC0_CPU` + `PACC1_CPU`,
  each the sum of its cores, their SRAM and its `_CPM` + `_CPM_SRAM` rails
  (all to within 1-8 mJ). counts_once admitted `DIE_n_EACC_CPM`,
  `DIE_n_PACC*_CPM` and their `_SRAM` rails.

On a host whose CPU rail is live, the counter would have summed that CPU
energy twice and sealed the result as measured. Neither chip is live on
macOS 27.0.1, so no reading so far was affected.

counts_once now also drops per-cluster accumulators (`PACC*`, `EACC*`,
`MACC*`) and CPU power managers (`*CPM*`). The aggregates, and the
domains that are no aggregate's summand (DRAM, AMCC, DCS, display, ISP,
AVE, MSR, FAB, PCIe), are still summed. counts_once_drops_subrails now
pins this on the real channel names of both chips.

Not changed: neither chip has an `ANE Energy` aggregate, and counts_once
drops `ANE0` (and `ANE0_0`/`ANE0_1`) as summands, so ANE energy is never
summed. Counting it needs a rule that looks at the whole sample, and that
is left for later.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-meter: a tick that spins on the wrong clock fails its test, not hangs it ... 0e18d6fe9b

In the_tick_spins_until_the_thread_cpu_clock_advanced, the scripted wall
clock advanced only when the spin read the CPU clock. So a spin_for_cpu
that measured its target on the wall clock never read the CPU clock, the
scripted time never moved, and the test hung until killed (900 s in the
mutation run) instead of failing.

Each scripted clock now advances on its own reads. The correct spin reads
each clock once per turn, so the expected values are unchanged. A spin that
targets wall time now stops at 28 ms of wall time and fails the assertion
that it burned 25 ms of CPU.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 14:32:47 -04:00

ci(standards-build): run all five IOReport meters on macOS, and fail a step whose test never ran (merge LAST) #102

dcharlot added 1 commit 2026-09-30 17:52:00 -04:00

eoc-meter: the tests drive the production liveness chain itself, and hold under load ... 33f6d78c2c

The round-5 review of 0e18d6fe found eight gaps. Four were on the
dangerous side:

- The group-only pin (median wall time per sample call < 50 ms) failed
  spuriously on the loaded M5 Max that runs macOS CI: 2 of 23 load runs.
- The once-per-process count sat in the cache wrapper, so a constructor
  that probed uncached through probe() passed. An instrumented run made 21
  real probes while the test printed 1.
- Mapping "no Energy Model subscription" to Live or CpuRailAbsent inline in
  probe_status, around the pure mapping, survived on both chips.
- try_detect()'s gate rewritten inline to decline only CpuRailFrozen
  survived on both chips.
- A stale sampler survived, because only the frozen CPU rails were
  bracketed.
- A probe_tick that spun 25 ms of wall time passed unloaded.
- the_counter_advances_when_a_rail_it_sums_advanced asserted nothing when
  GPU Energy did not rise.
- CPU_RAIL_LIVENESS_BUDGET's doc called a second of wall time "a second of
  single-thread CPU work".

What holds now.

The production chain is thin, branch-free, and the one the tests run.
host_status() adds one to a process-wide AtomicUsize, then returns
status_of(probe_with(real opener, real tick, thread CPU clock, wall
clock, budget).verdict()). ioreport_status() caches it in a OnceLock and
is its only production caller. probe_with is the only place that composes
the opener, the sampler, the tick and the clocks. An opener that fails
gives a run with no verdict, and the pure status_of maps no verdict to
Unavailable. try_detect() is try_detect_given(ioreport_status()).
try_detect_given and detect() both gate through the pure admits_counter(),
which admits Live only. The real tick is one call of the pure spin_for_cpu
with this thread's CPU clock and the wall clock injected.

Fault injection sits inside the real leaves. In test builds, a script
installed on a thread stands in for IOReport in open_energy_model() and
State::group_sample(), and nowhere above them. The tests call the real
host_status(), uncached, under these scripts:

- the opener fails: Unavailable, with no sample taken and no tick run
- a single batch jump: CpuRailFrozen
- two advances: Live, at the third sample
- one step on each of two dies: CpuRailFrozen
- a constant rail: CpuRailFrozen, and every production tick burned 25 ms
  of thread CPU by the test's own clock

A status installed on a thread likewise stands in for the cached one, so
the real try_detect() and detect() run under all four statuses.

The pins. No pin bounds wall time from above, so host load cannot fail
one.

- Group-only, by structure. Every sample the production probe took has
  exactly as many channels as the Energy Model group in the oracle's
  all-channel read, and every one is in the group: 364 of 11,886 channels
  on the M5 Max, 645 of 14,345 on the M3 Ultra. Also, the median thread CPU
  of a sample call must be under 50 ms. It measured 6-9 ms; every channel
  cost 110-120 ms in round 5.
- The clock, against the tests' own CLOCK_THREAD_CPUTIME_ID literal (16).
  The implementation's clock keeps pace while this thread burns. It does
  not move across a 100 ms sleep, or while another thread burns 100 ms.
  Each production tick is at least 25 ms of thread CPU or 500 ms of wall
  time, and never more CPU than wall. This is checked once unloaded, and
  once beside 2 x ncpu busy threads, where at least one tick must have
  shared its core.
- Freshness. The production probe runs, uncached and outside
  host_status(), between two oracle reads of the Energy Model group's CPU
  and GPU energy rails. The run records the (key, raw) values it handed
  the rule, GPU Energy included. They must carry the oracle's keys, EQUAL
  a rail that held, and lie within a rail that moved. When the oracle saw GPU Energy change, the
  record must show it changing too.
- Once per process. tests/liveness_probe_once_per_process.rs runs in its
  own process. It releases 8 threads at a cold cache through status,
  construction and selection, in three orders. Two counts must each be
  exactly 1: runs of host_status(), and opens of the probe's sampler. The
  second count catches a probe run outside the cache that never touched
  host_status().
- Declines before anything opens. Given Unavailable, CpuRailAbsent or
  CpuRailFrozen, try_detect_given returns Err(NoCounter), and detect_with,
  try_detect() and detect() decline too. None of them attempts to open the
  counter's subscription, by a per-thread count inside its opener. Given
  Live, the attempt is counted. This test runs on every macOS host,
  whether or not IOReport opens there.
- The counter's value. The counter is built past the gate on purpose, and
  its delta over a window of at least 3 s must lie between two rises of the
  oracle's sum of the same summed rails: the rise inside the window, and
  the rise around it. The oracle rounds rail by rail, as the counter does.
  0 == 0 is asserted too; no branch skips.

The budget doc now says the second is wall time. A loaded host gives the
probe fewer intervals, which fails safe.

Evidence. eoc-meter: 37 lib tests + the once-per-process test pass.

- M5 Max, unloaded: per-tick thread CPU 25.02-25.03 ms. Median thread CPU
  per sample call 6.5 ms. `ioreport liveness: status=CpuRailFrozen
  oracle=CpuRailMoved(false) selected=false`. The counter read 136,442 uJ
  against a bracket of 126,920..162,549 uJ; GPU Energy was the only rail
  that moved.
- M5 Max, 5 runs with a helper process sampling every IOReport channel in
  a tight loop: 5/5 ok, 37 passed each.
- M5 Max, 5 runs beside 1 x ncpu busy spinners (load average 23-39): 5/5
  ok, 37 passed each. The probe saw 4-11 intervals.
- M3 Ultra (10.10.10.2, macOS 27.0.1), 1 run: 37 passed, and the
  once-per-process test passed with 1 run and 1 open. Both DIE_n_CPU Energy
  rails were read equal to the oracle. 12 of 17 production ticks shared
  their core with 64 busy threads, each still burning 25.00-25.01 ms of
  CPU.
- The whole eoc workspace: 613 passed, 0 failed, 3 ignored. The ignored
  three are pre-existing ignored doctests.
- clippy --no-deps on eoc-meter, all targets: the only two warnings are
  from 1bae1ea1 (a duplicated #[link] attribute, and a b"...\0" literal).
  The aarch64-linux cross type-check and clippy are clean.

Mutations. Each was a temporary edit, then git checkout; the tree was
clean after every one. All 24 were caught (test that failed in brackets):

- compare with the first sample [9 pure rule tests + host_status]
- decrease counts as an advance [a_decrease_is_not_an_advance]
- no subscription -> Live, and -> CpuRailAbsent, inside host_status
  [host_status_is_the_rule_on_the_production_composition]
- single advance -> Live, inside host_status and inside the production
  sampler closure [the same]
- no-op tick; tick spinning 25 ms of wall time; tick timed on the wall
  clock [the production-tick and probe-tick tests]
- per-thread cache [once-per-process: "ran 8 times"]
- try_detect probing uncached, through host_status and through
  production_probe [once-per-process: 9 runs / "sampler was opened 9
  times"; only_a_live_status_constructs_or_selects_the_counter]
- try_detect's gate inlined to decline only CpuRailFrozen, and
  try_detect_given likewise [a_declined_status_declines_before_anything_opens]
- try_detect_given opening before it declines [the same: "left: 4, right:
  0"]
- sampler constant 0, in the FFI extraction and in the production closure
  [sampler/freshness tests, host_status]
- stale sampler, in the production closure and cached per thread below
  the script hook [freshness: "every one of the 39 samples ... read
  [126274575439834]: its sampler is not sampling"]
- thread clock = CLOCK_MONOTONIC and = CLOCK_PROCESS_CPUTIME_ID
  [the_probe_thread_clock_is_this_threads_cpu_time, and others]
- counts_once re-admitting PACC_* [counts_once_drops_subrails]
- the counter's sum skipping GPU Energy, or counting every rail twice
  [the_counter_advances_when_a_rail_it_sums_advanced]

Not done here, deliberately:

- A sampler stale BELOW the script hook is caught only through GPU Energy
  moving during the 1 s probe. On a host whose GPU rail holds, the
  freshness line says "NOT asserted" for it. Then a stale value is also
  the true value, and the held-rail equality still asserts. Staleness at
  or above the hook is caught by the scripted tests on any host.
- The crate was not rustfmt-clean before this branch, and it is not
  reformatted here, so the diff stays reviewable.
- Linux: a cross type-check and clippy only, not a run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 18:48:33 -04:00

eoc-meter: a rail is live only if it keeps advancing, never on a two-step refresh ... 65ff45ba75

The jouleclaw-energy verifier found the defect on 2026-09-30, on the M5 Max
(macOS 27.0.1). The liveness rule said Live when the same CPU rail advanced
in two separate sample intervals (LIVE_RAIL_ADVANCES = 2). That rule assumed
a batch refresh lands as one step. It does not always. A watcher sampling
the Energy Model group alone every ~16 ms saw `CPU Energy` step
2,643,843,378 -> 2,705,607,863 (+61,764,485), then 2,705,607,863 ->
2,705,611,298 (+3,435) 64 ms later. MCPU0, MCPU1 and PCPU each stepped twice
too. The probe samples every ~31 ms, so a probe window that holds such a
refresh sees two advancing intervals and says Live, and the status is cached
for the process. That happened: two real probes said Live while the oracle
saw no sustained movement, and both coincided with a refresh. Refreshes come
~30 minutes apart, so this is rare, but it is exactly the failure the rule
exists to prevent. On this M5 Max `CPU Energy` has held at 2,705,611,298,
that refresh's second step, in every run for this commit that printed it.

The rule now asks for a sustained advance. A rail is live only if the same
rail (same key: name + ordinal) meets three bars after some interval:

- it advanced in at least MIN_LIVE_ADVANCES = 5 separate intervals;
- those were at least 80% of every interval observed so far
  (advancing * 5 >= intervals * 4);
- the samples observed so far span at least MIN_LIVE_WINDOW = 300 ms of
  wall time, measured from the first usable sample.

It is evaluated after each interval, and Live is declared the first time all
three hold. A burst of steps fails the window while it is young, and fails
the share once it is not. The rule needs each sample's wall time, so
RailWatch::observe now takes it: watch() reads its injected clock as each
sample returns, and tests script that clock.

The probe keeps sampling until the rail is Live, or until the budget (1 s)
has elapsed and at least 5 intervals were observed, and never past 5
budgets. Otherwise the verdict is Frozen, Absent or Unavailable, as before.
Nothing else changed: per-rail keys, consecutive-sample pairing, a decrease
or a first appearance is not an advance, one advance per rail per interval,
the group-only subscription, the thread-CPU tick, the pure status mapping,
the OnceLock cache, the scripted leaves, try_detect_given / admits_counter,
counts_once, and the "ioreport liveness: status=" line.

The cost: a live rail is declared after 300 ms at the least, not ~60 ms. A
frozen verdict still takes the ~1 s budget.

The test oracle had the same weakness. It said CpuRailMoved(true) when one
rail changed at two reads, so a refresh that coincided with a false Live
vouched for it. Now it is CpuRailMoved(true) only if one rail changed at
least 5 times, at no fewer than 80% of the read pairs it compared. It keeps
reading past its window until it has compared 5 pairs (6 reads), so a live
rail could meet the bar, and it stops at 5 windows. Too few pairs is still a
test failure, never a verdict. The per-read tally is now a pure OracleTally,
so the bar runs on scripted reads too.

The docs that described the old two-step rule are updated: the constants,
CpuRail, RailWatch, the macos_ioreport module comment,
IoReportStatus::{CpuRailFrozen, Live}, CPU_RAIL_LIVENESS_BUDGET and
ioreport_status.

Tests. The rule tests use a scripted 31 ms interval unless noted. (*) marks
the ones that must fail under the old ">= 2 advances" rule. That was
checked by temporarily restoring the old rule: 13 tests failed, every (*)
test among them. The F9 test failed on its two-step script: "a refresh in
two steps (5 samples) must be frozen", left: Live.

- (*) h_the_two_step_refresh_is_frozen: the observed +61,764,485 / +3,435
  refresh is Frozen at every position tried. The steps land in intervals
  (3,4), (1,2), (1,3), (3,5), (20,21) and (31,33), beside a moving GPU
  rail and sub-rails that step with it.
- (*) i_a_refresh_in_three_to_five_steps_at_the_start_is_frozen: 3, 4 or 5
  steps within 150 ms at the start (30 ms a sample) are Frozen. At five
  steps every interval so far has advanced; only the window stops it.
- (*) j_the_m3_ultra_two_steps_on_each_die_are_frozen: DIE_0 and DIE_1
  each step twice within 64 ms, together or one interval apart, at the
  start and mid-probe, amid same-named per-die channels. Frozen.
- d_a_rail_advancing_every_interval_is_live_at_five_advances_and_300_ms:
  not Live at 279 ms; Live at the 10th interval (310 ms), and the probe
  stops there. At 60 ms a sample it is Live at the 5th interval (exactly
  300 ms); at 59 ms, only at the 6th.
- k_a_live_rail_missing_one_interval_in_five_is_live_two_is_not_yet:
  missing 1 interval in every 5 is Live at 8 of 10. Missing 2 of the first
  5 (100 ms a sample) is not Live at 3 of 5 or at 7 of 9; it is Live at
  8 of 10. Missing 2 in every 5 is Frozen.
- l_a_live_rail_on_a_loaded_host_is_live_after_five_intervals: at 100 ms
  intervals, Live after exactly 5 intervals (500 ms).
- f_the_budget_stops_the_probe_only_once_five_intervals_are_seen: with
  fewer than 5 intervals the probe keeps sampling past the budget, and the
  5-budget cap stops it (at 2 s ticks: 3 intervals, all advancing, Frozen).
- sustained_needs_five_advances_80_percent_and_300_ms: each bar at its
  edge.
- probe_with_the_two_step_refresh_is_frozen and
  probe_with_a_sustained_advance_is_live: the same rule through
  probe_with's fakes. The sustained case is Live at 12 intervals, 300 ms.
- F9, the real host_status() on scripted IOReport leaves. The old "two
  advances -> Live at the third sample" script is now a sustained advance:
  Live after 6..=14 samples and >= 300 ms of real wall time. Also added:
  (*) the two-step refresh through host_status() gives CpuRailFrozen.
- the_oracle_does_not_count_a_refresh_as_movement: the two-step refresh,
  a single step and a five-step burst are no movement; 5 of 5, 16 of 20
  and every read are movement; 15 of 20 and 4 of 4 are not.
- Minimums follow the rule. A frozen verdict holds at least 5 ticks, and
  the budget comments say five intervals.

Mutations. Each was a temporary edit, then `git checkout`; the tree was
clean after every one. All four were caught (tests that failed):

- MIN_LIVE_ADVANCES = 2: f_..., l_..., sustained_...
- 80% share check removed: a_decrease_is_not_an_advance (a rail rising
  and falling in turn), i_..., k_..., sustained_...
- 300 ms window check removed: c_..., d_..., i_..., k_...,
  probe_with_a_sustained_advance_is_live, sustained_...,
  host_status_is_the_rule_on_the_production_composition
- oracle back to ">= 2 changes":
  the_oracle_does_not_count_a_refresh_as_movement

Evidence.

- M5 Max, unloaded: eoc-meter has 45 lib tests (37 before) and the
  once-per-process test, all passing.
  `ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
  selected=false (oracle: 7 reads over 3.062004833s with a helper thread
  burning CPU; changes per CPU rail [0], per GPU rail [5], of 6 compared
  pairs)`
- M5 Max, 3 runs of all eoc-meter tests beside a helper process sampling
  every IOReport channel in a tight loop: 3/3 ok, 45 + 1 passed each. The
  oracle read past its 3 s window to reach 5 compared pairs:
  `ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
  selected=false (oracle: 6 reads over 3.664727125s with a helper thread
  burning CPU; changes per CPU rail [0], per GPU rail [4], of 5 compared
  pairs)`, then 11 reads over 3.54 s, then 9 reads over 3.57 s. The
  fresh probe said Frozen every time (12-14 intervals over ~1 s).
- M3 Ultra (10.10.10.2, macOS 27.0.1), 1 run of the lib and
  once-per-process test binaries: 45 + 1 passed.
  `ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
  selected=false (oracle: 18 reads over 3.101584s with a helper thread
  burning CPU; changes per CPU rail [0, 0], per GPU rail [17], of 17
  compared pairs)`
- The whole eoc workspace: 621 passed, 0 failed, 3 ignored. The ignored
  three are the pre-existing ignored doctests.
- clippy --no-deps on eoc-meter, all targets: no new warnings. The only
  two are the pre-existing ones (a duplicated #[link] attribute, and a
  b"...\0" literal). The aarch64-linux cross clippy could not run here:
  blake3's C build finds no Linux sysroot.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 19:00:00 -04:00

jcp-energy: IOReport only when its CPU rail keeps advancing, never on a batch refresh #110

dcharlot referenced this pull request 2026-09-30 19:00:02 -04:00

sandbox-meter: a CPU counter that does not keep moving is never a measurement, and no consumer relabels an estimate #114

dcharlot referenced this pull request 2026-09-30 19:00:05 -04:00

proof-cascade: never meter CPU work through an IOReport CPU rail that does not keep moving #115

dcharlot changed title from WIP: eoc-meter: never build the IOReport counter on a CPU rail that does not keep moving to eoc-meter: never build the IOReport counter on a CPU rail that does not keep moving 2026-09-30 19:00:06 -04:00

dcharlot referenced this pull request 2026-09-30 19:00:09 -04:00

jouleclaw-energy: IOReport only when its CPU rail keeps moving; consumers never zero-fill or relabel #117

dcharlot added 4 commits 2026-09-30 20:35:56 -04:00

eoc-cascade: a cache hit claims no measurement, and a total is measured only if every microjoule was read ... f84f0b2fd0

c6378629 listed two defects here as not done. `JouleCost::zero()` was tagged
`Measured`, and eoc-cache and the cascade's memoizer stamp every cache hit with
it, so each hit claimed a measured 0 µJ although no meter was read for it. The
receipt seals that tag. And `attach_cascade_cost` added the meter's delta over
the whole resolve to the stage's own cost, then tagged the sum `Measured` as
soon as the delta was positive. With a live counter (RAPL, NVML, a live
IOReport rail) a 50 J estimate plus a 500 µJ reading came out as 50,000,500 µJ
measured. A stage that meters itself on the cascade's counter reads a span
nested inside the cascade's own span, so its reading was counted twice: 300 µJ
inside a 500 µJ span came out as 800 µJ.

The rule now, as the owner set it: a live vendor counter's figure is
`Measured`, and nothing that was not read from one may be. So a total is
measured only when every microjoule of it was read from a live counter.

- `JouleCost::zero()` is `Estimated`. Its doc says what it is: a figure no
  counter read, which is what a cache hit costs by assumption. Every caller
  becomes honest at once: eoc-cache, the memoizer, and eoc-route-learned's two
  empty fallback responses, which also claimed a measured zero. No caller
  needed a measured zero; a counter that read but did not move has measured
  nothing. `JouleSource`'s and `JouleCost::measured`'s docs now say what
  measured means.
- `attach_cascade_cost` takes the span only when it is live: both reads
  succeeded and the counter advanced. Otherwise it adds nothing, and the
  stage's cost stands, figure and tag, as before.
- A live span and an estimated stage: the estimate is added, and the total is
  an estimate. The estimate may be energy the counter never saw (a remote
  backend), so it is added, not dropped.
- A live span and a measured stage: the stage read a counter on this host
  during the resolve, so inside the span. The total is the larger of the two
  readings, measured, never their sum. That is exact when the stage read the
  cascade's counter, and a lower bound if it read another; the cost carries no
  counter identity, so the cascade cannot tell.
- A live span and an estimated zero (a cache hit): the zero adds nothing, so
  the total is the span's reading alone and is measured. Every microjoule of
  it was read. Tagging it estimated would under-claim a real reading.

The eoc CLI test's hardware-counter branch asserted the old rule (measured
whenever the counter added anything). It now asserts the new one: the
backend's estimate plus the reading, tagged estimated. That branch does not run
on this Mac, where IOReport declines and the meter is the powermetrics stand-in
("eoc cost on this host: meter=powermetrics cost=50000000 µJ (estimated)").

Tests. The cascade is driven with a scripted counter that one test shares with
a self-metering neural backend, as a host's one counter would be. An estimate
plus a live 1,000 -> 1,500 span is `estimated(50_000_500)`, and its receipt
verifies. A stage that read 1,100 -> 1,400 inside 1,000 -> 1,500 costs
`measured(500)`, not 800. A cache hit inside a live 2,000 -> 2,250 span is
`measured(250)`. A counter that holds, fails its first read, or fails its last
leaves `estimated(42)` alone. Memoized and hand-inserted hits through a stub
cascade are `estimated(0)`. A table test covers every branch of the cost rule
and of the liveness check.

Falsified on the uncommitted tree. Each mutant was restored byte-for-byte and
re-touched so cargo rebuilt from the real source; a diff fingerprint taken
before and after matched:
- `zero()` back to `Measured` fails three tests, in eoc-core, eoc-cache and
  eoc-cascade;
- the cache stamping hits `measured(0)` fails the hand-inserted hit
  (`left: JouleCost { microjoules: 0, source: Measured }`);
- the memoizer stamping `measured(0)` fails the memoized hit;
- tagging estimate + span `Measured` fails with
  `left: JouleCost { microjoules: 50000500, source: Measured }`;
- summing a measured stage onto the span fails with `left: ... 800 ...`,
  `right: ... 500 ...`;
- HEAD's rule, verbatim, fails three of the new tests;
- counting a counter that held (`e >= s`) as live fails two.

eoc-bench, a separate workspace that builds against eoc-core, eoc-cache and
eoc-cascade, passes too.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-carbon: an SCI score is only as measured as the joules it was computed from ... 0db9b8be02

`sci_from_eval_report` wrapped every eval report's energy in
`JouleCost::measured`, whatever produced it. An eval run against the echo
backend, a vendor API's token estimate or a stub-metered cascade came out as a
measured energy term. It could not do better: `EvalReport` dropped the
provenance of the costs it summed, and `SciScore` had nowhere to put one. The
end-to-end test `sci_for_ai` did the same in miniature. Its comment said
"pretend the meter reported 2000 J on the stub backend", and it built that
figure with `JouleCost::measured`.

What holds now:

- `EvalReport` gains `joule_source`. It is `Measured` only when every case's
  cost was read from a live counter. One estimated case makes the total an
  estimate, and so does a run with no cases, since nothing was read. Each case
  is separate work that ran, so an unmeasured case is not covered by any other
  reading. A report serialized before the field existed reads back as
  `Estimated` (`#[serde(default)]`): nothing in it says it was measured.
- `SciScore` gains `energy_source`, the input `JouleCost`'s provenance
  unchanged. It has the same serde default. The numbers do not change.
- `sci_from_eval_report` builds the energy term with the report's own
  `joule_source`.
- `sci_for_ai` uses `JouleCost::estimated` for its fixed 2000 J. No counter
  read that figure. It now asserts that the score says so.

Tests: a measured backend gives a measured report. An estimated one gives an
estimated report, and so does one that turns estimated from its third answer
on. An empty run is estimated. A report and a score without provenance read
back as estimated. A score carries its input's provenance, with the same SCI
either way. Under `eval-bridge`, a report's score keeps the report's
provenance.

Falsified, each mutant restored byte-for-byte:
- HEAD's `JouleCost::measured(total_microjoules)` in the bridge fails
  `an_eval_reports_score_keeps_the_reports_provenance` (`left: Measured`,
  `right: Estimated`). That test is behind the `eval-bridge` feature, which
  nothing in the workspace enables, so a plain `cargo test --workspace` never
  compiles it. It was run with `cargo test -p eoc-carbon --features
  eval-bridge`, 39 + 8 passing.
- `SciScore` ignoring its input's provenance fails two tests, one in the
  default build.
- the runner ignoring per-case provenance fails
  `the_report_is_measured_only_when_every_case_was`.
- `sci_for_ai` tagging its figure measured again fails that test.

Not done: `CarbonAccount` still drops provenance (`account_request` is the
other public way in). And a report with zero correct answers has
`joules_per_correct == u64::MAX` as a sentinel, which `sci_from_eval_report`
multiplies by `correct.max(1)` into a u64::MAX µJ energy term.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-streaming: a stream nothing metered has no energy figure, never 0 µJ ... 6a8d7c68ab

467acb38 made a stream's energy measured only when its counter advanced. It
left this as not done: `StreamAccount` had no way to say "not metered". So a
stream metered by the stub, by a counter whose read failed, or by one that did
not move reported `0 µJ (estimated)`. That is work that ran, presented as a
figure of zero. Saying so honestly needed a change to a public type. Nothing
outside this file uses `StreamAccount` or `StreamMeter`, in this workspace or
any other in the monorepo, so the change is contained.

The fix is the honest one, not a duration-based estimate: the account does not
invent a figure. Any per-second or per-token coefficient would be a number
nobody measured, and callers already hold their own estimates.

- `StreamAccount.microjoules` is now `Option<u64>`: `Some` only when the
  counter read at the start and at the latest checkpoint and advanced between
  them, and `None` otherwise. The separate `measured: bool` is gone, since the
  `Option` says it.
- `joule_cost()` returns `Option<JouleCost>`: `Some(JouleCost::measured(..))`
  for a metered stream and `None` for one nothing metered. There is no
  estimated zero; a caller with no reading supplies its own estimate and tags
  it as one.
- Token counts are kept, metered or not. A failed later read still keeps the
  last reading, and a failed start still never turns a later read into the
  counter's since-boot total.

Tests: a counter that holds gives `{ tokens: 3, microjoules: None }` and no
cost. A counter that advances 450 µJ gives `Some(measured(450))`. A failed
start gives `None`. A stream that advanced 200 µJ keeps that figure when a
later read fails. The stub gives five tokens and no figure, and so does
`StreamAccount::default()`.

Falsified, each mutant restored byte-for-byte:
- recording a zero delta as the stream's figure fails three tests
  (`left: Some(0)`, `right: None`);
- returning HEAD's `0 µJ (estimated)` for an unmetered stream fails three
  (`left: Some(JouleCost { microjoules: 0, source: Estimated })`).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-meter, eoc-local: say what a counter's provenance is, and what each backend reports without one ... 722d80d90e

Docs only. Each statement changed was false against the code.

- eoc-meter's module doc said `StubCounter` "reports its source as
  `JouleSource::Estimated`". `JouleCounter` carries no source; it only reads.
  The doc now says consumers decide: eoc-cascade, eoc-streaming and eoc-local's
  `mlx`, `mlc` and `onnx` call a delta measured only when the counter advanced
  over their span. It also says what each does when nothing was read: no
  figure, the stage's own cost, or an estimate.
- Where the IOReport counter is described (the module doc and
  `MacosIoReportCounter`): a live IOReport reading is a vendor energy counter's
  reading, so its delta is `Measured`, as RAPL's and NVML's are. That holds only
  while the rail is live, which is why the counter is never constructed when it
  is not. The struct's "Measured-grade" tag is replaced by that sentence.
- The powermetrics stand-in's read comment said the error makes "the cascade
  attach `JouleSource::Estimated`". The cascade keeps the stage's own cost,
  whatever its tag; the comment now says no consumer gets a figure from it.
  The rung's doc, that it "never degrades to `StubCounter`", was already
  corrected in 3b827e2a.
- eoc-local's "Joule attribution" said every backend reports the delta as
  measured and that a `StubCounter` yields zero cost. With nothing measured,
  `mlx` falls back to its per-token coefficient; `mlc` and `onnx` report 0 µJ,
  estimated; `llamacpp` reports the delta since its first read, estimated if
  any chunk saw a zero delta. Each is now stated per backend. The crate intro
  said local backends measure; they can, when the host has a live counter.
- mlx's module doc said attribution "uses `powermetrics`". It reads whatever
  counter it is given, and the powermetrics stand-in never reads. The
  `joule_attribution` test's doc said the stub always yields 0, but its own mlx
  test asserts the per-token estimate.

Checked: `cargo doc` raises no new warning on the crates touched, and
`cargo test -p eoc-local --features mlc,gguf` passes. The `mlx` and
`llamacpp` modules need native dependencies (mlx-rs, llama.cpp) that are not
built here, so their doc changes are not compiled. They are comment text and
an intra-doc link to `JouleCounter`, which that module imports.

Not done, found while making these true. Both are behaviour, not docs:
- `llamacpp` takes a failed first read as 0 (`unwrap_or(0)`), so a later
  successful read reports the counter's since-boot total, tagged measured. This
  is the defect 467acb38 fixed in eoc-streaming.
- `mlc` and `onnx` report 0 µJ for inference that ran when nothing was
  measured. It is tagged estimated, but it is still a zero presented as a
  figure.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 3 commits 2026-09-30 21:10:01 -04:00

eoc-local: a span is measured only when both reads succeed and the counter advanced, and inference that ran is never 0 µJ ... 15cf07c2ee

722d80d9 listed two defects here as not done.

`llamacpp` took a failed first counter read as 0 (`unwrap_or(0)`), so the next
successful read reported the counter's whole since-boot total as the
generation's energy. Its drain then tagged that figure measured unless some
chunk had seen a zero delta. This is the defect 467acb38 fixed in
eoc-streaming. And `mlc` and `onnx` reported 0 µJ for inference that ran
whenever nothing was measured: tagged estimated, but still a zero presented as
a figure.

The rule now lives once, in a new `span` module, and every backend meters
through it:

- `MeterSpan::open` reads the counter; `reading` gives `Some(delta)` only when
  that opening read succeeded, the read now succeeds, and the counter advanced
  between them. A failed read at either end, a counter that held, one that
  went down, or the stub: `None`, never 0 and never a since-boot total.
- `span_cost` makes a reading `JouleCost::measured` and asks for the estimate
  only when there was none.
- `per_token_estimate` is tokens × `fallback_joules_per_token`, to the nearest
  µJ, and never 0: at least one token is counted, a coefficient that is not a
  finite positive number gives way to the default (0.05 J/token, mlx's figure
  since the start), and a product under half a µJ is 1 µJ.

Per backend:

- `llamacpp`: `TokenChunk::cumulative_microjoules` is now `Option<u64>`, the
  reading so far, `None` when nothing was measured. The inference's cost is
  the final chunk's reading, measured, or the per-token estimate over the
  tokens streamed. An earlier chunk's reading covers part of the generation,
  so it is never taken as the cost of the whole. The backend gains
  `fallback_joules_per_token` and its builder. Nothing outside this file uses
  `TokenChunk`.
- `mlc` and `onnx` gain `fallback_joules_per_token` (default 0.05 J) and its
  builder, and estimate over the placeholder's own token count, as `mlx` does.
- `mlx` already had the right rule; it now goes through the same functions, so
  a zero coefficient no longer yields 0 µJ.

Factored so it tests without native code. Neither the `llamacpp` nor the `mlx`
module binds a single symbol of its native crate (both generators are
placeholders), so this crate's tests compile them without their features
(`cfg(any(feature, test))`; mlx on macOS). Their metering now runs in every
`cargo test --workspace`, where before it ran in none: llama.cpp is not built
here, and `--features mlx` fails in mlx-sys's Metal kernel build on this
toolchain. The `not(feature = "llamacpp")` error stub, unreachable since the
module itself was gated on the feature, is gone.

Tests, all on a scripted counter that reads a list (`None` is a failed read)
and counts its reads:

- span: an advance of 450 µJ reads 450; a failed first read followed by
  9,000,000 reads nothing; so does a failed last read, a held counter, a
  counter that went down, and the stub. The estimate is 800,000 µJ for 8 × 0.1
  J, 100,000 for 0 tokens at 0.1 J, the default for 0, -1, NaN and infinity,
  1 µJ for 1e-12 J, and saturates at u64::MAX.
- llamacpp, through `stream` and `infer`: the stub gives the 4-token estimate
  (200,000 µJ, estimated) and no chunk a reading; a failed first read then 9 J
  gives no chunk a reading and the estimate; 1,000 -> 1,100 ... 1,450 gives
  chunk readings 100, 200, 300, 400 and a measured 450 over exactly six reads;
  a held counter and a failed last read give the estimate. A table test pins
  that only the final chunk's reading is a cost.
- mlc, onnx, mlx: measured 450 for an advance; the estimate (800,000 µJ) for a
  held counter, a failed first read and a failed last read; the stub's
  estimate, never 0. The `joule_attribution` integration test now asserts the
  mlc stub's estimate (500,000 µJ, estimated) where it asserted 0.

Falsified, each mutant applied to the uncommitted tree, run, and restored
byte-for-byte (sha256 checked):

- a failed start taken as 0 fails 5 tests (llamacpp, mlc, mlx, onnx, span),
  the llamacpp one with every chunk at `Some(9000000)`;
- `>=` in place of `>` (a held counter measured) fails 9;
- llamacpp reading the counter's absolute value per chunk (HEAD's rule with a
  0 baseline) fails 3;
- a partial chunk reading kept as the cost fails the tally test;
- mlc's and onnx's HEAD estimate of 0 fail 2 each;
- the estimate without its 1 µJ floor, without the one-token minimum, or
  without the coefficient check fails the estimate test.
One mutant survived and is equivalent: `final_reading = reading.or(previous)`
cannot differ, since nothing sets the field before the final chunk.

`cargo test -p eoc-local` passes (36 + 1), with `--features mlc,gguf` (47 +
3) and `--features onnx --lib` (43). The onnx leg runs `--lib` because the
`joule_attribution` test binary cannot link ort-sys without ONNX Runtime
installed when mlc is also enabled; that is unchanged. No clippy warning in
this crate, before or after.

Not done: the `llamacpp` error path still returns 0 µJ, estimated, with the
error text. Whether any work ran before an error is unknown there.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-carbon: an account and an SCI score are measured only where every part was, and a run with no correct answer has no score ... 6c37e29980

0db9b8be listed three things here as not done. `CarbonAccount` dropped
provenance, and `account_request` is a public way in. A report with zero
correct answers carries `joules_per_correct == u64::MAX` as a sentinel, which
`sci_from_eval_report` multiplied by `correct.max(1)` into a u64::MAX µJ
energy term (18.4 TJ). And the test that pins the bridge's provenance sat
behind the `eval-bridge` feature, which nothing in the workspace enables, so
`cargo test --workspace` never compiled it, nor any mutation it catches.

What holds now:

- `CarbonAccount` gains `it_energy_source`, the input `JouleCost`'s provenance
  unchanged, and `facility_energy_source`. The facility joules are `it_joules`
  × PUE, so they are measured only when every part of them was: the IT joules
  were measured, and the PUE added no overhead. A PUE above 1 is a published
  or assumed facility ratio applied to the reading, energy no counter on the
  host read; a figure containing any of it is an estimate. A PUE of 1 adds
  nothing, so measured IT joules stay measured, as an estimated zero adds
  nothing to a live span in eoc-cascade. Grid intensity has its own provenance
  (the reading's provider and kind) and does not enter the tag. Both fields
  read back as `Estimated` when absent. Nothing outside this crate builds a
  `CarbonAccount`.
- `SciScore::energy_source` is now the account's facility provenance, because
  the score's energy term is facility energy. Before, it was the input's
  provenance, so measured IT joules at the default PUE of 1.58 gave a
  "measured" term of which 37% no counter read. This changes what 0db9b8be
  set: an SCI score at PUE > 1 is now estimated even from measured joules.
  The SCI numbers do not change.
- The bridge's pure part is compiled in every build. `EvalEnergy`
  (micro-joules per correct answer, correct answers, provenance) and
  `sci_per_correct_answer` hold all the scoring, including the `JouleCost`
  construction 0db9b8be's mutation targeted. `sci_from_eval_report`, still
  behind `eval-bridge`, only copies the report's three fields across.
- A run with no correct answer has no score per correct answer:
  `sci_per_correct_answer` and `sci_from_eval_report` return `None` for
  `correct == 0` and for the `u64::MAX` sentinel, whatever `correct` says
  beside it. The sentinel is never read as energy. `sci_from_eval_report` now
  returns `Option<SciScore>`; its only callers are this crate's tests.

Tests in the default build: an account's (IT, facility) tags are (Measured,
Measured) at PUE 1, (Measured, Estimated) at 1.2, (Estimated, Estimated) from
an estimate at either, and (Measured, Measured) at a PUE of 0.5, which is
clamped to 1; the figures are the same whichever tag. An account round-trips
and an old one reads back estimated. A score is measured at PUE 1, estimated
at 1.2 from measured joules, with the same SCI. An eval run's score keeps the
run's provenance (2 × 1 J × 1.2 = 2.4 J over R = 2), and a run with 0
correct, or the sentinel with 3 correct, has none. `account_per_request` now
asserts its tags. Under `eval-bridge`, a report's score equals its run's score
for both provenances and both PUEs, and a 0-correct report has none.

Falsified, each mutant restored byte-for-byte (sha256 checked); every one but
the last runs in the default build:

- 0db9b8be's mutant, `JouleCost::measured(total)` in the bridge, now fails
  `an_eval_runs_score_keeps_the_runs_provenance` in a plain `cargo test`;
- an account dropping the IT provenance fails 1 test;
- the facility tag ignoring the PUE fails 3, ignoring the IT provenance 3;
- the score taking the input's provenance (0db9b8be's rule) fails 2;
- removing the no-correct-answer guard fails 1, and so does checking
  `correct == 0` without the sentinel;
- the `eval-bridge` wrapper tagging every report measured fails
  `an_eval_reports_score_is_its_runs_score` under `--features eval-bridge`.
That wrapper's three-field copy is the only part a default build cannot see.

`cargo test -p eoc-carbon` passes (42 + 8), and with `--features eval-bridge`
(43 + 8). No clippy warning in this crate, before or after.

Not done: `joules_per_correct` is an integer quotient, so the total
reconstructed from it can fall short of the run's by less than one µJ per
correct answer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-meter: RAPL and NVML are selected only when their counter keeps advancing, as IOReport's CPU rail is ... 2c353d68cd

eoc-meter has both vendor counters: `LinuxRaplCounter` (Linux, package 0's
powercap `energy_uj`) and `NvmlCounter` (feature `cuda`, GPU 0's total-energy
counter). `detect()` selected RAPL whenever the file existed and NVML whenever
it initialised. Neither was held to the rule every IOReport backend in the
family now uses, so a counter that held, or was republished in lumps (a
virtualised powercap tree can do either), was sealed as a reading by every
consumer that tags a live counter's delta `Measured`. And `energy_uj` is
root-only on most current kernels, so on those hosts RAPL was "selected" with
every read failing.

The rule is now one rule for every rung. `rail_liveness`, which held the
IOReport rule, is compiled on every platform and generalised: a `RailWatch`
carries which channels are rails (`default()`: IOReport's CPU energy
aggregates; `every_channel()`: a single vendor counter's one channel), and
`probe_counter` runs the same `watch` loop, stopping rule and thresholds over a
counter that opens. A counter is live only if it advanced in at least five
sample intervals, at least 80% of those observed, over at least 300 ms. A fall
(RAPL's wrap at `max_energy_range_uj`) is no advance.

- `CounterStatus` (`Unavailable`, `Frozen`, `Live`) is public, with
  `rapl_status()` and `nvml_status()`, each probed once per process and
  cached, as `ioreport_status()` is. Off Linux, or without `cuda`, they are
  `Unavailable` without probing.
- RAPL's probe burns 25 ms of the calling thread's own CPU between samples,
  on its CPU clock, so the package counter is watched under load. Where
  `energy_uj` cannot be read it is `Unavailable` without burning anything.
- NVML's probe waits 100 ms between samples instead: burning CPU does not load
  the GPU, and a powered GPU's own draw advances a live counter. That shows
  the counter is published continuously, not that it tracks a load.
- `LinuxRaplCounter::try_detect` and `NvmlCounter::try_detect` go through
  `try_detect_given(status)`, which declines every status but `Live` before
  NVML is touched. `detect_with` takes each rung's status, asked only when the
  ladder reaches that rung.
- The thread CPU clock, the wall clock and the CPU-burning tick moved out of
  `macos_ioreport` into a `clocks` module (macOS and Linux, the crate's second
  scoped `allow(unsafe_code)`, one `clock_gettime`), so RAPL's probe burns
  exactly what IOReport's does.

Behaviour on a Linux host whose `energy_uj` is root-only: `detect()` now
returns the stub, where it returned a RAPL counter whose every read failed.
Consumers got no reading either way. On a host whose RAPL is readable, the
first `detect()` now costs about 300 ms of CPU (a second if the counter is not
live).

Tests. The rule for a single counter runs on scripted samples and a scripted
clock on every platform: advancing every 25 ms is live at 12 intervals, 13
samples (at 100 ms, at 5 intervals); a held counter is frozen after 41
samples; one step, a two-step refresh, and advancing in 2 of every 3 intervals
are frozen, while advancing in 4 of every 5 is live; a wrap is no advance; a
counter that never reads is sampled to the five-budget cap (201 samples) and
is unavailable, and one that does not open is never sampled or ticked; a
vendor watch counts every channel where IOReport's counts only `CPU Energy`.
The RAPL module is compiled for tests on a Mac too, so its own probe runs on
scripted reads (live; held, one step: frozen; EACCES on every read, or one
read then none: unavailable), its sampler is pinned (raw value, saturating at
`i64::MAX`), and only `Live` constructs it. Selection: a frozen or unavailable
RAPL or NVML status never selects that rung, and a live RAPL status selects
it exactly on Linux. Under `cuda`, a declined NVML status never even tries to
initialise NVML (counted per thread), and `Live` tries once. On Linux, a
hardware test checks the cached RAPL status against the counter itself across
200 ms of CPU work, and that `detect()` never picks a RAPL counter that is not
live.

Falsified, each mutant restored byte-for-byte (sha256 checked):
- the gate admitting `Frozen` fails 2 tests;
- a frozen verdict mapped to `Live` fails 3;
- the vendor probe watching only CPU rails fails 4;
- RAPL's constructor ignoring its status fails 1;
- NVML's constructor ignoring its status fails 1 under `cuda` (it survived
  until the open attempt was counted: on a host with no NVIDIA GPU, opening
  fails either way);
- two advances in place of five fails 4.

What ran where. On this M5 Max (macOS 27.0.1): `cargo test -p eoc-meter`
57 + 1 passed, with `--features cuda` 58 + 1, and the whole eoc workspace 665
passed, 0 failed, 3 ignored. For Linux, `cargo check -p eoc-meter --tests
--target x86_64-unknown-linux-gnu`, with and without `cuda`, is clean. The
Linux production probe and the RAPL hardware test were type-checked, not run:
there is no Linux host with readable RAPL here, and no NVIDIA GPU, so neither
rung has been seen live. clippy raises no new warning; the crate's two
existing ones (a manual C-string, a duplicated `link` attribute) predate this.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 21:29:56 -04:00

eoc-meter: a live per-process CPU energy counter for in-process cascades #124

dcharlot added 3 commits 2026-09-30 21:46:19 -04:00

eoc-meter: IOReport is live only when every CPU rail it sums keeps advancing, each die's on its own ... b2e8464c79

The IOReport liveness rule said Live once ANY CPU rail sustained its advance
(five intervals, 80% of them, 300 ms), but the counter it admits SUMS every CPU
rail. On a two-die part (an M3 Ultra's `DIE_0_CPU Energy` and `DIE_1_CPU
Energy`), one live die beside a frozen one would have selected a counter whose
figure leaves the frozen die out, or carries its refreshes, while claiming the
whole CPU, tagged Measured. RAPL and NVML were never exposed to this: each
hands its one counter as the only channel, so "any" and "every" agree there.

`RailWatch::observe` now says Live the first time EVERY rail seen in an
interval passes `sustained`, and at least one does (`all` of no rails is
vacuously true, and a sample with no CPU rail is Absent, never Live). Each rail
is still counted on its own; a rail that joins late is held to every interval
observed since the first, so it is live only once its own advances reach 80% of
all of them. The rule stays one rule: for a single vendor counter, every rail is
its one channel. `CpuRail`, `IoReportStatus` and the crate docs say so.

The hardware oracle is held to the same property. `oracle_moved` required one
CPU rail to change at 80% of its reads; it now requires every CPU rail it found
to, so on a multi-die host the oracle can no longer vouch for a Live verdict that
one die alone earned.

Tests, all pure:

- `m_every_die_must_sustain_its_own_advance`: two dies advancing at every 31 ms
  interval are Live at the tenth (310 ms), exactly where one die is (d); DIE_0
  live with DIE_1 holding, stepping once, or stepping twice in a refresh is
  Frozen over the whole budget and over 1,000 samples; DIE_1 at 60% is Frozen;
  DIE_1 frozen for 20 intervals then live is Live at 80 of 100, never at DIE_0's
  tenth, and Frozen within one probe budget.
- `probe_with_one_die_live_and_one_frozen_is_frozen`: the same through
  `probe_with`, both dies' own values handed to the rule; both live is Live at
  12 intervals, as one die is.
- `host_status_is_the_rule_on_the_production_composition` gains the mixed case:
  `CpuRailFrozen` through the real opener closure, tick and clocks.
- (c) changes meaning: of two same-named `CPU Energy` rails, one live and one
  held was Live and is now Frozen; both live is Live.
- The oracle test now pins `[0, 11]` and `[11, 2]` as not moving and `[11, 9]`
  as moving, and an empty set as not moving.
- Single-die behaviour is unchanged: (a)-(l), the RAPL and NVML tests and the
  composition's single-rail cases pass untouched.

Falsified, each mutant applied to the uncommitted tree, run, and restored
byte-for-byte (sha256 checked):

- `any` in place of `all` (HEAD's rule) fails 4: (m), with "DIE_0 live and
  DIE_1 frozen (holds) counted as live: the counter would leave DIE_1 out ...
  left: Some(Live) right: Some(Frozen)", plus (c), the probe_with mixed test
  and the production-composition test;
- `all` without the non-empty guard fails 2 (`no_cpu_rail_is_absent`,
  `a_rail_new_in_a_sample_only_sets_its_baseline`);
- the oracle back on `any` fails the oracle test.

On hardware, macOS 27.0.1: the M5 Max (one `CPU Energy` rail) and the M3 Ultra
(`DIE_0_`/`DIE_1_CPU Energy`) both report `status=CpuRailFrozen
oracle=CpuRailMoved(false) selected=false`; the M3 Ultra's oracle saw changes
per CPU rail [0, 0] over 19 compared pairs, the GPU rail [19]. eoc-meter's lib
tests pass 59/59 on each.

Not done: on a part whose idle die's rail is live but does not advance while
the probe burns one thread elsewhere, every-die declines the rung. That fails
safe (estimated, never over-claimed) and is not observable on these hosts, where
every CPU rail is frozen to an unprivileged reader.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-agent: a step is charged what the meter measured only when both reads succeed and the counter advanced, never a since-boot total ... 52d9d1e296

`AgentLoop::read_meter` took a failed read as 0 (`unwrap_or(0)`), and
`measured` charged `after - before`. A failed first read followed by a good one
therefore charged the counter's whole since-boot total to one ReAct tool call
or CodeAct execution, which on any real counter exhausts a joule budget at once.
This is the defect 15cf07c2 fixed in eoc-local and 467acb38 in eoc-streaming.

The same rule now holds here:

- `read_meter` returns `Option<u64>`: `None` for a failed read, never 0. Its
  only callers are in this file; nothing else in the monorepo uses it.
- `span_reading(start)` is `Some(delta)` only when the opening read succeeded,
  a read now succeeds, and the counter advanced. Without an opening reading the
  counter is not read again.
- `measured` charges the larger of that delta and the step's own estimate, as
  before, when there is a delta; otherwise the estimate alone.

Tests, on a scripted counter that reads a list (`None` is a failed read) and
counts its reads:

- a failed read is `None`, the next good one `Some`;
- a failed first read then 9 J charges the estimate (250, or 0), with one read;
- 1,000 -> 1,450 charges 450, or the estimate when larger;
- a failed last read, a held counter, one that went down, and the stub charge
  the estimate;
- through `ReActLoop`, with a 1 J budget and a counter that fails once then
  reads 9 J, the agent finishes `Done` having charged its two replies (2,000
  µJ), and the tool step's span carries 0.

Falsified, each mutant applied to the uncommitted tree, run, and restored
byte-for-byte (sha256 checked):

- the failed first read taken as 0 in `measured` fails 2: the unit test with
  `left: (9000000, 2) right: (250, 1)`, and the ReAct test, which stopped
  with `BudgetExhausted("joules")` after the tool step instead of `Done`;
- `read_meter` as at HEAD (`Some(unwrap_or(0))`) fails 3.

`cargo test -p eoc-agent` passes (25 unit, 9 integration); no new clippy
warning (the two in `codeact.rs` and `react.rs`'s parsers predate this).

Not done: ReAct and CodeAct pass an estimate of 0 µJ for a tool step, so an
unmeasured tool step is charged 0 and its span carries 0, and `Span` has no
field to say whether its figure was measured. Giving tool steps an estimate,
and spans a provenance, are separate decisions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-local: a llamacpp generation that errors costs the per-token estimate over the tokens it streamed, never 0 µJ ... 1535b8eb78

15cf07c2 left one path at 0 µJ: when the stream carried an error, `infer`
returned the error text with `JouleCost { microjoules: 0, source: Estimated }`.
The prompt may have been processed and tokens generated before the error, so
that is a zero presented as a figure for inference that may have run.

The drain is now a function of the stream, `drain(rx, id, joules_per_token)`,
which `infer` calls. On an error it returns the same error text, costed at
`span::per_token_estimate` over the tokens streamed before the error — at least
one, as every other unmeasured path — tagged estimated. A reading an earlier
chunk carried covers only part of the generation, so it is not taken as the
cost, as on the normal path.

Test `an_error_costs_the_per_token_estimate_never_zero`, on an injected channel:
three tokens then an error cost 150,000 µJ estimated at 0.05 J/token, with or
without a reading on the earlier chunks, and with the payload
"[llamacpp-error: backend error (llamacpp): decode failed]"; an error before any
token costs one token, 50,000 µJ.

Falsified, each mutant applied to the uncommitted tree, run, and restored
byte-for-byte (sha256 checked): HEAD's 0 µJ fails it ("left: JouleCost {
microjoules: 0, .. } right: 150000"); an estimate that ignores the tokens
streamed fails it (50,000 for 150,000).

`cargo test -p eoc-local` passes (37 + 1), and with `--features mlc,gguf` (48 +
3). No clippy warning in this crate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 2 commits 2026-09-30 22:11:57 -04:00

eoc-meter: RAPL is the sum of every package zone, each of which must be live and each wrapped at its own range ... 26165a77ec

The RAPL rung read one file, `/sys/class/powercap/intel-rapl:0/energy_uj`, and
its liveness probe watched that one file. On a multi-socket server that figure
is package 0's energy alone while the counter claims the CPU, so every other
socket's energy was missing from every span it measured. And nothing makes zone
0 a package: `intel-rapl:<n>` is only an identifier, and each zone says what it
is in its `name` file. A host whose zone 0 is the platform zone (`psys`) or a
`dram` zone had that domain read and probed as the package. proof-cascade does
this right since 9334ec0c; this follows it.

- The package set is listed from the powercap directory: every top-level
  `intel-rapl:<n>` zone whose `name` reads `package-<digits>`. Left out wherever
  they sit, zone 0 included: `psys` (the platform zone, whose energy includes
  the packages', so summing it beside them counts them twice), `dram`, the
  `core`/`uncore`/`dram` sub-zones (`intel-rapl:0:0`), the control type
  (`intel-rapl`), and the MMIO interface (`intel-rapl-mmio:0`, also named
  `package-0`, which republishes a package). The listing reads the directory
  rather than trying fixed indices, and takes the root path as an argument, so
  the tests run it on fake trees.
- Liveness: each package counter is its own rail in the sample, named for its
  zone, and the existing every-rail rule (the one IOReport's dies are held to)
  makes RAPL live only when EVERY package keeps advancing. A host with no
  package zone (only `psys`) is `Unavailable` with nothing read or burned, so
  no counter is constructed.
- The probe's verdict is cached together with the package set it watched
  (`linux_rapl::host()`, one `OnceLock`). `rapl_status()` and the counter's
  constructor both read that cache, so the counter sums exactly the packages
  that were proved live.
- The counter returns a running total: the first read's package counters
  summed, then each package's delta added, each taken modulo its own
  `max_energy_range_uj`. Summing the raw counters would make one package's
  wrap a fall of the sum, and with sockets at different ranges no single range
  is right. Before this, a wrap of the one zone was a fall, which callers take
  as no reading; now a wrap is read across. The files are read under the
  counter's lock, so two threads' reads are folded in the order they were
  taken, and an older read is never taken for a wrap. A package that cannot be
  read fails the read: no figure that leaves a package out. A package whose
  counter falls while its range is unknown (absent, or 0) fails that read and
  every later one, because the total would be short by energy no one can know.

Tests, on fake powercap directories under the temp dir (removed when done), so
they run on macOS:
- a two-socket server (the control type, package-0 and package-1 with core and
  dram sub-zones, psys, a top-level dram, intel-rapl-mmio:0) lists packages 0
  and 1 only, and with every zone moving, the counter's delta is the two
  packages' energy (250,000 + 400,000 uJ) and nothing else;
- zone 0 named psys beside package zone 1: zone 1 alone is listed, sampled
  and summed;
- a psys zone 0 moving at every tick beside a held package: Frozen;
- both packages advancing: Live; package 1 held while package 0 and psys move:
  Frozen; scripted, either package held: Frozen;
- per-zone wrap: package 0 wraps at 262,143,328,850 while package 1 advances,
  then package 1 wraps at 65,532,610,987: deltas of 1,700 and 2,500 uJ; taken
  at the other zone's range, package 1's wrap would be off by the difference
  of the two ranges;
- a package with no range, absent or 0, is summed while it rises; its fall
  fails that read and the next;
- an unreadable package fails the read;
- an only-psys host, an empty directory and a missing one: Unavailable, no
  tick, no counter; a counter forced over no package never reads;
- the name and directory rules: `package-<digits>`, `intel-rapl:<digits>`.
On Linux, the hardware test reads every package in the cached set and, when it
is Live, requires each one to move across 200 ms of CPU work.

Falsified, each mutant applied to this commit, run, and restored with
`git checkout` (blob hash equal to HEAD's):
- back to `intel-rapl:0` only, whatever its name: 8 of the 12 RAPL tests fail;
- every top-level zone summed (no name check): 5 fail;
- every package wrapped at package 0's range: 1 fails, the per-zone wrap test;
- the liveness sample holding the first package only: 3 fail.

What ran where. On this M5 Max (macOS 27.0.1): `cargo test -p eoc-meter` 68 + 1
passed, with `--features cuda` 69 + 1; `cargo test --workspace` (with the
eoc-agent commit that follows) 686 passed, 0 failed, 3 ignored. For
x86_64-unknown-linux-gnu, `cargo check --all-targets` and `cargo clippy
--all-targets` over every workspace member except the five that link
aws-lc-sys (eoc-embeddings, -multimodal, -rerank, -tools, -vendor-api, whose
build script needs `x86_64-linux-gnu-gcc`, not installed here) pass, with no
warning in eoc-meter, with or without `cuda`. The sysfs reader, the host cache
and the hardware test are type-checked, not run: there is no Linux host with
readable RAPL here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-agent: a tool step the meter did not measure is charged its duration at a stated power, never 0 µJ ... 371fbe5eaa

ReAct and CodeAct passed `measured` an estimate of 0 µJ for a tool step. Where
the meter measured nothing (the `StubCounter` that `AgentLoop::new` builds every
loop with, a failed read, a counter that held), the step was charged 0 and its
span carried 0, for a tool or a block of code that ran. 52d9d1e2 left this open.

This crate has no energy model for a tool, and what a model step is charged is
the provider's own figure (`LlmReply::microjoules`), which a tool has no
counterpart of. So a tool step's estimate is time by power:

- `duration_estimate(elapsed, watts)`: elapsed times watts, in micro-joules,
  rounded. Never 0, because the step ran: under half a micro-joule is 1 µJ.
  A power that is not finite and positive gives way to the default.
- `DEFAULT_TOOL_WATTS` = 15 W, about one busy core: a reference figure, said to
  be one, to be replaced by one calibrated on the host, as eoc-local's
  `DEFAULT_FALLBACK_JOULES_PER_TOKEN` is.
- `AgentLoop::measured_with(estimate, f)` times the step on the monotonic clock
  and makes the estimate from its duration once it has run. `measured` is now
  `measured_with` with a fixed estimate, its behaviour unchanged, and
  `measured_tool(f)` is `measured_with` at the default power.
- ReAct and CodeAct charge tool steps through `measured_tool`, under the rule
  `measured` already had: the meter's reading when both reads succeed and the
  counter advanced, or the estimate if that is larger; otherwise the estimate.
  A tool that fails ran too, and is charged for it.

Tests:
- the estimate: 2 s at 15 W is 30,000,000 µJ, 20 ms is 300,000, 10 µs at
  2.5 W is 25; zero and 1 ns are 1 µJ; zero, negative, NaN and infinite power
  fall back to 15 W; `Duration::MAX` saturates;
- `measured_tool` over a step of at least 20 ms charges at least 300,000 µJ
  with the stub, with a failed first read, and with a held counter (and less
  than the 9 J since-boot total), and charges a 50 J reading when the counter
  advanced by that;
- ReAct with the stub meter and a tool of at least 20 ms: the act span carries
  at least 300,000 µJ, and the budget the two replies plus that;
- CodeAct with the stub meter and a runtime of at least 20 ms, once succeeding
  and once failing: the same;
- the existing failed-first-read ReAct test now expects the tool step's
  non-zero estimate on the span and in the budget (2,000 plus the span), still
  far below the 9 J since-boot total. Its budget is 5 J instead of 1 J, so a
  slow scheduler cannot push a microsecond echo's estimate past it (1 J is
  67 ms at 15 W).

Falsified, each mutant applied to this commit, run, and restored with
`git checkout`:
- `measured_tool`'s estimate back to 0: 4 tests fail;
- ReAct back to `measured(0, ..)`: 2 fail;
- CodeAct back to `measured(0, ..)`: 1 fails.

`cargo test -p eoc-agent`: 29 unit and 9 integration tests pass, and the eoc
workspace 686 passed, 0 failed, 3 ignored, on an M5 Max (macOS 27.0.1).
`cargo check` and `cargo clippy --all-targets` for x86_64-unknown-linux-gnu
pass; clippy's two collapsible-if warnings in the ReAct and CodeAct parsers
predate this.

Not done: Plan-and-Execute charges each worker step `charge_tool(0)` without
metering it, the same defect in a third loop; and `Span` still has no field to
say whether its figure was measured or estimated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 2 commits 2026-09-30 22:38:57 -04:00

eoc-agent: a Plan-and-Execute step is metered like a tool step, never charged 0 µJ ... 6cf8b60e4b

Plan-and-Execute charged each worker step `charge_tool(0)` and put no figure on
its span, and its plan step was charged nothing at all, under a comment saying
it was charged a notional cost. A planner or worker that ran was 0 µJ in the
trace and against the budget. 371fbe5e fixed this for ReAct and CodeAct and
left it open here.

Neither a `Planner` nor a `Worker` reports what its step cost: the traits
return subgoals and text, and `LlmPlanner` and `LlmWorker` drop the reply's
figure. So each step is metered in the loop, through `measured_tool`, under
the rule ReAct and CodeAct now follow: the meter's reading when both reads
succeed and the counter advanced, or the step's duration at
`DEFAULT_TOOL_WATTS` if that is larger; otherwise that duration estimate,
which is never 0.

- The plan step's span carries its figure, and the figure is charged against
  the joule cap. It is not a tool call, so it takes none of the tool-call
  quota. A plan step that spends the cap stops the loop before any worker runs.
- Each worker step's span carries its figure, charged with `charge_tool`.
- A step that fails ran too. Before, a planner or worker error was returned at
  once, unrecorded and uncharged; now the span records `ERROR: ...` with the
  step's figure, the budget is charged, and then the error is returned.
- The comment claiming planners self-meter through `charge_llm_step` is gone:
  a `Planner` has no access to the loop and cannot.

Tests:
- with the stub meter, a planner and two workers that each take at least
  20 ms: every span carries at least 300,000 µJ, the budget is charged
  exactly the trace's total, and two tool calls;
- with a scripted meter: 40 J over the plan step and 50 J over worker `b` are
  charged as read, and worker `a`, over which the counter held, its own
  estimate (above 0, below 1 J for an echo); the budget is 90 J plus that;
- a worker that fails after 20 ms: the error is returned, its span says
  `ERROR:` and carries at least 300,000 µJ, the budget is charged that and one
  tool call;
- a 1 µJ joule cap stops the loop after the plan step, before any worker.

Falsified, each mutant applied to this commit, run, and restored with
`git checkout`:
- a worker step charged `charge_tool(0)` again, its span keeping the figure:
  3 tests fail;
- a worker step's figure 0 on the span and in the charge: 3 fail;
- the plan step's figure 0: 3 fail;
- the plan step not charged against the budget: 4 fail;
- a failed worker step returned before it is recorded and charged: 1 fails.

`cargo test -p eoc-agent`: 33 unit and 9 integration tests pass. clippy on
eoc-agent adds no warning; its two collapsible-if warnings in the ReAct and
CodeAct parsers predate this.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

eoc-agent: every span says whether its joule figure was measured or estimated ... 1352aeea07

A trace `Span` carried a micro-joule figure and nothing about where it came
from. A provider's own figure, a tool step's duration estimate, a zero for an
observation, and a delta a live counter read all looked the same, so nothing
reading a trace could tell a measurement from an estimate. Every other eoc
surface that reports energy says which it is (`JouleCost`, eoc-eval's
`joule_source`); an agent trace did not.

`Span` gains `joule_source: JouleSource`, eoc-core's tag, under eoc-core's
rule: `Measured` only for what a live counter read over the step — both reads
succeeded and the counter advanced — and `Estimated` for anything else.

- The field is additive on the wire. It is `#[serde(default)]`, as eoc-eval's
  report field is: a trace written before it reads back with every span
  `Estimated`, since nothing in it says it was measured. Nothing in the
  monorepo outside eoc-agent uses eoc-agent, so no struct literal of `Span`
  breaks.
- `Span::new` is `Estimated` (0 µJ that no counter read). `with_joule_cost`
  attaches a figure and its tag together, and `joule_cost()` reads them back.
  `with_microjoules` attaches a bare figure, which is an estimate: it no
  longer leaves an earlier `Measured` tag on a figure the counter did not read.
- `AgentLoop::measured_with` and `measured_tool` return a `JouleCost` instead
  of a bare `u64`. The figure is unchanged — the reading, or the estimate when
  larger, or the estimate alone — and it is `Measured` exactly when it is the
  reading: a reading smaller than the estimate is not what is charged, so the
  estimate charged in its place is `Estimated`. `measured` still returns the
  bare figure, as on main. Both changed functions are new on this branch.
- Model steps: `charge_llm_step` tags the provider's own figure `Estimated`;
  `LlmProvider`'s contract calls it an estimate and no counter here read it.
  This covers every loop's model steps (ReAct, CodeAct, Reflexion, ToT, LATS).
- Tool steps: ReAct and CodeAct put `measured_tool`'s tag on the act span, and
  Plan-and-Execute on its plan and execute spans.
- Other steps: ReAct's observation of an unknown tool is `Span::new`'s
  estimated 0, as nothing ran.

Tests:
- a new span and a bare figure are estimates; a measured cost survives
  `with_joule_cost`, and a bare figure attached after it is an estimate;
- a trace round-trips through JSON with a `Measured` and an `Estimated` span,
  and a trace in the old format, without the field, reads back all
  `Estimated`;
- `measured_tool`: the stub, a failed first or last read, a counter that held
  or went down, and one that advanced 10 µJ over a 20 ms step are all
  `Estimated` and charged at least the 300,000 µJ estimate; a 50 J reading is
  `Measured`. `measured_with` at a fixed estimate: a 450 µJ reading is
  `Measured` at estimates of 100 and 450, and an estimate of 1,000 is charged
  as `Estimated`;
- `charge_llm_step` records the reply's figure as `Estimated`;
- ReAct, with a counter that advances 50 J over the tool step: think spans
  `Estimated` at 1,000 µJ, the unknown tool's observation `Estimated` at 0,
  the act span `Measured` at 50 J, and the budget 3,000 µJ plus 50 J; with
  the stub, every span `Estimated`;
- CodeAct, the same over code that succeeds and code that fails;
- Plan-and-Execute: with the stub every span `Estimated`; with the scripted
  meter the plan step (40 J) and worker `b` (50 J) `Measured`, and worker `a`,
  over which the counter held, `Estimated`.

Falsified, each mutant applied to this commit, run, and restored with
`git checkout`:
- every span pushed to a trace tagged `Measured`: 8 tests fail;
- `Span::new` and `with_joule_cost` always `Measured`: 9 fail;
- every span pushed to a trace tagged `Estimated`: 4 fail;
- a span without the field reading back `Measured`: 1 fails;
- `measured_with` tagging any reading `Measured`, including when the
  estimate is what is charged: 1 fails;
- `charge_llm_step` tagging the provider's figure `Measured`: 5 fail;
- `with_microjoules` keeping an earlier tag: 1 fails;
- ReAct, CodeAct, and Plan-and-Execute each dropping the tag
  (`with_microjoules(cost.microjoules)`): 1 fails in each.

`cargo test -p eoc-agent`: 38 unit and 9 integration tests pass.
The eoc workspace (`cargo test --workspace`): 695 passed, 0 failed,
3 ignored, on an M5 Max (macOS 27.0.1). clippy for x86_64-unknown-linux-gnu
on eoc-agent: the same two pre-existing warnings and no others.
On macOS too clippy adds no warning (the two collapsible-if warnings predate
this), and rustdoc reports nothing in the lines changed here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot merged commit 997471e045 into main 2026-10-01 05:50:58 -04:00

dcharlot referenced this pull request from a commit 2026-10-01 05:51:00 -04:00

Merge pull request 'eoc-meter: never build the IOReport counter on a CPU rail that does not keep moving' (#116) from eoc-meter/cpu-rail-liveness into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!116

No description provided.