sandbox-meter: a CPU counter that does not keep moving is never a measurement, and no consumer relabels an estimate #114

Merged

dcharlot merged 20 commits from sandbox-meter/cpu-rail-liveness into main 2026-10-01 05:45:12 -04:00

Conversation 0 Commits 20 Files changed 43 +7875 -526

dcharlot commented 2026-09-30 14:31:46 -04:00

Owner

Copy link

sandbox-meter's IOReport probe sealed a CPU counter that never moved as a hardware measurement. Four consumers then relabelled whatever they were given.

What was wrong: on main, measured on both chips

• joule-meter, 1 s of all-core burn:
  • M5 Max: Cpu: 0 µJ [HwShunt] · Npu: 0 µJ [HwShunt] · Gpu: 226790 µJ [HwShunt]
  • M3 Ultra: three 0 µJ [HwShunt]
• Span::stop sealed every class delta, 0 included, as measured_uj: Some(delta) + HwShunt. That breaks the module's own promise: "a probe can degrade but it can never lie … not a zero dressed as a measurement".
• run_attributed stayed attributed = true with soc_cpu_uj == 0.
• jcp-gateway sealed every call as Energy::measured (HwShunt). Its "metered and debited" tests asserted only inside if metering_is_hardware(). They passed while the GPU drew, and on the M3 Ultra they failed (3/3 and 2/3). This is the root cause of the jcp-gateway failures seen on main.
• joule-code-remediate signed the attributed figure as ModelBased whether or not it was attributed; on these hosts that is an apportioned 0 J for real cargo work. Its end-to-end test failed on main every time (report.total_uj() > 0).
• joule-ui-serve sealed each served interaction as 0 µJ.

What this does

1. Selection. IoReportProbe::new() declines unless ioreport_status() is Live, so PowerMeter falls back to its estimator: wall × TDP, Estimator, no measured_uj.
2. Per span. A span whose CPU-class counter did not advance is not a measurement. The whole span becomes the estimator record. The same holds for a RAPL package counter that held; a GPU-only (NVML) probe set is unaffected.
3. Attribution. attributed is true only when the window's CPU record is a HwShunt reading > 0.
4. Consumers take their label from the span they were given, never from is_hardware(), a trait default or a hard-coded grade:
   • jcp-gateway: energy_of() uses jcp-schema's existing constructors: estimator, measured, measured_partial for a CPU-less hardware reading, and modeled. No wire fields are added.
   • joule-code-remediate: a run is ModelBased only if every cargo window was attributed. Otherwise the receipts carry the unapportioned total as an estimate (JCR-1 Estimator; JouleContract Unaccounted, with joules_spent set to that figure). It is never an apportioned 0.
   • joule-ui-serve: SpanMeter seals each interaction with its own span's figure and grade. The joule-ui-interaction examples get the same fix, and the Meter trait is unchanged.
   • sandbox-proof says "measured" only when every record is a reading.
   • meter_cmd no longer prints a floored 0 as "the workload's honest share".
5. A pre-existing leak is fixed: IoReportProbe::open() never released desired.

After, on both chips: spans seal Cpu: ~20,000,000 µJ/s [Estimator] and state the reason.

• jcp-gateway prints allowed, 2309714 µJ [Estimator] where it used to print 1541728 µJ measured.
• joule-fix prints 44.352 J estimated, not attributed (Unaccounted) where it used to print 1.446 J (attributed).

Verification of this PR (head d35c99af)

• Mutations: the round-6 run caught 20 of 20. Restoring the old two-advance rule fails 17 tests.
• Robustness: 0 of 6 runs failed under all-channel contention. In one run the median wall time per sample call was 58.7 ms, which the old wall-clock pin would have failed; its thread CPU was 8.4 ms.
• M3 Ultra: passes.
• Consumer suites: jcp-gateway + jcp-action-receipt, joule-code-remediate (with and without a git identity), joule-ui-interaction + joule-ui-serve, and the rest of sandbox-rs are green.
• Pre-existing defect fixed on the way (41d80434): joule-code-remediate discarded the results of git checkout -b, git add and git commit. It reported shipped: true for fixes that were never committed, which happens in any environment without a git identity, including the CI container. It could also commit onto the wrong branch. A failed ship step now returns HarnessError::Ship, saying where the verified change was left. (That commit's message has two errors, corrected in 4b20ce2a: the two-run test came from cee66773, and the falsification fails five tests.)

Later rounds on this branch

• f2f23c2d, 7609bfea, 1b7a6169 and ccbec44a:
  • jcp-gateway puts the grade in plaintext beside measured_uj (energy_provenance, plus energy_coverage when partial).
  • joule-ui:
    • Meter defaults to Estimator; the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical.
    • Authorizer::charge carries the full figure.
    • Work that ran with no reading gets a never-zero estimate, so a budget binds on a host with no counter.
  • remediate's z3 tests assert in both branches; JOULE_FIX_EXPECT_Z3=1 makes a missing solver a failure.
• 6b693aa4 and 7539c62e: RAPL and NVML are gated, and every summed rail must be live. joule-ui seals a GPU-only span as an Estimator figure for the interaction, because it has no coverage field.
• 5e945eed: IOReport is live only if every CPU rail it sums keeps advancing.
• 7cf1f036: RAPL sums only the package-N zones, never psys or dram. Each zone wraps at its own range; an unknown range on a fall gives no figure and breaks the running count.
• 8a3465e1: sandbox-exec-microvm's RAPL meter reuses the gated package reader. The MSR meter reads every package and die, and declines on hosts it can't fully see.
• a16e36cf: remediate's tests give each scratch crate its own target directory, so they pass with or without CARGO_TARGET_DIR. Production joule-fix still honours it.
• 3d70348a: a microVM run whose CPU counter held is a model estimate, never a measured zero.

Why: what IOReport actually does on macOS 27.0.1

Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1:

• CPU rails are republished in rare lumps. For an unprivileged reader, CPU Energy and DIE_<n>_CPU Energy do not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each.
• A lump can arrive in steps. A watcher sampling every ~16 ms caught one as +61,764,485 mJ, then +3,435 mJ 64 ms later.
• GPU Energy is live on the M5 Max (it changes every ~28 ms) and sporadic on the M3 Ultra.

A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it.

The rule: identical in all five backends (#110, #114, #115, #116, #117)

• The probe opens its own subscription to the "Energy Model" group. A sample costs about 6–12 ms of thread CPU, against about 115 ms for all channels.
• It decides per CPU rail (CPU Energy, or each DIE_<n>_CPU Energy), keyed by name and ordinal:
  • it compares raw integers from consecutive samples;
  • a decrease, or a rail's first appearance, is not an advance;
  • a rail advances at most once per interval.
• Live means every rail the meter sums kept advancing: in at least 5 intervals, in at least 4 of every 5 intervals so far, over a window of at least 300 ms.
  • A live rail meets that in about 300 ms. A lump does not, even one published in several steps.
  • Every rail is required because the meter sums them. One live die beside a frozen die would otherwise give a figure that silently misses the frozen die while claiming the whole CPU.
• The same rule gates RAPL and NVML.
  • RAPL sums the top-level zones whose name reads package-N, on every socket, and each must be live. It never sums psys, which already includes the packages, nor dram or the sub-zones.
  • Each zone wraps at its own max_energy_range_uj. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading.
  • An NVML (GPU-only) figure is Partial where the format can say so. Where it can't, it is never a hardware reading of the whole task.
• It samples until Live, or until the 1 s budget has elapsed and at least 5 intervals have been seen, with a cap of 5 budgets.
  • Each tick burns 25 ms of this thread's CPU time, not wall time, with a 500 ms wall cap. So the probe keeps working on a loaded host.
• The result is cached once per process. Anything but Live declines the backend, so the crate's existing honest fallback runs. The reason is public: IoReportStatus, ioreport_status(), rapl_status(), nvml_status().
• The production status is one branch-free composition, counted each time it runs. Every selection path reads it only through the cached status and a pure admits(status) (Live only).
• A live vendor counter's reading is a hardware reading: HwShunt, or Measured in eoc. That follows the owner's position that vendor counters are real readings and the trust gap is provenance. It holds only while the counter is live, per the MUST in #119.

How it was verified

Six rounds of implementation, each checked by independent adversarial verification.

Pure tests on scripted samples and a scripted clock. They cover:

• the real refreshes (one step, two steps, 3–5 steps, the two-die shape);
• decreases, first appearances and failed samples;
• live rails (every interval, 80% of intervals, on a loaded host);
• the sampling-loop bounds.

Fault hooks inside the real IOReport opener and sampler. The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test.

Hardware tests against an independent oracle. The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check:

• the probe's sampled values equal the oracle's reads on either side;
• the live GPU rail shows up in the probe's record, which proves the reads are fresh;
• every probe sample is group-only, by channel count;
• thread CPU per sample and per tick is checked against the tests' own clock;
• the probe runs exactly once per process, checked in a fresh process.

Mutation-tested. Each of the following was applied as a temporary edit and failed a test:

• restore the old one- or two-advance rule;
• compare every sample with the first;
• treat a decrease as an advance;
• map "no subscription" to Live inside the production status;
• use a no-op tick, a tick that sleeps, or a tick on the wrong clock;
• use a per-thread or non-atomic cache;
• probe uncached from the constructor;
• a sampler that returns constant, stale or summed values;
• an all-channel subscription;
• decide on the summed total or on the GPU rail;
• an admits() that accepts CpuRailAbsent.

The counts per crate are below.

Robustness. Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra.

CI: these tests are behind cfg(target_os = "macos"), and this workspace runs on a linux runner. They reach CI through #102, which merges last. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not print ioreport liveness:.

Merge order: #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them.

Open questions, now decided and implemented (per "fix it all")

• One grade for a live vendor counter. It is HwShunt, or Measured in eoc, everywhere: a vendor counter is a real reading, and the gap is provenance. It holds only while the counter is live (the MUST in #119).
• The GPU rail as a partial reading of CPU work: no. The GPU rail is machine-wide (display compositing moves it). Booking it to a task would misattribute energy and under-debit budgets.
• Wire keys named measured_uj that carry estimates. Their grade now sits beside them in plaintext: jcp-gateway's _meta gets energy_provenance, plus energy_coverage when partial. The keys stay, for compatibility.
• eoc cache hits are an estimated zero, never Measured.
• joule-ui. Meter defaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. Authorizer::charge carries the figure's provenance.
• A live CPU energy source on macOS 27 exists. It is the kernel's per-process counter ri_energy_nj: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privileged powermetrics is live too, but machine-wide and root-only, so it is not used.)

Still open

• The joule-ui demo on prod. It has run on macOS 27 since 2026-09-15. Its sealed chain was not inspected, because the read was blocked as a production read. After #114 merges, redeploying joule-ui-serve gives it honest grades, and the per-process probe in #121 gives it real readings.
• Linux RAPL and NVML have never run against real hardware. No Linux host with readable RAPL and no NVIDIA GPU were available. They are covered by fake-tree and scripted tests, type-checked for Linux, and declined fail-safe.

🤖 Generated with Claude Code

sandbox-meter's IOReport probe sealed a CPU counter that never moved as a hardware measurement. Four consumers then relabelled whatever they were given. ### What was wrong: on main, measured on both chips - **`joule-meter`, 1 s of all-core burn:** - M5 Max: `Cpu: 0 µJ [HwShunt] · Npu: 0 µJ [HwShunt] · Gpu: 226790 µJ [HwShunt]` - M3 Ultra: three `0 µJ [HwShunt]` - **`Span::stop` sealed every class delta, 0 included, as `measured_uj: Some(delta)` + HwShunt.** That breaks the module's own promise: *"a probe can degrade but it can never lie … not a zero dressed as a measurement"*. - **`run_attributed` stayed `attributed = true` with `soc_cpu_uj == 0`.** - **jcp-gateway** sealed every call as `Energy::measured` (HwShunt). Its "metered and debited" tests asserted only inside `if metering_is_hardware()`. They passed while the GPU drew, and on the M3 Ultra they failed (3/3 and 2/3). **This is the root cause of the jcp-gateway failures seen on main.** - **joule-code-remediate** signed the attributed figure as ModelBased whether or not it was attributed; on these hosts that is an apportioned 0 J for real cargo work. Its end-to-end test failed on main every time (`report.total_uj() > 0`). - **joule-ui-serve** sealed each served interaction as `0 µJ`. ### What this does 1. **Selection.** `IoReportProbe::new()` declines unless `ioreport_status()` is Live, so `PowerMeter` falls back to its estimator: wall × TDP, `Estimator`, no `measured_uj`. 2. **Per span.** A span whose CPU-class counter did not advance is not a measurement. The whole span becomes the estimator record. The same holds for a RAPL package counter that held; a GPU-only (NVML) probe set is unaffected. 3. **Attribution.** `attributed` is true only when the window's CPU record is a HwShunt reading > 0. 4. **Consumers take their label from the span they were given**, never from `is_hardware()`, a trait default or a hard-coded grade: - **jcp-gateway:** `energy_of()` uses jcp-schema's existing constructors: `estimator`, `measured`, `measured_partial` for a CPU-less hardware reading, and `modeled`. No wire fields are added. - **joule-code-remediate:** a run is ModelBased only if *every* cargo window was attributed. Otherwise the receipts carry the unapportioned total as an estimate (JCR-1 `Estimator`; JouleContract `Unaccounted`, with `joules_spent` set to that figure). It is never an apportioned 0. - **joule-ui-serve:** `SpanMeter` seals each interaction with its own span's figure and grade. The `joule-ui-interaction` examples get the same fix, and the `Meter` trait is unchanged. - **sandbox-proof** says "measured" only when every record is a reading. - **`meter_cmd`** no longer prints a floored 0 as "the workload's honest share". 5. **A pre-existing leak is fixed:** `IoReportProbe::open()` never released `desired`. **After, on both chips:** spans seal `Cpu: ~20,000,000 µJ/s [Estimator]` and state the reason. - jcp-gateway prints `allowed, 2309714 µJ [Estimator]` where it used to print `1541728 µJ measured`. - joule-fix prints `44.352 J estimated, not attributed (Unaccounted)` where it used to print `1.446 J (attributed)`. ### Verification of this PR (head `d35c99af`) - **Mutations:** the round-6 run caught 20 of 20. Restoring the old two-advance rule fails 17 tests. - **Robustness:** 0 of 6 runs failed under all-channel contention. In one run the median wall time per sample call was 58.7 ms, which the old wall-clock pin would have failed; its thread CPU was 8.4 ms. - **M3 Ultra:** passes. - **Consumer suites:** jcp-gateway + jcp-action-receipt, joule-code-remediate (with and without a git identity), joule-ui-interaction + joule-ui-serve, and the rest of sandbox-rs are green. - **Pre-existing defect fixed on the way (`41d80434`):** joule-code-remediate discarded the results of `git checkout -b`, `git add` and `git commit`. It reported `shipped: true` for fixes that were never committed, which happens in any environment without a git identity, including the CI container. It could also commit onto the wrong branch. A failed ship step now returns `HarnessError::Ship`, saying where the verified change was left. (That commit's message has two errors, corrected in `4b20ce2a`: the two-run test came from `cee66773`, and the falsification fails five tests.) ### Later rounds on this branch - `f2f23c2d`, `7609bfea`, `1b7a6169` and `ccbec44a`: - **jcp-gateway** puts the grade in plaintext beside `measured_uj` (`energy_provenance`, plus `energy_coverage` when partial). - **joule-ui:** - `Meter` defaults to Estimator; the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. - `Authorizer::charge` carries the full figure. - Work that ran with no reading gets a never-zero estimate, so a budget binds on a host with no counter. - **remediate's z3 tests** assert in both branches; `JOULE_FIX_EXPECT_Z3=1` makes a missing solver a failure. - `6b693aa4` and `7539c62e`: RAPL and NVML are gated, and every summed rail must be live. joule-ui seals a GPU-only span as an Estimator figure for the interaction, because it has no coverage field. - `5e945eed`: IOReport is live only if every CPU rail it sums keeps advancing. - `7cf1f036`: RAPL sums only the `package-N` zones, never psys or dram. Each zone wraps at its own range; an unknown range on a fall gives no figure and breaks the running count. - `8a3465e1`: sandbox-exec-microvm's RAPL meter reuses the gated package reader. The MSR meter reads every package and die, and declines on hosts it can't fully see. - `a16e36cf`: remediate's tests give each scratch crate its own target directory, so they pass with or without `CARGO_TARGET_DIR`. Production `joule-fix` still honours it. - `3d70348a`: a microVM run whose CPU counter held is a model estimate, never a measured zero. ### Why: what IOReport actually does on macOS 27.0.1 Measured by reading channels directly on the M5 Max (dev) and the M3 Ultra (10.10.10.2), both on macOS 27.0.1: - **CPU rails are republished in rare lumps.** For an unprivileged reader, `CPU Energy` and `DIE_<n>_CPU Energy` do not move under load for seconds to minutes. On the M3 Ultra both dies held for at least 22 minutes, then jumped about 1.3 kJ each. - **A lump can arrive in steps.** A watcher sampling every ~16 ms caught one as `+61,764,485 mJ`, then `+3,435 mJ` 64 ms later. - **GPU Energy is live on the M5 Max** (it changes every ~28 ms) and sporadic on the M3 Ultra. A meter that sums rails therefore reports CPU work as GPU-only energy (M5 Max) or 0 (M3 Ultra), labelled as a reading. A liveness check that accepts one advance, or two, gives a false Live whenever its window holds a refresh. After that, each later ~1.3 kJ lump is booked whole to whichever span straddles it. ### The rule: identical in all five backends (#110, #114, #115, #116, #117) - **The probe opens its own subscription to the "Energy Model" group.** A sample costs about 6–12 ms of thread CPU, against about 115 ms for all channels. - **It decides per CPU rail** (`CPU Energy`, or each `DIE_<n>_CPU Energy`), keyed by name and ordinal: - it compares raw integers from consecutive samples; - a decrease, or a rail's first appearance, is not an advance; - a rail advances at most once per interval. - **Live means every rail the meter sums kept advancing:** in at least 5 intervals, in at least 4 of every 5 intervals so far, over a window of at least 300 ms. - A live rail meets that in about 300 ms. A lump does not, even one published in several steps. - Every rail is required because the meter sums them. One live die beside a frozen die would otherwise give a figure that silently misses the frozen die while claiming the whole CPU. - **The same rule gates RAPL and NVML.** - RAPL sums the top-level zones whose `name` reads `package-N`, on every socket, and each must be live. It never sums `psys`, which already includes the packages, nor `dram` or the sub-zones. - Each zone wraps at its own `max_energy_range_uj`. A wrap whose range is unknown gives no figure; it never becomes a ~1.8×10¹⁹ µJ reading. - An NVML (GPU-only) figure is Partial where the format can say so. Where it can't, it is never a hardware reading of the whole task. - **It samples until Live,** or until the 1 s budget has elapsed and at least 5 intervals have been seen, with a cap of 5 budgets. - Each tick burns 25 ms of this thread's CPU time, not wall time, with a 500 ms wall cap. So the probe keeps working on a loaded host. - **The result is cached once per process.** Anything but `Live` declines the backend, so the crate's existing honest fallback runs. The reason is public: `IoReportStatus`, `ioreport_status()`, `rapl_status()`, `nvml_status()`. - **The production status is one branch-free composition,** counted each time it runs. Every selection path reads it only through the cached status and a pure `admits(status)` (Live only). - **A live vendor counter's reading is a hardware reading:** HwShunt, or Measured in eoc. That follows the owner's position that vendor counters are real readings and the trust gap is provenance. It holds only while the counter is live, per the MUST in #119. ### How it was verified Six rounds of implementation, each checked by independent adversarial verification. **Pure tests on scripted samples and a scripted clock.** They cover: - the real refreshes (one step, two steps, 3–5 steps, the two-die shape); - decreases, first appearances and failed samples; - live rails (every interval, 80% of intervals, on a loaded host); - the sampling-loop bounds. **Fault hooks inside the real IOReport opener and sampler.** The tests drive the real production status through each script: opener fails, a single lump, a two-step lump, constant values, and a sustained advance. So a bypass written into the glue fails a test. **Hardware tests against an independent oracle.** The oracle has its own subscription, reads absolute values, burns on a helper thread, and counts a rail as moving only on a sustained change. The tests check: - the probe's sampled values equal the oracle's reads on either side; - the live GPU rail shows up in the probe's record, which proves the reads are fresh; - every probe sample is group-only, by channel count; - thread CPU per sample and per tick is checked against the tests' own clock; - the probe runs exactly once per process, checked in a fresh process. **Mutation-tested.** Each of the following was applied as a temporary edit and failed a test: - restore the old one- or two-advance rule; - compare every sample with the first; - treat a decrease as an advance; - map "no subscription" to Live inside the production status; - use a no-op tick, a tick that sleeps, or a tick on the wrong clock; - use a per-thread or non-atomic cache; - probe uncached from the constructor; - a sampler that returns constant, stale or summed values; - an all-channel subscription; - decide on the summed total or on the GPU rail; - an `admits()` that accepts CpuRailAbsent. The counts per crate are below. **Robustness.** Each crate passes under artificial CPU load and while another process samples every IOReport channel in a tight loop. The latter is what made an earlier wall-clock pin flaky, so the job now runs on the shared CI host without that flake. Each also passes on the M3 Ultra. **CI:** these tests are behind `cfg(target_os = "macos")`, and this workspace runs on a linux runner. They reach CI through **#102**, which merges **last**. #102 runs every backend's lib and integration tests on macOS, and fails any step whose frozen-rail test did not print `ioreport liveness:`. **Merge order:** #110, #114, #115, #116 and #117 are independent of each other. #102 goes after all of them. ### Open questions, now decided and implemented (per "fix it all") - **One grade for a live vendor counter.** It is HwShunt, or Measured in eoc, everywhere: a vendor counter is a real reading, and the gap is provenance. It holds only while the counter is live (the MUST in #119). - **The GPU rail as a partial reading of CPU work: no.** The GPU rail is machine-wide (display compositing moves it). Booking it to a task would misattribute energy and under-debit budgets. - **Wire keys named `measured_uj` that carry estimates.** Their grade now sits beside them in plaintext: jcp-gateway's `_meta` gets `energy_provenance`, plus `energy_coverage` when partial. The keys stay, for compatibility. - **eoc cache hits** are an estimated zero, never Measured. - **joule-ui.** `Meter` defaults to Estimator, and the conformance fixture keeps HwShunt explicitly, so the vectors are byte-identical. `Authorizer::charge` carries the figure's provenance. - **A live CPU energy source on macOS 27 exists.** It is the kernel's per-process counter `ri_energy_nj`: unprivileged, live on both Macs, and attributed to the work. It is wired in as stacked PRs #120–#124, only where the work runs in-process. (Privileged `powermetrics` is live too, but machine-wide and root-only, so it is not used.) ### Still open - **The joule-ui demo on prod.** It has run on macOS 27 since 2026-09-15. Its sealed chain was not inspected, because the read was blocked as a production read. After #114 merges, redeploying `joule-ui-serve` gives it honest grades, and the per-process probe in #121 gives it real readings. - **Linux RAPL and NVML have never run against real hardware.** No Linux host with readable RAPL and no NVIDIA GPU were available. They are covered by fake-tree and scripted tests, type-checked for Linux, and declined fail-safe. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dcharlot added 6 commits 2026-09-30 14:31:47 -04:00

sandbox-meter: a CPU energy counter that did not move is never a measurement ... 3b6a7c67c2

The crate's contract, in its probe module doc: "A probe can degrade but it
can never lie: if the hardware path fails mid-span, the reading comes back
`Estimator`, not a zero dressed as a measurement." On current macOS the
IOReport probe broke that on every Apple Silicon host.

Ground truth, measured 2026-09-29 by dumping the Energy Model channels while
burning CPU, not by re-running tests:

- M5 Max, macOS 27.0.1: 364 channels. Of the three that `class_for` maps,
  `CPU Energy` (mJ) held at 480,083,160 and `ANE0` held across 3 s of burn;
  `GPU Energy` (nJ) advanced 682,543,073. Nothing else moved.
- M3 Ultra, macOS 27.0.1: 645 channels, none moved, including
  `DIE_0_CPU Energy`, `DIE_1_CPU Energy`, `ANE0_0`, `ANE0_1`, `GPU Energy`.

The defect. `IoReportProbe::new()` accepted the probe once a subscription
opened and one snapshot was non-empty, and `Span::stop` sealed every class
delta, zero included, as `measured_uj: Some(delta)` under `HwShunt`. main's
own `joule-meter 1`, busy on all cores, sealed and verified:

    M5 Max:    Cpu: 0 µJ [HwShunt]  Npu: 0 µJ [HwShunt]  Gpu: 226790 µJ [HwShunt]
    M3 Ultra:  Cpu: 0 µJ [HwShunt]  Npu: 0 µJ [HwShunt]  Gpu: 0 µJ [HwShunt]

On the M5 Max the display's GPU energy is passed off as the energy of CPU
work; on the M3 Ultra, three measured zeros. The same figure flows on:
`sample_uj()` sums the classes, and `run_attributed` apportioned a 0 J
"measurement" with `attributed = true`.

That is why dependants fail on main on these machines. This crate's
`attributed_run_captures_a_busy_child` fails at attribute.rs:297 ("a busy
window costs SoC energy"). joule-code-remediate's
`verified_fix_end_to_end_ships_with_receipt` fails with
`report.total_uj() > 0`: it apportions the frozen rail's zero. jcp-gateway's
`granted_tool_is_allowed_metered_and_debited` (lib.rs:585, `measured_uj > 0`)
and `budget_exhaustion_denies_further_calls_and_the_log_verifies` (lib.rs:638,
"the budget eventually exhausts") assert, under `metering_is_hardware()`, that
a span's summed total moved. That holds only while the GPU rail happens to
tick: 40/40 green on the M5 Max this morning with the display compositing,
red when it idles, and 3/3 and 2/3 red on the M3 Ultra.

The fix applies the contract in the three places the decision is made.

1. Selection. `IoReportProbe::new()` declines unless `ioreport_status()` is
   `Live`. The CPU rail itself (`CPU Energy`, or `DIE_<n>_CPU Energy` on
   multi-die parts, matched after stripping the die prefix) must advance
   while the probing thread burns CPU. It is sampled every ~25 ms for up to
   `CPU_RAIL_LIVENESS_BUDGET` (1 s), probed once per process and cached in a
   OnceLock. The probe reads the same absolute samples `snapshot()` reads, on
   the probe's own subscription, so it tests the path a span will use. It
   decides per rail on the raw integer, before unit conversion, paired by
   position and name. It never decides on a summed total, which the live GPU
   rail advances by itself. Otherwise `new()` returns `ProbeError::Unavailable`
   naming the frozen `CPU Energy` rail, and `PowerMeter` falls back to its
   estimator: wall × TDP, `Estimator`, no `measured_uj`. A probe whose snapshot
   carries no CPU class is refused too, because GPU and ANE alone would be the
   same defect.
2. The span. In a span this process ran in, a CPU-class delta of 0 is a
   counter that held, not a reading. The whole span now returns the
   estimator record, exactly as when every probe vanished. That covers a rail
   that freezes after the per-process probe, and RAPL: a package counter that
   did not advance is no measurement either. A probe set with no CPU class
   (NVML alone) is not subject to it. I checked for tests that would break:
   none expects a short span to be `HwShunt`. jcp-gateway's hardware-only
   asserts need only a positive figure, which an estimate is. Whether the
   Linux runners' containers can read RAPL was not verified here.
3. Attribution. `run_attributed` sets `attributed` only when the window's
   CPU-class record is an `HwShunt` reading that advanced, on top of the
   existing rusage and busy-tick conditions. An estimate is carried as
   context and never apportioned as a reading.

`IoReportStatus` (`Unavailable`, `CpuRailAbsent`, `CpuRailFrozen`, `Live`,
with a Display), `ioreport_status()` and `CPU_RAIL_LIVENESS_BUDGET` are
public and re-exported at the crate root, so a caller or CLI can say why an
Apple Silicon host is estimated. `joule-meter` now does. Its "machine-wide
counters; subtract idle" note is printed only for a measured span, because it
is false for an estimate. Grade names are untouched: a live IOReport reading
is still `HwShunt`.

Tests. Every IOReport test asserts against an independent oracle. The
implementation reads absolute samples on its own subscription and strips a
die prefix. The oracle opens its OWN subscription, reads DELTA samples with
`IOReportCreateSamplesDelta` (the implementation never calls it), matches
`ends_with("CPU Energy")`, and burns for three times the probe's budget.
No branch passes without asserting.

- `the_ioreport_probe_is_never_selected_on_a_frozen_cpu_rail` checks the
  property and prints "ioreport liveness: status=.. oracle=.. selected=..".
- `the_liveness_probe_never_reports_a_frozen_rail_as_live` checks the
  dangerous direction, and checks every other status claim against the
  oracle.
- `a_selected_ioreport_probe_meters_cpu_work`: when selected, a CPU burn
  meters an `HwShunt` CPU record > 0. When declined, the status is not Live,
  the span is the estimator's, and `sample_uj()` is None.
- `a_cpu_rail_is_the_rail_itself_on_one_die_or_many` and
  `liveness_is_decided_per_cpu_rail_on_the_raw_integer` cover the matcher
  and the pairing, using the dumps' names and values.
- Scripted-probe span tests replay the dumps and are deterministic on every
  platform: `a_span_whose_cpu_counter_held_is_estimated_even_though_the_gpu_advanced`
  (M5 Max), `a_span_where_every_counter_held_is_estimated` (M3 Ultra),
  `a_held_cpu_counter_estimates_the_whole_composite_span` (RAPL held, NVML
  moved), `a_span_whose_cpu_counter_advanced_is_measured`, and
  `a_probe_set_without_a_cpu_class_is_not_subject_to_the_cpu_rule`.
- `detect_never_fails_and_records_validate` used to assert `total_uj() > 0`,
  which the GPU rail satisfied. It now asserts the CPU record or the
  estimator record, and on Apple Silicon that the probe exists exactly when
  the status is Live.
- `attributed_run_captures_a_busy_child` used to assert `soc_cpu_uj > 0` and
  fails on main. It now asserts per status: Live means attributed; otherwise
  not hardware, not attributed, and an estimator figure > 0. Two scripted
  tests pin the rule itself: `a_window_whose_cpu_counter_held_is_never_attributed`,
  and `a_window_whose_cpu_counter_advanced_is_attributed` for the positive
  path, which no macOS 27 host reaches for real. Their child now runs
  2–3 s, bounded by the clock and an iteration cap. The machine tick
  counters behind `busy_cpu_secs` update in bursts up to ~990 ms apart on
  both chips. A ~0.3 s child saw `busy_cpu_secs = 0.0` on the M3 Ultra and
  would have declined for the wrong reason.

Observed with the same test binary on both chips, 5 runs each, 31/31 every
time:

    M5 Max:   ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false
    M3 Ultra: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false

Falsified. Each mutation was restored with git checkout, leaving a clean tree:
- F1, the probe always says Live: the frozen-rail test fails ("the oracle saw
  the CPU rail hold under 3s of CPU burn", left: Live, right: CpuRailFrozen),
  and so do the never-Live, metering and attribution tests.
- F2, main's ungated `new()`: it fails with "IOReport probe selected although
  its CPU rail did not move under 3s of CPU burn — every span would report
  GPU-only energy, or zero, as a measurement of CPU work", and 3 others fail.
- F3, the per-span rule removed: the three held-counter span tests fail. The
  M5 Max one reproduces main's seal exactly: Cpu Some(0), Npu Some(0),
  Gpu Some(682543), all HwShunt.
- F4, main's attribution rule: `a_window_whose_cpu_counter_held_is_never_attributed`
  fails because the estimator's 27.8 J is attributed as a reading.

Dependants, run on main and on this branch: sandbox-attest, -runtime,
-exec-wasm, -exec-microvm, -conformance (and its suite: all classes, no
skips), -proof, -proof-wasm, -mcp, -build, -cli, -jcp, jcp-gateway,
jcp-action-receipt, joule-ui-interaction and joule-ui-serve pass on the
branch. joule-code-remediate passed 15 of 16 branch runs. The failure was
`verified_fix_end_to_end_ships_with_receipt` with the assertion it fails
with on every main run, `report.total_uj() > 0`. On the branch that figure
is estimate × share, and the share is 0 when no machine tick burst lands in
any of its cargo windows. jcp-gateway passes because its hardware-only
asserts are now skipped. Both relabel the estimate this crate now returns:
see below. Clippy is clean on this crate (--tests), and so is
x86_64-unknown-linux-gnu.

Not done here, deliberately:
- Consumers that relabel the estimate this crate now returns: jcp-gateway
  debits and seals `Energy::measured(measurement.total_uj())` (lib.rs:255-259)
  and labels its eval by `meter.is_hardware()` (lib.rs:228, 430). Its
  hardware-only asserts are now skipped on Apple Silicon.
  joule-code-remediate seals an estimate-apportioned figure as `ModelBased`
  (lib.rs:656), and its end-to-end test passes here only because
  `total_uj()` is now an estimate > 0. jcp-action-receipt
  (examples/end_to_end.rs:139), sandbox-proof (lib.rs:1321) and joule-ui
  label by `is_hardware()` rather than by the span. They are for the next
  change, on this branch.
- A GPU or ANE zero beside a CPU counter that advanced, and an NVML-only
  set's zero GPU delta, are still sealed `HwShunt` 0 (probe/mod.rs:340).
- Reporting the live GPU rail as an explicitly partial reading instead of
  declining the probe is a receipt and wire decision.
- `class_for`'s substring match (ioreport.rs:101-111) sums every
  `DIE_n_CPU Energy`, which is right for an Ultra. It would also sum a
  sub-rail named "…CPU Energy", and it takes any channel containing "ANE".
  On these two dumps only the top-level rails match; other chips need a
  dump.
- `sample_uj()` sums every class, so its before/after users cannot apply the
  held-CPU rule. `Attribution` has no provenance for `soc_cpu_uj`. Both are
  API changes.
- `busy_cpu_secs` inherits the ~1 s tick bursts, so windows under ~1 s are
  declined or skewed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

sandbox-meter consumers: a figure keeps its span's provenance, and work that ran is never 0 J ... 3d9eb150be

The contract, stated in jcp-energy and sandbox-meter alike: measured when the
platform exposes it, estimated when it doesn't, "and the provenance always
says which". The parent commit makes sandbox-meter keep it. On macOS 27.0.1
no Apple Silicon host checked exposes a live CPU energy counter, so
PowerMeter declines IOReport and every span is the estimator's wall × TDP
record. On a live host, a span whose CPU counter held is estimated too. The
meter's consumers then threw that provenance away.

Ground truth: the consumers' own output at the parent commit. The same
binaries ran on an M5 Max and an M3 Ultra, both on macOS 27.0.1; the M5 Max
comes first.

- jcp-gateway printed "metering=estimator", then "allowed, 1541728 µJ
  measured", and sealed energy_provenance "HwShunt" in the receipt. It
  debited Energy::measured(total_uj()) whatever the span was. On the M3
  Ultra: "999293 µJ measured", sealed HwShunt.
- joule-ui-serve sealed every interaction as Estimator with estimated_uj 0,
  for 24-56 ms of work. Its sampler summed raw counters under the Meter
  trait's default HwShunt. It returned None once no hardware probe was left,
  and run_interaction turns None into 0 J.
- joule-fix printed "energy per verified fix: 1.446 J (attributed)" for an
  estimate apportioned by CPU share, and sealed it ModelBased in the JCR-1
  meter envelope (M3 Ultra: 7.377 J, ModelBased). The share is 0 whenever
  the machine tick counters miss every window, so the figure can be 0 J. The
  parent commit gave that as the likely cause of verified_fix_end_to_end's
  1-in-16 failure.
- sandbox-proof said "ESTIMATED" here only because it asked is_hardware().
  On a live host it would say "measured" for a span the meter estimated.

The defect. Each consumer labelled a span by the meter instead of by the
span's own records: is_hardware(), a trait default, or a hard-coded
"ModelBased" or "measured". Or it turned a meter with no reading into 0 J.

The fix. Every consumer derives the label from the Measurement it got: the
weakest of its records, because a sum is only as good as its weakest part.

- jcp-gateway. energy_of() builds the ledger entry with jcp-schema's own
  constructors:
    - Energy::estimator for the estimator's record;
    - Energy::measured for a hardware reading that has a CPU record;
    - Energy::measured_partial for a hardware reading without one (a GPU
      counter alone, NVML without RAPL);
    - Energy::modeled for ModelBased.
  The figure is unchanged, so a runaway is still debited. Only what the
  receipt says about it changes, through fields it already had:
  energy_provenance, and energy_coverage when Partial. The flow monitor's
  scan is sealed with its own span's provenance (flow_energy_provenance)
  instead of is_hardware()'s. The binary's summary line reads the grade
  back from the receipt it just sealed ("[Estimator]") instead of printing
  "measured". The docs and the crate description said "metered on real
  hardware" and "the MEASURED joules"; both are corrected. jcp-gateway now
  depends on sandbox-schema directly, for the CPU class; the crate was
  already in the lock graph.
- joule-code-remediate. An EnergyTally grades the run by its cargo windows.
  The run is ModelBased only when every window was attributed. main needed
  only one, and never reset the flag between runs. An attributed run's
  receipts carry the attributed figure, anchored as before. Any other run's
  receipts carry the unapportioned total over the windows, as an estimate:
    - the JCR-1 meter envelope seals it with the Estimator grade;
    - the JouleContract receipt seals it as Unaccounted;
    - total_uj() and uj_per_verified_fix() return it.
  An apportioned estimate is no better than an estimate, and apportioning can
  put 0 J on the receipt for work that ran. joule-fix prints "estimated, not
  attributed" and "(estimated)" to match.
- joule-ui-serve. SpanMeter replaces the summing sampler. Every sample closes
  the span the previous sample opened, adds its figure to a running counter,
  and opens the next. provenance() returns that span's own grade, which
  run_interaction asks for right after its second sample. An estimated span
  is sealed as Estimator with the estimator's figure, not 0 J, and a hardware
  span as HwShunt. The Meter trait is untouched. The binary's doc and its
  Cargo.toml said every receipt's HwShunt was true; corrected.
- sandbox-proof. prove_local says "measured" only when every record of the
  span is a hardware reading (energy_word). The records were, and still are,
  sealed verbatim into the receipt, the attestation and the proof cell.
- jcp-action-receipt. The end_to_end example sealed energy_provenance from
  is_hardware(); it now takes the span's.

Checked and left alone:
- sandbox-jcp debits a declared ceiling as Energy::estimator and never reads
  a PowerMeter.
- joule-ui-interaction's library labels nothing as a hardware reading that
  its meter did not declare one. Its jcp_metered example wraps whatever figure
  it is charged in Energy::measured, but only in a JCP runtime decision that
  it neither seals nor prints. See below.

After the fix, with the same binaries on both chips:
- jcp-gateway prints "allowed, 2309714 µJ [Estimator]" and seals Estimator
  (M3 Ultra: 973555 µJ, Estimator).
- joule-ui-serve seals 482410 and 1121739 µJ as Estimator for intensities 3
  and 7 (M3 Ultra: 481236 and 1121485 µJ).
- joule-fix prints "44.352 J estimated, not attributed" and seals Estimator
  and Unaccounted (M3 Ultra: 20.723 J).
- sandbox-proof still says ESTIMATED.

Tests. The same test binaries ran 5/5 green on each chip. On the M3 Ultra,
joule-code-remediate's the_shipped_formal_template_parses_and_validates
cannot find formal.example.json, whose path is fixed at compile time on the
dev machine, so the five runs there skip that one test. That host also has no
z3, so the two existing formal-proof tests return early there.
- jcp-gateway a_metered_span_enters_the_ledger_as_what_it_is: every span
  shape maps to its constructor, and the debit is the span's own figure.
- jcp-gateway a_calls_receipt_says_how_its_joules_were_obtained: end to end,
  the sealed energy_provenance is Estimator when the meter has no hardware
  source ("gateway call: hardware=false sealed=Estimator cost_uj=30621").
  Otherwise it is a reading or an estimate. cost_uj > 0 either way.
- jcp-gateway granted_tool_is_allowed_metered_and_debited and
  budget_exhaustion_denies_further_calls_and_the_log_verifies asserted only
  under metering_is_hardware(), so on these hosts they passed without
  asserting. They now assert on every host: an estimate is debited, and the
  budget exhausts.
- The exfiltration test now also checks flow_energy_provenance.
- joule-code-remediate
  an_unattributed_run_is_sealed_as_an_estimate_never_an_apportioned_zero and
  a_run_is_attributed_only_when_every_window_was pin the tally.
- joule-code-remediate verified_fix_end_to_end_ships_with_receipt now also
  checks that joules_spent equals total_uj(), the JouleContract class, and
  the grade inside the sealed meter envelope. It also checks that nothing is
  attributed without a live CPU rail.
- joule-ui-serve each_served_interaction_carries_its_own_spans_provenance
  replays a HwShunt span, then an Estimator span, through the real Host.
- joule-ui-serve the_served_meter_never_seals_zero_or_an_estimate_as_a_reading
  runs the served meter on this host.
- sandbox-proof a_local_proof_says_estimated_for_an_estimated_span.

Falsified. Each mutation was restored with git checkout, leaving a clean tree.
- energy_of returns Energy::measured(total), main's rule: both new gateway
  tests fail. The end-to-end one fails with left "HwShunt", right
  "Estimator".
- eval_prov is sealed as HwShunt, which is what main sealed here: the
  exfiltration test fails with "an estimated scan sealed as a reading".
  Reverting eval_prov to is_hardware()'s answer is not caught on these hosts,
  because with no hardware source both say Estimator.
- energy_of returns Energy::measured(0), as main did on the M3 Ultra: four
  tests fail, including the two formerly hardware-only ones ("a real span
  costs joules, measured or estimated", "the budget eventually exhausts").
- figure_uj returns the apportioned sum: three joule-code tests fail. The
  unit tests see 10000000 against 60000000; end to end, 1.373789 J sealed
  against 65.634219 J.
- The meter envelope is always graded ModelBased: three tests fail. End to
  end: "an estimate sealed in the meter envelope as ModelBased".
- main's any-window attribution: the mixed-window test fails.
- main's total_uj, the apportioned figure: the end-to-end test fails
  (45.799836 against 0.776724).
- SpanMeter without its provenance() override, so the trait's HwShunt
  applies: both joule-ui-serve tests fail.
- served_meter returns the parent commit's HardwareMeter: the end-to-end test
  fails with "~24 ms of work sealed at 0 J: Energy { measured_uj: None,
  estimated_uj: 0, provenance: Estimator }".
- energy_word always says "measured": the sandbox-proof test fails.
Not separately tested: the per-run reset of the tally. It is observable only
with attributed windows, which need a live CPU rail.

Clippy --no-deps --all-targets is clean on every changed crate. It is also
clean for x86_64-unknown-linux-gnu, except that sandbox-proof could not be
checked for that target: zstd-sys's build script needs a Linux C cross
compiler, which is not installed here. One pre-existing warning, in
jcp-action-receipt's ar-verify.rs, is untouched.

Not done here, deliberately:
- The joule-ui Meter trait: its default provenance is HwShunt, and its
  provenance is per meter, not per sample. SpanMeter relies on
  run_interaction asking for provenance right after its second sample.
  run_interaction also turns a None sample into Estimator(0), so an
  Authorizer charges 0 for work that ran. Authorizer::charge carries no
  provenance. Both joule-ui-interaction examples still use the summing
  sampler. These are public API and SPEC §5/§6: David's call.
- Wire keys named measured_uj now carry estimates: jcp-gateway's _meta and
  Relayed::Allowed, and joule-ui-serve's POST /action response. The sealed
  provenance beside each key is right.
- A GPU counter alone (NVML without RAPL): the gateway seals it Partial but
  still debits only the GPU figure, so CPU work goes undebited there. Its two
  budget tests would fail on such a host, correctly. joule-ui has no coverage
  field. Not observed: no such host here.
- JouleContract §4.2 says an Unaccounted figure seals its method reference
  (energy_anchor). The harness leaves it None, as it always has.
- proof-cascade's joules: FormalOutcome, "(real z3, ModelBased)" in joule-fix,
  and formal_fix's "µJ measured". Another change owns that meter.
- joule-code-remediate's Cargo.toml description still says hardware-measured.
  This commit does not touch that file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

sandbox-meter: a CPU rail is live only when it keeps advancing, never on one step ... 67f821e6a5

The family contract: measured when the platform exposes it, estimated when
it doesn't, "and the provenance always says which ... A reading never
silently upgrades its own provenance." 3b6a7c67 applied it to this crate's
IOReport probe, but its liveness probe called a CPU rail live on a single
advance, and on current macOS a single advance is exactly what a frozen rail
produces now and then.

Ground truth, measured 2026-09-29 on an M5 Max and an M3 Ultra (both macOS
27.0.1) by reading the channels, not by re-running tests:

- For an unprivileged reader the Energy Model CPU rails (`CPU Energy`;
  `DIE_<n>_CPU Energy` on the Ultra) do not advance under CPU load over
  seconds to minutes, but they are not frozen for good: they are refreshed
  in rare batches. The M3 Ultra's two dies held for at least 22 minutes, then
  stepped +1,344,869 and +1,391,637 mJ within one 100 ms sample, and held
  again. The M5 Max's rail read 480,083,160 mJ at ~10:30 and 541,893,638 mJ
  by 10:47, then did not change in 13,614 samples over a 25-minute watch.
  Cadence and trigger are unknown.
- `GPU Energy` is live on the M5 Max (a change about every 28 ms) and moves
  only sporadically on the M3 Ultra.
- One IOReportCreateSamples over the Energy Model group takes ~6 ms and
  returns the same absolute values as one over all channels.

The defect. `probe_cpu_rail` compared every sample with the first one and
returned Live on the first `a > b` of any CPU rail. A probe window that
straddles a batch refresh sees one large step and calls it live: that
happened with the same rule at ~10:46 on the M5 Max, in the jouleclaw-energy
work. The probe's answer is cached for the process, so the IOReport probe is
then used for its whole life, and each later ~1.3 kJ refresh lands whole in
whichever span straddles it. `Span::stop`'s rule (a CPU delta of 0 means the
span is estimated) cannot catch that: the refresh is not 0, so it is sealed
`HwShunt`. The hardware oracle had the same blind spot: it counted any change
at all, as a delta from its first sample, so one refresh landing in both
windows would have vouched for the false Live.

The fix ports the family liveness rule (jouleclaw-energy b9d7e945's
RailWatch, eoc-meter c6378629's LIVE_AFTER_STEPS), with no new dependency:

- `RailWatch`, a pure struct, is the rule. Per CPU rail (channel name, after
  stripping `DIE_<n>_`, equal to `CPU Energy` ignoring case), keyed by
  position and name, it counts the intervals in which that rail's raw
  integer rose between consecutive samples. It is never applied to a summed
  total. LIVE means one rail advanced in at least `LIVE_RAIL_ADVANCES` (2)
  separate intervals. Two dies stepping once each, or two same-named channels
  stepping once each, are not live.
- `wants_more` keeps the probe sampling, with ~25 ms of spin on the calling
  thread between samples, until Live, or until `CPU_RAIL_LIVENESS_BUDGET`
  (1 s) has passed with at least two intervals seen. Hard cap: five budgets.
- `verdict`: Live, Frozen (a CPU rail exists, none advanced twice), Absent
  (no CPU rail), or none (not one interval could be sampled). None now maps
  to `Unavailable`. It used to report Absent when the first sample failed.
- `rail_advances` pairs consecutive samples by position AND name. If the
  channel list changes between samples, no rail advanced in that interval
  (the safe side).
- The probe already sampled its own Energy Model subscription
  (IOReportCopyChannelsInGroup, then IOReportCreateSubscription), so the
  "polled every ~25 ms" claim was true here. It is now pinned by a test. The
  counter's read path is the same group-only subscription, unchanged.

Still cached once per process (OnceLock). Anything but Live declines the
rung, as before. The public API (`IoReportStatus`, `ioreport_status()`,
`CPU_RAIL_LIVENESS_BUDGET`) and the grade names are unchanged. Docs that the
rule made false are corrected: the status variants and their Display text
("did not advance" is now "did not keep advancing ... (it held, or stepped
once)"; Unavailable now also covers "or sampled"), the budget and
`ioreport_status` docs, `PowerMeter::detect`'s "up to the budget" (the probe
can sample past it, never past five budgets), the module doc, and one
test-constant comment in attribute.rs.

Span::stop: no plausibility bound was added for a CPU delta too large for
its wall time. There is no evidence to set one. No host here has a Live
probe, so there is no measured live-rail power ceiling to bound against. The
only straddled refreshes observed were on rails the new rule now declines.
Picking a wattage would be inventing a threshold. Under this rule a Live
verdict needs the same rail to step twice within about a second. A rail that
behaves like that and then falls back to rare batches was never observed. If
it is, it needs a per-read witness, not a guessed bound.

Tests. The rule tests need no hardware:
- a_single_jump_is_a_batch_refresh_not_a_live_rail (the M5 Max's
  480,083,160 to 541,893,638 mJ step, fed through rail_advances);
- one_jump_on_each_die_is_not_a_live_rail (the observed same-sample step on
  both dies, and one step per die in different intervals);
- two_channels_that_share_a_name_are_two_rails;
- two_advances_of_the_same_rail_are_live (adjacent or not, either die,
  through the raw-integer pairing too);
- no_interval_gives_no_verdict_and_no_cpu_rail_is_absent;
- the_probe_samples_until_live_or_long_enough_to_tell (including the
  five-budget cap with zero and with one interval);
- an_interval_is_read_per_cpu_rail_on_the_raw_integer, which replaces the
  test for the old single-comparison helper.

The hardware oracle now opens its OWN subscription to EVERY channel (the
probe never does). It reads ABSOLUTE values (no delta samples, no deltas from
a first sample) of every channel whose name ends_with("CPU Energy") in any
group. A HELPER thread burns CPU for the whole 3 s window (3x the budget), so
the window stays loaded even while the reading thread blocks in an
all-channel sample. The oracle returns CpuRailMoved(true) only if one rail's
absolute value changed at two or more separate reads, a literal threshold of
its own. It is not blind: pointed at `GPU Energy` on the M5 Max, it counted
16 changes in 16 reads and 17 in 17.

- the_ioreport_probe_is_never_selected_on_a_frozen_cpu_rail prints the CI
  line "ioreport liveness: status=.. oracle=.. selected=.. available=..".
- the_liveness_probe_never_reports_a_frozen_rail_as_live now also runs the
  probe uncached on a fresh subscription, beside the cached status. It
  checks both against the oracle, checks that Frozen rests on at least two
  intervals and that the probe respects its cap, and prints the probe's
  cost.
- the_probe_samples_the_energy_model_group_alone (new): every channel in the
  probe's sample belongs to the Energy Model group, and a CPU rail is among
  them unless the status says Absent.
- a_selected_ioreport_probe_meters_cpu_work is adapted to the new rule.

Observed with the same test binaries on both chips, 38/38 in every run (6 on
the M5 Max, 5 on the M3 Ultra; the crate had 31 tests):
  M5 Max:   ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false
            ioreport oracle: 16-20 reads over 3s under a helper-thread burn, changes per CPU rail [0]
  M3 Ultra: ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false available=false
            ioreport oracle: 17 reads over 3s under a helper-thread burn, changes per CPU rail [0, 0]
Probe cost, uncached: M5 Max 1.096-1.125 s for 30-33 intervals over 364
channels, one group sample 5.96-10.49 ms. M3 Ultra 1.103-1.153 s for 31-33
intervals over 645 channels, one group sample 5.71-7.32 ms. That is ~34 ms
per interval: 25 ms of spin plus the sample.

Falsified. Each mutation was applied to this change and restored with git
checkout, leaving a clean tree:
- F-rule, LIVE_RAIL_ADVANCES = 1: five rule tests fail, e.g.
  a_single_jump_is_a_batch_refresh_not_a_live_rail "one refresh counted as a
  live rail: RailWatch { intervals: 4, rails: [(0, "CPU Energy", 1)] }";
  one_jump_on_each_die_is_not_a_live_rail "left: Some(Live) right:
  Some(Frozen)"; two_channels_that_share_a_name_are_two_rails (same);
  two_advances_of_the_same_rail_are_live "one advance is not yet live";
  the_probe_samples_until_live_or_long_enough_to_tell "one interval is not
  enough to tell".
- F1, the probe always says Live: on both chips the frozen-rail test fails,
  "the oracle did not see the CPU rail keep moving under 3s of CPU burn
  left: Live right: CpuRailFrozen" (status=Live selected=true). So do the
  probe test ("the cached status said Live, the oracle did not see the CPU
  rail keep moving"), the metering test and the attribution test.
- F2, IoReportProbe::new() ungated: on both chips the frozen-rail test fails,
  "IOReport probe selected although its CPU rail did not keep moving under 3s
  of CPU burn — every span would report GPU-only energy, zero, or a stale
  refresh as a measurement of CPU work" (status=CpuRailFrozen selected=true),
  as do detect_never_fails_and_records_validate, the metering test and the
  attribution test.
- F-sum, liveness decided on the summed raw total of all Energy Model
  channels: the M5 Max probe says Live after 2 intervals (167-185 ms), and
  the frozen-rail and probe tests fail ("the uncached probe said Live, the
  oracle did not see the CPU rail keep moving"). It failed the same way on
  the M3 Ultra too, Live after 10 intervals.

Dependants, run on the M5 Max. sandbox-attest (49), -build (7), -conformance
(3), -exec-microvm (12), -exec-wasm (13), -jcp (4), -mcp (3 + roundtrip 2),
-proof (16), -proof-wasm (4), -runtime (13) pass. So do jcp-gateway (12),
jcp-action-receipt (17 + conformance 3), joule-code-remediate (10),
joule-ui-interaction (6) and joule-ui-serve (4 + 2).

The same binaries on the M3 Ultra: jcp-gateway 12, jcp-action-receipt 17,
joule-ui-interaction 6, joule-ui-serve 4 + 2, sandbox-proof 16, sandbox-attest
49, -build 7, -exec-microvm 12, -jcp 4, -mcp 3, -proof-wasm 4 and -runtime 13
pass. joule-code-remediate passes 9 of 10 (9 with that one skipped),
sandbox-conformance 2 of 3 and sandbox-exec-wasm 12 of 13. Eight tests in
five places fail there, for a reason that is not this change: each reads a file, or spawns a
binary, at a path fixed at compile time on the dev machine, and that
worktree does not exist on the M3 Ultra ("No such file or directory"):
- jcp-action-receipt tests/conformance (3, "conformance dir must exist next
  to the spec");
- joule-code-remediate the_shipped_formal_template_parses_and_validates
  ("template present");
- sandbox-mcp tests/roundtrip (2, "spawn server");
- sandbox-conformance committed_vector_files_verify ("read
  content_addressing.json");
- sandbox-exec-wasm wasmtime_version_matches_the_lockfile ("workspace
  Cargo.lock").

clippy -p sandbox-meter --no-deps --all-targets is clean on
aarch64-apple-darwin and x86_64-unknown-linux-gnu. On x86_64-apple-darwin
its one warning (unused_mut on `probes` in `PowerMeter::detect`) is in a
line this change does not touch. rustdoc's two warnings (contention.rs:14,
probe/mod.rs:6) are likewise untouched lines.

Not done here, deliberately:
- `IoReportProbe::open()` never releases the `desired` channel dictionary
  when IOReportCreateSubscription returns a separate `subbed` one: one
  CFDictionary per open. The leak is older than this change and small.
- A rail that stops tracking after the per-process probe said Live still
  needs a per-read witness (CPU Stats residency, say). A plausibility bound
  in Span::stop needs a live host to measure against (see above).
- Reporting the live GPU rail as an explicitly partial reading, instead of
  declining the rung, is a receipt and wire decision.
- The other IOReport backends in the family are separate crates and
  separate branches.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

sandbox-meter: the liveness probe's sampling loop is tested, and every sample is read against the one before it ... cee6677357

The parent commit's rule is sound, but a round-2 adversarial review showed that
nothing tested the loop that fed it samples. That loop is where the original
defect lived: it compared every sample with the first one. Deleting
`prev = cur;` put that defect back, and all 38 tests still passed on the M5 Max
and on the M3 Ultra. Only a batch refresh landing inside the 1 s probe window
could have caught it. Four more mutations also survived:
- the once-per-process cache removed;
- a probe with no verdict reported as CpuRailAbsent instead of Unavailable;
- the probe sleeping between samples instead of burning CPU;
- joule-code-remediate's per-run tally reset deleted.

The same review found two more problems in this branch. First,
examples/meter_cmd.rs printed "marginal: 0.000 J (work − idle, the workload's
honest share)" on every estimated host. Work and idle there are both
wall × TDP over the same wall time, so the difference is 0 by construction.
Second, IoReportProbe::open() leaked a CFDictionary, despite a module doc that
said every Copy/Create was released.

The rule, apart from the FFI (probe/liveness.rs). The module is compiled on
every platform under test, so the rule's tests run where `ioreport` compiles
out, including the Linux runner that runs this workspace in CI. It has four
parts:

- RailWatch consumes each sample whole: every CPU rail's (channel name, raw
  integer), in channel order. It compares each sample with the previous one
  itself. The first sample is the baseline. A rail is keyed by its name and its
  ordinal among the same-named channels in that sample, and it counts at most
  one advance per interval. A channel list that changed between two samples
  still counts as no advance.
- watch(sample, tick, now, budget) is the probe loop, generic over its sampler,
  tick and clock. The real probe passes the FFI sampler, a 25 ms CPU spin and
  Instant. The FFI side now only extracts (name, raw) pairs.
- status_of maps a verdict to IoReportStatus: None to Unavailable, Absent to
  CpuRailAbsent, Frozen to CpuRailFrozen, Live to Live.
- spin is the tick.

One behaviour change, by design: a sample that cannot be read is skipped, not
fatal. Before, a failed first sample returned at once. Now a host whose
IOReport subscription opens but never yields a sample keeps sampling to the
cap of 5 budgets, then reports Unavailable, once per process.

The leak. When IOReportCreateSubscription returns a separate `subbed`
dictionary, open() now releases the `desired` one it asked for, and the
module doc is true again. I measured it with `leaks --atExit` on a
test-only loop of open()+sample+drop (the loop was not committed):
- without the fix: N=1 gave 2812 leaks, 176272 bytes; N=200 gave 562400
  leaks, 35254400 bytes, which is 200 root leaks of one ~172 KB
  NSMutableDictionary each;
- with the fix: 0 leaks at N=0, N=1 and N=200.
So the leak is one dictionary per open, not one per process as the review
reported. In practice that is one or two opens per process: one for the
probe, and one for new() when the rail is Live.

Tests. The probe loop runs on scripted samples and a scripted clock, and its
scripted sampler fails instead of hanging if the loop never stops:
- the M5 Max refresh 480,083,160 -> 541,893,638 mJ, then held, is Frozen with
  one advance;
- the M3 Ultra's two dies stepping together once is Frozen;
- two same-named `CPU Energy` channels stepping together once is Frozen;
- a rail that steps every sample is Live after exactly 2 intervals, and
  the call order is sample, tick, sample, tick, sample;
- samples that never read go on to the 5-budget cap and give no verdict,
  which is Unavailable;
- past the budget, two intervals stop the loop and one does not;
- DIE_0 stepping once and DIE_1 stepping once, in different intervals, is
  Frozen.

The pure RailWatch tests now feed whole samples, and a_verdict_maps_to_its_status
pins the mapping. The hardware tests gain four checks:
- the_liveness_probe_runs_once_per_process: a test-only counter of real probe
  runs is exactly 1 after repeated ioreport_status(), IoReportProbe::new()
  and PowerMeter::detect() calls.
- the_probe_tick_burns_cpu_on_the_calling_thread: at least a quarter of the
  tick in thread CPU time. A sleep uses ~10 µs.
- The uncached-probe test now asserts cadence. A Frozen or Absent verdict
  must rest on at least 10 intervals; a subscription to every channel gets
  2 to 8. A None verdict must have run to the cap.
- The same test checks that the probe burns. It subtracts the CPU cost of
  as many samples taken back to back from the probe's thread CPU, and the
  rest must cover a quarter of the ticks. The bound is a quarter, not all,
  because this machine ran at load average 19-24 on 18 cores from other
  sessions during these runs. In one run the probe got 49% of a core, which
  a "CPU >= half the wall time" bound would have failed.

The declined path meters 120 ms of real CPU work as one Estimator record
equal to wall × TDP, with no measured figure and a non-zero total.

meter_cmd. It now mirrors joule-meter:
- a span that is not fully measured prints its estimate with its provenance
  and no work − idle marginal;
- it says "estimated" when no record is measured, and "partly estimated"
  only when some are;
- the idle baseline runs only for a measured span;
- a measured work figure that does not exceed idle prints "not resolved"
  instead of a floored 0.
On this host it prints "estimate: 17.165 J (estimated) — wall-clock × TDP,
not a reading; no work − idle marginal".

joule-code-remediate. a_second_run_seals_only_its_own_cargo_windows runs
run_once twice on one Harness. It adds a 1 GJ stale window to the tally
between the runs, and asserts that run 2's figure, its receipt and the tally
hold only run 2's windows. The reset's comment gave only the attribution
reason; it now also says a run would seal the previous run's figure, on
every host.

joule-ui-interaction examples. Only the labels changed; the Meter trait, the
Authorizer API and each example's charge path are untouched. Each line is now
labelled from the sealed record, not from the meter:
- a 0 µJ figure for work that ran reads "no figure", never a measurement;
- the totals say "measured" only when every record is a HwShunt reading that
  moved;
- jcp_metered reports the µJ actually debited, and says the budget was never
  spent when nothing was denied;
- on Apple Silicon the header names the IOReport status.
On this host real_meter prints "3 of 3 interaction(s) have no figure (0 µJ
for work that ran): not a measurement". Before, it printed "0 µJ total".
jcp_metered prints "6 had no figure, 0 µJ for work that ran; 0 of the
15000000 µJ budget was debited; 0 denied by the real JCP runtime, as the
budget was never spent". Before, it printed "6 allowed (real joules, debited
from a real grant)".

Doc corrections:
- the all-channel sample cost is 0.13–0.8 s on the M5 Max, not 0.4–0.8 s;
- a group sample costs ~6–12 ms;
- IoReportStatus::Unavailable now says "fewer than two samples could be read
  before the cap";
- the "Cached: the probe runs once per process" comment asserted only
  equality; it now points at the counting test.

Claims in earlier commits of this branch that were false:
- 3d9eb150: "Not separately tested: the per-run reset of the tally. It is
  observable only with attributed windows, which need a live CPU rail."
  Wrong. soc_uj carries over on any host: with the reset deleted, run 2
  sealed 1,000,000,045,883,686 µJ. It is tested now.
- 3d9eb150: jcp_metered "wraps whatever figure it is charged in
  Energy::measured, but only in a JCP runtime decision that it neither seals
  nor prints." It did print "(real joules, debited from a real grant)" while
  debiting 0 µJ. The label is corrected; the Energy::measured wrap is
  unchanged (see below).
- 3b6a7c67 and 67f821e6, module doc: "Every Copy/Create is released." It was
  not, until this commit.

Runs. The same lib test binary ran on both machines:
- M5 Max, 3 of 3: "test result: ok. 48 passed; 0 failed". The probe took
  1.004, 1.030 and 1.012 s over 30, 30 and 20 intervals, and burned 724,
  615 and 333 ms against bounds of 188, 188 and 125 ms. It printed "ioreport
  liveness: status=CpuRailFrozen oracle=CpuRailMoved(false) selected=false
  available=false".
- M3 Ultra (macOS 27.0.1), 1 of 1: 48 passed. The probe took 1.010 s over 32
  intervals, with rails [("DIE_0_CPU Energy", 0, 0), ("DIE_1_CPU Energy", 0,
  0)] and 793 ms burned. The binary was deleted afterwards.

Consumers on the M5 Max, all exit 0:
- jcp-gateway 12, with "gateway call: hardware=false sealed=Estimator";
- jcp-action-receipt 17, plus conformance 3;
- joule-ui-interaction 6;
- joule-ui-serve 4, plus 2 in its bin;
- joule-code-remediate 11 (on the M3 Ultra, 10 with the compile-time-path
  template test skipped as before);
- every other sandbox-rs crate (cargo test --workspace), with the
  conformance suite reporting "All classes passed with no skips."
clippy --no-deps --all-targets is clean on sandbox-meter, joule-code-remediate
and joule-ui-interaction, and on sandbox-meter for x86_64-unknown-linux-gnu. For
x86_64-apple-darwin the only warning is the pre-existing unused_mut in
detect(). Neither Mac has Rosetta, so the x86_64 build compiles but was not
run.

Falsified. Each mutation was applied to this exact tree and the lib tests
run on the M5 Max; each was then restored with git checkout, leaving git
status clean:
- M1, RailWatch keeps the first sample as `prev` (the original defect): 7
  failed, e.g. "left: Some(Live) right: Some(Frozen)" for the M5 Max refresh.
- M1-loop, the loop re-feeds the baseline before each sample: 4 failed.
- M5, no OnceLock: "the real probe ran more than once in this process
  left: 23 right: 1".
- M5b, new() probes afresh: "left: 10 right: 1".
- M8, None mapped to CpuRailAbsent: 2 failed ("left: CpuRailAbsent right:
  Unavailable").
- M9a, spin sleeps: "a 25ms tick used 10.458µs"; the probe check failed too.
- M9b, the probe's tick sleeps: "24.094709ms burned" failed the bound; on the
  M3 Ultra "the probe used 163.3475ms of CPU over 25 intervals, 164.845625ms
  of it sampling".
- A failed sample breaks the loop: 2 failed ("gave up at 43ms, before the
  cap").
- No tick: 4 failed.
- An all-channel subscription: "a CpuRailFrozen verdict on 2 intervals in
  1.157435166s", and the group-only test failed too.
- C3, joule-code-remediate reset deleted: "run 2 sealed an earlier run's
  windows: 1000000045883686 µJ".
The mutations caught in round 2 are still caught:
- threshold 1: 11 failed;
- name-only key: 2 failed;
- no die strip: 1 failed on the M5 Max, and 2 on the M3 Ultra ("status=
  CpuRailAbsent");
- deciding on the GPU rail: 5 failed;
- stopping at the budget with fewer than 2 intervals: 3 failed;
- the declined meter's sample_uj returning Some(0): 1 failed;
- summing advances across rails: 5 failed;
- no cap: 2 failed;
- gating new() only on Unavailable: 4 failed;
- a verdict from zero intervals: 2 failed.

Left for David. None of these is changed here:
- jcp-gateway's `_meta` key measured_uj carries an estimate on every declined
  host, and only the sealed receipt says Estimator. That is a wire decision.
- The joule-ui Meter trait still defaults to HwShunt, its provenance is per
  meter, not per sample, and a None sample becomes Estimator(0).
- Authorizer::charge carries no provenance, so jcp_metered still wraps its
  figure in Energy::measured.
- CI: this workspace runs only on the Linux runner. The liveness rule's
  tests now run there; the IOReport hardware tests and the "ioreport
  liveness:" line need a macOS job, which the workflow owner decides.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

sandbox-meter: the probe's tick burns this thread's CPU time, and no hardware pin rests on a wall-clock ratio ... d7adbe6cc2

The liveness probe's hardware tests run in CI on the macOS runner, which is
the M5 Max this was written on, at load averages of 24-104. The parent
commit's tick spun for 25 ms of WALL time, so on a loaded machine each tick
burned only the CPU the scheduler gave the thread. Its tests pinned the burn
and the group-only subscription with load-dependent bounds: burned CPU at
least a quarter of the ticks, and a Frozen verdict on at least 10 intervals.
With the tick below, the probe completes 11-13 intervals in its 1 s budget
under 2 x ncpu busy-loop spinners (29 on an idle machine), so that floor
would have sat one to three intervals from failing.

The tick (probe/liveness.rs):

- spin(cpu, wall) burns until the thread's CPU clock has advanced PROBE_TICK
  (25 ms), with a 500 ms wall cap per tick. The real probe passes
  CLOCK_THREAD_CPUTIME_ID (thread_cpu in ioreport.rs) and a monotonic
  clock. A loaded machine makes a tick longer, never lighter.
- Both clocks are injected, so the rule is tested with scripted ones: a full
  tick on a third of a core, and a thread that gets no CPU ending at the
  cap. The loop still takes an injected tick, and the scripted-loop tests
  are unchanged.

What the probe records as it runs (ProbeTrace, returned beside the
RailWatch):

- the wall time of every FFI sample call;
- per interval, the thread CPU its ticks burned, measured around the ticks,
  apart from the samples.

The hardware test the_liveness_probe_never_reports_a_frozen_rail_as_live
now asserts:

- every interval burned at least PROBE_TICK/2 (only the wall cap can cut a
  tick short, and the message says so);
- the median sample call took under 50 ms (a group sample is ~6-12 ms,
  one over every channel 0.13-0.8 s), and it prints that median.

The ">= 10 intervals" floor, the "burned >= a quarter of the ticks" bound
and the ">= 2 intervals for Frozen" check are gone. The cap check stays,
with generous slack (5 budgets + 5.5 s).

The sampler's values, against the oracle. The new test
the_probe_sampler_reads_each_cpu_rails_own_value brackets each of three
probe samples between two oracle reads on the oracle's own all-channel
subscription. It matches each rail by name and ordinal in the Energy Model
group:

- where the oracle saw the rail hold, the probe's value must equal it;
- where the rail moved, the value must lie between the two reads.

This catches a sampler that returns 0, a sum over rails, or another rail's
value. the_probe_samples_the_energy_model_group_alone also checks every
sampled name with is_cpu_rail, so the FFI filter is pinned on any host
(round-3 nit).

The oracle reads for its window, and on past it until it has two reads,
never past four windows. Fewer than two reads is a test failure with a
message, never CpuRailMoved(false). Its load thread runs until the oracle
stops reading.

Pure pins, each its own test:

- a decrease between consecutive samples is not an advance;
- a rail new in a sample only sets its baseline;
- probe_host is the open-watch-map path the real probe now goes through. A
  subscription that cannot open, or never yields a sample, goes through
  status_of to Unavailable, and is never watched;
- status_of(None) is Unavailable.

The once-per-process cache is pinned process-wide. Six threads, released by
one barrier, ask for the status and build the backend at once, and the
static PROBE_RUNS counter must read 1.

The test that prints "ioreport liveness: status=" keeps its name, and no
test was renamed.

Evidence, from one lib test binary of this tree (55 tests):

- M5 Max (macOS 27.0.1), 10 runs under 36 busy-loop spinners (2 x
  hw.ncpu), load averages 124-149 (other sessions included). 10/10
  passed. Every run printed:
  'ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
  selected=false available=false'
  - median sample cost: 6.93-9.01 ms;
  - intervals per probe: 11-13;
  - least burn in one interval: 25.0009-25.0015 ms;
  - oracle reads: 7-12.
  Afterwards all 36 spinner PIDs were confirmed gone.
- M3 Ultra (macOS 27.0.1), 1 run: 55 passed; median 8.28 ms over 30
  calls; 29 intervals; least burn 25.0003 ms; 17 oracle reads; both dies
  equal to the held oracle values.

Mutations, each restored with git checkout (git status clean after each).
Every one fails a test on the M5 Max except the one marked:

- a decrease counts as an advance: 2 failed;
- a rail new in a sample counts as an advance: 2 failed;
- no subscription maps to Live: 1 failed;
- per-thread cache: 1 failed ('left: 16 right: 1');
- sampler returns 0: 1 failed ('held at 1863990848 ...; the probe read 0');
- sampler reports the GPU rail as CPU Energy: 5 failed;
- FFI filter also keeps the GPU rail: 6 failed;
- spin sleeps: 4 failed ('a 25ms tick used 8.875µs');
- the real tick sleeps: 2 failed ('interval 0 of 26: the probe burned
  8.541µs');
- spin is a no-op: 4 failed;
- the real tick is a no-op: 2 failed;
- the tick counts wall time: 3 failed;
- all-channel subscription: 2 failed ('median sample call took
  481.672458ms');
- None maps to CpuRailAbsent: 4 failed;
- oracle cut to 1 ms: 2 failed ('the oracle got 1 read(s)');
- PROBE_TICK = 100 ms: 1 failed;
- sampler gives each CPU rail the sum of all CPU rails: EQUIVALENT on the
  M5 Max, which has one CPU rail (55 passed). On the M3 Ultra it fails:
  'DIE_0_CPU Energy #0 held at 28237793 ...; the probe read 59082835'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-ui-interaction examples: each interaction is sealed with its own span's figure, never 0 J for work that ran ... 20ad7bc386

The round-3 review found that real_meter sealed "0 µJ [Estimator]" for each
~150 ms interaction that ran. It sampled PowerMeter::sample_uj, a sum of
raw counters that is None wherever no hardware probe exists (every macOS
27.0.1 host checked), and run_interaction turns None into
Energy::estimator(0). The printed labels were already honest; the sealed
records were not. jcp_metered had the same defect, and for the same reason
it debited nothing, so its 15 J grant never bound.

No API change was needed. joule-ui-serve already seals served interactions
through a `Meter` over PowerMeter spans (its SpanMeter): each sample closes
the span the previous one opened, so two consecutive samples differ by
exactly one span's figure, and `provenance()` returns that span's own. The
same adapter now lives in examples/support/span_meter.rs, which both
examples include with #[path]. Each interaction is sealed as:

- `HwShunt` only when its span was a hardware reading whose CPU counter
  advanced;
- otherwise the meter's `Estimator` figure, wall-clock × TDP for its span.

jcp_metered also charges with the right provenance. Authorizer::charge
carries only a number, so the example's authorizer reads the provenance
from the span meter it shares with run_interaction: the span just closed,
whose figure it is being charged. It hands the JCP runtime
Energy::estimator (or modeled, or measured) accordingly. It no longer
wraps every figure in Energy::measured. The summary now reports the sum of
the sealed figures and, separately, what the grant was actually debited:
the runtime never debits past the budget.

Output on this M5 Max (CpuRailFrozen):

- real_meter:
  - 'render #1: allow · 3003067 µJ [Estimator, an estimate]' (and #2, #3);
  - 'sealed chain: 3 links verified, 9008244 µJ total — estimated
    (wall-clock × TDP over each interaction's span), not measured'.
- jcp_metered:
  - 'apply #1: allow · 3002215 µJ [Estimator, an estimate] · grant
    11997785 µJ left';
  - through 'apply #5: ... grant 0 µJ left';
  - 'apply #6: deny · not run';
  - '5 allowed; 15012116 µJ sealed, estimated ...; 15000000 of the
    15000000 µJ budget debited; 1 denied by the real JCP runtime once the
    budget was spent.'

Left for David, unchanged:

- Authorizer::charge carries no provenance;
- the Meter trait's provenance defaults to HwShunt;
- run_interaction seals a None sample as Estimator(0).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 14:32:47 -04:00

ci(standards-build): run all five IOReport meters on macOS, and fail a step whose test never ran (merge LAST) #102

dcharlot added 1 commit 2026-09-30 14:38:30 -04:00

joule-code-remediate: a fix is shipped only when it is committed ... 41d804348f

The ship step ran `git checkout -b`, `git add` and `git commit` and discarded
all three results. Two consequences, both pre-existing on main:

- A failed `commit` still reported `shipped: true` with a branch name while
  the verified fix sat uncommitted in the work tree. Any environment with no
  git identity does this: the CI container for joule-code-rs has none, which
  is how this surfaced. The two-run test added in d7adbe6c failed there with
  `Err(Dirty)` on its second run, because the first run's "shipped" fix was
  never committed. main's single-run tests never checked that a commit
  happened, so its CI has been green over the same silent failure (including
  the formal-obligation and L3-skill tests, which also ship).
- A failed `checkout -b` (the fix's branch already exists) let the commit
  land on whatever branch was checked out.

Now each step's result is checked. A failure returns the new
`HarnessError::Ship`, whose message says where the verified change was left
(uncommitted in the tree, or staged on the fix's branch), and nothing is
reported as shipped. No crate outside joule-code-remediate uses
HarnessError.

Tests:
- scratch_crate gives each scratch repo a repo-local identity, so the
  harness's own ship commit works in any environment.
- verified_fix_end_to_end_ships_with_receipt asserts that shipped means
  committed: HEAD is on the reported branch, one commit above the start,
  with a clean tree.
- a_fix_whose_commit_is_rejected_is_never_reported_shipped: a pre-commit
  hook that rejects makes run_once return Ship("git commit ..."), and HEAD
  does not move.
- a_fix_never_lands_on_the_current_branch_when_its_own_branch_exists:
  pre-creating joule-fix/<rev> makes run_once return Ship("git checkout -b
  ..."), with the current branch and HEAD unchanged.

Reproduced the CI failure locally by removing the git identity
(GIT_CONFIG_GLOBAL=/dev/null, GIT_CONFIG_NOSYSTEM=1, user.useConfigOnly=true):
before, a_second_run_seals_only_its_own_cargo_windows failed with
`Err(Dirty)` at the same line as in CI. After, all 13 tests pass both with
and without a git identity.

Falsified (each restored with git checkout):
- ignoring commit's result again fails the rejected-commit test;
- ignoring checkout -b's result again fails the existing-branch test;
- scratch repos without the repo-local identity, run with no identity, fail
  the four shipping tests with Err(Ship) rather than passing over an
  uncommitted fix.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 17:51:38 -04:00

sandbox-meter: the status the meter is gated on is the run the tests drive, and no hardware pin rests on wall time ... 4b20ce2a50

Round-5 verification of 41d80434 (its sandbox/ tree is 20ad7bc3's) found
three kinds of problem in the IOReport liveness probe's tests.

- A flake on correct code. The R3 pin asserted that the probe's median
  sample call took under 50 ms of WALL time. When another process samples
  IOReport over every channel, a group-only sample call waits behind that
  process's 0.25-0.8 s samples. Beside one such reader, 3 of 6 unmodified
  copies failed (medians 85, 165 and 171 ms). The other backends' test
  oracles are such readers.
- Two dangerous-side survivors in production glue no test ran. The cached
  run was probe_status -> probe_host -> watch_cpu_rail, and it only ever ran
  against this host's real, frozen IOReport. Mapping a failed open straight
  to Live inside probe_status passed every test, and so did lifting a single
  advance (one batch refresh) to Live inside its closure.
- Fail-safe survivors: a failed open mapped to CpuRailAbsent inside
  probe_status, a no-op tick in the cached run, a sampler that returns a
  stale sample, and thread_cpu() reading CLOCK_MONOTONIC or
  CLOCK_PROCESS_CPUTIME_ID. The tick and its pins all read the same
  thread_cpu(), so a wrong clock passed them all.

What now holds.

One production chain, with no branch of its own. ioreport_status() caches
host_status() in a process-wide OnceLock, and nothing else in production
runs the probe. host_status bumps a process-wide AtomicUsize first, then
returns status_of(real_run().verdict()). real_run is liveness::probe_with
over the real opener (IoReportProbe::open, whose sampler is
IoReportProbe::read), the real tick (probe_tick), this thread's CPU clock and
a monotonic clock, for CPU_RAIL_LIVENESS_BUDGET. probe_with is generic over
all of those. A subscription that does not open is never sampled or ticked.
probe_status, probe_host, watch_cpu_rail, cpu_rail_raw and ProbeTrace are
gone. The counter is read through probe::ioreport_probe_runs()
(#[doc(hidden)]).

Selection only through the cached status and one pure gate. detect() is
detect_given(ioreport_status()). IoReportProbe::given(status) replaces
IoReportProbe::new(), which read the cache itself. Both ask
liveness::admits(status), true for Live alone. admits, given and
detect_given are each tested for all four statuses, with nothing probing.

The run records what it did. For every sample call: when it started and
ended on the run's clock, the thread CPU it used, how many channels the
sample held, whether all of them were in the Energy Model group, and each
GPU Energy rail's (name, raw). RailWatch keeps every CPU sample it was
handed (observed, which replaces prev). For every interval: the tick CPU,
measured apart from the samples.

Fault injection at the FFI leaf. A #[cfg(test)] thread-local script (mod
seam) sits inside IoReportProbe::open and IoReportProbe::read. It decides
whether the subscription opens and what each sample holds. Every other
thread, and every build that ships, reads the real IOReport. Tests call the
real, uncached host_status on the scripted thread:

- the open fails: Unavailable, one open, zero sample calls, no tick;
- one batch refresh (480,083,160 -> 541,893,638 mJ, then holding): Frozen;
- a rail that advances at every sample: Live after exactly three samples;
- a rail that holds: Frozen, watched for at least 0.9 s of the 1 s budget;
- two dies that each step once, in different intervals: Frozen;
- no CPU rail: CpuRailAbsent.

The script stamps every sample call on the tests' own clock. From those
stamps, each test checks that every tick burned at least 25 ms of this
thread's CPU (or ran into the 500 ms wall cap), and no more CPU than wall
time.

No wall-clock pin. the_liveness_probe_never_reports_a_frozen_rail_as_live
runs real_run itself, uncounted and uncached, and pins from its record:

- every sample held the Energy Model group alone, with as many channels as
  the all-channel oracle counts in that group (364 of 11,886 on the M5 Max,
  645 of 14,345 on the M3 Ultra);
- the median sample call used under 50 ms of this thread's CPU;
- every interval burned at least PROBE_TICK/2.

The median wall time is printed, not asserted.

The clock is pinned on the tests' own clock. The tests read
CLOCK_THREAD_CPUTIME_ID through their own literal (16, from macOS
<time.h>). thread_cpu() must not move over a 100 ms sleep, nor while
another thread burns 100 ms. It must track this thread's own 50 ms burn to
within 2 ms. probe_tick is checked five times with literal bounds.

Freshness. the_real_probe_hands_the_rule_each_cpu_rails_own_fresh_value
runs real_run while a helper thread reads the tests' own Energy Model
subscription over and over. Each sample is bracketed by a timeline read
that ended before it and one that started after it. It asserts:

- each CPU value the rule got, and each recorded GPU value, lies within its
  bracket, so it equals the bracket where the rail held;
- the CPU keys equal the Energy Model CPU rails the all-channel oracle lists;
- where the timeline saw GPU Energy rise between the run's first sample and
  its last, the run's own GPU values rose too.

Once per process, in a process of its own. The concurrent test moved to
tests/ioreport_probe_once_per_process.rs. Eight threads start behind a
barrier: half ask for the status first and half call PowerMeter::detect()
first, then each does the other. It asserts the real-run counter is 0
before and exactly 1 after (0 off Apple Silicon macOS).

class_for, checked against the channel names in 7aa42418 and a live dump
of the Energy Model group on both chips. Only `CPU Energy` (M5 Max) and
`DIE_0_CPU Energy` / `DIE_1_CPU Energy` (M3 Ultra) contain "CPU Energy". No
CPU sub-rail does (`MCPU*`, `PCPU*`, `PACC_*`, `*CPM*`, `DIE_n_EACC_CPU`,
`DIE_n_PACC*_CPU*`, `*DTL*`). So it is unchanged. Its test now covers every
name 7aa42418 lists, plus the domains that are no aggregate's summand.

No test was removed or renamed; the once-per-process test moved to its own
file under the same name. The frozen-rail test still prints
"ioreport liveness: status=".

Corrections to 41d80434's message:

- "The two-run test added in d7adbe6c": it was added in cee66773
  (`git log -S a_second_run_seals_only_its_own_cargo_windows`). d7adbe6c
  touches only sandbox-meter.
- "fail the four shipping tests": it fails five. Reproduced here with no git
  identity: "FAILED. 8 passed; 5 failed". The five are
  a_second_run_seals_only_its_own_cargo_windows,
  a_proven_formal_obligation_lets_the_fix_ship,
  verified_fix_end_to_end_ships_with_receipt,
  shipped_l3_fix_induces_a_skill_that_closes_the_recurrence_without_a_model
  and l3_fix_passes_the_same_gates, each Err(Ship("git commit on
  joule-fix/...: Author identity unknown ...")).

Evidence, on an M5 Max, macOS 27.0.1, unless noted.

- cargo test -p sandbox-meter: 73 lib tests and 1 integration test pass.
- 20 mutations, each applied, built, run on both test binaries, and restored
  with git checkout; the tree was clean after each. All 20 are caught.
  - Rule: comparing with the first sample (12 failures); a decrease counted
    as an advance (2); LIVE_RAIL_ADVANCES = 1 (19).
  - No subscription inside host_status: -> Live when the run did not open;
    -> Live when an open fails before the run; -> CpuRailAbsent.
  - A single advance lifted to Live: after the run in real_run; inside its
    sampler closure.
  - Ticks: a no-op tick passed by real_run; a no-op probe_tick.
  - A per-thread cache: caught by the integration test ("real probe
    runs=9").
  - Sampler: every value 0; stale per probe and process-wide, below the test
    seam (caught on hardware: "sample 0: GPU rail ... read 126274113687220;
    the timeline read 126274125371248 just before"); stale inside real_run's
    closure.
  - thread_cpu() on CLOCK_MONOTONIC ("advanced 101.266ms while this thread
    slept 100 ms") and on CLOCK_PROCESS_CPUTIME_ID.
  - admits() accepting CpuRailAbsent (3).
  - Span::stop's held-CPU rule removed (3).
  - run_attributed reverted to meter.is_hardware() (1).
- Contention: the lib tests 5x while a helper process sampled IOReport over
  every channel in a tight loop: 5/5 passed, with a median thread CPU per
  sample call of 7.4-8.3 ms. In the round-5 flake's own shape (3 copies at
  once beside that reader, 2 rounds), 6/6 passed. In one of them the median
  WALL time reached 58.7 ms, which the old pin would have failed, while the
  median thread CPU was 8.4 ms.
- M3 Ultra (macOS 27.0.1), the same binaries, 1x: "test result: ok. 73
  passed", "ioreport liveness: status=CpuRailFrozen
  oracle=CpuRailMoved(false) selected=false available=false", "30 samples,
  each 645 channels ... median thread CPU per sample call 7.621667ms", "29
  intervals, the least thread CPU burned in one 25.001125ms", and "8
  threads, status=CpuRailFrozen, real probe runs=1".
- Dependants pass:
  - jcp-gateway (12) and jcp-action-receipt (17 + 3);
  - joule-code-remediate, 13, with and without a git identity;
  - joule-ui-interaction (6) and joule-ui-serve (4 + 2);
  - sandbox-rs `cargo test --workspace`, 260 tests.
- clippy --all-targets --no-deps on sandbox-meter is clean.
- cargo check --tests passes for x86_64-unknown-linux-gnu, and for
  x86_64-apple-darwin with only the pre-existing unused_mut on
  `let mut probes`.

Not done, deliberately:

- CI does not run these hardware tests. standards-build runs sandbox-rs on
  the linux runner, where the integration test takes its non-macOS branch.
  The macOS runner reaches sandbox-meter only through joule-ui-rs. Moving
  the job is a workflow change and is out of scope here.
- The seam scripts open() and read(), the liveness path. snapshot(), the
  span path, still reads IOReport directly. In production it runs only
  after a Live status admitted the probe, which neither host here reaches.
- A per-rail sum inside the FFI read is invisible on a one-rail M5 Max. The
  scripted two-die tests pin the pure split, and only an M3 Ultra run
  covers the FFI.
- Still left for David, unchanged from earlier rounds: jcp-gateway `_meta`
  measured_uj, Authorizer::charge without provenance, the Meter trait's
  default HwShunt, the macOS CI job, and rust_string's 128-byte buffer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 18:51:22 -04:00

sandbox-meter: a rail is live only if it keeps advancing, never on a two-step refresh ... d35c99affa

The CPU-rail liveness rule said Live once the same rail advanced in two
separate sample intervals. On 2026-09-30 the jouleclaw-energy verifier, on
the M5 Max under macOS 27.0.1, showed that two advances can be one batch
refresh. A group-only watcher sampling every ~16 ms saw `CPU Energy` go
2,643,843,378 -> 2,705,607,863 (+61,764,485) and then, 64 ms later,
-> 2,705,611,298 (+3,435); `MCPU0`, `MCPU1` and `PCPU` each stepped twice
with it. This probe samples about every 31 ms, so a probe window that holds
such a refresh sees two advancing intervals and says Live, cached for the
process. Two real probes did exactly that while the independent oracle saw no
sustained movement, coinciding with a refresh. Refreshes are ~30 minutes
apart, so this is rare, but it is the failure the rule exists to prevent: a
Live status selects IOReport for the life of the process, and every later
refresh lands whole in whichever span straddles it, sealed as a measurement.

The rule now asks whether one rail KEEPS advancing. Live iff the same rail
(name + ordinal) advanced in at least MIN_LIVE_ADVANCES = 5 separate
intervals, those advancing intervals are at least 80% of all intervals
observed so far (advancing * 5 >= intervals * 4), and the samples so far span
at least MIN_LIVE_WINDOW = 300 ms of wall time from the first sample. It is
evaluated after every interval and latches Live as soon as all three hold.
A refresh of fewer than five steps can never pass; one of five or more steps
inside 150 ms cannot pass at any steady sampling pace (five advances in
150 ms need intervals of 37.5 ms or less, and then 300 ms holds eight
intervals or more, of which the burst advanced in under two thirds). A live
rail at the probe's idle cadence proves itself at the tenth interval, ~310 ms.

The rule needs each interval's wall time, so RailWatch::observe takes the
wall-clock reading with each sample, and the loop (`watch`) passes its clock
read just after the sample returned; RailWatch records them in `at`.
LIVE_RAIL_ADVANCES (2) becomes MIN_LIVE_ADVANCES (5), with MIN_LIVE_WINDOW
and a pure `sustained(advances, intervals)` beside it. The loop keeps
sampling until Live, or until the 1 s budget has elapsed AND at least five
intervals were observed; the hard cap stays five budgets. Frozen, Absent and
Unavailable map exactly as before. Unchanged: per-rail keys, reading each
interval against the previous sample, a decrease or a first appearance not
being an advance, one advance per rail per interval, the group-only
subscription, the thread-CPU tick, the pure status mapping, the OnceLock
cache, the F9 fault seam, the Span::stop and run_attributed rules, and the
"ioreport liveness: status=" line.

The test oracle's "moved" verdict had the same hole: CpuRailMoved(true) on
two changes, so the refresh that fooled the probe could also vouch for it.
It now says moved only if one rail changed at five reads or more, at four
reads in five at least (oracle_saw_it_keep_moving, its own literals). It
reads for its whole window and past it until it has five reads, never past
four windows, and still fails the test on fewer. Under contention it did
extend: two runs took 5 reads over 3.64 s and 3.19 s of a 3 s window.

Tests added (scripted clock, ~31 ms an interval unless stated). Those marked
(*) fail under the old two-advance rule, as the old-rule run below shows.

- (*) the_observed_two_step_refresh_is_frozen: [a,a,a,a+61764485,
  a+61764485+3435], then flat for a 1 s budget, and the same with the steps
  two intervals apart: Frozen, never live at any interval.
- (*) a_refresh_in_three_to_five_steps_at_the_start_is_frozen: three, four
  and five steps in consecutive intervals inside 150 ms: Frozen.
- (*) an_m3_ultra_refresh_stepping_twice_per_die_is_frozen: DIE_0 and DIE_1
  each stepping twice within 64 ms, in step and apart: Frozen.
- a_rail_advancing_every_interval_is_live_not_before_five_advances_and_300_ms:
  live first at interval 10 at 31 ms, 5 at 60 ms, 5 at 75 ms (where 300 ms
  arrives at the fourth), and stays live.
- a_live_rail_missing_one_interval_in_five_is_live (80%, wherever the miss
  falls: interval 10 at 31 ms, 6 at 100 ms) and
  a_rail_missing_two_intervals_in_five_is_not_yet_live (60% never, and a
  rail that missed two of its first five is live only at the tenth, 8 of 10).
- a_loaded_host_with_a_live_rail_is_live_after_five_intervals: 100 ms
  intervals, live at the fifth (500 ms), not the third or fourth; the loop
  stops there with six samples and five ticks.
- two_rails_that_advance_in_turn_are_not_one_live_rail, for two dies and
  for twin channels, so per-rail keying still decides a verdict now that two
  single steps no longer could.
- Keep-sampling: the_probe_samples_until_live_or_long_enough_to_tell and
  the_probe_loop_needs_the_budget_and_five_intervals_to_stop (continues past
  the budget while fewer than five intervals, stops at five, stops at the
  cap with three); the loop hands the rule each sample with the clock just
  after it (the_probe_loop_stops_as_soon_as_a_rail_is_live pins `at`).
- the_oracle_says_moved_only_for_a_rail_that_changed_at_four_reads_in_five.

Real-host-composition (F9) tests, through host_status on a scripted
IOReport: the "two advances -> Live" script is now a rail advancing at every
sample (Live after 6..=14 samples, and a scripted real_run shows from its own
record that it stopped at the first interval the rule allows); new
the_production_status_of_the_observed_two_step_refresh_is_frozen
(CpuRailFrozen, watched >= 900 ms of the budget); the two-dies test now has
the dies advancing in turn for the whole budget, so a sampler that summed
the dies would be called Live; frozen-status tests expect five intervals.
The same in-turn change was made to the probe_with sum test.

Mutations, each a temporary edit restored with git checkout (tree clean
after each):
- the old rule (MIN_LIVE_ADVANCES = 2, no fraction, no window): 17 failed,
  including all three (*) tests and the F9 two-step refresh ("a two-step
  refresh was called Live").
- MIN_LIVE_ADVANCES = 2: 6 failed (e.g. the loaded host went live at the
  third interval, Some(3)).
- fraction check removed: 5 failed (five-step refresh live at interval 10;
  in-turn dies live; 60% rail live).
- window check removed: 7 failed (five-step refresh live at interval 5).
- oracle back to ">= 2 changes": 1 failed, the oracle verdict test.

Runs. sandbox-meter lib 83 passed, integration 1 passed, on this M5 Max
(macOS 27.0.1). Three runs of the lib and integration binaries beside an
all-channel IOReport reader (~337 ms a sample): 0 of 6 failed, each
"ioreport liveness: status=CpuRailFrozen oracle=CpuRailMoved(false)
selected=false available=false". The M3 Ultra (macOS 27.0.1), same binary:
83 passed, "ioreport probe: probed=CpuRailFrozen cached=CpuRailFrozen
oracle=CpuRailMoved(false) took=1.240480083s intervals=28
window=1.002887292s rails=[("DIE_0_CPU Energy", 0, 0), ("DIE_1_CPU Energy",
0, 0)]". Consumers: jcp-gateway + jcp-action-receipt 32 passed,
joule-code-remediate 13 passed, joule-ui-interaction + joule-ui-serve 12
passed, the rest of sandbox-rs 186 passed. clippy --no-deps on
sandbox-meter, all targets: no warnings on aarch64-apple-darwin or
x86_64-unknown-linux-gnu (x86_64-apple-darwin shows one pre-existing
unused-mut in detect_given, untouched here).

Not done here, deliberately:
- A refresh of five or more steps whose last interval is stalled well past
  its tick (a heavily loaded host) could still meet all three conditions;
  no such refresh has been seen, and the observed ones have two steps.
- The same rule in the four sibling meters is being changed separately.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 19:00:00 -04:00

jcp-energy: IOReport only when its CPU rail keeps advancing, never on a batch refresh #110

dcharlot changed title from WIP: sandbox-meter: a CPU counter that does not keep moving is never a measurement, and no consumer relabels an estimate to sandbox-meter: a CPU counter that does not keep moving is never a measurement, and no consumer relabels an estimate 2026-09-30 19:00:01 -04:00

dcharlot referenced this pull request 2026-09-30 19:00:05 -04:00

proof-cascade: never meter CPU work through an IOReport CPU rail that does not keep moving #115

dcharlot referenced this pull request 2026-09-30 19:00:07 -04:00

eoc-meter: never build the IOReport counter on a CPU rail that does not keep moving #116

dcharlot referenced this pull request 2026-09-30 19:00:09 -04:00

jouleclaw-energy: IOReport only when its CPU rail keeps moving; consumers never zero-fill or relabel #117

dcharlot referenced this pull request 2026-09-30 20:01:17 -04:00

specs: a vendor counter is a hardware reading only while it is demonstrably live #119

dcharlot added 4 commits 2026-09-30 20:10:47 -04:00

jcp-gateway: the plaintext figure beside a receipt says whether it is a reading ... f2f23c2dce

An allowed `tools/call` comes back with `_meta["science.transaction/jcp"]`:
the sealed receipt, `measured_uj` and `budget_remaining_uj`. Since this branch
made the gateway seal each call's real provenance, `measured_uj` is the call's
cost, which is a hardware reading only where the span was one. On a host with
no live energy counter (every macOS 27 host checked) it is the meter's
estimate, and the plaintext said nothing about which: the grade existed only
inside `receipt_b64`, so a client that read the number without decoding the
COSE envelope took an estimate for a measurement.

The key keeps its name, because clients already read it. Beside it the block
now carries `energy_provenance` (`HwShunt`, `ModelBased` or `Estimator`) and,
when the figure reads only part of the call (a GPU counter with no CPU
record), `energy_coverage: "Partial"`: the same two labels, with the same
omit-when-Whole rule, that the sealed receipt carries. `attach_receipt` takes
the cost `Energy` instead of its bare `u64`, so the label cannot drift from
the figure. The change is additive: nothing is removed from the block, the
receipt bytes are unchanged, and no published JCP or AR-1 vector moves.

AR-1's `joules_uj` was documented as "Measured cost in microjoules". It is the
work's cost however it was obtained, graded by `energy_provenance`; the field
doc, the family table ("measured joules") and AR-1-spec §2.2 now say so, and
§9's energy cross-check says it holds only where the figure is `HwShunt`.
Doc-only: the encoding and the pack are untouched.

Tests. `the_caller_gets_its_receipt_back` now decodes the receipt and asserts
that the plaintext `energy_provenance` and `energy_coverage` equal what it
sealed, and that a non-hardware meter's call is labelled `Estimator`; on this
host it observes `"energy_provenance":"Estimator","measured_uj":26577`. A new
`the_plaintext_figure_carries_its_grade` covers every cost shape: Estimator,
HwShunt whole, HwShunt Partial, ModelBased.

Falsified: removing the `energy_provenance` insert fails both tests
(`energy_provenance beside measured_uj`; `left: None, right:
Some("Estimator")`). jcp-gateway 13/13, jcp-action-receipt 17 + 3 conformance
pass, and `jcp-conformance` reports all checks passed. Clippy is clean on
jcp-gateway; its one jcp-action-receipt warning is in the untouched
`ar-verify` binary.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-ui: a meter is an estimator until it opts in, and a charge carries its grade ... 7609bfeaef

Three places where joule-ui let a figure be read as more than it was.

The `Meter` trait defaulted `provenance()` to `HwShunt`. Any meter that did
not think to override it, a deterministic stub included, had its figures
sealed as hardware readings with `measured_uj` set. The default is now
`Estimator`: a meter opts in to `ModelBased` for a model and to `HwShunt` only
for readings of a hardware counter it has seen to be live. In this workspace
the only meters that are hardware readings are the two `SpanMeter` adapters
(the examples' and joule-ui-serve's), and both already return their span's
own grade, so nothing that reads hardware changes. The stubs in the
joule-ui-interaction, joule-ui-serve and joule-ui-mcp tests and the
`metered_surface` example now come out `Estimator`, as they should.

The conformance pack's `StepMeter` now declares `HwShunt` explicitly. C2 and
C4 were sealed with it standing in for a hardware counter, so they carry
`measured_uj` and exercise that path of the signed core; the explicit
override, documented as a fixture grade that attests nothing (like the
pack's steward key), is what keeps them byte-identical. SPEC §5 and §8 and
the conformance README say so.

`Authorizer::charge(action, measured_uj: u64)` handed the authorizer a bare
number, so an authorizer that keeps its own record of the debit (a JCP grant)
could not tell an estimate from a reading; the `jcp_metered` example worked
around it by reading the provenance back off the meter. `charge` now takes
`&Energy`, the exact figure the interaction is sealed with, provenance
included. Every implementor and caller in the workspace is updated, and
`jcp_metered`'s authorizer maps the charged figure's own provenance onto
`jcp_schema::Energy` instead of holding an `Rc` to the meter.

`run_interaction` turned a missing sample into `Energy::estimator(0)`. Work
that ran was sealed at 0 µJ and charged nothing, so on a host with no reading
a budget never bound, which SPEC §6 says is exactly what the budget is for.
Now, when either sample is `None`, the handler's own wall-clock time is
charged at the meter's `estimator_milliwatts()` (default
`DEFAULT_ESTIMATOR_MILLIWATTS` = 20 W, the Sandbox estimator's default),
rounded up and never 0 (`estimate_uj`), graded `Estimator`. A denial, whose
handler never ran, still records 0 µJ. The signed core and its profile tag
are unchanged.

SPEC: §5 states the default and carries the shared vendor-counter liveness
MUST verbatim (HwShunt only while the counter is demonstrably live for the
domain it claims; for IOReport the CPU rail advancing in >= 5 intervals, in
>= 4 of every 5, over >= 300 ms while the probe burns CPU; a frozen or lumped
counter never yields a hardware-graded figure; no claim of a domain the
counter does not read). The "real RAPL/IOReport" wording at §5 and "the
hardware Meter" at §9 now describe the reference meter as a hardware reading
while a vendor counter is live and its labelled estimate otherwise. §6.2
specifies the no-reading estimate and forbids recording work that ran at
0 µJ; §6.3 becomes "charge the cost, with its provenance".

Tests (joule-ui-interaction, new): `a_meter_is_an_estimator_unless_it_opts_in`,
`work_that_ran_without_a_reading_is_estimated_never_zero`,
`the_estimate_is_wall_time_at_the_declared_power_rounded_up`,
`a_budget_binds_on_a_host_with_no_reading` (a 1 J budget stops ~20 ms
interactions at 20 W within three, every debit `Estimator`), and
`charge_is_handed_the_sealed_figure_and_its_provenance`. All 31 tests across
joule-ui-interaction, -conformance (incl. the published-pack tests), -serve
and -mcp pass; the workspace builds with all targets; the three examples run
(`jcp_metered` here: five ~3 J `Estimator` interactions spend the 15 J grant
and the real JCP runtime denies the sixth). `gen_vectors` regenerated into a
scratch directory is byte-identical to `conformance/v1/` (all five files,
MANIFEST included).

Falsified, each restored and re-run green:
- default back to `HwShunt` with the fixture override removed: the vectors
  regenerate identical (under the old default they rested on the default),
  and `a_meter_is_an_estimator_unless_it_opts_in`,
  `charge_is_handed_the_sealed_figure_and_its_provenance`, joule-ui-serve's
  `firing_an_action_meters_seals_and_returns_the_next_surface` and
  joule-ui-mcp's `an_mcp_app_action_is_metered_sealed_and_attributed` fail;
- new default, override removed: `c2_interaction_is_reproduced_from_the_pack`
  and `c2_interaction_is_metered_and_allowed` fail, and the regenerated c2,
  c4 and MANIFEST differ (`"measured_uj": null, "provenance": "Estimator"`),
  so the override is what keeps the pack;
- `charge` handed `Energy::measured(uj)` regardless of grade:
  `charge_is_handed_the_sealed_figure_and_its_provenance` and
  `a_budget_binds_on_a_host_with_no_reading` fail;
- the no-reading arm back to `Energy::estimator(0)`:
  `work_that_ran_without_a_reading_is_estimated_never_zero` and
  `a_budget_binds_on_a_host_with_no_reading` fail.

Clippy --no-deps on the changed crates adds nothing; the remaining warnings
are in untouched code (joule-ui-core's tests, the conformance pack test).

Not done here: joule-ui's energy figure has no coverage field, and both
`SpanMeter` adapters grade a span whose only records are hardware GPU
records (NVML without a CPU probe) `HwShunt`, which claims the whole
interaction. §5's coverage clause forbids that; fixing it needs either a
coverage field in the signed core (a new profile tag) or a downgrade rule,
which is a decision, not a cleanup.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

sandbox: the README's metering guarantee says HwShunt only while a counter is live ... 1b7a61692a

The README's metering guarantee said the reference meter reads IOReport,
RAPL and NVML "each tagged `HwShunt`", and that "where no counter exists the
reading degrades to a labeled estimator, never a fabricated measurement". On
macOS 27 the IOReport CPU energy rails exist but are frozen, or republished in
lumps; read at face value, the sentence promised `HwShunt` there, which is the
fabricated measurement it disclaims. The meter on this branch does not do
that, so the README was describing the old behaviour.

It now says what the meter does: a CPU reading is `HwShunt` only while the
CPU counter is demonstrably live; IOReport is used only if its CPU energy rail
keeps advancing while the probe burns CPU (in at least 5 sample intervals, at
least 4 of every 5, over at least 300 ms), and a span across which the CPU
counter held is never a reading. A counter that is absent, or present but not
live, yields no hardware-graded figure: the span is the labelled `Estimator`
(wall-clock × TDP), and `ioreport_status()` says why an Apple Silicon host is
estimated. The heterogeneity paragraph's "`HwShunt` where the device exposes
an energy interface" becomes "a live energy interface" to match.

README only: no code, spec text or vector changes. sandbox-rs tests pass
(`cargo test --workspace`), and the vanity-metrics scanner is clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-code-remediate: the description grades its joules, and the z3 tests never pass silently ... ccbec44ab2

The crate description promised "a JouleContract receipt with
hardware-measured joules sealed by a sandbox meter attestation". The crate
does not do that on a host without a live CPU energy counter (every macOS 27
host checked): there its receipts carry an estimate, sealed as JCR-1
`Estimator` / JouleContract `Unaccounted`, as its own module doc says. The
description now says the joules are graded by how they were obtained: a
hardware reading apportioned to the run and anchored to a sealed meter
envelope only where the host exposes a live CPU counter, an estimate sealed
as one everywhere else. Two `FormalEntry`/`FormalOutcome` docs that called
the proof cascade's `ModelBased` accounting "measured joules" are corrected
too.

`a_proven_formal_obligation_lets_the_fix_ship` and
`a_refuted_formal_obligation_reverts_the_fix` returned before asserting
anything when `z3` was absent, so on a solverless host (the joule-code-rs CI
job installs none) they passed in 0.00 s whatever the formal gate did. They
now follow the convention proof-solvers and joulecontract-proof already use,
and that the standards-build workflow's "Install z3" step documents: the
suite passes with or without a solver and asserts something either way. With
`z3` the solver's verdict is asserted as before. Without it, both assert the
gate fails closed: the obligation is Unproved (neither proved nor refuted),
the `formal-proof` gate fails while every other gate passes, the fix is
reverted and not shipped, and the discharge evidence is still written. Each
logs which branch it asserted. Where a solver is supposed to be installed,
`JOULE_FIX_EXPECT_Z3=1` turns a missing `z3` into a test failure instead of
the solverless branch; this is documented on the test helper and in
Cargo.toml beside the proof dependencies. No workflow is changed, so CI
still runs the solverless branch until a job installs z3 and sets the
variable.

The formal gate's detail, which a failed gate copies into its receipt entry,
and joule-fix's summary line both said "real z3" even with no solver on
PATH. They now say "no z3 on PATH: SMT goals left undecided" in that case.

Verified, 13/13 each way: with z3 on PATH; with z3 removed from PATH; with no
git identity (GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 and
user.useConfigOnly=true); and with no identity and no z3.
`JOULE_FIX_EXPECT_Z3=1` with no z3 fails both tests ("JOULE_FIX_EXPECT_Z3=1,
but there is no working `z3` on PATH"); with z3 present it passes.

Falsified: letting an unproved obligation pass the gate (`all_proved` without
`unproved == 0`) fails both tests with no z3 on PATH. The same mutation
against the previous tests passes both, in 0.00 s. Clippy --no-deps on the
crate is clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request from a commit 2026-09-30 20:12:10 -04:00

ci: run joule-code-remediate's formal gate against a real z3

dcharlot added 2 commits 2026-09-30 20:47:43 -04:00

sandbox-meter: RAPL and NVML are hardware readings only while every counter they sum keeps advancing ... 6b693aa46f

The shared rule for vendor energy counters is that a reading is a hardware
reading only while the counter is demonstrably live for the domain the figure
claims. IOReport has been held to it since d35c99af. The Linux probes were
not: `RaplProbe::new()` and `NvmlProbe::new()` were selected the moment
`energy_uj` or `nvmlDeviceGetTotalEnergyConsumption` answered once, so a
counter that held, or was republished in lumps, would seal zero or a stale
refresh as a `HwShunt` figure.

Both are now gated at construction by the same sustained-advance rule, probed
once per process and cached (`rapl_status()`, `nvml_status()`, a new public
`CounterStatus` of `Unavailable`, `Frozen` or `Live`). A counter is live only
if it advanced in at least five sample intervals, at least four in five of
those observed, over at least 300 ms, while the probing thread burned CPU,
each interval read against the sample before it. One difference from IOReport
is deliberate: these probes sum their rails into one figure per class (every
RAPL package into `Cpu`, every NVML GPU into `Gpu`), and a sum claims every
rail in it, so EVERY rail must keep advancing (`Quorum::Every`). IOReport's
liveness stays one CPU rail's (`Quorum::Any`). The sampling interval is the
probe's choice, not part of the rule: RAPL ticks at IOReport's 25 ms, NVML at
150 ms, because NVML does not document how often its counter is published and
a live counter updated every 100 ms advances in only about one 25 ms interval
in four. A host where powercap is unreadable or NVML is absent is
`Unavailable` at once, with no burn.

The rule, the run and the gate are pure and live in `liveness`, which is now
compiled on Linux as well; `rapl.rs` and `nvml.rs` only open their counters and
read raw integers. `RaplProbe::given` and `NvmlProbe::given` are
`liveness::gated` over the cached status and nothing else, and
`detect_given` takes a `Statuses` (IOReport, RAPL, NVML) so every gate is
tested for every status without probing. libc becomes a Linux dependency for
the thread CPU clock the tick burns against (Cargo.lock unchanged).

Two small additions serve callers that grade a whole unit of work by one
figure (joule-ui, next commit): `Measurement::reads_cpu()` (a `Cpu` record that
is `HwShunt` with a measured figure) and `PowerMeter::estimate(wall)` (the
single estimator record `Span::stop` itself gives a span with no reading). A
GPU-only span is honest per class here; it is not a reading of work that ran
on the CPU, and a format with no coverage field must not grade it as one.

Tests. The scripted ones run on every platform (macOS here):
`a_counter_whose_only_rail_keeps_advancing_is_live` (a RAPL package moving
every ~1 ms is live at the 12th 25 ms interval, and the probe stops there),
`a_counter_with_one_rail_that_held_is_frozen` (and the same samples ARE live
under IOReport's one-rail quorum), `a_counter_whose_every_rail_keeps_advancing_is_live`,
`a_counter_that_steps_in_a_burst_is_frozen` (1 to 5 steps),
`a_counter_that_wraps_once_is_still_live_and_one_that_falls_is_not`,
`a_counter_that_cannot_be_opened_or_read_is_unavailable` (never sampled or
ticked when it does not open), `an_nvml_counter_is_judged_at_its_own_tick` (a
100 ms counter is declined at 25 ms and live at 150 ms; a 400 ms one is
declined), `a_gated_counter_is_opened_only_when_live`,
`only_a_live_counter_admits_its_probe`, `a_counter_tick_burns_its_own_length`,
`the_real_counter_tick_burns_this_threads_cpu`,
`detect_selects_rapl_and_nvml_only_for_a_live_status`,
`the_counter_statuses_explain_themselves`,
`a_span_reads_the_cpu_only_with_a_hardware_cpu_record`, and a new test file,
`counter_probes_once_per_process` (eight threads; each real probe runs once on
Linux, never elsewhere). Linux-only, against an independent oracle that reads
every counter's absolute value before and after three budgets:
`the_rapl_rung_is_never_selected_over_a_counter_that_held`,
`the_nvml_rung_is_never_selected_over_a_counter_that_held`, and the two
`given` gate tests. Those compile here (`cargo check --tests --target
x86_64-unknown-linux-gnu`, and clippy for that target is clean) but have not
run: no Linux host was used, so they first run on the Linux CI runner.

sandbox-meter: 97 lib tests + both once-per-process files pass on this M5 Max;
every sandbox-rs crate that depends on the meter passes (225 tests); clippy is
clean for aarch64-apple-darwin and x86_64-unknown-linux-gnu.

Falsified, each restored from git and the suite re-run green: counter_run on
IOReport's one-rail quorum, and the Every quorum counting any rail, fail
`a_counter_with_one_rail_that_held_is_frozen`; a gate that admits every
status fails `only_a_live_counter_admits_its_probe` and
`a_gated_counter_is_opened_only_when_live`; `gated` that opens without asking
fails the latter; NVML at the 25 ms tick fails
`an_nvml_counter_is_judged_at_its_own_tick`; `reads_cpu` ignoring the class
fails `a_span_reads_the_cpu_only_with_a_hardware_cpu_record`. Not falsifiable
here: a `RaplProbe::given`/`NvmlProbe::given` that bypassed `gated`, or a
`real_run` on the wrong tick, is caught only by the Linux tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-ui: a span with no CPU reading is sealed as an estimate of the interaction, never as HwShunt ... 7539c62e19

joule-ui's energy record carries one figure per interaction and has no
coverage field. Both `SpanMeter` adapters (joule-ui-serve's, and the
examples' `support/span_meter.rs`) graded a span by its records alone, so a
span from an NVML-only probe set (a Linux host whose only live counter is the
GPU's: a `Gpu` record, `HwShunt`, and no CPU record at all) was sealed
`HwShunt` for the whole interaction: the GPU's energy, passed off as a reading
of a handler that ran on the CPU. `Span::stop`'s held-CPU rule does not reach
it, because there is no CPU counter to have held.

The rule now: a span with no hardware reading of the CPU
(`Measurement::reads_cpu`) is sealed as the meter's `Estimator` figure for
that span's wall time (`PowerMeter::estimate`, wall x the meter's TDP, never 0
for work that ran). A span that reads the CPU passes through unchanged, with
any GPU record beside it (RAPL and NVML: the sum, `HwShunt`). A span that was
already the estimator's record is replaced by the same figure. The rule is
one function, `interaction_figure`, applied where each adapter closes a span.
The examples' adapter is now generic over the same `Spans` seam
joule-ui-serve's has (defaulting to `PowerMeter`), so it can be driven over
scripted spans, and a new integration test includes it by `#[path]`.

SPEC §5 gains the sentence the shared vendor-counter MUST implies for this
format: with no coverage field, a figure from a counter that does not read the
domain the work ran on (a GPU counter alone, for a handler on the CPU) MUST
NOT be graded `HwShunt` for the interaction. joule-ui-serve's Cargo comment no
longer lists NVML alone as a source of a `HwShunt` receipt.

Tests: `a_gpu_only_span_is_sealed_as_an_estimate_never_as_a_reading`, in
joule-ui-serve (through the real `Host`) and in joule-ui-interaction's new
`tests/span_meter.rs` (through the real `run_interaction` and the examples'
adapter): a scripted 8 ms GPU-only span (9,000 uJ `HwShunt`) is sealed as
160,000 uJ `Estimator` with no measured figure; a CPU+GPU span is sealed
`HwShunt` at their sum (10,500 uJ); a CPU-only span as itself; the chain
verifies. joule-ui-serve, -interaction, -conformance, -mcp, -core, -a2ui and
-render pass (72 tests); clippy adds nothing (one existing joule-ui-core
lint); `real_meter` and `jcp_metered` run as before on this M5 Max (every
interaction `Estimator`, the 15 J grant spent and the sixth denied).
`gen_vectors` regenerated into a scratch directory is byte-identical to
`conformance/v1/` (`diff -r` clean, MANIFEST included): the conformance
pack's `StepMeter` is not a `SpanMeter`, and nothing in the signed core
changed. Cargo.lock gains only the new sandbox-schema dev-dependency edge.

Falsified, each restored from git and re-run green: either adapter sealing the
span as it stands (`interaction_figure` returning it unchanged) fails its
`a_gpu_only_span_is_sealed_as_an_estimate_never_as_a_reading`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-09-30 21:09:53 -04:00

sandbox-meter: a live per-process CPU energy probe for in-process work, and a real reading for a direct child #121

dcharlot added 1 commit 2026-09-30 21:24:36 -04:00

sandbox-meter: IOReport is a hardware reading only while every CPU rail it sums keeps advancing ... 5e945eed4a

The IOReport liveness rule said Live as soon as ONE CPU rail sustained its
advances (`Quorum::Any`), but `snapshot` sums every CPU rail into the one
`Cpu` figure. On a multi-die part that is two rails, `DIE_0_CPU Energy` and
`DIE_1_CPU Energy` on an M3 Ultra, so one die live beside a frozen one would
have selected the probe and sealed one die's energy as the whole CPU's, as
`HwShunt`. RAPL and NVML were already held to every rail they sum
(6b693aa4); IOReport now is too.

The rule: Live only when every CPU rail present in the samples advanced in
at least five intervals, four in five of those observed, over at least
300 ms; Frozen otherwise; Absent with no CPU rail. A rail is present if it is
in any sample the probe read, the baseline included, and it is judged over
every interval observed. Before, a rail seen only in the first sample was
never counted, because rails were registered from the later sample of each
interval; now the baseline's rails are seen with no advance. That holds for
the RAPL and NVML runs too, which share the watch. With one rule left there
is no quorum to choose, so `Quorum` and `watch_with` are gone: `RailWatch` is
the rule, and `counter_run` is `watch` over the counter.

The oracle behind the real-IOReport tests now says "moved" only when every
CPU rail it reads changed at four reads in five or more, so a Live claim is
checked against every die. It reads exactly the probe's rails on both hosts:
one on this M5 Max, the two dies on the M3 Ultra.

Docs: the liveness and ioreport module docs, `CpuRail`, `RailWatch`,
`IoReportStatus` and its Display text, `ioreport_status`, the probe module
doc, and the sandbox README's metering guarantee.

Tests, all scripted (macOS here). The rule:
`two_dies_that_both_keep_advancing_are_live` (live at the tenth interval, or
at the fifteenth when DIE_1 misses its first three),
`one_die_that_keeps_advancing_beside_a_frozen_one_is_frozen` (the other die
held, stepped once, stepped five times, or advanced three intervals in five;
in either listing order; and twin `CPU Energy` channels),
`one_die_that_keeps_advancing_with_no_other_die_in_the_samples_is_live`, and
`a_die_in_only_some_samples_is_a_rail_the_watch_needs`. A whole probe run:
`a_probe_run_calls_two_dies_that_keep_advancing_live`,
`a_probe_run_calls_one_live_die_beside_a_frozen_one_frozen`,
`a_probe_run_calls_one_die_alone_that_keeps_advancing_live`. The production
status through the scripted IOReport seam:
`the_production_status_of_two_dies_that_keep_advancing_is_live`,
`the_production_status_of_one_live_die_beside_a_frozen_one_is_frozen`. The
single-rail tests are unchanged. `a_counter_with_one_rail_that_held_is_frozen`
asserted that the IOReport loop called its two-rail samples live; it now
asserts Frozen.

sandbox-meter: 106 lib tests and both once-per-process files pass on this M5
Max. The same lib test binary passes 106/106 on the M3 Ultra (macOS
27.0.1), where the real probe gave `CpuRailFrozen` with `DIE_0_CPU Energy`
and `DIE_1_CPU Energy` at 0 advances each in 28 intervals over 991 ms, and
the oracle saw 0 changes on both dies in 17 reads. The sandbox-rs crates that
depend on the meter pass (126 tests), as do jcp-gateway and
jcp-action-receipt (33), joule-code-remediate (13), and joule-ui-interaction
and joule-ui-serve (19). joule-ui `gen_vectors` regenerated into a scratch
directory is byte-identical to `conformance/v1/`. Clippy is clean for
aarch64-apple-darwin and x86_64-unknown-linux-gnu.

Falsified, each restored from git and the suite re-run green: the rule back
to any one rail fails the three one-live-die-beside-a-frozen-one tests (rule,
run, production status), `two_dies_that_both_keep_advancing_are_live`,
`a_die_in_only_some_samples_is_a_rail_the_watch_needs` and
`a_counter_with_one_rail_that_held_is_frozen`; not seeing the baseline's
rails fails `a_die_in_only_some_samples_is_a_rail_the_watch_needs`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 22:18:46 -04:00

sandbox-meter: the RAPL figure sums the package zones by name, never psys or dram ... 7cf1f0366c

`RaplProbe::open` took every top-level powercap directory named
`intel-rapl:<n>` as a package and summed them into the `Cpu` figure, and the
liveness probe's `raw_sample` watched every one of them. A top-level zone is
not necessarily a package. Many Intel hosts expose `psys`, the platform zone,
at the top level, and its count already includes the packages', so the
`Cpu` figure counted package energy twice and sealed it `HwShunt`; some
platforms put a `dram` zone at the top level, which is not CPU energy at
all. The liveness rule was judged over the same wrong set: a live `psys`
counted toward a package counter being live, and a held `dram` zone could
decline a host whose packages were live.

The fix is the rule proof-cascade (9334ec0c) and jcp-energy (25bfdcd9)
follow: the figure is the sum of every top-level zone whose `name` file
reads `package-<digits>`, every socket's, and nothing else. `psys`, `dram`,
a package's subzones and the second interface to the same packages
(`intel-rapl-mmio:<n>`) are neither summed nor probed; `raw_sample` reads
the opened packages, so every package must be live and only packages are
judged. A host with no package zone (only `psys`) does not open and is
`Unavailable` with no burn.

Every package or none. A package zone whose `energy_uj` did not read was
skipped, so a two-socket host could seal one socket's energy as the CPU's;
now the probe does not open, and neither does it when a top-level zone's
`name` does not read (it may be a package).

Wrap handling, checked per zone. Each package was already corrected at its
own range, but an unreadable `max_energy_range_uj` defaulted to `u64::MAX`,
so a wrap would add about 1.8e19 µJ to the cumulative count, `HwShunt`. The
probe is cumulative across spans, so a per-span "no figure" cannot be
expressed without a later span, bracketing the read from both sides,
coming back measured but short. So a package with no readable, non-zero
range does not open, and a fall from above the declared range (a counter
contradicting its range, the one unknown delta left) breaks the count for
good: that snapshot and every later one fail, and the meter estimates every
span. Snapshots now read every package before updating any.

To test this on macOS, the zone listing takes the powercap root
(`open_in(root)`, with `open()` the sysfs tree), the liveness run takes it
too (`run_over(root, tick)`, with `real_run` the sysfs tree and the CPU-burning
tick), and `rapl` is built under test on every platform, its Linux-only
parts (`given`, `open`, `host_status`) gated as before. Production builds
are unchanged off Linux.

Tests, on fake powercap trees in the temp directory, running on macOS here
and on Linux in CI:
- `the_probe_sums_and_samples_the_package_zones_only`: a two-socket tree with
  `psys`, a top-level `dram` zone, `core`/`dram` subzones and an MMIO zone.
  The probe holds exactly `intel-rapl:0` and `intel-rapl:2` with their own
  ranges; the liveness sample is those two; a span through
  `PowerMeter::measure` over which every zone rose is `Cpu` 8,000 µJ, the
  packages' rise alone; package 1 wrapping at its own 65,532,610,987 µJ
  range while `psys` and `dram` fall is 12,487 µJ.
- `the_liveness_run_watches_the_package_zones_only`: the run `rapl_status`
  uses, over the fake tree with a sleeping tick, is Live with both packages
  advancing and `psys`/`dram` holding, Frozen with `psys`, `dram`, the MMIO
  zone and package 0 advancing while package 1 holds; a `psys`-only tree is
  Unavailable and never ticked.
- `the_probe_never_leaves_a_package_out_or_takes_another_zone`: `psys` and
  `dram` with a `package-0` subzone, an unreadable package beside a
  readable one, an unnamed top-level zone, and a package with no or a zero
  range each fail to open, `Unavailable`, saying why.
- `a_package_wraps_at_its_own_range_and_a_contradiction_breaks_the_count`:
  the per-zone delta; a snapshot sequence through a wrap; a span opened at
  5,000 µJ, sampled mid-span at 10 (a fall from above the 1,000 µJ range)
  and closed at 6,000 is `Estimator`, as is every later read and span.
- `a_package_is_a_top_level_zone_named_package_n`: the matchers.
- The Linux-only oracle test against the real sysfs is unchanged.

Falsified, each restored through git before the next:
- selecting by directory name again (the `name` check disabled) fails
  three: the probe held `intel-rapl:0..3`, `psys` and `dram` among them; the
  run was Frozen ("both packages advancing, psys and dram holding, was not
  live"); the `psys`/`dram` tree opened `["intel-rapl:0", "intel-rapl:1"]`;
- skipping an unreadable package again fails the every-package test: the
  tree opened `["intel-rapl:0"]` beside an unreadable `intel-rapl:1`;
- an unreadable range defaulting to `u64::MAX` again fails it too: a package
  with no range opened;
- dropping the sticky break fails the wrap test: "a span over an unknown
  delta was measured: [... measured_uj: Some(1000) ... HwShunt]".

sandbox-meter (112 lib tests on macOS, 106 before: the six in `rapl` now run
here, five of them new; and both once-per-process tests) and the whole
sandbox-rs workspace pass on macOS (M5 Max, macOS 27.0.1), as does the
sandbox-conformance binary ("All classes passed with no skips");
jcp-gateway, jcp-action-receipt, joule-code-remediate, joule-ui-interaction
and joule-ui-serve pass, and joule-ui's `gen_vectors` regenerated into a
scratch directory is byte-identical to `conformance/v1/`. `cargo clippy
--all-targets -p sandbox-meter` is clean for aarch64-apple-darwin and
x86_64-unknown-linux-gnu. Not run on a Linux host with a real RAPL tree
here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 2 commits 2026-09-30 22:50:44 -04:00

sandbox-exec-microvm: the CPU figure is every package's, read only while every package counter is live ... 8a3465e18d

The S5 executor's two CPU meters broke the rule sandbox-meter's RAPL probe
follows since 7cf1f036, and each sealed what it read as `HwShunt`.

`RaplMeter` read one file, `intel-rapl:0/energy_uj`, through `at(path)` or
`intel_package0()`, the constructor `prove_microvm` uses. It never read the
zone's `name`, so nothing confirmed `intel-rapl:0` is a package, and on a
two-socket host the second socket's energy was left out of a figure sealed
as the CPU's. It had no wrap handling of its own: a wrap read as a fall,
which the executor's `a >= b` happened to turn into an estimate. And nothing
checked that the counter was live: one that held, or stepped only in a batch
refresh, was sealed as a reading of the run. `MsrRaplMeter` read
`MSR_PKG_ENERGY_STATUS` through `/dev/cpu/0/msr` alone, package 0's counter,
the same one-socket figure on a multi-socket host, also with no liveness
check.

`RaplMeter` is now sandbox-meter's own RAPL probe rather than a second copy
of it. sandbox-meter gains `RaplPackages`, its `RaplProbe` behind the
liveness gate with no ungated constructor: `host()` opens the sysfs tree only
when the cached `rapl_status()` is `Live` (and is `Unavailable` off Linux),
`live_in(root, tick)` runs the same liveness run over another tree and opens
only if it is `Live`, and `cumulative_uj()` is the probe's snapshot. So the
figure is the sum of every top-level zone whose `name` is `package-<digits>`
and nothing else, every package or none, each wrap-corrected at its own
`max_energy_range_uj`; a package with no readable range does not open, and a
fall from above a package's range breaks the count for good. `RaplMeter::host()`
replaces `intel_package0()` and `at()`, which are gone; an unopened meter
samples `None` and says why (`unavailable()`), and `prove_microvm` uses it.
To make that possible on every platform, `rapl` and `liveness` are now built
everywhere, not only on Linux and under test: both take their tree or their
sampler as an argument, and only Linux opens sysfs.

`MsrRaplMeter` now reads every package, and every die of a multi-die
package, since the register is per die there. It groups every CPU under
`/sys/devices/system/cpu` by `topology/physical_package_id` and
`topology/die_id`, reads each domain through the first of its CPUs whose
`msr` device reads both registers, scales each by its own energy unit, and
sums them. It does not open when a domain reads through none of its CPUs,
when a domain's unit is zero, or when a CPU has no readable topology: an
offline CPU's package is unknown and could be one no online CPU covers, so
the sum could not be shown to be the whole CPU's. That makes a host with
offline CPUs (`nosmt` included) estimated, which is the safe side. Each
domain's counter is 32 bits, so a fall is a wrap at 2^32 counts, its
architectural range; the reserved upper half is ignored. It opens only once
every domain's raw counter has kept advancing under the same liveness rule,
through a new public `sandbox_meter::probe::counter_liveness(open, tick)`
(with `counter_liveness_tick()`, the production CPU-burning tick), probed
once per process for the host. `MsrRaplMeter::host()` replaces `package0()`
and returns why it did not open; `boot_microvm` prints it.

Tests, on fake trees in the temp directory, on macOS here and on Linux in
CI. The MSR tree is a sysfs CPU directory beside fake `msr` devices holding
the two registers at their offsets.
- `the_rapl_meter_sums_every_package_zone_and_no_other`: two sockets with
  `psys`, a top-level `dram`, subzones and the MMIO zone, opened live; with
  every zone rising, a difference is the packages' 8,000 µJ alone; package 1
  wraps at its own 65,532,610,987 µJ range (11,487 µJ); a run through
  `MicroVmExecutor` whose VMM raises both packages and `psys` seals
  `Some(3000)`, `HwShunt`.
- `the_rapl_meter_reads_nothing_unless_every_package_is_live`: package 1
  holding beside a live `psys`, `dram`, MMIO zone and package 0 leaves the
  meter unopened, `None`, and the executor seals `ModelBased`; a `psys`-only
  tree and a package with no range never open and are never ticked.
- `a_fall_from_above_a_package_range_stops_the_rapl_meter_for_good`.
- `the_msr_meter_reads_every_package_and_die`: three domains, one read
  through cpu3 because cpu2's device is absent; 256, 512 and 16,384 counts at
  ESU 14 sum to 1,046,875 µJ with package 0's reserved bits changing; a run
  through the executor seals a joule, `HwShunt`.
- `the_msr_meter_wraps_each_domain_at_32_bits`: 512 counts across the wrap.
- `the_msr_meter_declines_what_it_cannot_cover`: a domain holding beside
  live ones; an offline CPU; no `die_id`; a domain none of whose CPUs read; a
  zero unit; no CPU; the unopened ones never ticked.
- sandbox-meter: `the_public_package_counter_opens_only_over_live_packages`
  and `counter_liveness_holds_a_callers_counter_to_the_every_rail_rule`.

Falsified, each restored from a saved copy before the next:
- `RaplMeter` reading `intel-rapl:0/energy_uj` alone again (the liveness
  gate kept) fails two: "not the two packages' rise alone, left: 3000,
  right: 8000", and "a fall from above the range gave a figure, left:
  Some(10)";
- the MSR meter keeping package 0 only fails three, "not one CPU per package
  and die" among them, and the holding-domain case opens;
- the MSR wrap as a saturating difference fails "512 counts across the wrap,
  left: 0, right: 31250";
- the MSR gate opening on a Frozen status fails the declines test ("opened
  [...] with package 1 die 1 holding");
- an offline CPU skipped instead of declining fails it too.

sandbox-exec-microvm (18) and the whole sandbox-rs workspace (308 tests)
pass on macOS (M5 Max, macOS 27.0.1), as does the sandbox-conformance binary
("All classes passed with no skips"); jcp-gateway, jcp-action-receipt,
joule-code-remediate, joule-ui-interaction and joule-ui-serve pass, and
joule-ui's `gen_vectors` regenerated into a scratch directory is
byte-identical to `conformance/v1/`. `cargo clippy --all-targets -p
sandbox-meter -p sandbox-exec-microvm` is clean for aarch64-apple-darwin and
x86_64-unknown-linux-gnu; sandbox-proof is clean on the host, and its Linux
cross-check needs a cross C compiler for zstd-sys, which this machine lacks.
Not run on a Linux host with a real RAPL tree or MSRs here.

Not done here: the executor seals a zero CPU delta between two reads of a
live meter as a measured `HwShunt` zero (`a >= b`), where sandbox-meter's
span estimates a CPU counter that held.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

joule-code-remediate: a test's scratch crate builds in its own target directory, whatever CARGO_TARGET_DIR says ... a16e36cf81

With `CARGO_TARGET_DIR` exported, four to eight of the crate's 13 tests
failed. The tests build scratch crates in the temp directory and run the
harness over them, and the harness's `cargo` runs inherit the test process's
environment. Every scratch crate is a package named `scratch` at 0.1.0, and
cargo names its artifacts without the package's path, so under one exported
target directory the concurrent tests' crates built over each other's
artifacts and fingerprints: after one run the shared directory held three
`scratch-<hash>` builds for all thirteen tests' crates. Reproduced here at
7cf1f036 by running the test binary with `CARGO_TARGET_DIR` set to a scratch
directory: 9 passed, 4 failed (`l3_fix_passes_the_same_gates`,
`l3_garbage_is_caught_by_the_gate_and_reverted`,
`shipped_l3_fix_induces_a_skill_that_closes_the_recurrence_without_a_model`,
`failed_gate_reverts_and_never_ships`).

The fix is in test builds only. Under `cfg(test)`, `Harness` carries the
workspace's own `target/`, and every `cargo` run it makes (`clippy`, `clippy
--fix`, `test`, `build`) is handed `--target-dir <workspace>/target` after
the subcommand, which outranks an inherited `CARGO_TARGET_DIR`: each scratch
crate builds where it does with no `CARGO_TARGET_DIR` at all. Production
builds have neither the field nor the flag; `joule-fix` passes cargo its
arguments as before, and the user's environment, `CARGO_TARGET_DIR`
included, decides where it builds.

`verified_fix_end_to_end_ships_with_receipt` now checks both halves: the
arguments a test harness hands cargo carry the flag, and after the run the
scratch crate's own `target/debug` exists.

13/13, three runs each, with and without `CARGO_TARGET_DIR` exported to a
scratch directory, which stayed empty; and `CARGO_TARGET_DIR` exported to
the workspace's own target directory around `cargo test -p
joule-code-remediate`, 13/13. Falsified: compiling the flag out again fails
five with `CARGO_TARGET_DIR` exported (the four above and the end-to-end
test) and one without it (the end-to-end test's argument check). `cargo
clippy --all-targets -p joule-code-remediate` is clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot added 1 commit 2026-09-30 23:00:57 -04:00

sandbox-exec-microvm: a CPU counter that held across a run is not a measured zero ... 3d70348a0d

MicroVmExecutor::cpu_energy sealed any `after >= before` as a HwShunt
reading. So a run whose vCPUs did work while the host's CPU energy counter
held sealed `measured_uj: Some(0)`, HwShunt: a zero dressed as a
measurement. That happens when a counter is frozen, or between the lumps
of a lump-refreshed one. sandbox-meter's Span::stop already refuses this
(a span whose CPU counter held is estimator-class), and the family rule is
that work that ran is never a measured zero.

The CPU figure is now a reading only if the counter advanced (`after >
before`). Otherwise it is the model estimate (vCPU-seconds x
uj_per_vcpu_second, ModelBased). The accelerator path is unchanged: a zero
accelerator delta can be an idle device. This matches sandbox-meter, which
keeps a GPU/NPU zero beside a CPU counter that advanced.

Test: a_cpu_counter_that_held_across_a_run_is_not_a_measured_zero. Held ->
ModelBased, measured None, 48,000 µJ estimated. Advanced -> HwShunt 3,000.
Restoring `a >= b` fails it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dcharlot referenced this pull request 2026-10-01 01:26:58 -04:00

jcp-gateway, joule-ui-serve: name the plaintext figure cost_uj #128

dcharlot merged commit c8c174609d into main 2026-10-01 05:45:12 -04:00

dcharlot referenced this pull request from a commit 2026-10-01 05:45:14 -04:00

Merge pull request 'sandbox-meter: a CPU counter that does not keep moving is never a measurement, and no consumer relabels an estimate' (#114) from sandbox-meter/cpu-rail-liveness into main

dcharlot referenced this pull request from a commit 2026-10-01 18:40:40 -04:00

ci: run joule-code-remediate's formal gate against a real z3

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Transaction-Science/open-standards!114

No description provided.